Repository: juanfont/headscale

Issue:
[Feature] Adding an option to prevent tailscale clients adding `~.` to its DNS search domains for systemd-resolved.
### Use case

systemd-resolved works as a multicast DNS server, which means by default when it queries domains, it will try all available providers, and use the fastest result, unless the domain matches one of DNS search domains of one provider, then it will only use this provider to query.

Tailscale by default registers itself as one of providers of systemd-resolved, but it adds `~.` to its DNS search domains, this means all domains will match, and makes systemd-resolved's multicast invalid, all DNS queries go to tailscale's DNS. User may only wants to query tailscale domains via tailscale.

### Description

Setting `tailcfg.DNSConfig.FallbackResolvers` instead of `tailcfg.DNSConfig.Resolvers` can prevent tailscale clients from adding `~.`. In previous version this behavior is controlled by `override_local_dns`, but this option is removed because it does not work as intended, and then we lose a way to control this behavior.

Is there any chance to add an option to control setting `tailcfg.DNSConfig.FallbackResolvers` instead of `tailcfg.DNSConfig.Resolvers`?

### Contribution

- [ ] I can write the design doc for this feature
- [X] I can contribute this feature

### How can it be implemented?

Just bring back <https://github.com/juanfont/headscale/pull/905/files#diff-0e426a43248661127a0c0ee115aef7a1093b635f8993b3f7ebb1dd9f05b8f249R406-R410>, but with a better option to describe what it does?

Hints from issue comments:
@nblock: The Tailscale SaaS behaves as follows when changing the setting [Override local DNS](https://tailscale.com/kb/1054/dns#override-local-dns):

When enabled in the control plane, Tailscale configures systemd-resolved as default route (`+DefaultRoute`) and also
configures a route-only domain `~.` so all domains will match and Tailscale will receive all DNS requests.

```shell
$ resolvectl status tailscale0
Link 4 (tailscale0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.100.100.100
       DNS Servers: 100.100.100.100
        DNS Domain: taile8db11.ts.net ~.
     Default Route: yes
```

When disabled in the control plane, Tailscale will not set the default route (`-DefaultRoute`) and the route-only domain
is not present in the list of DNS domains. It will only handle requests that match the list of domains.

```shell
$ resolvectl status tailscale0
Link 4 (tailscale0)
    Current Scopes: DNS
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.100.100.100
       DNS Servers: 100.100.100.100
        DNS Domain: taile8db11.ts.net ~0.e.1.a.c.5.1.1.a.7.d.f.ip6.arpa ~100.100.in-addr.arpa ~101.100.in-addr.arpa ~102.100.in-addr.arpa ~103.100.in-addr.arpa ~104.100.in-addr.arpa ~105.100.in-addr.arpa ~106.100.in-addr.arpa
                    ~107.100.in-addr.arpa ~108.100.in-addr.arpa ~109.100.in-addr.arpa ~110.100.in-addr.arpa ~111.100.in-addr.arpa ~112.100.in-addr.arpa ~113.100.in-addr.arpa ~114.100.in-addr.arpa ~115.100.in-addr.arpa ~116.100.in-addr.arpa
                    ~117.100.in-addr.arpa ~118.100.in-addr.arpa ~119.100.in-addr.arpa ~120.100.in-addr.arpa ~121.100.in-addr.arpa ~122.100.in-addr.arpa ~123.100.in-addr.arpa ~124.100.in-addr.arpa ~125.100.in-addr.arpa ~126.100.in-addr.arpa
                    ~127.100.in-addr.arpa ~64.100.in-addr.arpa ~65.100.in-addr.arpa ~66.100.in-addr.arpa ~67.100.in-addr.arpa ~68.100.in-addr.arpa ~69.100.in-addr.arpa ~70.100.in-addr.arpa ~71.100.in-addr.arpa ~72.100.in-addr.arpa
                    ~73.100.in-addr.arpa ~74.100.in-addr.arpa ~75.100.in-addr.arpa ~76.100.in-addr.arpa ~77.100.in-addr.arpa ~78.100.in-addr.arpa ~79.100.in-addr.arpa ~80.100.in-addr.arpa ~81.100.in-addr.arpa ~82.100.in-addr.arpa
                    ~83.100.in-addr.arpa ~84.100.in-addr.arpa ~85.100.in-addr.arpa ~86.100.in-addr.arpa ~87.100.in-addr.arpa ~88.100.in-addr.arpa ~89.100.in-addr.arpa ~90.100.in-addr.arpa ~91.100.in-addr.arpa ~92.100.in-addr.arpa
                    ~93.100.in-addr.arpa ~94.100.in-addr.arpa ~95.100.in-addr.arpa ~96.100.in-addr.arpa ~97.100.in-addr.arpa ~98.100.in-addr.arpa ~99.100.in-addr.arpa ~ts.net
     Default Route: no
```

---

As a workaround, one may configure domains and the default route explicitly on the `tailscale0` link and "undo" the DNS override:

```
resolvectl domain tailscale0 tn.example.com
resolvectl default-route tailscale0 off
```

Likely this will only last until the next network change where tailscale gets a chance to reconfigure DNS.
@st0nie: Another workaround is to set headscale's global dns to empty, in which case the system will not use tailscale's dns to resolve other domains even if `tailscale --accept-dns`. But the disadvantage of this workaround is that if the linux system does not use `systemd-resolved` but uses `100.100.100.100` as the dns, the system won't be able to resolve domains except headscale domain (because the upstream dns is not set).@Toomoch: This is a big problem, as I can't resolve local domains served by my router's DNS server (.lan) and tailscale names at the same time. It's one or the other with `--accept-dns`. Interestingly, in Android everything works correctly. @Devristo: This has hit me in the head a few times as well. I think it also breaks DNS resolution inside my docker containers / k3s containers from time to time, but have not been able to create a clear reproducer. However, `sude tailscale set --accept-dns=false` fixes everything.@VolatileDream: I recently ran into this issue as a result of noticing that mDNS was broken only on Headscale connected Windows machines (which might be a bug, and the behaviour is tracked in https://github.com/tailscale/tailscale/issues/15093). And then noticed that DNS resolution was subtly wrong across a bunch of other devices that connected to Headscale.

I'll lend a perhaps contradictory opinion to this feature request: I **don't** mind if there's not a setting for this, I think the default should be to use `tailcfg.DNSConfig.FallbackResolvers` (equivalent to `override_local_dns=false`) and would be fine if Headscale folks decided that for simplicity sake they wouldn't add the feature.

I have worked around the current behaviour by using the config that follows, since my machines are configured to use something other than `100.100.100.100` for DNS.
```
dns:
  // Setting is needed to enable Tailscale to do DNS resolution.
  magic_dns: true
  // Since Tailscale uses two lists instead of an override setting,
  // passing an empty tailcfg.DNSConfig.Resolvers causes clients
  // to not override system DNS.
  nameservers:
    global: []
```

edit: I **don't** mind if there's no setting. 🤦 I dropped the "don't" during editing.@nblock: > I'll lend a perhaps contradictory opinion to this feature request: I mind if there's not a setting for this, I think the default should be to use `tailcfg.DNSConfig.FallbackResolvers` (equivalent to `override_local_dns=false`) and would be fine if Headscale folks decided that for simplicity sake they wouldn't add the feature.

I agree, PR https://github.com/juanfont/headscale/pull/2438 also changes the default to align with Tailscale's default behavior.

Solution patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e4c0fd81..f39c3a2b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -92,6 +92,8 @@ The new policy can be used by setting the environment variable
 - node FQDNs in the netmap will now contain a dot (".") at the end. This aligns
   with behaviour of tailscale.com
   [#2503](https://github.com/juanfont/headscale/pull/2503)
+- Restore support for "Override local DNS"
+  [#2438](https://github.com/juanfont/headscale/pull/2438)
 
 ## 0.25.1 (2025-02-25)
 
diff --git a/config-example.yaml b/config-example.yaml
index 9d6b82d6..edd0586d 100644
--- a/config-example.yaml
+++ b/config-example.yaml
@@ -270,6 +270,10 @@ dns:
   # `hostname.base_domain` (e.g., _myhost.example.com_).
   base_domain: example.com
 
+  # Whether to use the local DNS settings of a node (default) or override the
+  # local DNS settings and force the use of Headscale's DNS configuration.
+  override_local_dns: false
+
   # List of DNS servers to expose to clients.
   nameservers:
     global:
diff --git a/hscontrol/types/config.go b/hscontrol/types/config.go
index 0b69a1a4..588d6a71 100644
--- a/hscontrol/types/config.go
+++ b/hscontrol/types/config.go
@@ -102,6 +102,7 @@ type Config struct {
 type DNSConfig struct {
 	MagicDNS         bool   `mapstructure:"magic_dns"`
 	BaseDomain       string `mapstructure:"base_domain"`
+	OverrideLocalDNS bool   `mapstructure:"override_local_dns"`
 	Nameservers      Nameservers
 	SearchDomains    []string            `mapstructure:"search_domains"`
 	ExtraRecords     []tailcfg.DNSRecord `mapstructure:"extra_records"`
@@ -287,6 +288,7 @@ func LoadConfig(path string, isFile bool) error {
 
 	viper.SetDefault("dns.magic_dns", true)
 	viper.SetDefault("dns.base_domain", "")
+	viper.SetDefault("dns.override_local_dns", true)
 	viper.SetDefault("dns.nameservers.global", []string{})
 	viper.SetDefault("dns.nameservers.split", map[string]string{})
 	viper.SetDefault("dns.search_domains", []string{})
@@ -351,9 +353,9 @@ func validateServerConfig() error {
 	depr.fatalIfNewKeyIsNotUsed("policy.path", "acl_policy_path")
 
 	// Move dns_config -> dns
-	depr.warn("dns_config.override_local_dns")
 	depr.fatalIfNewKeyIsNotUsed("dns.magic_dns", "dns_config.magic_dns")
 	depr.fatalIfNewKeyIsNotUsed("dns.base_domain", "dns_config.base_domain")
+	depr.fatalIfNewKeyIsNotUsed("dns.override_local_dns", "dns_config.override_local_dns")
 	depr.fatalIfNewKeyIsNotUsed("dns.nameservers.global", "dns_config.nameservers")
 	depr.fatalIfNewKeyIsNotUsed("dns.nameservers.split", "dns_config.restricted_nameservers")
 	depr.fatalIfNewKeyIsNotUsed("dns.search_domains", "dns_config.domains")
@@ -417,6 +419,12 @@ func validateServerConfig() error {
 		)
 	}
 
+	if viper.GetBool("dns.override_local_dns") {
+		if global := viper.GetStringSlice("dns.nameservers.global"); len(global) == 0 {
+			errorText += "Fatal config error: dns.nameservers.global must be set when dns.override_local_dns is true\n"
+		}
+	}
+
 	if errorText != "" {
 		// nolint
 		return errors.New(strings.TrimSuffix(errorText, "\n"))
@@ -616,6 +624,7 @@ func dns() (DNSConfig, error) {
 
 	dns.MagicDNS = viper.GetBool("dns.magic_dns")
 	dns.BaseDomain = viper.GetString("dns.base_domain")
+	dns.OverrideLocalDNS = viper.GetBool("dns.override_local_dns")
 	dns.Nameservers.Global = viper.GetStringSlice("dns.nameservers.global")
 	dns.Nameservers.Split = viper.GetStringMapStringSlice("dns.nameservers.split")
 	dns.SearchDomains = viper.GetStringSlice("dns.search_domains")
@@ -721,7 +730,11 @@ func dnsToTailcfgDNS(dns DNSConfig) *tailcfg.DNSConfig {
 
 	cfg.Proxied = dns.MagicDNS
 	cfg.ExtraRecords = dns.ExtraRecords
-	cfg.Resolvers = dns.globalResolvers()
+	if dns.OverrideLocalDNS {
+		cfg.Resolvers = dns.globalResolvers()
+	} else {
+		cfg.FallbackResolvers = dns.globalResolvers()
+	}
 
 	routes := dns.splitResolvers()
 	cfg.Routes = routes
diff --git a/integration/hsic/config.go b/integration/hsic/config.go
index 256fbd76..297cbd9f 100644
--- a/integration/hsic/config.go
+++ b/integration/hsic/config.go
@@ -23,6 +23,7 @@ func DefaultConfigEnv() map[string]string {
 		"HEADSCALE_PREFIXES_V6":                       "fd7a:115c:a1e0::/48",
 		"HEADSCALE_DNS_BASE_DOMAIN":                   "headscale.net",
 		"HEADSCALE_DNS_MAGIC_DNS":                     "true",
+		"HEADSCALE_DNS_OVERRIDE_LOCAL_DNS":            "false",
 		"HEADSCALE_DNS_NAMESERVERS_GLOBAL":            "127.0.0.11 1.1.1.1",
 		"HEADSCALE_PRIVATE_KEY_PATH":                  "/tmp/private.key",
 		"HEADSCALE_NOISE_PRIVATE_KEY_PATH":            "/tmp/noise_private.key",



Test patch:
diff --git a/hscontrol/types/config_test.go b/hscontrol/types/config_test.go
index 511528df..e7afee69 100644
--- a/hscontrol/types/config_test.go
+++ b/hscontrol/types/config_test.go
@@ -7,6 +7,7 @@ import (
 	"testing"
 
 	"github.com/google/go-cmp/cmp"
+	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/spf13/viper"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -34,8 +35,9 @@ func TestReadConfig(t *testing.T) {
 				return dns, nil
 			},
 			want: DNSConfig{
-				MagicDNS:   true,
-				BaseDomain: "example.com",
+				MagicDNS:         true,
+				BaseDomain:       "example.com",
+				OverrideLocalDNS: false,
 				Nameservers: Nameservers{
 					Global: []string{
 						"1.1.1.1",
@@ -70,7 +72,7 @@ func TestReadConfig(t *testing.T) {
 			want: &tailcfg.DNSConfig{
 				Proxied: true,
 				Domains: []string{"example.com", "test.com", "bar.com"},
-				Resolvers: []*dnstype.Resolver{
+				FallbackResolvers: []*dnstype.Resolver{
 					{Addr: "1.1.1.1"},
 					{Addr: "1.0.0.1"},
 					{Addr: "2606:4700:4700::1111"},
@@ -99,8 +101,9 @@ func TestReadConfig(t *testing.T) {
 				return dns, nil
 			},
 			want: DNSConfig{
-				MagicDNS:   false,
-				BaseDomain: "example.com",
+				MagicDNS:         false,
+				BaseDomain:       "example.com",
+				OverrideLocalDNS: false,
 				Nameservers: Nameservers{
 					Global: []string{
 						"1.1.1.1",
@@ -135,7 +138,7 @@ func TestReadConfig(t *testing.T) {
 			want: &tailcfg.DNSConfig{
 				Proxied: false,
 				Domains: []string{"example.com", "test.com", "bar.com"},
-				Resolvers: []*dnstype.Resolver{
+				FallbackResolvers: []*dnstype.Resolver{
 					{Addr: "1.1.1.1"},
 					{Addr: "1.0.0.1"},
 					{Addr: "2606:4700:4700::1111"},
@@ -181,6 +184,40 @@ func TestReadConfig(t *testing.T) {
 			},
 			wantErr: "",
 		},
+		{
+			name:       "dns-override-true-errors",
+			configPath: "testdata/dns-override-true-error.yaml",
+			setup: func(t *testing.T) (any, error) {
+				return LoadServerConfig()
+			},
+			wantErr: "Fatal config error: dns.nameservers.global must be set when dns.override_local_dns is true",
+		},
+		{
+			name:       "dns-override-true",
+			configPath: "testdata/dns-override-true.yaml",
+			setup: func(t *testing.T) (any, error) {
+				_, err := LoadServerConfig()
+				if err != nil {
+					return nil, err
+				}
+
+				dns, err := dns()
+				if err != nil {
+					return nil, err
+				}
+
+				return dnsToTailcfgDNS(dns), nil
+			},
+			want: &tailcfg.DNSConfig{
+				Proxied: true,
+				Domains: []string{"derp2.no"},
+				Routes:  map[string][]*dnstype.Resolver{},
+				Resolvers: []*dnstype.Resolver{
+					{Addr: "1.1.1.1"},
+					{Addr: "1.0.0.1"},
+				},
+			},
+		},
 		{
 			name:       "policy-path-is-loaded",
 			configPath: "testdata/policy-path-is-loaded.yaml",
@@ -254,6 +291,7 @@ func TestReadConfigFromEnv(t *testing.T) {
 			configEnv: map[string]string{
 				"HEADSCALE_DNS_MAGIC_DNS":          "true",
 				"HEADSCALE_DNS_BASE_DOMAIN":        "example.com",
+				"HEADSCALE_DNS_OVERRIDE_LOCAL_DNS": "false",
 				"HEADSCALE_DNS_NAMESERVERS_GLOBAL": `1.1.1.1 8.8.8.8`,
 				"HEADSCALE_DNS_SEARCH_DOMAINS":     "test.com bar.com",
 
@@ -272,8 +310,9 @@ func TestReadConfigFromEnv(t *testing.T) {
 				return dns, nil
 			},
 			want: DNSConfig{
-				MagicDNS:   true,
-				BaseDomain: "example.com",
+				MagicDNS:         true,
+				BaseDomain:       "example.com",
+				OverrideLocalDNS: false,
 				Nameservers: Nameservers{
 					Global: []string{"1.1.1.1", "8.8.8.8"},
 					Split:  map[string][]string{
@@ -301,7 +340,7 @@ func TestReadConfigFromEnv(t *testing.T) {
 			conf, err := tt.setup(t)
 			require.NoError(t, err)
 
-			if diff := cmp.Diff(tt.want, conf); diff != "" {
+			if diff := cmp.Diff(tt.want, conf, cmpopts.EquateEmpty()); diff != "" {
 				t.Errorf("ReadConfig() mismatch (-want +got):\n%s", diff)
 			}
 		})
diff --git a/hscontrol/types/testdata/base-domain-in-server-url.yaml b/hscontrol/types/testdata/base-domain-in-server-url.yaml
index 401f2a49..10a0b82a 100644
--- a/hscontrol/types/testdata/base-domain-in-server-url.yaml
+++ b/hscontrol/types/testdata/base-domain-in-server-url.yaml
@@ -13,3 +13,4 @@ server_url: "https://server.derp.no"
 dns:
   magic_dns: true
   base_domain: derp.no
+  override_local_dns: false
diff --git a/hscontrol/types/testdata/base-domain-not-in-server-url.yaml b/hscontrol/types/testdata/base-domain-not-in-server-url.yaml
index 80b4a08f..e78cd6f8 100644
--- a/hscontrol/types/testdata/base-domain-not-in-server-url.yaml
+++ b/hscontrol/types/testdata/base-domain-not-in-server-url.yaml
@@ -13,3 +13,4 @@ server_url: "https://derp.no"
 dns:
   magic_dns: true
   base_domain: clients.derp.no
+  override_local_dns: false
diff --git a/hscontrol/types/testdata/dns-override-true-error.yaml b/hscontrol/types/testdata/dns-override-true-error.yaml
new file mode 100644
index 00000000..c11e2fca
--- /dev/null
+++ b/hscontrol/types/testdata/dns-override-true-error.yaml
@@ -0,0 +1,16 @@
+noise:
+  private_key_path: "private_key.pem"
+
+prefixes:
+  v6: fd7a:115c:a1e0::/48
+  v4: 100.64.0.0/10
+
+database:
+  type: sqlite3
+
+server_url: "https://server.derp.no"
+
+dns:
+  magic_dns: true
+  base_domain: derp.no
+  override_local_dns: true
diff --git a/hscontrol/types/testdata/dns-override-true.yaml b/hscontrol/types/testdata/dns-override-true.yaml
new file mode 100644
index 00000000..359cea56
--- /dev/null
+++ b/hscontrol/types/testdata/dns-override-true.yaml
@@ -0,0 +1,20 @@
+noise:
+  private_key_path: "private_key.pem"
+
+prefixes:
+  v6: fd7a:115c:a1e0::/48
+  v4: 100.64.0.0/10
+
+database:
+  type: sqlite3
+
+server_url: "https://server.derp.no"
+
+dns:
+  magic_dns: true
+  base_domain: derp2.no
+  override_local_dns: true
+  nameservers:
+    global:
+      - 1.1.1.1
+      - 1.0.0.1
diff --git a/hscontrol/types/testdata/dns_full.yaml b/hscontrol/types/testdata/dns_full.yaml
index 62bbd3ab..d27e0fee 100644
--- a/hscontrol/types/testdata/dns_full.yaml
+++ b/hscontrol/types/testdata/dns_full.yaml
@@ -7,6 +7,7 @@ dns:
   magic_dns: true
   base_domain: example.com
 
+  override_local_dns: false
   nameservers:
     global:
       - 1.1.1.1
diff --git a/hscontrol/types/testdata/dns_full_no_magic.yaml b/hscontrol/types/testdata/dns_full_no_magic.yaml
index 2f35c3db..4fb25d65 100644
--- a/hscontrol/types/testdata/dns_full_no_magic.yaml
+++ b/hscontrol/types/testdata/dns_full_no_magic.yaml
@@ -7,6 +7,7 @@ dns:
   magic_dns: false
   base_domain: example.com
 
+  override_local_dns: false
   nameservers:
     global:
       - 1.1.1.1
diff --git a/hscontrol/types/testdata/policy-path-is-loaded.yaml b/hscontrol/types/testdata/policy-path-is-loaded.yaml
index da0d29cd..94f60b74 100644
--- a/hscontrol/types/testdata/policy-path-is-loaded.yaml
+++ b/hscontrol/types/testdata/policy-path-is-loaded.yaml
@@ -15,4 +15,6 @@ policy:
   type: file
   path: "/etc/policy.hujson"
 
-dns.magic_dns: false
+dns:
+  magic_dns: false
+  override_local_dns: false