Change:
Needed for new DNS schema transition. was reverted in because of Need to add 'metatdata' to e2e names-to-resolve list for GCE
OK, so we have a real problem. if libc resolver has a limit of 6 search paths, and we intend to prepend at least 3 ( ) there's a very real possibility that a random machine on which someone wants to run kube will have more than 3 search domains. Even worse, what we really want is to prepend 4 (otherwise we have to do a release with which you MUST update your DNS server), and even GCE has 3 search paths. So the question is how to handle it? 1) Use less search paths ourselves. Force users to use more specific names or even FQDNs. Mitigates but does not eliminate the problem. 2) Log an error in kubelet, but soldier on: "some of your DNS search domains may get dropped". It really is "may" since we don't know what every resolver lib does. Need a way for people to fix this, e.g. a kubelet flag --dns-search-paths="," which trumps 3) Make kubelet hard-stop. Log an event, set a Condition, refuse to run. Still need a way to fix this as in (2). for consult.
I suggest the following strategy to move forward: checks for the number of search paths in host's and mark the node unhealthy if it has more than a maximum number of search paths. We can also add a kubelet flag to let users provide custom search paths for pods. being backwards compatible while introducing the new "svc" sub-domain.
For reference RHEL recommends running dnsmasq on systems affected by this, because dnsmasq isn't affected - links to some discussion about the ABI on it.
Could we drop '' for now from the search path? I think i'm less concerned about that one if it's unlikely people change it.
I like it. If we drop for now, we can keep compat and squeek by on GCE. We still need to solve the problem overall.
How about the following: kube dns pod to be upgraded as part of the next release (v0.18.0) kube dns create dns records with and without sub-domain (This is the current plan) kubelet to replace existing search paths with and in the next release (v0.19). Existing pods will not face any resolving issues as long as they don't use FQDNs or have their namespace set to . Pipelining the changes provides some time for users to update their pods if they use FQDNs. Does this work for Openshift , Tim Hockin wrote:
We already dropped the old paths for support, so what you've proposed is fine for us. ----- Original Message -----
raised a good point - Clayton's suggestion doesn't work until search paths are transitioned. We need to be sure: a) running pods can resolve "svc" and "" to "" b) new pods can resolve "svc" and "" to "" or "". depending on search paths Then we can transition search paths to So I think Vish is right, but minor nits: kube-dns release that supports both old and new names. users to upgrade kube-dns after 0.18 before 0.19. only new search paths in 0.19; running pods are fine, new pods are fine : Make kube-dns drop support for old names This will let us squeek by on GCE. In the mean time, we can consider how to make kubelet warn when it detects a problem, and how to mitigate said problem.
/cc
re: dnsmasq, I played with it and got it doing what we need wrt nameservers but NOT wrt search paths. As far as I can tell, it expects to pass queries on unmolested, and that the client has proper search paths. Is there some config I am not seeing that makes it apply search paths to incoming queries before forwarding them? Otherwise we're still stuck at 6.
Is the instruction I got internally. ----- Original Message -----
dnsmasq manpage specifically says it ignores everything but 'nameserver' directives from any file. That means it does not do search path expansion, which is exactly what I observe. DNS forwarding works but it only forwards the queries, not expand them.

Discussion:
history of clients that, on purpose or on accident, do not respect DNS TTLs (see previous remark about Pod IPs changing). <del> ## DNS Name format for Services Services get a DNS name with the format my-svc.my-namespace.svc.cluster.local 'svc' should not be used a namespace label to avoid conflicts. The old format of my-svc.my-namespace.cluster.local has been deprecated. </del> <ins> ## Supported DNS schema The following sections detail the supported record types and layout that is supported.  Any other layout or names or queries that happen to work are considered implementation details and are subject to change without warning. ### A records "Normal" (not headless) Services are assigned a DNS A record for a name of the form `my-svc.my-namespace.svc.cluster.local`.  This resolves to the cluster IP of the Service. "Headless" (without a cluster IP) Services are also assigned a DNS A record for a name of the form `my-svc.my-namespace.svc.cluster.local`.  Unlike normal Services, this resolves to the set of IPs of the pods selected by the Service. Clients are expected to consume the set or else use standard round-robin selection from the set. ### Backwards compatibility Previous versions of kube-dns made names of the for `my-svc.my-namespace.cluster.local` (the 'svc' level was added later).  For compatibility, kube-dns supports both names for the time being.  Users should avoid creating a namespace named 'svc', to avoid conflicts.  The old name format is deprecated and will be removed in a future release. </ins> ## How do I find the DNS server? The DNS server itself runs as a Kubernetes Service.  This gives it a stable IP

variables), pulls service info from the master, and writes that to etcd for SkyDNS to find. <ins> ## Inheriting DNS from the node When running a pod, kubelet will prepend the cluster DNS server and search paths to the node's own DNS settings.  If the node is able to resolve DNS names specific to the larger environment, pods should be able to, also.  See "Known issues" below for a caveat. </ins> ## Known issues Kubernetes installs do not configure the nodes' resolv.conf files to use the cluster DNS by default, because that process is inherently distro-specific. This should probably be implemented eventually. <ins> Linux's libc is impossibly stuck ([see this bug from 2005](https://bugzilla.redhat.com/show_bug.cgi?id=168253)) with limits of just 3 DNS `nameserver` records and 6 DNS `search` records.  Kubernetes needs to consume 1 `nameserver` record and 3 `search` records.  This means that if a local installation already uses 3 `nameserver`s or uses more than 3 `search`es, some of those settings will be lost.  As a partial workaround, the node can run `dnsmasq` which will provide more `nameserver` entries, but not more `search` entries. </ins> ## Making changes Please observe the release process for making changes to the `kube2sky` image that is documented in [RELEASES.md](kube2sky/RELEASES.md). Any significant changes

dns = append([]string{kl.clusterDNS.String()}, hostDNS...) } if kl.clusterDomain != "" { <del> nsDomain := fmt.Sprintf("%s.%s", pod.Namespace, kl.clusterDomain) dnsSearch = append([]string{nsDomain, kl.clusterDomain}, hostSearch...) </del> <ins> nsSvcDomain := fmt.Sprintf("%s.svc.%s", pod.Namespace, kl.clusterDomain) svcDomain := fmt.Sprintf("svc.%s", kl.clusterDomain) dnsSearch = append([]string{nsSvcDomain, svcDomain, kl.clusterDomain}, hostSearch...) </ins> } return dns, dnsSearch, nil }