Code change:
This one involves systemd, DNS, musl, and Kubelet v1.25 Starting in v1.25.0, pods running on Kubelets on certain nodes with a FQDN hostname (more on that later) will not be able to resolve DNS queries if they're using musl (e.g. alpine:3.16.2, busybox, etc). Our story starts with systemd. Back in 2020, systemd a "search ." in stub resolvers if it detected the hostname to be a FQDN (to solve a separate problem). Check for a static hostname with . In , a change was made to Kubelet to propagate that "search ." into pods (to solve a separate DNS problem). That change landed in and appears in . Unfortunatey, musl based DNS resolvers struggle with this "search ." EDIT: details in musl-based DNS resolvers (depite their other quirks) should continue to function To see if you have a suitable repro node, check your (likely a symlink to . Next, start a container (any CNI) that makes some use of musl resolvers. Confirm that starting in Kubelet v1.25.0, the container now includes the "search ." if it was present on the host. Try using musl-based DNS resolution. Note that uses its own DNS resolver, its not a reproducer. Note also that pod-to-pod traffic and outbound masquerade are fine too. This is not a masquerade problem. Edit the in the container to remove the "." and your musl problems instantly disappear. Add the dot and they're back. I've been spelunking for a while. I can confirm this is not related to a specific CNI - flannel, Calico, and Cilium all reproduce the issue. Not networking related OpenShift is not affected. In 2021, RedHat folks noticed DNS "breaks" and reverted the systemd change on their OS cc <details</details<detailsThis isn't cloud provider specific. However, the most likely cases where folks use FQDN's are on-prem imo. So you're likely to see reports originating from there. </details<details</details<detailsThis was found while preparing Typhoon's v1.25.0 release of Kubernetes v1.25.0 </details<detailscontainerd 1.6.6 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Could either revert (which propagates "search ." into containers) or find a different solution to the empty "search " problem it was trying to solve (does OpenShift even need that fix if they already ? cc A workaournd to problem is to configure hosts to strip "search ." from ala
has a great packet capture and this has been reported to musl
/sig network /cc
impressive investigation in the linked issue , that seems to conclude that is a muslc bug, since it happens outside of Kubernetes too. Just reverting the PR in Kubernetes doesn't looks right, it fixed a known bug ... and this is really a muslc bug .... maybe always removing the single dot from ? I think that for Kubernetes we should understand first what does and means and its implications, maybe some developer can help? you know someone?
Its discussed by the musl folks what means. It does not sound trivial to implement on the musl side, its a new case they did not anticipate supporting. Unfortunately this is subtle, and folks using alpine or related ecosystem things will get bitten and eventually end up here. I agree its a musl issue, but Kubelet now propagating triggers this I think its worth noting that did not set out to propagate , it was merely trying to prevent an empty . There may be better ways to achive that.
IIUC means "try the input name without any suffix". Resolvers will do this ANYWAY, but it's implementation-defined whether they do it first or last or in parallel, so allowing lets node owners be explicit (though, curiously, it seems at least glibc may try the same name AGAIN at the end of the search list). This is pretty clearly a bug in MUSL, but I wonder if, instead of , we should have treated as rather than or the errant ? It nullifies the control that provides, but it wouldn't tickle musl like this.
/cc I foresee a new wave of bug reports caused because of this muslc bug , better to keep CoreDNS friends in the loop
The patch for this got merged for the upcoming 1.25.1 and 1.26.

Related discussion:
text: {"nameserver t 1.2.3.4", []string{"1.2.3.4"}, []string{}, []string{}, false}, {"nameserver 1.2.3.4nnameserver 5.6.7.8", []string{"1.2.3.4", "5.6.7.8"}, []string{}, []string{}, false}, {"nameserver 1.2.3.4 #comment", []string{"1.2.3.4"}, []string{}, []string{}, false}, <del> {"search ", []string{}, []string{}, []string{}, false}, // search empty {"search .", []string{}, []string{"."}, []string{}, false}, </del> <ins> {"search ", []string{}, []string{}, []string{}, false},  // search empty {"search .", []string{}, []string{}, []string{}, false}, // ignore lone dot {"search . foo", []string{}, []string{"foo"}, []string{}, false}, {"search foo .", []string{}, []string{"foo"}, []string{}, false}, {"search foo .  bar", []string{}, []string{"foo", "bar"}, []string{}, false}, </ins> {"search foo", []string{}, []string{"foo"}, []string{}, false}, {"search foo bar", []string{}, []string{"foo", "bar"}, []string{}, false}, {"search foo. bar", []string{}, []string{"foo", "bar"}, []string{}, false},