Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: My broke after , after much debugging we discovered that the API server could not reach itself on , further debugging showed that DNS lookups for localhost were occuring. Long story short, while we have entries for localhost, the api server was not respecting them. What you expected to happen: our standard control plane deployments should not lookup localhost via dns. they should respect /etc/hosts How to reproduce it (as minimally and precisely as possible): create a cluster and monitor network traffic. then try again after adding a minimal to the api server pod. Anything else we need to know?: Prior art: This happens because we build with go's build in DNS resolution, which respects glibc's , but we build images based on busybox for some components (like the apiserver) which does not ship with one. Environment: kubernetes using Kubernetes version (use ): /sig release
/assign cc Thanks a million to and for looking into this with me! this was tricky to resolve.
We should be able to fix this by just adding a minimal to the images make go's resolver happy. alternatively we can standardize on eg which already ships one
oh wow. tough one to debug :)
/milestone v1.13 /priority important-soon
I have a PR under way, will file after lunch!
For now adding the , but discussing with and it sounds like switching to debian-base everywhere is the best long term answer. It will actually save a mb or so on each node :sweat_smile: ... and give a more consistent base for component images to build on.
I just got bit by this.
This issue will bite anyone deploying a statically linked go binary, not just the api server. Can we get kubelet to slide in a default that is sane so it wont keep biting people? Similar to
We'd need a feature request to sig-node and probably sig-network. I do like this idea, but, is used for things other than networking so injecting one might be slightly problematic. In this case it was fine because we control the images and the binaries. People deploying go images need to make sure they have a sane for now, or not deploy staticly linked binaries, unfortunately.
For 1.13, we should just document the problem
would it be difficult to add the file only if it doesn't exist in the image?
workaround for 1.12: echo hosts: files dns myhostname and add to the ClusterConfig:
Yeah that work around for now, I'm not sure how difficult / expensive it would be for the kubelet to conditionally add this, and it might be a surprising behavior change for arbitrary containers / pods.
it may be surprising, but its very surprising now too, that dns takes precident to hosts file unexpectedly. :( I'd rather it be secure by default and surprise them that its safe. I didn't even know glibc had this backwards behavior when was missing. I haven't run up against a glibc based system that didn't have an nsswitch file. All distro's add one.
hmm... etcd is crashing too.... I hope there is an equiv of etcdExtraVolumes....
never mind on the etcd thing that was something else. The above yaml tweaks did solve the issue.
We have just wasted a TON of time on this when a misconfigured Samsung TV on the other side of the world, plus lack of rigor or our DDNS policy broke our Kube 1.12 clusters. I really don't like that these control plane images are hitting DNS for every lookup of localhost. I am very keen to see implemented.
The control plane images in future releases already contain , which could be cherry picked back. Doing this in an automated fashion like that issue would require bringing a feature to sig-network and sig-node for discussion. I suspect supplying a file only if it doesn't exist in the image would be somewhat expensive compared to what we do currently, and it's definitely a breaking change. In the meantime, if you're containerizing go and not using CGO for networking, you need to add this to your own images. On Tue, Nov 20, 2018, 10:00 nickperry wrote:
I would argue, that 1.12 was a broken release. 1.11 and below did not have the issue. As K8S is maturing, some of us, especially in more enterprise like environments, are not upgrading as fast, so the issues are being found slower. I think backporting the fix for the control plane images is appropriate to resolve the regression until a more universal fix can be put in place.
that sounds like a great WHODUNIT war story :)
generally I'm not upgrading production workloads particularly quickly either :-) We can file cherry picks back to 1.12, but lack of an in the control plane images is not new: EG does not contain an . IMO, setting on control plane pods was certainly broken, which I questioned, but not just for this reason, and that change was eventually reverted IIRC. /cc Regarding -- because Kubernetes wants to be stable there are processes for proposing and vetting features prior to adding them, particularly for user facing changes! If you'd like to see this, please discuss it with sig-network and sig-node. I don't personally have the bandwidth for this right now and I'm not even sure that it is even the right solution.
yes the change was reverted. it also spawned a little bit of a discussion here for a rather complicated workaround: given this affects images that are already deployed at GCR, it's unlikely that it can be cherry picked. the images do support revisions, but this would also mean porting changes to all the bundled tools that use them as hardcoded tags. also this means revisions for all patch release images. overall i see this as something that will not be approved by sig-network and the GCR maintainers.
I guess to prevent a future regression, we probably need a test that inserts a <some other addressrecord somewhere during the test.

Discussion:
package(default_visibility = ["//visibility:public"]) load("@io_bazel_rules_docker//docker:docker.bzl", "docker_build", "docker_bundle") <ins> load("@io_bazel_rules_docker//container:container.bzl", "container_image") </ins> load("@io_kubernetes_build//defs:build.bzl", "release_filegroup") filegroup(

tags = ["automanaged"], ) <ins> # ensure /etc/nsswitch.conf exists so go's resolver respects /etc/hosts container_image( name = "busybox-with-nsswitch", base = "@official_busybox//image", directory = "/etc", files = ["nsswitch.conf"], mode = "0644", ) </ins> # This list should roughly match kube::build::get_docker_wrapped_binaries() # in build/common.sh. DOCKERIZED_BINARIES = { "cloud-controller-manager": { <del> "base": "@official_busybox//image", </del> <ins> "base": ":busybox-with-nsswitch", </ins> "target": "//cmd/cloud-controller-manager:cloud-controller-manager", }, "kube-apiserver": { <del> "base": "@official_busybox//image", </del> <ins> "base": ":busybox-with-nsswitch", </ins> "target": "//cmd/kube-apiserver:kube-apiserver", }, "kube-controller-manager": { <del> "base": "@official_busybox//image", </del> <ins> "base": ":busybox-with-nsswitch", </ins> "target": "//cmd/kube-controller-manager:kube-controller-manager", }, "kube-scheduler": { <del> "base": "@official_busybox//image", </del> <ins> "base": ":busybox-with-nsswitch", </ins> "target": "//cmd/kube-scheduler:kube-scheduler", }, "kube-proxy": {

rm -rf "${docker_build_path}" mkdir -p "${docker_build_path}" ln "${binary_dir}/${binary_name}" "${docker_build_path}/${binary_name}" <del> printf " FROM ${base_image} n ADD ${binary_name} /usr/local/bin/${binary_name}n" > "${docker_file_path}" </del> <ins> ln "${KUBE_ROOT}/build/nsswitch.conf" "${docker_build_path}/nsswitch.conf" chmod 0644 "${docker_build_path}/nsswitch.conf" cat <<EOF > "${docker_file_path}" FROM ${base_image} COPY ${binary_name} /usr/local/bin/${binary_name} EOF # ensure /etc/nsswitch.conf exists so go's resolver respects /etc/hosts if [[ "${base_image}" =~ busybox ]]; then echo "COPY nsswitch.conf /etc/" >> "${docker_file_path}" fi </ins> "${DOCKER[@]}" build --pull -q -t "${docker_image_tag}" "${docker_build_path}" >/dev/null "${DOCKER[@]}" tag "${docker_image_tag}" "${deprecated_image_tag}" >/dev/null "${DOCKER[@]}" save "${docker_image_tag}" "${deprecated_image_tag}" > "${binary_dir}/${binary_name}.tar"

<ins> # ensure go's non-cgo resolver respects /etc/hosts hosts: files dns </ins>