Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ? 00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10

Discussion:
<ins> # 3.1 * The pause container gains a signal handler to clean up orphaned zombie processes. ([#36853](https://prs.k8s.io/36853), [@verb](https://github.com/verb)) * `pause -v` will return build information for the pause binary. ([#56762](https://prs.k8s.io/56762), [@verb](https://github.com/verb)) # 3.0 * The pause container was rewritten entirely in C. ([#23009](https://prs.k8s.io/23009), [@uluyol](https://github.com/uluyol)) </ins>

) const ( <del> validTmpl    = "image: {{ .ImageRepository }}/pause-{{ .Arch }}:3.0" validTmplOut = "image: gcr.io/google_containers/pause-amd64:3.0" doNothing    = "image: gcr.io/google_containers/pause-amd64:3.0" </del> <ins> validTmpl    = "image: {{ .ImageRepository }}/pause-{{ .Arch }}:3.1" validTmplOut = "image: gcr.io/google_containers/pause-amd64:3.1" doNothing    = "image: gcr.io/google_containers/pause-amd64:3.1" </ins> invalidTmpl1 = "{{ .baz }/d}" invalidTmpl2 = "{{ !foobar }}" )

) const ( <del> // When these values are updated, also update test/e2e/framework/util.go </del> <ins> // When these values are updated, also update test/utils/image/manifest.go </ins> defaultPodSandboxImageName    = "gcr.io/google_containers/pause" <del> defaultPodSandboxImageVersion = "3.0" </del> <ins> defaultPodSandboxImageVersion = "3.1" </ins> // From pkg/kubelet/rkt/rkt.go to avoid circular import defaultRktAPIServiceEndpoint = "localhost:15441" )

kube::test::get_object_assert pods "{{range.items}}{{$image_field}}:{{end}}" 'changed-with-yaml:' ## Patch pod from JSON can change image # Command <del> kubectl patch "${kube_flags[@]}" -f test/fixtures/doc-yaml/admin/limitrange/valid-pod.yaml -p='{"spec":{"containers":[{"name": "kubernetes-serve-hostname", "image": "gcr.io/google_containers/pause-amd64:3.0"}]}}' # Post-condition: valid-pod POD has image gcr.io/google_containers/pause-amd64:3.0 kube::test::get_object_assert pods "{{range.items}}{{$image_field}}:{{end}}" 'gcr.io/google_containers/pause-amd64:3.0:' </del> <ins> kubectl patch "${kube_flags[@]}" -f test/fixtures/doc-yaml/admin/limitrange/valid-pod.yaml -p='{"spec":{"containers":[{"name": "kubernetes-serve-hostname", "image": "gcr.io/google_containers/pause-amd64:3.1"}]}}' # Post-condition: valid-pod POD has image gcr.io/google_containers/pause-amd64:3.1 kube::test::get_object_assert pods "{{range.items}}{{$image_field}}:{{end}}" 'gcr.io/google_containers/pause-amd64:3.1:' </ins> ## If resourceVersion is specified in the patch, it will be treated as a precondition, i.e., if the resourceVersion is different from that is stored in the server, the Patch should be rejected ERROR_FILE="${KUBE_TEMP}/conflict-error"

"containers": [ { "name": "kubernetes-pause", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> } ], "restartPolicy": "Never",

) const ( <del> defaultSandboxImage = "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> defaultSandboxImage = "gcr.io/google_containers/pause-amd64:3.1" </ins> // Various default sandbox resources requests/limits. defaultSandboxCPUshares int64 = 2

"spec": { "containers": [{ "name": "test-container", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> }] } }

) const ( <del> // When these values are updated, also update cmd/kubelet/app/options/options.go // A copy of these values exist in e2e/framework/util.go. </del> <ins> // When these values are updated, also update cmd/kubelet/app/options/container_runtime.go // A copy of these values exist in test/utils/image/manifest.go </ins> currentPodInfraContainerImageName    = "gcr.io/google_containers/pause" <del> currentPodInfraContainerImageVersion = "3.0" </del> <ins> currentPodInfraContainerImageVersion = "3.1" </ins> ) // GetServerArchitecture fetches the architecture of the cluster's apiserver.

NoSnatTest               = ImageConfig{e2eRegistry, "no-snat-test", "1.0", true} NoSnatTestProxy          = ImageConfig{e2eRegistry, "no-snat-test-proxy", "1.0", true} NWayHTTP                 = ImageConfig{e2eRegistry, "n-way-http", "1.0", true} <del> // When these values are updated, also update cmd/kubelet/app/options/options.go Pause               = ImageConfig{gcRegistry, "pause", "3.0", true} </del> <ins> // When these values are updated, also update cmd/kubelet/app/options/container_runtime.go Pause               = ImageConfig{gcRegistry, "pause", "3.1", true} </ins> Porter              = ImageConfig{e2eRegistry, "porter", "1.0", true} PortForwardTester   = ImageConfig{e2eRegistry, "port-forward-tester", "1.0", true} Redis               = ImageConfig{e2eRegistry, "redis", "1.0", true}