Change:
func (p crashingExecProber) Probe(_ exec.Cmd) (probe.Result, string, error) { panic("Intentional Probe crash.") } <ins> func TestOnHoldOnLivenessCheckFailure(t *testing.T) { m := newTestManager() w := newTestWorker(m, liveness, api.Probe{SuccessThreshold: 1, FailureThreshold: 1}) status := getTestRunningStatus() m.statusManager.SetPodStatus(w.pod, getTestRunningStatus()) // First probe should fail. m.prober.exec = fakeExecProber{probe.Failure, nil} msg := "first probe" expectContinue(t, w, w.doProbe(), msg) expectResult(t, w, results.Failure, msg) if !w.onHold { t.Errorf("Prober should be on hold due to liveness check failure") } // Set fakeExecProber to return success. However, the result will remain // failure because the worker is on hold and won't probe. m.prober.exec = fakeExecProber{probe.Success, nil} msg = "while on hold" expectContinue(t, w, w.doProbe(), msg) expectResult(t, w, results.Failure, msg) if !w.onHold { t.Errorf("Prober should be on hold due to liveness check failure") } // Set a new container ID to lift the hold. The next probe will succeed. status.ContainerStatuses[0].ContainerID = "test://newCont_ID" m.statusManager.SetPodStatus(w.pod, status) msg = "hold lifted" expectContinue(t, w, w.doProbe(), msg) expectResult(t, w, results.Success, msg) if w.onHold { t.Errorf("Prober should not be on hold anymore") } } </ins>

Discussion:
Happened in GKE suite:
Does it need a liveness probe, maybe?
to delegate
Saw
Can you take a look at this one? Thanks!
FWIW, I saw some fishy messages in in (node: 104.197.48.84) It seems like docker was giving conflicting view of the container status, i.e., whether the container was running or not (and whether kubelet should kill it). kubelet's first attempt to kill the container : All the exec well after the grace period still shows that the container is the "Running: true" state () Later, kubelet tried killing the container again and failed because container does not exist. Note that after the failed kill also shows that the container was in the Running state. At the same time, cadvisor observed the container being stopped but then detected the container and then tried adding it again, but failed to find the /cc UPDATE: According to supervisord, docker was not killed during this time.
I didn't find result for the first Killing, does that really return? What is the log level, I remember that you told me before the log level in GKE is 2?
AFAICT, the pod worker may have never returned because the specific pod worker didn't log anything beyond that point. I was looking at (docker stop hangs) to see if it's relevant. didn't print any error though.
Yeah, the last SyncPod log was which is before the first Killing. Then the pod worker was stuck in
This really looks similar with I tried to reproduce it following the comment. The container was not actually killed. Docker stop hangs. Docker ps shows the container is still running. But the difference is that when I tried to kill it with another , it still hung there.
I was able to reproduce this quite easily by running 30 pods with the same spec in the test and with :
From
My theory is that during may cause docker to leave the container in a unrecoverable state (yep, a bug). I can consistently reproduce this by simply creating (which always fails), and leave the cluster for < 10 min. The restart counts of some of the pods will stop increasing after sometime, and after ssh'ing in to the node, one can easily confirm the same symptoms posted above. If pods use the http probe, they do not have this issue.

For the workaround, we can instruct the probe workers to stop probing after detecting a liveness check failure, until the container has been restarted (with a new container ID). Note that this is just a workaround, as it doesn't stop other components or users to run during this window. I've verified that with this fix, there are no zombie docker containers anymore. Will upload a PR.
FWIW, I also tested it with docker 1.8.3, which has the same problem.
Thanks !