Change:
What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated. We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.
Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:

Discussion:
w.pod.Spec.RestartPolicy != v1.RestartPolicyNever } <ins> // Graceful shutdown of the pod. if w.pod.ObjectMeta.DeletionTimestamp != nil && (w.probeType == liveness || w.probeType == startup) { klog.V(3).Infof("Pod deletion requested, setting %v probe result to success: %v - %v", w.probeType.String(), format.Pod(w.pod), w.container.Name) if w.probeType == startup { klog.Warningf("Pod deletion requested before container has fully started: %v - %v", format.Pod(w.pod), w.container.Name) } // Set a last result to ensure quiet shutdown. w.resultsManager.Set(w.containerID, results.Success, w.pod) // Stop probing at this point. return false } </ins> // Probe disabled for InitialDelaySeconds. if int32(time.Since(c.State.Running.StartedAt.Time).Seconds()) < w.spec.InitialDelaySeconds { return true

if c.Started != nil && *c.Started { // Stop probing for startup once container has started. if w.probeType == startup { <del> return true </del> <ins> return false </ins> } } else { // Disable other probes until container has started.

failedStatus.Phase = v1.PodFailed tests := []struct { <del> probe          v1.Probe podStatus      *v1.PodStatus expectContinue bool expectSet      bool expectedResult results.Result </del> <ins> probe                v1.Probe podStatus            *v1.PodStatus expectContinue       map[string]bool expectSet            bool expectedResult       results.Result setDeletionTimestamp bool </ins> }{ { // No status. <del> expectContinue: true, </del> <ins> expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> }, { // Pod failed podStatus: &failedStatus, }, <ins> { // Pod deletion podStatus:            &runningStatus, setDeletionTimestamp: true, expectSet:            true, expectContinue: map[string]bool{ readiness.String(): true, }, expectedResult: results.Success, }, </ins> { // No container status <del> podStatus:      &otherStatus, expectContinue: true, </del> <ins> podStatus: &otherStatus, expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> }, { // Container waiting <del> podStatus:      &pendingStatus, expectContinue: true, </del> <ins> podStatus: &pendingStatus, expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> expectSet:      true, expectedResult: results.Failure, },

expectedResult: results.Failure, }, { // Probe successful. <del> podStatus:      &runningStatus, expectContinue: true, </del> <ins> podStatus: &runningStatus, expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> expectSet:      true, expectedResult: results.Success, },

probe: v1.Probe{ InitialDelaySeconds: -100, }, <del> expectContinue: true, </del> <ins> expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> expectSet:      true, expectedResult: results.Success, },

if test.podStatus != nil { m.statusManager.SetPodStatus(w.pod, *test.podStatus) } <del> if c := w.doProbe(); c != test.expectContinue { t.Errorf("[%s-%d] Expected continue to be %v but got %v", probeType, i, test.expectContinue, c) </del> <ins> if test.setDeletionTimestamp { now := metav1.Now() w.pod.ObjectMeta.DeletionTimestamp = &now } if c := w.doProbe(); c != test.expectContinue[probeType.String()] { t.Errorf("[%s-%d] Expected continue to be %v but got %v", probeType, i, test.expectContinue[probeType.String()], c) </ins> } result, ok := resultsManager(m, probeType).Get(testContainerID) if ok != test.expectSet {

} } <ins> func expectStop(t *testing.T, w *worker, c bool, msg string) { if c { t.Errorf("[%s - %s] Expected to stop, but did not", w.probeType, msg) } } </ins> func resultsManager(m *manager, probeType probeType) results.Manager { switch probeType { case readiness:

// startupProbe fails, but is disabled m.prober.exec = fakeExecProber{probe.Failure, nil} msg = "Started, probe failure, result success" <del> expectContinue(t, w, w.doProbe(), msg) </del> <ins> expectStop(t, w, w.doProbe(), msg) </ins> expectResult(t, w, results.Success, msg) }