Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Shutdown a node gracefully. E.g. with . Expected to see pods being evicted, and pods being started on another nodes. Let a deployment create one pod, configure Graceful Node Shutdown as described in , then just that node. Then, the pod will get a state of or , but no new pod is being created and scheduled on a different node. Then, when the node comes back, the terminated pod stays in such a state: Effectively, the pod is running on the node which has been rebooted gracefully, but is in a state. This worked as expected in v1.21 Kubernetes version (use ):Cloud provider or hardware configuration: bare metal OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others:
/sig api-machinery
/sig node /cc
/remove-sig api-machinery
As I tested in both 1.21 and 1.22, the big difference was whether pod's is or (Please see details below). I'm suspecting that doesn't change phase to for node shutdown manager (While eviction manager is changing it, ). I'm not familiar with the codes around it, so I'm not sure if it is intended or not, though. <details[k8s 1.21] [k8s 1.22] </details
Note that Graceful Node Shutdown in k8s 1.21 seems to work well even with Statefulset with persistent volumes (Please also see the details below for the console log). It may be out of the scope of this issue, but sharing here just for your information. <details- Deploy statefulset with persistent volumes Rebooting node1 </details
Also, sharing additional test results for "Statefulset with persistent volume" case with k8s 1.21. didn't work well, but worked well (Please also see the details below). Tested with ShutdownGracePeriod=30s, and ShutdownGracePeriodCriticalPods=10s, as described in the blog. And all the nodes are running on CentOS 8. <details[With ] on node1After manually start the node1 [With ] on node1 </details
Also sharing the results with to the current master branch. I just replaced the kubelet to the patched version on the nodes of k8s 1.22 environment that is previously used. Some of the Pods still remain , but the situation seems improving (Not working perfectly and not patched to the k8s 1.22 release, so I can't say that the patch solved the issue nor the patch did improve). I think that further investigation will be much beyond my current knowledge, so I stop here. I hope that this series of information helps in solving this issue. Feel free to ask me to try another tests with some other configuration. I will keep this environment as they are for a while. <details[Deployment without persistent volume] node2 (Sorry. No pods are scheduled on node1, so I did a bit different test from previous ones.)After node2 is back [StatefulSet with persistent volumes] node2After node2 is back (Or five minutes?) </details
Hi It seems that the node status is not synchronized in time Could you remove the go and try again to see if it can be avoided
/assign
Thank you for your comment. Tested with and confirmed that pods constantly become . As all the pods don't become , they remained in the nodes. <details- After on node1 After node1 is back, again </details
My suspicion is that the graceful shutdown duration of the Pod is the same as that of NodeShutdown. Node shuts down first and the Pod data is not synchronized to ApiServer. Caused the container status not to be updated. Could you try this patch
Commented the test result for the patch in .
/triage accepted /cc
Basically, this was working as expected, at least as expected by documentation in 1.21. Is it not a regression?
This is a regression in 1.22, caused by The change was intentional, since restarting a node should not necessarily stop all pods/containers on that node (see ), but for people depending on the existing behavior, it is definitely a regression. I think we should do the following: Fix the regression and backport to 1.22.x and 1.23.x to restore the delete-on-shutdown behavior In 1.24, add a kubelet option to permit choosing whether pods are terminated on node shutdown or not and change the default to not terminate. People who want the prior behavior can opt into setting the "terminate-on-shutdown" option in 1.24. This should be done before promoting graceful node shutdown to GA
I agree with Jordan, that is what we should have done. People using graceful shutdown on non-terminal pods should be informed of this behavior change as well (someone deploying in 1.22 not using transient nodes may be impacted by this in a way that causes data loss, for instance if they have data in pod local dirs, and have started using graceful deletion in 1.22 unaware of the previous behavior). Jordan argued that as a beta feature, changing the default in 1.23-1.24 is acceptable, whereas changing the default in a back port fix is unusual, and per our normal behavior with regressions, even with enough of a gap that people might anticipate the new behavior, we try to go back to our previous behavior in patch versions.
see upthread, Jordan and I also briefly discussed back porting the config flag that would set the default as well (we should explore that in the fix), and for distros that want to use beta graceful shutdown but are not using transient nodes, we should recommend that they leverage that flag in order to avoid data loss on pods (if we indeed decide to include the flag).
I dont know if its related, but since 1.22, I frequently see Terminated pods still running. It happens if I accidentally pull the power plug, or something like that. And those pods are up and running after the nodes come back, but their state is Terminated: It seems that these pods are daemonset pods.
The plan in makes sense. I've opened up the following changes to revert to the previous 1.21 behavior. master - 1.23 - 1.22 - We will followup with option to make it configurable in 1.24 as we discussed.

Discussion:
klog.V(1).InfoS("Shutdown manager killing pod with gracePeriod", "pod", klog.KObj(pod), "gracePeriod", gracePeriodOverride) if err := m.killPodFunc(pod, false, &gracePeriodOverride, func(status *v1.PodStatus) { <ins> // set the pod status to failed (unless it was already in a successful terminal phase) if status.Phase != v1.PodSucceeded { status.Phase = v1.PodFailed } </ins> status.Message = nodeShutdownMessage status.Reason = nodeShutdownReason }); err != nil {

} } <del> return pod.Status.Message == podShutdownMessage && pod.Status.Reason == podShutdownReason && hasContainersNotReadyCondition </del> <ins> return pod.Status.Message == podShutdownMessage && pod.Status.Reason == podShutdownReason && hasContainersNotReadyCondition && pod.Status.Phase == v1.PodFailed </ins> }