Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods remaining in running but not ready state after node become unready for few seconds. What you expected to happen: Pods should become ready when node is ready. How to reproduce it (as minimally and precisely as possible): Created large deployment. Make node unready for few seconds. Pods will remain not ready. However is requires node lifecycle controler to be in eviction pass, so my guess it will not be reproducible in 100% of cases. Anything else we need to know?: I achieve this state lock with kubemark run. There is chance that vanilla k8s is unaffected. Environment: Kubernetes version (use ): 18 Cloud provider or hardware configuration: gce - kubemark OS (e.g: ): linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
My guess of what happened is: nodes become not ready. lifecycle controller store node statuses become ready again pass (based on stored status) marked pods not ready even though nodes were ready at that time
/cc
kube-controller-manager log: The error was created here: Assuming that node still existed at that time, it didn't changed execution of the logic. OTOH support for kubemark provider would be nice.
There is a possibility that node lifecycle controller didn't spot node updates. I'm bumping up log level for this case:
Apiserver logs (ragrding node status): Updating pod (from node hollow-node-pz68p):
Can someone give a hint what sets the pod ready status to true, when node is ready again?
I also find this bug and It is introduced by when I rollback this PR, This bug is fixed
If you have a chance, can you see if the following fixes the issue you (without reverting ) ?
I haven't find a way to reproduce this issue every time, so it is hard for me to say if it fixes the issue. Can you say more about what issue this patch addresses? Is there a problem with statusManager having wrong/old pod status?
I've re-run the test. There was also a problem with hollow-nodes. This time test didn't have problem with pods. k8s build included patch posted above. It looks like your patch helps. I still would like to hear some backstory of what happens there.
My patch refines the code to achieve what intended to do. Can you say more about problem with hollow-node ? Please post log snippet if you have it. Thanks
hi, . I have already test your fix in and it can solve this bug. Still, I have a suggestion. Could you use UpdatePod() function in podManager instead of directly changing pod status in podManager by p.Status=status? If some other codes also use this direct way to modify pod infos in podManager, it will be very hard to trace pod infos are modified by which code , still there is no any log show it has been changed.
- FYI
I have log for status update over w.r.t. using UpdatePod() function in podManager, the review / test may take more time. I will think about it meanwhile.
Is this issue the same problem? Sure looks like it: I've been following that issue in hopes of a fix as we see this happen a few times a day (several large clusters). Very happy to see progress.
Can you attach kubelet log (without the patch) somewhere and share the URL ? Thanks
We are also hit by this on a regular basis with kublets v1.15.2 The following issues are reporting the same bug I think: As pointed out in this inconsistency seems to be trigged when a kubelet reconnects to the apiserver. At least this is what we observe.
this as well /cc
Maybe add a bug label? I believe there are at least 4 independent bug reports about this rather serious inconsistency that requires a kubelet restart to fix.
Reiterating the mentions to trigger a notification: <detailsIn response to : Maybe add a bug label? I believe there are at least 4 independent bug reports about this rather serious inconsistency that requires a kubelet restart to fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ah, nvm. It is already attached. Sorry for the noise. I got confused with the other related issues. All good.
Is a different fix for the same underlying problem? I wouldn't normally ask, but we're trying to understand if there are two different problem and fixes we need, or competing solutions for the same problem.

Discussion:
"k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/pkg/kubelet/config" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" <ins> kubelettypes "k8s.io/kubernetes/pkg/kubelet/types" </ins> utilnode "k8s.io/kubernetes/pkg/util/node" )

// a kubelet running without apiserver requires an additional // update of the static pod status. See #57106 for _, p := range pods { <del> if status, ok := kl.statusManager.GetPodStatus(p.UID); ok { p.Status = status </del> <ins> if kubelettypes.IsStaticPod(p) { if status, ok := kl.statusManager.GetPodStatus(p.UID); ok { klog.V(2).Infof("status for pod %v updated to %v", p.Name, status) p.Status = status } </ins> } } return pods