Code change:
After restart Kubelet, node will become notReady in first kubelet update period. node condition is : in kubelet's logs: when go in to next period (default 10s), node status resume to Ready. Restart Kubelet should not make node status change to notReady. Restart Kubelet, and watch node status for second The root case maybe here: When kubelet start, it will start two go routine, one detect runtime status, one check node status and update to APIServer; detect runtime status: in it will update : in it will check runtimeStatus include : is one of the notReady setter func: So if nodestatus execute before , kubelet will update a node status with notReady condition: In kubelet starting, both go routine and will be called before , so the first node status is notReady. In our cluster is v1.22.17, but I belive it can reproduce in latest version. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce this issue in v1.30 When the container runtime is healthy, the kubelet should not report with the reason We called in the function to update the status of the container runtime, but we did not do this in , We only need to execute function once before and , just like we did in the function, than this problem can be avoided. But in , seems to have some different opinions...
Yeah, in our env, we modify kubelet the same way as you said, to let it execute once before , it worked. I can submit a PR later.
I may be missing something but what happens if something has changed that caused the kubelet to not be ready ... imagine, the Network plugin is not working ... during the initial period until the check changes the state pods will be scheduled on the node and will fail .... I think that we are operating the assumption that this is a restart and during that time nothing changed, but is this a safe assumption or can we guarantee somehow that nothing has changed that could impact the node readiness state?
In this issue, what I found is that nothing has changed, every time I restart kubelet, node will become notReady only in first sync period, in next period it will be ready; It's not an assumption. or other problem of course will cause node notReady, but they are not in this issue's scope, in this scope, notReady reason is message, have not relationship with other problem;
ok, I misread this issue sorry, so it is not to blindly set the node to ready, is just to perform the runtime check before the other checks ... I think you both are right ... actually it seems that if wins the race then there is no problem, righ ?
Yes, but and run in different goroutines, so we cannot guarantee that will complete faster. So it seems like a good choice to execute before running , or do you have better suggestions?
We need to put the goroutines in the right order to make the startup more predictable
Yes, if execute very fast, may there is no problem, because in it will execute . I have test in v1.28 (which code in fastStatusUpdateOnce almost same as 1.30 ) , it will not always reproduce, but still occurr some times. In 1.22, will sleep for about 100ms before everything, it can be reproduced everytimes.
Arranging the goroutines in order can solve our problems in most cases, but in the method, we need to call the containerRuntime api to get the status. I am worried that its response will not be timely enough in special scenarios, there may be a risk that will complete faster.
moving will execute first
I think I must not have expressed clearly. What I'm worried about is that under special circumstances, calls the API of the container runtime and cannot return immediately (not sure if this situation really exists). Even if the method is executed first, it will still complete later than :)
Add some detail: in it call runtime api to check runtime status, then set variable . in , it will check variable, if is nil, this issue's problem will occurr. Even if goroutine call firstly, call container runtime check will still need some time, it can't guarantee that when firstly call, is setted.
are you suggesting to lock syncNodeStatus on updateRuntimeUp? I'm afraid of some corner cases we can hit ... if the container runtime does not return do we block forever?
Container runtime API call have timeout.
Can you try this on a newer version? 1.22 is out of support and we have made changes to node readiness since. /triage needs-information
I have tried in 1.28, it can reproduce, can reproduce in 1.30.
/remove-triage needs-information

Related discussion:
text: // Introduce some small jittering to ensure that over time the requests won't start // accumulating at approximately the same time from the set of nodes due to priority and // fairness effect. <del> go wait.JitterUntil(kl.syncNodeStatus, kl.nodeStatusUpdateFrequency, 0.04, true, wait.NeverStop) </del> <ins> go func() { // Call updateRuntimeUp once before syncNodeStatus to make sure kubelet had already checked runtime state // otherwise when restart kubelet, syncNodeStatus will report node notReady in first report period kl.updateRuntimeUp() wait.JitterUntil(kl.syncNodeStatus, kl.nodeStatusUpdateFrequency, 0.04, true, wait.NeverStop) }() </ins> go kl.fastStatusUpdateOnce() // start syncing lease