Repository: kubernetes/kubernetes (language: go)

Issue:
After restart Kubelet, node will become notReady in first kubelet update period.
### What happened?

After restart Kubelet, node will become notReady in first kubelet update period.

node condition is :
```
  - lastHeartbeatTime: "2024-04-19T06:40:55Z"
    lastTransitionTime: "2024-04-19T06:40:55Z"
    message: container runtime status check may not have completed yet
    reason: KubeletNotReady
    status: "False"
    type: Ready
```

in kubelet's logs:
```
I0419 14:40:55.510166 1902465 setters.go:577] "Node became not ready" node="mjq-k8stestarsenalwork-1" condition={Type:Ready Status:False LastHeartbeatTime:2024-04-19 14:40:55.510127183 +0800 CST m=+5.226928586 LastTransitionTime:2024-04-19 14:40:55.510127183 +0800 CST m=+5.226928586 Reason:KubeletNotReady Message:container runtime status check may not have completed yet}
```

when go in to next period (default 10s), node status resume to Ready.

### What did you expect to happen?

Restart Kubelet should not make node status change to notReady.

### How can we reproduce it (as minimally and precisely as possible)?

Restart Kubelet, and watch node status for second
```
watch -n 1 'kubectl get no <node-name>'
```


### Anything else we need to know?

The root case maybe here:
When kubelet start, it will start two go routine, one detect runtime status, one check node status and update to APIServer;
detect runtime status:

in `updateRuntimeUp` it will update `lastBaseRuntimeSync`:
https://github.com/kubernetes/kubernetes/blob/bf07ef3950d80256cb366b035712b6d60b8a3f4c/pkg/kubelet/kubelet.go#L2909

in `kl.runtimeState.runtimeErrors` it will check runtimeStatus include `lastBaseRuntimeSync`:

https://github.com/kubernetes/kubernetes/blob/bf07ef3950d80256cb366b035712b6d60b8a3f4c/pkg/kubelet/runtime.go#L108

`kl.runtimeState.runtimeErrors` is one of the notReady setter func:
https://github.com/kubernetes/kubernetes/blob/bf07ef3950d80256cb366b035712b6d60b8a3f4c/pkg/kubelet/kubelet_node_status.go#L748

So if nodestatus execute before `updateRuntimeUp`, kubelet will update a node status with notReady condition:
https://github.com/kubernetes/kubernetes/blob/bf07ef3950d80256cb366b035712b6d60b8a3f4c/pkg/kubelet/kubelet.go#L1635-L1641
In kubelet starting,  both `syncNodeStatus` go routine and `fastStatusUpdateOnce` will be called before `updateRuntimeUp`,  so the first node status is notReady.

### Kubernetes version

In our cluster is v1.22.17, but I belive it can reproduce in latest version.

<details>

```console
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T11:47:36Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T11:42:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>



Discussion / hints:
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the `triage/accepted` label and provide further guidance.

The `triage/accepted` label can be added by org members by writing `/triage accepted` in a comment.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
/sig  node
@AllenXu93: The label(s) `sig/` cannot be applied, because the repository doesn't have them.

<details>

In response to [this](https://github.com/kubernetes/kubernetes/issues/124397#issuecomment-2065897186):

>/sig  node


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
I can reproduce this issue in v1.30

When the container runtime is healthy, the kubelet should not report `KubeletNotReady` with the reason `runtime status check may not have completed yet`

We called `kl.updateRuntimeUp()` in the `fastStatusUpdateOnce` function to update the status of the container runtime, but we did not do this in `syncNodeStatus`, 
https://github.com/kubernetes/kubernetes/blob/37ca0377d542d9eb5b6a7ebac2093b8ec6fc9942/pkg/kubelet/kubelet_node_status.go#L467-L469

We only need to execute `kl.updateRuntimeUp()` function once before `fastStatusUpdateOnce` and `syncNodeStatus`, just like we did in the `fastStatusUpdateOnce` function, than this problem can be avoided.

But in https://github.com/kubernetes/kubernetes/issues/122338#issuecomment-1912243387, @aojea seems to have some different opinions...
> I can reproduce this issue in v1.30
> 
> When the container runtime is healthy, the kubelet should not report `KubeletNotReady` with the reason `runtime status check may not have completed yet`
> 
> We called `kl.updateRuntimeUp()` in the `fastStatusUpdateOnce` function to update the status of the container runtime, but we did not do this in `syncNodeStatus`,
> 
> https://github.com/kubernetes/kubernetes/blob/37ca0377d542d9eb5b6a7ebac2093b8ec6fc9942/pkg/kubelet/kubelet_node_status.go#L467-L469
> 
> We only need to execute `kl.updateRuntimeUp()` function once before `fastStatusUpdateOnce` and `syncNodeStatus`, just like we did in the `fastStatusUpdateOnce` function, than this problem can be avoided.
> 
> But in [#122338 (comment)](https://github.com/kubernetes/kubernetes/issues/122338#issuecomment-1912243387), @aojea seems to have some different opinions...

Yeah, in our env, we modify kubelet  the same way as you said, to let it execute `kl.updateRuntimeUp` once before `syncNodeStatus`,   it worked. 
I can submit a PR later.

Fix patch:
diff --git a/pkg/kubelet/kubelet.go b/pkg/kubelet/kubelet.go
index af74a09562816..761a040a0ce3e 100644
--- a/pkg/kubelet/kubelet.go
+++ b/pkg/kubelet/kubelet.go
@@ -1636,7 +1636,13 @@ func (kl *Kubelet) Run(updates <-chan kubetypes.PodUpdate) {
 		// Introduce some small jittering to ensure that over time the requests won't start
 		// accumulating at approximately the same time from the set of nodes due to priority and
 		// fairness effect.
-		go wait.JitterUntil(kl.syncNodeStatus, kl.nodeStatusUpdateFrequency, 0.04, true, wait.NeverStop)
+		go func() {
+			// Call updateRuntimeUp once before syncNodeStatus to make sure kubelet had already checked runtime state
+			// otherwise when restart kubelet, syncNodeStatus will report node notReady in first report period
+			kl.updateRuntimeUp()
+			wait.JitterUntil(kl.syncNodeStatus, kl.nodeStatusUpdateFrequency, 0.04, true, wait.NeverStop)
+		}()
+
 		go kl.fastStatusUpdateOnce()
 
 		// start syncing lease


Test patch: