Change:
If the kubelet Run method panics (e.g. ) Run gets called multiple times: That would start multiple goroutines calling relist() concurrently From a PR that doesn't touch the PLEG (but might stress it?), panic in kubelet from pleg:
/sig node /priority important-soon
Could be related to but not seeing how yet
I don't see how this map could have escaped the PLEG unless it was started twice. I see no signs of that. This appears to be code that should never fail this way if the go runtime is correct (I don't see the pod record object escaping this type, and I see no public interface that touches a field outside of an atomic accessor).
Note nothing in this package has changed since last year in July (all the metrics stuff should already be thread safe)
So far appears to be able to consistently reproduce it (because we improve the efficiency of some failure modes and change some loops), but I don't see any other kind jobs failing on it. So this PR exacerbates (but does not cause?) the issue, potentially.
now no longer causes this after a rebase. I am suspicious that this may have been something related to how the code was built in go (environment, optimizations, strange merge) but I don't have any evidence to support that. Will keep watching.
I think different relist period which happens every second may trigger race of . What do you think ? see
Looking at the logs there are 3 fatal errors that occur: , (2x), and . All are caused by concurrent calls.
these concurrent operations happens in different place, I wonder two goroutines use podRecords at the same time (after all 1 second relist period is too short).
I think the kubelet is panicing somewhere else (e.g. ) and Run is getting called multiple times That would start multiple goroutines calling relist() concurrently
it looks like the kubelet does not exit on panics (ReallyCrashForTesting defaults false):
The first panic seen in that kubelet log is:
ah I see. For the record, it does look like the kubelet crashes on the s. kubelet pid changes after each one. But it makes sense if the kubelet isn't fully crashing on the first panic, leading to concurrent runs of the , thus leading to the eventual fatal error.
Yeah.... I think that is the real issue
the panic makes another Run loop start, which sets up a concurrent map read/write, which actually causes the process exit :-)
retitled and bumped priority
We were just talking about this in sig-node yesterday; changing the kubelet to crash on panic. This is definitely evidence that we should do that. I didn't realize this was possible with the panic recovery. /cc
So there are two issues here: was a bug in the PR under test, fixed before merge the kubelet to exit on panic, not live on in zombie state
looks like the particular panic was an actual bug in that PR (ContainerStatuses indexed in a range of InitContainerStatuses): the bug was
Should we change the default for ReallyCrashForTesting to true ? In that case, the flag doesn't need to exist.
Options: . Run() exiting is never expected, and calling it again is never correct. ReallyCrashForTesting completely and leave the panic handler at its default behavior of exiting. This would affect all uses of HandleCrash, which at first glance, looks like these for the kubelet:
Working on a patch for option now.
Yes we talked about making kubelet actually crash in the sig meeting yesterday. I think we had this consensus back in , but kubelet wasn't changed because of the . Yesterday I think we agreed it's good to make kubelet crash for debugging purpose. One concern was if there was a panic caused by plugin / container runtime, kubelet could end up in a crashlooping state. For that concern, we talked about every cluster setup should have some monitoring around how many times / how frequent kubelet restarted. systemd service has a restart counter. I haven't dig into whether it has the restart interval / frequency information I'm working on a PR to remove the ReallyCrashForTesting flag. I think we should do that in 1.19 and have release notes about the behavior change and monitoring suggestions.
We should never wait.Until things that don't tolerate it. So removing the wait.Until is appropriate.
I would recommend removing the wait.Until around the Run() invocation in 1.15-1.18, since that is known to be wrong, and consider the removal of ReallyCrashForTesting in 1.19
Agreed, that is the safest approach.

Discussion:
func startKubelet(k kubelet.Bootstrap, podCfg *config.PodConfig, kubeCfg *kubeletconfiginternal.KubeletConfiguration, kubeDeps *kubelet.Dependencies, enableCAdvisorJSONEndpoints, enableServer bool) { // start the kubelet <del> go wait.Until(func() { k.Run(podCfg.Updates()) }, 0, wait.NeverStop) </del> <ins> go k.Run(podCfg.Updates()) </ins> // start the kubelet server if enableServer {