Repository: kubernetes/kubernetes (language: go)

Issue:
kubelet crash: fatal error: concurrent map writes
### What happened?

While looking into three failing E2E tests in https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1854326383216431104 (an AllAlpha job), I found that Kubelet had crashed ([logs](https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1854326383216431104/artifacts/bootstrap-e2e-minion-group-h288/kubelet.log)):

```
Nov 07 01:09:07.338827 bootstrap-e2e-minion-group-h288 kubelet[1937]: fatal error: concurrent map writes
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]: goroutine 29497 [running]:
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet/cm/containermap.ContainerMap.Add(...)
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/cm/containermap/container_map.go:36
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet/cm/cpumanager.(*manager).AddContainer(0xc0008a5ad0, 0xc002175688, 0xc002d8a780, {0xc002d21ec0, 0x40})
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/cpu_manager.go:276 +0x165
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet/cm.(*internalContainerLifecycleImpl).PreStartContainer(0xc000a2a060, 0xc002175688, 0xc002d8a780, {0xc002d21ec0, 0x40})
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/cm/internal_container_lifecycle.go:42 +0x42
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).startContainer(0xc0009c4180, {0x334ff38, 0xc001184050}, {0xc0017ab600, 0x40}, 0xc000323420, 0xc0026aad58, 0xc002175688, 0xc002856c60, {0x4de4ae0, ...}, ...)
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:271 +0xcdb
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod.func1({0x334ff38, 0xc001184050}, {0x2e91ac3, 0x9}, {0x2e91ac3, 0x9}, 0xc0026aad58)
Nov 07 01:09:07.342609 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1249 +0x8c7
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod(0xc0009c4180, {0x334ff38, 0xc001184050}, 0xc002175688, 0xc002856c60, {0x4de4ae0, 0x0, 0x0}, 0xc00095f220)
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1318 +0x37d9
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).SyncPod(0xc000431008, {0x334ff00, 0xc001133860}, 0x2, 0xc002175688, 0x0, 0xc002856c60)
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:2000 +0x1923
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop.func1({0x0, {0x2, {0xc1c322abaa0c010a, 0x7267b7d9ca, 0x4dc1f60}, 0xc002175688, 0x0, 0x0, 0x0}}, 0xc000ad6960, ...)
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1286 +0x1ca
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop(0xc000ad6960, {0xc000e53bc0, 0x24}, 0xc002574150)
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1291 +0x49b
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod.func1()
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:950 +0x118
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]: created by k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod in goroutine 250
Nov 07 01:09:07.344799 bootstrap-e2e-minion-group-h288 kubelet[1937]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:945 +0x20db
```

### What did you expect to happen?

No crash.

### How can we reproduce it (as minimally and precisely as possible)?

Don't know, but we have logs from a prow job that encountered it. It's possible that this crash requires a particular alpha feature gate, since this was an AllAlpha job.

### Anything else we need to know?

_No response_

### Kubernetes version

<details>
```
v1.32.0-beta.0.205+4c487b00afb20b
```
</details>

### Cloud provider

<details>
GCE
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>



Discussion / hints:
/sig node
Thought to search for similar errors to see if there were other examples of the same crash (https://storage.googleapis.com/k8s-triage/index.html?text=10250%3A%20connect%3A%20connection%20refused) and found some that look related.

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-ec2-eks-al2023/1853917686065532928 ([logs](https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-e2e-ec2-eks-al2023/1853917686065532928/artifacts/logs/i-0f7be39e04f78bc14/kubelet.log)):

```
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: fatal error: concurrent map iteration and map write
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: goroutine 247 [running]:
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/containermap.ContainerMap.Visit(...)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/containermap/container_map.go:75
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/memorymanager.(*manager).removeStaleState(0xc000896ab0)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/memorymanager/memory_manager.go:334 +0x3f2
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/memorymanager.(*manager).Allocate(0xc000896ab0, 0xc002b3d208, 0xc0029fad80)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/memorymanager/memory_manager.go:243 +0x4c
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/topologymanager.(*scope).allocateAlignedResources(0x0?, 0xc002b3d208, 0xc0029fad80)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/topologymanager/scope.go:152 +0x5c
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/topologymanager.(*scope).admitPolicyNone(0xc000432ff0, 0xc002b3d208)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/topologymanager/scope.go:140 +0x1ab
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/topologymanager.(*noneScope).Admit(0x1?, 0xc0030708e0?)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/topologymanager/scope_none.go:45 +0x27
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/topologymanager.(*manager).Admit(0xc0007a3ce0, 0xc000d6e540)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/topologymanager/topology_manager.go:218 +0x88
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).canAdmitPod(0xc000309008, {0xc0031dbba0, 0x16, 0x1a}, 0xc002b3d208)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:2320 +0x109
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).HandlePodAdditions(0xc000309008, {0xc001ec4298, 0x1, 0x1})
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:2610 +0x7ac
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).syncLoopIteration(0xc000309008, {0x325ecd8, 0x4c58700}, 0xc00049b500, {0x3268080, 0xc000309008}, 0xc0004ad110, 0xc0004ad1f0, 0xc00049d7a0)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:2428 +0xbba
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).syncLoop(0xc000309008, {0x325ecd8, 0x4c58700}, 0xc00049b500, {0x3268080, 0xc000309008})
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:2371 +0x325
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run(0xc000309008, 0xc00049b500)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:1716 +0x8cf
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: created by k8s.io/kubernetes/cmd/kubelet/app.startKubelet in goroutine 1
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:1304 +0x96
```

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-cos-containerd-e2e-cos-gce/1853677098250539008 ([logs](https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-cos-containerd-e2e-cos-gce/1853677098250539008/artifacts/bootstrap-e2e-minion-group-mfdb/kubelet.log)):

```
Nov 05 06:09:05.838398 bootstrap-e2e-minion-group-mfdb kubelet[1874]: fatal error: concurrent map writes
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: goroutine 26977 [running]:
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet/cm/containermap.ContainerMap.Add(...)
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/cm/containermap/container_map.go:36
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet/cm/cpumanager.(*manager).AddContainer(0xc0009ebad0, 0xc001b11208, 0xc001d44c00, {0xc000e5c540, 0x40})
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/cpu_manager.go:276 +0x165
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet/cm.(*internalContainerLifecycleImpl).PreStartContainer(0xc000a50e70, 0xc001b11208, 0xc001d44c00, {0xc000e5c540, 0x40})
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/cm/internal_container_lifecycle.go:42 +0x42
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).startContainer(0xc0009e8480, {0x3271318, 0xc001cbf920}, {0xc00192b080, 0x40}, 0xc00042e460, 0xc002368c08, 0xc001b11208, 0xc0015fbef0, {0x4c73a20, ...}, ...)
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:271 +0xcdb
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod.func1({0x3271318, 0xc001cbf920}, {0x2dc3d3a, 0x9}, {0x2dc3d3a, 0x9}, 0xc002368c08)
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1249 +0x8c7
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod(0xc0009e8480, {0x3271318, 0xc001cbf920}, 0xc001b11208, 0xc0015fbef0, {0x4c73a20, 0x0, 0x0}, 0xc0000c3130)
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1318 +0x37d9
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).SyncPod(0xc0001d0c08, {0x32712e0, 0xc001ef6be0}, 0x2, 0xc001b11208, 0x0, 0xc0015fbef0)
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:1982 +0x2523
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop.func1({0x0, {0x2, {0xc1c28b7ec384d573, 0x82237b02ec, 0x4c50ea0}, 0xc001b11208, 0x0, 0x0, 0x0}}, 0xc0008e65a0, ...)
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1286 +0x1ca
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop(0xc0008e65a0, {0xc000931fb0, 0x24}, 0xc0020a0310)
Nov 05 06:09:05.842397 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1291 +0x49b
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod.func1()
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:950 +0x118
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: created by k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod in goroutine 235
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:945 +0x20db
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: goroutine 1 [select, 9 minutes]:
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/cmd/kubelet/app.run({0x32712e0, 0xc0003d77c0}, 0xc00071e008, 0xc0007a1040, {0x7854a8080858, 0xc0000c65b0})
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:926 +0x21e5
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/cmd/kubelet/app.Run({0x32712e0, 0xc0003d77c0}, 0xc00071e008, 0xc0007a1040, {0x7854a8080858, 0xc0000c65b0})
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:526 +0x333
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0xc000484308, {0xc000144010, 0x12, 0x13})
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:292 +0xb9f
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: github.com/spf13/cobra.(*Command).execute(0xc000484308, {0xc000144010, 0x12, 0x13})
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000484308)
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: github.com/spf13/cobra.(*Command).Execute(...)
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         github.com/spf13/cobra@v1.8.1/command.go:1041
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/component-base/cli.run(0xc000484308)
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/component-base/cli/run.go:143 +0x245
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: k8s.io/component-base/cli.Run(0xc0000061c0?)
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/component-base/cli/run.go:44 +0x17
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]: main.main()
Nov 05 06:09:05.845449 bootstrap-e2e-minion-group-mfdb kubelet[1874]:         k8s.io/kubernetes/cmd/kubelet/kubelet.go:36 +0x18
```

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-cos-containerd-e2e-ubuntu-gce/1853571903961174016 ([logs](https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-cos-containerd-e2e-ubuntu-gce/1853571903961174016/artifacts/bootstrap-e2e-minion-group-csh0/kubelet.log)):

```
Nov 04 23:11:39.134387 bootstrap-e2e-minion-group-csh0 kubelet[2170]: fatal error: concurrent map writes
Nov 04 23:11:39.141903 bootstrap-e2e-minion-group-csh0 kubelet[2170]: goroutine 46314 [running]:
Nov 04 23:11:39.142199 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet/cm/containermap.ContainerMap.Add(...)
Nov 04 23:11:39.142199 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/cm/containermap/container_map.go:36
Nov 04 23:11:39.142199 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet/cm/memorymanager.(*manager).AddContainer(0xc00099ea20, 0xc0023d1688, 0xc0018bdd88, {0xc002ec7dc0?, 0x40?})
Nov 04 23:11:39.142199 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/cm/memorymanager/memory_manager.go:197 +0x110
Nov 04 23:11:39.142199 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet/cm.(*internalContainerLifecycleImpl).PreStartContainer(0xc000a529f0, 0xc0023d1688, 0xc0018bdd88, {0xc002ec7dc0, 0x40})
Nov 04 23:11:39.142199 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/cm/internal_container_lifecycle.go:46 +0x72
Nov 04 23:11:39.142199 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).startContainer(0xc000990480, {0x326b558, 0xc0009b78e0}, {0xc002ec61c0, 0x40}, 0xc001a33b20, 0xc00103ac08, 0xc0023d1688, 0xc001f277a0, {0x4c6a9e0, ...}, ...)
Nov 04 23:11:39.142199 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:271 +0xcdb
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod.func1({0x326b558, 0xc0009b78e0}, {0x2dbee5a, 0x9}, {0x2dbee5a, 0x9}, 0xc00103ac08)
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1249 +0x8c7
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet/kuberuntime.(*kubeGenericRuntimeManager).SyncPod(0xc000990480, {0x326b558, 0xc0009b78e0}, 0xc0023d1688, 0xc001f277a0, {0x4c6a9e0, 0x0, 0x0}, 0xc000588eb0)
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:1318 +0x37d9
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet.(*Kubelet).SyncPod(0xc0001d0c08, {0x326b520, 0xc002cbf4a0}, 0x2, 0xc0023d1688, 0x0, 0xc001f277a0)
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/kubelet.go:1982 +0x2523
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop.func1({0x0, {0x2, {0xc1c2730a89772d94, 0x83eee24cb3, 0x4c47e60}, 0xc0023d1688, 0x0, 0x0, 0x0}}, 0xc000050a00, ...)
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1286 +0x1ca
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).podWorkerLoop(0xc000050a00, {0xc000f62390, 0x24}, 0xc001974f50)
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:1291 +0x49b
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod.func1()
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:950 +0x118
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: created by k8s.io/kubernetes/pkg/kubelet.(*podWorkers).UpdatePod in goroutine 234
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/pkg/kubelet/pod_workers.go:945 +0x20db
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: goroutine 1 [select, 9 minutes]:
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/cmd/kubelet/app.run({0x326b520, 0xc00045c3c0}, 0xc000283808, 0xc0008541a0, {0x7f4894e93ae0, 0xc000477a40})
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:926 +0x21e5
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/cmd/kubelet/app.Run({0x326b520, 0xc00045c3c0}, 0xc000283808, 0xc0008541a0, {0x7f4894e93ae0, 0xc000477a40})
Nov 04 23:11:39.142888 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:526 +0x333
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0xc00071e008, {0xc000142010, 0x13, 0x13})
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/cmd/kubelet/app/server.go:292 +0xb9f
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]: github.com/spf13/cobra.(*Command).execute(0xc00071e008, {0xc000142010, 0x13, 0x13})
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]: github.com/spf13/cobra.(*Command).ExecuteC(0xc00071e008)
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]: github.com/spf13/cobra.(*Command).Execute(...)
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         github.com/spf13/cobra@v1.8.1/command.go:1041
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/component-base/cli.run(0xc00071e008)
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/component-base/cli/run.go:143 +0x245
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]: k8s.io/component-base/cli.Run(0xc0000061c0?)
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/component-base/cli/run.go:44 +0x17
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]: main.main()
Nov 04 23:11:39.144356 bootstrap-e2e-minion-group-csh0 kubelet[2170]:         k8s.io/kubernetes/cmd/kubelet/kubelet.go:36 +0x18
```
I think this may by the cause.

The `containerMap` is passed as a parameter to  three mangers, so it may be read/written by them  concurrently. 

https://github.com/kubernetes/kubernetes/blob/4c487b00afb20b59e66099ba5e955e6f8ff992c0/pkg/kubelet/cm/container_manager_linux.go#L567-L585

https://github.com/kubernetes/kubernetes/blob/4c487b00afb20b59e66099ba5e955e6f8ff992c0/pkg/kubelet/cm/container_manager_linux.go#L646-L648
/assign
/triage accepted
/priority important-soon
/cc
the first suspect in line is admission racing with reconciliation loops of the cpu and memory managers
> the first suspect in line is admission racing with reconciliation loops of the cpu and memory managers

```
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: fatal error: concurrent map iteration and map write
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: goroutine 247 [running]:
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/containermap.ContainerMap.Visit(...)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/containermap/container_map.go:75
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]: k8s.io/kubernetes/pkg/kubelet/cm/memorymanager.(*manager).removeStaleState(0xc000896ab0)
Nov 05 22:15:13 ip-172-31-22-196.ec2.internal kubelet[4849]:         k8s.io/kubernetes/pkg/kubelet/cm/memorymanager/memory_manager.go:334 +0x3f2
```

confirmed!
the root cause is likely a side effect of the GA graduation of memory manager. We share now the same ContainerMap across cpumanager and memory manager, and both can modify it.
The map is also shared with device manager, which (IIRC, but a quick look seems to confirm) only accesses the map in read only mode so it's less problematic.

An initial fix is to build the map once (no point in build it repeatedly) but clone it and pass each instance to each manager.

I see yet no obvious benefit in using a shared map across managers, even though there are possible synchronization concerns which are worth a second look later.
OTOH a specific map per manager is how things operated till recently, so let's stop the bleeding first.
/priority critical-urgent
/assign

Fix patch:
diff --git a/pkg/kubelet/cm/container_manager_linux.go b/pkg/kubelet/cm/container_manager_linux.go
index 9d335791f4cb7..efbd90c1d5f24 100644
--- a/pkg/kubelet/cm/container_manager_linux.go
+++ b/pkg/kubelet/cm/container_manager_linux.go
@@ -575,13 +575,13 @@ func (cm *containerManagerImpl) Start(ctx context.Context, node *v1.Node,
 	}
 
 	// Initialize CPU manager
-	err := cm.cpuManager.Start(cpumanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService, containerMap)
+	err := cm.cpuManager.Start(cpumanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService, containerMap.Clone())
 	if err != nil {
 		return fmt.Errorf("start cpu manager error: %w", err)
 	}
 
 	// Initialize memory manager
-	err = cm.memoryManager.Start(memorymanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService, containerMap)
+	err = cm.memoryManager.Start(memorymanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService, containerMap.Clone())
 	if err != nil {
 		return fmt.Errorf("start memory manager error: %w", err)
 	}
@@ -643,7 +643,7 @@ func (cm *containerManagerImpl) Start(ctx context.Context, node *v1.Node,
 	}
 
 	// Starts device manager.
-	if err := cm.deviceManager.Start(devicemanager.ActivePodsFunc(activePods), sourcesReady, containerMap, containerRunningSet); err != nil {
+	if err := cm.deviceManager.Start(devicemanager.ActivePodsFunc(activePods), sourcesReady, containerMap.Clone(), containerRunningSet); err != nil {
 		return err
 	}
 
diff --git a/pkg/kubelet/cm/container_manager_windows.go b/pkg/kubelet/cm/container_manager_windows.go
index 5cbbc30021828..3c36daeffb546 100644
--- a/pkg/kubelet/cm/container_manager_windows.go
+++ b/pkg/kubelet/cm/container_manager_windows.go
@@ -25,10 +25,11 @@ package cm
 import (
 	"context"
 	"fmt"
+	"sync"
+
 	utilfeature "k8s.io/apiserver/pkg/util/feature"
 	kubefeatures "k8s.io/kubernetes/pkg/features"
 	"k8s.io/kubernetes/pkg/kubelet/cm/memorymanager"
-	"sync"
 
 	"k8s.io/klog/v2"
 	"k8s.io/mount-utils"
@@ -96,14 +97,14 @@ func (cm *containerManagerImpl) Start(ctx context.Context, node *v1.Node,
 
 	// Initialize CPU manager
 	if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.WindowsCPUAndMemoryAffinity) {
-		err := cm.cpuManager.Start(cpumanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService, containerMap)
+		err := cm.cpuManager.Start(cpumanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService, containerMap.Clone())
 		if err != nil {
 			return fmt.Errorf("start cpu manager error: %v", err)
 		}
 	}
 
 	// Starts device manager.
-	if err := cm.deviceManager.Start(devicemanager.ActivePodsFunc(activePods), sourcesReady, containerMap, containerRunningSet); err != nil {
+	if err := cm.deviceManager.Start(devicemanager.ActivePodsFunc(activePods), sourcesReady, containerMap.Clone(), containerRunningSet); err != nil {
 		return err
 	}
 
diff --git a/pkg/kubelet/cm/containermap/container_map.go b/pkg/kubelet/cm/containermap/container_map.go
index 7dc4689af623b..1d3fce813b9ff 100644
--- a/pkg/kubelet/cm/containermap/container_map.go
+++ b/pkg/kubelet/cm/containermap/container_map.go
@@ -20,23 +20,35 @@ import (
 	"fmt"
 )
 
-// ContainerMap maps (containerID)->(*v1.Pod, *v1.Container)
-type ContainerMap map[string]struct {
+// cmItem (ContainerMap ITEM) is a pair podUID, containerName
+type cmItem struct {
 	podUID        string
 	containerName string
 }
 
+// ContainerMap maps (containerID)->(podUID, containerName)
+type ContainerMap map[string]cmItem
+
 // NewContainerMap creates a new ContainerMap struct
 func NewContainerMap() ContainerMap {
 	return make(ContainerMap)
 }
 
+// Clone creates a deep copy of the ContainerMap
+func (cm ContainerMap) Clone() ContainerMap {
+	ret := make(ContainerMap, len(cm))
+	for key, val := range cm {
+		ret[key] = val
+	}
+	return ret
+}
+
 // Add adds a mapping of (containerID)->(podUID, containerName) to the ContainerMap
 func (cm ContainerMap) Add(podUID, containerName, containerID string) {
-	cm[containerID] = struct {
-		podUID        string
-		containerName string
-	}{podUID, containerName}
+	cm[containerID] = cmItem{
+		podUID:        podUID,
+		containerName: containerName,
+	}
 }
 
 // RemoveByContainerID removes a mapping of (containerID)->(podUID, containerName) from the ContainerMap


Test patch:
diff --git a/pkg/kubelet/cm/containermap/container_map_test.go b/pkg/kubelet/cm/containermap/container_map_test.go
index 9dd5cbb24a4a9..737bfa209898f 100644
--- a/pkg/kubelet/cm/containermap/container_map_test.go
+++ b/pkg/kubelet/cm/containermap/container_map_test.go
@@ -20,6 +20,65 @@ import (
 	"testing"
 )
 
+func TestContainerMapCloneEqual(t *testing.T) {
+	cm := NewContainerMap()
+	// add random fake data
+	cm.Add("fakePodUID-1", "fakeContainerName-a1", "fakeContainerID-A")
+	cm.Add("fakePodUID-2", "fakeContainerName-b2", "fakeContainerID-B")
+	cm.Add("fakePodUID-2", "fakeContainerName-c2", "fakeContainerID-C")
+	cm.Add("fakePodUID-3", "fakeContainerName-d3", "fakeContainerID-D")
+	cm.Add("fakePodUID-3", "fakeContainerName-e3", "fakeContainerID-E")
+	cm.Add("fakePodUID-3", "fakeContainerName-f3", "fakeContainerID-F")
+
+	cloned := cm.Clone()
+	if !areEqual(cm, cloned) {
+		t.Fatalf("clone %+#v different from original %+#v", cloned, cm)
+	}
+}
+
+func TestContainerMapCloneUnshared(t *testing.T) {
+	cm := NewContainerMap()
+	// add random fake data
+	cm.Add("fakePodUID-1", "fakeContainerName-a1", "fakeContainerID-A")
+	cm.Add("fakePodUID-2", "fakeContainerName-b2", "fakeContainerID-B")
+	cm.Add("fakePodUID-2", "fakeContainerName-c2", "fakeContainerID-C")
+	cm.Add("fakePodUID-3", "fakeContainerName-d3", "fakeContainerID-D")
+	cm.Add("fakePodUID-3", "fakeContainerName-e3", "fakeContainerID-E")
+	cm.Add("fakePodUID-3", "fakeContainerName-f3", "fakeContainerID-F")
+
+	// early sanity check, random ID, no special meaning
+	podUID, containerName, err := cm.GetContainerRef("fakeContainerID-C")
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err)
+	}
+	if podUID != "fakePodUID-2" || containerName != "fakeContainerName-c2" {
+		t.Fatalf("unexpected data: uid=%q name=%q", podUID, containerName)
+	}
+	if cID, err := cm.GetContainerID(podUID, containerName); err != nil || cID != "fakeContainerID-C" {
+		t.Fatalf("unexpected data: cid=%q err=%v", cID, err)
+	}
+
+	cloned := cm.Clone()
+	cloned.RemoveByContainerRef("fakePodUID-2", "fakeContainerName-c2")
+	// check is actually gone
+	if cID, err := cloned.GetContainerID("fakePodUID-2", "fakeContainerName-c2"); err == nil || cID != "" {
+		t.Fatalf("unexpected data found: cid=%q", cID)
+	}
+
+	// check the original copy didn't change
+	// early sanity check, random ID, no special meaning
+	podUIDRedo, containerNameRedo, err2 := cm.GetContainerRef("fakeContainerID-C")
+	if err != nil {
+		t.Fatalf("unexpected error: %v", err2)
+	}
+	if podUIDRedo != "fakePodUID-2" || containerNameRedo != "fakeContainerName-c2" {
+		t.Fatalf("unexpected data: uid=%q name=%q", podUIDRedo, containerNameRedo)
+	}
+	if cID, err := cm.GetContainerID(podUIDRedo, containerNameRedo); err != nil || cID != "fakeContainerID-C" {
+		t.Fatalf("unexpected data: cid=%q", cID)
+	}
+}
+
 func TestContainerMap(t *testing.T) {
 	testCases := []struct {
 		podUID         string
@@ -84,3 +143,19 @@ func TestContainerMap(t *testing.T) {
 		}
 	}
 }
+
+func areEqual(cm1, cm2 ContainerMap) bool {
+	if len(cm1) != len(cm2) {
+		return false
+	}
+	for key1, item1 := range cm1 {
+		item2, ok := cm2[key1]
+		if !ok {
+			return false
+		}
+		if item1 != item2 {
+			return false
+		}
+	}
+	return true
+}