Change:
What happened: One pod stuck at CreateContainerError and can't get recovered automatically. Kubelet log shows that the requested name (let's say Container2) is already in use. shows the Container2 Running, and it's created before Container1 This should never happen. How to reproduce it (as minimally and precisely as possible): I wish i knew. Anything else we need to know?: I know this can be caused by time shifting, but I didn't find any time shifting sign in system log and chrony log, and there is no rebooting between the create timestamp of Container1 and Container2, so this should be caused by something else. This happens intermittently in our env, like 3 times in half a year, and I didn't find any ime shifting sign each time. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:18Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-08-28T11:18:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): Linux node-1 3.10.0-957.21.3.el7.x8664 SMP Wed Jul 17 06:57:50 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig node
Any update on this issue, we are also facing the same issue. EKS version : 1.14.9 Docker version: 19.03.6-ce AWS CNI: 1.6.2 Intermittently we are also facing this , what might the issue, could please guide us here? Below are the kubelet logs. )" with CreatePodSandboxError: "CreatePodSandbox for pod "-f0fe-11ea-9872-)" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "aiqx-server-dep--b9lmv": operation timeout: context deadline exceeded" Sep 07 11:40:40 kubelet[2899]: W0907 11:40:40. 2899 ] No ref for container "docker://7b9c86b529530f15d941f8e68973a202c8f348ab5db864a8371e8c82ddbf9dc4" (r(-e845-11ea-bd96-) Sep 07 11:40:43 kubelet[2899]: W0907 11:40:43. 2899 ] Unable to create pod sandbox due to conflict. Attempting to remove sandbox "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8" Sep 07 11:40:43 kubelet[2899]: E0907 11:40:43. 2899 ] Failed to remove the conflicting container "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8": Error: No such container: a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
statuses[container.Name] = status } <ins> // Copy the slice before sorting it containerStatusesCopy := make([]*kubecontainer.Status, len(podStatus.ContainerStatuses)) copy(containerStatusesCopy, podStatus.ContainerStatuses) </ins> // Make the latest container status comes first. <del> sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(podStatus.ContainerStatuses))) </del> <ins> sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(containerStatusesCopy))) </ins> // Set container statuses according to the statuses seen in pod status containerSeen := map[string]int{} <del> for _, cStatus := range podStatus.ContainerStatuses { </del> <ins> for _, cStatus := range containerStatusesCopy { </ins> cName := cStatus.Name if _, ok := statuses[cName]; !ok { // This would also ignore the infra container.

"reflect" "sort" "testing" <ins> "time" </ins> "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require"

}) } } <ins> func TestConvertToAPIContainerStatusesDataRace(t *testing.T) { pod := podWithUIDNameNs("12345", "test-pod", "test-namespace") testTimestamp := time.Unix(123456789, 987654321) criStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, ContainerStatuses: []*kubecontainer.Status{ {Name: "containerA", CreatedAt: testTimestamp}, {Name: "containerB", CreatedAt: testTimestamp.Add(1)}, }, } testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kl := testKubelet.kubelet // convertToAPIContainerStatuses is purely transformative and shouldn't alter the state of the kubelet // as there are no synchronisation events in that function (no locks, no channels, ...) each test routine // should have its own vector clock increased independently. Golang race detector uses pure happens-before // detection, so would catch a race condition consistently, despite only spawning 2 goroutines for i := 0; i < 2; i++ { go func() { kl.convertToAPIContainerStatuses(pod, criStatus, []v1.ContainerStatus{}, []v1.Container{}, false, false) }() } } </ins>