Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Pod worker panic when trying to re-allocating the device plugin resource during the node restart. The issue is potentially caused by an un-matching pod id. The map will be later used in an , which caused panic. ( func by itself is not thread-safe) Stack trace: DeviceManager Allocate() should be able to handle such race safely. Customer reported issue after upgrading to 1.18+. This is the new behavior introduced since 1.18+ from . Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/assign looking into the issue
I think your PR is related to this issue.
/unassign seems pretty relatable to PR cc
/triage accepted /priority critical-urgent
/remove-priority critical-urgent I don't think this is a release blocker at this point given the history reading through ; we can fix this in a cherry-pick.
Hi all, We are the GCP/GKE customer from which this issue originates. I can provide more information: it happened multiple times on our production cluster since we upgraded to 1.18 it happens in 2 scenarios: when GCP has a "host maintenance" event on our VMs with GPU (they don't support VM live migrations when GPU are involved), it forces a VM reboot: when multiple pods on that k8s node request a GPU prior to the reboot, we sometimes get the kubelet panic on node restart. when GKE upgrades our node-pools: new nodes (fresh VMs) get multiple scheduled pods requesting a GPU, sometimes leading to the kubelet panic. =this happens outside of our control The end result is a Pod scheduled on a node, but with no container ever created: our production workload is effectively broken until we manually delete the stuck pods. More details about our device usage: we use a of the each node has one GPU, and our device plugin declares N devices per real GPU; nvidia/cuda supports multi processing so it effectively shares the GPU among N pods/containers. (our workload is memory-gpu bound, and uses <1/Nth of the gpu memory each, so no conflict) it worked great for almost 3 years now, with N=8
/cc
Now that we have confirmation that it occurs in live cluster I will take a closer look later this week.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

Discussion:
return nil } <ins> // checkPodActive checks if the given pod is still in activePods list func (m *ManagerImpl) checkPodActive(pod *v1.Pod) bool { activePods := m.activePods() for _, activePod := range activePods { if activePod.UID == pod.UID { return true } } return false } </ins> // GetDeviceRunContainerOptions checks whether we have cached containerDevices // for the passed-in <pod, container> and returns its DeviceRunContainerOptions // for the found one. An empty struct is returned in case no cached state is found.

if err != nil { return nil, err } <ins> if !m.checkPodActive(pod) { klog.ErrorS(nil, "pod deleted from activePods, skip to reAllocate", "podUID", podUID) continue } </ins> // This is a device plugin resource yet we don't have cached // resource state. This is likely due to a race during node // restart. We re-issue allocate request to cover this race.

} <ins> func TestGetDeviceRunContainerOptions(t *testing.T) { res1 := TestResource{ resourceName:     "domain1.com/resource1", resourceQuantity: *resource.NewQuantity(int64(2), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev1", "dev2"}}, topology:         true, } res2 := TestResource{ resourceName:     "domain2.com/resource2", resourceQuantity: *resource.NewQuantity(int64(1), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev3", "dev4"}}, topology:         false, } testResources := make([]TestResource, 2) testResources = append(testResources, res1) testResources = append(testResources, res2) podsStub := activePodsStub{ activePods: []*v1.Pod{}, } as := require.New(t) tmpDir, err := ioutil.TempDir("", "checkpoint") as.Nil(err) defer os.RemoveAll(tmpDir) testManager, err := getTestManager(tmpDir, podsStub.getActivePods, testResources) as.Nil(err) pod1 := makePod(v1.ResourceList{ v1.ResourceName(res1.resourceName): res1.resourceQuantity, v1.ResourceName(res2.resourceName): res2.resourceQuantity, }) pod2 := makePod(v1.ResourceList{ v1.ResourceName(res2.resourceName): res2.resourceQuantity, }) activePods := []*v1.Pod{pod1, pod2} podsStub.updateActivePods(activePods) err = testManager.Allocate(pod1, &pod1.Spec.Containers[0]) as.Nil(err) err = testManager.Allocate(pod2, &pod2.Spec.Containers[0]) as.Nil(err) // when pod is in activePods, GetDeviceRunContainerOptions should return runContainerOpts, err := testManager.GetDeviceRunContainerOptions(pod1, &pod1.Spec.Containers[0]) as.Nil(err) as.Equal(len(runContainerOpts.Devices), 3) as.Equal(len(runContainerOpts.Mounts), 2) as.Equal(len(runContainerOpts.Envs), 2) activePods = []*v1.Pod{pod2} podsStub.updateActivePods(activePods) testManager.UpdateAllocatedDevices() // when pod is removed from activePods,G etDeviceRunContainerOptions should return error runContainerOpts, err = testManager.GetDeviceRunContainerOptions(pod1, &pod1.Spec.Containers[0]) as.Nil(err) as.Nil(runContainerOpts) } </ins> func TestInitContainerDeviceAllocation(t *testing.T) { // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers.