Change:
It is suspected to be a concurrent bug of kubelet. When we delete the last pod that requires GPU on the node and create two or more pods that require GPU at the same time, kubelet may panic on as following. I think it's a concurrency issue. a pod with GPU request, then kubelet will allocate resource for it. Kubelet try call device plugin to choose preferred device for it. It will unlock the mutex!. we delete the last pod that requires GPU. the second pod with GPU request, kubelet will allocate resource for it also. Kubelet will call to recalculate device (delete allocated device if pod deleted). It will update the allocatedDevices map of ManagerImpl, because of mutex is unlock by step 1. Kubelet continues with the first pod. If allocatedDevices has changed and GPU device is nil, kubelet will panic in the closure allocateRemainingFrom. Don't panic. Delete the last pod that requires GPU on the node and create two or more pods that require GPU at the same time. No response <details</details<detailsexternal </details<details</detailsNOOP <detailsContainerd </detailsNOOP
/assign
/sig node
Is this reproduced in the master branch?
It is possible from the code. I will try to reproduce it later. Hope someone can give me some ideas also.
/cc Could you take a look? The issue scenario looks real to me.
hi, this looks legit, but OTOH a key assumption in the resource management allocation is that all the pods are admitted serially (e.g. NOT concurrently) one after the another. In other words, a key assumption is we never try allocating resources while another allocation is ongoing, e.g. not completed. So is not obvious to me which flows are conflicting. [EDIT] could still be if there is a client accessing the podresources API:
/triage accepted /priority important-soon I think the issue described here is real but I'm not sure how often it appears. Panics are always bad, but how often do we know this is happening? Let's start with as priority [EDIT] subejct to further adjustments as the investigation progresses.
Sure. Kubelet will allocate resources to pods when admitting. Admit is sequential. As long as it is not device re-allocate, the problem will not occur. Podresources API is suspicious.
I deployed the dcgm exporter and it does call the interface of podresources.
This is a genuine and reproducible issue. We have observed the kubelet crashing due to such a panic when inserting a device into a map (): In our observation, right after deleting the pod, the podresources API was called. This removed the state of for the GPU resource. This happened when the was called as soon as the mutex got unlocked in the . Moreover, such a race seems being triggered when Device Plugin periodically calls podresources API.
Thanks, this information matches with other findings. The podresources endpoint intentionally triggers to report the current up to date allocation; otherwise the device manager would clean up lazily the allocation state when a new admission is attempted. We will work on a fix.
Just to add we've also seen this a few times on 1.23 and 1.24 clusters in EKS with the nvidia-device-plugin + DCGM-Exporter
This looks like a real problem. I unit tests to verify that the PR works.
not taking for 1.29, but may consider for a cherry-pick later. Depending on the fix complexity
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted /priority important-longterm because low capacity
/remove-priority important-soon

Discussion:
// Create a closure to help with device allocation // Returns 'true' once no more devices need to be allocated. allocateRemainingFrom := func(devices sets.Set[string]) bool { <ins> // When we call callGetPreferredAllocationIfAvailable below, we will release // the lock and call the device plugin. If someone calls ListResource concurrently, // device manager will recalculate the allocatedDevices map. Some entries with // empty sets may be removed, so we reinit here. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.New[string]() } </ins> for device := range devices.Difference(allocated) { m.allocatedDevices[resource].Insert(device) allocated.Insert(device)

return false } <del> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.New[string]() } </del> // Allocates from reusableDevices list first. if allocateRemainingFrom(reusableDevices) { return allocated, nil

} <ins> func TestDevicesToAllocateConflictWithUpdateAllocatedDevices(t *testing.T) { podToAllocate := "podToAllocate" containerToAllocate := "containerToAllocate" podToRemove := "podToRemove" containerToRemove := "containerToRemove" deviceID := "deviceID" resourceName := "domain1.com/resource" socket := filepath.Join(os.TempDir(), esocketName()) devs := []*pluginapi.Device{ {ID: deviceID, Health: pluginapi.Healthy}, } p, e := esetup(t, devs, socket, resourceName, func(n string, d []pluginapi.Device) {}) waitUpdateAllocatedDevicesChan := make(chan struct{}) waitSetGetPreferredAllocChan := make(chan struct{}) p.SetGetPreferredAllocFunc(func(r *pluginapi.PreferredAllocationRequest, devs map[string]pluginapi.Device) (*pluginapi.PreferredAllocationResponse, error) { waitSetGetPreferredAllocChan <- struct{}{} <-waitUpdateAllocatedDevicesChan return &pluginapi.PreferredAllocationResponse{ ContainerResponses: []*pluginapi.ContainerPreferredAllocationResponse{ { DeviceIDs: []string{deviceID}, }, }, }, nil }) testManager := &ManagerImpl{ endpoints:             make(map[string]endpointInfo), healthyDevices:        make(map[string]sets.Set[string]), unhealthyDevices:      make(map[string]sets.Set[string]), allocatedDevices:      make(map[string]sets.Set[string]), podDevices:            newPodDevices(), activePods:            func() []*v1.Pod { return []*v1.Pod{} }, sourcesReady:          &sourcesReadyStub{}, topologyAffinityStore: topologymanager.NewFakeManager(), } testManager.endpoints[resourceName] = endpointInfo{ e: e, opts: &pluginapi.DevicePluginOptions{ GetPreferredAllocationAvailable: true, }, } testManager.healthyDevices[resourceName] = sets.New[string](deviceID) testManager.podDevices.insert(podToRemove, containerToRemove, resourceName, nil, nil) go func() { <-waitSetGetPreferredAllocChan testManager.UpdateAllocatedDevices() waitUpdateAllocatedDevicesChan <- struct{}{} }() set, err := testManager.devicesToAllocate(podToAllocate, containerToAllocate, resourceName, 1, sets.New[string]()) assert.NoError(t, err) assert.Equal(t, set, sets.New[string](deviceID)) } </ins> func TestGetDeviceRunContainerOptions(t *testing.T) { res1 := TestResource{ resourceName:     "domain1.com/resource1",