Code change:
return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

Related discussion:
text: PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.