jiebi/CodeConvo huggingface dataset:
Code change:
} return feature.DefaultFeatureGate.Enabled(features.JobPodFailurePolicy) && job.Spec.PodFailurePolicy != nil } <ins> func (jm *Controller) backupRemovePodFinalizers(job *batch.Job) { // Listing pods shouldn't really fail, as we are just querying the informer cache. selector, err := metav1.LabelSelectorAsSelector(job.Spec.Selector) if err != nil { utilruntime.HandleError(fmt.Errorf("parsing deleted job selector: %v", err)) return } pods, _ := jm.podStore.Pods(job.Namespace).List(selector) for _, pod := range pods { if metav1.IsControlledBy(pod, job) && hasJobTrackingFinalizer(pod) { jm.enqueueOrphanPod(pod) } } } </ins>

Related discussion:
text: When a Job is deleted (via Background cascade), we have a backup logic to remove finalizers from pods This logic could also be applied for any finished Job. While investigating a case similar to , we observed that the job controller didn't delete finalizers from Pods where the Job was deleted with Foreground cascading policy. Since the Job was already marked as Finished, it only makes sense to trigger the backup logic to remove finalizers, so that we can guard against any possible bug where a Job is marked as Finished but not all pod finalizers are removed. Note that I'm not aware of any other bug in tracking with finalizers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/remove-kind feature /kind cleanup
/wg batch
I am happy to give this a go.
/assign
Just to check if I have the logic chain right: We are talking about adding this logic If so, I should probably turn the backup-logic code into a function that then apply that in both places? I also would be unsure of where in the unit tests this would be tested, are you able to point me to how this is tested in the deletePod function? I assume we add a pod that is never included in the "uncounted" lists.
I would put it in the event handler, to be more of a "catch all" Yes, that's a good idea. I think you would have to force it quite a bit, as we have fixed all the bugs that we have found. For example, from the test: the controller, but do not start the workers (this will setup the event handlers) a job the pods the job the workers some inspiration here
Alright, thank you :))
Should the issue title be: Remove finalizers from Pods belonging to finished Jobs ?
If someone has set a custom finalizer on a Pod, even one owned by a Job, (when) would we remove that finalizer?
/retitle Remove tracking finalizers from Pods belonging to finished Jobs No, only the finalizer specific to job tracking
Code change:
// Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators What happened: In the context of the device plugin, when using init containers in your pod spec, the current behavior will not reuse devices between init containers and regular containers. This is problematic as Pod scheduling is currently wrong: , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is . However, the pod predicate ran on the node is the sum of devices used by regular and init containers. Meaning that your pod can bounce back between the scheduler and a node infinitely. Also given that many of the extended resources are typically more expensive that the built-in ones like CPU, Memory and Storage this bug has very real financial drawbacks in terms of resource usage (e.g: using 4 GPUs on a pod that only uses 2 GPUs at a time). Furthermore, this unexpected behavior was caught during implementation (but we did not catch the scheduling implications) and decision was taken to ignore it for the first implementation. We need to document it for 1.9 What you expected to happen: Devices can be reused by Init Containers Correct resource accounting In terms of the current behavior of the Allocate call to the device plugin, I'd expect the allocate call to be done before every Init containers start and once at the end of the last Init container (for regular containers). /assign
Yep, I realized the inconsistency between the predicate behavior of scheduler and that of node too. Having said that, considering we treat a pod as a whole when predicating currently, it's kind of difficult to reuse the runtime informations between init containers and regular containers. The real crux of the matter might be whether the scenario of requiring extended resources in init containers is common, if so, the issue should be addressed ASAP.
If I remembered correctly, the current design to improve DP includes a reconciliation in admit phase of Pod, it may be able to fix Allocate call for
can you point me to said design ? I'm not sure that the Allocate call for init containers is planned to be fixed as described.
Can you guys please point to the code where is happening. Looks to me that same logic is executed on both the scheduler and the node to calculate a pod's resource request and which is  On the node, function call tree looks like this: Is it possible to add few more words in explanation?
Code change:
// Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Related discussion:
text: PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.
Code change:
// Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Related discussion:
text: do you think if you have time to implement the solution you mentioned in the above comment We can try to get it into a 1.9 sub release. I think for now, lets just support device reuse among initContainers and normal containers. We can revisit this part when we need to guarantee an Allocate call per container.
Right what I am mentioning here is that it is not possible to do that. The point of the device plugin is so that we can execute vendor specific code before making the device available in the container (e.g: Scrub the GPU memory). I don't think the proposed solution addresses this problem.
Sure. But to clarify, I think if we dont't expect big shock on current implementation, we could only provide limited support on the point, that is, without re-generate runtime information for regular containers, only those whose requested resource is equal to or is a subset of that in init containers could make a resuse(e.g. if we have two init containers, each requests 2 devices, then a regular container that request 3 devices still could not reuse the devices.) Besides, if a initcontainer requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them. In sum, since we don't re-generate runtime information, we could only reuse devices per container, not per device, like we do in scheduler. Is that what you mean?
Hmm, that is a good question. I forgot that we now return the combined runtime information in AllocResponse. Then for now, perhaps we should just return an error in device plugin manager Allocate() in case both init containers and containers request the same device plugin resource?
You mean we give up reuse and prohibit users to request same kind of device in init containers and regular containers? that seems a bit weird. And if we decide to do so, we might need to do the check at the first beginning, i.e. in API validation, not wait until it goes into DevicePlugin.
I think for 1.9, we just return an error when both init containers and normal containers of a pod request the same device resource, and document this in the device plugin user guide.
Code change:
// Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Related discussion:
text: The reason is because in the current device plugin implementation, we don't have a good way to free devices used by init containers of an active pod. It is hard to do this kind of validation at API level or even scheduler because we can't know whether an extended resource is device resource or not. So for now, I think we should probably just return a clear error message during admission handler. I am not sure yet whether there will be strong use cases that need to share devices between init containers and normal containers. We can revisit this based on user feedback. If there is strong user demand to support this, something we could do is to differentiate init containers and normal containers in podDevices, and free up devices used by init containers when any normal container is about to run. That would request some code structure change, and we may wait to see whether there is strong demand to support this.
It is hard to do this at API level or even scheduler because we can't know whether an extended resource is device resource or not. Oh, you are right, I ignored the resource awareness issue. I also agree that it's hard to address the issue without big change to current implementation. But I still feel the inconsistency of behavior between API and kubelet might confusing users. However, its effect should be sustainable, since we haven't received feedback about init containers up to now.
Right. So for now, perhaps just return an error when this happens, to give clear error signals.
Oh, I'm afraid we miss something, the check may not work as expected. As I commented, because init-containers run sequentially, we did reuse between init containers in , that is, if we have two init containers, each request 1 device, then we expect 1 device in total in scheduler, but expect 2 in deviceplugin. The check could not cover the scenario.
Thinking about this problem a bit more, I wonder whether we may tackle this issue from the following approaches, ordered by their complexities: 1) Just return an error whenever an init container requests a device plugin resource, and document in the device plugin user guide that requiring device plugin resource in init containers are not supported. This may be the short-term change we can try to get into 1.9. 2) Maintain a list of devices allocated to init containers in ManagerImpl.Allocate().
Code change:
// Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Related discussion:
text: Try to allocate from this list first for later init containers and normal containers. The required no. of devices we pass to ManagerImpl.devicesToAllocate() should subtract the no. of devices we have allocated to init containers. This may require some code structure change around ManagerImpl.Allocate(), ManagerImpl.allocateContainerResources(), and ManagerImpl.devicesToAllocate(). However, Allocate gRPC call is still issued during Admission all together, and it is possible that we issue an Allocate gRPC for [dev1] (say dev1 allocated to initContianer1), followed by another Allocate gRPC call for [dev1, dev2] (say dev1 and dev2 allocated to initContainer2), with none of containers started or finished. This assumes containers running in the same pod share the same security context, and device plugin doesn't need to perform per-container allocation operations. We just use Allocate gRPC call to learn container runtime settings. 3) Take a further step from 2), during Admission, we just determine what devices to allocate to containers without issuing the actual Allocate gRPC calls. I.e., we only fill deviceIds field in podDevices.deviceAllocateInfo but leaves allocResp field empty. During ManagerImpl.GetDeviceRunContainerOptions(), we check for this case and issue the actual Allocate gRPC call for the container. This ensures we can issue a gRPC Allocate call per container. However, if a pod is restarted multiple times, we don't guarantee Allocate gRPC call will be issued during subsequent pod runs. 4) Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. My suggestion is to implement 1) now and gather some user feedbacks to determine whether we need to implement the following approaches.
Code change:
// Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Related discussion:
text: I see this a very big limitation. Special devices/resources in init containers is a very common use case IMO. Therefore we should try to fix this. This approach perfectly makes sense to me. What is your concern in implementing this? Even today, IIUC, Allocate call is made per container. Is there any other value that this approach is adding over 1) ? I am not able to see any gains with 2) or 3) over 1). Comments please.
I feel we may not be able to implement 2) for 1.9 timeframe. Option 2) also doesn't address the case mentioned above: Option 3) covers that case for pods without restarts. Option 4) fully covers the requirement we we will scrub old container data before giving it to a new container, even if they both run in the same pod, bug it does have the mentioned drawback during device plugin failure.
Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. How about making the allocation call optional? That is , if we find a contaienr using re-allocated devices, we'd try to issue an Allocate gRPC call first, and still use the cached setting if allocation failed. And, if we decide not to address the issue in v1.9, which means we have enough time to work on it later, I think approach 2) might not be a good choice. Because it has a premise that devices are stateless, recycle actions before reuse(e.g: Scrub the GPU memory mentioned above) could not be supported. And this way, as I commented above, we could only reuse devices per container, not per device, like we do in scheduler.
Code change:
// Returns list of device Ids we need to allocate with Allocate rpc call. // Returns empty list in case we don't need to issue the Allocate rpc call. <del> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int) (sets.String, error) { </del> <ins> func (m *ManagerImpl) devicesToAllocate(podUID, contName, resource string, required int, reusableDevices sets.String) (sets.String, error) { </ins> m.mutex.Lock() defer m.mutex.Unlock() needed := required

Related discussion:
text: I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).