Code change:
as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

Related discussion:
text: There are three containers in pod, cont-init(requested 2 unit or resource X), cont-reg1(requested 1 unit of resource X) and cont-reg2(requested 1 unit of resource X). Now in approach 2), gRPC Allocate call is made for cont-init requesting X1 and X2. Next, gRPC Allocate call is made for cont-reg1. This time resource id will be send in Allocate from the previously allocated [X1, X2]. Lets say X1 is requested for cont-reg1 Next, gRPC Allocate for cont-reg2 and resource with ID X2. All this allocation info gets stored in internal cache. Now when container runs, runtimeOptions will be collected from internal cache. Where do you see a problem in this flow?
/cc
Yep, it could work if the plugin behavior the way you described, but I'm not sure if all kinds of devices could support "pre-allocation", that is, doing allocation on a same device multiple times at the same time, to prepare runtime informations for several containers that reuse the device later. Also I have no idea if all plugin developers consider their plugins "stateless", that is, managing initialization in init-containers. As I don't have enough knowledge on hardware, would like to hear opinions from other guys :)
This would defeat the purpose of a container. In the case of GPUs, when you run a container you expect the GPU to always be in the same state. Furthermore GPUs can't be shared (exclusive usage) between containers meaning that Init containers should behave in the same way as regular containers. With the only difference being that they are executed sequentially and before regular containers. Additionally, operations to setup a device before making them available in a container, whether functionally (load the driver, insert libraries, ....) or security wise (scrub the memory), should be handled by the infrastructure not the users pod spec.
I agree that it's a sane behavior to forbid the use of Device Plugin Resources in Init Containers for the 1.9 release
moving forward,
/cc I think what long-term solution we decide to implement will largely depend on use-case requirements from different device plugins. Would like to hear some thoughts from different device plugin developers on this. Can people cc-ed in this issue take a look at the proposed options mentioned in and provide some feedbacks on which one may better fit your use cases?