Change:
The following behaviour was reproduced on openshift 4.10 ("Kubelet version" kubeletVersion="v1.23.3+") but I believe the root cause is on kubernetes. Let's consider a pod requiring resources, among them a device (point in case: dpdk network device) the pods starts and run just fine up until the node is restarted. On restart, the device manager correctly recover its state from the checkpoint data: As expected, the device plugin capacity is set to zero until the device plugin correctly re-register itself: A bit later, the recovery process from the apiserver begins: we can see already that the workload pod (testpmd) is going to be recovered BEFORE the device plugin. When the workload pod enters the admission flow (as per recovered pods), because the devicemanager restored its state and has data about this pod, we see: When the topology manager requests device allocation, we end up in the devicemanager's function. This function checks if there already a pre-allocated device (and there is, thanks to the recovered data), and if so exits early, skipping the health check which would have failed, because the device plugin has not re-registered itself yet. The end result is that containers can be admitted and start running without the requested device actually available, causing runtime errors in the workload. The kubelet should reject the admission of the pod until the device plugins re-registered themselves; then the pods requiring devices should be processed again. This seems the only meaningful way to recover pods while taking in consideration the actualy health of the devices. The behaviour here depends on the order on which pods are recovered, so it's essentially a race. a test pod requiring devices. Make sure this test pod clearly fails if the requested devices are not available the node (can take few attempts) the health of the test pod No response <details</details<detailsbare metal </details<details</details<details</details<details</details<details</details
/sig node
/triage accepted /priority important-soon
Changing the priority to important-longterm as we should consider the kubelet restart best practices here. We need to look into whether it is reasonable to restart kubelet without draining/cordoning the node. /priority important-longterm
/remove-priority important-soon
Hi Is this Issue open to work on. I am new to k8s would like to work on this .
Well this issue require very specific kubelet knowledge so I don't recommend it as first issue
I think that's how I will learn . Is this issue time bound ? Can you help me with areas I will have to learn about to work on this. Else do you have something else I can start with . All the good-first-issue are assigned to someone.
Sure, maybe just have a look to issue tagged "help wanted" and/or "good first issue" to see if there is something friendlier available; if not, and/or if you have interest in this specific area, we can work this out. For example:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
A possible workaround, to be evaluated, could be to remove the device manager state file at node boot time, and only at node boot time. We need to keep the state file across kubelet (but not node) reboots.
I managed to reproduce this issue on a kind based Kubernetes cluster (v1.24.7) so the root cause is certainly Kubernetes. I Will provide additional updates as I have 'em!
I have been spending some time to get a better understanding of this issue and want to capture my learning here: On node restart, here is the flow of events on kubelet: Manager . we haven't removed the checkpoint file, is called using which the following information is populated: podDevices (which contains pod to allocated device mapping) m.allocatedDevices which stores the list of allocated devices. registered device plugin resources the registered devices plugins resources obtained in the step above, healthyDevices/unhealthyDevices : sets.NewString(). part of the Node sync loop, is called where we inspect both healthy (and unhealthy devices). Because of the point above, capacity of of the device plugin resource is updated to the length of healthy (and unhealthy) devices present on the system which is 0 in this case as all the devices have been reset. This is done to allow the device plugin to re-register itself so that capacity can be updated appropriately. a scenario where we have an application pod requesting a device provided by the device plugin pod coming before device plugin pod has had a chance to start or re-register itself, we reach the allocation phase for the application pod: - -. deviceToAllocate, we obtain podDevices which returns the devices that were previously allocated to the application pod means that the same number of devices are being requested as were previously allocated making needed=0 without any obvious error. I tried but it didn't solve the issue. This is because healthyDevices are reset to empty set of strings keyed by resource name (step 3 above) because of which the evaluates to true as the device plugin had registered itself prior to the node reboot). In order to fix this, we need to check that we have healthy resources to fulfill the pod request. If we don't have healthy devices, the pod admission should fail. I am working on it and will be testing the fix in the coming days.
xref: possibly related
Thanks for pointing to this PR, it is definitely related.
reopening since is reverted in
/triage accepted
This sounds like this adds additional requirements to kubelet admission - the ability to defer for retry admission for some time, rather than rejecting right away? When you say "reject" are you returning a failure in kubelet admission such that the pod is marked failed? Or do you mean reject in a different way?
IMO that would be very nice and handy. If kubelet knows it can admit a pod, but not just yet, I think that would be a nice addition. Cases I can think of: very case: pods requesting devices, it the device plugin pod is not recovered yet the application pod should either be postponed or fail admission and retry later in progress, kubelet knows resources are about to be freed /available soon enough, not yet though I think the best we can have now is a hard reject: pod fails admission and is marked failed. This is what I suggested as desired behavior and what the current fix implements. In general though this can be improved, for example retrying the admission later.
Ok, that's useful. I'll add that to the tracking list for improving admission - the problem of "i can't admit this now but probably can later" comes in with other places (apparmor, waiting for pods to be gracefully deleted). Thanks
I agree that this kind of behavior would be useful but it is worth mentioning that currently we rely on the fact we don't use naked pods and typically a pod is part of a deployment. New instances would be spinned up and at some stage a pod would succeed at admission and that's when the deployment would succeed. EDIT: The selling point of the semantic of "i can't admit this now but probably can later" is that we don't end up with runaway pod creation due to attempts of having a successful deployment.
Hi I am now going through this issue and the related PRs and cherry-picks. May I ask: have you taken into account the case when there are running workloads using some devices and only kubelet is restarted? In KubeVirt project, we now see a regression in such a scenario when running on Kubernetes (with the change ). KubeVirt runs virtual machines inside pods and uses a device plugin to advertise e.g. on the nodes. Now if I run any VM and then , then the workload pod gets terminated with . Is it now the expected behavior? Or do we just miss some handling / hit a bug?
A gentle ping here. With the fix , if kubelet is restarted on a node, then all the existing and running workloads that use devices are terminated: This also affects 1.25.x and 1.26.x branches, since the fix has been cherry-picked there. I guess this happens because of the race between device plugin registration and the code in . Anyway, this does not seem right to me. A potential restart of kubelet should not interrupt the running workloads, right? Any thought on that?
Yes, something is not fine here and this deserve discussion, please file a separate issue, backlinking this one, for visibility. Thanks!

Discussion:
return nil, fmt.Errorf("pod %q container %q changed request for resource %q from %d to %d", string(podUID), contName, resource, devices.Len(), required) } } <del> if needed == 0 { // No change, no work. return nil, nil } </del> klog.V(3).InfoS("Need devices to allocate for pod", "deviceNumber", needed, "resourceName", resource, "podUID", string(podUID), "containerName", contName) <ins> healthyDevices, hasRegistered := m.healthyDevices[resource] </ins> // Check if resource registered with devicemanager <del> if _, ok := m.healthyDevices[resource]; !ok { </del> <ins> if !hasRegistered { </ins> return nil, fmt.Errorf("can't allocate unregistered device %s", resource) } <ins> // Check if registered resource has healthy devices if healthyDevices.Len() == 0 { return nil, fmt.Errorf("can't allocate unhealthy devices %s", resource) } // Check if all the previously allocated devices are healthy if !healthyDevices.IsSuperset(devices) { return nil, fmt.Errorf("previously allocated devices are no longer healthy; can't allocate unhealthy devices %s", resource) } if needed == 0 { // No change, no work. return nil, nil } </ins> // Declare the list of allocated devices. // This will be populated and returned below. allocated := sets.NewString()

} <ins> func TestPodContainerDeviceToAllocate(t *testing.T) { resourceName1 := "domain1.com/resource1" resourceName2 := "domain2.com/resource2" resourceName3 := "domain2.com/resource3" as := require.New(t) tmpDir, err := os.MkdirTemp("", "checkpoint") as.Nil(err) defer os.RemoveAll(tmpDir) testManager := &ManagerImpl{ endpoints:        make(map[string]endpointInfo), healthyDevices:   make(map[string]sets.String), unhealthyDevices: make(map[string]sets.String), allocatedDevices: make(map[string]sets.String), podDevices:       newPodDevices(), } testManager.podDevices.insert("pod1", "con1", resourceName1, constructDevices([]string{"dev1", "dev2"}), constructAllocResp(map[string]string{"/dev/r2dev1": "/dev/r2dev1", "/dev/r2dev2": "/dev/r2dev2"}, map[string]string{"/home/r2lib1": "/usr/r2lib1"}, map[string]string{"r2devices": "dev1 dev2"})) testManager.podDevices.insert("pod2", "con2", resourceName2, checkpoint.DevicesPerNUMA{nodeWithoutTopology: []string{"dev5"}}, constructAllocResp(map[string]string{"/dev/r1dev5": "/dev/r1dev5"}, map[string]string{"/home/r1lib1": "/usr/r1lib1"}, map[string]string{})) testManager.podDevices.insert("pod3", "con3", resourceName3, checkpoint.DevicesPerNUMA{nodeWithoutTopology: []string{"dev5"}}, constructAllocResp(map[string]string{"/dev/r1dev5": "/dev/r1dev5"}, map[string]string{"/home/r1lib1": "/usr/r1lib1"}, map[string]string{})) // no healthy devices for resourceName1 and devices corresponding to // resource2 are intentionally omitted to simulate that the resource // hasn't been registered. testManager.healthyDevices[resourceName1] = sets.NewString() testManager.healthyDevices[resourceName3] = sets.NewString() // dev5 is no longer in the list of healthy devices testManager.healthyDevices[resourceName3].Insert("dev7") testManager.healthyDevices[resourceName3].Insert("dev8") testCases := []struct { description              string podUID                   string contName                 string resource                 string required                 int reusableDevices          sets.String expectedAllocatedDevices sets.String expErr                   error }{ { description:              "Admission error in case no healthy devices to allocate present", podUID:                   "pod1", contName:                 "con1", resource:                 resourceName1, required:                 2, reusableDevices:          sets.NewString(), expectedAllocatedDevices: nil, expErr:                   fmt.Errorf("can't allocate unhealthy devices %s", resourceName1), }, { description:              "Admission error in case resource is not registered", podUID:                   "pod2", contName:                 "con2", resource:                 resourceName2, required:                 1, reusableDevices:          sets.NewString(), expectedAllocatedDevices: nil, expErr:                   fmt.Errorf("can't allocate unregistered device %s", resourceName2), }, { description:              "Admission error in case resource not devices previously allocated no longer healthy", podUID:                   "pod3", contName:                 "con3", resource:                 resourceName3, required:                 1, reusableDevices:          sets.NewString(), expectedAllocatedDevices: nil, expErr:                   fmt.Errorf("previously allocated devices are no longer healthy; can't allocate unhealthy devices %s", resourceName3), }, } for _, testCase := range testCases { allocDevices, err := testManager.devicesToAllocate(testCase.podUID, testCase.contName, testCase.resource, testCase.required, testCase.reusableDevices) if !reflect.DeepEqual(err, testCase.expErr) { t.Errorf("DevicePluginManager error (%v). expected error: %v but got: %v", testCase.description, testCase.expErr, err) } if !reflect.DeepEqual(allocDevices, testCase.expectedAllocatedDevices) { t.Errorf("DevicePluginManager error (%v). expected error: %v but got: %v", testCase.description, testCase.expectedAllocatedDevices, allocDevices) } } } </ins> func TestGetDeviceRunContainerOptions(t *testing.T) { res1 := TestResource{ resourceName:     "domain1.com/resource1",

import ( "context" <ins> "errors" </ins> "fmt" "os" "path/filepath"

"time" v1 "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/resource" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/uuid" runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1" "k8s.io/klog/v2" </ins> kubeletpodresourcesv1 "k8s.io/kubelet/pkg/apis/podresources/v1" "k8s.io/kubernetes/pkg/kubelet/apis/podresources" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager"

e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" <ins> e2etestfiles "k8s.io/kubernetes/test/e2e/framework/testfiles" testutils "k8s.io/kubernetes/test/utils" </ins> "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" <ins> "github.com/onsi/gomega/gcustom" "github.com/onsi/gomega/types" </ins> ) const (

}) }) <ins> /* This end to end test is to simulate a scenario where after kubelet restart/node reboot application pods requesting devices appear before the device plugin pod exposing those devices as resources. The happy path is where after node reboot/ kubelet restart, the device plugin pod appears before the application pod. This PR and this e2e test aims to tackle the scenario where device plugin either does not appear first or doesn't get the chance to re-register itself. Since there is no way of controlling the order in which the pods appear after kubelet restart/node reboot, we can't guarantee that the application pod recovers before device plugin pod (the scenario we want to exercise here). If the device plugin pod is recovered before the test pod, we still can meaningfully reproduce the scenario by NOT sending the registration command. To do so sample device plugin is enhanced. For implementation details, refer to: `test/images/sample-device-plugin/sampledeviceplugin.go`. This enhancement allows auto-registration of the plugin to be controlled with the help of an environment variable: REGISTER_CONTROL_FILE. By default this environment variable is not present and the device plugin autoregisters to kubelet. For this e2e test, we use sample device plugin spec with REGISTER_CONTROL_FILE=/var/lib/kubelet/device-plugins/sample/registration to allow manual registeration of the plugin to allow an application pod (requesting devices) to successfully run on the node followed by kubelet restart where device plugin doesn't register and the application pod fails with admission error. Breakdown of the steps implemented as part of this e2e test is as follows: 1. Create a file `registration` at path `/var/lib/kubelet/device-plugins/sample/` 2. Create sample device plugin with an environment variable with `REGISTER_CONTROL_FILE=/var/lib/kubelet/device-plugins/sample/registration` that waits for a client to delete the control file. 3. Trigger plugin registeration by deleting the abovementioned directory. 4. Create a test pod requesting devices exposed by the device plugin. 5. Stop kubelet. 6. Remove pods using CRI to ensure new pods are created after kubelet restart. 7. Restart kubelet. 8. Wait for the sample device plugin pod to be running. In this case, the registration is not triggered. 9. Ensure that resource capacity/allocatable exported by the device plugin is zero. 10. The test pod should fail with `UnexpectedAdmissionError` 11. Delete the test pod. 12. Delete the sample device plugin pod. 13. Remove `/var/lib/kubelet/device-plugins/sample/` and its content, the directory created to control registration */ ginkgo.Context("With sample device plugin", func() { var deviceCount int = 2 var devicePluginPod *v1.Pod var triggerPathFile, triggerPathDir string // this test wants to reproduce what happened in https://github.com/kubernetes/kubernetes/issues/109595 ginkgo.BeforeEach(func(ctx context.Context) { ginkgo.By("Wait for node to be ready") gomega.Eventually(ctx, e2enode.TotalReady). WithArguments(f.ClientSet). WithTimeout(time.Minute). Should(gomega.BeEquivalentTo(1)) ginkgo.By("Setting up the directory and file for controlling registration") triggerPathDir = filepath.Join(devicePluginDir, "sample") if _, err := os.Stat(triggerPathDir); errors.Is(err, os.ErrNotExist) { err := os.Mkdir(triggerPathDir, os.ModePerm) if err != nil { klog.Errorf("Directory creation %s failed: %v ", triggerPathDir, err) panic(err) } klog.InfoS("Directory created successfully") triggerPathFile = filepath.Join(triggerPathDir, "registration") if _, err := os.Stat(triggerPathFile); errors.Is(err, os.ErrNotExist) { _, err = os.Create(triggerPathFile) if err != nil { klog.Errorf("File creation %s failed: %v ", triggerPathFile, err) panic(err) } } } ginkgo.By("Scheduling a sample device plugin pod") data, err := e2etestfiles.Read(SampleDevicePluginControlRegistrationDSYAML) if err != nil { framework.Fail(err.Error()) } ds := readDaemonSetV1OrDie(data) dp := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: SampleDevicePluginName, }, Spec: ds.Spec.Template.Spec, } devicePluginPod = e2epod.NewPodClient(f).CreateSync(ctx, dp) go func() { // Since autoregistration is disabled for the device plugin (as REGISTER_CONTROL_FILE // environment variable is specified), device plugin registration needs to be triggerred // manually. // This is done by deleting the control file at the following path: // `/var/lib/kubelet/device-plugins/sample/registration`. defer ginkgo.GinkgoRecover() framework.Logf("Deleting the control file: %q to trigger registration", triggerPathFile) err := os.Remove(triggerPathFile) framework.ExpectNoError(err) }() ginkgo.By("Waiting for devices to become available on the local node") gomega.Eventually(ctx, isNodeReadyWithSampleResources). WithArguments(f). WithTimeout(5 * time.Minute). Should(BeReady()) framework.Logf("Successfully created device plugin pod") devsLen := int64(deviceCount) // shortcut ginkgo.By("Waiting for the resource exported by the sample device plugin to become available on the local node") gomega.Eventually(ctx, isNodeReadyWithAllocatableSampleResources). WithArguments(f, devsLen). WithTimeout(5 * time.Minute). Should(HaveAllocatableDevices()) }) ginkgo.It("should deploy pod consuming devices first but fail with admission error after kubelet restart in case device plugin hasn't re-registered", func(ctx context.Context) { var err error podCMD := "while true; do sleep 1000; done;" ginkgo.By(fmt.Sprintf("creating a pods requiring %d %q", deviceCount, SampleDeviceResourceName)) pod := makeBusyboxDeviceRequiringPod(SampleDeviceResourceName, podCMD) testPod := e2epod.NewPodClient(f).CreateSync(ctx, pod) ginkgo.By("making sure all the pods are ready") err = e2epod.WaitForPodCondition(ctx, f.ClientSet, testPod.Namespace, testPod.Name, "Ready", 120*time.Second, testutils.PodRunningReady) framework.ExpectNoError(err, "pod %s/%s did not go running", testPod.Namespace, testPod.Name) framework.Logf("pod %s/%s running", testPod.Namespace, testPod.Name) ginkgo.By("stopping the kubelet") startKubelet := stopKubelet() ginkgo.By("stopping all the local containers - using CRI") rs, _, err := getCRIClient() framework.ExpectNoError(err) sandboxes, err := rs.ListPodSandbox(ctx, &runtimeapi.PodSandboxFilter{}) framework.ExpectNoError(err) for _, sandbox := range sandboxes { gomega.Expect(sandbox.Metadata).ToNot(gomega.BeNil()) ginkgo.By(fmt.Sprintf("deleting pod using CRI: %s/%s -> %s", sandbox.Metadata.Namespace, sandbox.Metadata.Name, sandbox.Id)) err := rs.RemovePodSandbox(ctx, sandbox.Id) framework.ExpectNoError(err) } ginkgo.By("restarting the kubelet") startKubelet() ginkgo.By("waiting for the kubelet to be ready again") // Wait for the Kubelet to be ready. gomega.Eventually(ctx, e2enode.TotalReady). WithArguments(f.ClientSet). WithTimeout(2 * time.Minute). Should(gomega.BeEquivalentTo(1)) ginkgo.By("making sure all the pods are ready after the recovery") var devicePluginPodAfterRestart *v1.Pod devicePluginPodAfterRestart, err = e2epod.NewPodClient(f).Get(ctx, devicePluginPod.Name, metav1.GetOptions{}) framework.ExpectNoError(err) err = e2epod.WaitForPodCondition(ctx, f.ClientSet, devicePluginPodAfterRestart.Namespace, devicePluginPodAfterRestart.Name, "Ready", 120*time.Second, testutils.PodRunningReady) framework.ExpectNoError(err, "pod %s/%s did not go running", devicePluginPodAfterRestart.Namespace, devicePluginPodAfterRestart.Name) framework.Logf("pod %s/%s running", devicePluginPodAfterRestart.Namespace, devicePluginPodAfterRestart.Name) ginkgo.By("Waiting for the resource capacity/allocatable exported by the sample device plugin to become zero") // The device plugin pod has restarted but has not re-registered to kubelet (as AUTO_REGISTER= false) // and registration wasn't triggered manually (by writing to the unix socket exposed at // `/var/lib/kubelet/device-plugins/registered`). Because of this, the capacity and allocatable corresponding // to the resource exposed by the device plugin should be zero. gomega.Eventually(ctx, isNodeReadyWithAllocatableSampleResources). WithArguments(f, int64(0)). WithTimeout(5 * time.Minute). Should(HaveAllocatableDevices()) ginkgo.By("Checking that pod requesting devices failed to start because of admission error") // NOTE: The device plugin won't re-register again and this is intentional. // Because of this, the testpod (requesting a device) should fail with an admission error. gomega.Eventually(ctx, getPod). WithArguments(f, testPod.Name). WithTimeout(time.Minute). Should(HaveFailedWithAdmissionError(), "the pod succeeded to start, when it should fail with the admission error") ginkgo.By("removing application pods") e2epod.NewPodClient(f).DeleteSync(ctx, testPod.Name, metav1.DeleteOptions{}, 2*time.Minute) }) ginkgo.AfterEach(func(ctx context.Context) { ginkgo.By("Deleting the device plugin pod") e2epod.NewPodClient(f).DeleteSync(ctx, devicePluginPod.Name, metav1.DeleteOptions{}, time.Minute) ginkgo.By("Deleting the directory and file setup for controlling registration") err := os.RemoveAll(triggerPathDir) framework.ExpectNoError(err) ginkgo.By("Deleting any Pods created by the test") l, err := e2epod.NewPodClient(f).List(context.TODO(), metav1.ListOptions{}) framework.ExpectNoError(err) for _, p := range l.Items { if p.Namespace != f.Namespace.Name { continue } framework.Logf("Deleting pod: %s", p.Name) e2epod.NewPodClient(f).DeleteSync(ctx, p.Name, metav1.DeleteOptions{}, 2*time.Minute) } ginkgo.By("Waiting for devices to become unavailable on the local node") gomega.Eventually(ctx, isNodeReadyWithoutSampleResources). WithArguments(f). WithTimeout(5 * time.Minute). Should(BeReady()) }) }) </ins> }) func compareSRIOVResources(expected, got *sriovData) {

sort.Strings(entries) return strings.Join(entries, ", ") } <ins> func makeBusyboxDeviceRequiringPod(resourceName, cmd string) *v1.Pod { podName := "device-manager-test-" + string(uuid.NewUUID()) rl := v1.ResourceList{ v1.ResourceName(resourceName): *resource.NewQuantity(2, resource.DecimalSI), } return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{{ Image: busyboxImage, Name:  podName, // Runs the specified command in the test pod. Command: []string{"sh", "-c", cmd}, Resources: v1.ResourceRequirements{ Limits:   rl, Requests: rl, }, }}, }, } } // BeReady verifies that a node is ready and devices have registered. func BeReady() types.GomegaMatcher { return gomega.And( // This additional matcher checks for the final error condition. gcustom.MakeMatcher(func(ready bool) (bool, error) { if !ready { return false, fmt.Errorf("Expected node to be ready=%t", ready) } return true, nil }), BeInReadyPhase(true), ) } // BeInReadyPhase matches if node is ready i.e. ready is true. func BeInReadyPhase(isReady bool) types.GomegaMatcher { return gcustom.MakeMatcher(func(ready bool) (bool, error) { return ready == isReady, nil }).WithTemplate("Expected Node Ready {{.To}} be in {{format .Data}}nGot instead:n{{.FormattedActual}}").WithTemplateData(isReady) } func isNodeReadyWithSampleResources(ctx context.Context, f *framework.Framework) (bool, error) { node, ready := getLocalTestNode(ctx, f) if !ready { return false, fmt.Errorf("Expected node to be ready=%t", ready) } if CountSampleDeviceCapacity(node) <= 0 { return false, fmt.Errorf("Expected devices to be advertised") } return true, nil } // HaveAllocatableDevices verifies that a node has allocatable devices. func HaveAllocatableDevices() types.GomegaMatcher { return gomega.And( // This additional matcher checks for the final error condition. gcustom.MakeMatcher(func(hasAllocatable bool) (bool, error) { if !hasAllocatable { return false, fmt.Errorf("Expected node to be have allocatable devices=%t", hasAllocatable) } return true, nil }), hasAllocatable(true), ) } // hasAllocatable matches if node is ready i.e. ready is true. func hasAllocatable(hasAllocatable bool) types.GomegaMatcher { return gcustom.MakeMatcher(func(hasAllocatableDevices bool) (bool, error) { return hasAllocatableDevices == hasAllocatable, nil }).WithTemplate("Expected Node with allocatable {{.To}} be in {{format .Data}}nGot instead:n{{.FormattedActual}}").WithTemplateData(hasAllocatable) } func isNodeReadyWithAllocatableSampleResources(ctx context.Context, f *framework.Framework, devCount int64) (bool, error) { node, ready := getLocalTestNode(ctx, f) if !ready { return false, fmt.Errorf("Expected node to be ready=%t", ready) } if CountSampleDeviceCapacity(node) != devCount { return false, fmt.Errorf("Expected devices capacity to be: %d", devCount) } if CountSampleDeviceAllocatable(node) != devCount { return false, fmt.Errorf("Expected devices allocatable to be: %d", devCount) } return true, nil } func isNodeReadyWithoutSampleResources(ctx context.Context, f *framework.Framework) (bool, error) { node, ready := getLocalTestNode(ctx, f) if !ready { return false, fmt.Errorf("Expected node to be ready=%t", ready) } if CountSampleDeviceCapacity(node) > 0 { return false, fmt.Errorf("Expected devices to be not present") } return true, nil } // HaveFailedWithAdmissionError verifies that a pod fails at admission. func HaveFailedWithAdmissionError() types.GomegaMatcher { return gomega.And( gcustom.MakeMatcher(func(hasFailed bool) (bool, error) { if !hasFailed { return false, fmt.Errorf("Expected pod to have failed=%t", hasFailed) } return true, nil }), hasFailed(true), ) } // hasFailed matches if pod has failed. func hasFailed(hasFailed bool) types.GomegaMatcher { return gcustom.MakeMatcher(func(hasPodFailed bool) (bool, error) { return hasPodFailed == hasFailed, nil }).WithTemplate("Expected Pod failed {{.To}} be in {{format .Data}}nGot instead:n{{.FormattedActual}}").WithTemplateData(hasFailed) } func getPod(ctx context.Context, f *framework.Framework, podName string) (bool, error) { pod, err := e2epod.NewPodClient(f).Get(ctx, podName, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("Expected node to get pod=%q got err=%q", pod.Name, err) } expectedStatusReason := "UnexpectedAdmissionError" expectedStatusMessage := "Allocate failed due to can't allocate unhealthy devices" // This additional matcher checks for the final error condition. if pod.Status.Phase != v1.PodFailed { return false, fmt.Errorf("Expected pod to reach phase %q, got final phase %q instead.", v1.PodFailed, pod.Status.Phase) } if pod.Status.Reason != expectedStatusReason { return false, fmt.Errorf("Expected pod status reason to be %q, got %q instead.", expectedStatusReason, pod.Status.Reason) } if !strings.Contains(pod.Status.Message, expectedStatusMessage) { return false, fmt.Errorf("Expected pod status reason to contain %q, got %q instead.", expectedStatusMessage, pod.Status.Message) } return true, nil } </ins>

} else { e2epod.ImagePrePullList.Insert(gpuDevicePluginImage) } <del> if samplePluginImage, err := getSampleDevicePluginImage(); err != nil { </del> <ins> if samplePluginImage, err := getContainerImageFromE2ETestDaemonset(SampleDevicePluginDSYAML); err != nil { </ins> klog.Errorln(err) } else { e2epod.ImagePrePullList.Insert(samplePluginImage) } <ins> if samplePluginImageCtrlReg, err := getContainerImageFromE2ETestDaemonset(SampleDevicePluginControlRegistrationDSYAML); err != nil { klog.Errorln(err) } else { e2epod.ImagePrePullList.Insert(samplePluginImageCtrlReg) } </ins> } func getNodeProblemDetectorImage() string {

return ds.Spec.Template.Spec.Containers[0].Image, nil } <del> func getSampleDevicePluginImage() (string, error) { data, err := e2etestfiles.Read(SampleDevicePluginDSYAML) </del> <ins> func getContainerImageFromE2ETestDaemonset(dsYamlPath string) (string, error) { data, err := e2etestfiles.Read(dsYamlPath) </ins> if err != nil { <del> return "", fmt.Errorf("failed to read the sample plugin yaml: %w", err) </del> <ins> return "", fmt.Errorf("failed to read the daemonset yaml: %w", err) </ins> } ds, err := e2emanifest.DaemonSetFromData(data) if err != nil { <del> return "", fmt.Errorf("failed to parse daemon set for sample plugin: %w", err) </del> <ins> return "", fmt.Errorf("failed to parse daemonset yaml: %w", err) </ins> } if len(ds.Spec.Template.Spec.Containers) < 1 { <del> return "", fmt.Errorf("failed to parse the sample plugin image: cannot extract the container from YAML") </del> <ins> return "", fmt.Errorf("failed to parse the container image: cannot extract the container from YAML") </ins> } return ds.Spec.Template.Spec.Containers[0].Image, nil }