Change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /cc /priority critical-urgent /milestone v1.19
Appears to have started failing after , which seems unrelated. /cc
according to the build log , the test timed out because it could not find gpu on any nodes. Is it probably a test-infra issue?
I think you are right, I don't see anything changing in the config so must be underlying infra issue /cc
yes, the issue is that the nodes do not have the GPU required to run the test The same infra commit () was used for the last passing job () and the first failing job ()
not sure who owns the node setup for the gpu tests, but I would guess
test-infra-maintainers owns e.g. prow, I don't think anything there relates to the GPUs the infra commit (test-infra commit) not changing suggests perhaps a change to the scripts / test harness in this repo?
the delta between the passing and failing runs only changed an ingress e2e test ()
The image is used in when installing the Nvidia driver. Given that ([1]) is lastly updated on July 17, 2020 at 12:22:11 AM UTC-7 (latest tag points to v2.0.0), does it seem this version is incompatible? BTW: the latest successful run pulled the image with SHA: 8d86a652759f80595cafed7d3dcde3dc53f57f9bc1e33b27bc3cfa7afea8d483 ([2]), maybe we can try that SHA specifically? [1]: [2]:
Forgot to mention, the shows that the deamonset was pending:
yes, and the log also shows that the pod was pending because the init container failed to complete could we change the image to  whose digest is exactly .

Discussion:
"//test/e2e/framework/resource:go_default_library", "//test/e2e/framework/service:go_default_library", "//test/e2e/framework/skipper:go_default_library", <ins> "//test/e2e/framework/testfiles:go_default_library", </ins> "//test/utils:go_default_library", "//test/utils/image:go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library",

"regexp" "time" <ins> appsv1 "k8s.io/api/apps/v1" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

"k8s.io/kubernetes/test/e2e/framework/providers/gce" e2eresource "k8s.io/kubernetes/test/e2e/framework/resource" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" <ins> e2etestfiles "k8s.io/kubernetes/test/e2e/framework/testfiles" </ins> imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo"

var ( gpuResourceName v1.ResourceName <del> dsYamlURL       string </del> ) func makeCudaAdditionDevicePluginTestPod() *v1.Pod {

func SetupNVIDIAGPUNode(f *framework.Framework, setupResourceGatherer bool) *framework.ContainerResourceGatherer { logOSImages(f) <ins> var err error var ds *appsv1.DaemonSet </ins> dsYamlURLFromEnv := os.Getenv("NVIDIA_DRIVER_INSTALLER_DAEMONSET") if dsYamlURLFromEnv != "" { <del> dsYamlURL = dsYamlURLFromEnv </del> <ins> // Using DaemonSet from remote URL framework.Logf("Using remote nvidia-driver-installer daemonset manifest from %v", dsYamlURLFromEnv) ds, err = e2emanifest.DaemonSetFromURL(dsYamlURLFromEnv) framework.ExpectNoError(err, "failed get remote") </ins> } else { <del> dsYamlURL = "https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/daemonset.yaml" </del> <ins> // Using default local DaemonSet framework.Logf("Using default local nvidia-driver-installer daemonset manifest.") data, err := e2etestfiles.Read("test/e2e/testing-manifests/scheduling/nvidia-driver-installer.yaml") framework.ExpectNoError(err, "failed to read local manifest for nvidia-driver-installer daemonset") ds, err = e2emanifest.DaemonSetFromData(data) framework.ExpectNoError(err, "failed to parse local manifest for nvidia-driver-installer daemonset") </ins> } gpuResourceName = e2egpu.NVIDIAGPUResourceName <del> framework.Logf("Using %v", dsYamlURL) // Creates the DaemonSet that installs Nvidia Drivers. ds, err := e2emanifest.DaemonSetFromURL(dsYamlURL) framework.ExpectNoError(err) </del> ds.Namespace = f.Namespace.Name _, err = f.ClientSet.AppsV1().DaemonSets(f.Namespace.Name).Create(context.TODO(), ds, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create nvidia-driver-installer daemonset")

<ins> # This DaemonSet was originally referenced from # https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/master/daemonset.yaml # The Dockerfile and other source for this daemonset are in # https://github.com/GoogleCloudPlatform/cos-gpu-installer apiVersion: apps/v1 kind: DaemonSet metadata: name: nvidia-driver-installer namespace: kube-system labels: k8s-app: nvidia-driver-installer spec: selector: matchLabels: k8s-app: nvidia-driver-installer updateStrategy: type: RollingUpdate template: metadata: labels: name: nvidia-driver-installer k8s-app: nvidia-driver-installer spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator operator: Exists tolerations: - operator: "Exists" hostNetwork: true hostPID: true volumes: - name: dev hostPath: path: /dev - name: vulkan-icd-mount hostPath: path: /home/kubernetes/bin/nvidia/vulkan/icd.d - name: nvidia-install-dir-host hostPath: path: /home/kubernetes/bin/nvidia - name: root-mount hostPath: path: / initContainers: - image: gcr.io/cos-cloud/cos-gpu-installer:v20200701 name: nvidia-driver-installer resources: requests: cpu: 0.15 securityContext: privileged: true env: - name: NVIDIA_INSTALL_DIR_HOST value: /home/kubernetes/bin/nvidia - name: NVIDIA_INSTALL_DIR_CONTAINER value: /usr/local/nvidia - name: VULKAN_ICD_DIR_HOST value: /home/kubernetes/bin/nvidia/vulkan/icd.d - name: VULKAN_ICD_DIR_CONTAINER value: /etc/vulkan/icd.d - name: ROOT_MOUNT_DIR value: /root volumeMounts: - name: nvidia-install-dir-host mountPath: /usr/local/nvidia - name: vulkan-icd-mount mountPath: /etc/vulkan/icd.d - name: dev mountPath: /dev - name: root-mount mountPath: /root containers: - image: "gcr.io/google-containers/pause:3.2" name: pause </ins>  No newline at end of file