Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.8 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): COS Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node /priority important-soon
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
FYI, the started to pass now with the merged PR. Will keep watching to make sure it is not flaky.

Discussion:
import ( "os/exec" <ins> "path/filepath" </ins> "regexp" "time"

By("Waiting for GPUs to become available on the local node") Eventually(func() bool { return framework.NumberOfNVIDIAGPUs(getLocalNode(f)) > 0 <del> }, 10*time.Second, time.Second).Should(BeTrue()) </del> <ins> }, 10*time.Second, framework.Poll).Should(BeTrue()) </ins> if framework.NumberOfNVIDIAGPUs(getLocalNode(f)) < 2 { Skip("Not enough GPUs to execute this test (at least two needed)")

// TODO: Find a uniform way to deal with systemctl/initctl/service operations. #34494 func restartKubelet(f *framework.Framework) { <ins> beforeSocks, err := filepath.Glob("/var/lib/kubelet/device-plugins/nvidiaGPU*.sock") framework.ExpectNoError(err) Expect(len(beforeSocks)).NotTo(BeZero()) </ins> stdout, err := exec.Command("sudo", "systemctl", "list-units", "kubelet*", "--state=running").CombinedOutput() framework.ExpectNoError(err) regex := regexp.MustCompile("(kubelet-[0-9]+)")

kube := matches[0] framework.Logf("Get running kubelet with systemctl: %v, %v", string(stdout), kube) stdout, err = exec.Command("sudo", "systemctl", "restart", kube).CombinedOutput() <del> if err == nil { return } framework.Failf("Failed to restart kubelet with systemctl: %v, %v", err, stdout) </del> <ins> framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) Eventually(func() ([]string, error) { return filepath.Glob("/var/lib/kubelet/device-plugins/nvidiaGPU*.sock") }, 5*time.Minute, framework.Poll).ShouldNot(ConsistOf(beforeSocks)) </ins> } func getDeviceId(f *framework.Framework, podName string, contName string, restartCount int32) string { // Wait till pod has been restarted at least restartCount times. Eventually(func() bool { p, err := f.PodClient().Get(podName, metav1.GetOptions{}) <del> framework.ExpectNoError(err) </del> <ins> if err != nil || len(p.Status.ContainerStatuses) < 1 { return false } </ins> return p.Status.ContainerStatuses[0].RestartCount >= restartCount <del> }, time.Minute, time.Second).Should(BeTrue()) </del> <ins> }, 5*time.Minute, framework.Poll).Should(BeTrue()) </ins> logs, err := framework.GetPodLogs(f.ClientSet, f.Namespace.Name, podName, contName) if err != nil { framework.Failf("GetPodLogs for pod %q failed: %v", podName, err)