Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-serial Which test(s) are failing: [sig-storage] PersistentVolumes-local Stress with local volumes [Serial] should be able to process many pods and reuse local volumes Since when has it been failing: since introduction Testgrid link: https://k8s- Reason for failure: Anything else we need to know:
/assign
/reopen 1 failure out of recent 4 builds since is merged, will make it stable
Reopened this issue. <detailsIn response to : /reopen 1 failure out of 4 builds since is merged, will make it stable Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This failure is because kubelet is restarted serval times during testing. fail time of test: Note that after , the rest 3 pods cannot finish. This is because kubelet is restarted serval times in this period of time. Output of : The rest 3 jobs was assigned to this node, it cannot finished because kubelet is restarted repeatedly. Because all tests in this job is run serially. I'm not sure what caused kubelet to restart during this test is running. kubelet.service is configured with following configurations: With , kubelet may be restarted by clean/unclean signal, exit code or watchdog, timeout. And may kill it when it failed to , here is kubelet-
I guess these kubelet restarts are not expected, but fixing it is beyond my ability. One workaround is to increase timeout to tolerate them, what do you think? Because we check pod conditions periodically, it will not slow down test.
Maybe we need to get some help from node team to figure out why kubelet is restarting. cc
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
This test has not flaked in almost 2 weeks. /close
Closing this issue. <detailsIn response to : This test has not flaked in almost 2 weeks. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
BeforeEach(func() { setupStorageClass(config, &waitMode) <del> for _, node := range config.nodes { </del> <ins> for i, node := range config.nodes { </ins> By(fmt.Sprintf("Setting up %d local volumes on node %q", volsPerNode, node.Name)) <del> allLocalVolumes[node.Name] = setupLocalVolumes(config, volType, &node, volsPerNode) </del> <ins> allLocalVolumes[node.Name] = setupLocalVolumes(config, volType, &config.nodes[i], volsPerNode) </ins> } By(fmt.Sprintf("Create %d PVs", volsPerNode*len(config.nodes))) var err error

}) AfterEach(func() { <ins> By("Stop and wait for recycle goroutine to finish") close(stopCh) wg.Wait() By("Clean all PVs") </ins> for nodeName, localVolumes := range allLocalVolumes { By(fmt.Sprintf("Cleaning up %d local volumes on node %q", len(localVolumes), nodeName)) cleanupLocalVolumes(config, localVolumes) } cleanupStorageClass(config) <del> By("Wait for recycle goroutine to finish") close(stopCh) wg.Wait() </del> }) It("should be able to process many pods and reuse local volumes", func() {