Change:
cc
There were: 2 failures as part of "gce-flaky" suite: 2170, 2171 a bunch of failures inside "gce-parallel-flaky" suite
I looked a bit into logs and in all cases it seems that we are simply not able to attach the volume. [Example logs from 2472 run in gce-parallel-flaky]: This log is repeating every ~10 seconds for 5 minutes and then the test fails.
Moving to post since AIUI this is a test stability fix.
I'm not sure - this might be either test problem or a problem with real code. I wasn't able to find the cause of it yet.
Haven't had a chance to dig in to this yet. But it may well be:
I don't think this is because all the failures I've seen here are at the level of attaching PDs, before any dettaching is done. Basically, what is happening here is: create one PD attach this PD in RW mode to pod1 on host1 attach this PD in RO mode to pod2 on host1 attach this PD in RO mode to pod3 on host2 only then do some deletions All the recent failure are before we even start deleting anything - it's just that we cannot start pod3 on host2, because it fails attaching PD to it.
I was trying to reproduce this issue locally, but unsuccessfully :( (although it is still failing on Jenkins). However, I don't think this is a test issue - I think this is a real issue. After looking through the code, I think that what is happening is: we send the request to attach a PD: it doesn't return an error, so it is "accepted" by GCE however, verifyAttached method is not able to verify it (disc is not being attached for the next 100 seconds): retrying that operation doesn't help: The fact that retrying doesn't help is slightly strange - maybe it's something wrong with the PD itself? Any other ideas?
BTW, this test is not unstable on : . Given the chasm, you're probably looking for something since the branchpoint ().
I still suspect it's . Although that is a detach issue, and the symptoms here are on attach, it could still be the same cause: i.e. the PD fails to actually detach from the first node, therefore it fails to attach to the second node. I'll tackle this while or after I'm done with .
This test is failing on virtually every run in kubernetes-e2e-gce in the last 24 hours (except the ones that were red).
- thanks for working on this - I'm assigning this to you then.
I got a repro of this while testing the fix for by running the test for a few hours. It does appear to be a slightly different issue than that, but my hunch seems to be correct that the disk is not detached from the previous node therefore it can't attach to the new node. Here is what I think is happening: When the test moves the pod with the PD from the first host to the second, the pod on the second host remains in the pending state until the PD is successfully attached to the new host. However, that won't happen until the the PD is successfully detached from the original host. Detaching from the original host can take a variable amount of time and it appears that the test sometimes timesout before this happens. If that is true, then this is a test issue. And the fix would be to increase the amount of time this test waits for the pod to move out of pending state on the second host. I'm trying to capture another repro to verify my hunch. The logs from my repro below. Kubelet logs from Node 1: PD is successfully detached from node 1 @ 05:29:25. Kubelet logs from Node 2: Node 2 tries to attach PD starting at 05:24:02. Node 2 gives up trying to attach PD starting at 05:29:24.

Discussion:
. "github.com/onsi/gomega" ) <ins> const ( gcePDDetachTimeout  = 10 * time.Minute gcePDDetachPollTime = 10 * time.Second ) </ins> var _ = Describe("Pod Disks", func() { var ( podClient client.PodInterface

expectNoError(err, "Failed to create rwPod") expectNoError(framework.WaitForPodRunning(rwPod.Name)) expectNoError(podClient.Delete(rwPod.Name, nil), "Failed to delete host0Pod") <ins> expectNoError(waitForPDDetach(diskName, host0Name)) </ins> By("submitting host0ROPod to kubernetes") _, err = podClient.Create(host0ROPod)

return pod } <ins> // Waits for specified PD to to detach from specified hostName func waitForPDDetach(diskName, hostName string) error { if testContext.Provider == "gce" || testContext.Provider == "gke" { for start := time.Now(); time.Since(start) < gcePDDetachTimeout; time.Sleep(gcePDDetachPollTime) { zone := testContext.CloudConfig.Zone cmd := exec.Command("gcloud", "compute", "--project="+testContext.CloudConfig.ProjectID, "instances", "describe", "--zone="+zone, hostName) data, err := cmd.CombinedOutput() if err != nil { Logf("Error waiting for PD %q to detach from node %q. 'gcloud compute instances describe' failed with %s (%v)", diskName, hostName, string(data), err) return err } dataStr := strings.ToLower(string(data)) diskName = strings.ToLower(diskName) if !strings.Contains(string(dataStr), diskName) { // Specified disk does not appear to be attached to specified node Logf("GCE PD %q appears to have successfully detached from %q.", diskName, hostName) return nil } Logf("Waiting for GCE PD %q to detach from %q.", diskName, hostName) } return fmt.Errorf("Gave up waiting for GCE PD %q to detach from %q after %v", diskName, hostName, gcePDDetachTimeout) } return nil } </ins>