Change:
Which jobs are flaking: gce-cos-master-alpha-features Which test(s) are flaking: [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] Testgrid link: Reason for failure: It takes 6 minutes to start a pod with PVC that has been restored from a snapshot, while the test waits only 5 minutes: csi-attacher sidecar says: Perhaps the provisioner should wait until the volume is ready to be used? That may not help with this test timeout, as PVC is provisioned after the Pod is scheduled and the total time could be the same. All this is caused by my PR to test block snapshots, It reworked the tests from using . If snapshots are slow, should all persistent storage tests use / ?
/sig storage
Generally, cutting a snapshot is quick, but uploading can take longer. Does provisioner wait until the snapshot is "ready" before returning success?
cc
(But yes, we should increase the timeout because overall time will still be longer no matter which order we take)
The provisioner does not wait. If snapshot is not readyToUse, creating a volume from it will fail. I'll have to check to see if there's logic to wait in the e2e test. Regarding GCE PD, at restore time, do we need to download the snapshot from object store? If that's the case, it could take a while to download as well.
volume controller should keep trying until snapshot is ready. Looks like attacher tried to attach while disk is not ready. It is strange because I think attach only happen after volume is successfully provisioned.
AFAIK, most cloud providers' snapshot ATM will be uploaded to some remote when taking, and will be downloaded when restoring. GCE PD is one of them.
It's not taking snapshots what slows the test, it's restoring it - returns quickly, but the returned volume is not really usable and attach fails for ~4 minutes until the snapshot is fully restored somewhere in GCE. Now that I am thinking about it, maybe it's better if the driver returns the volume early, so it does not end up orphaned when user deletes corresponding PVC and the provisioner is restarted at the same time. Anyway, I'll extend the timeout in volume checks. /assign
Which brings me to idea: what about other storage backends? Some of them may take even longer that 10 minutes to restore a snapshot. Or more than 5 minutes to provision an empty volume and run a pod. There should be some configuration of timeouts (timeout multiplier?) for
I opened to investigate from GCE side if this behavior is expected.

Discussion:
gomega.Expect(err).To(gomega.BeNil(), "Failed to delete pod %v in namespace %v", config.Prefix+"-server", config.Namespace) } <del> func runVolumeTesterPod(client clientset.Interface, config TestConfig, podSuffix string, privileged bool, fsGroup *int64, tests []Test) (*v1.Pod, error) { </del> <ins> func runVolumeTesterPod(client clientset.Interface, config TestConfig, podSuffix string, privileged bool, fsGroup *int64, tests []Test, slow bool) (*v1.Pod, error) { </ins> ginkgo.By(fmt.Sprint("starting ", config.Prefix, "-", podSuffix)) var gracePeriod int64 = 1 var command string

if err != nil { return nil, err } <del> err = e2epod.WaitForPodRunningInNamespace(client, clientPod) </del> <ins> if slow { err = e2epod.WaitForPodRunningInNamespaceSlow(client, clientPod.Name, clientPod.Namespace) } else { err = e2epod.WaitForPodRunningInNamespace(client, clientPod) } </ins> if err != nil { <ins> e2epod.DeletePodOrFail(client, clientPod.Namespace, clientPod.Name) </ins> e2epod.WaitForPodToDisappear(client, clientPod.Namespace, clientPod.Name, labels.Everything(), framework.Poll, framework.PodDeleteTimeout) return nil, err }

// and check that the pod sees expected data, e.g. from the server pod. // Multiple Tests can be specified to mount multiple volumes to a single // pod. <ins> // Timeout for dynamic provisioning (if "WaitForFirstConsumer" is set && provided PVC is not bound yet), // pod creation, scheduling and complete pod startup (incl. volume attach & mount) is pod.podStartTimeout. // It should be used for cases where "regular" dynamic provisioning of an empty volume is requested. </ins> func TestVolumeClient(f *framework.Framework, config TestConfig, fsGroup *int64, fsType string, tests []Test) { <del> clientPod, err := runVolumeTesterPod(f.ClientSet, config, "client", false, fsGroup, tests) </del> <ins> testVolumeClient(f, config, fsGroup, fsType, tests, false) } // TestVolumeClientSlow is the same as TestVolumeClient except for its timeout. // Timeout for dynamic provisioning (if "WaitForFirstConsumer" is set && provided PVC is not bound yet), // pod creation, scheduling and complete pod startup (incl. volume attach & mount) is pod.slowPodStartTimeout. // It should be used for cases where "special" dynamic provisioning is requested, such as volume cloning // or snapshot restore. func TestVolumeClientSlow(f *framework.Framework, config TestConfig, fsGroup *int64, fsType string, tests []Test) { testVolumeClient(f, config, fsGroup, fsType, tests, true) } func testVolumeClient(f *framework.Framework, config TestConfig, fsGroup *int64, fsType string, tests []Test, slow bool) { clientPod, err := runVolumeTesterPod(f.ClientSet, config, "client", false, fsGroup, tests, slow) </ins> if err != nil { framework.Failf("Failed to create client pod: %v", err) }

e2epod.WaitForPodToDisappear(f.ClientSet, clientPod.Namespace, clientPod.Name, labels.Everything(), framework.Poll, framework.PodDeleteTimeout) }() <del> framework.ExpectNoError(e2epod.WaitForPodRunningInNamespace(f.ClientSet, clientPod)) </del> testVolumeContent(f, clientPod, fsGroup, fsType, tests) }

if framework.NodeOSDistroIs("windows") { privileged = false } <del> injectorPod, err := runVolumeTesterPod(f.ClientSet, config, "injector", privileged, fsGroup, tests) </del> <ins> injectorPod, err := runVolumeTesterPod(f.ClientSet, config, "injector", privileged, fsGroup, tests, false /*slow*/) </ins> if err != nil { framework.Failf("Failed to create injector pod: %v", err) return

ExpectedContent: expectedContent, }, } <del> volume.TestVolumeClient(f, testConfig, nil, "", tests) </del> <ins> volume.TestVolumeClientSlow(f, testConfig, nil, "", tests) </ins> } l.testCase.TestDynamicProvisioning() })

ExpectedContent: expectedContent, }, } <del> volume.TestVolumeClient(f, myTestConfig, nil, "", tests) </del> <ins> volume.TestVolumeClientSlow(f, myTestConfig, nil, "", tests) </ins> } myTestCase.TestDynamicProvisioning() }(i)