Change:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-e2e-gce Which test(s) are failing: Since when has it been failing: unknown Testgrid link: Reason for failure: unknown Anything else we need to know: PR that introduces the new test: /assign /sig storage /kind flake /priority critical-urgent the job is PR blocking and is flaking mutliple times in a row.
/remove-kind bug
from SIG Testing mentioned that probably the whole job is timing out, so this might not be a SIG Storage problem per-se. we need the owners to triage the problem.
Here's one example run:
i've sent an email to k-dev to see who reacts. increasing the timeout is the preferred option for now.
To clarify I think this is a real failure that really happened at least twice, but it is likely not the primary cause of pull-kubernetes-e2e-gce currently being broken, and almost all failed pull-kubernetes-e2e-gce runs that I see had this test pass.
Let's use this bug to triage this specific test case
The first provision succeeded: 0404 18:32:48. 1 ] provision "multivolume-4799/pvc-mvjvq" class "multivolume-4799-csi-hostpath-multivolume-4799-scjmgx4": started The second one failed because it couldn't find the storage class: I0404 18:32:50. 1 ] provision "multivolume-4799/pvc-985xh" class "multivolume-4799-csi-hostpath-multivolume-4799-scm9tnq": started E0404 18:32:50. 1 ] provision "multivolume-4799/pvc-985xh" class "multivolume-4799-csi-hostpath-multivolume-4799-scm9tnq": error getting claim's StorageClass's fields: storageClass "multivolume-4799-csi-hostpath-multivolume-4799-scm9tnq" not found
The provisioner doesn't retry/requeue if the storageclass is not found. I guess it is flaky because the pvc is created immediately after the sc and even though the provisioner has watches on both, it's possible it sees pvc before sc? It's a bug in the external library, not an issue with in-tree because it retries everything every 15s. So I'll need to update the library, update hostpath plugin, then update our manifests here. btw, how did you find that log
There is probably something else wrong too* because every other dynamic provisioning test creates pvc immediately after the sc. Why does only this one flake?
We store all the CSI driver logs for each test case under artifacts
Should we just add wait before and ? Or are there any ways to ensure that storageClass is ready? (like checking that storageClass can be got after creation.)
I don't think wait is the solution. Even if the test does a Get after Create, there is no guarantee that the provisioner pod's informer has seen the StorageClass too. I've opened for now I am still unsure why only this test is flaking frequently, if the issue is race between controller seeing PVC first or SC first, one would expect a lot of other external provisioner tests to fail too...
I guess given the frequency of failures, waiting for SC to exist after the Create call could be a good stopgap solution. Fixing the lib-hostpath plugin-test will take a lot more time.
Can we wait until 's actual fix for this issue or do we need workaround to reduce this flake?
Let's go with 's fix. I don't think it should take too long

Discussion:
metadata: name: external-provisioner-runner rules: <del> - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list"] </del> <ins> # The following rule should be uncommented for plugins that require secrets # for provisioning. # - apiGroups: [""] #   resources: ["secrets"] #   verbs: ["get", "list"] </ins> - apiGroups: [""] resources: ["persistentvolumes"] <del> verbs: ["get", "list", "watch", "create", "delete"] </del> <ins> verbs: ["get", "list", "watch", "update", "create", "delete"] </ins> - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"]

namespace: default name: external-provisioner-cfg rules: <ins> # Only one of the following rules for endpoints or leases is required based on # what is set for `--leader-election-type`. Endpoints are deprecated in favor of Leases. </ins> - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "watch", "list", "delete", "update", "create"] <ins> - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "watch", "list", "delete", "update", "create"] </ins> --- kind: RoleBinding

serviceAccountName: csi-provisioner containers: - name: csi-provisioner <del> image: quay.io/k8scsi/csi-provisioner:v1.0.1 </del> <ins> image: quay.io/k8scsi/csi-provisioner:v1.1.0-rc2 </ins> args: - "--provisioner=csi-hostpath" - "--csi-address=$(ADDRESS)"