Change:
tests are failing atm on GCI https://k8s- because VOLUMEPLUGINDIR=/home/kubernetes/flexvolume set in takes precedence over KUBELETTESTVOLUMEPLUGINDIR="/etc/srv/kubernetes/kubelet-plugins/volume/exec". As far as I can tell /home/kubernetes/flexvolume is not readable yet by containerized kubelet so the test needs to keep using "/etc/srv/kubernetes/kubelet-plugins/volume/exec" and the bash needs to be shuffled around to solve this. cc
/sig storage
This is more an acknowledgment of the issue than anything, can't write a PR until later
cc /kind e2e-test-failure
this is currently blocking alpha.3 release
we are just waiting for approval
I think they’re still failing, not sure what the release process is, but these tests started running only yesterday and so failure indicates something wrong with the test themselves, they shouldn’t block release. (They succeed on gce cluster with Debian nodes but fail on gci)
well, let's see if we can get the kinks worked out before the beta release on 9/27 or we will need to disable it from release-blocking
/assign /reopen
I0824 17:49:28. 8562 ] operationExecutor.VerifyControllerAttachedVolume started for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-e2e-tests-flexvolume-wf48v/-88f4-11e7-a3f9--flex-volume-0") pod "flex-client" (UID: "-88f4-11e7-a3f9-") edit: VerifyControllerAttachedVolume must succeed since the plugin is non-attachalbe and no error is reported. But the voluem is absent from node.volumesInUse I0824 17:44:06. 4 ] AttachVolume.Attach succeeded for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") from node "bootstrap-e2e-minion-group-9913" W0824 17:44:06. 4 plugin-] flexVolume driver k8s/dummy-attachable-e2e-tests-flexvolume-j8962: using default GetVolumeName for volume 0x1c5df60 I0824 17:43:59. 3728 ] operationExecutor.VerifyControllerAttachedVolume started for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") pod "flex-client" (UID: "-88f3-11e7-a3f9-") E0824 17:43:59. 3728 ] Operation for ""flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0"" failed. No retries permitted until 2017-08-24 17:44:00. +0000 UTC (durationBeforeRetry 500ms). Error: Volume has not been to the list of VolumesInUse in the node's volume status for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") pod "flex-client" (UID: "-88f3-11e7-a3f9-") Seems there is something wrong with VerifyControllerAttachedVolume, still at a loss as to why it's showing up in GCI but not Debian edit: get node -o yaml shows: yet it errors because the volume hasn't been to volumesInUse??? Aug 24 20:31:42 mawong-e2e-minion-group-npjh kubelet[9431]: E0824 20:31:42. 9431 ] Operation for ""flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-z8xc4/flex-volume-0"" failed. No retries permitted until 2017-08-24 20:31:44. +0000 UTC (durationBeforeRetry 2s). Error: Volume has not been to the list of VolumesInUse in the node's volume status for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-z8xc4/flex-volume-0") pod "flex-client" (UID: "-890b-11e7-9392-")
Never mind the GCI vs Debian issue, our test fix would have worked but in between and , something else changed and is causing both GCI and Debian tests to fail.
might be the cause, so we might be looking at legitimate failures caught by the test at this point (yay?) and not faulty tests. investigating...

Discussion:
if [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "gci" ]] || [[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then cat >>$file <<EOF <del> VOLUME_PLUGIN_DIR: $(yaml-quote ${VOLUME_PLUGIN_DIR:-/home/kubernetes/flexvolume}) </del> <ins> VOLUME_PLUGIN_DIR: $(yaml-quote ${VOLUME_PLUGIN_DIR:-/etc/srv/kubernetes/kubelet-plugins/volume/exec}) </ins> EOF fi