Change:
By("Restart the nfs server") restartNfsServer(nfsServerPod) <del> By("Verify host running the deleted pod is now cleaned up") </del> <ins> By("Verify that the deleted client pod is now cleaned up") </ins> checkPodCleanup(c, pod, true) }) }

Discussion:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: See One /sig storage
/assign
[Slow] test suite was run. Glusterfs io stream test failed due to ns not being deleted, which was due to 1 pod remaining. The glusterfs-server pod was deleted. It appears that the glusterfs-client pod was not deleted even though returned no error. In cases where this test succeeds (which is most of the time) returns which is seen in the log as INFO: Ignore "not found" error above. Pod "gluster-io-client" successfully deleted. Sometimes when the plugin io streaming tests pass you'll see logged for the client pod. However, sometimes a client pod is deleted but the above log entry is not triggered. Eg in you'll see the nfs-client pod is deleted (the ns was able to be deleted) yet the above log msg stating to ignore the "not found" error was not generated. (scroll down to the line "I0724 20:27:20.364] Jul 24 20:27:00.559: INFO: Waiting for pod nfs-io-client in namespace 'e2e-tests-volume-io-pmdks' status to be 'terminated due to deadline exceeded'(found phase: "Running", readiness: true) (29.883170178s elapsed) I0724 20:27:20.365] [AfterEach] [sig-storage] NFS"). My guess is that in this case the plugin's client pod did not terminate in 30s -- WHY NOT THE FULL 5m? -- but did terminate in time for the ns to be deleted. And sometimes the client pod cannot be deleted in 30s (again WHY NOT THE FULL 5m?) and is still present when the ns is deleted. Note: if we cannot detect that the client pod was not actually deleted and then we delete the server pod, there is no chance that the ns can be deleted. Once the server pod is gone the client pod will not delete.
It looks like the client pod failed to terminate:
30s may not always be enough time for the gluster client pod to terminate?

also, why is it so common to see ~29 seconds for the client pods (nfs, gluster) to supposedly terminate. The wait should be for up to 5m yet I never see = 30s wait times... yet, we know that sometimes the gluster client pod is not terminated. Seems like maybe a bug in this area?
can you take a look? My findings so far are this test fails about 20% of the time. The gluster client pod's Status is set to Failed and therefore the pod won't be deleted. There is e2e code that waits for the pod to be deleted but escapes if the pod has "Failed" and if the passed-in reason matches the pod's Status.Reason. In the gluster pod test failures the pod's Status.Reason == "". This causes the test to incorrectly assume that the pod was deleted when it wasn't. This problem manifests itself when the namespace is deleted and that step fails. It seems like the Failed pod's Status.Reason should not be blank. cc
It looks like gluster unmount failed:
But in the log I looked at today (https://k8s-) I see the pod's Status=Running. Then the pod is deleted and the status remains Running for ~30s then changes to Failed. Even though the pod is Failed its Reason remains "". So 1) I wonder why the pod is marked as Running if its mount failed, and 2) once the pod is Failed why is Reason still blank?
It was the unmount that failed, not the mount.
ah, ok thanks, I missed that. Still need to know why Pod.Status.Reason is blank...
Below are some records from a related to the glusterfs-client pod, gluster-io-cliente2e-tests-volume-io-6h140(-778e-11e7-afad-)_ and the "e2e-tests-volume-io-6h140" ns:
I think the issue is that the test is deleting the gluster endpoints before deleting the pod.
Nm, the test does seem to be cleaning up in the correct order: client pod, then server pod and endpoints. So the issue seems to be that during Pod termination, it goes to Failed state.

Before, I thought it was because the unmount failed, but based on the timestamps of the logs, it seems like it fails before we start unmounting.
Just spoke to and he said that it's normal for a Pod to go to Failed state during the termination sequence. In between Failed state and Pod object deletion is when we unmount the volumes. So I think the issue here is that the WaitForPodTerminated function needs to wait only for the Pod object to be deleted, and not return early if Pod state is Failed.
do you have time to look at this? The test is still flaking
yes I can take a look. Do you know if there has been a pr started to address this issue? BTW, I will be gone most of tomorrow but will start tomorrow night.