Issue 1:
When a node is both master and minion --- that is, the provision-masterandnode function in util.sh is invoked, the flannel files for both master and minion are copied to the same node. That looks bad, the file names are the same but the contents are different.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): unable to unmount volume properly

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: vSphere

OS (e.g. from /etc/os-release):

RETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="https://bugs.debian.org/"

Kernel (e.g. uname -a):

Linux kubernetes-master 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux

Install tools: None

Others: None

What happened:

A deployment was created with the volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" mounted on the pod. Existing deployment was deleted and recreated. When I checked the logs during the delete of the deployment, I unmounts the volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" successfully. But, I also tries to unmount a volume "kubernetes.io/vsphere-volume/hello-master" which doesn't exist at all. This seems to happen only with kubernetes v1.4.5 but v1.4.4 works perfectly fine.

You can check the logs below.

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537562 23426 reconciler.go:166] Attempting to start UnmountVolume for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537760 23426 reconciler.go:188] UnmountVolume operation started for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.537944 23426 reconciler.go:166] Attempting to start UnmountVolume for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.538142 23426 reconciler.go:188] UnmountVolume operation started for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

It loops on forever trying to unmount to the volume "kubernetes.io/vsphere-volume/hello-master" which doesn't exist at all.

What you expected to happen:

All the pods in the recreated deployment comes up.

How to reproduce it (as minimally and precisely as possible):

See above.

Anything else do we need to know:

Here are some logs. I also see multiple calls to TearDown().

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: E1114 22:00:47.552683 23426 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master\" (\"d470b123-aadd-11e6-bd75-005056b52a82\")" failed. No retries permitted until 2016-11-14 22:00:48.052644231 -0500 EST (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (volume.spec.Name: "hello-master") pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82") with: Unmount failed: exit status 32

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: Unmounting arguments: /var/lib/kubelet/pods/d470b123-aadd-11e6-bd75-005056b52a82/volumes/kubernetes.io~vsphere-volume/hello-master

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: Output: umount: /var/lib/kubelet/pods/d470b123-aadd-11e6-bd75-005056b52a82/volumes/kubernetes.io~vsphere-volume/hello-master not mounted

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.639638 23426 reconciler.go:166] Attempting to start UnmountVolume for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.640356 23426 reconciler.go:320] Attempting to start UnmountDevice for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master")

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.640558 23426 reconciler.go:338] UnmountDevice operation started for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master")

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: W1114 22:00:47.642090 23426 mount.go:121] could not determine device for path: "/var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/hello-master"

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: W1114 22:00:47.642277 23426 util.go:72] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/hello-master

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: E1114 22:00:47.642490 23426 nestedpendingoperations.go:253] Operation for "\"kubernetes.io/vsphere-volume/hello-master\"" failed. No retries permitted until 2016-11-14 22:00:48.142466159 -0500 EST (durationBeforeRetry 500ms). Error: UnmountDevice.DeviceOpened failed for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master") with: PathIsDevice failed for path "": stat : no such file or directory

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.741368 23426 reconciler.go:166] Attempting to start UnmountVolume for volume "kubernetes.io/vsphere-volume/[datastore2] kubevols/sample-master" (spec.Name: "hello-master") from pod "d470b123-aadd-11e6-bd75-005056b52a82" (UID: "d470b123-aadd-11e6-bd75-005056b52a82").

> Nov 14 22:00:47 kubernetes-minion-2 kubelet[23426]: I1114 22:00:47.741641 23426 reconciler.go:320] Attempting to start UnmountDevice for volume "kubernetes.io/vsphere-volume/hello-master" (spec.Name: "hello-master")
>

Also, I see an orphan mount point on the node. Even after deleting this mount point on the node, I still see issues with the new deployment.

Also we see a similar issue #36269, however cherry picking the fix from jingxu97:syncAttach-10-15 for this bug did not solve the problem for us.