Change:
} if !nodeExist { <del> glog.Warningf( "Node %q does not exist. DiskIsAttached will assume vmdk %q is not attached to it.", vSphereInstance, volPath) return false, nil </del> <ins> glog.Errorf("DiskIsAttached failed to determine whether disk %q is still attached: node %q does not exist", volPath, vSphereInstance) return false, fmt.Errorf("DiskIsAttached failed to determine whether disk %q is still attached: node %q does not exist", volPath, vSphereInstance) </ins> } // Get VM device list

Discussion:
When a vm is reported as no longer present in cloud provider and is deleted by node controller, there are no attempts to detach respective volumes. For example, if a VM is powered off or paused, and pods are migrated to other nodes. In the case of vSphere, the VM cannot be started again because the VM still holds mount points to volumes that are now mounted to other VMs. In order to re-join this node again, you will have to manually detach these volumes from the powered off vm before starting it.
Once the node is powered off, the controller-manager doesn't send a detachVolume request to vSphere cloud provider. The controller-manager couldn't send the detachVolume request because it checks until the volume is unmounted on the node for a particular timeout. You can see the code over here - Since the volume is never unmounted, the controller manager skips the detachVolume request and issues the attachVolume() on another successfully. I think the controller-manager needs send a detachVolume() request even if the volume is mounted when the node is powered OFF. This is serious problem in our case (vSphere) as volume is now attached on another node and the node which is powered OFF cannot be powered ON again, as the node tries to acquire a lock on the volume to mount it on this node during the power ON. The acquire lock fails as it is already attached/mounted on other node. Also there are a few workarounds for this. When the Node is powered OFF, the cloud provider can detach the volume in NodeExists() - This method is called in DisksAreAttached() method. However i don't think this is a viable solution. : Can you provide your inputs on this. CCing
CC If a node goes offline (for any reason including being powered off) the attach/detach controller will eventually detach the volume even if the node never reports it as safely unmounted. Once a node becomes inaccessible, the node controller waits 5 min 40 seconds, before deleting all the pods that were scheduled to that node. When a pod referencing an attachable volume on that node is deleted, the attach/detach controller tries to detach the volume, but if the node has not reported the volume as safely unmounted (which would be the case if the node was powered off), the attach/detach controller waits 6 minutes before triggering detach anyway.

Once the volume is detached, any attempts to attach it to some other node will finally succeed. This means that it can take up to 15 minutes for this situation to rectify itself. Can you verify what amount of time you waited?
So do you mean that the volume can be attached to another node only after the volume is detached on the node which is powered OFF. In my case, i have waited for significant amount of time. I am sure it is more than 15 mins. And I see the volume succesfully attached to the new node but still not detached from the powered OFF node. In this case I cannot restart the node as explained above. Only if I delete the attached nodes manually I can power ON the node again.
If the storage system supports multi node attach for the access mode, then the attach to the new node should succeed immediately. If the storage systems supports only single node attach for the access mode then the attach calls to the new node fail fail until it is detached from the first node (this is the case with GCE PDs). If you are never seeing the volume detach from the original (powered off) node (even after 15 minutes), that sounds like a bug. AFAIK, this case is working fine in the GCE PD case ( tested this recently). If you have a consistent repro, to help debug, could you grab and share your controller-manager logs showing the original attach, subsequent attach, and lack of detach from the first node.
Sure. I will attach the logs. Between in our case we only support single node attach for a volume.
Between I am testing it on v1.4.7. Is this a change which is recently made to kubernetes code base to address this issue and not present in v1.4.7?
Please verify the logs from controller-manager. I see there is detach volume request being issued. I almost waited for 45 mins, but I didn't send any request attempting to detach volume as in - As in , since it failed to update node status, it couldn't make any further request for detachVolume(). Also, from the logs I see it is the kubernetes control plane which decides whether the storage plane is multi node attach or single node attach by attempting to send an attachVolume request on the new node.

And its not the cloud provider which says it to kubernetes control plane what type of storage plane it is. Correct me if I am wrong on this. : Can you please verify this.
That log is level 5. If your cluster is set up with log level 4, you will not see that log. When a node is down, the following events will happen with code release 4 or before. Once a node becomes inaccessible, the node controller waits for about 5 mins, and then deletes all the pods that were scheduled to that node. Once pods are deleted, Volume manager's reconciler will try to detach the volumes that belong to the pods. However, it will fail to issue to request first because it thinks the volumes are still mounted to the nodes according to the node status. (node status can be updated by the kubelet but since the node is down, the status is not updated). the reconciler fails to issue the detach request for a periods time, it times out (currently it is set to 6 minutes) and will issue detach request anyway and it should find that the volume is already detached since node is already terminated. So it will probably take more than 10 mins for reconciler marks the volume is detached to the node. We recently have a PR that change the above behavior during node termination because this PR adds a periodic check that whether the volumes are still attached. When node is terminated, the reconciler will very quickly (in a few seconds) find out the node is terminated and volume is detached and then mark this change in the actual state. {"log":"W1223 00:27:26. 1 ] Node "node2" does not exist. DisksAreAttached will assume vmdk [[vsanDatastore1] kubevols/redis-master] are not attached to it.n","stream":"stderr","time":"2016-12-23T00:27:26.429562004Z"} {"log":"I1223 00:27:26.

1 ] VolumesAreAttached: check volume "[vsanDatastore1] kubevols/redis-master" (specName: "redis-master-data") is no longer attachedn","stream":"stderr","time":"2016-12-23T00:27:26.44262004Z"} Please let me know if you have any questions about this part. Can you please verify this.
: I am using v1.4.7. Can you let me know if this holds true for v1.4.7? I don't see any behavior as mentioned by in the logs. Also you can see there is no detach call made on the node. As per this - if fails to update the node status after various attempts and hence detach call doesn't pass through. Also, from the logs I see it is the kubernetes control plane which decides whether the storage plane is multi node attach or single node attach by attempting to send an attachVolume request on the new node. And its not the cloud provider which says it to kubernetes control plane what type of storage plane it is. Also can you comment on the above question?
+1 on this. Same goes for k8s v1.5.2. Detach never happens when powering off a node with PVs attached. Disks has to be manually detached before being able to turn on the node again.
The reason detach never happens is as follows, is terminated controller periodically check whether volumes are still attached by sending requests to cloud provider. The function DisksAreAttached will check whether the node still exist. If not, it will assume the volume is no longer attached. If node does exist, check the information returned by the cloud provider to see whether volume is still attached. In this step, I believe it returns as volume is detached volume is already detached. So explicit detach will not be issued since step 2 and 3 already marks it as detached. I will check the function to see if there is potential problem in checking volume detached or not.
Cheers then I understand. I will also have a look.
is a workaround but ideally k8s should call detach for every attach that was issued, if the disk is already detached it should be a no-op. The ideal behavior should be that for each attach there will be a detach (which could be redundant).

The assumption in the current model seems to be that as long as the volume provider does not consider the disk attached the detach does not need to be issued which ignores the configuration for the VM. If the VM still considers the disk attached it should be reconfigured as well. The pod in this case is fine after being rescheduled to a new node but the VM is left with an invalid configuration.
Ok! The workaround seems legit for the time being. Thanks!