Change:
if err != nil { return fmt.Errorf("fc: failed to unmount: %snError: %v", deviceMountPath, err) } <ins> // GetDeviceNameFromMount from above returns an empty string if deviceMountPath is not a mount point // There is no need to DetachDisk if this is the case (and DetachDisk will throw an error if we attempt) if devName == "" { return nil } </ins> unMounter := volumeSpecToUnmounter(detacher.mounter) err = detacher.manager.DetachDisk(*unMounter, devName) if err != nil {

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Certain stress tests we run involve a large number of fibre channel persistent volumes shared amount multiple Kubernetes worker nodes. During the stress tests, any given one of these volumes may be bound by a PVC and mounted to a node, released, and bound by a different PVC and mounted to a different node. Given enough cycles (and generally the number is not that large), failures start to occur. The first symptom of a problem is a discrepancy between "volumesAttached" and "volumesInUse" for a node. Eventually, "volumesInUse" grows significantly larger than "volumesAttached", indicating that the attach/detach controller is forcefully detaching fibre channel volumes from a node even though the node's kubelet still believes those volumes are mounted. kubecontroller-manager logs like the following confirm that this is happening: W0911 15:57:53. 1 ] attacherDetacher.DetachVolume started for volume "array-00-ext-016" (UniqueName: "[]") on node "node4" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching Kubelet logs shed some light on what is happening: I0911 15:55:16. ] operationExecutor.UnmountDevice started for volume "array-00-xfs-050" (UniqueName: "[]") on node "node4" W0911 15:55:16. ] Warning: Unmount skipped because path does not exist: I0911 15:55:16. ] fc: DetachDisk devicePath: , dstPath: ., devices: [.] E0911 15:55:16. ] Operation for ""[]"" failed. No retries permitted until 2020-09-11 15:55:24. +0000 UTC m=+2146. (durationBeforeRetry 8s).

Error: "UnmountDevice failed for volume "array-00-xfs-050" (UniqueName: "[]") on node "node4" : fc: failed to detach disk: nError: fc: detachFCDisk failed. device: . err: fc detach disk: invalid device name: ." The kubelet's volume manager reconcile loop is repeatedly attempting to do an UnmountDevice on the fibre channel volume in question, but something in the code is broken. DetachFCDisk never receives the name of a device to detach. It throws and error, UnmountDevice fails, and the loop continues. After six minutes, the attach/detach controller gives up waiting for the kubelet to unmount the volume and allows it to be attached to a different node. The volume continues to appear in one node's "volumesInUse" list while it also appears in another node's "volumesAttached" and "volumesInUse" list. What you expected to happen: DetachFCDisk should receive a device name if a fibre channel volumes is being unmounted and needs to be detached. Otherwise, DetachFCDisk should not be called, and thus not throw an error that confuses the kubelet volume manager's reconcile loop. How to reproduce it (as minimally and precisely as possible): This may be difficult to reproduce, as it only seems to occur in the circumstances described above (a large number of fibre channel volumes shared by many nodes undergoing high churn). However, we will propose a simple fix that we have tested in our environment. The fix appears to eliminate the issue. Environment: Kubernetes version (use ): 1.14.3 Cloud provider or hardware configuration: bare-metal Lenovo servers OS (e.g: ): CentOS Linux release 7.7.1908 Kernel (e.g. ): 3.10.0-1062.18.1.el7.x8664 Install tools: kubeadm
/sig storage
/assign ejweber