Repository: harvester/harvester

Issue #2137: [BUG] Block device volume failed to unmount when it is detached unexpectedly
Created: 2022-04-11T09:39:25Z
Describe the bug
backport issue longhorn/longhorn#3778
To Reproduce
Steps to reproduce the behavior:

longhorn/longhorn#3778 (comment)

Expected behavior

Support bundle

Environment:

Harvester ISO version: v1.0.1
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630):

Additional context
Add any other context about the problem here.

Duplicate issue #2123: [BUG] VMs stuck terminating when provisioning through rancher
Created: 2022-04-07T06:21:48Z
Describe the bug
When provisioning through rancher, some nodes did not get to the final step to register with k8s. Rancher deleted the existing node and brought up a new one, however that existing node never deleted in harvester.
To Reproduce
Steps to reproduce the behavior:

provision a single node-per-role rke1 cluster through rancher using rke1
wait for nodes to become active

Expected behavior
all nodes should come to a healthy state. any node(s) removed in the process should be properly removed from harvester.
Support bundle

supportbundle_f97ef78f-9917-4e2e-bcec-e8cf206e1090_2022-04-07T06-10-02Z.zip
Environment:

Harvester ISO version:  1.0.1-rc2
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): dell poweredge r510

this setup has rancher running on one of the harvester VMs.

Closing comment (2022-04-28T08:13:14Z): In support-bundle, we can see the robustness of LH Volume is unknown. Also, logs in LH engine manager show Setting replica tcp://xxx to ERR due to: r/w timeout first, and then many I/O error: No backend available. So I think there may be network issue between the LH engine manager and replica manager.
I try to kill the connection between the engine manager and replica manager with the following steps. VM pod will be stuck in Terminating and its containerStatuses shows exitCode: 137. Also, logs in kubelet are similar to #2137.
E0428 08:10:24.558722   15240 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/driver.longhorn.io^pvc-d042adad-8254-4eba-b4dd-b378647e276f podName:c67e7a78-d489-4cf8-acb9-e8da287db975 nodeName:}" failed. No retries permitted until 2022-04-28 08:12:26.558706566 +0000 UTC m=+14240.221959502 (durationBeforeRetry 2m2s). Error: UnmapVolume.UnmapBlockVolume failed for volume "disk-0" (UniqueName: "kubernetes.io/csi/driver.longhorn.io^pvc-d042adad-8254-4eba-b4dd-b378647e276f") pod "c67e7a78-d489-4cf8-acb9-e8da287db975" (UID: "c67e7a78-d489-4cf8-acb9-e8da287db975") : blkUtil.DetachFileDevice failed. globalUnmapPath:/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d042adad-8254-4eba-b4dd-b378647e276f/dev, podUID: c67e7a78-d489-4cf8-acb9-e8da287db975, bindMount: true: failed to unmount linkPath /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d042ada
d-8254-4eba-b4dd-b378647e276f/dev/c67e7a78-d489-4cf8-acb9-e8da287db975: unmount failed: exit status 32
Unmounting arguments: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d042adad-8254-4eba-b4dd-b37864
7e276f/dev/c67e7a78-d489-4cf8-acb9-e8da287db975
Output: umount: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d042adad-8254-4eba-b4dd-b378647e276f
/dev/c67e7a78-d489-4cf8-acb9-e8da287db975: target is busy.

Reproduce steps:

Start a 1-node cluster with harvester version v1.0.1-rc2.
Create a VM.
Get volume name of VM.
Login instance-manager-r-xxx.
Use ss to list socket connections and ss -K to kill connection.

root@instance-manager-r-37fa71ad:/# ss
Netid    State    Recv-Q    Send-Q             Local Address:Port               Peer Address:Port     Process
tcp      ESTAB    0         0            [::ffff:10.52.0.37]:10001       [::ffff:10.52.0.36]:38406
tcp      ESTAB    0         0            [::ffff:10.52.0.37]:8500        [::ffff:10.52.0.26]:41918
tcp      ESTAB    0         0            [::ffff:10.52.0.37]:10016       [::ffff:10.52.0.36]:38832
tcp      ESTAB    0         0            [::ffff:10.52.0.37]:10031       [::ffff:10.52.0.36]:57766
root@instance-manager-r-37fa71ad:/# ss -K dport = 57766
Netid    State    Recv-Q    Send-Q             Local Address:Port               Peer Address:Port     Process
tcp      ESTAB    0         0            [::ffff:10.52.0.37]:10031       [::ffff:10.52.0.36]:57766

I try the above steps on harvester master-head and I cannot reproduce anymore,  so I think the root cause is the same as #2137.