Project: Longhorn (Runtime / Cloud Native Storage)
File: 20201106-disk-reconnection.md

When disks are reconnected/migrated to other Longhorn nodes, Longhorn should be able to figure out the disk reconnection and update the node ID as well as the data path for the related replicas (including failed replicas). https://github.com/longhorn/longhorn/issues/1269 The goal of this feature is to reuse the existing data of the failed replica when the corresponding disk is back. As for how to reuse the existing data and handle rebuild related feature, it is already implemented in #1304, which is not the intention of this enhancement. Identifying the disk that is previously used in Longhorn is not the the key point. The essential of this feature is that Longhorn should know where to reuse existing data of all related replicas when the disk is reconnected. In other words, the fields that indicating the replica data position should be updated when the disk is reconnected. Before the enhancement, there is no way to reuse the existing data when a disk is reconnected/migrated. After the enhancement, this can be done by: detach the volumes using the disk Reconnect the disk to the another node (both nodes keep running) reattach the related volumes Before the enhancement, there is no chance to reuse the failed replicas on the node. After the enhancement, Longhorn will update the path and node id for all failed replicas using the disks, then Longhorn can reuse the failed replicas during rebuilding. Detach all related volumes using the disk before the disk migration. Directly move the disk to the new node (physically or in cloud vendor) and mount the disk. Add the disk with the new mount point to the corresponding new Longhorn node in Longhorn Node page. Attach the volumes for the workloads. Directly shut down the node when there are replicas on the node. Then the replicas on the node will fail. Move the disks on the down node to other running nodes (physically or in cloud vendor). Add the disk with the new mount point to the corresponding new Longhorn node in Longhorn Node page. Wait then verify the failed replicas using the disk will be reused, and the node ID & path info will be updated. There is no API change. When a disk is ready, Longhorn can list all related replicas via `replica.Spec.DiskID` then sync up node ID and path info for these replicas. If a disk is not ready, the scheduling info will be cleaned

Longhorn won't be confused of updating replicas if multiple disconnected disks using the same Disk UUID. Need to add a disk related label for replicas. Store DiskUUID rather than the disk name in `replica.Spec.DiskID` Need to update `DiskID` for existing replicas during upgrade. Since the disk path of a replica may get changed but the data directory name is immutable. It's better to split `replica.Spec.DataPath` to `replica.Spec.DiskPath` and `replica.Spec.DataDirectoryName`. Then it's more convenient to sync up the disk path for replicas. Need to update the path fields for existing replicas during upgrade. Disable the node soft anti-affinity. Create a new host disk. Disable the default disk and add the extra disk with scheduling enabled for the current node. Launch a Longhorn volume with 1 replica. Then verify the only replica is scheduled to the new disk. Write random data to the volume then verify the data. Detach the volume. Unmount then remount the disk to another path. (disk migration) Create another Longhorn disk based on the migrated path. Verify the Longhorn disk state. The Longhorn disk added before the migration should become "unschedulable". The Longhorn disk created after the migration should become "schedulable". Verify the replica DiskID and the path is updated. Attach the volume. Then verify the state and the data. Set `ReplicaReplenishmentWaitInterval`. Make sure it's longer than the time needs for node replacement. Launch a Kubernetes cluster with the nodes in AWS Auto Scaling group. Then Deploy Longhorn. Deploy some workloads using Longhorn volumes. Wait for/Trigger the ASG instance replacement. Verify new replicas won't be created before reaching `ReplicaReplenishmentWaitInterval`. Verify the failed replicas are reused after the node recovery. Verify if workloads still work fine with the volumes after the recovery. Launch Longhorn v1.0.x Create and attach a volume, then write data to the volume. Directly remove a Kubernetes node, and shut down a node. Wait for the related replicas failure. Then record `replica.Spec.DiskID` for the failed replicas. Upgrade to Longhorn master Verify the Longhorn node related to the removed node is gone. Verify `replica.Spec.DiskID` on the down node is updated and the field of the replica on the gone node is unchanged. `replica.Spec.DataPath` for all replicas becomes empty. Remove all unscheduled replicas. Power on the down node. Wait for the failed replica on the down node being reused. Wait for a new replica being replenished and available. Need to update disk ID and data path for existing replicas.