Project: Longhorn (Runtime / Cloud Native Storage)
File: 20230616-automatic-offline-replica-rebuild.md

Currently, Longhorn does not have the capability to support online replica rebuilding for volumes utilizing the V2 Data Engine. However, an automatic offline replica rebuilding mechanism has been implemented as a solution to address this limitation. https://github.com/longhorn/longhorn/issues/6071 Support volumes using v2 data engine Support volumes using v1 data engine In the event of abnormal power outages or network partitions, replicas of a volume may be lost, resulting in volume degradation. Unfortunately, volumes utilizing the v2 data engine do not currently have the capability for online replica rebuilding. As a solution to address this limitation, Longhorn has implemented an automatic offline replica rebuilding mechanism. When a degraded volume is detached, this mechanism places the volume in maintenance mode and initiates the rebuilding process. After the rebuilding is successfully completed, the volume is detached according to the user's specified expectations. If a volume using the v2 data engine is degraded, the online replica rebuilding process is currently unsupported. If offline replica rebuilding feature is enabled when one of the conditions is met Global setting `offline-replica-rebuild` is `enabled` and `Volume.Spec.OfflineReplicaRebuilding` is `ignored` `Volume.Spec.OfflineReplicaRebuilding` is `enabled` The volume's `Spec.OfflineReplicaRebuildingRequired` is set to `true` if a volume is degraded. When a degraded volume is detached, this mechanism places the volume in maintenance mode and initiates the rebuilding process. After the rebuilding is successfully completed, the volume is detached according to the user's specified expectations. If a user attaches the volume without enabling maintenance mode while the replica rebuilding process is in progress, the ongoing replica rebuilding operation will be terminated. Settings Add global setting `offline-replica-rebuilding`. Default value is `enabled`. The available options are: `enabled` `disable` CRD Add `Volume.Spec.OfflineReplicaRebuilding`. The available options are: ignored`: The volume's offline replica rebuilding behavior follows the settings defined by the global setting `offline-replica-rebuilding`. `enabled`: Offline replica rebuilding of the volume is always enabled. `disabled`: Offline replica rebuilding of the volume is always disabled. Add `Volume.Status.OfflineReplicaRebuildingRequired` Controller Add `volume-rebuilding-controller` for creating and deleting `volume-rebuilding-controller` attachment ticket. Logics A volume-controller sets 'Volume.Status.OfflineReplicaRequired' to 'true' when it realizes a v2 data engine is degraded. If a volume's `Volume.Status.OfflineReplicaRebuildingRequired` is `true`, volume-rebuilding-controller creates a `volume-rebuilding-controller` attachment ticket with frontend disabled and lower priority than tickets with workloads. When the volume is detached, volume-attachment-controller attaches the volume with a `volume-rebuilding-controller` attachment ticket in maintenance mode. volume-controller triggers replica rebuilding. After finishing the replica rebuilding, the volume-controller sets `Volume.Status.OfflineReplicaRebuildingRequired` to `false` if a number of healthy replicas is expected. volume-rebuilding-controller deletes the 'volume-rebuilding-controller' attachment ticket. volume-attachment-controller is aware of the deletion of the `volume-rebuilding-controller` attachment ticket, which causes volume detachment. Degraded Volume lifecycle (creation, attachment, detachment and deletion) and automatic replica rebuilding