Project: Longhorn (Runtime / Cloud Native Storage)
File: 20221024-longhorn-volumeattachment.md

There are several cases related to (auto) volume attach/detach, right now we leverage the volume attributes to achieve that, but it's better to introduce a new resource (longhorn volumeattachement) to have complete context for each scenario. https://github.com/longhorn/longhorn/issues/3715 Introduce a new resource (Longhorn volumeattachement) to cover the following scenarios for Longhorn volume's AD: Traditional CSI attachment (pod -> csi-attacher -> Longhorn API) Traditional UI attachment (Longhorn UI -> Longhorn API) Auto attach/detach volume for K8s CSI snapshot Auto attach/detach volume for recurring jobs Auto attach/detach volume for volume cloning Auto attach/detach volume for auto salvage feature Refactor RWX mechanism's volume attachment/detachment (in share manager lifecycle) Volume live migration Consider how to upgrade from previous Longhorn version which doesn't have VA resource yet NA This is where we get down to the nitty-gritty of what the proposal actually is. Before this feature there are race conditions between Longhorn auto-reattachment logic and CSI volume attachment that sometimes result in the volume CR in a weird state that volume controller can nerve resolve. Ref https://github.com/longhorn/longhorn/issues/2527#issuecomment-966597537 After this feature, the race condition should not exist Before this feature, the user cannot take a CSI snapshot for detached Longhorn volume. After this feature, user should be able to do so Ref: https://github.com/longhorn/longhorn/issues/3726 Make the attaching/detaching more resilient and transparent. User will see clearly who is requesting the volume to be attached in the AD ticket. Also, volume controller will be able to reconcile the volume in any combination value of (volume.Spec.NodeID, volume.Status.CurrentNodeID, and volume.Status.State). See the Create a new CRD called VolumeAttachment with the following structure: ```go type AttachmentTicket struct { // The unique ID of this attachment. Used to differentiate different attachments of the same volume. // +optional ID string `json:"id"` // +optional Type AttacherType `json:"type"` // The node that this attachment is requesting // +optional NodeID string `json:"nodeID"` // Optional additional parameter for this attachment // +optional Parameters map[string]string `json:"parameters"` // A sequence number representing a specific generation of the desired state. // Populated by the system. Read-only. // +optional Generation int64 `json:"generation"` } type AttachmentTicketStatus struct { // The unique ID of this attachment. Used to differentiate different attachments of the same volume. // +optional ID string `json:"id"` // Indicate whether this attachment ticket has been satisfied Satisfied bool `json:"satisfied"` // Record any error when trying to fulfill this attachment // +nullable Conditions []Condition `json:"conditions"` // A sequence number representing a specific generation of the desired state. // Populated by the system.

// +optional Generation int64 `json:"generation"` } type AttacherType string const ( AttacherTypeCSIAttacher = AttacherType("csi-attacher") AttacherTypeLonghornAPI = AttacherType("longhorn-api") AttacherTypeSnapshotController = AttacherType("snapshot-controller") AttacherTypeBackupController = AttacherType("backup-controller") AttacherTypeVolumeCloneController = AttacherType("volume-clone-controller") AttacherTypeSalvageController = AttacherType("salvage-controller") AttacherTypeShareManagerController = AttacherType("share-manager-controller") AttacherTypeVolumeRestoreController = AttacherType("volume-restore-controller") AttacherTypeVolumeEvictionController = AttacherType("volume-eviction-controller") AttacherTypeVolumeExpansionController = AttacherType("volume-expansion-controller") AttacherTypeBackingImageDataSourceController = AttacherType("bim-ds-controller") AttacherTypeVolumeRebuildingController = AttacherType("volume-rebuilding-controller") ) const ( AttacherPriorityLevelVolumeRestoreController = 2000 AttacherPriorityLevelVolumeExpansionController = 2000 AttacherPriorityLevelLonghornAPI = 1000 AttacherPriorityLevelCSIAttacher = 900 AttacherPriorityLevelSalvageController = 900 AttacherPriorityLevelShareManagerController = 900 AttacherPriorityLevelSnapshotController = 800 AttacherPriorityLevelBackupController = 800 AttacherPriorityLevelVolumeCloneController = 800 AttacherPriorityLevelVolumeEvictionController = 800 AttacherPriorityLevelBackingImageDataSourceController = 800 AttachedPriorityLevelVolumeRebuildingController = 800 ) const ( TrueValue = "true" FalseValue = "false" AnyValue = "any" AttachmentParameterDisableFrontend = "disableFrontend" AttachmentParameterLastAttachedBy = "lastAttachedBy" ) const ( AttachmentStatusConditionTypeSatisfied = "Satisfied" AttachmentStatusConditionReasonAttachedWithIncompatibleParameters = "AttachedWithIncompatibleParameters" ) // VolumeAttachmentSpec defines the desired state of Longhorn VolumeAttachment type VolumeAttachmentSpec struct { // +optional AttachmentTickets map[string]*AttachmentTicket `json:"attachmentTickets"` // The name of Longhorn volume of this VolumeAttachment Volume string `json:"volume"` } // VolumeAttachmentStatus defines the observed state of Longhorn VolumeAttachment type VolumeAttachmentStatus struct { // +optional AttachmentTicketStatuses map[string]*AttachmentTicketStatus `json:"attachmentTicketStatuses"` } // VolumeAttachment stores attachment information of a Longhorn volume type VolumeAttachment struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec VolumeAttachmentSpec `json:"spec,omitempty"` Status VolumeAttachmentStatus `json:"status,omitempty"` } ``` Modify volume controller Repurpose the field `volume.Status.CurrentNodeID` so that `volume.Status.CurrentNodeID` is only set once we are fully attached and is only unset once we are fully detached. See this state flow for full detail: Deprecate `volume.Status.PendingNodeID` and the auto-salvage logic. We will have a dedicated `salvage-controller` as describe in the below section. Create a controller, VolumeAttachment controller (AD controller). This controller watches the VolumeAttachment objects of a volume. When AD controller sees a newly created ticket in `VolumeAttachment.Spec.AttachmentTickets` object. If `volume.Spec.NodeID` is non-empty Do nothing. It will wait for the volume to be fully detached first before setting `volume.Spec.NodeID` If `volume.Spec.NodeID` is empty Wait for `volume.Status.State` to be `detached` Then select an attachment ticket from `va.Spec.AttachmentTickets` based on priority level of the tickets. If 2 ticket has same priority, select the ticket with shorter name. Set the `vol.Spec.NodeID = attachmentTicket.NodeID` to attach the volume When AD controller check the list of tickets in `VolumeAttachment.Spec.AttachmentTickets`. If no ticket is requesting the `volume.Spec.NodeID`, Ad controller set `volume.Spec.NodeID` to empty AD controller watch volume CR and set ticket status accordingly in the `va.Status.AttachmentTicketStatuses` If the VolumeAttachment object is pending deletion, There is no special resource need to be cleanup, directly remove the finalizer for the VolumeAttachment Note that the priority of ticket is determine in the order: volume data restoring > user workload > snapshot/backup operations Traditional CSI attachment (pod -> csi-attacher -> Longhorn API) csi-attacher send attaching request to longhorn-csi-plugin longhorn-csi-plugin sends attaching request to longhorn-manager with pod-id and attacherType `csi-attacher` longhorn manager create a VolumeAttachment object with this spec: ```yaml metadata: finalizers: longhorn.io labels: longhornvolume: <volume-name> nodeID: <node-name> spec: attachers: csi-attacher: <pod-id>: id: <pod-id> type: "csi-attacher" volume: <volume-name> nodeID: <node-name> status: attached: false ``` longhorn-csi-plugin watch the `volumeAttachment.Status.Attached` and `volumeAttachment.Status.AttachError` and return corresponding respond to csi-attacher Traditional UI attachment (Longhorn UI -> Longhorn API) Longhorn UI send attaching request to longhorn-manager with attacherType `longhorn-api` longhorn manager create a VolumeAttachment object with this spec: ```yaml metadata: finalizers: longhorn.io labels: longhornvolume: <volume-name> nodeID: <node-name> spec: attachers: longhorn-api: "": id: "" type: "longhorn-api" volume: <volume-name> nodeID: <node-name> status: attached: false ``` Longhorn UI watches the `volumeAttachment.Status.Attached` and `volumeAttachment.Status.AttachError` and display the correct message Auto attach/detach volume Longhorn snapshot Snapshot controller watches the new Longhorn snapshot

If the snapshot CR request a new snapshot, snapshot controller create a new VolumeAttachment object with the content: ```yaml metadata: finalizers: longhorn.io labels: longhornvolume: <volume-name> nodeID: <node-name> spec: attachers: snapshot-controller: <snapshot-name>: id: <snapshot-name> type: "snapshot-controller" volume: <volume-name> nodeID: <node-name> status: attached: false ``` Snapshot controller wait for volume to be attached and take the snapshot Auto attach/detach volume Longhorn backup Backup controller watches the new Longhorn backup CR. If the backup CR request a new backup, backup controller create a new VolumeAttachment object with the content: ```yaml metadata: finalizers: longhorn.io labels: longhornvolume: <volume-name> nodeID: <node-name> spec: attachers: backup-controller: <backup-name>: id: <backup-name> type: "backup-controller" volume: <volume-name> nodeID: <node-name> status: attached: false ``` Backup controller wait for volume to be attached and take the backup Auto attach/detach volume for K8s CSI snapshot csi-snappshotter send a request to longhorn-csi-plugin to with snapshot name and volume name longhorn-csi-plugin send snapshot request to Longhorn manager Longhorn manager create a snapshot CR longhorn-csi-plugin watches different snapshot CR status and respond to csi-snapshotter Auto attach/detach volume for recurring jobs recurring job deploys backup/snapshot CR recurring job watch the status of backup/snapshot CR for the completion of the operation Auto attach/detach volume for volume cloning Create a new controller: cloning-controller This controller will watch new volume that need to be cloned For a volume that needs to be cloned, cloning controller deploy a VolumeAttachment for both target volume and old volume: ```yaml metadata: finalizers: longhorn.io labels: longhornvolume: <target-volume-name> nodeID: <node-name> spec: attachers: cloning-controller: <target-volume-name>: id: <target-volume-name> type: "cloning-controller" volume: <target-volume-name> nodeID: <node-name> status: attached: false metadata: finalizers: longhorn.io labels: longhornvolume: <source-volume-name> nodeID: <node-name> spec: attachers: cloning-controller: <target-volume-name>: id: <target-volume-name> type: "cloning-controller" volume: <source-volume-name> nodeID: <node-name> status: attached: false ``` Cloning controller watch for the cloning status and delete the VolumeAttachment upon completed Auto attach/detach volume for auto salvage feature We create a new controller to check if the volume is faulted and detach the volume for detach. After the volume is auto detach and replica is auto-salvaged by the volume controller, the AD controller will check the VolumeAttachment object and reattach the volume to the correct node With this design, we no longer need the volume.Status.PendingNodeID which was the source of some race conditions Refactor RWX mechanism's volume attachment/detachment (in share manager lifecycle) Each pod that use the RWX volume will directly request a CSI ticket. The share-manager controller watch the csi ticket and create share-manager ticket when there are one or more csi ticket exist. Then AD controller will attach the volume to the node that is being requested by share-manager ticket. AD controller ignore the CSI ticket for RWX volume. We are adding new APIs to operate on Snapshot CRD directly ```go "snapshotCRCreate": s.SnapshotCRCreate, "snapshotCRList": s.SnapshotCRList, "snapshotCRGet": s.SnapshotCRGet, "snapshotCRDelete": s.SnapshotCRDelete, ``` Refer to the test plan https://github.com/longhorn/longhorn/issues/3715#issuecomment-1563637861 In the upgrade path, list all volume and create VolumeAttachment for the them. For the volume that currently being used by CSI workload/Longhorn UI we create CSI ticket/Longhorn UI ticket to keep them being attached