Project: Longhorn (Runtime / Cloud Native Storage)
File: 20220420-longhorn-snapshot-crd.md

Supporting Longhorn snapshot CRD allows users to query/create/delete volume snapshots using kubectl. This is one step closer to making kubectl as Longhorn CLI. Also, this will be a building block for the future auto-attachment/auto-detachment refactoring for snapshot creation, deletion, volume cloning. https://github.com/longhorn/longhorn/issues/3144 Support Longhorn snapshot CRD to allow users to query/create/delete volume snapshots using kubectl. A building block for the future auto-attachment/auto-detachment refactoring for snapshot creation, deletion, volume cloning. Pay attention to scalability problem. A cluster with 1k volumes might have 30k snapshots. We should make sure not to overload the controller work-queue as well as making too many grpc calls to engine processes. Introduce a new CRD, snapshot CRD and the snapshot controller. The life cycle of a snapshot CR is as below: Create (by engine monitor/kubectl) When user create a new snapshot CR, Longhorn try to create a new snapshot When there is a snapshot in the volume that isn't corresponding to any snapshot CR, Longhorn will generate snapshot CR for that snapshot Update (by snapshot controller) Snapshot controller will reconcile the snapshot CR status with the snapshot info inside the volume engine Delete (by engine monitor/kubectl) When a snapshot CR is deleted (by user or by Longhorn), snapshot controller will make sure that the snapshot are removed from the engine before remove the finalizer and allow the deletion Deleting volume should be blocked until all of its snapshot are removed When there is a system generated snapshot CR that isn't corresponding to any snapshot info inside engine status, Longhorn will delete the snapshot CR Before this enhancement, users have to use Longhorn UI to query/create/delete volume snapshot. For user with only access to CLI, another option is to use our . However, the Python client are not as intuitive and easy as using kubectl. After this enhancement, users will be able to use kubectl to query/create/delete Longhorn snapshots just like what they can do with Longhorn backups. There is no additional requirement for users to use this feature. The experience details should be in the `User Experience In Detail` later. User wants to limit the snapshot count to save space. Snapshot RecurringJobs set to Retain X number of snapshots do not touch unrelated snapshots, so if one ever changes the name of the RecurringJob, the old snapshots will stick around forever. These then have to be manually deleted in the UI. There might be some kind of browser automation framework might also work for pruning large numbers of snapshots, but this feels janky. Having a CRD for snapshots would greatly simplify this, as one could prune snapshots using kubectl, much like how one can currently manage backups using kubectl due to the existence of the `backups.longhorn.io` CRD. There is no additional requirement for users to use this

We don't want to have disruptive changes in this initial version of snapshot CR (e.g., snapshot API create/delete shouldn't change. Snapshot status is still inside the engine status). We can wait for the snapshot CRD to be a bit more mature (no issue with scalability) and make the disruptive changes in the next version of snapshot CR (e.g., snapshot API create/delete changes to create/delete snapshot CRs. Snapshot status is removed from inside the engine status) Introduce a new CRD, snapshot CRD and the snapshot controller. The snapshot CRD is: ```yaml // SnapshotSpec defines the desired state of Longhorn Snapshot type SnapshotSpec struct { // the volume that this snapshot belongs to. // This field is immutable after creation. // Required Volume string `json:"volume"` // require creating a new snapshot // +optional CreateSnapshot bool `json:"createSnapshot"` // The labels of snapshot // +optional // +nullable Labels map[string]string `json:"labels"` } // SnapshotStatus defines the observed state of Longhorn Snapshot type SnapshotStatus struct { // +optional Parent string `json:"parent"` // +optional // +nullable Children map[string]bool `json:"children"` // +optional MarkRemoved bool `json:"markRemoved"` // +optional UserCreated bool `json:"userCreated"` // +optional CreationTime string `json:"creationTime"` // +optional Size int64 `json:"size"` // +optional // +nullable Labels map[string]string `json:"labels"` // +optional OwnerID string `json:"ownerID"` // +optional Error string `json:"error,omitempty"` // +optional RestoreSize int64 `json:"restoreSize"` // +optional ReadyToUse bool `json:"readyToUse"` } ``` The life cycle of a snapshot CR is as below: Create When a snapshot CR is created, Longhorn mutation webhook will: Add a volume label `longhornvolume: <VOLUME-NAME>` to the snapshot CR. This allow us to efficiently find snapshots corresponding to a volume without having listing potentially thoundsands of snapshots. Add `longhornFinalizerKey` to snapshot CR to prevent it from being removed before Longhorn has change to clean up the corresponding snapshot Populate the value for `snapshot.OwnerReferences` to uniquely identify the volume of this snapshot. This field contains the volume UID to uniquely identify the volume in case the old volume was deleted and a new volume was created with the same name. For user created snapshot CR, the field `Spec.CreateSnapshot` should be set to `true` indicating that Longhorn should provision a new snapshot for this CR. Longhorn snapshot controller will pick up this CR, check to see if there already is a snapshot inside the `engine.Status.Snapshots`. If there is there already a snapshot inside engine.Status.Snapshots, update the snapshot.Status with the snapshot info inside `engine.Status.Snapshots` If there isn't a snapshot inside `engine.Status.Snapshots` then: making a call to engine process to check if there already a snapshot with the same name. This is to make sure we don't accidentally create 2 snapshots with the same name. This logic can be remove after is resolved If the snapshot doesn't inside the engine process, make another call to create the snapshot For the snapshots that are already exist inside `engine.Status.Snapshots` but doesn't have corresponding snapshot CRs

system generated snapshots), the engine monitoring will generate snapshot CRs for them. The snapshot CR generated by engine monitoring with have `Spec.CreateSnapshot` set to `false`, Longhorn snapshot controller will not create a snapshot for those CRs. The snapshot controller only sync status for those snapshot CRs Update Snapshot CR spec and label are immutable after creation. It will be protected by the admission webhook Sync the snapshot info from `engine.Status.Snapshots` to the `snapshot.Status`. If there is any error or if the snapshot is marked as removed, set `snapshot.Status.ReadyToUse` to `false` If there there is no snapshot info inside `engine.Status.Snapshots`, mark the `snapshot.Status.ReadyToUse` to `false`and populate the `snapshot.Status.Error` with the lost message. This snapshot will eventually be updated again when engine monitoring update `engine.Status.Snapshots` or it may be cleanup as the section below Delete Engine monitor will responsible for removing all snapshot CRs that don't have a matching snapshot info and are in one of the following cases: The snapshot CRs with `Spec.CreateSnapshot: false` (snapshot CR that is auto generated by the engine monitoring) The snapshot CRs with `Spec.CreateSnapshot: true` and `snapCR.Status.CreationTime != nil` (snapshot CR that has requested a new snapshot and the snapshot has already provisioned before but no longer exist now) When a snapshot CR has deletion timestamp set, snapshot controller will: Check to see if the actual snapshot inside engine process exist. If it exist do: if has not been marked as removed, issue grpc call to engine process to remove the snapshot Check if the engine is in the purging state, if not issue a snapshot purge call to engine process If it doesn't exist, remove the `longhornFinalizerKey` to allow the deletion of the snapshot CR Integration test plan. For engine enhancement, also requires engine integration test plan. Anything that requires if user want to upgrade to this enhancement How do we address scalability issue? Controller workqueue Disable resync period for snapshot informer Enqueue snapshot only when: There is a change in snapshot CR There is a change in `engine.Status.CurrentState` (volume attach/detach event), `engine.Status.PurgeStatus` (for snapshot deletion event), `engine.Status.Snapshots` (for snapshot creation/update event) This enhancement proposal doesn't make additional call to engine process comparing to the existing design. For the special snapshot `volume-head`, we don't create a snapshot CR for this special snapshot because: From the usecase perspective, user cannot delete this snapshot anyway so there is no need to generate this snapshot The name `volume-head` is not globally uniquely, we might have to include volume name if we want to generate this snapshot CR We would have to implement special logic to prevent user from deleting this special CR On the flip side, if we generate this special CR, user will have a complete picture of the snapshot chain The VolumeHead CR may suddenly point to another actual file during the snapshot creation.