Project: Velero (Runtime / Cloud Native Storage)
File: new-prepost-backuprestore-plugin-hooks.md

Velero should provide a way to trigger actions before and after each backup and restore. Important: These proposed plugin hooks are fundamentally different from the existing plugin hooks, BackupItemAction and RestoreItemAction, which are triggered per resource item during backup and restore, respectively. The proposed plugin hooks are to be executed only once: pre-backup (before backup starts), post-backup (after the backup is completed and uploaded to object storage, including volumes snapshots), pre-restore (before restore starts) and post-restore (after the restore is completed, including volumes are restored). For the backup, the sequence of events of Velero backup are the following (these sequence depicted is prior upcoming changes for ): ``` New Backup Request |--> Validation of the request |--> Set Backup Phase "In Progress" | --> Start Backup | --> Discover all Plugins |--> Check if Backup Exists |--> Backup all K8s Resource Items |--> Perform all Volumes Snapshots |--> Final Backup Phase is determined |--> Persist Backup and Logs on Object Storage ``` We propose the pre-backup and post-backup plugin hooks to be executed in this sequence: ``` New Backup Request |--> Validation of the request |--> Set Backup Phase "In Progress" | --> Start Backup | --> Discover all Plugins |--> Check if Backup Exists |--> PreBackupActions are executed, logging actions on existent backup log file |--> Backup all K8s Resource Items |--> Perform all Volumes Snapshots |--> Final Backup Phase is determined |--> Persist Backup and logs on Object Storage |--> PostBackupActions are executed, logging to its own file ``` These plugin hooks will be invoked: PreBackupAction: plugin actions are executed after the backup object is created and validated but before the backup is being processed, more precisely before function . If the PreBackupActions return an err, the backup object is not processed and the Backup phase will be set as `FailedPreBackupActions`. PostBackupAction: plugin actions are executed after the backup is finished and persisted, more precisely after function . The proposed plugin hooks will execute actions that will have statuses on their own: `Backup.Status.PreBackupActionsStatuses` and `Backup.Status.PostBackupActionsStatuses` which will be an array of a proposed struct `ActionStatus` with PluginName, StartTimestamp, CompletionTimestamp and Phase. For the restore, the sequence of events of Velero restore are the following (these sequence depicted is prior upcoming changes for ): ``` New Restore Request |--> Validation of the request |--> Checks if restore is from a backup or a schedule |--> Fetches backup |--> Set Restore Phase "In Progress" |--> Start Restore |--> Discover all Plugins |--> Download backup file to temp |--> Fetch list of volumes snapshots |--> Restore K8s items, including PVs |--> Final Restore Phase is determined |--> Persist Restore logs on Object Storage ``` We propose the pre-restore and post-restore plugin hooks to be executed in this sequence: ``` New Restore Request |--> Validation of the request |--> Checks if restore is from a backup or a schedule |--> Fetches backup |--> Set Restore Phase "In Progress" |--> Start Restore |--> Discover all Plugins |--> Download backup file to temp |--> Fetch list of volumes snapshots |--> PreRestoreActions are executed, logging actions on existent backup log file |--> Restore K8s items, including PVs |--> Final Restore Phase is determined |--> Persist Restore logs on Object Storage |--> PostRestoreActions are executed, logging to its own file ``` These plugin hooks will be invoked: PreRestoreAction: plugin actions are executed after the restore object is created and validated and before the backup object is fetched, more precisely in function `runValidatedRestore` after function

If the PreRestoreActions return an err, the restore object is not processed and the Restore phase will be set a `FailedPreRestoreActions`. PostRestoreAction: plugin actions are executed after the restore finishes processing all items and volumes snapshots are restored and logs persisted, more precisely in function `processRestore` after setting . The proposed plugin hooks will execute actions that will have statuses on their own: `Restore.Status.PreRestoreActionsStatuses` and `Restore.Status.PostRestoreActionsStatuses` which will be an array of a proposed struct `ActionStatus` with PluginName, StartTimestamp, CompletionTimestamp and Phase. Increasingly, Velero is employed for workload migrations across different Kubernetes clusters. Using Velero for migrations requires an atomic operation involving a Velero backup on a source cluster followed by a Velero restore on a destination cluster. It is common during these migrations to perform many actions inside and outside Kubernetes clusters. Attention: these actions are not per resource item, but they are actions to be executed once before and/or after the migration itself (remember, migration in this context is Velero Backup + Velero Restore). One important use case driving this proposal is migrating stateful workloads at scale across different clusters/storage backends. Today, Velero's Restic integration is the response for such use cases, but there are some limitations: Quiesce/unquiesce workloads: Pod hooks are useful for quiescing/unquiescing workloads, but platform engineers often do not have the luxury/visibility/time/knowledge to go through each pod in order to add specific commands to quiesce/unquiesce workloads. Orphan PVC/PV pairs: PVCs/PVs that do not have associated running pods are not backed up and consequently, are not migrated. Aiming to address these two limitations, and separate from this proposal, we would like to write a Velero plugin that takes advantage of the proposed Pre-Backup plugin hook. This plugin will be executed once (not per resource item) prior backup. It will scale down the applications setting `.spec.replicas=0` to all deployments, statefulsets, daemonsets, replicasets, etc. and will start a small-footprint staging pod that will mount all PVC/PV pairs. Similarly, we would like to write another plugin that will utilize the proposed Post-Restore plugin hook. This plugin will unquiesce migrated applications by killing the staging pod and reinstating original `.spec.replicas` values after the Velero restore is completed. Other examples of plugins that can use the proposed plugin hooks are: PostBackupAction: trigger a Velero Restore after a successful Velero backup (and complete the migration operation). PreRestoreAction: pre-expand the cluster's capacity via Cluster API to avoid starvation of cluster resources before the restore. PostRestoreAction: call actions to be performed outside Kubernetes clusters, such as configure a global load balancer (GLB) that enables the new cluster. The post backup actions will be executed after the backup is uploaded (persisted) on the disk. The logs of post-backup actions will be uploaded on the disk once the actions are completed. The post restore actions will be executed after the restore is uploaded (persisted) on the disk. The logs of post-restore actions will be uploaded on the disk once the actions are completed. This design seeks to provide missing extension points. This proposal's scope is to only add the new plugin hooks, not the plugins themselves. Provide PreBackupAction, PostBackupAction, PreRestoreAction, and PostRestoreAction APIs for plugins to implement. Update Velero backup and restore creation logic to invoke registered PreBackupAction and PreRestoreAction plugins before processing the backup and restore respectively. Update Velero backup and restore complete logic to invoke registered PostBackupAction and PostRestoreAction plugins the objects are uploaded on disk. Create one `ActionStatus` struct to keep track of execution of the plugin hooks. This struct has PluginName, StartTimestamp, CompletionTimestamp and Phase. Add sub statuses for the plugins on Backup object: `Backup.Status.PreBackupActionsStatuses` and `Backup.Status.PostBackupActionsStatuses`. They will be flagged as optional and

They will be populated only each plugin registered for the PreBackup and PostBackup hooks, respectively. Add sub statuses for the plugins on Restore object: `Backup.Status.PreRestoreActionsStatuses` and `Backup.Status.PostRestoreActionsStatuses`. They will be flagged as optional and nullable. They will be populated only each plugin registered for the PreRestore and PostRestore hooks, respectively. that will be populated optionally if Pre/Post Backup/Restore. Specific implementations of the PreBackupAction, PostBackupAction, PreRestoreAction and PostRestoreAction API beyond test cases. For migration specific actions (Velero Backup + Velero Restore), add disk synchronization during the validation of the Restore (making sure the newly created backup will show during restore) The Velero backup controller package will be modified for `PreBackupAction` and `PostBackupAction`. The PreBackupAction plugin API will resemble the BackupItemAction plugin hook design, but with the fundamental difference that it will receive only as input the Velero `Backup` object created. It will not receive any resource list items because the backup is not yet running at that stage. In addition, the `PreBackupAction` interface will only have an `Execute()` method since the plugin will be executed once per Backup creation, not per item. The Velero backup controller will be modified so that if there are any PreBackupAction plugins registered, they will be The PostBackupAction plugin API will resemble the BackupItemAction plugin design, but with the fundamental difference that it will receive only as input the Velero `Backup` object without any resource list items. By this stage, the backup has already been executed, with items backed up and volumes snapshots processed and persisted. The `PostBackupAction` interface will only have an `Execute()` method since the plugin will be executed only once per Backup, not per item. If there are any PostBackupAction plugins registered, they will be executed after the backup is finished and persisted, more precisely after function . The Velero restore controller package will be modified for `PreRestoreAction` and `PostRestoreAction`. The PreRestoreAction plugin API will resemble the RestoreItemAction plugin design, but with the fundamental difference that it will receive only as input the Velero `Restore` object created. It will not receive any resource list items because the restore has not yet been running at that stage. In addition, the `PreRestoreAction` interface will only have an `Execute()` method since the plugin will be executed only once per Restore creation, not per item. The Velero restore controller will be modified so that if there are any PreRestoreAction plugins registered, they will be executed after the restore object is created and validated and before the backup object is fetched, more precisely in function `runValidatedRestore` after function . If the PreRestoreActions return an err, the restore object is not processed and the Restore phase will be set a `FailedPreRestoreActions`. The PostRestoreAction plugin API will resemble the RestoreItemAction plugin design, but with the fundamental difference that it will receive only as input the Velero `Restore` object without any resource list items. At this stage, the restore has already been executed. The `PostRestoreAction` interface will only have an `Execute()` method since the plugin will be executed only once per Restore, not per item. If any PostRestoreAction plugins are registered, they will be executed after the restore finishes processing all items and volumes snapshots are restored and logs persisted, more precisely in function `processRestore` after setting . To keep the status of the plugins, we propose the following struct: ```go type ActionStatus struct { // PluginName is the name of the registered plugin // retrieved by the PluginManager as id.Name // +optional // +nullable PluginName string `json:"pluginName,omitempty"` // StartTimestamp records the time the plugin started. // +optional // +nullable StartTimestamp *metav1.Time `json:"startTimestamp,omitempty"` // CompletionTimestamp records the time the plugin was

// +optional // +nullable CompletionTimestamp *metav1.Time `json:"completionTimestamp,omitempty"` // Phase is the current state of the Action. // +optional // +nullable Phase ActionPhase `json:"phase,omitempty"` } // ActionPhase is a string representation of the lifecycle phase of an action being executed by a plugin // of a Velero backup. // +kubebuilder:validation:Enum=InProgress;Completed;Failed type ActionPhase string const ( // ActionPhaseInProgress means the action has being executed ActionPhaseInProgress ActionPhase = "InProgress" // ActionPhaseCompleted means the action finished successfully ActionPhaseCompleted ActionPhase = "Completed" // ActionPhaseFailed means the action failed ActionPhaseFailed ActionPhase = "Failed" ) ``` The `Backup` Status section will have the follow: ```go type BackupStatus struct { (...) // PreBackupActionsStatuses contains information about the pre backup plugins's execution. // Note that this information is will be only populated if there are prebackup plugins actions // registered // +optional // +nullable PreBackupActionsStatuses *[]ActionStatus `json:"preBackupActionsStatuses,omitempty"` // PostBackupActionsStatuses contains information about the post backup plugins's execution. // Note that this information is will be only populated if there are postbackup plugins actions // registered // +optional // +nullable PostBackupActionsStatuses *[]ActionStatus `json:"postBackupActionsStatuses,omitempty"` } ``` The `Restore` Status section will have the follow: ```go type RestoreStatus struct { (...) // PreRestoreActionsStatuses contains information about the pre Restore plugins's execution. // Note that this information is will be only populated if there are preRestore plugins actions // registered // +optional // +nullable PreRestoreActionsStatuses *[]ActionStatus `json:"preRestoreActionsStatuses,omitempty"` // PostRestoreActionsStatuses contains information about the post restore plugins's execution. // Note that this information is will be only populated if there are postrestore plugins actions // registered // +optional // +nullable PostRestoreActionsStatuses *[]ActionStatus `json:"postRestoreActionsStatuses,omitempty"` } ``` In case the PreBackupActionsStatuses has at least one `ActionPhase` = `Failed`, it means al least one of the plugins returned an error and consequently, the backup will not move forward. The final status of the Backup object will be set as `FailedPreBackupActions`: ```go // BackupPhase is a string representation of the lifecycle phase // of a Velero backup. // +kubebuilder:validation:Enum=New;FailedValidation;FailedPreBackupActions;InProgress;Uploading;UploadingPartialFailure;Completed;PartiallyFailed;Failed;Deleting type BackupPhase string const ( (...) // BackupPhaseFailedPreBackupActions means one or more the Pre Backup Actions has failed // and therefore backup will not run. BackupPhaseFailedPreBackupActions BackupPhase = "FailedPreBackupActions" (...) ) ``` In case the PreRestoreActionsStatuses has at least one `ActionPhase` = `Failed`, it means al least one of the plugins returned an error and consequently, the restore will not move forward. The final status of the Restore object will be set as `FailedPreRestoreActions`: ```go // RestorePhase is a string representation of the lifecycle phase // of a Velero restore // +kubebuilder:validation:Enum=New;FailedValidation;FailedPreRestoreActions;InProgress;Completed;PartiallyFailed;Failed type RestorePhase string const ( (...) // RestorePhaseFailedPreRestoreActions means one or more the Pre Restore Actions has failed // and therefore restore will not run. RestorePhaseFailedPreRestoreActions BackupPhase = "FailedPreRestoreActions" (...) ) ``` The `PreBackupAction` interface is as follows: ```go // PreBackupAction provides a hook into the backup process before it begins. type PreBackupAction interface { // Execute the PreBackupAction plugin providing it access to the Backup that // is being executed Execute(backup *api.Backup) error } ``` `PreBackupAction` will be defined in `pkg/plugin/velero/prebackupaction.go`. The `PostBackupAction` interface is as follows: ```go // PostBackupAction provides a hook into the backup process after it completes. type PostBackupAction interface { // Execute the PostBackupAction plugin providing it access to the Backup that // has been completed Execute(backup *api.Backup) error } ``` `PostBackupAction` will be defined in `pkg/plugin/velero/postbackupaction.go`. The `PreRestoreAction` interface is as follows: ```go // PreRestoreAction provides a hook into the restore process before it begins. type PreRestoreAction interface { // Execute the PreRestoreAction plugin providing it access to the Restore that // is being executed Execute(restore *api.Restore) error } ``` `PreRestoreAction` will be defined in

The `PostRestoreAction` interface is as follows: ```go // PostRestoreAction provides a hook into the restore process after it completes. type PostRestoreAction interface { // Execute the PostRestoreAction plugin providing it access to the Restore that // has been completed Execute(restore *api.Restore) error } ``` `PostRestoreAction` will be defined in `pkg/plugin/velero/postrestoreaction.go`. For the persistence of the logs originated from the PostBackup and PostRestore plugins, create two additional methods on `BackupStore` interface: ```go type BackupStore interface { (...) PutPostBackuplog(backup string, log io.Reader) error PutPostRestoreLog(backup, restore string, log io.Reader) error (...) ``` The implementation of these new two methods will go hand-in-hand with the changes of uploading phases rebase. In `pkg/plugin/proto`, add the following: Protobuf definitions will be necessary for PreBackupAction in `pkg/plugin/proto/PreBackupAction.proto`. ```protobuf message PreBackupActionExecuteRequest { ... } service PreBackupAction { rpc Execute(PreBackupActionExecuteRequest) returns (Empty) } ``` Once these are written, then a client and server implementation can be written in `pkg/plugin/framework/prebackupactionclient.go` and `pkg/plugin/framework/prebackupactionserver.go`, respectively. Protobuf definitions will be necessary for PostBackupAction in `pkg/plugin/proto/PostBackupAction.proto`. ```protobuf message PostBackupActionExecuteRequest { ... } service PostBackupAction { rpc Execute(PostBackupActionExecuteRequest) returns (Empty) } ``` Once these are written, then a client and server implementation can be written in `pkg/plugin/framework/postbackupactionclient.go` and `pkg/plugin/framework/postbackupactionserver.go`, respectively. Protobuf definitions will be necessary for PreRestoreAction in `pkg/plugin/proto/PreRestoreAction.proto`. ```protobuf message PreRestoreActionExecuteRequest { ... } service PreRestoreAction { rpc Execute(PreRestoreActionExecuteRequest) returns (Empty) } ``` Once these are written, then a client and server implementation can be written in `pkg/plugin/framework/prerestoreactionclient.go` and `pkg/plugin/framework/prerestoreactionserver.go`, respectively. Protobuf definitions will be necessary for PostRestoreAction in `pkg/plugin/proto/PostRestoreAction.proto`. ```protobuf message PostRestoreActionExecuteRequest { ... } service PostRestoreAction { rpc Execute(PostRestoreActionExecuteRequest) returns (Empty) } ``` Once these are written, then a client and server implementation can be written in `pkg/plugin/framework/postrestoreactionclient.go` and `pkg/plugin/framework/postrestoreactionserver.go`, respectively. Similar to the `RestoreItemAction` and `BackupItemAction` plugins, restartable processes will need to be implemented (with the difference that there is no `AppliedTo()` method). In `pkg/plugin/clientmgmt/`, add `restartableprebackup_action.go`, creating the following unexported type: ```go type restartablePreBackupAction struct { key kindAndName sharedPluginProcess RestartableProcess } func newRestartablePreBackupAction(name string, sharedPluginProcess RestartableProcess) *restartablePreBackupAction { // ... } func (r *restartablePreBackupAction) getPreBackupAction() (velero.PreBackupAction, error) { // ... } func (r *restartablePreBackupAction) getDelegate() (velero.PreBackupAction, error) { // ... } // Execute restarts the plugin's process if needed, then delegates the call. func (r restartablePreBackupAction) Execute(input velero.PreBackupActionInput) (error) { // ... } ``` `restartablepostbackup_action.go`, creating the following unexported type: ```go type restartablePostBackupAction struct { key kindAndName sharedPluginProcess RestartableProcess } func newRestartablePostBackupAction(name string, sharedPluginProcess RestartableProcess) *restartablePostBackupAction { // ... } func (r *restartablePostBackupAction) getPostBackupAction() (velero.PostBackupAction, error) { // ... } func (r *restartablePostBackupAction) getDelegate() (velero.PostBackupAction, error) { // ... } // Execute restarts the plugin's process if needed, then delegates the call. func (r restartablePostBackupAction) Execute(input velero.PostBackupActionInput) (error) { // ... } ``` `restartableprerestore_action.go`, creating the following unexported type: ```go type restartablePreRestoreAction struct { key kindAndName sharedPluginProcess RestartableProcess } func newRestartablePreRestoreAction(name string, sharedPluginProcess RestartableProcess) *restartablePreRestoreAction { // ... } func (r *restartablePreRestoreAction) getPreRestoreAction() (velero.PreRestoreAction, error) { // ... } func (r *restartablePreRestoreAction) getDelegate() (velero.PreRestoreAction, error) { // ... } // Execute restarts the plugin's process if needed, then delegates the call. func (r restartablePreRestoreAction) Execute(input velero.PreRestoreActionInput) (error) { // ... } ``` `restartablepostrestore_action.go`, creating the following unexported type: ```go type restartablePostRestoreAction struct { key kindAndName sharedPluginProcess RestartableProcess } func newRestartablePostRestoreAction(name string, sharedPluginProcess RestartableProcess) *restartablePostRestoreAction { // ... } func (r *restartablePostRestoreAction) getPostRestoreAction() (velero.PostRestoreAction, error) { // ... } func (r *restartablePostRestoreAction) getDelegate() (velero.PostRestoreAction, error) { // ... } // Execute restarts the plugin's process if needed, then delegates the call. func (r restartablePostRestoreAction) Execute(input velero.PostRestoreActionInput) (error) { // ... } ``` Add the following methods to the `Manager` interface in `pkg/plugin/clientmgmt/manager.go`: ```go type Manager interface {

// Get PreBackupAction returns a PreBackupAction plugin for name. GetPreBackupAction(name string) (PreBackupAction, error) // Get PreBackupActions returns the all PreBackupAction plugins. GetPreBackupActions() ([]PreBackupAction, error) // Get PostBackupAction returns a PostBackupAction plugin for name. GetPostBackupAction(name string) (PostBackupAction, error) // GetPostBackupActions returns the all PostBackupAction plugins. GetPostBackupActions() ([]PostBackupAction, error) // Get PreRestoreAction returns a PreRestoreAction plugin for name. GetPreRestoreAction(name string) (PreRestoreAction, error) // Get PreRestoreActions returns the all PreRestoreAction plugins. GetPreRestoreActions() ([]PreRestoreAction, error) // Get PostRestoreAction returns a PostRestoreAction plugin for name. GetPostRestoreAction(name string) (PostRestoreAction, error) // GetPostRestoreActions returns the all PostRestoreAction plugins. GetPostRestoreActions() ([]PostRestoreAction, error) } ``` `GetPreBackupAction` and `GetPreBackupActions` will invoke the `restartablePreBackupAction` implementations. `GetPostBackupAction` and `GetPostBackupActions` will invoke the `restartablePostBackupAction` implementations. `GetPreRestoreAction` and `GetPreRestoreActions` will invoke the `restartablePreRestoreAction` implementations. `GetPostRestoreAction` and `GetPostRestoreActions` will invoke the `restartablePostRestoreAction` implementations. Getting Actions on `backup_controller.go` in `runBackup`: ```go backupLog.Info("Getting PreBackup actions") preBackupActions, err := pluginManager.GetPreBackupActions() if err != nil { return err } backupLog.Info("Getting PostBackup actions") postBackupActions, err := pluginManager.GetPostBackupActions() if err != nil { return err } ``` Calling the Pre Backup actions: ```go for _, preBackupAction := range preBackupActions { err := preBackupAction.Execute(backup.Backup) if err != nil { backup.Backup.Status.Phase = velerov1api.BackupPhaseFailedPreBackupActions return err } } ``` Calling the Post Backup actions: ```go for _, postBackupAction := range postBackupActions { err := postBackupAction.Execute(backup.Backup) if err != nil { postBackupLog.Error(err) } } ``` Getting Actions on `restore_controller.go` in `runValidatedRestore`: ```go restoreLog.Info("Getting PreRestore actions") preRestoreActions, err := pluginManager.GetPreRestoreActions() if err != nil { return errors.Wrap(err, "error getting pre-restore actions") } restoreLog.Info("Getting PostRestore actions") postRestoreActions, err := pluginManager.GetPostRestoreActions() if err != nil { return errors.Wrap(err, "error getting post-restore actions") } ``` Calling the Pre Restore actions: ```go for _, preRestoreAction := range preRestoreActions { err := preRestoreAction.Execute(restoreReq.Restore) if err != nil { restoreReq.Restore.Status.Phase = velerov1api.RestorePhaseFailedPreRestoreActions return errors.Wrap(err, "error executing pre-restore action") } } ``` Calling the Post Restore actions: ```go for _, postRestoreAction := range postRestoreActions { err := postRestoreAction.Execute(restoreReq.Restore) if err != nil { postRestoreLog.Error(err.Error()) } } ``` Velero plugins are loaded as init containers. If plugins are unloaded, they trigger a restart of the Velero controller. Not mentioning if one plugin does get loaded for any reason (i.e., docker hub image pace limit), Velero does not start. In other words, the constant load/unload of plugins can disrupt the Velero controller, and they cannot be the only method to run the actions from these plugins selectively. As part of this proposal, we want to give the velero user the ability to skip the execution of the plugins via annotations on the Velero CR backup and restore objects. If one of these exists, the given plugin, referenced below as `plugin-name`, will be skipped. Backup Object Annotations: ``` <plugin-name>/prebackup=skip <plugin-name>/postbackup=skip ``` Restore Object Annotations: ``` <plugin-name>/prerestore=skip <plugin-name>/postrestore=skip ``` An alternative to these plugin hooks is to implement all the pre/post backup/restore logic outside Velero. In this case, one would need to write an external controller that works similar to what does today when quiescing applications. We find this a viable way, but we think that Velero users can benefit from Velero having greater embedded capabilities, which will allow users to write or load plugins extensions without relying on an external components. The plugins will only be invoked if loaded per a user's discretion. It is recommended to check security vulnerabilities before execution. In terms of backward compatibility, this design should stay compatible with most Velero installations that are upgrading. If plugins are not present, then the backup/restore process should proceed the same way it worked before their inclusion. The implementation dependencies are roughly in the order as they are described in the section.