Backing Up and Restoring Runtime Fabric | MuleSoft Documentation
# Backing Up and Restoring Runtime Fabric
Use the following procedures to back up and restore Runtime Fabric services and the required state necessary for all deployed applications and their configurations.
Data about the state of Runtime Fabric is distributed across the Kubernetes cluster. To preserve applications in the event of system failure, confirm that Runtime Fabric is automatically backed up. When scheduling backups, follow best practices by configuring hourly backups. Store your backup on external storage outside the Runtime Fabric cluster.
Starting with `rtfctl` version 1.0.70, if you use an rtf-agent version higher than 2.6.52, the `rtfctl` backup bundle does not contain the Mule app deployments. The deployments are synched from the Runtime Manager UI at the time of restoration.
## When to Use the Back Up and Restore Process
Common scenarios for using the backup and restore process include:
  * Performing a major infrastructure upgrade, such as migrating nodes in a Runtime Fabric cluster to a new version of an operating system.
  * Migrating your Runtime Fabric cluster from on-premises to the cloud or migrating from one cloud provider to another.
  * Performing a failover to a standby cluster.


Do not use the backup and restore process to create concurrent running instances of Runtime Fabric. In other words, do not back up Runtime Fabric A, use that back up to create a new Runtime Fabric B while Runtime Fabric A is still active, and attempt to run them concurrently. This creates application deployment issues.  
---  
## What Gets Backed Up
The Runtime Fabric backup process affects only the Runtime Fabric cluster. The backup process does not back up data from the control plane.
For regular Kubernetes clusters, the following is backed up:
  * Runtime Fabric cluster details, such as controller and worker node configurations
  * Cluster-level secure properties
  * All data in the Runtime Fabric and application namespace, such as data about deployments, services and service accounts, secrets, ingress resources, role bindings, cluster role configurations, and ConfigMap settings
  * The current `rtfctl` version
  * The current Runtime Fabric agent version
  * Application details including:
    * Application configurations for pods, replicasets, application metadata, and templates
    * Application properties configured in Runtime Manager
  * Data from other Mulesoft configurations, including:
    * Object Store
    * Anypoint Monitoring and external logforwarders


For OpenShift clusters, the following is backed up:
  * Custom Resources
  * Application namespaces and for each one:
    * Secrets
    * Service Account
    * ConfigMap
    * Services
    * CronJobs
    * DaemonSets
    * Deployments
    * Ingress
    * Role Bindings
  * Ingress from Runtime Fabric namespace


## What Does Not Get Backed Up
  * Data about application state
  * Data from Ops Center
  * Data from the control plane


### Changes Made After Performing a Backup Are Not Restored
Any changes you make after you perform a backup will not be present if you restore the cluster from that backup. For example, if you perform a backup, delete an application, and then restore from that backup, the application you deleted will be recreated in the cluster, but it will be marked as deleted in Runtime Manager. As a result the application’s status will show as **applying** in Runtime Manager.
Runtime Fabric does not restore the following control plane changes if you make them after you take a backup:
  * Created apps
  * Delete apps
  * Updated app configurations
  * Agent upgrades


You need to manually resolve such inconsistencies by reapplying the changes you made after performing the backup, as these changes can’t be reapplied from the control plane.
## Create a Backup
Before creating a backup:
  * Verify that Runtime Fabric is installed and running successfully.
  * Verify that `rtfctl` command-line tool is upgraded to version 0.3.102 or later. Verify the version by running `rtfctl version` on any Runtime Fabric node.
  * Confirm that the `rtfctl` binary is present in the current directory and `kubectl` is present in the user `$PATH`. Additionally, confirm that you’ve set the Kubernetes (`kubectl`) context to the intended backup cluster.


Creating a backup per node, such as via a VM snapshot, is not supported.  
---  
To create a backup, run the following command:
```
./rtfctl backup <path_to_backup_file>
```

This command creates a backup of the current system state in `<path_to_backup_file>`, which can be any path in the file system that you have write access to, such as: `/opt/anypoint/runtimefabric/backup.tar.gz`.
## Perform a Restore
Before you restore a cluster, review the following information:
  * You must perform the restore on the same Runtime Fabric cluster that is listed in Runtime Manager (the same cluster in the control plane).
  * Configuration changes you make to deployed applications and management services after a back up are not restored.
  * Application monitoring metrics are not restored.
  * When restoring on an existing Runtime Fabric cluster, use the same version of the `rftctl` command-line utility that you used to create the backup.

  
---  
Runtime Fabric provides two target options when restoring a cluster from a backup:
  * Use an existing Runtime Fabric cluster.
  * Create a new Kubernetes cluster with the same configuration as the backed-up cluster. This includes the same number of servers, disks, etc.


To restore a cluster on Runtime Fabric, create a cluster **without** running the `rtfctl` install command. After you create the cluster, complete the restore process.
  1. Choose a target option for restoring a cluster.
  2. Ensure you have installed `rftctl` in the cluster.
  3. Copy the backup file you previously created, and make sure it is available to `rtfctl`.
  4. Confirm your Kubernetes (kubectl) context is set to the backed-up cluster, and scale down all Runtime Fabric components on the original backed-up cluster:
```
kubectl scale --replicas=0 -n rtf deployment.apps/agent
```

  5. Restore the cluster from the backup:


```
./rtfctl restore <path_to_backup_file>
```

This process may require several minutes to complete.
For Runtime Fabric, confirm the `rtfctl` binary is present in the current directory and the Kubernetes (kubectl) context is set to the cluster you are restoring to.
## Create a Backup and Restore for OpenShift Clusters
In the following example, you have a Runtime Fabric with a set of Mule applications and configurations deployed on cluster (named `cluster-1`) inside an OpenShift cluster. To create a backup and perform a restore, follow these steps:
  1. Run the backup command:
```
./rtfctl backup <path_to_backup_file> -n <rtf_namespace>
```

MuleSoft performs the back up based on the detected cluster you have. Though, you can also manually specify the type of back up to perform using the flag `--backup-type` string. The flag options are `auto`, `openshift`, and `full`. With the default `auto` flag option, MuleSoft automatically detects the infrastructure cluster provider to perform the back up.   
---  
  2. Shutdown `cluster-1`.
  3. Delete all Mule application deployments by running the following command:
```
kubectl delete ns <app_namespace>
```

Do not remove applications from the Runtime Manager UI.
  4. Remove the Runtime Fabric installation.
  5. Remove the Runtime Fabric operator from the OpenShift cluster (optional).
  6. Send a `curl` command to the management plane API endpoint to regenerate the activation ID. The Runtime Fabric shows as ready for activation. You can install and activate Runtime Fabric again on a new OpenShift cluster.
```
curl --location --request PATCH 'https://anypoint.mulesoft.com/runtimefabric/api/organizations/{org-id}/fabrics/{fabric-id}/regenerateActivationData' \
--header 'Authorization: Bearer {bearer-token}
```

  7. Install the Runtime Fabric operator on `cluster-2` using the regenerated activation data ID. The installation must be on the same Runtime Fabric namespace. All labels on the backup point to that installation.
  8. Run the restore command to pick the set of resources and restore them in the cluster:
```
./rtfctl restore <path_to_backup_file> -n <rtf_namespace>
```

  9. Check the status of your Runtime Fabric deployments.


## See Also
  * Hardening Runtime Fabric