Chapter 9. 3scale API Management backup and restore Note Red Hat 3scale API Management backup and restore is deprecated and no longer the focus of development. Refer to OpenShift APIs for Data Protection information and instructions. This section provides you, as the administrator of a 3scale installation, the information needed to: Set up the backup procedures for persistent data. Perform a restore from backup of the persistent data. In case of issues with one or more of the MySQL databases, you will be able to restore 3scale correctly to its operational state. 9.1. Prerequisites A 3scale 2.15 instance. For more information about how to install 3scale, see Installing 3scale API Management on OpenShift . An OpenShift Container Platform 4.x user account with one of the following roles in the OpenShift cluster: cluster-admin admin edit Note A user with an edit cluster role locally binded in the namespace of a 3scale installation can perform backup and restore procedures. The following contains information about how to set up the backup procedures for persistent data, perform a restore from backup of the persistent data. In case of a failure with one or more of the MySQL databases, I will then be able to restore 3scale correctly to its operational state. Persistent volumes and considerations Using data sets Backing up system databases Restoring system databases 9.2. Persistent volumes and considerations Persistent volumes In a 3scale API Management deployment on OpenShift : A persistent volume (PV) provided to the cluster by the underlying infrastructure. Storage service external to the cluster. This can be in the same data center or elsewhere. Considerations The backup and restore procedures for persistent data vary depending on the storage type in use. To ensure the backups and restores preserve data consistency, it is not sufficient to backup the underlying PVs for a database. For example, do not capture only partial writes and partial transactions. Use the database's backup mechanisms instead. Some parts of the data are synchronized between different components. One copy is considered the source of truth for the data set. The other is a copy that is not modified locally, but synchronized from the source of truth . In these cases, upon completion, the source of truth should be restored, and copies in other components synchronized from it. 9.3. Using data sets This section explains in more detail about different data sets in the different persistent stores, their purpose, the storage type used, and whether it is the source of truth . The full state of a 3scale deployment is stored across the following Deployment objects and their PVs: Name Description system-mysql MySQL database ( mysql-storage ) system-storage Volume for Files backend-redis Redis database ( backend-redis-storage ) system-redis Redis database ( system-redis-storage ) 9.3.1. Defining system-mysql system-mysql is a relational database which stores information about users, accounts, APIs, plans, and more, in the 3scale Admin Console. A subset of this information related to services is synchronized to the Backend component and stored in backend-redis . system-mysql is the source of truth for this information. 9.3.2. Defining system-storage system-storage stores files to be read and written by the System component. They fall into two categories: Configuration files read by the System component at run-time Static files, for example, HTML, CSS, JS , uploaded to system by its CMS feature, for the purpose of creating a Developer Portal Note System can be scaled horizontally with multiple pods uploading and reading said static files, hence the need for a ReadWriteMany (RWX) PersistentVolume . 9.3.3. Defining backend-redis backend-redis contains multiple data sets used by the Backend component: Usages : This is API usage information aggregated by Backend . It is used by Backend for rate-limiting decisions and by System to display analytics information in the UI or via API. Config : This is configuration information about services, rate-limits, and more, that is synchronized from System via an internal API. This is not the source of truth of this information, however System and system-mysql is. Queues : This is queues of background jobs to be executed by worker processes. These are ephemeral and are deleted once processed. 9.3.4. Defining system-redis system-redis contains queues for jobs to be processed in background. These are ephemeral and are deleted once processed. 9.4. Backing up system databases The following commands are in no specific order and can be used as you need them to back up and archive system databases. 9.4.1. Backing up system-mysql Execute MySQL Backup Command: 9.4.2. Backing up system-storage Archive the system-storage files to another storage: 9.4.3. Backing up backend-redis Backup the dump.rdb file from redis: 9.4.4. Backing up system-redis Backup the dump.rdb file from redis: 9.4.5. Backing up zync-database Backup the zync_production database: 9.4.6. Backing up OpenShift secrets and ConfigMaps The following is the list of commands for OpenShift secrets and ConfigMaps: 9.4.6.1. OpenShift secrets 9.4.6.2. ConfigMaps 9.5. Restoring system databases Important Prevent record creation by scaling down pods like system-app or disabling routes. In the commands and snippets examples that follow, replace USD{DEPLOYMENT_NAME} with the name you defined when you created your 3scale deployment. Note Ensure the output includes at least a pair of braces {} and is not empty. Procedure Store current number of replicas to scale up later: SYSTEM_SPEC=`oc get APIManager/USD{DEPLOYMENT_NAME} -o jsonpath='{.spec.system.appSpec}'` Verify the result of the command and check the content of USDSYSTEM_SPEC : echo USDSYSTEM_SPEC Patch the APIManager CR using the following command that scales the number of replicas to 0 : USD oc patch APIManager/USD{DEPLOYMENT_NAME} --type merge -p '{"spec": {"system": {"appSpec": {"replicas": 0}}}}' Alternatively, to scale down system-app , edit the existing APIManager/USD{DEPLOMENT_NAME} and set the number of system replicas to zero as shown in the following example: apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: <DEPLOYMENT_NAME> spec: system: appSpec: replicas: 0 Use the following procedures to restore OpenShift secrets and system databases: Restoring an operator-based deployment Restoring system-mysql Restoring system-storage Restoring zync-database Ensuring information consistency between backend and system 9.5.1. Restoring an operator-based deployment Use the following steps to restore operator-based deployments. Procedure Install the 3scale API Management operator on OpenShift . Restore secrets before creating an APIManager resource: USD oc apply -f system-smtp.json USD oc apply -f system-seed.json USD oc apply -f system-database.json USD oc apply -f backend-internal-api.json USD oc apply -f system-events-hook.json USD oc apply -f system-app.json USD oc apply -f system-recaptcha.json USD oc apply -f system-redis.json USD oc apply -f zync.json USD oc apply -f system-master-apicast.json Restore ConfigMaps before creating an APIManager resource: USD oc apply -f system-environment.json USD oc apply -f apicast-environment.json Deploy 3scale API Management with the operator using the APIManager CR. 9.5.2. Restoring system-mysql Procedure Copy the MySQL dump to the system-mysql pod: USD oc cp ./system-mysql-backup.gz USD(oc get pods -l 'deployment=system-mysql' -o json | jq '.items[0].metadata.name' -r):/var/lib/mysql Decompress the backup file: USD oc rsh USD(oc get pods -l 'deployment=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d USD{HOME}/system-mysql-backup.gz' Restore the MySQL DB Backup file: USD oc rsh USD(oc get pods -l 'deployment=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=USD{MYSQL_ROOT_PASSWORD}; mysql -hsystem-mysql -uroot system < USD{HOME}/system-mysql-backup' 9.5.3. Restoring system-storage Restore the Backup file to system-storage: USD oc rsync ./local/dir/system/ USD(oc get pods -l 'deployment=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system 9.5.4. Restoring zync-database Instructions to restore zync-database for a 3scale operator deployment. 9.5.4.1. Operator-based deployments Note Follow the instructions under Deploying 3scale API Management using the operator , in particular Deploying the APIManager CR to redeploy your 3scale instance. Procedure Store the number of replicas, by replacing USD{DEPLOYMENT_NAME} with the name you defined when you created your 3scale deployment: Scale down the zync Deployment to 0 pods: USD oc patch APIManager/USD{DEPLOYMENT_NAME} --type merge -p '{"spec": {"zync": {"appSpec": {"replicas": 0}, "queSpec": {"replicas": 0}}}}' Copy the zync database dump to the zync-database pod: USD oc cp ./zync-database-backup.gz USD(oc get pods -l 'deployment=zync-database' -o json | jq '.items[0].metadata.name' -r):/var/lib/pgsql/ Decompress the backup file: USD oc rsh USD(oc get pods -l 'deployment=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d USD{HOME}/zync-database-backup.gz' Restore zync database backup file: USD oc rsh USD(oc get pods -l 'deployment=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'psql zync_production -f USD{HOME}/zync-database-backup' Restore to the original count of replicas: USD oc patch APIManager/USD{DEPLOYMENT_NAME} --type json -p '[{"op": "replace", "path": "/spec/zync", "value":'"USDZYNC_SPEC"'}]' If the output of following command does not contain the replicas key: Then, run the following additional command to scale up zync : 9.5.4.2. Restoring 3scale API Management options with backend-redis and system-redis By restoring 3scale, you will restore backend-redis and system-redis . These components have the following functions: * backend-redis : The database that supports application authentication and rate limiting in 3scale. It is also used for statistics storage and temporary job storage. * system-redis : Provides temporary storage for background jobs for 3scale and is also used as a message bus for Ruby processes of system-app pods. The backend-redis component The backend-redis component has two databases, data and queues . In default 3scale deployment, data and queues are deployed in the Redis database, but in different logical database indexes /0 and /1 . Restoring data database runs without any issues, however restoring queues database can lead to duplicated jobs. Regarding duplication of jobs, in 3scale the backend workers process background jobs in a matter of milliseconds. If backend-redis fails 30 seconds after the last database snapshot and you try to restore it, the background jobs that happened during those 30 seconds are performed twice because backend does not have a system in place to avoid duplication. In this scenario, you must restore the backup as the /0 database index contains data that is not saved anywhere else. Restoring /0 database index means that you must also restore the /1 database index since one cannot be stored without the other. When you choose to separate databases on different servers and not one database in different indexes, the size of the queue will be approximately zero, so it is preferable not to restore backups and lose a few background jobs. This will be the case in a 3scale Hosted setup you will need to therefore apply different backup and restore strategies for both. The `system-redis`component The majority of the 3scale system background jobs are idempotent, that is, identical requests return an identical result no matter how many times you run them. The following is a list of examples of events handled by background jobs in system: Notification jobs such as plan trials about to expire, credit cards about to expire, activation reminders, plan changes, invoice state changes, PDF reports. Billing such as invoicing and charging. Deletion of complex objects. Backend synchronization jobs. Indexation jobs, for example with searchd. Sanitisation jobs, for example invoice IDs. Janitorial tasks such as purging audits, user sessions, expired tokens, log entries, suspending inactive accounts. Traffic updates. Proxy configuration change monitoring and proxy deployments. Background signup jobs, Zync jobs such as single sign-on (SSO) synchronization, routes creation. If you are restoring the above list of background jobs, 3scale's system maintains the state of each restored job. It is important to check the integrity of the system after the restoration is complete. 9.5.5. Ensuring information consistency between backend and system After restoring backend-redis a sync of the Config information from system should be forced to ensure the information in backend is consistent with that in system , which is the source of truth . 9.5.5.1. Managing the deployment configuration for backend-redis These steps are intended for running instances of backend-redis . Procedure Edit the redis-config configmap: USD oc edit configmap redis-config Comment SAVE commands in the redis-config configmap: Set appendonly to no in the redis-config configmap: Redeploy backend-redis to load the new configurations: USD oc rollout restart deployment/backend-redis Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/backend-redis Rename the dump.rdb file: USD oc rsh USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv USD{HOME}/data/dump.rdb USD{HOME}/data/dump.rdb-old' Rename the appendonly.aof file: USD oc rsh USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv USD{HOME}/data/appendonly.aof USD{HOME}/data/appendonly.aof-old' Move the backup file to the POD: USD oc cp ./backend-redis-dump.rdb USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb Redeploy backend-redis to load the backup: USD oc rollout restart deployment/backend-redis Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/backend-redis Create the appendonly file: USD oc rsh USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF' After a while, ensure that the AOF rewrite is complete: USD oc rsh USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progress While aof_rewrite_in_progress = 1 , the execution is in progress. Check periodically until aof_rewrite_in_progress = 0 . Zero indicates that the execution is complete. Edit the redis-config configmap: USD oc edit configmap redis-config Uncomment SAVE commands in the redis-config configmap: Set appendonly to yes in the redis-config configmap: Redeploy backend-redis to reload the default configurations: USD oc rollout restart deployment/backend-redis Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/backend-redis 9.5.5.2. Managing the deployment configuration for system-redis These steps are intended for running instances of system-redis . Procedure Edit the redis-config configmap: USD oc edit configmap redis-config Comment SAVE commands in the redis-config configmap: Set appendonly to no in the redis-config configmap: Redeploy system-redis to load the new configurations: USD oc rollout restart deployment/system-redis Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/system-redis Rename the dump.rdb file: USD oc rsh USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv USD{HOME}/data/dump.rdb USD{HOME}/data/dump.rdb-old' Rename the appendonly.aof file: USD oc rsh USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv USD{HOME}/data/appendonly.aof USD{HOME}/data/appendonly.aof-old' Move the Backup file to the POD: USD oc cp ./system-redis-dump.rdb USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb Redeploy system-redis to load the backup: USD oc rollout restart deployment/system-redis Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/system-redis Create the appendonly file: USD oc rsh USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF' After a while, ensure that the AOF rewrite is complete: USD oc rsh USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progress While aof_rewrite_in_progress = 1 , the execution is in progress. Check periodically until aof_rewrite_in_progress = 0 . Zero indicates that the execution is complete. Edit the redis-config configmap: USD oc edit configmap redis-config Uncomment SAVE commands in the redis-config configmap: Set appendonly to yes in the redis-config configmap: Redeploy system-redis to reload the default configurations: USD oc rollout restart deployment/system-redis Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/system-redis 9.5.6. Restoring backend-worker These steps are intended to restore backend-worker . Procedure Restore to the latest version of backend-worker : USD oc rollout restart deployment/backend-worker Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/backend-worker 9.5.7. Restoring system-app These steps are intended to restore system-app . Procedure To scale up system-app , edit the existing APIManager/USD{DEPLOYMENT_NAME} and change .spec.system.appSpec.replicas back to original number of replicas or run the following command to apply previously stored specification: USD oc patch APIManager/USD{DEPLOYMENT_NAME} --type json -p '[{"op": "replace", "path": "/spec/system/appSpec", "value":'"USDSYSTEM_SPEC"'}]' If the output of following command does not contain the replicas key: Then, run the following additional command to scale up system-app : Restore to the latest version of system-app : USD oc rollout restart deployment/system-app Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/system-app 9.5.8. Restoring system-sidekiq These steps are intended to restore system-sidekiq . Procedure Restore to the latest version of system-sidekiq : USD oc rollout restart deployment/system-sidekiq Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/system-sidekiq 9.5.8.1. Restoring system-searchd These steps are intended to restore system-searchd . Procedure Restore to the latest version of system-searchd : USD oc rollout restart deployment/system-searchd Check the status of the rollout to ensure it has finished: USD oc rollout status deployment/system-searchd 9.5.8.2. Restoring OpenShift routes managed by zync Force zync to recreate missing OpenShift routes: USD oc rsh USD(oc get pods -l 'deployment=system-sidekiq' -o json | jq '.items[0].metadata.name' -r) bash -c 'bundle exec rake zync:resync:domains'

Commands:
oc rsh USD(oc get pods -l 'deployment=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=USD{MYSQL_ROOT_PASSWORD}; mysqldump --single-transaction -hsystem-mysql -uroot system' | gzip > system-mysql-backup.gz
oc rsync USD(oc get pods -l 'deployment=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system ./local/dir
oc cp USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./backend-redis-dump.rdb
oc cp USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./system-redis-dump.rdb
oc rsh USD(oc get pods -l 'deployment=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'pg_dump zync_production' | gzip > zync-database-backup.gz
oc get secrets system-smtp -o json > system-smtp.json oc get secrets system-seed -o json > system-seed.json oc get secrets system-database -o json > system-database.json oc get secrets backend-internal-api -o json > backend-internal-api.json oc get secrets system-events-hook -o json > system-events-hook.json oc get secrets system-app -o json > system-app.json oc get secrets system-recaptcha -o json > system-recaptcha.json oc get secrets system-redis -o json > system-redis.json oc get secrets zync -o json > zync.json oc get secrets system-master-apicast -o json > system-master-apicast.json
oc get configmaps system-environment -o json > system-environment.json oc get configmaps apicast-environment -o json > apicast-environment.json
SYSTEM_SPEC=`oc get APIManager/USD{DEPLOYMENT_NAME} -o jsonpath='{.spec.system.appSpec}'`
echo USDSYSTEM_SPEC
oc patch APIManager/USD{DEPLOYMENT_NAME} --type merge -p '{"spec": {"system": {"appSpec": {"replicas": 0}}}}'
apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: name: <DEPLOYMENT_NAME> spec: system: appSpec: replicas: 0
oc apply -f system-smtp.json oc apply -f system-seed.json oc apply -f system-database.json oc apply -f backend-internal-api.json oc apply -f system-events-hook.json oc apply -f system-app.json oc apply -f system-recaptcha.json oc apply -f system-redis.json oc apply -f zync.json oc apply -f system-master-apicast.json
oc apply -f system-environment.json oc apply -f apicast-environment.json
oc cp ./system-mysql-backup.gz USD(oc get pods -l 'deployment=system-mysql' -o json | jq '.items[0].metadata.name' -r):/var/lib/mysql
oc rsh USD(oc get pods -l 'deployment=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d USD{HOME}/system-mysql-backup.gz'
oc rsh USD(oc get pods -l 'deployment=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=USD{MYSQL_ROOT_PASSWORD}; mysql -hsystem-mysql -uroot system < USD{HOME}/system-mysql-backup'
oc rsync ./local/dir/system/ USD(oc get pods -l 'deployment=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system
ZYNC_SPEC=`oc get APIManager/USD{DEPLOYMENT_NAME} -o json | jq -r '.spec.zync'`
oc patch APIManager/USD{DEPLOYMENT_NAME} --type merge -p '{"spec": {"zync": {"appSpec": {"replicas": 0}, "queSpec": {"replicas": 0}}}}'
oc cp ./zync-database-backup.gz USD(oc get pods -l 'deployment=zync-database' -o json | jq '.items[0].metadata.name' -r):/var/lib/pgsql/
oc rsh USD(oc get pods -l 'deployment=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d USD{HOME}/zync-database-backup.gz'
oc rsh USD(oc get pods -l 'deployment=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'psql zync_production -f USD{HOME}/zync-database-backup'
oc patch APIManager/USD{DEPLOYMENT_NAME} --type json -p '[{"op": "replace", "path": "/spec/zync", "value":'"USDZYNC_SPEC"'}]'
echo USDZYNC_SPEC
oc patch deployment/zync -p '{"spec": {"replicas": 1}}'
oc edit configmap redis-config
#save 900 1 #save 300 10 #save 60 10000
appendonly no
oc rollout restart deployment/backend-redis
oc rollout status deployment/backend-redis
oc rsh USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv USD{HOME}/data/dump.rdb USD{HOME}/data/dump.rdb-old'
oc rsh USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv USD{HOME}/data/appendonly.aof USD{HOME}/data/appendonly.aof-old'
oc cp ./backend-redis-dump.rdb USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb
oc rollout restart deployment/backend-redis
oc rollout status deployment/backend-redis
oc rsh USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF'
oc rsh USD(oc get pods -l 'deployment=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progress
oc edit configmap redis-config
save 900 1 save 300 10 save 60 10000
appendonly yes
oc rollout restart deployment/backend-redis
oc rollout status deployment/backend-redis
oc edit configmap redis-config
#save 900 1 #save 300 10 #save 60 10000
appendonly no
oc rollout restart deployment/system-redis
oc rollout status deployment/system-redis
oc rsh USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv USD{HOME}/data/dump.rdb USD{HOME}/data/dump.rdb-old'
oc rsh USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv USD{HOME}/data/appendonly.aof USD{HOME}/data/appendonly.aof-old'
oc cp ./system-redis-dump.rdb USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb
oc rollout restart deployment/system-redis
oc rollout status deployment/system-redis
oc rsh USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF'
oc rsh USD(oc get pods -l 'deployment=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progress
oc edit configmap redis-config
save 900 1 save 300 10 save 60 10000
appendonly yes
oc rollout restart deployment/system-redis
oc rollout status deployment/system-redis
oc rollout restart deployment/backend-worker
oc rollout status deployment/backend-worker
oc patch APIManager/USD{DEPLOYMENT_NAME} --type json -p '[{"op": "replace", "path": "/spec/system/appSpec", "value":'"USDSYSTEM_SPEC"'}]'
echo USDSYSTEM_SPEC
oc patch deployment/system-app -p '{"spec": {"replicas": 1}}'
oc rollout restart deployment/system-app
oc rollout status deployment/system-app
oc rollout restart deployment/system-sidekiq
oc rollout status deployment/system-sidekiq
oc rollout restart deployment/system-searchd
oc rollout status deployment/system-searchd
oc rsh USD(oc get pods -l 'deployment=system-sidekiq' -o json | jq '.items[0].metadata.name' -r) bash -c 'bundle exec rake zync:resync:domains'