Chapter 7. Backing up and restoring Red Hat Quay managed by the Red Hat Quay Operator Use the content within this section to back up and restore Red Hat Quay when managed by the Red Hat Quay Operator on OpenShift Container Platform 7.1. Optional: Enabling read-only mode for Red Hat Quay on OpenShift Container Platform Enabling read-only mode for your Red Hat Quay on OpenShift Container Platform deployment allows you to manage the registry's operations. Administrators can enable read-only mode to restrict write access to the registry, which helps ensure data integrity, mitigate risks during maintenance windows, and provide a safeguard against unintended modifications to registry data. It also helps to ensure that your Red Hat Quay registry remains online and available to serve images to users. When backing up and restoring, you are required to scale down your Red Hat Quay on OpenShift Container Platform deployment. This results in service unavailability during the backup period which, in some cases, might be unacceptable. Enabling read-only mode ensures service availability during the backup and restore procedure for Red Hat Quay on OpenShift Container Platform deployments. Note In some cases, a read-only option for Red Hat Quay is not possible since it requires inserting a service key and other manual configuration changes. As an alternative to read-only mode, Red Hat Quay administrators might consider enabling the DISABLE_PUSHES feature. When this field is set to true , users are unable to push images or image tags to the registry when using the CLI. Enabling DISABLE_PUSHES differs from read-only mode because the database is not set as read-only when it is enabled. This field might be useful in some situations such as when Red Hat Quay administrators want to calculate their registry's quota and disable image pushing until after calculation has completed. With this method, administrators can avoid putting putting the whole registry in read-only mode, which affects the database, so that most operations can still be done. For information about enabling this configuration field, see Miscellaneous configuration fields . Prerequisites If you are using Red Hat Enterprise Linux (RHEL) 7.x: You have enabled the Red Hat Software Collections List (RHSCL). You have installed Python 3.6. You have downloaded the virtualenv package. You have installed the git CLI. If you are using Red Hat Enterprise Linux (RHEL) 8: You have installed Python 3 on your machine. You have downloaded the python3-virtualenv package. You have installed the git CLI. You have cloned the https://github.com/quay/quay.git repository. You have installed the oc CLI. You have access to the cluster with cluster-admin privileges. 7.1.1. Creating service keys for Red Hat Quay on OpenShift Container Platform Red Hat Quay uses service keys to communicate with various components. These keys are used to sign completed requests, such as requesting to scan images, login, storage access, and so on. Procedure Enter the following command to obtain a list of Red Hat Quay pods: USD oc get pods -n <namespace> Example output example-registry-clair-app-7dc7ff5844-4skw5 0/1 Error 0 70d example-registry-clair-app-7dc7ff5844-nvn4f 1/1 Running 0 31d example-registry-clair-app-7dc7ff5844-x4smw 0/1 ContainerStatusUnknown 6 (70d ago) 70d example-registry-clair-app-7dc7ff5844-xjnvt 1/1 Running 0 60d example-registry-clair-postgres-547d75759-75c49 1/1 Running 0 70d example-registry-quay-app-76c8f55467-52wjz 1/1 Running 0 70d example-registry-quay-app-76c8f55467-hwz4c 1/1 Running 0 70d example-registry-quay-app-upgrade-57ghs 0/1 Completed 1 70d example-registry-quay-database-7c55899f89-hmnm6 1/1 Running 0 70d example-registry-quay-mirror-6cccbd76d-btsnb 1/1 Running 0 70d example-registry-quay-mirror-6cccbd76d-x8g42 1/1 Running 0 70d example-registry-quay-redis-85cbdf96bf-4vk5m 1/1 Running 0 70d Open a remote shell session to the Quay container by entering the following command: USD oc rsh example-registry-quay-app-76c8f55467-52wjz Enter the following command to create the necessary service keys: sh-4.4USD python3 tools/generatekeypair.py quay-readonly Example output Writing public key to quay-readonly.jwk Writing key ID to quay-readonly.kid Writing private key to quay-readonly.pem 7.1.2. Adding keys to the PostgreSQL database Use the following procedure to add your service keys to the PostgreSQL database. Prerequistes You have created the service keys. Procedure Enter the following command to enter your Red Hat Quay database environment: USD oc rsh example-registry-quay-app-76c8f55467-52wjz psql -U <database_username> -d <database_name> Display the approval types and associated notes of the servicekeyapproval by entering the following command: quay=# select * from servicekeyapproval; Example output id | approver_id | approval_type | approved_date | notes ----+-------------+----------------------------------+----------------------------+------- 1 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 | 2 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 | 3 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095 | 4 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235 | 5 | 1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 | ... Add the service key to your Red Hat Quay database by entering the following query: quay=# INSERT INTO servicekey (name, service, metadata, kid, jwk, created_date, expiration_date) VALUES ('quay-readonly', 'quay', '{}', '{<contents_of_.kid_file>}', '{<contents_of_.jwk_file>}', '{<created_date_of_read-only>}', '{<expiration_date_of_read-only>}'); Example output INSERT 0 1 , add the key approval with the following query: quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes') VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE", {include_notes_here_on_why_this_is_being_added}); Example output INSERT 0 1 Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs: UPDATE servicekey SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER') WHERE name = 'quay-readonly'; UPDATE 1 7.1.3. Configuring read-only mode Red Hat Quay on OpenShift Container Platform After the service keys have been created and added to your PostgreSQL database, you must restart the Quay container on your OpenShift Container Platform deployment. Important Deploying Red Hat Quay on OpenShift Container Platform in read-only mode requires you to modify the secrets stored inside of your OpenShift Container Platform cluster. It is highly recommended that you create a backup of the secret prior to making changes to it. Prerequisites You have created the service keys and added them to your PostgreSQL database. Procedure Enter the following command to read the secret name of your Red Hat Quay on OpenShift Container Platform deployment: USD oc get deployment -o yaml <quay_main_app_deployment_name> Use the base64 command to encode the quay-readonly.kid and quay-readonly.pem files: USD base64 -w0 quay-readonly.kid Example output ZjUyNDFm... USD base64 -w0 quay-readonly.pem Example output LS0tLS1CRUdJTiBSU0E... Obtain the current configuration bundle and secret by entering the following command: USD oc get secret quay-config-secret-name -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > config.yaml Edit the config.yaml file and add the following information: # ... REGISTRY_STATE: readonly INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid' INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem' # ... Save the file and base64 encode it by running the following command: USD base64 -w0 quay-config.yaml Scale down the Red Hat Quay Operator pods to 0 . This ensures that the Operator does not reconcile the secret after editing it. USD oc scale --replicas=0 deployment quay-operator -n openshift-operators Edit the secret to include the new content: USD oc edit secret quay-config-secret-name -n quay-namespace # ... data: "quay-readonly.kid": "ZjUyNDFm..." "quay-readonly.pem": "LS0tLS1CRUdJTiBSU0E..." "config.yaml": "QUNUSU9OX0xPR19..." # ... With your Red Hat Quay on OpenShift Container Platform deployment on read-only mode, you can safely manage your registry's operations and perform such actions as backup and restore. 7.1.3.1. Scaling up the Red Hat Quay on OpenShift Container Platform from a read-only deployment When you no longer want Red Hat Quay on OpenShift Container Platform to be in read-only mode, you can scale the deployment back up and remove the content added from the secret. Procedure Edit the config.yaml file and remove the following information: # ... REGISTRY_STATE: readonly INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid' INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem' # ... Scale the Red Hat Quay Operator back up by entering the following command: oc scale --replicas=1 deployment quay-operator -n openshift-operators 7.2. Backing up Red Hat Quay Database backups should be performed regularly using either the supplied tools on the PostgreSQL image or your own backup infrastructure. The Red Hat Quay Operator does not ensure that the PostgreSQL database is backed up. Note This procedure covers backing up your Red Hat Quay PostgreSQL database. It does not cover backing up the Clair PostgreSQL database. Strictly speaking, backing up the Clair PostgreSQL database is not needed because it can be recreated. If you opt to recreate it from scratch, you will wait for the information to be repopulated after all images inside of your Red Hat Quay deployment are scanned. During this downtime, security reports are unavailable. If you are considering backing up the Clair PostgreSQL database, you must consider that its size is dependent upon the number of images stored inside of Red Hat Quay. As a result, the database can be extremely large. This procedure describes how to create a backup of Red Hat Quay on OpenShift Container Platform using the Operator. Prerequisites A healthy Red Hat Quay deployment on OpenShift Container Platform using the Red Hat Quay Operator. The status condition Available is set to true . The components quay , postgres and objectstorage are set to managed: true If the component clair is set to managed: true the component clairpostgres is also set to managed: true (starting with Red Hat Quay v3.7 or later) Note If your deployment contains partially unmanaged database or storage components and you are using external services for PostgreSQL or S3-compatible object storage to run your Red Hat Quay deployment, you must refer to the service provider or vendor documentation to create a backup of the data. You can refer to the tools described in this guide as a starting point on how to backup your external PostgreSQL database or object storage. 7.2.1. Red Hat Quay configuration backup Use the following procedure to back up your Red Hat Quay configuration. Procedure To back the QuayRegistry custom resource by exporting it, enter the following command: USD oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields: metadata.creationTimestamp metadata.finalizers metadata.generation metadata.resourceVersion metadata.uid Backup the managed keys secret by entering the following command: Note If you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Red Hat Quay for the first time. These are stored in a secret called <quay_registry_name>-quay_registry_managed_secret_keys in the namespace of the QuayRegistry resource. USD oc get secret -n <quay_namespace> <quay_registry_name>_quay_registry_managed_secret_keys -o yaml > managed_secret_keys.yaml Edit the resulting managed_secret_keys.yaml file and remove the entry metadata.ownerReferences . Your managed_secret_keys.yaml file should look similar to the following: apiVersion: v1 kind: Secret type: Opaque metadata: name: <quayname>_quay_registry_managed_secret_keys> namespace: <quay_namespace> data: CONFIG_EDITOR_PW: <redacted> DATABASE_SECRET_KEY: <redacted> DB_ROOT_PW: <redacted> DB_URI: <redacted> SECRET_KEY: <redacted> SECURITY_SCANNER_V4_PSK: <redacted> All information under the data property should remain the same. Redirect the current Quay configuration file by entering the following command: USD oc get secret -n <quay-namespace> USD(oc get quayregistry <quay_registry_name> -n <quay_namespace> -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml Backup the /conf/stack/config.yaml file mounted inside of the Quay pods: USD oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml 7.2.2. Scaling down your Red Hat Quay deployment Use the following procedure to scale down your Red Hat Quay deployment. Important This step is needed to create a consistent backup of the state of your Red Hat Quay deployment. Do not omit this step, including in setups where PostgreSQL databases and/or S3-compatible object storage are provided by external services (unmanaged by the Red Hat Quay Operator). Procedure Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Red Hat Quay, mirror workers, and Clair (if managed). Your QuayRegistry resource should look similar to the following: apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: ... - kind: horizontalpodautoscaler managed: false 1 - kind: quay managed: true overrides: 2 replicas: 0 - kind: clair managed: true overrides: replicas: 0 - kind: mirror managed: true overrides: replicas: 0 ... 1 Disable auto scaling of Quay, Clair and Mirroring workers 2 Set the replica count to 0 for components accessing the database and objectstorage For Operator version 3.6 and earlier : Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay registry first and then the managed Red Hat Quay resources: USD oc scale --replicas=0 deployment USD(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print USD1}') -n <quay-operator-namespace> USD oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print USD1}') -n <quay-namespace> USD oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print USD1}') -n <quay-namespace> USD oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print USD1}') -n <quay-namespace> Wait for the registry-quay-app , registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by the Red Hat Quay Operator) to disappear. You can check their status by running the following command: USD oc get pods -n <quay_namespace> Example output: USD oc get pod Example output quay-operator.v3.7.1-6f9d859bd-p5ftc 1/1 Running 0 12m quayregistry-clair-postgres-7487f5bd86-xnxpr 1/1 Running 1 (12m ago) 12m quayregistry-quay-app-upgrade-xq2v6 0/1 Completed 0 12m quayregistry-quay-database-859d5445ff-cqthr 1/1 Running 0 12m quayregistry-quay-redis-84f888776f-hhgms 1/1 Running 0 12m 7.2.3. Backing up the Red Hat Quay managed database Use the following procedure to back up the Red Hat Quay managed database. Note If your Red Hat Quay deployment is configured with external, or unmanged, PostgreSQL database(s), refer to your vendor's documentation on how to create a consistent backup of these databases. Procedure Identify the Quay PostgreSQL pod name: USD oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}' Example output: quayregistry-quay-database-59f54bb7-58xs7 Obtain the Quay database name: USD oc -n <quay_namespace> rsh USD(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print USD4}' quayregistry-quay-database Download a backup database: USD oc -n <quay_namespace> exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database > backup.sql 7.2.3.1. Backing up the Red Hat Quay managed object storage Use the following procedure to back up the Red Hat Quay managed object storage. The instructions in this section apply to the following configurations: Standalone, multi-cloud object gateway configurations OpenShift Data Foundations storage requires that the Red Hat Quay Operator provisioned an S3 object storage bucket from, through the ObjectStorageBucketClaim API Note If your Red Hat Quay deployment is configured with external (unmanged) object storage, refer to your vendor's documentation on how to create a copy of the content of Quay's storage bucket. Procedure Decode and export the AWS_ACCESS_KEY_ID by entering the following command: USD export AWS_ACCESS_KEY_ID=USD(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d) Decode and export the AWS_SECRET_ACCESS_KEY_ID by entering the following command: USD export AWS_SECRET_ACCESS_KEY=USD(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d) Create a new directory: USD mkdir blobs Note You can also use rclone or sc3md instead of the AWS command line utility. Copy all blobs to the directory by entering the following command: USD aws s3 sync --no-verify-ssl --endpoint https://USD(oc get route s3 -n openshift-storage -o jsonpath='{.spec.host}') s3://USD(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs 7.2.4. Scale the Red Hat Quay deployment back up Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following: apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: ... - kind: horizontalpodautoscaler managed: true 1 - kind: quay 2 managed: true - kind: clair managed: true - kind: mirror managed: true ... 1 Re-enables auto scaling of Quay, Clair and Mirroring workers again (if desired) 2 Replica overrides are removed again to scale the Quay components back up For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay registry: USD oc scale --replicas=1 deployment USD(oc get deployment -n <quay_operator_namespace> | awk '/^quay-operator/ {print USD1}') -n <quay_operator_namespace> Check the status of the Red Hat Quay deployment by entering the following command: USD oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace> Example output: apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: ... name: registry namespace: <quay-namespace> ... spec: ... status: - lastTransitionTime: '2022-06-20T05:31:17Z' lastUpdateTime: '2022-06-20T17:31:13Z' message: All components reporting as healthy reason: HealthChecksPassing status: 'True' type: Available 7.3. Restoring Red Hat Quay Use the following procedures to restore Red Hat Quay when the Red Hat Quay Operator manages the database. It should be performed after a backup of your Red Hat Quay registry has been performed. See Backing up Red Hat Quay for more information. Prerequisites Red Hat Quay is deployed on OpenShift Container Platform using the Red Hat Quay Operator. A backup of the Red Hat Quay configuration managed by the Red Hat Quay Operator has been created following the instructions in the Backing up Red Hat Quay section Your Red Hat Quay database has been backed up. The object storage bucket used by Red Hat Quay has been backed up. The components quay , postgres and objectstorage are set to managed: true If the component clair is set to managed: true , the component clairpostgres is also set to managed: true (starting with Red Hat Quay v3.7 or later) There is no running Red Hat Quay deployment managed by the Red Hat Quay Operator in the target namespace on your OpenShift Container Platform cluster Note If your deployment contains partially unmanaged database or storage components and you are using external services for PostgreSQL or S3-compatible object storage to run your Red Hat Quay deployment, you must refer to the service provider or vendor documentation to restore their data from a backup prior to restore Red Hat Quay 7.3.1. Restoring Red Hat Quay and its configuration from a backup Use the following procedure to restore Red Hat Quay and its configuration files from a backup. Note These instructions assume you have followed the process in the Backing up Red Hat Quay guide and create the backup files with the same names. Procedure Restore the backed up Red Hat Quay configuration by entering the following command: USD oc create -f ./config-bundle.yaml Important If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists , you must delete your existing resource with USD oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with USD oc create -f ./config-bundle.yaml . Restore the generated keys from the backup by entering the following command: USD oc create -f ./managed-secret-keys.yaml Restore the QuayRegistry custom resource: USD oc create -f ./quay-registry.yaml Check the status of the Red Hat Quay deployment and wait for it to be available: USD oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace> 7.3.2. Scaling down your Red Hat Quay deployment Use the following procedure to scale down your Red Hat Quay deployment. Procedure Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). Your QuayRegistry resource should look similar to the following: apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: ... - kind: horizontalpodautoscaler managed: false 1 - kind: quay managed: true overrides: 2 replicas: 0 - kind: clair managed: true overrides: replicas: 0 - kind: mirror managed: true overrides: replicas: 0 ... 1 Disable auto scaling of Quay, Clair and Mirroring workers 2 Set the replica count to 0 for components accessing the database and objectstorage For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay registry first and then the managed Red Hat Quay resources: USD oc scale --replicas=0 deployment USD(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print USD1}') -n <quay-operator-namespace> USD oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print USD1}') -n <quay-namespace> USD oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print USD1}') -n <quay-namespace> USD oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print USD1}') -n <quay-namespace> Wait for the registry-quay-app , registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by Red Hat Quay Operator) to disappear. You can check their status by running the following command: USD oc get pods -n <quay-namespace> Example output: registry-quay-config-editor-77847fc4f5-nsbbv 1/1 Running 0 9m1s registry-quay-database-66969cd859-n2ssm 1/1 Running 0 6d1h registry-quay-redis-7cc5f6c977-956g8 1/1 Running 0 5d21h 7.3.3. Restoring your Red Hat Quay database Use the following procedure to restore your Red Hat Quay database. Procedure Identify your Quay database pod by entering the following command: USD oc get pod -l quay-component=postgres -n <quay-namespace> -o jsonpath='{.items[0].metadata.name}' Example output: Upload the backup by copying it from the local environment and into the pod: Open a remote terminal to the database by entering the following command: USD oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm Enter psql by running the following command: bash-4.4USD psql You can list the database by running the following command: Example output List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ----------------------------+----------------------------+----------+------------+------------+----------------------- postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | quayregistry-quay-database | quayregistry-quay-database | UTF8 | en_US.utf8 | en_US.utf8 | Drop the database by entering the following command: postgres=# DROP DATABASE "quayregistry-quay-database"; Example output DROP DATABASE Exit the postgres CLI to re-enter bash-4.4: \q Redirect your PostgreSQL database to your backup database: sh-4.4USD psql < /tmp/backup.sql Exit bash by entering the following command: sh-4.4USD exit 7.3.4. Restore your Red Hat Quay object storage data Use the following procedure to restore your Red Hat Quay object storage data. Procedure Export the AWS_ACCESS_KEY_ID by entering the following command: USD export AWS_ACCESS_KEY_ID=USD(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d) Export the AWS_SECRET_ACCESS_KEY by entering the following command: USD export AWS_SECRET_ACCESS_KEY=USD(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d) Upload all blobs to the bucket by running the following command: USD aws s3 sync --no-verify-ssl --endpoint https://USD(oc get route s3 -n openshift-storage -o jsonpath='{.spec.host}') ./blobs s3://USD(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') Note You can also use rclone or sc3md instead of the AWS command line utility. 7.3.5. Scaling up your Red Hat Quay deployment Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following: apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: ... - kind: horizontalpodautoscaler managed: true 1 - kind: quay 2 managed: true - kind: clair managed: true - kind: mirror managed: true ... 1 Re-enables auto scaling of Red Hat Quay, Clair and mirroring workers again (if desired) 2 Replica overrides are removed again to scale the Red Hat Quay components back up For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay registry again: USD oc scale --replicas=1 deployment USD(oc get deployment -n <quay-operator-namespace> | awk '/^quay-operator/ {print USD1}') -n <quay-operator-namespace> Check the status of the Red Hat Quay deployment: USD oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace> Example output: apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: ... name: registry namespace: <quay-namespace> ... spec: ... status: - lastTransitionTime: '2022-06-20T05:31:17Z' lastUpdateTime: '2022-06-20T17:31:13Z' message: All components reporting as healthy reason: HealthChecksPassing status: 'True' type: Available

Commands:
oc get pods -n <namespace>
example-registry-clair-app-7dc7ff5844-4skw5 0/1 Error 0 70d example-registry-clair-app-7dc7ff5844-nvn4f 1/1 Running 0 31d example-registry-clair-app-7dc7ff5844-x4smw 0/1 ContainerStatusUnknown 6 (70d ago) 70d example-registry-clair-app-7dc7ff5844-xjnvt 1/1 Running 0 60d example-registry-clair-postgres-547d75759-75c49 1/1 Running 0 70d example-registry-quay-app-76c8f55467-52wjz 1/1 Running 0 70d example-registry-quay-app-76c8f55467-hwz4c 1/1 Running 0 70d example-registry-quay-app-upgrade-57ghs 0/1 Completed 1 70d example-registry-quay-database-7c55899f89-hmnm6 1/1 Running 0 70d example-registry-quay-mirror-6cccbd76d-btsnb 1/1 Running 0 70d example-registry-quay-mirror-6cccbd76d-x8g42 1/1 Running 0 70d example-registry-quay-redis-85cbdf96bf-4vk5m 1/1 Running 0 70d
oc rsh example-registry-quay-app-76c8f55467-52wjz
sh-4.4USD python3 tools/generatekeypair.py quay-readonly
Writing public key to quay-readonly.jwk Writing key ID to quay-readonly.kid Writing private key to quay-readonly.pem
oc rsh example-registry-quay-app-76c8f55467-52wjz psql -U <database_username> -d <database_name>
quay=# select * from servicekeyapproval;
id | approver_id | approval_type | approved_date | notes ----+-------------+----------------------------------+----------------------------+------- 1 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 | 2 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 | 3 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095 | 4 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235 | 5 | 1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
quay=# INSERT INTO servicekey (name, service, metadata, kid, jwk, created_date, expiration_date) VALUES ('quay-readonly', 'quay', '{}', '{<contents_of_.kid_file>}', '{<contents_of_.jwk_file>}', '{<created_date_of_read-only>}', '{<expiration_date_of_read-only>}');
INSERT 0 1
quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes') VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE", {include_notes_here_on_why_this_is_being_added});
INSERT 0 1
UPDATE servicekey SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER') WHERE name = 'quay-readonly';
UPDATE 1
oc get deployment -o yaml <quay_main_app_deployment_name>
base64 -w0 quay-readonly.kid
ZjUyNDFm
base64 -w0 quay-readonly.pem
LS0tLS1CRUdJTiBSU0E
oc get secret quay-config-secret-name -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > config.yaml
REGISTRY_STATE: readonly INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid' INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
base64 -w0 quay-config.yaml
oc scale --replicas=0 deployment quay-operator -n openshift-operators
oc edit secret quay-config-secret-name -n quay-namespace
data: "quay-readonly.kid": "ZjUyNDFm..." "quay-readonly.pem": "LS0tLS1CRUdJTiBSU0E..." "config.yaml": "QUNUSU9OX0xPR19..."
REGISTRY_STATE: readonly INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid' INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
scale --replicas=1 deployment quay-operator -n openshift-operators
oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml
metadata.creationTimestamp metadata.finalizers metadata.generation metadata.resourceVersion metadata.uid
oc get secret -n <quay_namespace> <quay_registry_name>_quay_registry_managed_secret_keys -o yaml > managed_secret_keys.yaml
apiVersion: v1 kind: Secret type: Opaque metadata: name: <quayname>_quay_registry_managed_secret_keys> namespace: <quay_namespace> data: CONFIG_EDITOR_PW: <redacted> DATABASE_SECRET_KEY: <redacted> DB_ROOT_PW: <redacted> DB_URI: <redacted> SECRET_KEY: <redacted> SECURITY_SCANNER_V4_PSK: <redacted>
oc get secret -n <quay-namespace> USD(oc get quayregistry <quay_registry_name> -n <quay_namespace> -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml
oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: ... - kind: horizontalpodautoscaler managed: false 1 - kind: quay managed: true overrides: 2 replicas: 0 - kind: clair managed: true overrides: replicas: 0 - kind: mirror managed: true overrides: replicas: 0 ...
oc scale --replicas=0 deployment USD(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print USD1}') -n <quay-operator-namespace>
oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print USD1}') -n <quay-namespace>
oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print USD1}') -n <quay-namespace>
oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print USD1}') -n <quay-namespace>
oc get pods -n <quay_namespace>
oc get pod
quay-operator.v3.7.1-6f9d859bd-p5ftc 1/1 Running 0 12m quayregistry-clair-postgres-7487f5bd86-xnxpr 1/1 Running 1 (12m ago) 12m quayregistry-quay-app-upgrade-xq2v6 0/1 Completed 0 12m quayregistry-quay-database-859d5445ff-cqthr 1/1 Running 0 12m quayregistry-quay-redis-84f888776f-hhgms 1/1 Running 0 12m
oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}'
quayregistry-quay-database-59f54bb7-58xs7
oc -n <quay_namespace> rsh USD(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print USD4}' quayregistry-quay-database
oc -n <quay_namespace> exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database > backup.sql
export AWS_ACCESS_KEY_ID=USD(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)
export AWS_SECRET_ACCESS_KEY=USD(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)
mkdir blobs
aws s3 sync --no-verify-ssl --endpoint https://USD(oc get route s3 -n openshift-storage -o jsonpath='{.spec.host}') s3://USD(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: ... - kind: horizontalpodautoscaler managed: true 1 - kind: quay 2 managed: true - kind: clair managed: true - kind: mirror managed: true ...
oc scale --replicas=1 deployment USD(oc get deployment -n <quay_operator_namespace> | awk '/^quay-operator/ {print USD1}') -n <quay_operator_namespace>
oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace>
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: <quay-namespace> spec: status: - lastTransitionTime: '2022-06-20T05:31:17Z' lastUpdateTime: '2022-06-20T17:31:13Z' message: All components reporting as healthy reason: HealthChecksPassing status: 'True' type: Available
oc create -f ./config-bundle.yaml
oc create -f ./managed-secret-keys.yaml
oc create -f ./quay-registry.yaml
oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: ... - kind: horizontalpodautoscaler managed: false 1 - kind: quay managed: true overrides: 2 replicas: 0 - kind: clair managed: true overrides: replicas: 0 - kind: mirror managed: true overrides: replicas: 0 ...
oc scale --replicas=0 deployment USD(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print USD1}') -n <quay-operator-namespace>
oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print USD1}') -n <quay-namespace>
oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print USD1}') -n <quay-namespace>
oc scale --replicas=0 deployment USD(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print USD1}') -n <quay-namespace>
oc get pods -n <quay-namespace>
registry-quay-config-editor-77847fc4f5-nsbbv 1/1 Running 0 9m1s registry-quay-database-66969cd859-n2ssm 1/1 Running 0 6d1h registry-quay-redis-7cc5f6c977-956g8 1/1 Running 0 5d21h
oc get pod -l quay-component=postgres -n <quay-namespace> -o jsonpath='{.items[0].metadata.name}'
quayregistry-quay-database-59f54bb7-58xs7
oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql
oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm
bash-4.4USD psql
postgres=# \l
List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ----------------------------+----------------------------+----------+------------+------------+----------------------- postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | quayregistry-quay-database | quayregistry-quay-database | UTF8 | en_US.utf8 | en_US.utf8 |
postgres=# DROP DATABASE "quayregistry-quay-database";
DROP DATABASE
\q
sh-4.4USD psql < /tmp/backup.sql
sh-4.4USD exit
export AWS_ACCESS_KEY_ID=USD(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)
export AWS_SECRET_ACCESS_KEY=USD(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)
aws s3 sync --no-verify-ssl --endpoint https://USD(oc get route s3 -n openshift-storage -o jsonpath='{.spec.host}') ./blobs s3://USD(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: ... - kind: horizontalpodautoscaler managed: true 1 - kind: quay 2 managed: true - kind: clair managed: true - kind: mirror managed: true ...
oc scale --replicas=1 deployment USD(oc get deployment -n <quay-operator-namespace> | awk '/^quay-operator/ {print USD1}') -n <quay-operator-namespace>
oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: <quay-namespace> spec: status: - lastTransitionTime: '2022-06-20T05:31:17Z' lastUpdateTime: '2022-06-20T17:31:13Z' message: All components reporting as healthy reason: HealthChecksPassing status: 'True' type: Available