Chapter 17. Backing up and restoring Red Hat Quay on a standalone deployment Use the content within this section to back up and restore Red Hat Quay in standalone deployments. 17.1. Optional: Enabling read-only mode for Red Hat Quay Enabling read-only mode for your Red Hat Quay deployment allows you to manage the registry's operations. Red Hat Quay administrators can enable read-only mode to restrict write access to the registry, which helps ensure data integrity, mitigate risks during maintenance windows, and provide a safeguard against unintended modifications to registry data. It also helps to ensure that your Red Hat Quay registry remains online and available to serve images to users. Prerequisites If you are using Red Hat Enterprise Linux (RHEL) 7.x: You have enabled the Red Hat Software Collections List (RHSCL). You have installed Python 3.6. You have downloaded the virtualenv package. You have installed the git CLI. If you are using Red Hat Enterprise Linux (RHEL) 8: You have installed Python 3 on your machine. You have downloaded the python3-virtualenv package. You have installed the git CLI. You have cloned the https://github.com/quay/quay.git repository. 17.1.1. Creating service keys for standalone Red Hat Quay Red Hat Quay uses service keys to communicate with various components. These keys are used to sign completed requests, such as requesting to scan images, login, storage access, and so on. Procedure If your Red Hat Quay registry is readily available, you can generate service keys inside of the Quay registry container. Enter the following command to generate a key pair inside of the Quay container: USD podman exec quay python3 tools/generatekeypair.py quay-readonly If your Red Hat Quay is not readily available, you must generate your service keys inside of a virtual environment. Change into the directory of your Red Hat Quay deployment and create a virtual environment inside of that directory: USD cd <USDQUAY>/quay && virtualenv -v venv Activate the virtual environment by entering the following command: USD source venv/bin/activate Optional. Install the pip CLI tool if you do not have it installed: USD venv/bin/pip install --upgrade pip In your Red Hat Quay directory, create a requirements-generatekeys.txt file with the following content: USD cat << EOF > requirements-generatekeys.txt cryptography==3.4.7 pycparser==2.19 pycryptodome==3.9.4 pycryptodomex==3.9.4 pyjwkest==1.4.2 PyJWT==1.7.1 Authlib==1.0.0a2 EOF Enter the following command to install the Python dependencies defined in the requirements-generatekeys.txt file: USD venv/bin/pip install -r requirements-generatekeys.txt Enter the following command to create the necessary service keys: USD PYTHONPATH=. venv/bin/python /<path_to_cloned_repo>/tools/generatekeypair.py quay-readonly Example output Writing public key to quay-readonly.jwk Writing key ID to quay-readonly.kid Writing private key to quay-readonly.pem Enter the following command to deactivate the virtual environment: USD deactivate 17.1.2. Adding keys to the PostgreSQL database Use the following procedure to add your service keys to the PostgreSQL database. Prerequistes You have created the service keys. Procedure Enter the following command to enter your Red Hat Quay database environment: USD podman exec -it postgresql-quay psql -U postgres -d quay Display the approval types and associated notes of the servicekeyapproval by entering the following command: quay=# select * from servicekeyapproval; Example output id | approver_id | approval_type | approved_date | notes ----+-------------+----------------------------------+----------------------------+------- 1 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 | 2 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 | 3 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095 | 4 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235 | 5 | 1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 | ... Add the service key to your Red Hat Quay database by entering the following query: quay=# INSERT INTO servicekey (name, service, metadata, kid, jwk, created_date, expiration_date) VALUES ('quay-readonly', 'quay', '{}', '{<contents_of_.kid_file>}', '{<contents_of_.jwk_file>}', '{<created_date_of_read-only>}', '{<expiration_date_of_read-only>}'); Example output INSERT 0 1 , add the key approval with the following query: quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes') VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE", {include_notes_here_on_why_this_is_being_added}); Example output INSERT 0 1 Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs: UPDATE servicekey SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER') WHERE name = 'quay-readonly'; UPDATE 1 17.1.3. Configuring read-only mode for standalone Red Hat Quay After the service keys have been created and added to your PostgreSQL database, you must restart the Quay container on your standalone deployment. Prerequisites You have created the service keys and added them to your PostgreSQL database. Procedure Shutdown all Red Hat Quay instances on all virtual machines. For example: USD podman stop <quay_container_name_on_virtual_machine_a> USD podman stop <quay_container_name_on_virtual_machine_b> Enter the following command to copy the contents of the quay-readonly.kid file and the quay-readonly.pem file to the directory that holds your Red Hat Quay configuration bundle: USD cp quay-readonly.kid quay-readonly.pem USDQuay/config Enter the following command to set file permissions on all files in your configuration bundle folder: USD setfacl -m user:1001:rw USDQuay/config/* Modify your Red Hat Quay config.yaml file and add the following information: # ... REGISTRY_STATE: readonly INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid' INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem' # ... Distribute the new configuration bundle to all Red Hat Quay instances. Start Red Hat Quay by entering the following command: USD podman run -d --rm -p 80:8080 -p 443:8443 \ --name=quay-main-app \ -v USDQUAY/config:/conf/stack:Z \ -v USDQUAY/storage:/datastorage:Z \ {productrepo}/{quayimage}:{productminv} After starting Red Hat Quay, a banner inside in your instance informs users that Red Hat Quay is running in read-only mode. Pushes should be rejected and a 405 error should be logged. You can test this by running the following command: USD podman push <quay-server.example.com>/quayadmin/busybox:test Example output 613be09ab3c0: Preparing denied: System is currently read-only. Pulls will succeed but all write operations are currently suspended. With your Red Hat Quay deployment on read-only mode, you can safely manage your registry's operations and perform such actions as backup and restore. Optional. After you are finished with read-only mode, you can return to normal operations by removing the following information from your config.yaml file. Then, restart your Red Hat Quay deployment: # ... REGISTRY_STATE: readonly INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid' INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem' # ... USD podman restart <container_id> 17.1.4. Updating read-only expiration time The Red Hat Quay read-only key has an expiration date, and when that date passes the key is deactivated. Before the key expires, its expiration time can be updated in the database. To update the key, connect your Red Hat Quay production database using the methods described earlier and issue the following query: quay=# UPDATE servicekey SET expiration_date = 'new-date' WHERE id = servicekey_id; The list of service key IDs can be obtained by running the following query: SELECT id, name, expiration_date FROM servicekey; 17.2. Backing up Red Hat Quay on standalone deployments This procedure describes how to create a backup of Red Hat Quay on standalone deployments. Procedure Create a temporary backup directory, for example, quay-backup : USD mkdir /tmp/quay-backup The following example command denotes the local directory that the Red Hat Quay was started in, for example, /opt/quay-install : Change into the directory that bind-mounts to /conf/stack inside of the container, for example, /opt/quay-install , by running the following command: USD cd /opt/quay-install Compress the contents of your Red Hat Quay deployment into an archive in the quay-backup directory by entering the following command: USD tar cvf /tmp/quay-backup/quay-backup.tar.gz * Example output: config.yaml config.yaml.bak extra_ca_certs/ extra_ca_certs/ca.crt ssl.cert ssl.key Back up the Quay container service by entering the following command: Redirect the contents of your conf/stack/config.yaml file to your temporary quay-config.yaml file by entering the following command: USD podman exec -it quay cat /conf/stack/config.yaml > /tmp/quay-backup/quay-config.yaml Obtain the DB_URI located in your temporary quay-config.yaml by entering the following command: USD grep DB_URI /tmp/quay-backup/quay-config.yaml Example output: Extract the PostgreSQL contents to your temporary backup directory in a backup .sql file by entering the following command: USD pg_dump -h 172.24.10.50 -p 5432 -d quay -U <username> -W -O > /tmp/quay-backup/quay-backup.sql Print the contents of your DISTRIBUTED_STORAGE_CONFIG by entering the following command: DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - s3_bucket: <bucket_name> storage_path: /registry s3_access_key: <s3_access_key> s3_secret_key: <s3_secret_key> host: <host_name> s3_region: <region> Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 7: USD export AWS_ACCESS_KEY_ID=<access_key> Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 7: USD export AWS_SECRET_ACCESS_KEY=<secret_key> Sync the quay bucket to the /tmp/quay-backup/blob-backup/ directory from the hostname of your DISTRIBUTED_STORAGE_CONFIG : USD aws s3 sync s3://<bucket_name> /tmp/quay-backup/blob-backup/ --source-region us-east-2 Example output: It is recommended that you delete the quay-config.yaml file after syncing the quay bucket because it contains sensitive information. The quay-config.yaml file will not be lost because it is backed up in the quay-backup.tar.gz file. 17.3. Restoring Red Hat Quay on standalone deployments This procedure describes how to restore Red Hat Quay on standalone deployments. Prerequisites You have backed up your Red Hat Quay deployment. Procedure Create a new directory that will bind-mount to /conf/stack inside of the Red Hat Quay container: USD mkdir /opt/new-quay-install Copy the contents of your temporary backup directory created in Backing up Red Hat Quay on standalone deployments to the new-quay-install1 directory created in Step 1: USD cp /tmp/quay-backup/quay-backup.tar.gz /opt/new-quay-install/ Change into the new-quay-install directory by entering the following command: USD cd /opt/new-quay-install/ Extract the contents of your Red Hat Quay directory: USD tar xvf /tmp/quay-backup/quay-backup.tar.gz * Example output: Recall the DB_URI from your backed-up config.yaml file by entering the following command: USD grep DB_URI config.yaml Example output: postgresql://<username>:test123@172.24.10.50/quay Run the following command to enter the PostgreSQL database server: USD sudo postgres Enter psql and create a new database in 172.24.10.50 to restore the quay databases, for example, example_restore_registry_quay_database , by entering the following command: USD psql "host=172.24.10.50 port=5432 dbname=postgres user=<username> password=test123" postgres=> CREATE DATABASE example_restore_registry_quay_database; Example output: Connect to the database by running the following command: postgres=# \c "example-restore-registry-quay-database"; Example output: You are now connected to database "example-restore-registry-quay-database" as user "postgres". Create a pg_trmg extension of your Quay database by running the following command: example_restore_registry_quay_database=> CREATE EXTENSION IF NOT EXISTS pg_trgm; Example output: CREATE EXTENSION Exit the postgres CLI by entering the following command: \q Import the database backup to your new database by running the following command: USD psql "host=172.24.10.50 port=5432 dbname=example_restore_registry_quay_database user=<username> password=test123" -W < /tmp/quay-backup/quay-backup.sql Example output: Update the value of DB_URI in your config.yaml from postgresql://<username>:test123@172.24.10.50/quay to postgresql://<username>:test123@172.24.10.50/example-restore-registry-quay-database before restarting the Red Hat Quay deployment. Note The DB_URI format is DB_URI postgresql://<login_user_name>:<login_user_password>@<postgresql_host>/<quay_database> . If you are moving from one PostgreSQL server to another PostgreSQL server, update the value of <login_user_name> , <login_user_password> and <postgresql_host> at the same time. In the /opt/new-quay-install directory, print the contents of your DISTRIBUTED_STORAGE_CONFIG bundle: USD cat config.yaml | grep DISTRIBUTED_STORAGE_CONFIG -A10 Example output: DISTRIBUTED_STORAGE_CONFIG: default: DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - s3_bucket: <bucket_name> storage_path: /registry s3_access_key: <s3_access_key> s3_region: <region> s3_secret_key: <s3_secret_key> host: <host_name> Note Your DISTRIBUTED_STORAGE_CONFIG in /opt/new-quay-install must be updated before restarting your Red Hat Quay deployment. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 13: USD export AWS_ACCESS_KEY_ID=<access_key> Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 13: USD export AWS_SECRET_ACCESS_KEY=<secret_key> Create a new s3 bucket by entering the following command: USD aws s3 mb s3://<new_bucket_name> --region us-east-2 Example output: USD make_bucket: quay Upload all blobs to the new s3 bucket by entering the following command: USD aws s3 sync --no-verify-ssl \ --endpoint-url <example_endpoint_url> 1 /tmp/quay-backup/blob-backup/. s3://quay/ 1 The Red Hat Quay registry endpoint must be the same before backup and after restore. Example output: upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d to s3://quay/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 to s3://quay/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec to s3://quay/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec ... Before restarting your Red Hat Quay deployment, update the storage settings in your config.yaml: DISTRIBUTED_STORAGE_CONFIG: default: DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - s3_bucket: <new_bucket_name> storage_path: /registry s3_access_key: <s3_access_key> s3_secret_key: <s3_secret_key> s3_region: <region> host: <host_name>

Commands:
podman exec quay python3 tools/generatekeypair.py quay-readonly
cd <USDQUAY>/quay && virtualenv -v venv
source venv/bin/activate
venv/bin/pip install --upgrade pip
cat << EOF > requirements-generatekeys.txt cryptography==3.4.7 pycparser==2.19 pycryptodome==3.9.4 pycryptodomex==3.9.4 pyjwkest==1.4.2 PyJWT==1.7.1 Authlib==1.0.0a2 EOF
venv/bin/pip install -r requirements-generatekeys.txt
PYTHONPATH=. venv/bin/python /<path_to_cloned_repo>/tools/generatekeypair.py quay-readonly
Writing public key to quay-readonly.jwk Writing key ID to quay-readonly.kid Writing private key to quay-readonly.pem
deactivate
podman exec -it postgresql-quay psql -U postgres -d quay
quay=# select * from servicekeyapproval;
id | approver_id | approval_type | approved_date | notes ----+-------------+----------------------------------+----------------------------+------- 1 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 | 2 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 | 3 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095 | 4 | | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235 | 5 | 1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
quay=# INSERT INTO servicekey (name, service, metadata, kid, jwk, created_date, expiration_date) VALUES ('quay-readonly', 'quay', '{}', '{<contents_of_.kid_file>}', '{<contents_of_.jwk_file>}', '{<created_date_of_read-only>}', '{<expiration_date_of_read-only>}');
INSERT 0 1
quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes') VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE", {include_notes_here_on_why_this_is_being_added});
INSERT 0 1
UPDATE servicekey SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER') WHERE name = 'quay-readonly';
UPDATE 1
podman stop <quay_container_name_on_virtual_machine_a>
podman stop <quay_container_name_on_virtual_machine_b>
cp quay-readonly.kid quay-readonly.pem USDQuay/config
setfacl -m user:1001:rw USDQuay/config/*
REGISTRY_STATE: readonly INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid' INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
podman run -d --rm -p 80:8080 -p 443:8443 --name=quay-main-app -v USDQUAY/config:/conf/stack:Z -v USDQUAY/storage:/datastorage:Z {productrepo}/{quayimage}:{productminv}
podman push <quay-server.example.com>/quayadmin/busybox:test
613be09ab3c0: Preparing denied: System is currently read-only. Pulls will succeed but all write operations are currently suspended.
REGISTRY_STATE: readonly INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid' INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
podman restart <container_id>
quay=# UPDATE servicekey SET expiration_date = 'new-date' WHERE id = servicekey_id;
SELECT id, name, expiration_date FROM servicekey;
mkdir /tmp/quay-backup
podman run --name quay-app -v /opt/quay-install/config:/conf/stack:Z -v /opt/quay-install/storage:/datastorage:Z registry.redhat.io/quay/quay-rhel8:v3.12.8
cd /opt/quay-install
tar cvf /tmp/quay-backup/quay-backup.tar.gz *
config.yaml config.yaml.bak extra_ca_certs/ extra_ca_certs/ca.crt ssl.cert ssl.key
podman inspect quay-app | jq -r '.[0].Config.CreateCommand | .[]' | paste -s -d ' ' - /usr/bin/podman run --name quay-app -v /opt/quay-install/config:/conf/stack:Z -v /opt/quay-install/storage:/datastorage:Z registry.redhat.io/quay/quay-rhel8:v3.12.8
podman exec -it quay cat /conf/stack/config.yaml > /tmp/quay-backup/quay-config.yaml
grep DB_URI /tmp/quay-backup/quay-config.yaml
postgresql://<username>:test123@172.24.10.50/quay
pg_dump -h 172.24.10.50 -p 5432 -d quay -U <username> -W -O > /tmp/quay-backup/quay-backup.sql
DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - s3_bucket: <bucket_name> storage_path: /registry s3_access_key: <s3_access_key> s3_secret_key: <s3_secret_key> host: <host_name> s3_region: <region>
export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<secret_key>
aws s3 sync s3://<bucket_name> /tmp/quay-backup/blob-backup/ --source-region us-east-2
download: s3://<user_name>/registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a to registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a download: s3://<user_name>/registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d to registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d
mkdir /opt/new-quay-install
cp /tmp/quay-backup/quay-backup.tar.gz /opt/new-quay-install/
cd /opt/new-quay-install/
tar xvf /tmp/quay-backup/quay-backup.tar.gz *
config.yaml config.yaml.bak extra_ca_certs/ extra_ca_certs/ca.crt ssl.cert ssl.key
grep DB_URI config.yaml
postgresql://<username>:test123@172.24.10.50/quay
sudo postgres
psql "host=172.24.10.50 port=5432 dbname=postgres user=<username> password=test123" postgres=> CREATE DATABASE example_restore_registry_quay_database;
CREATE DATABASE
postgres=# \c "example-restore-registry-quay-database";
You are now connected to database "example-restore-registry-quay-database" as user "postgres".
example_restore_registry_quay_database=> CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION
\q
psql "host=172.24.10.50 port=5432 dbname=example_restore_registry_quay_database user=<username> password=test123" -W < /tmp/quay-backup/quay-backup.sql
SET SET SET SET SET
cat config.yaml | grep DISTRIBUTED_STORAGE_CONFIG -A10
DISTRIBUTED_STORAGE_CONFIG: default: DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - s3_bucket: <bucket_name> storage_path: /registry s3_access_key: <s3_access_key> s3_region: <region> s3_secret_key: <s3_secret_key> host: <host_name>
export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<secret_key>
aws s3 mb s3://<new_bucket_name> --region us-east-2
make_bucket: quay
aws s3 sync --no-verify-ssl --endpoint-url <example_endpoint_url> 1 /tmp/quay-backup/blob-backup/. s3://quay/
upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d to s3://quay/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 to s3://quay/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec to s3://quay/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec
DISTRIBUTED_STORAGE_CONFIG: default: DISTRIBUTED_STORAGE_CONFIG: default: - S3Storage - s3_bucket: <new_bucket_name> storage_path: /registry s3_access_key: <s3_access_key> s3_secret_key: <s3_secret_key> s3_region: <region> host: <host_name>