Chapter 8. Repository mirroring 8.1. Repository mirroring Red Hat Quay repository mirroring lets you mirror images from external container registries, or another local registry, into your Red Hat Quay cluster. Using repository mirroring, you can synchronize images to Red Hat Quay based on repository names and tags. From your Red Hat Quay cluster with repository mirroring enabled, you can perform the following: Choose a repository from an external registry to mirror Add credentials to access the external registry Identify specific container image repository names and tags to sync Set intervals at which a repository is synced Check the current state of synchronization To use the mirroring functionality, you need to perform the following actions: Enable repository mirroring in the Red Hat Quay configuration file Run a repository mirroring worker Create mirrored repositories All repository mirroring configurations can be performed using the configuration tool UI or by the Red Hat Quay API. 8.2. Repository mirroring compared to geo-replication Red Hat Quay geo-replication mirrors the entire image storage backend data between 2 or more different storage backends while the database is shared, for example, one Red Hat Quay registry with two different blob storage endpoints. The primary use cases for geo-replication include the following: Speeding up access to the binary blobs for geographically dispersed setups Guaranteeing that the image content is the same across regions Repository mirroring synchronizes selected repositories, or subsets of repositories, from one registry to another. The registries are distinct, with each registry having a separate database and separate image storage. The primary use cases for mirroring include the following: Independent registry deployments in different data centers or regions, where a certain subset of the overall content is supposed to be shared across the data centers and regions Automatic synchronization or mirroring of selected (allowlisted) upstream repositories from external registries into a local Red Hat Quay deployment Note Repository mirroring and geo-replication can be used simultaneously. Table 8.1. Red Hat Quay Repository mirroring and geo-replication comparison Feature / Capability Geo-replication Repository mirroring What is the feature designed to do? A shared, global registry Distinct, different registries What happens if replication or mirroring has not been completed yet? The remote copy is used (slower) No image is served Is access to all storage backends in both regions required? Yes (all Red Hat Quay nodes) No (distinct storage) Can users push images from both sites to the same repository? Yes No Is all registry content and configuration identical across all regions (shared database)? Yes No Can users select individual namespaces or repositories to be mirrored? No Yes Can users apply filters to synchronization rules? No Yes Are individual / different role-base access control configurations allowed in each region No Yes 8.3. Using repository mirroring The following list shows features and limitations of Red Hat Quay repository mirroring: With repository mirroring, you can mirror an entire repository or selectively limit which images are synced. Filters can be based on a comma-separated list of tags, a range of tags, or other means of identifying tags through Unix shell-style wildcards. For more information, see the documentation for wildcards . When a repository is set as mirrored, you cannot manually add other images to that repository. Because the mirrored repository is based on the repository and tags you set, it will hold only the content represented by the repository and tag pair. For example if you change the tag so that some images in the repository no longer match, those images will be deleted. Only the designated robot can push images to a mirrored repository, superseding any role-based access control permissions set on the repository. Mirroring can be configured to rollback on failure, or to run on a best-effort basis. With a mirrored repository, a user with read permissions can pull images from the repository but cannot push images to the repository. Changing settings on your mirrored repository can be performed in the Red Hat Quay user interface, using the Repositories Mirrors tab for the mirrored repository you create. Images are synced at set intervals, but can also be synced on demand. 8.4. Mirroring configuration UI Start the Quay container in configuration mode and select the Enable Repository Mirroring check box. If you want to require HTTPS communications and verify certificates during mirroring, select the HTTPS and cert verification check box. Validate and download the configuration file, and then restart Quay in registry mode using the updated config file. 8.5. Mirroring configuration fields Table 8.2. Mirroring configuration Field Type Description FEATURE_REPO_MIRROR Boolean Enable or disable repository mirroring Default: false REPO_MIRROR_INTERVAL Number The number of seconds between checking for repository mirror candidates Default: 30 REPO_MIRROR_SERVER_HOSTNAME String Replaces the SERVER_HOSTNAME as the destination for mirroring. Default: None Example : openshift-quay-service REPO_MIRROR_TLS_VERIFY Boolean Require HTTPS and verify certificates of Quay registry during mirror. Default: true REPO_MIRROR_ROLLBACK Boolean When set to true , the repository rolls back after a failed mirror attempt. Default : false 8.6. Mirroring worker Use the following procedure to start the repository mirroring worker. Procedure If you have not configured TLS communications using a /root/ca.crt certificate, enter the following command to start a Quay pod with the repomirror option: If you have configured TLS communications using a /root/ca.crt certificate, enter the following command to start the repository mirroring worker: 8.7. Creating a mirrored repository When mirroring a repository from an external container registry, you must create a new private repository. Typically, the same name is used as the target repository, for example, quay-rhel8 . 8.7.1. Repository mirroring settings Use the following procedure to adjust the settings of your mirrored repository. Prerequisites You have enabled repository mirroring in your Red Hat Quay configuration file. You have deployed a mirroring worker. Procedure In the Settings tab, set the Repository State to Mirror : In the Mirror tab, enter the details for connecting to the external registry, along with the tags, scheduling and access information: Enter the details as required in the following fields: Registry Location: The external repository you want to mirror, for example, registry.redhat.io/quay/quay-rhel8 Tags: This field is required. You may enter a comma-separated list of individual tags or tag patterns. (See Tag Patterns section for details.) Start Date: The date on which mirroring begins. The current date and time is used by default. Sync Interval: Defaults to syncing every 24 hours. You can change that based on hours or days. Robot User: Create a new robot account or choose an existing robot account to do the mirroring. Username: The username for accessing the external registry holding the repository you are mirroring. Password: The password associated with the Username. Note that the password cannot include characters that require an escape character (\). 8.7.2. Advanced settings In the Advanced Settings section, you can configure SSL/TLS and proxy with the following options: Verify TLS: Select this option if you want to require HTTPS and to verify certificates when communicating with the target remote registry. Accept Unsigned Images: Selecting this option allows unsigned images to be mirrored. HTTP Proxy: Select this option if you want to require HTTPS and to verify certificates when communicating with the target remote registry. HTTPS PROXY: Identify the HTTPS proxy server needed to access the remote site, if a proxy server is needed. No Proxy: List of locations that do not require proxy. 8.7.3. Synchronize now Use the following procedure to initiate the mirroring operation. Procedure To perform an immediate mirroring operation, press the Sync Now button on the repository's Mirroring tab. The logs are available on the Usage Logs tab: When the mirroring is complete, the images will appear in the Tags tab: Below is an example of a completed Repository Mirroring screen: 8.8. Event notifications for mirroring There are three notification events for repository mirroring: Repository Mirror Started Repository Mirror Success Repository Mirror Unsuccessful The events can be configured inside of the Settings tab for each repository, and all existing notification methods such as email, Slack, Quay UI, and webhooks are supported. 8.9. Mirroring tag patterns At least one tag must be entered. The following table references possible image tag patterns. 8.9.1. Pattern syntax Pattern Description * Matches all characters ? Matches any single character [seq] Matches any character in seq [!seq] Matches any character not in seq 8.9.2. Example tag patterns Example Pattern Example Matches v3* v32, v3.1, v3.2, v3.2-4beta, v3.3 v3.* v3.1, v3.2, v3.2-4beta v3.? v3.1, v3.2, v3.3 v3.[12] v3.1, v3.2 v3.[12]* v3.1, v3.2, v3.2-4beta v3.[!1]* v3.2, v3.2-4beta, v3.3 8.10. Working with mirrored repositories Once you have created a mirrored repository, there are several ways you can work with that repository. Select your mirrored repository from the Repositories page and do any of the following: Enable/disable the repository : Select the Mirroring button in the left column, then toggle the Enabled check box to enable or disable the repository temporarily. Check mirror logs : To make sure the mirrored repository is working properly, you can check the mirror logs. To do that, select the Usage Logs button in the left column. Here's an example: Sync mirror now : To immediately sync the images in your repository, select the Sync Now button. Change credentials : To change the username and password, select DELETE from the Credentials line. Then select None and add the username and password needed to log into the external registry when prompted. Cancel mirroring : To stop mirroring, which keeps the current images available but stops new ones from being synced, select the CANCEL button. Set robot permissions : Red Hat Quay robot accounts are named tokens that hold credentials for accessing external repositories. By assigning credentials to a robot, that robot can be used across multiple mirrored repositories that need to access the same external registry. You can assign an existing robot to a repository by going to Account Settings, then selecting the Robot Accounts icon in the left column. For the robot account, choose the link under the REPOSITORIES column. From the pop-up window, you can: Check which repositories are assigned to that robot. Assign read, write or Admin privileges to that robot from the PERMISSION field shown in this figure: Change robot credentials : Robots can hold credentials such as Kubernetes secrets, Docker login information, and Mesos bundles. To change robot credentials, select the Options gear on the robot's account line on the Robot Accounts window and choose View Credentials. Add the appropriate credentials for the external repository the robot needs to access. Check and change general setting : Select the Settings button (gear icon) from the left column on the mirrored repository page. On the resulting page, you can change settings associated with the mirrored repository. In particular, you can change User and Robot Permissions, to specify exactly which users and robots can read from or write to the repo. 8.11. Repository mirroring recommendations Best practices for repository mirroring include the following: Repository mirroring pods can run on any node. This means that you can run mirroring on nodes where Red Hat Quay is already running. Repository mirroring is scheduled in the database and runs in batches. As a result, repository workers check each repository mirror configuration file and reads when the sync needs to be. More mirror workers means more repositories can be mirrored at the same time. For example, running 10 mirror workers means that a user can run 10 mirroring operators in parallel. If a user only has 2 workers with 10 mirror configurations, only 2 operators can be performed. The optimal number of mirroring pods depends on the following conditions: The total number of repositories to be mirrored The number of images and tags in the repositories and the frequency of changes Parallel batching For example, if a user is mirroring a repository that has 100 tags, the mirror will be completed by one worker. Users must consider how many repositories one wants to mirror in parallel, and base the number of workers around that. Multiple tags in the same repository cannot be mirrored in parallel.

Commands:
sudo podman run -d --name mirroring-worker -v USDQUAY/config:/conf/stack:Z registry.redhat.io/quay/quay-rhel8:v3.13.3 repomirror
sudo podman run -d --name mirroring-worker -v USDQUAY/config:/conf/stack:Z -v /root/ca.crt:/etc/pki/ca-trust/source/anchors/ca.crt:Z registry.redhat.io/quay/quay-rhel8:v3.13.3 repomirror