Chapter 3. Preparing for Red Hat Quay (high availability) Note This procedure presents guidance on how to set up a highly available, production-quality deployment of Red Hat Quay. 3.1. Prerequisites Here are a few things you need to know before you begin the Red Hat Quay high availability deployment: Either Postgres or MySQL can be used to provide the database service. Postgres was chosen here as the database because it includes the features needed to support Clair security scanning. Other options include: Crunchy Data PostgreSQL Operator: Although not supported directly by Red Hat, the CrunchDB Operator is available from Crunchy Data for use with Red Hat Quay. If you take this route, you should have a support contract with Crunchy Data and work directly with them for usage guidance or issues relating to the operator and their database. If your organization already has a high-availability (HA) database, you can use that database with Red Hat Quay. See the Red Hat Quay Support Policy for details on support for third-party databases and other components. Ceph Object Gateway (also called RADOS Gateway) is one example of a product that can provide the object storage needed by Red Hat Quay. If you want your Red Hat Quay setup to do geo-replication, Ceph Object Gateway or other supported object storage is required. For cloud installations, you can use any of the following cloud object storage: Amazon S3 (see S3 IAM Bucket Policy for details on configuring an S3 bucket policy for Quay) Azure Blob Storage Google Cloud Storage Ceph Object Gateway OpenStack Swift CloudFront + S3 NooBaa S3 Storage The haproxy server is used in this example, although you can use any proxy service that works for your environment. Number of systems: This procedure uses seven systems (physical or virtual) that are assigned with the following tasks: A: db01: Load balancer and database : Runs the haproxy load balancer and a Postgres database. Note that these components are not themselves highly available, but are used to indicate how you might set up your own load balancer or production database. B: quay01, quay02, quay03: Quay and Redis : Three (or more) systems are assigned to run the Quay and Redis services. C: ceph01, ceph02, ceph03, ceph04, ceph05: Ceph : Three (or more) systems provide the Ceph service, for storage. If you are deploying to a cloud, you can use the cloud storage features described earlier. This procedure employs an additional system for Ansible (ceph05) and one for a Ceph Object Gateway (ceph04). Each system should have the following attributes: Red Hat Enterprise Linux (RHEL) 8: Obtain the latest Red Hat Enterprise Linux 8 server media from the Downloads page and follow the installation instructions available in the Product Documentation for Red Hat Enterprise Linux 9 . Valid Red Hat Subscription : Configure a valid Red Hat Enterprise Linux 8 server subscription. CPUs : Two or more virtual CPUs RAM : 4GB for each A and B system; 8GB for each C system Disk space : About 20GB of disk space for each A and B system (10GB for the operating system and 10GB for docker storage). At least 30GB of disk space for C systems (or more depending on required container storage). 3.2. Using podman This document uses podman for creating and deploying containers. If you do not have podman available on your system, you should be able to use the equivalent docker commands. For more information on podman and related technologies, see Building, running, and managing Linux containers on Red Hat Enterprise Linux 8 . Note Podman is strongly recommended for highly available, production quality deployments of Red Hat Quay. Docker has not been tested with Red Hat Quay 3.10, and will be deprecated in a future release. 3.3. Setting up the HAProxy load balancer and the PostgreSQL database Use the following procedure to set up the HAProxy load balancer and the PostgreSQL database. Prerequisites You have installed the Podman or Docker CLI. Procedure On the first two systems, q01 and q02 , install the HAProxy load balancer and the PostgreSQL database. This configures HAProxy as the access point and load balancer for the following services running on other systems: Red Hat Quay (ports 80 and 443 on B systems) Redis (port 6379 on B systems) RADOS (port 7480 on C systems) Open all HAProxy ports in SELinux and selected HAProxy ports in the firewall: # setsebool -P haproxy_connect_any=on # firewall-cmd --permanent --zone=public --add-port=6379/tcp --add-port=7480/tcp success # firewall-cmd --reload success Configure the /etc/haproxy/haproxy.cfg to point to the systems and ports providing the Red Hat Quay, Redis and Ceph RADOS services. The following are examples of defaults and added frontend and backend settings: After the new haproxy.cfg file is in place, restart the HAProxy service by entering the following command: # systemctl restart haproxy Create a folder for the PostgreSQL database by entering the following command: USD mkdir -p /var/lib/pgsql/data Set the following permissions for the /var/lib/pgsql/data folder: USD chmod 777 /var/lib/pgsql/data Enter the following command to start the PostgreSQL database: USD sudo podman run -d --name postgresql_database \ -v /var/lib/pgsql/data:/var/lib/pgsql/data:Z \ -e POSTGRESQL_USER=quayuser -e POSTGRESQL_PASSWORD=quaypass \ -e POSTGRESQL_DATABASE=quaydb -p 5432:5432 \ registry.redhat.io/rhel8/postgresql-13:1-109 Note Data from the container will be stored on the host system in the /var/lib/pgsql/data directory. List the available extensions by entering the following command: USD sudo podman exec -it postgresql_database /bin/bash -c 'echo "SELECT * FROM pg_available_extensions" | /opt/rh/rh-postgresql96/root/usr/bin/psql' Example output name | default_version | installed_version | comment -----------+-----------------+-------------------+---------------------------------------- adminpack | 1.0 | | administrative functions for PostgreSQL ... Create the pg_trgm extension by entering the following command: USD sudo podman exec -it postgresql_database /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS pg_trgm;" | /opt/rh/rh-postgresql96/root/usr/bin/psql -d quaydb' Confirm that the pg_trgm has been created by entering the following command: USD sudo podman exec -it postgresql_database /bin/bash -c 'echo "SELECT * FROM pg_extension" | /opt/rh/rh-postgresql96/root/usr/bin/psql' Example output extname | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition ---------+----------+--------------+----------------+------------+-----------+-------------- plpgsql | 10 | 11 | f | 1.0 | | pg_trgm | 10 | 2200 | t | 1.3 | | (2 rows) Alter the privileges of the Postgres user quayuser and grant them the superuser role to give the user unrestricted access to the database: USD sudo podman exec -it postgresql_database /bin/bash -c 'echo "ALTER USER quayuser WITH SUPERUSER;" | /opt/rh/rh-postgresql96/root/usr/bin/psql' Example output ALTER ROLE If you have a firewalld service active on your system, run the following commands to make the PostgreSQL port available through the firewall: # firewall-cmd --permanent --zone=trusted --add-port=5432/tcp # firewall-cmd --reload Optional. If you do not have the postgres CLI package installed, install it by entering the following command: # yum install postgresql -y Use the psql command to test connectivity to the PostgreSQL database. Note To verify that you can access the service remotely, run the following command on a remote system. Example output Password for user test: psql (9.2.23, server 9.6.5) WARNING: psql version 9.2, server version 9.6. Some psql features might not work. Type "help" for help. test=> \q 3.4. Set Up Ceph For this Red Hat Quay configuration, we create a three-node Ceph cluster, with several other supporting nodes, as follows: ceph01, ceph02, and ceph03 - Ceph Monitor, Ceph Manager and Ceph OSD nodes ceph04 - Ceph RGW node ceph05 - Ceph Ansible administration node For details on installing Ceph nodes, see Installing Red Hat Ceph Storage on Red Hat Enterprise Linux . Once you have set up the Ceph storage cluster, create a Ceph Object Gateway (also referred to as a RADOS gateway). See Installing the Ceph Object Gateway for details. 3.4.1. Install each Ceph node On ceph01, ceph02, ceph03, ceph04, and ceph05, do the following: Review prerequisites for setting up Ceph nodes in Requirements for Installing Red Hat Ceph Storage . In particular: Decide if you want to use RAID controllers on OSD nodes . Decide if you want a separate cluster network for your Ceph Network Configuration . Prepare OSD storage (ceph01, ceph02, and ceph03 only). Set up the OSD storage on the three OSD nodes (ceph01, ceph02, and ceph03). See OSD Ansible Settings in Table 3.2 for details on supported storage types that you will enter into your Ansible configuration later. For this example, a single, unformatted block device ( /dev/sdb ), that is separate from the operating system, is configured on each of the OSD nodes. If you are installing on metal, you might want to add an extra hard drive to the machine for this purpose. Install Red Hat Enterprise Linux Server edition, as described in the RHEL 7 Installation Guide . Register and subscribe each Ceph node as described in the Registering Red Hat Ceph Storage Nodes . Here is how to subscribe to the necessary repos: Create an ansible user with root privilege on each node. Choose any name you like. For example: 3.4.2. Configure the Ceph Ansible node (ceph05) Log into the Ceph Ansible node (ceph05) and configure it as follows. You will need the ceph01, ceph02, and ceph03 nodes to be running to complete these steps. In the Ansible user's home directory create a directory to store temporary values created from the ceph-ansible playbook Enable password-less ssh for the ansible user. Run ssh-keygen on ceph05 (leave passphrase empty), then run and repeat ssh-copy-id to copy the public key to the Ansible user on ceph01, ceph02, and ceph03 systems: Install the ceph-ansible package: Create a symbolic between these two directories: Create copies of Ceph sample yml files to modify: Edit the copied group_vars/all.yml file. See General Ansible Settings in Table 3.1 for details. For example: Note that your network device and address range may differ. Edit the copied group_vars/osds.yml file. See the OSD Ansible Settings in Table 3.2 for details. In this example, the second disk device ( /dev/sdb ) on each OSD node is used for both data and journal storage: Edit the /etc/ansible/hosts inventory file to identify the Ceph nodes as Ceph monitor, OSD and manager nodes. In this example, the storage devices are identified on each node as well: Add this line to the /etc/ansible/ansible.cfg file, to save the output from each Ansible playbook run into your Ansible user's home directory: Check that Ansible can reach all the Ceph nodes you configured as your Ansible user: Run the ceph-ansible playbook (as your Ansible user): At this point, the Ansible playbook will check your Ceph nodes and configure them for the services you requested. If anything fails, make needed corrections and rerun the command. Log into one of the three Ceph nodes (ceph01, ceph02, or ceph03) and check the health of the Ceph cluster: On the same node, verify that monitoring is working using rados: 3.4.3. Install the Ceph Object Gateway On the Ansible system (ceph05), configure a Ceph Object Gateway to your Ceph Storage cluster (which will ultimately run on ceph04). See Installing the Ceph Object Gateway for details. 3.5. Set up Redis With Red Hat Enterprise Linux 8 server installed on each of the three Red Hat Quay systems (quay01, quay02, and quay03), install and start the Redis service as follows: Install / Deploy Redis : Run Redis as a container on each of the three quay0* systems: Check redis connectivity : You can use the telnet command to test connectivity to the redis service. Type MONITOR (to begin monitoring the service) and QUIT to exit: Note For more information on using podman and restarting containers, see the section "Using podman" earlier in this document.

Commands:
setsebool -P haproxy_connect_any=on firewall-cmd --permanent --zone=public --add-port=6379/tcp --add-port=7480/tcp success firewall-cmd --reload success
#--------------------------------------------------------------------- common defaults that all the 'listen' and 'backend' sections will use if not designated in their block #--------------------------------------------------------------------- defaults mode tcp log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- main frontend which proxys to the backends #--------------------------------------------------------------------- frontend fe_http *:80 default_backend be_http frontend fe_https *:443 default_backend be_https frontend fe_redis *:6379 default_backend be_redis frontend fe_rdgw *:7480 default_backend be_rdgw backend be_http balance roundrobin server quay01 quay01:80 check server quay02 quay02:80 check server quay03 quay03:80 check backend be_https balance roundrobin server quay01 quay01:443 check server quay02 quay02:443 check server quay03 quay03:443 check backend be_rdgw balance roundrobin server ceph01 ceph01:7480 check server ceph02 ceph02:7480 check server ceph03 ceph03:7480 check backend be_redis server quay01 quay01:6379 check inter 1s server quay02 quay02:6379 check inter 1s server quay03 quay03:6379 check inter 1s
systemctl restart haproxy
mkdir -p /var/lib/pgsql/data
chmod 777 /var/lib/pgsql/data
sudo podman run -d --name postgresql_database -v /var/lib/pgsql/data:/var/lib/pgsql/data:Z -e POSTGRESQL_USER=quayuser -e POSTGRESQL_PASSWORD=quaypass -e POSTGRESQL_DATABASE=quaydb -p 5432:5432 registry.redhat.io/rhel8/postgresql-13:1-109
sudo podman exec -it postgresql_database /bin/bash -c 'echo "SELECT * FROM pg_available_extensions" | /opt/rh/rh-postgresql96/root/usr/bin/psql'
name | default_version | installed_version | comment -----------+-----------------+-------------------+---------------------------------------- adminpack | 1.0 | | administrative functions for PostgreSQL
sudo podman exec -it postgresql_database /bin/bash -c 'echo "CREATE EXTENSION IF NOT EXISTS pg_trgm;" | /opt/rh/rh-postgresql96/root/usr/bin/psql -d quaydb'
sudo podman exec -it postgresql_database /bin/bash -c 'echo "SELECT * FROM pg_extension" | /opt/rh/rh-postgresql96/root/usr/bin/psql'
extname | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition ---------+----------+--------------+----------------+------------+-----------+-------------- plpgsql | 10 | 11 | f | 1.0 | | pg_trgm | 10 | 2200 | t | 1.3 | | (2 rows)
sudo podman exec -it postgresql_database /bin/bash -c 'echo "ALTER USER quayuser WITH SUPERUSER;" | /opt/rh/rh-postgresql96/root/usr/bin/psql'
ALTER ROLE
firewall-cmd --permanent --zone=trusted --add-port=5432/tcp
firewall-cmd --reload
yum install postgresql -y
psql -h localhost quaydb quayuser
Password for user test: psql (9.2.23, server 9.6.5) WARNING: psql version 9.2, server version 9.6. Some psql features might not work. Type "help" for help. test=> \q
subscription-manager repos --disable=* subscription-manager repos --enable=rhel-7-server-rpms subscription-manager repos --enable=rhel-7-server-extras-rpms subscription-manager repos --enable=rhel-7-server-rhceph-3-mon-rpms subscription-manager repos --enable=rhel-7-server-rhceph-3-osd-rpms subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
USER_NAME=ansibleadmin useradd USDUSER_NAME -c "Ansible administrator" passwd USDUSER_NAME New password: ********* Retype new password: ********* cat << EOF >/etc/sudoers.d/admin admin ALL = (root) NOPASSWD:ALL EOF chmod 0440 /etc/sudoers.d/USDUSER_NAME
USER_NAME=ansibleadmin sudo su - USDUSER_NAME [ansibleadmin@ceph05 ~]USD mkdir ~/ceph-ansible-keys
USER_NAME=ansibleadmin sudo su - USDUSER_NAME [ansibleadmin@ceph05 ~]USD ssh-keygen [ansibleadmin@ceph05 ~]USD ssh-copy-id USDUSER_NAME@ceph01 [ansibleadmin@ceph05 ~]USD ssh-copy-id USDUSER_NAME@ceph02 [ansibleadmin@ceph05 ~]USD ssh-copy-id USDUSER_NAME@ceph03 [ansibleadmin@ceph05 ~]USD exit #
yum install ceph-ansible
ln -s /usr/share/ceph-ansible/group_vars /etc/ansible/group_vars
cd /usr/share/ceph-ansible cp group_vars/all.yml.sample group_vars/all.yml cp group_vars/osds.yml.sample group_vars/osds.yml cp site.yml.sample site.yml
ceph_origin: repository ceph_repository: rhcs ceph_repository_type: cdn ceph_rhcs_version: 3 monitor_interface: eth0 public_network: 192.168.122.0/24
osd_scenario: collocated devices: - /dev/sdb dmcrypt: true osd_auto_discovery: false
[mons] ceph01 ceph02 ceph03 [osds] ceph01 devices="[ '/dev/sdb' ]" ceph02 devices="[ '/dev/sdb' ]" ceph03 devices="[ '/dev/sdb' ]" [mgrs] ceph01 devices="[ '/dev/sdb' ]" ceph02 devices="[ '/dev/sdb' ]" ceph03 devices="[ '/dev/sdb' ]"
retry_files_save_path = ~/
USER_NAME=ansibleadmin sudo su - USDUSER_NAME [ansibleadmin@ceph05 ~]USD ansible all -m ping ceph01 | SUCCESS => { "changed": false, "ping": "pong" } ceph02 | SUCCESS => { "changed": false, "ping": "pong" } ceph03 | SUCCESS => { "changed": false, "ping": "pong" } [ansibleadmin@ceph05 ~]USD
[ansibleadmin@ceph05 ~]USD cd /usr/share/ceph-ansible/ [ansibleadmin@ceph05 ~]USD ansible-playbook site.yml
ceph health HEALTH_OK
ceph osd pool create test 8 echo 'Hello World!' > hello-world.txt rados --pool test put hello-world hello-world.txt rados --pool test get hello-world fetch.txt cat fetch.txt Hello World!
mkdir -p /var/lib/redis chmod 777 /var/lib/redis sudo podman run -d -p 6379:6379 -v /var/lib/redis:/var/lib/redis/data:Z registry.redhat.io/rhel8/redis-5
yum install telnet -y telnet 192.168.122.99 6379 Trying 192.168.122.99 Connected to 192.168.122.99. Escape character is '^]'. MONITOR +OK +1525703165.754099 [0 172.17.0.1:43848] "PING" QUIT +OK Connection closed by foreign host.