Chapter 6. Automatically building Dockerfiles with Build workers Red Hat Quay supports building Dockerfiles using a set of worker nodes on OpenShift or Kubernetes. Build triggers, such as GitHub webhooks can be configured to automatically build new versions of your repositories when new code is committed. This document will walk you through enabling builds with your Red Hat Quay installation and setting up one or more OpenShift/K8s clusters to accept builds from Red Hat Quay. With Red Hat Quay 3.4, the underlying Build Manager has been completely re-written as part of Red Hat Quay's migration from Python 2 to Python 3. As a result, builder nodes are now dynamically created as Kubernetes Jobs versus builder nodes that ran continuously in Red Hat Quay 3.3 and earlier. This greatly simplifies how Red Hat Quay manages builds and provides the same mechanism quay.io utilizes to handle thousands of container image builds daily. Customers who are currently running static ("Enterprise" builders under Red Hat Quay 3.3) will be required to migrate to a Kubernetes-based build mechanism. 6.1. Architecture Overview The Red Hat Quay Build system is designed for scalability (since it is used to host all builds at quay.io). The Build Manager component of Red Hat Quay provides an orchestration layer that tracks build requests and ensures that a Build Executor (OpenShift/K8s cluster) will carry out each request. Each build is handled by a Kubernetes Job which launches a small virtual machine to completely isolate and contain the image build process. This ensures that container builds do not affect each other or the underlying build system. Multiple Executors can be configured to ensure that builds are performed even in the event of infrastructure failures. Red Hat Quay will automatically send builds to a different Executor if it detects that one Executor is having difficulties. Note The upstream version of Red Hat Quay provides instructions on how to configure an AWS/EC2 based Executor. This configuration is not supported for Red Hat Quay customers. 6.1.1. Build manager The build manager is responsible for the lifecycle of scheduled build. Operations requiring updating the build queue, build phase and running jobs' status is handled by the build manager. 6.1.2. Build workers' control plane Build jobs are run on separate worker nodes, and are scheduled on separate control planes (executor). Currently, Red Hat Quay supports running jobs on AWS and Kubernetes. Builds are executed using quay.io/quay/quay-builder. On AWS, builds are scheduled on EC2 instances. On k8s, the builds are scheduled as job resources. 6.1.3. Orchestrator The orchestrator is used to store the state of currently running build jobs, and publish events for the build manager to consume. e.g expiry events. Currently, the supported orchestrator backend is Redis. 6.2. OpenShift Requirements Red Hat Quay builds are supported on Kubernetes and OpenShift 4.5 and higher. A bare metal (non-virtualized) worker node is required since build pods require the ability to run kvm virtualization. Each build is done in an ephemeral virtual machine to ensure complete isolation and security while the build is running. In addition, your OpenShift cluster should permit the ServiceAccount associated with Red Hat Quay builds to run with the necessary SecurityContextConstraint to support privileged containers. 6.3. Orchestrator Requirements The Red Hat Quay builds need access to a Redis instance to track build status information. It is acceptable to use the same Redis instance already deployed with your Red Hat Quay installation. All build queues are managed in the Red Hat Quay database so there is no need for a highly available Redis instance. 6.4. Setting Up Red Hat Quay Builders With OpenShift 6.4.1. OpenShift TLS component The tls component allows you to control TLS configuration. Note Red Hat Quay 3.7 does not support builders when the TLS component is managed by the Operator. If you set tls to unmanaged , you supply your own ssl.cert and ssl.key files. In this instance, if you want your cluster to support builders, you must add both the Quay route and the builder route name to the SAN list in the cert, or alternatively use a wildcard. To add the builder route, use the following format: [quayregistry-cr-name]-quay-builder-[ocp-namespace].[ocp-domain-name] 6.4.2. Prepare OpenShift for Red Hat Quay Builds There are several actions that are needed on an OpenShift cluster before it can accept builds from Red Hat Quay. Create a project where builds will be run (e.g. 'builder') Create a ServiceAccount in this Project that will be used to run builds. Ensure that it has sufficient privileges to create Jobs and Pods . Copy the ServiceAccount 's token for use later. Identify the URL for the OpenShift cluster's API server. This can be found from the OpenShift Console. Identify a worker node label to be used when scheduling build Jobs . Because build pods need to run on bare metal worker nodes, typically these are identified with specific labels. Check with your cluster administrator to determine exactly which node label should be used. If the cluster is using a self-signed certificate, get the kube apiserver's CA to add to Red Hat Quay's extra certs. Get the name of the secret containing the CA: Get the ca.crt key value from the secret in the Openshift console. The value should begin with "-----BEGIN CERTIFICATE-----" Import the CA in Red Hat Quay using the ConfigTool. Ensure the name of this file matches K8S_API_TLS_CA . Create the necessary security contexts/role bindings for the ServiceAccount : 6.4.3. Enable Builders and add Build Configuration to Red Hat Quay's Configuration Bundle Ensure that you've got Builds enabled in your Red Hat Quay configuration. Add the following to your Red Hat Quay configuration bundle, replacing each value with a value specific to your installation. Note Currently only the Build feature itself can be enabled via the Red Hat Quay Config Tool. The actual configuration of the Build Manager and Executors must be done manually in the config.yaml file. BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ ORCHESTRATOR: REDIS_HOST: quay-redis-host REDIS_PASSWORD: quay-redis-password REDIS_SSL: true REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: builder K8S_API_SERVER: api.openshift.somehost.org:6443 K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 5120Mi CONTAINER_CPU_LIMITS: 1000m CONTAINER_MEMORY_REQUEST: 3968Mi CONTAINER_CPU_REQUEST: 500m NODE_SELECTOR_LABEL_KEY: beta.kubernetes.io/instance-type NODE_SELECTOR_LABEL_VALUE: n1-standard-4 CONTAINER_RUNTIME: podman SERVICE_ACCOUNT_NAME: ***** SERVICE_ACCOUNT_TOKEN: ***** QUAY_USERNAME: quay-username QUAY_PASSWORD: quay-password WORKER_IMAGE: <registry>/quay-quay-builder WORKER_TAG: some_tag BUILDER_VM_CONTAINER_IMAGE: <registry>/quay-quay-builder-qemu-rhcos:v3.4.0 SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 SSH_AUTHORIZED_KEYS: - ssh-rsa 12345 someuser@email.com - ssh-rsa 67890 someuser2@email.com Each configuration field is explained below. ALLOWED_WORKER_COUNT Defines how many Build Workers are instantiated per Red Hat Quay Pod. Typically this is '1'. ORCHESTRATOR_PREFIX Defines a unique prefix to be added to all Redis keys (useful to isolate Orchestrator values from other Redis keys). REDIS_HOST Hostname for your Redis service. REDIS_PASSWORD Password to authenticate into your Redis service. REDIS_SSL Defines whether or not your Redis connection uses SSL. REDIS_SKIP_KEYSPACE_EVENT_SETUP By default, Red Hat Quay does not set up the keyspace events required for key events at runtime. To do so, set REDIS_SKIP_KEYSPACE_EVENT_SETUP to false . EXECUTOR Starts a definition of an Executor of this type. Valid values are 'kubernetes' and 'ec2' BUILDER_NAMESPACE Kubernetes namespace where Red Hat Quay builds will take place K8S_API_SERVER Hostname for API Server of OpenShift cluster where builds will take place K8S_API_TLS_CA The filepath in the Quay container of the build cluster's CA certificate for the Quay app to trust when making API calls. KUBERNETES_DISTRIBUTION Indicates which type of Kubernetes is being used. Valid values are 'openshift' and 'k8s'. CONTAINER_* Define the resource requests and limits for each build pod. NODE_SELECTOR_* Defines the node selector label name/value pair where build Pods should be scheduled. CONTAINER_RUNTIME Specifies whether the builder should run docker or podman . Customers using Red Hat's quay-builder image should set this to podman . SERVICE_ACCOUNT_NAME/SERVICE_ACCOUNT_TOKEN Defines the Service Account name/token that will be used by build Pods. QUAY_USERNAME/QUAY_PASSWORD Defines the registry credentials needed to pull the Red Hat Quay build worker image that is specified in the WORKER_IMAGE field. Customers should provide a Red Hat Service Account credential as defined in the section "Creating Registry Service Accounts" against registry.redhat.io in the article at https://access.redhat.com/RegistryAuthentication . WORKER_IMAGE Image reference for the Red Hat Quay builder image. registry.redhat.io/quay/quay-builder WORKER_TAG Tag for the builder image desired. The latest version is v3.4.0. BUILDER_VM_CONTAINER_IMAGE The full reference to the container image holding the internal VM needed to run each Red Hat Quay build ( registry.redhat.io/quay/quay-builder-qemu-rhcos:v3.4.0 ). SETUP_TIME Specifies the number of seconds at which a build times out if it has not yet registered itself with the Build Manager (default is 500 seconds). Builds that time out are attempted to be restarted three times. If the build does not register itself after three attempts it is considered failed. MINIMUM_RETRY_THRESHOLD This setting is used with multiple Executors; it indicates how many retries are attempted to start a build before a different Executor is chosen. Setting to 0 means there are no restrictions on how many tries the build job needs to have. This value should be kept intentionally small (three or less) to ensure failovers happen quickly in the event of infrastructure failures. You must specify a value for this setting. E.g Kubernetes is set as the first executor and EC2 as the second executor. If we want the last attempt to run a job to always be executed on EC2 and not Kubernetes, we would set the Kubernetes executor's MINIMUM_RETRY_THRESHOLD to 1 and EC2's MINIMUM_RETRY_THRESHOLD to 0 (defaults to 0 if not set). In this case, kubernetes' MINIMUM_RETRY_THRESHOLD > retries_remaining(1) would evaluate to False, thus falling back to the second executor configured SSH_AUTHORIZED_KEYS List of ssh keys to bootstrap in the ignition config. This allows other keys to be used to ssh into the EC2 instance or QEMU VM 6.5. OpenShift Routes Limitation Note This section only applies if you are using the Quay Operator on OpenShift with managed route component. Due to a limitation of OpenShift Routes to only be able to serve traffic to a single port, additional steps are required to set up builds. Ensure that your kubectl or oc CLI tool is configured to work with the cluster where the Quay Operator is installed and that your QuayRegistry exists (not necessarily the same as the bare metal cluster where your builders run). Ensure that HTTP/2 ingress is enabled on the OpenShift cluster by following these steps . The Quay Operator will create a Route which directs gRPC traffic to the build manager server running inside the existing Quay pod(s). If you want to use a custom hostname (such as a subdomain like builder.registry.example.com ), ensure that you create a CNAME record with your DNS provider which points to the status.ingress[0].host of the created Route : Using the OpenShift UI or CLI, update the Secret referenced by spec.configBundleSecret of the QuayRegistry with the build cluster CA certificate (name the key extra_ca_cert_build_cluster.cert ), and update the config.yaml entry with the correct values referenced in the builder config above (depending on your build executor) along with the BUILDMAN_HOSTNAME field: BUILDMAN_HOSTNAME: <build-manager-hostname> BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 600 ORCHESTRATOR: REDIS_HOST: quay-redis-host REDIS_PASSWORD: quay-redis-password REDIS_SSL: true REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: builder ... The extra configuration field is explained below: BUILDMAN_HOSTNAME The externally accessible server hostname which the build jobs use to communicate back to the build manager. Default is the same as SERVER_HOSTNAME . For OpenShift Route , it is either status.ingress[0].host or the CNAME entry if using a custom hostname. BUILDMAN_HOSTNAME needs to include the port number, e.g somehost:443 for Openshift Route, as the gRPC client used to communicate with the build manager does not infer any port if omitted. 6.6. Troubleshooting Builds The builder instances started by the build manager are ephemeral. This means that they will either get shut down by Red Hat Quay} on timeouts/failure or garbage collected by the control plane (EC2/K8s). This means that in order to get the builder logs, one needs to do so while the builds are running. 6.6.1. DEBUG config flag A DEBUG flag can be set in order to prevent the builder instances from getting cleaned up after completion/failure. To do so, in the desired executor configuration, set DEBUG to true. For example: EXECUTORS: - EXECUTOR: ec2 DEBUG: true ... - EXECUTOR: kubernetes DEBUG: true ... When set to true, DEBUG will prevent the build nodes from shutting down after the quay-builder service is done or fails, and will prevent the build manager from cleaning up the instances (terminating EC2 instances or deleting k8s jobs). This will allow debugging builder node issues, and should not be set in a production environment. The lifetime service will still exist. i.e The instance will still shutdown after approximately 2 hours (EC2 instances will terminate, k8s jobs will complete) Setting DEBUG will also affect ALLOWED_WORKER_COUNT, as the unterminated instances/jobs will still count towards the total number of running workers. This means the existing builder workers will need to manually be deleted if ALLOWED_WORKER_COUNT is reached to be able to schedule new builds. Use the followings steps: The guest VM forwards its SSH port (22) to its host's (the pod) port 2222. Port forward the builder pod's port 2222 to a port on localhost. e.g SSH into the VM running inside the container using a key set from SSH_AUTHORIZED_KEYS: Get the quay-builder service logs: Step 2-3 can also be done in a single SSH command: 6.7. Setting up GitHub builds (optional) If your organization plans to have builds be conducted via pushes to GitHub (or GitHub Enterprise), continue with Creating an OAuth application in GitHub .

Commands:
[quayregistry-cr-name]-quay-builder-[ocp-namespace].[ocp-domain-name]
oc new-project builder
oc create sa -n builder quay-builder oc policy add-role-to-user -n builder edit system:serviceaccount:builder:quay-builder oc sa get-token -n builder quay-builder
oc get sa openshift-apiserver-sa --namespace=openshift-apiserver -o json | jq '.secrets[] | select(.name | contains("openshift-apiserver-sa-token"))'.name
apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: quay-builder priority: null readOnlyRootFilesystem: false requiredDropCapabilities: null runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - '*' supplementalGroups: type: RunAsAny volumes: - '*' allowHostDirVolumePlugin: true allowHostIPC: true allowHostNetwork: true allowHostPID: true allowHostPorts: true allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: - '*' allowedUnsafeSysctls: - '*' defaultAddCapabilities: null fsGroup: type: RunAsAny --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: quay-builder-scc namespace: builder rules: - apiGroups: - security.openshift.io resourceNames: - quay-builder resources: - securitycontextconstraints verbs: - use --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: quay-builder-scc namespace: builder subjects: - kind: ServiceAccount name: quay-builder roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: quay-builder-scc
FEATURE_BUILD_SUPPORT: True
BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ ORCHESTRATOR: REDIS_HOST: quay-redis-host REDIS_PASSWORD: quay-redis-password REDIS_SSL: true REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: builder K8S_API_SERVER: api.openshift.somehost.org:6443 K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 5120Mi CONTAINER_CPU_LIMITS: 1000m CONTAINER_MEMORY_REQUEST: 3968Mi CONTAINER_CPU_REQUEST: 500m NODE_SELECTOR_LABEL_KEY: beta.kubernetes.io/instance-type NODE_SELECTOR_LABEL_VALUE: n1-standard-4 CONTAINER_RUNTIME: podman SERVICE_ACCOUNT_NAME: ***** SERVICE_ACCOUNT_TOKEN: ***** QUAY_USERNAME: quay-username QUAY_PASSWORD: quay-password WORKER_IMAGE: <registry>/quay-quay-builder WORKER_TAG: some_tag BUILDER_VM_CONTAINER_IMAGE: <registry>/quay-quay-builder-qemu-rhcos:v3.4.0 SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 SSH_AUTHORIZED_KEYS: - ssh-rsa 12345 someuser@email.com - ssh-rsa 67890 someuser2@email.com
kubectl get -n <namespace> route <quayregistry-name>-quay-builder -o jsonpath={.status.ingress[0].host}
BUILDMAN_HOSTNAME: <build-manager-hostname> BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 600 ORCHESTRATOR: REDIS_HOST: quay-redis-host REDIS_PASSWORD: quay-redis-password REDIS_SSL: true REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: builder
EXECUTORS: - EXECUTOR: ec2 DEBUG: true - EXECUTOR: kubernetes DEBUG: true
kubectl port-forward <builder pod> 9999:2222
ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost
systemctl status quay-builder journalctl -f -u quay-builder
ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost 'systemctl status quay-builder' ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost 'journalctl -f -u quay-builder'