Chapter 3. Bare metal builds with Red Hat Quay on OpenShift Container Platform The procedures in this section explain how to create an environment for bare metal builds for Red Hat Quay on OpenShift Container Platform. 3.1. Configuring bare metal builds for Red Hat Quay on OpenShift Container Platform Use the following procedure to configure bare metal builds for Red Hat Quay on OpenShift Container Platform. Note If you are using the Red Hat Quay Operator on OpenShift Container Platform with a managed route component in your QuayRegistry CRD, see "Red Hat Quay on OpenShift Container Platform builds limitations with self-managed routes ". Prerequisites You have an OpenShift Container Platform cluster provisioned with the Red Hat Quay Operator running. You have set the tls component to unmanaged and uploaded custom SSL/TLS certificates to the Red Hat Quay Operator. For more information, see SSL and TLS for Red Hat Quay . You are logged into OpenShift Container Platform as a cluster administrator. Procedure Enter the following command to create a project where Builds will be run, for example, bare-metal-builder : USD oc new-project bare-metal-builder Create a new ServiceAccount in the the bare-metal-builder namespace by entering the following command: USD oc create sa -n bare-metal-builder quay-builder Enter the following command to grant a user the edit role within the bare-metal-builder namespace: USD oc policy add-role-to-user -n bare-metal-builder edit system:serviceaccount:bare-metal-builder:quay-builder Enter the following command to retrieve a token associated with the quay-builder service account in the bare-metal-builder namespace. This token is used to authenticate and interact with the OpenShift Container Platform cluster's API server. If your OpenShift Container Platform cluster is version 4.11+, enter the following command: oc create token quay-builder -n bare-metal-builder --duration 24h If your OpenShift Container Platform cluster is earlier than version 4.11, for example, version 4.10, enter the following command: USD oc sa get-token -n bare-metal-builder quay-builder Identify the URL for the OpenShift Container Platform cluster's API server. This can be found in the OpenShift Container Platform web console. Identify a worker node label to be used when scheduling build jobs . Because build pods must run on bare metal worker nodes, typically these are identified with specific labels. Check with your cluster administrator to determine exactly which node label should be used. Obtain the Kube API Server's certificate authority (CA) to add to Red Hat Quay's extra certificates. On OpenShift Container Platform versions 4.15+, enter the following commands to obtain the name of the secret containing the CA: USD oc extract cm/kube-root-ca.crt -n openshift-apiserver USD mv ca.crt build_cluster.crt On OpenShift Container Platform versions earlier than 4.15, for example, 4.14, enter the following command: USD oc get sa openshift-apiserver-sa --namespace=openshift-apiserver -o json | jq '.secrets[] | select(.name | contains("openshift-apiserver-sa-token"))'.name Obtain the ca.crt key value from the secret in the OpenShift Container Platform Web Console. The value begins with "-----BEGIN CERTIFICATE-----"` . Import the CA to Red Hat Quay. Ensure that the name of this file matches the K8S_API_TLS_CA field used in Step 9. Create the following SecurityContextConstraints resource for the ServiceAccount : apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: quay-builder priority: null readOnlyRootFilesystem: false requiredDropCapabilities: null runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - '*' supplementalGroups: type: RunAsAny volumes: - '*' allowHostDirVolumePlugin: true allowHostIPC: true allowHostNetwork: true allowHostPID: true allowHostPorts: true allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: - '*' allowedUnsafeSysctls: - '*' defaultAddCapabilities: null fsGroup: type: RunAsAny --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: quay-builder-scc namespace: bare-metal-builder rules: - apiGroups: - security.openshift.io resourceNames: - quay-builder resources: - securitycontextconstraints verbs: - use --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: quay-builder-scc namespace: bare-metal-builder subjects: - kind: ServiceAccount name: quay-builder roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: quay-builder-scc Update the config.yaml file of your Red Hat Quay on OpenShift Container Platform deployment to include an appropriate bare metal builds configuration by using the OpenShift Container Platform web console. Click Operators Installed Operators Red Hat Quay Quay Registry . Click the name of your registry, for example, example-registry . Under Config Bundle Secret , click the name of your configuration bundle, for example, extra-ca-certificate-config-bundle-secret . Click Actions Edit Secret . Add the following information to your Red Hat Quay config.yaml file, replacing each value with information that is relevant to your specific installation: FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - <superusername> FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_BUILD_SUPPORT: True BUILDMAN_HOSTNAME: USD{BUILDMAN_HOSTNAME}:443 1 BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 10 ORCHESTRATOR_PREFIX: buildman/production/ ORCHESTRATOR: REDIS_HOST: <sample_redis_hostname> 2 REDIS_PASSWORD: "" REDIS_SSL: false REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: <sample_builder_namespace> 3 K8S_API_SERVER: <sample_k8s_api_server> 4 K8S_API_TLS_CA: <sample_crt_file> 5 VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 1G 6 CONTAINER_CPU_LIMITS: 300m 7 CONTAINER_MEMORY_REQUEST: 1G 8 CONTAINER_CPU_REQUEST: 300m 9 NODE_SELECTOR_LABEL_KEY: beta.kubernetes.io/instance-type NODE_SELECTOR_LABEL_VALUE: n1-standard-4 CONTAINER_RUNTIME: podman SERVICE_ACCOUNT_NAME: <sample_service_account_name> SERVICE_ACCOUNT_TOKEN: <sample_account_token> 10 QUAY_USERNAME: <quay_username> QUAY_PASSWORD: <quay_password> WORKER_IMAGE: <registry>/quay-quay-builder WORKER_TAG: <some_tag> BUILDER_VM_CONTAINER_IMAGE: quay.io/quay/quay-builder-qemu-fedoracoreos:latest SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 SSH_AUTHORIZED_KEYS: 11 - <ssh-rsa 12345 someuser@email.com> - <ssh-rsa 67890 someuser2@email.com> HTTP_PROXY: <http://10.0.0.1:80> HTTPS_PROXY: <http://10.0.0.1:80> NO_PROXY: <hostname.example.com> 1 Obtained by running the following command: USD oc get route quayregistry-quay-builder -n USD{QUAY_PROJECT} -o jsonpath='{.spec.host}' . 2 The hostname for your Redis service. 3 Set to match the name of your bare metal builds namespace. This example used bare-metal-builder . 4 The K8S_API_SERVER is obtained by running USD oc cluster-info . 5 You must manually create and add your custom CA cert, for example, K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt . 6 Defaults to 5120Mi if left unspecified. 7 Defaults to 1000m if left unspecified. 8 Defaults to 3968Mi if left unspecified. 9 Defaults to 500m if left unspecified. 10 Obtained when running USD oc create sa . 11 Allows public SSH keys to be added to the build environment for remote troubleshooting access. This key, or keys, should correspond to the private key that an admin or developer will use to SSH into the build worker for debugging purposes. This key can be obtained by establishing an SSH connection to the remote host using a specific SSH key and port. For example: USD ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost . Restart your Red Hat Quay registry to enable the builds feature. 3.1.1. Red Hat Quay on OpenShift Container Platform builds limitations with self-managed routes The following limitations apply when you are using the Red Hat Quay Operator on OpenShift Container Platform with a managed route component: Currently, OpenShift Container Platform routes are only able to serve traffic to a single port. Additional steps are required to set up Red Hat Quay Builds. Ensure that your kubectl or oc CLI tool is configured to work with the cluster where the Red Hat Quay Operator is installed and that your QuayRegistry exists; the QuayRegistry does not have to be on the same bare metal cluster where builders run. Ensure that HTTP/2 ingress is enabled on the OpenShift cluster by following these steps . The Red Hat Quay Operator creates a Route resource that directs gRPC traffic to the Build manager server running inside of the existing Quay pod, or pods. If you want to use a custom hostname, or a subdomain like <builder-registry.example.com> , ensure that you create a CNAME record with your DNS provider that points to the status.ingress[0].host of the create Route resource. For example: Using the OpenShift Container Platform UI or CLI, update the Secret referenced by spec.configBundleSecret of the QuayRegistry with the build cluster CA certificate. Name the key extra_ca_cert_build_cluster.cert . Update the config.yaml file entry with the correct values referenced in the build configuration that you created when you configured Red Hat Quay builds , and add the BUILDMAN_HOSTNAME CONFIGURATION FIELD: BUILDMAN_HOSTNAME: <build-manager-hostname> 1 BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 600 ORCHESTRATOR: REDIS_HOST: <quay_redis_host REDIS_PASSWORD: <quay_redis_password> REDIS_SSL: true REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: builder ... 1 The externally accessible server hostname which the build jobs use to communicate back to the build manager . Default is the same as SERVER_HOSTNAME . For an OpenShift route resource, it is either status.ingress[0].host or the CNAME entry if using a custom hostname. BUILDMAN_HOSTNAME must include the port number, for example, somehost:443 for an OpenShift Container Platform route resource, as the gRPC client used to communicate with the build manager does not infer any port if omitted.

Commands:
oc new-project bare-metal-builder
oc create sa -n bare-metal-builder quay-builder
oc policy add-role-to-user -n bare-metal-builder edit system:serviceaccount:bare-metal-builder:quay-builder
create token quay-builder -n bare-metal-builder --duration 24h
oc sa get-token -n bare-metal-builder quay-builder
oc extract cm/kube-root-ca.crt -n openshift-apiserver
mv ca.crt build_cluster.crt
oc get sa openshift-apiserver-sa --namespace=openshift-apiserver -o json | jq '.secrets[] | select(.name | contains("openshift-apiserver-sa-token"))'.name
apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: quay-builder priority: null readOnlyRootFilesystem: false requiredDropCapabilities: null runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - '*' supplementalGroups: type: RunAsAny volumes: - '*' allowHostDirVolumePlugin: true allowHostIPC: true allowHostNetwork: true allowHostPID: true allowHostPorts: true allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: - '*' allowedUnsafeSysctls: - '*' defaultAddCapabilities: null fsGroup: type: RunAsAny --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: quay-builder-scc namespace: bare-metal-builder rules: - apiGroups: - security.openshift.io resourceNames: - quay-builder resources: - securitycontextconstraints verbs: - use --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: quay-builder-scc namespace: bare-metal-builder subjects: - kind: ServiceAccount name: quay-builder roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: quay-builder-scc
FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - <superusername> FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_BUILD_SUPPORT: True BUILDMAN_HOSTNAME: USD{BUILDMAN_HOSTNAME}:443 1 BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 10 ORCHESTRATOR_PREFIX: buildman/production/ ORCHESTRATOR: REDIS_HOST: <sample_redis_hostname> 2 REDIS_PASSWORD: "" REDIS_SSL: false REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: <sample_builder_namespace> 3 K8S_API_SERVER: <sample_k8s_api_server> 4 K8S_API_TLS_CA: <sample_crt_file> 5 VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 1G 6 CONTAINER_CPU_LIMITS: 300m 7 CONTAINER_MEMORY_REQUEST: 1G 8 CONTAINER_CPU_REQUEST: 300m 9 NODE_SELECTOR_LABEL_KEY: beta.kubernetes.io/instance-type NODE_SELECTOR_LABEL_VALUE: n1-standard-4 CONTAINER_RUNTIME: podman SERVICE_ACCOUNT_NAME: <sample_service_account_name> SERVICE_ACCOUNT_TOKEN: <sample_account_token> 10 QUAY_USERNAME: <quay_username> QUAY_PASSWORD: <quay_password> WORKER_IMAGE: <registry>/quay-quay-builder WORKER_TAG: <some_tag> BUILDER_VM_CONTAINER_IMAGE: quay.io/quay/quay-builder-qemu-fedoracoreos:latest SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 SSH_AUTHORIZED_KEYS: 11 - <ssh-rsa 12345 someuser@email.com> - <ssh-rsa 67890 someuser2@email.com> HTTP_PROXY: <http://10.0.0.1:80> HTTPS_PROXY: <http://10.0.0.1:80> NO_PROXY: <hostname.example.com>
kubectl get -n <namespace> route <quayregistry-name>-quay-builder -o jsonpath={.status.ingress[0].host}
BUILDMAN_HOSTNAME: <build-manager-hostname> 1 BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 600 ORCHESTRATOR: REDIS_HOST: <quay_redis_host REDIS_PASSWORD: <quay_redis_password> REDIS_SSL: true REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: builder