Chapter 14. Tutorial: Dynamically issuing certificates using the cert-manager Operator on ROSA While wildcard certificates provide simplicity by securing all first-level subdomains of a given domain with a single certificate, other use cases can require the use of individual certificates per domain. Learn how to use the cert-manager Operator for Red Hat OpenShift and Let's Encrypt to dynamically issue certificates for routes created using a custom domain. 14.1. Prerequisites A ROSA cluster (HCP or Classic) A user account with cluster-admin privileges The OpenShift CLI ( oc ) The Amazon Web Services (AWS) CLI ( aws ) A unique domain, such as *.apps.example.com An Amazon Route 53 public hosted zone for the above domain 14.2. Setting up your environment Configure the following environment variables: USD export DOMAIN=apps.example.com 1 USD export EMAIL=email@example.com 2 USD export AWS_PAGER="" USD export CLUSTER=USD(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}" | sed 's/-[a-z0-9]\{5\}USD//') USD export OIDC_ENDPOINT=USD(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed 's|^https://||') USD export REGION=USD(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") USD export AWS_ACCOUNT_ID=USD(aws sts get-caller-identity --query Account --output text) USD export SCRATCH="/tmp/USD{CLUSTER}/dynamic-certs" USD mkdir -p USD{SCRATCH} 1 Replace with the custom domain you want to use for the IngressController . 2 Replace with the e-mail you want Let's Encrypt to use to send notifications about your certificates. Ensure all fields output correctly before moving to the section: USD echo "Cluster: USD{CLUSTER}, Region: USD{REGION}, OIDC Endpoint: USD{OIDC_ENDPOINT}, AWS Account ID: USD{AWS_ACCOUNT_ID}" Note The "Cluster" output from the command may be the name of your cluster, the internal ID of your cluster, or the cluster's domain prefix. If you prefer to use another identifier, you can manually set this value by running the following command: USD export CLUSTER=my-custom-value 14.3. Preparing your AWS account When cert-manager requests a certificate from Let's Encrypt (or another ACME certificate issuer), Let's Encrypt servers validate that you control the domain name in that certificate using challenges . For this tutorial, you are using a DNS-01 challenge that proves that you control the DNS for your domain name by putting a specific value in a TXT record under that domain name. This is all done automatically by cert-manager. To allow cert-manager permission to modify the Amazon Route 53 public hosted zone for your domain, you need to create an Identity Access Management (IAM) role with specific policy permissions and a trust relationship to allow access to the pod. The public hosted zone that is used in this tutorial is in the same AWS account as the ROSA cluster. If your public hosted zone is in a different account, a few additional steps for Cross Account Access are required. Retrieve the Amazon Route 53 public hosted zone ID: Note This command looks for a public hosted zone that matches the custom domain you specified earlier as the DOMAIN environment variable. You can manually specify the Amazon Route 53 public hosted zone by running export ZONE_ID=<zone_ID> , replacing <zone_ID> with your specific Amazon Route 53 public hosted zone ID. USD export ZONE_ID=USD(aws route53 list-hosted-zones-by-name --output json \ --dns-name "USD{DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///') Create an AWS IAM policy document for the cert-manager Operator that provides the ability to update only the specified public hosted zone: USD cat <<EOF > "USD{SCRATCH}/cert-manager-policy.json" { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "route53:GetChange", "Resource": "arn:aws:route53:::change/*" }, { "Effect": "Allow", "Action": [ "route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets" ], "Resource": "arn:aws:route53:::hostedzone/USD{ZONE_ID}" }, { "Effect": "Allow", "Action": "route53:ListHostedZonesByName", "Resource": "*" } ] } EOF Create the IAM policy using the file you created in the step: USD POLICY_ARN=USD(aws iam create-policy --policy-name "USD{CLUSTER}-cert-manager-policy" \ --policy-document file://USD{SCRATCH}/cert-manager-policy.json \ --query 'Policy.Arn' --output text) Create an AWS IAM trust policy for the cert-manager Operator: USD cat <<EOF > "USD{SCRATCH}/trust-policy.json" { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "USD{OIDC_ENDPOINT}:sub": "system:serviceaccount:cert-manager:cert-manager" } }, "Principal": { "Federated": "arn:aws:iam::USDAWS_ACCOUNT_ID:oidc-provider/USD{OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF Create an IAM role for the cert-manager Operator using the trust policy you created in the step: USD ROLE_ARN=USD(aws iam create-role --role-name "USD{CLUSTER}-cert-manager-operator" \ --assume-role-policy-document "file://USD{SCRATCH}/trust-policy.json" \ --query Role.Arn --output text) Attach the permissions policy to the role: USD aws iam attach-role-policy --role-name "USD{CLUSTER}-cert-manager-operator" \ --policy-arn USD{POLICY_ARN} 14.4. Installing the cert-manager Operator Create a project to install the cert-manager Operator into: USD oc new-project cert-manager-operator Important Do not attempt to use more than one cert-manager Operator in your cluster. If you have a community cert-manager Operator installed in your cluster, you must uninstall it before installing the cert-manager Operator for Red Hat OpenShift. Install the cert-manager Operator for Red Hat OpenShift: USD cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-cert-manager-operator-group namespace: cert-manager-operator spec: targetNamespaces: - cert-manager-operator --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-cert-manager-operator namespace: cert-manager-operator spec: channel: stable-v1 installPlanApproval: Automatic name: openshift-cert-manager-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF Note It takes a few minutes for this Operator to install and complete its set up. Verify that the cert-manager Operator is running: USD oc -n cert-manager-operator get pods Example output NAME READY STATUS RESTARTS AGE cert-manager-operator-controller-manager-84b8799db5-gv8mx 2/2 Running 0 12s Annotate the service account used by the cert-manager pods with the AWS IAM role you created earlier: USD oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn=USD{ROLE_ARN} Restart the existing cert-manager controller pod by running the following command: USD oc -n cert-manager delete pods -l app.kubernetes.io/name=cert-manager Patch the Operator's configuration to use external nameservers to prevent DNS-01 challenge resolution issues: USD oc patch certmanager.operator.openshift.io/cluster --type merge \ -p '{"spec":{"controllerConfig":{"overrideArgs":["--dns01-recursive-nameservers-only","--dns01-recursive-nameservers=1.1.1.1:53"]}}}' Create a ClusterIssuer resource to use Let's Encrypt by running the following command: USD cat << EOF | oc apply -f - apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-production spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: USD{EMAIL} # This key doesn't exist, cert-manager creates it privateKeySecretRef: name: prod-letsencrypt-issuer-account-key solvers: - dns01: route53: hostedZoneID: USD{ZONE_ID} region: USD{REGION} secretAccessKeySecretRef: name: '' EOF Verify the ClusterIssuer resource is ready: USD oc get clusterissuer.cert-manager.io/letsencrypt-production Example output NAME READY AGE letsencrypt-production True 47s 14.5. Creating a custom domain Ingress Controller Create and configure a certificate resource to provision a certificate for the custom domain Ingress Controller: Note The following example uses a single domain certificate. SAN and wildcard certificates are also supported. USD cat << EOF | oc apply -f - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: custom-domain-ingress-cert namespace: openshift-ingress spec: secretName: custom-domain-ingress-cert-tls issuerRef: name: letsencrypt-production kind: ClusterIssuer commonName: "USD{DOMAIN}" dnsNames: - "USD{DOMAIN}" EOF Verify the certificate has been issued: Note It takes a few minutes for this certificate to be issued by Let's Encrypt. If it takes longer than 5 minutes, run oc -n openshift-ingress describe certificate.cert-manager.io/custom-domain-ingress-cert to see any issues reported by cert-manager. USD oc -n openshift-ingress get certificate.cert-manager.io/custom-domain-ingress-cert Example output NAME READY SECRET AGE custom-domain-ingress-cert True custom-domain-ingress-cert-tls 9m53s Create a new IngressController resource: USD cat << EOF | oc apply -f - apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: custom-domain-ingress namespace: openshift-ingress-operator spec: domain: USD{DOMAIN} defaultCertificate: name: custom-domain-ingress-cert-tls endpointPublishingStrategy: loadBalancer: dnsManagementPolicy: Unmanaged providerParameters: aws: type: NLB type: AWS scope: External type: LoadBalancerService EOF Warning This IngressController example will create an internet accessible Network Load Balancer (NLB) in your AWS account. To provision an internal NLB instead, set the .spec.endpointPublishingStrategy.loadBalancer.scope parameter to Internal before creating the IngressController resource. Verify that your custom domain IngressController has successfully created an external load balancer: USD oc -n openshift-ingress get service/router-custom-domain-ingress Example output NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE router-custom-domain-ingress LoadBalancer 172.30.174.34 a309962c3bd6e42c08cadb9202eca683-1f5bbb64a1f1ec65.elb.us-east-1.amazonaws.com 80:31342/TCP,443:31821/TCP 7m28s Prepare a document with the necessary DNS changes to enable DNS resolution for your custom domain Ingress Controller: USD INGRESS=USD(oc -n openshift-ingress get service/router-custom-domain-ingress -ojsonpath="{.status.loadBalancer.ingress[0].hostname}") USD cat << EOF > "USD{SCRATCH}/create-cname.json" { "Comment":"Add CNAME to custom domain endpoint", "Changes":[{ "Action":"CREATE", "ResourceRecordSet":{ "Name": "*.USD{DOMAIN}", "Type":"CNAME", "TTL":30, "ResourceRecords":[{ "Value": "USD{INGRESS}" }] } }] } EOF Submit your changes to Amazon Route 53 for propagation: USD aws route53 change-resource-record-sets \ --hosted-zone-id USD{ZONE_ID} \ --change-batch file://USD{SCRATCH}/create-cname.json Note While the wildcard CNAME record avoids the need to create a new record for every new application you deploy using the custom domain Ingress Controller, the certificate that each of these applications use is not a wildcard certificate. 14.6. Configuring dynamic certificates for custom domain routes Now you can expose cluster applications on any first-level subdomains of the specified domain, but the connection will not be secured with a TLS certificate that matches the domain of the application. To ensure these cluster applications have valid certificates for each domain name, configure cert-manager to dynamically issue a certificate to every new route created under this domain. Create the necessary OpenShift resources cert-manager requires to manage certificates for OpenShift routes. This step creates a new deployment (and therefore a pod) that specifically monitors annotated routes in the cluster. If the issuer-kind and issuer-name annotations are found in a new route, it requests the Issuer (ClusterIssuer in this case) for a new certificate that is unique to this route and which will honor the hostname that was specified while creating the route. Note If the cluster does not have access to GitHub, you can save the raw contents locally and run oc apply -f localfilename.yaml -n cert-manager . USD oc -n cert-manager apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml The following additional OpenShift resources are also created in this step: ClusterRole - grants permissions to watch and update the routes across the cluster ServiceAccount - uses permissions to run the newly created pod ClusterRoleBinding - binds these two resources Ensure that the new cert-manager-openshift-routes pod is running successfully: USD oc -n cert-manager get pods Example result NAME READY STATUS RESTARTS AGE cert-manager-866d8f788c-9kspc 1/1 Running 0 4h21m cert-manager-cainjector-6885c585bd-znws8 1/1 Running 0 4h41m cert-manager-openshift-routes-75b6bb44cd-f8kd5 1/1 Running 0 6s cert-manager-webhook-8498785dd9-bvfdf 1/1 Running 0 4h41m 14.7. Deploying a sample application Now that dynamic certificates are configured, you can deploy a sample application to confirm that certificates are provisioned and trusted when you expose a new route. Create a new project for your sample application: USD oc new-project hello-world Deploy a hello world application: USD oc -n hello-world new-app --image=docker.io/openshift/hello-openshift Create a route to expose the application from outside the cluster: USD oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls --hostname hello.USD{DOMAIN} Verify the certificate for the route is untrusted: USD curl -I https://hello.USD{DOMAIN} Example output curl: (60) SSL: no alternative certificate subject name matches target host name 'hello.example.com' More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. Annotate the route to trigger cert-manager to provision a certificate for the custom domain: USD oc -n hello-world annotate route hello-openshift-tls cert-manager.io/issuer-kind=ClusterIssuer cert-manager.io/issuer-name=letsencrypt-production Note It takes 2-3 minutes for the certificate to be created. The renewal of the certificate will automatically be managed by the cert-manager Operator as it approaches expiration. Verify the certificate for the route is now trusted: USD curl -I https://hello.USD{DOMAIN} Example output HTTP/2 200 date: Thu, 05 Oct 2023 23:45:33 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: 52e4465485b6fb4f8a1b1bed128d0f3b=68676068bb32d24f0f558f094ed8e4d7; path=/; HttpOnly; Secure; SameSite=None cache-control: private 14.8. Troubleshooting dynamic certificate provisioning Note The validation process usually takes 2-3 minutes to complete while creating certificates. If annotating your route does not trigger certificate creation during the certificate create step, run oc describe against each of the certificate , certificaterequest , order , and challenge resources to view the events or reasons that can help identify the cause of the issue. USD oc get certificate,certificaterequest,order,challenge For troubleshooting, you can refer to this helpful guide in debugging certificates . You can also use the cmctl CLI tool for various certificate management activities, such as checking the status of certificates and testing renewals.

Commands:
export DOMAIN=apps.example.com 1 export EMAIL=email@example.com 2 export AWS_PAGER="" export CLUSTER=USD(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}" | sed 's/-[a-z0-9]\{5\}USD//') export OIDC_ENDPOINT=USD(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed 's|^https://||') export REGION=USD(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") export AWS_ACCOUNT_ID=USD(aws sts get-caller-identity --query Account --output text) export SCRATCH="/tmp/USD{CLUSTER}/dynamic-certs" mkdir -p USD{SCRATCH}
echo "Cluster: USD{CLUSTER}, Region: USD{REGION}, OIDC Endpoint: USD{OIDC_ENDPOINT}, AWS Account ID: USD{AWS_ACCOUNT_ID}"
export CLUSTER=my-custom-value
export ZONE_ID=USD(aws route53 list-hosted-zones-by-name --output json --dns-name "USD{DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')
cat <<EOF > "USD{SCRATCH}/cert-manager-policy.json" { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "route53:GetChange", "Resource": "arn:aws:route53:::change/*" }, { "Effect": "Allow", "Action": [ "route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets" ], "Resource": "arn:aws:route53:::hostedzone/USD{ZONE_ID}" }, { "Effect": "Allow", "Action": "route53:ListHostedZonesByName", "Resource": "*" } ] } EOF
POLICY_ARN=USD(aws iam create-policy --policy-name "USD{CLUSTER}-cert-manager-policy" --policy-document file://USD{SCRATCH}/cert-manager-policy.json --query 'Policy.Arn' --output text)
cat <<EOF > "USD{SCRATCH}/trust-policy.json" { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "USD{OIDC_ENDPOINT}:sub": "system:serviceaccount:cert-manager:cert-manager" } }, "Principal": { "Federated": "arn:aws:iam::USDAWS_ACCOUNT_ID:oidc-provider/USD{OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF
ROLE_ARN=USD(aws iam create-role --role-name "USD{CLUSTER}-cert-manager-operator" --assume-role-policy-document "file://USD{SCRATCH}/trust-policy.json" --query Role.Arn --output text)
aws iam attach-role-policy --role-name "USD{CLUSTER}-cert-manager-operator" --policy-arn USD{POLICY_ARN}
oc new-project cert-manager-operator
cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-cert-manager-operator-group namespace: cert-manager-operator spec: targetNamespaces: - cert-manager-operator --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-cert-manager-operator namespace: cert-manager-operator spec: channel: stable-v1 installPlanApproval: Automatic name: openshift-cert-manager-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
oc -n cert-manager-operator get pods
NAME READY STATUS RESTARTS AGE cert-manager-operator-controller-manager-84b8799db5-gv8mx 2/2 Running 0 12s
oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn=USD{ROLE_ARN}
oc -n cert-manager delete pods -l app.kubernetes.io/name=cert-manager
oc patch certmanager.operator.openshift.io/cluster --type merge -p '{"spec":{"controllerConfig":{"overrideArgs":["--dns01-recursive-nameservers-only","--dns01-recursive-nameservers=1.1.1.1:53"]}}}'
cat << EOF | oc apply -f - apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-production spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: USD{EMAIL} # This key doesn't exist, cert-manager creates it privateKeySecretRef: name: prod-letsencrypt-issuer-account-key solvers: - dns01: route53: hostedZoneID: USD{ZONE_ID} region: USD{REGION} secretAccessKeySecretRef: name: '' EOF
oc get clusterissuer.cert-manager.io/letsencrypt-production
NAME READY AGE letsencrypt-production True 47s
cat << EOF | oc apply -f - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: custom-domain-ingress-cert namespace: openshift-ingress spec: secretName: custom-domain-ingress-cert-tls issuerRef: name: letsencrypt-production kind: ClusterIssuer commonName: "USD{DOMAIN}" dnsNames: - "USD{DOMAIN}" EOF
oc -n openshift-ingress get certificate.cert-manager.io/custom-domain-ingress-cert
NAME READY SECRET AGE custom-domain-ingress-cert True custom-domain-ingress-cert-tls 9m53s
cat << EOF | oc apply -f - apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: custom-domain-ingress namespace: openshift-ingress-operator spec: domain: USD{DOMAIN} defaultCertificate: name: custom-domain-ingress-cert-tls endpointPublishingStrategy: loadBalancer: dnsManagementPolicy: Unmanaged providerParameters: aws: type: NLB type: AWS scope: External type: LoadBalancerService EOF
oc -n openshift-ingress get service/router-custom-domain-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE router-custom-domain-ingress LoadBalancer 172.30.174.34 a309962c3bd6e42c08cadb9202eca683-1f5bbb64a1f1ec65.elb.us-east-1.amazonaws.com 80:31342/TCP,443:31821/TCP 7m28s
INGRESS=USD(oc -n openshift-ingress get service/router-custom-domain-ingress -ojsonpath="{.status.loadBalancer.ingress[0].hostname}") cat << EOF > "USD{SCRATCH}/create-cname.json" { "Comment":"Add CNAME to custom domain endpoint", "Changes":[{ "Action":"CREATE", "ResourceRecordSet":{ "Name": "*.USD{DOMAIN}", "Type":"CNAME", "TTL":30, "ResourceRecords":[{ "Value": "USD{INGRESS}" }] } }] } EOF
aws route53 change-resource-record-sets --hosted-zone-id USD{ZONE_ID} --change-batch file://USD{SCRATCH}/create-cname.json
oc -n cert-manager apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml
oc -n cert-manager get pods
NAME READY STATUS RESTARTS AGE cert-manager-866d8f788c-9kspc 1/1 Running 0 4h21m cert-manager-cainjector-6885c585bd-znws8 1/1 Running 0 4h41m cert-manager-openshift-routes-75b6bb44cd-f8kd5 1/1 Running 0 6s cert-manager-webhook-8498785dd9-bvfdf 1/1 Running 0 4h41m
oc new-project hello-world
oc -n hello-world new-app --image=docker.io/openshift/hello-openshift
oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls --hostname hello.USD{DOMAIN}
curl -I https://hello.USD{DOMAIN}
curl: (60) SSL: no alternative certificate subject name matches target host name 'hello.example.com' More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.
oc -n hello-world annotate route hello-openshift-tls cert-manager.io/issuer-kind=ClusterIssuer cert-manager.io/issuer-name=letsencrypt-production
curl -I https://hello.USD{DOMAIN}
HTTP/2 200 date: Thu, 05 Oct 2023 23:45:33 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: 52e4465485b6fb4f8a1b1bed128d0f3b=68676068bb32d24f0f558f094ed8e4d7; path=/; HttpOnly; Secure; SameSite=None cache-control: private
oc get certificate,certificaterequest,order,challenge