Chapter 4. OpenShift Service Mesh and cert-manager The cert-manager tool is a solution for X.509 certificate management on Kubernetes. It delivers a unified API to integrate applications with private or public key infrastructure (PKI), such as Vault, Google Cloud Certificate Authority Service, Let's Encrypt, and other providers. Important The cert-manager tool must be installed before you create and install your Istio resource. The cert-manager tool ensures the certificates are valid and up-to-date by attempting to renew certificates at a configured time before they expire. 4.1. About integrating Service Mesh with cert-manager and istio-csr The cert-manager tool provides integration with Istio through an external agent called istio-csr . The istio-csr agent handles certificate signing requests (CSR) from Istio proxies and the controlplane in the following ways: Verifying the identity of the workload. Creating a CSR through cert-manager for the workload. The cert-manager tool then creates a CSR to the configured CA Issuer, which signs the certificate. Note Red Hat provides support for integrating with istio-csr and cert-manager. Red Hat does not provide direct support for the istio-csr or the community cert-manager components. The use of community cert-manager shown here is for demonstration purposes only. Prerequisites One of these versions of cert-manager: Red Hat cert-manager Operator 1.10 or later community cert-manager Operator 1.11 or later cert-manager 1.11 or later Red Hat OpenShift Service Mesh 3.0 or later An IstioCNI instance is running in the cluster Istio CLI ( istioctl ) tool is installed jq is installed Helm is installed 4.2. Installing cert-manager You can integrate cert-manager with OpenShift Service Mesh by deploying istio-csr and then creating an Istio resource that uses the istio-csr agent to process workload and control plane certificate signing requests. This example creates a self-signed Issuer , but any other Issuer can be used instead. Important You must install cert-manager before installing your Istio resource. Procedure Create the istio-system namespace by running the following command: USD oc create namespace istio-system Create the root issuer by creating an Issuer object in a YAML file. Create an Issuer object similar to the following example: Example issuer.yaml file apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: selfsigned namespace: istio-system spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: istio-ca namespace: istio-system spec: isCA: true duration: 87600h # 10 years secretName: istio-ca commonName: istio-ca privateKey: algorithm: ECDSA size: 256 subject: organizations: - cluster.local - cert-manager issuerRef: name: selfsigned kind: Issuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: istio-ca namespace: istio-system spec: ca: secretName: istio-ca --- Create the objects by running the following command: USD oc apply -f issuer.yaml Wait for the istio-ca certificate to contain the "Ready" status condition by running the following command: USD oc wait --for=condition=Ready certificates/istio-ca -n istio-system Copy the istio-ca certificate to the cert-manager namespace so it can be used by istio-csr: Copy the secret to a local file by running the following command: USD oc get -n istio-system secret istio-ca -o jsonpath='{.data.tls\.crt}' | base64 -d > ca.pem Create a secret from the local certificate file in the cert-manager namespace by running the following command: USD oc create secret generic -n cert-manager istio-root-ca --from-file=ca.pem=ca.pem steps To install istio-csr , you must follow the istio-csr installation instructions for the type of update strategy you want. By default, spec.updateStrategy is set to InPlace when you create and install your Istio resource. You create and install your Istio resource after you install istio-csr . Installing the istio-csr agent by using the in place update strategy Installing the istio-csr agent by using the revision based update strategy 4.2.1. Installing the istio-csr agent by using the in place update strategy Istio resources use the in place update strategy by default. Follow this procedure if you plan to leave spec.updateStrategy as InPlace when you create and install your Istio resource. Procedure Add the Jetstack charts repository to your local Helm repository by running the following command: USD helm repo add jetstack https://charts.jetstack.io --force-update Install the istio-csr chart by running the following command: USD helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr \ --install \ --namespace cert-manager \ --wait \ --set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" \ --set "volumeMounts[0].name=root-ca" \ --set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" \ --set "volumes[0].name=root-ca" \ --set "volumes[0].secret.secretName=istio-root-ca" \ --set "app.istio.namespace=istio-system" steps Installing your Istio resource 4.2.2. Installing the istio-csr agent by using the revision based update strategy Istio resources use the in place update strategy by default. Follow this procedure if you plan to change spec.updateStrategy to RevisionBased when you create and install your Istio resource. Procedure Specify all the Istio revisions to your istio-csr deployment. See "istio-csr deployment". Add the Jetstack charts to your local Helm repository by running the following command: USD helm repo add jetstack https://charts.jetstack.io --force-update Install the istio-csr chart with your revision name by running the following command: USD helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr \ --install \ --namespace cert-manager \ --wait \ --set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" \ --set "volumeMounts[0].name=root-ca" \ --set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" \ --set "volumes[0].name=root-ca" \ --set "volumes[0].secret.secretName=istio-root-ca" \ --set "app.istio.namespace=istio-system" \ --set "app.istio.revisions={default-v1-23-0}" Note Revision names use the following format, <istio-name>-v<major_version>-<minor_version>-<patch_version> . For example: default-v1-23-0 . Additional resources istio-csr deployment steps Installing your Istio resource 4.2.3. Installing your Istio resource After you have installed istio-csr by following the procedure for either an in place or revision based update strategy, you can install the Istio resource. You need to disable Istio's built in CA server and tell istiod to use the istio-csr CA server. The istio-csr CA server issues certificates for both istiod and user workloads. Procedure Create the Istio object as shown in the following example: Example istio.yaml object apiVersion: sailoperator.io/v1alpha1 kind: Istio metadata: name: default spec: version: v1.23.0 namespace: istio-system values: global: caAddress: cert-manager-istio-csr.cert-manager.svc:443 pilot: env: ENABLE_CA_SERVER: "false" volumeMounts: - mountPath: /tmp/var/run/secrets/istiod/tls name: istio-csr-dns-cert readOnly: true Note If you installed your CSR agent with a revision based update strategy, then you need to add the following to your Istio object YAML: kind: Istio metadata: name: default spec: updateStrategy: type: RevisionBased Create the Istio resource by running the following command: USD oc apply -f istio.yaml Wait for the Istio object to become ready by running the following command: USD oc wait --for=condition=Ready istios/default -n istio-system 4.2.4. Verifying cert-manager installation You can use the sample httpbin service and sleep application to check communication between the workloads. You can also check the workload certificate of the proxy to verify that the cert-manager tool is installed correctly. Procedure Create the sample namespace by running the following command: USD oc new-project sample Find your active Istio revision by running the following command: USD oc get istiorevisions Add the injection label for your active revision to the sample namespace by running the following command: USD oc label namespace sample istio.io/rev=<your-active-revision-name> --overwrite=true Deploy the sample httpbin service by running the following command: USD oc apply -n sample -f https://raw.githubusercontent.com/openshift-service-mesh/istio/refs/heads/master/samples/httpbin/httpbin.yaml Deploy the sample sleep application by running the following command: USD oc apply -n sample -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/sleep/sleep.yaml Wait for both applications to become ready by running the following command: USD oc rollout status -n sample deployment httpbin sleep Verify that sleep application can access the httpbin service by running the following command: USD oc exec "USD(oc get pod -l app=sleep -n sample \ -o jsonpath={.items..metadata.name})" -c sleep -n sample -- \ curl http://httpbin.sample:8000/ip -s -o /dev/null \ -w "%{http_code}\n" Example of a successful output 200 Run the following command to print the workload certificate for the httpbin service and verify the output: USD istioctl proxy-config secret -n sample USD(oc get pods -n sample -o jsonpath='{.items..metadata.name}' --selector app=httpbin) -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode | openssl x509 -text -noout Example output ... Issuer: O = cert-manager + O = cluster.local, CN = istio-ca ... X509v3 Subject Alternative Name: URI:spiffe://cluster.local/ns/sample/sa/httpbin 4.3. Updating istio-csr agents with revision-based update strategies If you deployed your Istio resource using the revision based update strategy, you must pass all revisions each time you update your control plane. You must perform the update in the following order: Update the istio-csr deployment with the new revision. Update the value of Istio.spec.version parameter/field. Example update for RevisionBased control plane In this example, the controlplane is being updated from v1.23.0 to 1.23.1. Update the istio-csr deployment with the new revision by running the following command: USD helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr \ --wait \ --reuse-values \ --set "app.istio.revisions={<old_revision>,<new_revision>}" where: old_revision Specifies the old revision in the <istio-name>-v<major_version>-<minor_version>-<patch_version> format. For example: default-v1-23-0 . new_revision Specfies the new revision in the <istio-name>-v<major_version>-<minor_version>-<patch_version> format. For example: default-v1-23-1 . Update the istio.spec.version in the Istio object similar to the following example: Example istio.yaml file apiVersion: sailoperator.io/v1alpha1 kind: Istio metadata: name: default spec: version: <new_revision> 1 1 Update to the new revision prefixed with the letter v , such as v1.23.1 Remove the old revision from your istio-csr deployment by running the following command: helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr \ --install \ --namespace cert-manager \ --wait \ --reuse-values \ --set "app.istio.revisions={default-v1-23-1}"

Commands:
oc create namespace istio-system
apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: selfsigned namespace: istio-system spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: istio-ca namespace: istio-system spec: isCA: true duration: 87600h # 10 years secretName: istio-ca commonName: istio-ca privateKey: algorithm: ECDSA size: 256 subject: organizations: - cluster.local - cert-manager issuerRef: name: selfsigned kind: Issuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: istio-ca namespace: istio-system spec: ca: secretName: istio-ca ---
oc apply -f issuer.yaml
oc wait --for=condition=Ready certificates/istio-ca -n istio-system
oc get -n istio-system secret istio-ca -o jsonpath='{.data.tls\.crt}' | base64 -d > ca.pem
oc create secret generic -n cert-manager istio-root-ca --from-file=ca.pem=ca.pem
helm repo add jetstack https://charts.jetstack.io --force-update
helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr --install --namespace cert-manager --wait --set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" --set "volumeMounts[0].name=root-ca" --set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" --set "volumes[0].name=root-ca" --set "volumes[0].secret.secretName=istio-root-ca" --set "app.istio.namespace=istio-system"
helm repo add jetstack https://charts.jetstack.io --force-update
helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr --install --namespace cert-manager --wait --set "app.tls.rootCAFile=/var/run/secrets/istio-csr/ca.pem" --set "volumeMounts[0].name=root-ca" --set "volumeMounts[0].mountPath=/var/run/secrets/istio-csr" --set "volumes[0].name=root-ca" --set "volumes[0].secret.secretName=istio-root-ca" --set "app.istio.namespace=istio-system" --set "app.istio.revisions={default-v1-23-0}"
apiVersion: sailoperator.io/v1alpha1 kind: Istio metadata: name: default spec: version: v1.23.0 namespace: istio-system values: global: caAddress: cert-manager-istio-csr.cert-manager.svc:443 pilot: env: ENABLE_CA_SERVER: "false" volumeMounts: - mountPath: /tmp/var/run/secrets/istiod/tls name: istio-csr-dns-cert readOnly: true
kind: Istio metadata: name: default spec: updateStrategy: type: RevisionBased
oc apply -f istio.yaml
oc wait --for=condition=Ready istios/default -n istio-system
oc new-project sample
oc get istiorevisions
oc label namespace sample istio.io/rev=<your-active-revision-name> --overwrite=true
oc apply -n sample -f https://raw.githubusercontent.com/openshift-service-mesh/istio/refs/heads/master/samples/httpbin/httpbin.yaml
oc apply -n sample -f https://raw.githubusercontent.com/istio/istio/refs/heads/master/samples/sleep/sleep.yaml
oc rollout status -n sample deployment httpbin sleep
oc exec "USD(oc get pod -l app=sleep -n sample -o jsonpath={.items..metadata.name})" -c sleep -n sample -- curl http://httpbin.sample:8000/ip -s -o /dev/null -w "%{http_code}\n"
200
istioctl proxy-config secret -n sample USD(oc get pods -n sample -o jsonpath='{.items..metadata.name}' --selector app=httpbin) -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode | openssl x509 -text -noout
Issuer: O = cert-manager + O = cluster.local, CN = istio-ca X509v3 Subject Alternative Name: URI:spiffe://cluster.local/ns/sample/sa/httpbin
helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr --wait --reuse-values --set "app.istio.revisions={<old_revision>,<new_revision>}"
apiVersion: sailoperator.io/v1alpha1 kind: Istio metadata: name: default spec: version: <new_revision> 1
helm upgrade cert-manager-istio-csr jetstack/cert-manager-istio-csr --install --namespace cert-manager --wait --reuse-values --set "app.istio.revisions={default-v1-23-1}"