Chapter 7. Serving transport encryption You can enable OpenShift Serverless Serving transport encryption to allow transporting data over secured and encrypted HTTPS connections using TLS. Important OpenShift Serverless Serving transport encryption is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Important Serving Transport Encryption is only available for Kourier as an ingress layer. For Red Hat OpenShift Service Mesh, use the service mesh mTLS capabilities to ensure encrypted traffic. 7.1. Overview of Serving transport encryption There are three parts to OpenShift Serverless Serving transport encryption: External domain encryption Transport encryption on the ingress layer external to the cluster. For example, a cluster-external domain, such as myapp-<namespace>.example.com . Cluster-local encryption Transport encryption on the ingress layer internal to the cluster. For example, cluster-local domains, such as myapp.<namespace>.svc.cluster.local . System-internal encryption Transport encryption between the ingress-gateway , activator , and queue-proxy Knative internal components. Important Control-plane traffic, including Kubernetes PreStopHooks, metadata, and metrics, contains no user data and is not encrypted. The different parts are independent of each other and can be enabled and disabled individually. They can use the same or different Certificate Authorities (CAs) to sign the necessary certificates. Note For diagrams illustrating transport encryption, see OpenShift Serverless Serving Transport Encryption . 7.1.1. External domain encryption The transport encryption for external domains is handled by the ingress layer of the cluster, which is either the OpenShift Container Platform ingress or Red Hat OpenShift Service Mesh. 7.1.2. Cluster-local encryption Cluster-local encryption enables transport encryption for cluster-local domains. It has the following properties: The Certificate Common Name (CN) or Subject Alternative Name (SAN) contains the cluster-local domains of a Knative Service, for example myapp.namespace.svc.cluster.local , myapp.namespace.svc , myapp.namespace . The cluster-local endpoint of the ingress-controller component uses SNI to select the certificates. To create the certificates, Knative relies on cert-manager , which needs to be installed and configured for the feature to work. For more information, see cert-manager Operator for Red Hat OpenShift . Important The caller must trust the CA that signed the cluster-local certificates. This is out of the scope of OpenShift Serverless. 7.1.3. System-internal encryption System-internal encryption enables transport encryption for the ingress-gateway , activator , and queue-proxy Knative internal components. These components host TLS endpoints when this configuration is used. The following prerequisites must be satisfied to use this feature: For OpenShift Serverless to get the certificates, cert-manager must be installed and configured. Specific SANs are used to verify each connection. Each component must trust the CA that signed the certificates. To satisfy this requirement, OpenShift Serverless system components consume and trust a bundle of CAs. The CA bundle must be provided by the cluster administrator. 7.2. Choice of a certificate issuer Issuers refer to cert-manager issuers and cluster issuers. They represent certificate authorities (CAs) that can generate signed certificates by honoring certificate signing requests. For more information, see cert-manager documentation on issuers . Depending on the encryption features that you use, OpenShift Serverless requires your certificate issuer to be able to sign certain certificates. To identify your certificate issuer, refer to the list of cert-manager integrations , which contains examples for the following: A custom CA stored in a Kubernetes secret HTTP-01 challenges DNS-01 challenges Self-signed issuers 7.2.1. Compatible certificate issuers Not all issuer types work for each Knative Serving encryption feature. For cluster-local encryption, the issuer must be able to sign certificates for the following cluster-local domain types: myapp.<namespace> myapp.<namespace>.svc myapp.<namespace>.svc.cluster.local As the CA usually is not within the cluster, verification using the Automated Certificate Management Environment (ACME) protocol (DNS01/HTTP01) is not possible. You can use an issuer that allows creating these certificates, such as the cert-manager CA issuer. For system-internal encryption, the issuer must be able to sign certificates with the following Subject Alternative Names (SANs): kn-routing names of format kn-user-<namespace> , where <namespace> is a namespace where Knative Services are created data-plane.knative.dev Knative requires these SANs to verify connections between the internal components. Because this is not possible using the ACME protocol (DNS01/HTTP01), you must configure an issuer that allows creating these certificates, for example, cert-manager CA issuer. 7.3. Setting up OpenShift Serverless transport encryption Prerequisites You have access to an OpenShift Container Platform account with cluster administrator access. Install the {oc-first}. Install the cert-manager Operator for Red Hat OpenShift. Install the OpenShift Serverless Operator. Important If you install the OpenShift Serverless Operator before installing the cert-manager Operator for Red Hat OpenShift, you must restart the controller and activator deployments in the knative-serving namespace. Failure to restart these deployments prevents Knative from creating the necessary cert-manager resources, which results in pending Knative Services and prevents enabling the Knative Serving cert-manager integration. 7.3.1. Configuring a SelfSigned cluster issuer The following procedure uses a SelfSigned issuer as the root certificate. For information about the implications and limitations of this method, see the SelfSigned cert-manager documentation . If you manage your own company-specific Private Key Infrastructure (PKI), use the CA issuer. For more information, see cert-manager documentation on CA issuers . Procedure Create a SelfSigned ClusterIssuer custom resource (CR): Example ClusterIssuer CR apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: knative-serving-selfsigned-issuer spec: selfSigned: {} Apply the ClusterIssuer CR by running the following command: USD oc apply -f <filename> Create a root certificate that refers to the ClusterIssuer CR: Example root certificate apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: knative-serving-selfsigned-ca namespace: cert-manager 1 spec: secretName: knative-serving-ca 2 isCA: true commonName: selfsigned-ca privateKey: algorithm: ECDSA size: 256 issuerRef: name: knative-serving-selfsigned-issuer kind: ClusterIssuer group: cert-manager.io 1 The cert-manager Operator for Red Hat OpenShift namespace, cert-manager by default. 2 Secret name later used for the ClusterIssuer CR for Knative Serving. Apply the Certificate CR by running the following command: USD oc apply -f <filename> 7.3.2. Creating a ClusterIssuer to be used by Serving To enable the use of certificates by Serving, you must create a cluster issuer. Procedure Create the knative-serving-ca-issuer ClusterIssuer for Serving: apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: knative-serving-ca-issuer spec: ca: secretName: knative-serving-ca 1 1 Secret name in the cert-manager Operator for Red Hat OpenShift namespace ( cert-manager by default) containing the certificate that can be used by OpenShift Serverless Serving components for new certificates. Apply the ClusterIssuer resource by running the following command: USD oc apply -f <filename> 7.3.3. Configuring transport encryption Configuring transport encryption consists of two parts: Specifying the ClusterIssuer issuer to use: clusterLocalIssuerRef : issuer for cluster-local-domain certificates used for ingress. systemInternalIssuerRef : issuer for certificates for system-internal-tls certificates used by Knative internal components. Specifying transport encryption features to use: cluster-local-domain-tls : Enables the transport encryption feature for cluster-local domains system-internal-tls : Enables the transport encryption feature for OpenShift Serverless Serving internal components. Procedure Enable transport encryption in the KnativeServing resource: apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: ... config: certmanager: clusterLocalIssuerRef: | kind: ClusterIssuer name: knative-serving-ca-issuer 1 systemInternalIssuerRef: | kind: ClusterIssuer name: knative-serving-ca-issuer 2 network: cluster-local-domain-tls: Enabled 3 system-internal-tls: Enabled 4 1 Define the cluster issuer for each feature. The same or individual cluster issuers can be used. 2 Define the cluster issuer. 3 Enable the cluster-local-domain-tls feature. This and other features can be enabled or disabled individually. 4 Enable the system-internal-tls feature. Apply the KnativeServing resource by running the following command: USD oc apply -f <filename> Optionally, change the defaultCertificate value in the Ingress Controller: apiVersion: operator.openshift.io/v1 kind: IngressController ... spec: defaultCertificate: name: ca-ingress-cert If you changed the defaultCertificate value, you must specify the custom certificate name in the openshift-ingress-default-certificate field in the KnativeServing custom resource. For example, if the custom certificate name is ca-ingress-cert , add the following configuration: ... spec: config: network: system-internal-tls: Enabled openshift-ingress-default-certificate: "ca-ingress-cert" ... If you enabled cluster-local-domain-tls or system-internal-tls , restart the Controller component by running the following command. Important When either the cluster-local-domain-tls or the system-internal-tls feature is enabled, you must restart the Controller component to enable the Knative Serving cert-manager integration. USD oc rollout restart deploy/controller -n knative-serving If you enabled system-internal-tls , restart the Activator component by running the following command. Important When the system-internal-tls feature is activated, you must restart the Activator component to reconfigure its internal web server, as this is not possible during runtime. USD oc rollout restart deploy/activator -n knative-serving 7.4. Trust configuration When you enable any of the transport encryption features, you must make sure that all clients calling trust the Certificate Authority (CA) issuing the certificates used for the transport encryption. There are multiple places where trust must be ensured: Cluster external client, such as Browser or other application. This is out of the scope of OpenShift Serverless. OpenShift Serverless system components, such as Activator, Queue-Proxy, and Ingress-Controller. Cluster internal client, such as a Knative Service or other workload. 7.4.1. Trust configuration for OpenShift Serverless Serving components and Knative Services To ensure that OpenShift Serverless Serving components and Knative Services trust the CA that issues certificates, you can create a ConfigMap in the following namespaces with the label networking.knative.dev/trust-bundle: true : knative-serving for the system components of OpenShift Serverless Serving. knative-serving-ingress for the ingress layer of OpenShift Serverless Serving. istio-system or your own Service Mesh namespace when the Service Mesh integration is enabled. Knative reads all data keys in ConfigMaps with this label, regardless of the name. One key can contain one or multiple CAs or intermediate certificates. If they are valid, they are added to the trust store of the Knative components. This is an example ConfigMap: apiVersion: v1 data: cacerts.pem: | 1 -----BEGIN CERTIFICATE----- MIIDDTCCAfWgAwIBAgIQMQuip05h7NLQq2TB+j9ZmTANBgkqhkiG9w0BAQsFADAW MRQwEgYDVQQDEwtrbmF0aXZlLmRldjAeFw0yMzExMjIwOTAwNDhaFw0yNDAyMjAw OTAwNDhaMBYxFDASBgNVBAMTC2tuYXRpdmUuZGV2MIIBIjANBgkqhkiG9w0BAQEF AAOCAQ8AMIIBCgKCAQEA3clC3CV7sy0TpUKNuTku6QmP9z8JUCbLCPCLACCUc1zG FEokqOva6TakgvAntXLkB3TEsbdCJlNm6qFbbko6DBfX6rEggqZs40x3/T+KH66u 4PvMT3fzEtaMJDK/KQOBIvVHrKmPkvccUYK/qWY7rgBjVjjLVSJrCn4dKaEZ2JNr Fd0KNnaaW/dP9/FvviLqVJvHnTMHH5qyRRr1kUGTrc8njRKwpHcnUdauiDoWRKxo Zlyy+MhQfdbbyapX984WsDjCvrDXzkdGgbRNAf+erl6yUm6pHpQhyFFo/zndx6Uq QXA7jYvM2M3qCnXmaFowidoLDsDyhwoxD7WT8zur/QIDAQABo1cwVTAOBgNVHQ8B Af8EBAMCAgQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB/zAd BgNVHQ4EFgQU7p4VuECNOcnrP9ulOjc4J37Q2VUwDQYJKoZIhvcNAQELBQADggEB AAv26Vnk+ptQrppouF7yHV8fZbfnehpm07HIZkmnXO2vAP+MZJDNrHjy8JAVzXjt +OlzqAL0cRQLsUptB0btoJuw23eq8RXgJo05OLOPQ2iGNbAATQh2kLwBWd/CMg+V KJ4EIEpF4dmwOohsNR6xa/JoArIYH0D7gh2CwjrdGZr/tq1eMSL+uZcuX5OiE44A 2oXF9/jsqerOcH7QUMejSnB8N7X0LmUvH4jAesQgr7jo1JTOBs7GF6wb+U76NzFa 8ms2iAWhoplQ+EHR52wffWb0k6trXspq4O6v/J+nq9Ky3vC36so+G1ZFkMhCdTVJ ZmrBsSMWeT2l07qeei2UFRU= -----END CERTIFICATE----- kind: ConfigMap metadata: labels: networking.knative.dev/trust-bundle: "true" name: knative-bundle 2 namespace: knative-serving 1 Serving components trust all keys containing valid PEM-encoded CA bundles. 2 You can use an arbitrary name. Important When a CA bundle ConfigMap is created or updated, the Serving components automatically pick them up and add the CAs or intermediate certificates to their CA trust store. The trust store is refreshed for every new HTTP connection. 7.4.2. Trust configuration on your custom workload As OpenShift Serverless Serving does not control all workloads and managing trust is highly dependent on your runtime and language, custom workloads are out of the scope of OpenShift Serverless. The following are other options for custom workloads: Adding a CA bundle to a Container image on build-time. Note that this complicates CA rotation, as you must rebuild and redeploy every application when the CA rotates. Mounting a CA bundle to the filesystem, such as from a Secret or ConfigMap, and making sure your application uses it to verify TLS connections. Reading a CA bundle from an environment variable and making sure that your application uses it to verify TLS connections. Accessing a CA bundle from a secret or ConfigMap using Kubernetes API and making sure your application uses it to verify TLS connections. 7.5. Ensuring seamless CA rotation Ensuring seamless CA rotation is essential to avoid service downtime, or to deal with an emergency. Procedure Create a new CA certificate. Add the public key of the new CA certificate to the CA trust bundles as described in the "Trust configuration for OpenShift Serverless Operator Serving components and Knative Services" section. Retain the public key of the existing CA. Ensure that all clients have consumed the latest set of CA trust bundles. OpenShift Serverless Serving components will automatically reload the changed CA trust bundles. If you have custom workload consuming trust bundles, reload or restart them accordingly. Update the knative-serving-ca-issuer cluster issuer to reference the secret containing the new CA certificate. Either wait for cert-manager to renew all your certificates or enforce it to renew all the certificates. For more information, see the cert-manager documentation. Once the CA rotation is fully completed, you can remove the public key of the old CA from the trust bundle configmap. Allow enough time for all components to apply the changes. Additional resources cert-manager documentation on reissuance triggered by user actions 7.6. Verifying transport encryption is enabled Procedure Create a Knative Service: apiVersion: serving.knative.dev/v1 kind: Service metadata: name: test-webapp namespace: test-namespace spec: template: spec: containers: - image: docker.io/openshift/hello-openshift env: - name: RESPONSE value: "Hello Serverless!" Apply the Knative Service YAML by running the following command: USD oc apply -f <filename> Examine status of the Knative Service: Example command USD oc get ksvc -n test-namespace -o yaml Example output apiVersion: serving.knative.dev/v1 kind: Service metadata: name: test-webapp namespace: test-namespace # spec: # ... status: address: # cluster-local-domain: url: https://helloworld.test.svc.cluster.local 1 1 If you have enabled cluster-local-domain-tls , you will see the HTTPS URL. To verify that system-internal-tls is enabled, check the output of Queue-Proxy logs by running the following command: Example command USD oc logs your-pod -n test-namespace -c queue-proxy | grep -E 'certDir|Certificate|tls' If you see lines similar to the following, system-internal-tls is enabled: {"severity":"INFO","timestamp":"2024-01-03T07:07:32.892810888Z","logger":"queueproxy","caller":"certificate/watcher.go:62","message":"Starting to watch the following directories for changes{certDir 15 0 /var/lib/knative/certs <nil>} {keyDir 15 0 /var/lib/knative/certs <nil>}","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.89397512Z","logger":"queueproxy","caller":"certificate/watcher.go:131","message":"Certificate and/or key have changed on disk and were reloaded.","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.894232939Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server admin:8022","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.894268548Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server main:8112","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}

Commands:
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: knative-serving-selfsigned-issuer spec: selfSigned: {}
oc apply -f <filename>
apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: knative-serving-selfsigned-ca namespace: cert-manager 1 spec: secretName: knative-serving-ca 2 isCA: true commonName: selfsigned-ca privateKey: algorithm: ECDSA size: 256 issuerRef: name: knative-serving-selfsigned-issuer kind: ClusterIssuer group: cert-manager.io
oc apply -f <filename>
apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: knative-serving-ca-issuer spec: ca: secretName: knative-serving-ca 1
oc apply -f <filename>
apiVersion: operator.knative.dev/v1beta1 kind: KnativeServing metadata: name: knative-serving namespace: knative-serving spec: config: certmanager: clusterLocalIssuerRef: | kind: ClusterIssuer name: knative-serving-ca-issuer 1 systemInternalIssuerRef: | kind: ClusterIssuer name: knative-serving-ca-issuer 2 network: cluster-local-domain-tls: Enabled 3 system-internal-tls: Enabled 4
oc apply -f <filename>
apiVersion: operator.openshift.io/v1 kind: IngressController spec: defaultCertificate: name: ca-ingress-cert
spec: config: network: system-internal-tls: Enabled openshift-ingress-default-certificate: "ca-ingress-cert"
oc rollout restart deploy/controller -n knative-serving
oc rollout restart deploy/activator -n knative-serving
apiVersion: v1 data: cacerts.pem: | 1 -----BEGIN CERTIFICATE----- MIIDDTCCAfWgAwIBAgIQMQuip05h7NLQq2TB+j9ZmTANBgkqhkiG9w0BAQsFADAW MRQwEgYDVQQDEwtrbmF0aXZlLmRldjAeFw0yMzExMjIwOTAwNDhaFw0yNDAyMjAw OTAwNDhaMBYxFDASBgNVBAMTC2tuYXRpdmUuZGV2MIIBIjANBgkqhkiG9w0BAQEF AAOCAQ8AMIIBCgKCAQEA3clC3CV7sy0TpUKNuTku6QmP9z8JUCbLCPCLACCUc1zG FEokqOva6TakgvAntXLkB3TEsbdCJlNm6qFbbko6DBfX6rEggqZs40x3/T+KH66u 4PvMT3fzEtaMJDK/KQOBIvVHrKmPkvccUYK/qWY7rgBjVjjLVSJrCn4dKaEZ2JNr Fd0KNnaaW/dP9/FvviLqVJvHnTMHH5qyRRr1kUGTrc8njRKwpHcnUdauiDoWRKxo Zlyy+MhQfdbbyapX984WsDjCvrDXzkdGgbRNAf+erl6yUm6pHpQhyFFo/zndx6Uq QXA7jYvM2M3qCnXmaFowidoLDsDyhwoxD7WT8zur/QIDAQABo1cwVTAOBgNVHQ8B Af8EBAMCAgQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB/zAd BgNVHQ4EFgQU7p4VuECNOcnrP9ulOjc4J37Q2VUwDQYJKoZIhvcNAQELBQADggEB AAv26Vnk+ptQrppouF7yHV8fZbfnehpm07HIZkmnXO2vAP+MZJDNrHjy8JAVzXjt +OlzqAL0cRQLsUptB0btoJuw23eq8RXgJo05OLOPQ2iGNbAATQh2kLwBWd/CMg+V KJ4EIEpF4dmwOohsNR6xa/JoArIYH0D7gh2CwjrdGZr/tq1eMSL+uZcuX5OiE44A 2oXF9/jsqerOcH7QUMejSnB8N7X0LmUvH4jAesQgr7jo1JTOBs7GF6wb+U76NzFa 8ms2iAWhoplQ+EHR52wffWb0k6trXspq4O6v/J+nq9Ky3vC36so+G1ZFkMhCdTVJ ZmrBsSMWeT2l07qeei2UFRU= -----END CERTIFICATE----- kind: ConfigMap metadata: labels: networking.knative.dev/trust-bundle: "true" name: knative-bundle 2 namespace: knative-serving
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: test-webapp namespace: test-namespace spec: template: spec: containers: - image: docker.io/openshift/hello-openshift env: - name: RESPONSE value: "Hello Serverless!"
oc apply -f <filename>
oc get ksvc -n test-namespace -o yaml
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: test-webapp namespace: test-namespace spec: status: address: # cluster-local-domain: url: https://helloworld.test.svc.cluster.local 1
oc logs your-pod -n test-namespace -c queue-proxy | grep -E 'certDir|Certificate|tls'
{"severity":"INFO","timestamp":"2024-01-03T07:07:32.892810888Z","logger":"queueproxy","caller":"certificate/watcher.go:62","message":"Starting to watch the following directories for changes{certDir 15 0 /var/lib/knative/certs <nil>} {keyDir 15 0 /var/lib/knative/certs <nil>}","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.89397512Z","logger":"queueproxy","caller":"certificate/watcher.go:131","message":"Certificate and/or key have changed on disk and were reloaded.","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.894232939Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server admin:8022","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"} {"severity":"INFO","timestamp":"2024-01-03T07:07:32.894268548Z","logger":"queueproxy","caller":"sharedmain/main.go:282","message":"Starting tls server main:8112","commit":"86420f2-dirty","knative.dev/key":"first/helloworld-00001","knative.dev/pod":"helloworld-00001-deployment-75fbb7d488-qgmxx"}