Chapter 9. Working with certificates Certificates are used by various components in OpenShift to validate access to the cluster. For clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle and use the CA bundle in Red Hat OpenShift AI. You can also use self-signed certificates in a custom CA bundle that is separate from the cluster-wide bundle. Administrators can add a CA bundle, remove a CA bundle from all namespaces, remove a CA bundle from individual namespaces, or manually manage certificate changes instead of the system. 9.1. Understanding certificates in OpenShift AI For OpenShift clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle ( ca-bundle.crt ) and use the CA bundle in Red Hat OpenShift AI. You can also use self-signed certificates in a custom CA bundle ( odh-ca-bundle.crt ) that is separate from the cluster-wide bundle. 9.1.1. How CA bundles are injected After installing OpenShift AI, the Red Hat OpenShift AI Operator automatically creates an empty odh-trusted-ca-bundle configuration file (ConfigMap), and the Cluster Network Operator (CNO) injects the cluster-wide CA bundle into the odh-trusted-ca-bundle configMap with the label "config.openshift.io/inject-trusted-cabundle". The components deployed in the affected namespaces are responsible for mounting this configMap as a volume in the deployment pods. After the CNO operator injects the bundle, it updates the ConfigMap with the ca-bundle.crt file containing the certificates. 9.1.2. How the ConfigMap is managed By default, the Red Hat OpenShift AI Operator manages the odh-trusted-ca-bundle ConfigMap. If you want to manage or remove the odh-trusted-ca-bundle ConfigMap, or add a custom CA bundle ( odh-ca-bundle.crt ) separate from the cluster-wide CA bundle ( ca-bundle.crt ), you can use the trustedCABundle property in the Operator's DSC Initialization (DSCI) object. In the Operator's DSCI object, you can set the spec.trustedCABundle.managementState field to the following values: Managed : The Red Hat OpenShift AI Operator manages the odh-trusted-ca-bundle ConfigMap and adds it to all non-reserved existing and new namespaces (the ConfigMap is not added to any reserved or system namespaces, such as default , openshift-\* or kube-* ). The ConfigMap is automatically updated to reflect any changes made to the customCABundle field. This is the default value after installing Red Hat OpenShift AI. Removed : The Red Hat OpenShift AI Operator removes the odh-trusted-ca-bundle ConfigMap (if present) and disables the creation of the ConfigMap in new namespaces. If you change this field from Managed to Removed , the odh-trusted-ca-bundle ConfigMap is also deleted from namespaces. This is the default value after upgrading Red Hat OpenShift AI from 2.7 or earlier versions to 2.18. Unmanaged : The Red Hat OpenShift AI Operator does not manage the odh-trusted-ca-bundle ConfigMap, allowing for an administrator to manage it instead. Changing the managementState from Managed to Unmanaged does not remove the odh-trusted-ca-bundle ConfigMap, but the ConfigMap is not updated if you make changes to the customCABundle field. In the Operator's DSCI object, you can add a custom certificate to the spec.trustedCABundle.customCABundle field. This adds the odh-ca-bundle.crt file containing the certificates to the odh-trusted-ca-bundle ConfigMap, as shown in the following example: 9.2. Adding a CA bundle There are two ways to add a Certificate Authority (CA) bundle to OpenShift AI. You can use one or both of these methods: For OpenShift clusters that rely on self-signed certificates, you can add those self-signed certificates to a cluster-wide Certificate Authority (CA) bundle ( ca-bundle.crt ) and use the CA bundle in Red Hat OpenShift AI. To use this method, log in to the OpenShift as a cluster administrator and follow the steps as described in Configuring the cluster-wide proxy during installation . You can use self-signed certificates in a custom CA bundle ( odh-ca-bundle.crt ) that is separate from the cluster-wide bundle. To use this method, follow the steps in this section. Prerequisites You have admin access to the DSCInitialization resources in the OpenShift cluster. You installed the OpenShift command line interface ( oc ) as described in Installing the OpenShift CLI . You are working in a new installation of Red Hat OpenShift AI. If you upgraded Red Hat OpenShift AI, see Adding a CA bundle after upgrading . Procedure Log in to the OpenShift. Click Operators Installed Operators and then click the Red Hat OpenShift AI Operator. Click the DSC Initialization tab. Click the default-dsci object. Click the YAML tab. In the spec section, add the custom certificate to the customCABundle field for trustedCABundle , as shown in the following example: Click Save . Verification If you are using a cluster-wide CA bundle, run the following command to verify that all non-reserved namespaces contain the odh-trusted-ca-bundle ConfigMap: If you are using a custom CA bundle, run the following command to verify that a non-reserved namespace contains the odh-trusted-ca-bundle ConfigMap and that the ConfigMap contains your customCABundle value. In the following command, example-namespace is the non-reserved namespace and examplebundle123 is the customCABundle value. 9.3. Removing a CA bundle You can remove a Certificate Authority (CA) bundle from all non-reserved namespaces in OpenShift AI. This process changes the default configuration and disables the creation of the odh-trusted-ca-bundle configuration file (ConfigMap), as described in Understanding certificates in OpenShift AI . Note The odh-trusted-ca-bundle ConfigMaps are only deleted from namespaces when you set the managementState of trustedCABundle to Removed ; deleting the DSC Initialization does not delete the ConfigMaps. To remove a CA bundle from a single namespace only, see Removing a CA bundle from a namespace . Prerequisites You have cluster administrator privileges for your OpenShift cluster. You installed the OpenShift command line interface ( oc ) as described in Installing the OpenShift CLI . Procedure In the OpenShift web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator. Click the DSC Initialization tab. Click the default-dsci object. Click the YAML tab. In the spec section, change the value of the managementState field for trustedCABundle to Removed : Click Save . Verification Run the following command to verify that the odh-trusted-ca-bundle ConfigMap has been removed from all namespaces: The command should not return any ConfigMaps. 9.4. Removing a CA bundle from a namespace You can remove a custom Certificate Authority (CA) bundle from individual namespaces in OpenShift AI. This process disables the creation of the odh-trusted-ca-bundle configuration file (ConfigMap) for the specified namespace only. To remove a certificate bundle from all namespaces, see Removing a CA bundle . Prerequisites You have cluster administrator privileges for your OpenShift cluster. You installed the OpenShift command line interface ( oc ) as described in Installing the OpenShift CLI . Procedure Run the following command to remove a CA bundle from a namespace. In the following command, example-namespace is the non-reserved namespace. Verification Run the following command to verify that the CA bundle has been removed from the namespace. In the following command, example-namespace is the non-reserved namespace. The command should return configmaps "odh-trusted-ca-bundle" not found . 9.5. Managing certificates After installing OpenShift AI, the Red Hat OpenShift AI Operator creates the odh-trusted-ca-bundle configuration file (ConfigMap) that contains the trusted CA bundle and adds it to all new and existing non-reserved namespaces in the cluster. By default, the Red Hat OpenShift AI Operator manages the odh-trusted-ca-bundle ConfigMap and automatically updates it if any changes are made to the CA bundle. You can choose to manage the odh-trusted-ca-bundle ConfigMap instead of allowing the Red Hat OpenShift AI Operator to manage it. Prerequisites You have cluster administrator privileges for your OpenShift cluster. Procedure In the OpenShift web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator . Click the DSC Initialization tab. Click the default-dsci object. Click the YAML tab. In the spec section, change the value of the managementState field for trustedCABundle to Unmanaged , as shown: Click Save . Note that changing the managementState from Managed to Unmanaged does not remove the odh-trusted-ca-bundle ConfigMap, but the ConfigMap is not updated if you make changes to the customCABundle field. Verification In the spec section, set or change the value of the customCABundle field for trustedCABundle , for example: Click Save . Click Workloads ConfigMaps . Select a project from the project list. Click the odh-trusted-ca-bundle ConfigMap. Click the YAML tab and verify that the value of the customCABundle field did not update. 9.6. Accessing S3-compatible object storage with self-signed certificates To use object storage solutions or databases that are deployed in an OpenShift cluster that uses self-signed certificates, you must configure OpenShift AI to trust the cluster's certificate authority (CA). Each namespace has a ConfigMap called kube-root-ca.crt that contains the CA certificates of the internal API Server. Use the following steps to configure OpenShift AI to trust the certificates issued by kube-root-ca.crt . Alternatively, you can add a custom CA bundle by using the OpenShift console, as described in Adding a CA bundle . Prerequisites You have cluster administrator privileges for your OpenShift cluster. You have downloaded and installed the OpenShift command-line interface (CLI). See Installing the OpenShift CLI . You have an object storage solution or database deployed in your OpenShift cluster. Procedure In a terminal window, log in to the OpenShift CLI as shown in the following example: Run the following command to fetch the current OpenShift AI trusted CA configuration and store it in a new file: Add the cluster's kube-root-ca.crt ConfigMap to the OpenShift AI trusted CA configuration: Update the OpenShift AI trusted CA configuration to trust certificates issued by the certificate authorities in kube-root-ca.crt : Verification You can successfully deploy components that are configured to use object storage solutions or databases that are deployed in the OpenShift cluster. For example, a pipeline server that is configured to use a database deployed in the cluster starts successfully. Note You can verify your new certificate configuration by following the steps in the OpenShift AI fraud detection tutorial. Run the script to install local object storage buckets and create connections, and then enable data science pipelines. For more information about running the script to install local object storage buckets, see Running a script to install local object storage buckets and create connections . For more information about enabling data science pipelines, see Enabling data science pipelines . 9.7. Using self-signed certificates with OpenShift AI components Some OpenShift AI components have additional options or required configuration for self-signed certificates. 9.7.1. Using certificates with data science pipelines If you want to use self-signed certificates, you have added them to a central Certificate Authority (CA) bundle as described in Working with certificates (for disconnected environments, see Working with certificates ). No additional configuration is necessary to use those certificates with data science pipelines. 9.7.1.1. Providing a CA bundle only for data science pipelines Perform the following steps to provide a Certificate Authority (CA) bundle just for data science pipelines. Procedure Log in to OpenShift. From Workloads ConfigMaps , create a ConfigMap with the required bundle in the same data science project or namespace as the target data science pipeline: Add the following snippet to the .spec.apiserver.caBundle field of the underlying Data Science Pipelines Application (DSPA): The pipeline server pod redeploys with the updated bundle and uses it in the newly created pipeline pods. Verification Perform the following steps to confirm that your CA bundle was successfully mounted. Log in to the OpenShift console. Go to the OpenShift project that corresponds to the data science project. Click the Pods tab. Click the pipeline server pod with the ds-pipeline-dspa-<hash> prefix. Click Terminal . Enter cat /dsp-custom-certs/dsp-ca.crt . Verify that your CA bundle is present within this file. You can also confirm that your CA bundle was successfully mounted by using the CLI: In a terminal window, log in to the OpenShift cluster where OpenShift AI is deployed. Set the dspa value: Set the dsProject value, replacing USDYOUR_DS_PROJECT with the name of your data science project: Set the pod value: Display the contents of the /dsp-custom-certs/dsp-ca.crt file: Verify that your CA bundle is present within this file. 9.7.2. Using certificates with workbenches Important Self-signed certificates apply by default to workbenches that you create after configuring the certificates centrally as described in Working with certificates (for disconnected environments, see Working with certificates ). To apply centrally configured certificates to an existing workbench, stop and then restart the workbench. Self-signed certificates are stored in /etc/pki/tls/custom-certs/ca-bundle.crt . Workbenches are preset with an environment variable that points packages to this path, and that covers many popular HTTP client packages. For packages that are not included by default, you can provide this certificate path. For example, for the kfp package to connect to the data science pipeline server:

Commands:
apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/part-of: opendatahub-operator config.openshift.io/inject-trusted-cabundle: 'true' name: odh-trusted-ca-bundle
apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/part-of: opendatahub-operator config.openshift.io/inject-trusted-cabundle: 'true' name: odh-trusted-ca-bundle data: ca-bundle.crt: | <BUNDLE OF CLUSTER-WIDE CERTIFICATES>
spec: trustedCABundle: managementState: Managed customCABundle: ""
apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/part-of: opendatahub-operator config.openshift.io/inject-trusted-cabundle: 'true' name: odh-trusted-ca-bundle data: ca-bundle.crt: | <BUNDLE OF CLUSTER-WIDE CERTIFICATES> odh-ca-bundle.crt: | <BUNDLE OF CUSTOM CERTIFICATES>
spec: trustedCABundle: managementState: Managed customCABundle: | -----BEGIN CERTIFICATE----- examplebundle123 -----END CERTIFICATE-----
oc get configmaps --all-namespaces -l app.kubernetes.io/part-of=opendatahub-operator | grep odh-trusted-ca-bundle
oc get configmap odh-trusted-ca-bundle -n example-namespace -o yaml | grep examplebundle123
spec: trustedCABundle: managementState: Removed
oc get configmaps --all-namespaces | grep odh-trusted-ca-bundle
oc annotate ns example-namespace security.opendatahub.io/inject-trusted-ca-bundle=false
oc get configmap odh-trusted-ca-bundle -n example-namespace
spec: trustedCABundle: managementState: Unmanaged
spec: trustedCABundle: managementState: Unmanaged customCABundle: example123
login api.<cluster_name>.<cluster_domain>:6443 --web
get dscinitializations.dscinitialization.opendatahub.io default-dsci -o json | jq -r '.spec.trustedCABundle.customCABundle' > /tmp/my-custom-ca-bundles.crt
get configmap kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}" >> /tmp/my-custom-ca-bundles.crt
patch dscinitialization default-dsci --type='json' -p='[{"op":"replace","path":"/spec/trustedCABundle/customCABundle","value":"'"USD(awk '{printf "%s\\n", USD0}' /tmp/my-custom-ca-bundles.crt)"'"}]'
kind: ConfigMap apiVersion: v1 metadata: name: custom-ca-bundle data: ca-bundle.crt: | # contents of ca-bundle.crt
apiVersion: datasciencepipelinesapplications.opendatahub.io/v1 kind: DataSciencePipelinesApplication metadata: name: data-science-dspa spec: apiServer: cABundle: configMapName: custom-ca-bundle configMapKey: ca-bundle.crt
login
dspa=dspa
dsProject=USDYOUR_DS_PROJECT
pod=USD(oc get pod -n USD{dsProject} -l app=ds-pipeline-USD{dspa} --no-headers | awk '{print USD1}')
-n USD{dsProject} exec USDpod -- cat /dsp-custom-certs/dsp-ca.crt
from kfp.client import Client with open(sa_token_file_path, 'r') as token_file: bearer_token = token_file.read() client = Client( host='https://<GO_TO_ROUTER_OF_DS_PROJECT>/', existing_token=bearer_token, ssl_ca_cert='/etc/pki/tls/custom-certs/ca-bundle.crt' ) print(client.list_experiments())