Chapter 1. Using Tekton Chains for OpenShift Pipelines supply chain security Tekton Chains is a Kubernetes Custom Resource Definition (CRD) controller. You can use it to manage the supply chain security of the tasks and pipelines created using Red Hat OpenShift Pipelines. By default, Tekton Chains observes all task run executions in your OpenShift Container Platform cluster. When the task runs complete, Tekton Chains takes a snapshot of the task runs. It then converts the snapshot to one or more standard payload formats, and finally signs and stores all artifacts. To capture information about task runs, Tekton Chains uses Result objects. When the objects are unavailable, Tekton Chains the URLs and qualified digests of the OCI images. 1.1. Key features You can sign task runs, task run results, and OCI registry images with cryptographic keys that are generated by tools such as cosign and skopeo . You can use attestation formats such as in-toto . You can securely store signatures and signed artifacts using OCI repository as a storage backend. 1.2. Configuring Tekton Chains The Red Hat OpenShift Pipelines Operator installs Tekton Chains by default. You can configure Tekton Chains by modifying the TektonConfig custom resource; the Operator automatically applies the changes that you make in this custom resource. To edit the custom resource, use the following command: USD oc edit TektonConfig config The custom resource includes a chain: array. You can add any supported configuration parameters to this array, as shown in the following example: apiVersion: operator.tekton.dev/v1alpha1 kind: TektonConfig metadata: name: config spec: addon: {} chain: artifacts.taskrun.format: tekton config: {} 1.2.1. Supported parameters for Tekton Chains configuration Cluster administrators can use various supported parameter keys and values to configure specifications about task runs, OCI images, and storage. 1.2.1.1. Supported parameters for task run artifacts Table 1.1. Chains configuration: Supported parameters for task run artifacts Key Description Supported values Default value artifacts.taskrun.format The format for storing task run payloads. in-toto , slsa/v1 in-toto artifacts.taskrun.storage The storage backend for task run signatures. You can specify multiple backends as a comma-separated list, such as "tekton,oci" . To disable storing task run artifacts, provide an empty string "" . tekton , oci , gcs , docdb , grafeas tekton artifacts.taskrun.signer The signature backend for signing task run payloads. x509 , kms x509 Note slsa/v1 is an alias of in-toto for backwards compatibility. 1.2.1.2. Supported parameters for pipeline run artifacts Table 1.2. Chains configuration: Supported parameters for pipeline run artifacts Parameter Description Supported values Default value artifacts.pipelinerun.format The format for storing pipeline run payloads. in-toto , slsa/v1 in-toto artifacts.pipelinerun.storage The storage backend for storing pipeline run signatures. You can specify multiple backends as a comma-separated list, such as "tekton,oci" . To disable storing pipeline run artifacts, provide an empty string "" . tekton , oci , gcs , docdb , grafeas tekton artifacts.pipelinerun.signer The signature backend for signing pipeline run payloads. x509 , kms x509 artifacts.pipelinerun.enable-deep-inspection When this parameter is true , Tekton Chains records the results of the child task runs of a pipeline run. When this parameter is false , Tekton Chains records the results of the pipeline run, but not of its child task runs. "true", "false" "false" Note slsa/v1 is an alias of in-toto for backwards compatibility. For the grafeas storage backend, only Container Analysis is supported. You can not configure the grafeas server address in the current version of Tekton Chains. 1.2.1.3. Supported parameters for OCI artifacts Table 1.3. Chains configuration: Supported parameters for OCI artifacts Parameter Description Supported values Default value artifacts.oci.format The format for storing OCI payloads. simplesigning simplesigning artifacts.oci.storage The storage backend for storing OCI signatures. You can specify multiple backends as a comma-separated list, such as "oci,tekton" . To disable storing OCI artifacts, provide an empty string "" . tekton , oci , gcs , docdb , grafeas oci artifacts.oci.signer The signature backend for signing OCI payloads. x509 , kms x509 1.2.1.4. Supported parameters for KMS signers Table 1.4. Chains configuration: Supported parameters for KMS signers Parameter Description Supported values Default value signers.kms.kmsref The URI reference to a KMS service to use in kms signers. Supported schemes: gcpkms:// , awskms:// , azurekms:// , hashivault:// . See Providers in the Sigstore documentation for more details. 1.2.1.5. Supported parameters for storage Table 1.5. Chains configuration: Supported parameters for storage Parameter Description Supported values Default value storage.gcs.bucket The GCS bucket for storage storage.oci.repository The OCI repository for storing OCI signatures and attestation. If you configure one of the artifact storage backends to oci and do not define this key, Tekton Chains stores the attestation alongside the stored OCI artifact itself. If you define this key, the attestation is not stored alongside the OCI artifact and is instead stored in the designated location. See the cosign documentation for additional information. builder.id The builder ID to set for in-toto attestations https://tekton.dev/chains/v2 builddefinition.buildtype The build type for in-toto attestation. When this parameter is https://tekton.dev/chains/v2/slsa , Tekton Chains records in-toto attestations in strict conformance with the SLSA v1.0 specification. When this parameter is https://tekton.dev/chains/v2/slsa-tekton , Tekton Chains records in-toto attestations with additional information, such as the labels and annotations in each TaskRun and PipelineRun object, and also adds each task in a PipelineRun object under resolvedDependencies . https://tekton.dev/chains/v2/slsa , https://tekton.dev/chains/v2/slsa-tekton https://tekton.dev/chains/v2/slsa If you enable the docdb storage method is for any artifacts, configure docstore storage options. For more information about the go-cloud docstore URI format, see the docstore package documentation . Red Hat OpenShift Pipelines supports the following docstore services: firestore dynamodb Table 1.6. Chains configuration: Supported parameters for docstore storage Parameter Description Supported values Default value storage.docdb.url The go-cloud URI reference to a docstore collection. Used if the docdb storage method is enabled for any artifacts. firestore://projects/[PROJECT]/databases/(default)/documents/[COLLECTION]?name_field=name If you enable the grafeas storage method for any artifacts, configure Grafeas storage options. For more information about Grafeas notes and occurrences, see Grafeas concepts . To create occurrences, Red Hat OpenShift Pipelines must first create notes that are used to link occurrences. Red Hat OpenShift Pipelines creates two types of occurrences: ATTESTATION Occurrence and BUILD Occurrence. Red Hat OpenShift Pipelines uses the configurable noteid as the prefix of the note name. It appends the suffix -simplesigning for the ATTESTATION note and the suffix -intoto for the BUILD note. If the noteid field is not configured, Red Hat OpenShift Pipelines uses tekton-<NAMESPACE> as the prefix. Table 1.7. Chains configuration: Supported parameters for Grafeas storage Parameter Description Supported values Default value storage.grafeas.projectid The OpenShift Container Platform project in which the Grafeas server for storing occurrences is located. storage.grafeas.noteid Optional: the prefix to use for the name of all created notes. A string without spaces. storage.grafeas.notehint Optional: the human_readable_name field for the Grafeas ATTESTATION note. This attestation note was generated by Tekton Chains Optionally, you can enable additional uploads of binary transparency attestations. Table 1.8. Chains configuration: Supported parameters for transparency attestation storage Parameter Description Supported values Default value transparency.enabled Enable or disable automatic binary transparency uploads. true , false , manual false transparency.url The URL for uploading binary transparency attestations, if enabled. https://rekor.sigstore.dev Note If you set transparency.enabled to manual , only task runs and pipeline runs with the following annotation are uploaded to the transparency log: chains.tekton.dev/transparency-upload: "true" If you configure the x509 signature backend, you can optionally enable keyless signing with Fulcio. Table 1.9. Chains configuration: Supported parameters for x509 keyless signing with Fulcio Parameter Description Supported values Default value signers.x509.fulcio.enabled Enable or disable requesting automatic certificates from Fulcio. true , false false signers.x509.fulcio.address The Fulcio address for requesting certificates, if enabled. https://v1.fulcio.sigstore.dev signers.x509.fulcio.issuer The expected OIDC issuer. https://oauth2.sigstore.dev/auth signers.x509.fulcio.provider The provider from which to request the ID Token. google , spiffe , github , filesystem Red Hat OpenShift Pipelines attempts to use every provider signers.x509.identity.token.file Path to the file containing the ID Token. signers.x509.tuf.mirror.url The URL for the TUF server. USDTUF_URL/root.json must be present. https://sigstore-tuf-root.storage.googleapis.com If you configure the kms signature backend, set the KMS configuration, including OIDC and Spire, as necessary. Table 1.10. Chains configuration: Supported parameters for KMS signing Parameter Description Supported values Default value signers.kms.auth.address URI of the KMS server (the value of VAULT_ADDR ). signers.kms.auth.token Authentication token for the KMS server (the value of VAULT_TOKEN ). signers.kms.auth.oidc.path The path for OIDC authentication (for example, jwt for Vault). signers.kms.auth.oidc.role The role for OIDC authentication. signers.kms.auth.spire.sock The URI of the Spire socket for the KMS token (for example, unix:///tmp/spire-agent/public/api.sock ). signers.kms.auth.spire.audience The audience for requesting a SVID from Spire. 1.3. Secrets for signing data in Tekton Chains Cluster administrators can generate a key pair and use Tekton Chains to sign artifacts using a Kubernetes secret. For Tekton Chains to work, a private key and a password for encrypted keys must exist as part of the signing-secrets secret in the openshift-pipelines namespace. Currently, Tekton Chains supports the x509 and cosign signature schemes. Note Use only one of the supported signature schemes. To use the x509 signing scheme with Tekton Chains, store the x509.pem private key of the ed25519 or ecdsa type in the signing-secrets Kubernetes secret. 1.3.1. Signing using cosign You can use the cosign signing scheme with Tekton Chains using the cosign tool. Prerequisites You installed the Cosign tool. For information about installing the Cosign tool, see the Sigstore documentation for Cosign . Procedure Generate the cosign.key and cosign.pub key pairs by running the following command: USD cosign generate-key-pair k8s://openshift-pipelines/signing-secrets Cosign prompts you for a password and then creates a Kubernetes secret. Store the encrypted cosign.key private key and the cosign.password decryption password in the signing-secrets Kubernetes secret. Ensure that the private key is stored as an encrypted PEM file of the ENCRYPTED COSIGN PRIVATE KEY type. 1.3.2. Signing using skopeo You can generate keys using the skopeo tool and use them in the cosign signing scheme with Tekton Chains. Prerequisites You installed the skopeo tool. Procedure Generate a public/private key pair by running the following command: USD skopeo generate-sigstore-key --output-prefix <mykey> 1 1 Replace <mykey> with a key name of your choice. Skopeo prompts you for a passphrase for the private key and then creates the key files named <mykey>.private and <mykey>.pub . Encode the <mykey>.pub file using the base64 tool by running the following command: USD base64 -w 0 <mykey>.pub > b64.pub Encode the <mykey>.private file using the base64 tool by running the following command: USD base64 -w 0 <mykey>.private > b64.private Encode the passhprase using the base64 tool by running the following command: USD echo -n '<passphrase>' | base64 -w 0 > b64.passphrase 1 1 Replace <passphrase> with the passphrase that you used for the key pair. Create the signing-secrets secret in the openshift-pipelines namespace by running the following command: USD oc create secret generic signing-secrets -n openshift-pipelines Edit the signing-secrets secret by running the following command: Add the encoded keys in the data of the secret in the following way: apiVersion: v1 data: cosign.key: <Encoded <mykey>.private> 1 cosign.password: <Encoded passphrase> 2 cosign.pub: <Encoded <mykey>.pub> 3 immutable: true kind: Secret metadata: name: signing-secrets # ... type: Opaque 1 Replace <Encoded <mykey>.private> with the content of the b64.private file. 2 Replace <Encoded passphrase> with the content of the b64.passphrase file. 3 Replace <Encoded <mykey>.pub> with the content of the b64.pub file. 1.3.3. Resolving the "secret already exists" error If the signing-secret secret is already populated, the command to create this secret might output the following error message: Error from server (AlreadyExists): secrets "signing-secrets" already exists You can resolve this error by deleting the secret. Procedure Delete the signing-secret secret by running the following command: USD oc delete secret signing-secrets -n openshift-pipelines Re-create the key pairs and store them in the secret using your preferred signing scheme. 1.4. Authenticating to an OCI registry Before pushing signatures to an OCI registry, cluster administrators must configure Tekton Chains to authenticate with the registry. The Tekton Chains controller uses the same service account under which the task runs execute. To set up a service account with the necessary credentials for pushing signatures to an OCI registry, perform the following steps: Procedure Set the namespace and name of the Kubernetes service account. USD export NAMESPACE=<namespace> 1 USD export SERVICE_ACCOUNT_NAME=<service_account> 2 1 The namespace associated with the service account. 2 The name of the service account. Create a Kubernetes secret. USD oc create secret registry-credentials \ --from-file=.dockerconfigjson \ 1 --type=kubernetes.io/dockerconfigjson \ -n USDNAMESPACE 1 Substitute with the path to your Docker config file. Default path is ~/.docker/config.json . Give the service account access to the secret. USD oc patch serviceaccount USDSERVICE_ACCOUNT_NAME \ -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}" -n USDNAMESPACE If you patch the default pipeline service account that Red Hat OpenShift Pipelines assigns to all task runs, the Red Hat OpenShift Pipelines Operator will override the service account. As a best practice, you can perform the following steps: Create a separate service account to assign to user's task runs. USD oc create serviceaccount <service_account_name> Associate the service account to the task runs by setting the value of the serviceaccountname field in the task run template. apiVersion: tekton.dev/v1 kind: TaskRun metadata: name: build-push-task-run-2 spec: taskRunTemplate: serviceAccountName: build-bot 1 taskRef: name: build-push ... 1 Substitute with the name of the newly created service account. 1.5. Creating and verifying task run signatures without any additional authentication To verify signatures of task runs using Tekton Chains with any additional authentication, perform the following tasks: Create an encrypted x509 key pair and save it as a Kubernetes secret. Configure the Tekton Chains backend storage. Create a task run, sign it, and store the signature and the payload as annotations on the task run itself. Retrieve the signature and payload from the signed task run. Verify the signature of the task run. Prerequisites Ensure that the following components are installed on the cluster: Red Hat OpenShift Pipelines Operator Tekton Chains Cosign Procedure Create an encrypted x509 key pair and save it as a Kubernetes secret. For more information about creating a key pair and saving it as a secret, see "Signing secrets in Tekton Chains". In the Tekton Chains configuration, disable the OCI storage, and set the task run storage and format to tekton . In the TektonConfig custom resource set the following values: apiVersion: operator.tekton.dev/v1alpha1 kind: TektonConfig metadata: name: config spec: # ... chain: artifacts.oci.storage: "" artifacts.taskrun.format: tekton artifacts.taskrun.storage: tekton # ... For more information about configuring Tekton Chains using the TektonConfig custom resource, see "Configuring Tekton Chains". To restart the Tekton Chains controller to ensure that the modified configuration is applied, enter the following command: USD oc delete po -n openshift-pipelines -l app=tekton-chains-controller Create a task run by entering the following command: USD oc create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/taskruns/task-output-image.yaml 1 1 Replace the example URI with the URI or file path pointing to your task run. Example output taskrun.tekton.dev/build-push-run-output-image-qbjvh created Check the status of the steps by entering the following command. Wait until the process finishes. USD tkn tr describe --last Example output [...truncated output...] NAME STATUS ∙ create-dir-builtimage-9467f Completed ∙ git-source-sourcerepo-p2sk8 Completed ∙ build-and-push Completed ∙ echo Completed ∙ image-digest-exporter-xlkn7 Completed To retrieve the signature from the object stored as base64 encoded annotations, enter the following commands: USD tkn tr describe --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-taskrun-USDTASKRUN_UID}" | base64 -d > sig USD export TASKRUN_UID=USD(tkn tr describe --last -o jsonpath='{.metadata.uid}') To verify the signature using the public key that you created, enter the following command: 1 Replace path/to/cosign.pub with the path name of the public key file. Example output Verified OK 1.5.1. Additional resources Section 1.3, "Secrets for signing data in Tekton Chains" Section 1.2, "Configuring Tekton Chains" 1.6. Using Tekton Chains to sign and verify image and provenance Cluster administrators can use Tekton Chains to sign and verify images and provenances, by performing the following tasks: Create an encrypted x509 key pair and save it as a Kubernetes secret. Set up authentication for the OCI registry to store images, image signatures, and signed image attestations. Configure Tekton Chains to generate and sign provenance. Create an image with Kaniko in a task run. Verify the signed image and the signed provenance. Prerequisites Ensure that the following tools are installed on the cluster: Red Hat OpenShift Pipelines Operator Tekton Chains Cosign Rekor jq Procedure Create an encrypted x509 key pair and save it as a Kubernetes secret: USD cosign generate-key-pair k8s://openshift-pipelines/signing-secrets Provide a password when prompted. Cosign stores the resulting private key as part of the signing-secrets Kubernetes secret in the openshift-pipelines namespace, and writes the public key to the cosign.pub local file. Configure authentication for the image registry. To configure the Tekton Chains controller for pushing signature to an OCI registry, use the credentials associated with the service account of the task run. For detailed information, see the "Authenticating to an OCI registry" section. To configure authentication for a Kaniko task that builds and pushes image to the registry, create a Kubernetes secret of the docker config.json file containing the required credentials. USD oc create secret generic <docker_config_secret_name> \ 1 --from-file <path_to_config.json> 2 1 Substitute with the name of the docker config secret. 2 Substitute with the path to docker config.json file. Configure Tekton Chains by setting the artifacts.taskrun.format , artifacts.taskrun.storage , and transparency.enabled parameters in the chains-config object: USD oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.format": "in-toto"}}' USD oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.storage": "oci"}}' USD oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"transparency.enabled": "true"}}' Start the Kaniko task. Apply the Kaniko task to the cluster. USD oc apply -f examples/kaniko/kaniko.yaml 1 1 Substitute with the URI or file path to your Kaniko task. Set the appropriate environment variables. USD export REGISTRY=<url_of_registry> 1 USD export DOCKERCONFIG_SECRET_NAME=<name_of_the_secret_in_docker_config_json> 2 1 Substitute with the URL of the registry where you want to push the image. 2 Substitute with the name of the secret in the docker config.json file. Start the Kaniko task. USD tkn task start --param IMAGE=USDREGISTRY/kaniko-chains --use-param-defaults --workspace name=source,emptyDir="" --workspace name=dockerconfig,secret=USDDOCKERCONFIG_SECRET_NAME kaniko-chains Observe the logs of this task until all steps are complete. On successful authentication, the final image will be pushed to USDREGISTRY/kaniko-chains . Wait for a minute to allow Tekton Chains to generate the provenance and sign it, and then check the availability of the chains.tekton.dev/signed=true annotation on the task run. USD oc get tr <task_run_name> \ 1 -o json | jq -r .metadata.annotations { "chains.tekton.dev/signed": "true", ... } 1 Substitute with the name of the task run. Verify the image and the attestation. USD cosign verify --key cosign.pub USDREGISTRY/kaniko-chains USD cosign verify-attestation --key cosign.pub USDREGISTRY/kaniko-chains Find the provenance for the image in Rekor. Get the digest of the USDREGISTRY/kaniko-chains image. You can search for it ing the task run, or pull the image to extract the digest. Search Rekor to find all entries that match the sha256 digest of the image. USD rekor-cli search --sha <image_digest> 1 <uuid_1> 2 <uuid_2> 3 ... 1 Substitute with the sha256 digest of the image. 2 The first matching universally unique identifier (UUID). 3 The second matching UUID. The search result displays UUIDs of the matching entries. One of those UUIDs holds the attestation. Check the attestation. USD rekor-cli get --uuid <uuid> --format json | jq -r .Attestation | base64 --decode | jq 1.7. Additional resources Installing OpenShift Pipelines

Commands:
oc edit TektonConfig config
apiVersion: operator.tekton.dev/v1alpha1 kind: TektonConfig metadata: name: config spec: addon: {} chain: artifacts.taskrun.format: tekton config: {}
chains.tekton.dev/transparency-upload: "true"
cosign generate-key-pair k8s://openshift-pipelines/signing-secrets
skopeo generate-sigstore-key --output-prefix <mykey> 1
base64 -w 0 <mykey>.pub > b64.pub
base64 -w 0 <mykey>.private > b64.private
echo -n '<passphrase>' | base64 -w 0 > b64.passphrase 1
oc create secret generic signing-secrets -n openshift-pipelines
oc edit secret -n openshift-pipelines signing-secrets
apiVersion: v1 data: cosign.key: <Encoded <mykey>.private> 1 cosign.password: <Encoded passphrase> 2 cosign.pub: <Encoded <mykey>.pub> 3 immutable: true kind: Secret metadata: name: signing-secrets type: Opaque
Error from server (AlreadyExists): secrets "signing-secrets" already exists
oc delete secret signing-secrets -n openshift-pipelines
export NAMESPACE=<namespace> 1 export SERVICE_ACCOUNT_NAME=<service_account> 2
oc create secret registry-credentials --from-file=.dockerconfigjson \ 1 --type=kubernetes.io/dockerconfigjson -n USDNAMESPACE
oc patch serviceaccount USDSERVICE_ACCOUNT_NAME -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}" -n USDNAMESPACE
oc create serviceaccount <service_account_name>
apiVersion: tekton.dev/v1 kind: TaskRun metadata: name: build-push-task-run-2 spec: taskRunTemplate: serviceAccountName: build-bot 1 taskRef: name: build-push
apiVersion: operator.tekton.dev/v1alpha1 kind: TektonConfig metadata: name: config spec: chain: artifacts.oci.storage: "" artifacts.taskrun.format: tekton artifacts.taskrun.storage: tekton
oc delete po -n openshift-pipelines -l app=tekton-chains-controller
oc create -f https://raw.githubusercontent.com/tektoncd/chains/main/examples/taskruns/task-output-image.yaml 1
taskrun.tekton.dev/build-push-run-output-image-qbjvh created
tkn tr describe --last
[...truncated output...] NAME STATUS ∙ create-dir-builtimage-9467f Completed ∙ git-source-sourcerepo-p2sk8 Completed ∙ build-and-push Completed ∙ echo Completed ∙ image-digest-exporter-xlkn7 Completed
tkn tr describe --last -o jsonpath="{.metadata.annotations.chains\.tekton\.dev/signature-taskrun-USDTASKRUN_UID}" | base64 -d > sig
export TASKRUN_UID=USD(tkn tr describe --last -o jsonpath='{.metadata.uid}')
cosign verify-blob-attestation --insecure-ignore-tlog --key path/to/cosign.pub --signature sig --type slsaprovenance --check-claims=false /dev/null 1
Verified OK
cosign generate-key-pair k8s://openshift-pipelines/signing-secrets
oc create secret generic <docker_config_secret_name> \ 1 --from-file <path_to_config.json> 2
oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.format": "in-toto"}}' oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"artifacts.taskrun.storage": "oci"}}' oc patch configmap chains-config -n openshift-pipelines -p='{"data":{"transparency.enabled": "true"}}'
oc apply -f examples/kaniko/kaniko.yaml 1
export REGISTRY=<url_of_registry> 1 export DOCKERCONFIG_SECRET_NAME=<name_of_the_secret_in_docker_config_json> 2
tkn task start --param IMAGE=USDREGISTRY/kaniko-chains --use-param-defaults --workspace name=source,emptyDir="" --workspace name=dockerconfig,secret=USDDOCKERCONFIG_SECRET_NAME kaniko-chains
oc get tr <task_run_name> \ 1 -o json | jq -r .metadata.annotations { "chains.tekton.dev/signed": "true", }
cosign verify --key cosign.pub USDREGISTRY/kaniko-chains cosign verify-attestation --key cosign.pub USDREGISTRY/kaniko-chains
rekor-cli search --sha <image_digest> 1 <uuid_1> 2 <uuid_2> 3
rekor-cli get --uuid <uuid> --format json | jq -r .Attestation | base64 --decode | jq