Chapter 1. Protect your signing data As a systems administrator, protecting the signing data of your software supply chain is critical when there is data loss due to hardware failure or accidental data deletion. The OpenShift API Data Protection (OADP) product provides data protection to applications running on Red Hat OpenShift. By using the OADP product, this can help us get the software developers back to signing and verifying code as quickly as possible. After installing and configuring the OADP operator you can start backing up and restoring your Red Hat Trusted Artifact Signer (RHTAS) data. 1.1. Installing and configuring the OADP operator The OpenShift API Data Protection (OADP) operator gives you the ability to backup OpenShift application resources and internal container images. You can use the OADP operator to backup and restore your Trusted Artifact Signer data. Important This procedure uses Amazon Web Services (AWS) Simple Storage Service (S3) to create a bucket for illustrating how to configure the OADP operator. You can choose to use a different supported S3-compatible object storage platform instead of AWS, such as Red Hat OpenShift Data Foundation . Prerequisites Red Hat OpenShift Container Platform version 4.13 or later. Access to the OpenShift web console with the cluster-admin role. The ability to create an S3-compatible bucket. A workstation with the oc , and aws binaries installed. Procedure Open a terminal on your workstation, and log in to OpenShift: Syntax oc login --token= TOKEN --server= SERVER_URL_AND_PORT Example Note You can find your login token and URL for use on the command line from the OpenShift web console. Log in to the OpenShift web console. Click your user name, and click Copy login command . Offer your user name and password again, if asked, and click Display Token to view the command. Create a new bucket: Syntax export BUCKET= NEW_BUCKET_NAME export REGION= AWS_REGION_ID export USER= OADP_USER_NAME aws s3api create-bucket \ --bucket USDBUCKET \ --region USDREGION \ --create-bucket-configuration LocationConstraint=USDREGION Example USD export BUCKET=example-bucket-name USD export REGION=us-east-1 USD export USER=velero USD USD aws s3api create-bucket \ --bucket USDBUCKET \ --region USDREGION \ --create-bucket-configuration LocationConstraint=USDREGION Create a new user: Example USD aws iam create-user --user-name USDUSER Create a new policy: Example USD cat > velero-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeVolumes", "ec2:DescribeSnapshots", "ec2:CreateTags", "ec2:CreateVolume", "ec2:CreateSnapshot", "ec2:DeleteSnapshot" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:DeleteObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Resource": [ "arn:aws:s3:::USD{BUCKET}/*" ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads" ], "Resource": [ "arn:aws:s3:::USD{BUCKET}" ] } ] } EOF Associate this policy to the new user: Example USD aws iam put-user-policy \ --user-name USDUSER \ --policy-name velero \ --policy-document file://velero-policy.json Create an access key: Example USD aws iam create-access-key --user-name USDUSER --output=json | jq -r '.AccessKey | [ "export AWS_ACCESS_KEY_ID=" + .AccessKeyId, "export AWS_SECRET_ACCESS_KEY=" + .SecretAccessKey ] | join("\n")' Create a credentials file with your AWS secret key information: Syntax cat << EOF > ./credentials-velero [default] aws_access_key_id=USDAWS_ACCESS_KEY_ID aws_secret_access_key=USDAWS_SECRET_ACCESS_KEY EOF Log in to the OpenShift web console with a user that has the cluster-admin role. From the Administrator perspective, expand the Operators navigation menu, and click OperatorHub . In the search field, type oadp , and click the OADP Operator tile provided by Red Hat. Click the Install button to show the operator details. Accept the default values, click Install on the Install Operator page, and wait for the installation to finish. After the operator installation finishes, from your workstation terminal, create a secret resource for OpenShift with your AWS credentials: Example USD oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero From the OpenShift web console, click the View Operator button. Click Create instance on the DataProtectionApplication (DPA) tile. On the Create DataProtectionApplication page, select YAML view . Edit the following values in the resource file: Under the metadata section, replace velero-sample with velero . Under the spec.configuration.nodeAgent section, replace restic with kopia . Under the spec.configuration.velero section, add resourceTimeout: 10m . Under the spec.configuration.velero.defaultPlugins section, add - csi . Under the spec.snapshotLocations section, replace the us-west-2 value with your AWS regional value. Under the spec.backupLocations section, replace the us-east-1 value with your AWS regional value. Under the spec.backupLocations.objectStorage section, replace my-bucket-name with your bucket name. Replace velero with your bucket prefix name, if you use a different prefix. Click the Create button. Additional resources About installing OADP . Creating a S3 bucket on AWS . Organizing objects by using prefixes . 1.2. Backing up your Trusted Artifact Signer data With the OpenShift API Data Protection (OADP) operator installed and an instance deployed, you can create a volume snapshot resource, and a backup resource to backup your Red Hat Trusted Artifact Signer data. Prerequisites Red Hat OpenShift Container Platform version 4.13 or later. Access to the OpenShift web console with the cluster-admin role. Installation of the OADP operator . A workstation with the oc binary installed. Procedure Open a terminal on your workstation, and log in to OpenShift: Syntax oc login --token= TOKEN --server= SERVER_URL_AND_PORT Example USD oc login --token=sha256~ZvFDBvoIYAbVECixS4-WmkN4RfnNd8Neh3y1WuiFPXC --server=https://example.com:6443 Note You can find your login token and URL for use on the command line from the OpenShift web console. Log in to the OpenShift web console. Click your user name, and click Copy login command . Offer your user name and password again, if asked, and click Display Token to view the command. Find and edit the VolumeSnapshotClass resource: Example USD oc get VolumeSnapshotClass -n openshift-adp USD oc edit VolumeSnapshotClass csi-aws-vsc -n openshift-adp Update the following values in the resource file: Under the metadata.labels section, add the velero.io/csi-volumesnapshot-class: "true" label. Save your changes, and quit the editor. Create a Backup resource: Example USD cat <<EOF | oc apply -f - apiVersion: velero.io/v1 kind: Backup metadata: name: rhtas-backup labels: velero.io/storage-location: velero-1 namespace: openshift-adp spec: schedule: 0 7 * * * hooks: {} includedNamespaces: - trusted-artifact-signer includedResources: [] excludedResources: [] snapshotMoveData: true storageLocation: velero-1 ttl: 720h0m0s EOF Add the schedule property to enable Cron scheduling for running this backup. In the example, this backup resource runs everyday at 7:00 a.m. By default, all resources are backed up within the trusted-artifact-signer namespace. You can specify what resources you want to include or exclude by using the includeResources or excludedResources properties respectively. Important Depending on the storage class of the backup target, persistent volumes cannot be actively in-use for the backup to be successful. 1.3. Restoring your Trusted Artifact Signer data With the Red Hat Trusted Artifact Signer (RHTAS) and OpenShift API Data Protection (OADP) operators installed, and a backup resource for RHTAS namespace, you can restore your data to an OpenShift cluster. Prerequisites Red Hat OpenShift Container Platform version 4.13 or later. Access to the OpenShift web console with the cluster-admin role. Installation of the RHTAS operator . Installation of the OADP operator . A backup resource of the trusted-artifact-signer namespace structure. A workstation with the oc binary installed. Procedure Disable the RHTAS operator: Example USD oc scale deploy rhtas-operator-controller-manager --replicas=0 -n openshift-operators Create the Restore resource: Example USD cat <<EOF | oc apply -f - apiVersion: velero.io/v1 kind: Restore metadata: name: rhtas-restore namespace: openshift-adp spec: backupName: rhtas-backup includedResources: [] restoreStatus: includedResources: - securesign.rhtas.redhat.com - trillian.rhtas.redhat.com - ctlog.rhtas.redhat.com - fulcio.rhtas.redhat.com - rekor.rhtas.redhat.com - tuf.rhtas.redhat.com - timestampauthority.rhtas.redhat.com excludedResources: - pod - deployment - nodes - route - service - replicaset - events - cronjob - events.events.k8s.io - backups.velero.io - restores.velero.io - resticrepositories.velero.io - pods - deployments restorePVs: true existingResourcePolicy: update EOF If restoring your RHTAS data to a different OpenShift cluster, do the following steps. Delete the secret for the Trillian database: Example USD oc delete secret securesign-sample-trillian-db-tls USD oc delete pod trillian-db-xxx Note The RHTAS operator recreates the secret and restarts the pod. Run the restoreOwnerReferences.sh script. Enable the RHTAS operator: Example USD oc scale deploy rhtas-operator-controller-manager --replicas=1 -n openshift-operators Important Immediately starting the RHTAS operator after starting the restore ensures the claim of the persistent volume. 1.4. Restore owner references script This Bash script is for restoring the ownerReferences when restoring Red Hat Trusted Artifact Signer (RHTAS) data to a different OpenShift cluster. #!/bin/bash # List of resources to check RESOURCES=("Fulcio" "Rekor" "Trillian" "TimestampAuthority" "CTlog" "Tuf") function validate_owner() { local RESOURCE=USD1 local ITEM=USD2 local OWNER_NAME=USD3 # Check all the labels exist and are the same LABELS=("app.kubernetes.io/instance" "app.kubernetes.io/part-of" "velero.io/backup-name" "velero.io/restore-name") for LABEL in "USD{LABELS[@]}"; do PARENT_LABEL=USD(oc get Securesign "USDOWNER_NAME" -o json | jq -r ".metadata.labels[\"USDLABEL\"]") CHILD_LABEL=USD(oc get USDRESOURCE "USDITEM" -o json | jq -r ".metadata.labels[\"USDLABEL\"]") if [[ -z "USDCHILD_LABEL" || USDCHILD_LABEL == "null" ]]; then echo " USDLABEL label missing in USDRESOURCE" return 1 elif [[ -z "USDPARENT_LABEL" || USDPARENT_LABEL == "null" ]]; then echo " USDLABEL label missing in Securesign" return 1 elif [[ "USDCHILD_LABEL" != "USDPARENT_LABEL" ]]; then echo " USDLABEL labels not matching: USDCHILD_LABEL != USDPARENT_LABEL" return 1 fi done return 0 } for RESOURCE in "USD{RESOURCES[@]}"; do echo "Checking USDRESOURCE ..." # Get all resources missing ownerReferences MISSING_REFS=USD(oc get USDRESOURCE -o json | jq -r '.items[] | select(.metadata.ownerReferences == null) | .metadata.name') for ITEM in USDMISSING_REFS; do echo " Missing ownerReferences in USDRESOURCE/USDITEM" # Find the expected owner based on labels OWNER_NAME=USD(oc get USDRESOURCE "USDITEM" -o json | jq -r '.metadata.labels["app.kubernetes.io/name"]') if [[ -z "USDOWNER_NAME" || "USDOWNER_NAME" == "null" ]]; then echo " Skipping USDRESOURCE/USDITEM: name not found in labels" continue fi if ! validate_owner USDRESOURCE USDITEM USDOWNER_NAME; then echo " Skipping ..." continue fi # Try to get the owner's UID from Securesign OWNER_UID=USD(oc get Securesign "USDOWNER_NAME" -o jsonpath='{.metadata.uid}' 2>/dev/null) if [[ -z "USDOWNER_UID" || "USDOWNER_UID" == "null" ]]; then echo " Failed to find Securesign/USDOWNER_NAME UID, skipping ..." continue fi echo " Found owner: Securesign/USDOWNER_NAME (UID: USDOWNER_UID)" # Patch the object with the restored ownerReference oc patch USDRESOURCE "USDITEM" --type='merge' -p "{ \"metadata\": { \"ownerReferences\": [ { \"apiVersion\": \"rhtas.redhat.com/v1alpha1\", \"kind\": \"Securesign\", \"name\": \"USDOWNER_NAME\", \"uid\": \"USDOWNER_UID\", \"controller\": true, \"blockOwnerDeletion\": true } ] } }" echo "Restored ownerReferences for USDRESOURCE/USDITEM" done done echo "Done"

Commands:
login --token= TOKEN --server= SERVER_URL_AND_PORT
oc login --token=sha256~ZvFDBvoIYAbVECixS4-WmkN4RfnNd8Neh3y1WuiFPXC --server=https://example.com:6443
export BUCKET= NEW_BUCKET_NAME export REGION= AWS_REGION_ID export USER= OADP_USER_NAME aws s3api create-bucket --bucket USDBUCKET --region USDREGION --create-bucket-configuration LocationConstraint=USDREGION
export BUCKET=example-bucket-name export REGION=us-east-1 export USER=velero USD aws s3api create-bucket --bucket USDBUCKET --region USDREGION --create-bucket-configuration LocationConstraint=USDREGION
aws iam create-user --user-name USDUSER
cat > velero-policy.json <<EOF { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeVolumes", "ec2:DescribeSnapshots", "ec2:CreateTags", "ec2:CreateVolume", "ec2:CreateSnapshot", "ec2:DeleteSnapshot" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:DeleteObject", "s3:PutObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Resource": [ "arn:aws:s3:::USD{BUCKET}/*" ] }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads" ], "Resource": [ "arn:aws:s3:::USD{BUCKET}" ] } ] } EOF
aws iam put-user-policy --user-name USDUSER --policy-name velero --policy-document file://velero-policy.json
aws iam create-access-key --user-name USDUSER --output=json | jq -r '.AccessKey | [ "export AWS_ACCESS_KEY_ID=" + .AccessKeyId, "export AWS_SECRET_ACCESS_KEY=" + .SecretAccessKey ] | join("\n")'
cat << EOF > ./credentials-velero [default] aws_access_key_id=USDAWS_ACCESS_KEY_ID aws_secret_access_key=USDAWS_SECRET_ACCESS_KEY EOF
oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero
login --token= TOKEN --server= SERVER_URL_AND_PORT
oc login --token=sha256~ZvFDBvoIYAbVECixS4-WmkN4RfnNd8Neh3y1WuiFPXC --server=https://example.com:6443
oc get VolumeSnapshotClass -n openshift-adp oc edit VolumeSnapshotClass csi-aws-vsc -n openshift-adp
cat <<EOF | oc apply -f - apiVersion: velero.io/v1 kind: Backup metadata: name: rhtas-backup labels: velero.io/storage-location: velero-1 namespace: openshift-adp spec: schedule: 0 7 * * * hooks: {} includedNamespaces: - trusted-artifact-signer includedResources: [] excludedResources: [] snapshotMoveData: true storageLocation: velero-1 ttl: 720h0m0s EOF
oc scale deploy rhtas-operator-controller-manager --replicas=0 -n openshift-operators
cat <<EOF | oc apply -f - apiVersion: velero.io/v1 kind: Restore metadata: name: rhtas-restore namespace: openshift-adp spec: backupName: rhtas-backup includedResources: [] restoreStatus: includedResources: - securesign.rhtas.redhat.com - trillian.rhtas.redhat.com - ctlog.rhtas.redhat.com - fulcio.rhtas.redhat.com - rekor.rhtas.redhat.com - tuf.rhtas.redhat.com - timestampauthority.rhtas.redhat.com excludedResources: - pod - deployment - nodes - route - service - replicaset - events - cronjob - events.events.k8s.io - backups.velero.io - restores.velero.io - resticrepositories.velero.io - pods - deployments restorePVs: true existingResourcePolicy: update EOF
oc delete secret securesign-sample-trillian-db-tls oc delete pod trillian-db-xxx
oc scale deploy rhtas-operator-controller-manager --replicas=1 -n openshift-operators
#!/bin/bash List of resources to check RESOURCES=("Fulcio" "Rekor" "Trillian" "TimestampAuthority" "CTlog" "Tuf") function validate_owner() { local RESOURCE=USD1 local ITEM=USD2 local OWNER_NAME=USD3 # Check all the labels exist and are the same LABELS=("app.kubernetes.io/instance" "app.kubernetes.io/part-of" "velero.io/backup-name" "velero.io/restore-name") for LABEL in "USD{LABELS[@]}"; do PARENT_LABEL=USD(oc get Securesign "USDOWNER_NAME" -o json | jq -r ".metadata.labels[\"USDLABEL\"]") CHILD_LABEL=USD(oc get USDRESOURCE "USDITEM" -o json | jq -r ".metadata.labels[\"USDLABEL\"]") if [[ -z "USDCHILD_LABEL" || USDCHILD_LABEL == "null" ]]; then echo " USDLABEL label missing in USDRESOURCE" return 1 elif [[ -z "USDPARENT_LABEL" || USDPARENT_LABEL == "null" ]]; then echo " USDLABEL label missing in Securesign" return 1 elif [[ "USDCHILD_LABEL" != "USDPARENT_LABEL" ]]; then echo " USDLABEL labels not matching: USDCHILD_LABEL != USDPARENT_LABEL" return 1 fi done return 0 } for RESOURCE in "USD{RESOURCES[@]}"; do echo "Checking USDRESOURCE ..." # Get all resources missing ownerReferences MISSING_REFS=USD(oc get USDRESOURCE -o json | jq -r '.items[] | select(.metadata.ownerReferences == null) | .metadata.name') for ITEM in USDMISSING_REFS; do echo " Missing ownerReferences in USDRESOURCE/USDITEM" # Find the expected owner based on labels OWNER_NAME=USD(oc get USDRESOURCE "USDITEM" -o json | jq -r '.metadata.labels["app.kubernetes.io/name"]') if [[ -z "USDOWNER_NAME" || "USDOWNER_NAME" == "null" ]]; then echo " Skipping USDRESOURCE/USDITEM: name not found in labels" continue fi if ! validate_owner USDRESOURCE USDITEM USDOWNER_NAME; then echo " Skipping ..." continue fi # Try to get the owner's UID from Securesign OWNER_UID=USD(oc get Securesign "USDOWNER_NAME" -o jsonpath='{.metadata.uid}' 2>/dev/null) if [[ -z "USDOWNER_UID" || "USDOWNER_UID" == "null" ]]; then echo " Failed to find Securesign/USDOWNER_NAME UID, skipping ..." continue fi echo " Found owner: Securesign/USDOWNER_NAME (UID: USDOWNER_UID)" # Patch the object with the restored ownerReference oc patch USDRESOURCE "USDITEM" --type='merge' -p "{ \"metadata\": { \"ownerReferences\": [ { \"apiVersion\": \"rhtas.redhat.com/v1alpha1\", \"kind\": \"Securesign\", \"name\": \"USDOWNER_NAME\", \"uid\": \"USDOWNER_UID\", \"controller\": true, \"blockOwnerDeletion\": true } ] } }" echo "Restored ownerReferences for USDRESOURCE/USDITEM" done done echo "Done"