Chapter 11. Tutorial: Using AWS Secrets Manager CSI on ROSA with STS The AWS Secrets and Configuration Provider (ASCP) provides a way to expose AWS Secrets as Kubernetes storage volumes. With the ASCP, you can store and manage your secrets in Secrets Manager and then retrieve them through your workloads running on Red Hat OpenShift Service on AWS (ROSA). 11.1. Prerequisites Ensure that you have the following resources and tools before starting this process: A ROSA cluster deployed with STS Helm 3 aws CLI oc CLI jq CLI Additional environment requirements Log in to your ROSA cluster by running the following command: USD oc login --token=<your-token> --server=<your-server-url> You can find your login token by accessing your cluster in pull secret from Red Hat OpenShift Cluster Manager . Validate that your cluster has STS by running the following command: USD oc get authentication.config.openshift.io cluster -o json \ | jq .spec.serviceAccountIssuer Example output "https://xxxxx.cloudfront.net/xxxxx" If your output is different, do not proceed. See Red Hat documentation on creating an STS cluster before continuing this process. Set the SecurityContextConstraints permission to allow the CSI driver to run by running the following command: USD oc new-project csi-secrets-store USD oc adm policy add-scc-to-user privileged \ system:serviceaccount:csi-secrets-store:secrets-store-csi-driver USD oc adm policy add-scc-to-user privileged \ system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws Create environment variables to use later in this process by running the following command: USD export REGION=USD(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") USD export OIDC_ENDPOINT=USD(oc get authentication.config.openshift.io cluster \ -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||') USD export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text` USD export AWS_PAGER="" 11.2. Deploying the AWS Secrets and Configuration Provider Use Helm to register the secrets store CSI driver by running the following command: USD helm repo add secrets-store-csi-driver \ https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts Update your Helm repositories by running the following command: USD helm repo update Install the secrets store CSI driver by running the following command: USD helm upgrade --install -n csi-secrets-store \ csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver Deploy the AWS provider by running the following command: USD oc -n csi-secrets-store apply -f \ https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml Check that both Daemonsets are running by running the following command: USD oc -n csi-secrets-store get ds \ csi-secrets-store-provider-aws \ csi-secrets-store-driver-secrets-store-csi-driver Label the Secrets Store CSI Driver to allow use with the restricted pod security profile by running the following command: USD oc label csidriver.storage.k8s.io/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted 11.3. Creating a Secret and IAM Access Policies Create a secret in Secrets Manager by running the following command: USD SECRET_ARN=USD(aws --region "USDREGION" secretsmanager create-secret \ --name MySecret --secret-string \ '{"username":"shadowman", "password":"hunter2"}' \ --query ARN --output text); echo USDSECRET_ARN Create an IAM Access Policy document by running the following command: USD cat << EOF > policy.json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], "Resource": ["USDSECRET_ARN"] }] } EOF Create an IAM Access Policy by running the following command: USD POLICY_ARN=USD(aws --region "USDREGION" --query Policy.Arn \ --output text iam create-policy \ --policy-name openshift-access-to-mysecret-policy \ --policy-document file://policy.json); echo USDPOLICY_ARN Create an IAM Role trust policy document by running the following command: Note The trust policy is locked down to the default service account of a namespace you create later in this process. USD cat <<EOF > trust-policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "USD{OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"] } }, "Principal": { "Federated": "arn:aws:iam::USDAWS_ACCOUNT_ID:oidc-provider/USD{OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF Create an IAM role by running the following command: USD ROLE_ARN=USD(aws iam create-role --role-name openshift-access-to-mysecret \ --assume-role-policy-document file://trust-policy.json \ --query Role.Arn --output text); echo USDROLE_ARN Attach the role to the policy by running the following command: USD aws iam attach-role-policy --role-name openshift-access-to-mysecret \ --policy-arn USDPOLICY_ARN 11.4. Create an Application to use this secret Create an OpenShift project by running the following command: USD oc new-project my-application Annotate the default service account to use the STS Role by running the following command: USD oc annotate -n my-application serviceaccount default \ eks.amazonaws.com/role-arn=USDROLE_ARN Create a secret provider class to access our secret by running the following command: USD cat << EOF | oc apply -f - apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: my-application-aws-secrets spec: provider: aws parameters: objects: | - objectName: "MySecret" objectType: "secretsmanager" EOF Create a deployment by using our secret in the following command: USD cat << EOF | oc apply -f - apiVersion: v1 kind: Pod metadata: name: my-application labels: app: my-application spec: volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "my-application-aws-secrets" containers: - name: my-application-deployment image: k8s.gcr.io/e2e-test-images/busybox:1.29 command: - "/bin/sleep" - "10000" volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" readOnly: true EOF Verify the pod has the secret mounted by running the following command: USD oc exec -it my-application -- cat /mnt/secrets-store/MySecret 11.5. Clean up Delete the application by running the following command: USD oc delete project my-application Delete the secrets store csi driver by running the following command: USD helm delete -n csi-secrets-store csi-secrets-store-driver Delete the security context constraints by running the following command: USD oc adm policy remove-scc-from-user privileged \ system:serviceaccount:csi-secrets-store:secrets-store-csi-driver; oc adm policy remove-scc-from-user privileged \ system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws Delete the AWS provider by running the following command: USD oc -n csi-secrets-store delete -f \ https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml Delete AWS Roles and Policies by running the following command: USD aws iam detach-role-policy --role-name openshift-access-to-mysecret \ --policy-arn USDPOLICY_ARN; aws iam delete-role --role-name openshift-access-to-mysecret; aws iam delete-policy --policy-arn USDPOLICY_ARN Delete the Secrets Manager secret by running the following command: USD aws secretsmanager --region USDREGION delete-secret --secret-id USDSECRET_ARN

Commands:
oc login --token=<your-token> --server=<your-server-url>
oc get authentication.config.openshift.io cluster -o json | jq .spec.serviceAccountIssuer
"https://xxxxx.cloudfront.net/xxxxx"
oc new-project csi-secrets-store oc adm policy add-scc-to-user privileged system:serviceaccount:csi-secrets-store:secrets-store-csi-driver oc adm policy add-scc-to-user privileged system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
export REGION=USD(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") export OIDC_ENDPOINT=USD(oc get authentication.config.openshift.io cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|^https://||') export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text` export AWS_PAGER=""
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo update
helm upgrade --install -n csi-secrets-store csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver
oc -n csi-secrets-store apply -f https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
oc -n csi-secrets-store get ds csi-secrets-store-provider-aws csi-secrets-store-driver-secrets-store-csi-driver
oc label csidriver.storage.k8s.io/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted
SECRET_ARN=USD(aws --region "USDREGION" secretsmanager create-secret --name MySecret --secret-string '{"username":"shadowman", "password":"hunter2"}' --query ARN --output text); echo USDSECRET_ARN
cat << EOF > policy.json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], "Resource": ["USDSECRET_ARN"] }] } EOF
POLICY_ARN=USD(aws --region "USDREGION" --query Policy.Arn --output text iam create-policy --policy-name openshift-access-to-mysecret-policy --policy-document file://policy.json); echo USDPOLICY_ARN
cat <<EOF > trust-policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "USD{OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"] } }, "Principal": { "Federated": "arn:aws:iam::USDAWS_ACCOUNT_ID:oidc-provider/USD{OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF
ROLE_ARN=USD(aws iam create-role --role-name openshift-access-to-mysecret --assume-role-policy-document file://trust-policy.json --query Role.Arn --output text); echo USDROLE_ARN
aws iam attach-role-policy --role-name openshift-access-to-mysecret --policy-arn USDPOLICY_ARN
oc new-project my-application
oc annotate -n my-application serviceaccount default eks.amazonaws.com/role-arn=USDROLE_ARN
cat << EOF | oc apply -f - apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: my-application-aws-secrets spec: provider: aws parameters: objects: | - objectName: "MySecret" objectType: "secretsmanager" EOF
cat << EOF | oc apply -f - apiVersion: v1 kind: Pod metadata: name: my-application labels: app: my-application spec: volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "my-application-aws-secrets" containers: - name: my-application-deployment image: k8s.gcr.io/e2e-test-images/busybox:1.29 command: - "/bin/sleep" - "10000" volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" readOnly: true EOF
oc exec -it my-application -- cat /mnt/secrets-store/MySecret
oc delete project my-application
helm delete -n csi-secrets-store csi-secrets-store-driver
oc adm policy remove-scc-from-user privileged system:serviceaccount:csi-secrets-store:secrets-store-csi-driver; oc adm policy remove-scc-from-user privileged system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws
oc -n csi-secrets-store delete -f https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml
aws iam detach-role-policy --role-name openshift-access-to-mysecret --policy-arn USDPOLICY_ARN; aws iam delete-role --role-name openshift-access-to-mysecret; aws iam delete-policy --policy-arn USDPOLICY_ARN
aws secretsmanager --region USDREGION delete-secret --secret-id USDSECRET_ARN