Question: How do I troubleshoot an OIDC provider and IRSA in Amazon EKS?

My pods can't use the AWS Identity and Access Management (IAM) role permissions with the Amazon Elastic Kubernetes Service (Amazon EKS) account token.

Short descriptionTo troubleshoot issues with the OpenID Connect (OIDC) provider and IAM roles for service accounts (IRSA) in Amazon EKS, complete the steps in one of the following sections:Check if you have an existing IAM OIDC provider for your clusterCheck if your IAM role has an attached needful IAM policy with required permissionsVerify that the IAM role trust relations are correctly setCheck if you created a service accountVerify that the service account has the correct IAM role annotationsVerify that you correctly specified the serviceAccountName in your podCheck the environment variables and permissionsVerify that the application uses a supported AWS SDKCheck the pod user and groupRecreate podsVerify that the audience is correctVerify that you configured the correct thumbprintFor the AWS China Region, check the AWS_DEFAULT_REGION environment variable

ResolutionCheck if you have an existing IAM OIDC provider for your clusterIf a provider already exists, then you receive an error that's similar to the following message:"WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E\n\tstatus code: 400"1.    Check your cluster's OIDC provider URL:$ aws eks describe-cluster --name cluster_name --query "cluster.identity.oidc.issuer" --output textSee the following example output:https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E2.    List the IAM OIDC providers in your account. Replace EXAMPLED539D4633E53DE1B716D3041E (include < >) with the value returned from the previous command:aws iam list-open-id-connect-providers | grep EXAMPLED539D4633E53DE1B716D3041ESee the following example output:"Arn": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"If the preceding command returns an output, then you already have a provider for your cluster. If the command doesn't return an output, then you must create an IAM OIDC provider.Check if your IAM role has an attached needful IAM policy with required permissions1.    Open the IAM console.2.    In the navigation pane, choose Roles.3.    Choose the role that you want to verify.4.    Under the Permissions tab, verify if this role has the required policy attached.Verify that the IAM role trust relations are correctly setWith the AWS Management Console:1.    Open the IAM console.2.    In the navigation pane, choose Roles.3.    Choose the role that you want to check.4.    Choose the Trust Relationships tab to verify that the format of your policy matches the format of the following JSON policy:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"      },      "Action": "sts:AssumeRoleWithWebIdentity",      "Condition": {        "StringEquals": {          "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME",          "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud": "sts.amazonaws.com"        }      }    }  ]}To verify trust relations, run following command with your role name in the AWS Command Line Interface (AWS CLI):$ aws iam get-role --role-name EKS-IRSANote: Replace EKS-IRSA with your IAM role name.In the output JSON, look for the AssumeRolePolicyDocument section.See the following example output:{  "Role": {    "Path": "/",    "RoleName": "EKS-IRSA",    "RoleId": "AROAQ55NEXAMPLELOEISVX",    "Arn": "arn:aws:iam::ACCOUNT_ID:role/EKS-IRSA",    "CreateDate": "2021-04-22T06:39:21+00:00",    "AssumeRolePolicyDocument": {      "Version": "2012-10-17",      "Statement": [        {          "Effect": "Allow",          "Principal": {            "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"          },          "Action": "sts:AssumeRoleWithWebIdentity",          "Condition": {            "StringEquals": {              "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud": "sts.amazonaws.com",              "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME"            }          }        }      ]    },    "MaxSessionDuration": 3600,    "RoleLastUsed": {      "LastUsedDate": "2021-04-22T07:01:15+00:00",      "Region": "AWS_REGION"    }  }}Note: Check that you have specified the correct AWS Region, Kubernetes service account name, and Kubernetes namespace.Check if you created a service accountUse the following command:$ kubectl get sa -n YOUR_NAMESPACENote: Replace YOUR_NAMESPACE with your Kubernetes namespace.See the following example output:NAME      SECRETS   AGEdefault   1         28dirsa      1         66mIf you don't have a service account, see Configure service accounts for pods (from the Kubernetes website).Verify that the service account has the correct IAM role annotationsUse the following command:$ kubectl describe sa irsa -n YOUR_NAMESPACENote: Replace irsa with your Kubernetes service account name. Replace YOUR_NAMESPACE with your Kubernetes namespace.See the following example output:Name:                irsaNamespace:           defaultLabels:              noneAnnotations:         eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAMEImage pull secrets:  noneMountable secrets:   irsa-token-v5rtcTokens:              irsa-token-v5rtcEvents:              noneVerify that you correctly specified the serviceAccountName in your podUse the following command:$ kubectl get pod POD_NAME  -o yaml -n YOUR_NAMESPACE| grep -i serviceAccountName:Note: Replace POD_NAME and YOUR_NAMESPACE with your Kubernetes pod and namespace.See the following example output:serviceAccountName: irsaCheck the environment variables and permissionsLook for AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE in the pod's environment variables:$ kubectl -n YOUR_NAMESPACE exec -it POD_NAME -- env | grep AWSSee the following example output:AWS_REGION=ap-southeast-2AWS_ROLE_ARN=arn:aws:iam::111122223333:role/EKS-IRSAAWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/tokenAWS_DEFAULT_REGION=ap-southeast-2Verify that the application uses a supported AWS SDKThe SDK version must be greater than or equal to the following values:Java (Version 2) — 2.10.11Java — 1.11.704Go — 1.23.13Python (Boto3) — 1.9.220Python (botocore) — 1.12.200AWS CLI — 1.16.232Node — 3.15.0Ruby — 2.11.345C++ — 1.7.174.NET — 3.3.659.1PHP — 3.110.7To check for latest supported SDK version, see Using a supported AWS SDK.Check the pod user and groupUse the following command:$ kubectl exec -it POD_NAME -- iduid=0(root) gid=0(root) groups=0(root)Note: By default, only containers that run as root have the proper file system permissions to read the web identity token file.If your containers aren't running as root, then you can receive the following errors:Error: PermissionError: [Errno 13] Permission denied: '/var/run/secrets/eks.amazonaws.com/serviceaccount/token-or-WebIdentityErr: failed fetching WebIdentity token: \ncaused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission deniedTo provide the proper file system permissions, make sure that your containers run as root. For clusters 1.18 or lower, provide the following security context for the containers in your manifest:apiVersion: apps/v1kind: Deploymentmetadata:  name: my-appspec:  template:    metadata:      labels:        app: my-app    spec:      serviceAccountName: my-app      containers:      - name: my-app        image: my-app:latest      securityContext:        fsGroup: 1337...Note: The fsGroup ID is arbitrary. You can choose any valid group ID. The preceding security context setting is not required for clusters 1.19 or later.Recreate podsIf you created pods before you applied IRSA, then recreate the pods.See the following example command:$ kubectl rollout restart deploy nginxSee the following example output:deployment.apps/nginx restartedFor daemonsets or statefulsets deployments, you can use the following command:$ kubectl rollout restart deploy DEPLOYMENT_NAMEIf you have created only one pod, then you must delete the pod and recreate it.See the following example command to delete the pod:$ kubectl delete pod POD_NAMESee the following example command to recreate the pod:$ kubectl apply -f SPEC_FILENote: Replace SPEC_FILE with your Kubernetes manifest file path and file name.Verify that the audience is correctIf you created the OIDC provider with the incorrect audience, then you receive the following error:Error - An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Incorrect token audienceCheck the IAM identity provider for your cluster. Your ClientIDList is sts.amazonaws.com:$ aws iam get-open-id-connect-provider --open-id-connect-provider-arn arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041ESee the following example output:{  "Url": "oidc.eks.AWS_REGION.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E",  "ClientIDList": [    "sts.amazonaws.com"  ],  "ThumbprintList": [    "9e99a48a9960b14926bb7f3b02e22da2b0ab7280"  ],  "CreateDate": "2021-01-21T04:29:09.788000+00:00",  "Tags": []}Verify that you configured the correct thumbprintIf the thumbprint that's configured in the IAM OIDC is not correct, you can receive the following error:failed to retrieve credentials caused by: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprintTo automatically configure the correct thumbprint, use eksctl or the AWS Management Console to create the IAM identity provider. For other ways to obtain a thumbprint, see Obtaining the thumbprint for an OpenID Connect identity provider.For the AWS China Region, check the AWS_DEFAULT_REGION environment variableIf you use IRSA for a pod or daemonset that's deployed to a cluster in the AWS China Region, then set the AWS_DEFAULT_REGION environment variable in the pod specification. If you don't, the pod or daemonset can receive the following error:An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalidUse the following example to add the AWS_DEFAULT_REGION environment variable to your pod or daemonset specification:apiVersion: apps/v1kind: Deploymentmetadata:  name: my-appspec:  template:    metadata:      labels:        app: my-app    spec:      serviceAccountName: my-app      containers:      - name: my-app        image: my-app:latest        env:        - name: AWS_DEFAULT_REGION          value: "AWS_REGION"...