Question: Why can't I use an IAM role for the service account in my Amazon EKS pod?

I try to use an AWS Identity and Access Management (IAM) role for a service account. But my Amazon Elastic Kubernetes Service (Amazon EKS) pod fails to assume the assigned IAM role. I receive an authorization error. Or, my pod tries to use the default IAM role assigned to the Amazon EKS node instead of the IAM role assigned to my pod.

Short descriptionTo resolve issues using the IAM roles for service accounts feature, try the following:Verify that you have an IAM OpenID Connect (OIDC) identity provider for your Amazon EKS clusterValidate your IAM role policies and trust policy configurationConfirm that your service account exists and has a properly formatted annotation for the IAM role's ARNUse a test pod to verify that the service account is workingNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

ResolutionVerify that you have an IAM OIDC identity provider for your Amazon EKS clusterCreate an IAM OIDC provider for your cluster, if you don't already have one. You must have an OIDC identity provider for your cluster to use an IAM role for your service account.Next, verify that the OIDC identity provider is configured correctly:1.    Open the IAM console, and then choose Identity providers from the navigation pane.2.    In the Provider column, identify and note the OIDC provider URL.3.    In a separate tab or window, open the Amazon EKS console, and then choose Clusters from the navigation pane.4.    Choose your cluster, and then choose the Configuration tab.5.    In the Details section, note the value of the OpenID Connect provider URL property.6.    Verify that the OIDC provider URL from the Amazon EKS console (step 5) matches the OIDC provider URL from the IAM console (step 2).If the OIDC provider URL for your Amazon EKS cluster doesn't match any of the OIDC provider URLs in the IAM console, then you must create a new IAM OIDC provider.Validate your IAM role policies and trust policy configurationYour IAM role might not have the full range of permissions that you expect are assigned to that role. Your IAM role's trust relationship policy could also have syntax errors, if you created your IAM role using the AWS Management Console or AWS CLI.To validate your IAM role policies and check for syntax errors in your trust policy, do the following:1.    Open the IAM console.2.    In the navigation pane, choose Roles, and then choose your role.3.    Choose the Permissions tab on your role's page, and then verify that all your required permissions are assigned to the role.4.    Choose the Trust Relationships tab, and then choose Edit trust relationship.5.    In the policy document for your trust relationship, verify that the format of your policy matches the format of the following JSON policy:"Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Federated": "arn:aws:iam::your-account-id:oidc-provider/oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER"      },      "Action": "sts:AssumeRoleWithWebIdentity",      "Condition": {        "StringEquals": {          "oidc.eks.your-region-code.amazonaws.com/id/EXAMPLE_OIDC_IDENTIFIER:sub": "system:serviceaccount:your-namespace:your-service-account"        }      }    }  ]}In the preceding JSON policy, review the format of the Federated property line and the StringEquals property line. In the Federated line, confirm that your AWS Region code (your-region-code), account ID (your-account-id), and unique OIDC identifier (EXAMPLE_OIDC_IDENTIFIER) are formatted correctly. In the StringEquals line, confirm that your Region code (your-region-code), OIDC unique identifier (EXAMPLE_OIDC_IDENTIFIER), Kubernetes namespace (your-namespace), and Kubernetes service account name (your-namespace) are formatted correctly.6.    If you edit your policy document to correct formatting errors, then choose Update Trust Policy.Confirm that your service account exists and has a properly formatted annotation for the IAM role's ARN1.    Confirm that your Kubernetes service account exists:$ kubectl get serviceaccount YOUR_ACCOUNT_NAME -n YOUR_NAMESPACE -o yamlNote: Replace YOUR_ACCOUNT_NAME with your account name. Replace YOUR_NAMESPACE with your namespace.If the preceding command doesn't return a service account name, then create a service account (from the Kubernetes website).2.    Confirm that your service account has the name that you expect and that its role-arn annotation is correctly formatted. For example:apiVersion: v1kind: ServiceAccountmetadata:  annotations:    eks.amazonaws.com/role-arn: arn:aws:iam::012345678912:role/my-example-iam-role  name: my-example-serviceaccount  namespace: my-test-namespaceUse a test pod to verify that the service account is workingYou can verify if the service account is working correctly by running a test pod. Then, check if the pod can mount environment variables correctly and can assume the specified IAM role.Note: You can experience an issue related to your application's credentials even if the IAM roles for service accounts feature work correctly. You can also experience this issue if your pod has the correct environment variables. To prevent this issue, you can use an AWS CLI container image for verification.1.    Create a local YAML file called awscli-pod.yaml. For example:apiVersion: v1kind: Podmetadata:  name: awscli  labels:    app: awsclispec:  serviceAccountName: YOUR_SERVICE_ACCOUNT  containers:  - image: amazon/aws-cli    command:      - "sleep"      - "604800"    imagePullPolicy: IfNotPresent    name: awscli  restartPolicy: AlwaysNote: Replace YOUR_SERVICE_ACCOUNT with your Kubernetes service account name.2.    Create the test pod (from the YAML file) in your namespace:$ kubectl apply -f ./awscli-pod.yaml -n YOUR_NAMESPACENote: Replace YOUR_NAMESPACE with your namespace.3.    Confirm that the awscli pod has the correct environment variables:$ kubectl exec -n YOUR_NAMESPACE awscli env | grep AWSOutput:AWS_ROLE_ARN=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAMEAWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token4.    Confirm that the test pod is using the correct IAM role:$ kubectl exec -it awscli -n YOUR_NAMESPACE -- aws sts get-caller-identityOutput:{    "UserId": "REDACTEDY471234567890:botocore-session-1632772568",    "Account": "012345678912",    "Arn": "arn:aws:sts::012345678912:assumed-role/your-iam-role/botocore-session-1632772568"}Note the Arn value, including the IAM role name (your-iam-role), that you receive in the output from the command in step 4.5.    Delete the awscli pod after you verify the IAM role:$ kubectl delete -f ./awscli-pod.yaml -n YOUR_NAMESPACEIf the awscli pod shows the correct IAM role, then the IAM roles for service accounts feature are working correctly.The preceding steps confirm that the IAM token is correctly mounted to the pod. If your application still can't use the token file correctly, then there is likely an issue at the application or SDK level. This issue is likely related to how the application ingests AWS credentials. For more information, see Using the Default Credential Provider Chain and Credentials (from the Boto3 website).