Question: How do I troubleshoot IRSA errors in Amazon EKS?

When I use AWS Identity and Access Management (IAM) roles for service accounts (IRSA) with Amazon Elastic Kubernetes Service (Amazon EKS), I get errors.

Short descriptionTo troubleshoot issues with IRSA in Amazon EKS, complete one or more of the following actions based on your use case:Check the formatting of the IAM Amazon Resource Name (ARN).Check whether you have an IAM OpenID Connect (OIDC) provider for your AWS account.Verify the audience of the OIDC provider.Verify that you created the OIDC resource with a root certificate thumbprint.Check the configuration of your IAM role's trust policy.Verify that your pod identity webhook configuration exists and is valid.Verify that your pod identity webhook is injecting environment variables to your pods using IRSA.Verify that you're using supported AWS SDKs.

ResolutionCheck the formatting of the IAM ARNIf you set your IAM ARN in the relative service account annotation with incorrect formatting, then you get the following error:An error occurred (ValidationError) when calling the AssumeRoleWithWebIdentityoperation: Request ARN is invalidHere's an example of an incorrect ARN format: eks.amazonaws.com/role-arn: arn:aws:iam::::1234567890:role/exampleThis ARN format is incorrect because it has an extra colon ( : ). To verify correct ARN format, see IAM ARNs.Check if you have an IAM OIDC provider for your AWS accountIf you didn't create an OIDC provider, then you get the following error:An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: No OpenIDConnect provider found in your account for https://oidc.eks.region.amazonaws.com/id/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxTo troubleshoot this, get the IAM OIDC provider URL:aws eks describe-cluster --name cluster name --query "cluster.identity.oidc.issuer" --output textNote: Replace cluster name with your cluster name.You get an output that's similar to the following example:https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041ETo list the IAM OIDC providers, run the following command:aws iam list-open-id-connect-providers | grep EXAMPLED539D4633E53DE1B716D3041ENote: Replace EXAMPLED539D4633E53DE1B716D3041E with the value that the previous command returned.If the OIDC provider doesn't exist, then use the following eksctl command to create one:eksctl utils associate-iam-oidc-provider --cluster cluster name --approveNote: Replace cluster name with your cluster name.You can also use the AWS Management Console to create an IAM OIDC provider for your cluster.Verify the audience of the IAM OIDC providerWhen you create an IAM OIDC provider, you must use sts.amazonaws.com as your audience. If the audience is incorrect, then you get the following error:An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Incorrect token audienceTo check the audience of the IAM OIDC provider, run the following command:aws iam get-open-id-connect-provider --open-id-connect-provider-arn ARN-of-OIDC-providerNote: Replace ARN-of-OIDC-provider with the ARN of your OIDC provider.-or-Complete the following steps:1.    Open the Amazon EKS console.2.    Select the name of your cluster, and then choose the Configuration tab.3.    In the Details section, note the value of the OpenID Connect provider URL.4.    Open the IAM console.5.    In the navigation pane, under Access Management, choose Identity Providers.6.    Select the provider that matches the URL for your cluster.To change the audience, complete the following steps:1.    Open the IAM console.2.    In the navigation pane, under Access Management, choose Identity Providers.3.    Select the provider that matches the URL for your cluster.4.    Choose Actions, and then choose Add audience.5.    Add sts.amazonaws.com.Verify that you created the IAM OIDC resource with a root certificate thumbprintIf you didn't use a root certificate thumbprint to create the OIDC provider, then you get the following error:An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprintNote: Non-root certificate thumbprints renew yearly, and root certificate thumbprints renew every decade. It's a best practice to use a root certificate thumbprint when you create an IAM OIDC.Suppose that you used one of the following to create your IAM OIDC:AWS Command Line Interface (AWS CLI)AWS Tools for PowerShellIAM API to create your IAM OIDCIn this case, you must manually obtain the thumbprint. If you created your IAM OIDC in the IAM console, then it's a best practice to manually obtain the thumbprint. With this thumbprint, you can verify that the console fetched the correct IAM OIDC.Obtain the root certificate thumbprint and its expiration date:echo | openssl s\_client -servername oidc.eks.your-region-code.amazonaws.com -showcerts -connect oidc.eks.your-region-code.amazonaws.com:443 2>/dev/null | awk '/-----BEGIN CERTIFICATE-----/{cert=""} {cert=cert $0 "\\n"} /-----END CERTIFICATE-----/{last\_cert=cert} END{printf "%s", last\_cert}' | openssl x509 -fingerprint -noout -dates | sed 's/://g' | awk -F= '{print tolower($2)}'Note: Replace your-region-code with the AWS Region that your cluster is located in.You receive an output that's similar to the following example:9e99a48a9960b14926bb7f3b02e22da2b0ab7280 sep 2 000000 2009 gmt jun 28 173916 2034 gmtIn this output, 9e99a48a9960b14926bb7f3b02e22da2b0ab7280 is the thumbprint, sep 2 000000 2009 gmt is the certificate start date, and jun 28 173916 2034 is the certificate expiration date.Check the configuration of your IAM role's trust policyIf the trust policy of the IAM role is misconfigured, then you get the following error:An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentityTo resolve this issue, make sure that you're using the correct IAM OIDC provider. If the IAM OIDC provider is correct, then use the IAM role configuration guide to check if the trust policy's conditions are correctly configured.Verify that your pod identity webhook configuration exists and is validThe pod identity webhook is responsible for injecting the necessary environment variables and projected volume. If you accidentally deleted or changed your webhook configuration, then IRSA stops working.To verify that your webhook configuration exists and is valid, run the following command:kubectl get mutatingwebhookconfiguration pod-identity-webhook  -o yamlVerify that your pod identity webhook is injecting environment variables to your pods that use IRSARun one of the following commands:kubectl get pod <pod-name> -n <ns> -o yaml | grep aws-iam-token-or-kubectl get pod <pod-name> -n <ns> -o yaml | grep AWS\_WEB\_IDENTITY\_TOKEN\_FILEVerify that you're using supported AWS SDKsMake sure that you're using an AWS SDK version that supports assuming an IAM role through the OIDC web identity token file.

Related informationWhy can't I use an IAM role for the service account in my Amazon EKS pod?How do I troubleshoot an OIDC provider and IRSA in Amazon EKS?