Question: Why do I get the "webidentityerr" error when using AWS Load Balancer Controller in Amazon EKS?

When I try to use the AWS Load Balancer Controller in Amazon Elastic Kubernetes Service (Amazon EKS), I receive a "WebIdentityErr" error message.

Short descriptionWhen you use the AWS Load Balancer Controller in Amazon EKS, you might see the following error:"failed to find existing LoadBalancer due to WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403"The error occurs for the following reasons:Incorrect service account configurationsIncorrect trust relationship of the AWS Identity and Access Management (IAM) role that you use in the service accountWith AWS Load Balancer Controller, worker nodes perform the tasks. You must use IAM permissions to grant these worker nodes access to the Application Load Balancer or Network Load Balancer resources. To set up the IAM permissions, use IAM roles for the service account. Or, attach the IAM permissions directly to the worker node's IAM roles. For more information, see Load Balancer Controller installation on the Kubernetes website.

ResolutionIncorrect service account configurationsTo check if you properly configured your service account, complete the following steps:1.    Verify the service account name that's defined in your deployment:kubectl describe deploy LOAD_BALANCER_CONTROLLER -n kube-system | grep -i "Service Account"2.    Describe the service account:kubectl describe sa LOAD_BALANCER_CONTROLLER -n kube-system3.    Verify the service account annotation for the IAM role:Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/AMAZON_EKS_LOAD_BALANCER_CONTROLLER_ROLE4.    If this annotation is missing or incorrect, then update the annotation. Make sure that you properly associated the IAM role to a service account:kubectl annotate serviceaccount -n SERVICE_ACCOUNT_NAMESPACE SERVICE_ACCOUNT_NAME \ eks.amazonaws.com/role-arn=arn:aws:iam::ACCOUNT_ID:role/IAM_ROLE_NAME5.    Delete and recreate any existing Amazon EKS pods that are associated with the service account. Be sure to apply the credential environment variables:kubectl delete pods AWS_LOAD_BALANCER_CONTROLLER_POD -n KUBBE_SYSTEMIncorrect trust relationship between the IAM role used and service accountThe following examples show some common mistakes when establishing the trust relationship between your IAM role and service account:Example 1: The IAM role or trust relationship isn't properly defined for the "sts:AssumeRoleWithWebIdentity" actionVerify that the trust relationship is properly defined for the sts:AssumeRoleWithWebIdentity action (and not the sts:AssumeRole action).The following example is a trust relationship that isn't properly defined:{      "Sid": "",      "Effect": "Allow",      "Principal": {        "Service": "xxxxx.amazonaws.com"      },      "Action": "sts:AssumeRole"}To correct this issue, define the trust relationship for the sts:AssumeRoleWithWebIdentity action:{    "Version": "2012-10-17",    "Statement": [      {        "Effect": "Allow",        "Principal": {          "Federated": "arn:aws:iam::AWS_ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC-PROVIDER_ID"        },        "Action": "sts:AssumeRoleWithWebIdentity",        "Condition": {          "StringEquals": {            "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC_PROVIDER_ID:sub": "system:serviceaccount:kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",            "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"          }        }      }    ]  }Note: Be sure to replace all variables with your own values.To use the same IAM role for multiple clusters in one account, define the trust relationship as in the following example:{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Principal": {        "Federated": "arn:aws:iam::AWS-ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC-PROVIDER_ID"      },      "Action": "sts:AssumeRoleWithWebIdentity",      "Condition": {        "StringEquals": {          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:sub": "system:serviceaccount: kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"        }      }    },    {      "Effect": "Allow",      "Principal": {        "Federated": "arn:aws:iam::AWS_ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID"      },      "Action": "sts:AssumeRoleWithWebIdentity",      "Condition": {        "StringEquals": {          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID:sub": "system:serviceaccount: kube-system:LOAD_BALANCER_CONTROLLER_SERVICE_ACCOUNT",          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_2_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"        }      }    }  ]}Example 2: Incorrect OIDC provider ID when creating an Amazon EKS clusterProperly create and verify an OpenID Connect (OIDC) provider for your Amazon EKS cluster. Verify that the OIDC provider ID and the associated AWS Region are correctly listed. Otherwise, you receive a WebIdentityErr error.Example 3: Service account name or its namespace not correctly enteredWhen you update your AWS Load Balancer Controller deployment, enter the correct service account name and its namespace.Example 4: Missing "sts.amazonaws.com" steps from the trust relationshipIf the service role that's associated with your EKS pod can't perform the STS operation on the AssumeRoleWithWebIdentity action, then update the trust relationship. To perform an STS operation, the trust relationship must include sts.amazonaws.com:"Condition": {        "StringEquals": {          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:sub": "system:serviceaccount:kube-system:LOAD_BALANCER_CONTROLLER_ACCOUNT",          "oidc.eks.REGION.amazonaws.com/id/EKS_CLUSTER_1_OIDC_PROVIDER_ID:aud": "sts.amazonaws.com"        }}For more information about IAM conditions with multiple keys or values, see Creating a condition with multiple keys or values.