Question: How do I troubleshoot the error "InvalidIdentityToken - OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint" when I'm using the Amazon EKS IAM role to access the service account?

The thumbprint for my Amazon Elastic Kubernetes Service (Amazon EKS) cluster has changed, causing the Application Load Balancer controller to fail updates.-or-My Amazon EKS pods are in failed state with the following error: "WebIdentityErr: failed to retrieve credentials\r\ncaused by: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint\r\n".

ResolutionAll Amazon EKS service accounts use the OpenID Connect (OIDC) to authenticate. When you create an AWS Identity and Access Management (IAM) OIDC provider for your Amazon EKS cluster, the thumbprint generated uses the root certificate. The Amazon root Certificate Authority (CA) has a validation period of around 25 years. You get the "HTTPS certificate doesn't match configured thumbprint" error under either of the following conditions:The thumbprint used in the OIDC provider is expired.The thumbprint doesn't match the CA.To troubleshoot this issue and obtain a thumbprint, you must install and configure the OpenSSL command line tool.To obtain a thumbprint for the OIDC provider, do the following:1.    Find the URL for the OIDC identity provider (IdP) by doing the following:Open the Amazon EKS console.In the navigation pane, Choose Clusters.Select the cluster that you want to check.Select the Configuration tab.Note the OICD provider URL under the Details section.Example: https://oidc.eks.us-east-1.amazonaws.com/id/1111222233334444555566667777888F/Include /.well-known/openid-configuration at the end of the OICD provider URL to form the URL for the IdP's configuration document.Example: https://oidc.eks.us-east-1.amazonaws.com/id/1111222233334444555566667777888F/.well-known/openid-configurationAccess this URL in a web browser and make a note of the value of jwks_uri from the output. The browser output looks similar to the following:{"issuer":"https://oidc.eks.us-east-1.amazonaws.com/id/1111222233334444555566667777888F","jwks_uri":"https://oidc.eks.us-east-1.amazonaws.com/id/1111222233334444555566667777888F/keys","authorization_endpoint":"urn:kubernetes:programmatic_authorization","response_types_supported":["id_token"],"subject_types_supported":["public"],"claims_supported":["sub","iss"],"id_token_signing_alg_values_supported":["RS256"]}2.    Use the OpenSSL command line tool to run the following command to display all the certificates used:Note: Be sure to replace oidc.eks.us-east-2.amazonaws.com with the domain name returned in Step 1.openssl s_client -connect oidc.eks.us-east-2.amazonaws.com:443 -showcertsThe output looks similar to the following:[root@ip-172-31-1-202 ~]# openssl s_client -connect oidc.eks.us-east-2.amazonaws.com:443 -showcertsCONNECTED(00000003)depth=4 C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authorityverify return:1depth=3 C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2verify return:1depth=2 C = US, O = Amazon, CN = Amazon Root CA 1verify return:1depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazonverify return:1depth=0 CN = *.execute-api.us-east-2.amazonaws.comverify return:1---Certificate chain 0 s:/CN=*.execute-api.us-east-2.amazonaws.com   i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon-----BEGIN CERTIFICATE-----CERTIFICATE Redacted-----END CERTIFICATE----- 1 s:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon   i:/C=US/O=Amazon/CN=Amazon Root CA 1-----BEGIN CERTIFICATE-----CERTIFICATE Redacted-----END CERTIFICATE----- 2 s:/C=US/O=Amazon/CN=Amazon Root CA 1   i:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2-----BEGIN CERTIFICATE-----CERTIFICATE Redacted-----END CERTIFICATE----- 3 s:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2   i:/C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority-----BEGIN CERTIFICATE-----MIIEdTCCA12gAwIBAgIJAKcOSkw0grd/MA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNVBAYTAlVTMSUwIwYDVQQKExxTdGFyZmllbGQgVGVjaG5vbG9naWVzLCBJbmMuMTIwMAYDVQQLEylTdGFyZmllbGQgQ2xhc3MgMiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wOTA5MDIwMDAwMDBaFw0zNDA2MjgxNzM5MTZaMIGYMQswCQYDVQQGEwJVUzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTElMCMGA1UEChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjE7MDkGA1UEAxMyU3RhcmZpZWxkIFNlcnZpY2VzIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IC0gRzIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVDDrEKvlO4vW+GZdfjohTsR8/y8+fIBNtKTrID30892t2OGPZNmCom15cAICyL1l/9of5JUOG52kbUpqQ4XHj2C0NTm/2yEnZtvMaVq4rtnQU68/7JuMauh2WLmo7WJSJR1b/JaCTcFOD2oR0FMNnngRoOt+OQFodSk7PQ5E751bWAHDLUu57fa4657wx+UX2wmDPE1kCK4DMNEffud6QZW0CzyyRpqbn3oUYSXxmTqM6bam17jQuug0DuDPfR+uxa40l2ZvOgdFFRjKWcIfeAg5JQ4W2bHO7ZOphQazJ1FTfhy/HIrImzJ9ZVGif/L4qL8RVHHVAYBeFAlU5i38FAgMBAAGjgfAwge0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0OBBYEFJxfAN+qAdcwKziIorhtSpzyEZGDMB8GA1UdIwQYMBaAFL9ft9HO3R+G9FtVrNzXEMIOqYjnME8GCCsGAQUFBwEBBEMwQTAcBggrBgEFBQcwAYYQaHR0cDovL28uc3MyLnVzLzAhBggrBgEFBQcwAoYVaHR0cDovL3guc3MyLnVzL3guY2VyMCYGA1UdHwQfMB0wG6AZoBeGFWh0dHA6Ly9zLnNzMi51cy9yLmNybDARBgNVHSAECjAIMAYGBFUdIAAwDQYJKoZIhvcNAQELBQADggEBACMd44pXyn3pF3lM8R5V/cxTbj5HD9/GVfKyBDbtgB9TxF00KGu+x1X8Z+rLP3+QsjPNG1gQggL4+C/1E2DUBc7xgQjB3ad1l08YuW3e95ORCLp+QCztweq7dp4zBncdDQh/U90bZKuCJ/Fp1U1ervShw3WnWEQt8jxwmKy6abaVd38PMV4s/KCHOkdp8Hlf9BRUpJVeEXgSYCfOn8J3/yNTd126/+pZ59vPr5KW7ySaNRB6nJHGDn2Z9j8Z3/VyVOEVqQdZe4O/Ui5GjLIAZHYcSNPYeehuVsyuLAOQ1xk4meTKCRlb/weWsKh/NEnfVqn3sF/tM+2MR7cEXAMPLE=-----END CERTIFICATE--------Server certificatesubject=/CN=*.execute-api.us-east-2.amazonaws.comissuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon---If you see more than one certificate in the output, then look for the last certificate displayed at the end of the output. The last certificate is the root CA in the certificate authority chain.3.    Create a certificate file (example: certificate.crt), and the copy the contents of the last certificate to this file. Then, run the following command:openssl x509 -in certificate.crt -textThe output looks similar to the following:[root@ip-172-31-1-202 ~]# openssl x509 -in certificate.crt -textCertificate:    Data:        Version: 3 (0x2)        Serial Number:            a7:0e:4a:4c:34:82:b7:7f    Signature Algorithm: sha256WithRSAEncryption        Issuer: C=US, O=Starfield Technologies, Inc., OU=Starfield Class 2 Certification Authority        Validity            Not Before: Sep  2 00:00:00 2009 GMT            Not After : Jun 28 17:39:16 2034 GMTYou can check the validity of the certificate from the values in the Not Before and Not After fields. From the output, you can see that the validity of Amazon CA is around 25 years.4.    If the output indicates that the certificate is expired, then you must renew the certificate with your OIDC provider. After you renew the certificate, run the following command using the OpenSSL command line tool to get the latest thumbprint:openssl x509 -in certificate.crt -fingerprint -nooutThe output looks similar to the following:SHA1 Fingerprint=9E:99:A4:8A:99:60:B1:49:26:BB:7F:3B:02:E2:2D:A2:B0:AB:72:80Delete the colons (:) from this string to get the final thumbprint:9E99A48A9960B14926BB7F3B02E22DA2B0AB7280-or-Run the following command to get the latest thumbprint after excluding the semicolons:$ openssl x509 -in certificate.crt -fingerprint -noout | sed s/://g5.    If the current thumbprint has expired, then use the latest thumbprint from step 4 to replace it. You can do so from the IAM console or using the AWS Command Line Interface (AWS CLI).To replace the thumbprint using the console, do the following:Open the IAM console.In the navigation pane, choose Identity providers.Choose the identity provider that you want to update.In the Thumbprints section, choose Manage.Choose Add thumbprint, and enter the new value.Choose Save changes.-or-Run a command similar to the following using the AWS CLI:aws iam update-open-id-connect-provider-thumbprint --open-id-connect-provider-arn arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/5ECB2797CB1324A37FC79E3C46851CED --thumbprint-list 9E99A48A9960B14926BB7F3B02E22DA2B0AB7280Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

Related informationObtaining the root CA thumbprint for an OpenID Connect Identity ProviderCreating OpenID Connect (OIDC) identity providersCreate an IAM OIDC provider for your cluster