Question: How do I troubleshoot HTTP 403 Forbidden errors from an API Gateway custom domain name that requires mutual TLS?

My Amazon API Gateway custom domain name that has mutual Transport Layer Security (TLS) authentication activated returns HTTP 403 Forbidden errors. I don't know why this is happening.

Short descriptionNote: API Gateway can return 403 Forbidden errors for a variety of reasons. This article addresses 403 Forbidden errors related only to mutual TLS. For information on troubleshooting other types of 403 Forbidden errors, see How do I troubleshoot HTTP 403 Forbidden errors from API Gateway?To invoke an API Gateway API using a custom domain name that requires mutual TLS, clients must present a trusted certificate in the API request. When a client attempts to invoke the API, API Gateway looks for the client certificate's issuer in your truststore.The following conditions cause API Gateway to fail the TLS connection, and return a 403 status code:API Gateway can't find the issuer of the client certificate in your truststore.The client certificate is using an insecure signature algorithm.The client certificate is self signed.If Amazon CloudWatch logging is activated for your API, then an error message that indicates the cause of the error appears in your execution logs.Important: If the API request doesn't produce any CloudWatch Logs after logging is activated, then the 403 Forbidden error isn't related to mutual TLS.For REST APIsIf you set up Amazon CloudWatch logging for your REST API, then one of the following error messages also appears in your execution logs:Access denied. Reason: Could not find issuer for certificateAccess denied. Reason: Client cert using an insecure Signature AlgorithmAccess denied. Reason: self signed certificateFor HTTP API operationsHTTP APIs don't support execution logging. To troubleshoot 403 Forbidden errors returned by a custom domain name that requires mutual TLS and invokes an HTTP API, you must do the following:1.    Create a new API mapping for your custom domain name that invokes a REST API for testing only.Note: If you don't have a REST API available for testing, then use the example PetStore REST API. Then, deploy the example API to a new stage, and create a new API mapping that uses your custom domain name.2.    Follow the instructions in the Resolution section of this article using the new API mapping that you created to your REST API.3.    After the error is identified and resolved, reroute the API mapping for your custom domain name back to your HTTP API.

ResolutionConfirm the cause of the error1.   Turn on CloudWatch logging for your REST API. Then, configure execution and access logging.When configuring access logging for this use case, it's a best practice to use the following $context variables. These variables do two things:They tell API Gateway to generate CloudWatch Logs when your custom domain name that requires mutual TLS returns a 403 Forbidden error.They make it easier to identify the caller that tried to invoke your API operation when you review your CloudWatch Logs.Recommended $context variables for CloudWatch access logging that will allow API Gateway to generate execution and access logs{ "accountId":"$context.accountId", "apiId":"$context.apiId", "domainName":"$context.domainName", "domainPrefix":"$context.domainPrefix", "error.message":"$context.error.message", "error.responseType":"$context.error.responseType", "extendedRequestId":"$context.extendedRequestId", "httpMethod":"$context.httpMethod", "identity.sourceIp":"$context.identity.sourceIp", "identity.clientCert.clientCertPem":"$context.identity.clientCert.clientCertPem", "identity.clientCert.subjectDN":"$context.identity.clientCert.subjectDN", "identity.clientCert.issuerDN":"$context.identity.clientCert.issuerDN", "identity.clientCert.serialNumber":"$context.identity.clientCert.serialNumber", "identity.clientCert.validity.notBefore":"$context.identity.clientCert.validity.notBefore", "identity.clientCert.validity.notAfter":"$context.identity.clientCert.validity.notAfter", "identity.userAgent":"$context.identity.userAgent", "path":"$context.path", "protocol":"$context.protocol", "requestId":"$context.requestId", "requestTime":"$context.requestTime", "requestTimeEpoch":"$context.requestTimeEpoch", "resourceId":"$context.resourceId", "resourcePath":"$context.resourcePath", "stage":"$context.stage", "responseLatency":"$context.responseLatency", "responseLength":"$context.responseLength", "status":"$context.status" }2.    Identify what's causing the 403 Forbidden errors by viewing your REST API's execution logs in CloudWatch. If a 403 Forbidden error related to mutual TLS is logged, then you receive an error message similar to the following one:Example CloudWatch Logs error message for when a custom domain name that requires mutual TLS returns a 403 Forbidden errorExtended Request Id: {extendedRequestId} Access denied. Reason: {reason} ForbiddenException Forbidden: {requestId}To resolve "Access denied. Reason: Could not find issuer for certificate" errorsVerify that the issuer of the client certificate in the API request is included in the custom domain name's truststoreThe issuer of the client certificate (client.pem) in the API request must be included in your custom domain name's truststore. The issuer must also be included as part of the certificate bundle (bundle.pem) in Amazon Simple Storage Service (Amazon S3).To verify if the issuer of the client certificate is included in the required truststore, run the following OpenSSL command:$ openssl verify -CAfile bundle.pem client.pem-or-If the certificate bundle contains intermediate certificate authorities, then run the following OpenSSL command:$ openssl verify -CAfile rootCA.pem -untrusted intCA.pem client.pemIf the issuer of the client certificate in the API request is included in the required truststore, then the command returns an OK response.If the issuer of the client certificate isn't included in the required truststore, then the command returns the following error: "error X at Y depth lookup: unable to get local issuer certificate"Verify that all of the client certificates in your custom domain name's truststore are validIf one of the client certificate's in your custom domain name's truststore isn't valid, then some clients might not be able to access your API.To verify whether all of the client certificates in your truststore are valid, do the following:1.    Open the API Gateway console.2.    In the left navigation pane, choose Custom domain names. Then, choose your custom domain name that requires mutual TLS.3.    In the Details section, check for the the following error message: There is an invalid certificate in your truststore bundle.4.    If you see the error message, then you must decode the certificates in your truststore to identify which certificate produced the warning.Note: The following OpenSSL command displays the contents of a certificate, including its subject:$ openssl x509 -in certificate.crt -text -noout5.    Update or remove the certificates that produced the warning. Then, upload a new truststore to Amazon S3.For more information, see Troubleshooting certificate warnings.Note: If their certificate chain is preserved, then API Gateway accepts client certificates signed directly by the root certificate authority or any other intermediate certificate authority. To validate client certificates signed by only the last intermediate certificate authority, use a request parameter-based AWS Lambda authorizer. You can use your custom validation algorithm at the Lambda function level by accepting the client certificate as input from the API request.To resolve "Access denied. Reason: Client cert using an insecure Signature Algorithm" errorsVerify that your truststore text file uses a supported hashing algorithmAPI Gateway supports the following hashing algorithms in the truststore:SHA-256 or strongerRSA-2048 or strongerECDSA-256 or strongerTo verify if your truststore text file uses a supported hashing algorithm, run the following OpenSSL command:$ openssl x509 -in client.crt -text -noout | grep 'Signature Algorithm'The command response returns your truststore's signature algorithm.For more information, see Configuring your truststore.To resolve "Access denied. Reason: self signed certificate" errorsVerify that the self-signed client certificate in the API request isn't altered or corruptedThe following must match exactly:The modulus of the private key (private.key) that's used to sign the self-signed certificate within the truststore in S3 (bundle.crt or bundle.pem).The modulus from the client's certificate that's passed in the API request (client.crt).To compare the two moduli, run the following OpenSSL commands:$ openssl rsa -noout -modulus -in private.key$ openssl x509 -noout -modulus -in bundle.crt$ openssl x509 -noout -modulus -in client.crtNote: To produce a shorter hash value for easier comparison, you can PIPE-ing the output modulus into a cryptographic hash function. See the following openssl sha1 example:$ openssl [operation] -noout -modulus -in [data] | openssl sha1Valid command output examples2143831a73a8bb28467860df18550c696c03fbcb2143831a73a8bb28467860df18550c696c03fbcb2143831a73a8bb28467860df18550c696c03fbcbTo confirm data integrity, verify that there isn't any data modification at the content level by running the following diff command:$ diff client.crt bundle.crtFor more information, see Configuring your truststore.