Bug: HEAD OBJECT returns only 400 BAD REQUEST when token is expired
Status: Resolved | Priority: Major | Resolution: Information Provided
Created: 20/Jul/22 17:42
Affects Version/s: 3.3.3

I tried reproducing this today by changing this test [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java#L116] . Getting a session token for 15 mins and trying every one mins and finally it fails after 15 mins. 
 
Looks like the AWS SDK is not having the Expired Token error message as I could see the same in access logs but I see BadRequest on the SDK logs. 
 
*S3A Connector logs with SDK debug enabled.*

2022-07-13 15:44:15,318 [JUnit-testSTS] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(184)) - Using credentials from TemporaryAWSCredentialsProvider
2022-07-13 15:44:15,319 [JUnit-testSTS] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1285)) - Sending Request: HEAD [https://mthakur-us-west-1.s3.us-west-1.amazonaws.com|https://mthakur-us-west-1.s3.us-west-1.amazonaws.com/] /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e Headers: (amz-sdk-invocation-id: 41e6e504-1c2b-2701-09bb-ae692dff2515, Content-Type: application/octet-stream, Referer: [https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318], User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10 vendor/Oracle_Corporation cfg/retry-mode/legacy, )
2022-07-13 15:44:15,623 [JUnit-testSTS] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1846)) - \{*}Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID{*}: WMGQ0KC4MHEMZTQC; S3 Extended Request ID: IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=; Proxy: null), S3 Extended Request ID: IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
2022-07-13 15:44:15,624 [JUnit-testSTS] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:close(3814)) - Filesystem s3a://mthakur-us-west-1 is closed


*AWS access logs*

183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8 mthakur-us-west-1 [13/Jul/2022:20:44:15 +0000] 67.79.115.98 - WMGQ0KC4MHEMZTQC REST.HEAD.OBJECT test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e "HEAD /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e HTTP/1.1" *400 ExpiredToken* 556 - 5 - "[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-00000098/?op=op_create&p1=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e&pr=mthakur&ps=46c6d232-80aa-4405-9e39-5df880932fdc&id=ca2778f8-085e-4d1f-aef3-73794869f275-00000098&t0=11&fs=ca2778f8-085e-4d1f-aef3-73794869f275&t1=11&ts=1657745055318]" "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7 Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10 vendor/Oracle_Corporation cfg/retry-mode/legacy" - IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q= SigV4 ECDHE-RSA-AES128-SHA AuthHeader [mthakur-us-west-1.s3.us-west-1.amazonaws.com|http://mthakur-us-west-1.s3.us-west-1.amazonaws.com/] TLSv1.2 -
 
I tested by running repeatedly ITestCustomSigner in S3A, and also just ListObjectsV2 on loop… I did just notice your test is failing with HEAD, and *I can reproduce* by running this after credential expiry.

 

aws s3api head-object --bucket djonesoa-us-west-2 --region us-west-2 --key test-object –debug

 
To summarise:
 * If I run ListObjectsV2, I get “400 ExpiredToken”{+}{+}{+}{+}
 * If I run HeadObject, I get “400 Bad Request”{+}{+}{+}{+}
 * If I run GetObject, I get “400 ExpiredToken”