aws/aws-sdk-java — pull request #265 by nickpaulryan (2014-08-14 11:36:10+00:00)
Title: Fix to allow retries of uploadPartsInSeries when source is an input stream

We were having problems using the TransferManager to do uploads from a source input stream.

Occasionally the S3 server would not respond and the entire transfer would be aborted. This seemed to occur very often one day when running from EC2 and not at all when running from a local machine. The next days it happened occasionally locally and occasionally from EC2.

So we got the following:

Caused by: com.amazonaws.AmazonClientException: Encountered an exception and couldn't reset the stream to retry
at com.amazonaws.http.AmazonHttpClient.resetRequestAfterError(AmazonHttpClient.java:612) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:529) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:257) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3636) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:2759) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadPartsInSeries(UploadCallable.java:252) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInParts(UploadCallable.java:186) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:118) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.upload(UploadMonitor.java:176) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:134) [aws-java-sdk-1.8.4.jar:]
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:50) [aws-java-sdk-1.8.4.jar:]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_45]
... 3 more
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95) [httpclient-4.2.jar:4.2]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62) [httpclient-4.2.jar:4.2]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) [httpcore-4.2.jar:4.2]
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) [httpcore-4.2.jar:4.2]
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) [httpclient-4.2.jar:4.2]
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) [httpclient-4.2.jar:4.2]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) [httpcore-4.2.jar:4.2]
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66) [aws-java-sdk-1.8.4.jar:]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) [httpcore-4.2.jar:4.2]
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713) [httpclient-4.2.jar:4.2]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518) [httpclient-4.2.jar:4.2]
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) [httpclient-4.2.jar:4.2]
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) [httpclient-4.2.jar:4.2]
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:447) [aws-java-sdk-1.8.4.jar:]
... 14 more

This was because the retry logic in AmazonHttpClient.executeHelper(...) was calling reset() on the input stream as part of resetRequestAfterError() - here this didn't work as prior to the initial request attempt AmazonHttpClient.executeHelper() had called content.mark(-1).

This isn't really necessary but did require one other change which was when the reset() call before a retry did now succeed we need to reset the MD5 digest. We need to reset this for the MD5DigestCalculatingInputStream even if mark() had not been called on this stream previously.

Now when we simulate an exception in the call to httpClient.execute() we get a nice retry on the part instead of having to fail the entire TransferManager.upload() request and retrying the entire upload again.