Repository: DataBiosphere/toil

Issue:
AWS S3 HeadObject 404
At the end of a workflow run on an AWS cluster, when hundreds of files were being exported, the job failed when a 404 was received for a `head_object` request. The job was restarted in the hopes that this was a freak occurrence. It began exporting all the files again, from the beginning, and failed again with the same 404, but for a different file.

I was able to work around this problem by using `wait_until_exists` on the S3 ObjectSummary, before making the `head_object` request to retrieve version information. Stack trace follows, and a PR too, shortly.

```
ip-10-25-20-134.ec2.internal 2019-09-07 16:01:49,981 MainThread DEBUG botocore.hooks: Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f8b6c825510>>
Traceback (most recent call last):
  File "/usr/local/bin/toil-cwl-runner", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 1255, in main
    destBucket=options.destBucket)
  File "/usr/local/lib/python2.7/dist-packages/toil/cwl/cwltoil.py", line 424, in toilStageFiles
    file_store.exportFile(p.resolved[7:], destUrl)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 973, in exportFile
    self._jobStore.exportFile(jobStoreFileID, dstUrl)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/abstractJobStore.py", line 353, in exportFile
    self._exportFile(otherCls, jobStoreFileID, dstUrl)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 423, in _exportFile
    info.copyTo(dstKey)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 1143, in copyTo
    copySourceSseKey=self._getSSEKey())
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/utils.py", line 306, in copyKeyMultipart
    **destEncryptionArgs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
Traceback (most recent call last):
  File "./run-rosmap.py", line 109, in <module>
    "main-paired-get-index.cwl", "main.json"
  File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-430)
┆Issue Number: TOIL-430



Discussion:

Resolving patch:
diff --git a/src/toil/jobStores/aws/utils.py b/src/toil/jobStores/aws/utils.py
--- a/src/toil/jobStores/aws/utils.py
+++ b/src/toil/jobStores/aws/utils.py
@@ -297,6 +297,10 @@ def copyKeyMultipart(srcBucketName, srcKeyName, srcKeyVersion, dstBucketName, ds
 
     dstObject.copy(copySource, ExtraArgs=copyEncryptionArgs)
 
+    # Wait until the object exists before calling head_object
+    object_summary = s3.ObjectSummary(dstObject.bucket_name, dstObject.key)
+    object_summary.wait_until_exists(**destEncryptionArgs)
+
     # Unfortunately, boto3's managed copy doesn't return the version
     # that it actually copied to. So we have to check immediately
     # after, leaving open the possibility that it may have been


Test patch: