Repository: DataBiosphere/toil

Issue:
fileJobStore.exportFile not working with large files
Happened twice trying to export files to s3. Both times with large files (7.5 and 12.5Gb) The small 500kb files exported fine.

This is on toil==3.5.2
```
---TOIL WORKER OUTPUT LOG---
J/6/jobqT5KFM    INFO:toil:Running Toil version 3.5.2-378fffa320ded1ed1ebade5ec7d01138699db3f6.
J/6/jobqT5KFM    WARNING:toil.resource:Can't find resource for leader path '/home/mesosbox/protect/src/protect'
J/6/jobqT5KFM    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/mesosbox/protect/src', name='protect.alignment.common', fromVirtualEnv=False)
J/6/jobqT5KFM    INFO:toil.fileStore:Starting job ('index_bamfile' J/6/jobqT5KFM) with ID (bfb5cdd356e6ac8e95eaea8e16ecc31ca1629a5d).
J/6/jobqT5KFM    WARNING:toil.resource:Can't find resource for leader path '/home/mesosbox/protect/src/protect'
J/6/jobqT5KFM    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/mesosbox/protect/src', name='protect.alignment.common', fromVirtualEnv=False)
J/6/jobqT5KFM    INFO:toil.fileStore:LOG-TO-MASTER: Running samtools-index on TCGA-CH-5766:normal_dna
J/6/jobqT5KFM    INFO:toil.fileStore:LOG-TO-MASTER: Exporting 7/X/tmpTpNJ7J.tmp to output location
J/6/jobqT5KFM    WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'index_bamfile' J/6/jobqT5KFM used 100.02% (12.5 GB [13439741952B] used, 12.5 GB [13436562076B] requested) at the end of its run.
J/6/jobqT5KFM    Traceback (most recent call last):
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/worker.py", line 340, in main
J/6/jobqT5KFM        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
J/6/jobqT5KFM        returnValues = self._run(jobGraph, fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
J/6/jobqT5KFM        return self.run(fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1383, in run
J/6/jobqT5KFM        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
J/6/jobqT5KFM      File "/home/mesosbox/protect/src/protect/alignment/common.py", line 55, in index_bamfile
J/6/jobqT5KFM        export_results(job, bamfile, os.path.splitext(out_bai)[0], univ_options, subfolder='alignments')
J/6/jobqT5KFM      File "/home/mesosbox/protect/src/protect/common.py", line 337, in export_results
J/6/jobqT5KFM        job.fileStore.exportFile(fsid, output_url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/fileStore.py", line 729, in exportFile
J/6/jobqT5KFM        self.jobStore.exportFile(jobStoreFileID, dstUrl)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 328, in exportFile
J/6/jobqT5KFM        return self._exportFile(otherCls, jobStoreFileID, dstUrl)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 185, in _exportFile
J/6/jobqT5KFM        super(FileJobStore, self)._exportFile(otherCls, jobStoreFileID, url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 344, in _exportFile
J/6/jobqT5KFM        otherCls._writeToUrl(readable, url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 441, in _writeToUrl
J/6/jobqT5KFM        dstKey.set_contents_from_string(readable.read())
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string
J/6/jobqT5KFM        encrypt_key=encrypt_key)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
J/6/jobqT5KFM        chunked_transfer=chunked_transfer, size=size)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
J/6/jobqT5KFM        chunked_transfer=chunked_transfer, size=size)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
J/6/jobqT5KFM        query_args=query_args
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
J/6/jobqT5KFM        retry_handler=retry_handler
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
J/6/jobqT5KFM        retry_handler=retry_handler)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
J/6/jobqT5KFM        raise ex
J/6/jobqT5KFM    error: [Errno 104] Connection reset by peer
J/6/jobqT5KFM    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-11-111
J/6/jobqT5KFM    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'index_bamfile' J/6/jobqT5KFM with ID J/6/jobqT5KFM to 0
Job 'index_bamfile' J/6/jobqT5KFM with ID J/6/jobqT5KFM is completely failed
```
Retry azure import/export on timeout



Discussion:
I imagine that using multipart uploads would fix this issue, I will look into it.
Related to https://github.com/BD2KGenomics/toil/issues/845, looks like the solution is to do multipart uploading with retries
Clarified with @arkal, this happening with the fileJobStore exporting to S3. 
Testing with awsJobStore to S3 on  whole run now. Will keep you posted
```
WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'index_bamfile' 3db20321-e09d-4523-b7a7-e297e89e2bde used 100.04% (7.7 GB [8249569280B] used, 7.7 GB [8246023407B] requested) at the end of its run.
Traceback (most recent call last):
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/mesosbox/protect/src/protect/alignment/common.py", line 55, in index_bamfile
    export_results(job, bamfile, os.path.splitext(out_bai)[0], univ_options, subfolder='alignments')
  File "/home/mesosbox/protect/src/protect/common.py", line 337, in export_results
    job.fileStore.exportFile(fsid, output_url)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/fileStore.py", line 729, in exportFile
    self.jobStore.exportFile(jobStoreFileID, dstUrl)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 328, in exportFile
    return self._exportFile(otherCls, jobStoreFileID, dstUrl)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 423, in _exportFile
    info.copyTo(dstKey)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 1142, in copyTo
    headers=headers)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 1166, in _copyKey
    headers=headers)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 400 Bad Request
<Error><Code>InvalidRequest</Code><Message>The specified copy source is larger than the maximum allowable size for a copy source: 5368709120</Message><RequestId>A2764BA8AC8F2871</RequestId><HostId>vApI5JL3PT4uF2UTq0ihykpCjqNfEF+kv/Lif9eqkBZIpfXSegbVonEIYfo7p/ujyLx/X91kkBc=</HostId></Error>
```

using an aws job store and exporting to an s3 bucket. Are you sure it is using multi-part upload for the copy?
http://stackoverflow.com/questions/10355941/how-can-i-copy-files-bigger-than-5-gb-in-amazon-s3
The multipart logic is there and for some reason it isn't being triggered for this file. The condition for using multipart is just
```
if srcKey.size > self.outer.partSize:
    use multipart
else:
    use copy_key
```
where partSize is 5mb. Maybe key.size is giving us wrong information?
Turns out the key is being initialized with the parameter `validate=False`, which tells boto just to create the object in-memory without hitting S3 for information about the key. 


Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -1130,8 +1130,11 @@ def copyTo(self, dstKey):
                         dstKey.set_contents_from_string(self.content)
             elif self.version:
                 for attempt in retry_s3():
-                    srcKey = self.outer.filesBucket.get_key(self.fileID,
-                                                            validate=False)
+                    encrypted = True if self.outer.sseKeyPath else False
+                    if encrypted:
+                        srcKey = self.outer.filesBucket.get_key(self.fileID, headers=self._s3EncryptionHeaders())
+                    else:
+                        srcKey = self.outer.filesBucket.get_key(self.fileID)
                     srcKey.version_id = self.version
                     with attempt:
                         headers = {k.replace('amz-', 'amz-copy-source-', 1): v
@@ -1145,6 +1148,7 @@ def copyTo(self, dstKey):
 
         def _copyKey(self, srcKey, dstBucketName, dstKeyName, headers=None):
             headers = headers or {}
+            assert srcKey.size is not None 
             if srcKey.size > self.outer.partSize:
                 return copyKeyMultipart(srcKey=srcKey,
                                         dstBucketName=dstBucketName,


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -1091,11 +1091,13 @@ def _prepareTestFile(self, containerName, size=None):
         return url, hashlib.md5(content).hexdigest()
 
     def _hashTestFile(self, url):
-        from toil.jobStores.azureJobStore import AzureJobStore
+        from toil.jobStores.azureJobStore import AzureJobStore, retry_azure
         url = urlparse.urlparse(url)
         blob = AzureJobStore._parseWasbUrl(url)
-        content = blob.service.get_blob_to_bytes(blob.container, blob.name)
-        return hashlib.md5(content).hexdigest()
+        for attempt in retry_azure():
+            with attempt:
+                content = blob.service.get_blob_to_bytes(blob.container, blob.name)
+                return hashlib.md5(content).hexdigest()
 
     def _createExternalStore(self):
         from toil.jobStores.azureJobStore import _fetchAzureAccountKey