Repository: DataBiosphere/toil

Issue:
fileJobStore.exportFile not working with large files
Happened twice trying to export files to s3. Both times with large files (7.5 and 12.5Gb) The small 500kb files exported fine.

This is on toil==3.5.2
```
---TOIL WORKER OUTPUT LOG---
J/6/jobqT5KFM    INFO:toil:Running Toil version 3.5.2-378fffa320ded1ed1ebade5ec7d01138699db3f6.
J/6/jobqT5KFM    WARNING:toil.resource:Can't find resource for leader path '/home/mesosbox/protect/src/protect'
J/6/jobqT5KFM    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/mesosbox/protect/src', name='protect.alignment.common', fromVirtualEnv=False)
J/6/jobqT5KFM    INFO:toil.fileStore:Starting job ('index_bamfile' J/6/jobqT5KFM) with ID (bfb5cdd356e6ac8e95eaea8e16ecc31ca1629a5d).
J/6/jobqT5KFM    WARNING:toil.resource:Can't find resource for leader path '/home/mesosbox/protect/src/protect'
J/6/jobqT5KFM    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/mesosbox/protect/src', name='protect.alignment.common', fromVirtualEnv=False)
J/6/jobqT5KFM    INFO:toil.fileStore:LOG-TO-MASTER: Running samtools-index on TCGA-CH-5766:normal_dna
J/6/jobqT5KFM    INFO:toil.fileStore:LOG-TO-MASTER: Exporting 7/X/tmpTpNJ7J.tmp to output location
J/6/jobqT5KFM    WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'index_bamfile' J/6/jobqT5KFM used 100.02% (12.5 GB [13439741952B] used, 12.5 GB [13436562076B] requested) at the end of its run.
J/6/jobqT5KFM    Traceback (most recent call last):
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/worker.py", line 340, in main
J/6/jobqT5KFM        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
J/6/jobqT5KFM        returnValues = self._run(jobGraph, fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
J/6/jobqT5KFM        return self.run(fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1383, in run
J/6/jobqT5KFM        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
J/6/jobqT5KFM      File "/home/mesosbox/protect/src/protect/alignment/common.py", line 55, in index_bamfile
J/6/jobqT5KFM        export_results(job, bamfile, os.path.splitext(out_bai)[0], univ_options, subfolder='alignments')
J/6/jobqT5KFM      File "/home/mesosbox/protect/src/protect/common.py", line 337, in export_results
J/6/jobqT5KFM        job.fileStore.exportFile(fsid, output_url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/fileStore.py", line 729, in exportFile
J/6/jobqT5KFM        self.jobStore.exportFile(jobStoreFileID, dstUrl)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 328, in exportFile
J/6/jobqT5KFM        return self._exportFile(otherCls, jobStoreFileID, dstUrl)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 185, in _exportFile
J/6/jobqT5KFM        super(FileJobStore, self)._exportFile(otherCls, jobStoreFileID, url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 344, in _exportFile
J/6/jobqT5KFM        otherCls._writeToUrl(readable, url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 441, in _writeToUrl
J/6/jobqT5KFM        dstKey.set_contents_from_string(readable.read())
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string
J/6/jobqT5KFM        encrypt_key=encrypt_key)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
J/6/jobqT5KFM        chunked_transfer=chunked_transfer, size=size)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
J/6/jobqT5KFM        chunked_transfer=chunked_transfer, size=size)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
J/6/jobqT5KFM        query_args=query_args
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
J/6/jobqT5KFM        retry_handler=retry_handler
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
J/6/jobqT5KFM        retry_handler=retry_handler)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
J/6/jobqT5KFM        raise ex
J/6/jobqT5KFM    error: [Errno 104] Connection reset by peer
J/6/jobqT5KFM    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-11-111
J/6/jobqT5KFM    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'index_bamfile' J/6/jobqT5KFM with ID J/6/jobqT5KFM to 0
Job 'index_bamfile' J/6/jobqT5KFM with ID J/6/jobqT5KFM is completely failed
```
fileJobStore.exportFile not working with large files
Happened twice trying to export files to s3. Both times with large files (7.5 and 12.5Gb) The small 500kb files exported fine.

This is on toil==3.5.2
```
---TOIL WORKER OUTPUT LOG---
J/6/jobqT5KFM    INFO:toil:Running Toil version 3.5.2-378fffa320ded1ed1ebade5ec7d01138699db3f6.
J/6/jobqT5KFM    WARNING:toil.resource:Can't find resource for leader path '/home/mesosbox/protect/src/protect'
J/6/jobqT5KFM    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/mesosbox/protect/src', name='protect.alignment.common', fromVirtualEnv=False)
J/6/jobqT5KFM    INFO:toil.fileStore:Starting job ('index_bamfile' J/6/jobqT5KFM) with ID (bfb5cdd356e6ac8e95eaea8e16ecc31ca1629a5d).
J/6/jobqT5KFM    WARNING:toil.resource:Can't find resource for leader path '/home/mesosbox/protect/src/protect'
J/6/jobqT5KFM    WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/mesosbox/protect/src', name='protect.alignment.common', fromVirtualEnv=False)
J/6/jobqT5KFM    INFO:toil.fileStore:LOG-TO-MASTER: Running samtools-index on TCGA-CH-5766:normal_dna
J/6/jobqT5KFM    INFO:toil.fileStore:LOG-TO-MASTER: Exporting 7/X/tmpTpNJ7J.tmp to output location
J/6/jobqT5KFM    WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'index_bamfile' J/6/jobqT5KFM used 100.02% (12.5 GB [13439741952B] used, 12.5 GB [13436562076B] requested) at the end of its run.
J/6/jobqT5KFM    Traceback (most recent call last):
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/worker.py", line 340, in main
J/6/jobqT5KFM        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
J/6/jobqT5KFM        returnValues = self._run(jobGraph, fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
J/6/jobqT5KFM        return self.run(fileStore)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1383, in run
J/6/jobqT5KFM        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
J/6/jobqT5KFM      File "/home/mesosbox/protect/src/protect/alignment/common.py", line 55, in index_bamfile
J/6/jobqT5KFM        export_results(job, bamfile, os.path.splitext(out_bai)[0], univ_options, subfolder='alignments')
J/6/jobqT5KFM      File "/home/mesosbox/protect/src/protect/common.py", line 337, in export_results
J/6/jobqT5KFM        job.fileStore.exportFile(fsid, output_url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/fileStore.py", line 729, in exportFile
J/6/jobqT5KFM        self.jobStore.exportFile(jobStoreFileID, dstUrl)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 328, in exportFile
J/6/jobqT5KFM        return self._exportFile(otherCls, jobStoreFileID, dstUrl)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 185, in _exportFile
J/6/jobqT5KFM        super(FileJobStore, self)._exportFile(otherCls, jobStoreFileID, url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 344, in _exportFile
J/6/jobqT5KFM        otherCls._writeToUrl(readable, url)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 441, in _writeToUrl
J/6/jobqT5KFM        dstKey.set_contents_from_string(readable.read())
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1426, in set_contents_from_string
J/6/jobqT5KFM        encrypt_key=encrypt_key)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
J/6/jobqT5KFM        chunked_transfer=chunked_transfer, size=size)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
J/6/jobqT5KFM        chunked_transfer=chunked_transfer, size=size)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
J/6/jobqT5KFM        query_args=query_args
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
J/6/jobqT5KFM        retry_handler=retry_handler
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
J/6/jobqT5KFM        retry_handler=retry_handler)
J/6/jobqT5KFM      File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
J/6/jobqT5KFM        raise ex
J/6/jobqT5KFM    error: [Errno 104] Connection reset by peer
J/6/jobqT5KFM    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-172-31-11-111
J/6/jobqT5KFM    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'index_bamfile' J/6/jobqT5KFM with ID J/6/jobqT5KFM to 0
Job 'index_bamfile' J/6/jobqT5KFM with ID J/6/jobqT5KFM is completely failed
```


Discussion:
I imagine that using multipart uploads would fix this issue, I will look into it.
Related to https://github.com/BD2KGenomics/toil/issues/845, looks like the solution is to do multipart uploading with retries
Clarified with @arkal, this happening with the fileJobStore exporting to S3. 
Testing with awsJobStore to S3 on  whole run now. Will keep you posted
```
WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'index_bamfile' 3db20321-e09d-4523-b7a7-e297e89e2bde used 100.04% (7.7 GB [8249569280B] used, 7.7 GB [8246023407B] requested) at the end of its run.
Traceback (most recent call last):
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/mesosbox/protect/src/protect/alignment/common.py", line 55, in index_bamfile
    export_results(job, bamfile, os.path.splitext(out_bai)[0], univ_options, subfolder='alignments')
  File "/home/mesosbox/protect/src/protect/common.py", line 337, in export_results
    job.fileStore.exportFile(fsid, output_url)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/fileStore.py", line 729, in exportFile
    self.jobStore.exportFile(jobStoreFileID, dstUrl)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 328, in exportFile
    return self._exportFile(otherCls, jobStoreFileID, dstUrl)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 423, in _exportFile
    info.copyTo(dstKey)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 1142, in copyTo
    headers=headers)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 1166, in _copyKey
    headers=headers)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 400 Bad Request
<Error><Code>InvalidRequest</Code><Message>The specified copy source is larger than the maximum allowable size for a copy source: 5368709120</Message><RequestId>A2764BA8AC8F2871</RequestId><HostId>vApI5JL3PT4uF2UTq0ihykpCjqNfEF+kv/Lif9eqkBZIpfXSegbVonEIYfo7p/ujyLx/X91kkBc=</HostId></Error>
```

using an aws job store and exporting to an s3 bucket. Are you sure it is using multi-part upload for the copy?
http://stackoverflow.com/questions/10355941/how-can-i-copy-files-bigger-than-5-gb-in-amazon-s3
The multipart logic is there and for some reason it isn't being triggered for this file. The condition for using multipart is just
```
if srcKey.size > self.outer.partSize:
    use multipart
else:
    use copy_key
```
where partSize is 5mb. Maybe key.size is giving us wrong information?
Turns out the key is being initialized with the parameter `validate=False`, which tells boto just to create the object in-memory without hitting S3 for information about the key. 
@cket i'm still seeing this error in the latest dev toil version
What version? Did any of the previous releases work or did this never fix your issue?
I imagine that using multipart uploads would fix this issue, I will look into it.
Related to https://github.com/BD2KGenomics/toil/issues/845, looks like the solution is to do multipart uploading with retries
Clarified with @arkal, this happening with the fileJobStore exporting to S3. 
Testing with awsJobStore to S3 on  whole run now. Will keep you posted
```
WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'index_bamfile' 3db20321-e09d-4523-b7a7-e297e89e2bde used 100.04% (7.7 GB [8249569280B] used, 7.7 GB [8246023407B] requested) at the end of its run.
Traceback (most recent call last):
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/worker.py", line 340, in main
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1270, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1217, in _run
    return self.run(fileStore)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/job.py", line 1383, in run
    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
  File "/home/mesosbox/protect/src/protect/alignment/common.py", line 55, in index_bamfile
    export_results(job, bamfile, os.path.splitext(out_bai)[0], univ_options, subfolder='alignments')
  File "/home/mesosbox/protect/src/protect/common.py", line 337, in export_results
    job.fileStore.exportFile(fsid, output_url)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/fileStore.py", line 729, in exportFile
    self.jobStore.exportFile(jobStoreFileID, dstUrl)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/abstractJobStore.py", line 328, in exportFile
    return self._exportFile(otherCls, jobStoreFileID, dstUrl)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 423, in _exportFile
    info.copyTo(dstKey)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 1142, in copyTo
    headers=headers)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/toil/jobStores/aws/jobStore.py", line 1166, in _copyKey
    headers=headers)
  File "/home/mesosbox/protect/venv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 888, in copy_key
    response.reason, body)
S3ResponseError: S3ResponseError: 400 Bad Request
<Error><Code>InvalidRequest</Code><Message>The specified copy source is larger than the maximum allowable size for a copy source: 5368709120</Message><RequestId>A2764BA8AC8F2871</RequestId><HostId>vApI5JL3PT4uF2UTq0ihykpCjqNfEF+kv/Lif9eqkBZIpfXSegbVonEIYfo7p/ujyLx/X91kkBc=</HostId></Error>
```

using an aws job store and exporting to an s3 bucket. Are you sure it is using multi-part upload for the copy?
http://stackoverflow.com/questions/10355941/how-can-i-copy-files-bigger-than-5-gb-in-amazon-s3
The multipart logic is there and for some reason it isn't being triggered for this file. The condition for using multipart is just
```
if srcKey.size > self.outer.partSize:
    use multipart
else:
    use copy_key
```
where partSize is 5mb. Maybe key.size is giving us wrong information?
Turns out the key is being initialized with the parameter `validate=False`, which tells boto just to create the object in-memory without hitting S3 for information about the key. 
@cket i'm still seeing this error in the latest dev toil version
What version? Did any of the previous releases work or did this never fix your issue?

Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -16,9 +16,7 @@
 
 from contextlib import contextmanager, closing
 import logging
-from multiprocessing import cpu_count
 
-import os
 import re
 import uuid
 import base64
@@ -42,7 +40,6 @@
 from boto.exception import S3CreateError
 from boto.s3.key import Key
 from boto.exception import SDBResponseError, S3ResponseError
-from concurrent.futures import ThreadPoolExecutor
 
 from toil.jobStores.abstractJobStore import (AbstractJobStore,
                                              NoSuchJobException,
@@ -57,7 +54,8 @@
                                       monkeyPatchSdbConnection,
                                       retry_s3,
                                       bucket_location_to_region,
-                                      region_to_bucket_location)
+                                      region_to_bucket_location, copyKeyMultipart,
+                                      uploadFromPath, chunkedFileUpload, fileSizeAndTime)
 from toil.jobStores.utils import WritablePipe, ReadablePipe
 from toil.jobGraph import JobGraph
 import toil.lib.encryption as encryption
@@ -65,84 +63,6 @@
 log = logging.getLogger(__name__)
 
 
-def copyKeyMultipart(srcKey, dstBucketName, dstKeyName, partSize, headers=None):
-    """
-    Copies a key from a source key to a destination key in multiple parts. Note that if the
-    destination key exists it will be overwritten implicitly, and if it does not exist a new
-    key will be created. If the destination bucket does not exist an error will be raised.
-
-    :param boto.s3.key.Key srcKey: The source key to be copied from.
-    :param str dstBucketName: The name of the destination bucket for the copy.
-    :param str dstKeyName: The name of the destination key that will be created or overwritten.
-    :param int partSize: The size of each individual part, must be >= 5 MiB but large enough to
-           not exceed 10k parts for the whole file
-    :param dict headers: Any headers that should be passed.
-
-    :rtype: boto.s3.multipart.CompletedMultiPartUpload
-    :return: An object representing the completed upload.
-    """
-
-    def copyPart(partIndex):
-        if exceptions:
-            return None
-        try:
-            for attempt in retry_s3():
-                with attempt:
-                    start = partIndex * partSize
-                    end = min(start + partSize, totalSize)
-                    part = upload.copy_part_from_key(src_bucket_name=srcKey.bucket.name,
-                                                     src_key_name=srcKey.name,
-                                                     src_version_id=srcKey.version_id,
-                                                     # S3 part numbers are 1-based
-                                                     part_num=partIndex + 1,
-                                                     # S3 range intervals are closed at the end
-                                                     start=start, end=end - 1,
-                                                     headers=headers)
-        except Exception as e:
-            if len(exceptions) < 5:
-                exceptions.append(e)
-                log.error('Failed to copy part number %d:', partIndex, exc_info=True)
-            else:
-                log.warn('Also failed to copy part number %d due to %s.', partIndex, e)
-            return None
-        else:
-            log.debug('Successfully copied part %d of %d.', partIndex, totalParts)
-            # noinspection PyUnboundLocalVariable
-            return part
-
-    totalSize = srcKey.size
-    totalParts = (totalSize + partSize - 1) / partSize
-    exceptions = []
-    # We need a location-agnostic connection to S3 so we can't use the one that we
-    # normally use for interacting with the job store bucket.
-    with closing(boto.connect_s3()) as s3:
-        for attempt in retry_s3():
-            with attempt:
-                dstBucket = s3.get_bucket(dstBucketName)
-                upload = dstBucket.initiate_multipart_upload(dstKeyName, headers=headers)
-        log.info("Initiated multipart copy from 's3://%s/%s' to 's3://%s/%s'.",
-                 srcKey.bucket.name, srcKey.name, dstBucketName, dstKeyName)
-        try:
-            # We can oversubscribe cores by at least a factor of 16 since each copy task just
-            # blocks, waiting on the server. Limit # of threads to 128, since threads aren't
-            # exactly free either. Lastly, we don't need more threads than we have parts.
-            with ThreadPoolExecutor(max_workers=min(cpu_count() * 16, totalParts, 128)) as executor:
-                parts = list(executor.map(copyPart, xrange(0, totalParts)))
-                if exceptions:
-                    raise RuntimeError('Failed to copy at least %d part(s)' % len(exceptions))
-                assert len(filter(None, parts)) == totalParts
-        except:
-            with panic(log=log):
-                upload.cancel_upload()
-        else:
-            for attempt in retry_s3():
-                with attempt:
-                    completed = upload.complete_upload()
-                    log.info("Completed copy from 's3://%s/%s' to 's3://%s/%s'.",
-                             srcKey.bucket.name, srcKey.name, dstBucketName, dstKeyName)
-                    return completed
-
-
 class AWSJobStore(AbstractJobStore):
     """
     A job store that uses Amazon's S3 for file storage and SimpleDB for storing job info and
@@ -484,7 +404,20 @@ def _readFromUrl(cls, url, writable):
     def _writeToUrl(cls, readable, url):
         dstKey = cls._getKeyForUrl(url)
         try:
-            dstKey.set_contents_from_string(readable.read())
+            canDetermineSize = True
+            try:
+                readable.seek(0, 2)  # go to the 0th byte from the end of the file, indicated by '2'
+                fileSize = readable.tell()  # tells the current position in file - in this case == size of file
+                readable.seek(0)  # go to the 0th byte from the start of the file
+            except:
+                canDetermineSize = False
+            if canDetermineSize and fileSize > (5 * 1000 * 1000):  # only use multipart when file is above 5 mb
+                log.debug("Uploading %s with size %s, will use multipart uploading", dstKey.name, fileSize)
+                chunkedFileUpload(readable=readable, bucket=dstKey.bucket, fileID=dstKey.name, file_size=fileSize)
+            else:
+                # we either don't know the size, or the size is small
+                log.debug("Can not use multipart uploading for %s, uploading whole file at once", dstKey.name)
+                dstKey.set_contents_from_string(readable.read())
         finally:
             dstKey.bucket.connection.close()
 
@@ -1042,57 +975,15 @@ def save(self):
                     raise
 
         def upload(self, localFilePath):
-            file_size, file_time = self._fileSizeAndTime(localFilePath)
+            file_size, file_time = fileSizeAndTime(localFilePath)
             if file_size <= self._maxInlinedSize():
                 with open(localFilePath) as f:
                     self.content = f.read()
             else:
                 headers = self._s3EncryptionHeaders()
-                if file_size <= self.outer.partSize:
-                    key = self.outer.filesBucket.new_key(key_name=self.fileID)
-                    key.name = self.fileID
-                    for attempt in retry_s3():
-                        with attempt:
-                            key.set_contents_from_filename(localFilePath, headers=headers)
-                    self.version = key.version_id
-                else:
-                    with open(localFilePath, 'rb') as f:
-                        for attempt in retry_s3():
-                            with attempt:
-                                upload = self.outer.filesBucket.initiate_multipart_upload(
-                                    key_name=self.fileID,
-                                    headers=headers)
-                        try:
-                            start = 0
-                            part_num = itertools.count()
-                            while start < file_size:
-                                end = min(start + self.outer.partSize, file_size)
-                                assert f.tell() == start
-                                for attempt in retry_s3():
-                                    with attempt:
-                                        upload.upload_part_from_file(fp=f,
-                                                                     part_num=next(part_num) + 1,
-                                                                     size=end - start,
-                                                                     headers=headers)
-                                start = end
-                            assert f.tell() == file_size == start
-                        except:
-                            with panic(log=log):
-                                for attempt in retry_s3():
-                                    with attempt:
-                                        upload.cancel_upload()
-                        else:
-                            for attempt in retry_s3():
-                                with attempt:
-                                    self.version = upload.complete_upload().version_id
-                for attempt in retry_s3():
-                    with attempt:
-                        key = self.outer.filesBucket.get_key(self.fileID,
-                                                             headers=headers,
-                                                             version_id=self.version)
-                assert key.size == file_size
-                # Make resonably sure that the file wasn't touched during the upload
-                assert self._fileSizeAndTime(localFilePath) == (file_size, file_time)
+                self.version = uploadFromPath(localFilePath, partSize=self.outer.partSize,
+                                              bucket=self.outer.filesBucket, fileID=self.fileID,
+                                              headers=headers)
 
         @contextmanager
         def uploadStream(self, multipart=True, allowInlining=True):
@@ -1289,10 +1180,6 @@ def _s3EncryptionHeaders(self):
             else:
                 return {}
 
-        def _fileSizeAndTime(self, localFilePath):
-            file_stat = os.stat(localFilePath)
-            file_size, file_time = file_stat.st_size, file_stat.st_mtime
-            return file_size, file_time
 
         def __repr__(self):
             r = custom_repr
diff --git a/src/toil/jobStores/aws/utils.py b/src/toil/jobStores/aws/utils.py
--- a/src/toil/jobStores/aws/utils.py
+++ b/src/toil/jobStores/aws/utils.py
@@ -15,14 +15,23 @@
 from __future__ import absolute_import
 import base64
 import bz2
+import os
 import socket
 import logging
 import types
 
 import errno
+from contextlib import closing
 from ssl import SSLError
+from multiprocessing import cpu_count
 
 # Python 3 compatibility imports
+import itertools
+
+
+import boto
+from bd2k.util.exceptions import panic
+from concurrent.futures import ThreadPoolExecutor
 from six import iteritems
 
 from bd2k.util.retry import retry
@@ -169,6 +178,155 @@ def attributesToBinary(cls, attributes):
 from boto.sdb.connection import SDBConnection
 
 
+def fileSizeAndTime(localFilePath):
+    file_stat = os.stat(localFilePath)
+    file_size, file_time = file_stat.st_size, file_stat.st_mtime
+    return file_size, file_time
+
+
+def uploadFromPath(localFilePath, partSize, bucket, fileID, headers):
+    """
+    Uploads a file to s3, using multipart uploading if applicable
+
+    :param str localFilePath: Path of the file to upload to s3
+    :param int partSize: max size of each part in the multipart upload, in bytes
+    :param boto.s3.Bucket bucket: the s3 bucket to upload to
+    :param str fileID: the name of the file to upload to
+    :param headers: http headers to use when uploading - generally used for encryption purposes
+    :return: version of the newly uploaded file
+    """
+    file_size, file_time = fileSizeAndTime(localFilePath)
+    if file_size <= partSize:
+        key = bucket.new_key(key_name=fileID)
+        key.name = fileID
+        for attempt in retry_s3():
+            with attempt:
+                key.set_contents_from_filename(localFilePath, headers=headers)
+        version = key.version_id
+    else:
+        with open(localFilePath, 'rb') as f:
+            version = chunkedFileUpload(f, bucket, fileID, file_size, headers, partSize)
+    for attempt in retry_s3():
+        with attempt:
+            key = bucket.get_key(fileID,
+                                 headers=headers,
+                                 version_id=version)
+    assert key.size == file_size
+    # Make reasonably sure that the file wasn't touched during the upload
+    assert fileSizeAndTime(localFilePath) == (file_size, file_time)
+    return version
+
+
+def chunkedFileUpload(readable, bucket, fileID, file_size, headers=None, partSize=50 << 20):
+    for attempt in retry_s3():
+        with attempt:
+            upload = bucket.initiate_multipart_upload(
+                key_name=fileID,
+                headers=headers)
+    try:
+        start = 0
+        part_num = itertools.count()
+        while start < file_size:
+            end = min(start + partSize, file_size)
+            assert readable.tell() == start
+            for attempt in retry_s3():
+                with attempt:
+                    upload.upload_part_from_file(fp=readable,
+                                                 part_num=next(part_num) + 1,
+                                                 size=end - start,
+                                                 headers=headers)
+            start = end
+        assert readable.tell() == file_size == start
+    except:
+        with panic(log=log):
+            for attempt in retry_s3():
+                with attempt:
+                    upload.cancel_upload()
+    else:
+        for attempt in retry_s3():
+            with attempt:
+                version = upload.complete_upload().version_id
+    return version
+
+
+def copyKeyMultipart(srcKey, dstBucketName, dstKeyName, partSize, headers=None):
+    """
+    Copies a key from a source key to a destination key in multiple parts. Note that if the
+    destination key exists it will be overwritten implicitly, and if it does not exist a new
+    key will be created. If the destination bucket does not exist an error will be raised.
+
+    :param boto.s3.key.Key srcKey: The source key to be copied from.
+    :param str dstBucketName: The name of the destination bucket for the copy.
+    :param str dstKeyName: The name of the destination key that will be created or overwritten.
+    :param int partSize: The size of each individual part, must be >= 5 MiB but large enough to
+           not exceed 10k parts for the whole file
+    :param dict headers: Any headers that should be passed.
+
+    :rtype: boto.s3.multipart.CompletedMultiPartUpload
+    :return: An object representing the completed upload.
+    """
+
+    def copyPart(partIndex):
+        if exceptions:
+            return None
+        try:
+            for attempt in retry_s3():
+                with attempt:
+                    start = partIndex * partSize
+                    end = min(start + partSize, totalSize)
+                    part = upload.copy_part_from_key(src_bucket_name=srcKey.bucket.name,
+                                                     src_key_name=srcKey.name,
+                                                     src_version_id=srcKey.version_id,
+                                                     # S3 part numbers are 1-based
+                                                     part_num=partIndex + 1,
+                                                     # S3 range intervals are closed at the end
+                                                     start=start, end=end - 1,
+                                                     headers=headers)
+        except Exception as e:
+            if len(exceptions) < 5:
+                exceptions.append(e)
+                log.error('Failed to copy part number %d:', partIndex, exc_info=True)
+            else:
+                log.warn('Also failed to copy part number %d due to %s.', partIndex, e)
+            return None
+        else:
+            log.debug('Successfully copied part %d of %d.', partIndex, totalParts)
+            # noinspection PyUnboundLocalVariable
+            return part
+
+    totalSize = srcKey.size
+    totalParts = (totalSize + partSize - 1) / partSize
+    exceptions = []
+    # We need a location-agnostic connection to S3 so we can't use the one that we
+    # normally use for interacting with the job store bucket.
+    with closing(boto.connect_s3()) as s3:
+        for attempt in retry_s3():
+            with attempt:
+                dstBucket = s3.get_bucket(dstBucketName)
+                upload = dstBucket.initiate_multipart_upload(dstKeyName, headers=headers)
+        log.info("Initiated multipart copy from 's3://%s/%s' to 's3://%s/%s'.",
+                 srcKey.bucket.name, srcKey.name, dstBucketName, dstKeyName)
+        try:
+            # We can oversubscribe cores by at least a factor of 16 since each copy task just
+            # blocks, waiting on the server. Limit # of threads to 128, since threads aren't
+            # exactly free either. Lastly, we don't need more threads than we have parts.
+            with ThreadPoolExecutor(max_workers=min(cpu_count() * 16, totalParts, 128)) as executor:
+                parts = list(executor.map(copyPart, xrange(0, totalParts)))
+                if exceptions:
+                    raise RuntimeError('Failed to copy at least %d part(s)' % len(exceptions))
+                assert len(filter(None, parts)) == totalParts
+        except:
+            with panic(log=log):
+                upload.cancel_upload()
+        else:
+            for attempt in retry_s3():
+                with attempt:
+                    completed = upload.complete_upload()
+                    log.info("Completed copy from 's3://%s/%s' to 's3://%s/%s'.",
+                             srcKey.bucket.name, srcKey.name, dstBucketName, dstKeyName)
+                    return completed
+
+
 def _put_attributes_using_post(self, domain_or_name, item_name, attributes,
                                replace=True, expected_value=None):
     """
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -22,6 +22,8 @@
 import sys
 
 # Python 3 compatibility imports
+from _ssl import SSLError
+
 from six import iteritems
 from six.moves import xrange
 
@@ -146,8 +148,13 @@ def descriptionMatches(ami):
         if coreOSAMI is not None:
             return coreOSAMI
         # that ownerID corresponds to coreOS
-        coreOSAMI = [ami for ami in ctx.ec2.get_all_images(owners=['679593333241']) if
-                     descriptionMatches(ami)]
+
+        for attempt in retry(predicate= lambda e : isinstance(e, SSLError)):
+            # SSLError is thrown when get_all_images times out
+            with attempt:
+                amis = ctx.ec2.get_all_images(owners=['679593333241'])
+
+        coreOSAMI = [ami for ami in amis if descriptionMatches(ami)]
         logger.debug('Found the following matching AMIs: %s', coreOSAMI)
         assert len(coreOSAMI) == 1
         return coreOSAMI.pop().id


Test patch: