Repository: DataBiosphere/toil

Issue:
Importing files to S3 JobStore not reliable
At the beginning of my workflow, I import about 100Gigs of data from S3 into my S3 jobstore using toil.importFile.  This fails regularly (I'd say easily 10% of the time).  I run with --retryCount 3 if that's relevant.  

I've complained about this type of thing before (#1469), and still maintain that boto3 provides a more simple and reliable API for doing multipart transfers in parallel than what's running now in jobstores/aws/utils.py. 

I've long since switched to bypassing Toil and writing output files with boto3, as it's so much more painful to crash at that point.  But there's no way that I see to get around going through Toil when importing, so it'd be nice to get this fixed. 

```
(...)
ip-172-31-19-120.us-west-2.compute.internal 2017-08-04 17:31:46,870 Thread-4 DEBUG toil.jobStores.aws.utils: Successfully copied part 418 of 561.
ip-172-31-19-120.us-west-2.compute.internal 2017-08-04 17:31:57,620 Thread-13 DEBUG toil.jobStores.aws.utils: Successfully copied part 405 of 561.
ip-172-31-19-120.us-west-2.compute.internal 2017-08-04 17:32:01,817 Thread-12 DEBUG toil.jobStores.aws.utils: Successfully copied part 419 of 561.
  File "/venv/bin/toil-vg", line 11, in <module>
    sys.exit(main())
  File "/venv/local/lib/python2.7/site-packages/toil_vg/vg_toil.py", line 325, in main
    call_main(context, args)
  File "/venv/local/lib/python2.7/site-packages/toil_vg/vg_call.py", line 602, in call_main
    inputXGFileID = toil.importFile(options.xg_path)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 943, in importFile
    return self._jobStore.importFile(srcUrl, sharedFileName=sharedFileName)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/abstractJobStore.py", line 282, in importFile
    return self._importFile(otherCls, srcUrl, sharedFileName=sharedFileName)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 363, in _importFile
    info.copyFrom(srcKey)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 1071, in copyFrom
    headers=self._s3EncryptionHeaders()).version_id
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 1109, in _copyKey
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/utils.py", line 320, in copyKeyMultipart
    upload.cancel_upload()
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/utils.py", line 316, in copyKeyMultipart
    raise RuntimeError('Failed to copy at least %d part(s)' % len(exceptions))
RuntimeError: Failed to copy at least 1 part(s)
```

Importing files to S3 JobStore not reliable
At the beginning of my workflow, I import about 100Gigs of data from S3 into my S3 jobstore using toil.importFile.  This fails regularly (I'd say easily 10% of the time).  I run with --retryCount 3 if that's relevant.  

I've complained about this type of thing before (#1469), and still maintain that boto3 provides a more simple and reliable API for doing multipart transfers in parallel than what's running now in jobstores/aws/utils.py. 

I've long since switched to bypassing Toil and writing output files with boto3, as it's so much more painful to crash at that point.  But there's no way that I see to get around going through Toil when importing, so it'd be nice to get this fixed. 

```
(...)
ip-172-31-19-120.us-west-2.compute.internal 2017-08-04 17:31:46,870 Thread-4 DEBUG toil.jobStores.aws.utils: Successfully copied part 418 of 561.
ip-172-31-19-120.us-west-2.compute.internal 2017-08-04 17:31:57,620 Thread-13 DEBUG toil.jobStores.aws.utils: Successfully copied part 405 of 561.
ip-172-31-19-120.us-west-2.compute.internal 2017-08-04 17:32:01,817 Thread-12 DEBUG toil.jobStores.aws.utils: Successfully copied part 419 of 561.
  File "/venv/bin/toil-vg", line 11, in <module>
    sys.exit(main())
  File "/venv/local/lib/python2.7/site-packages/toil_vg/vg_toil.py", line 325, in main
    call_main(context, args)
  File "/venv/local/lib/python2.7/site-packages/toil_vg/vg_call.py", line 602, in call_main
    inputXGFileID = toil.importFile(options.xg_path)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 943, in importFile
    return self._jobStore.importFile(srcUrl, sharedFileName=sharedFileName)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/abstractJobStore.py", line 282, in importFile
    return self._importFile(otherCls, srcUrl, sharedFileName=sharedFileName)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 363, in _importFile
    info.copyFrom(srcKey)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 1071, in copyFrom
    headers=self._s3EncryptionHeaders()).version_id
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/jobStore.py", line 1109, in _copyKey
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/utils.py", line 320, in copyKeyMultipart
    upload.cancel_upload()
  File "/usr/local/lib/python2.7/dist-packages/toil/jobStores/aws/utils.py", line 316, in copyKeyMultipart
    raise RuntimeError('Failed to copy at least %d part(s)' % len(exceptions))
RuntimeError: Failed to copy at least 1 part(s)
```



Discussion:
+1. I've run into this as well, though not nearly as often. Also see #1793 for another annoying bug with s3 imports.

We don't use boto3 currently, but I think you can even do this with boto2, http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.key.Key.copy .
+1. I've run into this as well, though not nearly as often. Also see #1793 for another annoying bug with s3 imports.

We don't use boto3 currently, but I think you can even do this with boto2, http://boto.cloudhackers.com/en/latest/ref/s3.html#boto.s3.key.Key.copy .

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -42,6 +42,7 @@ def runSetup():
                 'psutil==3.0.1'],
             'aws': [
                 botoRequirement,
+                'boto3==1.4.7',
                 'futures==3.0.5',
                 'pycrypto==2.6.1'],
             'azure': [
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -35,7 +35,7 @@
 import urllib.request, urllib.parse, urllib.error
 
 # Python 3 compatibility imports
-from six.moves import xrange, StringIO, reprlib
+from six.moves import StringIO, reprlib
 from six import iteritems
 
 from bd2k.util import strict_bool
@@ -46,6 +46,7 @@
 from boto.exception import S3CreateError
 from boto.exception import SDBResponseError, S3ResponseError
 
+from toil.fileStore import FileID
 from toil.jobStores.abstractJobStore import (AbstractJobStore,
                                              NoSuchJobException,
                                              ConcurrentFileModificationException,
@@ -382,6 +383,7 @@ def getEmptyFileStoreID(self, jobStoreID=None):
     def _importFile(self, otherCls, url, sharedFileName=None):
         if issubclass(otherCls, AWSJobStore):
             srcKey = self._getKeyForUrl(url, existing=True)
+            size = srcKey.size
             try:
                 if sharedFileName is None:
                     info = self.FileInfo.create(srcKey.name)
@@ -395,7 +397,7 @@ def _importFile(self, otherCls, url, sharedFileName=None):
                 info.save()
             finally:
                 srcKey.bucket.connection.close()
-            return info.fileID if sharedFileName is None else None
+            return FileID(info.fileID, size) if sharedFileName is None else None
         else:
             return super(AWSJobStore, self)._importFile(otherCls, url,
                                                         sharedFileName=sharedFileName)
@@ -1096,10 +1098,13 @@ def copyFrom(self, srcKey):
             if srcKey.size <= self._maxInlinedSize():
                 self.content = srcKey.get_contents_as_string()
             else:
-                self.version = self._copyKey(srcKey=srcKey,
-                                             dstBucketName=self.outer.filesBucket.name,
-                                             dstKeyName=self._fileID,
-                                             headers=self._s3EncryptionHeaders()).version_id
+                self.version = copyKeyMultipart(srcBucketName=srcKey.bucket.name,
+                                                srcKeyName=srcKey.name,
+                                                srcKeyVersion=srcKey.version_id,
+                                                dstBucketName=self.outer.filesBucket.name,
+                                                dstKeyName=self._fileID,
+                                                sseAlgorithm='AES256',
+                                                sseKey=self._getSSEKey())
 
         def copyTo(self, dstKey):
             """
@@ -1120,38 +1125,16 @@ def copyTo(self, dstKey):
                         srcKey = self.outer.filesBucket.get_key(bytes(self.fileID))
                     srcKey.version_id = self.version
                     with attempt:
-                        headers = {k.replace('amz-', 'amz-copy-source-', 1): v
-                                   for k, v in iteritems(self._s3EncryptionHeaders())}
-                        self._copyKey(srcKey=srcKey,
-                                      dstBucketName=dstKey.bucket.name,
-                                      dstKeyName=dstKey.name,
-                                      headers=headers)
+                        copyKeyMultipart(srcBucketName=srcKey.bucket.name,
+                                         srcKeyName=srcKey.name,
+                                         srcKeyVersion=srcKey.version_id,
+                                         dstBucketName=dstKey.bucket.name,
+                                         dstKeyName=dstKey.name,
+                                         copySourceSseAlgorithm='AES256',
+                                         copySourceSseKey=self._getSSEKey())
             else:
                 assert False
 
-        def _copyKey(self, srcKey, dstBucketName, dstKeyName, headers=None):
-            headers = headers or {}
-            assert srcKey.size is not None 
-            if srcKey.size > self.outer.partSize:
-                return copyKeyMultipart(srcKey=srcKey,
-                                        dstBucketName=dstBucketName,
-                                        dstKeyName=dstKeyName,
-                                        partSize=self.outer.partSize,
-                                        headers=headers)
-            else:
-                # We need a location-agnostic connection to S3 so we can't use the one that we
-                # normally use for interacting with the job store bucket.
-                with closing(boto.connect_s3()) as s3:
-                    for attempt in retry_s3():
-                        with attempt:
-                            dstBucket = s3.get_bucket(dstBucketName)
-                            return dstBucket.copy_key(new_key_name=bytes(dstKeyName),
-                                                      src_bucket_name=srcKey.bucket.name,
-                                                      src_version_id=srcKey.version_id,
-                                                      src_key_name=bytes(srcKey.name),
-                                                      metadata=srcKey.metadata,
-                                                      headers=headers)
-
         def download(self, localFilePath):
             if self.content is not None:
                 with open(localFilePath, 'w') as f:
@@ -1203,20 +1186,25 @@ def delete(self):
                             store.filesBucket.delete_key(key_name=bytes(self.fileID),
                                                          version_id=self.previousVersion)
 
-        def _s3EncryptionHeaders(self):
+        def _getSSEKey(self):
             sseKeyPath = self.outer.sseKeyPath
+            if sseKeyPath is None:
+                return None
+            else:
+                with open(sseKeyPath) as f:
+                    sseKey = f.read()
+                    return sseKey
+
+        def _s3EncryptionHeaders(self):
             if self.encrypted:
-                if sseKeyPath is None:
-                    raise AssertionError('Content is encrypted but no key was provided.')
-                else:
-                    with open(sseKeyPath) as f:
-                        sseKey = f.read()
-                    assert len(sseKey) == 32
-                    encodedSseKey = base64.b64encode(sseKey)
-                    encodedSseKeyMd5 = base64.b64encode(hashlib.md5(sseKey).digest())
-                    return {'x-amz-server-side-encryption-customer-algorithm': 'AES256',
-                            'x-amz-server-side-encryption-customer-key': encodedSseKey,
-                            'x-amz-server-side-encryption-customer-key-md5': encodedSseKeyMd5}
+                sseKey = self._getSSEKey()
+                assert sseKey is not None, 'Content is encrypted but no key was provided.'
+                assert len(sseKey) == 32
+                encodedSseKey = base64.b64encode(sseKey)
+                encodedSseKeyMd5 = base64.b64encode(hashlib.md5(sseKey).digest())
+                return {'x-amz-server-side-encryption-customer-algorithm': 'AES256',
+                        'x-amz-server-side-encryption-customer-key': encodedSseKey,
+                        'x-amz-server-side-encryption-customer-key-md5': encodedSseKeyMd5}
             else:
                 return {}
 
diff --git a/src/toil/jobStores/aws/utils.py b/src/toil/jobStores/aws/utils.py
--- a/src/toil/jobStores/aws/utils.py
+++ b/src/toil/jobStores/aws/utils.py
@@ -15,6 +15,7 @@
 from __future__ import absolute_import
 from __future__ import division
 from builtins import str
+from past.builtins import str as oldstr
 from builtins import next
 from builtins import range
 from past.utils import old_div
@@ -27,17 +28,13 @@
 import types
 
 import errno
-from contextlib import closing
 from ssl import SSLError
-from multiprocessing import cpu_count
 
 # Python 3 compatibility imports
 import itertools
 
 
-import boto
 from bd2k.util.exceptions import panic
-from concurrent.futures import ThreadPoolExecutor
 from six import iteritems
 
 from bd2k.util.retry import retry
@@ -46,6 +43,7 @@
                             S3ResponseError,
                             S3CreateError,
                             S3CopyError)
+import boto3
 
 log = logging.getLogger(__name__)
 
@@ -257,83 +255,57 @@ def chunkedFileUpload(readable, bucket, fileID, file_size, headers=None, partSiz
     return version
 
 
-def copyKeyMultipart(srcKey, dstBucketName, dstKeyName, partSize, headers=None):
+def copyKeyMultipart(srcBucketName, srcKeyName, srcKeyVersion, dstBucketName, dstKeyName, sseAlgorithm=None, sseKey=None,
+                     copySourceSseAlgorithm=None, copySourceSseKey=None):
     """
     Copies a key from a source key to a destination key in multiple parts. Note that if the
     destination key exists it will be overwritten implicitly, and if it does not exist a new
     key will be created. If the destination bucket does not exist an error will be raised.
 
-    :param boto.s3.key.Key srcKey: The source key to be copied from.
+    :param str srcBucketName: The name of the bucket to be copied from.
+    :param str srcKeyName: The name of the key to be copied from.
+    :param str srcKeyVersion: The version of the key to be copied from.
     :param str dstBucketName: The name of the destination bucket for the copy.
     :param str dstKeyName: The name of the destination key that will be created or overwritten.
-    :param int partSize: The size of each individual part, must be >= 5 MiB but large enough to
-           not exceed 10k parts for the whole file
-    :param dict headers: Any headers that should be passed.
+    :param str sseAlgorithm: Server-side encryption algorithm for the destination.
+    :param str sseKey: Server-side encryption key for the destination.
+    :param str copySourceSseAlgorithm: Server-side encryption algorithm for the source.
+    :param str copySourceSseKey: Server-side encryption key for the source.
 
-    :rtype: boto.s3.multipart.CompletedMultiPartUpload
-    :return: An object representing the completed upload.
+    :rtype: str
+    :return: The version of the copied file (or None if versioning is not enabled for dstBucket).
     """
-
-    def copyPart(partIndex):
-        if exceptions:
-            return None
-        try:
-            for attempt in retry_s3():
-                with attempt:
-                    start = partIndex * partSize
-                    end = min(start + partSize, totalSize)
-                    part = upload.copy_part_from_key(src_bucket_name=srcKey.bucket.name,
-                                                     src_key_name=bytes(srcKey.name),
-                                                     src_version_id=srcKey.version_id,
-                                                     # S3 part numbers are 1-based
-                                                     part_num=partIndex + 1,
-                                                     # S3 range intervals are closed at the end
-                                                     start=start, end=end - 1,
-                                                     headers=headers)
-        except Exception as e:
-            if len(exceptions) < 5:
-                exceptions.append(e)
-                log.error('Failed to copy part number %d:', partIndex, exc_info=True)
-            else:
-                log.warn('Also failed to copy part number %d due to %s.', partIndex, e)
-            return None
-        else:
-            log.debug('Successfully copied part %d of %d.', partIndex, totalParts)
-            # noinspection PyUnboundLocalVariable
-            return part
-
-    totalSize = srcKey.size
-    totalParts = old_div((totalSize + partSize - 1), partSize)
-    exceptions = []
-    # We need a location-agnostic connection to S3 so we can't use the one that we
-    # normally use for interacting with the job store bucket.
-    with closing(boto.connect_s3()) as s3:
-        for attempt in retry_s3():
-            with attempt:
-                dstBucket = s3.get_bucket(dstBucketName)
-                upload = dstBucket.initiate_multipart_upload(bytes(dstKeyName), headers=headers)
-        log.info("Initiated multipart copy from 's3://%s/%s' to 's3://%s/%s'.",
-                 srcKey.bucket.name, srcKey.name, dstBucketName, dstKeyName)
-        try:
-            # We can oversubscribe cores by at least a factor of 16 since each copy task just
-            # blocks, waiting on the server. Limit # of threads to 128, since threads aren't
-            # exactly free either. Lastly, we don't need more threads than we have parts.
-            with ThreadPoolExecutor(max_workers=min(cpu_count() * 16, totalParts, 128)) as executor:
-                parts = list(executor.map(copyPart, range(0, totalParts)))
-                if exceptions:
-                    raise RuntimeError('Failed to copy at least %d part(s)' % len(exceptions))
-                assert len([_f for _f in parts if _f]) == totalParts
-        except:
-            with panic(log=log):
-                upload.cancel_upload()
-        else:
-            for attempt in retry_s3():
-                with attempt:
-                    completed = upload.complete_upload()
-                    log.info("Completed copy from 's3://%s/%s' to 's3://%s/%s'.",
-                             srcKey.bucket.name, srcKey.name, dstBucketName, dstKeyName)
-                    return completed
-
+    s3 = boto3.resource('s3')
+    dstBucket = s3.Bucket(oldstr(dstBucketName))
+    dstObject = dstBucket.Object(oldstr(dstKeyName))
+    copySource = {'Bucket': oldstr(srcBucketName), 'Key': oldstr(srcKeyName)}
+    if srcKeyVersion is not None:
+        copySource['VersionId'] = oldstr(srcKeyVersion)
+
+    # The boto3 functions don't allow passing parameters as None to
+    # indicate they weren't provided. So we have to do a bit of work
+    # to ensure we only provide the parameters when they are actually
+    # required.
+    destEncryptionArgs = {}
+    if sseKey is not None:
+        destEncryptionArgs.update({'SSECustomerAlgorithm': sseAlgorithm,
+                                   'SSECustomerKey': sseKey})
+    copyEncryptionArgs = {}
+    if copySourceSseKey is not None:
+        copyEncryptionArgs.update({'CopySourceSSECustomerAlgorithm': copySourceSseAlgorithm,
+                                   'CopySourceSSECustomerKey': copySourceSseKey})
+    copyEncryptionArgs.update(destEncryptionArgs)
+
+    dstObject.copy(copySource, ExtraArgs=copyEncryptionArgs)
+
+    # Unfortunately, boto3's managed copy doesn't return the version
+    # that it actually copied to. So we have to check immediately
+    # after, leaving open the possibility that it may have been
+    # modified again in the few seconds since the copy finished. There
+    # isn't much we can do about it.
+    info = boto3.client('s3').head_object(Bucket=dstObject.bucket_name, Key=dstObject.key,
+                                          **destEncryptionArgs)
+    return info.get('VersionId', None)
 
 def _put_attributes_using_post(self, domain_or_name, item_name, attributes,
                                replace=True, expected_value=None):


Test patch:
diff --git a/src/toil/test/jobStores/jobStoreTest.py b/src/toil/test/jobStores/jobStoreTest.py
--- a/src/toil/test/jobStores/jobStoreTest.py
+++ b/src/toil/test/jobStores/jobStoreTest.py
@@ -40,7 +40,7 @@
 
 # Python 3 compatibility imports
 from six.moves.queue import Queue
-from six.moves import xrange, socketserver as SocketServer, SimpleHTTPServer
+from six.moves import SimpleHTTPServer
 from six import iteritems
 import six.moves.urllib.parse as urlparse
 from six.moves.urllib.request import urlopen, Request
@@ -52,9 +52,9 @@
 from mock import patch
 
 from toil.common import Config, Toil
+from toil.fileStore import FileID
 from toil.job import Job, JobNode
-from toil.jobStores.abstractJobStore import (AbstractJobStore,
-                                             NoSuchJobException,
+from toil.jobStores.abstractJobStore import (NoSuchJobException,
                                              NoSuchFileException)
 from toil.jobStores.fileJobStore import FileJobStore
 from toil.test import (ToilTest,
@@ -464,6 +464,7 @@ def testImportExportFile(self, otherCls, size):
                 srcUrl, srcMd5 = other._prepareTestFile(store, size)
                 # Import into job store under test
                 jobStoreFileID = self.master.importFile(srcUrl)
+                self.assertTrue(isinstance(jobStoreFileID, FileID))
                 with self.master.readFileStream(jobStoreFileID) as f:
                     fileMD5 = hashlib.md5(f.read()).hexdigest()
                 self.assertEqual(fileMD5, srcMd5)
@@ -1017,33 +1018,6 @@ def testInaccessableLocation(self):
             args, kwargs = mock_log.warn.call_args
             self.assertTrue('Could not determine location' in args[0])
 
-    @slow
-    def testMultiPartImportFailures(self):
-        # This should be less than the number of threads in the pool used by the MP copy.
-        num_parts = 10
-        i = count()
-
-        # noinspection PyUnusedLocal
-        def fail(*args, **kwargs):
-            # The sleep ensures that all tasks are scheduled in the thread pool. Without it,
-            # there is a chance that one task fails before another is scheduled, causing the
-            # latter to bail out immediatly and failing the assertion that ensure the number of
-            # failing tasks.
-            time.sleep(.25)
-            if next(i) % 2 == 0:
-                raise RuntimeError()
-
-        with patch('boto.s3.multipart.MultiPartUpload.copy_part_from_key',
-                   new_callable=lambda: fail):
-            self.master.partSize = self.mpTestPartSize
-            bucket = self._externalStore()
-            url, md5 = self._prepareTestFile(bucket, self.mpTestPartSize * num_parts)
-            try:
-                self.master.importFile(url)
-            except RuntimeError as e:
-                self.assertEquals(e.message, 'Failed to copy at least %d part(s)' % (old_div(num_parts, 2)))
-            else:
-                self.fail('Expected a RuntimeError to be raised')
     def testOverlargeJob(self):
         master = self.master
         masterRequirements = dict(memory=12, cores=34, disk=35, preemptable=True)
diff --git a/src/toil/test/src/multipartTransferTest.py b/src/toil/test/src/multipartTransferTest.py
deleted file mode 100644
--- a/src/toil/test/src/multipartTransferTest.py
+++ /dev/null
@@ -1,81 +0,0 @@
-# Copyright (C) 2015 UCSC Computational Genomics Lab
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from __future__ import absolute_import
-
-from builtins import str
-import logging
-import os
-import uuid
-from contextlib import contextmanager, closing
-
-from toil.test import ToilTest, make_tests, slow
-
-partSize = 2 ** 20 * 5
-logger = logging.getLogger(__name__)
-
-
-@contextmanager
-def openS3(keySize=None):
-    """
-    Creates an AWS bucket. If keySize is given, a key of random bytes is created and its handle
-    is yielded. If no keySize is given an empty bucket handle is yielded. The bucket and all
-    created keys are cleaned up automatically.
-
-    :param int keySize: Size of key to be created.
-    """
-    import boto.s3
-    from toil.jobStores.aws.utils import region_to_bucket_location
-    if keySize is not None and keySize < 0:
-        raise ValueError('Key size must be greater than zero')
-    with closing(boto.s3.connect_to_region(AWSMultipartCopyTest.awsRegion())) as s3:
-        bucket = s3.create_bucket('multipart-transfer-test-%s' % uuid.uuid4(),
-                                  location=region_to_bucket_location(AWSMultipartCopyTest.awsRegion()))
-        try:
-            keyName = 'test'
-            if keySize is None:
-                yield bucket
-            else:
-                key = bucket.new_key(keyName)
-                content = os.urandom(keySize)
-                key.set_contents_from_string(content)
-
-                yield bucket.get_key(keyName)
-        finally:
-            for key in bucket.list():
-                key.delete()
-            bucket.delete()
-
-
-@slow
-class AWSMultipartCopyTest(ToilTest):
-
-    @classmethod
-    def makeTests(cls):
-        def multipartCopy(self, threadPoolSize):
-            from toil.jobStores.aws.jobStore import copyKeyMultipart
-            # key size is padded to ensure some threads are reused
-            keySize = int((threadPoolSize * partSize) * 1.3)
-            with openS3(keySize) as srcKey:
-                with openS3() as dstBucket:
-                    copyKeyMultipart(srcKey, dstBucket, 'test', partSize)
-                    self.assertEqual(srcKey.get_contents_as_string(),
-                                     dstBucket.get_key('test').get_contents_as_string())
-
-        make_tests(multipartCopy,
-                   AWSMultipartCopyTest,
-                   threadPoolSize={str(x): x for x in (1, 2, 16)})
-
-
-AWSMultipartCopyTest.makeTests()