Repository: aws/aws-cli

Issue:
aws s3 mv leaks memory when uploading large file
Using aws-cli 1.1.0 I was attempting to upload a 1.7GB plain text file using `aws s3 mv localfile s3://bucket_name/remotefile`. And it used up all the memory on the server dipping into the swap memory.

```
$ aws --version
aws-cli/1.1.0 Python/2.7.3 Linux/3.2.0-40-virtual
$ pip freeze -l
awscli==1.1.0
bcdoc==0.10.0
botocore==0.17.0
colorama==0.2.5
distribute==0.6.24
docutils==0.11
jmespath==0.0.2
ply==3.4
pyasn1==0.1.7
python-dateutil==2.1
requests==1.2.3
rsa==3.1.2
six==1.4.1
```

Seemed like it was stuck in a never-ending loop. Had to kill -9 the process, Interrupting it didn't work.

Output of `vmstat 1` on Ubuntu 12.04.2 LTS (keep an eye out on the 3rd column)

```
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
1  0  32180 3509484   1696 235628    0    0     0     0  256  363  0  1 99  0
0  0  32180 3509112   1696 236064    0    0     0     0  634  661  2  4 94  0
0  0  32180 3508988   1708 236140    0    0     4    52  274  383  0  1 98  1
0  0  32180 3508740   1952 236160    0    0   244     0  359  455  1  1 93  5
0  0  32180 3507980   2076 236216    0    0   156     0  273  418  2  1 93  4
0  0  32180 3507996   2076 236276    0    0     0     0  293  396  0  0 100  0
0  0  32180 3507996   2076 236340    0    0     0     0  240  345  0  1 99  0
0  1  32180 3459896   2084 252980    0    0 16588   160  864 1205 21 16 18 45
0  1  32180 3344948   2084 310368    0    0 57344     0 1770 2566  2  9  0 89
0  1  32180 3223944   2084 370828    0    0 60416     0 1849 2769  1 11  0 88
1  1  32180 3099808   2084 432832    0    0 61952     0 1897 2661  0  9  0 89
1  1  32180 2985972   2084 489816    0    0 56576     0 2230 2778  3 11  0 86
0  1  32180 2888892   2092 538224    0    0 48384    72 1665 2309  0  7  0 93
0  1  32180 2771832   2092 596640    0    0 58368     0 1795 2565  1 11  0 88
0  1  32180 2655384   2092 654808    0    0 58112     0 1812 2621  0  9  0 91
0  1  32180 2539812   2100 712460    0    0 57600    12 1943 2628  1 10  0 89
1  0  32180 2420140   2100 772172    0    0 59648     0 1918 2672  2 10  0 87
0  1  32180 2308784   2108 827756    0    0 55552   136 1907 2556  0 12  0 88
0  1  32180 2212696   2108 875776    0    0 48000     0 1603 2069  0  5  0 95
0  1  32180 2100728   2108 931672    0    0 55808     0 1869 2558  1  6  0 92
0  1  32180 2002904   2108 980476    0    0 48768     0 1639 2184  0  9  0 90
1  1  32180 1895028   2108 1034488    0    0 53632     0 1961 2639  4  9  0 87
0  1  32180 1801048   2116 1081384    0    0 46848   740 1522 2172  1  7  0 92
0  1  32180 1690568   2116 1136612    0    0 55168     0 1810 2622  0  8  0 92
1  1  32180 1583800   2116 1189916    0    0 53248     0 1716 2461  0 10  0 90
0  1  32180 1480024   2116 1241660    0    0 51712     0 1802 2518  2  6  0 92
0  1  32180 1367180   2116 1298004    0    0 56320     0 1773 2488  1  9  0 90
0  1  32180 1267496   2124 1347748    0    0 49664   868 1707 2310  1  8  0 90
0  1  32180 1155892   2124 1403468    0    0 55680     0 1720 2419  0 10  0 90
1  1  32180 1045776   2124 1458464    0    0 54912     0 1702 2449  1  6  0 92
0  1  32180 924616   2124 1518896    0    0 60416     0 1601 2567  0  7  0 93
0  1  32180 808548   2124 1577012    0    0 57728     0 2035 2653  2 10  0 88
1  0  32180 726612   2132 1617848    0    0 40832  1688 1416 2063  1  8  0 90
0  1  32180 611908   2132 1675168    0    0 57216    12 1757 2446  0  9  0 91
0  1  32180 499560   2132 1731236    0    0 56064     0 1668 2514  1  9  0 90
1  0  32180 391684   2132 1785068    0    0 53760     0 1732 2541  1  6  0 92
2  0  32180  19024   1160 1757132    0    0 16136     0  834 1086 14 56  0 30
1  0  32180  19048   1160 1429552    0    0   136    64  519  608 21 79  0  0
1  0  32180  18340   1144 1027924    0    0     0     0  411  545 29 71  0  0
1  0  32180  20200   1128 594712    0    0     0     0  520  734 26 74  0  0
0  0  32180  19976   1120 227936    0    0   244     0  663  753 28 63  4  5
```



Discussion:
Did you see any output from the tool?  Any messages about parts being uploaded?  Or did it just hang with no output at all?

I ran the command again, this time waited for it to clean up after Interrupting it.
It took about 90 seconds for it to clean up.

And if I wouldn't have put --debug into the command I wouldn't have seen anything.
Basically I just needed to stop the process since it used up all the memory.

```
# aws --debug s3 mv localfile s3://bucket_name/remotefile
2013-09-20 16:01:50,095 - botocore.hooks - DEBUG - Event building-operation-table.s3: calling handler <function add_commands at 0x259c848>
2013-09-20 16:01:50,096 - botocore.hooks - DEBUG - Event building-parameter-table.s3.mv: calling handler <function add_cmd_params at 0x259c8c0>
2013-09-20 16:01:50,099 - botocore.service - DEBUG - Creating service object for: s3
2013-09-20 16:01:50,099 - botocore.base - DEBUG - Attempting to load: aws/s3
2013-09-20 16:01:50,139 - botocore.base - DEBUG - Found data file: /opt/aws/local/lib/python2.7/site-packages/botocore/data/aws/s3.json
2013-09-20 16:01:50,139 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x22b5410>
2013-09-20 16:01:50,140 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2013-09-20 16:01:50,140 - botocore.credentials - INFO - Found credentials in config file.
2013-09-20 16:01:50,141 - botocore.service - DEBUG - Creating service object for: s3
2013-09-20 16:01:50,142 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x22b5410>
2013-09-20 16:01:50,142 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2013-09-20 16:01:50,147 - botocore.service - DEBUG - Creating service object for: s3
2013-09-20 16:01:50,147 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x22b5410>
2013-09-20 16:01:50,147 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2013-09-20 16:01:50,148 - botocore.service - DEBUG - Creating service object for: s3
2013-09-20 16:01:50,148 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x22b5410>
2013-09-20 16:01:50,148 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
****** This is where it used pretty much all of the memory ******
2013-09-20 16:01:56,869 - botocore.service - DEBUG - Creating operation objects for: Service(s3)
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 846, in emit
    msg = self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 723, in format
    return fmt.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
MemoryError
Logged from file operation.py, line 57
2013-09-20 16:01:56,909 - botocore.operation - DEBUG - Creating parameter objects for: Operation:PutObject
2013-09-20 16:01:56,912 - botocore.endpoint - DEBUG - Making request for Operation:PutObject (verify_ssl=True) with params: {'headers': {}, 'uri_params': {u'Bucket': u'bucket_name', u'Key': u'remotefile'}, 'payload': <botocore.payload.Payload object at 0x2859050>}
2013-09-20 16:01:56,912 - botocore.endpoint - DEBUG - Building URI for rest endpoint.
2013-09-20 16:01:56,912 - botocore.endpoint - DEBUG - Templated URI path: /{Bucket}/{Key}
2013-09-20 16:01:56,912 - botocore.endpoint - DEBUG - Templated URI query_params:
2013-09-20 16:01:56,913 - botocore.endpoint - DEBUG - Rendered path: /bucket_name/remotefile
2013-09-20 16:01:56,913 - botocore.endpoint - DEBUG - Rendered query_params:
2013-09-20 16:01:56,913 - botocore.hooks - DEBUG - Event before-auth.s3: calling handler <function fix_s3_host at 0x22b5398>
2013-09-20 16:01:56,913 - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3-eu-west-1.amazonaws.com/bucket_name/remotefile
2013-09-20 16:01:56,914 - botocore.handlers - DEBUG - URI updated to: https://bucket_name.s3.amazonaws.com/remotefile
2013-09-20 16:01:56,914 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2013-09-20 16:01:56,914 - botocore.auth - DEBUG - HTTP request method: PUT
2013-09-20 16:01:56,915 - botocore.auth - DEBUG - StringToSign:
PUT


Fri, 20 Sep 2013 16:01:56 GMT
/bucket_name/remotefile
2013-09-20 16:01:56,991 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
****** Sent CTRL+C *******
Cleaning up. Please wait...
2013-09-20 16:03:47,976 - botocore.response - DEBUG - Response Body:

2013-09-20 16:03:47,977 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x283c090>
2013-09-20 16:03:47,977 - botocore.retryhandler - DEBUG - No retry needed.
2013-09-20 16:03:47,977 - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <awscli.errorhandler.ErrorHandler object at 0x25a7450>
2013-09-20 16:03:47,978 - awscli.errorhandler - DEBUG - HTTP Response Code: 200
```

Okay, thanks for that additional input.  We are investigating this now and will report back on this issue with more info.

Confirmed, I see what the issue is, working on a fix.

In the meantime, the workaround is to just do a cp followed by a local rm:

```
aws s3 cp localfile s3://bucket/key/
rm localfile
```


Resolving patch:
diff --git a/awscli/customizations/s3/s3handler.py b/awscli/customizations/s3/s3handler.py
--- a/awscli/customizations/s3/s3handler.py
+++ b/awscli/customizations/s3/s3handler.py
@@ -129,12 +129,9 @@ def _enqueue_tasks(self, files):
         for filename in files:
             filename.set_session(self.session, self.params['region'])
             num_uploads = 1
-            is_multipart_task = False
+            is_multipart_task = self._is_multipart_task(filename)
             too_large = False
             if hasattr(filename, 'size'):
-                is_multipart_task = (
-                    filename.size > self.multi_threshold and
-                    filename.operation == 'upload')
                 too_large = filename.size > MAX_UPLOAD_SIZE
             if too_large and filename.operation == 'upload':
                 warning = "Warning %s exceeds 5 TB and upload is " \
@@ -152,11 +149,29 @@ def _enqueue_tasks(self, files):
             total_parts += num_uploads
         return total_files, total_parts
 
+    def _is_multipart_task(self, filename):
+        # First we need to determine if it's an operation that even
+        # qualifies for multipart upload.
+        if hasattr(filename, 'size'):
+            is_multipart_operation = (
+                filename.operation == 'upload' or
+                filename.operation == 'move' and (
+                    filename.src_type == 'local' and
+                    filename.dest_type == 's3'))
+            above_multipart_threshold = filename.size > self.multi_threshold
+            return is_multipart_operation and above_multipart_threshold
+        else:
+            return False
+
     def _enqueue_multipart_tasks(self, filename):
         num_uploads = 1
         chunksize = self.chunksize
         if filename.operation == 'upload':
             num_uploads = self._enqueue_multipart_upload_tasks(filename)
+        elif filename.operation == 'move':
+            # Already checked that this is a local -> s3 move.
+            num_uploads = self._enqueue_multipart_upload_tasks(
+                filename, remove_local_file=True)
         elif filename.operation == 'download':
             num_uploads = int(filename.size / chunksize)
             filename.set_multi(executer=self.executer,
@@ -165,7 +180,7 @@ def _enqueue_multipart_tasks(self, filename):
                                 chunksize=chunksize)
         return num_uploads
 
-    def _enqueue_multipart_upload_tasks(self, filename):
+    def _enqueue_multipart_upload_tasks(self, filename, remove_local_file=False):
         # First we need to create a CreateMultipartUpload task,
         # then create UploadTask objects for each of the parts.
         # And finally enqueue a CompleteMultipartUploadTask.
@@ -192,4 +207,8 @@ def _enqueue_multipart_upload_tasks(self, filename):
             print_queue=self.print_queue, upload_context=upload_context)
         self.executer.submit(complete_multipart_upload_task)
         self._multipart_uploads.append((upload_context, filename))
+        if remove_local_file:
+            remove_task = tasks.RemoveFileTask(local_filename=filename.src,
+                                               upload_context=upload_context)
+            self.executer.submit(remove_task)
         return num_uploads
diff --git a/awscli/customizations/s3/tasks.py b/awscli/customizations/s3/tasks.py
--- a/awscli/customizations/s3/tasks.py
+++ b/awscli/customizations/s3/tasks.py
@@ -298,9 +298,26 @@ def __call__(self):
             print_op = print_operation(self.filename, False,
                                        self.parameters['dryrun'])
             print_task = {'result': print_op}
+            self._upload_context.announce_completed()
         self.print_queue.put(print_task)
 
 
+class RemoveFileTask(BasicTask):
+    def __init__(self, local_filename, upload_context):
+        self._local_filename = local_filename
+        self._upload_context = upload_context
+        # This 'filename' attr has to be here because other objects
+        # introspect tasks objects.  This should eventually be removed
+        # but it's needed for now.
+        self.filename = None
+
+    def __call__(self):
+        LOGGER.debug("Waiting for upload to complete.")
+        self._upload_context.wait_for_completion()
+        LOGGER.debug("Removing local file: %s", self._local_filename)
+        os.remove(self._local_filename)
+
+
 class MultipartUploadContext(object):
     """Context object for a multipart upload.
 
@@ -333,6 +350,7 @@ def __init__(self, expected_parts):
         self._lock = threading.Lock()
         self._upload_id_condition = threading.Condition(self._lock)
         self._parts_condition = threading.Condition(self._lock)
+        self._upload_complete_condition = threading.Condition(self._lock)
         self._state = self._UNSTARTED
 
     def announce_upload_id(self, upload_id):
@@ -362,6 +380,13 @@ def wait_for_upload_id(self):
                 self._upload_id_condition.wait(timeout=1)
             return self._upload_id
 
+    def wait_for_completion(self):
+        with self._upload_complete_condition:
+            while not self._state == self._COMPLETED:
+                if self._state == self._CANCELLED:
+                    raise UploadCancelledError("Upload has been cancelled.")
+                self._upload_complete_condition.wait(timeout=1)
+
     def cancel_upload(self, canceller=None, args=None, kwargs=None):
         """Cancel the upload.
 
@@ -420,5 +445,6 @@ def announce_completed(self):
         This should be called after a CompleteMultipartUpload operation.
 
         """
-        with self._lock:
+        with self._upload_complete_condition:
             self._state = self._COMPLETED
+            self._upload_complete_condition.notifyAll()


Test patch:
diff --git a/tests/unit/customizations/s3/test_tasks.py b/tests/unit/customizations/s3/test_tasks.py
--- a/tests/unit/customizations/s3/test_tasks.py
+++ b/tests/unit/customizations/s3/test_tasks.py
@@ -34,25 +34,6 @@ def join_threads(self):
         for thread in self.threads:
             thread.join()
 
-    def test_normal_non_threaded(self):
-        # The context object is pretty straightforward.
-        # This shows the non threaded usage of this object.
-        context = MultipartUploadContext(expected_parts=3)
-        # First you can announce an upload id.
-        context.announce_upload_id('my_upload_id')
-        # Then a thread that was waiting on the id would be notified.
-        self.assertEqual(context.wait_for_upload_id(), 'my_upload_id')
-        # Then thread would chug away at the parts.
-        context.announce_finished_part(etag='etag1', part_number=1)
-        context.announce_finished_part(etag='etag2', part_number=2)
-        context.announce_finished_part(etag='etag3', part_number=3)
-        # Then a thread that was waiting for all the parts to finish
-        # would be notified.
-        self.assertEqual(context.wait_for_parts_to_finish(), [
-            {'ETag': 'etag1', 'PartNumber': 1},
-            {'ETag': 'etag2', 'PartNumber': 2},
-            {'ETag': 'etag3', 'PartNumber': 3}])
-
     def upload_part(self, part_number):
         # This simulates what a thread would do if it wanted to upload
         # a part.  First it would wait for the upload id.
@@ -77,6 +58,16 @@ def complete_upload(self):
             return
         with self.call_lock:
             self.calls.append(('complete_upload', upload_id, parts))
+            self.context.announce_completed()
+
+    def wait_for_upload_complete(self):
+        try:
+            self.context.wait_for_completion()
+        except Exception as e:
+            self.caught_exception = e
+            return
+        with self.call_lock:
+            self.calls.append(('arbitrary_post_complete_operation',))
 
     def create_upload(self, upload_id):
         with self.call_lock:
@@ -87,6 +78,28 @@ def start_thread(self, thread):
         thread.start()
         self.threads.append(thread)
 
+    def test_normal_non_threaded(self):
+        # The context object is pretty straightforward.
+        # This shows the non threaded usage of this object.
+        context = MultipartUploadContext(expected_parts=3)
+        # First you can announce an upload id.
+        context.announce_upload_id('my_upload_id')
+        # Then a thread that was waiting on the id would be notified.
+        self.assertEqual(context.wait_for_upload_id(), 'my_upload_id')
+        # Then thread would chug away at the parts.
+        context.announce_finished_part(etag='etag1', part_number=1)
+        context.announce_finished_part(etag='etag2', part_number=2)
+        context.announce_finished_part(etag='etag3', part_number=3)
+        # Then a thread that was waiting for all the parts to finish
+        # would be notified.
+        self.assertEqual(context.wait_for_parts_to_finish(), [
+            {'ETag': 'etag1', 'PartNumber': 1},
+            {'ETag': 'etag2', 'PartNumber': 2},
+            {'ETag': 'etag3', 'PartNumber': 3}])
+        context.announce_completed()
+        # This will return right away since we've already announced completion.
+        self.assertIsNone(context.wait_for_completion())
+
     def test_basic_threaded_parts(self):
         # Now while test_normal_non_threaded showed the conceptual idea,
         # the real strength of MultipartUploadContext is that it works
@@ -107,6 +120,13 @@ def test_basic_threaded_parts(self):
         complete_upload_thread = threading.Thread(target=self.complete_upload)
         self.start_thread(complete_upload_thread)
 
+        # We'll also have some other arbitrary thread that's just waiting for
+        # the whole upload to be complete.  This is not the same as
+        # complete_upload_thread, as that thread is used to complete the
+        # upload.  This thread wants to know when *that* process is all done.
+        arbitrary_waiting_thread = threading.Thread(target=self.wait_for_upload_complete)
+        self.start_thread(arbitrary_waiting_thread)
+
         # Then finally the CreateMultipartUpload completes and we
         # announce the upload id.
         self.create_upload('my_upload_id')
@@ -114,12 +134,16 @@ def test_basic_threaded_parts(self):
         # multipart upload thread.
         self.join_threads()
 
+        self.assertIsNone(self.caught_exception)
         # We can verify that the invariants still hold.
+        self.assertEqual(len(self.calls), 4)
         # First there should be three calls, create, upload, complete.
-        self.assertEqual(len(self.calls), 3)
         self.assertEqual(self.calls[0][0], 'create_multipart_upload')
         self.assertEqual(self.calls[1][0], 'upload_part')
         self.assertEqual(self.calls[2][0], 'complete_upload')
+        # Then anything that was waiting for the operation to complete should
+        # be called afterwards.
+        self.assertEqual(self.calls[3][0], 'arbitrary_post_complete_operation')
 
         # Verify the correct args were used.
         self.assertEqual(self.calls[0][1], 'my_upload_id')
@@ -151,6 +175,7 @@ def test_randomized_stress_test(self):
                 threading.Thread(target=self.complete_upload),
                 threading.Thread(target=self.create_upload,
                                 args=('my_upload_id',)),
+                threading.Thread(target=self.wait_for_upload_complete),
             ]
             for i in range(1, expected_parts + 1):
                 all_threads.append(
@@ -161,9 +186,11 @@ def test_randomized_stress_test(self):
                 self.start_thread(thread)
             self.join_threads()
             self.assertEqual(self.calls[0][0], 'create_multipart_upload')
-            self.assertEqual(self.calls[-1][0], 'complete_upload')
+            self.assertEqual(self.calls[-1][0],
+                             'arbitrary_post_complete_operation')
+            self.assertEqual(self.calls[-2][0], 'complete_upload')
             parts = set()
-            for call in self.calls[1:-1]:
+            for call in self.calls[1:-2]:
                 self.assertEqual(call[0], 'upload_part')
                 self.assertEqual(call[2], 'my_upload_id')
                 parts.add(call[1])
@@ -186,3 +213,17 @@ def test_can_cancel_tasks(self):
             self.context.wait_for_upload_id()
         with self.assertRaises(UploadCancelledError):
             self.context.wait_for_parts_to_finish()
+
+    def test_cancel_threads_waiting_for_completion(self):
+        # So we have a thread waiting for the entire upload to complete.
+        arbitrary_waiting_thread = threading.Thread(target=self.wait_for_upload_complete)
+        self.start_thread(arbitrary_waiting_thread)
+
+        # And as it's waiting, something happens and we cancel the upload.
+        self.context.cancel_upload()
+
+        # The thread should exit.
+        self.join_threads()
+
+        # And we should have seen an exception being raised.
+        self.assertIsInstance(self.caught_exception, UploadCancelledError)