Repository: aws/aws-cli

Issue:
S3 sync bug in setting modified time on multi-part downloads
Hi there,

I don't have a reproducible test case, but multi-part downloads incorrectly set the mtime of the file _before_ closing it.  This can cause the time to be reset to now when the file is closed.

I have a very simple patch to fix this issue as a pull request and have been running this patched version on our production servers since Nov 10.

Please do take a look so I don't have to keep my patch around.

https://github.com/aws/aws-cli/pull/994

best,
-Kirat

fix for aws s3 sync intermittently setting mtime on a large multipart download to now
Hi folks,

I've noticed a lot of unnecessary downloads with aws s3 sync but only for large multi-part files.

Looking over the code I see that CompleteDownloadTask in awscli/customizations/s3/tasks.py sets the file mtime before it is closed.

I can't reliably reproduce it, but off-hand this seems wrong.  I suspect the file has unflushed data, and closing it updates the mtime to now.

So I've made a pretty straightforward fix where IOCloseRequest takes an optional last_updated.  If this is not None, then after the file is closed the IOWriterThread sets the mtime.  And I removed the os.utime call in CompleteDownloadTask since it is now redundant.

In my tests I could not reproduce the problem after the fix was applied.

Please let me know if this is a reasonable change and apologies in advance if I haven't followed protocol, this is the first time I've submitted a pull request here.

best,
-Kirat



Discussion:
Thanks, I'll take a look.  One of the things we typically do before merging bug fix PRs is to try to get a test case written, both to ensure that a) the bug is confirmed fixed and b) we can prevent this bug from surfacing again.  I'll try to see if I can get a test case written for this.

I see what's happening here.  This is definitely a race condition which is why this doesn't always happen.  The mtime is set immediately before enqueueing the IOCloseRequest.  However if the IO thread still has writes queued to that file, then it will continue writing to the file (which updates the mtime) after the original `os.utime` call. This works most of the time because the IO thread writes out IO tasks quickly.  However if you're writing out to a slow disk then this problem becomes much more likely.

The proposed fix here is definitely a better way.  The mtime of the file should be set in the IO thread at file close time rather than at enqueueing time.  In order to merge this though I'll need to add some tests to ensure this issue is fixed.


Resolving patch:
diff --git a/awscli/customizations/s3/executor.py b/awscli/customizations/s3/executor.py
--- a/awscli/customizations/s3/executor.py
+++ b/awscli/customizations/s3/executor.py
@@ -10,6 +10,7 @@
 # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
+import os
 import logging
 import sys
 import threading
@@ -175,6 +176,9 @@ def run(self):
                 if fileobj is not None:
                     fileobj.close()
                     del self.fd_descriptor_cache[task.filename]
+                if task.desired_mtime is not None:
+                    os.utime(task.filename, (task.desired_mtime,
+                                             task.desired_mtime))
 
     def _cleanup(self):
         for fileobj in self.fd_descriptor_cache.values():
diff --git a/awscli/customizations/s3/tasks.py b/awscli/customizations/s3/tasks.py
--- a/awscli/customizations/s3/tasks.py
+++ b/awscli/customizations/s3/tasks.py
@@ -310,12 +310,12 @@ def __call__(self):
         self._context.wait_for_completion()
         last_update_tuple = self._filename.last_update.timetuple()
         mod_timestamp = time.mktime(last_update_tuple)
-        os.utime(self._filename.dest, (int(mod_timestamp), int(mod_timestamp)))
+        desired_mtime = int(mod_timestamp)
         message = print_operation(self._filename, False,
                                   self._parameters['dryrun'])
         print_task = {'message': message, 'error': False}
         self._result_queue.put(PrintTask(**print_task))
-        self._io_queue.put(IOCloseRequest(self._filename.dest))
+        self._io_queue.put(IOCloseRequest(self._filename.dest, desired_mtime))
 
 
 class DownloadPartTask(OrderableTask):
diff --git a/awscli/customizations/s3/utils.py b/awscli/customizations/s3/utils.py
--- a/awscli/customizations/s3/utils.py
+++ b/awscli/customizations/s3/utils.py
@@ -452,4 +452,7 @@ def __new__(cls, message, error=False, total_parts=None, warning=None):
                        ['filename', 'offset', 'data', 'is_stream'])
 # Used to signal that IO for the filename is finished, and that
 # any associated resources may be cleaned up.
-IOCloseRequest = namedtuple('IOCloseRequest', ['filename'])
+_IOCloseRequest = namedtuple('IOCloseRequest', ['filename', 'desired_mtime'])
+class IOCloseRequest(_IOCloseRequest):
+    def __new__(cls, filename, desired_mtime=None):
+        return super(IOCloseRequest, cls).__new__(cls, filename, desired_mtime)


Test patch:
diff --git a/tests/unit/customizations/s3/test_executor.py b/tests/unit/customizations/s3/test_executor.py
--- a/tests/unit/customizations/s3/test_executor.py
+++ b/tests/unit/customizations/s3/test_executor.py
@@ -13,12 +13,13 @@
 import os
 import tempfile
 import shutil
-from awscli.compat import six
-from six.moves import queue
+import time
 import sys
 
 import mock
 
+from awscli.compat import six
+from six.moves import queue
 from awscli.testutils import unittest, temporary_file
 from awscli.customizations.s3.executor import IOWriterThread
 from awscli.customizations.s3.executor import ShutdownThreadRequest
@@ -74,6 +75,17 @@ def test_multiple_files_in_queue(self):
         with open(second_file, 'rb') as f:
             self.assertEqual(f.read(), b'otherstuff')
 
+    def test_mtime_set_at_file_close_time(self):
+        # We're picking something other than the close time so that can verify
+        # that the IOCloseRequest can specify what the mtime should be.
+        now_time = int(time.time() - 100)
+        self.queue.put(IORequest(self.filename, 0, b'foobar', False))
+        self.queue.put(IOCloseRequest(self.filename, now_time))
+        self.queue.put(ShutdownThreadRequest())
+        self.io_thread.run()
+        actual_mtime = int(os.stat(self.filename).st_mtime)
+        self.assertEqual(actual_mtime, now_time)
+
     def test_stream_requests(self):
         # Test that offset has no affect on the order in which requests
         # are written to stdout. The order of requests for a stream are