Repository: aws/aws-cli

Issue:
aws s3 sync racing with local files
I'm having an issue with S3 sync bombing when local files (which would be excluded anyway) are removed by another process.

I found issue #2333, and updated to the latest release, but the issue persists.  I think there is more that needs to be done to avoid these races.




Discussion:
The test `test_sync_skips_over_files_deleted_between_listing_and_transfer` is testing for `get_file_stat` raising `OSError`, but `get_file_stat` actually translates `OSError` to `ValueError`.  I started down the path of making the test patch `os.stat` instead of `get_file_stat`, but it's a bit gnarly because there are unrelated calls to calls to `os.stat` that shouldn't be interfered with and then there are 6 calls to `os.stat` on the file used in the test, which means there may be several ways for a race to manifest here.


Interesting. Do you happen to have the debug logs on you when you run into this? You can get those by appending `--debug` to the command you ran. I'm specifically interested in the Traceback that you are getting. If you are unable to get those, is there at least a description coming back from the `ValueError`? The fix may be as easy as catching the `ValueError` here as well: https://github.com/aws/aws-cli/blob/1ae25c8773eb452a6d1a61d27ce8f99c47d99356/awscli/customizations/s3/filegenerator.py#L217
I do happen to have the traceback handy still:
```
Traceback (most recent call last):
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/utils.py", line 211, in get_file_stat
    stats = os.stat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/log/sterling/production.2017-11-14.b55df55ad9ce70642'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/s3handler.py", line 173, in call
    for fileinfo in fileinfos:
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/fileinfobuilder.py", line 31, in call
    for file_base in files:
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/comparator.py", line 79, in call
    src_file = advance_iterator(src_files)
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/filters.py", line 118, in call
    for file_info in file_infos:
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/filegenerator.py", line 142, in call
    for src_path, extra_information in file_iterator:
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/filegenerator.py", line 210, in list_files
    stats = self._safely_get_file_stats(file_path)
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/filegenerator.py", line 216, in _safely_get_file_stats
    size, last_update = get_file_stat(file_path)
  File "/usr/local/opt/awscli/lib/python3.5/site-packages/awscli/customizations/s3/utils.py", line 214, in get_file_stat
    path, e))
ValueError: Could not retrieve file stat of "/var/log/sterling/production.2017-11-14.b55df55ad9ce70642": [Errno 2] No such file or directory: '/var/log/sterling/production.2017-11-14.b55df55ad9ce70642'
```

I don't see that anything else calls `get_file_stat`, so the translation of OSError to ValueError could probably just be removed.
Confirmed bug, thanks for reporting.
Proposed fix is in #3026.

Resolving patch:
diff --git a/awscli/customizations/s3/filegenerator.py b/awscli/customizations/s3/filegenerator.py
--- a/awscli/customizations/s3/filegenerator.py
+++ b/awscli/customizations/s3/filegenerator.py
@@ -214,7 +214,7 @@ def list_files(self, path, dir_op):
     def _safely_get_file_stats(self, file_path):
         try:
             size, last_update = get_file_stat(file_path)
-        except OSError:
+        except (OSError, ValueError):
             self.triggers_warning(file_path)
         else:
             last_update = self._validate_update_time(last_update, file_path)


Test patch:
diff --git a/tests/functional/s3/test_sync_command.py b/tests/functional/s3/test_sync_command.py
--- a/tests/functional/s3/test_sync_command.py
+++ b/tests/functional/s3/test_sync_command.py
@@ -166,7 +166,37 @@ def test_sync_with_delete_on_downloads(self):
 
         self.assertFalse(os.path.exists(full_path))
 
-    def test_sync_skips_over_files_deleted_between_listing_and_transfer(self):
+    # When a file has been deleted after listing,
+    # awscli.customizations.s3.utils.get_file_stat may raise either some kind
+    # of OSError, or a ValueError, depending on the environment. In both cases,
+    # the behaviour should be the same: skip the file and emit a warning.
+    #
+    # This test covers the case where a ValueError is emitted.
+    def test_sync_skips_over_files_deleted_between_listing_and_transfer_valueerror(self):
+        full_path = self.files.create_file('foo.txt', 'mycontent')
+        cmdline = '%s %s s3://bucket/' % (
+            self.prefix, self.files.rootdir)
+
+        # FileGenerator.list_files should skip over files that cause an
+        # IOError to be raised because they are missing when we try to
+        # get their stats. This IOError is translated to a ValueError in
+        # awscli.customizations.s3.utils.get_file_stat.
+        def side_effect(_):
+            os.remove(full_path)
+            raise ValueError()
+        with patch(
+                'awscli.customizations.s3.filegenerator.get_file_stat',
+                side_effect=side_effect
+                ):
+            self.run_cmd(cmdline, expected_rc=2)
+
+        # We should not call PutObject because the file was deleted
+        # before we could transfer it
+        self.assertEqual(len(self.operations_called), 1, self.operations_called)
+        self.assertEqual(self.operations_called[0][0].name, 'ListObjects')
+
+    # This test covers the case where an OSError is emitted.
+    def test_sync_skips_over_files_deleted_between_listing_and_transfer_oserror(self):
         full_path = self.files.create_file('foo.txt', 'mycontent')
         cmdline = '%s %s s3://bucket/' % (
             self.prefix, self.files.rootdir)