Repository: aws/aws-cli

Issue:
[Errno 2] No such file or directory
I know this has been brought up before but I think I have found another scenario where this error can occur.

So I'm running the following command to sync files to S3.

`aws s3 sync ./source/ s3://bucket/destination/ --exclude '*' --include '*.mp3' --storage-class STANDARD_IA --no-follow-symlink`

Great, works fine. But every now and again the sync fails with the error:

`[Errno 2] No such file or directory`

I had a look around and saw people were talking about broken symlinks. I though it was quite strange seeing as my source is just a folder with MP3's - but I tried the `--no-follow-symlink` flag anyway. This still didn't work.

So I have a theory which I hope will be correct and result in a fix. When I run the sync command, it gathers a list of all the files in the directory then syncs them. This list doesn't change from the initial gathering. So I'm trying to sync a folder which regularly changes. So my theory is that while I'm trying to sync files, at the same time some of them are being deleted by their respective users, but because the initial file list is only retrieved at the beginning of the sync - it fails because the file doesn't exist anymore.

It would be nice just to ignore these failures. Let me know if I'm correct in thinking this.



Discussion:
That definitely sounds like the likely cause for the error you're seeing.

If that's the case, we'd also have the same problem even once the upload process has started.  Each task thread gets its own file handle, so it's possible that if the file is deleted while it's being uploaded one of the task threads will get a file not found error.

We should be able to fix the case you've mentioned though.  Let me discuss with the others.

I still can't sync an important directory because of this issue. Is there any way to make this a priority if I paid something? Also although I don't know python I can give it a shot myself trying to fix it but would you be kind enough to point me in the general direction of which file I'd be looking to edit? Thanks

Also experiencing the same issue here. In my case, I rsync to a folder and then aws sync that to S3. Rsync by default adds a `.tmp` extension to files as they copy, and I exclude them with `--exclude *.tmp`. However, when rsync finishes and renames it back to the original filename, I get 

`[Errno 2] No such file or directory`

Shouldn't aws be excluding the `.tmp` file? Why is it coming back to it to notice that it's no longer there? Seems like a bug to me.

I am having a similar problem.  In my case the sync does not continue.  It stops there. I have discovered today that you can't get it ignore anything that starts with ".".  Also I am getting an error with the --no-follow-symlinks command.  The error is "Unknown options: --no-follow-symlinks"

I'm also having this issue syncing anything starting with `.` that also has a `~` in the file name, getting the exact same generic `[Errno 2] No such file or directory`  Think of it like this for example: `./6025/Proj_Name/Foo/Review/031416/RFJTJ~7`

I ran a `find . -iname "*~*"` on my directory and found a slew of other files that happen to fall into this area of concern, but I don't necessarily want to go and remove that character or remove those offending "characters to avoid" as they maybe important, but I don't know, I'm just told to "archive all of it."

What's the best way to overcome this concern?  I need to finish uploading this stuff.  I've tried s4cmd, s3cmd, Cyberduck and some others as a workaround to AWS' exclude feature leaving something left to be desired... all run into the same issue.  

Guys I really don't understand why you can't fix this issue. It's such a simple solution of checking the existence of the file first and not throwing an exception if it doesn't exist!! 

The problem is it takes a huge amount of time for the CLI to iterate through a large directory, when it does it stores the file list in cache while it runs through and syncs to S3. In the time that it's created the file list and when it actually get's to some of the files...some files may have been deleted - resulting in this problem.

I've done a debug and here is the stack trace

```
2016-11-15 00:48:54,237 - MainThread - awscli.customizations.s3.results - DEBUG - Exception caught during command execution: [Errno 2] No such file or directory: '/var/www/mp3s/file.mp3'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/s3handler.py", line 170, in call
    for fileinfo in fileinfos:
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/fileinfobuilder.py", line 31, in call
    for file_base in files:
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/comparator.py", line 79, in call
    src_file = advance_iterator(src_files)
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/filegenerator.py", line 142, in call
    for src_path, extra_information in file_iterator:
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/filegenerator.py", line 211, in list_files
    size, last_update = get_file_stat(file_path)
  File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/s3/utils.py", line 212, in get_file_stat
    stats = os.stat(path)
OSError: [Errno 2] No such file or directory: '/var/www/mp3s/file.mp3'
fatal error: [Errno 2] No such file or directory: '/var/www/mp3s/file.mp3'
2016-11-15 00:48:54,247 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.
```

Please guys can you look into this, it renders the CLI useless on large constantly changing directories. There's not a whole lot of solutions out there other than the official AWS CLI.

We are running into this same issue using `aws s3 sync` as part of log rotation (a file is rotated while the sync is running which causes issues even though it isn't part of the `--include` pattern).
I believe the issue is that there is a race condition between the check for the [file existing in `issues_warning`](https://github.com/aws/aws-cli/blob/develop/awscli/customizations/s3/filegenerator.py#L285) and when the [`stat` is looked up](https://github.com/aws/aws-cli/blob/develop/awscli/customizations/s3/filegenerator.py#L211). In this brief period of time, the file gets removed.

I'm going to look into whether it's reasonable to EAFP here.

Resolving patch:
diff --git a/awscli/customizations/s3/filegenerator.py b/awscli/customizations/s3/filegenerator.py
--- a/awscli/customizations/s3/filegenerator.py
+++ b/awscli/customizations/s3/filegenerator.py
@@ -172,10 +172,9 @@ def list_files(self, path, dir_op):
         error, listdir = os.error, os.listdir
         if not self.should_ignore_file(path):
             if not dir_op:
-                size, last_update = get_file_stat(path)
-                last_update = self._validate_update_time(last_update, path)
-                yield path, {'Size': size, 'LastModified': last_update}
-
+                stats = self._safely_get_file_stats(path)
+                if stats:
+                    yield stats
             else:
                 # We need to list files in byte order based on the full
                 # expanded path of the key: 'test/1/2/3.txt'  However,
@@ -208,13 +207,18 @@ def list_files(self, path, dir_op):
                         for x in self.list_files(file_path, dir_op):
                             yield x
                     else:
-                        size, last_update = get_file_stat(file_path)
-                        last_update = self._validate_update_time(
-                            last_update, path)
-                        yield (
-                            file_path,
-                            {'Size': size, 'LastModified': last_update}
-                        )
+                        stats = self._safely_get_file_stats(file_path)
+                        if stats:
+                            yield stats
+
+    def _safely_get_file_stats(self, file_path):
+        try:
+            size, last_update = get_file_stat(file_path)
+        except OSError:
+            self.triggers_warning(file_path)
+        else:
+            last_update = self._validate_update_time(last_update, file_path)
+            return file_path, {'Size': size, 'LastModified': last_update}
 
     def _validate_update_time(self, update_time, path):
         # If the update time is None we know we ran into an invalid tiemstamp.


Test patch:
diff --git a/tests/functional/s3/test_sync_command.py b/tests/functional/s3/test_sync_command.py
--- a/tests/functional/s3/test_sync_command.py
+++ b/tests/functional/s3/test_sync_command.py
@@ -12,6 +12,7 @@
 # language governing permissions and limitations under the License.
 from awscli.testutils import set_invalid_utime
 from awscli.testutils import BaseAWSCommandParamsTest, FileCreator
+from mock import patch
 import os
 
 from awscli.compat import six
@@ -99,7 +100,7 @@ def test_glacier_sync_with_force_glacier(self):
         self.assertEqual(len(self.operations_called), 2, self.operations_called)
         self.assertEqual(self.operations_called[0][0].name, 'ListObjects')
         self.assertEqual(self.operations_called[1][0].name, 'GetObject')
-    
+
     def test_handles_glacier_incompatible_operations(self):
         self.parsed_responses = [
             {'Contents': [
@@ -164,3 +165,25 @@ def test_sync_with_delete_on_downloads(self):
         self.assertEqual(self.operations_called[0][0].name, 'ListObjects')
 
         self.assertFalse(os.path.exists(full_path))
+
+    def test_sync_skips_over_files_deleted_between_listing_and_transfer(self):
+        full_path = self.files.create_file('foo.txt', 'mycontent')
+        cmdline = '%s %s s3://bucket/' % (
+            self.prefix, self.files.rootdir)
+
+        # FileGenerator.list_files should skip over files that cause an
+        # OSError to be raised because they are missing when we try to
+        # get their stats.
+        def side_effect(_):
+            os.remove(full_path)
+            raise OSError()
+        with patch(
+                'awscli.customizations.s3.filegenerator.get_file_stat',
+                side_effect=side_effect
+                ):
+            self.run_cmd(cmdline, expected_rc=2)
+
+        # We should not call PutObject because the file was deleted
+        # before we could transfer it
+        self.assertEqual(len(self.operations_called), 1, self.operations_called)
+        self.assertEqual(self.operations_called[0][0].name, 'ListObjects')