Repository: aws/aws-cli

Issue:
S3 Sync with "--delete" option is deleting files that have folders with the same name!
If you have a folder in your S3 bucket and a file name that contains the same starting characters as the folder name, the target files will be intermittently be deleted upon each run.

For example, create a folder in your S3 bucket:

s3://my-bucket/test/

Now create some txt files in the bucket with 'test' as the first 4 characters of the file name.. ie:

s3://my-bucket/test-123.txt
s3://my-bucket/test-321.txt
s3://my-bucket/test.txt

Run `aws s3 sync --delete s3://my-bucket /my/local/folder`.. Do this 3 or 4 times. You will see each file 'test-123.txt', 'test-321.txt', 'test.txt' will get intermittently be deleted and downloaded with each run. Each run produces different results.

Having files unexpectedly being deleting is a big concern as we use 's3 sync --delete' for daily backup's.

Please see AWS post for originating report https://forums.aws.amazon.com/message.jspa?messageID=497335#497335



Discussion:
I'm trying to repro this and I'm not able to.  How are you creating your folder in the first step?  I tried first creating a folder in the S3 console (clicking the "Create Folder" button).  Then I followed the remaining steps.

```
touch test-123.txt
touch test-321.txt
touch test.txt
aws s3 cp test-123.txt s3://bucket-name/
aws s3 cp test-321.txt s3://bucket-name/
aws s3 cp test.txt s3://bucket-name/
```

When I ran `s3 sync` several times I didn't see anything change:

```
$ aws s3 sync --delete s3://bucket-name/ /tmp/testsync/
$ aws s3 sync --delete s3://bucket-name/ /tmp/testsync/
$ for i in $(seq 10); do aws s3 sync --delete s3://bucket-name/ /tmp/testsync/; done
```

S3 still has the same contents:

```
      LastWriteTime     Length Name
      -------------     ------ ----
                           PRE test/
2013-10-24 17:45:33          0 test-123.txt
2013-10-24 17:45:42          0 test-321.txt
2013-10-24 17:45:47          0 test.txt
```

and the local dir still has the same contents:

```
$ ls
test-123.txt test-321.txt test.txt
```

Perhaps there's a step I'm missing?

Also, what version of the CLI are you using `aws --version`?

Make sure there is a file within the 'test' folder to reproduce the issue:

```
touch test/example.txt
```

Then create your files:

```
touch test-123.txt
touch test-321.txt
touch test.txt
```

Now try running `aws s3 sync --delete s3://my-bucket /home/username/my-bucket` several times

```
delete: s3://my-bucket/test.txt
delete: s3://my-bucket/test-123.txt           
delete: s3://my-bucket/test-321.txt             
upload: ../../my-bucket/test-123.txt to s3://my-bucket/test-123.txt
upload: ../../my-bucket/test-321.txt to s3://my-bucket/test-321.txt
upload: ../../my-bucket/test.txt to s3://my-bucket/test.txt
```

run 2:

```
delete: s3://my-bucket/test-123.txt           
delete: s3://my-bucket/test-321.txt           
upload: ../../my-bucket/test-321.txt to s3://my-bucket/test-321.txt
delete: s3://my-bucket/test.txt               
upload: ../../my-bucket/test-123.txt to s3://my-bucket/test-123.txt
upload: ../../my-bucket/test.txt to s3://my-bucket/test.txt
```

...results are different everytime

Version: aws-cli/1.1.0 Python/2.6.8 Linux/3.4.62-53.42.amzn1.x86_64

Also, I just tested reversing the direction of the sync by swapping source and destination... Looks like the same issue is present when sync'ing back up to S3.

Thanks, I'm now able to repro this issue.  Working on a fix now.


Resolving patch:
diff --git a/awscli/customizations/s3/filegenerator.py b/awscli/customizations/s3/filegenerator.py
--- a/awscli/customizations/s3/filegenerator.py
+++ b/awscli/customizations/s3/filegenerator.py
@@ -80,10 +80,28 @@ def list_files(self, path, dir_op):
             size, last_update = get_file_stat(path)
             yield path, size, last_update
         else:
-            names = sorted(listdir(path))
+            # We need to list files in byte order based on the full
+            # expanded path of the key: 'test/1/2/3.txt'  However, listdir()
+            # will only give us contents a single directory at a time, so we'll
+            # get 'test'.  At the same time we don't want to load the entire
+            # list of files into memory.  This is handled by first going
+            # through the current directory contents and adding the directory
+            # separator to any directories.  We can then sort the contents,
+            # and ensure byte order.
+            names = listdir(path)
+            for i, name in enumerate(names):
+                file_path = join(path, name)
+                if isdir(file_path):
+                    names[i] = name + os.path.sep
+            names.sort()
             for name in names:
                 file_path = join(path, name)
                 if isdir(file_path):
+                    # Anything in a directory will have a prefix of this
+                    # current directory and will come before the
+                    # remaining contents in this directory.  This means we need
+                    # to recurse into this sub directory before yielding the
+                    # rest of this directory's contents.
                     for x in self.list_files(file_path, dir_op):
                         yield x
                 else:


Test patch:
diff --git a/tests/integration/customizations/s3/test_plugin.py b/tests/integration/customizations/s3/test_plugin.py
--- a/tests/integration/customizations/s3/test_plugin.py
+++ b/tests/integration/customizations/s3/test_plugin.py
@@ -323,6 +323,41 @@ def test_sync_with_empty_files(self):
         self.assertTrue(
             self.key_exists(bucket_name=bucket_name, key_name='bar.txt'))
 
+    def test_sync_with_delete_option_with_same_prefix(self):
+        # Test for issue 440 (https://github.com/aws/aws-cli/issues/440)
+        # First, we need to create a directory structure that has a dir with
+        # the same prefix as some of the files:
+        #
+        #  test/foo.txt
+        #  test-123.txt
+        #  test-321.txt
+        #  test.txt
+        bucket_name = self.create_bucket()
+        # create test/foo.txt
+        nested_dir = os.path.join(self.files.rootdir, 'test')
+        os.mkdir(nested_dir)
+        self.files.create_file(os.path.join(nested_dir, 'foo.txt'),
+                               contents='foo.txt contents')
+        # Then create test-123.txt, test-321.txt, test.txt.
+        self.files.create_file('test-123.txt', 'test-123.txt contents')
+        self.files.create_file('test-321.txt', 'test-321.txt contents')
+        self.files.create_file('test.txt', 'test.txt contents')
+
+        # Now sync this content up to s3.
+        p = aws('s3 sync %s s3://%s/' % (self.files.rootdir, bucket_name))
+
+        # Now here's the issue.  If we try to sync the contents down
+        # with the --delete flag we should *not* see any output, the
+        # sync operation should determine that nothing is different and
+        # therefore do nothing.  We can just use --dryrun to show the issue.
+        p = aws('s3 sync s3://%s/ %s --dryrun' % (
+            bucket_name, self.files.rootdir))
+        # These assertion methods will give better error messages than just
+        # checking if the output is empty.
+        self.assertNotIn('download:', p.stdout)
+        self.assertNotIn('delete:', p.stdout)
+        self.assertEqual('', p.stdout)
+
 
 class TestUnicode(BaseS3CLICommand):
     """
diff --git a/tests/unit/customizations/s3/test_filegenerator.py b/tests/unit/customizations/s3/test_filegenerator.py
--- a/tests/unit/customizations/s3/test_filegenerator.py
+++ b/tests/unit/customizations/s3/test_filegenerator.py
@@ -12,6 +12,10 @@
 # language governing permissions and limitations under the License.
 import os
 import unittest
+import tempfile
+import shutil
+
+import six
 
 from awscli.customizations.s3.filegenerator import FileGenerator
 from awscli.customizations.s3.fileinfo import FileInfo
@@ -99,6 +103,59 @@ def test_local_directory(self):
             compare_files(self, result_list[i], ref_list[i])
 
 
+class TestListFilesLocally(unittest.TestCase):
+    maxDiff = None
+
+    def setUp(self):
+        self.directory = six.text_type(tempfile.mkdtemp())
+
+    def tearDown(self):
+        shutil.rmtree(self.directory)
+
+    def test_list_files_is_in_sorted_order(self):
+        p = os.path.join
+        open(p(self.directory, 'test-123.txt'), 'w').close()
+        open(p(self.directory, 'test-321.txt'), 'w').close()
+        open(p(self.directory, 'test123.txt'), 'w').close()
+        open(p(self.directory, 'test321.txt'), 'w').close()
+        os.mkdir(p(self.directory, 'test'))
+        open(p(self.directory, 'test', 'foo.txt'), 'w').close()
+
+        file_generator = FileGenerator(None, None, None, None)
+        values = list(el[0] for el in file_generator.list_files(
+            self.directory, dir_op=True))
+        self.assertEqual(values, list(sorted(values)))
+
+    def test_list_local_files_with_unicode_chars(self):
+        p = os.path.join
+        open(p(self.directory, u'a'), 'w').close()
+        open(p(self.directory, u'a\u0300'), 'w').close()
+        open(p(self.directory, u'a\u0300-1'), 'w').close()
+        open(p(self.directory, u'a\u03001'), 'w').close()
+        open(p(self.directory, u'z'), 'w').close()
+        open(p(self.directory, u'\u00e6'), 'w').close()
+        os.mkdir(p(self.directory, u'a\u0300a'))
+        open(p(self.directory, u'a\u0300a', u'a'), 'w').close()
+        open(p(self.directory, u'a\u0300a', u'z'), 'w').close()
+        open(p(self.directory, u'a\u0300a', u'\u00e6'), 'w').close()
+
+        file_generator = FileGenerator(None, None, None, None)
+        values = list(el[0] for el in file_generator.list_files(
+            self.directory, dir_op=True))
+        expected_order = [os.path.join(self.directory, el) for el in [
+            u"a",
+            u"a\u0300",
+            u"a\u0300-1",
+            u"a\u03001",
+            u"a\u0300a%sa" % os.path.sep,
+            u"a\u0300a%sz" % os.path.sep,
+            u"a\u0300a%s\u00e6" % os.path.sep,
+            u"z",
+            u"\u00e6"
+        ]]
+        self.assertEqual(values, expected_order)
+
+
 class S3FileGeneratorTest(unittest.TestCase):
     def setUp(self):
         self.session = FakeSession()