Repository: aws/aws-cli

Issue:
aws s3 sync --delete behaves differently if source is <LocalPath> rather than <S3Path>
This issue was raised on: https://stackoverflow.com/questions/32304326/amazon-s3-sync-deleting-excluded-files

Basically, doing `aws s3 sync . s3://bucket2 --delete` behaves differently to `aws s3 sync s3://bucket1 s3://bucket2 --delete` even if the two source directories are identical.

It would appear that the logic for `--delete` behaves differently between Local and S3. An example is well-documented in the stackoverflow issue.



Discussion:
I can confirm the file being unexpectedly deleted. The files being copied over from the s3 bucket to other s3 bucket is as expected because the objects in the origin bucket are newer than the objects in the destination bucket.

Confirmed on both OSX and Linux, investigating


Resolving patch:
diff --git a/awscli/customizations/s3/filters.py b/awscli/customizations/s3/filters.py
--- a/awscli/customizations/s3/filters.py
+++ b/awscli/customizations/s3/filters.py
@@ -36,12 +36,16 @@ def create_filter(parameters):
             # the bucket to be the root dir.
             src_rootdir = _get_s3_root(source_location,
                                        parameters['dir_op'])
-            dst_rootdir = _get_local_root(parameters['dest'],
-                                          parameters['dir_op'])
         else:
             src_rootdir = _get_local_root(parameters['src'], parameters['dir_op'])
+
+        destination_location = parameters['dest']
+        if destination_location.startswith('s3://'):
             dst_rootdir = _get_s3_root(parameters['dest'],
                                        parameters['dir_op'])
+        else:
+            dst_rootdir = _get_local_root(parameters['dest'],
+                                          parameters['dir_op'])
 
         return Filter(real_filters, src_rootdir, dst_rootdir)
     else:


Test patch:
diff --git a/tests/unit/customizations/s3/test_filters.py b/tests/unit/customizations/s3/test_filters.py
--- a/tests/unit/customizations/s3/test_filters.py
+++ b/tests/unit/customizations/s3/test_filters.py
@@ -202,6 +202,26 @@ def test_create_root_s3_no_dir_op(self):
         filtered = list(s3_filter.call(s3_files))
         self.assertEqual(len(filtered), 0)
 
+    def test_create_filter_s3_to_s3(self):
+        source = 'bucket/'
+        destination = 'bucket-2/'
+        pattern = '*'
+        parameters = {'filters': [['--exclude', pattern],
+                                  ['--include', '*.jpg']],
+                      'dir_op': True,
+                      'src': 's3://' + source,
+                      'dest': 's3://' + destination}
+        s3_filter = self.create_filter(parameters=parameters)
+
+        source_pattern = s3_filter.patterns[0][1]
+        destination_pattern = s3_filter.dst_patterns[0][1]
+        self.assertEquals(source_pattern, source + pattern)
+        self.assertEquals(destination_pattern, destination + pattern)
+
+        filtered = list(s3_filter.call(self.s3_files))
+        self.assertEquals(len(filtered), 2)
+        for filtered_file in filtered:
+            self.assertFalse('.txt' in filtered_file.src)
 
 if __name__ == "__main__":
     unittest.main()