Repository: aws/aws-cli

Issue:
Define semantics of the root of include/exclude filters
Our docs currently don't specify _what_ an include/exclude filter is matched against.

Implementation-wise, we currently expand patterns into full paths.  The full paths are constructed based on our current working directory, which provides unintuitive behavior in certain cases, particularly:

1) when matching a prefix pattern (anything that doesn't start with a `*`)
2) when specifying a source directory that's not the same as the root of the source directory.

For example:

If I'm currently in `/a/b/c` and I run `aws s3 cp /z/y/x s3://bucket --recursive --exclude "*"` then nothing will be excluded this is because we expand the pattern `"*"` to an absolute path based on our CWD and try to match it against the abspath of each file.  So we'd have:

```
/z/y/x/file1  matched against /a/b/c/*   -> does not match
/z/y/x/file2  matched against /a/b/c/*   -> does not match
```

I believe most of these issues can be resolved by running the queries against the root of the source location so in the example above our source location is `/z/y/x` so our full exclude filter would be `/z/y/x/*` If a file is specified, we'd run it against the dirname of the file.

This should be a backwards compatible change.



Discussion:
I missed your issue when I was searching but yeah, same thing. I agree, your idea would make the most sense for handling the situation.

However, there would be one use case where it would break and that would be an include/exclude that includes directories. It seems from my look at the code earlier, that currently, if you specify a relative path on the command line, you could use --exclude "some/path/*" and the CWD would be pre-pended and you'd match against files in that sub directory. If you prepend the dirname of the file to the pattern, as proposed, then the pattern would be limited to only matching on file name.

For my uses, pattern matching on the filename is fine, and the documentation seems a bit ambiguous to my reading about what the intended design would be. 

Another possible way to fix it might be to make the patterns absolute when they are parsed, so presumably you would also have access to the command line argument for the source absolute path. This shouldn't affect the relative path case as in that case you'd be pre-pending the CWD no matter where you do it. EDIT: Although this would probably break the non-local patterns. So maybe keep track of the root path somewhere? Have the process chdir to the root path?


Resolving patch:
diff --git a/awscli/customizations/s3/filters.py b/awscli/customizations/s3/filters.py
--- a/awscli/customizations/s3/filters.py
+++ b/awscli/customizations/s3/filters.py
@@ -14,24 +14,61 @@
 import fnmatch
 import os
 
+from awscli.customizations.s3.utils import split_s3_bucket_key
+
 
 LOG = logging.getLogger(__name__)
 
 
+def create_filter(parameters):
+    """Given the CLI parameters dict, create a Filter object."""
+    # We need to evaluate all the filters based on the source
+    # directory.
+    if parameters['filters']:
+        cli_filters = parameters['filters']
+        real_filters = []
+        for filter_type, filter_pattern in cli_filters:
+            real_filters.append((filter_type.lstrip('-'),
+                                 filter_pattern))
+        source_location = parameters['src']
+        if source_location.startswith('s3://'):
+            # This gives us (bucket, keyname) and we want
+            # the bucket to be the root dir.
+            rootdir = split_s3_bucket_key(source_location)[0]
+        else:
+            if parameters.get('dir_op'):
+                rootdir = os.path.abspath(parameters['src'])
+            else:
+                rootdir = os.path.abspath(os.path.dirname(parameters['src']))
+        return Filter(real_filters, rootdir)
+    else:
+        return Filter({}, None)
+
+
 class Filter(object):
     """
     This is a universal exclude/include filter.
     """
-    def __init__(self, parameters=None):
+    def __init__(self, patterns, rootdir):
         """
-        :var self.patterns: A list of patterns. A pattern consits of a list
-            whose first member is a string '--exclude' or '--include'.
+        :var patterns: A list of patterns. A pattern consits of a list
+            whose first member is a string 'exclude' or 'include'.
             The second member is the actual rule.
+        :var rootdir: The root directory where the patterns are evaluated.
+            This will generally be the directory of the source location.
+
         """
-        if 'filters' in parameters:
-            self.patterns = parameters['filters']
-        else:
-            self.patterns = []
+        self.patterns = self._full_path_patterns(patterns, rootdir)
+
+    def _full_path_patterns(self, original_patterns, rootdir):
+        # We need to transform the patterns into patterns that have
+        # the root dir prefixed, so things like ``--exclude "*"``
+        # will actually be ['exclude', '/path/to/root/*']
+        full_patterns = []
+        for pattern in original_patterns:
+            full_patterns.append(
+                (pattern[0], os.path.join(rootdir, pattern[1])))
+        return full_patterns
 
     def call(self, file_infos):
         """
@@ -50,29 +87,24 @@ def call(self, file_infos):
         for file_info in file_infos:
             file_path = file_info.src
             file_status = (file_info, True)
-
             for pattern in self.patterns:
                 pattern_type = pattern[0]
                 if file_info.src_type == 'local':
                     path_pattern = pattern[1].replace('/', os.sep)
-                    full_path_pattern = os.path.abspath(path_pattern)
-
                 else:
                     path_pattern = pattern[1].replace(os.sep, '/')
-                    full_path_pattern = os.path.join(file_path.split('/')[0],
-                                                     path_pattern)
-                is_match = fnmatch.fnmatch(file_path, full_path_pattern)
-                if is_match and pattern_type == '--include':
+                is_match = fnmatch.fnmatch(file_path, path_pattern)
+                if is_match and pattern_type == 'include':
                     file_status = (file_info, True)
                     LOG.debug("%s matched include filter: %s",
-                              file_path, full_path_pattern)
-                elif is_match and pattern_type == '--exclude':
+                              file_path, path_pattern)
+                elif is_match and pattern_type == 'exclude':
                     file_status = (file_info, False)
                     LOG.debug("%s matched exclude filter: %s",
-                              file_path, full_path_pattern)
+                              file_path, path_pattern)
                 else:
                     LOG.debug("%s did not match %s filter: %s",
-                              file_path, pattern_type[2:], full_path_pattern)
+                              file_path, pattern_type[2:], path_pattern)
             LOG.debug("=%s final filtered status, should_include: %s",
                       file_path, file_status[1])
             if file_status[1]:
diff --git a/awscli/customizations/s3/s3.py b/awscli/customizations/s3/s3.py
--- a/awscli/customizations/s3/s3.py
+++ b/awscli/customizations/s3/s3.py
@@ -27,7 +27,7 @@
 from awscli.customizations.s3.fileformat import FileFormat
 from awscli.customizations.s3.filegenerator import FileGenerator
 from awscli.customizations.s3.fileinfo import TaskInfo
-from awscli.customizations.s3.filters import Filter
+from awscli.customizations.s3.filters import create_filter
 from awscli.customizations.s3.s3handler import S3Handler
 from awscli.customizations.s3.description import add_command_descriptions, \
     add_param_descriptions
@@ -558,41 +558,42 @@ def run(self):
         s3handler = S3Handler(self.session, self.parameters)
 
         command_dict = {}
-        command_dict['sync'] = {'setup': [files, rev_files],
-                                'file_generator': [file_generator,
-                                                   rev_generator],
-                                'filters': [Filter(self.parameters),
-                                            Filter(self.parameters)],
-                                'comparator': [Comparator(self.parameters)],
+        if self.cmd == 'sync':
+            command_dict = {'setup': [files, rev_files],
+                                    'file_generator': [file_generator,
+                                                    rev_generator],
+                                    'filters': [create_filter(self.parameters),
+                                                create_filter(self.parameters)],
+                                    'comparator': [Comparator(self.parameters)],
+                                    's3_handler': [s3handler]}
+        elif self.cmd == 'cp':
+            command_dict = {'setup': [files],
+                                'file_generator': [file_generator],
+                                'filters': [create_filter(self.parameters)],
+                                's3_handler': [s3handler]}
+        elif self.cmd == 'rm':
+            command_dict = {'setup': [files],
+                                'file_generator': [file_generator],
+                                'filters': [create_filter(self.parameters)],
+                                's3_handler': [s3handler]}
+        elif self.cmd == 'mv':
+            command_dict = {'setup': [files],
+                                'file_generator': [file_generator],
+                                'filters': [create_filter(self.parameters)],
+                                's3_handler': [s3handler]}
+        elif self.cmd == 'mb':
+            command_dict = {'setup': [taskinfo],
+                                's3_handler': [s3handler]}
+        elif self.cmd == 'rb':
+            command_dict = {'setup': [taskinfo],
                                 's3_handler': [s3handler]}
 
-        command_dict['cp'] = {'setup': [files],
-                              'file_generator': [file_generator],
-                              'filters': [Filter(self.parameters)],
-                              's3_handler': [s3handler]}
-
-        command_dict['rm'] = {'setup': [files],
-                              'file_generator': [file_generator],
-                              'filters': [Filter(self.parameters)],
-                              's3_handler': [s3handler]}
-
-        command_dict['mv'] = {'setup': [files],
-                              'file_generator': [file_generator],
-                              'filters': [Filter(self.parameters)],
-                              's3_handler': [s3handler]}
-
-        command_dict['mb'] = {'setup': [taskinfo],
-                              's3_handler': [s3handler]}
-
-        command_dict['rb'] = {'setup': [taskinfo],
-                              's3_handler': [s3handler]}
-
-        files = command_dict[self.cmd]['setup']
+        files = command_dict['setup']
 
         while self.instructions:
             instruction = self.instructions.pop(0)
             file_list = []
-            components = command_dict[self.cmd][instruction]
+            components = command_dict[instruction]
             for i in range(len(components)):
                 if len(files) > len(components):
                     file_list.append(components[i].call(*files))
diff --git a/awscli/customizations/s3/utils.py b/awscli/customizations/s3/utils.py
--- a/awscli/customizations/s3/utils.py
+++ b/awscli/customizations/s3/utils.py
@@ -69,6 +69,19 @@ def find_bucket_key(s3_path):
     return bucket, s3_key
 
 
+def split_s3_bucket_key(s3_path):
+    """Split s3 path into bucket and key prefix.
+
+    This will also handle the s3:// prefix.
+
+    :return: Tuple of ('bucketname', 'keyname')
+
+    """
+    if s3_path.startswith('s3://'):
+        s3_path = s3_path[5:]
+    return find_bucket_key(s3_path)
+
+
 def get_file_stat(path):
     """
     This is a helper function that given a local path return the size of


Test patch:
diff --git a/tests/integration/customizations/s3/test_plugin.py b/tests/integration/customizations/s3/test_plugin.py
--- a/tests/integration/customizations/s3/test_plugin.py
+++ b/tests/integration/customizations/s3/test_plugin.py
@@ -22,6 +22,7 @@
 import tempfile
 import shutil
 import platform
+import contextlib
 
 import botocore.session
 
@@ -31,6 +32,16 @@
 from awscli.customizations.s3 import constants
 
 
+@contextlib.contextmanager
+def cd(directory):
+    original = os.getcwd()
+    try:
+        os.chdir(directory)
+        yield
+    finally:
+        os.chdir(original)
+
+
 class BaseS3CLICommand(unittest.TestCase):
     """Base class for aws s3 command.
 
@@ -62,7 +73,7 @@ def create_bucket(self):
         self.addCleanup(self.delete_bucket, bucket_name)
         return bucket_name
 
-    def put_object(self, bucket_name, key_name, contents):
+    def put_object(self, bucket_name, key_name, contents=''):
         operation = self.service.get_operation('PutObject')
         http = operation.call(self.endpoint, bucket=bucket_name,
                               key=key_name, body=contents)[0]
@@ -603,5 +614,82 @@ def test_create_website_configuration(self):
         self.assertEqual(parsed['IndexDocument']['Suffix'], 'index.html')
 
 
+class TestIncludeExcludeFilters(BaseS3CLICommand):
+    def assert_no_files_would_be_uploaded(self, p):
+        self.assert_no_errors(p)
+        # There should be no output.
+        self.assertEqual(p.stdout, '')
+        self.assertEqual(p.stderr, '')
+
+    def test_basic_exclude_filter_for_single_file(self):
+        full_path = self.files.create_file('foo.txt', 'this is foo.txt')
+        # With no exclude we should upload the file.
+        p = aws('s3 cp %s s3://random-bucket-name/ --dryrun' % full_path)
+        self.assert_no_errors(p)
+        self.assertIn('(dryrun) upload:', p.stdout)
+
+        p2 = aws('s3 cp %s s3://random-bucket-name/ --dryrun --exclude "*"'
+                 % full_path)
+        self.assert_no_files_would_be_uploaded(p2)
+
+    def test_explicitly_exclude_single_file(self):
+        full_path = self.files.create_file('foo.txt', 'this is foo.txt')
+        p = aws('s3 cp %s s3://random-bucket-name/ --dryrun --exclude foo.txt'
+                 % full_path)
+        self.assert_no_files_would_be_uploaded(p)
+
+    def test_cwd_doesnt_matter(self):
+        full_path = self.files.create_file('foo.txt', 'this is foo.txt')
+        with cd(os.path.expanduser('~')):
+            p = aws('s3 cp %s s3://random-bucket-name/ --dryrun --exclude "*"'
+                    % full_path)
+        self.assert_no_files_would_be_uploaded(p)
+
+    def test_recursive_exclude(self):
+        # create test/foo.txt
+        nested_dir = os.path.join(self.files.rootdir, 'test')
+        os.mkdir(nested_dir)
+        self.files.create_file(os.path.join(nested_dir, 'foo.txt'),
+                               contents='foo.txt contents')
+        # Then create test-123.txt, test-321.txt, test.txt.
+        self.files.create_file('test-123.txt', 'test-123.txt contents')
+        self.files.create_file('test-321.txt', 'test-321.txt contents')
+        self.files.create_file('test.txt', 'test.txt contents')
+        # An --exclude test* should exclude everything here.
+        p = aws('s3 cp %s s3://random-bucket-name/ --dryrun --exclude "*" '
+                 '--recursive' % self.files.rootdir)
+        self.assert_no_files_would_be_uploaded(p)
+
+        # We can include the test directory though.
+        p = aws('s3 cp %s s3://random-bucket-name/ --dryrun '
+                '--exclude "*" --include "test/*" --recursive'
+                % self.files.rootdir)
+        self.assert_no_errors(p)
+        self.assertRegexpMatches(p.stdout,
+                                 r'\(dryrun\) upload:.*test/foo.txt.*')
+
+    def test_s3_filtering(self):
+        # Should behave the same as local file filtering.
+        bucket_name = self.create_bucket()
+        self.put_object(bucket_name, key_name='foo.txt')
+        self.put_object(bucket_name, key_name='bar.txt')
+        self.put_object(bucket_name, key_name='baz.jpg')
+        p = aws('s3 rm s3://%s/ --dryrun --exclude "*" --recursive'
+                % bucket_name)
+        self.assert_no_files_would_be_uploaded(p)
+
+        p = aws(
+            's3 rm s3://%s/ --dryrun --exclude "*.jpg" --exclude "*.txt" '
+            '--recursive' % bucket_name)
+        self.assert_no_files_would_be_uploaded(p)
+
+        p = aws('s3 rm s3://%s/ --dryrun --exclude "*.txt" --recursive'
+                % bucket_name)
+        self.assert_no_errors(p)
+        self.assertRegexpMatches(p.stdout, r'\(dryrun\) delete:.*baz.jpg.*')
+        self.assertNotIn(p.stdout, 'bar.txt')
+        self.assertNotIn(p.stdout, 'foo.txt')
+
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/tests/unit/customizations/s3/test_filters.py b/tests/unit/customizations/s3/test_filters.py
--- a/tests/unit/customizations/s3/test_filters.py
+++ b/tests/unit/customizations/s3/test_filters.py
@@ -12,11 +12,20 @@
 # language governing permissions and limitations under the License.
 import os
 import unittest
+import platform
 
 from awscli.customizations.s3.fileinfo import FileInfo
 from awscli.customizations.s3.filters import Filter
 
 
+def platform_path(filepath):
+    # Convert posix platforms to windows platforms.
+    if platform.system().lower() == 'windows':
+        filepath = filepath.replace('/', os.sep)
+        filepath = 'C:' + filepath
+    return filepath
+
+
 class FiltersTest(unittest.TestCase):
     def setUp(self):
         self.local_files = [
@@ -42,8 +51,15 @@ def file_info(self, filename, src_type='local'):
                         dest_type=dest_type, operation_name='',
                         service=None, endpoint=None)
 
+    def create_filter(self, filters=None, root=None):
+        if root is None:
+            root = os.getcwd()
+        if filters is None:
+            filters = {}
+        return Filter(filters, root)
+
     def test_no_filter(self):
-        exc_inc_filter = Filter({})
+        exc_inc_filter = self.create_filter()
         matched_files = list(exc_inc_filter.call(self.local_files))
         self.assertEqual(matched_files, self.local_files)
 
@@ -51,8 +67,8 @@ def test_no_filter(self):
         self.assertEqual(matched_files2, self.s3_files)
 
     def test_include(self):
-        patterns = [['--include', '*.txt']]
-        include_filter = Filter({'filters': [['--include', '*.txt']]})
+        patterns = [['include', '*.txt']]
+        include_filter = self.create_filter([['include', '*.txt']])
         matched_files = list(include_filter.call(self.local_files))
         self.assertEqual(matched_files, self.local_files)
 
@@ -60,7 +76,7 @@ def test_include(self):
         self.assertEqual(matched_files2, self.s3_files)
 
     def test_exclude(self):
-        exclude_filter = Filter({'filters': [['--exclude', '*']]})
+        exclude_filter = self.create_filter([['exclude', '*']])
         matched_files = list(exclude_filter.call(self.local_files))
         self.assertEqual(matched_files, [])
 
@@ -68,8 +84,8 @@ def test_exclude(self):
         self.assertEqual(matched_files, [])
 
     def test_exclude_include(self):
-        patterns = [['--exclude', '*'], ['--include', '*.txt']]
-        exclude_include_filter = Filter({'filters': patterns})
+        patterns = [['exclude', '*'], ['include', '*.txt']]
+        exclude_include_filter = self.create_filter(patterns)
         matched_files = list(exclude_include_filter.call(self.local_files))
         self.assertEqual(matched_files, [self.local_files[0]])
 
@@ -77,8 +93,8 @@ def test_exclude_include(self):
         self.assertEqual(matched_files, [self.s3_files[0]])
 
     def test_include_exclude(self):
-        patterns = [['--include', '*.txt'], ['--exclude', '*']]
-        exclude_all_filter = Filter({'filters': patterns})
+        patterns = [['include', '*.txt'], ['exclude', '*']]
+        exclude_all_filter = self.create_filter(patterns)
         matched_files = list(exclude_all_filter.call(self.local_files))
         self.assertEqual(matched_files, [])
 
@@ -100,7 +116,7 @@ def test_prefix_filtering_consistent(self):
             self.file_info('bucket/nottest1.txt', src_type='s3'),
         ]
         # If I apply the filter to the local to the local files.
-        exclude_filter = Filter({'filters': [['--exclude', 't*']]})
+        exclude_filter = self.create_filter([['exclude', 't*']])
         filtered_files = list(exclude_filter.call(local_files))
         self.assertEqual(len(filtered_files), 1)
         self.assertEqual(os.path.basename(filtered_files[0].src),
@@ -108,6 +124,7 @@ def test_prefix_filtering_consistent(self):
 
         # I should get the same result if I apply the same filter to s3
         # objects.
+        exclude_filter = self.create_filter([['exclude', 't*']], root='bucket')
         same_filtered_files = list(exclude_filter.call(remote_files))
         self.assertEqual(len(same_filtered_files), 1)
         self.assertEqual(os.path.basename(same_filtered_files[0].src),
@@ -120,14 +137,31 @@ def test_bucket_exclude_with_prefix(self):
             self.file_info('bucket/dir1/notkey3.txt', src_type='s3'),
         ]
         filtered_files = list(
-            Filter({'filters': [['--exclude', 'dir1/*']]}).call(s3_files))
+            self.create_filter([['exclude', 'dir1/*']],
+                               root='bucket').call(s3_files))
         self.assertEqual(filtered_files, [])
 
         key_files = list(
-            Filter({'filters': [['--exclude', 'dir1/key*']]}).call(s3_files))
+            self.create_filter([['exclude', 'dir1/key*']],
+                               root='bucket').call(s3_files))
         self.assertEqual(len(key_files), 1)
         self.assertEqual(key_files[0].src, 'bucket/dir1/notkey3.txt')
 
+    def test_root_dir(self):
+        p = platform_path
+        local_files = [self.file_info(p('/foo/bar/baz.txt'), src_type='local')]
+        local_filter = self.create_filter([['exclude', 'baz.txt']],
+                                          root=p('/foo/bar/'))
+        filtered = list(local_filter.call(local_files))
+        self.assertEqual(filtered, [])
+
+        # However, if we're at the root of /foo', then this filter won't match.
+        local_filter = self.create_filter([['exclude', 'baz.txt']],
+                                          root=p('/foo/'))
+        filtered = list(local_filter.call(local_files))
+        self.assertEqual(len(filtered), 1)
+        self.assertEqual(filtered[0].src, p('/foo/bar/baz.txt'))
+
 
 if __name__ == "__main__":
     unittest.main()