Repository: aws/aws-cli

Issue:
AWS CLI can't remove objects containing "\n" in Key
Here is how to reproduce the issue.

```
~ $ aws s3 mb s3://j3tm0t0-rmtest
make_bucket: s3://j3tm0t0-rmtest/

# upload file with \n in filename
~ $ aws s3 cp hoge s3://j3tm0t0-rmtest/hoge^Mpuge <-- Ctrl+v Ctrl+m
pugead: ./hoge to s3://j3tm0t0-rmtest/hoge

# file exists
~ $ aws s3 ls s3://j3tm0t0-rmtest/
2014-02-25 21:18:23          5 hoge
puge

# direct API shows "\n" in the key
 $ aws s3api list-objects --bucket j3tm0t0-rmtest --output json --query Contents[].Key
[
    "hoge\npuge"
]

# remove bucket --force pretends to delete it, but it doesn't actually and failed to remove bucket
~ $ aws s3 rb s3://j3tm0t0-rmtest --force
delete: s3://j3tm0t0-rmtest/hoge
puge
remove_bucket failed: s3://j3tm0t0-rmtest/ A client error (BucketNotEmpty) occurred when calling the DeleteBucket operation: The bucket you tried to delete is not empty

# it still exists
~ $ aws s3 ls s3://j3tm0t0-rmtest/
2014-02-25 21:18:23          5 hoge
puge

# rm --recursive also the same
~ $ aws s3 rm s3://j3tm0t0-rmtest/ --recursive
delete: s3://j3tm0t0-rmtest/hoge
puge
~ $ aws s3api list-objects --bucket j3tm0t0-rmtest --output json --query Contents[].Key
[
    "hoge\npuge"
]
```



Discussion:
Looking into this.  From what I can tell it's only when we try to do a recursive type of delete (either `s3 rb --force` or `s3 rm --recursive`).  Trying to delete the file directly seems to work for me:

```
$ aws s3 cp bar.txt s3://bucket/newlinetest/bar^M.txt
$ aws s3 ls bucket/newlinetest/
2014-02-26 07:56:04         11 bar
.txt
$ aws s3 rm s3://bucket/newlinetest/bar^M.txt
.txtte: s3://bucket/newlinetest/bar
$ aws s3 ls bucket/newlinetest/
$
```

It seems to be that in the recursive case we send:

```
"DELETE /newlinetest/bar%0A.txt HTTP/1.1" 204 0
```

But when I specify the single file directly we send:

```
"DELETE /newlinetest/bar%0D.txt HTTP/1.1" 204 0
```

Though it's interesting that S3 sends a 204 response in both cases.  I would have expected the first request with %0A to generate a 404.

So I believe the issue here is our use of cElementTree as an XML parser.  I can repro the issue using just the xml parser alone.  What's happening is that if we give the parser keys with a carriage return, it will automatically convert them to a new line char.  This is why it does not work in the recursive delete case.  We need to parse the XML response to know what keys to delete.  But when a single file is explicitly specified by the user, we properly encode the character.

```
>>> from xml.etree import cElementTree
>>> parser = cElementTree.XMLParser(target=cElementTree.TreeBuilder(), encoding='utf-8')
>>> parser.feed(b'<Node>KeyWithCarriageReturn\r.txt</Node>')
>>> tree = parser.close()
>>> tree
<Element 'Node' at 0x10157c390>
>>> tree.text
'KeyWithCarriageReturn\n.txt'
```

I'll need to do some investigation with our XML parser to see what options we have.

One potential option we have is to use the [encoding-type](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html) parameter to force the key name to be urlencoded.

I'll need to investigate what, if anything, this does to performance as we'd now have to urldecode every key name, but I can confirm that it works:

```
$ aws s3api list-objects --bucket bucket-name --prefix newline --encoding url --query 'Contents[].Key'
[
    "newline/foo%0D.txt"
]
```


Resolving patch:
diff --git a/awscli/customizations/s3/filegenerator.py b/awscli/customizations/s3/filegenerator.py
--- a/awscli/customizations/s3/filegenerator.py
+++ b/awscli/customizations/s3/filegenerator.py
@@ -12,7 +12,6 @@
 # language governing permissions and limitations under the License.
 import os
 import sys
-import datetime
 
 import six
 from dateutil.parser import parse
@@ -20,6 +19,7 @@
 
 from awscli.customizations.s3.fileinfo import FileInfo
 from awscli.customizations.s3.utils import find_bucket_key, get_file_stat
+from awscli.customizations.s3.utils import BucketLister
 from awscli.errorhandler import ClientError
 
 
@@ -162,28 +162,22 @@ def list_objects(self, s3_path, dir_op):
             yield self._list_single_object(s3_path)
         else:
             operation = self._service.get_operation('ListObjects')
-            iterator = operation.paginate(self._endpoint, bucket=bucket,
-                                          prefix=prefix)
-            for html_response, response_data in iterator:
-                contents = response_data['Contents']
-                for content in contents:
-                    src_path = bucket + '/' + content['Key']
-                    size = content['Size']
-                    last_update = parse(content['LastModified'])
-                    last_update = last_update.astimezone(tzlocal())
-                    if size == 0 and src_path.endswith('/'):
-                        if self.operation_name == 'delete':
-                            # This is to filter out manually created folders
-                            # in S3.  They have a size zero and would be
-                            # undesirably downloaded.  Local directories
-                            # are automatically created when they do not
-                            # exist locally.  But user should be able to
-                            # delete them.
-                            yield src_path, size, last_update
-                    elif not dir_op and s3_path != src_path:
-                        pass
-                    else:
-                        yield src_path, size, last_update
+            lister = BucketLister(operation, self._endpoint)
+            for key in lister.list_objects(bucket=bucket, prefix=prefix):
+                source_path, size, last_update = key
+                if size == 0 and source_path.endswith('/'):
+                    if self.operation_name == 'delete':
+                        # This is to filter out manually created folders
+                        # in S3.  They have a size zero and would be
+                        # undesirably downloaded.  Local directories
+                        # are automatically created when they do not
+                        # exist locally.  But user should be able to
+                        # delete them.
+                        yield source_path, size, last_update
+                elif not dir_op and s3_path != source_path:
+                    pass
+                else:
+                    yield source_path, size, last_update
 
     def _list_single_object(self, s3_path):
         # When we know we're dealing with a single object, we can avoid
diff --git a/awscli/customizations/s3/utils.py b/awscli/customizations/s3/utils.py
--- a/awscli/customizations/s3/utils.py
+++ b/awscli/customizations/s3/utils.py
@@ -21,7 +21,9 @@
 
 from six import PY3
 from six.moves import queue
+from dateutil.parser import parse
 from dateutil.tz import tzlocal
+from botocore.compat import unquote_str
 
 from awscli.customizations.s3.constants import MAX_PARTS
 from awscli.customizations.s3.constants import MAX_SINGLE_UPLOAD_SIZE
@@ -285,6 +287,31 @@ def __iter__(self):
         return iter([])
 
 
+def _date_parser(date_string):
+    return parse(date_string).astimezone(tzlocal())
+
+
+class BucketLister(object):
+    """List keys in a bucket."""
+    def __init__(self, operation, endpoint, date_parser=_date_parser):
+        self._operation = operation
+        self._endpoint = endpoint
+        self._date_parser = date_parser
+
+    def list_objects(self, bucket, prefix=None):
+        kwargs = {'bucket': bucket, 'encoding_type': 'url'}
+        if prefix is not None:
+            kwargs['prefix'] = prefix
+        pages = self._operation.paginate(self._endpoint, **kwargs)
+        for response, page in pages:
+            contents = page['Contents']
+            for content in contents:
+                source_path = bucket + '/' + unquote_str(content['Key'])
+                size = content['Size']
+                last_update = self._date_parser(content['LastModified'])
+                yield source_path, size, last_update
+
+
 IORequest = namedtuple('IORequest', ['filename', 'offset', 'data'])
 # Used to signal that IO for the filename is finished, and that
 # any associated resources may be cleaned up.


Test patch:
diff --git a/tests/integration/customizations/s3/test_plugin.py b/tests/integration/customizations/s3/test_plugin.py
--- a/tests/integration/customizations/s3/test_plugin.py
+++ b/tests/integration/customizations/s3/test_plugin.py
@@ -241,6 +241,25 @@ def test_mv_to_nonexistent_bucket(self):
         self.assertEqual(p.rc, 1)
 
 
+class TestRm(BaseS3CLICommand):
+    def test_rm_with_newlines(self):
+        bucket_name = self.create_bucket()
+
+        # Note the carriage return in the key name.
+        foo_txt = self.files.create_file('foo\r.txt', 'this is foo.txt')
+        p = aws('s3 cp %s s3://%s/foo\r.txt' % (foo_txt, bucket_name))
+        self.assert_no_errors(p)
+
+        # Make sure object is in bucket.
+        self.assertTrue(self.key_exists(bucket_name, key_name='foo\r.txt'))
+
+        # Then delete the file.
+        p = aws('s3 rm s3://%s/ --recursive' % (bucket_name,))
+
+        # And verify it's gone.
+        self.assertFalse(self.key_exists(bucket_name, key_name='foo\r.txt'))
+
+
 class TestCp(BaseS3CLICommand):
 
     def test_cp_to_and_from_s3(self):
diff --git a/tests/unit/customizations/s3/test_utils.py b/tests/unit/customizations/s3/test_utils.py
--- a/tests/unit/customizations/s3/test_utils.py
+++ b/tests/unit/customizations/s3/test_utils.py
@@ -4,13 +4,13 @@
 import shutil
 import ntpath
 
-from six.moves import queue
 import mock
 
 from awscli.customizations.s3.utils import find_bucket_key, find_chunksize
 from awscli.customizations.s3.utils import ReadFileChunk
 from awscli.customizations.s3.utils import relative_path
 from awscli.customizations.s3.utils import StablePriorityQueue
+from awscli.customizations.s3.utils import BucketLister
 from awscli.customizations.s3.constants import MAX_SINGLE_UPLOAD_SIZE
 
 
@@ -193,5 +193,58 @@ def test_priority_attr_is_missing(self):
         self.assertIs(q.get(), a)
 
 
+class TestBucketList(unittest.TestCase):
+    def setUp(self):
+        self.operation = mock.Mock()
+        self.endpoint = mock.sentinel.endpoint
+        self.date_parser = mock.Mock()
+        self.date_parser.return_value = mock.sentinel.now
+
+    def test_list_objects(self):
+        now = mock.sentinel.now
+        self.operation.paginate.return_value = [
+            (None, {'Contents': [
+                {'LastModified': '2014-02-27T04:20:38.000Z',
+                 'Key': 'a', 'Size': 1},
+                {'LastModified': '2014-02-27T04:20:38.000Z',
+                 'Key': 'b', 'Size': 2},]}),
+            (None, {'Contents': [
+                {'LastModified': '2014-02-27T04:20:38.000Z',
+                 'Key': 'c', 'Size': 3},
+            ]}),
+        ]
+        lister = BucketLister(self.operation, self.endpoint, self.date_parser)
+        objects = list(lister.list_objects(bucket='foo'))
+        self.assertEqual(objects, [('foo/a', 1, now), ('foo/b', 2, now),
+                                   ('foo/c', 3, now)])
+
+    def test_urlencoded_keys(self):
+        # In order to workaround control chars being in key names,
+        # we force the urlencoding of the key names and we decode
+        # them before yielding them.  For example, note the %0D
+        # in foo.txt:
+        now = mock.sentinel.now
+        self.operation.paginate.return_value = [
+            (None, {'Contents': [
+                {'LastModified': '2014-02-27T04:20:38.000Z',
+                 'Key': 'bar%0D.txt', 'Size': 1}]}),
+        ]
+        lister = BucketLister(self.operation, self.endpoint, self.date_parser)
+        objects = list(lister.list_objects(bucket='foo'))
+        # And note how it's been converted to '\r'.
+        self.assertEqual(objects, [('foo/bar\r.txt', 1, now)])
+
+    def test_urlencoded_with_unicode_keys(self):
+        now = mock.sentinel.now
+        self.operation.paginate.return_value = [
+            (None, {'Contents': [
+                {'LastModified': '2014-02-27T04:20:38.000Z',
+                 'Key': '%E2%9C%93', 'Size': 1}]}),
+        ]
+        lister = BucketLister(self.operation, self.endpoint, self.date_parser)
+        objects = list(lister.list_objects(bucket='foo'))
+        # And note how it's been converted to '\r'.
+        self.assertEqual(objects, [(u'foo/\u2713', 1, now)])
+
 if __name__ == "__main__":
     unittest.main()