Repository: aws/aws-cli

Issue:
s3 sync from local to s3 bucket re-uploads files with version 1.3.6
It is probably related to #718 but it is still happening on my OSX Mavericks with Python 2.7. I sync local files to S3 using "aws s3 sync" but some files always want to re-upload. I did a debug and comparator thinks that "file does not exist at destination".

I even removed all scripts and python packages for awscli, botocore and jmespath, and then reinstalled latest 1.3.6 but no luck. I use awscli for s3 sync since version 1.0 without any issues and have only started to experiences such issues starting with 1.3.2.

Appreciate if you could look into it. Let me know if additional information was needed.



Discussion:
Can you share the filename?  If it's related to 718 it would have whitespace in the filename.

There certainly are whitespaces in full path (not necessarily in filename) as below. Should it not have been fixed with #718?

```
2014-04-09 09:30:06,383 - awscli.customizations.s3.comparator - DEBUG - syncing: /Users/OSXUser/Documents/Work/MEGA SHOTS/EmptyRoad/Hiring/Jeff So/iPhone Apps/MyFirstApp/build/Debug-iphonesimulator/Animation.app/ChooseStyle.nib -> megabucket/Documents/Work/MEGA SHOTS/EmptyRoad/Hiring/Jeff So/iPhone Apps/MyFirstApp/build/Debug-iphonesimulator/Animation.app/ChooseStyle.nib, file does not exist at destination
```

Yep.  Seems to be working, here's what I tried:

```
$ aws --version
aws-cli/1.3.6 Python/2.7.5 Darwin/12.5.0
$ mkdir /tmp/synctest && cd /tmp/synctest
$ echo "test contents" > "file with spaces.txt"
$ aws s3 sync . s3://bucket-name/testwithspaces/
upload: ./file with spaces.txt to s3://bucket-name/testwithspaces/file with spaces.txt
$ aws s3 sync . s3://bucket-name/testwithspaces/   # <--- note did not resync the file.
$
```

Perhaps for some reason after determining that the file needs to be synced the actual file transfer fails.  Can you confirm that the sync commands exits cleanly (RC of 0) and that the file in question is actually uploaded to S3 successfully?

Your test works for me too, but there is something else going on with the files I have.

Files are indeed re-uploaded every time successfully. In fact my local files and S3 files (50,000+) are in sync except for 13 files, and 1.3.6 want to upload 1200+ files every time.

I just did some tests with same set of local files and S3 bucket and:

| awscli Version | Behaviour |
| :-: | :-- |
| 1.2.13 | Works as expected |
| 1.3.1 | Works as expected |
| 1.3.2. | Has the behaviour described in #718. Wants to upload half of my files with spaces |
| 1.3.6 | Doesn't have same behaviour as #718 but it still want to upload 1200+ every time. That is even if let the upload complete for all files, or even delete files in S3 and re-sync. |

Please let me know if you need anything from me to identify this issue. I've downgraded to 1.3.1 (last working version) for now.

This is similar to issue #648 regarding `awscli.customizations.s3.comparator` yielding a `file does not exist at destination` error.

In that issue files move the opposite direction (S3 to local disk) but there are no spaces in the file names (we replace them with underscores).

I was just looking into this a little bit. Seems to be that the two lists being analyzed in the comparator get out of sync at some point. From what I can tell, it's going along just fine and then suddenly a retry is needed. Then there's a big chunk of xml showing another set of keys. I think it's overwriting dest_files with this new list instead of appending.

I added a simple print of src_file.compare_key and dest_file.compare_key.

Before retry:
2014-04-14 17:18:04,966 - awscli.customizations.s3.comparator - DEBUG - src_file = My Pictures/October/IMG_0308.JPG, dest_file = My Pictures/October/IMG_0308.JPG
2014-04-14 17:18:04,967 - awscli.customizations.s3.comparator - DEBUG - src_file = My Pictures/October/IMG_0314.JPG, dest_file = My Pictures/October/IMG_0314.JPG
2014-04-14 17:18:04,967 - awscli.customizations.s3.comparator - DEBUG - src_file = My Pictures/October/IMG_0315.JPG, dest_file = My Pictures/October/IMG_0315.JPG
2014-04-14 17:18:04,968 - awscli.customizations.s3.comparator - DEBUG - src_file = My Pictures/October/IMG_0319.JPG, dest_file = My Pictures/October/IMG_0319.JPG
2014-04-14 17:18:04,969 - awscli.customizations.s3.comparator - DEBUG - src_file = My Pictures/October/IMG_0320.JPG, dest_file = My Pictures/October/IMG_0320.JPG

...

Then the retry.

Then:

2014-04-14 17:18:05,902 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x1e7bc08>
2014-04-14 17:18:05,902 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2014-04-14 17:18:05,902 - awscli.customizations.s3.executor - DEBUG - Submitting task: <awscli.customizations.s3.tasks.BasicTask object at 0x29c3510>
2014-04-14 17:18:05,902 - awscli.customizations.s3.comparator - DEBUG - src_file = My Pictures/October/IMG_0323.JPG, dest_file = Photos/2009/11/24/IMG_1336-1.CR2
2014-04-14 17:18:05,903 - awscli.customizations.s3.executor - DEBUG - Worker thread invoking task: <awscli.customizations.s3.tasks.BasicTask object at 0x29c3510>
2014-04-14 17:18:05,903 - awscli.customizations.s3.comparator - DEBUG - syncing: /home/jon/Pictures/My Pictures/October/IMG_0323.JPG -> 8342d92d-3b90-42e2-ac14-4877ed8aaee7/Pictures/My Pictures/October/IMG_0323.JPG, less_than, file does not exist at destination
2014-04-14 17:18:05,905 - botocore.service - DEBUG - Creating service object for: s3
2014-04-14 17:18:05,905 - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler <function signature_overrides at 0x1e7bde8>
2014-04-14 17:18:05,904 - awscli.customizations.s3.executor - DEBUG - Received print task: {'message': u'(dryrun) upload: ./IMG_0322.JPG to s3://8342d92d-3b90-42e2-ac14-4877ed8aaee7/Pictures/My Pictures/October/IMG_0322.JPG', 'error': False}
2014-04-14 17:18:05,906 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x1e7bc08>
2014-04-14 17:18:05,906 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
yyyyyyyyyyyy My Pictures/October/IMG_0323.JPG
(dryrun) upload: ./IMG_0322.JPG to s3://8342d92d-3b90-42e2-ac14-4877ed8aaee7/Pictures/My Pictures/October/IMG_0322.JPG
Completed 2 part(s) with ... file(s) remaining^M2014-04-14 17:18:05,907 - awscli.customizations.s3.executor - DEBUG - Submitting task: <awscli.customizations.s3.tasks.BasicTask object at 0x29c3790>
2014-04-14 17:18:05,907 - awscli.customizations.s3.comparator - DEBUG - src_file = My Pictures/October/IMG_0324.JPG, dest_file = Photos/2009/11/24/IMG_1336-1.CR2
2014-04-14 17:18:05,908 - awscli.customizations.s3.executor - DEBUG - Worker thread invoking task: <awscli.customizations.s3.tasks.BasicTask object at 0x29c3790>
2014-04-14 17:18:05,908 - awscli.customizations.s3.comparator - DEBUG - syncing: /home/jon/Pictures/My Pictures/October/IMG_0324.JPG -> 8342d92d-3b90-42e2-ac14-4877ed8aaee7/Pictures/My Pictures/October/IMG_0324.JPG, less_than, file does not exist at destination

Again, the dest file that is not working is the first file returned in the new chunk of xml.

Hope that helps..

Hi,

I don't really know how to use github, but I have a patch that fixes the issue. The problem is botocore/paginate.py is not unquoting the "next_marker" key. When it creates the query for the next request, the "+" in the key gets encoded as %2B.

```
diff --git a/botocore/paginate.py b/botocore/paginate.py
index c6efff6..95fedb4 100644
--- a/botocore/paginate.py
+++ b/botocore/paginate.py
@@ -11,6 +11,7 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.

+from compat import unquote_str
 from itertools import tee
 try:
     from itertools import zip_longest
@@ -251,7 +252,7 @@ class PageIterator(object):
                 return [None]
         next_tokens = []
         for token in self._output_token:
-            next_tokens.append(token.search(parsed))
+            next_tokens.append(unquote_str(token.search(parsed)))
         return next_tokens

     def result_key_iters(self):
```

Awesome, thanks for your help.  Looking at this now...

Just wanted to give an update here.  I'm able to repro the issue and can confirm what @jonbrock has said.  I want to work on a set of test cases here to ensure we don't regress on this again.  I'll send a PR once I have this worked out.


Resolving patch:
diff --git a/awscli/customizations/s3/utils.py b/awscli/customizations/s3/utils.py
--- a/awscli/customizations/s3/utils.py
+++ b/awscli/customizations/s3/utils.py
@@ -302,14 +302,39 @@ def list_objects(self, bucket, prefix=None):
         kwargs = {'bucket': bucket, 'encoding_type': 'url'}
         if prefix is not None:
             kwargs['prefix'] = prefix
-        pages = self._operation.paginate(self._endpoint, **kwargs)
-        for response, page in pages:
-            contents = page['Contents']
-            for content in contents:
-                source_path = bucket + '/' + unquote_str(content['Key'])
-                size = content['Size']
-                last_update = self._date_parser(content['LastModified'])
-                yield source_path, size, last_update
+        # This event handler is needed because we use encoding_type url and
+        # we're paginating.  The pagination token is the last Key of the
+        # Contents list.  However, botocore does not know that the encoding
+        # type needs to be urldecoded.
+        with ScopedEventHandler(self._operation.session, 'after-call.s3.ListObjects',
+                                self._decode_keys):
+            pages = self._operation.paginate(self._endpoint, **kwargs)
+            for response, page in pages:
+                contents = page['Contents']
+                for content in contents:
+                    source_path = bucket + '/' + content['Key']
+                    size = content['Size']
+                    last_update = self._date_parser(content['LastModified'])
+                    yield source_path, size, last_update
+
+    def _decode_keys(self, parsed, **kwargs):
+        for content in parsed['Contents']:
+            content['Key'] = unquote_str(content['Key'])
+
+
+class ScopedEventHandler(object):
+    """Register an event callback for the duration of a scope."""
+
+    def __init__(self, session, event_name, handler):
+        self._session = session
+        self._event_name = event_name
+        self._handler = handler
+
+    def __enter__(self):
+        self._session.register(self._event_name, self._handler)
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self._session.unregister(self._event_name, self._handler)
 
 
 IORequest = namedtuple('IORequest', ['filename', 'offset', 'data'])


Test patch:
diff --git a/tests/integration/customizations/s3/test_plugin.py b/tests/integration/customizations/s3/test_plugin.py
--- a/tests/integration/customizations/s3/test_plugin.py
+++ b/tests/integration/customizations/s3/test_plugin.py
@@ -1,4 +1,3 @@
-
 # -*- coding: utf-8 -*-
 # Copyright 2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 #
@@ -25,6 +24,7 @@
 import signal
 
 import botocore.session
+import six
 
 from tests.integration import aws
 from tests.unit.customizations.s3 import create_bucket as _create_bucket
@@ -68,6 +68,17 @@ def extra_teardown(self):
         # Subclasses can use this to define extra teardown steps.
         pass
 
+    def assert_key_contents_equal(self, bucket, key, expected_contents):
+        if isinstance(expected_contents, six.StringIO):
+            expected_contents = expected_contents.getvalue()
+        actual_contents = self.get_key_contents(bucket, key)
+        # The contents can be huge so we try to give helpful error messages
+        # without necessarily printing the actual contents.
+        self.assertEqual(len(actual_contents), len(expected_contents))
+        if actual_contents != expected_contents:
+            self.fail("Contents for %s/%s do not match (but they "
+                      "have the same length)" % (bucket, key))
+
     def create_bucket(self):
         bucket_name = _create_bucket(self.session)
         self.addCleanup(self.delete_bucket, bucket_name)
@@ -147,8 +158,7 @@ def test_mv_local_to_s3(self):
         # When we move an object, the local file is gone:
         self.assertTrue(not os.path.exists(full_path))
         # And now resides in s3.
-        contents = self.get_key_contents(bucket_name, 'foo.txt')
-        self.assertEqual(contents, 'this is foo.txt')
+        self.assert_key_contents_equal(bucket_name, 'foo.txt', 'this is foo.txt')
 
     def test_mv_s3_to_local(self):
         bucket_name = self.create_bucket()
@@ -179,14 +189,13 @@ def test_mv_s3_to_s3(self):
     def test_mv_s3_to_s3_multipart(self):
         from_bucket = self.create_bucket()
         to_bucket = self.create_bucket()
-        file_contents = 'abcd' * (1024 * 1024 * 10)
+        file_contents = six.StringIO('abcd' * (1024 * 1024 * 10))
         self.put_object(from_bucket, 'foo.txt', file_contents)
 
         p = aws('s3 mv s3://%s/foo.txt s3://%s/foo.txt' % (from_bucket,
                                                            to_bucket))
         self.assert_no_errors(p)
-        contents = self.get_key_contents(to_bucket, 'foo.txt')
-        self.assertEqual(contents, file_contents)
+        self.assert_key_contents_equal(to_bucket, 'foo.txt', file_contents)
         # And verify that the object no longer exists in the from_bucket.
         self.assertTrue(not self.key_exists(from_bucket, key_name='foo.txt'))
 
@@ -194,7 +203,7 @@ def test_mv_s3_to_s3_multipart_recursive(self):
         from_bucket = self.create_bucket()
         to_bucket = self.create_bucket()
 
-        large_file_contents = 'abcd' * (1024 * 1024 * 10)
+        large_file_contents = six.StringIO('abcd' * (1024 * 1024 * 10))
         small_file_contents = 'small file contents'
         self.put_object(from_bucket, 'largefile', large_file_contents)
         self.put_object(from_bucket, 'smallfile', small_file_contents)
@@ -211,29 +220,28 @@ def test_mv_s3_to_s3_multipart_recursive(self):
         self.assertTrue(self.key_exists(to_bucket, key_name='smallfile'))
 
         # And the contents are what we expect.
-        self.assertEqual(self.get_key_contents(to_bucket, 'smallfile'),
-                         small_file_contents)
-        self.assertEqual(self.get_key_contents(to_bucket, 'largefile'),
-                         large_file_contents)
+        self.assert_key_contents_equal(to_bucket, 'smallfile',
+                                       small_file_contents)
+        self.assert_key_contents_equal(to_bucket, 'largefile',
+                                       large_file_contents)
 
     def test_mv_with_large_file(self):
         bucket_name = self.create_bucket()
         # 40MB will force a multipart upload.
-        file_contents = 'abcd' * (1024 * 1024 * 10)
-        foo_txt = self.files.create_file('foo.txt', file_contents)
+        file_contents = six.StringIO('abcd' * (1024 * 1024 * 10))
+        foo_txt = self.files.create_file('foo.txt', file_contents.getvalue())
         p = aws('s3 mv %s s3://%s/foo.txt' % (foo_txt, bucket_name))
         self.assert_no_errors(p)
         # When we move an object, the local file is gone:
         self.assertTrue(not os.path.exists(foo_txt))
         # And now resides in s3.
-        contents = self.get_key_contents(bucket_name, 'foo.txt')
-        self.assertEqual(len(contents), len(file_contents))
+        self.assert_key_contents_equal(bucket_name, 'foo.txt', file_contents)
 
         # Now verify we can download this file.
         p = aws('s3 mv s3://%s/foo.txt %s' % (bucket_name, foo_txt))
         self.assert_no_errors(p)
         self.assertTrue(os.path.exists(foo_txt))
-        self.assertEqual(os.path.getsize(foo_txt), len(file_contents))
+        self.assertEqual(os.path.getsize(foo_txt), len(file_contents.getvalue()))
 
     def test_mv_to_nonexistent_bucket(self):
         full_path = self.files.create_file('foo.txt', 'this is foo.txt')
@@ -315,16 +323,12 @@ def test_cp_without_trailing_slash(self):
     def test_cp_s3_s3_multipart(self):
         from_bucket = self.create_bucket()
         to_bucket = self.create_bucket()
-        file_contents = 'abcd' * (1024 * 1024 * 10)
+        file_contents = six.StringIO('abcd' * (1024 * 1024 * 10))
         self.put_object(from_bucket, 'foo.txt', file_contents)
 
         p = aws('s3 cp s3://%s/foo.txt s3://%s/foo.txt' % (from_bucket, to_bucket))
         self.assert_no_errors(p)
-        contents = self.get_key_contents(to_bucket, 'foo.txt')
-        # Don't use assertEqual() here, this will spit out a huge
-        # 20mb diff of 'abcd' chars.  Just let the user know we failed.
-        if contents != file_contents:
-            self.fail("Downlaoded contents of 10mb file are not the same.")
+        self.assert_key_contents_equal(to_bucket, 'foo.txt', file_contents)
         self.assertTrue(self.key_exists(from_bucket, key_name='foo.txt'))
 
     def test_guess_mime_type(self):
@@ -342,18 +346,19 @@ def test_guess_mime_type(self):
     def test_download_large_file(self):
         # This will force a multipart download.
         bucket_name = self.create_bucket()
-        foo_contents = 'abcd' * (1024 * 1024 * 10)
+        foo_contents = six.StringIO('abcd' * (1024 * 1024 * 10))
         self.put_object(bucket_name, key_name='foo.txt', contents=foo_contents)
         local_foo_txt = self.files.full_path('foo.txt')
         p = aws('s3 cp s3://%s/foo.txt %s' % (bucket_name, local_foo_txt))
         self.assert_no_errors(p)
-        self.assertEqual(os.path.getsize(local_foo_txt), len(foo_contents))
+        self.assertEqual(os.path.getsize(local_foo_txt),
+                         len(foo_contents.getvalue()))
 
     @unittest.skipIf(platform.system() not in ['Darwin', 'Linux'],
                     'SIGINT not supported on Windows.')
     def test_download_ctrl_c_does_not_hang(self):
         bucket_name = self.create_bucket()
-        foo_contents = 'abcd' * (1024 * 1024 * 20)
+        foo_contents = six.StringIO('abcd' * (1024 * 1024 * 20))
         self.put_object(bucket_name, key_name='foo.txt', contents=foo_contents)
         local_foo_txt = self.files.full_path('foo.txt')
         process = aws('s3 cp s3://%s/foo.txt %s' % (bucket_name, local_foo_txt), wait_for_finish=False)
@@ -398,6 +403,23 @@ def test_download_non_existent_key(self):
 
 
 class TestSync(BaseS3CLICommand):
+    def test_sync_with_plus_chars(self):
+        # 1. Create > 1000 files with '+' in the filename.
+        # 2. Sync up to s3.
+        # 3. Sync up to s3
+        # 4. Verify nothing was synced up down from s3 in step 3.
+        bucket_name = self.create_bucket()
+        filenames = []
+        for i in range(2000):
+            # Create a file with a space char and a '+' char in the filename.
+            filenames.append(self.files.create_file('foo +%06d' % i, contents=''))
+        p = aws('s3 sync %s s3://%s/' % (self.files.rootdir, bucket_name))
+        self.assert_no_errors(p)
+        time.sleep(1)
+        p2 = aws('s3 sync %s s3://%s/' % (self.files.rootdir, bucket_name))
+        self.assertNotIn('upload:', p2.stdout)
+        self.assertEqual('', p2.stdout)
+
     def test_sync_to_from_s3(self):
         bucket_name = self.create_bucket()
         foo_txt = self.files.create_file('foo.txt', 'foo contents')
diff --git a/tests/unit/customizations/s3/fake_session.py b/tests/unit/customizations/s3/fake_session.py
--- a/tests/unit/customizations/s3/fake_session.py
+++ b/tests/unit/customizations/s3/fake_session.py
@@ -70,6 +70,12 @@ def emit(self, *args, **kwargs):
     def emit_first_non_none_response(self, *args, **kwargs):
         pass
 
+    def register(self, name, handler):
+        pass
+
+    def unregister(self, name, handler):
+        pass
+
 
 class FakeService(object):
     """
diff --git a/tests/unit/customizations/s3/test_utils.py b/tests/unit/customizations/s3/test_utils.py
--- a/tests/unit/customizations/s3/test_utils.py
+++ b/tests/unit/customizations/s3/test_utils.py
@@ -6,11 +6,13 @@
 
 import mock
 
+from botocore.hooks import HierarchicalEmitter
 from awscli.customizations.s3.utils import find_bucket_key, find_chunksize
 from awscli.customizations.s3.utils import ReadFileChunk
 from awscli.customizations.s3.utils import relative_path
 from awscli.customizations.s3.utils import StablePriorityQueue
 from awscli.customizations.s3.utils import BucketLister
+from awscli.customizations.s3.utils import ScopedEventHandler
 from awscli.customizations.s3.constants import MAX_SINGLE_UPLOAD_SIZE
 
 
@@ -196,13 +198,23 @@ def test_priority_attr_is_missing(self):
 class TestBucketList(unittest.TestCase):
     def setUp(self):
         self.operation = mock.Mock()
+        self.emitter = HierarchicalEmitter()
+        self.operation.session.register = self.emitter.register
+        self.operation.session.unregister = self.emitter.unregister
         self.endpoint = mock.sentinel.endpoint
         self.date_parser = mock.Mock()
         self.date_parser.return_value = mock.sentinel.now
+        self.responses = []
+
+    def fake_paginate(self, *args, **kwargs):
+        for response in self.responses:
+            self.emitter.emit('after-call.s3.ListObjects', parsed=response[1])
+        return self.responses
 
     def test_list_objects(self):
         now = mock.sentinel.now
-        self.operation.paginate.return_value = [
+        self.operation.paginate = self.fake_paginate
+        self.responses = [
             (None, {'Contents': [
                 {'LastModified': '2014-02-27T04:20:38.000Z',
                  'Key': 'a', 'Size': 1},
@@ -224,7 +236,8 @@ def test_urlencoded_keys(self):
         # them before yielding them.  For example, note the %0D
         # in bar.txt:
         now = mock.sentinel.now
-        self.operation.paginate.return_value = [
+        self.operation.paginate = self.fake_paginate
+        self.responses = [
             (None, {'Contents': [
                 {'LastModified': '2014-02-27T04:20:38.000Z',
                  'Key': 'bar%0D.txt', 'Size': 1}]}),
@@ -236,7 +249,8 @@ def test_urlencoded_keys(self):
 
     def test_urlencoded_with_unicode_keys(self):
         now = mock.sentinel.now
-        self.operation.paginate.return_value = [
+        self.operation.paginate = self.fake_paginate
+        self.responses = [
             (None, {'Contents': [
                 {'LastModified': '2014-02-27T04:20:38.000Z',
                  'Key': '%E2%9C%93', 'Size': 1}]}),
@@ -246,5 +260,15 @@ def test_urlencoded_with_unicode_keys(self):
         # And note how it's been converted to '\r'.
         self.assertEqual(objects, [(u'foo/\u2713', 1, now)])
 
+
+class TestScopedEventHandler(unittest.TestCase):
+    def test_scoped_session_handler(self):
+        session = mock.Mock()
+        scoped = ScopedEventHandler(session, 'eventname', 'handler')
+        with scoped:
+            session.register.assert_called_with('eventname', 'handler')
+        session.unregister.assert_called_with('eventname', 'handler')
+
+
 if __name__ == "__main__":
     unittest.main()