Repository: boto/boto

Issue:
Glacier uploads are broken on python2
Hi, if I try and upload a _large binary_ file to glacier (ie one which needs multipart) with boto, it fails with a UnicodeDecodeError. It works fine with small files (ie non-multiparters).

I've figured out why, but first here is my code and the error message:

My code:

```
glacierConn = boto.glacier.connect_to_region("eu-west-1",
                                             aws_access_key_id=ACCESS,
                                             aws_secret_access_key=SECRET)
vault = glacierConn.get_vault("myvault")
archiveId = vault.upload_archive("/tmp/test", description="test")
```

Error message:

```
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/adq/dev/tvsquared-backend/tvsquared/tmp/adq/gtest.py", line 11, in <module>
    archiveId = vault.upload_archive("/tmp/test", description="test")
  File "/usr/lib/python2.7/site-packages/boto/glacier/vault.py", line 87, in upload_archive
    return self.create_archive_from_file(filename, description=description)
  File "/usr/lib/python2.7/site-packages/boto/glacier/vault.py", line 177, in create_archive_from_file
    writer.write(data)
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 219, in write
    self.partitioner.write(data)
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 61, in write
    self._send_part()
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 75, in _send_part
    self.send_fn(part)
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 222, in _upload_part
    self.uploader.upload_part(self.next_part_index, part_data)
  File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 129, in upload_part
    content_range, part_data)
  File "/usr/lib/python2.7/site-packages/boto/glacier/layer1.py", line 1279, in upload_part
    response_headers=response_headers)
  File "/usr/lib/python2.7/site-packages/boto/glacier/layer1.py", line 114, in make_request
    data=data)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1053, in make_request
    retry_handler=retry_handler)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 926, in _mexe
    request.body, request.headers)
  File "/usr/lib/python2.7/httplib.py", line 995, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1029, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 991, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 842, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 86: ordinal not in range(128)
```

I am passing non-unicode strs into boto. The problem is caused during the get_vault() call: it connects to the API, and retrieves the vault details, including the vault name, as unicode strings.

When I later call upload_archive(), httplib ends up being passed the vault name as a unicode string, which causes it to try and interpret my binary file as a unicode string (which fails, surprise!)

As a quick bodge, if I forcibly convert the vault name back to unicode, it works fine again:

```
glacierConn = boto.glacier.connect_to_region("eu-west-1",
                                             aws_access_key_id=ACCESS,
                                             aws_secret_access_key=SECRET)
vault = glacierConn.get_vault("myvault")
vault.name = str(vault.name)
archiveId = vault.upload_archive("/tmp/test", description="test")
```

This problem started a month or so ago, I suspect some of the python3 changes have broken it.

I'm using boto 2.32.1, python 2.7.8 on Arch Linux (x64).



Discussion:
Works fine with 2.31.1. Most likely commit a4c9a781f47a61ddde2b3a3802b93c1ed29cdf16 busted this?

Hi, just checked with 2.33.0 -- seems to be working fine without my bodge now.

Seeing this on 2.33.0. Ubuntu 14.04, Python 2.7.6, installed boto using pip.

This is still happening for me as well with 2.33.0.

On Wed, Oct 15, 2014 at 10:49 AM, Janez Urevc notifications@github.com
wrote:

> Seeing this on 2.33.0. Ubuntu 14.04, installed using pip.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/boto/boto/issues/2603#issuecomment-59217304.


Resolving patch:
diff --git a/boto/connection.py b/boto/connection.py
--- a/boto/connection.py
+++ b/boto/connection.py
@@ -368,7 +368,7 @@ def authorize(self, connection, **kwargs):
             for key in self.headers:
                 val = self.headers[key]
                 if isinstance(val, six.text_type):
-                    safe = '!"#$%&\'()*+,/:;<=>?@[\\]^`{|}~'
+                    safe = '!"#$%&\'()*+,/:;<=>?@[\\]^`{|}~ '
                     self.headers[key] = quote(val.encode('utf-8'), safe)
             setattr(self, '_headers_quoted', True)
 
diff --git a/boto/glacier/layer1.py b/boto/glacier/layer1.py
--- a/boto/glacier/layer1.py
+++ b/boto/glacier/layer1.py
@@ -1273,7 +1273,7 @@ def upload_part(self, vault_name, upload_id, linear_hash,
                    'x-amz-sha256-tree-hash': tree_hash,
                    'Content-Range': 'bytes %d-%d/*' % byte_range}
         response_headers = [('x-amz-sha256-tree-hash', u'TreeHash')]
-        uri = 'vaults/%s/multipart-uploads/%s' % (vault_name, upload_id)
+        uri = 'vaults/%s/multipart-uploads/%s' % (str(vault_name), upload_id)
         return self.make_request('PUT', uri, headers=headers,
                                  data=part_data, ok_responses=(204,),
                                  response_headers=response_headers)


Test patch:
diff --git a/tests/unit/glacier/test_layer1.py b/tests/unit/glacier/test_layer1.py
--- a/tests/unit/glacier/test_layer1.py
+++ b/tests/unit/glacier/test_layer1.py
@@ -2,8 +2,10 @@
 import copy
 import tempfile
 
+from tests.unit import unittest
 from tests.unit import AWSMockServiceTestCase
 from boto.glacier.layer1 import Layer1
+from boto.compat import six
 
 
 class GlacierLayer1ConnectionBase(AWSMockServiceTestCase):
@@ -80,6 +82,33 @@ def test_get_archive_output(self):
         self.assertEqual(self.job_content, response.read())
 
 
+class TestGlacierUploadPart(GlacierLayer1ConnectionBase):
+    def test_upload_part_content_range_header(self):
+        fake_data = b'\xe2'
+        self.set_http_response(status_code=204)
+        self.service_connection.upload_part(
+            u'unicode_vault_name', 'upload_id', 'linear_hash', 'tree_hash',
+            (1,2), fake_data)
+        self.assertEqual(
+            self.actual_request.headers['Content-Range'], 'bytes 1-2/*')
+
+    def test_upload_part_with_unicode_name(self):
+        fake_data = b'\xe2'
+        self.set_http_response(status_code=204)
+        self.service_connection.upload_part(
+            u'unicode_vault_name', 'upload_id', 'linear_hash', 'tree_hash',
+            (1,2), fake_data)
+        self.assertEqual(
+            self.actual_request.path,
+            '/-/vaults/unicode_vault_name/multipart-uploads/upload_id')
+        # If the path is unicode in python2, it triggers the following bug
+        # noted in this PR: https://github.com/boto/boto/pull/2697
+        # httplib notices that if the path is unicode, it will try to encode
+        # body which may be impossible if there is the data is binary.
+        self.assertIsInstance(self.actual_request.body, six.binary_type)
+        self.assertEqual(self.actual_request.body, fake_data)
+
+
 class GlacierUploadArchiveResets(GlacierLayer1ConnectionBase):
     def test_upload_archive(self):
         fake_data = tempfile.NamedTemporaryFile()
diff --git a/tests/unit/test_connection.py b/tests/unit/test_connection.py
--- a/tests/unit/test_connection.py
+++ b/tests/unit/test_connection.py
@@ -522,7 +522,7 @@ def test_get_status_error(self):
 
 class TestHTTPRequest(unittest.TestCase):
     def test_user_agent_not_url_encoded(self):
-        headers = {'Some-Header': u'should be url encoded',
+        headers = {'Some-Header': u'should be encoded \u2713',
                    'User-Agent': UserAgent}
         request = HTTPRequest('PUT', 'https', 'amazon.com', 443, None,
                               None, {}, headers, 'Body')
@@ -539,7 +539,7 @@ def mock_add_auth(req, **kwargs):
         # Ensure the headers at authorization are as expected i.e.
         # the user agent header was not url encoded but the other header was.
         self.assertEqual(mock_connection.headers_at_auth,
-                         {'Some-Header': 'should%20be%20url%20encoded',
+                         {'Some-Header': 'should be encoded %E2%9C%93',
                           'User-Agent': UserAgent})
 
     def test_content_length_str(self):