Repository: boto/boto

Issue:
URL encoding of headers not decoded by S3
Hi,

In boto/connection.py, the following snippet URL encodes headers, but the problem is S3 doesn't **decode**, resulting in encoded headers being stored in S3 and sent to browsers:

<pre>
Cache-control: public%2C max-age=xxx
</pre>


``` python
    def authorize(self, connection, **kwargs):
        for key in self.headers:
            val = self.headers[key]
            if isinstance(val, unicode):
                self.headers[key] = urllib.quote_plus(val.encode('utf-8'))

        connection._auth_handler.add_auth(self, **kwargs)
```



Discussion:
It looks like #1829 is a fix for this, but it is causing a couple of tests to fail. If anybody has time to look into how to fix those then it can be merged in.

Firstly, why do we even need non-ASCII chars in HTTP headers? 

If non-ASCII chars are really needed, no matter what kind of fanciful encoding/escaping method you use in boto, S3 still **has** to decode them. So I guess the fundamental problem is S3 doesn't decode HTTP headers...

Also, I would think quote_plus is a wrong method to use in this case... We should only escape non-ASCII chars into %xx.  Chars such as commas, dots, semi-colons are allowed in HTTP header values... and do NOT require urlencoding...

If you require a use case like:

<pre>
Content-disposition: filename="{some non-English lang}"
</pre>

Then I suppose the user of boto should have the initiative to encode the filename first before passing it to boto ...


Resolving patch:
diff --git a/boto/connection.py b/boto/connection.py
--- a/boto/connection.py
+++ b/boto/connection.py
@@ -372,7 +372,8 @@ def authorize(self, connection, **kwargs):
         for key in self.headers:
             val = self.headers[key]
             if isinstance(val, unicode):
-                self.headers[key] = urllib.quote_plus(val.encode('utf-8'))
+                safe = '!"#$%&\'()*+,/:;<=>?@[\\]^`{|}~'
+                self.headers[key] = urllib.quote_plus(val.encode('utf-8'), safe)
 
         connection._auth_handler.add_auth(self, **kwargs)
 


Test patch:
diff --git a/tests/integration/s3/test_key.py b/tests/integration/s3/test_key.py
--- a/tests/integration/s3/test_key.py
+++ b/tests/integration/s3/test_key.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 # Copyright (c) 2012 Mitch Garnaat http://garnaat.org/
 # All rights reserved.
 #
@@ -27,6 +28,7 @@
 from tests.unit import unittest
 import time
 import StringIO
+import urllib
 from boto.s3.connection import S3Connection
 from boto.s3.key import Key
 from boto.exception import S3ResponseError
@@ -394,3 +396,19 @@ def test_header_casing(self):
 
         check = self.bucket.get_key('test_header_case')
         self.assertEqual(check.content_type, 'application/json')
+
+    def test_header_encoding(self):
+        key = self.bucket.new_key('test_header_encoding')
+
+        key.set_metadata('Cache-control', 'public, max-age=500')
+        key.set_metadata('Content-disposition', u'filename=Schöne Zeit.txt')
+        key.set_contents_from_string('foo')
+
+        check = self.bucket.get_key('test_header_encoding')
+
+        self.assertEqual(check.cache_control, 'public, max-age=500')
+        self.assertEqual(check.content_disposition, 'filename=Sch%C3%B6ne+Zeit.txt')
+        self.assertEqual(
+            urllib.unquote_plus(check.content_disposition).decode('utf-8'),
+            'filename=Schöne Zeit.txt'.decode('utf-8')
+        )