Repository: falconry/falcon

Issue:
resp.location double-encodes urlencoded strings
I've got a URL which contains a url-encoded URL as a parameter, and when setting resp.location, that results in it being double-encoded:

resp.location = "http://something?redirect_uri=http%3A%2F%2Fsite"

This should result in:
Location: http://something?redirect_uri=http%3A%2F%2Fsite...

But what actually happens is:
Location: http://something?redirect_uri=http%253A%252F%252Fsite

I worked around by raising a HTTPStatus() object, but this doesn't seem like the ideal situation.



Discussion:
Hi @alertedsnake, thanks for the report. I agree that `resp.location` should be smarter about this. Work on Falcon 0.4 is wrapping up, but I've scheduled this bug fix for the following release.

Same issue  here.
Please try to fix it as soon as possible because we have planned to use it in production.

Till then, any solution to fix this issue is welcomed. 
Thank you.

Possibly related: #782 

Hi, @alertedsnake, @pressyfdo:

Thanks for reporting this issue! 

I've been taking a fairly deep look at this and discussing with @kgriffs and others. I'm curious if you could fill me in with more details with the specific issue you are having. Specifically, I'm trying to understand why you might have an encoded URL from within your application (if the URL is coming in through Falcon in the usual way(s) it should be automatically decoded). If you know that the URL was previously encoded, why not just decode it using urllib before assigning to `resp.location`? 

The crux of the problem:
- Any value in the Location header (and others) needs to be urlencoded. 
- There is no good way to determine if URI is already encoded or not (any suggestions here appreciated). There are some heuristics that can be used to achieve a fairly high level of certainty, but there are no deterministic approaches that will work 100% of the time. 
- The typical approach for dealing with url encoding is to be explicit about when urls are encoded or not and perform encoding/decoding as necessary.

The URL I am assigning to resp.location is not encoded - it contains URL encoded parameters. This data is not coming in from something in Falcon that should have bit didn't decode it, it is something application specific that we are generating. As was stated in my example, it is a url which has a url-encoded url as a parameter. 

Your suggestion is that I decode this data before setting resp.location? The issue I see with this is that I am passing what is a perfectly valid URL to an argument called "url" and Falcon is altering it in an unexpected way. I can see no reason for Falcon to alter this data in any way. 

At the very least, the documentation should be very explicit here if the behavior is to remain as it is now. 

Does that help? 

Now that I look too, this is definitely the same issue as with #782 and was also broken by the solution to #135. 

Ugh, cop out.

Hi @alertedsnake, we are actually still looking at this. The issue was closed prematurely. 


Resolving patch:
diff --git a/falcon/util/uri.py b/falcon/util/uri.py
--- a/falcon/util/uri.py
+++ b/falcon/util/uri.py
@@ -60,6 +60,7 @@ def _create_char_encoder(allowed_chars):
 def _create_str_encoder(is_value):
 
     allowed_chars = _UNRESERVED if is_value else _ALL_ALLOWED
+    allowed_chars_plus_percent = allowed_chars + '%'
     encode_char = _create_char_encoder(allowed_chars)
 
     def encoder(uri):
@@ -67,10 +68,32 @@ def encoder(uri):
         if not uri.rstrip(allowed_chars):
             return uri
 
+        if not uri.rstrip(allowed_chars_plus_percent):
+            # NOTE(kgriffs): There's a good chance the string has already
+            # been escaped. Do one more check to increase our certainty.
+            tokens = uri.split('%')
+            for token in tokens[1:]:
+                hex_octet = token[:2]
+
+                if not len(hex_octet) == 2:
+                    break
+
+                if not (hex_octet[0] in _HEX_DIGITS and
+                        hex_octet[1] in _HEX_DIGITS):
+                    break
+            else:
+                # NOTE(kgriffs): All percent-encoded sequences were
+                # valid, so assume that the string has already been
+                # encoded.
+                return uri
+
+            # NOTE(kgriffs): At this point we know there is at least
+            # one unallowed percent character. We are going to assume
+            # that everything should be encoded. If the string is
+            # partially encoded, the caller will need to normalize it
+            # before passing it in here.
+
         # Convert to a byte array if it is not one already
-        #
-        # NOTE(kgriffs): Code coverage disabled since in Py3K the uri
-        # is always a text type, so we get a failure for that tox env.
         if isinstance(uri, six.text_type):
             uri = uri.encode('utf-8')
 


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -177,6 +177,35 @@ def test_uri_encode(self):
                     '?limit=3&e%C3%A7ho=true')
         self.assertEqual(uri.encode(url), expected)
 
+    def test_uri_encode_double(self):
+        url = 'http://example.com/v1/fiz bit/messages'
+        expected = 'http://example.com/v1/fiz%20bit/messages'
+        self.assertEqual(uri.encode(uri.encode(url)), expected)
+
+        url = u'http://example.com/v1/fizbit/messages?limit=3&e\u00e7ho=true'
+        expected = ('http://example.com/v1/fizbit/messages'
+                    '?limit=3&e%C3%A7ho=true')
+        self.assertEqual(uri.encode(uri.encode(url)), expected)
+
+        url = 'http://example.com/v1/fiz%bit/mess%ages/%'
+        expected = 'http://example.com/v1/fiz%25bit/mess%25ages/%25'
+        self.assertEqual(uri.encode(uri.encode(url)), expected)
+
+        url = 'http://example.com/%%'
+        expected = 'http://example.com/%25%25'
+        self.assertEqual(uri.encode(uri.encode(url)), expected)
+
+        # NOTE(kgriffs): Specific example cited in GH issue
+        url = 'http://something?redirect_uri=http%3A%2F%2Fsite'
+        self.assertEqual(uri.encode(url), url)
+
+        hex_digits = 'abcdefABCDEF0123456789'
+        for c1 in hex_digits:
+            for c2 in hex_digits:
+                url = 'http://example.com/%' + c1 + c2
+                encoded = uri.encode(uri.encode(url))
+                self.assertEqual(encoded, url)
+
     def test_uri_encode_value(self):
         self.assertEqual(uri.encode_value('abcd'), 'abcd')
         self.assertEqual(uri.encode_value(u'abcd'), u'abcd')