requests.get() URL Encode Issue

Hi,

I noticed URL encode issue in get() --

In [32]: r = requests.get(r'http://i.ebayimg.com/00/s/ODAzWDEyODA=/$%28KGrHqV,!o8E63YcElkoBPFMhH2vUQ~~60_1.JPG')
In [33]: r.status_code
Out[33]: 404

In [34]: r = urllib2.urlopen("http://i.ebayimg.com/00/s/ODAzWDEyODA=/$%28KGrHqV,!o8E63YcElkoBPFMhH2vUQ~~60_1.JPG")
In [35]: r.getcode()
Out[35]: 200

In Wireshark, I see that it's because requests further encodes the URL (path) to """/00/s/ODAzWDEyODA%3D/%24%28KGrHqV%2C%21o8E63YcElkoBPFMhH2vUQ%7E%7E60_1.JPG"""

Is it a bug?


No, escaping urls is proper. Someone's working on a patch to make this optional, though.


Thanks Kenneth (though this implies the default behavior of urllib2.urlopen is improper...)


I'm having this issue with a project I'm working on and would like the patch for this being optional. Do you know who that "someone" working on the change is? Otherwise I could give it a go - I'd just like to avoid any duplicate work.


I'm pretty sure this is a bit wonky, somehow:

passing the URL: http://localhost:6060/test/United Kingdom/
hits the server as: [03/Feb/2012 09:41:27] GET /test/United%2520Kingdom/

passing the URL: http://localhost:6060/test/United%20Kingdom/
hits the server as: [03/Feb/2012 09:43:44] GET /test/United%2520Kingdom/

Both of which don't match what a browser sends, for example:
[03/Feb/2012 09:45:14] GET /test/United%20Kingdom/


In what way is this behaviour proper? It shouldn't be up to the library to re-encode parts of a URL, I'm pretty sure that doesn't even make sense.

For example, as far as I can tell it's impossible to GET any url with an escaped slash in it. e.g if I try to get "/%2F_blah", it will be escaped to "/%252F_blah", which is wrong.


This case is reopened to support the "canonical" behavior.


@stringfellow that's a bug. It's double-encoding. This happened when I added Python 3 support. I'm working on it.

@gfxmonk as I said, someone's working on a patch to make the escaping optional.


@kennethreitz but you seem to be indicating that it is correct behaviour for most (or even some) cases. I can't see how it could ever be right, so it bothers me to see it even implemented (regardless of whether you can turn it off).


`somesite.com/ø`


That's not a valid URL - this should be an error. The user _must_ encode their URL components before combining them, you can't possibly guess after the fact which characters were intended to be used literally and which weren't.

If you must support requesting a unicode URL, you should at least restrict the encoding to only affect non-ascii characters (since they couldn't have possibly be meant literally, and therefore encoding them is at least reasonable, if not strictly correct).


@gfxmonk it's not a URL, it's an IRI. Quoting needs to be optimized and optional.


@kennethreitz What do you mean by optimised? Quicker? Or restricted to non-ASCII characters?

I admit I hadn't heard of IRIs before now, but from a quick skim of the spec converting IRIs sounds like it must not escape an existing percent character, which the current code is doing.


That's exactly what I mean :)

I'm going to look closely at what chrome does. It seems to do this as we all expect.


Ahh ok, sounds good. Sorry for spamming the thread, but one last thing - you may want to look at: http://unspecified.wordpress.com/2012/02/12/how-do-you-escape-a-complete-uri/
This explains it pretty well, and with specifics. You should be able to construct a correct set of characters to pass in as quote's `safe` argument to achieve this.


Awesome, thanks.

You're always welcome to implement of course :)


Hi. I'm the author of the post @gfxmonk posted above. I've had a look at the requests library and fixed both of the bugs in this thread. My branch [py3-uri-encoding](https://github.com/mgiuca/requests/tree/py3-uri-encoding) contains both fixes, as well as test cases.

There are two separate issues: the double-encoding and the quoting of reserved characters. (Perhaps we should have two bugs and two branches, but I was short on time.)

First, the double-encoding is only a problem on Python 3 for me. In Python 2, it works great -- it unquotes and then quotes the path, which means that any illegal characters (such as '<') become quoted (to "%3C"), while any already-quoted characters (such as "%3E") remain quoted. It works on Unicode characters too. However, in Python 3, it has completely different behaviour: it simply encodes the path to UTF-8 (not necessary, since Python 3's quote automatically does this if given a Unicode string) and then quotes it. This results in double-quoting: '<' becomes "%3C", but "%3E" becomes "%253E".

My fix was basically to remove the Python 3 specific code and make it work the same as Python 2. I had to make a quick fix to utils.requote_path, and then it worked. On Python 3, it now just does the unquote and the requote, and that works.

As for the quoting of reserved characters, that is a more tricky issue. It is essentially the same problem as I outlined in my [blog post](http://unspecified.wordpress.com/2012/02/12/how-do-you-escape-a-complete-uri/). For some reason, all of this only concerns the path (I'd have applied it to the full URI, but for now, I'm just addressing how the path is handled). The current behaviour is to quote every character in the path except for '/', '%' and most of the unreserved characters. That violates [RFC 3986](http://tools.ietf.org/html/rfc3986#section-2.2):

> URIs that differ in the replacement of a reserved character with its
> corresponding percent-encoded octet are not equivalent.  Percent-
> encoding a reserved character, or decoding a percent-encoded octet
> that corresponds to a reserved character, will change how the URI is
> interpreted by most applications.  Thus, characters in the reserved
> set are protected from normalization and are therefore safe to be
> used by scheme-specific and producer-specific algorithms for
> delimiting data subcomponents within a URI.

You should NOT be tampering with any reserved characters. But I understand the desire to a) encode illegal characters (such as spaces and non-ASCII characters) so they form a valid URI, and b) unencode unreserved characters (such as '-') so they are easier to read without changing the semantics of the URI.

This took a bit of work, but I have rewritten requote_path so that it behaves the following way:
1. Unquote only the unreserved characters, leaving reserved and illegal characters as-is.
2. Quote only illegal characters, leaving reserved and unreserved characters as-is.

Note that I have also changed how the tilde ('~') character is treated -- for some reason Python treats it as a reserved character, which is wrong. It is unreserved, so it ought to be unquoted.

That means that percent-encoded unreserved characters will end up being unquoted (which is legal, because they are equivalent either way), illegal characters will end up being quoted (which is required for a valid URI), and reserved characters will remain exactly as they were. This solves the original problem of this bug report (I can successfully pull down that eBay URL) because the special characters in the path '=' and '$' are left unquoted.

There isn't any point in making this "optional" -- the current behaviour is a violation of the RFC and is erroneously tampering with valid URIs. If you must make it optional, you should make the correct behaviour the default, and add an option to quote all reserved characters in the path.

I have included comprehensive test cases for all of the behaviour described above.

P.S. As you mentioned you were interested, in my [blog post](http://unspecified.wordpress.com/2012/02/12/how-do-you-escape-a-complete-uri/) I described exactly how Chrome performs URI cleaning. It is largely the same as I have implemented here, but it has a few wonky characters that don't get escaped properly.


Is this issue now resolved?


Hopefully! I need to poke around a bit first.


Has this been resolved? I'm using 0.13.3 and Python 3.2 and "oauth_token=A%3" gets sent as "oauth_token=A%253". Is there a way I can get round to it?