%2e Being Decoded in URL

Hi there,

When a url-encoded period is included in a url (in path directory, path file, param name, or param value), it gets permanently converted to a period before the request is made (sometime before get_connection() is called in a subclassed HTTPAdapter). Other url-encoded things like single ticks (%27), double quotes (%22), semicolons (%3f), and percent symbols (%25) don't get decoded.

This means you can't successfully request a url with %2e anywhere in it because it gets changed to a '.', and if you try to double-url-encode it with %252e, it stays as %252e.

I've also tried passing %2e in via params=dict(...) and params='...bytes...', and nothing works.

Examples:
`requests.get('http://www.example.com/test.x%252ey/test.%252ey.html?test.x%252ey=test.x%252ey').url`
Output: u'http://www.example.com/test.x%252ey/test.%252ey.html?test.x%252ey=test.x%252ey'
First Line of Request When Viewing Through Proxy: GET /test.x%252ey/test.%252ey.html?test.x%252ey=test.x%252ey HTTP/1.1

`requests.get('http://www.example.com/', params=dict(a='x%2ey')).url`
Output: u'http://www.example.com/?a=x%252ey'
First Line of Request When Viewing Through Proxy: GET /?a=x%252ey HTTP/1.1

`requests.get('http://www.example.com/', params='a=x%2ey').url`
Output: u'http://www.example.com/?a=x.y'
First Line of Request When Viewing Through Proxy: GET /?a=x.y HTTP/1.1

`requests.get('http://www.example.com/', params='a=x.y').url`
Output: u'http://www.example.com/?a=x.y'
First Line of Request When Viewing Through Proxy: GET /?a=x.y HTTP/1.1

I just happen to be facing a use case where I need to be able to make a request containing a %2e in one of the parameters, so I'm a bit stuck on how to workaround this within the context of the Requests lib.

Best Regards,
-Justin


Thanks for this @correcthorsebatterystaple-!

Requests has strong opinions about the correct form of a URL: specifically, it does not percent-encode things that don't normally need percent encoding. I should note that your server is in this instance misbehaving: periods do not need to be percent encoded in query string and there's no reason to do it, and even if there were, for characters that do not need percent encoding both the encoded and non-encoded character are to be treated _exactly identically_.

Regardless, the pragmatic solution to this problem is the [prepared request flow](http://docs.python-requests.org/en/master/user/advanced/#prepared-requests), where you construct a `Request` object, `prepare()` it, and then edit the `url` field on the `PreparedRequest` object you get. Any changes to the URL field will then be kept, so you can (for example) double-encode the period in the query string and then do a `prepped.url.replace()` call to swap it out with the single-encoded period.


Thanks for the response, @Lukasa.

Fair point regarding equivalency and the server misbehaving. However, one doesn't always have control over the misbehaving server, so sometimes you have to have a way to send what needs to be sent, even if it's technically wrong. :-)

Thanks for the info on how to get around this with a PreparedRequest!

Out of curiosity, I understand a stance of not percent-encoding things that don't typically need it, but why percent-**de**code things (specifically the period in this case)? Does it benefit other parts of the codebase by normalizing encoded and decoded periods when performing operations?

Also, regarding Requests' stance on URL forms and encodings, is there documentation regarding what gets encoded/decoded where? For example, I noticed passing query string params as a dict results in encoding, but passing as bytes doesn't.

I'm mainly asking because I'm writing a tool that walks through a series of requests (to and from servers I don't have any control over ;-) ), and part of the challenge is to ensure certain pieces of data pass through the process in their intended form. If I pull a URL out of a Location header, for example, do I need to do any manually encoding on certain characters before I pass it to Requests (like plus signs)?

Thank you for your time on this! I'm a bit of a Python noob (maybe I've graduated to noob+ at this point ;-) ), and I'm even more of a Requests noob, so I appreciate you taking the time to field my feedback.


> Why percent-decode things (specifically the period in this case)? Does it benefit other parts of the codebase by normalizing encoded and decoded periods when performing operations?

The short answer is because we try to normalise URLs. This is because users are often inconsistent with their percent encoding, leading to URLs that are partially encoded and partially not. That causes problems when we want to come to percent-encode a URL ourselves, because blindly percent-encoding a URL that may be either unencoded, encoded, or partially encoded can lead to all kinds of nasty results. The easiest way to resolve the problem is to do what we currently do, which is _decode_ the URL (ignoring unencoded values rather than treating them as errors) and then _re-encoding_ it according to the standard encoding rules.

Generally speaking, Requests reserves the right to transform your request into a form that is semantically identical to the original in order to provide a better user experience for the majority of users: essentially, we try to save users from themselves at the cost of making it harder to be really specific about what you want.

> If I pull a URL out of a Location header, for example, do I need to do any manually encoding on certain characters before I pass it to Requests (like plus signs)?

You don't _need_ to, but you may want to manually handle it yourself if you need it to remain _identical_ to the Location header provided.


And this is exactly why PreparedRequests were created. 


Right on. Thanks!