Unquote+Quote cycle prohibits urls required by salesforce.com

Salesforce.com requires you %hex encode period (.) and slash (/) to issue a PATCH.

The unquote+quote cycle in `requote_uri(uri)` disallows this:  https://github.com/kennethreitz/requests/blob/master/requests/utils.py#L398

Since it's not prohibited by the http spec (see uri comparison at http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.2.2) but some web services like salesforce.com won't allow the unquoted values for period or slash, could this be disabled?  I'm glad to issue a pull request for disabling it but figured that I'd ask first for background before assuming I knew why this was here.


Thanks for raising this issue!

Firstly, I think the slashes are a non-starter. We always urlencode them if you pass them in the `params` dict, so that's not a problem. The sole problem seems to be with periods.

Happily, the answer is that you can disable it if you use the [PreparedRequest](http://docs.python-requests.org/en/latest/user/advanced/#prepared-requests) objects. Try this:

``` python
from requests import Request, Session

s = Session()
req = Request('GET', url)
prepped = s.prepare_request(req)
prepped.url = prepped.url.replace('.', '%2E')
resp = s.send(prepped)
```


My issue is not with params but with path. Salesforce.com requires the id in the URL has these both escaped.  I only need one of my periods to be quoted in this path, not all of them. I consider this to be a bug because the code is changing my valid (albeit wonky because of Salesforce.com) URL. 

Is there a use case for the unquote and quote cycle?

Sent from my iPhone

> On Jan 3, 2014, at 6:42 PM, Cory Benfield notifications@github.com wrote:
> 
> Thanks for raising this issue!
> 
> Firstly, I think the slashes are a non-starter. We always urlencode them if you pass them in the params dict, so that's not a problem. The sole problem seems to be with periods.
> 
> Happily, the answer is that you can disable it if you use the PreparedRequest objects. Try this:
> 
> from requests import Request, Session
> 
> s = Session()
> req = Request('GET', url)
> prepped = s.prepare_request(req)
> prepped.url = prepped.url.replace('.', '%2E')
> resp = s.send(prepped)
> —
> Reply to this email directly or view it on GitHub.


Ah, sorry.

Yes, there is. The problem is that Requests doesn't know whether it's going to get a quoted or an unquoted URI (or indeed a URI that is some combination of quoted and unquoted). In an attempt to be helpful, we do our best to normalise them: that is, we unquote everything, then requote again. That way, anything that was quoted remains quoted, and anything that wasn't becomes quoted.

This is valuable behaviour: most users don't understand the complexities of URL quoting (hell, _I_ don't understand all of them), and they don't want to have to have a class about it. They want to be able to pass a URL and have us do the right thing, and in 90% of cases the quote-unquote cycle does exactly that.

If you're interested, the relevant comment talking about this behaviour is [this one](https://github.com/kennethreitz/requests/issues/369#issuecomment-3953802), but the key point of that comment is this:

> [...] percent-encoded unreserved characters will end up being unquoted (which is legal, because they are equivalent either way), illegal characters will end up being quoted (which is required for a valid URI), and reserved characters will remain exactly as they were.

The period is unreserved, so is unquoted. This is _totally correct_ behaviour. Normally I wouldn't consider that a compelling argument (we do lots of incorrect things to work around bugs introduced by doing the correct thing), but in this case if we change this behaviour we will simply introduce bugs in other web services that make the opposite mistake to Salesforce.

The question then becomes, do we add a feature to the API to disable this? The answer is no: we are not changing Requests' API at this time. The expected way to work around this set of problems is to mutate the URL provided by the `PreparedRequest` object I showed you above. This is really the only way to guarantee that we'll do the correct incorrect thing for your specific use case. I appreciate that it's not very clean, but there's not a whole lot we can do about that.

Is that explanation clear? I'm very tired at the minute, so I may have been unclear: if I was, I apologise. =)


What I hear is that the intended behavior is to handle quoted and unquoted (and mixed) urls. I agree with this because it makes me have to think less. 

It looks like the unquote method is used to prevent double quoting. That unquote causes a problem because it modifies a valid url by unnecessarily unquoting text and sends something different than the user specified.

I am proposing it work without unquote and add logic that prevents double quoting so that people can use a mix of both but requests will not unnecessarily modify/destroy urls that are valid without telling the user. 

If this sounds reasonable I will open a pull. 

Thanks!

Sent from my iPhone

> On Jan 3, 2014, at 7:00 PM, Cory Benfield notifications@github.com wrote:
> 
> Ah, sorry.
> 
> Yes, there is. The problem is that Requests doesn't know whether it's going to get a quoted or an unquoted URI (or indeed a URI that is some combination of quoted and unquoted). In an attempt to be helpful, we do our best to normalise them: that is, we unquote everything, then requote again. That way, anything that was quoted remains quoted, and anything that wasn't becomes quoted.
> 
> This is valuable behaviour: most users don't understand the complexities of URL quoting (hell, I don't understand all of them), and they don't want to have to have a class about it. They want to be able to pass a URL and have us do the right thing, and in 90% of cases the quote-unquote cycle does exactly that.
> 
> If you're interested, the relevant comment talking about this behaviour is this one, but the key point of that comment is this:
> 
> [...] percent-encoded unreserved characters will end up being unquoted (which is legal, because they are equivalent either way), illegal characters will end up being quoted (which is required for a valid URI), and reserved characters will remain exactly as they were.
> 
> The period is unreserved, so is unquoted. This is totally correct behaviour. Normally I wouldn't consider that a compelling argument (we do lots of incorrect things to work around bugs introduced by doing the correct thing), but in this case if we change this behaviour we will simply introduce bugs in other web services that make the opposite mistake to Salesforce.
> 
> The question then becomes, do we add a feature to the API to disable this? The answer is no: we are not changing Requests' API at this time. The expected way to work around this set of problems is to mutate the URL provided by the PreparedRequest object I showed you above. This is really the only way to guarantee that we'll do the correct incorrect thing for your specific use case. I appreciate that it's not very clean, but there's not a whole lot we can do about that.
> 
> Is that explanation clear? I'm very tired at the minute, so I may have been unclear: if I was, I apologise. =)
> 
> —
> Reply to this email directly or view it on GitHub.


First, let's be clear. Yes, the unquote/quote modifies a valid URL, but it modifies the URL in a way that leaves the URL _completely synonymous_, as defined by the standard. The default position is that we have done nothing wrong here, so we need a good justification for removing this step.

Unfortunately, there simply isn't one. Salesforce is _wrong_ to require that a percent-encoded period has a different meaning to a literal one. That wrongness is dramatic: you would encounter the same problem with browsers. Using TCPdump with Chrome shows that Chrome has the same 'bug' Requests has:

```
Input:           http://httpbin.org/get/test.test
Wireshark path:  /get/test.test

Input:           http://httpbin.org/get/test%2Etest
Wireshark path:  /get/test.test
```

There is no way for you to turn this off that won't result in the opposite bug for someone else: that is, someone is being saved by us turning their percent-encoded period into a literal one. We've made our decision: we'll do what browsers do with their URL bars. If you need to change that to work with a misbehaving server, you can do that, using the `PreparedRequest` URL.


@jsullivanlive 
Has someone raised this issue with salesforce.com?


In the past quirks like this encountered on salesforce have never been addressed. I encourage someone to tell them but I wouldn't encourage you to get your hopes up


@sigmavirus24 and I have had the same experience, which is also the prevailing impression in #salesforce on freenode.  Oracle, Salesforce, and any other big companies have more concern over their slipping ship dates than their adherence to rfc and test coverage. This is why, after 20 years of development behind me, I touch data as little as possible. :)


requests-oauthlib has also had reports that Salesforce has not upgraded the version (draft) of OAuth 2.0 that they're using in spite of the fact that they were involved in its creation.