Handling of URLs with unicode in python 2.7

Consider the URL: `https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl`. This URL contains the unicode character u`è`. If you open in most major browsers there should be no problem, similarly cURLing like  `curl -L 'https://www.sainsburys.co.uk/shop/gb/groceries/chabllis-premi%C3%A8r-cru-brocard-75cl'` works without issue (i.e. we request the percent encoded utf8 encoded URL).

However if we try to get this url with python requests as follows:

    unicode_url = u'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl'
    r = requests.get(unicode_url)

then it will 404. At this point I considered that maybe I just need to UTF8 encode the url and pass the byte str to requests:

    r =  requests.get(unicode_url.encode('utf8'))

Nope, 404 again. Finally I considered also manually doing the percent encoding (although I'm sure requests deals with this):

    r = requests.get(urllib.quote(unicode_url.encode('utf8')))

Nope, it really doesn't like that.

I also tried `requests.get(unicode_url.encode('latin1'))` but now got a UnicodeDecodeError (presumably because requests is attempting to decode with utf8).

So how on earth can I request this URL in python-requests in python 2.7?
    



Just adding some local testing to see what the web server is seeing. With a local HTTP server running and then running this:
```python
import requests
requests.get(u'http://127.0.0.1:8000/è')
```
The web server receives this as the HTTP header:
`127.0.0.1 - - [15/May/2017 14:52:35] "GET /%C3%A8 HTTP/1.1" 404 -`

Similarly running this:
```python
import requests
requests.get(u'http://127.0.0.1:8000/%C3%A8')
```
nets the same result on the web server:
`127.0.0.1 - - [15/May/2017 14:52:45] "GET /%C3%A8 HTTP/1.1" 404 -`

Running this:
```python
import requests
requests.get(u'http://127.0.0.1:8000/è'.encode('utf-8'))
```
also nets the same result on the web server.
`127.0.0.1 - - [15/May/2017 14:55:18] "GET /%C3%A8 HTTP/1.1" 404 -`

To add to those local tests, I also tried fetching with Scrapy in the Scrapy shell:

    fetch(Request(u'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl'.encode('utf8')))

This is successful  with 200 status.

Testing against a local server with Scrapy: 

    fetch(Request(u'http://127.0.0.1:8000/è'.encode('utf8'), dont_filter=True))

gives `127.0.0.1 - - [15/May/2017 20:03:47] "GET /%C3%A8 HTTP/1.1" 404 -` 

so I'm slightly puzzled at how the local log looks the same for Scrapy and python-requests for the u`è` char, ye, the Scrapy request is successful (I tried setting the User-Agent in python requests too, just in case, but to no avail).

Whats `response.request.url` in the failing case?

@Lukasa Oh that's interesting, the `response.request.url` is the monsterous 

`'https://www.sainsburys.co.uk/webapp/wcs/stores/servlet/gb/groceries/chablis/chablis-premi%E8r-cru-brocard-75cl?langId=44&storeId=10151&krypto=avT10G07EA5Ef5vUiHvBmeJymCkV1iJKI%2BICpLkmAAfHIZJZ6oAoS5YpWh0JsNL4Y5DVN%2F%2Bx%2BLg23H1HqHhekHiyPEml4aL4FBurJg9kEqyOjrk%2BymN9sOje5plJkgTf9f8w05kssDkEByvy5LHAVUPfnEURk0xqW7M2Za9MJWY%3D&ddkey=https%3Agb%2Fgroceries%2Fchablis%2Fchablis-premi%C3%A8r-cru-brocard-75cl'`

in the failing case.

If I turn off redirects I get something more sensible.
```python
print(requests.get(u'http://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl', allow_redirects=False).request.url)
```
prints out `'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premi%C3%A8r-cru-brocard-75cl'`

Yeah, so this is unlikely to be directly related to the URL and much more likely to be about the rest of the Request. I wonder if one of the request headers is causing problems. 

Here are the headers for the HTTP redirect response, note the `LOCATION` field having `LATIN-1 \xe8` in place of our `UTF-8 è`.

```
CONTENT-LENGTH=0
CONTENT-TYPE=text/html;charset=UTF-8
CONTENT-LANGUAGE=en-GB
SET-COOKIE=Apache=10.173.10.13.1494879611612576; path=/, JSESSIONID=0000VFtqeHv_MVwpT3yTbZ_ESeK:17buic8rt; HTTPOnly; Path=/, WC_PERSISTENT=F1JXlopv9%2Fvb%2BgnuCpQnfmVuEsE%3D%0A%3B2017-05-15+21%3A20%3A11.615_1494879611615-630867_0; HTTPOnly; Expires=Wed, 14-Jun-17 20:20:10 GMT; Path=/, SESSION_COOKIEACCEPT=true; HTTPOnly; Path=/, WC_SESSION_ESTABLISHED=true; HTTPOnly; Path=/, WC_PERSISTENT=kbGcXW5r9J0l1UOqxJS3jGdEr8U%3D%0A%3B2017-05-15+21%3A20%3A11.624_1494879611615-630867_10151; HTTPOnly; Expires=Wed, 14-Jun-17 20:20:10 GMT; Path=/, WC_AUTHENTICATION_-1002=-1002%2CJLwz3JSOi44I7duGuk3ubPJibnI%3D; HTTPOnly; Path=/; Secure, WC_ACTIVEPOINTER=44%2C10151; HTTPOnly; Path=/, WC_USERACTIVITY_-1002=-1002%2C10151%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2CP4jUxIHNtjybZiX1clIkTMr4hqkqtEkvAIxD%2FjBtgOmmRSV2%2BQ8pSlCkdE16%2FPpdKxlgI5DnXS478Bd6xrPa4VXYvNhRPMujgCRlnF%2B%2BOFLZWIxk9XQATZ0yADjEJ8qATKm0GlxV3il0UPwL8JPngI9OCC03gHm16Asva4TFbGfceuE4G0WqHEODeNQhoTgflGUcYt0Xmzvoz4bRwaOL7w%3D%3D; HTTPOnly; Path=/, WC_GENERIC_ACTIVITYDATA=[18456303419%3Atrue%3Afalse%3A0%3Azx1HjzLxjqoYBsHke%2Bd7Ug4RlL0%3D][com.ibm.commerce.context.audit.AuditContext|1494879611615-630867][com.ibm.commerce.store.facade.server.context.StoreGeoCodeContext|null%26null%26null%26null%26null%26null][CTXSETNAME|Store][com.sol.commerce.context.SolBusinessContext|null%26null%26null%26null%26null%26null%26null%26null%26false%26false%26false%26null%26false%26null%26false%26false%26null%26false%26false%26false%26null%26false%26false][com.ibm.commerce.context.globalization.GlobalizationContext|44%26GBP%2644%26GBP][com.ibm.commerce.catalog.businesscontext.CatalogContext|10241%26null%26false%26false%26false][com.ibm.commerce.context.preview.PreviewContext|null%26null%26false][com.ibm.commerce.context.base.BaseContext|10151%26-1002%26-1002%26-1][com.ibm.commerce.context.experiment.ExperimentContext|null][com.ibm.commerce.context.entitlement.EntitlementContext|10502%2610502%26null%26-2000%26null%26null%26null]; HTTPOnly; Path=/, sbrycookie1=630263751;path=/;, sbrycookie2=FdrHbjOczkHbHbkHbHbjOc;path=/;domain=sainsburys.co.uk;, TS017d4e39=01a4f6ca7c6475495093834d7fd522443e41ba56ffd1d2b30a200be8149f886bad6c420c09120ee1903630ce23e061783b2d82ec56032188b7e79057cc61fc425a3ba8e099db7903081dfb4b084a60dde7e751a8c857388621c15d6db47574064f7f5b835cfa9ca301c69113251139456ef238a2fddc83350fdd1bae711a372b15aac6cf10d2f4af97ae38a37c42ca8bbe6d08b9a8201f4b85c31ede51f9a83cb56b17287ff5628ab383d33ee9d3d34e673aecd4c15be6b32839f1a5e76e6d04c84bc43cb4d7ba9114f0698080e2e9611f8f75dda6; Path=/, TS01cd64fc=01a4f6ca7c59e5c22be507a4b33c6594f8edbdc5d729b3fac53ed692e9fe894f1042e3fc38857b9063f35a2b81a9e7253e8aaf8ff8; path=/; domain=sainsburys.co.uk
EXPIRES=Thu, 01 Jan 1970 00:00:00 GMT
KEEP-ALIVE=timeout=15, max=4955
CONNECTION=Keep-Alive
LOCATION=https://www.sainsburys.co.uk/webapp/wcs/stores/servlet/gb/groceries/chablis/chablis-premi�r-cru-brocard-75cl?langId=44&storeId=10151&krypto=EhgWKRYw8NiBb2J%2B2NCnNPKz3cPG2ogpLvUgcZ7dD%2F71VThR7j0Sa%2BaCyf%2B5aGWf%2BXEqhAjOjcJr3xWVANaQnlPKA5uCK%2BnI36mOKAmflrzCj8mTkF1pcc85xQVnUMSMZMhnCBDRNRB0th%2B%2FJb6u4gBcjXp5FEpO5fNyWYz0TCI%3D&ddkey=https%3Agb%2Fgroceries%2Fchablis%2Fchablis-premi%C3%A8r-cru-brocard-75cl
PRAGMA=No-cache
CACHE-CONTROL=no-cache,no-store,max-age=0
DATE=Mon, 15 May 2017 20:20:11 GMT
X-FRAME-OPTIONS=SAMEORIGIN, SAMEORIGIN
TTFB=D=23924
REQ_TIMESTAMP=t=1494879611612501
```

I observed something odd:

```
In [1]: import requests

In [2]: s = requests.Session()

In [3]: unicode_url = u'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl'

In [4]: s.get(unicode_url)
Out[4]: <Response [404]>

In [5]: s.get(unicode_url)
Out[5]: <Response [200]>
```

Seems to work in a session the second time around! 

Tempting to say this is actually about cookies. The first request is without cookies and the webserver 404s because of that whilst still setting some session cookies. The next request sends the cookies and the webserver 200s.

**BUT**, notice that the second request now does not need a redirect; you could replace the second request with `s.get(unicode_url, allow_redirects=False)` and still get a 200, not a 302. Hence the only reason it works now is that redirection is being sidestepped with cookies. This indicates the encoding issues are taking place somewhere in the redirect chain.

**There is one weird thing too:**

    In [11]:   requests.get(u'http://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premiÃ¨r-cru-brocard-75cl')
    Out[11]: <Response [200]>

despite no cookies/sessions. I am struggling to understand that. The location header of the redirect in this case looks like:

     'Location':  'http://www.sainsburys.co.uk/webapp/wcs/stores/servlet/gb/groceries/chablis/chablis-premi\xc3\xa8r-cru-brocard-75cl?langId=44&storeId=10151&krypto=dZB7Mt97QsHQQ%2BGMpb1iMZwdVfmbg%2BbRUdkh%2FciAItm7%2F4VSUi8NRUiszN3mSofKSCyAv%2F0QRKSsjhHzoo1x7in7Ctd4vzPIDIW5CcjiksLKE48%2BFU9nLNGkVzGj92PknAgP%2FmIFz63xpKhvPkxbJrtUmwi%2FUpbXNW9XIygHyTA%3D&ddkey=http%3Agb%2Fgroceries%2Fchablis%2Fchablis-premi%C3%83%C2%A8r-cru-brocard-75cl'

i.e. we have the utf8 encoded u'è' instead of latin1 encoded.

Our `response.request.url` in this case was:

    'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premi%C3%83%C2%A8r-cru-brocard-75cl'

from which the original unicode url can be recovered with

    In [46]: print urllib.unquote(response.request.url).decode('utf8').encode('latin1').decode('utf8')
    https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl

So webserver is using latin1 it seems....



I observed this behaviour even if a clean session of Chrome, 404 on the first attempt at the URL and 200 on the second.

**Summary**

Given that the first request (the request that actually has no cookies and relies on redirects) fails in every platform I have tried - Chrome, Scrapy, python-requests - I would put this down to a bug on the host server itself. It latin1 encodes it's location header in the redirect, but wants utf8 encoded URLs and 404s when the browser actually requests that redirect location URL, because the server was actually expecting a utf8 encoded URL. It should really be utf8 encoding the location header of it's redirect response in order to be consistent with the URL encoding it is using elsewhere.

This is why when you cheat and use `u'http://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premiÃ¨r-cru-brocard-75cl'` you actually end up with the correct utf8 encoded location header in the redirect, because `u'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premiÃ¨r-cru-brocard-75cl'.encode('latin1')` is `'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premi\xc3\xa8r-cru-brocard-75cl'`, which happens to be the correct utf8 encoded byte str of `u'https://www.sainsburys.co.uk/shop/gb/groceries/chablis/chablis-premièr-cru-brocard-75cl'`, so when the browser redirects it works.

If you have cookies set from visiting the URL already or perhaps somewhere else on the website, you sidestep the need for a redirect, and avoid the broken redirect process altogether.

All this is absolutely true in a recent version of Chrome and Firefox too.


Issue is avoided in python3 because location header is str not bytes however.

I think this explains a lot:

https://github.com/kennethreitz/requests/blob/eae38b8d131e8b51c3daf3583e69879d1c02f9a4/requests/sessions.py#L101-L114

Ok, so there isn't really much Requests can do about this I think: the fact that the server is encoding the URL in an encoding that it doesn't itself accept is not really very sensible. We avoid this problem by sheer good luck on Python 3, but I don't think there is any sensible approach to fixing this generally. I think this might just be Sainsbury's crappy web server being crappy again.

@Lukasa I would agree with that. 

>> We avoid this problem by sheer good luck on Python 3

Could you point me to the relevant bit of the code that does this please? I'm just curious. I've been trying to work out how in the  python3 version the byte str received from the location headers is correctly decoded in latin1 to the correct str, and then re-encoded to utf8 byte before being sent.

The decode is actually done inside the standard library in `http.client`.

Thanks, well feel free to close this if you like.