Issue 1:
GET request to an API returned a response with broken encoding.

More specifically, the response body was encoded in ASCII even though the data sent via API is encoded in UTF-8.

The result is escaped non-ASCII characters in the response body.

This behaviour can be observed on r.content, r.text, and r.json().

Following the quickstart guide, I tried to "force encoding" with r.encoding = 'UTF-8' but it had no effect.

I made sure to include charset=UTF-8 in request header, which also didn't help.

What ended up working was json.loads(r.text).

Very simple I know, but locating the problem was difficult,

not to mention I had to be absolutely sure the API was sending a data that was encoded in UTF-8.

This problem is not mentioned or, at least, not easily found on Stackoverflow.

I thought I would mention it here, so that others can find it.

issue #5577 seems to be similar in nature, which only proves that Encoding is an issue that seems to pop up every now and then.

The answer given for that issue seems to be "there is an easy solution so there's no need to fix it on our ends".

But nonetheless, a bug is a bug.

Besides, it would be nice to mention how to troubleshoot encoding on the quickstart guide.

Sorry it's not so much of a "summary", but things had to be said.

Expected Result

A response body properly encoded in UTF-8.

Like so, {'body': '상품등록'}

Actual Result

A response body encoded in ASCII even though it's supposed to be UTF-8.

{'body': '\uc0c1\ud488\ub4f1\ub85'}

Reproduction Steps

import requests

url = 'https://www.sorryicannottellyoutheactualapi.com/api'
headers = {'Content-Type': 'application/json;charset=UTF-8'}
r = requests.get(url, headers=headers)
print(r.content)
print(r.text)
print(r.json())

System Information

chardet 3.0.4
cryptography 2.8
idna 2.9
implementation CPython 3.6.7
platform Linux 4.15.0-101-generic
pyOpenSSL 19.1.0 (openssl 1010101f)
requests 2.23.0
system_ssl 1010104f
urllib3 1.25.7
using_pyopenssl True

python 3.6.7

Issue 2:
The location header in a 302 redirect response is not being parsed correctly when it has special characters in it.

/Login.aspx?ReturnUrl=%2f becomes /Login.aspx%3FReturnUrl%3D/

When followed, this results in a 404 because it's not correct.

>>> import requests
>>> r = requests.get('http://affiliatenetwork.ist-apps.com')
>>> r.status_code
404
>>> print r.headers
{'content-length': '1245', 'x-powered-by': 'ASP.NET', 'date': 'Tue, 18 Oct 2011 09:15:08 GMT', 'content-type': 'text/html', 'connection': 'close', 'server': 'Microsoft-IIS/7.0'}
>>> r.url
'http://affiliatenetwork.ist-apps.com/Login.aspx%3FReturnUrl%3D/'
>>> r.history
[<Response [302]>]
>>> r.history[0].headers
{'content-length': '142', 'x-powered-by': 'ASP.NET', 'x-aspnet-version': '4.0.30319', 'server': 'Microsoft-IIS/7.0', 'connection': 'close', 'location': '/Login.aspx?ReturnUrl=%2f', 'cache-control': 'private', 'date': 'Tue, 18 Oct 2011 10:56:08 GMT', 'content-type': 'text/html; charset=utf-8'}

curl parses this correctly:

david@Shibuya ~: curl -v http://affiliatenetwork.ist-apps.com
* About to connect() to affiliatenetwork.ist-apps.com port 80 (#0)
* Trying 94.236.92.60... connected
* Connected to affiliatenetwork.ist-apps.com (94.236.92.60) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: affiliatenetwork.ist-apps.com
> Accept: */*
>
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=utf-8
< Location: /Login.aspx?ReturnUrl=%2f
< Server: Microsoft-IIS/7.0
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< Date: Tue, 18 Oct 2011 11:26:34 GMT
< Content-Length: 142
<
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="/Login.aspx?ReturnUrl=%2f">here</a>.</h2>
</body></html>
* Connection #0 to host affiliatenetwork.ist-apps.com left intact
* Closing connection #0

httplib also parses this correctly:

>>> conn = httplib.HTTPConnection('affiliatenetwork.ist-apps.com')
>>> conn.request("GET","/")
>>> res = conn.getresponse()
>>> print res.status
302
>>> print res.reason
Found
>>> print res.getheaders()
[('content-length', '142'), ('x-powered-by', 'ASP.NET'), ('x-aspnet-version', '4.0.30319'), ('server', 'Microsoft-IIS/7.0'), ('location', '/Login.aspx?ReturnUrl=%2f'), ('cache-control', 'private'), ('date', 'Tue, 18 Oct 2011 10:35:20 GMT'), ('content-type', 'text/html; charset=utf-8')]

Looks like the problem is caused by https://github.com/kennethreitz/requests/blob/master/requests/models.py#L237

>>> url = '/Login.aspx?ReturnUrl=%2f'
>>> o = urlparse.urlparse(url)
>>> print o
ParseResult(scheme='', netloc='', path='/Login.aspx', params='', query='ReturnUrl=%2f', fragment='')
>>> import urllib
>>> urllib.quote(urllib.unquote(url))
'/Login.aspx%3FReturnUrl%3D/'
>>> urllib.unquote(url)
'/Login.aspx?ReturnUrl=/'