German signs äÄöÖüÜß presented not correctly

Sending a response with special german signs and encoded as utf-8 does not really help
to get special german signs like äÄöÖüÜß in response.
Sending such signs results in empty response.
The answer object has as text field an empty field and can not use json(...) function member, ending with an exception.

## Expected Result

Was expecting responsed text in response object with german signs.

## Actual Result

Getting following exception:

File "C:\Python27\lib\site-packages\requests\models.py", line 886, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Python27\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Python27\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python27\lib\json\decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

## Reproduction Steps

Code that i use in parts:

For headers using following structure:

headers = {
                'Content-Type' : 'application/json; encoding="utf-8"',
                'Authorization' : 'authorization '
            }

Sending request with:

response = requests.post(
                url,
                headers = headers
            )
            response.close()

Trying to get text from response with:

response.text (But it is empty!) or a json object, because getting json in response
response.json() (But getting an exception described in ## Actual result)

## System Information

Python 2.7.5
requests 2.17.3


Are you actually sending any JSON data at all? You don't appear to be in your request: you only seem to be setting the url and the headers.

Hello Lukasa, it was just an example. I send JSON.
If you like to test encoding, that should be utf-8 per default =>
python =>
import requests
response = requests.get('https://www.tagesschau.de')
print(response.text)

Setting response.encoding='utf-8' explicitly does not help

print(response.text) still shows the following exception:

>>> print(response.text)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\encodings\cp850.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u2019' in position 99816: character maps to <undefined>
>>>

best regards 
Aleksey

This is not a Requests issue: it’s a Python one. If you remove the call to “print” and just access response.text you’ll encounter no error. The problem is that the text encoding your interpreter or shell is using, or rather that Python thinks it’s using, is not able to render all the characters in the decoded data. Python seems to think you’re using CP850. If you change that, you’ll be fine.



> On 6 Jun 2017, at 10:46, ashumilin <notifications@github.com> wrote:
> 
> Hello Lukasa, it was just an example. I send JSON.
> If you like to test encoding, that should be utf-8 per default =>
> python =>
> import requests
> response = requests.get('https://www.tagesschau.de <https://www.tagesschau.de/>')
> print(response.text)
> 
> Setting response.encoding='utf-8' explicitly does not help
> 
> print(response.text) still shows the following exception:
> 
> print(response.text)
> Traceback (most recent call last):
> File "", line 1, in 
> File "C:\Python27\lib\encodings\cp850.py", line 12, in encode
> return codecs.charmap_encode(input,errors,encoding_map)
> UnicodeEncodeError: 'charmap' codec can't encode character u'\u2019' in position 99816: character maps to
> 
> best regards
> Aleksey
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub <https://github.com/requests/requests/issues/4130#issuecomment-306436303>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUYnHDNb5ZoJanoqldtMjD10reOnOjqks5sBSASgaJpZM4Nw5ZB>.
> 



I see, but your suggestion does not really help :-).
At a particular time in code i have to load response.text as json(now it does not matter that i accept no json) - so i get an exception beacause of encoding.
Is there any solution to this problem(probably migration to another python version)? 

@ashumilin Sorry, again, there is a lack of clarity here.

Can you give an example of the actual code you're running that fails to work? I need to see what the actual data is.

of course, i get the following json structure(it should be available in response.text or provided by response.json)

{
	"kompott" : [
		{
			"zutat_1" : "äpfel",
			"zutat_1" : "pflaumen"
		}
	],
	"sorte" : [
		{
			"zutat_1" : "zimnije",
			"zutat_2" : "weiße"
		}
	]
}

response.text is empty, response.json() shows an exception,
if i replace ä with ae and ß with ss, i get the data without problems

This is not really what I need. What I need is a way to see the complete response, with the binary data exactly as it came from the server.

So basically, I think it's highly unlikely that Requests is barfing on *valid* JSON data. It seems much more likely that the data is not quite correct.

where do i get the binary data for response? 
the response is send as utf-8, an as i said without german signs, the code do the thing

The easiest way is to use tcpdump/wireshark.

Takes too much time, i replace them with Umlaute-Replacement-Algorithm ae, Ae, Oe, oe, ue, Ue, ss.
Thank you for your suggestions!
Aleksey

Sounds to me like we can close this now. If I'm mistaken, let us know and I'll reopen it.