Fix incorrect encoding for reason

HTTP status lines are latin1 and not utf-8

>    The TEXT rule is only used for descriptive field contents and values
>    that are not intended to be interpreted by the message parser. Words
>    of *TEXT MAY contain characters from character sets other than ISO-
>    8859-1 only when encoded according to the rules of RFC 2047

To quote the RFC. Nobody implements RFC 2047 so this can be disregarded.

<!-- Reviewable:start -->

---

This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/kennethreitz/requests/3538)

<!-- Reviewable:end -->


The UTF-8 decode was added here very deliberately, to handle the fact that many origin servers incorrectly serve UTF-8 reason phrases. Given that all defined reason phrases are ASCII, it seemed a reasonable middle ground to handle them as UTF-8. 

Do you have a specific problem you have encountered with this choice?


Of course, I would rather ignore the reason phrase entirely because it's stupid. 


See also: #3385.


@Lukasa it came up because Jira when served up with french locale sends french status reasons. Originally it was an exception and when we saw this was fixed I noticed that now the reason is garbled. I don't care _that_ much about it but the current code is definitely incorrect.


So I agree that the current code is incorrect per the spec, but I don't agree that it's incorrect per actual usage.

Fundamentally this boils down to the use cases of reason phrases. There are two schools of thought on them. The first (and I would argue the correct) one is that reason phrases are vestigial additional data that serves no meaningful purpose. Clients cannot be expected to take any action based on the reason phrase alone, and so it's basically nothing more than an opaque byte sequence with no meaning. This interpretation is supported by browsers and by the IETF: browsers ignore reason phrases, and the IETF stripped reason phrases from HTTP/2.

However, the persistent desire to _localize_ reason phrases reveals the other interpretation. The only point of localizing reason phrases is if you expect humans to read them. If you expect that, then it's bizarre to privilege ISO-8859-1: why do [these languages alone](https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Modern_languages_with_complete_coverage) get to localize their reason phrases, while everyone else is out in the cold? And, in fact, we have good evidence that reason phrases _are_ delivered in UTF-8 in some instances.

So the question becomes: given that people insist on localizing their reason phrases (which makes very little sense to me), what is the correct way to handle them? We could pass opaque byte strings around, but that doesn't work on Python 3. Decoding in either Latin 1 or UTF-8 risks mangling some reason phrases. Or we could just remove the reason phrase entirely from the debug output.

For my part, I don't think I care. UTF-8 with 'ignore' seemed like a best-effort choice to handle this insane server behaviour, but ISO-8859-1 would also do just fine. With that in mind, I think right now I'm inclined to prefer the current behaviour slightly, if for no better reason than anyone who knows enough to care about this charset choice also knows enough not to give a crap about what the reason phrase says.

Incidentally, JIRA should stop localizing its reason phrases.

Anyway, I'll let one of the other maintainers step in here. I don't really care, and when I don't care I tend to want to prefer the status quo. Roll on HTTP/2 where we'll never have to have this discussion again. =D


Another option is that, given that we're off the hot path, we could try to decode as UTF-8 first and then, if that fails, fall back to ISO-8859-1. That is probably the closest to best effort. 


> So I agree that the current code is incorrect per the spec, but I don't agree that it's incorrect per actual usage.

Actual usage: there are actual servers out there sending back latin1. Are there actual servers out there sending back utf-8? Even in case the latter is correct you can go from latin1 to utf-8 without loss, you cannot go the other way.


@mitsuhiko we changed it because there are servers sending back utf-8. I can't search for it at the moment, but you should have little problems finding that pull request and issue.


I linked it above: #3385.


> Even in case the latter is correct you can go from latin1 to utf-8 without loss, you cannot go the other way.

Sure, which is why we don't unconditionally convert the reason phrase. We only convert it during the processing for `raise_for_status`, specifically to build the Unicode error string. That string is _strictly_ best effort.

Regardless, I'm happy to remove the 'ignore' encoding directive and instead fall back to ISO-8859-1 if UTF-8 fails. That should cover the complete set of problems. 


Resolved by #3554 instead. Thanks @mitsuhiko!