Consistent String Encoding

Over the last few months there have been multiple issues raised about problems with string encodings. At best Requests is being inconsistent (#1250), and at worst perfectly valid combinations of input are failing (#1279).

I think it's time for Requests to enforce a very consistent behaviour when it comes to string encodings. We need to establish what format the data will be in when we pass it to urllib3, and where we're going to transform it. We need to be consistent so that when users inevitably encounter errors, we can make it very clear what they should have passed in.

@shazow has said that urllib3 wants the following input: bytes in the body, and native strings everywhere else. This defines our interface to urllib3.

To keep in sync with the above, we need to do the following things.
- On Python 2, any unicode strings anywhere on a `PreparedRequest` other than the body should be encoded as UTF-8. The body should also be encoded as UTF-8 if it is a unicode string (unlikely, but unless we can guarantee it won't be we should do the right thing).
- On Python 3, any bytestrings in a PreparedRequest should be decoded to unicode strings. Because we cannot make any intelligent guess, we will use Python's (stupid, stupid, _stupid_) locale-based decoding. This will almost-certainly throw exceptions if an incorrect encoding has been used, alerting the user.

Thoughts?


One thought I have is that I'm glad you didn't assign this to me. ;)

The only things I can think of are
- to make this a dumb function that lives in `requests/utils.py` and will do it for you.
- heavily document this so users might handle the encoding themselves or at least be able to find out why they're getting this new exception
- perhaps we should have our own exception so that people will be lead back to this area of the documentation mentioned above


Ok, so I've taken a first-pass at this. Right now this doesn't do anything about coercing the `body` parameter to bytes, which it needs to.

I want lots of feedback on this. In particular, at some point @kennethreitz is going to need to step in and make an API decision. This change is a massive API change and affects the output of the PreparedRequest. I think it's worth it, but if @kennethreitz doesn't then I'm happy to stop work on this.


Nice work. This mess with encodings should really be cleaned up.


It's a lot of work, but I definitely think it is worth it, regardless of the pains this will inflict on the API.


Good work. +1 for getting this sorted and consistent ASAP.


I'm no longer convinced by the approach I've taken here. I wonder if it might be better to use unicode internally the whole way through, and then encode at the Transport Adapter layer on Python 2.


Ugh, actually, that'll be a pain. I'd have to write a compatibility function for string literals to make that work, because Python 3.2 doesn't support `u''` literals. CURSE YOU PYTHON 3.2!


@Lukasa, the binary can't hear you.


I think things are working pretty well today and will just magically work perfectly once everyone's on Python 3. 

Assuming I'm correct, I think it's a good approach.


If you are prepared to wait for that day, so am I. =)


I think it's coming sooner rather than later. :)


Sounds good. I'll leave my branch on my fork as a reference in case anyone bumps into any problems.


Thanks for this work, by the way. Having conversations around this stuff, especially with code attached, is super important.


Always happy to do it. =)