TypeError after 2.7.0 -> 2.8.0 upgrade (cannot make memory view...)

I'm running into this traceback after upgrading to 2.8.0 from 2.7.0:

```
<snip>
    response = self.session.send(request)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/requests/sessions.py", line 579, in send
    r = adapter.send(request, **kwargs)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/requests/adapters.py", line 369, in send
    timeout=timeout
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/python/2.7.9/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/opt/python/2.7.9/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/opt/python/2.7.9/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/opt/python/2.7.9/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/opt/python/2.7.9/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 216, in sendall
    data = memoryview(data)
TypeError: cannot make memory view because object does not have the buffer interface
```

The problem goes away after downgrading to 2.7.0.

A full traceback can be found [here](https://travis-ci.org/simon-weber/gmusicapi/jobs/82365307).

Here are the versions of relevant packages:

```
httplib2-0.9.2
ndg-httpsclient-0.4.0
pyasn1-0.1.9
pyasn1-modules-0.0.8
pyopenssl-0.15.1
requests-2.8.0
```

And a full list of packages:

```
MechanicalSoup-0.3.1
appdirs-1.4.0
beautifulsoup4-4.4.1
cffi-1.2.1
cryptography-1.0.2
decorator-4.0.4
enum34-1.0.4
gmusicapi-7.0.1.dev0
gpsoauth-0.0.4
httplib2-0.9.2
idna-2.0
ipaddress-1.0.14
mutagen-1.31
ndg-httpsclient-0.4.0
oauth2client-1.5.1
proboscis-1.2.6.0
protobuf-2.6.1
pyasn1-0.1.9
pyasn1-modules-0.0.8
pycparser-2.14
pycrypto-2.6.1
pyopenssl-0.15.1
python-dateutil-2.4.2
requests-2.8.0
rsa-3.2
six-1.10.0
validictory-1.0.1
```

Any ideas?


Can I see a bit of sample code? I suspect you're passing a Unicode string somewhere. 


Good call. To reproduce this, you need both a unicode method and an https url, eg

```
python -c 'import requests; s = requests.Session(); r = requests.Request(u"POST", "https://httpbin.org/post"); pr = s.prepare_request(r); print s.send(pr).content'
```

In my case, the unicode method comes from [this line](https://github.com/hickford/MechanicalSoup/blob/0dd97c237cbad16e7c3f6a46dcbd6f067b319652/mechanicalsoup/browser.py#L36) in a library I'm using.


So, this is strictly a regression IMO: requests is supposed to handle this reasonably correctly. Clearly we regressed that somewhere down the line. I'll see if I can get a fix for this into 2.8.1.


The reason we regressed this is because urllib3's PyOpenSSL compatibility layer started using memoryviews. I think we can likely only reproduce this bug when using that, but we were definitely doing this wrong and should fix it.


Ok, proposed fix in #2821.


Nice; thanks for the quick fix.


I just got this error with requests 2.8.1. I can't be specific about what caused it, since I'm using a library that uses Tweepy, which uses requests. The test case posted by simon-weber above worked, but the actual POST didn't. I'm using Python 2.7 on Linux.

Downgrading to requests 2.7 fixed the issue.


@fitnr It's going to be really important to have a traceback/use-case. The problem will be mixing unicode/bytes, but we really need to see where the unicode is getting into our system. =)


@Lukasa I understand, but I don't have time to crack open Tweepy at the moment. The POST payload did contain unicode, I can tell you that.


The payload itself _should_ be fine, I'm more worried about the headers.


We received a report about this as well with requests 2.8.1


@jorilallo care to elaborate? Any details you can provide? Anything to help us help you?


@netanelkl reported this so it's probably better if he explains, I haven't had time to test this


I got this same issue when unicode strings were being passed as the header keys.
I'm now coding a test code to help you reproduce it.


This for example:
I stubbed this from the source lib that caused the problem.
Removing the single unicode header, solves the problem

``` Python
import hashlib
import hmac
import requests
from requests.auth import AuthBase
import time


class HMACAuth(AuthBase):

    def __call__(self, request):
        timestamp = str(int(time.time()))
        message = timestamp + request.method + request.path_url + (request.body or '')
        secret = "TEST"

        if not isinstance(message, bytes):
            message = message.encode()
        if not isinstance(secret, bytes):
            secret = secret.encode()

        signature = hmac.new(secret, message, hashlib.sha256).hexdigest()
        request.headers.update({
          'CB-VERSION': "",
          'CB-ACCESS-KEY': "",
          u'CB-ACCESS-SIGN': signature,
          'CB-ACCESS-TIMESTAMP': timestamp,
        })
        return request


session = requests.session()
session.auth = HMACAuth()
session.headers.update({'Accept': 'application/json',
                        'Content-Type': 'application/json',
                        'User-Agent': 'coinbase/python/2.0'})
a = session.get("https://www.google.com")
```


Ok, so this is a tricky one.

We validate header keys when they are passed in through the functional APIs at the top, and convert them to the native string on the platform. However, we do _not_ do that when you work directly with a `PreparedRequest`, which is what you're doing here. Neither should we: the purpose of the `PreparedRequest` is that we promise not to mess with it anymore.

In your case, you may find it useful to wrap each of the keys in `requests.utils.to_native_string`.


oh I see. Makes sense.
I guess the problem goes back to @jorilallo , this is a rather simple fix in the coinbase python library.
Thank you for checking this out!


This issue still occurs for me on 2.8.1 when posting a unicode string:

```
>>> import requests
>>> r = requests.post('https://example.com', data=u'unicode data')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 109, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 979, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 839, in _send_output
    self.send(message_body)
  File "/usr/lib/python2.7/httplib.py", line 811, in send
    self.sock.sendall(data)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 216, in sendall
    data = memoryview(data)
TypeError: cannot make memory view because object does not have the buffer interface
```

After fiddling a bit, this only happens if `ndg-httpsclient-0.4.0` is installed. When removed, the problem no longer occurs.

Of course sending byte strings in the post data also solves the problem.


It is a very bad idea to send unicode strings in body data. This is because the body data goes onto the wire as-is. The reason it appears to "work" in your example above is because of Python 2.7's loose handling of byte and unicode strings, but it's definitely wrong, and will explode if your unicode string contains any data outside of the default encoding on your system. For example, try changing your example to this:

``` python
>>> import requests
>>> r = requests.post('https://example.com', data=u'unicode data\ua000')
```

You'll find that this fails with a `UnicodeEncodeError`, as it should.

The rule here is: do not send unicode in the body.


@Lukasa It appears that example works with requests 2.7.0 and a default encoding of 'utf-8'. The issue we are encountering is that upgrading requests from 2.7.0 now breaks all requests that could have utf-8 data being sent in the data field, without prior warning. Would there be a way to avoid this breaking change?


@vinodc To be clear, the error you're encountering is a known bug in urllib3 that has been fixed (see shazow/urllib3#719). However, that fix will not be pulled into the released version of requests until urllib3 ships a new release and then requests ships 2.9.0. Until such time, you could apply the diff from the linked patch to your installation or, alternatively, simply encode the data yourself.

> It appears that example works with requests 2.7.0 and a default encoding of 'utf-8'.

The reason I encourage you to encode the data yourself is that the example only works _by sheer good luck_. There are many ways this could fail, and it could fail in other environments. Relying on Python 2.7's implicit encoding/decoding is extremely dangerous, and I strongly encourage you to resolve this problem.


@Lukasa Got it. Thanks for the detailed response.


Got an email about this today. Sounds like there's a demand for 2.9 :)


Yup. We're on track, but @shazow is taking a well-deserved birthday break so it'll be a little bit. 


Meow. What do we need from me? A urllib3 release?


Yarp. Let me know when you want to do it, and I can spend a few hours ahead of time looking at PRs and stuff you want to merge before a release. =)


Okely, thanks! Made an issue: shazow/urllib3#755