TypeError with pyOpenSSL, unicode headers, and python 2.x

```
>>> requests.get("https://google.com", headers={"X-Foo": u"foo"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.6/httplib.py", line 910, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.6/httplib.py", line 947, in _send_request
    self.endheaders()
  File "/usr/lib/python2.6/httplib.py", line 904, in endheaders
    self._send_output()
  File "/usr/lib/python2.6/httplib.py", line 776, in _send_output
    self.send(msg)
  File "/usr/lib/python2.6/httplib.py", line 755, in send
    self.sock.sendall(str)
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 353, in sendall
    return self.connection.sendall(data)
  File "/home/buildbot/joe-wip/virtualenv/lib/python2.6/site-packages/OpenSSL/SSL.py", line 969, in sendall
    raise TypeError("buf must be a byte string")
TypeError: buf must be a byte string
```


By comparison, both `requests.get("http://google.com", headers={"X-Foo": u"foo"})` and `requests.get("https://google.com", headers={"X-Foo": "foo"})` work.


We don't expect you to set unicode headers on Python 2. It's generally a bad idea anyway because the text encoding to use is really unclear: what bytes does that header turn into? The fact that it works in other requests is simple good luck, and you'll find that if you put non-ASCII characters into your unicode string it will also fail.

On Python 2 you should be encoding all your strings to bytestrings before passing them to requests. =)


@joeshaw do you mean to say this works without unicode headers with pyOpenSSL?


It would be nice if the validation happened at a higher level than the SSL layer.  It feels wrong that the error happens inconsistently across HTTP and HTTPs, in such a low-level.  (And I would assume utf-8 is a reasonable encoding for such things.)


@sigmavirus24 Both without unicode headers against HTTPS and with unicode headers against HTTP.  In my particular case the data I am sending is always ASCII, it just happens to be a unicode object because i got it from a different library that always returns unicode objects.


@joeshaw True, but as you've pointed out, it will break lots of currently working code. Even though that code is in error, I'm reluctant to break it because it does _happen_ to work. This is the reason Python 3 happened in the first place. =)

As for assuming UTF-8 is valid, you would be wrong. =) There is no clear ruling on the valid textual encoding for headers. Generally, ASCII is all you can _guarantee_ to be safe. Everything from then on is difficult to know. Better to make the user do this.


@joeshaw the fix is to call `string.encode` on each string that library gives you. You also seem to know the encoding (something requests should never guess at), so you can get the string from that library and then simply call `s.encode('utf-8')` on it.

As @Lukasa the only totally safe header encoding is ASCII. The next safest would be Latin-1 (ISO-8859-1). Other than that, there's not much we can do for you.


Ok, that's fair.  The inconsistency between SSL (pyOpenSSL, specifically, not the built-in SSL) and non-SSL is the most frustrating thing, and the thing that seems most like a bug to me.  This all started because a third-party service I was using switched to SNI.


Yeah, that inconsistency is a general pain in the neck. PyOpenSSL is currently looking to fix some of that up (see pyca/pyopenssl#15), so hopefully their 0.15 release will improve matters. If not, we might work on adding it to backports.ssl.