When path of requests contains non-ascii chars, it failed loading ca_certs

Tested with Python 3.4.3, requests2.6.0(Used by [wakatime](https://github.com/wakatime/wakatime)), windows 7
Also try with 2.7.0

```
Traceback (most recent call last):
  File "cli.py", line 18, in <module>
    sys.exit(wakatime.main(sys.argv))
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\base.py", line 460, in main

    if send_heartbeat(**kwargs):
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\base.py", line 377, in send_heartbeat
    proxies=proxies)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\sessions.py", line 507, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\sessions.py", line 464, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\adapters.py", line 370, in send
    timeout=timeout
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\connectionpool.py", line 341, in _make_request
    self._validate_conn(conn)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\connectionpool.py", line 762, in _validate_conn
    conn.connect()
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\connection.py", line 238, in connect
    ssl_version=resolved_ssl_version)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2015\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\util\ssl_.py", line 256, in ssl_wrap_socket
    context.load_verify_locations(ca_certs)
TypeError: cafile should be a valid filesystem path
```


OS language:zh-cn, default encoding: GBK


Can you show me some sample code please? I want to see if you're using a unicode string or a byte string.


Just an simple code

``` python
#!/usr/bin/env python
#fileencoding: utf-8

import sys
from packages import requests

r = requests.get('https://www.github.com/')
sys.stdout.write(r.text)

```

---

C:\Python34\python.exe --version
`Python 3.4.3`

---

C:\Users**\AppData\Roaming\Microsoft Corporation\Microsoft® Visual Studio® 2015\14.0.23107.0\wakatime-master\wakatime>C:\Python34\python.exe test.py

```
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    r = requests.get('https://www.github.com/')
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\api.py",line 69, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\api.py",line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\adapters.py", line 371, in send
    timeout=timeout
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\connectionpool.py", line 341, in _make_request
    self._validate_conn(conn)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\connectionpool.py", line 761, in _validate_conn
    conn.connect()
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\connection.py", line 238, in connect
    ssl_version=resolved_ssl_version)
  File "C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft\xae Visual Studio\xae 2**5\14.0.23107.0\wakatime-master\wakatime\packages\requests\packages\urllib3\util\ssl_.py", line 267, in ssl_wrap_socket
    context.load_verify_locations(ca_certs)
TypeError: cafile should be a valid filesystem path
```

"**" is my userName and be removed, "\xae" is "®" in the filesystem.

---

And when move folder `C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft® Visual Studio® 2015\14.0.23107.0\wakatime-master\wakatime` to `C:\Users\**\AppData\Roaming\Microsoft Corporation\wakatime-master\wakatime`(without non-ascii ®), it works well.


Ah, I see, the problem is that there's a non-ascii character in the location that requests is installed in. That makes sense.

So the path we generate is generated by doing `os.path.join(os.path.dirname(__file__), 'cacert.pem')`. Given that requests hasn't touched this at all, it looks like it's possible to get this wrong.

I'd like you to quickly `cd` into the directory that wakatime is in, and run the following in a Python interpreter and show me the output (censoring your username):

``` python
>>> import os
>>> import os.path
>>> os.getcwd()
>>> os.path.dirname(os.getcwd())
```

Right now this looks to me like the SSL library is not capable of handling Windows paths properly. @tiran?


```
C:\Users\**\AppData\Roaming\Microsoft Corporation\Microsoft® Visual Studio® 2015\14.0.23107.0\wakatime-master\wakatime>C:\Python34\python.exe
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import os.path
>>> os.getcwd()
'C:\\Users\\**\\AppData\\Roaming\\Microsoft Corporation\\Microsoft\xae Visual Studio\xae 2015\\14.0.23107.0\\wakatime-master\\wakatime'
>>> os.path.dirname(os.getcwd())
'C:\\Users\\**\\AppData\\Roaming\\Microsoft Corporation\\Microsoft\xae Visual Studio\xae 2015\\14.0.23107.0\\wakatime-master'
>>>
```


Right, so this error seems to be entirely in the stdlib. For some reason we're not able to pass a path that we got from the stdlib _to_ the stdlib. I'm hoping that when @tiran wakes up he'll confirm that I'm right, and then we can chase this up in the stdlib.


The SSL module uses PyUnicode_FSConverter() to convert the path from unicode to bytes using the FS default encoding. What's the output of 

```
>>> import os, sys
>>> sys.getfilesystemencoding()
>>> os.path.abspath(os.getcwd()).encode(sys.getfilesystemencoding())
```

on your system?

The problem might be another step further down the stack. OpenSSL accepts one char arguments. It has no wide char variants for e.g. file names. BIO_new_file() has some magic to detect UTF-8 encoded bytes. You could get lucky if you encode the file name from unicode to UTF-8 first.


Inside a Windows8 VM I was able to create a folder named `test®` and run some tests using kennethreitz/requests@3385c789991197d9b08da1de203615f6f40c2738.

```
C:\testr>\Python34\python.exe
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:43:06) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, os
>>> os.getcwd()
'C:\\test\xae'
>>> sys.getfilesystemencoding()
'mbcs'
>>> os.path.abspath(os.getcwd()).encode(sys.getfilesystemencoding())
b'C:\\test\xae'
>>>
```

However, I'm not able to reproduce the cafile `TypeError`. The request to github succeeds, but when printing the response content I get a `UnicodeEncodeError`:

```
C:\testr>\Python34\python.exe
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:43:06) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://www.github.com/')
>>> import sys
>>> sys.stdout.write(r.text)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python34\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2019' in position
10879: character maps to <undefined>
>>>
```

My first guess is we are using a different filesystem encoding.


Tested on another PC with Win10
C:\Users\*********\AppData\Roaming\Microsoft Corporation\Microsoft® Visual Studio® 2015\14.0.23107.0\wakatime-masteratime>"C:\Python34\python"

``` python
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys, locale
>>> sys.getfilesystemencoding()
'mbcs'
>>> sys.getdefaultencoding()
'utf-8'
>>> os.path.abspath(os.getcwd()).encode(sys.getfilesystemencoding())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'mbcs' codec can't encode characters in position 0--1: invalid character
>>> locale.getdefaultlocale()
('zh_CN', 'cp936')
>>> locale.getpreferredencoding()
'cp936'
>>> os.path.abspath(os.getcwd()).encode('cp936')#also known as gbk
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'gbk' codec can't encode character '\xae' in position 66: illegal multibyte sequence
```


Well, this is tricky. That's a real mess of encodings, none of which are going to play real nice together.


## I wonder if we should page Nick Coghlan and/or Steve Dower. I have no clue how we might handle this

Sent from my Android device with K-9 Mail. Please excuse my brevity.


Yeah, good idea: I paged Nick.


My first suggestion is that any experimentation with this be undertaken using either IPython Notebook or with https://pypi.python.org/pypi/win_unicode_console enabled. Otherwise you're going to run into confusingly unrelated issues related to the console's stdin and stdout handling that may obscure the underlying issues in interacting with OpenSSL (if you _do_ use win_unicode_console, feedback on the related CPython tracker issues would be appreciated - it would be nice to see that as the default console behaviour in 3.6).

My second suggestion would be to try _something else_ that uses OpenSSL on Windows (MinGW, perhaps?) and see if it can open that cert file in that location. If OpenSSL is still internally using the 8-bit mbcs Windows APIs rather than the UTF-16-LE ones, then it may be necessary to find a way load the cert bundle that lets Python code take care of the path handling.

In relation to that, the ssl module code in 3.4+ and 2.7.9+ to load certs from the Windows cert store may be relevant, although Christian Heimes & Steve Dower would be better folks to ask about that.


Oops, @tiran already confirmed that OpenSSL only accepts 8-bit paths, so there's no need to experiment to determine that.

The issues addressed by win_unicode_console are still a potentially confounding factor in some of the experiments above.

At this point, I'd say the suggestion of force encoding the cert path to UTF-8 (or UTF-8-BOM?) to try to trigger OpenSSL's encoding detection may be worth trying, but it will likely only work in this case if that actually prompts OpenSSL to switch to using the UTF-16-LE Windows APIs.


path with remote chars success with path.encode('utf8')

Test with an path that include ACSII + cp437??(remote) encoding chars(®) + cp936(local) encoding chars(中文)

```
C:\test®中文>C:\Python34\py.exe
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys, locale
>>> os.getcwd()
'C:\\test\xae中文'
>>> sys.getdefaultencoding()
'utf-8'
>>> sys.getfilesystemencoding()
'mbcs'
>>> locale.getdefaultlocale()
('zh_CN', 'cp936')
>>> locale.getpreferredencoding()
'cp936'
>>> import win_unicode_console
>>> win_unicode_console.enable()
>>> os.getcwd()
'C:\\test®中文'
```

put cacert.pem at 'C:\test®中文\'

```
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, ssl, sys
>>> origpath = os.getcwd()
>>> origpath
'C:\\Python34'
>>> relpath = 'cacert.pem'
>>> dir = 'C:\\test\xae中文\\'
>>> abspath = dir + relpath
>>>
>>> from ssl import SSLContext
>>> context = SSLContext(ssl.PROTOCOL_SSLv23)
>>> #failed with abspath
... context.load_verify_locations(abspath)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cafile should be a valid filesystem path
>>> #but work with encode('utf8')
... context.load_verify_locations(abspath.encode('utf8'))
>>> #also work with encode('utf8').decode('mbcs')
... context.load_verify_locations(abspath.encode('utf8').decode('mbcs'))
>>> abspath.encode('utf8').decode('mbcs')
'C:\\test庐涓\ue15f枃\\'
>>> #also work with relpath if relpath is ASCII, but os.chdir does not need encode??
... os.chdir(dir)
>>> context.load_verify_locations(relpath)
```

And if there is no 'C:\test®中文\cacert.pem', it will try loading'C:\test庐涓\ue15f枃\cacert.pem'(tested with monitor)
For path with only ASCII and local encoding chars, it also works without encode('utf8'), and seems do not make an fallback


So, is the suggested workaround that we should consider paths unicode strings? Is that valid? Can we always assume that `sys.getfilesystemencoding()` will be right?


We also have to remember Python2.
`str` in Python3 is unicode, so calling `.encode('utf-8')` on a `str` in Python3 works, but turns that `str` into `bytes`.
`str` in Python2 is a [string literal](https://docs.python.org/2/reference/lexical_analysis.html#strings) and needs to be decoded into unicode before it can be encoded. Running those tests in Python2 should raise a `UnicodeDecodeError` when calling `.encode('utf-8')`.

To support Python2, we need to:
- detect if running in Python2 from `is_py2` in [requests/compat.py](https://github.com/kennethreitz/requests/blob/master/requests/compat.py#L19)
- if is_py2, decode the `str` using the correct character encoding
- encode as unicode with `.encode('utf-8')`

I'm not sure the correct character encoding to use when decoding `str`, does anyone know? Should be one from `sys.getdefaultencoding()`, `sys.getfilesystemencoding()`, or `locale.getdefaultlocale()`.


It seems python2 can't recognize path with non-local encoding chars and replace them with '?', which can be recovered.
Tested with Portable Python 2.7.3.2
`Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32`


@yyjdelete we have a solution for Python3.4 (always encode as utf-8) but do we have a working solution for Python2? Can you run these tests on Python2:

``` python
import ssl, sys
from ssl import SSLContext

context = SSLContext(ssl.PROTOCOL_SSLv23)
context.load_verify_locations(abspath)
context.load_verify_locations(abspath.decode(sys.getfilesystemencoding()))
context.load_verify_locations(abspath.decode(sys.getfilesystemencoding()).encode('utf8'))
```


I made an mistake. In python2, `os.getcwdu()` and unicode string should be used instead of normal version, so we can still get the path.

@alanhamlett 
It's hard to test this case, for old python which doesn't has `ssl.SSLContext`(like python 2.7), wrapper in `urllib3/util/ssl_.py` is used instead, which doesn't crash with `load_verify_locations`

---

BTW: Is `os.path.supports_unicode_filenames` should be considered for the case?

Sorry for my poor English, and I'm not an python programmer.


Python 2.7.9+ includes the newer SSL module backported from Python 3.4, and may hit the same problem as the Python 3 version.


I'm not sure if it really works well with python2 without newer SSL module, because I didn't find an way to run .py with python2 at the path, for that python2 seems not use unicode to load .py and can not recognize the file with remote encoding.
Maybe I will do some test in interpreter later.

---

Tested(NOTE: in fact Python2 not work with .py, only some test with interpreter)
(Portable )Python 2.7.3.2(without newer SSL module): Failed, only 'ascii' is accepted.
Python 2.7.10: work with local encoding/unicode(the same as 3.4)


Ah, OK - Python 2 not running at all from that path is actually what I expected, but I interpreted some of your earlier comments as indicating it was working.

There are all sorts of "ASCII only" assumptions in the Python 2 import machinery and startup code that were only addressed properly through the more pervasive adoption of Unicode in Python 3.


@yyjdelete what is the value returned from `os.path.supports_unicode_filenames` on your machine?


@alanhamlett
`os.path.supports_unicode_filenames` is `True` in both  py2 and py3


This overwritten `where` function got rid of the `TypeError: cafile should be a valid filesystem path` exception for me:

https://github.com/wakatime/wakatime/blob/bba2015d9b6797f8eca94fef6dbd964c27734891/wakatime/packages/requests/certs.py#L25

Tested using a Windows 8.1 virtual machine with Simpified Chinese language. Was able to reproduce the `TypeError` exception only using Python 3.x, when testing with Python 2.x the original code worked.


Reproduced that problem on Windows 8.1 x64 with Python 3.5.1.

I had a pem file at the following path: `C:\Users\غازي\AppData\Local\Temp\_غازي_70e5wbxo\cacert.pem`.

```
c = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)

c.load_verify_locations(cafile=r"C:\Users\غازي\AppData\Local\Temp\_غازي_70e5wbxo\cacert.pem")
> TypeError: cafile should be a valid filesystem path

c.load_verify_locations(cafile=r"C:\Users\غازي\AppData\Local\Temp\_غازي_70e5wbxo\cacert.pem".encode(sys.getfilesystemencoding()))
> UnicodeEncodeError: 'mbcs' codec can't encode characters in positions 0--1: invalid character

c.load_verify_locations(cafile=r"C:\Users\غازي\AppData\Local\Temp\_غازي_70e5wbxo\cacert.pem".encode('utf-8'))
> ok
```


Closing due to inactivity.