Repository: pypa/pip

Issue:
Not handling locales return codes correctly for LANG=C
* Pip version: pip 9.0.1 from /opt/local/lib/python2.7/site-packages (python 2.7)
* Python version: Python 2.7.12
* Operating System:
```
root@unknown:/export/home/root/opensource# cat /etc/release 
                       Solaris 10 10/09 s10x_u8wos_08a X86
           Copyright 2009 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 16 September 2009
root@unknown:/export/home/root/opensource# 
```


### Description:
Not allowing for environments without a language encoding, namely LANG=C
```
root@unknown:/export/home/root/opensource# set | grep LANG
LANG=C
LANGUAGE=C
```

getting the following error;
```
134 + pip install --egg -r requirements.txt
.
137 Exception:
138 Traceback (most recent call last):
139   File "/opt/local/lib/python2.7/site-packages/pip/basecommand.py", line 209, in main
140     status = self.run(options, args)
141   File "/opt/local/lib/python2.7/site-packages/pip/commands/install.py", line 287, in run
142     wheel_cache
143   File "/opt/local/lib/python2.7/site-packages/pip/basecommand.py", line 289, in populate_requirement_set
144     wheel_cache=wheel_cache):
145   File "/opt/local/lib/python2.7/site-packages/pip/req/req_file.py", line 84, in parse_requirements
146     filename, comes_from=comes_from, session=session
147   File "/opt/local/lib/python2.7/site-packages/pip/download.py", line 414, in get_file_content
148     content = auto_decode(f.read())
149   File "/opt/local/lib/python2.7/site-packages/pip/utils/encoding.py", line 31, in auto_decode
150     **return data.decode(locale.getpreferredencoding(False))**
151 **TypeError: decode() argument 1 must be string, not None**
```

Problem is due to locale.getpreferredencoding(False) returns 'None' for LANG=C

Quick test to show issue
```
root@unknown:/export/home/root/opensource# cat mytest.py 
import locale
myloc = locale.getlocale()
mypref = locale.getpreferredencoding(False)
print myloc
print mypref
print "done"
```
returns
```
(None, None)
None
done

```


Discussion:
Found on Solaris but should be repeatable on a linux box with LANG=C, etc
I have also hit this on OneFS with `LANG=C` and `LANG=`.

```
Exception:
Traceback (most recent call last):
  File "/root/venv/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/root/venv/lib/python2.7/site-packages/pip/commands/install.py", line 312, in run
    wheel_cache
  File "/root/venv/lib/python2.7/site-packages/pip/basecommand.py", line 295, in populate_requirement_set
    wheel_cache=wheel_cache):
  File "/root/venv/lib/python2.7/site-packages/pip/req/req_file.py", line 84, in parse_requirements
    filename, comes_from=comes_from, session=session
  File "/root/venv/lib/python2.7/site-packages/pip/download.py", line 422, in get_file_content
    content = auto_decode(f.read())
  File "/root/venv/lib/python2.7/site-packages/pip/utils/encoding.py", line 31, in auto_decode
    return data.decode(locale.getpreferredencoding(False))
TypeError: decode() argument 1 must be string, not None
```

Setting `LANG="en_US.UTF-8"` seems to fix it.

*edit*: FWIW, I am not able to repro on my Debian workstation just by setting `LANG`.
Maybe this is a bug with `locale.getpreferredencoding`?

```
(venv) dtucker-onefs-1# export LANG=C  # Same results with LANG=
(venv) dtucker-onefs-1# python                        
Python 2.7.10 (default, Apr  5 2017, 16:39:48) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd11
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import locale
>>> locale.getpreferredencoding(), sys.getfilesystemencoding(), sys.getdefaultencoding()
(None, 'US-ASCII', 'ascii')
>>> import io
>>> io.open('/ifs/README.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: bad argument type for built-in operation
```
```
(venv) dtucker-onefs-1# export LANG="en_US.UTF-8"
(venv) dtucker-onefs-1# python                   
Python 2.7.10 (default, Apr  5 2017, 16:39:48) 
[GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd11
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import locale
>>> locale.getpreferredencoding(), sys.getfilesystemencoding(), sys.getdefaultencoding()
('UTF-8', 'UTF-8', 'ascii')
>>> import io
>>> io.open('/ifs/README.txt')
<_io.TextIOWrapper name='/ifs/README.txt' encoding='UTF-8'>
```

Resolving patch:
diff --git a/pip/utils/encoding.py b/pip/utils/encoding.py
--- a/pip/utils/encoding.py
+++ b/pip/utils/encoding.py
@@ -1,6 +1,7 @@
 import codecs
 import locale
 import re
+import sys
 
 
 BOMS = [
@@ -28,4 +29,6 @@ def auto_decode(data):
         if line[0:1] == b'#' and ENCODING_RE.search(line):
             encoding = ENCODING_RE.search(line).groups()[0].decode('ascii')
             return data.decode(encoding)
-    return data.decode(locale.getpreferredencoding(False))
+    return data.decode(
+        locale.getpreferredencoding(False) or sys.getdefaultencoding(),
+    )


Test patch:
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -461,6 +461,16 @@ def test_auto_decode_pep263_headers(self):
         latin1_req = u'# coding=latin1\n# Pas trop de café'
         assert auto_decode(latin1_req.encode('latin1')) == latin1_req
 
+    def test_auto_decode_no_preferred_encoding(self):
+        om, em = Mock(), Mock()
+        om.return_value = 'ascii'
+        em.return_value = None
+        data = u'data'
+        with patch('sys.getdefaultencoding', om):
+            with patch('locale.getpreferredencoding', em):
+                ret = auto_decode(data.encode(sys.getdefaultencoding()))
+        assert ret == data
+
 
 class TestBuildDirectory(object):
     # No need to test symlinked directories on Windows