Repository: conda/conda-build

Issue:
git source fails when the changeset's commit message is non-ASCII
This is a traceback we got on a Jenkins builder (Numba):

```
Traceback (most recent call last):
  File "/var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/bin/conda-build", line 5, in <module>
    sys.exit(main())
  File "/var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 118, in main
    args_func(args, p)
  File "/var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 329, in args_func
    args.func(args, p)
  File "/var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 288, in execute
    channel_urls=channel_urls, override_channels=args.override_channels)
  File "/var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/lib/python2.7/site-packages/conda_build/build.py", line 347, in build
    source.provide(m.path, m.get_section('source'))
  File "/var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/lib/python2.7/site-packages/conda_build/source.py", line 251, in provide
    git_source(meta, recipe_dir)
  File "/var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/lib/python2.7/site-packages/conda_build/source.py", line 135, in git_source
    git_info()
  File "/var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/lib/python2.7/site-packages/conda_build/source.py", line 163, in git_info
    print(stdout + u'\n')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 296: ordinal not in range(128)
```

You can see that changeset in the Numba git repository:

```
$ git checkout 6efdca690a4e13d10cc18e2c172c0b2050159c95
[...]
$ git log -n1
commit 6efdca690a4e13d10cc18e2c172c0b2050159c95
Merge: 49d01f4 e9e1a70
Author: Siu Kwan Lam <michael.lam.sk@gmail.com>
Date:   Tue Jun 16 13:20:44 2015 -0500

    Merge pull request #1231 from pitrou/cuda_del_warnings

    Fix #1164: avoid warnings at shutdown when finalizing a CUDA context …
```

Note the unicode ellpsis (`…` not `...`) in the commit message.



Discussion:
What is your `conda info`?

From the jenkins log:

```
++ conda info
Current conda install:

             platform : linux-32
        conda version : 3.13.0
  conda-build version : 1.13.0
       python version : 2.7.10.final.0
     requests version : 2.7.0
     root environment : /var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda  (writable)
  default environment : /var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda
     envs directories : /var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/envs
        package cache : /var/jenkins/workspace/Numba-Builder-Linux32@2/CONDA_NPY/19/CONDA_PY/34/label/CentOS5x32b/miniconda/pkgs
         channel URLs : https://repo.continuum.io/pkgs/free/linux-32/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/linux-32/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : None
    is foreign system : False
```

This log should be available at http://jenkins:8080/job/Numba-Builder-Linux32/CONDA_NPY=19,CONDA_PY=34,label=CentOS5x32b/31/console from the continuum internal network.

Note that you can reproduce locally if you force the C locale when calling conda-build:

```
$ LANG=C conda build --no-test buildscripts/condarecipe.jenkins/
[...]
  File "/home/antoine/miniconda3/lib/python3.4/site-packages/conda_build/source.py", line 163, in git_info
    print(stdout + u'\n')
UnicodeEncodeError: 'ascii' codec can't encode character '\u2026' in position 296: ordinal not in range(128)
```

You will need to apply the following patch to build the offending changeset:;

```
diff --git a/buildscripts/condarecipe.jenkins/meta.yaml b/buildscripts/condarecipe.jenkins/meta.yaml
index 59a94aa..6e24f41 100644
--- a/buildscripts/condarecipe.jenkins/meta.yaml
+++ b/buildscripts/condarecipe.jenkins/meta.yaml
@@ -4,6 +4,7 @@ package:

 source:
   git_url: git@github.com:numba/numba.git
+  git_tag: 6efdca690a4e13d10cc18e2c172c0b2050159c95

 build:
   number: {{ environ.get('GIT_DESCRIBE_NUMBER', 0) }}
```

Confirmed with conda-build-1.14.0.

What are we doing wrong https://github.com/conda/conda-build/blob/f5a62cf40e924cf8e1357f625175a7b074d7b53b/conda_build/source.py#L160? Should I nix the Unicode literals? 

Instead of:

```
        stdout = stdout.decode('utf-8')
        stderr = stderr.decode('utf-8')
```

I would suggest something like:

```
        encoding = locale.getpreferredencoding()
        stdout = stdout.decode(encoding, 'ignore')
        stderr = stderr.decode(encoding, 'ignore')
```

Wouldn't that make it still fail if the locale doesn't support the characters? According to ftp://www.kernel.org/pub/software/scm/git/docs/git-log.html (search for `--encoding`), `git log` defaults to `utf-8`. If that's true, wouldn't that code be wrong in any case where the `getpreferredencoding` is not utf-8. 

The "ignore" error handler is there precisely to avoid that issue - undecodable characters are simply skipped. For instance:

```
$ python -c "s = 'héhé'.decode('utf8'); print(s)"
héhé
$ LANG=C python -c "s = 'héhé'.decode('utf8'); print(s)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1: ordinal not in range(128)
$ python -c "import locale; s = 'héhé'.decode(locale.getpreferredencoding(), 'ignore'); print(s)"
héhé
$ LANG=C python -c "import locale; s = 'héhé'.decode(locale.getpreferredencoding(), 'ignore'); print(s)"
hh
```

(that's Python 2, by the way)

For the record, same example with Python 3:

```
$ python3 -c "s = b'h\xc3\xa9h\xc3\xa9'.decode('utf8'); print(s)"
héhé
$ LANG=C python3 -c "s = b'h\xc3\xa9h\xc3\xa9'.decode('utf8'); print(s)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128)
$ python3 -c "import locale; s = b'h\xc3\xa9h\xc3\xa9'.decode(locale.getpreferredencoding(), 'ignore'); print(s)"
héhé
$ LANG=C python3 -c "import locale; s = b'h\xc3\xa9h\xc3\xa9'.decode(locale.getpreferredencoding(), 'ignore'); print(s)"
hh
```

OK. I knew I remembered seeing issues with that function before so I searched and found https://github.com/conda/conda-build/issues/360. So I should probably incorporate that in the fix as well. 


Resolving patch:
diff --git a/conda_build/source.py b/conda_build/source.py
--- a/conda_build/source.py
+++ b/conda_build/source.py
@@ -5,6 +5,7 @@
 from os.path import join, isdir, isfile, abspath, expanduser
 from shutil import copytree, copy2
 from subprocess import check_call, Popen, PIPE
+import locale
 
 from conda.fetch import download
 from conda.utils import hashsum_file
@@ -148,8 +149,9 @@ def git_info(fo=None):
                 ('git status', True)]:
         p = Popen(cmd.split(), stdout=PIPE, stderr=PIPE, cwd=WORK_DIR, env=env)
         stdout, stderr = p.communicate()
-        stdout = stdout.decode('utf-8')
-        stderr = stderr.decode('utf-8')
+        encoding = locale.getpreferredencoding() or 'utf-8'
+        stdout = stdout.decode(encoding, 'ignore')
+        stderr = stderr.decode(encoding, 'ignore')
         if check_error and stderr and stderr.strip():
             raise Exception("git error: %s" % stderr)
         if fo:


Test patch: