Repository: spack/spack

Issue:
spack compiler find adds python unicode strings
Just noticed this interesting behavior. I deleted my `compilers.yaml` and `spack compiler find` created a new one that looks like:
```yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: sierra
    paths:
      cc: !!python/unicode '/Library/Developer/CommandLineTools/usr/bin/clang'
      cxx: !!python/unicode '/Library/Developer/CommandLineTools/usr/bin/clang++'
      f77: /usr/local/bin/gfortran
      fc: /usr/local/bin/gfortran
    spec: clang@8.1.0-apple
    target: x86_64
```
I'm going to investigate why it adds these `!!python/unicode` strings. Just wanted to report it so I can keep track of the bug.
Make Executable return str instead of unicode.
If I understood it correctly, the idea of calling decode('utf-8') was to get strings instead of binaries, which Popen.communicate() returns by default in Python3. Unfortunately, this leads to a problem described in  #5032. Another way to tell Popen.communicate() that we want to get strings is to set universal_newlines=True. After this change Executable returns <type 'str'> with Python2 and <class 'str'> with Python3.

Fixes #5032 
Fixes #4339 
Fixes #3842
Closes #4355


Discussion:
Having a little trouble tracking this one down. Note that this bug doesn't seem to affect Spack when run with Python 3, only with Python 2.
In my case (on linux, w/ python 2.7.10) ```spack compiler find``` does not add these tags

My guess would be that on sierra the file paths are read as unicode strings and then persist as such until written out by spack_yaml (which decides to preserve that in the representation)

I think this issue could be resolved by controlling how spack_yaml writes out strings. Is this creating problems? I could start working on this Monday
It isn't causing any problems, just looks sketchy. No rush.
Is this related to #4339 and #4355 by any chance?
TIL universal_newlines. Works for me but we upgraded to 2.7.9 since I submitted the PR so not entirely sure if it fixes those problems but I think it's good to go!
@becker33 @lee218llnl Can you also look at this as an alternative to #4355?
@adamjstewart, yes, this solves #4339. My first idea for the fix was like in #4355 (although, the condition there should be not version 2.7.9 but 2.7.3, according to the [documentation](https://docs.python.org/2/library/shlex.html) at least). I didn't like it because I thought that I would have to put that conditional conversion before each shlex usage or to write a wrapper for shlex to avoid code duplication. Now I see that shlex is used only once in the real code (other usages are for testing if I am not mistaken), so maybe that's not so bad.
Whoa, this solves #3842 too! I was wondering where all those unicode strings were coming from. Thanks!
Just a note from the [docs](https://docs.python.org/2/library/subprocess.html#subprocess.Popen):

> **Note:** This feature is only available if Python is built with universal newline support (the default). Also, the newlines attribute of the file objects `stdout`, `stdin` and `stderr` are not updated by the communicate() method.

I would be interested to see how common Python built without universal newline support is. I would also like to know whether a lack of support will cause this to crash completely, or just go back to the old behavior of returning unicode strings.

Either way, I am in favor of merging this and seeing if anyone reports any problems. It certainly wouldn't be hard to back out such a small change later if it causes us headaches.
I'll take a look at this (and probably merge it) tomorrow unless you'd like to @adamjstewart 
@scheibelp It's all you!
@adamjstewart starting [Python 2.4.0](https://hg.python.org/cpython/raw-file/v2.4/Misc/NEWS):

> Universal newline support can no longer be disabled (see PEP 11).

Resolving patch:
diff --git a/lib/spack/spack/util/executable.py b/lib/spack/spack/util/executable.py
--- a/lib/spack/spack/util/executable.py
+++ b/lib/spack/spack/util/executable.py
@@ -26,6 +26,7 @@
 import re
 import subprocess
 from six import string_types
+import sys
 
 import llnl.util.tty as tty
 import spack
@@ -184,9 +185,9 @@ def streamify(arg, mode):
             if output is str or error is str:
                 result = ''
                 if output is str:
-                    result += out.decode('utf-8')
+                    result += to_str(out)
                 if error is str:
-                    result += err.decode('utf-8')
+                    result += to_str(err)
                 return result
 
         except OSError as e:
@@ -223,6 +224,20 @@ def __str__(self):
         return ' '.join(self.exe)
 
 
+def to_str(content):
+    """Produce a str type from the content of a process stream obtained with
+       Popen.communicate.
+    """
+    # Prior to python3, Popen.communicate returns a str type. For python3 it
+    # returns a bytes type. In the case of python3 we decode the
+    # byte string to produce a str type. This will generate junk if the
+    # encoding is not UTF-8 (which includes ASCII).
+    if sys.version_info < (3, 0, 0):
+        return content
+    else:
+        return content.decode('utf-8')
+
+
 def which(*args, **kwargs):
     """Finds an executable in the path like command-line which.
 


Test patch: