Repository: spack/spack

Issue:
"spack compilers" abort - fedora26 gcc-7.2.1-2 compiler present
Hi,
I was trying to get spack going on another system and have run into this abort.
I just downloaded a fresh copy of the devel branch of spack from 
[jeg@localhost packages]$ git remote -v
origin	https://github.com/spack/spack.git (fetch)
origin	https://github.com/spack/spack.git (push)
upstream	https://github.com/spack/spack.git (fetch)
upstream	https://github.com/spack/spack.git (push)

[jeg@localhost packages]$ spack find
==> 0 installed packages.

[jeg@localhost packages]$ spack compilers
==> Available compilers
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jeg/jgalarowicz_spack/spack/lib/spack/spack/util/multiproc.py", line 37, in fun
    pipe.send(f(x))
  File "/home/jeg/jgalarowicz_spack/spack/lib/spack/spack/architecture.py", line 285, in <lambda>
    self.find_compiler(cmp_cls, *filtered_path),
  File "/home/jeg/jgalarowicz_spack/spack/lib/spack/spack/architecture.py", line 324, in find_compiler
    spec = spack.spec.CompilerSpec(cmp_cls.name, ver)
  File "/home/jeg/jgalarowicz_spack/spack/lib/spack/spack/spec.py", line 459, in __init__
    self.versions.add(ver(version))
  File "/home/jeg/jgalarowicz_spack/spack/lib/spack/spack/version.py", line 870, in ver
    return _string_to_version(obj)
  File "/home/jeg/jgalarowicz_spack/spack/lib/spack/spack/version.py", line 860, in _string_to_version
    return Version(string)
  File "/home/jeg/jgalarowicz_spack/spack/lib/spack/spack/version.py", line 117, in __init__
    raise ValueError("Bad characters in version string: %s" % string)
ValueError: Bad characters in version string: 
^C
==> Error: Keyboard interrupt.


[jeg@localhost packages]$ which gcc
/usr/bin/gcc


[jeg@localhost packages]$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC) 


[jeg@localhost packages]$ gcc --version
gcc (GCC) 7.2.1 20170915 (Red Hat 7.2.1-2)
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[jeg@localhost packages]$ 



Discussion:
What do `gcc -dumpversion` and `gcc -dumpfullversion` output?
Hi Todd,

jeg@localhost ~]$ gcc -dumpversion
7
[jeg@localhost ~]$ gcc -dumpfullversion
7.2.1

Thanks!
I upgraded to Fedora 27 with gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC) 
uname -a: Linux localhost.localdomain 4.13.13-300.fc27.x86_64 #1 SMP Wed Nov 15 15:47:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Still get the same traceback and failure.
spack compiler find
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jeg/new_spack/spack/lib/spack/spack/util/multiproc.py", line 37, in fun
    pipe.send(f(x))
  File "/home/jeg/new_spack/spack/lib/spack/spack/architecture.py", line 285, in <lambda>
    self.find_compiler(cmp_cls, *filtered_path),
  File "/home/jeg/new_spack/spack/lib/spack/spack/architecture.py", line 324, in find_compiler
    spec = spack.spec.CompilerSpec(cmp_cls.name, ver)
  File "/home/jeg/new_spack/spack/lib/spack/spack/spec.py", line 459, in __init__
    self.versions.add(ver(version))
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 870, in ver
    return _string_to_version(obj)
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 860, in _string_to_version
    return Version(string)
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 117, in __init__
    raise ValueError("Bad characters in version string: %s" % string)
ValueError: Bad characters in version string: 
^C
==> Error: Keyboard interrupt.


@scheibelp @becker33 - Could one of you look into this?
Additional information:

[jeg@vpna-c-user-192-168-244-79 spack]$ python --version
Python 2.7.14


spack -d install openspeedshop
==> Reading config file /home/jeg/new_spack/spack/etc/spack/defaults/packages.yaml
==> READ LOCK: /home/jeg/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquiring]
==> READ LOCK: /home/jeg/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquired]
==> READ LOCK: /home/jeg/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Released]
==> '/usr/bin/gcc49' '--version'
==> '/usr/bin/gcc' '--version'
==> '/usr/bin/gcc' '-dumpversion'
==> '/usr/bin/cc' '-V'
==> '/usr/bin/gcc' '-dumpfullversion'
==> '/usr/bin/gcc49' '-dumpversion'
==> '/usr/bin/gfortran' '--version'
==> Couldn't get version for compiler /usr/bin/cc
  Command exited with status 1:
    '/usr/bin/cc' '-V'
==> '/usr/bin/gfortran' '-dumpversion'
==> '/usr/bin/gfortran' '--version'
==> '/usr/bin/g++' '--version'
==> '/usr/bin/g++' '-dumpversion'
==> '/usr/bin/g++' '-dumpfullversion'
==> '/usr/bin/gfortran' '-dumpversion'
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jeg/new_spack/spack/lib/spack/spack/util/multiproc.py", line 37, in fun
    pipe.send(f(x))
  File "/home/jeg/new_spack/spack/lib/spack/spack/architecture.py", line 285, in <lambda>
    self.find_compiler(cmp_cls, *filtered_path),
  File "/home/jeg/new_spack/spack/lib/spack/spack/architecture.py", line 324, in find_compiler
    spec = spack.spec.CompilerSpec(cmp_cls.name, ver)
  File "/home/jeg/new_spack/spack/lib/spack/spack/spec.py", line 459, in __init__
    self.versions.add(ver(version))
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 870, in ver
    return _string_to_version(obj)
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 860, in _string_to_version
    return Version(string)
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 117, in __init__
    raise ValueError("Bad characters in version string: %s" % string)
ValueError: Bad characters in version string: 

[jeg@vpna-c-user-192-168-244-79 spack]$ which gcc
/usr/bin/gcc
[jeg@vpna-c-user-192-168-244-79 spack]$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,objc,obj-c++,fortran,ada,go,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 7.2.1 20170915 (Red Hat 7.2.1-2) (GCC) 


@mplegendre @becker33 @jgalarowicz I have self-assigned this and anticipate being able to look at it by the end of 12/12 (tomorrow)
Regarding

```
raise ValueError("Bad characters in version string: %s" % string)
ValueError: Bad characters in version string:
^C
==> Error: Keyboard interrupt.
```

To confirm: this hung for you for some length of time?

I'm having trouble replicating this but I have a suspicion of what the error is and I've created a debug patch. Could you apply the following (unzip and `git apply`, assuming you don't have any current modifications in your git repo and reverting this is easy) and then run `spack -d compiler find` and tell me if you get any output like `[6200] Missing version:`?

[debug_6200.zip](https://github.com/spack/spack/files/1554013/debug_6200.zip)

@scheibelp 

Hi - here is what I'm seeing after adding your patch:

[jeg@localhost spack]$ patch -p1 < debug_6200.patch 
patching file lib/spack/spack/compiler.py
patching file lib/spack/spack/version.py
[jeg@localhost spack]$ spack -d compiler find
==> Reading config file /home/jeg/new_spack/spack/etc/spack/defaults/modules.yaml
==> '/usr/bin/cc' '-V'
==> '/usr/bin/gfortran' '-dumpversion'
==> '/usr/bin/gfortran' '--version'
==> '/usr/bin/g++' '--version'
==> '/usr/bin/gfortran' '--version'
==> '/usr/bin/gcc' '--version'
==> '/usr/bin/gcc49' '--version'
==> '/usr/bin/gfortran' '-dumpversion'
==> '/usr/bin/g++' '-dumpversion'
==> '/usr/bin/g++' '-dumpfullversion'
==> '/usr/bin/gcc' '-dumpversion'
==> Couldn't get version for compiler /usr/bin/cc
  Command exited with status 1:
    '/usr/bin/cc' '-V'
==> '/usr/bin/gcc' '-dumpfullversion'
==> '/usr/bin/gcc49' '-dumpversion'
==> [6200] Missing version: /usr/bin/gcc49
==> [6200] Version provided: 7.2.1
==> [6200] Version provided: 
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jeg/new_spack/spack/lib/spack/spack/util/multiproc.py", line 37, in fun
    pipe.send(f(x))
  File "/home/jeg/new_spack/spack/lib/spack/spack/architecture.py", line 285, in <lambda>
    self.find_compiler(cmp_cls, *filtered_path),
  File "/home/jeg/new_spack/spack/lib/spack/spack/architecture.py", line 324, in find_compiler
    spec = spack.spec.CompilerSpec(cmp_cls.name, ver)
  File "/home/jeg/new_spack/spack/lib/spack/spack/spec.py", line 459, in __init__
    self.versions.add(ver(version))
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 872, in ver
    return _string_to_version(obj)
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 861, in _string_to_version
    return Version(string)
  File "/home/jeg/new_spack/spack/lib/spack/spack/version.py", line 118, in __init__
    raise ValueError("Bad characters in version string: %s" % string)
ValueError: Bad characters in version string: 

< here it hangs >
Thanks for running that!

What does `/usr/bin/gcc49 -dumpversion` output?
@scheibelp 
Looks like it gives back nothing:

[jeg@localhost spack]$ /usr/bin/gcc49 -dumpversion
[jeg@localhost spack]$ 

@scheibelp 
More info:
[jeg@localhost spack]$ /usr/bin/gcc49 -dumpversion
[jeg@localhost spack]$ /usr/bin/gcc49
[jeg@localhost spack]$ /usr/bin/gcc49 -v
[jeg@localhost spack]$ file /usr/bin/gcc49
/usr/bin/gcc49: Bourne-Again shell script, ASCII text executable
[jeg@localhost spack]$ wc /usr/bin/gcc49
 24  35 453 /usr/bin/gcc49
[jeg@localhost spack]$ cat /usr/bin/gcc49
#! /bin/bash
#set -x

gver=4.9.3
gcc_target_platform=$(uname -m)-fedoraunited-linux-gnu

if [ `getconf LONG_BIT` = "64" ]
then
lib=lib64
else
lib=lib
fi


export CC=/opt/gcc-$gver/bin/gcc
export CXX=/opt/gcc-$gver/bin/g++
export CPP=/opt/gcc-$gver/bin/cpp
export LD=/opt/gcc-$gver/bin/gcc
#

# gcc49
export LDFLAGS="-L/opt/gcc-$gver/$lib/gcc/$gcc_target_platform/$lib/" 
export CPPFLAGS="-I/opt/gcc-$gver/$lib/gcc/$gcc_target_platform/$gver/include/"
#
[jeg@localhost spack]$ 
@scheibelp 
The last output was messed up here is a cat -n /usr/bin/gcc49
     1	#! /bin/bash
     2	#set -x
     3	
     4	gver=4.9.3
     5	gcc_target_platform=$(uname -m)-fedoraunited-linux-gnu
     6	
     7	if [ `getconf LONG_BIT` = "64" ]
     8	then
     9	lib=lib64
    10	else
    11	lib=lib
    12	fi
    13	
    14	
    15	export CC=/opt/gcc-$gver/bin/gcc
    16	export CXX=/opt/gcc-$gver/bin/g++
    17	export CPP=/opt/gcc-$gver/bin/cpp
    18	export LD=/opt/gcc-$gver/bin/gcc
    19	#
    20	
    21	# gcc49
    22	export LDFLAGS="-L/opt/gcc-$gver/$lib/gcc/$gcc_target_platform/$lib/" 
    23	export CPPFLAGS="-I/opt/gcc-$gver/$lib/gcc/$gcc_target_platform/$gver/include/"
    24	#


Resolving patch:
diff --git a/lib/spack/spack/compiler.py b/lib/spack/spack/compiler.py
--- a/lib/spack/spack/compiler.py
+++ b/lib/spack/spack/compiler.py
@@ -278,27 +278,11 @@ def _find_matches_in_path(cls, compiler_names, detect_version, *path):
 
                     match = re.match(regex, exe)
                     if match:
-                        key = (full_path,) + match.groups()
+                        key = (full_path,) + match.groups() + (detect_version,)
                         checks.append(key)
 
-        def check(key):
-            try:
-                full_path, prefix, suffix = key
-                version = detect_version(full_path)
-                return (version, prefix, suffix, full_path)
-            except ProcessError as e:
-                tty.debug(
-                    "Couldn't get version for compiler %s" % full_path, e)
-                return None
-            except Exception as e:
-                # Catching "Exception" here is fine because it just
-                # means something went wrong running a candidate executable.
-                tty.debug("Error while executing candidate compiler %s"
-                          % full_path,
-                          "%s: %s" % (e.__class__.__name__, e))
-                return None
-
-        successful = [k for k in parmap(check, checks) if k is not None]
+        successful = [k for k in parmap(_get_versioned_tuple, checks)
+                      if k is not None]
 
         # The 'successful' list is ordered like the input paths.
         # Reverse it here so that the dict creation (last insert wins)
@@ -322,6 +306,28 @@ def __str__(self):
                 str(self.operating_system)))))
 
 
+def _get_versioned_tuple(compiler_check_tuple):
+    full_path, prefix, suffix, detect_version = compiler_check_tuple
+    try:
+        version = detect_version(full_path)
+        if (not version) or (not str(version).strip()):
+            tty.debug(
+                "Couldn't get version for compiler %s" % full_path)
+            return None
+        return (version, prefix, suffix, full_path)
+    except ProcessError as e:
+        tty.debug(
+            "Couldn't get version for compiler %s" % full_path, e)
+        return None
+    except Exception as e:
+        # Catching "Exception" here is fine because it just
+        # means something went wrong running a candidate executable.
+        tty.debug("Error while executing candidate compiler %s"
+                  % full_path,
+                  "%s: %s" % (e.__class__.__name__, e))
+        return None
+
+
 class CompilerAccessError(spack.error.SpackError):
 
     def __init__(self, path):


Test patch:
diff --git a/lib/spack/spack/test/compilers.py b/lib/spack/spack/test/compilers.py
--- a/lib/spack/spack/test/compilers.py
+++ b/lib/spack/spack/test/compilers.py
@@ -27,6 +27,7 @@
 
 import spack.spec
 import spack.compilers as compilers
+from spack.compiler import _get_versioned_tuple
 
 
 @pytest.mark.usefixtures('config')
@@ -49,6 +50,19 @@ def test_all_compilers(self):
         assert len(filtered) == 1
 
 
+def test_version_detection_is_empty():
+    no_version = lambda x: None
+    compiler_check_tuple = ('/usr/bin/gcc', '', r'\d\d', no_version)
+    assert not _get_versioned_tuple(compiler_check_tuple)
+
+
+def test_version_detection_is_successful():
+    version = lambda x: '4.9'
+    compiler_check_tuple = ('/usr/bin/gcc', '', r'\d\d', version)
+    assert _get_versioned_tuple(compiler_check_tuple) == (
+        '4.9', '', r'\d\d', '/usr/bin/gcc')
+
+
 def test_compiler_flags_from_config_are_grouped():
     compiler_entry = {
         'spec': 'intel@17.0.2',