Repository: spack/spack

Issue:
spack compiler find does not find FE compilers on Cray
I'm seeing a problem on Cray where `spack compiler find` does not
identify the FE compiler modules, at least for GNU.

Cray has several modules for compilers and separate FE and BE arch
types.  For GNU at least, the raw modules are front-end compilers.
It's only the Cray wrappers (cc, CC, ftn) that add the back-end
specific options.  I'm not sure how Intel, PGI or Clang handle this.

For example, on theta at ANL, I switch to `PrgEnv-gnu` and load module
`gcc/7.3.0`.  The FE C compiler is `/opt/gcc/7.3.0/bin/gcc` and the BE
compiler is the `cc` wrapper.

But spack fails to see `/opt/gcc/7.3.0/bin/gcc` as a FE compiler.

For example, again on theta at ANL, I remove my `compilers.yaml` file
and run `spack compiler find` from scratch.  I get these compilers.

```
$ spack compiler list
==> Available compilers
-- cce cnl6-any -------------------------------------------------
cce@8.7.7  cce@8.7.3  cce@8.6.5  cce@8.6.3  cce@8.6.1  cce@8.5.8
cce@8.7.6  cce@8.7.0  cce@8.6.4  cce@8.6.2  cce@8.6.0  cce@8.5.7

-- gcc cnl6-any -------------------------------------------------
gcc@8.2.0  gcc@7.3.0  gcc@7.2.0  gcc@7.1.0  gcc@6.3.0  gcc@6.1.0  gcc@5.3.0  gcc@4.9.3

-- gcc sles12-x86_64 --------------------------------------------
gcc@4.8

-- intel cnl6-any -----------------------------------------------
intel@19.0.3.199  intel@18.0.5.274  intel@18.0.2.199  intel@17.0.2.174  intel@16.0.3.210
intel@19.0.2.187  intel@18.0.3.222  intel@18.0.0.128  intel@17.0.1.132
intel@19.0.1.144  intel@18.0.2.199  intel@17.0.4.196  intel@17.0.0.098
```

Note: the only FE (sles12-x86_64) GNU compiler it finds is gcc 4.8.5
from `/usr/bin/gcc`.  And further attempts to find the compiler don't
work.  I have to add the full compiler: entry in compilers.yaml
manually.

```
$ module list
Currently Loaded Modulefiles:
  2) gcc/7.3.0
 23) PrgEnv-gnu/6.0.4
 ...

$ type -a gcc
gcc is /opt/gcc/7.3.0/bin/gcc
gcc is /usr/bin/gcc
gcc is /usr/bin/X11/gcc

$ /opt/gcc/7.3.0/bin/gcc --version
gcc (GCC) 7.3.0 20180125 (Cray Inc.)
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ ./spack compiler find /opt/gcc/7.3.0 
==> Found no new compilers
==> Compilers are defined in the following files:
    /home/krentel/.spack/cray/compilers.yaml
```

I see the same problem on cori at NERSC and theta at ANL, both Cray.

I'm quite sure this used to work.  I update my compilers.yaml so
infrequently that it's hard to say how far back it was when it worked.
Maybe 3 months, 6 months, maybe 12.  But I think something has changed
in how spack reads the FE compilers on Cray.

But since it used to work, at some point, I can go back and try old
revs and apply binary search.



Discussion:
Ok, I did the binary search and found the regression.
It's from 6d56d4545472 and #10190.
ping @alalazo

My memory was a bit off.  `spack compiler find` (with no arguments)
never found FE compilers on Cray, except for `/usr/bin/gcc`.

BUT, `spack compiler find /opt/gcc/7.3.0` did used to work and identify
this as a FE compiler.  Before the commit, I get,

```
$ ./spack compiler find /opt/gcc/7.3.0   
==> Added 1 new compiler to /home/krentel/.spack/cray/compilers.yaml
    gcc@7.3.0
```

And after (with a fresh compilers.yaml, of course),

```
$ ./spack compiler find /opt/gcc/7.3.0   
==> Found no new compilers
```

That's what I expect on machines with modules.  I load the
module and run either `spack compiler find` or else
`spack compiler find /path/to/prefix`.

And note: on Cray, the raw compiler `/opt/gcc/7.3.0/bin/gcc` is a
valid front-end compiler.  It's the `cc, CC, ftn` wrappers that
compile for the back end.

----------

This brings up an interesting question.  For systems with separate
FE/BE compilers, would it make sense to add an argument to
`spack compiler add` to tell spack this is FE or BE compiler?

For example, `spack compiler add /opt/gcc/7.3.0 arch=fe`.

@alalazo I'm assigning this to you since @mwkrentel bisected it down to a commit from your PR #10190.
I used to have access to Piz Daint, but I realize it's not the case anymore since a couple of days. I'll try to get a username on any Cray asap to test this. In the meanwhile if anybody could provide a `xfailing` unit test I could work on that.
@tgamblin We're trying to get Cray support in place for El Capitan and the failure to address this issue is a major stumbling block. I would conjecture that Mark Krentel is the only one who could successfully install HPCToolkit for Cray because this issue has not been addressed. 

Mark just walked me through an install and we had to go back to a commit from last year before running "spack compiler find" to get the right configuration for the front-end compilers. That is really not a sustainable model. 

This issue has sat without attention for ~9 months.
@jmellorcrummey 

> We're trying to get Cray support in place for El Capitan and the failure to address this issue is a major stumbling block. I would conjecture that Mark Krentel is the only one who could successfully install HPCToolkit for Cray because this issue has not been addressed

I understand that not being able to detect all compilers using a command is inconvenient. I recently got access again to Cori and Piz Daint, so I'll look into this issue asap. That said the compiler configuration file (as any other configuration file in Spack) is meant to be manually editable. You can work around this issue by adding manually the appropriate compiler entries in `compilers.yaml`, which would look similar to:
```yaml
- compiler:
    spec: gcc@7.3.0
    paths:
      cc: /opt/gcc/7.3.0/bin/gcc
      cxx: /opt/gcc/7.3.0/bin/g++
      f77: /opt/gcc/7.3.0/bin/gfortran
      fc: /opt/gcc/7.3.0/bin/gfortran
    flags: {}
    operating_system: sles12
    target: x86_64
    modules: 
      - PrgEnv-gnu
      - gnu/7.3.0
```
That should hopefully permit you to continue your work, while compiler detection is getting fixed. Let me know if you need help with that.
@becker33 I started on this. It should be fairly easy to recover the previous behavior. Also, even with the current behavior a valid front-end compiler can be detected automatically by loading the corresponding module and searching for new compilers:
```console
culpo@daint102:~/github/spack> spack compiler find
==> Added 19 new compilers to /users/culpo/.spack/cray/compilers.yaml
    pgi@20.1.0  pgi@19.5.0  intel@19.0.1.144  intel@18.0.1.163  gcc@8.3.0  gcc@8.1.0  gcc@7.3.0  cce@9.0.2  cce@9.0.1  cce@8.7.9
    pgi@19.7.0  pgi@19.4.0  intel@18.0.2.199  intel@17.0.4.196  gcc@8.2.0  gcc@7.4.1  gcc@6.1.0  cce@9.0.2  cce@9.0.1
==> Compilers are defined in the following files:
    /users/culpo/.spack/cray/compilers.yaml

culpo@daint102:~/github/spack> module load gcc/8.3.0
culpo@daint102:~/github/spack> spack compiler find
==> Added 1 new compiler to /users/culpo/.spack/cray/compilers.yaml
    gcc@8.3.0
==> Compilers are defined in the following files:
    /users/culpo/.spack/cray/compilers.yaml

culpo@daint102:~/github/spack> spack compiler list
==> Available compilers
-- cce cnl7-any -------------------------------------------------
cce@9.0.2  cce@9.0.2  cce@9.0.1  cce@9.0.1  cce@8.7.9

-- gcc cnl7-any -------------------------------------------------
gcc@8.3.0  gcc@8.2.0  gcc@8.1.0  gcc@7.3.0  gcc@6.1.0

-- gcc sles15-x86_64 --------------------------------------------
gcc@8.3.0  gcc@7.4.1

-- intel cnl7-any -----------------------------------------------
intel@19.0.1.144  intel@18.0.2.199  intel@18.0.1.163  intel@17.0.4.196

-- pgi cnl7-any -------------------------------------------------
pgi@20.1.0  pgi@19.7.0  pgi@19.5.0  pgi@19.4.0
```
@jmellorcrummey Is there any other fundamental issue that I am missing here?
@alalazo  Hmm, that's not the behavior I see.  On both cori at NERSC
and theta at ANL, I get the following with today's spack.

On cori, starting without a compilers.yaml file, modules for
PrgEnv-gnu and gcc/8.3.0, I run `spack compiler find` and then
`spack compiler list`.  gcc 7.5.0 is the one from /usr/bin.

```
$ spack compiler list
-- gcc sles15-x86_64 --------------------------------------------
gcc@7.5.0

$ module load gcc/8.3.0

$ type -a gcc
gcc is /opt/gcc/8.3.0/bin/gcc
gcc is /usr/bin/gcc

$ spack compiler find
==> Found no new compilers
==> Compilers are defined in the following files:
    /global/homes/k/krentel/.spack/cray/compilers.yaml

$ spack compiler find /opt/gcc/8.3.0
==> Found no new compilers
==> Compilers are defined in the following files:
    /global/homes/k/krentel/.spack/cray/compilers.yaml

$ spack compiler list
-- gcc sles15-x86_64 --------------------------------------------
gcc@7.5.0
```

What am I doing differently?
Is it the machine?  Something in my environment?
You're running this with PrgEnv-gnu, right?
Here is `module show gcc/8.3.0`, if that helps.

```
/opt/modulefiles/gcc/8.3.0:

conflict         gcc 
conflict         gcc-cross-aarch64 
prepend-path     PATH /opt/gcc/8.3.0/bin 
prepend-path     MANPATH /opt/gcc/8.3.0/snos/share/man 
prepend-path     INFOPATH /opt/gcc/8.3.0/snos/share/info 
prepend-path     LD_LIBRARY_PATH /opt/gcc/8.3.0/snos/lib64 
setenv           GCC_PATH /opt/gcc/8.3.0 
setenv           GCC_VERSION 8.3.0 
setenv           GNU_VERSION 8.3.0 
```

And both cori and theta use TCL modules, if that matters.
@mwkrentel The test I did so far were done only on Piz-Daint. I just checked and the default PE there is `PrgEnv-cray`, so I was using that. If I switch explicitly to `PrgEnv-gnu` I obtain your same results. I'll try to work on a fix and then also test it on Cori. 
To give more context, we do two separate searches to detect compilers for back-end and front-end. The one on front-end is based on:

https://github.com/spack/spack/blob/b26e93af3d19e4492cf049bc7479eb85bc8f7da7/lib/spack/spack/operating_systems/cray_frontend.py#L56-L65

This is the part of code that unloads the `PrgEnv` before trying to search executables in the PATH and thus prevents from detecting `gcc` on the frontend if `PrgEnv-gnu` is loaded. I'll probably need to extend the logic in that method.

Resolving patch:
diff --git a/lib/spack/spack/operating_systems/cray_frontend.py b/lib/spack/spack/operating_systems/cray_frontend.py
--- a/lib/spack/spack/operating_systems/cray_frontend.py
+++ b/lib/spack/spack/operating_systems/cray_frontend.py
@@ -5,8 +5,11 @@
 
 import contextlib
 import os
+import re
 
 import llnl.util.filesystem as fs
+import llnl.util.lang
+import llnl.util.tty as tty
 
 from spack.operating_systems.linux_distro import LinuxDistro
 from spack.util.environment import get_path
@@ -60,6 +63,42 @@ def compiler_search_paths(self):
         This prevents from detecting Cray compiler wrappers and avoids
         possible false detections.
         """
+        import spack.compilers
+
         with unload_programming_environment():
-            search_paths = fs.search_paths_for_executables(*get_path('PATH'))
-        return search_paths
+            search_paths = get_path('PATH')
+
+        extract_path_re = re.compile(r'prepend-path[\s]*PATH[\s]*([/\w\.:-]*)')
+
+        for compiler_cls in spack.compilers.all_compiler_types():
+            # Check if the compiler class is supported on Cray
+            prg_env = getattr(compiler_cls, 'PrgEnv', None)
+            compiler_module = getattr(compiler_cls, 'PrgEnv_compiler', None)
+            if not (prg_env and compiler_module):
+                continue
+
+            # It is supported, check which versions are available
+            output = module('avail', compiler_cls.PrgEnv_compiler)
+            version_regex = r'({0})/([\d\.]+[\d]-?[\w]*)'.format(
+                compiler_cls.PrgEnv_compiler
+            )
+            matches = re.findall(version_regex, output)
+            versions = tuple(version for _, version in matches)
+
+            # Now inspect the modules and add to paths
+            msg = "[CRAY FE] Detected FE compiler [name={0}, versions={1}]"
+            tty.debug(msg.format(compiler_module, versions))
+            for v in versions:
+                try:
+                    current_module = compiler_module + '/' + v
+                    out = module('show', current_module)
+                    match = extract_path_re.search(out)
+                    search_paths += match.group(1).split(':')
+                except Exception as e:
+                    msg = ("[CRAY FE] An unexpected error occurred while "
+                           "detecting FE compiler [compiler={0}, "
+                           " version={1}, error={2}]")
+                    tty.debug(msg.format(compiler_cls.name, v, str(e)))
+
+        search_paths = list(llnl.util.lang.dedupe(search_paths))
+        return fs.search_paths_for_executables(*search_paths)


Test patch:
diff --git a/lib/spack/spack/test/compilers/detection.py b/lib/spack/spack/test/compilers/detection.py
--- a/lib/spack/spack/test/compilers/detection.py
+++ b/lib/spack/spack/test/compilers/detection.py
@@ -4,6 +4,9 @@
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 """Test detection of compiler version"""
 import pytest
+import os
+
+import llnl.util.filesystem as fs
 
 import spack.compilers.arm
 import spack.compilers.cce
@@ -16,6 +19,9 @@
 import spack.compilers.xl
 import spack.compilers.xl_r
 
+from spack.operating_systems.cray_frontend import CrayFrontend
+import spack.util.module_cmd
+
 
 @pytest.mark.parametrize('version_str,expected_version', [
     ('Arm C/C++/Fortran Compiler version 19.0 (build number 73) (based on LLVM 7.0.2)\n' # NOQA
@@ -189,3 +195,41 @@ def test_xl_version_detection(version_str, expected_version):
 
     version = spack.compilers.xl_r.XlR.extract_version_from_output(version_str)
     assert version == expected_version
+
+
+@pytest.mark.parametrize('compiler,version', [
+    ('gcc', '8.1.0'),
+    ('gcc', '1.0.0-foo'),
+    ('pgi', '19.1'),
+    ('pgi', '19.1a'),
+    ('intel', '9.0.0'),
+    ('intel', '0.0.0-foobar')
+])
+def test_cray_frontend_compiler_detection(
+        compiler, version, tmpdir, monkeypatch, working_env
+):
+    """Test that the Cray frontend properly finds compilers form modules"""
+    # setup the fake compiler directory
+    compiler_dir = tmpdir.join(compiler)
+    compiler_exe = compiler_dir.join('cc').ensure()
+    fs.set_executable(str(compiler_exe))
+
+    # mock modules
+    def _module(cmd, *args):
+        module_name = '%s/%s' % (compiler, version)
+        module_contents = 'prepend-path PATH %s' % compiler_dir
+        if cmd == 'avail':
+            return module_name if compiler in args[0] else ''
+        if cmd == 'show':
+            return module_contents if module_name in args else ''
+    monkeypatch.setattr(spack.operating_systems.cray_frontend, 'module',
+                        _module)
+
+    # remove PATH variable
+    os.environ.pop('PATH', None)
+
+    # get a CrayFrontend object
+    cray_fe_os = CrayFrontend()
+
+    paths = cray_fe_os.compiler_search_paths
+    assert paths == [str(compiler_dir)]