Repository: spack/spack

Issue:
IntelPackage gets compiler wrong
IntelPackage queries compiler support libraries (libomp) before compiler modules are loaded. These paths are then added as rpaths.

```python
==> [2022-02-21-19:44:20.434120] 'cc' '--print-file-name' 'libgomp.so'
Traceback (most recent call last):
  File "/path/to/spack/bin/spack", line 98, in <module>
    sys.exit(spack.main.main())
  File "/path/to/spack/lib/spack/spack/main.py", line 900, in main
    return _main(argv)
  File "/path/to/spack/lib/spack/spack/main.py", line 883, in _main
    return _invoke_command(command, parser, args, unknown)
  File "/path/to/spack/lib/spack/spack/main.py", line 553, in _invoke_command
    return_val = command(parser, args)
  File "/path/to/spack/lib/spack/spack/cmd/build_env.py", line 16, in build_env
    env_utility.emulate_env_utility('build-env', 'build', args)
  File "/path/to/spack/lib/spack/spack/cmd/common/env_utility.py", line 67, in emulate_env_utility
    build_environment.setup_package(spec.package, args.dirty, context)
  File "/path/to/spack/lib/spack/spack/build_environment.py", line 787, in setup_package
    set_wrapper_variables(pkg, env_mods)
  File "/path/to/spack/lib/spack/spack/build_environment.py", line 449, in set_wrapper_variables
    update_compiler_args_for_dep(dspec)
  File "/path/to/spack/lib/spack/spack/build_environment.py", line 421, in update_compiler_args_for_dep
    dep_link_dirs.extend(query.libs.directories)
  File "/path/to/spack/lib/spack/spack/spec.py", line 948, in __get__
    value = f()
  File "/path/to/spack/lib/spack/spack/spec.py", line 937, in <lambda>
    callbacks_chain.append(lambda: getattr(pkg, self.attribute_name))
  File "/path/to/spack/lib/spack/spack/build_systems/intel.py", line 993, in libs
    result = self.blas_libs + result
  File "/path/to/spack/lib/spack/spack/build_systems/intel.py", line 785, in blas_libs
    threading_engine_libs = self.openmp_libs
  File "/path/to/spack/lib/spack/spack/build_systems/intel.py", line 692, in openmp_libs
    raise Exception(omp_lib_path)
Exception: /opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0/../../../../lib64/libgomp.so
```

On Cray where `cc` is clang by default, this means it picks up clang's compiler support libraries, which are in turn from some old GCC:

```console
$ cc --version
Cray clang version 12.0.3  (58badb8c0532a8b5fadda356f3c5fa8499b4918d)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/12.0.3/cce-clang/x86_64/share/../bin

$ cc --print-file-name libgomp.so
/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0/../../../../lib64/libgomp.so
```

And its directory contains `/opt/gcc/8.1.0/snos/lib64/libstdc++.so.6.0.25`, but the compiler registered in spack should use `/opt/gcc/9.3.0/snos/lib64/libstdc++.so.6.0.28`.

Then when you compile anything C++17 like a hello world `std::filesystem`, linking fails.


Discussion:
Simpler example that fails immediately:

```console
$ module switch PrgEnv-cray/6.0.10 PrgEnv-nvidia 
$ cc --version # nvidia compiler for cc
nvc 21.3-0 LLVM 64-bit target on x86-64 Linux -tp haswell-64

$ spack config get compilers
compilers:
- compiler:
    spec: gcc@9.3.0
    paths:
      cc: cc
      cxx: CC
      f77: ftn
      fc: ftn
    flags: {}
    operating_system: cnl7
    target: any
    modules:
    - PrgEnv-gnu
    - gcc/9.3.0
    environment: {}
    extra_rpaths: []

$ cat spack.yaml
spack:
  specs:
  - netlib-lapack %gcc@9.3.0 ^intel-mkl threads=openmp
  view: false

$ spack -e . build-env netlib-lapack 
nvc-Error-Unknown switch: --print-file-name
==> Error: Command exited with status 1:
'cc' '--print-file-name' 'libgomp.so'
```
@rscohn2 pinging you since your handle pops up in `spack maintainers intel-mkl`.

This issue a high priority issue, since it's making spack unusable on our systems.

Another person reports gcc 7.5.0's `libgfortran.so.4` is picked up instead of `libgfortran.so.5` corresponding to gcc 9.3.0.
I reproduced this issue
Which package has the issue?
`intel-mkl` is what I've tried.

The issue is in build_systems/intel.py:

- https://github.com/spack/spack/blob/033647c8b1969aacc6db859fb7f779ff658f48ed/lib/spack/spack/build_systems/intel.py#L689-L692

`self.compiler.cc` may just be the string `"cc"`. Compilers in Spack may require modules to be loaded that alter the PATH so that `cc` refers to the right `cc`. So, if you run `cc` before those modules are loaded, it can refer to anything that happens to be in the PATH.

The problem is even worse on Cray's programming environment, where there's only one executable `cc` for _all possible compilers_, so specifying the path to `cc` explicitly in `compiler.yaml` configuration is not a solution, you must load the correct compiler module before executing `cc`.

So, one solution would be pass `env` in `compiler = Executable(self.compiler.cc); compiler(..., env=...)` with a dict of the environment variables you get as a result of `module load compiler/module`, but retrieving those variables is awkward.

Another solution is to not invoke the compiler at all before entering the build environment.

What do you think?
@scheibelp: Any thoughts on the right way to do this? MKL package libs property needs to return the openmp runtime as one of the libraries. The openmp runtime is provided by the compiler. The implementation of the libs property invokes the compiler so it can query where the openmp runtime is located. Apparently, it is not invoking the compiler that is actually used for the build. @haampie says some modules must be loaded to make the actual compiler available.

Resolving patch:
diff --git a/lib/spack/spack/build_systems/intel.py b/lib/spack/spack/build_systems/intel.py
--- a/lib/spack/spack/build_systems/intel.py
+++ b/lib/spack/spack/build_systems/intel.py
@@ -686,15 +686,15 @@ def openmp_libs(self):
             # packages.yaml), specificially to provide the 'iomp5' libs.
 
         elif '%gcc' in self.spec:
-            gcc = Executable(self.compiler.cc)
-            omp_lib_path = gcc(
-                '--print-file-name', 'libgomp.%s' % dso_suffix, output=str)
+            with self.compiler.compiler_environment():
+                omp_lib_path = Executable(self.compiler.cc)(
+                    '--print-file-name', 'libgomp.%s' % dso_suffix, output=str)
             omp_libs = LibraryList(omp_lib_path.strip())
 
         elif '%clang' in self.spec:
-            clang = Executable(self.compiler.cc)
-            omp_lib_path = clang(
-                '--print-file-name', 'libomp.%s' % dso_suffix, output=str)
+            with self.compiler.compiler_environment():
+                omp_lib_path = Executable(self.compiler.cc)(
+                    '--print-file-name', 'libomp.%s' % dso_suffix, output=str)
             omp_libs = LibraryList(omp_lib_path.strip())
 
         if len(omp_libs) < 1:
@@ -735,8 +735,9 @@ def tbb_libs(self):
 
         # TODO: clang(?)
         gcc = self._gcc_executable     # must be gcc, not self.compiler.cc
-        cxx_lib_path = gcc(
-            '--print-file-name', 'libstdc++.%s' % dso_suffix, output=str)
+        with self.compiler.compiler_environment():
+            cxx_lib_path = gcc(
+                '--print-file-name', 'libstdc++.%s' % dso_suffix, output=str)
 
         libs = tbb_lib + LibraryList(cxx_lib_path.rstrip())
         debug_print(libs)
@@ -746,8 +747,9 @@ def tbb_libs(self):
     def _tbb_abi(self):
         '''Select the ABI needed for linking TBB'''
         gcc = self._gcc_executable
-        matches = re.search(r'(gcc|LLVM).* ([0-9]+\.[0-9]+\.[0-9]+).*',
-                            gcc('--version', output=str), re.I | re.M)
+        with self.compiler.compiler_environment():
+            matches = re.search(r'(gcc|LLVM).* ([0-9]+\.[0-9]+\.[0-9]+).*',
+                                gcc('--version', output=str), re.I | re.M)
         abi = ''
         if sys.platform == 'darwin':
             pass
diff --git a/lib/spack/spack/compiler.py b/lib/spack/spack/compiler.py
--- a/lib/spack/spack/compiler.py
+++ b/lib/spack/spack/compiler.py
@@ -325,7 +325,7 @@ def accessible_exe(exe):
 
         # setup environment before verifying in case we have executable names
         # instead of absolute paths
-        with self._compiler_environment():
+        with self.compiler_environment():
             missing = [cmp for cmp in (self.cc, self.cxx, self.f77, self.fc)
                        if cmp and not accessible_exe(cmp)]
             if missing:
@@ -407,7 +407,7 @@ def _get_compiler_link_paths(self, paths):
                     compiler_exe.add_default_arg(flag)
 
             output = ''
-            with self._compiler_environment():
+            with self.compiler_environment():
                 output = str(compiler_exe(
                     self.verbose_flag, fin, '-o', fout,
                     output=str, error=str))  # str for py2
@@ -523,7 +523,7 @@ def get_real_version(self):
         modifications) to enable the compiler to run properly on any platform.
         """
         cc = spack.util.executable.Executable(self.cc)
-        with self._compiler_environment():
+        with self.compiler_environment():
             output = cc(self.version_argument,
                         output=str, error=str,
                         ignore_errors=tuple(self.ignore_version_errors))
@@ -597,7 +597,7 @@ def __str__(self):
                 str(self.operating_system)))))
 
     @contextlib.contextmanager
-    def _compiler_environment(self):
+    def compiler_environment(self):
         # store environment to replace later
         backup_env = os.environ.copy()
 


Test patch: