Repository: spack/spack

Issue:
setup_dependent_environment overwritten by module-defined external compiler
## Problem

In Spack, a compiler definition that uses a `module:` token can interfere with or nullify any environment variable defined by `setup_dependent_environment()` or `setup_dependent_package()` of a package upstream in the DAG.

## Question

How does one get around this, with the fewest assumptions and least _a priori_ knowledge?

* Can one defer calling the `setup_dependent_*` hooks until _after_ the compiler is loaded?

  If I'm not mistaken, that's the canonical way of doing things outside of Spack. I can see why Spack would load compilers last, perhaps to largely (though not entirely) guarantee a consistent compiler environment?

* Blacklist "known" variables for `module load` in the specific situation of loading a compiler module? – I think that could only cure specific symptoms.

* Have the user modify the module that defines the compiler? – Ditto for symptoms-only, and I'd say that's impractical for many users.

  On my own cluster, that's what I had done, if unrelated to Spack: I create modulefiles for `intel-parallel-studio` by a script from `psxevars.sh` but _excised_ any MPI variables and paths and spirits those away into a separate `impi` package. This avoided a hit from the main example that follows:

## Main Example: I_MPI_ROOT

The present issue crystallized from the root cause for the failure in issue #8410, which concerns `$I_MPI_ROOT`, defined by both `intel-parallel-studio` (any edition) and `intel-mpi`. 

Consider the scenario where a user runs:
```
spack install clientpkg %intel ^intel-mpi
```
and where the user has in `compilers.yaml`:
``` yaml
- compiler:
    …
    spec:       intel@18.0.3
    # Module provided externally or perhaps even internally by Spack:
    modules:    [intel/18]
    paths:
        …
```

Then, there are two failure modes:

### Failure mode 1: MPI Runtime vs. MPI Development version

When `intel-parallel-studio` is `@composer.*` or `@professional.*`, it includes an MPI runtime, and therefore sets `I_MPI_ROOT`. This value supersedes any specific value or the internal default if unset for any prerequisite `^intel-mpi` version.

When compiling a client package, `mpicc` etc. from `intel-mpi` will error out due to missing static libs only found under `$I_MPI_ROOT` of `intel-mpi`, but not that of `intel-parallel-studio`. Sadly, that information is buried in `config.log`. The user sees the rather more dramatic message:
```
configure: error: C compiler cannot create executables
```

### Failure mode 2: Silent version overshadowing
When `intel-parallel-studio` is `@cluster.*`, a version mismatch occurs for MPI components: In the eyes of the user and Spack, the client package is compiled with `intel-mpi@foo`, whereas `mpicc` (which is from `intel-mpi@foo`) will take its libs etc. from `intel-parallel-studio`.

Disconcertingly, this will occur _silently_.

Now, a user would not typically need or use `^intel-mpi` when one has `%intel-parallel-studio@cluster.*`, but Spack will let you do it. It may not be clear to a new user that it's typically not needed, and it's bad for advanced users who specifically choose a different MPI version.

## Other affected packages: MKL, …

The same is true for the other `intel-foo` library-style packages, of which `intel-mkl` is most critical.

The MKL _is_ provided by all studio versions, so failure mode 1. does not occur, but failure mode 2. might, especially when the user has no access to newer compilers but does for the MKL, which has been available under a "simplified" (no-cost) license since 2017.

## Variable tracing
Here's a more detailed tracing of the MPI variables for the case discussed in  #8410 :
```
spack --debug -v -v install -v -v hdf5@1.10.2+mpi ^intel-mpi
```

Gives (formatted for readability):
```
==> Installing hdf5

==> setup_package.setup_dependent_package:
    spec 'intel-mpi@2018. ...' received .mpi* properties:
    {
	'MPICC':   '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiicc',
	'MPICXX':  '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiicpc',
	'MPIF77':  '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiifort',
	'MPIF90':  '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiifort'

	'MPIFC':   '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiifort',
    }
...
==> setup_dependent_environment._setup_dependent_env_callback.mpi_setup_dependent_environment:
    adding to spack_env:
    {
	'MPICC':      '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiicc',
	'MPICXX':     '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiicpc',
	'MPIF77':     '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiifort',
	'MPIF90':     '.../intel-mpi/2018.3.222-intel-18.0.3/.../intel64/bin/mpiifort',

	'I_MPI_CC':   '.../lib/spack/env/intel/icc',
	'I_MPI_CXX':  '.../lib/spack/env/intel/icpc',
	'I_MPI_F77':  '.../lib/spack/env/intel/ifort'
	'I_MPI_F90':  '.../lib/spack/env/intel/ifort',
	'I_MPI_FC':   '.../lib/spack/env/intel/ifort',
	'I_MPI_ROOT': '.../.../intel-mpi/2018.3.222-intel-18.0.3/Hipwooo/compilers_and_libraries_2018.3.222/linux/mpi',
    }


==> '/usr/bin/modulecmd' 'python' 'load' 'intel-parallel-studio/composer.2018.3/intel-18.0.3/Hnd7rbb'
...
```

.. and fails with:
```
  >> 33    configure: error: in `/tmp/stern/spack-stage/spack-stage-MQP6Az/hdf5-1.10.2':
  >> 34    configure: error: C compiler cannot create executables
```


```
$ module show intel-parallel-studio/composer.2018.3/intel-18.0.3/Hnd7rbb | sed 's,[ \t:]\+/,\n\t/,g'
```
(formatted for readability):
* MPI:
  ```
  prepend-path	LD_LIBRARY_PATH
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mpi/intel64/lib
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mpi/mic/lib
  prepend-path	CLASSPATH
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mpi/intel64/lib/mpi.jar
  setenv		I_MPI_ROOT
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mpi
  prepend-path	MANPATH
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mpi/man
  prepend-path	PATH
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mpi/intel64/bin
  ```
* MKL:
  ```
  prepend-path	CPATH
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mkl/include
  prepend-path	LD_LIBRARY_PATH
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mkl/lib/intel64_lin
  prepend-path	LIBRARY_PATH
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mkl/lib/intel64_lin
  setenv		MKLROOT
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mkl
  prepend-path	NLSPATH
	.../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mkl/lib/intel64_lin/locale/%l_%t/%N
  prepend-path	PKG_CONFIG_PATH
	  .../intel-parallel-studio/composer.2018.3-intel-18.0.3/.../mkl/bin/pkgconfig
  ```
* likewise for DAAL, IPP, etc.


#### Location where `I_MPI_ROOT` is used
This occurs right on top of `mpiicc `:
```
$ grep -C1 I_MPI  /home/SHARE/soft/spack/spack-stern/opt/spack/linux-centos6-x86_64/\
intel-mpi/2018.3.222-intel-18.0.3/Hipwooo/compilers_and_libraries_2018.3.222/linux/mpi/intel64/bin/mpiicc 
```
namely,
```
prefix=.../intel-mpi/2018.3.222-intel-18.0.3/Hipwooo/compilers_and_libraries_2018.3.222/linux/mpi
# The environment variable I_MPI_ROOT may be used to override installation folder path
if [ -n "$I_MPI_ROOT" ] ; then
    prefix=$I_MPI_ROOT;
fi
```
The lib missing  in  #8410 is specifically mentioned as well; it is available as static lib only, and is linked explicitly even for shared linkage:
```
...
    I_MPI_OTHERLIBS=" $libdir/libmpigi.a"
...
    I_MPI_OTHERLIBS=" -lmpigi"
...
```


Discussion:
> Can one defer calling the setup_dependent_* hooks until after the compiler is loaded?

At the moment the order is hard-coded (to load compiler modules after `setup_dependent_environment`). Do you think that the opposite order would be universally better? Or would you prefer a way to customize the order?
@scheibelp 
> At the moment the order is hard-coded […]. Do you think that the opposite order would be universally better?

As I wrote:
> [loading compilers first is] the canonical way of doing things outside of Spack. I can see why Spack would load compilers last, perhaps to largely (though not entirely) guarantee a consistent compiler environment?

The current Spack strategy will backfire for any omnibus development package that provides more than just a compiler. Today, perhaps only Parallel Studio fits that bill, but tomorrow other packages may come along that want to be all things to all people. With compilers loaded last (and in the form of modules), all such omnibus packages will _silently overshadow_ alternative sub-packages that a user may legitimately want, as shown for Intel-MPI and MKL in the intro.

I am aware that changing the order touches on perhaps deep architectural choices made for Spack that I don't have insight on. If the default order is not changed, a configurable order provides at least a way out, but it puts the burden on the user and it could may make issue replications tricky.
> I am aware that changing the order touches on perhaps deep architectural choices made for Spack

The order of compiler vs. module loads is not particularly difficult to change and I don't think there would be too much controversy around it. As an example, https://github.com/spack/spack/pull/8245 (which was closed in favor of https://github.com/spack/spack/pull/8346) changed the order.

@alalazo do you see a major reason to maintain the current order? I know in https://github.com/spack/spack/pull/8245 you mention the line

> All module loads that otherwise would belong in previous functions have to occur after the spack_env object has its modifications applied.

but I think that was primarily because the environment cleaning actions like unsetting `LD_LIBRARY_PATH` are currently bundled with the dependency environment modifications.
@mgsternberg @scheibelp The problem:

> In Spack, a compiler definition that uses a module: token can interfere with or nullify any environment variable defined by setup_dependent_environment() or setup_dependent_package() of a package upstream in the DAG

seems to be a generalization of #8346 (there we were only interested in `CC`, etc.). I think we could probably solve this issue with an extension of #8346 by:

1. Keeping track of which variables will be modified by `spack_env`
2. Passing that list to the  `preserve_environment` context manager

Would this solution work for you? Do you see major drawbacks?
@alalazo 
> I think we could probably solve this issue with an extension of #8346

That's what I described by:
> Blacklist "known" variables for `module load` in the specific situation of loading a compiler module? – I think that could only cure specific symptoms.

… and it certainly won't tackle the problem "with the fewest assumptions and least _a priori_ knowledge".


The relevant env. var names include `I_MPI_ROOT` and `MKLROOT`. While #8346 reasonably hard-codes `CC` & friends, the vars here are far more specialized and cover only today's needs. Therefore, a list of additional variables must be user-configurable and presumably will have to live in the `compilers.yaml` preferences(*), each in a sister leaf of any offending `modules:` tag. As tag name, `preserve_environment:` would suggest itself.  Don't use "blacklist" because that term is already used twice elsewhere in Spack in similar but different contexts and therefore could be confusing.

(*) _For goodness' sake,_ don't include that list in the hash definition of the compiler spec.
@mgsternberg 

> The relevant env. var names include I_MPI_ROOT and MKLROOT. While #8346 reasonably hard-codes CC & friends, the vars here are far more specialized and cover only today's needs. Therefore, a list of additional variables must be user-configurable and presumably will have to live in the compilers.yaml preferences(*), each in a sister leaf of any offending modules: tag. As tag name, preserve_environment: would suggest itself. Don't use "blacklist" because that term is already used twice elsewhere in Spack in similar but different contexts and therefore could be confusing.

I think you misunderstood my proposal (or I was not clear exposing it). What I want to do is to use the knowledge already in `spack_env` to prevent modifications to all the variables modified explicitly by a package in the DAG. In any case the modifications I had in mind were just a few lines, so see #8688
Sorry, yes, I missed the core idea. I commented at #8688, which I fear is going too far.

What about the idea of deferring the `setup_dependent_*` hooks until _after_ the compiler is loaded, i.e., switching the current sequence? I figured this would touch on architectural choices and thus have ramifications in kind, but @scheibelp mentioned that this would [not be out of the question](https://github.com/spack/spack/issues/8632#issuecomment-403915048).
> What about the idea of deferring the setup_dependent_* hooks until after the compiler is loaded, i.e., switching the current sequence? I figured this would touch on architectural choices and thus have ramifications in kind, but @scheibelp mentioned that this would not be out of the question.

I am not opposed to it, though it may come with its own issues (an incompatibility like `I_MPI_ROOT` might also go the other way around, i.e. being overridden by packages while it should stay as it was set by compiler modules). In that case I would probably proceed like @scheibelp did in #8245, but with a small modification in the order:

1. Clear the environment variables first (and emit a warning or at least a debug message if any is found?).
2. Load the modules for compilers. At this stage we should have a working compiler.
3. Construct and apply all the other modifications.

Thinking about it: we already have a factory for environment modifications called `EnvironmentModifications.from_sourcing_file` that returns the list of modifications to the environment you would have by sourcing a file. Would it be worth adding something like `EnvironmentModifications.from_loading_modules`? 

The added value is that if all the modifications are in the same object they can be analyzed together. It won't magically solve the issue, but at least in your case you would have seen a message like:
```console
==> Warning: Suspicious requests to set or unset 'CC' found
==> Warning: 	    	env.set('CC', join_path(link_dir, compiler.link_paths['cc'])) at /git/spack/lib/spack/spack/build_environment.py:146
==> Warning: 	--->	spack_env.set('CC', spack_cc) at /git/spack/var/spack/repos/builtin/packages/gdbm/package.py:49
```
pointing to a clear incompatibility among the two requests.
@alalazo 
>  proceed like @scheibelp did in #8245, but with a small modification in the order:
> 1. Clear the environment variables first (and emit a warning or at least a debug message if any is found?).
> 2. Load the modules for compilers. At this stage we should have a working compiler.
> 3. Construct and apply all the other modifications.

If I understand correctly, the _application_ of env. modifications in #8245 is already in the right order, done after the compiler setup at the very last step of `setup_package()` as `spack_env.apply_modifications()`, and your suggestion is _deferred construction_ so you could warn if a compiler setting is being touched. Thoughts on this:

* For the first step, the term "clear" is misleading since it suggests what `env -i` does.  Rather, "sanitize" in the function name would be more apt.
* For the problem on hand, I'd say a supervised `load_module()` via a `.from_loading_modules()` as you suggest is not needed for a compiler module if you _do_ load it first. I don't know about other occasions it might apply (external DAG packages?). I'd opine to keep things simple and not insert a layer into `load_module()`.
* In the construction of env. mods., deferred or otherwise, the `setup_dependent_*` hooks in the various package codes can only act on `EnvironmentModifications` objects `spack_env` and `run_env`. Thus, detection of overshadowing compiler-controlled vars would have to go into `EnvironmentModifications.{append,extend}` or perhaps an `execute` callback directly in `NameValueModifier`, rather than into a supervised `load_module()`.

  But I have reservations about trying to issue warnings as such:
  * You'd still need _a priori_ knowledge to decide which vars to warn about and which not, to keep the noise down, wouldn't you?
  * Moreover, compilers can be affected by env. variables that _have not been mentioned_ in the compiler modulefile or elsewhere. Spack cannot possibly know them all, and I'd say it should not try to interject.

I'd say #8245 needs to be reconsidered.

Resolving patch:
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -146,6 +146,33 @@ def __call__(self, *args, **kwargs):
         return super(MakeExecutable, self).__call__(*args, **kwargs)
 
 
+def clean_environment():
+    # Stuff in here sanitizes the build environment to eliminate
+    # anything the user has set that may interfere. We apply it immediately
+    # unlike the other functions so it doesn't overwrite what the modules load.
+    env = EnvironmentModifications()
+
+    # Remove these vars from the environment during build because they
+    # can affect how some packages find libraries.  We want to make
+    # sure that builds never pull in unintended external dependencies.
+    env.unset('LD_LIBRARY_PATH')
+    env.unset('LIBRARY_PATH')
+    env.unset('CPATH')
+    env.unset('LD_RUN_PATH')
+    env.unset('DYLD_LIBRARY_PATH')
+
+    # Remove any macports installs from the PATH.  The macports ld can
+    # cause conflicts with the built-in linker on el capitan.  Solves
+    # assembler issues, e.g.:
+    #    suffix or operands invalid for `movq'"
+    path = get_path('PATH')
+    for p in path:
+        if '/macports/' in p:
+            env.remove_path('PATH', p)
+
+    env.apply_modifications()
+
+
 def set_compiler_environment_variables(pkg, env):
     assert pkg.spec.concrete
     compiler = pkg.compiler
@@ -278,27 +305,6 @@ def set_build_environment_variables(pkg, env, dirty):
     # Install root prefix
     env.set(SPACK_INSTALL, spack.store.root)
 
-    # Stuff in here sanitizes the build environment to eliminate
-    # anything the user has set that may interfere.
-    if not dirty:
-        # Remove these vars from the environment during build because they
-        # can affect how some packages find libraries.  We want to make
-        # sure that builds never pull in unintended external dependencies.
-        env.unset('LD_LIBRARY_PATH')
-        env.unset('LIBRARY_PATH')
-        env.unset('CPATH')
-        env.unset('LD_RUN_PATH')
-        env.unset('DYLD_LIBRARY_PATH')
-
-        # Remove any macports installs from the PATH.  The macports ld can
-        # cause conflicts with the built-in linker on el capitan.  Solves
-        # assembler issues, e.g.:
-        #    suffix or operands invalid for `movq'"
-        path = get_path('PATH')
-        for p in path:
-            if '/macports/' in p:
-                env.remove_path('PATH', p)
-
     # Set environment variables if specified for
     # the given compiler
     compiler = pkg.compiler
@@ -629,6 +635,9 @@ def setup_package(pkg, dirty):
     spack_env = EnvironmentModifications()
     run_env = EnvironmentModifications()
 
+    if not dirty:
+        clean_environment()
+
     set_compiler_environment_variables(pkg, spack_env)
     set_build_environment_variables(pkg, spack_env, dirty)
     pkg.architecture.platform.setup_platform_environment(pkg, spack_env)
@@ -656,14 +665,12 @@ def setup_package(pkg, dirty):
     set_module_variables_for_package(pkg, pkg.module)
     pkg.setup_environment(spack_env, run_env)
 
-    # Make sure nothing's strange about the Spack environment.
-    validate(spack_env, tty.warn)
-    spack_env.apply_modifications()
-
     # Loading modules, in particular if they are meant to be used outside
     # of Spack, can change environment variables that are relevant to the
     # build of packages. To avoid a polluted environment, preserve the
     # value of a few, selected, environment variables
+    # With the current ordering of environment modifications, this is strictly
+    # unnecessary. Modules affecting these variables will be overwritten anyway
     with preserve_environment('CC', 'CXX', 'FC', 'F77'):
         # All module loads that otherwise would belong in previous
         # functions have to occur after the spack_env object has its
@@ -681,6 +688,10 @@ def setup_package(pkg, dirty):
 
         load_external_modules(pkg)
 
+    # Make sure nothing's strange about the Spack environment.
+    validate(spack_env, tty.warn)
+    spack_env.apply_modifications()
+
 
 def fork(pkg, function, dirty, fake):
     """Fork a child process to do part of a spack build.
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1478,7 +1478,7 @@ def to_node_dict(self, hash_function=None):
         if self.external:
             d['external'] = {
                 'path': self.external_path,
-                'module': bool(self.external_module)
+                'module': self.external_module
             }
 
         # TODO: restore build dependencies here once we have less picky
@@ -1916,7 +1916,7 @@ def concretize(self, tests=False):
                     dedupe(p + patches))
 
         for s in self.traverse():
-            if s.external_module:
+            if s.external_module and not s.external_path:
                 compiler = spack.compilers.compiler_for_spec(
                     s.compiler, s.architecture)
                 for mod in compiler.modules:


Test patch:
diff --git a/lib/spack/spack/test/build_environment.py b/lib/spack/spack/test/build_environment.py
--- a/lib/spack/spack/test/build_environment.py
+++ b/lib/spack/spack/test/build_environment.py
@@ -131,7 +131,6 @@ def _set_wrong_cc(x):
 
 @pytest.mark.usefixtures('config', 'mock_packages')
 def test_compiler_config_modifications(monkeypatch):
-
     s = spack.spec.Spec('cmake')
     s.concretize()
     pkg = s.package
@@ -209,3 +208,29 @@ def test_compiler_config_modifications(monkeypatch):
     os.environ.pop('PATH_LIST', None)
     os.environ.pop('EMPTY_PATH_LIST', None)
     os.environ.pop('NEW_PATH_LIST', None)
+
+
+@pytest.mark.regression('9107')
+def test_spack_paths_before_module_paths(config, mock_packages, monkeypatch):
+    s = spack.spec.Spec('cmake')
+    s.concretize()
+    pkg = s.package
+
+    module_path = '/path/to/module'
+
+    def _set_wrong_cc(x):
+        os.environ['PATH'] = module_path + ':' + os.environ['PATH']
+
+    monkeypatch.setattr(
+        spack.build_environment, 'load_module', _set_wrong_cc
+    )
+    monkeypatch.setattr(
+        pkg.compiler, 'modules', ['some_module']
+    )
+
+    spack.build_environment.setup_package(pkg, False)
+
+    spack_path = os.path.join(spack.paths.prefix, 'lib/spack/env')
+    paths = os.environ['PATH'].split(':')
+
+    assert paths.index(spack_path) < paths.index(module_path)