Repository: spack/spack

Issue:
Use of --dirty with externals packages and compilers (always required?)
If we have externally installed packages or compilers with `modules` and if we specify those modules in Spack config files (`packages.yaml`, `compilers.yaml`) then we should not require `--dirty` flag. At least this is expectation from Slack discussion. But this is not the case.

Here is an example:

We have gcc module as:

```
→ module show gcc_490
-------------------------------------------------------------------
/gpfs/bbp.cscs.ch/home/kumbhar/my_modules/gcc_490:

module-whatis	 Loads gcc 4.9.0 to your environment variables
prepend-path	 PATH /gpfs/bbp.cscs.ch//apps/viz/tools/gcc/gcc-4.9.0/install/bin
prepend-path	 LD_LIBRARY_PATH /gpfs/bbp.cscs.ch//apps/viz/tools/gcc/gcc-4.9.0/install/lib64
prepend-path	 LD_LIBRARY_PATH /gpfs/bbp.cscs.ch/apps/viz/tools/mpc/mpc-1.0.2/install/lib
prepend-path	 LD_LIBRARY_PATH /gpfs/bbp.cscs.ch/apps/viz/tools/mpfr/mpfr-3.1.2/install/lib
prepend-path	 LD_LIBRARY_PATH /gpfs/bbp.cscs.ch/apps/viz/tools/gmp/gmp-5.1.3/install/lib
prepend-path	 LD_RUN_PATH /gpfs/bbp.cscs.ch//apps/viz/tools/gcc/gcc-4.9.0/install/lib64
prepend-path	 CPATH /gpfs/bbp.cscs.ch//apps/viz/tools/gcc/gcc-4.9.0/install/include
prepend-path	 SEARCH_DIR /gpfs/bbp.cscs.ch//apps/viz/tools/gcc/gcc-4.9.0/install/lib64
```

We add entry to `compilers.yaml` as:

```
- compiler:
    flags: {}
    modules:
    - gcc_490
    operating_system: rhel6
    paths:
      cc: /gpfs/bbp.cscs.ch/apps/viz/tools/gcc/gcc-4.9.0/install/bin/gcc
      cxx: /gpfs/bbp.cscs.ch/apps/viz/tools/gcc/gcc-4.9.0/install/bin/g++
      f77: /gpfs/bbp.cscs.ch/apps/viz/tools/gcc/gcc-4.9.0/install/bin/gfortran
      fc: /gpfs/bbp.cscs.ch/apps/viz/tools/gcc/gcc-4.9.0/install/bin/gfortran
    spec: gcc@4.9.0
```

As we have specified module then it should work without dirty. Isn't it? But we get:

```
$ spack -d install -v everytrace %gcc@4.9.0
...
/gpfs/bbp.cscs.ch/apps/viz/tools/gcc/gcc-4.9.0/install/bin/../libexec/gcc/x86_64-unknown-linux-
gnu/4.9.0/cc1: error while loading shared libraries: libmpc.so.3: cannot open shared object file: No 
.... 
"/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/lib/
spack/env/gcc/gfortran"
  is not able to compile a simple test program.
....
```

If I look at `spack-build.env` with and without dirty then we see that `LD_LIBRARY_PATH` is set only with `--dirty`:

```
 → diff spack-build.env spack-build.env.dirty
84a85
> export CPATH="/gpfs/bbp.cscs.ch//apps/viz/tools/gcc/gcc-4.9.0/install/include:/gpfs/bbp.cscs.ch/apps/viz/tools/gmp/gmp-5.1.3/install/include:/gpfs/bbp.cscs.ch/apps/viz/tools/mpfr/mpfr-3.1.2/install/include:/gpfs/bbp.cscs.ch/apps/viz/tools/mpc/mpc-1.0.2/install/include"
112a114,115
> export LD_LIBRARY_PATH="/gpfs/bbp.cscs.ch/apps/viz/tools/mvapich2/mvapich2-2.0.1-nocuda-slurm-14.03.4.2/install/lib:/gpfs/bbp.cscs.ch//apps/viz/tools/gcc/gcc-4.9.0/install/lib64:/gpfs/bbp.cscs.ch/apps/viz/tools/gmp/gmp-5.1.3/install/lib:/gpfs/bbp.cscs.ch/apps/viz/tools/mpfr/mpfr-3.1.2/install/lib:/gpfs/bbp.cscs.ch/apps/viz/tools/mpc/mpc-1.0.2/install/lib:/gpfs/bbp.cscs.ch/apps/viz/tools/git/git-1.8.4.3/install/lib"
> export LD_RUN_PATH="/gpfs/bbp.cscs.ch//apps/viz/tools/gcc/gcc-4.9.0/install/lib64"
```

cc : @tgamblin @becker33 


Discussion:
@adamjstewart  @davydden : I have browsed through old GitHub issues / discussions but still confused about this. I wonder if you have come across this.
> We add entry to compilers.yaml as

IIRC `modules` in `compilers.yaml` work only for Cray. I once tried it on our Ubuntu cluster, but it did not work. This should certainly be documented.

EDIT: the problem is there for externally provided `mpi` https://github.com/LLNL/spack/issues/2563 , but i think I also tried it with `intel` compilers which have to be loaded on a cluster.
From Slack discussion this should work ("env is first cleaned, then we load the modules.").

I think above case should work. IMO `--dirty` is more for something like:

```
export LD_LIBRARY_PATH=...
# or
module load xxxx

spack install --dirty package
```

I would like to ask some of our end users to install the softwares we develop using Spack. Asking them:
```
 “Hey, use `spack install --dirty xyz %gcc@4.9.0` but make sure you don’t
have dirty `LD_LIBRARY_PATH`, `PATH`, `DYLD_LIBRARY_PATH`
```
 etc is bit inconvenient. As a developer/packager we can understand this but not all end users.
> the problem is there for externally provided mpi #2563 , but i think I also tried it with intel compilers which have to be loaded on a cluster.

I think the error `'NoneType' object has no attri....` in #2563 is may be more related to modules parsing/loading related (?). Recently I created #5273 to understand requirements for external modules.

In our cluster I use external mpi packages as:

```
    mpich:
        modules:
            mpich@3.2%pgi: pgi/mpich/16.5
        buildable: False
        version: [3.2]
    mvapich2:
        modules:
            mvapich2@2.0.1%gcc: mvapich2/2.0.1-nocuda
        version: [2.0.1]
        buildable: False
    openmpi:
        modules:
            openmpi@1.8.7: openmpi/1.8.7
        version: [1.8.7]
        buildable: False
```

And this is working fine (except the `dirty` requirement )

Resolving patch:
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -167,12 +167,6 @@ def set_compiler_environment_variables(pkg, env):
 
     env.set('SPACK_COMPILER_SPEC', str(pkg.spec.compiler))
 
-    for mod in compiler.modules:
-        # Fixes issue https://github.com/LLNL/spack/issues/3153
-        if os.environ.get("CRAY_CPU_TARGET") == "mic-knl":
-            load_module("cce")
-        load_module(mod)
-
     compiler.setup_custom_environment(pkg, env)
 
     return env
@@ -312,9 +306,6 @@ def set_build_environment_variables(pkg, env, dirty):
             if os.path.isdir(pcdir):
                 env.prepend_path('PKG_CONFIG_PATH', pcdir)
 
-    if pkg.architecture.target.module_name:
-        load_module(pkg.architecture.target.module_name)
-
     return env
 
 
@@ -484,7 +475,7 @@ def setup_package(pkg, dirty):
     set_compiler_environment_variables(pkg, spack_env)
     set_build_environment_variables(pkg, spack_env, dirty)
     pkg.architecture.platform.setup_platform_environment(pkg, spack_env)
-    load_external_modules(pkg)
+
     # traverse in postorder so package can use vars from its dependencies
     spec = pkg.spec
     for dspec in pkg.spec.traverse(order='post', root=False, deptype='build'):
@@ -511,6 +502,21 @@ def setup_package(pkg, dirty):
     validate(spack_env, tty.warn)
     spack_env.apply_modifications()
 
+    # All module loads that otherwise would belong in previous functions
+    # have to occur after the spack_env object has its modifications applied.
+    # Otherwise the environment modifications could undo module changes, such
+    # as unsetting LD_LIBRARY_PATH after a module changes it.
+    for mod in pkg.compiler.modules:
+        # Fixes issue https://github.com/LLNL/spack/issues/3153
+        if os.environ.get("CRAY_CPU_TARGET") == "mic-knl":
+            load_module("cce")
+        load_module(mod)
+
+    if pkg.architecture.target.module_name:
+        load_module(pkg.architecture.target.module_name)
+
+    load_external_modules(pkg)
+
 
 def fork(pkg, function, dirty):
     """Fork a child process to do part of a spack build.


Test patch: