Repository: spack/spack

Issue:
Issue setting up Spack on Stampede2
I'm trying to set up Spack on Stampede2, with system Intel compiler and MPI, but couldn't make it work. I have asked for help [on the mailing list](https://groups.google.com/d/msg/spack/VSZqELYczok/09O4TV13AgAJ), but didn't get it resolved there. Probably email is not as effective a media as GitHub issue tracker for this kind of discussion. So I decided to report it here.

### Steps to reproduce the issue

```console
$ spack install mpi
==> intel-mpi@2018.2.199 : has external module in impi/18.0.2
==> intel-mpi@2018.2.199 : is actually installed in /opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64
==> intel-mpi@2018.2.199 : generating module file
('debug', '() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then\n case "$-" in \n *v*x*)\n __lmod_sh_dbg=\'vx\'\n ;;\n *v*)\n __lmod_sh_dbg=\'v\'\n ;;\n *x*)\n __lmod_sh_dbg=\'x\'\n ;;\n esac;\n fi;\n if [ -n "${__lmod_sh_dbg')
==> Error: Error: unterminated ${ in format:'() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
 case "$-" in 
 *v*x*)
 __lmod_sh_dbg='vx'
 ;;
 *v*)
 __lmod_sh_dbg='v'
 ;;
 *x*)
 __lmod_sh_dbg='x'
 ;;
 esac;
 fi;
 if [ -n "${__lmod_sh_dbg'
```

### Error Message

```console
$ spack -d --stacktrace install mpi
lib/spack/spack/config.py:683 ==> [2019-05-21-00:04:23.068180] Reading config file /home1/03560/tg828592/opt/spack/etc/spack/defaults/modules.yaml
lib/spack/spack/config.py:683 ==> [2019-05-21-00:04:23.075014] Reading config file /home1/03560/tg828592/opt/spack/etc/spack/defaults/linux/modules.yaml
lib/spack/spack/config.py:683 ==> [2019-05-21-00:04:23.077788] Reading config file /home1/03560/tg828592/opt/spack/etc/spack/defaults/config.yaml
lib/spack/spack/cmd/__init__.py:98 ==> [2019-05-21-00:04:23.092772] Imported install from built-in commands
lib/spack/spack/cmd/__init__.py:98 ==> [2019-05-21-00:04:23.093802] Imported install from built-in commands
lib/spack/spack/config.py:683 ==> [2019-05-21-00:04:23.102649] Reading config file /home1/03560/tg828592/opt/spack/etc/spack/defaults/repos.yaml
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.175080] READ LOCK: /home1/03560/tg828592/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.175764] READ LOCK: /home1/03560/tg828592/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.176624] READ LOCK: /home1/03560/tg828592/.spack/cache/tags/.builtin-index.json.lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.178966] READ LOCK: /home1/03560/tg828592/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.179620] READ LOCK: /home1/03560/tg828592/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.187958] READ LOCK: /home1/03560/tg828592/.spack/cache/patches/.builtin-index.json.lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.190317] READ LOCK: /home1/03560/tg828592/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.190948] READ LOCK: /home1/03560/tg828592/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.207350] READ LOCK: /home1/03560/tg828592/.spack/cache/providers/.builtin-index.json.lock[0:0] [Released]
lib/spack/spack/config.py:683 ==> [2019-05-21-00:04:23.215037] Reading config file /home1/03560/tg828592/opt/spack/etc/spack/defaults/packages.yaml
lib/spack/spack/config.py:683 ==> [2019-05-21-00:04:23.225054] Reading config file /home1/03560/tg828592/.spack/packages.yaml
lib/spack/spack/config.py:683 ==> [2019-05-21-00:04:23.256745] Reading config file /home1/03560/tg828592/.spack/compilers.yaml
lib/spack/spack/util/module_cmd.py:112 ==> [2019-05-21-00:04:23.429715] Module name: impi/18.0.2
lib/spack/spack/util/module_cmd.py:119 ==> [2019-05-21-00:04:23.430045] Package directory variable prefix: IMPI
lib/spack/spack/package.py:1280 ==> [2019-05-21-00:04:23.433942] intel-mpi@2018.2.199 : has external module in impi/18.0.2
lib/spack/spack/package.py:1282 ==> [2019-05-21-00:04:23.434177] intel-mpi@2018.2.199 : is actually installed in /opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64
lib/spack/spack/database.py:214 ==> [2019-05-21-00:04:23.436924] DATABASE LOCK TIMEOUT: 120s
lib/spack/spack/database.py:218 ==> [2019-05-21-00:04:23.437221] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.437929] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.438622] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.440015] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.440560] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.441202] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.442337] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Released]
lib/spack/spack/package.py:1300 ==> [2019-05-21-00:04:23.443226] intel-mpi@2018.2.199 : generating module file
lib/spack/spack/hooks/sbang.py:102 ==> [2019-05-21-00:04:23.443938] SKIP: shebang filtering [external package]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.444753] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.445396] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.446628] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Released]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.447245] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.447963] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.449151] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Released]
lib/spack/spack/modules/common.py:709 ==> [2019-05-21-00:04:23.450514]  WRITE: intel-mpi@2018.2.199%intel@18.0.2 arch=linux-centos7-x86_64/gzav7jv [/home1/03560/tg828592/opt/spack/share/spack/modules/linux-centos7-x86_64/intel-mpi-2018.2.199-intel-18.0.2-gzav7jv]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.466402] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.467111] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
lib/spack/llnl/util/lock.py:334 ==> [2019-05-21-00:04:23.468208] READ LOCK: /home1/03560/tg828592/opt/spack/opt/spack/.spack-db/lock[0:0] [Released]
lib/spack/spack/build_systems/intel.py:41 ==> [2019-05-21-00:04:23.501325] normalize_path.normalize_suite_dir:  /opt/intel/compilers_and_libraries_2018.2.199
lib/spack/spack/build_systems/intel.py:41 ==> [2019-05-21-00:04:23.502480] file_to_source.normalize_path:       /opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/bin/mpivars.sh
lib/spack/spack/build_systems/intel.py:1002 ==> [2019-05-21-00:04:23.502765] sourcing /opt/intel/compilers_and_libraries_2018.2.199/linux/mpi/intel64/bin/mpivars.sh
('debug', '() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then\n case "$-" in \n *v*x*)\n __lmod_sh_dbg=\'vx\'\n ;;\n *v*)\n __lmod_sh_dbg=\'v\'\n ;;\n *x*)\n __lmod_sh_dbg=\'x\'\n ;;\n esac;\n fi;\n if [ -n "${__lmod_sh_dbg')
Traceback (most recent call last):
  File "/home1/03560/tg828592/opt/spack/bin/spack", line 48, in <module>
    sys.exit(spack.main.main())
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/main.py", line 695, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/main.py", line 446, in _invoke_command
    return_val = command(parser, args)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/cmd/install.py", line 334, in install
    install_spec(args, kwargs, abstract, concrete)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/cmd/install.py", line 206, in install_spec
    install(spec, kwargs)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/cmd/install.py", line 195, in install
    spec.package.do_install(**kwargs)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/package.py", line 1406, in do_install
    return self._process_external_package(explicit)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/package.py", line 1301, in _process_external_package
    spack.hooks.post_install(self.spec)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/hooks/__init__.py", line 54, in __call__
    hook(*args, **kwargs)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/hooks/module_file_generation.py", line 29, in <lambda>
    post_install = lambda spec: _for_each_enabled(spec, 'write')
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/hooks/module_file_generation.py", line 22, in _for_each_enabled
    getattr(generator, method_name)()
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/modules/common.py", line 735, in write
    context = self.context.to_dict()
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/tengine.py", line 67, in to_dict
    d = [(name, getattr(self, name)) for name in self.context_properties]
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/modules/common.py", line 615, in environment_modifications
    x.value = self.spec.format(x.value)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/spec.py", line 3107, in format
    return self.old_format(format_string, **kwargs)
  File "/home1/03560/tg828592/opt/spack/lib/spack/spack/spec.py", line 3419, in old_format
    "'%s'" % format_string)
ValueError: Error: unterminated ${ in format:'() {  if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
 case "$-" in 
 *v*x*)
 __lmod_sh_dbg='vx'
 ;;
 *v*)
 __lmod_sh_dbg='v'
 ;;
 *x*)
 __lmod_sh_dbg='x'
 ;;
 esac;
 fi;
 if [ -n "${__lmod_sh_dbg'
```

### Information on your system

platform: [Stampede2](https://www.tacc.utexas.edu/systems/stampede2)

```console
$ module list

Currently Loaded Modules:
  1) git/2.9.0       3) cmake/3.10.2   5) TACC           7) libfabric/1.7.0   9) python2/2.7.15
  2) autotools/1.1   4) xalt/2.6.5     6) intel/18.0.2   8) impi/18.0.2
```

`compilers.yaml`:
```yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: centos7
    paths:
      cc: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc
      cxx: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icpc
      f77: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort
      fc: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort
    spec: intel@18.0.2
    target: x86_64
```

`packages.yaml`:
```yaml
packages:
  all:
    compiler: [intel]
    providers:
      mpi: [intel-mpi]
  intel-mpi:
    buildable: False
    modules:
      intel-mpi@2018.2.199: impi/18.0.2
```

Env mod 4
### Introduction

This PR provides fixes for Spack to function with [Environment Modules 4.x](https://github.com/cea-hpc/modules) and to address minor coding issues.

I factored out the main fix addressed here from #7469. As mentioned in that PR in general, the issue is fundamental and would render Spack a non-starter for new users on a similar platform.

### Issues and Fixes

* Spack threw `UnboundLocalError` when using Environment Modules 4.x.
  - Solution: Add a `continue` statement in `lib/spack/spack/environment.py:from_sourcing_file()`.

  This is the absolute minimum needed to accommodate Env-Mod-4.x.  It turns out that a changed content of `BASH_FUNC_module()` triggered this problem (see next point). 

* `BASH_FUNC_module()` and `ENV` must be blacklisted to not show up in generated modulefiles.
  - Root cause: Env-Mod-4 initializes `module()` differently, depending on the tty being interactive or not, or more specifically, where stderr goes [(see reason in code)](https://github.com/cea-hpc/modules/blob/master/modulecmd.tcl.in#L3476):

  ```sh
  $ env - bash -l -c 'env' | grep 'module()'
  BASH_FUNC_module()=() {  _moduleraw $* 2>&1
  $ env - bash -l -c 'env' 2> /dev/null | grep 'module()'
  BASH_FUNC_module()=() {  if [ "$MODULES_SILENT_SHELL_DEBUG" = '1' ]; then
  ```
  and:
  ```sh
  $ ( env;  env - bash -l -c 'env' ) | grep ^ENV
  ENV=/home/username/.bashrc
  ENV=/usr/share/Modules/init/profile.sh
  ```
  Blacklisting `BASH_FUNC_module()` happens to sidestep the initial problem, but the latter should still be fixed to address the uninitialized variable issue.

* A TypeError/NoneType was thrown if `blacklist` was in the YAML config but had all entries commented out (as happened during debugging), or equivalently has no entries.
  - Solution: Extend a relevant `if` clause.


### Background
The Environment Modules software project recently saw some major changes when Xavier Delaruelle <xavier.delaruelle@cea.fr> took on maintanence. He froze the former C-based implementation at 3.x, since it had become difficult to maintain and extend, and instead made the formerly alternative Tcl implemention the primary engine. Like Modules-C, the Modules-Tcl goes back quite some time and was mature but nonetheless new and interesting features are available with 4.x.

Pre-release versions of Environment Modules-4, like Modules-Tcl previously, did _not work at all with Spack_ for their lack of a `modulecmd` script/executable. The 4.x release version conveniently added such a script, which Spack can latch on to. For end users, more involved interfaces are available in a `module` function for Bash and numerous scripting languages, including Python. Those front-end functions support the concept of _quarantining_ variables, in particular `LD_LIBRARY_PATH`, which is needed to ensure that any user-side environment modifications do not interfere with running the Tcl interpreter for the Modules engine itself. Since Spack itself isolates module commands, however, this quarantine may not strictly be needed.

### Development note
While implementing the above, I got the impression that `lib/spack/spack/environment.py:from_sourcing_file()` looks somewhat, shall we say, "organically grown", which is concerning given the complexity of its task. That function could benefit from being split into two tasks:

1. sourcing a file and reading the resulting environment, and
2. comparing two environments stored as `dict` objects.

Factoring out task 2 should make it rather more easy to test. It could be done here or in a separate PR. Of interest might be [createmodule.py](https://github.com/cea-hpc/modules/blob/master/contrib/scripts/createmodule.py) (it is under GPLv2). It uses a slightly different algorithm, jumping off with the rather nifty construct `(prepend,append) = env2[key].split(env1[key])`. There is also a [Bash-based version](https://github.com/cea-hpc/modules/blob/master/contrib/scripts/createmodule.sh) which, remarkably, is shorter.


Discussion:

Oh my – having factored out this PR from #7469, I realized there that `blacklist` in `lib/spack/spack/environment.py:from_sourcing_file()` is _wholly unrelated to_ the one or more module-flavor-specific [environment_blacklist](http://spack.readthedocs.io/en/latest/tutorial_modules.html?highlight=blacklist#filter-unwanted-modifications-to-the-environment) entries in `modules.yaml` 🤦‍♂️.

This decouples the issues above from one another, but doesn't change their impact, in particular for the primary one.
I rolled back the change to `lib/spack/spack/util/module_cmd.py` as it won't be relevant anymore once #8570 is merged.

The other changes are still very much needed to fix bugs in `environment_modifications()` and `from_sourcing_file()`, as well as to accommodate Environment Modules 4.x as described in the Intro.
@mgsternberg @alalazo it would be great to have this PR merged to enable the use of new Modules releases (4+). What can be done to help ?
@xdelaruelle As far as I am concerned a rebase + unit tests + reply to the few minor comments above would do.
Looking at the issues reported here. Concerning:

> A TypeError/NoneType was thrown if blacklist was in the YAML config but had all entries commented out (as happened during debugging), or equivalently has no entries.

This is what Spack at 617c1a37060b975d12f8bfa4bdfd0acb0fa8a2d0 reports:

```console
$ spack config blame modules
==> Error: /home/mculpo/PycharmProjects/spack/etc/spack/modules.yaml:3: None is not of type 'array'
```

so it doesn't seem to be an issue anymore.
> Spack threw UnboundLocalError when using Environment Modules 4.x.

> BASH_FUNC_module() and ENV must be blacklisted to not show up in generated modulefiles.

I also can't reproduce the two bugs above using 617c1a37060b975d12f8bfa4bdfd0acb0fa8a2d0 with the `intel` module and `environment-modules@4.2.2`


Resolving patch:
diff --git a/lib/spack/spack/util/environment.py b/lib/spack/spack/util/environment.py
--- a/lib/spack/spack/util/environment.py
+++ b/lib/spack/spack/util/environment.py
@@ -12,9 +12,11 @@
 import re
 import sys
 import os.path
-import subprocess
+
+import six
 
 import llnl.util.tty as tty
+import spack.util.executable as executable
 
 from llnl.util.lang import dedupe
 
@@ -171,6 +173,11 @@ def __init__(self, name, **kwargs):
 
         self.args.update(kwargs)
 
+    def __eq__(self, other):
+        if not isinstance(other, NameModifier):
+            return False
+        return self.name == other.name
+
     def update_args(self, **kwargs):
         self.__dict__.update(kwargs)
         self.args.update(kwargs)
@@ -185,6 +192,13 @@ def __init__(self, name, value, **kwargs):
         self.args = {'name': name, 'value': value, 'separator': self.separator}
         self.args.update(kwargs)
 
+    def __eq__(self, other):
+        if not isinstance(other, NameValueModifier):
+            return False
+        return self.name == other.name and \
+            self.value == other.value and \
+            self.separator == other.separator
+
     def update_args(self, **kwargs):
         self.__dict__.update(kwargs)
         self.args.update(kwargs)
@@ -483,122 +497,86 @@ def shell_modifications(self, shell='sh'):
         return cmds
 
     @staticmethod
-    def from_sourcing_file(filename, *args, **kwargs):
-        """Returns modifications that would be made by sourcing a file.
-
-        Parameters:
-            filename (str): The file to source
-            *args (list of str): Arguments to pass on the command line
-
-        Keyword Arguments:
-            shell (str): The shell to use (default: ``bash``)
-            shell_options (str): Options passed to the shell (default: ``-c``)
-            source_command (str): The command to run (default: ``source``)
-            suppress_output (str): Redirect used to suppress output of command
+    def from_sourcing_file(filename, *arguments, **kwargs):
+        """Constructs an instance of a
+        :py:class:`spack.util.environment.EnvironmentModifications` object
+        that has the same effect as sourcing a file.
+
+        Args:
+            filename (str): the file to be sourced
+            *arguments (list of str): arguments to pass on the command line
+
+        Keyword Args:
+            shell (str): the shell to use (default: ``bash``)
+            shell_options (str): options passed to the shell (default: ``-c``)
+            source_command (str): the command to run (default: ``source``)
+            suppress_output (str): redirect used to suppress output of command
                 (default: ``&> /dev/null``)
-            concatenate_on_success (str): Operator used to execute a command
+            concatenate_on_success (str): operator used to execute a command
                 only when the previous command succeeds (default: ``&&``)
-            blacklist ([str or re]): Ignore any modifications of these
+            blacklist ([str or re]): ignore any modifications of these
                 variables (default: [])
-            whitelist ([str or re]): Always respect modifications of these
-                variables (default: []). Has precedence over blacklist.
-            clean (bool): In addition to removing empty entries,
+            whitelist ([str or re]): always respect modifications of these
+                variables (default: []). has precedence over blacklist.
+            clean (bool): in addition to removing empty entries,
                 also remove duplicate entries (default: False).
-
-        Returns:
-            EnvironmentModifications: an object that, if executed, has
-                the same effect on the environment as sourcing the file
         """
         # Check if the file actually exists
         if not os.path.isfile(filename):
             msg = 'Trying to source non-existing file: {0}'.format(filename)
             raise RuntimeError(msg)
 
-        # Kwargs parsing and default values
-        shell                  = kwargs.get('shell', '/bin/bash')
-        shell_options          = kwargs.get('shell_options', '-c')
-        source_command         = kwargs.get('source_command', 'source')
-        suppress_output        = kwargs.get('suppress_output', '&> /dev/null')
-        concatenate_on_success = kwargs.get('concatenate_on_success', '&&')
-        blacklist              = kwargs.get('blacklist', [])
-        whitelist              = kwargs.get('whitelist', [])
-        clean                  = kwargs.get('clean', False)
-
-        source_file = [source_command, filename]
-        source_file.extend(args)
-        source_file = ' '.join(source_file)
-
-        dump_cmd = 'import os, json; print(json.dumps(dict(os.environ)))'
-        dump_environment = 'python -c "{0}"'.format(dump_cmd)
-
-        # Construct the command that will be executed
-        command = [
-            shell,
-            shell_options,
-            ' '.join([
-                source_file, suppress_output,
-                concatenate_on_success, dump_environment,
-            ]),
-        ]
-
-        # Try to source the file
-        proc = subprocess.Popen(
-            command, stdout=subprocess.PIPE, env=os.environ)
-        proc.wait()
-
-        if proc.returncode != 0:
-            msg = 'Sourcing file {0} returned a non-zero exit code'.format(
-                filename)
-            raise RuntimeError(msg)
-
-        output = ''.join([line.decode('utf-8') for line in proc.stdout])
-
-        # Construct dictionaries of the environment before and after
-        # sourcing the file, so that we can diff them.
-        env_before = dict(os.environ)
-        env_after = json.loads(output)
-
-        # If we're in python2, convert to str objects instead of unicode
-        # like json gives us.  We can't put unicode in os.environ anyway.
-        if sys.version_info[0] < 3:
-            env_after = dict((k.encode('utf-8'), v.encode('utf-8'))
-                             for k, v in env_after.items())
+        # Prepare a whitelist and a blacklist of environment variable names
+        blacklist = kwargs.get('blacklist', [])
+        whitelist = kwargs.get('whitelist', [])
+        clean = kwargs.get('clean', False)
 
         # Other variables unrelated to sourcing a file
-        blacklist.extend(['SHLVL', '_', 'PWD', 'OLDPWD', 'PS2'])
-
-        def set_intersection(fullset, *args):
-            # A set intersection using string literals and regexs
-            meta = '[' + re.escape('[$()*?[]^{|}') + ']'
-            subset = fullset & set(args)   # As literal
-            for name in args:
-                if re.search(meta, name):
-                    pattern = re.compile(name)
-                    for k in fullset:
-                        if re.match(pattern, k):
-                            subset.add(k)
-            return subset
-
-        for d in env_after, env_before:
-            # Retain (whitelist) has priority over prune (blacklist)
-            prune = set_intersection(set(d), *blacklist)
-            prune -= set_intersection(prune, *whitelist)
-            for k in prune:
-                d.pop(k, None)
+        blacklist.extend([
+            # Bash internals
+            'SHLVL', '_', 'PWD', 'OLDPWD', 'PS1', 'PS2', 'ENV',
+            # Environment modules v4
+            'LOADEDMODULES', '_LMFILES_', 'BASH_FUNC_module()', 'MODULEPATH',
+            'MODULES_(.*)', r'(\w*)_mod(quar|share)'
+        ])
+
+        # Compute the environments before and after sourcing
+        before = sanitize(
+            dict(os.environ), blacklist=blacklist, whitelist=whitelist
+        )
+        file_and_args = (filename,) + arguments
+        after = sanitize(
+            environment_after_sourcing_files(file_and_args, **kwargs),
+            blacklist=blacklist, whitelist=whitelist
+        )
+
+        # Delegate to the other factory
+        return EnvironmentModifications.from_environment_diff(
+            before, after, clean
+        )
 
+    @staticmethod
+    def from_environment_diff(before, after, clean=False):
+        """Constructs an instance of a
+        :py:class:`spack.util.environment.EnvironmentModifications` object
+        from the diff of two dictionaries.
+
+        Args:
+            before (dict): environment before the modifications are applied
+            after (dict): environment after the modifications are applied
+            clean (bool): in addition to removing empty entries, also remove
+                duplicate entries
+        """
         # Fill the EnvironmentModifications instance
         env = EnvironmentModifications()
-
         # New variables
-        new_variables = list(set(env_after) - set(env_before))
+        new_variables = list(set(after) - set(before))
         # Variables that have been unset
-        unset_variables = list(set(env_before) - set(env_after))
+        unset_variables = list(set(before) - set(after))
         # Variables that have been modified
-        common_variables = set(env_before).intersection(set(env_after))
-
+        common_variables = set(before).intersection(set(after))
         modified_variables = [x for x in common_variables
-                              if env_before[x] != env_after[x]]
-
+                              if before[x] != after[x]]
         # Consistent output order - looks nicer, easier comparison...
         new_variables.sort()
         unset_variables.sort()
@@ -616,21 +594,21 @@ def return_separator_if_any(*args):
         # Assume that variables with 'PATH' in the name or that contain
         # separators like ':' or ';' are more likely to be paths
         for x in new_variables:
-            sep = return_separator_if_any(env_after[x])
+            sep = return_separator_if_any(after[x])
             if sep:
-                env.prepend_path(x, env_after[x], separator=sep)
+                env.prepend_path(x, after[x], separator=sep)
             elif 'PATH' in x:
-                env.prepend_path(x, env_after[x])
+                env.prepend_path(x, after[x])
             else:
                 # We just need to set the variable to the new value
-                env.set(x, env_after[x])
+                env.set(x, after[x])
 
         for x in unset_variables:
             env.unset(x)
 
         for x in modified_variables:
-            before = env_before[x]
-            after = env_after[x]
+            before = before[x]
+            after = after[x]
             sep = return_separator_if_any(before, after)
             if sep:
                 before_list = before.split(sep)
@@ -844,3 +822,108 @@ def preserve_environment(*variables):
             msg += ' {0} was set to "{1}", will be unset'
             tty.debug(msg.format(var, os.environ[var]))
             del os.environ[var]
+
+
+def environment_after_sourcing_files(*files, **kwargs):
+    """Returns a dictionary with the environment that one would have
+    after sourcing the files passed as argument.
+
+    Args:
+        *files: each item can either be a string containing the path
+            of the file to be sourced or a sequence, where the first element
+            is the file to be sourced and the remaining are arguments to be
+            passed to the command line
+
+    Keyword Args:
+        env (dict): the initial environment (default: current environment)
+        shell (str): the shell to use (default: ``/bin/bash``)
+        shell_options (str): options passed to the shell (default: ``-c``)
+        source_command (str): the command to run (default: ``source``)
+        suppress_output (str): redirect used to suppress output of command
+            (default: ``&> /dev/null``)
+        concatenate_on_success (str): operator used to execute a command
+            only when the previous command succeeds (default: ``&&``)
+    """
+    # Set the shell executable that will be used to source files
+    shell_cmd = kwargs.get('shell', '/bin/bash')
+    shell_options = kwargs.get('shell_options', '-c')
+    source_command = kwargs.get('source_command', 'source')
+    suppress_output = kwargs.get('suppress_output', '&> /dev/null')
+    concatenate_on_success = kwargs.get('concatenate_on_success', '&&')
+
+    shell = executable.Executable(' '.join([shell_cmd, shell_options]))
+
+    def _source_single_file(file_and_args, environment):
+        source_file = [source_command]
+        source_file.extend(x for x in file_and_args)
+        source_file = ' '.join(source_file)
+
+        dump_cmd = 'import os, json; print(json.dumps(dict(os.environ)))'
+        dump_environment = 'python -c "{0}"'.format(dump_cmd)
+
+        # Try to source the file
+        source_file_arguments = ' '.join([
+            source_file, suppress_output,
+            concatenate_on_success, dump_environment,
+        ])
+        output = shell(source_file_arguments, output=str, env=environment)
+        environment = json.loads(output)
+
+        # If we're in python2, convert to str objects instead of unicode
+        # like json gives us.  We can't put unicode in os.environ anyway.
+        if sys.version_info[0] < 3:
+            environment = dict(
+                (k.encode('utf-8'), v.encode('utf-8'))
+                for k, v in environment.items()
+            )
+
+        return environment
+
+    current_environment = kwargs.get('env', dict(os.environ))
+    for f in files:
+        # Normalize the input to the helper function
+        if isinstance(f, six.string_types):
+            f = [f]
+
+        current_environment = _source_single_file(
+            f, environment=current_environment
+        )
+
+    return current_environment
+
+
+def sanitize(environment, blacklist, whitelist):
+    """Returns a copy of the input dictionary where all the keys that
+    match a blacklist pattern and don't match a whitelist pattern are
+    removed.
+
+    Args:
+        environment (dict): input dictionary
+        blacklist (list of str): literals or regex patterns to be
+            blacklisted
+        whitelist (list of str): literals or regex patterns to be
+            whitelisted
+    """
+
+    def set_intersection(fullset, *args):
+        # A set intersection using string literals and regexs
+        meta = '[' + re.escape('[$()*?[]^{|}') + ']'
+        subset = fullset & set(args)  # As literal
+        for name in args:
+            if re.search(meta, name):
+                pattern = re.compile(name)
+                for k in fullset:
+                    if re.match(pattern, k):
+                        subset.add(k)
+        return subset
+
+    # Don't modify input, make a copy instead
+    environment = dict(environment)
+
+    # Retain (whitelist) has priority over prune (blacklist)
+    prune = set_intersection(set(environment), *blacklist)
+    prune -= set_intersection(prune, *whitelist)
+    for k in prune:
+        environment.pop(k, None)
+
+    return environment


Test patch:
diff --git a/lib/spack/spack/test/data/sourceme_unset.sh b/lib/spack/spack/test/data/sourceme_unset.sh
new file mode 100644
--- /dev/null
+++ b/lib/spack/spack/test/data/sourceme_unset.sh
@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+#
+# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+unset UNSET_ME
diff --git a/lib/spack/spack/test/environment_modifications.py b/lib/spack/spack/test/environment_modifications.py
--- a/lib/spack/spack/test/environment_modifications.py
+++ b/lib/spack/spack/test/environment_modifications.py
@@ -14,6 +14,9 @@
 from spack.util.environment import filter_system_paths, is_system_path
 
 
+datadir = os.path.join(spack_root, 'lib', 'spack', 'spack', 'test', 'data')
+
+
 def test_inspect_path(tmpdir):
     inspections = {
         'bin': ['PATH'],
@@ -55,7 +58,7 @@ def test_exclude_paths_from_inspection():
 
 
 @pytest.fixture()
-def prepare_environment_for_tests():
+def prepare_environment_for_tests(working_env):
     """Sets a few dummy variables in the current environment, that will be
     useful for the tests below.
     """
@@ -67,11 +70,6 @@ def prepare_environment_for_tests():
     os.environ['PATH_LIST_WITH_SYSTEM_PATHS'] \
         = '/usr/include:' + os.environ['REMOVE_PATH_LIST']
     os.environ['PATH_LIST_WITH_DUPLICATES'] = os.environ['REMOVE_PATH_LIST']
-    yield
-    for x in ('UNSET_ME', 'EMPTY_PATH_LIST', 'PATH_LIST', 'REMOVE_PATH_LIST',
-              'PATH_LIST_WITH_SYSTEM_PATHS', 'PATH_LIST_WITH_DUPLICATES'):
-        if x in os.environ:
-            del os.environ[x]
 
 
 @pytest.fixture
@@ -109,19 +107,13 @@ def miscellaneous_paths():
 @pytest.fixture
 def files_to_be_sourced():
     """Returns a list of files to be sourced"""
-    datadir = os.path.join(
-        spack_root, 'lib', 'spack', 'spack', 'test', 'data'
-    )
-
-    files = [
+    return [
         os.path.join(datadir, 'sourceme_first.sh'),
         os.path.join(datadir, 'sourceme_second.sh'),
         os.path.join(datadir, 'sourceme_parameters.sh'),
         os.path.join(datadir, 'sourceme_unicode.sh')
     ]
 
-    return files
-
 
 def test_set(env):
     """Tests setting values in the environment."""
@@ -322,8 +314,119 @@ def test_preserve_environment(prepare_environment_for_tests):
     assert os.environ['PATH_LIST'] == '/path/second:/path/third'
 
 
+@pytest.mark.parametrize('files,expected,deleted', [
+    # Sets two variables
+    ((os.path.join(datadir, 'sourceme_first.sh'),),
+     {'NEW_VAR': 'new', 'UNSET_ME': 'overridden'}, []),
+    # Check if we can set a variable to different values depending
+    # on command line parameters
+    ((os.path.join(datadir, 'sourceme_parameters.sh'),),
+     {'FOO': 'default'}, []),
+    (([os.path.join(datadir, 'sourceme_parameters.sh'), 'intel64'],),
+     {'FOO': 'intel64'}, []),
+    # Check unsetting variables
+    ((os.path.join(datadir, 'sourceme_second.sh'),),
+     {'PATH_LIST': '/path/first:/path/second:/path/fourth'},
+     ['EMPTY_PATH_LIST']),
+    # Check that order of sourcing matters
+    ((os.path.join(datadir, 'sourceme_unset.sh'),
+      os.path.join(datadir, 'sourceme_first.sh')),
+     {'NEW_VAR': 'new', 'UNSET_ME': 'overridden'}, []),
+    ((os.path.join(datadir, 'sourceme_first.sh'),
+      os.path.join(datadir, 'sourceme_unset.sh')),
+     {'NEW_VAR': 'new'}, ['UNSET_ME']),
+
+])
+@pytest.mark.usefixtures('prepare_environment_for_tests')
+def test_environment_from_sourcing_files(files, expected, deleted):
+
+    env = environment.environment_after_sourcing_files(*files)
+
+    # Test that variables that have been modified are still there and contain
+    # the expected output
+    for name, value in expected.items():
+        assert name in env
+        assert value in env[name]
+
+    # Test that variables that have been unset are not there
+    for name in deleted:
+        assert name not in env
+
+
 def test_clear(env):
     env.set('A', 'dummy value')
     assert len(env) > 0
     env.clear()
     assert len(env) == 0
+
+
+@pytest.mark.parametrize('env,blacklist,whitelist', [
+    # Check we can blacklist a literal
+    ({'SHLVL': '1'}, ['SHLVL'], []),
+    # Check whitelist takes precedence
+    ({'SHLVL': '1'}, ['SHLVL'], ['SHLVL']),
+])
+def test_sanitize_literals(env, blacklist, whitelist):
+
+    after = environment.sanitize(env, blacklist, whitelist)
+
+    # Check that all the whitelisted variables are there
+    assert all(x in after for x in whitelist)
+
+    # Check that the blacklisted variables that are not
+    # whitelisted are there
+    blacklist = list(set(blacklist) - set(whitelist))
+    assert all(x not in after for x in blacklist)
+
+
+@pytest.mark.parametrize('env,blacklist,whitelist,expected,deleted', [
+    # Check we can blacklist using a regex
+    ({'SHLVL': '1'}, ['SH.*'], [], [], ['SHLVL']),
+    # Check we can whitelist using a regex
+    ({'SHLVL': '1'}, ['SH.*'], ['SH.*'], ['SHLVL'], []),
+    # Check regex to blacklist Modules v4 related vars
+    ({'MODULES_LMALTNAME': '1', 'MODULES_LMCONFLICT': '2'},
+     ['MODULES_(.*)'], [], [], ['MODULES_LMALTNAME', 'MODULES_LMCONFLICT']),
+    ({'A_modquar': '1', 'b_modquar': '2', 'C_modshare': '3'},
+     [r'(\w*)_mod(quar|share)'], [], [],
+     ['A_modquar', 'b_modquar', 'C_modshare']),
+])
+def test_sanitize_regex(env, blacklist, whitelist, expected, deleted):
+
+    after = environment.sanitize(env, blacklist, whitelist)
+
+    assert all(x in after for x in expected)
+    assert all(x not in after for x in deleted)
+
+
+@pytest.mark.parametrize('before,after,search_list', [
+    # Set environment variables
+    ({}, {'FOO': 'foo'}, [environment.SetEnv('FOO', 'foo')]),
+    # Unset environment variables
+    ({'FOO': 'foo'}, {}, [environment.UnsetEnv('FOO')]),
+    # Append paths to an environment variable
+    ({'FOO_PATH': '/a/path'}, {'FOO_PATH': '/a/path:/b/path'},
+     [environment.AppendPath('FOO_PATH', '/b/path')]),
+    ({}, {'FOO_PATH': '/a/path:/b/path'}, [
+        environment.AppendPath('FOO_PATH', '/a/path:/b/path')
+    ]),
+    ({'FOO_PATH': '/a/path:/b/path'}, {'FOO_PATH': '/b/path'}, [
+        environment.RemovePath('FOO_PATH', '/a/path')
+    ]),
+    ({'FOO_PATH': '/a/path:/b/path'}, {'FOO_PATH': '/a/path:/c/path'}, [
+        environment.RemovePath('FOO_PATH', '/b/path'),
+        environment.AppendPath('FOO_PATH', '/c/path')
+    ]),
+    ({'FOO_PATH': '/a/path:/b/path'}, {'FOO_PATH': '/c/path:/a/path'}, [
+        environment.RemovePath('FOO_PATH', '/b/path'),
+        environment.PrependPath('FOO_PATH', '/c/path')
+    ])
+])
+def test_from_environment_diff(before, after, search_list):
+
+    mod = environment.EnvironmentModifications.from_environment_diff(
+        before, after
+    )
+
+    for item in search_list:
+        assert item in mod