Repository: spack/spack

Issue:
MPI as external package not working
I'm on a Linux system and I use external packages for the compiler and MPI. The compiler works fine, but a very simply TPL won't build due to MPI issues (actually *hwloc*). This is a new issue that appeared over the last few weeks.

```
% spack install parmetis % intel@17.0.4 ^ openmpi@1.10.5
==> openmpi@1.10.5 : has external module in openmpi/1.10.5
==> openmpi@1.10.5 : is actually installed in /usr/projects/hpcsoft/toss3/snow/openmpi/1.10.5-intel-17.0.4
==> openmpi@1.10.5 : generating module file
==> Error: 'No spec with name hwloc in openmpi@1.10.5%intel@17.0.4~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-rhel7-x86_64'
```

* packages.yaml
```
packages:
  openmpi:
    version: [1.10.5]
    modules:
      openmpi@1.10.5: openmpi/1.10.5
```

* The spec seems happy:

```
% spack spec parmetis % intel@17.0.4 ^ openmpi@1.10.5
Input spec
--------------------------------
parmetis%intel@17.0.4
    ^openmpi@1.10.5

Normalized
--------------------------------
parmetis%intel@17.0.4
    ^cmake@2.8:
    ^metis@5:
    ^openmpi@1.10.5
        ^hwloc
            ^pkg-config

Concretized
--------------------------------
parmetis@4.0.3%intel@17.0.4~debug~gdb+shared arch=linux-rhel7-x86_64 
    ^cmake@3.9.0%intel@17.0.4~doc+ncurses+openssl+ownlibs~qt arch=linux-rhel7-x86_64 
    ^metis@5.1.0%intel@17.0.4~debug~gdb~int64~real64+shared arch=linux-rhel7-x86_64 
    ^openmpi@1.10.5%intel@17.0.4~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-rhel7-x86_64 
```

Also:
* Using `--dirty` doesn't help.
* Same error with different versions of openmpi.
* I'm not sure why spack needs to know about hwloc since only the MPI dependency needs it and MPI is *external*.

Suggestions or workarounds are welcome.


Discussion:
On a whim, I tried the `--only package` option to `spack install` and that allows me to move forward.  But this behavior seems broken.
Could you post debug output? (spack -d install ...)
I think I have seen this myself but don't remember precisely ATM.
Ah - I found the issue.  It was a stupid user issue.  I had an openmpi module already loaded in my environment.  I think spack was failing to load the new openmpi module (as specified in packages.yaml) due to a module-conflict.  

That '-d' option is really nice. I didn't know about it.  Thanks!

@KineticTheory : I am seeing this issue again if I switched to latest develop (see #3581). Are you using latest develop?
@pramodskumbhar Yes, I'm seeing this error again with the most recent 'develop' branch of spack.  I did a fresh clone into a new directory.  I backed up to a Aug 1 checkout of spack that is working for me right now.  With today's 'develop', I see the following:

```
% spack clean -a; spack -d install parmetis%intel@17.0.4 ^openmpi@2.1.2 
==> Removing all temporary build stages
==> Removing cached downloads
==> Removing cached information on repositories
==> '/bin/ibv_devices'
==> Reading config file /yellow/usr/projects/jayenne/devs/kellyt/build/spack/etc/spack/defaults/packages.yaml
==> Reading config file /yellow/usr/projects/jayenne/devs/kellyt/build/spack/etc/spack/linux/packages.yaml
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquiring]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquired]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Released]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.lanl-index.yaml.lock[0:0] [Acquiring]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.lanl-index.yaml.lock[0:0] [Acquired]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.lanl-index.yaml.lock[0:0] [Released]
==> Reading config file /yellow/usr/projects/jayenne/devs/kellyt/build/spack/etc/spack/linux/compilers.yaml
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'friendly-testing'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'load' 'friendly-testing'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'intel/17.0.4'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'load' 'intel/17.0.4'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'cmake/3.9.0'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'friendly-testing'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'load' 'friendly-testing'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'intel/17.0.4'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'load' 'intel/17.0.4'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'openmpi/2.1.2'
==> WRITE LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Acquiring]
==> WRITE LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> WRITE LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Released]
==> link /yellow/usr/projects/jayenne/devs/kellyt/build/spack/var/spack/stage/parmetis-4.0.3-ybxxde65n24qd5qg23e5llddfrr473zo -> /tmp/kellyt/spack-stage/spack-stage-5Ij4tt
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Acquiring]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Released]
==> Installing parmetis dependencies
==> openmpi@2.1.2 : has external module in openmpi/2.1.2
==> openmpi@2.1.2 : is actually installed in /usr/projects/hpcsoft/toss3/snow/openmpi/2.1.2-intel-17.0.4
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> openmpi@2.1.2 : generating module file
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> 	WRITE: openmpi@2.1.2%intel@17.0.4~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-rhel7-x86_64 /7jp3nk7 [/yellow/usr/projects/jayenne/devs/kellyt/build/spack/share/spack/modules/linux-rhel7-x86_64/openmpi-2.1.2-intel-17.0.4-7jp3nk7]
==> '/bin/ibv_devices'
Traceback (most recent call last):
  File "/users/kellyt/build/spack/bin/spack", line 81, in <module>
    sys.exit(spack.main.main())
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/main.py", line 548, in main
    return _main(command, parser, args, unknown)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/main.py", line 440, in _main
    return _invoke_spack_command(command, parser, args, unknown_args)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/main.py", line 356, in _invoke_spack_command
    return_val = command(parser, args)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/cmd/install.py", line 379, in install
    package.do_install(**kwargs)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/package.py", line 1286, in do_install
    **kwargs
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/package.py", line 1252, in do_install
    return self._process_external_package(explicit)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/package.py", line 1191, in _process_external_package
    spack.hooks.post_install(self.spec)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/hooks/__init__.py", line 73, in __call__
    hook(*args, **kwargs)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/hooks/module_file_generation.py", line 43, in <lambda>
    post_install = lambda spec: _for_each_enabled(spec, 'write')
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/hooks/module_file_generation.py", line 40, in _for_each_enabled
    getattr(generator, method_name)()
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/modules/common.py", line 640, in write
    context = self.context.to_dict()
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/tengine.py", line 85, in to_dict
    d = [(name, getattr(self, name)) for name in self.context_properties]
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/modules/common.py", line 464, in configure_options
    configure_args = getattr(pkg, attr)()
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/var/spack/repos/builtin/packages/openmpi/package.py", line 305, in configure_args
    config_args.append('--with-hwloc={0}'.format(spec['hwloc'].prefix))
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/spec.py", line 2647, in __getitem__
    raise KeyError("No spec with name %s in %s" % (name, self))
KeyError: 'No spec with name hwloc in openmpi@2.1.2%intel@17.0.4~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-rhel7-x86_64'
```
Just adding my findings here too:

```shell
mamelara@cori09 ~ $ spack install abinit%intel+hdf5+openmp+scalapack ^intel-mkl
==> libxc is already installed in /global/homes/m/mamelara/common/sw/cray-cnl6-haswell/libxc/2.2.2/intel/17.0.2.174/jvk2ots
==> hdf5@1.10.0.3 : has external module in cray-hdf5-parallel
==> hdf5@1.10.0.3 : is actually installed in /opt/cray/pe/hdf5-parallel/1.10.0.3/INTEL/16.0
==> hdf5@1.10.0.3 : generating module file
==> Error: 'No spec with name mpi in hdf5@1.10.0.3%intel@17.0.2.174+cxx~debug+fortran+mpi+pic~shared~szip~threadsafe arch=cray-cnl6-haswell'
```
I've traced the bug down to configure_options in the BaseContext class in modules/common.py

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/plasma/package.py b/var/spack/repos/builtin/packages/plasma/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin/packages/plasma/package.py
@@ -0,0 +1,90 @@
+##############################################################################
+# Copyright (c) 2017, Innovative Computing Laboratory
+# Produced at the Innovative Computing Laboratory.
+#
+# Created by Piotr Luszczek, luszczek@icl.utk.edu, All rights reserved.
+#
+# For details, see https://github.com/llnl/spack
+#
+##############################################################################
+#
+from spack import *
+
+
+class Plasma(MakefilePackage):
+    """Parallel Linear Algebra Software for Multicore Architectures, PLASMA is
+    a software package for solving problems in dense linear algebra using
+    multicore processors and Xeon Phi coprocessors. PLASMA provides
+    implementations of state-of-the-art algorithms using cutting-edge task
+    scheduling techniques. PLASMA currently offers a collection of routines for
+    solving linear systems of equations, least squares problems, eigenvalue
+    problems, and singular value problems."""
+
+    homepage = "https://bitbucket.org/icl/plasma/"
+    url      = "https://bitbucket.org/icl/plasma/downloads/plasma-17.1.tar.gz"
+
+    version("17.1", "64b410b76023a41b3f07a5f0dca554e1")
+
+    version("develop", hg="https://luszczek@bitbucket.org/icl/plasma")
+
+    depends_on("blas")
+    depends_on("lapack")
+
+    conflicts("atlas")  # does not have LAPACKE interface
+    conflicts("netlib-lapack@:2.999")  # missing LAPACKE features
+    # clashes with OpenBLAS declarations and has a problem compiling on its own
+    conflicts("cblas")
+    conflicts("openblas-with-lapack")  # incomplete LAPACK implementation
+    conflicts("veclibfort")
+
+    # only GCC 7+ and higher have sufficient support for OpenMP 4+ tasks+deps
+    conflicts("%gcc@:6.999")
+    conflicts("%cce")
+    conflicts("%clang")
+    conflicts("%intel")
+    conflicts("%nag")
+    conflicts("%pgi")
+    conflicts("%xl")
+    conflicts("%xl_r")
+
+    patch("remove_absolute_mkl_include.patch", when="@17.1")
+
+    def edit(self, spec, prefix):
+        # copy "make.inc.mkl-gcc" provided by default into "make.inc"
+        open("make.inc", "w").write(open("make.inc.mkl-gcc").read())
+
+        make_inc = FileFilter("make.inc")
+
+        if not spec.satisfies("^mkl"):
+            make_inc.filter("-DPLASMA_WITH_MKL", "")  # not using MKL
+
+        header_flags = ""
+        # accumulate CPP flags for headers: <cblas.h> and <lapacke.h>
+        for dep in ("blas", "lapack"):
+            try:  # in case the dependency does not provide header flags
+                header_flags += " " + spec[dep].headers.cpp_flags
+            except:
+                pass
+
+        make_inc.filter("CFLAGS +[+]=", "CFLAGS += " + header_flags + " ")
+
+        # pass prefix variable from "make.inc" to "Makefile"
+        make_inc.filter("# --*", "prefix={0}".format(self.prefix))
+
+        # make sure CC variable comes from build environment
+        make_inc.filter("CC *[?]*= * .*cc", "")
+
+    @property
+    def build_targets(self):
+        targets = list()
+
+        # use $CC set by Spack
+        targets.append("CC = {0}".format(self.compiler.cc))
+
+        if self.spec.satisfies("^mkl"):
+            targets.append("MKLROOT = {0}/mkl".format(env["MKLROOT"]))
+
+        # pass BLAS library flags
+        targets.append("LIBS = {0}".format(self.spec["blas"].libs.ld_flags))
+
+        return targets


Test patch: