Repository: spack/spack

Issue:
MPI as external package not working
I'm on a Linux system and I use external packages for the compiler and MPI. The compiler works fine, but a very simply TPL won't build due to MPI issues (actually *hwloc*). This is a new issue that appeared over the last few weeks.

```
% spack install parmetis % intel@17.0.4 ^ openmpi@1.10.5
==> openmpi@1.10.5 : has external module in openmpi/1.10.5
==> openmpi@1.10.5 : is actually installed in /usr/projects/hpcsoft/toss3/snow/openmpi/1.10.5-intel-17.0.4
==> openmpi@1.10.5 : generating module file
==> Error: 'No spec with name hwloc in openmpi@1.10.5%intel@17.0.4~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-rhel7-x86_64'
```

* packages.yaml
```
packages:
  openmpi:
    version: [1.10.5]
    modules:
      openmpi@1.10.5: openmpi/1.10.5
```

* The spec seems happy:

```
% spack spec parmetis % intel@17.0.4 ^ openmpi@1.10.5
Input spec
--------------------------------
parmetis%intel@17.0.4
    ^openmpi@1.10.5

Normalized
--------------------------------
parmetis%intel@17.0.4
    ^cmake@2.8:
    ^metis@5:
    ^openmpi@1.10.5
        ^hwloc
            ^pkg-config

Concretized
--------------------------------
parmetis@4.0.3%intel@17.0.4~debug~gdb+shared arch=linux-rhel7-x86_64 
    ^cmake@3.9.0%intel@17.0.4~doc+ncurses+openssl+ownlibs~qt arch=linux-rhel7-x86_64 
    ^metis@5.1.0%intel@17.0.4~debug~gdb~int64~real64+shared arch=linux-rhel7-x86_64 
    ^openmpi@1.10.5%intel@17.0.4~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-rhel7-x86_64 
```

Also:
* Using `--dirty` doesn't help.
* Same error with different versions of openmpi.
* I'm not sure why spack needs to know about hwloc since only the MPI dependency needs it and MPI is *external*.

Suggestions or workarounds are welcome.


Discussion:
On a whim, I tried the `--only package` option to `spack install` and that allows me to move forward.  But this behavior seems broken.
Could you post debug output? (spack -d install ...)
I think I have seen this myself but don't remember precisely ATM.
Ah - I found the issue.  It was a stupid user issue.  I had an openmpi module already loaded in my environment.  I think spack was failing to load the new openmpi module (as specified in packages.yaml) due to a module-conflict.  

That '-d' option is really nice. I didn't know about it.  Thanks!

@KineticTheory : I am seeing this issue again if I switched to latest develop (see #3581). Are you using latest develop?
@pramodskumbhar Yes, I'm seeing this error again with the most recent 'develop' branch of spack.  I did a fresh clone into a new directory.  I backed up to a Aug 1 checkout of spack that is working for me right now.  With today's 'develop', I see the following:

```
% spack clean -a; spack -d install parmetis%intel@17.0.4 ^openmpi@2.1.2 
==> Removing all temporary build stages
==> Removing cached downloads
==> Removing cached information on repositories
==> '/bin/ibv_devices'
==> Reading config file /yellow/usr/projects/jayenne/devs/kellyt/build/spack/etc/spack/defaults/packages.yaml
==> Reading config file /yellow/usr/projects/jayenne/devs/kellyt/build/spack/etc/spack/linux/packages.yaml
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquiring]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Acquired]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.builtin-index.yaml.lock[0:0] [Released]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.lanl-index.yaml.lock[0:0] [Acquiring]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.lanl-index.yaml.lock[0:0] [Acquired]
==> WRITE LOCK: /users/kellyt/.spack/cache/providers/.lanl-index.yaml.lock[0:0] [Released]
==> Reading config file /yellow/usr/projects/jayenne/devs/kellyt/build/spack/etc/spack/linux/compilers.yaml
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'friendly-testing'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'load' 'friendly-testing'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'intel/17.0.4'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'load' 'intel/17.0.4'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'cmake/3.9.0'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'friendly-testing'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'load' 'friendly-testing'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'intel/17.0.4'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'load' 'intel/17.0.4'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'list'
==> '/usr/share/lmod/lmod/libexec/lmod' 'python' 'show' 'openmpi/2.1.2'
==> WRITE LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Acquiring]
==> WRITE LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> WRITE LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Released]
==> link /yellow/usr/projects/jayenne/devs/kellyt/build/spack/var/spack/stage/parmetis-4.0.3-ybxxde65n24qd5qg23e5llddfrr473zo -> /tmp/kellyt/spack-stage/spack-stage-5Ij4tt
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Acquiring]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/prefix_lock[6933213328793099392:1] [Released]
==> Installing parmetis dependencies
==> openmpi@2.1.2 : has external module in openmpi/2.1.2
==> openmpi@2.1.2 : is actually installed in /usr/projects/hpcsoft/toss3/snow/openmpi/2.1.2-intel-17.0.4
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> openmpi@2.1.2 : generating module file
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> READ LOCK: /yellow/usr/projects/jayenne/devs/kellyt/build/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> 	WRITE: openmpi@2.1.2%intel@17.0.4~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-rhel7-x86_64 /7jp3nk7 [/yellow/usr/projects/jayenne/devs/kellyt/build/spack/share/spack/modules/linux-rhel7-x86_64/openmpi-2.1.2-intel-17.0.4-7jp3nk7]
==> '/bin/ibv_devices'
Traceback (most recent call last):
  File "/users/kellyt/build/spack/bin/spack", line 81, in <module>
    sys.exit(spack.main.main())
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/main.py", line 548, in main
    return _main(command, parser, args, unknown)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/main.py", line 440, in _main
    return _invoke_spack_command(command, parser, args, unknown_args)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/main.py", line 356, in _invoke_spack_command
    return_val = command(parser, args)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/cmd/install.py", line 379, in install
    package.do_install(**kwargs)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/package.py", line 1286, in do_install
    **kwargs
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/package.py", line 1252, in do_install
    return self._process_external_package(explicit)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/package.py", line 1191, in _process_external_package
    spack.hooks.post_install(self.spec)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/hooks/__init__.py", line 73, in __call__
    hook(*args, **kwargs)
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/hooks/module_file_generation.py", line 43, in <lambda>
    post_install = lambda spec: _for_each_enabled(spec, 'write')
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/hooks/module_file_generation.py", line 40, in _for_each_enabled
    getattr(generator, method_name)()
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/modules/common.py", line 640, in write
    context = self.context.to_dict()
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/tengine.py", line 85, in to_dict
    d = [(name, getattr(self, name)) for name in self.context_properties]
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/modules/common.py", line 464, in configure_options
    configure_args = getattr(pkg, attr)()
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/var/spack/repos/builtin/packages/openmpi/package.py", line 305, in configure_args
    config_args.append('--with-hwloc={0}'.format(spec['hwloc'].prefix))
  File "/yellow/usr/projects/jayenne/devs/kellyt/build/spack/lib/spack/spack/spec.py", line 2647, in __getitem__
    raise KeyError("No spec with name %s in %s" % (name, self))
KeyError: 'No spec with name hwloc in openmpi@2.1.2%intel@17.0.4~cuda fabrics=verbs ~java schedulers= ~sqlite3~thread_multiple+vt arch=linux-rhel7-x86_64'
```
Just adding my findings here too:

```shell
mamelara@cori09 ~ $ spack install abinit%intel+hdf5+openmp+scalapack ^intel-mkl
==> libxc is already installed in /global/homes/m/mamelara/common/sw/cray-cnl6-haswell/libxc/2.2.2/intel/17.0.2.174/jvk2ots
==> hdf5@1.10.0.3 : has external module in cray-hdf5-parallel
==> hdf5@1.10.0.3 : is actually installed in /opt/cray/pe/hdf5-parallel/1.10.0.3/INTEL/16.0
==> hdf5@1.10.0.3 : generating module file
==> Error: 'No spec with name mpi in hdf5@1.10.0.3%intel@17.0.2.174+cxx~debug+fortran+mpi+pic~shared~szip~threadsafe arch=cray-cnl6-haswell'
```
I've traced the bug down to configure_options in the BaseContext class in modules/common.py
I'll try to look at this tomorrow (9/29)
@KineticTheory @mamelara do you have relevant dependencies for your respective cases defined as externals in packages.yaml?
@scheibelp : in above OpenMPI case even if I add `hwloc` entry in `packages.yaml`, nothing changes.
@scheibelp I was pointed to this issue from a discussion in the modules channel on Slack. I'll probably have a fix ready by today - so don't worry diving into it.

Resolving patch:
diff --git a/lib/spack/spack/modules/common.py b/lib/spack/spack/modules/common.py
--- a/lib/spack/spack/modules/common.py
+++ b/lib/spack/spack/modules/common.py
@@ -457,16 +457,24 @@ def long_description(self):
     def configure_options(self):
         pkg = self.spec.package
 
+        # If the spec is external Spack doesn't know its configure options
+        if self.spec.external:
+            msg = 'unknown, software installed outside of Spack'
+            return msg
+
         # This is quite simple right now, but contains information on how
         # to call different build system classes.
         for attr in ('configure_args', 'cmake_args'):
             try:
                 configure_args = getattr(pkg, attr)()
                 return ' '.join(configure_args)
-            except (AttributeError, IOError):
+            except (AttributeError, IOError, KeyError):
+                # The method doesn't exist in the current spec,
+                # or it's not usable
                 pass
 
-        # The default is to return None
+        # Returning a false-like value makes the default templates skip
+        # the configure option section
         return None
 
     @tengine.context_property


Test patch:
diff --git a/lib/spack/spack/test/modules/lmod.py b/lib/spack/spack/test/modules/lmod.py
--- a/lib/spack/spack/test/modules/lmod.py
+++ b/lib/spack/spack/test/modules/lmod.py
@@ -236,3 +236,13 @@ def test_override_template_in_modules_yaml(
 
         content = modulefile_content('mpileaks arch=x86-linux')
         assert 'Override even better!' in content
+
+    @pytest.mark.usefixtures('config')
+    def test_external_configure_args(
+            self, factory
+    ):
+        # If this package is detected as an external, its configure option line
+        # in the module file starts with 'unknown'
+        writer, spec = factory('externaltool')
+
+        assert 'unknown' in writer.context.configure_options