Repository: spack/spack

Issue:
Want to pass `--with-verbs=DIR` to OpenMPI
When I install the OpenMPI package on Stampede (TACC), it doesn't detect that Infiniband is present. When I pass the `+verbs` option, then the configure stage fails, since OpenMPI doesn't correctly auto-detect the location of the libraries.

I can run `which(ibv_devices)` to find the location of this utility (`/opt/ofed`), and pass `--with-verbs=/opt/ofed` to OpenMPI, and this seems to work.

How should I build this into OpenMPI? Options are:
- do this in the OpenMPI package, changing how `+verbs` is interpreted, or even changing its default value
- add a new option `+verbs-autodetect` that does this
- look at an environment variable `VERBS_DIR` that the caller has to set



Discussion:
The 3rd idea looks like the easiest. If `VERBS_DIR` is set, use it. Otherwise, don't specify the directory and let OpenMPI search for it wherever it usually does.

@adamjstewart: I think passing a user-defined env var kind of goes against some of Spack's reproducibility aspirations, and adds another step for the user.

We've got two somewhat conflicting goals with Spack:
1. We want to run on HPC hardware, where the environment is fickle and we have to rely on vendor networks and potentially MPI implementations.
2. We want to be able to generate combinatorial versions of things, and record _exactly_ what was used, because scientific software is picky and often something special is needed to get things running.

(2) conflicts with (1) in the sense that if you use some env var, you're leaving a combinatorial decision up to the environment, which means a build with hash xyz123 could be subtly different between builds.  If the env var were actually set by the OS, I might say go for it, b/c that's the mechanism the platform likes to use, but in this case the suggestion is a user-defined variable.

I actually like the option to call `which(ibv_devices)`, because the auto-detection is at least automatic and only platform-dependent.  Spack already kind of punts on versioning any of the underlying OS libs (as they may be custom vendor things and we want to work with what is there) so detecting the network this way seems most similar to how Spack already works.

So I'd go with option 1, and maybe leave in support for the old verbs semantics so that we can still build old versions (if that is possible).

We are merging platform support soon (maybe next week -- #561) and it will include a "platform" class that could provide this information in a more robust way to the packages.  We could move the auto-detection there once it's merged.

Thoughts?

I may be wrong, but since yesterday variants should have become more than just a boolean. This, I think, opens up the possibility to say something like : 

```
spack install openmpi verbs=/opt/ofed
```

Is what I say above correct, @tgamblin , @becker33 ?

@alalazo: I was going to mention that but I think this case is more inline with what we've discussed (abstractly) about further parameterizing the platform/arch. Basically it's a vendor network with vendor drivers, so I think we want to auto detect what is there.

Of course, if you want to allow the package author to override, we could handle the platform case _and_ the combinatorial case with the `key=val` syntax, e.g. offer both:

```
    spack install openmpi    # verbs defaults to `autodetect` and grabs from platform.
    spack install openmpi verbs=/path/to/my/ofed    # user override
```

This would actually offer some flexibility for, say, trying new drivers or ofed versions.  Of course, then you might ask whether Spack should just have an `ofed` package and handle this in the DAG...  I think the case above could easily become an anti-pattern if used poorly.

I had a similar problem with tm:  our intel compiled mpi is referring to a custom installation of PBS: I used a workaround and defined a void package tm that is referred in package.yaml
Are The variant in path already available in develop? That would let me avoid this stupid void package hack.
Please let me know so I can rework the PR for letting intel compiler build a working openmpi on my platform

@tgamblin I get your point. What I had in mind is more in line with your second proposal : 

```
spack install openmpi  # Default is no verbs, so it will configure with --without-verbs
spack install openmp verbs=True  # Use --with-verbs and autodetection
spack install openmp verbs=<dir>  # Use --with-verbs=<dir>
```

@alalazo: I like it (`+verbs` would also work there).  The support in `develop` for `key=val` isn't quite ready for this yet, though.  We've put in support for `cflags` but not enough is there yet for arbitrary strings/bools/ints, so I'd go with `which(ibv_devices)` with this in mind.  I like the flexibility too.  I just don't want everyone putting in variants instead of dependencies.

Ok, so currently no support for general variant=value, so could go on with the current PR shape using void tm package


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/openmpi/package.py b/var/spack/repos/builtin/packages/openmpi/package.py
--- a/var/spack/repos/builtin/packages/openmpi/package.py
+++ b/var/spack/repos/builtin/packages/openmpi/package.py
@@ -27,6 +27,26 @@
 from spack import *
 
 
+def _verbs_dir():
+    """
+    Try to find the directory where the OpenFabrics verbs package is
+    installed. Return None if not found.
+    """
+    try:
+        # Try to locate Verbs by looking for a utility in the path
+        ibv_devices = which("ibv_devices")
+        # Run it (silently) to ensure it works
+        ibv_devices(output=str, error=str)
+        # Get path to executable
+        path = ibv_devices.exe[0]
+        # Remove executable name and "bin" directory
+        path = os.path.dirname(path)
+        path = os.path.dirname(path)
+        return path
+    except:
+        return None
+
+
 class Openmpi(Package):
     """Open MPI is a project combining technologies and resources from
        several other projects (FT-MPI, LA-MPI, LAM/MPI, and PACX-MPI)
@@ -54,7 +74,7 @@ class Openmpi(Package):
     variant('psm', default=False, description='Build support for the PSM library.')
     variant('psm2', default=False, description='Build support for the Intel PSM2 library.')
     variant('pmi', default=False, description='Build support for PMI-based launchers')
-    variant('verbs', default=False, description='Build support for OpenFabrics verbs.')
+    variant('verbs', default=_verbs_dir() is not None, description='Build support for OpenFabrics verbs.')
     variant('mxm', default=False, description='Build Mellanox Messaging support')
 
     variant('thread_multiple', default=False, description='Enable MPI_THREAD_MULTIPLE support')
@@ -113,7 +133,6 @@ def install(self, spec, prefix):
             # Fabrics
             '--with-psm' if '+psm' in spec else '--without-psm',
             '--with-psm2' if '+psm2' in spec else '--without-psm2',
-            ('--with-%s' % self.verbs) if '+verbs' in spec else ('--without-%s' % self.verbs),
             '--with-mxm' if '+mxm' in spec else '--without-mxm',
             # Other options
             '--enable-mpi-thread-multiple' if '+thread_multiple' in spec else '--disable-mpi-thread-multiple',
@@ -121,6 +140,14 @@ def install(self, spec, prefix):
             '--with-sqlite3' if '+sqlite3' in spec else '--without-sqlite3',
             '--enable-vt' if '+vt' in spec else '--disable-vt'
         ])
+        if '+verbs' in spec:
+            path = _verbs_dir()
+            if path is not None:
+                config_args.append('--with-%s=%s' % (self.verbs, path))
+            else:
+                config_args.append('--with-%s' % self.verbs)
+        else:
+            config_args.append('--without-%s' % self.verbs)
 
         # TODO: use variants for this, e.g. +lanl, +llnl, etc.
         # use this for LANL builds, but for LLNL builds, we need:


Test patch: