Repository: spack/spack

Issue:
Error installing OpenMPI+CUDA with external CUDA
I added a `packages.yaml` to take an external cuda installation via
```yaml
packages:
  cuda:
    paths:
      cuda@8.0.61%gcc@5.4.0 arch=linux-ubuntu16-x86_64: /usr/local/cuda
    buildable: False
```

When trying to install a cuda-aware MPI via `openmpi+cuda` it fails (on dev as of 4553b14) since it is kind of weirdly adding the `--with-cuda-libdir` as

`'--with-cuda=/usr/local/cuda' '--with-cuda-libdir=['"'"'/usr/local/cuda/targets/x86_64-linux/lib/stubs'"'"']'`

```
==> Building openmpi [AutotoolsPackage]
==> Error: ProcessError: Command exited with status 1:
    '/home/src/spack/var/spack/stage/openmpi-2.1.1-w22h44f3bzru6544vvhkioccwqiwbmze/openmpi-2.1.1/configure' '--prefix=/home/src/spack/opt/spack/linux-ubuntu16-x86_64/gcc-5.4.0/openmpi-2.1.1-w22h44f3bzru6544vvhkioccwqiwbmze' '--enable-shared' '--enable-static' '--enable-mpi-cxx' '--without-psm' '--without-psm2' '--without-pmi' '--without-verbs' '--without-mxm' '--without-alps' '--without-lsf' '--without-tm' '--without-slurm' '--without-sge' '--without-loadleveler' '--with-hwloc=/home/src/spack/opt/spack/linux-ubuntu16-x86_64/gcc-5.4.0/hwloc-1.11.7-nsd5ek5ld357qgjjhkn3ep2w3zzq2pmu' '--disable-java' '--disable-mpi-java' '--disable-mpi-thread-multiple' '--with-cuda=/usr/local/cuda' '--with-cuda-libdir=['"'"'/usr/local/cuda/targets/x86_64-linux/lib/stubs'"'"']'
```

Here is a docker file to reproduce the issue via `docker build .` (add the `packages.yaml` in the same dir):
```
FROM       nvidia/cuda:8.0-devel

# general environment for docker
ENV        HOME /home
ENV        DEBIAN_FRONTEND noninteractive

# spack environment
ENV        PATH "$HOME/src/spack/bin:$PATH"
COPY       packages.yaml $HOME/.spack/packages.yaml

# install minimal spack dependencies
#   also adds gfortran for spack's openmpi package
RUN        apt-get update \
           && apt-get install -y --no-install-recommends \
              autoconf \
              ca-certificates \
              curl \
              gfortran \
              git \
              build-essential \
              python \
           && rm -rf /var/lib/apt/lists/*

# install spack
RUN        git clone --depth 50 https://github.com/llnl/spack.git \
           $HOME/src/spack/

# install PIConGPU dependencies
RUN        spack install cmake \
           && spack install cuda \
           && spack install openmpi+cuda
```


Discussion:

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/openmpi/package.py b/var/spack/repos/builtin/packages/openmpi/package.py
--- a/var/spack/repos/builtin/packages/openmpi/package.py
+++ b/var/spack/repos/builtin/packages/openmpi/package.py
@@ -190,6 +190,7 @@ class Openmpi(AutotoolsPackage):
     depends_on('jdk', when='+java')
     depends_on('sqlite', when='+sqlite3@:1.11')
 
+    conflicts('+cuda', when='@:1.6')  # CUDA support was added in 1.7
     conflicts('fabrics=psm2', when='@:1.8')  # PSM2 support was added in 1.10.0
     conflicts('fabrics=pmi', when='@:1.5.4')  # PMI support was added in 1.5.5
     conflicts('fabrics=mxm', when='@:1.5.3')  # MXM support was added in 1.5.4
@@ -310,12 +311,29 @@ def configure_args(self):
                 config_args.append('--disable-mpi-thread-multiple')
 
         # CUDA support
+        # See https://www.open-mpi.org/faq/?category=buildcuda
         if spec.satisfies('@1.7:'):
             if '+cuda' in spec:
+                # OpenMPI dynamically loads libcuda.so, requires dlopen
+                config_args.append('--enable-dlopen')
+                # Searches for header files in DIR/include
                 config_args.append('--with-cuda={0}'.format(
                     spec['cuda'].prefix))
-                config_args.append('--with-cuda-libdir={0}'.format(
-                    spec['cuda'].libs.directories))
+                if spec.satisfies('@1.7:1.7.2'):
+                    # This option was removed from later versions
+                    config_args.append('--with-cuda-libdir={0}'.format(
+                        spec['cuda'].libs.directories[0]))
+                if spec.satisfies('@1.7.2'):
+                    # There was a bug in 1.7.2 when --enable-static is used
+                    config_args.append('--enable-mca-no-build=pml-bfo')
+                if spec.satisfies('%pgi^cuda@7.0:7.999'):
+                    # OpenMPI has problems with CUDA 7 and PGI
+                    config_args.append(
+                        '--with-wrapper-cflags=-D__LP64__ -ta:tesla')
+                    if spec.satisfies('%pgi@:15.8'):
+                        # With PGI 15.9 and later compilers, the
+                        # CFLAGS=-D__LP64__ is no longer needed.
+                        config_args.append('CFLAGS=-D__LP64__')
             else:
                 config_args.append('--without-cuda')
 


Test patch: