Repository: spack/spack

Issue:
Installation issue: gdal
@adamjstewart if I understand the error correctly, it appears that if any of gdal's dependencies have MPI enabled, MPI must be loaded for libgdal to link correctly.  Specifically, I propose we add an `mpi` variant to gdal and maybe couple it to enable/disable all the `mpi` aware dependencies of `gdal`.  Does that make sense?  If so I can create a pull request with the change.

### Steps to reproduce the issue

```console
$ spack env status
==> Using spack.yaml in current directory: /shared/stormcenter/Shen/src/MEX-2.3.0
$ spack install  # see spack.yaml below
...
  >> 3041    /tmp/xis14008/spack-stage/spack-stage-5eNDNt/spack-src/.libs/libgd
             al.so: undefined reference to `ompi_mpi_cxx_op_intercept'
  >> 3042    /tmp/xis14008/spack-stage/spack-stage-5eNDNt/spack-src/.libs/libgd
             al.so: undefined reference to `MPI::Datatype::Free()'
  >> 3043    /tmp/xis14008/spack-stage/spack-stage-5eNDNt/spack-src/.libs/libgd
             al.so: undefined reference to `MPI::Comm::Comm()'
  >> 3044    /tmp/xis14008/spack-stage/spack-stage-5eNDNt/spack-src/.libs/libgd
             al.so: undefined reference to `MPI::Win::Free()'
  >> 3045    collect2: error: ld returned 1 exit status
  >> 3046    make[1]: *** [gdalinfo] Error 1
     3047    make[1]: *** Waiting for unfinished jobs....
     3048    make[1]: Leaving directory `/tmp/xis14008/spack-stage/spack-stage-
             5eNDNt/spack-src/apps'
  >> 3049    make: *** [apps-target] Error 2

See build log for details:
  /shared/stormcenter/Shen/src/spack/var/spack/stage/gdal-3.0.1-4mvstpuneibqvhr45imvl3waobwa5hx6/spack-build-out.txt
```

Attached the full log file [2.3 MB]: [spack-build-out.txt](https://github.com/spack/spack/files/3543038/spack-build-out.txt)

### Platform and user environment

Please report your OS here:
```commandline
$ uname -a 
Linux cn01 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Nov 23 16:03:01 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -d
Description:    Red Hat Enterprise Linux Server release 6.7 (Santiago)
``` 

```yaml
# ~/.spack/linux/compilers.yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: rhel6
    paths:
      cc: /usr/bin/gcc
      cxx: /usr/bin/g++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    spec: gcc@4.4.7
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: rhel6
    paths:
      cc: /shared/stormcenter/Shen/src/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-6.3.0-m5d7f4djpmzkxwvwe2ggpqj2bl44enm6/bin/gcc
      cxx: /shared/stormcenter/Shen/src/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-6.3.0-m5d7f4djpmzkxwvwe2ggpqj2bl44enm6/bin/g++
      f77: /shared/stormcenter/Shen/src/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-6.3.0-m5d7f4djpmzkxwvwe2ggpqj2bl44enm6/bin/gfortran
      fc: /shared/stormcenter/Shen/src/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-6.3.0-m5d7f4djpmzkxwvwe2ggpqj2bl44enm6/bin/gfortran
    spec: gcc@6.3.0
    target: x86_64
```

```yaml
# spack.yaml

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs:
  - gdal +hdf4 +hdf5 +jasper +netcdf +crypto +grib +curl ^hdf5 +hl ^openmpi +pmi fabrics=auto
    %gcc@6.3.0
  view: false
  mirrors: {}
  repos: []
  upstreams: {}
  modules:
    enable: []
  definitions: []
  packages:
    cmake:
      buildable: false
      paths:
        cmake@3.10.2: /apps2/cmake/3.10.2
      version: []
      providers: {}
      modules: {}
      compiler: []
    gmake:
      buildable: false
      paths:
        gmake@3.81: /usr
      version: []
      providers: {}
      modules: {}
      compiler: []
  config: {}
```

### Additional information

```console
$ spack concretize -f
==> Concretized gdal+crypto+curl+grib+hdf4+hdf5+jasper+netcdf ^hdf5+hl ^openmpi%gcc@6.3.0 fabrics=auto +pmi
 -   4mvstpu  gdal@3.0.1%gcc@6.3.0~armadillo~cfitsio+crypto~cryptopp+curl~expat~geos~gif+grib+hdf4+hdf5+jasper~java+jpeg~kea~libiconv+liblzma+libtool+libz~mdb+netcdf~odbc~opencl~openjpeg~pcre~perl~pg~png~poppler+proj~python~qhull~sosi~sqlite3~xerces~xml2~zstd arch=linux-rhel6-x86_64
[+]  4vx5fvr      ^curl@7.63.0%gcc@6.3.0~darwinssl~gssapi~libssh~libssh2~nghttp2 arch=linux-rhel6-x86_64
[+]  b6r6fnb          ^openssl@1.1.1c%gcc@6.3.0+systemcerts arch=linux-rhel6-x86_64
[+]  w4idrsb              ^perl@5.26.2%gcc@6.3.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-rhel6-x86_64
[+]  qn2pvbe                  ^gdbm@1.18.1%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  gt74ike                      ^readline@7.0%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  rqwrhpn                          ^ncurses@6.1%gcc@6.3.0~symlinks~termlib arch=linux-rhel6-x86_64
[+]  cmluutb                              ^pkgconf@1.6.1%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  rjqs3cq              ^zlib@1.2.11%gcc@6.3.0+optimize+pic+shared arch=linux-rhel6-x86_64
[+]  yyiyegm      ^gmake@3.81%gcc@6.3.0~guile+nls patches=ca60bd9c1a1b35bc0dc58b6a4a19d5c2651f7a94a4b22b2c5ea001a1ca7a8a7f arch=linux-rhel6-x86_64
[+]  lyr63eg      ^hdf@4.2.14%gcc@6.3.0~szip arch=linux-rhel6-x86_64
[+]  hseynw6          ^bison@3.0.5%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  iej2wmo              ^diffutils@3.7%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  opcp74a              ^help2man@1.47.8%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  twor3y6                  ^gettext@0.19.8.1%gcc@6.3.0+bzip2+curses+git~libunistring+libxml2 patches=9acdb4e73f67c241b5ef32505c9ddf7cf6884ca8ea661692f21dca28483b04b8 +tar+xz arch=linux-rhel6-x86_64
[+]  x33qfwi                      ^bzip2@1.0.8%gcc@6.3.0+shared arch=linux-rhel6-x86_64
[+]  mw4lps6                      ^libxml2@2.9.9%gcc@6.3.0~python arch=linux-rhel6-x86_64
[+]  aso4k7z                          ^libiconv@1.15%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  o2xme3g                          ^xz@5.2.4%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  hhxm2xj                      ^tar@1.31%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  h3gzxvn              ^m4@1.4.18%gcc@6.3.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-rhel6-x86_64
[+]  kuyi76p                  ^libsigsegv@2.11%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  jay4qjn          ^flex@2.6.4%gcc@6.3.0+lex patches=09c22e5c6fef327d3e48eb23f0d610dcd3a35ab9207f12e0f875701c677978d3 arch=linux-rhel6-x86_64
[+]  3crh7lk              ^autoconf@2.69%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  6j77nxv              ^automake@1.16.1%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  of3s2pk              ^libtool@2.4.6%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  pu7o7ls          ^libjpeg-turbo@2.0.2%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  enxs2ae              ^cmake@3.10.2%gcc@6.3.0~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-rhel6-x86_64
[+]  5f2wqfv              ^nasm@2.14.02%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  u2ffeby      ^hdf5@1.10.5%gcc@6.3.0~cxx~debug~fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-rhel6-x86_64
[+]  oxprqis          ^openmpi@3.1.4%gcc@6.3.0~cuda+cxx_exceptions fabrics=auto ~java~legacylaunchers~memchecker+pmi schedulers=none ~sqlite3~thread_multiple+vt arch=linux-rhel6-x86_64
[+]  njk5j57              ^hwloc@1.11.11%gcc@6.3.0~cairo~cuda~gl+libxml2~nvml+pci+shared arch=linux-rhel6-x86_64
[+]  bkmqzbd                  ^libpciaccess@0.13.5%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  5u6bbgi                      ^util-macros@1.19.1%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  nxxlgnc                  ^numactl@2.0.12%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  e2ecn35      ^jasper@1.900.1%gcc@6.3.0 build_type=Release +jpeg~opengl patches=95a654159688eaffdf035a467914c5953068ad0fd62d18229de5779a17ec380b,db104400a2e72f610b8fa4d061a32282254819c70b024ef1cf99fef64aca67e3 +shared arch=linux-rhel6-x86_64
[+]  qsmbnxo      ^json-c@0.13.1%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  l5srte6      ^libgeotiff@1.5.1%gcc@6.3.0+jpeg patches=9485efc0a62a02207d34ac0c4d22e421c975b6ce85397c5e557c0105a232aaa3,ac81bd50db5950b42e48de78845ad91c60bc107ccaa1acd78836e03fdf4a2313,e0d45d3c34bf92df2d1d140957f110dc84759420e68a97e1e3d6ab90c81777d8 +proj+zlib arch=linux-rhel6-x86_64
[+]  poxlp53          ^libtiff@4.0.10%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  25tmaus          ^proj@6.1.0%gcc@6.3.0 arch=linux-rhel6-x86_64
[+]  qpmcb6k              ^sqlite@3.28.0%gcc@6.3.0+fts~functions arch=linux-rhel6-x86_64
[+]  gxgix6l      ^netcdf@4.7.0%gcc@6.3.0~dap~hdf4 maxdims=1024 maxvars=8192 +mpi~parallel-netcdf patches=10a1c3f7fa05e2c82457482e272bbe04d66d0047b237ad0a73e87d63d848b16c +pic+shared arch=linux-rhel6-x86_64
```



Discussion:
Hmm, there are a couple points of confusion I have. I've built GDAL with both `hdf5+mpi` and `hdf5~mpi` without any problems in the past. Also, if you build `gdal ^hdf5+mpi`, then Spack automatically adds transitive dependencies like OpenMPI to the compile line, so it should be RPATHed anyway.

A couple things you can try:

- Add `depends_on('mpi')` to the GDAL `package.py`
- Set `CC` and friends to `mpicc` and friends in the `package.py`. See other packages for examples

If neither of these work, I can try reproducing your issue, but your best bet is to submit a bug report to the GDAL developers.

When I build GDAL, I only use `+hdf4+hdf5+java+mdb+jasper+openjpeg+sqlite3+python`, so I've never tested `+netcdf`before. Maybe try building `~netcdf` and see if we can narrow down the problem.
My spec lists `hdf5+mpi`

I can get it to compile with your advice: `depends_on('mpi')` isn't sufficient; I also need to pass `mpicc`, `mpicxx`:

```diff
diff --git a/var/spack/repos/builtin/packages/gdal/package.py b/var/spack/repos/builtin/packages/gdal/package.py
index a8b4d85..03b96c0 100644
--- a/var/spack/repos/builtin/packages/gdal/package.py
+++ b/var/spack/repos/builtin/packages/gdal/package.py
@@ -91,6 +91,7 @@ class Gdal(AutotoolsPackage):
     depends_on('pkgconfig@0.25:', type='build')
 
     # Required dependencies
+    depends_on('mpi')
     depends_on('libtiff@3.6.0:')  # 3.9.0+ needed to pass testsuite
     depends_on('libgeotiff@1.2.1:')
     depends_on('libgeotiff@:1.4', when='@:2.4')
@@ -168,6 +169,8 @@ class Gdal(AutotoolsPackage):
 
         # Required dependencies
         args = [
+            'CC={0}'.format(self.spec['mpi'].mpicc),
+            'CXX={0}'.format(self.spec['mpi'].mpicxx),
             # https://trac.osgeo.org/gdal/wiki/TIFF
             '--with-libtiff={0}'.format(spec['libtiff'].prefix),
             '--with-geotiff={0}'.format(spec['libgeotiff'].prefix),
```

I'll revert and test `~netcdf` and other variants to find the culprit.
- I can reproduce the error simply with `gdal +hdf5 %gcc@6.3.0`.
- Maybe it has to do with my using `mpi` compiled by spack?  A lot of sites then to source their environmental modules for their `mpi` providers.  What `mpi` implementation do you use for your GDAL installation?

I need to use `%gcc@6.3.0` compiler for my HPC user because MATLAB pins their releases against gcc versions for compiling MEX files, and I'm trying to use the spack built gdal to compile the MEX file.  Generally MATLAB MEX files compile just as well with other gcc versions, but I would prefer not to play Monday night football with MATLAB support if I need their help diagnosing why my user's code intermittent crashes only on our Skylake CPU nodes.
> What mpi implementation do you use for your GDAL installation?

I'm using an external Cray Mpich installation and it seems to be linking just fine:
```console
$ ldd -r /mnt/a/u/sciteam/stewart1/spack/opt/spack/cray-cnl5-interlagos/gcc-6.3.0/gdal-3.0.1-ktbycl2kbp6i3e3hz5k7cit3hkjqnjsf/lib/libgdal.so | grep mpi
	libmpich_gnu_51.so.3 => /opt/cray/mpt/7.5.0/gni/mpich-gnu/5.1/lib/libmpch_gnu_51.so.3 (0x00002aaaaef7a000)
	libmpichcxx_gnu_51.so.3 => /opt/cray/mpt/7.5.0/gni/mpich-gnu/5.1/lib/limpichcxx_gnu_51.so.3 (0x00002aaaaf50d000)
	libsci_gnu_51_mpi.so.5 => /opt/cray/lib64/libsci_gnu_51_mpi.so.5 (0x00002aaaaf732000)
```
You could try mpich and see if the problem is specific to OpenMPI.
I also experienced the same problem. As a temporary workaround, I ended up with disabling mpi for hdf5 and netcdf for install to work:
```
gdal+hdf5+netcdf ^hdf5~mpi ^netcdf~mpi
```
It would be great to get this sorted out.
> You could try mpich and see if the problem is specific to OpenMPI.

Indeed, the MPI linking error is specific to OpenMPI because gdal compiles fine with mpich:

```bash
spack install gdal +hdf5 ^mpich netmod=ofi ^libfabric@:1.6 %gcc@6.3.0
```
@omsai In that case, I think you should try contacting the GDAL developers and see what they recommend. I don't want to hack the `package.py` unless that is the recommended way of building GDAL with MPI support.
It turns out this issue was reported in OSGeo/gdal#1260 and the lead developer suggested directly adding the OpenMPI library `ld` flags.  Which is exactly your suggestion from earlier which we know works around the problem:
 
>     * Add `depends_on('mpi')` to the GDAL `package.py`
>     * Set `CC` and friends to `mpicc` and friends in the `package.py`. See other packages for examples

Setting `mpicc` and `mpicxx` doesn't feel like a clean solution because GDAL doesn't make any MPI calls.  It points more to some sort of misbehavior of OpenMPI.

[Stackoverflow has one suggestion](https://stackoverflow.com/a/52899800) to set the `OMPI_SKIP_MPICXX` environmental variable to disable the C++ warppers which would make sense if none of the GDAL dependencies actually use the MPI C++ interface.  As you may know, the MPI standard only supported a C++ interface in MPI-2, but not MPI-3 and later dropped support for it.
Thanks @omsai. See if `OMPI_SKIP_MPICXX` works. If not, we'll have to set `CC` and friends.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/hdf5/package.py b/var/spack/repos/builtin/packages/hdf5/package.py
--- a/var/spack/repos/builtin/packages/hdf5/package.py
+++ b/var/spack/repos/builtin/packages/hdf5/package.py
@@ -111,6 +111,11 @@ class Hdf5(AutotoolsPackage):
           sha256='57cee5ff1992b4098eda079815c36fc2da9b10e00a9056df054f2384c4fc7523',
           when='@1.10.2%gcc@8:')
 
+    # Disable MPI C++ interface when C++ is disabled, otherwise downstream
+    # libraries fail to link; see https://github.com/spack/spack/issues/12586
+    patch('h5public-skip-mpicxx.patch', when='+mpi~cxx',
+          sha256='b61e2f058964ad85be6ee5ecea10080bf79e73f83ff88d1fa4b602d00209da9c')
+
     filter_compiler_wrappers('h5cc', 'h5c++', 'h5fc', relative_root='bin')
 
     def url_for_version(self, version):


Test patch: