Repository: spack/spack

Issue:
building zoltan fails if openmpi has been compiled with pmi/slurm-support (i.e. without --enable-static)
Configuration of _zoltan_ fails **if** _openmpi_ has been compiled with `+pmi schedulers=slurm` (and thus _without_ `--enable-static`)

### Steps to reproduce the issue

```console
$ spack install openmpi+pmi schedulers=slurm
$ spack install zoltan ^openmpi+pmi schedulers=slurm
==> Building zoltan [Package]
==> Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
    '/tmp/spack/var/spack/stage/zoltan-3.83-fixlptk6olvzooozkph2jjiutyvewaxm/Zoltan_v3.83/configure' '--prefix=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/zoltan-3.83-fixlptk6olvzooozkph2jjiutyvewaxm' '--with-cflags=-O3  -fPIC' '--with-cxxflags=-O3  -fPIC' '--with-fcflags=-O3  -fPIC' '--enable-f90interface' '--enable-mpi' 'RANLIB=echo' '--with-ar=$(CXX) -shared $(LDFLAGS) -o' '--with-libs=-lgfortran' 'CC=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpicc' 'CXX=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpic++' 'FC=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90' '--with-mpi=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr' '--with-mpi-libs=-lmpi_usempi_ignore_tkr -lmpi_usempif08 -lmpi_cxx -lompitrace -lmpi_mpifh -lmpi -lmca_common_ompio'

2 errors found in build log:
     62    checking whether to use zoltan-examples... yes
     63    checking whether additional library search paths defined... no
     64    checking whether additional include search paths defined... no
     65    checking how to get verbose linking output from /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90... -v
     66    checking for Fortran libraries of /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90...  -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/
           openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.
           5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0 -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2
           .0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../.. -lmpi_usem
           pi_ignore_tkr -lmpi_usempif08 -lmpi_cxx -lompitrace -lmpi_mpifh -lmpi -lmca_common_ompio -lgfortran -lm -lquadmath -lpthread
     67    checking for dummy main to link with Fortran libraries... unknown
  >> 68    configure: error: in `/tmp/hpcins0h/spack-stage/spack-stage-3IwHvo/Zoltan_v3.83/build':
  >> 69    configure: error: linking to Fortran libraries from C fails
     70    See `config.log' for more details

configure:6640: checking how to get verbose linking output from /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90
...
 /apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/libexec/gcc/x86_64-pc-linux-gnu/8.2.0/collect2 -plugin /apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh
3f63qrlb4mw6pmzzmgrit/libexec/gcc/x86_64-pc-linux-gnu/8.2.0/liblto_plugin.so -plugin-opt=/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/libexec/gcc/x86_64-pc-linux-gnu/8.2
.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccQPzMXR.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lquadmath -plugin-opt=-pass-through=-lm -plugin-opt=-pass-throu
gh=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -rpath /apps/SPACK/opt/linux-centos7-x8
6_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib:/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib64 --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-l
inux-x86-64.so.2 -o conftest /lib/../lib64/crt1.o /lib/../lib64/crti.o /apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/crtbegin.o -L/tmp/s
pack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -L/tmp/spack/
opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.
2.0 -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/apps/SPACK/opt/linux-centos
7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../.. /tmp/ccexUSTn.o -lmpi_usempi_ignore_tkr -lmpi_usempif08 -lmpi_cxx -lompitrace -lmpi_mpifh -lmpi -lmca_common_
ompio -lgfortran -lm -rpath /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -rpath /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw5
4uqkj5u5jfvscgjeib6jr/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgcc_s -lgcc -lquadmath -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8
.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/crtend.o /lib/../lib64/crtn.o
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_progress_is_registered'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `ompi_io_ompio_decode_datatype'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_bytes_per_agg'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_file_delete'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_coll_timing_info'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_component_progress'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_set_aggregator_props'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_fview_based_grouping'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_ompio_request_t_class'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `ompi_io_ompio_generate_current_file_view'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_get_bytes_per_agg'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_grouping_option'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_sharedfp_lazy_open'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_cycle_buffer_size'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_get_num_aggregators'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_finalize_initial_grouping'
collect2: error: ld returned 1 exit status
```

### Information on your system

- CentOS-7.5
- current develop branch without modifications

### Analysis and possible solutions

Compiling openmpi _without_ `--enabled-shared` results in "libmca_*.so" files ending up in openmpi's lib directory, including e.g. `libmca_common_ompio.so`.

The function `get_mpi_libs()` from `packages/zoltan/package.py` looks for any file in the MPI's library directory matching `r'^(lib)((\w*)mpi(\w*))\.((a)|({0}))$'.format(dso_suffix)`.

`libmca_common_ompio.so` is matched by that regular expression, however, should not be specified for (explicit) linkage.

I did not find a clever way to extend the regular expression to not match libmca* or *ompio*. Thus, my solution would be to introduce a blacklist and skip any of those, e.g. something like

```
    def get_mpi_libs(self):
        mpi_libs = set()

+       # never add one of the blacklisted libs even if they match the regex
+       blacklisted_libs = [ 'mca_common_ompio' ]

        for lib_path in glob.glob(join_path(self.spec['mpi'].prefix.lib, '*')):
            mpi_lib_match = re.match(
                r'^(lib)((\w*)mpi(\w*))\.((a)|({0}))$'.format(dso_suffix),
                os.path.basename(lib_path))
-           if mpi_lib_match:
+           if mpi_lib_match and mpi_lib_match.group(2) not in blacklisted_libs:
                mpi_libs.add(mpi_lib_match.group(2))

        return list(mpi_libs)
```
Any better idea?

At least for openmpi an empty mpi_libs list would be o.k. as the mpi-wrapper automatically pick up the required libs during the configure strep.

```
openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90 --showme:link
-fexceptions -pthread -I/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -Wl,-rpath -Wl,/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -Wl,-rpath -Wl,/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
```

Thus, an other solution might be

```
    def get_mpi_libs(self):
        if '^openmpi' in self.spec:
            return list()
        mpi_libs = set()
        ...
```

Opinions before I create a pull request for either of these?


Discussion:
Check out the [`LibraryList`](http://spack.readthedocs.io/en/latest/llnl.util.html#llnl.util.filesystem.LibraryList) and [`find_libraries`](http://spack.readthedocs.io/en/latest/llnl.util.html#llnl.util.filesystem.find_libraries). You can do something like:
```python
mpi_libs = find_libraries(['libfoo', 'libbar'], root=self.spec['mpi'].prefix.lib)
```
Just `mpi_libs = find_libraries(['lib*mpi*'], root=self.spec['mpi'].prefix.lib)` instead of the locally defined get_mpi_libs() function of course would have been a more elegant solution. But with the globs it will have the same problems, i.e. also detect **lib**mca_common_o**mpi**o.so. Going without globs is no option as the required libs (especially for Fortran linkage) probably are not known for all possible MPIs and none of the existing mpi packages defines its Fortran libs; there is only self.spec.mpicxx_shared_libs.
Thus, we still need a way for the "exclusion".
What if we define these libraries in each MPI provider? For example, if you check the `openmpi` package, it looks like you can explicitly query the `cxx` libraries like so: `spec['mpi:cxx'].libs`. In general, specifying these things in the package that provides these libraries is best. Otherwise, every single package that uses them has to figure out its own way to determining the correct libraries. Defining it in the MPI packages removes a lot of redundancies.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/zoltan/package.py b/var/spack/repos/builtin/packages/zoltan/package.py
--- a/var/spack/repos/builtin/packages/zoltan/package.py
+++ b/var/spack/repos/builtin/packages/zoltan/package.py
@@ -104,16 +104,11 @@ def install(self, spec, prefix):
 
             config_args.append('--with-mpi={0}'.format(spec['mpi'].prefix))
 
-            mpi_libs = self.get_mpi_libs()
-
-            # NOTE: Some external mpi installations may have empty lib
-            # directory (e.g. bg-q). In this case we need to explicitly
-            # pass empty library name.
-            if mpi_libs:
-                mpi_libs = ' -l'.join(mpi_libs)
-                config_args.append('--with-mpi-libs=-l{0}'.format(mpi_libs))
-            else:
-                config_args.append('--with-mpi-libs= ')
+            # NOTE: Zoltan assumes that it's linking against an MPI library
+            # that can be found with '-lmpi' which isn't the case for many
+            # MPI packages. We rely on the MPI-wrappers to automatically add
+            # what is required for linking and thus pass an empty list of libs
+            config_args.append('--with-mpi-libs= ')
 
         # NOTE: Early versions of Zoltan come packaged with a few embedded
         # library packages (e.g. ParMETIS, Scotch), which messes with Spack's
@@ -152,18 +147,3 @@ def install(self, spec, prefix):
     def get_config_flag(self, flag_name, flag_variant):
         flag_pre = 'en' if '+{0}'.format(flag_variant) in self.spec else 'dis'
         return '--{0}able-{1}'.format(flag_pre, flag_name)
-
-    # NOTE: Zoltan assumes that it's linking against an MPI library that can
-    # be found with '-lmpi,' which isn't the case for many MPI packages.  This
-    # function finds the names of the actual libraries for Zoltan's MPI dep.
-    def get_mpi_libs(self):
-        mpi_libs = set()
-
-        for lib_path in glob.glob(join_path(self.spec['mpi'].prefix.lib, '*')):
-            mpi_lib_match = re.match(
-                r'^(lib)((\w*)mpi(\w*))\.((a)|({0}))$'.format(dso_suffix),
-                os.path.basename(lib_path))
-            if mpi_lib_match:
-                mpi_libs.add(mpi_lib_match.group(2))
-
-        return list(mpi_libs)


Test patch: