Repository: spack/spack

Issue:
Installation issue: hdf5-1.8.21 lib link can't find -liconv when building against external openmpi
**snl-atdm-issue**

CC @becker33, @fryeguy52

When trying to build 'hdf5-1.8.21' using a pre-installed 'openmpi-1.10.1' (pulled in as an external package) the build of the 'libhdf5_cpp.so.16.0.0' library fails with the link error:

```
/usr/bin/ld: cannot find -liconv
```

However, if you build 'hdf5-1.8.21' with 'openmpi-1.10.1' in the same `spack install` command it works.   But we can't build 'openmpi' with the downstream TPL packages including 'hdf5' because we also want to build those with clang@5.0.1 and you can't build 'openmpi@1.10.1' with 'clang@5.0.1'.  Also, we should be able to point to a pre-installed MPI and spack should be able to build downstream packages using that MPI.

Though a lot of investigative study (documented at the end of this comment), I was able to determine that some spack package is putting `-liconv` on the link line but id not putting `-L<dir>` pointing to the spack-installed location of this library.  Iif you manually remove `-liconv`, from the raw link command:

<details>
<summary><b>Raw Link Command with -liconv</b></summary>
.

`/ascldap/users/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/bin/mpic++  -fPIC -DPIC -shared -nostdlib /usr/lib/../lib64/crti.o /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtbeginS.o  .libs/H5Exception.o .libs/H5IdComponent.o .libs/H5DataSpace.o .libs/H5PropList.o .libs/H5Library.o .libs/H5FaccProp.o .libs/H5FcreatProp.o .libs/H5LaccProp.o .libs/H5StrcreatProp.o .libs/H5LcreatProp.o .libs/H5AcreatProp.o .libs/H5DxferProp.o .libs/H5DcreatProp.o .libs/H5Location.o .libs/H5AbstractDs.o .libs/H5Attribute.o .libs/H5Object.o .libs/H5OcreatProp.o .libs/H5DataType.o .libs/H5AtomType.o .libs/H5PredType.o .libs/H5EnumType.o .libs/H5IntType.o .libs/H5FloatType.o .libs/H5StrType.o .libs/H5ArrayType.o .libs/H5VarLenType.o .libs/H5CompType.o .libs/H5DataSet.o .libs/H5CommonFG.o .libs/H5Group.o .libs/H5File.o   -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/src/.libs -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg/lib ../../src/.libs/libhdf5.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../.. /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_cxx.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.so -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.so -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.so -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.so -lrt -liconv /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.so -lm -lpthread -lc -lgcc_s /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtendS.o /usr/lib/../lib64/crtn.o    -pthread -Wl,-soname -Wl,libhdf5_cpp.so.16 -o .libs/libhdf5_cpp.so.16.0.0`

</details>

then that library links just fine.  Also, if you manually add the `-L<dir>` to the spack-installed location of the 'libiconv-1.15' lib dir in the raw link command:

<details>
<summary><b>Raw link command with -L libiconv-dir</b></summary>

.

`/ascldap/users/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/bin/mpic++  -fPIC -DPIC -shared -nostdlib /usr/lib/../lib64/crti.o /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtbeginS.o  .libs/H5Exception.o .libs/H5IdComponent.o .libs/H5DataSpace.o .libs/H5PropList.o .libs/H5Library.o .libs/H5FaccProp.o .libs/H5FcreatProp.o .libs/H5LaccProp.o .libs/H5StrcreatProp.o .libs/H5LcreatProp.o .libs/H5AcreatProp.o .libs/H5DxferProp.o .libs/H5DcreatProp.o .libs/H5Location.o .libs/H5AbstractDs.o .libs/H5Attribute.o .libs/H5Object.o .libs/H5OcreatProp.o .libs/H5DataType.o .libs/H5AtomType.o .libs/H5PredType.o .libs/H5EnumType.o .libs/H5IntType.o .libs/H5FloatType.o .libs/H5StrType.o .libs/H5ArrayType.o .libs/H5VarLenType.o .libs/H5CompType.o .libs/H5DataSet.o .libs/H5CommonFG.o .libs/H5Group.o .libs/H5File.o   -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/src/.libs -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg/lib ../../src/.libs/libhdf5.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../.. /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_cxx.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.so -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.so -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.so -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.so -lrt -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-bohu4az5xtxmslpnagcdwdr6oear4g5j/lib -liconv /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.so -lm -lpthread -lc -lgcc_s /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtendS.o /usr/lib/../lib64/crtn.o    -pthread -Wl,-soname -Wl,libhdf5_cpp.so.16 -o .libs/libhdf5_cpp.so.16.0.0`

</details>

it also links just fine.

I can't figure out what spack package is putting `-liconv` on the link line for these 'hdf5' library.  There is no mention of `iconv` at all in any of the generated build files for 'hdf5' so it must be getting pulled in from one of its upstream dependencies.

### Steps to reproduce the issue

We are using the multi-step process:

1. Install gcc@7.2.0 with spack (using [spack.compiler.yaml.txt](https://github.com/spack/spack/files/3477286/spack.compiler.yaml.txt))
2. Load the installed gcc@7.2.0 compiler as a spack compiler (which generates [compilers.yaml.txt](https://github.com/spack/spack/files/3477381/compilers.yaml.txt))
3. Build a set of "tools" packages with gcc@7.2.0 (using [spack.tools.yaml.txt](https://github.com/spack/spack/files/3477291/spack.tools.yaml.txt))
4. Build openmpi@1.10.1 using gcc@7.2.0 and pre-installed tools packages specified as external packages (using [spack.mpi.yaml.txt](https://github.com/spack/spack/files/3477296/spack.mpi.yaml.txt) and [packages_tools_for_mpi_build.yaml.txt](https://github.com/spack/spack/files/3477321/packages_tools_for_mpi_build.yaml.txt))
5. Build TPL packages (which includes 'hdf5') using gcc@7.2.0 and pre-installed openmpi@1.10.1 and tools packages specified as external packages (using [spack.tpls.yaml.txt](https://github.com/spack/spack/files/3477297/spack.tpls.yaml.txt), [packages_tools_for_tpls_build.yaml.txt](https://github.com/spack/spack/files/3477324/packages_tools_for_tpls_build.yaml.txt) and 
[packages_mpi_for_tpls_build.yaml.txt](https://github.com/spack/spack/files/3477327/packages_mpi_for_tpls_build.yaml.txt))

These all pull in:

* [common_options.yaml.txt](https://github.com/spack/spack/files/3477303/common_options.yaml.txt)

To really reproduce this, one should use the `install-trilinos-env.sh` script provided in the (currently protected repo):

* https://gitlab.sandia.gov/atdm/atdm-spack

(but I would need to push the branch 'cdofa-44-clang' for this work)

### Platform and user environment

Can report later ...

### Additional information

Someone just really needs to run out script ...

----

<details>

<summary><b>DETAILED NOTES:</b> (click to expand)</summary>

. 


I want to go back and build a fresh Spack env from scratch for the default gcc@7.2.0 + openmpi@1.10.1 build to make sure it works:

```
$ sss crf450

$ cd ~/Spack.base4/

$ rm *.out

$ . load-install-env.sh

$ time ./atdm-spack/install-trilinos-env.sh \
  --wipe-clean -a -j20 \
  &> install-trilinos-env.out

real    50m54.216s
user    230m23.372s
sys     26m46.407s
```

Hold up, this gave a build error for 'hdf5' as well:

```
==> Installing hdf5
==> Searching for binary cache of hdf5
==> Finding buildcaches in /ascldap/users/rabartl/Spack.base4/atdm-spack/atdm-spack-tarball-mirror/build_cache
==> No binary for hdf5 found: installing from source
==> Fetching file:///ascldap/users/rabartl/Spack.base4/atdm-spack/atdm-spack-tarball-mirror/hdf5/hdf5-1.8.21.tar.gz
==> Staging archive: /home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-ohmmzkjcpyq46a2yyrebk5px6xjxrw6b/hdf5-1.8.21.tar.gz
==> Created stage in /home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-ohmmzkjcpyq46a2yyrebk5px6xjxrw6b
==> No patches needed for hdf5
==> Building hdf5 [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j20'

5 errors found in build log:
     12480      CXX      H5CompType.lo
     12481      CXX      H5DataSet.lo
     12482      CXX      H5CommonFG.lo
     12483      CXX      H5Group.lo
     12484      CXX      H5File.lo
     12485      CXXLD    libhdf5_cpp.la
  >> 12486    /usr/bin/ld: cannot find -liconv
  >> 12487    collect2: error: ld returned 1 exit status
  >> 12488    make[2]: *** [libhdf5_cpp.la] Error 1
     12489    make[2]: Leaving directory `/tmp/rabartl/spack-stage/spack-stage-s_TBDI/hdf5-1.8.21/c++/src'
  >> 12490    make[1]: *** [all-recursive] Error 1
     12491    make[1]: Leaving directory `/tmp/rabartl/spack-stage/spack-stage-s_TBDI/hdf5-1.8.21/c++'
  >> 12492    make: *** [all-recursive] Error 1

See build log for details:
  /home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-ohmmzkjcpyq46a2yyrebk5px6xjxrw6b/hdf5-1.8.21/spack-build.out
```

So this may not be something wrong with clang@5.0.1 building 'hdf5'.  Something in the configurations settings that impacts 'hdf5'.  Could it be something different about the way that 'openmpi' is built?  Let's look into this in more detail.

Let's go back and build from scratch with the 'snl-atdm' branch and see what happens.  The repo versions are:

```
*** Base Git Repo: atdm-spack
b0bdb0c [Fri Aug 2 19:19:29 2019 -0600] <wg-atdm-devops-admins@sandia.gov>
Add back include of packages_tools.yaml in spack.tpls.yaml (CDOFA-52)
*** Git Repo: atdm-spack-tarball-mirror
d8a7c92 [Thu Aug 1 16:51:14 2019 -0600] <rabartl@sandia.gov>
Merge branch 'improvements-20190716-cleanup' into snl-atdm
*** Git Repo: spack
a8c458b [Thu Aug 1 16:51:15 2019 -0600] <rabartl@sandia.gov>
Merge branch 'improvements-20190716-cleanup' into snl-atdm
```

Then we do:

```
$ ssh crf450

$ cd ~/Spack.base5/

$ . load-install-env.sh

$ time ./atdm-spack/install-trilinos-env.sh \
  --wipe-clean -a -j20 \
  &> install-trilinos-env.out

real    56m48.757s
user    256m9.421s
sys     35m28.843s
```

Now to compare the conretizations of 'openmpi' and 'hdf5'.

Here is the concretization of 'openmpi' for the 'snl-atdm' branch:

```
 -   x2ipjvm              ^openmpi@1.10.1%gcc@7.2.0 cppflags="-fPIC" ~cuda+cxx_exceptions fabrics=auto ~java~legacylaunchers~memchecker patches=6c825b161b01c00d4069d002db9c43a1d8a658672e40b87d2415b228e040981d ~pmi schedulers=auto ~sqlite3~thread_multiple+vt arch=linux-rhel6-x86_64 
 -   2eoyo6v                  ^hwloc@1.11.11%gcc@7.2.0 cppflags="-fPIC" ~cairo~cuda~gl+libxml2~nvml+pci+shared arch=linux-rhel6-x86_64 
 -   zk2jyll                      ^libpciaccess@0.13.5%gcc@7.2.0 cppflags="-fPIC"  arch=linux-rhel6-x86_64 
 -   5u4ppqm                          ^util-macros@1.19.1%gcc@7.2.0 cppflags="-fPIC"  arch=linux-rhel6-x86_64 
```

Here is the concretization of 'openmi' for the 'cdofa-44-clang' branch:

```
 -   znrfpnz  openmpi@1.10.1%gcc@7.2.0~cuda+cxx_exceptions fabrics=auto ~java~legacylaunchers~memchecker patches=6c825b161b01c00d4069d002db9c43a1d8a658672e40b87d2415b228e040981d ~pmi schedulers=auto ~sqlite3~thread_multiple+vt arch=linux-rhel6-x86_64 
 -   dnaqlgt      ^hwloc@1.11.11%gcc@7.2.0~cairo~cuda~gl+libxml2~nvml+pci+shared arch=linux-rhel6-x86_64 
 -   q4netje          ^libpciaccess@0.13.5%gcc@7.2.0 arch=linux-rhel6-x86_64 
 -   yhum7yr              ^libtool@2.4.6%gcc@7.2.0 arch=linux-rhel6-x86_64 
[+]  dgpidmj              ^pkgconf@1.5.4%gcc@7.2.0 arch=linux-rhel6-x86_64 
 -   rjakshk              ^util-macros@1.19.1%gcc@7.2.0 arch=linux-rhel6-x86_64 
[+]  qckwa5f          ^libxml2@2.9.8%gcc@7.2.0~python arch=linux-rhel6-x86_64 
[+]  pnczpyw              ^libiconv@1.15%gcc@7.2.0 arch=linux-rhel6-x86_64 
[+]  4mfrw3e              ^xz@5.2.4%gcc@7.2.0 arch=linux-rhel6-x86_64 
[+]  aqhxxoj              ^zlib@1.2.11%gcc@7.2.0+optimize+pic+shared arch=linux-rhel6-x86_64 
 -   yl2h3h7          ^numactl@2.0.12%gcc@7.2.0 arch=linux-rhel6-x86_64 
 -   5hnreg6              ^autoconf@2.69%gcc@7.2.0 arch=linux-rhel6-x86_64 
 -   cbkyjff              ^automake@1.16.1%gcc@7.2.0 arch=linux-rhel6-x86_64 
 -   un6zokx              ^m4@1.4.18%gcc@7.2.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-rhel6-x86_64 
```

Let's compare these two specs a little more directly:

*'openmpi' for 'snl-atdm' branch:*

```
openmpi@1.10.1%gcc@7.2.0 cppflags="-fPIC" ~cuda+cxx_exceptions fabrics=auto ~java~legacylaunchers~memchecker patches=6c825b161b01c00d4069d002db9c43a1d8a658672e40b87d2415b228e040981d ~pmi schedulers=auto ~sqlite3~thread_multiple+vt arch=linux-rhel6-x86_64 
```

*'openmpi' for 'cdofa-44-clang' branch:*

```
openmpi@1.10.1%gcc@7.2.0 ~cuda+cxx_exceptions fabrics=auto ~java~legacylaunchers~memchecker patches=6c825b161b01c00d4069d002db9c43a1d8a658672e40b87d2415b228e040981d ~pmi schedulers=auto ~sqlite3~thread_multiple+vt arch=linux-rhel6-x86_64 
```

So the only difference that I can see is that 'cppflags=-fPIC' is used on the 'snl-atdm' branch and not the 'cdofa-44-clang' branch.  There are more dependencies built into the 'cdofa-44-clang' branch but I don't know if that is real or not.  It may just be that Spack does not show the full dependency tree.  I bet that Spack is only showing the unique conretizations and does not show every dependency over and over.

Therefore, other than 'cppflags=-fPIC', I don't see a difference between how 'openmpi' is concretized.

Let's move on to 'hdf5' concretization.

Here is how 'hdf5' is concretized on the 'snl-atdm' branch:

```
-   tmyizpm          ^hdf5@1.8.21%gcc@7.2.0+cxx~debug+fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-rhel6-x86_64 
 -   yl2h3h7              ^numactl@2.0.12%gcc@7.2.0 arch=linux-rhel6-x86_64 
 -   jpc636j                  ^autoconf@2.69%gcc@7.2.0 arch=linux-rhel6-x86_64 
 -   xruokvr                  ^automake@1.16.1%gcc@7.2.0 arch=linux-rhel6-x86_64 
 -   vdu6ks6                  ^libtool@2.4.6%gcc@7.2.0 cppflags="-fPIC"  arch=linux-rhel6-x86_64 
 -   zjo7etn                  ^m4@1.4.18%gcc@7.2.0
```

Here is how 'hdf5' is concretized on the 'snl-atdm' branch:

```
 -   ohmmzkj          ^hdf5@1.8.21%gcc@7.2.0+cxx~debug+fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-rhel6-x86_64 
[+]  yl2h3h7              ^numactl@2.0.12%gcc@7.2.0 arch=linux-rhel6-x86_64 
[+]  5hnreg6                  ^autoconf@2.69%gcc@7.2.0 arch=linux-rhel6-x86_64 
[+]  cbkyjff                  ^automake@1.16.1%gcc@7.2.0 arch=linux-rhel6-x86_64 
[+]  yhum7yr                  ^libtool@2.4.6%gcc@7.2.0 arch=linux-rhel6-x86_64 
[+]  un6zokx                  ^m4@1.4.18%gcc@7.2.0
```

Comparing just the 'hdf5' conretization.


*'hdf5' for 'snl-atdm' branch:*

```
hdf5@1.8.21%gcc@7.2.0+cxx~debug+fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-rhel6-x86_64 
```

*'hdf5' for 'cdofa-44-clang' branch:*

```
hdf5@1.8.21%gcc@7.2.0+cxx~debug+fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-rhel6-x86_64 
```

Crap, I don't see any difference in the conretization for 'hdf5'.  The only difference is using -fPIC for 'openmpi'.

So let's go back and put in 'cppflags=-fPIC' for building 'openmpi' on the 'cdofa-44-clang' branch.

I tried building again in ~/Spack.base4 on the updated 'cdofa-44-clang' branch:

```
$ cd ~/Spack.base4/

$ time ./atdm-spack/install-trilinos-env.sh -a -j20 \
  &> install-trilinos-env.3.out

real    9m10.853s
user    23m28.446s
sys     6m54.172s
```

but that failed again with the same error:

```
==> Installing hdf5
==> Searching for binary cache of hdf5
==> Finding buildcaches in /ascldap/users/rabartl/Spack.base4/atdm-spack/atdm-spack-tarball-mirror/build_cache
==> No binary for hdf5 found: installing from source
==> Using cached archive: /home/rabartl/Spack.base4/atdm-spack/spack/var/spack/cache/hdf5/hdf5-1.8.21.tar.gz
==> Staging archive: /home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21.tar.gz
==> Created stage in /home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b
==> No patches needed for hdf5
==> Building hdf5 [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j20'

5 errors found in build log:
     12480      CXX      H5CompType.lo
     12481      CXX      H5DataSet.lo
     12482      CXX      H5CommonFG.lo
     12483      CXX      H5Group.lo
     12484      CXX      H5File.lo
     12485      CXXLD    libhdf5_cpp.la
  >> 12486    /usr/bin/ld: cannot find -liconv
  >> 12487    collect2: error: ld returned 1 exit status
  >> 12488    make[2]: *** [libhdf5_cpp.la] Error 1
     12489    make[2]: Leaving directory `/tmp/rabartl/spack-stage/spack-stage-qVQQPe/hdf5-1.8.21/c++/src'
  >> 12490    make[1]: *** [all-recursive] Error 1
     12491    make[1]: Leaving directory `/tmp/rabartl/spack-stage/spack-stage-qVQQPe/hdf5-1.8.21/c++'
  >> 12492    make: *** [all-recursive] Error 1
```

So what is up with not fiking 'liconv'?  How can that be?  Let's try to directly debug this build error and see what happens.

```
$ ssh crf450

$ cd ~/Spack.base4/

$ . atdm-spack/spack/share/spack/setup-env.sh

$ spack build-env hdf5 bash

$ spack cd hdf5
```

So here is the raw command:

```
$ cd c++/src/

$ make -n
```

From that I got the libtool command for which I removed the {{--silent}} option and ran and it produced:

```
/ascldap/users/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/bin/mpic++  -fPIC -DPIC -shared -nostdlib /usr/lib/../lib64/crti.o /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtbeginS.o  .libs/H5Exception.o .libs/H5IdComponent.o .libs/H5DataSpace.o .libs/H5PropList.o .libs/H5Library.o .libs/H5FaccProp.o .libs/H5FcreatProp.o .libs/H5LaccProp.o .libs/H5StrcreatProp.o .libs/H5LcreatProp.o .libs/H5AcreatProp.o .libs/H5DxferProp.o .libs/H5DcreatProp.o .libs/H5Location.o .libs/H5AbstractDs.o .libs/H5Attribute.o .libs/H5Object.o .libs/H5OcreatProp.o .libs/H5DataType.o .libs/H5AtomType.o .libs/H5PredType.o .libs/H5EnumType.o .libs/H5IntType.o .libs/H5FloatType.o .libs/H5StrType.o .libs/H5ArrayType.o .libs/H5VarLenType.o .libs/H5CompType.o .libs/H5DataSet.o .libs/H5CommonFG.o .libs/H5Group.o .libs/H5File.o   -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/src/.libs -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg/lib ../../src/.libs/libhdf5.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../.. /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_cxx.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.so -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.so -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.so -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.so -lrt -liconv /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.so -lm -lpthread -lc -lgcc_s /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtendS.o /usr/lib/../lib64/crtn.o    -pthread -Wl,-soname -Wl,libhdf5_cpp.so.16 -o .libs/libhdf5_cpp.so.16.0.0
```

So we can see {{-liconv}}.  So where is that library?  Let's find it:

```
$ find atdm-spack/spack/opt/spack/linux-rhel6-x86_64/ -type f -name "libiconv.so*"
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-pnczpywuj4rnkqxkrsqo6fnehku4kqnr/lib/libiconv.so.2.6.0
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-bohu4az5xtxmslpnagcdwdr6oear4g5j/lib/libiconv.so.2.6.0
```

The library {{libiconv-1.15-pnczpywuj4rnkqxkrsqo6fnehku4kqnr}} is created in the building of the tools packages while building CMake:

```
[+]  rikhkft      ^cmake@3.13.4%gcc@7.2.0~doc+ncurses+openssl~ownlibs~qt arch=linux-rhel6-x86_64
...
[+]  gfsvl7p          ^libarchive@3.3.2%gcc@7.2.0 arch=linux-rhel6-x86_64
[+]  qckwa5f              ^libxml2@2.9.8%gcc@7.2.0~python arch=linux-rhel6-x86_64 
[+]  pnczpyw                  ^libiconv@1.15%gcc@7.2.0 arch=linux-rhel6-x86_64 
```

The libray {{libiconv-1.15-bohu4az5xtxmslpnagcdwdr6oear4g5j}} is created in the building of 'openmpi@1.10.1':

```
 -   gtxczlm  openmpi@1.10.1%gcc@7.2.0 cppflags="-fPIC" cxxflags="-fPIC" fflags="-fPIC" ~cuda+cxx_exceptions fabrics=auto ~java~legacylaunchers~memchecker patches=6c825b161b01c00d4069d002db9c43a1d8a658672e40b87d2415b228e040981d ~pmi schedulers=auto ~sqlite3~thread_multiple+vt arch=linux-rhel6-x86_64 
 -   agnjvnf      ^hwloc@1.11.11%gcc@7.2.0 cppflags="-fPIC" cxxflags="-fPIC" fflags="-fPIC" ~cairo~cuda~gl+libxml2~nvml+pci+shared arch=linux-rhel6-x86_64 
 -   fatkcjg          ^libpciaccess@0.13.5%gcc@7.2.0 cppflags="-fPIC" cxxflags="-fPIC" fflags="-fPIC"  arch=linux-rhel6-x86_64 
 -   jn77ea5              ^libtool@2.4.6%gcc@7.2.0 cppflags="-fPIC" cxxflags="-fPIC" fflags="-fPIC"  arch=linux-rhel6-x86_64 
 -   uf5tave              ^pkgconf@1.5.4%gcc@7.2.0 cppflags="-fPIC" cxxflags="-fPIC" fflags="-fPIC"  arch=linux-rhel6-x86_64 
 -   pvadxl3              ^util-macros@1.19.1%gcc@7.2.0 cppflags="-fPIC" cxxflags="-fPIC" fflags="-fPIC"  arch=linux-rhel6-x86_64 
 -   7pamd47          ^libxml2@2.9.8%gcc@7.2.0 cppflags="-fPIC" cxxflags="-fPIC" fflags="-fPIC" ~python arch=linux-rhel6-x86_64 
 -   bohu4az              ^libiconv@1.15%gcc@7.2.0 cppflags="-fPIC" cxxflags="-fPIC" fflags="-fPIC"  arch=linux-rhel6-x86_64 
```

The problem is that if you look above, you will not see any a {{-L}} entry for the 'libiconv' library.

If you take out '-libiconv' from the link line and run:

```
/ascldap/users/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/bin/mpic++  -fPIC -DPIC -shared -nostdlib /usr/lib/../lib64/crti.o /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtbeginS.o  .libs/H5Exception.o .libs/H5IdComponent.o .libs/H5DataSpace.o .libs/H5PropList.o .libs/H5Library.o .libs/H5FaccProp.o .libs/H5FcreatProp.o .libs/H5LaccProp.o .libs/H5StrcreatProp.o .libs/H5LcreatProp.o .libs/H5AcreatProp.o .libs/H5DxferProp.o .libs/H5DcreatProp.o .libs/H5Location.o .libs/H5AbstractDs.o .libs/H5Attribute.o .libs/H5Object.o .libs/H5OcreatProp.o .libs/H5DataType.o .libs/H5AtomType.o .libs/H5PredType.o .libs/H5EnumType.o .libs/H5IntType.o .libs/H5FloatType.o .libs/H5StrType.o .libs/H5ArrayType.o .libs/H5VarLenType.o .libs/H5CompType.o .libs/H5DataSet.o .libs/H5CommonFG.o .libs/H5Group.o .libs/H5File.o   -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/src/.libs -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg/lib ../../src/.libs/libhdf5.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../.. /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_cxx.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.so -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.so -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.so -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.so -lrt /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.so -lm -lpthread -lc -lgcc_s /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtendS.o /usr/lib/../lib64/crtn.o    -pthread -Wl,-soname -Wl,libhdf5_cpp.so.16 -o .libs/libhdf5_cpp.so.16.0.0
```

then running that command in:

```
/home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/c++/src
```

produces:

```
/home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/c++/src/.libs/libhdf5_cpp.so.16.0.0
```

Also, if I manually add:

```
-L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-bohu4az5xtxmslpnagcdwdr6oear4g5j/lib
```

to the link command:

```
/ascldap/users/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/bin/mpic++  -fPIC -DPIC -shared -nostdlib /usr/lib/../lib64/crti.o /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtbeginS.o  .libs/H5Exception.o .libs/H5IdComponent.o .libs/H5DataSpace.o .libs/H5PropList.o .libs/H5Library.o .libs/H5FaccProp.o .libs/H5FcreatProp.o .libs/H5LaccProp.o .libs/H5StrcreatProp.o .libs/H5LcreatProp.o .libs/H5AcreatProp.o .libs/H5DxferProp.o .libs/H5DcreatProp.o .libs/H5Location.o .libs/H5AbstractDs.o .libs/H5Attribute.o .libs/H5Object.o .libs/H5OcreatProp.o .libs/H5DataType.o .libs/H5AtomType.o .libs/H5PredType.o .libs/H5EnumType.o .libs/H5IntType.o .libs/H5FloatType.o .libs/H5StrType.o .libs/H5ArrayType.o .libs/H5VarLenType.o .libs/H5CompType.o .libs/H5DataSet.o .libs/H5CommonFG.o .libs/H5Group.o .libs/H5File.o   -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/src/.libs -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -Wl,-rpath -Wl,/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg/lib ../../src/.libs/libhdf5.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../.. /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_cxx.so -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64 /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.so -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.so -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.so /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.so -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.so -lrt -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-bohu4az5xtxmslpnagcdwdr6oear4g5j/lib -liconv /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.so -lm -lpthread -lc -lgcc_s /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/gcc/x86_64-pc-linux-gnu/7.2.0/crtendS.o /usr/lib/../lib64/crtn.o    -pthread -Wl,-soname -Wl,libhdf5_cpp.so.16 -o .libs/libhdf5_cpp.so.16.0.0
```

I ran that and it also linked just fine.

This shows that something in Spack or one of the intermediate package build systems is messed up and is adding `-liconv` without adding `-L<dir>` for its location.

*(8/6/2019)*

So what do I do about this?  First, it would be good to verify that the spack build where 'openmpi' is built along with 'hdf5' adds the proper `-L<dir>` option for `libiconv`.  So how do I do that?  That type of thing is so hard to do with spack ...

I did the uninstall of 'hdf5' using:

```
$ ssh crf450

$ cd ~/Spack.base5/

$ . load-install-env.sh

$ mv spack.yaml spack.yaml.moved

$ ./atdm-spack/spack/bin/spack uninstall -f hdf5
$ ./atdm-spack/spack/bin/spack uninstall -f snl-atdm-deps
```

I then tried to use the 'spack setup' function using:

```
$ mv spack.yaml.moved spack.yaml

$ . setup-env.sh

$  spack setup hdf5
==> Error: spack setup spec must have a single, concrete version. Did you forget a package version number?

$ spack setup  hdf5@1.8.21%gcc@7.2.0+cxx~debug+fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-rhel6-x86_64
==> Error: Support for AutotoolsPackage derived packages not yet implemented
```

So you can't use the 'spack setup' function :-(

Once again, it looks like I have to temporaroly break the hdf5 configuration so that I can introspect the build of the library 'libhdf5_cpp.so.16.0.0' and see what happens.

So I made the change:

```
diff --git a/var/spack/repos/builtin/packages/hdf5/package.py b/var/spack/repos/builtin/packages/hdf5/package.py
index 4b5a09d..ce32dc2 100644
--- a/var/spack/repos/builtin/packages/hdf5/package.py
+++ b/var/spack/repos/builtin/packages/hdf5/package.py
@@ -248,6 +248,8 @@ class Hdf5(AutotoolsPackage):
 
         extra_args.append('--with-zlib=%s' % self.spec['zlib'].prefix)
 
+        extra_args.append('--this-argument-does-not-exist')
+
         return extra_args
 
     @run_after('configure')
```

Then I ran it with:

```
$ ./atdm-spack/spack/bin/spack -d install

...

'/home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21/configure' '--prefix=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw' '--enable-unsupported' '--disable-threadsafe' '--enable-cxx' '--enable-hl' '--enable-fortran' '--without-szlib' '--enable-production' '--enable-fortran2003' '--enable-shared' 'CFLAGS=-fPIC' 'CXXFLAGS=-fPIC' 'FCFLAGS=-fPIC' '--enable-parallel' 'CC=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpicc' 'CXX=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpic++' 'FC=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpif90' '--with-zlib=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg' '--this-argument-does-not-exist'

1 error found in build log:
     3    ==> [2019-08-06-15:49:29.066393] '/home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21/configure' '--prefix=/home/
          rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw' '--enable-unsupported' '--disable-threadsafe' '--enable
          -cxx' '--enable-hl' '--enable-fortran' '--without-szlib' '--enable-production' '--enable-fortran2003' '--enable-shared' 'CFLAGS=-fPIC' 'CXXFLAGS=-fPIC' 'FCFLAGS=-fPIC' '--enable
          -parallel' 'CC=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpicc' 'CXX=/home/rabartl/Sp
          ack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpic++' 'FC=/home/rabartl/Spack.base5/atdm-spack/spack/opt/
          spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpif90' '--with-zlib=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_
          64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg' '--this-argument-does-not-exist'
  >> 4    configure: error: unrecognized option: `--this-argument-does-not-exist'
     5    Try `/home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21/configure --help' for more information

See build log for details:
  /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21/spack-build.out
Traceback (most recent call last):
  File "/home/rabartl/Spack.base5/atdm-spack/spack/lib/spack/spack/build_environment.py", line 767, in child_process
    return_value = function()
  File "/home/rabartl/Spack.base5/atdm-spack/spack/lib/spack/spack/package.py", line 1497, in build_process
    phase(self.spec, self.prefix)
  File "/home/rabartl/Spack.base5/atdm-spack/spack/lib/spack/spack/package.py", line 101, in phase_wrapper
    phase(spec, prefix)
  File "/home/rabartl/Spack.base5/atdm-spack/spack/lib/spack/spack/build_systems/autotools.py", line 256, in configure
    inspect.getmodule(self).configure(*options)
  File "/home/rabartl/Spack.base5/atdm-spack/spack/lib/spack/spack/util/executable.py", line 188, in __call__
    proc.returncode, long_msg)
ProcessError: Command exited with status 1:
    '/home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21/configure' '--prefix=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw' '--enable-unsupported' '--disable-threadsafe' '--enable-cxx' '--enable-hl' '--enable-fortran' '--without-szlib' '--enable-production' '--enable-fortran2003' '--enable-shared' 'CFLAGS=-fPIC' 'CXXFLAGS=-fPIC' 'FCFLAGS=-fPIC' '--enable-parallel' 'CC=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpicc' 'CXX=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpic++' 'FC=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/bin/mpif90' '--with-zlib=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg' '--this-argument-does-not-exist'
```

Now to debug this:

```
$  spack build-env hdf5 bash
==> Error: Query of package 'openmpi' for 'headers' failed
        prefix : /home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-3.1.3-rqcvehp3elb6t5ojxkze453wz5yfq6qs
        spec : openmpi@3.1.3%gcc@7.2.0~cuda+cxx_exceptions fabrics=auto ~java~legacylaunchers~memchecker~pmi schedulers=auto ~sqlite3~thread_multiple+vt arch=linux-rhel6-x86_64  ^autoconf@2.69%gcc@7.2.0 arch=linux-rhel6-x86_64  ^automake@1.16.1%gcc@7.2.0 arch=linux-rhel6-x86_64  ^hwloc@1.11.11%gcc@7.2.0~cairo~cuda~gl+libxml2~nvml+pci+shared arch=linux-rhel6-x86_64  ^libiconv@1.15%gcc@7.2.0 arch=linux-rhel6-x86_64  ^libpciaccess@0.13.5%gcc@7.2.0 arch=linux-rhel6-x86_64  ^libtool@2.4.6%gcc@7.2.0 arch=linux-rhel6-x86_64  ^libxml2@2.9.8%gcc@7.2.0~python arch=linux-rhel6-x86_64  ^m4@1.4.18%gcc@7.2.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-rhel6-x86_64  ^numactl@2.0.12%gcc@7.2.0 arch=linux-rhel6-x86_64  ^pkgconf@1.5.4%gcc@7.2.0 arch=linux-rhel6-x86_64  ^util-macros@1.19.1%gcc@7.2.0 arch=linux-rhel6-x86_64  ^xz@5.2.4%gcc@7.2.0 arch=linux-rhel6-x86_64  ^zlib@1.2.11%gcc@7.2.0+optimize+pic+shared arch=linux-rhel6-x86_64
        queried as : openmpi
        extra parameters : []
```

Darn, I can't even debug the build of hdf5 :-(

To try to debug this, I ran:

```
$ ./atdm-spack/spack/bin/spack -d install
```

and then killed it when it was doing the build:

```
==> [2019-08-06-15:59:26.655364] Using cached archive: /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/cache/hdf5/hdf5-1.8.21.tar.gz
==> [2019-08-06-15:59:26.692931] Staging archive: /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21.tar.gz
==> [2019-08-06-15:59:26.693752] '/bin/tar' '-xf' '/home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21.tar.gz'
==> [2019-08-06-15:59:27.385925] Created stage in /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw
==> [2019-08-06-15:59:27.386403] No patches needed for hdf5
==> [2019-08-06-15:59:27.386459] Building hdf5 [AutotoolsPackage]
==> [2019-08-06-15:59:27.386571] WRITE LOCK: /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/.lock[1672467032832099954:1] [Acquiring]
==> [2019-08-06-15:59:27.386750] WRITE LOCK: /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/.lock[1672467032832099954:1] [Acquired]
==> [2019-08-06-15:59:27.387201] WRITE LOCK: /home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/.spack-db/prefix_lock[5591296310307041496:1] [Acquiring]
==> [2019-08-06-15:59:27.387318] WRITE LOCK: /home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/.spack-db/prefix_lock[5591296310307041496:1] [Acquired]
==> [2019-08-06-15:59:27.389339] Executing phase: 'autoreconf'
==> [2019-08-06-15:59:27.392168] Executing phase: 'configure'
==> [2019-08-06-15:59:46.443968] Executing phase: 'build'
Signal received : entering python shell.
```

Now to look the stage space:

```
/home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw
```

which is a symlink:

```
$ ls -ld /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw
lrwxrwxrwx. 1 rabartl rabartl 43 Aug  6 15:59 /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw -> /tmp/rabartl/spack-stage/spack-stage-PNlUHR
```

Now I will try to introspect this thing:

```
$ cd /home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21/

$ find . -type f -exec grep -l libiconv {} \;
./c++/src/cpp_doc_config
./spack-build.env
./config.log
```

The file:

```
/home/rabartl/Spack.base5/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-tmyizpmttnq3d3lpjq3usvuhksngh3zw/hdf5-1.8.21/config.log
```

it shows:

```
CMAKE_PREFIX_PATH=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-zk2jyll4kclh6jtxpchjtx2nyzfmadsk:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-4mfrw3eoatxttabiyummdif5cfbx6kxo:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-2eoyo6vq3jatiet2ifbl6vbutbp5znij:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/numactl-2.0.12-yl2h3h7icf2ts3mrnndckbmz7wc4ql5o:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-qckwa5f4brssbbwbbtxe7tvdy4dnhwlw:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-pnczpywuj4rnkqxkrsqo6fnehku4kqnr:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj

...

SPACK_INCLUDE_DIRS=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-zk2jyll4kclh6jtxpchjtx2nyzfmadsk/include:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-4mfrw3eoatxttabiyummdif5cfbx6kxo/include:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-2eoyo6vq3jatiet2ifbl6vbutbp5znij/include:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg/include:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/numactl-2.0.12-yl2h3h7icf2ts3mrnndckbmz7wc4ql5o/include:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-qckwa5f4brssbbwbbtxe7tvdy4dnhwlw/include/libxml2:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-pnczpywuj4rnkqxkrsqo6fnehku4kqnr/include:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/include

...

SPACK_LINK_DIRS=/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-zk2jyll4kclh6jtxpchjtx2nyzfmadsk/lib:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-4mfrw3eoatxttabiyummdif5cfbx6kxo/lib:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-2eoyo6vq3jatiet2ifbl6vbutbp5znij/lib:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-aqhxxoj2tgtjgdkqpxtdyzo6sfwsi6vg/lib:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/numactl-2.0.12-yl2h3h7icf2ts3mrnndckbmz7wc4ql5o/lib:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-qckwa5f4brssbbwbbtxe7tvdy4dnhwlw/lib:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-pnczpywuj4rnkqxkrsqo6fnehku4kqnr/lib:/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-x2ipjvmrhhgydtcpz5i6s2ln6zjuzvqj/lib
```

So those show that Spack is putting in the stuff to to find `-liconv` under the directory:

```
/home/rabartl/Spack.base5/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-pnczpywuj4rnkqxkrsqo6fnehku4kqnr/
```

Going back to the 'cdofa-44-clang' branch build dir staging area for 'hdf5':

```
/ascldap/users/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21
```

looking for any mention of 'libiconv':

```
$ find . -type f -exec grep -l libiconv {} \;
./c++/src/cpp_doc_config
```

That file:

```
/ascldap/users/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/c++/src/cpp_doc_config
```

just shows:

```
$ find . -type f -exec grep -nH libiconv {} \;
./c++/src/cpp_doc_config:22:# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv
./c++/src/cpp_doc_config:23:# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv
./c++/src/cpp_doc_config:751:# libiconv (or the iconv built into libc) for the transcoding. See the libiconv
./c++/src/cpp_doc_config:752:# documentation (see: http://www.gnu.org/software/libiconv) for the list of
```

So this shows that Spack is **NOT** putting in any links to the correct links to the directory:

```
/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-bohu4az5xtxmslpnagcdwdr6oear4g5j/lib
```

to find `-liconv`.

So where the heck is `-liconv` comming from?  It is not coming from the hdf5 build system as shown by:

```
$ cd /ascldap/users/rabartl/Spack.base4/atdm-spack/spack/var/spack/stage/hdf5-1.8.21-d7t4gzu4w4iacxm7dbfe46773g7euf7b/hdf5-1.8.21/

$ find . -type f -exec grep -l iconv {} \;
./c++/src/cpp_doc_config
./spack-build.out
```

So one of the package that 'hdf5' is using must be putting in a bad `-liconv` without supplying the `-L<dir>` argument to find it.  What packages are doing that?

What packages does `hdf5' directly use?

```
$ ./atdm-spack/spack/bin/spack info hdf5

...

Link Dependencies:
    mpi  numactl  szip  zlib
```

Looking for the presents of 'iconv' in the installed package:

```
$ cd ~/Spack.base4/

$ find atdm-spack/spack/opt/spack/linux-rhel6-x86_64/ -type f -exec grep -l "iconv" {} \; | wc -l
809
```

Darn, 809 files.  That is too many files to try to search manually.  So what about looking for '-liconv':

```
$ cd ~/Spack.base4/

$ find atdm-spack/spack/opt/spack/linux-rhel6-x86_64/ -type f -exec grep -l "[-]liconv" {} \; | wc -l
61
```

Looking at those files, I am finding disturbing vars like:

```
$ find atdm-spack/spack/opt/spack/linux-rhel6-x86_64/ -type f -exec grep -nH "LIBICONV" {} \; | grep "\-liconv"
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/gdbm-1.18.1-scud3a6ekkn4ctjygv64fsm6u6gleyir/.spack/archived-files/config.log:8271:LIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/gdbm-1.18.1-scud3a6ekkn4ctjygv64fsm6u6gleyir/.spack/archived-files/config.log:8278:LTLIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-4mfrw3eoatxttabiyummdif5cfbx6kxo/.spack/archived-files/config.log:4064:LIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-4mfrw3eoatxttabiyummdif5cfbx6kxo/.spack/archived-files/config.log:4072:LTLIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/.spack/archived-files/config.log:4063:LIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/.spack/archived-files/config.log:4071:LTLIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libarchive-3.3.2-gfsvl7pvs7gkl4cxru4ilwblxehv443n/.spack/archived-files/config.log:48643:LIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libarchive-3.3.2-gfsvl7pvs7gkl4cxru4ilwblxehv443n/.spack/archived-files/config.log:48651:LTLIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gdbm-1.18.1-5tlzkazidylvnzafdeef5o7gm2ul4zry/.spack/archived-files/config.log:8258:LIBICONV='-liconv'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gdbm-1.18.1-5tlzkazidylvnzafdeef5o7gm2ul4zry/.spack/archived-files/config.log:8265:LTLIBICONV='-liconv'
```

This makes me concerned that these build systems are injecting `-liconv` without also injecting `-L<dir>` to find that library.

But I don't think that 'hdf5' has any direct dependencies on those TPLs.

But my bigger concern is actually 'openmp' itself as I see hits like:

```
$ find atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227 -type f -exec grep -nH "\-liconv" {} \;
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_usempi_ignore_tkr.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libgfortran.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libquadmath.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libquadmath.la'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/openmpi/libompi_dbg_msgq.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/liboshmem.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_mpifh.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.la -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libgfortran.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libquadmath.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libquadmath.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_cxx.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.la -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.la'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_usempif08.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi_mpifh.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libgfortran.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libquadmath.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libquadmath.la'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libvt-mpi.la:20:dependency_libs=' -L/tmp/rabartl/spack-stage/spack-stage-Bz5hag/openmpi-1.10.1/ompi/.libs -L/tmp/rabartl/spack-stage/spack-stage-Bz5hag/openmpi-1.10.1/ompi/contrib/vt/vt/../../../.libs /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libvt-mpi-unify.la -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libotfaux.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-trace-format.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libvt-mpi-unify.la:20:dependency_libs=' -L/tmp/rabartl/spack-stage/spack-stage-Bz5hag/openmpi-1.10.1/ompi/.libs -L/tmp/rabartl/spack-stage/spack-stage-Bz5hag/openmpi-1.10.1/ompi/contrib/vt/vt/../../../.libs /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libotfaux.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-trace-format.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.la -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.la'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libompitrace.la:20:dependency_libs=' -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libvt-hyb.la:20:dependency_libs=' -L/tmp/rabartl/spack-stage/spack-stage-Bz5hag/openmpi-1.10.1/ompi/.libs -L/tmp/rabartl/spack-stage/spack-stage-Bz5hag/openmpi-1.10.1/ompi/contrib/vt/vt/../../../.libs /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libvt-mpi-unify.la -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib -L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/zlib-1.2.11-sasjkfmh7fe56scibugcgvlrszr2i4bv/lib /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libotfaux.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-trace-format.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/gcc-7.2.0-ekndq5n53c7zwwn2fa7njakoi6atm4ss/lib/../lib64/libstdc++.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libmpi.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-rte.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227/lib/libopen-pal.la -lutil /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/hwloc-1.11.11-agnjvnfloiz5azhjr7s26icfejtoy5lp/lib/libhwloc.la -lnuma /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libpciaccess-0.13.5-fatkcjgscemiv5xuparjkrkwpu2owadt/lib/libpciaccess.la /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libxml2-2.9.8-7pamd475wc4lpzwjnfcw7cvekgtpfizg/lib/libxml2.la -ldl -lz /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/xz-5.2.4-vek6eren3xarnbj6h5lqut7rykzhwllu/lib/liblzma.la -lrt -liconv -lm'
```

So what you see is that it looks like 'openmpi' itself?

I am not sure how `-liconv` is getting onto the link line but it is clear that something is not also adding the proper `-L<dir>` argument to make this work :-(

</details>




Discussion:
@becker33, I think I can hack this to work if I can figure out how to get `-L<libiconv-dir>` on the link link.  I could hack the `hdf5/package.py` file to do this (and use an env var) but is there an easier way to get arbitrary link options onto the link line for all Spack packages?  It seems that it should be harmless to put `-L <libiconv-dir>` on every link line.

I was told by someone that I may have to write my own compiler wrappers to do stuff like this.  Is this really what I have to do to get Spack to do the right thing?

FYI: I verified that if I hack the hdf5/packages.py file as:

```
diff --git a/var/spack/repos/builtin/packages/hdf5/package.py b/var/spack/repos/builtin/packages/hdf5/package.py
index 4b5a09d..6692dd5 100644
--- a/var/spack/repos/builtin/packages/hdf5/package.py
+++ b/var/spack/repos/builtin/packages/hdf5/package.py
@@ -248,6 +248,8 @@ class Hdf5(AutotoolsPackage):
 
         extra_args.append('--with-zlib=%s' % self.spec['zlib'].prefix)
 
+        extra_args.append("LDFLAGS=-L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-bohu4az5xtxmslpnagcdwdr6oear4g5j/lib")
+
         return extra_args
 
     @run_after('configure')
```

I can get 'hdf5' to build.  But I was not able to pass this `-L<dir>` argument through the `ldflags` compiler wrapper argument to spack.  Actually, I can't even tell what arguments these are.  Does spack have a verbose mode where its compiler wrappers will be verbose showing the extra arguments it is passing into each command?  That would be very useful.

Doesn't this imply that `libiconv` should be a direct dependency of `hdf5`? Does this problem only occur with OpenMPI or does it also occur with `~mpi` or other MPI providers? Maybe OpenMPI wasn't rpathed to `libinconv` during installation, so `LD_LIBRARY_PATH` needs to be set to use it.
@adamjstewart,

As was explained in #12143, `RPATH` and `LD_LIBRARY_PATH` only impact runtime behavior of the binaries, they don't have any impact in linking.  If `-l<some-lib>` is passed into the link line, the linker needs to be be able to find that library by searching the passing in `-L<path>` paths or the standard system paths (as last resort).  So there are some bad `-liconv` listings in some package build system that is pulled in by 'hdf5' without regard for where the lib will be found :-(

FYI: Building of the 'parallel-netcdf' package also has build errors not finding `-liconv` and `-lnuma`.  I was able to hack it to get it to work with:

```
diff --git a/var/spack/repos/builtin/packages/parallel-netcdf/package.py b/var/spack/repos/builtin/packages/parallel-netcdf/package.py
index 62f81e3..507c5bf 100644
--- a/var/spack/repos/builtin/packages/parallel-netcdf/package.py
+++ b/var/spack/repos/builtin/packages/parallel-netcdf/package.py
@@ -69,6 +69,8 @@ class ParallelNetcdf(AutotoolsPackage):
         if '~fortran' in spec:
             args.append('--disable-fortran')
 
+        args.append("LDFLAGS=-L/home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/numactl-2.0.12-yl2h3h7icf2ts3mrnndckbmz7wc4ql5o/lib")
+
         return args
 
     def install(self, spec, prefix):
```

For the life of me I can't get `ldflags` to work with spack.  I am going to need to find a way to hack around `ldflags` support in spack not working.


@bartlettroscoe several points:

1. LD_LIBRARY_PATH is searched (after `-L` flags) at link time.
2. `spack install hdf5 ldflags="-L/path/to/iconv"` should install hdf5 and all of its dependencies with `-L/path/to/iconv` added to every link line. You are correct that it shouldn't cause any problems for packages that don't depend on libiconv.
3. @adamjstewart's point about OpenMPI is a good one. If the MPI compiler wrapper is emitting `-liconv` flags to expose it's dependency library for static builds, then you will need EITHER `-L` flags or `LD_LIBRARY_PATH` to point to libiconv at build time to find dependencies. Are you pulling the pre-built openmpi in via a path or via a module in your `package.yaml`?
> If the MPI compiler wrapper is emitting -liconv flags to expose it's dependency library for static builds, then you will need EITHER -L flags or LD_LIBRARY_PATH to point to libiconv at build time to find dependencies.

I am not sure what is injecting `-liconv` and `-lnuma` on the link line.  It is very hard to figure that out.  All I know is that something is adding them without the associated `-L<dir>` arguments so they can be found.

> Are you pulling the pre-built openmpi in via a path or via a module in your package.yaml?

I am pulling it in as the included file `packages_mpi_for_tpls_build.yaml` with contents:

```
packages:
  openmpi:
    buildable: False
    paths:
      openmpi@1.10.1 %gcc arch=linux-rhel6-x86_64: /ascldap/users/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/openmpi-1.10.1-gtxczlm3wtr7xm2gkuwmnk4s4vsjd227
```

Would pulling this in as a module avoid the need to manually set `ldflags="-L<dir1> -L<dir2>"?  If so, how would that look in the file `packages_mpi_for_tpls_build.yaml`?  Would it look like:

```
packages:
  openmpi:
    buildable: False
    modules:
      openmpi@1.10.1: spack-openmpi/1.10.1-gcc-7.2.0
```

?

The module `spack-openmpi/1.10.1-gcc-7.2.0` should be defined by the MPI installation step.

For that matter, should I be defining the package.yaml file entires for packages built by spack and being pulled in as external packages with `modules:` instead of `paths:`?  That would actually be more robust for packages that I know there are modules being created for them (which is the case for most of the 'tools' packages and MPI).






FYI: Turns out you can **not** apply those `-L<dir>` flags globally.  If you do, you get a 'superlu-dist' package failure:

```
  >> 500    /usr/bin/ar: -L/ascldap/users/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-7.2.0/libiconv-1.15-5kckwbisqsb43dd4hsgex2ua7225r3ps/lib: No such file or 
            directory
  >> 501    make[2]: *** [SRC/libsuperlu_dist.a] Error 1
     502    make[2]: Leaving directory `/tmp/rabartl/spack-stage/spack-stage-TlXG4W/superlu_dist-6.1.0/spack-build'
```

So now I am having to use the spec:

```
spack:
  specs: [
    snl-atdm-deps
      cflags="-fPIC"
      cxxflags="-fPIC"
      fflags="-fPIC"
      ^hdf5 ldflags="SPACK_HACK_LDLIBS_TO_REPLACE"
      ^parallel-netcdf ldflags="SPACK_HACK_LDLIBS_TO_REPLACE"
      ^SPACK_MPI_NAME_TO_REPLACE@SPACK_MPI_VER_TO_REPLACE
    ]
```

This is quite a tap dance we are having to do here to allow using an externally built MPI.

Why have other people net seen problems like this?  Is everyone just allowing spack to build MPI along with the packages?  But we can't do that because `clang@5.0.1` can't build `openmpi@1.10.1`.





What error message do you get for building `openmpi@1.10.1 %clang@5.0.1`? That's a bug in its own right, there's no reason that shouldn't build properly.
> What error message do you get for building `openmpi@1.10.1 %clang@5.0.1`?  hat's a bug in its own right, there's no reason that shouldn't build properly.

```
==> Installing openmpi
==> Searching for binary cache of openmpi
==> Finding buildcaches in /ascldap/users/rabartl/Spack.base/atdm-spack/atdm-spack-tarball-mirror/build_cache
==> No binary for openmpi found: installing from source
==> Using cached archive: /home/rabartl/Spack.base/atdm-spack/spack/var/spack/cache/openmpi/openmpi-1.10.2.tar.bz2
==> Staging archive: /home/rabartl/Spack.base/atdm-spack/spack/var/spack/stage/openmpi-1.10.2-xrivpliuwc735tk7kqik2owjnav7ma2x/openmpi-1.10.2.tar.bz2
==> Created stage in /home/rabartl/Spack.base/atdm-spack/spack/var/spack/stage/openmpi-1.10.2-xrivpliuwc735tk7kqik2owjnav7ma2x
==> No patches needed for openmpi
==> Building openmpi [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 1:
    '/home/rabartl/Spack.base/atdm-spack/spack/var/spack/stage/openmpi-1.10.2-xrivpliuwc735tk7kqik2owjnav7ma2x/openmpi-1.10.2/configure' '--prefix=/home/rabartl/Spack.base/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/clang-5.0.1/openmpi-1.10.2-xrivpliuwc735tk7kqik2owjnav7ma2x' '--enable-shared' '--with-wrapper-ldflags=' '--enable-static' '--without-pmi' '--without-psm' '--without-libfabric' '--without-ucx' '--without-mxm' '--without-verbs' '--without-psm2' '--without-alps' '--without-lsf' '--without-sge' '--without-slurm' '--without-tm' '--without-loadleveler' '--disable-memchecker' '--with-hwloc=/home/rabartl/Spack.base/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/clang-5.0.1/hwloc-1.11.11-wtlmh335kfu57csvrw6cdgbz72fvxnnr' '--disable-java' '--disable-mpi-java' '--without-sqlite3' '--disable-mpi-thread-multiple' '--without-cuda' '--enable-cxx-exceptions'

1 error found in build log:
     252    configure: WARNING:  -finline-functions has been added to CXXFLAGS
     253    checking if C and C++ are link compatible... yes
     254    checking for C++ optimization flags... -O3 -DNDEBUG -finline-functions
     255    checking size of bool... 0
     256    checking alignment of bool... configure: WARNING: *** Problem running configure test!
     257    configure: WARNING: *** See config.log for details.
  >> 258    configure: error: *** Cannot continue.

See build log for details:
  /home/rabartl/Spack.base/atdm-spack/spack/var/spack/stage/openmpi-1.10.2-xrivpliuwc735tk7kqik2owjnav7ma2x/openmpi-1.10.2/spack-build.out
```

But in any case, the SPARC and Sierra projects at SNL use gcc@4.9.3 to build openmpi@1.10.1 (not with spack obviously) and then use that to build packages with clang@5.0.1.  Therefore, we should be able to do that with Spack too.  Spack should be able to match the builds that people are already succeeding with.  We should not force people to build openmpi with clang if they want to build it with gcc.


It appears that the `.la` libtool files associated with the MPI libraries are not adding the location of libiconv, where they are for virtually all other libraries (numa is another exception). I think this is the source of the problem.

@bartlettroscoe I've found another case where this comes up internally, which is how I got the idea it could come from the `.la` files. I'll keep trying to figure out why.
We have a workaround for now which is to use scripting to find the install location of libiconv and add the `-L<path>` entry for that.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/libxml2/package.py b/var/spack/repos/builtin/packages/libxml2/package.py
--- a/var/spack/repos/builtin/packages/libxml2/package.py
+++ b/var/spack/repos/builtin/packages/libxml2/package.py
@@ -47,7 +47,8 @@ def headers(self):
     def configure_args(self):
         spec = self.spec
 
-        args = ['--with-lzma={0}'.format(spec['xz'].prefix)]
+        args = ['--with-lzma={0}'.format(spec['xz'].prefix),
+                '--with-iconv={0}'.format(spec['libiconv'].prefix)]
 
         if '+python' in spec:
             args.extend([


Test patch: