Repository: spack/spack

Issue:
Installation issue: mpich: undefined reference to `hwloc_topology_set_io_types_filter'
### Steps to reproduce the issue

I am trying to install firedrake on a cluster and would like to compile a new mpi. Part of the concretization results are listed below:
```console
 -   xffgfev          ^mpich@4.0.2%gcc@9.5.0~argobots~cuda+fortran+hwloc+hydra+libxml2+pci~rocm+romio~slurm~two_level_namespace~verbs+wrapperrpath datatype-engine=auto device=ch3 netmod=ofi pmi=pmi arch=linux-rhel6-ivybridge
[+]  yyr6dvc              ^findutils@4.9.0%gcc@9.5.0 patches=440b954 arch=linux-rhel6-ivybridge
[+]  i7hw5xi              ^hwloc@2.7.1%gcc@9.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm+shared arch=linux-rhel6-ivybridge
[+]  rjxaj6q                  ^libpciaccess@0.16%gcc@9.5.0 arch=linux-rhel6-ivybridge
[+]  otnpivf                      ^libtool@2.4.7%gcc@9.5.0 arch=linux-rhel6-ivybridge
[+]  f5lis6l                          ^m4@1.4.19%gcc@9.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-rhel6-ivybridge
[+]  edvnqxs                              ^libsigsegv@2.13%gcc@9.5.0 arch=linux-rhel6-ivybridge
[+]  edinkzp                      ^util-macros@1.19.3%gcc@9.5.0 arch=linux-rhel6-ivybridge
[+]  wenbrq5                  ^libxml2@2.9.13%gcc@9.5.0~python arch=linux-rhel6-ivybridge
[+]  bonjmeb                      ^xz@5.2.5%gcc@9.5.0~pic libs=shared,static arch=linux-rhel6-ivybridge
 -   tmynx25              ^libfabric@1.6.2%gcc@9.5.0~debug~kdreg fabrics=sockets,tcp,udp arch=linux-rhel6-ivybridge
```

In the building process, the linker will always reach to `/usr/lib64/libhwloc.so` other than the version installed by spack.
It seems the option `--with-hwloc-prefix` does not work. Even if I replace it with two other options: `-with-hwloc-include` and `--with-hwloc-lib`, the problem remains.

After I add the corresponding path of `hwloc` into `CFLAGS` and `LDFLAGS` in mpich's file `package.py`, the problem is gone.

@raffenet @yfguo, could you help find the reason? Thanks in advance!

### Error message

<details><summary>Error message</summary><pre>
/bin/sh ./libtool  --tag=CC   --mode=link /vol7/home/lrtfm/opt/spack/lib/spack/env/gcc/gcc  -g  -L.  -o mpiexec.hydra ui/mpich/mpiexec.o ui/mpich/utils.o -lpm -lhydra /tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/mpl/libmpl.la   -lpthread  -lhwloc
libtool: link: /vol7/home/lrtfm/opt/spack/lib/spack/env/gcc/gcc -g -o mpiexec.hydra ui/mpich/mpiexec.o ui/mpich/utils.o  -L. /tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra/.libs/libpm.a /tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra/.libs/libhydra.a /tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/mpl/.libs/libmpl.a -L/vol7/home/lrtfm/opt/spack/opt/spack/linux-rhel6-ivybridge/gcc-9.5.0/libfabric-1.14.1-2lx2gc4bktgpxgvkf3z2csotw6sq2wfc/lib -lpthread /usr/lib64/libhwloc.so -lm -lnuma -lxml2
/vol7/home/lrtfm/opt/spack/opt/spack/linux-rhel6-ivybridge//gcc-9.2.0vol7/binutils/-2.38home-/royhsidjkisjn7tmarni7kjty6n57k2i/lrtfmbin//ldopt:/ spack//tmpopt//lrtfm/spackspack-/stage/linuxspack--stagerhel6-mpich--4.0.2ivybridge-/konutm24znsihjf4vvbxugdr6rdwzumy/gccspack--src9.2.0/src//binutilspm/-hydra2.38/-.libsroyhsidjkisjn7tmarni7kjty6n57k2i//libhydra.a(bintopo_hwloc.o)/:ld :in  function/ `tmpHYDT_topo_hwloc_init/':lrtfm
//tmpspack/lrtfm-/stagespack/-stagespack/spack--stagestage--mpich-mpich4.0.2--konutm24znsihjf4vvbxugdr6rdwzumy/4.0.2spack--src/konutm24znsihjf4vvbxugdr6rdwzumysrc//pmspack/-hydra/srctools//toposrc/hwloc//topo_hwloc.cpm:/577:hydra undefined/ .reference libsto/ `libhydra.ahwloc_topology_set_io_types_filter('topo_hwloc.o
): in function `HYDT_topo_hwloc_init':
/tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter'
collect2: error: ld returned 1 exit status
collect2: error: ld returned 1 exit status
make[3]: *** [hydra_nameserver] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: *** [hydra_persist] Error 1
/vol7/home/lrtfm/opt/spack/opt/spack/linux-rhel6-ivybridge/gcc-9.2.0/binutils-2.38-royhsidjkisjn7tmarni7kjty6n57k2i/bin/ld: /tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init':
/tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter'
collect2: error: ld returned 1 exit status
make[3]: *** [hydra_pmi_proxy] Error 1
/vol7/home/lrtfm/opt/spack/opt/spack/linux-rhel6-ivybridge/gcc-9.2.0/binutils-2.38-royhsidjkisjn7tmarni7kjty6n57k2i/bin/ld: /tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra/.libs/libhydra.a(topo_hwloc.o): in function `HYDT_topo_hwloc_init':
/tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c:577: undefined reference to `hwloc_topology_set_io_types_filter'
collect2: error: ld returned 1 exit status
make[3]: *** [mpiexec.hydra] Error 1
make[3]: Leaving directory `/tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src/src/pm/hydra'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/lrtfm/spack-stage/spack-stage-mpich-4.0.2-konutm24znsihjf4vvbxugdr6rdwzumy/spack-src'
make: *** [all] Error 2

</pre></details>


### Information on your system

* **Spack:** 0.19.0.dev0 (6a240c9b392d9d519b5df14943732bd1831d25bf)
* **Python:** 3.8.13
* **Platform:** linux-rhel6-ivybridge
* **Concretizer:** clingo


### Additional information

[spack-build-env.txt](https://github.com/spack/spack/files/9220278/spack-build-env.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/9220280/spack-build-out.txt)


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
@lrtfm is the problem also present if you try to build the latest MPICH (mpich@develop)? We've made a lot of changes to dependency management in configure recently, but its possible not everything was backported to the 4.0.x branch.
I will try the `develop` version tomorrow.

Did you mean that this problem may only relate to the 4.0.x version, i.e. the 3.x is fine?
> I will try the `develop` version tomorrow.
> 
> Did you mean that this problem may only relate to the 4.0.x version, i.e. the 3.x is fine?

3.4.x is likely fine, since it will default to use the embedded hwloc included with MPICH and ignore whatever is installed on the system. In 4.0.x we switched to prefer an external hwloc library by default.
Thank you for your clarification!
> @lrtfm is the problem also present if you try to build the latest MPICH (mpich@develop)? We've made a lot of changes to dependency management in configure recently, but its possible not everything was backported to the 4.0.x branch.

@raffenet the problem is still there, and I notice that there are some failed operations in the configuration. The `spec` and logs are attached.
[spack-build-env.txt](https://github.com/spack/spack/files/9224153/spack-build-env.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/9224154/spack-build-out.txt)

```
$ spack spec mpich@develop
==> Warning: Missing a source id for mpich@develop
Input spec
--------------------------------
mpich@develop

Concretized
--------------------------------
mpich@develop%gcc@9.5.0~argobots~cuda+fortran+hwloc+hydra+libxml2+pci~rocm+romio~slurm~two_level_namespace~vci~verbs+wrapperrpath datatype-engine=auto device=ch4 netmod=ofi pmi=pmi arch=linux-rhel6-ivybridge
    ^autoconf@2.69%gcc@9.5.0 patches=35c4492,7793209,a49dd5b arch=linux-rhel6-ivybridge
        ^m4@1.4.19%gcc@9.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-rhel6-ivybridge
            ^diffutils@3.8%gcc@9.5.0 arch=linux-rhel6-ivybridge
                ^libiconv@1.16%gcc@9.5.0 libs=shared,static arch=linux-rhel6-ivybridge
            ^libsigsegv@2.13%gcc@9.5.0 arch=linux-rhel6-ivybridge
        ^perl@5.34.1%gcc@9.5.0+cpanm+shared+threads arch=linux-rhel6-ivybridge
            ^berkeley-db@18.1.40%gcc@9.5.0+cxx~docs+stl patches=b231fcc arch=linux-rhel6-ivybridge
            ^bzip2@1.0.8%gcc@9.5.0~debug~pic+shared arch=linux-rhel6-ivybridge
            ^gdbm@1.19%gcc@9.5.0 arch=linux-rhel6-ivybridge
                ^readline@8.1.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^ncurses@6.2%gcc@9.5.0~symlinks+termlib abi=none arch=linux-rhel6-ivybridge
                        ^pkgconf@1.8.0%gcc@9.5.0 arch=linux-rhel6-ivybridge
            ^zlib@1.2.12%gcc@9.5.0+optimize+pic+shared patches=0d38234 arch=linux-rhel6-ivybridge
    ^automake@1.16.5%gcc@9.5.0 arch=linux-rhel6-ivybridge
    ^findutils@4.9.0%gcc@9.5.0 patches=440b954 arch=linux-rhel6-ivybridge
    ^hwloc@2.7.1%gcc@9.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm+shared arch=linux-rhel6-ivybridge
        ^libpciaccess@0.16%gcc@9.5.0 arch=linux-rhel6-ivybridge
            ^libtool@2.4.7%gcc@9.5.0 arch=linux-rhel6-ivybridge
            ^util-macros@1.19.3%gcc@9.5.0 arch=linux-rhel6-ivybridge
        ^libxml2@2.9.13%gcc@9.5.0~python arch=linux-rhel6-ivybridge
            ^xz@5.2.5%gcc@9.5.0~pic libs=shared,static arch=linux-rhel6-ivybridge
    ^libfabric@1.14.1%gcc@9.5.0~debug~kdreg fabrics=sockets,tcp,udp arch=linux-rhel6-ivybridge
    ^python@3.9.13%gcc@9.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-rhel6-ivybridge
        ^expat@2.4.8%gcc@9.5.0+libbsd arch=linux-rhel6-ivybridge
            ^libbsd@0.11.5%gcc@9.5.0 arch=linux-rhel6-ivybridge
                ^libmd@1.0.4%gcc@9.5.0 arch=linux-rhel6-ivybridge
        ^gettext@0.21%gcc@9.5.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-rhel6-ivybridge
            ^tar@1.34%gcc@9.5.0 zip=pigz arch=linux-rhel6-ivybridge
                ^pigz@2.7%gcc@9.5.0 arch=linux-rhel6-ivybridge
                ^zstd@1.5.2%gcc@9.5.0+programs compression=none libs=shared,static arch=linux-rhel6-ivybridge
        ^libffi@3.4.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
        ^openssl@1.1.1q%gcc@9.5.0~docs~shared certs=mozilla patches=3fdcf2d arch=linux-rhel6-ivybridge
            ^ca-certificates-mozilla@2022-03-29%gcc@9.5.0 arch=linux-rhel6-ivybridge
        ^sqlite@3.38.5%gcc@9.5.0+column_metadata+dynamic_extensions+fts~functions+rtree arch=linux-rhel6-ivybridge
        ^util-linux-uuid@2.37.4%gcc@9.5.0 arch=linux-rhel6-ivybridge
    ^yaksa@0.2%gcc@9.5.0~cuda~rocm arch=linux-rhel6-ivybridge
        ^python@3.9.13%gcc@9.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix+tkinter~ucs4+uuid+zlib patches=0d98e93,f2fd060 arch=linux-rhel6-ivybridge
            ^tcl@8.6.12%gcc@9.5.0 arch=linux-rhel6-ivybridge
            ^tk@8.6.11%gcc@9.5.0+xft+xss arch=linux-rhel6-ivybridge
                ^libx11@1.7.0%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^inputproto@2.3.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^kbproto@1.0.7%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^libxcb@1.14%gcc@9.5.0 arch=linux-rhel6-ivybridge
                        ^libpthread-stubs@0.4%gcc@9.5.0 arch=linux-rhel6-ivybridge
                        ^libxau@1.0.8%gcc@9.5.0 arch=linux-rhel6-ivybridge
                            ^xproto@7.0.31%gcc@9.5.0 arch=linux-rhel6-ivybridge
                        ^libxdmcp@1.1.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
                        ^xcb-proto@1.14.1%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^xextproto@7.3.0%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^xtrans@1.3.5%gcc@9.5.0 arch=linux-rhel6-ivybridge
                ^libxft@2.3.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^fontconfig@2.13.1%gcc@9.5.0 arch=linux-rhel6-ivybridge
                        ^font-util@1.3.2%gcc@9.5.0 fonts=encodings,font-adobe-100dpi,font-adobe-75dpi,font-adobe-utopia-100dpi,font-adobe-utopia-75dpi,font-adobe-utopia-type1,font-alias,font-arabic-misc,font-bh-100dpi,font-bh-75dpi,font-bh-lucidatypewriter-100dpi,font-bh-lucidatypewriter-75dpi,font-bh-type1,font-bitstream-100dpi,font-bitstream-75dpi,font-bitstream-speedo,font-bitstream-type1,font-cronyx-cyrillic,font-cursor-misc,font-daewoo-misc,font-dec-misc,font-ibm-type1,font-isas-misc,font-jis-misc,font-micro-misc,font-misc-cyrillic,font-misc-ethiopic,font-misc-meltho,font-misc-misc,font-mutt-misc,font-schumacher-misc,font-screen-cyrillic,font-sun-misc,font-winitzki-cyrillic,font-xfree86-type1 arch=linux-rhel6-ivybridge
                            ^bdftopcf@1.0.5%gcc@9.5.0 arch=linux-rhel6-ivybridge
                                ^fontsproto@2.1.3%gcc@9.5.0 arch=linux-rhel6-ivybridge
                                ^libxfont@1.5.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
                                    ^freetype@2.11.1%gcc@9.5.0 arch=linux-rhel6-ivybridge
                                        ^libpng@1.6.37%gcc@9.5.0 arch=linux-rhel6-ivybridge
                                    ^libfontenc@1.1.3%gcc@9.5.0 arch=linux-rhel6-ivybridge
                            ^mkfontdir@1.0.7%gcc@9.5.0 arch=linux-rhel6-ivybridge
                                ^mkfontscale@1.1.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
                        ^gperf@3.1%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^libxrender@0.9.10%gcc@9.5.0 arch=linux-rhel6-ivybridge
                        ^renderproto@0.11.1%gcc@9.5.0 arch=linux-rhel6-ivybridge
                ^libxscrnsaver@1.2.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^libxext@1.3.3%gcc@9.5.0 arch=linux-rhel6-ivybridge
                    ^scrnsaverproto@1.2.2%gcc@9.5.0 arch=linux-rhel6-ivybridge
```

 
@lrtfm thanks for the logs. I think I see the issue. I will test a patch locally and tag if in the PR if it works.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/mpich/package.py b/var/spack/repos/builtin/packages/mpich/package.py
--- a/var/spack/repos/builtin/packages/mpich/package.py
+++ b/var/spack/repos/builtin/packages/mpich/package.py
@@ -280,7 +280,7 @@ def is_disabled(text):
         for exe in exes:
             variants = []
             output = Executable(exe)(output=str, error=str)
-            if re.search(r"--with-hwloc-prefix=embedded", output):
+            if re.search(r"--with-hwloc(-prefix)*=embedded", output):
                 variants.append("~hwloc")
 
             if re.search(r"--with-pm=hydra", output):
@@ -457,9 +457,6 @@ def configure_args(self):
         config_args = [
             "--disable-silent-rules",
             "--enable-shared",
-            "--with-hwloc-prefix={0}".format(
-                spec["hwloc"].prefix if "^hwloc" in spec else "embedded"
-            ),
             "--with-pm={0}".format("hydra" if "+hydra" in spec else "no"),
             "--{0}-romio".format("enable" if "+romio" in spec else "disable"),
             "--{0}-ibverbs".format("with" if "+verbs" in spec else "without"),
@@ -467,6 +464,18 @@ def configure_args(self):
             "--with-yaksa={0}".format(spec["yaksa"].prefix if "^yaksa" in spec else "embedded"),
         ]
 
+        # hwloc configure option changed in 4.0
+        if spec.satisfies("@4.0:"):
+            config_args.append(
+                "--with-hwloc={0}".format(spec["hwloc"].prefix if "^hwloc" in spec else "embedded")
+            )
+        else:
+            config_args.append(
+                "--with-hwloc-prefix={0}".format(
+                    spec["hwloc"].prefix if "^hwloc" in spec else "embedded"
+                )
+            )
+
         if "~fortran" in spec:
             config_args.append("--disable-fortran")
 


Test patch: