Repository: spack/spack

Issue:
Installation issue: gperftools
### Steps to reproduce the issue

```console
spack install gperftools
```

### Information on your system

* **Spack:** 0.16.2-3914-fbed679dd0
* **Python:** 3.9.6
* **Platform:** linux-ubuntu20.04-skylake
* **Concretizer:** original


### Additional information

Basically, this hits https://github.com/gperftools/gperftools/issues/424 because gperftools isn't configured to link with libunwind. Forcing a LDFLAGS="-L/path/to/libunwind -lunwind' bypasses the issue but it's an ugly solution. gperftools has cmake build now and we should switch the package to cmake build
@albestro  @eschnett  @msimberg  @teonnik

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
I can try to have a go at this. It looks like CMake support was added in 2.8.1 though, and I don't know if it's possible to support autotools for the older versions while using CMake for the newer versions? I'd be ok with dropping the older versions but I don't know if someone else needs them to stay around? I take it things build fine with CMake but not autotools, right?

Adding the needed LDFLAGS is not terrible either given that there's one and only one dependency.
There's currently a discussion in the Spack community about how to change build systems, i.e. how to support multiple build systems for packages. I don't know whether there's a good answer yet. In the mean time, I'd just limp forward with autotools, and setting the additional flag.
@Char-Aznable just to understand the problem, are you building an older version of `gperftools`? I just tried `spack install gperftools` on spack commit `edb1d75b1b` which installs `gperftools` 2.9.1 and `libunwind` 1.5.0. I'm on Ubuntu 21.04 and it worked.

The issue you've linked was closed as not a bug, so it's not clear to me in what situations `gperftools` isn't being linked to `libunwind`. I know nothing about `gperftools`' build system otherwise.
@msimberg I was also trying to build 2.9.1. To my understanding, libunwind doesn't have dependent environment set up in package.py so gperftools shouldn't be able to pick up the LDFLAGS from spack's libunwind, should it? I examined my build log and certainly didn't see the desired LDFLAGS. `spack` sometimes pick up dependent library from other directories (see https://github.com/spack/spack/issues/18802). Which libunwind was your build of gperftools linked to?

The issue I linked was probably a bug but the OP probably hacked the autotool config to work around it. 
Notice that you don't see the flags the compiler wrapper injects in the build logs:

```
$ spack build-env gperftools -- env SPACK_TEST_COMMAND=dump-args gcc foo.c | grep unwind
-I/home/user/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.3.0/libunwind-1.5.0-gu3uqvitxk2jr2l34hqboh7o7eye6d63/include
-L/home/user/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.3.0/libunwind-1.5.0-gu3uqvitxk2jr2l34hqboh7o7eye6d63/lib
-Wl,-rpath,/home/user/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.3.0/libunwind-1.5.0-gu3uqvitxk2jr2l34hqboh7o7eye6d63/lib
```

from my build logs `checking for backtrace in -lunwind... yes`, so it works with just `-lunwind`, provided everything goes through the compiler wrapper.

@Char-Aznable just FYI, I'm not doing any work in this direction at the moment. I'd like to understand why things are failing in your case first.
I still see the same build error on a new machine with a fresh spack setup. The `build-env` command @haampie showed above gave:

```
-L/home/aznb/spack/opt/spack/linux-ubuntu22.04-x86_64_v4/clang-homebrew/libunwind-1.6.2-r6fdxqrwhcd4ozgvbb3sdlzottbyvapj/lib
/home/aznb/spack/opt/spack/linux-ubuntu22.04-x86_64_v4/clang-homebrew/libunwind-1.6.2-r6fdxqrwhcd4ozgvbb3sdlzottbyvapj/lib
```
on my system, which lacks the necessary `-lunwind` flag. I am using clang from homebrew as the compiler instead of gcc
@Char-Aznable I'd be happy to add a `-lunwind` flag if that turns out to be required.

Just for comparison, could you check what your install logs say about `unwind`? This is my verbose install log (on nixos, which may make things a bit different... but still): https://gist.github.com/msimberg/1f2c9d40d04b276cb9a2c7a9b117516a. Note that for me `-lunwind` does explicitly end up in the linker command lines (without any patches, of course).
@msimberg I do see `-lunwind` here and there in the log but not at the place right where it needs to be, exactly the reason why we should move to a more maintainable build system. See the attached log here:
[spack-build-out.txt.zip](https://github.com/spack/spack/files/8953002/spack-build-out.txt.zip)

Manually adding `-lunwind` would work around this issue but it would be difficult to keep up with upstream changes. Anyway, thanks for looking at this. 
@Char-Aznable yeah, definitely. I just realized that this now exists in the documentation: https://spack.readthedocs.io/en/latest/build_systems/multiplepackage.html#autotools-cmake. I don't know if this is the multi-build system support that @eschnett was referring to but that looks workable. I'll try to get something going before the weekend.
@Char-Aznable could you try out https://github.com/spack/spack/pull/31231? Note that the 2.10 release notes also mention a CMake/libunwind issue being fixed, so maybe try both 2.10 and at least one of 2.8.1 or 2.9.1 (which are now both also built with CMake). If 2.10 works but the others don't we might want to add a conflict for `+libunwind` and the older versions.
Didn't realize you fixed this years ago @msimberg :) 

#43506 fixes it now.

Resolving patch:
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -68,6 +68,7 @@
 import spack.repo
 import spack.schema.environment
 import spack.spec
+import spack.stage
 import spack.store
 import spack.subprocess_context
 import spack.user_environment
@@ -80,7 +81,7 @@
 from spack.installer import InstallError
 from spack.util.cpus import determine_number_of_jobs
 from spack.util.environment import (
-    SYSTEM_DIRS,
+    SYSTEM_DIR_CASE_ENTRY,
     EnvironmentModifications,
     env_flag,
     filter_system_paths,
@@ -103,9 +104,13 @@
 # Spack's compiler wrappers.
 #
 SPACK_ENV_PATH = "SPACK_ENV_PATH"
+SPACK_MANAGED_DIRS = "SPACK_MANAGED_DIRS"
 SPACK_INCLUDE_DIRS = "SPACK_INCLUDE_DIRS"
 SPACK_LINK_DIRS = "SPACK_LINK_DIRS"
 SPACK_RPATH_DIRS = "SPACK_RPATH_DIRS"
+SPACK_STORE_INCLUDE_DIRS = "SPACK_STORE_INCLUDE_DIRS"
+SPACK_STORE_LINK_DIRS = "SPACK_STORE_LINK_DIRS"
+SPACK_STORE_RPATH_DIRS = "SPACK_STORE_RPATH_DIRS"
 SPACK_RPATH_DEPS = "SPACK_RPATH_DEPS"
 SPACK_LINK_DEPS = "SPACK_LINK_DEPS"
 SPACK_PREFIX = "SPACK_PREFIX"
@@ -418,7 +423,7 @@ def set_compiler_environment_variables(pkg, env):
 
     env.set("SPACK_COMPILER_SPEC", str(spec.compiler))
 
-    env.set("SPACK_SYSTEM_DIRS", ":".join(SYSTEM_DIRS))
+    env.set("SPACK_SYSTEM_DIRS", SYSTEM_DIR_CASE_ENTRY)
 
     compiler.setup_custom_environment(pkg, env)
 
@@ -546,9 +551,23 @@ def update_compiler_args_for_dep(dep):
     include_dirs = list(dedupe(filter_system_paths(include_dirs)))
     rpath_dirs = list(dedupe(filter_system_paths(rpath_dirs)))
 
-    env.set(SPACK_LINK_DIRS, ":".join(link_dirs))
-    env.set(SPACK_INCLUDE_DIRS, ":".join(include_dirs))
-    env.set(SPACK_RPATH_DIRS, ":".join(rpath_dirs))
+    spack_managed_dirs: List[str] = [
+        spack.stage.get_stage_root(),
+        spack.store.STORE.db.root,
+        *(db.root for db in spack.store.STORE.db.upstream_dbs),
+    ]
+
+    env.set(SPACK_MANAGED_DIRS, "|".join(f'"{p}/"*' for p in spack_managed_dirs))
+    is_spack_managed = lambda p: any(p.startswith(store) for store in spack_managed_dirs)
+    link_dirs_spack, link_dirs_system = stable_partition(link_dirs, is_spack_managed)
+    include_dirs_spack, include_dirs_system = stable_partition(include_dirs, is_spack_managed)
+    rpath_dirs_spack, rpath_dirs_system = stable_partition(rpath_dirs, is_spack_managed)
+    env.set(SPACK_LINK_DIRS, ":".join(link_dirs_system))
+    env.set(SPACK_INCLUDE_DIRS, ":".join(include_dirs_system))
+    env.set(SPACK_RPATH_DIRS, ":".join(rpath_dirs_system))
+    env.set(SPACK_STORE_LINK_DIRS, ":".join(link_dirs_spack))
+    env.set(SPACK_STORE_INCLUDE_DIRS, ":".join(include_dirs_spack))
+    env.set(SPACK_STORE_RPATH_DIRS, ":".join(rpath_dirs_spack))
 
 
 def set_package_py_globals(pkg, context: Context = Context.BUILD):
diff --git a/lib/spack/spack/util/environment.py b/lib/spack/spack/util/environment.py
--- a/lib/spack/spack/util/environment.py
+++ b/lib/spack/spack/util/environment.py
@@ -36,6 +36,8 @@
 
 SYSTEM_DIRS = [os.path.join(p, s) for s in SUFFIXES for p in SYSTEM_PATHS] + SYSTEM_PATHS
 
+#: used in the compiler wrapper's `/usr/lib|/usr/lib64|...)` case entry
+SYSTEM_DIR_CASE_ENTRY = "|".join(sorted(f'"{d}{suff}"' for d in SYSTEM_DIRS for suff in ("", "/")))
 
 _SHELL_SET_STRINGS = {
     "sh": "export {0}={1};\n",


Test patch:
diff --git a/lib/spack/spack/test/build_environment.py b/lib/spack/spack/test/build_environment.py
--- a/lib/spack/spack/test/build_environment.py
+++ b/lib/spack/spack/test/build_environment.py
@@ -63,7 +63,8 @@ def build_environment(working_env):
     os.environ["SPACK_LINKER_ARG"] = "-Wl,"
     os.environ["SPACK_DTAGS_TO_ADD"] = "--disable-new-dtags"
     os.environ["SPACK_DTAGS_TO_STRIP"] = "--enable-new-dtags"
-    os.environ["SPACK_SYSTEM_DIRS"] = "/usr/include /usr/lib"
+    os.environ["SPACK_SYSTEM_DIRS"] = "/usr/include|/usr/lib"
+    os.environ["SPACK_MANAGED_DIRS"] = f"{prefix}/opt/spack"
     os.environ["SPACK_TARGET_ARGS"] = ""
 
     if "SPACK_DEPENDENCIES" in os.environ:
diff --git a/lib/spack/spack/test/cc.py b/lib/spack/spack/test/cc.py
--- a/lib/spack/spack/test/cc.py
+++ b/lib/spack/spack/test/cc.py
@@ -15,7 +15,7 @@
 import spack.config
 import spack.spec
 from spack.paths import build_env_path
-from spack.util.environment import SYSTEM_DIRS, set_env
+from spack.util.environment import SYSTEM_DIR_CASE_ENTRY, set_env
 from spack.util.executable import Executable, ProcessError
 
 #
@@ -159,7 +159,8 @@ def wrapper_environment(working_env):
         SPACK_DEBUG_LOG_ID="foo-hashabc",
         SPACK_COMPILER_SPEC="gcc@4.4.7",
         SPACK_SHORT_SPEC="foo@1.2 arch=linux-rhel6-x86_64 /hashabc",
-        SPACK_SYSTEM_DIRS=":".join(SYSTEM_DIRS),
+        SPACK_SYSTEM_DIRS=SYSTEM_DIR_CASE_ENTRY,
+        SPACK_MANAGED_DIRS="/path/to/spack-1/opt/spack/*|/path/to/spack-2/opt/spack/*",
         SPACK_CC_RPATH_ARG="-Wl,-rpath,",
         SPACK_CXX_RPATH_ARG="-Wl,-rpath,",
         SPACK_F77_RPATH_ARG="-Wl,-rpath,",
@@ -907,3 +908,108 @@ def test_linker_strips_loopopt(wrapper_environment, wrapper_flags):
         result = cc(*(test_args + ["-loopopt=0", "-c", "x.c"]), output=str)
         result = result.strip().split("\n")
         assert "-loopopt=0" in result
+
+
+def test_spack_managed_dirs_are_prioritized(wrapper_environment):
+    # We have two different stores with 5 packages divided over them
+    pkg1 = "/path/to/spack-1/opt/spack/linux-ubuntu22.04-zen2/gcc-13.2.0/pkg-1.0-abcdef"
+    pkg2 = "/path/to/spack-1/opt/spack/linux-ubuntu22.04-zen2/gcc-13.2.0/pkg-2.0-abcdef"
+    pkg3 = "/path/to/spack-2/opt/spack/linux-ubuntu22.04-zen2/gcc-13.2.0/pkg-3.0-abcdef"
+    pkg4 = "/path/to/spack-2/opt/spack/linux-ubuntu22.04-zen2/gcc-13.2.0/pkg-4.0-abcdef"
+    pkg5 = "/path/to/spack-2/opt/spack/linux-ubuntu22.04-zen2/gcc-13.2.0/pkg-5.0-abcdef"
+
+    variables = {
+        # cppflags, ldflags from the command line, config or package.py take highest priority
+        "SPACK_CPPFLAGS": f"-I/usr/local/include -I/external-1/include -I{pkg1}/include",
+        "SPACK_LDFLAGS": f"-L/usr/local/lib -L/external-1/lib -L{pkg1}/lib "
+        f"-Wl,-rpath,/usr/local/lib -Wl,-rpath,/external-1/lib -Wl,-rpath,{pkg1}/lib",
+        # automatic -L, -Wl,-rpath, -I flags from dependencies -- on the spack side they are
+        # already partitioned into "spack owned prefixes" and "non-spack owned prefixes"
+        "SPACK_STORE_LINK_DIRS": f"{pkg4}/lib:{pkg5}/lib",
+        "SPACK_STORE_RPATH_DIRS": f"{pkg4}/lib:{pkg5}/lib",
+        "SPACK_STORE_INCLUDE_DIRS": f"{pkg4}/include:{pkg5}/include",
+        "SPACK_LINK_DIRS": "/external-3/lib:/external-4/lib",
+        "SPACK_RPATH_DIRS": "/external-3/lib:/external-4/lib",
+        "SPACK_INCLUDE_DIRS": "/external-3/include:/external-4/include",
+    }
+
+    with set_env(SPACK_TEST_COMMAND="dump-args", **variables):
+        effective_call = (
+            cc(
+                # system paths
+                "-I/usr/include",
+                "-L/usr/lib",
+                "-Wl,-rpath,/usr/lib",
+                # some other externals
+                "-I/external-2/include",
+                "-L/external-2/lib",
+                "-Wl,-rpath,/external-2/lib",
+                # relative paths are considered "spack managed" since they are in the stage dir
+                "-I..",
+                "-L..",
+                "-Wl,-rpath,..",  # pathological but simpler for the test.
+                # spack store paths
+                f"-I{pkg2}/include",
+                f"-I{pkg3}/include",
+                f"-L{pkg2}/lib",
+                f"-L{pkg3}/lib",
+                f"-Wl,-rpath,{pkg2}/lib",
+                f"-Wl,-rpath,{pkg3}/lib",
+                "hello.c",
+                "-o",
+                "hello",
+                output=str,
+            )
+            .strip()
+            .split("\n")
+        )
+
+    dash_I = [flag[2:] for flag in effective_call if flag.startswith("-I")]
+    dash_L = [flag[2:] for flag in effective_call if flag.startswith("-L")]
+    dash_Wl_rpath = [flag[11:] for flag in effective_call if flag.startswith("-Wl,-rpath")]
+
+    assert dash_I == [
+        # spack owned dirs from SPACK_*FLAGS
+        f"{pkg1}/include",
+        # spack owned dirs from command line & automatic flags for deps (in that order)]
+        "..",
+        f"{pkg2}/include",  # from command line
+        f"{pkg3}/include",  # from command line
+        f"{pkg4}/include",  # from SPACK_STORE_INCLUDE_DIRS
+        f"{pkg5}/include",  # from SPACK_STORE_INCLUDE_DIRS
+        # non-system dirs from SPACK_*FLAGS
+        "/external-1/include",
+        # non-system dirs from command line & automatic flags for deps (in that order)
+        "/external-2/include",  # from command line
+        "/external-3/include",  # from SPACK_INCLUDE_DIRS
+        "/external-4/include",  # from SPACK_INCLUDE_DIRS
+        # system dirs from SPACK_*FLAGS
+        "/usr/local/include",
+        # system dirs from command line
+        "/usr/include",
+    ]
+
+    assert (
+        dash_L
+        == dash_Wl_rpath
+        == [
+            # spack owned dirs from SPACK_*FLAGS
+            f"{pkg1}/lib",
+            # spack owned dirs from command line & automatic flags for deps (in that order)
+            "..",
+            f"{pkg2}/lib",  # from command line
+            f"{pkg3}/lib",  # from command line
+            f"{pkg4}/lib",  # from SPACK_STORE_LINK_DIRS
+            f"{pkg5}/lib",  # from SPACK_STORE_LINK_DIRS
+            # non-system dirs from SPACK_*FLAGS
+            "/external-1/lib",
+            # non-system dirs from command line & automatic flags for deps (in that order)
+            "/external-2/lib",  # from command line
+            "/external-3/lib",  # from SPACK_LINK_DIRS
+            "/external-4/lib",  # from SPACK_LINK_DIRS
+            # system dirs from SPACK_*FLAGS
+            "/usr/local/lib",
+            # system dirs from command line
+            "/usr/lib",
+        ]
+    )