Repository: spack/spack

Issue:
External system packages may hide spack compiler wrappers
I followed the advice in spack's Getting Started guide about preferring a vendor-supplied openssl for improved security. This involved creating a packages.yaml file with the following contents:

```
packages:
  openssl:
    paths:
      openssl@1.0.1e: /usr
    buildable: False
```

This seemed to work for many packages, but then I tried to build `qt@4.8.6`. One of the link steps failed with an unresolved symbol from libpng, even though libpng is listed as a dependency by the qt package. I eventually found the problem - `spack env qt@4.8.6 which c++` was resolving to `/usr/bin/c++`. After removing my customised packages.yaml file, spack's `c++` wrapper was found correctly.

What is the best way to force spack to use specified system packages (e.g. openssl) without breaking spack's build environment?


Discussion:
Hmm... I thought we had fixed this. Spack is supposed to add all build dependencies to your `PATH` so that the build process can find them. Of course, this includes external packages that may or may not be in `/usr`. At some point we noticed that this caused problems with linking (things were linking to installations in `/usr` even if they were built by Spack) so we made sure that Spack didn't add things like `/lib` or `/usr/lib64` to the link line. Maybe we forgot to do this for the `PATH` too?
can you please look in `build.env` and copy-paste `PATH` and `SPACK_DEPENDENCIES`? 

@adamjstewart 

> Maybe we forgot to do this for the PATH too?

we did not! We do filter here https://github.com/LLNL/spack/blob/develop/lib/spack/spack/build_environment.py#L272  and here https://github.com/LLNL/spack/blob/develop/lib/spack/spack/build_environment.py#L332 . I have a feeling that the problem is elsewhere...
Hi @davydden, here are the PATH and SPACK_DEPENDENCIES variables from the spack-build.env produced by `spack install qt@4.8.6`, with packages.yaml forcing the use of openssl from `/usr`. The main thing I notice is that `/usr/bin` occurs in PATH before `/data/mwoods/spack2/lib/spack/env`.

`PATH=/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/glib-2.49.7-6mvzrhcdryemn67ahp3ia4k7x5gc4b5m/bin:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libxml2-2.9.4-yk4fnt3lwvjsjkcewj35yvxotu7irlak/bin:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libtiff-4.0.6-4hwzr2lg34urtkfv6jwebo3rkdkbk6xc/bin:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/jpeg-9b-7mdezufg3chiskzd2y66ckxsehqlabwu/bin:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/icu4c-58.2-52crcjkhyy5g3bbxlpc2pt3ygamw43cs/bin:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libpng-1.6.27-4zz3phez54ulf7j3qaxy2qa52nlkmymv/bin:/usr/bin:/data/mwoods/spack2/lib/spack/env:/data/mwoods/spack2/lib/spack/env/case-insensitive:/data/mwoods/spack2/lib/spack/env/gcc:/data/mwoods/spack2/bin:/home/mwoods/opt/netcdf-4.1.1/bin:/home/mwoods/opt/udunits-2.1.10/bin:/home/mwoods/opt/udunits-1.11.7/bin:/home/mwoods/bin:/home/mwoods/opt/bin:/home/mwoods/opt/ssvnc/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin`

`SPACK_DEPENDENCIES=/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/glib-2.49.7-6mvzrhcdryemn67ahp3ia4k7x5gc4b5m:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/autoconf-2.69-wce2lz6nmutlrgyx67ofm54r7p7t4cy4:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/m4-1.4.18-uxuflkbdlyu53rs2usnvlh36ktyz2t7f:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libsigsegv-2.11-hp5fmaqw3e6e6y237kdmavr5gzb4ub43:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/automake-1.15-f4w47vogfk4nn6d7ukoksbovcrxyvuic:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/gettext-0.19.8.1-jvn64ebtlbievt32fa43mjdrof5ej6c4:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/bzip2-1.0.6-kleqq4isxniiwo2s7ctgb3gcjbls66kp:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libxml2-2.9.4-yk4fnt3lwvjsjkcewj35yvxotu7irlak:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/pkg-config-0.29.2-gvreigcsvldfiweh5qe37uu63mgevo7n:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/xz-5.2.3-p5kdjwwvkrbu6ddr4xx7nzfnoymg7iyt:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/zlib-1.2.11-p354t3uhiqf7btobzka3lphuh2x6ic5w:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/ncurses-6.0-tfbllisfhb2cauoaxevjk6qqzx6lkdqz:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/tar-1.29-ipcximnmozbcjvlwguztrryv4xzbt4qy:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libffi-3.2.1-5tualc4wpkzavyta546sttkoxjitmspn:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libtool-2.4.6-v7uc7asl44a5kacx2yr5lr2klgym4ddf:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/pcre-8.40-e6mwkklabmfe7yuifljfloipdpp3hxes:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/icu4c-58.2-52crcjkhyy5g3bbxlpc2pt3ygamw43cs:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/jpeg-9b-7mdezufg3chiskzd2y66ckxsehqlabwu:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libmng-2.0.2-2ltbe47nhl7ur42mphhlmlommkur6f25:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/lcms-2.6-gxcu5a2ecfh4m6iwej4oncrwrslro6gt:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libtiff-4.0.6-4hwzr2lg34urtkfv6jwebo3rkdkbk6xc:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libpng-1.6.27-4zz3phez54ulf7j3qaxy2qa52nlkmymv:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libxcb-1.12-5igo3baejurwam4trqwclttc36fdcmlf:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libpthread-stubs-0.3-6n2oinitdtkpuoxlgxgnf6nlq2t5umyb:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libxau-1.0.8-lvcfo5ahhj4cnukank3xtxg76jailgpo:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/util-macros-1.19.1-f35wwh47zyshnbap35icfbfwipwkjeru:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/xproto-7.0.29-mqpamn3rlhr7e4rwcslse2kcdwwczrui:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/libxdmcp-1.1.2-ylbihrhciylhzwefgncr2jwlln2kfnpu:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/python-2.7.13-5mif4q7uecrq3zeqiw4uzs2vdo533htc:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/readline-7.0-glm7nakfazghmuid2gb3gwsg3x4lb4lb:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/sqlite-3.8.5-wew6vi2uxfonfiqawra5pxn5t2iomus2:/apps/spack/opt/linux-centos6-x86_64/gcc-4.4.7/xcb-proto-1.12-gsjaibejapm7t7fh47m3mjc5otytfs2g`
>  /usr/bin occurs in PATH before /data/mwoods/spack2/lib/spack/env

@tgamblin @alalazo @adamjstewart do you recall where compiler wrappers are appended to PATH? From the post above it looks like this is done **after** `def set_build_environment_variables(pkg, env, dirty=False):` (where we filter out system paths to external packages) **by appending** instead of prepending.

I'm running into this problem now too. A lot of packages don't build with the old `ld` installed on CentOS 6 so I have a newer Spack installed version in my `PATH`. I also have a few external packages like `/usr/bin/m4` in `packages.yaml`. Spack is prepending `/usr` to my `PATH` and the build is picking up `/usr/bin/ld` instead. I'll try to track this down.
I think I found the problem.

As @davydden pointed out, [`set_build_environment_variables`](https://github.com/LLNL/spack/blob/develop/lib/spack/spack/build_environment.py#L238) is where Spack sets `PATH`. On [line 266](https://github.com/LLNL/spack/blob/develop/lib/spack/spack/build_environment.py#L266), we prepend Spack's compiler wrappers to the `PATH`. Then, on [line 344](https://github.com/LLNL/spack/blob/develop/lib/spack/spack/build_environment.py#L334), we prepend dependencies to the `PATH`.

As @davydden mentioned, immediately before both lines, we are checking for system paths `['/bin', '/usr/bin', '/usr/local/bin']`. But here's the catch. On [line 263](https://github.com/LLNL/spack/blob/develop/lib/spack/spack/build_environment.py#L263), we call [`filter_system_paths`](https://github.com/LLNL/spack/blob/develop/lib/spack/spack/util/environment.py#L34,L35), but on [line 332](https://github.com/LLNL/spack/blob/develop/lib/spack/spack/build_environment.py#L332) we instead call [`filter_system_bin_paths`](https://github.com/LLNL/spack/blob/develop/lib/spack/spack/util/environment.py#L38,L44), which has a completely different specification. The former removes system paths from the list while the latter puts system paths _after_ non-system paths. In my case, I was trying to install `gmp` which only had a single dependency (`m4`). Since I added `m4` as an external package, `/usr/bin` was at the beginning of my `PATH`.

I see a couple potential solutions to this problem. First of all, I see no reason to add dependencies to the `PATH` if they reside in system directories. These values should already be in the `PATH`. Of course, someone could theoretically have multiple copies of a dependency in their `PATH` and want to build with the system one, not the first one. But I don't see any other way to prevent the aforementioned problem. The problem with system paths is that they usually contain hundreds of installations in the same directory, so there's no easy way to control which ones get used and which don't.

Second, I think we should always ensure that Spack's compiler wrappers are at the front of the `PATH`, even before dependencies. I don't see why we would do it any other way.
> First of all, I see no reason to add dependencies to the PATH if they reside in system directories.

I actually don't remember why we decided to do in first place. But I agree that we may as well filter out system paths completely. @tgamblin @alalazo do you agree?

> Second, I think we should always ensure that Spack's compiler wrappers are at the front of the PATH, even before dependencies. I don't see why we would do it any other way.

fully agree on this account.
It looks like this was all done in #2083. @tgamblin had some arguments for why system paths should be in `PATH`.
indeed, here is the link to @tgamblin 's post https://github.com/LLNL/spack/pull/2083#discussion_r84586534
i would go for the second solution, and keep the discussion around the first solution for another day.
The second solution will solve @mjwoods' problem, but not mine. Go big or go home!
oh, i see. Then I am fine with applying both, and add a comment why we really need this (explain your problem) + regression test.
~~We also don't seem to be filtering system paths from `get_rpath_deps`...~~

Nvm, yes we are.

Resolving patch:
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -236,50 +236,47 @@ def set_compiler_environment_variables(pkg, env):
 
 
 def set_build_environment_variables(pkg, env, dirty=False):
-    """
-    This ensures a clean install environment when we build packages.
+    """Ensure a clean install environment when we build packages.
 
-    Arguments:
-    dirty -- skip unsetting the user's environment settings.
-    """
-    # Add spack build environment path with compiler wrappers first in
-    # the path. We add both spack.env_path, which includes default
-    # wrappers (cc, c++, f77, f90), AND a subdirectory containing
-    # compiler-specific symlinks.  The latter ensures that builds that
-    # are sensitive to the *name* of the compiler see the right name
-    # when we're building with the wrappers.
-    #
-    # Conflicts on case-insensitive systems (like "CC" and "cc") are
-    # handled by putting one in the <build_env_path>/case-insensitive
-    # directory.  Add that to the path too.
-    env_paths = []
-    compiler_specific = join_path(spack.build_env_path, pkg.compiler.name)
-    for item in [spack.build_env_path, compiler_specific]:
-        env_paths.append(item)
-        ci = join_path(item, 'case-insensitive')
-        if os.path.isdir(ci):
-            env_paths.append(ci)
-
-    env_paths = filter_system_paths(env_paths)
+    This involves unsetting pesky environment variables that may
+    affect the build. It also involves setting environment variables
+    used by Spack's compiler wrappers.
 
-    for item in reversed(env_paths):
-        env.prepend_path('PATH', item)
-    env.set_path(SPACK_ENV_PATH, env_paths)
+    Args:
+        pkg: The package we are building
+        env: The build environment
+        dirty (bool): Skip unsetting the user's environment settings
+    """
+    # Gather information about various types of dependencies
+    build_deps      = pkg.spec.traverse(root=False, deptype=('build'))
+    link_deps       = pkg.spec.traverse(root=False, deptype=('link'))
+    build_link_deps = pkg.spec.traverse(root=False, deptype=('build', 'link'))
+    rpath_deps      = get_rpath_deps(pkg)
+
+    build_prefixes      = [dep.prefix for dep in build_deps]
+    link_prefixes       = [dep.prefix for dep in link_deps]
+    build_link_prefixes = [dep.prefix for dep in build_link_deps]
+    rpath_prefixes      = [dep.prefix for dep in rpath_deps]
+
+    # Filter out system paths: ['/', '/usr', '/usr/local']
+    # These paths can be introduced into the build when an external package
+    # is added as a dependency. The problem with these paths is that they often
+    # contain hundreds of other packages installed in the same directory.
+    # If these paths come first, they can overshadow Spack installations.
+    build_prefixes      = filter_system_paths(build_prefixes)
+    link_prefixes       = filter_system_paths(link_prefixes)
+    build_link_prefixes = filter_system_paths(build_link_prefixes)
+    rpath_prefixes      = filter_system_paths(rpath_prefixes)
 
     # Prefixes of all of the package's dependencies go in SPACK_DEPENDENCIES
-    dep_prefixes = [d.prefix for d in
-                    pkg.spec.traverse(root=False, deptype=('build', 'link'))]
-    dep_prefixes = filter_system_paths(dep_prefixes)
-    env.set_path(SPACK_DEPENDENCIES, dep_prefixes)
+    env.set_path(SPACK_DEPENDENCIES, build_link_prefixes)
 
     # These variables control compiler wrapper behavior
-    env.set_path(SPACK_RPATH_DEPS, filter_system_paths([
-        d.prefix for d in get_rpath_deps(pkg)]))
-    env.set_path(SPACK_LINK_DEPS, filter_system_paths([
-        d.prefix for d in pkg.spec.traverse(root=False, deptype=('link'))]))
+    env.set_path(SPACK_RPATH_DEPS, rpath_prefixes)
+    env.set_path(SPACK_LINK_DEPS, link_prefixes)
 
     # Add dependencies to CMAKE_PREFIX_PATH
-    env.set_path('CMAKE_PREFIX_PATH', dep_prefixes)
+    env.set_path('CMAKE_PREFIX_PATH', build_link_prefixes)
 
     # Install prefix
     env.set(SPACK_PREFIX, pkg.prefix)
@@ -326,12 +323,33 @@ def set_build_environment_variables(pkg, env, dirty=False):
         env.set('SPACK_COMPILER_EXTRA_RPATHS', extra_rpaths)
 
     # Add bin directories from dependencies to the PATH for the build.
-    bin_dirs = reversed(
-        [d.prefix.bin for d in pkg.spec.dependencies(deptype='build')
-         if os.path.isdir(d.prefix.bin)])
-    bin_dirs = filter_system_bin_paths(bin_dirs)
-    for item in bin_dirs:
+    for prefix in build_prefixes:
+        for dirname in ['bin', 'bin64']:
+            bin_dir = os.path.join(prefix, dirname)
+            if os.path.isdir(bin_dir):
+                env.prepend_path('PATH', bin_dir)
+
+    # Add spack build environment path with compiler wrappers first in
+    # the path. We add both spack.env_path, which includes default
+    # wrappers (cc, c++, f77, f90), AND a subdirectory containing
+    # compiler-specific symlinks.  The latter ensures that builds that
+    # are sensitive to the *name* of the compiler see the right name
+    # when we're building with the wrappers.
+    #
+    # Conflicts on case-insensitive systems (like "CC" and "cc") are
+    # handled by putting one in the <build_env_path>/case-insensitive
+    # directory.  Add that to the path too.
+    env_paths = []
+    compiler_specific = join_path(spack.build_env_path, pkg.compiler.name)
+    for item in [spack.build_env_path, compiler_specific]:
+        env_paths.append(item)
+        ci = join_path(item, 'case-insensitive')
+        if os.path.isdir(ci):
+            env_paths.append(ci)
+
+    for item in reversed(env_paths):
         env.prepend_path('PATH', item)
+    env.set_path(SPACK_ENV_PATH, env_paths)
 
     # Working directory for the spack command itself, for debug logs.
     if spack.debug:
@@ -340,9 +358,9 @@ def set_build_environment_variables(pkg, env, dirty=False):
     env.set(SPACK_DEBUG_LOG_DIR, spack.spack_working_dir)
 
     # Add any pkgconfig directories to PKG_CONFIG_PATH
-    for pre in dep_prefixes:
+    for prefix in build_link_prefixes:
         for directory in ('lib', 'lib64', 'share'):
-            pcdir = join_path(pre, directory, 'pkgconfig')
+            pcdir = join_path(prefix, directory, 'pkgconfig')
             if os.path.isdir(pcdir):
                 env.prepend_path('PKG_CONFIG_PATH', pcdir)
 
diff --git a/lib/spack/spack/util/environment.py b/lib/spack/spack/util/environment.py
--- a/lib/spack/spack/util/environment.py
+++ b/lib/spack/spack/util/environment.py
@@ -25,23 +25,13 @@
 import os
 
 system_paths = ['/', '/usr', '/usr/local']
-suffixes = ['lib', 'lib64', 'include']
+suffixes = ['bin', 'bin64', 'include', 'lib', 'lib64']
 system_dirs = [os.path.join(p, s) for s in suffixes for p in system_paths] + \
     system_paths
-system_bins = [os.path.join(p, 'bin') for p in system_paths]
 
 
 def filter_system_paths(paths):
-    return [p for p in paths if p not in system_dirs]
-
-
-def filter_system_bin_paths(paths):
-    # Turn the iterable into a list. Assume it's a list from here on.
-    _paths = list(paths)
-    bins = [p for p in _paths if p in system_bins]
-    nobins = [p for p in _paths if p not in system_bins]
-    # put bins infront as PATH is set by: prepend_path('PATH', item)
-    return bins + nobins
+    return [p for p in paths if os.path.normpath(p) not in system_dirs]
 
 
 def get_path(name):


Test patch:
diff --git a/lib/spack/spack/test/environment.py b/lib/spack/spack/test/environment.py
--- a/lib/spack/spack/test/environment.py
+++ b/lib/spack/spack/test/environment.py
@@ -29,7 +29,7 @@
 from spack.environment import EnvironmentModifications
 from spack.environment import RemovePath, PrependPath, AppendPath
 from spack.environment import SetEnv, UnsetEnv
-from spack.util.environment import filter_system_paths, filter_system_bin_paths
+from spack.util.environment import filter_system_paths
 
 
 @pytest.fixture()
@@ -64,25 +64,18 @@ def miscellaneous_paths():
         '/usr/local/lib64',
         '/usr/local/opt/some-package/lib',
         '/usr/opt/lib',
+        '/usr/local/../bin',
         '/lib',
         '/',
         '/usr',
+        '/usr/',
+        '/usr/bin',
+        '/bin64',
         '/lib64',
         '/include',
+        '/include/',
         '/opt/some-package/include',
-    ]
-
-
-@pytest.fixture
-def bin_paths():
-    """Returns a list of bin paths, including system ones."""
-    return [
-        '/usr/local/Cellar/gcc/5.3.0/bin',
-        '/usr/local/bin',
-        '/usr/local/opt/some-package/bin',
-        '/usr/opt/bin',
-        '/bin',
-        '/opt/some-package/bin',
+        '/opt/some-package/local/..',
     ]
 
 
@@ -137,21 +130,8 @@ def test_filter_system_paths(miscellaneous_paths):
         '/usr/local/Cellar/gcc/5.3.0/lib',
         '/usr/local/opt/some-package/lib',
         '/usr/opt/lib',
-        '/opt/some-package/include'
-    ]
-    assert filtered == expected
-
-
-def test_filter_system_bin_paths(bin_paths):
-    """Tests that the filtering of system bin paths works as expected."""
-    filtered = filter_system_bin_paths(bin_paths)
-    expected = [
-        '/usr/local/bin',
-        '/bin',
-        '/usr/local/Cellar/gcc/5.3.0/bin',
-        '/usr/local/opt/some-package/bin',
-        '/usr/opt/bin',
-        '/opt/some-package/bin'
+        '/opt/some-package/include',
+        '/opt/some-package/local/..',
     ]
     assert filtered == expected