Repository: spack/spack

Issue:
/usr/local should not be at beginning of PKG_CONFIG_PATH
I have problems building `libxcb` on Cori. The problem is that this package finds a version of `xcb-proto` that is too old, although Spack just installed a newer of this package.

I tracked down the problem to this setting of `PKG_CONFIG_PATH` while Spack is running:

```
$ spack env libxcb | grep PKG_CONFIG_PATH
PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/lib64/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/xcb-proto-1.12-gvfp74jkzfuerqxvohhcybivqaw5ts3v/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/sqlite-3.8.5-esif7eg3rfoneqll2snxf4jz55xrdcom/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/zlib-1.2.11-b7dx6xah5twbll4f2vtbwwf2husbpaxl/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/openssl-1.0.2k-6t2y3u5htsrkkffpbanzmrwftn4dhtoy/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/ncurses-6.0-ndm4sflka535lhge3yhpxstbf4jspfsx/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/python-2.7.13-5tssjghckgwlldxlxlnbxhnintt2mkq2/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/libxdmcp-1.1.2-hyxfmfczedliy5rs6mgr3qoxcjaoc52h/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/xproto-7.0.29-rw2zmcnlbc6qqcj2fw7dxvr5a3yirxr3/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/util-macros-1.19.1-avbfnl2hx7fsimm64emh7kmnyx4c5pkj/share/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/libxau-1.0.8-l42xxi5panwmko6rgvarhr7k5o6r4swu/lib/pkgconfig:/global/project/projectdirs/m152/schnette/cori/src/spack/opt/spack/cray-CNL-haswell/gcc-6.3.0-spack/libpthread-stubs-0.3-45cokmhjghejkvvhi4fzyvgb6ddv4mo6/lib/pkgconfig:/opt/cray/rca/2.1.6_g2c60fbf-2.265/lib64/pkgconfig:/opt/cray/alps/6.3.4-2.21/lib64/pkgconfig:/opt/cray/xpmem/2.1.1_gf9c9084-2.38/lib64/pkgconfig:/opt/cray/gni-headers/5.0.11-2.2/lib64/pkgconfig:/opt/cray/dmapp/7.1.1-39.37/lib64/pkgconfig:/opt/cray/pe/pmi/5.0.10-1.0000.11050.0.0.ari/lib64/pkgconfig:/opt/cray/ugni/6.0.15-2.2/lib64/pkgconfig:/opt/cray/udreg/2.3.2-7.54/lib64/pkgconfig:/usr/common/software/darshan/3.0.1.1/lib/pkgconfig:/opt/cray/pe/craype/2.5.7/pkg-config:/opt/cray/pe/iobuf/2.0.8/lib/pkgconfig:/opt/cray/pe/atp/2.1.0/lib/pkgconfig:/opt/cray/pe/atp/2.0.3/lib/pkgconfig
```

Note the two system directories `/usr/local/lib64/pkgconfig:/usr/lib64/pkgconfig` in the beginning. They should not be there; if at all, they need to be listed in the end (see discussions about `LD_LIBRARY_PATH` etc.)


Discussion:
This also occurs on Blue Waters. Note that both Blue Waters and Cori are Cray system. I don't see the problem elsewhere, but that might be coincidence.
Or it might not be a coincidence:
https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/pkg-config/package.py#L46

@mamelara added this in #3378 to address #3167.
My comment on #3672 might be relevant here.

I agree, hardcoding `/usr/lib` etc. anywhere other than the END of the environment in a Spack context is wrong.  Any ideas on how to fix this?

> ... anywhere other than the END of the environment ... Any ideas on how to fix this?

`s/prepend_path/append_path/` in the `package.py`. If @mamelara agrees I can put out a PR.
Yes... thinking about it a few minutes, I think that's a good start.  To finish the job, we would need:

1. `AutotoolsPackage` should PREPEND `spec.prefix/lib/pkgconfig` when setting up build and module environments.

2. Since `pkg-config` is Autotools-based, it needs to override (1) with its own APPENDING logic discussed above.

I *think* this will do what we want, and I recommend both get submitted in the same PR.  But there's also likely something I haven't thought of here...

As per discussion on #3672, it looks like (1) above already happens.  We just need to figure out how to:

1. DISABLE it for `PkgConfig`
2. Make sure messing with `PKG_CONFIG_PATH` happens for build environments as well as modules.

@adamjstewart yes, should probably change to append


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/pkg-config/package.py b/var/spack/repos/builtin/packages/pkg-config/package.py
--- a/var/spack/repos/builtin/packages/pkg-config/package.py
+++ b/var/spack/repos/builtin/packages/pkg-config/package.py
@@ -49,8 +49,8 @@ def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
         """spack built pkg-config on cray's requires adding /usr/local/
         and /usr/lib64/  to PKG_CONFIG_PATH in order to access cray '.pc'
         files."""
-        spack_env.prepend_path('PKG_CONFIG_PATH', '/usr/lib64/pkgconfig')
-        spack_env.prepend_path('PKG_CONFIG_PATH', '/usr/local/lib64/pkgconfig')
+        spack_env.append_path('PKG_CONFIG_PATH', '/usr/lib64/pkgconfig')
+        spack_env.append_path('PKG_CONFIG_PATH', '/usr/local/lib64/pkgconfig')
 
     def configure_args(self):
         config_args = ['--enable-shared']


Test patch: