Repository: spack/spack

Issue:
darwin: cut DYLD_LIBRARY_PATH from default modules
This commit removes the `DYLD_LIBRARY_PATH` variable from the default
`modules.yaml` for darwin. The rationale behind deleting this
environment variable is that paths in this environment variable take
precedence over the default locations of libraries (usually the
install path of the library), which can lead to linking errors in some
circumstances. For example, executables intended to link with Apple's
system BLAS and LAPACK will instead link to a spack-installed
implementation (e.g., OpenBLAS), causing runtime errors.

These errors are resolved by instead relying on paths set in
`DYLD_FALLBACK_LIBRARY_PATH`, which is lower in precedence than default
locations of libraries.
Spack breaks some software when we are inside an environment
### Steps to reproduce the issue

- To reproduce this issue you need to be on macOS.
- Then after this, you need to create an environment, with the intention of replacing Homebrew/Anaconda, [reference](https://spack.readthedocs.io/en/latest/workflows.html)
- After that active any env, lets say for example, my env is named Neo, and here is the configuration:

```
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: [bash, coreutils]
  view: /opt/neo
  config: {}
  mirrors: {}
  modules:
    enable: []
  packages: {}
  repos: []
  upstreams: {}
  definitions: []
  concretization: together
```

- Now you need to run software, that you had installed in your Mac/Darwin machine, and it isn't installed by Spack, for example here I use `code` command line tool from visual studio code and otool, and we get an error message:

```dyld: Symbol not found: _iconv
  Referenced from: /usr/lib/libarchive.2.dylib
  Expected in: /opt/neo/lib/libiconv.2.dylib
 in /usr/lib/libarchive.2.dylib
/usr/local/bin/code: line 10: 26217 Abort trap: 6           ELECTRON_RUN_AS_NODE=1 "$ELECTRON" "$CLI" "$@"
```

- As we see here is using /opt/neo to search for libiconv, the same happens, in for example otool.
```spack ❯ otool -L code                                                                                                                        develop
/opt/local/libexec/llvm-9.0/bin/llvm-objdump: dyld: lazy symbol binding failed: Symbol not found: _setupterm
  Referenced from: /opt/local/libexec/llvm-9.0/bin/../lib/libLLVM.dylib
  Expected in: /opt/neo/lib/libncurses.6.dylib

dyld: Symbol not found: _setupterm
  Referenced from: /opt/local/libexec/llvm-9.0/bin/../lib/libLLVM.dylib
  Expected in: /opt/neo/lib/libncurses.6.dylib

Stack dump:
0.	Program arguments: /opt/local/libexec/llvm-9.0/bin/llvm-objdump -macho -dylibs-used -non-verbose code
0  libLLVM.dylib            0x0000000107aace68 llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 40
1  libLLVM.dylib            0x0000000107aad262 SignalHandler(int) + 188
2  libsystem_platform.dylib 0x00007fff708f55fd _sigtramp + 29
3  libsystem_platform.dylib 0x000000011701b590 _sigtramp + 18446603343308677040
4  libsystem_platform.dylib 0x000000011701a54f _sigtramp + 18446603343308672879
5  libsystem_platform.dylib 0x0000000117017bd2 _sigtramp + 18446603343308662258
6  libsystem_platform.dylib 0x0000000116f9fb01 _sigtramp + 18446603343308170529
7  libdyld.dylib            0x00007fff706fb692 dyld_stub_binder + 282
8  libLLVM.dylib            0x000000010a61b6e0 _dyld_private + 0
9  libLLVM.dylib            0x0000000107a8dd20 llvm::WithColor::WithColor(llvm::raw_ostream&, llvm::HighlightColor, bool) + 54
```

However, there is a workaround for this issue:
- First of all I dumped my environment variables using the `env` command, and this is the part that was particularly suspicious:

```
LIBRARY_PATH=/opt/neo/lib64:/opt/neo/lib:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib64:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib
CPATH=/opt/neo/include:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/include
PKG_CONFIG_PATH=/opt/neo/lib64/pkgconfig:/opt/neo/lib/pkgconfig:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib64/pkgconfig:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib/pkgconfig
DYLD_LIBRARY_PATH=/opt/neo/lib64:/opt/neo/lib:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib64:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib
LD_LIBRARY_PATH=/opt/neo/lib64:/opt/neo/lib:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib64:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib
ACLOCAL_PATH=/opt/neo/share/aclocal:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/share/aclocal
MANPATH=/opt/neo/share/man:/opt/neo/man:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/share/man:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/man
SPACK_LD_LIBRARY_PATH=/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib64:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib
SPACK_DYLD_LIBRARY_PATH=/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib64:/Users/dsmc/spack/var/spack/environments/neo/.spack-env/view/lib
SPACK_ENV=/Users/dsmc/spack/var/spack/environments/neo
_=/usr/bin/env
```

-So I had an idea, I know that on Darwin the environment variable `DYLD_LIBRARY_PATH` tells Darwin where to search for libraries, so what I did, I unset this environment variable with the following command:

`unset DYLD_LIBRARY_PATH`

-This made everything work as expect, I think the issue is that this is getting set in some part of the modules, I tried running some Spack installed programs, and they worked fine (Since these programs are linked properly), so unsetting doesn't cause an issue.
### Information on your system

```
ProductName:	Mac OS X
ProductVersion:	10.15.4
BuildVersion:	19E287
```

```
spack ❯ uname -a                                                                                                                             develop
Darwin epicMac 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
```


### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [ x] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
Apologies to @adamjstewart for the change; I encountered a weird LAPACK linking error in code I develop, and traced it down to spack modifications to `DYLD_LIBRARY_PATH`. Using `DYLD_FALLBACK_LIBRARY_PATH` seems to work, and resolves the linking error I encountered.
Hmm, how is this different than the behavior you would see on Linux where we set `LD_LIBRARY_PATH`? I would prefer to see the same behavior on both systems.
I agree, uniform behavior across all platforms is preferable. 

The tl;dr version is: This PR is a fix recommended by [this Stack Overflow answer](https://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s). I'm not sure that achieving Linux-like behavior on macOS is possible with spack. If it is possible, I don't think the obstacle is spack's architecture -- I think the issue is how build systems of various packages build shared libraries, and shared libraries on macOS probably need to use `@rpath/` in load commands instead of full, absolute paths. This sort of behavior might be fixable post-build (see #9065 for an example), but it would be better for the build system to build the libraries using `@rpath/` rather than hacking it using `install_name_tool`.

The longer version -- after multiple revisions -- is:

**Darwin searches for libraries using the full file name**
 
To quote the [`dyld` manpage](https://www.unix.com/man-page/osx/1/dyld/):

>  `DYNAMIC LIBRARY LOADING`
	      Unlike many other operating systems,  Darwin  does  not  locate  dependent  dynamic
	      libraries  via  their  leaf file name.  Instead the full path to each dylib is used
	      (e.g. `/usr/lib/libSystem.B.dylib`).  But there are times when a  full  path  is  not
	      appropriate;  for instance, may want your binaries to be installable in anywhere on
	      the disk.  To support that, there are three `@xxx/` variables that can be used  as	a
	      path  prefix.   At  runtime  `dyld`  substitutes a dynamically generated path for the
	      `@xxx/` prefix.

**Because libraries are located by full paths, the search path order works differently**

The best source I could find on `dyld` path search order precedence is [some Apple developer documentation](https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/DynamicLibraryUsageGuidelines.html#//apple_ref/doc/uid/TP40001928-SW12). To excerpt the relevant part:

> When the library name is a filename (that is, when it doesn’t include directory names), the dynamic loader searches for the library in several locations until it finds it, in the following order:
> 1. `$LD_LIBRARY_PATH`.
> 2. `$DYLD_LIBRARY_PATH`.
> 3. The process' working directory.
> 4. `$DYLD_FALLBACK_LIBRARY_PATH`.
>
> When the library name contains at least one directory name, that is, when the name is a pathname (relative or fully qualified), the dynamic loader searches for the library in the following order:
> 1.  `$DYLD_LIBRARY_PATH` using the filename.
> 2. The given pathname.
> 3. `$DYLD_FALLBACK_LIBRARY_PATH` using the filename.

The important thing to note in the second list is that the searches of `$DYLD_LIBRARY_PATH` and `$DYLD_FALLBACK_LIBRARY_PATH` will strip off the directories in a library name that contains at least one directory, and search for that filename in the paths stored in those environment variables. The developer documentation gives the example (note: formatting altered from source for my convenience):

> For example, say you set the environment variables introduced earlier as shown in the following table.
> * `LD_LIBRARY_PATH` set to `./lib`
> * `DYLD_LIBRARY_PATH` set to `/usr/local/dylibs`
> * `DYLD_FALLBACK_LIBRARY_PATH` set to `/usr/local/lib`
>
> If the app calls `dlopen` with the pathname `/libs/libCelsus.dylib`, the dynamic loader tries to find the library using these pathnames, in order:
> 1. `/usr/local/dylibs/libCelsus.dylib` (`DYLD_LIBRARY_PATH` environment variable)
> 2. `/libs/libCelsus.dylib` (Path as given)
> 3. `/usr/local/lib/libCelsus.dylib` (`DYLD_FALLBACK_LIBRARY_PATH` environment variable)

**Darwin's linker also has to work with Apple frameworks, a different binary format (Mach-O instead of ELF), etc.**

The [`dyld` manpage](https://www.unix.com/man-page/osx/1/dyld/) lists additional macOS-specific wrinkles, like paths for searching for framework libraries (e.g., the vecLib/Accelerate framework containing Apple's vendor BLAS/LAPACK), `LC_PATH`, etc.

**There are some analogies, but these analogues have to be encoded at build-system level**

Aside from some of the obvious parallels in environment variables above (e.g., `DYLD_LIBRARY_PATH` and similar), the macOS equivalent of  [`ld.so`'s  `$ORIGIN` variable](https://linux.die.net/man/8/ld.so) appears to be `@loader_path/`, and the macOS equivalent of `DT_RUNPATH` appears to be `@rpath/`. However, use of `@rpath/` must be enforced by the build system, and many build systems don't do this, or spack overrides their defaults so `@rpath` isn't used. #9065 is a good example of spack clobbering the CMake arguments of googletest so it doesn't use `@rpath`, even though the default configuration on macOS would use `@rpath`.

**How I got here: a BLAS/LAPACK linking bug**

I ran into buggy linking with CMake using its FindBLAS and FindLAPACK modules. Due to changes in the modules from version 2.8.x to 3.12.x, it is no longer possible to specify search directories using CMake variables. They recommend setting `BLA_VENDOR` to specify an implementation, and the only way to add Homebrew-installed or spack-installed OpenBLAS is to modify `DYLD_LIBRARY_PATH` or `DYLD_FALLBACK_LIBRARY_PATH` -- `LD_LIBRARY_PATH` won't work because CMake uses absolute paths that include directories. So I loaded OpenBLAS via lmod, and CMake found it, but then executing a binary with both `DYLD_LIBRARY_PATH` and `DYLD_FALLBACK_LIBRARY_PATH` set to `/usr/local/Cellar/openblas/0.3.3` yielded the runtime error:

```
dyld: Library not loaded: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
Referenced from: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
Reason: Incompatible library version: vecLib requires version 1.0.0 or later, but libopenblas_haswellp-r0.3.3.dylib provides version 0.0.0
```

I ran `otool -L` on every library my binary depended on and did not see vecLib's LAPACK in the output. Setting the `DYLD_PRINT_LIBRARIES` variable confirmed that vecLib's LAPACK was being linked, and unsetting `DYLD_LIBRARY_PATH` resolved the issue. It's also worth noting that Apple recommends only using `DYLD_LIBRARY_PATH` for library testing purposes, not as a production build configuration; [the Stack Overflow answer linked at the top](https://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s) gave me the insight to unset `DYLD_LIBRARY_PATH`.

#9249 should fix the `@rpath` issue for CMake. #9220 is an example of how to fix it in an Autotools-like build system.
Rebased on current `develop`.
@davydden @adamjstewart Is there anything I can do to help get this PR merged?

Here's a close-to-minimal example illustrating the problems I run into. I generated these results using macOS 10.12.6, spack commit `16aa37bdfe76640e5e660b1b9dc1f25c7d9d85a5` (on `develop`), and Homebrew commit `323556c27ff45cc4082119177bd17c0795eabad5` (on `master`), assuming Homebrew is installed with:

* `git`
* `lmod`
* `openblas`
* `python@2`
* `python`
* `gcc`

I also assume that `lmod` is already configured in shell configuration files via something like this `.bashrc` snippet:

```bash
# $HOME/.bashrc
#!/bin/bash
# Explicitly put /usr/local/ directories in higher precedence order
# than the corresponding /usr/ or / (root) directories
export PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/X11/bin"

## Autocompletions
[ -f /usr/local/etc/bash_completion ] && . /usr/local/etc/bash_completion
[ -f /usr/local/opt/lmod/init/bash ] && . /usr/local/opt/lmod/init/bash
[ -f /usr/local/opt/lmod/init/profile ] && . /usr/local/opt/lmod/init/profile

# path to lmod modules
#export MODULEPATH=/usr/local/opt/lmod/lmod/modulefiles

export SPACK_ROOT="$HOME/spack"

if [ -d "$SPACK_ROOT" ]
   then
       export PATH="$SPACK_ROOT:$PATH"
       [ -f  ${SPACK_ROOT}/share/spack/setup-env.sh ] && . ${SPACK_ROOT}/share/spack/setup-env.sh
       module unuse /Users/oxberry1/spack/share/spack/modules/darwin-sierra-x86_64
       module use /Users/oxberry1/spack/share/spack/lmod/darwin-sierra-x86_64/Core
       [[ $PS1 && -f %s/share/bash-completion/bash_completion ]] && \
	   . $(spack location -i bash-completion)/share/bash-completion/bash_completion
fi

export LMOD_COLORIZE=YES
```

I use the following configuration files in `$HOME/.spack`:

`.spack/modules.yaml`:
```yaml
modules:
  enable::
    - lmod
  lmod:
    core_compilers:
      - 'clang@9.0.0-apple'
    hierarchy:
      - mpi
      - lapack
    hash_length: 0
    ^python:
      autoload: 'direct'
    all:
      suffixes:
        '+debug': debug
  prefix_inspections:
    bin:
      - PATH
    man:
      - MANPATH
    share/man:
      - MANPATH
    share/aclocal:
      - ACLOCAL_PATH
    lib:
      - LIBRARY_PATH
      - LD_LIBRARY_PATH
    lib64:
      - LIBRARY_PATH
      - LD_LIBRARY_PATH
    include:
      - CPATH
    lib/pkgconfig:
      - PKG_CONFIG_PATH
    lib64/pkgconfig:
      - PKG_CONFIG_PATH
    '':
      - CMAKE_PREFIX_PATH
```

`.spack/packages.yaml`:
```yaml
packages:
  curl:
    paths:
      curl@7.62.0+darwinssl: /usr/local/Cellar/curl/7.62.0
  git:
    paths:
      git@2.19.1: /usr/local/Cellar/git/2.19.1
  lmod:
    paths:
      lmod@7.8 %clang@9.0.0-apple: /usr/local/Cellar/lmod/7.8
  openblas:
    paths:
      openblas@0.3.3 %clang@9.0.0-apple: /usr/local/Cellar/openblas/0.3.3
  python: # Homebrew calls python@2.7.15 via "python@2"
    paths:
      python@2.7.15 %clang@9.0.0-apple: /usr/local/Cellar/python@2/2.7.15_1
      python@3.7.1 %clang@9.0.0-apple: /usr/local/Cellar/python/3.7.1
  all:
    compiler: [clang@9.0.0-apple, clang, gcc]
    providers:
      awk: [gawk]
      blas: [openblas, atlas, intel-mkl, netlib-lapack]
      daal: [intel-daal]
      elf: [libelf, elftoolchain, elfutils]
      golang: [gcc]
      gl: [opengl, mesa]
      ipp: [intel-ipp]
      java: [jdk]
      lapack: [openblas, atlas, intel-mkl, netlib-lapack]
      mkl: [intel-mkl]
      mpe: [mpe2]
      mpi: [openmpi, mpich, intel-mpi]
      opencl: [pocl]
      openfoam: [openfoam-com, openfoam-org, foam-extend]
      pil: [py-pillow]
      pkgconfig: [pkgconf, pkg-config]
      scalapack: [netlib-scalapack, intel-mkl]
      szip: [libszip, libaec]
      tbb: [intel-tbb]
      jpeg: [libjpeg-turbo, libjpeg]
      cmake: [cmake%clang]
    paths: {}
    modules: {}
    buildable: true
    version: []
```

`.spack/darwin/compilers.yaml`:
```yaml
compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: sierra
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77: /usr/local/Cellar/gcc/8.2.0/bin/gfortran-8
      fc: /usr/local/Cellar/gcc/8.2.0/bin/gfortran-8
    spec: clang@9.0.0-apple
    target: x86_64
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: sierra
    paths:
      cc: /usr/local/Cellar/gcc/8.2.0/bin/gcc-8
      cxx: /usr/local/Cellar/gcc/8.2.0/bin/g++-8
      f77: /usr/local/Cellar/gcc/8.2.0/bin/gfortran-8
      fc: /usr/local/Cellar/gcc/8.2.0/bin/gfortran-8
    spec: gcc@8.2.0
    target: x86_64
```

In a new, clean `spack` install on macOS with the installed Homebrew packages mentioned above, and the spack configuration files above, run:
```console
spack install curl git lmod openblas python@2.7.15 python@3.7.1
```

Then create a new directory (wherever you want) containing the following two files:

`CMakeLists.txt`:
```cmake
cmake_minimum_required(VERSION 3.0)
project(test_lapack_link C CXX Fortran)

include(FortranCInterface)
FortranCInterface_HEADER("${CMAKE_BINARY_DIR}/test_lapack_link.hpp"
  MACRO_NAMESPACE "TEST_FC_"
  SYMBOLS ilaver)
FortranCInterface_VERIFY(CXX)

set(BLA_VENDOR "OpenBLAS")

find_package(BLAS REQUIRED)
find_package(LAPACK REQUIRED)

add_executable(test_lapack_link test_lapack_link.cpp)
target_include_directories(test_lapack_link PUBLIC ${CMAKE_BINARY_DIR})
target_link_libraries(test_lapack_link ${BLAS_LIBRARIES} ${LAPACK_LIBRARIES})
```

`test_lapack_link.cpp`:
```cpp
#include <iostream.hpp>
#include "test_lapack_link.hpp"

extern "C"
{
  int ilaver(int* major_version, int* minor_version, int* patch_version);
}

int main(int argc, char** argv)
{
  int major_version, minor_version, patch_version;
  ilaver(&major_version, &minor_version, &patch_version);

  std::cout << "This executable was linked to LAPACK version "
	    << major_version << "." << minor_version << "." << patch_version
	    << "." << std::endl;
  return 0;
}
```

Add to the directory containing two files a subdirectory called `build`. Run in this `build` subdirectory the following two commands:

```console
module purge
cmake ..
```
which should fail with something like the output below because BLAS cannot be found:

```console
-- The C compiler identification is AppleClang 9.0.0.9000039
-- The CXX compiler identification is AppleClang 9.0.0.9000039
-- The Fortran compiler identification is GNU 8.2.0
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - yes
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Check for working Fortran compiler: /usr/local/bin/gfortran
-- Check for working Fortran compiler: /usr/local/bin/gfortran  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /usr/local/bin/gfortran supports Fortran 90
-- Checking whether /usr/local/bin/gfortran supports Fortran 90 -- yes
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/CXX Compiler Compatibility
-- Verifying Fortran/CXX Compiler Compatibility - Success
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'blas'
--   No package 'blas' found
CMake Error at /usr/local/Cellar/cmake/3.12.4/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find BLAS (missing: BLAS_LIBRARIES)
Call Stack (most recent call first):
  /usr/local/Cellar/cmake/3.12.4/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/Cellar/cmake/3.12.4/share/cmake/Modules/FindBLAS.cmake:702 (find_package_handle_standard_args)
  CMakeLists.txt:13 (find_package)


-- Configuring incomplete, errors occurred!
See also "/Users/oxberry1/spack-lapack-bug-mwe/build/CMakeFiles/CMakeOutput.log".
```

Then run in that same `build` directory following three commands:

```console
rm -rf * # delete all CMake output files and directories
ml openblas
cmake ..
```

which should yield the following output:

```console
dyld: Library not loaded: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
  Referenced from: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
  Reason: Incompatible library version: vecLib requires version 1.0.0 or later, but libLAPACK.dylib provides version 0.0.0
```

Then run the following commands, all within the same `build` directory:

```console
echo $DYLD_LIBRARY_PATH  # should output /usr/local/Cellar/openblas/0.3.3/lib or similar
echo $LD_LIBRARY_PATH       # should output /usr/local/Cellar/openblas/0.3.3/lib or similar
echo $DYLD_FALLBACK_LIBRARY_PATH $ should also output /usr/local/Cellar/openblas/0.3.3/lib
unset DYLD_LIBRARY_PATH
rm -rf * # delete all CMake output files and directories
cmake .. && make && ./test_lapack_link
```

which should yield the following output:

```console
-- The C compiler identification is AppleClang 9.0.0.9000039
-- The CXX compiler identification is AppleClang 9.0.0.9000039
-- The Fortran compiler identification is GNU 8.2.0
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - yes
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Check for working Fortran compiler: /usr/local/bin/gfortran
-- Check for working Fortran compiler: /usr/local/bin/gfortran  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /usr/local/bin/gfortran supports Fortran 90
-- Checking whether /usr/local/bin/gfortran supports Fortran 90 -- yes
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/CXX Compiler Compatibility
-- Verifying Fortran/CXX Compiler Compatibility - Success
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Found BLAS: /usr/local/Cellar/openblas/0.3.3/lib/libopenblas.dylib  
-- Looking for Fortran cheev
-- Looking for Fortran cheev - found
-- A library with LAPACK API found.
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/oxberry1/spack-lapack-bug-mwe/build
Scanning dependencies of target test_lapack_link
[ 50%] Building CXX object CMakeFiles/test_lapack_link.dir/test_lapack_link.cpp.o
[100%] Linking CXX executable test_lapack_link
[100%] Built target test_lapack_link
This executable was linked to LAPACK version 3.8.0.
```

This extended example should illustrate the problems I'm running into if `DYLD_LIBRARY_PATH` is set by modules -- this variable tends to cause conflicts when resolving things like BLAS and LAPACK that are already installed on the system.
@goxberry I may not fully understand the issue here, but have you considered [defining a `filter`](https://spack.readthedocs.io/en/latest/module_file_support.html#filter-out-environment-modifications) in your `modules.yaml` config file, to remove the env var(s) that you don't want?
I also have to add the variable I do want, hence the swap. Also, there are
several Stack Overflow posts that suggest using DYLD_FALLBACK_LIBRARY_PATH
instead of DYLD_LIBRARY_PATH.

On Wed, Jan 16, 2019 at 20:34 Owen Solberg <notifications@github.com> wrote:

> @goxberry <https://github.com/goxberry> I may not fully understand the
> issue here, but have you considered defining a filter
> <https://spack.readthedocs.io/en/latest/module_file_support.html#filter-out-environment-modifications>
> in your modules.yaml config file, to remove the env var(s) that you don't
> want?
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/spack/spack/pull/9221#issuecomment-455039331>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AATnx69ihavzjMPodbhdetF9N8Wnr7KHks5vD_1qgaJpZM4Wks3P>
> .
>
-- 
Geoffrey Oxberry, Ph.D., E.I.T.
goxberry@gmail.com

@goxberry --

> I also have to add the variable I do want, hence the swap.

If you're talking about *environment variables*, you can both:

- filter unwanted ones from your module files ([docs here](https://spack.readthedocs.io/en/latest/tutorial_modules.html#filter-unwanted-modifications-to-the-environment)); 
- add new/custom variables ([docs here](https://spack.readthedocs.io/en/latest/tutorial_modules.html#add-custom-environment-modifications)); and
- go hog-wild and rework the templates ([docs here](https://spack.readthedocs.io/en/latest/tutorial_modules.html#working-with-templates)).

If you're monkeying with e.g. Cmake build variables, then the modulefile hacks won't help.
@hartzell I’m talking about environment variables, not CMake build variables. The issue is proper path resolution during dynamic linking — using `LD_LIBRARY_PATH` simply picks up the wrong path, so OS X’s Accelerate/vecLib BLAS and LAPACK get linked instead of OpenBLAS, causing errors.

To quote from https://stackoverflow.com/questions/3146274/is-it-ok-to-use-dyld-library-path-on-mac-os-x-and-whats-the-dynamic-library-s:

> As you've noted, DYLD_LIBRARY_PATH behaves like LD_LIBRARY_PATH on other *nix. However, there is another environment variable you should look at called DYLD_FALLBACK_LIBRARY_PATH.

> In general, these are (both on osx and linux) suggested only for development use as they can cause symbol lookup errors when you override with a library that does not have the same symbol table. A good example of this is when you attempt to override the default install of VecLib (e.g. blas lapack) with a custom install. This will cause symbol not found errors in applications linked to the system VecLib if DYLD_LIBRARY_PATH is set and the reverse (symbol lookup errors in custom applications) if it is not. This is due to the system blas/lapack not being a full implementation of the ATLAS libs.

> DYLD_FALLBACK_LIBRARY_PATH will not produce these problems.

> When installing libraries to a non-standard location, DYLD_FALLBACK_LIBRARY_PATH is much more sane. This will look for symbols in libraries provided in the default paths and if the symbol is not found there, fall back to the specified path.

> The benefit is that this process will not cause symbol lookup errors in applications compiled against the default libraries.

> In general, when libraries are installed to non-standard locations absolute paths should be specified which negates the ambiguity of the dynamic lookup.

There is no way, as far as I know, to set the absolute path for BLAS/LAPACK library lookup with recent versions of CMake. I have tried setting, e.g., `BLAS_DIR`, `LAPACK_DIR` as arguments to CMake build systems that use my OpenBLAS install to no avail. So, while I would use absolute paths, that does not seem to be possible at present. (See 
https://github.com/xianyi/OpenBLAS/issues/1522, marked closed, but other users still report seeing these sorts of unresolved dynamic linking errors.)

@goxberry said:
> @hartzell I’m talking about environment variables [...]

I may be off the rails, but I thought you were talking about needing to maintain a private branch of Spack with some changes that allowed you to set and unset various environment variables in the module files for various packages so that you could use them on the Mac.

If/to-the-extent that is true, I'm just pointing out that you can influence the contents of those module files with a local `modules.yaml` file that you apply to the main tree, you don't need to keep a separate tree.

If, on the other hand, you need to change the packages themselves and/or the *e.g.* Cmake build_system code, then you *do* need to keep your private branch going.

Sorry if I've mis-understood what you're trying to achieve.

@goxberry I encountered a similar library load error using a Homebrew-installed `cmake` after `module load`ing a bunch of Spack link-time prerequisites: one of Apple's frameworks was loading a Spack `libjpeg-turbo` install instead of its built-in libjpeg.

The solution for my problem (and perhaps yours) was to add an `environment_blacklist` entry to the `module.yaml` file:
```yaml
modules:
  enable::
    - lmod
  lmod:
    all:
      filter:
        environment_blacklist: ['DYLD_LIBRARY_PATH']
```

That prevents the dynamic lib path from being altered when a spack module is loaded, thus *not* overriding Apple's framework installation.

---
Note: my final configuration is
```yaml
modules:
  enable::
    - lmod
  lmod:
    #blacklist_implicits: true
    core_compilers:
      - 'clang@9.0.0-apple'
    hierarchy:
      - mpi
    hash_length: 0
    all:
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH', 'DYLD_LIBRARY_PATH']
      environment:
        set:
          '${PACKAGE}_ROOT': '${PREFIX}'
    ^python:
      autoload:  'direct'
```
Thanks @sethrj! This example is a good way to kludge around spack’s behavior at the moment.

Longer-term, I still believe that the defaults should be reverted or changed because I suspect this environment variable issue will be a pain point for new spack users on macOS.
I understand @adamjstewart 's desire for "exactly" equivalent dynamic library behavior between Mac and Linux systems, but I agree with you that it doesn't make as much sense on the mac (where so many more codes are linked against system libraries and frameworks).

Regardless, the problem and workaround are *definitely* worth writing up in the documentation somewhere.
Also I think a possible solution would be relying on `DYLD_FALLBACK_LIBRARY_PATH` instead since it is lower priority on Darwin, but still prevents anything from breaking on Spack.

Resolving patch:
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -145,6 +145,7 @@ def clean_environment():
     env.unset('CPATH')
     env.unset('LD_RUN_PATH')
     env.unset('DYLD_LIBRARY_PATH')
+    env.unset('DYLD_FALLBACK_LIBRARY_PATH')
 
     build_lang = spack.config.get('config:build_language')
     if build_lang:
diff --git a/lib/spack/spack/main.py b/lib/spack/spack/main.py
--- a/lib/spack/spack/main.py
+++ b/lib/spack/spack/main.py
@@ -692,7 +692,10 @@ def main(argv=None):
     # Spack clears these variables before building and installing packages,
     # but needs to know the prior state for commands like `spack load` and
     # `spack env activate that modify the user environment.
-    for var in ('LD_LIBRARY_PATH', 'DYLD_LIBRARY_PATH'):
+    recovered_vars = (
+        'LD_LIBRARY_PATH', 'DYLD_LIBRARY_PATH', 'DYLD_FALLBACK_LIBRARY_PATH'
+    )
+    for var in recovered_vars:
         stored_var_name = 'SPACK_%s' % var
         if stored_var_name in os.environ:
             os.environ[var] = os.environ[stored_var_name]
diff --git a/lib/spack/spack/user_environment.py b/lib/spack/spack/user_environment.py
--- a/lib/spack/spack/user_environment.py
+++ b/lib/spack/spack/user_environment.py
@@ -40,7 +40,7 @@ def prefix_inspections(platform):
 
     if platform == 'darwin':
         for subdir in ('lib', 'lib64'):
-            inspections[subdir].append('DYLD_LIBRARY_PATH')
+            inspections[subdir].append('DYLD_FALLBACK_LIBRARY_PATH')
 
     return inspections
 


Test patch: