Repository: spack/spack

Issue:
`install --use-cache` fails when installing into original location
I created a buildcache from things built in `~/spack` (aka `/home/hartzell/spack/opt/spack`).

Then I removed the package and tried to re-install it (simulating an install into my home directory on another machine).

Every package I tried (`tmux`, `the-silver-searcher`, `tree`) failed, complaining (details below in the "recreate" section):

```console
==> Relocating package from
  /home/hartzell/spack/opt/spack to /home/hartzell/spack/opt/spack.
==> Error:
 /tmp/tmpxEG6CD/tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl/bin/tmux
contains string
 /home/hartzell/spack/opt/spack
after replacing it in rpaths.
Package should not be relocated.
 Use -a to override.
```

I expected it to work.

The error message is mis-leading, there doesn't seem to be any place to specify the `-a` arg to `spack install`.

If I moved my spack tree to `~/foo/spack` I was able to install `tmux` from the buildcache:

```console
[hartzell@bunny ~]$ mkdir foo
[hartzell@bunny ~]$ mv spack foo
[hartzell@bunny ~]$ cd foo
[hartzell@bunny foo]$ cd spack/
[hartzell@bunny spack]$ . share/spack/setup-env.sh
[hartzell@bunny spack]$ which spack
~/foo/spack/bin/spack
[hartzell@bunny spack]$ spack uninstall tmux
==> The following packages will be uninstalled:

-- linux-centos7-x86_64 / gcc@5.5.0 -----------------------------
szc3vlr tmux@2.7%gcc

==> Do you want to proceed? [y/N] y
==> Successfully uninstalled tmux@2.7%gcc@5.5.0 arch=linux-centos7-x86_64 /szc3vlr
[hartzell@bunny spack]$ spack install --use-cache tmux
==> zlib is already installed in /home/hartzell/foo/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/zlib-1.2.11-ijtgxbh42fcxbwklab6vdvmlrjadep56
==> openssl is already installed in /home/hartzell/foo/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/openssl-1.0.2n-xduivakzy6qkikvpdwad4cndjrxxjrtm
==> libevent is already installed in /home/hartzell/foo/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/libevent-2.0.21-eb6ntdlm3v63a2yhrpxk3cobbrvttep4
==> pkgconf is already installed in /home/hartzell/foo/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/pkgconf-1.4.0-oq27yb7w5zuwmhkozaepre2g6qsh7fcm
==> ncurses is already installed in /home/hartzell/foo/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/ncurses-6.0-cknorkglnsg7wwc6txojd67o6ryvujmt
==> Installing tmux
==> Searching for binary cache of tmux
==> Finding buildcaches in /tmp/mirror//build_cache
==> Installing tmux from binary cache
==> Already downloaded /home/hartzell/foo/spack/var/spack/stage/build_cache/linux-centos7-x86_64-gcc-5.5.0-tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl.spack
gpg: Signature made Sun 20 May 2018 03:16:18 AM UTC using RSA key ID 2BD81240
gpg: Good signature from "George Hartzell (GPG created for Spack) <hartzell@alerce.com>"
==> Relocating package from
  /home/hartzell/spack/opt/spack to /home/hartzell/foo/spack/opt/spack.
==> Successfully installed tmux from binary cache
[+] /home/hartzell/foo/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl
[hartzell@bunny spack]$
```

### Steps to reproduce the issue

```console
[hartzell@bunny spack]$ spack install tmux
==> zlib is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/zlib-1.2.11-ijtgxbh42fcxbwklab6vdvmlrjadep56
==> openssl is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/openssl-1.0.2n-xduivakzy6qkikvpdwad4cndjrxxjrtm
==> libevent is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/libevent-2.0.21-eb6ntdlm3v63a2yhrpxk3cobbrvttep4
==> pkgconf is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/pkgconf-1.4.0-oq27yb7w5zuwmhkozaepre2g6qsh7fcm
==> ncurses is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/ncurses-6.0-cknorkglnsg7wwc6txojd67o6ryvujmt
==> Installing tmux
==> Using cached archive: /home/hartzell/spack/var/spack/cache/tmux/tmux-2.7.tar.gz
==> Staging archive: /home/hartzell/spack/var/spack/stage/tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl/tmux-2.7.tar.gz
==> Created stage in /home/hartzell/spack/var/spack/stage/tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl
==> No patches needed for tmux
==> Building tmux [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed tmux
  Fetch: 0.01s.  Build: 34.25s.  Total: 34.27s.
[+] /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl
[hartzell@bunny spack]$ spack buildcache create -r -a -f -d /tmp/mirror tmux
==> adding matching spec tmux@2.7%gcc@5.5.0 arch=linux-centos7-x86_64
==> recursing dependencies
==> adding dependency zlib@1.2.11%gcc@5.5.0+optimize+pic+shared arch=linux-centos7-x86_64
==> adding dependency openssl@1.0.2n%gcc@5.5.0+systemcerts arch=linux-centos7-x86_64
==> adding dependency libevent@2.0.21%gcc@5.5.0+openssl arch=linux-centos7-x86_64
==> adding dependency ncurses@6.0%gcc@5.5.0 patches=4110a40613b800da2b2888c352b64c75a82809d48341061e4de5861e8b28423f,f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks~termlib arch=linux-centos7-x86_64
==> adding dependency tmux@2.7%gcc@5.5.0 arch=linux-centos7-x86_64
==> writing tarballs to /tmp/mirror/build_cache
==> creating binary cache file for package zlib@1.2.11%gcc@5.5.0+optimize+pic+shared arch=linux-centos7-x86_64
==> creating binary cache file for package openssl@1.0.2n%gcc@5.5.0+systemcerts arch=linux-centos7-x86_64
==> creating binary cache file for package tmux@2.7%gcc@5.5.0 arch=linux-centos7-x86_64
==> creating binary cache file for package libevent@2.0.21%gcc@5.5.0+openssl arch=linux-centos7-x86_64
==> creating binary cache file for package ncurses@6.0%gcc@5.5.0 patches=4110a40613b800da2b2888c352b64c75a82809d48341061e4de5861e8b28423f,f84b2708a42777aadcc7f502a261afe10ca5646a51c1ef8b5e60d2070d926b57 ~symlinks~termlib arch=linux-centos7-x86_64
[hartzell@bunny spack]$ spack uninstall tmux
==> The following packages will be uninstalled:

-- linux-centos7-x86_64 / gcc@5.5.0 -----------------------------
szc3vlr tmux@2.7%gcc

==> Do you want to proceed? [y/N] y
==> Successfully uninstalled tmux@2.7%gcc@5.5.0 arch=linux-centos7-x86_64 /szc3vlr
[hartzell@bunny spack]$ spack install --use-cache tmux
==> zlib is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/zlib-1.2.11-ijtgxbh42fcxbwklab6vdvmlrjadep56
==> openssl is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/openssl-1.0.2n-xduivakzy6qkikvpdwad4cndjrxxjrtm
==> libevent is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/libevent-2.0.21-eb6ntdlm3v63a2yhrpxk3cobbrvttep4
==> pkgconf is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/pkgconf-1.4.0-oq27yb7w5zuwmhkozaepre2g6qsh7fcm
==> ncurses is already installed in /home/hartzell/spack/opt/spack/linux-centos7-x86_64/gcc-5.5.0/ncurses-6.0-cknorkglnsg7wwc6txojd67o6ryvujmt
==> Installing tmux
==> Searching for binary cache of tmux
==> Finding buildcaches in /tmp/mirror//build_cache
==> Installing tmux from binary cache
==> Already downloaded /home/hartzell/spack/var/spack/stage/build_cache/linux-centos7-x86_64-gcc-5.5.0-tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl.spack
gpg: Signature made Sun 20 May 2018 03:16:18 AM UTC using RSA key ID 2BD81240
gpg: Good signature from "George Hartzell (GPG created for Spack) <hartzell@alerce.com>"
==> Relocating package from
  /home/hartzell/spack/opt/spack to /home/hartzell/spack/opt/spack.
==> Error:
 /tmp/tmpxEG6CD/tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl/bin/tmux
contains string
 /home/hartzell/spack/opt/spack
after replacing it in rpaths.
Package should not be relocated.
 Use -a to override.
[hartzell@bunny spack]$
```

### Information on your system

CentOS 7 on a Digital Ocean droplet.

My spack tree was populated with the following script (apologies in advance for the bits of cruft that are still in there...):

```sh
#!/bin/sh
set -e

# export TMPDIR=/data/hartzell/tmp
SPACK="${SPACK:-${HOME}/spack}"

# having an app tree set up messes up the build.  Clear the field.
if [[ $(type -t module) == "function" ]]; then
    module purge
fi

# this host has an antique set of certificates, which prevents spack
# (curl) from fetching some applications tarballs.  This grabs a good one.
# if [[ ! -d ~/tmp/cert ]]; then
#     mkdir -p ~/tmp/cert
# fi
# if [[ -f ~/tmp/cert/cacert.pem ]]; then
#     rm ~/tmp/cert/cacert.pem
# fi

# wget -O ~/tmp/cert/cacert.pem http://curl.haxx.se/ca/cacert.pem
# export CURL_CA_BUNDLE=~/tmp/cert/cacert.pem

cat << "EOYAML" > ${SPACK}/etc/spack/config.yaml
# -------------------------------------------------------------------------
# This is the project specific spack configuration file.
# -------------------------------------------------------------------------
config:
  # Temporary locations Spack can try to use for builds.
  #
  # A value of $spack/var/spack/stage indicates that Spack should run
  # builds directly inside its install directory without staging them in
  # temporary space.
  #
  # The build stage can be purged with `spack purge --stage`.
  build_stage:
    - $spack/var/spack/stage
EOYAML

cat << "EOYAML" > ${SPACK}/etc/spack/packages.yaml
packages:
  all:
    compiler: [gcc@5.5.0, gcc@4.8.5, gcc@4.8.3]
EOYAML

cat << "EOYAML" > ${SPACK}/etc/spack/modules.yaml
modules:
  enable::
    - lmod
    - tcl
  tcl:
    hash_length: 0
    all:
      suffixes:
        '+jit': jit
    naming_scheme: '${PACKAGE}/${VERSION}'
  lmod:
    core_compilers:
      - 'gcc@4.8.5'
      - 'gcc@4.8.2'
      - 'gcc@4.8'
    hash_length: 0
    whitelist:
      - gcc
    blacklist:
      - '%gcc@4.8.2'
      - '%gcc@4.8.5'
    verbose_autoload: false
    all:
      suffixes:
        '+jit': jit
      filter:
        environment_blacklist: ['CPATH', 'LIBRARY_PATH', 'LD_LIBRARY_PATH']
      #environment:
      #  set:
      #    '${PACKAGE}_ROOT': '${PREFIX}'
    ^python:
      autoload:  'direct'
      filter:
        environment_whitelist: ['LD_LIBRARY_PATH']
EOYAML

my_compiler=gcc@5.5.0

if ! spack location -i ${my_compiler} > /dev/null 2>&1; then
    # Compiler

    ## HEADS UP: This is a bit fragile.  Make sure that you don't have
    ## a ~/.spack/.../compilers.yaml file that defines a 5.4.0,
    ## otherwise you're likely to build "my_compiler" with that one
    ## and lmod won't generate a set of Core modules.  Alt, we could
    ## grab the version from the system compiler and specify it here
    ## explicitly.  More work than it's worth for now.
    spack install ${my_compiler}

    # This might not do what you want if there's already
    # a 5.4.0 on the system....
    # Now all of the things will be built with %gcc@5.4.0
    spack compiler add --scope site $(spack location -i ${my_compiler})
fi

spack install aspell6-en; spack activate aspell6-en
spack install cask
spack install direnv
spack install emacs
spack install git
spack install htop
spack install httpie
spack install jq
spack install lmod
spack install perl
spack install the-silver-searcher
spack install tree
spack install tmux
```

I added a mirror entry to my personal (unintentionally) spack with:

```
spack mirror add mypkgs /tmp/mirror/
```




Discussion:
Actually, the resulting installation doesn't work:

```
[hartzell@bunny spack]$ opt/spack/linux-centos7-x86_64/gcc-5.5.0/tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl/bin/tmux
opt/spack/linux-centos7-x86_64/gcc-5.5.0/tmux-2.7-szc3vlr7qwrr5d6vpv4qmnnifrqj4zyl/bin/tmux: error while loading shared libraries: libncurses.so.6: cannot open shared object file: No such file or directory
[hartzell@bunny spack]$
```

I thought I'd try installing from a clean slate, using the heavy-handed blow-away `./opt` approach, only to discover that the key I'd used to sign things was also stashed down there.  I'd *AssUMe*-ed that it would be in the source tree or `~/.spack`.

```
[hartzell@bunny spack]$ rm -rf opt/
[hartzell@bunny spack]$ spack reindex
[hartzell@bunny spack]$ spack install --use-cache tmux
==> Installing zlib
==> Searching for binary cache of zlib
==> Finding buildcaches in /tmp/mirror//build_cache
==> Installing zlib from binary cache
==> Fetching file:///tmp/mirror//build_cache/linux-centos7-x86_64/gcc-5.5.0/zlib-1.2.11/linux-centos7-x86_64-gcc-5.5.0-zlib-1.2.11-ijtgxbh42fcxbwklab6vdvmlrjadep56.spack
######################################################################## 100.0%
gpg: keyring `/home/hartzell/foo/spack/opt/spack/gpg/pubring.gpg' created
gpg: Signature made Sun 20 May 2018 05:41:24 PM UTC using RSA key ID 2BD81240
gpg: Can't check signature: No public key
==> Error: Command exited with status 2:
    'gpg2' '--verify' '/tmp/tmpxgIaOT/linux-centos7-x86_64-gcc-5.5.0-zlib-1.2.11-ijtgxbh42fcxbwklab6vdvmlrjadep56.spec.yaml.asc' '/tmp/tmpxgIaOT/linux-centos7-x86_64-gcc-5.5.0-zlib-1.2.11-ijtgxbh42fcxbwklab6vdvmlrjadep56.spec.yaml'
[hartzell@bunny spack]$
```

There doesn't seem to be any way to skip signature checking on installation, so that's the end of this particular experiment.....
I've been playing around and can confirm that the buildcache mechanism generally works (now that I've confirmed that, you're probably all breathing a deep sigh of relief) and have also re-created this example in a simpler spack deployment:

```sh
git clone https://github.com/spack/spack
cd spack
export PATH=$(pwd)/bin:$PATH
spack install tree
spack gpg create "George Hartzell" ghartzell@audentestx.com
spack buildcache create -d /tmp/ghartzell-spack-mirror tree
spack uninstall -a
spack mirror add mine file:///tmp/ghartzell-spack-mirror/
spack install --use-cache tree
# That failed with:
# ==> Error:
#  /tmp/tmpZmTNOv/tree-1.7.0-52otbnhv4je235jvhstlem4e7axti562/bin/tree
# contains string
#  /home/hartzell/tmp/spack/opt/spack
# after replacing it in rpaths.
# Package should not be relocated.
#  Use -a to override.
pwd
cd ..
cp -rp spack spack-testing
cd spack-testing
./bin/spack uninstall -a
./bin/spack install --use-cache tree
./opt/spack/linux-centos7-x86_64/gcc-4.8.5/tree-1.7.0-52otbnhv4je235jvhstlem4e7axti562/bin/tree
# YAY!
```

Resolving patch:
diff --git a/lib/spack/spack/relocate.py b/lib/spack/spack/relocate.py
--- a/lib/spack/spack/relocate.py
+++ b/lib/spack/spack/relocate.py
@@ -355,6 +355,7 @@ def relocate_binary(path_names, old_dir, new_dir, allow_root):
                                 rpaths, deps, idpath,
                                 new_rpaths, new_deps, new_idpath)
             if (not allow_root and
+                old_dir != new_dir and
                 strings_contains_installroot(path_name, old_dir)):
                     raise InstallRootStringException(path_name, old_dir)
 
@@ -373,6 +374,7 @@ def relocate_binary(path_names, old_dir, new_dir, allow_root):
                                                   old_dir, new_dir)
                 modify_elf_object(path_name, new_rpaths)
                 if (not allow_root and
+                    old_dir != new_dir and
                     strings_contains_installroot(path_name, old_dir)):
                         raise InstallRootStringException(path_name, old_dir)
     else:


Test patch: