Repository: spack/spack

Issue:
R does not work after relocation
### Steps to reproduce

```console
$ docker run -it --rm spack/ubuntu-bionic:latest
:~# spack mirror add v0.21.0-build_systems https://binaries.spack.io/v0.21.0/build_systems
:~# spack buildcache keys --install --trust
:~# spack install --use-buildcache=only /eqteloqhjzix6ta373ruzt5imvvbcesc
:~# spack load r
:~# R
> help()
> install.packages('bit64')
```

Notice that R commands do not work because the default packages are not found.

### Error message

```console
:~# R

R version 4.3.0 (2023-04-21) -- "Already Tomorrow"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
> help()
Error in help() : could not find function "help"
> install.packages('bit64')

Error in install.packages("bit64") :
  could not find function "install.packages"
```

### Information on your system

```console
# spack debug report
* **Spack:** 0.21.0.dev0 (910190f55bb5467305dd75a4dac8c60f1f51e283)
* **Python:** 3.6.9
* **Platform:** linux-ubuntu18.04-skylake_avx512
* **Concretizer:** clingo
```

After struggling to understand why R installed from a buildcache was not working on our local systems (running CentOS 7), I reproduced the issue using Spack's own distributed ubuntu-bionic Docker image and Spack's own R binary from binaries.spack.io. 

If I force Spack to build the same R from source (i.e. `spack install --use-buildcache=package:never,dependencies:only /eqteloqhjzix6ta373ruzt5imvvbcesc`), then R _does_ work as expected. Some part of the relocation process seems to be missing or silently failing when the binary is used.

Here is what one might expect for a properly-working R:

```console
root@6949bf78224a:~# spack install --use-buildcache=package:never,dependencies:only /eqteloqhjzix6ta373ruzt5imvvbcesc
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/bzip2-1.0.8-5mghoqz4ngtwq3v67ewyun64xv4o53m3
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/libiconv-1.17-ogz4kf67zest4ajc6dtwxiokntbip24x
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/nghttp2-1.57.0-roxxir4emd7bmj7qv2733unvagl2xlg6
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/zlib-ng-2.1.4-syxyq4lwjyg6jcwdmueu3pcwz6vv4imc
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/gmake-4.4.1-s7ksjmgajvb7sjqiee7boqyapewgzirk
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/openjdk-11.0.20.1_1-emyzuxuhihfdjtxtq7raapcw2qufuym2
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/openssl-3.1.3-i7dl3uq4lad3dchztaghe2ae5vwiyloh
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/libunistring-1.1-iuznve7ejcwrfepyqq2f5icu7l4oeuc4
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/ncurses-6.4-cod5aaaoh43yzbzkf4ljb3nfvjr637hh
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/berkeley-db-18.1.40-dzr37itwibvfx66orqzx2oya5fj4jits
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/xz-5.4.1-q2vsdwtcjmoirhf7wt4ffijey7buklby
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/which-2.21-3j5ymd5gybivcdmkxy53tdgq5vzuk5ys
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/pcre2-10.42-quubhlacmkwpz43ygmqn3vnwslxpgdjy
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/icu4c-67.1-gw4x3xgvtys4b2aahemlcpqh2v7zmwiz
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/zstd-1.5.5-dnn2xfy6g4mybb4aqijdyxt66auzn6vf
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/pigz-2.7-5v3za7qyrp6mt3bcapyy4ofoolslr2hp
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/libidn2-2.3.4-uh2kdjvixrfblju2czgx3yzhnxp4xpkm
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/readline-8.2-abh5mvvbkpfvn6wrxf5n3m633t76b6kw
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/libxml2-2.10.3-hzocw3suyawfzi6ufnf7kkvso4zey5tk
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/tar-1.34-oolmzjtck3er63v7mqv2x2dqdfdffxq3
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/curl-8.4.0-5oclhntmwoiecmbssryqc5nsdtjrfp2z
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/gdbm-1.23-puhkytr3sslzfdbwpcbunu5orlqa34if
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/gettext-0.22.3-q4bymsxue2tqaq6sj4d32vu6v7rrvviv
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/perl-5.38.0-7lnnnoavjsd2ov5326dv7x3hrtquydyb
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/texinfo-7.0.3-zora4oj5z4bfjkz3oai324wgeam5eagn
==> Installing r-4.3.0-eqteloqhjzix6ta373ruzt5imvvbcesc [26/26]
==> Using cached archive: /opt/spack/var/spack/cache/_source-cache/archive/45/45dcc48b6cf27d361020f77fde1a39209e997b81402b3663ca1c010056a6a609.tar.gz
==> No patches needed for r
==> r: Executing phase: 'autoreconf'
==> r: Executing phase: 'configure'
==> r: Executing phase: 'build'
==> r: Executing phase: 'install'
==> r: Successfully installed r-4.3.0-eqteloqhjzix6ta373ruzt5imvvbcesc
  Stage: 1.49s.  Autoreconf: 0.00s.  Configure: 33.87s.  Build: 2m 36.27s.  Install: 1.65s.  Post-install: 0.62s.  Total: 3m 14.02s
[+] /opt/spack/opt/spack/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/r-4.3.0-eqteloqhjzix6ta373ruzt5imvvbcesc
root@6949bf78224a:~# spack load /eqteloqhjzix6ta373ruzt5imvvbcesc
root@6949bf78224a:~# R

R version 4.3.0 (2023-04-21) -- "Already Tomorrow"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>
```

Notice R gives no warnings about missing default packages.


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I think the issue is that R compresses the original install prefix and decompresses at runtime, `strace` shows it tries to execute `<old prefix>/bin/which` for example.
Okay, this is what happens:

R stores an absolute path to `<prefix>/bin/which` in byte code, and that byte code is compressed. It can be retrieved like this:

```shell
$ ./bin/R --quiet -e 'readRDS("./rlib/R/library/base/R/base.rdx")$variables$Sys.which' 2>/dev/null
> readRDS("./rlib/R/library/base/R/base.rdx")$variables$Sys.which
[1] 133838   2044
```

meaning the serialized stuff for `Sys.which` is stored at offset `133838` with length `2044` in `base.rdb`. That itself starts with a 4 byte integer, so those have to be skipped. Then a zlib header follows:

```shell
$ dd if=./rlib/R/library/base/R/base.rdb bs=1 skip=133842 count=2044 status=none | zlib-flate -uncompress
...
/home/software/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh/linux-ubuntu18.04-x86_64_v3/gcc-7.5.0/which-2.21-3j5ymd5gybivcdmkxy53tdgq5vzuk5ys/bin/which
...
```

So in short: a string we need to relocate is stored as compressed, serialized byte code in some sort of database for serialized objects. In principle it's possible to decompress the blocks in `base.rdb`, rewrite strings, store again, and update the offsets in `base.rdx` -- but that's pretty involved.

Hopefully R can use `which` from PATH instead of an absolute path?


> Hopefully R can use `which` from PATH instead of an absolute path?

```m4
AC_PATH_PROGS(WHICH, which, which)
if test "${WHICH}" = which ; then
  ## needed to build and run R
  ## ends up hard-coded in the utils package
  AC_MSG_ERROR([[which is required but missing]])
fi
```

looks like we can't.

Another alternative is to look if we can disable compression of byte code. (Edit: looks like you can, but now idk where the source code for `Sys.which` is coming from...)
Disabling byte compilation doesn't help, it still stores something compressed & serialized in `base.rdb`. The file is smaller, so I guess there's a difference between byte code and compiled byte code.

Further options:

- Easiest is to patch the R build system to allow `WHICH=which` and expect it to be in `PATH`.
- Disabling compression of `rdb` entries (probably a bad idea)
- Ask R people to create a symlink to `which` in their prefix, have them register that in their R sources, so no absolute paths are registered in bytecode.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/r/package.py b/var/spack/repos/builtin/packages/r/package.py
--- a/var/spack/repos/builtin/packages/r/package.py
+++ b/var/spack/repos/builtin/packages/r/package.py
@@ -108,6 +108,10 @@ class R(AutotoolsPackage):
     # temporary fix to lower the optimization level.
     patch("change_optflags_tmp.patch", when="%fj@4.1.0")
 
+    # Make R use a symlink to which in Sys.which, otherwise an absolute path
+    # gets stored as compressed byte code, which is not relocatable
+    patch("relocate-which.patch")
+
     build_directory = "spack-build"
 
     # R custom URL version


Test patch: