Repository: bazelbuild/bazel
Issue: Paths to Param Files in Command Lines Refer to Host Execroot Paths (instead of sandbox execroot paths) With `--incompatible_sandbox_hermetic_tmp=true`
Labels: type: bug
team-Local-Exec

## Issue

Building Bazel at HEAD with Bazel 7.0.0 (with `--incompatible_sandbox_hermetic_tmp=true`, as is the [default in Bazel 7](https://github.com/bazelbuild/bazel/pull/19943)) and an output base on `/tmp` yields the following error about a `.params` file not being found:
```console
❯ bazel-7.0.0-linux-x86_64 --output_user_root=/tmp/rrbutani-bazel build //src:bazel -c opt
ERROR: /src/dev/bazel/src/main/java/com/google/devtools/build/lib/analysis/BUILD:153:13: Compiling Java headers src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar (109 source files) failed: (Exit 1): linux-sandbox failed: error executing Turbine command
  (cd /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/execroot/_main && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
    PATH=/usr/local/bin:/usr/bin:/bin \
    TMPDIR=/tmp \
  /tmp/bazel-rrbutani/install/89a68939cbf63eb54205fdf943a58b15/linux-sandbox -W /tmp/bazel-working-directory/_main -t 15 -w /tmp/bazel-execroot/_main -w /tmp -w /dev/shm -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/execroot -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-execroot -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/execroot -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-working-directory -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/external/rules_java~7.3.1~toolchains~remote_java_tools_linux -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/0 -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/external/rules_java~7.3.1~toolchains~remotejdk21_linux -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/1 -M /src/dev/bazel -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/2 -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp -m /tmp -S /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/stats.out -D /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/debug.out -- external/rules_java~7.3.1~toolchains~remote_java_tools_linux/java_tools/turbine_direct_graal '-Dturbine.ctSymPath=external/rules_java~7.3.1~toolchains~remotejdk21_linux/lib/ct.sym' @bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params)
java.lang.AssertionError: params file does not exist: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params
        at com.google.turbine.options.TurbineOptionsParser.expandParamsFiles(TurbineOptionsParser.java:182)
        at com.google.turbine.options.TurbineOptionsParser.parse(TurbineOptionsParser.java:49)
        at com.google.turbine.options.TurbineOptionsParser.parse(TurbineOptionsParser.java:38)
        at com.google.turbine.main.Main.compile(Main.java:133)
        at com.google.turbine.main.Main.main(Main.java:89)
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.225s, Critical Path: 0.15s
INFO: 3 processes: 3 internal.
ERROR: Build did NOT complete successfully
```

If `--incompatible_sandbox_hermetic_tmp=false` is passed, the above command succeeds.

## Details

Running with `--sandbox_debug` produces the following:

<details><summary>Click to expand</summary>

```
DEBUG: Sandbox debug output for Turbine //src/main/java/com/google/devtools/build/lib/analysis:analysis_cluster: 1702418703.999694695: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1702418703.999718491: src/main/tools/linux-sandbox.cc:165: Netns is 0
1702418703.999720291: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1702418704.000568193: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 30502
1702418704.000590115: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1702418704.000652861: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1702418704.000775265: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/execroot -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-execroot
1702418704.000791794: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/execroot -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-working-directory
1702418704.000799418: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/external/rules_java~7.3.1~toolchains~remote_java_tools_linux -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/0
1702418704.000809258: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/external/rules_java~7.3.1~toolchains~remotejdk21_linux -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/1
1702418704.000815958: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /src/dev/bazel -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/2
1702418704.000834430: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp -> /tmp
1702418704.000848407: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp/bazel-execroot/_main
1702418704.000854112: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1702418704.000869304: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1702418704.000876579: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /tmp/bazel-working-directory/_main
1702418704.000949533: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1702418704.000959313: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1702418704.000963615: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1702418704.000967575: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1702418704.000971798: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1702418704.000975643: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1702418704.000979660: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1702418704.000993590: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1702418704.001141631: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1702418704.001206959: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /var
1702418704.001215631: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
<unrelated paths snipped>
1702418704.002958323: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-execroot
1702418704.002966211: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-execroot, nullptr, 2101281, nullptr) failure (No such file or directory) ignored
1702418704.002970402: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-working-directory
1702418704.002972290: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-working-directory, nullptr, 2101281, nullptr) failure (No such file or directory) ignored
1702418704.002974770: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/0
1702418704.002976413: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/0, nullptr, 2101281, nullptr) failure (No such file or directory) ignored
1702418704.002988558: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/1
1702418704.002990230: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/1, nullptr, 2101281, nullptr) failure (No such file or directory) ignored
1702418704.002997175: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/2
1702418704.002998951: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/2, nullptr, 2101281, nullptr) failure (No such file or directory) ignored
1702418704.003001364: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1702418704.003004727: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-execroot
1702418704.003008764: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-working-directory
1702418704.003012225: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/0
1702418704.003015831: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/1
1702418704.003029328: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/2
1702418704.003033205: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp/bazel-execroot/_main
1702418704.003036504: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1702418704.003039277: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-execroot
1702418704.003042065: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp/bazel-execroot/_main
1702418704.003044940: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-working-directory
1702418704.003047774: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/0
1702418704.003050505: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/1
1702418704.003057570: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/2
1702418704.003060582: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1702418704.003064295: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp/bazel-working-directory/_main
1702418704.003087515: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1702418704.003209294: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1702418704.005588552: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x100
1702418704.005591954: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 1
1702418704.034940719: src/main/tools/linux-sandbox.cc:243: child exited normally with code 1
```

</details>

Inspecting the mounts and their ordering above confirms that contents in Bazel source roots and execroot should be visible: these mounts are bind mounted into the hermetic tmp location (not `/tmp` directly) before `/tmp` is pivoted to the hermetic tmp location (the final `bind mount:` in the above).

---

By altering the `linux-sandbox` invocation above we can confirm that the `bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params` file is a dangling symlink, within the sandbox:
```bash
/tmp/bazel-rrbutani/install/89a68939cbf63eb54205fdf943a58b15/linux-sandbox \
    -W /tmp/bazel-working-directory/_main \
    -t 15 \
    -w /tmp/bazel-execroot/_main \
    -w /tmp \
    -w /dev/shm \
    -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/execroot \
        -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-execroot \
    -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/execroot \
        -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-working-directory \
    -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/external/rules_java~7.3.1~toolchains~remote_java_tools_linux \
        -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/0 \
    -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/external/rules_java~7.3.1~toolchains~remotejdk21_linux \
        -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/1 \
    -M /src/dev/bazel \
        -m /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp/bazel-source-roots/2 \
    -M /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/_hermetic_tmp \
        -m /tmp \
    -S /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/stats.out \
    -D /dev/null \
    -- ls --color -l bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params
```
```console-output
lrwxrwxrwx 1 rrbutani users 179 Dec 12 14:05 bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params
```

Inspecting `bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params` within this action's execroot (`/tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/execroot/`) confirms that the file is a symlink that refers to the host's execroot path rather than that of the sandbox (i.e. `/tmp/bazel-execroot/`):
```console
❯ ls /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params -l
lrwxrwxrwx 1 rrbutani users 179 Dec 12 14:05 /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params

❯ ls /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3031/execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/ -l | head
total 16
drwxr-x--- 2 rrbutani users 4096 Dec 12 14:05 libactions
lrwxrwxrwx 1 rrbutani users  128 Dec 12 14:05 libactions_provider-hjar.jar -> /tmp/bazel-execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libactions_provider-hjar.jar
lrwxrwxrwx 1 rrbutani users  179 Dec 12 14:05 libanalysis_cluster-hjar.jar-0.params -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params
lrwxrwxrwx 1 rrbutani users  141 Dec 12 14:05 libaspect_aware_attribute_mapper-hjar.jar -> /tmp/bazel-execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libaspect_aware_attribute_mapper-hjar.jar
lrwxrwxrwx 1 rrbutani users  129 Dec 12 14:05 libaspect_collection-hjar.jar -> /tmp/bazel-execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libaspect_collection-hjar.jar
lrwxrwxrwx 1 rrbutani users  130 Dec 12 14:05 libblaze_version_info-hjar.jar -> /tmp/bazel-execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libblaze_version_info-hjar.jar
lrwxrwxrwx 1 rrbutani users  134 Dec 12 14:05 libbuild_setting_provider-hjar.jar -> /tmp/bazel-execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libbuild_setting_provider-hjar.jar
drwxr-x--- 3 rrbutani users 4096 Dec 12 14:05 libconfig
lrwxrwxrwx 1 rrbutani users  129 Dec 12 14:05 libconfigured_target-hjar.jar -> /tmp/bazel-execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libconfigured_target-hjar.jar
```

(note how the other files in the above are symlinks to `/tmp/bazel-execroot/`)

## Cause

Poking around a bit, I think this is the relevant part of `rules_java`:
https://github.com/bazelbuild/bazel/blob/379ee5fd888c8f3a83ed3751bee700287fa5d7de/src/main/java/com/google/devtools/build/lib/rules/java/JavaHeaderCompileAction.java#L486-L489

Notably `rules_cc` generated param files for link actions do not seem to have this issue (note the `/tmp/bazel-execroot` in the destinations):
```console
❯ fd . /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/ | grep params | xargs ls -l --color
lrwxrwxrwx 1 rrbutani users 179 Dec 12 14:25 /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3006/execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params -> /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/execroot/_main/bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/analysis/libanalysis_cluster-hjar.jar-0.params
lrwxrwxrwx 1 rrbutani users  80 Dec 12 14:27 /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3098/execroot/_main/bazel-out/k8-dbg/bin/src/main/tools/daemonize-2.params -> /tmp/bazel-execroot/_main/bazel-out/k8-dbg/bin/src/main/tools/daemonize-2.params
lrwxrwxrwx 1 rrbutani users  83 Dec 12 14:27 /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3114/execroot/_main/bazel-out/k8-dbg/bin/src/main/tools/liblogging.a-2.params -> /tmp/bazel-execroot/_main/bazel-out/k8-dbg/bin/src/main/tools/liblogging.a-2.params
lrwxrwxrwx 1 rrbutani users 111 Dec 12 14:27 /tmp/bazel-rrbutani/705c9771f27ae01ebf0993be3b99a716/sandbox/linux-sandbox/3149/execroot/_main/bazel-out/k8-dbg/bin/src/main/java/net/starlark/java/eval/libcpu_profiler.so-2.params -> /tmp/bazel-execroot/_main/bazel-out/k8-dbg/bin/src/main/java/net/starlark/java/eval/libcpu_profiler.so-2.params
```

I _think_ that's because `rules_cc` generates its param files (for link actions) on its own using separate actions + files that are added as dependencies and doesn't use the `CommandLines.Builder` machinery:
https://github.com/bazelbuild/bazel/blob/0aacc74776ec824cc403e5c040d97a4618c64429/src/main/java/com/google/devtools/build/lib/rules/cpp/CppLinkActionBuilder.java#L950-L971

---

I am not familiar with this part of the Bazel codebase but it seems likely the problematic bit (placing host-execroot paths for the params file onto the command line) lives here:
https://github.com/bazelbuild/bazel/blob/29f1db231f2a1c1197ae6b71cb60c3360e7fcd30/src/main/java/com/google/devtools/build/lib/actions/CommandLines.java#L121-L154

And here:
https://github.com/bazelbuild/bazel/blob/41ca39a38857fbb78566f40da008f087afbe22e8/src/main/java/com/google/devtools/build/lib/analysis/actions/SpawnAction.java#L348-L360

(note the use of `getPrimaryOutput().getExecPath()` for `paramFileBasePath`)

Issue comments:
Thanks for the highly detailed report! I will look into this tomorrow.

Cc @lberki 
Woah, that was a very detailed report, thanks for investing the time and energy into it :) It helped me to zero in on the problem:


I can reproduce this. It's due to this snippet:

https://github.com/bazelbuild/bazel/blob/c6a5ebb9af3d3e114a055c566f9aca777263a945/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java#L506

It doesn't recognize `VirtualActionInput`s as something in the execroot, therefore, it doesn't transform its root to `/tmp/bazel-execroot`. Let me see if I can fix this with a well-aimed tiny change (I probably won't have the time to put together a test case, though)

The good news is that this tiny patch:

```
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java b/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java
index fb72d591e0..845d69be7b 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java
@@ -503,6 +503,8 @@ public final class SandboxHelpers {

         if (actionInput instanceof EmptyActionInput) {
           inputPath = null;
+        } else if (actionInput instanceof VirtualActionInput) {
+          inputPath = RootedPath.toRootedPath(withinSandboxExecRoot, actionInput.getExecPath());
         } else if (actionInput instanceof Artifact) {
           Artifact inputArtifact = (Artifact) actionInput;
           if (inputArtifact.isSourceArtifact() && sandboxSourceRoots != null) {
```

fixes the issue with the params files. The bad news is that it's _still_ not working, but the deploy jar creation (which is the action that fails for me) now fails with a different error. I probably won't have time to debug this any further today.


Reproduction:

```
touch WORKSPACE
mkdir -p java/a

cat > java/a/BUILD <<'EOF'
java_library(
    name = "a",
    srcs = ["A.java"],
)

java_binary(
    name = "b",
    srcs = ["B.java"],
    deps = [":a"],
)
EOF

cat > java/a/A.java <<'EOF'
package a;

public class A {
  public static int getValue() {
    return 3;
  }
}
EOF

cat > java/a/B.java <<'EOF'
import a.A;

public class B {
  public static void main(String[] args) {
    System.err.println(A.getValue());
  }
}
EOF

bazel --output_base=/tmp/ob build //java/a:b_deploy.jar

```
I continued debugging @lberki's reproducer and it turns out that the next issue is most likely https://github.com/bazelbuild/bazel/issues/20518, given that the failure points to this file:
https://github.com/bazelbuild/bazel/blob/48374dfd677248be8c1318b5c6fbe885d6c690a7/tools/build_defs/build_info/bazel_java_build_info.bzl#L70
@fmeum I think you're right -- I didn't get even far enough to find out that the missing file is a symlink but it indeed is.
BTW: I'm running it either this issue or a very similar one showing up when `external/bazel_tools/tools/test/generate-xml.sh` tries to run:

```
$ bazel --output_base=$(mktemp -d -t tmp-bazel-base-XXXXXXXXXX) test //:some_test --sandbox_debug
[...]
1702872574.903264941: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/162/_hermetic_tmp -> /tmp
1702872574.903285799: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp/bazel-execroot/wd
[...]
1702872574.973004420: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/_hermetic_tmp -> /tmp
1702872574.973021869: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp/bazel-execroot/wd
[...]
ERROR: /home/bcs/wd/BUILD:9:12: Testing //:some_test failed: (Exit 1): linux-sandbox failed: error executing TestRunner command 
  (cd /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/execroot/wd && \
[...]
  /home/bcs/.cache/bazel/_bazel_bcs/install/89a68939cbf63eb54205fdf943a58b15/linux-sandbox \
   -W /tmp/bazel-working-directory/wd \
   -t 15 \
   -w /tmp/bazel-execroot/wd \
   -w /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/execroot/wd/_tmp/270135f98d0932ba0f76c9b90a5c8ea3 \
   -w /tmp \
   -w /dev/shm \
   -M /tmp/tmp-bazel-base-ms8mQWfOXq/execroot \
   -m /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/_hermetic_tmp/bazel-execroot \
   -M /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/execroot \
   -m /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/_hermetic_tmp/bazel-working-directory \
   -M /tmp/tmp-bazel-base-ms8mQWfOXq/external/bazel_tools \
   -m /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/_hermetic_tmp/bazel-source-roots/0 \
   -M /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/_hermetic_tmp \
   -m /tmp \
   -S /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/stats.out \
   -D /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/debug.out \
   -- \
   external/bazel_tools/tools/test/generate-xml.sh \
      bazel-out/k8-fastbuild/testlogs/some_test/test.log \
      bazel-out/k8-fastbuild/testlogs/some_test/test.xml 0 1)
[...]
$ ll /tmp/tmp-bazel-base-ms8mQWfOXq/sandbox/linux-sandbox/163/execroot/wd/external/bazel_tools/tools/test/generate-xml.sh
```

Every path starting with `/tmp` that isn't under `/tmp/tmp-bazel-base-ms8mQWfOXq/` seems to be wrong. I'm seeing at least these three:
- `/tmp`
- `/tmp/bazel-working-directory/`
- `/tmp/bazel-execroot/`


@bazel-io flag
@bazel-io fork 7.1.0
@iancha1992 Forget to say that we should really have this target 7.0.1 - maybe not the full fix, but something that makes the default configuration work with it.
https://github.com/bazelbuild/bazel/pull/20603 is a draft fix for this (it contains the aforementioned two-line change and the more involved fix for #20518)
A fix for this issue has been included in [Bazel 7.1.0 RC1](https://releases.bazel.build/7.1.0/rc1/index.html). Please test out the release candidate and report any issues as soon as possible. Thanks!


Pull request review comments:

Patch that resolved the issue:
diff --git a/src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java b/src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
index 1afcd70e2b533b..be99bbd147220b 100644
--- a/src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
+++ b/src/main/java/com/google/devtools/build/lib/actions/FileArtifactValue.java
@@ -169,11 +169,19 @@ protected boolean couldBeModifiedByMetadata(FileArtifactValue lastKnown) {
   /**
    * Optional materialization path.
    *
-   * <p>If present, this artifact is a copy of another artifact. It is still tracked as a
-   * non-symlink by Bazel, but materialized in the local filesystem as a symlink to the original
-   * artifact, whose contents live at this location. This is used by {@link
-   * com.google.devtools.build.lib.remote.AbstractActionInputPrefetcher} to implement zero-cost
-   * copies of remotely stored artifacts.
+   * <p>If present, this artifact is a copy of another artifact whose contents live at this path.
+   * This can happen when it is declared as a file and not as an unresolved symlink but the action
+   * that creates it materializes it in the filesystem as a symlink to another output artifact. This
+   * information is useful in two situations:
+   *
+   * <ol>
+   *   <li>When the symlink target is a remotely stored artifact, we can avoid downloading it
+   *       multiple times when building without the bytes (see AbstractActionInputPrefetcher).
+   *   <li>When the symlink target is inaccessible from the sandboxed environment an action runs
+   *       under, we can rewrite it accordingly (see SandboxHelpers).
+   * </ol>
+   *
+   * @see com.google.devtools.build.lib.skyframe.TreeArtifactValue#getMaterializationExecPath().
    */
   public Optional<PathFragment> getMaterializationExecPath() {
     return Optional.empty();
@@ -214,6 +222,12 @@ public static FileArtifactValue createForSourceArtifact(
         xattrProvider);
   }
 
+  public static FileArtifactValue createForResolvedSymlink(
+      PathFragment realPath, FileArtifactValue metadata, @Nullable byte[] digest) {
+    return new ResolvedSymlinkFileArtifactValue(
+        realPath, digest, metadata.getContentsProxy(), metadata.getSize());
+  }
+
   public static FileArtifactValue createFromInjectedDigest(
       FileArtifactValue metadata, @Nullable byte[] digest) {
     return createForNormalFile(digest, metadata.getContentsProxy(), metadata.getSize());
@@ -439,7 +453,25 @@ public String toString() {
     }
   }
 
-  private static final class RegularFileArtifactValue extends FileArtifactValue {
+  private static final class ResolvedSymlinkFileArtifactValue extends RegularFileArtifactValue {
+    private final PathFragment realPath;
+
+    private ResolvedSymlinkFileArtifactValue(
+        PathFragment realPath,
+        @Nullable byte[] digest,
+        @Nullable FileContentsProxy proxy,
+        long size) {
+      super(digest, proxy, size);
+      this.realPath = realPath;
+    }
+
+    @Override
+    public Optional<PathFragment> getMaterializationExecPath() {
+      return Optional.of(realPath);
+    }
+  }
+
+  private static class RegularFileArtifactValue extends FileArtifactValue {
     private final byte[] digest;
     @Nullable private final FileContentsProxy proxy;
     private final long size;
@@ -462,7 +494,8 @@ public boolean equals(Object o) {
       RegularFileArtifactValue that = (RegularFileArtifactValue) o;
       return Arrays.equals(digest, that.digest)
           && Objects.equals(proxy, that.proxy)
-          && size == that.size;
+          && size == that.size
+          && Objects.equals(getMaterializationExecPath(), that.getMaterializationExecPath());
     }
 
     @Override
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/BUILD b/src/main/java/com/google/devtools/build/lib/sandbox/BUILD
index 87ac37c3644eb0..6d49080690bf08 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/BUILD
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/BUILD
@@ -17,6 +17,7 @@ java_library(
     deps = [
         "//src/main/java/com/google/devtools/build/lib/actions",
         "//src/main/java/com/google/devtools/build/lib/actions:artifacts",
+        "//src/main/java/com/google/devtools/build/lib/actions:file_metadata",
         "//src/main/java/com/google/devtools/build/lib/analysis:test/test_configuration",
         "//src/main/java/com/google/devtools/build/lib/cmdline",
         "//src/main/java/com/google/devtools/build/lib/vfs",
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/DarwinSandboxedSpawnRunner.java b/src/main/java/com/google/devtools/build/lib/sandbox/DarwinSandboxedSpawnRunner.java
index fad5cfa108b11e..2bf50dffa528d2 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/DarwinSandboxedSpawnRunner.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/DarwinSandboxedSpawnRunner.java
@@ -228,6 +228,7 @@ protected SandboxedSpawn prepareSpawn(Spawn spawn, SpawnExecutionContext context
     SandboxInputs inputs =
         helpers.processInputFiles(
             context.getInputMapping(PathFragment.EMPTY_FRAGMENT, /* willAccessRepeatedly= */ true),
+            context.getInputMetadataProvider(),
             execRoot,
             execRoot,
             packageRoots,
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/DockerSandboxedSpawnRunner.java b/src/main/java/com/google/devtools/build/lib/sandbox/DockerSandboxedSpawnRunner.java
index 3dacaeaa05ee9a..971b05cd11b99b 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/DockerSandboxedSpawnRunner.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/DockerSandboxedSpawnRunner.java
@@ -226,6 +226,7 @@ protected SandboxedSpawn prepareSpawn(Spawn spawn, SpawnExecutionContext context
     SandboxInputs inputs =
         helpers.processInputFiles(
             context.getInputMapping(PathFragment.EMPTY_FRAGMENT, /* willAccessRepeatedly= */ true),
+            context.getInputMetadataProvider(),
             execRoot,
             execRoot,
             packageRoots,
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java b/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java
index 2d593e7a28003e..9c623357e66dae 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java
@@ -281,6 +281,7 @@ protected SandboxedSpawn prepareSpawn(Spawn spawn, SpawnExecutionContext context
     SandboxInputs inputs =
         helpers.processInputFiles(
             context.getInputMapping(PathFragment.EMPTY_FRAGMENT, /* willAccessRepeatedly= */ true),
+            context.getInputMetadataProvider(),
             execRoot,
             withinSandboxExecRoot,
             packageRoots,
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/ProcessWrapperSandboxedSpawnRunner.java b/src/main/java/com/google/devtools/build/lib/sandbox/ProcessWrapperSandboxedSpawnRunner.java
index 2f7608dcc8b857..c9b1509ca06107 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/ProcessWrapperSandboxedSpawnRunner.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/ProcessWrapperSandboxedSpawnRunner.java
@@ -100,6 +100,7 @@ protected SandboxedSpawn prepareSpawn(Spawn spawn, SpawnExecutionContext context
     SandboxInputs inputs =
         helpers.processInputFiles(
             context.getInputMapping(PathFragment.EMPTY_FRAGMENT, /* willAccessRepeatedly= */ true),
+            context.getInputMetadataProvider(),
             execRoot,
             execRoot,
             packageRoots,
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java b/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java
index fb72d591e0781e..dfc18bda2c839a 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/SandboxHelpers.java
@@ -20,6 +20,7 @@
 import static com.google.devtools.build.lib.vfs.Dirent.Type.SYMLINK;
 
 import com.google.auto.value.AutoValue;
+import com.google.common.base.Function;
 import com.google.common.base.Preconditions;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
@@ -29,6 +30,8 @@
 import com.google.common.flogger.GoogleLogger;
 import com.google.devtools.build.lib.actions.ActionInput;
 import com.google.devtools.build.lib.actions.Artifact;
+import com.google.devtools.build.lib.actions.FileArtifactValue;
+import com.google.devtools.build.lib.actions.InputMetadataProvider;
 import com.google.devtools.build.lib.actions.Spawn;
 import com.google.devtools.build.lib.actions.UserExecException;
 import com.google.devtools.build.lib.actions.cache.VirtualActionInput;
@@ -444,6 +447,29 @@ private static RootedPath processFilesetSymlink(
             symlink.getPathString()));
   }
 
+  private static RootedPath processResolvedSymlink(
+      Root absoluteRoot,
+      PathFragment execRootRelativeSymlinkTarget,
+      Root execRootWithinSandbox,
+      PathFragment execRootFragment,
+      ImmutableList<Root> packageRoots,
+      Function<Root, Root> sourceRooWithinSandbox) {
+    PathFragment symlinkTarget = execRootFragment.getRelative(execRootRelativeSymlinkTarget);
+    for (Root packageRoot : packageRoots) {
+      if (packageRoot.contains(symlinkTarget)) {
+        return RootedPath.toRootedPath(
+            sourceRooWithinSandbox.apply(packageRoot), packageRoot.relativize(symlinkTarget));
+      }
+    }
+
+    if (symlinkTarget.startsWith(execRootFragment)) {
+      return RootedPath.toRootedPath(
+          execRootWithinSandbox, symlinkTarget.relativeTo(execRootFragment));
+    }
+
+    return RootedPath.toRootedPath(absoluteRoot, symlinkTarget);
+  }
+
   /**
    * Returns the inputs of a Spawn as a map of PathFragments relative to an execRoot to paths in the
    * host filesystem where the input files can be found.
@@ -458,6 +484,7 @@ private static RootedPath processFilesetSymlink(
    */
   public SandboxInputs processInputFiles(
       Map<PathFragment, ActionInput> inputMap,
+      InputMetadataProvider inputMetadataProvider,
       Path execRootPath,
       Path withinSandboxExecRootPath,
       ImmutableList<Root> packageRoots,
@@ -468,12 +495,24 @@ public SandboxInputs processInputFiles(
         withinSandboxExecRootPath.equals(execRootPath)
             ? withinSandboxExecRoot
             : Root.fromPath(execRootPath);
+    Root absoluteRoot = Root.absoluteRoot(execRootPath.getFileSystem());
 
     Map<PathFragment, RootedPath> inputFiles = new TreeMap<>();
     Map<PathFragment, PathFragment> inputSymlinks = new TreeMap<>();
     Map<VirtualActionInput, byte[]> virtualInputs = new HashMap<>();
     Map<Root, Root> sourceRootToSandboxSourceRoot = new TreeMap<>();
 
+    Function<Root, Root> sourceRootWithinSandbox =
+        r -> {
+          if (!sourceRootToSandboxSourceRoot.containsKey(r)) {
+            int next = sourceRootToSandboxSourceRoot.size();
+            sourceRootToSandboxSourceRoot.put(
+                r, Root.fromPath(sandboxSourceRoots.getRelative(Integer.toString(next))));
+          }
+
+          return sourceRootToSandboxSourceRoot.get(r);
+        };
+
     for (Map.Entry<PathFragment, ActionInput> e : inputMap.entrySet()) {
       if (Thread.interrupted()) {
         throw new InterruptedException();
@@ -503,23 +542,63 @@ public SandboxInputs processInputFiles(
 
         if (actionInput instanceof EmptyActionInput) {
           inputPath = null;
+        } else if (actionInput instanceof VirtualActionInput) {
+          inputPath = RootedPath.toRootedPath(withinSandboxExecRoot, actionInput.getExecPath());
         } else if (actionInput instanceof Artifact) {
           Artifact inputArtifact = (Artifact) actionInput;
-          if (inputArtifact.isSourceArtifact() && sandboxSourceRoots != null) {
-            Root sourceRoot = inputArtifact.getRoot().getRoot();
-            if (!sourceRootToSandboxSourceRoot.containsKey(sourceRoot)) {
-              int next = sourceRootToSandboxSourceRoot.size();
-              sourceRootToSandboxSourceRoot.put(
-                  sourceRoot,
-                  Root.fromPath(sandboxSourceRoots.getRelative(Integer.toString(next))));
-            }
-
-            inputPath =
-                RootedPath.toRootedPath(
-                    sourceRootToSandboxSourceRoot.get(sourceRoot),
-                    inputArtifact.getRootRelativePath());
-          } else {
+          if (sandboxSourceRoots == null) {
             inputPath = RootedPath.toRootedPath(withinSandboxExecRoot, inputArtifact.getExecPath());
+          } else {
+            if (inputArtifact.isSourceArtifact()) {
+              Root sourceRoot = inputArtifact.getRoot().getRoot();
+              inputPath =
+                  RootedPath.toRootedPath(
+                      sourceRootWithinSandbox.apply(sourceRoot),
+                      inputArtifact.getRootRelativePath());
+            } else {
+              PathFragment materializationExecPath = null;
+              if (inputArtifact.isChildOfDeclaredDirectory()) {
+                FileArtifactValue parentMetadata =
+                    inputMetadataProvider.getInputMetadata(inputArtifact.getParent());
+                if (parentMetadata.getMaterializationExecPath().isPresent()) {
+                  materializationExecPath =
+                      parentMetadata
+                          .getMaterializationExecPath()
+                          .get()
+                          .getRelative(inputArtifact.getParentRelativePath());
+                }
+              } else if (!inputArtifact.isTreeArtifact()) {
+                // Normally, one would not see tree artifacts here because they have already been
+                // expanded by the time the code gets here. However, there is one very special case:
+                // when an action has an archived tree artifact on its output and is executed on the
+                // local branch of the dynamic execution strategy, the tree artifact is zipped up
+                // in a little extra spawn, which has direct tree artifact on its inputs. Sadly,
+                // it's not easy to fix this because there isn't an easy way to inject this new
+                // tree artifact into the artifact expander being used.
+                //
+                // The best would be to not rely on spawn strategies for executing that little
+                // command: it's entirely under the control of Bazel so we can guarantee that it
+                // does not cause mischief.
+                FileArtifactValue metadata = inputMetadataProvider.getInputMetadata(actionInput);
+                if (metadata.getMaterializationExecPath().isPresent()) {
+                  materializationExecPath = metadata.getMaterializationExecPath().get();
+                }
+              }
+
+              if (materializationExecPath != null) {
+                inputPath =
+                    processResolvedSymlink(
+                        absoluteRoot,
+                        materializationExecPath,
+                        withinSandboxExecRoot,
+                        execRootPath.asFragment(),
+                        packageRoots,
+                        sourceRootWithinSandbox);
+              } else {
+                inputPath =
+                    RootedPath.toRootedPath(withinSandboxExecRoot, inputArtifact.getExecPath());
+              }
+            }
           }
         } else {
           PathFragment execPath = actionInput.getExecPath();
@@ -544,6 +623,7 @@ public SandboxInputs processInputFiles(
     return new SandboxInputs(inputFiles, virtualInputs, inputSymlinks, sandboxRootToSourceRoot);
   }
 
+
   /** The file and directory outputs of a sandboxed spawn. */
   @AutoValue
   public abstract static class SandboxOutputs {
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/WindowsSandboxedSpawnRunner.java b/src/main/java/com/google/devtools/build/lib/sandbox/WindowsSandboxedSpawnRunner.java
index c7996e38582fa1..505e2417850161 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/WindowsSandboxedSpawnRunner.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/WindowsSandboxedSpawnRunner.java
@@ -76,6 +76,7 @@ protected SandboxedSpawn prepareSpawn(Spawn spawn, SpawnExecutionContext context
     SandboxInputs readablePaths =
         helpers.processInputFiles(
             context.getInputMapping(PathFragment.EMPTY_FRAGMENT, /* willAccessRepeatedly= */ true),
+            context.getInputMetadataProvider(),
             execRoot,
             execRoot,
             packageRoots,
diff --git a/src/main/java/com/google/devtools/build/lib/skyframe/ActionOutputMetadataStore.java b/src/main/java/com/google/devtools/build/lib/skyframe/ActionOutputMetadataStore.java
index a1b1a77d3d0cbb..1bc2f4c7c0ac6a 100644
--- a/src/main/java/com/google/devtools/build/lib/skyframe/ActionOutputMetadataStore.java
+++ b/src/main/java/com/google/devtools/build/lib/skyframe/ActionOutputMetadataStore.java
@@ -264,7 +264,15 @@ private TreeArtifactValue constructTreeArtifactValueFromFilesystem(SpecialArtifa
     Path treeDir = artifactPathResolver.toPath(parent);
     boolean chmod = executionMode.get();
 
-    FileStatus stat = treeDir.statIfFound(Symlinks.FOLLOW);
+    FileStatus lstat = treeDir.statIfFound(Symlinks.NOFOLLOW);
+    FileStatus stat;
+    if (lstat == null) {
+      stat = null;
+    } else if (!lstat.isSymbolicLink()) {
+      stat = lstat;
+    } else {
+      stat = treeDir.statIfFound(Symlinks.FOLLOW);
+    }
 
     // Make sure the tree artifact root exists and is a regular directory. Note that this is how the
     // action is initialized, so this should hold unless the action itself has deleted the root.
@@ -332,12 +340,18 @@ private TreeArtifactValue constructTreeArtifactValueFromFilesystem(SpecialArtifa
     }
 
     // Same rationale as for constructFileArtifactValue.
-    if (anyRemote.get() && treeDir.isSymbolicLink() && stat instanceof FileStatusWithMetadata) {
-      FileArtifactValue metadata = ((FileStatusWithMetadata) stat).getMetadata();
-      tree.setMaterializationExecPath(
-          metadata
-              .getMaterializationExecPath()
-              .orElse(treeDir.resolveSymbolicLinks().asFragment().relativeTo(execRoot)));
+    if (lstat.isSymbolicLink()) {
+      PathFragment materializationExecPath = null;
+      if (stat instanceof FileStatusWithMetadata) {
+        materializationExecPath =
+            ((FileStatusWithMetadata) stat).getMetadata().getMaterializationExecPath().orElse(null);
+      }
+
+      if (materializationExecPath == null) {
+        materializationExecPath = treeDir.resolveSymbolicLinks().asFragment().relativeTo(execRoot);
+      }
+
+      tree.setMaterializationExecPath(materializationExecPath);
     }
 
     return tree.build();
@@ -509,7 +523,13 @@ private FileArtifactValue constructFileArtifactValue(
       return value;
     }
 
-    if (type.isFile() && fileDigest != null) {
+    boolean isResolvedSymlink =
+        statAndValue.statNoFollow() != null
+            && statAndValue.statNoFollow().isSymbolicLink()
+            && statAndValue.realPath() != null
+            && !value.isRemote();
+
+    if (type.isFile() && !isResolvedSymlink && fileDigest != null) {
       // The digest is in the file value and that is all that is needed for this file's metadata.
       return value;
     }
@@ -525,22 +545,30 @@ private FileArtifactValue constructFileArtifactValue(
           artifactPathResolver.toPath(artifact).getLastModifiedTime());
     }
 
-    if (injectedDigest == null && type.isFile()) {
+    byte[] actualDigest = fileDigest != null ? fileDigest : injectedDigest;
+
+    if (actualDigest == null && type.isFile()) {
       // We don't have an injected digest and there is no digest in the file value (which attempts a
       // fast digest). Manually compute the digest instead.
-      Path path = statAndValue.pathNoFollow();
-      if (statAndValue.statNoFollow() != null
-          && statAndValue.statNoFollow().isSymbolicLink()
-          && statAndValue.realPath() != null) {
-        // If the file is a symlink, we compute the digest using the target path so that it's
-        // possible to hit the digest cache - we probably already computed the digest for the
-        // target during previous action execution.
-        path = statAndValue.realPath();
-      }
 
-      injectedDigest = DigestUtils.manuallyComputeDigest(path, value.getSize());
+      // If the file is a symlink, we compute the digest using the target path so that it's
+      // possible to hit the digest cache - we probably already computed the digest for the
+      // target during previous action execution.
+      Path pathToDigest = isResolvedSymlink ? statAndValue.realPath() : statAndValue.pathNoFollow();
+      actualDigest = DigestUtils.manuallyComputeDigest(pathToDigest, value.getSize());
     }
-    return FileArtifactValue.createFromInjectedDigest(value, injectedDigest);
+
+    if (!isResolvedSymlink) {
+      return FileArtifactValue.createFromInjectedDigest(value, actualDigest);
+    }
+
+    PathFragment realPathAsFragment = statAndValue.realPath().asFragment();
+    PathFragment execRootRelativeRealPath =
+        realPathAsFragment.startsWith(execRoot)
+            ? realPathAsFragment.relativeTo(execRoot)
+            : realPathAsFragment;
+    return FileArtifactValue.createForResolvedSymlink(
+        execRootRelativeRealPath, value, actualDigest);
   }
 
   /**
diff --git a/src/main/java/com/google/devtools/build/lib/worker/WorkerSpawnRunner.java b/src/main/java/com/google/devtools/build/lib/worker/WorkerSpawnRunner.java
index 6718b7e709a561..c21eb8e6bdc6b2 100644
--- a/src/main/java/com/google/devtools/build/lib/worker/WorkerSpawnRunner.java
+++ b/src/main/java/com/google/devtools/build/lib/worker/WorkerSpawnRunner.java
@@ -188,6 +188,7 @@ public SpawnResult exec(Spawn spawn, SpawnExecutionContext context)
             helpers.processInputFiles(
                 context.getInputMapping(
                     PathFragment.EMPTY_FRAGMENT, /* willAccessRepeatedly= */ true),
+                context.getInputMetadataProvider(),
                 execRoot,
                 execRoot,
                 packageRoots,


Test patch:
diff --git a/src/test/java/com/google/devtools/build/lib/sandbox/BUILD b/src/test/java/com/google/devtools/build/lib/sandbox/BUILD
index 74873ef72d3311..671997db568c43 100644
--- a/src/test/java/com/google/devtools/build/lib/sandbox/BUILD
+++ b/src/test/java/com/google/devtools/build/lib/sandbox/BUILD
@@ -59,12 +59,14 @@ java_test(
     deps = [
         "//src/main/java/com/google/devtools/build/lib/actions",
         "//src/main/java/com/google/devtools/build/lib/actions:artifacts",
+        "//src/main/java/com/google/devtools/build/lib/actions:file_metadata",
         "//src/main/java/com/google/devtools/build/lib/exec:bin_tools",
         "//src/main/java/com/google/devtools/build/lib/exec:tree_deleter",
         "//src/main/java/com/google/devtools/build/lib/sandbox:sandbox_helpers",
         "//src/main/java/com/google/devtools/build/lib/sandbox:sandbox_options",
         "//src/main/java/com/google/devtools/build/lib/sandbox:sandboxed_spawns",
         "//src/main/java/com/google/devtools/build/lib/sandbox:tree_deleter",
+        "//src/main/java/com/google/devtools/build/lib/skyframe:tree_artifact_value",
         "//src/main/java/com/google/devtools/build/lib/vfs",
         "//src/main/java/com/google/devtools/build/lib/vfs:pathfragment",
         "//src/main/java/com/google/devtools/build/lib/vfs/inmemoryfs",
diff --git a/src/test/java/com/google/devtools/build/lib/sandbox/SandboxHelpersTest.java b/src/test/java/com/google/devtools/build/lib/sandbox/SandboxHelpersTest.java
index 8df844739de17e..78fe5c244ee282 100644
--- a/src/test/java/com/google/devtools/build/lib/sandbox/SandboxHelpersTest.java
+++ b/src/test/java/com/google/devtools/build/lib/sandbox/SandboxHelpersTest.java
@@ -24,17 +24,23 @@
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Iterables;
 import com.google.devtools.build.lib.actions.ActionInput;
+import com.google.devtools.build.lib.actions.Artifact;
+import com.google.devtools.build.lib.actions.Artifact.SpecialArtifact;
+import com.google.devtools.build.lib.actions.Artifact.TreeFileArtifact;
 import com.google.devtools.build.lib.actions.ArtifactRoot;
 import com.google.devtools.build.lib.actions.CommandLines.ParamFileActionInput;
+import com.google.devtools.build.lib.actions.FileArtifactValue;
 import com.google.devtools.build.lib.actions.ParameterFile.ParameterFileType;
 import com.google.devtools.build.lib.actions.PathMapper;
 import com.google.devtools.build.lib.actions.Spawn;
 import com.google.devtools.build.lib.actions.cache.VirtualActionInput;
 import com.google.devtools.build.lib.actions.util.ActionsTestUtil;
 import com.google.devtools.build.lib.exec.BinTools;
+import com.google.devtools.build.lib.exec.util.FakeActionInputFileCache;
 import com.google.devtools.build.lib.exec.util.SpawnBuilder;
 import com.google.devtools.build.lib.sandbox.SandboxHelpers.SandboxInputs;
 import com.google.devtools.build.lib.sandbox.SandboxHelpers.SandboxOutputs;
+import com.google.devtools.build.lib.skyframe.TreeArtifactValue;
 import com.google.devtools.build.lib.testutil.Scratch;
 import com.google.devtools.build.lib.testutil.TestUtils;
 import com.google.devtools.build.lib.vfs.DigestHashFunction;
@@ -68,6 +74,7 @@
 /** Tests for {@link SandboxHelpers}. */
 @RunWith(JUnit4.class)
 public class SandboxHelpersTest {
+  private static final byte[] FAKE_DIGEST = new byte[] {1};
 
   private final Scratch scratch = new Scratch();
   private Path execRootPath;
@@ -94,6 +101,79 @@ private RootedPath execRootedPath(String execPath) {
     return RootedPath.toRootedPath(execRoot, PathFragment.create(execPath));
   }
 
+  @Test
+  public void processInputFiles_resolvesMaterializationPath_fileArtifact() throws Exception {
+    ArtifactRoot outputRoot =
+        ArtifactRoot.asDerivedRoot(execRootPath, ArtifactRoot.RootType.Output, "outputs");
+    Path sandboxSourceRoot = scratch.dir("/faketmp/sandbox-source-roots");
+
+    Artifact input = ActionsTestUtil.createArtifact(outputRoot, "a/a");
+    FileArtifactValue symlinkTargetMetadata =
+        FileArtifactValue.createForNormalFile(FAKE_DIGEST, null, 0L);
+    FileArtifactValue inputMetadata =
+        FileArtifactValue.createForResolvedSymlink(
+            PathFragment.create("b/b"), symlinkTargetMetadata, FAKE_DIGEST);
+
+    FakeActionInputFileCache inputMetadataProvider = new FakeActionInputFileCache();
+    inputMetadataProvider.put(input, inputMetadata);
+
+    SandboxHelpers sandboxHelpers = new SandboxHelpers();
+    SandboxInputs inputs =
+        sandboxHelpers.processInputFiles(
+            inputMap(input),
+            inputMetadataProvider,
+            execRootPath,
+            execRootPath,
+            ImmutableList.of(),
+            sandboxSourceRoot);
+
+    assertThat(inputs.getFiles())
+        .containsEntry(
+            input.getExecPath(), RootedPath.toRootedPath(execRoot, PathFragment.create("b/b")));
+  }
+
+  @Test
+  public void processInputFiles_resolvesMaterializationPath_treeArtifact() throws Exception {
+    ArtifactRoot outputRoot =
+        ArtifactRoot.asDerivedRoot(execRootPath, ArtifactRoot.RootType.Output, "outputs");
+    Path sandboxSourceRoot = scratch.dir("/faketmp/sandbox-source-roots");
+    SpecialArtifact parent =
+        ActionsTestUtil.createTreeArtifactWithGeneratingAction(
+            outputRoot, "bin/config/other_dir/subdir");
+
+    TreeFileArtifact childA = TreeFileArtifact.createTreeOutput(parent, "a/a");
+    TreeFileArtifact childB = TreeFileArtifact.createTreeOutput(parent, "b/b");
+    FileArtifactValue childMetadata = FileArtifactValue.createForNormalFile(FAKE_DIGEST, null, 0L);
+    TreeArtifactValue parentMetadata =
+        TreeArtifactValue.newBuilder(parent)
+            .putChild(childA, childMetadata)
+            .putChild(childB, childMetadata)
+            .setMaterializationExecPath(PathFragment.create("materialized"))
+            .build();
+
+    FakeActionInputFileCache inputMetadataProvider = new FakeActionInputFileCache();
+    inputMetadataProvider.put(parent, parentMetadata.getMetadata());
+
+    SandboxHelpers sandboxHelpers = new SandboxHelpers();
+    SandboxInputs inputs =
+        sandboxHelpers.processInputFiles(
+            inputMap(childA, childB),
+            inputMetadataProvider,
+            execRootPath,
+            execRootPath,
+            ImmutableList.of(),
+            sandboxSourceRoot);
+
+    assertThat(inputs.getFiles())
+        .containsEntry(
+            childA.getExecPath(),
+            RootedPath.toRootedPath(execRoot, PathFragment.create("materialized/a/a")));
+    assertThat(inputs.getFiles())
+        .containsEntry(
+            childB.getExecPath(),
+            RootedPath.toRootedPath(execRoot, PathFragment.create("materialized/b/b")));
+  }
+
   @Test
   public void processInputFiles_materializesParamFile() throws Exception {
     SandboxHelpers sandboxHelpers = new SandboxHelpers();
@@ -106,7 +186,12 @@ public void processInputFiles_materializesParamFile() throws Exception {
 
     SandboxInputs inputs =
         sandboxHelpers.processInputFiles(
-            inputMap(paramFile), execRootPath, execRootPath, ImmutableList.of(), null);
+            inputMap(paramFile),
+            new FakeActionInputFileCache(),
+            execRootPath,
+            execRootPath,
+            ImmutableList.of(),
+            null);
 
     assertThat(inputs.getFiles())
         .containsExactly(PathFragment.create("paramFile"), execRootedPath("paramFile"));
@@ -127,7 +212,12 @@ public void processInputFiles_materializesBinToolsFile() throws Exception {
 
     SandboxInputs inputs =
         sandboxHelpers.processInputFiles(
-            inputMap(tool), execRootPath, execRootPath, ImmutableList.of(), null);
+            inputMap(tool),
+            new FakeActionInputFileCache(),
+            execRootPath,
+            execRootPath,
+            ImmutableList.of(),
+            null);
 
     assertThat(inputs.getFiles())
         .containsExactly(PathFragment.create("_bin/say_hello"), execRootedPath("_bin/say_hello"));
@@ -173,7 +263,12 @@ protected void setExecutable(PathFragment path, boolean executable) throws IOExc
               try {
                 var unused =
                     sandboxHelpers.processInputFiles(
-                        inputMap(input), customExecRoot, customExecRoot, ImmutableList.of(), null);
+                        inputMap(input),
+                        new FakeActionInputFileCache(),
+                        customExecRoot,
+                        customExecRoot,
+                        ImmutableList.of(),
+                        null);
                 finishProcessingSemaphore.release();
               } catch (IOException | InterruptedException e) {
                 throw new IllegalArgumentException(e);
@@ -181,7 +276,12 @@ protected void setExecutable(PathFragment path, boolean executable) throws IOExc
             });
     var unused =
         sandboxHelpers.processInputFiles(
-            inputMap(input), customExecRoot, customExecRoot, ImmutableList.of(), null);
+            inputMap(input),
+            new FakeActionInputFileCache(),
+            customExecRoot,
+            customExecRoot,
+            ImmutableList.of(),
+            null);
     finishProcessingSemaphore.release();
     future.get();
 
diff --git a/src/test/java/com/google/devtools/build/lib/skyframe/ActionOutputMetadataStoreTest.java b/src/test/java/com/google/devtools/build/lib/skyframe/ActionOutputMetadataStoreTest.java
index 430e809fb50f31..afcf8281bb36b8 100644
--- a/src/test/java/com/google/devtools/build/lib/skyframe/ActionOutputMetadataStoreTest.java
+++ b/src/test/java/com/google/devtools/build/lib/skyframe/ActionOutputMetadataStoreTest.java
@@ -82,10 +82,6 @@ private enum TreeComposition {
     FULLY_LOCAL,
     FULLY_REMOTE,
     MIXED;
-
-    boolean isPartiallyRemote() {
-      return this == FULLY_REMOTE || this == MIXED;
-    }
   }
 
   private final Map<Path, Integer> chmodCalls = Maps.newConcurrentMap();
@@ -422,11 +418,10 @@ public void fileArtifactMaterializedAsSymlink(
         .getPath(outputArtifact.getPath().getPathString())
         .createSymbolicLink(targetArtifact.getPath().asFragment());
 
-    PathFragment expectedMaterializationExecPath = null;
-    if (location == FileLocation.REMOTE) {
-      expectedMaterializationExecPath =
-          preexistingPath != null ? preexistingPath : targetArtifact.getExecPath();
-    }
+    PathFragment expectedMaterializationExecPath =
+        location == FileLocation.REMOTE && preexistingPath != null
+            ? preexistingPath
+            : targetArtifact.getExecPath();
 
     assertThat(store.getOutputMetadata(outputArtifact))
         .isEqualTo(createFileMetadataForSymlinkTest(location, expectedMaterializationExecPath));
@@ -436,7 +431,12 @@ private FileArtifactValue createFileMetadataForSymlinkTest(
       FileLocation location, @Nullable PathFragment materializationExecPath) {
     switch (location) {
       case LOCAL:
-        return FileArtifactValue.createForNormalFile(new byte[] {1, 2, 3}, /* proxy= */ null, 10);
+        FileArtifactValue target =
+            FileArtifactValue.createForNormalFile(new byte[] {1, 2, 3}, /* proxy= */ null, 10);
+        return materializationExecPath == null
+            ? target
+            : FileArtifactValue.createForResolvedSymlink(
+                materializationExecPath, target, target.getDigest());
       case REMOTE:
         return RemoteFileArtifactValue.create(
             new byte[] {1, 2, 3}, 10, 1, -1, materializationExecPath);
@@ -481,11 +481,8 @@ public void treeArtifactMaterializedAsSymlink(
         .getPath(outputArtifact.getPath().getPathString())
         .createSymbolicLink(targetArtifact.getPath().asFragment());
 
-    PathFragment expectedMaterializationExecPath = null;
-    if (composition.isPartiallyRemote()) {
-      expectedMaterializationExecPath =
-          preexistingPath != null ? preexistingPath : targetArtifact.getExecPath();
-    }
+    PathFragment expectedMaterializationExecPath =
+        preexistingPath != null ? preexistingPath : targetArtifact.getExecPath();
 
     assertThat(store.getTreeArtifactValue(outputArtifact))
         .isEqualTo(
@@ -814,7 +811,7 @@ public void fileArtifactValueForSymlink_readFromCache() throws Exception {
 
     var symlinkMetadata = store.getOutputMetadata(symlink);
 
-    assertThat(symlinkMetadata).isEqualTo(targetMetadata);
+    assertThat(symlinkMetadata.getDigest()).isEqualTo(targetMetadata.getDigest());
     assertThat(DigestUtils.getCacheStats().hitCount()).isEqualTo(1);
   }
 }
diff --git a/src/test/shell/bazel/bazel_sandboxing_test.sh b/src/test/shell/bazel/bazel_sandboxing_test.sh
index 3e65184291777a..86fa7ad1807cd4 100755
--- a/src/test/shell/bazel/bazel_sandboxing_test.sh
+++ b/src/test/shell/bazel/bazel_sandboxing_test.sh
@@ -334,6 +334,95 @@ EOF
   assert_contains GOOD bazel-bin/pkg/gen.txt
 }
 
+function test_symlink_with_output_base_under_tmp() {
+  if [[ "$PLATFORM" == "darwin" ]]; then
+    # Tests Linux-specific functionality
+    return 0
+  fi
+
+  create_workspace_with_default_repos WORKSPACE
+
+  sed -i.bak '/sandbox_tmpfs_path/d' $TEST_TMPDIR/bazelrc
+
+  mkdir -p pkg
+  cat > pkg/BUILD <<'EOF'
+load(":r.bzl", "symlink_rule")
+
+genrule(name="r1", srcs=[], outs=[":r1"], cmd="echo CONTENT > $@")
+symlink_rule(name="r2", input=":r1")
+genrule(name="r3", srcs=[":r2"], outs=[":r3"], cmd="cp $< $@")
+EOF
+
+  cat > pkg/r.bzl <<'EOF'
+def _symlink_impl(ctx):
+  output = ctx.actions.declare_file(ctx.label.name)
+  ctx.actions.symlink(output = output, target_file = ctx.file.input)
+  return [DefaultInfo(files = depset([output]))]
+
+symlink_rule = rule(
+  implementation = _symlink_impl,
+  attrs = {"input": attr.label(allow_single_file=True)})
+EOF
+
+  local tmp_output_base=$(mktemp -d "/tmp/bazel_output_base.XXXXXXXX")
+  trap "chmod -R u+w $tmp_output_base && rm -fr $tmp_output_base" EXIT
+
+  bazel --output_base="$tmp_output_base" build //pkg:r3
+  assert_contains CONTENT bazel-bin/pkg/r3
+  bazel --output_base="$tmp_output_base" shutdown
+}
+
+function test_symlink_to_directory_with_output_base_under_tmp() {
+  if [[ "$PLATFORM" == "darwin" ]]; then
+    # Tests Linux-specific functionality
+    return 0
+  fi
+
+  create_workspace_with_default_repos WORKSPACE
+
+  sed -i.bak '/sandbox_tmpfs_path/d' $TEST_TMPDIR/bazelrc
+
+  mkdir -p pkg
+  cat > pkg/BUILD <<'EOF'
+load(":r.bzl", "symlink_rule", "tree_rule")
+
+tree_rule(name="t1")
+symlink_rule(name="t2", input=":t1")
+genrule(name="t3", srcs=[":t2"], outs=[":t3"], cmd=";\n".join(
+    ["cat $(location :t2)/{a/a,b/b} > $@"]))
+EOF
+
+  cat > pkg/r.bzl <<'EOF'
+def _symlink_impl(ctx):
+  output = ctx.actions.declare_directory(ctx.label.name)
+  ctx.actions.symlink(output = output, target_file = ctx.file.input)
+  return [DefaultInfo(files = depset([output]))]
+
+symlink_rule = rule(
+  implementation = _symlink_impl,
+  attrs = {"input": attr.label(allow_single_file=True)})
+
+def _tree_impl(ctx):
+  output = ctx.actions.declare_directory(ctx.label.name)
+  ctx.actions.run_shell(
+    outputs = [output],
+    command = "export TREE=%s && mkdir $TREE/a $TREE/b && echo -n A > $TREE/a/a && echo -n B > $TREE/b/b" % output.path)
+  return [DefaultInfo(files = depset([output]))]
+
+tree_rule = rule(
+  implementation = _tree_impl,
+  attrs = {})
+
+EOF
+
+  local tmp_output_base=$(mktemp -d "/tmp/bazel_output_base.XXXXXXXX")
+  trap "chmod -R u+w $tmp_output_base && rm -fr $tmp_output_base" EXIT
+
+  bazel --output_base="$tmp_output_base" build //pkg:t3
+  assert_contains AB bazel-bin/pkg/t3
+  bazel --output_base="$tmp_output_base" shutdown
+}
+
 # The test shouldn't fail if the environment doesn't support running it.
 check_sandbox_allowed || exit 0