Repository: bazelbuild/bazel
Issue: linux-sandbox is not available occasionally since Bazel 6.0.0
Labels: type: bug
P2
team-Local-Exec

### Description of the bug:


We noticed that Bazel occasionally (about 5% in our env) fails due to `linux-sandbox` not being available.
```
ERROR: 'linux-sandbox' was requested for default strategies but no strategy with that identifier was registered. Valid values are: [processwrapper-sandbox, standalone, remote, worker, sandboxed, local]
```
It seems that recently a 1s timeout was introduced in checking if `linux-sandbox` available #15414, which might be too tight under load.

In our setup, we disable all other weaker sandboxes for hermeticity, which makes this fail reliably and easy to notice. I suspect this is happening on more environments, but people haven't noticed because of`processwrapper-sandbox` fallback.

CC @meisterT 

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.


run Bazel >=6.0.0 with `--spawn_strategy=worker,linux-sandbox` under a heavy load many times.

### Which operating system are you running Bazel on?


Linux

### What is the output of `bazel info release`?


_No response_

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.


_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?


_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Issue comments:
the 1s timeout was introduced for #15373
I am slightly confused here:

1. Why is the system under load? Bazel only runs this code upon startup (SandboxModule registration), before any action is executed. So the expectation here is the host system should have enough resources dedicated to running Bazel and downstream actions. Are you suffering from the noisy neighbor problem here? (i.e. running multiple Bazel JVM, isolated by containers without dedicated resources, on the same host?).

2. If the system does not have adequate resources to execute `/bin/true` inside a `linux-sandbox`, it's a pretty good sign to fail the build right there immediately instead of waiting and letting downstream actions also suffer the same faith?

3. Do you restart Bazel JVM a lot? Bazel JVM should be long-lived to retain the in-memory analysis cache. Restarting Bazel JVM from cold will make your build slower and increase the chance of this problem happening.

From the point above, I would imagine your current setup is something like this: 1 VM/baremetal host running Bazel in multiple containers. After each build, you are discarding your Bazel container, killing the Bazel JVM inside. You are overfitting the host system to saturate the CPU consumption which resulted in "noisy neighbor": builds from a few busier containers are consuming too many CPU, stopping new containers from being spun up successfully.

If that's indeed the case, my advice would be: to switch to using a set of "persistent containers" and re-use the containers + Bazel JVM inside between runs. Each container should be assigned a fixed set of CPU/RAM, enforced by the container runtime to ensure that they don't use more resources than they should.

I don't think the mitigation in #18151 will solve your issue at all. You are only delaying the noisy neighbor issue from Bazel's startup phase down to the action execution phase. The 5% failure rate would still retain as all of your action execution will be delayed in scheduling by the same wait time.

Pull request review comments:

Patch that resolved the issue:
diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java b/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java
index a50f4ba89f48be..cc0219bc340662 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java
@@ -34,6 +34,7 @@
 import com.google.devtools.build.lib.events.Reporter;
 import com.google.devtools.build.lib.exec.TreeDeleter;
 import com.google.devtools.build.lib.exec.local.LocalEnvProvider;
+import com.google.devtools.build.lib.exec.local.LocalExecutionOptions;
 import com.google.devtools.build.lib.exec.local.PosixLocalEnvProvider;
 import com.google.devtools.build.lib.profiler.Profiler;
 import com.google.devtools.build.lib.profiler.SilentCloseable;
@@ -94,10 +95,11 @@ public static boolean isSupported(final CommandEnvironment cmdEnv) throws Interr
 
   private static boolean computeIsSupported(CommandEnvironment cmdEnv, Path linuxSandbox)
       throws InterruptedException {
+    LocalExecutionOptions options = cmdEnv.getOptions().getOptions(LocalExecutionOptions.class);
     ImmutableList<String> linuxSandboxArgv =
         LinuxSandboxCommandLineBuilder.commandLineBuilder(
                 linuxSandbox, ImmutableList.of("/bin/true"))
-            .setTimeout(Duration.ofSeconds(1))
+            .setTimeout(options.getLocalSigkillGraceSeconds())
             .build();
     ImmutableMap<String, String> env = ImmutableMap.of();
     Path execRoot = cmdEnv.getExecRoot();


Test patch: