Repository: bazelbuild/bazel
Issue: Bazel @HEAD frequently kills the server process
Labels: type: bug
P1
untriaged
team-Core

### Description of the bug:


When running Bazel in a loop, it occasionally kills the server for no good reason, on about every second invocation, with the error message `Killed non-responsive server process (pid=<number>)`


### Which category does this issue belong to?


_No response_

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.


```
touch WORKSPACE
cat > BUILD <<'EOF'
genrule(
    name = "gg",
    outs = ["go"],
    cmd = "sleep 1; echo O > $@",
)
EOF
for i in $(seq 1 20); do bazel build //:gg; done
```

Looping 20 times is usually enough to reproduce the issue frequently.

### Which operating system are you running Bazel on?


Linux

### What is the output of `bazel info release`?


development version

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.


From HEAD, commithash 08407743e903627d7e64c669f27b3f2b81316e33.

This issue doesn't seem to happen with 7.0.0rc4.

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?


```text
See the commithash above.
```


### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.


Somewhere between 7.0.0rc4 and 08407743e903627d7e64c669f27b3f2b81316e33; I haven't done the bisection.

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Issue comments:
The issue seems not to be present in 77eacce3f56ce6b5b466337c123a5cdfd671f2ee .
Bisection says that the culprit is unfortunately 51bddee96bc357d8131efba606420227c7a8ebc5 , an Abseil version bump. I'm pretty certain: after this change, I got 2/20 restarts, before, 0/50 .
Interesting,

> This issue doesn't seem to happen with 7.0.0rc4.

The abseil-cpp upgrade is also backported to 7.0.0rc4 in https://github.com/bazelbuild/bazel/pull/20221, if that's the root cause, 7.0.0rc4 should have the same issue?
well that's alarming. I checked again and the confirmed the result of the bisection: 51bddee contains the bug, 39084edb2718865b6648274240ea94a29e62cdf7, its parent, does not and 7.0.0rc4 doesn't either (I did 50 runs each, 51bddee fails very frequently so I'm pretty sure 39084edb is not a false negative)

Maybe we started using some Abseil functionality slightly differently between HEAD and 7.0.0 that happens to break with the upgrade?
@lberki How did you build the binary at 51bddee96bc357d8131efba606420227c7a8ebc5?

I actually couldn't reproduce this issue with Bazelisk with `USE_BAZEL_VERSION=51bddee96bc357d8131efba606420227c7a8ebc5`
Wat. `git checkout 51bddee; bazel build //src:bazel`, then use the resulting Bazel binary.  I put the two binaries I have at `/google/data/ro/users/lb/lberki/bazel.restart/*` (commithashes in their names).

The issue readily reproduces on my workstation (asciinema recording on request :) )

Also, automatic bisection readily zeroed in on the right commit, which is a good indication that the issue is reproducible at will.
The difference is that Bazelisk uses `-c opt` builds. Without `-c opt`, the issue reproduces for me at 51bddee, with it, it doesn't. More alarmingly, the issue also reproduces when I build Bazel from `release-7.0.0pre4` (8c4b1c908e5b520126a2480b0855c0c1e58d1ae0) without `-c opt`.
The good news is that this is reproducible also with `bazel version`. The bad news is that it becomes much less frequent with `GRPC_TRACE=all GRPC_VERBOSITY=INFO` and not reproducible with `GRPC_TRACE=all GRPC_VERBOSITY=DEBUG`.

I teased these lines out from the gRPC logs:

```
I1121 14:04:48.250881833 4115421 retry_filter.cc:896]        chand=0x564e161284f0 calld=0x564e16111070 attempt=0x564e160f0cc0: adding batch (start replayable pending batch on call attempt):  SEND_INITIAL_METADATA{:path: /command_server.CommandServer/Ping, grpc-timeout: 30S} SEND_MESSAGE:flags=0x00000000:len=33 SEND_TRAILING_METADATA{} RECV_INITIAL_METADATA RECV_MESSAGE RECV_TRAILING_METADATA
I1121 14:04:48.250898079 4115421 retry_filter.cc:1113]       chand=0x564e161284f0 calld=0x564e16111070 attempt=0x564e160f0cc0: starting 1 retriable batches on lb_call=0x564e16111900
I1121 14:04:48.250907137 4115421 call_combiner.h:172]        CallCombinerClosureList executing closure while already holding call_combiner 0x564e1610ceb0: closure=0x564e16111840 error=OK reason=start replayable pending batch on call attempt
I1121 14:04:48.250942134 4115421 handshaker.cc:93]           handshake_manager 0x564e16119eb0: error=UNKNOWN:Invalid argument {file:"external/grpc~1.48.1.bcr.1/src/core/lib/iomgr/tcp_client_posix.cc", file_line:352, created_time:"2023-11-21T14:04:48.250314978+00:00", errno:22, os_error:"Invalid argument", syscall:"connect", target_address:"ipv6:%5B::1%5D:35397"} shutdown=0 index=1, args={endpoint=(nil), args=(nil) {size=0: {}}, read_buffer=(nil) (length=0), exit_early=0}
I1121 14:04:48.250972032 4115421 handshaker.cc:126]          handshake_manager 0x564e16119eb0: handshaking complete -- scheduling on_handshake_done with error=UNKNOWN:Invalid argument {file:"external/grpc~1.48.1.bcr.1/src/core/lib/iomgr/tcp_client_posix.cc", file_line:352, created_time:"2023-11-21T14:04:48.250314978+00:00", errno:22, os_error:"Invalid argument", syscall:"connect", target_address:"ipv6:%5B::1%5D:35397"}
```

Next step is `strace`, I guess, to see what exact system call fails and how it's different from the case when it doesn't fail?

@bazel-io fork 7.0.0
A quick look with strace on the client says:

```
connect(8, {sa_family=AF_INET6, sin6_port=htons(42657), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_scope_id=0}, 28) = -1 EINPROGRESS (Operation now in progress)
uname({sysname="Linux", nodename="<REDACTED>", ...}) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f48234c3000
prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, 0x7f48234c3000, 65536, "absl") = -1 EINVAL (Invalid argument)
epoll_ctl(6, EPOLL_CTL_ADD, 8, {events=EPOLLIN|EPOLLOUT|EPOLLET, data={u32=734601009, u64=93996093873969}}) = 0
```

And I have to stop debugging for today; it looks like the reported `EINVAL` comes from a `prctl()` call, but what that system call is, I know not. There are a number of other similar `prctl()` calls, so it could very well be a red herring.
After pondering this a bit, I propose that we treat this as a release blocker until we find out what's going on. Given that  this bug manifests itself without `-c opt`, but not with it, I find it difficult to come up with a theory that makes it benign.



wdyt @meteorcloudy @meisterT ?
I can try to check if newer abseil-cpp version fixes it.
`20230125.1` does not have the issue but I don't know how to verify whether it still includes the fix for https://github.com/bazelbuild/bazel/issues/20076
@meisterT Thanks, I can confirm `20230125.1` also fixes https://github.com/bazelbuild/bazel/issues/20076. I can send a change to update abseil to this version.
This is a gRPC bug: https://github.com/grpc/grpc/pull/35064
@benjaminp impressive sleuthing!

My guess is that you're right and this is in fact a gRPC bug and the reason why the Abseil update tickled it is that https://github.com/abseil/abseil-cpp/commit/3a46229c3cdd945ccc8bae1458148115ff8f88e7 added a `prctl()` call in some low-level allocation code that gets called somehow in between (dunno, e.g. by `grpc_sockaddr_to_uri()`). That commit was pushed in 2023 April, which is conveniently within the 2022 Jun 23 - 2023 Aug 2 time span covered by the Abseil update (it's a long time, though, so it's not a smoking gun). It also explains why using Abseil 20230125.1 fixes this issue.

The only part that's missing is explaining why this bug does not reproduce with `-c opt`. I haven't found a suspicious `#ifdef` but that doesn't mean that there isn't any and given that it's low level memory allocation code, it's not surprising that it gets called at different times depending on how much the compiler optimizes things.


A fix for this issue has been included in [Bazel 7.0.0 RC5](https://releases.bazel.build/7.0.0/rc5/index.html). Please test out the release candidate and report any issues as soon as possible. Thanks!

Pull request review comments:

Patch that resolved the issue:
diff --git a/MODULE.bazel b/MODULE.bazel
index a4fbb21e997f84..03f1dce5d811cf 100644
--- a/MODULE.bazel
+++ b/MODULE.bazel
@@ -53,7 +53,7 @@ local_path_override(
 # The following Bazel modules are not direct dependencies for building Bazel,
 # but are required for visibility from DIST_ARCHIVE_REPOS in repositories.bzl
 bazel_dep(name = "apple_support", version = "1.5.0")
-bazel_dep(name = "abseil-cpp", version = "20230802.0.bcr.1")
+bazel_dep(name = "abseil-cpp", version = "20230125.1")
 bazel_dep(name = "c-ares", version = "1.15.0")
 bazel_dep(name = "rules_go", version = "0.39.1")
 bazel_dep(name = "upb", version = "0.0.0-20220923-a547704")
diff --git a/MODULE.bazel.lock b/MODULE.bazel.lock
index 7e44db5d68a997..3646c7e15bfa9c 100644
--- a/MODULE.bazel.lock
+++ b/MODULE.bazel.lock
@@ -1,6 +1,6 @@
 {
   "lockFileVersion": 3,
-  "moduleFileHash": "fe0ab69d5b93d0aa2a00d76cfc5058fbc4ac5e63bd3b5e47cd7b5551bd33fcdb",
+  "moduleFileHash": "63625ac7809ba5bc83e0814e16f223ac28a98df884897ddd5bfbd69fd4e3ddbf",
   "flags": {
     "cmdRegistries": [
       "https://bcr.bazel.build/"
@@ -596,7 +596,7 @@
         "remoteapis": "remoteapis@_",
         "googleapis": "googleapis@_",
         "apple_support": "apple_support@1.5.0",
-        "abseil-cpp": "abseil-cpp@20230802.0.bcr.1",
+        "abseil-cpp": "abseil-cpp@20230125.1",
         "c-ares": "c-ares@1.15.0",
         "rules_go": "rules_go@0.39.1",
         "upb": "upb@0.0.0-20220923-a547704",
@@ -719,7 +719,7 @@
         "rules_proto": "rules_proto@5.3.0-21.7",
         "rules_java": "rules_java@7.1.0",
         "rules_pkg": "rules_pkg@0.9.1",
-        "com_google_abseil": "abseil-cpp@20230802.0.bcr.1",
+        "com_google_abseil": "abseil-cpp@20230125.1",
         "zlib": "zlib@1.3",
         "upb": "upb@0.0.0-20220923-a547704",
         "rules_jvm_external": "rules_jvm_external@5.2",
@@ -797,7 +797,7 @@
         "bazel_skylib": "bazel_skylib@1.4.1",
         "boringssl": "boringssl@0.0.0-20211025-d4f1ab9",
         "com_github_cares_cares": "c-ares@1.15.0",
-        "com_google_absl": "abseil-cpp@20230802.0.bcr.1",
+        "com_google_absl": "abseil-cpp@20230125.1",
         "com_google_protobuf": "protobuf@21.7",
         "com_googlesource_code_re2": "re2@2021-09-01",
         "rules_proto": "rules_proto@5.3.0-21.7",
@@ -1435,7 +1435,7 @@
       "toolchainsToRegister": [],
       "extensionUsages": [],
       "deps": {
-        "com_google_absl": "abseil-cpp@20230802.0.bcr.1",
+        "com_google_absl": "abseil-cpp@20230125.1",
         "platforms": "platforms@0.0.8",
         "rules_cc": "rules_cc@0.0.9",
         "bazel_tools": "bazel_tools@_",
@@ -1544,10 +1544,10 @@
         }
       }
     },
-    "abseil-cpp@20230802.0.bcr.1": {
+    "abseil-cpp@20230125.1": {
       "name": "abseil-cpp",
-      "version": "20230802.0.bcr.1",
-      "key": "abseil-cpp@20230802.0.bcr.1",
+      "version": "20230125.1",
+      "key": "abseil-cpp@20230125.1",
       "repoName": "abseil-cpp",
       "executionPlatformsToRegister": [],
       "toolchainsToRegister": [],
@@ -1556,7 +1556,6 @@
         "rules_cc": "rules_cc@0.0.9",
         "platforms": "platforms@0.0.8",
         "bazel_skylib": "bazel_skylib@1.4.1",
-        "com_google_googletest": "googletest@1.14.0",
         "bazel_tools": "bazel_tools@_",
         "local_config_platform": "local_config_platform@_"
       },
@@ -1564,14 +1563,14 @@
         "bzlFile": "@bazel_tools//tools/build_defs/repo:http.bzl",
         "ruleClassName": "http_archive",
         "attributes": {
-          "name": "abseil-cpp~20230802.0.bcr.1",
+          "name": "abseil-cpp~20230125.1",
           "urls": [
-            "https://github.com/abseil/abseil-cpp/archive/refs/tags/20230802.0.tar.gz"
+            "https://github.com/abseil/abseil-cpp/archive/refs/tags/20230125.1.tar.gz"
           ],
-          "integrity": "sha256-WdKXavnW7PABqBo1dJpuVRozW5SdNJGM+t4Hc3udk8U=",
-          "strip_prefix": "abseil-cpp-20230802.0",
+          "integrity": "sha256-gTEcF1mbNxIGne0gzKCaYqsL8qid+haZN4bIeCt+0UU=",
+          "strip_prefix": "abseil-cpp-20230125.1",
           "remote_patches": {
-            "https://bcr.bazel.build/modules/abseil-cpp/20230802.0.bcr.1/patches/module_dot_bazel.patch": "sha256-Ku6wJ7uNqANq3fVmW03ySSUddevratZDsJ67NqtXIEY="
+            "https://bcr.bazel.build/modules/abseil-cpp/20230125.1/patches/module_dot_bazel.patch": "sha256-L1wChhBmDOnRbPbD4MENVXHjOBT2KFrDxT6D+aoThxk="
           },
           "remote_patch_strip": 0
         }
@@ -1756,7 +1755,7 @@
         "bazel_skylib": "bazel_skylib@1.4.1",
         "rules_proto": "rules_proto@5.3.0-21.7",
         "com_google_protobuf": "protobuf@21.7",
-        "com_google_absl": "abseil-cpp@20230802.0.bcr.1",
+        "com_google_absl": "abseil-cpp@20230125.1",
         "platforms": "platforms@0.0.8",
         "bazel_tools": "bazel_tools@_",
         "local_config_platform": "local_config_platform@_"
@@ -2181,7 +2180,7 @@
         "bzlTransitiveDigest": "PjK+f/kxkhda9tRFlKVdGfNszPoXs7CDXZUi+ZGWGYU=",
         "accumulatedFileDigests": {
           "@@//src/test/tools/bzlmod:MODULE.bazel.lock": "10b96bd3c1eb194b0efe3a13fd06f2051abf36efb33414ad92048883ba471c7f",
-          "@@//:MODULE.bazel": "fe0ab69d5b93d0aa2a00d76cfc5058fbc4ac5e63bd3b5e47cd7b5551bd33fcdb"
+          "@@//:MODULE.bazel": "63625ac7809ba5bc83e0814e16f223ac28a98df884897ddd5bfbd69fd4e3ddbf"
         },
         "envVariables": {},
         "generatedRepoSpecs": {
@@ -2261,7 +2260,7 @@
             "attributes": {
               "name": "_main~bazel_build_deps~bootstrap_repo_cache",
               "repos": [
-                "abseil-cpp~20230802.0.bcr.1",
+                "abseil-cpp~20230125.1",
                 "apple_support~1.5.0",
                 "bazel_skylib~1.4.1",
                 "blake3~1.3.3.bcr.1",


Test patch: