Repository: bazelbuild/bazel
Issue: High rate of spurious CI failures on macOS machines
Labels: type: bug
more data needed
platform: apple

<!--
ATTENTION! Please read and follow:
- if this is a _question_ about how to build / test / query / deploy using Bazel, or a _discussion starter_, send it to bazel-discuss@googlegroups.com
- if this is a _bug_ or _feature request_, fill the form below as best as you can.
-->

### Description of the problem / feature request:

I run daily CI checks in my rulesets' GitHub Actions pipeline. The macOS pipelines, running on `macos-latest`, fail every few days with two kinds of spurious failures that I have never been able to reproduce locally:

Issue 1:
```
Starting local Bazel server and connecting to it...
... still trying to connect to local Bazel server after 10 seconds ...
... still trying to connect to local Bazel server after 20 seconds ...
... still trying to connect to local Bazel server after 30 seconds ...
... still trying to connect to local Bazel server after 40 seconds ...
... still trying to connect to local Bazel server after 50 seconds ...
... still trying to connect to local Bazel server after 60 seconds ...
... still trying to connect to local Bazel server after 70 seconds ...
... still trying to connect to local Bazel server after 80 seconds ...
... still trying to connect to local Bazel server after 90 seconds ...
... still trying to connect to local Bazel server after 100 seconds ...
... still trying to connect to local Bazel server after 110 seconds ...
FATAL: couldn't connect to server (1753) after 120 seconds.
Error: Process completed with exit code 37.
```

Issue 2:
```
ERROR: /Users/runner/work/rules_jni/rules_jni/tests/libjvm_stub/BUILD.bazel:116:12: Target '//libjvm_stub:HelloFromJava' depends on toolchain '@local_config_cc//:cc-compiler-darwin', which cannot be found: error loading package '@local_config_cc//': cannot load '@local_config_cc_toolchains//:osx_archs.bzl': no such file'
ERROR: Analysis of target '//libjvm_stub:HelloFromJava' failed; build aborted: Analysis failed
```

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have no way to consistently reproduce this issue, but it happens every few days on [rules_jni's CI schedule](https://github.com/fmeum/rules_jni/blob/main/.github/workflows/run-tests-externally.yml).

### What operating system are you running Bazel on?

macOS 10.15
### What's the output of `bazel info release`?

Over time, I have hit the issues on 4.2.2, 5.0.0rc3 and various last_green builds.

###  Have you found anything relevant by searching the web?

* https://github.com/bazelbuild/rules_go/issues/2221
* https://github.com/grailbio/bazel-toolchain/pull/75#issuecomment-775818020

### Any other information, logs, or outputs that you want to share?

I can make arbitrary changes to the CI config if that helps to gather more information on the cause of these issues.

Issue comments:
I believe it was waiting for xcode-locator, that can be super long if the machine has many Xcodes. 

Try this hack: https://www.smileykeith.com/2021/03/08/locking-xcode-in-bazel/
Thanks, I will try that. Do you think that this could also be related to Issue 2?
It could be, since that file is only created if Xcode is successfully detected.

https://github.com/bazelbuild/bazel/blob/b4b0c321910bc968736ef48e8140528ea7d323cd/tools/cpp/cc_configure.bzl#L73
@fmeum did @thii's suggestion fix the issue?
I just checked again and found no new failures in the two weeks since I started using a checked-in `xcode_version_config`. I will close this now as the workaround seems to have worked and open a new issue if it should show up again.
@sventiffe Reopening since the exact same issue appeared again today over at [rules_jni](https://github.com/fmeum/rules_jni).

I started pinning with `--xcode_version_config` in https://github.com/fmeum/rules_jni/commit/b5021809cd5ea4393291d0d592b2f195c6f4e1f5. Today, [the runs](https://github.com/fmeum/rules_jni/runs/4823744403) on macos-latest with Bazel 4.0.0 and Bazel 5.0.0rc3 failed. I attached the [CI logs](https://github.com/bazelbuild/bazel/files/7874710/logs.zip), the failing step is [here](https://github.com/fmeum/rules_jni/blob/ff4e93e428869246f4779becd56b027bb13b3003/.github/workflows/run-tests-externally.yml#L75) with the message (in the case of Bazel 5.0.0rc3 with bzlmod):

```
2022/01/15 00:35:14 Downloading https://releases.bazel.build/4.0.0/release/bazel-4.0.0-darwin-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 0e7ba86d-1a19-4b3a-a770-f0c3d58d166f
Loading: 
Loading: 0 packages loaded
Loading: 11 packages loaded
    currently loading: analysis/cc_jni_library ... (2 packages)
Loading: 11 packages loaded
    currently loading: analysis/cc_jni_library ... (2 packages)
Analyzing: 21 targets (13 packages loaded, 0 targets configured)
Analyzing: 21 targets (15 packages loaded, 1 target configured)
Analyzing: 21 targets (18 packages loaded, 8 targets configured)
Analyzing: 21 targets (18 packages loaded, 8 targets configured)
Analyzing: 21 targets (19 packages loaded, 8 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (21 packages loaded, 9 targets configured)
Analyzing: 21 targets (38 packages loaded, 206 targets configured)
Analyzing: 21 targets (38 packages loaded, 206 targets configured)
Analyzing: 21 targets (38 packages loaded, 206 targets configured)
ERROR: /Users/runner/work/rules_jni/rules_jni/tests/native_loader/src/test/java/com/example/math/BUILD.bazel:3:17: Target '//native_loader/src/test/java/com/example/math:math_remove_this_part_' depends on toolchain '@local_config_cc//:cc-compiler-darwin', which cannot be found: error loading package '@local_config_cc//': cannot load '@local_config_cc_toolchains//:osx_archs.bzl': no such file'
ERROR: Analysis of target '//native_loader/src/test/java/com/example/math:math' failed; build aborted: Analysis failed
INFO: Elapsed time: 216.940s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (39 packages loaded, 206 targets configured)
ERROR: Couldn't start the build. Unable to run tests
FAILED: Build did NOT complete successfully (39 packages loaded, 206 targets configured)
Error: Process completed with exit code 1.
```

@thii Are there any additional diagnostics that I could enable that would help diagnose the underlying issue?
@keith I just encountered another failure and attached the [the logs](https://github.com/bazelbuild/bazel/files/8044694/rules_jni.log). They include the content of the BUILD file, which turns out not to be empty.

That file actually looks pretty good to me, since yours didn't fail with osx_archs import, it's because that file doesn't contain `darwin` https://github.com/bazelbuild/bazel/blob/master/tools/osx/crosstool/osx_archs.bzl

I'm not sure where that's coming from but it should be darwin_x86_64 probably instead
> That file actually looks pretty good to me, since yours didn't fail with osx_archs import, it's because that file doesn't contain `darwin` https://github.com/bazelbuild/bazel/blob/master/tools/osx/crosstool/osx_archs.bzl
> 
> I'm not sure where that's coming from but it should be darwin_x86_64 probably instead

Nice catch. I think I found the root cause and submitted a fix as https://github.com/bazelbuild/bazel/pull/14796.
Nice, I'll keep an eye on that. My assumption for why this is similar to the flaky CI case is because the toolchain only falls back to this codepath on macOS when Xcode cannot be found, which could happen in the case of a timeout running the discovery logic for that. This would likely workaround that issue if folks needed full Xcode in their cases https://www.smileykeith.com/2021/03/08/locking-xcode-in-bazel/

Pull request review comments:

Patch that resolved the issue:
diff --git a/tools/osx/crosstool/BUILD.tpl b/tools/osx/crosstool/BUILD.tpl
index e80dffbd1db909..15230207a0f357 100644
--- a/tools/osx/crosstool/BUILD.tpl
+++ b/tools/osx/crosstool/BUILD.tpl
@@ -80,6 +80,14 @@ cc_toolchain_suite(
     for arch in OSX_TOOLS_ARCHS
 ]
 
+# When xcode_locator fails and causes cc_autoconf_toolchains to fall back
+# to the non-Xcode C++ toolchain, it uses the legacy cpu value to refer to
+# the toolchain, which is "darwin" for x86_64 macOS.
+alias(
+    name = "cc-compiler-darwin",
+    actual = ":cc-compiler-darwin_x86_64",
+)
+
 [
     cc_toolchain_config(
         name = arch,


Test patch: