Repository: kythe/kythe (language: rust)

Issue:
Kythe is broken with Bazel@HEAD on Bazel CI
https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2975#0187b105-58ad-4860-a44e-58412a0a4c30


Platform :  Kythe (:ubuntu: 20.04 LTS (OpenJDK 11, gcc 9.4.0))


Error logs:
```
kythe/java/com/google/devtools/kythe/extractors/java/standalone/Jdk9CompatibilityShims.java:41: error: incompatible types: String[] cannot be converted to List<String>
--
  | return Lists.newArrayList(CommandLine.parse(args));
  | ^
  | kythe/java/com/google/devtools/kythe/extractors/java/standalone/Jdk9CompatibilityShims.java:46: error: incompatible types: String[] cannot be converted to Iterable<String>
  | arguments.init("kythe_javac", args);
  | ^
  | (02:20:55) ERROR: io.reactivex.rxjava3.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling \| io.grpc.StatusRuntimeException: NOT_FOUND: Action result with digest "89a766ab3cad185c3ae67d97c531e118211a72fa4df68a38e07dd38d03ada8a7/232" not found
  | at io.reactivex.rxjava3.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:372)
  | at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:82)
  | at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onFailure(RxFutures.java:172)
  | at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
  | at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
  | at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
  | at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
  | at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
  | at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:572)
  | at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)
  | at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
  | at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
  | at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
  | at com.google.devtools.build.lib.remote.NetworkTimeInterceptor$NetworkTimeCall$1.onClose(NetworkTimeInterceptor.java:81)
  | at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
  | at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
  | at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
  | at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
  | at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
  | at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
  | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  | at java.base/java.lang.Thread.run(Unknown Source)
  | Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Action result with digest "89a766ab3cad185c3ae67d97c531e118211a72fa4df68a38e07dd38d03ada8a7/232" not found
  | at io.grpc.Status.asRuntimeException(Status.java:535)
  | ... 14 more
  | (02:20:55) INFO: Elapsed time: 58.117s, Critical Path: 7.41s
  | (02:20:55) INFO: 11678 processes: 4021 remote cache hit, 7493 internal, 47 linux-sandbox, 117 local.
  | (02:20:55) ERROR: Build did NOT complete successfully
  | (02:20:55)
  | Fetching bazel-out/k8-fastbuild/bin/kythe/cxx/extractor/commandline_test; 18s
  | Fetching bazel-out/k8-fastbuild/bin/kythe/cxx/common/path_utils_test; 13s
  | bazel build failed with exit code 1
  | 🚨 Error: The command exited with status 1

```
Note : Unable to reproduce this in local machine.

CC  @meteorcloudy 


Discussion / hints:
<del>Probably due to the upgrade of java_tools: https://github.com/bazelbuild/bazel/commit/403564a7ce8c9aff9879595cc6545ad27e4ea16d</del>
/cc @shahms can you take a look?
@meteorcloudy what changed as part of those Java tools? Kythe requires a JDK11-compatible source by default.
We'd love to be able to `select()` on actual Java language version, rather than just Java-version-as-specified but that's not currently possible.
/cc @comius @cushon can you give some pointers here?
@meteorcloudy Is there a way to get more precise date when this broke? Or bisection?

Autosheriff is pointing to a wrong commit. The issue was reported 2 days ago, there's an older date in our logs. There were more java_tools releases after the one mentioned.

Finding the right culprit is taking more time for me than what I would wish.

Culprit finder (https://buildkite.com/bazel/culprit-finder/builds/5407#_) points to https://github.com/bazelbuild/bazel/commit/75e0f7dfea71a7bd40b4bdb601f64217650e8d47, from the log, I don't think it's making mistake. I know this is a very old commit, but it's possible than when we just submitted this change it didn't break Kythe, but later some new commit in Kythe isn't compatible with this change (but is compatible with Bazel 6.x).
> Probably due to the upgrade of java_tools: https://github.com/bazelbuild/bazel/commit/403564a7ce8c9aff9879595cc6545ad27e4ea16d

So probably what I said here is wrong.
The determined culprit is much more likely, but I'm confused as to why. The error indicates a JDK other than JDK11 is being used to compile that file, but you're inheriting our `--java_language_version=11 --tool_java_language_version=11 --java_runtime_version=remotejdk_11 --tool_java_runtime_version=remotejdk_11` flags from .bazelrc and yet the command line indicates an other-than-specified JDK being used:
```
 external/remotejdk17_linux/bin/java '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.resources=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' '-Dsun.io.useCanonCaches=false' -XX:-CompactStrings -Xlog:disable '-Xlog:all=warning:stderr:uptime,level,tags' -jar external/remote_java_tools/java_tools/JavaBuilder_deploy.jar ...
```
This seems very likely to be https://github.com/bazelbuild/rules_java/issues/95

We worked around that locally by not registering the default toolchains, but, uh, as you can see that's not super reliable.
I think `remotejdk17_linux` might not be the root cause. The JDK used to run javac is configured by `java_toolchain.java_runtime`, not directly by `--{,tool_}java_language_version --{,tool_}java_runtime_version`, and it's valid to use a newer javac and cross-compile with javac's `-source -target -bootclasspath` flags to target older language versions

The `Jdk9CompatibilityShims` error suggests that the cross-compilation configuration is not correctly being set up, though

> Unable to reproduce this in local machine

Has there been any progress on a self-contained repro?
My reading of https://docs.oracle.com/en/java/javase/11/tools/javac.html is that cross-compilation doesn't work with Bazel for JDK versions 9 or greater when using JDK-internal classes:

* `-bootclasspath` only works for JDK < 9
* `--release` removes the ability to access JDK-internal classes from non-modular sources
* `--target`/`--source` have no effect on JDK API (from what I can tell)
The more recent docs at https://docs.oracle.com/en/java/javase/20/docs/specs/man/javac.html are the same with different formatting
`--system` provides equivalent functionality to `-bootclasspath` for JDK >= 9

Bazel has all of the pieces to use it for cross-compilation and we're using it internally, but it may not be getting set up correctly here
Ah, there are bugs in the JDK's handling of `--system` that have caused us problems. Specifically, javac doesn't validate that path correctly. Internally, it's b/144849002.

 (I don't know if that's a problem here or not, but it is a problem)
The java_toolchain.java_runtime comment helps a lot, considering: https://github.com/bazelbuild/rules_java/blob/master/toolchains/default_java_toolchain.bzl#L89
b/144849002 is now https://bugs.openjdk.org/browse/JDK-8307168, but I don't think it's related
Yeah, I think the issue is with the use of the JDK-11 default_java_toolchain, which uses `java_runtime=remotejdk17`.
That isn't a problem as long as `--system` is being set correctly?
Presumably it wouldn't be a problem if `--system` were being set correctly but creating and using a JDK11 toolchain which uses a JDK11 java_runtime seems simpler than figuring out what it would take to get a correct `--system`.

Fix patch:
diff --git a/buildenv/java/BUILD b/buildenv/java/BUILD
index 1d409b4c60..173bdf90d1 100644
--- a/buildenv/java/BUILD
+++ b/buildenv/java/BUILD
@@ -3,23 +3,21 @@ load("@rules_java//toolchains:default_java_toolchain.bzl", "default_java_toolcha
 
 package(default_visibility = ["//visibility:public"])
 
-# Bazel does not yet ship with a JDK-19 compatible JDK, so
-# fetch and install our own.
-default_java_toolchain(
-    name = "toolchain_jdk_19",
-    configuration = dict(),
-    java_runtime = "@remotejdk19_linux//:jdk",
-    source_version = "19",
-    target_version = "19",
-)
-
-default_java_toolchain(
-    name = "toolchain_jdk_20",
-    configuration = dict(),
-    java_runtime = "@remotejdk20_linux//:jdk",
-    source_version = "20",
-    target_version = "20",
-)
+# We need to have consistent JDK dependencies, which Bazel often doesn't provide out-of-the-box.
+# In particular, when a target requires a particular JDK version it generally depends on
+# internal class peculiar to that JDK, thus all of `source_version`, `target_version`,
+# and (critically) `java_runtime` must match. Some of the upstream toolchains use a `java_runtime`
+# which differs from the source and target version.
+[
+    default_java_toolchain(
+        name = "toolchain_jdk_{version}".format(version = version),
+        configuration = dict(),
+        java_runtime = "@remotejdk{version}_linux//:jdk".format(version = version),
+        source_version = version,
+        target_version = version,
+    )
+    for version in ("11", "17", "19", "20")
+]
 
 config_setting(
     name = "runtime_version_default",


Test patch: