Repository: bazelbuild/bazel
Issue: //src/main/java/com/google/devtools/build/lib:gen_command-line-reference is broken
Labels: type: documentation (cleanup)
breakage

Bisected to 09924273efc7050a2e342cd89b823a619ca073f8 (cc @anakanemison)

```
$ bazel build //src/main/java/com/google/devtools/build/lib:gen_command-line-reference --verbose_failures
...

INFO: Analyzed target //src/main/java/com/google/devtools/build/lib:gen_command-line-reference (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /Users/jingwen/bazels/github/src/main/java/com/google/devtools/build/lib/BUILD:756:1: Executing genrule //src/main/java/com/google/devtools/build/lib:gen_command-line-reference failed (Exit 37): bash failed: error executing command

...

  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; cat site/command-line-reference-prefix.html > bazel-out/darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/command-line-reference.html && TMP=`mktemp -d /tmp/tmp.XXXXXXXXXX` && bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer --jvm_flag=-Dio.bazel.EnableJni=0 --batch --install_base=${TMP} --output_base=${TMP}/output/ --output_user_root=${TMP} help everything-as-html >> bazel-out/darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/command-line-reference.html 2>/dev/null && cat site/command-line-reference-suffix.html >> bazel-out/darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/command-line-reference.html')
Execution platform: //:default_host_platform

Use --sandbox_debug to see verbose messages from the sandbox bash failed: error executing command
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; cat site/command-line-reference-prefix.html > bazel-out/darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/command-line-reference.html && TMP=`mktemp -d /tmp/tmp.XXXXXXXXXX` && bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer --jvm_flag=-Dio.bazel.EnableJni=0 --batch --install_base=${TMP} --output_base=${TMP}/output/ --output_user_root=${TMP} help everything-as-html >> bazel-out/darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/command-line-reference.html 2>/dev/null && cat site/command-line-reference-suffix.html >> bazel-out/darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/command-line-reference.html')
Execution platform: //:default_host_platform

Use --sandbox_debug to see verbose messages from the sandbox
Target //src/main/java/com/google/devtools/build/lib:gen_command-line-reference failed to build
INFO: Elapsed time: 1.082s, Critical Path: 0.88s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
```

Same error is happening on the Cloud Build job for docs.bazel.build. 

https://pantheon.corp.google.com/cloud-build/builds/90b4bb72-5e19-4379-948e-79267dec30fb?project=bazel-public

Issue comments:
cc @gregestren 
Figured out the problem:

```
$ TMP=`mktemp -d /tmp/tmp.XXXXXXXXXX` && bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer --jvm_flag=-Dio.bazel.EnableJni=0 --batch --install_base=${TMP} --output_base=${TMP}/output/ --output_user_root=${TMP} help everything-as-html
Apr 08, 2020 3:03:24 PM com.google.devtools.build.lib.analysis.BlazeVersionInfo logVersionInfo
WARNING: Bazel release version information not available
Apr 08, 2020 3:03:24 PM com.google.devtools.build.lib.runtime.BlazeRuntime batchMain
INFO: Running Bazel in batch mode with startup args [--batch, --install_base=/tmp/tmp.4SOLYWMRux, --output_base=/tmp/tmp.4SOLYWMRux/output/, --output_user_root=/tmp/tmp.4SOLYWMRux]
Apr 08, 2020 3:03:24 PM com.google.devtools.build.lib.bugreport.BugReport logCrash
SEVERE: Crash: java.lang.IllegalArgumentException: Bad --failure_detail_out option specified: 'null' java.lang.IllegalArgumentException: Bad --failure_detail_out option specified: 'null'
	at com.google.devtools.build.lib.runtime.BlazeRuntime.newRuntime(BlazeRuntime.java:1229)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.batchMain(BlazeRuntime.java:993)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:814)
	at com.google.devtools.build.lib.bazel.Bazel.main(Bazel.java:79)

Apr 08, 2020 3:03:24 PM com.google.devtools.build.lib.bugreport.BugReport sendBugReport
INFO: (Not a released binary; not logged.)
java.lang.IllegalArgumentException: Bad --failure_detail_out option specified: 'null'
	at com.google.devtools.build.lib.runtime.BlazeRuntime.newRuntime(BlazeRuntime.java:1229)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.batchMain(BlazeRuntime.java:993)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:814)
	at com.google.devtools.build.lib.bazel.Bazel.main(Bazel.java:79)
Apr 08, 2020 3:03:24 PM com.google.devtools.build.lib.bugreport.BugReport printThrowableTo
SEVERE: <unknown> crashed
java.lang.IllegalArgumentException: Bad --failure_detail_out option specified: 'null'
	at com.google.devtools.build.lib.runtime.BlazeRuntime.newRuntime(BlazeRuntime.java:1229)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.batchMain(BlazeRuntime.java:993)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:814)
	at com.google.devtools.build.lib.bazel.Bazel.main(Bazel.java:79)

ERROR: <unknown> crash in async thread:
java.lang.IllegalArgumentException: Bad --failure_detail_out option specified: 'null'
	at com.google.devtools.build.lib.runtime.BlazeRuntime.newRuntime(BlazeRuntime.java:1229)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.batchMain(BlazeRuntime.java:993)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:814)
	at com.google.devtools.build.lib.bazel.Bazel.main(Bazel.java:79)
```

We should consider adding the site build target (https://github.com/bazelbuild/bazel/pull/11093), or at least docgen targets, to presubmit.
Why does this call the server jar directly?
It uses the output of `bazel help everything-as-html` to generate the command line reference.
Why doesn't it call the launcher?
In https://bazel-review.googlesource.com/c/bazel/+/3880/2/src/main/java/com/google/devtools/build/lib/BUILD#982, @damienmg asked the same question, and @ulfjack said:

> Tried that, didn't work. By default, there's no JDK in the sandbox, and JAVA_HOME isn't set.

> Also, the packaging step is fairly expensive, compared to 'just' building the Java binary, so it seemed ok to go with this.
Interesting, I filed a breakage on this before too: https://github.com/bazelbuild/bazel/issues/4730

https://github.com/bazelbuild/bazel/issues/4730#issuecomment-369150294 laments that we don't have presubmits for this, which I'm adding in https://github.com/bazelbuild/bazel/pull/11096
I feel like that can be revisited, that was from 2016. Also, it seems like getting it to use the launcher would reduce future brittleness.
I can definitely do that, but for now, the site generation is broken/blocked, so the fix is rather high priority.
Opened https://github.com/bazelbuild/bazel/issues/11102

Pull request review comments:
This is obviously unneeded, but it also makes sense to copy over a tricky preamble like this without changes. So I'm calling it out as OK, in spite of it looking funny.
grep -q  ....
will just set an exit status without doing any output.
Nice, done.

Patch that resolved the issue:
diff --git a/site/BUILD b/site/BUILD
index d9b4b153d6f4ca..ef3d4ba3c00401 100644
--- a/site/BUILD
+++ b/site/BUILD
@@ -155,6 +155,15 @@ jekyll_build(
     srcs = [
         "@jekyll_tree_%s//file" % DOC_VERSION["version"].replace(".", "_")
         for DOC_VERSION in DOC_VERSIONS
-    ] + [":jekyll-tree"],  # jekyll-tree *must* come last to be the default.
+    ] + [":jekyll-tree"], # jekyll-tree *must* come last to use layouts and includes from master
     bucket = "docs.bazel.build",
 )
+
+# A smaller site build target. Contains just the docs for latest released version and master.
+jekyll_build(
+    name = "site_latest",
+    srcs = [
+        "@jekyll_tree_%s//file" % DOC_VERSIONS[0]["version"].replace(".", "_"),
+        ":jekyll-tree", # jekyll-tree *must* come last to use layouts and includes from master
+    ],
+)
diff --git a/src/main/java/com/google/devtools/build/lib/BUILD b/src/main/java/com/google/devtools/build/lib/BUILD
index ff1b09a84296e8..cdbc54ca9c36ab 100644
--- a/src/main/java/com/google/devtools/build/lib/BUILD
+++ b/src/main/java/com/google/devtools/build/lib/BUILD
@@ -711,6 +711,21 @@ java_library(
     ],
 )
 
+########################################################################
+#
+# Documentation generation
+#
+
+filegroup(
+    name = "generated_docs",
+    srcs = [
+        "//src/main/java/com/google/devtools/build/lib:gen_buildencyclopedia",
+        "//src/main/java/com/google/devtools/build/lib:gen_command-line-reference",
+        "//src/main/java/com/google/devtools/build/lib:gen_skylarklibrary",
+    ],
+    visibility = ["//src/test/shell/bazel:__pkg__"]
+)
+
 filegroup(
     name = "docs_embedded_in_sources",
     srcs = glob(["**/*.java"]) + [
@@ -747,10 +762,7 @@ genrule(
         "//src/main/java/com/google/devtools/build/docgen:docgen_bin",
         "//src/main/java/com/google/devtools/build/docgen:docgen_javalib",
     ],
-    visibility = [
-        "//site:__pkg__",
-        "//src/test/shell/bazel:__pkg__",
-    ],
+    visibility = ["//site:__pkg__"],
 )
 
 genrule(
@@ -766,15 +778,14 @@ genrule(
         "$(location //src/main/java/com/google/devtools/build/lib/bazel:BazelServer) " +
         "--jvm_flag=-Dio.bazel.EnableJni=0 --batch " +
         "--install_base=$${TMP} --output_base=$${TMP}/output/ --output_user_root=$${TMP} " +
+        "--failure_detail_out=$${TMP}/output/failure_detail.rawproto " +
         "help everything-as-html >> $@ 2>/dev/null && " +
         "cat $(location //site:command-line-reference-suffix.html) >> $@"
     ),
     tools = [
         "//src/main/java/com/google/devtools/build/lib/bazel:BazelServer",
     ],
-    visibility = [
-        "//site:__pkg__",
-    ],
+    visibility = ["//site:__pkg__"],
 )
 
 genrule(


Test patch:
diff --git a/src/test/shell/bazel/BUILD b/src/test/shell/bazel/BUILD
index c0f0772756f839..3f69d81de57ab9 100644
--- a/src/test/shell/bazel/BUILD
+++ b/src/test/shell/bazel/BUILD
@@ -597,8 +597,9 @@ sh_test(
     name = "bazel_docgen_test",
     size = "large",
     srcs = ["bazel_docgen_test.sh"],
-    data = ["//src/main/java/com/google/devtools/build/lib:gen_buildencyclopedia"],
+    data = ["//src/main/java/com/google/devtools/build/lib:generated_docs"],
     tags = ["no_windows"],
+    deps = ["@bazel_tools//tools/bash/runfiles"],
 )
 
 sh_test(
diff --git a/src/test/shell/bazel/bazel_docgen_test.sh b/src/test/shell/bazel/bazel_docgen_test.sh
index e17a059fba3e0d..11e6f16047e526 100755
--- a/src/test/shell/bazel/bazel_docgen_test.sh
+++ b/src/test/shell/bazel/bazel_docgen_test.sh
@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2015 The Bazel Authors. All rights reserved.
+# Copyright 2020 The Bazel Authors. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,10 +13,30 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-#
-# Test the Bazel documentation generation
-#
 
-# if we get to run this script, the prerequisite is there, and we're happy.
+# A smoke file test for generated doc files.
+
+set -euo pipefail
+
+# --- begin runfiles.bash initialization v2 ---
+# Copy-pasted from the Bazel Bash runfiles library v2.
+set -uo pipefail; f=bazel_tools/tools/bash/runfiles/runfiles.bash
+source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
+    source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
+    source "$0.runfiles/$f" 2>/dev/null || \
+    source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
+    source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
+    { echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
+# --- end runfiles.bash initialization v2 ---
+
+EXPECTED_ZIPS="build-encyclopedia.zip skylark-library.zip"
+
+for out in $EXPECTED_ZIPS; do
+  filepath="$(rlocation io_bazel/src/main/java/com/google/devtools/build/lib/$out)"
+  # Test that the file has a corrected ZIP structure.
+  zipinfo "$filepath" 1>/dev/null
+done
 
-echo PASS
+COMMAND_LINE_REF_FILE="$(rlocation io_bazel/src/main/java/com/google/devtools/build/lib/command-line-reference.html)"
+# Test that the file contains (at least) the title string.
+grep -q "Command-Line Reference" "$COMMAND_LINE_REF_FILE"