Repository: bazelbuild/bazel
Issue: Update Protobuf to 3.6.1
Labels: P1
type: process

### Description of the problem / feature request:

Please update Bazel's Protocol Buffer version to 3.6.1

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

This is breaking Bazel at HEAD when built with Visual C++ 2017 on Windows.

See https://github.com/bazelbuild/bazel/issues/6053#issuecomment-418031872.

### What operating system are you running Bazel on?

Windows 10

/cc @meteorcloudy @jin 

Issue comments:
Why am I assigned to this @laszlocsomor?
IIRC you updated Protobuf in the past. If so, you have domain expertise. Am I confusing it with gRPC though?
I have, so have other people. I currently have multiple P1s I am working on and don't have time to update protobuf anytime soon. There's a detailed description on how to update protobuf here: https://github.com/bazelbuild/bazel/tree/master/third_party/protobuf
Thanks for the info. Let me take a look.
If there's a cherrypick request, please add it to the release thread (https://github.com/bazelbuild/bazel/issues/5059).

Pull request review comments:

Patch that resolved the issue:
diff --git a/third_party/protobuf/README.md b/third_party/protobuf/README.md
index e2a428a2afa9dd..a678ab5b02426f 100644
--- a/third_party/protobuf/README.md
+++ b/third_party/protobuf/README.md
@@ -1,40 +1,100 @@
 # Updating protobuf
 
+You will create and merge the following Pull Requests.
 
-1) Fetch the desired protobuf version and copy it in a folder `new_proto` under
-`third_party/protobuf`.
-2) Bazel uses upstream protobuf from source, except for java, as we currently don't
-build protobuf java when bootstrapping bazel. So instead we use pre-built jars.
-So build the java proto library from source and in case you cloned an upstream version
-of protobuf, remove the .git folders:
-```
-cd new_proto
-bazel build :protobuf_java :protobuf_java_util
-cp bazel-bin/libprotobuf_java.jar .
-cp bazel-bin/libprotobuf_java_util.jar .
-bazel clean --expunge
-rm -rf .git .gitignore .gitmodules
-```
-3) Modify protobuf's `BUILD` file to not build java from source, but to use
-   the jars instead. To do that, in the BUILD file delete the rules listed
-   under `Java support`. Then, from the `third_party/protobuf/<old_proto>/BUILD file`
-   copy the rules under "Modifications made by bazel" to the new BUILD file.
-   The java rules in there should have the same names as the ones you just deleted under "Java support".
-   You might need to update the names of the jars in the rules sources to the ones you just build.
-4) Copy `third_party/protobuf/<old_proto>/com_google_protobuf_java.BUILD` to the new
-   directory.
-5) Copy the `licenses` declaration and the `srcs` filegroup from
-   `third_party/protobuf/<old_proto>/util/python/BUILD` and
-   `third_party/protobuf/<old_proto>/examples/BUILD` to the corresponding
-   file in the new directory.
-6) Name the `new\_proto` directory according to the protobuf version number.
-7) In `third\_party/protobuf/BUILD` update `PROTOBUF\_VERSION` to the name of the
-directory you just created.
-8) In the root `WORKSPACE` file update relative paths of protobuf to point to
-the new version.
-9) Update version number in `src/main/protobuf/BUILD` and `src/test/shell/testenv.sh`.
-10) Delete the `third_party/protobuf/<old_proto>` directory.
-11) Update this file if you found the :instructions to be wrong or incomplete.
+## 1st PR: add the new protobuf version to the Bazel tree
+
+1.  Fetch the desired protobuf version and copy it in a folder `new_proto` under
+   `third_party/protobuf`.
+
+    **Example:** to upgrade to 3.6.1, download and unpack
+    [protobuf-all-3.6.1.zip](https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protobuf-all-3.6.1.zip).
+
+1.  Build the Java proto library from source and, in case you cloned an upstream version
+    of protobuf, remove the .git folders:
+
+    ```
+    cd third_party/protobuf/new_proto
+    bazel build :protobuf_java :protobuf_java_util
+    cp bazel-bin/libprotobuf_java.jar .
+    cp bazel-bin/libprotobuf_java_util.jar .
+    bazel clean --expunge
+    rm -rf .git .gitignore .gitmodules
+    ```
+
+    **Reason:** Bazel uses upstream protobuf from source, except for Java, as Bazel's bootstrapping
+    scripts currently don't build protobuf Java when bootstrapping Bazel but use pre-built jars
+    instead.
+
+1.  Modify protobuf's `BUILD` file to not build java from source, but to use the jars instead:
+
+    1.  In the BUILD file delete the rules listed under `Java support`.
+
+    1.  From the `third_party/protobuf/<old_proto>/BUILD` file copy the rules under the
+        "Modifications made by bazel" section to the new BUILD file. The java rules in there should
+        have the same names as the ones you just deleted under "Java support". You might need to
+        update the names of the jars in the rules sources to the ones you just build.
+
+1.  Copy `third_party/protobuf/<old_proto>/com_google_protobuf_java.BUILD` to the new directory.
+
+1.  From `third_party/protobuf/<old_proto>/util/python/BUILD`,
+    `third_party/protobuf/<old_proto>/examples/BUILD`, and
+    `third_party/protobuf/<old_proto>/third_party/googletest/BUILD.bazel`:
+    copy the `licenses` declaration and the `srcs` filegroup to the corresponding file under
+    `third_party/protobuf/<new_proto>`.
+
+1.  In `third_party/protobuf/<new_proto>/BUILD`, in the `srcs` filegroup rule, update the version
+    number referring to the newly added `srcs` rules.
+
+1.  Rename `third_party/protobuf/<new_proto>` directory according to the protobuf version number.
+
+1.  In `third_party/protobuf/BUILD`:
+
+    1.  Add a new variable `_NEW_PROTOBUF_VERSION`, set to value of the version.
+
+    1.  In the `srcs` filegroup rule, add:
+
+        ```diff
+         srcs = [
+             "//third_party/protobuf/" + PROTOBUF_VERSION + ":srcs",
+        +    "//third_party/protobuf/" + _NEW_PROTOBUF_VERSION + ":srcs",
+         ],
+        ```
+
+1.  Create a PR of these changes and merge it directly to
+    https://bazel.googlesource.com/bazel/+/master (without the usual process of importing it to
+    the Google-internal version control).
+
+## 2nd and 3rd PRs: update references in the Bazel tree
+
+1.  In `third_party/protobuf/BUILD`:
+
+    1.  rename `PROTOBUF_VERSION` to `_OLD_PROTOBUF_VERSION`
+
+    1.  rename `_NEW_PROTOBUF_VERSION` to `PROTOBUF_VERSION`
+
+1.  In the root `WORKSPACE` file update relative paths of protobuf to point to the new version.
+
+1.  Update version number in `src/main/protobuf/BUILD` and `src/test/shell/testenv.sh`.
+
+1.  Update the current version in this file.
+
+1.  Create a PR of these changes and get it imported. Some files won't be imported (those that are
+    only hosted on GitHub); this is expected.
+
+2.  Wait for the imported PR to be pushed back to GitHub. Rebase the PR from the previous step, and
+    merge it to https://bazel.googlesource.com/bazel/+/master .
+
+## 4th PR: remove the old directory
+
+1.  Delete the `third_party/protobuf/<old_proto>` directory.
+
+1.  Remove `_OLD_PROTOBUF_VERSION` from `third_party/protobuf/BUILD`.
+
+1.  Create a PR of these changes and merge it directly to
+    https://bazel.googlesource.com/bazel/+/master .
+
+**Update this file if you found these instructions to be wrong or incomplete.**
 
 # Current protobuf version
 


Test patch: