Gerrit/Upgrade

We deploy Gerrit using the upstream war file published in Maven Central, it contains Gerrit itself as well as a collection of bundled plugin, on top of that we add plugins to customize our instance. The artifacts are stored in under a deploy/wmf/stable-X.Y branch and use git-lfs to offload the binaries payloads out of the git repositories. We build extra plugins from sources, which relies on the source code held in the Gerrit repository in branch wmf/stable-X.Y.
The upstream branches (master, stable-3.9) are manually kept in sync. We have a forked branch wmf/stable-3.9 which has our extra plugins registered as sub-modules as well as some Wikimedia specific build scripts at the root of the repository. Our branch can be used to cherry pick patches for plugins build purpose, but must eventually be send to upstream, reviewed and merged there. The branch overview:
 operations/software/gerrit branches layout
Branch 	Description
stable-3.9 	Gerrit core, updated from upstream
wmf/stable-3.9 	Fork holding our build script and our plugins as submodules
deploy/wmf/stable-3.9	scap deployment repository holding Gerrit .war and plugins .jar with git LFS
== Update our mirror repository ==
Even if we deploy Gerrit core from the upstream released .war, we still need an up-to-date Gerrit copy in order to build our plugins.
=== Update Gerrit core branch ===
#Clone Gerrit and submodules from Wikimedia, add upstream as a remote:
git clone https://gerrit.wikimedia.org/r/operations/software/gerrit
cd gerrit
git remote add upstream https://gerrit.googlesource.com/gerrit
If updating to a new minor version (for example from stable-3.9 to stable-3.9), you need to create new branches, else you would simply update them.
==== Create new branches ====
# Checkout the upstream stable-3.9 and create the branch in our Gerrit:
git fetch upstream --tags
git checkout -b stable-3.9 upstream/stable-3.9
git push --tags origin stable-3.9
# Create our new stable branch and update it:
git push origin wmf/stable-3.8:wmf/stable-3.9
git checkout -B wmf/stable-3.9 origin/wmf/stable-3.9
git merge v3.9.5
In .gitmodules, update the branch statements to point to the new stable branch (see below), then push for review git push origin HEAD:refs/for/wmf/stable-3.9.
==== Updating existing branch ====
#Our repository HEAD points to the branch holding the material for the deployment deploy/wmf/stable-3.9. Checkout the stable-3.9 branch which is a copy of the upstream release we currently use:
git checkout stable-3.9 origin/stable-3.9
# Fetch the upstream branch and fast-forward our copy of their branch to the latest tag. Then push the stable-3.9 directly to Gerrit:
git fetch --tags upstream stable-3.9
git merge --ff-only v3.9.5
git push --tags origin HEAD:stable-3.9
# Checkout wmf/stable-3.9 and merge the latest tag, push the resulting SINGLE merge commit up for review:
git checkout wmf/stable-3.9
git merge v3.9.5
git commit --amend --no-edit # add Change-Id
git push origin HEAD:refs/for/wmf/stable-3.9
=== Update upstream bundled plugins ===
Gerrit core comes with bundled plugins and a specific version of jgit which are registered as submodules. The .gitmodules file comes from upstream and uses relative urls. Since your workspace copy points to the Wikimedia Gerrit, the submodules would be fetched from our Gerrit which will would fail since the repositories do not exist. Instead we have to instruct git to rewrite the URLs to use upstream repositories. This can be done using the git configuration setting url.<base>.insteadOf = <our url>.
#Ensure you are in our forked branch:
git checkout wmf/stable-3.9
# update as needed
#Update git submodules from the upstream repository using: 
git -c url."https://gerrit.googlesource.com".insteadOf="$(dirname $(git config --get remote.origin.url))" submodule update --init
#Push for review. At this point, you will likely be ahead of wmf/stable-3.9 by...quite a bit (hundreds of patches). This is all within a single merge commit though, so Be Bold — push up your change for review:
git commit --amend --no-edit # to add Change-Id and Phabricator tracking bug to the merge commit
git push origin HEAD:refs/for/wmf/stable-3.9
Do NOT Code-Review +2 the resulting change yet, plugins need to be updated
=== Fetch our additional plugins ===
As our Gerrit instance runs some plugins which are not part of Gerrit upstream set of plugins, we have those extra plugins registered as submodules in our branch. They will need to be updated to match the new Gerrit version. You will need ssh admin access to Gerrit to be able to list the installed plugins as well as jq (sudo apt install jq).
List our plugins:
$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/rename-project
plugins/reviewers
plugins/zuul
$ 
Most plugins come with a set of stable- branches, some support multiple versions and solely have a master branch. The proper branch to use for our extra plugins is indicated in .gitmodules , we thus just have to update them from the remote branch (git submodule update --remote plugins/foo). To update our plugins:
$ ./wmf-plugins-update.sh
Updating Wikimedia specific plugins
<the list of our plugins is shown here>
<git submodule update if relevant>
Done
$ 
(as of October 2022 we no more fork gitiles) Plugins which we have forked on our Gerrit such as gitiles need the fork to be updated and upstream tag to be pushed:
cd plugins/gitiles
git remote add google https://gerrit.googlesource.com/plugins/gitiles
git fetch google
git merge google/stable-3.9
git commit --amend --no-edit # add Change-Id
git remote set-url --push origin ssh://gerrit.wikimedia.org:29418/operations/software/gerrit/plugins/gitiles.git
git push --tags origin HEAD:stable-3.9
Eventually review plugins changes:
$ git diff --submodule=log
Submodule plugins/healthcheck 5be80596ca..c84dfceab2:
 > Make documentation consistent
Submodule plugins/its-base 33380b5e25..84a2ea5694:
 > build: fix log(String) must be compile-time constants
 > Merge branch 'stable-3.9'
Once done updating our plugins submodules, git add them, update your commit and send it for review:
git add $(./wmf-plugins-list.sh)
git commit --amend
git push origin HEAD:refs/for/wmf/stable-3.9
CI will build Gerrit core as well as all the extra plugins, the resulting .war and .jar files are attached to the build.
== Build ==
For Gerrit core and its bundled plugins, just use the upstream released .war. This instructions are left here for reference in case we have to build Gerrit ourselves at some point although the logic has been ported to a Jenkins job and CI will happily build Gerrit and the plugins for you.
=== Build prerequisites ===
The full list of requirements can be found at https://gerrit.wikimedia.org/r/Documentation/dev-bazel.html or from source at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/stable-3.9/Documentation/dev-bazel.txt
Install OpenJDK 11, git, NodeJS, Python 3. On Debian:
$ sudo apt install git openjdk-11-jdk-headless nodejs python3
Gerrit and its plugins are build using Bazel. Since the exact version to use might differ between branches or can be updated by upstream at any point, one should install Bazelisk. It is a frontend to Bazel which automatically installs the needed Bazel version for the Gerrit branch you are working on. You will require the Go programming language in order to build it. The following will build Bazelisk 1.6.1 from source and copy the resulting binary in your user bin directory (~/.local/bin):
$ which go || sudo apt install golang-go
$ mkdir /tmp/bazelisk-build
$ (/tmp/bazelisk-build && go mod init . && go get github.com/bazelbuild/bazelisk@v1.6.1)
$ mkdir -p "$HOME"/.local/bin && cp /tmp/bazelisk-build/gopath/bin/bazelisk "$HOME"/.local/bin/bazelisk
$ rm -fR /tmp/bazelisk-build
=== Gerrit core ===
==== Build our own ====
As of Feb 2nd 2021 we no more build Gerrit ourselves, we instead use the upstream .war, see previous section. The build instructions are left there in case we have to build a fork.
# Checkout the wmf/stable-3.9 (as of May 2024)
git checkout -b wmf/stable-3.9 origin/wmf/stable-3.9
Production uses Java 8 as of November 19th 2020 which we will eventually switch to Java 11 ( https://phabricator.wikimedia.org/T268225 ). Meantime, Bazel has to be pointed to your local Java 8 installation: JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64.
# Build the bazel-bin/release.war
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 bazel build release
Note that if your building Gerrit v3.3 (or later) or master from 2020-10-06 (or later) and you're targeting Java 8, you'll also have to add the argument --java_toolchain=//tools:error_prone_warnings_toolchain (this is untested, but claimed on the gerrit mailing list). 
=== Plugins ===
Most plugins come with build instructions usually available at src/main/resources/Documentation/build.md.
Since we use the upstream war, we do not need to build the plugins provided with Gerrit core. Instead we only build our extra plugins. wmf-plugins-list.sh can list our plugins:
$ git checkout wmf/stable-3.9
$ ./wmf-plugins-list.sh 
plugins/go-import
plugins/healthcheck
plugins/its-base
plugins/its-phabricator
plugins/javamelody
plugins/lfs
plugins/metrics-reporter-jmx
plugins/metrics-reporter-prometheus
plugins/rename-project
plugins/reviewers
plugins/zuul
$ 
The whole build logic is implemented in the wmf-build.py script:
$ git checkout wmf/stable-3.9
$ ./wmf-build.py
Building Wikimedia specific plugins
> Building go-import
...
Done building
Copying build artifacts
plugins-wmf/go-import.jar
plugins-wmf/healthcheck.jar
plugins-wmf/its-base.jar
plugins-wmf/its-phabricator.jar
plugins-wmf/javamelody.jar
plugins-wmf/lfs.jar
plugins-wmf/metrics-reporter-jmx.jar
plugins-wmf/metrics-reporter-prometheus.jar
plugins-wmf/motd.jar
plugins-wmf/reviewers.jar
plugins-wmf/zuul.jar
Those should be copied to the wmf/deploy branch
Done
$ 
Make sure to remove all content from ./plugins before copying the build artifacts.
==== Add a new plugin ====
In our wmf/stable-3.9 branch and the plugin upstream repository as a submodule. It should point to the current stable branch (stable-3.9) if supported, else master.
git submodule add -f -b stable-3.9 -- https://gerrit.googlesource.com/plugins/rename-project plugins/rename-project
git add .gitmodules plugins/rename-project
git commit -m 'Add rename-project plugin'
Then follow the instructions from the previous section to build it and add the resulting .jar to our deployment branch.
If we were to fork a plugin, the process is similar. The repository is forked on our Gerrit instance and then registered as a submodule pointing to our Gerrit.
== Update artifacts ==
=== Setup git LFS ===
Since artifacts are rather large, we use git-lfs to offload the git repository:
# sudo apt install git-lfs
# Initialization: git lfs install
Git LFS determine the URL to fetch/push from the git remote, on our setup we have explicitly defined them in a .lfsconfig file. It will fetch over https and pushes over ssh using the same credentials used for regular git operations. It should essentially work out of the box.
==== Gerrit .war ====
The gerrit.war can be retrieved by running mvn package.
Checkout the deploy/wmf/stable-3.9 branch, edit pom.xml and adjust the gerrit.version property:
 <properties>
 <gerrit.version>3.8.6</gerrit.version>
 </properties>
Then run mvn package to download the Gerrit war released by Google on Maven Central, replaces the /gerrit.war at the root of the repository and unpack the bundled plugins to ./plugins/.
Git add the result (gerrit.war, plugins/*.jar and pom.xml) and craft the commit:
$ mvn package
...
[INFO] --- maven-dependency-plugin:3.1.2:copy (import upstream gerrit war) @ gerrit ---
[INFO] Configured Artifact: com.google.gerrit:gerrit-war:3.8.6:war
Downloading from wikimedia.mirrored: https://archiva.wikimedia.org/repository/mirrored/com/google/gerrit/gerrit-war/3.8.6/gerrit-war-3.8.6.pom
Downloaded from wikimedia.mirrored: https://archiva.wikimedia.org/repository/mirrored/com/google/gerrit/gerrit-war/3.8.6/gerrit-war-3.8.6.pom (2.8 kB at 1.5 kB/s)
Downloading from wikimedia.mirrored: https://archiva.wikimedia.org/repository/mirrored/com/google/gerrit/gerrit-war/3.8.6/gerrit-war-3.8.6.war
Downloaded from wikimedia.mirrored: https://archiva.wikimedia.org/repository/mirrored/com/google/gerrit/gerrit-war/3.8.6/gerrit-war-3.8.6.war (85 MB at 1.3 MB/s)
[INFO] Copying gerrit-war-3.8.6.war to ~/projects/operations/software/gerrit-deploy/gerrit.war
[INFO] 
[INFO] --- maven-dependency-plugin:3.1.2:unpack (extract upstream plugins) @ gerrit ---
[INFO] Configured Artifact: com.google.gerrit:gerrit-war:3.8.6:war
[INFO] Unpacking ~/.m2/repository/com/google/gerrit/gerrit-war/3.8.6/gerrit-war-3.8.6.war to ~/projects/operations/software/gerrit-deploy/plugins with includes "WEB-INF/plugins/*.jar" and excludes ""
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:10 min
[INFO] ------------------------------------------------------------------------
$ git add pom.xml gerrit.war plugins/*.jar
$ git commit -m 'Update Gerrit to vX.Y.Z'
==== TypeScript API ====
Update the @gerritcodereview/typescript-api module by checking whether a matching versions exists https://www.npmjs.com/package/@gerritcodereview/typescript-api?activeTab=versions . The API should only change on each minor version (ex: Gerrit 3.7 > 3.8).
Update it by changing the package.json then running npm install. Add the result:
$ git add package-lock.json package.json
$ git commit --amend
==== Plugins ====
After you have imported Gerrit releases.war from upstream and updated the plugins (either by building locally with wmf-build.py or retrieved from CI), the artifacts have to be added to the deploy/wmf/stable-X branch. 
If you have build locally:
#Copy all the previously build plugin jars from ./plugins-wmf to ./plugins in the deploy branch
If you want to rely on CI:
# Head to the last merged change of the wmf/stable-3.9 branch and follow the link to the gerrit-docker job build
# On the build page follow the Build artifacts link
# Download a zip via (all files in zip)
# Extract the archive: cd plugins && unzip -o -j /path/to/archive.zip
# Remove the release.war and its-base.jar built by CI: rm plugins/{release.war,its-base.jar}
If you get the release.war from upstream CI:
# https://gerrit-ci.gerritforge.com/job/Gerrit-bazel-stable-3.10/
# Download release.war
# unzip -j gerrit.war 'WEB-INF/plugins/*.jar' -d ./plugins/
==== Prepare the commit ====
Ensure Gerrit bundled plugins are not included since they are provided by the releases.war
* rm plugins/its-base.jar
Commit the changes and upload to Gerrit
* git add 'plugins/*.jar'
* Add a note if a plugin is no more present,
* git commit -m 'Update plugins for Gerrit 3.8.6'
* git push origin HEAD:refs/for/deploy/wmf/stable-3.9
The git push send the artifacts with git LFS.
== Deployment ==
When upgrading Gerrit releases.war to a new minor or major version: make sure to have taken in account upgrade steps that are mentioned in the upstream release notes.
Before deploying: 
* check the free disk space before deploying a new version (df -h and df -h -i)
* check Gerrit Icinga alerts manually during the maintenance to make sure no critical alert gets overlooked.
1) Schedule downtime for Gerrit hosts to avoid unwanted notifications in alerting channels. Tag SRE collaboration-services and request to schedule a downtime via the sre.hosts.downtime cookbook.
2) Disable Puppet
This make sure Puppet does not restart the service or alter the gerrit.config:
sudo puppet agent --disable 'Gerrit deployment'
3) Deploy artifacts
# SSH to the deployment master (ssh deployment.eqiad.wmnet)
# cd /srv/deployment/gerrit/gerrit
# Update the repository: git fetch
# Checkout the appropriate deploy/wmf/stable-X branch: git checkout deploy/wmf/stable-X
# Review differences: git log HEAD..HEAD@{u}
# git rebase if all is fine
Deploy artifacts to gerrit2002 to ensure that there are no errors with git LFS et al
#scap deploy -l gerrit2002.wikimedia.org 'Gerrit to [version] on gerrit2002 - TXXXXX'
Deploy to gerrit1003:
# scap deploy -l gerrit1003.wikimedia.org 'Gerrit to [version] on gerrit1003 - TXXXXX'
4) Upgrade Gerrit
; Accessing servers
: ssh gerrit2002.wikimedia.org (replica)
: ssh gerrit1003.wikimedia.org (primary)
We stop Gerrit and run the initialization steps. --skip-plugins is passed to prevent the extraction of bundled plugins, we already do that when preparing the artifacts and they are committed in the git repository.
#sudo service gerrit stop
#sudo -u gerrit2 -i
#cd review_site
#java -jar bin/gerrit.war init --batch --skip-plugins
5) Apply our config
Enable and run Puppet which starts Gerrit:
sudo puppet agent --enable
sudo puppet agent -tv
Take note of any change that might have been applied to gerrit.config: they should be transcribed back to Puppet.
6) Verify
Check the plugins versions, either on the web https://gerrit.wikimedia.org/r/admin/plugins or via ssh:
* ssh -p 29418 gerrit.wikimedia.org gerrit plugin ls
* ssh -p 29418 gerrit-replica.wikimedia.org gerrit plugin ls
Look at error log:
sudo journalctl -u gerrit -n 50 --follow
/var/log/gerrit/error_log
Logs are in Kibana https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV
7) Resume monitoring
* https://icinga.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=171&host=gerrit1003
* https://icinga.wikimedia.org/cgi-bin/icinga/cmd.cgi?cmd_typ=171&host=gerrit2002
Remove the scheduled downtime using the sre.hosts.remove-downtime cookbook.