nyuuzyou/wikis huggingface dataset:
Gerrit/Events ingestion

This was an experiment over the summer 2022. As of March 2023 there is no plan to complete it
Gerrit is able to emit events as JSON data. With the appropriate permission, events can be streamed via the stream-events ssh command which is how Zuul CI listens for new patches and comments. Internally an action, such as commenting on a patch, populates a Java class corresponding to the event, which is serialized with Gson and emitted on the stream events channel. Each event is represented by a Java class which all extend the abstract class com.google.gerrit.server.events.Event.
To publish Gerrit data to our Event Platform we could have:
* have the platform listen to the events directly much like CI is doing. This option has not been investigated.
* emit events directly to a Kafka topic. This got ruled out since any changes to the format of events would lead to side effects in the backend storage and the Kafka topics can take any arbitrary data
* HTTP POST to EventGate. This is the retained option which ensures data are validated and give a clear view of the data layout to the data engineering team.
EventGate can optionally validate any events it receives against a Json Schema and those schemas are required to setup the data backends (Hadoop, MySQL...). We thus had to provide a Json Schema for each events, rather than writing them manually, the schema are generated from the Java event classes. This is done by the events-wikimedia Gerrit plugin which relies on the jsonschema-generator library. Upon upgrading Gerrit, the schema are regenerated and send to the secondary event schema repository.
=== References ===
* send Gerrit events to our data lake
* Audit JSON schemas for Gerrit events
* Event Platform/Schemas
* Event_Platform/Schemas/Guidelines#Required_fields
Gerrit/Phabricator integration

Gerrit Notification Bot (aka gerritbot) is the username used by a Gerrit plugin to perform actions on Phabricator.
== Source ==
The bot is implemented as a Gerrit plugin: https://gerrit-review.googlesource.com/#/admin/projects/plugins/its-phabricator (GitHub)
The configuration is done via Gerrit daemon main config. See: gerrit.config.erb (search for "its-phabricator") and the files beneath /etc/its
== Functionality ==
* If Bug: TXXX is used as a footer in a Gerrit commit message, Gerrit Notification Bot will:
** Add a comment to task TXXX cross-referencing the Gerrit commit
** Add the tag #Patch-For-Review to task TXXX
* If a Gerrit commit is merged or abandoned that contains "Bug: TXXX" as a footer in its commit message, Gerrit Notification Bot will:
** Add a comment to bug TXXX cross-referencing the Gerrit commit
== Maintainer ==
If there's a problem, file a bug in Phabricator against #GerritBot.
== Archive ==
The feature was previously provided by script: grrrit-wm
Category:Services
Category:Gerrit
Category:Bots
Gerrit/Replica

gerrit-replica is a read-only replica of the Gerrit instance we run. It is used by Phabricator and ExtensionDistributor, among others.
== Usage ==
For services that can accept slightly delayed updates to repositories, using the replica will help in keeping the load on the main Gerrit instance low.
Urls are identical to the canonical instance, but use gerrit-replica.wikimedia.org, instead of gerrit.wikimedia.org.
If you have bots fetching from Gerrit (such as refreshing a list of repositories), consider using the replica with the following gitconfig:
 [url "https://gerrit-replica.wikimedia.org/r"]
 insteadOf = https://gerrit.wikimedia.org/r
Category:Gerrit
Gerrit/Users

== User storage ==
Gerrit's database is a bunch of git repos on the gerrit servers.
Users are stored in the All-Users repo on the gerrit servers in /srv/gerrit/git/All-Users.git
The All-Users repo has a ref called refs/meta/external-ids. Inside are a bunch of files that are stored by the sha1 of the file. They're all git config files used for different purposes.
Used for web login:
Used for SSH/REST API login:
(optional) Used for secondary email addresses:
the "mailto:" record only exists in the case where someone adds a secondary email
=== Finding user records ===
The easiest way to find them is to use git grep on a checkout of refs/meta/external-ids.
Inside the checkout, git grep email@example.com. You should get a result like:
XX/YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYF00: email = email@example.com
If you cat XX/YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYF00 you should be able to find the accountId of the user. From there you should be able to find all other information.
=== User preferences ===
user records are stored under refs/users/YY/XXYY where XXYY is the accountId from refs/meta/external-ids (that's all the user preferences)
=== Renaming users ===
This is difficult and requires some careful git surgery. For more information see: SRE/LDAP/Renaming_users/Gerrit
Git-lfs

"git large file storage" or git-lfs is a tool that helps with tracking large files inside a git repository while the file content is stored outside of git. The need for such functionality is necessitated by limitations in git performance which degrades as the size of the repository grows beyond a few megabytes. When storing files that are several hundred megabytes in size, git becomes prohibitively slow. git-lfsprovides one method of addressing this problem. Below you will find some tips on setting up git-lfs to work with Wikimedia projects and infrastructure.
=== git-lfs support in Gerrit ===
A few Wikimedia projects are beginning to experiment with using git-lfs to track large files associated with code that is hosted in Gerrit. Before git-lfs can be used it has to be enabled on the project's configuration in gerrit. Once lfs support is enabled, there are still a few more steps to use git-lfs: https://gerrit.wikimedia.org/r/plugins/lfs/Documentation/config.md
==== Installing git-lfs client ====
To work with projects that use git-lfs, you need to download and install the git-lfs client on your workstation. The client can be downloaded from https://git-lfs.github.com/, simply follow the instructions under "Getting Started" on the git-lfs site.
==== Setting a http password in gerrit ====
In order to use git-lfs you will need to use your gerrit http username and password. This can be set in the gerrit settings under http-password. Even when you use ssh to access a git repository, git-lfs uses http(s) to transfer large files.
==== Setting lfs.url in your project's git config ====
You need to set the lfs.url for your project to point to the gerrit lfs url. To do this you need to run a command similar to the following, replacing {username} and {project/repository} with values appropriate for the project you are setting up.
# git config lfs.url 'https://{username}@gerrit.wikimedia.org/r/{project/repository}/info/lfs'
==== Git credential store ====
To ease the annoyance of git password prompts, the use of git's credential store makes it fairly painless.
===== libsecret =====
You can enable libsecret credential helper so that git can store your https credentials securely:
git config --global credential.helper libsecret
===== libgnome-keyring =====
This is apparently deprecated, but retained for posterity:
For Gnome users, the `git-credential-gnome-keyring` helper is an option which can be enabled fairly easily:
sudo apt-get install libgnome-keyring-dev
cd /usr/share/doc/git/contrib/credential/gnome-keyring
sudo make
git config --global credential.helper /usr/share/doc/git/contrib/credential/gnome-keyring/git-credential-gnome-keyring
Snippet by James Ward CC-BY-SA 3.0
=== See also ==
* git-lfs tag in Phabricator
Git.wikimedia.org

Git.wikimedia.org may relate to several services:
* Gitblit (July 2013 - June 2016)https://phabricator.wikimedia.org/T111465#2415612
* Diffusion (June 2016 -)https://phabricator.wikimedia.org/T137224#2415573 (mirroring repositories)
== See Also ==
* Gerrit, our code review and repository hosting tool. Browse its code repositories via https://gerrit.wikimedia.org/g/
* GitLab, our code review and repository hosting tool. Browse its code repositories via https://gitlab.wikimedia.org/explore/groups
== References ==
 
Category:Services
GitLab/Abuse and rate limiting

This page summarizes the available tools against abuse and rate limiting bots/scraping.
=== Account approvals and blocking ===
Automatically created accounts added SEO spam to open merge requests and issues (see T334958). As a follow up all new accounts need a approval first (using this Phabricator form).
Furthermore GitLab accounts can be blocked in the admin menu. A cascading block for blocked wikitech accounts is work in progress: T359820. For obvious spam or vandalism it is also possible to "Delete user and contributions".
=== Malicious traffic pattern ("scraping") ===
GitLab is not behind the load balancing infrastructure, so tools like superset are not available to analyze traffic. However there are some Logstash dashboards which can be used to analyze the traffic:
Access logs dashboard: https://logstash.wikimedia.org/app/dashboards#/view/38dc4c50-7cae-11ef-b061-47f1245f2864
Application logs: https://logstash.wikimedia.org/goto/82d8dd92e36776ce581aeef953ff6891
=== Throttling using nftables ===
Even though GitLab is not behind the CDN some throttling and rate limiting capabilities were added. In T366882 a new Puppet module profile::firewall::nftables_throttling was implemented which allows basic rate limiting based on a maximum number of parallel TCP connections.
If the threshold is exceeded the offending IP address is blocked for 5 minutes. More information can be found in Firewall#Throttling with nftables.
Metrics for this throttling can be found in Grafana.
=== Blocked nets ===
Requestctl has a feature to ban single IPs or IP ranges with the abuse/blocked_nets ipblock. This IPs are dropped automatically on all wmf hosts using firewall rules. This is a simple and effective measure to ban a IP with little work.
For GitLab this feature is currently not working because of the recent migration to nftables. See T348734. But there is ongoing work to implement this feature also for nftables.
=== Blocking using apache config ===
GitLab has an option to inject a custom apache config using the GitLab config file, see insert-custom-nginx-settings-into-the-gitlab-server-block. This is also available in our Puppet code as profile::gitlab::custom_nginx_config. So in case a certain API endpoint or path is causing trouble, this option can be used to restrict access.
GitLab/Gitlab Runner/Shared Runners

=== Current Gitlab Runner setup (T287279) ===
We're currently relying on WMCS VPSs for shared runner capacity. There is a project named gitlab-runners in which to provision new instances, and a profile to help provision Docker based runners on those instances. Note that a standalone puppetmaster in the same project stores the runner registration token under /etc/puppet/secret, and Puppet autosigning is turned off to protect the token value.
==== Setting up a new shared runner ====
To set up a new shared runner, following these steps.
# Create a new WMCS VPS instance.
## Log in to https://horizon.wikimedia.org and navigate to the gitlab-runners project.
## Launch a new Debian bullseye instance, following the runner-{nnnn} naming convention.
## Choose the g3.cores8.ram24.disk20.ephemeral40.4xiops flavor
# Wait until the new instance has fully provisioned and you can successfully ssh to the running instance using your authorized key and sudo as root. (This typically takes a few minutes.)
# Check if there is an existing profile::gitlab::runner::token on the standalone puppetmaster in /etc/puppet/secret/hieradata. If not, generate a new token and add it there.
# Fully configure instance with volume, profile and project puppetmaster
## Add the following to the instance's Hiera Config under the Puppet Configuration tab in horizon.
profile::gitlab::runner::docker_volume: true
puppetmaster: gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud
## Add role::gitlab_runner to the instance's Puppet Classes under the Puppet Configuration tab.
# Do the little SSL dance that is required of instances that use a standalone puppetmaster.
## On the new runner (runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud).
### Run sudo rm -rf /var/lib/puppet/ssl to remove the existing SSL certs used by the default puppetmaster.
### Run sudo -i puppet agent --test --verbose --server gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud to have the puppet client generate a new SSL cert.
## On gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud sign the new instance's SSL cert.
### Run sudo -i puppet cert list and find the new instance in the list.
### Run sudo -i puppet cert sign runner-{nnnn}.gitlab-runners.eqiad1.wikimedia.cloud to sign the client cert.
# Run sudo -i puppet agent --test --verbose --server gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud on the runner to ensure it has fully provisioned the role::gitlab_runner role.
# Verify that the runner has successfully registered with our GitLab instance by viewing the runner list.
GitLab/Monitoring

GitLab is monitored by Prometheus and Icinga.
=== Metrics and dashboards ===
Metrics of GitLab instances are collected by Prometheus.https://gerrit.wikimedia.org/r/c/operations/puppet/+/707860 Multiple Grafana dashboards can be found in the folder GitLab.
A new GitLab Logstash dashboard with visualizations was added in phab:T375849.
Currently the following exporters are enabled:
Exporter 	Port	Path
nginx 	8060	/metrics
redis 	9121	/metrics
postgresql 	9187	/metrics
gitlab-workhorse 	9229	/metrics
rails 	8083	/metrics
sidekiq 	8082	/metrics
gitlab-server 	9168	/metrics
gitlay 	9236	/metrics
wmf_gitlab_exporter	9196	/
Related Task: T275170 and T354656
The wmf_gitlab_exporter is a custom exporter written in Python. The code can be found here: https://gitlab.wikimedia.org/repos/sre/gitlab-exporter. The exporter exposes metrics for special alerting and auditing use-cases mostly for the collaboration services team.
=== Alerting ===
Alerting happens with multiple Icinga and Prometheus blackbox checks. See the overview page. We mostly rely on the checks which come from the basic vm setup (like filesystem checks, puppet check). Additionally the public facing endpoints of production GitLab are checked as wellhttps://gerrit.wikimedia.org/r/c/operations/puppet/+/708530:
Host 	check
gitlab.wikimedia.org	https and cert check
gitlab.wikimedia.org	ssh check for git ssh daemon
Individual GitLab components (like Postgresql or Redis) are not monitored currently. 
There are additional alerts configured in Prometheus Alertmanager, see https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-sre/gitlab.yaml and all GitLab/Runbook.
In the future when individual GitLab components are separated adapting the existing upstream alerts is recommended.
GitLab/Runbook

A runbook is a set of instructions for a human what to do. More specifically what to do when a certain monitoring alert triggers. This page contains GitLab related runbooks, linked from alertmanager rules with the runbook annotation.
=== GitLabCIJobErrors ===
More than 50% of GitLab CI jobs are failing. 
Check what kind of job error is elevated https://grafana.wikimedia.org/d/Chb-gC07k/gitlab-ci-overview?orgId=1. 
Also check the past and currently running Jobs at https://gitlab.wikimedia.org/admin/jobs (admin privileges required).
=== GitLabCIPipelineErrors ===
More than 50% of GitLab CI pipelines are failing. 
Check what kind of pipeline error is elevated https://grafana.wikimedia.org/d/Chb-gC07k/gitlab-ci-overview?orgId=1. 
Also check the past and currently running Jobs at https://gitlab.wikimedia.org/admin/jobs (admin privileges required).
=== GitLabCIPipelineLatency ===
Pipeline createn takes more than 10 seconds.
Check number of active pipelines at https://grafana.wikimedia.org/d/Chb-gC07k/gitlab-ci-overview?orgId=1. 
Also check the past and currently running Jobs at https://gitlab.wikimedia.org/admin/jobs (admin privileges required).
Check if all runners are available and marked online at https://gitlab.wikimedia.org/admin/runners.
If needed, stop jobs which need too much resources.
=== GitLabRunnerTrustedConfigMissing ===
The configuration for the Trusted runners is missing a configuration setting. The runners have to be "locked", "protected" and must not run untagged jobs to ensure proper separation of trusted/reviewed jobs and unreviewed jobs. Sometimes this settings get lost by administrators accidentally removing them or by bugs/version upgrades.
This settings are handled by the gitlab-trusted-runner/ project. To re-apply the correct configuration, run a new pipeline, the runner-config job should re-apply the correct settings. (Note, the job runs every 30 minutes). Alternatively the edit button in the admin interface can be used to edit the settings directly. The correct settings are:
* run untagged jobs: false/unchecked
* protected: true/checked
* lock to current project: true/checked
GitLab/Stewardship

This table documents accountabilities for the various pieces of GitLab infrastructure.
What 	Accountable Team 	Consulted Teams/Groups	Informed Teams/Groups
Security Upgrades 	SRE Collaboration Services	NA 	Release Engineering
Major Release Upgrades 	SRE Collaboration Services	Release Engineering 	wikitech-l@, ops-l@, #engineering-all
Shared WMCS GitLab Runners 	SRE Collaboration Services	Release Engineering 	wikitech-l@, ops-l@, #engineering-all
Shared third-party cloud GitLab Runners 	Release Engineering 	 	wikitech-l@, ops-l@, #engineering-all
Trusted GitLab Runners/sotware packaging	SRE Collaboration Services	Release Engineering 	wikitech-l@, ops-l@, #engineering-all
Monitoring GitLab availability 	SRE Collaboration Services	❓ 	wikitech-l@, ops-l@, #engineering-all
Responding to outages 	SRE Collaboration Services	Release Engineering 	wikitech-l@, ops-l@, #engineering-all
Application-level technical support 	Release Engineering 	NA 	NA
GitLab/Test Instance

A production-like test instance can be used to test changes, updates and prototype code. The WMCS/VPS instance gitlab-prod-1002.devtools.eqiad1.wikimedia.cloud uses mostly identical Puppet configuration and is accessible using wmcloud SSO for users with wmf/nda permissions. The instance can be reached at https://gitlab.devtools.wmcloud.org/
The test instance does not share data (user, repos, groups) with the production instance. So if you need them for testing, you have to migrate this data or request certain groups.
==== Resetting test instance ====
Snapshot and restore is not supported in WMCS currently. If changes modify the test instance significantly, we advice to create an additional, temporary instance (see below). Minor changes can be tested by creating backups of certain config files, undo changes or disable puppet temporarily. However the test instance gitlab-prod-1002 must not differ from the actual puppet configuration.
==== Setup new test instances ====
If changes modify the test instance significantly, we advice to create an additional, temporary test instance in the WMCS/VPS project devtools. Follow steps below to create a additional instance:
* Create VM in WMCS project devtools, flavor g3.cores2.ram4.disk20, image debian-11.0-bullseye
* Create additional volume (~50gb) and assign it to VM (or re-map a existing volume from a old instance)
* Edit /etc/fstab and mount the volume at /srv/gitlab-backup
* Create floating IP (quota increase may needed)
* Create DNS A record <service-name> for floating IP in zone devtools.wmcloud.org
* Request floating IP to be mapped to a second, private IP/port (assistance from WMCS is needed here)
* Edit /etc/interfaces to make VM listen to both private addresses (similar to gitlab-prod-1001)
* Set at least the following hieradata for the host:
profile::gitlab::passive_host: '<service-name>.devtools.wmcloud.org'
profile::gitlab::cert_path: '/etc/letsencrypt/live/<service-name>.devtools.wmcloud.org/fullchain.pem'
profile::gitlab::key_path: '/etc/letsencrypt/live/<service-name>.devtools.wmcloud.org/privkey.pem'
profile::gitlab::service_ip_v4: '<second-private-ip>'
profile::gitlab::service_name: '<service-name>.devtools.wmcloud.org'
* Assign puppet role::gitlab to new instance
* Wait for puppet run/force puppet run on new instance
* Run initial cerbot command (see todo below, will be added to puppet):
certbot certonly --standalone --preferred-challenges http -d <service-name>.devtools.wmcloud.org
* Verify installation by login to the instance by using the new DNS entry. Login should be able using SSO (wmcloud idp has wildcard for wmcloud.org)
* Optionally:
** apply gitlab-settings
** make yourself admin
** create some groups, mirror some repos
** ...
==== Open todos ====
* move initial certbot command to puppet/verify if this is needed
* simplify hiera data so that hostname has to be set only once
* automate configuration for /etc/interfaces in puppet
* verify steps above
Gitblit

Our Gitblit installation is was at https://git.wikimedia.org/ and is ^demon's burden.
== Hardware ==
* antimony
== Operations ==
=== Restarting ===
The init script is /etc/init.d/gitblit. Puppet should ensure gitblit is running.
=== Delete a repo ===
* Remove the directory from /var/lib/git/*
Category:Services
Gitlab1001

gitlab1001 was the production Gitlab server (along with Gitlab2001 in codfw).
== Connecting ==
== Tickets ==
*phab:T274459 (VM creation request)
*phab:T296944 (Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints)
*phab:T295481 (Setup GitLab Runner in trusted environment)
Gitlab2001

gitlab1001 is the production Gitlab server (along with Gitlab1001 in eqiad).
== Connecting ==
== Tickets ==
*phab:T274459 (VM creation request)
*phab:T296944 (Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints)
*phab:T295481 (Setup GitLab Runner in trusted environment)
Grafana/Best practices/Getting Started with Thanos panels

This is a draft/WIP space for a Getting Started/Recipe/Cookbook page/section for common graphing with Grafana and Prometheus.
== Counters ==
=== Counter: rate of things per second ===
Use this for counters that typical count something during a web-request.
In almost all cases, you want to set the Min interval in the Query Options to 2m. The background-context for that is that the "scrape-interval" with which Prometheus gets a new data-point is the default 60 seconds. So at least 2 intervals are required to reliably cover the data-points, though note that the usual recommendations seems to be to use 4 intervalshttps://grafana.com/events/grafanacon/2020/prometheus-rate-queries-in-grafana/.
sum by(label1, label2) (
 rate(mediawiki_YourComponent_your_metric_name_total[$__rate_interval])
)
You can skip by(label1, label2) if you have no labels in your metric.
Unit: This will produce a metric that shows the number of counts per second averaged over the interval displayed (which depends dynamically on the time-range of your panel). So you should probably set the y-axis label to something like "things / s".
See also: https://grafana.wikimedia.org/d/e16fec87-72a6-405e-b931-d04a0bc26e48/cdanis-temporal-downsampling-considered-harmful-max-over-time-rate-interval-trick?orgId=1
TODO: add info here when that link is relevant
=== Counter: number of things per hour/day ===
TODO: add info here how to summarize per day
=== Counter from a maintenance script: number of things per hour/day ===
Not as useful for counters that are only incremented like once per hours via a maintenance script and actually the increment is the value of the thing.
== Timings ==
Grafana1001

grafana1001 is a virtual machine that runs in Ganeti. It can be controlled from ganeti1003 and runs on ganeti01. 
Grafana1001 runs grafana.wikimedia.org. (Previously ran on krypton.)
== Services ==
* include role::grafana grafana
== VM Info ==
[ganeti1003:~] $ sudo gnt-instance info grafana1001.eqiad.wmnet
..
- Instance name: grafana1001.eqiad.wmnet
 Creation time: 2018-12-03 15:35:25
 Modification time: 2018-12-03 18:19:23
 State: configured to be up, actual state is up
 Nodes: 
 - primary: ganeti1002.eqiad.wmnet
.. 
 Disks: 
 - disk/0: drbd, size 20.0G
 access mode: rw
 nodeA: ganeti1002.eqiad.wmnet, minor=1
 nodeB: ganeti1004.eqiad.wmnet, minor=4
...