nyuuzyou/wikis huggingface dataset:
Fundraising Monitoring

This page is intended to document the monitoring infrastructure that exists for the fundraising as well as keep track of desired monitoring functionality.
== Existing monitoring infrastructure ==
=== Log monitoring ===
==== Minfraud log ====
We currently monitor the 'minfraud' log (which gets aggregated form the payments cluster to Loudon). 
== Monitoring wishlist ==
See RT tickets #405
* Hudson
** Nagios check for alive-ness
** Nagios check for failed builds
*** Note: some scripts run by Hudson need to be modified to throw a non-successful exit status when they don't complete properly (eg send/receive mail scripts for civimail)
** Nagios check for too many files in build folders (if the limit of 63999 gets hit, builds will fail)
* ActiveMQ
** Nagios check for queues filling up too fast
* Service communication times
** Nagios checks for timeouts/unacceptably high communication times
* 3rd party service accessibility from payments cluster
** Nagios check for communications access to MaxMind/PayPal
Category: Fundraising
Category:Fundraising - Needs Updated
GDI/Equity Landscape/Dashboard

=Equity Landscape Dashboard=
==Setup==
* To setup the dashboard, you'll need two things:
** Copy the json files from Google Drive.
*** After uncompressing (unzip) the file run the below command (python version 3 is required):
**** OR
**** 
*** This will make the files available on your localhost.
** To update the files, you can use this notebook and store the files in the respective directories under the Webfiles directory.
* Initializing the dashboard.
** Clone the dashboard from GitLab (ensure you have git installed)
*** 
** cd to the dashboard directory:
*** 
** Run the below commands, make sure npm is installed first:
*** 
*** 
** If you see the below output, copy the link from your terminal and paste it on your browser to view the dashboard:
*** 
==Code==
===Stores===
* These are responsible for storing the application state data.
** Data Store
*** Responsible for storing metric, country and world data.
**** If you want to introduce a new metric group (tab), this is where the code would go.
*** Chart Store
**** Responsible for most of the chart initialising code for the dashboard.
*** App State
**** Responsible for storing the different application state data that is shared throughout the entire dashboard application.
===Components===
* These are the components that are used throughout the dashboard, with the Dashboard component being the central component and incorporating all the other components (except for Projects and Languages) which is defined as part of the views.
** It is called with various measurementType to specify which metrics the dashboard should display.
** These are defined under the data store as per the below:
***
** Each observation type has a list of metrics that it encompasses. 
*** Currently we have the below observations:
***
** Each of the observations correspond to a Tab on the dashboard (minus Projects and Languages),
*** If you want to add a new Observation (Tab), you would need to update both ObservationType and observations' respectfully.
===Styles===
* Styling is done via scss.
* All the styling files can be found under the scss folder.
* They correspond to the component/tab that they are styling.
===Charts===
* The charts are implemented with ECharts and Vue good table
* We currently have the following charts:
** World Map (ECharts)
*** World Maps use two datasets for mapping the countries, one being the world_data.json and country_data.json found on our fileserver.
*** These are joined together and used with each metric for the visualisations. 
** BarChart (Vertical/Horizontal) (ECharts)
** Tree Map (ECharts)
** Text Table (Vue good table)
*** Text Tables can be formatted using a format function, example:
***
** Format functions are useful for styling the data within the cells. This is done by defining a list of cell styles that are defined in the _chart.scss styles file.
Gadolinium

gadolinium is a Wikimedia esams bits event logging to vanadium relay (misc::logging::relay::eventlogging).
gadolinium is a Wikimedia log collector (role::logging).
gadolinium is a Wikimedia Webrequest log stream unicast to multicast relay (role::logging::relay::webrequest-multicast).
gadolinium is a Wikimedia udp2log data collection server (udp2log::logger).
Events:
* [Analytics] Udp2log consumers / Eventlogging issues from 2014-01-05
Ganeti evaluation

= Goal =
The goal is to evaluate ganeti as a virtualization cluster management tool for miscellaneous services
= Quick breakdown of design =
* Two machines, thallium/mercury, networked with access ports in the same vlan/IP public address space.
* One shared IP for the cluster (ganeti.wikimedia.org)
* No shared storage (usage of DRBD)
== Rammifications of the design ==
VMs can have no network which was considered OK for the eval. We can easily bypass that by creating a VLAN and giving them IPs or turning the host machines into routers. Should be discussed after the eval though. In general ganeti is made to work both in bridged as well as routed mode.
== Important info ==
* ganeti.wikimedia.org is a DNS A(AAAA) record that points to the current master everytime.
* Ganeti is/can be a shared NOTHING Virtualization cluster management suite
* The master is manually switched over. Failure of the master does NOT mean disruption of services. Just loss of management for VMs (and if the master has VMs, disruption for those too).
= Installation =
After puppet has run for the first time
On both nodes
 
 # aptitude install ganeti drbd8-utils ganeti-htools ganeti-instance-debootstrap qemu-kvm ndisc6
 # swapoff -a # Prerequisite for the next command since swap was on LVM
 # vgrename thallium-vg ganeti-vg # All nodes need the same LVM VG
 # swapon -a
 # cat > /etc/modprobe.d/drbd.conf <<EOF
 options drbd usermode_helper=/bin/true
 EOF
 
On the master only (thallium)
 
 # gnt-cluster init --candidate-pool-size=2 --enabled-hypervisors=kvm --master-netdev=eth0 --vg-name=ganeti-vg ganeti.wikimedia.org
 # gnt-cluster modify --nic-parameters=link=100,mode=routed
 # gnt-cluster modify -H kvm:kernel_path=/boot/vmlinuz-3.13.0-24-generic (not really used, but needed to be set at something to squash warnings)
 # gnt-node add mercury.wikimedia.org (a ganeti bug/problem with trusty was discovered here, patched on the fly, will probably need newer packages, already talking with the maintainer
 # gnt-cluster command cp /root/.ssh/id_dsa.pub /root/.ssh/authorized_keys2 (some commands need root)
 
We are good and ready to go
= Operations =
 Login to ganeti.wikimedia.org using ssh. You should elevate privileges to root before doing anything else
== Create a new VM ==
 # gnt-instance add -s 10g -t drbd -o debootstrap+default --no-name-check --no-ip-check --net 0:ip=192.168.0.1 testme
== Login in to the console of a VM ==
 # gnt-instance console testme
== Reboot/shutdown/startup a VM ==
 # gnt-instance reboot testme
 # gnt-instance shutdown testme # Will send ACPI shutdown signal, after 2 mins will forcefully kill the VM
 # gnt-instance startup testme
== Modify the VM attributes ==
 # gnt-instance modify -B mem=1g testme
== Live migrate a VM ==
 # gnt-instance migrate testme
== Non-Live migration of a VM ==
 # gnt-instance failover testme
== Run cluster verification ==
 # gnt-cluster verify 
 # gnt-cluster verify-disks 
== Node statistics ==
 # gnt-node list
And much much more, read the docs at http://docs.ganeti.org/ganeti/current/html/admin.html
Gdash.wikimedia.org

gdash.wikimedia.org was a dashboard to show a selection of graphs rendered by Graphite.
* Requires no authentication.
* Pre-configured graph render queries
** Maintained in puppet: https://github.com/wikimedia/operations-puppet/tree/production/files/gdash/dashboards
** Graphite DSL manual: https://github.com/ripienaar/graphite-graph-dsl/wiki
** Graphite manual: http://graphite.readthedocs.org/
== Service ==
Was hosted on graphite1001.
Previously hosted on tungsten, and before that on professor.
== See also ==
* graphite.wikimedia.org
* grafana.wikimedia.org
== External link ==
* https://gdash.wikimedia.org
Category:Decommissioned services
Generate an SSH Key

This page contains information about generating a public SSH Key
== Overview ==
This page contains information about generating a public SSH Key. If you are generating a key for production access, please see SRE/Production access#Generating your SSH key.
== Generate an SSH key ==
1) Enter the following in the terminal. Use the email you wish to associate with the SSH Key, and then press enter. 
 ssh-keygen -t ed25519 -C "your_email@youremail.com"
2) When prompted Enter file in which to save the key, press Enter to choose the default.
3) Type a secure passphrase, and press Enter.
4) Retype the secure passphrase, and press Enter.
The terminal will display information about the files your identification and public key will be saved in:
* ~/.ssh/id_ed25519 - private key (as the name suggests, keep this private!) 
* ~/.ssh/id_ed25519.pub - public key
 
A fingerprint and randomart image for the key will be generated.
== Add or Change a passphrase ==
Add or change the SSH passphrase using this command:
1) Enter the following into your terminal $ ssh-keygen -p, and press Enter.
2) When prompted Enter file in which the key is (/Users/you/.ssh/id_rsa):, type the location of your key and press Enter.
3) Enter your old passphrase, and press Enter.
4) Enter the same passphrase again, and press Enter.
5) our identification has been saved with the new passphrase.
== Notes ==
* Make sure to choose a passphrase. Do not leave it empty.
* ssh-agent can be used to store your passphrase securely, so you do not have to re-enter it.
* Mac OS X Leopard or later and Linux users SSH can save their passphrase in the system's keychain.
Generating CAPTCHAs

==Overview==
Captchas are randomly selected from storage when MediaWiki decides to serve them to users. Memcached is heavily used to cache the directory listings of captchas (there are several subdirectores based on hashes). To generate a new batch, the captchas must first be created by a python script in a temp directory and then copied into storage. Each captchas is just two English words concatenate together and stored in a png file. The file names contain information about the solution (though they are salted with a secret key) and thus listings of captchas in storage should be disabled for un-authenticated reads (e.g. end users).
Captchas should be made with a blacklist to avoid some of the more common offensive (though sometimes amusing) word combinations. They should also be readable (at least by CAPTCHA standards), so FreeMonoBoldOblique.ttf is a good font choice.
==Making new CAPTCHAs==
To create enough new captchas so that the rough total is, say, 10000, run the following command (on mwmaint1002):
mwscript extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php aawiki --wordlist=/etc/fancycaptcha/words --font=/usr/share/fonts/truetype/freefont/FreeMonoBoldOblique.ttf --blacklist=/etc/fancycaptcha/badwords --fill=10000 --verbose
Geolocation

Geolocation is based on the MaxMind GeoIP2 database paid for by the WMF, and is used in two ways:
* Varnish adds a cookie called GeoIP (only if the request does not already have one), with lifetime set to the current session, in the format <ISO 3166-1 country code>:<ISO 3166-2 region code>:<city name>:<lat>:<long>:<???>
* The analytics pipeline adds geolocation data to the geocoded_data field of the webrequest table, based on the IP address.
To look up data by hand, log in to mwlog1001 or mwmaint1002 and run mmdblookup --file /usr/share/GeoIP/GeoIP2-City.mmdb --ip <IP> (see maxmind's site for documentation of the returned data structure) or, if you just want a single field, something like mmdblookup --file /usr/share/GeoIP/GeoIP2-City.mmdb --ip <IP> country names en.
== History ==
Geolocation started as a Fundraising-Tech initiative introduced in 2009. Some links around how its various incarnations are/were used:
* Analytics/Geolocation, published in 2014
*Slowness of GeoIP lookup and workarounds
*gerrit:260316 documents the cookie format
*Analytics/Geowiki documents one way to use the geolocation data
*meta:Geonotice (discussing alternatives to current system for regional-level geolocation)
*"ryan lane says that our geoip service is probably not fine grained enough": meta:IRC office hours/Office hours 2012-01-12
*Extension:CentralNotice might happen to contain some information in a far future.
*meta:MaxMindCityTesting was a test of a geolocation DB data provider
*"Can 3rd parties use geoiplookup.wikimedia.org" (like ULS on another MediaWiki installation)? No known problems on WMF end.
*As of 2015, GeoIP information is set by varnish in a cookie, see phabricator:diffusion/OPUP/history/production/templates/varnish/geoip.inc.vcl.erb for more information and gerrit:190964 for an overview of the system.
*As of 2016, the geoiplookup service was shut down.
== Unknown country ==
You may encounter geo data where country is "Unknown" and country code is "--"
The primary (and perhaps only?) source of this is requests and edits made internally, such as bots running on Toolforge and other Wikimedia Cloud Services infrastructure.
They will have IP addresses starting with "10." – for example in the cu_changes table:
SELECT
 cuc_ip, cuc_agent,
 COUNT(1) as n_changes
FROM cu_changes
WHERE cuc_ip RLIKE '^10\\.'
GROUP BY cuc_ip, cuc_agent
ORDER BY n_changes DESC
One example is IP address "10.192.32.203" (with User-Agent ChangePropagation-JobQueue/WMF) and indeed, it is one of our servers (cf. codfw.wmnet).
If we geolocate that:
ADD JAR /srv/deployment/analytics/refinery/artifacts/refinery-hive-shaded.jar;
CREATE TEMPORARY FUNCTION get_geo_data as 'org.wikimedia.analytics.refinery.hive.GetGeoDataUDF';
SELECT get_geo_data('10.192.48.103') AS geo_data;
we get:
{
	"city" : "Unknown",
	"subdivision" : "Unknown",
	"timezone" : "Unknown",
	"country_code" : "--",
	"country" : "Unknown",
	"continent" : "Unknown"
}
Category:Fundraising
Gerrit

We use Gerrit for our code review system.
Gerrit is installed on gerrit1003.wikimedia.org and in the prefix /var/lib/gerrit2 and serves public request at https://gerrit.wikimedia.org/r/) . A replica runs on gerrit2002 which serves requests at https://gerrit-replica.wikimedia.org/r/ ), it is intended for a switch over and for various bots in order to offload the primary instance.
Gerrit is a Java daemon which listens for HTTP connections (TCP port 8080) and SSH connections (TCP port 29418). Apache proxies the relevant URLs on port 80 and 443 through to Gerrit. The SSH port provides a restricted shell for Git upload pack, reviews and administrative commands.
Since the Gerrit v3.2 upgrade in summer 2020, Gerrit no longer uses a conventional, relational database. So if you read somewhere about Gerrit's MySQL (or similar) database or "reviewdb": it is stale information. Instead of a relational database, Gerrit 3 stores the needed data directly in the git repositories (NoteDB). To speed up lookup, it creates indices also known has secondary index. These indices are Lucene backed indices and H2 (flat file database engine from the Java world).
Git repositories are stored at /srv/gerrit/git/.
Gerrit uses the Wikimedia developer accounts for authentication. Accounts under ou=people,dc=wikimedia,dc=org and groups under ou=groups,dc=wikimedia,dc=org are exposed to Gerrit. The Gerrit account name is the cn field in LDAP. In order to log in to Gerrit, a user needs to already have a Wikimedia developer account. See Help:Create a Wikimedia developer account for the process of getting a developer account.
As of July 2021, Gerrit runs with Java 11 ().
== End users documentations ==
* Gerrit online documentation
* Our own documentations:
** mw:Gerrit/Tutorial
** Commit message guidelines
** mw:Gerrit/Advanced usage
** Get a new gerrit repository
Alternatively, you can follow the Git & Gerrit codelab.
Category:Gerrit
Gerrit/Abuse and rate limiting

This page summarizes the available tools against abuse and rate limiting bots/scraping. It is uses GitLab/Abuse and rate limiting as a template.
=== Malicious traffic pattern ("scraping") ===
Gerrit is not behind the load balancing infrastructure, so tools like superset are not available to analyze traffic. However there are some Logstash dashboards which can be used to analyze the traffic:
* Access logs dashboard
* Application logs
=== Throttling using nftables ===
Even though Gerrit is not behind the CDN some throttling and rate limiting capabilities were added. In T366882 a new Puppet module profile::firewall::nftables_throttling was implemented which allows basic rate limiting based on a maximum number of parallel TCP connections.
If the threshold is exceeded the offending IP address is blocked for 5 minutes. More information can be found in Firewall#Throttling with nftables.
Metrics for this throttling can be found in Grafana.
=== Blocked nets ===
Requestctl has a feature to ban single IPs or IP ranges with the abuse/blocked_nets ipblock. This IPs are dropped automatically on all wmf hosts using firewall rules. This is a simple and effective measure to ban a IP with little work.
For Gerrit this feature is currently not working because of the recent migration to nftables. See T348734. But there is ongoing work to implement this feature also for nftables.
=== Blocking using Apache config ===
As seen in this patch, it is also possible to block low traffic incidents through Apache configuration. This method is not recommended for high traffic incident. The recommended method in such context would be to either use throttling or drop traffic.
Gerrit/Architecture

== Gerrit Architecture ==
=== Overview ===
This document provides a comprehensive overview of the current architecture of the Gerrit instance used at the Wikimedia Foundation. It includes details on why Gerrit is not behind a CDN, and the alerting mechanisms in place.
It operates on dedicated servers rather than behind a Content Delivery Network (CDN). This choice was influenced by historical decisions, operational requirements, and technical limitations discussed below.
=== Current Alerting Mechanisms ===
* Icinga Monitoring: Configured to check Gerrit's HTTPS availability. It sends IRC alerts to the the admin contact group.
* Prometheus Blackbox Monitoring: Configured to monitor HTTP(S) endpoints by checking for the HTTP status, following redirects, and verifying the presence of specific content (e.g., "Gerrit Code Review"). It sends alerts to IRC, creates log entries, and creates Phabricator tasks on service availability.
=== Decision to Not Use CDN ===
==== Historical Context ====
Gerrit is not positioned behind a CDN due to a decision made by the Wikimedia SRE team. The decision, which hasn't been documented previously, was made ~2013 and was at various times discussed between SREs, e.g. during onboarding. As the team grew, this practice no longer sufficed.
The decision to not place Gerrit behind a CDN was made to avoid additional points of failure and ensure the availability of critical tools independently of the CDN and it was based on the following considerations:
# Decoupling from the CDN: Ensuring Gerrit remains available without relying on the CDN, which could introduce additional points of failure.
# Technical Complexity: Gerrit requires the SSH port (29148) to be accessible on the same IP, which complicates using a CDN due to the need for TCP proxy functionality
==== Re-evaluation of CDN Usage ====
Despite the historical stance, in 2024 a debate started about whether the advantages of placing Gerrit behind a CDN, such as improved abuse mitigation, would outweigh the risks. With advancements in technology and anti-abuse measures within modern CDN setups, this decision may be revisited in light of new technologies and organizational needs. The phabricator task where the discussion is taking place is at T365259
A key consideration for this discussion is to explore potential solutions to address the technical challenges related to SSH and TCP proxy if the decision to use a CDN is revisited.
=== Emergency Procedures ===
In the event of Gerrit's downtime, emergency procedures are in place to ensure continuity of operations:
* DNS Updates: If Gerrit is unavailable, DNS changes can still be performed as outlined in the DNS Emergency Procedures.
* Direct Deployments: Operations requiring immediate changes can be handled directly on deployment servers, bypassing Gerrit until it is back online.
Gerrit/Bot account

A bot account is a user account reporting to Gerrit without human interactions, in Gerrit such accounts are referred to as being a service user. A typical example is the Zuul/Jenkins CI reporting as jenkins-bot. This page has a few guidelines for those accounts.
The account MUST be added to the Gerrit "Service Users" group which is a built-in group. Members of the group have a few behaviors adjusted:
* Sending a review will not add the bot account to the attention set 
* Their requests are handled independently from interactive human users https://gerrit.wikimedia.org/r/Documentation/access-control.html#service_users
* They are granted access to the JSON events stream ssh stream-events
Review comments send by bot accounts should add to their review a tag prefixed with autogenerated: which will filter older messages from the Web UI. Tags are free form, they can be used to differentiate between messages emitted by the bot, for example Zuul CI adds the pipeline (test, gate-and-submit) which triggered the message, respectively: tag: autogenerated:ci-test and tag: autogenerated:gate-and-submit.
== Checks UI ==
Message emitted by the bots can be integrated in the Checks web ui by using the JavaScript Plugin Checks API. This requires JavaScript code to be added to our Gerrit deployment repository (https://gerrit.wikimedia.org/g/operations/software/gerrit/ , branch deploy/wmf/stable-X.Y, file plugins/wm-checks-api.js). As of March 2023 and Gerrit 3.5 the file is https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/deploy/wmf/stable-3.5/plugins/wm-checks-api.js
Each message in Gerrit is a ChangeMessageInfo, the TypeScript API for Gerrit version 3.5 can be read at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/gerrit/+/refs/heads/stable-3.5/polygerrit-ui/app/api/rest-api.ts#434
/**
 * The ChangeMessageInfo entity contains information about a message attached
 * to a change.
 * https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#change-message-info
 */
export declare interface ChangeMessageInfo {
 id: ChangeMessageId;
 author?: AccountInfo;
 reviewer?: AccountInfo;
 updated_by?: AccountInfo;
 real_author?: AccountInfo;
 date: Timestamp;
 message: string;
 accounts_in_message?: AccountInfo[];
 tag?: ReviewInputTag;
 _revision_number?: PatchSetNum;
}
The Checks API has a TypeScript definition is at https://gerrit.googlesource.com/gerrit/+/master/polygerrit-ui/app/api/checks.ts ).
Additions should be requested via a task to Phabricator against the #gerrit tag with a subject prefixed by [wm-checks-api].
== References ==
* Gerrit "Service Users" group