nyuuzyou/wikis huggingface dataset:
Load balancing architecture

Originally created on January 2014, edited from a conversation between Roan Kattouw, Gabriel Wicke and Inez
LVS is the load balancer in front of the frontend varnishes. Every multi-server cluster has an LVS in front of it to load-balance requests. The workflow of a request to the Parsoid backend is thus:
 API LVS --> API server --> Varnish LVS --> Varnish frontend --> Varnish backend --> Parsoid LVS --> Parsoid server
How LVS load balancing and hashing work together? For every cluster of servers, there's something called a service IP. It's a public IP address that points to the LVS server, which selects a backend based on round-robin. LVS just does simple load balancing/fail-over and front-end varnishes do consistent hashing based on the request. Hence the selection of the backend server is very dumb. The only smart backend selection is from Varnish frontend to Varnish backend where the frontend knows exactly which backend has (or is supposed to have) the cache entry and so it hits its IP directly
The main purpose of splitting Varnish in frontend and backend is to do hashing on the frontend one. Basically the frontend Varnishes are glorified hash-based routers: they hash the URL and route the request to the right backend they also have smallish in-memory caches to serve very popular URLs from. As an implementation detail: the frontend Varnishes aren't actually separate boxes, each Varnish box runs one frontend and one backend, each on a different port, With 1/n probability, the relevant backend is on the same box and it uses loopback
In the parsoid case the front-end varnishes don't cache anything, they are pure routers.
Where can I see that hashing code? The puppet manifests for setting up our caches are in manifests/cache.pp (IIRC) in our puppet repo
== See also ==
* LVS
Local-charts/Tutorial

To better understand how to contribute to the local-charts environment, let's walk through it together.
In the walkthrough below we will show how to add a service that already has a helm chart to be deployed using local-charts.
Let's try adding the Blubberoid service from . Blubberoid creates a Dockerfile based on a yaml configuration file. Adding it to local-charts will let us run Blubberoid in Minikube.
== Goal ==
Deploy the Blubberoid service in Minikube by adding it to local-charts.
== Steps ==
=== Clone the local-charts repository ===
We'll be editing files in the local-charts repo: 
Run the installation script by running `make install` from within the repo's root directory.
=== Update requirements.yaml ===
First, we need to tell Helm that we require an additional chart in our deployment. Update helm/requirements.yaml like so:
=== Update values.example.yaml ===
Add blubberoid to the global enabled property:
Next, create a Blubberoid property in order to configure the Blubberoid chart. Possible values to configure can be seen in the Blubberoid chart in deployment-charts repo . For example, perhaps we want to enable Blubberoid's policy feature:
Now we're ready to deploy!
=== Run a deployment ===
First, run `make start` from within the local-charts root to start our Minikube cluster.
Second, run `make deploy values=values.example.yaml`
Blubberoid should now be deployed to Minikube!
Check it's running by running `kubectl get pods`:
=== Results ===
We can now try using Blubberoid!
first, get your minikube's IP by `minikube ip`, and the 5-digit blubberoid port by `kubectl get svc`.
Now try Blubberoid:
=== Clean up ===
You can delete your deployments by running `make undeploy` from the local-charts root.
Similarly, running `make stop` will stop the Minikube cluster
Locke

Dell PowerEdge 1950
* 2009-03-20: renamed from db6 to locke
* Management interface is still db6.mgmt Domas Mituzas 01:13, 16 October 2009 (UTC)
* Manual startup:
** inside screen: udp2log -p 8420 -f /etc/udp2log/squid
** as nobody (su -m nobody): cd /a/webstats/; bin/collector
Squid logging
Logmsgbot

logmsgbot is udprec piped into ircecho. It outputs !log messages to IRC. These and other (manual) !log messages are read by Stashbot and saved to Server Admin Log.
== Set up ==
Logmsgbot is provisioned via the Puppet class profile::tcpircbot which configures the tcpircbot class. In Wikimedia's production cluster, these Puppet classes are applied to alert1001/alert2001 and files are in /srv/tcpircbot/.
== Restart ==
Since the Icinga servers are on stretch, it's now using systemd. The service name is "tcpircbot-logmsgbot" which might be a bit confusing.
 systemctl status tcpircbot-logmsgbot
 systemctl restart tcpircbot-logmsgbot
== See also ==
* Wikimedia binaries: Most of these scripts run logmsgbot.
Category:Bots
Category:Bot and monitoring
Logstash/Extended Retention

=== Policy ===
See Privacy Policy and Data Retention Guidelines.
=== Request Process ===
# Ensure the log stream you want to preserve has only non-personal information not associated with a user account.
# Create a request to increase retention for a log stream on Phabricator, tagging the Observability team.
# Once the audit is complete and approved, the log stream will be tagged for inclusion into long-term retention indexes according to cluster capacity.
=== Audit Process ===
# Audit the log stream to determine it is non-personal information not associated with a user account.
# If in doubt, consult with the Security and/or Legal teams. Notify these teams when necessary.
=== Common Logging Schema fields indicating PII ===
These fields (as of ECS 1.7.0) have been identified as likely containing personal information.
* client.*
* error.message
* geo.*
* http.request.body.content
* http.response.body.content
* http.headers.*
* labels
* log.original
* message
* source.*
* tls.client.*
* user.*
* user_agent.*
Logstash/SRE onboard

= Shipping logs to Logstash via Kafka =
As part of FY2019 TEC6 goals SRE is implementing a new logging pipeline as part of taking ownership of the Logstash stack. See also the Logging infrastructure design document (source) for more details.
Implementation of the logging infrastructure is in progress as FY2019 Q2, this page aims at outline migration steps for existing applications.
== Syslog ==
Given how ubiquitous syslog is it has been chosen as the transport of choice, more specifically applications log using local syslog interface (i.e. syslog(3) or manually opening the /dev/log unix socket. The "identity" (also known as syslog tag or program name) is used to opt-in applications to ship their logs to Logstash (via arsyslog lookup table in puppet.git. The "level" as intended by syslog (i.e. info/err/warning/etc) is also set to its appropriate value per-message (see also structured logging below for more information).
== Structured logging ==
Most applications using Logstash nowadays are sending structured logs (in other words maps of key: value pairs), this use case is supported on the new logging infrastructure by sending JSON as the syslog message/payload. To have a robust way to detect JSON payloads the "cee cookie" (rsyslog documentation) is required to prefix said JSON messages, in other words a structured syslog message/payload would look like this (excluding syslog headers):
 @cee: {"foo": "bar"}
In case of parsing failure the message is regarded as a regular non-JSON message.
== Kafka and JSON ==
In preparation to sending to Kafka locally generated messages (structured or otherwise) are turned into JSON, message metadata (such as timestamp, source host, etc) is added and then JSON is shipped to Kafka. In the structured logging case the JSON parsed from the message is merged with the message metadata and then shipped to Kafka. Logstash will then consume from Kafka and ingest the message into its pipelines.
== UDP logging ==
A local UDP endpoint to accept the same messages/format as /dev/log can also be configured. This isn't the preferred interface and should be regarded as a compatibility layer only. There are several reasons for such an endpoint: for example the JVM doesn't support unix socket communication with /dev/log out of the box. Furthermore UDP provides a non-blocking interface, useful when transitions applications to local logging and blocking behavior of writing to /dev/log isn't known yet.
Logstash/Tasks

* Install Logstash stack in labs instance : http://logstash.instance-proxy.wmflabs.org/
* Direct beta cluster udp2log traffic to Logstash : 
* Start conversation with Ops about production hardware : 
* Write more filter rules to parse various message formats
* Get a syslog feed from beta
* Package logstash jar for deployment
* Make a puppet module
* Determine architecture for production deployment
** What log forwarding methods should be used?
*** syslog, udp2log, redis
** How many Logstash instances?
*** We will start with 3 nodes
** HA strategy?
*** Varnish, built-in Elasticsearch HA, Redis. There is no HA for udp2log at the moment.
** QoS terms? 
*** Not mission critical at this point. This will need to be revisited as usage increases but it seems unlikely that this will ever become the system of record for log storage.
* Implement Redis based transit of structured json log messages from MW
== See also ==
* Bug reports and feature requests: phab:tag/wikimedia-logstash/
MCS decommission

== Reasoning ==
We are in the middle of a big effort to sunset RESTBase (the underlying proxy between all services), including mobile content service (MCS). It's a multi-year effort with multiple teams and work-streams the final push to deprecate MCS and remove it from our infrastructure, a plan we had before this project started.
Moving forward this committed work will be prioritised and the Content Transform Team won't plan any new APIs or endpoints (JSON or not) unless it overlaps with the work for the Parser Unification (not related with the scope of interest of this topic).
== Timelines ==
Currently there's an active allowed list that won't receive any new additions. The allowed list will stop working soon (tentatively end of September) and the service fully removed from our servers. This allowed list is a temporary compromise for projects that already started migrating away from MCS and reached out before the deadline.
== Alternatives ==
A good candidate for extraction section information is a combination of these Action API requests:
* Get section information for page <PAGE> - https://en.wikipedia.org/w/api.php?action=parse&page=<PAGE>&prop=sections
* Render the output only for the specific section identifier <ID> - https://en.wikipedia.org/w/api.php?action=parse&page=<PAGE>§ion=<ID>
Machine Learning/LiftWing/Alerts

__TOC__
=== Kafka Consumer lag - ORESFetchScoreJobKafkaLag alert ===
When ORESFetchScoreJobKafkaLag alert fires it means that there is a lag between messages landing in Kafka topics and message consumption rate from Changeprop.
What may be happening:
* ORESFetchScoreJobs are taking too long to complete which results in an increased Job queue.
* Lift Wing may be returning errors so Changeprop is retrying thus taking longer to complete.
First thing to do is explore mediawiki-errors on logstash and review the grafana Jobqueue dashboard to spot the time of the increase by investigating the "Job run duration" chart. Check if there were any deployments at that time affecting either the ORES extension of Lift Wing.
Changes for Lift Wing can be tracked on the inferenceservices repository. For the ORES extension we can follow the MediaWiki deployment notes. For example if we notice that the job duration increased when MediaWIki_1.31/wmf.29 was deployed, we visit the changelog page for MediaWiki and check if there is any deployment related to the ORES extension repository.
We should also explore the logs on Lift Wing to figure out if there is an obvious reason for this lag.
If there have been changes that increase job duration and nothing can else can be done, increasing the job concurrency value will improve the job duration (and empty the queue faster), but we should keep in mind that these resources are shared among jobs so increasing the concurrency may affect other jobs. This can be changed in changeprop-jobqueue values in the deployment-charts repository.
=== Inference Services High Memory Usage - InfServiceHighMemoryUsage alert ===
This alert fires when memory utilization of the kserve-container of an Inference Service is above 90% of the container limit for more than 5 minutes. 
If a deployment has been made recently we focus on the recent changes to find something that is increasing memory usage.
Otherwise we explore the requests that have been made by looking at the logs to see if the issue is connected to the inputs.
In both cases exploring the memory utilization before the spike by looking at the "Kubernetes Container Details" Grafana dashboard will help us understand if the specified resources were tight or there is an issue that needs to be resolved.
=== LiftWingServiceErrorRate ===
Istio Dashboard
This alert fires if the percentage of error status codes (5XX and 0) from an Istio service is larger than a given threshold. Note that this means 3XXs and 4XXs are not considered errors, since they are usually client-caused. The error code 0 indicates that the connection was closed (usually by the client) before a normal HTTP response code could be sent.
Possible causes for this alert are breakage in the LW service itself, or upstream timing our or failing in a way that is not handled well by the LW service. It may also be due to a misbehaving client. Note that going over the rate limit would cause 429 responses either from LW or the API GW, and so this alert would not fire.
Machine Learning/LiftWing/ML-Sandbox/Usage-Examples

Examples on how to use the WMF ML-Sandbox environment to develop and test your isvc code.
Example 1: Work With and Test The model.py File
__NOTOC__
When moving ORES models to LiftWing, model.py relies on revscoring to load the model, extract features, handle errors and return a prediction.
To edit and test the model.py file we need an environment where all revscoring dependencies have been installed.
For example, if we want to work with the editquality model.py, one of the easiest ways would be to use the editquality docker image found on the Wikimedia docker registry.
The steps below are what I would take to work with the editquality model.py:
# Log into the ML sandbox
$ ssh ml-sandbox.machine-learning.eqiad1.wikimedia.cloud
# Pull the editquality docker image
$ docker pull docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-editquality:stable
# Run the docker container
$ docker run -it --entrypoint=/bin/bash docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-editquality:stable
# Edit the model.py and save your new changes.
I usually use vim for this but feel free to use a text/code editor of your choice.
# Test/Run model.py
# python3 model.py
Hacks I've found helpful:
# I usually download the model binary and point to it locally within the load() method of model.py 
#*Download model binary
# apt-get install wget -y && wget -O enwiki.damaging.gradient_boosting.model https://github.com/wikimedia/editquality/blob/master/models/enwiki.damaging.gradient_boosting.model?raw=true
#*Point to downloaded model in model.py
with open("enwiki.damaging.gradient_boosting.model") as f:
#Our LiftWing images on the Wikimedia docker registry are created using blubber. Blubber adds air-tight security restrictions like not being able to easily download the model binary or install software tools from the internet like text/code editors. Although these restrictions are good for the production environment, I find that they are a challenge in the dev environment since I like to have full autonomy and be able to tinker and test my ideas to make sure they work the best way possible. So to avoid the restrictions, on steps 2 & 3 above, I pull a docker image that I created without the blubber restrictions. 
#*Pull and run editquality docker image that doesn't have blubber restrictions
$ docker pull kevinbazira/kfserving-revscoring-model:v15
$ docker run -it --entrypoint=/bin/bash kevinbazira/kfserving-revscoring-model:v15
#* Steps 1, 4, and 5 remain the same regardless of the image I am using.
Magru data center

Magru (pronounced mah-groo) is a caching data center located at Ascenty São Paulo 3 in São Paulo, Brazil.
== History ==
Magru is the first Wikimedia data center not to follow the traditional Data center naming conventions of vendor initials + airport code. In December 2023, after two months of debate (SPGRU, BRGRU, CSGRU, IMGRU, GRU01, AAGRU), SRE DC Ops have decided on Magru as the location's name.P&T Weekly Status Updates: 2023-12-04, Wikimedia Foundation, Google Docs (restricted)
The datacenter came online on 2024-06-11. 
Read about some of the setup process in "The journey to open our first data center in South America".
== Notes ==
Category:Data centers
Mailman-htdig

This is a patched version of GNU Mailman which integrates support for searching list archives using htdig. This is integrated in the Wikimedia Mailman .deb.
=== Mailman patches ===
Apply these patches to the Mailman source before installation:
* archive adjustment for better searching
* htdig integration
The patch documentation describes how to enable it; this is not enabled in the default Mailman configuration file in the package.
=== htdig ===
htdig can be installed via yum on fedora and APT on Ubuntu.
Fun warning! Search result pages will come up blank with no error messages if htsearch is unable to locate the 'COMMON_DIR' which contains its HTML templates and images.
For some reason, the fedora package installs this directory at /usr/share/htdig but the binaries look for it at /var/www/html/htdig. I made a manual symlink to that disused location.
Additionally, that directory should be aliased in to /htdig on the mail web server so that the icons show up in result pages; for the current setup I just symlinked it in from the docroot.
== Index updating ==
Search indexes are updated by a nightly cronjob, at /var/lib/mailman/cron/nightly_htdig which is called from /etc/cron.d/mailman.
== Setup notes ==
Initially, lists are set up for htdig as they first receive a mail after the configuration is enabled; but they won't be indexed until the cronjob next runs. (It can also be manually run, of course.)
Mailman/Mailman2/Mailman encoding bug

Two messages in mediawiki-l archives from February 2006 triggered this exciting error when rebuilding archives with 'wipe':
Updating HTML for article 5494
Pickling archive state into /usr/local/mailman/archives/private/mediawiki-l/pipermail.pck
Traceback (most recent call last):
 File "bin/arch", line 200, in ?
 main()
 File "bin/arch", line 188, in main
 archiver.processUnixMailbox(fp, start, end)
 File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 585, in processUnixMailbox
 self.add_article(a)
 File "/usr/local/mailman/Mailman/Archiver/pipermail.py", line 626, in add_article
 filename))
 File "/usr/local/mailman/Mailman/Archiver/HyperArch.py", line 1116, in write_article
 f.write(article.as_text())
 File "/usr/local/mailman/Mailman/Archiver/HyperArch.py", line 579, in as_text
 '\g<1>' + _(' at ') + '\g<2>', body)
 File "/usr/local/python2.4/lib/python2.4/sre.py", line 142, in sub
 return _compile(pattern, 0).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 68: ordinal not in range(128)
As a workaround, I added this hack to skip over the affected items:
--- /usr/local/mailman/Mailman/Archiver/HyperArch.py.orig 2006-03-31 18:11:26.000000000 +0000
+++ /usr/local/mailman/Mailman/Archiver/HyperArch.py 2006-03-31 18:24:56.000000000 +0000
@@ -575,8 +575,11 @@
 otrans = i18n.get_translation()
 try:
 i18n.set_language(self._lang)
- body = re.sub(r'([-+,.\w]+)@([-+.\w]+)',
- '\g<1>' + _(' at ') + '\g<2>', body)
+ try:
+ body = re.sub(r'([-+,.\w]+)@([-+.\w]+)',
+ '\g<1>' + _(' at ') + '\g<2>', body)
+ except UnicodeDecodeError:
+ body = 'Mailman has a bug and Unicode decoding broke here.'
 finally:
 i18n.set_translation(otrans)
 return NL.join(headers) % d + '\n\n' + body + '\n'
Category:Mail
Mailman/Migration

This page documents comments made about past migrations. This is here to centralise specific notes and also to allow comments/experiences to expand into more detail in future.
= Migration from lily to sodium =
Configuration files can be copied to sodium just fine. Variables that may need to be changed are:
* reply_to_address
* host_name
Most of these can probably be done automatically.
Not present in the dumped list configuration file is the list's URLs. A fix_url withlist script is provided to change this.
Archives can be copied by just transferring the .mbox file, and then rebuilding the archive from scratch with arch --wipe.
* Files in dirs lists, archives and data should be rsynced again, with Mailman disabled on both sides. dpkg-reconfigure mailman should be rerun.
= Migration from sodium to fermium =
On 2015-09-18 we migrated from sodium (Ubuntu lucid) to fermium (Debian jessie), a VM running on the ganeti cluster. We use the default jessie package, so we are at 2.1.8 now.
T105756 was the tracking ticket for this migration and has many subtasks and a migration plan.
To copy list configs and archives we used the scripts rsync_lists and rsync_exim which can be found in /usr/local/sbin/ on the server and are puppetized in the mailman module. you can also see related gerrit changes.
After shutting down exim and mailman and rsyncing we also ran ./bin/update and ./bin/check_perms from the mailman package.
Mailman/Monitoring

Logs are in /var/log/mailman3 and /var/log/mailman3/web, rotated daily with a max retention of 30 days. In the future they may end up in logstash too.
== mailman3_runners ==
PROCS CRITICAL: 13 processes with UID = 38 (list), regex args '/usr/lib/mailman3/bin/runner'
Mailman3 has a number of job runners that process the in/out/bounces/virgin/etc. queues. There's a known bug upstream that the runners don't automatically restart when they crash. This alert detects that one of them has crashed and needs a restart. Currently the fix is to systemctl restart mailman3.
== mailman3_queue_size ==
CRITICAL: 1 mailman3 queues above limits: bounces is 1053 (limit: 25)
Monitors the size of the in, virgin and bounces queues. You can also see the size on this dashboard. If this alert is firing, something most likely has happened to one of the runners (and the above mailman3_runners alert has fired too) and needs a restart (systemctl restart mailman3).
You can look at the queue files in /var/lib/mailman3/queues/<name>. Each file in the queue is a pickled file, you can dump it with mailman-wrapper qfile <filename> or just pickle.load(open('<filename>')).
The out queue size will not alert because it is common for it to reach ~hundreds of emails when someone emails a large list.
== Misc stuff ==
We monitor a few key components:
* HTTP
** mailman archives: (hyperkitty) Are the archives for wikimedia-l reachable?
*** If not, check apache2.service and mailman3-web.service.
** mailman list info: (postorius) Is the subscribe page for wikimedia-l reachable?
*** If not, check apache2.service and mailman3-web.service.
* Queues
** Mailman outbound queue hours until empty: How long will it take to drain the out queue?
*** If it's going to take a long time, check:
**** Date and time of day. Historically, the first of the month has the longest queue of about 15+ hours long and 08:00 UTC each day about 1+ hours long.
**** dashboard
**** logs in /var/log/mailman3 and /var/log/exim4
**** you may have to look at the messages themselves in /var/lib/mailman3/queues/out to see what is queued up/not sending.
Map of database maintenance/Header

Here you can follow maintenance on our databases. This is useful to spot potential causes for issues in production or avoid interfering with another maintenance work.
A script parses dbctl SAL logs and add entries here. Furthermore, DBAs can add manual entries in this list by logging SAL. It is important that SAL must contain the word "dbmaint", a phabricator ticket id, a section name and name of the datacenter. For example: "!log dbmaint on s2@codfw (T12345)"