nyuuzyou/wikis huggingface dataset:
MariaDB/query performance

Here it is a recipe on how to measure query performance in order to compare 2 different servers, or the same server at two different periods of time or with two different mysql versions:
== Gathering data ==
Run the mysql command line client:
 # mysql --defaults-file=/root/.my.cnf
 SET @slow_query_log_file := @@GLOBAL.slow_query_log_file;
 SET @log_slow_rate_limit := @@GLOBAL.log_slow_rate_limit;
 SET @long_query_time := @@GLOBAL.long_query_time;
 SET @log_slow_verbosity := @@GLOBAL.log_slow_verbosity;
 SET @slow_query_log := @@GLOBAL.slow_query_log;
 
 SELECT @slow_query_log, @slow_query_log_file, @long_query_time, @log_slow_rate_limit, @log_slow_verbosity
 INTO OUTFILE '/tmp/variables.txt'; -- just in case we drop the connection
 
 SET GLOBAL slow_query_log_file := '/tmp/slow.log';
 SET GLOBAL slow_query_log := ON;
 SET GLOBAL log_slow_rate_limit := 20; -- set for an an appropriate rate, only 1 every log_slow_rate_limit queries will be logged
 SET GLOBAL long_query_time := 0; -- we log all queries
 SET GLOBAL log_slow_verbosity := 'query_plan'; -- extended logging
 
...
We now monitor on another session now the size of /tmp/slow.log, so it doesn't grow disproportionately.
I found that a rate of 20 for regular hosts, and 100 for busy hosts, for 24 hours is a good value.
If the client application uses some kind of connection pooling, you may want to restart at least one client application/connection pool/ORM.
After some time passes (e.g. 10GB of queries), we revert the changes:
...
 SET GLOBAL slow_query_log := @slow_query_log;
 SET GLOBAL long_query_time := @long_query_time;
 SET GLOBAL log_slow_rate_limit := @log_slow_rate_limit;
 SET GLOBAL slow_query_log_file := @slow_query_log_file;
 SET GLOBAL log_slow_verbosity := @log_slow_verbosity;
== Analysing data ==
If you do not want to affect your production server, copy the file /tmp/slow.log to a server where you can maximize the CPU usage without problem. pt-query-digest uses only one core, though.
I recommend compressing this file before sending it away: compressing it with something like bzip can shrink the size 10x!
Important: Never take the file out of the cluster, as it probably will contain sensitive data (query data)
In order to run pt-query-digest, just write:
 pt-query-digest --no-version-check --limit="100%" slow.log > /tmp/pt-query-digest.txt
If you have a compressed file, you do not need to decompress it first:
 pv slow.log.bz2 | bzip2 -d -c | pt-query-digest --no-version-check --limit="100%" > /tmp/pt-query-digest.txt
This new file, pt-query-digest.txt, will contain a report of all queries logged with its frequency, performance and statistics.
This log can also be used as the input for other tools like pt-upgrade.
== Reports generated ==
* Parser Cache (2015-05-25)
* External Storage (2015-06-04)
* Core Production: 
** 2015-07-01
** 2015-10-20
** main-traffic 2017-02-07
MariaDB/request new database

You need to provide:
* Brief summary (or pointer to a task) indicating how it is intended to be used (application or service)
* Previewed queries per second (worse case secenario)
* Total space needed and growth provision
* Availability constraints (can it suffer downtimes for maintenance?)
* Owner person or (preferred) team to contact
* DB Name
* User or users' names and their grants needed (recommended to separate admin accounts from service accounts, with limited rights)
* Backup policy
* From which ips the services will be accessed from (mysql client locations)
MariaDB Upgrade

=== MariaDB Upgrade ===
Only ever upgrade a slave. A master should be demoted first.
# Depool the box.
#* Check https://noc.wikimedia.org/dbtree/; box should disappear. 
#* Watch SHOW PROCESSLIST until connections go away.
# Disable icinga notifications for the host and it's services.
# /etc/init.d/mysql stop
#* Watch mwlog1001:/a/mw-log/dberror.log for sudden burst of connection errors.
#* May need to manually sync-common on relevant mw[0-9]{4}.
#* a host continually failing may indicate that they are not in the right dsh group or some other issue, check it.
# Disable puppet on the host (so it doesn't try to grab the apt db lock and break the upgrade in the next step)
# do-release-upgrade https://wikitech.wikimedia.org/wiki/Distribution_upgrades. Afterwards:
## rm /etc/apt/sources.list.d/*distUpgrade
## rm /etc/apt/sources.list.d/wikimedia-mysql.list
## apt-get update
## dpkg -l | grep mysql | awk '{print $2}' | xargs apt-get remove -y --purge
## rm /etc/mysql/*cnf*
## rm -rf /a/sqldata/*
##* Yes, everything in the datadir. Make sure it's the right box :-)
# Enable puppet on the host
# In puppet, site.pp:
#* mariadb => true
#* innodb_file_per_table => true
# puppetd --test
#* If stuff fails:
#*: puppet may have put the old mysql packages back on again, check if they are there and if so remove them
#*: now apt-get update and retry. The wikimedia-mariadb.list doesn't get added in time...
# xtrabackup clone: https://wikitech.wikimedia.org/wiki/Setting_up_a_MySQL_replica
#* The source box should also be depooled, or at least have it's MW load reduced.
# /etc/init.d/mysql start
#* innodb_file_per_table=1 on a shard with lots of databases/tables results in a slow startup. Wait it out.
# Use /a/sqldata/xtrabackup_slave_info to setup replication
#* CHANGE MASTER TO MASTER_HOST=..., MASTER_PORT=3306, MASTER_USER='repl', MASTER_PASSWORD='<in private repo>', MASTER_LOG_FILE=..., MASTER_LOG_POS=...;
# START SLAVE
#* Wait for it to catch up.
# Repool the box
#* Use a low load setting to let it warmup (hour or so?) then raise to full steam.
# Enable icinga notifications.
Maurus

maurus was the lead lucene search server in pmtpa.
* 2005-02-27: it is down. Brion said it was stealing IPs from another squid but wasn't up reliably enough to be reconfigured, so Kate shut down its switch port.
* 2005-04-15: up and idle for the last month
* 2005-08-22: has been running lucene (both daemons) for I don't know how long
== Services ==
* mwdaemon
* rsync for search index updates
** /usr/local/mwsearch as /mwsearch
== Batch jobs ==
* Will run search index updates on a regular basis
Mchenry

mchenry was a Dell PowerEdge 1950 with a single Intel Xeon 5130, with 4 GB of memory and 2x 300 GB hardware RAID. mchenry was Wikimedia's primary mail relay. Mailing lists emails were sent by sodium, emails such as bugzilla notifications and watchlist notifications by mchenry.
= History =
2007-12-18: Bad DIMM reported in DIMM2.
Record: 14
Date/Time: 11/30/2007 18:07:02
Source: system
Severity: 3
Description: Mem ECC Warning: memory sensor, transition to non-critical from OK ( DIMM2 )
-------------------------------------------------------------------------------
Record: 15
Date/Time: 12/14/2007 11:54:48
Source: system
Severity: 3
Description: Mem ECC Warning: memory sensor, transition to non-critical from OK ( DIMM2 )
-------------------------------------------------------------------------------
Record: 16
Date/Time: 12/16/2007 22:55:50
Source: system
Severity: 4
Description: Mem ECC Warning: memory sensor, transition to critical from less severe ( DIMM2 )
-------------------------------------------------------------------------------
2014-10-01: Decommissioned as part of the Tampa shutdown. Succeeded by Polonium in Ashburn.
MediaWiki Engineering

== Guides ==
== Development tools ==
Tools and services we provide for developers that maintain or contribute to MediaWiki. 
=== Frontend development ===
* ResourceLoader bundle size (Grafana): Stats about the size of your asset bundles.
* ResourceLoader module builds (Grafana): Stats about how long your asset bundles take to build during cache misses.
* ResourceLoader manifest size (Grafana): The size of your module metadata (name, dependencies, version hash) as sent to the browser in the site-wide "startup" request from all pageviews.
* mw.inspect: Inspect frontend build sizes in production or locally during development.
=== Backend development ===
* WANCache (Grafana): Performance stats for specific chunks of MediaWiki code in core and extensions.
* Backend Pageview Timing: Overall backend latency from MediaWiki when generating pageviews, e.g. to the CDN and to logged-in users.
* Backend Save Timing (Grafana): Overall latency when saving edits to articles. See also Save Timing breakdown by page type, account type, and entry point.
== Internal runbooks ==
These pages are mainly for use within the team. 
== Contact ==
=== Intake process ===
* Performance Review. Note that our main review starts after the change or feature is deployed to the Beta Cluster.
=== MediaWiki Platform Team ===
* Phabricator issue tracker: #MediaWiki-Platform-Team (triaged within 7 days)
* Libera Chat IRC: 
* Email: mediawiki-platform-team@wikimedia.org.
* or reach out directly to Larissa Gaulia.
=== MediaWiki Interfaces Team ===
* Phabricator: #mw-interfaces-team
* Libera Chat IRC: 
=== Content Transform Team ===
* Phabricator: #Content-Transform-Team
* Libera Chat IRC: 
* or reach out directly to Sérgio Lopes.
Category:MediaWiki Engineering
MediaWiki Engineering/Performance Review

The Performance Review process is a critical service provided by MediaWiki Engineering to help uphold the integrity of production software and infrastructure at Wikimedia Foundation.
The goal of these reviews is to reduce risk of a major change or new feature causing performance regressions detrimental to the user experience, or significant load or availability risk to Foundation infrastructure. 
Regressions in perceived performance can lead to user frustration and search engine ranking degradation, both likely to result in audience and traffic attrition. Performance reviews are a necessary step for all major changes deployed to Wikimedia production.
== Timeline ==
Performance reviews for a given quarter must be requested at least 21 calendar days prior to the beginning of the quarter.
As performance reviews can be time-consuming, these are usually considered quarterly goals for us.
Once the quarter has started and the preparation steps have been performed by the requester (see below), we commit to completing the performance review within 30 calendar days.
== Required preparation ==
Familiarise yourself with:
* Performance/Guides/Backend performance practices
* Performance/Guides/Frontend performance practices
Beyond setup, it's also necessary for the requesting team to do your own performance assessment first. Outline potential performance bottlenecks, potential optimisations and including performance measurements. Web performance should be a responsibility and concern owned and monitored by the individual engineering team (not offloaded to MediaWiki Engineering). New features and major changes should already have had their performance optimised to the best of the authors' abilities prior to being submitted to performance review.
The main prerequisite before our performance review can start is for the changes or features to be deployed to a Beta Cluster wiki.
If that isn't possible, the exception must be justified and the changes/feature will still have to be hosted on a wiki so that we can SSH into (for example, in Wikimedia Cloud VPS). It's insufficient to merely point us to the code. Reviewing must involve zero setup for us. This differs from code review where pushing the code to Gerrit is enough preparation for the reviewer.
== How to request a performance review ==
Use the "Request Performance Review" form on Phabricator and enter the required information. Thank you!
== See also ==
* mw:Writing an extension for deployment
Category:MediaWiki Engineering
__NOTOC__
MediaWiki Engineering/Runbook/Access control

== Phabricator ==
…
== Gerrit ==
…
== IRC ==
To add a team member to the channel, we grant the "Operator" flags. We have defined this to give the person a "voice" indicator (which helps others discover who's in the team and to easily remember their name), and to gives administrator rights (such as: edit the "topic" text, ban unwanted guests, and the viral ability to give Operator to other people). While in the channel, run these two commands:
 /cs flags #mediawiki-core TheirNicknameHere operator
For example, to grant Operator rights to Krinkle, you'd run /cs flags #mediawiki-core Krinkle operator.
To remove special rights from someone in the channel (this does not ban them), run these two commands:
 /cs flags #mediawiki-core TheirNicknameHere -*
The Operator group is defined as follows (see also ChanServ flags).
 /cs template #mediawiki-core operator +ARVefiorstv
== See also ==
* Technology/Onboarding/Checklists/Template on office.wikimedia.org (restricted).
* Wikipedia:IRC/Tutorial on Wikipedia.
* IRC/Instructions on Meta-Wiki.
* IRC/Channels list on Meta-Wiki.
Category:MediaWiki Engineering
MediaWiki HTTP cache headers

Varnish and Apache Traffic Server cache Wikipedia content.
MediaWiki caching headers are sent mainly in OutputPage.php, function sendCacheControl(). The headers sent depend mainly on the action and if a cookie is sent by the browser.
;cache-control
:* s-maxage: Tells intermediate caches how long they should consider the content to be valid without ever checking back. This needs to be hidden from caches we can't purge, otherwise users won't see changes.
:* max-age: For how long browsers should deem the content as up-to-date. We allow clients to keep the page (the "private" header allows this) but tell them to send a conditional if-modified-since request. For this the Last-modified header is needed; We set the header value to the last modification time or - if we don't have it - to the current time minus one hour. Images and stylesheets (including the generated ones that represent the user's preference selections) have max-age > 0 to avoid reloading those on each request. This is the reason why users have to refresh their cache after changing their preferences.
:* private: Allows browsers to cache content.
: Example: Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
;last-modified
:This is required for client-side caching, as without it browsers don't know what to base their if-modified-since requests on. If the page hasn't changed the cache will only respond with a 304 (unchanged) status code, and only the response code and headers are transferred.
;vary
:Tells downstream proxy caches to cache the content depending on some values — if those values are different, serve another page for the same url.
:* Accept-Encoding Clients that don't support gzip transfer-encoding don't get compressed pages.
:* Cookie Make sure logged-in users (which send a cookie) get pages with their user name and preferences.
:Example: vary: Accept-Encoding, Cookie
Category:Caching
Category:SRE Traffic
MediaWiki On Kubernetes/Migration Notes

== External traffic ramp-up ==
All numbers below per-datacenter.
+5% of external traffic:
* +12/13 replicas to mw-web
* +10 replicas to mw-api-ext
+22/23 replicas per 5% bump, times 5.6 CPUs/replica = 123 cores per traffic bump = ~3 servers (48 cores)
Available kubernetes cluster resources dashboard
MediaWiki UDP logging

All of the MediaWiki error logs should now be configured to be sent to udp://10.0.5.8:8420/[log_name]. On the relevant host, udp2log is installed (from udplog in svn). It runs in a root screen, with no command-line parameters. On the MediaWiki side, logging is configured via the global $wmgUdp2logDest.
All logs are sent to /srv/mw-log/[log_name].log on mwlog1002. They are rotated daily to /srv/mw-log/archive with 180 days kept, as per the defacto privacy policy. 
Configuration files are as follows:
===/etc/udp2log===
flush pipe 1 python /usr/local/bin/demux.py
flush file 1 /home/wikipedia/logs/everything.log
===/etc/logrotate.d/mw-udp2log===
/home/wikipedia/logs/*.log {
	daily
	olddir /home/wikipedia/logs/archive
	notifempty
	maxage 180
	rotate 1000
	dateext
	postrotate
		/usr/bin/killall -HUP udp2log
	endscript
}
===/usr/local/bin/demux.py===
Copied from svn with the base directory changed to /home/wikipedia/logs.
Category:Bot and monitoring
Category:MediaWiki production
MessageCache

This is Wikimedia-specific documentation for the MessageCache system.
The MediaWiki message cache stores text from the MediaWiki namespace for use during wfMsg() etc. When text is not present in the MediaWiki namespace, it calls the LocalisationCache instead, and passes the result back to the caller.
The message cache has 3 layers of caching:
# An in-process cache stored in MessageCache::singleton()->mCache
# A local cache, stored in $wgCacheDirectory (/tmp/mw-cache-1.18). The filename is of the form messages-<wiki>-<code>. 
# A shared cache, stored in memcached
To ensure that the local caches are properly synchronised, there is a content hash stored at the start of the local cache file, which is checked against a hash stored in memcached. 
The message cache is not normally completely rebuilt. On edit, MessageCache::replace() is called, which modifies the shared cache, replacing a single key. The hash is updated and all other servers reload their caches from the new shared cache.
Memcached keys:
; <wiki>:messages: : Stores the message texts below a certain size ($wgMaxMsgCacheEntrySize). This can expire without ill effects as long as the hash key is still present.
; <wiki>:messages:hash:: The hash key for the local message cache and <wiki>:messages:
; <wiki>:messages:individual:<title>: Stores the text of an individual message which was larger than $wgMaxMsgCacheEntrySize
== How to ==
; To clear the cache
: mwscript eval.php --wiki=<wiki>
: MessageCache::singleton()->clear();
Metrics Platform/API

API in Metrics Platform may refer to:
== Metrics Platform client API ==
Metrics Platform provides client libraries that you can use to build instruments:
* JavaScript API
* PHP API
* Java API
* Swift API
At present there is a 1:1 relationship between programming language and client: effectively, the JavaScript MPC is the MediaWiki frontend client; the PHP client is the MediaWiki backend client; the Java client is the Android app client; and the Swift client is the iOS app client. In principle, however, the libraries should be language-specific but platform-agnostic, and should be able to be shared by multiple clients implemented in the same language. Our goal is for all MPCs to provide the same API and behave the same way, and to be as clear as possible when they do not.
== See also ==
* See https://github.com/linehan/wmf-epc for early prototype implementations of the JavaScript, Java, and Swift MPCs
Category:Disambiguation pages
Metrics Platform/App schema

This page documents Metrics Platform's base schema for mobile apps. The schema includes three types of properties:
* core properties: fields describing the event itself
* interaction data: fields with custom values set by the instrument code to fit the needs of the experiment
* contextual attributes: fields describing the performer and environment of the event
This page helps you understand which properties of each type are available, so you can plan your instrumentation specification. For property descriptions and types, see the app base schema definition.
== Example event ==
[
 {
 "action": "cancel",
 "action_source": "2",
 "action_context": "2510",
 "agent": {
 "app_flavor": "prodrelease",
 "app_install_id": "9977d039-70d8-4951-94f3-6cfcb19359ee",
 "app_theme": "LIGHT",
 "app_version": 506,
 "client_platform": "android",
 "client_platform_family": "app",
 "device_language": "fr",
 "release_status": "prod",
 "app_version_name": "WikipediaApp/2.7.50506-r-2024-10-08",
 "device_family": "samsung SM-G973F"
 },
 "dt": "2024-10-21T18:25:36Z",
 "http": {
 "has_cookies": null,
 "method": null,
 "protocol": null,
 "request_headers": {
 "user-agent": "WikipediaApp/2.7.50506-r-2024-10-08 (Android 12; Phone; SM-G973F Build/SP1A.210812.016) Google Play"
 },
 "response_headers": null,
 "status_code": null,
 "client_ip": "2a01:e0a:333:a9d0:f46e:4a43:ef3e:2a3c"
 },
 "mediawiki": {
 "database": "frwiki"
 },
 "meta": {
 "domain": "fr.wikipedia.org",
 "dt": "2024-10-21T18:26:03.404Z",
 "id": "3fd8c1f8-b241-4e86-83c3-4ee796d9744d",
 "request_id": "1f0ebbd8-207f-4543-be1c-36ec566501f8",
 "stream": "android.product_metrics.article_link_preview_interaction",
 "uri": null
 },
 "page": {
 "title": "Notre_Parti",
 "content_language": "fr",
 "id": 13704218,
 "namespace_id": 0,
 "namespace_name": "MAIN",
 "revision_id": 219634859,
 "wikidata_qid": "Q7314695"
 },
 "performer": {
 "groups": [
 "*",
 "user",
 "autoconfirmed",
 "autopatrolled"
 ],
 "id": 87538527,
 "is_bot": null,
 "is_logged_in": true,
 "is_temp": null,
 "language_groups": "[fr, es, en, ca]",
 "language_primary": "fr",
 "name": "Rachimbourg",
 "pageview_id": "fda560e5cfc4b83edc92",
 "registration_dt": null,
 "session_id": "cb766dd9452da71e6b6e"
 },
 "sample": {
 "rate": null,
 "unit": null
 },
 "$schema": "/analytics/product_metrics/app/base/1.2.2"
 }
]
== Core properties ==
== Interaction data ==
== Contextual attributes ==
Contextual attributes provide information about the performer who triggered the event and the wiki where the event occurred. The values of contextual attributes are populated automatically by Metrics Platform when the event is generated as long as they have been added to the stream configuration.
All attributes must be enabled in the instrument's event stream configuration. For more information, see Contextual attributes.
== Code references ==
Metrics Platform/Architecture

This page provides a detailed overview of the Metrics Platform architecture.
== Components ==
The Metrics Platform architecture includes the following components:
* Extension:EventLogging
* Extension:EventBus
* Extension:WikimediaEvents
* Extension:EventStreamConfigs
* Extension:MetricsPlatform
* MPIC
We'll give a brief introduction to each of the extensions and MPIC, then we'll discuss how the extensions interact with one another to deliver instrumentation, and then how the MetricsPlatform extension specifically interacts with MPIC.
=== MediaWiki Extensions ===
Instrumentation developers use the EventLogging extension directly (or indirectly) to submit analytics events to an event stream in the Data Platform via the event intake service, EventGate.
If the instrumentation is written in JS, then EventLogging will submit the analytics event directly. If the instrumentation is written in PHP, the EventLogging extension will defer to EventBus to submit the analytics event. In both cases, the EventStreamConfigs extension is used to determine the details of the event stream that the analytics event is being submitted to.
=== EPIC (fka MPIC) - Experiment Platform Instrument Configurator ===
Instrumentation owners use EPIC to configure their instruments pre- and post-deployment. Instrumentation configuration is editable via a web app and served to Metrics Platform Clients via a RESTful API.
The EPIC application provides the following endpoints:
https://mpic.wikimedia.org/api/v1/instruments
https://mpic.wikimedia.org/api/v1/experiments
Limited to authorized end users, the EPIC application can be accessed at https://mpic.wikimedia.org/
=== Extension:MetricsPlatform and EPIC ===
Now that the components have been introduced, we can discuss how the MetricsPlatform extension and EPIC work together to deliver configuration to instruments running on the server and on the client.
The MetricsPlatform extension fetches instrument configuration from the EPIC RESTful API and dynamically creates event stream configurations for the EventStreamConfig extension to process.
Delivering instrumentation and its configuration is secondary to delivering content to the user quickly. As such, the MetricsPlatform extension caches the response from the EPIC RESTful API in the WAN Cache. The extension will attempt to fetch the cached response first before making a request to the EPIC RESTful API. If an error occurs or the request takes more than 250 ms to complete, then a cached response is returned. If there is no cached response, then an empty response is returned, which disables all EPIC-managed instrumentation.
=== GrowthBook JS SDK Analysis ===
One piece of code that is very interesting to us is the GrowthBook JavaScript SDK. This SDK has multiple functions, all of which are interesting to us. An analysis of the code will be conducted collaboratively here: Metrics_Platform/Architecture/GrowthBook_JS_SDK_Analysis