nyuuzyou/wikis huggingface dataset:
Performance/Runbooks/Analyze memcached

== memkeys ==
This is like "top" but for memcached traffic. Real-world examples:
* Memcached incident in 2022, T310532#8004500
* Traffic audit in 2021, T264604#7175809
* Traffic audit in 2017, T74024
# from an appserver, see local requests to the "on-host tier" instance
mw0011$ sudo memkeys -i lo -p 11210
# from an appserver, see outgoing requests to the memcached cluster
mw0011$ sudo memkeys -i lo -p 11213
# from a memcached server, see all incoming requests to that server
mc0032$ sudo memkeys -i eth0
mc0046$ sudo memkeys -i ens2f0np0
Example output:
# memcached key calls objsize req/sec
# WANCache:global:rdbms-server-readonly:deployment-db07:enwiki:ǀ#ǀv 3581 53 12.88
# WANCache:global:centralauth-user:44df0d9a65e7d3fad8664f4b2dc2ǀ#ǀv 2973 1178 10.40
# WANCache:enwiki:messages:en:hash:v1ǀ#ǀv 2954 124 10.19
Upstream:
* https://github.com/bmatheny/memkeys 
* https://gitlab.wikimedia.org/repos/sre/memkeys - WMF debian package
== cachedump ==
Inspied by @elukey via T203786#4999052.
 krinkle@mc1048:~$
 echo "stats cachedump 1 0" | nc -q 1 localhost 11211 > dump-01.txt
 echo "stats cachedump 16 0" | nc -q 1 localhost 11211 > dump-16.txt
 echo "stats cachedump 26 0" | nc -q 1 localhost 11211 > dump-25.txt
 echo "stats cachedump 32 0" | nc -q 1 localhost 11211 > dump-16.txt
 $ head cache-01.txt
 
 ITEM WANCache:global:SqlBlobStore-blob:commonswiki:tt%3A429993772|#|v [756 b; 1660458645 s]
 ITEM WANCache:global:revision-row-1.29:frwiki:6035826:144772360|#|v [790 b; 1660264028 s]
 ITEM WANCache:commonswiki:MessageBlobStore:ext.visualEditor.language:nl:2b6785d…|#|v [724 b; 1660719066 s]
 $ cat dump-*.txt | fgrep WANCache | cut -d " " -f 2| cut -d ':' -f3 | sort | uniq -c | sort -rn
 
   18669 SqlBlobStore-blob
 8855 filerepo-file-foreign-description
   1735 preprocess-hash
    596 getExtendedMetadata
    254 MessageBlobStore
Upstream:
* https://github.com/memcached/memcached/wiki/Commands#statistics
* https://github.com/memcached/memcached/wiki/ProgrammingFAQ#how-can-you-list-all-keys
== See also ==
* Memcached for MediaWiki#Mcrouter
* Memcached for MediaWiki#Runbooks
* T228970: Evaluate memsniff as alternative to memkeys
Memcached analyze
Performance/Synthetic testing/Cloud instance

== Synthetic tests on a cloud instance ==
We run our direct synthetic tests on a cloud servers using Hetzner to match our old setup where we used AWS running tests from the US. At the moment Hetzner do not provide bare metal servers in the US. The positive thing with the cloud servers are that they are cheap and work fine for our direct tests. We use four servers:
* first view direct tests using desktop
* user journeys direct tests using desktop
* first view direct tests using emulated mobile (Chrome in emulation mode)
* user journeys direct tests using emulated mobile
The servers runs Ubuntu and the tests runs in Docker containers.
== Installation ==
You create a new server in the Hetzner GUI. We use dedicated vCPU servers with 4 CPU (AMD) and 16 GB RAM.
When you create the server in the GUI, make sure to choose latest stable version of Ubuntu and a dedicated vCPU.
Then the setup follows the same instruction as setting up a bare metal server, except that you should not pin the CPU (you can't on the cloud server).
Performance/Synthetic testing/OnDemandTesting

=== On demand performance testing ===
We have setup for running on demand performance testing The big win with this setup is that we can easily re-use the hardware we use for performance monitoring for on demand tests. The performance monitoring tests use the hardware and continuously runs the test. With the on demand GUI/API you can add one off tests that will be the next job to run in the hardware.
At the moment we are testing the setup running desktop/emulates mobile tests. The plan is to have the setup also run Android tests.
The setup looks like this.
=== Servers ===
We have three servers (we can add more test runner servers):
* The queue queue.webperformancetest.eqiad1.wikimedia.cloud that runs KeyDB and handles the job queue.
* The database db.webperformancetest.eqiad1.wikimedia.cloud that runs PostgreSQL and a webserver front end for adding tests through the GUI.
One machine at Hetzner (138.201.135.103) that runs the actual tests. At the moment this server kicks of the jobs known as the direct tests (testing direct against Wikipedia servers) against the on demand testing framework.
=== The queue ===
The queue run KeyDB (an Open Source version of Redis). It stores jobs in memory.
When you login to the server, you will get information on how restart and how to change configuration.
The configuration for KeyDB exists in keydb.conf. The configuration file holds the password, set a max memory limit (to make sure the server do not run out of memory) and configuration to store the tests on disk using an append only file (aof) to make sure we can restart KeyDB without loosing any tests in the queue.
The KeyDB is setup as a service. You can restart the service using: sudo systemctl restart keydb-queue
=== The Database ===
The database store meta data about all the tests. The database uses PostgreSQL. 
There's a user: database that have access to everything. You can restart the database with:
sudo systemctl restart database.service
=== The GUI ===
The GUI run on the database server with the database user.
You can restart the GUI using sudo systemctl restart sitespeedserver.service
=== The test runner ===
At the moment there's one test runner/test server that runs the tests. The same server runs the tests for the direct tests for monitoring. If you pass on a test through the GUI, that test will be the first in line in the queue.
Performance/Synthetic testing/Result storage

== Synthetic test result storage ==
The synthetic test produces test results in the form of HTML, videos, screenshots and browser tracelogs from the tests. These results are useful when you need investigate a performance regression. We used to use S3 on AWS to upload the data, but we pushed a lot and compared to running our own server, S3 was much more expensive. Today we use server at Hetzner with 4 disks (two raids) where we store the data. The data is then served from a nginx instance running on that server.
=== Server setup ===
You can login to the server using: ssh USERNAME@116.202.221.47
The data is stored in /data/ and /home/data/ (it's two different disks). The synthetic test server SCP the content to this server.
The data is served using nginx and the setup for the instance it located in /etc/nginx/sites-available/wikimedia.sitespeed.io 
Old test results are removed after 35 days using the crontab for root (yep we should change that). It looks like this:
0 10 * * * find /data/* -mtime +35 -exec rm -rf {} \; >/dev/null 2>&1
0 20 * * * find /home/data/* -mtime +35 -exec rm -rf {} \; >/dev/null 2>&1
When other server uploads the data, it uses the user uploader that has access to the both directories that is used for uploading files.
Performance/Synthetic testing/Result storage/Runbook

This is the runbook for the result storage for synthetic testing.
==Meta==
*Issue tracker (Phabricator): synthetic-performance-testing
*Documentation: ResultStorage
== Debug ==
If results for the tests is missing or not working you will see that in Grafana by either that the "see test result" annotation do now show screenshots when you hover the metrics or when you to go to the result of a test and you get a 404.
If all results are missing there is probably a problem with nginx serving the data. If there's just one or a couple of tests that miss data, it could have problem uploading the data. 
=== Uploading problems ===
Servers that upload content use the user uploader that has access to the uploading folders. You can find if there's been problem uploading files by checking the Slack channel #synthetic-tests-error-reporting 
If there's been problem with uploading you will see logs looking something like this:
hetzner-1-baremetal [2024-10-15 04:24:49] ERROR: Error: Connection lost before handshake
hetzner-1-baremetal [2024-10-15 04:24:49] ERROR: Could not upload using scp Error: Connection lost before handshake
If that happens a lot you need to dig into why SCP has problems. If it happens a couple of times for month you can just ignore it.
=== Serving content problems ===
The nginx server is straightforward. If you think something is wrong with nginx you can check the status using:
sudo systemctl status nginx
If you the server Is down you can start it with sudo systemctl start nginx
If you need to change the configuration you reload it with sudo systemctl reload nginx
Category:Runbooks
Periodic jobs

We are targetting an early March date for migrating all MediaWiki periodic maintenance jobs from the maintenance servers to the mw-cron MediaWiki_On_Kubernetes deployment on Wikikube.
== Logs ==
 Logstash Dashboard
== Monitoring ==
TODO
== Job Migration ==
===General procedure===
====Code changes====
The jobs are still defined in puppet, using the profile::mediawiki::periodic_job resource.
Additional parameters are necessary to migrate a job to mw-cron and remove it from the maintenance servers.
* If the periodic jobs are defined in a subprofile of profile:mediawiki::maintenance, change the class definition to include the $helmfile_defaults_dir parameter
class profile::mediawiki::maintenance::subprofile(
 Stdlib::Unixpath $helmfile_defaults_dir = lookup('profile::kubernetes::deployment_server::global_config::general_dir', {default_value => '/etc/helmfile-defaults'}),
) {
* Include your subprofile in profile::kubernetes::deployment_server::mediawiki::periodic_jobs
* Add the following additional parameters:
 cron_schedule => '*/10 * * * *', # The interval must be converted from systemd-calendar intervals to crontab syntax. Keep the interval parameter as well if your job is used on beta
 kubernetes => true, # Create the CronJob resource in mw-cron, and remove the systemd-timer from the maintenance server
 team => 'job-owner-team', # For easier monitoring, log dashboard, and alerting
 script_label => 'scriptName-wikiName', # A label for monitoring, logging, and alerting, preferably the script name and its target
 description => 'A longer form description of the periodic job',
 helmfile_defaults_dir => $helmfile_defaults_dir, # Pass down the directory where the jobs will be defined
If you are not migrating all jobs in the subprofile at once, you will notice a lot of resources being created on the kubernetes deployment server. This is normal and an artifact of how we define the jobs. They will be no-op resources until the above additional parameters have been defined - if in doubt, ask in #wikimedia-serviceops. The example below makes the actual diff easier to read.
====Example====
We recommend the following procedure:
* Create a first change adding the subprofile to the profile::kubernetes::deployment_server::mediawiki::periodic_jobs profile, like . This will have all the no-op changes.
* Migrate the jobs in follow-up patches like 
====Deployment====
# Disable puppet on the maintenance server if the job can't be interrupted easily
# Merge the puppet change
# Run puppet on the deployment server, this will create the job definition, but it won't be deployed to mw-cron yet
# Stop the job on the maintenance server if it is running
# Deploy the mw-cron change with helmfile on the deployment server. The job will start on its next scheduled trigger.
# Enable puppet on the maintenance server and run it. This will delete the systemd timer for the maintenance job.
Category:MediaWiki production
Phabricator/Conduit API Tokens

Wikimedia's SRE and Release Engineering teams maintain several automated processes which utilize Phabricator Bot accounts to access Phabricator's Conduit API. This list is an attempt to document all of these bots and where their api tokens are used.
A list of all Phabricator bot accounts is available by using the "Active Bots" query.
== Bots maintained by RelEng & SRE ==
== Bots maintained by WMCS ==
== Bots maintained by the community ==
== See Also ==
* mw:Phabricator/Bots
Category:Bots
Phabricator/Disaster Recovery

=== Disaster recovery plan for phabricator.wikimedia.org ===
For reference, here is a list of patches required to migrate to a new phabricator server: https://gerrit.wikimedia.org/r/q/topic:phab1002
And here is a more recent migration from phab1001 to phab1003: https://gerrit.wikimedia.org/r/q/%2523phabricator-server-switch
=== High level overview ===
* Switch Phabricator production to phab2001 (in codfw)
** verify that git-ssh is working on phab2001
** shut down phd on phab1001
** manually rsync the git data from phab1001 to phab2001 (this is already rsync'd periodically, just need a one-time refresh)
** verify that phd works on phab2001
** switch `phabricator_active_server` to `phab2001` in hiera, `role/common/phabricator/main.yaml`
** test phabricator's web interface by locally overriding dns records to point to phab2001
* update dns / redirect traffic to phab2001
** Update SFP records for the phabricator hostname (example)
* Make phab1001 a warm standby for phab2001
** verifying that everything is installed correctly
** manually rsync the git repositories.
** make sure the rsync cron job is set up correctly to sync from phab2001
=== See Also ===
* Phabricator: Failure Scenarios
* The Phabricator task for developing a disaster recovery plan:
Phabricator/Mail debugging

For end-user management of mail, see mw:Phabricator/Help/Managing mail.
This a dump of rough notes on debugging notification mail on the Phabricator/Phorge instance. They may or may not be useful or correct.
Gathered while investigating phab:T370352 about missing notifications, which turned out to be a mail provider issue.
== Command-line tools ==
sudo bin/mail offers some diagnostics on mail, can resend specific mails, and simulate sending mail. It feels fairly limited, but is probably a good first stop.
== Database mail info ==
At the database level, mail info seems to be stored mostly in phabricator_metamta, and especially the table phabricator_metamta.metamta_mail.
describe metamta_mail;
Current database: phabricator_metamta
+--------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+------------------+------+-----+---------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| phid | varbinary(64) | NO | UNI | NULL | |
| actorPHID | varbinary(64) | YES | MUL | NULL | |
| parameters | longtext | NO | | NULL | |
| status | varchar(32) | NO | MUL | NULL | |
| message | longtext | YES | | NULL | |
| relatedPHID | varbinary(64) | YES | MUL | NULL | |
| dateCreated | int(10) unsigned | NO | MUL | NULL | |
| dateModified | int(10) unsigned | NO | | NULL | |
+--------------+------------------+------+-----+---------+----------------+
Of particular interest here are actorPHID, relatedPHID, and parameters.
Pretty much every entity in the database has a PHID. If looking for the mail related to a task, one can use a task PHID like so:
select count(*) from metamta_mail where relatedPHID = 'PHID-TASK-2iacqyb64oiku3uw35ct';
To retrieve the PHID for a task, you can use the conduit lookup here:
https://phabricator.wikimedia.org/conduit/method/maniphest.info/
parameters is a JSON blob containing various values, including stamps, which contains a list of recipients.
MySQL supports extracting JSON values like so:
select json_extract(parameters, '$.stamps') as stamps from metamta_mail where relatedPHID = 'PHID-TASK-2iacqyb64oiku3uw35ct';
...but an easier approach may be to use a brute force LIKE:
select count(*) from metamta_mail where relatedPHID = 'PHID-TASK-2iacqyb64oiku3uw35ct' and parameters like '%@username%';
Phabricator/Slowness

= Phabricator is Slow Troubleshooting =
Grafana dashboard for Phabricator
== Increased network traffic ==
Grafana link for performance data including network. Typical network traffic is between 3 and 5 Mbit/sec, a significant higher value (>10 Mbit/sec on Dec 15 2020 for example) can be an indication of higher than normal use and it is worth checking the apache access logs on the machine to see if there is a pattern that can be addressed, an IP or a certain distinctive User-Agent to ban.
Database traffic is usually correlated to high Phabricator usage.
 
== Apache Access Logs ==
Logstash link
The access log is in /var/log/apache2/phabricator_access.log. 400 MB/day is a typical size. Tailing and watching the file might give an indication what requests dominate. To look for repeat accesses: cat /var/log/apache2/phabricator_access.log | sed 's/@cee: //' | jq '."client.ip"' | sort -n | uniq -c | sort -n will give a list of IPs and count. Compare to cat /var/log/apache2/phabricator_access.log | wc to judge whether any number is to large. Example: On Dec 15 2020 we had about 560,000 accesses from 2 IPs that represented about 2/3 of all accesses and were responsible for a significant (3x) increase in network traffic.
cat /var/log/apache2/phabricator_access.log | sed 's/@cee: //' | jq '."user_agent.original"' | sort -n | uniq -c | sort -n does the same report for the user-agent. Example: On Dec 15 2020, we had 1,100,00 accesses from user-agent vscode-phabricator.
== Banning an IP or User-Agent ==
Best to use edge based banning: Requestctl IP addresses to ban are specified in requestctl/request-ipblocks/abuse/phabricator_abusers.yaml, User-agents are not implemented at the moment
Keywords: phab slow dos DDos phabbanlist
Phabricator/T373952

url 	taskTitle 	author 	assignee
phab:T43522 	Provide a log of actions which trigger the CAPTCHA 	He7d3r 	kostajh
phab:T223772	Extend #time parser function to display time in format specific to each language 	Jarekt 	tstarling
phab:T328754	IP reveal buttons not displaying on Safari 	Tchanders 	kostajh
phab:T333592	Gather user feedback about the temporary account IP access log 	Tchanders 	kostajh
phab:T344599	wikireplicas root access 	jbond 	fnegri
phab:T354542	Create new Rest API endpoint that takes an IP address and returns temporary accounts that have used that address 	Tchanders 	STran
phab:T365784	[S] Commons mobile view UploadWizard long translations breaking layout 	GPSLeo 	matthiasmullie
phab:T366271	Change Wikipedia: and Wikipedia_talk: namespaces for Mongolian (for Mongolian Wikipedia) 	chinneeb 	srishakatux
phab:T366485	Add Open Research Knowledge Graph SPARQL endpoint to allowed federated endpoints in Wikidata Query Service 	Fnielsen 	BTullis
phab:T369463	Add VideoWiki to translatewiki.net 	Hassan.m.amin 	Wangombe
phab:T369930	Update navigation between Special:Contributions and Special:IPContributions 	Tchanders 	kostajh
phab:T372392	Align the "current" marker with the edit summary on Special:Contribs in Minerva 	Tufor 	Jdlrobson
phab:T372615	[Android app] can't scroll all the way to bottom of source while editing, toolbar is in the way 	jeremyb-phone 	Dbrant
phab:T373028	Edit window freezes if the search query is 1 or 2 letters 	anoopspeaks 	Dbrant
phab:T373245	Table: Sortable column headers do not take up maximum available width 	Dogu 	CCiufo-WMF
phab:T373521	wfGetDB and ArgumentCountError on MediaWiki 1.42.1 	Danster2024 	Reedy
phab:T373763	Banners keep getting created which turn footer white in dark mode 	Kuldeepburjbhalaike	Pcoombe
phab:T373958	Requesting GitLab account activation for ChandraPratap25 	ChandraPratap25 	thcipriani
phab:T374199	Identities and Types are missing from the Object selector results so References to them cannot be used in new tests or implementations	GrounderUK 	gengh
phab:T374296	Create Diffusion Mirror the "Wikimedia CH Cassandra GLAM Tools" 	ValerioBoz-WMCH 	Aklapper
phab:T374389	[REQUEST] Assistance for proper implementation of Codex Dark Mode in Lexica 	Raisha_Abdillah 	egardner
phab:T374526	InvalidArgumentException: Invalid IP address error when loading IPInfo logs 	Dreamy_Jazz 	mszabo
phab:T374684	Unblock stuck global rename of IloveuFlyTek, Iosonopony, Mohamadanisahmad5, Monty.ch 	Syunsyunminmin 	DAlangi_WMF
phab:T374704	Create project tag for User-lens0021 	Lens0021 	Aklapper
phab:T374719	FlaggedRevs causing a lot of warnings for "SQL query did not specify the caller" 	Ladsgroup 	matmarex
phab:T374773	404 error opening a specific file on Commons (due to inconsistent state between two swift clusters) 	KTT-Commons 	MatthewVernon
phab:T374781	Responsive Web Design - The Viewport 	Gerges 	Krinkle
phab:T374796	Phabricator Form 102 header has a 404 URI anchor link 	JayCubby 	Reedy
phab:T374797	Server side upload for Koavf 	Koavf 	TheresNoTime
Phabricator/pushing over https

== How to git push over https to repos on Phabricator Diffusion ==
If you have a git repository [Phabricator (Diffusion)], this is how you can push changes over https (instead of ssh).
=== Set a VCS password ===
First, you are going to set a 'version control password' in your Phabricator profile. This is a "once only" global setting, not per repo.
Login, click your profile icon in the upper right corner, -> Settings -> Authentication -> VCS Password.
Because this is security-related change you will now enter "High Security Mode".
You will be asked for an "App Code". Enter a 2-factor code from your authentication app (Google Authenticator, Authy).
Now you set the VCS password to something random, separate from your user password and keep it in your password store. 
You can now leave "High Security Mode" as well.
=== Clone repo and push ===
Next go to 'Code' -> 'Source Repositories' which brings you to Diffusion.
Click 'Hosted on Phabricator' and find your repository in the list, click on it and then the "clone" button.
If you see multiple URIs copy only the https:// URI, ignore ssh:// and http:// URIs.
On your local computer run:
 git clone <paste link from above>
Now make changes as normal and 'git add ..', 'git commit' as usual.
Once you 'git push' you will be asked for your username and password. Enter your Phabricator username, and the VCS password you setup before.
~/httpspush$ git push
Username for 'https://phabricator.wikimedia.org': Dzahn
Password for 'https://Dzahn@phabricator.wikimedia.org': 
Hit enter and now changes should be pushed.
=== Afterwards ===
Once you confirmed this works for you please tell User:Dzahn, ideally via phab:T296022, that you are done and the name of the repo.
He will then deactivate the http:// and ssh:// URIs for your repo.
Phawikibugs

Phawikibugs is an instance of the Phabricator Chat Bot running under the wikibugs account on Portal:Toolforge. It uses the pywikibugs user account to communicate with Phabricator.
=== Starting and stopping ===
The bot is written as a Phabricator Daemon, which means we need a wrapper to run it on the Grid:
run_phab_bot.sh
#!/bin/bash
# because phabricator uses SIGUSR1 to communicate between child and parent, we need to catch this...
cd /data/project/wikibugs/phabricator
function on_EXIT()
{
 echo Stopping all daemons...
 phabricator/bin/phd stop
 echo Done. Bye-bye!
}
function on_USR1()
{
 echo Received SIGUSR1
}
trap on_USR1 USR1
trap on_EXIT EXIT
echo Starting daemon via PHD!
phabricator/bin/phd launch phabricatorbot `pwd`/phab_config.json
echo Started. Sleep for ALL times!
sleep infinity
#echo Attempting to EXEC daemon
#./libphutil/scripts/daemon/exec/exec_daemon.php PhabricatorBot --load-phutil-library=`pwd`/arcanist/src --load-phutil-library=`pwd`/phabricator/src --verbose -- `pwd`/phab_config.json
The helper script run_grid.sh submits the job to the grid:
#!/bin/bash
jstart -once -mem 1G ./run_phab_bot.sh
=== Configuration ===
In phab_config.json. Current configuration:
{
 "server" : "irc.freenode.net",
 "port" : 6667,
 "nick" : "phawikibugs",
 "join" : [
 "#mediawiki-feed",
 "#wikimedia-devtools"
 ],
 "handlers" : [
 "PhabricatorBotFeedNotificationHandler"
 ],
 "conduit.uri" : "https://phabricator.wikimedia.org",
 "conduit.user" : "pywikibugs",
 "conduit.cert" : "[redacted]",
 "macro.size" : 48,
 "macro.aspect" : 0.66,
 "notification.channels" : ["#phabot-test"]
}
(I think notification.channels is ignored?)
=== Status ===
In development. The basic bot works (as provided by Phabricator), but it needs WMF-specific adjustments. See the task list for details.
Category:IRC