nyuuzyou/wikis huggingface dataset:
Url-downloader

url-downloader is a Squid non-caching proxy for HTTP requests. It was originally created for MediaWiki's upload-by-URL functionality, these days it is being used by any service that wants to access resources outside the boundaries of the wikimedia production realm. It is not a highly available service and as such no assumptions should be made in code regarding availability.
== Service ==
: See also url_downloader in Puppet.
* For eqiad: Hosted on urldownloader100[34].wikimedia.org
* For codfw: Hosted on urldownloader200[34].wikimedia.org
In both datacenters only one url-downloader is active at any time, and is pointed at via the DNS record for url-downloader.$DC.wikimedia.org, and exposed at url-downloader.$DC.wikimedia.org:8080.
url-downloader.wikimedia.org:8080 points to eqiad for backward-compatibility.
In MediaWiki configuration, it would be typically obtained via $wmgLocalServices['urldownloader'] (which also provides an appropriate proxy URL on beta).
== See also ==
* HTTP proxy (another set of squid proxies for slightly different use cases)
* T254011: Why do we have 2 sets of squid proxies?
Category:Services
User-agent client hints

Client Hints are information about your device and browser which is sent on request to websites you visit by browsers such as Google Chrome and Edge. The extension requests and collects this data by default to support the community in reducing vandalism and abuse. This data is deleted after 90 days and can only be viewed by trusted users. For more high-level information, see mw:Extension:CheckUser/Client Hints. 
Collection of client hint data on all wikis began on August 22, 2023. 
=== Configuration ===
* wgCheckUserClientHintsEnabled , set to false to disable collection and storage of client hints data on a wiki
See mw:Extension:CheckUser#Configuration_2 for other configuration settings related to client hints.
=== Technical details ===
* on non-Special pages, we load the ext.checkUser.clientHints ResoureLoader module, which listens to the MediaWiki postEdit JavaScript hook. 
** After a client makes an edit, if it supports the client hints JavaScript API, it will POST the client hint data to w/rest.php/checkuser/v0/useragent-clienthints/{type}/{id}. 
** The UserAgentClientHintsHandler class in CheckUser stores the values in cu_useragent_clienthints and cu_useragent_clienthints_map. The cu_useragent_clienthints_map table links client hint data with a revision, or to a CheckUser log event.
** The purgeOldData.php maintenance script in CheckUser runs via puppet (code) and ensures that data older than 90 days is removed.
* on certain Special pages (see CheckUserClientHintsSpecialPages) and content pages with an action query parameter (see CheckUserClientHintsActionQueryParameter, currently it is used for ?action=history because users can perform one-click rollback edits from this UI), MediaWiki will request client hints from the client using a server header. On the subsequent request, the client will send client hint data, and CheckUser ingests these values updateCheckUserData
**CheckUser by default sends an empty Accept-CH header field; this instructs browsers to not send client hint data by default. We need to do this, because once you request client hint data with Accept-CH header, the browser will continue to send this data for the duration of the user's browsing session, and we only collect this information in specific scenarios (performing a rollback, creating an account, emailing a user, resetting a password, etc)
=== Resources ===
* Epic for implementing client hints functionality [EPIC] Support User-Agent Client Hints header in CheckUser (T257893)
* Phab project for related tasks 
* Technical contacts: User:Dreamy Jazz, User:Kosta Harlan
* Project plan (WMF only)
* WMF Slack #chrome-ua-deprecation
VRT System

Znuny, a community fork of OTRS which went closed source in version 7.x, forcing us to adopt Znuny, is installed on vtrs1001.eqiad.wmnet. 
== Terms ==
* VRT stands for Volunteer Response Team. 
** They use the platform to read and answer emails to a variety of incoming destinations, e.g. info-<language>@wikipedia.org, permissions-<language>@wikipedia.org, etc.
** Public details are at m:Volunteer Response Team
** Their private wiki is at vrtwiki:
* VRTS stands for Volunteer Response Team System. We currently use software by Znuny GmbH.
** Znuny stands for the software released by Znuny GmbH. It's a community fork of OTRS community edition which went belly up in 2021 after being closed source since version 7.x
** OTRS was how we previously referred to the software and team. Remnants remain. (T280392 - migration task)
* VRTS admins, volunteers who administer the queues and the agents. (~6 people)
* VRTS agents, individuals of that team. (~400+ people)
== Notes ==
* Public documentation for general public interacting with VRT members: https://meta.wikimedia.org/wiki/Volunteer_Response_Team
* URL is https://ticket.wikimedia.org
* The root user/pass is in the ops password repo
* We use mod_perl with ModPerl::Registry, so whenever a file is changed, an apache2 reload is required.
*The "News" messages on the main Znuny login screen can be editing by modifying /opt/otrs/Kernel/Output/HTML/Templates/Standard/Motd.tt.
There is no need to update config files to add email addresses to the system; Inbound MX servers will automatically see that the queue exists or has disappeared. However it is possible (due to negative caching at the secondary mail exchangers) that new addresses will take up to two hours to begin working.
== Backups ==
Our Znuny installation is almost fully database only. That is, all data is stored in a mysql database (m2 shard), and only configuration and code is stored locally on the hosting server. Znuny is open source and hence almost impossible to lose the code and most of the configuration is stored in puppet. There are a few configuration items that are stored locally on the server but this is temporary and those are in the end transferred to puppet. Hence we only care about database data being safe and backed up.
Database is regularly backed up once per week (on Wednesday currently). The infrastructure used is Bacula and most documentation from that page applies. The code doing the pre dump is in https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/role/templates/mariadb/backups/dumps-otrs.sh.erb, and bacula just backs up the resulting file.
Restoring at a previous point in time is quite easy and all it takes is restore the dump from bacula (covered in Bacula) and applying it to the db server via the mysql command. Restoring individual items (like an article being deleted) is possible but quite complicated and difficult and requires a DBA to help isolate the specific transaction and avoid replaying it while replaying logs between the last backup and the time of the incident. It has never been done, nor required up to now.
VRT System/Configuration

The source is being expanded in a version dependant directory under /opt, e.g. /opt/znuny-6.0.37.
A symlink is used to point to that then, e.g. /opt/otrs -> /opt/znuny-6.0.37. That symlink is updated at every upgrade.
The primary configuration file is /opt/otrs/Kernel/Config.pm
Configuration can be done either via the Web interface or by changing Kernel/Config.pm and shipping the change via puppet. The file overrides whatever changes happen via the Web Interface. When you change the config, reload apache to clear the mod_perl cache.
Znuny configuration is meant to be primarily done via the Web interface. Stick to that for most use cases.
VRT System/Customization

== Local Customizations ==
=== Packages ===
The codebase has some local customizations, which are applied as Znuny packages. The packages are stored in the database in otrs.package_repository and can be reapplied after update via the web UI or the command line The packages can also be downloaded from within the web interface, when logged in as an admin user:
 Admin-->Package Manager-->{packages listed under Local Repository}-->Download
via the command line, packages can be reapplied with:
 sudo -u www-data /opt/otrs/bin/otrs.Console.pl Admin::Package::ReinstallAll
The packages are in gerrit in the operations/software/otrs repo. Their structure is similar to the way the VRTS source code is structured. The reason for that is because OTRS/Znuny packages can override a "proper" file. e.g. 
* Custom / Kernel / Output / HTML / Templates / Standard / AgentDashboardUserOnline.tt
will override 
* Kernel / Output / HTML / Templates / Standard / AgentDashboardUserOnline.tt
That makes it a bit harder then just maintaining as patches as we have to keep the entirety of the files in sync with upstream and not just the differences (which are arguably very small). Maintaining consists of fetching the new files the package overrides per version and manually merging the new upstream stuff.
=== Building ===
Installing the packages requires that they are first built. Building the packages happens on the command line using:
 cd /tmp
 git clone https://gerrit.wikimedia.org/g/operations/software/otrs
 cd otrs
 sudo -u otrs /opt/otrs/bin/otrs.Console.pl Dev::Package::Build --module-directory packages/WikimediaTemplates/ packages/WikimediaTemplates/WikimediaTemplates.sopm /tmp
In the command above, the module directory is just a checkout of the repo mentioned above. /tmp is the output directory and the path to the .sopm file is so the code is bundled alongside the description. The .sopm file is an XML file and the format is pretty self-explanatory. 
=== Installing ===
Installing the package happens using:
 sudo -u www-data /opt/otrs/bin/otrs.Console.pl Admin::Package::Install /tmp/WikimediaTemplates-1.0.18.opm
The version (1.0.18 in the example above) will need adjustment per package version and is defined in the .sopm file above.
=== Errors ===
During installation of a custom package, you may encounter errors to do with the Repository setting. The following steps can be taken to resolve them:
 sudo -u otrs /opt/otrs/bin/otrs.Console.pl Maint::Config::Rebuild --cleanup
 sudo -u otrs /opt/otrs/bin/otrs.Console.pl Admin::Config::Update --setting-name Package::RepositoryList --valid 1
 sudo -u otrs /opt/otrs/bin/otrs.Console.pl Admin::Config::Read --setting-name Package::RepositoryList
 sudo -u otrs /opt/otrs/bin/otrs.Console.pl Admin::Config::Update --setting-name Package::RepositoryList --valid 0
VRT System/Failover

VRTS has one active host (currently vtrs1001) and one replica (vrts2001). 
== Introduction ==
For some time now, VRTS has had only one primary server in eqiad (vrts1001). Failing over the system is not a very straight-forward process. For one the database is very huge thus backing it up and restoring it could potentially take days and secondly the database shares a section with other unrelated services so it would potentially mean having to failover those services as well which is somewhat of an inconvenience for everyone involved. So this guide is only to be used in cases where there is some maintenance work in eqiad that forces us to switch traffic to codfw (vrts2001) at least temporarily until services are restored in eqiad.
== Prerequisites ==
The host to failover to should be a proper VRTS replica, meaning:
* is running the puppet role(vrts)
* has the same files as the primary in /opt. There is currently a rsync setup and can be run using sudo /usr/bin/rsync --rsh /usr/local/sbin/sync-vrts-ssl-wrapper -av --progress rsync://vrts1001.eqiad.wmnet/vrts /opt/
* check for a fresh backup MariaDB/Backups#Dashboard
This is mostly already done for the currently configured replica vrts2001.
== Planned Failover ==
A planned failover means the old production instance is responding and working properly. The following steps are needed to failover to a new host:
* Log in with an admin account to the VRTS dashboard and schedule new system maintenance for when you plan to do the failover. This can be done from Admin -> System Maintenance. This is important as one of the critical things we have to try and ensure during a failover is that no one is writing to the database. Maintenance mode ensures that only admins can login to the system and this goes a long way in reducing the number of people actively using the system and we can easily inform admins to not perform any critical tasks during a failover.
* Prepare DNS patch: In the DNS repo, open the wmnet template and change the record that ticket points to. This is under the "misc services without multiple backends section".
* Ensure your new host is listed as the active_host in the hieradata/role/common/vrts.yaml file. Since there are only two hosts, you can just invert the values of active_host and passive_host.
== Unplanned Failover ==
An unplanned failover means the old production instance is not responding/lost.
VRT System/Installation

== Installation ==
Setting up VRTS is a partly automated and manual process. The first step is to apply the Puppet role to the server that you'd like to turn into a VRTS server in site.pp in the Puppet repo i.e.
node 'otrs1001.eqiad.wmnet' {
 role(vrts)
}
This is also after ensuring that the values in hieradata exist i.e. the VRTS database user, password, host and the exim database values as well. This file is located in hieradata/role/common/vrts.yaml. If running the database on the same host as the VRTS installation (like we are currently with the VRTS instance on WMCS), you can use the following scripts to provision the database user and schema.
If running a local MySQL database, edit the /etc/my.cnf file and alter the following values and restart the mariadb service:
max_allowed_packet = 64M
innodb_log_file_size = 512M
-- VRTS
-- Create Database
CREATE DATABASE IF NOT EXISTS vrts;
-- Create Database User
CREATE USER '<vrts_user_in_hieradata>'@'localhost'
 IDENTIFIED BY '<password_in_hieradata>';
-- Grant Database User Permissions
GRANT ALL ON vrts.* TO 'vrts'@'localhost';
-- Set Database Character Set Encoding (this configuration is pretty important or the migrations will not work)
ALTER DATABASE vrts CHARACTER SET utf8 COLLATE utf8_unicode_ci;
-- Mail Database
CREATE DATABASE IF NOT EXISTS exim;
-- 
GRANT ALL ON exim.* TO 'vrts'@'localhost';
ALTER DATABASE exim CHARACTER SET utf8 COLLATE utf8_unicode_ci;
After provisioning this, navigate to the host's public URL https://vrts-prod.wmcloud.org/otrs/installer.pl. This will open the graphical VRTS installation wizard. Initially, this will display an error about SecureMode being active on the server. To fix this, you will temporarily have to disable Puppet on the host and edit the /opt/otrs/Kernel/Config.pm file to disable SecureMode. This parameter is currently on line 97 of this file; set it to 0 like so:
$Self->{SecureMode} = 0;
After performing these steps, restart the web server and vrts deamon on the host with the following:
sudo service restart apache2
sudo service restart vrts-daemon
The error about SecureMode being active should be gone and the installation will allow you to proceed. It will prompt you to provide information about the database (username, password and host) and email server settings. Enter the values as configured in the first steps earlier when creating the database stuff.
VRT System/SSL

To create SSL keys you can easily follow the steps here (https://wikitech.wikimedia.org/wiki/Cergen) but with the following exceptions:
* The template you create must not have a password defined as this will result in an encrypted key and envoy proxy will not be able to use it.
* To resolve the above, do not include a password in your template. A sample template is shown below:
ticket-test.discovery.wmnet:
 authority: puppet_ca
 expiry: null
 alt_names: ["name.example.com", ...]
 key:
 algorithm: ec
* After generating the keys, upload the public key to puppet in the SSL module. Example change (https://gerrit.wikimedia.org/r/c/operations/puppet/+/959272)
VRT System/Troubleshooting

==Troubleshooting ==
===Mail delivery===
*When user/group for the exim pipe were incorrect, otrs.Postmaster.pl logged permissions to /var/log/mail.log about errors on attempts to write in /opt/otrs/var/tmp/CacheFileStorable. We fixed this by configuring the exim pipe to use group=www-data.
*Mail hosts need mysql access to the otrs database. If mx IP addresses change or the database is inaccessible, mail defers on whichever mx is trying to do an address lookup. When we saw this happen, exim wasn't very informative about why. When in doubt double-check mysql access from the mx's command line.
*Spamassassin runs locally, and logs in /var/log/mail.log.
===Apache permissions errors===
*as user otrs run:
/opt/otrs/bin/otrs.SetPermissions.pl --otrs-user=otrs --otrs-group=otrs --web-user=www-data --web-group=www-data /opt/otrs
===SpamAssassin stops reporting Bayes results===
*This happened 2014-04-24, 2016-08-06, 2016-12-21 and we discovered it was unhappy about the bayes database.
*/var/log/syslog was full of this:
bayes db version 0 is not able to be used, aborting! at /usr/share/perl5/Mail/SpamAssassin/BayesStore/DBM.pm line 203, <GEN88>
*We tried backup/restore the database (the verify failed), and the database shrank from ~24M to ~14M and SA stopped complaining. But SA continued to pass through mail with no Bayes results (the BAYES_XX where XX in [00,90] Header added to the message was missing)
*So we moved aside the old database, modifed otrs.TicketExport2Mbox.pl not to skip previously-seen messages, and created one-time GenericAgent [within OTRS] jobs to re-export a couple of days worth of ham/spam. Then we ran train_spamassaassin manually to train on all this data. Note otrs.TicketExport2Mbox now has --rebuild mode to support this process.
*On the 2016-08-06 incident the log statement below was found in the logs:
 Aug 6 09:56:59.752 [1619] dbg: bayes: not available for scanning, only 126 ham(s) in bayes DB < 200
after running a:
 sudo -u debian-spamd spamassassin -D bayes < /tmp/sample_email.eml
and a 
 sudo -u debian-spamd sa-learn --dump magic
confirmed it.
The fix was the re-exporting and training of spamassassin as mentioned above. Extra care should be take to make sure the spam/ham messages exported are above 200 in every case. 
*On the 2016-12-21 incident both the hams and the spams in the database were below 200. That was not logged however as the message about the hams above, leading the investigation off track for a while. Exporting quite a few messages and training spamassasin on them as above fixed the issue. The database in this case was NOT being marked as corrupted by db_verify but it was truncated in the end just for good measure manually
VRT System/Upgrading

==Upgrading==
This section is general guidance for a patchlevel (e.g. 6.0.x -> 6.0.y) update only. 
Minor or Major version upgrades can be complicated by database schema changes and other issues. 
There's really no way around reading the upgrade documentation, 
and testing the updates on a real system based on our existing configuration and database.
#Fetch new VRTS, extract into /opt and read through any upgrade instruction:
#* :~# curl -L -O https://download.znuny.org/releases/znuny-$VERSION.tar.gz
#* :~# sudo tar xfz znuny-$VERSION.tar.gz -C /opt
#* https://doc.znuny.org/legacy/manual/admin/6.0/en/html/updating.html (as a sanity check, read through the upgrade instructions)
# Stop Puppet, Apache, Exim, and vrts-daemon & cron:
#* :~# disable-puppet "VRTS Upgrade"
#* :~# service apache2 stop
#* :~# service exim4 stop
#* :~# service cron stop
#* :~# service vrts-daemon stop
# Copy old config into new code tree & fix permissions:
#* :/opt# sudo cp /opt/otrs/Kernel/Config.pm /opt/znuny-$VERSION/Kernel
#* :/opt# sudo cp /opt/otrs/var/log/TicketCounter.log /opt/znuny-$VERSION/var/log/
#* :/opt# ./otrs/bin/otrs.SetPermissions.pl --otrs-user=otrs --otrs-group=otrs --web-user=www-data --web-group=www-data /opt/otrs
# Fix symbolic link, permissions and rebuild and install packages:
#* :/opt# sudo ln -sfn /opt/znuny-$1 /opt/otrs
#* :/opt# sudo -u root /opt/otrs/bin/otrs.SetPermissions.pl --web-group=www-data
#* :/opt# sudo -u otrs /opt/otrs/bin/otrs.Console.pl Maint::Config::Rebuild
#* :/opt# sudo -u otrs /opt/otrs/bin/otrs.Console.pl Maint::Cache::Delete
#* :/opt# sudo -u www-data /opt/otrs/bin/otrs.Console.pl Admin::Package::ReinstallAll
#Restart Apache, Exim, vrts-daemon, cron & Puppet:
#* :/opt# sudo service cron start
#* :/opt# sudo service exim4 start
#* :/opt# sudo service apache2 start
#* :/opt# sudo service vrts-daemon start
#* :/opt# enable-puppet "VRTS Upgrade"
#Run Puppet:
#* :/opt# puppet agent -tv
# After performing the upgrades, sync the replica to match the files on the primary:
#* aokoth@vrts2001:~$ sudo /usr/bin/rsync --rsh /usr/local/sbin/sync-vrts-ssl-wrapper -av --progress rsync://vtrs1001.eqiad.wmnet/vrts /opt/
#Test functionality:
#*Send a mail to e.g. info-en and check that it shows up in OTRS
Vanadium

vanadium was a server in eqiad that was the log collector for EventLogging generated by event.gif requests to the bits Varnish servers. It was decommissioned 2015-04-09.
It also collected logging events from server-side PHP code.
It also counted MediaWiki errors on the cluster for graphing in ganglia (see Wikimedia Grid > Miscellaneous eqiad > vanadium.eqiad.wmnet : mediawiki metrics
It was also (December 2012) used for Translate's translation memory through Solr.
Verify-copy.sh

You probably want to use compare.py instead and do a full verification.
A script comparing ES with metadata would be nice (does that exist?).
#!/bin/bash
#
# script to compare random samples from two external store databases and report whether they match
# asks one db for the rowcount
# generates 2000 random numbers within the rowcount range
# selects the blob with the random ID, md5sums it from both DBs
# compares the md5sum
# reports how many match / fail
#
# example usage:
# ./verify-copy.sh 10.0.2.158 yowiki blobs es1002.eqiad.wmnet yowiki blobs_cluster10
# passwords
mysqluser="wikiadmin"
mysqlpass="xxxxxxxxx"
# get the database name and the two hostnames from the command line
hosta=$1
dbnamea=$2
tablea=$3
hostb=$4
dbnameb=$5
tableb=$6
# print help
if [[ -z "$hosta" || $hosta == '-h' || $hosta == '--help' ]]
then
 echo "Compares random entries in the external database tables."
 echo "Example usage:"
 echo " ./verify-copy.sh 10.0.2.158 yowiki blobs es1002.eqiad.wmnet yowiki blobs_cluster10"
 exit 0
fi
## find out how many rows we've got
num_rows=$(mysql --batch --skip-column-names -u $mysqluser -p$mysqlpass -h $hosta $dbnamea -e "select count(*) from $tablea;")
## select 2000 random rows (in 10 groups of 200 rows) and md5sum them, compare, record status
num_failed=0
num_succeeded=0
for j in {1..10}
do
 idlist='1'
 for i in {1..200}
 do
 id=$RANDOM
 let "id %= $num_rows"
 idlist="${idlist},$id"
 done
 suma=$(mysql --batch --skip-column-names -u $mysqluser -p$mysqlpass -h $hosta $dbnamea -e "select blob_text from $tablea where blob_id in ($idlist)" | md5sum | cut -f 1 -d\ )
 sumb=$(mysql --batch --skip-column-names -u $mysqluser -p$mysqlpass -h $hostb $dbnameb -e "select blob_text from $tableb where blob_id in ($idlist)" | md5sum | cut -f 1 -d\ )
 if [ $suma == $sumb ]
 then
 let "num_succeeded += 1"
 else
 let "num_failed += 1"
 fi
done
echo "matched: $num_succeeded, failed: $num_failed"
Category:MySQL
Virt-install

== MAC Address ranges ==
The KVM MAC address range is: 52:54:00
The Xen MAC address range is: 00:16:3e
== Using virt-install ==
virt-install --connect qemu:///system -n <vmname> -r <amountOfMemory> --arch=<architectureType> --os-type=<typeOfOS> \
--hvm --virt-type kvm --accelerate --pxe --disk /a/images/<vmname>.qcow2,size=<size>,format=qcow2 --network=bridge=br0,mac=<MAC>