nyuuzyou/wikis huggingface dataset:
Analytics/Archive/Webstatscollector

:More current information may be available (or not) at Analytics/Pageviews.
Webstatscollector was a set of services that computed unsampled hourly per page pageviews from udp2log. It powered pagecounts-raw until 2015-01-01. Webstatscollector itself got turned off on 2015-01-29.
The output is made available through pagecounts-raw at dumps.wikimedia.org, and ends up powering for example http://stats.grok.se/, and parts of http://stats.wikimedia.org/.
== Architecture ==
Webstatscollector is made up of two separate processes:
* filter, and
* collector.
filter is run by udp2log (currently on oxygen). All udp2log logs are piped through filter. filter has hardcoded url matches that it will not count as pageviews.
The output of filter is then piped into log2udp which sends logs over to the collector process on gadolinium.
As of 2014-08-27 collector is continuously writing data to the disks at about 15MB/s. Since that puts stress on disks, collector is run in a tmpfs directory, so it is writing to tmpfs and does not stress disks. That tmpfs directory's “dumps” subdirectory (that's where the aggregated, hourly files get written to) symlinks to a real disk, so the aggregated, hourly files survive a reboot.
As of 2014-09-02 the collector's per-page Berkeley database grows up to an on-disk size of ~1GB for the busy hours. The per-project database is orders of magnitude smaller. 
=== Relevant sources ===
* filter is built from webstatscollector's filter.c
* collector is built from webstatscollector's collector.c
* log2udp is built from udplog's srcmisc/log2udp.ccp
== Updating and Deploying a New Version ==
webstatscollector is puppetized and installed using a (hacky) .deb package.
Build a new package by running:
 $version=0.2
 rm -f ../webstatscollector_$version.orig.tar.gz && tar -cvzf ../webstatscollector_$version.orig.tar.gz . && debuild -us -uc
Put this .deb into our apt repository following the instructions here: https://wikitech.wikimedia.org/wiki/Reprepro#Importing_packages.
Then, upgrade the package and restart the binaries:
On oxygen:
 apt-get update
 apt-get install webstatscollector
 # restart udp2log with the new filter binary
 service udp2log restart
On gadolinium:
 apt-get update
 apt-get install webstatscollector
 # restart the collector process
 service webstats-collector restart
Done!
== Used Page View definition ==
The flow diagram describes at https://phabricator.wikimedia.org/diffusion/ANME/browse/master/pageviews/webstatscollector/pageview_definition.png illustrates the used pageview definition.
As webstatscollector throws away requests coming from WMF internal IP addresses, it needs to be fed the requests arriving at the SSL endpoints to count SSL traffic.
Requests to mobile sites get aggregated across projects. So “en.mw" does not refer to “PageViews to mobile site of enwiki”, but “PageViews of mobile sites of counted english wikis”. So requests to each of http://en.m.wikivoyage.org/wiki/Main_Page, and http://en.m.wikipedia.org/wiki/Main_Page will get counted towards en.mw.
Annual Toolforge Survey/Opt out

The Cloud Services Annual Survey is conducted once per year to help the Wikimedia Cloud Services team get a sense of how their efforts to support Wikimedia technical contributors are being received. Announcements of each iteration of the survey are sent via email to Toolforge users and Cloud VPS project admins who:
* have a confirmed email address
* have not explicitly asked to be excluded from future surveys
To opt-out of future survey announcement emails:
* Add your Wikitech username to the list below between the <!-- BEGIN OPT-OUT LIST --> and <!-- END OPT-OUT LIST --> comments
* Usernames should be one per line
* Only use a bare username, not a link to your User page or User Talk page
If you want to unsubscribe confidentially: Update your email preferences on your wikitech account by removing your email address or unclicking "Allow other users to email me". If you wish to allow other users to e-mail you, but not to receive survey e-mails: this is not presently possible.
== Toolforge survey email opt-out list ==
<!-- BEGIN OPT-OUT LIST -->
CoolCanuck
Dachary
Bluegoblin7
Jackson Peebles
Catchpoint-monitor
Smith609
Nova Tools Bot
Nutz
Rschen7754
Steini
राजा उप्रेती
Alex Monk (Test 8)
Alex Monk (Test 9)
Demo Unicorn
Fredddie
Bamyers99
LoganDark
Czar
Hym411
Wmflabsdotorgadmin
Neutron
Striker test 2017-08-23
Striker test 2017-08-28
Meta Monitoring Canary
Osstackcanary
Ruhrfisch
Graham87
Sau226
Ineuw
Erfaan
Foks
Owner of abandoned tools
Taweetham
Tools-dns-manager
Hassen.Houssein
Katia
ZI Jony
GTirloni
Labdajiwa
aline bruenger WMDE
Niedzielski
Hjfocs
Elgei
Lahi
Stang
Psubhashish
<!-- END OPT-OUT LIST -->
Anycast authoritative DNS

Work in progress.
In order to improve latency and resilience of our authoritative DNS, this setup leverages BGP and anycast.
Tracking task: https://phabricator.wikimedia.org/T98006
== Limitation of a non-anycast setup ==
By definition, GeoDNS can't be used to redirect users to their closest nameserver (NS), like we do for websites.
When asked for a record (eg. fr.wikipedia.org), the .org zone presents all 3 of our NS to the client, to decide which one to use.
Client side implementations not being great , anycast offloads that decision to BGP.
== Configuration ==
=== Server side ===
The server side is a regular internal anycast setup.
modules/profile/manifests/dns/auth.pp and modules/profile/manifests/dns/recursor.pp include ::profile::bird::anycast
hieradata/role/common/dnsbox.yaml and hieradata/role/common/dns/auth.yaml
profile::bird::advertise_vips:
 nsa.wikimedia.org:
 address: 198.35.27.27 # VIP to advertise (limited to a /32)
 check_cmd: '/usr/lib/nagios/plugins/check_dns_query -H 198.35.27.27 -a -l -d www.wikipedia.org -t 1'
 ensure: present
 service_type: authdns
=== Routers side ===
Policy to only create (and thus advertise) the /24 anycast prefix if the router learns about it locally.
policy-options {
 policy-statement BGP_from_anycast {
 term BGP_local_anycast {
 from {
 protocol bgp;
 as-path local_anycast;
 }
 then accept;
 }
 then reject;
 }
 as-path local_anycast "^64605$";
}
routing-options {
 aggregate {
 route 198.35.27.0/24 policy BGP_from_anycast;
 }
}
== Troubleshooting ==
=== Know which server a client is routed to ===
$ dig +nsid @nsa.wikimedia.org en.wikipedia.org A |grep NSID
== Failure tests ==
=== Total local AuthDNS failure ===
# Stop gdnsd on all ulsfo servers
# The anycast prefix stops being advertised to the routers
# The routers don't have any contributing routes to the less specific prefix
# The routers stop advertising the prefix to their peers
# Start gdnsd back
# prefixes are re-advertised
== Limitations ==
* L3 header LB: ICMP packet too big sent by routers along the path will not consistently be router to the correct router
* Non-consistent hashing: if a routing change on the Internet causes a TCP packet to arrive through a different router, the router will not consistently route it to the proper server
Anycast recursive DNS

In order to improve resiliency of recursive DNS, this setup leverages BGP and anycast.
Task: https://phabricator.wikimedia.org/T186550
CR: https://gerrit.wikimedia.org/r/c/operations/puppet/+/397723/
== Limitation of a non-anycast setup ==
* Some services don't fail over fast enough to the 2nd server listed on resolv.conf when one fails
* If the two servers of a site (or the whole site) fails, servers relying on them will experience an outage
* LVS/pybal depends on DNS and thus making it a chicken/egg problem
== Configuration ==
=== Server side ===
modules/role/manifests/dnsrecursor.pp  include ::profile::bird::anycast
hieradata/role/common/recursor.yaml (global)
profile::bird::advertise_vips:
 recdns.anycast.wmnet:
 address: 10.3.0.1 # VIP to advertise (limited to a /32)
 check_cmd: '/usr/lib/nagios/plugins/check_dns_query -H 10.3.0.1 -l -d www.wikipedia.org -t 1'
 service_type: recdns
==== check_cmd ====
In this case we re-use an Icinga NRPE check, installed on all the servers:
/usr/lib/nagios/plugins/check_dns_query -H 10.3.0.1 -l -d www.wikipedia.org -t 1
== Troubleshooting ==
=== Know which server a client is redirected to ===
$ dig @10.3.0.1 CHAOS TXT id.server. +short
=== Ensure all servers can reach the VIP ===
bblack@cumin1001:~$ sudo cumin '*' 'dig @10.3.0.1 CHAOS TXT id.server. +short'
== Failure tests ==
=== Single local recursor failure ===
bblack@backup2001:~$ while [ 1 ]; do echo ======; date; dig @10.3.0.1 CHAOS TXT id.server. +short; sleep 1; done
"so I can see a result once a second, I've tried stopping just healthchecker, stopping or killing the recursor, etc"
Traffic routes to the one working local node within the second.
=== Double local recursor failure ===
Eg. take down dns2001/dns2002
Request end up on dns1001/dns1002
== Limitations ==
*If the DNS recursors have the anycast VIP as only resolver in resolv.conf, then processes depending on DNS will fail until pdnsd starts as they will try to connect to the local recdns service instead of being routed to the closest server.
**Workaround is to either hardcode real recursors IPs or have a daemon that remove the VIP loopback
== Future evolution ==
*Add Icinga monitoring to check local recursors work (eg. Icinga check on bastX hosts that check it's dnsX that reply and not dnsY)
Category:Runbooks
Apache log format

The WMF Apache access log format is derived from the cache log format. It has tab-separated fields.
Field#	Description 	Details
1 	time stamp 	The time at which the request started, in ISO 8601 format, according to the server's clock.
2 	Request service time 	Integer number of microseconds
3 	Client IP 	You might want to override this with the original client IP (field 17 below).
4 	Request handler, HTTP status code 	The Apache request handler, then a slash, then the HTTP status code, e.g. cgi-script/200
5 	Response size 	Size of response in bytes, excluding HTTP headers.
6 	Request method 	GET, POST, etc
7 	URL 	The URL, concatenated from the host header, the path and the query string. The protocol is always http, even if mod_ssl was used. In theory we could use SetEnvIf with "SSLOptions StdEnvVars", but the latter has a high performance overhead, according to the documentation.
8 	Empty 	For compatibility with the cache log format, this field is always just a hyphen.
9 	MIME content type 	The Content-Type response header
10 	Referer header 	
11 	X-Forwarded-For header 	This field should only be respected if the client IP (see above) is a known good IP (i.e.: in $wgSquidServersNoPurge of wmf-config/squid.php of operations/mediawiki-config).
12 	User-Agent header 	
13 	Accept_Language 	
14 	X-Analytics 	General purpose header for instrumentation, which received its own page at X-Analytics.
15 	User 	The username as determined by HTTP authentication
16 	User: Response header 	The username as determined by the server and added as the HTTP User: header. Since 2019-03-20
17 	Original client IP or if local Envoy proxy, the local IP	The original client IP as determined by mod_remoteip, which can walk X-Forwarded-For headers. This module currently needs to be configured on a per-site basis to give good results. If the module is not present, this will be a copy of field 3.
18 	Request ID 	Internally we generate a request ID header, X-Request-Id, that should allow to trace a request across logs of various services. Since 2020-02-03
19 	Original client IP 	The original client IP as determined by mod_remoteip, which can walk X-Forwarded-For headers. This module currently needs to be configured on a per-site basis to give good results. If the module is not present, this will be a copy of field 3.
A JSON version (called wmfjson in code) derived from this format is used for logs consumed by logstash.
Aphlict

Aphlict is a realtime-notification service used by Phabricator.
It uses web sockets to establish connections between end user clients and the Phabricator webserver.
This enables it do things like sending notification pop-ups and moving items on dashboards without users having to reload pages.
It was once part of the Phabricator servers like phab1001 but has since been moved to its own dedicated VM running on Ganeti.
The current VM is called aphlict1002.eqiad.wmnet and lives in eqiad.
phab:T322369 is a ticket to create its equivalent in codfw.
=== Rebooting aphlict server ===
When you reboot a VM running aphlict by simply "reboot", it can happen that the aphlict service does not come back properly after reboot.
In this case the fix is to manually delete the pid file and start the service.
 rm /var/run/aphlict/aphlict.pid
 systemtctl start aphlict
Alternatively, shut the service down and make sure the PID file is gone before rebooting.
Category:Services
Application servers/Move apache-config to tin

October 2013 - list of crapola we need to do to move the last apache/deployment/config/etc stuff off fenari to tin
RT ticket here: 
There's sync scripts and some binaries not on tin, the apache config repo isn't over there, there's some keys that probably have to be copied around too; see Apaches and Wikimedia binaries for some (outdated) information about those.
This will get formatted into a better list with things checked off, over time.
Ryan wants it converted to git deploy so
Sartoris#Adding_a_new_repo and also porting a few scripts to use salt.
Appserver-buster-upgrade-2021

== Observations of differences between MediaWiki servers on stretch vs buster, before and after , January 2021 ==
=== TCP errors ===
==== Are there more TCP errors ? ====
At first glance it appears as if TCP errors have been reduced when looking at these 2 example hosts. The data gap in the middle is when the reimaging to buster happened.
==== No, not really ====
But once you zoom out and look at an entire week before, it turns out it isn't actually a pattern.
=== disk utilization ===
==== Did disk utilization go up ? ====
Similary it first looks as if disk utilization went through the roof after the upgrade:
==== No, seems like spikes are unrelated ====
But once you zoom out.. you see we have these spikes separate from the upgrade event:
=== performance (avg response time) ===
==== Is it actually getting slower? ====
Looking at average response time it can appear as if a buster server is actually slower if we look at mw1268 (stretch) vs mw1267 (buster) over a 6 hour span:
Similarly if we compare these hosts over a week:
Or over a full 30 days (mw1267 was reimaged on Jan 8):
==== Compare another set of servers ====
===== App =====
But mw1267/mw1268 are really old hardware and will soon be replaced anyways. Let's take another set of servers, appservers, one on stretch and unchanged and the other during the reimaging, so before and after, mw1403 to see whether we can confirm this or not on the more modern hardware.
mw1403 (stretch) vs mw1405 (buster) - response time over 6 hours on 2021-01-27 - can't see a difference here
===== API =====
Let's do the same for API servers. mw1404 (buster) mw1406 (stretch->buster) - avg response time over 12 hours on 2021-01-27, no obvious difference
==== Let's check the same hardware before and after ====
To make really sure, let's check the same machines before and after reimaging.
===== Before =====
First let's take a screenshot of the baseline, before touching them. mw1402,mw1404 are API servers, both on stretch. mw1268,mw1269 are app servers, both on stretch in this image. looking at 24 hours and 7 days and also at the 95th percentile now:
===== After =====
Now after reimaging one of the 2 servers. stretch on the left, buster on the right. Here we are seeing a slower average response time on buster again.
But... the percentage of responses under 250ms is actually getting better
Same comparison for appservers instead of API. Here stretch and buster are reversed, buster on the left, stretch on the right.
Results still inconsistent? Is there a pattern?
== sources ==
example grafana links, dashboards used: host-overview, application-servers-red-dashboard-wkandek
* https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&from=now-7d&to=now&var-server=mw1268&var-datasource=thanos&var-cluster=appserver
* https://grafana.wikimedia.org/d/MLl49gLGk/application-servers-red-dashboard-wkandek?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=appserver&var-method=GET&var-code=200&var-serverA=mw1268&var-serverB=mw1267
Arc Lamp

This is the internal runbook for deploying and monitoring Arc Lamp. Arc Lamp collects stack traces from all MediaWiki production traffic, using php-excimer, and publishes daily aggregated flame graphs and trace logs to https://performance.wikimedia.org/php-profiling/. 
== Service ==
Arc Lamp data originates from the Arc Lamp client (configured in operations/mediawiki-config.git:/Profiler.php) that runs alongside MediaWiki on the application servers. The samples are collected with php-excimer, and sent to a Redis instance on the arclamp host.
The Arc Lamp service then reads the stream of samples from Redis, and produces trace logs on disk. The flame graphs are generated with brendangregg/FlameGraph.
Stats from the Arc Lamp client are sent to Graphite. Stats from the Arc Lamp service go to Prometheus.
===Provision===
*Source code: https://gerrit.wikimedia.org/g/performance/arc-lamp
*Server provisioning: modules/arclamp and profile::arclamp.
We currently run one arclamp#### host in each core data center (Eqiad and Codfw). There's also an equivalent deployment-webperf## host in the Beta Cluster.
=== Monitoring ===
* Grafana: Host overview
* Grafana: Redis (arclamp_redis)
* See also Redis
=== Host migration ===
For how to migrate to a new host, refer to past arclamp1001 upgrade (T319434).
== See also ==
* Excimer, the engine that powers the Arc Lamp.
* WikimediaDebug#Request profiling, which provides on-demand flame graphs for individual debug requests.
* Profiling PHP in production at scale, Timo Tijhof, 2021.
* How we made editing Wikipedia twice as fast, Ori Livneh, 2014.
== External links ==
* Grafana dashboard: Arc Lamp 
* Index page: https://performance.wikimedia.org/php-profiling/
* Public file listing: https://performance.wikimedia.org/arclamp/svgs/daily/
* Upstream brendangregg/FlameGraph.
Category:SRE Observability
Arc Lamp
Auto schema/examples

This page is for examples of check functions:
;Check if cul_user column on cu_log table has default value of 0
 default = db.get_columns('cu_log')['cul_user']['COLUMN_DEFAULT']
 return str(default) == '0'
;The same as above but only on enwiktionary
 default = db.get_columns('cu_log', 'enwiktionary')['cul_user']['COLUMN_DEFAULT']
 return str(default) == '0'
;If revsrc_user column on revsrc table is unsigned (on enwikivoyage only)
 column = db.get_columns('revsrc', 'enwikivoyage')['revsrc_user']
 return 'unsigned' in column['COLUMN_TYPE'].lower()
;If cul_user index exists on cu_log table
 return 'cul_user' not in db.get_indexes('cu_log')
;Same as above but only on frwikisource
 return 'cul_user' not in db.get_indexes('cu_log', 'frwikisource')
;Whether cul_user index on cu_log table is unique
 return db.get_indexes('cu_log')['cul_user']['unique']
;If column cul_actor exists in cul_user index on cu_log table
 return 'cul_actor' in db.get_indexes('cu_log')['cul_user']['columns']
;If column ft_title exists in flaggedtemplates but skip wikis that don't have flaggedtemplates table
 columns = db.get_columns('flaggedtemplates')
 if not columns:
 return True
 return 'ft_title' not in columns
BGPalerter

Replacing the now defunct BGPmon. BGPalerter is an opensource NodeJS tool to monitor network prefixes and ASN using RIPE's RIS Live.
Tracking task: https://phabricator.wikimedia.org/T230600
Relevant CRs (there are more): https://gerrit.wikimedia.org/r/q/topic:bgpalerter
Most of the configuration is done in: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/rpkivalidator.yaml
Especially the contacts to email if an issue is found.
Monitored prefixed are pulled from data.yaml network::external and network::customers
The daemon needs proxy access to reach the RIPE's RIS Live. It's also monitored using the generic Icinga systemd service check.
More recently there is an hosted version available on https://packetvis.com/#/home (contact me if you want to be added to the Wikimedia org)
Backport windows/Deployers/Script

The following Bash script automates the steps in Backport windows/Deployers#SSH Connections and Error Logs: Set up before deploying:
function backport {
 # open browser window
 firefox \
 https://wikitech.wikimedia.org/wiki/Deployments#!/deploycal/current \
 https://wikitech.wikimedia.org/wiki/Backport_windows/Deployers \
 https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors \
 https://logstash.wikimedia.org/app/dashboards#/view/mwdebug1002 \
 https://integration.wikimedia.org/zuul/ \
 https://versions.toolforge.org/
 # ensure the ssh key is in the agent and the ControlMaster connection is alive
 ssh mwdebug2002.codfw.wmnet true
 # ensure the new tmux session will still be able to connect to the agent
 # (in case the tmux server was started before the agent was set up)
 tmux set-environment -g SSH_AUTH_SOCK "$SSH_AUTH_SOCK"
 # set up the backport/config terminal window:
 # logspam-watch deployment
 # mwdebug1002 mwmaint1002
 gnome-terminal --maximize \
 tmux \
 new-session -s backport ssh mwlog2002.codfw.wmnet \; \
 split-window -h ssh deployment.eqiad.wmnet \; \
 split-window -t {left} -v ssh mwdebug2002.codfw.wmnet \; \
 split-window -t {right} -v ssh maintenance.eqiad.wmnet \; \
 send-keys -t {top-left} "exec logspam-watch" Enter \; \
 send-keys -t {top-right} "cd /srv/mediawiki-staging" Enter \; \
 select-pane -t {top-right} \;
}
It will open a new Firefox window with some relevant tabs,
and a GNOME Terminal window with four SSH sessions arranged like this:
logspam-watch on mwlog1002, keep an eye on this	deployment, run git fetch, scap sync-file etc. here
mwdebug1002, run scap pull here 	maintenance server, run mwscript here if necessary
Category:Deployment
Bacon

Bacon was a Wikimedia server. It was working as a thumb server for Wikimedia Commons.
== Basics ==
 RoleThumb server
 OSFedora Core
DECOMMISSIONED ON 2008-03-04
== Specifications ==
 Processor(s)2x Opteron 242 1.6GHz 1MB L2
 RAM2x 512MB PC2700 regd. ECC (6/8 slots in 2 banks available)
 Storage6x 200GB Seagate 7200.7 (7200RPM, 8MB buffer, RAID 10) [total: 600GB]
 I/O Controllers3Ware 9500S-8 8 port RAID controller; expandable to 1GB cache & batt. backup
 MotherboardRioworks HDAMA_F REV E
 BIOSPhoenix PMLC00-7.05/05/2004
 Power460W hot-swap redundant power supplies
 Misc. DrivesFDD, Sony 52x IDE CD-ROM
Part of the August 2004 hardware order.
Bast3001

bast3001 was the main bastion for the Esams cluster until March 2017. (Role previously held by hooft.esams.wikimedia.org, changed per phab:T123712).
It was replaced by bast3002! (per phab:T156506)
== Services ==
* bastion: ssh to other hosts on our internal subnet from here (ssh host fingerprint)
* TFTP install-server for Esams (replaced by bast3002)
* Ganglia aggregators for Esams (replaced by bast3002)
Bast3004

bast3004 was the main bastion for the Esams cluster. It replaced bast3002 as part of the 2019 ESAMS refresh (phab:T235805 phab:T236394), and was in turn replaced by bast3005 (phab:T257324).
== Services ==
* bastion: ssh to other hosts on our internal subnet from here (ssh host fingerprint)
* TFTP install-server for Esams
Bastion

A bastion is the main host on any given network for external users to log into. From the bastion, system administrators access other hosts on the same internal subnets.
While a bastion may not be the only host on a network with a public IP, it tends to be the only one with SSH enabled. Other public nodes are typically frontend web servers.
To verify fingerprints, refer to Help:SSH Fingerprints.
== See also ==
* Production shell access#SSH configuration
* List of Bastion hosts in Wikimedia clusters
* Read about "Bastion host" on Wikipedia
* experimental Bash script for local users to detect the correct bastion and auto-fix config: https://people.wikimedia.org/~dzahn/bastion.sh.txt