nyuuzyou/wikis huggingface dataset:
Grosley

gosley was the failover host for aluminium. See Grosley-Aluminium-Migration for notes from switchover in November 2011.
== Services ==
Various fundraising services, including:
* civicrm.wikimedia.org (Apache, PHP & CiviCRM)
* fundraising.wikimedia.org
* Non-web accessible fundraising scripts can be found in /opt/fundraising_misc.
== Hudson ==
Grosley has Hudson CI installed to facilitate the periodic running of maintenance scripts. It's flexibility, particularly for logging/capturing output, is why we are preferring it over cron. Hudson CI ver. 1.368 running. It is accessible at: http://grosley.wikimedia.org:8080/
=== Potential issues ===
==== Hitting ext3 subdir limit in builds directories ====
Every time a Hudson job runs, a new directory gets created on Hudson where information about that particular build gets stored. Build dirs get created in:
/var/lib/hudson/jobs/<job name>/builds
Particularly for frequently running jobs (eg queue consumption which runs every 5 minutes), we run the risk of hitting the ext3 limit on subirectories (32000). When this limit gets reached, build jobs will fail silently until the issue gets resolved
===== Prevention =====
We have prepared a Nagios script which we are hoping to have monitor the job directories. Warning thresholds should be set at 75% of subdirectory limit and critical at 85%.
===== Resolution =====
As the limit gets approached, build directories should be archived to storage3 and then cleaned out:
# tar czf hudson_builds_<build name>_<starting date>-<ending date>.tar.gz /var/lib/hudson/jobs/<build name>/builds
# scp hudson_builds_<build name>_<starting date>-<ending date>.tar.gz user@storage3.pmtpa.wmnet:/archive/fundraising/hudson_builds/
# rm -rf /var/lib/hudson/jobs/<build name>/builds/*
== Mail setup ==
grosley runs a simple Exim setup, which can send out e-mail directly to the outside world using its dnslookup router and remote_smtp transport. It's also configured to accept mail for *@donate.wikimedia.org, and deliver this mail into a Maildir box in /var/mail/civimail, using the civimail router and civimail_delivery transport. This is basically a simpler version of the configuration described in the Mail article.
Dovecot is installed for IMAP access by CiviMail. Its configuration is almost default, but it's configured to listen using non-SSL IMAP on the loopback interface only, and to retrieve mail using Maildir from /var/mail/user. Authentication is done using PAM and system users, so a system user civimail has been added.
== Configuration changes ==
* localhost IPs (v4 and v6) added to relay_from_hosts in exim4.conf --David Strauss 02:37, 7 May 2009 (UTC)
* Truncated /var/log/exim4/paniclog, which had an old configuration syntax error notice in it --David Strauss 02:38, 7 May 2009 (UTC)
HAProxyKafka

HAProxyKafka is a daemon running on all cp hosts to read logs produced by HAProxy and forward them to Kafka to be used in analytics pipelines.
The aim is to replace VarnishKafka to have a better observability of incoming requests, considering that HAProxy is our main entry point for all CDN requests. 
The daemon is written in Golang and uses the librdkafka library installed on the cache hosts (and NOT the embedded one provided by the Golang package).
Useful links:
* Code repository on GitLab
* Grafana Dashboard
== Building the package and deploy new version ==
GitLab CI should build the package just fine using the usual branch naming to choose the target distribution. Upload the binary package as usual to the internal APT repo and deploy it manually (or with something like cumin) as usual.
== HAProxy configuration ==
To let HAProxyKafka correctly parse and dispatch messages, HAProxy must be configured:
1. To log to a unix domain socket (created by HAProxyKafka in advance with the correct permissions)
2. To structure logs with the expected log-format. This includes the captured headers in the correct order, in RFC5424 format.
Luckily using Puppet this is configured automatically, along with the HAProxyKafka configuration file and services. Refer to existing puppet hiera/code for this.
== Procedures ==
TBD
== See also ==
TBD
Category:Services
Category:SRE Traffic
HHVM/Troubleshooting

== Problem determination and data collection ==
* hhvmadm check-health: emits useful statistics like load, queued reqs, etc..
* hhvm-dump-debug: captures a stack trace of HHVM and writes it to stdout and to /tmp/hhvm.$PID.bt. Run hhvm-dump-debug --full to generate a core dump as well.
* See Debugging in production#Locally.
* links -dump http://127.0.0.1/server-status -width 80: produce a full Apache status report, showing which requests are currently in flight.
* furl is a cURL-like tool for FastCGI. You can use it to bypass Apache and request pages directly from HHVM. For example: furl http://en.wikipedia.org/wiki/Main_Page. Requests always go to localhost.
* curl localhost:9002/dump-apc will dump all the current APC keys to /tmp/apc_dump.
* stacktraces, if available, are in /var/log/hhvm
* core dumps are in /var/tmp/core or sysctl kernel.core_pattern
== Light process dying ==
If you see light processes deaths in the logs it could be because they always log when they exit (noisy but ignorable). Please check the parent page for an explanation of what a light process is.
HTTPS/CAcert

http://www.cacert.org/
CAcert is the biggest of the off-brand SSL certificate authorities. They have the nice feature of being free (as in cost), and are based on a distributed trust network which is also ideologically favorable.
At the moment they're not in the default CA list for most major browsers, but are in process of being audited by Mozilla and should eventually get in. Even without this, they're better than using self-signed certificates, as it gives us protection against MitM attacks and generally makes one feel fuzzy and warm.
Wikimedia Foundation, Inc. has been set up with an organizational account. There are various team members who have access to the organizational account (brion & tomasz) and you should ask on IRC if you need a certificate signed.
HTTPS/WMF CA

= Introduction =
WMF has found occasionally the need to host it's own internal CA for some purposes. That is what WMF CA is. There are 2 WMF CAs at this point:
* wmf_ca
* wmf_ca_2014_2017
== wmf_ca ==
This has finally been completely phased out and any rerefences to it are historical. Any certificates by it have been reissued by the the second one and it has been removed from the puppet repo and any keystores. It should no longer be possible to use it, issue/revoke certs etc.
== wmf_ca_2014_2017 ==
This is the current WMF CA. It is set to expire in 2017 as you can tell by the name. Should we still need it by then we should very easily issue a new one and reissue the certificates. 
== Howto ==
puppet private repo holds all the information you might need to manipulate the wmf_ca_2014_2017. There is a README there that should cover the basics.
== Notes ==
If you want to re-issue a cert, don't forget to revoke the old one first or you will get a:
 
 failed to update database
 TXT_DB error number 2
Hardware Disambiguation

This page serves as a rough translation between informal names (db) and hardware (dell r510). All statements should be verified on specific hardware. Documentation for each type is in the category page in case it's not linked inline.
Legend:
* System Role: the generic definition of the hardware class
* Current Platform: What we are presently using for new purchases of this hardware class
* Past Platform: What we have used in the past, that is still in active rotation.
System Role 	Current Platform 	Past Platform(s)
apache (srv,mw) 	Dell PowerEdge R410 	Dell PowerEdge R610, Dell PowerEdge 1950
caching (sq,cp) 	Dell PowerEdge R610 	Dell PowerEdge 1950
database (db) 	Dell PowerEdge R510 	Sun SunFire X4240, Dell PowerEdge 2950
es 	Dell PowerEdge R510 + 12 2TB disks 	Sun SunFire X4500, Sun SunFire X4540
memcached (mc) 	Dell PowerEdge R610 	
ms 	Dell PowerEdge R510 + Dell PowerVault MD1200	Sun SunFire X4500, Sun SunFire X4540
ms-be 	Dell R720xd or C2100 + 12 or 14 2TB disks 	
misc 	Dell PowerEdge R310 	
high performance misc 	Dell PowerEdge R610 	
Search Indexer (searchidx)	Dell PowerEdge R610 	Sun SunFire X4150
Search Node (search) 	Dell PowerEdge R410 	Sun SunFire X4150
Category:Hardware
Category:Platform-specific documentation
Helm/Upstream Charts

This page holds a list of upstream charts that have been evaluated for usage in the Wikimedia Production Kubernetes clusters using Kubernetes/Upstream_Helm_charts_policy.
 type=create
 useve=1
 preload=Helm/Upstream_Charts/Template
 placeholder=Chart Name
 buttonlabel=Create new evaluation
 bgcolor=inherit
 break=no
 prefix=Helm/Upstream_Charts/
== Evaluated Charts ==
Evaluated Upstream Helm Charts
== Charts in progress ==
To Be Evaluated Upstream Helm Charts
Helm/Upstream Charts/Calico

Chart name: Calico
Chart version: v3.23.3
Reviewers: JMeybohm, Alexandros_Kosiaris
Link to repo: https://docs.tigera.io/calico/charts
Link to source code: https://github.com/projectcalico/calico/releases/download/v3.23.3/release-v3.23.3.tgz
Topics:
* network
Decision: pass
Reasoning:
It was decided to use the upstream chart as base and modify it to fit our needs/deployment methods.
Upstream calico relies heavily on host-path mounts to install cni binaries etc. which we roll out via puppet instead, so those parts have been removed completely. Corresponding RBAC permissions have been dropped as well.
Category:Evaluated Upstream Helm Charts
Helm/Upstream Charts/Template

Chart name: <insert name>
Chart version: Version of that chart that is evaluated
Phab task: Link to phab
Reviewers: <name #1>, <name #2>
Link to repo: Link to the OCI registry and/or helm chart repo holding the chart artifact
Link to source code: Link to the source code of the chart
Topics:
* e.g. machine learning
* frontend
* systems
Decision: [fail|pass|hold]
Reasoning:
Insert the reasoning for passing, failing or holding a helm chart review
Change the category to Evaluated Upstream Helm Charts once evaluation is done.
Category:To Be Evaluated Upstream Helm Charts
Helm/Upstream Charts/ceph-csi-cephfs

Chart name: ceph-csi-cephfs
Chart version: v3.7.2
Phab task:
Review: 
Use case: 
Reviewers: Balthazar Rouberol, Luca Toscano
Link to repo: https://ceph.github.io/csi-charts https://artifacthub.io/packages/helm/ceph-csi/ceph-csi-cephfs
Link to source code: https://github.com/ceph/ceph-csi/blob/v3.7.2/charts/ceph-csi-cephfs/Chart.yaml
Topics:
* storage
Decision: pass
Reasoning: https://phabricator.wikimedia.org/T376406#10221585
Category:Evaluated Upstream Helm Charts
Helm/Upstream Charts/ceph-csi-rbd

Chart name: ceph-csi-rbd
Chart version: v3.7.2
Phab task:
Review: 
Use case: 
Reviewers: Ben Tullis, Luca Toscano
Link to repo: https://ceph.github.io/csi-charts https://artifacthub.io/packages/helm/ceph-csi/ceph-csi-rbd
Link to source code: https://github.com/ceph/ceph-csi/blob/v3.7.2/charts/ceph-csi-rbd/Chart.yaml
Topics:
* storage
Decision: pass
Reasoning:
The helm chart is of sufficient quality and is actively developed. Upstream are responsive and receptive to patches intended to improve the functionality and/or security of the chart.
There are multiple components required to install and operate a Ceph CSI plugin. Some are provided by the kubernetes project, while others are shipped by the Ceph project. It would have been difficult for us to re-engineer what has been achieved in this chart by ourselves, using our standard scaffolding.
Several small modifications have been made:
# Allow selective disabling of the csi-snapshot functionality, since this requires additional configuration and we wish to start using ceph-csi without it.
# Prevent certain containers from requesting elevated privileges, which are only required in an SELinux environment.
There is one container that requires SYS_ADMIN capabilities and that is the cephrbd plugin container itself, which is running as part of the daemonset on each worker. We evaluated whether it would be practicable to extract this container from the pod and run it as a daemon on the bare-metal worker. Upon investigation, it was decided that this would likely create a more brittle configuration by coupling the daemonsets to an externally managed daemon, so we have permitted this container to run with the required privileges. All other privilege escalations have been disabled.
Category:Evaluated Upstream Helm Charts
Helm/Upstream Charts/cert-manager

Chart name: cert-manager
Chart version: v1.10.1
Reviewers: JMeybohm, Alexandros_Kosiaris
Link to repo: https://charts.jetstack.io
Link to source code: https://github.com/cert-manager/cert-manager/tree/v1.10.1/deploy/charts/cert-manager
Topics:
* TLS
* Operator
Decision: pass
Reasoning:
The chart is in very good quality and well maintained so it was decided to use the upstream chart with minimal adjustments.
Category:Evaluated Upstream Helm Charts
Helm/Upstream Charts/cloudnative-pg

Chart name: cloudnative-pg
Chart version: 0.21.4
Phab task: https://phabricator.wikimedia.org/T364797
Reviewers: Balthazar Rouberol, User:Alexandros Kosiaris
Link to repo: https://github.com/cloudnative-pg/charts/
Link to source code: https://github.com/cloudnative-pg/charts/tree/cloudnative-pg-v0.21.4/charts/cloudnative-pg
Topics:
* databases
* kubernetes operators
* postgresql
Decision: pass
Reasoning:
The chart was reviewed, and we adjusted the RBAC, that we found pretty lax. With some additional changes, we found it appropriate to integrate within our charts.
Category:Evaluated Upstream Helm Charts
Helm/Upstream Charts/eventrouter

Chart name: eventrouter
Chart version: v0.3.0
Reviewers: JMeybohm, Alexandros_Kosiaris
Link to repo: N/A
Link to source code: https://github.com/helm/charts/tree/master/stable/eventrouter
Topics:
* Observability
* Logging
Decision: pass
Reasoning:
The chart is in okay shape to use as base, but we will need to derive from upstream a bit (for NetworkPolicy support and some minor tweaks). As the chart is simple enough we decided to import it as base and go from there.
Category:Evaluated Upstream Helm Charts
Helm/Upstream Charts/kube-state-metrics

Chart name: kube-state-metrics
Chart version: 5.10.1
Phab task: https://phabricator.wikimedia.org/T264625
Reviewers: Kamila Součková, JMeybohm, Elukey
Link to repo: https://prometheus-community.github.io/helm-charts/
Link to source code: https://github.com/prometheus-community/helm-charts/tree/kube-state-metrics-5.10.1/charts/kube-state-metrics
Topics:
* observability
* kubernetes
* infrastructure
Decision: pass
Reasoning:
The chart is good quality and actively maintained by a community. It looks like a good fit for our use case and should not require significant changes.
Category:Evaluated Upstream Helm Charts
Heterogeneous deployment/Train deploys/GitStructure

< liw> when we do scap prep on a new train, it seem the extension repos have detached heads, e.g. /srv/mediawiki-staging/php-1.36.0-wmf.35/extensions/RSS right now
   < liw> is that intentional?
   < hashar> liw: yes
   < hashar>  /srv/mediawiki-staging/php-1.36.0-wmf.35  is a working copy of mediawiki/core  on branch wmf/1.36.0-wmf.35
   < hashar> in Gerrit the .gitmodules file registers the extensions / skins / vendor repositories
   < hashar> each with  a gerrit specific parameter of  branch = .
   < hashar> which is to indicate that gerrit should update the submodules when the repos get changes merged
   < liw> but /srv/mediawiki-staging/php-1.36.0-wmf.35 itself does not have a detached head
   < hashar> and thus  extensions/RSS is just a submodule , and they are typically in detached head
   < hashar> cause that is mediawiki/core and we explicitly checkout wmf/1.36.0-wmf.35 branch
   < hashar> you can think about it as a mono repo
   < hashar> of core + vendor + extensions + skins
   < hashar> implemented using git submodules
   < liw> that's just it: I can't think of it as a monorepo, when it's clearly a mess
   < hashar> so essentially one "just" need to:
   < hashar>  git clone --branch wmf/1.36.0-wmf.35 php-1.36.0-wmf.35
   < hashar> cd php-1.36.0-wmf.35
   < hashar> git submodule update --init
   < hashar> apply sec patches
   < hashar> {done}
   < hashar> and if one has to do a backport for an extension:
   < hashar> send patch to master branch, get it reviewed / CR+2
   < hashar> cherry pick to the wmf/1.36.0-wmf.35  branch, get it reviewd / CR+2
   < hashar> once CI submit the change
   < hashar> Gerrit automatically update the submodule registered in mediawiki/core  wmf/1.36.0-wmf.35 branch
   < hashar> so on the deployment server, one  does:  cd /srv/mediawiki-staging/php-1.36.0-wmf.35
   < hashar> git remote update
   < hashar> git log HEAD..HEAD@{u}  # to review the diff
   < hashar> git rebase
   < hashar> git submodule update --init
   < hashar> which is configured to automatically rebase submodules in case there is a security patch applied
   < hashar> scap deploy
   < hashar> done
   < hashar> I guess we could add that tldr in the overhauled train doc brennen plans to work on
   < hashar> !log Purging openjdk-8 packages from Jenkins agents # T269354
Heterogeneous deployment/old

Below are miscellaneous notes about past implementation of this idea. For the latest information, see Heterogeneous deployment.
== Roan's quick&dirty implementation ==
A q&d implementation of het deploy is currently used to serve 1.17 on test2wiki and will be used to do a partial deployment of 1.17 soon.
=== What lives where ===
* Files for 1.16 live in /home/wmf-deployment/common/wmf-deployment , files for 1.17 live in /home/wikipedia/common/php-1.17
* The 1.17 tree has a separate set of configuration files in php-1.17/wmf-config
* The list of wikis on 1.17 is in /home/wikipedia/common/1.17.dblist. Wikis not in this list are on 1.16
* The magic to switch between 1.16 and 1.17 is in /home/wikipedia/common/live-1.5/MWVersion.php. The original version without any het deploy-related changes has been preserved as MWVersion.php-before-Roan-started-messing-with-it
* URLs with /w/skins-1.5 and /w/extensions point to the 1.16 tree, /w/skins-1.17 and /w/extensions-1.17 point to the 1.17 tree. 1.17 wikis have $wgStylePath and $wgExtensionAssetsPath set up to point to the latter two
* On the Apaches, CDB cache for 1.16 lives in /tmp/mw-cache, for 1.17 it's in /tmp/mw-cache-1.17
=== How to do stuff ===
* To update the 1.17 code, simply run svn up in the php-1.17 directory
* To sync files in the 1.17 tree, use sync-common-file php-1.17/filename (this does not check PHP syntax!)
* The 1.17 tree will be synced in its entirety when sync-common-all or scap is run, along with many other things
* To switch a wiki to 1.17 or back to 1.16, add/remove it in /home/wikipedia/common/1.17.dblist and run sync-dblist
** After downgrading a wiki, run echo '$wgMessageCache->clear();' | php /home/wikipedia/common/wmf-deployment/maintenance/eval.php --wiki=blah to fix an issue with customized messages getting stuck
* Be careful when running maintenance scripts. The magic switch thing does NOT work for them, so you have to find out which version the relevant wiki is on and run the script from wmf-deployment for 1.16 wikis and from php-1.17 for 1.17 wikis
== Notes for future project ==
Project notes for multiple extension/core version deployment.
Requirements
* Multiple core versions
* Multiple extension versions
* Ability to have at least staging + production
Nice to have
* A/B testing
Issues:
* bits: $wgExtensionAssetsPath and $wgStyleSheetPath
* l10n cache
* separate $IP, but shared configuration
* other path elements in configuration that don't use $IP
* scap pushing multiple versions
* downgrading
* (figure out if there are any other shared caches)
* APC cache size
* maintenance script location (and other things that might need to be symlinked)
Plan:
# Replace scap with wmsync
# devise a way to list which sites get which version
# tweak MWVersion.php for 'live' (w/) stub scripts to check the above list and divert to appropriate $IP
# ...
# profit!
(ok....we don't have a full blown plan yet, but we know scap needs work)
History of job queue runners at WMF/Jobrunner

Redis is used for the MediaWiki job queue as a memcached replacement.
The Job queue in production is backed up by a Redis storage (see Redis for documentation on how to issue Redis commands). The jobs in the queue are run by the Jobrunner service.
==Status==
* https://grafana.wikimedia.org/dashboard/db/job-queue-rate
* https://grafana.wikimedia.org/dashboard/db/job-queue-health
== Cluster ==
Redis queues:
* rdb1001, rdb1002, rdb1003 and rdb1004 (as of September 2013 – check Puppet for current data).
15 job runners (not counting videoscalers):
* Eqiad: mw1161-1167 (7), mw1299-3006 (8). Wikimedia: operations/puppet.git:/conftool-data/node/eqiad.yaml#L10-L25 (on GitHub)
* Codfw: mw2153-mw2162 (10), mw2243 (1), mw2247-mw2250 (4). Wikimedia: operations/puppet.git:/conftool-data/node/codfw.yaml#L149-L163 (on GitHub)
==Job queue keys==
The queues have various pools which have names of the form
wikidbname:jobqueue:entrytype:setorlistorhashname (example: commonswiki:jobqueue:webVideoTranscode:h-data)
where the sets, lists and hashes are one of the following:
*z-claimed -- set of 'claimed' job queue entries to be run 
*z-abandoned -- set of job queue entries that have been retried max times and failed, hence we gave up
*z-delayed -- set of delayed entries (jobs set to run at a given time in the future)
*l-unclaimed -- list of job queue entries waiting to be claimed to be run
*h-data -- hash containing job id => job info for all jobs
and there may be others.
==Getting counts of queues (lists and sets)==
zcard <set-name> gives you the number of items in the set, so e.g. 
:zcard commonswiki:jobqueue:webVideoTranscode:z-abandoned
llen <list-name> gives you the number of items in the list, so e.g.
:llen commonswiki:jobqueue:webVideoTranscode:l-unclaimed
==Getting slices of sets==
zrange <set-name> <slice-start> <slice-end> (lrange for lists) gives you the values in the set for the given index range, so e.g.
:zrange commonswiki:jobqueue:webVideoTranscode:z-abandoned 0 50
will give you a list of items that look like
 1) "0e7b1d37856445f7a1d1eab1478f3ffd"
 2) "0f9f569e43324204bf3d8782ba09de63"
 3) "102e7ef6641644e8ac3654f73f3e7f35"
 4) "2e82cd1078044a1a893edbbc5fd24c8a"
etc. These entries are job ids.
==Examining the data for a job==
Once you have gotten the job ids by using a zrange statement as above, you can look at the data by asking for the value associated with the hash in h-data, e.g.
:hget commonswiki:jobqueue:webVideoTranscode:h-data 092cbbc0f75344d1b33dd85c65d33328
This will return a string of serialized data describing the job, for example:
a:8:{s:4:\"type\";s:17:\"webVideoTranscode\";s:9:\"namespace\";i:6;s:5:\"title\";s:35:\"Sublimation_of_dry_ice_on_water.ogv\";s:6:\"params\";a:2:{s:13:\"transcodeMode\";s:10:\"derivative\";s:12:\"transcodeKey\";s:9:\"480p.webm\";}s:10:\"rtimestamp\";i:0;s:4:\"uuid\";s:32:\"c0959e4ed0a2451b8677119361f7af73\";s:4:\"sha1\";s:31:\"5t4s7oe4nz9jneymz8bovt226somsve\";s:9:\"timestamp\";i:1370265154;}
Category:MediaWiki production
Hooper

hooper ran miscellaneous services
== Services ==
* was etherpad.wikimedia.org (THIS IS THE OLD etherpad, not etherpad-lite). was up as https://etherpad-old.wikimedia.org/
* racktables.wikimedia.org
=== Puppet Roles ===
hooper is a Wikimedia blog.wikimedia.org (misc::blog-wikimedia).
hooper is a Wikimedia blog.wikimedia.org (misc::blogs::wikimedia).
hooper is a Wikimedia Etherpad server (misc::etherpad).
hooper is a Wikimedia Racktables (misc::racktables).