nyuuzyou/wikis huggingface dataset:
Benthos

Benthos defines itself as fancy stream processing made operationally mundane.
==== Benthos on centrallog ====
As of Nov 2022 it is used at WMF to provide a real time (sampled) stream of Varnish requests (webrequest). The initial groundwork has been done as part of by Luca Toscano and Filippo Giunchedi. The plan is to expand its use for simple and stateless stream processing, inside and outside SRE.
Observability into the system is provided by the the Benthos Grafana dashboard.
The webrequest pipeline runs on centrallog hosts, reads the webrequest Kafka topics, samples and enriches the stream and writes back to webrequest_sampled Kafka topic for general consumption. The sampled topic then powers (via Druid) the Turnilo dashboard and the Superset dashboard (NDA-only links)
==== Benthos on cache hosts ====
As of Apr 2024 Benthos is also used to replace VarnishKafka on cache (cp) hosts, starting with the ulsfo DC as guinea pig. The activity is tracked on various tickets, mainly and .
The original log flow started with Varnish logs on cache hosts, read by VarnishKafka (using the Varnish API to read the log ring buffer) and sent to Kafka Jumbo cluster. This leads to some inconveniences, mainly the lack of observability at HAProxy level.
Then new flow starts with HAProxy writing logs (in syslog format) to a special UDP socket. A Benthos instance on each cache host reads from this socket, parses the logs and produces the required metrics, while sending the formatted message to the appropriate Kafka topic.
This will completely replace VarnishKafka, as Benthos message format is a superset of VarnishKafka ones, so downstream pipeline is already able to accept/process new messages from HAProxy.
==== How to add a new instance ====
Adding a new instance is simple enough:
* Add an entry to profile::benthos::instances in hieradata/role/common/syslog/centralserver.yaml:
 # The name of the instance
 foobar:
 # This is passed to your template as an env variable you can reference as $PORT.
 # Make sure to set this value to a port that is not used already by other Benthos
 # instances!
 port: 4151
 # Any additional env variables you want to be able to modify the behaviour of
 # your instance
 env_variables:
 # Yes, this is a troll about how bad yaml is
 - pinkunicorn: no
 kafka:
 cluster: jumbo
 # Be careful - each instance will only point to the kafka in one DC.
 # If you need both, either subscribe to the replicated topics or add
 # a second instance, or again run separately in the two sites
 site: eqiad
 topics:
 - "foobar"
 - "anothertopic"
* Add the configuration for your benthos instance as a file modules/profile/files/benthos/instances/foobar.yaml
Bits.wikimedia.org/Bits and pieces plan

The idea of bits and pieces is moving serving a group of certain most viewed static objects (css, js, etc) via another set of cache servers - that would be configured for many many connections, many many requests and not too much data with nearly no invalidations. 
== Rationale ==
Currently we have quite heavy mix of our standard "needs huge cache" and small assets workload:
* 18% of all squid load is for /skins*
* 10% of all squid load is for /w/extensions/* 
* 3% of hits are for centralnotice (even after fundraiser)
* 12% of hits are for action=raw css/js files
Frankly, that is most of our standard text squid load, so by moving it away we're making text cluster purely article-caching and mediawiki request routing. Though moving action=raw requests may not be feasible immediately, it is a possibility for future development. Initial target plan is using bits and pieces for /skins and /w/extensions.
This has already caused stability problems in past (e.g. new Usability Initiative assets have overloaded our systems), so we should do quite aggressive initial sizing for newly built platform (we can target for 100k-150k requests/s). 
== Software ==
* Such deployment is ideal for initial Varnish testing in our environment. Of course, there may be need to fall back to Squid or Sun Java Enterprise Cache. ;-) 
* Alternative is having software deployed to nginx or lighttpd instances on the edge. 
* We may also consider switch-based load balancing here, as workload will be uniform
== Deployment plan ==
* Pick name ;-) bitsandpieces.wikimedia.org looks fine! Alternatively we can use 'bits.wikimedia.org' or 'pieces.wikimedia.org' ;-) If we want to be industry buzzword compliant it would be called 'assets'.
** currently it is bits.wikimedia.org
* Configure virtual host on all apaches, that would be configured to statically deliver /w/extensions and /skins/, and not serve mediawiki itself (403 on PHPs, no redirects, etc - would do fine)
** done
* point the name to text cluster RRs
** done
* Make sure Squid deals fine with this new domain and files. 
** done
* Change inside MediaWiki to use new domain (URL structure can be same, just shared domain)
** done for core and UsabilityInitiative/*
* Set up new geo-balancing domain
** done: bits-geo.wikimedia.org 
* Set up new VIP on load-balancers
* Add new caches to new VIP
** During testing Varnish deadlock was hit - and escalated to Linpro (more on Bits varnish testing)
* Start changing VIP for 'bits' to new-cache-based pool
Bits.wikimedia.org/Varnish testing

Few notes:
* We hit a bug where all threads are writing to acceptor pipe, but acceptor thread doesn't seem to pick that up
* Originally thought as 2.6.24 kernel problem, a 2.6.32.3 was deployed, but still got same problem
* This happens with both poll and epoll acceptors (managed to hit it much earlier with poll acceptor, may be coincidence)
* Currently we are running:
** varnish 2.0-branch, Feb15 build (standard configure options)
** Following sysctl.conf additional changes loaded:
 net.core.rmem_max=16777216
 net.core.wmem_max=16777216
 net.ipv4.tcp_rmem=4096 87380 16777216
 net.ipv4.tcp_wmem=4096 65536 16777216
 net.ipv4.tcp_fin_timeout = 3
 net.core.netdev_max_backlog = 30000
 net.ipv4.tcp_no_metrics_save=1
 net.core.somaxconn = 262144
 net.ipv4.tcp_syncookies = 0
 net.ipv4.tcp_max_orphans = 262144
 net.ipv4.tcp_max_syn_backlog = 262144
 net.ipv4.tcp_synack_retries = 2
 net.ipv4.tcp_syn_retries = 2
** ulimit -s 128
** ulimit -n 500000
** varnishd -n /dev/shm -smalloc,1G -f /usr/local/etc/varnish/bits.vcl -T 127.0.0.1:6000 -w 2000 -a 0.0.0.0:80 -p thread_pool_add_delay=1 -p send_timeout=30 -p listen_depth=4096
Apparently when varnish reaches 100% cpu usage, accept thread gets into a state where it leaks worker threads. poll() reaches 100% cpu sooner than epoll, so effect visible way earlier. 
As sq1 is single-cpu/single-core, high ratio of context switches consumes way more CPU resources, than it would in multithreaded environment - this is why we saw much better scalability on multi-core machines.
Currently bits.pmtpa is being handled by db19, which is handling 14000 requests/s with 180% cpu load and ~100MByte/s traffic.
db19 currently is connected via bond0 over eth0/eth1 directly to core switch. 
VCL:
backend default {
.host = "10.2.1.1";
.port = "80";
}
sub vcl_recv {
 if (req.request != "GET" && req.request != "HEAD") {
 /* We only deal with GET and HEAD by default */
 error 403 "this is readonly domain";
 }
 if (req.http.host != "bits.wikimedia.org") { 
	error 403 "bad bad, very bad request"; 
 }
 return (lookup);
}
sub vcl_error {
 set obj.http.Content-Type = "text/html; charset=utf-8";
 synthetic {"
<!DOCTYPE html>
<html>
 <head>
 <title>"} obj.status " " obj.response {"</title>
 </head>
 <body>
 Error "} obj.status " " obj.response {"
 We didn't feel like serving it
 </body>
</html>
"};
 return (deliver);
}
Blogposts

Propose topics we could blog about, and spread love and knowledge.
== Upcoming posts ==
* Migration from HHVM to PHP7? (cf. 2014 switch to HHVM)
* More at https://phabricator.wikimedia.org/tag/technical-blog-posts/
== See also ==
* https://techblog.wikimedia.org/
* https://www.mediawiki.org/wiki/Wikimedia_technical_blog_editorial_guidelines
* https://www.mediawiki.org/wiki/Technical_communications/Tech_blog_activity
Blubber

Blubber is a BuildKit frontend for building application container images from a minimal set of declarative constructs in YAML. Its focus is on composability, determinism, cache efficiency, and secure default behaviors.
Blubber is used by Wikimedia CI to test code and publish production-ready Docker images.
== Developers ==
For information on how to contribute, see CONTRIBUTING.md in the Blubber repository on GitLab.
Blubber/Deployment

=Deploying blubber(oid) to Production=
Blubberoid is deployed to kubernetes in the staging, codfw, and eqiad clusters.
Follow the instructions on Deployments_on_kubernetes to deploy to each cluster.
==Testing the deployments==
If you supply a blubberfile, testing each deployment can be done from the deployment server by curling staging.svc.eqiad.wmnet:4666, blubberoid.svc.codfw.wmnet, and blubberoid.svc.eqiad.wmnet, respectively.
For example:
curl -s "https://staging.svc.eqiad.wmnet:4666/v1/production" -H 'content-type: application/yaml' --data-binary @- <<__EOF__
version: v4
base: docker-registry.wikimedia.org/wikimedia-stretch
variants:
 production:
 entrypoint: [echo, "Hello, world!"]
__EOF__
Bolt

Puppet Bolt supports a variety of uses cases, but one in particular is interesting for our use, masterless Puppet catalog applies. In a masterless apply rather than obtaining the catalog from the puppet master the catalog is compiled directly from the source repository. One advantage of a masterless apply is that it can offer faster feedback on code changes, since you don’t need to commit in order to apply or noop your changes against a destination host, i.e.:
# Edit
$ vi modules/example/manifests/init.pp
# Noop to see changes
$ bolt apply --noop -t node.example.com
# Commit
$ git commit -a -m 'add example module'
Other projects which support a masterless workflow:
* supply_drop
== Install Directions ==
=== Install Bolt, not currently in Debian unfortunately ===
$ wget https://apt.puppet.com/puppet-tools-release-bullseye.deb
$ sudo dpkg -i puppet-tools-release-bullseye.deb
$ sudo apt-get update
$ sudo apt-get install puppet-bolt
=== Delete Bolt’s system module ===
This module conflicts with our system module, Upstream pull request or rename our module
$ sudo find /opt/puppetlabs/bolt -depth -path '*bolt-modules/system' -exec rm -r '{}' \+
=== Add bolt utilities to your path ===
$ git clone gitlab.wikimedia.org:jhathaway/bolt-wmf.git
$ cd bolt-wmf
$ cp bolt-wmf puppet-out-grep ~/.local/bin/
== Usage ==
=== Noop a server ===
Saving its output into puppet.out, so we can go through it more carefully$ bolt-wmf -t mirror1001.wikimedia.org -- apply --noop | tee puppet.out
=== Look through output ===
Use the puppet-out-grep script to remove items from the console output that you don't care about
$ puppet-out-grep Exim Nagios
== Outstanding Issues ==
# Bolt's system modules conflicts with our system module, Upstream pull request or rename our module
# Bolt does not preserve symlinks on files with recurse and source directories, https://github.com/puppetlabs/bolt/issues/3056
# PuppetDB queries do not work, nor do exported resources
# acme_chief module does not work
# Nooping a server with Bolt will take out a Puppet lock, which would block a cron based Puppet run which began after the Bolt run
Category:Puppet
Category:SRE Infrastructure Foundations
Brewster

Brewster was a Dell PowerEdge R300, with 4GB of RAM and two 1T drives (really?), running Ubuntu 8.04.3. 
== Decom ==
Decommissioned in April 2014. Details at phabricator:T83137, phabricator:T83804
Replaced by carbon, then install1001 / install2001 and then install1002 / install2002 and then install1003 / install2003
== Services ==
Previously:
* squid gateway
* DHCP server
* TFTP install-server.
Even earlier:
* apt.wikimedia.org and ubuntu.wikimedia.org
* http proxy for the pmtpa datacenter hosts. (migrated to carbon in April 2014)
Bromine

bromine is a virtual machine that runs in Ganeti. It can be controlled from ganeti1003 and runs on ganeti01.
Bromine runs miscellaneous static HTML services that previously ran on zirconium.
== Services ==
* include role::webserver_misc_static (Microsites)
== VM Info ==
[ganeti1003:~] $ sudo gnt-instance info bromine.eqiad.wmnet
..
- Instance name: bromine.eqiad.wmnet
 Creation time: 2015-07-01 19:13:34
 Modification time: 2015-07-03 01:13:39
 State: configured to be up, actual state is up
 Nodes: 
 - primary: ganeti1003.eqiad.wmnet
..
 Disks: 
 - disk/0: drbd, size 20.0G
 access mode: rw
 nodeA: ganeti1003.eqiad.wmnet, minor=3
 nodeB: ganeti1001.eqiad.wmnet, minor=0
...
Building OpenJDK 8 backports

* Fetch openjdk-8 source package from testing (check https://packages.qa.debian.org/o/openjdk-8.html). Per the Debian backports rule it needs to enter testing first. In case of major testing migrate lag, we might simply waive that.
* Sync it to the builder host
 http_proxy=http://url-downloader.wikimedia.org:8080 dget -x LINK_TO_OPENJDK.dsc
* Add a changelog entry with "dch --bpo" (that should correctly set the distribution field to "jessie-backports" and update the version number by appending ~bpo8+1.
* Regenerate the control/rules file to use dependencies available in jessie:
 touch debian/control.in
 debian/rules debian/control distrel=jessie
* Rebuild for jessie (not jessie-wikimedia!) (preferrable in a screen session, the test suite run takes a long time): 
 DIST=jessie pdebuild
* Test the package
* Sign and upload to jessie-backports with dput $CHANGESFILE
CDN/Hardware

This is an overview of our currently deployed and active cache hardware at the Traffic layer.
== Hardware classes ==
We have purchased and retired multiple classes of server hardware over the years in staggered timeframes. In the general case we'll always have multiple overlapping classes of hardware as various warranty and support periods expire. These are the currently-active hardware configuration classes:
Label	Model 	CPU Type/Speed 	Phys Cores	RAM 	Cache storage 	NIC speed, type, driver	DC Ops Config
F2 	Dell R440	2x Xeon Gold 5118 @ 2.3Ghz 	24 	384 GB	1x Samsung PM1725b 1.6TB NVMe (HHHL Card)	10G, BCM57412, bnxt 	F-10G, +storage card
F3 	Dell R440	2x Xeon Gold 5118 @ 2.3Ghz 	24 	384 GB	1x Samsung PM1725b 1.6TB NVMe (HHHL Card)	10/25G, BCM57412, bnxt 	F-10G, +storage card, +10/25G NIC variant
F4 	Dell R450	2x Xeon Gold 5318Y @ 2.1Ghz	48 	512 GB	2x 6.4TB NVMe Card 	10/25G, BCM57414, bnxt 	Config F
== Deployed hardware ==
Currently deployed hardware by data center and caching cluster. 
Current as of May 2024
Data center	cache_text	cache_upload	total 	Approx Next 5 Yr Refresh - FY (CY)
codfw 	8x F2 	8x F2 	16x F2	FY25 Q3 (2025-Jan)
drmrs 	8x F3 	8x F3 	16x F3	FY27 Q2 (2026-Oct)
ulsfo 	8x F4 	8x F4 	16x F4	FY28 Q2 (2027-Oct)
eqsin 	8x F4 	8x F4 	16x F4	FY28 Q2 (2027-Oct)
esams 	8x F4 	8x F4 	16x F4	FY29 Q1 (2028-Jul)
eqiad 	8x F4 	8x F4 	16x F4	FY29 Q2 (2028-Oct)
magru 	8x F4 	8x F4 	16x F4	FY29 Q4 (2029-Apr)
Totals 	56 	56 	112 	
Category:Caching
Category:SRE Traffic
CX

CX refers to "content translation". You may be looking for:
* Content translation (mediawiki.org), manual for users. 
* Extension:ContentTranslation (mediawiki.org), developer manual of the MediaWiki extension (provides the UI and wiki integration).
* cxserver, the Node.js-powered backend that powers this feature:
** mediawiki/services/cxserver.git (source code)
** https://cxserver.wikimedia.org/v1/?doc (live instance and API docs)
Category:Disambiguation pages
Category:Services
Cache TLS termination

TLS in the caching cluster is terminated by nginx. It performs the following roles:
* TLS terminator
* HTTP/2 terminator
* Forward TLS & HTTP/2 metrics to varnish
= TLS termination =
TLS requirements for canonical domains and hence for the caching cluster are described at HTTPS. Puppetization happens in profile::cache::ssl::unified and more specifically in tlsproxy::localssl. The usual TLS configuration parameters (ciphersuites enabled and their preference order, DH parameters and optionally HSTS) happens in the puppet function ssl_ciphersuite.
== HSTS ==
HSTS header injection is performed by our varnish frontend layer in wikimedia-frontend.vcl.erb and it looks like this:
strict-transport-security: max-age=106384710; includeSubDomains; preload
== OCSP stapling ==
OCSP stapling is performed externally to nginx. So nginx just sends a prefetched OCSP stapling response that's obtained every 12 hours using the nginx configuration directive ssl_stapling_file and a custom patch to support dual-cert setups with prefetched OCSP stapling responses. A good post about why this is being done like this can be found in APNIC's blog.
OCSP stapling prefetching is performed by update-ocsp and it's puppetization lives in sslcert::ocsp::conf which of course is invoked in the already mentioned profile::cache::ssl::unified 
Category:Caching
Cadvisor

As of Aug 2023 all production hosts run cadvisor; therefore it is possible to query any systemd unit resource usage (at the very least CPU and Memory usage). See also for more context on this work.
The https://grafana.wikimedia.org/d/lxIVOKq4k/units-resource-usage-overview allows you to inspect a given cluster resources broken down by unit name.
For in-cluster metrics of k8s-managed services (which also use cadvisor) see Kubernetes/Metrics.
Canary

canary in Wikimedia technology may refer to:
* The debug servers, e. g. mwdebug1002, are sometimes called canaries. scap backport will deploy changes to them first and then wait for confirmation before deploying them to the rest of production.
* Scap has the notion of canary hosts: when deploying a change, it will first deploy it to a subset of servers, then pause for a bit and check their error rate. If enough canary servers experience a significantly higher error rate, the deployment is automatically aborted (though the canary servers continue to serve traffic with the deployed code).
* group0 wikis are sometimes called canary wikis. Problems with a new MediaWiki version may be discovered there before they start affecting the bigger wikis.
* canary events. These are artificially injected into WMF Event Platform streams to differentiate between empty streams and broken streams.
* cloudvirt-canary Cloud VPS project, running a test VM on each Cloud VPS hypervisor.
(All four meanings refer to the idiom of a canary in a coal mine.)
Category:Disambiguation pages
Capella

capella is (as of April 2015) currently spare in codfw
== History ==
Called mobile1 in pmtpa, which was a Dell PowerEdge 1950 running Ubuntu 8.04Special:Diff/47163, handling all requests for *.wikipedia.org from mobile devices, renamed to capella in T80523
Wikimedia IPv6 tunnel relay (6to4/Teredo) (role::ipv6relay) in pmtpa, eventually decommissioned in the tampa migration T83126, physically removed in T83687
solr3 (Dell PowerEdge R420, Dual Intel Xeon E5-2440 (2.40 GHz), 16 GB Memory, 300GB SSDSpecial:Diff/152418) renamed to capella in codfw in T84901 and racked (solr3 had been shipped from sdtpa to eqiad(?) in T83652)
Used in T84902 for OrientDB testing
Decommissioned again in T94474
Cassandra/PrometheusJmxExporter

The Prometheus JMX Exporter is a JVM agent that creates an HTTP listener to export MBean metrics for Prometheus.
== Source == 
Wikimedia maintains a Github branch of the repository, with the Maven pom.xml patched to deploy the jar to archiva.wikimedia.org.
== Deployment ==
The agent is deployed from https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/prometheus_jmx_exporter via Scap.
=== Updating the deploy repo ===
The deploy repository makes use of git-fat to sync the jar file to a working copy from Archiva, as necessary. If you need to deploy a new version of the strategy, the basic steps are:
# Build, and deploy to Archiva with Maven.
$ cat ~/.m2/settings.xml
<settings>
 <servers>
 <server>
 <id>wikimedia.releases</id>
 <username>archiva-deploy</username>
 <password>PASSWORD_GOES_HERE</password>
 </server>
 <server>
 <id>wikimedia.snapshots</id>
 <username>archiva-deploy</username>
 <password>PASSWORD_GOES_HERE</password>
 </server>
 </servers>
</settings>
$ mvn deploy
...
# Update the operations/software/prometheus_jmx_exporter with the jar
$ cp /path/to/exporter/jmx_prometheus_javaagent/target/jmx_prometheus_javaagent-<version>.jar lib/
$ git add lib/jmx_prometheus_javaagent-<version>.jar
$ git commit -m 'Adding jmx_prometheus_javaagent-<version>.jar via git-fat'
#: Note: Make sure that the jar you add to git is the same one uploaded in #1 (checksums should be matching). See Adding an artifact to your project for additional information.
# Deploy to the Cassandra nodes (see https://doc.wikimedia.org/mw-tools-scap/)
Cassandra/Staging

To obtain developer access, follow the usual process for submitting an access request.
After obtaining access, you can run cqlsh by using sudo and the surrogate user cassandra_devel. This user has been pre-configured for access using a dedicated Cassandra role (also named cassandra_devel).
user@home: ~$ ssh cassandra-dev2001.codfw.wmnet
user@cassandra-dev2001:~$ sudo -u cassandra_devel cqlsh cassandra-dev2001-a
Connected to cassandra-dev at cassandra-dev2001-a:9042
[cqlsh 6.1.0 | Cassandra 4.1.1 | CQL spec 3.4.6 | Native protocol v5]
Use HELP for help.
cassandra_devel@cqlsh>
As cassandra_devel you can read, write, and delete to all tables (not just those owned by you or your team), so be a good neighbor.
Cassandra/TimeWindowCompactionStrategy

TimeWindowCompactionStrategy is an alternative to DateTieredCompactionStrategy, developed by Jeff Jirsa. It ships with Cassandra 3.0.7 and later, (where DateTieredCompactionStrategy has been officially deprecated). For Cassandra versions < 3.0.7, it is possible to install time-window compaction out-of-tree.
== Source == 
Wikimedia maintains a Github branch of the repository, with the Maven pom.xml patched to deploy the jar to archiva.wikimedia.org.
== Deployment ==
The compaction strategy is deployed from https://gerrit.wikimedia.org/r/#/admin/projects/operations/software/cassandra-twcs via Scap.
=== Updating the deploy repo ===
The deploy repository makes use of git-fat to sync the jar file to a working copy from Archiva, as necessary. If you need to deploy a new version of the strategy, the basic steps are:
# Build, and deploy to Archiva with Maven.
$ cat ~/.m2/settings.xml
<settings>
 <servers>
 <server>
 <id>wikimedia.releases</id>
 <username>archiva-deploy</username>
 <password>PASSWORD_GOES_HERE</password>
 </server>
 <server>
 <id>wikimedia.snapshots</id>
 <username>archiva-deploy</username>
 <password>PASSWORD_GOES_HERE</password>
 </server>
 </servers>
</settings>
$ mvn deploy
...
# Update the operations/software/cassandra-twcs with the jar
$ cp /path/to/twcs/target/TimeWindowCompactionStrategy.jar lib/cassandra-v2.2
$ git add lib/cassandra-v2.2/TimeWindowCompactionStrategy.jar
$ git commit -m 'Adding TimeWindowCompactionStrategy.jar via git-fat'
#: Note: Make sure that the jar you add to git is the same one uploaded in #1 (checksums should be matching). See Adding an artifact to your project for additional information.
# Deploy to the Cassandra nodes (see https://doc.wikimedia.org/mw-tools-scap/)
# Update the Cassandra classpath entries as necessary (cassandra::extra_classpath in Puppet hiera)