nyuuzyou/wikis huggingface dataset:
Swift/Server layout

Category:Swift
= Swift Server Hardware, Partitioning, and OS information =
There are two types of servers involved in a Swift cluster - the front end proxy (ms-fe, standing for media store front end) and the back end storage node (ms-be).
== ms-fe ==
The front end proxies are running on high performance misc servers built with:
* two 250GB disks
* 16GB RAM
The disks are partitioned using the raid1-250G-1partition.cfg partman recipe:
* 200GB partition on each disk software RAID 1 for /, ext3
* the rest for two swap partitions (one on each disk)
== ms-be ==
The back end storage nodes are running on Dell R720xd machines with:
* twelve 2TB disks 
* two ssds 150GB each (on some hosts)
* 48GB RAM
For hosts with ssds, the disks are partitioned such that
* the last two disks have a 55GB partition for /, set up with software RAID 1, ext3
* the last two disks have a small (1GB) partition for swap
* the last two disks have a small (93GB) partition formatted for account and container storage
* all other disks are formatted with one large XFS partition for swift object storage.
The reason for this is that the ssds show up as /dev/sdm and /dev/sdn instead of /dev/sda and /dev/sdb, because of where they can be cabled up in the box.
Hosts without ssds are partitioned such that
* the first two disks have a 60GB partition for /, set up with software RAID 1, ext3
* the first two disks have a small (1G) partition for swap
* the first two disks have a large (the rest, or ~1.95T) partition formatted for object storage
* all other disks are formatted with one large XFS partition for swift object storage.
The partman recipe (ms-be-ssd.cfg or ms-be.cfg) only partitions the first or last two disks and doesn't create the XFS partition. Puppet partitions the rest.
------
The below applies only to previously installed hosts; these will soon be replaced.
The back end storage nodes are running on Dell C2100 machines with:
* twelve 2TB disks
* 48GB RAM
The disks are partitioned such that
* the first two disks have a small (1MB) partition for grub
* the first two disks have a 250GB partition for /, set up with software RAID 1, ext3
* the first two disks have a small (4G) partition for swap
* the first two disks have a large (the rest, or ~1.75T) partition formatted for object storage
* all other disks are formatted with one large XFS partition for swift object storage.
The partman recipe (raid1-2TB-1partition.cfg) only partitions the first two disks and doesn't create the XFS partition. Puppet partitions the rest.
Swift/Setup New Swift Cluster (labs)

This document is incomplete - it is a list of differences from Swift/Setup New Swift Cluster rather than a complete set of instructions.
Any heading that is empty should be used unaltered from the regular setup page.
==== update DNS ====
skip this step
==== Launch the instances ====
new step
* Create a project for your cluster (you may need help from a labs admin for this - ask in IRC
* Create a security group for the proxy in your project
** accept traffic on port 80 (from everyone or a specific other project or ...)
* Launch the instances for your cluster - minimum one front end proxy and 3 back end storage nodes
** m1.small is sufficient as the instance size
** put the proxy in the proxy security group
==== Set up filesystems ====
Do this on all back end storage nodes
* unmount /dev/vdb from /mnt
* reformat it as xfs
** sudo mkfs -t xfs -f -i size=512 -L swift-vdb /dev/vdb
* make the mountpoint swift expects
** mkdir -p /srv/swift-storage/vdb
* mount the partition with the appropriate options
** sudo mount /dev/vdb /srv/swift-storage/vdb/ -o rw,noatime,nodiratime,nobarrier,logbufs=8
* set the ownership to swift
** sudo chown -R swift:swift /srv/swift-storage/vdb
==== create the cluster hash ====
==== update puppet ====
==== build the rings ====
When building the rings, there's only one device per storage node - /srv/swift-storage/vdb. This simplifies adding devices to the rings a little bit. This example has four storage nodes. Everything else about building the rings remains the same.
for num in {1..4}; do 
 host="swift-be${num}.pmtpa.wmflabs"; 
 hostip=$(dig +short $host); 
 zone="z${num}-${hostip}"; 
 weight=100; 
 for dev in vdb; do
 sudo swift-ring-builder account.builder add z${zone}:6002/${dev} $weight;
 sudo swift-ring-builder container.builder add z${zone}:6001/${dev} $weight;
 sudo swift-ring-builder object.builder add z${zone}:6000/${dev} $weight;
 done;
done
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance
chown swift:swift *.ring.gz 
==== distribute the rings ====
==== reboot ====
skip this step. instead:
* sudo swift-init all stop; sleep 1; sudo pkill swift; sleep 1; sudo swift-init all start
==== set up auth tokens for the cluster ====
==== tell puppet about the auth tokens ====
==== update the proxies and restart the proxy service ====
==== make the containers necessary for thumbnails ====
==== make the containers readable by anonymous users ====
==== test the cluster ====
Category:Swift
Swift/Thumbnails Cleanup

Media thumbnails are generated on demand (of arbitrary width) and stored in swift, however MediaWiki doesn't know about the full list of widths being requested by users. Further, thumbnails are never cleaned up from swift, thus as part of Techops FY2016-2017 goals there has been investigation on how to periodically and effectively clean up thumbnails stored in swift.
To better understand what widths are in swift vs what the users request we have walked all thumbnails stored and imported in hive. For end-users requests a subset of webrequest data in hive has been used to drive decisions. Finally, data extracted has been summarized in this google spreadsheet. See also phabricator task T162796 for additional context.
Analyzing webrequest data showed that some prerendered widths were rarely requested but stored nevertheless (2560/2880 pixels wide). Thus prerendering for those widths was stopped and all thumbnails larger than 2000px cleaned up.
== Collect webrequest data ==
A subset of webrequest to gather insight on how thumbs are requested
create table filippo.webrequest_upload_thumb_width_201704 stored as parquet as select regexp_extract(webrequest.uri_path, '.*/.*?(\\d+)px-.*', 1) as width, response_size, cache_status, uri_path from wmf.webrequest where uri_path like '%/thumb/%px-%' and webrequest_source='upload' and year = 2017 and month = 4;
# export back to plaintext on hdfs
insert overwrite directory '/user/filippo/webrequest_upload_201704' row format delimited fields terminated by ' ' select width, count(width) as count, avg(response_size) as avg_size from filippo.webrequest_upload_thumb_width_201704 group by width;
== Collect swift data ==
Use thumbstats/swift-thumb-stats --hive_export to get a list of thumbnails present in swift and ready to be imported in hive for further processing and analyzing.
Once criteria for deletions are determined, the list of matching thumbnails in swift can be exported from hive and fed to thumbstats/swift-thumb-cleanup for the actual deletion. e.g. below for width greater than 2000px.
insert overwrite directory '/user/filippo/thumbs_to_delete_notcommons' row format delimited fields terminated by ' ' select container,path from thumbstats where container not like '%commons%' and width > 2000;
insert overwrite directory '/user/filippo/thumbs_to_delete_commons' row format delimited fields terminated by ' ' select container,path from thumbstats where container like '%commons%' and width > 2000;
== Further cleanup strategies ==
The first cleanup for widths greater than 2000px is the one that yielded the most space savings in the short term. Other cleanup strategies have been identified in the Phabricator, including storing only the top100 widths and having Thumbor process the rest instead.
Category:Swift
Switch Datacenter/DeploymentServer

This page describes the procedure to switch over the active deployment server from one host to another.
This is suitable for use either as part of the regularly scheduled datacenter switchover or more generally when replacing the active deployment server with a newer one.
In either case, you will want to schedule an exclusive window in the Deployments calendar (at least one hour) and coordinate with potential deployers.
=== Procedure ===
* Disable puppet on deployment servers. From a cluster-management host:
sudo cumin 'A:deployment-servers' 'disable-puppet "Deployment server switchover - TXXXXXX"'
* Merge a DNS change that points the deployment.eqiad.wmnet CNAME record to the new active host, see this change for an example.
** Apply your changes as described in DNS#Deploying_DNS_changes and remember to !log your change.
* Change deployment_server and scap::deployment_server variables in hiera, see this change for an example.
* Run puppet on the new active deployment server: sudo run-puppet-agent -e "Deployment server switchover - TXXXXXX"
* Run puppet on all the other servers (same command).
* Using sudo cumin 'A:deployment-servers' 'grep block_deployments /etc/scap.cfg' (on a cluster-management host), verify that:
** Deployments on the previously active server are blocked
** Deployments on the newly active server are not blocked
* Since keyholder configuration might have changed but not been reloaded on the newly active deployment server when it was a spare, restart the keyholder service there and test it:
 $ sudo systemctl restart keyholder-proxy.service
 $ SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i /etc/keyholder.d/deploy_jenkins -l deploy-jenkins releases1002.eqiad.wmnet
* Workaround for : Run the following on all deployment servers after replacing the deployment server URLs accordingly. For example, if switching from deploy1003 to deploy2002:
 $ sudo -i
 # find /srv/deployment -name DEPLOY_HEAD | xargs sed -i "s/git_server: deploy1003.eqiad.wmnet/git_server: deploy2002.codfw.wmnet/"
* Test a scap deployment, noting that this may take quite some time on the first attempt: scap sync-world "Test deployment to validate deployment server switchover - TXXXXXX". This will also test helmfile deployments.
* Email ops@ about the switch of active deployment server and update any IRC channels where the ongoing work is being tracked.
* Update Deployment_server to reflect the change in active server.
Switch Datacenter/MemcachedRedisSessions

This page describes the procedure to switch over safely the redis/memcached cluster in the process of switching mediawiki.
When in need to switch over mediawiki, the replication flow of redis must change.
The simplest and safest way to do this now is as follows:
# Stop puppet on all hosts of this cluster cumin 'A:memcached' 'puppet agent --disable "pre-switchover"'
# Switch the puppet configuration so that the primary datacenter is now the new one
# Run puppet on the machines in the new master datacenter (so that the replica stops flowing)
# Re-enable puppet in the whole cluster; the machines in the other datacenters will eventually catch up
Switches/Vlan-change

csw1:
 int range gi0/7 , gi0/29 , gi0/33
csw4:
 int range gi0/28 , gi0/37 , gi0/40 - 41
csw5:
 vlan 100
 untagged ethe 1/23 to 1/42 ethe 2/3 ethe 2/18 to 2/22 ethe 2/37 to 2/38 ethe 2/41
TCP Fast Open

== Metrics ==
/proc/net/netstat provides a bunch of TFO-related metrics:
* TCPFastOpenActive: number of successful outbound TFO connections
* TCPFastOpenActiveFail: number of SYN-ACK packets received that did not acknowledge data sent in the SYN packet and caused a retransmissions without SYN data. Note that the original SYN packet contained a cookie + data, this is not the number of connections to servers that didn’t support TFO
* TCPFastOpenPassive: number of successful inbound TFO connections
* TCPFastOpenPassiveFail: number of inbound SYN packets with TFO cookie that was invalid
* TCPFastOpenCookieReqd: number of inbound SYN packets requesting TFO with TFO set but no cookie
* TCPFastOpenListenOverflow: number of inbound SYN packets that will have TFO disabled because the socket has exceeded the max queue length
Other interesting metrics are:
* TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down retransmissions into SYN, fast-retransmits, timeout retransmits, etc.
* TCPOrigDataSent: number of outgoing packets with original data (excluding retransmission but including data-in-SYN). This counter is different from TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is more useful to track the TCP retransmission rate.
== Detecting server-side TFO support ==
The initial portion of the 3WHS can be used to check whether a remote TCP server supports TFO. For example, with scapy we can craft a SYN packet with the TFO option set and check whether the SYN/ACK response from the server includes the TFO option as well. If that is the case, TFO is enabled on the target server.
from scapy.all import sr1, IP, TCP
dst = "en.wikipedia.org"
dport = 443
res = sr1(IP(dst=dst)/TCP(dport=dport,flags="S",options=[('TFO', )]), verbose=False)
print 'TFO' in dict(res[1].options)
== Python client using BSD sockets ==
#!/usr/bin/env python
import sys
import socket
TCP_FASTOPEN = 0x20000000
dest_name = sys.argv[1]
port = 80
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.sendto("HEAD / HTTP/1.0\r\n\r\n", TCP_FASTOPEN, (dest_name, port))
data = s.recv(1024)
s.close()
print data
== CLI tools ==
tshark allows to easily filter packets with TFO option set:
tshark -f 'tcp[tcpflags] & tcp-syn != 0' -Y 'tcp.option_kind == 34'
curl version 7.49.0 or greater allows establishing a TCP connection with TFO, albeit not in combination with TLS as of July 2016.
curl --tcp-fastopen http://en.wikipedia.org
== Server key generation ==
RAND=$(openssl rand -hex 16)
NEWKEY=${RAND:0:8}-${RAND:8:8}-${RAND:16:8}-${RAND:24:8}
echo "net.ipv4.tcp_fastopen_key=$NEWKEY" > /etc/sysctl.d/50-tcp_fastopen_key.conf
chmod 600 /etc/sysctl.d/50-tcp_fastopen_key.conf; chown root /etc/sysctl.d/50-tcp_fastopen_key.conf
sysctl -p /etc/sysctl.d/50-tcp_fastopen_key.conf
unset RAND NEWKEY
== References ==
* Shaving your RTT with TCP Fast Open
TLS/Runbook

This page contains service runbooks to handle (among other things) certificate expiration alerts.
==== CertAlmostExpired ====
Every probe using TLS also exports the certificate (chain) earliest expiration date. The certificate(s) presented during the probe are about to expire. Check the linked dashboard(s) for more information. For service::catalog probes the instance label is in the form of service:port where service is the service's key in the catalog.
===== fooservice:443 =====
The alert will link to this anchor for fooservice:443
===== swift-https:443 =====
Process is documented at Swift/How_To#Update_internal_TLS_certificates
===== gitlab:22 =====
The GitLab ssh service is not running (used for git and not ops access). Make sure to check the status of this service:
systemctl status ssh-gitlab
And start/restart the service:
systemctl start ssh-gitlab
# or
systemctl restart ssh-gitlab
Category:TLS
Td-agent-bit

Fluent Bit/td-agent-bithttps://fluentbit.io/ is an open source and multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations. 
== Packaging ==
td-agent-bit provides native debian packaging (I could not find debian source packages, though).
To build a new version on package build host:
VERSION=1.5.3
wget http://github.com/fluent/fluent-bit/archive/v${VERSION}.zip
unzip v${VERSION}.zip
tar cfz td-agent-bit_${VERSION}.orig.tar.gz fluent-bit-${VERSION}
cd fluent-bit-${VERSION}/
dch -v ${VERSION}-0 -D buster-wikimedia --force-distribution "New upstream version ${VERSION}"
DIST=buster pdebuild
Techblog.wikimedia.org

techblog.wikimedia.org is a WordPress deployment on the Enterprise WordPress hosting platform. See Wikimedia technical blog editorial guidelines for more information about the process of getting an article published on the blog.
== Local dev setup ==
See https://github.com/bd808/vvv-wmf-techblog-site-template
== GitHub repos ==
* https://github.com/wpcomvip/wikimedia-techblog
** Public repo that configures the site
** admins: @bd808, @reedy, @varnent
== Visitor privacy ==
We have tried to remove 100% of 3rd party resource loads and tracking. See phab:T247569 for details.
* Jetpack stats disabled via PHP code
* Webfonts for Modern theme loaded from local plugin
* New Relic perf tracker disabled via PHP code
* Gravatar user avatars disabled via WordPress admin settings
== Piwik for analytics ==
We turned off the Jetpack analytics module and are using Analytics/Systems/Matomo instead. See phab:T247918 for more info.
* https://piwik.wikimedia.org
* SiteId: 22 (TECHBLOG.WIKIMEDIA.ORG)
* Login: design / design
== Monitoring ==
* https://rpm.newrelic.com/accounts/2054423/applications/382238569
** Access must be granted by wpvip.com account managers
==History==
The Wikimedia Technical Blog was created in March 2009, hosted at techblog.wikimedia.org. It was powered by WordPress and hosted as a Wikimedia production service. It was created to allow for more in-depth engineering focus toward a more technical audience (distinct from the official Wikimedia Blog at blog.wikimedia.org). See also The first Techblog post with its original skin and appearance.Techblog is online., techblog.wikimedia.org, 13 March 2009 by Rob Halsell. (Archived.)Wikimedia Tech joins Wikimedia Blog, blog.wikimedia.org, 27 March 2009 by Jay Walsh.
In June 2011, the blogs were merged, with the Techblog posts under a new "Technology" category on the official Wikimedia Blog, and the techblog domain redirecting to https://blog.wikimedia.org/c/technology/. See also The last archived version of this incarnation of the Techblog.
In 2013-2014, the Wikimedia Blog was moved to be hosted by Automattic using WordPress.com VIP.Migrate blog off our systems, Phabricator. 
In 2019, the current incarnation of the Techblog started as a new WordPress blog hosted via WordPress.com VIP. The archived posts remain available as part of Wikimedia Blog instead, which in 2020 was rebranded to Wikimedia Diff at diff.wikimedia.org.Switch blog.wikimedia.org to diff.wikimedia.org, Phabricator. Wikimedia Diff includes in its archive the original Techblog posts, with redirects in-place.
==See also==
*Blogposts
*kohl.wikimedia.org (2004-2006)
==External links==
*https://techblog.wikimedia.org
Category:Services
Tendril

Tendril is a tool for analytics and performance tuning of the MariaDB servers, developed by Springle (with features added in an ad-hoc fashion as required) and used by the DBA team. For security reasons it is only accessible to people with NDA.
Prior to October 2015, the Ishmael tool provided a similar service.
As of late 2021, it is planned to deprecate Tendril in favour of Orchestrator.
== Service ==
* Front end (vanilla PHP) runs on dbmonitor1001,2001.wikimedia.org; scripts in /srv/tendril/
* Backend database presently on db1115.eqiad.wmnet (former DB was db1011.eqiad.wmnet) using MariaDB event scheduler and FEDERATEDX engine.
* Uses a pull method for monitoring with one watchdog database client connection to each host.
* Entirely and intentionally separate from ganglia and mediawiki-config to allow sanity checks both ways.
* Uses Labs LDAP for authentication (restricted to people with NDA).
== Features ==
It houses a lot of functionality, but here are a few highlighted features:
=== Reports ===
: https://tendril.wikimedia.org/report
There are many different report views, such as:
* Slow queries. E.g. Slow queries from MediaWiki requests, slow analytics queries (from user 'research')
* An ad-hoc cluster-wide INFORMATION SCHEMA view (mostly standard schema plus server_id field).
* Cluster-wide query process list close to real time (10s intervals).
=== Replication tree ===
Based on actual reported configuration (no dependence on coredb or wmf-config/db-eqiad.php)
* https://tendril.wikimedia.org/tree
=== Host viewer ===
: https://tendril.wikimedia.org/host
Information about each database host (e.g. IP, RAM, uptime) and interactive graphs and charts indicating their recent query activity and various InnoDB statistics.
Also linked to from most other reports.
== Service dependencies ==
Tendril, or more accurately, its host and mariadb database server instance (as of Oct 2019, db1115 is the active host for it), provides the following services, documented here so that on maintenance, we know they will be affected:
* Tendril database monitoring, as documented on this page and serving queries from url https://tendril.wikimedia.org
* Dbtree, public version of tendril tree view at https://dbtree.wikimedia.org
** Both dbtree and tendril are served by dbmonitor frontend, and use the tendril schema
* Zarcillo database inventory, used manually by DBAs (dashboard url to be developed, some cli tools on cumin hosts)
* Database backups metadata, started on cumin + dbprov hosts, and using zarcillo and alerting on backup status on icinga (db1115)
* Systemd unit on 2+2 Grafana servers on eqiad and codfw, automatically adding production databases to monitoring from zarcillo (alerting on icinga)
** These last 3 use the zarcillo schema
== See also ==
* MariaDB/monitoring
* Orchestrator
== External link ==
* https://tendril.wikimedia.org
Category:Services
Category:Bot and monitoring
Category:MySQL
Terbium

terbium was the maintenance script host in eqiad, a Dell PowerEdge R420. It was replaced by mwmaint1001.
== Connect ==
== Services ==
* dbtree.wikimedia.org
* noc.wikimedia.org
* people.wikimedia.org
* scripts: People log in here to run shell scripts that take a lot of time or CPU, or to run various MySQL queries. See How to run a maintenance script.
== See also ==
* deployment.eqiad.wmnet, for deployments.
* wasat, maintenance script host in Codfw.
Test.wikipedia.org

test.wikipedia.org is a test wiki inside the production cluster. It runs the same code as other Group 0 wikis.
== Purpose ==
test.wikipedia.org differs from test2.wikipedia.org in the following ways:
* Debug log instrumentation is enabled. Output from wgDebugLog() is enabled (see Logs).
== Details ==
=== Before 2016 ===
Until January 2016, traffic intended for test.wikipedia.org automatically got a X-Wikimedia-Debug header set at the Varnish layer. This made all its requests automatically route to the canary server (mw1017, as of July 2013) for easy debugging in production. As all debug traffic, Varnish caching is automatically disabled for those test.wikipedia.org requests in Varnish.
=== Before 2013 ===
Prior to 2013, the testwiki canary server ran the MediaWiki code in /home/wikipedia/common mounted over NFS from the deployment server (then fenari, now tin). As of June 2012, the application server for testwiki was srv193.
== See also ==
* Hwcp
* Beta
* Debugging in production
* MediaWiki deployment setup
* mw1017
* test2.wikipedia.org
== External link ==
* https://test.wikipedia.org
Category:Services
Test2.wikipedia.org

test2.wikipedia.org (created in 2011) is a test wiki inside the production cluster (like test.wikipedia.org). test2 is primarily used to trial and debug global and cross-wiki features in conjunction with test.wikipedia.org and test.wikidata.org. It runs the same code as other Group 1 wikis.
== History ==
As of January 2016 test.wikipedia.org and test2.wikipedia.org both get the "next" wmf-branch in the first phase.
test2 was originally created to ensure testability of all Het deploy states throughout the deployment phases (where test.wikipedia.org would be the first wiki to get the next version and test2.wikipedia.org the last – together with the Wikipedias).
== Details ==
test.wikipedia and test2.wikipedia are both served from the production cluster behind the main load balancers serving live scapped code from the apache servers.
== See also ==
* test.wikipedia.org
== External link ==
* https://test2.wikipedia.org
Category:Services