nyuuzyou/wikis huggingface dataset:
Single Sign On

Wikimedia Production Web UIs use Single Sign On (SSO) (via Apereo CAS) for user authentication. The SSO portal is hosted at https://idp.wikimedia.org/login.
This page will help you understand what you need to know to log in (AKA authenticate) and use these UIs.
More detailed documentation can be found at CAS-SSO.
== What you need ==
To log in, you'll need
* A Wikimedia Developer Account (this creates an LDAP account for you).
* Membership in certain LDAP groups.
Usually, membership in either the wmf or nda LDAP group will suffice to authorize you to access most UIs. To request membership in an LDAP group, file a task in Phabricator with the LDAP-Access-Requests tag.
== Enabling 2FA using a YubiKey ==
WMF is slowly in the process of securing the SSO login process using 2FA. As of 2021-06, our only supported form of 2FA is with a hardware device using Universal 2nd Factor (U2F). WMF ITS will issue YubiKeys for this purpose.
===== Getting a YubiKey =====
# Figure out which YubiKey product is right for you.
# Email ITS at techsupport@wikimedia.org and ask them to send you a YubiKey, linking the product you'd like.
# ITS will purchase and ship you a YubiKey.
====== Using your YubiKey ======
# Open a Phabricator ticket with the LDAP-Access-Requests tag to ask SRE to enable YubiKey authentication.
# Once done
## Insert your YubiKey into your computer's USB port.
## logout of SSO (if you are already logged in)
## log back in. You will be prompted to enable authentication with your YubiKey.
From here on out, whenever you authenticate to the SSO portal you will be prompted to authenticate with your YubiKey after entering your username and password.
== What sites are SSO enabled? ==
As of 2021-06:
* https://alerts.wikimedia.org
* https://icinga.wikimedia.org
* https://librenms.wikimedia.org
* https://graphite.wikimedia.org
* https://superset.wikimedia.org
* https://people.wikimedia.org/~cdanis/sremap/
* https://piwik.wikimedia.org
* https://puppetboard.wikimedia.org
* https://rt.wikimedia.org
* https://tendril.wikimedia.org
* https://thanos.wikimedia.org
* https://turnilo.wikimedia.org
* https://yarn.wikimedia.org
Slow require once

Here are the syscalls made on the second and subsequent invocations of require_once("Setup.php"):
stat64("/usr/local/apache/common/php-1.5/Setup.php", 0xbfffafe0) = -1 ENOENT (No such file or directory)
stat64("/usr/local/apache/common/php-1.5/includes/Setup.php", {st_mode=S_IFREG|0664, st_size=8763, ...}) = 0
getcwd("/home/wikipedia/common/php-1.5/maintenance", 4096) = 43
lstat64("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/local", {st_mode=S_IFDIR|S_ISGID|0755, st_size=4096, ...}) = 0
lstat64("/usr/local/apache", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/usr/local/apache/common", {st_mode=S_IFLNK|0777, st_size=22, ...}) = 0
readlink("/usr/local/apache/common", "/home/wikipedia/common", 4096) = 22
lstat64("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat64("/home/wikipedia", {st_mode=S_IFDIR|0755, st_size=16384, ...}) = 0
lstat64("/home/wikipedia/common", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
lstat64("/home/wikipedia/common/php-1.5", {st_mode=S_IFDIR|0775, st_size=12288, ...}) = 0
lstat64("/home/wikipedia/common/php-1.5/includes", {st_mode=S_IFDIR|0775, st_size=16384, ...}) = 0
lstat64("/home/wikipedia/common/php-1.5/includes/Setup.php", {st_mode=S_IFREG|0664, st_size=8763, ...}) = 0
open("/home/wikipedia/common/php-1.5/includes/Setup.php", O_RDONLY) = 8
fstat64(8, {st_mode=S_IFREG|0664, st_size=8763, ...}) = 0
fstat64(8, {st_mode=S_IFREG|0664, st_size=8763, ...}) = 0
lseek(8, 0, SEEK_CUR) = 0
lseek(8, 0, SEEK_SET) = 0
close(8) = 0
Smokeping.wikimedia.org

smokeping.wikimedia.org was the Smokeping installation for Wikimedia. Smokeping is a network latency grapher. smokeping has been replaced by a Prometheus blackbox exporter based solution. See also https://phabricator.wikimedia.org/T169860 for more context and the following replacement dashboards:
* Network ICMP probes
* Network probes overview
* Network DNS probes
== External links ==
* smokeping.org, official website
Category:Services
Snuggle

Snuggle is a workflow tool for helping Wikipedian mentors find goodfaith newcomers to support in Wikipedia. 
== User documention ==
w:en:WP:Snuggle
== Code ==
github.com/halfak/snuggle 
== Development == 
TODO
== Deployment == 
TODO
== Orchestration ==
Start a medium VM with the lvm role for mounting at /srv. Then run these commands:
# Create directories in /srv
sudo mkdir /srv/snuggle/
sudo chown www-data:www-data /srv/snuggle/
sudo mkdir /srv/log
sudo mkdir /srv/log/snuggle
sudo chown www-data:www-data /srv/log/snuggle
# Install debian packages and initialize virtualenv
sudo apt-get install virtualenv build-essential python-dev python-numpy python-scipy libmysqlclient-dev uwsgi-plugin-python mongodb
sudo -u www-data virtualenv -p $(which python2.7) /srv/snuggle/venv --system-site-packages
# Clone repository
sudo -u www-data git clone https://github.com/halfak/snuggle snuggle/deploy
sudo -u www-data /srv/snuggle/venv/bin/pip install -r /srv/snuggle/deploy/requirements.txt
# Write out uwsgi configuration
sudo bash -c 'echo "[uwsgi]
chdir=/srv/snuggle/deploy
die-on-term=true
http-socket=0.0.0.0:8080
logformat=[pid: %(pid)] %(addr) (%(user)) {%(vars) vars in %(pktsize) bytes} [%(ctime)] %(method) %(uri) => generated %(rsize) bytes in %(m$
logger=file:/srv/log/snuggle/main.log
master=true
max-requests=200
memory-report=true
need-plugins=python
plugins=python, logfile
processes=32
venv=/srv/snuggle/venv
wsgi-file=/srv/snuggle/deploy/snuggle-en.wmflabs.wsgi" > /etc/uwsgi/apps-available/55-snuggle.ini'
sudo ln -s /etc/uwsgi/apps-available/55-snuggle.ini /etc/uwsgi/apps-enabled/55-snuggle.ini
# Set up mongodb
sudo mkdir /srv/snuggle/mongodb
sudo chown mongodb:nogroup /srv/snuggle/mongodb
sudo mkdir /srv/log/mongodb
sudo chown mongodb:nogroup /srv/log/mongodb
sudo cp /etc/mongodb.conf /etc/mongodb.backup.conf
sudo bash -c 'cat /etc/mongodb.backup.conf | grep mongodb | \
 sed "s*dbpath=/var/lib/mongodb*dbpath=/srv/snuggle/mongodb*" | \
 sed "s*logpath=/var/log/mongodb/mongodb.log*logpath=/srv/log/mongodb/mongodb.log*" > \
 /etc/mongodb.conf'
sudo service mongodb restart
sudo -u www-data mongo snuggle /srv/snuggle/deploy/docs/schema/indexes.js
sudo -u www-data mongo snuggle /srv/snuggle/deploy/docs/schema/sorting.js
# Start up the web service
sudo service uwsgi restart
# Start up the sync service. This should probably done in a screen -- or better an init script. 
sudo -u www-data /srv/snuggle/venv/bin/python /srv/snuggle/deploy/syncd /srv/snuggle/deploy/config/snuggle-en.wmflabs.yaml /srv/snuggle/deploy/config/enwiki.mediawiki.yaml
Solr

Solr is a Lucene-based search engine. WMF used it for translation memory and spatial searches, but switched to Elasticsearch.
== Puppet ==
modules/solr + manifests/role/solr.pp
== Operations ==
=== How to poke it with a stick ===
Checking if it's OK:
 maxsem@fenari:~$ curl http://solr1001:8983/solr/admin/cores?action=STATUS
 <?xml version="1.0" encoding="UTF-8"?>
 <response>
 <lst name="responseHeader"><int name="status">0</int><int name="QTime">2</int></lst> (loads of XML)
Checking if search works:
 $ curl 'http://solr1001:8983/solr/select/?q=*%3A*&version=2.2&start=0&rows=10&indent=on'
 (loads of XML)
Checking replication status
 $ curl http://solr1001:8983/solr/replication?command=details
Get schema remotely:
 $ curl 'http://solr1001:8983/solr/admin/file/?contentType=text/xml;charset=utf-8&file=schema.xml'
Restart 
 # service jetty restart
=== Upgrading schema ===
* ☠Delete existing documents☠:
 $ curl 'http://localhost:8983/solr/update?commit=true&stream.body=%3Cdelete%3E%3Cquery%3E*%3A*%3C%2Fquery%3E%3C%2Fdelete%3E'
* Stop the server:
 # service jetty stop
* Update schema.xml e.g. by forcing a puppet run:
 # puppetd -tv
* Reindex the data
=== Logs ===
* If jetty was restarted, it moves today's logs to *.<some number>
In /var/log/jetty:
* request.log - requests
* stderrout.log - errors and debug information
These logs are prefixed with "year_month_day.".
Examples:
* 2012_12_19.stderrout.log.100139764
* 2012_12_20.request.log
=== GeoData ===
==== Disabling updates ====
GeoData is updated every 30 minutes by cronjob on terbium. You can disable it by creating /tmp/disable-update-geodata on that server. Deleting the file will reenable updates.
==== Switching master ====
* Disable updates (see above). Wait some time in case update cronjob was already running.
* Change replication_master in mainfests/role/solr.pp. When deploying this change, it's preferrable to start with the new master.
* Update $wgGeoDataSolrMaster and $wgGeoDataSolrHosts in CommonSettings.php. Don't forget to reduce the weight on new master to avoid spikes during replication being too high.
== Packages ==
We use our own backports:
$ dpkg -l *solr*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-==========================-==========================-====================================================================
ii libsolr-java 3.6.0+dfsg-1 Enterprise search server based on Lucene - Java libraries
ii solr-common 3.6.0+dfsg-1 Enterprise search server based on Lucene3 - common files
ii solr-jetty 3.6.0+dfsg-1 Enterprise search server based on Lucene3 - Jetty integration
== Links ==
* Documentation (pretty crappy)
Category:How-To
Category:Search
Category:Decommissioned services
SonarQube

SonarQube is an open platform to manage code quality.
== Why another tool? Isn't PHPLint, puppet-lint, unit test, ... enough? ==
SonarQube is not a replacement for those tools, it is a way to improve visibility on what those tools are already reporting or it can replace some of them with its own analysis.
== Key Features ==
* Persistence: SonarQube brings persistence to you metrics, which brings a way to see their evolution over time.
* Quality Gates: Quality Gates let you define rules on what quality is acceptable. Compared to most tools, those rules can be of higher level and be aware of the history of the project. For example, we can define a rule that "Number of critical issues since last version has not increased", or "Unit tests on new code must have a branch coverage > 40%".
== Improving Quality ==
Most of our code has a history. Not always a happy one. We have ton of code that is ugly, but which has been working just fine in production for the last X years. Investing on that code is probably not worth it. We should instead focus on new code, on code that we touch regularly.
== Proof of Concept ==
A good demo is worth more than any Wiki page. A PoC SonarQube instance is available on labs.
== Analyzing Code ==
# You need the SonarQube scanner installed (see link at the top of its documentation page).
# create a `sonar-project.properties` file at the root of your project:
sonar.host.url=http://sonarqube-poc.wmflabs.org/
sonar.projectKey=<ID of your project>
sonar.projectName=<name of your project>
sonar.projectVersion=<version of your project>
sonar.sources=<path to the sources>
# export SONAR_RUNNER_HOME=/opt/sonar-scanner-2.5/
# run the analysis with sonar-runner
There are a lot more options to tune where your sources are, exclusions of some directories, integration of unit test results, ... but this should get you started.
== IDE integration ==
To get faster feedback during development, there are plugins for Eclipse, IntelliJ and CLI.
== Where do we go from here? ==
If there is any interest in moving forward, there are a few things we could do (in no particular order):
* install SonarQube on a real database
* integrate authentication with LDAP
* run analysis as part of continuous integration
* integrate analysis of unit tests
* configure quality profiles and quality gates
* integration with mediawiki/vagrant
* ...
Category:Bot and monitoring
Category:Software
Squids deployment

The 10 new Squid servers (single opterons with 4 disks) have been deployed as follows.
FIRST remove audit, as it blocks everything during a Yum upgrade on FC4:
 # yum remove audit
Add the domain pmtpa.wmnet to /etc/resolv.conf:
 domain pmtpa.wmnet
Add the /home NFS mount line to /etc/fstab:
 zwinger:/home /home nfs bg,soft,rsize=8192,wsize=8192,timeo=14,intr,nfsvers=3 0
Set up LDAP:
 # /home/wikipedia/bin/setup-ldap
Copy root SSH keys from zwinger's /root/.ssh
Remove standard bloat services:
 # for s in cups gpm; do /sbin/chkconfig --del $s; done
NTP
Initialize squid spool filesystems using:
 # mkfs -t reiserfs -s 0 device name
Add /etc/fstab lines:
 /dev/sda2 /a reiserfs nolog,notail 1 2
 /dev/sdb1 /b reiserfs nolog,notail 1 2
 /dev/sdc1 /c reiserfs nolog,notail 1 2
 /dev/sdd1 /d reiserfs nolog,notail 1 2
Create squid spool directories using:
 # for i in a b c d; mkdir /$i/squid && chown squid.squid /$i/squid && chmod 0750 /$i/squid; done
Stat1002

The stat1002 server was an Analytics Cluster client node for accessing Analytics MariaDB databases or Hadoop. It was replaced in July 2017 by stat1005.https://lists.wikimedia.org/pipermail/wiki-research-l/2017-July/005947.htmlhttps://lists.wikimedia.org/pipermail/analytics/2017-July/005955.html
It was intended to be the private webrequest access log storage host. Users should not use it for app development. Data processing on this machine is fine.
== Deprecation ==
== Usage ==
Users in the groups analytics-privatedata-users can log in here.
* Access to Hadoop. (Since May 2016, it has been suggested to use stat1004 for this instead.)
* Access Analytics MariaDB databases (EventLogging databases and research slaves of MediaWiki databases).
* Access to raw JSON data for EventLogging (/srv/eventlogging).
== Services ==
* stat1002 houses our Hive instance which provides access to various tables, including aggregated traffic data in pageview_hourly and projectview_hourly. 
* stat1002 provides access to Analytics/Requests stream.
* stat1002 provides access to Analytics/Mobile requests stream.
* stat1002 provides access to Analytics/Zero requests stream.
== See also ==
* Analytics/Data access#Stats machines
* Analytics/Data access#Host access granted
== Notes ==
Stat1003

stat1003 is an analytics server for accessing the Analytics MariaDB databases. Successor to Stat1.
== Deprecation ==
As of July 2017, use of stat1003 is deprecated. Please use stat1006 instead. https://lists.wikimedia.org/pipermail/wiki-research-l/2017-July/005947.html
== Usage ==
* Access Analytics MariaDB databases (EventLogging databases and research slaves of MediaWiki databases).
* Access to raw JSON data for EventLogging (/srv/eventlogging).
== Services ==
* Slave replication monitoring by User:QChris is run on stat1003.
== See also ==
* SSH host fingerprint
* Analytics/Data access#Stats machines
Stat1004

stat1004 is an Analytics Cluster client node for accessing Hadoop. 
== Usage ==
Users in the groups analytics-users or analytics-privatedata-users can log in here.
* Access to Hadoop. (stat1007 may be used for this as well, but stat1004 is dedicated for just Hadoop usagehttps://lists.wikimedia.org/pipermail/analytics/2016-May/005166.html)
== See also==
* Analytics/Systems/Clients
== Notes ==
Stat1008

The stat1008 server is a Data Platform client.
== GPU ==
stat1008 has an AMD GPU slated for use with training machine learning models. As of September 2019 the GPU is only available for users of the gpu-testers POSIX group. If you wish to get added to the group, please contact the Analytics team. For more info. see T148843.
Users in the group gpu-testers can use the AMD GPU that we deployed for T148843. The long term plan is to make it available for all the users logging in, but for the moment its access is restricted for a limited set of users to better test it (and avoid usage contention etc..). Please reach out to the Analytics team if you wish to get added to the group to test the GPU for your use case.
Static-codereview.wikimedia.org

static-codereview.wikimedia.org is the home of the static HTML dumps for the MediaWiki CodeReview system that was used during the subversion era. For the MediaWiki and pywikipedia SVN repositories, you can view commit diffs, reviewer comments and status changes.
The domain is hosted on a ganeti VM and configured using the profile::microsites::static_codereview puppet profile. The dump was generated on 2020-05-12 using the codereview-archiver script.
Static test notes

http://sage.knams.wikimedia.org/static/articles/s/e/r/Server_admin_log.html
unclosed tags -- are we running through tidy?
http://sage.knams.wikimedia.org/static/index.html
funkiness, possibly also unclosed tags
http://sage.knams.wikimedia.org/static/index.html and all other pages
missing logo
http://sage.knams.wikimedia.org/static/index.html
"current revision" tab links to localhost
other pages even link to the wrong page title, eg http://localhost/view/1.4_upgrade from http://sage.knams.wikimedia.org/static/articles/c/o/n/Help~Contents_22de.html
http://sage.knams.wikimedia.org/static/articles/w/h/a/Special~Whatlinkshere_Template~Document_me_2b96.html
special page links from sidebar are broken
http://sage.knams.wikimedia.org/static/articles/m/a/i/Main_Page_8ef9.html
^ search jump to 'Main Page' doesn't bring up the file, as it's actually at index.html
http://sage.knams.wikimedia.org/static/articles/s/e/r/Server_Roles_f792.html
^ search jumps to exact matches only and jumps to a 404 if you misspell anything
(ideal would be to have some sort of index page we can go through if the target match doesn't exist)
http://sage.knams.wikimedia.org/static/articles/1/2E/4/Talk~1.4_upgrade.html
red links on talk tabs bring up 404
http://sage.knams.wikimedia.org/static/articles/b/a/c/Category~Background_ccd5.html
bogus search links at top (from category page which doesn't have page text but does have members)
Statsd

statsd is a metrics aggregation server. We use it to aggregate metrics for Graphite. It flushes data points to Graphite at an interval of 60 seconds (the highest resolution supported by our Graphite configuration), otherwise only the last data point in a given minute would be stored. Statsd also adds various properties to the data such as .rate, .median, .p95, etc. More about that at Graphite#Extended properties.
== Service ==
Wikimedia currently uses the statsite implementation (package, website), written is a C program that is wire-compatible with Etsy's original Statsd (written in Node.js).
Several statsite deamons are hosted on the main Graphite servers, with a local load-balancer in front of them (statsd-proxy). You should access it through the canonical hostname "statsd.eqiad.wmnet".
== History ==
Use of Statsd was first deployed in 2015 using wikimedia/statsdlb, a StatsD-compatible load-balancer written in C by Ori Livneh. This ran in front of armon/statsite, a C implementation of statsd.
In 2016, the wikimedia/statsdlb service was replaced by hit9/statsd-proxy for performance reasons (T126447).
== See also ==
* Graphite
* statsite documentation
* etsy/statsd documentation (more elaborate)
* Metric types
Category:Services
Category:Metrics
Category:SRE Observability
Storage1

==History==
* 2007-09-19: Replaced disk 13.
*2007-03-15: Replacement server arrives from SM. Racked and loading FC4 on it to test that it works.
*2007-02-22: Attempted to do the HDD tests required by SM to replace server due to HDD slowdowns. Emailed SM to get the entire thing replaced, there have been too many issues.
*2007-02-19: Replaced PDUs and Mainboard for the issue of the system not wanting to reboot without pulling all the power and plugging it back in.
**Attempting to load Ubuntu shows the system experiencing the HDD throttling issue it first had out of the box. An email has been sent to SM about this on 02/17/07.
*2007-02-09: Loading Ubuntu.
Concerning the slow raid:
Kyle,
We will track this issue under Ticket: #15298.
It looks like the most recent storage server purchase you made was under 
order #S2025934 [SM24266, SM24267] for 2 x K300s.
We may have seen the issue you are describing to us before. The internal 
host drives have a feature where if there is too much vibration in the 
server the write speeds will slow way down in order to prevent damage to 
the platters. With these storage servers you would notice this issue if 
the server rack is vibrating more than usual or if the exhaust fans in 
the rear of the chassis are causing too much vibration. I would like to 
target the fans first.
If you are able to, please install lm_sensors on one of the storage 
servers and send us the fan speed output. This will allow us to narrow 
it down to which version of the exhaust fans your servers have in the 
rear of the chassis.
Thanks in advance,
Jonathan McQuistion
Silicon Mechanics Support
Tests: (All outcomes are the result of two tests averaged together.
 Tests done with FC5 CD in rescue mode. Raid controller in question is a 3ware 8006 with Raid 1 on 2 80GB drives)
Storage1
W/O Exhaust Fans and no raid fans (Only CPU Fans)
 dd if=/dev/zero of=test bs=1024 count=10000000 <-- 106MB/s
 dd if=test of=/dev/null <-- 241MB/
W/O Exhaust Fans - (3 Raid fans and CPU fans)
 dd if=/dev/zero of=test bs=1024 count=10000000 <-- 98MB/s
 dd if=test of=/dev/null <-- 223MB/s
W/ Exhaust Fans - (All fans on)
 dd if=/dev/zero of=test bs=1024 count=10000000 <-- 81MB/s
 dd if=test of=/dev/null <-- 252MB/s
With Exhaust fans and Storage 2 powered on. (Its racked below Storage1)
 dd if=/dev/zero of=test bs=1024 count=10000000 <-- 72MB/s
 dd if=test of=/dev/null <-- 237MB/s
If there is caching taking place, is consistent. (Whatever is cached with the fans off, 
the same thing is cached with the fans on, and its slower)
There are definitely slower writes taking place on this raid controller with more fans. 
Obviously we cannot function without fans.
Storm control

=What is Storm control?=
Storm means excessive broadcast, multicast or unicast traffic received on a particular interface in 1 second. Storm control on the other hand is the mechanism put in place to monitor and take actions such as to drop or shut down the interface when the storm control level is exceeded.
=How to setup Storm control?=
Right now, the goal is to run Storm control on both main management switches in codfw and eqiad (msw1-codfw and msw1-eqiad).
To setup Storm control the first step is to create a storm control profile with the name wmf-mgmt-storm and setup the bandwidth-level of 15000kbps. Next create an interface range called mgmt-switches and to finish, assign members to that interface range and apply the storm control profile.
==Create the storm control profile== 
 set storm-control-profiles wmf-mgmt-storm all bandwidth-level 15000kbps
==Create the interface range==
 set interfaces interface-range mgmt-switches unit 0 family ethernet-switching storm-control wmf-mgmt-storm
==Add members to the interface-range==
 set interfaces interface-range mgmt-switches member-range ge-0/0/0 to ge-0/0/31
==Add action-shutdown==
 edit
 edit forwarding-options storm-control-profiles wmf-mgmt-storm
 set action-shutdown
 commit
=View traffic statistics=
Pick two interfaces from the interface range ge-/0/0/0 to ge-/0/0/31 and in operational mode issue the command
 show interface interface_name detail 
We are going to just focus on the line that says "Input bytes"
For this example, we are going to use ge-0/0/1 and ge-0/0/16
 show interfaces ge-0/0/1 detail
 ---
 Traffic statistics:
 Input bytes : 8698005972 560 bps
 ---
 show interfaces ge-0/0/16 detail
 ---
 Traffic statistics:
 Input bytes : 16074924932 1392 bps
 ---
We can see that the ingress traffic rate in bits per second for both interface is within the storm control limit of 15,000 Kbps (15000000 bps)
=Restore the interfaces after a shutdown=