nyuuzyou/wikis huggingface dataset:
Sender Policy Framework

SPF is a framework for validating outgoing mail, which gives the receiving side useful information for spam filtering. The main goal is to cause spoofed @wikimedia.org mail to be correctly identified as such. It should also improve our odds of getting fundraiser mailings into inboxes rather than spam folders. 
The October 8 change will be simply a matter of adding a TXT record to the wikimedia.org DNS zone:
wikimedia.org IN TXT "v=spf1 ip4:91.198.174.0/24 ip4:208.80.152.0/22 ip6:2620:0:860::/46 include:_spf.google.com ip4:74.121.51.111 ?all"
The record is a list of subnets that we identify as senders (all wmf subnets, google apps, and the fundraiser mailhouse). The "?all" is a "neutral" policy--it doesn't state either way how mail should be handled.
Eventually we'll probably bump "?all" to a stricter "~all" aka SoftFail, which tells the receiving side that only mail coming from the listed subnets is valid. Most ISPs will route 'other' mail to a spam folder based on SoftFail.
I was under the impression that ~all softfail is not an assertion that something is not authorized and the only way to actually assert that is with -all hardfail.
Category:Mail
Category:Fundraising
Server Lifecycle/reclaim checklist

This checklist is able to be copied and pasted into phabricator hardware request tasks for reclaiming systems to spare or decom.
[] - all system services confirmed offline from production use
[] - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
[] - remove system from all lvs/pybal active configuration
[] - any service group puppet/hiera/dsh config removed
[] - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)
START NON-INTERRUPPTABLE STEPS
[] - disable puppet on host
[] - remove all remaining puppet references (include role::spare)
[] - power down host
[] - disable switch port
[] - switch port assignment noted on this task (for later removal)
[] - remove production dns entries
[] - puppet node clean, puppet node deactivate
END NON-INTERRUPPTABLE STEPS
[] - system disks wiped (by onsite)
[] - IF DECOM: system unracked and decommissioned (by onsite), update Netbox with result
[] - IF DECOM: switch port configration removed from switch once system is unracked.
[] - IF DECOM: mgmt dns entries removed.
[] - IF RECLAIM: system added back to spares tracking (by onsite)
Server history

* 2001-05-02 Wikipedia moves to a new server
* 2001-08-25 Wikipedia is on 64.78.205.6
* 2001-09-15 ross is 64.78.205.6, serving wikipedia-l
* 2001-12-27 major migration from one server facility to another one. Definitely to San Diego, perhaps from Chicago?
* 2002-01-05 Wikipedia is on 130.94.122.195
* 2002-04-02 Wikipedia, Nupedia and the mailing lists are on ross.bomis.com, which is 130.94.122.195
* 2002-07-03 Pliny ready for testing on address 130.94.122.197
* 2002-07-20 Switchover to pliny
So the most likely story is that ross was serving Wikipedia from 2001-05-02 until 2002-07-20, when pliny took over.
* 2003-04-29 Larousse ordered
* 2003-05-14 Larousse switched on
* 2003-06-20 pliny/larousse is still owned by Bomis after the creation of the Foundation. (Bomis was later taken over by Tim Shell.)
* 2004-01-13 Initial batch of 9 servers ordered
* 2004-02-12 Moved from San Diego to Tampa
* 2007-03-18: larousse is still in the rack in Florida.
* 2012-12-06: In response to a request from a museum, Christopher Johnson reports "Larousse is no longer onsite in Tampa. The server was donated last year along with most of the silicone mechanic servers Rob mentioned."
* 2012-12-12: Tim Shell reports: "I don't know if I ever saw Pliny. Ross is around somewhere but I cannot say it is in a very good state."
Server replacements mid 2008

Servers that are out of warranty, and the services they are running:
; srv1 : LDAP (being phased out), NIS (critical) sdtpa
:* internal ip
; srv2 : NIS (critical) pmtpa
:* internal ip
; srv9 : donate.wikimedia.org, civicrm sdtpa
:* external ip
; suda (temp running on db20): /home NFS (critical) sdtpa and pmtpa
:* internal ip
:* internal ip
; bayle : DNS (critical) sdtpa
:* external ip
; khaldun : Ubuntu mirror, installation server (critical) sdtpa
:* external ip
; zwinger : deployment (critical), ganglia (critical), NTP (critical), dhcpd (to be taken over by khaldun) sdtpa
:* external ip
; will : Network monitoring pmtpa
; browne : IRC, udpmcast, toolserver tunnels sdtpa
; anthony : Untrusted services (WAP) sdtpa
; isidore : bugzilla, blogs (critical), SVN proxy for CodeReview sdtpa
; bart : Nagios (critical), OTRS (critical), secure (critical), planet (critical) sdtpa
; yongle : Apple dictionary search, mobile gateway, PDF export sdtpa
; mobile gateway : (?) (merge with anthony replacement?) sdtpa
Hosts with strike through are merged into a new setup and do not need replacement by itself.
Service/Etcd

== Etcd ==
=== Description ===
etcd (https://etcd.io/) is an open source key-value store with a focus on reliability that is used to store configuration and state data for distributed systems. At WMF we run a number of etcd clusters, this document addresses the two etcd Main clusters, one each installed in the primary datacenters, eqiad and codfw. A number of applications, including mediawiki read/write configuration store state data on etcd.
=== Categories ===
Relevant service categories (wiki categories) for grouping by similar services, owner, etc.
=== Service Type ===
Etcd is a foundational service
=== Service Dependenies === 
No hard dependencies beyond hardware and networking. It is worth pointing out that server hardware and networking have their own failure rates that are in the 99% range. Etcd as configured is able to deal with a certain type of failures in a local datacenter.
=== Related Services ===
Confd: a lightweight configuration management tool focused on keeping local configuration files up-to-date using data stored in etcd
=== Ownership ===
Etcd is owned by the Service Operations SRE team, which is responsible for all aspects including operation, scalability, backups and software updates.
Technology Department / Site Reliability Engineering / Service Operations
*Escalation points and Key contacts:
** Code: etcd.io - Service Operations
** Data persistence: Service Operations
** Release pipelines: Service Operations
** Metrics, logging, alerting, tracing: Service Operations
** Product: Service Operations
** Security: Service Operations
** Systems & Infrastructure: Service Operations
=== Supporting documentation and relevant information ===
* Design documents
* Operational documentation
* Phabricator component query links
* Netbox links
* LibreNMS
* Icinga
* Links to other relevant SRE Tooling™
* Links to Runbooks
* Related service request types
* Any supporting or underpinning services (e.g. dependencies)
* Who is entitled to request/view the service
ServiceOwnership

As a result of the mobile gateway at en.m becoming more and more popular it has become necessary to draft some guidelines about what it is required from our dedicated operations staff to better support the service. The role of these guides would be a quick go to for anyone needing to administer the service. Separate pages could be used for anything more specific.
At a bare minimum the following criteria would be required:
== Software ==
* Software Stack
** OS if relevant
** LAMP
** Scary Salaris
Should be brief as this will be a operations guide not a detailed service spec
== Hardware ==
* What hosts do we live on? 
Links should follow to the hardware pages of the servers
This can get out of date really quickly
== Usage == 
* Who are the clients?
== Health ==
* Be able to easily identify the state of the application and if its meeting its sla's
== Capacity ==
* Can capacity be easily added and if so how?
== Dependencies ==
* Clearly identify what other services this piece of software requires and what other services require it
** What happens if the service is intermittent or goes down hard
== Start/Restart ==
* How do we bring it up and down
* How do we do migrate quickly if the server is on fire
== Configuration ==
* Where do they exist live and in source control
* What (additional) actions are needed to make configuration changes go live
== Typical Problems ==
* What commonly breaks
** And how are we fixing it
this should not be a section of this sucks and we just live with it
== Activity ==
* How do I see running activity of the service
** Logs
** Process stats
** Black Magic
== Monitoring ==
* How do I track the history of the services lifetime
== Backups ==
* How are backups of any data handled, next to the configuration of the service itself.
== Escalation Path ==
* Who do I call when the problem is far more then what I can handle
== Reference Links ==
* Pointers to more detailed installation/configuration instructions if needed
Template is available at Template:ServiceOperations
Service Catalog

service 	purpose 	number of machines	multi-dc 	runs on 	TLS behind ATS 	owner team 	tickets / workboard	puppet role
planet 	RSS feed aggregator 	2 	yes, active/passive	ganeti 	yes, envoy 	serviceops / dzahn 	wikimedia-planet 	role(planet)
install_servers 	OS installions 	2 	yes, active/active 	ganeti 	no, public 	 	 	role(installserver::light)
APT repos 	providing software packages 	2 	yes, active/passive	ganeti 	no, public 	 	 	role(apt_repo)
Icinga 	monitoring other services 	2 	yes, active/passive	metal 	no, monitoring tool	observability 	icinga 	role(alerting_host)
Phabricator 	issue tracker, repos, pastebin et.al	2 	yes, active/passive	metal 	 	releng / serviceops 	phabricator 	role(phabricator)
Gerrit 	code review 	2 	yes, active/passive	metal 	no, public 	releng / serviceops 	gerrit 	role(gerrit)
changeprop 	 	 	 	kubernetes 	 	 	 	role(scb)
cpjobqueue 	 	 	 	kubernetes 	 	 	 	role(scb)
graphoid 	 	 	 	kubernetes 	 	 	 	role(scb)
mobileapps 	 	 	 	kubernetes 	 	 	 	role(scb)
recommendation_api 	 	 	 	kubernetes 	 	 	 	role(scb)
apertium 	 	 	 	kubernetes 	 	 	 	role(scb)
acme_chief 	providing TLS certificates 	2 + 2 test 	 	ganeti 	 	traffic-team 	 	role(acme_chief)
hadoop 	 	 	 	metal & ganeti	 	analytics-team 	 	role(analytics_cluster::hadoop::master) et al
hue / yarn 	UI for hadoop 	 	 	ganeti 	 	analytics-team 	 	role(analytics_cluster::hadoop::ui)
turnilo 	 	 	 	ganeti 	 	analytics-team 	 	role(analytics_cluster::turnilo)
airflow 	 	 	 	ganeti 	 	 	 	role(search::airflow)
zookeeper 	 	 	 	ganeti 	 	analytics-team 	 	role(analytics_cluster::zookeeper)
presto 	 	 	 	metal 	 	analytics-team 	 	role(analytics_cluster::presto::server)
analytics query service (AQS)	 	 	 	metal 	 	analytics-team 	 	role(aqs)
archiva 	 	 	 	metal 	 	analytics-team 	 	role(archiva)
auth DNS 	 	 	 	metal 	 	traffic-team 	 	role(dns::auth)
backup 	 	 	 	metal 	 	data-persistence-team	 	role(backup)
bastionhost 	 	 	 	metal 	 	 	 	role(bastionhost::general), role(bastionhost::pop)
syslog 	 	 	 	metal 	 	 	 	role(syslog::central_server)
cescout 	 	 	 	 	 	 	 	role(cescout)
cloudceph 	 	 	 	 	 	 	 	
configcluster 	 	 	 	 	 	 	 	role(configcluster)
NFS 	 	 	 	 	 	wmcs 	 	role(wmcs::nfs::*)
contint 	 	 	 	 	 	releng,serviceops 	 	role(ci::master)
Services/MW-Vagrant

MediaWiki Vagrant is a convenient way for developers to rapidly set up a development environment containing a MediaWiki instance and any needed dependencies in a virtualised environment. Your service needs to be present there as well. Luckily, setting it up is very easy. First, in the vagrant directory, create the directory for your service's module and place this code inside <vagrant-dir>/puppet/modules/<service-name>/manifests/init.pp:
# == Class: <service-name>
#
# <a-short-description-of-the-service-here>
#
# === Parameters
#
# [*port*]
# Port the service listens on for incoming connections.
#
# [*log_level*]
# The lowest level to log (trace, debug, info, warn, error, fatal)
#
class <service-name>(
 $port,
 $log_level = undef,
) {
 service::node { '<service-name>':
 port => $port,
 log_level => $log_level,
 config => {},
 }
}
This is the minimum amount of code your service's Puppet module should have. As you can see, this definition does not provide any extra configuration for the service. If that is needed, simply add the configuration stanzas to the config hash as key/value pairs. Note that only configuration specific to your service should be listed here and not the whole configuration file, i.e. only the configuration parameters that your service code accesses via app.conf.*.
In order to configure the port (and any other parameters that you might have declared for the class), add the following contents to puppet/hieradata/common.yaml:
<service-name>::port: <service-port>
The last step is to create the role so that users may (de)activate it easily. Place the following Puppet code in puppet/module/role/<service-name>.pp:
# == Class: role::<service-name>
# This role installs <service-name>
#
class role::<service-name> {
 include ::<service-name>
}
Finally, the service's port must be exposed to the host environment; create the file puppet/modules/role/settings/<service-name>.yaml with:
forward_ports:
 <service-port>: <service-port>
You are done! You can now submit the patch for review and anybody will be able to profit from the service in the MediaWiki-Vagrant environment.
Services/Monitoring/tilerator

tilerator is a Maps service for generating map tiles from OSM data in postgres and then storing them in cassandra. The service being down could be an indication of issues with either postgres or with cassandra. Individual instances of tilerator being down will affect maps throughput but will not cause a service outage on their own. 
Check /srv/log/tilerator/main.log and /srv/log/tilerator/syslog.log for information about service performance and to help debug what is going on with the service.
Services/Operation

== Starting, Stopping, Restarting ==
If you have sudo rights on the target machines, then that's as simple as logging onto each of the targets and issuing the respective commands:
$ sudo service <service-name> start
$ sudo service <service-name> stop
$ sudo service <service-name> restart
== Monitoring ==
=== Logs ===
The service's logs are stored locally in /srv/log/<service-name>/main.log. To take a look, simply tail it:
$ tail -f /srv/log/<service-name>/main.log
Since the log entries are JSON-formatted, you may want to see them in a more presentable form. Use bunyan for that:
$ tail -f /srv/log/<service-name>/main.log | /srv/deployment/<service-name>/deploy/node_modules/.bin/bunyan
 
Category:How-To
SessionStorage

Session storage provides application servers with shared access to user session data, making this service critical to site functionality for logged in users. It utilizes Kask, a RESTful opaque key/value storage abstraction on top of Apache Cassandra, deployed in an active/active configuration in both the eqiad and codfw data-centers.
== Production Environment ==
=== Service endpoints ===
==== Discovery ====
https://sessionstore.discovery.wmnet:8081/sessions/v1
==== eqiad ====
https://sessionstore.svc.eqiad.wmnet:8081/sessions/v1
==== codfw ====
https://sessionstore.svc.codfw.wmnet:8081/sessions/v1
==== staging ====
https://staging.svc.eqiad.wmnet:8081
==== Prometheus metrics ====
https://sessionstore.discovery.wmnet:8081/metrics
==== Readiness ====
https://sessionstore.discovery.wmnet:8081/healthz
=== Cassandra ===
Given the sensitivity of the information stored, a dedicated Cassandra cluster is used.
== Monitoring and debugging ==
Sessionstore has a Grafana dashboard. 
Failures in sessionstore operation are very likely to have an impact upon user sessions in a variety of ways. Reports of CSRF token failures are a frequent giveaway. Spiking errors of save failures on the Edit Count dashboard and spiking MediaWiki login errors are also a common sign of issues with sessions.
== Common Tasks ==
=== Deployment ===
==== k8s ====
Follow the Kubernetes documentation for the sessionstore service. 
==== Deployment-prep ====
Unlike production, Docker is used to deploy the session storage service in the deployment-prep environment. 
In a web browser:
# Visit https://horizon.wikimedia.org
# Select the deployment-prep project
# Select Project → Compute → Instances (in the sidebar)
# Locate and select the deployment-sessionstore06 instance
# Select the Puppet Configuration tab
# Scroll to the bottom of the page, and click edit for Hiera Config (see below)
# Click "Apply Changes"
# Run sudo run-puppet-agent from deployment-sessionstore06
===== Sample Hiera Config =====
profile::cassandra::instances:
 deployment-sessionstore06.deployment-prep.eqiad1.wikimedia.cloud: {}
profile::docker::engine::declare_service: true
profile::docker::engine::settings: {}
profile::docker::engine::version: 1.12.6-0~debian-jessie
profile::docker::runner::service_defs:
 mediawiki-services-kask:
 config:
 base_uri: /sessions/v1
 cassandra:
 authentication:
 password: cassandra
 username: cassandra
 hosts:
 - deployment-sessionstore06.deployment-prep.eqiad1.wikimedia.cloud
 keyspace: sessions
 table: values
 default_ttl: 86400
 listen_address: 0.0.0.0
 listen_port: 8080
 openapi_spec: /go/src/gerrit.wikimedia.org/r/mediawiki/services/kask/openapi.yaml
 service_name: sessions
 namespace: wikimedia
 override_cmd: ./kask --config /etc/mediawiki-services-kask/config.yaml
 port: 8080
 version: v1.0.10
=== Pool and Depool ===
== See also ==
* Kask
Category:Data Persistence
Setup a new network device

Checklist to not get lost when setting up a new network device
=== Initial setup ===
NOTE: We can now use ZTP to handle many of these steps for Juniper QFX-series switches. See ZTP Network Device Provision for more info.
* Add to Netbox (dc-ops)
** Include serial number and asset tag
** Set Netbox status to 'planned'
* Connect power/console/mgmt
* Update console server's port (& test it)
* Assign mgmt IP and loopback IP(s) in Netbox
* Run DNS cookbook to add reverse DNS entries
* Add base config
** Management IP
** Static route to management router (in mgmt vrf where appropriate)
** Auth setup (user(s), ssh enabled)
* Add to Homer
* Push initial configuration
* Upgrade to latest recommended OS version
* Save rescue configuration
** request system configuration rescue save
* Set Netbox status to 'active'
* Run the sre.puppet.sync-netbox-hiera cookbook to import location to hiera
=== Monitoring ===
* Add to LibreNMS - LibreNMS#Add a device to LibreNMS
** You can set the "Disable alerting" and "Ignore alert tag" in the device's setting if it's not ready for prime time.
* Add to Rancid (Puppet: modules/rancid/files/core/router.db)
* Add to Icinga and Prometheus
** Puppet: hieradata/common.yaml - infra_devices
** Puppet: hieradata/common/monitoring.yaml 
** Puppet: modules/netops/manifests/monitoring.pp
Shell Request

:See Help:Accessing Cloud VPS instances for accessing Cloud Services instances.
:See Production shell access for accessing Wikimedia production servers.
Prior to mid-2015, new Cloud VPS users had to separately ask for "shell access" and be manually approved as members of the bastion project before they could become a member of another Cloud VPS project (like Toolforge). The subpages of this page are the requests and record of their approval/denial from this process.
See Special:PrefixIndex/Shell Request/ for the ~5,000 subpages. Or search:
type=fulltext
prefix=Shell Request/
searchbuttonlabel=Search requests
break=no
Showcase

==Background==
A lot of great tools are being developed on Labs but not everybody is aware of them or knows how they work. The Labs Showcase gives the opportunity to tool / labs developers to demonstrate their work and give short tutorials. Intended audience is everyone who wants to use Labs more and is just curious to see what's happening.
==Proposal==
;Length: 30 minutes
;Frequency: biweekly
;Inaugural showcase: first week of January 2014.
;Time: TBD, maybe using a Doodle or something similar. 
;Medium: Google Hangouts so we can publish the showcase to youtube. I realize that not everybody is a big fan of Google Hangouts and I am definitely open to alternatives but I think the ability of Hangouts to record a showcase is very useful and we know from past experience that it's working pretty well with larger groups of people.
==Participation==
Yes, I think this is a great idea and would attend the showcases on a regular basis!
* BryanDavis (talk) 19:00, 18 December 2013 (UTC)
* I'd at least watch the first few, and hopefully they'd hold my interest. Anomie (talk) 13:47, 19 December 2013 (UTC)
* --zfilipin (talk) 15:20, 19 December 2013 (UTC)
* ...
Not on a regular basis but i would watch some archives if they were on youtube
* tomasz (talk) 00:09, 19 December 2013 (UTC)
* would be glad to watch or listen rubin (talk) 12:22, 1 January 2014 (UTC)
* --Rutu (talk) 23:35, 1 January 2014 (UTC)
Category:Wikimedia infrastructure
Similarusers

Similarusers (formerly known as "Sockpuppet" or the "Sockpuppet API") is a service running in Kubernetes. It can be used to identify relative similarity between users on wikis. It was developed by the Research team, productionised by the Platform Engineering team, and currently used in the SimilarEditors extension; see that page for installation instructions and documentation. 
Development documentation can be found in the service README file. 
It runs on port 4110.
== What it does ==
The similar-users service uses analyses of the Dumps to determine how similar given users seem to be based on how and when their edits took place. 
== How it works ==
On a monthly basis as new dumps are released, a PySpark notebook runs on the Analytics cluster. This notebook analyses all edits and generates summaries of temporal and coedit data (see the similar-users README for more details on the process), and the results are stored in MariaDB. When the similar-users service receives a request for a particular user, it queries the database and the Mediawiki API, and then uses this information to gather information about similar users, which is then returned as JSON.
==== Service Documentation ====
The service is documented using the OpenAPI format.
A documentation endpoint is available at /apidocs. This is the authoritative documentation source. Developers can access the endpoint locally by running the service docker container:
 git clone "https://gerrit.wikimedia.org/r/mediawiki/services/similar-users"
 make docker
Visit
 http://0.0.0.0:5000/apidocs
The endpoint is protected by BasicAuth. Credentials are specified by the BASIC_AUTH_USERNAME and BASIC_AUTH_PASSWORD variables in the config file at
 similar_users/flask_config.yaml
See README for more information.
== Where it runs ==
The service will run in Kubernetes in the staging, codfw and eqiad clusters 
== Deploying ==
The similar-users service is deployed using the standard Helm deployment process. 
== Testing ==
== How it's monitored ==
== Related pages ==