nyuuzyou/wikis huggingface dataset:
Proxy access to cluster

This page document how to access web services on private hosts.
Presently, there is no VPN access to Wikimedia Foundation's production cluster. Cloud VPS does not provide this either.
== Tunnel ==
The below example will expose a web service from analytics1001.eqiad.wmnet:8088 at your localhost as https://localhost:9088.
ssh -N bast1002.wikimedia.org -L 9088:analytics1001.eqiad.wmnet:8088
The management network being restricted to the cluster management hosts (), you can reach hosts with for example:
ssh -L 8000:scs-eqsin.mgmt.eqsin.wmnet:443 cumin1001.eqiad.wmnet
Pointing then your web browser to:
https://localhost:8000, will actually show you (in this example) https://scs-eqsin.mgmt.eqsin.wmnet
If you need to reach an http port (eg. 80), don't forget to use http://localhost:8000 instead and for example :80 in the ssh command.
== FoxyProxy ==
The preferred method by Ops for accessing the web interfaces on private nodes is via the -D option of SSH in combination with FoxyProxy.
This document will outline how to setup this method of access. This method uses Firefox and the FoxyProxy addon linked above.
# Install FireFox and FoxyProxy.
# Setup an ssh -D session for the cluster(s) you need access to. (Repeat this as needed.)
screen
ssh cumin1001.eqiad.wmnet -D 8080 #cumin hosts have full mgmt vlan access
# Ctrl+a c (creates new screen window)
# Ctrl+a d (disconnects you from the screen session, allowing you to close or otherwise use terminal)
# Now when you load up the url patterns in the FoxyProxy settings, it will direct those URLs via your SSH tunnel to the correct cluster over the specified ports.
# Please note that if you use OS X, you may want to 'exec ssh-agent bash' in your terminal screen sessions before adding your ssh key to ensure they are independently keyed and not shared across sessions.
Patterns to route to localhost:8080: 
 *.wmnet
 10.*
== See also ==
* Production shell access#SSH configuration
Category:How-To
PublicDashBoard

What public facing reporting do we want?
* Mail server status
* Editing Lag
* XML snapshots lag
* API healthfead
* Overall cluster health
* History
* wikitech
* irc
* backups (:))
* Latency
* Rss feed of any status updates
* Ability to use !commands on irc to report status from portal
== Examples of Dashboards ==
* Twitter
* Facebook
* Google Apps
* Amazon AWS
* Ebay
* SecondLife
Puppet-ecdsacert

It is now possible to generate general-purpose ECDSA certificates using our internal puppet CA and then use those for serving HTTPS traffic internally.
All of the following assumes you're operating on our puppet active CA server.
Usage is pretty simple:
puppet-ecdsacert -a SAN1,SAN2,SAN3 CERTNAME
will take care of generating the cert, creating the CSR, submitting it to the puppet CA and signing it.
You can control the output directories with a config file, but by default the private key will be at:
/var/lib/puppet/ssl/private_keys/CERTNAME.pem
and the public signed cert will be at 
/var/lib/puppet/server/ssl/ca/signed/CERTNAME.pem
Once you've generated those files, you should move the private key to the private puppet repo under /srv/private as a secret, and the public cert inside the main puppet repository with .crt extension.
=== Use in labs with non-central puppetmasters ===
If you're using a different puppetmaster in labs, you probably want to do something like this:
echo 'puppetca: deployment-puppetmaster04.deployment-prep.eqiad.wmflabs' > /tmp/ecdsacertconfig
puppet-ecdsacert -a "deployment-etcd-01.deployment-prep.eqiad.wmflabs,deployment-etcd-01,etcd.deployment-prep.eqiad.wmflabs" etcd.deployment-prep.eqiad.wmflabs -c /tmp/ecdsacertconfig
This is because otherwise, puppetca will default to 'puppet', which in labs resolves to the central puppetmaster. This will cause the CSR to be sent to the central puppetmaster but the puppet cert sign call at the bottom of puppet-ecdsacert to fail
== See also ==
* Special:Prefixindex/HTTPS
Category:Puppet
Puppet/Bootstrap

When installing the puppet server for the first time there is a bit of a bootstrapping step specifically because we want to use out own CA.
== Standard Puppetserver ==
* First move the puppetserver to the pupertserver role and run puppet on the host
* once puppet has run on the new server run puppet on the following hosts to update all access lists.
sudo cumin "O:puppetserver or C:puppetmaster or O:cluster::management or C:swift::ring_manager" "run-puppet-agent"
* manually run puppet-merge on the new server with the current puppet sha e.g. sudo puppet-merge.py -o 64db1c37de4649d48f69ce80303eb2260040ed9b
* sync a copy of the private repo from the primary puppet server i.e. where yuo edit files in the private repo
$ sudo su -s /bin/bash - gitpuppet
$ cd /srv/private/
$ GIT_SSH=/srv/private/.git/ssh_wrapper.sh git push ssh://puppetserver1002.eqiad.wmnet/srv/git/private master
* run the various sync jobs to make sure all data is in sync
$ sudo systemctl start sync-puppet-ca.service geoipupdate.service geoip_update_ipinfo.service geoip_update_main.service sync-puppet-volatile.service
$ sudo systemctl status sync-puppet-ca.service geoipupdate.service geoip_update_ipinfo.service geoip_update_main.service sync-puppet-volatile.service 
* finally add the new server to the SRV record
== Puppet CA ==
To initialise the CA you will need the intermidate certificate for puppet. the private key is in the private repo modules/secret/secrets/puppetserver/puppet_ca.key. The public key is in the puppet repo modules/profile/files/puppetserver/puppet_ca.pem.
I generated the csr with the following command on the pki-root server
$ cd /etc/cfssl/ssl
$ for i in * ; do openssl x509 -noout -in ${i}/${i}.pem -serial ; done
# copy the serials to series.txt one serial per line
$ cfssl gencrl serials.txt signers/Wikimedia_Internal_Root_CA/ca/ca.pem signers/Wikimedia_Internal_Root_CA/ca/ca-key.pem | base64 -d | openssl crl -inform DER -out crl.pem
You then need to run the following command before puppetserver has initialised its ca
$ sudo puppetserver ca import --cert-bundle /etc/puppet/puppetserver/custom_ca/ca.pem --private-key /etc/puppet/puppetserver/custom_ca/ca.key --crl-chain /etc/puppet/puppetserver/custom_ca/crl.pem
Import succeeded. Find your files in /etc/puppet/puppetserver/ca
you should be able to start the puppet server at this point
also as we have different ssl dirs to support the cloud use case we need to copy the created server certs to the 
The following may not be needed check configs
$ cd /var/lib/puppet
$ cp server/ssl/private_keys/hostname.pem ssl/private_keys/
$ cp server/ssl/certs/hostname.pem ssl/certs/
$ cp server/ssl/certs/ca.pem ssl/certs/
Puppet/Cleanups

Our Puppet tree is continuously evolving. As such, it's normal that some enhancement doesn't immediately trickle into the wider Puppet tree. This page tracks some enhancements to possibly apply when updating a given Puppet module:
=== Redundant file ownerships for root ===
In https://gerrit.wikimedia.org/r/c/operations/puppet/+/809095/1 a default ownership for File resources was added (root for user/group). As such, an existing File definition like
file { '/usr/local/sbin/memcached-dump':
 ensure => file,
 mode => '0550',
 owner => 'root',
 group => 'root',
 source => 'puppet:///modules/apereo_cas/memcached-dump.py',
}
can be simplified to 
file { '/usr/local/sbin/memcached-dump':
 ensure => file,
 mode => '0550',
 source => 'puppet:///modules/apereo_cas/memcached-dump.py',
}
In some corner cases it might still make sense to keep the explicit root ownership, e.g. if you want to be very explicit that some config file contains sensitive data or similar.
=== Installation of packages using ensure_packages ===
Packages can be installed via a package definition: https://www.puppet.com/docs/puppet/7/types/package.html. These definitions are not guarded against re-declaration, if e.g. apache gets installed by two classes applied to a role, this causes a redeclaration error in the second class. The cleaner solution is to use the ensure_packages define instead. It guards against declaring packages multiple times internally:
ensure_packages(['python3-flask', 'python3-semver'])
ensure_packages('rsync')
ensure_packages isn't well suited for packages which are dependent on an $ensure variable. While ensure_packages allows passing an ensure parameter, this would also needed to be passed from all ensure_packages call sites, as such for such a special case it's better to stick with a bare package declaration.
Puppet/Runbooks

= Puppet Disabled =
Puppet should not be disabled for extended periods of time as it causes issues with things like shourt live certificates. users should instead leave puppet enabled but remove it from service and disable 
alerting if there is on going development.
The puppet disabled alert triggers if puppet has been disabled on a node for one week or more. When this alert fires you should look at the disabled message which should be displayed in the Motd text but you can also see this via the disable lock file
$ jq .disabled_message $(sudo puppet config print agent_disabled_lockfile)
"test - jbond"
The message should include the username of the person that disabled puppet. if so contact the user and ask them if the agent can be enabled. If not try to contact someone from the service owner team cat /etc/wikimedia/contacts.yaml
Category:Puppet
Python/Wmflib

A Python package that contains custom modules to interact with the WMF production infrastructure.
It can be used in any script throughout the fleet as it doesn't require any special privilege to be run, as opposed to Spicerack and its Cookbooks and removes the need to re-implement each time the same functionalities over and over again.
For the current available modules see its documentation.
It's available both on PyPI as wmflib and apt.wikimedia.org as python3-wmflib and its source code lives on Gerrit in the operations/software/pywmflib repository.
When releasing a new version don't forget to upgrade any venv-based deployment (usually with the *-deploy repository) that use it, like for example Netbox.
Category:SRE Infrastructure Foundations
Quarry/Quarry maintenance and administration

Learn more about how Quarry is administered and maintained.
== Overview ==
This page contains links to pages related to the maintenance and administration of the Quarry service.
== Main points ==
Quarry is run as a service on a web and a few worker nodes. These nodes can be viewed from []
=== Web ===
Code is found in /srv/quarry, it is managed by git and can be updated with git pull and a service restart as follows:
systemctl restart uwsgi-quarry-web.service
=== Worker ===
Code is found in /srv/quarry, it is managed by git and can be updated with git pull and a service restart as follows:
systemctl restart celery-quarry-worker.service
== Quarry source code ==
* Quarry source code - View the Quarry source code on GitHub.
== Nova Resource==
* Quarry/SAL
Query killer

The query killer on production databases is supposed to stop queries on replicas that take more than 60 seconds. It is implemented in native MySQL, see events_coredb_slave.sql. Under high load, it may not function properly, see db-kill for how to kill slow queries manually in emergencies. 
Historically this was implemented as a 60s pt-kill job, when the number of slow queries on a replica grows beyond a threshold the slowest one above 60s is sniped to keep the box alive. It was introduced after query spikes caused outages in November 2013.
== Statistics ==
Each killed query is recorded in the ops.event_log table. Events are removed after 24 hours.
MariaDB [(none)]> use ops
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [ops]> select * from event_log where event like "wmf_slave_wikiuser_slow%" limit 5;
+-----------+---------------------+-------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| server_id | stamp | event | content |
+-----------+---------------------+-------------------------------+-----------------------------------------------------------------------------------------------------------------------+
| 171974878 | 2021-10-14 20:49:03 | wmf_slave_wikiuser_slow (>60) | kill 2459844921; SELECT /* SpecialRecentChangesLinked::doMainQuery */ rc_id,rc_timestamp,rc_namespace,rc_title,rc_m |
| 171974878 | 2021-10-14 20:49:03 | wmf_slave_wikiuser_slow (>60) | kill 2459844921; SELECT /* SpecialRecentChangesLinked::doMainQuery */ rc_id,rc_timestamp,rc_namespace,rc_title,rc_m |
| 171974878 | 2021-10-14 20:50:03 | wmf_slave_wikiuser_slow (>60) | kill 2459862622; SELECT /* SpecialRecentChanges::doMainQuery */ /*! STRAIGHT_JOIN */ rc_id,rc_timestamp,rc_namespac |
| 171974878 | 2021-10-14 20:51:03 | wmf_slave_wikiuser_slow (>60) | kill 2459882167; SELECT /* SpecialRecentChanges::doMainQuery */ /*! STRAIGHT_JOIN */ rc_id,rc_timestamp,rc_namespac |
| 171974878 | 2021-10-14 20:52:03 | wmf_slave_wikiuser_slow (>60) | kill 2459901127; SELECT /* SpecialRecentChanges::doMainQuery */ /*! STRAIGHT_JOIN */ rc_id,rc_timestamp,rc_namespac |
+-----------+---------------------+-------------------------------+-----------------------------------------------------------------------------------------------------------------------+
5 rows in set (0.000 sec)
Category:MySQL
RANCID

RANCID is a tool that can monitor configuration changes on network devices, and mail the diffs periodically. It's installed on netmon1003 from the (rather alpha-quality) Debian/Ubuntu packages.
Rancid configuration is in /etc/rancid/ and /var/lib/rancid/.
You can clone the rancid repo with:
git clone ssh://netmon1003.wikimedia.org:/var/lib/rancid/core/ rancid-configs
Manually testing Rancid has access to devices:
sudo -u rancid SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh rancid@<hostname>
== Equipment notes ==
=== Juniper ===
The following configuration works on JUNOS:
system {
 login {
 class rancid {
 permissions [ view view-configuration ];
 }
 user rancid {
 uid 2002;
 class rancid;
 authentication {
 ssh-rsa "ssh-rsa rancid public-key"; ## SECRET-DATA
 }
 }
 }
}
=== OpenGear ===
See Platform-specific documentation/Opengear Serial Consoles#Initial Setup
And https://opengear.zendesk.com/hc/en-us/articles/216369543-RANCID-Support
Category:Network
Category:Bot and monitoring
REST Gateway

The REST Gateway is an Envoy instance for proxying and rewriting requests that would have previously gone to RESTBase as part of our migration away from the platform. It is a subconfiguration of the API Gateway chart. It runs as a Kubernetes service and is accessible via https://rest-gateway.discovery.wmnet:4113/ 
== Services currently supported ==
The following is a non-exhaustive list of services where traffic is routed via the REST gateway (and so avoid use of RESTBase): 
* Proton
* Wikifeeds
* geo-analytics (AQS 2)
* device-analytics (AQS 2)
* media-analytics (AQS 2)
* page-analytics (AQS 2)
* editor-analytics (AQS 2)
* edit-analytics (AQS 2)
* commons-impact-analytics (AQS 2)
== Gateway configuration ==
=== Per-service configuration and traffic routing ===
The service configuration should be queried for the most up to date list of proxied services. These services are configured in the gateway but may not be configured to route public traffic, verify this by checking Puppet. The gateway-checkLua script configured on ATS instances checks request paths for /rest_v1/ URLs and, if they are in the config, route the requests to the configured gateway. 
=== Headers ===
The REST Gateway automatically adds a set of headers to all responses as part of the RESTBase compatibility layer. See git for an up to date list. The REST Gateway will also add a content-security-policy header only if the header is absent. 
=== Acceptance criteria ===
For a service to be routed to via the rest-gateway, certain criteria should be met before routing public traffic: 
* The service must have a discovery DNS record configured ($service.discovery.wmnet)
* The service must be running in both eqiad and codfw
* The service must emit either an etag or last-modified header
* The service must emit a cache-control header
If a URL is being migrated from one service to another, the headers should match as closely as is reasonable with particular attention paid to the above.
=== Testing ===
The REST Gateway can be reached directly from inside the infrastructure. For instance, to query wikifeeds availability path:
curl -v -H 'de.wikipedia.org' https://rest-gateway.discovery.wmnet:4113/de.wikipedia.org/v1/feed/availability
RFP NfSen Contractor

RFP for a contractor for NfSen
Goals 
* Have a working network monitoring system. Nfsen is a basic starting point but other solutions (like pmacct) are also okay. 
* Have a puppetized network monitoring instance that is running on labs with minimal operations staff involvement
* Have plugins working that easily 
:* analyze traffic per port
:* analyze traffic per destination AS
:* analyze traffic per intermediate AS
* Use jflow5 or jflow 10 (juniper routers with trio chips). Preferably jflow10 (inline jflow)
* Have a frontend graphing that is not difficult to understand (possibly integrating into our current graphite system).
* Maintain contact and email updates with main ops team. 
Stretch Goals
* Set up email alerts at 50, 75, and 90% full thresholds
* some sort of ddos detection
Outcome : 
You will produce a set of puppet manifests, and, if necessary, packages for Ubuntu 12.04 LTS that will install a network monitoring system with no outside work. All work will be done on Wikimedia's public git repo. This work will be open sourced and released on git.wikimedia.org, as well as possibly on the puppet forge.
Limitations
* No direct access to production
RT/Reporting issues

== How to report an issue via RT ==
RT (RequestTracker) is an issue tracker used by the Operations Team. You do not need an account to report an issue or follow issues you have reported. To follow others' issues you must either have an account that lets you view that queue or you must have been added to the specific ticket as a reporter or observer.
=== Reporting an issue ===
Send email to ops-requests@rt.wikimedia.org with a description of the issue. This should have at least the following:
* A short subject with good keywords describing the issue. For example, if the issue is with a specific host having issues, make sure the host's name is in the subject line.
* What symptoms are being observed
* If it's not a report of something broken but rather a task you want done, describe how soon it's needed/how urgent it is.
* Link to any bugzilla reports, wiki pages, email threads etc. as appropriate.
You will get an automatically generated reply with the ticket number included in the subject and in the body of the message.
=== Following an issue ===
Any time someone answers the ticket, you'll get an email with your original subject line and [wikimedia #nnn] prepended to it, indicating the number of the ticket in our queue. Replying to that subject line will make sure your follow-up emails are attached to the ticket.
When the ticket is resolved, you'll also get an email notification about it.
=== Who to ask about new tickets ===
Every week there is a designated person from the operations team who looks at all incoming tickets and does a basic triage, handing off tickets to folks who know or are working on specific areas and doing basic requests themselves. The person on duty is reachable via irc in the #wikimedia-operations channel and their irc nick is in the topic of that channel. For more information on irc, see m:IRC/Instructions.
Category:RT
Raid and MegaCli

== Raid setup at Wikimedia ==
RAID setups for database:
* raid-10
* 256k stripe
* writeback cache
* no read ahead
 How to set up the initial raid group
== Querying the RAID card ==
=== Get event logs ===
# installs megacli if not present and output logs to all.txt
sudo apt install -qqy megacli && sudo megacli -AdpAliLog -a0>all.txt
=== Logical device info ===
Things to look for here:
* Stripe size
* Cache policy
root@db1047:/a/sqldata# megacli -LDInfo -Lall -Aall
 
Adapter 0 -- Virtual Drive Information:
Virtual Disk: 0 (Target Id: 0)
Name:
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
Size:1713408MB
State: Optimal
Stripe Size: 256kB
Number Of Drives:2
Span Depth:6
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default
Exit Code: 0x00
root@db1047:/a/sqldata# 
=== Firmware, serial number, and boatloads of other default config ===
root@db1047:/a/sqldata# megacli -AdpAllInfo -aALL | less
Adapter #0
==============================================================================
 Versions
 ================
Product Name : PERC H700 Integrated
Serial No : 11L006J
FW Package Build: 12.10.0-0025
 Mfg. Data
 ================
Mfg. Date : 01/22/11
Rework Date : 01/22/11
Revision No : A00 
Battery FRU : N/A 
 Image Versions In Flash:
 ================
BIOS Version : 3.18.00_4.09.05.00_0x0416A000
FW Version : 2.100.03-1046
Preboot CLI Version: 04.04-010:#%00008
Ctrl-R Version : 2.02-0025
Boot Block Version : 2.02.00.00-0000
MORE...
== Icinga checks ==
As of 2017, check_raid python script only checks the state of the controller (Optimal, failed, rebuilding, etc.), the logical disks and the pysical disk state:
 sudo /usr/local/lib/nagios/plugins/check_raid [megacli]
 OK: optimal, 1 logical, 2 physical
An optional parameter was introduced to also check the write cache policy:
 sudo /usr/local/lib/nagios/plugins/check_raid --policy WriteBack megacli
 OK: optimal, 1 logical, 2 physical, WriteBack policy
It will complain if for any reason the policy is different than the one given:
/usr/local/lib/nagios/plugins/check_raid --policy WriteThrough
CRITICAL: 1 LD(s) must have write cache policy WriteThrough, currently using: WriteBack
It does not have into account the state of the BBU, size and existence of the cache, etc.
It is enabled on puppet by setting the ::raid class parameter $write_cache_policy, normally set through profile::base::check_raid_policy
== External links ==
* http://erikimh.com/megacli-cheatsheet/
* http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS
== See Also ==
* Setting_up_a_MySQL_replica
Raskin

Raskin is a selenium cluster node for testing OS X browser compatibility.
Server name 	IP 	MAC 	Memory (in GBs)	CPUs	CPU cores	CPU speed	Storage allocated (in GBs)	NICs	OS 	Os version	Hardware type
raskin.usability.wikimedia.org	208.80.152.253	00:24:36:F4:09:9C	12 	1 	4 	2.26 	160 	1 	os x	10.6 	Xserve 3,1
Reading List Service

The Reading List Service is the combination of a few proxy endpoints in RESTBase and a MediaWiki extension. See mw:Reading/Reading_List_Service for the non-Wikimedia-specific bits.
The data lives in the x1 cluster.
The service is maintained by the Wikimedia Reading Infrastructure team, with support from the Services team.
* API sandbox: readinglists, query+readinglists, query+readinglistentries, RESTBase
* dashboard: https://grafana.wikimedia.org/d/000000457/reading-list-service
Category:Services
Reattach a global account

== Background ==
Stewards and all other users with the centralauth-delete permission can delete a global account using the CentralAuth page. That means that the account loses their unified login across all public Wikimedia wikis. With the implementation of the SUL finalisation all accounts are now global by default, and the need to delete global accounts should be rather scarce at this point. Sometimes, however, a global account can be deleted by mistake when stewards intended to globally lock the account instead, and a request to have the account globally reattached could appear.
== Steps to reattach a global account ==
# Access CentralAuth data for the affected account. You'll see that only local accounts exist and that there's no global account information.
# On mwmaint1002:
## Migrate the local accounts to SUL account using migrateAccount.php
## Attach them using attachAccount.php
# Log the reattachment on the Production Server Administration Log.
== See also ==
* 
* — specially this comment from a sysadmin explaining the process.