nyuuzyou/wikis huggingface dataset:
SRE/Infrastructure Foundations/Responsibilities

The Infrastructure Foundations (I/F) team focuses on building and maintaining our base platform (“metal cloud”) that forms the foundations upon which nearly everything else in our infrastructure builds upon.
= Responsibilities =
For most tools a single line description can be found also in SRE tooling.
 Bare metal Infrastructure 
Service 	Primary Contact 	Secondary Contact	SLOs 	Checklist
Install_server	User:Muehlenhoff	User:Jbond 	Install_server/SLO	Install_server/Checkilist
Ganeti 	User:Muehlenhoff	 	Ganeti/SLO 	Ganeti/Checkilist
 Configuration Management Systems
Service 	Primary Contact	Secondary Contact	SLOs 	Checklist
puppet (infrastructure)	User:Jbond 	 	puppet/SLO	puppet/Checkilist
PCC 	User:Jbond 	 	PCC/SLO 	PCC/Checkilist
puppetboard 	User:Jbond 	User:Volans 	 	
DebMonitor 	User:Volans 	 	 	
homer 	User:Volans 	User:Ayounsi 	homer/SLO 	homer/Checkilist
 Orchestration Tooling
Service 	Primary Contact 	Secondary Contact 	SLOs 	Checklist
cookbooks 	User:Volans 	Just for the repository in general, not each single cookbook	 	
spicerack 	User:Volans 	 	spicerack/SLO	spicerack/Checkilist
wmf-auto-reimage	User:Volans 	 	 	
debdeploy 	User:Muehlenhoff	User:Jbond 	 	
conftool 	User:CDanis 	User:Giuseppe Lavagetto 	 	
dbctl 	User:CDanis 	User:Volans 	 	
cumin 	User:Volans 	 	cumin/SLO 	cumin/Checkilist
wmflib 	User:Volans 	User:Elukey 	 	
 infrastructure security and packaging
Service 	Primary Contact 	Secondary Contact	SLOs 	Checklist
PKI 	User:Jbond 	User:Muehlenhoff 	PKI/SLO 	PKI/Checkilist
CAS-SSO 	User:Jbond 	User:Muehlenhoff 	CAS-SSO/SLO 	CAS-SSO/Checkilist
Reprepro 	User:Muehlenhoff	User:Jbond 	Reprepro/SLO 	Reprepro/Checkilist
Cowbuilder	User:Muehlenhoff	User:Jbond 	Cowbuilder/SLO	Cowbuilder/Checkilist
 infrastructure security
Service 	Primary Contact	Secondary Contact	SLOs 	Checklist
netbox 	User:Ayounsi 	 	netbox/SLO 	netbox/Checkilist
netmon 	User:Ayounsi 	 	netmon/SLO 	netmon/Checkilist
RPKI 	User:Ayounsi 	 	routinator/SLO	routinator/Checkilist
cloudflare	User:Ayounsi 	User:CDanis 	cloudflare/SLO	cloudflare/Checkilist
NEL 	User:CDanis 	 	NEL/SLO 	NEL/Checkilist
Miscellanea
Service	Primary Contact	Secondary Contact	SLOs	Checklist
failoid	User:Volans 	User:Muehlenhoff
SRE/LDAP/Groups/Request access

There are currently two paths to request access to an LDAP group:
* By logging into the Wikimedia Identity Management System at https://idm.wikimedia.org. (See guide below).
* By creating an LDAP-Access-Requests task in Phabricator.
Access requests are gradually being migrated to Wikimedia IDM. Current groups configured to be requested via IDM:
* wmf
All other groups require opening a Phabricator task.
=== Using the Wikimedia Identity Management System ===
* Sign in to https://idm.wikimedia.org.
* In the menu in the top left select Permissions and under Request new permissions select Request this permission next to group you want to request
* Please provide a rationale for this access under Comment. Alternatively if there is an existing Phabricator task about your access, you can also provide a reference under Ticket.
* After finalizing your request with Request access it's being processed. In most cases a request will need approval. The administrators of the permission/group are automatically notified about your request. Until this is acted on, your request will be shown in state Pending. Once processed, it will be shown as Approved or Rejected and you will be added to the underlying LDAP group, if approved.
SRE/LDAP/Renaming users

It is possible, but not trivial, to rename a LDAP user account. The complication occurs mostly because multiple "primary" data stores all reference each other in interdependent ways. 
=== Prerequisites ===
Open 2 tabs and stay logged in on Gerrit and Phabricator. On Phabricator, check the user profile page to make sure that the user to be renamed has enabled both LDAP and MediaWiki auth methods so that they do not lose access to Phabricator entirely during the rename.
=== Rename the user in LDAP ===
# Find the user's current information
#* ldaplist -l passwd <shell-account-name>
# Ensure the new CN doesn't conflict with another user's CN
#* ldapsearch -xLLL -P 3 -E pr=40000/noprompt -b"dc=wikimedia,dc=org" 'cn=<new-cn>'
# Change the user's CN using ldapmodify
#* 
$ ldapmodify -v -H 'ldap://ldap-rw.eqiad.wikimedia.org:389' -D 'uid=novaadmin,ou=people,dc=wikimedia,dc=org' -W -
Enter bind password:
dn: uid=<shell-account-name>,ou=people,dc=wikimedia,dc=org
changetype: modify 
replace: cn 
cn: <new-cn>
- 
replace: sn 
sn: <new-cn>
-
=== Icinga ===
Login to Icinga with your new creds. It should work for lowercase as well.
=== Gerrit ===
As of Gerrit 3.2, the user database is a git repository. See the LDAP/Renaming_users/Gerrit page for instructions.
=== Phabricator ===
* Got to your profile and click on Manage->Edit Settings->External Accounts (direct link)
* Unlink the Wikitech auth and relink it with your new wikitech username. 
*If you want a new phabricator username, you could create a new account (not tested)
=== Wikimedia Cloud Services (Horizon) ===
* You should be able to login using your new username.
* You should have access to all projects you were before.
Hope for the best.
SRE/LDAP/Renaming users/Gerrit

Renaming users in Gerrit involves fiddling in git quite a bit since Gerrit uses git as a database.
Gerrit uses the ldap cn as the user's username and full name. The uid is used for ssh and api access.
$ ssh [gerrit host]
user@gerrit:~ $ sudo su - gerrit2
gerrit2@gerrit:~ $ cd /srv/gerrit/All-Users
gerrit2@gerrit:~ $ git fetch origin refs/meta/external-ids:refs/meta/external-ids
gerrit2@gerrit:~ $ git checkout FETCH_HEAD
[run the rename account script]
gerrit2@gerrit:~ $ git add . && git commit -m 'Modify [user] commonname'
gerrit2@gerrit:~ $ git push origin HEAD:refs/meta/external-ids
gerrit2@gerrit:~ $ git fetch origin refs/users/[user-id]
gerrit2@gerrit:~ $ git checkout FETCH_HEAD
gerrit2@gerrit:~ $ git config -f account.config account.fullName [new username]
gerrit2@gerrit:~ $ git add account.config
gerrit2@gerrit:~ $ git commit -m 'modify user fullname'
gerrit2@gerrit:~ $ git push origin HEAD:refs/users/[user-id]
[back on your laptop reindex accounts]
$ ssh -p 29418 gerrit.wikimedia.org -- gerrit index start accounts --force
=== cn rename script ===
If changing the cn this is the script to use:
#!/bin/bash
set -euo pipefail
OLD_USERNAME="$1"
NEW_USERNAME="$2"
OLD_SHASUM=$(printf "gerrit:%s" "${OLD_USERNAME}" | shasum -a 1)
NEW_SHASUM=$(printf "gerrit:%s" "${NEW_USERNAME}" | shasum -a 1)
OLD_FILE=$(printf '%s/%s\n' "${OLD_SHASUM:0:2}" "${OLD_SHASUM:2:38}")
NEW_FILE=$(printf '%s/%s\n' "${NEW_SHASUM:0:2}" "${NEW_SHASUM:2:38}")
git mv "$OLD_FILE" "$NEW_FILE"
# Change username to lowercase in new file
sed -i "s/gerrit:${OLD_USERNAME}/gerrit:${NEW_USERNAME}/" "$NEW_FILE"
== uid rename account script ==
If changing the `uid` this is the script to use:
#!/bin/bash
set -euo pipefail
OLD_USERNAME="$1"
NEW_USERNAME="$2"
OLD_SHASUM=$(printf "username:%s" "${OLD_USERNAME}" | shasum -a 1)
NEW_SHASUM=$(printf "username:%s" "${NEW_USERNAME}" | shasum -a 1)
OLD_FILE=$(printf '%s/%s\n' "${OLD_SHASUM:0:2}" "${OLD_SHASUM:2:38}")
NEW_FILE=$(printf '%s/%s\n' "${NEW_SHASUM:0:2}" "${NEW_SHASUM:2:38}")
git mv "$OLD_FILE" "$NEW_FILE"
# Change username to lowercase in new file
sed -i "s/username:${OLD_USERNAME}/username:${NEW_USERNAME}/" "$NEW_FILE"
SRE/LDAP/Sysusers

A few special accounts are owned by a given team, but not representing a human user. They are stored under the special LDAP OU ou=sysusers. If you need to create a new account of such a type, please open a Phabricator task and tag it with Infrastructure-Foundations
 Systems users
username 	email address 	Team in charge 	Phabricator reference for original request
jenkinsrelapi	releng+jenkins-rel-api@wikimedia.org	Release Engineering	T338950
SRE/Mailing Lists

Please use sre@wikimedia.org for team-wide communications, plus the subteam ones for subteam-specific communication. Looking at ops-private's archive, it seems like its uses can be replaced by sre@ for the most part, and thus and unless there are objections, we will be filing a task for its closure. Given subscription to these groups is limited by design to the team members, the intention is for these to be used seldomly for async private communications ("agenda for our meeting next week", "OKRs due N/N", "follow up from our meeting the other day", "out sick", etc.) and not for siloing teams into their own spaces. More public mediums like Phabricator are more appropriate for anything that anyone else that could be helpful or interesting to folks across SRE, Tech, the org or the community.
All teams in SRE have a dedicated Google Group, with a working email address. This means the following can be used for emails, calendar invites, Google Docs permissions etc.
==== SRE Organization: ====
* sre@wikimedia.org - all engineers, managers and directors in SRE 
* sre-mgmt@wikimedia.org - managers and directors in SRE.
==== SRE Teams: ====
* sre-foundations@wikimedia.org - Infrastructure Foundations
* sre-observability@wikimedia.org - Observability
* datacenter-ops@wikimedia.org - Datacenter Operations
* sre-service-ops@wikimedia.org - Service Operations
* sre-traffic@wikimedia.org - Traffic
* sre-data-persistence@wikimedia.org - Data Persistence
==== All SREs in Technology ==== 
* sre-at-large@wikimedia.org - everyone in an SRE or SRE-adjacent role across all the teams in Technology. This is a partially hand-managed list, if you are outside of the SRE org and would like to be added please contact sre-mgmt@
==== Problems that we are trying to solve:====
* There is no easy way (without enumerating every team member) for someone in a subteam to address their peers, for a manager to address their subteam, or folks from another team to email or Cc a subteam.
* Similarly, there is no easy way to invite a team into e.g. a recurring calendar invite and make sure that it will be kept consistent as team members come and go. We see that e.g. with the introduction of new team members and recurring team meetings. Sometimes these are even maintained by outside of SRE, e.g. Linh's invites for team meetings with the CTO.
* ops-private@ is a legacy name from years ago when this team was called TechOps. "Operations" is gradually taking on a different meaning across the org: business operations, i.e. the COO's department. ops-private@ is another place to maintain membership, with membership criteria not even being well defined or consistently applied. Another way to phrase this is that we lack a source of truth :)
* ops@ (aka ops-l) suffers from a lot of these problems as well, but with 130+ members it's a beast of its own, so keeping it out of scope for now.
PS: Adapted from an an e-mail by Faidon Liambotis to ops-private and sre on 2020-05-15 Subject: [sre] Google Groups & mailing lists changes for SRE
SRE/Mentorship

Mentorship in the SRE team is a structured, personalized partnership focused on the professional and personal development of the mentee. This partnership is characterized by one-on-one engagement, where knowledge, skills, and experiences are shared within SRE practices to foster technical excellence, reliability principles, and career growth.
Mentorship is not defined by seniority, but rather by a mentee’s interest in learning and a mentor’s interest in teaching. People should seek out mentors, and also opportunities to mentor, regardless of where they are in their career.
== Mentorship core elements ==
* Building relationships between people
* One-on-one engagement
* Recurring meetings to teach and collaborate
* Creating goals for the mentee
* Listening to feedback from the mentee to improve your mentoring
== Methodologies for success ==
* A soft limit of no more than two mentees per mentor, allowing some wiggle room for those with more time to devote.
* Facilitate match-making using the table of available mentors on Officewiki.
* Mentoring relationships last for six months. After that time, the mentor and mentee should end the formal relationship so that they are free to join others, unless there is an extenuating circumstance. This term limit promotes a diversity of interactions on both sides and ensures we eventually clear out any queues of those requesting mentorship.
== Incentivizing mentorship ==
* Time spent in mentorship conversations should be accounted for like all other normal work. Mentorship should not be devalued relative to other priorities or seem like an excess burden. Managers should be sure to allocate time for people who are mentors or mentees and account for those time commitments during quarterly resource planning.
* Mentoring history should be explicitly included as part of the evaluation process for promotions. Mentorship’s inclusion in a promotion evaluation, however, does not denote a requirement. Some people will not have a desire to mentor and that is okay.
* We may be able to occasionally use travel budget, from the pool for conference attendance, to allow mentors and mentees to meet in real life, subject to budgetary constraints and tradeoffs, but this is not expected to be common at this time.
== Resources ==
This is a very lightly-curated initial list of links on the topic of mentoring that seem relevant. This is by no means a specific endorsement of the content of any given article. If you find other compelling resources around the Internet that may be helpful to others, feel free to add them here.
* https://blog.getace.io/all-posts/mentorship-in-tech
* https://blog.getace.io/all-posts/ultimate-guide-on-how-to-start-a-mentorship-program-at-work
* https://hbr.org/2018/07/how-to-mentor-someone-who-doesnt-know-what-their-career-goals-should-be
* https://hbr.org/2019/02/mentoring-someone-with-imposter-syndrome
* https://hbr.org/2019/03/stressed-at-work-mentoring-a-colleague-could-help
* https://hbr.org/2019/10/why-reverse-mentoring-works-and-how-to-do-it-right
* https://hbr.org/2022/03/how-to-mentor-in-a-remote-workplace
SRE/Observability/Documentation

== SRE Observability documentation ==
SRE ObservabilityThe starting point for observability resources at Wikimedia SRE.
===Alerts===
*icinga.w.o/alerts: view of legacy alerting. See also Icinga.
*alerts.w.o: view all active alerts. See also Alertmanager,
*Alerting infrastructure roadmap PDF
===Logs===
*Logstash: central logging platform. See also Logstash.
*Logging infrastructure design document PDF
===Metrics===
*grafana.w.o: central observability platform. See also Grafana.
*Prometheus, recommended and supported metrics toolkit
*Graphite, supported but deprecated time series framework
*Statsd, supported but deprecated metrics aggregation
*Observability/Dashboard_guidelines, ideas towards better dashboards
SRE/Observability/OKR

== FY2024/2025 Q1 Hypothesis ==
If we disable unused Graphite metrics, target migrating metrics using the db-prefixed data factory and increase our outreach efforts to other teams and the community in Q1, then we would be on track to achieve our goal of making Graphite read-only by Q3 FY24/25, by observing an increase of 30% in migration progress.
=== Context ===
We have been using Prometheus in production for several years as it offers several benefits over Graphite. Migrating MW off Graphite ensures we stay ahead with a supported, scalable metrics platform for more effective, multidimensional metrics analysis and storage. Prometheus provides more robust data labeling, storage, and query capabilities. This initiative is fundamental in unifying our metrics, enhancing monitoring, improving MW observability, and reducing tool fragmentation.
Last year, the team set out to test whether a new interface was viable and determined that long-term sustainability required us to migrate MediaWiki metrics to Prometheus, utilizing StatsLib, a new, internally developed, Prometheus-capable metrics interface. By the end of Q2, the team had successfully tested the component in production, and by the end of Q4, it had advanced about 41% along the migration. see Graphite metrics volume migration dashboard
As the WMF improves its culture around MW ecosystem sustainability, we are setting our goals to complete the migration of active, production, and in-use (by dashboards/alerts) metrics to Prometheus to enable read-only mode on the Graphite cluster by the end of Q3 FY 2024/2025.
For this exercise, we define as “in-use” any metric emitted to graphite mapped to a dashboard panel or alert active in Grafana. see Graphite Utilization Dashboard
Community-driven participation is crucial in this effort. This project also facilitates the improvement of our production metrics infrastructure and helps us deprecate legacy systems. 
RFC on Prometheus as a better interface for MW metrics: T249164.
SRE/SRE tooling

List of automation tools and libraries, reporting and authoritative Web UIs used by the SRE team.
=== Tools ===
* Python/Wmflib: a Python package that contains custom modules to interact with the WMF production infrastructure, installable in any host.
*Cumin: parallel remote execution CLI tool and Python library with fine-grained target selection and output deduplication.
* Conftool: high level CLI tool and Python library to manage the live state stored in etcd. Load Balanced Services And Conftool has further information on scripts used to operate on clustered services. 
** Dbctl: conftool extension to specifically manage MediaWiki's database configuration in etcd.
** requestctl: conftool extension to control access and routing of web requests
* Spicerack: centralized Python library to automate task, has modules to interact with most components of the infrastructure.
* Cookbooks: Python scripts that adhere to Spicerack's API and allow to automate tasks using Spicerack functionalities.
**Server Lifecycle/Reimage: fully automated OS (re)installation for physical hosts.
* Debdeploy: CLI tool to check and upgrade Debian packages.
* Homer: network devices configuration management CLI tool.
* WMFMariadbpy, work in progress library to automate administration of Wikimedia MySQL/MariaDB instances
* Country latency measurement: tool to measure per-country latency via RIPE Atlas.
=== Reporting Web UI ===
* icinga.w.o: central monitoring and alerting platform. See also Icinga.
* alerts.w.o real-time Alertmanager frontend. See also Prometheus.
* grafana.w.o: central observability platform. See also Grafana.
* OpenSearch Dashboards (a.k.a. logstash): central logging platform. See also Logstash.
* Server Admin Log (a.k.a. SAL): public log of all !log actions on the #wikimedia-operations IRC channel. Also available at toolforge:sal/production.
* noc.w.o: publicly exposed configuration files and live state related to MediaWiki.
* puppetboard.w.o: PuppetDB API Web UI to inspect all the data stored in PuppetDB (hosts, facts, catalogs, reports of last runs).
* debmonitor.w.o: Debian package tracker website and CLI tool to track installed and upgradable packages. See also DebMonitor.
* orchestrator.w.o: database cluster analytics and performance Web UI. See also Orchestrator.
*librenms.w.o: Network-specific monitoring platform. See also LibreNMS.
*turnilo.w.o: Managed by Analytics but also used by SRE, especially the wmf_netflow and webrequest_sampled_live datasets.
*Superset dashboards for webrequest sampled.
=== Authoritative Web UI ===
* netbox.w.o: data center infrastructure management (DCIM) and IP address management (IPAM) Web UI, a.k.a. "the source of truth". See also Netbox.
SRE/Service Operations

SRE Service Operations▼ 
SRE Service Operations - MediaWiki operations and supporting services (Kubernetes, memcached, redis, infrastructure for: Gitlab, VRTS, Phabricator).
The Service Operations team takes care of public and “user-visible” services alongside Technology and Product teams. This means, for example, our MediaWiki platform, but also the newer (micro)services that comprise our stack. It also includes miscellaneous services and components that we rely upon (think Phabricator, mail systems, VRTS, etc…). The team is also building our new service oriented architecture based on Kubernetes.
Our Workboard containing the work in progress and queued tasks can be found phabricator.wikimedia.org
SRE/Service Operations/About

SRE Service Operations▼ 
{|class="sortable" 
!
Service Operations (aka "serviceops") is a subteam of the Wikimedia Foundations's SRE team. The Service Operations team takes care of public and “user-visible” services alongside Technology and Product teams. This means, for example, our MediaWiki platform, but also the newer (micro)services that comprise our stack. The team is responsible for our new service infrastructure in production based on Kubernetes, plus a subset of the caching and storage service - memcached and redis. It also attends to the infrastructure of the supporting services and its components that we rely upon (think GitLab, Phabricator, mailing list systems, OTRS, etc…). Further we coordinate the efforts around the implementation of Service Level Objectives within the foundation.
SRE/Service Operations/Documentation/Reboots

There are times when we need to reboot our fleet. Here are some notes to help us go through it like
== Mediawiki Bare Metal ==
=== appservers ===
For each $cluster of parsoid, api_appserver, appserver, and jobrunner; for each $dc of eqiad and codfw:
sudo cookbook sre.hosts.reboot-cluster -D $dc -c $cluster -p 5 -s 45
The cookbook will depool and repool the hosts. Beware that it may cause issues with deployments as hosts are still scap targets or proxies while rebooting.
== Datastores ==
=== Memcache cluster ===
Servers of this cluster need no depooling of any sort, and we have a pool of servers to pick up the traffic of any unavailable server. The cached data in the memcache cluster are crucial for our latency, so it is highly recommended to reboot 1 server at a time per DC, with a sleep time of 15'-20' minimum between reboots. 
* https://grafana-rw.wikimedia.org/d/000000316/memcache?orgId=1
=== Redis misc ===
Redis hosts can be rebooted one after the other, taking care of waiting for replication to be back up after the reboot.
For each instance in /etc/redis:
sudo -i
redis-cli -p $instance_port
AUTH <password>
INFO Replication
docker-registry doesn't use HA for redis, so the reboot of its redis nodes will cause a short unavailability.
=== Etcd ===
== Kubernetes ==
=== Production ===
==== Workers ====
sudo cookbook -d sre.k8s.reboot-nodes --batchsize 3 -g main --reason "Reason" --alias wikikube-worker-codfw
sudo cookbook -d sre.k8s.reboot-nodes --batchsize 3 -g main --reason "Reason" --alias wikikube-worker-eqiad
==== Control plane ====
For each host in the control plane (kubemaster, kubetcd)
sudo cookbook sre.hosts.reboot-single -r "Reason" $host
=== Staging ===
Category:SRE Service Operations
SRE/Service Operations/Intake Standards

Architecture: We are happy to talk to you about the operational architecture of your system, and discuss implementation options such as database, caching, shelling out or the operational aspects of running on kubernetes. The best way to start that conversation is to open a Phabricator task. You can use this template, New Task Example, but set the title to: Architecture Conversation Project ABC, if your project is named ABC. In addition tag the team "serviceops". See for a past tasks for the maps 2.0 project.
Implementation: Once you are ready to implement your system, the best way to restart the conversation about your project named "ABC" is to open a "New Service Request ABC" Phabricator task and in addition tag the team "serviceops". New Task Example
Contact:. Best via Phabricator - tell us what you need and how urgent it is. For quick questions we can be reached on .
See mw:Wikimedia Site Reliability Engineering#Service Operations for more.