nyuuzyou/wikis huggingface dataset:
SLO/Wiki Replicas

Status: draft
== Organizational ==
=== Service ===
https://wikitech.wikimedia.org/wiki/Wiki_Replicas
=== Teams ===
* Wikimedia Cloud Services
* Data Persistence
* Data Engineering SRE
== Architectural ==
Instructions
=== Environmental dependencies ===
Where does the service run?
=== Service dependencies ===
What are the hard and soft dependencies?
== Client-facing ==
Instructions
=== Clients ===
Who are the service's clients?
=== Request Classes ===
What are the request classes, and how is a request classified? If your service has only one request class, delete this section.
== Service Level Indicators (SLIs) ==
Instructions
What are the service level indicators? Copy a selection from the instructions, filling in the blanks.
== Operational ==
Instructions
=== Monitoring ===
How is the service monitored?
=== Troubleshooting ===
How complex is the service to troubleshoot?
=== Deployment ===
How is the service deployed?
== Service Level Objectives ==
Instructions
=== Realistic targets ===
What are the realistic targets for each SLI? Why?
=== Ideal targets ===
What are the ideal targets for each SLI? Why?
=== Reconciliation ===
Reconcile the realistic vs. ideal targets, documenting any decisions made along the way.
Once the SLO is final, consider collapsing the above three sections.
What are the agreed-upon SLOs, for each SLI and each request class?
SLO/ipoid

Status: draft
== Organizational ==
=== Service ===
IPoid provides an API for retrieving reputation data for individual IP addresses, as well as retrieving lists of IP addresses matching a particular set of labels (e.g. list of IP addresses that are associated with VPNs).
The service has two parts:
* A web application for serving requests to access the data
* A data pipeline for downloading data from an external data source (spur.us), and loading it into a MariaDB database
=== Teams ===
* Trust and Safety Product
* DBA
* Service ops
== Architectural ==
=== Environmental dependencies ===
Wikikube cluster
=== Service dependencies ===
==== Hard dependencies ====
* Production MariaDB on the misc cluster MariaDB/misc
==== Soft dependencies ====
* Spur.us API availability. If Spur.us is offline or we are unable to access, it means we don't update the data in the application's database. If we are unable to access the data source at Spur.us for more than N days, then the service is less useful, as the data is out of date.
== Client-facing ==
=== Clients ===
* MediaWiki
** via the IPInfo extension
** via CentralAuth PreSecondaryAuthenticationProvider (to be moved to Extension:IPReputation)
* ...
=== Request Classes ===
* Primary use case is look up for a single IP address
* Endpoints exist for looking up all IPs for a particular risk type (e.g. all VPNs)
There are no write requests.
== Service Level Indicators (SLIs) ==
The percentage of all requests that complete within [fill in] milliseconds and receive a non-error response, defined as percentage of requests that receive a non-5xx response.
== Operational ==
Instructions
=== Monitoring ===
How is the service monitored?
=== Troubleshooting ===
How complex is the service to troubleshoot?
=== Deployment ===
How is the service deployed?
== Service Level Objectives ==
Instructions
=== Realistic targets ===
What are the realistic targets for each SLI? Why?
=== Ideal targets ===
What are the ideal targets for each SLI? Why?
=== Reconciliation ===
Reconcile the realistic vs. ideal targets, documenting any decisions made along the way.
Once the SLO is final, consider collapsing the above three sections.
What are the agreed-upon SLOs, for each SLI and each request class?
SLO/logstash

== SLO Worksheet - Logstash ==
=== Service ===
Logstash is a free and open server-side data processing pipeline that ingests data from multiple sources, transforms it, and then outputs it for search. In our infrastructure Logstash is a component of the logging pipeline, which consists of Kafka -> Logstash -> OpenSearch <- OpenSearch Dashboards.
=== Teams ===
Logstash is owned by the SRE Observability team, which is responsible for operation, scalability, and software updates. Contact: sre-observability@wikimedia.org and https://office.wikimedia.org/wiki/Contact_list#Observability
=== Architectural ===
Logstash consists of two clusters per-site.
* A production cluster which consumes logs from Kafka, transforms them, and outputs to OpenSearch.
* A barebones legacy cluster which ingests logs directly via TCP/UDP and outputs them to Kafka for consumption by the production cluster.
==== Hard Dependencies ====
* OpenSearch - This is where log data is stored, logstash will block if OpenSearch becomes unavailable.
* Kafka - Logstash ingests log message from the kafka-logging cluster.
* Hardware - Both dedicated servers, Ganeti instances, and networking.
==== Soft Dependencies ====
none
=== Client-facing ===
==== Clients ====
software 	use 	connection 	interval 	failure mode(Logstash down)
Kafka 	Aggregates and queues log messages for consumption by logstash	Pull via TCP 	Continuous	Kafka consumer lag will spike and alarm
OpenSearch	Storage/archival of log data for search 	Push via TCP 	Continuous	Logstash will block and stop consuming log events, Kafka consumer lag will spike and alarm.
SCAP 	pre-flight error checks to support deployments 	Pull via TCP using logstash_checker.py in puppet	Manual 	False negative/positive result during deploy pre-flight deploy check
== Service Level Indicators (SLIs) ==
Errors - Percentage of logs which fail to be indexed by OpenSearch
Availability - Percentage of time Logstash is handling logs minute-to-minute
=== Monitoring ===
Logstash is monitored via a suite of health checks and metrics, including:
* Icinga checks - Host based service up/down checks
* Kafka consumer lag - Is Logstash able to consume logs from the Kafka queue faster (or as fast as) they appear, or is the Kafka queue growing faster than Logstash (and OpenSearch) can process?
* OpenSearch indexing failures - Is Logstash able to output events to OpenSearch, or do a significant number of log messages fail to be stored in OpenSearch
* Logstash event rate today vs. yesterday - Is the overall log volume significantly higher or lower than 24h ago?
=== Deployment ===
Logstash is installed via Debian package and its configuration is deployed via puppet.
== Service Level Objectives ==
* Errors - 99.5% of events are indexed successfully, per datacenter. Log producers may emit invalid log messages which cannot be parsed and are dropped, producers may exceed rate limits, or output excessive amounts of logs that cannot be reasonably ingested
* Availability - 99.95% of the time, per datacenter, Logstash is operational and actively processing logs
Category:Published SLOs
SMS/USSD

== SMS and USSD support for Wikipedia ==
The simplest Vumi system consists of a transport worker and an application worker.
A transport worker is responsible for sending messages to and receiving messages from users in the big wide world. 
Transport mechanisms Vumi supports include SMPP, XMPP, Twitter, IRC, HTTP and a variety of mobile network aggregator specific messaging protocols. The application worker processes messages from a transport and sends replies – it holds the application logic. Vumi workers communicate over RabbitMQ.
A XMPP based demo is available with the following jabber ids:
wikipediavumi@gmail.com and wikipediavumitest@gmail.com
You can see the code at: https://github.com/praekelt/vumi-wikipedia/
== Background ==
* History of the setup is captured in https://rt.wikimedia.org/Ticket/Display.html?id=2826
* Git Code Documentation - 
== Monitoring ==
Two servers (one in each data center) are deployed, namely zhen and silver.
* Application key monitoring indicators :
* Zhen - http://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Miscellaneous+pmtpa&h=zhen.wikimedia.org&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS
* silver - http://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&c=Miscellaneous+eqiad&h=silver.wikimedia.org&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS
== Troubleshooting ==
=== Possible problems (or known issues) are ===
==== Problem connecting to Redis ====
There will be log entries like the following:
:2013-01-08 18:23:42+0000 [-] Starting factory <vumi.persist.txredis_manager.VumiRedisClientFactory instance at 0x3784638>
:2013-01-08 18:25:43+0000 [VumiRedis,client] <twisted.internet.tcp.Connector instance at 0x3784758> will retry in 2 seconds
:2013-01-08 18:25:43+0000 [VumiRedis,client] Stopping factory <vumi.persist.txredis_manager.VumiRedisClientFactory instance at 0x3784638>
Recovery: Restart Redis, restart Vumi.
==== Bad message crashes transport: ====
There will be a stack trace in one or more of the transport logs.
Recovery: Restart Vumi, send logs to Vumi team for debugging.
==== Problem connecting to SMPP ====
There will be log entries like the Redis ones, but referencing the SMPP client.
Recovery: Check network connectivity, escalate to operator.
=== Vumi recovery process management: ===
stop application - "sudo supervisorctl stop all"
start application - "sudo supervisorctl start all"
restart application - "sudo supervisorctl restart all"
restart with updated supervisord config - "sudo supervisorctl reload"
==== Escalation (Application) Contact: ====
* Jeremy Thurgood / jeremy@praekelt.com
== Future Plans ==
SRE/Data Persistence/Currently working on

Tickets:
* phab:tag/data-persistence/
** phab:tag/dba/
*** phab:tag/blocked-on-schema-change/
*** See also: Map of database maintenance
** phab:tag/data-persistence-backup/
*** phab:tag/bacula/
*** phab:tag/database-backups/
*** phab:tag/media-backups/
** phab:tag/sre-swift-storage/
** phab:tag/cassandra/
SRE/Data Persistence/Documentation guidelines

== General guidelines ==
* If you notice a mistake / gap requiring a simple update, JUST DO IT™.
* If you notice a mistake / gap requiring a non-trivial update, add the documentation review include.
* Keep the documentation pages as short as possible. Any self-contained section should be its own page.
* If you realize we need to document something but don't have the time to do it, create a stub page with a short description and the documentation review and footer includes.
== Includes ==
==== Documentation review (pages requiring comprehensive review) ====
Add the following at the top of the page: {{SRE/Data Persistence/Documentation_review}}
==== Footer (all pages) ====
Add the following at the bottom of the page: {{SRE/Data Persistence/Footer}}
== Special pages ==
* Navigation
* All Data Persistence pages
* Pages requiring review / update
Category:Data Persistence
SRE/Dc-operations

This landing page will serve as an index for DC Ops team specific Wikitech pages. In addition to this landing page, Data Center Operations maintains a landing page in Phabricator.
== SLAs ==
We make every attempt to resolve all tasks and requests in a timely manner. We've implemented the following SLA targets.
Please note none of these start until both the clarified start time and with proper project tags. See details for each type of task request in their section below. Please use templates listed below.
Project 	Business Days to Resolve	SLA start
Procurement 	90 	Date of Task filing
Racking/Installation 	30 	Arrival of Hardware to DC site
Hardware Failure / Repair	10 	Date of Task filing
Decommission 	45 	When all sub-team steps are complete and task is assigned to on-site
== Other Info ==
Common Data center Specifications
SRE/Infrastructure Naming Conventions - shared SRE Department document of host-name standards.
Insetup roles per team can be browsed from the repo directy via: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/role/manifests/insetup/
Examples (not a complete list, included for formatting example)
* Data Engineering: role::insetup::data_engineering
* Machine Learning: role::insetup::machine_learning
== Runbooks ==
* Hardware Troubleshooting Runbook
* Provision cookbook troubleshooting
* Re-image cookbook troubleshooting
* Securely Erasing Media
* Rebuilding SW Raid with SGDISK & MDADM
* Platform Specific Documentation - vendor specific hardware details (Dell, HP, OpenGear, ServerTech)
Category:SRE Data Center Operations
SRE/Dc-operations/Platform-specific documentation

This page contains links to documentation about specific hardware platforms that are currently in use within the Wikimedia datacenter space. This page and sub-pages are currently maintained by the DC Operations team.
* /Dell Documentation - Dell iDrac[67] - PowerEdge R[4567][234]0.
/Supermicro Documentation 
* /Supermicro Documentation - Supermicro server documentation
* /HP Documentation - HP iLo4 - ProLiant DL360 & DL380 generations 8, 9, & 10.
* /ServerTech - PDUs
* /Opengear Serial Consoles : SCS devices.
* /Atlas Anchor - Atlas Anchor devices, basic flashing directions.
See also the generic Management Interfaces page for how to troubleshooting remote IPMI and management interfaces issues. Please note Management Interfaces is not managed/updated by the DC Ops team.
Category:Hardware
Category:Platform-specific documentation
Category:Management interfaces
SRE/Dc-operations/Platform-specific documentation/Supermicro Documentation

* power cycle: reset /system1/pwrmgtsvc1
* power off: stop /system1/pwrmgtsvc1
* power on: start /system1/pwrmgtsvc1
* serial console: start /system1/sol1    
To quit the serial console (NB it's capital T - i.e. shift-t):
 press <Enter>, <Esc>, and then <T> to terminate session
 (press the keys in sequence, one after the other)
TODO: How to get the equivalent of "racadm getsel" on the CLI
SRE/Dc-operations/Securely Erasing Media

Please note that the default Debian Live Image does not include hdparm.
Rob has used a small liveimage of [finnix] which includes hdparm and wipe by default.
In an ideal world, we would have a debian image available via network/PXE boot that loads a command line interface including all raid support (so we can break down raid arrays via the command line), wipe, and hdparm. This would allow 100% remote execution of secure erasure of disks.
== Securely Erasing Hard Disk Drives ==
=== wipe ===
One method of securely erasing HDDs requires use of the wipe utility. This is best done via multiple consoles via crash cart, as it allows each session to run the wipe on an individual disks concurrently rather than sequentially.
Please note that raids must be broken and disks presented individually to ensure proper wipe of all data.
Ensure you list off all the disks in a system:
 lshw -class disk
Run the wipe command
 wipe /dev/sda
You can then swap to alternate sessions to run the wipe command for each hard disk drive in the system.
== Securely Erasing Solid State Disks ==
=== gparted ===
GParted is a GUI interface that also supports a securely erase SSD function. Do NOT confuse the secure erase with simply writing zeros or re-partitioning.
The GParted image can be downloaded here: https://gparted.org
=== hdparm ===
One method of securely erasing SSDs requires use of the hdparm utility
List off the details for a given SSD: 
 hdparm -I /dev/sda 
Set the security password, which is required to securely erase an SSD: 
 hdparm --user-master u --security-set-pass pw /dev/sda 
Send the securely erase command to the SSD (this takes about 1-3 minutes per SSD tops, this command also removes the security set password as part of the secure erase): 
 time hdparm --user-master u --security-erase pw /dev/sda 
Display the info to show SSD back to normal state: 
 hdparm -I /dev/sda
SRE/Dc-operations/Sw raid rebuild directions

When a defective disk is swapped out on a sw raid, it is not automatically rebuilt. Rebuilding requires adding the new disk in with the following procedure (uses wdqs2007 disk replacement from this task as an example):
check to see if the new disk is detected along with the existing disks:
sudo lshw -class disk
copy the parition table of sda to sdh (sdh was replaced)
sudo sgdisk -R /dev/sdh /dev/sda
create a random guid for sdh
sudo sgdisk -G /dev/sdh
audit output of both disks to ensure they now match
sudo sgdisk -p /dev/sda
sudo sgdisk -p /dev/sdh
add the new SSD back into the array
sudo mdadm --manage /dev/md0 --add /dev/sdh2
In some cases, mdraid may automatically create and add the new disk to a separate sw raid. This will manifest with the above command returning
mdadm: Cannot open /dev/sdh2: Device or resource busy
as well as /proc/mdstat showing an inactive raid that isn't md0, for instance
 cat /proc/mdstat 
 Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
 md127 : inactive sda2[7](S)
 937267200 blocks super 1.2
 
 md0 : active raid1 sdb2[1]
 937267200 blocks super 1.2 [2/1] [_U]
 bitmap: 6/7 pages [24KB], 65536KB chunk
In that case, copy the inactive raid id and stop it
sudo mdadm --manage /dev/md127 --stop then retry adding the device to /dev/md0
check the status
 robh@wdqs2007:~$ cat /proc/mdstat
 Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
 md0 : active raid10 sdh2[8] sda2[0] sde2[4] sdf2[5] sdg2[6] sdc2[2] sdd2[3] sdb2[1]
 3749068800 blocks super 1.2 512K chunks 2 near-copies [8/7] [UUUUUUU_]
 [>....................]  recovery =  0.0% (132736/937267200) finish=470.6min speed=33184K/sec
 bitmap: 28/28 pages [112KB], 65536KB chunk
 unused devices: <none>
SRE/Infrastructure Foundations/About

Empowering Reliable Infrastructure and Streamlined Operations
The SRE Infrastructure Foundations team (SRE I/F) builds and maintains the foundational components of our bare-metal infrastructure and global network, ensuring the reliability and efficiency of our systems.
With a dedicated focus on Configuration Management, Infrastructure Security, Orchestration, Automation, and Networking, the team stands at the forefront of creating a reliable, scalable, and secure environment that powers our operations.
Configuration Management: We are implementing robust configuration management practices, leveraging state-of-the-art tools and methodologies. We carefully design and maintain configuration templates, manage version control, and ensure consistent and scalable deployments across our infrastructure. By optimizing configuration processes, we minimize downtime, enhance system performance, and streamline operations.
Infrastructure Security: Security is paramount in today's digital landscape, and as an SRE I/F team, we remain steadfast in protecting our infrastructure. We meticulously assess and fortify our systems against potential vulnerabilities, implementing rigorous security measures at every layer. From conducting regular audits to implementing detection systems and access controls, we continuously monitor, evaluate, and enhance our infrastructure's security posture.
Orchestration and Automation: To facilitate seamless operations, we leverage advanced orchestration and automation tools. We design and maintain orchestration frameworks, enabling efficient resource allocation, workload balancing, and automated scaling. By automating routine tasks, we reduce human error, increase operational efficiency, and free up valuable resources for strategic initiatives.
Networking: We are responsible for designing and maintaining our global network architecture. We optimize network connectivity, ensuring low latency, high availability, and fault tolerance. We implement advanced networking protocols and technologies to enhance the reliability and performance of our systems. Our expertise lies in creating scalable and secure network configurations that enable seamless communication and data transfer across our infrastructure.
Our collective technical excellence and collaborative approach serve as the bedrock of Wikimedia's infrastructure. By continuously refining and enhancing our systems, we empower our other SRE and development teams to deliver exceptional products and services to our users.
Category:SRE Infrastructure Foundations
SRE/Infrastructure Foundations/Contact

=Communication channels=
=== IRC ===
We primarily use IRC as our communication platform, and our public IRC channel can be accessed at 
. You can reach out to us through this channel for the following:
# Asking for help: If you need assistance with any technical or operational matters, feel free to ask for help on the channel.
# General questions: If you have general inquiries or seek information about our team or processes, you can ask your questions on the channel.
# Seeking advice: If you require guidance or advice related to things under our ownership or any relevant topics, feel free to seek advice from our team members on the channel.
SRE/Infrastructure Foundations/Currently working on

==Welcome to our Progress Tracker!==
Here, you can explore the list of tasks currently in progress by our dedicated team. 
We utilize a comprehensive Dashboard that provides separate panels for each area of ownership. Within these panels, you'll find tabs representing different components, each corresponding to a specific task or project. 
This organization allows us to efficiently manage and prioritize our workload.
In our streamlined workflow, most of our tickets progress swiftly from the open to the close status. However, we acknowledge that we currently lack a suitable representation for other task statuses. 
In Phabricator, when story points are used in tasks they represent a rough estimate of the effort as follows: 
Story points	Estimated effort
0.5 	1 hour
1 	1 day
2 	1 week
3 	1 month
4 	1 quarter
5 	multi quarter
If you're curious about our current focus or have any specific inquiries, we encourage you to join us on our IRC channel, #wikimedia-sre-foundations. 
Our team is always available to address your questions and provide updates on our ongoing initiatives. 
Together, we strive to ensure the seamless functioning and reliability of Wikimedia's infrastructure.
Category:SRE Infrastructure Foundations
SRE/Infrastructure Foundations/Debian-installer

This page collects various relevant information around our use of the Debian installer.
= Updating the netboot image =
The netboot image released by Debian does not contain non-free firmware (unless the regular installation media, this isn't for license reasons, but due to the complex process of generating the netboot images). After the release of a Debian point update we typically rebuild the installer image to include the necessary non-free firmware (which is needed to PXE boot). 
The images are stored in the volatile repository in the tftpboot directory. The only users of this directory are the install and apt servers and these are on Puppet 7, as such we only need to update the volatile directory on puppetserver1001.
Simply run it with the distro which has seen a point update as a argument, e.g.
 sudo update-netboot-image bookworm
The script will download the latest bookworm netinst image, append the non-free firmware and then print instructions where to copy the result to (/srv/puppet_fileserver/volatile/tftpboot). You can run the script many times, the only permanent step (moving the result to the target directory) is a manual step (which has instructions printed).
d-i will always pick the DISTRO-installer name, so e.g. bookworm-installer. Before moving the result to volatile you can e.g. move the old one to bookworm-installer-x.y or so that you can revert to a working installer if the new Debian installer has bugs (which didn't happen so far).