nyuuzyou/wikis huggingface dataset:
Rsyslog

rsyslog is the default Debian logging daemon and what's deployed fleet-wide at Wikimedia Foundation.
== Packaging ==
We currently have a set of different rsyslog versions/packages that we manage for different reasons, all using gbp build flow:
* Rebuilds of the debian buster upstream packages (8.1901.0) including the mmkubernetes plugin our Kubernetes/Logging pipeline is build on
* A backport of rsyslog 8.2008.0 to address issues with the debian upstream version (, ) which is used on centrallog hosts
=== Branches ===
* debian/buster-wikimedia-k8s: Published to component/rsyslog-k8s for buster-wikimedia; Used on Kubernetes nodes running buster.
* debian/bullseye-wikimedia-k8s: Published to component/rsyslog-k8s for bullseye-wikimedia; Used on Kubernetes nodes running bullseye.
* debian/stretch-wikimedia: Published to main for stretch-wikimedia; Used on Kubernetes nodes running stretch.
* UNKNOWN: Published to component/rsyslog for buster-wikimedia; Used on centrallog nodes.
=== Build ===
# Adapt the --branch argument to debian/buster-wikimedia-k8s in case you want to build that
BACKPORTS=yes DIST=stretch gbp buildpackage --git-pbuilder --git-no-pbuilder-autoconf --git-dist=$DIST -sa -uc -us --git-debian --branch=debian/$DIST-wikimedia
== Troubleshooting ==
=== rsyslog "stuck" ===
Servers to look for:
 Puppet: syslog::centralserver
 Currently: centrallog1002.eqiad.wmnet and centrallog2002.codfw.wmnet (Jul 2024)
rsyslog has been observed for getting stuck from time to time (its TLS listener stops responding). In these situations a restart "fixes" the problem, however before doing a restart it is important to capture the daemon's status:
 cd
 timeout 30s strace -f -p $(pidof rsyslogd) -s 65535 -o rsyslog_$(date -Im).strace
 lsof -p $(pidof rsyslogd) > rsyslog_$(date -Im).lsof
 gdb -p $(pidof rsyslogd) --batch -ex gcore
 gdb -p $(pidof rsyslogd) --batch -ex 'thread apply all bt full' > rsyslog_$(date -Im).threaddump
 systemctl restart rsyslog
Runbook

A runbook is a set of instructions for a human what to do. More specifically what to do when a certain monitoring alert triggers.
Prometheus alerting rules link to this page in a service-specific anchor based on the service name itself; from the service entry you can give more context, next actions, and link to other resources such as more runbooks, service dashboards, etc.
Pages that contain runbooks are linked from Icinga checks in puppet using the notes_url parameter, or alertmanager rules with the runbook annotation. 
There is a :Category:Runbooks for pages with runbooks.
Compare to cookbooks which are programs running in spicerack/cumin to do maintenance tasks.
== service-name:port ==
This is the entry linked from generic alerts for service-name:port, for example ProbeDown will link here when network probes for service-name fail. See also .
 
Category:Monitoring
== apt:80 ==
See APT_repository
== grafana:443 ==
See Grafana.wikimedia.org
== graphite:443 ==
See Graphite
== helm-charts:443 ==
This is the service powered by chartmuseum, see ChartMuseum
== jobrunner:443 ==
See Application servers/Runbook#Jobrunners
== librenms:443 ==
See LibreNMS
== netbox:443 ==
See Netbox
== puppetboard:443 ==
See Puppet
== puppetdb-api:8090 ==
See Puppet#Micro_Service
== releases:443 ==
See Releases.wikimedia.org
== thanos-query:443 ==
See Thanos#Alerts
== upload-https:443 ==
Check NEL
== videoscaler:443 ==
See Application servers/Runbook#Jobrunners
== tools-k8s-haproxy-3:30000 ==
See Portal:Toolforge/Admin/Runbooks/k8s-haproxy
== tools-k8s-haproxy-4:30000 ==
See Portal:Toolforge/Admin/Runbooks/k8s-haproxy
== gerrit1003:443 ==
Check Apache Dashboard for worker saturation. If so, run sudo systemctl restart apache2. See also Gerrit/Operations.
SLO/AQS/Geo Analytics

Status: draft
== Organizational ==
=== Service ===
This page provides information on the publicly accessible Geo analytics service, which is hosted at the base path: https://wikimedia.org/api/rest_v1/metrics/editors/by-country
In order to make a service call, it is necessary to append appropriate request parameters to the following URL: https://wikimedia.org/api/rest_v1/metrics/editors/by-country/{project}/{activity-level}/{year}/{month}
For example:
https://wikimedia.org/api/rest_v1/metrics/editors/by-country/en.wikipedia/5..99-edits/2020/12
=== Teams ===
The Data products team is the service owner of the geo analytics service.
== Architectural ==
=== Environmental dependencies ===
The Geo Analytics service runs on kubernetes, proxied through the rest gateway. It is available in both the eqiad and codfw data centers.
=== Service dependencies ===
Beyond the environmental dependencies above, its storage backend is Cassandra, used as a datastore.
== Client-facing ==
=== Clients ===
Community Members: 
Seeing impact in terms of Wikipedia project and the upper bound of their editor counts
Identifying growth opportunities
WMF Staff: 
Movement health and impact metrics
Geographic equity 
Audience reach
Editor acquisition
Essential and Core Metrics 
WMF Engineers and other Technical Contributors :
Provide them with accessible APIs that allow them to reliably accomplish their engineering and analytics tasks that make use of AQS endpoints in a self-serve and efficient way.
=== Request Classes ===
Requests are classified by HTTP method and by the API endpoint:
GET https://wikimedia.org/api/rest_v1/metrics/editors/by-country/en.wikipedia/5..99-edits/2020/12
== Service Level Indicators (SLIs) ==
Uptime (availability) percentage: The percentage of all requests receiving a non-error response, defined as one of: HTTP 200 (success).
Latency: percentage of requests that are completed within a certain deadline,
== Service Level Objectives ==
TBD
SLO/Docker-registry

Status: approved
== Organizational ==
=== Service ===
This page covers the highly available Docker registry hosted at docker-registry.wikimedia.org.
=== Teams ===
The Service Ops SRE team is the service owner of the Docker registry.
== Architectural ==
=== Environmental dependencies ===
The Docker registry runs on Ganeti, active/passive via discovery DNS in eqiad and codfw, with traffic load-balanced via PyBal to two VMs in each data center.
=== Service dependencies ===
Beyond the environmental dependencies above, the Docker registry's only hard dependency is Swift, its storage backend. Redis, used as a blob cache, is a soft dependency: during a Redis outage, pulling and pushing images would be slower but would still complete successfully.
== Client-facing ==
=== Clients ===
The Docker registry is used by Kubernetes (partly via Dragonfly) and by CI.
=== Request Classes ===
Requests are classified by HTTP method and by the API endpoint:
* Manifest reads are HTTP GET or HEAD requests to URL paths of the form /v2/<name>/manifests/<reference>.
* Tag reads are HTTP GET requests to URL paths of the form /v2/<name>/tags/list.
* Blob reads are HTTP GET or HEAD requests to URL paths of the form /v2/<name>/blobs/<digest>.
* Manifest writes are HTTP PUT requests to URL paths of the form /v2/<name>/manifests/<reference>.
All other requests are ineligible for the SLO. Only requests sent to the active data center are eligible for the SLO.
== Service Level Indicators (SLIs) ==
* Latency SLI: The 95th-percentile request latency, as measured at the server side.
* Availability SLI: The percentage of all requests receiving a non-server-error response, defined as HTTP status code less than 500. (200 is the successful response status for reads, 201 or 202 as appropriate for writes. 4xx response codes indicate request errors, like an attempt to fetch a manifest that doesn't exist; for the purpose of calculating availability, these requests are also counted as successful.)
Both SLIs are computed over the Foundation-standard reporting periods: three calendar months, phased one month earlier than the fiscal quarter.
== Service Level Objectives ==
TODO: Grafana links
* The 95th-percentile latency for manifest reads will be less than 2 seconds.
* The 95th-percentile latency for tag reads will be less than 2 seconds.
* The 95th-percentile latency for manifest writes will be less than 3 seconds.
* The availability for manifest, tag, and blob reads, measured together, will be at least 99%.
Note that not all request classes have an objective for each SLI.
Category:Docker
Category:Published SLOs
SLO/MariaDB

Status: draft
== Organizational ==
=== Service ===
MariaDB. We are currently only building SLOs for core databases (sX and xX).
=== Teams ===
Data persistence 
== Architectural ==
=== Environmental dependencies ===
The service sits in 2 datacenters and roughly 250 hosts. More details can be found in MariaDB#Sections and shards.
=== Service dependencies ===
Hard dependencies: server hardware, network, infra-layer software, etc.
Soft dependencies: Most dev team but specially teams in MW engineering group.
== Client-facing ==
=== Clients ===
* MediaWiki for all operations rely on the MariaDB databases.
* Many services rely on databases in misc cluster
**Phabricator
**Mailman
**Znuny
**toolhub
**...
=== Request Classes ===
* Web-based requests (using wikiuser):
** Read operations
** Write operations
* Maintenance script requests (using wikiadmin) aka CLI:
** Read operations
** Write operations
== Service Level Indicators (SLIs) ==
Mostly inspired from Database Reliability Engineering book
* Percentage of queries returned with error (killed due to being slow, syntax error, connection issues, etc.) to all queries returned.
* Average time spent in each MediaWiki request (to index.php) waiting for databases to return query results.
* Read-only time of each section (or: number of write queries affected by read-only time, estimated by measuring the drop in queries)
* Number of queries done divided by CapEx
== Operational ==
=== Monitoring ===
There are all-hands pages for these failure scenarios:
* Any replica or primary that is serving traffic going down.
* Any replica or primary that is serving traffic having replication lag above a certain threshold.
There are IRC alerts/warnings for lower priority issues. Any IRC alert, for a condition that threatens to violate the SLO if uncorrected, also has a paging alert at a higher threshold.
=== Troubleshooting ===
Many failure scenarios are covered under MariaDB/troubleshooting. Common cases cover vast majority of all issues but sometimes some cases require in-depth investigation. Further automation is helping us reduce complexity of troubleshooting in many cases (e.g. by just cloning the host from another replica).
=== Deployment ===
Debian packaging
== Service Level Objectives ==
Instructions
=== Realistic targets ===
What are the realistic targets for each SLI? Why?
=== Ideal targets ===
What are the ideal targets for each SLI? Why?
=== Reconciliation ===
Reconcile the realistic vs. ideal targets, documenting any decisions made along the way.
Once the SLO is final, consider collapsing the above three sections.
What are the agreed-upon SLOs, for each SLI and each request class?
Category:Published SLOs
SLO/Template

To get started, make a copy of this page, named after your service, as a subpage of SLO. (For example, "SLO/Foobaroid".) Also open the instructions, which guide you in more detail through the steps outlined here. Answer the questions in your new copy, removing all the yellow text, in mark tags, as you go. (Save early and often.)
Status: draft (Replace with "approved" when the SLO is complete, agreed on by all responsible teams, and subject to quarterly reporting. This is the final step. You can still update the SLO afterward, with all teams' agreement.)
== Organizational ==
Instructions
=== Service ===
What is the service?
=== Teams ===
Who are the responsible teams?
== Architectural ==
Instructions
=== Environmental dependencies ===
Where does the service run?
=== Service dependencies ===
What are the hard and soft dependencies?
== Client-facing ==
Instructions
=== Clients ===
Who are the service's clients?
=== Request Classes ===
What are the request classes, and how is a request classified? If your service has only one request class, delete this section.
== Service Level Indicators (SLIs) ==
Instructions
What are the service level indicators? Copy a selection from the instructions, filling in the blanks.
== Operational ==
Instructions
=== Monitoring ===
How is the service monitored?
=== Troubleshooting ===
How complex is the service to troubleshoot?
=== Deployment ===
How is the service deployed?
== Service Level Objectives ==
Instructions
=== Realistic targets ===
What are the realistic targets for each SLI? Why?
=== Ideal targets ===
What are the ideal targets for each SLI? Why?
=== Reconciliation ===
Reconcile the realistic vs. ideal targets, documenting any decisions made along the way.
Once the SLO is final, consider collapsing the above three sections.
What are the agreed-upon SLOs, for each SLI and each request class?
SLO/Template instructions

A service level objective (SLO) is an agreement, among everyone who works on a service, about how reliable that service needs to be.
Without such an agreement, different people may have different implicit notions of how much latency is tolerable, how many errors justify getting someone out of bed, or how much instability justifies rolling back a release. By defining a quantitative, measurable set of targets, and then measuring our performance against those targets, we ensure that we're on the same page.
At the same time, it allows teams to reason about the services they depend on. Suppose a frontend promises 100 ms responses at the 95th percentile, but it passes each request to a backend promising 500 ms at the 95th. The SLOs are incompatible -- the frontend can't reasonably promise to respond faster than the backend! But in the absence of explicitly-agreed SLOs, such mismatches in implicit expectations are common.
Thus an SLO also mediates the relationship between services. It's an agreement among the multiple teams invested in a service, but also a promise to the teams that depend on them. That means it's important that each SLO is consistently met. We should only miss an SLO due to truly unexpected circumstances -- and if that happens, we take it seriously and prioritize reliability fixes over other work, so that it doesn't happen again.
An SLO is only valuable if it can be meaningfully relied on; an SLO that's regularly not upheld is worse than none at all. Therefore, in writing your SLO, err on the side of caution. Set unambitious targets that you know you'll be able to meet right away, and plan to tighten them later if you like -- rather than failing to meet your initial goals and loosening them until they're achievable.
== Getting started ==
Make a copy of SLO/Template for your service, and fill it out section by section. This page has details on how to address each of the questions in the template -- each section will take some research, discussion, and sometimes negotiation. 
The next pages in this guide will walk you through addressing each of the questions in the template. Each section will take some research, discussion, and sometimes negotiation. Throughout, when you see the 📝 symbol, that's the prompt to return to your SLO document and write up your findings.
There are seven parts, to complete in order:
# /Organizational/
# /Architectural/
# /Client-facing/
# /Service level indicators/
# /Operational/
# /Service level objectives/
# /Finalizing/
(You might like to skim them all before getting started.)
Now, move on to the first section.
== References ==
* Jones, Wilkes, and Murphy with Smith, "Service Level Objectives" in Site Reliability Engineering, O'Reilly 2016 (free online)
* Alex Hidalgo, Implementing Service Level Objectives, O'Reilly 2020 (WMF Tech copy)
SLO/Template instructions/Client-facing

=== Who are the service's clients? ===
If all the clients are other internal WMF services, they should be identifiable. If some or all clients are external users (human or automated) characterize them in as much detail as possible, for the purpose of assessing their reliability needs.
For complicated services with a variety of distinct use cases, one way to catalogue your clients is to make a list of user journeys -- that is, scenarios like "a user logging into their account" or "a user editing an article" -- that eventually depend on your service's functionality. Then break down that list of user journeys by identifying, in each case, what piece of software directly contacts your service to play its role. ("For logged-in page views, service X calls us to fetch a key, but for edits, service Y fetches the key from us and then service Z writes it back.")
📝 List, or characterize, the service's clients. You don't need to list your clients' clients, unless they also depend on you directly.
=== What are the request classes? ===
Some services receive more than one kind of request, where each is subject to a different SLO. For example, read requests may have better latency guarantees than write requests.
Some classes of requests might be ineligible for any SLO guarantees, such as batch requests over a certain size. (These requests aren't invalid -- it goes without saying that malformed requests will be served errors. SLO-ineligible requests might still be served in a "best effort" fashion -- no guarantees, but in practice they'll often work.)
Ideally, the classes should be constructed such that a request can be classified based only on the request -- not on the response or server state. In other words, a client should be able to classify a request before sending it. In practice, this isn't always possible, and that's okay.
As above, one way to catalogue your request classes is to make a list of user journeys and identify what types of request are made in each case. (In the above example, maybe the fetches from services X and Y are functionally the same, making up one request class, and the write from service Z belongs to another. Or maybe the fetch from service Y requires a freshness guarantee and consequently has a longer latency deadline.)
📝 If your service has only one request class, you can delete this section. Otherwise, list all request classes, and the criteria for determining what class a request belongs to. If any classes are ineligible for the SLO as described above, label them.
Next, move on to writing the service level indicators.
SLO/Template instructions/Finalizing

Read over the whole document one more time. By now, all of the text highlighted in yellow should be gone; if any is left, you can clean it up (after making sure the instructions were followed).
If anything in the document isn't satisfactory to all the responsible teams, now's the time to reach an agreement. It's also strongly recommended to have the draft reviewed by the team managing SLOs at the Wikimedia Foundation; you can contact RLazarus from Service Ops SRE.
When that's all done, follow the Publication instructions in the SLO Runbook.
You're done; congratulations!
SLO/Template instructions/Operational

Every service experiences an outage sometimes, so the SLO should reflect its expected time to recovery. If the expected duration of a single outage exceeds the error budget, then the SLO reduces to "we promise not to make any mistakes." Relying on such an SLO is untenable.
Answer these questions for the service as it is, not as it ought to be, in order to arrive at a realistically supportable SLO. Alternatively, you may be able to make incremental improvements to the service as you progress through the worksheet. Resist the temptation to publish a more ambitious SLO than you can actually support immediately, even if it feels like you should be able to support it.
=== How is the service monitored? ===
Assuming that an outage ends when engineers mitigate or work around the underlying issue, you expect the outage to last at least as long as it takes someone to notice and respond to the outage. If all SLIs are monitored with paging alerts 24x7, this is the expected time between the time when the outage starts and the time when a responding engineer is hands-on-keyboard investigating it. (Remember to include any delay associated with the alert itself, such as a sampling interval or rolling-average window.)
If some SLIs don't have paging alerts, this period is likely much longer. For example, if some element of the service is only monitored during working hours, a breakage on Friday evening might go unnoticed all weekend. A single such outage would cause the service to miss a 99% quarterly uptime SLO.
=== How complex is the service to troubleshoot? ===
After an engineer begins working on the problem, how long will it take to identify the necessary mitigative action? This is the least scientific question on this worksheet; it will likely be informed in part by experience.
Questions to consider: Does the team receiving pages for the service also fully understand its internals, or will they have to escalate to the developers and wait for help? If the engineer responding to the page is relatively inexperienced, can they still find all the information they need -- how to interpret monitoring, diagnose problems, and take mitigative action -- in documentation that‘s complete, up-to-date, and discoverable?
=== How is the service deployed? ===
Production incidents are often resolved by rolling out a code or configuration change, so a slower deployment process means a slower resolution. If the normal rollout is intentionally slowed by canary checks, it's reasonable to assume here that they're skipped for a rollback to a known-safe version, as long as such a process exists.
📝 Answer all the operational questions realistically, explaining how long you expect each phase of an outage response to last in the ideal case, and explaining why you think so. Consider linking to past incident reports to use as a comparison.
Next, move on to writing the service level objectives.
SLO/Template instructions/Organizational

=== What is the service? ===
📝 Name it, and link to appropriate documentation. Define it clearly -- which binary or binaries are covered? If some hosts or clusters are covered but not others, be explicit. (If some requests aren't covered, we'll cover that later.)
=== Who are the responsible teams? ===
An SLO is generally an agreement between two or more teams (though a team might still benefit when solely operating a service for its own use -- and thus acting as both the service provider and the user). In order for the agreement to be useful, representatives from each team should be involved in drafting it, and must be involved in finalizing it.
The teams listed here should include everyone involved in handling or preventing incidents in the service: the people who receive production alerts, the people who write the code, and the people who build and deploy new releases. It's generally not reasonable to expect a team to reprioritize their own work to meet an SLO that was written without their participation.
Representatives from client services may also be consulted, to provide input on their service's needs.
📝 List all the responsible teams. You're probably already talking to all of them -- but if not, start that conversation before continuing to the next section.
Next, move on to ../Architectural/.