SLO/MediaWiki Content History

== Service ==
MediaWiki Content History (Dumps 2, Phase II) is an event-based system that makes mediawiki_wikitext_history data available every day instead of just once a month. It fixes data consistency problems by using a reconciliation algorithm that checks data directly against MediaWiki's databases.
The specific components implemented as part of the overall product of MediaWiki Content History are the following.
==== Batch processing ====
Currently (2024-11) the following data pipelines make up the batch component of the system:
* Daily merge of page_content_change events into table
* Daily find missing data and generate reconciliation events
* Merge reconciled event data into table
These pipelines are responsible for the processing of mediawiki.page_content_change events, reconciliation of missing events against MediaWiki MariaDB replicas, and insertion into mediawiki_content_history Iceberg table (AKA “dumps 2 intermediate table”)
==== Stream processing ====
mw-content-history-reconcile-enrich is a streaming application that consumes reconciliation events and enriches them with page content body and redirect info.
=== Teams ===
Data Platform Engineering is responsible for MediaWiki Content History. The sub-team currently in charge of developing and steering the system can be reached at #mediawiki-data-dumps-2.
== Architectural ==
Instructions
=== Environmental dependencies ===
Batch Processing
Data pipelines are implemented as Spark jobs and are executed on the WMF Data Lake (Hadoop), a cluster of bare-metal machines deployed in eqiad. Data pipeline orchestration is managed via Airflow DAGs, deployed on the analytics instance, which runs on a Ganeti VM in the eqiad data center.
Stream Processing
Reconciliation events enrichment is implemented as a PyFlink mediawiki-event-enrichment application, running on DSE Kubernetes in eqiad.
=== Service dependencies ===
For dependencies without an SLO yet, or dependencies that habitually miss their SLO, we assume that they maintain their historical performance, or worsen slightly but not dramatically (as recommended by https://wikitech.wikimedia.org/wiki/SLO/Template_instructions/Service_level_objectives).
==== Hard ====
MediaWiki Content History has hard dependencies on:
* Gobblin and Refine ingestion pipeline (no SLO)
* Airflow 
* YARN and HDFS
Degradation of hard dependencies will impact the system’s ability to produce data dumps on schedule.
==== Soft ====
MediaWiki Content History has soft dependencies on:
* MediaWiki Action APIs, queried for reconciliation event enrichment.
* Kubernetes (dse-k8s) with Flink Kubernetes Operator
* Mariadb analytics replicas (no SLO), queried for reconciliation events emission
Degradation of soft dependencies will result in the inability to reconcile missing events. with impact measurable dy MediaWiki Content History SLIs.
==== Indirect ====
The API boundaries for data pipelines are HDFS and Hive. However, these datasets are downstream of Event Platform services. Therefore, MediaWiki Content History has indirect dependencies on these services:
* EventBus (SLO) produces page_change data.
* mediawiki-page-content-change-enrich (SLO), consumes page_change and page_content_change events.
* Kafka (main, jumbo)
Client-facing
Instructions
Clients
Who are the service's clients?
The service clients are consumers of the mediawiki_content_history Iceberg table. 
Immediate client teams, internal to the Wikimedia Foundation include:
* Research
* Data Products
* Product Analytics
* Machine Learning
* MediaWiki Growth
* MediaWiki Structured Content
* Wikimedia Movement Insights
* WMDE WikiData teams
* MediaWiki Abstract Wikipedia
* Wikimedia Enterprise
* Community Tech
* Search Platform
* …
== Service Level Indicators (SLIs) ==
Instructions
What are the service level indicators? Copy a selection from the instructions, filling in the blanks.
We define system health by ensuring that data landing in the wmf_dumps.mediawiki_content_history Iceberg table meets objectives of:
* correctness: ensures that we are alerted about potential data errors in dumps data 
* freshness: ensures that we are alerted if data is not delivered on time.
We measure these two objectives with the following indicators:
* Latency SLI for batch processing: percentage of the data pipelines dag runs that complete on schedule. This metric informs data freshness (service latency) by measuring the lateness of datadelivery relative to their scheduled times.
* Availability SLI for batch processing: number of missing events over a period of time. This metric informs the correctness of data dumps by tracking events that are present in MariaDB but missing in the data dumps, indicating cases where the system failed to reconcile the data..
* Latency SLI for stream processing: the percentage of time that kafka topic lags above a threshold. This SLI informs about eventual service latency, that would cause mediawiki_content_history_reconcile_enriched events to lag behind mediawiki_content_history_reconcile.
* Availability SLI for stream processing: the percentage of mediawiki_content_history_reconcile events consumed that resulted in an enriched event being produced into enriched events. This is the amount of events that have been successfully enriched.
These SLIs are focused on ensuring operational correctness of the MediaWiki Content History system. Additional guarantees on the wmf_dumps.mediawiki_content_history table correctness are covered by Data Quality Metrics reporting and alerting.
Operational
Instructions
Monitoring
mediawiki_content_history data pipelines are monitored by a combination of Grafana, Logstash and Airflow UI dashboards.
=== Batch processing ===
The status of Spark jobs can be monitored in the analytics instance Airflow UI:
<add links once deployed>.
Airflow will alert on failure by sending an email to <rcpt>.
=== Stream processing ===
The Flink application emits time-series metrics (counters and gauges) for all SLIs, which are available in Grafana. Alerts are triggered and delivered by the Alert Manager.
== Troubleshooting ==
How complex is the service to troubleshoot?
As of 2024-11 no dedicated support SLA is provided. 
==== Batch processing ====
Data pipelines have been implemented according to  https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide guidelines. Generic troubleshooting and operations is described in https://wikitech.wikimedia.org/wiki/Data_Platform_Engineering/Ops_week
Upon failure, data pipelines should be re-run to backfill data.
==== Stream processing ====
mw-content-history-reconcile-enrich depends on Kafka and the Action API. Operational errors are expected to be correlated to the performance of either system. The application is deployed on k8s, integrates with the observability platform and follows deployment pipeline operational conventions.
The application emits errors (exceptions, invalid records, HTTP timeout after retries into a kafka error topic: eqiad.mw_content_history_reconcile_enrich.error. Logs are also forwarded to Logstash in ECS format.
The application is deployed with high-availability enabled within Kubernetes HA services, and checkpoints to ceph. Upon application failure, enrichment will restart from the latest processed kafka offset.
== Deployment ==
=== Batch Processing ===
Data pipelines are stored in the airflow-dags repository (under analytics/dags/dumps) and are manually deployed on the analytics Airflow instance using scap. The deployment steps are documented on Wikitech.
=== Stream Processing ===
The mw-content-history-reconcile-enrich application is deployed on dse-k8s-eqiad following deployment pipeline practices. The enrichment implementation is available in the mediawiki-event-enrichment monorepo and is packaged in the corresponding docker image.
== Service Level Objectives ==
Instructions
We will measure the system’s performance over a period of one quarter, and adjust expected and ideal targets accordingly.
=== Realistic targets ===
Based on our experience running airflow data pipelines and flink applications, the following are realistic targets:
* Latency SLI for batch processing: 80% of the data pipelines successfully complete over a 30 days period.
* Availability SLI for batch processing: no more than 20% of events are missing over a 24-hour period.
* Latency SLI for stream processing:  acceptable kafka consumer latency would be 80% of the time, the max lag is within 60 seconds threshold. 
* Availability SLI for stream processing: 80% of processed messages are enriched over a 24-hour period, with no particular upper bound of latency.
=== Ideal targets ===
* Latency SLI for batch processing: 99% of the data pipelines successfully complete over a 30 days period.
* Availability SLI for batch processing: no more of 0.1% of events are missing over a 24 hours period.
* Latency SLI for stream processing:  excessive kafka topic percentage would be 99% of the time, the max lag is within 60 seconds threshold. 
* Availability SLI for stream processing: 99% of processed messages are enriched over a 24-hour period, with no particular upper bound of latency.
=== Reconciliation ===
MariaDB replica lag, erroneous responses from the Mediawiki Action API, or changes on databases that can't be captured by EventBus hooks, will impact the availability of page_content_change events. 
See also https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment/SLO/Mediawiki_Page_Content_Change_Enrichment
== References ==
* Dumps 2.0 System Overview and Task Breakdown
* DRAFT: [Dumps 2] Reconciliation mechanism to detect and fetch missing/mismatched revisions