nyuuzyou/wikis huggingface dataset:
Event Platform/Beta Deployments

Event Platform services are deployed to k8s via Deployment pipeline. All services are available in staging, and that's where we usually test.
Sometime you might find it useful run integration tests in the Beta cluster . For this use case, services should follow Deployment prep practices.
While MediaWiki code is synced automagically Nova Resource:Deployment-prep/How code is updated#How do I get my code on the beta cluster?,
Event Platform services require a couple of manual steps.
tl;dr: version bump the image to deploy in the Horizon UI (do not submit a CR to the cloud infra git repo), ssh to the cloud vps box, run sudo puppet agent -t to pull changes and eventually redeploy the docker image.
To be able to deploy to Beta you need to:
* Have a valid Cloud vps account with a Generate an SSH Key. Deployment prep hosts run docker containers on cloud infra (*not* k8s), so generate a dedicated key.
* Have an ssh config that with cloudvps bastion hops. See Help:Accessing Cloud VPS instances#Setup
* Make sure you are part of the deployment-prep project in https://horizon.wikimedia.org/project/. 
* Make sure you are a project member of the service you want to deploy. 
** You need this to be able to make config changes.
** If the puppet config tab to horizon does not show an "Edit" button, you are not a project memebr. Ping Event Platform/Maintainers if you think you should.
As an example, let's say you want to test changes to Event Platform/EventStreams. The workflow would look something like this:
# Get patches merged to the service Gerrit repo
# On merge a new docker image will be created and pushed to Docker-registry.
# Go to the Deployment prep instance (Compute -> Instances -> Filter by name = event) page in Horizon
## Right now this would point to deployment-eventstreams-2
# Click on the Puppet Configuration tab
# Click on the "Edit" button and version bump the service docker image .
# Apply changes. Horizon should auto merge to the cloud instances gerrit repo.
# Login to the cloudvps host (assumes you have provided a pub key and configured bastion hops): ssh deployment-eventstreams-2.deployment-prep.eqiad1.wikimedia.cloud Where
## deployment-eventstreams-2: is the clouvps integration test instance
## deployment-prep.eqiad1.wikimedia.cloud: is the clouvps deployment-prep net
# Deploy the changes with puppet: sudo puppet agent -t
# You can edit https://simple.wikipedia.beta.wmflabs.org/ to test changes.
Event Platform/EventGate occasionally fails to ingest specific schemas

This page contains some notes and a post mortem of https://phabricator.wikimedia.org/T326002.
This incident might not technically an outage, but is a good use case for troubleshooting and op practices.
The root cause was an instance of EventGate configured with dynamic schema loading (no bundled schemas). Cache refreshes were requesting a larger config than what the EventStreamConfig was set to return, causing schema loading to fail.
==== Timeline ====
1. Dec 29 2022, 10:43 AM. The phab was open. It reports an issue with a data analytics job that fails to produce canary events.
2. Fri, Sep 15, 9:05 AM. Joseph flags the phab in slack. Joseph correctly identifies the root cause begin the EventGate service and escalates to an EventGate maintainer (Gabriele).
3. Fri, Sep 15, 1:06 PM. Joseph and Gabriele tirage, identify the issue only affecting the eventgate-analytics-external instance. Error rates correlate with eventbus log entries.
4. Fri, Sep 15, 2:50 PM. Sam helps narrow down the issue, and provides a workaround. The workaround patches the EventStreamConfig service (Sam is the maintainer).
5. Mon, Sep 18, 9:17 PM. f/up from Gabriele. OK on the patch. Proposal to improve alert coverage for all eventgate instances.
6. Now - EventStreamConfig patch is scheduled for deployment on the upcoming train. Alertmanager patch is in review.
==== Lessons learnt ====
1. This issue would have been caught before if we had alerting in place.
2. Severity was underestimated. Since the issue seemed to only relate to canary events, it was not prioritized.
==== Impact ====
This incident impacted an instance of eventgate that supports legacy instrumentation systems https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate#eventgate-analytics-external
mobile clients might have been impacted, but it's unclear to what extent.
==== Follow up ====
1. EventGate should be able to fetch configs with result pagination. We should evaluate the need for dynamic configs. We should improve logging reporting.
2. EventGate will be covered by an SLO.
3. EventBus will be covered by an SLO.
4. All eventgate instances should alerts on SLOs.
Event Platform/EventStreams/Powered By

Name/Description 	Author 	Link 	Source 	Screenshot
JavaScript EventStreams client 	ChlodAlejandro 	https://github.com/ChlodAlejandro/wikimedia-streams 	https://github.com/ChlodAlejandro/wikimedia-streams 	
EventStreams UI 	WMF Analytics Team 	https://stream.wikimedia.org/v2/ui/ 	https://gerrit.wikimedia.org/g/mediawiki/services/eventstreams/	
Recent changes statistics 	Krinkle 	https://codepen.io/Krinkle/full/BwEKgW 	https://codepen.io/Krinkle/pen/BwEKgW 	
Live Edit Charts 	Ethan Jewett 	https://esjewett.github.io/wm-eventsource-demo/ 	https://github.com/esjewett/wm-eventsource-demo 	
Change Type Radar Chart 	Calvin Bao 	http://calvinbao.com/wikimedia_event_streams/ 	https://github.com/csbao/wikimedia_event_streams 	
Edit IP Patrol 	Freddy2001 	https://tools.wmflabs.org/freddy2001/ipp/ 	https://github.com/freddy2001/ipp 	
Event Streams 	MusikAnimal 	https://tools.wmflabs.org/event-streams/ 	https://github.com/MusikAnimal/event-streams 	
Data Waltz Exhibit in LA 	Nate Imai, Matthew Conway, Rachel Lee, and Max Wong	https://woodbury.edu/event/data-waltz-wuho/, http://datawaltz.hatnote.com/ 	 	
WikiMedia recent changes DDP (Meteor) API 	Mitar 	https://wikimedia.meteorapp.com/ 	https://github.com/mitar/mediawiki-stream 	
LunchTime (Wikipedia changes) 	ahstro 	http://ahst.ro/lunchtime/ 	https://github.com/ahstro/lunchtime 	
Wikipedia edits from various countries around the world in real time	TIBCO Software Inc 	https://www.tibco.com/products/tibco-spotfire/learn/demos/real-time-wikipedia-edits-demo	 	
EditGroups 	Pintoch 	https://editgroups.toolforge.org/ 	https://github.com/Wikidata/editgroups 	
MajavahBot task 1, patrolling enwiki edit filter false positive page	User:Majavah 	https://majavah-bot.toolforge.org 
:w:en:WP:EFFP 	https://github.com/supertassu/majavah-bot 	
== Unavailable ==
Name/Description	Author 	Link 	Source 	Screenshot
Edits on a globe	Sacha Saint-Leger	https://0xmiel.github.io/wikimedia-challenge/	https://github.com/0xmiel/wikimedia-challenge
Event Platform/Event Utilities

Wikimedia Event Utilities are code libraries for interacting with Wikimedia's Event Platform, including Stream config, schemas and producing events to Kafka, and stream processing abstractions to aide building and deploying stream processing applications.
Event Utilities libraries that are used to produce events must follow Event Platform/Producer Requirements.
== Wikimedia Event Utilities ==
=== eventutilties-core (java) ===
This library has logic and abstractions for working with streams and schemas using configured Stream Config endpoints and Schema repositories / Schema services. 
Main entrypoints:
* EventStreamFactory - Uses Stream Configuration and Schemas to build EventStreams.
* EventStream - Helps for getting information, schemas, config, etc. about an Event Platform stream.
* JsonEventGenerator - Creates Jackson JSON ObjectNodes that are suitable for producing to an Event Platform stream following Event Platform/Producer Requirements.
This library is built and deployed to archiva.wikimedia.org using Jenkins.
=== eventutilities-flink (java) ===
This library has Flink + Event Platform integration abstractions. It contains converters to convert from Event Platform event JSONSchemas to Flink types, and classes that help instantiate Flink DataStreams or Tables representing Event Platform streams. 
The main entrypoints here are 
* EventDataStreamFactory - helps instantiate Flink DataStreams
* EventTableDescriptorBuilder - helps instantiate Flink Tables
* EventStreamCatalog - Flink SQL Catalog + Event Platform stream integration. Event Flink Catalog Wikitech docs.
This library is built and deployed to archiva.wikimedia.org using Jenkins.
=== eventutiltlies-python ===
A python library that aims to wrap the eventutillities Java libraries. As of 2023-05, most of the work here is done to enable and ease building simple PyFlink applications.
This library is built and deployed to its Gitlab package registry using Gitlab CI. To install a release via pip, you must add the Gitlab pip package registry as a pip package index. This is most easily done using the --extra-index-url pip option.
pip install eventutilities-python --extra-index-url https://gitlab.wikimedia.org/api/v4/projects/1014/packages/pypi/simple
Or, in a requirements.txt file, e.g:
--extra-index-url https://gitlab.wikimedia.org/api/v4/projects/1014/packages/pypi/simple
eventutilities-python==0.8.0
Installing eventutililties-python via a pip wheel like this will include some Java dependency jars in its 'lib/' path. When using eventutilties-python flink helpers, these dependencies will automatically be added to the Flink environment.
A Flink distribution must be manually provided. You can do this many ways, but the easiest way is to install pyflink via pip:
pip install apache-flink
Category:Event Platform
Event Platform/Flaws

Like any system, Event Platform has its flaws. This page aims to document them.
It has inherited design decisions that may have been good at the time, but with hindsight are not. 
== meta field ==
The meta field is very confusing. It was originally created as a way of referencing a single subobject field that contained fields EventBus events needed to operate. This allowed for easier copy/pasting the field between different schemas, which we had to do before we had jsonschema-tools and schema $refs and materialization. 
Ideally, the fields in meta would be top level and named appropriately. If we could get rid of meta we would. Doing so would be a lot of work. In 2023-01 the Event Platform team considered doing this work, but decided not to. (See also Event Platform/Decision Log decision #006).
== meta.domain field ==
The description of this field (as of 2024-02) is "Domain the event or entity pertains to".
The semantics of this field were never well defined. It is often used to hold a domain name the event pertains to, but it is also sometimes used as the 'business' domain. It is also used by the WMF canary (heartbeat) events system.
== dt fields ==
Every event needs to have an 'event time' field, specifying the time at which the event happened. Ideally, this would be the only field we'd need to require for all events. We would then use this field for Kafka timestamps and Hive hourly partitioning.
However, we accept events from unauthenticated external clients, so we can't totally trust them. A client might send an event with a timestamp in the distant past or future, which would cause issues for data ingestion. In cases where we can't trust the event time, we fall back to using the server side receive time.
So we need 2 timestamp fields: event time and server receive time. As of 2023-06, the intention is to always use meta.dt for server receive time and dt for event time. These field names are not particularly descriptive, but creating and using new dt fields is a non trivial amount of work.
See also T267648 Adopt conventions for server receive and client/event timestamps in non analytics event schemas.
== Kafka topic data center prefixes ==
Kafka#Data center topic prefixing design flaw
Event Platform/Intake

PLACEHOLDER: document event platform intake requirements
 * - initialization of default values
 * - set meta.dt as the ingestion time (current time is fine)
 * - set meta.stream to the proper value (if not set by caller)
 * - $schema as the schema that can validate the generated event (if not set by caller)
 * - set meta.id to a uuid if not set
 * - verification of schema vs stream (this requires looking up stream config)
 * - validation of the event against its schema
Event Platform/Maintainers

Every Event Platform component should have a steward with a manager, and at least 2 maintainers. If a component has fewer than 2 maintainers, it is in danger of being ill-supported. If maintainers leaves WMF, the steward/manager should find and assign a new maintainers.
Component 	Description 	Steward 	Maintainers
eventgate-wikimedia (4 deployments in prod k8s) 	Service 	Data Engineering	Andrew Otto, Gabriele Modena
EventGate 	NodeJS Library 	Data Engineering	Andrew Otto, Gabriele Modena
eventstreams (2 deployments in k8s) 	Service 	Data Engineering	Andrew Otto, Gabriele Modena
EventStreams 	NodeJS Library 	Data Engineering	Andrew Otto, Gabriele Modena, Thomas Chin
EventBus 	MW Extension 	Data Engineering	Andrew Otto, Gabriele Modena
wikimedia-event-utilities 	Java Library 	Data Engineering	Andrew Otto, David Causse
eventutilities-python 	Python / PyFlink library, built on wikimedia-event-utilities 	Data Engineering	Andrew Otto, Gabriele Modena, Thomas Chin
mediawiki-event-enrichment 	PyFlink based applications for streaming enrichment of MediaWiki events 	Data Engineering	Andrew Otto, Gabriele Modena, Thomas Chin
flink-app helm chart 	Helm chart for deploying Flink applications in Kubernetes 	Data Engineering	Andrew Otto, Gabriele Modena, Thomas Chin, David Cause, Brian King
flink-kubernetes-operator helm chart 	Helm chart for deploying flink-kubernetes-operator, a dependency for the flink-app chart	Data Engineering	Andrew Otto, Brian King
flink and flink-kubernetes-operator production docker images	Base image for Flink applications and flink-kubernetes-operator deployed in Kubernetes 	Data Engineering	Andrew Otto, Brian King
EventStreamConfig 	MW Extension: Lib and (action) API endpoint 	Data Engineering	Andrew Otto, Sam Smith
Schema Repositories (1, 2) and Service 	Repository / Service 	Data Engineering	Andrew Otto, Sam Smith
jsonschema-tools 	NodeJS library 	Data Engineering	Andrew Otto, Thomas Chin
Event Platform/Producer Requirements

In order to ensure that Event Platform based event streams are automatically integratable with all consumers and downstream data systems, producers must ensure that the events satisify specific requirements before producing them. If you are using a supported WMF Event Platform producer service or library, these requirements should be satisified already. However, if you are working in a language or area that does not have a Event Platform client library, the you will be producing events directly to Kafka yourself. 
This page describes the requirements that any Event Platform producer should satisfy.
= Requirements =
==== Events ====
* Events MUST have an event schema in a WMF Schema Repository.
* Events MUST have required event fields set.
* Events MUST have meta.stream set to the name of the stream they are destined to.
* Event streams MUST be declared in WMF Event Stream Config, with schema_title set to the title that matches the event schema's title they will conform to.
==== Producer libraries ====
A producer will need to interact with and lookup Event Stream Configuration and WMF Schema Repositories. The URI locations from which to look up event stream configuration and event schemas should be configurable. E.g. A user should be able to provide your producer library a local schema repository base path for development.
All generic producer clients and libraries:
* MUST look up the event's schema via its $schema URI field in the configured base schema repository URIs.
* MUST look for any event stream configuration defined for the stream the event will be produced to. The stream the event will be produced to should be in the event's meta.stream field.
* MUST ensure the event is allowed in its destination stream, as specified in the meta.stream field. This can be determined by checking that the stream's configured schema_title matches the event's schema's title.
* MUST ensure that the event has a dt field set specifying its ISO-8601 event time.
* SHOULD set the meta.dt field to indicate the event's ISO-8601 'system ingestion time' by the library.
* MUST ensure the event is valid according to the schema it declares in its $schema field.
* SHOULD use the message_key_fields event stream config setting to hoist event field values into a JSON object, and set this as the Kafka message key. (Example EventGate Javascript code that does this).
* MAY automatically add datacenter prefixes to the stream name to make the destination Kafka topic name.
* MAY produce to the datacenter specific Kafka topic. 
= Supported Event Platform producer clients =
* EventGate - An http producer proxy. At WMF, this takes events to produce over HTTP and produces them to Kafka.
* wikimedia-event-utilities Java library - A java library for working with Event Platform events and streams. Has APIs for preparing events for production to Kafka.
* eventutilities-python - A python library mostly (as of 2023-05) for building PyFlink + Event Platform applications
As new clients and libraries are implemented, please link to them here.
Event Platform/SLO/Flink Kubernetes Operator/

=== Service ===
A Kubernetes Operator that allows users to manage Flink applications and their lifecycle through native k8s tooling like kubectl. Wikimedia maintains an operator for the upstream Apache project at https://github.com/wikimedia/operations-deployment-charts/tree/master/charts/flink-kubernetes-operator. The Kubernetes operator enables the use of the FlinkDeployment CRD to deploy native Flink clusters in single application mode. All Wikimedia Flink clusters should be deployed atop this service using the provided flink-app chart template.
=== Teams ===
The Data Platform Engineering and the Search Team share responsibility for this service.
== Architectural ==
Instructions
=== Environmental dependencies ===
The operator runs on k8s main (eqiad, codfw), staging and dse-k8s. Deploying this service required admin privileges.
=== Service dependencies ===
== Client-facing ==
Instructions
=== Clients ===
A Helm chart template to deploy Flink applications using the operator is available at https://github.com/wikimedia/operations-deployment-charts/tree/master/charts/flink-app.
Clients are expected to implement their own deployments via a set of helmfiles.
=== Request Classes ===
What are the request classes, and how is a request classified? If your service has only one request class, delete this section.
== Service Level Indicators (SLIs) ==
Instructions
* Service uptime (availability): measured by the kubernetes POD uptime metric. When the PODs are down, the existing Flink cluster won’t be operational and new clusters can’t be deployed.
* Deployed clusters state: the percentage of Flink clusters in STABLE vs MISSING or ERROR state.
== Operational ==
Instructions
=== Monitoring ===
The Flink Kubernetes operator  emits timeseries metrics (counter, gauges) for all SLIs. They are available in Grafana.
Logs are available in Logstash (ECS index).
=== Troubleshooting ===
TBD. Needs input from SRE / admins.
=== Deployment ===
The service is deployed with deployment-charts. See https://github.com/wikimedia/operations-deployment-charts/tree/master/charts/flink-kubernetes-operator.
Deploying this operator implies a restart of the clusters that run atop it.
== Service Level Objectives ==
Instructions
=== Realistic targets ===
TBD: this is a relaxed target that would not be acceptable by dependend systems (e.g. WDQS has an SLA for serving pages with al lag lag < 10min 95%). Looking at metrics on main, this looks achievable, question is whether responsible teams can commit to it.
A realistic target for availability would be 80% operator pods uptime.
A realistic target for deployed clusters state would be deployments are STABLE 80% of the time.
=== Ideal targets ===
An ideal target for availability would be 99% operator pods uptime.
An ideal target for deployed clusters state would be deployments are STABLE 99% of the time.
=== Reconciliation ===
Reconcile the realistic vs. ideal targets, documenting any decisions made along the way.
Once the SLO is final, consider collapsing the above three sections.
What are the agreed-upon SLOs, for each SLI and each request class?
Event Platform/SPIKE/Should we enable compression on kafka jumbo

This page summarizes the learnings of https://phabricator.wikimedia.org/T345657
== [SPIKE] Should we enable compression on kafka jumbo? ==
* Author: Gabriele Modena <gmodena@wikimedia.org>
* Bug: https://phabricator.wikimedia.org/T345657
tl;dr: we should enable snappy compression at Flink producer level. Snappy compression is the default for events produced by EventGate,
and topics mirrored by MirrorMaker. Flink streaming applications bypass EventGate, but should guarantee configuration settings parity. 
=== Evaluation ===
This spike wanted the inform the following:
===== What are key metrics on topics that store large messages (total size, percentile message size, throughput, retention)? =====
A rich set of topic level metrics are exposed in Grafana. Config settings (retention, partitioning) need to be programmatically 
inspected on a per topic basis and are not available under version control.
Some rough numbers on 7 days of mediawiki.page_content_change.v1. The topic has a size in the neighborhood of 762GiB. Messages have a median size of 380KB and a max of 1MB.
Quartile breakdown:
 stat (bytes)
 mean 4.140364e+05 
 std 1.493413e+05 
 min 1.370000e+01 
 25% 3.171210e+05 
 50% 3.879230e+05 
 75% 4.627110e+05 
 max 1.032399e+06
For comparison, mediawiki.page_change.v1 in the neighborhood of 20-24GiB and significantly smaller records:
 stat (bytes)
 mean 11830.387352 
 std 2519.479806 
 min 5576.000000 
 25% 10159.000000 
 50% 11955.000000 
 75% 13253.000000 
 max 20642.000000
Page changes is produced from eventgate, so I assume the payload is snappy compressed.
===== Should we also enable compression on jumbo? =====
No need to. Topics are assumed to be compressed by producers (EventGate) and MirrorMaker.
===== Should we rather let producers take care of it? =====
This is the recommended path forward for producers that sidestep MirrorMaker.
==== Follow up work ====
The following work has been identified as actionable next steps:
* https://phabricator.wikimedia.org/T345805
* https://phabricator.wikimedia.org/T345806
Event Platform/Stream Configuration/Evaluate ESC and explore an alternative design

This page summarizes the learnings of https://phabricator.wikimedia.org/T365005 . That phab task explored an alternative architecture design to refactor stream configuration services (Stream Registry) to better integrate with a platform wide Config Store and MPIC use cases.
After re-scoping both Config Store and MPIC, we decided to not move forward with refactoring ESC at this stage. A Design Document is available at Stream Registry
= Stream Registry =
2024-05-17
URL:Stream Registry
Author: Gabriele Modena <gmodena@wikimedia.org>
Bug: T365005
This is a living document, and currently (as of 2024-05-23) under review. 
== Overview ==
This document describes the design of a Stream configuration specification and its reference implementation as a web service (Stream Registry), deployed on Kubernetes (multi data center, targeting wikikube). Stream Registry will be initially backed by Git and Cassandra, with the goal of moving to an s3 protocol compatible object store once CEPH is available. Stream Registry will not need to be directly exposed to the public internet, only the internal network or a selected pool of IPs.
Event Platform/Stream Processing/Flink/FailureScenarios

== Known/Observed Flink Failure Scenarios ==
=== Kubernetes Operator ===
The Flink Kubernetes Operator runs as an HA pair. We have observed a scenario in which the active master loses sync with its resources. In this scenario, API calls that involve resource updates (writes, in other words) are either ignored, or hang forever. 
==== Workaround ====
Deleting the Flink Operator's active master container will force a failover, which fixes the issue. To find the master: kubectl -n flink-operator get lease flink-operator-lease  -o yaml