Event Platform/Decision Log

ID 	Category 	Decision Description 	Alternatives Considered 	Responsible Parties 	Decision Date	Comments
#000	Mediawiki Event Carried State Transfer 	How to store MediaWiki state, including revisions content body, in events. 	 	Andrew Otto, Giuseppe Lavagetto 	2022-03 	https://www.mediawiki.org/wiki/Technical_decision_making/Decision_records/T291120#2022-03-01:_Otto_and_Giuseppe_discussion
#001	Dumps 	Flink is not ideal when writing to Iceberg in our environment 	Spark [Streaming] 	Dan Andreescu 	2022-11-21 	In trying to write a proof of concept job, we met with too many Java dependency problems. Building a maven pom with Flink, Iceberg, Kafka, and jars we need for our Hadoop environment runs into duplications and version mismatches. It's basically jar hell that may not be worth untangling. It's possible that in the near future the Iceberg team will solve this problem for us. It's also the case that Spark (Streaming or plain) seems to be working quite well for this use case.
#002	Event Streams: PyFlink 	Event Platform will support the API abstractions. Pyflink will be the preferred method to guide people to. Still allow people to use Java (if experienced)
Focus on abstractions of Python so you don't need to know Flink (which you would do for Java/Scala) 	 	Andrew Otto, Gabriele Modena, Luke Bowmaker, Thomas Chin 	2022-12-7 	
#003	Event streams: FlinkSQL 	We don't support it directly (not in .sql file) but sql can be embedded in Python code 	 	Andrew Otto, Gabriele Modena, Luke Bowmaker, Thomas Chin 	2022-12-7 	
#004	Event Streams: Java/Scala 	We support it for experienced Flink engineers for more complex jobs (windowing, state, etc.) 	 	Andrew Otto, Gabriele Modena, Luke Bowmaker, Thomas Chin 	2022-12-7 	
#005	Page Change schema model 	We will not overly flatten schemas for the sake of SQL querying. 	 	Andrew Otto, Gabriele Modena, Dan Andreescu 	2023-01 	T308017#8495502
#006	Page Change schema model 	We will not deprecate some meta fields now. 	 	Andrew Otto, Gabriele Modena, Dan Andreescu 	2023-1 	T308017#8549053
#007	Mediawiki Event Carried State Transfer streaming repo name 	mediawiki-event-enrichment 	https://etherpad.wikimedia.org/p/stream-enrichment-name-bikeshed 	Andrew Otto, Gabriele Modena, Dan Andreescu, Thomas Chin, Luke Bowmaker	2023-01 	
#008	Event Error streams 	Our convention for error event stream names is <job_name>.error, where job_name is a required setting for all jobs. E.g. mediawiki_page_content_change_enrichment is the job_name 	 	Andrew Otto, Gabriele Modena 	2023-04 	T326536
#009	Stream Major versioning 	Versioned streams should be declared with a major version suffix, like: mediawiki.page_change.v1 . Versioning streams is an opt in manual convention. Documented here: Stream_Configuration#Stream_versioning 	 	Andrew Otto, Gabriele Modena 	2023-05 	T332212
#010	User Entity Event Schema - user type booleans 	MediaWiki inconsistently models 'user types', e.g. temp, registered/anonymous, system, imported, etc. MediaWiki will one day refactor its own data model here, but we don't yet know what this data model will be. For now, we will represent these with boolean fields in the user entity data model, instead of trying to be future proof with a more flexible 'user types' field.	 	Andrew Otto, Gabriele Modena, Dan Andreescu 	2023-05 	T336506#8855469
#011	Flink HA metadata storage in zookeeper 	Using Zookeeper to store Flink HA metadata will avoid manual steps from Service Ops during kubernetes maintenance. The Zookeper cluster is owned by Service Ops. There is no formal onboarding process, but we have been given an explicit OK to use it. 	 	Andrew Otto, Gabriele Modena 	2023-05 	T331283#8858396
#012	Flink HA metadata storage in k8s ConfigMaps, until Zookeeper is available	As of 2023-07, Our deployed version of Zookeeper is too old to use for Flink HA. We will use k8s ConfigMaps instead. Supersedes #011 until we can use Zookeeper. 	 	Andrew Otto, Gabriele Modena 	2023-06 	T338233
#013	Kafka message compression 	Kafka topics produced by Flink applications should be snappy compressed. This happens by default with EventGate producers, that native Flink applications bypass. 	https://wikimedia.slack.com/archives/C03TA2WKGLB/p1693940979971639	Gabriele Modena, Luca Toscano, David Causse 	2023-09 	T345657
(Note: this doc was moved from a google sheet on 2023-06-27)
Category:Decision log