Analytics/Archive/Spark/Migration to Spark 3

The Data Engineering team has now upgraded to Spark 3 and no longer supports Spark 2. The official date upon which support ceased was 31 March 2023. The spark3 shuffler service was enabled on 5 July 2023. Please write all future jobs using Spark 3 and by now you should have finished migrating your existing jobs to Spark 3. If you still need help in doing so, please reach out.
If you are using Spark via Wmfdata-Python, you will just need to switch to a conda-analytics environment and Wmfdata-Python will automatically start using Spark 3.
== Spark 2 and Spark 3 ==
We have been supporting the execution of both Spark 2 (2.4.4) and Spark 3 (3.1.2) jobs in our Hadoop cluster for some time now. That said, Spark 3 has advantages over Spark 2 in several areas: the usual improvements in performance and security, as well as new features like better Python support (especially using Pandas) and SQL Catalog. So we prefer using Spark 3, also to stay on top of the latest Spark releases.
== There can be only one ==
We have successfully migrated many of our Spark 2 jobs to Spark 3 already, but we are facing an issue. Spark 2 and Spark 3 can not fully co-exist in the same cluster due to the Spark shuffler. The shuffler is a core component of the Spark system for which there can only be 1 version installed per cluster. Right now we’re using the Spark 2 shuffler, because Spark 3 supports it. However, Spark 3 will not work at its full capacity until we pair it with the corresponding Spark 3 shuffler, which has valuable new features like automatic handling of skewed data. Unfortunately, Spark 2 can not run with the Spark 3 shuffler. The spark3 shuffler has now been enabled and the spark2 shuffler has been decommissioned.
== Spark 2 deprecation ==
This is why we have decided to set a date at which we’ll stop supporting the execution of Spark 2 jobs in our Hadoop cluster, so we can finally switch to using the Spark 3 shuffler. The chosen date was the end of Q3 (March 31st, 2023). We put together a list of jobs that are currently using Spark 2 below. If your jobs are on the list, please migrate them as soon as possible. If you have Spark2 jobs that are not on the list, please add them, and then migrate them.
== Impact ==
This change will affect all teams that have Spark 2 jobs running in our Hadoop cluster. Teams will need to change their job submissions to point to the Spark 3 executable and libraries. It’s likely that the Spark 2 code needs (small) syntax adaptations, and we recommend that all migrated jobs are re-tested, and the data they produce re-checked. We haven’t had any major issues migrating our Spark 2 jobs to Spark 3 so far, but we recommend teams reserve some time for potential problems. Likely things to happen are: dependency issues (if you use internal Spark APIs extensively), or query plan changes that might alter the performance characteristics of computationally heavy jobs (most times for the better, but sometimes a regression can happen).
== Help ==
We Data Engineering will help teams solve their issues with Spark 3 migration as much as we can. You can reach out to us during our biweekly office hours (Tuesdays at 16:00 UTC), or via Slack using the #data-engineering channel. Here’s the official Spark docs on migrating from Spark 2 to Spark 3.
== List of Spark 2 jobs to migrate ==
Dataset / Project 	Job count 	Owners 	Stakeholders 	Needs Airflow migration	Risk 	Complexity	Spark 3
compatible	Spark 3
tested	Spark 3 in production	Comments
Puppet Druid loading 	8 (+1 in test cluster)	Data Engineering 	MediaWiki Core, Traffic 	Yes 	Medium	Low 	Yes 	No 	No 	All these jobs use the same shared Spark code. We only need to migrate this one, which should be used as well in the Airflow Druid loading jobs.
Refine 	4 (+2 in test cluster)	Data Engineering 	Data Engineering 	Yes 	High 	High 	Yes 	Yes	No 	All these jobs use the same shared Spark code. We only need to migrate this one, although it's large and complex (9 scala files, 4000 code lines).
MediaWiki History 	4 	Data Engineering 	Data Engineering 	Yes 	Medium	High 	Yes 	Yes	No 	This code is the most complex of all. It consists of 33 scala files, about 9000 lines of code. Currently 99.99% of Spark 3 output matches Spark 2.
HDFSCleaner 	4 	Data Engineering 	Data Engineering 	Yes 	Low 	Low 	Yes 	No 	No 	All these jobs use the same shared Spark code. We only need to migrate this one (1 scala file).
HistoricalProjectcountsRaw 	1 	Data Engineering 	Data Engineering 	No 	Low 	Low 	Yes 	No 	No 	Not sure we need to migrate this one. It is a legacy job that was made to be executed just once. And I think we won't have to run it again.
ProduceCanaryEvents 	1 	Data Engineering 	Data Engineering 	Yes 	Medium	Low 	Yes 	No 	No 	
WebrequestSubsetPartitioner 	1 	Data Engineering 	Data Engineering 	No? 	Low? 	Low 	Yes 	No 	No 	I think this job is currently not running anywhere? Not sure if we need to migrate it.
refinery-spark libraries 	8 libraries 	Data Engineering 	Data Engineering 	No 	Medium	Medium 	Yes 	No 	No 	Some of these libraries are used by the jobs listed in this sheet. But wanted to make them explicit, so that we don't forget any of them.
Glent 	7 	Search 	Search 	No 	Low 	Medium 	Unknown	No 	No 	
WDQS Spark tools 	6 libraries 	Search?, Wikidata?	Search, Wikidata 	No 	Medium	Medium 	Unknown	No 	No 	
Mjolnir 	8 	Search 	Search 	No 	Low 	High 	Unknown	No 	No 	
Search satisfaction 	2 	Search 	Search 	No 	Low 	 	Unknown	No 	No 	
Cirrus namespace map 	1 	Search 	Search 	No 	Low 	Low 	Unknown	No 	No 	
Relforge queries 	1 	Search 	Search 	No 	Low 	Low 	Unknown	No 	No 	
Head queries 	1 	Search 	Search 	No 	Low 	Low 	Unknown	No 	No 	
MediaWiki recommendation create 	1 	Search 	Search 	No 	Low 	Low 	Unknown	No 	No 	
MediaWiki revision predictions 	4 	Search 	Search 	No 	Low 	 	Unknown	No 	No 	
Convert to elasticsearch bulk 	5 	Search 	Search, SDoC? 	No 	Medium	Low 	Unknown	No 	No 	
Image suggestions to Cassandra 	9 queries 	Structured Data 	Structured Data 	Yes 	Low 	Medium 	Unknown	Yes	Yes 	Work done via https://phabricator.wikimedia.org/T323108
Commonswiki search index 	9 queries 	Structured Data 	Structured Data 	Yes 	Low 	Medium 	Unknown	Yes	Yes 	Work done via https://phabricator.wikimedia.org/T323108
Image suggestion indeces 	1 	Structured Data 	Structured Data 	Yes 	Low 	Medium 	Unknown	Yes	Yes 	Work done via https://phabricator.wikimedia.org/T323108
Section topics 	1 	Structured Data 	Structured Data 	No 	Low 	Low 	Yes 	Yes	Yes 	Performance decrease due to Spark's query plan change, see https://phabricator.wikimedia.org/T323107
Section Alignment 	 	Research 	 	 	 	 	Unknown	No 	No 	
Image Section Recommendation 	2 	Research 	 	Yes 	 	 	Unknown	Yes	Yes 	Work done via https://phabricator.wikimedia.org/T328641
Content gaps metrics 	 	 	 	 	 	 	 	 	 	
Project template 	 	 	 	 	 	 	 	 	 	
Image features 	 	 	 	 	 	 	 	 	 	
Link recommendation (add-a-link)	2 	Research 	Growth, ML 	No 	 	 	Unknown	No 	No 	2 spark jobs when training the link-recommendation model for a new language https://github.com/wikimedia/research-mwaddlink/blob/main/run-pipeline.sh
Welcome Survey aggregation 	1 	Growth 	Growth, Product Analytics	No 	Low 	Low 	Unknown	No 	No 	Runs monthly, https://github.com/nettrom/Growth-welcomesurvey-2018/blob/master/T275172_survey_aggregation.ipynb
iCloud Private Relay usage 	1 	Product Analytics 	Product Analytics 	No 	Low 	Low 	Unknown	No 	No 	Runs daily, stat1006:~nettrom/src/T289795/T292106-relay-pageviews.ipynb
Category:Spark