How to Screw up your Repartitioning! - Spark Partitioning (Part 11)
Channel: Data with Nikk the Greek
Date: 14/02/2024

In today's episode we will screw up repartitioning. On purpose of course 😉 By that we discover how the following Spark Partitioning and shuffling concepts work:
- Round Robin Partitioning
- Hash Partitioning
- Range Partitioning

You want to master Data Engineering with PySpark? Subscribe here: https://www.youtube.com/@DataNikktheGreek?sub_confirmation=1

Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.

Link to the code can be found here: https://github.com/datanikkthegreek/SparkDeltaDatabricksInternals/blob/main/Lection2%20-%20Spark%20Partitions/S02E11_RepartitionDetails.ipynb

00:00 - Intro
01:50 - Recap Repartition
03:30 - Round Robin Partitioning
11:15 - Hash Partitioning
18:55 - Range Partitioning
23:00 - Summary 

#spark #pyspark #dataengineering #dataengineeringessentials