The Secrets of Influencing Spark Partitions during reads - Spark Partitioning (Part 8)
Channel: Data with Nikk the Greek
Date: 2024-01-27

In today's episode we will deep dive into the influence factors on partitions and how we choose a good partition based on it. The behaviours include:
- Max Partitions Bytes
- Open Cost in Bytes
- Num of cores
- File size
- Num of files

Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.

Link to the code can be found here: https://github.com/datanikkthegreek/SparkDeltaDatabricksInternals/blob/main/Lection2%20-%20Spark%20Partitions/S02E08_InfluenceOfMaxPartitionsBytes.ipynb