Spark Partitions when Loading Files - Spark Partitioning (Part 6)
Channel: Data with Nikk the Greek
Date: 2024-01-21

In  this episode we have a look how the partitions are influenced by Spark when loading parquet files

Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.

Link to the code can be found here: https://github.com/datanikkthegreek/SparkDeltaDatabricksInternals/blob/main/Lection2%20-%20Spark%20Partitions/S02E06_PartitionsBasedOnFileSize.ipynb