Why are over partitioned tables poor data layout strategy


Over-partitioning creates physical boundaries beyond which the file compaction and z-ordering aren’t effective and you’re still left with the small files problems. With Delta, a better strategy is to use combination of partitioning and z-ordering to allow for better file layouts. 

Would be helpful to understand the business requirements & design details behind these highly partitioned tables you have in parquet today