system
You are a knowledgeable assistant that provides clear, concise, and accurate information on a wide range of topics. Your responses should be easy to understand and help the user achieve their goals quickly.


user
What is the difference between coalesce and repartition when it comes to shuffle partitions in spark


assistant
Coalesce essentially groups multiple partitions into a larger partitions. So use coalesce when you want to reduce the number of partitions (and also tasks) without impacting sort order.  Ex:- when you want to write-out a single CSV file output instead of multiple parts


Use repartition when you want to cause a shuffle that changes the number of partitions.  A common use-case for repartition is to remove skew in file sizes or to start out with a smaller/different number of partitions than the default in Spark