This tries to avoid having lots of partitions for just a few items. Since the nature of the algorithm is analyzing subsets of the initial candidates set, each time the RDD has fewer elements. If you don't coalesce the RDD you end up with an RDD with as many blocks as the original data and very few items.