I do believe that your understanding of map is correct. However, while the distinct is not necessary for correctness, I believe it is a performance optimization, as the distinct operation is instead run distributed through Spark. I think this should provide a reasonable improvement, as for a contig fragment length of 100,000, we'd still have O(3,000) fragments per contig for the largest contigs.