BTW, one other question on this -- have you tested it with 100K partitions again? If it's slow, then one solution would be to add a "bulk" version of getPreferredLocs that returns the locations for all partitions of an RDD at once as an array of lists, and do this just once in PartitionCoalescer, then look into that array instead of calling this.