The hash set requires extra memory, and in the worst case, this is proportional to the partition size. `RDD#distinct()` should be better.

But see my comments below about re-using `makeInLinkBlocks` and `makeOutLinkBlocks`.