This inner loop takes a bit to understand. It may be worth re-writing it (maybe outside of the mapPartition). Basically we want to get numTotal and initial numBigger for each partition. You can use mapPartitionsWithIndex to make this much more readable.