Why not just use a mapPartitions and work on one partition at a time? You don't want to collect all the data back to the driver.