Trying to keep slow stuff out of the RDD loading inner loop; seems to have made it slower.
Removing the join and sort, and cacheing RDDs as Frank recommended