[SPARK-2534] Avoid pulling in the entire RDD in various operators

This should go into both master and branch-1.0.
That makes sense.