[SPARK-2521] Broadcast RDD object (instead of sending it along with every task)

This is a resubmission of #1452. It was reverted because it broke the build.


Currently (as of Spark 1.0.1), Spark sends RDD object (which contains closures) using Akka along with the task itself to the executors. This is inefficient because all tasks in the same stage use the same RDD object, but we have to send RDD object multiple times to the executors. This is especially bad when a closure references some variable that is very large. The current design led to users having to explicitly broadcast large variables.

The patch uses broadcast to send RDD objects and the closures to executors, and use Akka to only send a reference to the broadcast RDD/closure along with the partition specific information for the task. For those of you who know more about the internals, Spark already relies on broadcast to send the Hadoop JobConf every time it uses the Hadoop input, because the JobConf is large.

The user-facing impact of the change include:

1. Users won't need to decide what to broadcast anymore, unless they would want to use a large object multiple times in different operations
2. Task size will get smaller, resulting in faster scheduling and higher task dispatch throughput. 

In addition, the change will simplify some internals of Spark, eliminating the need to maintain task caches and the complex logic to broadcast JobConf (which also led to a deadlock recently).


A simple way to test this:
```scala
val a = new Array[Byte](1000*1000); scala.util.Random.nextBytes(a);
sc.parallelize(1 to 1000, 1000).map { x => a; x }.groupBy { x => a; x }.count
```

Numbers on 3 r3.8xlarge instances on EC2
```
master branch: 5.648436068 s, 4.715361895 s, 5.360161877 s
with this change: 3.416348793 s, 1.477846558 s, 1.553432156 s
```

As the code currently stands, you don't need to remove the stage from runningStages here -- because it doesn't get added until below (line 738).  BUT, the test failure you're seeing is occurring because the stage is not yet in runningStages.  If you look on line 1049, we only post the SparkListenerStageCompleted event if the stage was in runningStages.  So, because the stage hasn't yet been added to runningStages when you call abortStage, the listener event is never posted.  You'll have to poke around to figure out the right way to fix this...I'm not sure whether it makes sense to just delete the check on line 1049 or if that's necessary for other reasons.