add http timeout for httpbroadcast
Merge pull request #210 from haitaoyao/http-timeout

add http timeout for httpbroadcast

While pulling task bytecode from HttpBroadcast server, there's no timeout value set. This may cause spark executor code hang and other task in the same executor process wait for the lock. I have encountered the issue in my cluster. Here's the stacktrace I captured  : https://gist.github.com/haitaoyao/7655830

So add a time out value to ensure the task fail fast.
Bugfix: SPARK-965 & SPARK-966

SPARK-965: https://spark-project.atlassian.net/browse/SPARK-965
SPARK-966: https://spark-project.atlassian.net/browse/SPARK-966

* Add back DAGScheduler.start(), eventProcessActor is created and started here.

  Notice that function is only called by SparkContext.

* Cancel the scheduled stage resubmission task when stopping eventProcessActor

* Add a new DAGSchedulerEvent ResubmitFailedStages

  This event message is sent by the scheduled stage resubmission task to eventProcessActor.  In this way, DAGScheduler.resubmitFailedStages is guaranteed to be executed from the same thread that runs DAGScheduler.processEvent.

  Please refer to discussion in SPARK-966 for details.
Updated some inline comments in DAGScheduler
More comments
Merge pull request #216 from liancheng/fix-spark-966

Bugfix: SPARK-965 & SPARK-966

SPARK-965: https://spark-project.atlassian.net/browse/SPARK-965
SPARK-966: https://spark-project.atlassian.net/browse/SPARK-966

* Add back `DAGScheduler.start()`, `eventProcessActor` is created and started here.

  Notice that function is only called by `SparkContext`.

* Cancel the scheduled stage resubmission task when stopping `eventProcessActor`

* Add a new `DAGSchedulerEvent` `ResubmitFailedStages`

  This event message is sent by the scheduled stage resubmission task to `eventProcessActor`.  In this way, `DAGScheduler.resubmitFailedStages()` is guaranteed to be executed from the same thread that runs `DAGScheduler.processEvent()`.

  Please refer to discussion in [SPARK-966](https://spark-project.atlassian.net/browse/SPARK-966) for details.
Added stageId <--> jobId mapping in DAGScheduler
  ...and make sure that DAGScheduler data structures are cleaned up on job completion.
  Initial effort and discussion at https://github.com/mesos/spark/pull/842
Actor instead of eventQueue for LocalJobCompleted
Fixed intended side-effects
Removed redundant residual re: reverted refactoring.
Improved comment
Refactoring to make job removal, stage removal, task cancellation clearer
Cleaned up job cancellation handling
Tightly couple stageIdToJobIds and jobIdToStageIds
Local jobs post SparkListenerJobEnd, and DAGScheduler data structure
cleanup always occurs before any posting of SparkListenerJobEnd.
Synchronous, inline cleanup after runLocally
SparkListenerJobStart posted from local jobs