apache/spark — pull request #2849 by olegz (2014-10-20 00:40:17+00:00)
Title: Initial commit to provide pluggable strategy to facilitate access to nat...

Initial commit to provide pluggable strategy to facilitate access to native Hadoop resources

Added HadoopExecutionContext trait and its default implementation DefaultHadoopExecutionContext
Modified SparkContext to instantiate and delegate to the instance of HadoopExecutionContext where appropriate

Changed HadoopExecutionContext to JobExecutionContext
Changed DefaultHadoopExecutionContext to DefaultExecutionContext
Name changes are due to the fact that when Spark executes outside of Hadoop having Hadoop in the name woudl be confusing
Added initial documentation and tests

polished scaladoc

annotated JobExecutionContext with @DeveloperAPI

eliminated TaskScheduler null checks in favor of NoOpTaskScheduler
to be used in cases where execution of Spark DAG is delegated to an external execution environment

added execution-context check to SparkSubmit

Added recognition of execution-context to SparkContext
updated spark-class script to recognize when 'execution-context:' is used

polished merge

changed annotations from @DeveloperApi to @Experimental as part of the PR suggestion

externalized persist and unpersist operations

added classpath hooks to spark-class