spotify/luigi — pull request #483 by tconklin (2014-10-03 19:51:00+00:00)
Title: add support for spark 1.x

The existing spark job class is compatible with 0.x versions of spark. This pull request adds support for 1.x versions of spark.

A spark job could have zero or many outputs and those outputs may not be well defined. Spark1xJob does not satisfy atomicity for this reason.

Added nosetest for Spark1xJob and kept original SparkJob.