spark-jobserver/spark-jobserver — pull request #32 by velvia (2014-12-25 01:09:55+00:00)
Title: Support SQLContext, different types of contexts

This is a preview of a pretty major change, which will break binary (though should not break source) compatibility. Namely, support for SQLContext, custom SparkContexts, and more!

- Each context can be separately configured to spin up a type of context using the context-factory config setting
- Jobs must implement a matching trait (`SparkJob`, `SparkSqlJob`, etc.)
- Jobs which do not match the underlying type of context will be rejected with a new error type
- New types of contexts and job interfaces can be created by extending base traits (see `SQLContextFactory` and `SparkSqlJob` for examples)

## Use Cases
- Separate jobs to load tables into an SQL Context, and query the tables
- Custom SparkContexts to save and manage RDDs per business logic