Hey Tom, one concern I have is with this being a singleton object. This will make it tough to have multiple Spark clients in the same JVM (e.g. if someone wants to submit jobs to multiple clusters, or even in unit tests). Is there any chance we can make this object part of SparkEnv instead and just pass it around by reference?

Other than that, the approach looks great, though I will look at specific pieces in more detail. Thanks for updating the documentation too.