Refactor SparkConf usage

Mostly cherry-picked from previous test patches.