apache/incubator-spark — pull request #55 by velvia (2013-10-12 07:15:25+00:00)
Title: SPARK-544 Part I: Replace system properties with Typesafe Config

This is the first part of an extensive change to Spark's configuration system, moving from one based on using System properties as global mutable state, to one based on an immutable Config object.

Summary of this PR:
- SparkContext now has an additional parameter, config, to allow passing overrides for the default config
- Order of config priorities: config object passed into SparkContext >> System properties >> application.conf >> reference.conf (defaults)
- All default config values will be centralized into one "reference.conf" in src/main/resources, with plenty of comments
- spark.driver.host, spark.driver.port, spark.akka.* converted to use the new Config system
- spark.hostPort removed entirely
- No more clearing spark.driver.port and spark.hostPort between tests -- this was removed from core tests
- Config objects, rather than lists of system properties, are now passed from the scheduler to Executors in both Standalone and Mesos modes

Long term benefits:
- No more pollution and hard to track bugs from System properties leakage
- Removes one major obstacle to running concurrent SparkContexts in the same JVM (can you say parallel tests?)
- Availability of rich config types (including lists, ints, time units) instead of just strings

Current status:
- All tests in core pass
- Lots of property settings still to be converted
- No work done in YARN mode, or in MLLib / Bagel / other modules