[SPARK-2098] All Spark processes should support spark-defaults.conf, config file

This is another implementation about #1256
cc @andrewor14 @vanzin
This whole block (from L56) is repeated three times in your patch. Could you add a helper method for this instead?

You could have something like:

    def loadPropertiesFile(file: Option[String], conf: SparkConf): Map[String, String] = {
      val path = file.getOrElse(getDefaultConfigFile()))
      getPropertiesFromFile(file)
        .filter { case (k, v) =>
          k.startsWith("spark.")
        }
        .foreach { case (k, v) =>
          conf.setIfMissing(k, v)
          sys.props.getOrElseUpdate(k, v)
        }
    }

And this whole block becomes a single method call. If you're creative you could even come up with an API that could be used by the SparkSubmit code too.