[SPARK-2889] Create Hadoop config objects consistently.

Different places in the code were intantiating Configuration / YarnConfiguration objects in different ways. This could lead to confusion for people who actually expected "spark.hadoop.*" options to end up in the configs used by Spark code, since that would only happen for the SparkContext's config.

This change modifies most places to use SparkHadoopUtil to initialize configs, and make that method do the translation that previously was only done inside SparkContext.

The places that were not changed fall in one of the following categories:
- Test code where this doesn't really matter
- Places deep in the code where plumbing SparkConf would be too difficult for very little gain
- Default values for arguments - since the caller can provide their own config in that case
It's the same as the rest of the codebase -- everything that is "private" should be marked `private[spark]`. Things that we need to make public for advanced developers are `@DeveloperApi`. In this case, this thing has been public so we can't remove it, but we could at least mark it to tell people not to depend on it.