Closes SPARK-4229 Create hadoop configuration in a consistent way

Unrelated to this PR?