[SPARK-4229] Create hadoop configuration in a consistent way

This should also be the configuration from the `sparkContext.hadoopConfiguration`