[SPARK-4229] Create hadoop configuration in a consistent way

I seem to recall there being potential thread safety issues related to
hadoop configuration objects, resulting in the need to create / clone them.

Quick search turned up e.g.

https://issues.apache.org/jira/browse/SPARK-2546

I'm not sure how relevant that is to all of these existing situations where
new Configuration() is being called.

On Tue, Dec 9, 2014 at 5:07 PM, Tathagata Das <notifications@github.com>
wrote:

> In sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
> <https://github.com/apache/spark/pull/3543#discussion-diff-21571141>:
>
> > @@ -262,7 +263,7 @@ class SQLContext(@transient val sparkContext: SparkContext)
> >    def createParquetFile[A <: Product : TypeTag](
> >        path: String,
> >        allowExisting: Boolean = true,
> > -      conf: Configuration = new Configuration()): SchemaRDD = {
>
> I think this should be using the "hadoopConfiguration" object in the
> SparkContext. That has all the hadoop related configuration already setup
> and should be what is automatically used. @marmbrus
> <https://github.com/marmbrus> should have a better idea.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3543/files#r21571141>.
>