mesos/spark — pull request #805 by woggle (2013-08-11 04:55:41+00:00)
Title: Use new Configuration() instead of slower new JobConf() in SerializableWritable

JobConf's constructor apparently loads default config files in some versions of Hadoop (e.g. https://github.com/apache/hadoop-common/blob/release-1.2.1/src/mapred/org/apache/hadoop/mapred/JobConf.java#L343 calls checkAndWarnDeprecation(), which calls get(), which it needs to check config files to see if deprecated configs are used.) This means that deserializing a SerializableWritable instances rereads these files every time. As far as I can tell, the Configuration passed to ObjectWritable is only used for its embedded ClassLoader, so besides this deprecation check, there would be no reason to load any config files otherwise. Fortunately, there seems to be no similar checks in hadoop.conf.Configuration, and that's actually the type expected by ObjectWritable anyways.