twitter/pycascading — Issue #2: Serializations errors with default configuration
Author: ebernhardson
Created: 2011-12-27 01:21:45+00:00
State: open

Upon initial setup of pycascading i could not get the examples to run, instead the following exception would be spit out:

WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
at org.apache.hadoop.io.serializer.SerializationFactory.getDeserializer(SerializationFactory.java:77)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:392)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:411)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

The class it is attempting to deserialize is cascading.tap.hadoop.MultiInputSplit, which should be handled by org.apache.hadoop.io.serializer.WritableSerialization but that is not available to the SerializationFactory for some reason.

The serializations available(as per jdb) are com.twitter.pycascading.bigintegerserialization.BigIntegerSerialization(id=3707), instance of cascading.tuple.hadoop.TupleSerialization

Adjusting pycascading/java/src/com/twitter/pycascading/Util.java:79 where the io.serializations property is set to include WritableSerialization, or commenting the line out completely, allows the examples to run to completion. I'm not entirely sure why WritableSerialization would be included when that line is commented out, but not when its used.

Software versions:

cascading-1.2.4-hadoop-0.19.2+
hadoop-0.20.204.0
jython-2.5.2