apache/incubator-spark — pull request #263 by MLnick (2013-12-15 14:57:06+00:00)
Title: [PySpark] Adding support for SequenceFile and arbitrary Hadoop InputFormat

This is more of an RFC than a merge-ready pull request. It is usable for common use cases, but I would like to get a few comments from other devs about it.

This adds support for reading from SequenceFiles and Hadoop InputFormats in PySpark. The relevant methods are part of the ```PythonRDD``` object. This uses ```MsgPack``` to serialize data from Java to Python, i.e. ```JavaRDD[Array[Byte]] -> PySpark```. MsgPack deserializes the data on the PySpark side (see ```SerDeUtil``` and ```serializers.py```). In PySpark the relevant methods are part of ```context.py``` and make use of similar mechanics as for ```sc.textFile```

I still need to add the methods for old Hadoop inputformat to PySpark, and add a few tests to PySpark.

This works for SequenceFiles of "standard" writables (IntWritable, DoubleWritable, FloatWritable, Text, etc). It also works for "simple" classes that have public fields (i.e. struct-like objects), as MsgPack is easily able to serialize these. It falls back to using ```toString``` in the event that serialization fails.

I'd like to get comments especially on:
1. Overall approach
2. Thoughts on MsgPack serialization use
3. Currently some verbose warnings are thrown by MsgPack, as for example trying to call ```register(String)``` throws a warning, but MsgPack is able to serialize String without registering it. Thus in ```SerDeUtil.serMsgPack``` and ```SerDeUtil.register``` I'd like to extract the correct type information and avoid a call to ```register``` if the type is e.g. ```String```, or a primitive, etc. I thought about ```TypeTag``` for this but ran into trouble with not having the implicit parameters available and since the ultimate method is called from Java via PySpark (```PythonRDD.newAPIHadoopFile```), I'm not sure this will work.
4. Approach to "key wrapper" and "value wrapper" that would be plug in functions to allow users to provide serialization functions for their custom key and value classes (not yet implemented), for more complex Writables and input formats where MsgPack or toString fails.