tarasglek/pydoop — pull request #3 by bsmedberg (2013-03-30 19:12:14+00:00)
Title: Despite this being called the "combiner" branch, this does everything except actually implement a combiner

Present in this PR:
* Implement "TypeWritable" which efficiently wraps python strings, ints, floats, and tuples of these.
* Start using jyson as a drop-in replacement for the jython "json" module which performs horribly.
* Implement working reducers
* Implement a final combiner which takes the input from multiple reducers and combines it into a single result file on the local machine.

This change has been tested against telemetry data on mango-gw and works.

Next steps:
* Implement the combiner
* Have TypeWritable implement RawComparator for extra speed
* Do some additional type checking in the python driver for better debugging before deployment
* Make the output serialization of tuples produce excel-tab CSV format
* Allow scripts to customize their output (already implemented in python driver, sorta)
* Change the dispatch mechanism to include all of "scripts" in the JAR file and select which script to run dynamically. Allow importing of utility modules i.e. "combiner = pydoop.countcombine; reduce = pydoop.countreduce"