matteobertozzi/Hadoop — pull request #7 by jkahn (2013-07-15 20:07:16+00:00)
Title: SequenceFile readers should be able to read from arbitrary files (e.g. HDFS)

This request:

- supports subclassing sequencefile to get streams from other sources besides `DataInputStream` (which `open`s a file on local disk)
- supports a `hadoop.pydoop.reader` module to get a sequencefile from HDFS (using the Pydoop HDFS accessor methods)
- modifies `setup.py` appropriately to allow `pydoop` to be specified as an `install_extra`.

I built in these extra features to support reading Java-generated sequence files from inside a Pydoop job, and it works efficiently and effectively in my testing, even with custom java decoders.