declassengine/declass — pull request #1 by langmore (2013-09-15 17:23:47+00:00)
Title: Convert gensim

### Gensim notebook

* `notebooks/gensim_lda_example_ian.ipynb`
* Demos how to stream from a directory structure containing flat files and do basic LSI, LDA.
* Most enhancements below are there to support this.

### High-performance streaming command line utilities

* `cmd/files_to_vw.py`
* `cmd/format_converter.py`
* Parallelized and memory independent

### Added `common.py` from jrl_utils repo

### Modifications to `declass/filefilter.py`

* Added generator functions
* Started to get it in compliance with Python (instead of e.g. Java) naming conventions

### Started `declass/gensim_helpers.py`

* Has a `TextFileCorpus` object that behaves like a gensim "corpus" and reads from text files defined by paths
* Implemented baseclass `Formatter` that `VWFormatter` and `SVMLightFormatter` inherit from
* Implemented `TokenStreamer` to stream tokens from files.