piskvorky/gensim — Issue #76: Add hashing dictionary
Author: piskvorky
Created: 2012-02-15 11:04:38+00:00
State: open
Labels: feature, medium difficult

Instead of collecting all token types (by one extra pass) and assigning each a unique id, as `Dictionary` does now, implement a "hashing dictionary" which simply hashes input features onto a predefined id range.

This will allow:

1. converting documents to a vector space online, as soon as they are seen (no extra corpus pass needed)
2. adding new features without re-training. New features are simply hashed into the same range. In fact, the whole concept of old/new features ceases to exist.

Downside: feature collisions.

Some articles about the "hashing trick": http://hunch.net/~jl/projects/hash_reps/index.html

Technically:

* create a `HashDict` class that has the same interface and functionality as `Dictionary` now (`corpora/hashdict.py`)
* the constructor of this `HashDict` will accept the desired id range = an integer
* will implement `doc2bow` and `__getitem__` methods by means of hashing
* maybe also implement `filter_extremes()`, if you feel daring :)
* unittests