added options to output additional information about features
fixed calculation of IG to use DF instead of TF
implemented indexing using a fixed file list
added support for term-frequency based tokenization
Merge branch 'master' of https://github.com/saffsd/langid.py