started work on a refactoring of langid.py training tools into finer-grained components
added a standalone corpus indexer
added a common functions module
added byte-NGram tokenizer
added DF-based feature selector
introduced a new contextmanager MapPool
added code to compute IG weights
replaced NaN with 0 in entropy calculation
fixed bug where lang and domain were swapped around
added LD feature selection based on IG weights