Merge pull request #7 from aitzol/master

Merge fix for places where 'instance' was used instead of 'text'
fixed encoding unicode strings not actually having any effect
added an error message if the temporary directory specified does not actually exist
added option to process piped input line-by-line
Merge branch 'master' of https://github.com/saffsd/langid.py

Conflicts:
	langid/langid.py
Merge branch 'master' of https://github.com/saffsd/langid.py

Conflicts:
	langid/langid.py
Added a parameter to allow use of only a random proportion of the training data
added the ability to specify a minimum domain count before we include a language
use islice in chunk
fixed typo
fixed calculation of chunk count
rewrote IG calculation. tests show this cuts LDfeatureselect runtime in half, apparently regardless of corpora size
started work on a refactoring of langid.py training tools into finer-grained components
added a standalone corpus indexer
added a common functions module
added byte-NGram tokenizer
added DF-based feature selector
introduced a new contextmanager MapPool
added code to compute IG weights
replaced NaN with 0 in entropy calculation