added a gitignore
added the ability to select tokens without conditioning on token order
replaced redundant third item in scanner tuple with a list of features
completed a TODO regarding using nm_arr instead of next_move
added the option to tokenize using a pre-trained Scanner
removed obsoleted training tools
made the train submodule importable
swapped order of arguments to write_weights to be consistent with write_features
added a classmethod to facilitate instansiating a scanner using pickled parameters
made scanner automatically upgrade the return array type as needed
renamed Tokenizer to NGramTokenizer
refactored to use new from_file classmethod of Scanner
changed chunk_size to chunksize for consistency with tokenize.py
added a single-file training tool for langid.py
made the argument to --doc_count option, with a default of DF_all
added a 'word' tokenization option using str.strip
added an option to skip first-pass feature selection
fixed an off-by-one bug
labelled tokenize tempdir with the tokenizer that produced it
remove temporary files from first-pass tokenization