lintool/Ivory — pull request #18 by ferhanture (2011-10-13 17:24:59+00:00)
Title: Changes to preprocessing/tokenization code

Tokenizer class made abstract, added default methods.
Added Chinese tokenizer, uses modified version of Stanford NLP package (I need to add jar file)
Various updates to CLIRUtils class
Fixed PreprocessWikipedia class: modified how arguments are passed to docnoMapping/refactoring classes.