Added command line tokenizer.
Disabled tokenizer test.