Made complex tokenizer test case more complex.
Fixed bug in tokenizer test suite.