some fixing & added a few unit tests for tokenizer