- Added a few more test cases for tokenizer and fixed a bug.

Signed-off-by: Kristian Duske <kristian.duske@gmail.com>