Added comments/doc to tokenizer