bug fixes for tokenizer