fixes a few flaws in the tokenizer