fix tokenizer bug where multiple delimiters would be treated as one