rewriting the tokenizer without regular expressions