[tokenizer] added abstractions for common state types, fixed previous regression