refactored tokenizer to use pattern matching in token constructor