added a tokenizer for general parsing of the tokens.