Added most of the core functionality to Tokenizer