Created a lexing framework that uses pattern specifiying tokenizers to generate tokens from an input string