This method is really a lexer in and of itself. Can this be redesigned as a second lexer and then control is passed back and forth between the that lexer and the original one?