Restructured lexical analyzer to parse multiple tokens.