Rebuilt lexical analyzer to deal with more complex scenarios