Add new feature to enable parser deal with tokens in different lexer states. (also some bug fixings)