lexer: improve Lexer.NextToken() to handle nextState == nil

Signed-off-by: Alejandro Mery <amery@geeks.cl>