lexer: close the tokens channel after the last state change

Signed-off-by: Alejandro Mery <amery@geeks.cl>