Ensure token tidying occurs in lexer

The lexer wasn't tidying up tokens that ended the input, resulting in
strange behaviour such as keywords being interpreted as identifiers. Now
the token loop takes one more pass at the end of input with the
character as null, which ends every token.