lexer: use range to iterate over tokens in test.
lexer: add package documentation.