puffnfresh/roy — pull request #188 by rtfeldman (2013-07-28 22:24:46+00:00)
Title: Lexer refactor

Refactored lexer in support of removing `let` (which is almost there - 2 failing test cases remaining).

The main changes are:

1. Put the keywords in a map, so we can easily check if something is a keyword or not by checking its presence in the map.
1. Parameterize the tokenize functions on `chunk`, so we can process a substring of `chunk` instead of the whole thing.
1. Pull out a `tokenise` function that can be used outside the main `exports.tokenise`.

This is a big enough change to a critical piece of code that I figured a PR would be appropriate.