initial commit for lexer
changed from straight text to use arrays. Now no need to parse rules, more readable.
added support for functions. Added support for multiple rules matching.
adding tests, refactoring.
adding one more file to commit
added more errors, refactored a bit, finished all tests
Characters are only consumed if there is a matching rule that returns a value or is ignored. Handled when JSLex.Ignore is returned from a function. Removed hasMoreTokens method. Instead, the user will call getNextToken until it returns JSLex.EndOfStream.
Instead of actually consuming the text in _input, just keep track of the index where the last match left off
renamed from JSLex to TokenJS
Merge https://github.com/jhewlett/token.js
renamed test file
removed html test file