ariya/esprima — pull request #162 by espadrine (2013-02-17 17:54:06+00:00)
Title: Add a Tokenizer API.

This adds a `tokenize(code, options)` function to the exported object.

It uses the same API as when `options.tokens` is set.

It also uses Tim Disney's algorithm documented here:
https://github.com/mozilla/sweet.js/wiki/design

http://code.google.com/p/esprima/issues/detail?id=398