Add a minimal documentation on the tokenizer API.

https://code.google.com/p/esprima/issues/detail?id=398
Fast path in advance() should check for semicolon instead of colon.

https://code.google.com/p/esprima/issues/detail?id=534
Revert "Fix escaped identifier being treated as a keyword."

https://code.google.com/p/esprima/issues/detail?id=496
Disallow escaped characters in a regular expression flag.

https://code.google.com/p/esprima/issues/detail?id=503
Automatically rewritten for npm.