tenderlove/rkelly — pull request #24 by nene (2013-06-13 14:21:12+00:00)
Title: Another approach to speed up tokenizer

My [previous PR for this](https://github.com/tenderlove/rkelly/pull/20) was somewhat hacky. So I took a bit of a different approach. Building a lookup table of all the characters - so at each tokenization step we quickly determine possible tokens that can follow by looking at the next character.

Combining this with some other optimization I get about 2.5 x speedup of the tokenizer.