tenderlove/rkelly — pull request #19 by nene (2013-06-04 13:48:21+00:00)
Title: Speed up Tokenizer about 30 x

Slicing the string after each token made the tokenizing process
really slow.

I eliminated this inneficency by using Ruby's builtin StringScanner,
that's well suited for this task.

Before this change, it took about 14 seconds to tokenize a 3000-line
JavaScript file. Now it takes just about 0.4 seconds. That's about
30 x speedup.

It's still slow in my opinion, but at least it's not dead-slow any more.