Merge pull request #19 from nene/master

Speed up Tokenizer about 30 x
Trim trailing whitespace in tokenizer.rb
Speed up Tokenizer by not looping over all lexemes.

Previously the tokenizer looped over all lexemes, picking the longest
one that matched.  But that's too much effort.  We can order the
lexemes so that we'll just need to look for the first one that matches.

Additionally I measured a bunch of JavaScript files to see which
lexemes appear most often.  The order was like this:

- :SINGLE_CHAR
- :S
- :RAW_IDENT
- :STRING
- :COMMENT
- :LITERALS
- :NUMBER
- :REGEXP

As the :SINGLE_CHAR matched any character I split it up to two lexemes:
one at the beginning that matches single chars that are never part of a
longer lexeme, and one at the end that matches the remaining.

Also I needed to tweak the :S lexeme to only match non-empty whitespace.

This all together gave about 2 x speedup.