ParrotFx/Parrot — pull request #4 by hiisi13 (2013-07-22 06:20:14+00:00)
Title: Tokenizer refactoring + Tests restructuring

I made some changes to Tokenizer in order remove duplications (mostly with token instances creation) and make code a little cleaner and better structured. Also I've split tests fixture into different classes and files so it's easier to navigate between tests and its results.

I decided not to squash commits so it would be easier to review - I tried to make changes in relatively small logical steps. Please let me know if you want me squash them first, though.