Added TokenRegExp and unit tests.
Code cleanup.