Tokenizer refactory.
Fix: using isIdentifierChar
3.2: Added support for the pattern $, _ matching any token, whereas
     $1 $2 (_1 _2 etc.) matching tokens that compare equal in the
     corresponding occurrences.