Leaf tokens like characters and assertions now inherit from a BasicToken which reduces some of the code duplication.
Clean up.