Added new objects that indicate the status of a token (however, they are not consistently used yet).
Added a CharacterRange class to encapsulate ranges in character classes better.