Tes4Mod:Oblivion Token Tables

The input character steam will be tokenized using a table based rules as defined in text files.

Type Definitions
The basic type definitions for all the classes needed for a table based tokenizer are given below:

   tokentableresult {
      int        Action  { ignore moveto return error continue jmpreturn jumpto }     //Type of action to take
      string     TableName            
      table*     Table                 //Destination for a table move/jump type
      string     TokenName            
      tokentype* TokenType             //Token to return for a return type
      string     Message               //User message for an error type
   }
  
   tokentablerow {
      int               MatchType { none charclass char string }    //Type of match to perform
      string            CharClassName  
      charclass*        CharClass      //Class to match for a class type row
      char              Char           //Character to match for a character type row
      string            StringValue    //String to match for a string type row
      tokentableresult  Result         //Action to take when the row matches
   }
  
   tokentable {
      string         Name              //Identifies the table
      tokentablerow  Rows[]            //Array of all rows in the table
      tokentablerow* DefaultRow        //Default row to be used when no matches are found
   }
  
   tokentablearray {
      tokentable   Tables[]            //Array of all tables
      tokentable*  StartTable          //The table to start tokenizing with
   }


Data File Format
The token tables are defined in a text file with the following format:

   [TableName]
      [RowMatch] = [Action]
      ...
   End
   ...

[RowMatch] can have the following values:
:*CharacterClass -- A character class name as defined in the current class file.
:*"[string]" -- A string surrounded by double quotes.
:*[Character] -- A single character.
:*Default -- The default row which is used when no match in the table is found.

The [Action] for a row can have the following values:
:*ignore -- Ignore the current character and restart parsing the current token table. 
:*moveto [TableName] -- Add the current character to the token and start parsing the given token table.
:*jumpto [TableName] -- Start parsing the given token table (does not add the current character to the token).
:*return [TokenType] -- Add the current character to the token and return the given token type. Clear the current token value.
:*jmpreturn [TokenType] -- Return the given token type (does not add the current character to the token). Clear the current token value.
:*continue -- Add the current character to the token and restart parsing the current table.
:*error [Message] -- Add the current character to the token and report an error using the supplied string. Clear the current token value.

Oblivion Token Tables
The following token tables (along with the matching Character Classes and Token Types) are capable of successfully tokenizing all of the 1687 scripts found in Oblivion.

   Begin
      Comment    = moveto Comment_Table
      Digit      = moveto Digit_Table
      Quote      = moveto String_Table
      IDStart    = moveto ID_Table
      (          = return LBracket
      )          = return RBracket
      AddOp      = return AddOp
      MultOp     = return MultOp
      &          = moveto And_Table
      |          = moveto Or_Table
      Equal      = moveto EQ_Table
      >          = moveto GT_Table
       ,  =, ==, and !=
   End
   
   GT_Table
      Equal   = return RelOp 
      Default = jmpreturn RelOp
   End
 
   LT_Table      Equal   = return RelOp
      Default = jmpreturn RelOp
   End
 
   NEQ_Table
      Equal   = return RelOp
      Default = error Unexpected ! found! Valid comparison operators are:  ,  =, ==, and !=
   End