Put tokenization into its own class.  Supports both state based tokenization and regular expression tokenization.