Fixes and refactoring for tokenizer.
Added more tests for tokenizer.

Includes some refactoring of the testing code, and the addition of some
valid manifest files.