Changed tokenizer to memory based tokenization. Need to add stream tokenizers that takes into account the differences between Linux and Windows when calling read() on files. Character vs. byte wise.