Broke some tokenizer functionality out to file_iter_t.