Revert "Added flag file tokenizer."

This reverts commit 85102c3e0901ff0a67e7c6eeee10aa6e9341b57b.