Removed corpus from tokenization and re-factored to generator functions.