Modify Tokenizer to provide a stream-like API.
Remove obsolete tests.