Refactor tokenization

We now use a TokenReader interface to abstract how tokens are retrieved. They
can now also be retrieved one at a time, as opposed to all at once.