Doxygen me up, Scotty!
Amalgamated tokeniser changes.

Remove the for_each, change from start-end to start-nread for a more
obvious API, and fix a broken assert.
More doxygenation.