More fixes in data flow to simplify it.
Reorganization of the code base, comments on it.
Fold token handling code into the main loop.