Rewrite inline tokenization

This makes the tokenization code significantly more readable and
allows much more powerful tokenization going further.