Moving to tokenize

Tests pass.  Some of the stuff I changed might need refactoring.
@bifodus it looks like this isn't being used.