Improve commenting. Remove testtokenizer.
Testtokenizer wasn't particularly likely to fail OR to catch
issues in the code.