Switch to using the tokenizer (not yet working, though)
Just checkpointing to avoid work-loss
Remove the old tests...
Still just playing around with making a reasonable code tokenizer
More tests ported from code-to-words into generic tokenizer

Minor fixes in tokenizer api to make it more flexible
Finish switching from codetowords to generic tokenizer
Command to show available contexts, get tokenizer-based context gen working