Pluggable tokenizer

Loads and uses a pluggable tokenizer. Comes with a sane default tokenizer.

@willkg r?
I think this makes the docstring incorrect.