Rename Tokenizer to BasicTokenizer.  Make several functions virtual.