Separated out the generated tokenizer code from the fixed tokenizer
code.