update tokenizer to be smarter, allow undefined tokenizer