alternate version using provided tokenizer