It's a perfectly valid choice to put this here, but since this is just a utility function I'd consider avoiding a clone of tokenizer.js here.