commit refactorings to eng-tokenizer