extracted and interfaced tokenizer