remove unused functions, add tokenizer function