integrate tokenizer into main