removed the main from tokenizer