added tokenizer function for individual functions