modified tokenizer to accept tokens from f(C)