Added instructor's tokenizer.c code and integrated it with the project