Tokenizer.c — Implements a tokenizer function.