It's a ANSI C tokenizer for the Comiler Principle experiment