Rewrite the preprocessor to parse tokens into an AST and evaluate it.

This makes the code actually readable and obvious. And eliminates
all of the bugs in the old code (which was written in an extremely
error-prone way).