Refactor: split pp-lexer from preprocessor, combine token classes

Split the lower level pp-lexer from the pre-processing logic.
Combine all token classes into one.