Adds basic tokenization to the compiler.