Total rewrite of tokenizer to make it simpler and easier to understand and extend. Not yet integrated into the compiler.