Functional implementation of an early tokenizer.

Works better than the previous one. We parse by character, not by line, so now it can be more exact if needed.
Also it specifies the correct line number upon error, or if a warning is triggered.