Added functions for reporting errors.
Adjusted main function in relation to recent tokenizer changes.
Improved on tokenizer.

* Refined set of produced tokens.
* Re-factored tokenization process.
* Sill in need of some tidying up.