Tokeniser code finished.

Code for tokenising input completed and commented.
More may be added in future as required.