Added rudimentary tokenizer. It still needs a bit of work especially on structured tokens but the basics are there.