rewrote tokenisation to fix stack smash issue