Code cleanup, tokenise method made private, streamlined interface
Added test