improved tokeniser and result sorting