Fixed a unicode issue and updated a test that was altered by changes to tokenisation.