I think that using encode/decode here with the arguments 'ascii' and 'ignore' has a chance of producing duplicate tokens.  If two tokens are the same, except for differing by one unicode character, the substituted value from ignore will cause them to be identical.  

What about using either the 'unicode_escape' codec, or alternatively using `binascii.b2a_qp`, which will produce output like an email header.  This should avoid potential collisions and leave the tokens rendered as generally readable values.