handle the case were the token passed in is a unicode string.

I struggled on whether or not this code change should be in my code or in here but I think its to more peoples benefit that we attempt to handle unicode terms. Slightly nervous about this fix since it involves unicode and sensitive stuff. All tests are ok though.