handle the case where unicode is passed in as a token.
mucky merge