handle the case were the token passed in is a unicode string.