don't insert original token into residual hash
use boost::locale::to_lower for utf8 support