Added some hacks to handle edge token cases.
Converted probability calculations to log space.