nltk/nltk — Issue #159: SimpleGoodTuringProbDist.prob
Author: alexrudnick
Created: 2012-01-17 06:27:25+00:00
State: open

I am using 2.0.1 nltk version

the function code:

def prob(self, sample):
"""
@param sample: sample of the event
@type sample: C{string}
@return: The sample's probability.
@rtype: C{float}
"""
count = self._freqdist[sample]
p = self._prob_measure(count)
if count == 0:
if self._bins == self._freqdist.B():
p = 0.0
else:
p = p / (1.0 * self._bins - self._freqdist.B())
else:
p = p * self._renormal
return p

1) If _renormal==0 the probability return will be also zero , then the NgramModel will have MathDomain (sending zero to a log calculation is forbidden)
can we return the probability itself if the _renormal equal to zero by fixing the "p = p * self._renormal" line? :
"p = p * self._renormal if self._renormal<>0 else p"

2) if we meet unseen word , count will be zero, in that case the probability will be smoothed by self._prob_measure(count).
by default bins will equal to self._bins - self._freqdist.B() and the probability returned will be 0.0 in-spite the calculation of "p = self._prob_measure(count)"

2 technical solution :
you should not calculate _prob_measure if bins==_freqdist.B and count==0.

or

you should return the smoothed probability without checking bins==_freqdist.B().

sorry , if my English is not so good.

Migrated from http://code.google.com/p/nltk/issues/detail?id=674