nltk/nltk — Issue #133: NgramModel fails to initialize with specified estimators
Author: alexrudnick
Created: 2012-01-17 06:26:49+00:00
State: open

What version of NLTK are you using? 2.0b9

What steps will reproduce the problem? (e.g. include Python source code)
from nltk import *
text = "test"
#Test 1
lmL = NgramModel(1, "test",nltk.LaplaceProbDist)
#Test 2
lmS = NgramModel(1, "test",nltk.SimpleGoodTuringProbDist)

What is the expected output? What do you see instead?

Expected:
lmL is initialized to language model using LaPlaceProbDist
Instead:
File "/Library/Python/2.6/site-packages/nltk/model/ngram.py", line 63, in __init__
self._model = ConditionalProbDist(cfd, estimator, len(cfd))
File "/Library/Python/2.6/site-packages/nltk/probability.py", line 2016, in __init__
**factory_kw_args)
File "/Library/Python/2.6/site-packages/nltk/probability.py", line 884, in __init__
LidstoneProbDist.__init__(self, freqdist, 1, bins)
File "/Library/Python/2.6/site-packages/nltk/probability.py", line 810, in __init__
'to create it (%d).' % freqdist.N())
ValueError:
The number of bins in a Laplace distribution (1) must be greater than or equal to
the number of bins in the FreqDist used to create it (4).

Expected:
lmS is initialized to language model using SimplegoodTuringProbDist
Instead:
File "/Library/Python/2.6/site-packages/nltk/model/ngram.py", line 63, in __init__
self._model = ConditionalProbDist(cfd, estimator, len(cfd))
File "/Library/Python/2.6/site-packages/nltk/probability.py", line 2016, in __init__
**factory_kw_args)
File "/Library/Python/2.6/site-packages/nltk/probability.py", line 1398, in __init__
'Bins parameter must not be less than freqdist.B()'
AssertionError: Bins parameter must not be less than freqdist.B()
Please use labels and text to provide additional information.

Let me know if I'm doing something wrong. It looks like bins is not getting set when initializing certain estimators? I'm trying this in order to get estimated probs for contexts that are missing in the training set.

All best, thanks,

Peter

Migrated from http://code.google.com/p/nltk/issues/detail?id=616