nltk/nltk — Issue #167: Unseen token error in nltk.model.NgramModel.entropy
Author: alexrudnick
Created: 2012-01-17 06:27:35+00:00
State: open

NLTK version 2.0b9

I'm attempting to compute the entropy of a piece of text with respect to a given model and there seems to be an issue with unseen tokens (even when an estimator is used for the model).

from nltk.model import NgramModel
from nltk.corpus import brown
from nltk.probability import WittenBellProbDist

estimator = lambda fdist, bins: WittenBellProbDist(fdist,0.2)
lm = NgramModel(3, brown.words(categories='news'), estimator)

sent = "this is a sentence with the word aardvark".split()
lm.entropy(sent)

Traceback (most recent call last):
File "<pyshell#28>", line 1, in <module>
lm.entropy("this is a sentence with the word aardvark".split())
File "C:\Python26\lib\site-packages\nltk\model\ngram.py", line 133, in entropy
e += self.logprob(token, context)
File "C:\Python26\lib\site-packages\nltk\model\ngram.py", line 98, in logprob
return -log(self.prob(word, context), 2)
File "C:\Python26\lib\site-packages\nltk\model\ngram.py", line 79, in prob
return self._alpha(context) * self._backoff.prob(word, context[1:])
File "C:\Python26\lib\site-packages\nltk\model\ngram.py", line 82, in prob
"context %s" % (word, ' '.join(context)))
TypeError: not all arguments converted during string formatting

An exception is not raised (and a value is returned) if the entropy is computed using the same sentence but with 'aardvark' removed, which leads me to believe it's an error with unseen tokens. However, my understanding is that the model should be equipped to handle unseen tokens as indicated by the estimator.

Migrated from http://code.google.com/p/nltk/issues/detail?id=694

earlier comments

SilaiSantai said, at 2011-07-29T11:23:30.000Z:

I am having the exact same problem (though I use a LidstoneProbDist as my estimator). I also looked at it and it seems that it is unable to handle unseen words in the text.

This would seem to be a major issue for the NgramModel method as it is not much use if it cannot handle unseen words.

SilaiSantai said, at 2011-07-29T12:20:45.000Z:

The fix written here:

http://code.google.com/p/nltk/issues/detail?id=673&can=1&q=ngrammodel

Solves the issue it seems. At least now it yields a result.