nltk/nltk — Issue #157: Non-standard calculation of (Shannon) entropy in NgramModel
Author: alexrudnick
Created: 2012-01-17 06:27:22+00:00
State: open

This applies to 2.01rc1 at least.

The code for calculating the entropy of a text in regard to the learned n-gram model calculates the entropy as the sum of the negative log-probabilities.

/trunk/nltk/nltk/model/ngram.py line 133:
133: e += self.logprob(token, context)

Shouldn't this be:
e += self.prob(token, context) * self.logprob(token, context)
Shannon entropy is the -1 \times the sum of the product of the probability and the log-probability.

This could be re-implemented to call the probability calculation function only once by doing:
token_prob = self.prob(token, context)
e += token_prob * -log(token_prob, 2)

Migrated from http://code.google.com/p/nltk/issues/detail?id=672

earlier comments

rico.sennrich@gmx.ch said, at 2011-07-21T16:02:23.000Z:

for "weighting" each log probability, you don't want to use the model probability, but the "true" probability, which we don't know.

A common approximation is to add all log probabilities of a text together, then divide the result by the number of tokens in the text.

source: http://en.wikipedia.org/wiki/Cross-entropy