But to avoid this expensive approach one can use the information theory quantity of entropy to get an estimate of how good a LM might be. The basic idea is to average the log probability on per word basis for a piece of new text not used in building the language model. Denote by p the true distribution, that is unknown to us, of a segment of new text x of k words. To determine the probability of this segment of text we will use our language model denoted by which is different from the true unknown distribution p of the new text. Obviously our goal is to find that LM which has an average logprob that is as close as possible to the entropy of the text.
Perplexity is defined as .
Perplexity is, crudely speaking, a measure of the size of the set of words from which the next word is chosen given that we observe the history of spoken words. The perplexity of a LM depends on the domain of discourse. For radiology reports, one expects less variation in the sentences than in general English. Perplexity of trigram models for different domains. The error rate of a speech recognizer is no less than the percentage of spoken words that are not in its vocabulary V .