Question: Why are statistics from finite corpora unreliable?
Answer: $\mathit {PMI}(w,c)$ goes to negative infinity when the word-context pair $(w,c)$ does not appear in the training corpus
Alternative answer: A finite corpora may entirely omit rare word combinations