nltk/nltk — pull request #384 by kmike (2013-04-08 21:58:17+00:00)
Title: More efficient FreqDist._Nr_cache

FreqDist._Nr_cache is very sparse in real word. E.g. for Brown corpus there is only 553 non-zero values and 62160 zero values; for bigger corpora issue is worse, e.g. Russian Wikipedia has < 10K non-zero values and > 11M zero values.

Storing numbers in Python list is also quite inefficient: list of integers is an array of pointers to integers (and each integer is a Python object with a pointer to type and a refcount number).

This change saves me about 100M of memory for FreqDist with unigrams from Russian Wikipedia.