mutual information calculation for a large doc set.Scalable. Tested on 4K doc set (2% the size of the original repo)