Repository: piskvorky/gensim

Issue:
Dictionary gensim.corpora should update the collection frequencies cfs after filtering with filter_extremes
#### Problem description

I am using the Dictionary class gensim.corpora.dictionary.Dictionary , in particular the filter_extremes method and the cfs property (returning a collection frequencies dictionary mapping token_id to tokenfrequency). 
It appears that after using  filter_extremes, the cfs still shows unfiltered token_id's can be absent from the filtered Dictionary instance. This can lead to KeyErrors. 

#### Steps to reproduce

```python
from gensim.corpora import Dictionary
corpus = [['common','single'],['common']]
dct = Dictionary(corpus)
len(dct)  #2
dct.cfs   #{0: 2, 1: 1}
dct.filter_extremes(no_below=1, no_above=0.5, keep_n=1)
len(dct)  #1
dct.cfs   #{1: 1}
dct[1]
```

The last line fails with 'KeyError: 1', as the filtered Dictionary has only one element. I suspect the cfs dictionary should be updated once the Dictionary instance dct gets filtered by filter_extremes. The documentation has this warning for the filter_extremes method, but I do not think it is relevant for this case: "After the pruning, resulting gaps in word ids are shrunk. Due to this gap shrinking, the same word may have a different word id before and after the call to this function!"

#### Versions

Please provide the output of:

```python
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)

Linux-4.15.0-50-generic-x86_64-with-LinuxMint-19-tara
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0]
NumPy 1.16.3
SciPy 0.19.1
gensim 3.8.0
FAST_VERSION 1
```



Discussion:
Thank you for reporting. Looks like a bug to me. Are you able to make a PR?
Not at the moment

Resolving patch:
diff --git a/gensim/corpora/dictionary.py b/gensim/corpora/dictionary.py
--- a/gensim/corpora/dictionary.py
+++ b/gensim/corpora/dictionary.py
@@ -472,6 +472,7 @@ def compactify(self):
         self.token2id = {token: idmap[tokenid] for token, tokenid in iteritems(self.token2id)}
         self.id2token = {}
         self.dfs = {idmap[tokenid]: freq for tokenid, freq in iteritems(self.dfs)}
+        self.cfs = {idmap[tokenid]: freq for tokenid, freq in iteritems(self.cfs)}
 
     def save_as_text(self, fname, sort_by_word=True):
         """Save :class:`~gensim.corpora.dictionary.Dictionary` to a text file.


Test patch:
diff --git a/gensim/test/test_corpora_dictionary.py b/gensim/test/test_corpora_dictionary.py
--- a/gensim/test/test_corpora_dictionary.py
+++ b/gensim/test/test_corpora_dictionary.py
@@ -124,8 +124,10 @@ def testMerge(self):
     def testFilter(self):
         d = Dictionary(self.texts)
         d.filter_extremes(no_below=2, no_above=1.0, keep_n=4)
-        expected = {0: 3, 1: 3, 2: 3, 3: 3}
-        self.assertEqual(d.dfs, expected)
+        dfs_expected = {0: 3, 1: 3, 2: 3, 3: 3}
+        cfs_expected = {0: 4, 1: 3, 2: 3, 3: 3}
+        self.assertEqual(d.dfs, dfs_expected)
+        self.assertEqual(d.cfs, cfs_expected)
 
     def testFilterKeepTokens_keepTokens(self):
         # provide keep_tokens argument, keep the tokens given