Repository: piskvorky/gensim

Issue:
Can not build similarity matrix when the dictionary contains word indices are not continuous
<!--
If your issue is a usage or a general question, please submit it here instead:
- Mailing List: https://groups.google.com/forum/#!forum/gensim
For more information, see Recipes&FAQ: https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ
-->

<!-- Instructions For Filing a Bug: https://github.com/RaRe-Technologies/gensim/blob/develop/CONTRIBUTING.md -->

#### Description
For some reasons, the gensim.corpora.Dictionary contains word indices are not continuous. I met the `KeyError` in _similarity_matrix()_ function when it tried to re-index from 0 to n-1. 

<!-- Example: Vocabulary size is not what I expected when training Word2Vec. -->

#### Steps/Code/Corpus to Reproduce
```
import gensim.downloader as api

def get_similarity_matrix(dictionary):
    '''
    In some cases, the dictionary contains word indices are not continuous.
    For example, the dictionary contains theses items: (1, u'car'), (2, u'so'), (4, 'nice')
    '''
    w2v_model = api.load('glove-wiki-gigaword-50', return_path=False)
    return w2v_model.wv.similarity_matrix(dictionary, nonzero_limit=100)
```
<!--
Example:
```
from gensim.models import word2vec

sentences = ['human', 'machine']
model = word2vec.Word2Vec(sentences)
print(model.syn0.shape) 
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
Term similarity matrix in `scipy.sparse.csc_matrix` format
<!-- Example: Expected shape of (100,2).-->

#### Actual Results
```
    return w2v_model.wv.similarity_matrix(dictionary, nonzero_limit=100)
  File "/home/jen/anaconda2/lib/python2.7/site-packages/gensim/models/keyedvectors.py", line 510, in similarity_matrix
    w1 = dictionary[w1_index]
  File "/home/jen/anaconda2/lib/python2.7/site-packages/gensim/corpora/dictionary.py", line 104, in __getitem__
    return self.id2token[tokenid]  # will throw for non-existent ids
KeyError: 0
ERROR: Non-zero return code '1' from command: Process exited with status 1
```
<!-- Example: Actual shape of (100,5). 

Please paste or specifically describe the actual output or traceback. -->

#### Versions
('Python', '2.7.14 |Anaconda custom (64-bit)| (default, Mar 27 2018, 17:29:31) \n[GCC 7.2.0]')
('NumPy', '1.14.2')
('SciPy', '1.0.1')
('gensim', '3.4.0')
('FAST_VERSION', 1)

<!--
Please run the following snippet and paste the output below.
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
-->


<!-- Thanks for contributing! -->




Discussion:
Have same issue.

In my current system have already trained a vocabulary (word-index, word-term) the word-index is not a continuous series (1,2,3,4,5, ...) but it is (1,4,5,...) randomly , So what we want is: try to convert this vocab to the same format dictionary of gensim and get this error.

Thanks
I could be completely wrong but this is an idea that comes to mind:

Words in the dictionary are indeed assigned continuous IDs, but some of them are filtered out by the default filters (i.e their occurence count is lower than the threshold). Could you check if the issue persists when disabling the filters? 
Looks like a bug to me. There is no expectation that the Dictionary id range must be contiguous.

CC @Witiko `similarity_matrix` seems to be your code -- can you please fix + add a test?

In the meanwhile: @quandb @tanthml can you call `.compactify()` on your dictionary, to remove the gaps? Why does your dictionary contain gaps in the first place?
@piskvorky All my contributed code assumes that dictionaries are contiguous. I discussed this with @menshikh-iv to confirm that this was a correct assumption. The response was that in the current version of Gensim, dictionaries were automatically compactified on construction and that this assumption was therefore valid. I assume that there are still ways to obtain an uncompactified dictionary, then?
@Witiko Yes, there are. For example `filter_extremes()` (the most common way to filter a Dictionary) will call `compactify()` automatically. But if the user implements their own filtering, it is a quite possible there will be gaps.

There is no API contract that the ids must be contiguous (though they typically are). Let's wait for @quandb and @tanthml 's reponse regarding their use case for having gaps.
Hi all, 

In summary: there is another app perform the mapping between word and id, some tokens have been removed due to some reasons. Then I have to build the Gensim `dictionary` from processed vocab (have word and id already).

In detail: the use case here is we build a product that includes multiple independent apps such as data ingestion, pre-processing, vectorization, topic modeling, etc.
The task i'm working on is to build a `similarity_matrix`. And the `similarity_matrix` function have to consume a dictionary. The thing here's we have to keep the token and the `token_id` consistent across the system. we can't build the dictionary from original corpus since we have another mapper did that.
Then I build a dictionary by refactor the `Dictionary.load_from_text()` function to load the vocabulary follow our format. We can't even use the `compactify()` due to the token_id must consistent for other apps.

Let me know if you need more information,

Resolving patch:
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -58,6 +58,7 @@
 """
 from __future__ import division  # py3 "true division"
 
+from collections import deque
 import logging
 
 try:
@@ -554,16 +555,17 @@ def similarity_matrix(self, dictionary, tfidf=None, threshold=0.0, exponent=2.0,
         num_skipped = 0
         # Decide the order of rows.
         if tfidf is None:
-            word_indices = range(matrix_order)
+            word_indices = deque(sorted(dictionary.keys()))
         else:
             assert max(tfidf.idfs) < matrix_order
-            word_indices = [
+            word_indices = deque([
                 index for index, _
                 in sorted(tfidf.idfs.items(), key=lambda x: (x[1], -x[0]), reverse=True)
-            ]
+            ])
 
         # Traverse rows.
-        for row_number, w1_index in enumerate(word_indices):
+        for row_number, w1_index in enumerate(list(word_indices)):
+            word_indices.popleft()
             if row_number % 1000 == 0:
                 logger.info(
                     "PROGRESS: at %.02f%% rows (%d / %d, %d skipped, %.06f%% density)",
@@ -578,8 +580,8 @@ def similarity_matrix(self, dictionary, tfidf=None, threshold=0.0, exponent=2.0,
             if matrix_order <= nonzero_limit + 1:  # Traverse all columns.
                 columns = (
                     (w2_index, self.similarity(w1, dictionary[w2_index]))
-                    for w2_index in range(w1_index + 1, matrix_order)
-                    if w1_index != w2_index and dictionary[w2_index] in self.vocab)
+                    for w2_index in word_indices
+                    if dictionary[w2_index] in self.vocab)
             else:  # Traverse only columns corresponding to the embeddings closest to w1.
                 num_nonzero = matrix_nonzero[w1_index] - 1
                 columns = (


Test patch: