Repository: piskvorky/gensim

Issue:
IndexError loading word2vec model from previous version
I'm seeing an issue in 1.0.0 that did not occur in 0.13.4.1.  I'm simply loading the pre-trained gensim models from this site:

https://zenodo.org/record/162792

And then when I try to lookup a word it seems to be in the vocab but something weird is happening when accessing syn0.  What's very strange is that model.vocab[word] returns an index which is smaller than the first access of syn0 but I still get an index error (I have to admit I did not try to track it down in the code).  Here's some sample code and the exception:

```
# check the word 'bon' is in the model
'bon' in model:  True
# extract the index using model.vocab['bon']
index 618
# check the size of syn0
model size:  (475475, 500)
# check that we can access syn0 via model.syn0[0].size
a vector:  (500,)
# check that we can access that vector via model.syn0[618].size
a further vector:  (500,)
# yet still we get an error when we try model['bon']
Get vectors for word 'bon': 
...
    print "Get vectors for word 'bon': ",w2v_mdl['bon']
  File "/usr/local/lib/python2.7/site-packages/gensim/models/word2vec.py", line 1188, in __getitem__
    return self.wv.__getitem__(words)
  File "/usr/local/lib/python2.7/site-packages/gensim/models/keyedvectors.py", line 567, in __getitem__
    return self.word_vec(words)
  File "/usr/local/lib/python2.7/site-packages/gensim/models/keyedvectors.py", line 271, in word_vec
    return self.syn0[self.vocab[word].index]
IndexError: list index out of range
```

It should be easy to reproduce this error by just trying any french word (e.g. 'bon') for the model on that site.


Discussion:
To get a better sense of what's going on, could you please report the values of...

    model.wv.vocab['bon'].index

and

    model.wv.syn0.shape




Got the same issue (since 1.0 update) on another dictionary, testing word existance works and returns true, but getting it raises the IndexError
@mathrb Could you please report the values of `model.wv.vocab['bon'].index` and `model.wv.syn0.shape`?

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -1291,14 +1291,15 @@ def load(cls, *args, **kwargs):
         return model
 
     def _load_specials(self, *args, **kwargs):
+        super(Word2Vec, self)._load_specials(*args, **kwargs)
         # loading from a pre-KeyedVectors word2vec model
         if not hasattr(self, 'wv'):
             wv = KeyedVectors()
             wv.syn0 = self.__dict__.get('syn0', [])
+            wv.syn0norm = self.__dict__.get('syn0norm', None)
             wv.vocab = self.__dict__.get('vocab', {})
             wv.index2word = self.__dict__.get('index2word', [])
             self.wv = wv
-        super(Word2Vec, self)._load_specials(*args, **kwargs)
 
     @classmethod
     def load_word2vec_format(cls, fname, fvocab=None, binary=False, encoding='utf8', unicode_errors='strict',


Test patch:
diff --git a/gensim/test/test_data/word2vec_pre_kv_py2 b/gensim/test/test_data/word2vec_pre_kv_py2
Binary files a/gensim/test/test_data/word2vec_pre_kv_py2 and b/gensim/test/test_data/word2vec_pre_kv_py2 differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_py3 b/gensim/test/test_data/word2vec_pre_kv_py3
Binary files a/gensim/test/test_data/word2vec_pre_kv_py3 and b/gensim/test/test_data/word2vec_pre_kv_py3 differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_py3_4 b/gensim/test/test_data/word2vec_pre_kv_py3_4
Binary files a/gensim/test/test_data/word2vec_pre_kv_py3_4 and b/gensim/test/test_data/word2vec_pre_kv_py3_4 differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py2 b/gensim/test/test_data/word2vec_pre_kv_sep_py2
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py2 and b/gensim/test/test_data/word2vec_pre_kv_sep_py2 differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py2.neg_labels.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py2.neg_labels.npy
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/word2vec_pre_kv_sep_py2.neg_labels.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn0.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn0.npy
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn0.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn0_lockf.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn0_lockf.npy
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn0_lockf.npy and b/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn0_lockf.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn1neg.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn1neg.npy
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn1neg.npy and b/gensim/test/test_data/word2vec_pre_kv_sep_py2.syn1neg.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py2.wv.syn0.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py2.wv.syn0.npy
deleted file mode 100644
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py2.wv.syn0.npy and /dev/null differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3 b/gensim/test/test_data/word2vec_pre_kv_sep_py3
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py3 and b/gensim/test/test_data/word2vec_pre_kv_sep_py3 differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3.neg_labels.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3.neg_labels.npy
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/word2vec_pre_kv_sep_py3.neg_labels.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn0.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn0.npy
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn0.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn0_lockf.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn0_lockf.npy
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn0_lockf.npy and b/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn0_lockf.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn1neg.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn1neg.npy
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn1neg.npy and b/gensim/test/test_data/word2vec_pre_kv_sep_py3.syn1neg.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3.wv.syn0.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3.wv.syn0.npy
deleted file mode 100644
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py3.wv.syn0.npy and /dev/null differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4 b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4 and b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4 differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.neg_labels.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.neg_labels.npy
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.neg_labels.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn0.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn0.npy
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn0.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn0_lockf.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn0_lockf.npy
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn0_lockf.npy and b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn0_lockf.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn1neg.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn1neg.npy
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn1neg.npy and b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.syn1neg.npy differ
diff --git a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.wv.syn0.npy b/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.wv.syn0.npy
deleted file mode 100644
Binary files a/gensim/test/test_data/word2vec_pre_kv_sep_py3_4.wv.syn0.npy and /dev/null differ