Repository: piskvorky/gensim

Issue:
_rollback_optimization is not performed on some old FastText models
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

Model `araneum_none_fasttextskipgram_300_5_2018` from https://rusvectores.org/ru/models/
is loading incorrectly with `gensim.models.fasttext.FastText.load`
This method creates model with `compatible_hash = True` and `hash2index` param, so that ngram indexes are computed incorrectly. 

#### Steps/code/corpus to reproduce

```python

from gensim.models.fasttext import FastText

ft2 = FastText.load('./data/araneum_none_fasttextskipgram_300_5_2018.model')
ft2.wv.word_vec('слово123')
```

gives the following stacktrace

```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-14-5fac1e42e24a> in <module>()
----> 1 ft2.wv.word_vec('слово123')

~/anaconda3/lib/python3.6/site-packages/gensim/models/keyedvectors.py in word_vec(self, word, use_norm)
   2111                 return word_vec
   2112             for nh in ngram_hashes:
-> 2113                 word_vec += ngram_weights[nh]
   2114             return word_vec / len(ngram_hashes)
   2115 

IndexError: index 1369364 is out of bounds for axis 0 with size 27355
```
#### Versions

```
Linux-4.2.0-27-generic-x86_64-with-debian-jessie-sid
Python 3.6.8 |Anaconda 4.4.0 (64-bit)| (default, Dec 30 2018, 01:22:34) 
[GCC 7.3.0]
NumPy 1.16.0
SciPy 1.1.0
gensim 3.7.2
FAST_VERSION 1
```

_rollback_optimization is not performed on some old FastText models
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

Model `araneum_none_fasttextskipgram_300_5_2018` from https://rusvectores.org/ru/models/
is loading incorrectly with `gensim.models.fasttext.FastText.load`
This method creates model with `compatible_hash = True` and `hash2index` param, so that ngram indexes are computed incorrectly. 

#### Steps/code/corpus to reproduce

```python

from gensim.models.fasttext import FastText

ft2 = FastText.load('./data/araneum_none_fasttextskipgram_300_5_2018.model')
ft2.wv.word_vec('слово123')
```

gives the following stacktrace

```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-14-5fac1e42e24a> in <module>()
----> 1 ft2.wv.word_vec('слово123')

~/anaconda3/lib/python3.6/site-packages/gensim/models/keyedvectors.py in word_vec(self, word, use_norm)
   2111                 return word_vec
   2112             for nh in ngram_hashes:
-> 2113                 word_vec += ngram_weights[nh]
   2114             return word_vec / len(ngram_hashes)
   2115 

IndexError: index 1369364 is out of bounds for axis 0 with size 27355
```
#### Versions

```
Linux-4.2.0-27-generic-x86_64-with-debian-jessie-sid
Python 3.6.8 |Anaconda 4.4.0 (64-bit)| (default, Dec 30 2018, 01:22:34) 
[GCC 7.3.0]
NumPy 1.16.0
SciPy 1.1.0
gensim 3.7.2
FAST_VERSION 1
```



Discussion:
Possible workaround is to manually apply this changes on loaded model:

```python
from gensim.models.keyedvectors import _rollback_optimization

ft2.wv.compatible_hash = False
_rollback_optimization(ft2.wv)

```
Good catch. We'll add logic to handle such models.
Possible workaround is to manually apply this changes on loaded model:

```python
from gensim.models.keyedvectors import _rollback_optimization

ft2.wv.compatible_hash = False
_rollback_optimization(ft2.wv)

```
Good catch. We'll add logic to handle such models.

Resolving patch:
diff --git a/gensim/models/fasttext.py b/gensim/models/fasttext.py
--- a/gensim/models/fasttext.py
+++ b/gensim/models/fasttext.py
@@ -1023,30 +1023,22 @@ def load(cls, *args, **kwargs):
         """
         try:
             model = super(FastText, cls).load(*args, **kwargs)
-            if hasattr(model.wv, 'hash2index'):
-                gensim.models.keyedvectors._rollback_optimization(model.wv)
 
             if not hasattr(model.trainables, 'vectors_vocab_lockf') and hasattr(model.wv, 'vectors_vocab'):
                 model.trainables.vectors_vocab_lockf = ones(model.wv.vectors_vocab.shape, dtype=REAL)
             if not hasattr(model.trainables, 'vectors_ngrams_lockf') and hasattr(model.wv, 'vectors_ngrams'):
                 model.trainables.vectors_ngrams_lockf = ones(model.wv.vectors_ngrams.shape, dtype=REAL)
 
-            if not hasattr(model.wv, 'compatible_hash'):
-                logger.warning(
-                    "This older model was trained with a buggy hash function. "
-                    "The model will continue to work, but consider training it "
-                    "from scratch."
-                )
-                model.wv.compatible_hash = False
-
             if not hasattr(model.wv, 'bucket'):
                 model.wv.bucket = model.trainables.bucket
-
-            return model
         except AttributeError:
             logger.info('Model saved using code from earlier Gensim Version. Re-loading old model in a compatible way.')
             from gensim.models.deprecated.fasttext import load_old_fasttext
-            return load_old_fasttext(*args, **kwargs)
+            model = load_old_fasttext(*args, **kwargs)
+
+        gensim.models.keyedvectors._try_upgrade(model.wv)
+
+        return model
 
     @deprecated("Method will be removed in 4.0.0, use self.wv.accuracy() instead")
     def accuracy(self, questions, restrict_vocab=30000, most_similar=None, case_insensitive=True):
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -1981,12 +1981,7 @@ def __init__(self, vector_size, min_n, max_n, bucket, compatible_hash):
     @classmethod
     def load(cls, fname_or_handle, **kwargs):
         model = super(WordEmbeddingsKeyedVectors, cls).load(fname_or_handle, **kwargs)
-        if not hasattr(model, 'compatible_hash'):
-            model.compatible_hash = False
-
-        if hasattr(model, 'hash2index'):
-            _rollback_optimization(model)
-
+        _try_upgrade(model)
         return model
 
     @property
@@ -2478,3 +2473,16 @@ def _unpack(m, num_rows, hash2index, seed=1):
         m[[h, i]] = m[[i, h]]  # swap rows i and h
 
     return m
+
+
+def _try_upgrade(wv):
+    if hasattr(wv, 'hash2index'):
+        _rollback_optimization(wv)
+
+    if not hasattr(wv, 'compatible_hash'):
+        logger.warning(
+            "This older model was trained with a buggy hash function. "
+            "The model will continue to work, but consider training it "
+            "from scratch."
+        )
+        wv.compatible_hash = False


Test patch:
diff --git a/gensim/test/test_fasttext.py b/gensim/test/test_fasttext.py
--- a/gensim/test/test_fasttext.py
+++ b/gensim/test/test_fasttext.py
@@ -780,9 +780,8 @@ def testLoadOldModel(self):
         self.assertTrue(model.trainables.vectors_lockf.shape == (12, ))
         self.assertTrue(model.vocabulary.cum_table.shape == (12, ))
 
-        self.assertEqual(len(model.wv.hash2index), 202)
-        self.assertTrue(model.wv.vectors_vocab.shape == (12, 100))
-        self.assertTrue(model.wv.vectors_ngrams.shape == (202, 100))
+        self.assertEqual(model.wv.vectors_vocab.shape, (12, 100))
+        self.assertEqual(model.wv.vectors_ngrams.shape, (2000000, 100))
 
         # Model stored in multiple files
         model_file = 'fasttext_old_sep'
@@ -795,9 +794,8 @@ def testLoadOldModel(self):
         self.assertTrue(model.trainables.vectors_lockf.shape == (12, ))
         self.assertTrue(model.vocabulary.cum_table.shape == (12, ))
 
-        self.assertEqual(len(model.wv.hash2index), 202)
-        self.assertTrue(model.wv.vectors_vocab.shape == (12, 100))
-        self.assertTrue(model.wv.vectors_ngrams.shape == (202, 100))
+        self.assertEqual(model.wv.vectors_vocab.shape, (12, 100))
+        self.assertEqual(model.wv.vectors_ngrams.shape, (2000000, 100))
 
     def compare_with_wrapper(self, model_gensim, model_wrapper):
         # make sure we get >=2 overlapping words for top-10 similar words suggested for `night`