Repository: piskvorky/gensim

Issue:
fastText  fixes in 3.7 break compatibility with old models
Recent fixes to Gensim's fastText implementation introduced in #2313 are great. Unfortunately, they also break compatibility with fastText models trained by older Gensim versions - if the models are stored as a KeyedVectors() object. One can load such a model, but as soon as you try to do anything useful (like `most_similar()`, etc), it fails, because the `compatible_hash` attribute is missing.
If this attribute is added manually after the loading, everything goes fine.

#### Steps/Code/Corpus to Reproduce
```
import gensim

model = gensim.models.KeyedVectors.load(ANY_KEYED_VECTORS_FASTTEXT_MODEL)
model.most_similar(positive=ANY_WORD)
```
#### Expected Results
The `compatible_hash` attribute is automatically assigned the False value on load, and the model works as before.

#### Actual Results
```
/usr/local/lib/python3.5/dist-packages/gensim/models/keyedvectors.py in word_vec(self, word, use_norm)
   2057 
   2058         """
-> 2059         hash_fn = _ft_hash if self.compatible_hash else _ft_hash_broken
   2060 
   2061         if word in self.vocab:

AttributeError: 'FastTextKeyedVectors' object has no attribute 'compatible_hash'

```

#### Versions
Linux-4.15.0-43-generic-x86_64-with-LinuxMint-18.3-sylvia
Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609]
NumPy 1.14.5
SciPy 1.1.0
gensim 3.7.0
FAST_VERSION 1


Discussion:
@mpenkov 
@menshikh-iv 
'Full' fastText models (not KeyedVectors objects) trained in older Gensim versions can be loaded and worked with. There is even  a warning message in the logs about the hash function being buggy. Unfortunately, this message itself is buggy and fails to show properly:
```
2019-01-19 20:36:08,303 : INFO : loaded test_fasttext.model
--- Logging error ---
Traceback (most recent call last):
  File "/projects/ltg/python3/lib/python3.5/logging/__init__.py", line 986, in emit
    msg = self.format(record)
  File "/projects/ltg/python3/lib/python3.5/logging/__init__.py", line 836, in format
    return fmt.format(record)
  File "/projects/ltg/python3/lib/python3.5/logging/__init__.py", line 573, in format
    record.message = record.getMessage()
  File "/projects/ltg/python3/lib/python3.5/logging/__init__.py", line 336, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "test.py", line 6, in <module>
    model = gensim.models.fasttext.FastText.load('test_fasttext.model')
  File "/projects/ltg/python3/lib/python3.5/site-packages/gensim/models/fasttext.py", line 845, in load
    "The model will continue to work, but consider training it "                                                                                                                             
Message: 'This older model was trained with a buggy hash function.  '                                                                                                                        
Arguments: ('The model will continue to work, but consider training it from scratch.',)     
```
Hello @akutuzov,

thanks for the fast report :+1: 

About "full" model message - fix already here: #2339.
About KeyedVectors - ~~can you share any `ANY_KEYED_VECTORS_FASTTEXT_MODEL`~~ reproduced in https://github.com/RaRe-Technologies/gensim/issues/2341#issuecomment-455837449 (we'll fix it ASAP in that case, I'm not sure, but `3.7.1` can appears in next 2 weeks)

upd: @akutuzov I reproduced KV problem (no need additional info from you)

Reproduce backward compatibility bug

1. Train FT & save KV in `gensim==3.6.0`
   ```python
   from gensim.test.utils import common_texts
   from gensim.models import FastText

   m = FastText(common_texts, min_count=0)
   m.wv.save("ft_kv.model")
   ```
   produced file (gzipped after, for uploading to github): [ft_kv.model.gz](https://github.com/RaRe-Technologies/gensim/files/2776280/ft_kv.model.gz)

2. Load KV in `gensim==3.7.0` and use it
   ```python
   from gensim.models.keyedvectors import FastTextKeyedVectors, KeyedVectors

   m = KeyedVectors.load("ft_kv.model")
   m.most_similar("human")  # exception "AttributeError: 'FastTextKeyedVectors' object has no attribute 'compatible_hash'"

   m = FastTextKeyedVectors.load("ft_kv.model")
   m.most_similar("human")  # exception "AttributeError: 'FastTextKeyedVectors' object has no attribute 'compatible_hash'"

   ```

Full trace
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-340e13f11fe0> in <module>()
      2 
      3 m = KeyedVectors.load("ft_kv.model")
----> 4 m.most_similar("human")  # exception "AttributeError: 'FastTextKeyedVectors' object has no attribute 'compatible_hash'"

/home/ivan/.virtualenvs/abc_g37/local/lib/python2.7/site-packages/gensim/models/keyedvectors.pyc in most_similar(self, positive, negative, topn, restrict_vocab, indexer)
    541                 mean.append(weight * word)
    542             else:
--> 543                 mean.append(weight * self.word_vec(word, use_norm=True))
    544                 if word in self.vocab:
    545                     all_words.add(self.vocab[word].index)

/home/ivan/.virtualenvs/abc_g37/local/lib/python2.7/site-packages/gensim/models/keyedvectors.pyc in word_vec(self, word, use_norm)
   2057 
   2058         """
-> 2059         hash_fn = _ft_hash if self.compatible_hash else _ft_hash_broken
   2060 
   2061         if word in self.vocab:

AttributeError: 'FastTextKeyedVectors' object has no attribute 'compatible_hash'
```
preliminary variant of fix
CC @mpenkov 
```diff
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
index d9dad1cc..881aaf18 100644
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -1974,6 +1974,14 @@ class FastTextKeyedVectors(WordEmbeddingsKeyedVectors):
         self.num_ngram_vectors = 0
         self.compatible_hash = compatible_hash
 
+    @classmethod
+    def load(cls, fname_or_handle, **kwargs):
+        model = super(WordEmbeddingsKeyedVectors, cls).load(fname_or_handle, **kwargs)
+        if not hasattr(model, 'compatible_hash'):
+            model.compatible_hash = False
+
+        return model
+
     @property
     @deprecated("Attribute will be removed in 4.0.0, use self.vectors_vocab instead")
     def syn0_vocab(self):
@@ -2012,7 +2020,7 @@ class FastTextKeyedVectors(WordEmbeddingsKeyedVectors):
         if word in self.vocab:
             return True
         else:
-            hash_fn = _ft_hash if self.compatible_hash else _ft_hash_broken
+            hash_fn = _ft_hash if getattr(self, "compatible_hash", False) else _ft_hash_broken
             char_ngrams = _compute_ngrams(word, self.min_n, self.max_n)
             return any(hash_fn(ng) % self.bucket in self.hash2index for ng in char_ngrams)
 
@@ -2056,7 +2064,7 @@ class FastTextKeyedVectors(WordEmbeddingsKeyedVectors):
             If word and all ngrams not in vocabulary.
 
         """
-        hash_fn = _ft_hash if self.compatible_hash else _ft_hash_broken
+        hash_fn = _ft_hash if getattr(self, "compatible_hash", False) else _ft_hash_broken
 
         if word in self.vocab:
             return super(FastTextKeyedVectors, self).word_vec(word, use_norm)
@@ -2237,7 +2245,7 @@ class FastTextKeyedVectors(WordEmbeddingsKeyedVectors):
         if self.bucket == 0:
             return
 
-        hash_fn = _ft_hash if self.compatible_hash else _ft_hash_broken
+        hash_fn = _ft_hash if getattr(self, "compatible_hash", False) else _ft_hash_broken
 
         for w, v in self.vocab.items():
             word_vec = np.copy(self.vectors_vocab[v.index])
```
Fixed partially (https://github.com/RaRe-Technologies/gensim/issues/2341#issuecomment-455809275) in #2339 
Waiting #2340 for full fix

Resolving patch:
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -1512,6 +1512,15 @@ def get_keras_embedding(self, train_embeddings=False):
         )
         return layer
 
+    @classmethod
+    def load(cls, fname_or_handle, **kwargs):
+        model = super(WordEmbeddingsKeyedVectors, cls).load(fname_or_handle, **kwargs)
+        if isinstance(model, FastTextKeyedVectors):
+            if not hasattr(model, 'compatible_hash'):
+                model.compatible_hash = False
+
+        return model
+
 
 KeyedVectors = Word2VecKeyedVectors  # alias for backward compatibility
 
@@ -1974,6 +1983,14 @@ def __init__(self, vector_size, min_n, max_n, bucket, compatible_hash):
         self.num_ngram_vectors = 0
         self.compatible_hash = compatible_hash
 
+    @classmethod
+    def load(cls, fname_or_handle, **kwargs):
+        model = super(WordEmbeddingsKeyedVectors, cls).load(fname_or_handle, **kwargs)
+        if not hasattr(model, 'compatible_hash'):
+            model.compatible_hash = False
+
+        return model
+
     @property
     @deprecated("Attribute will be removed in 4.0.0, use self.vectors_vocab instead")
     def syn0_vocab(self):


Test patch:
diff --git a/gensim/test/test_data/ft_kv_3.6.0.model.gz b/gensim/test/test_data/ft_kv_3.6.0.model.gz
new file mode 100644
Binary files /dev/null and b/gensim/test/test_data/ft_kv_3.6.0.model.gz differ
diff --git a/gensim/test/test_keyedvectors.py b/gensim/test/test_keyedvectors.py
--- a/gensim/test/test_keyedvectors.py
+++ b/gensim/test/test_keyedvectors.py
@@ -15,7 +15,8 @@
 import numpy as np
 
 from gensim.corpora import Dictionary
-from gensim.models import KeyedVectors as EuclideanKeyedVectors, WordEmbeddingSimilarityIndex
+from gensim.models.keyedvectors import KeyedVectors as EuclideanKeyedVectors, WordEmbeddingSimilarityIndex, \
+    FastTextKeyedVectors
 from gensim.test.utils import datapath
 
 import gensim.models.keyedvectors
@@ -279,6 +280,19 @@ def test_set_item(self):
         for ent, vector in zip(entities, vectors):
             self.assertTrue(np.allclose(self.vectors[ent], vector))
 
+    def test_ft_kv_backward_compat_w_360(self):
+        kv = EuclideanKeyedVectors.load(datapath("ft_kv_3.6.0.model.gz"))
+        ft_kv = FastTextKeyedVectors.load(datapath("ft_kv_3.6.0.model.gz"))
+
+        expected = ['trees', 'survey', 'system', 'graph', 'interface']
+        actual = [word for (word, similarity) in kv.most_similar("human", topn=5)]
+
+        self.assertEqual(actual, expected)
+
+        actual = [word for (word, similarity) in ft_kv.most_similar("human", topn=5)]
+
+        self.assertEqual(actual, expected)
+
 
 class L2NormTest(unittest.TestCase):
     def test(self):