Repository: piskvorky/gensim

Issue:
FastText OOV word embedding are calculated incorrectly when passing `use_norm=True` 
#### Problem description

FastText OOV word embedding are calculated incorrectly when passing `use_norm=True` (particularly, when looking for most similar words).  The library first normalizes n-gram vectors, then averages them. But it should average them first; otherwise, results are inconsistent.

What happens: cosine similarities used for neighbor retrieval are different from similarities calculated directly from word vectors. 

Why it happens:
* usually when calculating vectors for OOV words fasttext calculates average of n-gram vectors
* but if we pass `use_norm=True`, then fasttext calculates average of *L2-normalized* n-gram vectors ([code](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/keyedvectors.py#L2090)). And it is wrong!
* when we lookup for most similar words, we use just this option, `use_norm=True` ([code](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/keyedvectors.py#L831)), how unfortunate!
* why averaging normalized vectors is wrong: because it was never done when model was trained, and is normally never done when the model is applied, so such vectors are most probably meaningless.
* how to do it right: *first* average n-gram vectors, and *then* normalize them. 

#### Call to action: 
Rewrite `word_vec` method for FastTextKeyedVectors to apply normalization and averaging in the rigth order. 

#### Steps/code/corpus to reproduce
```
word = 'some_oov_word'
pairs = model.most_similar(word)
top_neighbor, top_simil = pairs[0]
print(top_simil)
print(model.cosine_similarities(model[word], model[top_neighbor].reshape(1, -1))[0])
```
these two prints are expected to produce identical numbers (similarity between the oov words and its closest neighbor), but in fact the numbers are different.

This [notebook](https://gist.github.com/avidale/c6b1d13b32a36f19750cd01148560561) reproduces the problem with a particular model for Russian language, but it is relevant for any language. 

#### Versions

The output of 
```python
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
```
is
```
Linux-4.14.137+-x86_64-with-Ubuntu-18.04-bionic
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]
NumPy 1.17.5
SciPy 1.4.1
gensim 3.8.1
FAST_VERSION 1
```




Discussion:

Resolving patch:
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -2114,10 +2114,6 @@ def word_vec(self, word, use_norm=False):
             raise KeyError('cannot calculate vector for OOV word without ngrams')
         else:
             word_vec = np.zeros(self.vectors_ngrams.shape[1], dtype=np.float32)
-            if use_norm:
-                ngram_weights = self.vectors_ngrams_norm
-            else:
-                ngram_weights = self.vectors_ngrams
             ngram_hashes = ft_ngram_hashes(word, self.min_n, self.max_n, self.bucket, self.compatible_hash)
             if len(ngram_hashes) == 0:
                 #
@@ -2131,8 +2127,11 @@ def word_vec(self, word, use_norm=False):
                 logger.warning('could not extract any ngrams from %r, returning origin vector', word)
                 return word_vec
             for nh in ngram_hashes:
-                word_vec += ngram_weights[nh]
-            return word_vec / len(ngram_hashes)
+                word_vec += self.vectors_ngrams[nh]
+            result = word_vec / len(ngram_hashes)
+            if use_norm:
+                result /= sqrt(sum(result ** 2))
+            return result
 
     def init_sims(self, replace=False):
         """Precompute L2-normalized vectors.


Test patch:
diff --git a/gensim/test/test_fasttext.py b/gensim/test/test_fasttext.py
--- a/gensim/test/test_fasttext.py
+++ b/gensim/test/test_fasttext.py
@@ -326,6 +326,15 @@ def test_load_model_non_utf8_encoding(self):
         except KeyError:
             self.fail('Unable to access vector for cp-852 word')
 
+    def test_oov_similarity(self):
+        word = 'someoovword'
+        most_similar = self.test_model.wv.most_similar(word)
+        top_neighbor, top_similarity = most_similar[0]
+        v1 = self.test_model.wv[word]
+        v2 = self.test_model.wv[top_neighbor]
+        top_similarity_direct = self.test_model.wv.cosine_similarities(v1, v2.reshape(1, -1))[0]
+        self.assertAlmostEqual(top_similarity, top_similarity_direct, places=6)
+
     def test_n_similarity(self):
         # In vocab, sanity check
         self.assertTrue(np.allclose(self.test_model.wv.n_similarity(['the', 'and'], ['and', 'the']), 1.0))