Repository: piskvorky/gensim

Issue:
Doc2Vec.infer_vector learning rate decays extremely fast (non-linearly)
I am working with a corpus of very short documents and noticed that the inferred vectors for the same document were very different.

```python
from scipy.spatial.distance import pdist, squareform
testdoc = "This is a small sample document."
vectors = [d2vmod.infer_vector(testdoc) for _ in range(5)]
squareform(pdist(vectors, "cosine"))
```

```
array([[0.        , 0.05987812, 0.06183155, 0.06931093, 0.05466599],
       [0.05987812, 0.        , 0.03724874, 0.05006329, 0.04789369],
       [0.06183155, 0.03724874, 0.        , 0.04771786, 0.05983109],
       [0.06931093, 0.05006329, 0.04771786, 0.        , 0.0367826 ],
       [0.05466599, 0.04789369, 0.05983109, 0.0367826 , 0.        ]])
```

More training steps makes things worse in this case:
```python
vectors = [d2vmod.infer_vector(testdoc, 10000) for _ in range(5)]
squareform(pdist(vectors, "cosine"))
```
```
array([[0.        , 0.27392197, 0.308742  , 0.51374501, 0.45744246],
       [0.27392197, 0.        , 0.14912033, 0.32902151, 0.1822687 ],
       [0.308742  , 0.14912033, 0.        , 0.2895444 , 0.27019636],
       [0.51374501, 0.32902151, 0.2895444 , 0.        , 0.38096254],
       [0.45744246, 0.1822687 , 0.27019636, 0.38096254, 0.        ]])
```
Note: This is more extreme than what I'm seeing with more domain-specific sample documents, where start to get more consistent after about 5000 steps.

I believe this is happening because the learning rate decays extremely rapidly:
 https://github.com/RaRe-Technologies/gensim/blob/8b810918d59781116794a6679999afdc76b857ef/gensim/models/doc2vec.py#L565

```python
alpha = 0.025
min_alpha = 0.001
steps = 100
for i in range(steps):
    print(alpha)
    alpha = ((alpha - min_alpha) / (steps - i)) + min_alpha
```
```
0.025
0.00124
0.0010024242424242424
0.0010000247371675943
...
```
Notice that `alpha` is very close to `min_alpha` after the first step and this is exaggerated even more when the number of steps is larger. 

When I change Doc2Vec to have a linear decay in learning rate
```python
alpha_delta = (alpha-min_alpha)/(steps-1)
for i in range(steps):
    # ...
    alpha -= alpha_delta
```

I get much better results. With 20 steps, we get pairwise cosine distances of
```
array([[0.        , 0.01617053, 0.02467067, 0.01828433, 0.01834735],
       [0.01617053, 0.        , 0.01879757, 0.00910884, 0.01358116],
       [0.02467067, 0.01879757, 0.        , 0.01521225, 0.01392789],
       [0.01828433, 0.00910884, 0.01521225, 0.        , 0.01121792],
       [0.01834735, 0.01358116, 0.01392789, 0.01121792, 0.        ]])
```
, with 100 we get
```
array([[0.        , 0.00282428, 0.00373375, 0.00331408, 0.00362875],
       [0.00282428, 0.        , 0.0036147 , 0.0028999 , 0.00210812],
       [0.00373375, 0.0036147 , 0.        , 0.0032986 , 0.00361321],
       [0.00331408, 0.0028999 , 0.0032986 , 0.        , 0.00318849],
       [0.00362875, 0.00210812, 0.00361321, 0.00318849, 0.        ]])
```
, and with 1000 steps:
```
array([[0.        , 0.00055459, 0.000633  , 0.00074271, 0.00036596],
       [0.00055459, 0.        , 0.00067211, 0.00075522, 0.00058975],
       [0.000633  , 0.00067211, 0.        , 0.00109709, 0.00049239],
       [0.00074271, 0.00075522, 0.00109709, 0.        , 0.00072527],
       [0.00036596, 0.00058975, 0.00049239, 0.00072527, 0.        ]])
```


Discussion:

Resolving patch:
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -546,6 +546,8 @@ def infer_vector(self, doc_words, alpha=0.1, min_alpha=0.0001, steps=5):
         if not self.sg:
             neu1 = matutils.zeros_aligned(self.trainables.layer1_size, dtype=REAL)
 
+        alpha_delta = (alpha - min_alpha) / (steps - 1)
+
         for i in range(steps):
             if self.sg:
                 train_document_dbow(
@@ -562,7 +564,7 @@ def infer_vector(self, doc_words, alpha=0.1, min_alpha=0.0001, steps=5):
                     self, doc_words, doctag_indexes, alpha, work, neu1,
                     learn_words=False, learn_hidden=False, doctag_vectors=doctag_vectors, doctag_locks=doctag_locks
                 )
-            alpha = ((alpha - min_alpha) / (steps - i)) + min_alpha
+            alpha -= alpha_delta
 
         return doctag_vectors[0]
 


Test patch: