Repository: piskvorky/gensim

Issue:
Enable to master the window size variations during training
**Context:** I am trying to use `gensim` on a corpus in which the order of words has no importance at all. So my idea is to give the constructor of the class `Word2Vec` a large window size, so that it trains each word with the full rest of the sentence (the iterable I give as parameter `sentences` to the `Word2Vec` constructor yields sentences in which word order is irrelevant).

I noticed [here](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/word2vec_inner.pyx#L572) in the code that during training, the window size was reduced of a size drawn uniformly at random. If I get it right, it feels like saying, if words are too far from one another, sometimes they carry less sense together. And at any rate, adding some randomness very often gives better results in machine learning in general. In general I agree with these two ideas.
- However, I find it too bad that this is hard-coded, and that the user cannot choose to keep the window fix-sized, with the size they enter. 
-  Another cool option would be to be able to input a minimal and a maximal size, instead of the current behaviour where the window size varies from the full length provided by the user to 0.

**Propositions:**
 From these observations I propose to do one or all of the following changes:
- Add a parameter to choose whether or not the window size randomly reduces. This could be something like `change_window_size`, set to `True` by default to be consistent with the current behaviour. 
- Add parameters for lower and higher bounds of the final window size. Note that if these bounds are equal, this amounts to no random reduction at all and this replaces the behaviour of the previous proposition.
- [Bonus] We could also imagine that instead of drawing sizes uniformly at random, the user could choose the probability distribution. But this is more of an idea I'm throwing into the air, rather than one I'm ready to implement right now. I don't know if it is relevant enough, considering the code changes it may imply.

What do you think of this ? If you're ok with it, I plan on doing the modifications myself.


Discussion:
This request comes up from time-to-time, or other variants for customizing the window behavior - assymetric or 1-way windows (#1600, #2172), or other influence/sampling weights per position (#2114 or the learned-weighting of @Witiko in #2840).

As you've noted, setting a very-large window, where nearly every random draw will be larger than your actual texts, should approximate a 'whole text' window fairly well. (For example, given the 10,000-token implementation limit, setting `window` to `2^30` would mean less than 1-in-a-million training windows are ever smaller than 10,)000.

Still, I'd welcome a way to make this more configurable, given how often it's requested, if it was of minimial API/performance impact. (For people who want their whole window used each time, the generation of the random windows is an unnecessary calculation - for exaple, at https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/word2vec_inner.pyx#L573 & https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/word2vec_inner.pyx#L669). WDYT, @piskvorky ?

In keeping with the existing naming of the behavior in the source code, I'd name a flag to turn this on/off as `reduced_windows`, default `True` (as traditional in word2vec/etc). 

While the idea of a min/max window would achieve the goal, I'm not sure there are any needs for a higher min other than min=max, and such flexibility might make the code less clear for a wholly-theoretically need. Similarly, other weightings/distribution options could be considered if there were experimental results that they offer big wins, but otherwise, the non-standard complication would be hard to justify. 




It makes sense to me, but I worry the (optimized) code paths are already convoluted enough. Adding another param would need to be done with care – easy to miss some code paths, apply it inconsistently across the already existing param combinations: `corpus_file`, `skip gram`, `negative sampling`, inference… at least we got rid of the Python code paths already!

And of course, a PR will need a final benchmark too. Users rely on Gensim being super fast, so we don't want any noticable performance regressions for the common case, just to add support for a new marginal one.

FWIW, the reason the `reduced_windows` values are done in a batch here is that in an earlier version, of generating a reduced window each loop through, my profiling showed a lot of time doing that - a batch per text gave a noticeable speedup. There's likely potential for further optimization here, perhaps via an even-faster PRNG or reuse of some rarely-refreshed buffer of values, etc - that's still good enough for the totally-non-adversarial variety needed here.

Resolving patch:
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -158,7 +158,7 @@ def count(self, new_val):
 class Doc2Vec(Word2Vec):
     def __init__(self, documents=None, corpus_file=None, vector_size=100, dm_mean=None, dm=1, dbow_words=0, dm_concat=0,
                  dm_tag_count=1, dv=None, dv_mapfile=None, comment=None, trim_rule=None, callbacks=(),
-                 window=5, epochs=10, **kwargs):
+                 window=5, epochs=10, shrink_windows=True, **kwargs):
         """Class for training, using and evaluating neural networks described in
         `Distributed Representations of Sentences and Documents <http://arxiv.org/abs/1405.4053v2>`_.
 
@@ -248,6 +248,12 @@ def __init__(self, documents=None, corpus_file=None, vector_size=100, dm_mean=No
 
         callbacks : :obj: `list` of :obj: `~gensim.models.callbacks.CallbackAny2Vec`, optional
             List of callbacks that need to be executed/run at specific stages during training.
+        shrink_windows : bool, optional
+            New in 4.1. Experimental.
+            If True, the effective window size is uniformly sampled from  [1, `window`]
+            for each target word during training, to match the original word2vec algorithm's
+            approximate weighting of context words by distance. Otherwise, the effective
+            window size is always fixed to `window` words to either side.
 
         Some important internal attributes are the following:
 
@@ -294,6 +300,7 @@ def __init__(self, documents=None, corpus_file=None, vector_size=100, dm_mean=No
             callbacks=callbacks,
             window=window,
             epochs=epochs,
+            shrink_windows=shrink_windows,
             **kwargs,
         )
 
diff --git a/gensim/models/fasttext.py b/gensim/models/fasttext.py
--- a/gensim/models/fasttext.py
+++ b/gensim/models/fasttext.py
@@ -276,7 +276,7 @@ def __init__(self, sentences=None, corpus_file=None, sg=0, hs=0, vector_size=100
                  max_vocab_size=None, word_ngrams=1, sample=1e-3, seed=1, workers=3, min_alpha=0.0001,
                  negative=5, ns_exponent=0.75, cbow_mean=1, hashfxn=hash, epochs=5, null_word=0, min_n=3, max_n=6,
                  sorted_vocab=1, bucket=2000000, trim_rule=None, batch_words=MAX_WORDS_IN_BATCH, callbacks=(),
-                 max_final_vocab=None):
+                 max_final_vocab=None, shrink_windows=True,):
         """Train, use and evaluate word representations learned using the method
         described in `Enriching Word Vectors with Subword Information <https://arxiv.org/abs/1607.04606>`_,
         aka FastText.
@@ -385,6 +385,12 @@ def __init__(self, sentences=None, corpus_file=None, sg=0, hs=0, vector_size=100
             ``min_count```.  If the specified ``min_count`` is more than the
             automatically calculated ``min_count``, the former will be used.
             Set to ``None`` if not required.
+        shrink_windows : bool, optional
+            New in 4.1. Experimental.
+            If True, the effective window size is uniformly sampled from  [1, `window`]
+            for each target word during training, to match the original word2vec algorithm's
+            approximate weighting of context words by distance. Otherwise, the effective
+            window size is always fixed to `window` words to either side.
 
         Examples
         --------
@@ -432,7 +438,8 @@ def __init__(self, sentences=None, corpus_file=None, sg=0, hs=0, vector_size=100
             max_vocab_size=max_vocab_size, max_final_vocab=max_final_vocab,
             min_count=min_count, sample=sample, sorted_vocab=sorted_vocab,
             null_word=null_word, ns_exponent=ns_exponent, hashfxn=hashfxn,
-            seed=seed, hs=hs, negative=negative, cbow_mean=cbow_mean, min_alpha=min_alpha)
+            seed=seed, hs=hs, negative=negative, cbow_mean=cbow_mean,
+            min_alpha=min_alpha, shrink_windows=shrink_windows)
 
     def _init_post_load(self, hidden_output):
         num_vectors = len(self.wv.vectors)
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -240,7 +240,7 @@ def __init__(
             max_vocab_size=None, sample=1e-3, seed=1, workers=3, min_alpha=0.0001,
             sg=0, hs=0, negative=5, ns_exponent=0.75, cbow_mean=1, hashfxn=hash, epochs=5, null_word=0,
             trim_rule=None, sorted_vocab=1, batch_words=MAX_WORDS_IN_BATCH, compute_loss=False, callbacks=(),
-            comment=None, max_final_vocab=None,
+            comment=None, max_final_vocab=None, shrink_windows=True,
         ):
         """Train, use and evaluate neural networks described in https://code.google.com/p/word2vec/.
 
@@ -345,6 +345,12 @@ def __init__(
             :meth:`~gensim.models.word2vec.Word2Vec.get_latest_training_loss`.
         callbacks : iterable of :class:`~gensim.models.callbacks.CallbackAny2Vec`, optional
             Sequence of callbacks to be executed at specific stages during training.
+        shrink_windows : bool, optional
+            New in 4.1. Experimental.
+            If True, the effective window size is uniformly sampled from  [1, `window`]
+            for each target word during training, to match the original word2vec algorithm's
+            approximate weighting of context words by distance. Otherwise, the effective
+            window size is always fixed to `window` words to either side.
 
         Examples
         --------
@@ -377,6 +383,7 @@ def __init__(
         self.min_alpha = float(min_alpha)
 
         self.window = int(window)
+        self.shrink_windows = bool(shrink_windows)
         self.random = np.random.RandomState(seed)
 
         self.hs = int(hs)
@@ -910,12 +917,12 @@ def _do_train_epoch(
         if self.sg:
             examples, tally, raw_tally = train_epoch_sg(
                 self, corpus_file, offset, cython_vocab, cur_epoch,
-                total_examples, total_words, work, neu1, self.compute_loss,
+                total_examples, total_words, work, neu1, self.compute_loss
             )
         else:
             examples, tally, raw_tally = train_epoch_cbow(
                 self, corpus_file, offset, cython_vocab, cur_epoch,
-                total_examples, total_words, work, neu1, self.compute_loss,
+                total_examples, total_words, work, neu1, self.compute_loss
             )
 
         return examples, tally, raw_tally
@@ -1039,7 +1046,7 @@ def train(
             msg=(
                 f"training model with {self.workers} workers on {len(self.wv)} vocabulary and "
                 f"{self.layer1_size} features, using sg={self.sg} hs={self.hs} sample={self.sample} "
-                f"negative={self.negative} window={self.window}"
+                f"negative={self.negative} window={self.window} shrink_windows={self.shrink_windows}"
             ),
         )
 
@@ -1970,6 +1977,8 @@ def _load_specials(self, *args, **kwargs):
                 self.syn1 = self.syn1
                 del self.syn1
             del self.trainables
+        if not hasattr(self, 'shrink_windows'):
+            self.shrink_windows = True
 
     def get_latest_training_loss(self):
         """Get current value of the training loss.


Test patch:
diff --git a/gensim/test/test_doc2vec.py b/gensim/test/test_doc2vec.py
--- a/gensim/test/test_doc2vec.py
+++ b/gensim/test/test_doc2vec.py
@@ -589,6 +589,44 @@ def test_dmc_neg_fromfile(self):
             )
             self.model_sanity(model)
 
+    def test_dmm_fixedwindowsize(self):
+        """Test DMM doc2vec training with fixed window size."""
+        model = doc2vec.Doc2Vec(
+            list_corpus, vector_size=24,
+            dm=1, dm_mean=1, window=4, shrink_windows=False,
+            hs=0, negative=10, alpha=0.05, min_count=2, epochs=20
+        )
+        self.model_sanity(model)
+
+    def test_dmm_fixedwindowsize_fromfile(self):
+        """Test DMM doc2vec training with fixed window size, from file."""
+        with temporary_file(get_tmpfile('gensim_doc2vec.tst')) as corpus_file:
+            save_lee_corpus_as_line_sentence(corpus_file)
+            model = doc2vec.Doc2Vec(
+                corpus_file=corpus_file, vector_size=24,
+                dm=1, dm_mean=1, window=4, shrink_windows=False,
+                hs=0, negative=10, alpha=0.05, min_count=2, epochs=20
+            )
+            self.model_sanity(model)
+
+    def test_dbow_fixedwindowsize(self):
+        """Test DBOW doc2vec training with fixed window size."""
+        model = doc2vec.Doc2Vec(
+            list_corpus, vector_size=16, shrink_windows=False,
+            dm=0, hs=0, negative=5, min_count=2, epochs=20
+        )
+        self.model_sanity(model)
+
+    def test_dbow_fixedwindowsize_fromfile(self):
+        """Test DBOW doc2vec training with fixed window size, from file."""
+        with temporary_file(get_tmpfile('gensim_doc2vec.tst')) as corpus_file:
+            save_lee_corpus_as_line_sentence(corpus_file)
+            model = doc2vec.Doc2Vec(
+                corpus_file=corpus_file, vector_size=16, shrink_windows=False,
+                dm=0, hs=0, negative=5, min_count=2, epochs=20
+            )
+            self.model_sanity(model)
+
     def test_parallel(self):
         """Test doc2vec parallel training with more than default 3 threads."""
         # repeat the ~300 doc (~60000 word) Lee corpus to get 6000 docs (~1.2M words)
diff --git a/gensim/test/test_fasttext.py b/gensim/test/test_fasttext.py
--- a/gensim/test/test_fasttext.py
+++ b/gensim/test/test_fasttext.py
@@ -397,12 +397,12 @@ def test_wm_distance(self):
         dist = self.test_model.wv.wmdistance(doc, oov_doc)
         self.assertNotEqual(float('inf'), dist)
 
-    def test_cbow_hs_training(self):
+    def test_cbow_hs_training(self, shrink_windows=True):
 
         model_gensim = FT_gensim(
             vector_size=48, sg=0, cbow_mean=1, alpha=0.05, window=5, hs=1, negative=0,
             min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
-            sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET)
+            sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET, shrink_windows=shrink_windows)
 
         lee_data = LineSentence(datapath('lee_background.cor'))
         model_gensim.build_vocab(lee_data)
@@ -429,12 +429,12 @@ def test_cbow_hs_training(self):
             overlap_count, 2,
             "only %i overlap in expected %s & actual %s" % (overlap_count, expected_sims_words, sims_gensim_words))
 
-    def test_cbow_hs_training_fromfile(self):
+    def test_cbow_hs_training_fromfile(self, shrink_windows=True):
         with temporary_file('gensim_fasttext.tst') as corpus_file:
             model_gensim = FT_gensim(
                 vector_size=48, sg=0, cbow_mean=1, alpha=0.05, window=5, hs=1, negative=0,
                 min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
-                sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET * 4)
+                sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET * 4, shrink_windows=shrink_windows)
 
             lee_data = LineSentence(datapath('lee_background.cor'))
             utils.save_as_line_sentence(lee_data, corpus_file)
@@ -465,12 +465,12 @@ def test_cbow_hs_training_fromfile(self):
                 overlap_count, 2,
                 "only %i overlap in expected %s & actual %s" % (overlap_count, expected_sims_words, sims_gensim_words))
 
-    def test_sg_hs_training(self):
+    def test_sg_hs_training(self, shrink_windows=True):
 
         model_gensim = FT_gensim(
             vector_size=48, sg=1, cbow_mean=1, alpha=0.025, window=5, hs=1, negative=0,
             min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
-            sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET)
+            sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET, shrink_windows=shrink_windows)
 
         lee_data = LineSentence(datapath('lee_background.cor'))
         model_gensim.build_vocab(lee_data)
@@ -497,12 +497,12 @@ def test_sg_hs_training(self):
             overlap_count, 2,
             "only %i overlap in expected %s & actual %s" % (overlap_count, expected_sims_words, sims_gensim_words))
 
-    def test_sg_hs_training_fromfile(self):
+    def test_sg_hs_training_fromfile(self, shrink_windows=True):
         with temporary_file('gensim_fasttext.tst') as corpus_file:
             model_gensim = FT_gensim(
                 vector_size=48, sg=1, cbow_mean=1, alpha=0.025, window=5, hs=1, negative=0,
                 min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
-                sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET)
+                sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET, shrink_windows=shrink_windows)
 
             lee_data = LineSentence(datapath('lee_background.cor'))
             utils.save_as_line_sentence(lee_data, corpus_file)
@@ -533,6 +533,18 @@ def test_sg_hs_training_fromfile(self):
                 overlap_count, 2,
                 "only %i overlap in expected %s & actual %s" % (overlap_count, expected_sims_words, sims_gensim_words))
 
+    def test_cbow_hs_training_fixedwindowsize(self):
+        self.test_cbow_hs_training(shrink_windows=False)
+
+    def test_cbow_hs_training_fixedwindowsize_fromfile(self):
+        self.test_cbow_hs_training_fromfile(shrink_windows=False)
+
+    def test_sg_hs_training_fixedwindowsize(self):
+        self.test_sg_hs_training(shrink_windows=False)
+
+    def test_sg_hs_training_fixedwindowsize_fromfile(self):
+        self.test_sg_hs_training_fromfile(shrink_windows=False)
+
     def test_cbow_neg_training(self):
 
         model_gensim = FT_gensim(
diff --git a/gensim/test/test_word2vec.py b/gensim/test/test_word2vec.py
--- a/gensim/test/test_word2vec.py
+++ b/gensim/test/test_word2vec.py
@@ -686,6 +686,38 @@ def test_cbow_neg_fromfile(self):
         )
         self.model_sanity(model, with_corpus_file=True)
 
+    def test_sg_fixedwindowsize(self):
+        """Test skipgram with fixed window size. Use NS."""
+        model = word2vec.Word2Vec(
+            sg=1, window=5, shrink_windows=False, hs=0,
+            negative=15, min_count=5, epochs=10, workers=2
+        )
+        self.model_sanity(model)
+
+    def test_sg_fixedwindowsize_fromfile(self):
+        """Test skipgram with fixed window size. Use HS and train from file."""
+        model = word2vec.Word2Vec(
+            sg=1, window=5, shrink_windows=False, hs=1,
+            negative=0, min_count=5, epochs=10, workers=2
+        )
+        self.model_sanity(model, with_corpus_file=True)
+
+    def test_cbow_fixedwindowsize(self, ranks=None):
+        """Test CBOW with fixed window size. Use HS."""
+        model = word2vec.Word2Vec(
+            sg=0, cbow_mean=1, alpha=0.1, window=5, shrink_windows=False,
+            hs=1, negative=0, min_count=5, epochs=10, workers=2
+        )
+        self.model_sanity(model, ranks=ranks)
+
+    def test_cbow_fixedwindowsize_fromfile(self):
+        """Test CBOW with fixed window size. Use NS and train from file."""
+        model = word2vec.Word2Vec(
+            sg=0, cbow_mean=1, alpha=0.1, window=5, shrink_windows=False,
+            hs=0, negative=15, min_count=5, epochs=10, workers=2
+        )
+        self.model_sanity(model, with_corpus_file=True)
+
     def test_cosmul(self):
         model = word2vec.Word2Vec(sentences, vector_size=2, min_count=1, hs=1, negative=0)
         sims = model.wv.most_similar_cosmul('graph', topn=10)