Repository: piskvorky/gensim

Issue:
Runtime error in phrases.py
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

Trying to use export_phrases function on a phrases model.    
Instead getting Runtime error

#### Steps/code/corpus to reproduce

```python
from gensim.models.phrases import Phrases, ENGLISH_CONNECTOR_WORDS

documents = ["I am interested in machine learning projects", 
             "machine learning projects can be useful sometimes",
            "I love working on machine learning projects",
            "interested does not mean the same thing as likes",
            "i am interested in blockchain"]

sentence_stream = [doc.split(" ") for doc in documents]
bigrams = Phrases(sentence_stream, min_count=2, threshold=1, connector_words=ENGLISH_CONNECTOR_WORDS)
trigrams = Phrases(bigrams[sentence_stream], min_count=2, threshold=1)
trigrams.export_phrases()
```

```
RuntimeError                              Traceback (most recent call last)
<ipython-input-190-0f7e41471301> in <module>
----> 1 trigrams.export_phrases()

~\Anaconda3\lib\site-packages\gensim\models\phrases.py in export_phrases(self)
    716         """
    717         result, source_vocab = {}, self.vocab
--> 718         for token in source_vocab:
    719             unigrams = token.split(self.delimiter)
    720             if len(unigrams) < 2:

RuntimeError: dictionary changed size during iteration
```



#### Versions

Please provide the output of:

```python
Windows-10-10.0.17763-SP0
Python 3.8.3 (default, Jul  2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)]
Bits 64
NumPy 1.19.5
SciPy 1.5.2
gensim 4.0.0beta
FAST_VERSION 0
```

Runtime error in phrases.py
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

Trying to use export_phrases function on a phrases model.    
Instead getting Runtime error

#### Steps/code/corpus to reproduce

```python
from gensim.models.phrases import Phrases, ENGLISH_CONNECTOR_WORDS

documents = ["I am interested in machine learning projects", 
             "machine learning projects can be useful sometimes",
            "I love working on machine learning projects",
            "interested does not mean the same thing as likes",
            "i am interested in blockchain"]

sentence_stream = [doc.split(" ") for doc in documents]
bigrams = Phrases(sentence_stream, min_count=2, threshold=1, connector_words=ENGLISH_CONNECTOR_WORDS)
trigrams = Phrases(bigrams[sentence_stream], min_count=2, threshold=1)
trigrams.export_phrases()
```

```
RuntimeError                              Traceback (most recent call last)
<ipython-input-190-0f7e41471301> in <module>
----> 1 trigrams.export_phrases()

~\Anaconda3\lib\site-packages\gensim\models\phrases.py in export_phrases(self)
    716         """
    717         result, source_vocab = {}, self.vocab
--> 718         for token in source_vocab:
    719             unigrams = token.split(self.delimiter)
    720             if len(unigrams) < 2:

RuntimeError: dictionary changed size during iteration
```



#### Versions

Please provide the output of:

```python
Windows-10-10.0.17763-SP0
Python 3.8.3 (default, Jul  2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)]
Bits 64
NumPy 1.19.5
SciPy 1.5.2
gensim 4.0.0beta
FAST_VERSION 0
```



Discussion:
OK, I can see the issue. Phrases employs `defaultdict`, which modifies its content on (unsuccessful) access: `self.vocab[not_in_vocab]` changes `self.vocab`.

The proper fix will be to get rid of this unwanted mutation – either by replacing `vocab[x]` by `vocab.get(x, 0)` in all const functions, or (better, easier to reason about) by replacing the entire `defaultdict` by a plain `dict`.

@thalishsajeed are you up for it?
@piskvorky Yep, I vote for replacing defaultdict. If you agree with that, I'll make a PR with the change. (after checking for any other unwanted issues)


Yes, please go ahead.

You can re-use #3030 , no need to start a new PR.
@piskvorky Is it okay to replace instances of the following code pattern

`self.vocab[word] + = 1`

with

`self.vocab[word] = vocab.get(word, 0) + 1`

I don't know of any way to do updates in-place in a regular dict.

There's also this line - which needs to be modified for similar reasons and making sure generator isn't called twice if you use .get method.

https://github.com/RaRe-Technologies/gensim/blob/a21d9cc768598640f38e4bd03d368f8712a9aa77/gensim/models/phrases.py#L596 

Still comfortable moving to dict right? Also I read some SO posts that defaultdict is more performant that dict, wondering if that is still the case and needs to be considered for this change.
Yes, that's the way to do it!

And yes, I'd expect defaultdict to be (slightly) more performant than dict. But having correct, readable code is more important.

If we ever optimize Phrases, it'll be by translating its code to Cython / C, for a proper 10x+ boost. Not chasing a few percent here and there with defaultdict.
@piskvorky  Thanks! Well, I'm done with the changes, can you point me to how I can run the unit test suite locally? 
Hmm. @mpenkov will `python setup.py test` work, for local testing? I don't see that on the [Developer page](https://github.com/RaRe-Technologies/gensim/wiki/Developer-page).
I usually do `pytest gensim`.

You may need to do something like `pip install -e .[test]` to get the dependencies installed first.
@mpenkov Thanks! Do you have anything for me where I can read regarding the current tests which are passing? 

Also I need some help with writing test cases, why is the test case called testExportPhrases present but with another function i.e find phrases being called inside.? Do i need to create a separate test case or is there an existing test case for export_phrases that I need to fix.

https://github.com/RaRe-Technologies/gensim/blob/a21d9cc768598640f38e4bd03d368f8712a9aa77/gensim/test/test_phrases.py#L215 

Yes, add a new test case please.

I don't know why a test called `testExportPhrases` doesn't test exporting phrases, that's really weird! Can you maybe rename that (to `testFindPhrases`?), and call your new test `testExportPhrases`?
> Thanks! Do you have anything for me where I can read regarding the current tests which are passing?

Other than the developer wiki, which @piskvorky mentioned above, no, I'm not aware of any unit-test related documentation.

Looks like export_phrases got renamed to find_phrases, but the test case wasn't renamed. Renaming the test case should resolve the problem. 
@thalishsajeed did you manage?
> @thalishsajeed did you manage?

Hi yes, I'll work on this tomorrow and update you :) 
@piskvorky : Soo, just a quick update. The current implementation of analyze_sentence seems to modify the vocabulary (the count is zero naturally because defaultdict)

So doing something like - 

`bigrams.analyze_sentence(["dis", "is", "good", "machine", "interested"])`

Will modify the vocabulary and so that bigams.vocab would end up looking like this - 

`'dis': 0, 'is': 0, 'good': 0, 'machine_interested': 0}`

This then seems to be affecting `bigrams.find_phrases` where you end up with different scores every time you run it after performing `bigrams.analyze_sentence`

I guess my question is - is this expected behavior? 

P.S - I hope I was able to explain the issue otherwise , let me know if you want like a more detailed write up of this behaviour.    

Yes, not desirable, as discussed above. Can you replace the `defaultdict` with normal `dict`?
@piskvorky Yes,  I've done that bit. I stumbled upon this while trying to create a test case. My test cases kept failing because the phrase scores were not matching up when I compared the currrent branch to the BugFix branch and it took some time to figure out what was happening. 
> Yes, I've done that bit

Wait, this still happens *after* getting rid of defaultdict? Something else must be afoot then.
> Wait, this still happens _after_ getting rid of defaultdict? Something else must be afoot then.

No no. Let me explain. So after getting rid of default dict I tried to make sure everything else works exactly the same. That's when I stumbled upon this phenomenon where the scores for phrases were different between the develop branch and the bugfix branch. Naturally I assumed that I was making some mistake while getting rid of defaultdict which is why the scores were different.  After diving a little deeper in the debug mode I realized the root cause i.e the vocab being changed when calling analyze_sentence in the develop branch so just wanted to be sure that I'm not messing with some expected behavior. 
No at all, you found a nasty hidden bug – well done!
OK, I can see the issue. Phrases employs `defaultdict`, which modifies its content on (unsuccessful) access: `self.vocab[not_in_vocab]` changes `self.vocab`.

The proper fix will be to get rid of this unwanted mutation – either by replacing `vocab[x]` by `vocab.get(x, 0)` in all const functions, or (better, easier to reason about) by replacing the entire `defaultdict` by a plain `dict`.

@thalishsajeed are you up for it?
@piskvorky Yep, I vote for replacing defaultdict. If you agree with that, I'll make a PR with the change. (after checking for any other unwanted issues)


Yes, please go ahead.

You can re-use #3030 , no need to start a new PR.
@piskvorky Is it okay to replace instances of the following code pattern

`self.vocab[word] + = 1`

with

`self.vocab[word] = vocab.get(word, 0) + 1`

I don't know of any way to do updates in-place in a regular dict.

There's also this line - which needs to be modified for similar reasons and making sure generator isn't called twice if you use .get method.

https://github.com/RaRe-Technologies/gensim/blob/a21d9cc768598640f38e4bd03d368f8712a9aa77/gensim/models/phrases.py#L596 

Still comfortable moving to dict right? Also I read some SO posts that defaultdict is more performant that dict, wondering if that is still the case and needs to be considered for this change.
Yes, that's the way to do it!

And yes, I'd expect defaultdict to be (slightly) more performant than dict. But having correct, readable code is more important.

If we ever optimize Phrases, it'll be by translating its code to Cython / C, for a proper 10x+ boost. Not chasing a few percent here and there with defaultdict.
@piskvorky  Thanks! Well, I'm done with the changes, can you point me to how I can run the unit test suite locally? 
Hmm. @mpenkov will `python setup.py test` work, for local testing? I don't see that on the [Developer page](https://github.com/RaRe-Technologies/gensim/wiki/Developer-page).
I usually do `pytest gensim`.

You may need to do something like `pip install -e .[test]` to get the dependencies installed first.
@mpenkov Thanks! Do you have anything for me where I can read regarding the current tests which are passing? 

Also I need some help with writing test cases, why is the test case called testExportPhrases present but with another function i.e find phrases being called inside.? Do i need to create a separate test case or is there an existing test case for export_phrases that I need to fix.

https://github.com/RaRe-Technologies/gensim/blob/a21d9cc768598640f38e4bd03d368f8712a9aa77/gensim/test/test_phrases.py#L215 

Yes, add a new test case please.

I don't know why a test called `testExportPhrases` doesn't test exporting phrases, that's really weird! Can you maybe rename that (to `testFindPhrases`?), and call your new test `testExportPhrases`?
> Thanks! Do you have anything for me where I can read regarding the current tests which are passing?

Other than the developer wiki, which @piskvorky mentioned above, no, I'm not aware of any unit-test related documentation.

Looks like export_phrases got renamed to find_phrases, but the test case wasn't renamed. Renaming the test case should resolve the problem. 
@thalishsajeed did you manage?
> @thalishsajeed did you manage?

Hi yes, I'll work on this tomorrow and update you :) 
@piskvorky : Soo, just a quick update. The current implementation of analyze_sentence seems to modify the vocabulary (the count is zero naturally because defaultdict)

So doing something like - 

`bigrams.analyze_sentence(["dis", "is", "good", "machine", "interested"])`

Will modify the vocabulary and so that bigams.vocab would end up looking like this - 

`'dis': 0, 'is': 0, 'good': 0, 'machine_interested': 0}`

This then seems to be affecting `bigrams.find_phrases` where you end up with different scores every time you run it after performing `bigrams.analyze_sentence`

I guess my question is - is this expected behavior? 

P.S - I hope I was able to explain the issue otherwise , let me know if you want like a more detailed write up of this behaviour.    

Yes, not desirable, as discussed above. Can you replace the `defaultdict` with normal `dict`?
@piskvorky Yes,  I've done that bit. I stumbled upon this while trying to create a test case. My test cases kept failing because the phrase scores were not matching up when I compared the currrent branch to the BugFix branch and it took some time to figure out what was happening. 
> Yes, I've done that bit

Wait, this still happens *after* getting rid of defaultdict? Something else must be afoot then.
> Wait, this still happens _after_ getting rid of defaultdict? Something else must be afoot then.

No no. Let me explain. So after getting rid of default dict I tried to make sure everything else works exactly the same. That's when I stumbled upon this phenomenon where the scores for phrases were different between the develop branch and the bugfix branch. Naturally I assumed that I was making some mistake while getting rid of defaultdict which is why the scores were different.  After diving a little deeper in the debug mode I realized the root cause i.e the vocab being changed when calling analyze_sentence in the develop branch so just wanted to be sure that I'm not messing with some expected behavior. 
No at all, you found a nasty hidden bug – well done!

Resolving patch:
diff --git a/gensim/models/phrases.py b/gensim/models/phrases.py
--- a/gensim/models/phrases.py
+++ b/gensim/models/phrases.py
@@ -62,7 +62,6 @@
 """
 
 import logging
-from collections import defaultdict
 import itertools
 from math import log
 import pickle
@@ -412,7 +411,7 @@ def load(cls, *args, **kwargs):
             if not isinstance(word, str):
                 logger.info("old version of %s loaded, upgrading %i words in memory", cls.__name__, len(model.vocab))
                 logger.info("re-save the loaded model to avoid this upgrade in the future")
-                vocab = defaultdict(int)
+                vocab = {}
                 for key, value in model.vocab.items():  # needs lots of extra RAM temporarily!
                     vocab[str(key, encoding='utf8')] = value
                 model.vocab = vocab
@@ -554,7 +553,7 @@ def __init__(
         self.min_count = min_count
         self.threshold = threshold
         self.max_vocab_size = max_vocab_size
-        self.vocab = defaultdict(int)  # mapping between token => its count
+        self.vocab = {}  # mapping between token => its count
         self.min_reduce = 1  # ignore any tokens with count smaller than this
         self.delimiter = delimiter
         self.progress_per = progress_per
@@ -579,7 +578,7 @@ def __str__(self):
     def _learn_vocab(sentences, max_vocab_size, delimiter, connector_words, progress_per):
         """Collect unigram and bigram counts from the `sentences` iterable."""
         sentence_no, total_words, min_reduce = -1, 0, 1
-        vocab = defaultdict(int)
+        vocab = {}
         logger.info("collecting all words and their counts")
         for sentence_no, sentence in enumerate(sentences):
             if sentence_no % progress_per == 0:
@@ -590,10 +589,11 @@ def _learn_vocab(sentences, max_vocab_size, delimiter, connector_words, progress
             start_token, in_between = None, []
             for word in sentence:
                 if word not in connector_words:
-                    vocab[word] += 1
+                    vocab[word] = vocab.get(word, 0) + 1
                     if start_token is not None:
                         phrase_tokens = itertools.chain([start_token], in_between, [word])
-                        vocab[delimiter.join(phrase_tokens)] += 1
+                        joined_phrase_token = delimiter.join(phrase_tokens)
+                        vocab[joined_phrase_token] = vocab.get(joined_phrase_token, 0) + 1
                     start_token, in_between = word, []  # treat word as both end of a phrase AND beginning of another
                 elif start_token is not None:
                     in_between.append(word)
@@ -654,7 +654,7 @@ def add_vocab(self, sentences):
             logger.info("merging %i counts into %s", len(vocab), self)
             self.min_reduce = max(self.min_reduce, min_reduce)
             for word, count in vocab.items():
-                self.vocab[word] += count
+                self.vocab[word] = self.vocab.get(word, 0) + count
             if len(self.vocab) > self.max_vocab_size:
                 utils.prune_vocab(self.vocab, self.min_reduce)
                 self.min_reduce += 1
@@ -666,17 +666,17 @@ def add_vocab(self, sentences):
 
     def score_candidate(self, word_a, word_b, in_between):
         # Micro optimization: check for quick early-out conditions, before the actual scoring.
-        word_a_cnt = self.vocab[word_a]
+        word_a_cnt = self.vocab.get(word_a, 0)
         if word_a_cnt <= 0:
             return None, None
 
-        word_b_cnt = self.vocab[word_b]
+        word_b_cnt = self.vocab.get(word_b, 0)
         if word_b_cnt <= 0:
             return None, None
 
         phrase = self.delimiter.join([word_a] + in_between + [word_b])
         # XXX: Why do we care about *all* phrase tokens? Why not just score the start+end bigram?
-        phrase_cnt = self.vocab[phrase]
+        phrase_cnt = self.vocab.get(phrase, 0)
         if phrase_cnt <= 0:
             return None, None
 


Test patch:
diff --git a/gensim/test/test_phrases.py b/gensim/test/test_phrases.py
--- a/gensim/test/test_phrases.py
+++ b/gensim/test/test_phrases.py
@@ -213,15 +213,34 @@ def dumb_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, co
 class TestPhrasesModel(PhrasesCommon, unittest.TestCase):
 
     def test_export_phrases(self):
-        """Test Phrases bigram export phrases."""
+        """Test Phrases bigram and trigram export phrases."""
+        bigram = Phrases(self.sentences, min_count=1, threshold=1, delimiter=' ')
+        trigram = Phrases(bigram[self.sentences], min_count=1, threshold=1, delimiter=' ')
+        seen_bigrams = set(bigram.export_phrases().keys())
+        seen_trigrams = set(trigram.export_phrases().keys())
+
+        assert seen_bigrams == set([
+            'human interface',
+            'response time',
+            'graph minors',
+            'minors survey',
+        ])
+
+        assert seen_trigrams == set([
+            'human interface',
+            'graph minors survey',
+        ])
+
+    def test_find_phrases(self):
+        """Test Phrases bigram find phrases."""
         bigram = Phrases(self.sentences, min_count=1, threshold=1, delimiter=' ')
         seen_bigrams = set(bigram.find_phrases(self.sentences).keys())
 
-        assert seen_bigrams == {
+        assert seen_bigrams == set([
             'response time',
             'graph minors',
             'human interface',
-        }
+        ])
 
     def test_multiple_bigrams_single_entry(self):
         """Test a single entry produces multiple bigrams."""
@@ -441,7 +460,7 @@ def test_multiple_bigrams_single_entry(self):
             'human interface',
         ])
 
-    def test_export_phrases(self):
+    def test_find_phrases(self):
         """Test Phrases bigram export phrases."""
         bigram = Phrases(self.sentences, min_count=1, threshold=1, connector_words=self.connector_words, delimiter=' ')
         seen_bigrams = set(bigram.find_phrases(self.sentences).keys())
@@ -453,6 +472,21 @@ def test_export_phrases(self):
             'lack of interest',
         ])
 
+    def test_export_phrases(self):
+        """Test Phrases bigram export phrases."""
+        bigram = Phrases(self.sentences, min_count=1, threshold=1, delimiter=' ')
+        seen_bigrams = set(bigram.export_phrases().keys())
+        assert seen_bigrams == set([
+            'and graph',
+            'data and',
+            'graph of',
+            'graph survey',
+            'human interface',
+            'lack of',
+            'of interest',
+            'of trees',
+        ])
+
     def test_scoring_default(self):
         """ test the default scoring, from the mikolov word2vec paper """
         bigram = Phrases(self.sentences, min_count=1, threshold=1, connector_words=self.connector_words)
@@ -510,9 +544,9 @@ def test__getitem__(self):
         assert phrased_sentence == ['data_and_graph', 'survey', 'for', 'human_interface']
 
 
-class TestFrozenPhrasesModelCompatibilty(unittest.TestCase):
+class TestFrozenPhrasesModelCompatibility(unittest.TestCase):
 
-    def test_compatibilty(self):
+    def test_compatibility(self):
         phrases = Phrases.load(datapath("phrases-3.6.0.model"))
         phraser = FrozenPhrases.load(datapath("phraser-3.6.0.model"))
         test_sentences = ['trees', 'graph', 'minors']