Repository: piskvorky/gensim

Issue:
different Phrases behavior for same code/corpus between 0.13.4.1 and 2.1.0
#### Description
Hi friends -- I've been using gensim 0.13.4.1 to generate bigrams and trigrams (Phrase model), then to train a word2vec model on a stream of sentences run thru the trigrams phrase model. Works great. 

But I've tried to upgrade to 1.0.0 and 2.1.0 and the results are way way worse -- with the exact same code. I've scoured the release notes but haven't found an explanation. Don't see deprecation warnings in 0.13.4.1 either.

I should clarify -- on gensim 2.1.0, everything "works", i.e. there are no exceptions. It's just that the answers it gives for similarity are incorrect. The classic man : king :: woman : __ analogy gives "chernow" in the 2.1.0-trained model, but "queen" (the expected answer) in the 0.13.4.1-trained model.

I'm not an expert on any of this, so this could be a silly mistake. (I posted about this in the gitter channel and was directed to open an issue.)

#### Steps/Code/Corpus to Reproduce
Snippet:
```
  model = gensim.models.Word2Vec(
                                  ngrams_models[ngrams_model](sentences) if ngrams_model else sentences, 
                                  workers=4, 
                                  min_count=min_count, 
                                  size=size, 
                                  sample=downsampling, 
                                  # sg=(1 if use_skipgrams else 0)
                                )

  model.init_sims(replace=True)
  try:
    model_name = "model_%s_%s_%s_min_count_%s_size_%s_downsampling_%s_%s.bin" % (sentences_filename.split("/")[-1].split(".")[0], "stemmed" if stemming else "raw_words", ngrams_model, min_count, size, downsampling, "sg" if use_skipgrams else "cbow")
  except:
    model_name = "model.bin"
  model.save(model_name)

  with open("most_recent_model_filename.txt", "w") as f:
    f.write(model_name)

  print(model.most_similar_cosmul(positive=['woman', 'king'], negative=['man']))
```

FULL training script (slightly sanitized): https://github.com/jeremybmerrill/ineffable_abracadabra_of_cleaning/blob/master/train_in_vain.py. The only difference it takes to get the different results is changing `requirements.txt` to specify gensim 2.1.0 and re-running `pip install -r requirements.txt`

#### Expected Results
`print(model.most_similar_cosmul(positive=['woman', 'king'], negative=['man']))`
should have `queen` as the first answer, e.g.
```[(u'queen', 0.8925966620445251), (u'royal', 0.8144874572753906), (u'wales', 0.7882706522941589), (u'kingston_ontario', 0.7752816677093506), (u'helen_mirren', 0.7718634009361267), (u'bismarck', 0.7668762803077698), (u'princess', 0.7609482407569885), (u'queen_elizabeth_ii', 0.76094651222229), (u'prince_philip', 0.7592762112617493), (u'duchess', 0.7587381601333618)]```

this is the result from in 0.13.4.1

#### Actual Results
gives `chernow` as the first answer and a bunch of other irrelevant stuff, ```("chernow",0.8686840534210205),("anxiety_disorders",0.8328080177307129),("marlinga",0.8300415277481079), ...```

(This is with a model trained in 2.1.0 and calling `model.most_similar_cosmul` in 2.1.0. I haven't tested what happens using a model trained in 0.13.4.1, but loading the model and calling `model.most_similar_cosmul` in 2.1.0.)

#### Versions
````
Linux-4.12.0-999-generic-x86_64-with-Ubuntu-17.04-zesty
>>> import sys; print("Python", sys.version)
('Python', '2.7.13 (default, Jan 19 2017, 14:48:08) \n[GCC 6.3.0 20170118]')
>>> import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.12.1')
>>> import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.19.0')
>>> import gensim; print("gensim", gensim.__version__)
('gensim', '2.1.0')
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
('FAST_VERSION', 1)
````
I appreciate any help y'all can offer! (Great software, love it, so much fun, :D etc. )
different Phrases behavior for same code/corpus between 0.13.4.1 and 2.1.0
#### Description
Hi friends -- I've been using gensim 0.13.4.1 to generate bigrams and trigrams (Phrase model), then to train a word2vec model on a stream of sentences run thru the trigrams phrase model. Works great. 

But I've tried to upgrade to 1.0.0 and 2.1.0 and the results are way way worse -- with the exact same code. I've scoured the release notes but haven't found an explanation. Don't see deprecation warnings in 0.13.4.1 either.

I should clarify -- on gensim 2.1.0, everything "works", i.e. there are no exceptions. It's just that the answers it gives for similarity are incorrect. The classic man : king :: woman : __ analogy gives "chernow" in the 2.1.0-trained model, but "queen" (the expected answer) in the 0.13.4.1-trained model.

I'm not an expert on any of this, so this could be a silly mistake. (I posted about this in the gitter channel and was directed to open an issue.)

#### Steps/Code/Corpus to Reproduce
Snippet:
```
  model = gensim.models.Word2Vec(
                                  ngrams_models[ngrams_model](sentences) if ngrams_model else sentences, 
                                  workers=4, 
                                  min_count=min_count, 
                                  size=size, 
                                  sample=downsampling, 
                                  # sg=(1 if use_skipgrams else 0)
                                )

  model.init_sims(replace=True)
  try:
    model_name = "model_%s_%s_%s_min_count_%s_size_%s_downsampling_%s_%s.bin" % (sentences_filename.split("/")[-1].split(".")[0], "stemmed" if stemming else "raw_words", ngrams_model, min_count, size, downsampling, "sg" if use_skipgrams else "cbow")
  except:
    model_name = "model.bin"
  model.save(model_name)

  with open("most_recent_model_filename.txt", "w") as f:
    f.write(model_name)

  print(model.most_similar_cosmul(positive=['woman', 'king'], negative=['man']))
```

FULL training script (slightly sanitized): https://github.com/jeremybmerrill/ineffable_abracadabra_of_cleaning/blob/master/train_in_vain.py. The only difference it takes to get the different results is changing `requirements.txt` to specify gensim 2.1.0 and re-running `pip install -r requirements.txt`

#### Expected Results
`print(model.most_similar_cosmul(positive=['woman', 'king'], negative=['man']))`
should have `queen` as the first answer, e.g.
```[(u'queen', 0.8925966620445251), (u'royal', 0.8144874572753906), (u'wales', 0.7882706522941589), (u'kingston_ontario', 0.7752816677093506), (u'helen_mirren', 0.7718634009361267), (u'bismarck', 0.7668762803077698), (u'princess', 0.7609482407569885), (u'queen_elizabeth_ii', 0.76094651222229), (u'prince_philip', 0.7592762112617493), (u'duchess', 0.7587381601333618)]```

this is the result from in 0.13.4.1

#### Actual Results
gives `chernow` as the first answer and a bunch of other irrelevant stuff, ```("chernow",0.8686840534210205),("anxiety_disorders",0.8328080177307129),("marlinga",0.8300415277481079), ...```

(This is with a model trained in 2.1.0 and calling `model.most_similar_cosmul` in 2.1.0. I haven't tested what happens using a model trained in 0.13.4.1, but loading the model and calling `model.most_similar_cosmul` in 2.1.0.)

#### Versions
````
Linux-4.12.0-999-generic-x86_64-with-Ubuntu-17.04-zesty
>>> import sys; print("Python", sys.version)
('Python', '2.7.13 (default, Jan 19 2017, 14:48:08) \n[GCC 6.3.0 20170118]')
>>> import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.12.1')
>>> import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.19.0')
>>> import gensim; print("gensim", gensim.__version__)
('gensim', '2.1.0')
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
('FAST_VERSION', 1)
````
I appreciate any help y'all can offer! (Great software, love it, so much fun, :D etc. )
different Phrases behavior for same code/corpus between 0.13.4.1 and 2.1.0
#### Description
Hi friends -- I've been using gensim 0.13.4.1 to generate bigrams and trigrams (Phrase model), then to train a word2vec model on a stream of sentences run thru the trigrams phrase model. Works great. 

But I've tried to upgrade to 1.0.0 and 2.1.0 and the results are way way worse -- with the exact same code. I've scoured the release notes but haven't found an explanation. Don't see deprecation warnings in 0.13.4.1 either.

I should clarify -- on gensim 2.1.0, everything "works", i.e. there are no exceptions. It's just that the answers it gives for similarity are incorrect. The classic man : king :: woman : __ analogy gives "chernow" in the 2.1.0-trained model, but "queen" (the expected answer) in the 0.13.4.1-trained model.

I'm not an expert on any of this, so this could be a silly mistake. (I posted about this in the gitter channel and was directed to open an issue.)

#### Steps/Code/Corpus to Reproduce
Snippet:
```
  model = gensim.models.Word2Vec(
                                  ngrams_models[ngrams_model](sentences) if ngrams_model else sentences, 
                                  workers=4, 
                                  min_count=min_count, 
                                  size=size, 
                                  sample=downsampling, 
                                  # sg=(1 if use_skipgrams else 0)
                                )

  model.init_sims(replace=True)
  try:
    model_name = "model_%s_%s_%s_min_count_%s_size_%s_downsampling_%s_%s.bin" % (sentences_filename.split("/")[-1].split(".")[0], "stemmed" if stemming else "raw_words", ngrams_model, min_count, size, downsampling, "sg" if use_skipgrams else "cbow")
  except:
    model_name = "model.bin"
  model.save(model_name)

  with open("most_recent_model_filename.txt", "w") as f:
    f.write(model_name)

  print(model.most_similar_cosmul(positive=['woman', 'king'], negative=['man']))
```

FULL training script (slightly sanitized): https://github.com/jeremybmerrill/ineffable_abracadabra_of_cleaning/blob/master/train_in_vain.py. The only difference it takes to get the different results is changing `requirements.txt` to specify gensim 2.1.0 and re-running `pip install -r requirements.txt`

#### Expected Results
`print(model.most_similar_cosmul(positive=['woman', 'king'], negative=['man']))`
should have `queen` as the first answer, e.g.
```[(u'queen', 0.8925966620445251), (u'royal', 0.8144874572753906), (u'wales', 0.7882706522941589), (u'kingston_ontario', 0.7752816677093506), (u'helen_mirren', 0.7718634009361267), (u'bismarck', 0.7668762803077698), (u'princess', 0.7609482407569885), (u'queen_elizabeth_ii', 0.76094651222229), (u'prince_philip', 0.7592762112617493), (u'duchess', 0.7587381601333618)]```

this is the result from in 0.13.4.1

#### Actual Results
gives `chernow` as the first answer and a bunch of other irrelevant stuff, ```("chernow",0.8686840534210205),("anxiety_disorders",0.8328080177307129),("marlinga",0.8300415277481079), ...```

(This is with a model trained in 2.1.0 and calling `model.most_similar_cosmul` in 2.1.0. I haven't tested what happens using a model trained in 0.13.4.1, but loading the model and calling `model.most_similar_cosmul` in 2.1.0.)

#### Versions
````
Linux-4.12.0-999-generic-x86_64-with-Ubuntu-17.04-zesty
>>> import sys; print("Python", sys.version)
('Python', '2.7.13 (default, Jan 19 2017, 14:48:08) \n[GCC 6.3.0 20170118]')
>>> import numpy; print("NumPy", numpy.__version__)
('NumPy', '1.12.1')
>>> import scipy; print("SciPy", scipy.__version__)
('SciPy', '0.19.0')
>>> import gensim; print("gensim", gensim.__version__)
('gensim', '2.1.0')
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
('FAST_VERSION', 1)
````
I appreciate any help y'all can offer! (Great software, love it, so much fun, :D etc. )


Discussion:
Certainly a concern and maybe a worrisome regression. 

Without yet looking deeply into specifics of your process, some things worth checking:

- do you see a similar discrepancy with plain `most_similar()` (as opposed to `most_similar_cosmul()`)? 
- with logging on (say to INFO level), is there anything different/fishy about the reporting/progress-stats/timing when running the 2.0.1 variant – perhaps suggesting less training is happening, or the same corpus is being interpreted as a different size?
- since you are able to see the error with a unigram test, what if you eliminate all the Phrases setup – do you see the same drop-off in results-quality? 
- if it's easy to re-try your process changing just the version of gensim, do you see the same problem in intervening versions? (Especially: 1.0.1 and 2.0.0.)
- it would be interesting to test the variant you parenthetically mention: if a model trained in 0.13.4.1, `save()`d, but then `load()`ed in 2.0.1 shows the better or worse results
- I don't believe there have been any significant changes in defaults between those versions (though there were earlier); still, to be certain you may want to log the actual in-model values used, for each of the things that *can* be specified in the model-initialization (`Word2Vec.__init__()` parameters as copied into model properties)


 
Hi @gojomo; thanks.

> it would be interesting to test the variant you parenthetically mention: if a model trained in 0.13.4.1, save()d, but then load()ed in 2.0.1 shows the better or worse results

This one is easy to check. `load`ing a model in 2.0.1 that I `save`d in 0.13.4.1 gives the "right" answers. I.e. "queen" not "chernow".

I will check on the rest of the stuff. 
> with logging on (say to INFO level), is there anything different/fishy about the reporting/progress-stats/timing when running the 2.0.1 variant – perhaps suggesting less training is happening, or the same corpus is being interpreted as a different size?

Yes, it turns out.

```
2017-06-09 09:09:36,280 : WARNING : train() called with an empty iterator (if not intended, be sure to provide a corpus that offers restartable iteration = an iterable).

2017-06-09 09:09:36,282 : WARNING : under 10 jobs per worker: consider setting a smaller `batch_words' for smoother alpha decay
2017-06-09 09:09:36,282 : WARNING : supplied example count (0) did not equal expected count (27454375)
```

Looks like quite the hint. I train the model like this (which presumably calls `train()` internally):
```
  model = gensim.models.Word2Vec(
                                  trigrams_model[bigrams_model[sentences]], 
                                  workers=4, 
                                  min_count=min_count, 
                                  size=size, 
                                  sample=downsampling, 
                                  # sg=(1 if use_skipgrams else 0)
                                )
```
I'm trying right now to train it without the ngram stuff, just the plain sentences iterator.

You are right about the ngrams -- removing that and throwing in the plain sentences iterator on 2.1.0 works fine.

Any idea what might have caused the change? 
This is helpful narrowing. It might be a Phrases regression, we'll have to take a look at recent changes there, especially around https://github.com/RaRe-Technologies/gensim/commit/34759be7af4394a48a0627ad4b21ffd7869ba464 @tmylk @ELind77 

A practical workaround could be to apply the (two-level) phrase-ification just once, writing the results to a new file – which is then iterated over multiple times as a simple text corpus. (This could offer performance benefits too - the repeated phrase-promotion on each training iteration can be expensive duplicate work.)
Certainly a concern and maybe a worrisome regression. 

Without yet looking deeply into specifics of your process, some things worth checking:

- do you see a similar discrepancy with plain `most_similar()` (as opposed to `most_similar_cosmul()`)? 
- with logging on (say to INFO level), is there anything different/fishy about the reporting/progress-stats/timing when running the 2.0.1 variant – perhaps suggesting less training is happening, or the same corpus is being interpreted as a different size?
- since you are able to see the error with a unigram test, what if you eliminate all the Phrases setup – do you see the same drop-off in results-quality? 
- if it's easy to re-try your process changing just the version of gensim, do you see the same problem in intervening versions? (Especially: 1.0.1 and 2.0.0.)
- it would be interesting to test the variant you parenthetically mention: if a model trained in 0.13.4.1, `save()`d, but then `load()`ed in 2.0.1 shows the better or worse results
- I don't believe there have been any significant changes in defaults between those versions (though there were earlier); still, to be certain you may want to log the actual in-model values used, for each of the things that *can* be specified in the model-initialization (`Word2Vec.__init__()` parameters as copied into model properties)


 
Hi @gojomo; thanks.

> it would be interesting to test the variant you parenthetically mention: if a model trained in 0.13.4.1, save()d, but then load()ed in 2.0.1 shows the better or worse results

This one is easy to check. `load`ing a model in 2.0.1 that I `save`d in 0.13.4.1 gives the "right" answers. I.e. "queen" not "chernow".

I will check on the rest of the stuff. 
> with logging on (say to INFO level), is there anything different/fishy about the reporting/progress-stats/timing when running the 2.0.1 variant – perhaps suggesting less training is happening, or the same corpus is being interpreted as a different size?

Yes, it turns out.

```
2017-06-09 09:09:36,280 : WARNING : train() called with an empty iterator (if not intended, be sure to provide a corpus that offers restartable iteration = an iterable).

2017-06-09 09:09:36,282 : WARNING : under 10 jobs per worker: consider setting a smaller `batch_words' for smoother alpha decay
2017-06-09 09:09:36,282 : WARNING : supplied example count (0) did not equal expected count (27454375)
```

Looks like quite the hint. I train the model like this (which presumably calls `train()` internally):
```
  model = gensim.models.Word2Vec(
                                  trigrams_model[bigrams_model[sentences]], 
                                  workers=4, 
                                  min_count=min_count, 
                                  size=size, 
                                  sample=downsampling, 
                                  # sg=(1 if use_skipgrams else 0)
                                )
```
I'm trying right now to train it without the ngram stuff, just the plain sentences iterator.

You are right about the ngrams -- removing that and throwing in the plain sentences iterator on 2.1.0 works fine.

Any idea what might have caused the change? 
This is helpful narrowing. It might be a Phrases regression, we'll have to take a look at recent changes there, especially around https://github.com/RaRe-Technologies/gensim/commit/34759be7af4394a48a0627ad4b21ffd7869ba464 @tmylk @ELind77 

A practical workaround could be to apply the (two-level) phrase-ification just once, writing the results to a new file – which is then iterated over multiple times as a simple text corpus. (This could offer performance benefits too - the repeated phrase-promotion on each training iteration can be expensive duplicate work.)
Certainly a concern and maybe a worrisome regression. 

Without yet looking deeply into specifics of your process, some things worth checking:

- do you see a similar discrepancy with plain `most_similar()` (as opposed to `most_similar_cosmul()`)? 
- with logging on (say to INFO level), is there anything different/fishy about the reporting/progress-stats/timing when running the 2.0.1 variant – perhaps suggesting less training is happening, or the same corpus is being interpreted as a different size?
- since you are able to see the error with a unigram test, what if you eliminate all the Phrases setup – do you see the same drop-off in results-quality? 
- if it's easy to re-try your process changing just the version of gensim, do you see the same problem in intervening versions? (Especially: 1.0.1 and 2.0.0.)
- it would be interesting to test the variant you parenthetically mention: if a model trained in 0.13.4.1, `save()`d, but then `load()`ed in 2.0.1 shows the better or worse results
- I don't believe there have been any significant changes in defaults between those versions (though there were earlier); still, to be certain you may want to log the actual in-model values used, for each of the things that *can* be specified in the model-initialization (`Word2Vec.__init__()` parameters as copied into model properties)


 
Hi @gojomo; thanks.

> it would be interesting to test the variant you parenthetically mention: if a model trained in 0.13.4.1, save()d, but then load()ed in 2.0.1 shows the better or worse results

This one is easy to check. `load`ing a model in 2.0.1 that I `save`d in 0.13.4.1 gives the "right" answers. I.e. "queen" not "chernow".

I will check on the rest of the stuff. 
> with logging on (say to INFO level), is there anything different/fishy about the reporting/progress-stats/timing when running the 2.0.1 variant – perhaps suggesting less training is happening, or the same corpus is being interpreted as a different size?

Yes, it turns out.

```
2017-06-09 09:09:36,280 : WARNING : train() called with an empty iterator (if not intended, be sure to provide a corpus that offers restartable iteration = an iterable).

2017-06-09 09:09:36,282 : WARNING : under 10 jobs per worker: consider setting a smaller `batch_words' for smoother alpha decay
2017-06-09 09:09:36,282 : WARNING : supplied example count (0) did not equal expected count (27454375)
```

Looks like quite the hint. I train the model like this (which presumably calls `train()` internally):
```
  model = gensim.models.Word2Vec(
                                  trigrams_model[bigrams_model[sentences]], 
                                  workers=4, 
                                  min_count=min_count, 
                                  size=size, 
                                  sample=downsampling, 
                                  # sg=(1 if use_skipgrams else 0)
                                )
```
I'm trying right now to train it without the ngram stuff, just the plain sentences iterator.

You are right about the ngrams -- removing that and throwing in the plain sentences iterator on 2.1.0 works fine.

Any idea what might have caused the change? 
This is helpful narrowing. It might be a Phrases regression, we'll have to take a look at recent changes there, especially around https://github.com/RaRe-Technologies/gensim/commit/34759be7af4394a48a0627ad4b21ffd7869ba464 @tmylk @ELind77 

A practical workaround could be to apply the (two-level) phrase-ification just once, writing the results to a new file – which is then iterated over multiple times as a simple text corpus. (This could offer performance benefits too - the repeated phrase-promotion on each training iteration can be expensive duplicate work.)

Resolving patch:
diff --git a/gensim/models/phrases.py b/gensim/models/phrases.py
--- a/gensim/models/phrases.py
+++ b/gensim/models/phrases.py
@@ -104,6 +104,7 @@ def _is_single(obj):
     is a corpus if it is an iterable of documents.
     """
     obj_iter = iter(obj)
+    temp_iter = obj_iter
     try:
         peek = next(obj_iter)
         obj_iter = it.chain([peek], obj_iter)
@@ -113,9 +114,12 @@ def _is_single(obj):
     if isinstance(peek, string_types):
         # It's a document, return the iterator
         return True, obj_iter
+    if temp_iter == obj:
+        # Checking for iterator to the object
+        return False, obj_iter
     else:
         # If the first item isn't a string, assume obj is a corpus
-        return False, obj_iter
+        return False, obj
 
 
 class SentenceAnalyzer(object):


Test patch:
diff --git a/gensim/test/test_phrases.py b/gensim/test/test_phrases.py
--- a/gensim/test/test_phrases.py
+++ b/gensim/test/test_phrases.py
@@ -160,6 +160,16 @@ def setUp(self):
         self.bigram_unicode = Phrases(
             self.unicode_sentences, min_count=1, threshold=1, common_terms=self.common_terms)
 
+    def testEmptyPhrasifiedSentencesIterator(self):
+        bigram_phrases = Phrases(self.sentences)
+        bigram_phraser = Phraser(bigram_phrases)
+        trigram_phrases = Phrases(bigram_phraser[self.sentences])
+        trigram_phraser = Phraser(trigram_phrases)
+        trigrams = trigram_phraser[bigram_phraser[self.sentences]]
+        fst, snd = list(trigrams), list(trigrams)
+        self.assertEqual(fst, snd)
+        self.assertNotEqual(snd, [])
+
     def testEmptyInputsOnBigramConstruction(self):
         """Test that empty inputs don't throw errors and return the expected result."""
         # Empty list -> empty list