internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: piskvorky/gensim

Issue:
Word2Vec w/ iterable input, make sure not to exhaust generator
In attempting to use an iterator/generator combination as input to my [Word2Vec model](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Word2Vec), I ran into a similar gotcha as [this person](http://comments.gmane.org/gmane.comp.ai.gensim/4656) where my iterator wasn't re-initializing my generator.

I had an iterator that read from a generator, like this (simplified):

```
def generator():
    for i in range(10):
        yield i

class MyIterator(object):
    def __init__(self):
        self.generator = generator()

    def __iter__(self):
        return self

    def next(self):
        return self.generator.next()
```

BUT, it needed to be like this:

```
def generator():
    for i in range(10):
        yield i

class MyIterator(object):
    def __iter__(self):
        self.generator = generator()
        return self

    def next(self):
        return self.generator.next()
```

because Word2Vec training re-uses the iterator.

Of course, this is user error, but the resulting error that gets thrown by gensim is:

`UnboundLocalError: local variable 'job_no' referenced before assignment`

and that doesn't make my mistake very obvious. I suggest putting some sort of check in place that validates that if an iterator has elements the first time but not the second time, some sort of error indicating so is thrown.



Discussion:

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -737,6 +737,7 @@ def worker_loop():
         done_jobs = 0
         next_alpha = self.alpha
         jobs_source = enumerate(utils.grouper(sentences, chunksize))
+        job_no = -1
         # fill jobs queue with (sentence, alpha) job tuples
         while True:
             try:
@@ -755,6 +756,12 @@ def worker_loop():
                         next_alpha = self.alpha - (self.alpha - self.min_alpha) * (pushed_words / total_words)
                     next_alpha = max(next_alpha, self.min_alpha)
             except StopIteration:
+                if job_no == -1 and self.train_count == 0:
+                    logger.warning(
+                        "train() called with empty iterator (if not intended, "
+                        "be sure to provide a corpus that offers restartable "
+                        "iteration)."
+                    )
                 logger.info(
                     "reached end of input; waiting to finish %i outstanding jobs",
                     job_no - done_jobs + 1)


Test patch:
Repository: piskvorky/gensim

Issue:
Word2vec to use GPU
Add an option (likely subclassing `Word2vec`) to train word2vec model using GPU.



Discussion:

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -1466,7 +1466,7 @@ def load(cls, *args, **kwargs):
         for v in model.vocab.values():
             if hasattr(v, 'sample_int'):
                 break  # already 0.12.0+ style int probabilities
-            else:
+            elif hasattr(v, 'sample_probability'):
                 v.sample_int = int(round(v.sample_probability * 2**32))
                 del v.sample_probability
         if not hasattr(model, 'syn0_lockf') and hasattr(model, 'syn0'):


Test patch:
Repository: piskvorky/gensim

Issue:
DTMWrapper does not support Python 3
TypeError: 'str' does not support the buffer interface

``` python
/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/gensim/models/wrappers/dtmmodel.py in convert_input(self, corpus, time_slices)
    173
    174         with utils.smart_open(self.ftimeslices(), 'wb') as fout:
--> 175             fout.write(str(len(self.time_slices)) + "\n")
    176             for sl in time_slices:
    177                 fout.write(str(sl) + "\n")

TypeError: 'str' does not support the buffer interface
```

On Python 3 this might need to be:

``` python
with utils.smart_open(self.ftimeslices(), 'wb') as fout:
    fout.write(bytes(str(len(self.time_slices))), 'utf-8'))
    for sl in time_slices:
        fout.write(bytes(str(sl)+'\n'), 'utf-8'))
```



Discussion:
Thanks for reporting.

We support both py2 and py3 from the same code base, using `six`. Can you think of a way to fix this so it works on both?


Resolving patch:
diff --git a/gensim/models/wrappers/dtmmodel.py b/gensim/models/wrappers/dtmmodel.py
--- a/gensim/models/wrappers/dtmmodel.py
+++ b/gensim/models/wrappers/dtmmodel.py
@@ -26,6 +26,7 @@
 import os
 from subprocess import PIPE
 import numpy as np
+import six
 
 from gensim import utils, corpora, matutils
 from gensim.utils import check_output
@@ -172,9 +173,9 @@ def convert_input(self, corpus, time_slices):
         corpora.BleiCorpus.save_corpus(self.fcorpustxt(), corpus)
 
         with utils.smart_open(self.ftimeslices(), 'wb') as fout:
-            fout.write(str(len(self.time_slices)) + "\n")
+            fout.write(six.u(str(len(self.time_slices)) + "\n"))
             for sl in time_slices:
-                fout.write(str(sl) + "\n")
+                fout.write(six.u(str(sl) + "\n"))
 
     def train(self, corpus, time_slices, mode, model):
         """


Test patch:
Repository: piskvorky/gensim

Issue:
More detailed/consistent logging around word2vec words dropped
```
2016-08-25 19:24:15 [INFO] gensim.models.word2vec (2913): min_count=10 retains 164581 unique words (drops 100300)
2016-08-25 19:24:15 [INFO] gensim.models.word2vec (2913): min_count leaves 30532905 word corpus (98% of original 30881130)
```

On first glance, it shows that `98%` of the corpus is left intact. You feel all warm and fuzzy inside.

But just before reading that, in spite of telling us that it dropped `100,300` unique words, it doesn't tell us that this accounts for `37.9%` of our unique words! This should stand out.



Discussion:

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -578,24 +578,26 @@ def scale_vocab(self, min_count=None, sample=None, dry_run=False, keep_raw_vocab
             self.min_count = min_count
             self.sample = sample
             self.vocab = {}
-        drop_unique, drop_total, retain_total, original_total = 0, 0, 0, 0
+        retain_total = drop_total = drop_unique = 0
         retain_words = []
         for word, v in iteritems(self.raw_vocab):
             if keep_vocab_item(word, v, min_count, trim_rule=trim_rule):
                 retain_words.append(word)
                 retain_total += v
-                original_total += v
                 if not dry_run:
                     self.vocab[word] = Vocab(count=v, index=len(self.index2word))
                     self.index2word.append(word)
             else:
                 drop_unique += 1
                 drop_total += v
-                original_total += v
-        logger.info("min_count=%d retains %i unique words (drops %i)",
-                    min_count, len(retain_words), drop_unique)
-        logger.info("min_count leaves %i word corpus (%i%% of original %i)",
-                    retain_total, retain_total * 100 / max(original_total, 1), original_total)
+        original_unique_total = len(retain_words) + drop_unique
+        retain_unique_pct = len(retain_words) * 100 / max(original_unique_total, 1)
+        logger.info("min_count=%d retains %i unique words (%i%% of original %i, drops %i)",
+                    min_count, len(retain_words), retain_unique_pct, original_unique_total, drop_unique)
+        original_total = retain_total + drop_total
+        retain_pct = retain_total * 100 / max(original_total, 1)
+        logger.info("min_count=%d leaves %i word corpus (%i%% of original %i, drops %i)",
+                    min_count, retain_total, retain_pct, original_total, drop_total)
 
         # Precalculate each vocabulary item's threshold for sampling
         if not sample:


Test patch:
Repository: piskvorky/gensim

Issue:
Remove "No handlers could be found for logger "gensim.models.word2vec" message during import
Instead print the actual message that word2vec is trying to print



Discussion:
@tmylk Check [logging.basicConfig](https://docs.python.org/2/library/logging.html#logging.basicConfig). 

> Does basic configuration for the logging system by creating a StreamHandler with a default Formatter and adding it to the root logger. The functions debug(), info(), warning(), error() and critical() will call basicConfig() automatically **if no handlers are defined for the root logger.**

I believe adding logging.basicConfig() to _gensim.models.word2vec_ should do the trick.

Implementing this on my system pops the following WARNING in compliance with the logger warning [here](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/word2vec.py#L107)
![image](https://cloud.githubusercontent.com/assets/11366315/19198816/9b9bb9f2-8cde-11e6-836c-c19338e69644.png)

Do you want to print this message?

Yes, that text is very important. Let's print it 

On OSX, the following message, is printed 

```
Slow version of gensim.models.word2vec is being used
Slow version of gensim.models.doc2vec is being used
```

So, I guess, it works correctly on OSX.


Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -96,6 +96,7 @@
 from types import GeneratorType
 
 logger = logging.getLogger(__name__)
+logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s')
 
 try:
     from gensim.models.word2vec_inner import train_batch_sg, train_batch_cbow
@@ -280,7 +281,7 @@ def sigmoid(p):
         return exp(p) / (1 + exp(p))
     else:
         raise ValueError
-        
+
 def train_cbow_pair(model, word, input_word_indices, l1, alpha, learn_vectors=True, learn_hidden=True):
     neu1e = zeros(l1.shape)
 
@@ -1613,7 +1614,7 @@ def n_similarity(self, ws1, ws2):
         v2 = [self[word] for word in ws2]
         return dot(matutils.unitvec(array(v1).mean(axis=0)),
                    matutils.unitvec(array(v2).mean(axis=0)))
-        
+
 
     def init_sims(self, replace=False):
         """


Test patch:
Repository: piskvorky/gensim

Issue:
Remove "No handlers could be found for logger "gensim.models.word2vec" message during import
Instead print the actual message that word2vec is trying to print



Discussion:
@tmylk Check [logging.basicConfig](https://docs.python.org/2/library/logging.html#logging.basicConfig). 

> Does basic configuration for the logging system by creating a StreamHandler with a default Formatter and adding it to the root logger. The functions debug(), info(), warning(), error() and critical() will call basicConfig() automatically **if no handlers are defined for the root logger.**

I believe adding logging.basicConfig() to _gensim.models.word2vec_ should do the trick.

Implementing this on my system pops the following WARNING in compliance with the logger warning [here](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/word2vec.py#L107)
![image](https://cloud.githubusercontent.com/assets/11366315/19198816/9b9bb9f2-8cde-11e6-836c-c19338e69644.png)

Do you want to print this message?

Yes, that text is very important. Let's print it 

On OSX, the following message, is printed 

```
Slow version of gensim.models.word2vec is being used
Slow version of gensim.models.doc2vec is being used
```

So, I guess, it works correctly on OSX.


Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -96,16 +96,15 @@
 from types import GeneratorType
 
 logger = logging.getLogger(__name__)
-logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s')
+
 
 try:
     from gensim.models.word2vec_inner import train_batch_sg, train_batch_cbow
     from gensim.models.word2vec_inner import score_sentence_sg, score_sentence_cbow
     from gensim.models.word2vec_inner import FAST_VERSION, MAX_WORDS_IN_BATCH
-    logger.debug('Fast version of {0} is being used'.format(__name__))
 except ImportError:
     # failed... fall back to plain numpy (20-80x slower training than the above)
-    logger.warning('Slow version of {0} is being used'.format(__name__))
+    # Can't log here because logger is usually not configured at import time
     FAST_VERSION = -1
     MAX_WORDS_IN_BATCH = 10000
 
@@ -432,6 +431,12 @@ def __init__(
         texts are longer than 10000 words, but the standard cython code truncates to that maximum.)
 
         """
+
+        if FAST_VERSION == -1:
+            logger.warning('Slow version of {0} is being used'.format(__name__))
+        else:
+            logger.debug('Fast version of {0} is being used'.format(__name__))
+
         self.vocab = {}  # mapping from a word (string) to a Vocab object
         self.index2word = []  # map from a word's matrix index (int) to word (string)
         self.sg = int(sg)


Test patch:
Repository: piskvorky/gensim

Issue:
Doc2Vec.DocvecsArray.doctag_syn0norm not ignored in save()
Unlike with Word2Vec.syn0norm, oversized `doctag_syn0norm` arrays aren't ignored during native `save()`. As a result, even if they're a shared-reference to `doctag_syn0` after an `init_sims(replace=True)`, both `doctag_syn0` and `doctag_syn0norm` will be written and then read back in, using more storage/RAM than necessary. (See https://groups.google.com/d/msg/gensim/fCcHoZCyV3M/djEZgNWEAAAJ )

Doc2Vec.DocvecsArray.doctag_syn0norm not ignored in save()
Unlike with Word2Vec.syn0norm, oversized `doctag_syn0norm` arrays aren't ignored during native `save()`. As a result, even if they're a shared-reference to `doctag_syn0` after an `init_sims(replace=True)`, both `doctag_syn0` and `doctag_syn0norm` will be written and then read back in, using more storage/RAM than necessary. (See https://groups.google.com/d/msg/gensim/fCcHoZCyV3M/djEZgNWEAAAJ )



Discussion:
@tmylk - Commit typo! That commit related to #879 not #789.

@tmylk - Commit typo! That commit related to #879 not #789.


Resolving patch:
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -353,6 +353,11 @@ def __contains__(self, index):
         else:
             return index in self.doctags
 
+    def save(self, *args, **kwargs):
+        # don't bother storing the cached normalized vectors
+        kwargs['ignore'] = kwargs.get('ignore', ['syn0norm'])
+        super(DocvecsArray, self).save(*args, **kwargs)
+
     def borrow_from(self, other_docvecs):
         self.count = other_docvecs.count
         self.doctags = other_docvecs.doctags


Test patch:
Repository: piskvorky/gensim

Issue:
default-value of 'hs' (hierarchical-softmax) parameter changed w/o explicit warning in 0.12.4
In vs. 0.12.4, the new default of 'hs=0' stays in accordance with the online documenation, but is in contrast to the default-behavior of versions <= 0.12.2. 
If not set explicitely set to 'hs=1', the w2vec.score() function will fail with the error message "we have only implemented score for hs". In a productive environment, this error message might appear as not trivial, and is assumed to be more productive when pointing directly to the change-need of the model-parameters.

default-value of 'hs' (hierarchical-softmax) parameter changed w/o explicit warning in 0.12.4
In vs. 0.12.4, the new default of 'hs=0' stays in accordance with the online documenation, but is in contrast to the default-behavior of versions <= 0.12.2. 
If not set explicitely set to 'hs=1', the w2vec.score() function will fail with the error message "we have only implemented score for hs". In a productive environment, this error message might appear as not trivial, and is assumed to be more productive when pointing directly to the change-need of the model-parameters.



Discussion:

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -968,7 +968,9 @@ def score(self, sentences, total_sentences=int(1e6), chunksize=100, queue_factor
             raise RuntimeError("you must first build vocabulary before scoring new data")
 
         if not self.hs:
-            raise RuntimeError("we have only implemented score for hs")
+            raise RuntimeError("We have currently only implemented score \
+                    for the hierarchical softmax scheme, so you need to have \
+                    run word2vec with hs=1 and negative=0 for this to work.")
 
         def worker_loop():
             """Train the model, lifting lists of sentences from the jobs queue."""


Test patch:
Repository: piskvorky/gensim

Issue:
Word2Vec.build_vocab(…, update=True) gives "ValueError: all the input array dimensions except for the concatenation axis must match exactly"
The`build_vocab` function does not work with the ability to update online.  

i.e.:

    model.build_vocab(sentences, update=False)

works fine, but

    model.build_vocab(sentences, update=True)

does not.

It provides the following error:

    ValueError: all the input array dimensions except for the concatenation axis must match exactly


Discussion:
Are you by chance passing the same `sentences` in consecutively, and thus on the 'update', there are no new words? 

There is somewhat of an expectation that the corpus for an update would contain new words, or else you wouldn't need to be calling `build_vocab(…, update=True)` at all. But it would probably be better for this to be a no-op, ideally with some return value hinting that no update occurred, than give an error. 

Separately, those cases where there are no words – perhaps because the new material is very small or very redundant with previous training – are also cases where subsequent training may not be a net benefit to the model. (This incremental-vocabulary-expansion option is best considered an experimental feature to should be evaluated carefully after each use – not a true 'online' training option where incremental new examples always or even usually lead to net improvements.) So I'd be wary of any process where an update-vocab is being casually called with small/redundant new batches. 
Thank you for your response, I did not realize I was using this improperly.  
My thought was this was a feature that allowed you to train the model such that when it encountered new data when being used, it would have the ability to incorporate that new data into it.  
I am hoping to use word embeddings in a domain for which new words are created within the text fairly often, so I was hoping to find a manner to do this without it throwing an error and halting the program at every new word that it finds after the model has been trained.
I apologize if i misunderstood the use for this feature.
Thank you for your help,  I will continue searching. 
It is not an issue but a better error message is needed.

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -1090,6 +1090,13 @@ def update_weights(self):
         for i in xrange(len(self.wv.syn0), len(self.wv.vocab)):
             # construct deterministic seed from word AND seed argument
             newsyn0[i-len(self.wv.syn0)] = self.seeded_vector(self.wv.index2word[i] + str(self.seed))
+
+        # Raise an error if an online update is run before initial training on a corpus
+        if not len(self.wv.syn0):
+            raise RuntimeError("You cannot do an online vocabulary-update of a model which has no prior vocabulary. " \
+                "First build the vocabulary of your model with a corpus " \
+                "before doing an online update.")
+
         self.wv.syn0 = vstack([self.wv.syn0, newsyn0])
 
         if self.hs:


Test patch:
Repository: piskvorky/gensim

Issue:
1.0.1 reset_from bug
   code should be:


def reset_from(self, other_model):
        """
        Borrow shareable pre-built structures (like vocab) from the other_model. Useful
        if testing multiple models in parallel on the same corpus.
        """
        self.wv.vocab = other_model.wv.vocab
        self.wv.index2word = other_model.wv.index2word



Discussion:

Resolving patch:
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -738,8 +738,8 @@ def reset_from(self, other_model):
         Borrow shareable pre-built structures (like vocab) from the other_model. Useful
         if testing multiple models in parallel on the same corpus.
         """
-        self.wv.vocab = other_model.vocab
-        self.wv.index2word = other_model.index2word
+        self.wv.vocab = other_model.wv.vocab
+        self.wv.index2word = other_model.wv.index2word
         self.cum_table = other_model.cum_table
         self.corpus_count = other_model.corpus_count
         self.reset_weights()


Test patch:
diff --git a/gensim/test/test_word2vec.py b/gensim/test/test_word2vec.py
--- a/gensim/test/test_word2vec.py
+++ b/gensim/test/test_word2vec.py
@@ -678,6 +678,16 @@ def test_sentences_should_not_be_a_generator(self):
     def testLoadOnClassError(self):
         """Test if exception is raised when loading word2vec model on instance"""
         self.assertRaises(AttributeError, load_on_instance)
+
+    def test_reset_from(self):
+        """Test if reset_from() uses pre-built structures from other model"""
+        model = word2vec.Word2Vec(sentences, min_count=1)
+        other_model = word2vec.Word2Vec(new_sentences, min_count=1) 
+        other_vocab = other_model.wv.vocab
+        model.reset_from(other_model)
+        self.assertEqual(model.wv.vocab, other_vocab)
+        
+
 #endclass TestWord2VecModel
 
 class TestWMD(unittest.TestCase):
Repository: piskvorky/gensim

Issue:
Passing `sentences=` to Doc2Vec() gives AttributeError: 'Doc2Vec' object has no attribute 'docvecs'
Hi to all,
as stated in the issue's title, when I create the Doc2Vec object I receive the error _'Doc2Vec' object has no attribute 'docvecs'_ using this code:

```
    labeled_docs = [gensim.models.doc2vec.TaggedDocument(words=words, tags=[label])
                         for words, label in sentences]
    model = gensim.models.Doc2Vec(sentences=labeled_docs) 
```

I'm using the latest gensim version (0.12.2) on Archlinux. I've tried also with these versions:
- gensim 0.12.1 - same error
- gensim 0.10.3 - correctly works (using the appropriate method's parameters)

I think that there is something wrong in the current version of the library. 
May you help me with this kind of issue? 

Thank you in advance,
Alessandro



Discussion:
In gensim 0.12+, the parameter to provide the (iterable) corpus in the Doc2Vec constructor is called `documents`, not `sentences`, to better hint that each text may be quite long (many sentences, often including punctuation as tokens). 

(I'll look into improving the error message that's triggered, since this is a natural mistake to make.)

@gojomo Thank you for your answer. I have solved the problem changing the constructor call to:

```
  model = gensim.models.Doc2Vec(documents=labeled_docs, **parameters)
```

Effectively, using _sentences_ as a parameter, generates the error that I've reported in the issue. This happens only with the latest versions. 

Cheers,
Alessandro 

You're welcome!

(Re-opening as a TODO for myself to make the error better.)


Resolving patch:
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -614,6 +614,9 @@ def __init__(self, documents=None, dm_mean=None,
         of the model.
         """
 
+        if 'sentences' in kwargs:
+            raise DeprecationWarning("'sentences' in doc2vec was renamed to 'documents'. Please use documents parameter.")
+
         super(Doc2Vec, self).__init__(
             sg=(1 + dm) % 2,
             null_word=dm_concat, **kwargs)


Test patch: