internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: piskvorky/gensim

Issue:
Documentation error in Doc2Vec
This is super simple, but the documentation says the default value of the `sample` parameter in Doc2Vec is 0:

> sample = threshold for configuring which higher-frequency words are randomly downsampled;
default is 0 (off), useful value is 1e-5.

But the actual default value is 1e-3 (0.001), based on the fact that the DocVec inherits from WordVec, where sample is set to 1e-3. See: https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/word2vec.py#L367

I can confirm this by simply initiating a model with all defaults:

```
>>> from gensim.models import Doc2Vec
>>> model = Doc2Vec()
>>> print(model.sample)
0.001
```

Not sure if the desire behavior is what's reflected in the documentation, in which case the default should be changed, or if the documentation should be updated. Simple fix either way, I imagine, but as is it causes some confusion.


Discussion:
The comment should accurately reflect the current inherited default.

Resolving patch:
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -580,7 +580,7 @@ def __init__(self, documents=None, dm_mean=None,
         need about 1GB of RAM. Set to `None` for no limit (default).
 
         `sample` = threshold for configuring which higher-frequency words are randomly downsampled;
-                default is 0 (off), useful value is 1e-5.
+                default is 1e-3, useful value is 1e-5.
 
         `workers` = use this many worker threads to train the model (=faster training with multicore machines).
 
@@ -616,7 +616,8 @@ def __init__(self, documents=None, dm_mean=None,
 
         super(Doc2Vec, self).__init__(
             sg=(1 + dm) % 2,
-            null_word=dm_concat, **kwargs)
+            null_word=dm_concat,
+            **kwargs)
 
         self.load = call_on_class_only
 


Test patch:
Repository: piskvorky/gensim

Issue:
Minimum version for numpy/scipy
In the last release, I had to "bump" versions for numpy/scipy. This may be a problem for *ops people (need to upgrade old versions). 

For this reason, we should downgrade numpy/scipy version and don't break anything.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -229,7 +229,7 @@ def finalize_options(self):
 test_env = [
     'testfixtures',
     'unittest2',
-    'Morfessor==2.0.2a4',
+    'Morfessor == 2.0.2a4',
     'scikit-learn',
     'pyemd',
     'annoy',
@@ -289,7 +289,7 @@ def finalize_options(self):
     ],
     install_requires=[
         'numpy >= 1.11.3',
-        'scipy >= 0.19.0',
+        'scipy >= 0.18.1',
         'six >= 1.5.0',
         'smart_open >= 1.2.1',
     ],


Test patch:
Repository: piskvorky/gensim

Issue:
ShardedCorpus skips the first value of a generator
#### Description
ShardedCorpus skips the first value of a generator. This is possibly caused by ShardedCorpus not using the fixed corpus that is returned from is_corpus method, but I haven't verified this yet.

#### Steps/Code/Corpus to Reproduce

```
from gensim.corpora.sharded_corpus import ShardedCorpus

def my_generator():
    yield [(0,1)]
    yield [(1,1)]
    yield [(2,1)]

corpus = ShardedCorpus("corpus", my_generator(), dim=3, overwrite=True)

print(len(corpus))
print(corpus[0])
```

#### Expected Results

Expected output:
```
3
[ 1.  0.  0.]
```

#### Actual Results

Actual output:
```
2
[ 0.  1.  0.]
```

I.e. The first item in the generator has been skipped and is missing from the resulting corpus

#### Versions

Darwin-16.7.0-x86_64-i386-64bit
Python 3.6.1 |Anaconda 4.4.0 (x86_64)| (default, May 11 2017, 13:04:09) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.12.1
SciPy 0.19.0
gensim 2.3.0
FAST_VERSION 1



Discussion:
@karkkainenk1 thanks for reporting and for the minimal example!

Some destructive peeking, or incorrect use of `is_corpus`, would be my guess too. Can you look into it?

Resolving patch:
diff --git a/gensim/corpora/sharded_corpus.py b/gensim/corpora/sharded_corpus.py
--- a/gensim/corpora/sharded_corpus.py
+++ b/gensim/corpora/sharded_corpus.py
@@ -251,7 +251,8 @@ def __init__(self, output_prefix, corpus, dim=None,
     def init_shards(self, output_prefix, corpus, shardsize=4096, dtype=_default_dtype):
         """Initialize shards from the corpus."""
 
-        if not gensim.utils.is_corpus(corpus):
+        is_corpus, corpus = gensim.utils.is_corpus(corpus)
+        if not is_corpus:
             raise ValueError(
                 "Cannot initialize shards without a corpus to read from! (Got corpus type: {0})".format(type(corpus))
             )


Test patch:
diff --git a/gensim/test/test_sharded_corpus.py b/gensim/test/test_sharded_corpus.py
--- a/gensim/test/test_sharded_corpus.py
+++ b/gensim/test/test_sharded_corpus.py
@@ -256,6 +256,18 @@ def test_resize(self):
             fname = dataset._shard_name(n)
             self.assertTrue(os.path.isfile(fname))
 
+    def test_init_with_generator(self):
+
+        def data_generator():
+            yield [(0, 1)]
+            yield [(1, 1)]
+
+        gen_tmp_fname = self.tmp_fname + '.generator'
+        corpus = ShardedCorpus(gen_tmp_fname, data_generator(), dim=2)
+
+        self.assertEqual(2, len(corpus))
+        self.assertEqual(1, corpus[0][0])
+
 
 if __name__ == '__main__':
     suite = unittest.TestSuite()
Repository: piskvorky/gensim

Issue:
LdaSeqModel word probabilities (too rough rounding)
***Problem:*** Probabilities of words don't sum up to 1 because we have `round(_, 3)` in `print_topic` method
***Solution:*** Remove `round` from `print_topic` from [here](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/ldaseqmodel.py#L361)

Based on [message from maillist](https://groups.google.com/d/msg/gensim/mL4C6-XaEFo/XnSSrWqwBgAJ)


@bhargavvader what's a reason to use `round` in this place?








Discussion:
@menshikh-iv and @bhargavvader, thanks so much for looking into this issue! Do you have any idea when this problem will be fixed? Many thanks in advance!
@myrthevandieijen It's very easy to fix, but I want to get an explanation from @bhargavvader about this behavior.
I really can't seem to remember why i had used `round` over here - can anyone make the fix and then see if the tests still pass fine?
Any progress on this issue so far?Thanks! 

Resolving patch:
diff --git a/gensim/models/ldaseqmodel.py b/gensim/models/ldaseqmodel.py
--- a/gensim/models/ldaseqmodel.py
+++ b/gensim/models/ldaseqmodel.py
@@ -358,7 +358,7 @@ def print_topic(self, topic, time=0, top_terms=20):
         topic = np.exp(topic[time])
         topic = topic / topic.sum()
         bestn = matutils.argsort(topic, top_terms, reverse=True)
-        beststr = [(self.id2word[id_], round(topic[id_], 3)) for id_ in bestn]
+        beststr = [(self.id2word[id_], topic[id_]) for id_ in bestn]
         return beststr
 
 


Test patch:
Repository: piskvorky/gensim

Issue:
gensim import is very slow
#### Description
Gensim import works very long if you'll have installed Keras+TF, report from [maillist](https://groups.google.com/forum/#!msg/gensim/HY90TZiBiAA/td4fsmwFBgAJ)

#### Steps/Code/Corpus to Reproduce
```
import gensim
```

#### Expected Results
Fast import (less than 0.5 sec) without any `stdout`

#### Actual Results
Slow import (more that 1 sec) + `Using TensorFlow backend.`




Discussion:

Resolving patch:
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -78,11 +78,6 @@
 from six import string_types, iteritems
 from six.moves import xrange
 from scipy import stats
-try:
-    from keras.layers import Embedding
-    KERAS_INSTALLED = True
-except ImportError:
-    KERAS_INSTALLED = False
 
 
 logger = logging.getLogger(__name__)
@@ -833,11 +828,13 @@ def init_sims(self, replace=False):
             else:
                 self.syn0norm = (self.syn0 / sqrt((self.syn0 ** 2).sum(-1))[..., newaxis]).astype(REAL)
 
-    def get_embedding_layer(self, train_embeddings=False):
+    def get_keras_embedding(self, train_embeddings=False):
         """
         Return a Keras 'Embedding' layer with weights set as the Word2Vec model's learned word embeddings
         """
-        if not KERAS_INSTALLED:
+        try:
+            from keras.layers import Embedding
+        except ImportError:
             raise ImportError("Please install Keras to use this function")
         weights = self.syn0
 


Test patch:
diff --git a/gensim/test/test_keras_integration.py b/gensim/test/test_keras_integration.py
--- a/gensim/test/test_keras_integration.py
+++ b/gensim/test/test_keras_integration.py
@@ -66,7 +66,7 @@ def testEmbeddingLayerCosineSim(self):
         keras_w2v_model = self.model_cos_sim
         keras_w2v_model_wv = keras_w2v_model.wv
 
-        embedding_layer = keras_w2v_model_wv.get_embedding_layer()
+        embedding_layer = keras_w2v_model_wv.get_keras_embedding()
 
         input_a = Input(shape=(1,), dtype='int32', name='input_a')
         input_b = Input(shape=(1,), dtype='int32', name='input_b')
@@ -135,7 +135,7 @@ def testEmbeddingLayer20NewsGroup(self):
         keras_w2v.build_vocab(texts_w2v)
         keras_w2v.train(texts, total_examples=keras_w2v.corpus_count, epochs=keras_w2v.iter)
         keras_w2v_wv = keras_w2v.wv
-        embedding_layer = keras_w2v_wv.get_embedding_layer()
+        embedding_layer = keras_w2v_wv.get_keras_embedding()
 
         # create a 1D convnet to solve our classification task
         sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
Repository: piskvorky/gensim

Issue:
FastText wrapper returns inconsistent dtypes
#### Description
`gensim.models.wrappers.FastText` returns inconsistent dtypes.

#### Steps/Code/Corpus to Reproduce
```python
from gensim.models.wrappers import FastText
embeds = FastText.load_fasttext_format(...)
```
For an existing word:
```python
embeds['the'].dtype == dtype('float32')
```
For an "imputed" word (missing from the vocabulary). The word embedding is computed as the sum of embedding for n-grams:

```python
embeds['ttttt'].dtype == dtype('float64')
```

The problem in `models/wrappers/fasttext.py::FastTextKeyedVectors.word_vec`. In the case of a missing word, the zero vector is initialised to be a 64-bit float array to which a bunch of 32-bit embeddings are added to. 

#### Versions
Linux-4.4.0-97-generic-x86_64-with-Ubuntu-16.04-xenial
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609]
NumPy 1.13.3
SciPy 0.19.1
gensim 3.0.1
FAST_VERSION 1



Discussion:

Resolving patch:
diff --git a/gensim/models/wrappers/fasttext.py b/gensim/models/wrappers/fasttext.py
--- a/gensim/models/wrappers/fasttext.py
+++ b/gensim/models/wrappers/fasttext.py
@@ -90,7 +90,7 @@ def word_vec(self, word, use_norm=False):
         if word in self.vocab:
             return super(FastTextKeyedVectors, self).word_vec(word, use_norm)
         else:
-            word_vec = np.zeros(self.syn0_ngrams.shape[1])
+            word_vec = np.zeros(self.syn0_ngrams.shape[1], dtype=np.float32)
             ngrams = compute_ngrams(word, self.min_n, self.max_n)
             ngrams = [ng for ng in ngrams if ng in self.ngrams]
             if use_norm:


Test patch:
diff --git a/gensim/test/test_fasttext_wrapper.py b/gensim/test/test_fasttext_wrapper.py
--- a/gensim/test/test_fasttext_wrapper.py
+++ b/gensim/test/test_fasttext_wrapper.py
@@ -337,6 +337,17 @@ def testHash(self):
         ft_hash = fasttext.ft_hash('word')
         self.assertEqual(ft_hash, 1788406269)
 
+    def testConsistentDtype(self):
+        """Test that the same dtype is returned for OOV words as for words in the vocabulary"""
+        vocab_word = 'night'
+        oov_word = 'wordnotpresentinvocabulary'
+        self.assertIn(vocab_word, self.test_model.wv.vocab)
+        self.assertNotIn(oov_word, self.test_model.wv.vocab)
+
+        vocab_embedding = self.test_model[vocab_word]
+        oov_embedding = self.test_model[oov_word]
+        self.assertEqual(vocab_embedding.dtype, oov_embedding.dtype)
+
 
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
Repository: piskvorky/gensim

Issue:
Update show_topics method from Mallet wrapper
Need to update https://github.com/RaRe-Technologies/gensim/blob/3.1.0/gensim/models/wrappers/ldamallet.py#L250 (replace `num_words` to `topn`) to prevent warning (report comes from https://groups.google.com/forum/#!topic/gensim/r3YIaqPRmzU)


Discussion:

Resolving patch:
diff --git a/gensim/models/wrappers/ldamallet.py b/gensim/models/wrappers/ldamallet.py
--- a/gensim/models/wrappers/ldamallet.py
+++ b/gensim/models/wrappers/ldamallet.py
@@ -258,7 +258,7 @@ def show_topics(self, num_topics=10, num_words=10, log=False, formatted=True):
             if formatted:
                 topic = self.print_topic(i, topn=num_words)
             else:
-                topic = self.show_topic(i, num_words=num_words)
+                topic = self.show_topic(i, topn=num_words)
             shown.append((i, topic))
             if log:
                 logger.info("topic #%i (%.3f): %s", i, self.alpha[i], topic)


Test patch:
Repository: piskvorky/gensim

Issue:
Pyro4.naming exception in `get_my_ip`
On a multi home nic computer, get_my_ip() is failing to use socket connection to nameserver to retrieve the external ip address to use. The code is proceeding to the default case to use hostname and to look up by ip address, which is currently resolving to virtualbox ethernet adapter that is not reachable from the dispatcher.

Specifically, the line 
`ns = Pyro4.naming.locateNS()`
is throwing exception 
`module 'Pyro4' has no attribute 'naming'`

when using Pyro4 version 4.62, thus get_my_ip() is defaulting to use
`result = socket.gethostbyname(socket.gethostname())`

The solution is to add 
`import Pyro4.naming`
Pyro4.naming exception in `get_my_ip`
On a multi home nic computer, get_my_ip() is failing to use socket connection to nameserver to retrieve the external ip address to use. The code is proceeding to the default case to use hostname and to look up by ip address, which is currently resolving to virtualbox ethernet adapter that is not reachable from the dispatcher.

Specifically, the line 
`ns = Pyro4.naming.locateNS()`
is throwing exception 
`module 'Pyro4' has no attribute 'naming'`

when using Pyro4 version 4.62, thus get_my_ip() is defaulting to use
`result = socket.gethostbyname(socket.gethostname())`

The solution is to add 
`import Pyro4.naming`


Discussion:

Resolving patch:
diff --git a/gensim/utils.py b/gensim/utils.py
--- a/gensim/utils.py
+++ b/gensim/utils.py
@@ -625,9 +625,9 @@ def get_my_ip():
     """
     import socket
     try:
-        import Pyro4
+        from Pyro4.naming import locateNS
         # we know the nameserver must exist, so use it as our anchor point
-        ns = Pyro4.naming.locateNS()
+        ns = locateNS()
         s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
         s.connect((ns._pyroUri.host, ns._pyroUri.port))
         result, port = s.getsockname()


Test patch:
Repository: piskvorky/gensim

Issue:
Move building documentation to CircleCI
Travis has problems with some "latex-related" packages, than needed for us for build documentation, for example `dvipng`, `texlive-latex-extra`, etc. 

The problem happens first time in #1714, where we need some additional stuff for png-rendering from latex (because we fix formulas).

Also, this related to really useful feature #1725 (than easier to implement based on `CircleCI`)




Discussion:

Resolving patch:
diff --git a/docs/src/conf.py b/docs/src/conf.py
--- a/docs/src/conf.py
+++ b/docs/src/conf.py
@@ -25,7 +25,7 @@
 
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ['sphinx.ext.autodoc', 'sphinxcontrib.napoleon']
+extensions = ['sphinx.ext.autodoc', 'sphinxcontrib.napoleon', 'sphinx.ext.imgmath']
 autoclass_content = "both"
 
 # Add any paths that contain templates here, relative to this directory.
diff --git a/gensim/topic_coherence/direct_confirmation_measure.py b/gensim/topic_coherence/direct_confirmation_measure.py
--- a/gensim/topic_coherence/direct_confirmation_measure.py
+++ b/gensim/topic_coherence/direct_confirmation_measure.py
@@ -21,7 +21,7 @@ def log_conditional_probability(segmented_topics, accumulator, with_std=False, w
     """
     This function calculates the log-conditional-probability measure
     which is used by coherence measures such as U_mass.
-    This is defined as: m_lc(S_i) = log[(P(W', W*) + e) / P(W*)]
+    This is defined as :math:`m_{lc}(S_i) = log \\frac{P(W', W^{*}) + \epsilon}{P(W^{*})}`.
 
     Args:
         segmented_topics (list): Output from the segmentation module of the segmented


Test patch:
Repository: piskvorky/gensim

Issue:
Wrong distance in Lexical entailment experiment for Poincare embeddings
The following line https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/poincare.py#L1522
should be:
```
return -1 * (1 + self.alpha * (norm_2 - norm_1)) * min_distance
```
otherwise `distance` is the last encountered distance, not the smallest one.


Discussion:
Thanks for report @octavian-ganea, looks like a typo (that generates bug here).
@jayantj can you confirm?
Yeah, that looks like a bug. Good catch.
Hi, Can I take this one?
@hachibaka of course, feel free to contribute!

Resolving patch:
diff --git a/gensim/models/poincare.py b/gensim/models/poincare.py
--- a/gensim/models/poincare.py
+++ b/gensim/models/poincare.py
@@ -1519,7 +1519,7 @@ def score_function(self, embedding, trie, term_1, term_2):
         assert min_term_1 is not None and min_term_2 is not None
         vector_1, vector_2 = embedding.word_vec(min_term_1), embedding.word_vec(min_term_2)
         norm_1, norm_2 = np.linalg.norm(vector_1), np.linalg.norm(vector_2)
-        return -1 * (1 + self.alpha * (norm_2 - norm_1)) * distance
+        return -1 * (1 + self.alpha * (norm_2 - norm_1)) * min_distance
 
     @staticmethod
     def find_matching_terms(trie, word):


Test patch:
Repository: piskvorky/gensim

Issue:
cannot import name 'LabeledSentence'

#### Description
LabeledSentence is not being imported from gensim.models.doc2vec. 

``from gensim.models.doc2vec import LabeledSentence``

the error I am getting is 
``cannot import name 'LabeledSentence'``



cannot import name 'LabeledSentence'

#### Description
LabeledSentence is not being imported from gensim.models.doc2vec. 

``from gensim.models.doc2vec import LabeledSentence``

the error I am getting is 
``cannot import name 'LabeledSentence'``





Discussion:
@thepurpleowl hello, thanks for the report, right now you can use 

`from gensim.models.deprecated.doc2vec import LabeledSentence` (FYI this is deprecated stuff),
 or better use 
`from gensim.models.doc2vec import TaggedDocument`


CC: @manneshiva 
@thepurpleowl hello, thanks for the report, right now you can use 

`from gensim.models.deprecated.doc2vec import LabeledSentence` (FYI this is deprecated stuff),
 or better use 
`from gensim.models.doc2vec import TaggedDocument`


CC: @manneshiva 

Resolving patch:
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -70,6 +70,7 @@
 from gensim.models.base_any2vec import BaseWordEmbeddingsModel
 from gensim.models.keyedvectors import Doc2VecKeyedVectors
 from types import GeneratorType
+from gensim.utils import deprecated
 
 logger = logging.getLogger(__name__)
 
@@ -247,6 +248,12 @@ def __str__(self):
         return '%s(%s, %s)' % (self.__class__.__name__, self.words, self.tags)
 
 
+# for compatibility
+@deprecated("Class will be removed in 4.0.0, use TaggedDocument instead")
+class LabeledSentence(TaggedDocument):
+    pass
+
+
 class Doctag(namedtuple('Doctag', 'offset, word_count, doc_count')):
     """A string document tag discovered during the initial vocabulary
     scan. (The document-vector equivalent of a Vocab object.)


Test patch:
Repository: piskvorky/gensim

Issue:
WikiCorpus scans corpus to determine vocabulary when an empty dictionary is provided
I have the following code:

```python
wiki = WikiCorpus(inp, dictionary={}, lemmatize=False)
```

When I use it, it tries to scan through the corpus to determine vocabulary, while a dictionary **is** provided.

It clearly violates what is being said in the documentation:
> dictionary (Dictionary, optional) – Dictionary, if not provided, this scans the corpus once, to determine its vocabulary (this needs really long time).


Discussion:

Resolving patch:
diff --git a/gensim/corpora/wikicorpus.py b/gensim/corpora/wikicorpus.py
--- a/gensim/corpora/wikicorpus.py
+++ b/gensim/corpora/wikicorpus.py
@@ -558,7 +558,11 @@ def __init__(self, fname, processes=None, lemmatize=utils.has_pattern(), diction
         self.token_min_len = token_min_len
         self.token_max_len = token_max_len
         self.lower = lower
-        self.dictionary = dictionary or Dictionary(self.get_texts())
+
+        if dictionary is None:
+            self.dictionary = Dictionary(self.get_texts())
+        else:
+            self.dictionary = dictionary
 
     def get_texts(self):
         """Iterate over the dump, yielding list of tokens for each article.


Test patch:
Repository: piskvorky/gensim

Issue:
Turn off support of Google Style docstrings
**All docstrings should be refactored first**

To prevent contributors from using Google Style docstrings, we need to set

`napoleon_google_docstring = False`,

[like explained here](https://samnicholls.net/2016/06/15/how-to-sphinx-readthedocs/).


Discussion:
Can I take this up @menshikh-iv @anotherbugmaster ?
@kaustubhhiware no, because we didn't finished with docstrings yet
@anotherbugmaster FYI, #1605 seems closed. Is this one up for grabs?
@jxub we'll make it after finishing all docstrings (that happens soon, I hope)

Resolving patch:
diff --git a/docs/src/conf.py b/docs/src/conf.py
--- a/docs/src/conf.py
+++ b/docs/src/conf.py
@@ -28,6 +28,8 @@
 extensions = ['sphinx.ext.autodoc', 'sphinxcontrib.napoleon', 'sphinx.ext.imgmath', 'sphinxcontrib.programoutput']
 autoclass_content = "both"
 
+napoleon_google_docstring = False  # Disable support for google-style docstring
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 


Test patch: