internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: piskvorky/gensim

Issue:
ZeroDivisionError: float division by zero
Getting error :  ZeroDivisionError: float division by zero

https://github.com/RaRe-Technologies/gensim/blob/9481915915bf61aa6e4e719a2f26d509677e6779/gensim/summarization/pagerank_weighted.py#L53


![error](https://user-images.githubusercontent.com/11848354/40821543-3283e832-6585-11e8-9a66-3a4d8ba95eeb.JPG)

ZeroDivisionError: float division by zero
Getting error :  ZeroDivisionError: float division by zero

https://github.com/RaRe-Technologies/gensim/blob/9481915915bf61aa6e4e719a2f26d509677e6779/gensim/summarization/pagerank_weighted.py#L53


![error](https://user-images.githubusercontent.com/11848354/40821543-3283e832-6585-11e8-9a66-3a4d8ba95eeb.JPG)



Discussion:
Hi @anujgupta82, please give us more information about this problem

- OS version
- Python version
- Gensim version
- Code example for reproducing this error (with all needed data)
ping @anujgupta82 
Hi @anujgupta82, please give us more information about this problem

- OS version
- Python version
- Gensim version
- Code example for reproducing this error (with all needed data)
ping @anujgupta82 

Resolving patch:
diff --git a/gensim/summarization/keywords.py b/gensim/summarization/keywords.py
--- a/gensim/summarization/keywords.py
+++ b/gensim/summarization/keywords.py
@@ -512,6 +512,9 @@ def keywords(text, ratio=0.2, words=None, split=False, scores=False, pos_filter=
 
     _remove_unreachable_nodes(graph)
 
+    if not graph.edges():
+        return _format_results([], [], split, scores)
+
     # Ranks the tokens using the PageRank algorithm. Returns dict of lemma -> score
     pagerank_scores = _pagerank(graph)
 


Test patch:
diff --git a/gensim/test/test_keywords.py b/gensim/test/test_keywords.py
--- a/gensim/test/test_keywords.py
+++ b/gensim/test/test_keywords.py
@@ -95,6 +95,12 @@ def test_text_keywords_with_small_graph(self):
         kwds = keywords(text, words=1, split=True)
         self.assertTrue(len(kwds))
 
+    def test_text_keywords_without_graph_edges(self):
+        # regression test, we get graph with no edges on this text
+        text = 'Sitio construcción. Estaremos línea.'
+        kwds = keywords(text, deacc=False, scores=True)
+        self.assertFalse(len(kwds))
+
 
 if __name__ == '__main__':
     logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
Repository: piskvorky/gensim

Issue:
aliasing chunkize to chunkize_serial warning, on Windows
On Windows, I have it that after initial installation, when importing gensim for the first time in python (Jupyter Notebook) we get:

> UserWarning: detected Windows; aliasing chunkize to chunkize_serial 
    warnings.warn("detected Windows; aliasing chunkize to chunkize_serial")

I've been unable to gather what trouble or missing functionality should we expect, the warning is rather opaque, yet I assume it is there in the code to tell us something, as API users. Which is my lame excuse for having an issue here, so that there's an official comment on what to understand from this warning.


Discussion:
Yes, that warning is rather opaque, sorry about that.

Some algorithms in Gensim (mostly the distributed/parallelized versions) call a function called `chunkize`, which splits an input stream of records into batches. It works in a streaming manner (lazy batch iteration). `chunkize` has an optional parameter that can actively prepare and buffer data batches in advance: not quite lazy, but also not eager (buffers a limited fixed number of batches in advance).

This optional functionality is not available on Windows, because it uses multi-processing and is slower. So on Windows, only `chunkize_serial` is available (no buffering). It's aliased to `chunkize` for API compatibility reason.

This is a rather technical point, related to performance on slow I/O input streams on Windows. You can probably ignore it.

A PR that clarifies the warning will be very welcome! We should probably only emit that warning when that situation actually happens (`chunkize` called with `maxsize > 0` on Windows), rather than always on import.

Resolving patch:
diff --git a/gensim/utils.py b/gensim/utils.py
--- a/gensim/utils.py
+++ b/gensim/utils.py
@@ -1215,8 +1215,6 @@ def run(self):
 
 
 if os.name == 'nt':
-    warnings.warn("detected Windows; aliasing chunkize to chunkize_serial")
-
     def chunkize(corpus, chunksize, maxsize=0, as_numpy=False):
         """Split `corpus` into fixed-sized chunks, using :func:`~gensim.utils.chunkize_serial`.
 
@@ -1237,6 +1235,8 @@ def chunkize(corpus, chunksize, maxsize=0, as_numpy=False):
             "chunksize"-ed chunks of elements from `corpus`.
 
         """
+        if maxsize > 0:
+            warnings.warn("detected Windows; aliasing chunkize to chunkize_serial")
         for chunk in chunkize_serial(corpus, chunksize, as_numpy=as_numpy):
             yield chunk
 else:


Test patch:
Repository: piskvorky/gensim

Issue:
Phrases documentation for threshold argument is misleading
https://github.com/RaRe-Technologies/gensim/blob/37e49971efa74310b300468a5b3cf531319c6536/gensim/models/phrases.py#L252-L255

It feels to me like this should have said the opposite: "**Heavily** depends on concrete scoring-function" rather than "**Hardly** depends on concrete socring-function".

For example, if you choose npmi instead of the default, the threshold has to between -1 and 1, which makes the default (10.0) make no sense. The current documentation suggests that 10.0 will be OK.


Discussion:
I suspect the intended meaning was, as you surmise, "heavily depends"... and the docs around the 'npmi' option should make that acceptable range clear. 
**Hardly** has already been replaced by **Heavily**. Would it be a good idea to add the **range** of the score under `returns` in the documentation of the scoring function?

Resolving patch:
diff --git a/gensim/models/phrases.py b/gensim/models/phrases.py
--- a/gensim/models/phrases.py
+++ b/gensim/models/phrases.py
@@ -676,6 +676,11 @@ def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count
     corpus_word_count : int
         Not used in this particular scoring technique.
 
+    Returns
+    -------
+    float
+        Score for given bi-gram, greater than or equal to 0.
+
     Notes
     -----
     Formula: :math:`\frac{(bigram\_count - min\_count) * len\_vocab }{ (worda\_count * wordb\_count)}`.
@@ -703,6 +708,11 @@ def npmi_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, co
     corpus_word_count : int
         Total number of words in the corpus.
 
+    Returns
+    -------
+    float
+        Score for given bi-gram, in the range -1 to 1.
+
     Notes
     -----
     Formula: :math:`\frac{ln(prop(word_a, word_b) / (prop(word_a)*prop(word_b)))}{ -ln(prop(word_a, word_b)}`,


Test patch:
Repository: piskvorky/gensim

Issue:
Unimportant 'Warning' in phrases.py
Upon 1st use of `PhrasesTransformation.__getitem__()`, there's a UserWarning issued "For a faster implementation, use the gensim.models.phrases.Phraser class". See:

https://github.com/RaRe-Technologies/gensim/blob/7fca36f3eb0df70e5f823e2fb4852193b4ad7393/gensim/models/phrases.py#L655

It's kind of excessive/alarming to display a "warning" to deliver a low-severity usage recommendation - thus leading to user inquiries like this at SO: https://stackoverflow.com/questions/54112456/warnings-in-gensim-phrasestransformer

I suggest no "warning" here, as there's nothing wrong or especially concerning. Subtle efficiency tips should go elsewhere.




Discussion:

Resolving patch:
diff --git a/gensim/models/phrases.py b/gensim/models/phrases.py
--- a/gensim/models/phrases.py
+++ b/gensim/models/phrases.py
@@ -39,7 +39,6 @@
 import sys
 import os
 import logging
-import warnings
 from collections import defaultdict
 import functools as ft
 import itertools as it
@@ -658,8 +657,6 @@ def __getitem__(self, sentence):
             ...     pass
 
         """
-        warnings.warn("For a faster implementation, use the gensim.models.phrases.Phraser class")
-
         return _sentence2token(self, sentence)
 
 


Test patch:
Repository: piskvorky/gensim

Issue:
fastText  fixes in 3.7 break compatibility with old models
Recent fixes to Gensim's fastText implementation introduced in #2313 are great. Unfortunately, they also break compatibility with fastText models trained by older Gensim versions - if the models are stored as a KeyedVectors() object. One can load such a model, but as soon as you try to do anything useful (like `most_similar()`, etc), it fails, because the `compatible_hash` attribute is missing.
If this attribute is added manually after the loading, everything goes fine.

#### Steps/Code/Corpus to Reproduce
```
import gensim

model = gensim.models.KeyedVectors.load(ANY_KEYED_VECTORS_FASTTEXT_MODEL)
model.most_similar(positive=ANY_WORD)
```
#### Expected Results
The `compatible_hash` attribute is automatically assigned the False value on load, and the model works as before.

#### Actual Results
```
/usr/local/lib/python3.5/dist-packages/gensim/models/keyedvectors.py in word_vec(self, word, use_norm)
   2057 
   2058         """
-> 2059         hash_fn = _ft_hash if self.compatible_hash else _ft_hash_broken
   2060 
   2061         if word in self.vocab:

AttributeError: 'FastTextKeyedVectors' object has no attribute 'compatible_hash'

```

#### Versions
Linux-4.15.0-43-generic-x86_64-with-LinuxMint-18.3-sylvia
Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609]
NumPy 1.14.5
SciPy 1.1.0
gensim 3.7.0
FAST_VERSION 1


Discussion:

Resolving patch:
diff --git a/gensim/models/fasttext.py b/gensim/models/fasttext.py
--- a/gensim/models/fasttext.py
+++ b/gensim/models/fasttext.py
@@ -841,7 +841,7 @@ def load(cls, *args, **kwargs):
 
             if not hasattr(model.wv, 'compatible_hash'):
                 logger.warning(
-                    "This older model was trained with a buggy hash function.  ",
+                    "This older model was trained with a buggy hash function.  "
                     "The model will continue to work, but consider training it "
                     "from scratch."
                 )


Test patch:
Repository: piskvorky/gensim

Issue:
Off-by-one counting error in TFIDF
Building a TFIDF model outputs this in the log:

```
INFO - 2019-02-06 16:09:54,670 - resulting dictionary: Dictionary(10000 unique tokens: [u'writings', u'foul', u'prefix', u'woods', u'hanging']...)
INFO - 2019-02-06 16:10:01,718 - collecting document frequencies
INFO - 2019-02-06 16:10:01,718 - PROGRESS: processing document #0
INFO - 2019-02-06 16:10:02,162 - calculating IDF weights for 1701 documents and 9999 features (2220559 matrix non-zeros)
```

Note the "9999 features", despite the dictionary having 10000 features. (any dictionary and any number of features -- the actual values don't matter, it's always off-by-one).

This seems to be a bug introduced in [this refactoring](https://github.com/RaRe-Technologies/gensim/commit/af01bc702f7ad48ed9ecacb256d9f6f7da6d3bc4#diff-896942304f9417b1b8842823b19d7431R128).

Not sure if this is an isolated problem or what other places are affected by similar "optimizations".


Discussion:

Resolving patch:
diff --git a/gensim/models/tfidfmodel.py b/gensim/models/tfidfmodel.py
--- a/gensim/models/tfidfmodel.py
+++ b/gensim/models/tfidfmodel.py
@@ -390,10 +390,9 @@ def initialize(self, corpus):
         self.num_nnz = numnnz
         self.dfs = dfs
         # and finally compute the idf weights
-        n_features = max(dfs) if dfs else 0
         logger.info(
             "calculating IDF weights for %i documents and %i features (%i matrix non-zeros)",
-            self.num_docs, n_features, self.num_nnz
+            self.num_docs, max(dfs.keys()) + 1 if dfs else 0, self.num_nnz
         )
         self.idfs = precompute_idfs(self.wglobal, self.dfs, self.num_docs)
 


Test patch:
Repository: piskvorky/gensim

Issue:
replace 'sum(time_slice) == num_topics' to 'sum(time_slice) == num_documents'
https://github.com/RaRe-Technologies/gensim/blob/355ecc68a6ccb07f38418e8c80784b70aac84442/gensim/models/ldaseqmodel.py#L75

As the comments said 'Number of documents in each time-slice', maybe the num_documents is the true meaning for sum(time_slice) ??


Discussion:
СС @bhargavvader 

Resolving patch:
diff --git a/gensim/models/ldaseqmodel.py b/gensim/models/ldaseqmodel.py
--- a/gensim/models/ldaseqmodel.py
+++ b/gensim/models/ldaseqmodel.py
@@ -79,7 +79,7 @@ def __init__(self, corpus=None, time_slice=None, id2word=None, alphas=0.01, num_
         time_slice : list of int, optional
             Number of documents in each time-slice. Each time slice could for example represent a year's published
             papers, in case the corpus comes from a journal publishing over multiple years.
-            It is asummed that `sum(time_slice) == num_topics`.
+            It is assumed that `sum(time_slice) == num_documents`.
         id2word : dict of (int, str), optional
             Mapping from word IDs to words. It is used to determine the vocabulary size, as well as for
             debugging and topic printing.


Test patch:
Repository: piskvorky/gensim

Issue:
Incomplete DTM docs
In https://radimrehurek.com/gensim/models/wrappers/dtmmodel.html#gensim.models.wrappers.dtmmodel.DtmModel : 

```
top_chain_var (int, optional) – Hyperparameter that affects.
```




Discussion:

Resolving patch:
diff --git a/gensim/models/wrappers/dtmmodel.py b/gensim/models/wrappers/dtmmodel.py
--- a/gensim/models/wrappers/dtmmodel.py
+++ b/gensim/models/wrappers/dtmmodel.py
@@ -97,8 +97,10 @@ def __init__(self, dtm_path, corpus=None, time_slices=None, mode='fit', model='d
              Max em optimization iterations in LDA.
         alpha : int, optional
             Hyperparameter that affects sparsity of the document-topics for the LDA models in each timeslice.
-        top_chain_var : int, optional
-            Hyperparameter that affects.
+        top_chain_var : float, optional
+            This hyperparameter controls one of the key aspect of topic evolution which is the speed at which
+            these topics evolve. A smaller top_chain_var leads to similar word distributions over multiple timeslice.
+
         rng_seed : int, optional
              Random seed.
         initialize_lda : bool, optional


Test patch:
Repository: piskvorky/gensim

Issue:
Python 3.7.3, gensim 3.8.1: UnboundLocalError: local variable 'doc_no2' referenced before assignment
Client code:
`    model = LogEntropyModel(corpus=data_corpus, normalize=True)
`
Referenced code:
https://github.com/RaRe-Technologies/gensim/blob/44ea7931c916349821aa1c717fbf7e90fb138297/gensim/models/logentropy_model.py#L115

Exception thrown:

```
  File "/anaconda3/lib/python3.7/site-packages/gensim/models/logentropy_model.py", line 76, in __init__
    self.initialize(corpus)
  File "/anaconda3/lib/python3.7/site-packages/gensim/models/logentropy_model.py", line 115, in initialize
    if doc_no2 != doc_no:
UnboundLocalError: local variable 'doc_no2' referenced before assignment

```


Discussion:
Looks like a programming bug to me. `doc_no2` should be initialized to zero, in case the corpus/generator is empty (which is what this block is actually testing).

Thanks for reporting.
I can try to patch it. I'll update and cut a PR when ready. Cheers!

Resolving patch:
diff --git a/gensim/models/logentropy_model.py b/gensim/models/logentropy_model.py
--- a/gensim/models/logentropy_model.py
+++ b/gensim/models/logentropy_model.py
@@ -108,6 +108,9 @@ def initialize(self, corpus):
             self.n_docs, len(glob_freq), self.n_words
         )
         logger.debug('iterating over corpus')
+
+        # initialize doc_no2 index in case corpus is empty
+        doc_no2 = 0
         for doc_no2, bow in enumerate(corpus):
             for key, freq in bow:
                 p = (float(freq) / glob_freq[key]) * math.log(float(freq) / glob_freq[key])


Test patch:
diff --git a/gensim/test/test_logentropy_model.py b/gensim/test/test_logentropy_model.py
--- a/gensim/test/test_logentropy_model.py
+++ b/gensim/test/test_logentropy_model.py
@@ -19,9 +19,23 @@
 
 
 class TestLogEntropyModel(unittest.TestCase):
+    TEST_CORPUS = [[(1, 1.0)], [], [(0, 0.5), (2, 1.0)], []]
+
     def setUp(self):
         self.corpus_small = MmCorpus(datapath('test_corpus_small.mm'))
         self.corpus_ok = MmCorpus(datapath('test_corpus_ok.mm'))
+        self.corpus_empty = []
+
+    def test_generator_fail(self):
+        """Test creating a model using a generator as input; should fail."""
+        def get_generator(test_corpus=TestLogEntropyModel.TEST_CORPUS):
+            for test_doc in test_corpus:
+                yield test_doc
+        self.assertRaises(ValueError, logentropy_model.LogEntropyModel, corpus=get_generator())
+
+    def test_empty_fail(self):
+        """Test creating a model using an empty input; should fail."""
+        self.assertRaises(ValueError, logentropy_model.LogEntropyModel, corpus=self.corpus_empty)
 
     def testTransform(self):
         # create the transformation model
Repository: piskvorky/gensim

Issue:
collections.Mapping deprecated, to be removed in py3.9
#### Problem description

`python -Wdefault -c 'from gensim.corpora import dictionary'`

reports

```
gensim/corpora/dictionary.py:11: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  from collections import Mapping, defaultdict
```

expected no warning.


#### Versions

Linux-5.4.10-200.fc31.x86_64-x86_64-with-fedora-31-Thirty_One
Python 3.7.6 (default, Dec 19 2019, 22:52:49) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]
NumPy 1.18.1
SciPy 1.4.1
gensim 3.8.1
FAST_VERSION 1


Discussion:

Resolving patch:
diff --git a/gensim/corpora/dictionary.py b/gensim/corpora/dictionary.py
--- a/gensim/corpora/dictionary.py
+++ b/gensim/corpora/dictionary.py
@@ -8,7 +8,8 @@
 
 from __future__ import with_statement
 
-from collections import Mapping, defaultdict
+from collections import defaultdict
+from collections.abc import Mapping
 import sys
 import logging
 import itertools
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -70,7 +70,8 @@
 except ImportError:
     from Queue import Queue  # noqa:F401
 
-from collections import namedtuple, defaultdict, Iterable
+from collections import namedtuple, defaultdict
+from collections.abc import Iterable
 from timeit import default_timer
 
 from numpy import zeros, float32 as REAL, empty, ones, \
diff --git a/gensim/models/fasttext.py b/gensim/models/fasttext.py
--- a/gensim/models/fasttext.py
+++ b/gensim/models/fasttext.py
@@ -285,7 +285,7 @@
 import numpy as np
 from numpy import ones, vstack, float32 as REAL
 import six
-from collections import Iterable
+from collections.abc import Iterable
 
 import gensim.models._fasttext_bin
 


Test patch:
diff --git a/gensim/test/test_corpora_dictionary.py b/gensim/test/test_corpora_dictionary.py
--- a/gensim/test/test_corpora_dictionary.py
+++ b/gensim/test/test_corpora_dictionary.py
@@ -7,8 +7,7 @@
 Unit tests for the `corpora.Dictionary` class.
 """
 
-
-from collections import Mapping
+from collections.abc import Mapping
 from itertools import chain
 import logging
 import unittest
Repository: piskvorky/gensim

Issue:
xml.etree.cElementTree was deprecated and removed in Python 3.9 in favor of ElementTree
#### Problem description

xml.etree.cElementTree was deprecated and removed in Python 3.9 in favor of ElementTree

Ref : https://github.com/python/cpython/pull/19108

#### Versions

Python 3.9

I will raise a PR for this issue.


Discussion:

Resolving patch:
diff --git a/gensim/corpora/wikicorpus.py b/gensim/corpora/wikicorpus.py
--- a/gensim/corpora/wikicorpus.py
+++ b/gensim/corpora/wikicorpus.py
@@ -25,8 +25,12 @@
 import re
 import signal
 from pickle import PicklingError
-from xml.etree.cElementTree import \
-    iterparse  # LXML isn't faster, so let's go with the built-in solution
+# LXML isn't faster, so let's go with the built-in solution
+try:
+    from xml.etree.cElementTree import iterparse
+except ImportError:
+    from xml.etree.ElementTree import iterparse
+
 
 from gensim import utils
 # cannot import whole gensim.corpora, because that imports wikicorpus...
diff --git a/gensim/scripts/segment_wiki.py b/gensim/scripts/segment_wiki.py
--- a/gensim/scripts/segment_wiki.py
+++ b/gensim/scripts/segment_wiki.py
@@ -61,7 +61,10 @@
 import multiprocessing
 import re
 import sys
-from xml.etree import cElementTree
+try:
+    from xml.etree import cElementTree as ET
+except ImportError:
+    from xml.etree import ElementTree as ET
 from functools import partial
 
 from gensim.corpora.wikicorpus import IGNORED_NAMESPACES, WikiCorpus, filter_wiki, find_interlinks, get_namespace, utils
@@ -183,7 +186,7 @@ def extract_page_xmls(f):
         XML strings for page tags.
 
     """
-    elems = (elem for _, elem in cElementTree.iterparse(f, events=("end",)))
+    elems = (elem for _, elem in ET.iterparse(f, events=("end",)))
 
     elem = next(elems)
     namespace = get_namespace(elem.tag)
@@ -192,7 +195,7 @@ def extract_page_xmls(f):
 
     for elem in elems:
         if elem.tag == page_tag:
-            yield cElementTree.tostring(elem)
+            yield ET.tostring(elem)
             # Prune the element tree, as per
             # http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
             # except that we don't need to prune backlinks from the parent
@@ -221,7 +224,7 @@ def segment(page_xml, include_interlinks=False):
         (Optionally) [(interlink_article, interlink_text), ...]).
 
     """
-    elem = cElementTree.fromstring(page_xml)
+    elem = ET.fromstring(page_xml)
     filter_namespaces = ('0',)
     namespace = get_namespace(elem.tag)
     ns_mapping = {"ns": namespace}


Test patch:
Repository: piskvorky/gensim

Issue:
Change parameter used in dtm_coherence() (in DTM wrapper) to avoid persistent warning
Within the DTM wrapper, using `dtm_coherence()` always produces a warning:

> "The parameter `num_words` is deprecated, will be removed in 4.0.0, use `topn` instead."

Although the function works, obviously the intention is to deprecate the parameter at some point. 

This can be tracked back to `show_topic()`, which is where `num_words` has switched to `topn`. It is a very simple fix to just change the parameter used in the `dtm_coherence()` function accordingly. 

PR incoming with fix. 


Discussion:

Resolving patch:
diff --git a/gensim/models/wrappers/dtmmodel.py b/gensim/models/wrappers/dtmmodel.py
--- a/gensim/models/wrappers/dtmmodel.py
+++ b/gensim/models/wrappers/dtmmodel.py
@@ -604,7 +604,7 @@ def dtm_coherence(self, time, num_words=20):
         """
         coherence_topics = []
         for topic_no in range(0, self.num_topics):
-            topic = self.show_topic(topicid=topic_no, time=time, num_words=num_words)
+            topic = self.show_topic(topicid=topic_no, time=time, topn=num_words)
             coherence_topic = []
             for prob, word in topic:
                 coherence_topic.append(word)


Test patch: