Repository: piskvorky/gensim

Issue:
BM25 does not support generator as corpus
``__init__`` method in BM25 class takes a "list of list of str" as corpus instead of a generator. More precisely, this is what it looks like right now:

```
def __init__(self, corpus):
        """
        Parameters
        ----------
        corpus : list of list of str
            Given corpus.
        """
        self.corpus_size = len(corpus)
        self.avgdl = 0
        self.doc_freqs = []
        self.idf = {}
        self.doc_len = []
        self._initialize(corpus)
```
As we know, considering a generator instead of a list would be great to handle large collections of documents that do not fit in memory.


Discussion:
@Witiko can you please have a look?

Models (any models) in Gensim not supporting streaming is definitely a bug.
I think the tricky problem here, using a generator as a collection of documents, is computing the average document length in the text collection. As I can see, in Python, there's no "proper" way of getting the "length" of a generator. A trivial (and not so efficiently) way could be the following (supposing `corpus` is a generator of lists):

```python
init = (0, 0)
total_sum, total_docs = reduce(lambda pair1, pair2: (pair1[0] + pair2[0], pair1[1] + pair2[1]), map(lambda d: (len(d), 1), corpus), init)
avgdl = total_sum/total_docs
```


We'd definitely want to avoid huge `reduce` / `map` (not Pythonic), but the code sounds trivial in any case.

The main question for me is, why does the current implementation ask for a `list` in the first place? What design choices lead to this? How did it pass our reviews? (if indeed true that streaming is not supported)

CC @fbarrios @menshikh-iv @horpto do you remember, can you help? Cheers.
@piskvorky There is no apparent reason why `corpus` can't be an iterable. Instead of initializing `self.corpus_size` to `len(corpus)` in `__init__`, `corpus_size` can be accumulated in the `for document in corpus` loop in the `_initialize` method. The `_initialize` method only assumes that `corpus` is iterable.

As for why this code was merged in the first place, I can't really tell. It has been in place since Gensim 3.7.2 (see ac7486a69da6ce5d9873216bad86afa1a10191fc). The corresponding pull request (#324) does not discuss the `BM25` class at all. It seems that the entire `gensim.summarization.bm25` module was originally designed to be just a private API.
Thanks @Witiko , makes sense.

@perezzini can you open a PR with a fix, as per @Witiko 's suggestion?
@piskvorky I'll make a PR as soon as possible!

Thanks for replying!

Resolving patch:
diff --git a/gensim/summarization/bm25.py b/gensim/summarization/bm25.py
--- a/gensim/summarization/bm25.py
+++ b/gensim/summarization/bm25.py
@@ -74,7 +74,7 @@ def __init__(self, corpus):
             Given corpus.
 
         """
-        self.corpus_size = len(corpus)
+        self.corpus_size = 0
         self.avgdl = 0
         self.doc_freqs = []
         self.idf = {}
@@ -86,6 +86,7 @@ def _initialize(self, corpus):
         nd = {}  # word -> number of documents with word
         num_doc = 0
         for document in corpus:
+            self.corpus_size += 1
             self.doc_len.append(len(document))
             num_doc += len(document)
 


Test patch:
diff --git a/gensim/test/test_BM25.py b/gensim/test/test_BM25.py
--- a/gensim/test/test_BM25.py
+++ b/gensim/test/test_BM25.py
@@ -24,6 +24,15 @@ def test_max_match_with_itself(self):
             predicted = doc_weights[index]
             self.assertAlmostEqual(expected, predicted)
 
+    def test_with_generator(self):
+        """ Check above function with input as generator """
+        text_gen = (i for i in common_texts)
+        weights = get_bm25_weights(text_gen)
+        for index, doc_weights in enumerate(weights):
+            expected = max(doc_weights)
+            predicted = doc_weights[index]
+            self.assertAlmostEqual(expected, predicted)
+
     def test_nonnegative_weights(self):
         """ All the weights for a partiular document should be non negative """
         weights = get_bm25_weights(common_texts)