Repository: piskvorky/gensim

Issue:
Unreasonable Query Result
<!--
**IMPORTANT**:

- Use the [Gensim mailing list](https://groups.google.com/forum/#!forum/gensim) to ask general or usage questions. Github issues are only for bug reports.
- Check [Recipes&FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) first for common answers.

Github bug reports that do not include relevant information and context will be closed without an answer. Thanks!
-->

#### Problem description

      **The query result seems not correct. The code is self-explained. Thank you!**

#### Steps/code/corpus to reproduce

Include full tracebacks, logs and datasets if necessary. Please keep the examples minimal ("minimal reproducible example").

```
from gensim.summarization.bm25 import BM25, get_bm25_weights


text1 = "A constellation is a group of stars that are considered to form imaginary outlines or meaningful patterns on the celestial sphere."
text2 = "The 88 modern constellations are formally defined regions of the sky together covering the entire celestial sphere."
text = [text1, text2]

corpus = [text1.split(" "), text2.split(" ")]
print(f'corpus: {corpus}')

query = text2.split(" ")

bm25 = BM25(corpus)
scores = bm25.get_scores(query)
scores = [(s, i) for i, s in enumerate(scores)]
scores.sort(key=lambda t: t[0], reverse=True)
print(f'scores:         {scores}')

for s, idx in scores:
  print(f'{s}\t{idx}: {text[idx]}')
```

Output:

```
-0.3601521710456333         0: A constellation is a group of stars that are considered to form imaginary outlines or meaningful patterns on the celestial sphere.
-0.44989406787023367     1: The 88 modern constellations are formally defined regions of the sky together covering the entire celestial sphere.
```

#### Versions

Please provide the output of:

```python
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import gensim; print("gensim", gensim.__version__)
from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
```

Output:
```
macOS-10.14.6-x86_64-i386-64bit
Python 3.8.0 (default, Nov  6 2019, 15:49:01)
[Clang 4.0.1 (tags/RELEASE_401/final)]
NumPy 1.17.4
SciPy 1.3.3
gensim 3.8.1
FAST_VERSION 0
```



Discussion:
@Witiko could you please have a look?
@JiaqiLiu Your issue seems to be due to the inverse document frequency term in the BM25 formula. Since the corpus is so small (only 2 documents), most words have zero or negative inverse document frequency. Can you try using a larger corpus and see if the results become more reasonable?
Hi Witiko, thank you for your reply. You are right, and the result is correct when trying larger corpus.
Thank you!
Can you summarize what the issue is (was)? Why should a small corpus give incorrect / useless results because of inverse document frequencies? Sounds fishy to me.
@piskvorky BM25 is a fairly simple model, but still sufficiently complex to surprise. The IDF term in [the BM25 formula](https://en.wikipedia.org/wiki/Okapi_BM25) is negative when a word occurs in more than half the corpus. In general, this penalizes stopwords. However, in this example, the IDF for all words is either zero (a word occurs in one document) or negative (a word occurs in both documents). Hence the negative scores.
Understood, but is that a desirable result?

If the formula breaks down for some inputs, isn't it better to raise an error? Or is such score "expected", interpretable?

Resolving patch:
diff --git a/gensim/summarization/bm25.py b/gensim/summarization/bm25.py
--- a/gensim/summarization/bm25.py
+++ b/gensim/summarization/bm25.py
@@ -37,6 +37,7 @@
 """
 
 
+import logging
 import math
 from six import iteritems
 from six.moves import range
@@ -48,6 +49,8 @@
 PARAM_B = 0.75
 EPSILON = 0.25
 
+logger = logging.getLogger(__name__)
+
 
 class BM25(object):
     """Implementation of Best Matching 25 ranking function.
@@ -116,6 +119,13 @@ def _initialize(self, corpus):
                 negative_idfs.append(word)
         self.average_idf = float(idf_sum) / len(self.idf)
 
+        if self.average_idf < 0:
+            logger.warning(
+                'Average inverse document frequency is less than zero. Your corpus of {} documents'
+                ' is either too small or it does not originate from natural text. BM25 may produce'
+                ' unintuitive results.'.format(self.corpus_size)
+            )
+
         eps = EPSILON * self.average_idf
         for word in negative_idfs:
             self.idf[word] = eps


Test patch: