Repository: castorini/pyserini

Issue:
Better error message when index doesn't store doc vectors to support RM3 expansion
This allows us to use RM3 query expansion.


Discussion:
In more detail:

```python -m pyserini.search \
    --topics dl19-passage \
    --index msmarco-passage \
    --output run.dl19-passage.txt \
    --rm3
```

Gives:

```
java.lang.NullPointerException
        at io.anserini.rerank.lib.Rm3Reranker.createdFeatureVector(Rm3Reranker.java:194)
        at io.anserini.rerank.lib.Rm3Reranker.estimateRelevanceModel(Rm3Reranker.java:151)
        at io.anserini.rerank.lib.Rm3Reranker.rerank(Rm3Reranker.java:83)
        at io.anserini.rerank.RerankerCascade.run(RerankerCascade.java:64)
        at io.anserini.search.SimpleSearcher.search(SimpleSearcher.java:594)
        at io.anserini.search.SimpleSearcher.search(SimpleSearcher.java:546)
```

At the very least, we should have better error messages...
Hello, I spent some time looking into this and can see where the traceback comes from.

https://github.com/castorini/pyserini/blob/a052748ea669125a129138fd2f53402a423b14bb/pyserini/search/lucene/_searcher.py#L131

Which calls this:

https://github.com/castorini/anserini/blob/2e8e9fdfedc1b81522d9e22a805e08e6a7105762/src/main/java/io/anserini/search/SimpleSearcher.java#L583

My intention was to add a test `search` call at the point where RM3 is set which is:

https://github.com/castorini/pyserini/blob/a052748ea669125a129138fd2f53402a423b14bb/pyserini/search/lucene/_searcher.py#L231

This way I should be able to add the custom error message during error handling right at the point where RM3 is being set. Something like:

```
self.object.setRM3(fb_terms, fb_docs, original_query_weight, rm3_output_query, rm3_filter_terms)
i = self.object.search("Test", 1)
# find someway to identify the error here and raise an error with a better error message
# if i has error:
#  raise Index Without DocVectors Don't Support RM3 Expansion
```

However, I noticed:

1. the error does not seem to cause the program to crash
2. I wasn't able to use a try and except to catch the error
3. I couldn't get the error using `traceback`

So, how do I go about triggering something in the actual Python code when something goes wrong in the Java part? Should we be handling this in Pyserini or is it better handled in Anserini?
Also, am I approaching this the wrong way?
> This way I should be able to add the custom error message during error handling right at the point where RM3 is being set. Something like:
> 
> ```
> self.object.setRM3(fb_terms, fb_docs, original_query_weight, rm3_output_query, rm3_filter_terms)
> i = self.object.search("Test", 1)
> # find someway to identify the error here and raise an error with a better error message
> # if i has error:
> #  raise Index Without DocVectors Don't Support RM3 Expansion
> ```

Yup, I was thinking something along these lines. At the point of `setRM3`, determine if RM3 can be successfully executed. If no, then print a warning and the method should be a no-op.

See if there's a way you can introspect the index (i.e., directly find out of doc vectors are stored)... instead of trying a test query and catching an exception.

Resolving patch:
diff --git a/pyserini/search/lucene/_searcher.py b/pyserini/search/lucene/_searcher.py
--- a/pyserini/search/lucene/_searcher.py
+++ b/pyserini/search/lucene/_searcher.py
@@ -228,7 +228,10 @@ def set_rm3(self, fb_terms=10, fb_docs=10, original_query_weight=float(0.5), rm3
         rm3_filter_terms: bool
             Whether to remove non-English terms.
         """
-        self.object.setRM3(fb_terms, fb_docs, original_query_weight, rm3_output_query, rm3_filter_terms)
+        if self.object.reader.getTermVectors(0):
+            self.object.setRM3(fb_terms, fb_docs, original_query_weight, rm3_output_query, rm3_filter_terms)
+        else:
+            raise TypeError("RM3 is not supported for indexes without document vectors.")
 
     def unset_rm3(self):
         """Disable RM3 query expansion."""


Test patch:
diff --git a/tests/test_search.py b/tests/test_search.py
--- a/tests/test_search.py
+++ b/tests/test_search.py
@@ -42,6 +42,20 @@ def setUp(self):
 
         self.searcher = LuceneSearcher(f'{self.index_dir}lucene-index.cacm')
 
+        # Create index without document vectors
+        # The current directory depends on if you're running inside an IDE or from command line.
+        curdir = os.getcwd()
+        if curdir.endswith('tests'):
+            corpus_path = '../tests/resources/sample_collection_json'
+        else:
+            corpus_path = 'tests/resources/sample_collection_json'
+        self.no_vec_index_dir = 'no_vec_index'
+        cmd1 = f'python -m pyserini.index.lucene -collection JsonCollection ' + \
+               f'-generator DefaultLuceneDocumentGenerator ' + \
+               f'-threads 1 -input {corpus_path} -index {self.no_vec_index_dir}'
+        os.system(cmd1)
+        self.no_vec_searcher = LuceneSearcher(self.no_vec_index_dir)
+
     def test_basic(self):
         self.assertTrue(self.searcher.get_similarity().toString().startswith('BM25'))
 
@@ -220,6 +234,9 @@ def test_rm3(self):
         self.assertEqual(hits[9].docid, 'CACM-1457')
         self.assertAlmostEqual(hits[9].score, 1.43700, places=5)
 
+        with self.assertRaises(TypeError):
+            self.no_vec_searcher.set_rm3()
+
     def test_doc_int(self):
         # The doc method is overloaded: if input is int, it's assumed to be a Lucene internal docid.
         doc = self.searcher.doc(1)
@@ -283,6 +300,7 @@ def tearDown(self):
         self.searcher.close()
         os.remove(self.tarball_name)
         shutil.rmtree(self.index_dir)
+        shutil.rmtree(self.no_vec_index_dir)
 
 
 if __name__ == '__main__':