Repository: deepset-ai/haystack

Issue:
`top_k` in `ElasticsearchFilterOnlyRetriever` has no effect when using `ElasticsearchDocumentStore`
**Describe the bug**
Using `top_k` with `ElasticsearchFilterOnlyRetriever` has no effect because `ElasticsearchDocumentStore.query` does not consider `top_k` if `query is None`. It seems the default number returned is `10`, which happens to be the default `top_k`.

**Expected behavior**
If there are more than `top_k` documents after applying filters, then `ElasticsearchFilterOnlyRetriever.retrieve` should return `top_k` documents.
If there are less than `top_k` documents after applying filters, then `ElasticsearchFilterOnlyRetriever.retrieve` should return all documents.

**Cause**
The `if query is None` branch does not apply `body["size"] = str(top_k)`, hence why `top_k` has no effect.
https://github.com/deepset-ai/haystack/blob/692cde11e736884cbd1a5f99416cee94c4f0921b/haystack/document_stores/elasticsearch.py#L1053-L1059

**Happy to implement this and make a PR if I get the go-ahead :)**


Discussion:
Hi @adri1wald thanks for pointing out this issue and great to see that you already suggest a fix. 👍 I think the reason why the "Naive retrieval without BM25, only filtering" part of the code doesn't use the `top_k` parameter is that the returned documents have no relevance scores and thus cannot be sorted. Therefore, it is not well defined which of the relevant documents are returned if there are more than `top_k`.
I would say the expected behavior of the ElasticsearchFilterOnlyRetriever is to return all documents that match the given filter and not use a `top_k` at all. Do you know whether that's currently the case or does it always return no more than 10 documents? 
Just to add to this. I think we recently had a similar case for `get_documents_by_id` and I'd agree with @julian-risch that we shouldn't use top_k here because it is not meaningful. For the case of `get_documents_by_id` we had to adjust the query like this:
https://github.com/deepset-ai/haystack/blob/692cde11e736884cbd1a5f99416cee94c4f0921b/haystack/document_stores/elasticsearch.py#L472

Happy if you would contribute a fix there @adri1wald :)
Ah, thanks for the reminder @mathislucka . Yes, the limitation to 10 documents most likely comes from Elasticsearch. @adri1wald A quick fix could be to add `"size"` to the query body, setting it to the maximum that Elasticsearch allows by default for example:
```
 body = {"size": 10000, "query": {"bool": {"must": {"match_all": {}}}}}  # type: Dict[str, Any]
```
A better solution would be to use the `scan` functionality as used in the `get_all_documents` method:
https://github.com/deepset-ai/haystack/blob/692cde11e736884cbd1a5f99416cee94c4f0921b/haystack/document_stores/elasticsearch.py#L898

Could you maybe just call `get_all_documents` with the filters set within `ElasticsearchFilterOnlyRetriever.retrieve` instead of the `ElasticsearchDocumentStore.query` method?
Hey @julian-risch @mathislucka!

Thanks for taking a look at this.

My usecase is definitely met by getting all the filtered documents. I really like the idea of using `get_all_documents` in `ElasticsearchFilterOnlyRetriever.retrieve`. I will implement that for our project now.

Happy to make the same change in haystack and submit a PR once I've made it work for us! Also, if the semantics of `top_k` without a `query` don't make sense, might it also be reasonable to entirely remove the code for handling `query is None` in `ElasticsearchDocumentStore.query` (and change the method signature to `query: str`)?

If you're interested:

I'm trying to build a question generation pipeline that retrieves all the documents given some filters, samples `k` documents, and then runs question generation. The sampling is useful because question generation is slow (there can be > 1000 docs after filtering), and even if it was fast it would generate an unmanageable amount of questions for a user. With this pipeline a user gets a reasonable amount of questions back, and they can generate questions on a different sample by pressing a refresh button.

I think a more effective way to achieve this would be something like a `DocumentStore.sample_documents(self, sample_k: int, filters...)` method that performs the sampling within the store backend, but I don't know if this is possible with Elasticsearch (or indeed any other store backend).
I think the storage backends do not support random samples. I think what I'd do when implementing this as a custom node:

- have a separate index where you store all ids as metadata in a single document
- retrieve all ids from that index
- take a random sample of the id list
- query the actual document index with the sampled ids and `get_documents_by_id`
At least for elasticsearch, could this be used for random sampling? https://www.elastic.co/guide/en/elasticsearch/guide/current/random-scoring.html

Also sorry if this is off topic for this issue - happy to create a separate one to discuss this further
@adri1wald 
> Happy to make the same change in haystack and submit a PR once I've made it work for us! 

Looking forward to these changes!

> Also, if the semantics of `top_k` without a `query` don't make sense, might it also be reasonable to entirely remove the code for handling `query is None` in `ElasticsearchDocumentStore.query` (and change the method signature to `query: str`)?

I'm a bit hesitant to remove these lines of code. Although their effect and the lack of `top_k`  in that context can be unintuitive, it could still well be that somebody is relying on `query` being optional. Let's not address it in this PR but in a separate one where we can make sure that all document stores use the same signature with `query: str` in future. 

The random scoring  could help for the sampling functionality that you are looking for. It's an interesting use case! You could maybe try to use `custom_query` for that? 

Resolving patch:
diff --git a/haystack/nodes/retriever/sparse.py b/haystack/nodes/retriever/sparse.py
--- a/haystack/nodes/retriever/sparse.py
+++ b/haystack/nodes/retriever/sparse.py
@@ -157,20 +157,16 @@ def retrieve(
         Scan through documents in DocumentStore and return a small number documents
         that are most relevant to the query.
 
-        :param query: The query
+        :param query: Has no effect, can pass in empty string
         :param filters: A dictionary where the keys specify a metadata field and the value is a list of accepted values for that field
-        :param top_k: How many documents to return per query.
+        :param top_k: Has no effect, pass in any int or None
         :param index: The name of the index in the DocumentStore from which to retrieve documents
         :param headers: Custom HTTP headers to pass to elasticsearch client (e.g. {'Authorization': 'Basic YWRtaW46cm9vdA=='})
                 Check out https://www.elastic.co/guide/en/elasticsearch/reference/current/http-clients.html for more information.
         """
-        if top_k is None:
-            top_k = self.top_k
         if index is None:
             index = self.document_store.index
-        documents = self.document_store.query(
-            query=None, filters=filters, top_k=top_k, custom_query=self.custom_query, index=index, headers=headers
-        )
+        documents = self.document_store.get_all_documents(filters=filters, index=index, headers=headers)
         return documents
 
 


Test patch:
diff --git a/test/test_retriever.py b/test/test_retriever.py
--- a/test/test_retriever.py
+++ b/test/test_retriever.py
@@ -563,3 +563,25 @@ def test_embeddings_encoder_of_embedding_retriever_should_warn_about_model_forma
             "You may need to set 'model_format='sentence_transformers' to ensure correct loading of model."
             in caplog.text
         )
+
+
+@pytest.mark.parametrize("retriever", ["es_filter_only"], indirect=True)
+@pytest.mark.parametrize("document_store", ["elasticsearch"], indirect=True)
+def test_es_filter_only(document_store, retriever):
+    docs = [
+        Document(content="Doc1", meta={"f1": "0"}),
+        Document(content="Doc2", meta={"f1": "0"}),
+        Document(content="Doc3", meta={"f1": "0"}),
+        Document(content="Doc4", meta={"f1": "0"}),
+        Document(content="Doc5", meta={"f1": "0"}),
+        Document(content="Doc6", meta={"f1": "0"}),
+        Document(content="Doc7", meta={"f1": "1"}),
+        Document(content="Doc8", meta={"f1": "0"}),
+        Document(content="Doc9", meta={"f1": "0"}),
+        Document(content="Doc10", meta={"f1": "0"}),
+        Document(content="Doc11", meta={"f1": "0"}),
+        Document(content="Doc12", meta={"f1": "0"}),
+    ]
+    document_store.write_documents(docs)
+    retrieved_docs = retriever.retrieve(query="", filters={"f1": ["0"]})
+    assert len(retrieved_docs) == 11