Repository: deepset-ai/haystack
Pull request: feat: Adding filters param to MostSimilarDocumentsPipeline run and run_batch
### Related Issues
- fixes #3288

### Proposed Changes:
Adding a param called "filters" to methods run and run_batch of MostSimilarDocumentsPipeline class, this param is sent to the query_by_embedding method of document_store, which already implements the logic to filter the results

### How did you test it?
Existent test:
Ran cd test & pytest
Added test:
Ran pytest -v pipelines/test_standard_pipelines.py::test_most_similar_documents_pipeline_with_filters
Ran pytest -v pipelines/test_standard_pipelines.py::test_most_similar_documents_pipeline_with_filters_batch

### Notes for the reviewer
Copied the test test_most_similar_documents_pipeline and test_most_similar_documents_pipeline_batch adding an extra assert to check the filters
`assert document.meta["source"] in ["wiki3", "wiki4", "wiki5"]`
Didn't include the test parameter of test_most_similar_documents_pipeline `("embedding", "milvus1")` since FAISS document store does not implement filters


### Checklist
- [X] I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- [X] I have updated the related issue with new insights and changes
- [X] I added tests that demonstrate the correct behavior of the change
- [X] I've used the [conventional commit convention](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title
- [X] I documented my code
- [X] I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
Add filters param to MostSimilarDocumentsPipeline
I tried to get the most similar documents given a document identifier, which is essentially the functionality of MostSimilarDocumentsPipeline. However, I would like the returned documents follow certain rules, for instance, only return documents which belongs to a category.

I think the easiest and most flexible approach is including a filter parameter to the run mehod in the pipeline, because this method calls the document_store.query_by_embedding method, which already includes this parameter

I don't know if all of the paramaters of document_store.query_by_embedding have sense in the context of MostSimilarDocumentsPipeline but maybe it could be good to include all of them in the MostSimilarDocumentsPipeline.run method

Hints from issue discussion:
Hi @JacdDev, thanks for opening the issue! I think that is a great idea. Would you be willing to open a PR with the suggested changes? We'd be happy to help answer any questions you have about the process. 
Sure, I'm going to read the Contributor Guidelines and try to help, thanks.

Commit message: Adding filters param to MostSimilarDocumentsPipeline run and run_batch

Patch to review:
diff --git a/haystack/pipelines/standard_pipelines.py b/haystack/pipelines/standard_pipelines.py
index 6275b0272c..7ef798f5e0 100644
--- a/haystack/pipelines/standard_pipelines.py
+++ b/haystack/pipelines/standard_pipelines.py
@@ -714,9 +714,15 @@ def __init__(self, document_store: BaseDocumentStore):
         self.pipeline.add_node(component=document_store, name="DocumentStore", inputs=["Query"])
         self.document_store = document_store
 
-    def run(self, document_ids: List[str], top_k: int = 5):
+    def run(
+        self,
+        document_ids: List[str],
+        filters: Optional[Dict[str, Union[Dict, List, str, int, float, bool]]] = None,
+        top_k: int = 5,
+    ):
         """
         :param document_ids: document ids
+        :param filters: Optional filters to narrow down the search space to documents whose metadata fulfill certain conditions
         :param top_k: How many documents id to return against single document
         """
         similar_documents: list = []
@@ -725,16 +731,17 @@ def run(self, document_ids: List[str], top_k: int = 5):
         for document in self.document_store.get_documents_by_id(ids=document_ids):
             similar_documents.append(
                 self.document_store.query_by_embedding(
-                    query_emb=document.embedding, return_embedding=False, top_k=top_k
+                    query_emb=document.embedding, filters=filters, return_embedding=False, top_k=top_k
                 )
             )
 
         self.document_store.return_embedding = False  # type: ignore
         return similar_documents
 
-    def run_batch(self, document_ids: List[str], top_k: int = 5):  # type: ignore
+    def run_batch(self, document_ids: List[str], filters: Optional[Dict[str, Union[Dict, List, str, int, float, bool]]] = None, top_k: int = 5):  # type: ignore
         """
         :param document_ids: document ids
+        :param filters: Optional filters to narrow down the search space to documents whose metadata fulfill certain conditions
         :param top_k: How many documents id to return against single document
         """
-        return self.run(document_ids=document_ids, top_k=top_k)
+        return self.run(document_ids=document_ids, filters=filters, top_k=top_k)
diff --git a/test/pipelines/test_standard_pipelines.py b/test/pipelines/test_standard_pipelines.py
index 21034f22e7..bdcfdf5fb8 100644
--- a/test/pipelines/test_standard_pipelines.py
+++ b/test/pipelines/test_standard_pipelines.py
@@ -200,6 +200,39 @@ def test_most_similar_documents_pipeline(retriever, document_store):
             assert isinstance(document.content, str)
 
 
+@pytest.mark.parametrize(
+    "retriever,document_store", [("embedding", "milvus1"), ("embedding", "elasticsearch")], indirect=True
+)
+def test_most_similar_documents_pipeline_with_filters(retriever, document_store):
+    documents = [
+        {"id": "a", "content": "Sample text for document-1", "meta": {"source": "wiki1"}},
+        {"id": "b", "content": "Sample text for document-2", "meta": {"source": "wiki2"}},
+        {"content": "Sample text for document-3", "meta": {"source": "wiki3"}},
+        {"content": "Sample text for document-4", "meta": {"source": "wiki4"}},
+        {"content": "Sample text for document-5", "meta": {"source": "wiki5"}},
+    ]
+
+    document_store.write_documents(documents)
+    document_store.update_embeddings(retriever)
+
+    docs_id: list = ["a", "b"]
+    filters = {"source": ["wiki3", "wiki4", "wiki5"]}
+    pipeline = MostSimilarDocumentsPipeline(document_store=document_store)
+    list_of_documents = pipeline.run(document_ids=docs_id, filters=filters)
+
+    assert len(list_of_documents[0]) > 1
+    assert isinstance(list_of_documents, list)
+    assert len(list_of_documents) == len(docs_id)
+
+    for another_list in list_of_documents:
+        assert isinstance(another_list, list)
+        for document in another_list:
+            assert isinstance(document, Document)
+            assert isinstance(document.id, str)
+            assert isinstance(document.content, str)
+            assert document.meta["source"] in ["wiki3", "wiki4", "wiki5"]
+
+
 @pytest.mark.parametrize("retriever,document_store", [("embedding", "memory")], indirect=True)
 def test_most_similar_documents_pipeline_batch(retriever, document_store):
     documents = [
@@ -213,6 +246,37 @@ def test_most_similar_documents_pipeline_batch(retriever, document_store):
     document_store.write_documents(documents)
     document_store.update_embeddings(retriever)
 
+    docs_id: list = ["a", "b"]
+    filters = {"source": ["wiki3", "wiki4", "wiki5"]}
+    pipeline = MostSimilarDocumentsPipeline(document_store=document_store)
+    list_of_documents = pipeline.run_batch(document_ids=docs_id, filters=filters)
+
+    assert len(list_of_documents[0]) > 1
+    assert isinstance(list_of_documents, list)
+    assert len(list_of_documents) == len(docs_id)
+
+    for another_list in list_of_documents:
+        assert isinstance(another_list, list)
+        for document in another_list:
+            assert isinstance(document, Document)
+            assert isinstance(document.id, str)
+            assert isinstance(document.content, str)
+            assert document.meta["source"] in ["wiki3", "wiki4", "wiki5"]
+
+
+@pytest.mark.parametrize("retriever,document_store", [("embedding", "memory")], indirect=True)
+def test_most_similar_documents_pipeline_with_filters_batch(retriever, document_store):
+    documents = [
+        {"id": "a", "content": "Sample text for document-1", "meta": {"source": "wiki1"}},
+        {"id": "b", "content": "Sample text for document-2", "meta": {"source": "wiki2"}},
+        {"content": "Sample text for document-3", "meta": {"source": "wiki3"}},
+        {"content": "Sample text for document-4", "meta": {"source": "wiki4"}},
+        {"content": "Sample text for document-5", "meta": {"source": "wiki5"}},
+    ]
+
+    document_store.write_documents(documents)
+    document_store.update_embeddings(retriever)
+
     docs_id: list = ["a", "b"]
     pipeline = MostSimilarDocumentsPipeline(document_store=document_store)
     list_of_documents = pipeline.run_batch(document_ids=docs_id)


Code review comments:
diff_hunk: @@ -213,6 +246,37 @@ def test_most_similar_documents_pipeline_batch(retriever, document_store):
     document_store.write_documents(documents)
     document_store.update_embeddings(retriever)
 
+    docs_id: list = ["a", "b"]
+    filters = {"source": ["wiki3", "wiki4", "wiki5"]}
+    pipeline = MostSimilarDocumentsPipeline(document_store=document_store)
+    list_of_documents = pipeline.run_batch(document_ids=docs_id, filters=filters)
path: test/pipelines/test_standard_pipelines.py
text: @user1:
Could you move this `run_batch` call with the `filters` option specified to under the test you made called `test_most_similar_documents_pipeline_with_filters_batch`? I think you misplaced it in the normal batch test. 

@author:
you right, solved