Repository: deepset-ai/haystack

Issue:
Multiple filters not supported in MilvusDocumentStore?
I am trying to use `get_all_documents` and `delete_documents` with filters.

Adding on filter arg, e.g. `{"my_filter_arg": ["value"]}` gives the correct results, but using two (`AND` expected) gives an empty result, e.g. `{"my_filter_arg_1": ["value1"], "my_filter_arg_2": ["value2"]}`

Is this known behaviour? And if so, what is a possible roadmap adding that functionality? The docs suggest this is working.

Thanks!


Discussion:
Hi @stphnthiel , I tested out the MilvusDocumentStore with Milvus 1.1.1 and I found that filtering doesn't seem to be working for querying or for deleting documents

```
from haystack.utils import launch_milvus, launch_es
from haystack.document_stores import MilvusDocumentStore, ElasticsearchDocumentStore
from haystack.nodes import EmbeddingRetriever, DensePassageRetriever
from haystack.pipelines import DocumentSearchPipeline
from haystack.utils import print_documents

launch_milvus()
document_store = MilvusDocumentStore()
document_store.delete_all_documents()

# launch_es()
# document_store = ElasticsearchDocumentStore()
# document_store.delete_all_documents()

docs = [
    {"content": "Here is some text",
     "meta": {"year": 1,
              "source": "a"}},
    {"content": "Here is more text",
     "meta": {"year": 1,
              "source": "b"}},
    {"content": "Here is yet more text",
     "meta": {"year": 2,
              "source": "b"}}
]

document_store.write_documents(docs)

retriever = DensePassageRetriever(document_store)

document_store.update_embeddings(retriever)

p = DocumentSearchPipeline(retriever)
res = p.run(query="here", params={"filters": {"source": ["a"]}})
print_documents(res, print_meta=True)

document_store.delete_documents(filters={"source": ["a"]})
documents = document_store.get_all_documents()
print(documents)

document_store.delete_documents(filters={"source": ["b"], "year": [1]})
documents = document_store.get_all_documents()
print(documents)
```
I definitely think we should have support for this ASAP. @tholor could we plan this into a coming sprint?
Yes, delete and get documents should work with filters. Let's work on it in the upcoming sprint.
Filtering at query time is not supported in Milvus and we are actually raising already a warning here (filtering will come with Milvus 2.0). 

Hey both,

Thanks for getting back so quickly and confirming this (and providing
Haystack as open source in the first place!)

What’s the rough timeline for Milvus 2.0? For my use case, filtering on
query is quite essential, so I am wondering whether I should switch to
using a different store. I moved away from FAISS because I thought Milvus
would be better suited for larger collections, but it’s probably a
good idea to use that until Milvus 2.0 support?

Thanks!

On Tue, Nov 9, 2021 at 4:58 PM Malte Pietsch ***@***.***>
wrote:

> Yes, delete and get documents should work with filters. Let's work on it
> in the upcoming sprint.
> Filtering at query time is not supported in Milvus and we are actually
> raising already a warning here (filtering will come with Milvus 2.0).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/deepset-ai/haystack/issues/1712#issuecomment-964287236>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABS5FRVDNXGPPTJ2RHX74TULFAIZANCNFSM5HUX5PYQ>
> .
>

One more thing to notice: I think this bug actually comes from the `_query` function of `SQLDocumentStore`, and indeed I found that `FAISSDocumentStore` seems to be affected by this as well (just use the snippet above and replace `MilvusDocumentStore` with `FAISSDocumentStore`). There are probably other functions affected, like `update_embeddings`, which also uses `_query`. I'm quite surprised this issue went unnoticed for so long...

@stphnthiel you didn't come across this issue with FAISS?
> What’s the rough timeline for Milvus 2.0?

We are just waiting for their official release ;)
It was initially planned for September, I believe, but they have a few open tasks they want to get straight. 

We already have an experimental version in Haystack to support it here: https://github.com/deepset-ai/haystack/pull/1473
However, the Milvus side is not stable yet and only allows certain filters. I wouldn't recommend it for production.

How large is your collection? At the moment, the only doc stores that support filtering and ANN (which become important  for > 1mio vectors) are `OpenSearchDocumentStore` and `WeaviateDocumentStore`. Elasticsearch is already working on it as well and will probably release something soonish (see https://github.com/elastic/elasticsearch/issues/78473#issuecomment-934554963).
@ZanSara I didn't do a full test but was rather trying to figure out an alternative for now.

My guess why this might not have come up is that many users are separating their document collections via indices? I am working with one index only and instead rely heavily on using filters to group documents. But I only discovered it now as implementation became more specific to adding/removing docs to/from the store.

And IIRC it appeared only with more than one filter. One filter seemed fine. But I'd have to validate this to be certain.

Would Weaviate be an alternative to switch to or can I contribute to fixing the filter issues with Milvus 1.1 somehow?

Resolving patch:
diff --git a/haystack/document_stores/sql.py b/haystack/document_stores/sql.py
--- a/haystack/document_stores/sql.py
+++ b/haystack/document_stores/sql.py
@@ -179,7 +179,7 @@ def get_all_documents(
         filters: Optional[Dict[str, List[str]]] = None,
         return_embedding: Optional[bool] = None,
     ) -> List[Document]:
-        documents = list(self.get_all_documents_generator(index=index, filters=filters))
+        documents = list(self.get_all_documents_generator(index=index, filters=filters, return_embedding=return_embedding))
         return documents
 
     def get_all_documents_generator(
@@ -201,7 +201,6 @@ def get_all_documents_generator(
         :param return_embedding: Whether to return the document embeddings.
         :param batch_size: When working with large number of documents, batching can help reduce memory footprint.
         """
-
         if return_embedding is True:
             raise Exception("return_embeddings is not supported by SQLDocumentStore.")
         result = self._query(
@@ -239,13 +238,14 @@ def _query(
         ).filter_by(index=index)
 
         if filters:
-            documents_query = documents_query.join(MetaDocumentORM)
             for key, values in filters.items():
-                documents_query = documents_query.filter(
-                    MetaDocumentORM.name == key,
-                    MetaDocumentORM.value.in_(values),
-                    DocumentORM.id == MetaDocumentORM.document_id
-                )
+                documents_query = documents_query. \
+                    join(MetaDocumentORM, aliased=True). \
+                    filter(
+                        MetaDocumentORM.name == key,
+                        MetaDocumentORM.value.in_(values),
+                    )
+
         if only_documents_without_embedding:
             documents_query = documents_query.filter(DocumentORM.vector_id.is_(None))
         if vector_ids:
@@ -446,9 +446,13 @@ def get_document_count(self, filters: Optional[Dict[str, List[str]]] = None, ind
         query = self.session.query(DocumentORM).filter_by(index=index)
 
         if filters:
-            query = query.join(MetaDocumentORM)
             for key, values in filters.items():
-                query = query.filter(MetaDocumentORM.name == key, MetaDocumentORM.value.in_(values))
+                query = query. \
+                    join(MetaDocumentORM, aliased=True). \
+                    filter(
+                        MetaDocumentORM.name == key,
+                        MetaDocumentORM.value.in_(values),
+                    )
 
         count = query.count()
         return count
@@ -537,11 +541,12 @@ def delete_documents(self, index: Optional[str] = None, ids: Optional[List[str]]
             document_ids_to_delete = self.session.query(DocumentORM.id).filter(DocumentORM.index==index)
             if filters:
                 for key, values in filters.items():
-                    document_ids_to_delete = document_ids_to_delete.filter(
-                        MetaDocumentORM.name == key,
-                        MetaDocumentORM.value.in_(values),
-                        DocumentORM.id == MetaDocumentORM.document_id
-                    )
+                    document_ids_to_delete = document_ids_to_delete. \
+                        join(MetaDocumentORM, aliased=True). \
+                        filter(
+                            MetaDocumentORM.name == key,
+                            MetaDocumentORM.value.in_(values),
+                        )
             if ids:
                 document_ids_to_delete = document_ids_to_delete.filter(DocumentORM.id.in_(ids))
 


Test patch:
diff --git a/test/test_faiss_and_milvus.py b/test/test_faiss_and_milvus.py
--- a/test/test_faiss_and_milvus.py
+++ b/test/test_faiss_and_milvus.py
@@ -14,12 +14,12 @@
 from haystack.nodes.retriever.dense import EmbeddingRetriever
 
 DOCUMENTS = [
-    {"name": "name_1", "content": "text_1", "embedding": np.random.rand(768).astype(np.float32)},
-    {"name": "name_2", "content": "text_2", "embedding": np.random.rand(768).astype(np.float32)},
-    {"name": "name_3", "content": "text_3", "embedding": np.random.rand(768).astype(np.float64)},
-    {"name": "name_4", "content": "text_4", "embedding": np.random.rand(768).astype(np.float32)},
-    {"name": "name_5", "content": "text_5", "embedding": np.random.rand(768).astype(np.float32)},
-    {"name": "name_6", "content": "text_6", "embedding": np.random.rand(768).astype(np.float64)},
+    {"meta": {"name": "name_1", "year": "2020", "month": "01"}, "content": "text_1", "embedding": np.random.rand(768).astype(np.float32)},
+    {"meta": {"name": "name_2", "year": "2020", "month": "02"}, "content": "text_2", "embedding": np.random.rand(768).astype(np.float32)},
+    {"meta": {"name": "name_3", "year": "2020", "month": "03"}, "content": "text_3", "embedding": np.random.rand(768).astype(np.float64)},
+    {"meta": {"name": "name_4", "year": "2021", "month": "01"}, "content": "text_4", "embedding": np.random.rand(768).astype(np.float32)},
+    {"meta": {"name": "name_5", "year": "2021", "month": "02"}, "content": "text_5", "embedding": np.random.rand(768).astype(np.float32)},
+    {"meta": {"name": "name_6", "year": "2021", "month": "03"}, "content": "text_6", "embedding": np.random.rand(768).astype(np.float64)},
 ]
 
 
@@ -217,6 +217,38 @@ def test_delete_docs_with_filters(document_store, retriever):
     assert {doc.meta["name"] for doc in documents} == {"name_5", "name_6"}
 
 
+@pytest.mark.slow
+@pytest.mark.parametrize("retriever", ["dpr"], indirect=True)
+@pytest.mark.parametrize("document_store", ["faiss", "milvus"], indirect=True)
+def test_delete_docs_with_filters(document_store, retriever):
+    document_store.write_documents(DOCUMENTS)
+    document_store.update_embeddings(retriever=retriever, batch_size=4)
+    assert document_store.get_embedding_count() == 6
+
+    document_store.delete_documents(filters={"year": ["2020"]})
+
+    documents = document_store.get_all_documents()
+    assert len(documents) == 3
+    assert document_store.get_embedding_count() == 3
+    assert all("2021" == doc.meta["year"] for doc in documents)
+
+
+@pytest.mark.slow
+@pytest.mark.parametrize("retriever", ["dpr"], indirect=True)
+@pytest.mark.parametrize("document_store", ["faiss", "milvus"], indirect=True)
+def test_delete_docs_with_many_filters(document_store, retriever):
+    document_store.write_documents(DOCUMENTS)
+    document_store.update_embeddings(retriever=retriever, batch_size=4)
+    assert document_store.get_embedding_count() == 6
+
+    document_store.delete_documents(filters={"month": ["01"], "year": ["2020"]})
+
+    documents = document_store.get_all_documents()
+    assert len(documents) == 5
+    assert document_store.get_embedding_count() == 5
+    assert "name_1" not in {doc.meta["name"] for doc in documents}
+
+
 @pytest.mark.slow
 @pytest.mark.parametrize("retriever", ["dpr"], indirect=True)
 @pytest.mark.parametrize("document_store", ["faiss", "milvus"], indirect=True)
@@ -261,6 +293,50 @@ def test_delete_docs_by_id_with_filters(document_store, retriever):
     assert all(doc_id in all_ids_left for doc_id in ids_not_to_delete)
 
 
+@pytest.mark.slow
+@pytest.mark.parametrize("retriever", ["dpr"], indirect=True)
+@pytest.mark.parametrize("document_store", ["faiss", "milvus"], indirect=True)
+def test_get_docs_with_filters_one_value(document_store, retriever):
+    document_store.write_documents(DOCUMENTS)
+    document_store.update_embeddings(retriever=retriever, batch_size=4)
+    assert document_store.get_embedding_count() == 6
+
+    documents =  document_store.get_all_documents(filters={"year": ["2020"]})
+
+    assert len(documents) == 3
+    assert all("2020" == doc.meta["year"] for doc in documents)
+
+
+@pytest.mark.slow
+@pytest.mark.parametrize("retriever", ["dpr"], indirect=True)
+@pytest.mark.parametrize("document_store", ["faiss", "milvus"], indirect=True)
+def test_get_docs_with_filters_many_values(document_store, retriever):
+    document_store.write_documents(DOCUMENTS)
+    document_store.update_embeddings(retriever=retriever, batch_size=4)
+    assert document_store.get_embedding_count() == 6
+
+    documents = document_store.get_all_documents(filters={"name": ["name_5", "name_6"]})
+
+    assert len(documents) == 2
+    assert {doc.meta["name"] for doc in documents} == {"name_5", "name_6"}
+
+
+@pytest.mark.slow
+@pytest.mark.parametrize("retriever", ["dpr"], indirect=True)
+@pytest.mark.parametrize("document_store", ["faiss", "milvus"], indirect=True)
+def test_get_docs_with_many_filters(document_store, retriever):
+    document_store.write_documents(DOCUMENTS)
+    document_store.update_embeddings(retriever=retriever, batch_size=4)
+    assert document_store.get_embedding_count() == 6
+
+    documents = document_store.get_all_documents(filters={"month": ["01"], "year": ["2020"]})
+
+    assert len(documents) == 1
+    assert "name_1" == documents[0].meta["name"]
+    assert "01" == documents[0].meta["month"]
+    assert "2020" == documents[0].meta["year"]
+
+
 
 @pytest.mark.parametrize("retriever", ["embedding"], indirect=True)
 @pytest.mark.parametrize("document_store", ["faiss", "milvus"], indirect=True)