Repository: deepset-ai/haystack

Issue:
Expected behavior of Embedding Retriever with missing embeddings
**Question**
What is the expected behavior of Embedding based retriever if some documents do not have embeddings

**Additional context**
Unlike tutorial setup where write to document store and retrieval are sequential, in production setup both the writing of documents and retrieval can happen in parallel. However, the write of key-value pair happens faster to document store but embedding creation (update_embeddings) and indexing may take longer. During that time if any other embedding retriever tries to do cosine similarity the code is likely to throw errors as few records may not have embeddings.

If there is an issue it will be a serious concern for production systems where read and writes from document stores happen concurrently and embeddings may be temporarily missing. We are observing a few issues and may be related to this aspect.

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

@julian-risch @tholor



Discussion:
Hi @cvgoudar I understand the concerns and agree that we should think about a different solution than the one we have in place at the moment. 
The document stores handle the documents without embeddings differently. ElasticsearchDocumentStore indeed raises an error message if you try to query by embedding and there are documents without embeddings: https://github.com/deepset-ai/haystack/blob/892ce4a760c6b7d7d108aa01eee985ab6ecf8483/haystack/document_stores/elasticsearch.py#L822
In contrast to that, WeaviateDocumentStore creates dummy embeddings for new documents so query by embedding cannot run into the same problem.

Embedding creation and indexing may take longer than querying, yes. However, we think that parallelization with ray could drastically speed up embedding creation and indexing.
What behavior of embedding-based retrievers would be best for your use case? Allowing queries if some documents don't have embeddings and excluding these documents from the search? Or do you have a different solution in mind? How frequently (rough estimation) would you like to add documents to the document store? 
Thanks @julian-risch 

For our use case we are good with the following option:   
**Allowing queries if some documents don't have embeddings and excluding these documents from the search**

My suggestion is that this makes a good default option. Based on the document_store type and available option we can support other functionality as well. In summary, we need a parameter that controls the behavior with how retriever works with documents not having embeddings. My recommendation would be to skip the records not having embeddings as the default method or None which makes the default choice based on document_store type.
Related to https://github.com/deepset-ai/haystack/issues/1630#issuecomment-962413324

@cvgoudar I like your suggestion and I can see that we restrict the search in ElasticsearchDocumentStore to docs that already have an embedding. Are you interested in creating a PR that modifies the query that you mention in https://github.com/deepset-ai/haystack/issues/1630#issuecomment-962413324?  
The solution ideally should be generic and some additional parameter may have to be passed as well. The change may have to be beyond elasticsearch itself. I can possibly make the change for elasticsearch.  However, we need to think carefully about the parameters as it has implications for other document store types as well.
@cvgoudar if you could create a PR with the changes to elasticsearch that would be great. 👍  Regarding the parameter: skipping documents without embeddings or failing if there are any are the two main options I see. Calculating missing embeddings on the fly is possible in theory but does not seem practical to me.
Nice, let's start with elasticsearch then. We can later take over and extend it to other documentstores if meaningful. 
I also only see the two options @julian-risch mentioned. So probably a param like `skip_missing_embeddings=True/False` is enough :thinking: 
We should cover the case though where no embedding at all is present after the first indexing of documents. Instead of returning an empty list of results we should trigger the same error as currently asking the user to run `update_embeddings()`  
Does this change make sense. I did sanity check on one test case and it seems to work fine.

From:  

`"query": {"match_all": {}}`

To:  
```
                "query": {
                    "bool": {
                        "filter": {
                            "bool": {
                                "must": [
                                    {
                                        "exists": {
                                            "field": self.embedding_field
                                        }
                                    }
                                ]
                            }
                        }
                    }
                }
```
Sounds reasonable to me. What was your test case? I am not sure if the current `exists` also excludes docs that do have the embedding field (i.e  the key) but no value set (i.e. null or empty array)
The test case I checked is as follows:
1. Indexed 8 records with embeddings
2. Added 2 more records without embeddings
3. Checked the documents in Elasticsearch with 8 records having embedding + 2 docs without embeddings
4. Ran Retriever and observed that only 8 records where fetched for top 10 matches with filtering condition added
As per my understanding, Elasticsearch doesn't index keys where the value is null. Please confirm. I can add that test case as well and check

Resolving patch:
diff --git a/haystack/document_stores/elasticsearch.py b/haystack/document_stores/elasticsearch.py
--- a/haystack/document_stores/elasticsearch.py
+++ b/haystack/document_stores/elasticsearch.py
@@ -51,7 +51,8 @@ def __init__(
         return_embedding: bool = False,
         duplicate_documents: str = 'overwrite',
         index_type: str = "flat",
-        scroll: str = "1d"
+        scroll: str = "1d",
+        skip_missing_embeddings: bool = True
     ):
         """
         A DocumentStore using Elasticsearch to store and query the documents for our search.
@@ -104,6 +105,10 @@ def __init__(
         :param scroll: Determines how long the current index is fixed, e.g. during updating all documents with embeddings.
                        Defaults to "1d" and should not be larger than this. Can also be in minutes "5m" or hours "15h"
                        For details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html
+        :param skip_missing_embeddings: Parameter to control queries based on vector similarity when indexed documents miss embeddings.
+                                        Parameter options: (True, False)
+                                        False: Raises exception if one or more documents do not have embeddings at query time
+                                        True: Query will ignore all documents without embeddings (recommended if you concurrently index and query)
 
         """
         # save init parameters to enable export of component config as YAML
@@ -114,7 +119,8 @@ def __init__(
             custom_mapping=custom_mapping, excluded_meta_data=excluded_meta_data, analyzer=analyzer, scheme=scheme,
             ca_certs=ca_certs, verify_certs=verify_certs, create_index=create_index,
             duplicate_documents=duplicate_documents, refresh_type=refresh_type, similarity=similarity,
-            timeout=timeout, return_embedding=return_embedding, index_type=index_type, scroll=scroll
+            timeout=timeout, return_embedding=return_embedding, index_type=index_type, scroll=scroll,
+            skip_missing_embeddings=skip_missing_embeddings
         )
 
         self.client = self._init_elastic_client(host=host, port=port, username=username, password=password,
@@ -140,6 +146,7 @@ def __init__(
         self.index: str = index
         self.label_index: str = label_index
         self.scroll = scroll
+        self.skip_missing_embeddings: bool = skip_missing_embeddings
         if similarity in ["cosine", "dot_product", "l2"]:
             self.similarity = similarity
         else:
@@ -817,6 +824,11 @@ def query_by_embedding(self,
             logger.debug(f"Retriever query: {body}")
             try:
                 result = self.client.search(index=index, body=body, request_timeout=300)["hits"]["hits"]
+                if len(result) == 0:
+                    count_embeddings = self.get_embedding_count(index=index)
+                    if count_embeddings == 0:
+                        raise RequestError(400, "search_phase_execution_exception",
+                                           {"error": "No documents with embeddings."})
             except RequestError as e:
                 if e.error == "search_phase_execution_exception":
                     error_message: str = "search_phase_execution_exception: Likely some of your stored documents don't have embeddings." \
@@ -845,9 +857,28 @@ def _get_vector_similarity_query(self, query_emb: np.ndarray, top_k: int):
         else:
             raise Exception("Invalid value for similarity in ElasticSearchDocumentStore constructor. Choose between \'cosine\' and \'dot_product\'")
 
+        # To handle scenarios where embeddings may be missing
+        script_score_query: dict = {"match_all": {}}
+        if self.skip_missing_embeddings:
+            script_score_query = {
+                "bool": {
+                    "filter": {
+                        "bool": {
+                            "must": [
+                                {
+                                    "exists": {
+                                        "field": self.embedding_field
+                                    }
+                                }
+                            ]
+                        }
+                    }
+                }
+            }
+
         query = {
             "script_score": {
-                "query": {"match_all": {}},
+                "query": script_score_query,
                 "script": {
                     # offset score to ensure a positive range as required by Elasticsearch
                     "source": f"{similarity_fn_name}(params.query_vector,'{self.embedding_field}') + 1000",


Test patch:
diff --git a/test/test_document_store.py b/test/test_document_store.py
--- a/test/test_document_store.py
+++ b/test/test_document_store.py
@@ -2,6 +2,8 @@
 import pandas as pd
 import pytest
 from elasticsearch import Elasticsearch
+from elasticsearch.exceptions import RequestError
+
 
 from conftest import get_document_store
 from haystack.document_stores import WeaviateDocumentStore
@@ -884,3 +886,38 @@ def test_get_document_count_only_documents_without_embedding_arg():
                                              filters={"meta_field_for_count": ["c"]}) == 1
     assert document_store.get_document_count(only_documents_without_embedding=True,
                                              filters={"meta_field_for_count": ["b"]}) == 2
+
+
+@pytest.mark.elasticsearch
+def test_skip_missing_embeddings():
+    documents = [
+        {"content": "text1", "id": "1"},  # a document without embeddings
+        {"content": "text2", "id": "2", "embedding": np.random.rand(768).astype(np.float64)},
+        {"content": "text3", "id": "3", "embedding": np.random.rand(768).astype(np.float32).tolist()},
+        {"content": "text4", "id": "4", "embedding": np.random.rand(768).astype(np.float32)}
+    ]
+    document_store = ElasticsearchDocumentStore(index="skip_missing_embedding_index")
+    document_store.write_documents(documents)
+
+    document_store.skip_missing_embeddings = True
+    retrieved_docs = document_store.query_by_embedding(np.random.rand(768).astype(np.float32))
+    assert len(retrieved_docs) == 3
+
+    document_store.skip_missing_embeddings = False
+    with pytest.raises(RequestError):
+        document_store.query_by_embedding(np.random.rand(768).astype(np.float32))
+
+    # Test scenario with no embeddings for the entire index
+    documents = [
+            {"content": "text1", "id": "1"},
+            {"content": "text2", "id": "2"},
+            {"content": "text3", "id": "3"},
+            {"content": "text4", "id": "4"}
+        ]
+
+    document_store.delete_documents()
+    document_store.write_documents(documents)
+
+    document_store.skip_missing_embeddings = True
+    with pytest.raises(RequestError):
+        document_store.query_by_embedding(np.random.rand(768).astype(np.float32))