Repository: deepset-ai/haystack

Issue:
WeaviateDocumentStore bug - weaviate has an internal query limit, so `.update_embeddings()` will loop forever if hits that
**Describe the bug**
FYI - @ZanSara @bogdankostic 
The latest version of the `WeaviateDocumentStore` has a bug, which comes up when calling `.update_embedddings()` on a collection with more than 10k documents.
The bug was introduced by recent commit https://github.com/deepset-ai/haystack/commit/ae712fe6bf087c717f3e38e4e87d2347165fc12b#diff-d4e4f2566db6c2fbe852debffcf7830447ecb969d4cdcce6d796fd773eba379bR682

**Error message**
No error, it gets into a forever loop never completing - due to a `while` loop never turning false.

**Expected behavior**
The `WeaviateDocumentStore` should be able to update the embeddings of ALL documents and should not be limited by the Weaviate query limit (`QUERY_MAXIMUM_RESULTS`, which is 10k by default). Or if it is limited, then at minimum it should throw an error explaining that it cannot update all documents only up to the `QUERY_MAXIMUM_RESULTS`, instead of getting into a forever loop when hitting the 10k records and not being able to go further.

**Additional context**"
Unfortunately fixing it is not as simple as just increasing the `QUERY_MAXIMUM_RESULTS` to infinity.
Quoting the Weaviate documentation, it is NOT recommended to increase the `QUERY_MAXIMUM_RESULTS` to too high. See:
_"Due to the increasing cost of each page outlined above, there is a limit to how many objects can be retrieved using pagination. By default setting the sum of offset and limit to higher than 10,000 objects, will lead to an error. If you must retrieve more than 10,000 objects, you can increase this limit by setting the environment variable QUERY_MAXIMUM_RESULTS=<desired-value>. Warning: **Setting this to arbitrarily high values can make the memory consumption of a single query explode and single queries can slow down the entire cluster.** We recommend setting this value to the lowest possible value that does not interfere with your users’ expectations."_
https://weaviate.io/developers/weaviate/current/graphql-references/filters.html

**To Reproduce**
Try to run the Haystack-Weaviate example published by Laura Ham: 
YouTube: https://www.youtube.com/watch?v=BkozaOnZpJI
GitHub: https://github.com/semi-technologies/weaviate-examples/tree/main/harrypotter-qa-haystack-weaviate

This example dataset has 13k records and uses default Weaviate settings, which has `QUERY_MAXIMUM_RESULTS` set to 10k, so it will hang forever when calling `.update_embeddings(), which will hit that forever loop at https://github.com/deepset-ai/haystack/blob/caf13364243807b873d74b9f4f2448ce2cec60f0/haystack/document_stores/weaviate.py#L693 (as the query can only return 10k records, but it is expecting to receive 13k, so it will keep querying forever).

**FAQ Check**
- [ X ] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: Ubuntu
 - GPU/CPU: GPU
 - Haystack version (commit or version number): 1.4.0
 - DocumentStore: Weaviate
 - Reader: deepset/tinyroberta-squad2
 - Retriever: sentence-transformers/multi-qa-mpnet-base-dot-v1



Discussion:

Resolving patch:
diff --git a/haystack/document_stores/weaviate.py b/haystack/document_stores/weaviate.py
--- a/haystack/document_stores/weaviate.py
+++ b/haystack/document_stores/weaviate.py
@@ -22,12 +22,17 @@
 from haystack.document_stores.base import get_batches_from_generator
 from haystack.document_stores.filter_utils import LogicalFilterClause
 from haystack.document_stores.utils import convert_date_to_rfc3339
+from haystack.errors import DocumentStoreError
 
 
 logger = logging.getLogger(__name__)
 UUID_PATTERN = re.compile(r"^[\da-f]{8}-([\da-f]{4}-){3}[\da-f]{12}$", re.IGNORECASE)
 
 
+class WeaviateDocumentStoreError(DocumentStoreError):
+    pass
+
+
 class WeaviateDocumentStore(BaseDocumentStore):
     """
 
@@ -697,17 +702,37 @@ def _get_all_documents_in_index(
                 query = query.with_where(filter_dict)
 
             if all_docs:
-                # .with_limit() must be used with .with_offset, of the latter won't work properly
-                #   https://weaviate-python-client.readthedocs.io/en/latest/weaviate.gql.html?highlight=offset#weaviate.gql.get.GetBuilder.with_offset
+                # Passing offset:0 raises an error, so we pass it only after the first round
+                # `.with_limit()` must be used with `.with_offset`, or the latter won't work properly
+                # https://weaviate-python-client.readthedocs.io/en/latest/weaviate.gql.html?highlight=offset#weaviate.gql.get.GetBuilder.with_offset
                 query = query.with_limit(100).with_offset(offset=len(all_docs))
 
-            result = query.do()
+            try:
+                result = query.do()
+            except Exception as e:
+                raise WeaviateDocumentStoreError(f"Weaviate raised an exception: {e}")
+
+            if "errors" in result:
+                raise WeaviateDocumentStoreError(f"Query results contain errors: {result['errors']}")
+
+            # If `query.do` didn't raise and `result` doesn't contain errors,
+            # we are good accessing data
+            docs = result.get("data").get("Get").get(index)
+
+            # `docs` can be empty if the query returned less documents than the actual
+            # number. This can happen when the number of document stored is greater
+            # than QUERY_MAXIMUM_RESULTS.
+            # See: https://weaviate.io/developers/weaviate/current/graphql-references/filters.html#offset-argument-pagination
+            if not docs:
+                logger.warning(
+                    "The query returned less documents than expected: this can happen when "
+                    "the value of the QUERY_MAXIMUM_RESULTS environment variable is lower than "
+                    "the total number of documents stored. See Weaviate documentation for "
+                    "more details."
+                )
+                break
 
-            if result and "data" in result and "Get" in result.get("data"):
-                if result.get("data").get("Get").get(index):
-                    all_docs += result.get("data").get("Get").get(index)
-            else:
-                raise ValueError(f"Weaviate returned ad exception: {result}")
+            all_docs += docs
 
         yield from all_docs
 
@@ -1156,7 +1181,7 @@ def delete_all_documents(
             raise NotImplementedError("WeaviateDocumentStore does not support headers.")
 
         logger.warning(
-            """DEPRECATION WARNINGS: 
+            """DEPRECATION WARNINGS:
                 1. delete_all_documents() method is deprecated, please use delete_documents method
                 For more details, please refer to the issue: https://github.com/deepset-ai/haystack/issues/1045
                 """


Test patch:
diff --git a/test/document_stores/test_weaviate.py b/test/document_stores/test_weaviate.py
--- a/test/document_stores/test_weaviate.py
+++ b/test/document_stores/test_weaviate.py
@@ -1,8 +1,11 @@
+import uuid
+
 import numpy as np
 import pytest
+
 from haystack.schema import Document
 from ..conftest import get_document_store
-import uuid
+
 
 embedding_dim = 768
 
@@ -105,3 +108,14 @@ def test_query(document_store_with_docs):
 
     docs = document_store_with_docs.query(filters={"content": ["live"]})
     assert len(docs) == 3
+
+
+@pytest.mark.weaviate
+def test_get_all_documents_unaffected_by_QUERY_MAXIMUM_RESULTS(document_store_with_docs, monkeypatch):
+    """
+    Ensure `get_all_documents` works no matter the value of QUERY_MAXIMUM_RESULTS
+    see https://github.com/deepset-ai/haystack/issues/2517
+    """
+    monkeypatch.setattr(document_store_with_docs, "get_document_count", lambda **kwargs: 13_000)
+    docs = document_store_with_docs.get_all_documents()
+    assert len(docs) == 3