Repository: deepset-ai/haystack

Issue:
_get_all_documents_in_index from Class including Cross-reference property
Hi,

I'm trying to write documents and update embeddings using WeaviateDocumentStore, 
and it seems to be a problem in [**weaviate.py**](https://github.com/deepset-ai/haystack/blob/main/haystack/document_stores/weaviate.py) when you have created a **class with cross-reference property**

### The problem is here : 

https://github.com/deepset-ai/haystack/blob/444a3116c42d2c8852d27aa8093ac92c8e85ab88/haystack/document_stores/weaviate.py#L727-L756

In this code, you get a list of all properties by running **__get_current_properties()_** function.
then, weaviate_client.query.get(index, properties) return all documents


### But, if the class contains **cross-reference property**, this error occurs (**belongTo** is the cross-ref property) :

>   File "/usr/local/lib/python3.9/dist-packages/haystack/document_stores/weaviate.py", line 1377, in update_embeddings
    for result_batch in get_batches_from_generator(result, batch_size):
  File "/usr/local/lib/python3.9/dist-packages/haystack/document_stores/base.py", line 931, in get_batches_from_generator
    x = tuple(islice(it, n))
  File "/usr/local/lib/python3.9/dist-packages/haystack/document_stores/weaviate.py", line 779, in _get_all_documents_in_index
    raise WeaviateDocumentStoreError(f"Query results contain errors: {result['errors']}")
haystack.document_stores.weaviate.WeaviateDocumentStoreError: Query results contain errors: [{'locations': [{'column': 188, 'line': 1}], 'message': 'Field "belongTo" of type "[BenefitBelongToObj]" must have a sub selection.', 'path': None}]

This is because you are using the same result from **__get_current_properties()_** that contains only the list of all properties name.
**Cross-reference property** must contain its subselection property in list of properties, not just its property name : 

**`result = client.query.get("Article", ["title", "url", "wordCount", "inPublication {... on Publication {name }}"])
`**

Not **`result = client.query.get("Article", ["title", "url", "wordCount", "inPublication"])
`**

### So I resolved this problem temporarily by changing **Line 400** like this :

`cur_properties = [item["name"] for item in class_item["properties"] if item["dataType"][0] in ['string', 'string[]', 'int','number','text','date','boolean']]` by ignoring cross-reference dataType when you get current properties.

But, it doesn't look good because you have to specify all the data types besides cross-ref in the code.

Is there a way to resolve this without fixing your **weaviate.py** code?


Discussion:

Resolving patch:
diff --git a/haystack/document_stores/weaviate.py b/haystack/document_stores/weaviate.py
--- a/haystack/document_stores/weaviate.py
+++ b/haystack/document_stores/weaviate.py
@@ -28,6 +28,20 @@
 
 logger = logging.getLogger(__name__)
 UUID_PATTERN = re.compile(r"^[\da-f]{8}-([\da-f]{4}-){3}[\da-f]{12}$", re.IGNORECASE)
+SUPPORTED_PROPERTY_TYPES = {
+    "string",
+    "string[]",
+    "int",
+    "int[]",
+    "boolean",
+    "boolean[]",
+    "number",
+    "number[]",
+    "date",
+    "date[]",
+    "text",
+    "text[]",
+}
 
 
 class WeaviateDocumentStoreError(DocumentStoreError):
@@ -391,13 +405,21 @@ def _sanitize_id(self, id: str, index: Optional[str] = None) -> str:
 
     def _get_current_properties(self, index: Optional[str] = None) -> List[str]:
         """
-        Get all the existing properties in the schema.
+        Get all the existing properties in the schema, excluding those with complex types
+        like cross-reference
         """
         index = self._sanitize_index_name(index) or self.index
         cur_properties = []
         for class_item in self.weaviate_client.schema.get()["classes"]:
             if class_item["class"] == index:
-                cur_properties = [item["name"] for item in class_item["properties"]]
+                cur_properties = [
+                    item["name"]
+                    for item in class_item.get("properties", [])
+                    # dataType should be always there and contain only one item unless
+                    # it's a cross-reference but here we try to be defensive against
+                    # unexpected schemas
+                    if set(item.get("dataType", [])).issubset(SUPPORTED_PROPERTY_TYPES)
+                ]
 
         return cur_properties
 


Test patch:
diff --git a/test/document_stores/test_weaviate.py b/test/document_stores/test_weaviate.py
--- a/test/document_stores/test_weaviate.py
+++ b/test/document_stores/test_weaviate.py
@@ -1,16 +1,14 @@
+import uuid
+import json
+from unittest import mock
+
 import pytest
+import numpy as np
 
 from haystack.document_stores.weaviate import WeaviateDocumentStore
 from haystack.schema import Document
 from haystack.testing import DocumentStoreBaseTestAbstract
 
-import uuid
-from unittest.mock import MagicMock
-
-import numpy as np
-import pytest
-
-from haystack.schema import Document
 
 embedding_dim = 768
 
@@ -207,7 +205,7 @@ def test_deleting_by_id_or_by_filters(self, ds, documents):
         ds.write_documents(documents)
         # This test verifies that deleting an object by its ID does not first require fetching all documents. This fixes
         # a bug, as described in https://github.com/deepset-ai/haystack/issues/2898
-        ds.get_all_documents = MagicMock(wraps=ds.get_all_documents)
+        ds.get_all_documents = mock.MagicMock(wraps=ds.get_all_documents)
 
         assert ds.get_document_count() == 9
 
@@ -258,3 +256,30 @@ def test_get_embedding_count(self, ds, documents):
         """
         ds.write_documents(documents)
         assert ds.get_embedding_count() == 9
+
+    @pytest.mark.unit
+    def test__get_current_properties(self):
+        with mock.patch("haystack.document_stores.weaviate.client") as mocked_client:
+            mocked_client.Client().is_ready.return_value = True
+            mocked_client.Client().schema.contains.return_value = False
+            mocked_client.Client().schema.get.return_value = json.loads(
+                """
+{
+  "classes": [{
+    "class": "Document",
+    "properties": [
+        {
+        "name": "hasWritten",
+        "dataType": ["Article"]
+        },
+        {
+        "name": "hitCounter",
+        "dataType": ["int"]
+        }
+    ]
+  }]
+} """
+            )
+            ds = WeaviateDocumentStore()
+            # Ensure we dropped the cross-reference property
+            assert ds._get_current_properties() == ["hitCounter"]