Repository: deepset-ai/haystack

Issue:
TransformersDocumentClassifier results can't be written back to SQL based DocumentStores
This issue is similar to the issue reported in #2792 

The `TransformersDocumentClassifier` results are added to metadata as dictionary within dictionary. However, SQL based document stores don't accept them, they seem to only expect key-string pairs in metadata.  Which means you can't re-write the results from the classifier back to your document store.

@Zeni69 had this issue with `FAISSDocumentStore`. The error message when you call document_store.write_documents() is below.

**Error message**
```
(sqlite3.InterfaceError) Error binding parameter 2 - probably unsupported type.
[SQL: INSERT INTO meta_document (id, name, value, document_id, document_index) VALUES (?, ?, ?, ?, ?)]
[parameters: ('e0bf178a-7318-449c-b5bc-95c57cdd0811', 'classification', {'sequence': '"Deep Frying Toxins" Frying is not just bad because it destroys more nutrition, and oil is such a waste of calories. We should never dee ... (441 characters truncated) ... nut, safflower, or', 'labels': ['plant', 'meat', 'dairy'], 'scores': [0.4199756383895874, 0.37468278408050537, 0.20534150302410126], 'label': 'plant'}, '431e5e627e22b35db8455b48368066e1', 'document')]
(Background on this error at: https://sqlalche.me/e/14/rvf5) (Background on this error at: https://sqlalche.me/e/14/7s2a)
```

**Additional context**
The reported workaround was re-formatting the resulting documents before writing them back to document_store. The reformatting was done as follows:

```
new_docs = list()
for doc in dtemp:
  tempdoc = dict()
  tempdoc['content'] = doc.content
  tempdoc['content_type'] = doc.content_type
  tempdoc['score'] = doc.score
  tempdoc['meta'] = {'name': doc.meta['name'], 'label':(doc.meta['classification'])['label'], '_split_id':doc.meta['_split_id']}
  tempdoc['embedding'] = doc.embedding
  tempdoc['id'] = doc.id
  new_docs.append(tempdoc)
```

**To Reproduce**
Use `TransformersDocumentClassifier` and write the results to a document store like `FAISSDocumentStore`:

```
document_store = FAISSDocumentStore(faiss_index_factory_str="Flat", embedding_dim=128)

from haystack.nodes import TransformersDocumentClassifier

doc_classifier = TransformersDocumentClassifier(
    model_name_or_path="facebook/bart-large-mnli",
    task="zero-shot-classification",
    labels=["meat", "plant", "dairy"],
    batch_size=16,
)
classified_docs = doc_classifier.predict_batch(docs2)
retriever = EmbeddingRetriever(
    document_store=document_store,
    embedding_model="vblagoje/retribert-base-uncased",
)
document_store.delete_documents()
document_store.write_documents(docs2)
```

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?




Discussion:
The problem seems clear: we are trying to store into SQL-based DocumentStores some unsuitable data, expressed in a nested format (as deeply analyzed in #2792).

I see two possible solutions, each with some drawbacks:

1. Write the `meta classification` in a different flat format, only if the used `DocumentStore` is SQL-based.
This doesn't seem optimal, because we then must consider this different behavior when filtering.

2. Write the `meta classification` in a different flat format, for every `DocumentStore`.
I like this approach more. I don't know if it can be considered a breaking change, but it would certainly represent a discontinuity from the past.

What do you think, guys of the Deepset team? Any better insights/advice?
@ZanSara 

Hey @anakin87, your analysis is totally correct: the way to overcome this issue is to flatten the metadata in such a way that an SQL database will accept it.

However, at a first glance I believe it would be better to implement this flattening only in docstores that require it, because that's only one: `SQLDocumentStore`. `FAISSDocumentStore` is just a subclass of it, so it will inherit the new behavior.

Plus, both metadata writing and metadata filtering are already done differently for each docstore, do introducing this difference for `SQLDocumentStore` won't be awkward.

It seems to be a quite straightforward bug. If you decide to submit a PR please tag me and I'll take care of reviewing it :blush:
While thinking about this issue, I discovered a problem with inconsistent output for `TransformersDocumentClassifier`: https://github.com/deepset-ai/haystack/issues/3167.

Resolving patch:
diff --git a/haystack/document_stores/sql.py b/haystack/document_stores/sql.py
--- a/haystack/document_stores/sql.py
+++ b/haystack/document_stores/sql.py
@@ -399,6 +399,8 @@ def write_documents(
             docs_orm = []
             for doc in document_objects[i : i + batch_size]:
                 meta_fields = doc.meta or {}
+                if "classification" in meta_fields:
+                    meta_fields = self._flatten_classification_meta_fields(meta_fields)
                 vector_id = meta_fields.pop("vector_id", None)
                 meta_orms = []
                 for key, value in meta_fields.items():
@@ -785,3 +787,14 @@ def _windowed_query(self, q, column, windowsize):
         for whereclause in self._column_windows(q.session, column, windowsize):
             for row in q.filter(whereclause).order_by(column):
                 yield row
+
+    def _flatten_classification_meta_fields(self, meta_fields: dict) -> dict:
+        """
+        Since SQLDocumentStore does not support dictionaries for metadata values,
+        the DocumentClassifier output is flattened
+        """
+        meta_fields["classification.label"] = meta_fields["classification"]["label"]
+        meta_fields["classification.score"] = meta_fields["classification"]["score"]
+        meta_fields["classification.details"] = str(meta_fields["classification"]["details"])
+        del meta_fields["classification"]
+        return meta_fields


Test patch:
diff --git a/test/conftest.py b/test/conftest.py
--- a/test/conftest.py
+++ b/test/conftest.py
@@ -106,24 +106,6 @@
 requests_cache.install_cache(urls_expire_after={"huggingface.co": timedelta(hours=1), "*": requests_cache.DO_NOT_CACHE})
 
 
-def _sql_session_rollback(self, attr):
-    """
-    Inject SQLDocumentStore at runtime to do a session rollback each time it is called. This allows to catch
-    errors where an intended operation is still in a transaction, but not committed to the database.
-    """
-    method = object.__getattribute__(self, attr)
-    if callable(method):
-        try:
-            self.session.rollback()
-        except AttributeError:
-            pass
-
-    return method
-
-
-SQLDocumentStore.__getattribute__ = _sql_session_rollback
-
-
 def pytest_collection_modifyitems(config, items):
     # add pytest markers for tests that are not explicitly marked but include some keywords
     name_to_markers = {
diff --git a/test/document_stores/test_document_store.py b/test/document_stores/test_document_store.py
--- a/test/document_stores/test_document_store.py
+++ b/test/document_stores/test_document_store.py
@@ -99,7 +99,6 @@ def test_write_with_duplicate_doc_ids_custom_index(document_store: BaseDocumentS
 
 
 def test_get_all_documents_without_filters(document_store_with_docs):
-    print("hey!")
     documents = document_store_with_docs.get_all_documents()
     assert all(isinstance(d, Document) for d in documents)
     assert len(documents) == 5
diff --git a/test/document_stores/test_sql.py b/test/document_stores/test_sql.py
--- a/test/document_stores/test_sql.py
+++ b/test/document_stores/test_sql.py
@@ -62,6 +62,42 @@ def test_sql_write_different_documents_same_vector_id(self, ds):
         with pytest.raises(Exception, match=r"(?i)unique"):
             ds.write_documents([doc2], index="index3")
 
+    @pytest.mark.integration
+    def test_sql_get_documents_using_nested_filters_about_classification(self, ds):
+        documents = [
+            Document(
+                content="That's good. I like it.",
+                id="1",
+                meta={
+                    "classification": {
+                        "label": "LABEL_1",
+                        "score": 0.694,
+                        "details": {"LABEL_1": 0.694, "LABEL_0": 0.306},
+                    }
+                },
+            ),
+            Document(
+                content="That's bad. I don't like it.",
+                id="2",
+                meta={
+                    "classification": {
+                        "label": "LABEL_0",
+                        "score": 0.898,
+                        "details": {"LABEL_0": 0.898, "LABEL_1": 0.102},
+                    }
+                },
+            ),
+        ]
+        ds.write_documents(documents)
+
+        assert ds.get_document_count() == 2
+        assert len(ds.get_all_documents(filters={"classification.score": {"$gt": 0.1}})) == 2
+        assert len(ds.get_all_documents(filters={"classification.label": ["LABEL_1", "LABEL_0"]})) == 2
+        assert len(ds.get_all_documents(filters={"classification.score": {"$gt": 0.8}})) == 1
+        assert len(ds.get_all_documents(filters={"classification.label": ["LABEL_1"]})) == 1
+        assert len(ds.get_all_documents(filters={"classification.score": {"$gt": 0.95}})) == 0
+        assert len(ds.get_all_documents(filters={"classification.label": ["LABEL_100"]})) == 0
+
     # NOTE: the SQLDocumentStore behaves differently to the others when filters are applied.
     # While this should be considered a bug, the relative tests are skipped in the meantime