Repository: deepset-ai/haystack

Issue:
Metadata are not written to SQL Document store when duplicate_documents!="overwrite"
**Describe the bug**
It seems that in `SQLDocumentStore`, if you call `write_documents` with skip/fail values for the `duplicate_documents` parameter, document metadata are not written to DB.

**Error message**
No error

**Expected behavior**
The metadata should be correctly saved.

**To Reproduce**
```python
from haystack.document_stores.sql import SQLDocumentStore
from haystack.schema import Document

ds = SQLDocumentStore()
documents = [
    Document(id="1", content="content", meta={"meta_field": "test1"})
]
ds.write_documents(documents, duplicate_documents="fail") # with 'skip', there is the same behavior
print(ds.get_document_by_id("1").to_dict())

# {'content': 'content', 'content_type': 'text', 'score': None, 'meta': {}, 'embedding': None, 'id': '1'}

```

**Additional context**
Seems related to these lines of code:
https://github.com/deepset-ai/haystack/blob/af96e002a4f2fbf6c844252c238684c441eedeef/haystack/document_stores/sql.py#L409-L426

Over time, this part has undergone several changes for performance reasons. Probably something has gone out of control...
A solution should be found that combines correct functionality with performance.

**FAQ Check**
- [X] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: Ubuntu 22.04
 - GPU/CPU: CPU
 - Haystack version (commit or version number): 1.10.0
 - DocumentStore: SQLDocumentStore



Discussion:
IMO, the issue seems related to `self.session.bulk_insert_mappings(DocumentORM, docs_orm)`
From [SQLAlchemy docs](https://docs.sqlalchemy.org/en/14/orm/session_api.html#sqlalchemy.orm.Session.bulk_insert_mappings):

> The bulk insert feature allows plain Python dictionaries to be used as the source of simple INSERT operations which can be more easily grouped together into higher performing “executemany” operations. ...
> The values within the dictionaries as given are typically passed without modification into Core constructs, after organizing the values within them across the tables to which the given mapper is mapped.

In the reported example, `docs_orm` is:
`[{'id': '1', 'content': 'content', 'content_type': 'text', 'vector_id': None, 'meta': [<haystack.document_stores.sql.MetaDocumentORM object at 0x7fd9ed8d1590>, <haystack.document_stores.sql.MetaDocumentORM object at 0x7fd9ed5d6ad0>], 'index': ...}]`

`meta` is composed of MetaDocumentORM objects (corresponding to `name` and `meta_field`). And all this metadata is lost: the meta field names and values don't even appear in SQLAlchemy logs.

- closely related PR: #2264
We can find a better way to perform this bulk operation. Otherwise, it seems better to revert this change.

- [interesting Medium post about SQLAlchemy bulk inserts](https://towardsdatascience.com/how-to-perform-bulk-inserts-with-sqlalchemy-efficiently-in-python-23044656b97d)
  To perform a bulk operation that supports objects instead of plain dictionaries, `bulk_save_objects` seems a reasonable compromise (slower but safer).
I would like to open a PR that tries to refactor the way documents are saved to DB,
taking into account two aspects:

- correctness (all documents and metadata are properly stored)
- performance (bulk operations are preferable to single ones)

@ZanSara and Deepset team, I would like to know your opinion first and collect useful advice.

Resolving patch:
diff --git a/haystack/document_stores/sql.py b/haystack/document_stores/sql.py
--- a/haystack/document_stores/sql.py
+++ b/haystack/document_stores/sql.py
@@ -406,24 +406,23 @@ def write_documents(
                         meta_orms.append(MetaDocumentORM(name=key, value=value))
                     except TypeError as ex:
                         logger.error("Document %s - %s", doc.id, ex)
-                doc_mapping = {
-                    "id": doc.id,
-                    "content": doc.to_dict()["content"],
-                    "content_type": doc.content_type,
-                    "vector_id": vector_id,
-                    "meta": meta_orms,
-                    "index": index,
-                }
+                doc_orm = DocumentORM(
+                    id=doc.id,
+                    content=doc.to_dict()["content"],
+                    content_type=doc.content_type,
+                    vector_id=vector_id,
+                    meta=meta_orms,
+                    index=index,
+                )
                 if duplicate_documents == "overwrite":
-                    doc_orm = DocumentORM(**doc_mapping)
                     # First old meta data cleaning is required
                     self.session.query(MetaDocumentORM).filter_by(document_id=doc.id).delete()
                     self.session.merge(doc_orm)
                 else:
-                    docs_orm.append(doc_mapping)
+                    docs_orm.append(doc_orm)
 
             if docs_orm:
-                self.session.bulk_insert_mappings(DocumentORM, docs_orm)
+                self.session.add_all(docs_orm)
 
             try:
                 self.session.commit()


Test patch:
diff --git a/test/document_stores/test_base.py b/test/document_stores/test_base.py
--- a/test/document_stores/test_base.py
+++ b/test/document_stores/test_base.py
@@ -82,11 +82,12 @@ def test_write_labels(self, ds, labels):
     @pytest.mark.integration
     def test_write_with_duplicate_doc_ids(self, ds):
         duplicate_documents = [
-            Document(content="Doc1", id_hash_keys=["content"]),
-            Document(content="Doc1", id_hash_keys=["content"]),
+            Document(content="Doc1", id_hash_keys=["content"], meta={"key1": "value1"}),
+            Document(content="Doc1", id_hash_keys=["content"], meta={"key1": "value1"}),
         ]
         ds.write_documents(duplicate_documents, duplicate_documents="skip")
         assert len(ds.get_all_documents()) == 1
+        assert ds.get_all_documents()[0] == duplicate_documents[0]
         with pytest.raises(Exception):
             ds.write_documents(duplicate_documents, duplicate_documents="fail")