Repository: deepset-ai/haystack

Issue:
Duplicate document bulk index error with Faiss document store
Hi,

Hope you are all well !

I think that a parameter is missing for the faiss document store while writing documents like you made for elasticesarch.        

:param update_existing_documents: Whether to update any existing documents with the same ID when adding
                                          documents. When set as True, any document with an existing ID gets updated.
                                          If set to False, an error is raised if the document ID of the document being
                                          added already exists.

Because, I tried to load some documents and I have the following error preventing me to load the full dataset into the faiss store.

```sh
Error:This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry '52713153-stackoverflow.com' for key 'PRIMARY'")
[SQL: INSERT INTO document (id, text, `index`, vector_id) VALUES (%s, %s, %s, %s)]
[parameters: (('52713148-stackoverflow.com', b'Question: \nBehaviorSubject next method not updating changes for another component \n <p>Hello I have developed user service, working of that user s ... (63 characters truncated) ... lationship with them. if i am changing data from peoplecompoent it is not detected by inboxcomponent. Here i am posnting code for \n\n Answer: \nBody', 'document', None), ('52713149-stackoverflow.com', b"Question: \nposition of floating button changes with number of elements in flatlist in react native \n <p>I have a flatlist below which I need to pl ... (80 characters truncated) ... bottom edge of the screen; instead, I want the flatlist window to be a fixed size and scrollable to accommodate more rows. Below t\n\n Answer: \nBody", 'document', None), ('52713150-stackoverflow.com', b"Question: \nRails 6, Devise and Active Storage \n <p>I'm using Devise and Active Storage in my rails 6 application. Currently using <code>before_act ... (27 characters truncated) ... de> disables all images for non users, which is great, except I want to allow images with the <code>record_type</code> 'News' in m\n\n Answer: \nBody", 'document', None), ('52713152-stackoverflow.com', b'Question: \nfish shell shows error in piping - new error \n "<p><a href=""https://i.stack.imgur.com/wyBue.jpg"" rel=""nofollow noreferrer""><img src ... (37 characters truncated) ... g"" alt=""enter image description here"" /></a>I am getting multiple errors from shell init scripts which more or less seem the sa\n\n Answer: \nBody', 'document', None), ('52713153-stackoverflow.com', b'Question: \nApplying CSS to Datatables that has NEXT Page \n "<p>I have this Script that works and does what I need, but it will only apply to the f ... (289 characters truncated) ... our $.each loop each time the page changes or (I think better) use <a href=""https://datatables.net/reference/option/createdRow"" rel=""nofollow nore', 'document', None), ('52713154-stackoverflow.com', b'Question: \nNumber of rows in a datetime index sampled by day \n "<p>I am trying to count the number of rows of the index in Data frame X and then g ... (42 characters truncated) ... as Data frame Y showing 11 rows on 7/10/2020 and 12 rows in 7/11/2020</p> <p><img src=""https://i.stack.imgur.com/rLX5o.jpg"" alt=\n\n Answer: \nBody', 'document', None), ('52713157-stackoverflow.com', b'Question: \nIs it possible to turn a gameObject (using DontDestroyOnLoad()) into a child of a gameObject from another scene? \n <p>I am making a 2d  ... (127 characters truncated) ... ite Renderer (in the game scene) is enabled and all other sprites are disabled... so I made an empty object containing the skins (\n\n Answer: \nBody', 'document', None), ('52713159-stackoverflow.com', b'Question: \nRobotframework for loop continue with next test \n <p>I have below code:</p> <pre><code>*** Settings *** Library     OperatingSystem Lib ... (291 characters truncated) ... ramework.org/robotframework/latest/RobotFrameworkUserGuide.html#templates-with-for-loops"" rel=""nofollow noreferrer"">Templates with for loops</a>, ', 'document', None)  ... displaying 10 of 500 total bound parameter sets ...  ('52714181-stackoverflow.com', b'Question: \nLogin & registration redirects - don\'t redirect login when on checkout page in WooCommerce \n "<p>My code works as expected but it shou ... (84 characters truncated) ... code>is_checkout()</code> to try and detect this but it is not working as expected.</p> <pre class=""lang-php prettyprint-override\n\n Answer: \nBody', 'document', None), ('52714183-stackoverflow.com', b'Question: \nHow to activate autocomplete option of tomtom in my html code? \n "<p>I have the following simple code for using tomtom which works:</p> ... (61 characters truncated) ... \'use-all-space\'&gt;     &lt;head&gt;     &lt;meta http-equiv=\'X-UA-Compatible\' content=\'IE=Edge\' /&gt;     &lt;meta charset=\n\n Answer: \nBody', 'document', None))]
(Background on this error at: http://sqlalche.me/e/13/gkpj) (Background on this error at: http://sqlalche.me/e/13/7s2a)
document_store.write_documents=1557
Bulk not indexed
```

Cheers,
X


Discussion:
@tholor Any ideas how to fix it ?
Hey @x0rzkov, 

Sorry, it's been crazily busy days on our side. It's correct that FAISSDocumentStore does not support `update_existing_documents` yet. What is your intended workflow here? Do you really want to update a subset of your existing documents?

If you are just trying to load documents and face this error, it probably means that you still have documents in SQL from a previous run while the FAISS index does not know about them. You can call  `document_store.delete_all_documents()` or manually drop the tables in SQL. 
Hi @tholor ,

Hope you are all well !

I am converting the xml stackoverflow dump, extracting pairs of question with their validated answers. Each time, I insert those pairs in the db and update the delta of pairs that have changed.

Would be also convenient to have an ignore flag if duplicate meanwhile you implement the ```update_existing_documents``` feature ^^

Is there a quick workaround ?

Cheers,
X
@x0rzkov Two dirty hack you can do -
1) If you want to update docs on duplicate then, change `session.add(` to `session.merge(` at this place https://github.com/deepset-ai/haystack/blob/master/haystack/document_store/sql.py#L135
2) If you want to skip on duplicate then, wrap this line under `try/catch`. https://github.com/deepset-ai/haystack/blob/master/haystack/document_store/sql.py#L135

For more information please refer https://stackoverflow.com/questions/6611563/sqlalchemy-on-duplicate-key-update#:~:text=ON%20DUPLICATE%20KEY%20UPDATE%20functionality,question%20is%20a%20primary%20key.
I forgot to mention that in either hack you have to clear `faiss_index` before calling `update_embeddings` function. You can do as follows -
```
faiss_document_store.faiss_index.reset()
```

@lalitpagaria 

Hope you are all well !

Thanks, I ll have a try this morning ^^

Resolving patch:
diff --git a/haystack/document_store/faiss.py b/haystack/document_store/faiss.py
--- a/haystack/document_store/faiss.py
+++ b/haystack/document_store/faiss.py
@@ -37,6 +37,8 @@ def __init__(
         faiss_index_factory_str: str = "Flat",
         faiss_index: Optional[faiss.swigfaiss.Index] = None,
         return_embedding: Optional[bool] = True,
+        update_existing_documents: bool = False,
+        index: str = "document",
         **kwargs,
     ):
         """
@@ -63,6 +65,11 @@ def __init__(
         :param faiss_index: Pass an existing FAISS Index, i.e. an empty one that you configured manually
                             or one with docs that you used in Haystack before and want to load again.
         :param return_embedding: To return document embedding
+        :param update_existing_documents: Whether to update any existing documents with the same ID when adding
+                                          documents. When set as True, any document with an existing ID gets updated.
+                                          If set to False, an error is raised if the document ID of the document being
+                                          added already exists.
+        :param index: Name of index in document store to use.
         """
         self.vector_dim = vector_dim
 
@@ -73,7 +80,11 @@ def __init__(
 
         self.index_buffer_size = index_buffer_size
         self.return_embedding = return_embedding
-        super().__init__(url=sql_url)
+        super().__init__(
+            url=sql_url,
+            update_existing_documents=update_existing_documents,
+            index=index
+        )
 
     def _create_new_index(self, vector_dim: int, index_factory: str = "Flat", metric_type=faiss.METRIC_INNER_PRODUCT, **kwargs):
         if index_factory == "HNSW" and metric_type == faiss.METRIC_INNER_PRODUCT:
@@ -99,12 +110,18 @@ def write_documents(self, documents: Union[List[dict], List[Document]], index: O
         # vector index
         if not self.faiss_index:
             raise ValueError("Couldn't find a FAISS index. Try to init the FAISSDocumentStore() again ...")
+
         # doc + metadata index
         index = index or self.index
         document_objects = [Document.from_dict(d) if isinstance(d, dict) else d for d in documents]
 
         add_vectors = False if document_objects[0].embedding is None else True
 
+        if self.update_existing_documents and add_vectors:
+            logger.warning("You have enabled `update_existing_documents` feature and "
+                           "`FAISSDocumentStore` does not support update in existing `faiss_index`.\n"
+                           "Please call `update_embeddings` method to repopulate `faiss_index`")
+
         for i in range(0, len(document_objects), self.index_buffer_size):
             vector_id = self.faiss_index.ntotal
             if add_vectors:
@@ -134,6 +151,9 @@ def update_embeddings(self, retriever: BaseRetriever, index: Optional[str] = Non
         if not self.faiss_index:
             raise ValueError("Couldn't find a FAISS index. Try to init the FAISSDocumentStore() again ...")
 
+        # Faiss does not support update in existing index data so clear all existing data in it
+        self.faiss_index.reset()
+
         index = index or self.index
         documents = self.get_all_documents(index=index)
 
diff --git a/haystack/document_store/sql.py b/haystack/document_store/sql.py
--- a/haystack/document_store/sql.py
+++ b/haystack/document_store/sql.py
@@ -1,3 +1,4 @@
+import logging
 from typing import Any, Dict, Union, List, Optional
 from uuid import uuid4
 
@@ -10,6 +11,10 @@
 from haystack import Document, Label
 from haystack.preprocessor.utils import eval_data_from_file
 
+
+logger = logging.getLogger(__name__)
+
+
 Base = declarative_base()  # type: Any
 
 
@@ -37,7 +42,7 @@ class MetaORM(ORMBase):
 
     name = Column(String(100), index=True)
     value = Column(String(1000), index=True)
-    document_id = Column(String(100), ForeignKey("document.id", ondelete="CASCADE"), nullable=False)
+    document_id = Column(String(100), ForeignKey("document.id", ondelete="CASCADE", onupdate="CASCADE"), nullable=False)
 
     documents = relationship(DocumentORM, backref="Meta")
 
@@ -45,7 +50,7 @@ class MetaORM(ORMBase):
 class LabelORM(ORMBase):
     __tablename__ = "label"
 
-    document_id = Column(String(100), ForeignKey("document.id", ondelete="CASCADE"), nullable=False)
+    document_id = Column(String(100), ForeignKey("document.id", ondelete="CASCADE", onupdate="CASCADE"), nullable=False)
     index = Column(String(100), nullable=False)
     no_answer = Column(Boolean, nullable=False)
     origin = Column(String(100), nullable=False)
@@ -58,13 +63,30 @@ class LabelORM(ORMBase):
 
 
 class SQLDocumentStore(BaseDocumentStore):
-    def __init__(self, url: str = "sqlite://", index="document"):
+    def __init__(
+        self,
+        url: str = "sqlite://",
+        index: str = "document",
+        label_index: str = "label",
+        update_existing_documents: bool = False,
+    ):
+        """
+        :param url: URL for SQL database as expected by SQLAlchemy. More info here: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls
+        :param index: The documents are scoped to an index attribute that can be used when writing, querying, or deleting documents. 
+                      This parameter sets the default value for document index.
+        :param label_index: The default value of index attribute for the labels.
+        :param update_existing_documents: Whether to update any existing documents with the same ID when adding
+                                          documents. When set as True, any document with an existing ID gets updated.
+                                          If set to False, an error is raised if the document ID of the document being
+                                          added already exists. Using this parameter coud cause performance degradation for document insertion. 
+        """
         engine = create_engine(url)
         ORMBase.metadata.create_all(engine)
         Session = sessionmaker(bind=engine)
         self.session = Session()
         self.index = index
-        self.label_index = "label"
+        self.label_index = label_index
+        self.update_existing_documents = update_existing_documents
 
     def get_document_by_id(self, id: str, index: Optional[str] = None) -> Optional[Document]:
         documents = self.get_documents_by_id([id], index)
@@ -132,8 +154,19 @@ def write_documents(self, documents: Union[List[dict], List[Document]], index: O
             vector_id = meta_fields.get("vector_id")
             meta_orms = [MetaORM(name=key, value=value) for key, value in meta_fields.items()]
             doc_orm = DocumentORM(id=doc.id, text=doc.text, vector_id=vector_id, meta=meta_orms, index=index)
-            self.session.add(doc_orm)
-        self.session.commit()
+            if self.update_existing_documents:
+                # First old meta data cleaning is required
+                self.session.query(MetaORM).filter_by(document_id=doc.id).delete()
+                self.session.merge(doc_orm)
+            else:
+                self.session.add(doc_orm)
+        try:
+            self.session.commit()
+        except Exception as ex:
+            logger.error(f"Transaction rollback: {ex.__cause__}")
+            # Rollback is important here otherwise self.session will be in inconsistent state and next call will fail
+            self.session.rollback()
+            raise ex
 
     def write_labels(self, labels, index=None):
 


Test patch:
diff --git a/test/test_db.py b/test/test_db.py
--- a/test/test_db.py
+++ b/test/test_db.py
@@ -4,7 +4,6 @@
 
 from haystack import Document, Label
 from haystack.document_store.elasticsearch import ElasticsearchDocumentStore
-from haystack.document_store.faiss import FAISSDocumentStore
 
 
 @pytest.mark.elasticsearch
@@ -85,6 +84,36 @@ def test_get_document_count(document_store):
     assert document_store.get_document_count(filters={"meta_field_for_count": ["b"]}) == 3
 
 
+@pytest.mark.elasticsearch
+@pytest.mark.parametrize("document_store", ["elasticsearch", "sql", "faiss"], indirect=True)
+@pytest.mark.parametrize("update_existing_documents", [True, False])
+def test_update_existing_documents(document_store, update_existing_documents):
+    original_docs = [
+        {"text": "text1_orig", "id": "1", "meta_field_for_count": "a"},
+    ]
+
+    updated_docs = [
+        {"text": "text1_new", "id": "1", "meta_field_for_count": "a"},
+    ]
+
+    document_store.update_existing_documents = update_existing_documents
+    document_store.write_documents(original_docs)
+    assert document_store.get_document_count() == 1
+
+    if update_existing_documents:
+        document_store.write_documents(updated_docs)
+    else:
+        with pytest.raises(Exception):
+            document_store.write_documents(updated_docs)
+
+    stored_docs = document_store.get_all_documents()
+    assert len(stored_docs) == 1
+    if update_existing_documents:
+        assert stored_docs[0].text == updated_docs[0]["text"]
+    else:
+        assert stored_docs[0].text == original_docs[0]["text"]
+
+
 @pytest.mark.elasticsearch
 def test_write_document_meta(document_store):
     documents = [
@@ -112,9 +141,8 @@ def test_write_document_index(document_store):
     document_store.write_documents([documents[0]], index="haystack_test_1")
     assert len(document_store.get_all_documents(index="haystack_test_1")) == 1
 
-    if not isinstance(document_store, FAISSDocumentStore):  # addition of more documents is not supported in FAISS
-        document_store.write_documents([documents[1]], index="haystack_test_2")
-        assert len(document_store.get_all_documents(index="haystack_test_2")) == 1
+    document_store.write_documents([documents[1]], index="haystack_test_2")
+    assert len(document_store.get_all_documents(index="haystack_test_2")) == 1
 
     assert len(document_store.get_all_documents(index="haystack_test_1")) == 1
     assert len(document_store.get_all_documents()) == 0