Repository: deepset-ai/haystack

Issue:
Using multiple indices in FAISS seems broken
**Describe the bug**
I'm trying to use FAISS document store and want to store multiple use cases in the same document store, i.e. specify different "index" during writing and querying. Everything works as expected for the first index I create, but for the subsequent ones, the query results are unexpectedly limited. 

I looked into it, and I think what's happening is: 

in the FAISS db, documents in each new index are keyed by vector_id, and this gets reset to 0 in each index. However, in the SQL db, the "vector_id" key isn't reset to 0 for each index. So when `get_documents_by_vector_ids` gets called with the vector_id's from the FAISS lookup, you get the wrong results, and most likely no results at all, when querying any of the subsequently created indices.

**Error message**
No error

**Expected behavior**
The vector ids in the SQL db should match those in the FAISS db for a given index.

**Additional context**
Add any other context about the problem here, like document types / preprocessing steps / settings of reader etc.

**To Reproduce**

```python
from haystack.document_stores import FAISSDocumentStore
from haystack.nodes import EmbeddingRetriever
from haystack.pipelines import DocumentSearchPipeline
document_store = FAISSDocumentStore(sql_url="sqlite://", similarity='cosine', embedding_dim=384)
retriever = EmbeddingRetriever(document_store=document_store, embedding_model='sentence-transformers/all-MiniLM-L6-v2')
pipeline = DocumentSearchPipeline(retriever)                   

#vector_id's in the SQL db vs FAISS db (maybe not exact order)
document_store.write_documents([{"content":"dogs are great"},  #vector_id:0 vs 0
                                {"content":"dogs are OK"},      #vector_id:1 vs 1
                                {"content":"cats are awful"}],    #vector_id:2 vs 2
                                    index="index1")
document_store.write_documents([{"content":"I love cake"},       #vector_id:3 vs 0
                                {"content":"I like cake"},     #vector_id:4 vs 1
                                {"content":"I hate cake"}],    #vector_id:5 vs 2
                                index="index2")

document_store.update_embeddings(retriever, index="index1")
document_store.update_embeddings(retriever, index="index2")

print(len(pipeline.run("I love dogs", params={"Retriever": {"index": "index1", "top_k":1}})['documents']))
# => 1 
print(len(pipeline.run("I like cake", params={"Retriever": {"index": "index2", "top_k":1}})['documents']))
# => 0
```


**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: MacOS 12.6
 - GPU/CPU: CPU
 - Haystack version (commit or version number): 1.9.0
 - DocumentStore: FAISSDocumentStore
 - Reader: None
 - Retriever: EmbeddingRetriever



Discussion:
Hi @einarbmag I was able to reproduce the bug that you described. Thank you for the detailed description. I think you are right about why the bug occurs, which already points to a way to fix it. 🙂 If you want, you would be more than welcome to try to create a pull request with a fix yourself. Otherwise, I'll discuss with the team how to address this issue in our next sprint.
Hey guys, as you know, I like to :pick: dig deep into code history.

### `vector_id` is unique
In  #556, `vector_id` was defined as unique:
https://github.com/deepset-ai/haystack/blob/db6e5754cd71dd822f466bebe3882dc61de00c44/haystack/document_stores/sql.py#L55

**Is this behavior still desired?** I don't see why `vector_id` with the same value cannot exist in different indexes. But I'm not aware of all the internals...

Anyway, this uniqueness generated bugs, as reported in #1954.
So, in #1961, in `update_embeddings`, `vector_id = self.faiss_indexes[index].ntotal` was replaced by line 347: https://github.com/deepset-ai/haystack/blob/db6e5754cd71dd822f466bebe3882dc61de00c44/haystack/document_stores/faiss.py#L331-L347

### Current bug
It is caused by `vector_id = sum(index.ntotal for index in self.faiss_indexes.values()) `.
In fact, in `update_embeddings`, `index2` is reset, but `vector_id` starts from 3, making the retrieval ineffective.

### Possible solutions
If you agree that `vector_id` may not be unique, a quite easy solution is to revert #1961.

In any case, I am eager to hear your opinion and I am ready for the challenge of opening a PR :smile:
@ZanSara any thoughts? 



Hey @anakin87! :blush: Thank you once more for this research! I believe your analysis is correct and I don't see why we should have such mismatch between the `vector_id` in FAISS and in SQL. I'd imagine this was an early oversight and, as you discovered, it was later worked around.

So:
1. Yes, let's remove the UNIQUE constraint on vector_id and revert #1961

2. Then, let's test. Can we add one or few tests making sure the vector_id behave correctly?

3. We should also run these tests on something else than SQLite to make sure it all works there. Feel free to use either PostgreSQL or MySQL or anything else similar, just not SQLite. I want to underline this because unfortunately SQLite does not really care about foreign key relationships, so if this uniqueness was used somehow, SQLite won't always complain.
   - Testing on DBs other than SQLite is... weird with our test suite. Try using this line here and let me know if you face issues or you observe other tests failing. Currently we run them on SQLite only :see_no_evil: https://github.com/deepset-ai/haystack/blob/main/test/conftest.py#L89



Resolving patch:
diff --git a/haystack/document_stores/faiss.py b/haystack/document_stores/faiss.py
--- a/haystack/document_stores/faiss.py
+++ b/haystack/document_stores/faiss.py
@@ -344,7 +344,7 @@ def update_embeddings(
             return
 
         logger.info("Updating embeddings for %s docs...", document_count)
-        vector_id = sum(index.ntotal for index in self.faiss_indexes.values())
+        vector_id = self.faiss_indexes[index].ntotal
 
         result = self._query(
             index=index,
diff --git a/haystack/document_stores/sql.py b/haystack/document_stores/sql.py
--- a/haystack/document_stores/sql.py
+++ b/haystack/document_stores/sql.py
@@ -19,6 +19,7 @@
         text,
         JSON,
         ForeignKeyConstraint,
+        UniqueConstraint,
     )
     from sqlalchemy.ext.declarative import declarative_base
     from sqlalchemy.orm import relationship, sessionmaker, validates
@@ -52,10 +53,12 @@ class DocumentORM(ORMBase):
     content_type = Column(Text, nullable=True)
     # primary key in combination with id to allow the same doc in different indices
     index = Column(String(100), nullable=False, primary_key=True)
-    vector_id = Column(String(100), unique=True, nullable=True)
+    vector_id = Column(String(100), nullable=True)
     # speeds up queries for get_documents_by_vector_ids() by having a single query that returns joined metadata
     meta = relationship("MetaDocumentORM", back_populates="documents", lazy="joined")
 
+    __table_args__ = (UniqueConstraint("index", "vector_id", name="index_vector_id_uc"),)
+
 
 class MetaDocumentORM(ORMBase):
     __tablename__ = "meta_document"


Test patch:
diff --git a/test/conftest.py b/test/conftest.py
--- a/test/conftest.py
+++ b/test/conftest.py
@@ -983,7 +983,7 @@ def setup_postgres():
 
     with engine.connect() as connection:
         try:
-            connection.execute(text("DROP SCHEMA public CASCADE"))
+            connection.execute(text("DROP SCHEMA IF EXISTS public CASCADE"))
         except Exception as e:
             logging.error(e)
         connection.execute(text("CREATE SCHEMA public;"))
diff --git a/test/document_stores/test_document_store.py b/test/document_stores/test_document_store.py
--- a/test/document_stores/test_document_store.py
+++ b/test/document_stores/test_document_store.py
@@ -489,7 +489,7 @@ def test_write_document_meta(document_store: BaseDocumentStore):
 
 
 @pytest.mark.parametrize("document_store", ["sql"], indirect=True)
-def test_write_document_sql_invalid_meta(document_store: BaseDocumentStore):
+def test_sql_write_document_invalid_meta(document_store: BaseDocumentStore):
     documents = [
         {
             "content": "dict_with_invalid_meta",
@@ -512,6 +512,23 @@ def test_write_document_sql_invalid_meta(document_store: BaseDocumentStore):
     assert document_store.get_document_by_id("2").meta == {"name": "filename2", "valid_meta_field": "test2"}
 
 
+@pytest.mark.parametrize("document_store", ["sql"], indirect=True)
+def test_sql_write_different_documents_same_vector_id(document_store: BaseDocumentStore):
+    doc1 = {"content": "content 1", "name": "doc1", "id": "1", "vector_id": "vector_id"}
+    doc2 = {"content": "content 2", "name": "doc2", "id": "2", "vector_id": "vector_id"}
+
+    document_store.write_documents([doc1], index="index1")
+    documents_in_index1 = document_store.get_all_documents(index="index1")
+    assert len(documents_in_index1) == 1
+    document_store.write_documents([doc2], index="index2")
+    documents_in_index2 = document_store.get_all_documents(index="index2")
+    assert len(documents_in_index2) == 1
+
+    document_store.write_documents([doc1], index="index3")
+    with pytest.raises(Exception, match=r"(?i)unique"):
+        document_store.write_documents([doc2], index="index3")
+
+
 def test_write_document_index(document_store: BaseDocumentStore):
     document_store.delete_index("haystack_test_one")
     document_store.delete_index("haystack_test_two")
diff --git a/test/document_stores/test_faiss.py b/test/document_stores/test_faiss.py
--- a/test/document_stores/test_faiss.py
+++ b/test/document_stores/test_faiss.py
@@ -196,6 +196,39 @@ def test_faiss_write_docs(document_store, index_buffer_size, batch_size):
         assert np.allclose(original_doc["embedding"] / np.linalg.norm(original_doc["embedding"]), stored_emb, rtol=0.01)
 
 
+@pytest.mark.parametrize("document_store", ["faiss"], indirect=True)
+def test_faiss_write_docs_different_indexes(document_store):
+    document_store.write_documents(DOCUMENTS, index="index1")
+    document_store.write_documents(DOCUMENTS, index="index2")
+
+    docs_from_index1 = document_store.get_all_documents(index="index1", return_embedding=False)
+    assert len(docs_from_index1) == len(DOCUMENTS)
+    assert {int(doc.meta["vector_id"]) for doc in docs_from_index1} == set(range(0, 6))
+
+    docs_from_index2 = document_store.get_all_documents(index="index2", return_embedding=False)
+    assert len(docs_from_index2) == len(DOCUMENTS)
+    assert {int(doc.meta["vector_id"]) for doc in docs_from_index2} == set(range(0, 6))
+
+
+@pytest.mark.parametrize("document_store", ["faiss"], indirect=True)
+def test_faiss_update_docs_different_indexes(document_store):
+    retriever = MockDenseRetriever(document_store=document_store)
+
+    document_store.write_documents(DOCUMENTS, index="index1")
+    document_store.write_documents(DOCUMENTS, index="index2")
+
+    document_store.update_embeddings(retriever=retriever, update_existing_embeddings=True, index="index1")
+    document_store.update_embeddings(retriever=retriever, update_existing_embeddings=True, index="index2")
+
+    docs_from_index1 = document_store.get_all_documents(index="index1", return_embedding=False)
+    assert len(docs_from_index1) == len(DOCUMENTS)
+    assert {int(doc.meta["vector_id"]) for doc in docs_from_index1} == set(range(0, 6))
+
+    docs_from_index2 = document_store.get_all_documents(index="index2", return_embedding=False)
+    assert len(docs_from_index2) == len(DOCUMENTS)
+    assert {int(doc.meta["vector_id"]) for doc in docs_from_index2} == set(range(0, 6))
+
+
 @pytest.mark.skipif(sys.platform in ["win32", "cygwin"], reason="Test with tmp_path not working on windows runner")
 @pytest.mark.parametrize("index_factory", ["Flat", "HNSW", "IVF1,Flat"])
 def test_faiss_retrieving(index_factory, tmp_path):