Repository: deepset-ai/haystack

Issue:
Semantic search is not working for MongoDBAtlasDocumentStore
**Describe the bug**
EmbeddingRetriever doesn't return any results when used with MongoDBAtlasDocumentStore. The reason seems like MongoDBAtlasDocumentStore

**Error message**
No error

**Expected behavior**
Semantic search should be possible with EmbeddingRetriever 

**Additional context**
There's no problem in `write_documents`, `update_embeddings` and `get_document_count()` methods of `MongoDBAtlasDocumentStore`. The code below works for InMemoryDocumentStore properly. 

Issue #6632 can be related to this issue.

**To Reproduce**
```python
from haystack.document_stores.mongodb_atlas import MongoDBAtlasDocumentStore
from haystack.nodes import EmbeddingRetriever
from haystack import Document, Pipeline
import os

ds=MongoDBAtlasDocumentStore(
    mongo_connection_string="mongodb+srv://{mongo_atlas_username}:{mongo_atlas_password}@{mongo_atlas_host}/?{mongo_atlas_params_string}",
    database_name="database_name",
    collection_name="collection_name",
    embedding_dim=1536
)

## Write Documents into MongoDBAtlasDocumentStore
ds.write_documents([Document("Berlin is an amazing city."), Document("Berlin is the capital of Germany"), Document("I love Berlin.")])
retriever = EmbeddingRetriever(document_store=ds, embedding_model="text-embedding-ada-002", api_key=os.getenv("OPENAI_API_KEY"))
ds.update_embeddings(retriever)
print(ds.get_document_count()) ## returns 3

## Document search pipeline
p = Pipeline()
p.add_node(component=retriever, name="EmbeddingRetriever", inputs=["Query"])
pipe_result = p.run(query="Berlin")
print(pipe_result) ## returns {'documents': [], 'root_node': 'Query', 'params': {}, 'query': 'Berlin', 'node_id': 'EmbeddingRetriever'}

## Use Retriever standalone
result = retriever.retrieve(query="Berlin", top_k=2)
print(result) ## returns []
```

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: MacOS 
 - GPU/CPU: CPU
 - Haystack version (commit or version number): 1.23
 - DocumentStore: MongoDBAtlasDocumentStore
 - Reader: - 
 - Retriever: EmbeddingRetriever



Discussion:
Hi @bilgeyucel,

My issue was indeed the same, it looks like Haystack uses the old way to query Mongo with embeddings. With the latest [MongoDB Atlas update](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/) there are two ways of searching with mongo:

- Sparse BM Search with the $search aggregation
- Dense search with $vectorSearch aggregation

I would love to see these two functions integrated, so the Sparse search for `MongoDBAtlasDocumentStore(bm25=True)`
and vectorsearch with an `EmbeddingRetriever` 

I fixed your issue with semantic search by changing the pipeline to the following, but this only works with the new VectorSearch index, this index is `vector_index`:

```
       pipeline = [
            {
                "$vectorSearch": {
                    "index": "vector_index",
                    "queryVector": query_emb.tolist(),
                    "path": "embedding", 
                    "numCandidates": 100,
                    "limit": top_k,
                }

            }
        ]
```


Just checking if this issue is going to be updated for 1.x?  It appears to simply be the pipeline jvollmuller-risk mentioned.  I get 0 results with retriever from MongoDB Atlas currently.  If I change the pipeline to the new search syntax and specify the search index (the index field, not the collection) it works, but in my environment I would be required to change this each time the instance spins up.
Hi @scottroot, we're working on the issue right now. If you'd like to help out, feel free to submit a PR to fix it. Your contribution would speed up the resolution process! :)

Resolving patch:
diff --git a/haystack/document_stores/mongodb_atlas.py b/haystack/document_stores/mongodb_atlas.py
--- a/haystack/document_stores/mongodb_atlas.py
+++ b/haystack/document_stores/mongodb_atlas.py
@@ -26,6 +26,7 @@ def __init__(
         mongo_connection_string: Optional[str] = None,
         database_name: Optional[str] = None,
         collection_name: Optional[str] = None,
+        vector_search_index: Optional[str] = None,
         embedding_dim: int = 768,
         return_embedding: bool = False,
         similarity: str = "cosine",
@@ -41,6 +42,7 @@ def __init__(
         :param mongo_connection_string: MongoDB Atlas connection string in the format: "mongodb+srv://{mongo_atlas_username}:{mongo_atlas_password}@{mongo_atlas_host}/?{mongo_atlas_params_string}".
         :param database_name: Name of the database to use.
         :param collection_name: Name of the collection to use.
+        :param vector_search_index: The name of the index to use for vector search. To use the search index it must have been created in the Atlas web UI before. None by default.
         :param embedding_dim: Dimensionality of embeddings, 768 by default.
         :param return_embedding: Whether to return document embeddings when returning documents.
         :param similarity: The similarity function to use for the embeddings. One of "euclidean", "cosine" or "dotProduct". "cosine" is the default.
@@ -67,6 +69,7 @@ def __init__(
         self.index = collection_name
         self.return_embedding = return_embedding
         self.recreate_index = recreate_index
+        self.vector_search_index = vector_search_index
 
         if self.recreate_index:
             self.delete_index()
@@ -127,7 +130,7 @@ def delete_documents(
 
     def delete_index(self, index=None):
         """
-        Deletes the collection named by index or the collection speicifed when the
+        Deletes the collection named by index or the collection specified when the
         driver was initialized.
         """
         self._get_collection(index).drop()
@@ -340,6 +343,10 @@ def query_by_embedding(
                             If true (default) similarity scores (e.g. cosine or dot_product) which naturally have a different value range will be scaled to a range of [0,1], where 1 means extremely relevant.
                             Otherwise raw similarity scores (e.g. cosine or dot_product) will be used.
         """
+        if not self.vector_search_index:
+            raise ValueError(
+                "No vector_search_index is set for MongoDBAtlasDocumentStore. Create a vector_search_index in the Atlas web UI and specify it in the init parameters of MongoDBAtlasDocumentStore. https://www.mongodb.com/docs/atlas/atlas-vector-search/create-index/#std-label-avs-create-index"
+            )
         if headers:
             raise NotImplementedError("MongoDBAtlasDocumentStore does not support headers.")
 
@@ -357,9 +364,12 @@ def query_by_embedding(
 
         pipeline = [
             {
-                "$search": {
-                    "index": self.collection_name,
-                    "knnBeta": {"vector": query_emb.tolist(), "path": "embedding", "k": top_k},
+                "$vectorSearch": {
+                    "index": self.vector_search_index,
+                    "queryVector": query_emb.tolist(),
+                    "path": "embedding",
+                    "numCandidates": 100,
+                    "limit": top_k,
                 }
             }
         ]
@@ -367,7 +377,7 @@ def query_by_embedding(
             pipeline.append({"$match": mongo_filter_converter(filters)})
         if not return_embedding:
             pipeline.append({"$project": {"embedding": False}})
-        pipeline.append({"$set": {"score": {"$meta": "searchScore"}}})
+        pipeline.append({"$set": {"score": {"$meta": "vectorSearchScore"}}})
         documents = list(collection.aggregate(pipeline))
 
         if scale_score:


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -122,10 +122,10 @@ jobs:
         include:
           - topic: document_stores
             os: ubuntu-latest
-            dependencies: elasticsearch8,faiss,weaviate,pinecone,opensearch,inference,crawler,preprocessing,file-conversion,pdf,ocr,metrics,dev
+            dependencies: elasticsearch8,faiss,weaviate,pinecone,opensearch,mongodb,inference,crawler,preprocessing,file-conversion,pdf,ocr,metrics,dev
           - topic: document_stores
             os: windows-latest
-            dependencies: elasticsearch8,faiss,weaviate,pinecone,opensearch,inference,crawler,preprocessing,file-conversion,pdf,ocr,metrics,dev
+            dependencies: elasticsearch8,faiss,weaviate,pinecone,opensearch,mongodb,inference,crawler,preprocessing,file-conversion,pdf,ocr,metrics,dev
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
diff --git a/test/document_stores/test_mongodb_atlas.py b/test/document_stores/test_mongodb_atlas.py
new file mode 100644
--- /dev/null
+++ b/test/document_stores/test_mongodb_atlas.py
@@ -0,0 +1,64 @@
+from unittest.mock import MagicMock, patch
+
+import pymongo
+import pytest
+from numpy import float32, random
+
+from haystack.document_stores import mongodb_atlas
+from haystack.document_stores.mongodb_atlas import MongoDBAtlasDocumentStore
+
+
+class TestMongoDBDocumentStore:
+    @pytest.fixture
+    def mocked_ds(self):
+        class DSMock(MongoDBAtlasDocumentStore):
+            # We mock a subclass to avoid messing up the actual class object
+            pass
+
+        mongodb_atlas._validate_mongo_connection_string = MagicMock()
+        mongodb_atlas._validate_database_name = MagicMock()
+        mongodb_atlas._validate_collection_name = MagicMock()
+        mongodb_atlas._get_collection = MagicMock()
+        pymongo.MongoClient = MagicMock()
+
+        mocked_ds = DSMock(
+            mongo_connection_string="mongodb+srv://{mongo_atlas_username}:{mongo_atlas_password}@{mongo_atlas_host}/?{mongo_atlas_params_string}",
+            database_name="test_db",
+            collection_name="test_collection",
+            embedding_dim=1536,
+        )
+
+        return mocked_ds
+
+    @pytest.mark.unit
+    def test_error_is_raised_if_vector_index_name_is_not_set_for_vector_search(self, mocked_ds):
+        with pytest.raises(ValueError):
+            mocked_ds.vector_search_index = None
+            mocked_ds.query_by_embedding(query_emb=random.rand(768))
+
+    @pytest.mark.unit
+    def test_vector_index_name_is_set_for_vector_search(self, mocked_ds):
+        mocked_ds.vector_search_index = "vector_search_index"
+        with patch.object(mocked_ds, "_get_collection", return_value=MagicMock()) as mock_get_collection:
+            query_emb = random.rand(768)
+            mocked_ds.query_by_embedding(query_emb=query_emb)
+            expected_emb_in_call = query_emb.astype(float32)
+            mocked_ds.normalize_embedding(expected_emb_in_call)
+            # check that the correct arguments are passed to collection.aggregate()
+            mock_get_collection().aggregate.assert_called()
+            mock_get_collection().aggregate.assert_called_once_with(
+                [
+                    {
+                        "$vectorSearch": {
+                            "index": "vector_search_index",
+                            "queryVector": expected_emb_in_call.tolist(),
+                            "path": "embedding",
+                            "numCandidates": 100,
+                            "limit": 10,
+                        }
+                    },
+                    {"$match": {}},
+                    {"$project": {"embedding": False}},
+                    {"$set": {"score": {"$meta": "vectorSearchScore"}}},
+                ]
+            )