Repository: deepset-ai/haystack

Issue:
Leaky scores in `BM25Retriever` from `InMemoryDocumentStore`
**Describe the bug**

When using the `BM25Retriever` and `InMemoryDocumentStore`, the scores produced in previous outputs are not fixed and change as more queries are computed, indicating scores are referenced somewhere in memory without being copied.

**Expected behavior**

The score(s) per unique query should remain the same regardless of order or number of queries fed in.

**To Reproduce**

```python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from copy import deepcopy

from haystack import Document
from haystack.document_stores import InMemoryDocumentStore
from haystack.nodes import BM25Retriever

# create document store for BM25
raw_docs = [
    "This is a really short document. It states that the weather is good.",
    "This is a document about test sets. A test set is a set for testing.",
]
docs = [Document(content=content) for content in raw_docs]
document_store = InMemoryDocumentStore(use_bm25=True)
document_store.write_documents(docs)

# create pipeline
retriever = BM25Retriever(
    document_store,
    scale_score=False,
)

# run pipeline on queries via direct and deep-copies
queries = [
    "Is the weather nice today?",
    "What is a test set?",
    "Do we have a short document?",
]
results_direct = [retriever.retrieve(query) for query in queries]
results_copied = [deepcopy(retriever.retrieve(query)) for query in queries]

# assert equivalence
assert results_direct == results_copied
```

An `AssertionError` will be produced. The differences lie in the scores. Everything else appears to be the same.

I tested the `EmbeddingRetriever` with `FAISSDocumentStore` and was not able to reproduce this error. I did not test other variants/combinations.

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: `Linux 6.1.20-1-lts x86_64`
 - GPU/CPU: `CPU`
 - Haystack version (commit or version number): `1.15.1`
 - DocumentStore: `InMemoryDocumentStore`
 - Retriever: `BM25Retriever`


Discussion:
@masci or @atreyasha Is there already some progress on this bug? Otherwise, I would like to take it on!
@benheckmann I was looking at some possible causes and believe it could be somewhere in these lines:

https://github.com/deepset-ai/haystack/blob/98947e4c3c394aa4a849863de71eaa5595c21d14/haystack/document_stores/memory.py#L989-L1001

Was thinking of debugging these lines, but I won't have time to do this anytime soon so help would be great!
@atreyasha Sorry for the delay. The issue is that `InMemoryDocumentStore.query` does two things: update the documents' scores with the latest query result and return a sorted sublist based on the scores. 

The new score overwrites the old one here:
```python
doc.score = docs_scores[i]
```

Opened a small PR to create a new document before returning the sorted list. Memory will not be an issue since the new documents will just share the content reference with the original ones.

Resolving patch:
diff --git a/haystack/document_stores/faiss.py b/haystack/document_stores/faiss.py
--- a/haystack/document_stores/faiss.py
+++ b/haystack/document_stores/faiss.py
@@ -1,3 +1,4 @@
+import copy
 from typing import Union, List, Optional, Dict, Generator
 
 import json
@@ -634,16 +635,18 @@ def query_by_embedding(
         scores_for_vector_ids: Dict[str, float] = {
             str(v_id): s for v_id, s in zip(vector_id_matrix[0], score_matrix[0])
         }
+        return_documents = []
         for doc in documents:
             score = scores_for_vector_ids[doc.meta["vector_id"]]
             if scale_score:
                 score = self.scale_to_unit_interval(score, self.similarity)
             doc.score = score
-
             if return_embedding is True:
                 doc.embedding = self.faiss_indexes[index].reconstruct(int(doc.meta["vector_id"]))
+            return_document = copy.copy(doc)
+            return_documents.append(return_document)
 
-        return documents
+        return return_documents
 
     def save(self, index_path: Union[str, Path], config_path: Optional[Union[str, Path]] = None):
         """
diff --git a/haystack/document_stores/memory.py b/haystack/document_stores/memory.py
--- a/haystack/document_stores/memory.py
+++ b/haystack/document_stores/memory.py
@@ -1,3 +1,4 @@
+import copy
 from typing import Any, Dict, List, Optional, Union, Generator, Literal
 
 import time
@@ -959,7 +960,7 @@ def query(
         scale_score: bool = True,
     ) -> List[Document]:
         """
-        Scan through documents in DocumentStore and return a small number documents
+        Scan through documents in DocumentStore and return a small number of documents
         that are most relevant to the query as defined by the BM25 algorithm.
         :param query: The query.
         :param top_k: How many documents to return per query.
@@ -995,13 +996,13 @@ def query(
         top_docs_positions = np.argsort(docs_scores)[::-1][:top_k]
 
         textual_docs_list = [doc for doc in self.indexes[index].values() if doc.content_type in ["text", "table"]]
-        top_docs = []
+        return_documents = []
         for i in top_docs_positions:
             doc = textual_docs_list[i]
             doc.score = docs_scores[i]
-            top_docs.append(doc)
-
-        return top_docs
+            return_document = copy.copy(doc)
+            return_documents.append(return_document)
+        return return_documents
 
     def query_batch(
         self,
@@ -1015,7 +1016,7 @@ def query_batch(
         scale_score: bool = True,
     ) -> List[List[Document]]:
         """
-        Scan through documents in DocumentStore and return a small number documents
+        Scan through documents in DocumentStore and return a small number of documents
         that are most relevant to the provided queries as defined by keyword matching algorithms like BM25.
         This method lets you find relevant documents for list of query strings (output: List of Lists of Documents).
         :param query: The query.
diff --git a/haystack/document_stores/pinecone.py b/haystack/document_stores/pinecone.py
--- a/haystack/document_stores/pinecone.py
+++ b/haystack/document_stores/pinecone.py
@@ -1,3 +1,4 @@
+import copy
 import json
 from typing import Set, Union, List, Optional, Dict, Generator, Any
 
@@ -1181,13 +1182,16 @@ def query_by_embedding(
 
         # assign query score to each document
         scores_for_vector_ids: Dict[str, float] = {str(v_id): s for v_id, s in zip(vector_id_matrix, score_matrix)}
+        return_documents = []
         for doc in documents:
             score = scores_for_vector_ids[doc.id]
             if scale_score:
                 score = self.scale_to_unit_interval(score, self.similarity)
             doc.score = score
+            return_document = copy.copy(doc)
+            return_documents.append(return_document)
 
-        return documents
+        return return_documents
 
     def _get_documents_by_meta(
         self,


Test patch:
diff --git a/test/document_stores/test_memory.py b/test/document_stores/test_memory.py
--- a/test/document_stores/test_memory.py
+++ b/test/document_stores/test_memory.py
@@ -1,4 +1,5 @@
 import logging
+from copy import deepcopy
 
 import pandas as pd
 import pytest
@@ -6,6 +7,7 @@
 import numpy as np
 
 from haystack.document_stores.memory import InMemoryDocumentStore
+from haystack.nodes import BM25Retriever
 from haystack.schema import Document
 from haystack.testing import DocumentStoreBaseTestAbstract
 
@@ -124,3 +126,17 @@ def test_memory_query_by_embedding_docs_wo_embeddings(self, ds, caplog):
             docs = ds.query_by_embedding(query_emb=query_embedding, top_k=1)
             assert "Skipping some of your documents that don't have embeddings" in caplog.text
         assert len(docs) == 0
+
+    @pytest.mark.integration
+    def test_bm25_scores_not_changing_across_queries(self, ds, documents):
+        """Test that computed scores which are returned to the user should not change when running multiple queries."""
+        ds.write_documents(documents)
+        retriever = BM25Retriever(ds, scale_score=False)
+        queries = ["What is a Foo Document?", "What is a Bar Document?", "Tell me about a document without embeddings"]
+        results_direct = []
+        results_direct = [retriever.retrieve(query) for query in queries]
+        results_copied = [deepcopy(retriever.retrieve(query)) for query in queries]
+        scores_direct = [rd.score for rds in results_direct for rd in rds]
+        scores_copied = [rc.score for rcs in results_copied for rc in rcs]
+
+        assert scores_direct == scores_copied