Repository: deepset-ai/haystack
Pull request: bug: fix MRR and MAP calculations
### Related Issues

- fixes #7758

### Proposed Changes:

Fixed errors of MRREvaluator:
* Corrected the logic for calculation of MRR 
* Added a flag to break the loop when condition is fulfilled
* Optimized the loops

Fixed errors of MAPEvaluator:
* Corrected the logic for calculation of MAP
* Optimized the loops

Fixed errors in pytest file of `test_document_map.py`:
* Corrected values for `test_run_with_complex_data()`

### How did you test it?

Ran unit tests for both evaluators

### Notes for the reviewer

Verify the correctness of the calculated scores for an example.

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- I documented my code
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
MAP and MRR wrong for multiple gold documents
**Describe the bug**
Both the MAP and the MRR show wrong values. It seems we calculate the score for single gold documents and then override it after each instead of calculating it for the whole batch of gold documents. 

**Expected behavior**
correct values

**To Reproduce**
```
retrieved_docs = ["one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten"]
gold_docs = ["one", "two", "three","four","seven"]

from haystack.components.evaluators import DocumentMAPEvaluator, DocumentMRREvaluator
from haystack import Document

mapevaluator = DocumentMAPEvaluator()
mrrevaluator = DocumentMRREvaluator()
mapresult = mapevaluator.run(
    ground_truth_documents=[[Document(content=content) for content in gold_docs]],
    retrieved_documents=[[Document(content=content) for content in retrieved_docs]])

mrrresult = mrrevaluator.run(
    ground_truth_documents=[[Document(content=content) for content in gold_docs]],
    retrieved_documents=[[Document(content=content) for content in retrieved_docs]])

print(mapresult["individual_scores"])
print(mrrresult["individual_scores"])
print(mapresult["score"])
print(mrrresult["score"])
```

Hints from issue discussion:
Hi @ju-gu I'm trying to understand the example above. You are providing 10 "retrievals" but have only 5 gold truth docs? In other words, you have:

| Actual retrieved | True doc |
| --------------- | --------------- | 
|  one |  one  |
|  two |  two  |
|  three |  three  |
|  four |  four  |
|  five |  seven  |
|  six |  ?  |
|  seven |  ?  |
|  eight |  ?  |
|  nine |  ?  |
|  ten |  ?  |

I don't think this is a typical use case, are you sure this is what you're trying to do?

Or are you trying to do this instead:

| Actual retrieved | True doc |
| --------------- | --------------- | 
|  ["one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten"] |  ["one", "two", "three","four","seven"]  |
I am doing the latter. The two lists retrieved docs and true docs become a list of lists in these lines: 

ground_truth_documents=[[Document(content=content) for content in gold_docs]]
retrieved_documents=[[Document(content=content) for content in retrieved_docs]])

So it is just one set of retrieved docs and one set of gold documents

Commit message: removed unnecessary flag

Patch to review:
diff --git a/haystack/components/evaluators/document_map.py b/haystack/components/evaluators/document_map.py
index f18a0e675b..df3a6a7776 100644
--- a/haystack/components/evaluators/document_map.py
+++ b/haystack/components/evaluators/document_map.py
@@ -69,24 +69,26 @@ def run(
 
         for ground_truth, retrieved in zip(ground_truth_documents, retrieved_documents):
             score = 0.0
+            average_precision = 0.0
+            relevant_documents = 0
+            ground_truth_content = []
+
             for ground_document in ground_truth:
                 if ground_document.content is None:
                     continue
+                ground_truth_content.append(ground_document.content)
 
-                average_precision = 0.0
-                relevant_documents = 0
-
-                for rank, retrieved_document in enumerate(retrieved):
-                    if retrieved_document.content is None:
-                        continue
+            for rank, retrieved_document in enumerate(retrieved):
+                if retrieved_document.content is None:
+                    continue
 
-                    if ground_document.content in retrieved_document.content:
-                        relevant_documents += 1
-                        average_precision += relevant_documents / (rank + 1)
-                if relevant_documents > 0:
-                    score = average_precision / relevant_documents
+                if retrieved_document.content in ground_truth_content:
+                    relevant_documents += 1
+                    average_precision += relevant_documents / (rank + 1)
+            if relevant_documents > 0:
+                score = average_precision / relevant_documents
             individual_scores.append(score)
 
-        score = sum(individual_scores) / len(retrieved_documents)
+        score = sum(individual_scores) / len(ground_truth_documents)
 
         return {"score": score, "individual_scores": individual_scores}
diff --git a/haystack/components/evaluators/document_mrr.py b/haystack/components/evaluators/document_mrr.py
index 64a98cc177..66cff12417 100644
--- a/haystack/components/evaluators/document_mrr.py
+++ b/haystack/components/evaluators/document_mrr.py
@@ -67,17 +67,19 @@ def run(
 
         for ground_truth, retrieved in zip(ground_truth_documents, retrieved_documents):
             score = 0.0
+            ground_truth_content = []
+
             for ground_document in ground_truth:
                 if ground_document.content is None:
                     continue
+                ground_truth_content.append(ground_document.content)
 
-                for rank, retrieved_document in enumerate(retrieved):
-                    if retrieved_document.content is None:
-                        continue
-
-                    if ground_document.content in retrieved_document.content:
-                        score = 1 / (rank + 1)
-                        break
+            for rank, retrieved_document in enumerate(retrieved):
+                if retrieved_document.content is None:
+                    continue
+                if retrieved_document.content in ground_truth_content:
+                    score = 1 / (rank + 1)
+                    break
             individual_scores.append(score)
 
         score = sum(individual_scores) / len(retrieved_documents)
diff --git a/releasenotes/notes/fix-issue-7758-d35b687ca226a707.yaml b/releasenotes/notes/fix-issue-7758-d35b687ca226a707.yaml
new file mode 100644
index 0000000000..b1accdd2ac
--- /dev/null
+++ b/releasenotes/notes/fix-issue-7758-d35b687ca226a707.yaml
@@ -0,0 +1,4 @@
+---
+fixes:
+  - |
+    Fixed the calculation for MRR and MAP scores.
diff --git a/test/components/evaluators/test_document_map.py b/test/components/evaluators/test_document_map.py
index 9c481e3d6c..7c7b26c089 100644
--- a/test/components/evaluators/test_document_map.py
+++ b/test/components/evaluators/test_document_map.py
@@ -62,7 +62,17 @@ def test_run_with_complex_data():
             ],
         ],
     )
-    assert result == {"individual_scores": [1.0, 0.8333333333333333, 1.0, 0.5, 0.0, 1.0], "score": 0.7222222222222222}
+    assert result == {
+        "individual_scores": [
+            1.0,
+            pytest.approx(0.8333333333333333),
+            1.0,
+            pytest.approx(0.5833333333333333),
+            0.0,
+            pytest.approx(0.8055555555555555),
+        ],
+        "score": pytest.approx(0.7037037037037037),
+    }
 
 
 def test_run_with_different_lengths():


Code review comments:
diff_hunk: @@ -67,17 +67,19 @@ def run(
 
         for ground_truth, retrieved in zip(ground_truth_documents, retrieved_documents):
             score = 0.0
+            ground_truth_content = []
+
             for ground_document in ground_truth:
                 if ground_document.content is None:
                     continue
+                ground_truth_content.append(ground_document.content)
path: haystack/components/evaluators/document_mrr.py
text: @user1:
Same as above, this can be a list comprehension

diff_hunk: @@ -69,24 +69,26 @@ def run(
 
         for ground_truth, retrieved in zip(ground_truth_documents, retrieved_documents):
             score = 0.0
+            average_precision = 0.0
+            relevant_documents = 0
+            ground_truth_content = []
+
             for ground_document in ground_truth:
                 if ground_document.content is None:
                     continue
+                ground_truth_content.append(ground_document.content)
path: haystack/components/evaluators/document_map.py
text: @user1:
```suggestion
            ground_truth_content = [doc.content for doc in ground_truth if doc.content is not None]
```