Repository: deepset-ai/haystack

Issue:
Document Recall Evaluator Multi Hit calculation returns confusing results because of non-uniqueness
**Describe the bug**
https://github.com/deepset-ai/haystack/blob/e3f9da13d0df40b5898e480c725ee329ca175858/haystack/components/evaluators/document_recall.py#L95-L100



**Expected behavior**
Should be: 
```python
    def _recall_multi_hit(self, ground_truth_documents: List[Document], retrieved_documents: List[Document]) -> float:
        unique_truths = {g.content for g in ground_truth_documents}
        unique_retrievals = {p.content for p in retrieved_documents}
        retrieved_ground_truths = unique_truths.intersection(unique_retrievals)

        if len(unique_truths) == 1 and not list(unique_truths)[0]:
            print(
                "Warning: No relevant documents found in ground truth or retrieved documents. Returning Recall score of 0.0.")
            return 0.0
        else:
            return len(retrieved_ground_truths) / len(unique_truths)
``` 

This returns the unique recall metric. 

If you expect the previous behavior, I suggest to add a DuplicateDocumentEvaluator which checks for duplicates.

Discussion:
@julian-risch: Hi @deep-rloebbert , thank you for opening this issue! I agree that we should divide by `len(unique_truths)` instead of `len(ground_truth_documents).` Could you please explain a bit more the idea behind `if len(unique_truths) == 1 and not list(unique_truths)[0]:`?

I understand that `if len(unique_truths) == 1` means that the content of all documents in `ground_truth_documents` is the same or there are no documents at all. With `not list(unique_truths)[0]` you would like to check whether that content is either `""` or `None`? Doesn't it make more sense to show a message then that none of the ground truth document has a content?

Side note: Instead of `print` we should use `logger.warning` when we address this issue.@deep-rloebbert: Hi @julian-risch, good question! This code was added by @ju-gu, I agree that we should check and log a warning for none or empty content, both in retrieved and ground truth. @davidsbatista: To help clarify and add more detail to the issue, do you have, by any chance, a simple use case with sample documents that illustrate the problem?

Solution patch:
diff --git a/haystack/components/evaluators/document_recall.py b/haystack/components/evaluators/document_recall.py
index 00c3c72e7c..6ab56f9171 100644
--- a/haystack/components/evaluators/document_recall.py
+++ b/haystack/components/evaluators/document_recall.py
@@ -5,9 +5,11 @@
 from enum import Enum
 from typing import Any, Dict, List, Union
 
-from haystack import component, default_to_dict
+from haystack import component, default_to_dict, logging
 from haystack.dataclasses import Document
 
+logger = logging.getLogger(__name__)
+
 
 class RecallMode(Enum):
     """
@@ -97,7 +99,21 @@ def _recall_multi_hit(ground_truth_documents: List[Document], retrieved_document
         unique_retrievals = {p.content for p in retrieved_documents}
         retrieved_ground_truths = unique_truths.intersection(unique_retrievals)
 
-        return len(retrieved_ground_truths) / len(ground_truth_documents)
+        if not unique_truths or unique_truths == {""}:
+            logger.warning(
+                "There are no ground truth documents or all of them have an empty string as content. "
+                "Score will be set to 0."
+            )
+            return 0.0
+
+        if not unique_retrievals or unique_retrievals == {""}:
+            logger.warning(
+                "There are no retrieved documents or all of them have an empty string as content. "
+                "Score will be set to 0."
+            )
+            return 0.0
+
+        return len(retrieved_ground_truths) / len(unique_truths)
 
     @component.output_types(score=float, individual_scores=List[float])
     def run(
diff --git a/releasenotes/notes/updated-doc-recall-eval-uniqueness-59b09082cf8e7593.yaml b/releasenotes/notes/updated-doc-recall-eval-uniqueness-59b09082cf8e7593.yaml
new file mode 100644
index 0000000000..132b857e60
--- /dev/null
+++ b/releasenotes/notes/updated-doc-recall-eval-uniqueness-59b09082cf8e7593.yaml
@@ -0,0 +1,5 @@
+---
+enhancements:
+  - |
+    The `DocumentRecallEvaluator` was updated. Now, when in `MULTI_HIT` mode, the division is over the unique ground truth documents instead of the total number of ground truth documents.
+    We also added checks for emptiness. If there are no retrieved documents or all of them have an empty string as content, we return 0.0 and log a warning. Likewise, if there are no ground truth documents or all of them have an empty string as content, we return 0.0 and log a warning.


Test patch:
diff --git a/test/components/evaluators/test_document_recall.py b/test/components/evaluators/test_document_recall.py
index 2c438441cc..b7010a793f 100644
--- a/test/components/evaluators/test_document_recall.py
+++ b/test/components/evaluators/test_document_recall.py
@@ -13,6 +13,14 @@ def test_init_with_unknown_mode_string():
         DocumentRecallEvaluator(mode="unknown_mode")
 
 
+def test_init_with_string_mode():
+    evaluator = DocumentRecallEvaluator(mode="single_hit")
+    assert evaluator.mode == RecallMode.SINGLE_HIT
+
+    evaluator = DocumentRecallEvaluator(mode="multi_hit")
+    assert evaluator.mode == RecallMode.MULTI_HIT
+
+
 class TestDocumentRecallEvaluatorSingleHit:
     @pytest.fixture
     def evaluator(self):
@@ -186,3 +194,27 @@ def test_from_dict(self):
         }
         new_evaluator = default_from_dict(DocumentRecallEvaluator, data)
         assert new_evaluator.mode == RecallMode.MULTI_HIT
+
+    def test_empty_ground_truth_documents(self, evaluator):
+        ground_truth_documents = [[]]
+        retrieved_documents = [[Document(content="test")]]
+        score = evaluator.run(ground_truth_documents, retrieved_documents)
+        assert score == {"individual_scores": [0.0], "score": 0.0}
+
+    def test_empty_retrieved_documents(self, evaluator):
+        ground_truth_documents = [[Document(content="test")]]
+        retrieved_documents = [[]]
+        score = evaluator.run(ground_truth_documents, retrieved_documents)
+        assert score == {"individual_scores": [0.0], "score": 0.0}
+
+    def test_empty_string_ground_truth_documents(self, evaluator):
+        ground_truth_documents = [[Document(content="")]]
+        retrieved_documents = [[Document(content="test")]]
+        score = evaluator.run(ground_truth_documents, retrieved_documents)
+        assert score == {"individual_scores": [0.0], "score": 0.0}
+
+    def test_empty_string_retrieved_documents(self, evaluator):
+        ground_truth_documents = [[Document(content="test")]]
+        retrieved_documents = [[Document(content="")]]
+        score = evaluator.run(ground_truth_documents, retrieved_documents)
+        assert score == {"individual_scores": [0.0], "score": 0.0}