Repository: deepset-ai/haystack

Issue:
`ContextRelevanceEvaluator`: statements extraction can be made more robust
Slightly different inputs produce very different outputs in the `ContextRelevanceEvaluator`.

**Example in the docs**:
```python
from haystack.components.evaluators import ContextRelevanceEvaluator

questions = ["Who created the Python language?"]
contexts = [
    [
        "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects."
    ],
]

evaluator = ContextRelevanceEvaluator()
result = evaluator.run(questions=questions, contexts=contexts)
print(result["score"])
# 1.0
print(result["individual_scores"])
# [1.0]
print(result["results"])
# [{'statements': ['Python, created by Guido van Rossum in the late 1980s.'], 'statement_scores': [1], 'score': 1.0}]
```

**A different example**:
```python
from haystack.components.evaluators import ContextRelevanceEvaluator
questions = ["Who created the Python language?"]
contexts = [
    [
        "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.",
        "Java is a high-level, class-based, object-oriented programming language. It allows you to write once, run anywhere, meaning that compiled Java code can run on all platforms that support Java without the need for recompilation.",
        "Scala is a high-level, statically typed programming language.",
    ],
]
evaluator = ContextRelevanceEvaluator()
result = evaluator.run(questions=questions, contexts=contexts)

print(result)
# {'results': [{'statements': ['Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language.', 'Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.'], 'statement_scores': [1, 0], 'score': 0.5}], 'score': 0.5, 'individual_scores': [0.5]}
```

In this case, the first context is broken into 2 statements, while the 2nd and 3rd statements are not considered.

**Expected behavior**
The extraction of statements should be more robust/consistent.

**Additional context**
From @julian-risch:
> I checked that the prompt sent to the LLM correctly contains all three contexts. The problem is that the LLM extract statements only for the first context at least in this example. It seems that the prompt instructions aren't good enough. You could try extending the _DEFAULT_EXAMPLES with an example that has multiple contexts for one question.


**System:**
 - Haystack version (commit or version number): 2.2.0


Discussion:
just to confirm, the expected behaviour here would be that every sentence/statement would have a score and not only the related ones?

Resolving patch:
diff --git a/haystack/components/evaluators/context_relevance.py b/haystack/components/evaluators/context_relevance.py
--- a/haystack/components/evaluators/context_relevance.py
+++ b/haystack/components/evaluators/context_relevance.py
@@ -23,8 +23,23 @@
         },
     },
     {
-        "inputs": {"questions": "What is the capital of France?", "contexts": ["Berlin is the capital of Germany."]},
-        "outputs": {"statements": ["Berlin is the capital of Germany."], "statement_scores": [0]},
+        "inputs": {
+            "questions": "What is the capital of France?",
+            "contexts": [
+                "Berlin is the capital of Germany and was founded in 1244.",
+                "Europe is a continent with 44 countries.",
+                "Madrid is the capital of Spain.",
+            ],
+        },
+        "outputs": {
+            "statements": [
+                "Berlin is the capital of Germany.",
+                "Berlin was founded in 1244.",
+                "Europe is a continent with 44 countries.",
+                "Madrid is the capital of Spain.",
+            ],
+            "statement_scores": [0, 0, 0, 0],
+        },
     },
     {
         "inputs": {"questions": "What is the capital of Italy?", "contexts": ["Rome is the capital of Italy."]},
@@ -113,7 +128,8 @@ def __init__(
             "Your task is to judge how relevant the provided context is for answering a question. "
             "First, please extract statements from the provided context. "
             "Second, calculate a relevance score for each statement in the context. "
-            "The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant."
+            "The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant. "
+            "Each statement should be scored individually."
         )
         self.inputs = [("questions", List[str]), ("contexts", List[List[str]])]
         self.outputs = ["statements", "statement_scores"]


Test patch:
diff --git a/test/components/evaluators/test_context_relevance_evaluator.py b/test/components/evaluators/test_context_relevance_evaluator.py
--- a/test/components/evaluators/test_context_relevance_evaluator.py
+++ b/test/components/evaluators/test_context_relevance_evaluator.py
@@ -23,7 +23,8 @@ def test_init_default(self, monkeypatch):
             "Your task is to judge how relevant the provided context is for answering a question. "
             "First, please extract statements from the provided context. "
             "Second, calculate a relevance score for each statement in the context. "
-            "The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant."
+            "The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant. "
+            "Each statement should be scored individually."
         )
         assert component.inputs == [("questions", List[str]), ("contexts", List[List[str]])]
         assert component.outputs == ["statements", "statement_scores"]
@@ -41,9 +42,21 @@ def test_init_default(self, monkeypatch):
             {
                 "inputs": {
                     "questions": "What is the capital of France?",
-                    "contexts": ["Berlin is the capital of Germany."],
+                    "contexts": [
+                        "Berlin is the capital of Germany and was founded in 1244.",
+                        "Europe is a continent with 44 countries.",
+                        "Madrid is the capital of Spain.",
+                    ],
+                },
+                "outputs": {
+                    "statements": [
+                        "Berlin is the capital of Germany.",
+                        "Berlin was founded in 1244.",
+                        "Europe is a continent with 44 countries.",
+                        "Madrid is the capital of Spain.",
+                    ],
+                    "statement_scores": [0, 0, 0, 0],
                 },
-                "outputs": {"statements": ["Berlin is the capital of Germany."], "statement_scores": [0]},
             },
             {
                 "inputs": {"questions": "What is the capital of Italy?", "contexts": ["Rome is the capital of Italy."]},
@@ -242,3 +255,30 @@ def test_live_run(self):
         assert all(field in result for field in required_fields)
         nested_required_fields = {"score", "statement_scores", "statements"}
         assert all(field in result["results"][0] for field in nested_required_fields)
+
+    @pytest.mark.skipif(
+        not os.environ.get("OPENAI_API_KEY", None),
+        reason="Export an env var called OPENAI_API_KEY containing the OpenAI API key to run this test.",
+    )
+    @pytest.mark.integration
+    def test_all_statements_are_scored(self):
+        from haystack.components.evaluators import ContextRelevanceEvaluator
+
+        questions = ["Who created the Python language?"]
+        contexts = [
+            [
+                "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming "
+                "language. Its design philosophy emphasizes code readability, and its language constructs aim to help "
+                "programmers write clear, logical code for both small and large-scale software projects.",
+                "Java is a high-level, class-based, object-oriented programming language which allows you to write once, "
+                "run anywhere, meaning that compiled Java code can run on all platforms that support Java without the "
+                "need for recompilation.",
+                "Scala is a high-level, statically typed programming language.",
+            ]
+        ]
+
+        evaluator = ContextRelevanceEvaluator()
+        result = evaluator.run(questions=questions, contexts=contexts)
+
+        assert len(result["results"][0]["statements"]) == 4
+        assert len(result["results"][0]["statement_scores"]) == 4