Repository: deepset-ai/haystack

Issue:
LLM-based evaluators shouldn't return `NaN`
**Is your feature request related to a problem? Please describe.**
- `{'statements': [], 'statement_scores': [], 'score': nan}`
- The `score` should be a number instead.
 
**Possible solution**

- check for empty lists of extracted statements in [ContextRelevance](https://github.com/deepset-ai/haystack/blob/94467149c1d2ac63cc48d4b47b8529804c3f9edd/haystack/components/evaluators/context_relevance.py#L135) and [Faithfulness](https://github.com/deepset-ai/haystack/blob/94467149c1d2ac63cc48d4b47b8529804c3f9edd/haystack/components/evaluators/faithfulness.py#L153), which causes `NaN` and return `0` instead. 


Discussion:
coveralls: ## Pull Request Test Coverage Report for [Build 9061648326](https://coveralls.io/builds/67450962)


### Details

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage increased (+**0.003%**) to **90.417%**

---



|  Totals | [![Coverage Status](https://coveralls.io/builds/67450962/badge)](https://coveralls.io/builds/67450962) |
| :-- | --: |
| Change from base [Build 9030307905](https://coveralls.io/builds/67417579): |  0.003% |
| Covered Lines: | 6529 |
| Relevant Lines: | 7221 |

---
##### 💛  - [Coveralls](https://coveralls.io)

davidsbatista: you are right, I forgot that one - the error only occur to me on the `FaithfullnessEvaluator`, but I will add code + tests for the `ContextRelevanceEvaluator`

Patch:
diff --git a/docs/pydoc/config/converters_api.yml b/docs/pydoc/config/converters_api.yml
index f5c7d7c4..3ac4beb6 100644
--- a/docs/pydoc/config/converters_api.yml
+++ b/docs/pydoc/config/converters_api.yml
@@ -6,6 +6,7 @@ loaders:
         "azure",
         "html",
         "markdown",
+        "pdfminer",
         "pypdf",
         "tika",
         "txt",
diff --git a/haystack/components/evaluators/context_relevance.py b/haystack/components/evaluators/context_relevance.py
index a5e33707..9988bdeb 100644
--- a/haystack/components/evaluators/context_relevance.py
+++ b/haystack/components/evaluators/context_relevance.py
@@ -136,7 +136,10 @@ class ContextRelevanceEvaluator(LLMEvaluator):
 
         # calculate average statement relevance score per query
         for res in result["results"]:
-            res["score"] = np_mean(res["statement_scores"])
+            if not res["statements"]:
+                res["score"] = 0
+            else:
+                res["score"] = np_mean(res["statement_scores"])
 
         # calculate average context relevance score over all queries
         result["score"] = np_mean([res["score"] for res in result["results"]])
diff --git a/haystack/components/evaluators/faithfulness.py b/haystack/components/evaluators/faithfulness.py
index bf9487b2..2bcbb9b0 100644
--- a/haystack/components/evaluators/faithfulness.py
+++ b/haystack/components/evaluators/faithfulness.py
@@ -154,7 +154,10 @@ class FaithfulnessEvaluator(LLMEvaluator):
 
         # calculate average statement faithfulness score per query
         for res in result["results"]:
-            res["score"] = np_mean(res["statement_scores"])
+            if not res["statements"]:
+                res["score"] = 0
+            else:
+                res["score"] = np_mean(res["statement_scores"])
 
         # calculate average answer faithfulness score over all queries
         result["score"] = np_mean([res["score"] for res in result["results"]])
diff --git a/releasenotes/notes/avoid-LLM-based-evaluators-returning-NaN-579bc4593febb691.yaml b/releasenotes/notes/avoid-LLM-based-evaluators-returning-NaN-579bc4593febb691.yaml
new file mode 100644
index 00000000..5245f352
--- /dev/null
+++ b/releasenotes/notes/avoid-LLM-based-evaluators-returning-NaN-579bc4593febb691.yaml
@@ -0,0 +1,4 @@
+---
+fixes:
+  - |
+    `FaithfullnessEvaluator` and `ContextRelevanceEvaluator` now return `0` instead of `NaN` when applied to an empty context or empty statements.


Test patch:
diff --git a/test/components/evaluators/test_context_relevance_evaluator.py b/test/components/evaluators/test_context_relevance_evaluator.py
index 464a7255..ecbc215d 100644
--- a/test/components/evaluators/test_context_relevance_evaluator.py
+++ b/test/components/evaluators/test_context_relevance_evaluator.py
@@ -121,6 +121,38 @@ class TestContextRelevanceEvaluator:
             "score": 0.75,
         }
 
+    def test_run_no_statements_extracted(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = ContextRelevanceEvaluator()
+
+        def generator_run(self, *args, **kwargs):
+            if "Football" in kwargs["prompt"]:
+                return {"replies": ['{"statements": ["a", "b"], "statement_scores": [1, 0]}']}
+            else:
+                return {"replies": ['{"statements": [], "statement_scores": []}']}
+
+        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+
+        questions = ["Which is the most popular global sport?", "Who created the Python language?"]
+        contexts = [
+            [
+                "The popularity of sports can be measured in various ways, including TV viewership, social media "
+                "presence, number of participants, and economic impact. Football is undoubtedly the world's most "
+                "popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and "
+                "Messi, drawing a followership of more than 4 billion people."
+            ],
+            [],
+        ]
+        results = component.run(questions=questions, contexts=contexts)
+        assert results == {
+            "individual_scores": [0.5, 0],
+            "results": [
+                {"score": 0.5, "statement_scores": [1, 0], "statements": ["a", "b"]},
+                {"score": 0, "statement_scores": [], "statements": []},
+            ],
+            "score": 0.25,
+        }
+
     def test_run_missing_parameters(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = ContextRelevanceEvaluator()
diff --git a/test/components/evaluators/test_faithfulness_evaluator.py b/test/components/evaluators/test_faithfulness_evaluator.py
index 6160c122..e493b709 100644
--- a/test/components/evaluators/test_faithfulness_evaluator.py
+++ b/test/components/evaluators/test_faithfulness_evaluator.py
@@ -149,6 +149,42 @@ class TestFaithfulnessEvaluator:
             "score": 0.75,
         }
 
+    def test_run_no_statements_extracted(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = FaithfulnessEvaluator()
+
+        def generator_run(self, *args, **kwargs):
+            if "Football" in kwargs["prompt"]:
+                return {"replies": ['{"statements": ["a", "b"], "statement_scores": [1, 0]}']}
+            else:
+                return {"replies": ['{"statements": [], "statement_scores": []}']}
+
+        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+
+        questions = ["Which is the most popular global sport?", "Who created the Python language?"]
+        contexts = [
+            [
+                "The popularity of sports can be measured in various ways, including TV viewership, social media "
+                "presence, number of participants, and economic impact. Football is undoubtedly the world's most "
+                "popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and "
+                "Messi, drawing a followership of more than 4 billion people."
+            ],
+            [],
+        ]
+        predicted_answers = [
+            "Football is the most popular sport with around 4 billion followers worldwide.",
+            "I don't know.",
+        ]
+        results = component.run(questions=questions, contexts=contexts, predicted_answers=predicted_answers)
+        assert results == {
+            "individual_scores": [0.5, 0],
+            "results": [
+                {"score": 0.5, "statement_scores": [1, 0], "statements": ["a", "b"]},
+                {"score": 0, "statement_scores": [], "statements": []},
+            ],
+            "score": 0.25,
+        }
+
     def test_run_missing_parameters(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = FaithfulnessEvaluator()