Repository: deepset-ai/haystack

Issue:
SASEvaluator returns score even when `predicted_answers` contains `None`
In a RAG pipeline, there are cases where the prompt instructs the LLM to return a placeholder answer if there is missing information in the context, such as "No answer". See an [example](https://github.com/deepset-ai/haystack-evaluation/blob/cbfcde5afcaf7574aa82b1ad6b5ce307c8a7c97e/evaluations/architectures/basic_rag.py#L10) in our evaluation repository.

In that case, it makes sense to replace the predicted answer in the RAG pipeline with `None`, instead of computing any metric on the placeholder answer. However, the SASEvaluator returns a score even if `None` is passed.

**Example**:
```
from haystack.components.evaluators import SASEvaluator

sas = SASEvaluator(model="sentence-transformers/all-mpnet-base-v2")
sas.warm_up()
sas.run(predicted_answers=[None, "Whatever"], ground_truth_answers=["Whatever", "Another whatever"])
```

the output is:

```
{'score': 0.46286168694496155,
 'individual_scores': [0.26153379678726196, 0.6641895771026611]}
```

**Expected**

```
{'score': 0.6641895771026611,
 'individual_scores': [None, 0.6641895771026611]}
```


Discussion:
> In that case, it makes sense to replace the predicted answer in the RAG pipeline with None

This is something that I'm manually doing in my workflow, but I would like to see a component for. We have `DocumentCleaner`, so I think it would make sense to have an `AnswerCleaner` or a `GeneratedAnswerCleaner` which can take in a "stop word" parameter to scan the answer for a specific word/phrase.

The handling of 'None' might need to be applied to LLMEvaluator as well. I run a forked version and it accepts 'None' as a parameter and returns an answer. I will try running it on the main Haystack code to see what happens.
![image](https://github.com/deepset-ai/haystack/assets/30765968/7f59c0ad-8032-4fa6-9450-20197f7d2e03)

The same issue needs to be handled in LLMEvaluators.
`SASEvaluator` expects `List[str]` for the input parameter `predicted_answers`. We can extend the input parameter validation here https://github.com/deepset-ai/haystack/blob/4c59000c211901d0badaacc18de5b0cb337a88df/haystack/components/evaluators/sas_evaluator.py#L158 to catch cases where `None` is a value in the input list. In that case, we should raise a TypeError and let the user know that all values in the list are expected to be str.

Resolving patch:
diff --git a/haystack/components/evaluators/sas_evaluator.py b/haystack/components/evaluators/sas_evaluator.py
--- a/haystack/components/evaluators/sas_evaluator.py
+++ b/haystack/components/evaluators/sas_evaluator.py
@@ -157,6 +157,9 @@ def run(self, ground_truth_answers: List[str], predicted_answers: List[str]) ->
         if len(ground_truth_answers) != len(predicted_answers):
             raise ValueError("The number of predictions and labels must be the same.")
 
+        if any(answer is None for answer in predicted_answers):
+            raise ValueError("Predicted answers must not contain None values.")
+
         if len(predicted_answers) == 0:
             return {"score": 0.0, "individual_scores": [0.0]}
 


Test patch:
diff --git a/test/components/evaluators/test_sas_evaluator.py b/test/components/evaluators/test_sas_evaluator.py
--- a/test/components/evaluators/test_sas_evaluator.py
+++ b/test/components/evaluators/test_sas_evaluator.py
@@ -73,6 +73,21 @@ def test_run_with_different_lengths(self):
         with pytest.raises(ValueError):
             evaluator.run(ground_truth_answers=ground_truths, predicted_answers=predictions)
 
+    def test_run_with_none_in_predictions(self):
+        evaluator = SASEvaluator()
+        ground_truths = [
+            "A construction budget of US $2.3 billion",
+            "The Eiffel Tower, completed in 1889, symbolizes Paris's cultural magnificence.",
+            "The Meiji Restoration in 1868 transformed Japan into a modernized world power.",
+        ]
+        predictions = [
+            "A construction budget of US $2.3 billion",
+            None,
+            "The Meiji Restoration in 1868 transformed Japan into a modernized world power.",
+        ]
+        with pytest.raises(ValueError):
+            evaluator.run(ground_truth_answers=ground_truths, predicted_answers=predictions)
+
     def test_run_not_warmed_up(self):
         evaluator = SASEvaluator()
         ground_truths = [