Repository: deepset-ai/haystack
Pull request #9116: refactor!: use Chat Generator in LLM evaluators
Author: login: anakin87
Labels: name: topic:tests
Created: 2025-03-26T10:02:28Z | Merged: 2025-03-26T14:38:57Z

### Related Issues

- part of #9062 (I decided to split the work to keep PRs small)

### Proposed Changes:
- refactor `LLMEvaluator` and child classes (`ContextRelevanceEvaluator` and `FaithfulnessEvaluator`) to use a Chat Generator instead of a Generator

- breaking: replace the public attribute `self.generator` with `self._chat_generator`
  This would not be strictly required but:
    - in a future PR we will allow users to pass a `chat_generator` instance and I think exposing a `generator` attribute would be confusing
    - I believe that the `generator`/`_chat_generator` attribute should not be part of the public API of this component

### How did you test it?
CI; adapted tests


### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:` and added `!` in case the PR includes breaking changes.
- I documented my code
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue



Diff:
diff --git a/haystack/components/evaluators/llm_evaluator.py b/haystack/components/evaluators/llm_evaluator.py
index ac1aab7c85..da6d089b33 100644
--- a/haystack/components/evaluators/llm_evaluator.py
+++ b/haystack/components/evaluators/llm_evaluator.py
@@ -9,7 +9,8 @@
 
 from haystack import component, default_from_dict, default_to_dict, logging
 from haystack.components.builders import PromptBuilder
-from haystack.components.generators import OpenAIGenerator
+from haystack.components.generators.chat.openai import OpenAIChatGenerator
+from haystack.dataclasses.chat_message import ChatMessage
 from haystack.utils import Secret, deserialize_secrets_inplace, deserialize_type, serialize_type
 
 logger = logging.getLogger(__name__)
@@ -110,7 +111,7 @@ def __init__(  # pylint: disable=too-many-positional-arguments
             generator_kwargs = {**self.api_params}
             if api_key:
                 generator_kwargs["api_key"] = api_key
-            self.generator = OpenAIGenerator(**generator_kwargs)
+            self._chat_generator = OpenAIChatGenerator(**generator_kwargs)
         else:
             raise ValueError(f"Unsupported API: {api}")
 
@@ -200,12 +201,13 @@ def run(self, **inputs) -> Dict[str, Any]:
         list_of_input_names_to_values = [dict(zip(input_names, v)) for v in values]
 
         results: List[Optional[Dict[str, Any]]] = []
-        metadata = None
+        metadata = []
         errors = 0
         for input_names_to_values in tqdm(list_of_input_names_to_values, disable=not self.progress_bar):
             prompt = self.builder.run(**input_names_to_values)
+            messages = [ChatMessage.from_user(prompt["prompt"])]
             try:
-                result = self.generator.run(prompt=prompt["prompt"])
+                result = self._chat_generator.run(messages=messages)
             except Exception as e:
                 if self.raise_on_failure:
                     raise ValueError(f"Error while generating response for prompt: {prompt}. Error: {e}")
@@ -214,15 +216,15 @@ def run(self, **inputs) -> Dict[str, Any]:
                 errors += 1
                 continue
 
-            if self.is_valid_json_and_has_expected_keys(expected=self.outputs, received=result["replies"][0]):
-                parsed_result = json.loads(result["replies"][0])
+            if self.is_valid_json_and_has_expected_keys(expected=self.outputs, received=result["replies"][0].text):
+                parsed_result = json.loads(result["replies"][0].text)
                 results.append(parsed_result)
             else:
                 results.append(None)
                 errors += 1
 
-            if self.api == "openai" and "meta" in result:
-                metadata = result["meta"]
+            if result["replies"][0].meta:
+                metadata.append(result["replies"][0].meta)
 
         if errors > 0:
             logger.warning(
@@ -231,7 +233,7 @@ def run(self, **inputs) -> Dict[str, Any]:
                 len=len(list_of_input_names_to_values),
             )
 
-        return {"results": results, "meta": metadata}
+        return {"results": results, "meta": metadata or None}
 
     def prepare_template(self) -> str:
         """
diff --git a/releasenotes/notes/llm-evaluators-chat-generator-bf930fa6db019714.yaml b/releasenotes/notes/llm-evaluators-chat-generator-bf930fa6db019714.yaml
new file mode 100644
index 0000000000..0185a1fb5e
--- /dev/null
+++ b/releasenotes/notes/llm-evaluators-chat-generator-bf930fa6db019714.yaml
@@ -0,0 +1,6 @@
+---
+upgrade:
+  - |
+    `LLMEvaluator`, `ContextRelevanceEvaluator`, and `FaithfulnessEvaluator` now internally use a
+    `ChatGenerator` instance instead of a `Generator` instance.
+    The public attribute `generator` has been replaced with `_chat_generator`.
diff --git a/test/components/evaluators/test_context_relevance_evaluator.py b/test/components/evaluators/test_context_relevance_evaluator.py
index 3e33a9f4c4..78d7506c46 100644
--- a/test/components/evaluators/test_context_relevance_evaluator.py
+++ b/test/components/evaluators/test_context_relevance_evaluator.py
@@ -11,6 +11,7 @@
 from haystack import Pipeline
 from haystack.components.evaluators import ContextRelevanceEvaluator
 from haystack.utils.auth import Secret
+from haystack.dataclasses.chat_message import ChatMessage
 
 
 class TestContextRelevanceEvaluator:
@@ -18,7 +19,7 @@ def test_init_default(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = ContextRelevanceEvaluator()
         assert component.api == "openai"
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.instructions == (
             "Please extract only sentences from the provided context which are absolutely relevant and "
             "required to answer the following question. If no relevant sentences are found, or if you "
@@ -65,7 +66,7 @@ def test_init_with_parameters(self):
                 {"inputs": {"questions": "Football is the most popular sport."}, "outputs": {"custom_score": 0}},
             ],
         )
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.api == "openai"
         assert component.examples == [
             {"inputs": {"questions": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
@@ -107,7 +108,7 @@ def test_from_dict(self, monkeypatch):
         }
         component = ContextRelevanceEvaluator.from_dict(data)
         assert component.api == "openai"
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.examples == [{"inputs": {"questions": "What is football?"}, "outputs": {"score": 0}}]
 
         pipeline = Pipeline()
@@ -118,13 +119,13 @@ def test_run_calculates_mean_score(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = ContextRelevanceEvaluator()
 
-        def generator_run(self, *args, **kwargs):
-            if "Football" in kwargs["prompt"]:
-                return {"replies": ['{"relevant_statements": ["a", "b"], "score": 1}']}
+        def chat_generator_run(self, *args, **kwargs):
+            if "Football" in kwargs["messages"][0].text:
+                return {"replies": [ChatMessage.from_assistant('{"relevant_statements": ["a", "b"], "score": 1}')]}
             else:
-                return {"replies": ['{"relevant_statements": [], "score": 0}']}
+                return {"replies": [ChatMessage.from_assistant('{"relevant_statements": [], "score": 0}')]}
 
-        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+        monkeypatch.setattr("haystack.components.evaluators.llm_evaluator.OpenAIChatGenerator.run", chat_generator_run)
 
         questions = ["Which is the most popular global sport?", "Who created the Python language?"]
         contexts = [
@@ -152,13 +153,13 @@ def test_run_no_statements_extracted(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = ContextRelevanceEvaluator()
 
-        def generator_run(self, *args, **kwargs):
-            if "Football" in kwargs["prompt"]:
-                return {"replies": ['{"relevant_statements": ["a", "b"], "score": 1}']}
+        def chat_generator_run(self, *args, **kwargs):
+            if "Football" in kwargs["messages"][0].text:
+                return {"replies": [ChatMessage.from_assistant('{"relevant_statements": ["a", "b"], "score": 1}')]}
             else:
-                return {"replies": ['{"relevant_statements": [], "score": 0}']}
+                return {"replies": [ChatMessage.from_assistant('{"relevant_statements": [], "score": 0}')]}
 
-        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+        monkeypatch.setattr("haystack.components.evaluators.llm_evaluator.OpenAIChatGenerator.run", chat_generator_run)
 
         questions = ["Which is the most popular global sport?", "Who created the Python language?"]
         contexts = [
@@ -188,13 +189,13 @@ def test_run_returns_nan_raise_on_failure_false(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = ContextRelevanceEvaluator(raise_on_failure=False)
 
-        def generator_run(self, *args, **kwargs):
-            if "Python" in kwargs["prompt"]:
+        def chat_generator_run(self, *args, **kwargs):
+            if "Python" in kwargs["messages"][0].text:
                 raise Exception("OpenAI API request failed.")
             else:
-                return {"replies": ['{"relevant_statements": ["c", "d"], "score": 1}']}
+                return {"replies": [ChatMessage.from_assistant('{"relevant_statements": ["c", "d"], "score": 1}')]}
 
-        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+        monkeypatch.setattr("haystack.components.evaluators.llm_evaluator.OpenAIChatGenerator.run", chat_generator_run)
 
         questions = ["Which is the most popular global sport?", "Who created the Python language?"]
         contexts = [
diff --git a/test/components/evaluators/test_faithfulness_evaluator.py b/test/components/evaluators/test_faithfulness_evaluator.py
index de92388ece..f92d280db5 100644
--- a/test/components/evaluators/test_faithfulness_evaluator.py
+++ b/test/components/evaluators/test_faithfulness_evaluator.py
@@ -10,6 +10,7 @@
 from haystack import Pipeline
 from haystack.components.evaluators import FaithfulnessEvaluator
 from haystack.utils.auth import Secret
+from haystack.dataclasses.chat_message import ChatMessage
 
 
 class TestFaithfulnessEvaluator:
@@ -17,7 +18,7 @@ def test_init_default(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = FaithfulnessEvaluator()
         assert component.api == "openai"
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.instructions == (
             "Your task is to judge the faithfulness or groundedness of statements based "
             "on context information. First, please extract statements from a provided predicted "
@@ -84,7 +85,7 @@ def test_init_with_parameters(self):
                 },
             ],
         )
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.api == "openai"
         assert component.examples == [
             {"inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
@@ -132,7 +133,7 @@ def test_from_dict(self, monkeypatch):
         }
         component = FaithfulnessEvaluator.from_dict(data)
         assert component.api == "openai"
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.examples == [
             {"inputs": {"predicted_answers": "Football is the most popular sport."}, "outputs": {"score": 0}}
         ]
@@ -145,13 +146,17 @@ def test_run_calculates_mean_score(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = FaithfulnessEvaluator()
 
-        def generator_run(self, *args, **kwargs):
-            if "Football" in kwargs["prompt"]:
-                return {"replies": ['{"statements": ["a", "b"], "statement_scores": [1, 0]}']}
+        def chat_generator_run(self, *args, **kwargs):
+            if "Football" in kwargs["messages"][0].text:
+                return {
+                    "replies": [ChatMessage.from_assistant('{"statements": ["a", "b"], "statement_scores": [1, 0]}')]
+                }
             else:
-                return {"replies": ['{"statements": ["c", "d"], "statement_scores": [1, 1]}']}
+                return {
+                    "replies": [ChatMessage.from_assistant('{"statements": ["c", "d"], "statement_scores": [1, 1]}')]
+                }
 
-        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+        monkeypatch.setattr("haystack.components.evaluators.llm_evaluator.OpenAIChatGenerator.run", chat_generator_run)
 
         questions = ["Which is the most popular global sport?", "Who created the Python language?"]
         contexts = [
@@ -186,13 +191,15 @@ def test_run_no_statements_extracted(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = FaithfulnessEvaluator()
 
-        def generator_run(self, *args, **kwargs):
-            if "Football" in kwargs["prompt"]:
-                return {"replies": ['{"statements": ["a", "b"], "statement_scores": [1, 0]}']}
+        def chat_generator_run(self, *args, **kwargs):
+            if "Football" in kwargs["messages"][0].text:
+                return {
+                    "replies": [ChatMessage.from_assistant('{"statements": ["a", "b"], "statement_scores": [1, 0]}')]
+                }
             else:
-                return {"replies": ['{"statements": [], "statement_scores": []}']}
+                return {"replies": [ChatMessage.from_assistant('{"statements": [], "statement_scores": []}')]}
 
-        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+        monkeypatch.setattr("haystack.components.evaluators.llm_evaluator.OpenAIChatGenerator.run", chat_generator_run)
 
         questions = ["Which is the most popular global sport?", "Who created the Python language?"]
         contexts = [
@@ -229,13 +236,15 @@ def test_run_returns_nan_raise_on_failure_false(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
         component = FaithfulnessEvaluator(raise_on_failure=False)
 
-        def generator_run(self, *args, **kwargs):
-            if "Python" in kwargs["prompt"]:
+        def chat_generator_run(self, *args, **kwargs):
+            if "Python" in kwargs["messages"][0].text:
                 raise Exception("OpenAI API request failed.")
             else:
-                return {"replies": ['{"statements": ["c", "d"], "statement_scores": [1, 1]}']}
+                return {
+                    "replies": [ChatMessage.from_assistant('{"statements": ["c", "d"], "statement_scores": [1, 1]}')]
+                }
 
-        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+        monkeypatch.setattr("haystack.components.evaluators.llm_evaluator.OpenAIChatGenerator.run", chat_generator_run)
 
         questions = ["Which is the most popular global sport?", "Who created the Python language?"]
         contexts = [
diff --git a/test/components/evaluators/test_llm_evaluator.py b/test/components/evaluators/test_llm_evaluator.py
index e0feb6c0f4..b9fa900ffc 100644
--- a/test/components/evaluators/test_llm_evaluator.py
+++ b/test/components/evaluators/test_llm_evaluator.py
@@ -9,6 +9,7 @@
 from haystack import Pipeline
 from haystack.components.evaluators import LLMEvaluator
 from haystack.utils.auth import Secret
+from haystack.dataclasses.chat_message import ChatMessage
 
 
 class TestLLMEvaluator:
@@ -23,7 +24,7 @@ def test_init_default(self, monkeypatch):
             ],
         )
         assert component.api == "openai"
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}}
         assert component.instructions == "test-instruction"
         assert component.inputs == [("predicted_answers", List[str])]
@@ -64,7 +65,7 @@ def test_init_with_parameters(self):
                 },
             ],
         )
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 43}}
         assert component.api == "openai"
         assert component.examples == [
@@ -239,7 +240,7 @@ def test_from_dict(self, monkeypatch):
         }
         component = LLMEvaluator.from_dict(data)
         assert component.api == "openai"
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}}
         assert component.instructions == "test-instruction"
         assert component.inputs == [("predicted_answers", List[str])]
@@ -317,10 +318,10 @@ def test_run_with_different_lengths(self, monkeypatch):
             ],
         )
 
-        def generator_run(self, *args, **kwargs):
-            return {"replies": ['{"score": 0.5}']}
+        def chat_generator_run(self, *args, **kwargs):
+            return {"replies": [ChatMessage.from_assistant('{"score": 0.5}')]}
 
-        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+        monkeypatch.setattr("haystack.components.evaluators.llm_evaluator.OpenAIChatGenerator.run", chat_generator_run)
 
         with pytest.raises(ValueError):
             component.run(questions=["What is the capital of Germany?"], predicted_answers=[["Berlin"], ["Paris"]])
@@ -342,10 +343,10 @@ def test_run_returns_parsed_result(self, monkeypatch):
             ],
         )
 
-        def generator_run(self, *args, **kwargs):
-            return {"replies": ['{"score": 0.5}']}
+        def chat_generator_run(self, *args, **kwargs):
+            return {"replies": [ChatMessage.from_assistant('{"score": 0.5}')]}
 
-        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+        monkeypatch.setattr("haystack.components.evaluators.llm_evaluator.OpenAIChatGenerator.run", chat_generator_run)
 
         results = component.run(questions=["What is the capital of Germany?"], predicted_answers=["Berlin"])
         assert results == {"results": [{"score": 0.5}], "meta": None}
@@ -471,7 +472,7 @@ def test_init_with_base_url(self):
                 },
             ],
         )
-        assert component.generator.client.api_key == "test-api-key"
+        assert component._chat_generator.client.api_key == "test-api-key"
         assert component.api_params == {
             "generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42},
             "api_base_url": "http://127.0.0.1:11434/v1",


Reviews:
user: login: davidsbatista
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T12:55:31Z
user: login: davidsbatista
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T12:56:48Z
user: login: davidsbatista
body: LGTM! just a small suggestion for the release notes
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T13:04:46Z
user: login: anakin87
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T13:40:58Z

Review comments:
diff_hunk: @@ -200,12 +201,13 @@ def run(self, **inputs) -> Dict[str, Any]:
         list_of_input_names_to_values = [dict(zip(input_names, v)) for v in values]
 
         results: List[Optional[Dict[str, Any]]] = []
-        metadata = None
+        metadata = []
path: haystack/components/evaluators/llm_evaluator.py
user: login: davidsbatista
body: 👍🏽 nice - I think we should avoid `None` as much as possible
created_at: 2025-03-26T12:55:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,6 @@
+---
+upgrade:
+  - |
+    `LLMEvaluator`, `ContextRelevanceEvaluator`, and `FaithfulnessEvaluator` now internally use a
+    `ChatGenerator` instance instead of a `Generator` instance.
+    The public attribute `generator` has been removed and replaced with `_chat_generator`.
path: releasenotes/notes/llm-evaluators-chat-generator-bf930fa6db019714.yaml
user: login: davidsbatista
body: ```suggestion
    The public attribute `generator` has been replaced with `_chat_generator`.
```
created_at: 2025-03-26T12:56:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -200,12 +201,13 @@ def run(self, **inputs) -> Dict[str, Any]:
         list_of_input_names_to_values = [dict(zip(input_names, v)) for v in values]
 
         results: List[Optional[Dict[str, Any]]] = []
-        metadata = None
+        metadata = []
path: haystack/components/evaluators/llm_evaluator.py
user: login: anakin87
body: TBH, I am still returning `None` if the list is empty to avoid breaking the existing behavior.
created_at: 2025-03-26T13:40:58Z
author_association: MEMBER

PR comments:
user: login: coveralls
created_at: 2025-03-26T10:07:28Z
body: ## Pull Request Test Coverage Report for [Build 14084787058](https://coveralls.io/builds/72932967)


### Details

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* **6** unchanged lines in **1** file lost coverage.
* Overall coverage increased (+**0.001%**) to **90.15%**

---


|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [components/evaluators/llm_evaluator.py](https://coveralls.io/builds/72932967/source?filename=components%2Fevaluators%2Fllm_evaluator.py#L213) | 6 | 95.04% |
<!-- | **Total:** | **6** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/72932967/badge)](https://coveralls.io/builds/72932967) |
| :-- | --: |
| Change from base [Build 14060960165](https://coveralls.io/builds/72911230): |  0.001% |
| Covered Lines: | 9894 |
| Relevant Lines: | 10975 |

---
##### 💛  - [Coveralls](https://coveralls.io)

author_association: COLLABORATOR
user: login: davidsbatista
created_at: 2025-03-26T13:04:20Z
body: LGTM! 

I initially thought a deprecation period would be needed but since the generator attribute was never marked as public API this is good to go.


author_association: CONTRIBUTOR
user: login: anakin87
created_at: 2025-03-26T13:47:17Z
body: > I initially thought a deprecation period would be needed but since the generator attribute was never marked as public API this is good to go.

`self.generator` is kind of part of the public API (otherwise it would have been `self._generator`).

Replacing it with `self._chat_generator` is technically a breaking change, and we should follow our policy in general.

In this case, I propose to make an exception for two reasons:
- I don't expect many users to directly access `self.generator`
- following the deprecation policy in this case would also mean delaying the second part of #9062, where we will allow users to pass their own `ChatGenerator` instance.


author_association: MEMBER