Repository: deepset-ai/haystack
Pull request: feat: Add new LLMEvaluator component
### Related Issues

- fixes #7023 

### Proposed Changes:
- Add a new component LLMEvaluator that is limited to the OpenAI api for now and evaluates inputs based on provided instructions and examples

### How did you test it?

- New unit tests
- I used an [example](https://github.com/deepset-ai/haystack/blob/521c80aef077eb4a7282bdd903148a383c8df053/haystack/components/evaluators/example.py) that can be adjusted to become an integration test or end to end test.

### Notes for the reviewer

An open question currently is how to enforce proper json formatting in the rendered prompt template. Main issue I see is " or ' being used inconsistently and thus maybe causing worse generated results. 

Current output is `{'score': 1, 'name': 'llm'}` in the style of the evaluation framework integrations.
We can make `llm` customizable as an additional parameter of `LLMEvaluator` later if it makes sense or leave it out completely.

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- I documented my code
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
LLM Eval - Implement custom LLM evaluator component in core
Implement a `LLMEvaluator` component with the following parameters:
- `api` - API to use (openai, cohere, etc)
- `api_key` - API secret
- `inputs` - tuple of expected inputs
    - Will act as the sockets for the component
- `outputs` - keys to look up in the output JSON
- `instruction` - prompt to the LLM describing what it should do
    - Prompt should always return a json string
    - We can (try to )ensure this by modifying the prompt internally
- `examples` - One or more examples conforming to the input and output format


The component sets up the backend, validates/formats the inputs, calls the backend, validates the output and returns them.

Hints from issue discussion:

Commit message: make outputs parameter optional. default:

Patch to review:
diff --git a/haystack/components/evaluators/__init__.py b/haystack/components/evaluators/__init__.py
index 9550a5f42d..88e78e5756 100644
--- a/haystack/components/evaluators/__init__.py
+++ b/haystack/components/evaluators/__init__.py
@@ -1,3 +1,4 @@
 from .answer_exact_match import AnswerExactMatchEvaluator
+from .llm_evaluator import LLMEvaluator
 
-__all__ = ["AnswerExactMatchEvaluator"]
+__all__ = ["AnswerExactMatchEvaluator", "LLMEvaluator"]
diff --git a/haystack/components/evaluators/llm_evaluator.py b/haystack/components/evaluators/llm_evaluator.py
new file mode 100644
index 0000000000..4e5a2d99a7
--- /dev/null
+++ b/haystack/components/evaluators/llm_evaluator.py
@@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
+    ):
+        """
+        Creates an instance of LLMEvaluator.
+
+        :param instructions:
+            The prompt instructions to use for evaluation.
+            Should be a question about the inputs that can be answered with yes or no.
+        :param inputs:
+            The inputs that the component expects as incoming connections and that it evaluates.
+            Each input is a tuple of an input name and input type. Input types must be lists.
+        :param outputs:
+            Optional output names of the evaluation results. They correspond to keys in the output dictionary.
+            The default is a single key "score".
+        :param api:
+            The API to use for calling an LLM through a Generator.
+            Supported APIs: "openai".
+        :param api_key:
+            The API key.
+        :param examples:
+            Optional few-shot examples conforming to the expected input and output format as defined in the `inputs` and
+             `outputs` parameters.
+            Each example is a dictionary with keys "inputs" and "outputs"
+            They contain the input and output as dictionaries respectively.
+        """
+        self.instructions = instructions
+        self.inputs = inputs
+        self.outputs = outputs or ["score"]
+        self.api = api
+        self.api_key = api_key
+        self.examples = examples
+        expected_inputs = dict(inputs)
+        if api == "openai":
+            self.generator = OpenAIGenerator(api_key=api_key)
+        else:
+            raise ValueError(f"Unsupported API: {api}")
+
+        component.set_input_types(self, **expected_inputs)
+
+    @component.output_types(results=List[List[Dict[str, Any]]])
+    def run(self, **inputs) -> Dict[str, Any]:
+        """
+        Run the LLM evaluator.
+
+        :param inputs:
+            The input values to evaluate. The keys are the input names and the values are lists of input values.
+        :returns:
+            A dictionary with a single `results` entry that contains a list of results.
+            Each result is a dictionary containing the keys as defined in the `outputs` parameter of the LLMEvaluator
+            and the evaluation results as the values.
+        """
+        self.validate_input_parameters(dict(self.inputs), inputs)
+        self.validate_lengths(*inputs.values())
+
+        template = self.prepare_template()
+        builder = PromptBuilder(template=template)
+
+        # inputs is a dictionary with keys being input names and values being a list of input values
+        # We need to iterate through the lists in parallel for all keys of the dictionary
+        input_names, values = inputs.keys(), list(zip(*inputs.values()))
+        list_of_input_names_to_values = [dict(zip(input_names, v)) for v in values]
+
+        results = []
+        for input_names_to_values in list_of_input_names_to_values:
+            prompt = builder.run(**input_names_to_values)
+            result = self.generator.run(prompt=prompt["prompt"])
+
+            self.validate_outputs(expected=self.outputs, received=result["replies"][0])
+            parsed_result = json.loads(result["replies"][0])
+            parsed_result["name"] = "llm"
+            results.append(parsed_result)
+
+        return {"results": results}
+
+    def prepare_template(self) -> str:
+        """
+        Combine instructions, inputs, outputs, and examples into one prompt template.
+
+        :returns:
+            The prompt template.
+        """
+        inputs_section = (
+            "{" + ",".join([f'"{input_socket[0]}": {{{{ {input_socket[0]} }}}}' for input_socket in self.inputs]) + "}"
+        )
+        examples_section = ""
+        if self.examples:
+            for example in self.examples:
+                examples_section += (
+                    "Inputs:\n" + json.dumps(example["inputs"]) + "\nOutputs:\n" + json.dumps(example["outputs"]) + "\n"
+                )
+        return (
+            f"Respond only in JSON format with a key {json.dumps(self.outputs)} and a value of either 0 for FALSE "
+            f"or 1 for TRUE.\n{self.instructions}\n{examples_section}Inputs:\n{inputs_section}\nOutputs:\n"
+        )
+
+    def _get_telemetry_data(self) -> Dict[str, Any]:
+        """
+        Data that is sent to Posthog for usage analytics.
+        """
+        return {"api": self.api}
+
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Serialize this component to a dictionary.
+
+        :returns:
+            The serialized component as a dictionary.
+        """
+        return default_to_dict(
+            self,
+            instructions=self.instructions,
+            inputs=self.inputs,
+            outputs=self.outputs,
+            api=self.api,
+            api_key=self.api_key.to_dict(),
+            examples=self.examples,
+        )
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "LLMEvaluator":
+        """
+        Deserialize this component from a dictionary.
+
+        :param data:
+            The dictionary representation of this component.
+        :returns:
+            The deserialized component instance.
+        """
+        deserialize_secrets_inplace(data["init_parameters"], keys=["api_key"])
+        return default_from_dict(cls, data)
+
+    @staticmethod
+    def validate_lengths(*lists):
+        """
+        Validate that all input lists have the same length.
+
+        :param lists:
+            The lists to validate.
+
+        :raises ValueError:
+            If not all input lists have the same length
+        """
+        length = len(lists[0])
+        if all(len(lst) == length for lst in lists[1:]):
+            return True
+        else:
+            msg = (
+                f"LLM evaluator expects all input lists to have the same length but received {lists} with lengths "
+                f"{[len(lst) for lst in lists]}."
+            )
+            raise ValueError(msg)
+
+    @staticmethod
+    def validate_input_parameters(expected: Dict[str, Any], received: Dict[str, Any]) -> None:
+        """
+        Validate the input parameters.
+
+        :param expected:
+            The expected input parameters.
+        :param received:
+            The received input parameters.
+
+        :raises ValueError:
+            If not all expected inputs are present in the received inputs
+        """
+        for param in expected.keys():
+            if param not in received:
+                msg = f"LLM evaluator expected input parameter '{param}' but received only {received.keys()}."
+                raise ValueError(msg)
+
+    @staticmethod
+    def validate_outputs(expected: List[str], received: str) -> None:
+        """
+        Validate the output.
+
+        :param expected:
+            Names of expected outputs
+        :param received:
+            Names of received outputs
+
+        :raises ValueError:
+            If not all expected outputs are present in the received outputs
+        """
+        parsed_output = json.loads(received)
+        if not all(output in parsed_output for output in expected):
+            msg = f"Expected response from LLM evaluator to be JSON with keys {expected}, got {received}."
+            raise ValueError(msg)
diff --git a/releasenotes/notes/llmevaluator-0ae63b2b9715fb9b.yaml b/releasenotes/notes/llmevaluator-0ae63b2b9715fb9b.yaml
new file mode 100644
index 0000000000..3b68f16a33
--- /dev/null
+++ b/releasenotes/notes/llmevaluator-0ae63b2b9715fb9b.yaml
@@ -0,0 +1,4 @@
+---
+features:
+  - |
+    Add a new LLMEvaluator component that leverages LLMs through the OpenAI api to evaluate pipelines.
diff --git a/test/components/evaluators/test_llm_evaluator.py b/test/components/evaluators/test_llm_evaluator.py
new file mode 100644
index 0000000000..da2816f8de
--- /dev/null
+++ b/test/components/evaluators/test_llm_evaluator.py
@@ -0,0 +1,162 @@
+import os
+from typing import List
+
+import pytest
+
+from haystack.components.evaluators import LLMEvaluator
+from haystack.utils.auth import Secret
+
+
+class TestLLMEvaluator:
+    def test_init_default(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = LLMEvaluator(instructions="test-instruction", inputs=[("responses", List[str])])
+        assert component.api == "openai"
+        assert component.generator.client.api_key == "test-api-key"
+        assert component.instructions == "test-instruction"
+        assert component.inputs == [("responses", List[str])]
+        assert component.outputs == ["score"]
+        assert component.examples == None
+
+    def test_init_fail_wo_openai_api_key(self, monkeypatch):
+        monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+        with pytest.raises(ValueError, match="None of the .* environment variables are set"):
+            LLMEvaluator(api="openai", instructions="test-instruction", inputs=[("responses", List[str])])
+
+    def test_init_with_parameters(self):
+        component = LLMEvaluator(
+            instructions="test-instruction",
+            api_key=Secret.from_token("test-api-key"),
+            inputs=[("responses", List[str])],
+            outputs=["custom_score"],
+            api="openai",
+            examples=[
+                {"inputs": {"responses": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
+                {"inputs": {"responses": "Football is the most popular sport."}, "outputs": {"custom_score": 0}},
+            ],
+        )
+        assert component.generator.client.api_key == "test-api-key"
+        assert component.api == "openai"
+        assert component.examples == [
+            {"inputs": {"responses": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
+            {"inputs": {"responses": "Football is the most popular sport."}, "outputs": {"custom_score": 0}},
+        ]
+        assert component.instructions == "test-instruction"
+        assert component.inputs == [("responses", List[str])]
+        assert component.outputs == ["custom_score"]
+
+    def test_to_dict_default(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = LLMEvaluator(instructions="test-instruction", inputs=[("responses", List[str])])
+        data = component.to_dict()
+        assert data == {
+            "type": "haystack.components.evaluators.llm_evaluator.LLMEvaluator",
+            "init_parameters": {
+                "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
+                "api": "openai",
+                "instructions": "test-instruction",
+                "inputs": [("responses", List[str])],
+                "outputs": ["score"],
+                "examples": None,
+            },
+        }
+
+    def test_to_dict_with_parameters(self, monkeypatch):
+        monkeypatch.setenv("ENV_VAR", "test-api-key")
+        component = LLMEvaluator(
+            instructions="test-instruction",
+            api_key=Secret.from_env_var("ENV_VAR"),
+            inputs=[("responses", List[str])],
+            outputs=["custom_score"],
+            api="openai",
+            examples=[
+                {"inputs": {"responses": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
+                {"inputs": {"responses": "Football is the most popular sport."}, "outputs": {"custom_score": 0}},
+            ],
+        )
+        data = component.to_dict()
+        assert data == {
+            "type": "haystack.components.evaluators.llm_evaluator.LLMEvaluator",
+            "init_parameters": {
+                "api_key": {"env_vars": ["ENV_VAR"], "strict": True, "type": "env_var"},
+                "api": "openai",
+                "instructions": "test-instruction",
+                "inputs": [("responses", List[str])],
+                "outputs": ["custom_score"],
+                "examples": [
+                    {"inputs": {"responses": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
+                    {"inputs": {"responses": "Football is the most popular sport."}, "outputs": {"custom_score": 0}},
+                ],
+            },
+        }
+
+    def test_run_with_different_lengths(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = LLMEvaluator(
+            instructions="test-instruction",
+            inputs=[("questions", List[str]), ("responses", List[List[str]])],
+            outputs=["score"],
+        )
+
+        def generator_run(self, *args, **kwargs):
+            return {"replies": [{"score": 0.5}]}
+
+        monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
+
+        with pytest.raises(ValueError):
+            component.run(questions=["What is the capital of Germany?"], responses=[["Berlin"], ["Paris"]])
+
+        with pytest.raises(ValueError):
+            component.run(
+                questions=["What is the capital of Germany?", "What is the capital of France?"], responses=[["Berlin"]]
+            )
+
+    def test_prepare_template_wo_examples(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = LLMEvaluator(instructions="test-instruction", inputs=[("responses", List[str])], outputs=["score"])
+        template = component.prepare_template()
+        assert (
+            template
+            == 'Respond only in JSON format with a key ["score"] and a value of either 0 for FALSE or 1 for TRUE.\ntest-instruction\nInputs:\n{"responses": {{ responses }}}\nOutputs:\n'
+        )
+
+    def test_prepare_template_with_examples(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = LLMEvaluator(
+            instructions="test-instruction",
+            inputs=[("responses", List[str])],
+            outputs=["score"],
+            examples=[
+                {"inputs": {"responses": "Damn, this is straight outta hell!!!"}, "outputs": {"score": 1}},
+                {"inputs": {"responses": "Football is the most popular sport."}, "outputs": {"score": 0}},
+            ],
+        )
+        template = component.prepare_template()
+        assert (
+            template
+            == 'Respond only in JSON format with a key ["score"] and a value of either 0 for FALSE or 1 for TRUE.\ntest-instruction\nInputs:\n{"responses": "Damn, this is straight outta hell!!!"}\nOutputs:\n{"score": 1}\nInputs:\n{"responses": "Football is the most popular sport."}\nOutputs:\n{"score": 0}\nInputs:\n{"responses": {{ responses }}}\nOutputs:\n'
+        )
+
+    def test_invalid_input_parameters(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = LLMEvaluator(instructions="test-instruction", inputs=[("responses", List[str])])
+        with pytest.raises(ValueError):
+            component.validate_input_parameters(expected={"responses": List[str]}, received={"questions": List[str]})
+
+        with pytest.raises(ValueError):
+            component.validate_input_parameters(
+                expected={"responses": List[str], "questions": List[str]}, received={"questions": List[str]}
+            )
+
+    def test_invalid_outputs(self, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")
+        component = LLMEvaluator(instructions="test-instruction", inputs=[("responses", List[str])])
+        with pytest.raises(ValueError):
+            component.validate_outputs(expected=["score", "another_expected_output"], received="{'score': 1.0}")
+
+        with pytest.raises(ValueError):
+            component.validate_outputs(expected=["score"], received="{'wrong_name': 1.0}")
+
+    def test_unsupported_api(self):
+        with pytest.raises(ValueError):
+            LLMEvaluator(api="unsupported_api", instructions="test-instruction", inputs=[("responses", List[str])])


Code review comments:
diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
+    ):
+        """
+        Creates an instance of LLMEvaluator.
+
+        :param instructions:
+            The prompt instructions to use for evaluation.
+            Should be a question about the inputs that can be answered with yes or no.
+        :param inputs:
+            The inputs that the component expects as incoming connections and that it evaluates.
+            Each input is a tuple of an input name and input type. Input types must be lists.
+        :param outputs:
+            Optional output names of the evaluation results. They correspond to keys in the output dictionary.
+            The default is a single key "score".
+        :param api:
+            The API to use for calling an LLM through a Generator.
+            Supported APIs: "openai".
+        :param api_key:
+            The API key.
+        :param examples:
+            Optional few-shot examples conforming to the expected input and output format as defined in the `inputs` and
+             `outputs` parameters.
+            Each example is a dictionary with keys "inputs" and "outputs"
+            They contain the input and output as dictionaries respectively.
+        """
+        self.instructions = instructions
+        self.inputs = inputs
+        self.outputs = outputs or ["score"]
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
```suggestion
        self.outputs = outputs
```
See above. We also need some basic validation here.

@author:
validation is added

diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
This should not be optional.

@author:
Why not? A single key `"score"` seems like a good default to me. That default would be consistent with all the statistical metric evaluators we are adding. We agreed that all of them will have a key `"score"` in the results.

@user1:
Since the idea here is to provide a generic component that is meant to be subclassed/wrapped by another that provides specific logic to interpret/process the output of the LLM, I'd argue that this component shouldn't be opinionated on what the specific inputs and outputs should be - The details aren't part of its concern.

diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
+    ):
+        """
+        Creates an instance of LLMEvaluator.
+
+        :param instructions:
+            The prompt instructions to use for evaluation.
+            Should be a question about the inputs that can be answered with yes or no.
+        :param inputs:
+            The inputs that the component expects as incoming connections and that it evaluates.
+            Each input is a tuple of an input name and input type. Input types must be lists.
+        :param outputs:
+            Optional output names of the evaluation results. They correspond to keys in the output dictionary.
+            The default is a single key "score".
+        :param api:
+            The API to use for calling an LLM through a Generator.
+            Supported APIs: "openai".
+        :param api_key:
+            The API key.
+        :param examples:
+            Optional few-shot examples conforming to the expected input and output format as defined in the `inputs` and
+             `outputs` parameters.
+            Each example is a dictionary with keys "inputs" and "outputs"
+            They contain the input and output as dictionaries respectively.
+        """
+        self.instructions = instructions
+        self.inputs = inputs
+        self.outputs = outputs or ["score"]
+        self.api = api
+        self.api_key = api_key
+        self.examples = examples
+        expected_inputs = dict(inputs)
+        if api == "openai":
+            self.generator = OpenAIGenerator(api_key=api_key)
+        else:
+            raise ValueError(f"Unsupported API: {api}")
+
+        component.set_input_types(self, **expected_inputs)
+
+    @component.output_types(results=List[List[Dict[str, Any]]])
+    def run(self, **inputs) -> Dict[str, Any]:
+        """
+        Run the LLM evaluator.
+
+        :param inputs:
+            The input values to evaluate. The keys are the input names and the values are lists of input values.
+        :returns:
+            A dictionary with a single `results` entry that contains a list of results.
+            Each result is a dictionary containing the keys as defined in the `outputs` parameter of the LLMEvaluator
+            and the evaluation results as the values.
+        """
+        self.validate_input_parameters(dict(self.inputs), inputs)
+        self.validate_lengths(*inputs.values())
+
+        template = self.prepare_template()
+        builder = PromptBuilder(template=template)
+
+        # inputs is a dictionary with keys being input names and values being a list of input values
+        # We need to iterate through the lists in parallel for all keys of the dictionary
+        input_names, values = inputs.keys(), list(zip(*inputs.values()))
+        list_of_input_names_to_values = [dict(zip(input_names, v)) for v in values]
+
+        results = []
+        for input_names_to_values in list_of_input_names_to_values:
+            prompt = builder.run(**input_names_to_values)
+            result = self.generator.run(prompt=prompt["prompt"])
+
+            self.validate_outputs(expected=self.outputs, received=result["replies"][0])
+            parsed_result = json.loads(result["replies"][0])
+            parsed_result["name"] = "llm"
+            results.append(parsed_result)
+
+        return {"results": results}
+
+    def prepare_template(self) -> str:
+        """
+        Combine instructions, inputs, outputs, and examples into one prompt template.
+
+        :returns:
+            The prompt template.
+        """
+        inputs_section = (
+            "{" + ",".join([f'"{input_socket[0]}": {{{{ {input_socket[0]} }}}}' for input_socket in self.inputs]) + "}"
+        )
+        examples_section = ""
+        if self.examples:
+            for example in self.examples:
+                examples_section += (
+                    "Inputs:\n" + json.dumps(example["inputs"]) + "\nOutputs:\n" + json.dumps(example["outputs"]) + "\n"
+                )
+        return (
+            f"Respond only in JSON format with a key {json.dumps(self.outputs)} and a value of either 0 for FALSE "
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
Let's modify the template so that the final output looks something like this:
```
Instruction:
<User instruction>

Generate the response in JSON format with the following keys: 
<list of output keys>
Consider the instruction and the examples below to determine those values.

Examples:
<dump of the examples>
```
We shouldn't force the model to return boolean values for all keys - we might want to get explanations too, for instance. Hopefully, that's generic enough to get the job done, but we can iterate on the prompt when we implement the downstream metrics.

@author:
Okay, I made the prompt template more generic now. 👍

diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
+    ):
+        """
+        Creates an instance of LLMEvaluator.
+
+        :param instructions:
+            The prompt instructions to use for evaluation.
+            Should be a question about the inputs that can be answered with yes or no.
+        :param inputs:
+            The inputs that the component expects as incoming connections and that it evaluates.
+            Each input is a tuple of an input name and input type. Input types must be lists.
+        :param outputs:
+            Optional output names of the evaluation results. They correspond to keys in the output dictionary.
+            The default is a single key "score".
+        :param api:
+            The API to use for calling an LLM through a Generator.
+            Supported APIs: "openai".
+        :param api_key:
+            The API key.
+        :param examples:
+            Optional few-shot examples conforming to the expected input and output format as defined in the `inputs` and
+             `outputs` parameters.
+            Each example is a dictionary with keys "inputs" and "outputs"
+            They contain the input and output as dictionaries respectively.
+        """
+        self.instructions = instructions
+        self.inputs = inputs
+        self.outputs = outputs or ["score"]
+        self.api = api
+        self.api_key = api_key
+        self.examples = examples
+        expected_inputs = dict(inputs)
+        if api == "openai":
+            self.generator = OpenAIGenerator(api_key=api_key)
+        else:
+            raise ValueError(f"Unsupported API: {api}")
+
+        component.set_input_types(self, **expected_inputs)
+
+    @component.output_types(results=List[List[Dict[str, Any]]])
+    def run(self, **inputs) -> Dict[str, Any]:
+        """
+        Run the LLM evaluator.
+
+        :param inputs:
+            The input values to evaluate. The keys are the input names and the values are lists of input values.
+        :returns:
+            A dictionary with a single `results` entry that contains a list of results.
+            Each result is a dictionary containing the keys as defined in the `outputs` parameter of the LLMEvaluator
+            and the evaluation results as the values.
+        """
+        self.validate_input_parameters(dict(self.inputs), inputs)
+        self.validate_lengths(*inputs.values())
+
+        template = self.prepare_template()
+        builder = PromptBuilder(template=template)
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
This looks like it can be moved to the `init` method?

diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
Same as above.

@author:
I wouldn't limit us to always providing examples. Instructions can also work without them, no?

@user1:
That's a fair point, and it also relates to the aforementioned question. 

IMO, this component should only be prescriptive in providing a general shape of the inputs (in the larger sense) that are required by the component to performs its function. Almost always, few-shot prompting is better than zero-shot prompting. So, we should push the consumer of this component to provide as much information as possible to facilitate a high quality response (particularly since we're talking about evaluation here).

@author:
Okay, I made outputs and examples non-optional. If we want to make them optional at a later point it's not a breaking change so let's move fast. 👍

diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
+    ):
+        """
+        Creates an instance of LLMEvaluator.
+
+        :param instructions:
+            The prompt instructions to use for evaluation.
+            Should be a question about the inputs that can be answered with yes or no.
+        :param inputs:
+            The inputs that the component expects as incoming connections and that it evaluates.
+            Each input is a tuple of an input name and input type. Input types must be lists.
+        :param outputs:
+            Optional output names of the evaluation results. They correspond to keys in the output dictionary.
+            The default is a single key "score".
+        :param api:
+            The API to use for calling an LLM through a Generator.
+            Supported APIs: "openai".
+        :param api_key:
+            The API key.
+        :param examples:
+            Optional few-shot examples conforming to the expected input and output format as defined in the `inputs` and
+             `outputs` parameters.
+            Each example is a dictionary with keys "inputs" and "outputs"
+            They contain the input and output as dictionaries respectively.
+        """
+        self.instructions = instructions
+        self.inputs = inputs
+        self.outputs = outputs or ["score"]
+        self.api = api
+        self.api_key = api_key
+        self.examples = examples
+        expected_inputs = dict(inputs)
+        if api == "openai":
+            self.generator = OpenAIGenerator(api_key=api_key)
+        else:
+            raise ValueError(f"Unsupported API: {api}")
+
+        component.set_input_types(self, **expected_inputs)
+
+    @component.output_types(results=List[List[Dict[str, Any]]])
+    def run(self, **inputs) -> Dict[str, Any]:
+        """
+        Run the LLM evaluator.
+
+        :param inputs:
+            The input values to evaluate. The keys are the input names and the values are lists of input values.
+        :returns:
+            A dictionary with a single `results` entry that contains a list of results.
+            Each result is a dictionary containing the keys as defined in the `outputs` parameter of the LLMEvaluator
+            and the evaluation results as the values.
+        """
+        self.validate_input_parameters(dict(self.inputs), inputs)
+        self.validate_lengths(*inputs.values())
+
+        template = self.prepare_template()
+        builder = PromptBuilder(template=template)
+
+        # inputs is a dictionary with keys being input names and values being a list of input values
+        # We need to iterate through the lists in parallel for all keys of the dictionary
+        input_names, values = inputs.keys(), list(zip(*inputs.values()))
+        list_of_input_names_to_values = [dict(zip(input_names, v)) for v in values]
+
+        results = []
+        for input_names_to_values in list_of_input_names_to_values:
+            prompt = builder.run(**input_names_to_values)
+            result = self.generator.run(prompt=prompt["prompt"])
+
+            self.validate_outputs(expected=self.outputs, received=result["replies"][0])
+            parsed_result = json.loads(result["replies"][0])
+            parsed_result["name"] = "llm"
+            results.append(parsed_result)
+
+        return {"results": results}
+
+    def prepare_template(self) -> str:
+        """
+        Combine instructions, inputs, outputs, and examples into one prompt template.
+
+        :returns:
+            The prompt template.
+        """
+        inputs_section = (
+            "{" + ",".join([f'"{input_socket[0]}": {{{{ {input_socket[0]} }}}}' for input_socket in self.inputs]) + "}"
+        )
+        examples_section = ""
+        if self.examples:
+            for example in self.examples:
+                examples_section += (
+                    "Inputs:\n" + json.dumps(example["inputs"]) + "\nOutputs:\n" + json.dumps(example["outputs"]) + "\n"
+                )
+        return (
+            f"Respond only in JSON format with a key {json.dumps(self.outputs)} and a value of either 0 for FALSE "
+            f"or 1 for TRUE.\n{self.instructions}\n{examples_section}Inputs:\n{inputs_section}\nOutputs:\n"
+        )
+
+    def _get_telemetry_data(self) -> Dict[str, Any]:
+        """
+        Data that is sent to Posthog for usage analytics.
+        """
+        return {"api": self.api}
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
I'd remove this for now - Let's not add telemetry to components proactively unless we have a solid use for the data.

diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
+    ):
+        """
+        Creates an instance of LLMEvaluator.
+
+        :param instructions:
+            The prompt instructions to use for evaluation.
+            Should be a question about the inputs that can be answered with yes or no.
+        :param inputs:
+            The inputs that the component expects as incoming connections and that it evaluates.
+            Each input is a tuple of an input name and input type. Input types must be lists.
+        :param outputs:
+            Optional output names of the evaluation results. They correspond to keys in the output dictionary.
+            The default is a single key "score".
+        :param api:
+            The API to use for calling an LLM through a Generator.
+            Supported APIs: "openai".
+        :param api_key:
+            The API key.
+        :param examples:
+            Optional few-shot examples conforming to the expected input and output format as defined in the `inputs` and
+             `outputs` parameters.
+            Each example is a dictionary with keys "inputs" and "outputs"
+            They contain the input and output as dictionaries respectively.
+        """
+        self.instructions = instructions
+        self.inputs = inputs
+        self.outputs = outputs or ["score"]
+        self.api = api
+        self.api_key = api_key
+        self.examples = examples
+        expected_inputs = dict(inputs)
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
Let's perform some basic validation here to ensure that the inputs are of the correct format.

diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
+    ):
+        """
+        Creates an instance of LLMEvaluator.
+
+        :param instructions:
+            The prompt instructions to use for evaluation.
+            Should be a question about the inputs that can be answered with yes or no.
+        :param inputs:
+            The inputs that the component expects as incoming connections and that it evaluates.
+            Each input is a tuple of an input name and input type. Input types must be lists.
+        :param outputs:
+            Optional output names of the evaluation results. They correspond to keys in the output dictionary.
+            The default is a single key "score".
+        :param api:
+            The API to use for calling an LLM through a Generator.
+            Supported APIs: "openai".
+        :param api_key:
+            The API key.
+        :param examples:
+            Optional few-shot examples conforming to the expected input and output format as defined in the `inputs` and
+             `outputs` parameters.
+            Each example is a dictionary with keys "inputs" and "outputs"
+            They contain the input and output as dictionaries respectively.
+        """
+        self.instructions = instructions
+        self.inputs = inputs
+        self.outputs = outputs or ["score"]
+        self.api = api
+        self.api_key = api_key
+        self.examples = examples
+        expected_inputs = dict(inputs)
+        if api == "openai":
+            self.generator = OpenAIGenerator(api_key=api_key)
+        else:
+            raise ValueError(f"Unsupported API: {api}")
+
+        component.set_input_types(self, **expected_inputs)
+
+    @component.output_types(results=List[List[Dict[str, Any]]])
+    def run(self, **inputs) -> Dict[str, Any]:
+        """
+        Run the LLM evaluator.
+
+        :param inputs:
+            The input values to evaluate. The keys are the input names and the values are lists of input values.
+        :returns:
+            A dictionary with a single `results` entry that contains a list of results.
+            Each result is a dictionary containing the keys as defined in the `outputs` parameter of the LLMEvaluator
+            and the evaluation results as the values.
+        """
+        self.validate_input_parameters(dict(self.inputs), inputs)
+        self.validate_lengths(*inputs.values())
+
+        template = self.prepare_template()
+        builder = PromptBuilder(template=template)
+
+        # inputs is a dictionary with keys being input names and values being a list of input values
+        # We need to iterate through the lists in parallel for all keys of the dictionary
+        input_names, values = inputs.keys(), list(zip(*inputs.values()))
+        list_of_input_names_to_values = [dict(zip(input_names, v)) for v in values]
+
+        results = []
+        for input_names_to_values in list_of_input_names_to_values:
+            prompt = builder.run(**input_names_to_values)
+            result = self.generator.run(prompt=prompt["prompt"])
+
+            self.validate_outputs(expected=self.outputs, received=result["replies"][0])
+            parsed_result = json.loads(result["replies"][0])
+            parsed_result["name"] = "llm"
+            results.append(parsed_result)
+
+        return {"results": results}
+
+    def prepare_template(self) -> str:
+        """
+        Combine instructions, inputs, outputs, and examples into one prompt template.
+
+        :returns:
+            The prompt template.
+        """
+        inputs_section = (
+            "{" + ",".join([f'"{input_socket[0]}": {{{{ {input_socket[0]} }}}}' for input_socket in self.inputs]) + "}"
+        )
+        examples_section = ""
+        if self.examples:
+            for example in self.examples:
+                examples_section += (
+                    "Inputs:\n" + json.dumps(example["inputs"]) + "\nOutputs:\n" + json.dumps(example["outputs"]) + "\n"
+                )
+        return (
+            f"Respond only in JSON format with a key {json.dumps(self.outputs)} and a value of either 0 for FALSE "
+            f"or 1 for TRUE.\n{self.instructions}\n{examples_section}Inputs:\n{inputs_section}\nOutputs:\n"
+        )
+
+    def _get_telemetry_data(self) -> Dict[str, Any]:
+        """
+        Data that is sent to Posthog for usage analytics.
+        """
+        return {"api": self.api}
+
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Serialize this component to a dictionary.
+
+        :returns:
+            The serialized component as a dictionary.
+        """
+        return default_to_dict(
+            self,
+            instructions=self.instructions,
+            inputs=self.inputs,
+            outputs=self.outputs,
+            api=self.api,
+            api_key=self.api_key.to_dict(),
+            examples=self.examples,
+        )
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "LLMEvaluator":
+        """
+        Deserialize this component from a dictionary.
+
+        :param data:
+            The dictionary representation of this component.
+        :returns:
+            The deserialized component instance.
+        """
+        deserialize_secrets_inplace(data["init_parameters"], keys=["api_key"])
+        return default_from_dict(cls, data)
+
+    @staticmethod
+    def validate_lengths(*lists):
+        """
+        Validate that all input lists have the same length.
+
+        :param lists:
+            The lists to validate.
+
+        :raises ValueError:
+            If not all input lists have the same length
+        """
+        length = len(lists[0])
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
I'd fold this function into `validate_input_parameters` and add an extra sanity check to make sure that all the inputs are lists (or nested lists).

diff_hunk: @@ -0,0 +1,230 @@
+import json
+from typing import Any, Dict, List, Optional, Tuple, Type
+
+from haystack import component, default_from_dict, default_to_dict
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class LLMEvaluator:
+    """
+    Uses an LLM to evaluate inputs based on a prompt containing instructions and examples.
+
+    The default API requires an OpenAI API key to be provided as an environment variable "OPENAI_API_KEY".
+    The inputs are lists that are user-defined depending on the desired metric.
+    The output is a dictionary with a key `results` containing a list of evaluation results.
+    Each result is a dictionary with user-defined keys and values of either 0 for FALSE or 1 for TRUE respectively.
+
+    Usage example:
+    ```python
+    from haystack.components.evaluators import LLMEvaluator
+    evaluator = LLMEvaluator(
+        instructions="Is this answer problematic for children?",
+        inputs=[("responses", List[str])],
+        outputs=["score"],
+    )
+    RESPONSES = [
+        "Football is the most popular sport with around 4 billion followers worldwide",
+        "Python language was created by Guido van Rossum.",
+    ]
+    results = evaluator.run(responses=RESPONSES)
+    ```
+    """
+
+    def __init__(
+        self,
+        instructions: str,
+        inputs: List[Tuple[str, Type]],
+        *,
+        outputs: Optional[List[str]] = None,
+        api: str = "openai",
+        api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        examples: Optional[List[Dict[str, Any]]] = None,
+    ):
+        """
+        Creates an instance of LLMEvaluator.
+
+        :param instructions:
+            The prompt instructions to use for evaluation.
+            Should be a question about the inputs that can be answered with yes or no.
+        :param inputs:
+            The inputs that the component expects as incoming connections and that it evaluates.
+            Each input is a tuple of an input name and input type. Input types must be lists.
+        :param outputs:
+            Optional output names of the evaluation results. They correspond to keys in the output dictionary.
+            The default is a single key "score".
+        :param api:
+            The API to use for calling an LLM through a Generator.
+            Supported APIs: "openai".
+        :param api_key:
+            The API key.
+        :param examples:
+            Optional few-shot examples conforming to the expected input and output format as defined in the `inputs` and
+             `outputs` parameters.
+            Each example is a dictionary with keys "inputs" and "outputs"
+            They contain the input and output as dictionaries respectively.
+        """
+        self.instructions = instructions
+        self.inputs = inputs
+        self.outputs = outputs or ["score"]
+        self.api = api
+        self.api_key = api_key
+        self.examples = examples
+        expected_inputs = dict(inputs)
+        if api == "openai":
+            self.generator = OpenAIGenerator(api_key=api_key)
+        else:
+            raise ValueError(f"Unsupported API: {api}")
+
+        component.set_input_types(self, **expected_inputs)
+
+    @component.output_types(results=List[List[Dict[str, Any]]])
path: haystack/components/evaluators/llm_evaluator.py
text: @user2:
```suggestion
    @user1.output_types(results=List[Dict[str, Any]])
```
Right?

@author:
Right 👍