Repository: deepset-ai/haystack

Issue:
Add support for `AzureOpenAI` in `LLMEvaluator`
**Is your feature request related to a problem? Please describe.**

`LLMEvaluator` currently only supports `OpenAI`, it would be nice if we could use it with the OpenAI models via Azure too.

**Describe the solution you'd like**

I'd like to use evaluators with Azure OpenAI (e.g. `ContextRelevanceEvaluator(api='azure-openai')`)

In addition, I propose to slightly change the design of `LLMEvaluator` to allow more flexibility.
Currently the params `api_key=Secret.from_env_var("OPENAI_API_KEY")` forces the user to provide an env var that is specific to OpenAI, and would not be used by other generators.

What about having something like:

```python
@component
class LLMEvaluator:
    def __init__(
        self,
        instructions: str,
        ...
        api: str = "openai",
        generator_kwargs: Dict[str, Any] = ..., # instead of api_key
    ):
        ...
        self.generator = OpenAIGenerator(**generator_kwargs)
```
?

This wouldn't force the user to provide to `LLMEvaluator` anything specific to the generator. It gives the flexibility to pass anything that the generator can take (e.g. api keys, api version, or `azure_deployment` in case of Azure) via the `generator_kwargs`.
At the same time, if the user doesn't pass anything, the generator would still look for its required env vars during instantiation.

I guess `api_key` needs to enter the deprecation cycle before being removed. Maybe we could just change to `api_key=Secret.from_env_var("OPENAI_API_KEY", strict=False)` until deprecated, so that that var will not be required for other generators.

**Describe alternatives you've considered**

Subclassing the `LLMEvaluator` (and all the child classes ) into a custom component

**Additional context**

Happy to hear your thoughts, also in case there are other better solutions I didn't consider. :)

I'm currently a bit busy with other things, but I may be able to raise PR with the proposal in the next days.


Discussion:
I solved this in my PR for local evaluation support but decided to not proceed with the PR: https://github.com/deepset-ai/haystack/pull/7745

You can take what I built, strip the llama.cpp bits, and keep the generation_kwargs sections.

Resolving patch:
diff --git a/haystack/components/evaluators/context_relevance.py b/haystack/components/evaluators/context_relevance.py
--- a/haystack/components/evaluators/context_relevance.py
+++ b/haystack/components/evaluators/context_relevance.py
@@ -93,6 +93,7 @@ def __init__(
         progress_bar: bool = True,
         api: str = "openai",
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        api_params: Optional[Dict[str, Any]] = None,
         raise_on_failure: bool = True,
     ):
         """
@@ -121,6 +122,8 @@ def __init__(
             Supported APIs: "openai".
         :param api_key:
             The API key.
+        :param api_params:
+            Parameters for an OpenAI API compatible completions call.
         :param raise_on_failure:
             Whether to raise an exception if the API call fails.
 
@@ -137,6 +140,7 @@ def __init__(
         self.examples = examples or _DEFAULT_EXAMPLES
         self.api = api
         self.api_key = api_key
+        self.api_params = api_params or {}
 
         warnings.warn(
             "The output of the ContextRelevanceEvaluator will change in Haystack 2.4.0. "
@@ -152,6 +156,7 @@ def __init__(
             examples=self.examples,
             api=self.api,
             api_key=self.api_key,
+            api_params=self.api_params,
             raise_on_failure=raise_on_failure,
             progress_bar=progress_bar,
         )
@@ -202,6 +207,7 @@ def to_dict(self) -> Dict[str, Any]:
             api_key=self.api_key.to_dict() if self.api_key else None,
             examples=self.examples,
             progress_bar=self.progress_bar,
+            api_params=self.api_params,
             raise_on_failure=self.raise_on_failure,
         )
 
diff --git a/haystack/components/evaluators/faithfulness.py b/haystack/components/evaluators/faithfulness.py
--- a/haystack/components/evaluators/faithfulness.py
+++ b/haystack/components/evaluators/faithfulness.py
@@ -88,6 +88,7 @@ def __init__(
         progress_bar: bool = True,
         api: str = "openai",
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        api_params: Optional[Dict[str, Any]] = None,
         raise_on_failure: bool = True,
     ):
         """
@@ -117,6 +118,8 @@ def __init__(
             Supported APIs: "openai".
         :param api_key:
             The API key.
+        :param api_params:
+            Parameters for an OpenAI API compatible completions call.
         :param raise_on_failure:
             Whether to raise an exception if the API call fails.
 
@@ -133,6 +136,7 @@ def __init__(
         self.examples = examples or _DEFAULT_EXAMPLES
         self.api = api
         self.api_key = api_key
+        self.api_params = api_params or {}
 
         super(FaithfulnessEvaluator, self).__init__(
             instructions=self.instructions,
@@ -141,6 +145,7 @@ def __init__(
             examples=self.examples,
             api=self.api,
             api_key=self.api_key,
+            api_params=self.api_params,
             raise_on_failure=raise_on_failure,
             progress_bar=progress_bar,
         )
@@ -193,6 +198,7 @@ def to_dict(self) -> Dict[str, Any]:
             self,
             api=self.api,
             api_key=self.api_key.to_dict() if self.api_key else None,
+            api_params=self.api_params,
             examples=self.examples,
             progress_bar=self.progress_bar,
             raise_on_failure=self.raise_on_failure,
diff --git a/haystack/components/evaluators/llm_evaluator.py b/haystack/components/evaluators/llm_evaluator.py
--- a/haystack/components/evaluators/llm_evaluator.py
+++ b/haystack/components/evaluators/llm_evaluator.py
@@ -58,6 +58,7 @@ def __init__(
         raise_on_failure: bool = True,
         api: str = "openai",
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        api_params: Optional[Dict[str, Any]] = None,
     ):
         """
         Creates an instance of LLMEvaluator.
@@ -84,6 +85,8 @@ def __init__(
             Supported APIs: "openai".
         :param api_key:
             The API key.
+        :param api_params:
+            Parameters for an OpenAI API compatible completions call.
 
         """
         self.validate_init_parameters(inputs, outputs, examples)
@@ -94,12 +97,16 @@ def __init__(
         self.examples = examples
         self.api = api
         self.api_key = api_key
+        self.api_params = api_params or {}
         self.progress_bar = progress_bar
 
+        default_generation_kwargs = {"response_format": {"type": "json_object"}, "seed": 42}
+        user_generation_kwargs = self.api_params.get("generation_kwargs", {})
+        merged_generation_kwargs = {**default_generation_kwargs, **user_generation_kwargs}
+        self.api_params["generation_kwargs"] = merged_generation_kwargs
+
         if api == "openai":
-            self.generator = OpenAIGenerator(
-                api_key=api_key, generation_kwargs={"response_format": {"type": "json_object"}, "seed": 42}
-            )
+            self.generator = OpenAIGenerator(api_key=api_key, **self.api_params)
         else:
             raise ValueError(f"Unsupported API: {api}")
 
@@ -282,6 +289,7 @@ def to_dict(self) -> Dict[str, Any]:
             examples=self.examples,
             api=self.api,
             api_key=self.api_key.to_dict(),
+            api_params=self.api_params,
             progress_bar=self.progress_bar,
         )
 


Test patch:
diff --git a/test/components/evaluators/test_context_relevance_evaluator.py b/test/components/evaluators/test_context_relevance_evaluator.py
--- a/test/components/evaluators/test_context_relevance_evaluator.py
+++ b/test/components/evaluators/test_context_relevance_evaluator.py
@@ -100,6 +100,7 @@ def test_to_dict_with_parameters(self, monkeypatch):
             "init_parameters": {
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": True, "type": "env_var"},
                 "api": "openai",
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "examples": [{"inputs": {"questions": "What is football?"}, "outputs": {"score": 0}}],
                 "progress_bar": False,
                 "raise_on_failure": False,
diff --git a/test/components/evaluators/test_faithfulness_evaluator.py b/test/components/evaluators/test_faithfulness_evaluator.py
--- a/test/components/evaluators/test_faithfulness_evaluator.py
+++ b/test/components/evaluators/test_faithfulness_evaluator.py
@@ -109,6 +109,7 @@ def test_to_dict_with_parameters(self, monkeypatch):
             "init_parameters": {
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": True, "type": "env_var"},
                 "api": "openai",
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "examples": [
                     {"inputs": {"predicted_answers": "Football is the most popular sport."}, "outputs": {"score": 0}}
                 ],
diff --git a/test/components/evaluators/test_llm_evaluator.py b/test/components/evaluators/test_llm_evaluator.py
--- a/test/components/evaluators/test_llm_evaluator.py
+++ b/test/components/evaluators/test_llm_evaluator.py
@@ -1,6 +1,7 @@
 # SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
 #
 # SPDX-License-Identifier: Apache-2.0
+import os
 from typing import List
 
 import pytest
@@ -23,6 +24,7 @@ def test_init_default(self, monkeypatch):
         )
         assert component.api == "openai"
         assert component.generator.client.api_key == "test-api-key"
+        assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}}
         assert component.instructions == "test-instruction"
         assert component.inputs == [("predicted_answers", List[str])]
         assert component.outputs == ["score"]
@@ -47,6 +49,7 @@ def test_init_with_parameters(self):
         component = LLMEvaluator(
             instructions="test-instruction",
             api_key=Secret.from_token("test-api-key"),
+            api_params={"generation_kwargs": {"seed": 43}},
             inputs=[("predicted_answers", List[str])],
             outputs=["custom_score"],
             api="openai",
@@ -62,6 +65,7 @@ def test_init_with_parameters(self):
             ],
         )
         assert component.generator.client.api_key == "test-api-key"
+        assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 43}}
         assert component.api == "openai"
         assert component.examples == [
             {"inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
@@ -202,6 +206,7 @@ def test_to_dict_default(self, monkeypatch):
         assert data == {
             "type": "haystack.components.evaluators.llm_evaluator.LLMEvaluator",
             "init_parameters": {
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
                 "api": "openai",
                 "instructions": "test-instruction",
@@ -220,6 +225,7 @@ def test_from_dict(self, monkeypatch):
         data = {
             "type": "haystack.components.evaluators.llm_evaluator.LLMEvaluator",
             "init_parameters": {
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
                 "api": "openai",
                 "instructions": "test-instruction",
@@ -233,6 +239,7 @@ def test_from_dict(self, monkeypatch):
         component = LLMEvaluator.from_dict(data)
         assert component.api == "openai"
         assert component.generator.client.api_key == "test-api-key"
+        assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}}
         assert component.instructions == "test-instruction"
         assert component.inputs == [("predicted_answers", List[str])]
         assert component.outputs == ["score"]
@@ -263,6 +270,7 @@ def test_to_dict_with_parameters(self, monkeypatch):
         assert data == {
             "type": "haystack.components.evaluators.llm_evaluator.LLMEvaluator",
             "init_parameters": {
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": True, "type": "env_var"},
                 "api": "openai",
                 "instructions": "test-instruction",
@@ -442,3 +450,63 @@ def test_unsupported_api(self):
                     {"inputs": {"predicted_answers": "Football is the most popular sport."}, "outputs": {"score": 0}}
                 ],
             )
+
+    def test_init_with_base_url(self):
+        component = LLMEvaluator(
+            instructions="test-instruction",
+            api_key=Secret.from_token("test-api-key"),
+            api_params={"api_base_url": "http://127.0.0.1:11434/v1"},
+            inputs=[("predicted_answers", List[str])],
+            outputs=["custom_score"],
+            api="openai",
+            examples=[
+                {
+                    "inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"},
+                    "outputs": {"custom_score": 1},
+                },
+                {
+                    "inputs": {"predicted_answers": "Football is the most popular sport."},
+                    "outputs": {"custom_score": 0},
+                },
+            ],
+        )
+        assert component.generator.client.api_key == "test-api-key"
+        assert component.api_params == {
+            "generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42},
+            "api_base_url": "http://127.0.0.1:11434/v1",
+        }
+        assert component.api == "openai"
+        assert component.examples == [
+            {"inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
+            {"inputs": {"predicted_answers": "Football is the most popular sport."}, "outputs": {"custom_score": 0}},
+        ]
+        assert component.instructions == "test-instruction"
+        assert component.inputs == [("predicted_answers", List[str])]
+        assert component.outputs == ["custom_score"]
+
+    @pytest.mark.skipif(
+        not (os.environ.get("API_BASE_URL") and os.environ.get("MODEL_NAME")),
+        reason="Export env vars API_BASE_URL and MODEL_NAME containing the OpenAI API compatible server URL and the model name to run this test.",
+    )
+    @pytest.mark.integration
+    def test_run_with_base_url(self):
+        component = LLMEvaluator(
+            instructions="test-instruction",
+            api_key=Secret.from_token("test-api-key"),
+            api_params={"api_base_url": os.environ["API_BASE_URL"], "model": os.environ["MODEL_NAME"]},
+            inputs=[("predicted_answers", List[str])],
+            outputs=["custom_score"],
+            api="openai",
+            examples=[
+                {
+                    "inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"},
+                    "outputs": {"custom_score": 1},
+                },
+                {
+                    "inputs": {"predicted_answers": "Football is the most popular sport."},
+                    "outputs": {"custom_score": 0},
+                },
+            ],
+        )
+        component.run(predicted_answers=["Damn, this is straight outta hell!!!", "Football is the most popular sport."])
+        assert component.outputs == ["custom_score"]