Repository: deepset-ai/haystack

Issue:
Add support for `AzureOpenAI` in `LLMEvaluator`
**Is your feature request related to a problem? Please describe.**

`LLMEvaluator` currently only supports `OpenAI`, it would be nice if we could use it with the OpenAI models via Azure too.

**Describe the solution you'd like**

I'd like to use evaluators with Azure OpenAI (e.g. `ContextRelevanceEvaluator(api='azure-openai')`)

In addition, I propose to slightly change the design of `LLMEvaluator` to allow more flexibility.
Currently the params `api_key=Secret.from_env_var("OPENAI_API_KEY")` forces the user to provide an env var that is specific to OpenAI, and would not be used by other generators.

What about having something like:

```python
@component
class LLMEvaluator:
    def __init__(
        self,
        instructions: str,
        ...
        api: str = "openai",
        generator_kwargs: Dict[str, Any] = ..., # instead of api_key
    ):
        ...
        self.generator = OpenAIGenerator(**generator_kwargs)
```
?

This wouldn't force the user to provide to `LLMEvaluator` anything specific to the generator. It gives the flexibility to pass anything that the generator can take (e.g. api keys, api version, or `azure_deployment` in case of Azure) via the `generator_kwargs`.
At the same time, if the user doesn't pass anything, the generator would still look for its required env vars during instantiation.

I guess `api_key` needs to enter the deprecation cycle before being removed. Maybe we could just change to `api_key=Secret.from_env_var("OPENAI_API_KEY", strict=False)` until deprecated, so that that var will not be required for other generators.

**Describe alternatives you've considered**

Subclassing the `LLMEvaluator` (and all the child classes ) into a custom component

**Additional context**

Happy to hear your thoughts, also in case there are other better solutions I didn't consider. :)

I'm currently a bit busy with other things, but I may be able to raise PR with the proposal in the next days.

Discussion:
coveralls: ## Pull Request Test Coverage Report for [Build 9821988850](https://coveralls.io/builds/68495058)


### Details

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* **6** unchanged lines in **1** file lost coverage.
* Overall coverage increased (+**0.006%**) to **90.009%**

---


|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [components/evaluators/llm_evaluator.py](https://coveralls.io/builds/68495058/source?filename=components%2Fevaluators%2Fllm_evaluator.py#L208) | 6 | 94.96% |
<!-- | **Total:** | **6** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/68495058/badge)](https://coveralls.io/builds/68495058) |
| :-- | --: |
| Change from base [Build 9804009410](https://coveralls.io/builds/68478044): |  0.006% |
| Covered Lines: | 6775 |
| Relevant Lines: | 7527 |

---
##### 💛  - [Coveralls](https://coveralls.io)

julian-risch: @shadeMe Could you please take over the review for this PR? 
coveralls: ## Pull Request Test Coverage Report for [Build 9862971185](https://coveralls.io/builds/68548660)

### Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

- For more information on this, see <a target="_blank" href="https://docs.coveralls.io/build-types#tracking-coverage-changes-with-pull_request-builds">Tracking coverage changes with pull request builds</a>.
- To avoid this issue with future PRs, see these <a target="_blank" href="https://docs.coveralls.io/build-types#recommended-ci-configurations">Recommended CI Configurations</a>.
- For a quick fix, <a target="_blank" href="https://github.blog/changelog/2022-02-03-more-ways-to-keep-your-pull-request-branch-up-to-date/#update-your-pull-request-branch-by-rebasing">rebase this PR at GitHub</a>. Your next report should be accurate.

### Details

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* **64** unchanged lines in **9** files lost coverage.
* Overall coverage increased (+**0.04%**) to **90.046%**

---


|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [utils/filters.py](https://coveralls.io/builds/68548660/source?filename=utils%2Ffilters.py#L295) | 2 | 98.65% |
| [components/converters/pypdf.py](https://coveralls.io/builds/68548660/source?filename=components%2Fconverters%2Fpypdf.py#L154) | 3 | 90.32% |
| [components/generators/azure.py](https://coveralls.io/builds/68548660/source?filename=components%2Fgenerators%2Fazure.py#L115) | 3 | 92.5% |
| [components/generators/chat/azure.py](https://coveralls.io/builds/68548660/source?filename=components%2Fgenerators%2Fchat%2Fazure.py#L132) | 3 | 92.31% |
| [components/generators/hugging_face_local.py](https://coveralls.io/builds/68548660/source?filename=components%2Fgenerators%2Fhugging_face_local.py#L202) | 3 | 91.57% |
| [components/evaluators/llm_evaluator.py](https://coveralls.io/builds/68548660/source?filename=components%2Fevaluators%2Fllm_evaluator.py#L208) | 6 | 94.96% |
| [components/fetchers/link_content.py](https://coveralls.io/builds/68548660/source?filename=components%2Ffetchers%2Flink_content.py#L144) | 8 | 79.79% |
| [components/embedders/azure_text_embedder.py](https://coveralls.io/builds/68548660/source?filename=components%2Fembedders%2Fazure_text_embedder.py#L87) | 13 | 66.67% |
| [components/embedders/azure_document_embedder.py](https://coveralls.io/builds/68548660/source?filename=components%2Fembedders%2Fazure_document_embedder.py#L97) | 23 | 53.62% |
<!-- | **Total:** | **64** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/68548660/badge)](https://coveralls.io/builds/68548660) |
| :-- | --: |
| Change from base [Build 9804009410](https://coveralls.io/builds/68478044): |  0.04% |
| Covered Lines: | 6794 |
| Relevant Lines: | 7545 |

---
##### 💛  - [Coveralls](https://coveralls.io)

EdoardoAbatiTR: Thank you very much @lbux for looking into the issue!! :)

I think this currently will not fix #7946, because the AzureOpenAI client is a bit different (or am I missing something?)
I may have time tonight or tomorrow to submit a PR with my proposal, maybe it will be easier to see what I had in mind.
lbux: > Thank you very much @lbux for looking into the issue!! :)
> 
> I think this currently will not fix #7946, because the AzureOpenAI client is a bit different (or am I missing something?) I may have time tonight or tomorrow to submit a PR with my proposal, maybe it will be easier to see what I had in mind.

This is not going to 100% solve what you are asking for, but it will lay the foundation for adding like 2-3 lines of code as an else if to allow "azure" as the API (which you can make a PR for if this gets merged). api_params can handle any of the parameters defined by the Azure component and the generation_kwargs: https://docs.haystack.deepset.ai/reference/generators-api#:~:text=Module%20azure-,AzureOpenAIGenerator,-A%20Generator%20component

Patch:
diff --git a/haystack/components/evaluators/context_relevance.py b/haystack/components/evaluators/context_relevance.py
index b3dbae58..d6fe3adf 100644
--- a/haystack/components/evaluators/context_relevance.py
+++ b/haystack/components/evaluators/context_relevance.py
@@ -93,6 +93,7 @@ class ContextRelevanceEvaluator(LLMEvaluator):
         progress_bar: bool = True,
         api: str = "openai",
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        api_params: Optional[Dict[str, Any]] = None,
         raise_on_failure: bool = True,
     ):
         """
@@ -121,6 +122,8 @@ class ContextRelevanceEvaluator(LLMEvaluator):
             Supported APIs: "openai".
         :param api_key:
             The API key.
+        :param api_params:
+            Parameters for an OpenAI API compatible completions call.
         :param raise_on_failure:
             Whether to raise an exception if the API call fails.
 
@@ -137,6 +140,7 @@ class ContextRelevanceEvaluator(LLMEvaluator):
         self.examples = examples or _DEFAULT_EXAMPLES
         self.api = api
         self.api_key = api_key
+        self.api_params = api_params or {}
 
         warnings.warn(
             "The output of the ContextRelevanceEvaluator will change in Haystack 2.4.0. "
@@ -152,6 +156,7 @@ class ContextRelevanceEvaluator(LLMEvaluator):
             examples=self.examples,
             api=self.api,
             api_key=self.api_key,
+            api_params=self.api_params,
             raise_on_failure=raise_on_failure,
             progress_bar=progress_bar,
         )
@@ -202,6 +207,7 @@ class ContextRelevanceEvaluator(LLMEvaluator):
             api_key=self.api_key.to_dict() if self.api_key else None,
             examples=self.examples,
             progress_bar=self.progress_bar,
+            api_params=self.api_params,
             raise_on_failure=self.raise_on_failure,
         )
 
diff --git a/haystack/components/evaluators/faithfulness.py b/haystack/components/evaluators/faithfulness.py
index 25b38673..5cdfceaf 100644
--- a/haystack/components/evaluators/faithfulness.py
+++ b/haystack/components/evaluators/faithfulness.py
@@ -88,6 +88,7 @@ class FaithfulnessEvaluator(LLMEvaluator):
         progress_bar: bool = True,
         api: str = "openai",
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        api_params: Optional[Dict[str, Any]] = None,
         raise_on_failure: bool = True,
     ):
         """
@@ -117,6 +118,8 @@ class FaithfulnessEvaluator(LLMEvaluator):
             Supported APIs: "openai".
         :param api_key:
             The API key.
+        :param api_params:
+            Parameters for an OpenAI API compatible completions call.
         :param raise_on_failure:
             Whether to raise an exception if the API call fails.
 
@@ -133,6 +136,7 @@ class FaithfulnessEvaluator(LLMEvaluator):
         self.examples = examples or _DEFAULT_EXAMPLES
         self.api = api
         self.api_key = api_key
+        self.api_params = api_params or {}
 
         super(FaithfulnessEvaluator, self).__init__(
             instructions=self.instructions,
@@ -141,6 +145,7 @@ class FaithfulnessEvaluator(LLMEvaluator):
             examples=self.examples,
             api=self.api,
             api_key=self.api_key,
+            api_params=self.api_params,
             raise_on_failure=raise_on_failure,
             progress_bar=progress_bar,
         )
@@ -193,6 +198,7 @@ class FaithfulnessEvaluator(LLMEvaluator):
             self,
             api=self.api,
             api_key=self.api_key.to_dict() if self.api_key else None,
+            api_params=self.api_params,
             examples=self.examples,
             progress_bar=self.progress_bar,
             raise_on_failure=self.raise_on_failure,
diff --git a/haystack/components/evaluators/llm_evaluator.py b/haystack/components/evaluators/llm_evaluator.py
index 22885f87..511bc1cd 100644
--- a/haystack/components/evaluators/llm_evaluator.py
+++ b/haystack/components/evaluators/llm_evaluator.py
@@ -58,6 +58,7 @@ class LLMEvaluator:
         raise_on_failure: bool = True,
         api: str = "openai",
         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+        api_params: Optional[Dict[str, Any]] = None,
     ):
         """
         Creates an instance of LLMEvaluator.
@@ -84,6 +85,8 @@ class LLMEvaluator:
             Supported APIs: "openai".
         :param api_key:
             The API key.
+        :param api_params:
+            Parameters for an OpenAI API compatible completions call.
 
         """
         self.validate_init_parameters(inputs, outputs, examples)
@@ -94,12 +97,16 @@ class LLMEvaluator:
         self.examples = examples
         self.api = api
         self.api_key = api_key
+        self.api_params = api_params or {}
         self.progress_bar = progress_bar
 
+        default_generation_kwargs = {"response_format": {"type": "json_object"}, "seed": 42}
+        user_generation_kwargs = self.api_params.get("generation_kwargs", {})
+        merged_generation_kwargs = {**default_generation_kwargs, **user_generation_kwargs}
+        self.api_params["generation_kwargs"] = merged_generation_kwargs
+
         if api == "openai":
-            self.generator = OpenAIGenerator(
-                api_key=api_key, generation_kwargs={"response_format": {"type": "json_object"}, "seed": 42}
-            )
+            self.generator = OpenAIGenerator(api_key=api_key, **self.api_params)
         else:
             raise ValueError(f"Unsupported API: {api}")
 
@@ -282,6 +289,7 @@ class LLMEvaluator:
             examples=self.examples,
             api=self.api,
             api_key=self.api_key.to_dict(),
+            api_params=self.api_params,
             progress_bar=self.progress_bar,
         )
 
diff --git a/releasenotes/notes/allow-parameters-for-openai-generator-8ce86ba1bac80817.yaml b/releasenotes/notes/allow-parameters-for-openai-generator-8ce86ba1bac80817.yaml
new file mode 100644
index 00000000..0c5e2b93
--- /dev/null
+++ b/releasenotes/notes/allow-parameters-for-openai-generator-8ce86ba1bac80817.yaml
@@ -0,0 +1,6 @@
+---
+features:
+  - |
+    LLM based evaluators can pass in supported OpenAIGenerator parameters via api_params.
+    This allows for custom generation_kwargs, changing the api_base_url (for local evaluation),
+    and all other supported parameters as described in the OpenAIGenerator docs.


Test patch:
diff --git a/test/components/evaluators/test_context_relevance_evaluator.py b/test/components/evaluators/test_context_relevance_evaluator.py
index 1a7daa0b..89f5b5ae 100644
--- a/test/components/evaluators/test_context_relevance_evaluator.py
+++ b/test/components/evaluators/test_context_relevance_evaluator.py
@@ -100,6 +100,7 @@ class TestContextRelevanceEvaluator:
             "init_parameters": {
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": True, "type": "env_var"},
                 "api": "openai",
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "examples": [{"inputs": {"questions": "What is football?"}, "outputs": {"score": 0}}],
                 "progress_bar": False,
                 "raise_on_failure": False,
diff --git a/test/components/evaluators/test_faithfulness_evaluator.py b/test/components/evaluators/test_faithfulness_evaluator.py
index 7616a884..8164ad9c 100644
--- a/test/components/evaluators/test_faithfulness_evaluator.py
+++ b/test/components/evaluators/test_faithfulness_evaluator.py
@@ -109,6 +109,7 @@ class TestFaithfulnessEvaluator:
             "init_parameters": {
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": True, "type": "env_var"},
                 "api": "openai",
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "examples": [
                     {"inputs": {"predicted_answers": "Football is the most popular sport."}, "outputs": {"score": 0}}
                 ],
diff --git a/test/components/evaluators/test_llm_evaluator.py b/test/components/evaluators/test_llm_evaluator.py
index edab8734..21d5fd02 100644
--- a/test/components/evaluators/test_llm_evaluator.py
+++ b/test/components/evaluators/test_llm_evaluator.py
@@ -1,6 +1,7 @@
 # SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
 #
 # SPDX-License-Identifier: Apache-2.0
+import os
 from typing import List
 
 import pytest
@@ -23,6 +24,7 @@ class TestLLMEvaluator:
         )
         assert component.api == "openai"
         assert component.generator.client.api_key == "test-api-key"
+        assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}}
         assert component.instructions == "test-instruction"
         assert component.inputs == [("predicted_answers", List[str])]
         assert component.outputs == ["score"]
@@ -47,6 +49,7 @@ class TestLLMEvaluator:
         component = LLMEvaluator(
             instructions="test-instruction",
             api_key=Secret.from_token("test-api-key"),
+            api_params={"generation_kwargs": {"seed": 43}},
             inputs=[("predicted_answers", List[str])],
             outputs=["custom_score"],
             api="openai",
@@ -62,6 +65,7 @@ class TestLLMEvaluator:
             ],
         )
         assert component.generator.client.api_key == "test-api-key"
+        assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 43}}
         assert component.api == "openai"
         assert component.examples == [
             {"inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
@@ -202,6 +206,7 @@ class TestLLMEvaluator:
         assert data == {
             "type": "haystack.components.evaluators.llm_evaluator.LLMEvaluator",
             "init_parameters": {
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
                 "api": "openai",
                 "instructions": "test-instruction",
@@ -220,6 +225,7 @@ class TestLLMEvaluator:
         data = {
             "type": "haystack.components.evaluators.llm_evaluator.LLMEvaluator",
             "init_parameters": {
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "api_key": {"env_vars": ["OPENAI_API_KEY"], "strict": True, "type": "env_var"},
                 "api": "openai",
                 "instructions": "test-instruction",
@@ -233,6 +239,7 @@ class TestLLMEvaluator:
         component = LLMEvaluator.from_dict(data)
         assert component.api == "openai"
         assert component.generator.client.api_key == "test-api-key"
+        assert component.api_params == {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}}
         assert component.instructions == "test-instruction"
         assert component.inputs == [("predicted_answers", List[str])]
         assert component.outputs == ["score"]
@@ -263,6 +270,7 @@ class TestLLMEvaluator:
         assert data == {
             "type": "haystack.components.evaluators.llm_evaluator.LLMEvaluator",
             "init_parameters": {
+                "api_params": {"generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42}},
                 "api_key": {"env_vars": ["ENV_VAR"], "strict": True, "type": "env_var"},
                 "api": "openai",
                 "instructions": "test-instruction",
@@ -442,3 +450,63 @@ class TestLLMEvaluator:
                     {"inputs": {"predicted_answers": "Football is the most popular sport."}, "outputs": {"score": 0}}
                 ],
             )
+
+    def test_init_with_base_url(self):
+        component = LLMEvaluator(
+            instructions="test-instruction",
+            api_key=Secret.from_token("test-api-key"),
+            api_params={"api_base_url": "http://127.0.0.1:11434/v1"},
+            inputs=[("predicted_answers", List[str])],
+            outputs=["custom_score"],
+            api="openai",
+            examples=[
+                {
+                    "inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"},
+                    "outputs": {"custom_score": 1},
+                },
+                {
+                    "inputs": {"predicted_answers": "Football is the most popular sport."},
+                    "outputs": {"custom_score": 0},
+                },
+            ],
+        )
+        assert component.generator.client.api_key == "test-api-key"
+        assert component.api_params == {
+            "generation_kwargs": {"response_format": {"type": "json_object"}, "seed": 42},
+            "api_base_url": "http://127.0.0.1:11434/v1",
+        }
+        assert component.api == "openai"
+        assert component.examples == [
+            {"inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"}, "outputs": {"custom_score": 1}},
+            {"inputs": {"predicted_answers": "Football is the most popular sport."}, "outputs": {"custom_score": 0}},
+        ]
+        assert component.instructions == "test-instruction"
+        assert component.inputs == [("predicted_answers", List[str])]
+        assert component.outputs == ["custom_score"]
+
+    @pytest.mark.skipif(
+        not (os.environ.get("API_BASE_URL") and os.environ.get("MODEL_NAME")),
+        reason="Export env vars API_BASE_URL and MODEL_NAME containing the OpenAI API compatible server URL and the model name to run this test.",
+    )
+    @pytest.mark.integration
+    def test_run_with_base_url(self):
+        component = LLMEvaluator(
+            instructions="test-instruction",
+            api_key=Secret.from_token("test-api-key"),
+            api_params={"api_base_url": os.environ["API_BASE_URL"], "model": os.environ["MODEL_NAME"]},
+            inputs=[("predicted_answers", List[str])],
+            outputs=["custom_score"],
+            api="openai",
+            examples=[
+                {
+                    "inputs": {"predicted_answers": "Damn, this is straight outta hell!!!"},
+                    "outputs": {"custom_score": 1},
+                },
+                {
+                    "inputs": {"predicted_answers": "Football is the most popular sport."},
+                    "outputs": {"custom_score": 0},
+                },
+            ],
+        )
+        component.run(predicted_answers=["Damn, this is straight outta hell!!!", "Football is the most popular sport."])
+        assert component.outputs == ["custom_score"]