Repository: deepset-ai/haystack
Pull request: feat: LLM-based evaluators return meta info from OpenAI
### Related Issues

- fixes [#7905](https://github.com/deepset-ai/haystack/issues/7905)

### Proposed Changes:

pass the `meta` entry from openai api to the result for all the LLM-based evaluators, containing the following fields:
- model
- prompt tokens
- answer tokens
- total tokens

### How did you test it?

manual veritifcation + run local tests + end2endtests

### Checklist

- I have read the [contributors guidelines](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md) and the [code of conduct](https://github.com/deepset-ai/haystack/blob/main/code_of_conduct.txt)
- I have updated the related issue with new insights and changes
- I added unit tests and updated the docstrings
- I've used one of the [conventional commit types](https://www.conventionalcommits.org/en/v1.0.0/) for my PR title: `fix:`, `feat:`, `build:`, `chore:`, `ci:`, `docs:`, `style:`, `refactor:`, `perf:`, `test:`.
- I documented my code
- I ran [pre-commit hooks](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md#installation) and fixed any issue


Problem statement:
LLM-based Evaluators should return `meta` information provided by OpenAIGenerator
**Is your feature request related to a problem? Please describe.**
To calculate the cost of the evaluation, we need a token count. Most generator components in Haystack provide that information in the `meta`  field.

**Describe the solution you'd like**
LLM-based evaluators should return token count information with the result. This would be handy for connecting evaluation pipelines to monitoring tools such as Langfuse

**Describe alternatives you've considered**
Leave it as it is

**Additional context**
N/A

Hints from issue discussion:

Commit message: fixing live tests

Patch to review:
diff --git a/haystack/components/evaluators/llm_evaluator.py b/haystack/components/evaluators/llm_evaluator.py
index 671ca57a58..6da3ccde2e 100644
--- a/haystack/components/evaluators/llm_evaluator.py
+++ b/haystack/components/evaluators/llm_evaluator.py
@@ -171,10 +171,12 @@ def run(self, **inputs) -> Dict[str, Any]:
         :param inputs:
             The input values to evaluate. The keys are the input names and the values are lists of input values.
         :returns:
-            A dictionary with a single `results` entry that contains a list of results.
+            A dictionary with a `results` entry that contains a list of results.
             Each result is a dictionary containing the keys as defined in the `outputs` parameter of the LLMEvaluator
             and the evaluation results as the values. If an exception occurs for a particular input value, the result
             will be `None` for that entry.
+            If the API is "openai" and the response contains a "meta" key, the metadata from OpenAI will be included
+            in the output dictionary, under the key "metadata".
         :raises ValueError:
             Only in the case that  `raise_on_failure` is set to True and the received inputs are not lists or have
             different lengths, or if the output is not a valid JSON or doesn't contain the expected keys.
@@ -187,6 +189,7 @@ def run(self, **inputs) -> Dict[str, Any]:
         list_of_input_names_to_values = [dict(zip(input_names, v)) for v in values]
 
         results: List[Optional[Dict[str, Any]]] = []
+        metadata = None
         errors = 0
         for input_names_to_values in tqdm(list_of_input_names_to_values, disable=not self.progress_bar):
             prompt = self.builder.run(**input_names_to_values)
@@ -208,11 +211,14 @@ def run(self, **inputs) -> Dict[str, Any]:
                 results.append(None)
                 errors += 1
 
+            if self.api == "openai" and "meta" in result:
+                metadata = result["meta"]
+
         if errors > 0:
             msg = f"LLM evaluator failed for {errors} out of {len(list_of_input_names_to_values)} inputs."
             warn(msg)
 
-        return {"results": results}
+        return {"results": results, "metadata": metadata}
 
     def prepare_template(self) -> str:
         """
diff --git a/releasenotes/notes/adding-metadata-info-from-OpenAI-f5309af5f59bb6a7.yaml b/releasenotes/notes/adding-metadata-info-from-OpenAI-f5309af5f59bb6a7.yaml
new file mode 100644
index 0000000000..b4b9dc0599
--- /dev/null
+++ b/releasenotes/notes/adding-metadata-info-from-OpenAI-f5309af5f59bb6a7.yaml
@@ -0,0 +1,5 @@
+---
+
+enhancements:
+  - |
+    When using "openai" for the LLM-based evaluators the metadata from OpenAI will be in the output dictionary, under the key "metadata".
diff --git a/test/components/evaluators/test_context_relevance_evaluator.py b/test/components/evaluators/test_context_relevance_evaluator.py
index 1214bd0cf1..5c9a9c007c 100644
--- a/test/components/evaluators/test_context_relevance_evaluator.py
+++ b/test/components/evaluators/test_context_relevance_evaluator.py
@@ -160,6 +160,7 @@ def generator_run(self, *args, **kwargs):
                 {"score": 1, "statement_scores": [1, 1], "statements": ["c", "d"]},
             ],
             "score": 0.75,
+            "metadata": None,
         }
 
     def test_run_no_statements_extracted(self, monkeypatch):
@@ -192,6 +193,7 @@ def generator_run(self, *args, **kwargs):
                 {"score": 0, "statement_scores": [], "statements": []},
             ],
             "score": 0.25,
+            "metadata": None,
         }
 
     def test_run_missing_parameters(self, monkeypatch):
@@ -256,6 +258,11 @@ def test_live_run(self):
         nested_required_fields = {"score", "statement_scores", "statements"}
         assert all(field in result["results"][0] for field in nested_required_fields)
 
+        assert "metadata" in result
+        assert "prompt_tokens" in result["metadata"][0]["usage"]
+        assert "completion_tokens" in result["metadata"][0]["usage"]
+        assert "total_tokens" in result["metadata"][0]["usage"]
+
     @pytest.mark.skipif(
         not os.environ.get("OPENAI_API_KEY", None),
         reason="Export an env var called OPENAI_API_KEY containing the OpenAI API key to run this test.",
diff --git a/test/components/evaluators/test_faithfulness_evaluator.py b/test/components/evaluators/test_faithfulness_evaluator.py
index c24df9458a..272aef389d 100644
--- a/test/components/evaluators/test_faithfulness_evaluator.py
+++ b/test/components/evaluators/test_faithfulness_evaluator.py
@@ -179,6 +179,7 @@ def generator_run(self, *args, **kwargs):
                 {"score": 1, "statement_scores": [1, 1], "statements": ["c", "d"]},
             ],
             "score": 0.75,
+            "metadata": None,
         }
 
     def test_run_no_statements_extracted(self, monkeypatch):
@@ -215,6 +216,7 @@ def generator_run(self, *args, **kwargs):
                 {"score": 0, "statement_scores": [], "statements": []},
             ],
             "score": 0.25,
+            "metadata": None,
         }
 
     def test_run_missing_parameters(self, monkeypatch):
@@ -282,3 +284,9 @@ def test_live_run(self):
         assert all(field in result for field in required_fields)
         nested_required_fields = {"score", "statement_scores", "statements"}
         assert all(field in result["results"][0] for field in nested_required_fields)
+
+        # assert that metadata is present in the result
+        assert "metadata" in result
+        assert "prompt_tokens" in result["metadata"][0]["usage"]
+        assert "completion_tokens" in result["metadata"][0]["usage"]
+        assert "total_tokens" in result["metadata"][0]["usage"]
diff --git a/test/components/evaluators/test_llm_evaluator.py b/test/components/evaluators/test_llm_evaluator.py
index d3ec4d29e4..b4bfdbf5b9 100644
--- a/test/components/evaluators/test_llm_evaluator.py
+++ b/test/components/evaluators/test_llm_evaluator.py
@@ -339,7 +339,7 @@ def generator_run(self, *args, **kwargs):
         monkeypatch.setattr("haystack.components.generators.openai.OpenAIGenerator.run", generator_run)
 
         results = component.run(questions=["What is the capital of Germany?"], predicted_answers=["Berlin"])
-        assert results == {"results": [{"score": 0.5}]}
+        assert results == {"results": [{"score": 0.5}], "metadata": None}
 
     def test_prepare_template(self, monkeypatch):
         monkeypatch.setenv("OPENAI_API_KEY", "test-api-key")


Code review comments:
diff_hunk: @@ -208,11 +211,14 @@ def run(self, **inputs) -> Dict[str, Any]:
                 results.append(None)
                 errors += 1
 
+            if self.api == "openai" and "meta" in result:
+                metadata = result["meta"]
+
         if errors > 0:
             msg = f"LLM evaluator failed for {errors} out of {len(list_of_input_names_to_values)} inputs."
             warn(msg)
 
-        return {"results": results}
+        return {"results": results, "metadata": metadata}
path: haystack/components/evaluators/llm_evaluator.py
text: @user1:
for consistency with the rest of the code base, I would call it `meta` instead of `metadata`.

@author:
👍🏽