Repository: stanfordnlp/dspy
Pull request: Change the output interface of evaluate
In this PR, we change the output interface of `Evaluate.__call__`.
Instead of returning either score, (score, outputs), (score, scores, outputs) based on arguments, it will always return EvaluationResult containing the following fields:

- score: A float percentage score (e.g., 67.30) representing overall performance
- results: a list of (example, prediction, score) tuples for each example in devset

Since this is a breaking change, this should be released in the next minor release rather than a patch release.

Resolve https://github.com/mlflow/mlflow/issues/15476

Problem statement:
[FR] DSPy : Log examples evaluation scores
### Willingness to contribute

Yes. I can contribute this feature independently.

### Proposal Summary

The current implementation only logs the overall evaluation score, while individual scores for each example are not captured or recorded.

### Motivation

> #### What is the use case for this feature?
This enhancement enables the inspection of individual scores for each example, providing greater granularity and insight into the evaluation process.
> #### Why is this use case valuable to support for MLflow users in general?
This feature is particularly beneficial for DSPy users, as it allows them to analyze the performance of individual examples, facilitating more targeted debugging, optimization.
> #### Why is this use case valuable to support for your project(s) or organization?
Since we're conducting strict evaluations, it's crucial to inspect why each generated output may not meet the desired quality. This capability allows us to identify specific issues at the individual example level, enabling more precise debugging and targeted improvements in the model's performance.
> #### Why is it currently difficult to achieve this use case?
I couldn’t find any existing method to extract the score for each individual example.



### Details

_No response_

### What component(s) does this bug affect?

- [x] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [x] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [x] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [x] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations

Hints from issue discussion:
@TomeHirata I thought the autologging records the evaluation result table that includes scores for individual samples 
https://github.com/mlflow/mlflow/blob/a68ee413624f5067ec33fa05ba1785fab613fa24/mlflow/dspy/callback.py#L271

@Nasreddine Could you share an example code where the row-level result is not available? There might be an edge case that the result table is not logged.
Hi, @Nasreddine. Thank you for the report. The individual example level evaluation score will be available once https://github.com/stanfordnlp/dspy/pull/8003 is released. Please wait for a while, it will be available in the next minor release of DSPy.

Commit message: introduce EvaluationResult class

Patch to review:
diff --git a/docs/docs/tutorials/agents/index.ipynb b/docs/docs/tutorials/agents/index.ipynb
index 8b5521fb66..ef7cc56c45 100644
--- a/docs/docs/tutorials/agents/index.ipynb
+++ b/docs/docs/tutorials/agents/index.ipynb
@@ -500,23 +500,20 @@
     "        metric=top5_recall,\n",
     "        num_threads=16,\n",
     "        display_progress=True,\n",
-    "        # To record the outputs and detailed scores to MLflow\n",
-    "        return_all_scores=True,\n",
-    "        return_outputs=True,\n",
     "    )\n",
     "\n",
     "    # Evaluate the program as usual\n",
-    "    aggregated_score, outputs, all_scores = evaluate(cot)\n",
+    "    result = evaluate(cot)\n",
     "\n",
     "    # Log the aggregated score\n",
-    "    mlflow.log_metric(\"top5_recall\", aggregated_score)\n",
+    "    mlflow.log_metric(\"top5_recall\", result.score)\n",
     "    # Log the detailed evaluation results as a table\n",
     "    mlflow.log_table(\n",
     "        {\n",
     "            \"Claim\": [example.claim for example in eval_set],\n",
     "            \"Expected Titles\": [example.titles for example in eval_set],\n",
-    "            \"Predicted Titles\": outputs,\n",
-    "            \"Top 5 Recall\": all_scores,\n",
+    "            \"Predicted Titles\": [output[1] for output in result.results],\n",
+    "            \"Top 5 Recall\": [output[2] for output in result.results],\n",
     "        },\n",
     "        artifact_file=\"eval_results.json\",\n",
     "    )\n",
diff --git a/docs/docs/tutorials/classification_finetuning/index.ipynb b/docs/docs/tutorials/classification_finetuning/index.ipynb
index 4864183b1c..4baac6df63 100644
--- a/docs/docs/tutorials/classification_finetuning/index.ipynb
+++ b/docs/docs/tutorials/classification_finetuning/index.ipynb
@@ -568,23 +568,20 @@
     "        metric=extraction_correctness_metric,\n",
     "        num_threads=16,\n",
     "        display_progress=True,\n",
-    "        # To record the outputs and detailed scores to MLflow\n",
-    "        return_all_scores=True,\n",
-    "        return_outputs=True,\n",
     "    )\n",
     "\n",
     "    # Evaluate the program as usual\n",
-    "    aggregated_score, outputs, all_scores = evaluate_correctness(people_extractor)\n",
+    "    result = evaluate_correctness(people_extractor)\n",
     "\n",
     "    # Log the aggregated score\n",
-    "    mlflow.log_metric(\"exact_match\", aggregated_score)\n",
+    "    mlflow.log_metric(\"exact_match\", result.score)\n",
     "    # Log the detailed evaluation results as a table\n",
     "    mlflow.log_table(\n",
     "        {\n",
     "            \"Text\": [example.text for example in devset],\n",
     "            \"Expected\": [example.example_label for example in devset],\n",
-    "            \"Predicted\": outputs,\n",
-    "            \"Exact match\": all_scores,\n",
+    "            \"Predicted\": [output[1] for output in result.results],\n",
+    "            \"Exact match\": [output[2] for output in result.results],\n",
     "        },\n",
     "        artifact_file=\"eval_results.json\",\n",
     "    )\n",
diff --git a/docs/docs/tutorials/entity_extraction/index.ipynb b/docs/docs/tutorials/entity_extraction/index.ipynb
index 10a96a5d59..95798912ee 100644
--- a/docs/docs/tutorials/entity_extraction/index.ipynb
+++ b/docs/docs/tutorials/entity_extraction/index.ipynb
@@ -514,23 +514,20 @@
         "        metric=extraction_correctness_metric,\n",
         "        num_threads=24,\n",
         "        display_progress=True,\n",
-        "        # To record the outputs and detailed scores to MLflow\n",
-        "        return_all_scores=True,\n",
-        "        return_outputs=True,\n",
         "    )\n",
         "\n",
         "    # Evaluate the program as usual\n",
-        "    aggregated_score, outputs, all_scores = evaluate_correctness(people_extractor)\n",
+        "    result = evaluate_correctness(people_extractor)\n",
         "\n",
         "    # Log the aggregated score\n",
-        "    mlflow.log_metric(\"exact_match\", aggregated_score)\n",
+        "    mlflow.log_metric(\"exact_match\", result.score)\n",
         "    # Log the detailed evaluation results as a table\n",
         "    mlflow.log_table(\n",
         "        {\n",
         "            \"Tokens\": [example.tokens for example in test_set],\n",
         "            \"Expected\": [example.expected_extracted_people for example in test_set],\n",
-        "            \"Predicted\": outputs,\n",
-        "            \"Exact match\": all_scores,\n",
+        "            \"Predicted\": [output[1] for output in result.results],\n",
+        "            \"Exact match\": [output[2] for output in result.results],\n",
         "        },\n",
         "        artifact_file=\"eval_results.json\",\n",
         "    )\n",
diff --git a/docs/docs/tutorials/math/index.ipynb b/docs/docs/tutorials/math/index.ipynb
index eb64396302..651511ef67 100644
--- a/docs/docs/tutorials/math/index.ipynb
+++ b/docs/docs/tutorials/math/index.ipynb
@@ -369,21 +369,21 @@
     "\n",
     "# Start an MLflow Run to record the evaluation\n",
     "with mlflow.start_run(run_name=\"math_evaluation\"):\n",
-    "    kwargs = dict(num_threads=THREADS, display_progress=True, return_all_scores=True, return_outputs=True)\n",
+    "    kwargs = dict(num_threads=THREADS, display_progress=True)\n",
     "    evaluate = dspy.Evaluate(devset=dataset.dev, metric=dataset.metric, **kwargs)\n",
     "\n",
     "    # Evaluate the program as usual\n",
-    "    aggregated_score, outputs, all_scores = evaluate(module)\n",
+    "    result = evaluate(module)\n",
     "\n",
     "    # Log the aggregated score\n",
-    "    mlflow.log_metric(\"correctness\", aggregated_score)\n",
+    "    mlflow.log_metric(\"correctness\", result.score)\n",
     "    # Log the detailed evaluation results as a table\n",
     "    mlflow.log_table(\n",
     "        {\n",
     "            \"Question\": [example.question for example in dataset.dev],\n",
     "            \"Gold Answer\": [example.answer for example in dataset.dev],\n",
-    "            \"Predicted Answer\": outputs,\n",
-    "            \"Correctness\": all_scores,\n",
+    "            \"Predicted Answer\": [output[1] for output in result.results],\n",
+    "            \"Correctness\": [output[2] for output in result.results],\n",
     "        },\n",
     "        artifact_file=\"eval_results.json\",\n",
     "    )\n",
diff --git a/docs/docs/tutorials/multihop_search/index.ipynb b/docs/docs/tutorials/multihop_search/index.ipynb
index 1c5998b424..f83f3faf25 100644
--- a/docs/docs/tutorials/multihop_search/index.ipynb
+++ b/docs/docs/tutorials/multihop_search/index.ipynb
@@ -534,23 +534,20 @@
     "        metric=top5_recall,\n",
     "        num_threads=16,\n",
     "        display_progress=True,\n",
-    "        # To record the outputs and detailed scores to MLflow\n",
-    "        return_all_scores=True,\n",
-    "        return_outputs=True,\n",
     "    )\n",
     "\n",
     "    # Evaluate the program as usual\n",
-    "    aggregated_score, outputs, all_scores = evaluate(Hop())\n",
+    "    result = evaluate(Hop())\n",
     "\n",
     "    # Log the aggregated score\n",
-    "    mlflow.log_metric(\"top5_recall\", aggregated_score)\n",
+    "    mlflow.log_metric(\"top5_recall\", result.score)\n",
     "    # Log the detailed evaluation results as a table\n",
     "    mlflow.log_table(\n",
     "        {\n",
     "            \"Claim\": [example.claim for example in eval_set],\n",
     "            \"Expected Titles\": [example.titles for example in eval_set],\n",
-    "            \"Predicted Titles\": outputs,\n",
-    "            \"Top 5 Recall\": all_scores,\n",
+    "            \"Predicted Titles\": [output[1] for output in result.results],\n",
+    "            \"Top 5 Recall\": [output[2] for output in result.results],\n",
     "        },\n",
     "        artifact_file=\"eval_results.json\",\n",
     "    )\n",
diff --git a/docs/docs/tutorials/rag/index.ipynb b/docs/docs/tutorials/rag/index.ipynb
index 79c9374f87..a650707c4f 100644
--- a/docs/docs/tutorials/rag/index.ipynb
+++ b/docs/docs/tutorials/rag/index.ipynb
@@ -731,24 +731,21 @@
     "        metric=metric,\n",
     "        num_threads=24,\n",
     "        display_progress=True,\n",
-    "        # To record the outputs and detailed scores to MLflow\n",
-    "        return_all_scores=True,\n",
-    "        return_outputs=True,\n",
     "    )\n",
     "\n",
     "    # Evaluate the program as usual\n",
-    "    aggregated_score, outputs, all_scores = evaluate(cot)\n",
+    "    result = evaluate(cot)\n",
     "\n",
     "\n",
     "    # Log the aggregated score\n",
-    "    mlflow.log_metric(\"semantic_f1_score\", aggregated_score)\n",
+    "    mlflow.log_metric(\"semantic_f1_score\", result.score)\n",
     "    # Log the detailed evaluation results as a table\n",
     "    mlflow.log_table(\n",
     "        {\n",
     "            \"Question\": [example.question for example in eval_set],\n",
     "            \"Gold Response\": [example.response for example in eval_set],\n",
-    "            \"Predicted Response\": outputs,\n",
-    "            \"Semantic F1 Score\": all_scores,\n",
+    "            \"Predicted Response\": [output[1] for output in result.results],\n",
+    "            \"Semantic F1 Score\": [output[2] for output in result.results],\n",
     "        },\n",
     "        artifact_file=\"eval_results.json\",\n",
     "    )\n",
@@ -1471,6 +1468,11 @@
     "\n",
     "The first step is to look at your system outputs, which will allow you to identify the sources of lower performance if any. While doing all of this, make sure you continue to refine your metric, e.g. by optimizing against your judgments, and to collect more (or more realistic) data, e.g. from related domains or from putting a demo of your system in front of users."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
   }
  ],
  "metadata": {
diff --git a/dspy/evaluate/evaluate.py b/dspy/evaluate/evaluate.py
index 484f9747c9..2c9395a996 100644
--- a/dspy/evaluate/evaluate.py
+++ b/dspy/evaluate/evaluate.py
@@ -41,6 +41,18 @@ def HTML(x: str) -> str:  # noqa: N802
 logger = logging.getLogger(__name__)
 
 
+class EvaluationResult(dspy.Prediction):
+    """
+    A class that represents the result of an evaluation.
+    It is a subclass of `dspy.Prediction` that contains the following fields
+
+    - score: An float value (e.g., 67.30) representing the overall performance
+    - results: a list of (example, prediction, score) tuples for each example in devset
+    """
+    def __repr__(self):
+        return f"EvaluationResult(score={self.score}, results=`{len(self.results)} results`)"
+
+
 class Evaluate:
     """DSPy Evaluate class.
 
@@ -57,8 +69,6 @@ def __init__(
         display_progress: bool = False,
         display_table: Union[bool, int] = False,
         max_errors: Optional[int] = None,
-        return_all_scores: bool = False,
-        return_outputs: bool = False,
         provide_traceback: Optional[bool] = None,
         failure_score: float = 0.0,
         **kwargs,
@@ -73,8 +83,6 @@ def __init__(
                 If a number is passed, the evaluation results will be truncated to that number before displayed.
             max_errors (Optional[int]): The maximum number of errors to allow before
                 stopping evaluation. If ``None``, inherits from ``dspy.settings.max_errors``.
-            return_all_scores (bool): Whether to return scores for every data record in `devset`.
-            return_outputs (bool): Whether to return the dspy program's outputs for every data in `devset`.
             provide_traceback (Optional[bool]): Whether to provide traceback information during evaluation.
             failure_score (float): The default score to use if evaluation fails due to an exception.
         """
@@ -84,8 +92,6 @@ def __init__(
         self.display_progress = display_progress
         self.display_table = display_table
         self.max_errors = max_errors
-        self.return_all_scores = return_all_scores
-        self.return_outputs = return_outputs
         self.provide_traceback = provide_traceback
         self.failure_score = failure_score
 
@@ -98,10 +104,8 @@ def __call__(
         num_threads: Optional[int] = None,
         display_progress: Optional[bool] = None,
         display_table: Optional[Union[bool, int]] = None,
-        return_all_scores: Optional[bool] = None,
-        return_outputs: Optional[bool] = None,
         callback_metadata: Optional[dict[str, Any]] = None,
-    ):
+    ) -> EvaluationResult:
         """
         Args:
             program (dspy.Module): The DSPy program to evaluate.
@@ -113,36 +117,20 @@ def __call__(
                 `self.display_progress`.
             display_table (Union[bool, int]): Whether to display the evaluation results in a table. if not provided, use
                 `self.display_table`. If a number is passed, the evaluation results will be truncated to that number before displayed.
-            return_all_scores (bool): Whether to return scores for every data record in `devset`. if not provided,
-                use `self.return_all_scores`.
-            return_outputs (bool): Whether to return the dspy program's outputs for every data in `devset`. if not
-                provided, use `self.return_outputs`.
             callback_metadata (dict): Metadata to be used for evaluate callback handlers.
 
         Returns:
-            The evaluation results are returned in different formats based on the flags:
-
-            - Base return: A float percentage score (e.g., 67.30) representing overall performance
-
-            - With `return_all_scores=True`:
-                Returns (overall_score, individual_scores) where individual_scores is a list of
-                float scores for each example in devset
-
-            - With `return_outputs=True`:
-                Returns (overall_score, result_triples) where result_triples is a list of
-                (example, prediction, score) tuples for each example in devset
-
-            - With both flags=True:
-                Returns (overall_score, result_triples, individual_scores)
-
+            The evaluation results are returned as a dspy.EvaluationResult object containing the following attributes:
+            
+            - score: A float percentage score (e.g., 67.30) representing overall performance
+            
+            - results: a list of (example, prediction, score) tuples for each example in devset
         """
         metric = metric if metric is not None else self.metric
         devset = devset if devset is not None else self.devset
         num_threads = num_threads if num_threads is not None else self.num_threads
         display_progress = display_progress if display_progress is not None else self.display_progress
         display_table = display_table if display_table is not None else self.display_table
-        return_all_scores = return_all_scores if return_all_scores is not None else self.return_all_scores
-        return_outputs = return_outputs if return_outputs is not None else self.return_outputs
 
         if callback_metadata:
             logger.debug(f"Evaluate is called with callback metadata: {callback_metadata}")
@@ -190,14 +178,11 @@ def process_item(example):
 
             self._display_result_table(result_df, display_table, metric_name)
 
-        if return_all_scores and return_outputs:
-            return round(100 * ncorrect / ntotal, 2), results, [score for *_, score in results]
-        if return_all_scores:
-            return round(100 * ncorrect / ntotal, 2), [score for *_, score in results]
-        if return_outputs:
-            return round(100 * ncorrect / ntotal, 2), results
+        return EvaluationResult(
+            score=round(100 * ncorrect / ntotal, 2),
+            results=results,
+        )
 
-        return round(100 * ncorrect / ntotal, 2)
 
     def _construct_result_table(
         self, results: list[Tuple[dspy.Example, dspy.Example, Any]], metric_name: str
diff --git a/dspy/primitives/prediction.py b/dspy/primitives/prediction.py
index 670b816b28..df01cef671 100644
--- a/dspy/primitives/prediction.py
+++ b/dspy/primitives/prediction.py
@@ -98,6 +98,15 @@ def __ge__(self, other):
             return self.__float__() >= float(other)
         raise TypeError(f"Unsupported type for comparison: {type(other)}")
 
+    def __eq__(self, other):
+        if isinstance(other, (float, int)):
+            return self.__float__() == other
+        elif isinstance(other, Prediction):
+            return self.__float__() == float(other)
+        else:
+            # we should return False when Prediction is compared with other types
+            return False
+
     @property
     def completions(self):
         return self._completions
diff --git a/dspy/teleprompt/bootstrap_finetune.py b/dspy/teleprompt/bootstrap_finetune.py
index 83b3808a67..44e8efc7bc 100644
--- a/dspy/teleprompt/bootstrap_finetune.py
+++ b/dspy/teleprompt/bootstrap_finetune.py
@@ -290,10 +290,10 @@ def wrapped_program(**kwargs):
 
                 return failed_pred, trace
 
-    _, outputs = evaluator(wrapped_program, metric=wrapped_metric)
+    results = evaluator(wrapped_program, metric=wrapped_metric).results
 
     data = []
-    for example_ind, (example, prediction, score) in enumerate(outputs):
+    for example_ind, (example, prediction, score) in enumerate(results):
         try:
             prediction, trace = prediction
         except ValueError as ve:
diff --git a/dspy/teleprompt/copro_optimizer.py b/dspy/teleprompt/copro_optimizer.py
index 6a3760d8db..65d09dc6f3 100644
--- a/dspy/teleprompt/copro_optimizer.py
+++ b/dspy/teleprompt/copro_optimizer.py
@@ -225,7 +225,7 @@ def compile(self, student, *, trainset, eval_kwargs):
                         f"At Depth {d+1}/{self.depth}, Evaluating Prompt Candidate #{c_i+1}/{len(candidates_)} for "
                         f"Predictor {p_i+1} of {len(module.predictors())}.",
                     )
-                    score = evaluate(module_clone, devset=trainset, **eval_kwargs)
+                    score = evaluate(module_clone, devset=trainset, **eval_kwargs).score
                     if self.prompt_model:
                         logger.debug(f"prompt_model.inspect_history(n=1) {self.prompt_model.inspect_history(n=1)}")
                     total_calls += 1
diff --git a/dspy/teleprompt/infer_rules.py b/dspy/teleprompt/infer_rules.py
index 70611962de..21ed154da4 100644
--- a/dspy/teleprompt/infer_rules.py
+++ b/dspy/teleprompt/infer_rules.py
@@ -119,9 +119,8 @@ def evaluate_program(self, program, dataset):
             max_errors=effective_max_errors,
             display_table=False,
             display_progress=True,
-            return_all_scores=True,
         )
-        score, _ = evaluate(program, metric=self.metric)
+        score = evaluate(program, metric=self.metric).score
         return score
 
 
diff --git a/dspy/teleprompt/mipro_optimizer_v2.py b/dspy/teleprompt/mipro_optimizer_v2.py
index f4013d48cd..fe1225f79e 100644
--- a/dspy/teleprompt/mipro_optimizer_v2.py
+++ b/dspy/teleprompt/mipro_optimizer_v2.py
@@ -527,9 +527,7 @@ def _optimize_prompt_parameters(
         )
         logger.info(f"== Trial {1} / {adjusted_num_trials} - Full Evaluation of Default Program ==")
 
-        default_score, _ = eval_candidate_program(
-            len(valset), valset, program, evaluate, self.rng, return_all_scores=True
-        )
+        default_score = eval_candidate_program(len(valset), valset, program, evaluate, self.rng).score
         logger.info(f"Default program score: {default_score}\n")
 
         trial_logs = {}
@@ -579,7 +577,7 @@ def objective(trial):
 
             # Evaluate the candidate program (on minibatch if minibatch=True)
             batch_size = minibatch_size if minibatch else len(valset)
-            score = eval_candidate_program(batch_size, valset, candidate_program, evaluate, self.rng)
+            score = eval_candidate_program(batch_size, valset, candidate_program, evaluate, self.rng).score
             total_eval_calls += batch_size
 
             # Update best score and program
@@ -816,7 +814,7 @@ def _perform_full_evaluation(
             param_score_dict, fully_evaled_param_combos
         )
         logger.info(f"Doing full eval on next top averaging program (Avg Score: {mean_score}) from minibatch trials...")
-        full_eval_score = eval_candidate_program(len(valset), valset, highest_mean_program, evaluate, self.rng)
+        full_eval_score = eval_candidate_program(len(valset), valset, highest_mean_program, evaluate, self.rng).score
         score_data.append({"score": full_eval_score, "program": highest_mean_program, "full_eval": True})
 
         # Log full eval as a trial so that optuna can learn from the new results
diff --git a/dspy/teleprompt/random_search.py b/dspy/teleprompt/random_search.py
index ce0272ac45..e189b07bd4 100644
--- a/dspy/teleprompt/random_search.py
+++ b/dspy/teleprompt/random_search.py
@@ -123,7 +123,9 @@ def compile(self, student, *, teacher=None, trainset, valset=None, restrict=None
                 display_progress=True,
             )
 
-            score, subscores = evaluate(program, return_all_scores=True)
+            result = evaluate(program)
+
+            score, subscores = result.score, [output[2] for output in result.results]
 
             all_subscores.append(subscores)
 
diff --git a/dspy/teleprompt/teleprompt_optuna.py b/dspy/teleprompt/teleprompt_optuna.py
index 46cc12361e..31724de496 100644
--- a/dspy/teleprompt/teleprompt_optuna.py
+++ b/dspy/teleprompt/teleprompt_optuna.py
@@ -48,9 +48,9 @@ def objective(self, trial):
             display_table=False,
             display_progress=True,
         )
-        score = evaluate(program2, return_all_scores=False)
+        result = evaluate(program2)
         trial.set_user_attr("program", program2)
-        return score
+        return result.score
 
     def compile(self, student, *, teacher=None, max_demos, trainset, valset=None):
         import optuna
diff --git a/dspy/teleprompt/utils.py b/dspy/teleprompt/utils.py
index 31f5d19a92..4d71259e17 100644
--- a/dspy/teleprompt/utils.py
+++ b/dspy/teleprompt/utils.py
@@ -44,31 +44,24 @@ def create_minibatch(trainset, batch_size=50, rng=None):
     return minibatch
 
 
-def eval_candidate_program(batch_size, trainset, candidate_program, evaluate, rng=None, return_all_scores=False):
+def eval_candidate_program(batch_size, trainset, candidate_program, evaluate, rng=None):
     """Evaluate a candidate program on the trainset, using the specified batch size."""
 
     try:
         # Evaluate on the full trainset
         if batch_size >= len(trainset):
-            return evaluate(
-                candidate_program,
-                devset=trainset,
-                return_all_scores=return_all_scores,
-                callback_metadata={"metric_key": "eval_full"},
-            )
+            return evaluate(candidate_program, devset=trainset, callback_metadata={"metric_key": "eval_full"})
         # Or evaluate on a minibatch
         else:
             return evaluate(
                 candidate_program,
                 devset=create_minibatch(trainset, batch_size, rng),
-                return_all_scores=return_all_scores,
-                callback_metadata={"metric_key": "eval_minibatch"},
+                callback_metadata={"metric_key": "eval_minibatch"}
             )
     except Exception:
         logger.error("An exception occurred during evaluation", exc_info=True)
-        if return_all_scores:
-            return 0.0, [0.0] * len(trainset)
-        return 0.0  # TODO: Handle this better, as -ve scores are possible
+        # TODO: Handle this better, as -ve scores are possible
+        return dspy.Prediction(score=0.0, results=[])
 
 
 def eval_candidate_program_with_pruning(
diff --git a/tests/evaluate/test_evaluate.py b/tests/evaluate/test_evaluate.py
index 6f03fdbbbb..595d88cecf 100644
--- a/tests/evaluate/test_evaluate.py
+++ b/tests/evaluate/test_evaluate.py
@@ -6,7 +6,7 @@
 import pytest
 
 import dspy
-from dspy.evaluate.evaluate import Evaluate
+from dspy.evaluate.evaluate import Evaluate, EvaluationResult
 from dspy.evaluate.metrics import answer_exact_match
 from dspy.predict import Predict
 from dspy.utils.callback import BaseCallback
@@ -251,3 +251,7 @@ def on_evaluate_end(
     assert callback.start_call_count == 1
     assert callback.end_call_outputs == 100.0
     assert callback.end_call_count == 1
+
+def test_evaluation_result_repr():
+    result = EvaluationResult(score=100.0, results=[(new_example("What is 1+1?", "2"), {"answer": "2"}, 100.0)])
+    assert repr(result) == "EvaluationResult(score=100.0, results=`1 results`)"
diff --git a/tests/teleprompt/test_utils.py b/tests/teleprompt/test_utils.py
index 5b3a59098c..f3e03bd916 100644
--- a/tests/teleprompt/test_utils.py
+++ b/tests/teleprompt/test_utils.py
@@ -1,7 +1,5 @@
 from unittest.mock import Mock
 
-import pytest
-
 import dspy
 from dspy.teleprompt.utils import eval_candidate_program
 
@@ -43,19 +41,12 @@ def test_eval_candidate_program_minibatch():
     assert called_kwargs["callback_metadata"] == {"metric_key": "eval_minibatch"}
     assert result == 0
 
-
-@pytest.mark.parametrize("return_all_scores", [True, False])
-def test_eval_candidate_program_failure(return_all_scores):
+def test_eval_candidate_program_failure():
     trainset = [1, 2, 3, 4, 5]
     candidate_program = DummyModule()
     evaluate = Mock(side_effect=ValueError("Error"))
     batch_size = 3
 
-    result = eval_candidate_program(
-        batch_size, trainset, candidate_program, evaluate, return_all_scores=return_all_scores
-    )
+    result = eval_candidate_program(batch_size, trainset, candidate_program, evaluate)
 
-    if return_all_scores:
-        assert result == (0.0, [0.0] * len(trainset))
-    else:
-        assert result == 0.0
+    assert result == 0.0


Code review comments:
diff_hunk: @@ -41,6 +41,18 @@ def HTML(x: str) -> str:  # noqa: N802
 logger = logging.getLogger(__name__)
 
 
+class EvaluationResult(dspy.Prediction):
+    """
+    A class that represents the result of an evaluation.
+    It is a subclass of `dspy.Prediction` that contains the following fields
+
+    - score: An float value (e.g., 67.30) representing the overall performance
+    - results: a list of (example, prediction, score) tuples for each example in devset
+    """
+    def __repr__(self):
+        return f"EvaluationResult(score={self.score}, results=`{len(self.results)} results`)"
path: dspy/evaluate/evaluate.py
text: @user1:
I feel a bit unnatural when reading "results=128 results", maybe we can do something like below?

```
 return f"EvaluationResult(score={self.score}, results=<list of {len(self.results)} items>)"
```

@author:
Yeah, I was wondering what is the least ugly message. Let me use <> style

diff_hunk: @@ -41,6 +41,18 @@ def HTML(x: str) -> str:  # noqa: N802
 logger = logging.getLogger(__name__)
 
 
+class EvaluationResult(dspy.Prediction):
path: dspy/evaluate/evaluate.py
text: @user1:
minor nit: use direct import instead of `dspy.Prediction`