AIME: AI System Optimization via Multiple LLM Evaluators

## Abstract

Abstract Text-based AI system optimization typically involves a feedback loop scheme where a single LLM generates an evaluation in natural language of the current output to improve the next iteration’s output. However, in this work, we empirically demonstrate that for a practical and complex task (code generation) with multiple criteria to evaluate, utilizing only one LLM evaluator tends to let errors in generated code go undetected, thus leading to incorrect evaluations and ultimately suboptimal test case performance. Motivated by this failure case, we assume there exists an optimal evaluation policy that samples an evaluation between response and ground truth. We then theoretically prove that a linear combination of multiple evaluators can approximate this optimal policy.
From this insight, we propose AI system optimization via Multiple LLM Evaluators (AIME) . AIME is an evaluation protocol that utilizes multiple LLMs that each independently generate an evaluation on separate criteria and then combine them via concatenation. We provide an extensive empirical study showing AIME outperforming baseline methods in code generation tasks, with up to 62 % percent 62 62\% 62 % higher error detection rate and up to 16 % percent 16 16\% 16 % higher success rate than a single LLM evaluation protocol on LeetCodeHard and HumanEval datasets. We also show that the selection of the number of evaluators and which criteria to utilize is non-trivial as it can impact pact success rate by up to 12 % percent 12 12\% 12 % .

## 1 Introduction

Pre-trained foundation models, such as Large Language Models (LLMs), have developed rapidly over the recent years (Achiam et al., ; Touvron et al., ). With these advancements, AI systems have grown in popularity for various tasks such as code generation (Chen et al., ; Gulwani, ), question-answering (Patel et al., ; Wang et al., ), mathematical reasoning (Trinh et al., ; Song et al., ), exploration (Dorbala et al., ; ; Ren et al., ), and information retrieval (Gao et al., ) etc. As the application complexity increases, the shift to AI systems containing multiple components such as LLM-based agents and web search (Xiong et al., ), will continue (Zaharia et al., ; Yuksekgonul et al., ). Thus, automatically optimizing these systems, AI system optimization (Yuksekgonul et al., ), becomes increasingly necessary.

An emerging paradigm is text-based optimization, also known as prompt optimization (Cheng et al., ; Wang et al., ; Zhou et al., ), whereby the natural language input prompt is tuned to generate an optimal output. This method requires no numerical gradient descent updates typical in optimization for machine learning models (Van Der Malsburg, ; Hassoun, ; Barto, ) and is thus appropriate for optimizing AI systems with fixed LLM components. Recently, there has been a growing class of iterative online methods for text-based optimization (Cheng et al., ; Yuksekgonul et al., ; Shinn et al., ), where a single LLM generates an evaluation based on the current output to help generate the next iteration’s prompt.

While prior art has compared the abilities of a single LLM for evaluations against those of multiple LLMs (Kocmi & Federmann, ; Ankner et al., ), in AI system optimization literature, there has been a lack of studies questioning the capabilities of using a single LLM evaluator to drive the optimization process. Recently, Yuksekgonul et al. () has viewed the evaluation as a text-based analogy to the objective function for backpropagation (Hinton et al., ; Rumelhart et al., ) in deep learning optimization. The objective function is a crucial element in optimizing machine learning models (Christiano et al., ; Mescheder et al., ; Chakraborty et al., ; Kingma & Welling, ). This importance motivates us to analyze and strengthen the evaluation protocol of state-of-the-art (SoTA) AI system optimization frameworks by addressing a critical research question: What are the failure cases or tasks of utilizing only one LLM-based evaluator for text-based AI system optimization?

For this question, we empirically demonstrate the shortcomings of a single evaluator protocol in judging complex outputs like code based on multiple diverse criteria, such as correctness, readability, and runtime. We emphasize its practical limitations to give optimal evaluation while being instructed to judge based on all criteria simultaneously. Figure 1 illustrates the suboptimality in the practice of an AI system optimization framework with a single-evaluator approach to code generation. Furthermore, by assuming there exists an optimal evaluation policy that in expectation samples the true evaluation between the generated response and ground truth, we also theoretically highlight that the suboptimality gap between a single evaluator and an optimal evaluator is fixed and cannot be reduced given the same output and problem task.
With this insight, we then naturally ask the following subsequent query: Can we develop a principled evaluation method for text-based optimization to handle multiple criteria? We address this question by assuming there exists an optimal evaluation policy that in expectation samples the true evaluation between the generated response and ground truth. We then theoretically prove that, under a linear additivity assumption, increasing the number of evaluators can reduce the suboptimality gap.
We capitalize on this theoretical insight by proposing AIME: AI system optimization via Multiple Evaluators.
AIME generates and combines via concatenation independent natural language evaluations from multiple evaluators based on different evaluation instructions.
We demonstrate on code generation tasks with LeetCodeHard and HumanEval benchmarks the superior performance of AIME over a single evaluator in code error detection and the success rate of test cases.

Figure: Figure 1: AI System Optimization Pipeline and Increased Error Detection and Success Rate with AIME-based Evaluation: [LEFT] Text-based AI system optimization with SoTA framework (Yuksekgonul et al., ) using our multiple LLM evaluator approach AIME (orange) and with single-evaluator approach (blue). [TOP RIGHT] The single-evaluator approach cannot detect an error in the generated code that fails all test cases. However, one of the evaluators of AIME could because the logical evaluator was independent from the correctness evaluator. [BOTTOM RIGHT] AIME-based optimization achieves $\sim 16\%$ higher success rate than a single-evaluator approach in code generation tasks.
Refer to caption: extracted/5961014/figures/cover_figure.png

Our main contributions are as follows:

- •
Novel Evaluation Approach for AI system Optimization: We propose using multiple LLM-based evaluators and introduce our AIME approach for iterative AI system optimization. We concatenate independent diverse samples from multiple LLM-based evaluation policies to better critique system outputs.
- •
Theoretical Motivation for Multiple Evaluators: We prove that through a linear additivity assumption increasing the number of evaluations can reduce the suboptimatity gap from an optimal evaluation policy while a single evaluator has a fixed gap. This theoretical result helps justify our formulation for a multiple evaluation-based protocol.
- •
Empirical Performance Over Single-Evaluation Approach: Using popular code generation dataset, LeetCodeHard (Shinn et al., ) and HumanEval (Chen et al., ), we perform an extensive study showing the superior prowess of AIME with $6$ evaluators over single evaluation to detect errors, with AIME achieve up to $62\%$ higher error detection rate than single evaluation. We then show that AIME-based optimization achieves up to a $16\%$ higher success rate on test cases than optimization with only a single evaluator. We also reveal that the choice of the number of evaluators and the combination of criteria to utilize can affect the success rate by up to $12\%$, emphasizing the design of AIME-based optimization is non-trivial. We provide a code repository. (^1^11Repository to code: [https://github.com/Bridge00/aime](https://github.com/Bridge00/aime))

## 2 Text-based AI System Optimization

Objective Function. In this section, we now characterize mathematically text-based prompt optimization as a system of LLM-based policies.
Let $\pi(\cdot|x)$ be the LLM-based AI system parameterized by fixed LLM-based policy that samples an output response $y\sim\pi(\cdot|x)$ given an input prompt $x\in\mathcal{X}$ from the set of prompts $\mathcal{X}$. We aim to sample a $y\sim\pi(\cdot|x^{*})$ by finding an input prompt $x^{*}$ corresponding to $x$ prompt such that $y$ is closer to the optimal response $y^{*}$. For code generation, $\pi_{\theta}$ would be the LLM generator; $x$ would be the input prompt; $y$ is the generated code; and the $y^{*}$ here would be a code snippet that is a readable, efficient solution to the problem.
Mathematically, we can write

$$ $x^{*}=\operatorname*{arg\,min}_{x\in\mathcal{X}}\mathbb{E}_{y\sim\pi_{\theta}( \cdot|x)}[l(y^{*},y)],$ (1) $$

where $l$ is a loss function to capture the closeness of sampled response $y$ to the ground truth $y^{*}$.

Iterative text-based optimization. Given an initial prompt $x_{1}$, we perform an iterative text-based optimization method to find $x^{*}$ as follows. For each iteration $t=1$ to $T$, we start by (i) sampling $y_{t}\sim\pi_{\theta}(\cdot|x_{t})$, (ii) evaluate the response $y_{t}$ to obtain evaluation $e_{t}=l(y^{*},y_{t})$, and then finally (iii) generate the next prompt $x_{t+1}\sim\pi(\cdot|y_{t},e_{t},x_{t})$.
Recent work by Yuksekgonul et al. () decompose step (iii) into two separate steps and (iii.a) first generate the feedback $f_{t}\sim\pi(\cdot|y_{t},e_{t},x_{t})$, and then (iii.b) generate the next prompt $x_{t+1}\sim\pi(\cdot|y_{t},f_{t},x_{t})$. For simplicity, we use the same variable $\pi$ for all LLM-based policies because the outputs are dependent on the input variables the policy is conditioned on, so the same LLM model can be utilized. In this paper, we use the same model, GPT-4o, for all steps. However, distinct LLM models can be employed at different steps.

Challenges. In an ideal setting, if we had the access to $y^{*}$ as in supervised learning (Tiwari, ), then we can achieve the optimal performance with larger data. However, in practice, they are hard to obtain or simply unknown for many tasks such as code generation (Chen et al., ). Therefore, a direct comparison to an optimal output $y^{*}$ and the resulting calculation of $e$ in step (ii) are both infeasible. Current SoTA work instead sample an evaluation $e$ from an evaluation policy conditioned by the response output $y$ and prompt $x$ as $e\sim\pi(\cdot|x,y)$. Let us denote $\pi_{e}=\pi(\cdot|x,y)$ for notation simplicitiy. Ideally, we would like the evaluation $e$ of $y$ to be $l(y^{*},y)$. More specifically, let’s assume the existence of an optimal evaluator LLM denoted by $\pi^{*}_{e}$, sampling from which will give us samples of the true loss function $l(y^{*},y)$.

Fixed Gap in Evaluation with Single Evaluation Policy from Prior SOTA.
As $\pi_{e}^{*}$ is unavailable as discussed before, current SoTA methods sample the evaluation loss from a single evaluator as $e\sim\pi_{e}$. Now, we know that in the majority of the scenarios $\pi_{e}$ will not be the true evaluator policy $\pi_{e}^{*}$. Thus $e=l(\hat{y},y)$, where $\hat{y}$ is an implicit approximation of $y^{*}$ from $\pi_{e}$. Under this scenario, we define the suboptimality gap in evaluation of prior SOTA as

$$ $\displaystyle\Delta^{\pi}_{\textsf{Eva-sub-opt}}$ $\displaystyle=\mathbb{E}_{e^{*}\sim\pi^{*}(\cdot|x,y)}\left[e^{*}\right]- \mathbb{E}_{e\sim\pi(\cdot|x,y)}\left[e\right]\leq|e|_{\text{max}}d_{\text{TV} }(\pi_{e}^{*}(\cdot|x,y),\pi(\cdot|x,y))$ (2) $$

where we first expand upon the sub-optimality in evaluation and then upper-bound using the total variation distance (Sriperumbudur et al., ). We see that the term $d_{\text{TV}}(\pi_{e}^{*}(\cdot|x,y),\pi(\cdot|x,y))$ is fixed and it cannot be improved once we have the evaluator $\pi$. This result shows the hardness of a single evaluator reaching $\pi_{e}^{*}$ due to this constant gap and it will only reduce if our current LLM evaluator is near-optimal which is not true in majority of the scenarios. Empirically, Figure 1 demonstrates a practical observation where a single evaluator lets code errors go undetected, causing a large suboptimality gap from oracle performance in code generation tasks.

## 3 AIME: AI System Optimization via Multiple LLM Evaluators

Our key idea is to utilize multiple evaluations than single evaluators used in state-of-the-art. The thought that multiple evaluators would work better than one sounds intuitive but a naive introduction of multiple evaluators does not work in practice. We theoretically prove the merit of multiple evaluators and then discuss how to introduce them into the pipeline described in Section 2.

### 3.1 Increasing Evaluations Reduces the Evaluation Suboptimality Gap

Let $\Pi=\{\pi_{k}(\cdot|x,y)\}_{k=1}^{K}$ be the set of diverse evaluators for $x,y$. We start our theoretical justification by defining the sub-optimality metric to measure the evaluation performance between $\pi^{*}_{e}$ and $\Pi$ as

$$ $\displaystyle\Delta^{\Pi}_{\textsf{Eva-sub-opt}}=\mathbb{E}_{e\sim\pi^{*}_{e}( \cdot|x,y)}\left[e\right]-\mathbb{E}_{\{e_{k}\sim\pi_{k}(\cdot|x,y)\}_{k=1}^{K }}\left[g(e_{1},\cdots,e_{K})\right],$ (3) $$

which is nothing but the difference between the expected value of the evaluation under the optimal unknown evaluation distribution, and the expected function $g(\cdots)$ which maps the $K$ different evaluations to one. In practice, $g$ can be seen as an aggregation function such as concatenation. Note that if we had access to the optimal evaluator $\pi_{e}^{*}$, we would have been able to get the ground-truth evaluation $e^{*}=l(y^{*},y)$ to perform the AI text optimization. However, in place of that, we have a diverse set of evaluators $\Pi=(\pi_{1},\pi_{2}\cdots\pi_{K})$ and $g(e_{1},e_{2}\cdots e_{K})$ is the aggregation function to combine the losses from the diverse evaluators. We provide the following theorem to relate the number of evaluations to the $\Delta^{\Pi}_{\textsf{Eva-sub-opt}}$.

###### Theorem 1 .

Let $d_{\text{TV}}$ denote the total variation distance between two distributions and let $\sum_{k=1}^{K}\alpha_{k}=1$. Assuming all pairs $\pi_{1},\pi_{2}\in\Pi$ are independent of one another,

$$ $\displaystyle\begin{split}\Delta^{\Pi}_{\textsf{Eva-sub-opt}}\leq|e^{*}|d_{ \text{TV}}(\pi^{*}_{e},\sum_{k=1}^{K}\alpha_{k}\pi_{k}).\end{split}$ (4) $$

Proof. First, we characterize the sub-optimality of our proposed evaluation method as $\Delta=\mathbb{E}_{e^{*}\sim\pi^{*}_{e}}\left[e^{*}\right]-\mathbb{E}_{e_{1}
\sim\pi_{1}(\cdot|x,y),e_{2}\sim\pi_{2}(\cdot|x,y)\cdots\pi_{K}}\left[g(e_{1},
e_{2},e_{3}\cdots e_{K})\right]$.
Note that if $\Delta$ is zero, we are doing the optimal evaluation. Thus, we want $\Delta$ to be as low as possible. For simplicity of the expression, we will keep to two evaluators and it can easily extend to $K$ without loss of generality.

$$ $\begin{split}&\Delta=\mathbb{E}_{e^{*}\sim\pi^{*}_{e}}\left[e^{*}\right]- \mathbb{E}_{e_{1}\sim\pi_{1}(\cdot|x,y),e_{2}\sim\pi_{2}(\cdot|x,y)}\left[g(e_ {1},e_{2})\right]\\ &=\underbrace{\mathbb{E}_{e^{*}\sim\pi^{*}_{e}}\left[e^{*}\right]-\mathbb{E}_{ e\sim\pi_{d}(\cdot|x,y)}\left[e\right]}_{\Delta_{1}}+\underbrace{\mathbb{E}_{e \sim\pi_{d}(\cdot|x,y)}\left[e\right]-\mathbb{E}_{e_{1}\sim\pi_{1}(\cdot|x,y), e_{2}\sim\pi_{2}(\cdot|x,y)}\left[g(e_{1},e_{2})\right]}_{\Delta_{2}}.\end{split}$ $$

where we add and subtract the terms $\mathbb{E}_{e\sim\pi_{d}(\cdot|x,y)}$, with $\pi_{d}=\alpha\pi_{1}+(1-\alpha)\pi_{2}$ ($0<\alpha<1$) and then separate the two terms as $\Delta_{1},\Delta_{2}$. We next individually analyze the terms $\Delta_{1},\Delta_{2}$.

We can now bound $\Delta_{1}$ as,

$$ $\displaystyle\Delta_{1}$ $\displaystyle=\mathbb{E}_{e^{*}\sim\pi^{*}_{e}}\left[e^{*}\right]-\mathbb{E}_{ e\sim\pi_{d}(\cdot|x,y)}\left[l\right]\leq|e^{*}|d_{\text{TV}}(\pi^{*},\pi_{d} )=|e^{*}|d_{\text{TV}}(\pi^{*},\alpha\pi_{1}+(1-\alpha)\pi_{2})$ $$

where we use the property of integral probability metric to bound $\Delta_{1}$ as the total variation distance between the optimal evaluation policy and the mixture evaluation policy. Next, we proceed to $\Delta_{2}$,

$$ $\displaystyle\Delta_{2}$ $\displaystyle=\mathbb{E}_{e\sim\pi_{d}(\cdot|x,y)}\left[e\right]-\mathbb{E}_{e _{1}\sim\pi_{1}(\cdot|x,y),e_{2}\sim\pi_{2}(\cdot|x,y)}\left[g(e_{1},e_{2})\right]$ $\displaystyle=\mathbb{E}_{e\sim\pi_{d}(\cdot|x,y)}\left[e\right]-\mathbb{E}_{e _{1}\sim\pi_{1}(\cdot|x,y),e_{2}\sim\pi_{2}(\cdot|x,y)}\left[\alpha e_{1}+(1- \alpha)e_{2}\right]$ $\displaystyle=\mathbb{E}_{e^{*}\sim\pi_{d}(\cdot|x,y)}\left[e^{*}\right]- \alpha\mathbb{E}_{e_{1}\sim\pi_{1}(\cdot|x,y)}\left[e_{1}\right]-(1-\alpha) \mathbb{E}_{e_{2}\sim\pi_{2}(\cdot|x,y)}\left[e_{2}\right]=0$ $$

where we expand upon the definition of $\Delta_{2}$ and use linear additivity assumption on the aggregation function, where we assume $g(e_{1},e_{2})=\alpha e_{1}+(1-\alpha)e_{2}$. Under this assumption, the two terms cancel out with the final result $\Delta_{2}=0$. Combining both terms concluded the proof.
This bound indicates that the sub-optimality in evaluation can be expressed as the total variation distance between the optimal evaluator and the available mixture of evaluators. We know from Blei et al. (); Nguyen et al. () that as we increase the number of mixture components and diversity amongst the components increase, it can approximate any distribution under certain assumptions.

Figure: Algorithm 1 AIME: AI System Optimization via Multiple LLM Evaluators

###### Theorem 1 .

### 3.2 Overview of AIME: Multiple Role-Specific Evaluators

Now that we have motivated utilizing multiple LLM-based evaluators, we now address the question on how to utilize multiple evaluators. To do so, we look at the ideas of roles. The LLM-based evaluation policy has an evaluation system prompt to specify what the evaluation should be based on. For tasks such as code generation, there may be multiple criteria or objectives to evaluate for such as correctness, clarity, and efficiency. Furthermore, aspects such correctness of code can rely on various aspects such as logic and syntax. Normally, with a single evaluator, all the criteria are specified together in the system prompt. However, we see from Figure 1 and later in Section 4 that this approach can fail significantly to reach the optimal performance.
We thus propose splitting the evaluation instruction across multiple evaluators, each one getting a specific role. We then aggregate via string concatenation them into a final evaluation. We chose concatenation as the aggregation method as it is analogous to creating a linear combination of the outputs (Yuksekgonul et al., ). We call this approach AIME: AI System Optimization with Multiple Evaluators.

Our AIME approach is a simple-to-implement approach that requires minimal changes to the already established methods (Yuksekgonul et al., ; Cheng et al., ) for system optimization. Our approach requires only modifying the evaluation step of the optimization pipeline from one evaluator to multiple. In Algorithm 1, given an output $y$, set of $k$ roles $R$, and pre-trained LLM $\pi_{\theta}$ we sample $k$ evaluations, $\{e_{k}\}_{k=1}^{K}$. We obtain $e_{i}$ by conditioning $\pi_{\theta}$ by $x$, $y$ and $R_{k}\in R$. Conditioning on $r_{k}$ is to specify the role in the evaluation system prompt.

## 4 Experiments and Results

We test the merits of our AIME approach via the code generation task because of its practicalness and its multiple plausible criteria (e.g., correctness, efficiency). Here, the AI system is an LLM generator that is given a code prompt and must produce a code snippet that passes the unit tests for that prompt. This code generation task is a form of instance optimization (Yuksekgonul et al., ), whereby the optimization variable, the input prompt, is defined as $x_{t+1}:=(y_{t},f_{t})$. $y_{0}$, $f_{0}$ are empty strings. We provide empirical results showing that AIME is superior to the single-evaluation (Single-Eval) approach in detecting code errors and that AIME-based optimization achieves higher success in test cases than Single-Eval-based optimization. Experiments were run on an Apple M1 Pro and macOS 14.5.

AIME and Single-Eval Implementation Details: We use TextGrad from Yuksekgonul et al. () to implement AIME and Single-Eval. We chose TextGrad because it separates the evaluation and feedback into two separate LLM calls, making it better to analyze the evaluation module in isolation. In TextGrad, the system prompt that generates the initial code, $p_{\text{init}}$, is different from the system prompt that updates the code in the following refinement iterations $p_{\text{update}}$. At $t=0$, $p_{\text{init}}$ specifies to the LLM that it is a code generator while the $p_{\text{update}}$ from $1\leq t\leq T$ specifies that it generates a new version $y_{t+1}$ given the current code $y_{t}$ and the feedback $f_{t}$. The transition from $p_{\text{init}}$ to $p_{\text{update}}$ is explicitly programmed and not caused by the optimization process. Because the scope of this paper lies within the evaluation protocol, our AI system is a single LLM generator. (^2^22We repeat the link to the repository: [https://github.com/Bridge00/aime](https://github.com/Bridge00/aime))

LLM Setup Details: We use GPT-4o for all LLM calls and run $10$ iterations of optimization for each coding problem. Across all trials for both methods, we use the same initial generated code for a given problem so both evaluation protocols can judge the same code in the initial iteration. For Single-Eval, the solitary LLM evaluator call is allowed $3600$ max output tokens. For our AIME approach, each of the $K$ evaluators is allowed $\frac{3600}{K}$ max output tokens. This decision is to model a uniform distribution of weights ${\alpha}$. Note that when $k=1$, Single-Eval and AIME are equivalent. We share the evaluation system prompt for both methods in Appendix A.1. We ablate on the temperature of the evaluation LLM. All other LLM calls in the Textgrad pipeline are given $2000$ max output tokens with call temperature set to 0 0 similar to Yuksekgonul et al. (). For all experiments, the $\text{top_p}=0.99$.

Roles for Evaluating Code: The set of evaluation roles $R$ we used for this task are as follows: syntax errors, logic errors, correctness, readability, runtime, and code redundancy. The following results are based on utilizing all these roles. We chose three roles that correlate to maximizing the number of passed test cases: correctness, logic, and syntax. We specifically chose these three to incorporate an overall correctness role with two more specific roles. We will see in Section 4.1 that having overlapping roles can help with the robustness of evaluation in terms of error detection. The three other roles (readability, runtime, redundancy), correlate to criteria such as clarity and efficiency. We will later see in Section 4.3.2 that utilizing only these roles for evaluation decreases the overall performance of the code generation task.

Datasets: We use the following two datasets, LeetCodeHard (Shinn et al., ) and HumanEval (Chen et al., ), where each dataset contains a set of coding problem prompts and multiple unit tests for each problem to evaluate the generated code. We use the entire LeetCodeHard dataset of $39$ problems with an average of $2.2$ unit tests per problem and the first $20$ problems of HumanEval with an average of $4.4$ unit tests per problem. We withhold giving any of the evaluators of either method any information on unit tests to simulate the scenario where unit tests may be unavailable to help judge (Chen et al., ).

### 4.1 AIME is Robust to Incorrect Evaluations

AIME has a higher chance to catch errors: Figure 1 displays portions of an evaluation generated by Single-Eval and AIME. In this scenario, the evaluations were generated for the same coding problem at the second iteration of optimization. For both Single-Eval and AIME, the code failed all test cases, thus meaning there exists some error in the code. The evaluation from Single-Eval for both correctness and logic states there is nothing wrong. For AIME, the correctness evaluator incorrectly states nothing is wrong with the generated code but the logic evaluator detects a logical error. In the next iteration of optimization, the code generated based on the Single-Eval evaluation still fails all cases but the code generated from AIME passes them all.

Figure: Figure 2: Using LeetCodeHard and HumanEval benchmarks we compare evaluations generated from Single-Eval against those of AIME in terms of [LEFT] EDR and [RIGHT] RAE scores. AIME has a higher EDR score on both datasets indicating it is less prone to letting errors go undetected. AIME has a higher resistance to an adversarial evaluator on LeetCodeHard and a comparable resistance on HumanEval, suggesting its robustness over Single-Eval
Refer to caption: extracted/5961014/figures/edr_rae.png

Error Detection Measurement: To quantitatively analyze the error detection of AIME, we develop a heuristic measurement, Error Detection Rate (EDR). For each optimization iteration that has at least one failed test case, if the given evaluation contains at least one phrase indicating failure, we consider that as an error was detected. For example if the phrase “has a logical error” appears in the evaluation, we count that as an error detected. We provide a complete list of phrases used for detection in Appendix A.2. Let $Z_{\text{fail}}$ be the set of iterations with at least one failed test case and let $q(z)=\mathbbm{1}_{\text{error detected}}$ be the indicator value of whether an error was detected at iteration $z\in Z_{\text{fail}}$. We calculate the EDR as $\frac{1}{|Z_{\text{fail}}|}\sum_{z\in Z_{\text{fail}}}q_{z}$. Left of Figure 2 shows AIME has up to $\sim 62\%$ higher EDR than Single-Eval. Table 2 in Appendix A.3 summarizes the EDR for Single-Eval and AIME across various evaluation call temperatures. AIME achieves $\sim 53-62\%$ higher error detection rate than Single-Eval on LeetCodeHard and $\sim 38-57\%$ higher rate on HumanEval. This demonstrates that multiple independent evaluators can ensure a more accurate assessment than conditioning a single evaluator with all roles at once.

Robustness to Adversarial Evaluator (RAE): To further highlight the robustness of AIME to incorrect evaluations, we introduce an adversarial evaluator. For AIME, we specify in the system prompt of the correctness evaluator to always generate an evaluation stating that the code solution works. Similarly, for Single-Eval, we specify in the system prompt of the single evaluator to output an evaluation claiming that code works when discussing correctness. We provide these adversarial system prompts in Figure 6. We run experiments with an evaluation temperature of 1. To measure the robustness to the adversarial evaluator (RAE), we calculate the percent decrease of the EDR from the non-adversarial setting to the adversarial one. We then report the absolute value of the percent decrease subtracted from $1$. Formally, let $p_{c}$ be the percent change of the EDR, our RAE metric is $1-|p_{c}|$. Right of Figure 2 reports the mean and standard deviation RAE over $3$ trials. AIME achieves $16\%$ higher RAE over Single-Eval on LeetCodeHard and comparable RAE over HumanEval, emphasizing AIME increased safety for AI systems.

AIME evaluations are more thorough: In Figure 3, we highlight the readability portions of the same evaluation in Figure 1. Even though both Single-Eval and AIME did not see errors in readability, AIME is more thorough and explains its evaluation while Single-Eval only gives a one-sentence judgment. We believe this also to be because of the independence of the readability evaluator in AIME as the evaluator does not feel the need to move on to the next role like in Single-Eval even though there is nothing to critique. AIME is thus more helpful in terms of explainability. Please see Appendix A.5 for more comparisons between evaluations AIME and Single-Eval.

Figure: Figure 3: Independent evaluator of AIME provides more thorough explanations: Example evaluations for readability generated by Single-Eval and AIME. Both evaluations are for the same coding task at the same iteration which failed all test cases. Even though both Single-Eval and AIME believe that the code is readable with no criticisms, AIME’s readability comment is more thorough. This result may be because it was generated independently from evaluations of other criteria. Without having other to worry about other roles, the readability evaluator was allowed to focus its entire output on readability.
Refer to caption: extracted/5961014/figures/eval_read_comparison2.png

### 4.2 AIME-based Optimization Achieves Higher Task Performance

Figure: Figure 4: [BAR PLOT] Success Rate and Completion Rate and [LINE PLOT] Best Completion Rate over max number of iterations for [LEFT] LeetCodeHard and [RIGHT] HumanEval. Over $10$ iterations for each coding problem, AIME has the highest SR and CR over both datasets.
Refer to caption: extracted/5961014/figures/leetcode_humaneval.png

Now that we have established the error detection capabilities of AIME over Single-Eval, we now focus on the overall performance of system optimization with AIME on the code generation task. For these experiments, we provide results with two additional baselines:
1) Zero-Shot: Initial generated code with no iterative optimization process; 2) Refinement with No Separate Text-based Evaluation Step (Implicit Eval): The evaluation and feedback steps are within the same LLM “reflection” call. The LLM reflection call is allowed $3600$ max output tokens and is sampled once per iteration. We implement this baseline with Reflexion by Shinn et al. ().

Metrics for Code Correctness:
We report the following metrics to inspect the correctness of the code generated; for AIME, Single-Eval, and Implicit Eval, we report these metrics using the best-performing code generated in the optimization process after the initial zero-shot generation: 1) Success Rate (SR), the percentage of test cases passed across the entire dataset; 2) Completion Rate (CR), the percentage of coding problems with all passed test cases.

Test Case Results: We plot the performance over $3$ trials on both datasets in Figure 4. Please see Table 3 in Appendix A.3 where we report the standard deviation and ablate the temperature of the evaluation LLM call. Over both datasets, AIME consistently has the highest SR and CR rates with up to $\sim 13\%$ higher SR and $\sim 18\%$ higher CR.

Remark: The analysis on EDR in Section 4.1 is specifically for comparing the error detection capabilities of the evaluation protocols, it does not take into account the downstream feedback LLM call in Textgrad system pipeline. This point may explain why Single-Eval can have a significantly lower error detection rate than AIME but then have a much smaller gap in SR and CR, as the feedback LLM is possibly also detecting errors and disregarding the incorrect evaluations. Another possibility for the low error detection rate could be more detection phrases are needed to give a better estimate for Single-Eval.

Figure: Figure 5: Increasing Number of Evaluator and Diversity Helps: [LEFT] When setting all the evaluators of AIME to the same role, correctness, and increasing the number of evaluators from $1\to 3\to 6$ increases EDR. This result shows that even if there is only one role, multiple independent evaluations can help catch errors. [RIGHT] With six evaluators, having 6 distinct roles has better SR, CR, and EDR, than all of the evaluators having the same role, correctness.
Refer to caption: extracted/5961014/figures/num_evals_diversity.png

### 4.3 Ablation Studies

#### 4.3.1 Increasing Number of Evaluators and Diversity of Roles Helps

We perform two experiments: 1) for AIME-based optimization, we ablate on the number of evaluators from $1\to 3\to 6$. However, each evaluator has the same role. Max output tokens in each experiment across all evaluators is $3600$. When all the evaluators have the correctness role (left of Figure 5), the EDR for AIME increases. This result emphasizes that AIME-based evaluations, even without role-specific evaluators, can detect more errors than Single-Eval. This finding then begs the question of whether there is a need for different roles to optimize for passed test cases if increasing the number of same-role evaluators already helps. When comparing the SR, CR, and EDR of AIME with 6 correctness evaluators against AIME with 6 distinct roles (correctness, logic, syntax, readability, runtime, redundancy), the increased diversity of roles raises these metrics (right of Figure 5). In the following study, we analyze which roles impact performance.

**Table 1: Utilzing Different Roles Affects SR and CR: This table summarizes the SR and CR for Single-Eval and AIME given different combinations of roles. We report the mean and standard deviation of $3$ trials. For the experiments with a single role, as in $K=1$, Single-Eval and AIME are the same. We see that SR and CR drops when not utilizing syntax, logic, or correctness evaluators. We also see that the SR and CR drop is not as significant for Single-Eval as it is for AIME, suggesting that Single-Eval protocol is less dependent on the roles correlated with maximizing passed test cases.**
| Syn-<br>tax | Correct-<br>ness | Logic | Read-<br>ability | Run-<br>Time | Code<br>Redun-<br>dancy | Metric<br>(%) | Single-<br>Eval | AIME<br>(Ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | SR | 83.70 $\pm$ 2.28 | 89.26 $\pm$ 2.10 |
| CR | 76.07 $\pm$ 1.21 | 82.91 $\pm$ 1.21 |  |  |  |  |  |  |
| ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | SR | 80.74 $\pm$ 2.10 | 77.41 $\pm$ 1.39 |
| CR | 66.67 $\pm$ 4.19 | 64.96 $\pm$ 3.20 |  |  |  |  |  |  |
| ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | SR | 87.78 $\pm$ 1.81 | 88.89 $\pm$ 0.91 |
| CR | 81.20 $\pm$ 1.21 | 80.34 $\pm$ 1.21 |  |  |  |  |  |  |
| ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | SR | 83.70 $\pm$ 1.05 |  |
| CR | 5.21 $\pm$ 1.21 |  |  |  |  |  |  |  |
| ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | SR | 85.55 $\pm$ 3.27 |  |
| CR | 75.21 $\pm$ 3.20 |  |  |  |  |  |  |  |
| ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | SR | 85.93 $\pm$ 2.28 |  |
| CR | 77.78 $\pm$ 3.20 |  |  |  |  |  |  |  |
| ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | SR | 87.04 $\pm$ 3.78 | 88.51 $\pm$ 1.89 |
| CR | 79.49 $\pm$ 5.54 | 80.34 $\pm$ 3.20 |  |  |  |  |  |  |
| ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | SR | 79.26 $\pm$ 1.39 |  |
| CR | 70.01 $\pm$ 3.20 |  |  |  |  |  |  |  |

#### 4.3.2 Combination of Evaluation Roles Affects Optimization Performance

We now analyze the effect the different roles have on SR and CR on LeetCodeHard. We perform this study for two reasons: 1) to see the change in performance due to utilizing various evaluation roles and 2) to see how the relative performance between Singl-Eval and our AIME changes based on the roles given. The total max output tokens for evaluation is still $3600$, and for AIME, it is distributed equally across the evaluators. Therefore, for experiments with $3$ evaluators, each one has max output tokens of $1200$.

Table 1 summarizes our results and reports the mean and standard deviation over $3$ trials for each experiment. All experiments were run with an evaluation temperature of 1. When only utilizing the readability, runtime, and code redundancy evaluators, SR and CR degrade by $\sim 12\%$ and $\sim 18\%$, respectively, for AIME. Interestingly, this combination of roles is also the only time in this ablation that Single-Eval performs higher in SR and CR than AIME. This outperformance is because the degradation in SR and CR for Single-Eval is significantly less than for AIME, suggesting that AIME was more dependent on the correctness, logic, and syntax roles for optimizing unit tests than Single-Eval. However, for all other experiments, AIME still has higher SR and CR, supporting the idea that separating the evaluation into role-specific policies allows for generally higher performance than a single evaluator across different combinations of roles.

Furthermore, for both Single-Eval and AIME, the SR drops by $3-5\%$ when going from using syntax, correctness, and logic, to using only one of them. This suggests that using all three in combination increases the evaluation in terms of maximizing passed unit tests. In Appendix A.4, we perform two similar ablation studies. In one study, we give the evaluators information on what test cases passed and failed. In the second study, we provide information on what passed and failed and include an explanation of each failure.

## 5 Related Works

AI System Optimization: Many prior works have studied the optimization of complex AI systems. Madaan et al. () was one of the first works to propose a text-based iterative feedback loop for refining LLMs, and Pryzant et al. () established text-based gradients, or Textual Gradients, as feedback to an AI system. DSPy (Khattab et al., ; ; Singhvi et al., ), Trace Cheng et al. (), and TextGrad (Yuksekgonul et al., ) have formulated LLM and AI-based systems as a network of multiple layers and provided methods to optimize these system analogous to backpropagation and autodifferentiation. Chakraborty et al. (); Ding et al. () used a bi-level optimization formulation to align AI agents and systems. Text-based reinforcement learning has also been used to improve LLM-based systems (Shinn et al., ). Decoding and RLHF is an alternative method to optimize or align an LLM with gradient descent (Chakraborty et al., ; Mudgal et al., ; Chakraborty et al., ). While these works have shown tremendous results, there has been a gap in the literature we aim to address analyzing the effect of using multiple independent evaluations to optimize the AI system for a complex task, code generation (Chen et al., ; Zeng et al., ; Zhang et al., ; Jha et al., ; Shinn et al., ; Yuksekgonul et al., ; Zan et al., ; Jiang et al., ; Chen et al., ; Gulwani, ).

LLM-based Evaluation: LLM-based evaluation, or LLM-as-a-Judge (Zheng et al., ), has been growing in interest due to the ability of LLMs to evaluate large outputs like text (Sellam et al., ; Kocmi & Federmann, ) quickly and to align with human preferences.
Verga et al. () showed a panel of smaller LLM judges can provide numeric scores correlating to human judgment than a single larger LLM model can. Prior work has also studied finetuning LLMs to be judges (Zhou et al., ). Ankner et al. () used LLM-generated critiques to augment the scalar reward from a reward model. Li et al. () used discussion between multiple LLMs to select a strong LLM-based evaluator for question-answering. Strong LLM judges have been shown to generalize across tasks (Huang et al., ). Weak LLM evaluators have been used to judge the debate between two stronger LLMs (Kenton et al., ). We are the first to use multiple LLM-based evaluators for iterative AI system optimization.

## 6 Conclusion, Limitations, and Further Works

In this work, we tackle AI system optimization by introducing AIME. AIME utilizes multiple LLM-based evaluators to provide natural language evaluation for the current system output, improving on prior methods that only use a single evaluator. Our key insight is to condition each evaluator with a specific role rather than giving all the roles to a single evaluator.
We prove that increasing the number of evaluations reduces the suboptimality evaluation gap, and empirically demonstrate that AIME outperforms Single-Eval in code generation tasks, analyzing success, completion, and error detection rates. Furthermore, we study AIME’s robustness to the adversarial evaluator that generate incorrect evaluations. We also provide ablations such as on the diversity of roles, role combinations, and evaluation temperature, consistently demonstrating AIME’s superior performance and the need for multiple evaluators.

Limitations and Further Work. We only empirically study our approach in code generation. Further work could extend this evaluation approach to other tasks that require multiple criteria like molecule optimization or text generation. In terms of system complexity, we only study multiple evaluators for AI systems comprising a single LLM-based agent, and using a compound system with multiple elements such as a web search agent (Agentic AI system) could be interesting. Another aspect of the work that can be explored further is weighting the different LLM-based evaluations. We gave uniform weighting to all evaluations by giving them the same max output tokens and concatenating them. Future research could investigate methods of weighting and aggregation, possibly using another LLM to summarize or perform best-of-N on the evaluations.

## Appendix A Appendix

### A.1 Evaluation System Prompt

We provide the evaluation system prompt in Figure 6. For Single-Eval the system prompt is given to only one LLM call and all the roles utilized are listed together in [INSERT UTILIZED ROLE]. For AIME, each evaluator gets one role specified in [INSERT UTILIZED ROLE].

Remark: It may seem conflicting that we specify conciseness in the evaluation system prompt and highlight that the evaluations from AIME are more descriptive in Figure 3. However, we would like to clarify that we do not believe that the evaluations are verbose, using more words without giving more information. The longer, thorough evaluations from AIME like in Figure 3 provide more information on their judgment, helping with the explainability of the evaluation model.

Figure: Figure 6: Evaluation System Prompt.
Refer to caption: extracted/5961014/figures/eval_system_prompt.png

### A.2 Error Detection Phrases

Below is the list of phrases we used to analyze the error detection of evaluations,

- •
has logical errors
- •
contains logical errors
- •
has a logical error
- •
contains a logical error
- •
is incorrect
- •
to be incorrect
- •
has a syntax error
- •
contains a syntax error
- •
contains syntax errors
- •
has syntax errors
- •
has several issues
- •
does not correctly
- •
appears to be mostly correct
- •
have several issues
- •
has several issues
- •
flaw
- •
incorrect
- •
not correct
- •
some issue
- •
there seems to be some issues
- •
has issue
- •
have issue

### A.3 Evaluation Temperature Ablation on EDR and Overall Performance

**Table 2: AIME detects more code errors than Single-Eval: Error Detection Rates of evaluations generated by Single-Eval and AIME. Over all temperatures, AIME detects has up to $61\%$ and $72\%$ higher rate than Single-Eval on LeetCodeHard and HumanEval, respectively. Thus, multiple independent role-specific evaluators are more likely to detect errors than a single evaluator with all roles.**
| Eval LLM Call Temp | Dataset | Single-Eval | AIME (Ours) |
| --- | --- | --- | --- |
| 0 | LeetCodeHard | 38.06 $\pm$ 6.80 | 91.20 $\pm$ 0.90 |
| HumanEval | 10.99 $\pm$ 2.33 | 49.0 $\pm$ 6.02 |  |
| 0.25 | LeetCodeHard | 34.19 $\pm$ 2.88 | 90.67 $\pm$ 1.05 |
| HumanEval | 19.65 $\pm$ 1.27 | 76.37 $\pm$ 12.88 |  |
| 0.50 | LeetCodeHard | 29.49 $\pm$ 4.06 | 91.93 $\pm$ 2.42 |
| HumanEval | 17.90 $\pm$ 9.15 | 55.80 $\pm$ 2.54 |  |
| 0.75 | LeetCodeHard | 35.43 $\pm$ 2.53 | 90.09 $\pm$ 0.39 |
| HumanEval | 3.80 $\pm$ 1.15 | 53.61 $\pm$ 5.98 |  |
| 1 | LeetCodeHard | 31.36 $\pm$ 4.25 | 91.07 $\pm$ 2.45 |
| HumanEval | 8.13 $\pm$ 5.80 | 60.45 $\pm$ 12.99 |  |

**Table 3: The success and completion rates for AIME (ours) and Single-Eval on LeetCodeHard code generation datasets with varying values for evaluating LLM call temperature. Consistent with Figure 4, AIME generally outperforms Single-Eval.**
| Eval LLM Temp | Dataset | Metric (%) | Single-Eval | AIME (Ours) |
| --- | --- | --- | --- | --- |
| 0 | LeetCodeHard | SR | 82.96 $\pm$ 3.44 | 87.41 $\pm$ 2.28 |
| CR | 75.21 $\pm$ 4.83 | 79.49 $\pm$ 2.09 |  |  |
| HumanEval | SR | 91.67 $\pm$ 2.14 | 93.18 $\pm$ 0.00 |  |
| CR | 93.33 $\pm$ 2.36 | 95.00 $\pm$ 0.00 |  |  |
| 0.25 | LeetCodeHard | SR | 82.96 $\pm$ 3.43 | 86.30 $\pm$ 1.04 |
| CR | 75.21 $\pm$ 4.83 | 77.78 $\pm$ 1.21 |  |  |
| HumanEval | SR | 91.28 $\pm$ 2.69 | 91.67 $\pm$ 2.41 |  |
| CR | 93.33 $\pm$ 2.36 | 91.67 $\pm$ 4.71 |  |  |
| 0.50 | LeetCodeHard | SR | 82.96 $\pm$ 1.04 | 89.30 $\pm$ 1.39 |
| CR | 72.65 $\pm$ 1.21 | 81.20 $\pm$ 3.20 |  |  |
| HumanEval | SR | 89.39 $\pm$ 1.42 | 92.42 $\pm$ 1.07 |  |
| CR | 90.00 $\pm$ 0.00 | 93.33 $\pm$ 2.36 |  |  |
| 0.75 | LeetCodeHard | SR | 83.70 $\pm$ 3.67 | 90.37 $\pm$ 3.19 |
| CR | 76.92 $\pm$ 5.54 | 83.76 $\pm$ 3.20 |  |  |
| HumanEval | SR | 91.29 $\pm$ 2.68 | 92.42 $\pm$ 1.07 |  |
| CR | 93.33 $\pm$ 2.36 | 93.33 $\pm$ 2.36 |  |  |
| 1 | LeetCodeHard | SR | 83.70 $\pm$ 2.28 | 89.26 $\pm$ 2.10 |
| CR | 76.07 $\pm$ 1.21 | 82.91 $\pm$ 1.21 |  |  |
| HumanEval | SR | 90.15 $\pm$ 2.34 | 93.18 $\pm$ 0.00 |  |
| CR | 91.76 $\pm$ 2.36 | 95.00 $\pm$ 0.00 |  |  |

### A.4 Giving evaluators test result information

**Table 4: Impact of different role combination like in Table 1. Here, we give the evaluators which test passed or failed [TOP] with failure explanations [BOTTOM] without failure explanation. Failure explanations could be runtime errors or incorrect return values.**
| Syn-<br>tax | Correct-<br>ness | Logic | Read-<br>ability | Run-<br>Time | Code<br>Redun-<br>dancy | Metric<br>(%) | Single-<br>Eval | AIME<br>(Ours) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Tests given with failure explanations |  |  |  |  |  |  |  |  |
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | SR | 88.15 $\pm$ 1.39 | 90.00 $\pm$ 1.57 |
| CR | 81.20 $\pm$ 2.42 | 82.91 $\pm$ 3.20 |  |  |  |  |  |  |
| ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | SR | 86.30 $\pm$ 0.52 | 89.26 $\pm$ 1.89 |
| CR | 79.49 $\pm$ 2.09 | 83.76 $\pm$ 3.20 |  |  |  |  |  |  |
| ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | SR | 87.78 $\pm$ 3.27 | 88.14 $\pm$ 1.39 |
| CR | 80.34 $\pm$ 4.36 | 80.34 $\pm$ 1.21 |  |  |  |  |  |  |
| Tests given with no failure explanation |  |  |  |  |  |  |  |  |
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | SR | 85.19 $\pm$ 0.52 | 90.37 $\pm$ 1.89 |
| CR | 79.49 $\pm$ 2.09 | 82.91 $\pm$ 2.42 |  |  |  |  |  |  |
| ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | SR | 84.44 $\pm$ 1.81 | 86.67 $\pm$ 0.91 |
| CR | 74.36 $\pm$ 3.63 | 78.63 $\pm$ 2.42 |  |  |  |  |  |  |
| ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | SR | 86.67 $\pm$ 1.81 | 86.30 $\pm$ 3.78 |
| CR | 79.49 $\pm$ 0.00 | 77.78 $\pm$ 4.36 |  |  |  |  |  |  |

### A.5 Example Evaluations

To emphasize the more thorough evaluations from our AIME method, we provide a few more comparisons of evaluations generated by AIME and Single-Eval.

Figure: Figure 7: Comparison of evaluations from Single-Eval and AIME for LeetCodeHard problem: minimum-time-to-visit-a-cell-in-a-grid. These evaluations are the full versions of the ones analyzed in the main body in Figures 1 and 3.
Refer to caption: extracted/5961014/figures/appendix_eval_comp/eval_comp1.png

Figure: Figure 8: Comparison of evaluations from Single-Eval and AIME for LeetCodeHard problem: paths-in-matrix-whose-sum-is-divisible-by-k.
Refer to caption: extracted/5961014/figures/appendix_eval_comp/eval_comp2.png

Figure: Figure 9: Comparison of evaluations from Single-Eval and AIME for LeetCodeHard problem: count-number-of-possible-root-nodes.
Refer to caption: extracted/5961014/figures/appendix_eval_comp/eval_comp3.png

Figure: Figure 10: Comparison of evaluations from Single-Eval and AIME for LeetCodeHard problem: minimum-number-of-visited-cells-in-a-grid.
Refer to caption: extracted/5961014/figures/appendix_eval_comp/eval_comp4.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - The results in Table 2 in the Appendix seem to indicate that for LeetCodeHard, the method is mostly invariant to temperature, whereas the difference in sampled responses for HumanEval seems more prominent. Could you explain why that is the case, perhaps with some qualitative examples of generations?
- Given the error detection phrases, if the LLM were to output something like "This code is correct, it has to be incorrect because it does not pass the first test", would that count as 2 errors or one? In other words, is there any deduplication done while computing the EDR and other metrics?
- There are 2 temperature settings mentioned: temperature = 0 in the LLM Setup Details in Section 4, and temperature = 1 in the Robustness to Adversarial Evaluator paragraph in Section 4.1. Could these choices be explained and also clarified in the main text?
- The SingleEval performance on HumanEval (Figure 2, left) is very low; could the authors provide some examples of generated output that lead to such a low value?
soundness: 2
strengths: - The method proposed is general, conceptually simple, and could also be applied to other domains where multiple objectives need to be optimised.
- Figure 1 clearly shows the approach and its advantages over a single LLM evaluation and clearly motivates the work.
- Most of the results show non-trivial improvements over the baselines considered.
- The role-wise analysis in Table 1 gives a nice understanding of the contribution of each role in determining performance.
summary: This paper proposes an alternative to evaluating LLMs along multiple axes by specifying different criteria such as logic, correctness, clarity, etc. The primary metric is the proportion of errors detected using their approach compared with a baseline in which all criteria are mentioned and evaluated in a single shot. The proposed mechanism, AIME, achieves higher error detection rates compared to the baselines and comparable or higher success rate than the baselines on the HumanEval and LeetCodeHard datasets.
weaknesses: - An important baseline that is missing and could potentially give superior results compared to the current baselines is: a single prompt containing descriptions of all criteria, as is the case with SingleEval, but with $k$ independent responses sampled from the LLM with temperature $> 0$. This would arguably give a much fairer comparison than allowing more tokens (3600 for SingleEval versus 3600/$k$ for AIME) since literature in LLMs contains evidence of majority voting or sampling methods outperforming greedy response selection [1, 2]. Furthermore, the results shown in Figure 5 clearly motivate this baseline: increasing the number of evaluators is shown to be helpful even when the same role is maintained.
- There is scope for improving the writing, especially Section 3.2, which contains the overview. It would also be helpful to have a clear description of the roles or criteria in one place and then discuss which ones are more consequential in obtaining the gain in performance (Section 4.3.2 mentions that readability, runtime, and code redundancy evaluators are secondary and leads to worse performance than the SingleEval baseline).
- There are some readability issues: the font size in the plots in Figure 4 is too small to be legible.
- From the results it seems that HumanEval is perhaps not the best dataset to demonstrate the strength of AIME; the performance in terms of success rate and completion rate for all methods is comparable, and the gains seem to be within noise (Tables 3 and 4 in Appendix).
- There is a claim made in the paper: "AIME is more thorough", which seems to be more of a qualitative observation than a quantitative result. It would be better if the paper were to clarify why longer responses are considered more thorough. The example shown in Figure 3 is clear enough, but the examples shown in the appendix seem to indicate that longer answers are considered to be more thorough, which may not always be the case. It would be better to substantiate the argument with what constitutes thoroughness in responses (whether it is the length of the response or something else).

[1] Learning to summarize from human feedback; Stiennon et al, 2020
[2] Self-Consistency Improves Chain of Thought Reasoning in Language Models; Wang et al, 2023.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Please respond to the above when possible as well.

- What does section 3.1 really say? How is this different to all the proofs that exist for ensembles?
- Figure 3 makes me think of chain-of-thought. Did you consider looking at the effect of that on the performances here?
- In Figure 4, why is the EDR so different? This seems like the defining difference between this work and the ensemble/the work mentioned above (and the thousand similar ones out there).
soundness: 4
strengths: - The paper is decently well-written and reads like a paper one would expect to see at ICLR.
- The evaluation is sufficiently extensive and shows strong performance over the baseline.
- The approach is quite modular, very practical, and easily extendable.
- I haven't seen a paper exposing exactly this result before.
summary: The authors present an ensemble-based system for prompt optimization in agentic systems. This is built by looking at the kinds of errors made by individual LLM evaluators. They conduct a detailed study showing the strength of their system.
weaknesses: - My biggest issue here is in the strength of the contribution. We know that combining several models will improve performance (i.e., ensemble methods). We know this holds true with LLMs as well when solving general tasks (Schmidhuber's Mindstorms paper to name just one). So what exactly is the new thing this work says? Is it just what roles the evaluators should take on to get good performance?
- I would have expected that there would be a fairly extensive discussion of ensemble methods in this paper and that it would be framed as one.
- The Rumelhard citation for backprop should be accompanied by the earlier work that preceded it.
- Less easy to address, I feel like there's something missing on top of the evaluators. Without that, it just feels like another paper saying ensembles work.

The above is my reading of this work. However, the reviewing process is noisy, and no matter how anyone feels about their evaluation, the data says that even the best reviewers can only provide truly medium-confidence reviews. Thus, I'm likely to acquiesce and change my score by a non-trivial amount if the other reviews disagree or the authors can correctly do away with my concerns.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Across the two benchmarks, why is the amount of performance improvement with AIME so significantly different (for example, Figures 2 and 4)?
soundness: 1
strengths: - Expansive empirical evaluation with insightful ablations.
- That having diversity of roles for evaluation helps is a useful takeaway, much like it has been shown to help for training [1].

---
[1] Rame, Alexandre, et al. "Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards." Advances in Neural Information Processing Systems 36 (2024).
summary: The paper studies the problem of optimizing LLMs using feedback in an iterative manner. Using other LLMs as a proxy for the evaluators, the proposed method recommends the use of *multiple* LLMs instead of a single LLM as is common in practice. Along with a rigorous empirical evaluation of this proposal, the paper also provides theoretical justification for this approach under some idealized assumptions.
weaknesses: - The idea of using multiple LLMs as evaluators has been proposed in some prior works [1,2]. Moreover, the theoretical justification is very idealized and disconnected from practical setups. This diminishes the overall contribution and the work is expected to have a heavy emphasis on its empirical contribution (more on this next).
- Theory:
  - The evaluation suboptimality bounds are for the case of one-shot evaluation rather than iterative evaluation and improvement, the latter being the setup of the paper.
  - The proof is strewn with typos and errors. A few examples:
    - Equation (4): Like Equation (2) the leading term should be $|e_\max|$ rather than $|e^*|$.
    - [L204]: The expression for $\Delta_1$ has an undefined $l$. This statement also introduces the above error.
    - Last equation in the equation block for $\Delta_2$ randomly introduces $e^*$.
  - The idea of using multiple evaluators heavily relies on the last statement “as we increase the number of mixture components [...] under certain assumptions.”. The assumptions have not been stated.
- Empirical evaluation:
  - EDR: The definition seems to have some error. $\frac{1}{|Z_\text{fail}|} \sum_{z \in Z_\text{fail}} q_z$ evaluates to 1, since $Z_\text{fail}$ by definition is the “set of iterations with at least one failed test case”. What is the actual definition? And why is this metric so significantly favorable for AIME relative to SR and CR?

Nitpicks:
- The paper is hastily written. This is indicated by typos in the paper, some examples of which are:
  - Third row of Table 3 highlights AIME as being more performant, whereas the numbers for Single-Eval are higher.
  - [L159] has an unaddressed comment “cite ipm”
  - [L185] “combine the losses”: the *evaluations* are combined, not losses.

---
[1] Verga, Pat, et al. "Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models." arXiv preprint arXiv:2404.18796 (2024).

[2] Kim, Seungone, et al. "The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models." arXiv preprint arXiv:2406.05761 (2024).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Is the evaluation considered as scalers, strings, or something else? Are they assumed to be different types in 3.1 and 3.2?
2. Is that true theorem 1 only applies to the case when the mixture constants $\alpha_k$s are the same for the mixture evaluator distribution and the linearly combined evaluator $g(\cdot)$?
soundness: 1
strengths: 1. The proposed approach is novel.
summary: This paper claims that when evaluating an AI-generated output in the context of text-based optimization, AI system optimization via multiple LLM evaluators (AIME) is preferred over a single LLM evaluator. The paper formulated the iterative text-based optimization problem and shows that the suboptimality of a linearly combined evaluator is bounded by the corresponding mixture distribution of the basis evaluators. The paper empirically shows that AIME significantly outperforms the single LLM evaluator approach in error detection and optimization gain on the LeetCodeHard and HumanEval datasets. Ablation studies are performed to demonstrate the impact of the combination of evaluators.
weaknesses: 1. The additive nature of the evaluations is not clearly stated in section 2.
2. The additivity assumption on the aggregation function, which I believe is the key to proving the theory, is not stated in the theorem statement nor anywhere in section 3 before it is used on line 210.
3. The proof of theorem one should include the case of more than two evaluators. 
4. In section 3.2, the representation of evaluation is changed from addable to concatable without a clear motivation. This makes the paper inconsistent.
5. The paper proposes multiple evaluators for general-purpose evaluation. However, the experiments only validate it on instance optimization for code generation.
6. The error detection measure evaluated in section 4.1 seems only to study the case when there are failed test cases. The aspect that a good evaluator should also be accurate when there is no error is not tested.
7. The scope claimed in the title is too broad. I recommend replaying "AI System Optimization" with "Text-based Optimization" or something similar.
8. There are mistakes in the writing, i.e., "cite ipm" in the parenthesis on line 158.