When is Tree Search Useful for LLM Planning? It Depends on the Discriminator (2024)

Abstract: In this paper, we examine how large language models (LLMs) solve multi-step problems under a language agent framework with three components: a generator, a discriminator, and a planning method. We investigate the practical utility of two advanced planning methods, iterative correction and tree search. We present a comprehensive analysis of how discrimination accuracy affects the overall performance of agents when using these two methods or a simpler method, re-ranking. Experiments on two tasks, text-to-SQL parsing and mathematical reasoning, show that: (1) advanced planning methods demand discriminators with at least 90% accuracy to achieve significant improvements over re-ranking; (2) current LLMs’ discrimination abilities have not met the needs of advanced planning methods to achieve such improvements; (3) with LLM-based discriminators, advanced planning methods may not adequately balance accuracy and efficiency. For example, compared to the other two methods, tree search is at least 10–20 times slower but leads to negligible performance gains, which hinders its real-world applications.1

heading: 1 Introduction
text: Planning plays a crucial role in intelligent behaviors of human and AI agents. Since the early stage of AI research, various methods have been proposed to build agents that can plan efficiently and accurately (Newell and Simon, 1956; Russell and Norvig, 2010). The problem-solving procedure in these AI agents usually involves three steps: searching for possible action sequences, predicting their expected outcomes with an internal world model, and finding an action sequence to achieve the best expected outcome (Russell and Norvig, 2010; Mattar and Lengyel, 2022). This procedure
1Code and data are available at https://github.com/ OSU-NLP-Group/llm-planning-eval.
shares common traits with how large language models (LLMs) solve multi-step tasks, including mathematical reasoning (Wei et al., 2022), multi-hop question answering (Yao et al., 2023b), and code generation (Yang et al., 2023). At each step, an LLM searches for possible next actions and generates their language representations (generation). To evaluate the actions, the LLM utilizes itself or another LLM to predict the outcomes of actions, in the form of rewards or correctness (discrimination). Afterwards, it incorporates the outcomes into its problem-solving process with some strategy to find the best action sequence (planning).
Motivated by the similarity, we critically examine how LLMs solve multi-step tasks from a language-agent view. We unify different problemsolving procedures of LLMs into an agent framework (Figure 1) consisting of a generator, a discriminator, and a planning method. Under this framework, we investigate the practical utility of more advanced planning methods, such as tree search, in comparison with simpler methods (e.g. re-ranking). We hypothesize that the discriminator may be a deciding factor and systematically investigate two research questions: (RQ1) How does discrimination accuracy affect the performance of language agents using different planning methods? (RQ2) Can LLM-based discriminators correctly assess language agents’ actions in practical settings?
13659
To this end, we analyze LLMs’ discrimination abilities and their impact on three categories of planning methods: re-ranking, iterative correction, and tree search. We comprehensively evaluate these methods on two real-world tasks, text-to-SQL parsing and mathematical reasoning, with opensource, proprietary, and fine-tuned LLM discriminators. First, we use oracle environmental information to simulate discriminators with different levels of accuracy. The simulation experiments exhibit a strong correlation between discrimination accuracy and overall task performance among all three types of planning methods. Then, in a non-oracle setting, we closely investigate the LLM-based discriminators and show how environmental observations can effectively improve them. Finally, we conduct endto-end evaluations of the discriminators and planning methods to verify and strengthen our findings. In summary, our experiments show that: (1) Advanced planning methods, i.e., iterative correction and tree search, demand highly accurate discriminators (≥ 90% accuracy) to achieve decent improvements over the simpler method, re-ranking. (2) Using environmental feedback, we improve the discrimination accuracy of LLMs by up to 30.2 and 8.4 absolute points on text-to-SQL parsing and mathematical reasoning, respectively. Yet, our endto-end evaluations suggest they have barely met the need for advanced planning methods to show significant improvements over re-ranking. (3) Meanwhile, advanced planning methods may not adequately balance accuracy and efficiency when using LLM-based discriminators. In our experiments, compared to the other two methods, tree search is at least 10–20 times slower but leads to negligible performance gains. This accuracyefficiency trade-off can impede the deployment of tree search in real-world applications.

heading: 2 Related Work
text: A lot of recent research efforts have focused on advanced planning methods for improving the multistep problem-solving abilities of LLMs (Li et al. 2023b; Madaan et al. 2023; Yao et al. 2023a,b; Zhou et al. 2023; Feng et al. 2024; inter alia). Despite different designs, all these methods use a discriminator to evaluate the agents’ actions, or planning steps. In fact, instead of planning methods, an agent’s discriminator could be the more critical component. Since incorrect outcome predictions could lead to suboptimal plans, discriminators may decide the performance of an agent, regardless of
its planning method (Mattar and Lengyel, 2022). While it is commonly believed that discrimination is easier than generation for human and AI agents (Gu et al., 2023), West et al. (2024) pose the hypothesis that state-of-the-art generative AI models, including LLMs, may not have discrimination abilities matching their generation abilities. This hypothesis coincides with the findings of Huang et al. (2024) and Wang et al. (2023a) that, without any external feedback or with obviously absurd feedback, LLMs may recognize some of their selfgenerated correct plans as wrong. Huang et al. (2024) also note that the performance gains of selfcorrection, a kind of iterative correction method, may rely on some high-quality external feedback, such as checking ground-truth labels or test sets for planning loop termination. However, such external feedback usually does not exist in practical applications because solutions to new problems are unknown, and annotating comprehensive test cases can be nontrivial and costly.
Distinct from these existing studies, our work focuses on studying the relationship between discriminators and planning methods, including but not limited to self-correction, and attempts to improve LLMs’ discrimination capability. Our findings can provide useful guidelines for choosing planning methods and implementing language agents in practice. In light of our findings, we encourage future research to thoroughly evaluate language agents with various practical, non-oracle discriminators. We also advocate that improving LLM-based discriminators is an important future direction to enhance agents’ accuracy and efficiency when using advanced planning methods.

heading: 3 Our Framework
text: As shown in Figure 1, we systematically analyze different planning methods in a unified generatordiscriminator framework. Our framework consists of a generator that proposes (partial) action sequences, a discriminator that evaluates the outcomes of these actions, and a planning method that ranks the actions according to their outcomes and manages the interaction between the two models. In this section, we describe each of the three components and how they are instantiated on text-to-SQL parsing and mathematical reasoning (Section 4.1).

heading: 3.1 Generator
text: For each planning step, we prompt the generator to sample action sequences (SQL queries or Python
programs for math reasoning). For text-to-SQL parsing, we use 1-shot prompting, where the example is retrieved from the training sets using BM25 (Robertson and Zaragoza, 2009). For math reasoning, we use a fixed 2-shot prompt adapted from Ni et al. (2023b). See prompts in Appendix D.

heading: 3.2 Discriminator
text: Given some (partial) action sequences, we formulate the discrimination task as binary question answering (Kadavath et al., 2022; Ke et al., 2023). The discrimination score of each tested example is the probability of “Yes” being generated as the next token. Specifically, we prompt the LLMs with the question “Is the SQL/python program correct given the utterance/problem?” to generate one single token with its probability as the score. With this formulation, we evaluate three types of LLMs in our experiments (Section 4.2). Similar to the generator, we use 1-shot prompting with BM25 retrieval for text-to-SQL parsing and a fixed 2-shot prompt for math reasoning. Prompts are in Appendix D.

heading: 3.3 Planning Methods
text: Re-ranking. Re-ranking is a straightforward planning method. After sampling a few complete action sequences from the generator, it uses the discriminator to score them and return the highest-scoring plan (Figure 2a). Although simple, it is commonly used for code generation (Ni et al., 2023a) and mathematical reasoning tasks (Wang et al., 2023b; Li et al., 2023b). We consider re-ranking as a baseline planning method for more advanced ones. Iterative correction. Like re-ranking, iterative correction starts with the generator proposing a complete action sequence. Then it leverages multiple rounds of revision to improve the initial plan based on the discriminator’s feedback (Figure 2b). When the generator and the discriminator are the same LLM, it becomes a prevalent planning method, selfcorrection (Madaan et al., 2023; Shinn et al., 2023; Yao et al., 2023b; Chen et al., 2024).
While some work uses greedy generation, our
implementation samples the same number of action sequences as other planning methods for fair comparison. Then, it uses the discriminator to select the best-scoring one for the next round’s revision. We allow up to 10 rounds of corrections, with early exiting when the best plan meets a threshold of discrimination score (> 0.99), or the score is not improved for 3 consecutive iterations. For fair comparison, we prompt the generator to revise plans with 0-shot instruction following (Appendix D) instead of few-shot, since in-context examples may introduce additional information. Tree Search. Tree search is another popular planning method for language agents, such as MonteCarlo Tree Search (Chaffin et al., 2022), Pangu (Gu et al., 2023), RAP (Hao et al., 2023), Tree of Thoughts (Yao et al., 2023a), and LATS (Zhou et al., 2023). It uses a memory structure (e.g., a heap) to store observed partial action sequences and their scores. For each iteration, it prompts the generator for possible next steps of the current best partial plan, calls the discriminator to evaluate the steps, and updates the memory with new plans and scores (Figure 2c). Our tree search implementation is a kind of MCTS (Zhang et al., 2023): (1) Selection: Find the highest scoring partial plan in the memory, implemented as a heap structure. (2) Expansion: Prompt the generator for the next step of this partial plan. We follow recent work to define a step to be a SQL clause (Chen et al., 2023c) or one line of Python code (Bui et al., 2022), which is semantically more meaningful. (3) Simulation: Reuse the generator to complete the partial plans as Monte-Carlo simulations. (4) Evaluation: Evaluate the simulations with the discriminator. The score for each new step is the maximum score of all simulations starting from it. (5) Backpropagation: Update the partial plan with the new step and score (if higher) and insert them into the heap memory. After the update, if there is a complete plan in the heap memory, we terminate the tree search and return this plan.

heading: 4 Experimental Setup

heading: 4.1 Tasks and Datasets
text: Text-to-SQL Parsing. Text-to-SQL parsing is a code generation task of mapping natural language utterances to SQL queries. It requires agents to ground utterances to database environment and generate multi-step plans as SQL queries, making it an appropriate testbed in our study. To evaluate language agents’ potential for text-to-SQL parsing, we adapt two widely used datasets, Spider (Yu et al., 2018) and Bird (Li et al., 2023a).
We use the entire training split in each dataset to prompt or fine-tune LLMs.2 For evaluation, due to resource and budget constraints, we randomly select 400 and 300 development set examples in Spider and Bird, respectively. We also note that model performance may be lower on our evaluation sets because we uniformly sampled examples from each difficulty level, while the original development sets have skewed distributions towards easier examples (Appendix A.1). Mathematical Reasoning. Mathematical reasoning is a common task for evaluating language agents’ multi-step reasoning and planning capabilities. With 500 random examples from GSM8K’s development set (Cobbe et al., 2021), we follow program of thoughts (Chen et al., 2023b) to test the agents’ ability to plan in Python programs and solve these grade school math word problems.

heading: 4.2 Models
text: In all experiments, we use CodeLlama-13BInstruct as the generator in our framework. We also evaluate three kinds of LLMs as the discriminator: (1) open-source LLMs: CodeLlama-7B-Instruct and CodeLlama-13B-Instruct (Rozière et al., 2024), (2) proprietary LLMs: GPT-3.5-Turbo (OpenAI, 2022) and GPT-4-Turbo (OpenAI, 2023), and (3) fine-tuned LLMs: CodeLlama-7B-Instruct-FT and CodeLlama-13B-Instruct-FT. For brevity, we will omit “Instruct” in model names.

heading: 4.3 Implementation Details
text: Prompting the Generator LM. We prompt CodeLlama-13B with temperature-based sampling for different programs as action sequences (Appendix D). We use the model checkpoint and generation function implemented by HuggingFace
2In Bird, we exclude training examples for one database, retail_world, due to annotation errors.
(Wolf et al., 2020). We set the maximum generation length (max_length) to 300, temperature (temperature) to 0.6, and number of samples (num_return_sequences) to 5. Prompting Discriminator LMs. For CodeLlama7B and CodeLlama-13B, we simply feed them the input prompt (Appendix D) to get the last logit’s values, which give us the token-level probability of “Yes” after applying the softmax function.
For GPT-3.5-Turbo (gpt-3.5-turbo-1106) and GPT-4-Turbo (gpt-4-1106-preview), we access them through the API of OpenAI (2022, 2023). We prompt the LLMs to generate one token and leverage the top_logprobs request to check the top-5 tokens and their probabilities.3 If “Yes” appears as one of the top-5 tokens, we take its probability p without any modifications. If “Yes” is missing and “No” appears as one of the top-5 tokens, we inverse its probability 1− p as the score. If both tokens are missing, our implementation returns 0, though this case should be rare in our experiments. Training Discriminator LMs. To get CodeLlama7B-FT and CodeLlama-13B-FT, we again use the checkpoints and trainer implemented by HuggingFace. We fine-tune the models with LoRA (Hu et al., 2022) to classify the correctness of a given program by generate one token: “Yes” or “No”. Our training uses the following hyperparameters:
• Number of epochs: 1 • Batch size: 128 • Learning rate: 1e-5 • Warmup ratio: 3% • Scheduler: cosine
The inference procedure of fine-tuned models is the same as how we prompt the pre-trained LLMs, but without using any in-context example. Computing Resources. All of our experiments on Spider and GSM8K use up to four NVIDIA RTX A6000 GPU (48GB). Experiments on Bird use up to four NVIDIA A100 Tensor Core GPU (80GB).

heading: 4.4 Evaluation
text: Intrinsic Evaluation. We measure the discrimination abilities of LLMs with four intrinsic metrics. (1) Discrimination accuracy (Acc): Given a pair of correct and wrong programs, we calculate the percentage where the correct program obtains a higher discrimination score than the wrong one
3https://platform.openai.com/docs/ api-reference/chat/create#chat-create-top_ logprobs
(Bai et al., 2022; Touvron et al., 2023). (2) Classification macro F1 (F1): We treat “correct” and “wrong” as two classes and compute the macro average of F1 scores on these two labels. (3) Hit@1 (H@1): Given a batch of candidate programs, we calculate the percentage where the highest scoring candidate is correct. (4) Mean reciprocal rank (MRR): We compute the standard MRR score by the highest-ranking correct program in the batches. End-to-End Evaluation. To show the impact of discriminators, we evaluate language agents’ endto-end performance using our three planning methods, with execution accuracy for text-to-SQL parsing and answer accuracy for math reasoning.

heading: 5 Simulation Experiments with Oracle

heading: 5.1 Oracle-Based Discriminator
text: To investigate how discrimination accuracy affects the overall performance of language agents using different planning methods (RQ1), we utilize oracle environmental feedback to simulate a discriminator with controllable accuracy. For text-to-SQL parsing, we compare the first five rows in the execution results of predicted and gold SQL queries and calculate their table cell overlaps (Appendix A.4). For mathematical reasoning, we compare the predicted Python programs’ answers with the ground truth.
We use a probability-based threshold τ to control the accuracy of each simulated discriminator (Gao et al., 2022). When evaluating each plan, the discriminator first computes a score s with oracle
information. Then, it uses a random function to generate a number p ∈ [0, 1). If p < τ , the discriminator returns the score s. Otherwise, it returns an inverted score 1− s. In this way, we ensure that the discriminator’s accuracy is at most τ .

heading: 5.2 Results and Analysis
text: As shown in Figure 3, discrimination accuracy closely correlates with the performance of agents on all three datasets, no matter which planning method is used. For instance, the performance of re-ranking agents improves linearly as we increase the discrimination accuracy threshold, setting up a strong baseline for agents using other planning methods. We also note that it takes around 80% discrimination accuracy for all agents to outperform greedy generation on text-to-SQL parsing, demonstrating the task’s difficulty. To answer RQ1, we further analyze the performance of agents using iterative correction and tree search as follows:
Advanced planning methods demand highly accurate discriminators. For iterative correction agents, their performance usually cannot distinguish from the re-ranking baselines until we maximize the threshold τ = 1.0 (Figure 3). This finding resonates with Huang et al. (2024) that high-quality feedback may be the key to the success of iterative correction. More interestingly, tree search agents consistently underperform the other two when the discrimination accuracy threshold τ ≤ 0.8. Moreover, when raising the threshold to 0.9, we observe a sharp increase of their performance, with which
they start to beat other kinds of agents.
Advanced planning methods may not adequately balance accuracy and efficiency. By calculating the average inference time per example (Figure 3), we find that our implementation of tree search is at least 10–20 times slower than the other two planning methods, mainly due to frequent generation of Monte-Carlo simulations (Zhang et al., 2023). While we can remove the simulations to be more efficient and evaluate partial plans, in our preliminary study, we find LLMs would struggle in this setting. This accuracy-efficiency trade-off may hinder real-world applications of tree search methods. Meanwhile, the inference time for iterative correction increases as the accuracy threshold is raised, suggesting more iterations are required to derive a correct answer (Appendix C). This indicates that developing efficient and accurate planning methods remains a key problem for AI agents.
Monte-Carlo tree search can be unstable, especially in the early stages. We observe that iterative correction outperforms tree search on Bird (Figure 3b) when the accuracy threshold is 1.0. This observation may be caused by the instability of Monte-Carlo tree search. We first note that McNemar’s test finds no difference between iterative correction and tree search (p > 0.05), despite their performance gap (29.3 vs 32.7). The rationales are discussed in Appendix B. Furthermore, we analyze all 25 examples of which iterative correction derives the correct answer but tree search fails. In 12 out of the 25 examples (48%), tree search fails to select the correct partial plan when the discrimination scores are the same. Especially, this can happen in the early stages of tree search, where a correct program has not yet been discovered and all the steps receive a score of 0 from the oracle discriminator. Thus, we consider this underperformance a consequence of search instability.

heading: 6 LLM-Based Discriminators
text: While we have shown that iterative correction and tree search work well with oracle discriminators, it remains unclear whether LLM-based discriminators can correctly assess language agents’ actions (RQ2). To answer this question, we leverage generator outputs in the simulation experiments and re-label them with ground-truths to evaluate the LLMs’ discrimination accuracy (Appendix A.2).

heading: 6.1 Naive Discriminators
text: As Table 1 shows, most open-source LLMs have mediocre discrimination abilities. To improve their accuracy (Arora and Kambhampati, 2023; Zhu et al., 2023), we further fine-tune the LLMs to classify the ground truth plans and incorrect ones sampled from the generator (Appendix A.3). After fine-tuning, CodeLlama-13B-FT could reach the same level of performance as GPT-3.5. In comparison, proprietary LLMs exhibit stronger discrimination abilities, with GPT-4 achieving the best performance across all three datasets. Nonetheless, due to its high cost, we will use GPT-3.5 as the representative proprietary LLM in our experiments.

heading: 6.2 Observation-Enhanced Discriminators
text: To improve LLMs’ discrimination abilities, we conduct an error analysis for CodeLlama-13B on its worst-performing intrinsic evaluation set, Bird. We sample 50 pairs of SQL queries from the Bird intrinsic evaluation set with incorrect predictions. In 25 of the 50 pairs (50%), CodeLlama-13B assigns a higher score to non-executable SQL queries. Consequently, no matter using which planning method, language agents could hardly perform well with such discriminators.
Motivated by our error analysis, we first propose to add a program executability check as a safeguard for LLMs. If a program is non-executable, our discriminator would discard LLMs’ score and return
EObservation-enhanced discriminators. ∗Statistically significant (p < 0.05; McNemar’s) compared to re-ranking with the same discriminator on each dataset. We only observe such improvement with the oracle discriminator.
0. Otherwise, it returns the original LLM score. Besides executability check, we incorporate the execution results of predicted programs (first 5 table rows of SQL queries or answer of Python program) into the in-context examples and fine-tuning data (Ni et al., 2023a). If a program is non-executable, we use ERROR to represent its execution result.
Evaluation results (Table 2) show that these two non-oracle environmental observations can effectively improve LLMs’ discrimination accuracy. Enhanced with environmental observations, CodeLlama-13B can obtain up to 25.4, 30.2, and 8.4 points absolute accuracy gain on Spider, Bird, and GSM8K, respectively. For the other two models, we also observe significant gains compared to the naive discriminator baseline. Such notable improvements also highlight the importance of filtering out non-executable programs, or invalid plans, during planning.

heading: 7 End-to-End Evaluation
text: While we have evaluated their discrimination abilities with a fixed test set, to answer RQ2, we wonder if LLMs can correctly assess constantly changing sets of programs in actual planning processes. To this end, we evaluate the end-to-end performance of language agents with LLM-based discriminators and the three planning methods.

heading: 7.1 Text-to-SQL Parsing
text: As shown in Table 3, agents using naive LLMbased discriminators do not perform well on textto-SQL parsing. On Spider, the re-ranking agent using CodeLlama-13B-FT has the best accuracy (61.5), which is still lower than greedy generation (62.3) that requires no planning and is more efficient. On Bird, GPT-3.5-Turbo and re-ranking show an accuracy of 18.0, which is slightly higher than greedy generation (16.0). In addition to the mediocre performance, we find that when using naive discriminators, iterative correction and tree search consistently show worse or the same performance as re-ranking. These results mostly agree with our findings in previous experiments that (1) advanced planning methods need strong discriminators, and (2) naive LLM-based discriminators are not accurate enough.
After enhancing the discriminators with two environmental observations (Section 6.2), we effectively improve the agents’ performance without any modifications to the generator or the planning methods. In 5 of the 6 experiments, CodeLlama-13BFTE results in the best execution accuracy among all discriminators. It also leads to the overall best performance on Spider with re-ranking (70.3) and on Bird with iterative correction (26.3), showing the effectiveness of fine-tuning LLMs for discrimination and using environmental observations.
While the performance gains are satisfying, our implementation also takes the latency issue into consideration (Section 5.2). For instance, on some BIRD databases with 10K or more rows, the runtime of a single SQL execution can take more than 10 minutes. To mitigate this issue, we speed up the executions by fetching only the first 5 rows. Additionally, we implement a 60-second timeout interruption for each SQL execution, which aligns to the limit in the official evaluation script.
It turns out that the environmental observations help to reduce the end-to-end latency (Table 4), especially for tree search. We think executability check is the main reason for this latency improvement. Since compilers and interpreters are heavily optimized, the check itself does not take much time. By quickly identifying incorrect programs, it helps to prune the search space, thus reducing the overall latency of re-ranking and tree search. For iterative correction, the latency is increased when using GPT-3.5-TurboE and CodeLlama-13B-FTE as discriminators. This is because these two discriminators are more accurate and allow iterative correction to run more planning loops, as stated in Section 5.2 and Appendix C.

heading: 7.2 Mathematical Reasoning
text: The most interesting result in mathematical reasoning evaluation (Table 5) is the failure of iterative correction with naive discriminators. When prompting the generator CodeLlama-13B for 0- shot correction, it would disregard the instruction to “generate a fixed python program” (Appendix D), copy the program to be modified, and generate explanations and correction steps in natural language. Such natural language steps, usually having some lexical overlap with the math problem, would increase the discrimination score of LLMs while being non-executable. As a result, our iterative correction agent only has 10.2 answer accuracy when using CodeLlama-13B to evaluate its own generation. While this issue also exists when using GPT-3.5-Turbo as the discriminator, it is less severe because GPT would sometimes assign a high score (> 0.99) to the initial Python program. These scores trigger an early exit condition in iterative correction (Section 3.3) and stop the agent from calling the generator to add any natural language, thus avoiding the issue. These findings echo related analysis on self-correction (Stechly et al., 2023; Valmeekam et al., 2023; Huang et al., 2024).
With an executability check, enhanced discrim-
Discriminators Re-ranking Iter. Correct. Tree Search‡
inators help mitigate this issue in iterative correction, which now achieves better performance (42.2 and 48.4) than greedy generation (39.4). Overall, the tree search agent using GPT-3.5-TurboE achieves the best answer accuracy. Nevertheless, McNemar’s test finds no difference (p > 0.05) between the performance of re-ranking (47.6) and that of iterative correction (48.4) or tree search (51.0).

heading: 7.3 Analysis
text: To better understand the end-to-end evaluation results, we conduct an in-depth analysis of examples where re-ranking returns the correct program, but iterative correction or tree search does not (Table 6). Specifically, we analyze cases of the strongest discriminators, CodeLlama-13B-FTE for text-toSQL parsing and GPT-3.5-TurboE for mathematical reasoning, and divide them into two kinds of errors. (1) Discrimination error: The discriminator assigns a higher score for wrong programs than correct ones, which is not recoverable by any planning method. (2) Exploration error: The planning method has not found the correct program before termination. Our analysis suggests that:
LLM-based discriminators have not yet met the needs of advanced planning methods. Across all datasets, 50% or more discrimination errors are observed in each planning method. On Spider, the number of such errors in iterative correction is as large as 29 out of 37 (78.4%). In fact, among the 29 errors, iterative correction has already found the correct SQL queries for 15 (40.5% of the total 37 errors) of them. However, not only does the discriminator fail to trigger early exits, but it also assigns a higher score for wrong SQL queries in new iterations. Consequently, these erroneous SQL queries override the originally correct ones, leading to an overall performance drop. The same issue is also serious in tree search. When an incorrect partial program receives a high discrimination score, tree search will commit to it and hardly explore other possibilities, including the correct partial programs. Such discrimination errors usually cannot be recovered by the planning methods themselves, unless they find another correct program with even higher scores. This finding also demonstrates that determining early exits using oracle information in iterative correction may introduce a larger benefit than previously thought (Huang et al., 2024).
Advanced planning methods need more thorough exploration. For the remaining cases, we observe that advanced planning methods have not found a correct program before terminating, which we call exploration errors. This kind of error circles our discussion back to the accuracy-efficiency trade-off mentioned in our simulation experiments with oracle (Section 5.2). Indeed, we can extend the exploration of planning methods in various ways, such as loosening termination conditions, increasing the number of generation samples for each step, and adjusting some hyperparameters for more diverse program samples. Yet, all these adjustments can slow down the planning methods and reduce the language agents’ efficiency. Additionally, we note that these strategies may not always result in better performance, as the discriminators may give unseen wrong programs a higher score.
For these reasons, iterative correction and tree
search cannot gain decent improvement over reranking with the same LLM-based discriminator. On text-to-SQL parsing, tree search even shows worse performance than re-ranking when using CodeLlama-13B-FTE (Table 3: 67.5 vs 70.3 on Spider; 21.7 vs 23.7 on Bird). More surprisingly, on GSM8K, advanced planning methods may not perform much better than re-ranking even with the oracle discriminator (p > 0.05; McNemar’s). Admittedly, some of the performance gains appear considerable, but McNemar’s tells us there are still decent chances of the simpler agent outperforming a more complex one (Appendix B).

heading: 8 Conclusions
text: This paper presents a thorough investigation into the relationship between discrimination accuracy and performance of planning methods in language agents. Through comprehensive experiments on text-to-SQL parsing and mathematical reasoning, we find that: Discrimination accuracy strongly correlates with the overall performance of language agents using different planning methods and also affects their efficiency (answer to RQ1). LLM-based discriminators can correctly assess a decent number of language agents’ actions with their environmental observations, but they are still not accurate enough for advanced planning methods (answer to RQ2). Future research should investigate the development of more accurate discrimination models for language agents, e.g. by improving their grounded understanding of execution results beyond error signals.
Limitations
Experiments with Other Models. In this study, we focus on studying the generation and discrimination of instruction-tuned LLMs that have seen code data during pre-training. This consideration is because: (a) They may have better in-context learning performance on our two tasks, text-to-SQL parsing and mathematical reasoning with programof-thought (Ni et al., 2023b); (b) We want to lever-
age their 0-shot instruction following capabilities in iterative correction for fair comparisons with other planning methods; (c) For GSM8K problems, LLMs tend to generate natural language plans instead of programs with 2-shot prompting, and some instructions other than in-context examples help to mitigate this issue. Future research may extend our study to other LLMs of code and conduct an ablation study of instruction-tuning’s impact on models’ discrimination accuracy. Experiments with Natural Language Plans. Our study focuses on the generation and discrimination of formal language plans, i.e., programs, as they can directly interact with the environment. Although feasible for mathematical reasoning (Wei et al., 2022), natural language plans require another semantic parsing step to convert them into actions defined in the corresponding environment, which may introduce intermediate errors and add noise to our analysis. Therefore, we conduct the experiments with formal language plans using LLMs trained on code data. As a future direction, it would be interesting to extend our study to natural language plans and see how the intermediate semantic parsing step would affect the overall performance of agents for mathematical reasoning. Impact of Generators on Planning Methods. While our work focuses on studying the relationship between different discriminators and planning methods, we acknowledge that the generator can also actively affect different planning methods. For example, we can transform the generator’s perplexity into a probability and multiply it by the discriminator’s score. We exclude such uses of the generator because in our preliminary experiments, we find that incorporating its perplexity leads to mixed results. These results make it even harder to analyze how language agents behave when using different planning methods. Thus, we exclude the generator to have a clear picture of how discriminators can affect planning methods. Nevertheless, it is worth studying the generator’s impact on planning methods in future work.

heading: Acknowledgements
text: We would like to thank colleagues from the OSU NLP group for their thoughtful comments. This research was supported in part by a sponsored award from Cisco Research, NSF IIS-1815674, NSF CAREER #1942980, NSF OAC-2112606, and Ohio Supercomputer Center (Center, 1987). The views
and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.

heading: A More Implementation Details
text: A.1 Text-to-SQL Parsing Evaluation Sets For text-to-SQL parsing, we sub-sample the development splits of each dataset, Spider and Bird, following three steps: (1) categorize development set examples by difficulty levels defined in each dataset, (2) randomly select a database and choose one associated example, and (3) repeat step 2 until we have 100 samples for each difficulty level. In this way, we ensure a uniform distribution across different difficulty levels and database. Since there are four and three difficulty levels in Spider and Bird, respectively, our evaluation sets have 400 and 300 examples for each dataset.
Text-to-SQL parsing models, including LLMs, may show lower performance on our evaluation sets because of their uniformly distributed difficulty (100 examples per level). In comparison, the original datasets have skewed distributions towards easier examples. Spider’s development set has 248
(24.0%) examples at easy level and 446 (43.1%) examples at medium level, while the hard and extra hard examples only sum up to 32.9 % of the 1,034 examples. In Bird, 925 out of the 1,534 (60.3%) development set examples are at simple level, 465 examples (30.3%) are at moderate level, and only 144 examples (9.4%) are at challenging level. Our evaluation sets normalize these skewed distributions and make the macro averages of model performance less biased (Section 4.1).
A.2 Intrinsic Evaluation Data To evaluate LLMs’ discrimination performance, we reuse the generation results from our oraclesimulation experiments (Section 6). Specifically, we use the evaluation scripts to re-label the generated programs in simulated re-ranking experiments (accuracy threshold τ = 1.0). Then, we construct our intrinsic evaluation sets based on the relabeled programs (Table A.1). Intuitively, the number of program batches for each dataset is the same as the end-to-end evaluation examples we have, and the the number of programs is all unique programs we can get from the batches. To pair the programs and calculate discrimination accuracy, we iterate through each batch and enumerate combinations of correct and wrong programs within the batch. We do not include cross-batch pairs, as those do not align with our end-to-end evaluation settings.
For discrimination accuracy, we enumerate pairs of correct and wrong programs and ask LLMs to select the better one. For classification F1, we let LLMs predict the correctness of each individual program. For Hit@1 and MRR, we use LLMs to score the batches of programs in simulation experiments.
A.3 Data for Discriminator LMs For text-to-SQL parsing, we perform 2-fold crossvalidation on the training sets to synthesize incorrect SQL queries for each example (Chen et al., 2023a). We prompt the LM using one pair of correct and wrong SQL queries (labeled with “Yes” and “No”), also retrieved by BM25 (Section 3.2). Alternatively, we fine-tune the LM on the entire training set with ground-truth and synthesized SQL
queries to generate “Yes” or “No.” For mathematical reasoning, we annotate two incorrect python programs for the two examples used in generator. Similar to text-to-SQL parsing, we use the two program pairs to prompt LMs for binary question answering. Since the training set of GSM8K is not annotated with program of thoughts, we are not able to fine-tune LMs on this dataset.
A.4 Implemendation of Oracle Discriminator For text-to-SQL parsing, our oracle uses the first five rows in execution results of the predicted and gold SQL query and calculate the table cell overlap. More specifically, the calculation is similar to span F1 in machine reading comprehension. Our oracle function first compares each row in the execution results head-to-head under a strong assumption that the rows are ordered. Although strict, this assumption is helpful for evaluating the correctness of SQL queries with an ORDER BY clause. Then, the function count how many table cells overlap with each other in an unordered manner. We divide the number of overlapping cells by the total number of cells in execution results of the gold SQL query (precision) and the predicted one (recall). Finally, we compute the harmonic mean of these two numbers to get the oracle score (F1).
For instance, given “-- countryid: 1, 2, 4, 5 -- countryname: usa, germany, japan, italy” as the gold execution result and “-- countryid: 1, 4, 6 -- countryname: usa, japan, japan” as the result of predicted SQL query. We compare (1, usa), (4, japan), and (6, japan) the first, second, and third row in the gold result, respectively. They have 2, 0, and 1 overlapping table cells, respectively. Thus, we have our precision to be 3/8 = 0.375 and recall to be 3/6 = 0.5. The oracle’s score would be:
2 · 0.375 · 0.5 0.375 + 0.5 = 0.43
For mathematical reasoning, our oracle directly checks if the predicted answer equals to the groundtruth. If the answer is None (non-executable program) or does not equal to the ground-truth, it returns 0. Otherwise, it returns 1.

heading: B McNemar’s Test for Statistical
text: Significance
We measure the statistical significance of performance gains using the exact McNemar’s Test4 (McNemar, 1947). We choose the test’s exact binomial version because our sample sizes are relatively small (Edwards, 1948), and the first two significant digits of p-values are the same for this binomial version and the original chi-square version in our tests. Intuitively, this test measures how likely the weaker method can still outperform the stronger one.
For example, we consider the comparison between tree search and iterative correction on Bird when using CodeLlama-13B-FTE as the discriminator (Section 5.2). By computing a 2× 2 contingency table (Table B.1), McNemar’s Test focuses on the 40 examples where only one of the two method have predicted correctly. Specifically, there are 25 examples that iterative correction finds the correct answer, but tree search does not, which is the source of performance gain. Also, there are 15 examples that iterative correction fails, but tree search succeeds. According to McNemar’s Test, these 15 (37.5% of the total 40) examples result in a p-value of 0.15, meaning there is still some chance for tree search to outperform iterative correction.
In contrast, suppose there are only 10 examples that iterative correction finds the correct answer, but tree search does not. Meanwhile, there are no examples that iterative correction fails, but tree search succeeds. Then, we can still observe the same number of accuracy gain, but it is now statistically different because it is almost impossible for tree search to outperform iterative correction (0 out of 10). The same rationale also applies to the results of other tests in Section 7.
4https://www.statsmodels.org/dev/generated/ statsmodels.stats.contingency_tables.mcnemar. html

heading: C More Analysis on Simulation Experiments with Oracle
text: In Section 5.2, we conclude that “the inference time for iterative correction increases as the accuracy threshold is raised, suggesting more iterations are required to derive a correct answer.” Though counter-intuitive, we verify that this conclusion is correct because an accurate discriminator would avoid triggering early exit by mistake in iterative correction.
To give a concrete example, we compare the number of iterations needed on Spider when using oracle-based discriminator with accuracy threshold 0.6 and 1.0. When the threshold is 0.6, the planning method takes 1.275 iterations on average. When the threshold is 1.0, the planning method takes 1.725 iterations on average, 1.35 times more than the previous case. This roughly aligns with the increase in average inference time: When the threshold is 0.6, iterative correction takes an average of 21.5 seconds per example. When the threshold is 1.0, the average inference time is 27.9, which is 1.29 times more.
This counter-intuitive observation is mainly caused by the early exit of iterative correction methods. In other words, if the discrimination score is high enough (>0.99) or is not improved for 3 consecutive iterations, the planning method would stop making more corrections (Lines 214–217). When the discriminator is not accurate, it would (a) give high scores to wrong plans or (b) keep assigning the same or even lower scores to improved plans. As a result, the planning method would stop quickly without finding a better plan. On the other hand, an accurate discriminator would predict less extreme scores and increase them for even small improvements in the plan, thus allowing the method to iterate 3 more times. This results in the increased inference time per example.

heading: D Prompt Examples

heading: Future Work
text: Future research should investigate the development of more accurate discrimination models for language agents, e. g. by improving their grounded understanding of execution results beyond error signals. Limitations
Experiments with Other Models. In this study, we focus on studying the generation and discrimination of instruction-tuned LLMs that have seen code data during pre-training. This consideration is because: (a) They may have better in-context learning performance on our two tasks, text-to-SQL parsing and mathematical reasoning with programof-thought (Ni et al. , 2023b); (b) We want to lever-
age their 0-shot instruction following capabilities in iterative correction for fair comparisons with other planning methods; (c) For GSM8K problems, LLMs tend to generate natural language plans instead of programs with 2-shot prompting, and some instructions other than in-context examples help to mitigate this issue. Future research may extend our study to other LLMs of code and conduct an ablation study of instruction-tuning’s impact on models’ discrimination accuracy. Experiments with Natural Language Plans. Our study focuses on the generation and discrimination of formal language plans, i. e. , programs, as they can directly interact with the environment. Although feasible for mathematical reasoning (Wei et al. , 2022), natural language plans require another semantic parsing step to convert them into actions defined in the corresponding environment, which may introduce intermediate errors and add noise to our analysis. Therefore, we conduct the experiments with formal language plans using LLMs trained on code data. As a future direction, it would be interesting to extend our study to natural language plans and see how the intermediate semantic parsing step would affect the overall performance of agents for mathematical reasoning. Impact of Generators on Planning Methods. While our work focuses on studying the relationship between different discriminators and planning methods, we acknowledge that the generator can also actively affect different planning methods. For example, we can transform the generator’s perplexity into a probability and multiply it by the discriminator’s score. We exclude such uses of the generator because in our preliminary experiments, we find that incorporating its perplexity leads to mixed results. These results make it even harder to analyze how language agents behave when using different planning methods. Thus, we exclude the generator to have a clear picture of how discriminators can affect planning methods. Nevertheless, it is worth studying the generator’s impact on planning methods in future work