Enhancing Language Model Agents using Diversity of Thoughts
Authors: Vijay Lingam
Behrooz Omidvar Tehrani
Sujay Sanghavi
Gaurav Gupta
Sayan Ghosh
Linbo Liu
Jun Huan
Anoop Deoras
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Reasoning
Programming
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: A popular approach to building agents using Language Models (LMs) involves iteratively prompting the LM, reflecting on its outputs, and updating the input prompts until the desired task is achieved. However, our analysis reveals two key shortcomings in the existing methods: $(i)$ limited exploration of the decision space due to repetitive reflections, which result in redundant inputs, and $(ii)$ an inability to leverage insights from previously solved tasks. To address these issues, we introduce DoT (Diversity of Thoughts), a novel framework that a) explicitly reduces redundant reflections to enhance decision-space exploration, and b) incorporates a task-agnostic memory component to enable knowledge retrieval from previously solved tasks—unlike current approaches that operate in isolation for each task. Through extensive experiments on a suite of programming benchmarks (HumanEval, MBPP, and LeetCodeHardGym) using a variety of LMs, DoT demonstrates up to a $\textbf{10}$% improvement in Pass@1 while maintaining cost-effectiveness. Furthermore, DoT is modular by design. For instance, when the diverse reflection module of DoT is integrated with existing methods like Tree of Thoughts (ToT), we observe a significant $\textbf{13}$% improvement on Game of 24 (one of the main benchmarks of ToT), highlighting the broad applicability and impact of our contributions across various reasoning tasks.

Published as a conference paper at ICLR 2025
ENHANCING LANGUAGE MODEL AGENTS USING DIVERSITY OF THOUGHTS
Vijay Lingam'
Behrooz Omidvar Tehrani
Sujay Sanghavi
Gaurav Gupta
Sayan Ghosh
Linbo Liu' $Amazon
Jun Huant
Anoop Deoras
AASS Al Labs
vilingame amazon... com
omidvart amazon com
{sujayrs, gauravaz, sghoshne, linbol, Lukehuan, aderaass)aaa com
ABSTRACT
A popular approach to building agents using Language Models (LMs) involves iteratively prompting the LM, reflecting on its outputs, and updating the input prompts until the desired task is achieved. However, our analysis reveals two key shortcomings in the existing methods: (i) limited exploration of the deci- sion space due repetitive reflections, which result in redundant inputs, and (ii) an inability to leverage insights from previously solved tasks. To address these issues, we introduce Do (Diversity of Thoughts), novel framework that a) explicitly reduces redundant reflections to enhance decision-span exploration, and b) incorporates a task-agnostic memory component enable knowledge re- trieval from previously solved tasks-unlike current approaches that operate in isolation for each task. Through extensive experiments on a suite of program- ming benchmarks (HumanEval, MBPP, and LeerCodeHardGyym) using variety LMs, DoT demonstrates up to a 10% improvement in Pass while main- taining cost-effectiveness.. Furthermore, DoT is modular by design. For instance, when the diverse reflection module of DoT is integrated with existing methods like Tree of Thoughts (ToT), observe significant 13% improvement Game of 24 (one of the main benchmarks of ToT), highlighting the broad applicability and impact of our contributions across various reasoning tasks.
INTRODUCTION
Developing autonomous Al-based frameworks leveraging Large Language Models (LLMs) to solve challenging reasoning and decision-making tasks is active area of research. Recent research works have focused on utilizing different LLMs as the backbone to develop AI agents that can understand, reason, and reflect on their own actions when solving task. An Al agent typically per- forms iterative inference to effectively reason and solve a task by leveraging LLMs and knowledge of tools if available. Examples of such agentic architectures
ReAct (Yao et al., 2023b), Tree of Thoughts (ToT) (Yao et al. 2023a). and Toolformer (Schick et al. 2023). which have explored the in-context learning abilities of LLMs, proposing iterative inference-level algorithms to mimic human-like learning. Such agent architectures promote generalistic approaches to reasoning and learning, which differs from prior reinforcement learning- solutions introduced by Mnih et al. (2033) Le et al. (2022) that were tailored towards specific task and demanded extensive compute and data for training.
Building upon ReAct, Reflexion (Shinn et al., 2023) improved the reasoning capabilities in the agent by introducing "selff-rffecctioss" a mechanism to steer the model away from previous failed trajectories. Similarly, LATS (Zhou et al.] (223) extended these ideas by enabling multiple reasoning paths and employing search algorithms like Monte Carlo Tree Search (MCTS) to identify optimal solutions. These methods represent significant strides in enhancing LLMs' reasoning capabilities through an iterative feedback.

---

Published as a conference paper at ICLR 2025
Table 1: Self-reflection counts and output token usage on HumanEval subset, highlighting redundancy across prob- lems. "Ref" denotes reflections.
Table Pass@ I and cost com- parison for OpenAl ol the LeerCodeHardGym benchmark. DoT outperforms Reflexion.
Problem Name
Total Ref
Unique Ref
#Tookens
145 order by points
36
5129
Method
Pass@1
Cost
130 tri
14,484
Base
45
$12.75
129 minPath
84
10
12,227
Reflexion
62
$24.24
132.1 is.nested
9
4292
84.solve
96
13
10,329
DoT
72.5
$33.47
Despite these advancements, current frameworks suffer from few critical limitations: (i) poor explo- ration of the decision space, primarily due to repetitive reflections, and (ii) an inadequate memory mechanism. To depict these shortcomings,
revisited the self-reflections generated by Reflex- ion and LATS. Specifically, we analyzed the generated self-reflections by these two approaches for the HumanEval dataset (Chen et al., 202Ib). Upon careful inspection, we observed that sev- eral self-reflections were repetitive. We employed an LLM (GPT-4o) to identify clusters of similar self-reflections and we manually inspected a random sample (see details of prompt used in Ap- pendix A.3). Our analysis revealed that significant portion of the self-reflections generated by LATS were repetitive and redundant, leading to poor decision-space exploration and excessive to- ken usage. We present key statistics for representative subset of the HumanEval dataset in Table I Similar patterns were observed across other datasets.
Reflexion and LATS incorporate local memory component to store reflections and failed imple- mentations, using as additional context to improve subsequent generations. However, this memory reset before tackling the next task. In contrast, Didolkar et al. (2024) demonstrated that recent LLMs can leverage metacognitive abilities to apply prior insights to new tasks, achieving signifi- cant performance gains. Their method involves curating in-context examples based skill exem- plars-insights from related tasks-but relies on a predefined set of exemplars, which necessitates additional training data and limits adaptability across diverse tasks.
To address these limitations, we propose Diversity of Thoughts (DoT), a novel framework that enhances decision space exploration by (a) reducing redundant reflections and (b) integrating a task-agnostic memory component. DoT promotes diverse reasoning attempts using variety of thought/step sampling strategies, ensuring effective intermediate reflections to guide decision- making. The task-agnostic memory bank dynamically retrieves relevant in-context examples from previously solved tasks, boosting the model's 's ability to generate informed reasoning paths.
This approach not only reduces redundant computations but also significantly improves efficiency, resulting in more cost-effective solution. For example, on the LeetCodeHarddGym benchmark, DoT achieved 10% gain in Pass I (Claude Sonnet 3.5) with 4x cost reduction compared to LATS (see Section 3.2. and Table 6). Additionally, DoT demonstrated substantial improvements on OpenAl in preliminary study, outperforming both the base model and Reflexion (see Table 2
In summary, our main contributions in this paper are mentioned below:
We identify redundancy a critical limitation in the existing reflection- seaeection-based reasoning frameworks (e.g.. Reflexion, LATS), as shown anecdotally in [Figure and quantitatively in [aalle I] Additionally, current frameworks handle each task in isolation, missing the opportunity for cross-task knowledge transfer.
We propose (i) DoT, novel framework that promotes diversity in reasoning trajectories, and (ii) DoT-bank, which further extends DoT by incorporating similar trajectories from a task aask-agnostic memory bank to enhance decision space exploration.
Our extensive experiments across multiple code generation datasets (HumanEval, MBPP, LeerCoddeHardGym) and models demonstrate the effectiveness of DoT. We achieve state-of-the-art results, with up to 10% improvement in Pass I on LeetCodeHardGyym
2

---

Published as a conference paper at ICLR 2025
Roflection-22 23
Roflection226 24
Roflection-27
implementation falled
where the both postiive negative rumbers
ippeemetttiio
The implementation failed the case where the liat containa both pasitive and negutive numbbers
lst containe both postive negative numbers. The asue with the sorting key fanction, which absolute negative
seve is that the lanction sorta the numbers based on sum of their di gins and then their actall valun.
the sorting key function, which sons the rumbers finst by the sum thair and approach doesn't correctly handle negaiive numbers, treats them the positive counterparts. adjust the kiyy function properly handle negat third sorting criterion that prioritzes negatve numbers
ralcultes the of the digits salue number. to be treated the same as their positive funetion To for this, ahould modil the
value.
have the same digit sum out different sggs in such cases, the negative number chould come before the need modily account: agn number We can de this by atiding a thid orter on to furetion that checks if the number negativn. will negativn postive ones when they have the same digtt sum.
their
sorsing To
the sign number should conober the of numbeess in the originel list, the current implomentation
befor
the original digit..
numbers with
Examples of repetitions in generated self-reflections from LATS problem 145 order by points s in the HumanEval dataset. The words highlighted in red are redundant across Reflection 23, 24 and 27.
Figure I:
for Claude Sonnet 3.5, while reducing token costs by 4x compared to LATS and 1.4x more expensive than Reflexion.
METHODOLOGY
Problem Setup. We briefly review the standard LLM-based reasoning and decision making setup- We are provided with an input x in natural language, along with a pretrained LLM po(-) param- eterized by frozen learned parameters 0. The goal is to generate an output response y that cor- responds to solution for reasoning tasks
set of actions for decision making. Traditional input-output prompting (y ~ po (z)) leads to sub-optimal performance. Following the observa- tions made by Brown et al. (2020), flurry of prompting techniques were proposed, which prepend additional input text with specific instructions or few-shot input-output examples to input query æ. Incorporating such additional context helps improve reasoning and decision making performance. prompt,p zo (1) denotes a generic stage in the process of transforming the given input prompt x into output y as: y ~ po (prromppt ro (z)). An illustration of this process is shown Figure
System Message:
You are an AI that only responds with python code, NOT ENGLISH. You will be given a function. si signature and its docstring by the user. Write your full implementation (restate the function signature).
User Message:
def cube Sum cube_Sum int) int:
Write a python function to find the cube sum of first even natural numbers.
Figure 2: An example of prompt (z)) programming tasks. The User Message denotes in the above example, and the prompt (1) prepended System Message specific a task.
Preliminaries. Our framework's design is inspired by Reflexion (Shinn et al. 2023). which en- hances reasoning through iterative interaction with an external API or environment. Reflexion
volves three agents agents-actor actor (M,), evaluator (Me), and self-reflection (Mar) workingg cyclically until a termination condition is met. For code generation, the process is as follows:
(a) The actor M, receives input and generates an output (e.g., ., code snippet).
(b) The evaluator M, scores this output (e.g., the number of unit tests passed).
(c) If the score is low (i.e., code fails), the self-rew reff-reflection agent M,r diagnoses the issue, suggests
fix, and adds to the input context along with the failed action. This new input given to the actor, and the cycle repeats until success trial limit is reached.
3

---

Published as a conference paper at ICLR 2025
DoT Agent
trajectories
External Feedback
Diverse Reflections (LM)
diverse reflections
Sample posential candicate
Task Agnostic
Evaluator
Expertance (long-ter memory)
Memory Bank
Trijectory (short-lerm memory)
Actor (LM)
Retfieve similar trajectories
Reward Score
Environment
Action
Overview of DoT and DoT-bank 's architecture. We introduce a new diverse-reflections model mitigate redundancy in reflections, and task agnostic memory bank to enable knowledge transfer across tasks.
Figure 3:
As noted in Table [ in several cases, the generated self-reflections are repetitive and redundant, thereby hindering the effective exploration the decision space.
2.1 DoT: PROPOSED FRAMEWORK
DoT builds upon Reflexion by replacing the self-reflection model with novel diverse-reflections model, Mar. We extend this further with DoT-bank, which introduces a task-agnostic memory bank, MB. Thus, DoT comprises three models: the actor M, and evaluator Me (similar Re- flexion) and the diverse-reflections model Már- In DoT-bank, MB is additionally leveraged. We provide a detailed description of each component and the workflow that illustrates how these pieces are integrated. An illustration is provided in Figure 3]
Actor (M,): The actor powered by an LLM of choice with specific instructions to generate actions conditioned on the state observations; these actions could be snippets of text, or other actions as per the setting. For example when the task code generation, the input the actor task description, associated history etc. as text string, and the output action is code snippet.
Evaluator (Ma): Computes a reward score for the action generated by the actor. Depending on the task, Me can be external environment (e.g., Python interpreter), Likert scale rating (Yao et al. 2023a Zhou et al. 2023), or another LLM (Shinn et al. 2023 Zhou et al. (023). For example, in the setting of code generation, the reward can be measured by the number of visible or synthetic unit tests passed.
propose the Diverse-Reflections module, which generates l diverse reflections in one shot using an explicit prompt: 24 Po (2 & (z)) The exact prompt structure (DivIO(z)) (see Appendix A.4). Reflections from previous iterations are also included in the context. We inves- tigate alternative approaches for generating diverse reflections in Section 3.3.3 and find one-shot sampling to be the most effective in terms of both performance and cost. Recent work (Hayati et al., 2024) supports that one-shot sampling can produce semantically diverse outputs. Additionally, we introduce diversity metric in Section 3 to quantitatively demonstrate how our Mar module effectively reduces redundancy generated reflections.
Diverse-Reflections (Mar): To address redundancy,
detailed in Figure
Task-Agnostic Memory Bank (MB): Our second contribution is persistent, task-agnostic mem- ory MB that stores successful trajectories of the DoT agent. trajectory is deemed successful if passes all visible or synthetic tests. Each entry in MB is indexed by unique task d and consists of a complete task trajectory. When the actor M,, is invoked, we retrieve j relevant trajectories from MB and include them in its input context. As more tasks solved, MB grows, facilitating knowl- edge transfer between tasks. We evaluate our framework in two configurations DoT (without M B) and
(with M and analyze the impact of the number of retrieved examples per-
4

---

Published as a conference paper at ICLR 2025
Algorithm DoT and DoT bank Framework
Require: Dataset D, Retriever, Max Trials T maz
1: Initialize: Actor Ma. Evaluator M,, Diverse Reflections Generator Masei
2: Memory Bank MB 0, Failed Tasks F
3:
4: Phase 1: DoT: solving and constructing memory bank
5: for each task & D do
6: Generate т using M, and evaluate using Me.
7: if r passes evaluation then
8: Add r to MB
9:
else
0
11:
while not solved or i < Tmaa do
Generate & diverse reflections using Mase
13:
Generate k new trajectories using by feeding these reflections and 7 to Ma for each generated trajectory т; do
15:
Evaluate using Me and record score T. passes evaluation then
16:
17:
Add to B, break the while loop end if end for
18:
19:
20:
none of the | trajectories passed then
21:
Select trajectory random (weighted by the corresponding score)
22:
end if Incrementi
24:
end while
no trajectory passes after Tmaz trials then Add to F 27: end
26:
28: end if
29:
end for
30:
31: Phase 2: DoT-bank: reattempting failed tasks using memory bank
32: for each failed task t do
33: Retrieve J similar trajectories from M
34: M,' Inject retrieved trajectories into context of actor
35: Repeat Phase with M, and update M B if successful
end for
formance in Section The orchestration these agents is presented as DoT framework in Algo- rithm I Additional implementation details and discussion on MB are provided in Appendix A.]
2.2 RELATION TO OTHER METHODS
Tree-of-Though (ToT) Yao et al. 2023a) extends Chain-of-Thoughh (CoT) by exploring multiple reasoning paths over intermediate states (also referred to "thoughts"). It constructs tree, where each node contains [x, 21...] and represents partial solution. Thoughts can either be sampled proposed using CoT: algorithms such as BFS/DFS are employed, combined with a language model- evaluator that assigns value to each node.
Pa (x, 21). To select the final solution, classic search tree
LATS Zhou et al. 2023) unifies reasoning, acting, and planning under one framework. It replaces the cyclic workflow in Reflexion with Monte Carlo Tree Search (MCTS) for proficient exploration of the decision space. During tree exploration, each node is assigned a value, which combination of LM based evaluation and self-consistency. A noteworthy detail is that current action agnostic at a given level of the search tree to previous actions: of ~ Pa (s.). We refer the reader to Algorithm 1 in (Zhou al. 2023) for additional details.
convex

---

Published as a conference paper at ICLR 2025
3 EXPERIMENTS
In this section, we begin with an overview of the models and datasets used in experiments. We then describe the experimenta setup in detail and conclude with a discussion of the results.
Summary: We perform comprehensive evaluation of our proposed framework, DoT and DoT-bank, using a wide range of LLMs and benchmarks spanning reasoning and programming tasks. The key findings are: (i) DoT and DoT-bank consistently outperformed baselines like Reflexion and LATS in terms of accuracy across datasets like HumanEval, LeyyCodeHardGym, and MBPP, while being significantly more cost-effective than LATS (see Section 3.2.1p. (ti) The performance gains were observed across different L.LM architectures and model sizes, with larger improvements seen for less powerful models (see Table and Table 8p. (iti) Quantitative analysis (see Section 3.3.1 revealed that DoT generates diverse and non-repetitive self-reflections, leading to more effective exploration of the decision space. (ir) Embedding- based retrieval of few-shot ex- amples from a memory bank (DoT-bank) further boosted performance. (v) The diversity strategy introduced in DoT also applicable to other reasoning frameworks like Tree of Thoughts, leading to substantial gains challenging datasets like Game of 24 (see Section 3.3.4.
3.1 EXPERIMENT SETUP
Models: We conduct comprehensive evaluation of DoT using a diverse suite of both open-source LLMs (Llama-3.1 8B and 1 70B) et al. (2024) and proprietary LLMs (Claude Sonnet 3.57 GPT--3.5-Turbo, GPT-4, GPT-4o, GPT-44-minn" This diverse selection allows for a holistic assessment of our framework's performance across different architectures and model sizes. Our em- pirical results demonstrate that the effectiveness of our contributions remains consistent, regardless of the specific L.M being used.
In this work, we experiment on variety of tasks spanning the domains of reasoning and programming. Tableß an overview of the tasks and their corresponding eval- uation metrics. For each method, we also report the dollar cost of invoking the respective LLM APIs.
Datasets and Metrics:
Table 3: Datasets used for Programming and Reasoning tasks.
Task Type
Dataset Name
Size
Metric
Programming Programming
HumanEvval MBPP
164 problems, ~33 visible test cases/problem
Pass@1
397 sampled problems
Pass @
Programming LoyroodeHardGyy
40 uncontaminated problems
Pass
Reasoning
Game of 24
100 puzzles
0-1 Acc
To evaluate the programming capabilities of DoT and -bank,,,, we conduct experiments on Hu-
manEvalsChen et al. 202 MBPP (Austin et al. 2021), and LeetCodeHardGyym (Shinn et al. 2023) benchmarks, using Pass @ I as our primary metric. During solution generation, only visible or synthetic test cases are used ensure the validity of @ass The final solution is then assessed on hidden test cases, assigning Pass@I score of 1 if all hidden tests are passed, and O otherwise.
Hyperparameters. We use the hyperparameters recommended by the authors of the respective baselines. For Reflexion, we set max max-iterations to k 3 for HumanEval and MBPP, and 1: = 5 for LeetCodeHardGymm. For LATS, h = 8 is used across all datasets. For DoT and DoT-bank, we set k for HumanEval and MBPP, and k = 5 for LeerCodeHar In Appendix [A2.4 show that naively increasing & (the total number of generated reflections) in Reflexion has minimal impact on performance, highlighting that DoT variants both more effective and cost-efficient. Note: unless specified, the number of retrieved trajectories is for all DoT-bank experiments.

---

Published as a conference paper at ICLR 2025
Table 4: HvmanEval Dataset Results Using Visible Test Cases. Bold indicates the best result and underscore denotes the second best result. Aaee is the relative improvement over the Base model. DoT and DoT-bank show consistent improvements performance.
Sonnet 3.5
Llama-3.1-8B
Llama-3.1-70B
Method
Pass @
ABss
Cost
Pass@1
ABose
Cost
Pass@1
ABse
Cost
Base
86.59
$0.39
59.76
S0.01
79.27
$0.04
Reflexion
88.41
+1.82
$0.89
72.56
+12.80
S0.07
87.2
+7.93
$0.17
LATS
88.41
+1.82
$7.64
74.39
+14.63
$2.00
84.76
+5,49
$4.56
DoT
91.46
+4.87
$1.14
73.17
+13.41
50.10
89.63
+00.36
$0.21
Don'-bank
93.9
+7.31
$1.67
78.66
+18.90
$0.17
93.29
+14.02
$0.31
3.2
EXPERIMENTS & RESULTS
3.2.1
PROGRAMMING Tasks
HumanEval Chen et al. 2021a) We use visible tests included in the problem doc-strings for all iterative baselines (Reflexion, LATS, DOT, and DoT-bank). Table 4compares DoT variants against the baselines across two key metrics: accuracy and cost. The results indicate that DoT achieves higher pass@ rate than LATS, Reflexion, and traditional input-output prompting. Moreover, DoT significantly more cost-effective than LATS, demonstrating that it strikes a more optimal balance between cost and accuracy. DoT-bank offers additional performance improvements over DoT with gains of up to 4% in pass @ as shown in Table 4 This highlights the significance of retaining insights from cross tasks. The performance improvement with Do'T is independent
the choice of LLM. We observe consistent gains in pass @ regardless of the underlying LLM, as shown in Ta- ble 4 Notably, the performance delta is greater for less powerful models like the Llama-3.1 family,
In Table 5| compare DoT against the baselines using synthetically generated tests. At the time the experiments were done, we discovered a bug in the official LATS implementation, where the "num_success" variable was mistakenly incremented even when an incorrect solution was generated. For comparison, we report both the original and bug-free versions of LATS in Table 5) After fixing the bug, there steep 10% drop Pass@ Overall, our results suggest that DoT delivers superior or competitive performance while being significantly more cost-efficient than LATS.
LeetCodeHardGym Shinn et al. (023 Table 6 compares the performance of DoT variants against the baselines using Claude Sonnet 3.5 as base LLM. While LATS incurs a notably high token consumption, leading increased costs, both Reflexion and LATS fail to deliver any perfor- mance gains on this dataset. In contrast, DoT effectively explores more diverse reasoning paths, leading to broader exploration of the decision space, which reflected the substantial 7.5% performance gains. DoT- DoT-bank further amplifies this improvement with an additional 2.5% gain in the performance. We present additional results using GPT-4o models in Appendix A.2.2 A.222
Table 5: Results for the HumanEyal dataset using GPT-3.5- model. Results with are obtained
from Zhou 2023) Acur shows the relative gains w.r.t CoT method.
Method
Pass@1
ACor
CoT*
46.9
ReAct
56,9
+10.00
Reflexion*
68.1
+21.20
ToT
54.4
+7.50
LATS*
83.8
+36.90
LATS (corrected)
73.9
+27.00
DoT
75
+28.10

---

Published as a conference paper at ICLR 2025
MBPP Austin et al. (2021) Since MBPP's problem prompts lack visible test cases by default, follow Shinn et al. (2023) and use synthetic unit tests to generate potential solutions for iterative baselines. Consistent with earlier findings, DoT and DoT-band leads notable performance improvements, with particularly significant gains observed in less powerful L.M backhones as il- lustrated in Table 8
To ensure the gains are not random, we repeat select experiments three times and report statistically significant findings in Section 3.3.1 A detailed token usage analysis is provided in Appendix A.2.3
Table 6: LeetcodeHarddGyy Dataset Results (Table 1). Bold highlights the best result, while underscore marks the second best. Base indicates the improvement over the Base model.
Table 7: Accuracy the Game of 24 Dataset (GPT-4). Augmenting diversity to the exist- ing reasoning frameworks results in signifi- cant gains.
Game of 24 (GPT-4)
Accuracy
Method
Pass
Base
Cost
Base
7.3%
Base
42.5
$0.18
CoT
4.0%
Reflexion
42.5
+0.00
$1.59
ToT
69.0%
LATS
42.5
+0.00
$14.75
ToT Diversity
82.0% (+13.0%)
DoT
50.0
+7.50
$3.18
DoT-bank
52.5
+10.0
$3.78
Table 8: Performance comparison MBPP using synthetic test cases. DoT and DoT-Bank outper- form other methods in Pass@ I, while maintaining cost-efficiency across models.
Sonnet 3,5
Llama-3.1-8B
Llama-3.1-708
Method
Pass@1
Auue
Cost
Pass@1
Ause
Cost
Pass
Cost
Base
77.08
$0.68
$0.02
72.80
$0.07
Reflexion
79.35
+2.27
$6.42
60.96
+12.35
$0.37
71.79
-1.01
$0.74
LATS
79.72
+2.64
$98.71
61.21
+12.60
$7.41
73.55
+0.57
$21.01
DoT
80.35
+3.27
$9.52
64.23
+15.62
$0.59
76.32
+3.52
$1.21
RoT-bank
84.63
+7.55
$9.74
66.50
+17.89
$0.71
78.34
+5.54
$1.34
3.3
ADDITIONAL. STUDIES
3.3.1
QUANTITATIVE ANALYSIS OF GENERATED SELF-REFLECTIONS
To quantify the diversity, we compute the average pairwise cosine similarity across generated self- reflections for each problem within a dataset, reporting both the mean and standard deviation. We use the 'all-Mini-LM--6' model from SentenceTTaansfomere]] to generate embeddings for these flections and tabulate our results in Table lower similarity score indicates greater diversity.
Table Average pairwise cosine similarity scores for generated reflections across baselines. DoT variants produce more diverse and less redundant reflections.
Dataset Model
Reflexion
LATS
DoT
DoT-Bank
HumanEval + 1lama-3.1-708
0.83
0.65+0.41
0.48
0.48
LeerCodeeHardGym Sonnet 3.5
0.83 +.. 13
0.70
0.62+n.13
0.61
hhpp://sherttnett
8

---

Published as a conference paper at ICLR 2025
3.3.2 MEMORY-BANC FEW-SHOT SELECTION'S 'S IMPACT ON PERFORMANCE
Recall from Algorithm| I that in phase 1, successful trajectories along with the problem's docstring embeddings are stored in the memory bank. In phase 2, for re-attempting failed problems, the docstring embedding is used to retrieve and inject similar trajectories as in-context examples. To evaluate the impact of retrieved few-shot examples from the memory bank, we conducted an (i) Random, which injects & random trajectories from MB into the context of Ma and (ii) Cosine-sim, which retrieves the k most similar (closest in embedding space) trajectories based on the cosine similarity their embeddings, [Table 10 demonstrates that embedding- based retrieval consistently outperforms random selection, with performance plateauing at 3. This highlights that injecting contextually similar examples improves model performance, consistent with recent findings in (Didolkar et al. 2024).
ablative study comparing two strategies:
Table 10: Performance on HumanEval with GPT-4o using varying numbers of memory bank examples: Random examples are uniformly sampled, while Cosine-sim examples are retrieved by cosine similarity.
#ICL Examples
5
Random
92.68
95.12
97.56
96.95
96.34
96.95
Cosine-sim
92.68
97.56
98.17
98.78
98.17
97.56
3.3.3 GENERATING DIVERSE REFLECTIONS
We analyze two variants for generating diverse reflections and empirically study their performance and the associated costs.
To encourage diverse reflections, condition each generated reflection all previous ones, incorporating them into the context using a structured prompt. Formally, 2, ~ PD (2: r, 21.. Vi E (1...k).
Iterative Sampling:
Table 11: Comparison of Sampling Methods for generating diverse reflections on the HumanEval Benchmark. The table shows pass@1 performance and cost for each model using one-shot and iterative sampling methods.
GPT-4o-mini
GPT-4o
OpenAl o1
Sampling Method
@ 1
Cost
Pass@
Cost
Pass
Cost
One Shot Iterative
92.07
$0.05
95.12
$1.01
99.39
$26.75
91.00
$0.07
93.29
$1.57
95.73
$37.37
From Table 11, we observe that one-shot sampling for generating diverse reflections consistently outperforms iterative sampling across all models. A recent study by |Hayati et al.) (2024) further supports our observation. GPT-4o-mini achieves a notable Pass@ score of 92.07 at a minimal cost ($0.05), making it the most cost-efficient. As model size increases, the cost disparity becomes more pronounced one-shot sampling is approximately $10 cheaper while delivering a 4% improvement.
Other sampling methods, such repetitive sampling using LLM to filter redundant reflections (similar to our motivation analysis), were not explored due to their higher costs, which compromise cost efficiency.
3.3.4 IMPACT OF DIVERSITY ON EXISTING REASONING FRAMEWORKS
The modular design of diversity strategy introduced in this work, aimed mitigating redundancies in generated steps and thoughts, broadly, is also applicable to the existing reasoning frameworks. As demonstration, we enhanced the Tree of Thoughts (ToT) framework by incorporating diversity into the thought sampling strategy. This was achieved through explicit prompting and by passing previously generated thoughts through context. We compared this enhanced version with the base

---

Published as a conference paper at ICLR 2025
ToT implementation the challenging Game-of-24 dataset, with the results presented in Table 7 We refer the reader to Appendix [A.4.] for the exact prompts been used. By introducing diversity into the proposed thoughts, we observed a substantial 13% improvement in 0-1 accuracy.
RELATED WORK
We categorize the literature on LLM reasoning capabilities into three main themes: (i) enhancing step-by-step reasoning abilities, (ii) promoting diversity in the reasoning process, and (iii) incor- porating memory mechanisms and external memory facilitate learning from past experiences.
Reasoning in LLMs. Since LLMs were identified as few-shot learners (Brown et al. 2020), sev- eral prompting techniques and inference strategies have been proposed to improve their reasoning abilities. Chain of Thought (Co'T) (Wei Wei et al. (2022) introduced step-by-step reasoning to enhance problem-solving. Self-Consistency Wang al. 2023) extended CoT by leveraging multiple inde- pendent CoT runs for improved outcomes. Tree of Thoughts (ToT) (Yao et al. (2023a) employed search algorithms like BFS/DFS with LLMMguided heuristics to further boost performance. Re- Act (Yao et al. 2023b) integrates reasoning and action steps, while Reflexion (Shinn et al., 2023) extends this approach by incorporating a "selffreffection" component, building upon the Self- framework (Madaan et al. (023). More recently, LATS (Zhou et al., 2023) integrated Monte Carlo Tree Search (MCTS) into Reflexion, achieving gains at the expense of higher token usage and costs. However, analysis revealed that Reflexion and LATS generate redundant self-reflections thereby hindering effective decision space exploration.
Diversity in Reasoning, Inducing diversity in LLM reasoning has been explored through methods like DIV-se (Naik et al., 2024), which uses varied prompts and personas (e.g., "Think like Alan Turing" or "Think like Math Professor"). However, it requires manual persona selection, limiting flexibility. Flow of Reasoning (Yu et al. 2024) uses GFlowNet to train LLMs to generate diverse reasoning without predefined personas but involves task-specific fine-tuning. In contrast, our framework, DoT, promotes diversity through structured prompts and one-shot sampling, leveraging task-agnostic memory bank without personas, manual interventions, or task-specific training. This enables DoT to explore multiple decision branches and adapt across domains scamlessly.
LLM Memory. Memory mechanisms enable LLMs to retain and learn from past experiences, en- hancing their ability to reason and adapt. MemoryBank (Zhong et al. 2024) enables LLMs to update and retrieve past interactions to align with user intent. MemoChat (Lu al. 2023) trains LLLMs to efficiently retrieve relevant dialogue history. Reflexion (Shinn al.) 2023) leverages memory in reinforcement learning manner, with past action described in verbal format. CLIN Majumder et al. (224) stores causal abstractions in evolving memory. In contrast, our task agnostic memory bank stores entire agent trajectories retrieving them as in-context examples, enriching decision- -maiing and reasoning.
External Memory for LLMs. External memory methods like Buffer of Thoughts (BoT) (Yang et al.) 2024) introduce high-level "thought templates" to assist reasoning, while Needle in Haystack Chaudhury et al. (2024) demonstrates the utility of external memory for tasks requir- ing long context lengths. Our approach differs by storing complete task trajectories in dynamic, task-agnostic memory bank, injecting relevant examples to improve reasoning. The memory-bank grows as the agent solves more tasks.
LIMITATIONS & CONCLUSION
DoT and DoT-bank significantly reduce costs compared to LATS but remain 1.4x more expen- sive than Reflexion and up to 8x higher than the base LLM. Iterative reasoning frameworks also suffer from increased latency due to repeated interactions with external environments (e.g., tools, APIs), limiting scalability. While our one-shot sampling mitigates redundancies in generated reflec- tions, not entirely foolproof, leaving room for future research on more efficient strategies to generate diverse reflections. This work addresses key limitations in self-reflection-ban frameworks, namely redundant reflections and missed opportunities for cross-lask knowledge trans- fer. Although focused programming tasks, extending these methods to other reasoning domains presents an exciting avenue for future exploration.
reasoning
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL httt::tt/////
2108 07
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, 1877-1901. Curran Associates, Inc.. 2020. URL http:::/prooceeeiogg...ee neuriss www.eeuiiss..cpppppeeeeeeeeeeeee cc/paper wttp://ppeeeeleeeeeeeeeeeeeeeeee files/paper/202// f1le/257706660000544411666662222
pdf
Subhajit Chaudhury, Soham Dan, Payel Das, Georgios Kollias, and Elliot Nelson. Needle in the haystack for memory based large language models. arXiv preprint arXiv:2407.01337, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021b.
Aniket Rajiv Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timo- thy P Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael Curtis Mozer, and Sanjeev Arora. Metacognitive capabilities of LLMs: An exploration mathematical problem solving. In The eightt- Annual Conference on Neural Information Processing Systems, 2024. URL net. forumrirroooooooo di 9UyP4HУK 9UDUUY
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and Akhil Mathur et. al. The llama herd of models, 2024. URL /tRppttts://axxv. brq/abs 2407.217777 21783
Shirley Anugrah Hayati, Minhwa Lee, Dheeraj Rajagopal, and Dongyeop Kang. How far can we extract diverse perspectives from large language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
Hung Lc. Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Hoi. CodeRL: Mastering code generation through pretrained models and deep reinforcement learning. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL httph:tt///cm.. httpp:://penreevee.... net forum?id= WaGvb70zySA
Junru Lu, Siyu An, Mingbao Lin, Gabriele Pergola, Yulan He, Di Yin, Xing Sun, and Yunsheng Wu. Memochat: Tuning Ilms to use memos for consistent long-range open-domain conversation. arXiv preprint 230i 208.03390008.9999, 2023.
11

---

Published as a conference paper at ICLR 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder. Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventh Conference on Neural Information Pro- httpttss:::
cessing Systems, 2023. URL htwph:ttps//t...... httpp:://ppenteerieeee.... net/ formm?form?? wtw.ooooummisusseo
Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison Burchh, and Peter Clark. CLIN: A continually learning language agent for rapid task adaptation and generalization. In First Conference on Language Modeling, 2024. URL httt:ss:///ppereeeeeeee
net/ / forum? mennmnddxxxxxxx555
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL
https: http://axxivv.x// aaagg//bb//aae 1ete./31.... 5602
Ranjita Naik, Varun Chandrasekaran, Mert Yuksekgonul, Hamid Palangi, and Besmira Nushi. DI- VERSITY OF THOUGHT IMPROVES REASONING ABILITIES OF LARGE LANGUAGE MODELS, 2024. URL /tttps://openeeviee
met/ffourm id=Fvdh
Timo Schick, Jane Dwivedi-Yu, Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In A. Oh, т. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36, pp. 68539-68551. Curran Associates, Inc.,
cctpsper.eppppppeeeeeeee forerr_flleee www.://aaeeeeeeeeeeeeeeaaee 2223/ 2023. URL https::/pproceeediggs...om Papeeaer---pPPPP-----
file/
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In A. Oh. T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 0634-8655 Curran Associates, Inc., neuris.eerrspp.. wwpperrrepppppppppepp/eeeeee...e sorr_fflll flless/pppee/e/2//////////
2023. URL
file/
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https: wptp/eeeeeeewecc/////// wwt.eviwwweeeeeeeeeeeeeeeeeeeeee
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thoughh prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824 24837. Curran Associates, Inc., 2022. URL httpps:/pproceeeeeeeeee wwwrpigs....u.up...e...........e fi e 000000061122444411111
Poreeaaeeeeaaeeeeeeereeeeeeeeee .d
Ling Yang. Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E. Gonza- lez, and Bin CUI. Buffer of thoughts: Thought-augmentee reasoning with large language models. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024. URL httt:::://ppenevveew i
forum forpee///iommmd/o d= 19JPtb
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 11809 11899.11122...11111111 Curran Associates, Inc., 2023a. URL https: http:::::s//pproor.eeeessoss/sss neuriss..eeuppppccoccmmmm cctp:sccccpppppeee htsp:r/sseecppppppeeeele/ee//ppp
ftlees//ppper/22223
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Con- ference on Learning Representations, 2023b. URL https: h/tp::///ppeeevieww.. net/ forum?
id=WE_ vluYUI-XX
12

---

Published as a conference paper at ICLR 2025
Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Efficient training of Ilm policy with divergent thinking. arXiv preprint arXiv:2406.05673 2024.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19724-19731, 2024.
Andy Zhou, Kai Yan, Michal Shlapentokh- Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL IMPLEMENTATION DETAILS
The memory bank is implemented as a hash map: unique-task-id: complete-task-traicce phase of our DoT algorithm, successful trajectories and the embedding of the task's doestring are stored. In phase 2, when re-attempting failed tasks, we generate an embedding for the task' s docstring and retrieve the top k most similar trajectories using cosine similarity. We use Cohere- Embed- BoatheddV--Baee V3-English for embedding generation. The & closest trajectories, based on cosine similarity, are considered relevant and serve as in-context examples for the actor agent.
In
A.2
ADDITIONAL EXPERIMENTS
A.2.1
EXPERIMENTS WITH STATISTICAL SIGNIFICANCE
To ensure the reported gains random, repeat subset of experiments three times and report both average and standard deviation in Table 12 These results further validate DoT's effectivenesss
Table 12:
HumanEval and LeetCodeHardGym using Sonnet-3.5. DoT variants show statistically significant improvements.
3-run average Pass @1
Model
HumanEval
LeetCodeHardGym
Base
85.57=0.94
40.83
Reflexion
88.21 +0.35
41.67
LATS
89.16
42.50=2.50
DoT
91.46
50.00 22.00
DoT-bank
94.10 10.35
53.33 1.44
A.2.2 LEETCODEHARDGYM
We replicate the LecrCodeHardGym experiments using GPT-4o and GPT-4o-mini, and summarize the results in Table 13 DoT achieves comparable or superior performance with a 4x reduction in cost. Notably, DoT-bank outperforms LATS by 12.5% on GPT-4o, while being 2.5x more cost-efficient.
Table 13: LeetcodeHHrddGym Dataset Results with Visible Test Cases. Bold highlights the best result, while underscore marks the second best. Aasee indicates the improvement over the Base model. DoT and DoT-bank consistently enhance performance across various base models.
GPT-4o-mini
GPT-4o
Method
Pass
ABuse
Cost
Pass @
ABose
Cost
Base
15.0
$0.05
25.0
$0.18
Reflexion
17.5
2.50
$0.10
27.5
2.50
$1.26
LATS
20.0
5.00
$0.72
32.5
7.50
$13.33
DoT
22.5
7.50
$0.15
32.5
7.50
$3.18
DoT-bank
22.5
7.50
$0.26
45.0
20.0
$5.48
A.2.3 LLM TOKEN USAGE ANALYSIS
Table 14 presents breakdown of average input and output token usage across different methods. DoT's 's token usage closely mirrors that of Reflexion, yet achieves significantly better performance. In contrast, LATS is order of magnitude more token- and cost-intensive than the other baselines.
14

---

Published as a conference paper at ICLR 2025
Table 14:
Token usage and cost analysis for different methods on the HvmanEval dataset using
Sonnet-3.5.
Method
Avg Input Tokens
Avg Output Tokens
Avg Cost per task (S)
Base
223
114
0.0024
Reflexion
702
221
0,0054
LATS
9370
1230
0,0466
DoT
904
282
0.0070
DoT-bank
1418
395
0.0102
A.2.4 IMPACT OF INCREASED ITERATIONS ON REFLEXION PERFORMANCE
Table 15 analyzes the impact of increasing the maximum iterations (A) performance (Pass @ 1) for the Reflexion method. The table shows that naively increasing the number of reflections has minimal impact performance, while DoT variants outperform Reflexion at 3.
Table 13: Performance and Cost Analysis for Different Values
Method
K=3
K=6
K=8
K=10
Reflexion
88.41 (S0.89)
89.02 ($1.36)
89.63 ($1.64)
89.63 ($1.87)
DoT
91.46 ($1.14)
DoT-bank
93.90 ($1.67)
A.3 PROMPT TO CLUSTER REFLECTIONS USING GPT-40
We use the prompt illustrated in Figure 4 to cluster reflections by using GPT-4o. In Table I6 show an example of how the output from GPT-4o looks like corresponding to the aforementioned prompt.
You are provi ded with 1ist sentences, each ident fied by a unique ID. Your task is to group these sentences into clusters, where each cluster contains sentences that convey the same meaning. The sentences might be phrased differently, but they should express the
same core idea or intent.
Instructions:
Review each sentence carefully, considering synonyme, paraphrasing, and variations in expression.
Group the sentences that share the same meaning into single cluster. For each cluster, list the unique IDs of the sentences included.
Input Format:
1ist of sentences with unique IDs.
Output Format:
cluster IDs along with concise summary of each cluster, presented in table format
Figure 4: Prompt to cluster reflections
15

---

Published as a conference paper at ICLR 2025
Cluster ID
Sentence IDs
Summary
0, 1, 3, 4, 6, 7,
The implementation fails to raise TypeError for non- integer inputs, particularly strings. Type checking needs to improved to explicitly raise TypeErrors for non-integer inputs. Various methods for fixing include adjusting condi- tions or using try-excepl blocks.
8, 9, 10, 11, 12, 13,
14, 15
16,
17.
18,
19,
20,
The implementation fails to raise TypeError for non- integer inputs, and relies on try-except blocks which do not handle all cases. The solution involves adding explicit type checks before calculations to ensure correct error handling.
21.
22
23,
24
25,
27,
28,
29,
30,
31.
32,
33,
34,
35,
36,
37,
38,
39, 40,
41,
42
43,
44, 45,
46,
47,
48,
49, 50,
51
Table 16: Clusters of sentences with similar meaning and their summaries
A.4 PROMPTS USED IN DoT
We use the prompt in Figure generate K diverse reflection. We use the prompt in Figure o to generate function implementations by using the reflections and few-shot examples for the program- ming tasks.
You are a Python programming assistant. You will be given a function implementation, unit tests, and previously generated reflections. Write multiple unique and diverse reflections to fix the problem. Each reflection should follow this structure:
Problem: la terse description of the identified problem). Fix: (proposed fix or hint to fix the identified problem}-
Ensure your reflections
accurate and leverage previous reflections
to avoid repetition. Aim for diversity in your explanations while prioritizing the correctress of your hints. Add "\n\n" at the end of each proposed reflection. Only provide the few sentence descriptions in your answer, not the implementation. RESTRICT 70 (K] UNIQUE REFLECTIONS.
Figure 5: Prompt for generating K diverse reflections
A.4.1 ADDING DIVERSITY TO FREE-OF-THOUGGES
We use the following prompt shown in Figure 7 to add diversity over Tree-of- Tree-of-thoughts for Game of 24 task (corresponds to results shown in Table
16

---

Published as a conference paper at ICLR 2025
Example 1:
[function impl]
"python
def
(nums: List [int],
target: int ->
[ist[int]: 1en (nums)
left, right 0, 0
max max_hength 0
current sum
result
while right n:
current_aum +- nums [right]
while current_sum target:
current_aum - nums [1eft]
left
right left + >= max _x__lenghh: max_length right left 1 result nums [lett:right+]] right +=
return result
[unit test results]:
Tests passing: hatpres_ababrraaaytt_eeettee
assert longest subarray with win
1imit ([1, 2, 3, 4, 51. 8) 11, 2, 31
assert [ogest_oubbrryy.itttsssssn.1111 ([1, 2, 3, 4, 5], 15) [1, 2,
3, 4, 51 assert boggeet_uubarrrrsg.tttttoonnnnnd -1, 2, -2, 3, -31, 2 11,
-1, 2, -2, 3]
assert congett_ubbrrrss_tttttsssm_ni_in
10) [] 0) == []
assert
assert Longest.cubbrrryy.ttt....aaatlat
-5) == 11
Tests failing:
assert
with
(1,, 6, 7, 8, 91, 4) 11
output: [5]
[self-reflection]:
Problem:
The implementation failed the test where no subarray fulfills the condition. The issue due to the use of instead of ">" in the condition to update the result.
Fix: Change the condition to only update the result when the current subarray length is strictly greater than the previous longest subarray length by replacing ">=" with ">" in the condition.
Problem:
The current implementation does not handle the case where the sum of any subarray exceeds the target, leading to incorrect
results.
Fix:
Add a condition to check the sum of the subarray exceeds the
target betore updating the result.
Problem:
The algorithm might not correctly identify the longest subarray when multiple subarrays have the same length but different sums.
Fix: Introduce an additional check to ensure that the subarray with the sum closest to the target is selected when lengths are equal.
Problem:
The code does not consider the possibility or a target that is too small to be achieved by any subarray. Fix: Implement preliminary check to return an empty array is the target is smaller than the smal lest element in "nums" or if "nums" is empty.
an empty array
17

---

Published as a conference paper at ICLR 2025
Example 2:
[function impll:
"'yythnn
def
(nums: List [int],
target: int ->
[ist[int]: 1en (nums)
left, right 0, 0
max max_tength 0
current_sum 0
result
while current_sum nums [right]
target:
current_aum += nums [right]
right += 1
while right
nums [right] while cumrent_sum target:
current_san
surrent_sum - nume [left]
Left +=
if right left
max Lex_ethth:
nax_ nax_length right left
result nums [left:rightt]]
right +-
return result
(unit test results]
Tests passing:
assert Lorgget_uubarrssy.ttttssssmllllt 10) []
assert (ongett_obbrrryyyttttmmmml..iii
assert longest_stberrryy.ttttt..mmmmm
-5) []
Tests failing: assert
([1, 2, 3, 4, 51, 8) == 11, 2, 31 output: dist index out of range
assert longest_ with_sum limit 2, 3, 4, 51, 15) 11, 2,
- 3, 4, 5: # output: list index out of range
assert output: list index out of range ([I, -1, -2, 3, -31, 11, 1, 2, -2, 3] # output: list index out of range
([5, 6, 7, 8, 91, 4) - []
[self-reflection]
Problem:
The implementation fails due accessing "nums [right]* without bounds checking,
IndexError caused by
Fix: Add a bounds check for the "right" variable in the "while current_sum + nums [right] <= target: loop by modifying the condition to "while right 1en (nums) and current_sum + nums (right) target:".
Problem:
The initial while doop does not account for the scenario where the right index exceeds the Length of the array.
Fix: Introduce boundary condition to ensure "right" remains within the array limits during the initial accumulation phase.
Problem:
The implementation incorrectly attempts to extend the subarray beyond the array's as bounds, leading to a runtime error. Fix: Implement check within the loop to exceed the length
"right" does "nums" before accessing "nums [right]'.
Problem: The code does not correctly handle cases where the target sum is less than any single element in the array, resulting in
failure to return empty array.
Fix:
Add a preliminare check to return an empty array if no subarray can possibly meet the target.
END OF EXAMPLES
18
Figure 6: Prompt for generating function implementations by using the reflections and few shot examples.

---

Published as a conference paper at ICLR 2025
Task: Game of 24 is mathematical reasoning challenge, where the goal is to use 4 numbers and basic arithmetic operations $ /1 to obtain 24.
You have the following numbers: (input)
Generate several possible next steps applying basic arithmetic operati ons (+, ", *, to pairs of numbers. Mach step should use a unique combination of numbers and unique operati on that has not been used in the current set of possibilities.
Possible operations include: Addition: a
Subtraction: - b or b - Multiplication: b Division: b b (provided the result is integer)
Instructions:
1. Do not repeat any operations that have already been listed.
2. Avoid using the same pair of numbers for the operation.
3. Provide the result of the operation and show the new set of numbers after applying the operation.
Example:
Input: 4, 5, 6, 7
Possible next steps:
(left: 9
+ 20 (ieft: 20 6 7)
5 -1 (left: -1 6 7)
4 11 (ieft: 1 1 7)
10 (left: 5 10)
24
(Left: 5 24)
= -2
(Left: -2 5 7)
4 + = 1:
(Left: 5 6 11)
28
(left: 5 6 28)
-3
(left:
6)
+ 1.
(Left: 4 11)
5 = 30
(left: 7 30)
+ 7 = 12
(left: 4 6 12)
35
(left:
35)
-2
(Left:
4 6)
+ 7 = 13
(Left: 4 5 13)
* 7 = 42
(Left: 4 5 42)
6 - -1 (left: -1 4 5)
Input: {input]
Possible next steps:
Figure 7: Prompt for obtaining diverse reasoning chains for the Game of task
19


Reviews:
[{"review_id": "RPCe8mxRQC", "reviewer": "Reviewer_7Wju", "summary": "Motivated by the observation of repetitive reflections, this paper proposes a method to improve the diversity of generated reasoning by incorporating task-agnostic memory and diversity of thoughts (DoT) prompting (on top of reflexion). The experiment results show clear improvement with various LLMs.", "questions": "1. Line 700 has un-referred figure", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The method is well-motivated. \n2. I appreciate the analysis and results in the Introduction to showcase the repetitive reasoning generation with existing methods, i.e., LATS, which is insightful and provides evidence.\n3. The paper is well-written and easy to follow.\n4. The results seem promising.", "weaknesses": "1. I am still unsure if diverse reflections can be guaranteed if prompting the Mdr agent with \"Ensure your reflections are accurate and leverage previous reflections to avoid repetition. Aim for diversity in your explanations while prioritising the correctness of your hints.\". Can you provide empirical evidence or analysis demonstrating how your prompting approach leads to diverse reflections?  For example, you could include metrics quantifying the diversity of generated reflections compared to baseline methods. You can also explain the reason behind it.\nTo me, it is more like a magic prompt. Can you explain why this simple prompt works? \n2. The whole framework is built on reflection, with Msr replaced by Mdr. Given my first comment and considering that utilizing memory to improve response diversity is widely explored, the contribution of DoT is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M614nAkPhX", "reviewer": "Reviewer_RxnW", "summary": "This paper proposes Diversity of Thought (DoT), a method that enhances LLM reasoning through diverse reflections. The paper first identifies two limitations of existing methods: (1) limited exploration due to repetitive reflections, and (2) inability to reuse knowledge from similar tasks. To address these, DoT reduces repetitive reflections and incorporates a task-agnostic memory. Experiments on three programming benchmarks and the Game of 24 demonstrate improvements of DoT compared to baselines.", "questions": "- L700 Figure reference is missing \n- The paper uses Reflexion as a key baseline, but other similar self-reflection methods, e.g. [1], should also be cited.\n- Several citation formats are not proper, e.g. some \\citet, and \\citep are misused, e..g L41, L105, L350, etc. \n- How is the cost of the Llama-3.1 model calculated?\n\n[1] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, Peter Clark. Self-Refine: Iterative Refinement with Self-Feedback", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper examines tasks requiring complex reasoning and proposes solutions to enhance LLMs' reasoning capabilities. Experiments show that DoT and DoT-bank outperform baselines across various benchmarks.\n- The paper provides some analysis, e.g. quantifying the redundancy of reflections generated using existing methods, to motivate the proposed method.", "weaknesses": "- The proposed method is motivated by two observed limitations of existing methods: lack of exploration and inability to leverage cross-task knowledge. While the first limitation is analyzed and discussed in Table 1, the second lacks supporting analysis. Therefore, adding a task-agnostic memory component to DoT appears somewhat disconnected from the paper’s discussion. Although intuitively it makes sense to reuse insights from previously solved tasks, whether it is useful in practice remains unclear. The authors need to provide more evidence to justify this proposed solution in order to create a more coherent story for the paper.\n- For the Reflexion baseline, is the number of reflections set to 1? If so, this may not be a fair comparison, as DoT uses many more reflections. A better comparison would be to Reflexion with k sampled reflections. This would more accurately indicate whether the proposed diverse reflection generation method (one-shot or iterative) is effective.\n- DoT-bank requires building a memory bank, and its size scales with the number of failed tasks using DoT based on Algorithm 1. Therefore, the reported cost of DoT-Bank is misleading, as it does not include the cost of building the memory bank. Also, for the DoT-Bank results in all tables, is the memory bank used for every task or only for failed tasks? \n- The choice of models used for different tasks seems arbitrary. Why are different models used across tasks? For example, why does HumanEval use Sonnet 3.5 and Llama-3.1, while LeetCodeHardGym only uses Sonnet 3.5? Additionally, why does GPT-3.5 have a different setting in Table 5 compared to other models in Table 4?\n- For incorporating DoT into ToT, is the only difference that ToT samples k thoughts in parallel, while ToT+Diversity samples k thoughts in one shot?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AE0g0pTaLg", "reviewer": "Reviewer_vtDD", "summary": "This paper introduces DoT (Diversity of Thoughts), a novel framework for improving language model agents's iterative reasoning tasks by addressing two key limitations in existing approaches: 1: redundant reflections that lead to inefficient exploration of the decision space, and 2:the inability to leverage insights across different tasks. The framework consists of a diverse-reflections model that reduces redundancy in reasoning paths and a task-agnostic memory bank that enables knowledge transfer between tasks. Through extensive experiments on programming benchmarks (HumanEval, MBPP, and LeetCodeHardGym) using various language models, DoT demonstrates up to 10% improvement in Pass@1 while maintaining cost-effectiveness compared to existing methods like LATS, and when integrated with Tree of Thoughts (ToT), its diverse reflection module improved performance by 13% on the Game of 24 benchmark.", "questions": "How is cost calculated for different model types (especially for Llama models where costs are in GPU hours rather than API calls)? Why is the cost difference between iterative and one-shot sampling surprisingly small in Table 10?\n\nWhat specific mechanism is used for retrieving \"relevant trajectories\" from the memory bank? How is relevance defined and measured?\n\nWhile the method reduces LLM inference calls by avoiding redundancy, does it require more tokens per call due to longer context? Can you provide a quantitative analysis of average input/output token lengths compared to baselines?\n\nWhat's the effect of varying k in k diverse-reflections? How reliable is the one-shot prompt approach, and what's the sensitivity to the number of retrieved trajectories?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper identifies and addresses concrete limitations in existing approaches (redundant reflections and isolated task solving) with a practical solution that shows consistent improvements across different models and benchmarks. The proposed modifications are simple to implement yet effective.", "weaknesses": "1:The paper's two main components (Diverse-Reflection and Memory Bank) appear to be independent modules without a strong theoretical connection or necessity to be combined. This suggests the work is more of an engineering effort combining separate improvements rather than a cohesive theoretical advancement. The lack of comprehensive ablation studies makes it unclear which component truly drives the performance gains.\n\n2: The paper's positioning around efficiency improvement over LATS is problematic because LATS and similar resource-intensive methods primarily aim to maximize accuracy, not efficiency. The experimental validation has several limitations: improvements are incremental (2-10%), test sets are relatively small, and no statistical significance testing is reported to validate that improvements are meaningful rather than random variation.\n\n3: The paper lacks detailed analysis of why and how diversity in reflections leads to better results. There's insufficient discussion about the reliability of one-shot sampling for generating diverse reflections and potential failure modes. Also, key technical details about the memory bank implementation and retrieval mechanism are not fully explained, including specifics about how \"relevant trajectories\" are defined and retrieved and how the diversity in reflection generation is ensured; all of these make reproduction challenging.\n\n4: The evaluation focuses mainly on programming tasks with visible or synthetic unit tests, limiting the method's demonstrated applicability to domains where clear evaluation metrics are available. This leaves questions about generalizability to broader reasoning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Y67piHM0n", "reviewer": "Reviewer_vf4f", "summary": "The paper presents an iterative prompting method for LLMs that aims to improve responses by repeatedly reflecting on its previous outputs similar to the recent Reflexion method. The key advance here over the Reflexion method is to encourage diversity in the outputs. The method also incorporates a task-agnostic memory component to enable knowledge retrieval from previously solved tasks.", "questions": "It isn't clear how an evaluator can be an LLM. Is the LLM asked to produce a numeric score? Or is there space for a textual evaluation?\n\nWhen you say that MB trajectories are picked based on cosine-similarity, do you mean that the K most similar are chosen? Is there a risk that there isn't much diversity in these retrieved trajectories? Or am I missing something?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is well laid out and clearly describes the method and justification for it. The method is general and can be widely applied as well as supporting integration with other orthogonal approaches (the example being tree of thoughts). The experiments are varied and well chosen with appropriate choice of base LLMs and tasks. The results show a consistent and clear improvement over other recent SoTA methods: Reflexion, LATS and ToT (where appropriate).\n\nI am particularly impressed with the modular nature of DoT and its generality.", "weaknesses": "The weaknesses are, in my opinion, minor. The following things came to mind:\n* The description of retrieval of previous trajectories could be a bit clearer, as could the experiment which produces the results in Table 9.\n* I am not sure that the t-SNE evaluation in Figure 4 is as robust as it could be. It may show that DoT produces more diverse self-reflections then LATS but as the embedding is done separately for the two sets of reflections it isn't entirely clear that this is the case. I would suggest embedding the two methods together and showing that the DoT reflections are more dispersed within the same space (maybe that is what was done but it isn't clear to me).\n* There are lots of results and space is at a premium but things do get less clear towards the end of these, perhaps partly due to the formatting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)