Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve
Authors: Yuanzhe Liu
Ryan Deng
Tim Kaler
Xuhao Chen
Charles Leiserson
Yao Ma
Jie Chen
Venue: NeurIPS 2025 poster ()
Keywords: Multi Agent
Large Language Model
Code Optimization
Primary area: applications
TL;DR: We propose a lesson-based framework for multiple LLM agents to collaboratively solve coding problems by learning from each other.

Abstract: Recent studies show that LLMs possess different skills and specialize in different tasks. In fact, we observe that their varied performance occur in several levels of granularity. For example, in the code optimization task, code LLMs excel at different optimization categories and no one dominates others. This observation prompts the question of how one leverages multiple LLM agents to solve a coding problem without knowing their complementary strengths a priori. We argue that a team of agents can learn from each other's successes and failures so as to improve their own performance. Thus, a lesson is the knowledge produced by an agent and passed on to other agents in the collective solution process. We propose a lesson-based collaboration framework, design the lesson solicitation--banking--selection mechanism, and demonstrate that a team of small LLMs with lessons learned can outperform a much larger LLM and other multi-LLM collaboration methods.

Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve
Yuanzhe Liu
Ryan Dengat 2t
Tim Kaler²
Xuhao Chen 2.3
Yao Ma² Jie Chen "Masachusettss Institute of Technology 3 Michigan State University *MIT-IBM Watson Al Lab, IBM Research edu (ryandeng, tfk, cxh, cel j@mi edu chenjie@us.. ibm.com
Charles E. Leiserson
Rensselaer Polytechnic Institute
Lliuy2,,my11322
Abstract
Recent studies show that LLMs possess different skills and specialize in different tasks. In fact, we observe that their varied performance occur in several levels of granularity. For example, in the code optimization task, code LLMs excel at different optimization categories and no one dominates others. This observation prompts the question of how leverages multiple L.LM agents solve coding problem without knowing their complementary strengths a priori. We argue that a team of agents can learn from each other's successes and failures so as to improve their own performance. Thus, lesson is the knowledge produced by an agent and passed on to other agents in the collective solution process. We propose lesson- based collaboration framework, design the lesson solicitation solicitation-bankingg banking-selection mechanism, and demonstrate that team of small LLMs with lessons learned can outperform a much larger LLM and other multi-LLM collaboration methods.
selection
1 Introduction
Code optimization refers to rewriting the code such that it performs the same task more efficiently [32]. The efficiency is predominantly measured by the runtime, but it could also refer to storage, energy, other resource consumptions. Code optimization a less explored coding task in the context of AI compared with the popularly studied code generation task (62). but it natural, and sometimes key, step in the software development cycle. Code performance can be improved in many ways, such as using a cower-complexity algorithm, caching and memorization, data alignment, vectorization. and parallclization [50] 5). Many of these approaches require low-level system knowledge that code LLMs are not explicitly trained with.
We are interested in exploring the use multiple LLM- --M-owwerd agents for code optimization. We multiple LLMs because
LLM performs the best on all problems, even if they trained with extensive code data. The common practice of benchmarking uses aggregated performance to rank models [6]. but the best-ranked model may be the best performer for every problem. We take the ParEval benchmark (39], which consists of programming problems in scientific computing, for example. When three open-source, small models and two GPT models are evaluated on this benchmark (see Appendix A for details), GPT-4o 43 is the overall winner. However, on the
"geometry" category of problems, Qwen7B 25 outperforms GPT-4o by 2.5x terms of speedup; and on the "histogram" category, Deepseck7B (20 and Qwen14B (25] outperform GPT-4o by 1.6x (Figure 5). Additionally, among the three open-source models, Qwen7B excels "search," while
Part of the work was done while YL visited MIT-IBM Watson AI Lab, IBM Research. Part the work was done while RD interned at MIT-IBM Watson Lab, IBM Research. Code implementation available at /ttttttt:///t gi thub. com/MI FastCoder/Lessonl
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Original code
for (int 1 0; 1 < ++1)
Naive implementation of matrix multiplication AB.
for (int for (int 0; n; ++k)
n; ++4)
cri][j] += A[il[k] B[k][j];
Improved code, round
for (int i 0; n; ++i)
Lesson: Reordering loops improves cache locality and inereases performance. The order of (i,k,j) out of 6 different permutations often performs the best, because of how caches work.
for (int 8; k n; ++k)
for (int j = 8; j n;; ++j)
c[i][j] += A[i][k] x B[k][]];
Improved code, round
#pragma omp parallel for
for (int i 0; < ++i)
Using OpenMP to parallelize the for-loop further improves performance. Parallelizing only the outermost loop performs the best, due sufficient par- allelism and less parallel scheduling overhead.
Lesson:
for (int k 8; k < n; ++k) for (int cri][j] += A[il[k] B[k][j];
0; j n; ++j)
Figure 1: Successively improving -atrixx multiplication in C with lessons.
Deepseek7B excels at "scan" and "dense_ia _aaeellla"a and Qwen148 excels at "sparse_la" "reduce," "fft," and "sort" (Table 5). Such varied performance suggests the opportunity of exploiting the complementary strengths of different LLMs to deliver the best solution.
How does one make use of multiple agents to solve coding problem? In this work, we advocate the concept lessons inspired by classroom experience. A student's problem-solving skills are not only taught by teacher at class but also developed through peer learning. For example, after receiving the graded homework, Student A, who cannot complete the solution to a problem may consult with Student B, who carns perfect score, on tips of the correct steps toward the solution. Mcanwhile, Student can also benefit from learning from Student C, who comes up with a wrong solution, to avoid making similar mistakes. Each student learns from the success and failure lessons of others to improve their own problem-solving skills. LLM agents behave similarly. Pre-training is analogous to classroom teaching, while pre-trainne prained LLM can improve its skills through prompted with lessons.
lesson means any information that helps an LLM better solve the problem hand. For code optimization, such lessons may be optimization strategies applicable to the current code, common pitfalls that programmers are trapped in, or performance feedback from profilers. Note that code's performance be improved by combined use of multiple strategies, step by step. Figure I shows the initial steps of a classic example of engineering the performance which can be improved by 3000x in speedup under extensive optimization [50). The applied strategies shown are loop reordering and parallelization of for-loops. In multi-LLM setting, such strategies, or lessons, can be iteratively summarized by one few LLMs and learned by others, so that they collectively improve the code performance.
matrix-mattre multiplications,
In this work, we propose the framework LessonL (pronouced as "lesson-nell") for multiple LLM agents to collaboratively solve problems. Central the framework the lesson mechanism for agents to improve their collective intelligence through learning from each other. The main technical innovation is solicitation banking bawking--eleettiiiinee selection framework that generates, deposits, and filters lessons incurred during the collective problem-solving process. Although code optimization is the focal application of LessonL, we demonstrate its use for other tasks (code generation) as well.
LessonL is a novel multi- aggent framework that resembles how humans learn to solve problems. Compared with other collaboration frameworks, such as where agents play different roles in the solution process 49
27 46), or where agents independently propose solutions that are (see Section for literature review), our framework has few advantages. First, agents do not need to be distinguished by pre-specified roles, as their complementary strengths for particular problem may be unknown a priori. Second, communication and prompt contents are economic since lessons are more concise than codes. Third,
12
23
subsequently communicated and aggregated [52 14 28
60)

---

Based an:
solution quality;
semantic relevance;
lesson effectiveness
Agent
Lesson
Grading
Lesson bank
Problem
good
Poode al resaoo
select lessons
feil
depo
passing
passing
good
Round =
Round t=
Figure 2: The LessonL. framework (which may repeat multiple rounds).
more importantly, lessons interpretable and reusable, allowing the explication of coding knowledge and the creation of educational materials.
Our work contributes the following:
1. finding that LLMs have complementary strengths even fine level of tasks (Appendix
novel desson-based framework for multiple agents to collectively solve problems (Section 3);
3. statc-of-the-art performance on code optimization and code generation benchmarks (Section 4.2)
4. empirical evidence that team of small LLMs significantly outperform much larger LLM under similar resource consumptions (Section 4.4);
5. representative code examples and lessons (Appendices K M
2 Related Work
Multi-agent collaboration. Recent advances in prompting techniques have enhanced LLMs' rea- soning and planning capabilities for complex tasks like mathematics and coding. CoT [54], Self- Consistency (53) ReAct (58). and ToT [57] demonstrate improved problem-solving skills with
reasoning while Reflexion [49 leverages self-generated feedback stored in episodic memory to en-
hance decision decision-making. As such, LLMs can be used autonomous agents and recent research efforts propose ways for multiple agents to collaboratively solve problems. Popular multi-agent frameworks either place agents in different roles (such planner, coder, debugger, reviewer, and tester in software project) or have agents independently propose solutions, which are collectively refined and consolidated. For role-based methods, Veent [12], MetaGPT (23]. MapCoder (27. Chat-
Dev [46), self-collaboration [3], SoA (26). AgentCoder (2), Agent Hospital 34 and DEI [6]] For
individual solution proposals, MoA [52]. LLM-Debate [14) LLM-Blender (28), DyLAN (37,
and AgentPrune (60). Our work LessonL. belongs to the latter category. A unique contribution of LessonI. is the lesson mechanism for agents learn from each other and the explication of coding knowledge as result of the collaborative learning.
Code optimization. It is important, but under-explored, work focuses on specialized models such as HPC-Coder [40] 4T and HPC-Coder- V2 [9] for high- performance computing, all requiring curating and/or generating code data and fine-tuning, Besides the usual fine-tuning mne-tuning approaches, PIE [51] proposes additional adaptation techniques, including retrieval-based prompting, proaches, SBLLM [T7] retrieves optimization examples from an external dataset and Self- Relf-Refine [38] iteratively refines the code based self-generated feedback. In contrast, multi-agent framework allows the use of more than one agent and it does not rely on an external code dataset.
of LLM for code. Prior
and self-play. Among the few agentic ap-
For extended discussions and more related work, Appendix B

---

3 The LessonL Framework
Our multi-agent collaboration framework centered around lessons, which can be any knowledge or information that helps an agent better solve the problem at hand. When a team of agents participates in the collective problem-solving process, such knowledge solicited through inspecting each agent's solution and is deposited into a bank for other agents to access, Hence, the solution process becomes iterative, looping between using the existing lessons update the solutions and using the updated solutions to generate new lessons. For code optimization, such an iterative process can progressively improve the code performance. This process is pictorially illustrated in Figure [ and sketched in Algorithm
Algorithm The LessonL. Framework (Sketched)
1: Each agent generates initial solution and the lesson for it. Deposit lessons the bank.
2: for round t = ...... T do
3: Select k lessons from the bank.
4: Based on the selected lessons, each agent generates an updated solution.
5: Each agent generates new lesson for the updated solution. Deposit lessons to the bank.
6: Adjust the effectiveness of the k selected lessons.
7: end for
8: Return the best solution.
In what follows, we elaborate on few key components of this framework. The full algorithm is We also discuss extension to other coding tasks, such as code generation.
given Appendix C
3.1 Lesson Solicitation
Every solution comes with lesson, either positive or negative, Such lessons explain why the solution correct what makes it wrong. For code optimization, multiple tools be used grade the output code, such as the compiler, the clock, test cases, and profilers. Consider four resulting scenarios:
(a) speed up, (b) slow down, (c) functional incorrectness, (d) syntax error.
We solicit lessons by referring the original code and the modified code, A and B, respectively, and prompting the agent with the resulting scenario and asking for explanations. For (a) and (b), we supplement the scenario description with the measured speedup and for (d), we include the error message reported by the compiler. Such information helps the agent reason with the code pair and articulate precise lessons. For (c), we do not include the test cases because LLMs tend to reason specifically about the test cases, resulting in insufficiently general lessons.
The detailed prompt templates are given in Appendix D
3.2 Lesson Banking and Selection
In each round, n lessons are generated and deposited to the bank, one from each agent. Then, after several rounds, there accumulate many lessons. It is unwise to feed all of them to the prompt when asking the agent to improve the code, because they may exceed the prompt capacity and also the token consumption can become too costly. Hence, the framework is run with at most & lessons in each round. In practice, it suffices to set k: be, slightly larger than, n.
set of lesson selection criteria is in place. First, naturally we want to use lessons about high speedups, because they point the right directions of optimization. However, positive lessons alone cannot address certain limitations of LLMs, such as the lack of guarantee on code correctness. Hence, negative lessons are still valuable, because they can help the agents avoid similar mistakes. It is, however, challenging to decide which negative lessons are more important than others. Finally, we also consider relevance, in case agent hallucinates irrelevant lessons. For this, treat the original code as the query and retrieve semantically relevant lessons from the bank, in a manner similar to retrieval augmented generation 33
Algorithmically, the selection mechanism goes as follows. If the bank has no more than & lessons, pick them all. Otherwise, sort the lessons according to speedup (more on this in the next subsection)

---

and pick the top [k/2]. Then, sort the remaining lessons according to their cosine similarity with the original code and pick the top |k/2]. Any powerful embedding models for code can be used to compute the cosine similarity; for example, CodeBERT [6]
3.3 Effectiveness Adjustment
The effectiveness of lesson is naturally measured by the speedup 8 - the higher the speedup, the more useful the lesson. Note that s was calculated when words, s is the speedup of code y over the original code. One's 's view on the effectiveness of a may change when this lesson is selected to apply later. For example, if the lesson later yields code y with a worse speedup s' < 8, should we keep selecting it only because the original 8 is great?
created together with code y. In other
To more effectively select high-speedus lessons, we need some adjustment to the speedup dynamically. Rather than sorting them according to s, we introduce an adjustment factor f and sort the lessons according to instead, where f accounts for the performance of the lesson when actually applied Specifically, when is applied in some round t, it incurs speedups a(*) for each agent j. We initialize correction variable with zero and add c to it whenever s 8, or add e when 8, for some E (0, 1). After looping over all n agents, the adjustment factor defined as c/ n. value greater than means more output codes enjoy a speedup > s by applying the lesson.
3.4 Extension to Other Coding Tasks
The LessonL framework is general. It can be extended to other coding tasks provided that the key components discussed above are properly adapted. For example, in Python code generation, lessons needed to be solicited for only one scenario: functional incorrectness, because if the output code passes all test cases, the iteration immediately terminates. We give the prompt templates in Appendix for completeness. Additionally, will still perform lesson banking, but A lessons selected based on the number of test cases passed and the semantic relevance instead of speedup.
4 Experiments
We perform a comprehensive set of experiments to evaluate LessonL. The main finding is that LessonL. enables an effective collaboration among LLMs to perform code optimization and other coding tasks. Using the LessonL. framework, a team of small LLMMs that collaborate through the sharing of learned lessons can outperform larger LLMs and multi-LLM collaboration methods.
4.1 Setup
Benchmarks. We use six coding benchmarks to perform the evaluation. (1) ParEval 39 includes 60 coding tasks (per programming mode) related to scientific and parallel computing. We experiment with the serial and OpenMP modes, as they are less demanding on the computational resources required for evaluation. ParEval was originally designed for code generation (i.e., write the code given verbal description). We adapt it to code optimization (i.e., write faster version of a given source code); see the adaptation details in Appendix FF] (2) PolyBench [45] contains 30 numerical tasks with static control flows from domains like linear algebra, image processing, physics, and statistics, adapted for code optimization by us. (3) HumanEval [T]] consists of 164 programming problems, assessing language comprehension, algorithms, and basic mathematics.
(4) HumanEval+ (36 extends HumanEval with 80x more test cases. (5) MBPP 4 consists of around 1,000 crowd-sourced entry-level Python problems covering fundamentals and standard library use. (6) MBPP+ (36) extends MBPP with 35x more test cases. The last four benchmarks evaluate code generation.
LLMs. We use seven models in total. Three are open-source, small models: Deepseek7B (deepseek- 7tw.5--Coorr--BBIIntieee
25). and Qwen14B (Qwen2.5- Coder- Mieee-11BBInnttuue Instruct) 25:; two are GPT models not trained with reasoning data: GPT-4o mini
Noder-7b-inttruct--l 20), Qwen7B
(gpt-4o-mini) [42 and GPT-40 (gpt-4o) 43; and two are reasoning models: DeepseckR1-14B
-1-st-------tt---ee 14B) [19 and OpenAl o3 (o3) 44 The open-source models are used to evaluate multi-agent collaborations; GPT-4o mini is closed-source but its size is comparable to the open-source models; while GPT-4o is much larger and it sets a strong baseline for single-agent
5

---

performance, Additionally, DeepSeckR1-144 and OpenAI o3 are two recent reasoning models that make use of the test-time scaling approach achieve strong reasoning capabilities.
Baselines. We compare Lessonl. with four categories of models/methods. (1) Single-agent standard prompting. All the above non-reasoning L.LMs are experimented with. Among the open-source models, preliminary findings suggest that Qwen I 4B slightly outperforms the other two (see Table 5 in Appendix A): hence, we omit the results of these two in subsequent tables to save space. (2) Single-agent trained with reasoning data. Both DeepSeckR1-14B and OpenAl o3 are off-the-shelf
models equipped with reasoning capabilities. (3) Single-agent reasoning or reflection. We use Qwen 14B as the agent and experiment with CoT [34] and Reflexion [49 CoT applies a chain-style thought process to reason about the steps, while Reflexion iteratively reflects on the task feedback to improve the solution. (4) Multi-agent collaboration. We experiment with MapCoder [27] and MoA [22. MapCoder uses agents for example retrieval, solution planning, coding, and debugging. For our purpose, all agents are Qwen14B. In contrast, each agent in MoA independently codes the solution and refines the aggregated solution. We use the open-source models as agents and use GPT-4o as the aggregator. Similarly, in our framework LessonL, the agents are open-source models.
For details on baseline implementation, hyperparameters, hardware, and timing, see Appendix G
4.2 Benchmark Results
We evaluated the performance of LessonL code optimization and code generation tasks. For code optimization, we studied the ability of LessonL to optimize serial and parallel code drawn from the ParEval and PolyBench benchmarks. For code generation, we investigated LessonL's 's performance on HumanEval, HumanEval+, MBPP, and MBPP+.
Code optimization task. In this task, models are given a correct program and are tasked with generating a faster version of that code while maintaining correctness evaluated by using test cases. The speedup achieved by model is measured the ratio of the runtime between the original and the new code. We evaluate the performance achieved over the set of codes in the benchmark. The geometric mean is preferred over the arithmetic mean because it is more resilient to large outliers that can cause a single code to unduly influence the average. For example, when using the arithmetic mean an algorithmic optimization that improves the asymptotic runtime of code from ((n²) to (nn log n) could result in a 1000 speedup for sufficiently large input, and result in the arithmetic average becoming poor measure for model's ability to optimize diverse sets of codes. In the case where a new code is incorrect or slower than the original, we consider the speedup to be 1. Similar to [51], which uses the same convention, our rationale is any new code may be discarded in favor of keeping the original code.
model by measuring the geometric mean speedup
Code optimization results. Table presents the results. We compare the performance of LessonL with three single-agent models (Qwen14B, GPT-4o-mini, and GPT-4o), two reasoning models (DeepseckRI-14D -14BB and OpenAl o3), and four prompting / agentic methods (CoT, Reflexion, MapCoder, and MoA). We report each method's 's average speedup. proportion of correct code, and the proportion of codes that achieve a speedup greater than 2x.
few observations follow. First, LessonL. achieves either the best or the second best results across metrics and benchmarks (with OpenAl 03 taking the other position generally). This highlights the attractiveness of LessonL's 's novel collaboration mechanism, which enables agents to learn form each other and iteratively improve solutions. Second, multi-agent collaborations generally surpass single-agent approaches (except using OpenAl o3 which comes with native reasoning). MapCoder and MoA, which use role specialization or solution aggregation, are often second to I LessonL. Third, achieving large speedups is more challenging for serial code than for OpenMP code. In serial mode, most methods yield an speedup below 2x, with less than 20% of solutions surpassing 2 speedup. Conversely, in OpenMP mode, most methods achieve more than 2x speedup, with more than 40% of solutions surpassing 2x. This is expected, as many scientific computing and image processing tasks are inherently parallelizable. For instance, LessonL. achieves an average 3.4x speedup with 8 threads across both benchmarks.
We additionally report the improvement of LessonL. over the single-agent models Appendix J.I Table 10 As motivated earlier, these models have complementary (but unknown a priori) strengths. I Lessonl. improves over them on majority of the problem categories (some highly significant), while experiences a minor degradation the remaining categories due to randomness in LLM generation.

---

Table Comparison of model/method performance for two code optimization benchmarks. "Correct" means correctness; "> 2x" means proportion of problems achieving speedup >> 2x; "Speedup" is the geometric mean speedup across all problems in the benchmark. The results are reported as the mean and standard deviation over three runs.
ParEval
Serial mode
OpenMP mode 2x
Correct
2x
Speedup
Correct
Speedup
Qwen14B
0.67 0.03
0.14 + 0.01
1.60 + 0.03
0.67 + 0.01
0.48
0.00
2.28 + 0.02
GPT-4o mini
0.77 + 0.03
0.14 + 0.01
1.57 + 0.12
0.70 + 0.03
0.55 + 0.04
2.72 0.00
GPT-40
0.80 0.00
0.16 0.03
1.12
0.73 0 0.05
0.58
0.05
2.93 0.30
DeepscckR1-14B
0.56 0.05
0.12 0.05
1.51
0.44
0.31 301+000 0.03
1.68 0..07
OpenAI o3
0.77 0.02
0.04
1
0.00
0
3.55 = 0..7
CoT
0.61 0.03
0.12 0.03
1.57
0.51
2.31 = 0.03
Reflexion
0.67 0.03
0.18 +100
1.88 0.20
0.64
0,46 0.02
2.27 1 0.06
MapCoder
0.88 + 0.02
0.15
1.85
0.83 = 0.05
0.58 0.02
3,43 = 0.17
MoA
0.73 0.04
0.16
1.76
0.74 - 0.04
0.59 0.02
2.77 = 0.11
LessonI.
0.91 0.02
0.21 + 0.01
2.16 + 0.11
0.11
0.86 0.01
0.62 0.02
3.46 0.03
PolyBench
Serial mode
OpenMP mode
Correct
2x
Speedup
Correct
2x
Speedup
Qwen14B
0,46 + 0.12
0.02 + 0.02
1.03 +: 0.04
0.49 + 0.02
0.46 = 0.02
2.21 +:: 0.06
GPT-4o mini
0.41 0.08
0.06 - 0.02
1.10 0.03
0.55 + 0.07
0.48 0.05
2.42 0.31
GPT-4o
0.51 0.02
0.06 0.02
1.12 + 0.03
0.60
0.55 0.04
2.73 0.21
DeepseckRI-14B
0.18 0.04
0.04
1.08 1 0.07
0.30 0.03
0.25 0.04
1 0.09
OpenAl o3
0.64 -.4:0000
0.28 0.02
1,71 = 0.21
0.21
0.72 0.04
0.56 0.02
3.15 3.15 0.23
CoT
0.03
0.02 - 0.04
1.06 + 0.04
0.39 +
0.37 0.04
1.116
Reflexion
0,41 0.02
0.08 0.07
1.12
0.56 0.23
0.52 = 0.04
=
MapCoder
0.64 + 0.02
0.04
1.07
0.55 + 0.04
0.44 1 0.05
2.23 = 0.18
MoA
0.32 0.07
0.08 + 0.02
+.04
0.61 0.07
0.54 0.05
2.70 0.34
LessonL
0.77 0.04
0.04
1.32
0.71
0.62 0.02
3,40 = 0.12
Table 2: Comparison of model/method performance (pass@) @1 1) for four code generation benchmarks. GPT-4o and GPT-4o mini results are taken from the EvalPlus leaderboard [36].
HumanEval
HumanEval+
MBPP
MBPP+
Qwen14h 14B
0.866
0.817
0.831
0.741
GPT-4o mini
0.884
0.835
0.854
0.722
GPT-4o
0.884
0,872
0.876
0.722
CoT
0.884
0.848
0.854
0.741
Reflexion
0.902
0.841
0.889
0.765
MapCoder
0.890
0,823
0.862
0.679
MoA
0.909
0.872
0.897
0.770
LessonL. 0.915
0.878
0.899
0.765
In the appendix section, show example (Problem 40 of ParEval) where agent critically relies on the lessons from other agents to successfully solve the problem.
Code generation. While code optimization is the main testbed of LessonL, also experiment with a widely studied coding fask-code generation. Table 2 compares the results for four commonly used benchmarks by using the correctness (pass @ 1) metric. We that LessonL is the best in three out of four benchmarks, with MoA coming next. In the remaining benchmark, the ranking positions of these two frameworks swapped. This observation echos that of code optimization, corroborating the usefulness of multi-agent collaboration for solving coding tasks and highlighting the effectiveness of our lesson framework.
4.3 Ablation Study
conducted additional studies on Lessonl. to investigate the impact of its design components, the number of collaboration rounds, and the number of agents.

---

Table Ablation study. Benchmark: ParEval (serial and OpenMP modes). The results are reported as the mean and standard deviation over three runs.
Serial mode
OpenMP mode
Correct
> 2x
Speedup
Correct
Speedup
(0) LessonL.
(1) high-speen speedup only
0.2 0.02
0.21 0.01
2.08 0.04
0.83 0.02
0.58 = 0.00
3.20 =
(2) & high-relevance only
0.91 + 0.02
0.18 + 0.02
196 + 0.88
0.84 = 0.02
0.61 + 0.01
3.40 0.16
(3) No speedup adjustment
0,92 0.03
0.21 0,03
2.05 + 0.05
0.86 = 0.02
0.61 + 0.02
3.28 = 0.10
(4) Random & lessons
0.91 0.02
0.19 0.02
2.03 0.05
0.82 0.02
0.61 +: 0.11
3,47 = 0.28
(5) No lessons
0.89 0.01
0.20 0.00
2.05
+
LessonL
2.2
MoA
Reflexion
Table 4: Performance comparison across dif- ferent numbers of agents. Benchmark: ParEval (serial mode).
2.1
MapCoder
Spee3ug Seedup
2.0
GPT-4u
1.9
1.8
#Agents
Correct
> 2x
Speedup
1.7
1 0.67
0.14
1.60
3
0.90
0.23
2.12
4
0.93
0.22
2.14
1.5
2
3 Round
5
5
0.93
0.20
2.09
6
0.95
0.23
2.21
Figure 3: Performance over rounds (or called "layers"). Benchmark: ParEval (serial mode).
Ablating components of the framework. We analyzed the lesson selection mechanisms. The results are illustrated in Table and compare five variations of the LessonL framework: (0) the full version of LessonL; (1) lessons selected based only on speedup; (2) lessons selected based only on relevance; (3) lessons selected based on speedup and relevance, but without speedup adjustment; (4) random selection of lessons; (5) no lessons used.
Table reports the results. We see that most ablated variants suffer a decrease of speedup. In serial mode, variants without high-speedup lesson prioritization (2,4) suffer most, while OpenMP mode, variants (1.3) show larger performance drops. This trend also applies to the proportion of problems with >2x speedup. This highlights different optimal strategies for each mode. Serial mode benefits from high-speedup lesson selection with dynamic adjustments, while OpenMP mode favors high-relevance lessons and speedup adjustments. Interestingly, in serial mode, the correctness metric sometimes benefits from ablations, even though the relationship between lessons and correctness is indecisive. While variant (4) marginally outperforms LessonL in OpenMP speedup (+0.01), LessonL demonstrates significantly better stability (0.03 vs 0.28 standard deviation). Finally, The consistent underperformance of variant (5) across metrics confirms that lessons are fundamental to LessonL.
Varying the number of iterations. Figure 3 plots the performance (speedup) achieved by Lessonl. and alternative methods when varying the number of collaboration rounds. The concept of "round" is different across methods: for MoA, a round is a "layer", and for MapCoder, it is "debugging round". The performance of GPT-4o also included to provide baseline for comparison. While LessonL and Reflexion consistently benefit from using more rounds (see Appendix [.2] Table II] where we push the number of rounds for LessonL to 10), the same not for MoA and MapCoder. The performance of MoA actually decreases when increasing the number of rounds, and there is no clear trend when using more than two rounds MapCoder. Even when using a much smaller model, Reflexion surpasses GPT-4o and continues to improve when further increasing the number of rounds. contrast, MoA's performance drops below GPT-4o when using more than three rounds, though MoA actually uses the GPT-4o model aggregator.

---

OpenAl 03
OpenAl 03
2.1
LessonL
2.1
LessonL
2.0
2.0
59eedup
Reflexion
Speedup
1.9
1.9
1.8
MapCoder
MapCoder MoA
GPT-4o
MoA
GPT-4o
Qwen14B(20)
Qwen148(20)
0wen143(1)
Owen144B(1
Deepseekfl11188 CoT(20)
BeepseekR]-1es
CoT(20)
1.6
GPP-44 mini
1.6
(Co1:11
(1)(1)
GPT-4o
0.4 Cost in dullars
10
20
30
40
50
60
PFLOPS
Figure 4: Performance versus costs and latency. Benchmark: ParEval (serial mode). The dashed line the Pareto front.
We studied the outputs of each method and provide examples in Appendix H For program that finds the ktth smallest element of an array, observe that MoA implements the divide-and-conquer algorithm in early rounds, achieving high speedup. In subsequent rounds, however, MoA introduces overheads and eventually resorts slower code. In contrast, in the example of performing convolutions an image, Lessonl. increases the speedup step by step by removing redundant zero-additions, separating boundary and interior computations, and avoiding extra data copies. These examples illustrate how the lesson mechanism can selectively inject useful information to aid improvements in coding solutions.
simple solution that calls std::thh_ed seeettt_eeemet,, which results in
Varying the number of agents. To see how LessonL. performs in larger scale, we added three more agents order: Llama-3.1 1...3.88-IIntteee models have similar size to the three already used. The results in Table 4 suggest that generally, using more agents lead to better performance. This observation uniform for all metries. However, the greatest improvement comes from using three agents; using more generates diminishing returns.
[8] Been2.5-7B-Instruue [48, and phi-4 (14B) (2] These
4.4 Cost Analysis
Code performance is not the only dimension that determines the effectiveness of a code model, considering the complex interplay of monetary costs, time costs and LLM calls in single-LLM and multi-LLM methods. We investigate where LessonL. stands for the code optimization task by considering budget constraints with respect to money and time. For this, we follow [52] and extract the pricing information of LLMs from API providers or guesstimate the price based on similar models. We also use the number of floating point operations (FLOPS) as a proxy of latency when considering the time cost. See Appendix [ for the detailed estimation. The estimation may be time sensitive (for example, price varies over time and across providers) and may include educated guesses (for example, the sizes of closed- source models), but it paints a reasonable picture of the performance landscape.
Figure 4 shows two preedup-versus-oon plots for ParEval, with the Pareto front indicated by the dashed line. Besides the afore--mnnioneed models/methods,, we add Qwen 44((20) and C2T((20), which prompt the respective model/method 20 times and select the fastest code (an inference-time scaling approach (7). We see a few clusters from the plots. First, Lessonl. Pareto optimal because of its superior speedup and reasonable costs compared with competitors. Second, Qwen 14B, DeepseckR1 1- 14B, GPT-4o mini, and CoT are Pareto optimal, or nearly optimal, because of the low costs in both money and time. Third, OpenAI o3 also Pareto optimal because of the superior speedup it achieves, despite at much higher cost. We consider all these models/methods to be cost-effective. For the remaining methods, inference-time scaling (CoT(20) and Qwenl not help and neither do Reflexion or multi-agent methods. In fact, MapCoder and MoA appear to be the least cost-effective, because they not only iterate multiple times, but also use multiple agents. This signifies the challenge of multi-agent methods, which are generally the most costly, and in contrast reflects the attractiveness of LessonL. Finally, compared with the much larger model GPT-4o, LessonL with small models yields significantly better speedups by consuming similar resources.
does

---

4.5 Case Study
few examples reveal the interesting lessons learned by the LLMs in our framework; see details in (aaandiessKGGaoooole (Geometry) (DFT). In the Geometry example, the original code finds the smallest distance among n points two dimensions by using the straightforward (nn)) implementation. The final optimized code uses divide-and-conque
Appendices K
algorithm that reduces the complexity to O(n log n) and achieves a 74.31x speedup. This optimization is a complete change of algorithm and it is nontrivial even for human programmers. Several of the lessons affirm the benefit of divide-and-conuee while others point out syntactic errors and discourage the agents to repeat them.
More interesting is the DFT example, where the original code implements the straightforward algorithu in (N2) cost. is well known that DFT can be implemented by using the FFT algorithm in O(N log N) cost and some agents try to implement it, but they fail with subtle bugs. The lessons alert these failures and eventually the agents choose to optimize the code by precomputing the repeatedly used exponentials, yielding considerable 10.83x speedup. Moreover, one lesson stands out by stating a syntactic error caused by using Intel intrinsic instructions. The evaluation pipeline and hardware do not support particular intrinsics and the involved agent later refrains from using them in the implementation.
There are also occasions when a problem is too challenging for the small LLMs to handle, even when they collaborate. We show an example in Appendix M (sum of prefix sum). The example involves various failures an LLM could produce: syntax errors, semantic errors, and lack of speedup. One mitigation integrate external knowledge to complement the self-reflected lessons.
5 Conclusions
We have presented desson-based framework LessonL for multiple agents to collaboratively solve coding problems. This framework allows each agent to learn from other agents' successes and failures to improve its own solution. Empirical evaluations indicate that with lessons learned over the course, our framework outperforms other methods to achieve state-of-the-arr performance, and a team of small L.LMs can significantly outperform a much larger LLM under similar resource budget. Case studies show that the agents perform many skillful optimizations, such performing divide-and-conquee and precomputation. An avenue future research is to improve the autonomy of agents, including their decision- making ability in lesson selection.
Limitations. Even though experiment results demonstrate competitive latency of LessonL compared with large LLMs, learning from lessons will defer the time to first token, because lesson solicitation and subsequent trials are overheads. This may affect user experience. However, the negative impact may be offset by the rich and educational Furthermore, LessonL. has so far been applied only to function-level code snippets rather than repository-level tasks such as those in SWE-bench (29). Extending LessonL. to broader coding scenarios remains an important direction for future work.
delivered together with the solution.
Broader impacts. Code optimization is an important step of the software development cycle, especially for domains such as high performance computing, real-time animation, and transaction systems. This work develops a cost-effective approach to improving programmer productivity, by using LLMs to suggest optimized code. The interpretability of lessons lowers the barrier of mastering the low-level system knowledge required for optimization. Nevertheless, LLM outputs are not without flaws and practitioners should verify the lessons if they plan to apply the knowledge elsewhere.
Acknowledgments and Disclosure of Funding
The research was supported by the National Science Foundation (NSF) under grant numbers NSF2406647 and NSF-2406648. This research was sponsored by the MIT-IBM Watson AI Lab and part by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19- 1--50-19---10000
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
10

---

References
[1] Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. Medec: A benchmark for medical error detection and correction in clinical notes. arXiv preprint arXiv:2412.19220 2024.
[2] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, and Piero Kauffmann et al. Phi-4 technical report. arXiv preprint wrXiv:2412.088055 2024.
[3] Andreas Abel and Jan Reineke. nanobench: A low-overhead tool for running microbenchmarks on x86 systems. In 2020 IEEE International Symposim on Performance Analysis of Systems and Software (ISPASS), page 34-46. IEEE, August 2020.
[4] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, and Quoe Le al. Program synthesis with large language models. arXiv preprint arXiv: 00730773, 2021.
[5] Denis Bakhvalov. Performance ninja class, 2022.
[6] Lei Zhang Binyuan Hui. An awesome and curated list of best code-llm for research. http:/b..oom/uuyyryyyy Saapteerr//weessmm-ooo-eeeeee
ht
2023.
[7] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini, Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv: arXiv:2407.21777, 2024.
[8] Tianle Cai, Xuezhi Wang. Tengyu Ma. Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023.
[9] Aman Chaturvedi, Daniel Nichols, Siddharth Singh, and Abhinav Bhatele. Hpc-coder-v2: Studying code ilms across low-resource parallel languages. arXiv preprint 15177,, 2024.
[10] Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305. 05176, 2023.
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
[12] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, and Chen Qian et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024.
[13] Yihong Dong, Xue Jiang, Zhi Jin and Ge Li. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engineering and Methodology, 33(7):1-38, 2024.
[14] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In Forty-first International Conference Machine Learning. 2023.
[15] Andrew Estorell and Yang Liu. Multi-LLM debate: Framework, principals, and interventions.
In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[16] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002. 08155, 2020.
[17] Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael Lyu. Search-based Ilms for code optimization. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 254-266. TEEE Computer Society, 2024.
[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan et al. The llama 3 herd of models. arXiv preprint 2rXiv:2407.21783 2024.
[19] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and Xiao Bi et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
11

---

[20] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, and YK Li et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401. /4/ 14/96, 2024.
[21] Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. Chatllm network: More brains, more intelligence. Al Open, 2025.
[22] Samuel Holt, Max Ruiz Luyten, and Mihaela der Schaar. 1.2mac: Large language model automatic computer for extensive code generation. arXiv preprint arXiv:2310.02003, 2023.
[23] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, and Liyang Zhou et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2338.003355 3(4):6, 2023.
[24] Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312. 13010, 2023.
[25] Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, and Keming Lu et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
[26] Yoichi Ishibashi and Yoshimasa Nishimura. Self-organized agents: A Ilm multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.02 2024.
[27] Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405. 1403, 2024.
[28] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.022 02000,,.2.11.. 2023.
[29] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770. 2023.
[30] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language 2020.
models. arXiv preprint arXiv:2001.088
[31] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[32] Charles E Leiserson, Neil C Thompson, Joel S Emer. Bradley Kuszmaul, Butler W Lampson. Daniel Sanchez, and Tao B Schardl. There's 's plenty of room at the top: What will drive computer performance after moore's law? Science, 368(445)) 2020.
[33] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, and Tim Rocktäschel et al. Retrieval- augmented generation for knowledge-intensiee nlp tasks. In Proceedings of the 34th Interna- tional Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc
[34] Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, and Weizhi Ma et al. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprini arXiv:2405.022 029557 2024.
[35] Jonathan Light, Yue WU, Yiyou Sun, Wenchao Yu, Yanchi Liu, Xujiang Zhao, Ziniu Hu, Haifeng Chen, and Wei Cheng. Sfs: Smarter code space search improves IIm inference scaling. In The Thirteenth International Conference on Learning Representations.
[36] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Shirth- Conference on Neural Information Processing Systems, 2023.
[37] Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic --owerd agent network for taik-triented agent collaboration. First Conference on Language Modeling, 2024.
12

---

[38] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uni Alon, Nouha Dziri, Shrimai Prabhumoye, and Yiming Yang al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534- 46594, 2023.
[39] Daniel Nichols, Joshua H. Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele. Can large language models write parallel code? In Proceedings of the 33rd International Symposium on Hirf-PPeforrance Parallel and Distributed Computing. HPDC '24, page 281-294. ACM, June 2024.
[40] Daniel Nichols, Aniruddha Marathe, Harshitha Menon, Todd Gamblin, and Abhinav Bhatele. Hpe-coder: Modeling parallel programs using large language models. In ISC High Performance 2024 Reseurch Paper Proceedings (39th International Conference), page 1-12. IEEE, May 2024.
[41] Daniel Nichols, Pranav Polasam, Harshitha Menon, Aniruddha Marathe, Todd Gamblin, and Abhinav Bhatele. Performance-atigned Ilms for generating fast code. arXiv preprint arXiv:2404. 18864, 2024.
[42] OpenAL. Gpt-4o mini: Advancing cost-efficient intelligence, July 2024. Accessed: 2025/02/14.
[43] OpenAl. GPT-4 Open System Card. Technical report, OpenAI, 2025. Accessed: 2025-02-13.
[44] OpenAl. Introducing openai o3 and 04-mini openai, 2025.
[45] Louis-Noël Pouchet, Tomofumi Yuki, and Matthias J. Reisinger. Polybench/c 4.2.1. https 2016. Accessed: 2025-01-22.
//github.
[46] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang. Weize Chen, Yusheng Su, and Xin Cong al. Chatdev: Communicative agents for software develop- ment. arXiv preprint 1rxXiv2307.079244 2023.
[47] Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, and Cheng Yang et al. Scaling large language model-based aueti-agent collaboration. In The Thirteenth International Conference on Learning Representations, 2025.
[48] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, and Dayiheng Liu et al. Qwen2.5 technical report, 2025.
[49] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8635 2023.
[50] Julian Shun and Charles E. Leiserson. Mit 6.2 ocw, 2018
[51]
Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Yiming Yang. Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, and Osbert Bastani et al. Learning performance-improving code edits. In The Twelfth International Conference on Learning Representations, 2024.
[52] Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities, 2024. URL https:/arxiv. org/ubs/2406.04669
[53] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
[54] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. -f prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
[55] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, and Morgan Funtowicz et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.0377) 2019.
[56] Qingyun WU, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, and Jiale Liu al. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling, 2024.
13

---

[57] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths. Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:1 1809-118222 2023.
[58] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[59] Zhangyue Yin, Qiushi Sun, Cheng Chang, Qipeng Guo, Junqi Dai, Xuanjing Huang, and Xipeng Qiu. Exchange-of thtthogght: Enhancing large language model capabilities through cross-model communication. arXiv preprint Xr3: /2.01823, 2023.
[60] Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng. Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for LLM-based multi-agent systems. In The Thirteenth International Conference on Learning Representations, 2025.
[61] Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei
Li, Renze Lou, and Jiacheng Xu Diversity empowers intelligence: Integrating expertise of software engineering agents. arXiv preprint arXiv:2408.07000
2024.
[62] Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. Unifying the perspectives of NLP and software engineering: survey language models for code. Transactions on Machine Learning Research, 2024.
[63]
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hin prompt- ing improves reasoning in large language models. arXiv preprint (rxXi:22304.09799) 2023.
[64] Andy Zhou, Kai Yan, Michal Saapenttoh-Roothaaae Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint wrXiv:2310.04406, 2023.
[65] Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Language agents optimizable graphs. arXiv preprint arXiv:2402.16823, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The information is Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The reproduction information is provided in Section 4 and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification: At the paper submission stage we do not include code because of the long running time perform one experiment (which compares large amount of baselines), but we will release the code on publication of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc/
ptgllic/uuudes//ddeuuubmissss seseesuubbssioooooccyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips cttpssccccbllliiccc gtt/giiiee///oeeeeeeeeeeeeee/e// ssssssuuisssssssssssiiissiiiiiis ssionPol coyyl for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The information is in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include error bars whenever appropriate.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The information is in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not release models (because do not train new models), This work will release a benchmark that is modified from an existing, publicly available benchmark. The modification (as described in Section (F) is mostly format but minorly in content. The modified content does not introduce new risks compared with the original content.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The benchmarks used in this paper are in the public domain and under permissive licenses. The benchmarks are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. wwt.cooo//aaaaeee datasets
19

---

has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The code assets produced from this work will be released upon publication in the public domain and under permissive licenses.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: This paper develops method for LLM agents collaboratively solve coding problems.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy http::///ferrrrssccccoooorrerrrs for what should or should not be described.
21

---

Appendix
Complementary Strengths of LLM Agents
24
More Related Work
25
Algorithm of LessonL
25
Prompt Templates for Code Optimization
26
D.I Initial Prompt
26
D.2 Lesson Solicitation
26
D.3 Subsequent Prompt With Lessons
27
Prompt Templates for Code Generation
28
E. Initial Prompt
28
Lesson Solicitation
29
Subsequent Prompt With Lessons
29
ParEval Modilication for Code Optimization
29
F1
Adaptation from ParEval
29
F2
Benchmarking
30
F3
Input Generation
30
Other Fixes
30
Additional Information of Evaluation Setup
30
Baseline Implementation for Code Optimization
30
G.2
Baseline Implementation for Code Generation
31
G.3
Hyperparameters, Hardware, and Timing
H
Supplementary Information for Figure] 3
32
Lesson
32
MoA
36
H.3 MapCoder
39
Reflexion
Supplementary Information for Figure 4
46
Additional Experiment Results
47
J.1
Complementary Strengths of Agents
47
J.2 Running LessonL for More Rounds
K Case Study: Geometry
48
Case Study: DFT
52
22

---

M Case Study: Sum of Prefix Sum (A Failure Example)
55
Supporting Code
57
23

---

A Complementary Strengths of LLM Agents
Section mootivate the use of multiple LLM agents to solve coding tasks, because is unknown priori which coding agent performs the best for each problem. We supplement the discussions with benchmarking results on the adapted ParEval benchmark [39] here (see Appendix F for the adaptation details). (egureeJcoopaane the overall performance of five LLMs, three open-source and two closed-source. Specifically, the models are:
depssekk-ooee77--sstuutttt..... abbreviated Deepseek7B;
Qwen2.5-Codder-BB-Be abbreviated as Qwen7B:
Monnn:5-Coder-14B-Imae abbreviated as Qwen14B:
gpt-4o-mini, abbreviated as GPT-4o mini;
gpt-4o, abbreviated as GPT-4o.
transform (1.04)
sparse la (1.77)
scan (8.67)
Figure 5: Comparison of models on their coding abilities for each category of the ParEval benchmark (serial mode). The coding ability is measured by the speedup when asked to optimize a code. Geometric mean of the speedups is taken over all problems in the same cat- egory. The number in the parenthesis the maximum speedup for the corre- sponding category over all models. Per- formance for each category normal- ized by the maximum speedup.
(2.21)
(1.57)
ftt (1.72)
graph (1.16)
geometry (8.09)
(1.07)
Deepseek/1
Owen7B
stencil (1.94)
sort (2.72)
Owen14B
dense la (1.11)
GPT-4o mini
GPT-40
Moreover, in Table 5 break down their performance into each category of the benchmark. We also highlight the best one among the open-source models. One sees that each model excels some of the categories but no dominates others.
Table 5: Speedup achieved by model for each category of the ParFvaal benchmark (serial mode). The speedup computed as the geometric mean. Three open-source models (i.e., excluding GPT-4o mini and GPT-4o) compared with the best boldfaced.
Deepseek7B
Qwen7B
Qwen14B
GPT-4o mini
GPT-4o
graph
1.00
1.00
1.00
1.16
1.00
histogram
1.57
1,00
1.52
1.00
1,00
scan
5.95
5.91
5.91
5.95
8.67
transform
1.01
1.00
1.01
1.04
1.01
sparse_la
1.45
1.00
1.70
1.65
1.77
reduce
2.17
2.18
2.21
1.89
2.16
fft
1.00
1.00
1.02
1.08
1.72
geometry
2.84
8.09
2.84
3.08
3.07
stencil
1.12
1.11
1.12
1.30
1.94
dense
1.02
1.01
1.01
1.01
1.11
sort
2.31
1.00
2.35
2.51
2.72
search
1.00
1.07
1.00
1.00
1.00
Overall
1.57
1.50
1.59
1.58
1.78

---

B More Related Work
8 proposes tooling framework An LLM can be used a problem solver in many ways. LATM wherein an LLM can create tools on its own and reuse them later. LATSS 64 shows that LLMs can be integrated with Monte Carlo tree search for more effective planning. PHP (3
is iterative framework where solutions are refined based on hints. L2MAC [22] introduces system that mirrors the von Neumann architecture of a computer, utilizing a prompt program and file storage generate long and coherent outputs. SFS [35] frames code generation as a black-box optimization problem, employing search techniques to enhance solution diversity and efficiency.
Moreover, multiple LLMs can collaborate for problem solving. MoA [52] orchestrates multiple LLM agents in layered structure aggregate and synthesize the solution. LLM-Debate [3] enables structured inter-agent debates to reach consensus. LLM-Blender (28) ranks, filters and fuses top responses from multiple L.Mss, improving the performance on instruction following. DyLAAN (37) employs dynamic team adjustment for different tasks. AgentPrune (60) eliminates redundant com-
munication through graph pruning. GPTSwarm [65] optimizes collaboration through graph-baser interactions. ChatLLM [21] facilitates interactions among multiple dialogue- based agents, enabling them to provide feedback and collectively improve the decision- makinng process. EoT [59] promotes cross-model communication among LLMs, integrating various communication paradigms and confidence evaluation mechanism to enhance reasoning accuracy. AutoGen [56] automates the development of agent-based applications, streamlining processes such as data collection, model training. and evaluation to efficiently adapt pre- trained models specialized tasks. FrugalGPT [TO] introduces strategies to reduce the computational and financial costs associated with deploying ILMMs, focusing on optimizing model architectures and inference techniques to achieve cost-effective
performance without significantly compromising accuracy. MacNet 477 utilizes directed acyclic graphs to organize LLM agents, streamlining their interactive reasoning via topological ordering.
C Algorithm of LessonL
The LessonL framework sketched in Algorithm (Section 3 elaborated in Algorithm 2 with details, together with subroutines Algorithms 5
Algorithm 2 The LessonI. Framework
Input: code 2; number of agents, number of rounds, T: number of selected lessons, &
1: Z..1 0
D. initialize the lesson bank initial round
for agent j = I.....r n do
3: Get new code v LLM, (x)
4: Get speedup " Exec(y) (0) and lesson 2,00 (L, M1
(0) (0))
5: Initialize factor 100
Zan. Zan.add(( (0) (0) (1
> insert lesson with speedup and factor into the lesson bank
7: end for
8: for round
1 1..... T do < k, ((9)
Zamt otherwise then Zn, Zremann = SELECCTHICHSEEDDUKZZ... ent = [k/2])
P> select lessons ( used in next round
10:
select half of the lessons
11:
Zg SELLECTHOOHRLLELWACCCZZZ...
ent [k/2])
select another half
12:
( Z. U 2y
> all selected lessons
13:
end if
14:
for agent j 1, do
next round
15:
Get new code y
(2)
LLM, (x, 2())
b use selected lessons ( to prompt
16:
Get speedup of Excc(y,") (1) and lesson 20 (x, (") (() (0)
17:
Initialize factor (1)
18: Zud.add(
(()))
insert lesson with speedup and factor into the lesson bank
19:
end for
20:
ADIUSTFACTOR (8 (1))
b adjust factors for selected lessons based on performance
21:
end for
22: return best among (1) ,..., yn (0)
(T)
(T)
.....
y
25

---

Algorithm 3 SELECTHICHISEEEDDPP Lilt, cnt)
1: Sort lessons in Zaul = {(2,s, (.2, f)} in descending order of 8 x f
> speedup x factor
2: Let ZA include top ent lessons whose x f threshold
3: Zremain = 7aa'' ? Zs
4: return 7.5; 7remain
Algorithm 4 SELECCTHCHIRLLEENNCC
cnt)
1: Sort lessons in Zremain {(2, 8, f)} in descending order of cos(2, 2t)
cos(lesson, code)
2: Let Zy include top ent lessons 3:
return Zi
Algorithm 5 ADDUSTFACTORR(2),15.)
for each lesson (z, f) in Z do Initialize correction <- 0
2:
3:
for agent j ,,...,, do
4: if < then
original speedup < yielded speedup when applying the lesson Update correction - c (1 e)
5:
6:
else
7:
Update correction
(1 6)
8:
end
9:
end for
10: Update factor f + c/n
11:
end for
D Prompt Templates for Code Optimization
This section provides the prompt templates for code optimization. The initial prompt differs from subsequent prompts that starts with lessons. The lesson solicitation prompt has several variants depending on the quality of the output code (speedup, slowdown, incorrect output, compilation error).
D.1 Initial Prompt
You given piece of code written in (language) Your task is rewrite it in the same language to improve its performance (i.e., execution time). [nse_openmp) Do not change the input/output behaviors of the code. Include the generated code between " (llanguage) and "".
// Code: (source_code)
our experiments, the variable language "C++". The variable use_openmy is set to "You should use OpenMP to parallelize the code." for OpenMP prompts; otherwise it is empty.
D.2 Lesson Solicitation
The following are two functionally equivalent codes. They are compiled by using the same compiler and executed in the same environment. Code B runs (faster_on _o_ vrsslower than Code with a speedup (speedup)> x. Explain why Code is [faster or slower). Be brief in the explanations. Use only one two sentences.
// Code A:
26

---

|source_cokee code]
// Code B: [target_code]
The following two codes functionally equivalent; that is, given the same input, they produce different outputs. Explain the reasons that make Code B nonequivalent to Code A. Be brief in the explanations. Use only one or sentences.
// Code A: (source_code)
// Code B: (target_code)
The following are two codes. Code B attempts to improve the performance of Code A, but has syntactic errors. Explain why Code B cannot be compiled. You may get hints from the compiler output provided after Code B. Be brief in the explanations. Use only one or two sentences.
// Code A: [source_code)
// Code B: [target_code]
Compiler output: (compiler_output)
D.3
Subsequent Prompt With Lessons
You are given a piece of code written in (language). Your task is to rewrite in the same language to improve its performance (i.e., execution time). (use_openel [es__pppmpp Do not change the input/output behaviors of the code. Some lessons regarding correctness and performance are provided help you rewrite the code. Include the generated code between ** "(language) and
Code:
|source_code]
While you rewrite the code, consider the following lessons. If Code A and Code B appear in the lessons, Code refers the given code and Code B refers to an attempted rewrite. Code B may not be optimal and it could be even worse than Code A.
[lesson[O])
[lesson[]]]
[lesson[2]]
[lesson[3]]
Besides the above lessons, consider other optimization strategies that can more significantly improve the performance of the given code.
27

---

Each item in the lesson list can be one of the following:
"Lesson {idx) slightly improves the code performance. [lesson_contents However, despite the code performance improvement, the speedup is only marginal."
"Lesson
[idx] significantly improves the code performance. [lesson_contentc content)"
"Lesson [idx] degrades the code performance. [lesson_content]
"Lesson [idx] compromises code equivalence. [lesson_content]]
"Lesson [idx] produces non-compilable code. [lesson_content]"
E
Prompt Templates for Code Generation
This section provides the prompt templates for code generation. The initial prompt differs from subsequent prompts in that it starts with no lessons. Following common practice, we include an example in the prompt, demonstrating the function signature, function description, and function completion.
E.1
Initial Prompt
You are given function signature in (language) together with docstring that explains what the function does. Your task is to implement the function according to the docstring. You should restate the function signature and docstring. Include the generated code between "I "[language)] and " For example, given the function signature and docstring
You should respond with
[example_generaatee
### Here is the function to implement:
{function (function_signaturee_ signaturggggeeeeeeeeeee _nd docstring)
In our experiments, the variable language "Python".
The variable example function function_signature signature and docstring is
def sam(a: float, b: float) -> float: *** Return the sum two floats a .**
The variable example_generaiee is
..' PPyyhhon def sam float, float) float:
nwa Return the sum two floats a and .*.
return a
28

---

E.2 Lesson Solicitation
The following completed code is incorrect; i.e., it does not exactly reflect the description in the docstring. The code passes only (num pass_ leaving (num_fat) cases) failed. Explain why the code is incorrect (that is, why it fails some test cases). Be brief in the explanations. Use only one or two sentences.
mnmm_psss_._ _ass_casess] test cases out of [num total _asal_ssss______________________
#### Completed code: [completed_codes
E.3
Subsequent Prompt With Lessons
You are given function signature in (language] together with docstring that explains what the function does. Your task is to implement the function according to the doestring. You should restate the function signature and docstring. Some lessons are provided to help you implement the function. Include the generated code between "" (llaguuaee and "' For example, given the function signature and docstring
[example_functin
docstric))
You should respond with
### Here is the function to implement: cinetiionssinaturee _nd docstring)
While you implement the function, consider the following lessons.
[lesson[O])
[lesson[1]]
[lesson[2]]
[Iesson[3])
Each item in the lesson list
"Lesson (idx) reasons why the code does pass all test cases. [lesson_content]
F
ParEval Modification for Code Optimization
We run experiments on code optimization benchmark adapted from ParEval [9), which is an existing benchmark for LLM code generation. At high level, our code optimization benchmark tasks an LLM with optimizing piece of source code. The LLM''s response evaluated based on its correctness as well the speedup it can achieve over the source code. In this section, we discuss our adaptation and improvement of ParEval.
E.1 Adaptation from ParEval
In the original ParEval benchmark, the LLM tasked with completing a function definition. test correctness, the LLM generated code is run custom test cases and is compared to provided baseline.
In our code optimization benchmark, we use the baseline as the source code to be optimized. Some changes are made to the source code for particular problems, such as adding definitions of custom types, that the source code can run on its own.
29

---

F.2 Benchmarking
ParEval performs benchmarking by taking the average runtime of the baseline code and the LLM generated code over fixed number of runs. This approach can produce noisy results if the execution time of the baseline code or LLM generated code is very short. To address this issue, we use Nanobench, a lightweight benchmarking library [3] Nanobench varies the number of benchmarking runs based on the runtime of the code as well as its variance, thereby producing stable results on codes with shorter runtimes. Overall, Nanobench produces consistent and stable results, which ensures that LLMs are accurately evaluated on the benchmark.
F.3 Input Generation
We ensure that the randomness used when generating inputs for the baseline code and the LLM generated code are the same. Fixing randomness among the two is necessary when benchmarking as the runtime of certain problems can vary greatly depending on the specific inputs provided. For example, consider the task of finding a value in list. If the value is the first element in the list, then the runtime will be much faster compared to when the value is not in the list. We ensure all inputs are generated from a seeded random generator, and ensure that the seed is the same when running the baseline and when running the LLM generated code.
We also improve the random input generation for some problems to ensure that all cases are considered and thoroughly tested. First, we look improving the input generation for graph problems, which includes finding the number of connected components in a graph, finding the size of the largest connected component in the graph, and finding the shortest path length in graph. Previously, ParEval generated random graphs using the G(n, p) Erdos-Renyí model. Each edge in the n-vertex graph would be independently added with probability p. The issue with this type of generation using fixed probability for large graphs is that the resulting graph is almost surely connected, which means the number of connected components is the size of the largest connected component is n, and there always exists path between any two vertices. Therefore. some incorrect codes can pass the provided test cases as the test cases are not varied enough. Therefore, mitigate this issue, we vary the distribution of generated graphs by implementing simple random graph algorithm, which first randomly selects the number of edges in the graph, and then for each edge randomly selects the two vertices incident to the edge. We limit the number of edges in the graphs to generate graphs with wide range of connected components for the purposes of these problems. Alternative random graph generation algorithms can be used, but we find this simple algorithm works for the specific graph problems the benchmark.
Next, we look at improving the input generation for search problem which tasks the user with finding a target value in a list. The current code from ParEval generates random inputs from a narrow range of values, which means that when benchmarking on large input sizes, all values in the range will be generated, which has the effect that any target value will almost surely be in the list. This allows incorrect codes, such as a function that always returns true, to potentially pass the provided test cases on large input sizes.
F.4
Other Fixes
We also fix several bugs in the existing ParEval benchmark when adapting it to the code optimization setting. Several problems encounter overflow issues when scaling up the input size, as the generated input values are quite large. We fix this behavior by changing the range of random values generated for these specific problems. In addition, we fix bugs in the baseline codes of several problems that produce incorrect results, such as dense and sparse matrix LU decomposition, finding the convex hull and computing its perimeter, and computing the maximum contiguous subarray sum.
G Addtionaa Information of Evaluation Setup
G.1 Baseline Implementation for Code Optimization
Co We prompt the agent by adding "Let's think step by step" at the end to solicit reasoning.
30

---

Reflexion was originally used for code generation, wherein each iteration, synthetic test cases are constructed and verbal feedback of the generated code is produced based on the code's correctness on these test cases. The iterations will terminate when the generated code is correct. For code optimization, we replace the testing of correctness with the measurement of speedup. We have Reflexion run ten iterations and choose the code with the best speedup over these iterations as the final solution.
Reflexion:
MapCoder: MapCoder was originally used for code generation. Given coding task, MapCoder first retrieves similar tasks as examples, then proposes few plans to solve the current task, and executes the plans one-by-on in the decreasing order of its confidence. For every plan, few iterations of coding debugging are performed. MapCoder will terminate when plan leads to correct code. For code optimization, we replace the testing of correctness in the debugging step with the measurement of speedup. We have MapCoder produce three plans and run five iterations for each plan. We choose the code with the best speedup over all plans and all iterations as the final solution.
MoA: We follow the default configuration, using three layers and having GPT-4o as the aggregator.
G.2 Baseline Implementation for Code Generation
Different from ParEval, the benchmarks for code generation offer a fixed set of test cases. Hu- manEval/MBPP offer very few test cases (which is motivation of developing HumanEval+/MBPPP)
Hence, in some of the baselines, there a choice between using the given set of test cases and using syntheticly generated test
in each iteration. The drawback of using the given set is that the generated code may overfit the test cases when the set is small, whereas the drawback of using a synthetic set, which is generated by an LLM, is that the test cases may be incorrect. We consider overfit doing more harm and use syntheticly generated test cases instead. Following Reflexion [49 each time, six test cases are generated by Owen14B.
For all four benchmarks, we use EvalPlus 36 to evaluate the generation.
LessonL: As opposed to the code optimization case, for code generation, LessonL terminates the iterations whenever correct code, as measured by the synthetic cases, is produced. The pass score is reported by using the given test cases.
CoТ: Same as the configuration in the code optimization case.
Reflexion: We use maximum ten iterations.
MapCoder: We use three plans and a maximum of five iterations each.
MoA: Same the configuration in the code optimization case.
G.3 Hyperparameters, Hardware, and Timing
Hyperparameters. The hyperparameters used by LessonL are summarized Table 6 unless otherwise stated (e.g., in ablation studies). We barely tune them. Additionally. LLM parameters for all models are also included in Table Both code optimization and code generate tasks share the same set of hyperparameters, except that code generation does not use SELECTHICHISEEDDUT
Table 6: Hyperparameters.
Number of agents, n Number of rounds, T
3
4
Number of selected lessons, &
4
Threshold in SELECTHIGHSPEEDUP c in ADJUSTFACTOR
1.1
0.1
LLM temperature
0.2
LLM frequency penalty
0.5
Hardware. For code optimization benchmarks (ParEval and PolyBench), the LLMs are hosted on an Amazon EC2 1 12xlarge instance through vLL.M (3I. This instance contains Nvidia L40SS GPUs (192 GB). Additionally, we use c5, 45.4xlarge instances, each with 16x Intel Xeon Platinum
31

---

vCPUs (3.4 GHz), to ensure resource isolation from the 12xlaree instance when evaluating code correctness and speedup. All baselines and methods use one single c5 c5.4xlarge exclusively for performance evalution to ensure consistent hardware resources. For code generation (HumanEval, HumanEval+, MBPP, MBPP+), we conduct experiments on a server with eight A6000 GPUs. The LLMMs are hosted on these resources.
Timing. A full of LessonL ParEval takes few hours. The runtime for other methods and/or other benchmarks falls in similar scale.
H
Supplementary Information for Figure
in the main text shows the performance variation as the number rounds increases for methods that do allow iterations (MoA, MapCoder, and Reflexion). In MoA, they are called "layers." " While prior work suggests that these methods' performance may improve with more iterations, we find that it does necessarily happen for code optimization. In what follows, for each method (including LessonL), we show an example of the output code at each round and discuss the speedup variation. For readability, the codes were edited by removing nonessential comments (such as problem description) and white spaces (including indentation).
Figure 3
H.1 LessonI.
Below are the outputs of Lessonl.. The speedup generally improves over rounds. The code is to perform convolution on image. In round O, the code is improved by removing the redundant zero-additions. The code in round identical to that round In round 2, the code handles the interior region and the boundary separately, yielding further improvement. Round 3 achieves significant further improvement by removing the temporary storage and avoiding extra data copies. The code in round is identical to that in round 3.
Original code
#pragma once
#include <vector>
const int (3113) (C-1, -1, -13, C-1, 8, -1), (-1, -1, -13)
/* Convolve the edge kernel with grayscale image. Each pixel will be replaced with the dot product Use a value of 0 for pixels outside the image's boundaries and clip outputs between and 255.
itself and its neighbors with the edge kernel.
imageIn and imageOut are NxN grayscale images stored in row-major. Store the output of the computation in imageOut.
Example:
input: [[112, 118, 141, 152],
[93, 101, 119, 203].
[45, 17, 16, 232].
[82, 31, 49, 1011]
output: [[255, 255, 255, 255],
[255, 147, 8, 255],
[36, 0, 0, 255].
[255, 39, 0, 255]]
*/ void convolveKernel vector <int> const& imageIn, red:ovectokintt & &imageOut, size_t N) { for (size_t i
N; i++)
For (size_t
0; j N; j+t){
int sum 0;
for (int k = -1; k 2; k++) {
for (int 1 -1; 1 < 2; 1++)
32

---

int i k;
int y j 1;
if ((x 0) II (x >= N) II (y < 8) 11 (y >= N))
sum += 8;
else
sum imageIn[x yl edgeKernel 1][1 11;
}
}
if (sum 0)
imageOut [i j] 0; } else 1f (sum 255) {
imageOut[i N + j] 255;
else imageOut[i
j] sum;
}
LessonL, round 0, speedup 1.71x
#pragma
#include <vector>
#include <algorithm>
const int [[]eeenne][3][3] [(-1, -1, -1), (-1, 8, -1), {-1, -1, -13);
void convolveKernel vector const& imageIn, std: ved::rrcror<int> &imageOut, size_t N) {
for (size_t i 0; i < N; i+t) for (size_t int for (int k -1; k 2; k++) {
0; j N; j+t)
0;
for (int 1 -1; < 2; 1++) {
int i k;
int y j+
(x
&& 0 N)
sum imageIn[x * yl edgeKernel Ik 1J[1 11;
imageOut [i j] std: clamp (sum, 0, 255): ;
LessonL., round 1, speedup 1.71x
#pragma once #include <vector>
#include <algorithm> // For std:: clamp
const int [[ecKernel[3][3] ((-1, -1, -1), (-1, 8, -1), (-1, -1, -10):
void convolvekernel (ett:: vector <int> consta imageIn, std:: vector &imageOut, size_t N) {
for (size_t i 0; < N; i++)
for (size_t = 0; j N; j++)
33

---

int for (int -1; k 2; k++) {
for (int 1 =1; 2; 1++) {
int x = i k;
int sum imageIn[x yl edgeKernel [k 1][1 11;
88 N)
imageOut[L j] std:: clamp (sum, 8, 255); ;
LessonL, round 2, speedup 2.09x
#pragma once
#include <vector>
#include <algorithm> For std::clam
const int ddgekerne1(3][3) ((-1, -1, -1}, (-1, 8, -1), (-1, -1, -13)
void convolveKernel :eett:::eettoriiii <int> const& imageIn, std:: : <int> &imageOut, size_t N) {
// Temporary vector to store intermediate results std:: ved::rrctosint> temp(N * N, 8);
Process the interior pixels for (size_t 1; < 1; i+t) { for (size_t t
j - 1; j+t)
int 0;
for (int k -1; k 2; k++) {
for (int -1; 2; 1++) {
sum += imageIn[(i k) NN (j 1)] * edgeKernel [k 1][1 1J;
tempCi i N il sum;
Process the boundary pixels
for (size_t i 0; i < N; i++)
for (size_t = 0; j N; j+t) {
if (1 == 0 11 i == 1 == 11 - 1)
int sum =0;
for (int k -1; << k++) {
for (int == =1-1
11:14 2; 1++)
int =
int j++l.
if (x &&
y >= 8 N) sum += imageIn[x * y] * edgeKernel[k: 1][1 + 1];
}
temp [i * N i] sum;
}
}
34

---

I/ Apply clipping to the results for (size_t t [i] std:: clamp (temp[il, 0, 255);
N; i++) {
LessonL, round 3, speedup 9.45x
#pragma once
#include <vector>
#include <algorithm> // For std::clamp
const int [3][33 ((-1, -1, -1), (-1, 8, -1), (-1, -1, -13)
void convolveKernel : vector <int> const& imageIn, std:: vectorkints
&imageOut, size_t N) {
// Process interior pixels
for (size_t i = 1; i N 1; i++) { for (size_t int sum 0;
1: j
1; j++)
for (int k = -1; k 2; k+t) {
for (int 1 = -1; 1 < 2; 1++) {
+= imageIn[(1 k)
(1 1)] edgeKernel[k 1][1
imageOut j] = std:: clamp (sum, 8, 255);
// Process boundary pixels for (size_t i
< N; i++)
for (size_t
0; j N; j++) {
if (i 0 ll i 1 j == 11 j N 1) t
int sum = 0;
for (int k -1; < k+t) {
for (int
-1; 1< 1++)
int
int y = k;
1;
if ((x < 0) Il (x >= N) 11 (y <) (y >= N)) {
sum 0;
} else
sum imageIn[x y] edgeKernel 1][1 13;
}
ImageOut j] std:: clamp (sum, 0, 255);
}
LessonL, round 4, speedup 9.50x
#pragma
#include <vector>
#include <algorithm> // For std:: clamp
const int edgeKernel [3][3] ((-1, -1, -1), (-1, 8, -1), {-1, -1, -13);
35

---

void convolveKernel (std: vector <int> const& imageIn, std: vector <int> &imageOut, size_t N) { // Process interior pixels
for (size_t i
i < N 1; i+t) { j+t)
for (size_t int sum 0;
1; j
for (int k = -1; k 2; k++) {
for (int 1 = -1; 1 2; 1++) {
+= imageIn[(1 k) (1 1)] edgeKernel 1][1 1];
imageOut[i j] =tdd: clamp (sum, 8, 255);
// Process boundary pixels for (size_t for (size_t
N; i++)
0; j N; j+t) {
if (i == 0 1l == - j == == = - 1)
int sum = 0;
for (int k -1; < k++) {
for (int =1; <<2 1++)
int
k;
int = 1;
if ((x < 0) Il (x >= N) 11 (y 0) 11 (y >= N)) {
} else
sum imageIn[x yl edgeKernel [k 1][1 + 13;
}
imageOut jl std:: clamp (sum, 0, 255);
H.2 MoA
Below the outputs of MoA. The speedup deteriorates over rounds/layers. The code finds the kth smallest element of array. The code in round 0 achieves the highest speedup by implementing the wwvid-aadd-coqqee algorithm. In round the speedup reduces because of the type cast from time to unsigned int. In round 2, the speedup further reduces because the implementation is changed from wovide-and-conquee using std::tth element. round 2.
_dd:met_elementt The codes in round 3, and are identical to that in
Original code
#pragma
#include <algorithm>
#include <vector>
/* Find the k-th smallest element of the vector Example:
input: x=[1, 7, 6, 0, 2, 2, 10, 6], k=4
output:
36

---

*/
int (std vector <int> constå x, int k) {
vedtorrcctsintt x_copy = X;
(xx:: coepx__cppbbbgin() b) x_copy.end()) return x_copy(k-1];
MoA, round 0, speedup 12.36x
#pragma #include <vector>
#include <cstdlib> // For std:: () and sta::
int pivotIndex) {
x, int left, int right, int int pivotValue x[pivotIndex]; sedp:::spxxxixioIndndd
x[right]); // Move pivot to end int storeIndex left;
for (int left: right; ++i) {
(x[i] pivotValue) {
sed::ssspo(oooooee **storeIndex;
x[i]);
(xd::swhppX(rihht),,. x[storeIndexx); Move pivot its final place return storeIndex;
int quickselect(std:: <intor x, left, int right, int k) { if (left == right) { // IF the list contains only one element, return x[left]; // return that element
Select a random pivotIndex int pivotIndex left sad: x (right left + 1);
Find the pivot position in sorted list pivotIndex partition(x, left, right, pivotIndex);
11 The pivot is in its final sorted position if (k == pivotIndex) {
return x[k];
else if (k < pivotIndex) {
return quickselect(x, left, pivotIndex - 1, k); else {
return quickselectxx, pivotIndex + 1, right, k);
}
int findKthSmallest (std: vector<int> const& x, int k) { :td::vector<int> x_copy x; std:: (time (0)); // Seed the random number generator
return quickselect (x_copy, 0, x_capy size - 1, k 1);
MoA, round speedup 8.10x
#pragma once #include <vector>
37

---

#include <estdlib> // For std:: and std:: s)and( #include <ctime> For std:: std:stime ()
int (artition(std:::. ( vectortttt pivotIndex) { int pivotValue x[pivotIndex];
x, int left, int right, int
(xd:::ssp((ppiitInneee) x[right]); Move pivot int storeIndex left;
for (int i left; right; +ti) {
if (x[i] pivotValue)
std:: swap (x[storeIndex), x[i]); **storeInde;;
(x.riihh, x[storeIndexx); Move pivot its final place return storeIndex;
int guickselectsstd: vectorriitt x, int left, int right, int k) (left right) { If the list contains only one element, return x[left]; // return that element
Select random pivotIndex int pivotIndex left std:: (right left 1);
// Find the pivot position in sorted list pivotIndex partition left, right, pivotIndex);
The pivot is in its final sorted position if (k == pivotIndex) {
return x[k];
else if (k pivotIndex) {
return quickselect(x, left, pivotIndex 1, k); else {
return quickselect(x, pivotIndex + 1, right, k);
}
int thakthssmallesssstt:seeeeree vector <int> const& x, int k) {
std:: vector x_copy = x;
s (static__asttsuniggggg
int>:std:: time (0))); // Seed the
random number generator
return quickselect (x_copy, 0, sicep 1, k 1);
}
MoA, round 2, speedup 6.71x
#pragma once
#include <vector> #include <algorithm>
int mooktthsmallestsstt:: (stt:t vectorkints constå x, int k) {
std:: vector< x_copy = x;
std::ennhelement (x_copy b)gin(), x_copy, x_copy.beging begin( k 1,
x_copy. end
return x_copy[k 13; }
MoA, round 3, speedup 6.72x
38

---

#pragma Vinclude <vector> #include <algorithm>
int Findkthhmmallesttstt:: vector vrt constà x, int k) {
vector<intt x_copy x;
std::nth_elemeet (x_copy. b) E.copy.beging 1,
x_copy (nd
return x_copy[k 13;
MoA, round 4, speedup 6.74x
#pragma #include <algorithm> #include <vector>
int findKthSmallest (std: :ector<<ntt const& int k) { vectorrcttitt x_copy x;
std::eth_element (x_copy (), x_copy. () 1,
x_copy end
return c_copy [k- 1];
MoA, round 5, speedup 6.72x
#pragma once #include <algorithm> #include <vector>
int findktthmmaleest v
vectorkints consta x, int k) {
vectorriit x_copy x:
std::nth_element (x_copy ()gin(), x_copy. ()gin() 1,
x_copy ());
return x_copy[ [k - 1];
H.3 MapCoder
Below are the outputs of MapCoder. The speedup does not follow monotonie trend over rounds. MapCoder works by using one agent to generate a few plans and evaluate each plan with a confidence rating. The plans executed in the decreasing order of confidence, but it does not necessarily mean that the earlier executed plans will lead to better performing codes. In this example, the original code compute the highest node degree in an undirected graph. This amounts computing the sum of each row of the adjacency and comparing different rows. All the plans perform some sort of loop unrolling, the idea of which is gaining less attraction nowadays because loop unrolling should be delegated to the compiler for better instruction parallelism and fewer comparisons. As a result, none of the output codes runs faster than the original code.
Oiginal code
#pragma once
#include <algorithm> #include <vector> #include <queue>
39

---

#include <stack>
#include <deque>
#include <limits>
/* Compute the highest node degree in the undirected graph. The graph defined the adjacency matrix
an NxN adjacency matrix stored in row-major. undirected.
Example:
input: [[e, 6, 11, [0, e, 11, [0, 11, D1, 1, 1, 8]]
output:
*/ int max vintor const& A, size_t N) {
int highestDegree = 0;
for (int 8; i N; += 1)
int degree 0;
for (int 8; j < N; + 1) {
degree += A[i * N il;
highestDegree std: (highestDegree, degree);
return highestDegree;
MapCoder, round 0, speedup 1.00x
#pragma once #include <algorithm> #include <vector> #include <numeric> // For raccumuuaee
int maxDegree std: vector <int>& A, size_t N) {
int highestDegree 0; N; 2) Loop Unrolling: Process
for (size_t rows time
int degreet std:: accumulate (A. (egin
N, A. begin + (i +
1) * N, 0);
int degree2 (i
acc::accumllate (A. begin() (1
N, A. begin() (i 2) N, 0) 0;
highestDegree std: max (highestDeegee,, std: max (degree1, degree2));
return highestDegree:
}
MapCoder, round 1, speedup 0.96x
#pragma once
#include <algorithm>
#include <vector>
#include <queue>
#include <stack>
#include <deque>
#include <limits>
inline int maxDegree (canst std:: www.veettoriiiii <int>& A, size_t N) int highestDegree 0;
for (size_t 0; < N; i += 4) // Loop unrolling by a factor of
int degreed degree1 = 0, degree2 0, degree3
40

---

if 8 N)
for (size_t j 0; N; j 4) {
//
Inner loop unrolling by
factor of 4
degreed += A[(i 8) * N 1 + 0];
degreed A[(i
N
11;
degreed AC(İ 0)
21;
degreed A[(i 8)
j 31;
if (i < N) [
degree1 += A[(i +
1)
01;
degree1 A[(i 1)
] 1];
degree1 AC(1 1)
2];
degreel #= A[(1 + 1) *
31;
if (i < N) {
degree2 A[(i 2)
]
01;
degree2 A[(i + 2)
]
11;
degree2 *= A[(i + 2)
2];
degree2 += A[(i + 2)
j
3];
(i N) {
degree3 t= A[(i + 3) *
]
0];
degree3 += AC(i + 3) * N
] 1];
degree3 += A[(i + 3)
2];
degree3 AC(i 3)
] 31;
}
highestDegree = std:: max ((higheestDeeree, degreea, degree1, degree2, degree3));
return highestDegree;
MapCoder, round 2, speedup 0.65x
#pragma once
#include <algorithm>
#include <vector>
#include <queue>
#include <stack>
#include <deque>
#include <limits>
int maxeegree std: :td::vectos vector <int>& A, size_t N) {
int highestDegree = 0; for (size_t int degree 0;
1 N;
1)
// Unrolling the inner loop
for (size_t = 0; j N; += 4) {
if (j + 0 < N) degree += A[i N
+ 0];
if (j
degree A[i
1];
if (j + N) degree A[i N
+ 2];
if (i + N) degree += A[i
N + 3];
}
highestDegree = std: max
max (highestDegree, degree);
return highestDegree:
}
MapCoder, round 3, speedup 0.74x
41

---

#pragma
#include <algorithm>
#include <vector>
#include <queue>
#include <stack>
#include <deque>
#include <limits>
int msx veetorrittt const& A, size_t N) {
int highestDegree 0;
for (int 8; N; 1)
int degree = 8;
for (int j = 8; j N; += 4) { // Unroll the loop by factor of
degree ACi if (j
il;
N) degree Ali N + 11;
if (j + 2 < N) degree += Ali N + 2]:
if (j + 3 < N) degree += Ali N + 3];
highestDegree std: max (highestDegree, degree);
return highestDegree:
MapCoder, round 4, speedup 1.00x
#pragma once
#include <algorithm>
#include <vector>
#include <queue>
#include <stack>
#include <deque>
#include <limits>
int :
const& A, size_t N) {
int highestDegree = 0;
for (int degree 0;
i N;
int = 8;
// Unroll the inner loop by factor of for (;
degree ACi 4; j += 4)
jl;
degree ACi
j 1];
degree += ACi k N j 2];
degree += ACi * ] 3];
Handle the remaining elements for (; j <N; j += 1) {
degree += A[i * N jl;
highestDegree std: max (highestDegree, degree);
return highestDegree:
}
H.4 Reflexion
Below are the outputs of Reflexion. The speedup generally improves over rounds. The code is to perform cell XOR. The code in round fails to compile. The code in round identical to the original code.
The code in round 2 is marginally slower than the original code. This is likely because
42

---

of the timing noise; in fact, this code precomputes the common expression i N N which can slightly improves the runtime. round 3. the code performance drastically improves, because half of the if-tests are eliminated. The code in round 4 is identical to that of round 3.
Original code
#pragma once #include <vector>
/* Set every cell's value to 1 if it has exactly neighbor that's 's it to
1. Otherwise
Note that we only consider neighbors and not input_(i,j} when computing (1,
input and output are NxN grids of ints in row-major. Example:
input: [[O, 1, 1, 80,
[1, 0, 8, 01,
[0, 0, 8, 0].
[0, 1, 8, 0]
output: [[e, 0, 1, 11,
[1, 0, 0, 11,
[0, 0, 1, 01.
[1, 0, 1, 8]]
*/
void cellsXOR : const& input, std: vector <int> &output, size_t N) {
for (size_t i = 0; i < N; ++)
for (size_t
0; j N; ++])
int count 0;
if (i > & input((i-1)*N
j] 1) ++count;
if (i < N-1 & input ((ppt((i+))t) + j] == 1) ++count;
if (j > & input[i*N + j-1] == 1) ++count;
if (j N-1 input[isN [+1]
++count;
output[ixN j] (count 1) ? 0;
Reflexion, round 0, compilation failed
#pragma once #include <vector>
Lookup table for the count neighbors const int ne neighhorCountLoouppP]
{
0, 0, 0, // ne neighbors
// neighbor
// neighbors
0 0, 0, // neighbors
0 0, 0, // neighbors
0, 0, // 5 neighbors
0, // neighbors
0, 0,
neighbors
0, 0,
// 8 neighbors
};
void DellsxOR(std::
seectrrkitt>> sentsxoo const& input, std:: vector <int> &output, size_t N) {
for (size_t 0: i N; ++i)
for (size_t = j N; ++5) {
43

---

int count
if (1 > 0) {
if (j > 0) count += input[(i-1)4) j-11;
count += input[(i-1))N + il;
if (i N-1) count += Input((i-1)+t) j+1];
}
if (j > 8) count input[iaN j-11;
if (j < N-1) count + input[iaN j+1];
if (i < N-1)
if (j 0) count 11) j-11;
count in 1];
if (1 N-1) count input inp j+1];
output[io] il reighhorcouttLoooppcoon]]]]
}
Reflexion, round 1, speedup 1.00x
#pragma once #include <vector>
void cellsXOR :ector const& input, std:: vector <int> <int> &output, size_t N) {
for (size_t i = 0; i N; ++i)
for (size_t
0; j N; ++5) {
int count
if (i >
input[(i-1))))
j] 1) ++count;
if (1 < N-1 & Input[(i+1)*)) + j] == 1) counnt
if (j > 0 && input[i*N + j-1] == 1) ++count;
if (1 N-1 & input[i*N ]+1]] == 1) ++count;
output[ixN j] (count 1) ? 0;
Reflexion, round 2, speedup 0.95x
#pragma once #include <vector>
void cellsXOR (std:: vector<int> const& input, std:: vector <int> &output, size_t N) {
for (size_t i for (size_t
N; ++i)
j N;; ++5) {
int count 0;
size_t idx
N j:
if (i > 0) {
if (inputCidx N] == 1) ++count;
}
if (i < 1) { if (input[idx N] = 1) ++count;
}
if (j 9) (inputCidx
1) ++count;
)
if (1 < 1) { if (input[idx 1] == 1) ++count;
44

---

)
[udxu (count 1) : 0;
Reflexion, round 3, speedup 10.81x
#pragma once
#include <vector>
void (allsxoR(std::: (eltssr vector<int> const& input, std:: vector <int> &output, size_t N) { for (size_t
N; ++i)
for (size_t
0; j N; ++]) {
int count 0;
size_t idx =i = j: if (i > 0)
count input[idx N];
if (1 < N 1) count += input[idx N];
if (i 0) count += input[idx 1];
}
if (1 < N 1) count input[idx 13;
Output[idx] (count 1) : 0;
Reflexion, round 4, speedup 10.71x
#pragma
#include <vector>
void fellsxoR(std:: vector<int>> const& input, std:: vector <int> &output, size_t for (size_t
N; ++1)
for (size_t
0; j N; ++5) {
int count 0;
size_t idx = i + i: if (i > 0)
count input[idx N];
if (i < N count += input[idx NJ;
}
if (i > 0) count += input[idx 1];
}
if (i < count indu 1];
{
output [idx] = (count == 1) 1 o;
45

---

}
I Supplementary Information for Figure
Figure in the main text shows the monetary cost and FLOPS for various models and methods. For this, several pieces of information need be estimated: the unit cost, FLOPS per token, and the number of tokens.
The pricing information for the open-source models is based on https://www.togette pricing and that for the GPT models is from https:/ openai com.emaaa.cmmbooc//rciinng Specifically, estimate the unit cost of Deepseek7B, Qwen7B, Qwen 14B, and DeepseckR]-14B to be $0.20, $0.20, $0.30, and $0.18, respectively, per 1M tokens. Meanwhile, GPT--4 mini costs 10.15/50.60 per 1M input/output tokens, while GPT--4o costs $2.50/$10.00 and OpenAl o3 costs $2.00/$8.00 00.0..00....00.. The pricing information was retrieved April 2025 (except for DeepseekR 1-14B and OpenAl o3) and it may be time-sensitive. The pricing for DeepseckR1-14B and OpenAl o3 was retrieved in October 2025 and it appears that the pricing is compatible with that of other models. The number of tokens estimated based on the CooeelJama-7--Instnutt tokenizer from Hugging- face [55]. For multi-agen, methods, we assume each agent consumes//eneraan the same amount of tokens. Table 7 summarizes the information for one experiment on the ParEval benchmark.
Table 7: (Estimated) Costs for running a ParEval experiment. For multi-agent methods, the listed number tokens is the sum over all agents. For MoA, the first row shows the number tokens for the open-source agents, while the second row for the GPT-4o aggregator (separately priced). For CoT, Reflexion, and MapCoder, all agents are QwenI4B.
Speedup
#Input tokens
#Output tokens
#Total tokens
Cost ($)
1.60
23421
33550
56971
0.017
Qwen14B(20)
1.70
468420
671001
1139421
0.342
GPT-4o mini
1.57
20815
26399
47214
0.019
GPT-40
1.72
20815
27800
48615
0.330
DeepseekR1-14B
1.59
24900
107733
132633
0.024
OpenAl o3
2.21
20875
98719
119594
0,832
CoT(I)
1.57
23681
37119
60800
0.018
CoT(20)
1.67
477220
742380
1219600
0.366
Reflexion
1.88
1126390
389520
1515910
0.455
MapCoder
1.85
1210143
713383
1923526
0.577
MoA
1.76
344316
172973
517289
0.612
79005
29441
108446
LessonI.
2.16
1009359
387994
1397353
0.326
For FLOPS per token, we take the approach of 30 and estimate it to 24 where Alayer is the number of layers and dmmdel is the hidden dimension. For the open-source models Deepseek, Qwen, and DeepseekR1 (which distilled from Qwen), these two hyperparameters come from (20 25). the other hand, the information of the GPT models is unknown. We treat GPT-4o mini an 8B model and GPT-4o 200B model, following []. We also treat OpenAl o3 the same size as GPT-4o, given no community information. Then, estimate Taaaer and dmodal by borrowing the information from the LLaMA models of similar sizes [18. Specifically, we reuse the configuration of LLaMA 8B for GPT-4o mini and that of LLaMA 405B for GPT-4o and OpenAl o3, except that for the latter case Taayrr is cut by half. The estimated FLOPS are given Table
Complementary to Figure that compares different models/methods based on monetary costs and inference latencies, in Table we list the number of LLM calls and the average number of tokens for solving one problem. The results are in three tiers. Qwen1 GPT-4o mini, GPT-4o,, and CoT make only one LLM call. The number of tokens per call is around 1000, with CoT costing slightly more because of its "thinking step by step" nature. DeepseckRI-14B and OpenAl o3 reasoning models
46

---

Table 8: (Estimated) Model configurations and FLOPS per token.
Model
Talayer
dmodel
FLOPS
DeepSeek7B
32
4096
12884901888
Qwen7B
28
3584
8631877632
Qwen14B
48
5120
30198988800
GPT-4o mini
32
4096
12884901 8888
GPT-4o
63
16384
405874409472
DeepscekR1-14B
48
5120
30198988800
OpenAl o3
63
16384
405874409472
Table 9: LLM calls and token consumption per problem.
#LLM calls
#Tokens / call
#Tokens
Qwen 14B
949.50
949.50
GPT-4o mini
1
786.90
786.90
GPT-4o
1
810.25
810.25
DeepseekR1-14B
I
2210.15
2210.15
OpenAl o3
1992.42
1992.41
CoT
1
1013.33
1013.33
Reflexion
20
1263.25
25265
MapCoder
16
2137.25
34196
MoA
10
957.90 $57.90 + 1087.43
20453
LessonL
24
970.38
23289
and they cost more tokens per call (around 2000). Reflexion, MapCoder, MoA, and LessonL all solve problem many rounds, making more LLM calls and costing considerably more tokens. Among these four methods, LessonL. requires the most number of calls but the per-call tokens are relatively economic, thanks to the concise design of lessons.
J Additional Experiment Results
J.1 Complementary Strengths of Agents
As shown in Table 5] the models Deepseek7B, Qwen7B, and Qwen have complementary (but unknown a priori) strengths. Here, we study how much Lessonl. improves over the best of them category by category. Table 10 lists the speedup results.
see that LessonL improves the speedup on many categories. Among them, "fft" and "stencil" enjoy the most significant improvement. Meanwhile, for categories that LessonL suffers from degradation of speedup, the degradation is only minor. theory, LessonL no worse than the best single agent. The degradation reflects randomness in LLM generation.
To investigate the complementary strengths of the agents, let us look into the "sort" category, on which Qwen7B performs poorly but Deepseek7B and Qwen 14B are quite good. Without Lessonl, Qwen7B fails offer any speedup; with LessonL., it achieves 2.35x speedup on average.
For a concrete example, consider Problem 40, sorting an array of complex numbers by magnitude. In this example, Qwen7B produces slower code initially but Deepseek7B and Dwenl4B produce fast codes. These codes generate the following lessons:
Lesson from Deepseek7B
Code B is faster because it uses the nstd: function, which calculates the square of the magnitude of complex number, instead of the 'std::ass function which caleulates the true
47

---

Table 10: Performance comparison between the best single agent and LessonL across categories. Benchmark: ParEval (serial mode) speedup.
Category
Best single agent
LessonL.
Relative improvement
graph histogram
1.00
1.04
4.00%
1.57
1.67
6.37%
scan
5.95
5.91
0.67%
transform
1,01
1.03
1.98%
sparse_ la
1.70
1.61
-5.29%
reduce
2.21
2.03
-8.14%
fft
1.02
2.23
118.63%
geometry
8.09
9.80
21.14%
stencil
1.12
3.23
188.39%
dense la
1.02
1.19
16.67%
sort search 1.07
2.35
2.69
14.47%
1.02
-4.67%
magnitude. Squaring the magnitude is faster and more efficient than calculating the square root, especially for large numbers.
Lesson from Qwen14B
Code B is significantly faster because it avoids recalculating the magnitude of each complex number multiple times during the sorting process by first storing all magnitudes in separate vector. This reduces redundant computations and leverages efficient sorting algorithms on precomputed data.
These two lessons point different optimizations: (1) using 'std:norm' rather than 'std:abs' to avoid a square root; (2) performing precomputation to avoid repetitively computing 'std::abs' inside the comparison function used for sorting. When given these lessons, Qwen7B generates code that achieves a speedup by following the guidance provided by the first lesson. When Qwen7B is not given any lessons, it fails optimize this problem; it often generates code that fails to compile or attempts misguided optimizations like replacing 'std:abs" with two calls to 'std:hypot" (slower).
J.2 Running LessonL for More Rounds
In addition Figure in the main text, increased the interaction rounds of LessonL and allowed the agents to interact longer. Table 11 lists the speedups. It shows that the speedup improves when more computational resources are spent. However, there a point after which the speedup nearly plateaus. This happens around round 8, indicating diminishing return afterward.
Table 11: Speedup over rounds for LessonL. Benchmark: ParEval (serial mode).
Round
3
5
7
9
10
Speedup
1.90
1.95
2.01
2.05
2.12
2.13
2.13
2.21
2.22
2.22
K
Case Study: Geometry
Let us study case about finding the smallest distance among n points in wwo-dimensional space. The original code performs naive computation by enumerating all possible pairs and finding the minimum distance, which takes Onn²) time. The final optimized code reduces the complexity from Onn²) to On log n) and achieves outstanding speedup of 74.31x. The algorithm starts by sorting the points based on their x-coordinates and then performs divide-and-conquer. The divide part separates point set in two equal halves. The conquer part (recursively) calculates the minimum
48

---

distance for each of the two halves; let us call the smaller of the two distances, d. Then, the combine part filters out points with an x-distance from the middle point greater than d, sorts the remaining on their y-coordinates, ignores point pairs with y-distance greater than d, and returns the smallest distance among the remaining points.
Qwen 4B summarizes the reason of the gain as the following:
"Code runs significantly faster than Code A because it sorts the points by their x-coordinate before comparing them pairwise. This sorting operation reduces the number of comparisons made in the nested loop, which a time-consuming operation in Code A. By sorting the points first, we can eliminate unnecessary comparisons and thus speed up the execution time."
We see that it attributes the initial sorting operation to be the main reason of the outstanding result. This reason is partially correct, but sorting only the precursor step of wwidd-and-conquee that leads to reduced Onn log n) cost. We proceed to investigate all the generated lessons over the course and find that indeed some lessons highlight the importance of wwiid-andd-onquees
Sorting is mentioned in lessons 1, 2, and 4, while divide-ande -nd-e--un--e is mentioned in lessons 1, 4, and 5. The remaining lessons all about compilation failures, including forgetting '#pragma once' (lesson 3), no implementation 'std::set for 'Point' (lesson 6), mismatch of argument types (lesson 7), undefined functions (lesson 8), and use of function before definition (lesson 9). These lessons reflect the complex implementation of aivide-and-conquer algorithm, which can cause many errors (even for human programmers). Overall, the lessons helpful guide an L.M to generate correct and high-performing codes.
The original code, the final code, and the full text of the lessons are provided below. For readability, the codes were edited by removing nonessential comments (such as problem description) and white spaces (including indentation).
Original code
#pragma once #include <vector> #include <limits>
struct Point {
double x, };
double distance(Point const& pl, Point const& p2) { return std:: sqrttsssstt:: pow (p2.x x-p.x, 2) std: y y-p1. 2));
/* Return the distance between the closest two points in vector points.
Example:
input: [[2, 3), (12, 30), {40, 50), (5, 1), (12, 18), (3, 4}] output: 1.41421
*/
double closestPair (ttierrrn const& points) // The polygon needs to have at least two points
if (points. size 2)
return 0;
auto getDist = [](Poi const& a, Point const& b) { return std:: (std po..::::..bb...... 2) std: (b.yaay., 2));
};
double minDist std: std:: hawe::uumeriiiimiii <double>::aax:
for (size_t i 0: i < points. points.size() 1; i++) { points. size (); j++) {
for (size_t =
49

---

struct Point const double dist getDist points [j]); if (dist minDist) { minDist dist: }
return minDist; }
Optimized code, speedup 74.31x
#include <vector>
#include <algorithma
#include <cmath>
#include <limits>
double };
double eestance(conss Point& pl, const Point& p2) { return s :::::::::st::::::: (p2.x pl.x, 2) std::pow(p2. - pl.y, 2));
double closestPairtHelper std: vector vector points, size_t start, size_t end)
if (end start double minDist std:: numeric__mimitt :ttp::liiiitttoooe::::::a::
{
for (size_t = start; < end; ++1)
for (size_t j ==i 1; j end; ++j) { minDist std::min(miiDDii }
(istance points(il));
}
return minDist;
size_t mid (start end) / 2;
double leftMin per (points, start, mid); double rightMin = closestPairtellpe (points, mid, end); double minDist min:::iinnn,, rightMin);
sed:: strip;
for (size_t start; end; ++i) { if (std:: (points[i].x points(mid].x) < minDist) { ptcip.ppsh_baccee (poontssii)))
std::sortsstrig begin(), Point& b) return a.y b.y: ));
[]onnt Point& const
for (size_t = i strip.size() +i)
for (size_t ==
strip.size() strip[il.y strip[il.y
< minDist: ++j) { minDist (tdi:iiinmiinDstt, distance strip(i]));
50

---

return minDist;
}
double closestPair (const (points. size () 2)
points)
return o;
ved:oorroorss sortedPoints points; rod::sott begin(), sortedPoints.eedd) end (contt Point&
a, const Point& b) {
return a.x < b.x; ));
return closestPai peelper (sortedPoints, oo e, sartedPoints. size ());
Lesson (selected)
Code B is significantly faster because sorts the points by their x-coordinate before comparing each pair of points. This sorting step reduces the number of comparisons made in the nested loop, which is time-consuming operation. By sorting the points first, we ensure that similar pairs of points (those with similar x-coordinates) are closer together in the sorted array, reducing the number of unnecessary distance calculations. This results in significant speedup because it eliminates redundant computations.
Lesson (selected)
Code B is significantly faster because it implements divide-and -nn-cuuue strategy, also known as the "plane sweep" algorithm, which more efficient for this type of problem. sorts the points based their x-coordinates and then uses strip-basee rrsi-bbsed approach find the closest pair of points. This approach reduces the time complexity from O(n?) to O log n), making it much faster for large datasets.
In Code A, the algorithm compares all pairs of points, leading to time complexity of O(n²). This makes it less efficient for larger datasets compared Code B, which has better time complexity.
Additionally, Code B optimizes by only considering points within a certain distance (the minimum distance found so far) in the strip, reducing unnecessary comparisons. This optimization not present in Code A and contributes to its slower performance.
Lesson (selected)
Code cannot be compiled because it missing the *#rragma once' directive the beginning of the file, which is used to prevent multiple inclusions of the same header file. Additionally, there are no syntactic errors in Code that would cause to fail compilation based on the provided compiler output.
Lesson (selected)
Code B is significantly faster because implements more efficient algorithm for finding the closest pair of points. Specifically. it uses divide-ands andid-- strategy combined with the concept of "strip" to reduce the time complexity from O(n²) to O(n log n.).
Code A. the algorithm compares every pair of points, which leads to time complexity of O(n²), This is because there are n (n 1)/2 comparisons in the worst case.
51

---

On the other hand, Code B uses -ode--andd-cuuuu approach. It first sorts the points by their x-coordinate and then recursively finds the minimum distance each half. This step has time complexity of (n log n).
Additionally, Code B also considers pairs of points that lie within a "strip" (a range of y-coordinates) around the vertical line that divides the two halves. This strip is sorted by y-coordinate and then checked for pairs that are closer than the current minimum distance. The stripClosest function has a time complexity of (n), which is used when there are more than 3 points in the strip.
Lesson
Code B is significantly faster because it implements the divide-and-conquer algorithm for finding the closest pair of points, which reduces the time complexity from O(n²) in Code A approximately (n log n).
Lesson
Code B cannot compiled because attempts to use 'std::set' to store 'Point' objects, but 'Point' does not have defined comparison operator. The compiler needs way to compare 'Point' objects to insert them into the set.
Lesson
Code cannot be compiled because the 'stripClosest' function is called with "std:vector<Point>> and 'double', but the function expects 'tdd:veccto<PPoin>>> and an 'int'. The compiler output would indicate a type mismatch error at the call to 'stripClosest'.
Lesson
Code B cannot be compiled because the function 'closestPairUtil' is declared but not defined, leading to linker error. The compiler output would indicate an unresolved external symbol for closestPairUtil..
Lesson
Code cannot be compiled because the 'stripClosest' function defined after is used in the "closestPairtti' function. In C++, functions must be declared before they are used, or defined inline within the scope where they are called. The compiler error indicates that does not recognize 'stripClosest' when it is first referenced in 'closestPairUtil..
L Case Study: DFT
Let us study case about performing the discrete Fourier transform (DFT). Mathematically, the DFT of length-N array is array X of the same length, where
N
N-1 In (coss(22)k// + (on(-2nk/NN),, k= 0,0,10 I.
Xx
In 2ank/N
Because of the periodicity of trigonometric functions, many nı values are congruent modulo N and thus many complex exponentials are repeatedly computed. A natural idea of improving the runtime the DFT is to precompute these exponentials and perform table lookup whenever needed. Indeed, the optimized code adopts this idea, reflected by lessons 1, 2, 5. 6, and 8, and agreed with by Qwen 4B's assessment:
"Code significantly faster than Code because precomputes the complex exponential factors, reducing redundant calculations in the inner loop. This
52

---

optimization minimizes the computational overhead and accelerates the DFT process." "
is interesting to note that the above mathematical definition of DFT implies an O(N2) computational cost, while is well known that divide-and-conguee algorithm, called the fast Fourier transform (FFT), can bring down the cost to ON log N). Some agents in some rounds do try FFT, as lessons 4, 7, and 9 suggest, but they fail. Two of these lessons say that the optimized code "is nonequivalent to" the original code, while another lesson says there is a logical error. We investigate into the resulting codes and find that they make error by flipping the sign (note the negative sign inside the trigonometric functions in I In other words, the incorrectness of the codes caused by sign error, rather than the nonequivalence between DFT and FFT. While the lessons not fully correct, they do discourage the agents to try the FFT idea, as the agents are probably not capable of precisely implementing FFT in context.
Another lesson (lesson 3) is incurred from compilation error.. The code attempts to use Intel intrinsic instructions but fails compilation. Such attempts requires CPU support and the of the corresponding compiler flags, which are not supported by our evaluation pipeline. This lesson discourages the of Intel intrinsics in the benchmarking.
The original code, the final code, and the full text of the lessons are provided below. readability, the codes were edited by removing nonessential comments (such as problem description) and white spaces (including indentation).
Original code
#pragma once
#include <vector>
#include <complex>
#include <cmath>
#if !defined (M_PI)
#define M_PI 3. 141 141111992255887933344444
#endif
/* Compute the discrete fourier transform of x. Store the result in output.
Example:
input: [1, 4, 9, 16]
output: [30-0i, -8-12i, -10-0i, -8+12i]
*/ void dft(std: : vector <double> const& x,
std: compooreed::: int N size
&output) {
output. resizee vector and initialize with 0
0)); // Resize the output
for (int k 0; k N; k++) [ // For each output element comllxxclelll sum(0, 0)
for (int 0; N; n++) [ // each input element double angle std::comploe <double>> <dubee c(std:: (angle), sed:::sinnnglll: // Euler's formula x[n] * c;
M_PI n
N;
output[k] sum;
Optimized code, speedup 10.83x
#pragma once
53

---

#include <vector>
#include <complex>
#include <cmath>
#if !defined (M_PI) #define M_PI 3.111199255388999223888 #endif
void dft(std: : <double< const&
std: <stt cometorrstt::coollexxcdouu int size
complex <double>>>> &output) {
output. resize complex sdouble>(O, 0)); // Resize the output vector and initialize with 0
Precompute the complex exponential factors
<stdor vector :ompoor compeexxcbbbeelblleeee <double>> W(N);
for (int 8; k N; k++) {
double angle = -2 M_PI * N;
W[k] std:: complexxdouble>(ttee congge)))) std:::in(angle));
for (int k 8; k < N; k++) // For each output element std:: comp: <double> sum(0, 8);
for (int 0; N; n++) { // For each input element sum x[n] w[(n k) N];
output[k] sum;
Lesson (selected)
Code B is significantly faster because it precomputes the complex exponential factors and stores them a vector, W. This avoids the need to compute the same factor multiple times within the nested loop, which can lead to significant performance improvements due to reduced computation time and increased cache efficiency. In Code B, each factor is computed only once and then reused in the inner loop, while in Code A, each factor is recomputed for every input element.
Lesson (selected)
Code B is significantly faster than Code A because it precomputes the complex exponential factors, reducing redundant calculations in the inner loop. This optimization minimizes the overhead associated with trigonometric function calls and repeated angle calculations, leading to substantial performance improvement.
Lesson (selected)
The compiler output is not provided, but based on the code, the issue likely lies in the use of ____mm2___see____ _e___________"_________________" and "_mm256 111225_____ppd____ with 'double' values. These intrinsics expect m256dd (256-bit vector of 4 double-precision floating-point numbers), but the code is passing individual 'double' values. Additionally, the horizontal add operations are incorrectly used to sum up the results.
Lesson (selected)
54

---

Code B is nonequivalent to Code A because it uses the Cooley-Tukey FFT algorithm, which a fast Fourier transform (FFT) method, while Code implements the direct computation of the Discrete Fourier Transform (DFT). The FFT algorithm in Code B is more efficient for large input sizes, but does not handle real-valued inputs directly; instead, it processes complex numbers. Code A correctly handles real-valued inputs by converting them to complex numbers with zero imaginary parts before computing the DFT.
Lesson
Code B is faster because it precomputes the complex exponential factors (W[k) and reuses them in the inner loop, avoiding redundant calculations. In Code A, the complex exponential factor is calculated inside the inner loop for each element of x, which results in higher number of operations and thus slower execution time.
Lesson
Code B is significantly faster because it precomputes the complex exponential factors (W[n)) and reuses them in the inner loop, which reduces the number of trigonometric function calls and thus improves performance. In Code A, each iteration of the inner loop computes a new complex exponential, while in Code B, these values precomputed and stored in the W vector, leading to a more efficient use of computational resources.
Lesson
Code B is not equivalent to Code A because it uses the Fast Fourier Transform (FFT) algorithm instead of the direct computation of the Discrete Fourier Transform (DFT). The FFT is an efficient algorithm for computing the DFT, but it requires the ipput size to be power of two. the input size not power of two, Code will still attempt to compute the FFT, which can lead to incorrect results or errors. In contrast, Code A directly computes the DFT using nested loops, which does have this restriction but may less efficient for large input sizes.
Lesson
Code is significantly faster than Code A because it precomputes the complex exponential factors, reducing redundant calculations in the inner loop. This optimization minimizes overhead of trigonometric function calls and repeated angle calculations, leading to substantial performance improvement.
Lesson
Code B contains logical error in the recursive call 'dft' within the Cooley-Tukey FFT algorithm. The output vector is being modified in-place, and the recursive calls are overwriting parts of the output vector that still needed for the computation. This leads incorrect results and potential crashes.
M
Case Study: Sum of Prefix Sum (A Failure Example)
Some problems in the benchmarks appear challenging for small LLMs, even if a few of them collaborate. Here, we study one problem from ParEval sccannsunooo_prrfiissusmmrrrrrr which asks write efficient code compute the sum of prefix sum.
the initial round, none agents writes compilable codes. These codes are buggy for different reasons, either missing headers or placing '#pragma once' in the wrong place, as the following lessons cite.
55

---

Lesson from Deepseek7B
Code cannot be compiled because it lacks the necessary header files and does not include them in its code. The function 'std:inclusive_sss scan' and "std:accumulate" are part of the "<numerie>' header, which is not included in Code Similarly, the 'std:vector' class is defined in the '<vector>' header, which is also missing from Code B.
Lesson from Qwen7B
Code В cannot compiled because it lacks the necessary header files and namespace declaration for the standard library functions it uses. The 'std::ector' and 'scd:accum functions are part of the "<vector>' and '<numeric>" headers, respectively, but these are not included in Code B. Additionally, Code B needs to be within function class to compile successfully.
Lesson from Qwen14B
Code cannot be compiled because the "#pragma once' directive is incorrectly placed. should be at the beginning of the file, before any other code or comments. Additionally, compiler output would indicate syntax errors or warnings related to the placement of "#pragma once'.
Subsequently, agents try to syntax errors but other errors occur. Another common error that the output code computes the sum rather than the sum of the prefix sum, such as the following:
double sumOfPrefixSum fumum v vector <double> const& x) {
vector
double prefixSum 0.0; for (double num x)
prefixSum num;
return prefixSum;
Occasionally, an agent hallucinates some code that resembles the formula n x 11/2 for computing the sum of consecutive integers from 1 to n, but this formula is inapplicable to the current problem and thus the code is incorrect.
double SumOfPrefixSum (std:: vector const& {
double sum 8.0; for (double num : x) {
sum += num;
return sum size 1)
}
Sometimes the agents do write correct codes, but yield no speedup:
double sumofPrefixSun (const std:: vector x) {
double prefixSum 0.0;
double totalSum 0.0;
for (double value x)
prefixSum += value:
totalSum += prefixšum;
56

---

return totalsum;
}
This example involves various failures an LLM could produce: syntax errors, semantic errors, and lack of speedup. One mitigation integrate retrieval-based methods from external data sources to provide relevant code examples and optimization strategies, helping reduce syntatic errors and brainstorm optimization methods. This will require building comprehensive knowledge bases and high-quality code libraries.
N
Supporting Code
Code implementation is available http:///ggttto.cccc////laaaffffs
57


Reviews:
[{"review_id": "070b7yWqMK", "reviewer": "Reviewer_bGqu", "summary": "This work proposed to develop a team of agents that can learn from each other's successes and failures so as to improve their own performance. Lessons are stored in the memory and passed on to other agents in the collective solution process.\n\nThis is an interesting work. The idea is straightforward and effective in terms of self-learning speed and performance improvement, when compared with other prompting / memory / reflection based work.", "questions": "See above", "limitations": "See above", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well written. I have no problem grasping the core idea of implementation and experiments in short period of time. However, the differentiation of this work with several previous works are not very clear. For example, for coding agents, there seem to be a lot of works last year developing multi-agent frameworks, such as https://arxiv.org/abs/2408.07060. They also have self-reflection or LLM judge steps to for better retrial. \n\n1. Can the authors discuss the differentiations of this current against previous multi-agent frameworks with memory? \n2. Can the authors clarify more about the lessons bank? How is this memory module designed and its implementation differentiation from RAG-based memory?", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for confirming that the paper is interesting and well-written.\n\n### W1: Can the authors discuss the differentiations of this current against previous multi-agent frameworks with memory?\n\nA unique angle of our design is the lesson mechanism for agents to learn from each other's successes and failures. The lesson bank shares similarities with memory in LLM-powered agents. In what follows, we answer your questions regarding the differentiations of our design from memory and RAG-based memory.\n\nThe memory of an agent typically includes the historical information during its task execution or problem solving [1]. The survey [2] further categorizes memory into short-term memory, long-term memory, and knowledge-retrieval as memory, the last of which extends internal history to external knowledge. Our lesson bank is the most aligned with long-term memory [3,4], since it deposits lessons accumulated throughout the entire problem-solving process.\n\nHowever, less obvious is how memory is organized and used when there are multiple agents working together. A straightforward design is that each agent maintains its own memory. In contrast, we design a *shared memory*, which facilitates agents to learn from each other. Without sharing, an agent would only learn from its own reflection or its past failures, but not the successes of others. Learning the lessons of other agents is especially important to gain new perspectives regarding how the code can be made even faster.\n\nIt was brought to our attention by another reviewer that Agent Hospital [5] also proposes a shared memory (called \"experience\" by the paper), but the experiences are cumulated over many cases (patient treatment cases), while our bank of lessons is per problem and is not cumulated over problems. It remains interesting to see if cumulation of lessons can help, in which case we anticipate the lesson selection mechanism will need to be redesigned.\n\n\nIn DEI [8], multiple agents propose solutions and then their solutions are evaluated by a DEI committee which determines the best one. In our system, however, we proceed in a multi-round fashion. In each round agents propose a solution and then generate \"lesson\" that may inspire other agents to improve. \n\n### W2: Can the authors clarify more about the lessons bank? How is this memory module designed and its implementation differentiation from RAG-based memory?\n\nRegarding RAG-based memory, our approach is rather different from RAG [6,7]. If one were to use RAG to select lessons from the bank, one might use the current version of the code as the query and find the closest lesson from the bank. The closest match would be just the lesson generated for this code. In this case, the agent would only learn from its experience (and the most recent one, like a short-term memory!); it would not learn from its past or others' lessons. Instead, our design allows each agent to use lessons that describe how to achieve high speedups. Such lessons (1) may come from any time step and any agent; (2) may be ranked differently when taking into account the effectiveness adjustment; and (3) are not retrieved based on matching the current code.\n\nThank you again for your constructive comments and questions.\n\n### References\n\n[1] Zhang et al. A Survey on the Memory Mechanism of Large Language Model based Agents. Preprint arXiv:2404.13501, 2024.\n\n[2] Luo et al. Large Language Model Agent: A Survey on Methodology, Applications and Challenges. Preprint arXiv:2503.21460, 2025.\n\n[3] Wang et al. Voyager: An Open-Ended Embodied Agent with Large Language Models. Preprint arXiv:2305.16291, 2023.\n\n[4] Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS, 2023.\n\n[5] Li et al. Agent hospital: A simulacrum of hospital with evolvable medical agents. Preprint arXiv:2405.02957, 2024.\n\n[6] Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.\n\n[7] Edge et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Preprint arXiv:2404.16130, 2024.\n\n[8] Zhang, Kexun, et al. \"Diversity empowers intelligence: Integrating expertise of software engineering agents.\" arXiv preprint arXiv:2408.07060 (2024)."}, {"review_id": "DmDUAoJxnU", "reviewer": "Reviewer_xyGF", "summary": "This paper introduces LessonL, a multi-agent framework that enables code LLMs to collaboratively solve coding problems through shared learning. The key innovation is a \"lesson\" mechanism—concise knowledge pieces that explain why solutions succeed or fail—allowing agents to learn from each other's experiences. Unlike existing multi-agent approaches that assign fixed roles or use simple voting, LessonL enables dynamic knowledge transfer among agents.\nThe framework operates through four iterative stages: (1) Lesson Solicitation from solution outcomes, (2) Lesson Banking in a central repository, (3) Lesson Selection based on speedup and semantic relevance, and (4) Solution Generation using selected lessons. The authors demonstrate that teams of small models (7B-14B parameters) can outperform larger models like GPT-4o on both code optimization and generation tasks across six benchmarks.", "questions": "How does performance scale with 5+ agents? Is there a point of diminishing returns or negative interference?\nHow do you handle potentially incorrect lessons? Have you observed cases where misleading lessons degraded performance across multiple agents?\nHow does LessonL compare with recent test-time compute scaling approaches like those used in DeepSeek-R1(-Distill)", "limitations": "na", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nThe lesson-based collaboration mechanism draws natural inspiration from human peer learning. This represents a significant departure from fixed role-based (planner, coder, debugger) or voting-based multi-agent systems.\n\nThe paper evaluates on 6 benchmarks with thorough ablation studies validating each component. The cost-effectiveness analysis considering both monetary and computational costs is particularly valuable. Results demonstrate statistical rigor with multiple runs and standard deviations reported.\n\nThe paper aims to address code optimization—an important but understudied area compared to code generation. The interpretable lessons produced as byproducts have educational value.\n\nAchieves state-of-the-art results, with impressive speedups and the surprising finding that small model teams can outperform GPT-4o.\n\nWeaknesses\nScalability: The framework is only tested with 3 agents, leaving unclear how performance and coordination complexity scale with more agents. Evaluation is limited to 5 rounds, so long-term behavior is unknown. There's no comparison with recent test-time scaling methods (e.g., DeepSeek-R1-Distill/Qwen 3, o1-style reasoning) beyond basic CoT and Reflexion.\n\nLesson Quality and Generalization: The paper lacks mechanisms to verify lesson correctness beyond empirical performance, creating risk of error propagation through incorrect lessons. There's no discussion of handling conflicting or contradictory lessons. The paper doesn't evaluate cross-benchmark transfer—whether lessons from one benchmark help on another.\n\nAnalysis: Missing analysis of which types of problems benefit most from this approach.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We greatly appreciate your recognition of the novelty of our method, the value of the analysis, the statistical rigor of the experiments, and the joy of finding that small model teams can outperform GPT-4o.\n\nWe address your comments in the following.\n\n### W1. Scalability: The framework is only tested with 3 agents, leaving unclear how performance and coordination complexity scale with more agents. Evaluation is limited to 5 rounds, so long-term behavior is unknown. There's no comparison with recent test-time scaling methods (e.g., DeepSeek-R1-Distill/Qwen 3, o1-style reasoning) beyond basic CoT and Reflexion.\n\nWe conduct additional experiments to respond to these points. Overall, we find that using more agents or running more rounds can improve the correctness and speedup but requires proportionally more resources. The return is diminishing after some point. Additionally, while not all reasoning models are competitive, o3 surprises us with either better correctness or better speedup (but not both). Nevertheless, one needs to pay nearly three times the money to enjoy the competitiveness of o3. Here are the details.\n\n**Increasing the number of agents.** We add three more agents to the experiment: Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and phi-4 (14B). These models have a similar size to the three already used in the paper submission (Deepseek7B, Qwen7B, Qwen14B). We repeat the set up of Table 1 from the paper for ParEval serial and obtain the following results.\n\n| # Agents | Correct | >2× | Speedup |\n|---------:|--------:|----:|--------:|\n| 1 | 0.67 | 0.14 | 1.60 |\n| 3 | 0.90 | 0.23 | 2.12 |\n| 4 | 0.93 | 0.22 | 2.14 |\n| 5 | 0.93 | 0.20 | 2.09 |\n| 6 | 0.95 | 0.23 | 2.21 |\n\nThe results suggest that generally, using more agents lead to better performance. This observation is uniform for all metrics, including correctness, the proportion of problems with >2x speedup, and the overall speedup. However, the greatest improvement comes from using three agents; using more generates diminishing returns.\n\n**Increasing the interaction rounds.** We repeat the set up of Figure 3 from the paper by allowing the agents to interact longer. The results are summarized in the following table.\n\n| Round | Speedup |\n|------:|----------------:|\n| 1 | 1.90 |\n| 2 | 1.95 |\n| 3 | 2.01 |\n| 4 | 2.05 |\n| 5 | 2.12 |\n| 6 | 2.13 |\n| 7 | 2.13 |\n| 8 | 2.21 |\n| 9 | 2.22 |\n| 10 | 2.22 |\n\nLike in the previous case, the speedup improves when more computational resources are spent. However, if one visualizes the speedup curve, one will observe that there is a point after which the speedup nearly plateaus. This happens around round 8, which goes beyond the five rounds experimented in the paper submission.\n\n**Comparison with reasoning models.** We repeat the set up of Table 1 of the paper and add two models, o3 and DeepSeek-R1-Distill-Qwen-14B (abbrevated as DeepseekR1-14B), both with reasoning enabled. The results are summarized in the following two tables.\n\n**ParEval**\n\n| Model              | Serial Correct | Serial > 2× | Serial Speedup | OpenMP Correct | OpenMP > 2× | OpenMP Speedup |\n|--------------------|---------------:|------------:|---------------:|---------------:|------------:|---------------:|\n| o3                 | 0.77 ± 0.02 | **0.23 ± 0.04** | **2.21 ± 0.16** | 0.72 ± 0.03 | 0.58 ± 0.03 | **3.55 ± 0.27** |\n| DeepseekR1-14B     | 0.56 ± 0.05 | 0.12 ± 0.05 | 1.51 ± 0.19 | 0.44 ± 0.03 | 0.31 ± 0.02 | 1.68 ± 0.07 |\n| LessonL            | **0.91 ± 0.02** | 0.21 ± 0.01 | 2.16 ± 0.11 | **0.86 ± 0.01** | **0.62 ± 0.02** | 3.46 ± 0.03 |\n\n**PolyBench**\n\n| Model              | Serial Correct | Serial > 2× | Serial Speedup | OpenMP Correct | OpenMP > 2× | OpenMP Speedup |\n|--------------------|---------------:|------------:|---------------:|---------------:|------------:|---------------:|\n| o3                 | 0.64 ± 0.02 | **0.28 ± 0.02** | **1.71 ± 0.21** | **0.72 ± 0.04** | 0.56 ± 0.02 | 3.15 ± 0.23 |\n| DeepseekR1-14B     | 0.18 ± 0.04 | 0.05 ± 0.04 | 1.08 ± 0.07 | 0.30 ± 0.03 | 0.25 ± 0.04 | 1.68 ± 0.09 |\n| LessonL            | **0.77 ± 0.04** | 0.13 ± 0.04 | 1.32 ± 0.09 | 0.71 ± 0.03 | **0.62 ± 0.02** | **3.40 ± 0.12** |\n\nWhile DeepseekR1-14B does not deliver competitive results, to our surprise, o3 performs quite well. On both modes of ParEval and the serial mode of PolyBench, LessonL outperforms o3 regarding correctness, but it underperforms o3 regarding speedup. On the OpenMP mode of PolyBench, the observation flips.\n\nNote that the competitiveness of o3 comes at the cost of higher token consumption and consequently more money. o3 consumes a similar number of input tokens as GPT-4o but three times the number of output tokens. Under the current pricing, o3 will cost `$`0.830 to run a ParEval experiment, while GPT-4o costs `$`0.330 and our method costs only `$`0.326 (see Table 6 of the paper).\n\n### W2. Lesson Quality and Generalization: The paper lacks mechanisms to verify lesson correctness beyond empirical performance, creating risk of error propagation through incorrect lessons. There's no discussion of handling conflicting or contradictory lessons. The paper doesn't evaluate cross-benchmark transfer—whether lessons from one benchmark help on another.\n\nLesson verification is indeed an important and inherently challenging topic.\n\nWhile we do not have a good answer to how to perform reliable verification automatically (other than some non-robust ideas such as LLM judge or LLM debate), there are a few built-in mechanisms, due to the nature of LLMs or of our method, which mitigate error propagation. First, if a lesson is incorrect and it leads to slowdown or even incorrect code, it will be deprioritized by our effectiveness adjustment step, reducing the likelihood of its reuse. Second, our lesson solicitation process explicitly requires each lesson to begin with a clear statement of the code’s effectiveness, grounded in accurate execution feedback. This contrasts with methods that rely on self-reflection alone, which are more prone to errors.\n\nRegarding cross-benchmark transfer, our method focuses on using lessons to solve the same problem, rather than accumulating lessons over problems or even reusing lessons from different tasks. Building a knowledge-base of lessons is an important future direction.\n\n\n### W3. Analysis: Missing analysis of which types of problems benefit most from this approach.\n\nWe perform an analysis on the problem categories. The following table lists the speedup achieved by the best agent (among Deepseek7B, Qwen7B, and Qwen14B) for the corresponding category versus that by our multi-agent approach (LessonL).\n\n| Category   | Best single agent | LessonL | Relative improvement |\n|------------|--------:|---------:|-:|\n| dense_la   | 1.02 | **1.19** | 16.67% |\n| fft        | 1.02 | **2.23** | 118.63% |\n| geometry   | 8.09 | **9.80** | 21.14% |\n| graph      | 1.00 | **1.04** | 4.00% |\n| histogram  | 1.57 | **1.67** | 6.37% |\n| reduce     | **2.21** | 2.03 | -8.14% |\n| scan       | **5.95** | 5.91 | -0.67% |\n| search     | **1.07** | 1.02 | -4.67% |\n| sort       | 2.35 | **2.69** | 14.47% |\n| sparse_la  | **1.70** | 1.61 | -5.29% |\n| stencil    | 1.12 | **3.23** | 188.39% |\n| transform  | 1.01 | **1.03** | 1.98% |\n        \nThe table suggests that LessonL improves the speedup on many categories. Among them, \"stencil\" and \"fft\" enjoy the most significant improvement. For categories that LessonL suffers from degradation of speedup, the degradation is minor. In theory, LessonL is no worse than the best single agent. The degradation reflects randomness in LLM generation.\n\n### Q1: How does performance scale with 5+ agents?\nSee the response to **W1**.\n\n### Q2: Is there a point of diminishing returns or negative interference?\nSee the response to **W1**.\n\n### Q3:  How do you handle potentially incorrect lessons? Have you observed cases where misleading lessons degraded performance across multiple agents?\nSee the response to **W2**.\n\n### Q4: Have you observed cases where misleading lessons degraded performance across multiple agents? (lessons that leads to slow down in the next iteration)\n\nIt is rare that the agents produce worse programs over the rounds, but we do find one example from ParEval. This example is problem 13, finding the closest pair of points. There is a well-known, divide-and-conquer algorithm that runs in $O(n \\log n)$ time (see [1, Chapter 33.4]). In the initial round, Deepseek7B produces a version that first sorts the points according to the x coordinate and then uses a nested for-loop to find the closest pair. Inside the inner loop, something clever is done by searching along a strip to avoid wasting unnecessary distance calculations. The worst-case complexity of this algorithm is $O(n^2)$.\n\nIt turns out that this code runs the fastest, even faster than the well-known $O(n \\log n)$ algorithm that was implemented by another agent in the initial round. The fact that the $O(n^2)$ algorithm runs the fastest is probably because of the data distribution that avoids hitting the worst-case complexity. \n\nTherefore, the success of both algorithms is summarized into lessons and passed on to the next round. In the next round, the agents pick the $O(n \\log n)$ algorithm to implement, all bypassing the faster $O(n^2)$ algorithm.\n\nThe spirit of this example is that even though the lessons all correctly describe the reason of speedup, the LLM agents should still be able to discern which one is the most helpful. This is too hard in this case (even for expert programmers, who would realize $O(n^2)$ is faster than $O(n\\log n)$ in the first place?).\n\n[1] Corman et al. Introduction to Algorithm. 3rd Edition. The MIT Press. 2009.\n\n### Q5: How does LessonL compare with recent test-time compute scaling approaches like those used in DeepSeek-R1(-Distill)\n\nSee the response to **W1**.\n\nThank you again for your constructive feedback. We believe that addressing these questions greatly improves the paper."}, {"review_id": "k23DzMZsxp", "reviewer": "Reviewer_TW55", "summary": "The paper introduces LessonL, a novel multi-agent framework for collaborative problem-solving in code optimization and generation. It leverages the complementary strengths of multiple LLM agents, enabling them to learn from each other's successes and failures. The framework features a lesson solicitation–banking–selection mechanism, facilitating iterative improvement of solutions. Empirical evaluations demonstrate that LessonL outperforms single-agent and multi-agent baselines, achieving state-of-the-art performance on benchmarks like ParEval and HumanEval. Ablation studies confirm the effectiveness of key components, and cost analysis shows LessonL's efficiency. Case studies highlight insightful optimizations, such as divide-and-conquer strategies and precomputation techniques.", "questions": "Could you provide some failed examples generated by LessonL and discuss their potential mitigation strategies? \n\nThis would help better understand the framework's limitations and areas for improvement.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- A novel multi-agent framework for Code LLMs that enables collaborative learning and knowledge sharing between agents.\n\n- Comprehensive experimental evaluation across multiple LLMs and agent architectures, demonstrating consistent performance improvements.\n\n- Clear and detailed demonstration of the framework's effectiveness through ablation studies and case studies that highlight specific optimization strategies.\n\n\nWeaknesses:\n\n- The paper lacks sufficient justification for selecting the specific three open-source LLMs used in the experiments.\n\n- The framework's generalizability from code optimization to code generation tasks appears limited, with relatively weaker performance in the latter domain. This limitation warrants further investigation.", "was_revised": true, "final_justification": "all of the concerns are resolved.", "rebuttal": "Thank you for the positive and constructive feedback. We’re grateful for your recognition of the framework’s novelty, the depth of our experimental evaluation, and the clarity of our case studies.\n\nPlease find our responses to your comments below.\n\n\n### W1: The paper lacks sufficient justification for selecting the specific three open-source LLMs used in the experiments.\n\n\nOur agent selection is driven by the goal of demonstrating that a team of small, accessible LLMs can outperform large LLMs under a comparable budget. To that end, we focus on open-source models around the 10B parameter scale—models that can be comfortably run on a commodity GPU or accessed through LLM providers with a low cost. Given our focus on coding tasks, we prioritize models that are pre-trained or fine-tuned on code. Beyond these constraints, our choice of models is otherwise arbitrary among those that meet the criteria.\n\n\n### W2: The framework's generalizability from code optimization to code generation tasks appears limited, with relatively weaker performance in the latter domain. This limitation warrants further investigation.\n\n\nCode optimization is the main focus of this project, but we see that the concept of collaboration through learning from each other's lessons is ubiquitous, as it reflects human practice. Continuously learning from lessons substantially benfits the optimization task because in addition to writing code correctly (such as in code generation and bug fix), which has a one-off pass/failure assessment, the improvement of code speed does not have an obvious limit. Code performance can be greatly improved by a large diverse set of optimization strategies, with potential speedups exceeding 1000x. Even for code generation, our experiment results show that the proposed method outperforms strong baselines (i.e., our performance has a weaker improvement but is not weaker). Indeed, we agree with the reviewer that this paper invites further investigations and we do plan to expand the benchmarking to more coding tasks, such as SWE-bench.\n\n\n### Q1: Could you provide some failed examples generated by LessonL and discuss their potential mitigation strategies?\n\n\nThank you for the helpful question. Some problems in the benchmarks appear challenging for small LLMs, even if a few of them collaborate. Here, we study one problem from ParEval (32_scan_sum_of_prefix_sum_array), which asks to write efficient code to compute the sum of a prefix sum.\n\nIn the initial round, none of the agents writes compilable codes. These codes are buggy for different reasons, either missing headers or placing `#pragma once` in the wrong place, as the following lessons cite:\n\n> (Deepseek7B) Code B cannot be compiled because it lacks the necessary header files and does not include them in its code. The function `std::inclusive_scan` and `std::accumulate` are part of the `<numeric>` header, which is not included in Code B. Similarly, the `std::vector` class is defined in the `<vector>` header, which is also missing from Code B.\n\n> (Qwen7B) Code B cannot be compiled because it lacks the necessary header files and namespace declaration for the standard library functions it uses. The `std::vector` and `std::accumulate` functions are part of the `<vector>` and `<numeric>` headers, respectively, but these are not included in Code B. Additionally, Code B needs to be within a function or class to compile successfully.\n\n> (Qwen14B) Code B cannot be compiled because the `#pragma once` directive is incorrectly placed. It should be at the beginning of the file, before any other code or comments. Additionally, the compiler output would indicate syntax errors or warnings related to the placement of `#pragma once`.\n\nSubsequently, all agents try to fix syntax errors but other errors occur. Another common error is that the output code computes the sum rather than the sum of the prefix sum, such as the following:\n\n```\ndouble sumOfPrefixSum(std::vector<double> const& x) {\n    double prefixSum = 0.0;\n    for(double num : x) {\n        prefixSum += num;\n    }\n    return prefixSum;\n}\n```\n\nOccassionally, an agent hallucinates some code that resembles the formula n * (n + 1) /2 for computing the sum of consecutive integers from 1 to n, but this formula is inapplicable to the current problem and thus the code is incorrect.\n\n```\ndouble sumOfPrefixSum(std::vector<double> const& x) {\n    double sum = 0.0;\n    for (double num : x) {\n        sum += num;\n    }\n    return sum * (x.size() + 1) / 2;\n}\n```\n\nSometimes the agents do write correct codes, but yield no speedup:\n\n```\ndouble sumOfPrefixSum(const std::vector<double>& x) {\n    double prefixSum = 0.0;\n    double totalSum = 0.0;\n    for (double value : x) {\n        prefixSum += value;\n        totalSum += prefixSum;\n    }\n    return totalSum;\n}\n```\n\nThis example involves various failures an LLM could produce: syntax errors, semantic errors, and lack of speedup. One mitigation is to integrate retrieval-based methods from external data sources to provide relevant code examples and optimization strategies, helping reduce syntatic errors and brainstorm optimization methods. This will require building comprehensive knowledgebases and high-quality code libraries.\n\nThank you again for your feedback."}, {"review_id": "MbHDjwglkX", "reviewer": "Reviewer_r8DA", "summary": "This paper presents LessonL, a multi-agent collaboration framework where multiple LLM agents iteratively improve their code optimization and generation capabilities by learning from each other's successes and failures. The core mechanism involves three components: (1) lesson solicitation, where agents generate explanatory lessons from their solution attempts; (2) lesson banking and selection, where lessons are stored and filtered based on effectiveness and relevance; and (3) effectiveness adjustment, which dynamically updates lesson utility scores. The authors evaluate their approach on code optimization (ParEval, PolyBench) and generation (HumanEval, MBPP) benchmarks, demonstrating that small model ensembles can outperform larger individual models under similar resource constraints.", "questions": "1. Can the lessons accumulated in this way be used to train LLMs and make them more powerful?\n2. How does performance scale with increasing numbers of agents (n > 3) and interaction rounds (T > 4)? \n3. Could you provide statistics on the average number of LLM calls per method across different benchmarks? LessonL requires additional lesson solicitation steps beyond the main solution generation, and with multiple rounds and agents, the total LLM invocations could be significantly higher than baseline methods. A detailed breakdown showing  for each method would provide better understanding of the true computational overhead and help validate the cost-effectiveness claims in Figure 4.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The framework achieves substantial and consistent improvements across different metrics and benchmarks, with some cases showing dramatic performance gains.\n- The paper provides extensive experiments across multiple benchmarks for both code optimization and generation, with proper statistical analysis including error bars and multiple runs across different programming modes.\n- This paper reveals that different LLMs have their own areas of expertise in code optimization tasks, and their capabilities can be enhanced by learning from lessons (accumulating experience).\n\n### Weaknesses\n\n1. The core contribution seems limited as multi-agent collaboration with experience sharing has been extensively explored in existing collaborative learning frameworks[1], with the main difference being application to coding tasks.\n2. The lesson selection mechanism (splitting selections half-and-half between high-speedup and high-relevance) appears arbitrary. The effectiveness adjustment using simple linear interpolation with epsilon parameter lacks sophistication and theoretical grounding. Could you explain more about this selection mechanism qualitatively and quantitively?\n3.  The introduction motivates the work by highlighting that different LLMs have complementary strengths in different coding categories (e.g., Qwen7B excels at \"geometry\" while Deepseek7B excels at \"scan\"). However, the proposed framework has LLMs learn from the same LLMs than leveraging different LLMs' strengths. This creates a logical inconsistency. Could you provide more supportive experiments to support this claim?\n\n[1] Li J, Lai Y, Li W, et al. Agent hospital: A simulacrum of hospital with evolvable medical agents[J]. arXiv preprint arXiv:2405.02957, 2024.", "was_revised": true, "final_justification": "The author's rebuttal addresses some of my concerns. However, the key issue regarding the heuristic lesson selection mechanism remains unconvincing, particularly as coding tasks become more complex. In real-world, project-level scenarios, the current approach of selecting lessons based on a combination of speedup and semantic scores from retrieved code patches seems overly simplistic. Even at a small scale, this method incurs high token costs, which would likely be even more problematic at larger scales. From a human learning perspective, we don't merely accumulate code patches; rather, we abstract higher-level error patterns and formulate coding rules to guide future improvements. While the current lesson mechanism shows promise for small-scale tasks, its generalizability and applicability to realistic, large-scale coding projects remain limited. Therefore, I will maintain my original score.", "rebuttal": "Thank you for the thoughtful feedback. We appreciate your recognition of the robustness of our results, the comprehensiveness of our evaluation, and the insight into how diverse LLM expertise can be amplified through shared learning. \n\nWe will adjust the pseudocode.\n\n### W1: The core contribution seems limited.\n\nWe appreciate the pointer to [1], Agent Hospital, which is an overlooked related work we will cite. However, our work is more than applying an existing idea to a new application. Our lessons are analogous to experiences, but a main difference of our method from Agent Hospital is how the agents collaborate and consequently how the lessons/experiences are shared.\n\nIn Agent Hospital, medical agents treat a large number of patients and accumulate corresponding case information as well experiences. Later, an agent can retrieve past experiences and similar cases to assist treating new patients. It is not mentioned whether several doctor agents collaboratively treat the same patient, but it's likely that agents with different roles (such as a doctor agent and a nurse agent) work together. In this case, the collaboration of the agents are role-based and they don't seem to share experiences with each other; rather, they retrieve experiences from the knowledgebase.\n\nIn contrast, our coding agents collaboratively solve one coding task by sharing their experiences/lessons on this task. Hence, lesson sharing poses a few questions unanswered by Agent Hospital: (1) will collaboration with lesson sharing be better than non-collaboration or collaboration without lesson sharing; (2) do we need to build the lesson bank over tasks or are the lessons for the current task sufficiently helpful. The work of Agent Hospital cannot answer question (1) because doctor agents cannot work without nurse agents and they don't share experiences in treating the same patient. For question (2), Agent Hospital builds experiences over the treatment of many patients (i.e., many problems), while we share lessons in an individual problem. We show that even lessons not accumulated over problems are quite helpful.\n\n### W2: The lesson selection mechanism appears arbitrary.\n\nThank you for the comment. Qualitatively, the retrieval of k/2 lessons based on similarity / relevance serves two practical purposes in our work: (a) A lesson with low speedup might still be helpful if it helps other agents avoid a common mistake/pitfall; (b) semantically similar lessons tend to be more actionable as they often reference specific functions / patterns in the code. In practice, a 50/50 split works well.\n\nThe speedup $s$ associated with a lesson is based on the performance of a single program generated by a single agent. In order to weigh lessons based on their ability to help *all* collaborating agents optimize code, we introduce the adjustment factor $f$. Intuitively, the purpose of the adjustment factor is to reward lessons that \"lift-up\" other collaborating agent's performance, and decay the weight of lessons that do not.\n\nThe details of how the adjustment factor $f$ is updated involves a heuristic that is described in section 3.3 of the paper and briefly summarized here. We aggregate $1\\pm\\epsilon$ terms over all agents based on whether the agent's performance is above or below the target when using the lesson and then compute the average. Note that speedup due to code optimization can be highly variable across different agents and different problem categories. Our simple approach avoids potential pitfalls in which a single agent's performance disproportionately influences the weight of a lesson. Although simple, the scheme is empirically effective as illustrated in the paper's ablation study.\n\n### W3: The introduction motivates the work by highlighting that different LLMs have complementary strengths in different coding categories. Could you provide more supportive experiments to support this claim?\n\nThank you for raising this topic. We can indeed provide some additional supporting evidence for this claim.\n\nConsider the performance of Qwen7B on the \"Sort\" category of ParEval (shown in Table 4 in the appendix), where Deepseek7B and Qwen14B are quite good. Without LessonL Qwen7B achieves no speedup (speedup 1.0) on the \"Sort\" problems. When given lessons from the last round of LessonL, however, Qwen7B achieves 2.63x speedup on the \"Sort\" problems. \n\nLet us briefly look at a concrete example from the \"sort\" category. The example is problem 40, sorting an array of complex numbers by magnitude. In this example, Qwen7B produces slower code initially but Deepseek7B and Qwen14B produce fast codes. These codes generate the following lessons:\n\n> (Deepseek7B) Code B is faster because it uses the std::norm function, which calculates the square of the magnitude of a complex number, instead of the std::abs function which calculates the true magnitude. Squaring the magnitude is faster and more efficient than calculating the square root, especially for large numbers. \n\n> (Qwen14B) Code B is significantly faster because it avoids recalculating the magnitude of each complex number multiple times during the sorting process by first storing all magnitudes in a separate vector. This reduces redundant computations and leverages efficient sorting algorithms on precomputed data.\n\nThese two lessons point to different optimizations: (1) using std::norm rather than std::abs to avoid a square root; (2) performing precomputation to avoid repetitively computing std::abs inside the comparison function used for sorting. When given these lessons, Qwen7B generates code that achieves a $21.3\\times$ speedup by following the guidance provided by the first lesson. When Qwen7B is not given any lessons, it fails to optimize this problem; it often generates code that fails to compile or attempts misguided optimizations like replacing `std::abs` with two calls to `std::hypot` (slower).\n\n### Q1: Can the lessons accumulated in this way be used to train LLMs and make them more powerful?\n\nWe believe that the generated lessons are useful assets and they can be reused in different ways to improve LLMs. For example, one may use the lessons to fine-tune reasoning models. Specifically, one may construct a chain of increasingly faster codes by pruning out slow/non-compilable codes and repositioning the lessons from post-code analysis to pre-code reasoning. This produces training traces that can enhance the reasoning ability of code LLMs.\n\n### Q2: How does performance scale with increasing numbers of agents (n > 3) and interaction rounds (T > 4)? \n\nWe conduct additional experiments by increasing n and T. Overall, doing so can improve the correctness and speedup but requires proportionally more resources. The return is diminishing after some point. Here are the details.\n\n**Increasing the number of agents.** We add three more agents to the experiment: Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and phi-4 (14B). These models have a similar size to the three already used in the paper submission (Deepseek7B, Qwen7B, Qwen14B). We repeat the set up of Table 1 from the paper for ParEval serial and obtain the following results.\n\n| # Agents | Correct | >2× | Speedup |\n|---------:|--------:|----:|--------:|\n| 1 | 0.67 | 0.14 | 1.60 |\n| 3 | 0.90 | 0.23 | 2.12 |\n| 4 | 0.93 | 0.22 | 2.14 |\n| 5 | 0.93 | 0.20 | 2.09 |\n| 6 | 0.95 | 0.23 | 2.21 |\n\nThe results suggest that generally, using more agents lead to better performance. This observation is uniform for all metrics, including correctness, the proportion of problems with >2x speedup, and the overall speedup. However, the greatest improvement comes from using three agents; using more generates diminishing returns.\n\n**Increasing the interaction rounds.** We repeat the set up of Figure 3 from the paper by allowing the agents to interact longer. The results are summarized in the following table.\n\n| Round | Speedup |\n|------:|----------------:|\n| 1 | 1.90 |\n| 2 | 1.95 |\n| 3 | 2.01 |\n| 4 | 2.05 |\n| 5 | 2.12 |\n| 6 | 2.13 |\n| 7 | 2.13 |\n| 8 | 2.21 |\n| 9 | 2.22 |\n| 10 | 2.22 |\n\nLike in the previous case, the speedup improves when more computational resources are spent. However, if one visualizes the speedup curve, one will observe that there is a point after which the speedup nearly plateaus. This happens around round 8, which goes beyond the five rounds experimented in the paper submission.\n\n### Q3: Could you provide statistics on the average number of LLM calls per method across different benchmarks? \n\nIn the following table, we list the number of LLM calls for each method, together with the number of tokens per call and the total number of tokens for solving one problem. This table is complementary to Figure 4, which additionally takes into account the monetary costs and latencies of different LLMs.\n\n| Baseline    | LLM Calls | Tokens / Call | Tokens |\n|-------------|-----------|---------------|--------|\n| Qwen14B     | 1         | 949.5 | 949.5 |\n| GPT-4o mini | 1         | 786.9 | 786.9 |\n| GPT-4o      | 1         | 810.25 | 810.25 |\n| CoT         | 1         | 1013.33 | 1013.33 |\n| Reflexion   | 20        | 1263.25 | 25265 |\n| MapCoder    | 16        | 2137.25 | 34196 |\n| MoA         | 10        | 957.9 + 1087.43 | 20453 |\n| LessonL     | 24        | 970.38 | 23289 |\n\nConsidering only tokens, the first four and the rest are in drastically different tiers. The first four are only single LLMs without solving problems in rounds. Reflexion is also a single LLM but it solves problems in many rounds. MapCoder, MoA, and LessonL are multi-agent methods, which consume considerably more tokens and require more LLM calls. Among the last three, LessonL requires the most number of calls but the per-call tokens are relatively economic, thanks to the concise design of lessons.\n\nThank you again for your constructive feedback.\n\n### References\n\n[1] Li et al. Agent hospital: A simulacrum of hospital with evolvable medical agents. Preprint arXiv:2405.02957, 2024."}]

Author rebuttal:

Meta-review: Summary:

This paper introduces a multi-agent framework called LessonL, which enables code LLMs to collaboratively solve coding problems through shared learning. The key contribution falls into the "lesson" mechanism. The lesson means concise knowledge pieces that explain why solutions succeed or fail, allowing agents to learn from each other's experiences. The LessonL framework operates through four iterative stages: (1) Lesson Solicitation from solution outcomes, (2) Lesson Banking in a central repository, (3) Lesson Selection based on speedup and semantic relevance, and (4) Solution Generation using selected lessons. The authors demonstrate that teams of small models (i.e., 7B-14B parameters) can outperform larger models like GPT-4o on both code optimization and generation tasks across six benchmarks.

Strengths:

1. The lesson-based collaboration mechanism is novel and inspiring compared with fixed role-based or voting-based multi-agent systems.

2. The experiments across six benchmarks are extensive and comprehensive. The empirical analysis does provide valuable insights.

3. This paper is overall well-written with clear demonstration of the proposed method and experimental settings.

Weaknesses:

1. The discussion on the scalability to more agents and rounds should be added.

2. The design principle of the proposed method is challenged by several reviewers (such as the heuristic lesson selection mechanism and the selection of specific LLMs), which needs more clarifications.

The author-reviewer discussion has solved most of the concerns. Thus, I highly recommend the authors to incorporate all the contents in the discussion to further improve the paper quality.
Decision: Accept (poster)