EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
Authors: Shihan Dou
Ming Zhang
Chenhao Huang
Jiayi Chen
Feng Chen
Shichun Liu
Yan Liu
Chenxiao Liu
CHENG ZHONG
Zongzhang Zhang
Tao Gui
Chao Xin
Wei Chengzhi
Lin Yan
Qi Zhang
Xuanjing Huang
Venue: NeurIPS 2025 poster ()
Keywords: benchmark
LLMs
evaluation
Primary area: evaluation
TL;DR: We introduce EvaLearn, a pioneering benchmark designed to evaluate LLMs on their learning capability and efficiency in challenging tasks.

Abstract: We introduce EvaLearn, a pioneering benchmark designed to evaluate large language models (LLMs) on their learning capability and efficiency in challenging tasks, a critical, yet underexplored aspect of model potential. EvaLearn contains 648 challenging problems across six task types, grouped into 182 sequences, each sequence dedicated to one task type. Diverging from most existing benchmarks that evaluate models in parallel, EvaLearn requires models to solve problems sequentially, allowing them to leverage the experience gained from previous solutions. EvaLearn provides five comprehensive automated metrics to evaluate models and quantify their learning capability and efficiency. We extensively benchmark nine frontier models and observe varied performance profiles: some models, such as Claude-3.7-sonnet, start with moderate initial performance but exhibit strong learning ability, while some models struggle to benefit from experience and may even show negative transfer. Moreover, we investigate model performance under two learning settings and find that instance-level rubrics and teacher-model feedback further facilitate model learning. Importantly, we observe that current LLMs with stronger static abilities do not show a clear advantage in learning capability across all tasks, highlighting that EvaLearn evaluates a new dimension of model performance. We hope EvaLearn provides a novel evaluation perspective for assessing LLM potential and understanding the gap between models and human capabilities, promoting the development of deeper and more dynamic evaluation approaches. All datasets, the automatic evaluation framework, and the results studied in this paper are available in the supplementary materials.

EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
Shihan Dou't Ming Zhang" Chenhao Huang', Jiayi Chen', Feng Chen
Shichun Liu', Yan Liu?, Chenxiao Liu?, Cheng Zhong?, Zongzhang Zhang², Tao Gui!
Chao Xin², Chengzhi Wei Lin Yan", Qi Zhang Xuanjing Huang Fudan University 2ByteDance Seed Nanjjngg University *Shanghai Innovation Institute Shanghai Collaborative Innovation Center of Intelligent Visual Computing {shdou24, mingzhang23) 8m. fudan. edu. cn [tgui, qz, xjhuang) @fudan edu. en
Abstract
We introduce EvaLearn, pioneering benchmark designed to evaluate large lan- guage models (LLMs) on their learning capability and efficiency in challenging tasks, a critical, yet underexplored aspect of model potential. Eval contains 648 challenging problems across six task types, grouped into 182 sequences, each sequence dedicated to one task type. Diverging from most existing benchmarks that evaluate models in parallel, EvaLearn requires models to solve problems sequen- tially, allowing them leverage the experience gained from previous solutions. EvaLearn provides five comprehensive automated metrics to evaluate models and quantify their learning capability and efficiency. We extensively benchmark nine frontier models and observe varied performance profiles: some models, such as Claude-3.7-sonnee start with moderate initial performance but exhibit strong learn- ing ability, while some models struggle to benefit from experience and may even show negative transfer. Moreover, investigate model performance under learning settings and find that instance-level rubrics and teacher-model feedback further facilitate model learning. Importantly, observe that current LLMs with stronger static abilities do not show a clear advantage in learning capability across all tasks, highlighting that EvaLearn evaluates new dimension of model perfor- mance. We hope EvaLearn provides a novel evaluation perspective for assessing LLM potential and understanding the gap between models and human capabilities, promoting the development of deeper and more dynamic evaluation approaches. All datasets, the automatic evaluation framework, and the results studied in this paper are available at htttt:://// //gp::///itee github.. Cactub.coo/BBBB wwwc Com/ByteDance. Seed/ EvaLLarn
1 Introduction
Large language models (LLMs) have advanced rapidly in recent years, driving remarkable progress in wide range of applications [90]9 26] [5, Rigorous evaluation of these models is essential
for understanding their current capabilities, identifying areas for improvement, and guiding the
development of more advanced LLMs [10
39
27
[02]
Although numerous benchmarks have been proposed assess various aspects of model performance,
the overwhelming majority rely on parallel evaluation paradigm [93] 71 43 7 47 40
33] 85).
In this setting, models are tested on independent and identically distributed (i.i.d)) samples, and their overall performance is reported as aggregate metrics. However, such benchmarks primarily
Equal contribution.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Problems
Sequentially solving, problems
comprohenssive metrics
Da sequeex scouray
venfyring
Tesk typee:
2-3
Sequential Reasoning
Seodback
Question: A knight placed the tup-lett corner an BrB grit. what the fewoutt nimber of moves to "trep' the knight...
Rasitin
JiMos
i
Camsnicaal Anever::
184) Mueera position and sakitin & Aosrang Offec offist lourzel adttion
1. Underrstand the knights move:... 2. Amalyae the problimn....
(1.1) (12)
Rubvice
The student's is answer must
Aoweage rumber connecttivn
esplicilly requtrement will result score of c..
sarmpp OCEEERY
Vartfying through instance-levd rubeic
Figure 1: Overview of EvaLearn. Instead of parallel evaluation, EvaLearn requires models solve problems sequentially, thereby systematically evaluating the LLM's learning ability and efficiency.
measure static abilities, while overlooking an equally important dimension: model's 's capability to learn and adapt within a specific task (i.c.. learning capability), as well as the speed at which this learning occurs (i.e.. learning efficiency). These dynamic learning abilities are fundamental indicators of human learning potential and intelligence [6] 20 79 36), yet remain largely unexplored in the evaluation of LLMs-mainly mainly because the prevailing parallel evaluation paradigm is inherently unable to capture such learning dynamics.
To address this gap, introduce EvaLearn, challenging benchmark designed systematically quantify the learning capability and efficiency of LLMs through a novel sequential evaluation paradigm. Considering the scarcity of carefully categorized and challenging related problems in existing benchmarks, we construct 648 challenging problems from scratch. These problems are organized into 182 sequences. with each sequence containing seven problems from the same task type and spanning six distinct task categories. Models are required to sequentially solve each problem within a sequence, with each solution automatically evaluated using a combination of instance-level rubrics and an LLM-as-a-judde [02 69 framework. EvaLearn is designed to assess these two core aspects by testing whether models can leverage experience gained from solving previous problems to improve their performance on subsequent ones. Moreover, EvaLearn includes suite of five metrics to comprehensively quantify the learning capability and efficiency of a model. Importantly, these metrics are decoupled from the specific learning methods employed (i.e., how models utilize prior experience from earlier problems), highlighting the extensibility and flexibility of Evaleeann
We conducted a comprchensive study with nine frontier LLMs on EvaLearn. We summarize several key findings: (a) Models exhibit diverse learning capabilities across different task types. They are generally more adept at leveraging prior experience to solve tasks involving mathematical and logical reasoning, while tasks such as summarization tend to rely more on knowledge acquired during pre-training and the model's inherent abilities. Moreover, learning efficiency also varies significantly across tasks. (b) Thinking- based LLMs typically outperform non -onnthinniin--bb based LLMs in both learning capability and learning efficiency. They are better able to utilize experience to solve new problems and also show greater learning stability, being more likely to solve multiple problems consecutively within a sequence. (c) Feedback learning, which enables models to solve problems with the help of feedback from previous solutions and rubric-based evaluations, significantly enhances both the learning capability and efficiency of models. Moreover, it is often more effective than demonstration learning, which simply provides previous problems and canonical solutions as context. (d) Indicators of learning capability and learning efficiency should be considered jointly. These metrics together provide a comprehensive assessment of a model's 's learning potential. Moreover, they not strongly correlated with static model capability. Even LLMs with higher static performance do not demonstrate a clear advantage in learning capability across all tasks.
More insightful findings are presented in Section 3 and Appendix G We hope Eval.earn offers new perspective on assessing the learning potential of L.LMs, which a key indicator of human- like intelligence. EvaLeam has significant potential to foster deeper understanding of model

---

capabilities within the community, promote the development of more effective learning methods, and serve as a pioneering step toward more dynamic and realistic evaluation paradigms.
EvaLearn
In this section, we first describe dataset composition and task types. We also provide an example to clearly illustrate the data structure. Section 2.2 presents the automated evaluation procedure based on instance-level rubries. Section 2.3 introduces five metries designed to quantify the learning capability and efficiency of LLMs. We also present the data annotation process in Appendix E
2.1 Datasets
EvaLearn is designed evaluate the LLM's learning capability and efficiency. It contains 648 challenging problems, all carefully constructed from scratch and organized into 182 sequences. Fach sequence contains seven problems under the same task category. Problems within a sequence are interrelated and collectively challenge the model's 's learning potential. Figure provides an overview of EvaLearn. Each problem within sequence includes a question and a canomical answer. Since most of these challenging questions cannot be reliably evaluated using pre-defined rules, each problem is also accompanied by a wurtten rubric that defines the criteria for assessing the correctness of model responses (details on the evaluation process are provided in Section 2.2
In contrast to most existing benchmarks that evaluate models in a parallel setting, EvaLearn requires models to solve problems sequentially within each sequence. This sequential setup is specifically designed to evaluate model's ability to accumulate experience and leverage feedback from previous problems, as reflected in its performance on subsequent problems within the same sequence.
Task types. EvaLearn comprises six distinct task types, with each sequence belonging to one of these categories. These task types include: (1) Summarization (Sum) that evaluates whether models can improve the accuracy and coverage of summaries by leveraging prior experience; (2)
Classification (Cla) that assesses a model's 's ability to enhance its categorization skills from solving a series of classification problems; (3) Extraction (Ex) that measures whether models can progressively improve the accuracy and completeness of key information extraction; (4) Logical Reasoning (LR)
that tests whether models can learn from previous errors and improve logic reasoning ability: (5)
Mathematical Reasoning (MR) that examines whether models can quickly master mathematical
problem-solving methods by utilizing feedback from earlier problems: (6) Sequential Reasoning (SR) that evaluates whether models can enhance their ability to solve sequence-basee problems by learning from historical experience, including clarifying event steps and reasoning logic.
Table I presents the statistics of EvaLearn. Across all tasks, the average word counts per question is 315.45. Solving these problems requires models to possess strong and diverse capabilities. The results and analysis for various models on EvaLearn are provided in Section B
An example of sequence. We use example sequence to showcase the relationships among problems within sequence, as illustrated in Figure which involves solving In this case, the puzzle consists of 16 tiles, one of which is a blank space (denoted by '>'), while the remaining tiles are filled with different symbols. By moving the blank space and swapping with adjacent tiles, the goal is to restore the puzzle from given initial state to a target configuration through a series of logical moves. This task is designed to assess the model's logical reasoning ability, and all problems within this sequence Figure preseett another problem from the same sequence, which differs mainly in grid size, the initial state of the puzzle, and symbol representation.
4 sliding puzzle.
centered around the sliding puzzles and logical reasoning.
These problems are related and together assess the model's capability for logical reasoning. Humans are able to gain experience and improve their performance by solving a series of such problems 38 766.. By requiring models to solve these problems sequentially within a sequence, EvaLearn evaluates whether models can similarly learn from experience, thereby measuring their learning efficiincy.
2.2 Automatic Evaluation
Most of the challenging problems in Eval earn cannot be reliably evaluated using rule-based verifiers, as many questions may have answers that are difficult to verify with pre-defined rules or may allow for
3

---

Table 1: Statistics of EvaLearn.
Problem rate
Average words per question
Average words per canonical answer
Task Type
# Problems
Sequences
Summarization
16
1.87
959.97
220.79
Classification
13
1.90
203.18
149.31
Extraction
60
17
1.98
674.89
78.60
Logicul Reasoning
360
102
1.98
227.49
81.98
Mathematical Reasoning
17
1.98
137.03
336.76
Sequential Reasoning
17
1.98
141.38
334.30
Overall
648
182
1.966
315.45
146.05
multiple correct answers. To address this, we employ an instance-level rubric combined with LLM- as-a-judge [25) to assess whether model outputs satisfy the corresponding rubrics. The evaluation prompt is shown in Figure which includes three demonstrations to ensure that the judging model follows the instructions. In all experiments, we use GPT-4o as the judging model.
We conduct an additional experiment to validate the effectiveness of our automatic evaluation process. Specifically, for each problem EvaLearn, we randomly select three models from model list (as shown in Table 3 to generate answers, which are then evaluated using our evaluation framework. Human annotators further verify whether the judging model can successfully assess the correctness of the model outputs through the instance- rubric. Results show that the evaluation accuracy exceeds 95% for all tasks, demonstrating the high reliability of our evaluation framework that combines instance-level rubrics with LL-Maa---juddee consistent with findings from previous studies (69; 91].
2.3 Evaluation Metries
In EvaLearn, models are required sequentially solve all problems within sequence. All evaluation metrics are computed based on the model's correctness across these problems, Notably, the choice of learning method, such as learning from demonstrations (i.e., in-context learning) B 53 or learning
from feedback [66], is decoupled from the evaluation metrics. This design ensures that EvaLcarn remains extensible and flexible, supporting a wide range of learning strategies.
Let N = 182 denote the number of sequences and M = 7 the number of problems in each sequence in Evaleern. Let Vn.m E {0, 1} indicate whether the m-th problem in the n-th sequence is answered correctly (1 for correct, 0 for incorrect). We define the following five metrics to comprehensively evaluate the learning potential of models in sequential problem-solving:
(1) Overall sequence accuracy (Acc). This metric reflects the model's overall performance in sequential problem-solvingg and serves an indirect indicator of its learning capability. It is computed as the average accuracy across all problems and all sequences:
Acc
Nn.m
(I)
NM
Higher values indicate better overall performance. We further compute the position- position-wise wise accuracy curve, i.e., the accuracy at each problem position across all sequences:
Acc..
1 N
Σ Vn.m
(2)
==1
Based on this curve, we define a helpful metric, i.e., the slope of the fitted accuracy curve, to capture dynamic learning behaviors.
(2) Slope of fitted accuracy curve (k): This metric measures the model's 's learning speed across all sequences by fitting straight line to the position-wise accuracy curve using least squares regression 288. It quantifies how quickly the model's accuracy improves as progresses through sequences.
Specifically, let Acem. denote the average accuracy at position m across all sequences. The fitted line given by y = kr + b, where the problem position and l is the estimated slope obtained by minimizing the sum of squared errors:
A: arg
(Accm (lom b))
(3)
m=1

---

A higher k indicates greater learning efficiency and a faster rate of improvement.
(3) Average position of first correct solution (Pira). This metric measures the average position in the sequence where the model achieves its first correct solution, indicating its initial ability, learning capability, and efficiency. For each sequence n, let pr, denote the position of the first correct solution (or M + 1 if none is correct). The metric is calculated as:
N
P Plestt
Σ Pa
(4)
N
Lower values indicate that the model can achieve its first correct solution earlier in the sequence.
(4) Average offset of first learned correct solution (Poffer). This metric measures how quickly a model begins to learn within a sequence while discounting pre-existing knowledge. For each sequence n, let 60)) be the position of the first problem answered incorrectly under zero-shot (parallel)
solving: problems are correct in zero-shot, set i(*) M+1. Let A Afsedback {0, 1] denote correctness at position i under zero-shot and under the sequential solving setting, respectively. We define the per-sequence offset to the first learned correct solution as:
d, min {i :())
((m)
An,i
feedback
1, 2 -
(5)
with the convention that d,, M+1 to if no such i exists, and d, when 10mh metric computed by averaging over sequences:
(n)
M+1. The
N
Poftset
Σ
dn
(6)
N
1=1
Lower values indicate that the model begins to learn earlier after excluding problems solved due to pre-existing knowledge.
(5) Average number of consecutive correct solutions (Noonse). This metric reflects the model's capability to leverage experience to consistently solve problems within sequence, resulting in fewer errors. It indirectly indicates how effectively and stably the model learns. For each sequence n, we compute the longest of consecutive correct solutions, then average across all sequences:
1
Nconsee
max (b a Vo.a = Vn.a+1 = yr 1} IsasbsM
(7)
N
Higher values indicate greater consistency, while lower values suggest intermittent errors and less stable learning.
(6) Post-warmup accuracy (Acepw-K), K). This metric reflects model performance after an initial "warmup" phase, i.e., after some experience has been accumulated. For each sequence, we exclude the first K problems and compute the average accuracy the remaining problems:
Accpw-K
N(M K
Vn.m
(8)
Higher values indicate better adaptation and learning after the warmup.
These metrics provide a comprehensive and nuanced evaluation of dynamic learning behaviors of models in sequential problem-solving, fully capturing learning capability and efficiency. In the following sections, we use these metrics to thoroughly analyze the learning potential of various LLMs.
3 Benchmarking LLMs with EvaLearn
In this section, we compare two problem-solving paradigms, including parallel solving and sequential solving, investigate the learning capability and efficiency of LLMs.
3.1 Setup
Parallel Solving. This paradigm includes two settings: (I) Zero-shot: Models solve each problem independently, without access to any experience from previous problems. This setting aligns with
5

---

Table 2: Comparison of accuracy between zero-shot (parallel solving) and feedback Learning (sequential solving). Values in parentheses indicate the difference between the two methods. Full task names for the abbreviations can be found in Section 2.1 Blue-colored cells indicate the best performance for each task category under few-shot and feedback learning settings, respectively. All
values are shown as %%
Model
Paradigu
Sum
Cla
Ex
MR
SR
Overall
Non- Non-thinking-Bae based
DeepSeck- V3
Zetursact
81.7
72,9
45.0
25.6
71.7
60.0
43.3
Feedback Leoming
76.8
74.7 (+1.8)
36.1 (4.99
(4.7)
(13.)
54.6 (-5.4)
(1.5
Clanle 3.7 Sonnet
Feedback Louming
Zero-shs
(..0
76.7
(+11.)
64.5
46.2
48.3
15.5
(57.4)
8.1
(3156))
48.3
(+16.)))3)
33.3
(57.2)
28.4
GPT-4o
Feedback Leaming
(-..)
73.6 (+2.8)
70.8
48.7 (+3.7)
(277)22.)
61.3 (+18.)
(+0.5
31.3
(2.7
Douban-1.5-Pro
Feedback Leuming
Zeco shot
(4.3))
71.7
68.1 (-2.))
70.8
42.9 (22.)
45.0
108 (+0.5)
10.3
(+97)))7)
61.7
36.1 (-5.6)
41.7
31.2(-011)
$1.3
Qwen2.5-326 --b Instract
Zeru-shet Feedback Leaminng
(6.9))
(42.2)
60.4
3033 (-1.4)
(+2.9)
(9.6
21,7
(+8.6)
(+1.7)
Thinking- haved
OpenAt-aS-mink
Feedback Leuming
Zoco shot
(+19.)
63.0
73.6 (+9.0)
04.6
47.1 (-1.2)
48,3
59.9 (+141)
45,8
80.7 (+7.4)
73.3
782 (+1.9)
73.3
64.8 (+10.5
24,3
Douban-1.3 Bomban-1.5-Thinkinng T.i.Ihii Pro
79.2
Feedback Leaming
(-2.9
(-8.9)
(:2.1)
(-..))
(3.77
51.6(-555
DeepSeekkRR R1
Zero-shot
86.7
99.6
48.3
41.7
78.3
66.7
55.7
Feedback Leaminn
89.3 (+2.6)
79.1 (:1.5)
48. (10.4)
29.6 (-12.)
748 (335)
(11.4)
46.4 (-9.3)
Clando-3 3.7-5ommss--
Feedback Leuming
Zoro-shet
(8.1)
802
66.7
48.7 (45.4)
43.3
188 (+6.3)
(+1.3)
46.7
46.2 (+14.5)
31,7
(+6.2)
31.2
G.mPini-...PPo
FeodBack Leuming
Zero-shet
89.3 (44,3)
85.0
75.8 (42.9)
72,9
$2.1 (-2.9
35.0
(-2662))
62.5
(-.3))
90.0
79 (43.1)
76.7
(-13
68.3
Gomini P.5-
FeedBack Leamiing
Zoro shot
(-19))
81.7
66.7
(-7.4)
(13(7)77)
38.3
(42.2)
16.1
(+11.)
70,0
5L.
(-15
36.3
the evaluation approach used in most existing benchmarks, assessing a model's inherent ability to solve challenging problems without any learning opportunity. The system prompt is shown in Figure T9 (II) Few-shot: For each problem, we provide three demonstrations from the same task as examples (i.e., 3-shot), offering models guidance on output format and problem-solving approach. The demonstrations are identical for all problems within each task type. The system prompt is shown in Figure 20
Sequential Solving. To investigate models' ability to learn from experience, we utilize two sequential learning paradigms: (I) Demonstration learning: Models are provided with all previous problems and their corresponding canonical answers from the same sequence before solving the current problem, similar to in-context learning. The system prompt is shown in Figure 21 (II) Feedback learning: When solving the current problem, models receive context all previous problems, their solutions, and detailed feedback on their own prior solutions, as assessed by a judge using instance-level rubrics. This setting evaluates whether models can leverage their previous experience to improve on subsequent problems. The system prompt is shown in Figure 22
In all experiments, we evaluate nine frontier LLMs, including both thinking-basedd and non- [4), DeepSeek-RI thinking-basee models: Claude-3.7-Sonmet 4). 1eoio--.--hiiiiin--P- [65], ApenAl-o3-mini (57]. 26). DeepSeck-V3 45). Doubao- Brabbao--15-Prri 1.5-Pro 19,
GPT-4o [55], and Qwen2.5-32b-Inste
[90). See Appendix F for details on these models.
3.2 RQ I: Can LLMs learn a task by engaging with sequence of problems?
LLMs exhibit varying abilities learn from problem sequences, with differences observed across both models and task types. Moreover, most models also demonstrate better performance after a warm-up phase.
Finding 1:
Table summarizes the differences in overall accuracy between the feedback learning and zero- shot paradigms. We observe that five models benefit from sequence learning, while four models experience slight declines, indicating that some models can effectively leverage prior experience to solve problems within the same task. Thinking-baas based models, in particular, exhibit more pronounced performance shifts, with OpenAl-o3-miní achieving the highest overall improvement (+10.5%)
Moreover, most L.LMs show improvement on mathematical reasoning and classification tasks when learning from experience. For example, GPT-4o and Claude-3. Slaude-3.--oomme 7-S-onnt--TT---- --Snnet-TThnking Thinking achieve gains of

---

OpenAl-o3-mini
OpenAl-o3-mini
Average
Average
4.65
Nonsee
4.08
4425
3.73
3 .D8
2.06
1.57
2.35
1:85
1.62
1.19
-1135
1.06
Cla
Ex
Sum
LR
MR
SR
Cla
Sum
LR
MR
SR
Task
Task
Figure 2: (Left) Average number of consecutive correct solutions (N (Nconsec) (Right) Average position of the first correct solution (Prna). Results are shown for OpenAl-o3-ming with each node representing a sequence.
18.0% and 13.5% in mathematical reasoning and classification, respectively. This improvement likely results from these tasks having clear solution steps, enabling feedhack pinpoint reasoning errors and help models effectively learn specific problem-solving strategies.
In contrast, most models experience performance declines the summarization task; for instance, 7 out of models perform worse after feedback in summarization. This may be because the summarization task rely more heavily on knowledge and instruction-folloving abilities acquired during pre-training, and additional experience may sometimes interfere with the model's 's ability to solve the current problem.
We also analyze models' Post- Dest-Warmup Accuracy (Accpw-K) K)) with results for Claude-3, s.aude-3.77- 3.7-Sonnet Thinking shown in the left side of Figure 3 and additional models presented in Appendix G.6 We observe that most models achieve higher accuracy the later stages of the sequence, particularly among thinking- based models. This further demonstrates that models can use early problems as practice, leveraging the experience gained to solve subsequent, related problems more effectively.
Learning stability varies significantly across different tasks and models. For certain tasks, such as summarization, current models are more adept at leveraging their inherent knowledge to solve problems rather than drawing experience gained from previous problems.
Finding 2:
We further investigate the learning stability of LLMSs by analyzing the average number of consecutive correct solutions (Nconsee. The left side of Figure 2 presents the results for o3-mini, with additional results for other models provided in Appendix G.5 Each colored node in the figure represents a sequence in the dataset, and nodes positioned higher indicate that the model solves more problems consecutively within that sequence.
From the results, we first observe that the average number of consecutive correct solutions varies notably across tasks. For most models, logical reasoning tasks present greater challenges for main- taining long streaks of correct solutions. For example, Sauude-.7--oomee and GPT-4o fail entirely or manage to solve only one problem consecutively in over half of the sequences. However, there are substantial differences between thinking-based and non-thinking wat-thiiiing--aaee Doubao-1.5-Prt struggles to maintain consecutive correct solutions logical reasoning, Doubao- 1.5- Thinking-Pae Pro exhibits much greater learning stability. Similar patterns appear in other model pairs, such as DeepSeek- V3 vs DeepSeek-RR RI and GPT-4o vs 03-mini, where the thinking-based variants generally achieve higher average numbers of consecutive correct solutions. For example, 03-mini achieves an average maximum streak of 3.42 consecutive correct solutions per sequence, compared to only 2.58 for GPT-4o. Combining these results with Table 2 conclude that Lanking-based models not only benefit more from prior experience to improve performance, but also maintain higher stability and are more likely to solve multiple related problems in succession.
models. For instance, while
Interestingly, in the summarization task, models tend have relatively high Nessse values. However, these same models show marked decline in AcCpw-K. indicating that their performance on later problems in the sequence worse than on earlier ones. This suggests that, for such tasks, the capability to solve multiple problems consecutively may rely more on the model's inherent knowledge and static ability, rather than on learning from experience within the sequence. This observation further supports the conclusion in Findings I.
7

---

Hattrarial Summetbakke Nerperty Reaueny
CkisFcalke
Nacornlicd
Sunmaiation
Patixtee
Separtal Moocee
Awrip
(laude..7-Sonnettie
DeepSeek-RI
100%
Avg. k= 0.29
80%
80%
Accuracy
Accuracy
60%
60%
40%
40%
20%
0%
0%
5
Warmup K
Position
Figure 3: (Left) Post-warmup accuracy (Accpw-K) results of Claude-3.1 3.7-Sonnet--Tinne (Right) Position-wise accuracy curve and its slope & of DeepSeck-R1.
Overall, these metrics allow us comprehensively assess the learning capability and stability of LLMs, providing deeper insight into their potential to learn from experience within a task.
Finding Learning capability provides new perspective for evaluating models, independent of their static performance, and reveals their underlying learning potential.
show that strong performance parallel solving does not necessarily imply strong learning capability. For example, although DeepSeek-RR -1 outperforms Sluude33.--oomge in the parallel setting, experiences 9% drop the sequential solving paradigm, while Hlaude-3.--oomme achieves 7. 7.2% gain. In the logic reasoning task, Soadd-3.--Sonnee--Tiee DeepSeek-RI or Doubao-15 1.5-Thinking-PPP in the zero-shot setting, yet demonstrates stronger learning capability by substantially improving its performance through experience. 03-mini, on the other hand, exhibits the best performance in this task under both zero-shot and feedback learning settings, showing remarkable learning capability by leveraging feedback to achieve a 14.1 percentage point improvement. Another example DeepSeek-R in the math reasoning task. Although DeepSeek-RI achieves higher zero-shot performance than 03-mini, it fails to learn from experience as effectively as o3-mini and instead suffers a decline in performance.
Results in Table show
does not outperform
Moreover, models with similar static abilities may exhibit markedly different learning capabilities. For instance, Claude-3.7 3.r.dd--.-- Thinking and GPT-4o both achieve 31.7% accuracy in the sequential reasoning task under the zero-shot paradigm, yet their learning capabilities differ significantly: the former improves by 14.5 percentage points, while the latter only improves by 0.5 points. However, GPT-4o demonstrates stronger learning capability than reasoning task. These results indicate that learning capability also varies across different tasks.
in the math
Taken together, these observations suggest that learning capability is an important and distinct metric for model evaluation, independent of static performance and varying across different tasks. This dimension of model assessment deserves further attention and systematic investigation by the research community. In Research Question III ($ 3.4), we further analyze model performance under various learning paradigms, providing additional evidence that learning capability is not solely determined by static ability.
3.3 RQ I: How efficient are LLMs at learning from a sequence problems?
Learning efficiency differs markedly across models and task types. On average, most non-thinking-baxt based models improve more rapidly with experience, while thinking- based models tend to achieve more stable gains.
Finding 4:
The right side of Figure presents the position-wel wise accuracy curve for ReepSeek-R1, with corre- sponding curves for other models shown in Appendix G.7 The heatmap in Figure summarizes the slope (k) of these curves, directly indicating model learning efficiency.
We observe substantial differences in learning efficiency across models. For instance, Claude- 3.7-Sonnet achieves the highest overall learning efficiency, with & = 2.08. On average, most non-thinking-basee based models exhibit steeper slopes in their position-wise accuracy curves compared to

---

mhinking-based models, as shown in Figure (e.g.. Sluude-3.--Soneed DeepSeek- V3, and GPT-4o). This may be because non-thine wwwttthhiinassssee based models often start from a lower baseline, allowing them to quickly capitalize on "low-hanging fruit" as they accumulate experience.
In contrast, learning in thinking-based models is more stable. For example, shown in Figure T6 o3-mini's pisi accuracy curve shows a steady upward trend, even when its initial accuracy already high. Non-thinking-bane models often display larger fluctuations between adjacent positions, such as GPT-4o, DeepSeek-V3, and Doubao- 1.5-Pro. This suggests that thinking- based models can better leverage information from previous problems, including feedback from the judging model, to reason more effectively about subsequent solutions, resulting in more stable performance gains. Moreover, most models display positive learning efficiency in math reasoning tasks. However, sequential reasoning tasks remain particularly challenging, with widespread performance declines; only Sonade...-Sonne---hinee
shows meaningful positive learning on these sequences (A: = 1.89). This indicates that even in reasoning tasks with clear solution paths and feedback, learning efficiency can vary significantly depending on the specific task, highlighting that depends on multiple factors.
Overall, these results suggest that static model capability, task type, and reasoning approach all influence learning efficiency. Non-thinking-basel models tend to improve performance more rapidly through experience, while thinking-basse models achieve more stable improvements. Case studies in Appendix G.I further illustrate how thinking baaseng-based models can utilize prior experience solve new problems.
3.4 RQ III: Do different learning methods lead to differences in performance?
Different solving approaches significantly affect model performance. Models can acquire experience from demonstrations, and feedback further enhances their learning. Moreover, learning capability is not strongly correlated with model's s inherent static ability.
Finding 5:
Table 5 and Figure O present model performance across four solving methods. We observe that, for most models, demonstration learning in the sequential setting generally yields better results than few-shot parallel solving. For instance, Claude-3.7-Soon 7-S7nn-t--tt-- Thinking achieves consistent performance improvements across five tasks compared to the few-shot setting. The key distinction that demon- stration learning allows models to access all previous problems and canonical solutions within the sequence, enabling them learn from prior experience.
feedback learning and demonstration learning in terms of post-warmup accuracy (Accpw-K) and the slope k with additional results provided in Appendix G Results show that, for most models, feedback learning achieves higher average overall accuracy and greater learning efficiency than demonstration learning. For example, DeepSeek-V3 and Doubao- Thinking- Pro show the ability to learn from feedback provided by the judging model and apply this experience subsequent problems across five tasks.
Furthermore, Figures 14 15
Notably, we find that each LLM exhibits strong learning capability on certain tasks, but no model can consistently achieve stable learning improvements across all tasks. Even the state-of-the-art LLMs, as widely recognized in recent studies 777 62 68 30 (8). such as 03-mini and Claude-3.7 3.7-one---- Thinking, do not demonstrate clear advantage in learning capability and efficiency across all tasks. Moreover, for every task, there are always some models that can learn from prior experience and improve their performance. For example, 03-mini exhibits strong learning capability in all tasks except sequential reasoning, whereas 3laude-3.7-Sonne can improve its performance on this task by leveraging experience from feedback.
These results indicate that each model has its own strengths, and that Icarning ability is not strongly correlated with static performance. Learning capability and efficiency thus provide a valuable new perspective for assessing model performance and understanding the gap between current models and human abilities. We also present case study in Appendix G.T to further illustrate how models learn from prior experience.
Finding 6: The average position of the first correct solution Pira and the average offset of the first learned correct solution Potfee vary across models and tasks, providing important insights into model potential. All metrics EvaLearn capture different aspects of a model's 's learning ability and efficiency, collectively revealing the model's learning potential.

---

We analyze the average position of the first correct solution Purs, which measures how quickly models achieve initial success within sequence. The right side of with results for other models provided in Appendix G.4 We observe that 03-mini and Doubao-1.5, Thinking- Pro achieve the best performance, with average positions of 1.57 and 1.80, respectively. These two models also achieve the highest overall accuracy in the feedback learning setting. Although Pirt is closely related to static model capability (since stronger models are more likely to solve problems earlier), notable exceptions exist. For example, DeepSeek-V V3 achieves the same firtt value as 03-mini on the mathematical reasoning task, yet its overall accuracy is 5.9% lower, indicating that o3-mini can better improve its problem-soke solviig ability through experience.
Pirtt for o3-mini,
We also find substantial variation in model performance across different tasks. For example, most models struggle to solve their first problem early in logical reasoning sequences, with only o3- mini and Doubao- Doubao-1.5-Thinking 1.5-Thinkin--Pre consistently achieving early success. Additionally, comparing demonstration learning and feedback learning settings, as shown in Figures] 11 and [T] seven out of nine models can solve a problem earlier when provided with feedback from the judging model. This indicates that, compared to simply providing canonical answers, feedback is more effective in facilitating model learning and mastery sometimes fails to solve any problems in sequence when using only demonstrations, but with feedback learning, it can solve at least one problem in every sequence.
task. For instance, in the logical reasoning task, 03-mini
Beyond Purst we also analyze the average offset of the first learned correct solution Pottee. as shown in Table 4 On average, frontier LLMs attain their first learned success relatively late in typicalllyy around three to four additional positions after the first zero-shot miss, such as DeepSeek-Ri and DeepSeek- V3. By contrast, o3-mini tends to reach this point earlier on average. Clear task effects emerge. Models generally learn earlier on classification and extraction than on logical reasoning. For models such as GPT-4o and Qwen2.5-32b-Instree the first learned correct solution typically appears earlier in classification and extraction, whereas logical reasoning usually requires more steps. Summarization displays substantial variability across models-early for 03-mini but much later for Doubao- 1.5- -ww.----iii--------- Pro and DeepSeek-RI.
@wen2.5-32b-Instrrue
The long bfain-offthouue reasoning paradigm provides a slight efficiency gain within families, For example. Claude -Jaude-3.7-Somee -o.7--tnt---TTe Thinking attains a lower average Potteet than slightly improves over Doubao- 1.5-Pro, while the DeepSeek pair mixed. Across families, the coouusee-ocuued 03-mini also tends to learn earlier than GPT-4o. We find that Pattee highlights systematic task effects and modest benefits from long-form reasoning, while revealing substantial cross-model variability in true test-time learning efficiency. Consistent with / Pirst. Poftse varies widely across models and tasks and does not align with overall accuracy, underscoring that EvaLeam captures dynamic learning dimension that is distinct from static capability.
and
1.5hhk--.5-Thhnnking
Overall, all metries comprehensively assess the learning performance from multiple perspectives. EvaLearn evaluates models in more realistic and dynamic way, helping researchers better understand the gap between models and humans, and driving the development of more powerful models.
4 Conclusion
We present EvaLearn, novel benchmark that sequentially evaluates the learning capability and efficiency of models within specific tasks. EvaLearn is equipped with a suite of comprehensive metrics, revealing significant performance differences among frontier models across diverse tasks, including both thinking- models can effectively leverage teacher model feedback previous solutions to enhance learning, others struggle to benefit from such feedback. EvaLearn offers a new perspective on assessing the potential of LLMs and serves as a pioneering step toward dynamic evaluation. The limitations and future work of EvaLearn are discussed in Appendix A and related work is presented in Appendix B
and non-thinking--an based models. Moreover, we find that while some
Acknowledgments
We gratefully acknowledge the significant contributions made by the annotation teams at ByteDance, whose diligent work was essential to the success of this paper. The core members of the annotation team include Di Cheng, Linhua Deng, Yanxi Fu, Yafei Qiao, Chaoqian Ren, Mei Su, Ying Wu, Baitong Yang, and Xingyu Zhu. We wish to express our sincere appreciation to undisclosed
10

---

third-part annotation company for their substantial support in data annotation. We also want to thank all individuals who participated in and supported this project for their valuable input. Finally, we wish to thank the AC and anonymous reviewers for their constructive comments. The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by the Science and Technology Commission of Shanghai Municipality (No. 2451 100 10), National Natural Science Foundation of China 16.66,,6 62441602. 62206057). Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai 1403500.
References
[1] Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930-769600 2024.
[2] Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms. In International Conference on Machine Learning, pages 787-812. PMLR, 2024.
[3] Anthropic. Claude 3,5 sonnet, 2025.
URL https://ww..nthrppc..oo//ess
[4] Anthropic. Claude 3.7 sonnet and claude code, 2025, URL http:://ww.arthrrpi.cc.
[5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv: 077.07 2021.
[6] Danielle S Bassett, Nicholas F Wymbs, Mason A Porter, Peter Mucha, Jean M Carlson, and Scott T Grafton. Dynamic reconfiguration of human brain networks during learning. Proceedings of the National Academy of Sciences, 108(18)77441-7646 2011.
[7] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are shw- learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[9] ByteDance.
Doubao-1.5-pro, 2025. URL
http://aaa..dbbooocoo www.ddoaao.ooo////eeeeiaaaaaaaae
foubao_1_5_pr
[10] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, al. A survey evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1-45, 2024.
[11] Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang. Tiejun Zhao, and Min Zhang. Dual-reflect: Enhancing large language models for reflective translation through dual learning feedback mechanisms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 693-704, 2024.
[12] Junzhe Chen, Xuming Hu, Shuodi Liu, Shiyu Huang. Wei-Wer Tu, Zhaofeng He, and Lijie Wen. LImarena: Assessing capabilities of large language models in dynamic multi-agent environments. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1 13055-13077, 2024.
[13] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating 2021.
large language models trained on code. arXiv preprint
11

---

[14] Wentong Chen. Yankai Lin. ZhenHao Zhou, Hong Yun Huang, Yantao Jia, Zhao Cao, and Ji-Rong Wen. Icleval: Evaluating in-context learning ability of large language models. In Proceedings of the 3 International Conference on Computational Linguistics, pages 10398- 10422, 2025.
[15] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference Learning Representations, 2024. URI. htt.hhttts//opppeeiiieeee wwwpeeewwwwweeeeeeeeeeeeeeeee
[16] Wei-Lin Chiang, Zhuohan Li. Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2()):,, 2023.
[17] François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.0154,, 2019.
[18] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv: 14168, 2021.
[19] Julian FornoF Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang, and Eric Schulz. Meta-in-context learning in large language models. Advances in Neural Information Processing Systems, 36:651 89-65201, 2023.
[20] Ian J Deary, Lars Penke, and Wendy Johnson. differences. Nature reviews neuroscience, 11(3):201-211, 2010.
The neuroscience of human intelligence
[21] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, al. survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107-1128, 2024.
[22] Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2368- 2378. Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.1 N0.18653/v1M 1/1/3//11/1111111 1246. URL https:/:::/// URU/ttts://acllanthoo..o http://aaaaanttogg...o.oo http:p:://.laaattooooogoorr///// www.i/11g...///-0--/-1 1246/
[23] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Ji- awei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. tion for large language models: A survey. ArXiv, abs/2312.10997,, 2023. URL https
genera-
//api. wew.aticcchccchllarr.. wwtpiholllrrrog/rrroooeeeeee
[24] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. Advances in Neural Information 2022.
Processing Systems,
[25] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15544 2024.
[26] Daya Guo. Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. /294. 2025.
[27] Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. Evaluating large language models: A comprehensive survey, arXiv preprint 2xXiv:2310.19736 2023.
[28] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, 2 edition, 2009.
12

---

[29] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL BtRLLttts///opeereeeie httpe//rwwieeeeeeeeeeeeeeeeeeeee net/ffrunl
mm=37
[30] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openzreview wet/:foorurrdddd7Byywwta..
[31] Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: evaluating lan- guage agents on machine learning experimentation, In Proceedings of the 41st International Conference on Machine Learning, pages 12230999,,, 2024.
[32] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-disciplin chinese evaluation suite for foundation models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
URL wtt.ttttts///pereeeeeeee
net/ wdw.mmdddddrmmmraa
[33] Yuxin Jiang, Yufei Wang, Xingshan Zeng. Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. FollowBence A multi-level fine-grained constraints following benchmark for large language models. In Lei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguisties (Volume 1: Long Papers), pages 4667-4688. Bangkok. Thailand, August 2024. Association for Computational Linguistics. /RLnttps:/acllattoollg or2/2024 acl- del-long.44 257
[34] Yimin Jing, Renren Jin, Jiahao Hu, Huishi Qiu, Xiaohua Wang, Peng Wang, and Deyi Xiong. Followeval: A multi-dimensions benchmark for assessing the instruction-followin capability of large language models. arXiv preprint arXiv:231 1.09829, 2023.
[35] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 881-905, 2024.
[36] Brenden M Lake, Tomer Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[37] Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao. Criticeval: Evaluating large-scale language model as critic. Advances in Neural Information Processing Systems, 17766907-669666 2024.
[38] Pat Langley, John E. Laird, and Seth Rogers. Cognitive architectures: Research issues and challenges. Cognitive Systems Research, 10(2):141-160, 2009. ISSN 1389-0417. doi: 10. 1w.11166...ooo. www..ss....0000.0000
[39] Md Tahmid Rahman Laskar, Sawsan Algahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, et al. systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations. In Proceedings of the 2024 Conference on 2024.
Empirical Methods in Natural Language Processing, pages
[40] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. / 1939, 2024.
[41] Yu Li, Josh Arnold, Feifan Yan, Weiyan Shi, and Zhou Yu. Legoeval: An open-source toolkit for dialogue system evaluation via crowdsourcing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I1th International Joint Conference Natural Language Processing: System Demonstrations, pages 317-324, 2021.
13

---

[42] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee. Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https
/ //openreviee.vo wtt.:///ooruummdddd8......EE
[43] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falschoods. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 3214-3252, 2022.
[44] Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. Critichench: Benchmarking Ilms for critique-correct reasoning. In Findings of the Association for Compu- tational Linguistics ACL 2024, pages 1552-1587, 2024.
[45] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint wrwiv:2412.19909000 19437, 2024.
[46] Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. Longgenbench: Long-context generation benehmark. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages
2024.
[47] Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. Alignbench: Benchmarking chinese aligmment of large language models. In ACL (1), pages 11621-11640, 2024. URL org/ 2024. acl-long.603 624
[48]
Xiao Liu, Hao Yu, Hanchen Zhang. Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. In The Twelffh International Conference on Learning Representations, 2024. URL htlLhhtps::/// //op://////eeeeieee/ wtwpa/rrviiieeneeforuuee. id=zAdUBOaCTQ
[49] Xinxi Lyu, Sewon Min, Beltagy, Luke Zettlemoyer and Hannaneh Hajishirzi. Z-icl: Zero- shot in-context learning with pseudo-demonstraie In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 2304-233 2317, 2023.
[50] Kaijing Ma, Xeron Du, Yunran Wang, Haoran Zhang, ZhoufutuWen, Xingwei Qu, Jian Yang, Jiaheng Liu, minghao liu, Xiang Yue, Wenhao Huang, and Ge Zhang. KOR-bench: Benchmarking language models on knowleage-orthogonnal reasoning tasks. In The Thirteenth International Conference Learning Representations, 2025. URL https: httpt:://oppeeeeiiiiie //pp/revie/eee bet//forumidd-SSRRRR
[51] Aman Madaan, Niket Tandon, Prakhar Gupta. Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 46594, 2023.
[52] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: benchmark for general Al assistants. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https://openrevieew wwt.:eeenfffieiccm fibxvahvs3
[53] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048-11064 Abu Dhabi, United Arab Emirates, December 2022. As- sociation for Computational Linguistics. doi: 10. 2o.18853/v1/022..oo http:/22...mmmmmiaana.mm.mm 759. URL
.rg 2022.02222222222 emnlp main. 759/
14

---

[54] Deepak Nathani, Lovish Madaan, Nicholas Roberts. Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, al. Mlgym: A new framework and benchmark for advancing ai research agents. arXiv preprint arXiv:2502. 14499, 2025.
[55] OpenAL. GPT4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[56] OpenAl. Learning reason with 1lms, 2024.
URL http::://ppenai.com/indd
learning wresonnwwthh-1e
[57]
OpenAL. Openai o3-mini, 2025. URL http:ppse/ppppaaacccc////a//pppp
[58] OpenAL. Introducing openai o3 and o4-mini, 2025. URL http::://peeai..com///// introducing--3- and--o-nn---ii//
[59] Long Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 2022.
[60] Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph www.sticscooooooa.......o. URL httpts//tpppppi
generation: A survey. ArXiv, abs/2408.0892// 2024. www.oollarrrrrrooooooeeo hetg/oorpuII::::00 hthpattps///pp.s.mmtiicccoola
271103110
[61] Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao. Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 13025-13048, 2024.
[62] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google- proof q&a benchmark. In First Conference Language Modeling, 2024. URL https //openreview mate/foouumideee id-71175846888
[63] Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback. arXiv preprint urXiv:2204.14144 2022.
[64] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36: 68539-68551, 2023.
[65] ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang. Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al. Seed-thinking-vl. 5: Advancing superb reasoning models with reinforcement learning. arXiv preprint 2rXiv:2504.139/1 2025.
[66] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8866 8652, 2023.
[67] Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 36:27801-27819, 2023.
[68] Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwatt Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei- Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. Global mmlu: Understanding and addressing cultural and linguistic biases multilingual evaluation, 2024. URL https:://// /arxivivv.. waa.iv..rrg/bb///111.1 03304
15

---

[69] Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed- Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. Multichallenge: realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501. 2025.
[70] Zayne Rea Sprague, Xi Ye. Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In The Tweifth International Conference Learning Representations, 2024. id jenyY
www.eewweeefoooe net/forum?
[71] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alos Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. TRANSACTIONS ON MACHINE LEARNING RESEARCH, 2022.
[72] Yi Su, Yunpeng Tai, Yixin Ji, Juntao Li, Yan Bowen, and Min Zhang. Demonstration aug- mentation for zero-shot in-context learning. In Findings of the Association for Computational Linguistics ACL 2024, pages 14232-14244, 2024.
[73] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery. Quoc Lc. Ed Chi, Denny Zhou, and Jason Wei. Chal- lenging BIG-bench tasks and whether chain-of- thouhht can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Com- putational Linguistics: ACL 2023, pages 13003- sociation for Computational Linguistics. doi: 10. 18653/11/2022 1/w../.innnnnss---.... attps://aclantholl. org 2023 www.iiinnnggannn acl. 824/
Toronto, Canada, July 2023. As-
URL
[74] Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et al. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-leve code. arXiv preprint arXiv:231 1..0935,, 2023.
[75] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https:///wenlla github. io/blog/quq-300 32b/
[76] Caitlin Tenison, Jon M. Fincham, and John R. Anderson. Phases of learning: How skill acquisition impacts cogmitive processing. Cognitive Psychology, 87:1-28, June 2016. ISSN 1095-5623. doi: 10. 10116//coggpyych2.16...00000
[77] Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL wttph:tpp::///t //wp:../wwwwww..m www.k//ggll/....le..e.. elg com/
Bat.aaets/hemmistterrrroo////// wwwpppooooooo..o..... set- 1983 2024
[78] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
[79] Jane X Wang. Meta-learning in natural and artificial intelligence. Current Opinion in Behav- ioral Sciences, 38:90-99 2021.
[80] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and chal- lenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL wwwp:/otttttoooood/o/o/ooaoa
[81] Yutong Wang, Jiali Zeng, Xuebo Liu, Fandong Meng, Jie Zhou, and Min Zhang. Taste: Teaching large language models to translate through self-reflection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1144-6158. 2024.
16

---

[82] Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, al. Benchmarking complex caaruction-follownn with multiple constraints composition. Advances in Neural Information Processing Systems, 37: 137610-137645, 2024.
[83] Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36:36637-3366 2023.
[84] Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang. Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprint arXiv:2406.0411 2024.
[85] Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang. Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. FOFO: A benchmark to evaluate LLMs' format-following capability. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: URL htt.:htsss/acllaantoooooooo org. 2024 a02. acl-long. 40/|
10. 10.18653/v 118855//1200222 1/653//1/202..allloong
[86] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language mod- els to follow complex instructions. In The Twelfth International Conference Learning Representations, 2024. URL B
[87] Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang. Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking Ilm agents on consequential real world tasks. arXiv preprini arXiv:2412.1416. 2024.
[88] Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. Sayself: Teaching ilms to express confidence with self-reflective rationales. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pages 3985-5998, 2024.
[89] Hanqi Yan, Qinglin Zhu, Xinyu Wang. Lin Gui, and Yulan He. Mirror: self-reflection method for knowledge-rich reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7086-7103, 2024.
[90] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu,, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui. Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
[91] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406. 12045, 2024.
[92] Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pages PMLR, 2023.
[93] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[94] Ming Zhang, Yue Zhang, Shichun Liu. Haipeng Yuan, Junzhe Wang, Yurui Dong, Jingyi Deng. Tao Gui, Qi Zhang, and Xuanjing Huang. Llmeval-2, 2023.
[95] Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang. Yue Zhang, Junzhe Wang, Shichun Liu, Shihan Dou, Huayu Sha, Qiyuan Peng, al. Llmeval-3: A large-scale longitudinal study on robust and fair evaluation of large language models. arXiv preprint 200i::22000005552... 2025.
17

---

[96] Xinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang, and Yongbin Li. lopo: Empowering llms with complex instruction following via input-output preference optimization. arXiv preprint arXiv:241 1.06208, 2024.
[97] Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. Limeval: A preliminary study on how to evaluate large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 19615-19622, 2024.
[98] Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. Agent-safetybench: Evaluating the safety of Ilm agents. arXiv preprini arXiv:2412. 14470, 2024.
[99] Henry Hengyuan Zhao, Wenqi Pei, Yifei Tao, Haiyang Mei, and Mike Zheng Shou. Inter- feedback: Unveiling interactive intelligence of large multimodal models via human feedback. arXiv preprint 1rXi:2252.15022 2025.
[100] Jun Zhao, Jingqi Tong, Yurong Mou, Ming Zhang, Qi Zhang, and Xuan-Jing Huang. Exploring the compositional deficiency large language models in mathematical reasoning through trap problems. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 16361-16376, 2024.
[101] Siyun Zhao, Yuqing Yang. Zilong Wang, Zhiyuan He, Luna K. Qiu, and Lili Qiu. Retrieval augmented generation (rag) and beyond: A comprehensive survey on how to make your 1lms use external data more wisely. ArXiv, Abs/2409.144244, 2024. URL https://api
semanticacholar. wrwpaaarrrr/ooooottttra/. 27:2827775
[102] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36:46595- 46623, 2023.
[103] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2024.
[104] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. issructioo-foolowingg evaluation for large language models. arXiv preprint arXiv:2311.079111 2023.
[105] Shuyan Zhou, Frank F. Xu. Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. httltttts///ppenni www.rrvvviieenee/fruu.r id=oKn9c6yLxx
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clarified our claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Justification: We have discussed the limitations and future work in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
In this work, we propose a novel benchmark that does not include theoretical
assumptions or proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have shown our experiment results in Section 3. We have also fully disclosed all experimental setups in Section and Appendix. We also open-source all datasets, the automatic evaluation framework, and the results studied in this paper in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We open-source all datasets, the automatic evaluation framework, and the results studied in this paper in the supplemental material. We have also included a full explanation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ltliiss.cc/publlcguuudeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In this work, we propose novel benchmark that does not require training models. We have included all experimental setups Section 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All models were evaluated via their respective APIs, and their implementations are provided in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In this work, we propose novel benchmark that does not require training models. We report the API costs in Appendix D.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't iidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. http:////eeaaupp.e.ecpp///c//caa inn
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive socictal impacts and negative socictal impacts of the work in Appendix A.
Guidelines:
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks. All the data in EvaLearn is used to measure the learning capability of LLMs, which does not contain any biased information.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original papers that produced the datasets used in our paper. We have included the version information and corresponding licenses in Appendix B. We have included the licenses for our assets the supplementary materials.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
23

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.paperssii paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included detailed descriptions and usage of the assets in the supple- mentary materials.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: In our work, all annotation was performed by company staff with relevant professional expertise, who were compensated with standard salaries.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
24

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only use LLM to help us polish our manuscript and make more readable. It does not impact the core methodology, scientific rigorousness, and originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standare components.
Please refer to our LLM policy (https://neure) wwt/oc////ooooooocccoooo 200t://eeccccoaeeeeee/////////// for what should or should be described.
25

---

A Limitations and Future Work
EvaLearn is the first benchmark to quantitatively evaluate the rapid learning capabilities of LLMs on single task. We discuss its limitations, the new insights it offers for the community, and the directions it opens for future research into the causes of learning differences between models and the development of learning methods.
Towards including highly challenging open-ended problems. EvaLearn focuses on problems that can be reliably evaluated with well-defined rubrics, ensuring trustworthy automated assessment. However, during the annotation process, we found that some highly challenging open-ended prob- lems-where systematic instance-level rubrics are difficult to construct- are not yet included, even though such problems can be valuable for human learning and reasoning. In the future, we aim to develop new evaluation strategies that can effectively assess model responses to open-ended tasks, further expanding the task coverage of EvaLearn.
Towards benchmarking more open-source LLMs. We benchmark current frontier LLMs and reveal significant differences in their task learning performance. While these advanced models substantially outperform existing open spen-source models. there is still limited understanding of how open- open-source models compare in terms of task learning ability. In future work, we plan to extend our evaluation to a broader range of open-source models, aiming to better quantify this gap and foster progress within the open-source L.LM community.
Exploring the roots of learning capability and efficiency differences. Our analysis quantifies the learning capability and efficiency of the models, and reveals that these are not strongly correlated with the models' static abilities. The fundamental factors driving differences in these aspects across models warrant deeper investigation. In future work, we plan conduct more detailed studies to uncover the underlying causes of these disparities. We hope that EvaLearn will inspire the community to further explore and understand the mechanisms underlying LLM learning capability and efficiency.
Broadening the exploration of learning methods. EvaLearn has investigated the impact of two learning methods on both learning capability and efficiency. However, LLMs may benefit from a broader range of strategies, such as integration with retrieval-augmmente generation (RAG) 23 101 60). In future work, will leverage EvaLearn systematically analyze the effectiveness of additional learning methods in task learning scenarios. We also hope that EvaLearn will encourage the community to innovate and develop more effective approaches for rapid adaptation new tasks.
B Related Work
Evaluation Perspective and Paradigm. Previous work evaluating LLMs has mainly focused on
evaluating their static capabilities 10
39
27
95 such as factual knowledge 62
29
32
97
94
80
68
reasoning (8 100,
78
37
44
103
70
50 42 instruction following 96
104
82
34
86
16], and code generation [3 5). Current evaluation paradigms can be broadly categorized
into two types: parallel evaluation and interactive evaluation (22] 73 69
64 17 41 91). The vast
majority of existing benchmarks adopt a parallel evaluation approach. This evaluation method only measures the static capabilities of models, providing little insight into their capability to learn or
adapt dynamically over time [93] 71, 43
33 85 46
99]. On the other hand, interactive
[05 12 98 3T 87 B5). This evaluation is commonly used in multi-ture turn behavior assessments [102]
paradigm emphasizes the dynamic evaluation of model's 's ability to engage in multi-turn interactions, where tasks themselves remain i.i.d., but actions or states are temporally dependent. Interactive evaluation more closely reflects real-world scenarios [48] 54 [2). Our work also falls under the category of dynamic evaluation. However, unlike these methods, EvaLearn requires models to sequentially solve a series of problems within the same task, evaluating learning capability and efficiency, providing a direct indication of the model's learning potential. fills a critical void in the current evaluation landscape.
Learning Paradigms. We compare two learning paradigms- learning from demonstrations g 53
21 and learning from feedback [66] in terms of their ability to stimulate model learning. Broadly, the former enables models to learn output formats and problem-solving strategies from demonstrations
49 72 24
67 83 92 14 2 while the latter allows models to leverage experience gained
from previous problems to improve subsequent performance [55 88 8T
89 51 [1] Both paradigms
can be incorporated into model training to enhance overall capability
|59 63 In this paper, we apply
26

---

both approaches at inference time, requiring models to sequentially solve a series of problems to assess whether they can benefit from prior canonical solutions or experience when tackling additional problems. Importantly, the evaluation metrics in Evaleeam are decoupled from the choice of learning method, ensuring the flexibility and general applicability.
C Broader Impact
In this work, we propose a pioneering benchmark, EvaLearn, which is designed to dynamically assess the learning ability and efficiency of I.LMs. We hope EvaLLeary provides a new evaluation perspective for assessing LLM potential and promotes the development of deeper and more dynamic evaluation approaches. All the data in Eval_ear is used to measure the learning capability of LLMs, which does not contain any biased information. We do not see any negative societal impacts of this work.
D License For Artifacts and Data Consent
All models used in this work, whether open-source or closed-source, are permitted for academic research. The licenses for DeepSeek-V. V3 and DeepSeek-R1 are MIT, while the license for Qwen2.5- 32B-Instruct Apache 2.0.
E Data Annotation Process
We adopt hybrid approach combining human expertise and advanced LLMs for data annotation. The annotation process consists of three main stages.
Problem Collection, We begin with collection of questions sourced from our model API platform, which are anonymized to remove any personally identifiable information identification markers. Four annotators independently and carefully review each question, identifying those for which humans could potentially improve their performance through repeated practice on similar problems. Only questions unanimously judged as "learnable" by all annotators are retained as seed problems. These seed problems are then categorized into six task categories. Next, we use Claude-3.7 4 to analyze each problem, identifying the required skill dimensions and suggesting potential directions for increasing problem complexity. With the aid of these model-generated strategies, nine annotators write additional similar and challenging problems for each seed question, expanding datasets with diverse variants.
Problem filtering and canonical answer annotation. To further increase the difficulty of the dataset, we leverage advanced LLMs screen the seed problems. Specifically, for each question, randomly select three LLMs from a model list (as shown in Table to generate candidate answers. This approach helps avoid bias toward against any specific model. Seven annotators then review these answers to write the correct canonical answer for each question, and assess the correctness of each model-generated response, providing explanations for their judgments. To ensure high data quality, an additional three annotators double-check both the written canonical answers and the explanations for the correctness judgments of the model responses. We retain only those challenging questions for which all three models fail to provide correct answer, thereby maintaining high level of difficulty among the seed problems.
All annotators involved in this work were fairly compensated in accordance with the labor standards of their respective countries.
Instance-level rubric annotation and validation. Eight annotators are responsible for writing rubrics for each retained challenging problem, which serve criteria for evaluating the correctness of model responses. Annotators first determine whether each problem has a unique correct answer. For problems with unique answer, the rubric is constructed answer. For problems with multiple valid answers, annotators write rubrics that comprehensively cover all acceptable solutions, clearly specifying the required conditions for correctness. Overly open-ended problems that are difficult to evaluate (e.g.. brainstorming tasks) are excluded from the dataset.
concise summary of the canonical
Fach rubric then validated by two annotators. Specifically, for every problem, three models are randomly selected from the model list to generate three candidate answers. Annotators assess whether
27

---

the rubric can accurately evaluate all responses and provide explanations. Rubrics that fail this validation step are revised and re-evaluated. An additional three annotators conduct final quality check on all fields, including the canonical answers, the explanations for the correctness judgments of the model responses, and the rubries. Finally, we construct problem sequences from the retained set of challenging problems. Each sequence consists of problems from the same task category, and the order of problems within each sequence randomized.
Table 3: The model list used for ensuring the difficulty of datasets and validating the effectiveness of instance-leved rubrics.
Model
Published date
Claude-3.7 3.nde--37-Sonet
February 25, 2025
Claude-3.77 77-Sonnet-Thinkeee
February 25, 2025
3.5uee-.5-Sonett
June 21, 2024
DeepSeck- V3
45
December 26, 2024
DeepSeek- R1
26
January 15, 2025
Doubao- 1.5-rr-32k 9
May 9, 2025
Doubao-1.5-Thinkee 1.t.:-..i--hiiiigg-P--a
65
April 17, 2025
Doubao-1.5-PPr-22 9
May 9, 2025
Qwen2.5-77-Instruct 990
September 19, 2024
Qwen2.5-32b-Instruuc 900
September 19, 2024
QwQ-32b [75]
March 6, 2025
OpenAI-o3-mini [57
January 31, 2025
OpenAI-ol [56]
September 12, 2024
OpenAl-o4-mini 558
April 16, 2025
F List of Evaluation Models
In this work, we comprehensively evaluate the learning capability and efficiency of nine state-of-the. art LLMs. These LLMs contain:
(1) Claude-3.7-Sonnet [4] is the most intelligent model released by Anthropic to date, particularly in terms of coding capabilities, which is upgraded version of Claude 3.5 Sonnet.
the extended thinking mode of Claude 3.eee3..-SSnneee which performs self-reflection before answering in order to improve performance on tasks such as math, physics, instruction-followingg coding, and many others.
(2) Shaude-37-Sonne--Thinkee
(3) DeepSeek-V2 V3 45 is a strong Mixture-of- -xpeerttt- (MoE) language model with 671B total parameters with 37B activated for each token.
(4) DeepSeek-Ri [26] is the first-generation reasoning model developed by DeepSeek AI, which incorporates stati training and cold-start data before reinforcement learning, enabling it to demonstrate strong reasoning capabilities.
(5) Doubao-1.5-PP 99 is nigh-performance sparse MoE large language model that focuses on achieving an optimal balance between inference performance and model capability developed by the Doubao team.
(6) Baubaao-.5-Thinkiing
shares the same model base Seed- Seed-Thinking-vl.. which an 665
advancing superb reasoning model with reinforcement learning. It is a MoE model with a relatively small size, featuring 20B activated and 200B total parameters. It is capable of reasoning through thinking before responding, resulting in improved performance on a wide range of benchmarks.
(7) Swee2.55-32-IInstrue (90] is the instruction-tunel 32B version of the Qwen2.5 series, which is an improved iteration based on the Qwen2 family.
(8) GPT-4o 55 is a state-of-the-art LLM developed by OpenAI, which demonstrates strong perfor- mance across wide range tasks, including natural language understanding, complex reasoning, instruction following, code generation, and problem solving. It achieves competitive results
28

---

Table Results of Potiset. Full task names for the abbreviations can be found in Section 2.1 colored cells indicate the best performance for each task category.
Blue-
Model
Sum
Ex
LR
MR
SR
Overall
Non-thinkiig-bbase
DeepSeek-V3 V3
3.82
4.0
3.88
5.27
3.27
4.00
4.04
Claude-3.7-Sond
4.08
3.08
3.71
5.69
3.24
1.94
3.62
GPT-4o
4.08
3.45
3.24
2.41
3.29
3.75
Doubao-1.55 1.5-Pro
3.54
3.58
3.71
6.75
3.24
3.06
3.98
Qwen2.5-32b-4 Inntttrtt
3.93
3.69
3.35
6.38
3.65
3.53
4.09
Thinking-
Whaseing-based based
OpenAl-o3-ming
3.42
3.71
3.62
3.28
Doubuo- 1.5- Thinking--iie
4.9
3.83
2.88
3.44
3.67
4.81
3.92
DeepSeek-R R1
4.7
4.5
2.76
4.7
3.62
4.19
4.08
Claude-3.7 1--7---nt------ Sonnet-Thinking Thinking
3.8
2.77
3.18
5.55
3.41
2.53
3.54
Gemini-2.. 2.5-Pro
4.31
3.47
3.66
2.71
3.46
3.44
Gemini 1omini-22.5-a 2.5-Flash
4.08
3.0
2.76
6.55
3.4
3.18
3.83
challenging benchmarks in mathematics, logic, and programming, and maintains high performance across diverse evaluation settings.
(9) OpenA - [7 is cost-efficient model in the reasoning series, previewed in December 2024, which pushes the boundaries what small models can achieve. It delivers exceptional STEM capabilities- with particular strengths in science, math, and coding-while maintaining the low cost and reduced latency of OpenAl-ol -mini.
All models are evaluated via their respective APIs For won-thinking--aa models, the temperature set to 0.2, while thinking-basee based models do not support temperature adjustment. Moreover, both input and output lengths are set to the maximum supported by each model.
The total cost for running all experiments in study approximately $4,357.14, which includes the evaluation of all models across all tasks and learning paradigms on the entire benchmark.
G A Addtional Experimental Results
G.1 Case Study: Do LLMs truly utilize prior judgments solve problems?
Case Study 1: Comparison with Parallel Solving (Zero-shot) and Sequence Solving (Feedback Learning)
Figure illustrates how model performance differs between the parallel solving (zero-shot) and sequence solving (feedback learning) paradigms in a sequential reasoning task involving maze navigation. the zero-shot setting (Response A), Claude-3.7-Sonnet makes a critical directional enror in the final segment, incorrectly assuming the direction after turn and arriving at the wrong coordinate (7, 4). This highlights the challenge of maintaining spatial orientation without feedback.
In contrast, under the feedback learning paradigm (Response B), the same model correctly tracks all movements and reaches the accurate exit coordinate 12). The model explicitly defines directional conventions at the outset ("Let me define directions first...") and methodically tracks orientation after each turn, demonstrating clear evidence of learning from previous feedback on similar problems.
Case Study 2: Evidence of learning from prior experience
Figure 5 demonstrates how models utilize prior experience in sequential reasoning tasks. In this case, DeepSeek-PP RI's thinking process while solving a 4x4 sliding puzzle explicitly references knowledge gained from previous problems.
29

---

Table Comparison of overall accuracy across four solving methods, including two parallel methods (i.e., zero-shot and few-shot) and two sequential methods (i.e., demonstration learning and feedback learning). Sum denotes the summarization task. Cla denotes the classification task. Ex denotes the extraction task. LR denotes the logical reasoning task. MR denotes the mathematical reasoning task.
SR denotes the sequential reasoning task. All values are shown as %.
Model
Paradigm
Cla
Ex
LR
MR
SR
Sum
Overall
Non-thinkkingbbssed
Zero-shot
70.8
45.0
10.3
61.7
41.7
71.7
31.3
Few-shot
60.0
8.4
54.4
31.6
63.2
25.7
Doubao-1.5. 1.5-Pro
Demonstration Learning
8.8
63.9
28.6
71.4
29.0
Feedback Learning
10.8
71.4
36.1
71.4
31.2
Zero-shot
72.9
45.0
25.6
71.7
60.0
81.7
43.5
Few-shot
75.6
40.4
23.0
61.4
57.9
78.9
40.0
DeepSeek-V
Demonstration Learning
70.3
40.3
17.6
68.9
51.3
74.1
36.4
Feedback Learning
74.7
36.1
24.9
74.8
54.6
76.8
41.5
Zero-shot
64.6
48.3
48.3
33.3
76.7
28.4
Few-shot
80.0
45.6
10.4
56.1
43.9
70.2
31.1
Claude-3.7-Some
Demonstration Learning
64.8
42,9
10.8
61.3
43.7
75.0
31.1
Feedback Learning
75.8
46.2
15.5
63.9
49.6
75.0
35.6
Zero-shot
60.4
31.7
6.9
46.7
21.7
66.7
23.8
Few-shot
60.0
31.6
6.4
38.6
24.6
66.7
22.5
Qwen2.5-32b-Inm
Demonstration Learning
69.2
34.5
7.4
43.7
21.8
64.3
24.1
Feedback Learning
62.6
30.3
9.8
49.6
30.3
59.8
25.5
Zero-shot
70.8
45.0
15.6
43.3
31.7
81.7
32.6
Few-shot
77.8
40.4
15.4
54.4
40.4
70.2
32.9
GPT-4o
Demonstration Learning
71.4
35.3
8.5
56.3
34.5
75.0
28.3
Feedhack Learning
73.6
48.7
12.9
61.3
32.2
78.6
32.7
Thinking-based
Zero-shot
89.6
48.3
41.7
78.3
66.7
86.7
55.7
Few-shot
75.6
49.1
23.5
73.7
54.4
78.9
41.9
DeepSeck-R1
Demonstration Learning
82.4
31.4
79.8
52.9
88.4
48.2
Feedback Learning
79.1
29.6
74.8
51.3
89.3
46.4
Zero-shot
66.7
43.3
12.5
46.7
31.7
86.7
31.2
Few-shot
60.0
42.1
13.4
43.9
43.9
77.2
30.6
Claude-3.7 Claude3..-Soonett Thinking
Demonstration Learning
74.7
48.7
16.1
59.7
42.0
77.7
35.2
Feedback Learning
80.2
48.7
18.8
58.0
46.2
78.6
37.4
Zero-shot
64.6
45.8
73.3
73.3
65.0
$4.3
Few-shot
80.0
50.9
50.7
75.4
84,2
73.7
60.0
OpenAl-o3-mini
Demonstration Learning
73.6
51.3
53.1
78.2
73.9
67.9
60.0
Feedhack Learning
73.6
47.1
59.9
80.7
78.2
75.9
64.8
Zero-shot
79.2
55.0
42.5
85.0
73.3
85.0
57.1
Few-shot
82.2
49.1
38.9
71.9
64.9
73.7
51.4
Doubao-La 1.5- Thinking-PiP -Pr
Demonstration Learning
73.6
52.1
42.0
77.3
64.7
83.9
54.3
Feedback Learning
70.3
52.9
39.4
77.3
55.5
82.1
51.6
30

---

The highlighted portions of the thinking process show: "After struggling, I think the correct sequence should be similar the user's previous solution, involving multiple 'Up' moves followed by 'Right' and "Down'." ." The model further reasons: "Thhe correct answer might involve series of 'Up" moves to position "M" at the top, then moving other numbers into place."
Ultimately, the model applies this learned pattern to formulate an 11-step solution: ["Up". "Up", "Up". "Right", "Down", "Down", "Down", "Right", "Up". "Up". "Up"], well within the 12-move limit.
This case illustrates how models extract patterns from prior experience and apply them to new related problems, demonstrating genuine experiential learning capability.
The case used for Case Study 1
Task Type: Sequential Reasoning
Model: (ouude-3--soonee-22255555
Question: You enter 12 x 12 maze. The coordinates this maze are represented by (x,, v). where æ is horizontal (left and right directions) and y is vertical (up and down directions). The bottom left square is at coordinates (1, 1) and the top right square is at coordinates (12, 12). You enter the maze at coordinates (3, 1). Following are the movements you take to reach the exit, where you will give the coordinates after each movement, the final exit point, which can be anywhere on any coordinate of the maze:
well as the coordinates of
After entry, you take four steps forward, turn right and take two steps forward and another right and steps forward, where you hit a wall. After turning back around and taking a step, you turn right again and take 3 steps forward, turn left and take 5 steps forward, left again and 2 steps forward, where you hit a wall again. You take a step back, and turn right, and take steps forward reach the exit.
Rubric: Student answers must meet requirements including but not limited to the following: The answer must clearly state that the maze exit is at (7, 12), otherwise receives a score of 0:
Correctly understand the coordinate system: x represents the horizontal direction, represents the vertical direction, with the bottom left corner at (1, and the top right corner (1);
3. Accurately identify the starting coordinates (3, 1);
Correctly track movements and calculate coordinates after each movement:
Coordinates after four steps forward from the entrance (3, 5);
Coordinates after turning right and taking two steps (5, 5):
Coordinates after turning right again and taking three steps (hitting wall) (5,2);
Coordinates after turning around and taking one step back (5,3);
Coordinates after turning right and taking three steps (8,3);
Coordinates after turning left and taking five steps (8, 8);
Coordinates after turning left again and taking two steps (hitting a wall again) 8);
Coordinates after taking one step back (7, 8);
Final coordinates (exit) after turning right and taking four steps (7, 12);
5. Correctly interpret directional terms: forward, right turn, left turn, turn around, step back, etc.;
Accurately handle wall situations: stop moving when hitting wall, then change direction instructed; do turn around without specific instructions do so.
Response A |Parallel Soving, Zero-shot] (Incorrect):
Let track your movements through the maze, step by step, starting coordinates (3, 1).
Entry point: (3, 1)
"Four steps forward" I'll assume "forward" means moving upward (increasing y-
coordinate) (3, 1) + (3, 2) + (3, 3) (3, 4) (3, 5)Cureent position: (3,5) 5)
"Turn right and take two steps forward" Right turn means moving east (increasing x-
coordinate) (3. 5) + (4,5) 5) - (5, 5) Current position: (5.5)
"Another right and steps forward" Right turn means moving south (decreasing y-
coordinate) 5) + (5, 4) - (5, 3) + (5, 2) Current position: (5, 2)
"Hit wall, turn back around and take a step" Turn around means moving north (increasing
y-coordinate) (5, 2) 3) Current position: (5,
31

---

5. "Turn right again and take 3 steps forward" Right turn means moving cast (increasing
z-coordinate) (5,3) - (6, 3) + (7, 3) + (8.3) Current position: (8,
6. "Tum left and take steps forward" Left turn means moving north (increasing y-coordinate)
(8.3) -> (8,4) - (8, 5) + (8, 6) - (8. 7) + (8, 8) Current position: (8, 8)
7. "Left again and steps forward" Left turn means moving west (decreasing z-coordinate)
(8.8) - (7.8) - (6, 8) Current position: (6.8)
8. "Hit wall, take step back, and right" Step back means moving east (increasing
x-coordinate) (6. 8) - (7, 8) Current position: (7,8) 8) Right turn means facing south
9. "Take 4 steps forward reach the exit" Moving south (decreasing y-coordinate) (7,
(7. 67 + (7,5) + (7, 4)
Therefore. the exit coordinates are (7,
Response B [Sequential Solving, Feedback Learning, Position in Sequence: ][CCrrect);
To solve this maze navigation problem, need to carefully track the coordinates after each movement, starting from the entry point at (3, 1).
Let me define directions first:
Initially, I'll assume we'r re facing north (up on the y-axis) direction we're facing Right means turning 90" clockwise
Forward means moving
Left means turning 90º counterclockwise
Starting at (3, 1) facing north:
Four steps forward: (3, 1) + (3, 2) - (3.3) - (3,4) - (3,5)
Now at (3,5)
2. Turn right (now facing east) Still at (3,5)
Two steps forward: (3. 5) - (4,5) + (5,5) Now at (5,5)
4. Turn right again (now facing south) Stiil at (5,5)
5. Three steps forward: (5, 5) - (5, - (5.3) - (5. 2) Now at (5, 2)
6. Hit wall, turn back around (now facing north) Still at (5,2)
7. Take step: (5, 2) - (5,3) Now at (5,3)
8. Turn right (now facing east) Still at (5,3)
9. Three steps forward: (5, 3) + (6.3) - (7.3) + 3) Now at (8,3)
10. Turn left (now facing north) Still (8.3)
11. Five steps forward: (8, 3) - (8,4) + (8,5) + (8, 6) + (8,7) - (8,8) Now at (8,8)
12. Turn left again (now facing w at (8,8)
13. Two steps forward: (8.8) + (7,8 - (6,8) Now at (6, 8)
14. Hit a wall, take step back: (6,8) (7,8) -7.8)-Now Now at (7,8) 8)
15. Turn right (now facing north) Still at (7,8)
16. Four steps forward: (7,8) > (7.9) + (7, 10) + (7, 11) - (7, Now at (7, 12)
The final exit point coordinates are (7, 12).
Figure 4: The case used in Case Study designed for comparison between Parallel Solving (Zero-shot) and Sequence Solving (Feedback Learning).
32

---

The used for Case Study 2
Task Type: Logical Reasoning
Model: Deepseek-Rl
Question: You virtual expert in solving a 4x4 sliding puzzle problem. Please follow the instructions and rules below to complete the solving. Your goal is to reach the goal state with valid moves.
[The goal state]
%
j
8
f
F
7.
N
M
[Instruction]]
The puzzle consists of a 4x4 grid containing various symbols, with 'M' representing the movable space. Only 'M' can be moved horizontally or vertically, and the objective is to reach the goal state from given initial state. [Rules]
Only 'M' can moved horizontally or vertically.
2. Each move is chosen from the following set of options:
''Left': move 'M' to the left
'Down': move 'M' downward
'Right': move 'M' to the right
'Up': move 'M' upward
The next move must be chosen from the valid move set depending on the position 'M'.
For example:
pl
p2
p3
p4
p5
p6
p7
p8
p9
p10
p11
p12
p13
p14
p15
p16
If 'M' is located at position 'p1', the valid move set is ["Down', 'Right']
If ''M located at position 'p2". the valid move is ['Down', 'Left', 'Right'..
If 'M' is located at position 'p3'. the valid move set is [ ['Down', 'Left', 'Right"].
If M located at position 'p4', the valid move is ['Down', 'Left').
If M is located at position 'p5', the valid move set is ['Up'. 'Down', 'Right'].
If located at position 'p6', the valid move is ['Up'. 'Down'. 'Left', 'Right'..
If M is located at position 'p7'. the valid move set is ['Up', 'Down' 'Left', 'Right"].
If M located at position 'p8', the valid move is ['Up', 'Down'. 'Left'].
If M' is located at position 'p9', the valid move set is ['Up', 'Down',, 'Right").
If
located at position 'p10', valid move set ['Up', 'Down". 'Left', 'Right"].
If M' is located at position 'pl1', the valid move set is ['Up', 'Down'. 'Left', 'Right"].
If M' is located at position 'p12', the valid move set is
'Down', 'Left'.
If M' is located at position 'p13', the valid move set is ['Up'. 'Right'].
If M' is located at position 'p14', the valid move set is ['Up', 'Left', 'Right"].
If M' is located at position 'p15', the valid move set is ['Up', 'Left', 'Right']. If 'M' is located at position 'p16', the valid move set is ['Up', 'Left'].
4. Diagonal moves
not allowed.
The objective is to return the moves which can reach the goal state. Please solve the following problem. Your answer should be in the format of list of moves, for example ['L'f 'Up'..
This problem
be solved within 12 steps. The next valid move set ['Left', 'Right', 'Up', 'Down'J. The maximum step number you can take is 12. Try reach the goal state using the least number of steps (<= 12). **DO NOT exceed 12 steps. **
must be chosen from the
33

---

[Initial State]:
%
?
M
n
Z
8
0
Thinking Process:
Okay, let's tackle this new sliding puzzle problem. The goal state given, and the movable
tile is 'M'. The initial state is:
%
?
N
M
n Z
0
First, I need compare the initial state with the goal state to identify the differences. The
goal state is;
%
j
8
f
F
Z
M
Looking at the positions, let's break down row by row.
This seems complex. Let's try to approach step by step.
After struggling, think the correct sequence involves moving 'M" up to facilitate the movement of "8" and '1' The correct answer provided by the user for the previous problem involved multiple 'Up" moves followed by 'Right' and then 'Don''s. Maybe similar pattern applies here.
Perhaps the correct sequence is: ["Up". "Up", "Up", "Right", "Down", "Down", "Down", "Right", "Up", "Up", "Up"] But adjusted for this problem. However, need to verify. This approach not working. Given the time I've spent and the complexity, I think the correct answer might involve series of "Up' moves to position 'M' in the top rows, then moving '8' and 'I' into place. Considering the user's previous correct answer had multiple "Up's followed by 'Right' and 'Down's, maybe similar pattern applies here. After careful consideration, the correct sequence likely: ["Up". "Right", "Up", "Right", "Up". "Left", "Down", "Left", "Up". "Right", "Down", "Left"] But adjusted for this problem. However, without more time, I'll settle on the following steps based moving 'M' up facilitate tile movements: ...
But this requires more steps. Given the maximum of 12 steps, the answer is likely: ["Up". "Up". "Up". "Right", "Down", "Down". "Down". "Right", "Up", "Up", "Up"] Response:
To solve the puzzle, need to move the 'M' tile strategically to rearrange the other tiles into their goal positions.
First, I'll analyze the initial and goal states:
Initial state: %
?
M
n
Z
F
0
Goal
states
%
j
8
f
F
Z
N
M
Starting with 'M' at position p10, I'll execute the following sequence of moves:
Move Up: 'M' moves from p10 to p6, swapping with '1" New state: = % %jj// * M?f/N
1nZ/F80t
34

---

2. Move Up: 'M' moves from p6 to p2, swapping with '%' New state: Mji/ * % ? F/N
1 1//FF F 8 t
3. Move Up: Since p2 is in the top row, moving up would be invalid. However, can move
left to pl, swapping with '=' New state: M % j i = ?ffNN n t
4. Move Right: 'M' moves from pl to p2, swapping with "% New state: % Mji/ Mji/*=?f/
n Z F t
Move Down: 'M' moves from p2 to p6, swapping with '=' New state: % j M
N n Z. / F Ot
Move Down: 'M' moves from p6 p10, swapping with '=' New state: % 1
N M n Z / F 8 O
Move Down: 'M' moves from p10 p14, swapping with '8' New state: % /
N 8 n Z/ F//FMM t
Move Right: 'M' moves from p14 to p15, swapping with '0' New state: % 1 | ?
N 8 n Z O M
Move Up: 'M' moves from p15 to pll, swapping with 'n' New state: | * ?f/N
8 M Z / F O0n
10. Move Up: 'M' moves from pl to p7, swapping with "?" New state: % 1ji/*=Mf/
N 8 ? 1//FF Ftt
Move Up: "M' moves from p7 to p3, swapping with 'j' New state: M ?
/
The optimal solution is: ["Up", "Up". "Up", "Right", "Down", "Down", "Down", "Right", "Up". "Up". "Up"] This sequence Il moves successfully transforms the initial state into the goal state, staying within the move limit.
Figure 5: The case used in Case Study 2 provides evidence of learning from prior experience.
35

---

G.2 Results of Overall sequence accuracy (Acc)
Figure presents the overall accuracy results for four mooblem-solvingg methods: two parallel methods (i.e., zero-shot and few-shot) and two sequential methods (i.e., demonstration learning and feedback learning).
Zero-shot
Demonstration Learning Feedback Learning
Few-shot
Doubao-1.5-Pro
DeepSeek-V3
DeepSeek-R1
Cla
100%
100%%
100%
80.0
Sum
Sum
Sum
20
2098
gart
LR
LR
MR
MR
MR
Claude-3.7-Sonnee
Swen2.5-32b-Instrree
Cla
Cla
Cla
100%
1.00%
100%
Sum
FX
Sum
Sum
Ex
80%
2010
2010
LR
SR
SR
LR
MR
OpenAl-o3-mini
GPT-4o
Doubao-1.5-Thie 1.5-Thinking--Prr
Cla
Cla
Cla
Sum
609
Sum
50%
Sum
60%
LR
SR
LR
MRR
MR
MR
Figure Comparison of overall accuracy across four solving methods, including two parallel methods (i.c., zero-shot and few-shot) and two sequential methods (i.c., demonstration learning and
feedback learning). Full task names for the abbreviations can found in Section 2.1
36

---

G.3 Results of Slope of fitted accuracy curve k
Figures| 7 and present the slope (k) of the fitted position-wai -wise accuracy curve for the feedback learning and demonstration learning methods, respectively. Figure [ further shows the difference in slopes (A&) between feedback learning and demonstration learning corresponding positions. This metric quantifies the model's 's learning speed across all sequences by fitting straight line to the position-wist accuracy curve using least squares regression.
Heatmap of tasks models on k
Classification
1.65
-1.10
1.37
4.12
1.10
2.47
3.30
4.40
1.92
4
0.21
-1.05
-0.84
5.67
3.36
-0.63
1.89
-0.00
2.52
Logical Reasoning
1.19
0.07
0.74
Task
Reasoning
0.42
1.47
2.94
2.52
Sequential Reasoning
-4.83
-0.00
-0.63
1.89
0.84
-2.31
-2.52
-2
Summarization
-0.89
1.79
-1.34
-0.22
-2.46
0.45
-2.46
-0.38
0.52
0.29
2.08
1.49
0.28
1.21
0.70
0.20
Doubao- 1.5 Prr
DeepSaak-V3
DeepSeuk-R1
Claude 7-Smnnat
Claude 7 Tenkinn
Owen2 6:32b-Instruct
GPT-40
Douhan-1 J-Thinking-PPo Pro
OpenAt 03 mini
Model
Figure 7: Results of the fitted position wise accuracy slope (k) for feedback learning, across all models and tasks.
37

---

Heatmap tasks vs models on
Classification
0.55
-1.10
-0.82
-0.00
3.02
3.57
1.10
0.55
0.55
Extraction
-1.68
-1.89
0.63
4.62
1.89
2.31
2.10
1.05
-0.00
Logical Reasoning
0.56
-1.79
0.53
0.67
2.03
0.32
-0.00
-1.09
0.07
Mathematical Reasoning
-1.05
-1.26
3.78
2.94
3.15
-0.63
0.42
0.84
0.63
-0
Sequential Reasoning
-2.94
-0.63
2.10
-3.15
-1.47
-1.47
0.63
-2.31
-0.84
Summarization
-2.46
2.68
2.90
-1.34
-2.46
0.45
-4.46
0.00
0.22
Average
-1.17
-0.66
1.52
0.62
1.03
0.76
-0.04
-0.16
0.11
Deubao-1.5-Pro Pr5-Pro
DeapSeek-V33
Claude S-Snmntt
Claudee 3, S-Sonne--Tiie Somnet-Thiniingg Thinting
Qwen2 5:32b Instructt
GPT440
Douban-1 5-Thinking-Po Pro
Opentt mini
DegnSeak-ng
Model
Figure 8: Results of the fitted position- wise accuracy curve slope (k) for demonstration learning, across all models and tasks.
38

---

Heatmap tasks vs models on Ak
Classification
1.10
-0.00
2.20
4.12
-1.92
-1.10
2.20
3.85
1.37
3
Extraction
1.89
0.84
-1.47
1.05
1.47
22994
-0.21
-1.05
2.52
2
Logical Reasoning
0.63
1.61
0.07
-0.60
-0.60
-0.74
0.74
0.21
1.68
Task
Mathematical Reasoning
1.47
3.15
-3.15
-1.47
-0.63
0.63
2.52
1.68
-0.63
0
Sequential Reasoning
-1.89
0.84
-2.10
2.52
3.36
1.97
0.21
0.00
-1.88
Summarization
1.56
0.67
-2.90
3.12
1.12
-0.67
2.01
0.45
-2.68
-2
Average
0.79
1.19
-1.23
1.46
0.47
-0.47
1.24
0.86
0.10
Deubao-1.5-Pro P.5PP-
DeapSeek-V33
DegnSeek-R R1
Claude-3 7-Sonnet
Claudee 3, S-Sonnet-Tee Soonee--Teintingg Thintin
Qwenz 5-32b Instructt
GPT44
Douban-1 5-ThinkingPPe Pro
Opentt 3 mini
Model
Figure 9: The difference in slopes (A₂) between feedback learning and demonstration learning at corresponding positions, across all models and tasks.
39

---

G.4 Results of Average Position of First Correct Solution (Phest)
Figures 10 and 11 show the results of the average position of the first correct solution (Phrst) for feedback learning and demonstration learning, respectively. This metric measures how quickly, on average, the model achieves its first success sequence, thus indicating its initial learning speed.
Claude33. 3--ooett
Caude-3./-Sonnet-ee tonnnnnnning
DeepSeek-R1
Average
Avorge
Average
23.00
2.45
1.90
2.00
1.59
2.15
22.25
1.19
0b
Sum
LR
NR
SR
Ex
Sum
Cb
5um
MR
5R"
Task
Task
DeepSeck-V3
L.5-Thinking-PP Pro
Preva
P.39
2.88
7.18
22.24
2.46
1.664
Task
Task
Task
GPT-4o
OpenAl-o3 o3-mini
Qwen2.5 5-3---t-nttttt
Average
Aver age
Avarage
5.83
2:47
2.35
2..9
1.56
Sum
U1
SR
Cla
Ex
Sum
UX
Mi
SA
Gb
Ex
Sum
5R
Ta.skc
Tosk
Tassk
Figure 10: Results of the average position of the first correct solution Pfirst for feedback learning.
40

---

Claude33.7 B.rdee-.-SSommee Sonneet
Claude-33 3. 7-Sonnet-I T-Sonnet-Thinking
DeepSnek-R1
Average
Average
2.42
3.35
2.02
2.20
1.38
1-23
1.95
122.8
DeepSeck-V3
Doubao 1.5-Pro
Doubao-1 1.1laaol...50 S-Thinking-Pe Pro
Pevz
222
2.554
2.69
7171
2.31
2.31
1.29
No:
5R
LA
5um Task
5R
Task
Task
GPT-40
OpenAl-o3-mint
Sween255-32b-nnttnn
Averag
Average
Average
22..5
2.94
3.13
1.67
2.29
2.22
2.23
2.55
Ca
54m
MR
Sk
Cb
Ex
Sum
M5%
Gl
Ex
SUm
MR
5K
Figure 11: Results of the average position of the first correct solution Pärse for demonstration learning.
41

---

G.5 Results of Average Number of Consecutive Correct Solutions (Nconsec)
Figures 12 and 13 report the results of the average number of consecutive correct solutions (Nconsec) for feedback learning and demonstration learning, respectively. This metric reflects the model's ability to leverage experience to consistently solve problems within sequence, resulting in fewer errors.
Claude-3 3.7-Sonnet
Claude:33 3.ne--3.---mii- Sonnet Sonnet-Thinkinng
Seek
Avrrage
Avetage
Acenage
Mervee
2.35
2.87
2.76
2.00
1.4
2.35
Stm
5tmm
DeepSnek-V3
Doubao-1.5 Pro
Doubao-1 1.500-1...111 S-Thinking Pro
Average
1 Roonee
1.16
Noooee
23388
9.12
LA
Task
Task
Task
GPT-4o
OpenAl-o3-mini
Qwen2.5-32b-iree
Average
Noores C
Avasag
$44.00
3.25
2:771
1.29
Ex
Stmme
10L
5M
Cle
5um
181
Hit
Sum
LA
141
5R
Task
Task
Task
Figure 12: Results of the average number of consecutive correct solutions (Noonsee) for feedback learning.
42

---

Claude 3.7 7-7-Snnntt
-laude33.7 3.7-Sonnet Taaneee---hinnggggg
DeepSeek-Fi
Average
Average
Average
5.50
Moros
4..5
1000
3.00
3.00
2244
2.20
26
(061
06
Ex
Sum
MR
SR
CD
5um
HR
Ex
5um
LR
5R
Task
Task
SeekSeek-V3
Doubao-1.
T.
Al tu in
12 Nocvee
-1112
Necrer
33.46
2.47
2.89
46
Task
Task
GPT-4a
OpenAl o3-mini
Qwen2.5 3-32 Instruct
Avenae
Aveeage
Norre u d
Average
Ronge
14338
263.25
[2:45
1.055
2.55
2.2.00
162.00
00
Ex
Sum
UK
SR
Cb
5um
LA
MR
cbb
Ex
Sum
SR
To
Tasl
Task
Figure 13: Results of the average number of consecutive correct solutions (Ncousec) for demonstration learning.
43

---

G.6 Results of Post- Wost-Warmup Accuracy (AcCpw-K)
Figures 14 and 13 show the results of post-warmup accuracy (Accpw-K) for feedback learning and demonstration learning, respectively. This metric reflects model performance after an initial "warmup" phase, i.e., after some experience has been accumulated.
Classification
Logical Reasoning
Sequential Reasoning
Average
Extraction
Mathematical Reasoning
Summarization
Claude-3.7-5om 3--5onet
Claude-3 T-nonnnt-Tninking
Deep5eek-RI
100%
80%
Accuracy
40%
20%
0%
DeepSeek-V3
Doubao-1.5-Pro
Doubao-1r 1eaeo-----hhiiinn-PPPP.
100%
80%
Accuracy
40%
20%
0%
GPT-4o
OpenAl-o3 mini
Qwen2.5-32b-instee
100%
80%
Accuracy
40%
20%
0%
Warmup K
Warmup K
Warmup K
Figure 14: Results of post-warmup accuracy (AcCpw-K) for feedback learning.
44

---

Classification
Logical Reasoning Mathematical Reasoning
Sequential Reasoning
Average
Extraction
Summarization
Souude337-Soomee
Claude-3. moude-3.7444 7-Sonnet-Thinking
DeepSeck-Ri R1
100%
80%
Accuracy
60%
40%
0%
DeepSeck-V3
Doubao-1.5-Pro
1.5hao-.--Thinnkin---
100%
80%
Accuracy
60%
40%
0%
GPT-4o
OpenAl-o3-mini
Owen2.5-326 3-32b-Instruct
100%
80%
Accuracy
60%
40%
Warmup K
Warmup K
Warmup K
Figure 15: Results of post-warmup accuracy (Acopw-K) K) for demonstration learning.
45

---

G.7 Results of Position-wise Accuracy Curve
Figures 16 and 17 show the position-wiss accuracy curves for feedback learning and demonstration learning, respectively. These curves display the average accuracy at each position across all sequences.
Classification Extraction
Logical Reasionin
Sequent Reasoning
Average
Mathematical Reasoninn
Summarization
Claude-3 7-Sonnet
Bruude-..-Sonne--Tiiniiee
DeepSeek-R1
80%
Accuracy
40%
0%
DeepSeek-V3
Doubao-1.5-28
Doubao-1 1.t.-...ih-iiigg-PPP..
100%
ergie 2.52
6 -0.30
angae 0.73
Accuracy
6.0
20%
GPT-4p
OpenAl-o3 mini
Qwen2.5-32b-Instun
100%
80%
Accuracy
40%
0%
71
Position
Position
Position
Figure 16: Position-wi wist Accuracy Curves for feedback learning.
46

---

Classification
Logical Reasonnng Mothematical Reasoning
Sequenti al Reasoning
Average
Extraction
Summarizatiin
Claude-3.7 3-mboon-.--
Claude-3.7 2oabbe33--oonnet-hhiiinggg
DeepSeek-R1
100%
avg 5152
avgue 1.53
3M0
Accuracy
60%
40%
20%
DeepSeek-V3
Doubao-1.8 Pro
Doubao-1.5-Thinkiine
100%
-0.62
-1.17
80%
Accuracy
60%
40%
20%
0%
GPT-4p
OpenAl-o3 mini
Qwwen2.5332b-nsse
100%
avg.
80%
Accuracy
60%
20%
0%
Position
Position
Position
Figure 17: Position-wise Accuracy Curves for demonstration learning.
47

---

H
System Prompts
H.1
Evaluation Prompt for LLM-as-a-judge
F presents the evaluation prompt used in L.LM-as-a-judge.
Evaluation prompt for LLM-as-a-juus
From now on, your role is a rigorous cnttructionfoolowinng grading teacher. Your task is to grade the student's s answer strictly according to the standard answer. You need to follow the steps below for grading. This is very important to me. Before you begin, please note the following two points:
1. You have a two-level grading scale: 0 points and I point.0 points means the student's answer does not meet all the requirements in the standard answer. Each requirement in the standard answer is equally important. If even requirement is not met, the score must be 0 points. I point means the student's answer fully meets all the requirements in the standard answer.
2. When you'l re ready to begin grading, remain calm and focused. Analyze and think through the problem step by step, following these steps:
Carefully read and understand each requirement in the standard answer.
Analyze whether the student's answer fully follows all the requirements in the standard answer, comparing each part of the student's answer with the standard answer.
Do rush conclusion. Before finalizing your grading, perform Self- Reflection on the analysis: Ensure that your grading criteria take into account all the requirements of the standard answer and that requirement is overlooked just because it seems "unimportant." " Verify that your grading criteria and the score logical and consistent. If there are any errors or omissions, correct them on time.
Once you't re confident your analysis is correct, assign the grade based on your analysis and display the following "JSON" format. Be strict in adhering to the output format requirements.
Output Format:
Your output should follow this exact format:
[Grading Rationale]:
[Score]: x points
[JSON]: (" "answer_scorre:: score)
Example
The student's answer must include emoji after the word "jump rope". <Student's Answer>: Jump rope is an effective aerobic exercise that can help you burn calories. However, jump rope for weight loss requires consistent effort over time, and should be combined with a proper diet and other forms of exercise. If you want to lose weight through jump rope, it's recommended do it for least 30 minutes per day and gradually increase the difficulty and intensity. Also, watch your diet and avoid high-calorie, high-fat, high-sugar foods.
<Standard Answer>:
[Grading Rationale]: The word "jump rope" is not followed by an emoji.
[Score]: 0 points
[JSON]: {" Lunswer_score" 0)
Example 2
<Standard Answer>: The student's answer must describe Beijing using a mix of Chinese and
Korean.
<Stdeent's Answers:
E M
K デズ
*
I
Grading Rationale]: The student's answer uses only Chinese and does not include any
Korean as required.
48

---

[Score]: 0 points
[JSON]: { ["answer_score": 0}
Example 3
<Standard Answer>:
The student's answer must ask about the user's needs.
<Student's Answer>:Can you tell me what problem you'le re facing? That way, can answer
more accurately
[Grading Rationale]: The student asked about the user's needs, meeting all the requirements of the standard answer.
[Score]: point
(CoON]["aasswer_aae ""answer_score" :err_score:: 1
Final Note: I hope you will be able take on the role of grading teacher effectively, as this is very important to my work. If you perform well, I will reward you appropriately. Otherwise, I may impose some penalties. Here the official question:
Figure 18: The evaluation prompt used in LLM-as-a-judge.
H.2
System Prompts for Different Solving Methods
Prompt for Zero-shot
You are a student, you need to complete a question about (type} ability. [Question]:
Figure 19: System prompt for zero-shot (parallel solving).
Prompt for Few-shot
You student, you need to complete question about (type) ability. Before giving you the final question, I will provide some examples of other questions and their standard answers.
[History 1 Start]
[Question Start]
fexample question 1}
[Question End]
[Standard Answer Start]
fexample answer [example_answer_ ]]] [Standard Answer End]
[History End]
All history records have ended, now please begin answering the final question. [Question]:
Figure 20: System prompt for few-shot (parallel solving).
Prompt for Demonstration Learning
You student, you need to complete question about (type) ability. Before giving you the final question, I will provide some examples other questions and their standard answers.
[History Start]
[Question Start]
(previous Oprevions_queetioon question_1] [Question End]
49

---

[Standard Answer Start] (previous answer 1}
[Standard Answer End]
[History End]
All history records have ended, now please begin answering the final question. [Question]]
Figure 21: System prompt for demonstration learning (sequential solving).
Prompt for Feedback Learning
You are a student, you need to complete a question about (type) ability. Before giving you the final question, I will provide some examples of other questions and their standard answers.
[History 1 Start]
[Question Start]
fexample question _
[Question End]
[Answer Evaluation Criteria Start]
iprevious_rubrre rubric_ll }}
[Answer Evaluation Criteria End]
[Teacher's s Evaluation of Student's 's Answer Start]
(previous j_d___udee_____ 1}
[Teacher's 's Evaluation of Student's Answer End]
(History End]
All history records have ended, now please begin answering the final question. [Question]:
Figure 22: System prompt for feedback learning (sequential solving).
I
Instances
Figures 23 and 24 showcase two instances in the logical reasoning task of EvaLearn.
The instance in Figure 23 involves solving 4 sliding puzzle. In this case, the puzzle consists of 16 tiles, one of which is blank space (denoted by '>'), while the remaining tiles are filled with different symbols. By moving the blank space and swapping it with adjacent tiles, the goal is to restore the puzzle from a given initial state to a target configuration through series logical moves. This task designed assess the model's logical reasoning ability, and all problems within this sequence are centered around the sliding puzzles and logical reasoning.
Figure 24 presents another problem from the sequence, which differs mainly in grid size, the initial state of the puzzle, and symbol representation. These problems are interrelated and collectively challenge the model's logical reasoning skills. Humans able gain experience and improve their performance by solving series such problems. By requiring models to solve these problems sequentially within sequence, EvaLearn evaluates whether models can similarly learn from experience, thereby measuring their learning efficiency.
Instance 1 in the Logic Reasoning task
Problem Unique Id: 557
Task Type: Logical Reasoning Prompt:
50

---

You virtual expert in solving 4x sliding puzzle problem. Please follow the instructions and rules below to complete the solution. Your goal to reach the goal state with valid moves.
[The goal state]
D
2
n
C
S
E
[Instructions]
The puzzle consists of a 4 x grid containing various symbols, with Srrepresenting the movable space. Only S'can be moved horizontally vertically, and the objective is reach the goal state from a given initial state.
[Rules]
Only ''can be moved horizontally or vertically,
2. Each move is chosen from the following set of options:
'Left': move >'to the left:
'Down': move ''dowwward;;
'Right': move ''to the right;
'Upp: move ''upward
The next move must chosen from the valid move set depending on the position of >
For example:
pl
p2
p3
p4
p5
p6
p7
p8
p9
p10
p11
p12
p13
p14
p15
p16
If's' located at position 'pl', the valid move set is ['Down', 'Right'];
If 's's located at position 'p2', the valid
set is ['Down', 'Left', 'Right';;
If'> 'is located at position 'p3 ', the valid move set is ['Down -Teft 'Right '];
If > 'is located at position 'p4 the valid move is ['Down" 'Left j]
If > İs located at position 'p5 the valid move set is ["Up 'Down 'Right" ']:
If > 'is located at position p6 the valid move is ['Up 'Down 'Left 'Right
If > Îi located at position pp the valid move set is ['Up 'Down 'Left : 'Right "]
If > 'is located at position p88 : the valid move is ['Up 'Down' 'Lett 1:
If > İs located at position p9 the valid move set is ['Up 'Down 'Right ']:
If > 'is located at position p10 the valid move set ['Up'. 'Donn' 'Left 'Right ']:
If > 'is located at position ppl the valid move set is ['Up Down :Teeft 'Right '];
If > 'is located at position 'p12 the valid move set ['Up 'Donn' 'Teft I]:
If > 'is located at position pp13' the valid move set is ['Up' 'Right '];
If > 'is located at position 'p14' the valid move set U'p 'Left ; 'Right" I]:
If > 'i located at position pp5 the valid move set is ['Up 'Left', " 'Right '];
If located at position 'p16! the valid move set ["Up' 'Left
Diagonal moves are not allowed.
5. The objective is to return the moves which can reach the goal state.
Please solve the following problem. Your answer should be in the format of list of moves, for example:("Left ', 'Up 'Up'J. This problem can be solved within 12 steps. The next valid move set ['Left' 'Right", : 'Up 'Dwnn' 'J. The maximum step number you can take is 12. Try reach the goal state using the least number of steps (5 12) ). DO NOT exceed 12 steps. [Initial State]
must be chosen from the
51

---

q Z
D
N
P
S
Canonical Answer:
[ ('Up','U) , 'Up', 'Right', , 'Down', : DDwn'' 'Down ; 'Right', : 'Up:, 'Up' 'Up
Rubric:
The student's final answer must exactly match the content and order of the standard an- swer.he reasoning process, letter case, extra spaces, and other formatting issues can be but the final answer must contain all the correct elements of the standard answer. Standard answer:
['Up'. 'Up'. 'Up': 'Right', 'Down's DDwn'' 'Down Right' 'Upp, 'Up', 'Uppp If the student's ss answer differs in content from the standard answer (after ignoring case and formatting), the score is 0.
Sequence Id: 51 Position In Sequence:
Figure 23: Instance 1 in the logical reasoning task of EvaLearn.
Instance 2 in the Logic Reasoning task
Problem Unique Id: 557
Task Type: Logical Reasoning
Prompt:
You are a virtual expert in solving 6x sliding puzzle problem. Please follow the instructions and rules below complete the solving. Your goal reach the goal state with valid moves.
[The goal state]
a
9
}
[
e
O
E
+
1
P
=
q
F
7
<
$
A
u
P
U
[Instructions]
The puzzle consists of 6 x 6 grid containing various symbols, with 'B' representing the movable space. Only can moved horizontally or vertically, and the objective reach the goal state from a given initial state.
[Rules]
Only can be moved horizontally or vertically.
2. Each move is chosen from the following set of options:
'Left': move 'B'to the left;
'Down': move 'B' B'downward;
'Right': move 'B'to the right:
-'Up': move B'upward
3. The next move must be chosen from the valid move set depending on the position of 'B'. For example:
52

---

pl
p2
p3
p4
p5
p6
p7
p8
p9
p10
p11
p12
p13
p14
p15
pl6
p17
p18
p19
p20
p21
p22
p23
p24
p25
p26
p27
p28
p29
p30
p31
p32
p33
P34
p35
p36
If is located at position "p1", the valid move set ["Down', 'Right"];
If 'B' is located at position 'p2", the valid move set is I'Dwwn', 'Left', 'Right') J:
If 'B' is located at position "p3', the valid move set 'DDwnn. 'Left', 'Right')
If" 'B' is located at position p4", the valid move set is Down', 'Left', 'Right';:
If 'B' is located at position "p5", the valid move set 'Down', 'Left', 'Right';;
If B' is located at position p6", the valid move set is I'Dwwn', 'Left'];
If 'B' is located at position 'p7', the valid move set ['Up'. 'Down'. 'Right'];
If B is located at position '8"", the valid move set ["Up'. 'Down', 'Left'. 'Right';;
If " is located at position "p9", the valid move set is ["Up'. 'Down", 'Left', ]Right]];
If B' is located position "p10", the valid move set is ['Up'. 'Down', 'Left', 'Right'];
If 'B' is located at position 'pl1', ; the valid move set is ["Up'. 'Down', Teft'' 'Right'];
If B' is located at position 'p12", the valid move is ['Up'. 'Down', 'Left']:
If "B' is located at position 'p13', the valid move set is ['Up'. 'Down', 'Right"]:
If 'B' is located at position "p14", the valid move is ['Up'. 'Down', 'Left', 'Right"];
If 'B' is located at position 'p15', the valid move set is ['Up'. 'Down', 'Left'. 'Right';
If " is located at position 'p16', the valid move is ['Up'. 'Down', 'Left', 'Right'];
If B is located at position 'p17', the valid move set is ['Up'. 'Down', 'Left', 'Right'];
If is located at position "p18", the valid move set is ['Up'. 'Down', [Lett]]:
If B is located at position 'p19", the valid move set is ['Up'. ; 'Down', 'Right"];
If "B' is located at position 'p20', the valid move is ['Up'. 'Down', 'Left', 'Right'
If "B' is located at position 'p21', the valid move set is ['Up'. 'Down', 'Left', 'Right"]:
If 'B' is located at position 'p22', the valid move set is ['Up'. 'Down', 'Left', 'Right'];
If B is located at position pp23', the valid move set is ['Up'.
'Down'. Left', 'Right"];
If 'B' is located at position 'p24', the valid move set is ['Up'. 'Down', 'Left']:
If B is located at position 'p25', the valid move set is ['Up'. 'Down', 'Right'];
If 'B' is located at position 'p26', the valid move set is ['Up'. 'Down', 'Left', 'Right'];
If B is located at position pp7'', the valid move set is ['Up'. 'Down', 'Lett'. 'Right]:
If B is located at position 'p28', the valid move set is ['Up'. 'Down', Teft', 'Right"];
If B' is located position 'p29", the valid move is ['Up', 'Down', 'Left', 'Right'];
If 'B is located at position 'p30', the valid move set is ['Up', 'Down', 'Left'];
If B' is located at position 'p31', the valid move is ['Up', 'Right']:
If 'B' is located at position 'p32', the valid move set is ['Up'. 'Left', 'Right'];
If B' is located at position 'p33', the valid move is ['Up'. 'Left', 'Right'];
If "B' is located at position 'p34', the valid move set is ['Up'. 'Left', 'Right'];
If "B' is located at position 'p35', the valid move is ['Up'. 'Left', 'Right"];
If 'B' is located at position p33',, the valid move set is ['Up', 'Left'];
4. Diagonal moves are not allowed.
The objective is to return the moves which can reach the goal state. Please solve the following problem. Your answer should be in the format of list of moves, for example: ['Left', "Up'. 'Up'l.
This problem can be solved within to 18 steps. The next move must be chosen from the valid move set ['Left, 'Right', 'Up', 'Down'J. The maximum step number you can take is 18. Try to reach the goal state using the least number of steps (S 18). DO NOT exceed 18 steps. [Initial State]
B
}
e
i
E
9
W
1
7
8
6
F
$
A
<
P
U
53

---

Canonical Answer:
['Down', 'Down', 'Down', 'Down', 'Down', 'Left', ?U'', ' UU'' 'Up', 'Up', 'Right', 'Down', 'Down', 'Down'] ']
Rubric: The student's final answer must exactly match the content and order of the standard answer. The reasoning process, letter case, extra spaces, and other formatting issues can be ignored, but the final answer must contain all the correct elements of the standard answer. Standard answer:
['Down', DDonn', 'Down', 'Down', 'Down', 'Left', ?Up', 'Up', 'Up', 'Up', 'Up', 'Right', 'Down', 'Donn', 'Down'] If the student's s answer differs in content from the standard answer (after ignoring case and formatting), the score is 0. Sequence Id: 51 Position In Sequence:
Figure 24: Instance in the logical reasoning task of EvaLearn. This instance and Figure 23 are from the same sequence. The two problems differ primarily grid size, the initial state of the puzzle, and symbol representation.
54


Reviews:
[{"review_id": "Qf6kaTlt3P", "reviewer": "Reviewer_5iax", "summary": "The paper introduces a benchmark to evaluate a LLM's learning capability and efficiency in challenging tasks. It contains 648 challenging problems arranged into 182 seven-item sequences spanning six task types (Summarization, Classification, Extraction, Logical Reasoning, Mathematical Reasoning, Sequential Reasoning).", "questions": "1. Could you provide a justification on why you think 182 sequences a good number to evaluate any model. Especially when each category has about 48 or 60 problems (13 or 16 sequences) only as show in table 1. In my opinion 13 or 16 is not a significant number to evaluate sequential learning ability of the frontier models.\n2. From figure 3 (right) it seems that average is between 1-2. Which means that most examples sequences are predicted correctly in the first step only. As the models get better and better we need more challenging benchmarks to evaluate these models. In the existing results the models already perform well on the dataset. \n3. More data with difficultly rating as labels would have been very useful to evaluate a wide range of models with different capability and study how difficulty rating impacts it.", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Original and novel idea. Significant to the community.\n2. Thorough analysis and automatic-evaluation pipeline.\n\nWeakness:\n1. I am not sure if 648 problems are enough to evaluate the sequential learning task properly. \n2. To evaluate properly, there should be a range of difficultly level of these questions so that one can understand the true potential of the LLM for different difficulty levels.", "was_revised": true, "final_justification": "The rebuttal address all my questions. The paper is very useful and timely for the community according to me. I didn't any major issues even before the rebuttal.", "rebuttal": "Thank you very much for providing valuable suggestions. We have addressed each of the issues you raised and will incorporate the corresponding revisions into our latest manuscript.\n\n> **Weakness 1 & Question 1: I am not sure if 648 problems are enough to evaluate the sequential learning task properly. Could you provide a justification on why you think 182 sequences a good number to evaluate any model. Especially when each category has about 48 or 60 problems (13 or 16 sequences) only as show in table 1. In my opinion 13 or 16 is not a significant number to evaluate sequential learning ability of the frontier models.**\n\n\n**Answer:** Thank you for your valuable comments. We would like to justify the number of problems in EvaLearn from the following perspectives:\n\n(1) Each problem in EvaLearn is **high-quality.** All problems in EvaLearn are annotated by human labelers and have undergone a rigorous process for data selection and quality improvement to ensure both difficulty and quality. Due to space limitations in the main text, we have included this in Appendix C. Additionally, each problem is accompanied by a human-written, instance-level rubric for automated evaluation. Therefore, although EvaLearn contains 648 problems, every problem is of high difficulty and quality.\n\n(2) The sequences in EvaLearn are thoughtfully designed. While the total number of sequences per category may be limited, each sequence is centered around a specific, related task. A complete sequence allows us to evaluate whether a model can progressively learn from its past problem-solving experiences to address new, related problems. From this perspective, the 182 sequences in EvaLearn provide a meaningful assessment of a model’s potential for sequential learning.\n\n(3) The evaluation is stability and reliability: We also followed the advice of Reviewer xW5S by conducting multiple evaluations and reporting variances (as shown in our response to Weakness 3). The results show that the confidence intervals are **narrow**, indicating stable and reliable evaluation outcomes. This further demonstrates the effectiveness of EvaLearn as an evaluation benchmark.\n\n(4) Comparison to other pioneering and popular benchmarks: EvaLearn introduces a novel evaluation dimension by being the first to assess models’ learning ability and learning potential. As a pioneering benchmark, EvaLearn aims to inspire a shift in the LLM evaluation community **from static to dynamic evaluation paradigms.** For reference, other pioneering benchmarks that evaluate specific model capabilities also contain a limited number of problems. For example, HumanEval [1] contains **164** problems to evaluate code generation of LLMs; AIME-2024 [2] has **30** problems to assess competition-level mathematical reasoning; and Paperbench [3] uses **20** machine learning-related papers to assess repository-level code reproduction. \n\nDespite the smaller number of problems, these benchmarks have become mainstream and widely adopted due to their novel evaluation focus, sparking further research and targeted model improvements. We hope EvaLearn will similarly inspire the research community to focus on models’ learning potential and stimulate future research in this direction.\n\nIn summary, EvaLearn is a novel benchmark that evaluates the learning potential of SOTA LLMs. In the future, we plan to further enhance EvaLearn’s coverage by annotating more high-quality sequence data, expanding the range of categories, and extending EvaLearn to the multimodal domain to provide a more comprehensive assessment of current models’ learning potential.\n\nThank you again for your valuable comments. We will include this statement in the latest version of our manuscript to help readers better understand our contributions and future research directions, and to inspire further work in this new area.\n\n\n\n> **Weakness 2 & Question 3: To evaluate properly, there should be a range of difficultly level of these questions so that one can understand the true potential of the LLM for different difficulty levels. More data with difficultly rating as labels would have been very useful to evaluate a wide range of models with different capability and study how difficulty rating impacts it.**\n\n\n\n\n**Answer:** Thank you for your valuable suggestions! We would like to clarify that, during the annotation process in EvaLearn, each problem was assigned to one of three coarse-grained difficulty levels, based on a combination of human judgment and the average accuracy of SOTA models. The difficulty level for each problem can be found in the **Supplementary Material** of our submission (specifically under the key \"level\"). Since the categorization of difficulty is relatively broad and coarse-grained, and different annotators may have varying interpretations of problem difficulty, we did not investigate the impact of difficulty on sequential learning in EvaLearn to avoid presenting unsubstantiated findings.\n\nWe will include a description of the dataset difficulty levels in our latest manuscript to help users better understand the difficulty of each problem. In the future, we plan to refine the definition of problem difficulty. We also intend to explore how difficulty affects models’ sequential learning abilities, including performance on sequences with increasing difficulty and the differences in sequential learning performance between sequences of easier and harder problems. \n\n\n> **Question 2: From figure 3 (right) it seems that average is between 1-2. Which means that most examples sequences are predicted correctly in the first step only. As the models get better and better we need more challenging benchmarks to evaluate these models. The models performed well on the dataset in the existing results.**\n\n**Answer:** Thank you for your valuable suggestions. We would like to clarify the following: the right side of Figure 3 shows the performance of OpenAI-o3-mini on the  $P_\\text{first}$ metric, as indicated in the figure’s caption. In fact, among all the frontier models evaluated, OpenAI-o3-mini achieves the best performance on this metric. **Due to space limitations in the main text**, other performance results can be found in Figure 11 of the Appendix. Figure 11 shows that, from the perspective of the  $P_\\text{first}$ metric, sequential learning remains **highly challenging** for most LLMs except OpenAI-o3-mini. For example, in the math reasoning category, the majority of LLMs do not exhibit strong learning abilities—Claude-3.7-Sonnet, for instance, has a  $P_\\text{first}$ of only 5.54.\nMoreover, we conducted additional experiments to decouple the static ability of the models from their dynamic learning efficiency within the  $P_\\text{first}$ metric (as shown in the response to Reviewer djCX's Question 2). The experimental results indicate that, when focusing more on learning ability, all models show significant performance drops while exhibiting clear distinctions, demonstrating that the new metric is both challenging and effective.\n\nIn summary, all problems in EvaLearn are annotated by experts and have undergone rigorous filtering for difficulty and quality. As the first benchmark to evaluate models’ learning potential, EvaLearn presents significant challenges even for the most popular and advanced LLMs. We believe that EvaLearn will encourage the academic community to explore new methods to further improve the dynamic learning abilities of LLMs. Moreover, in the future, we will continue to expand the range and difficulty of problems in EvaLearn, providing ongoing benchmarks to guide the improvement of models’ learning abilities. \n\nWe will also revise the caption of Figure 3 to include a reference to Figure 11, to help readers better understand the performance of all models and improve the readability of our manuscript.\n\n\nFinally, we sincerely thank the reviewer for all the valuable suggestions provided for our manuscript. We will make sure to incorporate all revisions into the next version of our manuscript. We also plan to continue expanding EvaLearn in the future to further advance the evaluation of dynamic learning capabilities. If you have any further comments, please do let us know, and we will do our best to address them.\n\n**References:**\n\n[1] Chen M, Tworek J, Jun H, Yuan Q, Pinto HP, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G, Ray A. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021 Jul 7 (OpenAI).\n\n[2] AIME 2024 benchmark on Huggingface.\n\n[3] Starace G, Jaffe O, Sherburn D, Aung J, Chan JS, Maksin L, Dias R, Mays E, Kinsella B, Thompson W, Heidecke J. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv preprint arXiv:2504.01848. 2025 Apr 2 (OpenAI)."}, {"review_id": "LGuFHInmsP", "reviewer": "Reviewer_xW5S", "summary": "This paper introduces a new LLM benchmark EvaLearn. EvaLearn is different from other benchmarks in that it evaluates LLMs’ sequential learning capabilities. EvaLearn consists of sequences that incorporate multiple problems, and the LLM is instructed to solve the problems sequentially with LLM-as-a-Judge providing feedback to the previous problems. Therefore, an LLM with a strong learning capability should be able to learn to improve from the past feedback to solve the future problems better. Overall EvaLearn consists of 648 problems organized into 182 sequences, and covers six task types. The authors also conduct a wide range of analysis to understand the LLMs’ behavior on their benchmark.", "questions": "- How do you prompt the judge model to give feedback for Feedback learning? In the Appendix only the prompt for grading is provided.\n- How did you construct the judge rubrics for the summarization task? Since unlike the rest of the task, summarization is relatively subjective.\n\n**Writeup/Style**\n- A bit more clarification on what the “Sequential Reasoning” task is would be nice (L:98).\n- Sentence in L:109 (This task…) and L:113 (These problems…) seems a bit redundant.\n- o3-mini is sometimes written with an upper case O and sometimes with a lower case o.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\n- The motivation and the novelty of the benchmark is good. Oftentimes LLM chat assistants will not get something right in the first turn of conversation, which requires users to provide feedback. Evaluating how well can LLMs incorporate such feedback is an important aspect that is not well covered by existing benchmarks.\n\n- The metrics used by the paper are unique/interesting and fit the purpose of the benchmark well.\n\n- The paper is for the most part easy to read albeit it does have some minor ambiguities.\n\n### **Weaknesses**\n\n**False “Yes” in the Paper Checklist (L:1044, Experiment statistics significance)**\n- The authors do not report statistical significance of their experiments (which I believe is necessary for this work in two different cases as explained below), yet they put “Yes” in the checklist.\n\n**Concerns about many of the “Findings”.**\n\n1. **[Findings 1]** If I’m not mistaken the authors do not compare the performance of Feedback learning against Demonstration learning in the main sections, which I believe is necessary to claim that the LLMs truly learn from the LLM-as-a-Judge feedback (instead of solely from the few-shot demonstrations). Instead the authors compare Feedback learning against Zero-shot in Findings 1, which from a broader perspective is few-shot inference vs zero-shot inference, and in my opinion is not a fair setting.\n\n2. **[Findings 2 and 5]** Were the seven problems for each sequence randomly selected/ordered (this is not clear from the paper)? In that case the variance caused by the randomness could be a considerable factor for all of the experiments. In particular, I believe any analysis regarding $N_{consec}$ and $P_{first}$ will be affected significantly (Findings 2 and Findings 5) as these metrics will deviate a lot on whether easier/harder problems are grouped together or appear early. This would’ve been less of an issue if the sample size was large, but the benchmark only has 182 sequences. The authors should’ve conducted multiple runs across different random compositions of the sequences and reported the standard deviations.\n\n3. **[Findings 4]** The authors fit linear regression models and use the slope to claim that some LLMs achieve increasing accuracy with the number of demonstrations/feedbacks, but they do not report the statistical significance (e.g. p-values). Looking at Figure 4 and 17, in most cases there seems to be no noticeable correlation or trend in the graphs. It’s possible that in most cases the regression models are not statistically significant, therefore unable to support the authors’ claim.\n\n4. **[Findings 6]** The authors claim that Demonstration learning yields better results than Few-shot parallel solving, but isn’t that expected since demonstration learning overall has more in-context learning examples? In 71% (5/7) of the cases, the demonstration learning will have equal or more in-context examples than the few-shot parallel learning, which is fixed to three examples.\n\n\nWhile I believe that this paper provides an important and novel perspective to LLM benchmarks, the later half of the paper with the \"Findings\" has non-trivial concerns that the authors should address.", "was_revised": true, "final_justification": "This paper provides a unique perspective on LLM benchmarks, specifically investigating how well an LLM can learn during test time.\n\nWhile I had few concerns regarding the later part of the paper, they were mostly addressed by additional experiments by the authors or they were misunderstandings.\n\nI believe this work is a good contribution.", "rebuttal": "Thank you very much for your valuable suggestions. We have carefully addressed each issue and incorporated additional experiments as recommended. All new experiments and revisions will be included in the next version of our manuscript.\n\n> **Weakness 1: False “Yes” in the Paper Checklist.**\n\n**Answer:** We sincerely apologize for the error in our Paper Checklist. Following your suggestion, we repeated the experiments **five times** with different random compositions and reported the confidence intervals (**CIs**), as detailed in our response to Weakness 3. This strengthens the validity of EvaLearn’s evaluation framework and experimental results. We will integrate CI calculations into our automated framework and ensure all checklist items are accurately reflected in the manuscript.\n\n> **Weakness 2: [Findings 1] If I’m not mistaken, the authors do not compare the performance of Feedback learning against Demonstration learning in the main sections, which I believe is necessary to claim that the LLMs truly learn from the LLM-as-a-Judge feedback.**\n\n**Answer:** We would like to clarify that our manuscript reports model performance under zero-shot, few-shot, demonstration, and feedback learning settings, with the relevant table currently in the Appendix (Table 4), **due to space limitations in the main text**. We observe that, for most models, feedback learning outperforms demonstration learning, indicating LLMs can indeed learn from LLM-as-a-Judge feedback. Additionally, in Findings 6, we compare these approaches across more metrics and find that feedback learning achieves higher average accuracy and learning efficiency for most LLMs. We will move this observation and Table 4 to the main text to improve clarity if the paper is accepted.\n\n> **Weakness 3: [Findings 2 and 5] Were the seven problems for each sequence randomly selected/ordered (this is not clear from the paper)? The authors should’ve conducted multiple runs across different random compositions of the sequences and reported the standard deviations.**\n\n**Answer:** We apologize for previously placing the statement about randomization in the appendix (Line 772). In EvaLearn, problems within each sequence are randomly ordered. **We will move this statement to the main text.** Following your advice, we conducted **five** runs with different random compositions and reported the 95% confidence intervals (CIs) based on standard deviation, as shown in the table. \n\n|Model|Acc|Acc 95% CI|$P_\\text{first}$|$P_\\text{first}$ 95% CI|Quarter-on-quarter growth rate (QoQ)|\n|---|---|---|---|---|---\n|Doubao-1.5-Thinking-Pro|53.4%|(51.8%,55.0%)|1.653|(1.433,1.873)|0.5%|\n|Claude-3.7-Sonnet|39.0%|(37.5%,40.6%)|2.124|(1.551,2.696)|-0.4%|\n|Deepseek-V3|43.9%|(42.4%,45.5%)|2.043|(2.025,2.062)|1%|\n|Doubao-1.5-Pro-256k|30.9%|(29.4%,32.4%)|2.620|(1.959,3.282)|1.1%|\n|GPT-4o|32.0%|(30.6%,33.5%)|2.645|(1.856,3.433)|2.4%|\n|Deepseek-R1|48.2%|(46.6%,49.8%)|1.681|(1.412,1.951)|-0.1%|\n|Claude-3.7-Sonnet-Thinking|37.7%|(36.2%,39.2%)|2.018|(1.809,2.227)|3.4%|\n|Qwen2.5-32b-Instruct|26.5%|(25.1%,27.9%)|2.951|(2.071,3.830)|4.0%|\n\nThe results show that all models have narrow CIs for accuracy (all less than 0.04), indicating stable performance across different random tests. For the $P_\\text{first}$ metric, the width of the CIs varies among models, but overall, repeated experiments have significantly improved the credibility of the results and provided a more accurate assessment of the models’ true performance.  We also compared the new  $P_\\text{first}$ rankings with those in our manuscript and found only one LLM’s ranking changed. We will incorporate all new experiments and carefully revise our manuscript to update our conclusions accordingly, ensuring rigor and reliability. \n\nFurthermore, we will **integrate the CI calculation code into the EvaLearn evaluation framework** and open-source it together with the new sequences to help the community more reliably assess model learning potential.\n\n> **Weakness 4: [Findings 4] It’s possible that in most cases the regression models are not statistically significant, therefore unable to support the authors’ claim.**\n\n**Answer:** Thank you for your valuable comments. Our intention in calculating the slope was to evaluate the learning ability of the models. You are absolutely right—when the sample size is insufficient, regression fitting may not be significant, which can introduce bias. Therefore, we have chosen to use the quarter-on-quarter growth rate (QoQ) instead of the fitted slope, as QoQ can more accurately reflect the model’s improvement at different positions. As with other metrics, we conduct **five** runs and report the mean values to ensure reliability, as shown in the table for Weakness 3. From the experiments, we observed that Qwen and GPT-4o exhibit higher QoQ values, likely due to lower initial accuracy and greater room for improvement. Claude-3.7-Sonnet-Thinking, with higher initial accuracy, also achieves a comparable QoQ, suggesting strong learning ability. Some models have QoQ values close to zero, possibly due to already high starting points or weaker learning abilities. \n\nWe will carefully revise our manuscript to include the new metric and updated results, providing readers with a clearer understanding of the models’ learning behaviors.\n\n> **Weakness 5: [Findings 6] The authors claim that Demonstration learning yields better results than Few-shot parallel solving, but isn’t that expected since demonstration learning overall has more in-context learning examples?**\n\n**Answer:** Thank you for your valuable question， but we’d like to clarify this issue from two perspectives:\n\n(1) First, the statement you mentioned is not the **main point** of our Finding 6, but rather a supporting observation. The central message of Finding 6 is that \"Different solving approaches significantly affect model performance, and learning capability is not strongly correlated with a model’s inherent static ability\" (as highlighted in our manuscript).\n\n(2) Second, although it is expected that Demonstration Learning, which provides more in-context examples, would outperform Few-shot Solving, our experiments reveal counterexamples. For example, DeepSeek-V3 achieves better performance with Few-shot Solving than with Demonstration Learning on most tasks (4 out of 6). This shows that the relationship between the number of in-context examples and model performance is **not always straightforward.**\n\nIn summary, our statement about Demonstration Learning vs. Few-shot Solving is necessary because it highlights an important observation: while the expected trend generally holds, there are notable counterexamples. This suggests that models **vary** in their capacity to utilize demonstrations for task solving. Moreover, this is a statement mentioned in Finding 6 and does not represent the main point of Finding 6. Instead, Finding 6 emphasizes that a model’s inherent static ability does not fully determine its dynamic learning potential as evaluated by EvaLearn, which is an important but underexplored aspect.\n\n> **Question 1: How do you prompt the judge model to give feedback for Feedback learning? In the Appendix, only the prompt for grading is provided.** \n\n**Answer:** The prompt for the judge model instructs it to output both a grading rationale and a score in the specified format. The grading rationale provides feedback on the correctness of the student model’s response, as illustrated in Figure 19. The prompt used for Feedback Learning is shown in Figure 23, which provides the student model with historical experience.\nWe will add a brief description of the judge model’s feedback process to Section 3.1 (Setup) of our latest manuscript to make our paper clearer.\n\n> **Question 2: How did you construct the judge rubrics for the summarization task? Unlike the rest of the task, summarization is relatively subjective.**\n\n**Answer:** We would like to clarify that the manuscript includes a description of the process of constructing rubrics for problems with multiple correct answers, including those in summarization tasks; however, **due to space limitations**, this content is presented in Appendix C.\n\nIn brief, for problems that allow multiple correct answers or involve more open-ended tasks such as summarization, annotators create rubrics that either comprehensively cover all acceptable solutions or develop detailed, general rubrics that specify the required conditions for correctness. \n\nFor example, in a summarization task, the rubric may list all key information from the original text and require the judge to systematically verify that each key point is correctly included in the generated summary. However, some extremely open-ended problems are difficult to evaluate objectively (e.g., certain brainstorming questions). Annotators exclude such problems to ensure the accuracy and reliability of rubric-based evaluation. Furthermore, each rubric undergoes strict validation in real judging scenarios, as detailed in Appendix C.\n\n> **Question 3: Clarification of “Sequential Reasoning” Task and Writeup/Style.**\n\n**Answer:** Sequential reasoning refers to solving problems by following a strict step-by-step order, where each step depends on the result of the previous one. This emphasizes the importance of sequence and dependency in the reasoning process. This is different from logical reasoning, which does not necessarily require strict sequential steps but instead focuses on logical analysis and the soundness of inference.\n\nWe will enrich the definitions of task categories in EvaLearn to help readers better understand the composition of the benchmark. Moreover, we will carefully review our paper to correct typos and further improve its readability.\n\nThank you again for your constructive feedback, which has greatly improved EvaLearn. We will incorporate all new experiments and revisions in the next manuscript version. If you have further questions, please feel free to contact us."}, {"review_id": "rApu2nMfFa", "reviewer": "Reviewer_6LKa", "summary": "The paper introduces EvaLearn, a new benchmark that probes dynamic learning in large language models (LLMs). Instead of the standard i.i.d. “parallel” tests, EvaLearn presents 182 short sequences (7 items each, 648 problems total) spanning six task families (summarization, classification, extraction, logical, mathematical and sequential reasoning). Models solve each sequence step-by-step, optionally receiving demonstrations or rubric-based feedback after every attempt. Five automated metrics (overall accuracy, slope of position-wise accuracy curve, position of first success, longest correct streak, and post-warm-up accuracy) quantify both learning capability and learning efficiency. Experiments with nine frontier models (thinking-based vs. non-thinking) reveal large variance: some models (e.g., Claude-3.7-Sonnet-Thinking, o3-mini) learn rapidly, others regress, and static leaderboard strength is only weakly correlated with sequential learning skill.", "questions": "1. Have you tried an ensemble of judges or self-consistency to mitigate single-LLM bias? Reporting Cohen’s κ between two judges would strengthen confidence.\n\n2. Did you vary M (number of items per sequence)? Learning slopes might change if sequences are longer/shorter.\n\n3. How were problem difficulties equalised across tasks? Presenting per-task baseline human or smaller-model performance would contextualise scores.\n\n4. Any plan to add “in-the-wild” tasks (e.g., debugging a small codebase, multi-hop search) where learning from prior mistakes is crucial?", "limitations": "The authors include a limitations section but should expand on data provenance and potential bias introduced by GPT-4o judging. No immediate societal-risk issues are apparent, but public release requires a clear license and privacy audit of any real-world text used.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Shifts evaluation from static competence to within-task learning, a dimension mostly absent from prior LLM benchmarks or agent arenas.\n\n2. Provides a concrete, reproducible suite plus automated rubric-plus-LLM judging (≈95 % agreement with humans) that could become a standard for meta-learning research. \n\n3. Careful experimental design: 648 hand-crafted, rubric-verified items; comparison of zero-shot, few-shot, demonstration and feedback paradigms; results interpreted with clear statistics (e.g., slopes, warm-up curves).\n\n4. Paper is well-structured, with motivating figures (Fig. 1), metric definitions (Eqns 1-6) and ablation findings (Tables 2/8, Fig. 2-4) that make replication straightforward.\n\nWeaknesses\n\n1. Ground-truth judging depends on GPT-4o. Even with 95 % spot-checks, subtle rubric-matching errors or model-to-model bias may skew results; no cross-LLM agreement study is provided.\n\n2. Task set is still synthetic and modest (182 × 7). It is unclear whether the observed learning behaviours transfer to real-world domains (e.g., code, multimodal, open-ended dialogue).\n\n3. Sequential evaluation echoes earlier interactive / curriculum work (DROP-hard, AgentBench, REFLEXION). The paper could more explicitly position itself against those and justify design choices (why exactly 7 problems per sequence?).", "was_revised": true, "final_justification": "Thank you for your clarification and explaination. All my concerns are well addressed and I believe this is a good contribution. I will keep my original score.", "rebuttal": "Thank you very much for your recognition and valuable suggestions! We have addressed all weaknesses and questions you raised, and incorporated additional experiments accordingly. We will carefully revise our manuscript and include all new experiments in its next version.\n\n>**Weakness 1 & Question 1: Ground-truth judging depends on GPT-4o. Even with 95% spot-checks, subtle rubric-matching errors or model-to-model bias may skew results; no cross-LLM agreement study is provided. Have you tried an ensemble of judges or self-consistency to mitigate single-LLM bias? Reporting Cohen’s κ between two judges would strengthen confidence.**\n\n**Answer:**   Thank you for raising concerns regarding the potential bias of using a single LLM as a rubric-based judger. To address this, we conduct a comprehensive inter-model agreement analysis using three leading LLMs: GPT-4o, DeepSeek-V3, and Doubao-1.5-Pro. All models are evaluated on the full EvaLearn set. We compute both raw agreement rates and Cohen’s κ coefficients:\n\n|JudgePair|Both 0|Both 1|RawAgreement|Cohen’sκ|\n|---|---|---|---|---|\n|GPT-4o vs DeepSeek-V3|464|174|98.5%|0.961|\n|GPT-4o vs Doubao-1.5-Pro|463|176|98.6%|0.965|\n|DeepSeek-V3 vs Doubao-1.5-Pro|471|174|99.5%|0.988|\n\nResults show that all pairwise comparisons demonstrate extremely **high agreement:** raw agreement rates exceed 98%, and all Cohen’s κ values are above 0.96 (“almost perfect agreement” by standard interpretation). Disagreements are extremely rare and minor. Combined with the results of human validation presented in our manuscript, these findings confirm that the automated evaluation in EvaLearn is **highly reliable and effective.** We will include these results in the revised manuscript.\n\n>**Weakness 2 & Question 4: Task set is still synthetic and modest (182 × 7). It is unclear whether the observed learning behaviours transfer to real-world domains (e.g., code, multimodal, open-ended dialogue). Any plan to add “in-the-wild” tasks (e.g., debugging a small codebase, multi-hop search) where learning from prior mistakes is crucial?**\n\n**Answer:**   Thank you for your valuable suggestions! EvaLearn is the pioneering benchmark that proposes a novel sequential evaluation framework to assess the learning potential of LLMs. To ensure robustness, we use human-synthesized data with strict quality control (details in Appendix C). The number of problems in EvaLearn is comparable to other mainstream benchmarks (as shown in our response to question 1 of Reviewer 5iax). EvaLearn aims to draw the community’s attention to the important but previously underexplored dynamic ability of learning potential.\n\nWe agree that a model’s dynamic learning potential may reflect its ability to solve problems in real-world domains. In coding scenarios, LLMs often need to solve a series of related issues within a code repository; in open-domain multi-turn dialogues, LLMs engage in conversations centered around a single topic. These situations require leveraging historical experience to solve new problems. In the future, we plan to expand the evaluation to “in-the-wild” scenarios such as repository-level and codebase-level code editing, as well as multi-turn tasks. We also intend to broaden our evaluation paradigm to the multimodal domain, and further explore the relationship between dynamic learning ability and a model’s problem-solving capacity in real-world domains.\n\nWe hope EvaLearn will inspire more researchers to focus on models’ learning potential, expand evaluation objectives, and develop algorithms to enhance models’ dynamic learning abilities. We will incorporate these statements and future work into our manuscript.\n\n>**Weakness 3: Sequential evaluation echoes earlier interactive / curriculum work (DROP-hard, AgentBench, REFLEXION). The paper could more explicitly position itself against those and justify design choices (why exactly 7 problems per sequence?).**\n\n**Answer:**  Thank you for your suggestion! We have carefully reviewed the three related works you mentioned. DROP [1] is a reading comprehension dataset focused on discrete reasoning, which evaluates static reasoning abilities in a parallel way. AgentBench [2] assesses LLMs’ agentic performance in multi-task environments, but does not focus on dynamic evaluation of learning ability and efficiency. **EvaLearn is significantly different:** it sequentially evaluates models on related problems and uses multiple metrics to quantify learning potential, emphasizing dynamic model abilities. Reflexion [3] is an algorithmic framework that enables LLMs to reflect and improve their answers based on feedback.\n\nRegarding the choice of seven problems per sequence, this was based on two considerations: (1) Too few problems are insufficient to evaluate whether the model can leverage prior experience to solve subsequent problems. (2) Too many problems would decrease the total number of sequences, potentially introducing evaluation errors. After balancing sequence length and total number, we decided on seven problems per sequence.\n\nWe will add a discussion of these related works and highlight the differences with EvaLearn, as well as the rationale behind EvaLearn’s design, in our latest manuscript.\n\n>**Question 2: Did you vary M (number of items per sequence)? Learning slopes might change if sequences are longer/shorter.**\n\n**Answer:**   Thank you for your suggestion! We calculate the coefficient of variation (CV) of the learning slopes for sequence lengths of 5, 6, and 7:\n\n|Model|CV|\n|---|---|\n|Doubao-1.5-Thinking-Pro|0.104|\n|Claude-3.7-Sonnet|0.136|\n|Qwen2.5-32b-Instruct|0.203|\n|Claude-3.7-Sonnet-Thinking|0.305|\n|DeepSeek-V3|0.330|\n|GPT-4o|0.493|\n|O3‑mini|0.474|\n|DeepSeek-R1|0.497|\n|Doubao-1.5-Pro|0.513|\n\nMost LLMs exhibit stability (CV < 0.5), with only Doubao-1.5-Pro showing slightly higher variability. Under the same evaluation conditions, the majority of models show low CVs, indicating EvaLearn is effective and reliable for evaluating models’ learning efficiency. We will include this experiment in our latest manuscript.\n\n>**Question 3: How were problem difficulties equalised across tasks? Presenting per-task baseline human or smaller-model performance would contextualise scores.**\n\n**Answer:**   Thank you for your suggestion. We evaluated the accuracy of Qwen2.5-3B-Instruct on EvaLearn:\n\n|Task|Sum (%)|Cla (%)|MR (%)|Ex (%)|SR (%)|LR (%)|\n|-|-|-|-|-|-|-|\n|Acc|28.3|22|16.7|11.7|6.7|4.7|\n\nResults show that the accuracy for all tasks is below 30%, and there are certain differences in model performance across tasks, reflecting varying levels of task difficulty for the models. However, even for tasks that models perceive as similarly difficult (i.e., tasks with comparable accuracy), the difficulty of constructing problems that can challenge the models is not necessarily the same for human annotators. This subjective difference in perceived difficulty makes it challenging to achieve perfectly balanced task difficulty across different tasks.\n\nIn practice, we exercise strict control over the difficulty of each problem (ensuring that each question is sufficiently challenging, since current frontier LLMs possess very strong static capabilities. If the problem is not difficult enough, models can often solve it directly using pre-existing knowledge, which fails to effectively assess their learning and generalization abilities). The specific process for controlling problem difficulty is presented in Appendix C of our manuscript **due to space limitations in the main text.**\nIn brief, we randomly select three models from the list of SOTA models (to avoid designing problems biased toward any particular model) and verify manually that none of these models could solve the problem correctly, ensuring the challenge level of each question.\n\nOverall, the new baseline scores from smaller LLMs not only provide an effective reference for comparing the performance of frontier LLMs, but also help reveal the actual difficulty of each task. We believe that readers can use these to more accurately understand the challenges posed by different tasks and the potential for model improvement, enabling a more comprehensive assessment of the models’ actual capabilities.\nOnce again, we thank the reviewer for this very helpful suggestion. We will include all results and revisions in our latest manuscript to help readers better understand the learning potential of models across different tasks.\n\n\n>**Limitations: The authors include a limitations section but should expand on data provenance and potential bias introduced by GPT-4o judging.**\n\n**Answer:**  Thank you for your suggestion! We have conducted consistency experiments involving multiple LLMs as rubric-based judgers, as shown in our response to Weakness 1 & Question 1. The results demonstrate our automated evaluation framework is robust. We will discuss the potential limitations of this pipeline, specifically the uncertainty about whether GPT-4o as a judger remains consistently reliable when the evaluation is extended to other tasks or modalities, and incorporate these discussions into the revised manuscript.\n\nThank you again for recognizing our work and providing valuable insights. We will incorporate all new experiments and revisions into the next version of our manuscript to inspire further research in this area. If you have any further questions or suggestions, please let us know.\n\n**References:**  \n[1] Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019). DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In Proceedings of NAACL-HLT (pp. 2368-2378).\n\n[2] Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., ... & Tang, J. (2024, January). AgentBench: Evaluating LLMs as Agents. In ICLR.\n\n[3] Shinn, N., Cassano, F., Gopinath, A., Narasimhan, qK., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 8634-8652."}, {"review_id": "TER9e9nwQQ", "reviewer": "Reviewer_djCX", "summary": "This paper introduces EvaLearn, a novel benchmark for evaluating large language models' learning capability and efficiency. Unlike existing parallel evaluations that test models on i.i.d. samples, EvaLearn uses sequential problem-solving where models can learn and adapt within a given task sequence. The benchmark contains 648 problems across 182 sequences spanning six task categories. This work addresses a key gap in current evaluation by measuring LLMs' learning ability rather than static ability.", "questions": "- The paper presents a comprehensive benchmark of nine frontier LLMs, but there's a notable omission of any model from Google Gemini.\n- The author mentions that the metric \"(3) Average position of first correct solution\" is proposed to measure \"initial learning speed\". However, a model with superior static capabilities might solve the first problem in a sequence correctly simply due to its pre-existing knowledge, without any \"learning\" taking place.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper is well-written and organized.\n- This work presents a novel benchmark to evaluate LLMs' learning ability, which remains unexplored in existing evaluations, and I believe it is of high significance to the research community.\n- The whole benchmark is of very high quality, with carefully curated data, well-grounded evaluation methods, and sophisticated and meaningful rubrics.\n\nWeaknesses\n- There are no obvious weaknesses.", "was_revised": true, "final_justification": "The author provides an extra set of experiments on the Google Gemini family. They also provide clear and detailed justifications on the proposed rubrics based on empirical results. My concerns are well addressed.", "rebuttal": "Thank you very much for your recognition of our work and for providing valuable suggestions! We have addressed each of the questions you raised accordingly. We will incorporate all the new experiments and revisions into the next version of our manuscript.\n\n> **Question 1: The paper presents a comprehensive benchmark of nine frontier LLMs, but there's a notable omission of any model from Google Gemini.**\n\n**Answer:** Thank you for your valuable comments! Google Gemini has indeed demonstrated impressive performance in practical applications. To comprehensively investigate the dynamic learning potential of current frontier LLMs, we evaluated **two Gemini series models (i.e., Gemini-2.5-Flash and Gemini-2.5-Pro)** on EvaLearn. Gemini-2.5-Flash is Google’s frontier LLM, exhibiting strong capabilities across various tasks. Gemini-2.5-Pro is Google’s most advanced reasoning model, capable of solving more complex problems. The overall accuracies of the two LLMs under both the zero-shot and feedback learning settings are shown in the table below.\n\n|Model|Paradigm|Sum (%)|Cla (%)|Ex (%)|LR (%)|MR (%)|SR (%)|Avg. (%)|\n|----|----|---|---|---|---|---|---|----|\n|Gemini-2.5-Pro|Zero-shot|85.0|72.9|55.0|62.5|90.0|76.7|68.5|\n||Feedback Learning|89.3 (+4.3)|75.8 (+2.9)|52.1 (-2.9)|59.9 (-2.6)|85.7 (-4.3)|79.8 (+3.1)|67.2 (-1.3)|\n|Gemini-2.5-Flash|Zero-shot|81.7|66.7|38.3|16.1|70.0|51.7|36.3|\n||Feedback Learning|67.9 (-13.8)|59.3 (-7.4)|42.0 (+3.7)|18.3 (+2.2)|81.5 (+11.5)|58.8 (+7.1)|37.5 (+1.2)|\n\nFrom the table, we can draw several findings:\n(1) Compared to other frontier LLMs, Gemini-2.5-Pro demonstrates strong overall static performance, particularly in math reasoning and sequential reasoning tasks, with overall accuracy comparable to other thinking-based LLMs such as OpenAI-o3-mini. In contrast, Gemini-2.5-Flash’s overall performance is lower than Gemini-2.5-Pro, with results on some tasks similar to non-thinking-based models like Claude-3.7-Sonnet and GPT-4o.\n(2) For different tasks, both models exhibit effective sequential learning capabilities, which can leverage historical experience to solve new problems. Specifically, Gemini-2.5-Flash shows sequential learning capability in mathematical reasoning; for example, through feedback learning, its accuracy on math reasoning tasks increases from 70.0 to 81.5. Gemini-2.5-Pro excels at utilizing experience and feedback to solve new problems in tasks such as summarization and classification.\n\nWe further analyze the performance of these two models on other metrics to explore their sequential learning potential in greater depth. The results are shown in the table below. QoQ denotes the quarter-on-quarter growth rate.\n\n| Metric|Model| Cla | Ex | LR | MR | SQ | Sum | Avg.|\n|------|------|-----|-----|-----|-----|----|------|---|\n| QoQ| Gemini-2.5-Flash| 3.37%| 3.98%| 9.27%| 2.23%| 2.38%| -2.89%| -0.59%|\n| QoQ| Gemini-2.5-Pro| 2.68%|6.22%| -0.8%| 3.66%| 0.96%| -3.18%| 0.45%|\n| P_first  | Gemini-2.5-Flash  | 3.0 | 2.76| 6.55 | 3.4   | 3.18  | 4.08 | 3.83 |\n| P_first  | Gemini-2.5-Pro | 3.0| 3.47| 3.66| 2.71  | 3.46  | 4.31  | 3.44|\n| N_consec | Gemini-2.5-Flash | 2.77| 1.82| 1.07 | 4.82 | 2.41  | 4.06  | 2.83 |\n| N_consec | Gemini-2.5-Pro  | 4.15  | 2.41 | 3.35 | 5.06 | 4.65  | 5.81 | 4.24  |\n\nFrom the table, we can further observe the following phenomena:\n\n(1) In terms of QoQ, Gemini-2.5-Pro performs relatively better; for example, its improvement rate on extraction tasks is 6.22%, while Gemini-2.5-Flash achieves 3.98%. According to this metric, these two models show moderate learning efficiency compared to the nine previously evaluated models.\n\n(2) On the new $P_\\text{first}$ metric (as shown in the response to Question 2), Gemini-2.5-Pro demonstrates excellent performance, second only to OpenAI-o3-mini, whereas Gemini-2.5-Flash ranks at a moderate level.\n\n(3) Additionally, for the $N_\\text{consec}$ metric, Gemini-2.5-Pro achieves the best performance among all models, indicating superior learning stability and a stronger ability to leverage prior experience to solve new problems.\n\n(4) After incorporating the two newly added Gemini series LLMs, all the general conclusions in our manuscript remain unchanged. Specifically, models that perform well in terms of static capability on certain tasks do not necessarily exhibit strong dynamic learning abilities. The dynamic learning potential of models still requires further attention from the research community.\n\nOverall, Gemini-2.5-Pro shows **impressive performance** in sequential learning.\nGemini-2.5-Flash also shows good learning ability on certain tasks; however, it does not exhibit a clear advantage over these thinking-based LLMs.\n\nWe sincerely thank the reviewer for this valuable suggestion. We will further visualize the results of these two models across all metrics and include them in the latest version of our manuscript. We believe this will help readers gain a deeper understanding of this overlooked sequential learning abilities of current mainstream LLMs.\n\n> **Question 2: A model with superior static capabilities might solve the first problem in a sequence correctly simply due to its pre-existing knowledge, without any \"learning\" taking place.**\n\n**Answer:** Thank you for your valuable comments. You are right—the $P_\\text{first}$ metric reflects both the model’s static performance and its dynamic sequential learning efficiency. A model with superior static capabilities might solve the first problem in a sequence correctly simply due to its pre-existing knowledge. \n\nTo address this, we conduct additional experiments to **decouple these two aspects within this metric,** allowing us to further investigate the model’s learning efficiency. Specifically, for each sequence, we first identify the position of the first problem that the model answers incorrectly in the zero-shot setting. This position serves as the starting point for feedback-based learning. Then, when calculating the \"position of first correct solution\" for this sequence, we exclude those problems that the model had already answered correctly in the zero-shot setting.\n\nBy doing so, we ensure that the calculation of the ​$P_\\text{first}$ metric excludes problems solved due to pre-existing knowledge, thus providing a more accurate measure of the model’s learning efficiency. Under this condition, $P_\\text{first}$ can be formulated as follows:\n\n$$\nP_{\\text{first}} = \\min (i - i_0),\\ \\text{for } i \\geq i_0,\\ A^{\\text{feedback}}_i = 1,\\ A^{\\text{zero}}_i = 0\n$$\n\nwhere $ i_0 $ denotes the position of the first problem that the model answers incorrectly in the zero-shot setting. $ A_i $ is a binary indicator representing whether the model's answer at position $ i $ is correct (1) or incorrect (0).\nThe experimental results regarding $P_\\text{first}$ are shown in the below table: \n\n|Model|Cla|Ex|Sum|LR|MR|SR|Avg.|\n|---|---|---|---|---|---|---|---|\n|Doubao-1.5-Pro|3.58|3.71|3.54|6.75|3.24|3.06|3.98|\n|DeepSeek-V3|4.0|3.88|3.82|5.27|3.27|4.00|4.04|\n|DeepSeek-R1|4.5|2.76|4.7|4.7|3.62|4.19|4.08|\n|Gemini-2.5-Flash|3.0|2.76|4.08|6.55|3.4|3.18|3.83|\n|Claude-3.7-Sonnet|3.08|3.71|4.08|5.69|3.24|1.94|3.62|\n|Gemini-2.5-Pro|3.0|3.47|4.31|3.66|2.71|3.46|3.44|\n|Claude-3.7-Sonnet-Thinking|2.77|3.18|3.8|5.55|3.41|2.53|3.54|\n|GPT-4o|3.45|3.24|4.08|6.0|2.41|3.29|3.75|\n|Qwen2.5-32b-Instruct|3.69|3.35|3.93|6.38|3.65|3.53|4.09|\n|Doubao-1.5-Thinking-Pro|3.83|2.88|4.9|3.44|3.67|4.81|3.92|\n|OpenAI-o3-mini|3.42|3.71|2.6|2.93|3.4|3.62|3.28|\n\nFrom the table, we can draw the following conclusions:\n\n(1) After excluding problems that the LLM could already solve in the zero-shot setting, the average position of the first correct solution in sequential learning is relatively late for frontier LLMs. This indicates that these models face challenges in leveraging past problem-solving experiences to solve new problems.\n\n(2) There are differences in the $P_\\text{first}$ across different task types. For example, LLMs generally achieve the first correct solution earlier in classification and extraction tasks compared to other tasks. This suggests that current LLMs may find it easier to summarize and utilize experience in these discriminative and extraction tasks.\n\n(3) The long chain-of-thought reasoning paradigm provides a slight improvement in the learning efficiency of models. Most thinking-based LLMs show slightly better learning efficiency compared to their non-thinking-based counterparts within the same series.\n\n(4) Finding 5 in our manuscript remains unchanged: The position of the first correct solution varies significantly across models and tasks, and does not align with overall accuracy. This highlights that EvaLearn assesses a novel aspect of models’ dynamic learning potential, distinct from their static capabilities. EvaLearn encourages the research community to further investigate the dynamic abilities of models, promoting the development of more powerful and human-like LLMs.\n\nIn summary, the $P_\\text{first}$ under the new conditions decouples the model’s static capability from its dynamic learning efficiency, allowing readers to better understand the dynamic learning potential of current LLMs. We will include these new experiments, findings, and further analyses in our latest manuscript to make the paper more comprehensive and robust.\n\n\nFinally, we sincerely appreciate the reviewer’s recognition of our work, careful review, and valuable suggestions. We will **incorporate all additional experiments and revisions** into our latest manuscript. If you have any further comments, please do let us know, and we will do our best to address them."}]

Author rebuttal:

Meta-review: The paper introduces EvaLearn, a benchmark to measure within-task learning of LLMs via sequences of related problems across six task families. Models are evaluated sequentially within every sequences from 648 challenging problems arranged into 182 sequences spanning six task families, and evaluated automatically using instance-level rubrics and LLM-as-a-judge. Ultimately, the model are then evaluated using five proposed metrics of overall accuracy, slope of position-wise accuracy curve, position of first success, longest correct streak, and post-warm-up accuracy, to quantify learning capability and efficiency. The author conducted wide range of analysis, and interestingly, static leaderboard strength is only weakly correlated with sequential learning skill.

The work is novel, timely, methodologically sound, and addresses the key gap of current evaluation paradigm. Rebuttal additions convincingly resolve the main methodological concerns (statistics, insights, metric robustness, protocol clarity, judge reliability). Overall, I believe the contribution of this work would be meaningful and interesting to the community.

__Strengths__
* Evaluation shift from static competence to within-task learning, is novel and timely
* Benchmark is high-quality and evaluation methods are meaningful
* Analysis are thorough
* Paper is well-written and organized

__Weaknesses__
- The rebuttals have addressed most—if not all—of the weaknesses and concerns.
Decision: Accept (poster)