Title: EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
Venue: NeurIPS 2025
Accepted: yes
Keywords: benchmark

LLMs

evaluation
Abstract: We introduce EvaLearn, a pioneering benchmark designed to evaluate large language models (LLMs) on their learning capability and efficiency in challenging tasks, a critical, yet underexplored aspect of model potential. EvaLearn contains 648 challenging problems across six task types, grouped into 182 sequences, each sequence dedicated to one task type. Diverging from most existing benchmarks that evaluate models in parallel, EvaLearn requires models to solve problems sequentially, allowing them to leverage the experience gained from previous solutions. EvaLearn provides five comprehensive automated metrics to evaluate models and quantify their learning capability and efficiency. We extensively benchmark nine frontier models and observe varied performance profiles: some models, such as Claude-3.7-sonnet, start with moderate initial performance but exhibit strong learning ability, while some models struggle to benefit from experience and may even show negative transfer. Moreover, we investigate model performance under two learning settings and find that instance-level rubrics and teacher-model feedback further facilitate model learning. Importantly, we observe that current LLMs with stronger static abilities do not show a clear advantage in learning capability across all tasks, highlighting that EvaLearn evaluates a new dimension of model performance. We hope EvaLearn provides a novel evaluation perspective for assessing LLM potential and understanding the gap between models and human capabilities, promoting the development of deeper and more dynamic evaluation approaches. All datasets, the automatic evaluation framework, and the results studied in this paper are available in the supplementary materials.


Peer reviews:
comments: SUMMARY:
This paper introduces EvaLearn, a novel benchmark for evaluating large language models' learning capability and efficiency. Unlike existing parallel evaluations that test models on i.i.d. samples, EvaLearn uses sequential problem-solving where models can learn and adapt within a given task sequence. The benchmark contains 648 problems across 182 sequences spanning six task categories. This work addresses a key gap in current evaluation by measuring LLMs' learning ability rather than static ability.

STRENGTHS AND WEAKNESSES:
Strengths
- The paper is well-written and organized.
- This work presents a novel benchmark to evaluate LLMs' learning ability, which remains unexplored in existing evaluations, and I believe it is of high significance to the research community.
- The whole benchmark is of very high quality, with carefully curated data, well-grounded evaluation methods, and sophisticated and meaningful rubrics.

Weaknesses
- There are no obvious weaknesses.

QUESTIONS:
- The paper presents a comprehensive benchmark of nine frontier LLMs, but there's a notable omission of any model from Google Gemini.
- The author mentions that the metric "(3) Average position of first correct solution" is proposed to measure "initial learning speed". However, a model with superior static capabilities might solve the first problem in a sequence correctly simply due to its pre-existing knowledge, without any "learning" taking place.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces EvaLearn, a new benchmark that probes dynamic learning in large language models (LLMs). Instead of the standard i.i.d. “parallel” tests, EvaLearn presents 182 short sequences (7 items each, 648 problems total) spanning six task families (summarization, classification, extraction, logical, mathematical and sequential reasoning). Models solve each sequence step-by-step, optionally receiving demonstrations or rubric-based feedback after every attempt. Five automated metrics (overall accuracy, slope of position-wise accuracy curve, position of first success, longest correct streak, and post-warm-up accuracy) quantify both learning capability and learning efficiency. Experiments with nine frontier models (thinking-based vs. non-thinking) reveal large variance: some models (e.g., Claude-3.7-Sonnet-Thinking, o3-mini) learn rapidly, others regress, and static leaderboard strength is only weakly correlated with sequential learning skill.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Shifts evaluation from static competence to within-task learning, a dimension mostly absent from prior LLM benchmarks or agent arenas.\n\n2. Provides a concrete, reproducible suite plus automated rubric-plus-LLM judging (≈95 % agreement with humans) that could become a standard for meta-learning research. \n\n3. Careful experimental design: 648 hand-crafted, rubric-verified items; comparison of zero-shot, few-shot, demonstration and feedback paradigms; results interpreted with clear statistics (e.g., slopes, warm-up curves).\n\n4. Paper is well-structured, with motivating figures (Fig. 1), metric definitions (Eqns 1-6) and ablation findings (Tables 2/8, Fig. 2-4) that make replication straightforward.\n\nWeaknesses\n\n1. Ground-truth judging depends on GPT-4o. Even with 95 % spot-checks, subtle rubric-matching errors or model-to-model bias may skew results; no cross-LLM agreement study is provided.\n\n2. Task set is still synthetic and modest (182 × 7). It is unclear whether the observed learning behaviours transfer to real-world domains (e.g., code, multimodal, open-ended dialogue).\n\n3. Sequential evaluation echoes earlier interactive / curriculum work (DROP-hard, AgentBench, REFLEXION). The paper could more explicitly position itself against those and justify design choices (why exactly 7 problems per sequence?).\n\nQUESTIONS:\n1. Have you tried an ensemble of judges or self-consistency to mitigate single-LLM bias? Reporting Cohen’s κ between two judges would strengthen confidence.\n\n2. Did you vary M (number of items per sequence)? Learning slopes might change if sequences are longer/shorter.\n\n3. How were problem difficulties equalised across tasks? Presenting per-task baseline human or smaller-model performance would contextualise scores.\n\n4. Any plan to add “in-the-wild” tasks (e.g., debugging a small codebase, multi-hop search) where learning from prior mistakes is crucial?\n\nLIMITATIONS:\nThe authors include a limitations section but should expand on data provenance and potential bias introduced by GPT-4o judging. No immediate societal-risk issues are apparent, but public release requires a clear license and privacy audit of any real-world text used.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new LLM benchmark EvaLearn. EvaLearn is different from other benchmarks in that it evaluates LLMs’ sequential learning capabilities. EvaLearn consists of sequences that incorporate multiple problems, and the LLM is instructed to solve the problems sequentially with LLM-as-a-Judge providing feedback to the previous problems. Therefore, an LLM with a strong learning capability should be able to learn to improve from the past feedback to solve the future problems better. Overall EvaLearn consists of 648 problems organized into 182 sequences, and covers six task types. The authors also conduct a wide range of analysis to understand the LLMs’ behavior on their benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- The motivation and the novelty of the benchmark is good. Oftentimes LLM chat assistants will not get something right in the first turn of conversation, which requires users to provide feedback. Evaluating how well can LLMs incorporate such feedback is an important aspect that is not well covered by existing benchmarks.\n\n- The metrics used by the paper are unique/interesting and fit the purpose of the benchmark well.\n\n- The paper is for the most part easy to read albeit it does have some minor ambiguities.\n\n### **Weaknesses**\n\n**False “Yes” in the Paper Checklist (L:1044, Experiment statistics significance)**\n- The authors do not report statistical significance of their experiments (which I believe is necessary for this work in two different cases as explained below), yet they put “Yes” in the checklist.\n\n**Concerns about many of the “Findings”.**\n\n1. **[Findings 1]** If I’m not mistaken the authors do not compare the performance of Feedback learning against Demonstration learning in the main sections, which I believe is necessary to claim that the LLMs truly learn from the LLM-as-a-Judge feedback (instead of solely from the few-shot demonstrations). Instead the authors compare Feedback learning against Zero-shot in Findings 1, which from a broader perspective is few-shot inference vs zero-shot inference, and in my opinion is not a fair setting.\n\n2. **[Findings 2 and 5]** Were the seven problems for each sequence randomly selected/ordered (this is not clear from the paper)? In that case the variance caused by the randomness could be a considerable factor for all of the experiments. In particular, I believe any analysis regarding $N_{consec}$ and $P_{first}$ will be affected significantly (Findings 2 and Findings 5) as these metrics will deviate a lot on whether easier/harder problems are grouped together or appear early. This would’ve been less of an issue if the sample size was large, but the benchmark only has 182 sequences. The authors should’ve conducted multiple runs across different random compositions of the sequences and reported the standard deviations.\n\n3. **[Findings 4]** The authors fit linear regression models and use the slope to claim that some LLMs achieve increasing accuracy with the number of demonstrations/feedbacks, but they do not report the statistical significance (e.g. p-values). Looking at Figure 4 and 17, in most cases there seems to be no noticeable correlation or trend in the graphs. It’s possible that in most cases the regression models are not statistically significant, therefore unable to support the authors’ claim.\n\n4. **[Findings 6]** The authors claim that Demonstration learning yields better results than Few-shot parallel solving, but isn’t that expected since demonstration learning overall has more in-context learning examples? In 71% (5/7) of the cases, the demonstration learning will have equal or more in-context examples than the few-shot parallel learning, which is fixed to three examples.\n\n\nWhile I believe that this paper provides an important and novel perspective to LLM benchmarks, the later half of the paper with the \"Findings\" has non-trivial concerns that the authors should address.\n\nQUESTIONS:\n- How do you prompt the judge model to give feedback for Feedback learning? In the Appendix only the prompt for grading is provided.\n- How did you construct the judge rubrics for the summarization task? Since unlike the rest of the task, summarization is relatively subjective.\n\n**Writeup/Style**\n- A bit more clarification on what the “Sequential Reasoning” task is would be nice (L:98).\n- Sentence in L:109 (This task…) and L:113 (These problems…) seems a bit redundant.\n- o3-mini is sometimes written with an upper case O and sometimes with a lower case o.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper introduces a benchmark to evaluate a LLM's learning capability and efficiency in challenging tasks. It contains 648 challenging problems arranged into 182 seven-item sequences spanning six task types (Summarization, Classification, Extraction, Logical Reasoning, Mathematical Reasoning, Sequential Reasoning).

STRENGTHS AND WEAKNESSES:
Strengths:
1. Original and novel idea. Significant to the community.
2. Thorough analysis and automatic-evaluation pipeline.

Weakness:
1. I am not sure if 648 problems are enough to evaluate the sequential learning task properly. 
2. To evaluate properly, there should be a range of difficultly level of these questions so that one can understand the true potential of the LLM for different difficulty levels.

QUESTIONS:
1. Could you provide a justification on why you think 182 sequences a good number to evaluate any model. Especially when each category has about 48 or 60 problems (13 or 16 sequences) only as show in table 1. In my opinion 13 or 16 is not a significant number to evaluate sequential learning ability of the frontier models.
2. From figure 3 (right) it seems that average is between 1-2. Which means that most examples sequences are predicted correctly in the first step only. As the models get better and better we need more challenging benchmarks to evaluate these models. In the existing results the models already perform well on the dataset. 
3. More data with difficultly rating as labels would have been very useful to evaluate a wide range of models with different capability and study how difficulty rating impacts it.

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-04