Title: SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM-Based Agents

Self-Evolution

Multi-Step Reasoning
Abstract: Large Language Model (LLM)-based agents have recently shown impressive capabilities in complex reasoning and tool use via multi-step interactions with their environments. While these agents have the potential to tackle complicated tasks, their problem-solving process—agents' interaction trajectory leading to task completion—remains underexploited. These trajectories contain rich feedback that can navigate agents toward the right directions for solving problems correctly. Although prevailing approaches, such as Monte Carlo Tree Search (MCTS), can effectively balance exploration and exploitation, they ignore the interdependence among various trajectories and lack the diversity of search spaces, which leads to redundant reasoning and suboptimal outcomes. To address these challenges, we propose SE-Agent, a Self-Evolution framework that enables Agents to optimize their reasoning processes iteratively. Our approach revisits and enhances former pilot trajectories through three key operations: revision, recombination, and refinement. This evolutionary mechanism enables two critical advantages: (1) it expands the search space beyond local optima by intelligently exploring diverse solution paths guided by previous trajectories, and (2) it leverages cross-trajectory inspiration to efficiently enhance performance while mitigating the impact of suboptimal reasoning paths. Through these mechanisms, SE-Agent achieves continuous self-evolution that incrementally improves reasoning quality. We evaluate SE-Agent on SWE-bench Verified to resolve real-world GitHub issues. Experimental results across five strong LLMs show that integrating SE-Agent delivers up to 55% relative improvement, achieving state-of-the-art performance among all open-source agents on SWE-bench Verified.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SE-Agent, a framework that enhances LLM agent performance by treating their reasoning paths as \"trajectories\" for optimization. The system employs evolutionary operation to iteratively improve a pool of trajectories and discover superior solutions. Validated on the SWE-bench benchmark, the method demonstrates significant performance improvements over baseline models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The empirical results are strong, showing substantial relative improvements over established baselines on the challenging SWE-bench benchmark. The method appears to generalize well across multiple open-source and closed-source LLMs.\n\nWeaknesses:\n1. The idea of collecting a pool of trajectories or experiences to guide or improve subsequent reasoning attempts is a well-established concept in agent learning, seen in frameworks like ExpeL[1] and AvaTar[2], which also learns from diverse, past experiences to improve agent performance. The \"self-evolution\" framing appears to be a new wrapper for a familiar concept, and its distinction from prior work in experience-based improvement or iterative refinement is not sufficiently articulated.\n2. The empirical evaluation, while strong, is narrowly focused on a single, highly specific benchmark: SWE-bench.  This makes it difficult to assess the generality of the SE-Agent framework. Results across a wider range of tasks are needed to demonstrate that this isn't just a bespoke solution for fixing GitHub issues. Its applicability to other code-related benchmarks like TerminalBench for terminal coding, LiveCodeBench for competitive programming.\n3. The paper lacks the concrete, low-level details needed for a deep understanding and confident replication of the method. The descriptions of the Revision, Recombination, and Refinement operators remain at a very high level. For example, how exactly does the\nCrossover function identify \"high-performing trajectory segments\" and combine them? What does the output of a Mutate operation actually look like?\n4. The case study presented in Figure 6 is too vague to be truly insightful. It claims that traditional agents get stuck editing `_validation.py` while SE-Agent discovers the root cause in `multioutput.py`, but it fails to show how the evolutionary process enabled this discovery. A step-by-step walkthrough is needed to illustrate the point.\n5. The proposed method generates an initial pool of ten trajectories and then iteratively evolves them. This process could be expensive in terms of number of API calls, tokens, and wall-clock time. The analysis in Figure 5, which plots performance against \"Maximum Cost,\" is insufficient as the \"Cost\" unit is not defined. A transparent discussion of its resource overhead compared to baselines is required.\n\n[1] ExpeL: LLM Agents Are Experiential Learners. AAAI 24\n\n[2] AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning. NIPS 24\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed an inference-time scaling framework, SE-Agent, for coding tasks. The motivation of this paper is to utilize the interdependence among various trajectories. Specifically, the SE-Agent framework uses the pilot trajectories with operations of revision, recombination, and refinement. For revision, the SE-Agent uses self-reflection to identify the critical point of the trajectory. For recombination, the SE-Agent combines the different trajectories based on their strengths. For Refinement, the agent optimizes the trajectories by improving the efficiency. In the experiment, the SE-Agnet achieves good performance (61.2 pass@1 with backbone model of Calude 3.7 Sonnet) on SWE-bench verified.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghth\n- The idea is simple but effective. \n- The paper is well-written with a clear structure and well-articulated mathematical formulations (Section 4)\n- The performance on the widely known SWE-bench verified benchmark is strong and demonstrates the performance gains achieved by the proposed framework. The ablation study of each component (Figure 3) provides a clear comparison between the revision and recombination parts.\n\nWeaknesses\n- Lack of inference time comparison. In line 98, the statement is \"our work introduces a novel approach within the ReAct paradigm that incorporates strategic reflection and mutation at critical steps, combining multiple trajectories to generate optimized execution paths **without** requiring extended computation time like Test-Time Scaling techniques.\" However, I did not find the inference time related experiment in the following sections.\n\nQUESTIONS:\n- How is the computation cost and inference time for the proposed method? This is not critical, but it would be good to include it to show the statement on lines 98-100.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SE-Agent, a so called ''self-evolution'' framework for improving multi-step reasoning in LLM-based agents.  SE-Agent introduces a trajectory-level optimization process featuring three key operations: revision, recombination, and refinement. Some of these mechanisms enable the agent to explore and generate solution paths for complex tasks. The framework is evaluated on the SWE-benc demonstrating  performance gains across five LLMs and outperforming strong open-source baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper thoroughly motivates the need for more diverse and optimized multi-step reasoning in LLM agents. \n\n- The three-step mechanism—revision, recombination, and refinement—constitutes a non-trivial advancement over existing trajectory sampling and MCTS approache.\n\nWeeknesses\n\n- I am somewhat confused about how the use of the term ''self-evolution paradigm.'' From my reading, the approach mainly involves generating different sampled trajectories rather than any genuine form of evolution as typically ablity improvement. Therefore, I suggest that the authors reconsider the terminology here. \n\n- One of my main concerns is the soundness of some of the experimental results. If I understand correctly, in Figure 2, the results for SE-Agent are based on Claude-3.7-Sonnet, while the other baselines use Claude-3.5. This raises questions about whether the comparisons are truly fair.\n\n- The ablation study only investigates the impact of the Revision and Recombination modules, but does not include an ablation for the Refinement operation. This omission experiment and explanation makes it difficult to fully understand the contribution of each component to the overall performance.\n\n- Some important experimental details are not clearly described in the current manuscript. For example, the process for generating ''pilot trajectories'' is not explicitly explained in the Experimental Setup section. Furthermore, when comparing with other baseline methods, it is unclear whether these ''pilot trajectories'' or similar auxiliary information are also made available to the baselines.\n\nQUESTIONS:\nQuestions refer to the Weaknesses.\n\nTypo:\n\nline 120: Objetive of Agent -> Objective of Agent\n\nLIMITATIONS:\nWhile certain potential limitations may be implicitly touched upon in the Conclusion, the discussion is not explicitly outlined or clearly labeled in the main text or additional section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper proposes a self-evolutionary pipeline for constructing better trajectories based on the previous ones. The pipeline consists of 3 steps: revision (revise single trajectory), recombination (combine multiple trajectories) and refinement (enhancing trajectories via rewards). The authors evaluated an agent with this pipeline on SWE-bench Verified, and obtained better performance than the baselines. Ablation study showed that each step contributed to the improvement. Compared to existing approach like MCTS, the proposed method tried to use cross-trajectory information to improve the agent.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The proposed idea is interesting which tries to utilize and improve the previous trajectories.
2. Experiments and ablation study showed the agent with this method obtained better performance.

Weaknesses:
1. Many of method descriptions (the 3 steps) are only at high-level and lack specific details. For example, it's not clear what Mutate(), Reflect(), Revise(), Crossover(), Transfer() and Restructure() really do under the hood.
2. It is not clear whether groundtruth label is being used to select the trajectories. In subsection 4.4.1, the paper mentions auto eval based on test pass rate is being used to compute the reward and select the trajectories. Isn't this a leak of the groundtruth label?
3. No examples are given for how each step revises / combines / refines the trajectory. Including some examples can help improve explain the methods.

QUESTIONS:
Please see "Strengths And Weaknesse" section.

LIMITATIONS:
I did not find discussion on limitations in the paper.
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03