Large Language Model-based Data Science Agent: A Survey
Authors: Ke Chen
Peiran Wang
Yaoning Yu
Xianyang Zhan
Haohan Wang
Venue: Accepted by TMLR ()

Abstract: The rapid advancement of Large Language Models (LLMs) has driven novel applications across diverse domains, with LLM-based agents emerging as a crucial area of exploration. This survey presents a comprehensive analysis of LLM-based agents designed for data science tasks, summarizing insights from recent studies. From the agent perspective, we discuss the key design principles, covering agent roles, execution, knowledge, and reflection methods. From the data science perspective, we identify key processes for LLM-based agents, including data preprocessing, model development, evaluation, visualization, etc. Our work offers two key contributions: (1) a comprehensive review of recent developments in applying LLM-based agents to data science tasks; (2) a dual-perspective framework that connects general agent design principles with the practical workflows in data science.

Published in Transactions on Machine Learning Research (02/2026)
Large Language Model-based Data Science Agent: A Survey
Ke Chen University of Illinois Urbona- Champaign
kec kec Dec1O@rllinoii.co edu
Peiran Wang
whillbug@@mmail.com com
University of Ilimois Urbana- Chanpnng
Yaoning Yu
yyn0005000@mmallccom
University of Illinos Urbona- Chanapgghaa
Xianyang Zhan
zhan390illimois.. eduu
University of Ilinois Urbana- Channaaga
Haohan Wang
hachanw@rilimoois..d
University of Illinois Urbena- Caanaa--aapeignn
Reviewed on OpenReview: https: openrviiee. net/ forum? ? id= 2T5S.JNOCS
Abstract
The rapid advancement of Large Language Models (LLMs) has driven novel applications across diverse domains, with LLM-based agents emerging as a crucial area of exploration. This survey presents a comprehensive analysis of LLM-based agents designed for data science tasks, summarizing insights from recent studies. From the agent perspective, we discuss the key design principles, covering agent roles, execution, knowledge, and reflection methods. From the data science perspective, we identify key processes for LLM-based agents, including data preprocessing, model development, evaluation, visualization, etc. Our work offers two key contributions: (1) comprehensive review of recent developments in applying LLM- based agents to data science tasks; (2) a dual-perspective framework that connects general agent design principles with the practical workflows data science.
1 Introduction
In recent years, the rapid development of Large Language Models (LLMs) has driven significant innovations across various domains. Leveraging their remarkable capabilities in understanding and generating human- like text, LLMs have become foundational in creating intelligent agents capable of performing complex tasks autonomously. These agents have demonstrated substantial potential in diverse fields, including healthcare Qiu et al. (2024), finance Yu et al. (2024), education Zhang et al. (2025b), and software engineering Hong ct al. (0023).
Among these fields, data science has emerged particularly critical area. for applying L.M--based agents Sun et al. (2024b). Data science involves extracting meaningful insights from vast and diverse datasets, process that traditionally requires extensive manual effort and expertise. Consequently, LLM-based data science agents (DS Agents) have altracted attention for their ability to automate and optimize data analysis, model development, and decision-making processes.
A central goal of this survey is to clarify how LLM agents can be organized perform the functions of data scientist. This leads a sequence of increasingly detailed research questions. (RQ1) At the organizational level, how should agents' roles and responsibilities be defined so that complex analytical tasks
1 These authors contributed equally this work.

---

Published in Transactions on Machine Learning Research (02/2026)
can be effectively decomposed? (RQ2) Given these roles, how should each agent carry out its reasoning and tool use to ensure reliable execution? (RQ3) Execution usually depends on information beyond the agent's ' internal context. What external knowledge should agents access, and how should such knowledge be integrated into their decisions? (RQ4) Even with access to the necessary knowledge, agents still make mistakes, raising the question of what reflection and self-correction mechanisms are required detect and repair errors during operation. (RQ5) Finally, data science workflows are inherently iterative, prompting the question of how these agent-level capabilities can support end-to-end, loop-based processes rather than isolated single-step tasks.
To address these research questions, we examine LLM-based data science agents from two complementary perspectives: agent design and data science application. From the agent design perspective, we summa- rize key architectural dynamic agent generation-ane and analyze core components relevant to RQ1-RQ4, including how the agent structure is designed ($3.1), how the reasoning of LLM within agents is performed ($3.2). where the knowl- edge comes from ($3.3). and the reflection of the agent ($3.4). such agent roles, execution strategies, knowledge integration, and reflection mechanisms. From the data science perspective $4, we explore how LLM agents are applicd across major workflow stages such as data preprocessing. modeling, evaluation, and tasks,
billimms-inccudnng single-agent systems, collaborative multi-agent structures, and
visualization. We also outline common task types, including model-building and and characterize the iterative nature of the data science loop. This perspective directly informs RQ5 by illustrating how LLM agents support each stage an end to end data science workflow. Additionally, our survey goes beyond mere documentation by synthesizing insights from recent studies to identify rescarch opportunities and future directions in this evolving field [ahuu et al. (2024); Fan al. (2023
In summary, this survey makes two primary contributions.
It provides comprehensive review of recent efforts to apply LLM-based agents to data science tasks, synthesizing work across arcas such data preprocessing, modeling, evaluation, and visualization.
It proposes a persppeeeee framework that bridges the gap between general agent design princi- ples-suce such as role allocation, execution, and reflection-and the specific operational needs of data science workflows, offering a structured lens understand and develop LLM-baseh data science systems.
2
Related Works
Recent surveys on LLM-based multi-agent systems have introduced various taxonomies from architectural, task-specific, and coordination perspectives. We briefly review them and highlight how our approach differs.
2.1 Modular Architectures
Existing surveys often decompose perception, and action. For example, Guo et al. (2024b) identifies components like agent-environment interface and capability acquisition; similar structures appear in Liu et al. (2024a). Sun et al. 2024bb and
agents into functional modules such planning, memory,
others.
Our work adopts this modular view but anchors the context of data science workflows. Instead of listing capabilities, we emphasize how modules interact during task execution e.g., how reasoning, planning, and knowledge access are coordinated in dynamic static execution ($3.2), and how external knowledge sources are integrated into decision-making ($3.3) We also highlight how modularity supports runtime adaptation through reflection mechanisms( 3.4 where agents adjust behaviors based on feedback, errors, or performance comuols-nabbiin dynamic coordination across modules.
2.2 Collaboration and Communication
Agent collaboration commonly categorized by structure (contralized, decentralized) or mode (cooperation, competition, etc.), as seen in Tran al. (2025). Related works Guo et al. (2024b). Li et al. (202gg) discuss role-based layered designs.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Modular Architectures 8
Related Works
-
Collaboration and Coumunication
Capability Evolusion and Redrectio
Single Agent
Two Agents 5.1.2
Agent Role 8
Multiple Agents
Dynamic Ageats $.1.4
Summary of Agent Role 3.1.5
Dynamic Execution Planning
3.222
Exceution Structure
Fixed Workflow
Agent Perspectives 3
Summury Execution Strueture 32.
External Knowlodge Methods 3.3.1
External Knowledlge
Summary of Exterraal Knowlodge 3.3.2
I.-M-based Agents DS
Rellection Techniqus
Refioction
3.4
Summary Rellection
Taoks 9
Data Pregmose
4:211
Stadistioal Compatation
1.22
Peature Engineering
223
DS (Data Science) Perspectives $
D88 Loop 8
A.2
Model Training 124
Exaluation $
125
Visuxlisation $
2.6
Summary of DS perspectives 4.
Benchmarks $
Research Opportunitics c
Figure 1: Structure of This Survey
However, these are mostly static views. In our work, we also examine dynamic orchestration, where agents adaptively coordinate by adjusting roles or workflows during execution. This includes settings where task allocation evolves based on feedback, or agents are restructured at runtime to respond to changing demands ($3.1.4 3.2
2.3 Capability Evolution and Reflection
Several surveys recognize that agent systems can adapt via feedback, memory updates, reflection for example, Li et al. (2024g) includes an evolution phase with memory consolidation, and Wang et al. (2024a) discusses reflective planning. Similar notions appear in Guo et al. (2024b) and others.
In contrast to treating reflection as an auxiliary feature, we emphasize it as a cross-stage mechanism that drives dynamic exccution adjustments ($3.4). We outline three key dimensions: the driver, level of impact), and adaptability of reflection, framing reflection as a central control process for progress monitoring and adaptive behavior.
3

---

Published in Transactions on Machine Learning Research (02/2026)
developer metadata
web search
api
calls
multi-aged
mcts
function calling
case-based
reasoning
single
agent
base
checkpoint
progra
rewir
orchestration
debugging
error
handl
ing
react t
rag
test
sel
cases
self-ref
performance metric
knowledge
search engine
p
paanerr
task decomposition
coder
Alamainder
unit
unit
tests
r
task graph
executor
human
feedback
hunan-in-the-
Inoelagge graph
eetrivaan
tool calling
retriever
retrieval
augmented
generation
reviewer
ref r leffecioonoomaii carlo tree search
knowledge
Figure 2: Keyword cloud summarizing the most frequent concepts across the surveyed LLM-based data science agent papers. Larger words represent higher occurrence frequency. The colors in the cloud correspond to the major analysis dimensions discussed in this survey; red words represent agent role design ($3. 1. blue
words represent execution structure ($3.2), green words represent external knowledge integration purple words represent reflection mechanisms ($3.4..
33.3 and
3 Analysis from Agent Perspective
LLMs provide strong linguistic and generative capabilities, but they exhibit structural limitations that be- come particularly evident in data science workflows. First, LLMs are prone to hallucination, e.g., fabricating column names, statistical results, entire processing steps. Second, LLM-generated code is brittle. Even when logically correct in natural language, produced scripts frequently break due to issues such type in- consistencies, missing dependencies, unstable pipelines, non-reproducible execution paths. Third, LLMs struggle with morizorizon tasks. Their limited ability to retain and manipulate extended context leads to goal drift, forgotten assumptions, and inconsistencies across multi-step analytical processes. These limita- tions show that raw LLMs are not reliable as end-to-end data science systems and need structured agent architectures provide the capabilitics they lack and to address common failure modes.
Motivated by these limitations, recent work has increasingly turned agent architectures that introduce structure, modularity, and external support, enabling LLMs to compensate for their inherent limitations. For example, grounding and verification mechanisms help reduce hallucination, controlled execution improves reliability, and explicit role decomposition mitigates long-context instability. Agent systems thus do not merely wrap LLMs they provide the organizational scaffolding needed to transform a general-purpose model into a dependable data science practitioner.
Seen from this perspective, our analysis focuses on four design dimensions: agent role design 3.1 execution
structure ($3.2), external knowledge integration ($3.3,, and reflection mechanisms ($3.4). They form structured solution to the inherent limitations of LLMs in data science settings. The remainder of this section examines each dimension in turn, showing how existing systems operationalize these design principles in practice.
3.1 Agent Role Design
In this section, we discuss the role design of LLM-bbasod agents, focusing on their agent role specification. Starting from single-agent designs, which manage all tasks independently, we summarize the transition two-agent systems that introduce role separation, such planner-executor and coder-neviewer frameworks.
4

---

Published in Transactions on Machine Learning Research (02/2026)
Agent Role
4.2 Execution Structure
OO
Code
Role: Coder You are a coder agent, you prompting to write code
Role: Reviewer
Role: Analyzer You are a Andlyzer agent, you prompting to run code to analyze..
Execute
You are reviewer
Coder
Reviewer
agent, you prompting to review and revise code
Comments
Integrate
4.3 External Knowledge
4.4 Reflection
OO
Reflect
Analyzer
Code
Search Engine
Database
Error
Log
Figure 3: We illustrate the basic components for current data science agents: 1) agent role; 2) execution structure; 3) knowledge and 4) reflection.
Furthermore, we conclude different multi-agent systems, covering software engineering style systems, mini- mum function agents, etc. Lastly, we introduce some dynamic agent role design frameworks where agents are generated adaptively rather than predefined.
3.1.1 Single Agent
Reasoning files, ...
Output Nome, Moth, Science, Engliishh 85,98,88 Bob,75,80,76
write analysis
davo.csv, chich
System Prorpt
Action
are a dota science agent. You can call tools: tenninal, python,
eche import pondos as pd
Figure The basic structure for single agent structure, with only the agent and execution environment.
The single-agent design is the simplest design structure as shown in Figure 4 Most existing single-agent
works directly follow the ReAct design (a prompting technique where LLMs are used to generate both
reasoning lraces and task-specific actions) Freimanis & Andersson Rhodin 22024 Chen et al. (2024a] Sun
et al. 2024a) Jing et al. (2024); Hu et al.
(2024a) : Deng et al. (2024); Le et al. (2023); Zhang et al. (2024a)
Gupta et al. 2024 (02d et al. (2025 Xu et al. (2025). where the single agent will perform thought,
action, and observation processes iteratively on its own.
Single-agent designs offer minimal overhead and are easy deploy, making them attractive for low-latency low-cost settings. However, since all reasoning, memory, and execution concentrated in single unmitigated, generated code is fragile,
model, they inherit most of the LLM limitations: hallucinations and long-horizon tasks often suffer from goal drift and lost context. As result, single-agent systems are often insufficient for complex or multi-stage workflows that require long-horizon reasoning, consistent state tracking, or robust code execution.
3.1.2 Two Agents
Evolving from the single agent, the two-agent design is introduced to decompose the single agent's functions, dividing the functions into two specialized roles, particularly either a plannerkexecutor or code&revvewerr structure.
Planner and executor. Different from the single agent discussed previously, some works split the single agent into two agents Huang al. (2024b): Liu et al. (2024b); Zhang et al. (2023b): Chi al.) (2024): Li et al. (2024e) Wang al.] (2025a). planner and executor. an example shown Figure 5 the planner will get observations from previous execution results users' requests, and generate next-step plan,
5

---

Published in Transactions on Machine Learning Research (02/2026)
Output: Name, , Scienhe,, Alice, 85, ,888888 8ob,75,88,77
Plan: Please try to write and execute a script to read the data.csv
Planner
Command: echo
df pe..eed_cssvdatt.cc httppcccscaaacaae
AI
<
analyze.py
Environment
Executor
Figure 5: An example of planner and exccutor agent structure, where the planner generates plan for the executor to execute in detail.
whole plan advance. Then, the executor is prompted interact with the environments and will follow the generated plans to execute.
Code
import numpy np
Train model didel.ffitx_trrana y_train, epochs=10, batch_size-32)
Comments
Coder
There is no validation set the code. for example:
Reviewer
model.fit(xxtrrall y_train, epochs-10, batch_size=32, vol volidation_splite....
In the coder&reviewer structure, the coder generates the code, while the reviewer will make comments to revise the code for the coder.
Figure 6:
Coder and reviewer. Another type of two-agent structure is the "coder and reviewer" style Trirat et al. (2 Huang et al. (2023) as shown in Figure 0 In such a design, the coder is responsible for completing tasks following the typical ReAct structure. Another agent, the reviewer is introduced to check the validity of the code generated by the coder, [Trirat et al. (2024) allows the reviewer to check the generated response at each step of generation, while Huang al. (2023) only allows the reviewer to function the end of the entire generation process.
Two-agent designs improve over single-agent systems by splitting responsibilities into complementary roles. Planner exeeutrr structures provide more structured task decomposition, and coder-reviewer structures introduce lightweight verification, catching some errors or hallucinations that a single agent would miss. However, the approach remains sensitive to failures such as planner hallucinating an incorrect plan or a reviewer failing identify subtle logical errors. Communication cost. also increases, and the reliability gains depend heavily on how well the two roles align. As result, two-agent systems offer a moderate balance between simplicity and robustness, but their effectiveness is bounded by the underlying LLM's consistency and self-correction ability.
3.1.3 Multiple Agents
Multi-agent systems enhance problem-solving capabilities by enabling collaboration among multiple agents, each with distinct roles and expertise. In such systems, agents are assigned specialized responsibilities, allowing them to focus different tasks while exchanging progress and information.
Software engineering-sty/ team. The Software Engineering (SE) team-style agent design draws inspira- tion from traditional human software development teams Qian et al. 2024 Zhao et al. (2024b); Trirat et al. 2024h; Hong el al. (2023); ; Tao et al. (2024): Lin el. al. (2024) Nguyen el al. 2024). In an SE team-style framework, agents are typically assigned roles that correspond to key human roles in software development, here we state some example roles: The product manager defines the product vision and organizes require- ments. The requirements analyst translates user needs into detailed software specifications. The scrum master facilitates task planning and sprint coordination. Hierarchical roles like team leader, module Leader,

---

Published in Transactions on Machine Learning Research (02/2026)
and function coordinator handle task decomposition at varying levels of gramularity. The executor for the task is the developer, who is responsible for implementing the code, while the senior developer refines it. Finally, the QA engineer ensures quality through rigorous testing, and the tester validates specific func- tionalitics. Some SE team-style agents also perform hicrarchical roles which allow some agents with high permissions to manage others.
AI
File: /src/utils.py
def func():
Line: 42
Code: def
async def func():
Code Searcher
Code Generator
try: print(array(ii) except IndexError:
IndexError
File: /src/main.py
Line: 23
Exeception Handler
Exeception Locator
For agents with minimum functions, each agent is only responsible for a minimum function, such as search code, run code, etc.
Figure 7:
Minimum function agents. Minimum function agents are designed to handle narrowly scoped and atomic lasks separately, shown in Figure] 7
Across existing frameworks, minimum function agents are given very small and specific functions to handle. For instance, code search agents locate relovant files, classes, or methods within a repository, while fault localization agents identify buggy code sections using debugging Lechniques like spectrum-based fault local- ization Zhang al. (2024e). Other agents specialize in generating patches to fix identified issues, executing tosts to validate code correctness,
systematically building repository structures from high-level descrip- tions Zan et al. (2024] Arora et al. (2024) In the context of exception handling, some agents detect fragile code, identily exception types, and implement robust handling mechanisms to enhance code reliability /Zhang
et al. 2024b). All works emphasize task decomposition and modularity, with outputs from one agent often
serving as inputs for another, forming structured and collaborative workflows Zan et al. 2024 Phan et al.
(0244;
Arora el al. (2024 el. al. (2025): You et al. (2025): Ou al. 2025
AT
Task Planning
Task Scheduling
Central Controller/Server
Client agent manage
AI
AD
AI
AI
Client A
Client A
Client A
Client
Client
Figure 8: In the client-server agent structure, normally there will be a central server controls all the other clients.
Client-server design. Client-server agents, adopt a hierarchical architecture where central controller agent manages and coordinates the operations multiple specialized client agents Zhang et al. 2023a Vang et al. (2024a): Gandhi al. (2024): Zhao al. (2024a).
The controller or server agent functions as project manager, planning entire workflows of the given tasks, and allocates the tasks to the client agents. Client agents perform roles like software engineers testers, focusing on executing specific subtasks such as data analysis, modeling, feedback generation. These roles allow for clear separation of tasks, while also enabling dynamic adjustments based on task requirements or
external feedback Zhang et al. (2023a): Bai et al. (2024); Zhao et. al. 2024a Shen el al. 2024).
Multi-agent architectures offer the highest scalability and strongest resistance to long-context forgetting, as each agent operates within narrowly scoped role. They can achieve high reliability through division of labor, modular workflows, and cross-agent. verification. Yet. the increased number of agents introduces

---

Published in Transactions on Machine Learning Research (02/2026)
coordination overhead, susceptibility to miscommunication, and longer execution latency. These systems are powerful but harder to control, and their effecti veness depends on careful manual workflow design.
3.1.4 Dynamic Agents
Dynamic agents represent a class of multi-agent systems that emphasize adaptability by dynamically creating, modifying, or expanding agents during runtime. Unlike static agents, which follow predefined workflows and configurations, dynamic agents are designed respond the complexity or variability of tasks by adjusting the agents' internal structure (prompt, etc.) introducing agents (Hu al.) (2024b); [shibashi & Nishimura (2024). Current frameworks for dynamic agent creation adopt two primary paradigms:
AI
Controller
Al
Layer 1
Al
AI
AI
Al
Layer 2
Figure 9: Hierarchical agent generation.
Hierarchical agent generation. This paradigm involves parent agent or high-level controller (e-g., Mother Agent in SoA Tshibashi & Nishimural (2024)) that decomposes complex tasks into subtasks and creates child agents handle the subtasks. Each child agent operates independently on its specific subtask. It is particularly effective in managing tasks with clear functional divisions, such as modular code generation or system-level software design Ishibashi & Nishimura (2024).
Former Agent Workflow
Failure Reason Analysis
Reflected Agent Workflow
AI
>>>
Code
Erv
Error
Andlyzon
Updator
Figure 10: Iterative agent generation through foedback.
Iterative agent generation through feedback. Dynamic agents adjust their structures and behaviors iteratively based on real-time feedback from their environment or other agents (Figure TO]. EvoMAC Hu et al. (202bb) exemplifies this paradigm, employing a collaborative rather than hicrarchical approach, where agents refine their outputs and workflows through mechanisms analogous to backpropagation such as a loss agent computing crrors and an update agent adjusting agent workflows accordingly. This textual feodback enables the dynamic creation or reconfiguration of agents, supporting adaptability for tasks with evolving requirements, such changing software specifications (Figure 10 Hu et al. (2024b).
Dynamic agent generation offers high flexibility by creating agents on demand rather than rolying on manually designed roles, prompts, and pipeline. This allows the system to adjust its structure at runtime and better accommodate diverse or evolving data science tasks. However, dynamically spawning agents also amplifies LLM uncertainty, reduces reproducibility, and increases computational and coordination overhead. As a result, these systems provide strong adaptability but exhibit lower predictability and stability, making them challenging to deploy in stable production settings.
8

---

Published in Transactions on Machine Learning Research (02/2026)
3.1.5 Summary of Agent Role
Beyond specific agent role design, we summarize the key features of agent roles, including agent structure, agent relationship, role lask allocation, and lask granularity. These key features reveal the core thoughts through the design of agent roles.
Framework RuvoCodeRover
CS
FL
PG
VA
RI
ED
EH
v
V
CODES
HYPERAGENT
V
MASAI
V
Sceker
Table 1: This table summarizes the roles Minimum Function Agents in different frameworks. The columns represent specific functions: Code Search (CS), Fault Localization (FL), Patch Generation (PG), Validation
(VA), Repository Initialization (RI), Exception Detection (ED), and Exception Handling (EH). A checkmark indicates that the framework supports the corresponding function.
Role
PM
RA
AR
SM
TL
ML
FC
DE
SD
QA
TE
AgileCoder
V
AutoMMI-Agent
ChatDev
FlowGen
V
v
MetaGPT
V
MAGIS
VisionCoder
Table 2: Summary of SE Team Roles in Agent Designs: Product Manager (PM), Requirements Analyst (RA), Architect (AR), Scrum Master (SM), Team Leader (TL), Module Leader (ML), Function Coordinator (FC), Developer (DE), Senior Developer (SD), QA Engineer (QA), Tester (TE).
Agent structure. The structure of LLM-based agents can be categorized into several types, each with its advantages and challenges:
1. With manager: In this structure, central agent manages and controls all agents. Software engineering- style agents and Client-server agents mainly have this structure. For example, in the AutoML-GP MAMoML-GPTTriraall et al. (2024) framework, central LLM serves as the controller, managing the entire pipeline by integrating specialized agents for subtasks such as model design and hyperparameter tuning.
2. Without a manager: Each agent operates independently and solves tasks autonomously. Minimum func- tion agents mainly pose this structure, since all the agents with minimum function share the same position. An example of this the MASAT Arora el al. (2024) framework, which utilizes decentralized agents that collaborate on machine learning and data science tasks by sharing results but central management system.
3. Hierarchical managers: A higher-level agent controls lower-level agents in layered structure. Hierar- chically generated agents and part of software engineering style agents mainly pose such structure. An example of this can be found in Hierarchical agent generation, such as in EvoMAC Hu et al. (2024b). where a parent agent dynamically creates child agents to handle specific subtasks during runtime.
Agent relationship. The relationship between agents within the system can vary significantly depending on the design philosophy:
1. Compete: Agents work against each other complete task, often in the coder-reviewer paradigm (similar the adversarial concept in GAN). In frameworks like MASAI Arora et al. (2024), agents engage in competitive strategies to address machine learning and data science challenges. The competition

---

Published in Transactions on Machine Learning Research (02/2026)
Dimension
Single Agent
Two-A Twwo-Agent
Multi-Agent
Dynamic Agents
Reliability
Scalability
Coordination Cost
Predictability
Stability
Industrial
Quick analyses, ad-hoc queries, and low-risk automation
Medium-scale workflows requiring basic verification and predictable execution
Production pipelines with modular stages, quality checks, and stable data dependencies
Exploratory analytics, quick pipeline prototyping, changeable environment
Applicability
Table 3: Trade-off summary across agent role designs.
strong,
moderate, O= weak.
between reviewers and coders allows the iterative refinement of the code. Some works also allow multiple agents to propose multiple plans to compete.
2. Collaborate: Agents work together toward a shared objective. For example, the MAGIS Tao et al. (2024) framework, agents assume different roles like Manager, Developer, and QA Engineer collaborate resolving GitHub issues. Their tasks are divided to ensure modular development, with continnous collaboration between agents.
3. Hybrid:
Agents alternate between competing and collaborating based on the task requirements. For instance, in AutoCodeRover Zhang et al. (2024c). agents work together to localize faults and generate patches, but may compete in terms of optimizing solutions or strategies based on the specific issue at hand.
A Agent role task allocation. LLM-based agents can allocate tasks in either a static or dynamic manner:
1. Static Task Allocation: In some systems, agents are assigned a fixed set of tasks that they perform. For example, in the Data Director Hong al. (2024) framework, agents follow a static task allocation, where the tasks are predefined and agents work through structured stepwise execution.
2. Dynamic Task Allocation: or environment. An example of dynamic task allocation can be seen in the EvoMAC Hu et al. (022bb) framework, where agents adjust dynamically based on environmental feedback, creating or dismissing agents as needed to refine or expand their tasks.
Tasks are allocated based on the real-time neods and feodback from the system
Agent Role Task Granularity. The granularity of tasks assigned agents influences both the precision and complexity of their execution:
1. Coarse Granularity: Some agents are given broader, less detailed tasks. For example, in AutoMI-GPT Trirat et al. (2024), the central controller agent coordinates the entire machine learning pipeline, handling coarse-grained tasks such as managing the overall workflow rather than focusing on the details each individual task.
2. Fine Granularity: In other cases, tasks are broken down into smaller units for more specific execution. For example, in MapCoder Islam et al. (2024a). multiple agents collaborate, each handling fine- ane-grained task such as code generation, debugging, or retrieval. This detailed task assignment ensures high accuracy but requires more computation overhead.
10

---

Published in Transactions on Machine Learning Research (02/2026)
3.2
Execution Structure
In this section, we summarize the execution strategies of LLM-based agents, emphasizing their approaches how to complete the tasks.
3.2.1 Static Execution
Static execution refers to a workflow-style structure where agents follow a predefined sequence of actions to accomplish tasks. In this paradigm, the workflow is rigidly designed, ensuring that each step is executed deterministically without deviations. This type of workflow is particularly useful in scenarios where tasks require consistent, repeatable processes or involve complex subtasks that must be systematically handled. By defining clear workflows, these systems ensure reliability, transparency, and ease of evaluation, as agents operate within well-defined boundaries, common feature of static execution structures the division of tasks into soquential, predefined stagesSoo et al. (2025]; Li ct al. (2025p; Xu al. (2025): Ou et al. (2025). These workflows often start with data lask interpretation, followed by intermediale processing steps such
as feature selection, data transformation(2 Qi & Wang (2024), subtask decomposition Luo et al. (2024. and
conclude with result generation Gu ct al. 20024 or validation Shen et al. (024].
3.2.2 Dynamic Execution
LLM Brain
Environment
Planning
Command
Perception/
Al
Observation
Next Step
Execution Result
Figure 11: In just-in-time planning, one agent responsible for planning and execution simultancously.
Just-in-time plan. The "just-in-time" planning approach represents a dynamic and iterative execution strategy widely adopted by modern LLM- agents (see Figure 11). Unlike pre-defined static workflows, this structure enables agents generate and refine plans based on real-time feedback from previous execution steps Rasheed et al. [2024]; Yang et al. (2024a) Bendinelli et al. (2025). Specifically, agents observe the outcomes of each execuled step, use these observalions to reevaluate the task's context, and then dynamically devise the next step Cao et al. (2024;; Zhao et al. (0244).. Just-in-ting -imeinetmme planning is particularly effective
in domains where the environment or task requirements are dynamic Zhhoo ct al. (2024a). as it roduces redundancy and enhances precision by continuously aligning actions with the latest data or results [Zhag
et al. 2024dh; b Liu et al. (2024b).
Plan then execute. The "plan- -eeeettteeee framework is a widely adopted structure in LLM agent systems, particularly for tasks requiring complex reasoning and multi-stage problem-solving (sce Figure 5 for an example). This framework divides the agent's workflow into two distinct phases. In the planning phase, the agent formulates a high-level strategy by breaking down the overarching task into smaller, manageable sub-tasks. In the execution phase, the agent performs these sub-tasks sequentially or iteratively, strictly adhering to the initial plan while refining the process based on intermediate results or environmental feedback. Such a design mirrors human problem-solving rategies, offering modularity, scalability, and adaptability for diverse tasks, from software development Qian et al. (2024) program repair Zhang et al. (024c) and exception handling Zhang al. 20244b).
Hierarchy execution is a structured approach where agents decompose complex tasks into smaller, manageable subtasks organized hicrarchically. While the core principle of decomposing tasks into subtasks and refining them when needed remains consistent across inplementations, structural details and execution strategies vary significantly among frameworks. For example, SELA uses tree- based hierarchies and Monte Carlo Tree Search (MCTS) to optimize AutoML workflows Chi al. (0024), CodeTree employs explicit task decomposition identify and evaluate coding strategies with execution feedback for dynamic optimization Li al. (2024d). and LATS integrates MCTS into a languagge-ggentddriven framework using model-driven value functions
11

---

Published in Transactions on Machine Learning Research (02/2026)
and self-reflection Zhou et al. (2023 Mcanwhile, MapCoder and AGILECODER introduce collaborative multi-agent systems for task decomposition Islam et al. (2024a); Nguyen et al. (2024). Data Interpreter
uses dynamic graph-based hierarchies for flexible task management Hong et al. (024). VisionCoder adopts
role-based docomposition mirroring traditional software engineering workflows Zhao et al. (2024b). and
Science Aeeenngnnnneeleeeeee and Self-Organized Agents incorporate feedback-driven planning to refine subsequent
cycles Chen et al. (2024c): Ishibashi & Nishimura (2024).
3.2.3
Summary of Execution Structure
Framework
Tree
Graph
Role
Dynamic Adjust
Feedback
Multi-Agent
MCTS
SELA
VisionCoder
AGILECODER
MASAI
Data Interpreter
MapCoder
CodeTree
Bent
LATS
Table 4: Summary of Hicrarchy Planning in Different Frameworks. The columns represent the type of plan- ning structure employed, including Tree-Based, Graph-Based, Role-Based, Dynamic Adjustment, Feedback- Driven, Multi-Agent Collaboration, and Monte Carlo Tree Search (MCTS). checkmark (v) indicates the framework supports the corresponding structure.
This section provides a summary of the execution structures employed by LLM-based agents, highlighting detailed execution dimensions including task execution, task routing, user interaction, and error handling, These execution dimensions significantly affect how agents adapt to dynamic environments and handle complex tasks. Below the key execution dimensions with specific examples from $ of the survey.
Exccution flexibilityy. Exccution flexibility describes how agents handle task allocation during exccution, ranging from static to dynamic allocation:
1. Static Execution: In this case, task allocation is predefined before execution and remains unchanged throughout the process. An example is the Data Director Hong et al. (2024) framework, where agents follow a fixed workflow with predefined tasks and are not dynamically adjusted during the execution.
2. Dynamic Execution: The task allocation changes during execution as the agent receives feedback or as the environment evolves. EvoMAC Huu et al. (2024b) employs a dynamic execution strategy where agents adjust their internal structure, adding or removing agents based on real-time feodback and task complexity.
3. Hybrid Execution: This approach combines both static and dynamic task allocation. For example, AutoML-GPT Zhang al. (2023a), task allocation is primarily managed by the central controller but dynamically adjust based the results from specialized agents performing tasks like hyperparameter luning model design.
Task routing. Task routing governs how lasks are passed between agents in multi-agent system:
1. Rule-based Routing: Tasks follow a specific rule or sequence to move from one agent to another. For instance, in MASAI Arora et al. (2024), tasks follow predefined rules based on the nature of the task, with each agent handling tasks according
strict order.
2. Agent- based Routing: In this type, one central agent controls the flow of tasks, deciding which agent will handle each task. An example is found in AutoMI-GPT Zhang al.) (2023a), where the central LLM oversees task assignment and ensures tasks are routed the appropriate agents based on the task requirements.
12

---

Published in Transactions on Machine Learning Research (02/2026)
3. Role-based Routing: Agents handle tasks according to their predefined roles. In MAGIS Tao et al. (2024) for instance, different agents like Manager, Developer, and QA Engineer assume roles in the task llow with each agent taking task when its role is responsible for the incoming part.
User interaction. User interaction defines the extent to which users are involved in task execution:
1. Fully-auto: In this case, no user intervention is required. For example, AutoCodeRover Zhang et al. (2024c) operates entirely automatically, with no need for human input during task exccution.
2. Human intervene: User interaction is frequent, and users must intervene with the system regularly, MAP- Coder [slam et al. (2024a) requires user feedback to ensure that agents are following the correct task path, especially in error-prone stages of task execution.
3. Hybrid: Some systems balance user input with autonomous execution. In MetaGPT Hong et al. (2023) for instance, the system functions autonomously but allows users to step in for oversight or specific corrections when needed, such as when complex decision-making process requires a human touch.
Plan exccution. This dimension describes whether planning and exccution are integrated or separated:
1. Plan-Execution in One: One agent handles both planning and execution. In EvoMAC Hu et al. (2024b). for example, the same agent can dynamically plan the next steps and execute them without relying on separate agents for cach task.
2. Plan-Execution Separate: The planning and execution roles are handled by separate agents. Parsel Zelikman et al. (2023), for example, separates the planning phase (where tasks are decomposed and strategized) from the execution phase (where the steps are carried out by different agents based on the plan generated).
Task decomposition. Task decomposition determines how lasks are broken down and managed:
1. Not Decomposed: In some systems, tasks are handled in their entirety without decomposition. Au- toCodeRover Zhang al. (2024e) may process some high-level tasks whole without further division, especially in simple workflows
2. Vertical Decomposition: Tasks are broken down hierarchically, with agents responsible for different levels of the task. This is seen in EvoMAC Hu et al. (2024b), where tasks are decomposed into sub-tasks at various levels, with parent agents overseeing child agents performing specific subtasks
3. Horizontal Decomposition: Tasks are divided into equal parts that are handled concurrently by multiple agents. MASAT Arora et al. (2024) employs horizontal decomposition, dividing a task into multiple smaller subtasks which is handled one following another.
4. Hybrid Decomposition: Some systems use combination of vertical and horizontal decomposition. AutoML-GPT Trirat et al. (2024). for example, combines both hierarchical task breakdowns (for oversee- ing large workflows) and parallel execution (for handling repetitive tasks such as data preprocessing)
Error handling. Error handling determines how a system deals with problems encountered during execu- tion:
1. Solve in Next Steps: Errors are handled in subsequent steps, either by a different agent or in the following phase of execution. For instance, AutoML-GPT Trirat et al. (2024) might handle errors in model design or tuning by adjusting parameters in later steps of the pipeline.
2. Traceback: The error is traced back to the previous steps to regenerate or correct the output. In MASAI Arora el al. (2024), if an error occurs during task execution, the system might trace back to earlier stages of the task to identify and resolve the root cause before continuing with execution.
3.3 External Knowledge
In this section, we summarize current methods for external knowledge acquisition in LLM-based agents. While pretrained LLMs have extensive internal knowledge, external sources often needed to address out-
13

---

Published in Transactions on Machine Learning Research (02/2026)
3.3.1 External Database
3.3.2 Retrival-based Approach
3.3.3 API Calls and Search Engine Integration
Embedding
SQL Query
AI
AI
Search Keywords
RAG
Figure 12: Overview of common external knowledge sources for DS agents.
dated or domain-specifie information. We specifically discuss external databases, retrieval- approaches, API calls, and search engine integration, as well as hybrid mothods that combine these techniques.
3.3.1 External Knowledge Methods
External Database External databases are collections of organized information stored independently of
LLMs, designed to serve as reliable sources of well-defined external knowledge for LLM-based agents Gu et al.
(2 Zhang et al. (2023a); ; Li et al. 22024e f); Jing et al. 2024); Liu et al. (2024b); Pietruszka et al. (2024):
Hassan et al. (2023 ; Chen et al. (2024a). For example, some utilize historical logs and past experimental
results as a structured knowledge base Liu et al.] (2024b); Xu et al. (2025), while Hassan et al.) (2023)
incorporates user-provided datasets with structured databases to enhance contextual understanding. This
method provides structured and consistent. data, particularly useful for domain-specific lasks that require
precision and stability.
Retrieval-based Approach Beyond external databases, LLM-based agents employ different retrieval-based approaches to dynamically obtain external knowledge from unstructured sources. For example, TTang et al. 2023 leverages the BM25 retriever, which uses ranking search to identify the most relevant code and docu- mentation based on word frequency and importance, to extract relevant segments based on given instructions. Mcanwhile, Li et al. (002cc) uses RAG Snetrrival-Augmeenter Generation) to improve response accuracy and reduce hallucination by retrieving relevant external data and integrating it into the generation process. Fur- thermore, Guo et al. (024a) extends Casc-Based Reasoning, which retrieves and adapts past cases rather than just ranking generating text, and andCCa et al. (2024) employs Llamalndex, data framework for RAG, to efficiently structure, retrieve, and inject knowledge into LLMs. Retrieval-based approaches, especially RAG, are widely used in applications, such as bias detection Li et al. (2025), geospatial analysis Chen et al. (2024b), and financial forccasting Yang et al.] (2024a), supporting broader contextual understanding and enhancing adaptability in handling structured and unstructured information.
API Calls and Search Engine Integration Another widely adopted approach involves direct interaction with external repositories and search engines, allowing LLM-based agents to directly interact with external repositories or retrieve real-time data from the internet. For instance, Liao ct al. (2024) integrates API calls to handle lime series analysis by retrieving Prophet models, statistical forecasting tool that models trends and seasonal variations in time-series data, while Bogin et al. (2024) accesses GitHub repositories and datasets from platforms like Hugging Face. By enabling agents to retrieve external knowledge on demand, API calls and search engine integration provide critical flexibility and responsiveness in dynamic environments.
Hybrid Approach To enhance knowledge acquisition, many systems adopt hybrid approaches by combining multiple methods. A common strategy is integrating both APT calls and external databases agents' accessible knowledge sources Merrill et al. (2024); Huang et al. (202cch; Li al. (2024b) Zhang et al. (002bb
Luo et al. [004).. For example, Merril et al. (2024) employs Google Search API alongside anonymized wearable health data for retrieving relevant health information. Moreover, some works combine retrieval approaches with an external databases to equip the system with expert-level knowledgeOu et al. (2025)
Additionally, several works combine API calls and search engines with relrieval-based approaches for dynamic retrieval Grosnit et al. (2024); Chen et al. (2024b); Tang et al. (2023), while others adopt a fully hybridized approach incorporating all three strategies Guo al. (2024a); Cao et al. (2024). For instance, Yang et al. (2024a) utilizes third-party APIs for financial data retrieval, applies RAG for financial sentiment analysis, and manages an external database for knowledge storage and retrieval.
14

---

Published in Transactions on Machine Learning Research (02/2026)
3.3.2 Summary of External Knowledge
External database, retrieval-based approach, and external API and search engine integration represent three primary methodologies adopted by LLM--based agents for external knowledge acquisition. External databases provide structured and reliable domain-specific information, ensuring precision and stability for targeted tasks. Retrieval-based approaches dynamically extract relevant segments from sources, enhancing contextual comprehension and adaptability. APT calls and search engines deliver real-time and frequently updated data, supporting immodiate responsiveness dynamic scenarios. Colloctively, these methods enable LLM-basod agents to cover a wide range of external knowledge and, moreover, empower LLMs to gauge, verify, enrich, even refine their internal knowledge, thereby significantly improving their overall accuracy and reliability of their responses.
3.4 Reflection
Apent Feodback
Code: Enror
Unit Test
History Aincow
Nodel Netrics
Hunan Feedbock
Unit
Agant
rnnsent
Agent
Unit Test
Agent
Agent
Juli
Codo Enpor
Exceptton
Motrics
Code
Figure 13: Overview of common reflection techniques in LLM multi-agent systemss.
LLM multi-agent systems rely on reflection mechanisms to iteratively refine their performance, enhance robustness, and adapt to complex environments. Reflection, in this context, refers to a system's ability to evaluate its past outputs, identify errors or inefficiencies and adjust its strategies accordingly enabling continuous self-improvement.
In this section, we discuss how LLM multi-agent systems employ various reflection mechanisms to improve output quality and system reliability. These mechanisms enable automated agents to iteratively refine their responses based on exccution outcomes, predefined evaluation metrics, or external foedback.
3.4.1 Reflection Methods
Agent Feedback: Many LLM multi-agent systems rely on agent feedback mechanisms, where one or more agents review the outputs of other agents and provide corrective guidance. In code generation tasks, for instance, DA-Code Huang et al. (2024c) BIASINSPECTOR Li et al. (2025), and AUTOMIND Ou el al.) (2025) applies a reviewer agent to evaluate generated scripts, detect syntax or logical errors, and suggest improvements. Similarly, in multimodal tasks, visual language models (VLMs) serve as reviewers assess image-based outputs for correctness and coherence. MatPlotAgent Yang al. (024bb exemplifies this by evaluating visualizations generated from inputs.
Code Error Handling: Automated error-handling mechanisms are essential for ensuring reliability of systems that generate and execute code, such as BudgetMLAgent Gandhi et al. (2024). WaitGPT Xie et al. (2024), and DatawiseAgent You al. (2025). These mechanisms monitor execution failures, capture error messages, and diagnose potential causes. Upon detecting an error, systems analyze faulty outputs and refine code iteratively without external intervention. This allows systems mimic human programmers reflecting errors, identifying root causes, and progressively improving the solution.
Unit testing is a structured validation mechanism where an agent generates test cases and evaluates whether the system's output meets predefined functional requirements. If a lest. fails, the agent refines the output and reruns the tests iteratively until all test cases pass. This method is widely used in LLM- driven programming tasks, where models generate executable code Guoo al.] (2024a). By automatically verifying whether the generated code meets expected functionality, unit testing helps to detect syntax errors, logical flaws, and compatibility issues before execution.
Unit Testing:
15

---

Published in Transactions on Machine Learning Research (02/2026)
Model Metrics Feedback: In tasks with clear quantitative performance indicators, model metrics feed- back enables systematic, data-driven optimization. Rather than relying on external evaluations, systems refine their outputs using predefined performance thresholds, such as accuracy, Fl score, or loss reduction. Some implementations use prreshold-based optimization, itcratively revising until desired metrics are met. For example, FinRobot Yang et al. (2024a) uses a composite scoring system that integrales normalized performance metrics with weighted evaluation criteria select or fine-tune models until the target score is achieved. Others adopt exploratory search strategies, such as Monte Carlo Tree Search (MCTS) Chen et al.) (2024b), to explore different refinement paths and select the most effective one. This structured approach allows LLM multi-agent systems to self-improve efficiently without requiring human or agent-based feedback at each step.
History Window: Unlike mechanisms that focus on immediate corrections, history window mechanisms enable long-term learning by maintaining a log of past outputs and errors [oon et al. (2024) Wang et al.. (2025a); Sco al. (2025); Xu et al. (2025). These logs help systems track recurring mistakes and recognize patterns across multiple ilerations. Some implementations enhance this capability with checkpointing, pe- riodically saving stable system states Zhao et al. (2024a). If a later refinement degrades performance, the system can revert to a previous checkpoint. By leveraging historical insights, history window mechanisms prevent repeated failures, allowing systems to refine their adjustments.
making over time and avoid ineffective
Human Feedback: Despite advancements automated reflection, human feedback remains indispensable in high-stakes applications. For example, TableAnalyst Freimanis & Andersson Rhodin (2024), a Human- in-the-loop system, allows experts to review and refine outputs, particularly when L.M--genrrtted responses require contextual understanding (RLHF) is a prominent example, where human reviewers assess model-generated responses and provide corrective guidance. Although more interactions required, human feedback mechanisms ensure that outputs align with domain-specific expectations and maintain interprotability in critical applications.
ethical considerations. Reinforcement learning from human feedback
3.4.2 Summary of Reflection
Beyond specific implementations, reflection mechanisms can be understood in terms of three fundamental dimensions: the driver, the level, and the adaptability of reflection. These dimensions help contextualize the broader implications reflection in multi-agent systems, providing insights into how different strategies contribute to long-term performance improvements.
Drivers of Reflection. Reflection in LLM multi-agent systems driven by different mechanisms that shape how the system refines its performance. Some systems improve through internal Hong et al. (2024) or external feedback Merrill et al. 20024) either from agents or human reviewers. Others adopt goal-driven approaches, continuously optimizing their outputs based on predefined performance criteria without relying on explicit external feodback Chen et al. (2024b).
1. Feelbback-driven reflection operates based on internal or external evaluation and corrective feedback, where agents or human evaluators assess outputs and provide improving guidance. For example, in agent feedback mechanisms, LLM agents critique one another's outputs, engaging in cycles of feodback and revision Triral et al.] (2024). This process is particularly common in multi-agent collaborations where specialized agents, such as debugging agents in code generation tasks Huang et al.] (2024c), verify and refine responses. Human feedback mechanisms introduce expert oversight, ensuring model outputs align with qualitative expectations Luo et al. (2024). While feedback-driven reflection allows flexibility and adaptation to dynamic environments, it also introduces challenges such as response latency and inconsis- tency in external evaluations.
2. Goal-driven reflection, in contrast, follows a more quantitative optimization approach, where systems refine their outputs based on predefined performance criteria rather than external evaluation. Many multi-agent systems employ metric- based optimization, where refinements are guided by quantitative performance indicators such as BLEU scores in machine translation or loss minimization model training. Some implementations, such as reinforcement learning and self-play strategies, enable systems to optimize through iterative self-improvement. For instance, AlphaGo Granter et al. (2017) refines its strategies
16

---

Published in Transactions on Machine Learning Research (02/2026)
without relying on external critique by playing against itself. The key advantage of goal-driven reflection lies in its predictability and efficiency, allowing systems to make systematic progress without requiring continuous human or agent-based feedback loops. However, it also risks overfitting to specific metrics, which can reduce gencralization and overlook qualitative aspects of task performance.
Levels of Reflection. Reflection does not operate uniformly across all components of system; rather, it varies in scope, with some mechanisms focusing on fine-grained, localized improvements, while others engage in system-wide analysis and optimization.
1. Local reflection focuses on refining individual task iterations, ensuring immediate performance improve- ments. Common techniques include unit testing Guo et al. (2024a), execution error diagnosis Zhang et al.. (2023b), and targeted debugging routines Huang et al. (0221c). The primary advantage of local reflection lies in efficiency, as it enables rapid refinement without requiring the system to analyze historical data restructure workflows. Additionally, it enhances precision by addressing specific issues within isolated tasks, preventing minor errors from propagating. However, because these mechanisms operate in isola- tion, they may fail to detect systematic inefficiencies. For example, in code generation Liao et al. (2024), a debugging agent may repeatedly fix syntax errors in isolated function calls without recognizing an un- derlying flaw in the model's broader logic generation capabilities. As a result, errors may be corrected in the short term but persist across different tasks.
2. Global reflection analyzes patterns across multiple iterations, extracting insights that guide long long-term optimization. History window mechanisms exemplify this approach by enabling systems to track recurring mistakes and refine their responses accordingly [ogiin et al. 2004). Some implementations further enhance global reflection through checkpointing Hong et al. (2024), where stable system states are periodically saved. If a newer version fails to improve performance, the system can revert to a prior state instead of reinforcing ineffective updates. This method is particularly valuable in preventing cyclical failures, such a conversational agent repeatedly generating redundant responses due to misaligned reinforcement signals. However, implementing global reflection requires sophisticated memory management and comes with higher computational costs, making it more suitable for large-scale, complex multi-agent systems.
Adaptability of Reflection. Another essential consideration is the degree of flexibility and adaptability within the reflection process. While some methods follow fixed, predefined correction strategies, others adjust dynamically based on performance patterns observed during exccution.
1. Structured reflection mechanisms operate according to fixed evaluation criteria, ensuring a stable and re- peatable refinement process. This category includes unit testing, where outputs must pass predefined test cases Guo et al. (2024a), and threshold- continue until performance metrics such as accuracy or loss reach a set threshold. The primary benefit of structured reflection is its predictability, making it particularly useful in well-defined problem spaces re- quiring strict correctness- suchh automated hyperparameter tuning in machine learning, where models are optimized based on predefined performance metrics. However, its rigidity limits adaplability, making it unsuitable for handling unexpected edge cases or unstructured, open-ended tasks, where predefined evaluation criteria may not capture qualitative aspects of performance.
metric optimization Yang et al. (2024a), where refinements
2. Adaptive reflection mechanisms dynamically adjust their evaluation strategies based on previous attempts or insights gathered from different parts of the system. Systems employing adaptive agent feedback [oogg] lt al. (2024), self-modifying history windows Qi & Wang (2024) reinforcement learning refine their reflection processes they accumulate more information. This flexibility allows LLM multi-agent systems to not only correct mistakes but also optimize their refinement strategies over time. For example, in dialogue systems, such as SageCopilot Liao et al.] (2024), , an adaplive reflection mechanism may modily its response-ge gennuaatnnn strategy if users repeatedly indicate dissatisfaction, shifting towards more context- aware replies rather than just fixing specific errors. Despite its advantages, adaptive reflection introduces challenges in complexity and computational overhead. Dynamically evolving strategies require greater processing power and careful tuning to prevent unintended biases or instability in the reflection process.
17

---

Published in Transactions on Machine Learning Research (02/2026)
4 Analysis from Data Science Perspective
4.1 Data Science Tasks
Data science agents are increasingly being integrated into diverse workflows to automate and optimize data- centric tasks. These tasks can be broadly categorized based on their objectives, this section outlines two representative types of tasks where AI agents are commonly deployed: (i) tasks that focus on building and refining machine learning models, and (ii) tasks centered on the generation of insights and outputs from data.
Building Machine Learning Models Building machine learning models is a core objective within data science, focusing on creating predictive, explanatory, or generative models to solve domain-specific problems These tasks typically aim to maximize the accuracy, efficiency and generalizability of the model across datasets by automating key processes such as feature engineering, hyperparameter optimization, and model selection [Tang et al. (2023). The automation of ML, model building uses iterative workflows, where each stage data preprocessing, model evaluation, and refinement is informed by prior outputs, often through multi-agent collaborations
tree-based optimization strategies Chi et al. (2024). Additionally, they are characterized by robust integration of tools alongside domain-specilic libraries tailored for computer vision, NLP, and tabular data analysis Grosnit et al. (2024); Xue et al. (2025). Those features enable efficient scaling and adaptation to diverse data environments, providing robust solutions to challenges in domains ranging
from financial analysis to biomedical research Yang et al. 2024a): Gandhi et al. (2024); Chen et al. (024a
Output Analysis Tasks Output analysis tasks focus on extracting, interpreting, and communicating in- sights derived from data. These tasks aim to generate clear and actionable narratives through visualization, summarization, or benchmarking while prioritizing interpretability and relevance Zhao al. (2024a). LLM- based agents often used to annotate visualizations and generate and refine textual explanations. For example, output analysis tasks leverage sophisticated tools like Vega-Lite for visualization and natural lan- guage models for insight generation, to enhance communication of data-driven findings Xie et al. (2024). In addition, agents are used for data storytelling through automated animated videos that transform raw data into engaging narratives by coordinating visual, textual, and auditory elements Shen et al. 2024).
Analysis tasks such as data cleaning output evaluation focus on quality dimensions like completeness, accu- racy, and consistency Li al. (0024 Agents typically employ rule-based metrics statistical heuristics to assess whether cleaned datasets meet project-specific requirements Li et al. (2024f). These evaluations often integrate automated validation pipelines, where agents verify the success of cleaning operations (e.g., deduplication, missing value imputation) and provide reports summarizing detected issues and corrective actions taken Huang et al. (2024c).
4.2
Data Science Loop
The data science loop serves as a structured blueprint for how LLM-based agents can systematically enhance and automate key phases of data workflows. Each step from initial data retrieval and cleaning to advanced statistical analysis, model development, and visualization can be augmented by LLM-based agents improve the efficiency and quality of decision-making. As shown in Figure 14 this loop captures the full lifecycle of data-driven project and provides a framework for understanding where agent- gased systems integrate into and optimize the process.
4.2.1 Data Preprocess
The data preprocessing phase includes steps for getting, cleaning, and preparing data. It starts with collecting data from different sources like databases, APIs, web scraping, sensor logs, and rescarch repositorics Guo et al.] (2024a). Structured data comes from relational databases using SQL, while unstructured text. is gathered through APIs or web crawlers Huang et al.
(002cc) Some systems combine different types of (002)): ; Cao et al. (2024). Automated pipelines use useful data, making sure that it high quality and
data, such as tables, images, audio, or text Luo et al. retrieval-angmentedd generation (RAG) to find and pull relevant. for later steps. Li et al. (2024h); Sun et, al.
2024a Rasheed et. al. (2024): Yang et al. (024a)
18

---

Published in Transactions on Machine Learning Research (02/2026)
Dsta Propracess
Statistical computation
Feature Enginsering
Ivalustion
Dola Retrieval
Deta Cleoning
Descriptive Stotistic
Inferential Tests
Calagorica cal Encoding
Dinension Reducttion
Clessification metrics
Visun lizetinn
Model Trainine
Retrievol
Bako Cleaning
Transfer Leam. Meta-Leamn
Adaptive Learn. Strotegiees
hyperporoneter Tunting
Algonithe Selection
Data Cleontng
Figure 14: Typical data science loop
After collection, the data is cleaned to improve accuracy and consistency. This step fixes missing values, duplicates, outliers, and inconsistencies that could cause problems later analysis. If important values are missing, agents will fill gaps or remove incomplete rows basod on their domain knowledge and pre-defined rules. Duplicates are found and removed using hashing or fuzzy matching methods like Levenshtein distance for text. Outliers are detected using basic statistical methods like Z-score and IQR filtering, while more advanced techniques like isolation forests and autoencoders handle complex cases. Hong et al. (2024): Li
et al. 2024h).
4.2.2 Statistical Computation
The statistical computation phase uses statistical methods to analyze data, find patterns, and show re- lationships. These methods help understand distributions and correlations in the data. Basic techniques include descriptive statistics (mean, median, variance), inferential tests (e.g., t-tests, chi-square tests), and correlation analysis are widely employed to establish baselines and validate data integrity Gu et al. (2024). Methods like hypothesis testing and parametric or non-parametric methods are also utilized to derive insights under uncertainty Zhang et al. (003b)). Parallelized frameworks like Dask or Spark are used to improve computational efficiency for large-scale or distributed datasets Jing et al. (022)).
4.2.3 Feature Engineering
In the feature engineering phase, raw data is transformed into meaningful representations that improve model performance. This phase involves creating, selecting, and refining features ensure they are both relevant and discrininative for the underlying predictive task PPietruszaa al. (2024). A well-designed fea- ture engincering process enables machine learning models to capture patterns effectively while mitigating overfitling and reducing noise. The core techniques in feature engineering include handling missing values, categorical encoding, numerical transformations, and dimensionality reduction. Standard approaches such as one-hot encoding and label encoding allow categorical variables to be numorically represented, whilc scaling techniques like min-max normalization and standardization ensure consistent feature magnitudes Tang et al. (2023). Advanced feature engineering techniques involve feature construction and selection strategies. Poly- nomial feature expansion, interaction terms, and domain-specifie transformations (e.g., log transformations for skewed distributions) enhance model's ability to capture complex relationships. Additionally, meth- ods like Principal Component. Analysis (PCA) and t-Distributed Stochastic Neighbor Embedding (I-SNE) facilitate dimensionality reduction, improving computational efficiency and reducing redundant information. Feature importance techniques, such as SHAP (SHapley Additive exPlanations) and permutation impor- tance, guide the selection of the most predictive variables Chi et al. 2024).
4.2.4 Model Training
In the model training phase, refined data and features are input into machine lcarning algorithms to cre- ale predictive models. This phase encompasses algorithm selection, hyperparameter tuning, and iterative validation to optimize performance. Commonly used algorithms range from traditional methods like linear regression and decision trees to modern neural architectures designed for high-dimensional and multi-modal data Huang al. (2024a). Agents in training pipclines integrate and use ML libraries and tools facilitate model creation, while hyperparameter optimization frameworks such as Optuna and Ray Tune enhance the
19

---

Published in Transactions on Machine Learning Research (02/2026)
search for optimal configurations Liu al. (2024b). Real-time monitoring and adaptive learning strategies are often employed to refine models, particularly in dynamic environments where data evolves over time Chen et al. (2024b). Advanced techniques, such as transfer learning and meta-learning, enable models to leverage knowledge from pre-trained networks, reducing training time and improving performance Luo et al. (2024.. The model training phase ilerative by nature, with feedback loops that refine both the model and its underlying assumptions, ensuring robustness and generalizability across unseen data Zhang et al. (002d)).
4.2.5 Evaluation
The evaluation phase is to assese the performance and reliability of machine lcarning models. This stop involves the use metrics tailored to the task at hand, such as accuracy, precision, recall, and Fl score for classification tasks, or RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) for regression analyses Li al. (2024e). For unsupervised tasks, metrics like silhouette score and Davies-Bouldie index are employed to evaluate clustering quality Jing et al. (2024). Pross-validation techniques, such k-fold leave-one-out, are widely used to estimate model performance unseen data, ensuring that results generalize beyond the training set Chen et al. (2024c). The integration of automated evaluation frameworks allows for streamlined reporting and comparison, cnabling itcrative improvement and enhancing the credibility of the final model Xie et al. 2024
4.2.6 Visualization
The visualization phase turns data into easy-lo-undersland images that help with decision-making. Clear visuals change complex data into simple forms like charts, plots, and dashboards. These help people see patterns, trends, and unusual points. This step uses both fixed and interactive visuals, letting users explore the data in different ways.
visualization libraries and tools create visualizations Modern data science agents leverage variety of tailored to different analytical needs Hong et al. (2024); Li et al. (2024h); Liao et al. (2024). In addition to traditional visualizations like line plots and histograms, more advanced visualizations, such model interpretability plots (e.g., SHAP values), are used to explain model outputs in transparent manner.
4.3 Summary of Data Science Perspective
Data science agents typically follow structured workflows in different stages such data preprocessing, feature selection, and model training et al. (2025). LLMM-based agent sys- tems like Qi & Wang (2024) and Li al. (2024f) automate data preprocessing strategics and address data quality issues like missing values, inconsist encies, and duplications. Feature selection mechanismę et al. (2024b)further enhance model performance by identifying the most data features, thus optimizing capabil- itics. Additionally, advanced agent systems, such as Chi et al. (2024): Trirat et al. (2024), dynamically generale optimized data science pipelines.
Fixed Data Science Pipeline
Self-planning Training Data science agents exhibit self-planning training capabilities, autonomously se- lecting optimal algorithms, configuring hyperparameters, and iteratively validating model performance dur- ing the training ShaseSeo al. (2025). This methodology parallels traditional Neural Architecture Search (NAS), automated process that optimizes neural network designs, but significantly leverages agent-driven decision-making framework et al. 2024b). By systematically exploring and refining model configura- tions based on ongoing feedback, these agents ensure robust and adaptive performance tailored to specific contexts.
setrics-based Feedback In contrast to traditional coding-oriented oriented agents, data science agents utilize rig- orous modcl-based motries feodback for itcrative improvement. FairOPTJung et al. (2025) leverages quanti- tative metrics such as accuracy, precision, recall, and F1 scores assess model outputs and systematically enhance their performance through structured refinement cycles. This metrics-driven feedback mechanism
20

---

Published in Transactions on Machine Learning Research (02/2026)
ensures continuous performance enhancement and enables agents to maintain consistent accuracy across diverse analytical scenarios.
Visualization Analysis
Data science agents distinctly emphasize the visualization of analytical outcomes.
recognizing its critical role in data interpretation and communication. Visualization- systems Zhao et al. (2024a) generate clear, insightful visual representations that facilitate a comprehensive understanding of complex analytical results. Furthermore, specialized agents like Matplot-gent Yang et al. (2024b) employ visualization explicitly as an informative feedback mechanism for detecting and correcting analytical errors. By integrating visual leedback into their error-handling processes, these agents significantly enhance the interpretability, accuracy, and overall effectiveness of data-driven decision-making
5 Benchmark
A wide range of benchmarks has been developed to evaluate LLM-based data science agents, each reflecting different assumptions about what "data science competence" entails. Table 5 summarizes the major suites. General-purpose benchmarks such as ML-Bench Tang et al. (2023) and DSBench Jing et al. (2024) assess core analytical skills across preprocessing, modeling, feature engineering, and multimodal reasoning. Other benchmarks specialize in particular workflow stages, such as visualization in MatPlotAgennt-E
Yang al.)
(202bb) or geospatial analysis in GeoAgent-Bendd Liao et al. (2024). Domain-focused benchmarks, such
Food Puzzle Huang et al.] 2024b)) GenoTEX Liu & Wang (2024) or AgentClinic [Shhiddall et. al. (2024) measure scientific and driven reasoning under disciplinesseecffcc constraints. In addition, TheAgent- Company Xu et al. (2024) provides a practical evaluation framework covering end-to-end automation tasks in corporate environments, including data science, engineering, and business operations.
While these benchmarks provide broad coverage and standardized evaluation scttings, they also exhibit structural limitations that may mischaracteriize an agent's true capabilities. First, most datasets evaluate short, isolated tasks. Suites such as DSBench ML-Bench typically test single operation (e.g., computing statistic, performing a small transformation, fitting lightweight model), thus missing the errors accu- mulated across multi-stage workflows. Second, rigid task formats allow models to exploit surface regularities rather than reasoning. In PlolAgent-- templates (e.g., generating a 2x3 grid of boxplots), so an agent may match the template instead of demon- strating genuine analytical understanding and reasoning. Third, benchmark inputs are often clean, complete, and unambiguous. But LLM agents frequently fail when inputs are messy or underspecified (e-g., irregular schemas missing values), scenario that current benchmarks like BroAAennt-Bendee still rarely capture.
for example, many lasks share highly similar visualixation
Each benchmark therefore captures only a particular slice of the capabilitics required for real data science practice. Gereral-purpose suites test broad coverage but not long-horizon robustness; workflow-sperifte and domain-specific benchmarks evaluate deeper skills but in narrower settings. A comprehensive evaluation therefore requires viewing performance across multiple benchmarks, each revealing different aspects of reli- ability, reasoning, and robustness.
6 Future Research Opportunity
Building the limitations discussed, several research opportunities emerge for improving the reliability of LLM-based data science agents. The core challenges of current systems lie in their vulnerability to hallucina- tion, tendency to generate fragile and non-reprodueible code, difficulty maintaining context over long- horizon workflows, and reliance on rigid agent pipelines. The following subsections outline three promising directions, each targeting a different aspect of these limitations and suggesting how next sextt-eeneatiin agent systems may overcome them.
6.1 Data-Centric Diagnostics
As discussed carlier, many high-impact failures in LLM-bbsed data science agents originate not from the agent's reasoning, but from properties of the data itself. Agents frequently misinterpret irregular schemas, treat categorical columns as numerical, overlook missing-value structures that distort statistical estimates.
21

---

Published in Transactions on Machine Learning Research (02/2020)
InfiAgent:AAeench (Accurasy)
DS Bench (Relative Performance
SWE- (Resolied Rate)
0.7
0.975
Oeta Intergaaber (GPT
2.5
(Claad 3.7 Scrnet 0f ni-0
0.950
0.6
PharmaDety
Data Inderpontt
0.5
0.900
0.875
0.45
Resolvves
0.4-
(Deuher)
0.850
0.40
0.3
[GPT
& Opas!
0.825
[0F7 -40
0.35
(GP7 -611
0.2
(OFT -da
2024-31
2024-25
2322-62 2822-20
2324-1212446
2023
Figure 15: Temporal performance trends across three benchmarks. The figure shows the evolution of state- of-the-art (SOTA) performance on InfiAgent-DABenchh (accuracy), DS-Bench (relative performance gap, RPG), and SWE-bench Lite (resolved rate). Each data point corresponds to the best-performing system reported at the time of publication and is annotated with the system name and backbone LLM.
or apply methods incompatible with the underlying distribution. Current reflection mechanisms are blind to such data issues: they only inspect the agent's generated text code, lcaving foundational data errors undelected. This limitation explains why agents often produce "correctly reasoned" but fundamentally invalid analytical results. Future research should therefore develop data-centric diagnostic modules that automatically evaluate schema consistency, detect anomalous or distribution-shifted subsets, identify type or semantic mismatches, and trace how specific data defects influence downstream modeling outcomes. By elevating data quality analysis to a first-class reflective capability, agents would be better equipped to prevent systematic failure modes that current systems cannot even observe.
6.2 Uncertainty-Aware Workflow Planning
Another highlighted limitation that current LLM-based agents operate without any awareness of uncer- tainty. When the model is unsure about a column's 's meaning, a user's intent, or the correctness of an interme- diate step, its downstream behavior does not change and the uncertain output carries the same operational consequences confident one. Because agents lack mechanisms to detect low-confidence reasoning, they do not pause execution to request clarification, reconsider assumptions, expand the search space, branch the workflow. As a result, ambignous or underspecified steps often trigger confident hallucinations that steer the entire pipeline in an incorrect direction, with errors compounding across subsequent stages. Future research may therefore focus on developing uncertainty-aware workflow planning, where agents explicitly model their confidence and adapt their behavior accordingly. Such systems could flag uncertain decisions, trigger query refinement, invoke stronger verification modes, or explore multiple candidate workflows when ambiguity is high. Introducing uncertainty first-class signal would allow agents to dynamically modulate autonomy and mitigate the error-propagation patterns that are pervasive in morgzoorizon data science tasks.
6.3 Pipeline-Level Reflection
A further limitation that existing reflection mechanisms operate only at the step level. Reviewers check the latest code snippet, self-reflection revises the previous response, and unit tests validate single functions. Yet data science workflows are inherently multi-stage, and small upstream issues, such incorrect filtering, misaligned merges, omitted preprocessing, often propagate silently into feature engineering, model training, and evaluation. Current agents lack the ability to revisit carlier stages, attribute downstream crrors to upstream decisions, or assess the global coherence of the entire analytical pipeline. Future research may address this gap by developing pipeline-level reflection: constructing explicit workflow graphs, tracing how information and errors flow across steps, identifying high-risk nodes, performing stago-level sanity checks, and enabling global revisions that correct the pipeline holistically rather than locally. Such mechanisms would transform reflection from shallow post-hoc editing into a principled, system-level reliability process.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Benchmark
Source
# of Tasks
Task Types
BLADE Gu et al.
2024
Literature
714
multiple-choice and ground truth-analysis studies
InfiAgent- Bench
Hu et al.
GitHlub
257
Data Analysis and ML
2024a
ML-Beecch
Tang et al.
2023
Gitllub
9641(168)
Multimodal,, time scries, audio, LLM, vision, biomedical, graphs
DevBeuch Li al.
1021a
Github
22
DL, CV, and NLP
SUPER
Bogin el al
2024
GitHub
739
Research data science challenges
DA-Code
Huang es
2024c
Kagyle, GitHub
Data wrangling, EDA, ML
FeasFnng
Pictrusska
2024
Kaggle
Classification, regression, feature engineering
Tapilot Li ct al.
2024c
Kaggle
1024
Data analysis, request clarification
MLE-Bench
Chan et al
20224
Kaggle
75
Modelling Tiske video, LLMs, tabular)
DSBench
Jing et al.
2024
ModelOff. Kaggle
540
Data analysis, modeling
Py Bench
Zbang
20246
Kaggle, Arxiv, multimedia files
Data Analysis, MI, image, text, and audio analysis
DSEval Zhang
al.
2024d
Tutorials, StackOverfoww Kaggle
825
Data analysis
Spider2-V Cao 05
2024
Tutorials, enterprise applications
494
Warchonsing, transformation, visualization
MatPPottAgent-I
Yang
Matplotlib, OriginLab
advanced visualiaatin
2024b
ScienceAgent Bench
Clen et xl.
Peer-reviewed publications
102
Data processing, modeling, visualization
2024c
SagePilot Liao et al.
2024
Exterual datasets
276
SQL-related tasks
Text2Analysia
et sl.
2022
LLM and human-gemersted
2219
Data analysis, modeling
DataNarrative Islam
Pew Research, Tableau Public, GapMinder
1449
Data-driven visualization, storytelling
2024b
Insigt-Bench
ServiceNow platform
Analysis
MMAU
Yin
2024
In-house, Kaggle, DeopMind-Mati
20
DS/ML, contest-level coding, math
GooAgent- Bench
GitHfu, tutorials, generation
19,504
Single-turn, multi-turn Gon-spatial analysis
2024
MLAgentBennch Huang et al.
Kaggle, canonical datasets
Research Image, text, graph, tabular, time series ML tasies
2023
Merrill et al.
2024
Human experts, wearable data
4172
Numerical, open-ended reasoning in personal health
AleneClinic Schmidgall et al.
USMLE, MIMICIV, NEJM, MedAQ
335
Patient interaction, multimodal data collection cliuics
2021
GenoTEX Liu & Wang 2024
GEO, TCGA, NCBI Gene
1146
Dataeet selection, processioning, statistical analysis in Genomics
TheAgent Company (2024)
Xu al.
Company websites, human examples
175
DS tasks in company settings
PoodPuzsle
Huang al.
2024b
FlavorDB
2744
Molecular prediction and profile completion flavor science
MLGym Nathani
2025
Publications datasets
DS, CV, RL, game theory Data Analysis, modeling. data Visualization
DataSeiBench Zhang
CodeGeeX, BCB, and human
519
2025a
Chen
2024a
CREEDS
Drug Hepprppsing medical fields
BIODSA-1K
Wang et al.
2025b
Publications
1029
Biomedical hypothesis and analysis
DS-100
Stackoverflow
Code generation DS
Bench
Wang et. al.
2024b
Published studies, TCGA-type genomics, and clinical data
293
Biomedical coding tasks
MlL-Dev-Benchl Padigela
Uuknown
processing, modeling, API integration
2026
TimeScriesGym
Kaggle, Github, publications, hand-crafted
Timo-series problem
Table 5: Summary of Benchmarks in Data Science Perspective
7 Conclusion
This survey provides a comprehensive analysis of Large Language Model (LLM)-based agents in data science, addressing key aspects from both agent design and data science perspectives. Tt. systematically explores
23

---

Published in Transactions on Machine Learning Research (02/2026)
different agent roles, execution structures, external knowledge acquisition methods, and reflection techniques. The study introduces dual-perspective framework bridging general agent design principles with specific data science requirements, covering structures from single-agent to dynamic multi-agent systems and execution methods from dynamic planning to static workflows. Overall, this paper offers a comprchensive summary of existing work on LLM-based data science agents.
24

---

Published in Transactions on Machine Learning Research (02/2026)
References
Daman Arora, Atharv Sonwane, Nalin Wadhwa, Abhav Mchrotra, Saäteja Utpala, Ramakrishna Bairi, Aditya Kanade, and Nagarajan Natarajan. Masai: Modular architecture for software-engineering ai agents. arv preprint arXie:2406. 1 1638, 2024.
Tianyi Bai, Ling Yang, Zhen Hao Wong, Jiahui Peng, Xinlin Zhuang, Chi Zhang, Lijun Wu, Jiantao Qiu, Wentao Zhang, Binhang Yuan, et al. Multi-agent collaborative data selection for efficient Ilm pretraining. arXiv preprint arXiv:2410.08/ 00.0000, 2024.
Tommaso Bendinelli, Artur Dox, and Christian Holz. Exploring I1m agents for cleaning labular machine learning datasets. arXiv preprint orXiv:2088.06664 2025.
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. Super: Evaluating agents on setting up and executing tasks from research repositories. arXiv preprinl or 0rxii22409.074 2024.
Yifu Cai, Xinyu Li, Mononito Goswami, Michal Wiliissii Gus Welter, and Artur Dubrawski. Time- seriesgym: scalable benchmark for (time series) machine learning engineering agents. arXiv preprint arXiv:2505.13211 2025.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? arXiv preprint arXiv:2407.10990 2024.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.0717 07095, 2024.
Haoran Chen, Shengxiao Zhang, Lizhong Zhang, Jie Geng, Jinqi Lu, Chuandong Hou, Peifeng He, and Xuechun Lu. Multi role chatgpt framework for transforming medical data analysis. Scientific Reports, 14 (1):13930, 2024a.
Yuxing Chen, Weijie Wang, Sylvain Lobry, and Camille Kurtz. An Ilm agent for automatic geospatial data analysis. Xii preprint Xiv:24 10. 18792, 2024b.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, al. Scienceagentbench:: Toward rigorous asscssment of language agents for data-driven scientific discovery. arXie preprind Xr 0.t:2410.05086 2024c.
Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang, Jacky Kwok, Ceyao Zhang, et al. Sela: Tree-search enhanced 1lm agents for automated machine learning. arXiv preprint Xxic:2410.17255 2024.
Gelci Deng, Yi Liu, Victor Mayoral- Peng Liu, Yuckang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. (PentesIGPT]: Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 847 864, 2024.
Gang Fan, Xiaoheng Xie, Xunjin Zheng, Yinan Liang, and Peng Di. Static code analysis in the ai era: An in-depth exploration of the concept, function, and potential of intelligent code analysis agents. arXiv preprint Xrx::22310. 08837, 2023.
Andris Freimanis and Patrick Andersson Rhodin. Tableanalyst: an llm-agent for tabular data analysis- implementation and evaluation on tasks of varying complexity. 2024.
Shubham Gandhi, Manasi Patwardhan, Lovekesh Vig, and Gautam Shroff. Budgetmlagent: A cost-cffective Im multi-agent system for automating machine learning tasks. arXio preprint or 1ii:2411.07464 2024.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Scott R Granter, Andrew H. Beck, and David J Papke Jr. Alphago, deep learning, and the future of the human microscopist. Archiues of Pathology & Laboratory Medicine, (41(5):619-621 621, 2017.
Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio lacobacci, Abdelhakim Benechchab, et al. Large language models orchestrating structured reasoning achieve kaggle grandmaster level. arXiv preprint 8rXiv:2411.03560 2024.
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, al. Blade: Benchmarking language model agents for data-driven science. arXiv preprint trXie:2408..000 2024.
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang, Ds-agent: Automated data science by empowering large language models with case-based reasoning, arXiv preprint arXiv:2402. 17453, 2024a.
Tianyu Guo, Xiaozhi Chen, Yujia Wang, et al. Large language model based multi-agents: A survey of org/ abs/ 220220100000000000
progress and challenges, 2024b. URL
Tanmay Gupta, Luca Weihs, and Aniruddha Kembhavi. Codenav: Beyond tool-use to using real-world codebases with Ilm agents. arXiv preprint arXiv:2406. 12276, 2024.
Md Mahadi Hassan, Alex Knipper, and Shubhra Kanti Karmaker Santu. Chatgpt as your personal data scientist. arXiv preprint wrXiv:2305.16657 2023.
Xinyi He, Mengyu Zhou, Xinrun Xu, Xiaojun Ma, Rui Ding, Lun Du, Yan Gao, Ran Jia, Xu Chen, Shi Han, et al. Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries. In Proceedings of the AAAT Conference on Arlificial Intelligence, volume 38, pp. 18206-18215, 2024.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuhong Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.003: 2023.
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An Ilm agent for data science. arXiv preprint 2rxiv:2402. 18679, 2024.
Xucyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. Infiagent-dabench: Evaluating agents on data analysis tasks. arXiv preprint arXiv:2401.05500 2024a.
Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, and Si- heng Chen. Self-evolving multi-agent collaboration networks for software development. arXiv preprint arXiv:24 10. 16946, 2024b.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovee. Benchmarking large language models as rescarch agents. In NewrIPS 2023 Foundation Models for Decision Making Workshop, 2023.
Qian Huang, Jian Vora Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024a.
Tenghao Huang, Donghee Lee, John Sweeney, Jiatong Shi, Emily Steliotes, Matthew Lange, Jonathan May, and Muhao Chen. Foodpuzzle: Developing large language model agents as flavor scientists. arXiv preprint arXiv:2409.1288 2024b.
Yiming Huang Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, et al. Da-code: Agent data science code generation benchmark for large language models. arXiv preprint Xiv::22410 07331, 2024c.
26

---

Published in Transactions on Machine Learning Research (02/2026)
Yoichi Ishibashi and Yoshimasa Nishimura. Self-organized agents: A Ilm multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXin:204.022 02183, 2024.
Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez, Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405. 1 11408, 2024a.
Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Datanarrative: Automated data-driven storytelling with visualizations and texts. arXiv preprint or arXiv:2408. 25.:2400.00366, 2024b.
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents to becoming data science experts? arxiv preprint ar 2rv: 07700, 2024.
Minscok Jung, Cynthia Fucrtes Panizo, Liam Dugan, Yi R., Fung, Pin-Yu Chen, and Paul Pu Liang. Group- adaptive threshold optimization for robust ai-generated text detection, 2025. URL //tp://///xiroroo..
0as/2502.04522
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zeutlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds 1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319 18345. PMLR, 2023.
Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. Codechain: Towards modular code generation through chain of self-revisions with representalive sub-modules. orXiv preprinl arXiv:2310.0800 2023.
Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. Devbench: comprehensive benchmark for software development. arXiv preprint 2403::440300600 3, 2024a.
Dawei Li, Zhen Tan, and Huan Liu. Exploring large language models for feature selection: A data-centric perspective. arXiv preprint arXiv:2408. 12025, 2024b.
Guoliang Li, Xuanhe Zhou, and Xinyang Zhao. Llm for data management. Proceedings of the VLDB
Endowment, 17(12):4213 4216, 2024c.
Haoxuan Li, Mingyu Derek Ma, Jen-tse Huang, Zhaotian Weng, Wei Wang, and Jieyu Zhao. Biasinspector:
Detecting bias in structured data through llm agents. arxiv preprint #xXi:2254.04455,, 2025.
Jicrui Li, Hung Le, Yinbo Zhou, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Codetroe: Agent-guided (ree search for code generation with large language models. arXiv preprinl arXio:2411.0432 2024d.
Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Yurong Wu, Chenhao Ma, Jian Slan-Guang Lon, and Reynold Cheng. Tapilot-crossing: Benchmarking and evolving Ilms towards interactive data analysis agents. arXiv preprint ar Xrxiv22403.000 053007 2024e.
Lan Li, Liri Fang, and Vetle Torvik. Antodcworkflow: Llm-based data cleaning workflow anto-generation and benchmark. Xiv preprint ar arXiv:2412.06724 2024f.
Xinyu Li, Shihan Wang, Siyang Zeng, ct al. A survey on Ilm-based multi-agent systems: Workflow, infras- tructure, and challenges. Vicinagearth, 1(1):9, 2024g.
Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions. arXiv preprint 1rXix:2400.2442.. 2024h.
Yuan Liao, Jiang Bian, Yuhui Yun, Shuo Wang, Yubo Zhang, Jiaming Chu, Tao Wang, Kewei Li, Yuchen Li, Xuhong Li, et al. Towards automated data sciences with natural language and sagecopilot: Practices and lessons learned. or Xie preprint orXin:2407.21104 2024.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Feng Lin, Dong Jac Kim, Tus-Husn, and Chen. Soen-101: Code generation by emulating software process models using large language model agents, 2024. URL httpp:////aaii.or//aa/////////// 15852
Haoyang Liu and Haohan Wang. Genotex: A benchmark for evaluating llm-based exploration of gene expression data in alignment with bioinformaticians. arXiv preprint arXiv:2406. 153:1, 2024.
Jiaqi Liu, Kening Wang, Yuxuan Chen, et al. Large language model-based agents for software engineering: A survey, 2024a. URL
Siyi Liu, Chen Gao, and Yong Li. Large language model agent for hyper-parameter optimization. arv preprint arXiv:2402.01888 01881, 2024b.
Daqin Luo, Chengjian Feng, Yuxuan Nong, and Yiqing Shen. Autom3l: An automated multimodal machine learning framework with large language models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8586 8594, 2024.
Mike A Merrill, Akshay Paruchuri, Naghmeh Rezaci, Geza Kovacs, Javier Perez, Yun Liu, Erik Schenck, Nova Hammerquist, Jake Sunshine, Shyam Tailor, et al. Transforming wearable data into health insights using large language model agents. arXiv preprint orXiv:2406.0644 2024.
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. Migym: new framework and benchmark for advancing ai research agents. Xiviv preprinl urXiv:2502.1449 14499, 2025.
Minh Huynh Nguyen, Thang Phan Chau, Phong X Nguyen, and Nghi DQ Bui. Agilecoder: Dynamic col- laborative agents for software development based on agile methodology. arXiv preprint arXiv:2406. 11912,
2024.
Yixin Ou, Yujie Luo, Jingsheng Zheng, Lanning Wei, Shuofei Qiao, Jintian Zhang, Da Zheng, Huajun Chen, and Ningyu Zhang. Automind: Adaptive knowledgeable agent for automated data science. arXiv preprint arXiv:2506. 10074, 2025.
Harshith Padigela, Chintan Shah, and Dinkar Juyal. MI-dev-bench: Comparative analysis of ai agents ml development workflows. arXie preprint TTN::2502000002 2025.
Huy Nhat. Phan, Tien N Nguyen, Phong X Nguyen, and Nghi DQ Bui. Hyperagent: Generalist. software engineering agents to solve coding tasks at scale. arXiv preprint (XNi:2409.11999 2024.
Michal Pietruszka, Lukasz Borchmann, Aleksander Jedrosz, and Pawel Morawiecki. Can models help create better models? evaluating ilms data scientists. arXiv preprint arXiv:2410.23331, 2024.
Danrui Qi and Jiannan Wang. Cleanagent: Automating data standardization with Ilm-based agents. arXiv preprint ar Xr:::2403.08890 2024.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15174 15186, 2024.
Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. Llm-based agentic systems in medicine and healthcare. Nature Machine Intelligence, 6(12):1418- 1420, 2024.
Zeeshan Rasheed, Muhammad Waseem, Aakash Ahmad, Kai-Kristian Kemell, Wang Xiaofeng, Anh Nguyen Duc, and Pekka Abrahamsson. Can large language models serve as data analysts? a multi-agent assisted approach for qualitative data analysis. orXiv preprint urXiv:2402.01388 2024.
Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, et al. Insightbench: Eval- uating business analytics agents through multi-step insight generation. arXiv preprint orXiv:2407.06425, 2024.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Samuel Schmidgall, Rojin Ziaci, Carl Harris, Eduardo Reis, Jeftrey Jopling, and Michael Moor. Agent- clinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments. arXie preprinl WwXiv:2405.07960 2024.
Wonduk Seo, Juhyeon Lee, and Yi Bu. Spio: Ensemble and selective strategies via llm-based multi-agent planning in automated data science. arXiv preprint arXiv:2508.2333 2025.
Leixian Shen, Haotian Li, Yun Wang, and Huamin Qu. From data to story: Towards automatic animated data video creation with Imm-based multi-agent systems. In 2024 IEEE VIS Workshop on Data Storytelling in an Era of Generative Al (GEN4DS), pp. 20-27. IEEE, 2024.
Maojun Sun, Ruijian Han, Binyan Jiang, Houduo Qi, Defeng Sun, Yancheng Yuan, and Jian Huang. Lambda: A large model based data agent. arXiv preprint arXiv:2407. 17535, 2024a.
Maojun Sun, Ruijian Han, Binyan Jiang, Houduo Qi, Defeng Sun, Yancheng Yuan, and Jian Huang. A survey on large language model-based agents for statistics and data science, 2024b. URL /sttps//arxivv org/ ht5/abb/24412 14222
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, et al. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. arXiv e-prints, pp. arXiv 2311, 2023.
Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. Magis: LIm-based multi-agent framework for github issuc resolution. arXiv preprint 1rXiv:2408.17997 2024.
Khai T. Tran, Duy Dao, Minh D. Nguyen, et al. Multi-agent collaboration mechanisms: A survey of llms, 2025. URL https://arxiw..rrg hww.:.org//bba...a httss//50100000
Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent 1lm framework for full-pipeline automl. arXiv preprint or 1rXiv:241000200 2024.
He Wang, Alexander Hanbo Li, Yiqun Hu, Sheng Zhang, Hideo Kobayashi, Jiani Zhang, Henry Zhu, Chung- Wei Hang, and Patrick Ng. Dsmentor: Enhancing data science agents with curriculum learning and online knowledge accumulation. arXiv preprinl urXiv:2505.14 14 163, 2025a.
Lingchen Wang, Chao Ma, Xiaohan Feng, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024a.
Zifeng Wang, Benjamin Danek, Ziwei Yang, Zheng Chen, and Jimeng Sun. Can large language models replace data scientists in clinical research? arXie preprint arXiv:2410.22 21591, 2024b.
Zifeng Wang, Benjamin Danck, and Jimeng Sun. Biodsa- 1k: Benchmarking data science agents for biomed- ical research. Xriie preprint NXi::2505.16100 2025b.
Liwenhan Xie, Chengbo Zheng, Haijun Xia, Huamin Qu, and Chen Zhu-Tian. Waitgpt: Monitoring and steering conversational Ilm agent in data analysis with on-the-fly code visualization. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pp. 1-14, 2024.
Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking 11m agents on consequential real world tasks. arXiv preprint 2xx::241..1411,, 2024.
Wenyi Xu, Yuren Mao, Xiaolu Zhang, Chao Zhang, Xuemei Dong, Mengfei Zhang, and Yunjun Gao. Dagent: A relational databaso-driven data analysis report generation agent. arXiv preprint 1xXi:2503.13269, 2025.
Eric Xue, Zeyi Huang, Yuyang Ji, and Haohan Wang. Improve: Iterative model pipeline refinement and optimization leveraging Ilm agents, 2025. URL https://arxiv..rgg/ www.o/gggg..........ooo 18530
Hongyang Yang, Boyu Zhang, Neng Wang, Cheng Guo, Xiaoli Zhang, Likun Lin, Junlin Wang, Tianyu Zhou, Mao Guan, Runjia Zhang, et al. Finrobot: An open-source ai agent platform for financial applications using large language models. arXio preprint (rv: 2024a.
29

---

Published in Transactions on Machine Learning Research (02/2026)
Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Luu, Dong Yu, et al. Matplotagent: Method and evaluation for llm-based agentic scientific data visualiza- 2024b.
tion. arXic preprint arXiv:2402.114
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, et al. Mmau: A holistic benchmark of agent capabilities across diverse domains.
arXiv preprint ar vxxi:2407.11966 2024.
Ziming You, Yumiao Zhang, Dexuan Xu, Yiwei Lou, Yandong Yan, Wei Wang, Huaming Zhang, and Yu Huang. Datawiseagent: A notebook-centric Ilm agent framework for automated data science. arXiv preprint arXi::2503.0000 07044, 2025.
Yangyang Yu, Zhiyuan Yao, Haohang Li, Zhiyang Deng, Yuechen Jiang, Yupeng Cao, Zhi Chen, Jordan Suchow, Zhenyu Cui, Rong Liu, et al. Fincon: A syuthesized Ilm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making. Advances in Neural Information Processing Systems, 37:137010-137045, 2024.
Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, et al. Codes: Natural language to code repository via multi-layer sketch. arxiv preprint arXiv:2403. 164433 2024.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: Algorithmic reasoning with language models by composing decompositions, Advances in Neural Information Processing Systems, 36:31466-31555 2023.
Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. Datascibench: An Ilm agent benchmark for data science. arXie preprint arXiv:2502.1 13897, 2025a.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool- integrated agent systems for real-world repo-level coding challenges. arXiv preprint 0rX1v:2401.07399 2024a.
Shujian Zhang, Chengyue Gong, Lemeng Wu, Xingchao Liu, and Mingyuan Zhou. Automl-gpt: Automatic machine learning with gpt. arXiv preprint 05i::2905.02499, 2023a.
Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yucting Zhuang. Data-copilot: Bridging billions of data and humans with aulonomous workflow. Xrxio preprinl wrXiw:2306.07200 2023b.
Xuanming Zhang, Yuxuan Chen, Yuan Yuan, and Minlie Huang. Seeker: Enhancing exception handling in code with basdd multi-agent approach. arXiv preprint 2Xxt:2410.069499 2024b.
Xueqiao Zhang, Chao Zhang, Jianwen Sun, Jun Xiao, Yi Yang, and Yawei Luo. Eduplanner: Llm-based multi-agent systems for customized and intelligent instructional design. IEEE Transactions on Learning Technologies, 2025b.
Yaolun Zhang, Yinxu Pan, Yudong Wang, and Jie Cai. Pybench: Evaluating I1m agent on various real-world coding lasks. arXiv preprini 0177:22407.16732, 2024c.
Yuge Zhang, Qiyang Jiang, Xingyu Han, Nan Chen, Yuqing Yang, and Kan Ren. Benchmarking data science agents. Xiu preprint Xiv:2400 17168, 2024d.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1592-1604, 2024c.
Yuheng Zhao, Junjic Wang, Linbin Xiang, Xiaowen Zhang, Zifei Guo, Cagatay Turkay, Yu Zhang, and Siming Chen. Lightva: Lightweight visual analytics with Ilm agent- based task planning and execution. IEFE. Transactions on Visualization and Computer Graphics, 2024a.
30

---

Published in Transactions on Machine Learning Research (02/2026)
Zixiao Zhao, Jing Sun, Zhiyuan Wei, Cheng-Hao Cai, Zhe Hou, and Jin Song Dong. Visioncoder: Empowering multi-agent auto-programming for image processing with hybrid llss. arxiv preprinl orXim:24 10. 19245, 2024b.
Andy Zhou, Kai Yan, Michal Stlapentokl-Rochman, Haohan Wang, and Yu-Xiong Wang. Language agent tree scarch unifics reasoning acting and planning in language models. arXiv preprint arXiv:2310.04 044.0.400.444.. 2023.
31


Reviews:
[{"review_id": "hvgtVT4rhZ", "reviewer": "Reviewer_Rh6q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper provides a comprehensive analysis of Large Language Model (LLM)-based agents designed for data science tasks. The authors adopt a dual-perspective framework: \n- (1) from the agent design perspective, examining agent roles, execution structures, external knowledge integration, and reflection mechanisms; and \n- (2) from the data science perspective, analyzing how agents are applied across key workflow stages including data preprocessing, model development, evaluation, and visualization. \nThe paper also provides an extensive review of benchmarks and discusses future research opportunities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The paper's organization around both agent design principles and data science workflows is well-structured and provides a holistic view of the field. This dual lens effectively bridges general agent architectures with domain-specific applications.\n2. The paper provides detailed categorizations across multiple dimensions: Agent roles (single, two-agent, multi-agent, dynamic), Execution structures (static vs. dynamic), External knowledge methods (databases, retrieval, APIs), Reflection mechanisms (agent feedback, code error handling, unit testing, etc.)", "audience_yn": "Yes", "audience_explanation": "1. While, this paper indeed presents a lot of agent works with comprehensive taxonimies, and the contents can be useful for the beginners to learn the context of existing research.\n2.Table 4 provides an extensive compilation of 35+ benchmarks with details on their sources, number of tasks, and task types, which is highly valuable for researchers.\n3. The survey covers a broad range of recent work with detailed tables summarizing different frameworks and their characteristics (e.g., Tables 1-4), making it a valuable reference resource.", "requested_changes": "1. As a survey, this paper mainly listed some existing works and organize them well. However, it seems that there is no quantative analysis tha help people understand better for the overall trending, such as keywork anlaysis, benchmark performnace thrending vs dates, etc.\n2. Please add more insights beyond a simple categorization. For example, what are the effective methods that able to be scaled? What's the limitation of the method? Are these methods only applicable to open-source models or close-source models (i.e. will better model backbone bering better performance).\n3. Section 6 on \"Future Research Opportunity\" is underdeveloped (only 3 subsections covering ~30 lines). Given the comprehensive survey of existing work, more substantive guidance on open problems and research gaps would be valuable.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LIxNzTjFwm", "reviewer": "Reviewer_EwxZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey provides a comprehensive analysis of LLM-based agents specifically designed for data science tasks. The authors propose a dual-perspective framework to structure their review. From the \"agent perspective,\" they examine the fundamental design principles of these agents, including their roles, execution structures, knowledge integration, and reflection mechanisms. From the \"data science perspective,\" they map the application of these agents across the typical data science workflow, such as data preprocessing, model development, evaluation, and visualization. The paper's main contributions are a thorough review of recent advancements in this area and the introduction of a framework that connects general agent design with the practicalities of data science.", "claims_and_evidence_yn": "No", "claims_explanation": "Strengths :- \n\n- The survey provides a systematic breakdown of agent design principles. The \"Analysis from Agent Perspective\" section methodically addresses key components, including agent roles, execution structures, external knowledge, and reflection. For instance, in discussing \"Agent Role Design,\" the paper outlines a clear progression from \"single-agent designs, which manage all tasks independently,\" to more intricate \"two-agent systems\" and \"multi-agent systems\". This demonstrates a comprehensive approach to cataloging the current state of agent architecture.\n- The document is organized in a way that enhances readability and comprehension. A key feature is \"Figure 1: Structure of This Survey,\" which functions as a visual table of contents, clearly outlining the paper's logical flow for the reader. The authors employ direct and functional prose, making the dense technical subject matter relatively straightforward to follow.\n\n\nWeaknesses :- \n- The survey presents a very optimistic view of LLM-based agents, focusing almost exclusively on their capabilities and potential. There is no substantive discussion of the inherent limitations of current LLMs, such as their propensity for hallucination, their difficulties with complex mathematical reasoning, or the brittleness of their code generation. A more balanced review would have dedicated a section to these challenges and how current research is attempting to mitigate them.\n\n- While the survey is broad, it sometimes lacks technical depth. For instance, in the discussion of \"Reflection,\" the paper lists several techniques like \"agent feedback for self-correction, model metrics feedback for optimization, code error handling for reliability, and history window mechanisms for long-term learning\". However, it does not delve into the specific algorithms or implementation details of these mechanisms, which would be highly beneficial for a technical audience.\n\n- The paper includes a section on benchmarks and provides an extensive table of existing ones. However, it does not critically analyze the limitations of these benchmarks. For example, it doesn't discuss the potential for \"Goodhart's Law\" (where a metric ceases to be a good measure once it becomes a target) in the context of LLM agent evaluation or the challenges of creating benchmarks that truly reflect real-world data science complexity.\n\n-  The \"Future Research Opportunity\" section identifies important areas like \"Trainable architecture,\" \"Advanced Reflection Mechanisms,\" and \"Multimodal Processing\". However, these points could have been more deeply integrated with the weaknesses and gaps identified throughout the survey. A more impactful conclusion would have directly linked the identified limitations in the current state-of-the-art to these future research directions.", "audience_yn": "No", "audience_explanation": "While this paper is a well-structured and comprehensive survey, it is unlikely to be of significant interest to a substantial portion of the TMLR audience due to a fundamental mismatch in its contribution style and the journal's focus. TMLR's readership primarily seeks original, cutting-edge research that presents novel algorithms, theoretical insights, or rigorous empirical results that advance the field. This paper's central contributions--a review of existing literature and an organizational framework--are inherently descriptive rather than generative. Consequently, it lacks the deep, critical analysis and presentation of new, validated techniques that a research-focused audience expects. A researcher reading TMLR is typically looking for the next breakthrough to build upon, not a retrospective summary of what has already been established. Therefore, despite its quality as a survey, its findings are organizational rather than empirical or theoretical, making it better suited for a dedicated review journal rather than a top-tier venue for original machine learning research.", "requested_changes": "- Add a dedicated analysis of the inherent limitations of LLM agents (e.g., hallucination, brittleness, computational cost) and the shortcomings of current evaluation benchmarks to provide a more balanced and scientifically grounded perspective.\n- Move beyond cataloging different agent designs and reflection mechanisms by providing a comparative analysis of their trade-offs, helping readers understand the relative strengths, weaknesses, and ideal use cases for each approach.\n- Revise the conclusion to explicitly link the limitations and gaps identified throughout the survey to the proposed \"Future Research Opportunities,\" framing the future work as a direct and well-motivated agenda to address the field's key challenges.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zVwoEPp5L6", "reviewer": "Reviewer_cYJu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive and well-structured survey of Large Language Model (LLM)-based agents for data science. It does a good job of linking together two major views: how agents are designed (roles, execution, knowledge, reflection) and how they operate in data science workflows (data preprocessing, modeling, evaluation, and visualization). The structure is clear, the coverage of related work is strong, and the dual-perspective framework effectively captures one of the key trends in this area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. Although the paper’s claims are solidly supported by abundant references to recent studies, some sections would benefit from stronger analytical framing.", "audience_yn": "Yes", "audience_explanation": "Yes. The survey targets a timely and rapidly evolving area that aligns closely with TMLR’s audience in machine learning, AI systems, and applied research.", "requested_changes": "The paper could be improved in several aspects. First, several subsections—particularly Agent Role Design (§3.1), Execution Structure (§3.2), External Knowledge (§3.3), Reflection (§3.4), and Data Science Loop (§4.2)—should explicitly state the research questions that the surveyed studies aim to address. At the moment, the summaries describe what each work does, rather than explaining why the research was conducted or what gap it was intended to fill. Including such guiding questions would make the discussion more analytical and less descriptive.\n\nSecond, there is a noticeable lack of discussion regarding real-world or industry applications. Although the paper summarizes academic advances very well, it does not yet address how these agent systems are being—or could be—deployed in industrial settings.\n\nLastly, some sections would benefit from mathematical formalization to add precision and scientific rigor. Specifically:\nExecution Structure (§3.2) could formalize task planning and coordination (e.g., using MDP or DAG formulations).\nReflection (§3.4) could mathematically articulate iterative feedback and performance optimization.\nExternal Knowledge (§3.3) could include retrieval scoring or hybrid knowledge fusion equations.\nData Science Loop (§4.2) could describe workflow stages using composable operators under a unified optimization objective.\n\nIn all, this is a strong and timely survey. With clearer articulation of research questions, expanded discussion of industry relevance, and a math-backed analytical approach, it could become one of the most authoritative analyses of LLM-based data science agents to date.", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)