AutoData: A Multi-Agent System for Open Web Data Collection
Authors: Tianyi Ma
Yiyue Qian
Zheyuan Zhang
Zehong Wang
Xiaoye Qian
Feifan Bai
Yifan Ding
Xuwei Luo
Shinan Zhang
Keerthiram Murugesan
Chuxu Zhang
Yanfang Ye
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Agents
LLMs
Web Data Collection
Primary area: deep_learning

Abstract: The exponential growth of data-driven systems and AI technologies has intensified the demand for high-quality web-sourced datasets. 
While existing datasets have proven valuable, conventional web data collection approaches face significant limitations in terms of human effort and scalability. 
Current data collecting solutions fall into two categories: wrapper-based methods that struggle with adaptability and reproducibility, and large language model (LLM)-based approaches that incur substantial computational and financial costs. 
To address these challenges, we propose AutoData, a novel multi-agent system for Automated web Data collection, that requires minimal human intervention, i.e., only necessitating a natural language instruction specifying the desired dataset. 
In addition, AutoData is designed for a robust multi-agent architecture, featuring a novel oriented message hypergraph coordinated by a central task manager, to efficiently organize agents across research and development squads. 
Besides, we introduce a novel hypergraph cache system to advance the multi-agent collaboration process that enables efficient automated data collection and mitigates the token cost issues prevalent in existing LLM-based systems. 
Moreover, we introduce Instruct2DS, a new benchmark dataset supporting live data collection from web sources across three domains: academic, finance, and sports. 
Comprehensive evaluations over Instruct2DS and three existing benchmark datasets demonstrate AutoData's superior performance compared to baseline methods.
Case studies on challenging tasks such as picture book collection and paper extraction from surveys further validate its applicability.

AutoData: A Multi-Agent System for Open Web Data Collection
Tianyi Malt, Yiyue Qian Zheyuan Zhang', Zehong Wang', Xiaoye Qian' Feifan Bait, Yifan Ding Xuwei Luo', Shinan Zhang", Keerthiram Murugesan", Chuxu Zhang': and Yanfang Ye
tma2@nd, edu, yyqian5@gmail.cmmmm com, {zzhang42, zwang43) (nee433))dnddddie edu, gianxxaoye199@@hhmmilll com, xiaobai@uv.c edu zhangshinan@gaiil. com,
dyf01250gmail. com, com, 1uo446ppurdue edu, edu, Keerthiram Muruuessaii bm. com, chuxu. zhangguconn edu, yye7 edu University of Notre Dame, UUiiversity of Connecticut, AAmazon, Univeersity of Washington, 'Purdue University, "IBM Research
Abstract
The exponential growth of data-driven systems and AI technologies has intensified the demand for high-quality web-sourced datasets. While existing datasets have proven valuable, conventional web data collection approaches face significant limi- tations in terms of human effort and scalability. Current data-collecting solutions fall into two categories: wrapper-based methods that struggle with adaptability and reproducibility, and large language model (LLMM)-based approaches that incur sub- stantial computational and financial costs. To address these challenges, we propose AutoData. novel multi-agent system for Automated web Data collection, that requires minimal human intervention, i.c., only necessitating a natural language instruction specifying the desired dataset. In addition, AutoData is designed with a robust multi-agent architecture, featuring a novel oriented message hypergraph coordinated by central task manager, to efficiently organize agents across research and development squads. Besides, we introduce a novel hypergraph cache system advance the multi-agent collaboration process that enables efficient automated data collection and mitigates the token cost issues prevalent in existing LLM-based systems. Moreover, introduce Instruct2DS new benchmark dataset supporting live data collection from web sources across three domains: academic, finance, and sports. Comprehensive evaluations over Instruct2DS and three existing bench- mark datasets demonstrate AutoData's 's superior performance compared to baseline methods. Case studies on challenging tasks such picture book collection and paper extraction from surveys further validate its applicability. Our source code and dataset are available here!
Introduction
State-of-the-art Al models rely heavily on high high-quality data to learn patterns, forecast trends, and self-optimize. As the World Wide Web hosts an unparalleled breadth real-world signals, it has become the default reservoir for large-scale data acquisition. Landmark web-sourced data [3,10) have already accelerated research in various domains, such as natural language processing and computer vision.
Data is the fuel that powers modern data-centric intelligence systems 1
Botth authors contributed equally. Corresponding author. "The work is related to the position at corresponding institution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input
Ouiput Dataset
Collect all papers sccepted in
Manager Agent
ICML 2024. Data
Paper
Paper
Step
Step
Research Squad
Oriented HyperGraph Cache System (OHCache)
Develop Squad
1) Oriented Message Hypergraph
Plan
Tinks mas
Agents Wip
Convert
E1
Engineer
Revisc
Debug
Plan
Tasks
P2
Code
Tioks
Messages
C3
E2
Web Result
Sourch
X
7F3
Retrieve
Test
Knowledge
2) Hyperedge Formatter
Local Cache System
Dataset
ISON
Valid
BP
Blueprint
Mevsages
Cache IDs
Figure 1: The overall framework of AutoData. With the input instruction, agents in the research squad collaborate to generate a development blueprint by browsing the web pages. Afterward, the development squad builds the program based the blueprint and executes the program to obtain the desired dataset. To ensure efficient and effective multi-agent collaboration, we introduce a novel oriented hypergraph cache system, called OHCache, for information sharing. As shown in the figure, plan agent en sends the message my to agents V2, ig and Us. resulting an oriented hyperedge é1- Next, the web agent (1a) retrieves messages m and my from the oriented message hypergraph & for decision making, In addition, design hyperedge formatter formalize the agent messages and local cache system to store valuable artifacts for subsequent agents to retrieve in on-demand manners.
Yet demand for richer, domain-specific, and ever-larger datasets is exploding |. Conventional web-scraping pipelines buckle under three bottlenecks: (i) Wrapper-based crawlers are brittle, i.e., each new site layout demands time-consuming, manual rule engineering. (ii) offload the parsing logic proprietary models but incur substantial temporal and financial overhead. (iii) REST-API harvesters still require human experts to interpret each endpoint parameter and limit by rate rules. The community therefore faces pivotal question: "How can build an end-to-end fully automatic open web data collection system that simultaneously maximizes coverage, uccuracy, and efficiency?"
scrapers
In response, inspired by the rapid advancement of AI agents that equip LLMs with task-specific tools [12144) we introduce novel multi-agent system for Automatic Data collection from web sources, called AutoData. AutoData orchestrates two specialized agent squads, i.e., research and development, under a central task manager (MGR). Specifically, upon receiving data collection instruction, specialized agents in the research squad break down the instruction into small steps, follow the steps to extract knowledge from web sources, and design development blueprint. Afterward, development squad converts the blueprint into executable code, runs the program, and validates the collected data. Unlike existing multi-agent systems for general tasks that ignore the extensive token costs for open web data collection tasks, we introduce novel Oriented HyperGraph Cache System, called OHCache, for effective and cost-efficient multi-agent collaboration in open web data collection tasks. OHCache consists of (i) An oriented message hypergraph that models inter-agent message flow as oriented hyperedges. (ii) An oriented hyperedge formatter to enforce the structured communication schema and hyperedge message accumulation. (iii) A local cache system to store reusable artifacts for agents to retrieve demand.
In addition to system contribution, we identify an absence of benchmark datasets to evaluate the performance of models for open web data collection tasks. The most relevant benchmark datasets [15 16 i.e., web information extraction tasks, are merely crawled on static and archived web pages, failing to test on open web data collection. In light of this, we introduce newly collected

---

Instruction to DataSet benchmark called Instruct2DS, the first open web data collection benchmark, spanning three domains, i.e., Academic, Finance, and Sport.
Extensive experiments on Instruct2DS and three existing benchmark datasets demonstrate the ef- fectiveness and efficiency of AutoData. Moreover, additional case studies further highlight its applicability and plug-and-play adaptability. We summarize our contributions as follows:
We develop fully automatic multi-agent system, consisting of eight specialized agents and a novel oriented hypergraph cache system for efficient and effective multi-agent collaboration, that turns a one-sentence data requirement into ready-to-use dataset. New Benchmark Dataset: We introduce a newly collected benchmark Instruct2DS that covers three domains, i.e., Academic, Finance, and Sport. To the best of our knowledge, this is the first benchmark dataset to evaluate the performance of models for open web data collection tasks. Comprehensive Evaluation: We conduct comprehensive experiments on AutoData and base- line methods over Instruct2DS and three existing benchmark datasets, i.e., SWDE, EXTENDED WSDE, and HUMANEVAL. The experiment results demonstrate the effectiveness, efficiency, and applicability of AutoData in open web data collection tasks.
Novel Multi-Agent System:
2 Preliminary
Definition 2.1. Oriented Hypergraph. Let 9 {V.E} denote a hypergraph, where V (vi,-.., ,,N) is the set of nodes and & = (ei,...,eM eM eM) is the set of hyperedges. Unlike pair- wise edges in graphs (17719), each hyperedge ey & connects multiple nodes and represents complex interactions among nodes. In an oriented hypergraph 9 (V, E), each hyperedge ēj E E contains two subsets of nodes E = (S, T), where the subset S C V contains the source nodes and the subset T C V contains the target nodes to depict a direction among nodes in Ej-
Instrnct in natural languages, the objective is to design a fully automated system f [[nnirrutt W) = [S] that can effectively and efficiently collect data ps from open web sources W that meet the requirements in Instruct
Problem 1. Open Web Data Collection. Given data instruction
3 AutoData: Multi-Agent System for Web Data Collection
AutoData is a multi-agent system (MAS) that is capable of open web data collection through not only web crawling but also REST API calls (20), effectively and efficiently. Specifically, AutoData consists of eight specialized agents that are further divided into the research squad and the development squad to perform web data collection workflow collaboratively. The design of AutoData motivated by the traditional web data collection process [21 in which the individual first distills data collection logic from web sources and extracts the logic. then codes and executes the program to obtain the datasets. In this section, we first discuss the specialized agents in AutoData. Afterward, we present details of novel multi-agent collaboration mechanism called the oriented hypergraph cache system (OHCache), which is designed to optimize agent synergy and deliver unparalleled operational efficiency. The overall framework is illustrated in Figure |
3.1 Specialized Agents
Recent advances underscore the remarkable efficacy of multi-agent collaboration, particularly when agents possess diverse skills and domain expertise, in tackling complex tasks, such software development 285, science debate [29 30), and recommendation systems [3]] Inspired by this, design AutoData, which comprises eight specialized agents strategically organized into two synergistic squads, i.c., research and development, coordinated by a central manager agent (MGR), for open web data collection through adaptive agent interaction.
Research Squad. The research squad, denoted as Vies. is designed to collaboratively extract, synthesize, and operationalize knowledge critical for web data collection workflows. This group consists of four specialized agents: (i) the plan agent Gplan: (ii) the web agent Uweb: (iii) the tool agent Vool, and (iv) the blueprint agent Ubg- Each agent is entrusted with a distinct, complementary role. Specifically, the plan agent Uplan decomposes the overarching data collection objective into granular, actionable steps. The web agent Uwe autonomously navigates diverse web sources to extract pivotal
3

---

knowledge and procedural logic. The tool agent Pooo leverages advanced utilities (e.g., Google search, file conversion, and HTML cleaner) to augment the squad's operational capabilities. Moreover, the blueprint agent thp consolidates the acquired insights into coherent, actionable development blueprint, laying the foundation for downstream execution.
Development Squad. Leveraging the comprehensive blueprint synthesized by the research squad, the development squad Váev applies its coding ability to program construction, debugging, execution, and data validation. This squad comprises (i) the engineering agent Vengr. (ii) the test agent 'hest, and (iii) the validation agent Uval- The engineering agent Peagr implements the data collection pipeline in strict alignment with the prescribed blueprint. The test agent Uest is responsible for debugging and executing the program, while the validation agent Vaal designs and executes sophisticated test cases to ensure data integrity and reliability.
Manager Agent. Beyond the specialized research and development agents, introduce a central manager agent (MGR) Umgr. which orchestrates the end-to-end workflow and mediates inter-group coordination, as illustrated in Figure MGR ensures seamless task allocation, progress tracking, and dynamic adaptation across all agent roles.
Collectively, these specialized agents operate as an integrated, adaptive collective to address the multifaceted challenges of open web data collection. Each agent V is characterized by a unique profile that encapsulates its objectives, name, and operational constraints, formalized as prompt P.. Agents adhere to the ReAct paradigm [33], engaging in deliberate reasoning before action to minimize hallucinations and enhance robustness. Formally, the execution of agent U; time is:
mf = LLM(M Fi)
where m is the generated output by agent v; at time l via the function LLM(-), M, represents the message history (see Section 3.2). and F; specifies the available function tools for agent U;
3.2 Multi-Agent Collaboration
However, current MAS paradigms exhibit critical limitations that hinder their practical deployment in open web data collection scenarios: (i) Information overload from broadcast communication. Traditional MAS approaches often rely on broadcast-styl messaging, where every message is disseminated to all agents and each agent processes the entire message history for decision decision-making. This strategy not only creates significant information overhead but also increases the risk of hallucination and cognitive overload, especially as message histories grow (38) (ii) Unstructured natural language interfaces. Most existing MAS frameworks [3941 utilize unstructured natural language for inter-agent commu- nication. While flexible, this practice makes it challenging for subsequent agents to extract salient information, impeding efficient problem-solving in complex, multi-stage tasks [26). (iii) Propagation of artifact-embedded messages. In open web data collection, agents may inadvertently transmit bulky artifacts (e.g., HTML files, raw web data) within messages, cluttering the communication channel and further compounding inefficiency.
Effective communication is the linchpin of successful multi-agent collaboration 34
To address these challenges, we propose a novel oriented hypergraph cache system, called OHCache, communication architecture specifically engineered to optimize information flow, structure message content, and manage artifacts efficiently in MAS environments. Specifically, OHCache includes three components: (i) Oriented Message Hypergraph, (ii) Oriented Hyperedge Formatter, and (iii) Local Cache System. The core insight behind OHCache is that naive one-to-one or broadcast communication topologies introduce unnecessary complexity and inefficiency [35] 42 Moreover, in practice, merely a few agents require access to the same information source. For instance, both the engineering agent Vengr and validation agent Uval depend on the development blueprint generated by the blueprint agent Uhp for their respective roles in program construction and validation. Oriented hyperedges in a hypergraph elegantly capture this multi-recipient communication, enabling targeted information dissemination that reduces message redundancy and mitigates the risk of overload. Overall, OHCache streamlines communication and enhances overall system scalability and precision. Next, we present each component in OHCache in detail.
Oriented Message Hypergraph. To overcome broadcast- innrucctt--nueeddd information overload, we introduce the oriented message hypergraph G. which enables selective and efficient message routing:
G = M,, = (S,, T,) E, where Sy; T3 C V. * U,T, + 0, and S, nT, 0. (2)

---

Here, V represents the set of agent nodes in Section 3.1 i.e., Vies U Vaer U (mmger = V with size [V] 8, and message hyperedges E depict the information exchange of messages M among agents. Particularly, a hyperedge ēj = TT) E E, corresponding to the message m E M at time t, comprises source node set S C V and target node set I3 S V. The source node set Sj includes the agent responsible for sending the message, which is consistent with size 1, i.e. Vēj E E |S,] Conversely, the target node set 7, identifies the group of agents designated to receive the message. This design models real-world scenarios where message from a single agent relevant a specific subset of agents. For any agent scheduled to execute at time I, its decision context is constructed from the chronological set of messages M directed to it:
Mf {m jē (5,, E E, ET <}.
Moreover, the oriented message hypergraph 9 is dynamically augmented as new messages are recorded. In detail, after an agent execution, the generated message m via Eq. will be incorporated into the oriented message hypergraph õ through the oriented hyperedge formatter (discussed below). Notably, message delivery does not trigger immediate agent execution; instead, the MGR Umnrr orchestrates the workflow, leveraging the oriented message hypergraph ensure efficient, sequential information flow.
Oriented Hyperedge Formatter. To address the ambiguity of unstructured natural language, introduce the oriented hyperedge formatter g(-), which standardizes message content and manages hyperedge insertion. When agent v generates message m the formatter transforms it into structured message m and inserts the structured message into the hypergraph:
M' M E' = 8 ( where m (m,)) and ē. (S,,T)).
(4)
(5)
The formatter applies agent-specific rules to ensure that all critical information is explicit and machine- interpretable, facilitating downstream processing. Afterward, the oriented hyperedge formatter updates the oriented message hypergraph as g = ", In practice, the structured formats of agents are designed based on their roles in the system, and individuals are requested provide necessary information for the formatter to generate the structured message. Besides, the relationship among the message m/ i.e., hyperedge é, generated based the role-specific requirement, For instance, for a hyperedge ε that the source node set contains plan agent Uplan i.e., S = {Tptan) the target node set is T, (Uweb, Utool: Ubp. Umgr).
Local Cache System. To prevent unnecessary artifact-embeddedd messages, we devise a local cache system that serves as a special node Ules in oriented message hypergraph 9 to store necessary artifacts. For example, when the web agent Uweh encounters valuable artifact, i.e., HTML file, that assist the engineering agent Vengr for programming, node nees stores the file locally and broadcasts a concise global message containing a cache identifier. The corresponding hyperedge Ey is formulated as:
(6) This approach ensures that artifacts are accessible all agents on demand, without polluting the communication channel. Moreover, agents can subsequently retrieve artifacts by referencing the cache ID, enabling efficient artifact-aware collaboration.
Ej =S,, (S,,T)), T)), where Sy- { and T₃ V/
4 Instruct2DS: Benchmark Dataset for Open Web Data Collection
As previously discussed, there is currently no publicly available benchmark dataset tailored for data collection tasks from open web sources. To address this gap and enable comprehensive evaluation of Al agent performance, we introduce a novel instruction to dataset benchmark, termed Instruct2DS. designed specifically for web-based data collection across three domains: academie, finance, and sports. In this section, we focus on the academic domain as a representative example. More details about Instruct2DS in the other domains are provided in Appendix
4.1 Task Definition
The primary objective of Instruct2DS to assess the performance of models to complete data collection tasks from real-world web sources based on task instructions. We design symbolic
5

---

templates for the generation of a diverse set of instruction variants and adjust complexity levels to better explore the robustness capabilitiess of models in data collection tasks. Specifically, the dataset Instruct2DS contains the following components:
Database. The database D serves the authoritative data source for constructing the ground truth dataset [GT-DS] which is generated according to the specified task instruction [nstruct] The resulting ground truth dataset [GT-DS] is used as reference evaluate the quality and completeness of the dataset DS collected by the model. In the academic domain, we comprehensively collect all papers from selected conferences to populate the database. Importantly. during the data collection process, MAS methods are provided only with the task instruction [istruct] and have no access to the underlying database. Instead, these methods are required autonomously gather data directly from open web sources. An illustrative example of a paper entry in the database is shown in Figure 2
Instruction Templates. These are predefined templates T to build task instructions Instrrnct] for data collection. Figure 2 lists three instruction templates (122,33 for academic paper collection tasks. The placeholder variables P in templates T., e.g., [Conference] and [Year], etc, represent the specific requirements for the tasks.
Dataset Construction Algorithm. We further manually build an algorithm g(-) for ground truth dataset construction from the database D. Formally, given an instruction template T, E T, and corresponding placeholder variables P, the ground truth dataset GT-DS is obtained as:
GT-DS
(T,,
(7)
For example, feeding the instruction template Tj with placeholder variables "conférence==NuurPS" and "year=2017", into the algorithm g(-) will result a ground truth dataset GT-DS that contains all accepted paper in NeurlPS 2017, and example of paper is shown in Figure
A Example of Paper Information
TITLE:
Attention is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin
AUTHORS:
ABSTRACT: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attention mechanism. (skipped for clarity.)
CONFERENCE: Advances in Neural Information Processing Systems 30 (NIPS 2017)
ABBR: NeurIPS
TRACK: Main Conference Track
PAPER LINK: (Ppeer Link
BIBTEX LINK: Bibtex Link
SUPP LINK: None
Instruction Templates for Academic Paper Data Collection
T₁. Collect all papers accepted in [Conference] [Year].
T2. Collect all papers accepted in [Conference] [Year] [Track]. Ty. Collect all papers accepted in [Conference] from [Start Year] [End Year].
Figure 2: A sample in Instruct2DS and examples of instruction templates Iastrrctt
4.2
Data Collection
To ensure the long-term availability and stability of Instruct2DS, we selected three domains charac- terized by persistent and accessible data sources: academic, finance, and sports. For the academic domain, we targeted several top-tier conferences and developed custom Python web crawlers to systematically collect all papers directly from official conference websites. Subsequently, researchers performed thorough crossvvaldattio to verify the completeness, quality, and integrity of the resulting

---

Table 1: Performance comparison with baseline methods over Instruct2DS. Time indicates the period from the task instruction release to starting data collection, i.e., execute crawler program.
ACADEMIC
STOCK
SPORT
Time (L)
Expense (U)
Model
F1
Prec.
Reca.
F1
Prec.
Roca.
Prec.
Reca.
Mins
USD (S)
Human
85.57
90.63
80.81
91.66
95.62
88.95
89.50
93,40
85,46
186,98
Cline
83.50
89.65
77.12
91.37
95.20
87.17
87.07
91.38
84.53
83.29
Cursor
46
84.37
88.53
81.02
90.23
94.24
86.31
88.70
92.98
84.38
71.60
Manus
69.27
72.76
66.10
95.24
97.10
93.47
87.48
93.12
81.80
2.49
AutoAgent
48
65.79
67.67
63.84
73.54
94.37
52.71
75.72
88,30
63.62
10.46
1.28
OpenManus
49
67.39
69.54
64.40
79.75
97.30
66.85
85.74
90.21
80.45
6.84
1.08
AutoAgents
63.42
70.80
57.34
75.52
95.45
87.50
71.48
1.36
AutoData
91.85
93.74
90.51
96.75
98.37
94.17
90.14
95.63
85.28
5.58
0.57
Table 2: The performance comparison with baseline methods over SWDE and EXTENDED SWDE Exec-P. and Exce-R. denote the execution precision and recall, respectively.
SWDB
EXTENDED SWDE
Model
F1
Prec.
Reca.
Correct
Exee-P.
Exce-R.
F1
Prec.
Reca.
Correct
Exce-P.
Exce-R. R.
COT
3
76.95
87.75
79.90
61.88
12.50
7.19
65.08
85.15
68.35
56.10
2.44
7.32
Reflexion
82,40
93.28
82,76
67.50
13.75
4.37
75.85
87.39
77,81
64.81
4.18
5.57
Manus +7
89.22
93.60
88.52
73.59
14.85
4.86
75.64
81.70
79.66
63.53
4.20
6.04
AutoSrraper
88.69
92.49
89.13
71.56
14.06
5.31
76.21
82.71
80.25
64.11
3.48
6.27
AutoData
89.25
92.43
90.59
74.17
14.51
5.12
77.44
80.81
81.40
65.39
3.82
5.52
database. Afterward, validate the correctness of the dataset construction algorithm, these re- searchers further code separately, and cross-validate the algorithm by testing the dataset generated by every valid task instruction. This rigorous process guarantees both the reliability and reproducibility of Instruct2DS. More detail about data collection is provided in Appendix A
4.3 Comparison with Existing Works
Instruct2DS introduces distinctive of research challenges, advancing the development of MAS methods for real-world web environments. Unlike existing benchmark datasets (15] 16 45 [4]
Instruct2DS stands out in several key dimensions: (i) Open Web Setting: Instruct2DS requires MAS agents to interact with live, dynamic web sources for data collection, in contrast to prior datasets that restrict agents static, locally archived pages. (ii) Diverse Data Acquisition Modalities: The benchmark extends beyond traditional web crawling from HTML sources, incorporating tasks that necessitate data collection through REST API calls, thereby broadening the operational scope and testing the versatility of MAS solutions. (iii) Symbolic Information Extraction: Instruct2DS supports symbolic-style information extraction (see placeholders in Figure 2), challenging agents to extract and reason over structured representations, whereas existing benchmarks are limited to extracting fixed, predefined information from web pages. Moreover, the curated databaseInstruct2DDS serves as valuable resource for the broader research community, enabling wide range of downstream applications, including academic recommendation systems, time-series forecasting in finance, game score prediction in sports analytics, and beyond.
Experiments
To comprehensively evaluate the performance of AutoData, we conduct experiments over Our newly collected dataset Instruct2DS, IE benchmark datasets SWDE, EXTENDED SWDE, and Coding benchmark HumanEval. Implementation details about environments, evaluation metrics, and experiment setup are provided Appendix [D.T] D.2 and D.7 respectively. Additional experiments with LLM variants and error analysis are reported in Appendix E. and B.2 respectively.
7

---

5.1 Experiments over Dataset InstrucI2DS
We report the performance of AutoData and baseline methods over dataset Instruct2DS in Table The bolded numbers are the best results and the underlined numbers indicate the runner-up results. We choose existing general MAS methods as baseline methods, including Manus [47) AutoAgent [48]. OpenManus [49], and Agentt [50] Besides the general Al agents in the literature, we also report the performance data collection by human efforts, via Cline [45], and Cursor 46 in Table
Detailed discussion about baseline methods is provided in Appendix
According to the table, we find that: (i) Manual web data collection exhibits superior performance compared to most general MAS systems, albeit with the increased implementation time, which serves as the primary motivation for our research; (ii) The baseline method, Manus, demonstrates enhanced performance relative to all general MAS approaches, while it requires more time and incurs greater expenses complete the tasks; (iii) Our method AutoData surpasses all baseline methods in every domain, necessitating less implementation time and incurs lower expenses, underscoring the efficiency and cost efficiency of our proposed approach comparison to existing MAS methods.
5.2 Experiments over IE Benchmark Datasets
To further validate the effectiveness of our proposed method AutoData, we conduct experiments over IE benchmark datasets, i.e., SWDE [I5] and EXTENDED SWDE 43 44], for web crawling tasks and list the result in Table We employ the same evaluation metrics in AutoScraper [53 to assess the performance of our proposed method and baseline methods. Detailed discussion about baseline methods, evaluation metrics, and experiment settings is provided in Appendix Appendix D.2 and Appendix D.5 respectively.
According to Table 2 we find that: (i) The baseline method, Manus, demonstrates competitive performance many metries. However, its results still leave room for improvement these benchmark datasets, highlighting the benefits of developing a system specifically tailored for web crawling tasks. (ii) Our proposed method AutoData outperforms baseline methods across most evaluation criteria, demonstrating its effectiveness in web crawling tasks.
5.3 Experiments over Coding Benchmark Dataset
As our method contains a group of cod- ing agents to develop web data collection programs, we conduct another experiment over the coding benchmark dataset HU- MANEVAL [55) The experiment settings are provided in Appendix D.6 and the per- formance is listed in Table According to the table, we observe that: (i) Although AutoData is specifically designed for code generation tasks, its performance is comparable that of the baseline method MetaGPT, through the LLM backbone GPT-4. (ii) Our proposed method AutoData, paired with GPT-4, significantly outperforms GPT-4 and, when using GPT-4o the backbone, it also shows incremental improvements over GPT-4o, underscoring its effectiveness in programming.
Table 3: Performance comparison with baseline meth- ods over dataset HUMANEVAL.
Model
Pass
Model
Pass @
PaLM
54
36.0
Codex- 175B (55)
47.0
Self-Col 27
74.4
MetaGPT 26
85.9
GPT-4 56
67.0
GPT-4o 57
90.2
Ours GPT-4
86.9
Ours + GPT-4o
92.5
5.4 Ablation Study
Agents Fies
A3: w/o OHCache
cache
Vawe
96
0.9 98
1.5
We conduct ablation study to analyze the contributions agents and components OHCache, as illustrated in Figure 3
F1 Score
90
0.6 88
Cost
Effectiveness of Agents. As removing certain agents will result in unworkable codes, we keep the fundamental agents, i.e., MGR Ungrs web agent tweb, and en- gineering agent Vengrs and remove the rest of agents by squads, i.c., research squad
78
Ours
A5
Figure 3: Ablation studies for AutoData.
8

---

(res(A1) and development squad Vae (A2), separately. According to the Figure B] although lower cost, all the ablation settings perform worse than our original setting, indicating that each group of agents contributes to our multi-agent system to some extent.
Effective of OHCache. We further validate the effectiveness of each component in OHCache, i.e., the entire OHCache (A3), the oriented hyperedge formatter (A4), and the local cache system (A5), by removing it separately. First, we remove the entire OHCache (A3), which the system switches to broadcast-style communications, omits message formatting, and disables the local cache system. The notable decline in performance and the increase in cost confirm the effectiveness and cost efficiency of OHCache. Next, we remove the oriented hyperedge formatter (A4), which means that we employ a plain natural language for communication, and the performance decline indicates the effectiveness of the oriented hyperedge formatter. Then, we remove the local cache system (A5), which means that the artifacts are embedded in communication. Although performance changes are marginal, substantial cost reduction demonstrates the cost efficiency of the local cache system in our system.
5.5
Case Study
To validate the applicability of our method AutoData, we conduct two web data collection tasks for case studies, i.e., children picture book collection and paper collection from surveys. Compared to the tasks in Instruct2DS, these two tasks are more challenging, involving multi-level, in-depth HTML crawling and separate crawling of individual sources.
Children Picture Book Collection. We collaborate with researchers the educa- tion domain for children's 's picture book col- lections. Picture books are essential for young learner development, and datasets for child picture books hold significant value for contributors in the education fields [58 59). See Appendix for more discussion. Based on the request from do- main experts, we choose the Children's ' Pic-
Table 4: Human evaluation for picture book collection task in accuracy (Acc), completeness (Comp), unique- ness (Uniq), and cost.
Model
Acc. (t)
Comp. (t)
Uniq. (4)
Cost (L)
Manus
63.93
79.76
1.51
1.86
AutoData
89.58
98.13
0.00
0.91
ture Book Database at Miami University the web source, which provides collection of picture book abstracts searchable by topics, concepts, and skills for building content area literacy across all academic subjects (60). We compare our proposed system AutoData with the best baseline method Manus to collect all picture books provided in the web source. Afterward, we randomly draw 500 samples from each dataset and conduct the human evaluation. The human evaluation setup provided in Appendix DD3 and the result is provided in Table
According the table, find that (i) Our method consistently outperforms Manus in all human evaluation aspects for the picture book collection task, underscoring the effectiveness of our approach; (ii) AutoData demonstrates significantly lower data collection costs compared to Manus, highlighting its cost-efficiency; (iii) Specifically, AutoData incurs a cost of only $0.91, half that Manus, emphasizing both its applicability and cost advantage. Mention that the collected picture book dataset is shared with researchers in the education domain for further utilization.
Paper Collection from Survey. We con- duct another case study to crawl BibTeX formatted references from papers in ArXiv (Experimental HTML). To make the study more challenging and account for the fact that surveys often contain more references than regular papers, we selected five sur- veys in the MAS field [61.55]. Crawling BibTeX from surveys is highly valuable for researchers it offers an extensive, structured collection of citations that can pinpoint key works and emerging trends within the field. The experiment settings and evaluation metrics are provided in Appendix D.7 and the result is listed in Table
Table 5: Performance for paper collection from survey.
Model
F1. (t)
Prec. (*)
Reca. (t)
Cost (L)
Manus
74.70
87.96
61.52
2.55
AutoData 91.16
94.28
88.46
1.40
As shown in Table 5 AutoData consistently surpasses Manus across all evaluation metrics, under- scoring the superior effectiveness and practical applicability of our approach. (ii) Our method incurs

---

a lower overall cost compared to Manus, highlighting its enhanced cost-efficiency in challenging web data collection tasks.
Related Works
Autonomous Agents for Web Applications. Website browsers have been of the major driving forces of LLM agents, as the web sources include all the available applications with up-to-date, domain-specific, and diverse resources of information. Some of the best-known commercial web applications include AI co-scientist [66] and Manus [47]. At the similar times, open-sourced have been proposed as counter-competiooss to speed up the development of this field. However, most existing web applications are created based on general purposes, thus ignoring the specific contexts and challenges of open web data collection tasks, such the demands of token-intensive operations. As a consequence, these LLM-based web agents have visible performance gaps and redundant LLM calls. Inspired by this, the goal of this work is to develop LLM-based multi-agent system to automatically collect datasets from open web sources effectively and efficiently.
alternatives 49
48 67
Web Information Extraction. Most websites have semi-structured layouts, c.g., HTML layout, and require models with strong generalizability to understand structural and semantic contexts to extract accurate results [68]. Traditional methods mainly formulate this problem span-based information extraction tasks 69 such as named entity recognition (70) and entity linking 71], and solve it with sequential labeling [72]. These methods mainly fine-tuned pre-trained language models (PLMs). Therefore, high-quality annotated labels or weakly supervised labels are required, which limits its ability automatically understand diverse layouts of different websites. Recent work 53 leverages generative LLMs to extract web information with in-context learning or prompt tuning. However, these methods require reading every webpage for information extraction, which is expensive and inapplicable for large-scale data collection. In light of this, we propose a multi-agent system that first extracts knowledge for building data collection programs from web sources and further employs development agents to build the scripts for scalable open web data collection.
7 Conclusion
In this study, we present novel multi-agent system (MAS) called AutoData for fully automatic open web data collection. It consists of eight specialized agents that divided into research and development squads for effective and efficient open web data collection. To tackle the limitations of existing MAS methods in multi-agent collaboration for open data collection tasks, we introduce novel oriented hypergraph cache system OHCache that includes an oriented hypergraph to depict the complex message sharing process among agents, oriented hyperedge formatter to structure the messages based on agent profiles, and a local cache system to store the necessary artifacts for agents to access in on-demand manner. Extensive experiments over newly collected dataset Instruct2DS,, and three existing benchmark datasets demonstrate the effectiveness, efficiency, and applicability of our proposed method, compared with SOTA baseline methods.
Acknowledgements
The work was partially supported by the NSF under grants IIS-2533550, IIS ASS-2321504, IIS-2340346, IIS-2217239, IIS 2528400 CNS-2426514, and CMM1-2146076, ND-IBM Tech Ethics Lab Program, Notre Dame Strategic Framework Research Grant (2025), and Notre Dame Poverty Research Package (2025). Any expressed opinions, findings, and conclusions or recommendations are those of the authors and do not necessarily reflect the views of the sponsors.
10

---

References
[1] Yuqi Wang, Ke Cheng, Jiawei He, Qitai Wang, Hengchen Dai, Yuntao Chen, Fei Xia, and Zhao- Xiang Zhang. Drivingdojo dataset: Advancing interactive and knowledge-enriched driving world model. In NeurlPS, 2024.
[2] Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. In NeurIPS, 2024.
[3] Yiyue Qian, Tianyi Ma. Chuxu Zhang, and Yanfang Ye. Adaptive graph enhancement for imbalanced multi-relation graph learning. In CIKM, 2025.
[4] Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. In NeurIPS, 2024.
[5] Zheyuan Zhang, Zehong Wang, Shifu Hou, Evan Hall, Landon Bachman, Jasmine White, Vincent Galassi, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. Diet-odin: A novel framework for opioid misuse detection with interpretable dietary patterns. In KDD, 2024.
[6] Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In ICLR, 2023.
[7] Yiyue Qian, Tianyi Ma, Chuxu Zhang, and Yanfang Ye. Dual-level hypergraph contrastive learning with adaptive temperature enhancement. In WWW, 2024.
[8] Tianyi Ma, Yiyue Qian, Chuxu Zhang, and Yanfang Ye. Hypergraph contrastive learning for drug trafficking community detection. In ICDM, 2023.
[9] Zheyuan Zhang, Yiyang Li, Nhi Ha Lan Le, Zehong Wang, Tianyi Ma, Vincent Galassi, Keerthiram Murugesan, Nuno Moniz, Werner Geyer, Nitesh V Chawla, et al. Ngqa: A nutritional graph question answering benchmark for personalized health-aware nutritional reasoning, arXiv preprint arXiv:2412.155444 2024.
[10] Zehong Wang, Sidney Liu, Zheyuan Zhang, Tianyi Ma, Chuxu Zhang, and Yanfang Ye. Can Iims convert graphs to text-attributed graphs? In NACCL, 2025.
[11] Minghao Liu, Zonglin Di, Jiaheng Wei, Zhongruo Wang, Hengxiang Zhang, Ruixuan Xiao, Haoyu Wang, Jinlong Pang, Hao Chen, Ankit Shah, et al. Automatic dataset construction (adc): Sample collection, data curation, and beyond. arXiv preprint arXiv:2408. 13338, 2024.
[12] Yanming Liu, Xinyue Peng. Jiannan Cao, Shi Bo, Yuwei Zhang. Xuhong Zhang, Sheng Cheng, Xun Wang, Jianwei Yin, and Tianyu Du. Tool-planner: Task planning with clusters across multiple tools. In ICLR, 2025.
[13] Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Xuanhe Zhou, Yufei Huang. Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang. Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong. Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang. Tongshuang Wu, Heng Ji, Guoliang Li, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. ACM Comput. Surv., 57(4), December 2024. ISSN 0360-0300. doi: 10. 1111/114755 3704435
10.1 10.1145/3704435. URL
[14] Tianyi Ma, Yiyue Qian, Zehong Wang, Zheyuan Zhang, Shinan Zhang, Chuxu Zhang, and Yanfang Ye. Lim-empowered class imbalanced graph prompt learning for online drug trafficking detection. In ACL, 2025.
[15] Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. From one tree to forest: a unified solution for structured web data extraction. In SIGIR, 2011.
[16] Adi Omari, Sharon Shoham, and Eran Yahav. Synthesis of forgiving data extractors. In WSDM, pages 385-394, 2017.
11

---

[17] Tianyi Ma, Yiyue Qian, Shinan Zhang, Chuxu Zhang, and Yanfang Ye. Adaptive expansion for hypergraph learning. arXiv preprint arXiv:2502. 155644 2025.
[18] Zehong Wang, Zheyuan Zhang, Tianyi Ma, Nitesh Chawla, Chuxu Zhang, and Yanfang Ye. Towards learning generalities across graphs via task-trees. In ICML, 2025.
[19] Zehong Wang, Zheyuan Zhang, Tianyi Ma, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. Neural graph pattern machine. In ICML, 2025.
[20] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong. Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In ICLR, 2024.
[21] Ryan Mitchell. Web scraping with Python: Collecting more data from the modern web. O'Reilly Media, Inc., 2018.
[22] Daniel Glez- Peña, Anália Lourengo, Hugo López-Fermánder Miguel Reboiro- Jato,, and Flo- rentino
Web scraping technologies in api world. Briefings in bioinformatics, 15(5):788-799 797, 2014.
[23] Moaiad Ahmad Khder. Web scraping or web crawling: State of art, techniques, approaches and application. International Journal of Advances in Soft Computing & Its Applications, 13(3), 2021.
[24] Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. Datasets for large language models: A comprehensive survey. arXiv preprint arXiv:2402.18044 2024.
[25] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang. Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative agents for software development. In ACL, 2024.
[26] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024. htttt:://ppenrreeeeeeee
[27] Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Seff-collaboratiion code generation via chatgpt. ACM Transactions on Software Engineering and Methodology, 33(7):1-33 1-38, 2024.
[28] Zheyuan Zhang, Lin Ge, Hongjiang Li, Weicheng Zhu, Chuxu Zhang, and Yanfang Ye. Mapro: Recasting multi-agent prompt optimization as maximum posteriori inference. arXiv, 2025.
[29] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In ICML, 2023.
[30] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llmebased evaluators through multi-agent debate. In ICLR, 2024.
[31] Junjie Zhang, Yupeng Hou, Ruobing Xie, Wenqi Sun, Julian McAuley, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Agentef: Collaborative learning with autonomous language agents for recommender systems. In WWW, 2024.
[32] Zheyuan Zhang, Zehong Wang, Tianyi Ma, Varun Sameer Taneja, Sofia Nelson, Nhi Ha Lan Le, Keerthiram Murugesan, Mingxuan Ju, Nitesh V Chawla, Chuxu Zhang, et al. Mopi-hfrs: A multi-objective personalized health-awar food recommendation system with lim-enhanced interpretation. In KDD, 2025.
[33] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2023.
[34] Xiaohe Bo, Zeyu Zhang, Quanyu Dai, Xueyang Feng, Lei Wang, Rui Li, Xu Chen, and Ji-Rong Wen. Reflective multi-agent collaboration based on large language models. In NewrIPS, 2024.
12

---

[35] Guohao Li. Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem, Camel: Communicative agents for" mind" exploration of large scale language model society. In NeurlPS, 2023.
[36] Zheyuan Zhang, Kaiwen Shi, Zhengqing Yuan, Zehong Wang, Tianyi Ma, Keerthiram Muruge- san, Vincent Galassi, Chuxu Zhang, and Yanfang Ye. Agentrouter: A knowledge-grappe guided Ilm router for collaborative multi-agent question answering. arXiv, 2025.
[37] Kaiwen Shi, Zheyuan Zhang, Zhengqing Yuan, Keerthiram Murugesan, Vincent Galass, Chuxu Zhang, and Yanfang Ye. Ng-router: Graph-supervised multi-agent collaboration for nutrition question answering. arXiv, 2025.
[38] Genki Kusano, Kosuke Akimoto, and Kumihiro Takeoka. Are longer prompts always bet- ter? prompt selection in large language models for recommendation systems. arXiv preprint arXiv:2412. /4454, 2024.
[39] Xu Li, Mingming Sun, and Ping Li. Multi-agent discussion mechanism for natural language generation. In AAAI, 2019.
[40] Qingyun Wu, Gagan Bansal, Jieyu Zhang. Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang. Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. arXiv preprint arXiv:2308.0811 2023.
[41] Joon Sung Park, Joseph O' O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In UIST, 2023.
[42] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. In ICLR, 2024.
[43] Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. Openceres: When open information extraction meets the semi-structured web. In NAACI, 2019.
[44] Colin Lockard, Prashant Shiralkar, Xin Luna Dong, and Hannaneh Hajishirzi. Zeroshotceres: Zero-shot relation extraction from semi-structured webpages. In ACL, 2020.
[45] Cline Team. Cline, the collaborative ai coder.
https: http:://cccine bot/ 2024.
[46] Cursor Team. Cursor ide. https://cursor.com com 2025.
[47] Manus Team. Leave it manus. https://manus.ing 2025.
[48] Jiabin Tang, Tianyu Fan, and Chao Huang. Autoagent: A fully-automated and zero-code framework for Ilm agents. arXiv e-prints, pages arXiv-2502, 2025.
[49] Xinbin Liang, Jinyu Xiang, Zhaoyang Yu, Jiayi Zhang, Sirui Hong, Sheng Fan, and Xiao Tang. Openmanus: An open-source framework for building general ai agents, 2025. URL
https://doi..rg// htt.://..r//...../eeeeme//
5001/22001020/ 15186407
[50] Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Sesay Jaward, Karlsson Börje. Jie Fu, and Yemin Shi. Autoagents: The automatic agents generation framework. In IJCAI, 2024.
[51] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models, In NeurlPS, 2022.
[52] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In NewrIPS, 2023.
[53] Wenhao Huang, Zhouhong Gu, Chenghao Peng, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Liqian Wen, and Zulong Chen. Autoscraper: A progressive understanding web agent for web scraper generation. In EMNLP. 2024.
13

---

[54] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):
1-113, 2023.
[55] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (rXiv:2107.03374, 2021.
[56] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08 08774, 2023.
[57] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[58] Maria Nikolajeva. Verbal and visual literacy: the role of picturebooks in the reading experience of young children. Handbook of early childhood literacy, pages 235-248, 2003.
[59] Rudine Sims Bishop. Mirrors, windows, and sliding glass doors. perspectives: Choosing and using books for the classroom, 6 (3). Perspectives: Choosing and using books for the classroom, 6(3):ixxii, 1990.
[60] Children's picture book database at miami university. htttps://dl..ii: mi amich edu/ picturebook/ 2007.
[61] Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.13501, 2024.
[62] Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. From Ilms to Ilm-based agents for software engineering: survey of current, challenges and future. arXiv preprint arXiv:2408. 02479, 2024.
[63] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey progress and challenges. arXiv preprint w:Xi::2402.01680 2024.
[64] Lijun Sun, Yijun Yang, Qiqi Duan, Yuhui Shi, Chao Lyu, Yu-Cheng Chang, Chin-Teng Lin, and Yang Shen. Multi-agent coordination across diverse applications: A survey. arXiv preprint arXiv:2502. 14743, 2025.
[65] Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. The landscape of emerging agent architectures for reasoning, planning, and tool calling: survey, arXiv preprint 2404v:2404. /584, 2024.
[66] Google Research. Accelerating scientific breakthroughs with an ai co-scientist, 2024.
[67] Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. In NewrIPS, 2023.
[68] Alireza Mika. AutoScraper: A smart, automatic, fast and lightweight web scraper for python. http::://gghuu..cmmm/rrrreaaaai autoscraper) 2022.
[69] Yifan Ding, Michael Yankoski, and Tim Weninger. Span-oriented information extraction-
unifying perspective on information extraction. arXiv preprint arXiv:2403.15453. 2024.
[70] Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A survey on deep learning for named entity recognition. IEEE transactions on knowledge and data engineering, 34(1):50-70, 2020.
[71] Wei Shen, Jianyong Wang, and Jiawei Han. Entity linking with knowledge base: Issues, techniques, and solutions. IEEE Tronsactions on Knowledge and Data Engineering, 27(2): 443-460, 2014.
14

---

[72] Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional LSTM-CNNs-
CRF. In Katrin Erk and Noah A. Smith, editors, ACL, 2016.
[73] Jeremy Siegel and Jeremy D Schwartz. The Long-term Returns on the Original S 500 Firms. Rodney L. White Center for Financial Research, 2004.
[74] Xiao- Yang Liu, Hongyang Yang. Qian Chen, Runjia Zhang, Liuqing Yang, Bowen Xiao, and Christina Dan Wang. Finrl: A deep reinforcement learning library for automated stock trading in quantitative finance. In NeurlPS, 2022.
15

---

Appendix: Table of Contents
New Dataset Instruct2DS
17
A.1
Overview
17
A.2
Academic
18
A.3
Finance
18
Sports
18
B
Benchmark Datasets
19
C
Baseline Methods
19
D
Implementation Details
23
D.1
Environments
23
D.2
Evaluation Metrics
23
D.3
Human Evaluation Setup for Case Study
23
D.4
Experiment Setup for Dataset Instruct2DS
23
D.5
Experiment Setup for Benchmark Datasets SWDE and EXTENDED SWDE
24
D.6
Experiment Setup for Benchmark Dataset HUMANEVAL
24
D.7
Experiment Setup for Case Study Paper Collection from Survey
E
Additional Experiments
24
E.1
Performance of AutoData with different LLMs
24
E.2
Error Analysis
24
Additional Discussion for Children Picture Book Collection
25
G
Limitations
25
H
Broader Impacts
25
16

---

A New Dataset InstrucI2DS
A.1 Overview
To comprchensively evaluate the performance of Al agents for data collection from open web sources, we build a new Instruction to DataSet benchmark, called Instruct2DS. The goal this benchmark dataset is to access the Al agent performance in accomplishing complex and multi-steps tasks in web data collection. Benchmark dataset Instruct2DS contains comprehensive data from various domains, which we will discuss in the following sections. All data in the benchmark dataset Instruct2DS is collected manually, and cross-validated by researchers to ensure the data completeness, quality, and integrity. These data serve
anchor database for ground truth dataset construction based on the instruction Instruct Given [nstruct] as input, Al agents follow the instruction to collect dataset DS from web sources. Besides, benchmark dataset constructs [T.DS]] according the Instruct 1 and we access the performance of the model by comparing the collected dataset [DS] with the ground truth
dataset GT-DS Mention that, all models merely take Instruct as input, and have access the data in Instruct2DS in the collection process. The constructed ground truth dataset GT-DS is merely used
for evaluation, An example of Instruct and GT-DS is shown in Figure
Task Instruction
Collect all accepted papers in NeurIPS 2017 Main Conference Track.
A Sample in Ground Truth Dataset
TITLE:
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin
AUTHORS:
ABSTRACT: The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in encoder and decoder configuration. The best perform- ing such models also connect the encoder and decoder through an attentionn echanisms. We propose novel, simple network architecture based solely onan attention mechanism, dispens- ing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27.5 BLEU EaEnnliss-tt-Grrmmae result by over BLEU. On English-to- French translation, we outperform the previoussingle state-of-the- art with model by 0.7 BLEU, achieving BLEU score of 41.1.
translation, improving over the existing best ensemble
CONFERENCE: Advances in Neural Information Processing Systems 30 (NIPS 2017)
ABBR: NeurIPS
TRACK: Main Conference Track
PAPER_ LINK: (Paeer Link
BIBTEX LINK: Bibtex Link
SUPP_LIN: None
Figure 4: Example of instruction
Instruct and a sample in corresponding GTTDS
There are several concerns in benchmark dataset construction:
Static of Web Sources: Due to the continuous evolution of web content, ground truth data may change time, potentially undermining the reproducibility of experimental results. For example, on social media platforms, posts can be edited or deleted, leading to inconsistencies in the collected ground truth across different periods.
Ethical and privacy concern: Data collection from the open web must adhere to ethical guidelines and respect user privacy. Sensitive or personally identifiable information should be excluded to prevent ethical violations and ensure compliance with relevant data protection regulations.
17

---

Based on the aforementioned concerns, we choose three domains that provides static web sources to ensure reproducibility, i.e., academic, finance, and sports. Next, we will discuss each domain detail.
A.2 Academic
We choose nine conferences in three fields, ie., machine learning (NeurIPS, ICLR, and ICML), natural language processing (ACL, EMNLP, and NAACL), and computer vision (CVPR, ICCV, and ECCV), and collect all paper data in these conferences, following the data structure provided in Figure 5 papers, define three instruction templates (1,2.3) to build instructions [Instruct & listed in Figure 5 With these instruction templates, we implement ground truth dataset construction algorithm to obtain the ground truth dataset GT-DS from Instruct2DS that meets instruction requirements, i.c., [Conference], [Year], [Track], [Start Year] and [End Year].
To systematically evaluate the performance of models for data collection tasks in academic
Academic Paper Data Structure
TITLE: # Paper title.
AUTHORS: # Author names, separated by comma.
ABSTRACT: # Paper abstract.
CONFERENCE: # Full conference name.
ABBR: # Abbr of the conference name, i.e., NeurIPS.
TRACK: Conference track of the paper.
PAPER LINK:
The permanent paper link provided by conference.
BIBTEX LINK: # The permanent bibtex link provided by conference.
LINK: # The permanent supplemental source link provided by conference.
Instruction Templates for Academic Paper Data Collection
Ti. Collect all papers accepted in [Conference] [Year].
T2. Collect all papers accepted in [Conference] [Year] [Track].
T3. Collect all papers accepted in [Conference] from [Start Year] to [End Year].
Figure 5: Academic paper data structure and instruction templates.
A.3
Finance
In addition to academic papers, our benchmark dataset incorporates financial market data from the S&P 500 index. The dataset spans comprehensive time period from January 1st, 2015, December 31st, 2024, capturing daily trading information for all S&P 500 constituent companies 73 The stock dataset includes both standard OHLCV (Open, High, Low, Close, Volume) data 774 and corporate action-adjusted attributes, including adjusted opening price (adjOpen), adjusted daily high (adjHigh), adjusted daily low (adjLow), adjusted closing price (adjClose), adjusted trading volume (adjVolume) and cash dividends per share (divCash). The complete data structure and instruction templates are presented Figure Our ground truth dataset construction algorithm follows the instruction Instruct to obtain the ground truth dataset GT-DS that meets the requirements, including [Stock], [Year]. [Start Date], [End Date], [Stocks], and [Date].
A.4 Sports
Sport another data source that meets our requirements. We build benchmark dataset among two popular sport league, including National Baseketball Association (NBA) and Major League Baseball (MLB).
NBA. Our benchmark NBA dataset is sourced from the official NBA website] The dataset encom- passes comprehensive statisties from 1946 to 2024, covering two distinct levels: team player. For
www.NBA.com
18

---

Stock Market Data Structure Team Level
Date: The date format YYYY-MM-DD
Open: # Open trading day price.
High:
# Highest price within the trading day.
Low: # Lowest price within the trading day.
Close: Close trading day price.
Volume: Total number of shares traded within the trading day.
adjOpen: # Open trading day price, adjusted by coporate actions.
adjHiigh: Highest price within the trading day, adjusted by coporate actions.
adLow: # Lowest price within the trading day, adjusted by coporate actions.
adjCCose: # Close trading day price, adjusted by coporate actions.
adjVolume: # Total number of shares traded within the trading day, adjusted for stock splits.
diyCash: The cash dividend distrbuted per share.
Instruction Templates for Stock Market Data Collection
T.. Collect daily stock information for [Stock] [Year].
Tç. Collect daily stock information for [Stock] between [Start Date] and [End Datel.
To. Collect daily stock information for [Stocks] on [Date].
Figure 6: Stock data structure and instruction templates.
consistency and comparability, focus exclusively regular season data, excluding pre-season and playoff games, as not all teams participate in these additional competitions. The data structure for team and player levels, and corresponding instruction templates are provided in Figure 7 and Figure respectively.
MLB. Similarly, dataset is obtained from the official MLB websitc] The dataset encompasses comprehensive statistics from 1876 to 2024, covering team-level statistics from two aspects: hitting and pitching. Again, merely focus regular-season data. The data structure for hitting and pitching and corresponding instruction templates are provided in Figure Soureeaadd Figure 10 respectively.
B Benchmark Datasets
SWDE. [5] SWDE is a large, real-world collection of web pages created to support research on the automatic extraction of structured data-such attribute-value pairs of entities-from the web. contains webpages from 80 websites in 8 domains, with 124,291 webpages. Each of the websites from the same domains focuses 3-5 attributes in the web pages.
Extended SWDE. 43 44] This extension provides additional annotations for 21 websites across domains, labeling all binary relations involving the main entity on each page, rather than just limited set of attributes as in the original SWDE. The new benchmark includes both the extracted object values and the predicate strings, offering a richer and more comprehensive ground truth for evaluating extraction systems.
HumanEval. (55) The HumanEval dataset a benchmark developed by OpenAl to evaluate the code generation capabilities of LLMs. It consists of 164 hand-crafted problems and has become standard reference for assessing how well models can understand, reason about, and generate correct code solutions for programming tasks.
C Baseline Methods
To evaluate the effectiveness of our proposed methods, compare AutoData with baseline methods in the literature and industry, which can be categorized into fthe ollowing groups
https//www.mlb.com/
19

---

NBA Data Structure Team Level
TEAM NAME:
# The team city with team name.
YEAR:
# The calendar year of the season, i.., 1949 means the season 1949-50.
GP:
Games Played this Year.
W: Total Wins in this Year.
L: Total Losses in this Year.
FGM: Field Goals Made.
FGA: # Field Goals Attempted.
FG3M: # Three-Point Field Goals Made.
FG3A: # Three-Point Field Goals Attempted.
FTM: # Free Throws Made.
FTA: # Free Throws Attempted.
OREB: # Offensive Rebounds.
DREB: # Defensive Rebounds.
REB: Total Rebounds.
AST: # Assists.
TOV:
Turnovers.
STL: # Steals.
BLK: Blocks.
PF: # Personal Fouls.
PTS: # Total Points Scored.
Instruction Templates for NBA Team Level Data Collection
T7. Collect NBA [Team Name] all regular season stats until 2024. Tg. Collect NBA [Team Name] regular season stats from [Start Year] to [End Year]. Tg. Collect all NBA regular season stats in [Year].
Figure 7: NBA data structure at the team level and instruction templates.
Genetic Multi-Agent Systems. We choose four SOTA multi-agent systems industry and literature as baseline methods.
Manus Manus an Al agent designed to assist with wide range of tasks, including information gathering, coding, process automation, and general problem-solving using computer and internet resources. Given the high costs of Manus, we conducted our experiments on representative subset of tasks to ensure cost-effectivenes: while maintaining experimental validity. Manus employs proprietary credit-based pricing system rather than conventional token-based billing. In accordance with their pricing structure, we quantify the expenses using credits, with each credit unit valued $0.01.
AutoAgent, 48 It introduces natural language-driven system that enables non-technical users to automatically create, customize, and deploy L.LM agents through conversational commands, eliminating programming requirements via its modular architecture and self-managing workflows.
Open [49 It is an open-source general multi-agent system to solve complex tasks. We obtain their source code from Githhb
AutoAgents. [50] It is framework that dynamically generates, coordinates, and refines multiple specialized agents with distinct roles to form collaborative Al teams tailored to specific tasks, enabling adaptive and coherent multi-agent problem-solving.
Programming Assistants and Human Programming. Conventional methods for data collection involve programming to efficiently collect raw data from web sources and clean up the raw data to ready-to-use dataset [ We employ existing programming assistant methods in the industry for data collection, which include:
Cursor. 446 Al-empowered code editor that acts as coding assistant, helping developers to work efficiently.
20

---

NBA Data Structure Player Level
PLAYER NAME:
The full name of the player.
YEAR: # The year which the statistics are recorded.
BIRTHDATE: # The birthdate of the player.
COUNTRY: # The country of origin.
HEIGHT: # The height of the player (feet-inch).
WEIGHT: The weight of the player (pounds).
DRAFT YEAR: # The year the player was drafted into the NBA.
DRAFT ROUND: # The round in which the player was drafted.
DRAFT NUMBER: # The overall draft pick number for the player.
TEAM ABBR: # The Abbr Team Name player associated with during YEAR.
GP: Games Played.
GS: # Games Started Number of games the player started.
MIN: Total Played Minutes.
FGM: Field Goals Made.
FGA: Field Goals Attempted.
FG3M: # Three-Point Field Goals Made.
FG3A: Three-Point Field Goals Attempted.
FTM: # Free Throws Made.
FTA: # Free Throws Attempted.
OREB: # Offensive Rebounds.
DREB: # Defensive Rebounds.
REB; Total Rebounds.
AST: # Assists.
STL; # Steals.
BLK: # Blocks.
TOV: Turnovers.
PF: # Personal Fouls.
PTS: # Points.
Instruction Templates for NBA Player Level Data Collection
Too- Collect NBA player [Name] all regular season stats until 2024. Tn. Collect NBA player [Name] regular season stats from [Start Year] to [End Year].
T12. Collect NBA players [Names] regular season stats in [Year].
Figure 8: NBA data structure at player level and instruction template.
Cline is an open-source AI coding assistant (Copilot) with dual Plan/Act modes, terminal execution, and Model Context Protocol (MCP) for VS Code]
Cline. 45
Human. Besides programming assistants in industry, also conduct data collection via human efforts as a baseline method. Specifically, we require three experts in web crawling to manually program data collection scripts obtain the datasets nn based the instructions Instruct]. Similar to Manus, we leverage a representative subset of tasks to ensure labor effectiveness while maintaining experimental validity.
For each data collection task, we feed the instructions to the programming assistant to generate data collection Python scripts. Afterward, we run the code for data collection output. Note that since these methods typically lack direct access to web sources for coding, besides the dataset instructions, also supply programming assistants with guidelines, including target data sources preferred collection methods, such as web crawling or REST APIs
Web Crawler. We also compare our method with three baseline methods over IE benchmark datasets,
including SWDE [5] and EXTENDED SWDE 43 44
https://coe.vvisualle vist.sssiiiiiiuallstdioooo.oo
21

---

MLB Data Structure Hitting
TEAM NAME: # The full name of the team.
TEAM ABBR: # The abbreviation of the team.
YEAR: # The calendar year of the season.
AB: # The number of official at-bats.
R: # The number of runs scored.
HR: # The number of home runs.
H: The number of hits reaching least first base.
2B: # The number of doubles (reaching second base).
3B: The number of triples (reaching third base).
RBI: # The number of runs batted in.
BB: The number of walks (bases on balls).
SO: # The number of strikeouts.
SB: # The number stolen bases.
CS: # The number of times caught stealing.
Instruction Templates for MLB Hitting Data Collection
T13- Collect MLB [Team] all regular season hitting stats until 2024. T14- Collect MLB [Team] regular season hitting stats from [Start Year] to [End Year].
T15. Collect all MLB teams regular season hitting stats in [Year].
Figure 9: MLB hitting data structure and instruction templates.
MLB Data Structure Pitching
TEAM NAME: # The full name of the team.
TEAM ABBR:
# The abbreviation of the team.
YEAR: # The calendar year of the season.
W: # The number of wins.
L: The number of losses.
ERA: The earned run average (runs allowed per nine innings).
G: The number of games pitched.
GS: The number of games started.
CG: # The number of complete games pitched.
SHO: The number of shutouts pitched.
SV: # The number of saves.
IP: # The number of innings pitched.
H: # The number of hits allowed.
R: The number of runs allowed.
ER: The number of earned runs allowed.
HR: # The number of home runs allowed.
HB: # The number of hit batters.
BB: The number of walks allowed (bases on balls).
SO: # The number of strikeouts.
Instruction Templates for MLB Pitching Data Collection
Ti6- Collect MLB [Team] all regular season pitching stats until 2024. Ti7. Collect MLB [Team] regular season pitch stats from [Start Year] [End Year].
T Collect all MLB teams' regular season pitching stats in [Year].
Figure 10: MLB pitching data structure and instruction templates.
22

---

AutoScraper. [53] An SOTA LLM-powwed powered framework that generates adaptable web scrapers through hierarchical HTML analysis and progressive learning.
COT. [51] This work introduces -f-thuhht prompting for LLMs to decompose the problems into intermediate reasoning steps to improve the performance of LLMs over complex tasks.
Reflexion. [52] It is framework that enables language agents to improve their decision-min making and task performance by generating and storing verbal self-reflections based on feedback from their actions, allowing them to iteratively learn from experience without updating model weights.
D Implementation Details
D.1 Environments
All experiments are conducted on Linux servers equipped with four Nvidia A40 GPUs. The models are implemented using PyTorch 2.4.0 with CUDA 12.1 and Python 3.11.5. We choose GPT-4o (56 as our default LLM backbone for baseline methods and AutoData.
D.2 Evaluation Metrics
As our focus is on textual data collection, we utilize standard evaluation metrics commonly employed in information extraction tasks, i.c., Fl score, precision, and recall. Furthermore, we also report the time needed to finish the tasks. Mention that, do not consider the time in program execution, it highly uncontrollable due to the several facts, e.g., web sources, Rest API call limits, etc. Instead, we focus the time that involve human efforts, which is the period from the task instruction release to start data collection. For benchmark datasets, i.e. SWDE [5] and EXTENDED SWDE 43 besides the standard evaluation metrics, also employ execution evaluation metrics [53], which include correct, execution precision (Exec-Prec, Prec.,, and execution recall (Exec-Reca.). We treat each attribute within the data samples as individual unit, evaluating and reporting the mean scores across the ground truth dataset GT-DS] and collected data DS] To maintain consistency in numerical comparisons, round all numerical attributes to four decimal places.
D.3 Human Evaluation Setup for Case Study
We collaborate with researchers in the education domain for case study children's picture book collections. With the collected dataset, we randomly draw 500 samples and conduct a comprehensive human evaluation based on the following aspects:
Completeness: Whether the sample contains all the attributes provided in the web source.
Accuracy: Whether the value of each attribute correctly reflects the corresponding information on the web source.
Uniqueness: Whether there is any unwanted duplication in the attribute values, ensuring that each attribute represents unique information.
We regard each aspect binary classification task, and three researchers evaluate the result separately. Afterward, we conduct a majority vote among three researchers as the final annotation result.
D.4 Experiment Setup for Dataset Instruct2DS
To ensure comprehensive evaluation across different domains, we sample valid tasks from each instruction template, where a valid task is defined as one that yields non-empty results containing the necessary data samples specified in the instruction. Specifically, for academic paper collection, we generated distinct tasks per conference, resulting tasks. Similarly, for stock market data collection, we created 27 unique tasks per template, leading 81 stock market collection tasks. In the sports domain, we sampled valid tasks for each instruction template, producing 72 sports-related data collection tasks. This sampling strategy resulted in diverse evaluation set comprising 234 unique tasks spanning three distinct domains within our dataset Instruct2DS.
total of 81 academic paper collection
23

---

D.5 Experiment Setup for Benchmark Datasets SWDE and EXTENDED SWDE
For experiments over IE benchmark datasets, i.e, SWDE [5] and EXTENDED SWDE 43 we follow the setup in AutoScraper [53] that chooses three seed webpages for models to identify knowledge for web crawler programming. and the rest for testing. We strictly follow the source code of AutoScraper to reproduce the experiment result. Furthermore, to ensure fair comparison with AutoScraper, we also employ GPT4-Turbo in our experiments for methods, with the exception of Manus, where the choice of backbone LLMs is not available.
D.6 Experiment Setup for Benchmark Dataset HUMANEVAL
We follow the settings in MetaGPT [26] to reproduce the experiment results. For AutoData, we simply pass the coding task instruction to the manager agent and request it to manage the workflow among research and development squads for code generation. Moreover, similar to MetaGPT, we slightly modify the prompt align with the format requirements in the dataset address format-specific issues, i.c., Python problems.
D.7 Experiment Setup for Case Study Paper Collection from Survey
For the survey- paper case study, we reuse the precision, recall, and F1 metries defined Appendix [D.2] Due to the absent of authoritative reference set, two researchers independently extracted every BibTeX entry cited in the target surveys, then cross-validate ensure the correctness of the BibTex. We use the manually collected BibТex as ground truth dataset GT-DS to evaluate the collected dataset DS via AutoData and the baseline method Manus.
E
Additional Experiments
E.1 Performance of AutoData with different LLMs
To further validate the performance of AutoData with different LLM backbones, randomly sample one task description from each unique instruction template in dataset Instruct2DS, and evaluate the performance of L.M backbones, i.e., GPT-4o, GPT-4o-mini, DeepSeek R1 70B, and DeepSeek R1 450B. Besides, for better comparison, we also provide the performance of the best baseline method, Manus in T
According to the table, we find that: (i) For simple tasks, such as STOCK and SPORT, all LLM variants perform similarly, but GPT-4o mini achieves faster execution times and lower costs. (ii) The open-source LLM, DeepSeek RI 70B, shows the lowest performance among all LLMs; however, still performs comparably to the best baseline method, Manus. (iii) GPT-4o offers the best overall performance in exchange for a relatively high expense among all LLMs.
Table The performance of AutoData with different LLMs backbone over Instruct2DS
ACADEMIC
STOCK
SPORT
Time (L)
Expense (1)
Model
F1
Prec.
Reca.
Fl
Prec.
Reca.
Fl
Prec.
Reca.
Mins
USD ($)
Manus
69.27
72.76
66.10
95.24
97.10
93.47
93.12
15.37
DeepSeek RI 70B
78.33
85.45
72.43
94.64
96.90
91.83
85.75
90.94
80.63
13.30
DeepSeck RI 450B
90.20
92.52
88.19
95.74
98.46
93.26
87.70
94.27
80.36
4.45
0.14
GPT-4o-mini
86.97
90.32
83.10
95.84
97.36
93.94
88.18
93.97
83.40
3.71
0.01
GPTT-4
91.85
93.74
90.51
96.75
98.37
94.17
90.14
95.63
85.28
5.58
0.57
E.2 Error Analysis
We perform analysis by looking the collected datasets DS via AutoData with ground truth datasets GT-DS), and identify the following common failure modes.
Unclear Conference Name and Track. Some conferences do not expose the conference name or track on individual paper pages. For instance, in NL.P venues, the page only lists "Volume" string, e.g., "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics

---

(Volume 1: Long Papers)", and the track information is implicitly encoded inside that field. In these cases AutoData misses or mis-parses the attributes. More task-specific prompts could help. but as our goal is general-purpose MAS for open web data collection, we do not hard-code such heuristics.
Data Collection Approaches. Although the system can both crawl HTML and call REST APIs, it occasionally chooses the less reliable route. For MLB statistics, both (i) direct HTML scraping from URL and (ii) the well-maintained third-party APIs are viable. AutoData sometimes opts for crawling, which is more error-prone, instead of the cleaner API path. Better decision policies or explicit user hints may alleviate this issue.
F
Additional Discussion for Children Picture Book Collection
Picture books, with their creative imagery and engaging narratives, help young learners connect reading with visualization, allowing them to make sense of text through images. Picture books can also serve as powerful tools for students explore the world that they live in [58]. Moreover, an earlier study [59] discusses how picture books can function as windows, mirrors, and sliding doors, providing glimpses into the lives of others, reflecting readers' own identities, and opening pathways to new perspectives.
The dataset holds significant value for educators, authors, illustrators, and publishers, serving as a powerful resource for guiding content selection and shaping future publications. Developing a database to identify picture books aligned with specific teaching themes offers an essential tool for analyzing recurring themes in children literature. This not only supports educators in selecting books that promote positive impact in the classroom, but also helps content creators, such as authors, illustrators or publishers, recognize gaps in the field and develop more relevant, representative content.
G Limitations
Despite the strong empirical results, AutoData still has several limitations. First, the entire pipeline depends on large-scale proprietary language models (GPT-4 / GPT-4o in our experiments). Any change in pricing, rate limits, or model quality immediately affects latency, reproducibility, and cost, the same for the proprietary baseline method, i.e., Manus. Although our abstraction layer allows the use of open-weigh weight models such as DeepSeek-R1, the superior performance is not guaranteed, as illustrated in Appendix E.T Second, our benchmark and case studies focus on publicly accessible HTML pages or via REST APIs. Websites rendered entirely client-side, protected by login walls, CAPTCHAs, or aggressive anti-bot policies, remain largely out of reach. Incorporating a headless- browser agent and credential- management modules is left for future work. Third, the current system assumes that the requested data is legally retrievable. Consequently, there is a risk of unintentionally violating copyright contractual constraints. Users must be mindful of ethical concerns, such as bias in the data, and ensure responsible use of the AutoData in different applications. Fourth, our proposed system merely focuses on text-centric data collection. Multimodal assets (images, audio, video) sources are not supported, and we leave it as further work.
H Broader Impacts
AutoData's ability to turn single sentence into a curated dataset carries both promising benefits and non-trivial risks. On the positive side, the system lowers the barrier to data acquisition for researchers, educators, journalists, and NGOs that lack extensive engineering support. By automating repetitive scraping and cleaning, frees human time for higher-level analysis and could accelerate meta-studies or reproducibility audits in scientific fields. Our picture-book case study illustrates how domain experts can quickly build specialised corpora that would otherwise take weeks of manual effort. Economically. the framework may increase efficiency and foster new data-driven products, but it also threatens traditional web-scraping or data- anntaaanntttiion jobs. Carefully designing re-skilling programs will be important if adoption scales. There are, however, serious dual-use concerns. Uncontrolled harvesting could capture personal data, pay-walled content, or copyrighted media, exposing operators and downstream users to legal liability. A determined adversary might exploit AutoData to assemble large disinformation corpora or to feed models that generate convincing deepfakes. The environmental footprint is another worry: multi-agent LLM inference, if run at web-scale, consumes non-negligible
25

---

energy. To mitigate these issues, before we release the code, we will opt-in following safeguards: (i) an extensible module that parses common licenses, issuing warnings or blocking nom-compliant requests; (ii) a PII detector that redacts sensitive attributes before storage; (iii) rate-limiting and respect for API quotas; and (iv) auditable provenance logs through OHCache so third parties can verify data origin. We also share only task descriptions and checksums of collected artifacts by default, leaving raw dumps under their original licenses. Ultimately, we hope the community will treat AutoData as research platform-uselee for exploring responsible automation, but requiring continuous refinement legal, ethical, and technical safeguards before it can deployed safely at scale.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
No theoretical part in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer:
[Yes]
Justification:
See link in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwiiiccguie//eeeeeeeeeis CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
29

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: httpp///////rrss..//// wwwpurips.ccpuuuue http://ss.....u.....hhccccculcaa
Answer:
[Yes]
Justification:
We strictly follow the NeurIPS guidance.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
30

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: See Appendix H
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all of essential tools and papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
31

---

Answer: [Yes]
Justification: We have well documented and provided it alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not relevant work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
32

---

Justification:
The role of LLMs in this paper is only related to polish.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htte:pas//eerrpppocccc//o/eeeees for what should or should not be described.
33


Reviews:
[{"review_id": "wtyeqZe2NY", "reviewer": "Reviewer_ddG4", "summary": "This paper proposes AutoData, a novel multi-agent system for automated open web data collection. It introduces a structured communication framework called OHCache to reduce message redundancy and improve collaboration efficiency, and presents Instruct2DS, a new benchmark spanning academic, finance, and sports domains. Experimental results demonstrate that AutoData achieves superior performance in accuracy, cost-efficiency, and execution time compared to strong baselines across multiple datasets and real-world tasks. The authors also explicitly acknowledge ethical and privacy concerns in web data collection and design their benchmark accordingly.", "questions": "1. Could you provide more details on the “cost” metric used in the experiments? For example, how much would it approximately cost to collect 1,000 samples using GPT-4? Is the data collection cost sensitive to the nature or structure of different web sources?\n\n2. Could the proposed OHCache framework be generalized to other multi-agent domains beyond data collection, such as tool-using or decision-making systems?\n\n3. What is the failure rate or error distribution of AutoData in real-world web scraping scenarios not represented in the Instruct2DS benchmark?", "limitations": "While the authors briefly acknowledge certain limitations—such as web content volatility and ethical/privacy concerns—in the appendix, these issues are not sufficiently addressed in the main paper. Critical points that deserve more discussion include: \n**Societal Impact:** The potential for misuse (e.g., scraping sensitive or copyrighted content) and alignment with ethical scraping norms (e.g., robots.txt, rate limits) is not explored in depth.\n\nThe authors are encouraged to integrate a dedicated section in the main paper on limitations and societal impact. Concrete discussion on compliance with data usage policies, safety checks during scraping, and value alignment would improve the paper’s transparency and trustworthiness.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths \n1. Strong Practical Motivation: The paper targets a highly practical and relevant problem—web data collection—which remains a complicated and labor-intensive workload for developers.\n\n2. Effective System Design: The proposed system integrates multiple techniques to address key bottlenecks in multi-agent system (MAS) design, notably in improving both effectiveness and efficiency.\n\n3. New Benchmark (Instruct2DS): The introduction of a live, domain-diverse benchmark is a substantial contribution, helping to fill the gap in realistic evaluation datasets for open web scraping tasks.\n\n### Weaknesses\n\n1. Data Quality and Coverage: The practicality of AutoData may be limited by real-world data diversity and quality issues. This could stem from differences in regional data collection policies, as well as engineering challenges such as deduplication, detoxification, safety filtering, and value alignment.\n\n2. Limited Discussion of Failure Modes: While ethical and volatility-related concerns are briefly acknowledged in the appendix, the main paper lacks an in-depth discussion on how the system handles adversarial, highly dynamic, or JavaScript-heavy web environments.", "was_revised": true, "final_justification": "The authors provided a clear rebuttal that addresses most of my concerns. However, the main paper still lacks sufficient discussion on data quality and coverage, which are critical to the system’s practical utility and reliability. Considering this limitation, I prefer to keep my original score.", "rebuttal": "We appreciate the reviewer's thoughtful and valuable feedback. We are glad to see the reviewer recognize novelty of AutoData, our strong practical motivation, effective model design, and the contribution of our newly introduced dataset.\n\n> **Q1: Data Quality and Coverage: The practicality of AutoData may be limited by real-world data diversity and quality issues. This could stem from differences in regional data collection policies, as well as engineering challenges such as deduplication, detoxification, safety filtering, and value alignment.**\n\nWe thank the reviewer for raising this question. We acknowledge current limitations in safety filtering, toxicity detection, and value alignment. AutoData benefits from two layers of built-in protection: (1) our workflow begins with search engines that employ \"safe search\" filtering by default, and (2) our backbone LLM, i.e., GPT-4o, incorporates extensive human alignment training. These safeguards provide a baseline level of content safety and value alignment. Moreover, we want to emphasize that **these challenges represent fundamental obstacles faced by all web crawling systems.** \n\nBesides the built-in protection, AutoData incorporates several mechanisms to address these data quality and diversity challenges. Specifically, our test and validation agent in AutoData can handle the multi-level quality checks, including schema validation, completeness verification, and deduplication. As demonstrated in column Uniqueness Table 4 for the case study, AutoData is free of duplication issues.  \n\nWe would like to further clarify that AutoData usage should strictly follow the NeurIPS Code of Ethics and the regional data collection policies. While this may hinder AutoData's practicality to some extent, we prioritize ethical and legitimate practices over comprehensive coverage. Moreover, AutoData should be used only for **research or non-commercial purposes**. We hope our method can contribute to not only the **traditional AI/ML community** but also **research domains relevant to trustworthiness.** \n\n> **Q2: Limited Discussion of Failure Modes.**\n\nThank you for pointing out the question regarding failure modes. As a research paper, we mainly focus on methodology. The solutions to handle failure modes belong to development details, which we omit in the submission. To alleviate your concern, we will provide a detailed discussion as follows.\n\nRegarding adversarial environments, we may extend AutoData with **an authentic agent** to interact with the user for assistance to pass CAPTCHAs, or request authentication requirements, e.g., OAuth, form-based login, API keys, etc. Moreover, this agent would integrate a secure credential storage solution to manage user-provided credentials safely. We can leverage headless browser solutions, e.g., Playwright, Selenium, etc., for highly dynamic or JavaScript-heavy web sources to render pages in a real browser environment for data collection. Again, while technical solutions exist, AutoData usage should strictly align with the NeurIPS Code of Ethics and the legitimate requirements. We will include these discussions in our next revision.\n\n> **Q3:Could you provide more details on the “cost” metric used in the experiments? For example, how much would it approximately cost to collect 1,000 samples using GPT-4? Is the data collection cost sensitive to the nature or structure of different web sources?**\n\nWe appreciate your question. The expense column in Table 1 is reported based on the average cost per task. As discussed in Appendix D.1, we choose GPT-4o as our default LLM backbone for AutoData, and all baseline methods, except Manus. We utilize the OpenAI dashboard to monitor the cost. For Manus, it employs a proprietary credit-based pricing system rather than conventional token-based billing. Following their pricing structure, we quantify the expenses using credits, with each credit unit valued at $0.01.\n\nWe do not have measurement for the cost to collect 1,000 samples using GPT-4o, as (i) the number of samples is dependent on the tasks; (ii) the cost in our method is mainly on explore crawling logics for program development, and if we have finalized the crawling program for a specific task, the number of samples to collect is independent to the cost. For task-level expense, the average cost is $0.57.\n\nThe data collection cost is sensitive to the structure of different web sources. Specifically, the main token usage is reading the webpages to extract crawling logic for programming. For academic papers, the web page is roughly 60K tokens, while for Sport-related tasks, the corresponding webpage is only 18K tokens.\n\n> **Q4: Could the proposed OHCache framework be generalized to other multi-agent domains beyond data collection, such as tool-using or decision-making systems?**\n\nWe thank the reviewer for recognizing the broader potential of OHCache. **OHCache framework is applicable to domains beyond open web data collection** with minimal adaptation: (i) Defining agent roles and communication patterns; (ii) Specifying message formats for the domain; and (iii) Identifying reusable artifacts to cache. \n\nFor instance, in a financial trading system, specialized agents would include market monitors (tracking prices), news analyzers (processing events), technical analysts (computing indicators and analysis information), risk managers (evaluating exposure), and execution agents (placing trades). When the news agent identifies breaking information, OHCache creates a directed hyperedge from {news agent} → {technical analyst, risk manager}, ensuring only relevant agents receive the message while excluding others, e.g., the execution agent, who don't need raw news. The cache system would store computationally expensive artifacts like historical market data, processed news summaries, and technical indicators, preventing redundant calculations and API calls. This selective message sharing and caching would reduce both latency and computational costs compared to broadcast-based communications.\n\n\n> **Q5: What is the failure rate or error distribution of AutoData in real-world web scraping scenarios not represented in the Instruct2DS benchmark?**\n\nThank you for the question. To alleviate your concern, we provide the error distribution in our case studies that examine AutoData in real-world web scraping beyond Instruct2DS. \n\n| Method | Parsing Error | Structure Error | Access Error | \n| --- |  --- |--- |--- |\n| Manus | 10.89% | 6.75% | **0%**|\n| AutoData | **6.37%** |  **0.27 %** |  **0%** |\n\nThe table above categorizes errors into three types: parsing errors (incorrect attribute values), structure errors (missing attributes), and access errors (caused by rate limits or connection refusal). The results show that AutoData consistently achieves lower error rates than Manus across all evaluation criteria, demonstrating AutoData's superior performance in real-world open web data collection tasks.\n\n\n> **Q6: While the authors briefly acknowledge certain limitations—such as web content volatility and ethical/privacy concerns—in the appendix, these issues are not sufficiently addressed in the main paper. Critical points that deserve more discussion include: Societal Impact: The potential for misuse (e.g., scraping sensitive or copyrighted content) and alignment with ethical scraping norms (e.g., robots.txt, rate limits) is not explored in depth. The authors are encouraged to integrate a dedicated section in the main paper on limitations and societal impact. Concrete discussion on compliance with data usage policies, safety checks during scraping, and value alignment would improve the paper’s transparency and trustworthiness.**\n\nThank you for your suggestions. Due to the limited space in the main paper, we leave the discussion about limitations and the broader social impact in the appendix. In our next revision, we will extend a detailed discussion in our main paper as follows. \n\nWhile AutoData offers a significant practical solution to collect data from open web sources, it remains in the scope of **common academic practice**, and should be used only for **research and non-commercial purposes**. We hope AutoData can contribute to not only the traditional AI/ML community but also research domains relevant to trustworthiness. \n\nMoreover, we acknowledge several critical societal and ethical considerations that must be addressed:\n\n**Data Usage Compliance**: our released version incorporates built-in safeguards to respect website policies, including an extensible module in the research squad that parses common licenses, robots.txt files, implementation with respect to rate limits, and adherence to API usage guidelines when available. \n\n**Privacy and Sensitivity Protection**: To ensure protection of privacy and sensitive information, our released source code includes a personally identifiable information (PII) detector that automatically identifies and redacts PII before data storage. We strongly discourage users from targeting sensitive or private content. \n\n**Copyright and Legal Compliance**: The system provides warnings when encountering potentially copyrighted material and maintains detailed logs through our OHCache system to ensure data traceability and accountability. \n\nAlthough the aforementioned features may hinder AutoData in performance, cost efficiency, or robustness, we prioritize ethical practices over comprehensive coverage. We strictly follow the NeurIPS Code of Ethics and regional legislations, i.e., never attempting to break security measures, only accessing content the user has legitimate rights to access, respecting rate limits, and server resources. Notice that the objective of Autodata is to automate **legitimate open web data collection**, not to circumvent security measures."}, {"review_id": "VUwkDpxjWj", "reviewer": "Reviewer_2gdK", "summary": "This paper introduces AutoData, a fully automated multi-agent system for web-scale data collection from open sources. AutoData organizes agents into two functional squads, research and development. They are coordinated by a manager agent and connected through a novel communication architecture called OHCache (Oriented Hypergraph Cache). OHCache formalizes agent messaging as a directed hypergraph, which enables structured, scoped, and efficient communication. The paper also contributes Instruct2DS, a new benchmark for open web data collection spanning academic, finance, and sports domains. AutoData outperforms both LLM-based and traditional scraper baselines across accuracy, cost, and time.", "questions": "- How are role-to-target mappings maintained or adapted? Are they hard-coded or learned over time?\n- Could the OHCache framework generalize to other tasks beyond web data collection? Which part of the design would require major changes?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed method encodes inter-agent messaging as oriented hyperedges, which allows targeted message delivery and avoid message broadcasting.\n- It introduces token-aware artifact offloading via local cache + cache ID routing, which effectively avoids context bloating \n- The proposed method demonstrates reduced cost and execution time in addition to strong empirical performance.\n\nWeaknesses:\n- Agent-specific contexts are not uncommon in multi-agent system design, which limits the novelty.\n- How the system handles underspecified or vague user instructions and out-of-domain webpages isn’t deeply explored, which raises doubts about its robustness.\n- It’s unclear how the manager agent prioritizes conflicting or simultaneous message paths.", "was_revised": true, "final_justification": "I think the rebuttal addressed my minor concerns about novelty and unstructured user input. Therefore I will maintain my original positive ratings.", "rebuttal": "We thank the reviewer’s valuable and constructive comments. We appreciate the reviewer's recognition of our key contributions, particularly the design of an oriented message hypergraph for efficient inter-agent communication, the artifact caching mechanism that addresses context bloating challenges, and the demonstrated improvements in both computational efficiency and empirical performance. In this rebuttal, we aim to alleviate your concerns as follows.\n\n> **Q1: Agent-specific contexts are not uncommon in multi-agent system design, which limits the novelty.**\n\nWe would like to clarify that OHCache serves as a communication architecture rooted in **a novel data structure**, called the Oriented Message Hypergraph, to store and retrieve messages or cache files. Every component in OHCache is necessary to address three critical limitations in MAS for open-web data collection tasks: (i) **information overload from broadcast messaging**, (ii) **ineffective unstructured natural language interface**, and (iii) **propagation of unnecessary artifact-embedded messages**. \n\nSpecifically, to tackle the information overload issue, unlike existing works, e.g., AgentPrune [1] and G-Designer [2], that employ pairwise graphs for agent communication, we design a novel data structure called Oriented Message Hypergraph to enable one-to-many message sharing. \n\nExisting works [3, 4] commonly employ natural language for communication, which is ineffective in message sharing. To handle this challenge, AutoData leverages the oriented hypergraph formatter to structure the messages into JSON. The structured messages allow subsequent agents to capture salient information. Moreover, the oriented hypergraph formatter is responsible for inserting the message into the oriented message hypergraph based on the pre-defined relationship. \n\nTo prevent the token waste in the propagation of artifact-embedded messages, we introduce a local cache system to store artifacts so that all agents are able to access cached artifacts on demand, without polluting the communication channel.\n\nWe further emphasize the contributions of our work: \n\n1. We introduce a fully automatic multi-agent system, called AutoData, that consists of eight specialized agents and a novel oriented hypergraph cache system for efficient and effective multi-agent collaboration, that turns a **one-sentence data requirement into a ready-to-use dataset.**\n2. We introduce a newly collected benchmark,k Instruct2DS, the **first benchmark dataset** to evaluate the performance of models for open web data collection tasks.\n\n> **Q2: How the system handles underspecified or vague user instructions and out-of-domain webpages isn’t deeply explored, which raises doubts about its robustness.**\n\nWe appreciate the reviewer's valuable comments. For underspecified or vague user instructions, AutoData can extend with a simple interface agent on top of the current system to interact with users to clarify the underspecified or vague requirements. This feature is similar to the design in Manus and OpenAI Deep Research. Specifically, interface agent **is activated** when the user's instruction is unclear. This agent interacts with the user to clarify unclear requirements. After refining the instruction, AutoData **remains automatic for open web data collection** with no human effort. In our experiments, since Instruct2DS has **well-defined instructions**, and for **a rigorous and fair comparison** with baseline methods, we do not include this agent.\n\nTo validate the robustness of AutoData beyond the Instruct2DS benchmark, we have conducted two case studies, i.e., children's picture book collection and paper collection from the survey in Section 5.5, to assess the performance of AutoData in domains not included in the benchmark. These two tasks are **more challenging**, involving multi-level, in-depth HTML crawling and separate crawling of individual sources. We compared it with the industry-level method, i.e., Manus. The results demonstrate AutoData's applicability, cost efficiency, and robustness.\n\n> **Q3: It’s unclear how the manager agent prioritizes conflicting or simultaneous message paths**\n\nAutoData employs a sequential agent workflow, as each agent's output influences subsequent workflow decisions. The manager agent serves as the central orchestrator, i.e., determines the agent to execute next based on previous results and messages stored in OHCache. OHCache functions purely as a message-sharing and caching system, which means message delivery itself does not trigger agent execution. This separation ensures the manager agent maintains full control over the execution flow while OHCache optimizes communication among agents. Thank you for the question. We will clarify the workflow of AutoData in our next revision.\n\n> **Q4: How are role-to-target mappings maintained or adapted? Are they hard-coded or learned over time?**\n\nOur role-to-target mappings are simply hard-coded. During development, we have explored agents to dynamically select message recipients. However, this approach hinders agents from performing their primary tasks and increases system complexity without corresponding benefits. Our current role-based message sharing already achieves superior performance compared to baseline methods while maintaining simplicity. The predefined communication patterns ensure agents remain focused on their specialized functions rather than communication logistics, resulting in both better task performance and lower computational overhead.\n\n> **Q5: Could the OHCache framework generalize to other tasks beyond web data collection? Which part of the design would require major changes?**\n\nOHCache framework **is applicable to domains beyond open web data collection** with minimal adaptation: (i) Defining agent roles and communication patterns; (ii) Specifying message formats for the domain; and (iii) Identifying reusable artifacts to cache. \n\nFor instance, in a financial trading system, specialized agents would include a market monitor (tracking prices), a news analyzer (processing events), technical analysts (computing indicators/ analysis information), a risk manager (evaluating exposure), and an execution agent (placing trades). When the news agent identifies breaking information, OHCache creates a directed hyperedge from {news agent} → {technical analyst, risk manager}, ensuring only relevant agents receive the message while excluding others (e.g., execution agent) who don't need raw news. The cache system would store computationally expensive artifacts like historical market data, processed news summaries, and technical indicators, preventing redundant calculations and API calls. This selective message sharing and caching would reduce both latency and computational costs compared to broadcast-based communications.\n\n\n[1] Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems. ICLR’25\n\n[2] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks. ICLR’25 Workshop.\n\n[3] CAMEL: Communicative Agents for \"Mind\" Exploration of Large Language Model Society. NeurIPS’23.\n\n[4] Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding. Arxiv."}, {"review_id": "DaBatsbezQ", "reviewer": "Reviewer_8H7h", "summary": "The research introduces AutoData, a novel multi-agent system designed for automated web data collection that minimizes human input, requiring only a natural language instruction. This system features a robust multi-agent architecture, coordinated by a central task manager, which efficiently organizes agents into research and development squads. A key innovation is the Oriented HyperGraph Cache System (OHCache), which enhances collaboration among agents and significantly reduces token costs commonly associated with existing Large Language Model (LLM)-based systems. To validate its effectiveness, the authors also present Instruct2DS, a new benchmark dataset supporting live data collection from various web sources, demonstrating AutoData's superior performance in terms of efficiency and accuracy compared to current methods.", "questions": "1. The paper indicates a dependency on proprietary LLMs like GPT-4 and GPT-4o, noting that the superior performance with open-weight models (such as DeepSeek-R1) is not guaranteed. Given that GPT-4o offers the best overall performance at a relatively high expense, are there specific plans to further optimize AutoData's architecture or conduct targeted fine-tuning on open-weight LLMs to reduce reliance on proprietary models while maintaining competitive performance and improving cost-efficiency for wider adoption?\n2.  The current system is limited to text-centric data collection and does not support multimodal assets like images, audio, or video. What is the roadmap for extending AutoData's capabilities to include diverse multimodal data collection, and would this require a significant overhaul of the existing agent architecture or the introduction of new specialized agents?\n3. The error analysis points to two key failure modes: mis-parsing attributes when information is implicitly encoded or conference names/tracks are unclear, and sometimes opting for less reliable HTML scraping even when cleaner REST API options are available. How do the authors plan to enhance AutoData's reasoning and decision policies to overcome these issues, particularly in distinguishing optimal data acquisition methods (e.g., API vs. scraping) and handling ambiguous or implicitly structured web content?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths - \n1. AutoData features a robust multi-agent architecture with a novel oriented message hypergraph coordinated by a central task manager that efficiently organizes agents across research and development squads. The system is composed of eight specialized agents, strategically organized into two synergistic squads (research and development), coordinated by a central manager agent (MGR). \n2. AutoData addresses the significant token costs prevalent in existing Large Language Model (LLM)-based systems through its novel Oriented HyperGraph Cache System (OHCache). \n3. The paper introduces Instruct2DS, the first open web data collection benchmark, which addresses a significant gap in evaluating AI agent performance for such tasks. This dataset spans three domains (academic, finance, and sports). Instruct2DS is distinctive because it requires MAS agents to interact with live, dynamic web sources, supports diverse data acquisition modalities (including REST API calls), and enables symbolic-style information extraction. \n\nweaknesses - \n1. The current system primarily focuses on publicly accessible HTML pages or REST APIs. It struggles with websites that are entirely client-side rendered, protected by login walls, CAPTCHAs, or aggressive anti-bot policies. \n2. The proposed system is currently limited to text-centric data collection and does not support multimodal assets such as images, audio, or video. \n3. There is a risk of unintentionally violating copyright or contractual constraints as the system assumes requested data is legally retrievable. Users must be mindful of ethical concerns and responsible use, as uncontrolled harvesting could lead to legal liability. \n4. Running multi-agent LLM inference at a web-scale consumes non-negligible energy, raising environmental concerns.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We are pleased that the reviewer recognizes the innovative contributions of AutoData and our benchmark contribution of Instruct2DS, i.e., the first open web data collection benchmark. In this rebuttal, we aim to address the reviewer’s concerns as follows.\n\n> **Q1: The current system primarily focuses on publicly accessible HTML pages or REST APIs. It struggles with websites that are entirely client-side rendered, protected by login walls, CAPTCHAs, or aggressive anti-bot policies.**\n\nWe acknowledge AutoData's limitations when encountering websites with strong anti-crawling protections. In our submission, to strictly follow the NeurIPS Code of Ethics, we limit our approach to publicly accessible webpages and REST APIs that explicitly permit automated access, prioritizing ethical practices over comprehensive coverage.\n\nIf beyond the scope of submission, we are aware that technical solutions exist to collect data among entirely client-side rendered, protected by login walls, CAPTCHAs, or aggressive anti-bot policies. Specifically, to access login-protected content, we can simply extend AutoData with **an authentic agent to interact with the user to request authentication requirements**, e.g., OAuth, form-based login, or API keys, etc. Moreover, this agent would integrate a secure credential storage solution to manage user-provided credentials safely. For CAPTCHA, we advocate an ethical approach that respects website security measures. That is, when AutoData encounters a CAPTCHA, it will pause automation and **request human assistance.** For websites that are client-side rendered or contain aggressive anti-bot policies, we can leverage headless browser solutions, e.g., Playwright, and Selenium, etc, to **render pages in a real browser environment for data collection.**\n\nWe would like to clarify that our proposed AutoData currently remains in the scope of **common academic practice**, and should be used only for research or non-commercial purposes. We strictly follow the NeurIPS Code of Ethics and regional legislations, i.e., never attempting to break security measures, only accessing content the user has legitimate rights to access, respecting rate limits, and server resources. We acknowledge that some web sources will remain inaccessible if they have strong anti-automation policies, and this is by design. The objective of Autodata is to automate **legitimate data collection**, not to circumvent security measures.\n\n\n> **Q2: The proposed system is currently limited to text-centric data collection and does not support multimodal assets such as images, audio, or video (originally W2&Q2).**\n\nThank you for your valuable feedback. As pioneering work in open-source web data collection methodologies, we focus on textual data to establish foundational principles and demonstrate core capabilities.\n\nExtending AutoData to multimedia domains would necessitate substantial architectural modifications, including specialized agents, tools, and multimodal LLM backbones. Specifically, A minimum overhaul would require: (i) **a media agent within the research squad to identify and evaluate multimedia content against user specifications**, (ii) **specialized tools for multimedia processing and storage**, and (iii) **multimodal LLM architectures capable across media and textual modalities.**\n\nBeyond these technical requirements, multimedia data collection introduces several unique challenges that distinguish it from text-based approaches:\n\n**Format Heterogeneity.**  Unlike textual data, which predominantly uses standardized encodings (UTF-8, ASCII), multimedia content exhibits significant format diversity. Images alone encompass various formats, e.g., JPEG, WebP, SVG, AVIF, and GIF, etc. Each requires distinct parsing algorithms, quality assessment metrics, and storage optimization strategies. Furthermore, content delivery networks often serve different formats based on client capabilities (browser type, device specifications, bandwidth), necessitating format detection and adaptive conversion pipelines.\n\n**Deduplication Complexity.** Multimedia deduplication presents substantially greater complexity than text-based approaches. Identical content may appear across multiple resolutions, compression levels, and watermarks, etc. Traditional hash-based deduplication becomes inadequate, requiring more advanced deduplication solutions.\n\n**Legal and Ethical Complexities.** While text-centric collection raises important legal and ethical considerations, media domains exponentially increase these concerns. Media content often involves complex copyright ownership, personality rights, and privacy implications. Notice that several source websites may lack clear licensing information, creating cascading legal uncertainties that are difficult to resolve programmatically.\n\nThese challenges represent active areas of research requiring expertise across multiple domains. While we remain optimistic about future solutions, addressing these complexities is beyond the scope of the current work.\n\n> **Q3: There is a risk of unintentionally violating copyright or contractual constraints as the system assumes requested data is legally retrievable. Users must be mindful of ethical concerns and responsible use, as uncontrolled harvesting could lead to legal liability.**\n\nWe appreciate the reviewer for pointing out this concern. As discussed previously, AutoData remains in the scope of **common academic practice** and should be used only for **research or non-commercial purposes**. Moreover, AutoData can be easily extended to include (i) **an extensible module that parses common licenses, issuing warnings or blocking non-compliant requests**; (ii) **employ a PII detector to prevent collecting sensitive attributes before storage**; (iii) **auditable provenance logs through OHCache so users can verify data origin**, to tackle the aforementioned challenges. We already discussed these solutions in Appendix H.  \n\n> **Q4: Running multi-agent LLM inference at a web-scale consumes non-negligible energy, raising environmental concerns.**\n\nWe agree that energy consumption is indeed a concern for web-scale multi-agent systems. AutoData addresses this through two key design choices: (i) the OHCache system minimizes redundant LLM calls through **efficient information sharing** with an oriented message hypergraph and artifact caching. Experiment over Instruct2DS demonstrates that AuoData reduces token usage by 47% compared with OpenManus (lowest cost among baseline methods); and (ii) AutoData generates **executable code for data collection rather than using LLMs** to process each webpage individually, which further reduces energy consumption.\n\n\n> **Q5: The paper indicates a dependency on proprietary LLMs like GPT-4 and GPT-4o, noting that the superior performance with open-weight models (such as DeepSeek-R1) is not guaranteed. Given that GPT-4o offers the best overall performance at a relatively high expense, are there specific plans to further optimize AutoData's architecture or conduct targeted fine-tuning on open-weight LLMs to reduce reliance on proprietary models while maintaining competitive performance and improving cost-efficiency for wider adoption?**\n\nThank you for this observation. During the development process, we have obtained extensive data consisting of agent-specific input-output pairs that capture the requisite reasoning patterns for web data collection tasks. Open-weight models (e.g., DeepSeek-R1) fine-tuned over our collected gold agent input-output pairs could achieve competitive performance with GPT-4o, addressing the quality disparities observed in our experiments. This represents a promising direction to reduce the reliance on proprietary models and maintain cost-efficient and competitive performance. We regard this direction as our future work.\n\n> **Q6: The error analysis points to two key failure modes: mis-parsing attributes when information is implicitly encoded or conference names/tracks are unclear, and sometimes opting for less reliable HTML scraping even when cleaner REST API options are available. How do the authors plan to enhance AutoData's reasoning and decision policies to overcome these issues, particularly in distinguishing optimal data acquisition methods (e.g., API vs. scraping) and handling ambiguous or implicitly structured web content?**\n\n**Deciding optimal data acquisition methods.** \nAutoData can handle this issue by adding a simple interface agent on top of the current system to interact with users to clarify their preferences for data acquisition methods. This feature is similar to the design in Manus and OpenAI Deep Research. Specifically, interface agent **is only activated** when the user's instruction is unclear or multiple collection methods are available. This agent interacts with the user to clarify unclear instructions and chooses among valid data acquisition methods (The trade-off analysis, e.g., cost, accuracy, and latency, is also provided by the agent). After refining the instruction and clarify data collection method, AutoData **remains automatic for open web data collection** with no human effort. In our experiments, we do not include this agent for **a rigorous and fair comparison** with baseline methods.\n\n**Handling ambiguous or implicitly structured web content.** Intuitively, a more task-specific prompt may alleviate the issue. However, as our goal is to build a general-purpose MAS for open web data collection, we do not hard-code such heuristics. In the long term, as discussed previously, we plan to fine-tune open-weight LLMs, e.g., DeepSeek R1, with our collected agent input-output pairs, which can explicitly tackle these errors."}, {"review_id": "gbuvl3fM3p", "reviewer": "Reviewer_ehRe", "summary": "The paper presents a multi-agent system for collecting data from the web. The proposed MAS has three components, a Research Squad which handles the job for decomposing the instruction into subgoals and forming plans to extract information from the web; a Development Squad takes the plan and try to write code to execute the plan (including testing and validation); and a Manager agent which orchestrates the entire workflow. The MAS protocol is implemented as an Oriented Message Hypergraph which is claimed to be more efficient in terms of cost and empirically have better performance. Considering the difficulty of evaluating the open web data collection task, the paper also proposes a new benchmark, Instruct2DS. Empirical results on the Instruct2DS benchmark and also additional Information Extraction benchmark show that the proposed AutoData outperformed general Web or Coding agents such as Manus and Cursor.", "questions": "1. In Ablation setting 3, when removing the graph structure, how does the agent know how to communicate and who to perform first? What’s exactly the implementation of this ablation here?", "limitations": "The authors did not specifically discussed about limitations; please see weakness for suggestions on improvements", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The paper is well-written with comprehensive evaluations.\n2. The proposed benchmark is a good contribution: especially the setting where we consider collecting the latest data from the web is both realistic and useful. The impact can extend to never-ending learning of recent large foundation models.\n3. The empirical results against mainstream web/code agents is promising, especially considering a faster turn around time and lower expense\n\n# Weakness\n\n1. Some important comparison for showing the necessity and effectiveness of the proposed MAS frame seems unclear. Specifically, in Table 1, one important comparison should be comparing previous/baseline agentic frameworks equipped with the same backbone model (i.e., GPT-4o) as the AutoData. It is unclear what backbone models are used for most baselines in the table. \n  - Also a very related prior work is not compared with nor mentioned: “INFOGENT: An Agent-Based Framework for Web Information Aggregation” (https://arxiv.org/pdf/2410.19054)\n\n2. I feel there is potentially an overcomplication / a lack of novelty in the Hypergraph and Cache System. If I understand correctly, all nodes and edges are predefined. Thus, simply keeping a shared information pool + each agent takes only partial information from the pool should work? This is not a new idea which has been investigated in multiple previous work, such as “Meta-Prompting:Enhancing Language Models with Task-Agnostic Scaffolding” (https://arxiv.org/pdf/2401.12954) and “Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks” (https://arxiv.org/pdf/2501.11733)\n\n3. See question 1.", "was_revised": true, "final_justification": "I thank the authors for the detailed rebuttal and additional analysis; I think my major questions are addressed; I will keep my already positive rating.", "rebuttal": "We sincerely thank the reviewer for the detailed and thoughtful evaluation of our work. We are pleased to hear that the paper is well-written, our experiments are comprehensive, and the reviewer recognizes the contribution of our new benchmark dataset and the promising empirical results against mainstream web/code agents. In this rebuttal, we aim to answer your questions as follows.\n\n> **Q1: Some important comparison for showing the necessity and effectiveness of the proposed MAS frame seems unclear. Specifically, in Table 1, one important comparison should be comparing previous/baseline agentic frameworks equipped with the same backbone model (i.e., GPT-4o) as the AutoData. It is unclear what backbone models are used for most baselines in the table.**\n\nWe appreciate your valuable feedback. We choose GPT-4o as our default LLM backbone for AutoData and **all baseline methods except Manus**. For Manus, we do not have options to choose the backbone model, and we use the default model provided by Manus. Thank you for your suggestion. We will clarify this in our next revision.\n\n> **Q2: Performance comparison with prior work INFOGENT: An Agent-Based Framework for Web Information Aggregation.**\n\nThank you for pointing out INFOGENT. We compare it with AutoData over our benchmark dataset, Instruct2DS, and report the performance in Table 1. \n\nTable 1. Performance comparison (F1) between INFOGENT and AutoData over Instruct2DS.\n| Model | ACADEMIC | STOCK | SPORT  |\n|---|---|---|---|\n| INFOGENT (API) | --  | 68.39 | 63.40| \n| INFOGENT (Visual) | -- | -- | -- |\n| AutoData | 91.85 | 96.75 | 90.14 |\n\nWe compare AutoData with INFOGENT in direct API and interactive visual modes. According to Table 1, INFOGENT in both modes fails on tasks in the academic domain. For tasks in stock and sport domains, AutoData has superior performance to INFOGENT, which again demonstrates the effectiveness of our proposed method in open-web data collection tasks.\n\n\n\n> **Q3: I feel there is potentially an overcomplication / a lack of novelty in the Hypergraph and Cache System. If I understand correctly, all nodes and edges are predefined. Thus, simply keeping a shared information pool + each agent takes only partial information from the pool should work? This is not a new idea which has been investigated in multiple previous work, such as “Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding” and “Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks”.**\n\n\nThank you for your valuable comments. We would like to clarify that OHCache serves as a communication architecture rooted in **a novel data structure**, called the Oriented Message Hypergraph, to store and retrieve messages or cache files. OHCache introduces **a new perspective** to view the multi-agent collaboration process as an oriented hypergraph. We believe every component in OHCache **is necessary** to address three critical limitations in MAS for open-web data collection tasks:\n\n**Information overload from broadcast messaging.** Unlike existing works, e.g., AgentPrune [1] and G-Designer [2], that employ pairwise graphs for agent communication, we design a novel data structure called Oriented Message Hypergraph to enable one-to-many message sharing. \n\n**Ineffective unstructured natural language interface.** Existing works, e.g., Meta-Prompting and Mobile-Agent-E, commonly employ natural language for communication, which may be ineffective in message sharing. To handle this challenge, AutoData leverages the oriented hypergraph formatter to structure the messages into JSON format. The structured messages allow subsequent agents to capture salient information for efficient complex problem-solving. Moreover, the oriented hypergraph formatter is responsible for inserting the message into the oriented message hypergraph based on the pre-defined relationship. During development, we have tested agents to select message recipients dynamically. However, this approach **hinders agents from performing their primary tasks** and **increases system complexity without corresponding benefits**. Our current role-based solution already achieves superior performance compared to baseline methods while maintaining simplicity. \n\n**Propagation of unnecessary artifact-embedded messages.** We introduce a local cache system to store artifacts so that all agents can access these artifacts on demand, without polluting the communication channel and preventing the token waste in the propagation of artifact-embedded messages.\n\nThe ablation study in Section 5.4 **validates the necessity of each component in our OHCache**. As illustrated in Figure 3, the performance drop and cost increase by removing the entire OHCache, or a partial mechanism, i.e., the formatter and cache system, demonstrate the necessity of each component in OHCache.\n\nWe further emphasize the contributions of our work as follows: \n1. We design a **fully automatic multi-agent system** called AutoData for open web data collection tasks, which consists of eight specialized agents and a novel oriented hypergraph cache system **for efficient and effective multi-agent collaboration.** \n2. We introduce a newly collected benchmark Instruct2DS, which is **first benchmark dataset** to evaluate the performance of models in open web data collection settings.\n\n\n> **Q4: In Ablation setting 3, when removing the graph structure, how does the agent know how to communicate and who to perform first? What’s exactly the implementation of this ablation here?**\n\n\nThe entire workflow is orchestrated by `Manager Agent`, which is responsible for deciding the next agent for execution. For ablation A3, we remove our proposed `OHCache`, and employ broadcast-style communications, same as existing work, e.g., CAMEL [3], etc, where every message is disseminated to all agents and each agent processes the entire message history for decision-making.\n\n\n> **Q5: The authors did not specifically discuss limitations; please see the weakness for suggestions on improvements.**\n\nDue to the limited space in the main paper, we have detailed discussions for limitations and potential social impact in **Appendix G and H**, respectively. Thank you for your suggestions. We will add the above discussion in our next revision.\n\n\n[1] Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems. ICLR’25.\n\n[2] G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks. ICLR’25 Workshop.\n\n[3] CAMEL: Communicative Agents for \"Mind\" Exploration of Large Language Model Society. NeurIPS’23."}]

Author rebuttal:

Meta-review: **summary**
This paper introduces AutoData, a multi-agent system designed to automate web data collection using only a natural language instruction. The system's core innovation is the Oriented Hypergraph Cache (OHCache), a communication architecture that enables efficient, structured messaging and reduces the high token costs associated with LLM-based agents. The authors also contribute Instruct2DS, a new benchmark for evaluating live data collection tasks across multiple domains.

**strengths**
- The paper addresses the highly practical and relevant problem of automated web data collection, offering a novel system that significantly reduces manual effort.
- The proposed OHCache architecture is an effective design for optimizing communication in multi-agent systems, successfully reducing token usage and execution time compared to baselines.
- The introduction of the Instruct2DS benchmark is a valuable contribution, providing a much-needed resource for standardized and realistic evaluation of web scraping agents.

**weaknesses**
- The system's robustness on complex websites that rely heavily on JavaScript, require logins, or use CAPTCHAs is not fully demonstrated, limiting its real-world applicability.
- The current implementation is limited to text-centric data and does not support multimodal data collection (e.g., images, video), which is a significant part of web content.
- The system's best performance is dependent on expensive proprietary models like GPT-4o, and the path to achieving similar results with open-weight models remains future work.

**final descision**
Accept. The paper presents a well-designed system for an important problem, and its contributions, particularly the novel OHCache architecture and the Instruct2DS benchmark, are significant and well-supported by strong empirical results.
Decision: Accept (poster)