Title: AutoData: A Multi-Agent System for Open Web Data Collection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Agents

LLMs

Web Data Collection
Abstract: The exponential growth of data-driven systems and AI technologies has intensified the demand for high-quality web-sourced datasets. 
While existing datasets have proven valuable, conventional web data collection approaches face significant limitations in terms of human effort and scalability. 
Current data collecting solutions fall into two categories: wrapper-based methods that struggle with adaptability and reproducibility, and large language model (LLM)-based approaches that incur substantial computational and financial costs. 
To address these challenges, we propose AutoData, a novel multi-agent system for Automated web Data collection, that requires minimal human intervention, i.e., only necessitating a natural language instruction specifying the desired dataset. 
In addition, AutoData is designed for a robust multi-agent architecture, featuring a novel oriented message hypergraph coordinated by a central task manager, to efficiently organize agents across research and development squads. 
Besides, we introduce a novel hypergraph cache system to advance the multi-agent collaboration process that enables efficient automated data collection and mitigates the token cost issues prevalent in existing LLM-based systems. 
Moreover, we introduce Instruct2DS, a new benchmark dataset supporting live data collection from web sources across three domains: academic, finance, and sports. 
Comprehensive evaluations over Instruct2DS and three existing benchmark datasets demonstrate AutoData's superior performance compared to baseline methods.
Case studies on challenging tasks such as picture book collection and paper extraction from surveys further validate its applicability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a multi-agent system for collecting data from the web. The proposed MAS has three components, a Research Squad which handles the job for decomposing the instruction into subgoals and forming plans to extract information from the web; a Development Squad takes the plan and try to write code to execute the plan (including testing and validation); and a Manager agent which orchestrates the entire workflow. The MAS protocol is implemented as an Oriented Message Hypergraph which is claimed to be more efficient in terms of cost and empirically have better performance. Considering the difficulty of evaluating the open web data collection task, the paper also proposes a new benchmark, Instruct2DS. Empirical results on the Instruct2DS benchmark and also additional Information Extraction benchmark show that the proposed AutoData outperformed general Web or Coding agents such as Manus and Cursor.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The paper is well-written with comprehensive evaluations.\n2. The proposed benchmark is a good contribution: especially the setting where we consider collecting the latest data from the web is both realistic and useful. The impact can extend to never-ending learning of recent large foundation models.\n3. The empirical results against mainstream web/code agents is promising, especially considering a faster turn around time and lower expense\n\n# Weakness\n\n1. Some important comparison for showing the necessity and effectiveness of the proposed MAS frame seems unclear. Specifically, in Table 1, one important comparison should be comparing previous/baseline agentic frameworks equipped with the same backbone model (i.e., GPT-4o) as the AutoData. It is unclear what backbone models are used for most baselines in the table. \n  - Also a very related prior work is not compared with nor mentioned: “INFOGENT: An Agent-Based Framework for Web Information Aggregation” (https://arxiv.org/pdf/2410.19054)\n\n2. I feel there is potentially an overcomplication / a lack of novelty in the Hypergraph and Cache System. If I understand correctly, all nodes and edges are predefined. Thus, simply keeping a shared information pool + each agent takes only partial information from the pool should work? This is not a new idea which has been investigated in multiple previous work, such as “Meta-Prompting:Enhancing Language Models with Task-Agnostic Scaffolding” (https://arxiv.org/pdf/2401.12954) and “Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks” (https://arxiv.org/pdf/2501.11733)\n\n3. See question 1.\n\nQUESTIONS:\n1. In Ablation setting 3, when removing the graph structure, how does the agent know how to communicate and who to perform first? What’s exactly the implementation of this ablation here?\n\nLIMITATIONS:\nThe authors did not specifically discussed about limitations; please see weakness for suggestions on improvements", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research introduces AutoData, a novel multi-agent system designed for automated web data collection that minimizes human input, requiring only a natural language instruction. This system features a robust multi-agent architecture, coordinated by a central task manager, which efficiently organizes agents into research and development squads. A key innovation is the Oriented HyperGraph Cache System (OHCache), which enhances collaboration among agents and significantly reduces token costs commonly associated with existing Large Language Model (LLM)-based systems. To validate its effectiveness, the authors also present Instruct2DS, a new benchmark dataset supporting live data collection from various web sources, demonstrating AutoData's superior performance in terms of efficiency and accuracy compared to current methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths - \n1. AutoData features a robust multi-agent architecture with a novel oriented message hypergraph coordinated by a central task manager that efficiently organizes agents across research and development squads. The system is composed of eight specialized agents, strategically organized into two synergistic squads (research and development), coordinated by a central manager agent (MGR). \n2. AutoData addresses the significant token costs prevalent in existing Large Language Model (LLM)-based systems through its novel Oriented HyperGraph Cache System (OHCache). \n3. The paper introduces Instruct2DS, the first open web data collection benchmark, which addresses a significant gap in evaluating AI agent performance for such tasks. This dataset spans three domains (academic, finance, and sports). Instruct2DS is distinctive because it requires MAS agents to interact with live, dynamic web sources, supports diverse data acquisition modalities (including REST API calls), and enables symbolic-style information extraction. \n\nweaknesses - \n1. The current system primarily focuses on publicly accessible HTML pages or REST APIs. It struggles with websites that are entirely client-side rendered, protected by login walls, CAPTCHAs, or aggressive anti-bot policies. \n2. The proposed system is currently limited to text-centric data collection and does not support multimodal assets such as images, audio, or video. \n3. There is a risk of unintentionally violating copyright or contractual constraints as the system assumes requested data is legally retrievable. Users must be mindful of ethical concerns and responsible use, as uncontrolled harvesting could lead to legal liability. \n4. Running multi-agent LLM inference at a web-scale consumes non-negligible energy, raising environmental concerns.\n\nQUESTIONS:\n1. The paper indicates a dependency on proprietary LLMs like GPT-4 and GPT-4o, noting that the superior performance with open-weight models (such as DeepSeek-R1) is not guaranteed. Given that GPT-4o offers the best overall performance at a relatively high expense, are there specific plans to further optimize AutoData's architecture or conduct targeted fine-tuning on open-weight LLMs to reduce reliance on proprietary models while maintaining competitive performance and improving cost-efficiency for wider adoption?\n2.  The current system is limited to text-centric data collection and does not support multimodal assets like images, audio, or video. What is the roadmap for extending AutoData's capabilities to include diverse multimodal data collection, and would this require a significant overhaul of the existing agent architecture or the introduction of new specialized agents?\n3. The error analysis points to two key failure modes: mis-parsing attributes when information is implicitly encoded or conference names/tracks are unclear, and sometimes opting for less reliable HTML scraping even when cleaner REST API options are available. How do the authors plan to enhance AutoData's reasoning and decision policies to overcome these issues, particularly in distinguishing optimal data acquisition methods (e.g., API vs. scraping) and handling ambiguous or implicitly structured web content?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper introduces AutoData, a fully automated multi-agent system for web-scale data collection from open sources. AutoData organizes agents into two functional squads, research and development. They are coordinated by a manager agent and connected through a novel communication architecture called OHCache (Oriented Hypergraph Cache). OHCache formalizes agent messaging as a directed hypergraph, which enables structured, scoped, and efficient communication. The paper also contributes Instruct2DS, a new benchmark for open web data collection spanning academic, finance, and sports domains. AutoData outperforms both LLM-based and traditional scraper baselines across accuracy, cost, and time.

STRENGTHS AND WEAKNESSES:
Strengths:
- The proposed method encodes inter-agent messaging as oriented hyperedges, which allows targeted message delivery and avoid message broadcasting.
- It introduces token-aware artifact offloading via local cache + cache ID routing, which effectively avoids context bloating 
- The proposed method demonstrates reduced cost and execution time in addition to strong empirical performance.

Weaknesses:
- Agent-specific contexts are not uncommon in multi-agent system design, which limits the novelty.
- How the system handles underspecified or vague user instructions and out-of-domain webpages isn’t deeply explored, which raises doubts about its robustness.
- It’s unclear how the manager agent prioritizes conflicting or simultaneous message paths.

QUESTIONS:
- How are role-to-target mappings maintained or adapted? Are they hard-coded or learned over time?
- Could the OHCache framework generalize to other tasks beyond web data collection? Which part of the design would require major changes?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AutoData, a novel multi-agent system for automated open web data collection. It introduces a structured communication framework called OHCache to reduce message redundancy and improve collaboration efficiency, and presents Instruct2DS, a new benchmark spanning academic, finance, and sports domains. Experimental results demonstrate that AutoData achieves superior performance in accuracy, cost-efficiency, and execution time compared to strong baselines across multiple datasets and real-world tasks. The authors also explicitly acknowledge ethical and privacy concerns in web data collection and design their benchmark accordingly.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n1. Strong Practical Motivation: The paper targets a highly practical and relevant problem—web data collection—which remains a complicated and labor-intensive workload for developers.\n\n2. Effective System Design: The proposed system integrates multiple techniques to address key bottlenecks in multi-agent system (MAS) design, notably in improving both effectiveness and efficiency.\n\n3. New Benchmark (Instruct2DS): The introduction of a live, domain-diverse benchmark is a substantial contribution, helping to fill the gap in realistic evaluation datasets for open web scraping tasks.\n\n### Weaknesses\n\n1. Data Quality and Coverage: The practicality of AutoData may be limited by real-world data diversity and quality issues. This could stem from differences in regional data collection policies, as well as engineering challenges such as deduplication, detoxification, safety filtering, and value alignment.\n\n2. Limited Discussion of Failure Modes: While ethical and volatility-related concerns are briefly acknowledged in the appendix, the main paper lacks an in-depth discussion on how the system handles adversarial, highly dynamic, or JavaScript-heavy web environments.\n\nQUESTIONS:\n1. Could you provide more details on the “cost” metric used in the experiments? For example, how much would it approximately cost to collect 1,000 samples using GPT-4? Is the data collection cost sensitive to the nature or structure of different web sources?\n\n2. Could the proposed OHCache framework be generalized to other multi-agent domains beyond data collection, such as tool-using or decision-making systems?\n\n3. What is the failure rate or error distribution of AutoData in real-world web scraping scenarios not represented in the Instruct2DS benchmark?\n\nLIMITATIONS:\nWhile the authors briefly acknowledge certain limitations—such as web content volatility and ethical/privacy concerns—in the appendix, these issues are not sufficiently addressed in the main paper. Critical points that deserve more discussion include: \n**Societal Impact:** The potential for misuse (e.g., scraping sensitive or copyrighted content) and alignment with ethical scraping norms (e.g., robots.txt, rate limits) is not explored in depth.\n\nThe authors are encouraged to integrate a dedicated section in the main paper on limitations and societal impact. Concrete discussion on compliance with data usage policies, safety checks during scraping, and value alignment would improve the paper’s transparency and trustworthiness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}