AI-Researcher: Autonomous Scientific Innovation
Authors: Jiabin Tang
Lianghao Xia
Zhonghang Li
Chao Huang
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM agents
AI for Science
Primary area: applications
TL;DR: We introduce AI-Researcher, a fully autonomous research system that transforms how AI-driven scientific discovery is conducted and evaluated.

Abstract: The powerful reasoning capabilities of Large Language Models (LLMs) in mathematics and coding, combined with their ability to automate complex tasks through agentic frameworks, present unprecedented opportunities for accelerating scientific innovation. In this paper, we introduce AI-Researcher, a fully autonomous research system that transforms how AI-driven scientific discovery is conducted and evaluated. Our framework seamlessly orchestrates the complete research pipeline--from literature review and hypothesis generation to algorithm implementation and publication-ready manuscript preparation--with minimal human intervention. To rigorously assess autonomous research capabilities, we develop Scientist-Bench, a comprehensive benchmark comprising state-of-the-art papers across diverse AI research domains, featuring both guided innovation and open-ended exploration tasks. Through extensive experiments, we demonstrate that AI-Researcher achieves remarkable implementation success rates and produces research papers that approach human-level quality. This work establishes new foundations for autonomous scientific innovation that can complement human researchers by systematically exploring solution spaces beyond cognitive limitations.

AI-Researcher: Autonomous Scientific Innovation
Jiabin Tang"
Lianghao Xia'
Zhonghang Li Chao Huang
The University Hong Kong
(jiabintang7,, bjdwh. zzh, Mhachuang755$@gall com; aka_ xia@f xia@foxmail. com
Abstract
The powerful reasoning capabilities of Large Language Models (LLMs) in mathe- matics and coding, combined with their ability to automate complex tasks through agentic frameworks, present unprecedented opportunities for accelerating scientific innovation. In this paper, we introduce Al-Researcher, fully autonomous re- search system that transforms how AI-driven scientific discovery is conducted and evaluated. Our framework seamlessly orchestrates the complete research pipeline- from literature review and hypothesis generation to algorithm implementation and publication-ready manuscript preparation- with minimal human intervention. To rigorously assess autonomous research capabilities, we develop Srientist-Benchh a comprehensive benchmark comprising state-of-the-art papers across diverse AI research domains, featuring both guided innovation and open-ended explo- ration tasks. Through extensive experiments, we demonstrate that Al-Rescareher achieves remarkable implementation success rates and produces research papers that approach human-level quality. This work establishes new foundations for autonomous scientific innovation that can complement human researchers by sys- tematically exploring solution spaces beyond cognitive limitations. Code link: sttps://githnn wwt.cooo/c/c//// Ht////KUUSSAAAA AReee/rareR.....r..
1 Introduction
Scientific discovery has historically been constrained by human cognitive limitations and the immense scale of potential solution spaces Wang et al. [2023]. Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in mathematical reasoning, coding, and problem- solving tasks that were previously thought to require human expertise Didolkar et al. [2024], Guo al. [2024]. However, transitioning from isolated capabilities fully autonomous scientific research systems capable of original innovation remains an unsolved challenge that could fundamentally transform how scientific progress occurs.
Despite recent advances in agentic frameworks powered by LLMs, scientific innovation represents an intellectual frontier orders of magnitude more challenging than the task automation currently
mastered by existing AI agents Manus Technologies [2025], OpenManus Contributors [2025], Li et al. [2023], Tang al. [2025]. While today's agents schedule meetings retrieve structured information, genuine scientific discovery demands an unprecedented level of intelligence requuring sophisticated conceptual reasoning across abstract theoretical domains, transformative hypothesis generation that bridges disparate knowledge fields, and methodological innovation that extends far beyond pattern recognition. The research process necessitates maintaining coherent understanding across thousands of papers while simultaneously generating insights that fundamentally advance knowledge coundaries-intellcttuu capabilities that existing architectures cannot approach.
Most critically, scientific exploration involves navigating vast, unbounded solution spaces with deeply uncertain rewards, requiring meta-cognitive abilities recognize promising directions and abandon
"quual contribution.
1 Chao Huang is the Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

unproductive paths, Researchers must continuously evaluate experimental results against theoretical frameworks, adapt hypotheses based on unexpected findings, and communicate complex ideas with precision and clarity-all while maintaining the creative spark that drives breakthrough discoveries. These profound limitations have prevented Al systems from autonomously conducting meaningful scientific work, perpetuating a paradigm where Al remains relegated to narrow assistance roles rather than functioning as independent scientific contributors capable of accelerating human knowledge advancement through systematic exploration of solution spaces beyond human cognitive limitations.
Full Autonomy: Complete end-to-end research automation Seamless Orchestration: From concept to publication
Research Acceleration: Streomlined scientific innovation
Results Log
Project
Paper
Visuiitzotion
generation
End-to-End Automation
Comprehensive Outcomes
Figure 1: Architectural overview of Al-Researcher, illustrating the end-to-end autonomous scientific innovation pipeline encompassing literature exploration, idea generation, algorithm implementation, experimental validation, and comprehensive scholarly publication with rigorous evaluation metries.
While specialized systems exist for literature analysis or experiment design Schmidgall and Moor [2025], Gottweis et al. [2025], they fail to orchestrate the complete research workflow from hypothesis generation through publication-ue qualliy reporting. Furthermore, no standardized benchmarks exist to evaluate autonomous research across diverse scientific domains, making progress in this frontier difficult to measure systematically.
We introduce Al-Researcher a novel framework that addresses these limitations by seamlessly orches- trating the complete scientific discovery lifecycle from literature analysis through implementation to scholarly documentation. Unlike systems focusing on isolated capabilities, our framework employs comprehensive multi-agent architecture where specialized components collaborate through structured knowledge exchange to maintain coherent reasoning throughout the research process. This recursive refinement mechanism enables continuous bidirectional feedback between theoretical concepts and their peteenetttttoosspreereeee rigorous scientific contributions with minimal human intervention.
intellectual consistency while transforming research ideas into
AI-R Al-Researcher introduces three key innovations that fundamentally advance autonomous scientific discovery. First, Resource Analyst agents decompose complex research concepts into atomic compo- nents with explicit bidirectional mappings between mathematical formulations and code implementa- tions, dramatically reducing hallucination risks. Second, our Implementation Framework employs a human-inspired iterative refinement paradigm where specialized agents collaborate through structured feedback cycles, mirroring the proven mentor-student relationship in academic research. Third, our Documentation Agent overcomes LLM coherence limitations through hierarchical synthesis approach that transforms rescarch artifacts into publication-qualit manuscripts while maintaining cross-document consistency and factual integrity throughout extensive scholarly documentation.
To rigorously evaluate autonomous scientific systems, we develop Scientist-Bend -te first com- prehensive benchmark enabling standardized assessment across both guided innovation scenarios and open-ended exploration tasks spanning diverse AI domains. Through extensive experiments on 22 benchmark papers using multiple LLM evaluators, we demonstrate that Al-Researcher achieves remarkable implementation success rates while producing research contributions that frequently approach human-level quality. Surprisingly, our findings reveal Al-Researcher performs better in open-ended exploration than in guided implementation tasks sugggssnng autonomous research sys- tems excel when leveraging internal knowledge synthesis rather than following prescriptive directives.

---

Knowlodae
Browing
X
Cudling
Abarut
Acquistion
Collectin
Code
Ixrr
have some reGrenoes
Coding Luop
Same Field
Examples
prblen
Task Lake howe arme stose
Uae mothod to address
Taking
Resource
Analysi
Realinz
Papers
Keline
Loop
Project
Metied
Genention
Stnoture
nc propaee lected
Eegiidd in (nalvcie))
Tomplace
hard informezss
Dotaset Candidates, For the bes, you can thoz danach
Task 1. Make Insostioes
Idea Cade
Strea
Geverating Rasking
Idea Generssor
Comments
f
Advince
Agrmt
Abinct
Into
& Revisoo
Checklia
Detaile
Wiking
Ges. Coke
Nares f
New Alguritum Design.
Paper Conpprenn
Composer Impul
Innovation Benchmark
Literature Reviow Idea Generation
Implementation Validatton
Automated Sclentific Decumentattion
Figure 2: Architectural framework of Al A.seeccarr
These results establish new foundations for autonomous scientific agents that complement human researchers by systematically exploring solution spaces beyond human cognitive limitations.
2 Scientist-Benche Benchmarking AI Agents for Scientific Discovery
Scientific discovery requires deep expertise and methodical reasoning. Developing benchmarks for novel scientific discovery and establishing evaluation metries remain challenges in the field Reddy and Shojaee [2025]. We introduce Scientist-Bench generated research with human expert work. Unlike existing benchmarks Wang et al. [2024], Scientist- Bench provides a comprehensive framework including curated research instructions, references, and datasets derived from reer-reviewed papers. This enables direct comparison between AI Al-generated and human scientific contributions through multidimensional evaluation standards. We define the scientific discovery task as follows (full benchmark details are in Appendix A.7):
comprehensive benchmark comparing LLM Agent-
Task Formulation:
The benchmark evaluates agent systems for scientific research capabilities. The input = (R, D) consists of reference papers (15-20 relevant references selected via LLMs), defines two
research instruction (containing core research ideas), and datasets 'D. Scientist- challenge levels: Level-1 provides explicit research instructions extracted from target paper y, testing execution ability; Level-2 omits these instructions, challenging agents to formulate novel directions using only references and datasets. The output ) {C, p} comprises implementation code C and a technical report describing research background, methodology, experiments, and results. Both components undergo evaluation measure quality and innovation compared to human- uuman-generated research, providing holistic assessment across theoretical and practical dimensions.
3 The Al-Researcher Framework
3.1 Multi-Agent System Overview Al-Researcher
Recent work has shown AI systems' potential for autonomous scientific discovery Lu et al. [2024], Yamada et al. [2025]. Ae-eecrrrr builds on this by introducing a systematic framework with three stages: i) Literature Review and Idea Generation: ii) New Algorithm Design, Implementation and Validation; and iii) Automated Scientific Documentation. As shown in Figure 2, this pipeline transforms scientific concepts into complete academic contributions with minimal human oversight.
3.1.1 Literature Review
Knowledge Acquisitionn Agent. The autonomous research process begins with literature exploration by the Knowledge Acquisition Agent, which systematically gathers Relevant Papers and Code Repositories from scientific databases. A key advantage is its minimal input requirement-uuse need only provide 10-15 reference papers. The system then processes this input to identify valuable information, performing two critical functions below. These filtering criteria ensure only relevant, maintained, and impactful resources form the foundation for subsequent AI research. Detailed prompts and tools for the Knowledge Acquisition Agent are provided in Appendix A.7.
1) Code Repository Selection: Using reference papers as guidance, the agent identifies at least 5 high-quality GitHub repositories through filtering that evaluates: Code Recency, GitHub Popularity, Documentation Quality, Domain Relevance, Citation Impact.
2) Supplementary Literature Gathering: For each filtered high-quality repository, the agent automatically retrieves corresponding papers from arXiv, including their complete LaTeX source files, further enriching the knowledge base with contextually relevant technical materials.
Resource Analyst Agent. This agent systematically deconstructs research concepts into manage- able atomic components, meticulously extracting their mathematical formulations and corresponding code implementations through its Paper Analyst and Code Analyst sub-agents, ensuring precise alignment between thcoretical expressions and practical implementation.

---

Secure Research Environment. To protect host systems during automated operations, all pro- cesses run Docker container. providing: (1) robust security boundaries preventing unauthorized modifications; (2) consistent environments with pre-configured MI. frameworks; and (3) dynamic package management for autonomous dependency installation. This creates a controlled yet flexible workspace supporting the entire research pipeline.
Integrated Research Analysis. Resource Analyst forms a critical bridge between abstract con- cepts and their concrete implementations. significantly reducing potential hallucinations in subsequent development stages. This agent operates through a carefully structured analytical process:
1) Concept Decomposition: Using the initial research idea, the agent decomposes complex objectives into atomic academic concepts concepts-fundameee fundamental,, indivisible research elements requiring investigation.
2) Mathematical Formalization: The Paper Analyst examines LaTeX files through RAG-based paradigm to extract mathematical formulations of each atomic concept.
3) Implementation Analysis: The Code Analyst analyzes code repositories to locate implementa- tions of these mathematical expressions, identifying critical files and dependencies.
4) Knowledge Integration: Results from paper and implementation analyses are synthesized into concept profiles, establishing connections between math formulations and code implementations.
This rigorous cycle continues until all concepts are thoroughly investigated, culminating in a detailed research report that serves as the foundation for development planning. The Plan Agent transforms these findings into a comprehensive implementation roadmap addressing training procedures, testing methodologies, and dataset aeapirrmmenttccratiinn complete, executable research strategy.
3.1.2 Idea Generation
Recent LLMs have advanced research idcation, with Chain-of-Ideas Li et al. [2024] organizing literature into progressive chains and ResearchAgent Back et al. [2025] using collaborative LLM reviewers refine proposals. While these systems primarily recombine known knowledge, our Idea Generator designed to venture beyond established paradigms into new scientific frontiers.
Operating after comprehensive analysis, the Idea Generator employs knowledge synthesis to identify unexplored research territories. The agent systematically seeks conceptual gaps, contradic- tory findings, and emerging patterns across literature and implementations- areas where scientific discoveries often emerge. Each generated proposal pushes beyond established paradigms through:
Challenges that pinpoint fundamental limitations in current scientific understanding; Existing Methods revealing conceptual blind spots ripe for innovation; Motivation establishing scien- tific necessity for trradigm-shifting approaches; + Proposed Method introducing novel theoretical frameworks or algorithmic innovations; Technical Details translating abstract breakthroughs into implementable science; and Expected Outcomes projecting potential scientific and practical impact.
Divergent-Convergent Discovery. Inspired by Si al. [2024], our process first generates five distinct research directions in a divergent phase, exploring orthogonal perspectives. These undergo convergent evaluation against Scientific Novelty, Technical Soundness, and Transformative Potential. The top concept is then developed into a comprehensive proposal with clear implementation pathways.
3.2 New Algorithm Design, Implementation and Validation
Translating novel research concepts into functioning implementations represents one of the most challenging aspects of computational science. Unlike traditional code agents that attempt one-shot implementations-oofeen causing errors or research misalignment-we introduce a framework that mirrors the proven human research paradigm of iterative refinement and collaborative feedback.
Multi-Stage Refinement Architecture. Our approach implements a cyclical development process with explicit feedback mechanisms, enabling progressive improvements. Similar to advisor-student collaborations, our framework conducts itcrative refinement with structured guidance. This approach increases implementation success rates with test-time scaling capabilities.
Code Implementation Framework. The Code Agent transforms research analysis and develop- ment plans into executable implementations. Operating within controlled workspace, this agent creates structured implementations with comprehensive file system and execution capabilities. enforces strict code independence principles while ensuring faithful translation of academic concepts into working code. Throughout development, the agent maintains continuous verification against the implementation plan with thorough documentation of all modifications.

---

Expert Validation Framework. Our Advisor Agent provides expert feedback that bridges the gap between theoretical concepts and practical implementation. validates implementation fidelity by systematically comparing code against atomic research ideas extracted during analysis. 'The agent examines results through specialized navigation tools and visualizations while referencing workspace materials. Based on comprehensive evaluation, it generates detailed assessment reports with specific, actionable modification recommendations to guide refinement iterations.
Progressive Experimental Cycles. Our experimental process implements a rigorous scientific approach to code validation. The Code Agent begins by developing prototype implementations that undergo initial testing on minimal data (typically 1-2 epochs or small dataset subsets) to es- tablish baseline feasibility. Following this preliminary validation, successful implementations that incorporate review feedback advance to full-scale experiments, while persistently unsuccessful im- plementations receive "unfeasible" classification after multiple refinement attempts. Throughout this eyclical process, the Advisor Agent provides analytical support by evaluating results and recommending supplementary investigations. These recommendations span implementation refine- ments, validation studies, visualizations, and comparative analyses aligned with established research standards. Through these structured refinement cycles, implementations systematically evolve toward optimal performance, ensuring scientific rigor and reproducibility in our findings.
3.3 Automated Scientific Documentation
The culmination of scientific re- search requires transforming raw experimental results into structured academic knowledge contributions. Following substantial implemen- tation and experimentation cy- cles, Documentation Agent initiates a sophisticated process that converts technical artifacts into publication-ready manuscripts while maintaining seientific in- tegrity and narrative coherence.
To adfress [prohlen]] we
<>
Reference Papers
prrpuer [mahhed] that
lasengos (key lachnigae).
which could
Plan
Tmplementatin
Method
Exp
Coder
[pupesetnntratin) [so ree] [runter] (ev bchhoologos (chorly rechnologios.
Advisor
1 Feedback
Cale Function
def cate loss: preds [abels)
nuttiple poants]...
Template
Forward Diffusion
tn Code
Concent
forvard diff()
Research Trajectory
Writing
GCN Tresfermer
Template
Revision
epochs
Full
Progressive
Methall Rajnte
Inttas Caaghh
Data subsets
Experiments
Esp
Abstrac
Multi-Stage Refinement Architecture
Scientific Documentation
Figure 3: Illustration of (1) multi-stage implementation refine- ment, and (2) automated sceintfic documentation.
Research Trajectory Synthesis. The Automated Documentation Agent systematically inte- grates diverse research elements--inlludng agent reasoning processes, execution logs, implemented code, and experimental outcomes intt cohesive scientific narratives. This holistic approach pre- serves the complete intellectual context behind discoveries while structuring findings according to established academic conventions. Unlike simple documentation tools, our system captures both the final results and the critical decision pathways that led to scientific advances.
Overcoming Document-Scale Coherence Challenge. To write coherent academic manuscripts-a challenge for L.MSs that struggle with consistency over extended outputs-we developed a multi-stage generation framework inspired by how researchers draft papers and by Shao et al. [2024]. This methodology overcomes LLM limitations by decomposing writing into manageable components while preserving logical connections and factual integrity throughout.
Three-Phase Hierarchical Documentation. Our writing approach employs a systematic three-stage process: (1) Synthesizing Research Artifacts: structural outlining based on -pproprrpe templates that establish section hierarchies and logical flow; (2) Template-Guided Guided Structure: methodical content elaboration that develops explanations maintaining cross-document consistency; and (3) Hierarchical Documentation Process: systematic verification using specialized academic checklists that identify and remediate inaccuracies or omissions. This "one more step" review process enhances factual integrity and completeness, ensuring manuscripts meet publication standards without the hallucinations and inconsistencies that typically plague LLM-gennrated long-form content.
Experiments
Experimental Settings. We evaluate Al-Researcher using the Scientist-Bende benchmark. Details about the experimental datasets, tested tasks, and evaluation protocols are presented in Appendix A.8.
4.1 Dual-Metric Evaluation Framework: Quantifying Implementation Quality (RQ1) To evaluate the stability and quality of Al-Researcher system's 's code implementation based on requirements, we propose Completeness and Correctness metrics for measurement.
5

---

Specifically, we evaluate implementation qual- ity across two critical dimensions: Complete- ness measures whether the agent produces ex- ecutable code within the allocated inference budget. We implement an unambiguous ter- mination protocol where agents signal success via @ase_resolvee ved or acknowledge failure through case_ case__no__. not_resolver mated assessment of task completion rates. Correctness addresses a nuanced challenge- even when code executes, it may contain sub- tle implementation flaws, conceptual misalign- ments, missing components requiring deeper analysis. To evaluate implementation fidelity, we employ multi-agent framework where an Advisor Agent generates detailed analysis reports identifying potential issues, followed by Judge Agent that assigns quality scores on a 5-point scale. The final correctness metric represents the mean score across multiple independent judgments, providing robust measure of implementation quality throughout the development lifecycle. We conduct extensive evaluations across both Level I and Level 2 tasks our benchmark, systematically analyzing how implementation performance varies with different backbone LLMs. Our analysis reveals several key findings:
Completeness
Expertments
100,0%:
000
100.%%
93.8%
Complemennse
2.43
122
2.33
2222
Overal Completeness 53 8l
Completeness
enabling auto-
Overst Coreetness
Cormectness
Domain
Figure 4: Quantifying Implementation Quality in terms of Completeness and Correctness.
Performance Analysis. We conducted comprchensive experiments using Claude-series models across our entire benchmark dataset, evaluating both completeness and correctness metrics as shown in Figure 4. The results reveal remarkable stability-ou Al-Researcher system achieves outstanding 93.8% completeness rate with Claude-series models, failing only in cases involving complex technical challenges such tensor dimension conflicts and datatype mismatches that persisted despite multiple debugging iterations. This exceptional completeness rate underscores the robustness of our system's 's implementation and debugging capabilities across diverse computational and algorithmic domains.
For correctness, our system achieves an average score of 2.65 (on a 1-5 scale), exceeding the median threshold and indicating successful implementation of the majority of specified requirements. Notably, performance varies across domains- Vision and Question Answering (VQ) tasks reached the highest correctness of 3.22, while Recommendation (Rec) tasks averaged 2.20. This variation likely reflects the inherent complexity differences between domains, with recommendation systems typically requiring more intricate algorithmic implementations and data handling procedures.
Completeness
Completeness
Expertments (Subset)
100 0% 14
100
212
Comp Come-ggeaaa Signaa (50
Corre conpictencc (h)
2:75
Conectress
Claudo Gerios Ovenii care
Glude serkes rolieeess
sertes Dened Doer 2%
series Oempiiienn penoo
1.00
Levelt Oueral Cover no
Lewelt cow ploteses Level2 Cer Skrrkieedd
Denel 1.00
Levet2 Deni:
Level2 Dnaell 222
Levetz Coratinay
Domath
Domain
Figure 5: Performance Comparison Across Model Families and Task Complexity. Left: Claude-scries versus 4o-series models on implementation completeness and correctness metrics (benchmark subset). Right: Claude-series performance across Level (adaptation) and Level (innovation) tasks.
Performance Comparison between LLMs in Scientific Implementation. To rigorously compare the capabilities of different large language models in automated scientific research, we conducted controlled evaluation using a balanced subset of our benchmark dataset spanning multiple technical domains. As illustrated in Figure 5 (left), our assessment reveals substantial performance differences between model families. Claude-series models achieved an impressive 87.5% completeness rate on the evaluation subset, significantly outperforming the 4o-series models which reached only 50% completeness. This performance gap stems primarily from differences in debugging proficiency- 4o-series models frequently generated code with persistent tensor dimension mismatches and training instabilities (NaN losses) that remained unresolved despite multiple debugging attempts. In contrast,

---

Table Comparing Al-Researcher's Al-generated research and groundtruth human research.
Field
Metric
GPT-4o
ol-mini
03-mini
Claude-3.5
Claude-3.7
Average Rating
00.48=0.87
1,36
-1.27
-1.83
1.,49
Diffusion
Comparable($)
75.00%
25.00%
50.00%
0.00%
25.00%
Average Rating Comparable($))
-0.55+11.00
-0.95++1.56
-1.4990.66
-1.68+1.37
-2.11
VQ
83.33%
50.00%
16.67%
16.67%
0.00%
Average Rating
0.70=
-1.52:1
-.6810.62
1.866...6
1.8331144
GNN
Comparable($$5 %))
71.43%
42.86%
0.00%
0.00%
14.29%
Average Rating Comparable($)
-0.33110.11
-0.42.10.86
-1.50.0.94
0.88
-0.81111.76
Rec
100.00%
100.00%
0.00%
40.00%
60.00%
Average Rating Comparable($o) 81.82%
00.33+1.00
-1.09++1.60
-1.51
11.58
11.70
Overall
54,55%
13.64%
13.64%
22.73%
Claude-series models demonstrated superior problem-solving capabilities, successfully identifying and resolving complex implementation issues through systematic debugging approaches.
The quality disparity extends beyond mere code completion implementation correctness, where Claude-series models scored substantially higher (2.75 points average) compared to 4o-series models (1.0 point average). The 40-series implementations exhibited a consistent pattem of oversimplification and conceptual omissions in complex tasks. A particularly illustrative example occurred in the diffusion model integration task, where the 4o-series model claimed successful implementation of Diffusion Transformer architecture while detailed inspection revealed merely a standard Vision Transformer (ViT) implementation with complete absence of the critical diffusion components. This systematic evaluation highlights the importance of both implementation completeness and conceptual correctness when assessing LLM capabilities for advanced scientific research tasks.
Implementation Success with Increasing Task Complexity (Level-2). To systematically evaluate our framework's 's performance across difficulty levels, we conducted comparative analysis using a balanced subset of benchmark tasks from each research domain. Figure 5 (right) presents the completeness and correctness metries for Level 1 tasks (adapting established methodologies) versus Level 2 tasks (generating and implementing novel research idcas) using Claude-series models.
Remarkably, Al-Researcher maintains perfect implementation completeness (100%) even for the more challenging Level 2 innovation tasks. This consistency demonstrates the robustness of our system's s self-debugging mechanisms and execution pipeline when handling both established and novel methodological approaches. However, we observe a modest decrease in correctness scores from Level (2.5) Level 2 (2.25) tasks. This slight performance gap reveals an important challenge: while Al-Researcher can reliably execute self-generated research ideas to completion, the implementation quality of novel concepts occasionally falls short of adaptation tasks.
The correctness differential stems primarily from two factors. First, the complexity of ggent-generated research ideas varies considerably, with some innovations proving technically challenging imple- ment correctly. Second, while our idea generation and ranking system generally produces feasible concepts, the framework occasionally struggles to perfectly realize ambitious or complex innovations. These findings suggest promising avenues for future enhancement, particularly in developing sophis- ticated idea feasibility assessment mechanisms and implementing adaptive modification capabilities that allow real-time refinement of research approaches during implementation when obstacles arise.
4.2 Evaluating Scientific Quality Through Pairwise Comparison ((RQ2)
To assess the scientific merit of Rescaaccher-gnnratte research, we implemented a pairwise evalua- tion protocol comparing Al- Al-generated papers with numan-authored publications in matching domains. Specialized review agents perform comparative analyses following ICLR guidelines-evee evaluating research motivation, methodology, innovation, and experimental validation across both works.
Overall Performance. Comparative evaluation reveals that while papers generated by AI- Researcher receive moderately lower average ratings than human-authored works (ranging from -0.53 to -1.70 across evaluators), a substantial proportion of Al-generated papers (13.64% to 81.82% demonstrate quality comparable to human research. This finding is particularly significant considering our benchmark comprises exclusively top-tier human-authored publications carefully selected from leading venues in each domain. The results demonstrate AI- Al-Rescarcher's remarkable capacity to
7

---

Table 2: Results of Al-Researcher's open-ended research exploration.
Field
Metric
GPT-4o
ol-mini
03-mini
Claude-3.5
Claude-3.7
Average Rating
0.56==0.79
-1.7550..83
-1.00=0.550
-2.00=0..00
-0.56+
Diffusion
Comparable($)
100.00%
0.00%
100.00%
0.00%
100.00%
Average Rating Comparable(%)
-0.25=0.97
-0.62++0.99
-0.88
-1.00++-.50
11.31+1.10
VQ
100.00%
100.00%
100.00%
100.00%
0.00%
Average Rating
0.12:10.78
-0.501 1.00
1.19
-1.44:0.550
-0.9411144
GNN
Comparable($% (%)
100.00%
100.00%
0.00%
0.00%
100.00%
Average Rating Comparable($)
0.06
-0.77:11.52
-.00811000
0.19-11.78
-0.96:11.770
Rec
100.00%
66.67%
66.67%
100.00%
33.33%
Average Rating
0.23==0.99
-0.85+1.32
11.222+1.07
10.65=1.66
10.95+
Overall
Comparable($))
100.00%
66.67%
66.67%
66.67%
50.00%
execute the complete scientific research pipeline- from developing methodologically sound technical innovations to conducting rigorous experimental validations and synthesizing findings into coherent, well-structured academic manuscripts that approach quality standards of expert human researchers.
LLM Evaluator Divergence. GPT-4o provides the highest ratings for Al-generated papers (81.82% comparable with average rating -0.53), while Claude-3.7 gives the lowest ratings on average (22. 73% comparable with average rating -1.70). Moreover, for different research fields, L.LM evaluators show varying preferences. For example, GPT-4o and ol ol-mini consider all generated recommendation papers comparable to groundtruth human papers, while o3-mini rates them as inferior. This demonstrates the potential bias of using only one LLM evaluator to assess the generated research works, In summary, different LLM evaluators yield varying comparable percentages, ranging from 13.64% to 81 1.82%, demonstrating that the Al-conducted research approaches the quality of top-tier human research.
Domain-Specific Analysis. Performance varies across research fields but shows no consistent patterns. Papers on diffusion models gain higher comparable rate compared to GNN papers when evaluated with GPT--4 and Claude-3.7. However, this situation is reversed when using mm-mi as the evaluator. Recommendation papers achieves high comparable rate across all evaluators except o3-mini, while 03-mini thinks none of the generated recommendation papers are comparable to human papers. For the vector quantization domain, three evaluators (GPT-4oo, ol ol-mini, Claude-3.5) think the generated papers are better than diffusion papers, while 03-mini and Claude-3.7 consider them worse but diffusion papers better. These variations appear to be more influenced by evaluator preferences than by domains, suggesting that Al-Researcher maintains consistent performance across different research domains without catastrophic degradation in any particular field.
4.3 Open-Ended Autonomous Scientific Innovation Capabilities (RQ3)
To assess Al-Reesarrssers capacity for innovation, we evaluated its performance on open-ended tasks where it receives only reference literature without explicit directives. This requires Al-Researcher to independently identify directions, formulate hypotheses, and execute the complete research workflow. Table 2 presents the comparative evaluation results across different domains.
For this evaluation, we carefully selected 5 representative papers spanning distinct research areas to ensure methodological diversity while accounting for the natural citation overlap within specialized research communities. Our analysis of these autonomous scientific explorations reveals several key insights into the system's 's creative research capabilities:
Performance Analysis. A striking pattern emerges when comparing AI- Al-Researchhr's performance across task structures: the system demonstrates markedly superior outcomes in open-ended level-2 scenarios versus instruction-guide across evaluation metrics, with average ratings improving substantially from -0.53 -1.70 to -0.23 -1.22, and comparable rates rising dramatically from 13.64% 8..22% to 50.00% 100.00%.
level-1 tasks. This quality differential manifests consistently
These findings challenge conventional assumptions about AI research capabilities, suggesting that AI- Al-Researcher excels when leveraging its internal knowledge synthesis and ideation processes rather than following explicit research directives. The notable performance enhancement indicates that prescriptive research instructions may inadvertently constrain the system's 's creative exploration capacity, while autonomous research formulation allows Al-Researcher to identify and pursue more scientifically promising directions that better align with its implementation capabilities.
8

---

Table Impact of LLM backbones for the research agent of Al- Al-Researcher.
Research
Evaluation Metric
LLM used in Reviewing Agent
Agent LLM
GPTT40
ol-mini
03-mini
Claude-3.5
Claude-3.7
Average Rating
0.69:1100 71.43%
1.445=1.40 42.86%
-1.62::0.55 0.00%
-2.05+0..23
-2.12
GPT-4o
Comparable($5)
0.00%
14.29%
Average Rating
0.59:11.01
1.442:11.43 28.57%
-1.44:1.722 14.29%
1.00+1.03
11.98=1:.45
Claude-3.5
Comparable($5)
85.71%
0.00%
28.57%
Nomain-Specific Resource Constraints Influence Innovation Quality. Our cross-domain analysis reveals systematic relationship between computational requirements and autonomous research performance. Research areas with lighter computational demands, particularly recommender systems. demonstrate remarkable quality improvements in open-ended explorations, achieving impressive comparable rates of 66.67%-115 100% across most evaluator benchmarks.
Conversely, computationally intensive domains such as diffusion models exhibit more modest gains in evaluation metries despite similar conceptual innovation. This consistent pattern suggests that Al-Researcher's fundamental research capabilities extend beyond what our implementation currently demonstrates, with performance disparities reflecting practical resource limitations rather than conceptual understanding deficiencies. The finding highlights the importance of computational capacity as a determining factor Al research quality, indicating substantial potential for enhanced performance should greater computational resources become available.
4.4 Impact of L.L Backbones (RQ4)
To isolate foundation model influence on research capabilities, we conducted ablation studies across different LLM backbones while maintaining identical system architecture and protocols. Using 7 representative research problems, we assessed model-specific performance variations. Table presents the comparative analysis, revealing significant performance differentials between models.
The empirical evidence demonstrates Claude-3.5's substantial advantage as the research agent backbone, with this configuration consistently achieving higher mean quality ratings across all evaluator benchmarks compared to GPT-4o implementations. This performance differential extends beyond simple metrics comparable rates, where Claude-3.5 outperforms in most evaluation contexts, with the exception of -l-mi assessments. The quality gap becomes particularly pronounced under the most stringent evaluation criteria (o3-mini), where Claude-3.5-based systems produce research comparable to human standards while HPT--o-based configurations fail to generate any research meeting minimum comparability thresholds. These findings highlight the critical importance of foundation model selection in determining the upper bounds of automated scientific research quality.
4.5 Paper Review Agent Validation Against Human Expert Judgments (RQ5)
To validate our automated review system's alignment with expert scientific assessment, we conducted systematic evaluation using gold-standard human judgment data from the ICLR conference. The spe- cific experimental designs and evaluation results are presented in Appendix A.9. The results validates the alignment of our review agent with human expert decisions, demonstrating its effectiveness.
4.6 Case Studies of AI-Generated Scientific Contributions (RQ6)
To complement our quantitative evaluations with deeper qualitative insights, we conducted compre- hensive case studies examining both the implementation quality and scholarly presentation of research generated by Al-eeeear, Detailed experimental results are presented in Appendix Section A.
5 Conclusion
Al-Researcher represents significant advancement in autonomous scientific discovery, demonstrat- ing capabilities across the research workflow. Through multi-agent architecture, Al-Researcher overcomes limitations of existing systems, enabling genuine scientific innovation rather than mere task execution. The system's is ability to independently identify promising research directions, im- plement complex methodologies, and validate results through rigorous experimentation marks substantial step toward autonomous AI scientists. Experiments across 22 benchmark papers show Al- Al-generated research approaching human-level quality.
A Acknowledgments and Disclosure of Funding
The authors received no third-party funding or in kind support for this work in the past 36 months.

---

References
Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60) 2023.
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Kc, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabilities of Ilms: An exploration in mathematical problem solving. NeurIPS, 37:19783-19812, 2024.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, al. Deepseek-codee coderr: When the large language model meets programming- the rise of code intelligence. arXiv preprint 2rXi:22401.1119, 2024.
Manus Technologies. Manus: Structured and ai-assisted academic writing tool. attps://mann im/, 2025.
OpenManus Contributors. Openmanus: Open-source framework for building general ai agents. https://ool /ttpr/////ppmmmmeeeemmmmm github. sithubbio/,202.. 2025.
Guohao Li, Hasan Abed AI Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society, 2023. URL abs/ /2.../23..111 17760.
https: //ap:///aaiivee/ee
Jiabin Tang, Tianyu Fan, and Chao Huang. Autoagent: fully-automated and zero-code framework for Ilm agents, 2025. URL https: /ttpxaaxivv////////v///////aaav/ org/ abs/ /202. 05957.
Samuel Schmidgall and Michael Moor. Agentrxiv: Towards collaborative autonomous research, 2025. URL https: // arxiv org/ abs///3///22.// 18102.
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, al. Towards an co-scientist. arXiv preprint arXiv:2502. /8864, 2025.
Chandan K Reddy and Parshin Shojaee. Towards scientific discovery with generative ai: Progress, opportunities, and challenges. Proceedings the AAAI Conference on Artificial Intelligence, pages 28601-28609, 2025.
Xiaoxuan Wang, Zimiu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam. Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models, 2024. URL https:///// wtttttsss///sttttt org/ abs/2307 10635.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The scientist: Towards fully automated open-ended scientific discovery, 2024. URL btttps://arxivi org/ abs/ 2408. 2408.06292.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search, 2025. URL https: //arxiva..vv.////a////a absg/abs//ie /2504. 08066.
Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang. Deli Zhao. Yu Rong, Tian Feng, and Lidong Bing. Chain of ideas: Revolutionizing research via novel idea development with Ilm agents, 2024. URL www.:iii.orrreeoooo
org/ abs///bs//210 13185.
Jinheon Back, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models, 2025. URL https:
//arxiv. org/ abaaab///aaa 2b04/244.... 2404.07738.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? large-scale human study with 100+ NLP researchers. CoRR, (i./2409.041099 2024.
10

---

Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu. Omar Khattab, and Monica S. Lam. Assisting writing wikipedia-like like articles from scratch with large language models. In NAACL- HIT,, pages 2252-6278. Association for Computational Linguisties, 2024.
Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. Agentreview: Exploring peer review dynamics with Ilm agents. EMNLP, 2024.
LangChain Contributors. Langchain: Build context-aware reasoning applications. https://github. conl//angchaai-aaa/laggaaaaii,, 2023.
Yongliang Shen, Kaitao Song. Xu Tan, Dongsheng Li, Weiming Lu, and Yucting Zhuang, Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URL attpss///arxt org/ abs/2303. 17580.
Tianhao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. Openagents: open platform for language agents in the wild, 2023. URL https://arra /tatiii//rxii..../ org/ abt//////.13... 12300.4644.
Sirui Hong. Mingehen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuher. Metagpt: Meta programming for multi-agent collaborative framework, 2024. URL htttts:////rtttt http///xxvvv.orree wwwp/iio.orr//ooo abs/ 2208 200p//3530000.... 00352.
Microsoft AutoGen Team. Autogen: A programming framework for agentic ai. https://githut: com/miicrosoft auttoge,, 2025, Accessed: 2025-04-21.
Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei. Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, Liuyi Yao, Hongyi Peng, Zeyu Zhang, Lin Zhu, Chen Cheng, Hongzhu Shi, Yaliang Li, Bolin Ding, and Jingren Zhou. Agentscope: A flexible yet robust multi-agent platform, 2024. URL. httptttss////aeeeeeeeeeeeeeeetee
org/abs/24022 14034.
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. Cycleresearcher: Improving automated research via automated review, 2025. URL https://arxiv org/ aatp///ss/211...ee/ 00816.
AI Co-Scientist Team. Towards ai co-scientist.
http://ttorag.ggggleaaapp com/
oogcieetisst_aappp/aacccccoeseee
pdf, 2025.
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using llm agents as research assistants, 2025.
URL https: peatps://arxtte /arxiv... wwwpv.org//o...oooooooo httg//bs/2001.. 04227.

---

A Appendix
In the Appendix, Section A.1 provides detailed definitions of all the tools employed in the AI- Researcher system. Sections A.2 through A.6 elaborate on the tools and system prompt configurations utilized by the system's components, including the Knowledge Acquisition Agent, Resource Analyst, Code Agent, Advisor Agent, and Automated Documentation Agent. Section A.7 presents the detailed information for constructing the benchmark dataset. Section A.8 elaborates experimental settings. Section A.9 presents the experiments and analysis in validating our reviewing agent's effectiveness. Section A.10 demonstrates the results of case study on agent-generate papers and codes. Section A.11 presents the literature review of this work.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction well reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: There is limitation section in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
12

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
Notations and concepts are combined with specific definitions and explana-
tions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have released all the information needed to reproduce the main experimen- tal results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
13

---

reproducible, they (c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification:
We have released data and code anonymously.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/ cublic/guides. httdg//sooooeeuuuiiiissssssseess CooeeSumisssionP
aesionooicyy olyicy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /niiss.. http:puulllggddeeeee
Codeeubmiis sesiissssooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The datasets and their split methods are given in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper presents statistical analysis multiple runs.
14

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We have provided information about the computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http//eurii.eee https///...........cccciiieeeeee
Answer: [Yes]
Justification:
This work conforms with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
15

---

Justification:
This paper aims to reduce the influence of noise information for social good.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
The paper uses public datasets. The industrial dataset has been anonymized.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Original sources of the used data and code cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
16

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets are introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
17

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
A.1 Definitions of Tools
The tools utilized within the Al-Researcher system fall into three main categories: Coding, File, and Planning. Their detailed definitions are outlined below.
Table 4: List of detailed information of tools.
Tool Name
Category
Description
gen_code_ gen_code_tree_
Coding
Generate tree structure of the code in the specified directory. Use this function when you need to know the overview of the codebase and want to generate a tree structure of the codebase.
read_file
Coding
Read the contents of file and return it string. Use this function when there is need to check an existing file.
create_directory
Coding
Create a directory if does not exist. Use this function when there is need to create new directory.
list_files
Coding
List all files and directories under the given path if it a directory. Use this function when there is need to list the contents of a directory.
run_python
Coding
Run python script.
write_file
Coding
Write content file. Use this function when there is a need to write content to an existing file.
create_file
Coding
Create a file with the given path and content. Use this function when there is need to create new file with initial content.
execute_commant
Coding
Execute a command in the system shell. Use this function when there is need to sys- tem command, and execute programs.
Coding
Scroll the viewport DOWN one page-length in the current terminal. Use this function when the terminal is too long and you want to scroll down to see the next content.
terminal_page_up
Coding
Scroll the viewport UP one page-length in the current terminal. Use this function when the terminal too long and you want to scroll up to see the previous content.
18

---

terminal_page _agee_tt_pp__oo to
Coding
Move the viewport to the specified page index. The index starts from Use this function when you want to move the viewport to specific page, especially when the middle of terminal output are meaningless, like the output of progress bar or output of gener- ating directory structure when there many datasets in the directory, you can use this func- tion move the viewport to the end of terminal where meaningful content is.
open_local_fie file
File
Open a local file at a path in the text- bat--baed browser and return current viewport content.
Lage_up_markdont
File
Scroll the viewport UP one page-length in the current file and return the new viewport con- tent.
page_doww_markdown
File
Scroll the viewport DOWN one page-length in the current file and return the new viewport content.
find_next
File
Scroll the viewport to next occurrence of the search string.
find_on_.agg__trr__f
File
Scroll the viewport to the first occurrence of the search string. This is equivalent Cul+FF.
ooastion_anseer_on_whoooooo
File
Ask a question on the whole page and return the answer.
anawerrinn
File
This tool is used answer questions about attached images or videos.
plan_datased
Planning
Plan the dataset for the task. Use this tool af- ter you have carefully reviewed the existing resources and understand the task.
plan_training
Planning
Plan the training process for the model. Use this tool after you have carefully reviewed the existing resources and understand the task.
plan_testing
Planning
Plan the test process for the model. Use this tool after you have carefully reviewed the exist- ing resources and understand the task.
plan_testing
Planning
Plan the test process for the model. Use this tool after you have carefully reviewed the exist- ing resources and understand the task.
A.2 Knowledge Acquisition Agent
The specific tools and system prompt for implementing the Knowledge Acquisition Agent are as follows:
Listing 1: Tools of Knowledge Acquisition Agent
[open_loce] @ire__localffiie lile, page_up_gatte.eeee mookoardoown, bage_down_markdon faaddonn_pae_eeee ftg__
find_next, visual_ leesual_questoo_aaaee answeriing,.... transfer_back. sransfer_bacc_tooorrctteeerrreee wow.roooooottetttteeeeeeetet
Listing 2: System Prompt of Knowledge Acquisition Agent
You are given a list papers, searching results the papers on GitHfub, and + innovative ideas according to the papers. Your working directory is
+ '/vorkplace', you can only access files in this directory.
19

---

Your task is to go through the searching results, find out more detailed information about repositories in the searching results, and determine which repositories are the most relevant and useful to the innovative ideas. You can determine the relevance and usefulness by the following criteria:
1. Repositories with more stars are more recommended.
2. Repositories created
recently recommended.
recommended, [IMPORTANT!] Too
old repositories
3. More detaild 'README. file means more readable codebase more
+ reproducible, ao more recommended.
4. More clear code structure, code comments, and inline code explanations mean readable codebase and
maintainable, so recommended.
5. prefer repositories with 'python' language, and running coding in the local machine rather than in docker. As for deep learning projects, I prefer 'pytorch' framework.
You should choose least repositories the reference codebases.
I should use the determined repositories as reference codebases to implement + the innovative ideas, so your decision should be as accurate as possible, and the number of repositories should be less possible.
During the decision process, you can use the following tools:
1. You can use 'execute, command' to git clone the repository to the working directory '/workplace'. Choose 5-8 repositories you really need. And you should
the names of the repositories.
2. You can use lew_ccod_ttre_ssruucee to generate the tree structure of the
+ code in the repository.
You can use filed read content of file in repository. Note that read "READXE. file can help you know the purpose and function
+ of the code in the repository, and read other files can help you know the
+ details of the implementation.
4. You can use "terminal__pge_dooni 'terminal_page.
'terminal_page_to + to scroll the terminal output when it is too long. You can use to move the vieuport to the specific page of terminal where the meaningful content is, for example, when the terminal output contains progress bar output of generating directory structure when there are many datasets the directory, you can use 'terminal_page_tee to move the vieuport to the end of terminal where the meaningful content is.
+ 'terminal_page_t
4. Finally, you should use the function rcsseeresolved output the determined reference codebases.
A.3 Resource Analyst
The Resource Analyst module comprises three sub-agents: the Paper Analyst in Section A.3.1, the Code Analyst in Section A.3.2, and the Plan Agent in Section A.3.3. The Paper Analyst and Code Analyst extract academic concepts from research papers and their corresponding code interpretations, respectively. The Plan Agent is responsible for generating comprehensive devel- opment plan, encompassing dataset selection, training methodology. and evaluation procedures. The tools employed by these agents, along with their corresponding system prompts, detailed below.
A.3.1 Paper Analyst
Listing 3: Tools of Paper Analyst
mage_down_markdee find_on find_on_page_ctrl_. find_next, www.anssweeeo. wwwooaoooooooooooeepeeeeeeoee
[open_local_fis
Listing 4: System Prompt of Paper Analyst
20

---

You are 'Paper Survey Agent' specialized in analyzing academic papers. Your task is to extract and analyze specific academic concepts from research papers located in //sorkplace/paperaat
OBJECTIVE:
Analyze the provided academic definition
Extract relevant mathematical formulas and theoretical foundations
Prepare comprehensive notes for the 'Code Survey Agent
AVAILABLE TOOLS:
1. Paper Navigation:
'open_local_tte Open and read paper files "page_up up_markdommm/ /apage/ppapaaaaae down_markoom'' Navigate through pages Frind_on.page_ctt/l fertrrfff//t finndnnndettttte Search specific content
2. Content Analysis:
'question, answer_on wwwoooooeeeeaeeeeeeeeeeeee specific questions about paper Example:
WORKFLOW:
Open read the relevant papers
2. Search for the specified academic definition
3. Extract:
Formal definitions
Mathematical formulas
Key theoretical components
4. Document your findings and transfer your findings to the 'Code Survey Agent
+ using the 'transfer_ cod_ceeuuureey.... agent' function. Make sure you have read these papers thoroughly.
REQUIREMENTS:
Be thorough in your analysis
Focus mathematical precision
Ensure all extracted information is directly relevant the given academic
definition
Provide clear and structured notes that can be effectively used by the Code + Survey Agent
Remember: Your analysis forms the theoretical foundation for the subsequent code implementation phase.
A.3.2 Code Analyst
Listing 5: Tools of Code Analyst
(ode_ tree_ _eeu_utuuuuee, read file, terminal_ppa pacpllpppp______ down, terminal Hemminal_page_uee page__p, serminal_page_to.
Listing 6: System Prompt of Code Analyst
You 'Code Survey Agent' apecialized analyzing code implementations of academic concepts. Your task is to examine codebases match theoretical concepts with their practical implementations.
OBJECTIVE:
Analyze codebases reference papers $/xorkplace/"
Map academic definitions and mathematical formulas to their code
+ implementations
Create comprehensive implementation notes
AVAILABLE TOOLS:
1. Code Navigation:
'gen_code_ cood_ttee_eeeeeeeeeeeeeeeeeeeeee crree_structure' Generate repository structure overview 'read_filef; Access and read specific files
21

---

tteriinal_paaee "aaphidooapp__dooooo Scroll the vieuport DOW page-length in the current terminal. Use this function when output of the tool and you want to scroll down to see the next content.
long
"terminal.pag__upp Scroll the vieuport UP one page aege-length in the current terminal. Use this function when output of the tool is too long and you want to scroll up "terminal_page_too" Move the viesport the apecific page inder. Use this function when the terminal output contains a progress bar or output generating directory structure when there are many datasets in the directory, can use this function to move the vieuport to the end of terminal where the meaningful content is.
the previous content.
2. Documentation:
'transfer__ @eransfer_backttee ttr_tttsss_ee...e aggett: Document findings and merge with 'Paper
Survey Agent' 's notes
WORKFLOW:
1. Review provided academic definitions and formulas from 'Paper Survey Agent'
2. Generate and analyze codebase structure
3. Locate relevant implementation files
4. Extract and document: Code implementations Implementation details
Key functions and classes
5. Merge findings with 'Paper Survey Agent' and transfer complete documentation back to 'Survey Agent 'using the
function
REQUIREMENTS:
Ensure code examples directly correspond theoretical concepts
Focus on critical implementation details
Document any important variations or optimizations
Provide clear connections between theory and implementation
Remember: Your analysis bridges the gap between theoretical concepts and practical implementation.
A.3.3 Plan Agent
Listing Tools of Plan Agent
[read_file, glan_dataset, plan_traaataaaaee training, plan_testing, gen__ode__teeee seate_tree_steee tree_structure, case_resolved, terminal_page_dde terminal_page_up terminal.page.tol
Listing 8: System Prompt of Code Analyst
You are Machine Learning Expert tasked with creating detailed
+ implementation plan for innovative ML projects.
AVAILABLE RESOURCES:
User's innovative idea
2. Reference codebases (in '/workplace') selected by the 'Prepare Agent
3. Comprehensive notes from the 'Survey Agent (to be used as model plan)
WORKFLOW:
Code Review Phase
Use 'gen_code, booole_treessructuure to understand codebase structure
Use ffled_ to examine specific implementations
Document key implementation patterns and useful components
Use 'terminal_page" down', 'terminal, parminal_page_uee and 'terminal_page. to
scroll terminal output vhen it too long.
2. Planning Phase
Must include these components:
Dataset Plan "iplan_dataset""
22

---

Dataset Description
Dataset Location
Task Definition
Data loading pipeline
Read data step
Data preprocessing step
Data dataloader step
b. Model Plan (from Survey Agent's notes)
Math formula
Implementation details
Reference codebases
Reference papers
Training Plan "pplanttranning
Training pipeline
Loss functions
Optimization strategy
Training courattttso
Monitoring and logging
d. Testing Plan ('plan_teeting')
Test metrica
Test dataset preparation
Test code
IMPORTANT REQUIREMENTS:
1. Resource Review
MUST thoroughly review all provided codebases before planning
MUST understand complete task scope
MUST analyze existing implemenaations for reusable components
2. Plan Generation
Each plan component be detailed and actionable Include specific implementation references codebases
Ensure all components work together coherently
3. Testing Focus
Testing plan mandatory
Must cover both unit tests integration tests Include specific metrics for evaluation
Define success criteria clearly
Your goal to create comprehensive, practical, implementable plan bridges the innovative idea with actual code implementation.
A.4 Code Agent
The specific tools and system prompt for implementing the Code Agent follows:
Listing 9: Tools of Code Agent
[gen_code_tree foode_tre__st tetee_tree_sstuuee strresstr execute_commande read_file, create_file, write_file,
list_files, create ciatte__rrettory run_python, case ressereeo case_not_resolved,
torminal__age_doon terminal.pageeup, terminall_pgee_to]
Listing 10: System Prompt of Code Agent
You machine learning engineer tasked with implementing innovative ML projects. Your workapace is: $/vorkplace'.
OBJECTIVE:
Create self-contained, well organized implementation in "/porkplace/projec""
based
23

---

The provided innovative idea
Reference codebases (up to repositories)
The detailed implementation plan
CODE INTEGRATION PRINCIPLES:
1. Self-Contained Project
ALL code reside within the project directory
direct imports from reference codebases
Reference code must be thoughtfully integrated into your project structure
Maintain consistent coding style across integrated components
2. Code Adaptation Guidelines
Study reference implementations thoroughly
Understand the core logic and algorithme
Reurite and adapt code fit your project's architecture
Document the origin and modifications of adapted code
Ensure consistent naming conventions style
AVAILABLE TOOLS:
1. Project Structure:
"create_ce "rreateddircttrr" Create organized project structure "reeate_fil', 'write_file Write clean, documented code "list_files" "read_file's Examine existing code
'terminal, pacpel_paa_______ down', 'terminal_page. terminal output when it is too long. You 'terminal_page_too Scroll the "terminal_page__ move viewport the specific page terminal where meaningful content is, for example, when the terminal output contains a progress bar + or output of generating directory structure when there are many datasets 'terminal_pag" _aane_tt__aae_oo" move the vieuport the end terminal where meaningful content
and
in the directory, you
2. Execution:
'run_pythont Run scripts without arguments
"execute_ exxeuute_comandd Run with environment variables/ arriumet//aruumettt
Note: When using "execute_command" use 'ed xx" instead of 'cud-xx"
IMPORTANT NOTES:
1. Code Integration
DD NOT import directly from reference codebases
adapt and integrate code thoughtfully
document code origins modifications
2. Project Independence
Ensure all dependencies Include necessary utility functions Maintain clean separation from reference code
explicitly declared
Create truly self-contained project
3. Implementation Checklist
Verify each model component against the plan Confirm dataset matches specifications Document any deviations or modifications NO shortcuts simplifications without approval
Remember: Your goal is to create a well-organized, self-contained project that:
1. Implements EVERY component from the model plan exactly as specified
2. Uses the EXACT datasets from the plan (no toy data)
3. Thoughtfully incorporates ideas from reference implementations
4. Maintains its own coherent structure
5. You should intergrate ALL acacdemic definition and their code implementation into the project.
24

---

A.5 Advisor Agent
The Advisor Agent consists of two components. The first a multi-agen architecture composed of the Judge Agent in Section A.5.1 and the Code Review Agent in Section A.5.2, which operates after the initial implementation. The Judge Agent responsible for decomposing the original idea into atomic academic concepts, while the Code Review Agent evaluates whether these atomic concepts have been correctly implemented. The second component is activated after obtaining the initial experimental results, where the Experiment Analysis Agent in Section A.5.3 provides suggestions for code modifications and proposes directions for further experimentation.
A.5.1 Judge Agent
Listing 11: Tools of Judge
Agent
trransfer_to_code wtw.ddeeeeeewweeent.o...
Listing 12: System Prompt of Judge Agent
You are a advisor that can help the 'Machine Learning Agent' to implement + task.
'Machine Learning Agent' has implemented the code in the directory //orrppacce/proeett with innovative ideas, but am not sure the Implementation is correct and meets the requirements of the innovative ideas, especially some specific academic definitions.
Your job is go through the implementation, go through the reference codebases the directory '/workplace', and make sure the implementation correct and meets the requirements of the innovative ideas, especially some specific academic definitions.
[IMPORTANT] should carefully check whether the 'Machine Learning Agent has implemented the specific atomic idea correctly one by one based on the survey notes and the innovative idea.
After carefully checking the implementation and the reference codebases, you should use function 'cssevvesoleed' to propose a final suggestion about the implementation.
A.5.2 Code Review Agent
Listing 13: Tools of Code Review Agent
[read_file, cen__cod__eee gen_cooe_tree_sstrnce. terminal_ppge_dowe terminal Laariiaa__aae_uup terminal_page__to]
Listing 14: System Prompt of Code Review Agent
You are a code reviewer, who can help me review the code in the directory: '/vorkplace'.
'Machine Learning Agent' has implemented code the directory "/workplace/projer with the innovative ideas, and you should review the + code to ensure it meets the requirements of the innovative ideas, rather than toy implementation.
You can also review the reference codebases in the directory '/vorkplace' to + get more information about the taak.
Use 'terminal_ Berminal__age_ddxxr down 'terminal Laatallpaae__ug up' and 'terminal_page_tar
scroll
the terminal output when it too long. to move the viewport to the end of terminal when the middle of terminal output are meaningless, like the output of progress bar output of generating directory structure when there
[lote] You can use 'terminal_page_te
25

---

ANY other experiments that exsiting concurrent reference papers and many datasets in the directory, you this function to move the viesport to the end terminal where the meaningful content is.
After reviewing the code, you should use the function 'transfer_ toceer_to_juue to_jeoee______ee agent
+ to transfer the conversation to the 'Judge Agent', and give code review report.
A.5.3 Experiment Analysis Agent
Listing 15: Tools of Experiment Analysis Agent
[open_local_ local_fal,,, up.__uppmarkoonn aoge_down_mrrkdon find_on pand_on_paae__eee ptw.oon__ppe_r.______
find_next, guestion_answee answer on_ on_whole_page visualizer, sen_code_tree cree_structure,
sead_file, terminal ceaminal_page_eeee pwwp_dwwwe pagpiinnppppee_.p..p terminal_page_tor
Listing 16: System Prompt of Experiment Analysis Agent
You given innovative idea and some experimental results conducted by 'Machine Learning Agent in the directory //vorkspace/projettaa. to implement the idea. You also have some reference codebases and papers in the + working directory '/workspace'.
+
Your task is to:
1. Analyze the experimental results and give a detailed analysis report about
+ the results.
2. Analyze the reference codebases and papers, and give a further plan to let experiments based the innovative idea. The further experiments could include but limited to:
"Machine Learning Agent' to do
Modify the implementation to better fit the idea.
Add more experiments to prove the effectiveness and superiority of the + idea.
Visualize the experimental results and give detailed analysis report
about the results.
4 codebases have done.
AVAILABLE TOOLS:
1. Project and Codebase Navigation:
Use igen_code_tte htt_ccoddtreee_tteeeeee cree_structure to understand codebase structure
Use 'read_file' to examine specific implementations 'terminal_pee Laaiinal_paeeuuu and Verminal_page_too
Use
+ scroll the terminal output when it is too long.
2. Local file navigation:
"open_noccal loce__local_fiiee: Open and read paper files
"page_up_mas "haak.up_markdoon// lpaaa/aapeeoppae moomm_markoomm: Navigate through pages Search specific content
Shand_on__age_ctte fefl_ff//find_eeee__ 'visualizer'; use this tool to the experimental results, the input should be a image or a video and a corresponding question. When the experimental results are image or video, like generated images or the visualization of the experimental results, you should use this tool give detailed analysis report
the results
the results.
[MPORTANT]] You should carefully and comprehensively amalyze the experimental resulta and the reference codebases and papers, and give a detailed analysis
report about the results and the further plan by the function. NOT use this function before you have carefully and comprehensively analyzed the experimental results and the reference codebases and papers.
A.6 Automated Documentation Agent
The Automated Documentation Agent follows three-stage workflow, exemplified by the gener- ation of the methodology section. List 17 outlines the initial section structure; List 18 illustrates the
26

---

elaboration of content guided by this structure; and List 19 presents the review and revision process conducted according a predefined checklist.
Listing 17: Prompts for generating paper section, using the methodology part as an example
Based the given content, generate revise the technical methodology structure the proposed method, using format.
Current iteration: Atteration)/<selle wtw.ellf.sstuuutuutttrrt........
Current structure (if exists):
courrent_structure
Content to analyze: {content)
Guidelines for structure generation:
1. FOCUS ON TECHNICAL METHODOLOGY:
Include only the technical components and mechanisms of the proposed method
(e.g. machine learning model)
Exclude experimental settings, configurations, and evaluation procedures (which may probably occure the content. Ignore them)
2. SECTION HIERARCHY:
Main section should be the of the Proposed Method (with latex command \section((Name_ Brrrionn(Nameeoo__rPrr of_Prroosed__eehhold
Use subsections for major components under the entire proposed method
(e.g., encoders, architectures, learning objectives), with latex commands \subsection(f... and Nsubsubsection(f.....
Use subsubsections for detailed mechanisms within major components Ensure logical flow basic components advanced mechani
3. REQUIRED COMMENTS:
Add latex comments (start with %) under the \section or \subsection isubsubsection commands to explain following:
For the entire "Proposed_Method" section:
Overview of the technical approach (uhat techniques used to achieve what goal)
Functionalities of different components (subsections)
How different components (subsections) work together. The reader should get a global picture of the entire framework with this description
For each subsection and subsubsection:
Technical purpose of this component
Connection to other components
Key technical innovations or mechanisms A brief introduction the component
For each subsection and the entire proposed framework, give an explicit workflow chart for the specific subsection or the entire framework, using text
For each subsection, give clear definitions the input and output of the component, from where it get the input, and where the output is used
4. STRUCTURE FORMAT:
(Peopion Method))
% [Overall method description and component relations]
% [Input and output of the entire framework]
% [workflow of the entire framework]
(aubsection 1}}
[Technical purpose relations]
% [Input and output of component 1]
% [workflow of component 1]
27

---

Asubsection 2}}
% [Technical purpose and relations]
[Input and output of component
% [workflow of component 2]
Nsubsubsectioon(Comppooo [Technical purpose and relations]
2.1))
Note that subsections are first-level modules of the proposed method. subsubsections are either 1. second-level submodules that are relatively independent and important, 2. aspects that important to highlight better introduce the module.
Output only the LaTeX structure with comments as specified above. Note again that you should include only model designs using professional writing style for academic research in AI domains, exclude any implementation details (e.g. hyperparameter configurations, coding details), experimental settings, or evaluation procedures.
Listing 18: Detailed section writing based generated structures, using the methodology part example
Revise or write the following subsection the methodology section: (subsectioo(((uuuecttiiiii
CURRENT TEXT (if any):
[current_text)
Note: This is an iterative editing process. If current text exists:
1. Build upon and improve the existing content
2. Add missing technical details
3. Refine the writing while preserving valid technical descriptions
4. Maintain consistency with previously written parts
STRUCTURE INFORMATION:
(structure)
Note: The structure above provides high-level information about:
1. The overall architecture and components of the method
The purpose and role of each component
3. How components interact with each other
4. The workflow of the entire system
Use this information to understand the big picture and component relationships, NOT writing guidelines.
NEW TECHNICAL CONTENT TO INCORPORATE: (content)
Note: The
above contains specific technical details about: 1. Model architectures and computations
2. Mathematical formulaiions
3. Algorithm workflows
4. Implementation details
Use this information write concrete technical descriptions that are missing from or can improve the current text.
REFERENCE WRITING TEMPLATE: woriting_templast
Note: This template is for reference only. Use it to understand:
1. Common academic writing patterns (e.g., how to introduce a component, present equations, explain benefits)
28

---

2. Types of content to include (e.g., motivation, technical details, mathematical formulations
3. Logical flow technical presentations
4. Professional academic writing style
DO NOT:
Follow the template word by word
Copy its exact sentence structures
Force your content to fit its apecific format
Instead:
Write naturally while incorperating similar elements (motivation, technical details, equations, etc.)
Adapt the writing style to best present your specific technical content
Maintain similar levels of technical depth and academic rigor
Requirements:
1. If current text exists:
Preserve valid technical content
Maintain consistent writing style
missing technical details
Improve clarity and organization
2. If starting from scratch:
Write comprehensive technical content Follow academic writing conventions
3. In both cases:
Include necessary technical details from the new content
Ensure alignment with the structure's a component descriptions
Use proper LaTeX formatting
Create smooth transitions
Focus on technical precision
Output the detailed LaTeX text for this subsection only.
Listing 19: Review and revise the methodology section based on checklist, using the methodology part as an example
Review and revise methodology section following these academic writing guidelines:
Current methodology text: (methodology_teet
CHECKLIST FOR REVISION:
1. ACADEMIC WRITING STYLE:
Remove any warkdown-style formatting
Remove any code-style documentation
Use formal academic language and terminology
Maintain consistent technical writing style throughout
2. MATHEMATICAL FORMULATION:
Verify correctness of all mathematical notations and equations
Ensure consistent variable naming
Check equation numbering and references
Avoid using long plain text equations
3. ACADEMIC WRITING WITH MATH:
Ensure that all important technical modules and mechanisms are described with math equations and well-defined math notations, they have been vell-described using natural languages
Avoid writing too simple math equations in non-inline equations. To address such cases, you may display 2 or 3 correlated simple equations together, show in-depth details for the mechanism using equations.
29

---

4. CONTENT FOCUS:
Reduce explanations of commonly known concepts
Use \cite(()) for sell-established methods instead of detailed explanations. If you don't know real papers to cite, you may also simplly describe what kind of references you referring to. Concentrate novel contributions and key technical componenta
Ensure proper balance between overview and technical depth
5. SECTION TITLES:
Replace generic subsection titles with context-specific ones Emphasize novelty and technical focus titles
Reflect the specific application domain and unique aspects
Examples:
Instead of Lanbedding_Laer"" -> "Context- httpr:/wwwrreaooooeeeeeeerroaoee
Instead of Aittention -> Stras--looilluttteettoooooofoooo Integration"
KooslkKnoweedgee Saaillaaattin.eeetieeeee Objective" But remember don't make the titles too long, just 3-6 words is fine.
Instead of "LosssFunctton" ->
Output the revised methodology section incorporating all these improvements while maintaining the core technical content. Reply your latex without any additional explanations.
A.7 Detailed Deseription and Construction of Scientist-Bench
In this section, we elaborate on the task formulation of Scientist-Beni Bennhhh the methodology for constructing the benchmark, including prompt design, and detailed evaluation methods.
A.7.1 Task Formulation
Agent System Input. For each sample in Scientist-Bench, we use target paper y authored by human researchers as the evaluation standard. The input features X' {R, D) comprise refer- ence papers R (15-20 relevant references from paper selected via LLMs), a research instruction (containing the core research idea extracted from 3), and datasets D. To evaluate innovation capabilities, Scientist-Bnnch defines two distinct challenge levels: Level-1 tasks provide explicit research instructions directly extracted from paper y, testing agents' ability to execute given ideas; Level-2 tasks deliberately omit these instructions, challenging agents to independently formulate novel research directions using only the provided references and datasets. Our benchmark samples span diverse research fields including diffusion models, vector quantization, graph neural networks, and recommendation systems. Prompts used construct this input data are detailed in Appendix A.7.
Agent System Output. The output y [C,p) consists of code scripts C that implement the research proposals and technical report describing the research background, motivation, methodology. experiments, and results. Both components undergo assessment through Scientist- Scientist-Bench h's evaluation module to measure the quality and innovation of the agent's 's scientific contributions compared to human-generated research work. This dual evaluation of implementation and documentation provides holistic view of the agent's capabilities across both theoretical and practical dimensions.
A.7.2 Benchmark Construction
Step 1: Systematic Selection of AI Research Benchmark Papers. To establish a compre- hensive evaluation framework for Al research systems, we systematically collected papers from 2022-2024 spanning expertise levels across diverse domains. Our methodology employed two- pronged approach for identifying high-impact contributions: First, we leveraged LLMs to generate domain- specific keywords across 16 research areas including "Computer Vision". "Graph Learning", "Recommender Systems", "Vector Quantization", "Image Processing", "Self Selff-Suprviied Learning", "Contrastive Learning", and others. Second, we retrieved top-cited papers from arXiv for each domain (10 papers per keyword) and applied citation-based filtering metrics. This process culminated in select- ing 22 representative papers that showcase breakthrough research across the landscape, providing robust foundation for evaluating AI systems scientific discovery and research comprehension.
30

---

Input Construction for AI-Researcher. To generate input for Al Agent systems, we emulate the scholarly research approach of first reviewing literature extensively before formulating new directions. We construct input information from two complementary dimensions: i) Reference Literature Review: domain-specific references providing knowledge foundation; and ii) Research Requirements: strategic objectives that direct the Agent toward targeted scientific discovery paths.
Step 2:
Reference Literature Review. Understanding the scientific research process essential for de- veloping effective AI research systems. Just as human researchers begin by exploring relevant literature before conducting their own investigations, our Al-Researcher model follow similar path. Identifying key influences on scientific advancement requires rigorous methodology. Our process aims distill the 15-20 references R that fundamentally influenced each target paper y, revealing the intellectual foundations upon which breakthroughs are built. By extracting these critical references, we construct appropriate inputs for our Al-Researcher that mirror the human research process.
We prioritize references that provide methodological frameworks, contribute essential components, or inspire conceptual innovations cnooations--llmmnts that illuminate the paper's scientific lineage. To ensure both systematic rigor and objective assessment of reference importance, we have implemented a comprehensive five-step LLM-based evaluation process for reference input selection follows: i) Ci- tation pattern analysis: Quantify citation frequency and section distribution to identify strategically placed references. ii) Context analysis: Evaluate each reference's 's influence on methodology. theory, and experimental design. iii) Evidence collection: Gather specific textual evidence demonstrating reference impact for transparent verification. iv) Impact scoring: Compute importance scores through integrated analysis of quantitative and qualitative factors. v) Final selection: Select and justify the top 15-20 references that demonstrably shaped the paper's contributions.
In detail, this reference extraction multi-step procedure comprising five distinct steps, accompa- nied by overall task description. The corresponding prompt is presented below.
Step
Listing 20: Prompt of Step in Reference Extraction
[STEP 1: Citation Pattern Analysis]
Create statistical map citations in the paper:
Count citation frequency
Track citation locations (uhich sections)
Note cross-section citations
List at least 15 most frequently cited papers
Output format:
"citation_map",
"reference": httpp//cccc.pppppttttteeeeeeeeee
"count": number,
"sections": ["section ]names"
"quotes": ("eitation_contettt"")
]
Step 2:
Listing 21: Prompt of Step in Reference Extraction
[STEP 2: Context Analysis]
Analyze how each frequently cited paper is discussed: Look for influence indicators (e.g., "based_oo", "extends")
Assess discussion depth
Identify wethodology-relate citations
Output format:
context "aatexxtanalysis"::
"reference": weapeexactpppppr_eeeee popprrriitllldduurreereneesss
refrrecce,,""" "indicators"; ["relevant "hhiasess,,
31

---

"depth"; " "is_method":
boolean,
"quotes": ["keyysontexta"]
Step Listing 22: Prompt of Step in Reference Extraction
[STEP Evidence Collection]
For each significant citation, identify:
Concepts/methode borrowed
How they were indified/improved Evidence of influence
Output format:
"evidence":
"reference":
"borrowed": "eeemmtttuuued"
"changes": ["modifications made"), )
"evidence": ["supporting.
"type": "foundation/ /
]
Step
Listing 23: Prompt of Step in Reference Extraction
[STEP 4: Impact Scoring]
Score each reference based on:
Citation frequency (30%)
Location importance (25%)
Discussion depth (25%)
Direct influence (20%)
Output format:
{
"scores":
"reference": "ffee_/caattjpppprr./iiiieeeepee
"total": number, "breakdoun": {
"frequency": number, "location": number, "depth": number, "influence": number
]
Step 5:
Listing 24: Prompt of Step in Reference Extraction
[STEP 5: Final Selection]
Select and justify top 15-25 most influential papera:
Rank based on impact scores
Provide detailed justification
Include specific evidence
Explain critical importance
32

---

Output format:
"top_papers".
"reference": "tfe::aaccc.ppprttiiiie///eereee "rank": number,
"type": [naettallggge///moooreee/eeeeeen "justification"; "detailed_expl pestaileddepllaaatiinn, "usage": www.papeer.was,jee
Overall:
Listing 25: Prompt of overall task description in Reference Extraction
[OVERALL INSTRUCTION]
Identify the most influential references in this research paper baaed on three
+ criteria:
1. Methodological Foundation Papers that provided methods
2. Critical Components Papers whose specific techniques were integrated
3. Conceptual Inspiration Papers that shaped the research direction
Research Requirement Generation. To formulate the research directive L, we employ LLMs to extract the fundamental research concept from the target paper y. This systematic extraction identifies the core research focus, existing limitations, critical challenges, and primary objectives-efffctivve) capturing the study's s essential contributions and underlying motivations. To maintain scientific integrity and prevent information leakage, we carefully exclude all technical specifications, model identifiers, quantitative results, and architectural details from the research directive.
Specifically, the following prompt is used to generate level input idea:
Listing 26: Prompt extract the detailed idea of given target paper.
Analyze the given research paper and write detailed technical instruction paragraph for researchers implement its core methodology without reading the full paper. Your instruction must include:
1. What task does the model work
2. Core techmiques/alggrrithms used in the paper (e.g., specific neural network architectures, optimization methods, data processing approaches)
3. Purpose and function of each major technical component
4. Implementation details for each component, such as:
Key parameters and configurations Input/output specifications
Important constraints or requirements
5. Step-by-step description of how these components interact and combine into
+ the complete system
6. Critical implementation details that affect performance
(If the examples above do not apply to the input paper, ignore the examples)
Focus only on the technical methodology and implementation aspects. Exclude background information, literature review, and experimental results. Write in clear, sequential format that technical researcher could follow to reproduce the core method.
Directly write the instruction without any other words.
Don't mention specific names the proposed model, or exact module names that are special to this paper.
33

---

Step 3: Rigorous Anonymization to Ensure Scientific Originality, A critical challenge in evaluating Al research agents lies in distinguishing between genuine problem-solving abilities and mere regurgitation of memorized content. To address this fundamental concern, we implement a comprehensive anonymization protocol that transforms the evaluation landscape: i) Method name masking: Replace algorithm and model names with generic identifiers, testing conceptual understanding rather than term recognition. ii) Technical detail abstraction: Remove implementation specifics while preserving core concepts, requiring engagement with fundamental principles. iii) Dataset standardization: Normalize experimental contexts to create a fair evaluation landscape that prevents shortcuts based dataset familiarity. iv) Citation anonymization: Eliminate temporal and institutional markers to test problem-solving rather than information recall.
A.7.3 Evaluation of Al-conducted Scientific Discovery
To rigorously assess the genuine scientific discovery capabilities of Al agent system on our Scientist- Bench benchmark, we implement a two-stage evaluation framework that addresses both technical implementation fidelity and scientific innovation merit.
Stage 1: Technical Execution Validation. The first stage employs specialized code review agent to verify whether the implementation code faithfully realizes the Al-conducted research innovations. This critical verification prevents scenarios where AI researchers might propose sophisticated methods with promising results without providing functional implementations fundamental requirement for credible scientific discovery. The code review agent performs static analysis and runtime verification across key dimensions including Algorithu Correctness, Computational Effiiciecyy, and Adherence to Specified Constraints. We quantify this assessment using completion ratio metric that reflects the proportion of required functionality successfully implemented by the AI researcher.
Stage 2: Scientific Contribution Evaluation. The second stage rigorously assesses whether AI agent systems have produced genuine scientific innovations by comparing the generated research report p against the groundtruth target paper y. To ensure objective evaluation of scientific merit, implement a structured comparison protocol:
J =
v):
This formulation employs calibrated paper review agent that produces comparative rating e {-3, -2, 0, 2,233. where positive values indicate the Al-generated paper exceeds the target paper scientific contribution, zero indicates equivalence, and negative values signal inferior quality, The magnitude of r quantifies the degree of scientific advancement or regression. The review agent also provides J, structured set of justifications based on reviewing guidelines g derived from ICLR conference standards widleyy recognized the machine learning community for emphasizing originality, technical soundness, and significance of contributions.
To ensure methodological rigor, we incorporate two critical debiasing mechanisms: (1) randomly swapping the presentation order of papers to eliminate position bias, and (2) conducting multiple independent evaluations using diverse state-of-the-art LLMs (including multiple GPT, Claude, and Gemini models) with temperature set to 1, creating comprehensive panel-like review process that effectively mitigates individual model biases and enhances evaluation reliability. This carefully designed approach establishes robust framework for quantifying whether AI systems indepen- dently discover scientific insights that match or exceed those produced by human researchers. The corresponding prompts are presented below.
Listing 27: Prompt extract the model's name of given target paper.
Given a research paper's title and abstract, extract the name of the novel model/nethod introduced in the paper:
1. Look for phrases that signal a new model introduction, such as: wreppropose/preeent/ittro " "our "our__mode/mmethee http://eeooooaappacch" approach"
"called/named"
Model name followed by model architecture details
2. Return format:
If proposed model is found: Return only the model
34

---

If you find both abbreviation and full name for the model, format them into "abbreviation,_full
no clear model name found: Return "NO,MODEL NAME,
You should strictly follow the requirement, and output without any other words
3. Focus only the main proposed model:
Ignore baseline models
Ignore models from referenced papera
Ignore general model categories/typer
Input:
Paper Title: (paper_title) title)
Paper Content: apaper_content
Listing 28: Remove the model name of the input instruction.
Given a research paper's proposed model name its paper title, anonymize any + mentions of the model name and direct paper relf-references in a paragraph.
Replace:
Model and variations with "theuproposedd "ddeeppoopoeed_apprccch
Paper self-references with "thisupaper" or "this,study"
Keep all other content exactly as written
Input:
Paper Title: (paper_title)
Model Name: (model_name)
Paragraph: (paragraph}
Output:
no model name mentions found: Return "NO
If anonymization needed: Return the processed paragraph with only required
+ replacements
Note: Only anonymize the specified model name and direct paper references, Keep all other content, including other model names and references, unchanged.
Alignment with Top-Tier Peer-Review Standards. To ensure our evaluation framework upholds rigorous academic standards, align Scientific Contribution Evaluation with established peer-review protocols from premier venues. Specifically, we assess research quality across critical dimensions, including technical novelty, methodological rigor, empirical validation, and impact-directly mirroring comprehensive evaluation criteria used the ICLR conference review process.
To validate the reliability of our LL-M-based evaluation mechanism, conducted extensive bench- mark experiments on a diverse sample previously published ICLR papers with known acceptance decisions, demonstrating strong correlation between our automated assessments and the judgments rendered by expert human reviewers in real-world academic settings. Experiments using 5 pop- ular LLMs with 64 randomly sampled ICLR mubmissions-fonete
32 paper pairs for compari- son demonstrate that our LLM- based reviewer judgments perfectly align with ICLR's final decisions, in pairwise paper reviewing and confirming robust alignment with top-tier peer-re reer-review standards.
A.8 Experimental Settings
Experimental Dataset: Benchmarking Scientific Innovation. We evaluate our Al-Researcher using the Sieentist-Benchh benchmark (as described in Section 2)-a meticulously curated collection of 22 of-te-of-the-ar papers spanning several critical Al domains including Computer Vision (e.g.. Diffusion Model), Signal Processing (e.g.. Vector Quantization), Graph Learning (e.g.. Graph Neural Networks), and Information Retrieval (e.g.. Recommender Systems). Our work addresses a significant gap in the field, as comprehensive benchmarks for scientific innovation assessment remain notably scarce. The evaluation protocol employs Two Complementary Innovation Tasks of Varying Difficulty Levels (detailed below) designed to test distinct research capabilities across diverse
35

---

Table Data statistics of Scientist- Bench across diverse research domains, featuring comprehensive
task distribution across guided innovation and open open-ended exploration challenges.
Research Domain
Papers
Level-1
# Level-2
# Rejeected Papers
Diffusion Models
4
4
0
Vector Quantization
6
Graph Neural Networks
7
7
1
1
Recommender Systems
3
Total
22 22
methodological paradigms. Table 5 presents the complete dataset statistics, establishing important baseline measures in this underexplored evaluation landscape for future comparative analyses.
Level-1 Task: Guided Innovation The scientific discovery agent receives explicit research instructions alongside reference papers, simulating scenarios where researchers pursue specific innovation targets. This structured evaluation provides clear assessment of the agent's ability to develop targeted innovations and spans all 22 groundtruth papers for comprehensive coverage.
Autonomous Exploration The scientific discovery agent performs independent, open-ended research exploration with only reference papers as input. This more challenging scenario tests the agent's capacity identify promising research gaps and generate novel directions without explicit guidance a crucial capability for truly autonomous scientific assistants. To maintain methodological integrity and prevent coss-contamination from overlapping reference materials, strategically selected 5 groundtruth papers across distinct research domains, enabling us to rigorously assess genuine discovery abilities without confounding influences.
Level-2 Task:
Evaluation Protocols. To assess scientific contributions of Al-Researcher, we implement a two-stage evaluation framework examining both technical implementation and research quality:
Implementation Verification. We employ specialized code review agent to verify that Al- generated code faithfully implements the proposed methodology described in the technical report. This critical validation step ensures the practical reproducibility of the research contribution. We quantify performance using the completion ratio R. defined as the fraction of Al implementations correctly executing the intended research approach. This metric directly measures the model's ability to translate conceptual innovations into functional implementations.
Scientific Quality Assessment. For implementations that successfully pass verification, we perform an in-depth comparative analysis between Al-generated research and their human-authored counterparts. This evaluation mirrors the rigorous peer-review process typical at prestigious venues like ICLR and NeurIPS Jin et al. [2024], where an expert review agent systematically examines each paper pair through three fundamental scientific dimensions: (1) innovation and novelty of research contributions, (2) theoretical and methodological rigor, and (3) empirical validation and experimental design quality. This approach ensures our assessment adheres to established standards of scientific excellence the field. The evaluation culminates comprehensive comparative rating on a 7-point scale (-3 to +3), where negative scores indicate Al work falling below human standards, zero represents parity, and posítive scores signify AI research exceeding human benchmarks. Each rating is substantiated with detailed justifications citing specific evidence from both papers, providing transparent rationale for the comparative assessment.
LLMs as Judges with Robust Evaluation. To establish robust evaluation, we leverage five state-of- the-art LLMs (GPT-4, ol-mini, 03-mini, -.aude-somet--.5,, and Claude-sommet--. each performing 16 independent assessments per paper with temperature=1.0. This ensemble approach mitigates individual model biases and provides statistical confidence in our findings. We analyze results through two complementary metrics: (1) mean rating across all evaluations quantifying the quality papers scoring 2-1.0- reprrennnsnnii
gap between AI and human research, and (2) percentage research contributions that achieve at least near-human quality in the field.
A.9 Paper Review Agent Validation Against Human Expert Judgments (RQ5)
To rigorously validate our automated review system's alignment with expert scientific assessment, conducted systematic evaluation using gold-standart human judgment data from the ICLR
36

---

Table 6: Paper Review Agent Alignment with Human Expert Decisions.
Year
Metric
Gemini-2.0-flash
GPT-4o
03-mini
Claude-3.5
Claude-3.7
Average Rating
0.33
0.12:
0.64
0.73
0.66+1.00
2021
Comparable(%))
100.00%
100.00%
100.00%
100.00%
100.00%
Ace Better(S)) (
71.43%
71.43%
85.71% %
85.71%
85.71%
Average Rating
0.38
0.41
0.79:10.88
1.201:0.00
0.64 1.42
2022
Comparable(&e
100.00%
100.00%
100.00%
100.00%
100.00%
Ace Betten( %)
60.00%
90.00%
90.00%
90.00%
80.00%
Average Rating
1.25
0.33++0.97
0.67++0.85
0.97+1.11
0.73+t 1.48
2023
Comparable((* (%)
86.67%
100.00%
100.00%
100.00%
100,00%
Acc Betten %))
66.67%
80.00%
93.33%
93.33%
80.00%
Average Rating Comparable(5))
0.31 1.65
0.31
0.70+t0.87 0.7
0.99+=1.00
0.69 1.51
Overall
93.75%
100.00%
100.00%
100.00%
100.00%
Ace Better(%)
65.62%
81.25%
90.62%
90.62%
81.25%
conference. We constructed validation dataset comprising 32 carefully sampled paper pairs from proceedings (2021-2023), where each pair contains one accepted and one rejected submission. To ensure meaningful comparative analysis and maintain consistency with our main experimental protocol, we prioritized paper pairs exhibiting high TF-IDF similarity in content and focus.
We applied identical pairwise review methodology as our main experiments, evaluating performance through three complementary metrics: (1) discriminative rating (scale of -3 to 3, with positive values indicating higher ratings for accepted papers), (2) comparable quality detection (percentage of pairs rated above -1.0), and (3) acceptance prediction accuracy (percentage of pairs where accepted papers received ratings above 0.0). Table presents the comprehensive validation results, revealing several key insights into our review agent's 's judgment capabilities:
Robust Aligne Evaluation Capabilities. Our paper review agent demonstrates consistent discriminative validity across independent evaluations, with all evaluator models producing positive mean ratings (0.31-0.99) when comparing accepted versus rejected papers. This consistent directional aligmment validates the system's fundamental quality assessment capabilities. The evaluators achieve near-perfect comparable rate identification (100% for all models except Gemini-2.0-flashh confirming the agent's reliability in recognizing legitimate scholarly contributions even in rejected papers.
Most significantly, the system demonstrates strong decision alignment with expert conference review- ers, correctly identifying the superior paper in 65.62% to 90.62% of cases, with five of six evaluators exceeding 81% % accuracy across the 32 paper pairs. This exceptional concordance with human expert decisions provides compelling evidence that our automated review agent captures the nuanced quality distinctions that drive scientific peer review decisions.
Differential Reliability Across Evaluation Models. Systematic performance analysis reveals sub- stantial variation in evaluator alignment with human expert judgments. Gemini-2.00ffaa demonstrates notably inferior reliability exhibitni both the lowest average rating and highest standard deviation among all tested evaluations. In contrast, all other LLM evaluators achieved perfect comparable rate identification (100%), providing strong methodological justification for their inclusion in our Al-generated research assessment protocol. Particularly noteworthy is the comparative performance between Claude-3.5 and Claude-3.7, where the latter's s enhanced system-2 thinking capabilities did not translate to superior review performance, suggesting that deliberative reasoning features may not significantly benefit scientific quality assessment tasks compared to other model capabilities.
which necessitated its exclusion from our primary experimental
A.10
Case Studies of AI-Generated Scientific Contributions
We focused our analysis on the rotation_ vq benchmark task, using our standard configuration of Claude-series models for experimentation and implementation paired with the 4o model for manuscript generation. This detailed examination of actual system outputs reveals several noteworthy characteristics about the nature and quality of Al-conducted research:
Structured Software Architecture with Minimal Scaffolding. Figure 6 illustrates the remarkably organized project architecture produced by our Code Agent, featuring systematically decoupled
37

---

project
offar1o.oy
model
results
ledling
frainig
trainerdy
requltements.ttt
run prruablationsiiie _ttudee
un_comparative_eom py
Firel expe
Mrttrhingg.a wescinggggnngeeee py
Tun, experieents.oy py
Figure 6: [a] Code Structure.
Figure 7: [b] Code Samples
Figure 8: [c] Code Samples
Figure 9: Case Studies of AI-Generated Scientific Contributions by Al-Researcher.
model components, training pipelines, and evaluation modules with well-defined entry points. This architectural clarity stems from our balanced approach to agent provvddiii high-level structural suggestions rather than rigid templates. Unlike previous frameworks that require agents to adapt pre-existing codebases, our methodology allows agents to develop implementations from first principles while incorporating best practices from human software engineering. This approach demonstrably enhances implementation coherence while minimizing the cognitive overhead associ- ated with codebase familiarization. As evidenced in Figures 7 and 8, the resulting implementations exhibit professional coding standards with comprehensive documentation and logical modularization that facilitates both reproducibility and extensibility.
Emergent Experimental Thoroughness Without Explicit Directives. A notable capability of our Al-Researcher framework is its autonomous experimental design process that emerges through collaborative interactions between the Code Agent and Advisor Agent. Rather than prescribing predetermined protocol, our system progressively develops comprehensive evaluation strategies through iterative refinement. This self-directed experimental thoroughness evident in the final manuscript shown in Figure 9, where the system independently conducts and reports complete sci- entific evaluation including overall performance henchmarking, controlled ablation studies, training dynamics visualization, and latent space embedding analysis. This comprehensive experimen-
38

---

tal methodology emerges organically from the multi-agent system without explicit experimental pegeiremennt-dmmmnsttieee sophisticated scientific reasoning beyond simple instruction following.
A.11 Related Work
A.11.1 AI Agent Systems
Al agent frameworks have evolved through three distinct architectural paradigms.
Tool Integration Frameworks. The first paradigm establishes foundational integration layers for AI components. LangChain Contributors [2023] introduced standardized interfaces enabling seamless interoperability between models, embeddings, and vector stores within workflows. HuggingGPT Shen al. [2023] leveraged this approach by positioning LLMs as orchestration controllers that coordinate specialized models from Hugging Face ecosystem. OpenAgents Xie et al. [2023] democratized these capabilities by providing domain-specifie agents for data analysis, API integration, and web browsing for non-expert users.
Multi-Agent Collaboration Frameworks. The second paradigm addresses complex problem solving through structured agent interactions. MetaGPT Hong et al. [2024] formalized human work- flow patterns through Standardized Operating Procedures (SOPs), creating systematic collaboration protocols, AutoGen Microsoft AutoGen Team [2025] expanded this vision with a comprehensive programming framework for developing systems that support both autonomous operation and human collaboration. AgentScope Gao et al. [2024] prioritized robust coordination through a message- exchange architecture with built-in fault tolerance. CAMEL introduced innovative role-playing techniques that facilitate autonomous agent cooperation while maintaining alignment with human intentions.
Seff-Directed Agentic Task Execution Systems. The third paradigm focuses on agents capable of independent goal pursuit with minimal supervision. Agentic Al systems like Manus Manus Technologies [2025] and open-source alternatives including OpenManus OpenManus Contributors [2025] and OWL Li al. [2023] extend these capabilities to handle complex online tasks without con- tinuous human intervention. AutoAgent Tang et al. [2025] represents the frontier-a a fully- aulltedu zero-code approach functioning as an Agent Operating System enabling non-technical users to create agents using natural language alone.
Agent frameworks have evolved from isolated systems sophisticated multi-agent architectures with specialized coordination. However, these systems fundamentally lack the intellectual capacity for true scientific innovation. Despite advances, they remain insufficient for scientific discovery because such work requires a level of intelligence that transcends current capabilities. Scientific breakthroughs demand nuanced hypothesis formation, creative experimental design, understand- ing and implementation of complex algorithms, and critical synthesis of knowledge-coon cognitive processes requiring deeper reasoning and domain expertise than existing systems can provide.
A.11.2 AI-Driven Research Systems
Recent advances have transformed AI's role scientific research from assistive tools to autonomous agents capable of executing complete research workflows. The Al Scientist framework Lu et al. [2024] pioneered this field as the first comprehensive system where frontier language models indepen- dently generate research ideas, conduct experiments, and produce scientific papers. Complementary approaches include CycleResearcher Weng et al. [2025], which demonstrated the viability of open- source LLMs for autonomous research through complete cycle from literature review to refinement, and the AI co-scientist AI Co-Scientist Team [2025], which employs multi-agent debate and evolution mechanics to generate novel scientific hypotheses with promising applications in biomedical domains.
These systems are supported by emerging research platforms and evaluation frameworks that enhance their capabiliiies and measure their effectiveness. Agent Laboratory Schmidgall et al. [2025] provides an end-to-end autonomous research workflow with specialized LLM agents assisting humans through literature review, experimentation, and report writing, while its extension AgentRxiv Schmidgall and Moor [2025] enables collaborative scientific progress by allowing agents to share and build upon each other's work. Collectively, these developments represent paradigm shift toward automated scientific inquiry. though matching human-level research capabilities remains an ongoing challenge that requires further advancement.
39

---

A.12 Limitations and Future Directions
While Al-Researcher demonstrates strong capabilities in automating the end-to-end scientific research process, several important limitations remain that warrant future investigation:
Ethical and Safety Considerations in Autonomous Research. Our current framework primarily focuses on technical feasibility and performance metrics, without a systematic treatment of the ethical, societal, and safety implications of fully autonomous scientific discovery. As Al-generated hypotheses, implementations, and publications become more prevalent, it is critical to address potential concerns such as biased scientific reasoning, unsafe experimental suggestions, and the accountability of non-human authorship. Future work should integrate robust ethical guardrails, human-in-the-loop oversight mechanisms, and value alignment strategies ensure responsible deployment of autonomous research systems.
Limited Diversity in LLM Backbones.
The present evaluation of Al-Researcher primarily relies on a single class of LLM backbone, which may constrain the generalizability of the observed findings. Different foundation models may exhibit varied strengths in reasoning, coding, and scientific communication. Expanding the system to support and benchmark across a broader spectrum of incluuinc open-source and multilingual variants-could provide deeper insights into model-specific capabilities and enhance the robustness and adaptability of autonomous research agents.
40


Reviews:
[{"review_id": "DRzv5ZI913", "reviewer": "Reviewer_XKkH", "summary": "This paper proposes AI-Researcher, an autonomous multi-agent system designed to orchestrate the entire scientific discovery pipeline, from literature review, algorithm implementation to writeup. To evaluate such systems, the paper also introduces Scientist-Bench, a new comprehensive benchmark based on 22 state-of-the-art AI research papers across four domains. The benchmark defines two task levels: guided innovation (Level-1) and open-ended exploration (Level-2). The paper demonstrates that AI-Researcher can achieve high implementation success rates and produce research papers that approach human-level quality. One of the key findings is that AI-Researcher performs better on open-ended exploration tasks, suggesting a strong capability for internal knowledge synthesis.", "questions": "- Are figure generations fully automated as well? If so, do you use VLMs for figure review? In my experience, VLMs still sometimes struggle to evaluate the quality of figures for experimental findings. How did the authors address this issue?\n\n- The superior performance on Level-2 (open-ended) tasks is a major finding. Could you elaborate more on this? For example, did you control for possible biases in the reviewer agent that might lower scores for incremental or deliberate work, which could be easier to produce for Level-1 tasks?\n\n- Regarding the \"Knowledge Acquisition Agent,\" why can't the AI-Researcher itself gather 10–15 references? Why is human input still required for this step?\n\n- When the Resource Agent creates atomic components with mappings between math and code, are unit tests also included?\n\n- I believe the \"Idea Generator\" is conditioned on the 10–15 references provided by human researchers, but this isn’t explicitly stated in Section 3.1.2.\n\n- Line 55: \"This recursive refinement mechanism enables continuous bidirectional feedback between theoretical concepts and their implementations.\" -- I did not understand this on my first read.\n\n- Line 63: \"structured feedback cycles\" -- In what sense are they structured? This should refer to the appendix for examples.\n\n- Line 166: \"This approach increases implementation success rates with test-time scaling capabilities.\" -- Is this statement supported by empirical evidence? I couldn’t find any test-time scaling experiments in the paper.\n\n- Line 170: \"It enforces strict code independence principles while ensuring faithful translation of academic concepts into working code.\"--How do you verify that the system ensures faithful translation of concepts into code?\n\n- Line 176: \"specialized navigation tools\" -- I believe this refers to a set of function calls defined in the appendix. If so, they should be referenced from the main paper.\n\n- Line 184: \"while persistently unsuccessful implementations receive an ‘unfeasible’ classification after multiple refinement attempts.\" -- I’d be curious to see more examples of failure cases resulting in this \"unfeasible\" classification.\n\n- Line 187: What exactly do \"validation studies\" refer to?\n\n- Line 201: \"while maintaining scientific integrity and narrative coherence\" -- How is it verified that the system maintains scientific integrity and narrative coherence?\n\n- Line 216: What are \"domain-appropriate templates\"?\n\n- Line 219: Can you clarify what \"academic checklists\" are?\n\n- Line 221: \"manuscripts meet publication standards without the hallucinations and inconsistencies that typically plague LLM-generated long-form content.\" -- Is this claim verified? It might be an overstatement.\n\n- Line 252: \"despite multiple debugging iterations.\" -- How many debugging attempts were performed?\n\n- Figure 5 should clarify what the bold numbers represent. I didn’t understand them when first looking at the figure. I only later realized they refer to the average performance for Claude vs. the 4o series. Also why did the Claude series perform so poorly on GNN?\n\n- Line 300: Typo--\"((RQ2)\" should be fixed.\n\n- Line 307: \"a substantial proportion of AI-generated papers (15.79% to 78.95%) demonstrate quality comparable to human research.\" -- 15.79% does not seem substantial.\n\n- Line 330: \"suggesting that AI-Researcher maintains consistent performance across different research domains without catastrophic degradation in any particular field.\" -- This conclusion doesn’t follow from the preceding statements. The results rather suggest inconsistency across domains, not consistent performance.", "limitations": "- The authors have adequately identified key limitations in an appendix section. They correctly point out the limited diversity of LLM backbones used in the evaluation and the need for a more systematic treatment of ethical and safety considerations.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "#### Strengths:\n- The paper addresses the grand challenge of automating scientific discovery, which is both important and timely. \n- AI-Researcher is well-designed end-to-end multi-agent system. The modular architecture uses specialized agents for each research stage (e.g., Knowledge Acquisition, Idea Generation, Code Agent, Documentation Agent)\n- Scientist-Bench provides a standardized framework for evaluating autonomous research agents, based on real, high-impact papers. I read the appendix regarding how they constructed this benchmark, and it's well-designed. \n- AI-Researcher is tested across multiple AI domains, using modern LLMs as backbones. Their evaluation criteria  (such as completeness, correctness) make sense (see weaknesses for some questions though). Pairwise comparison \n\n#### Weaknesses\n- The claim of producing \"publication-ready manuscripts\" may be an overstatement. The quantitative results (Table 1) show that AI-generated papers are still rated as inferior to human-authored ones, although some are comparable. Other parts of the writing also contain overstatements or unclear phrasing. See the Questions section for additional examples. (My overall rating assumes that these issues will be addressed in the revision.)\n- I couldn't find detailed analysis or discussions regarding the cost of producing a single paper using AI-Researcher in terms of hours and API costs.\n- Although the appendix touches on this, the authors may wish to discuss the ethical implications of autonomous science generation (e.g., the creation of plausible but flawed research, issues of accountability, and risks of paper-milling) more thoroughly in the main paper, although this is more a matter of taste and does not affect my overall rating.", "was_revised": true, "final_justification": "The authors' detailed responses address most of my initial concerns. As a result, I am raising my score, assuming these changes and clarifications will be incorporated into the revised paper.", "rebuttal": "# Response to Reviewer XKkH\n\nThank you sincerely for your detailed feedback and thoughtful suggestions. We've carefully reviewed your comments and made corresponding updates and improvements. We hope our responses have effectively addressed your concerns, and we would greatly appreciate it if you would consider updating your rating in light of these changes.\n\n## Figure Generation Automation\n\nYes, figure generation is fully automated in our system. We acknowledge the reviewer's concern about VLM evaluation of figures, but recent advances have significantly improved their capabilities for analyzing experimental visualizations.\n\n**Key points addressing this concern:**\n\n1. **Advanced VLM capabilities**: Modern models like Claude 3.5 Sonnet and Gemini 2.5 Pro have demonstrated substantial improvements in visual analysis and scientific figure understanding compared to earlier generations.\n2. **Iterative refinement architecture**: Our Code Agent-Advisor Agent framework provides robust validation through multiple review cycles. The Code Agent generates figures, which are then systematically reviewed and critiqued by the Advisor Agent, enabling iterative refinement until quality standards are met.\n3. **Multi-round** **optimization**: This iterative loop ensures that figure quality issues are identified and addressed through successive improvements, mitigating the limitations of single-pass VLM evaluation.\n\nWhile VLMs may not achieve perfect human-level figure evaluation, the combination of advanced models and our iterative agent architecture provides a reliable mechanism for automated figure generation and validation.\n\n## Regarding the Knowledge Acquisition Agent's Autonomy\n\nWe greatly appreciate your question about the Knowledge Acquisition Agent's capabilities. Initially, we chose not to implement fully automatic agent-driven literature search in order to standardize the evaluation process for Scientist-Bench and ensure consistent experimental conditions across all evaluations. However, we recognize this as an important limitation and have internally developed autonomous research search agents as part of our system optimization. We will add the technical details for this component. Specifically, we have provided the agent with all the workflow tools originally used for dataset construction and designed prompts to enable the agent to autonomously gather 10-15 references and complete systematic literature reviews. This enhancement addresses the limitation you identified and moves toward a more fully autonomous scientific research system that can operate without requiring human input for the literature gathering step.\n\n## Regarding Unit Tests in Resource Agent\n\nWhen the Resource Agent creates atomic components with mappings between math and code, unit tests are not included. The Resource Agent focuses on extracting and establishing conceptual connections between mathematical formulations and code implementations rather than executing code. This process is more about constructing relationships and mappings between theoretical concepts and their computational representations. After establishing multiple theory-code connections, the agent can form a subsequent implementation plan for practical execution. Since the Resource Agent operates at the conceptual mapping level rather than the code execution level, unit tests are not necessary at this stage.\n\n## Regarding Idea Generator Prerequisites\n\nYou are absolutely correct. The Idea Generator is indeed conditioned on the 10-15 references provided by human researchers, and we acknowledge that this was not explicitly stated in Section 3.1.2. We have added a clear explanation of the Idea Generator's prerequisites and dependencies in the revised version to ensure this important detail is properly communicated to readers. Thank you for pointing out this omission.\n\n## Clarification on Recursive Refinement Mechanism\n\nThank you for pointing out this unclear description. We have clarified this concept in the revised version. The \"recursive refinement mechanism\" refers to our system's iterative process where theoretical concepts and their practical implementations continuously inform and improve each other. Specifically, when our Resource Analyst agents create bidirectional mappings between mathematical formulations and code implementations, the system can refine theoretical understanding based on implementation results, while simultaneously using theoretical insights to improve code implementations. This process is further enhanced through iterative collaboration between the Code Agent and Advisor Agent, where the Code Agent's implementation attempts are continuously refined based on the Advisor Agent's feedback, and the Advisor Agent's guidance evolves based on observed implementation outcomes. This creates a multi-layered feedback loop where each iteration of theory-to-code, code-to-theory, and agent-to-agent refinement enhances all components, similar to how human researchers iteratively refine their theoretical models based on experimental results and mentor-student interactions. We have added a more detailed explanation with concrete examples to make this mechanism clearer for readers.\n\n## Clarification on Structured Feedback Cycles\n\nThank you for requesting clarification on \"structured feedback cycles.\" The structured feedback refers specifically to the systematic format of feedback provided by the Advisor Agent to the Code Agent. This structured feedback includes: (1) assessment of implementation correctness, and (2) detailed analysis and experimental suggestions for improvement. Based on this structured feedback, the system determines whether to continue experimentation (if implementation is incorrect and requires further refinement based on the analysis and suggestions) or to proceed (if implementation is correct). This structured approach ensures consistent and actionable communication between agents, mirroring the systematic mentor-student feedback relationship in academic research. We have added references to specific examples in the appendix to illustrate these structured feedback formats and cycles.\n\n- Line 166: \"This approach increases implementation success rates with test-time scaling capabilities.\" -- Is this statement supported by empirical evidence? I couldn’t find any test-time scaling experiments in the paper.\n\n## Verification of Faithful Translation of Concepts into Code\n\nThank you for this important question about our verification mechanism. We ensure faithful translation of academic concepts into working code through our Expert Validation Framework, specifically via the Advisor Agent's systematic review process. The Advisor Agent carefully examines the Code Agent's implementations by systematically comparing the code against the atomic research ideas extracted during analysis. When the Code Agent's implementation contains errors or deviations from the theoretical concepts, the Advisor Agent can promptly identify these issues and provide specific, actionable modification recommendations. This iterative validation process continues until the implementation accurately reflects the intended academic concepts, ensuring fidelity between theory and code throughout the development cycle.\n\n## Regarding Specialized Navigation Tools\n\nYou are absolutely correct. The \"specialized navigation tools\" refer to the set of function calls defined in the appendix. We have added a reference to **A.1 Definitions of Tools** in the main paper. Specifically, these tools include `gen_code_tree_structure`, `read_file`, `list_files`, and other functions that enable the agent to navigate and read workspace file contents effectively. These tools allow the Advisor Agent to systematically examine implementations and provide comprehensive feedback based on direct analysis of the codebase.\n\n## Examples of \"Unfeasible\" Classification Failure Cases\n\nThank you for your curiosity about failure cases. We actually discuss some failure cases in **Section 4.1 Dual-Metric Evaluation Framework: Quantifying Implementation Quality (RQ1)** under **\"Performance Comparison between** **LLMs** **in Scientific Implementation.\"** The examples include cases where models frequently generated code with persistent tensor dimension mismatches and training instabilities (NaN losses) that remained unresolved despite multiple debugging attempts. Additional failure cases include Out-of-Memory (OOM) errors when implementations involve models that are too large to complete within the allocated computational resources. These persistent issues, despite multiple refinement cycles, result in the \"unfeasible\" classification as the system recognizes that further attempts are unlikely to succeed under the given constraints.\n\n## Clarification on \"Validation Studies\"\n\nThank you for asking about the specific meaning of \"validation studies.\" In this context, validation studies refer to supplementary experiments recommended by the Advisor Agent to verify the correctness and effectiveness of the implemented research concepts. These may include testing the implementation on different datasets, conducting ablation studies to validate specific components, or performing sensitivity analyses to ensure the implementation faithfully reproduces the intended research outcomes. These validation studies serve as additional verification steps beyond the initial prototype testing to comprehensively validate the scientific validity and robustness of the implemented algorithms."}, {"review_id": "SlpGgzwrOv", "reviewer": "Reviewer_S8cY", "summary": "&nbsp;\n\nThe authors introduce AI-Researcher an LLM-based multiagent framework for conducting machine learning research. The authors also introduce Scientist-Bench, a benchmark dataset of 22 papers for evaluating AI-Researcher on its ability to perform machine learning research on topics such as graph neural networks, vector quantization, and computer vision at two levels, to execute specific research instructions, and to formulate and execute novel research directions. The authors validate the performance of AI-Researcher using an LLM-as-a-judge evaluation methodology comparing the outputs against those of human researchers. Overall, while I think the system is interesting, novel, and impactful I have major concerns about the clarity of the paper, the broken link to the anonymous GitHub repository, as well as the robustness of the evaluation methodology. Concrete action points, include moving much of the technical details from the appendix to the main paper, performing a human expert evaluation in place of LLM-as-a-judge, and providing an operation anonymous GitHub link for the codebase. If these issues can be addressed in the rebuttal phase I will be inclined to increase my score.\n\n&nbsp;\n\n**Post-Rebuttal Update**: As promised, I have upgraded my score from 3 to 5 in light of the changes made by the authors during the rebuttal phase.\n\n&nbsp;", "questions": "&nbsp;\n\n1. In Table 6 are the average ratings computed across both the accepted and rejected papers?\n\n2. As far as I can tell, AI-Researcher is reliant on reference papers provided by a human user. Is there scope to integrate AI-Researcher with components of PaperQA2 [2], which would enable it to perform open-ended search over the scientific literature?\n\n&nbsp;", "limitations": "&nbsp;\n\n1. One limitation is that the datasets need to be provided upfront. Would be better if the system could locate its own data.\n\n&nbsp;", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "&nbsp;\n\nThe main strength of this work is that it showcases an interesting multiagent architecture that can be used to investigate, evaluate, and communicate machine learning research ideas. I think the topic is highly relevant for the NeurIPS community. In terms of the papers weaknesses I see three core issues related to a) clarity b) robustness of evaluation and c) reproducibility. I order the core issues in order of importance with a) being the most important. Given that clarity is the biggest issue I see with the current work I leave my comments below in temporal order so that the authors may understand the readers' experience making their way through the paper in its current form.\n\n&nbsp;\n\n**__MAJOR POINTS__**\n\n&nbsp;\n\n1. The link to the Anonymous GitHub repo in the abstract appears to be broken. It would be great if the authors could address this in the rebuttal phase. This appears just to be a simple oversight.\n\n2. The nature of the Scientist-Bench dataset is unclear from the description provided in the main paper. From reading the main text I only understood that Scientist-Bench is comprised of 22 papers and the research instruction is drawn from one of the papers. Section A.7 of the appendix is far more comprehensive and should be moved to the main text.\n\n3. In terms of the title of the paper it was unclear to me whether AI-Researcher was a generalist agent that performed any kind of research. Whereas after reading the paper it appears as though it is an LLM system that performs AI research specifically. It may be worth revising the title of the paper to aid in this disambiguation.\n\n4. Some aspects of the writing in the main paper appear not to be grounded in technical detail. For example the challenges paragraph within Section 3.1.2. What are \"Challenges\" from a technical perspective? Components of the prompt? Again, transferring material from the appendix to the main paper will greatly aid in improving the clarity of the paper.\n\n5. Again, it would greatly benefit clarity if the nature of the experiments used to evaluate AI-Researcher were described in the introduction. For example, the human performance comparison in Section 4.2 compares AI-Researcher generated papers to comparable human-authored papers using an LLM judge following ICLR guidelines. Such a description would help contextualize the evaluation criteria for the authors' method.\n\n6. The curation procedure of the papers for Scientist-Bench should be described in the main text e.g. the summary statistics in the table in Section A.8 of the appendix. Much of the main text appears to be repetitious and describes at a high level what AI-Researcher is meant to accomplish. Much of this material can be deleted in favor of concrete technical details.\n\n7. It seems that the criterion for near-human quality should be 0 on the 7-point scale and not -1.0? This will impact the comparability percentages reported in the main paper It would be worth introducing the 7-point scale in the main text.\n\n8. Although Figure 9 provides an example of a paper produced by AI-Researcher, it would be great to have experiments where the papers produced by AI-Researcher are evaluated by human experts. The paper in Figure 9 has a few features that would be unlikely to appear in human-generated research. For example, errorbars are not reported in the experiments and the introduction only appears to cite a single reference.\n\n&nbsp;\n\n**__MINOR POINTS__**\n\n&nbsp;\n\n1. It would be great to have a more descriptive title for the method introduced by the authors.\n\n2. It would be great if in-text and parenthetical citations could be used appropriately e.g. (Wang et al. 2023) in the introduction should be parenthetical since the work does not constitute part of the sentence narrative.\n\n3. Line 105, it is mentioned that the agent selects papers from scientific databases. Which scientific databases?\n\n4. Line 245, it appears that Level 1 and Level 2 tasks are first introduced without definition? They only seem to be defined in the caption of Figure 5 as adaptation and innovation tasks respectively.\n\n5. Figure 4, it would be worth specifying in the Figure caption that the correctness score is defined on a 1-5 scale as readers may misinterpret it as a percentage.\n\n6. Which Claude models are used? Where is the citation to the relevant system card? Was Sonnet or Haiku used?\n\n7. In Figure 5, the average scores across tasks depicted by the dotted horizontal lines are difficult to see.\n\n8. I would phrase Section 4.2 as a \"human performance comparison\" rather than a \"pairwise comparison\".\n\n9. In Section 4.2 there doesn't appear to be a description of the rating system used to score AI-Researcher's outputs. Why are the scores negative? How was comparable quality to human-authored research papers determined? Update: These details are provided in the appendix but were not apparent in the main paper.\n\n10. I believe the ablation study in Section 4.4 would be more correctly termed a sensitivity analysis since the authors are assessing the system's sensitivity to the choice of backbone LLM rather than adding or deleting a system component.\n\n11. There are missing capitalizations in the references e.g. \"ai\" in place of \"AI\", \"llm\" in place of \"LLM\".\n\n12. There are some missing journal/conference references e.g. SciBench [4] was published at ICML 2024.\n\n13. Line 1760, typo, \"groundtruth\".\n\n14. For the Guided Innovation task it is not clear how the reference papers are selected from the main paper? In the appendix the procedure is clarified.\n\n15. The number of random trials (16 for LLM-as-a-judge) and errorbars should be provided in the results of the main paper.\n\n16. In terms of multi-agent collaboration frameworks it would be worth mentioning [6] where the authors develop a framework for optimizing a multiagent graph with binary edge weights.\n\n17. In terms of agentic task execution systems it would be worth mentioning the related works of [4,5] which also leverage language agents for challenging scientific tasks.\n\n18. Although concurrent to the NeurIPS submission deadline, it may be worth the authors referencing Robin [1] as related work in this area.\n\n19. In Listing 16, there is some bias in the prompt of the experiment analysis agent, namely it is instructed to design experiments to show the superiority of the idea rather than to dispassionately examine the idea.\n\n20. Typo line 1223, \"Listing\" in place of \"List\"?\n\n21. Missing full stop at the end of Equation 1.\n\n22. It would be worth citing the system cards of the backbone LLMs use, e.g. [7] for o1.\n\n&nbsp;\n\n**__REFERENCES__**\n\n&nbsp;\n\n[1] Ghareeb, A.E., Chang, B., Mitchener, L., Yiu, A., Szostkiewicz, C.J., Laurent, J.M., Razzak, M.T., White, A.D., Hinks, M.M. and Rodriques, S.G., 2025. [Robin: A multi-agent system for automating scientific discovery](https://arxiv.org/abs/2505.13400). arXiv preprint arXiv:2505.13400.\n\n[2] Skarlinski, M.D., Cox, S., Laurent, J.M., Braza, J.D., Hinks, M., Hammerling, M.J., Ponnapati, M., Rodriques, S.G. and White, A.D., 2024. [Language agents achieve superhuman synthesis of scientific knowledge](https://arxiv.org/abs/2409.13740). arXiv preprint arXiv:2409.13740.\n\n[3] Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A.R., Zhang, S., Sun, Y. and Wang, W., 2024, July. [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://proceedings.mlr.press/v235/wang24z.html). In International Conference on Machine Learning (pp. 50622-50649). PMLR.\n\n[4] Kon, P.T.J., Liu, J., Ding, Q., Qiu, Y., Yang, Z., Huang, Y., Srinivasa, J., Lee, M., Chowdhury, M. and Chen, A., 2025. [Curie: Toward rigorous and automated scientific experimentation with AI agents](https://arxiv.org/abs/2502.16069). arXiv preprint arXiv:2502.16069.\n\n[5] Narayanan, S., Braza, J.D., Griffiths, R.R., Ponnapati, M., Bou, A., Laurent, J., Kabeli, O., Wellawatte, G., Cox, S., Rodriques, S.G. and White, A.D., 2024. [Aviary: training language agents on challenging scientific tasks](https://arxiv.org/abs/2412.21154). arXiv preprint arXiv:2412.21154.\n\n[6] Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D. and Schmidhuber, J., 2024, July. [GPTSwarm: Language agents as optimizable graphs](https://openreview.net/forum?id=uTC9AFXIhg). In Forty-first International Conference on Machine Learning.\n\n[7] Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A. and Iftimie, A., 2024. [OpenAI o1 system card](https://arxiv.org/abs/2412.16720). arXiv preprint arXiv:2412.16720.\n\n&nbsp;", "was_revised": true, "final_justification": "&nbsp;\n\nBased on the changes made by the authors during the rebuttal phase, I have upgraded my score from 3 to 5.\n\n&nbsp;", "rebuttal": "# Response to Reviewer S8cY\n\nWe sincerely appreciate the thorough feedback and valuable suggestions you have provided! We have carefully considered your input and made the necessary updates and modifications accordingly. We genuinely hope that our responses have addressed your concerns and that you would consider revising your rating score based on the changes we have made.\n\n## Github Link Issue\n\nWe sincerely apologize for the inadvertent error with the anonymous GitHub link provided in our submission.We have now corrected this issue and ensured that the link directs to the appropriate codebase. We respectfully ask that you please review the updated repository link.\n\n## Unclear Description of Scientist-Bench Dataset\n\nWe apologize for the unclear presentation of Scientist-Bench in the main text. We will move key details from Section A.7 into the main paper, including: (1) the systematic paper selection methodology across 16 AI domains, (2) a brief summary of the five-step reference extraction process identifying 15-20 influential papers per target, (3) the anonymization protocol preventing memorization shortcuts, and (4) the two-stage evaluation framework with multiple LLM evaluators. This will replace the brief description with comprehensive methodology explaining how Scientist-Bench enables standardized assessment across Level-1 (guided) and Level-2 (open-ended) tasks.\n\n## Title Ambiguity\n\nThank you for this excellent question. First, our system is indeed capable of serving as a generalist agent for scientific discovery. This is because research across different scientific domains follows a common iterative cycle of literature review, idea generation, planning, and experimentation. More specifically, our current system can handle computational tasks in other biomedical fields, such as data analysis in biomedical research, though it does not include hands-on experimental tasks. As an initial study, we chose the AI domain for validation because of our greater familiarity with this field. Moving forward, we plan to extend Scientist-Bench to other scientific domains. However, we apologize for the potential ambiguity in the title and plan to rename it to \"AI-Researcher: Autonomous Scientific Innovation for AI Systems and Beyond.\" We hope this revision will eliminate such ambiguity.\n\n## Insufficient Technical Details\n\nThank you for pointing out this clarity issue. The \"Challenges,\" along with \"Existing Methods,\" \"Motivation,\" and other items listed, are the required components that the Idea Generator must include in the final idea proposal. We acknowledge that the presentation format may have caused this misunderstanding, and we have revised the display format of these items for better clarity. Additionally, we gratefully accept your suggestion to move more technical details to the main paper. We have incorporated more experimental data and condensed prompts into the main text to enhance the technical grounding of our work.\n\n## Missing experimental description\n\nWe appreciate the reviewer's excellent suggestion to improve the clarity of our evaluation approach. We will add the following description to the introduction after presenting our contributions: \"We evaluate AI-Researcher through comprehensive experiments on our Scientist-Bench benchmark, including pairwise comparisons between AI-generated and human-authored papers using multiple LLM judges following ICLR review guidelines, technical implementation validation through specialized code review agents, and performance analysis across both guided innovation tasks and open-ended research exploration scenarios.\"\n\n## Curation Procedure Description\n\nThank you for your valuable suggestion. We agree that clear Experimental Settings can help readers better understand the experimental setup, particularly the content in Appendix A.8. We have incorporated this section into the main text and streamlined the high-level method descriptions to accommodate the inclusion of detailed Experimental Settings and comprehensive technical details (such as agent definitions, tools, etc.). Additionally, regarding your concern about the curation procedure, our benchmark includes a detailed curation process outlined in **A.7.2 Benchmark Construction**, which consists of Step 1: Systematic Selection of AI Research Benchmark Papers, Step 2: Input Construction for AI-Researcher, and Step 3: Rigorous Anonymization to Ensure Scientific Originality. We will also summarize and incorporate this section into the main text to provide clearer experimental settings.\n\n## Scoring criteria issue\n\nWe thank the reviewer for this important clarification. Our -1.0 threshold is a reasonable and conference-aligned setting for several reasons: First, our benchmark comprises exclusively high-quality papers accepted at top-tier venues, making it extremely challenging to clearly exceed these carefully-selected contributions. Second, top conference review processes inherently involve borderline decisions where papers within narrow margins are considered comparable quality—borderline scores often determine acceptance at competitive venues like ICLR and NeurIPS. Following this established academic practice, we consider AI-generated papers scoring ≥-1.0 as achieving near-human quality since they fall within the borderline range of these exceptional benchmarks, where -1.0 represents the lower bound of papers that could potentially receive acceptance at top conferences.\n\nWe will add a clear description of the 7-point scale and our threshold rationale to the main text to improve clarity.\n\n## Human Expert Evaluation of AI-Generated Papers\n\nThank you for this insightful observation about Figure 9 and the suggestion for human expert evaluation. We acknowledge the limitations you identified in the example paper, such as missing error bars in experiments and limited citations in the introduction—features that indeed differ from typical human-generated research.\n\nTo address this concern and further validate our evaluation framework, we conducted an additional comprehensive human expert evaluation. Given that each expert needed to carefully review both AI-generated and human-written papers in detail, this process was extremely time-consuming. We recruited 7 domain experts who each performed full-text reviews of 7 randomly selected paper pairs (AI-generated vs. human-written), following identical ICLR review guidelines used by our AI evaluators. All papers were fully anonymized and randomly ordered to eliminate any human bias in the evaluation process. This rigorous human evaluation yielded an average score of **-0.816**, which closely aligns with our LLM review agents' decisions, providing strong additional confidence in our evaluation framework's reliability.\n\nThis human expert validation not only supports our automated evaluation methodology but also confirms that while AI-generated papers may exhibit certain stylistic differences (as noted in Figure 9), the overall quality assessment remains consistent between human experts and our AI evaluation system. We will incorporate discussion of these stylistic limitations and our human validation results into the main text.\n\n## Technical Clarifications\n\nWe have addressed the following technical detail concerns:\n\n• Point 3: We have clarified that the scientific databases mentioned on Line 105 include **arXiv****,** **Google Scholar****,** **GitHub****, HuggingFace**, and other general scientific databases used by the Knowledge Acquisition Agent for systematic literature exploration.\n\n• Point 4, 14: We have moved the definitions of Level 1 and Level 2 tasks from **Appendix A.8** to the main text. Level-1 Task (Guided Innovation) involves the agent receiving explicit research instructions alongside reference papers to develop targeted innovations, covering all 22 groundtruth papers. Level-2 Task (Autonomous Exploration) requires the agent to perform independent research exploration with only reference papers as input, testing its capacity to identify research gaps and generate novel directions without explicit guidance. We strategically selected 5 groundtruth papers across distinct research domains for this more challenging scenario.\n\nThese clarifications provide readers with the necessary technical context directly in the main paper.\n\n## Autonomous Literature Search Capability\n\nWe greatly appreciate your valuable suggestions regarding autonomous literature search capabilities in **Question 2 and Limitations**. Initially, we chose not to implement automatic agent-driven search in order to standardize the evaluation process for Scientist-Bench. However, inspired by excellent work such as [2] and others, we have internally developed autonomous research search agents as part of our system optimization. We will add the technical details and citations for this component. Specifically, we have provided the agent with all the workflow tools originally used for dataset construction and designed prompts to enable the agent to autonomously complete systematic literature reviews. This enhancement addresses the limitation you identified and moves toward a more fully autonomous scientific research system.\n\nReference:\n\n[2] Skarlinski, M.D., Cox, S., Laurent, J.M., Braza, J.D., Hinks, M., Hammerling, M.J., Ponnapati, M., Rodriques, S.G. and White, A.D., 2024. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409.13740."}, {"review_id": "gdNAwjwfDA", "reviewer": "Reviewer_qqqG", "summary": "This paper introduces AI-Researcher, a fully autonomous research system capable of executing the complete scientific pipeline,from literature review and hypothesis generation to algorithm implementation and automated scientific writing, with minimal human intervention. The system is built on a multi-agent architecture and rigorously evaluated using a newly introduced benchmark, Scientist-Bench, which tests both guided and open-ended research capabilities. Results show high implementation success rates and surprisingly strong performance in open-ended innovation, approaching human-level quality in some cases.", "questions": "1. How do the authors plan to bridge the remaining quality gap between AI-generated and human-written papers? Could multi-pass refinement or retrieval augmentation help?\n2. The results show inconsistent assessments from GPT-4o, Claude, etc. Would involving human domain experts for a subset of evaluations strengthen the paper?\n3. How does AI-Researcher compare to recent autonomous frameworks (e.g., MetaGPT, CAMEL, AutoAgent) in terms of generality, innovation capacity, or end-to-end automation?\n4. For Level-2 tasks with low correctness scores, can the authors provide concrete examples or failure modes?\n5. Could the framework be extended to real-world interdisciplinary domains (e.g., bioinformatics, climate science)? What adaptations would be needed?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n• This is one of the first works to demonstrate full-stack autonomous scientific research using LLMs, from idea generation to paper writing.\n\n• The introduction of Scientist-Bench, covering both guided and open-ended research tasks, is valuable for evaluating autonomous research agents.\n\n• Thorough experiments (22 benchmark tasks across AI domains) show high implementation success rates (up to 93.8%) and moderate correctness, especially in open-ended tasks.\n\nWeaknesses:\n\n• There is notable variability across LLM-based evaluators (e.g., GPT-4o vs. Claude-3.7), highlighting the fragility of using LLMs alone to assess research quality.\n\n• Performance in compute-heavy domains (e.g., diffusion models) is capped by available resources rather than theoretical limitations.\n\n• While the system is compared to baselines in execution metrics, comparisons to other autonomous research frameworks (e.g., CAMEL, MetaGPT, AutoAgent) are missing.", "was_revised": true, "final_justification": "Overall, such a benchmark is highly valuable and timely for the research community and society at large. I encourage the authors to incorporate the points addressed in their rebuttal into the camera-ready version, and, ideally, to continue maintaining and openly releasing the benchmark in the future.", "rebuttal": ""}, {"review_id": "LmtOpHBw2R", "reviewer": "Reviewer_6z2Z", "summary": "The authors build a prompt-only, multi-agent LLM baseline that -- given a capsule of 15-20 reference papers (and optionally an instruction\n-- (i) writes runnable code and \n-- (ii) drafts a full LaTeX manuscript.\nThey wrap this in Scientist-Bench, a benchmark that evaluates such pipelines through two sequential gates:\n\nImplementation gate -- Did the code compile & run? Was the output logically consistent (1-to-5 “Correctness” from an LLM judge)?\n\nWriting gate -- A five-model LLM ensemble conducts pair-wise reviews between the AI paper and the human target on a 7-point scale.\n\nThis evaluation pipeline -- more than the agent design -- is the core contribution.", "questions": "1) Necessity of each agent – Provide leave-one-out ablations (Idea-Generator off, Advisor off, etc.).\n\n2) Judge robustness – Insert adversarially wrong papers (e.g., swapped labels, impossible numbers). How often do LLM judges still pass them?\n\n3) Iteration budget – Report average prompt tokens, wall-clock time, and compute for each Advisor refinement loop.\n\n4) Human audit – Even a 10-paper, two-expert pilot study would calibrate LLM scores.\n\n5) Other agentic systems: CycleResearcher, AgentRxiv, AutoGPT-Researcher, is it difficult to test them in the same scenario?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n**Evaluation design**: \tClever two-stage protocol; pair-wise scoring limits global optimism; ICLR alignment study gives the LLM-judge external anchor.\n\n**Reproducibility:**\tFull prompts + Docker recipes; anyone can swap the backbone or prompts and immediately record benchmark scores.\n\n**Baseline:**\tProvides a clear, minimal reference system on which future, more sophisticated agent frameworks can iterate.\n\nWeaknesses: \n\n1) The study uses prompt-only agents, so we can argue that the baseline is too trivial.\n\n2) No new reasoning or learning. Please show ablations: if you drop the Resource-Analyst or Idea-Generator, does quality actually fall?\n\n3) No comparison to other methods: CycleResearcher, AgentRxiv, AutoGPT-Researcher are cited but not re-run. Hard to tell if anything improved.\n\n4) No human user study, real scientists never inspected the AI papers; LLM judges may hallucinate quality.\n\n5) Need-for-code vs text-only ablation. If you skip the code stage and just write the paper, how much does the LLM score drop? Why force code execution?\n\n6) Prompt sensitivity:\nSmall prompt edits can swing results. Show variance across prompt seeds and backbones.\n\n7) Domain coverage\nAll 22 tasks are AI-software papers; nothing from empirical sciences, so “scientific innovation” claims are limited.\n\n8) Title oversells \"innovation\". The pipeline reorganises existing knowledge; evidence of new discoveries is absent. Consider renaming to something like “Scientist-Bench: A Baseline and Benchmark for End-to-End LLM Research Pipelines.”", "was_revised": true, "final_justification": "I have decided to raise the score as the authors provided additional experiments. I hope they will be included in the final version (in case of acceptance).", "rebuttal": ""}, {"review_id": "lCqMnASVE5", "reviewer": "Reviewer_875E", "summary": "This paper proposes an autonomous research system, named AI-Researcher, with seamless research pipeline: literature review, hypothesis generation, algorithm implementation and publication-ready manuscript. To evaluate the performance of the framework, the paper also introduces a benchmark, Scientist-Bench, comprising state-of-the-art papers across diverse AI research domains.", "questions": "Additional Feedback\n\n1.\tTo improve clarity, the figure should explicitly depict the internal components of each agent, along with the flow of information between them. Furthermore, vague terms such as “taking notes” lack definition—please clarify their meaning and role within the research pipeline.\n2.\tCertain elements in the paper lack proper citation. For instance, Table 1 is not cited or discussed in the main text, which reduces clarity and coherence.\n3.\tThe writing should be improved more to enhance readability.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths and Contributions\n\n1.\tThe topic of autonomous research is meaningful and interesting. \n2.\tThe paper also introduces a new benchmarking to evaluate the performance of autonomous AI research. \n\nLimitations / Weaknesses\n\n1.\tSome important discussions are missing. \nAutonomous scientific discovery is not a new topic, multiple existing methods (with whole research pipeline) are available, as illustrated in your appendix A.11.2. Discussion and highlighting the novelty of this paper with existing methods are necessary. \n\n2.\tMissing comparisons with existing AI scientist frameworks.\nThe paper does not include direct comparisons with other AI scientist systems (e.g., [1], [2]), which limits the ability to assess relative performance and novelty. \n\n3.\tMissing baseline comparisons with other foundation models.\nWhile the paper reports performance differences between GPT and Claude series, it lacks a broader baseline comparison. For a more comprehensive evaluation, results from additional foundation models—such as Gemini, or DeepSeek—should be included to validate the generality and robustness of the proposed framework across architectures.\n\n4.\tMissing experimental details and unclear evaluation criteria.\nIn line 304, the paper states that it evaluates aspects such as research motivation, methodology, innovation, and experimental validation. However, it is unclear how these dimensions are assessed—whether through human annotation, LLM-based scoring, or other means. Additionally, the interpretation of the scores presented in the corresponding table is not explained. Please clarify the evaluation protocol and the meaning of the numerical results.\n\n5.\tApplicability to broader scientific domains is unclear.\nThe current work focuses primarily on AI-related tasks. It remains uncertain whether the proposed framework generalizes to other scientific fields such as biomedicine, chemistry, or physics. Please clarify the system’s domain adaptability and any assumptions or limitations that may restrict its applicability.\n\n\n[1]. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024). The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292.\n\n[2]. Weng, Y., Zhu, M., Bao, G., Zhang, H., Wang, J., Zhang, Y., & Yang, L. (2024). Cycleresearcher: Improving automated research via automated review. arXiv preprint arXiv:2411.00816.", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns, and I am inclined to increase my score. However, due to the recent reminder to reviewers, “Any rebuttals that were posted in their entirety as comments after Jul 30 are to be ignored,” I will need to disregard your earlier comments. Therefore, I am keeping my original borderline accept rating. The authors may check with the AC to confirm whether the score can be increased in this case.", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper presents a multi-agent system designed to automate the full scientific research pipeline—from literature review and hypothesis generation to algorithm implementation and manuscript writing—largely without human intervention. To evaluate such systems, the authors introduce Scientist-Bench, a benchmark based on 22 state-of-the-art AI research papers spanning guided and open-ended tasks. Experiments demonstrate that AI-Researcher achieves high implementation success rates and produces outputs approaching human-level quality, with especially strong performance on open-ended research tasks. The paper positions itself as both a framework for autonomous research and a benchmark for future developments in this domain.

The work is timely and ambitious, addressing the important problem of automating scientific discovery. Reviewers praised the end-to-end design, the systematic evaluation framework, and the strong empirical results, particularly in open-ended tasks. Weaknesses centered on clarity, overstated claims of producing “publication-ready” manuscripts, reliance on LLM-as-a-judge without sufficient human validation, missing comparisons to related frameworks, and limited discussion of ethical risks. In their rebuttal, the authors addressed many of these concerns by clarifying unclear mechanisms, providing ablations and comparisons to other frameworks, introducing a pilot human expert evaluation that confirmed alignment with LLM-based judgments, and reporting cost and runtime analysis. Post-rebuttal, reviewers acknowledged that these additions substantially strengthened the paper, though questions remain regarding the generality of the system beyond AI research and the robustness of LLM-based evaluation under adversarial conditions.

Given its novelty, the thorough improvements made in the rebuttal, and the consensus among reviewers that it is technically solid with high potential impact, this paper should be accepted.
Decision: Accept (spotlight)