Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper
Authors: Atsuyuki Miyai
Mashiro Toyooka
Takashi Otonari
Zaiying Zhao
Kiyoharu Aizawa
Venue: Accepted by TMLR ()

Abstract: AI Scientist systems are autonomous agents capable of conducting scientific research. Understanding their current capabilities and risks is essential for ensuring trustworthy and sustainable AI-driven scientific progress while preserving the integrity of the academic ecosystem. To this end, we develop Jr. AI Scientist, a state-of-the-art autonomous AI scientist system that mimics the core research workflow of a novice student researcher: Given the baseline paper from the human mentor, it analyzes its limitations, formulates novel hypotheses for improvement, validates them through rigorous experimentation, and writes a paper with the results. Unlike previous approaches that assume full automation or operate on small-scale code, Jr. AI Scientist follows a well-defined research workflow and leverages modern coding agents to handle complex, multi-file implementations, leading to scientifically valuable contributions.  Through our experiments, the Jr. AI Scientist successfully generated new research papers that build upon real NeurIPS, IJCV, and ICLR works by proposing and implementing novel algorithms. For evaluation, we conducted automated assessments using AI Reviewers, author-led evaluations, and submissions to Agents4Science, a venue dedicated to AI-driven scientific contributions. The findings demonstrate that Jr. AI Scientist generates papers receiving higher review scores by DeepReviewer than existing fully automated systems. Nevertheless, we identify important limitations from both the author evaluation and the Agents4Science reviews, indicating the potential risks of directly applying current AI Scientist systems and key challenges for future research. Finally, we comprehensively report various risks identified during development. We believe this study clarifies the current role and limitations of AI Scientist systems, offering insights into the areas that still require human expertise and the risks that may emerge as these systems evolve. The generated papers and the authors' annotations on them are included in the supplementary materials. Issues, comments, and questions are all welcome in https://github.com/Agent4Science-UTokyo/Jr.AI-Scientist.

Published in Transactions on Machine Learning Research (02/2026)
Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper
Atsuyuki Miyai* The University of Tokyo
migui@Bcm.ttuutooop...o..
Mashiro Toyooka'
togpoka@@hal.....ookoo.....j
The University of Tokyo
Takashi Otonari The University of Tokyo
otonarr@eom.tt.u.ookpg......
Zaiying Zhao The University of Tokyo
cheo@Bom.t....okkyo......
Kiyoharu Aizawa
aiaawa@hal...-tokgg..ooo
The University of Tokyo Tokyo University of Science
Reviewed on OpenReview: https: opwp:/eiee..........mmmm net/ perm/ id= DeV0628SS
Abstract
AI Scientist systems are autonomous agents capable of conducting scientific rescarch. Un- derstanding their current capabilities and risks is essential for ensuring trustworthy and sustainable AT-driven scientific progress while preserving the integrity of the academic ecosystem. To this end, we develop Jr. Scientist, a state-of-the-art autonomous Al scientist system that mimics the core research workflow of novice student researcher: Given the baseline paper from the human mentor, it analyzes its limitations, formulates novel hypotheses for improvement, validates them through rigorous experimentation, and writes paper with the results. Unlike previous approaches that assume full automation operate on small-scale code, Jr. Al Scientist follows a well-defined research workflow and leverages modern coding agents to handle complex, multi-file implementations, loading to scientifi- cally valuable contributions. Through our experiments, the Jr. AI Scientist successfully generated new research papers that build upon real NeurIPS, IJCV, and ICLR works by proposing and implementing novel algorithms. For evaluation, we conducted automated assessments using AI Reviewers, author-led evaluations, and submissions to Agents4Science,, venue dedicated to AI-driven scientific contributions. The findings demonstrate that Jr. Al Scientist generates papers receiving higher review scores by DeepReviewer than existing fully automated systems. Nevertheless, we identify important limitations from both the author evaluation and the Agents4Science reviews, indicating the potential risks of directly applying current AI Scientist systems and key challenges for future research. Finally, comprehensively report various risks identified during development. We be- lieve this study clarifies the current role and limitations of AI Scientist systems, offering insights into the areas that still require human expertise and the risks that may emerge as these systems evolve. The generated papers and the authors' annotations on them are included in the supplementary materials. Issues, comments, and questions are all welcome in http://hht...oooaaeeo//aaaaaaaaa
AI-Scientist
* Equal contribution.

---

Published in Transactions on Machine Learning Research (02/2026)
Idea Generation
Experimentation
Paper Write-Up
Baseline Resources
PDF
</>
PDF
Code
PDF
Limitation
Stagel Implemeent Proposed Method
Manuscript Creation
Analysis
Idea Generation
Stage2 Improve Proposed Method
L.M Feedback
Review Reflection
Latex
Novelty Check
Stage3 Conduct Ablation Studies
Feedback Reflection
LLM Review
Page Adjustment
Figure 1: Jr. AI Scientist Workflow. We provide the baseline paper, its LaTeX source files, and the associated codebase. By effectively utilizing these resources across all phases, the system significantly improves the quality of the generated paper.
1 Introduction
Understanding the current upper bound of capabilitics AI Scientist systems, autonomous agents capable of conducting scientific research, is crucial for promoling sustainable, AI-driven scientific progress. Nevertheless, developers must remain conscious of the potential risks these systems pose to the academic ecosystem and commit to advancing them responsibly. Since 2025, contributions, the Agents4Science conference (Zou et al., 2025), has emerged. Through such a platform, developers AT Scientist systems are encouraged to engage in responsible research and development, ensuring both the protection of the academic ecosystem and the long-term sustainability scientific progress.
venue dedicated to evaluating Al-driven scientific
In recent years, several works have explored the concept AI Scientists (Lu et al., 2024 Weng et al., 2025a Yamada al.] 2025 Tang et al. 2025). ). However, the quality of research papers produced by these systems remains insufficient. One major reason is that the problem setting of achieving fully automated science is overly ambitious and often lacks clearly defined scientific goals for AI Scientists. Without specific goal, these systems tend to generate undirected discoveries that appear to lack genuine scientific value. Another limitation is that current systems are limited to small-scale code experiments (Zhu et al. 2025b Lu et al., 2024 Yamada et al. 2025), lacking the scale and complexity needed for meaninglul science. Achieving real scientific contributions requires not just ideas but strong implementation capability handle complex codebases.
As an initial step toward enabling AI Scientists to produce genuine scientific value, from how student researchers begin their research. When a student first joins research lab, a common and meaningful process as follows: the mentor assigns a key paper, the student analyzes its limitations, proposes an improvement hypothesis, implements the idea on the baseline code, validates the hypothesis through thorough experiments, and finally writes paper summarizing the results. Through this process, the student learns the fundamental workflow of scientific rescarch and gains the skills and experience needed for more creative work later Also, improving baseline method not only an important stage in early research training but also valuable research goal in many fields where advancing task performance remains central scientific pursuit.
take inspiration
In this paper, we introduce Jr. AI Scientist, a new Al Scientist that mimics the essential research workflow of novice student researcher: Starting from a baseline paper, it identifies key limitations, proposes an improvement hypothesis, validates it through rigorous experimentation, and writes a paper with the results. Specifically, A Jr. AI Scientist is defined as an AI Scientist that is given bascline resources and focuses on extending the baseline. This setting is novice student analog, where student builds upon existing paper
2

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Comparison of the starting point, code complexity, and review scores among existing Al Scientist systems. Previous methods often made overly ambitious assumptions in their problem formulation and were limited to handling only simple, single-file codebases, which resulted in significantly lower review scores. In contrast, our system can substantially improve review scores by utilizing the baseline paper and its associated codebase.
AI Scientist Systems
Starting Point
Code Complexity
Review Score
AI Scientist-vi
Lu et al.
2024
Template code
Single file
3.30
AI Scientist-v2
Yamada el al. 2025
General idea
Single file
2.75
AT Researcher
lang al.
2025
15-20 existing papers
Multiple files
3.25
Jr. AI Scientist (ours)
One baseline paper and code
Multiple files
5.75
which the mentor gave. The workflow of Jr. AI Scientist is shown in Figure 1 We provide the baseline paper, its LaTeX source files, and the associated codebase for each stage. Table D shows a comparison of the problem setting with existing rescarch. This problem sctting reframes previously ambitious goals into a more specific objective, providing a clear optimization direction for the Al Scientist. Moreover, because the framework operates on the actual codebase of baseline papers, it can generate results with genuine scientific value. These aspects collectively represent an essential first step toward the autonomous generation of reliable and high-quality research papers.
Our Jr. AI Scientist consists of three main components: (1) automatic idea generation based on the limitations of a given paper, (2) automatic implementation and thorough validation of the proposed ideas, and (3)
automatic writing of research paper based on the obtained results. This system is built upon AI Scientist
v2 (Yamada et al. )025), but our work differs from prior studies (Lu et al. 2024 Weng et al. 2025a Yamada
et al. 2025 Tang et al. 2025) in several key aspects: First, by leveraging the latest coding agents (e.g. Claude Code (Anthropic 2025)), our system can perform meaningful improvements and edits on real multi-fle codebases, which were challenging for previous AT Scientist systems. Second, by incorporating the full set. of resources from a given baseline paper, the system exploits all available artifacts such as LaTeX sources, PDFs, and codebases, thereby substantially improving the scope and quality of every stage in the research pipeline. Finally, by refining every stage of the process, our framework enables the autonomous generation of research papers that are both higher in quality and more trustworthy.
For the baseline papers, we selected papers for which we obtained permission from the original authors. Specifically, we used three papers: NeurIPS 2023 paper (Miyai et al. 2023) and IJCV 2025 paper (Miyai et al. 2025bb on out-of-distribution (OOD) detection (a task that aims to detect semantic classes outside the predefined set of semantic classes) (Heddrycks & Gimpel 2017 Yang ct al., 2024), and an ICLR 2025 spotlight paper (Zhang el al.) 2025)) on pre-training data delection for large language models (LLMs). Refer to Section [1.]] for the detailed rationale behind the selection of these papers. For the evaluation,
we conducted three evaluations: (1) an automated assessment using DeepReviower (Zhu et al.] 2025a),
(2) an author-led evaluation, and (3) submission to the Agents&Science conference (Zou et al. 2025 DeepReviewer automatically compared our generated papers with existing A AT-generated works to assess overall quality. The author evaluation examined the outputs for hallucinations or fabricated content. Finally, the Agents4Science [Zou et al., 2025) platform provides rigorous evaluation and feedback from the community platform.
Through our experiments, the Jr. Al Scientist successfully generated new research papers that build upon the above top-tier venues works by proposing and implementing novel algorithms. As for the evaluation using DeepReviewer (Zhu et al.) (2025a), the papers generated by Jr. Al Scientist achieved substantially higher review scores compared to the cxisting Al-gencrated papers. Therefore, our Jr. AI Scientist can be regarded as the most capable autonomous AI Scientist. However, we also observed that Jr. AI Scientist still exhibits some failures and unresolved challenges through the author evaluation and Agents4Science conference. To share these challenges and lessons with the research community, we analyze the feodback and evaluation results from the Agents4Science conference and include the author evaluation, which helps clarify what is required to further improve the quality of Jr. AT Scientist systems.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Finally, we perform in-depth report of the risks encountered during the development of our system. Although few existing studies have provided comprehensive discussion of these issues, we believe that accurately documenting such risks is essential to avoid overestimating current Al Scientists' capabilities and to build a clear understanding of their remaining challenges. Our risk report highlights several critical issues, including the potential for review-score hacking and difficulties in ensuring proper citation, interpreting results, and detecting fabricated descriptions. We believe that these findings provide valuable guidance on the potential risks that exist both in the current Al Scientist research and as this field continues to grow. Through this comprehensive report, aim to foster deeper understanding of current AI Scientist systems and contribute to their safe and trustworthy development.
Our contributions are summarized as follows:
Development of a New AI Scientist: We developed Jr. AI Scientist, a new system that starts from a baseline paper and its associated codebase, and is capable of handling complex, multi-file implementations, overcoming major limitation of previous AI Scientist systems.
Revealing Strengths and Limitations of Jr. AI Scientist: We conducted extensive evaluations using open-source AT reviewers, Agents4Science, and author evaluation. The results demonstrate that Jr. AI Scientist generates higher-quality research papers than existing Al Scientists, while also revealing key challenges for future improvement.
Thorough Risk Report: We report the observed risks throughout the project. We believe these reports offer insights into the areas that still require human expert and the risks that may emerge as these systems evolve.
2 Related Work
Automated Scientific Discovery. Recent progress has significantly reshaped the role of AI in automating end-ta-end scientific research. AT Scientist-vl (Lu et al. 2024) was an early milestone, showcasing how advanced language models can autonomously generate research ideas, run experiments, and draft scientific papers. This work was followed by a scries of subsequent studies machine learning fields (Intology 2025
Tang et al., 2025) and diverse scientific disciplines Mallaescusa-Navarry et al. 2025 Mitchener et al. 2025
that further advanced this line of research. However, these approaches often suffer from overly ambitious problem setting that aims to achieve fully automated science and tend to lack clearly defined scientific objectives for AI Scientists. Without specific goals, such systems often produce undirected discoveries that lack gemuine scientific value. To address this issue, our Jr. Al Scientist builds on existing baselines and conducts research within a well-defined research workflow, aiming to generate higher-quality scientific papers. As concurrent work, DeepScientist. (Weng et al., 2025b) also adopts a baseline basedd approach. DeepScientist (Weng et al., (2025)) focuses mainly on experimental performance, formalizing discovery as Bayesian Optimization problem. However, its overall framework design and workflow intogration are outlined at high level, with limited discussion of implementation details. In contrast, our Jr. AI Scientist explicitly articulates each stage of the research process and further aims to contribute to the community by comprehensively reporting the failures and risks encountered throughout scientific exploration.
AI-Assisted Scientific Research. Research specialized for each element of the research process has also been actively explored (Chen al.] (025). For the idea generation phase, Si et al. (2025b) investigates the novelty of the LLM--gnerated ideas, and Si et al. (025a)) investigates the ideation execution gap. For the survey phase, OpenScholar (Asai et al. 2026) have been developed to support literature review. For the experimental phase, AlphaEvolve (Novikov et al.. 2025) leverages large-scale trial-and-error strategies to enhance the performance. For the writing and review phase, CycleRescarcher (Weng ct al., 2025a) provides a learning framework specialized for scientific writing, while DeepReviewer (Zhu et. al. 2025aa locuses on the review process. Recent studies provide a comprehensive overview of automated review, outlining key challenges, proposing a practical review pipeline for real- world implementation, and constructing a largo-scale
dataset to support automated review research (Zhuang et al. 2025 Lin al. [023a b). Beyond these,
rather than pursuing full automation like AT Scientists, AT Co-Scientist. (Gottweis et al. 2025 emphasizes
4

---

Published in Transactions on Machine Learning Research (02/2026)
Stage1
Stage2
Baseline Codebase
Idea implementation
Improvement
Buggy
Baseline: 91%
Worse than baseline
90
Non-Buggy
Non-Bugsy than baseline
Copy Edit with Coding Agent
Debug with Coding Agent
Stage3
Hyperparameter Ablation
Component Ablation
Figure 2: Jr. AI Scientist Workflow for the Experiment Phase. The workflow consists of three stages. Through bug management and performance tracking, our system passes the most promising experimental nodes to the next stage.
collaboration between humans and AI. In this work, instead of focusing on individual parts of the research process, we investigate the entire end-to-end research cycle, aiming to rigorously evaluate both the performance and the limitations.
Failures and Risk Analysis for AI Scientist Systems. There are very few studies that thoroughly analyze report the risks and failure cases of AT Scientist systems. Although Tang al.) (2024) summarizes the risks that AI Scientists may pose, it focuses
dypothesis--asson potential risks, rather than empirically observed risks or failures. While Beel et al.) (2025) provides an in-depth analysis of failure cases in AI Scientist-vl (Lu et al. 2024p, these findings are based on the early Al Scientist, and the analysis was not conducted from the developer's perspective. While Luo et al. (2025) examines four failure modes (benchmark selection, data leakage, metric misuse, and post-hoe bias), their analysis is limited to experimental diagnostics and early AT Scientists without modern coding. Thus, their analysis remains somewhat limited in scope.
Therefore, will provide of our state-of-the-art AI Scientist, in order to deepen the community's understanding of AI Scientists.
comprchensive report the various risks identified during the development
3 Jr. Al Scientist
In this section, we describe the mechanisms behind the three components of the Jr. Al Scientist: idea generation, experimentation, and writing. First, in Section 3.1 we describe the necessary preparation. Next, in Section 3.2 we explain the methods for idea generation. Next, in Section 3..] we discuss how agents can execute and manage experiments. Finally, Section 3.4. we explain the writing process.
3.1 Preparation: Baseline Paper Selection
The preparation stage involves selecting a baseline paper, obtaining its LaTeX source files and PDF, and the basclinc code. This sctup realistic because many recent publications are relcased on arXiv with LaTeX sources, and their implementation code shared on GitHub. While there might be some augment that AI agents should automatically select baseline and reproduce the code, current reprodueibility rates from papers
5

---

within Claude Published in Transactions on Machine Learning Research (02/2026)
are still limited (Siegel et al. 2025 Xiang et al. 2025 Starace et al. 2025 making complete automation premature. Since our goal is to emulate how a. human scientist engages in early-stage research under the guidance of a mentor, we explicitly include this preparatory stage.
When constructing the baseline code, we followed AI Scientist v1 Lu et al. 2024) and made only minor modifications the existing implementation so that the experiments could be executed via baseline,, py and the results could be visualized via plot. py. Defining such an experimental entry point facilitated easier management and reproducibility of the execution process in the experimental section.
3.2 Idea Generation Phase
We provide baseline paper text to an LLM (e.g. 04-mini (OpenAT 2025b)) and prompt it to output the limitations of the work. Based on both the baseline paper and the limitations, the LLM is then guided to propose potential research ideas. Following AI Scientist v2 (Yamada et al. 2025), the system evaluates the originality of proposed ideas through literature review tools such as Semantic Scholar, which review papers citing the bascline work and papers with similar concepts. If conceptually similar idcas are identified, they are refined; of herwise, they are clearly distinguished from prior work. These steps define the preliminary research idea.
3.3 Experiment Phase
The Experiment Phase mainly involves implementing and iterating on the implementations through exper- iments. It is divided into three stages: Stage 1: Idea Implemention, Stage 2: Iterative Improvement, and Stage 3: Ablation Study. The workflow each stage is shown Figure 2 We first describe the general procodure for using the coding agent, followed by an explanation of the implementation at cach stage.
3.3.1 Preliminary: Coding Agent Usage
A powerful coding agent (e.g. Claude Code (Anthropic) 2025)) is employed to translate research ideas into concrete implementations. We provide the coding agent with working directory that contains the bascline implementations (prepared at Section 3.1 and give it detailed instructions through input prompts. The agent is informed of how to use the main scripts-basel ine py. which serves as the experimental entry point, and plot. which visualizes the experimental results. We use claude-sonnet-4-22255 Code (version 1.0.24).
The coding agent allowed read and write any files within the working directory. For efficient directory exploration, it permitted use commands such as is and grep, while commands that may cause side effects (e.g. python or other execution commands) are not allowed. After the agent generates runnable file (e.g. proposeed_eethee py in Stagel), our system mechanically executes the specified command. The coding agent is generally given up to 30 turns to complete cach assigned task.
3.3.2 Stagel: Idea Implemention
The system manages four experimental nodes running in parallel, each responsible for implementing and testing a proposed idea independently. Within each node, the coding agent receives the baseline code and a research idea, and writes a directly executable script named proposed_mette finishes writing the implementation, the system sequentially runs proposed method. py and plot.py. py. If a result file is successfully generated, the codebase is marked Non-Buggy; otherwise, it is labeled visualization image is also produced, it is further marked as Son-Plot-Buugy otherwise, as Plot- Blgg.Buggy. Each iteration of this process, coding and execution, is counted as one trial and is repeated until bug-free implementation is obtained.
py. Once the coding agent.
Buggy If
As shown Figure 2] if any node completes successfully without encountering bugs, its codebase is carried forward to Stage 2. During execution, four experimental nodes are selected. If all currently running nodes fail, the system selects the next nodes, either by initializing new nodes from the baseline code or by debugging previously generated buggy codebases. When debugging buggy codebases, provide the coding agent with

---

Published in Transactions on Machine Learning Research (02/2026)
Coding Agent
Draft Writing
Reflection
Adjustment
Temp /ate_LaeeX/
Collection
Bib
Validation
Citation
Logical Consistency
Reviews
CLAUDE. md
Basel inc_ LaTo/
Method
Related Work
Page-length Adjustment
Basel inne Code/
Writing
Rewriting
Proposed_ /ode/
Formatting
Figures
Exp Summary Files
Structure
Detailed
templ ate. tex
Writing
Writing
Figure 3: Jr. AI Scientist Workflow for the Writing Phase. The Writing process consists of three steps: Draft Writing, Reflection, and Adjustment.
detailed runtime feedback, such standard output and error messages, to guide iterative debugging until the issue is resolved. We set this stage to run for 12 iterations.
3.3.3 Stage2: Iterative Improvement
Stage focuses on iteratively improving the method implemented in Stage 1 until its performance metrics surpass those of the baseline. In each trial, the coding agent first proposes an improvement idea to the experimental code, and then applies the modification based on the improved idea. To avoid overwriting the Stage results, we instruct the coding agent use a new entry file named improved porpoooeddrppeedd______ perpposed_methhd.... py the implementation target. The system then executes this script, followed by plot.py py, to generate results and visualizations in the same manner Stage 1.
As shown in Figure for each trial, the experimental codebase is selected probabilistically from either (1) the Stage 1 implementation or (2) the node containing the best-performing implementation observed so far. Stage 2 ends when a bug-free implementation surpasses the baseline performance. The resulting code is then passod Stage 3. We set this stage to run for 50 iterations.
3.3.4 Stage3: Ablation Study
Stage 3 performs ablation studies on the improved method implemented in Stage 2. In each trial, the system uses LLM to generate ablation study ideas and then employs the coding agent to implement corresponding scripts based on those ideas. To encourage higher-quality ablation ideas, we first instruct the coding agent to produce a textual description of the Stage method, which is then provided to the LLM as context for generating more meaningful ablation idcas. The generated ablation idcas include hyperparameter ablations, which analyze the sensitivity of the method to key hyperparameters, and component-level ablations, which assess the contribution of each component to the overall performance. To avoid overwriting Stage entry files named hyperparam ablation studsstuuupprr and
2's code, instruct the coding agent to component__ awllaailllttautteeeeee py as the implementation target. The iterations are executed until a sufficient number of experimental results are obtained.
3.4 Writing Phase
We primarily used a coding agent (e.g. Clande Code (Anthropic) 2025)) as a writing agent for the writing process. As shown in Figure 3 the Writing Phase consists of three stages Draft Writing, Reflection, and Adjustment. Below, describe the resources provided to the writing agent and the details of each stage.
3.4.1 Preliminary: Resources Provided to Writing Agent
Conference LaTeX Template. We provide the conference LaTeX template to the writing agent. We give the Agents4Science template, and the corresponding directory is set as the working directory where the writing agent operates.
Instruction Markdown File for the Writing Agent. An instruction file in Markdown format is provided to the writing agent. This document. defines the overall structure of the paper, outlining how each section
7

---

Published in Transactions on Machine Learning Research (02/2026)
should be organized, and offers detailed guidelines on the key points and considerations for writing each part of the manuscript.
Baseline LaTeX Files and Code. We provide the writing agent with the baseline LaTeX files and code. These resources are mainly used to explain the baseline method in the Method section.
Stage Proposed Method Code. The writing agent is also given the Stage 2 code (the proposed method). This is primarily referenced in the Method section when explaining the proposed approach.
Experiment Summaries for Each Stage. Following the protocol of Al Scientist v2, we pro- vide the writing agent with summarized JSON files containing key experimental results for each stage (baseline_sumnell www.sruummsaooooo json, improved_reeearee summary. component_ component_ablation Sernnnssuumarrrae summary-json, and hyperparam hatpeearramablaaiiiioaaaaa iellatiionsummaane json). These files include essential information such as experimental de- scriptions, results, and paths for visualization results, which are crucial for the writing agent when writing the experimental results section. In addition, for ablation studies, we not only provide the JSON files but also aulomatically convert them into LaTeX table files (component, tttpeenttabbltioonn ablation, ablation_summary hyperparam_ hatarpprammabaation_eeeee abllaiin_summary_coo tablee. tex). This conversion has significantly reduced numerical transcrip- tion errors in the paper.
tex and
The only materials that humans need to prepare for each paper generation are the LaTeX source files and the baseline codebase of the baseline paper. The CLAUDE.md file and the conference LaTeX template are shared across all experiments. The code and experimental files for the proposed methods are automatically generated during the experimental stage. Using these inputs, the AI agent automatically writes all sections and captions in the paper.
3.4.2 Draft Writing
As shown in Figure 8 the Draft Writing stage follows a multi-step process: it begins with the collection of BibTeX entries, followed by the writing of the Method section, the generation of the paper structure, and finally the full-paper writing. Afterward, the system performs a rewrite of the Related Work section and subsequently validates the correctness of citations. Here, we first explain how we determined the writing order and then describe the detailed procedures for each stage of this workflow.
Rationale of Writing Orders. Following Al Scientist v2 (Yamada et al., (025), we initially generated the entire paper at once, but this resulted in decline in the quality of the Method section. We therefore adopted step-by-step writing process to improve overall consistency and quality. When determining the writing order, we considered it most important for the writing agent to first accurately undersland and describe the proposed method, as this understanding serves as the foundation for correctly writing other sections. Therefore, we instruct the writing agent to focus exclusivoly on accurately writing the Method section. In addition, for the paper structure, we followed the approach of Al-Researcher (Tang el al. 2025 and introduced an intermediate step of summarizing the paper structure, in which the writing agent briefly outlines the content of each section before full-paper generation. These refinements made it possible to produce more consistent and accurate descriptions throughout paper.
Collection of BibTeX Entries. To ensure accurate citation, it is essential to collect complete and correct. set of BibTeX entries. Following Al Scientist v1 (Lu et al., 2024), we use the Semantic Scholar API to retrieve BibTeX records. However, this approach alone often yields an insufficient mumber of references. To address this limitation, we adopt a practical strategy commonly used by human researchers: using the baseline paper's BibTeX file as a starting point. This approach allows the system to gather a sufficient number of references while also can expect correct citation by referring to the baseline's 's LaTeX source. Since the baseline's BibTeX file does not include the entry for the baseline reference set, we explicitly add it to the reference set.
Method Section Writing. For the Method scction, we instruct the writing agent to write both preview of the baseline method and a detailed description of the proposed method. To ensure an accurate descriplion, we refer the writing agent to the LaTeX source of the baseline paper so that it can correctly describe the existing method. We also instruct the writing agent to describe the proposed method based on the Stage implementation code, ensuring that the technical details are precisely reflected in the text. This process yielded more accurate and consistent Method sections.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Related Work Section Rewriting. To clearly define the position and novelty of the generated research, we instruct the writing agent to rewrite the Related Work section after completing the full paper draft. Since Jr. Al Scientist aims to update the baseline paper, the Related Work section of the baseline serves as a valuable summary of the research field and provides useful guidance on writing style and structure. Therefore, we instruct the writing agent to refer to the Related Work section in the baseline's LaTeX file when generating its own version.
Citation Validation. We introduce a citation verification phase at the end of the Draft Writing stage. Since Jr. Al Scientist updates the baseline paper, it correctly reuse many of the original citations from the bascline paper. In this step, the writing agent compares the generated paper with the bascline La file in terms of the quantity and quality of citations, and is instructed to add missing references and remove inappropriate ones to ensure accurate and consistent citation practices.
3.4.3 Reflection
To improve the overall quality of the generated paper, we incorporated multiple reflection processes into our workflow. We repeat these reflections three times.
(1) Feedback Generation and Reflection on Logical Consistency. This process aims to enhance the academic reliability of the generated text by producing specific and actionable feedback. In particular, this process examines several key aspects essential for ensuring the logical soundness of paper, such logical consistency, validity of supporting citations, and alignment between experimental results and textual descriptions, and whether each section contains a sufficient amount of content. We first instruct the writing agent to generate feedback regarding the above aspects, and then use it to revise the draft based on that feedback. This process encourages the generation of more logically coherent and trustworthy papers.
(2) Reflection on Formatting and Presentation. Following AT Scientist. v2 (aamada et al. 2025). we also introduce reflection phase focused on formatting and presentation quality. In this phase, the writing agent generates feedback such as: "Are there any LaTeX syntax errors style violations we can fix? Refer to the chktex output below", "Are there short sections (one two sentences) that could be combined into single paragraph?" This process helps produce final draft that is well-formatted and stylistically consistent.
(3) Feedback Generation and Reflection on Figures. Following AT Scientist v2 (Yamada et al., 2023 ; we perform figure-level reflection in the refinement stages by integrating a Large Multimodal Model (LMM)-based foedback mechanism. This process aims to improve the quality, clarity, and alignment of generated figures, captions, and their corresponding textual interpretations. Specifically, the LMM is used to identify figures that are uninformative or make little contribution to the paper's scientific value, and such figures are either removed moved to the Appendix. This ensures that all figures presented in the main paper contain adequate informational value. To achieve this, we provide the LMM with the paper's abstract, figure captions, and figure images to generate targeted feedback, which is then used to guide the reflection and revision process.
(4) Feedback Generation and Reflection from AI Reviews. Following CycleResearcher (Weng et al., 2025al, we adopt review-based reflection, where the system improves its manuscript. based on reviewer feedback. In this step, the writing agent revises the generated paper according to reviewer comments such as "the Method section is unclear", "important parameter details are missing", or "the writing is overly verbose". For generating such feedback, we employ Al reviewers in Al Scientist v1 (text-only evaluation) and v2 (ovaluation including figures). These AI reviewers use GPT-4o HHusst et al. 2024h and are prompted to evaluate papers in the official NeurIPS review format.
3.4.4 Adjustment: Page-length Adjustment
We also introduced a new design to the page-length adjustment process. In AI Scientist. v2 Yamada et al., 2025), when the generated paper exceeded the predefined page limit, the system attempted to adjust the length within a single LIM call. However, this approach often resulted in over-trimming, causing the paper to become significantly shorter than the target length. To address this issue, our method performs iterative and gradual page-length reduction until the manuscript reaches the target length, thereby improving the
9

---

Published in Transactions on Machine Learning Research (02/2026)
stability of page adjustment. As result, the final papers consistently fell within + 1 page of the specified page limit. We set the page layout to 8 pages.
4 Experiment
4.1 Experimental Setting
Bascline Paper Selection. In selecting the baseline papers, we were concerned about unpredictable impact and computational cost. For the former, we considered the potential socictal impact of accelerating research through AT Scientist systems, which might pose a risk of confusing the research fields. To mitigate such risks, we selected only those for which we obtained explicit permission from the original authors. For the latter, we selected papers that require relatively few GPU hours, ensuring that the experiments can be conducted even in our academic laboratories. Although this limits the ability conduct large-scale experiments, it does not undermine objectives to evaluate the capability and risks of AT Scientist systems during their development.
As a result, we selected three papers: LoCoOp (NeurIPS2023) Miyai et al, 2023 and GL-MCM (IJCV2025) (Miyai et al. 2025b) in the field of out-of-distribution (OOD) detection (a task that aims to detect semantic classes outside the predefined set of semantic classes), and Min-K%+ (ICLR2025 spot- light) (Zhang al., 2025) in the field of pre-training data detection for LLMs. LoCoOp is a few-shot learning method with CLIP (Radford et al. 2021), GL-MCM an inference-only method with CLIP, and Min-K%++ is an inference-only task with LLMs. Both rescarch areas have recently attracted incrcasing attention (Shi et al. 2024 Miyai el al.) 2025a). Accurately evaluating how much current AI ist systems can advance these research fields is crucial for deepening the understanding of their capabilities and limitations.
Cost. Claude Code the most resource-intensive agent in our pipeline. However, by using the Claude Code Max plan (USD 200 per month), it possible run two experiments in parallel within the usage limit, enabling paper generation without incurring substantial cost.
Human Involvement. In framework, humans were involved only in verifying the outputs. Publicly available papers are often curated and therefore may not accurately represent. the typical quality of each system's outputs. Therefore, for evaluation, following this common practice, we selected the papers that appeared to be of the highest quality among six generated ones. As the selection criterion, one of the authors who is highly familiar with the baseline paper carefully examined the content the generated papers, the obtained results, and the code, and selected those that were considered to be of high quality. We include three generated papers in the supplementary material.
4.2 Results with Public Al Reviewers
Comparison Methods. As comparison methods, we used papers generated by existing AI Scientist systems. Specifically, we included AT Scientist-vl [Tu et al. 2024). AT Scientist-v2 (Yamada et al. 2025
Al-Researcher (Tang et al. 2025), CycleResearcher [Weng et al. 2025a and Zochi [Intology 2025).
Evaluation Metrics. For evaluation, we employed DeepReviewer (Zu et al. 2025a an AI model designed to comprehensively assess rescarch papers in manner similar to expert reviewers. DeepReviewer is a 14B-parameter language model built by fine-tuning Phi-4 on the DeepReview-13K dataset, which contains structured, human-like review reasoning trajectories. During evaluation, DeepReviewer focuses technical soundness, experimental validity, and logical consistency, grounding its judgments in explicit evidence from the manuscript. This design enables efficient and reliable evaluation while maintaining strong alignment with human reviewer judgments. This adopts standardized scoring, where the overall rating score is from to 10, and soundness, presentation, and contribution are from 1 to 4. Compared to existing Al reviewers (Weng ct al. 2025a Lu et al., 2024: Yamada et al., (025), DecpReviewer exhibits substantially stronger alignment with human reviewer evaluations, enabling efficient and reliable assessment of Al-generated paper reviews. For the implementation, we utilize single 1100 80G GPU.
10

---

Published in Transactions on Machine Learning Research (02/2026)
thraitt Bakg
Figure 4: An example of a generated paper. Our Jr. Al Scientist can generate full-length research papers with appendices.
We evaluated papers generated by our system (an extension paper of LoCoOp is shown in Figure All generated papers, along with detailed author comments, are provided in the supplementary material. We present our experimental results in Table 2 Here, among the three papers generated, the LoCoOp extension paper received a score of 6, the GL-MCM extension paper received a score of 5, and the Min-K%++ extension paper received a score of 6.25. The table shows the average scores across multiple papers (overall score), the score the paper with the highest rating, and that of the paper with the lowest rating. From these results, we observe that our papers outperform the publicly available AT Scientist papers in all criteria (Soundness, Presentation, Contribution, and Rating).
Results.
Score Distribution among Generated Papers. Figure shows the score distribution among all our generated papers before the author selection. First, the average score over all 18 papers is 5.30. Although this is lower than the score of 5.75 reported in Table 2 it can still be considered higher than those of existing Al-generated papers. Second, while the authors' selection criteria show a similar Lendency to the review results produced by DeepReviewer, the selected papers are not necessarily those with the highest review scores. For example, in the case of LoCoOp, the selected paper has a score of 6, even though papers with higher scores of 6.25 and 6.50 exist. Upon manually reviewing these higher-scoring papers, we found that they contain more hallucinations in terms of numerical values and claims (Detailed hallucinations and risks
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Evaluation of Al-generated papers produced by various Scientist systems. Scores represent the
ratings given by DeepReviewer-145 (Zhu et al. 2025a) across public papers.
(a) Overall Score
AI Scientist Systems
Number
Soundness
Presentation
Contribution
Rating
AI SCIENTIST-V1
10
2.03
2.05
1.83
3.30
AI Researcher
1.86
1.79
1.79
3.25
AI SCIENTIST-V2
1.67
1.50
1.58
2.75
CycleResearcher-128
2.25
2.25
2.04
3.92
Zochi
2
2.50
2.75
2.38
4.50
Jr. Al Scientist (Ours)
3
2.75
2.75
2.75
5.75
(b) Score for the Max Rating Paper
AI Scientist Systems
Number
Soundness
Presentation
Contribution
Rating
AI SCIENTIST-V1
2.25
2.50
2.25
4.25
AI Researcher
2.25
2.25
2.00
4.25
AI SCIENTIST-V2
1.75
1.75
1.75
3.25
CycleRescaacherr-222
6
2.75
2.75
2.75
5.00
Zochi
2.50
3.00
2.50
5.00
Jr. AI Scientist (Ours)
3.00
3.00
3.00
6.25
(c) Score for the Minimum Rating Paper
AI Scientist Systems
Number
Soundness
Presentation
Contribution
Rating
AI SCIENTIST-VI
10
1.75
1.25
1.75
2.00
AI Researcher
1.25
1.00
1.25
2.50
AI SCIENTIST-V2
1.50
1.50
1.50
2.50
CycleResearcher-1222
2.00
2.50
1.50
3.00
Zochi
2.50
2.50
2.25
4.00
Jr. AI Scientist. (Ours)
2.50
2.50
2.50
5.00
#aaper
Paaper
2
#Paper
2.5
7.0
6.0
6.5
7.0
30
Score
Score
Score
(a) LoCoOp
(b) GL-MCM
(c) Min-K%
Figure 5: A score distribution among all our generated papers before the author selection.
are discussed in Section and Section 7 ). These findings indicate that evaluation based solely on review scores is insufficient, and highlight the importance of author inspection and honest reporting of hallucinations and associated risks.
5
Agents4Science Conference Submission
5.1 Overview of Agents4Science.
Agents4Science Zou al. 2025) is a conference jointly organized by Stanford University and Together AI, where AT systems both the primary authors and reviewers of research papers. The first edition of the
12

---

Published in Transactions on Machine Learning Research (02/2026)
conference was held in 2025. It is the first venue in which AI authorship is not only allowed but required, enabling open evaluation of AT-generated research and the development of guidelines for responsible AT participation in science. This conference targets a wide range of Al-driven contributions, including papers authored by AI Scientists, as well as those that allow human involvement. The conference provides an ideal platform for evaluating our work, so we submitted our paper to this venue to receive leedback from AT reviewers.
5.2 Al Reviewer in Agents4Science.
The Al reviewers used in Agents4Science are based on GPT-5 (OpenAl 2025a), Gemini 2.5 (DeepMind 2025), and Claude Somnet 4 (Anthropic, 2025). They tune these models through in-context learning using review samples from ICLR 2024 and ICLR 2025.
5.3 Review Results.
We submitted papers generated by the earlier version of AI Scientist. Although these are identical to the newer papers in this study, the reviews discussed here mainly apply to the newer papers well. We summarize below the representative comments from the submitted reviews In terms of strengths, the reviewers generally noted that the work is technically sound, includes comprehensive ablation studies, and is clearly presented. As for the weaknesses, we identified four key issues that we consider particularly important, as summarized below.
Weakness1
Limited Improvement over Baselines.
This comment is valid. While Jr. Al Scientist achieved higher scores than the baseline, the performance gap is not significant enough to claim a substantial improvement. To address this limitation, it would be necessary to increase the number of experimental trials and explore more innovative search strategies for selecting experimental nodes.
Weakness2
Moderate Novelty and Incremental Contribution.
This observation also reasonable. Since Scientist is designed to build upon a given baseline, a certain degree of incremental progress is inevitable. Achieving more innovative ideas would likely require human intervention during the idea generation phase.
Weakness3
Insufficient Experiments. No Comparison with Other Methods.
We agree that comparisons limited to the baseline are not sufficient. However, expanding the comparative methods would require appropriate selection of comparison methods and accurate reproduction of them, which remain beyond the current. level of autonomous AT Scientists. Therefore, human intervention would also be necessary in this part.
Weakness4:
Shallow Theoretical Justification.
Detailed reviews available the following URLs:
LoCoOp: httpss//openreviev..ccm wttp://www...ee///o///////////o.
GL-MOM: httt::///pparreeeeeeee//rrrrrrre
13

---

Published in Transactions on Machine Learning Research (02/2026)
This comment is fair. Jr. AI Scientist follows experimental, performance-driven design that repeatedly edits and improves code until it surpasses the baseline. Therefore, it does not include mechanism to theoretically validate why a particular modification works. As result, some successful solutions may have been discovered only by chance, and their effectiveness might not generalize to other datasets.
For these reasons, our submission was rejected from the Agents4Science conference. However, we would like emphasize that most of the accepted papers at this venue involved human intervention, the rejection does not necessarily indicate that the capability of our AI Scientist is low. The feedback received clearly highlights the current limitations, and we believe these points will serve as important directions for the future development of AI Scientists.
6 Authors Evaluation
We conducted an internal review of the generated papers with authors. Recent Al-generated papers include some degree of manual post-editing (Intology 2025; Weng el al.) 2025b), and only a few studies have carefully examined the raw, unedited outputs of systems (Yamada et al. 2025 However, evaluating the raw, uncdited outputs is essential for accurately understanding the current limitations of AI Scientist systems.
The review here does not evaluate whether the paper has the level of contribution, impact, experimental results required for acceptance at conference. Instead, review focuses whether the writing contains misinterpretations of results, incorrect methodological descriptions, inaccurate citations, or hallucinations. Therefore, we cross-check the manuscripts against the actual code and experimental results to precisely identify such issues. The issues of our review are summarized as follows. More detailed reviews for each paper are provided in the supplementary material.
Positive aspects are that none of these papers contained citations non-existent works developed invalid methods, such leak-in methods. The issues in these papers follows:
Issuel
Frequent
Irrelevant Citations.
We found that there are some irrelevant citations in these papers. This issue arises when adding new BibTeX entries that are not included in the baseline papers. (The reason for this is discussed detail Writing Risk 2 of Section
Issue2
Ambiguous Method Descriptions.
We found that while the method descriptions are generally accurate, they still contain ambiguities. For example, in the LoCoOp extended paper, the parameter appearing around Line 156 is not clearly explained, making it difficult to fully understand the method. Similarly, Min-K%++ extended paper (Lines (26-129). although the corresponding code exists, the process implemented as an optional component and is not actually utilized. This occurs because the coding agent makes numerous modifications during Stage 2 in the Experimental Phase, increasing code complexity. This suggests that accurately transferring experimental code into faithful methodological description remains an open challenge.
Issne3
Misinterpretation of Figure Results.
These papers include the overinterpretation of the figure results, making unsupported claims that appear plausible. For instance, in Min-K%++ extended paper (L177) and LoCoOp extended paper (L160-162), they report findings not evident from the figures. This highlights that precise result interpretation remains difficult. for current Al Scientist systems.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Issued
Descriptions of Auxiliary Experiments That Were Never Conducted.
We found several cases where these papers describe auxiliary experiments that were never actually conducted, such in LoCoOp extended paper (Lines 183 184) and GL-MCM extended paper (Lines 208 213). This issuc occurred even though the writing agent explicitly instructed not include nonexistent experimental results. This problem is especially tricky because hallucinations do not appear in the main results, which are easy to notice, but they often appear in parts like ablation or analysis. Therefore, even human rev reviewers might not notice them unless they carefully check the draft. Such cases illustrate that the risk of hallucination remains inherent in the current system.
7 Observed Risks During the Project
In this section, we describe the risks identified during this project. In the previous section (Section G mainly identified issues related to the papers released in this study. In this section, present various risks encountered during the development process. Sharing such risks is essential to prevent overreliance on these systems and to promote deeper understanding of AT Scientists within the research community.
7.1
Idea Generation
Idea Risk1:
Identifying a successful idea is highly computationally expensive.
The ideas generated by AI do not always work, which holds true for human scientists. In our case, we aimed to generate one successful idea for each baseline paper. To this end, we generated approximately ten ideas and evaluated them. Some were filtered out through human review, while others did not outperform the baseline. Finally, only one idea proved be successful.
From this perspective, more extensive validation conducted in some recent works Liu al. 2025 Weng et al. 2025b). For example, the concurrent work DeepScientist (Weng et al. 2025b) performed a comprehensive large-scale study. They report that, out of approximately 5,000 unique scientific ideas generated, only 21 ultimately led to genuine scientific innovations. Our experiments require less time because our limitation analysis is effective, and our goal modest, aiming to find one successful idea rather than exploring more successful ideas.
Validating large-scale ideas highly computationally expensive and often infeasible for many academic laboratories. Future research will therefore focus developing more efficient idea theaiiguning mechanisms, efficient tree search algorithm for experiments, or incorporating human feedback.
7.2 Experiment
Experiment Risk1
Lacking domain expertise, the coding agent sometimes produces code that leads incorrect imple- mentations and false performance gains.
Because the coding agent is unaware of domain-specific conventions, it often improves performance in undesirable or invalid ways. This issue frequently appeared in the experiments on GL-MCM (Miyai et al. 2025b), which we describe in detail below.
Background. GL-MCM
task of zero-shot out-of-distribution (OOD) detection Miyai et al. 2025a
OOD detection aims distinguish between samples belonging predefined in-distribution (ID) class set (e.g., the 1000 classes of ImageNet) and those belonging to classes with different semantics YYng al. 2024
15

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm GL-MCM implementation (Python-like) psendocode).
Require: TD dataloader DrD; OOD dataloader Doon, method f, scoring function S(.)
Ensure: AUROC value scores
labels =
3: for batch in DID do
4: ood score = S[
5: scores. scores.appendlood_ score)
(abels.append(0)
0 ID sample
end for
for batch in DooD do
ood_score = ((f(batc))
10: scores.a scores.append(ood_sooo
11:
labels.append(1)
= OOD sample
12: end for
13: auroc AUROC(scores, labels)
14: return auroc
In the GL-MCM setting, the model uses CLIP RRadford et al. 2021 and is required to discriminate between ID and OOD data without any training, given only the ID class names.
a convention this research area, the source code typically written as shown in Algorithm Specifically, the ID and OOD dataloaders are definod separately. batch is first sampled from the ID dataloader to obtain OOD score, followed by another batch from the OOD dataloader compute its OOD score. Finally, the OOD scores and the corresponding ID/OOD labels used to compute the AUROC.
Mistake by Jr. AI Scientist. Our Jr. AI Scientist wrote code that applied bach-level normalization and statistical operations within the method for each batch. However, as shown in Algorithm ! each batch contains only ID or OOD samples, not both. As a result, the batch-level statistics are biased toward either the ID or OOD distribution. Human experts can immediately recognize that normalization should not be performed on a per-batch basis. Nevertheless, during numerous attempts to improve performance, the Jr. AI Scientist often arrived such invalid solutions. We believe this issue will persist even as the performance of coding agents contimues improve. This obscrvation highlights the importance of human rescarchers possessing sufficient domain expertise to verify whether the observed performance improvements are indeed valid.
7.3 Writing
Writing Riskl
When foodback is provided, fabrication of experimental results can casily occur.
We found that feedback can sometimes become a major source of fabrication. For example, when the AI Reviewer commented that "validation through thorough ablation studies insufficient", the writing agent often responded by fabricating non-cxistent ablation studics in the subsequent revision, which unfortunately led to an improvement in the review score. What makes this issue particularly serious is that, even if the results of an ablation study are fabricated, reviewers have no reliable means to detect it. In practice, the human author would have to mamuall examine all the actual experiment result files to determine whether the reported results are true or not.
To address this issue, we experimented with two approaches: (i) Adding an explicit instruction to the writing agent such as "If a feedback requests new experiment, a comparison with data you do not have, or an analysis that impossible with the provided information, DO NOT INVENT DATA OR RESULTS." explicitly prohibit fabrication falsification. (ii) Providing the writing agent with experimental results
16

---

Published in Transactions on Machine Learning Research (02/2026)
in structured summary format that was both easy to parse and contained detailed descriptions of each setting and its corresponding outcomes. The second approach proved particularly important. Evenn when the writing agent was explicitly instructed not to fabricate data or results, it still tended to do so unless it was provided with a sufficient amount correct experiment information. For larger-scale experiments, exploring the effective format and structure of the experimental results will likely become an important. research considcration.
Despite these improvements, hallucinations still occur, shown in Section 6 Hence, human verification is necessary to ensure the absence of hallucination.
Writing Risk2
Making appropriate citations in the right context remains challenging.
In our system, making appropriate citations in the right context still remains challenging, Through several design improvements, (i) we have prevented the agent from citing non-existent papers, and (2) it can correctly handle citations to papers included in the basoline. However, issues remain with nowly added BibTeX entries. The agent sometimes cites these papers in irrelevant contexts. This problem mainly arises from the current. framework, in which the agent searches for related papers through the Semantic Scholar API, extracts their BibTeX entries and abstracts, summarizes them, and refers to these summaries when writing the manuscript. Because abstracts alone do not contain sufficient information for proper citation, such contextual mismatches frequently occur. Therefore, enabling an AI system to make appropriate citations likely requires deeper, human-level understanding of the referenced papers, which remains highly challenging problem.
Writing Risk3
The result interpretation is unreliable.
We found that the writing agent sometimes makes unreliable or unfounded interpretations of the results. For example, when the proposed method performs better in a table, the agent writes plausible but groundless explanations for why it performs well. Similarly, when referring to figures, tends to exaggerate the effectiveness of the method beyond what can actually be seen. This shows that accurately interpreting experimental results is still difficult task for our Al Scientist system.
Writing Risk4
A mechanism is needed to prevent the agent from generating non-cxistent citations.
During the reflection stage, observed that the agent occasionally modified the BibТeX file its own for example, by introducing incorrect author information or adding entries for papers that do not actually exist. To address this issue, we adopted an agentic framework in which, whenever a citation required during feedback-basee revision, any references to revised or added are dynamically retrieved through the Semantic Scholar API. Tn addition, since the writing agent sometimes automatically generated new .bib file and referenced that instead, we explicitly instruct the agent to refer only entries stored in the verified BibTeX file that contains the correct entries obtained from the Semantic Scholar API.
7.4 Review
Review Riskl
Current Al reviewers cannot detect discrepancies between the actual experimental results and the written descriptions.
Current AI Reviewers primarily evaluate the written content papers and lack any mechanism to detect discrepancies between the text and the actual experimental results. For instance, even if all the reported
17

---

Published in Transactions on Machine Learning Research (02/2026)
ablation studies were fabricated, there is no way for the reviewer to identify such inconsistencies. A similar observation was also reported in (Jiang et al. 2025): To address this issue, it would be necessary to develop a reviewing agent that can access and analyze all associated code files and result data in addition to the manuscript. Developing AI reviewers that can incorporate not only textual information but also experimental code and data will be an important direction for future research.
8 Conclusion
In this paper, we aimed thoroughly investigate the current AI Scientist capabilities and the associated risks. To this end, we developed Jr. AT Scientist, an AT Scientist specialized in extending given baseline paper. By combining carefully designed mechanisms at each stage with the latest powerful coding agents, Jr. Al Scientist is capable of autonomously generating research papers of higher quality than those produced by existing systems. This provides valuable insights into the capability of our Jr. AT Scientist. However, through the author evaluation and the evaluation of Agents4Science, several important challenges have become apparent, which will be important future work. Finally, we present specific examples of the risks and failures identified during development. We hope these insights will help deepen the understanding of both the current progress and the potential risks in AI Scientist research and development.
Limitations and Future Work
Use of Multiple Agents. We built our AT Scientist on top of single coding agent (i.e. Claude Code) and a single family of large language models. We did not explore the potential benefits of combining multiple LLMs or coding agents to further improve performance. Investigating multi-agent or multi-model configurations remains an important direction for future work.
Fragility of Novelty Verification via Semantic Scholar. Our novelty verification relies on Semantic Scholar, which remains fragile and incomplete. How to more reliably ensure and validate the novelty of generated research is an important open problem and key challenge for future research.
Addressing Observed Risks and Identified Limitations. In this study, we place emphasis on accurately reporting observed risks and limitations identified through projects. Developing methods to mitigate these risks and address the identified limitations will be an important direction for future efforts.
Broader Impact Statement
Through this work, we developed a Jr. AT Scientist. However, we do not recommend the use of Jr. AT Scientist or similar systems in the preparation of actual conference submission papers. The primary objective of this paper is to help the community gain clearer and more comprehensive understanding of the current status of AT Sstentists. As discussed throughout the paper, AT Scientists entail various risks. Therefore, when preparing manuscripts for conference submission, it is essential that all content be carefully inspected and validated by human authors prior to publication.
Furthermore, we argue that researchers working on AT Scientists bear a responsibility to report risks, including concrete failure cases and negative outcomes. The study of AI Scientists has significant implications for the academic ecosystem. As these systems continue to advance in capability and autonomy, it is essential that developers remain fully aware of the potential risks they may introduce and commit to their responsible development and deployment. Responsible advancement requires not only the reporting of successful outcomes, but also the systematic disclosure of limitations, failure cases, and potential systemic risks.
This paper adheres to such a responsible research principle. In addition to presenting the capabilities of AI Scientists, we systematically document the risks and shortcomings observed in our study. Through this transparent reporting, we hope to help the community gain a clearer and more comprehensive understanding of the current. status of AT Scientists.
18

---

Published in Transactions on Machine Learning Research (02/2026)
References
Anthropic. System card: Claude opus 4 and clande somnet 4. Technical report, Anthropic, 2025. URL https
Accessed: 2025-09-14.
Anthropic.
Claude code, 2025. URL httpp://cocc.aaurppp.....oo/oo/d
Claude Code: Deep coding at terminal velocity.
Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D D'arcy, et al. Synthesizing scientific literature with retrieval aerieval-ungmented language models. Nature, 2026. ISSN 1476-4687.
Joeran Beel, Min- Yen Kan, and Moritz Baungart. Evaluating sakana's ai scientist for autonomous research: Wishful thinking or emerging reality towards' artificial rescarch intelligence(aari ar preprint arXiv:2502. 14297, 2025.
Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, et al. Ai4research: A survey of artificial intelligence for scientific research. arXiv preprint arXiv:2507.0190) 2025.
Google DeepMind. Gemini 2.5: Our most intelligent model, 2025.
URL https :
byblog.goggle. wwwgoo///..ccoooooggooooooooo/.. hooloologgggggglleoooooooooooooe in hete///mmiiiiiiii gewini-model-theeeeeeeee mooel-thinnking--ee wawphng--ddttt--maaa-aea 2022/
#gemini-2-5-thinkinn Accessed: 2025-04-01.
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint ftt::55021188.4 2025.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distributio examples in neural networks. In ICLR, 2017.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2400.2122 21276, 2024.
Intology. Zochi technical report. 2025. URL http::/ww..htooog..//////ooooooo cessed: 2025-10-17.
Ac-
Fengqing Jiang, Yichen Feng, Yuetai Li, Luyao Niu, Basel Alomair, and Radha Poovendran. Badscientist: Can a research agent write convincing but unsound papers that fool Ilm reviewers? arXiv preprint arXiv:2510. 18003, 2025.
Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. Automated scholarly paper review: Concepts, technologies, and challenges. Information Fusion, 98:101830, 2023a. ISSN 1566-2535 doi: 10.1016/j 201f. URL https:///ai..or/110..00 online 12 May 2023.
inffus. 2023. 101830
Available
Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. Moprd: A multidisciplinary open peer review datasct. Neural Computing and Applications, (5(34):24191-2226, 2023b.
Yixiu Liu, Yang Nan, Weixian Xu, Xiangkun Hu, Lyumanshan Ye, Zhen Qin, and Pengfei Liu. Alphago moment for model architecture discovery. arXiv preprint arXiv:2507.1807 2025.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv: 240.0.622, 2024.
Ziming Luo, Atoosa Kasirzadch, and Nihar B Shah. The more you automate, the less you see: Hidden pitfalls of ai scientist systems. ar Xive preprint or 250:200000,, 2025.
19

---

Published in Transactions on Machine Learning Research (02/2026)
Ludovico Mitchener, Angela Yiu, Benjamin Chang, Mathien Bourdenx, Tyler Nadolski, Arvis Sulovari, Eric C Landsness, Daniel I, Barabasi, Siddharth Narayanan, Nicky Evans, et al. Kosmos: An ai scientist for autonomous discovery. arXiv preprint arXic:2511. 02822, 2025.
Atsuyuki Miyai, Qing Yu, Go Trie, and Kiyoharu Aizawa. Locoop: Few-shot out-of-distributioo detection via prompt learning. NewrIPS, 36:76298 76310, 2023.
Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Yueqian Lin, Qing Yu, Go Trie, Shafiq Joty, Yixuan Li, Hai Li, al. Generalized out-of-distribution detection and beyond in vision language model era: A survey. TMLR, 2025a.
Atsuyuki Miyai, Qing Yu, Go Irie, and Kiyoharu Aizawa. Gl-mem: Global and local maximum concept matching for zero-shot -istrrbiiitiiitti detection. IJCV, (33(6))3586 3596, 2025b.
Alexander Novikov, Ngăn Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. Technical report, Technical report, Google DecpMind, 05 2025. CRL
googleapis .... 2025.
OpenAL. Gpt-5 system card. Technical report, OpenAI, 2025a. gpt-5-system- pdf Accessed: 2025-09-14.
URL https://cdnnooenai...
OpenAl. Openai o3 and o4-mini system card. Technical report, OpenAI, 2025b. URL httpp::ttt...ooopptte openai
sytes card.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In ICLR, 2024.
Chenglei Si, Tatsunori Hashimoto, and Diyi Yang. The idcation-cxoeution gap: Execution outcomes of llm-generated versus human research ideas. Xixiv preprind rrXi::2506.20800 2025a.
Chonglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research idcas? a largo-scale human study with 100+ ulp researchers. In ICLR, 2025b.
Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Coro-bench: Fostering the credibility of published research through computational reproducibility agent benchmark. TMLR, 2025.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Teon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating al's ability to replicate ai research. In ICML, 2025.
Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang, Ai-researcher: Autonomous scientific innovation. In NewrIPS, 2025.
Xiangru Tang, Qiao Jin, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, Meng Qu, Yilun Zhao, Jian Tang, Zhuosheng Zhang, et al. Risks of ai scientists: Prioritizing safeguarding over autonomy. In arviv, 2024.
Francisco Villaescusa-Navaal Boris Bolliet, Pablo Fillamevaa-Dominge Adrian Bayer, Aidan Acquah,
Chetana Amancharla, Almog Barzilay-Siegal, Pablo Bermejo, Camille Bilodeau, Pablo Cárdenas Ramírez, ct al. The denario project: Deep knowledge ai agents for scientifie discovery. arXiv preprint ar arXiu:2510228877, 2025.
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. Cycleresearcher: Improving automated research via. automated review. In ICLR, 2025a.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Yixuan Weng, Minjun Zhu, Qiujie Xie, Qiyao Sun, Zhen Lin, Sifan Liu, and Yue Zhang. Deepscientist: Advancing frontier-pushing scientific findings progressively. arXiv preprint arXiv:2509.2600 2025b.
Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. Scireplicate- bench: Benchmarking Ilms in agent-driven algorithmic reproduction from research papers. In COLM, 2025.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXie preprint arXiv:2504.0006 08066, 2025.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. IJV, 132((2))5633 5662, 2024.
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. In ICLR, 2025.
Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. Deepreview: Improving imm-ased paper review with human-like deep thinking process. In ACL, 2025a.
Minjun Zhu, Qiujie Xie, Yixuan Weng, Jian Wu, Zhen Lin, Linyi Yang, and Yue Zhang. Ai scientists fail without strong implementation capability. arXiv preprint arXiv:2506. 01372, 2025b.
Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. Large language models for automated scholarly paper review: A survey. Information Fusion, 124:103332, 2025. ISSN 1566-2535.
James Zou, Owen Queen, Nitya Thakkar, Eric Sun, and Federico Bianchi. Open conference of ai agents for science 2025. http:///wgenttssscccoo.sssstaaae www.sord..aeeeeoommm edu/ 2025. Accessed: 2025-10-17.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix
A Public Release of Codebase and Detail Prompts
We plan to gradually release a codebase. This decision is motivated by the potential risks associated with making the Jr. Scientist easy to use. As discussed in main paper, Al Scientist systems, including the Jr. AI Scientist, also involve various risks, and the potential impact on the academic community must be carefully considered. Therefore, plan to release codebase after evaluating the risks that this system may pose the rescarch community.
B Main Prompts for Jr. Al Scientist
An example of the prompt in the Idea Generation Phase is shown below. Since the Idea Generation Phase consists of three stages, different prompts are provided for each stage.
B.1 Idea Generation
Prompt for Limitation Analysis
Please analyze the following paper text and identify limitations (constraints/Challenge)) from a broad perspective.
Paper Text: (PAPER TEXT}
Please analyze limitations from the following perspectives:
1. Method/ Batthod/AAggrttaue Algorithm limitations
For each limitation:
Explain the specifie problem
Explain why it is problematic
Propose improvement directions
Additional Requirement:
Include the perspective of "Al autonomously conducting research". You may assume that the Al is provided with access to the codebase of this research. For each limitation, explain how an autonomous AI research agent could refine this limitation through self experiments, automated ablation, meta-learning.
Constraints:
Focus on limitations that can be addressed through relatively simple implementation changes but would have significant impact if improved.
Do not focus too much on innovativenoss: instead, identify limitations with substantial potential for solid performance improvement.
Output the analysis in a structured format.
Output Structure:
## Method Am Limitations
# # Limitation 1: [Short Title]
Specific Problem:
Why Problematic:
Improvement Directions:
SSmple fix or adjustment]
Autonomous AI research perspective]
22

---

Published in Transactions on Machine Learning Research (02/2026)
### # Limitation 2: [Short Title]
# Limitation 3:
Short Title]
Prompt for Idea Generation
Please generate research idea based on the following paper limitation analysis.
Original Paper: (PAPER TEXT}
Limitation Analysis: [LIMITATIONS]
Already generated ideas: {PREV IDEAS}
Constraints:
1. The proposal should not require large computational resources extensive data collection.
2. Research proposal that is executable within the scope of an academic lab
3. Research proposal of quality that can be submitted to top ML conferences
4. The method need not be broadly applicable; effectiveness on the given model and dataset is sufficient.
5. Aim for simple, effective implementation rather than complex overly innovative one.
6. Do not focus too much innovativeness; instead, propose method with solid performance improvement.
7. This research should be conducted by an autonomous AI agent, which means human rescarch activities (e.g., human-in-the-loop) should be excluded.
Please generate one research idca in the following format. Take a different approach from existing ideas and address different limitations.
Important **; Always respond in the following JSON format. Include "'json.
< jsson
{
"idea":
"Name": "Short name for research idea (lowercase, underscores alloved)",
"Title": "Catchy and informative title",
"Short Hypothesis": "Concise explanation of main hypothesis or research question. Clearly explain why this direction is needed, confirm that this setting is optimal for investigating this idea, and confirm that there are no other obviously simpler ways to answer the question.",
"Related Work": "Brief discussion of most relevant related work and how the proposal clearly distinguishes from them, and is not trivial extension.",
"Abstract": "Conference format proposal summary (approximately 250 words)", "Experiments": "List of experiments to be conducted to validate the proposal. Ensure these are simple and feasible. Explain specifically how you would test the hypothesis and detail precise algorithmic changes. Include evaluation metrics you would use.".
"Risk Factors and Limitations"; "List of potential risks and limitations of the proposal"
23

---

Published in Transactions on Machine Learning Research (02/2026)
B.2
Experimentation
An example of the prompt given the Coding Agent to make it write the code required for the Experiment Phase is shown below. Since the Experiment Phase consists of four stages, different prompts are provided for each stage. In addition, even within the same stage, elements such as <memory>> may vary slightly across different executions of the agent.
The <introduction>> Seirectory_structurre and <research idea elements are shared across all stages.
The <rescarch idea> directly contains the research idea generated in the Idea Generation Phase, providing the Coding Agent with high-level contextual overview of the functionality to be imple- mented.
In Stage 2 and Stages 3-4, <perfornance_impooeeeeee improvement, idea> and cablation_staade idea> are inserted, respectively. These guide the implementation strategy.
The <memory>> field embeds summary of the experiments conducted so far within the corresponding stage. This includes results and metrics from successful experiments, as well as causes of failed experiments and error logs, summarized by the LLM.
The <required functionality>> specifies the main functionalities that the Coding Agent is expected to implement. While this content differs across stages, the differences limited to short descriptive text and the name of the entry-point filc. In Stage 1, where the proposed method is implemented, this field is set to <proposed method implementations>. In Stage 2, which focuses on improving the performance of the proposed method, the required functionality specified as <improved proposed method>.
Prompt for Coding Agent (An Example in Stage 1)
<introduction>>
You are an experienced Al researcher.
You are provided with a codebase which implements a baseline experiment. Follow the instructions below and modify the code as necessary. /introduction>>
directory directory_sttucturee structure>
baseline.py>>
A python script that runs the baseline experiment.
Don't t modify this file because it works fine.
Args:
-utpuut-dir (str): path to the output directory where results will be saved.
< //baseline.py>
<plot.py>
A python script that generates plots for the experimental results.
Don't modify this file because it works fine
This script loads 'scores.npz' file and plots the score distributions.
All experimental results, including ablation studies, must plotted using this script; do not write a separate plotting script.
Args:
input-file (str): path to the input 'scores.npz filc.
24

---

Published in Transactions on Machine Learning Research (02/2026)
output-dir (str): path to the output directory where plots will be saved.
</plot.py>
<bsseline ceoeeeeooo_____eeeeeeeeeeeeeeeee dir>
This directory is the original baseline code directory.
Don't t modify this directory. You should keep this directory as it is.
This directory contains the baseline code.
By comparing with this directory and your code, you can understand what change you did in the proposed method.
< /asselne code dir>
< /ddrectory_struetteeee structure>
<rescarch idea>>
[RESEARCH RCH IDEA]
< </rescarch <icbarrchiida>>>>>>
<performance iitrroemmeeptttttteeeeeee idea>>
[PERFORMANCE IMPROVEMENT IDEA}
</performance _ipprovempppttttttm idea>>
<ablation study idea>>
[ABLATION STUDY IDEA}
ioolation_studfe studd_iddax> _dea>>
<memory>>
[MEMORY]
</memory>
<required functionality>
<proposed method
_roposee_meethddd_e implementation>
You implement the proposed method based on research idea exploring novel improvements or revealing
new insights.
You should implement only the core part of the idca, without changing the input /output file formats.
The entry script must take the same arguments as baseline.py py and produce output files.
The output file schema (npz, json) must also be the same as in baseline.py.
Keep codebase modifications to minimum, except where changes are necessary to implement the idca.
The entry script must be named proposed method.py. Example:
'<*
python proposed_method.pyy -output-dir .results
<important notes>>
Write implementation plan in _aoposeo method. md file. </important notes>>
proposed method implemennation>>
required ffnniiiolttttt
dataset ditasst_dircctory>>
The necessary datasets are stored in the following directory: /datasets/LoCoOOp
No additional datasets should be downloaded.
</dataset didrecorry
25

---

Published in Transactions on Machine Learning Research (02/2026)
B.3 Writing
The prompl used for Draft Writing in the Writing Phase is shown below. The portion denoted as (RESEARCH IDEA) } is identical to the text presented in B.2 (RESOURCE OVERVIEW] is the resource explanation provided to the writing agent. [PLOT LIST and {PLOT DESCRIPTION) correspond to experimental results and may vary across different executions.
Prompt for Detailed Writing
Your goal to write up the following idea:
"markdown
[RESEARCH IDEA]
<<<
Note that idea text represents a preliminary hypothesis and may not necessarily align with the experiments that were eventually performed.
First, make sure refer to the experiment summaries contained in baseline summaryyjson, hyperparam ablation summary.json and com- improved research summary-json, Important: The experiment results in hyper- ponent ablation a summary.json.
param @rram_ablation lllaaaaaolssuumnnnn summary.json and component component_ablation _bllationssummaraaya. summary.json are summarized in the tables hyperparam ablation summary table.tex and component ablation summary table.tex. these .tex tables, the columns ablation name and ablation ablation_descriitionn correspond to the name and description of each experiment's ablation idea. Do not directly copy these descriptions into the .txx tables. However, you should refer them when writing the ablation study section in the text, You may modify how the non-numerical content is written in the Lex files, but numerical values must. remain unchanged and must be used exactly as provided.
(You may, however, change the rounding percentage formatting of the numerical values.)
Available plots for the writeup (use these filenames):
ree
(PLOT LIST}
Please also consider which plots can naturally be grouped logether subfigures.
We also have VLM-based figure descriptions:
'*e
(PLOT DESCRIPTION] ree
To better understand the methodology, please also refer to: (RESOURCE OVERVIEW)
Please read the current template.tex file and update it to produce a complete, coherent, and scientifically accurate paper.
This must be an acceptable complete LaTeX writeup, suitable for 8-page single-column paper. Make sure to the citations from the references.bib file and report results accurately based the experimental data provided. Start by reading template.tes to understand the current state, then odit it to incorporate all the information above into a complete paper.
Please note: For the bibliography, do not use the \begin filecontents) (references.bib envi- ronment. Instead, all citations should refer to an external file named references. bib.
26

---

Published in Transactions on Machine Learning Research (02/2026)
Please use the structure plan saved in paper structure.md as guidance for writing the full paper.
Prompt for Citation Validation
Your task is to validate and improve the citations in the paper, particularly in the Related Work section, by comparing with a high-quality baseline paper.
Research idea:
"mmrkddon
[RESEARCH IDEA]
<<c
Note that idea_ text represents a preliminary hypothesis and may not necessarily align with the experiments that were eventually performed.
Please perform the following steps:
1. ** Analyze Citation Standards from Bascline Paper:**
Read a paper from baseline lex/ to understand the citation standards in this field
Pay attention to:
The typical number of citations per section
How citations are grouped and discussed thematically
The balance between classic/fous founiaaiooaaa papers and recent work
Citation density in the Related Work section
2. **Evaluate Current Citations:**
Read the current template.ter tex file Count the total number of unique citations
-Analyze citation distribution across sections
-Check if all citations in references. bib are actually used
3. **Identiyy Citation Gaps:**
Compare your citation count and distribution with the baseline papers
Identify areas where additional citations would strongthen the paper
Check any important research areas or foundational works are missing
4. **Improve Citations:**
Add citations where the paper makes claims that should be supported
Ensure the Related Work section has comprehensive coverage
Make sure recent relevant work (within last 3-5 years) is adoquately cited Verify that seminal/foundational papers in the field are included
5. **Citation Quality *heck:**
Ensure citations are properly formatted and consistent
Check that each citation adds value and discussed meaningfully
Avoid citation padding each citation should serve clear purpose
Please use the Read, Edit, and MuluiEdit tools to:
First analyze baseline_ tex/ papers to understand citation standards
Then review and improve the citations template.tex
Focus especially on the Related Work section but also check other sections
27

---

Published in Transactions on Machine Learning Research (02/2026)
After your analysis, make the necessary edits to improve the citation quality and coverage to match the standards of the field.
Please nole: For the bibliography, do not use the Abegin (ilecontents) ronment. Instead, all citations should refer to an external file named references. bib.
[ (references.bib) envi-
Next, present the prompt used for Reflection.
Prompt for Feedback Generation
# Role You are an AT assistant specializing in reviewing scientific manuscripts. Your mission is to analyze the provided artifacts (manuscript, data, code, etc.) from multiple angles and generate precise, constructive leedback based on the specified criteria.
# Instructions
Read the content from the provided file paths and information, then generate feedback report that focuses the following criteria. For each criterion, state clearly if issues are found, provide specific details if problems potential improvements identified.
# Feedback Criteria
1. ** Verification for Falsification & Inconsistency:**
Thoroughly cross-reference the claims made the manuscript ,/template.tex including text, tables, and figures-againe objective evidence from Experiment Summaries (JSON format), proposed method code/improved propooos proposed method-py, and Plot Descriptions (from VL.M).
**era the Research Idea as background context only and do not flag discrepancies between it and the final manuscript." **
* Identify and report all suspected inconsistencies, such as numerical mismatches, discrepancies between the described methodology and the source code logic, and biased interpretations of plots. ** All numerical discrepancies resulting from rounding are to ignored." **
2. **Verification for Guideline Adherence:**
Check if the mamuscript complies with the rules and guidelines provided in CLAUDE.MD file. Report any deviations from the guidelines, including issues with formatting, style, mathematical notation, or section structure.
3. **Verffiaatinn for Missing Citations:**
Fxamine the manuscript for claims that require a citation but lack one, particularly for background concepts, related work, and datasets.
Using the content from ./template.tex **For each missing citation, create descriptive string that clearly identifies the paper to be found (e.g., 'The original Transformer paper, "Attention Is All You Need." '). These descriptions will form a list to be saved in a single JSON file. **
reference, suggest specific citations that should be added.
4. **Suggestions for Section-by-Sectinn Improvements:***
For each major section of the manuscript (Abstract, Introduction, Method, Experiments, Conclusion, etc.), propose concrete improvements to enhance clarity, persuasiveness, and impact. Provide actionable advice on aspects like logical flow, readability, and the effective use of figures and tables.
* Please ensure that all suggestions are actionable within the scope of the provided experiments and code, without requiring new experimental runs.
Refuse to suggest new experiments, visualizations, comparisons, or analyses. All suggestions must be limited encouraging deeper discussion based on the provided text, figures, and existing data.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Focus on suggestions that amplify the manuscript's strengths and maximize the impact of its contributions. Instead of flagging claims overstated, suggest how rephrase or restructure the narralive to make them more compelling based on the existing evidence.
# Inputs ## Research Idea:
< mmrrddonn
[RESEARCH IDEA} r.c
Note that idea text represents preliminary hypothesis and may not necessarily align with the experiments that were eventually performed.
## Experiment Summaries (JSON format)
Refer to the experiment summaries contained in baseline sunmary.jjon, im- gaato___eearchssummaee nent ablation summary.json.
_ariaprrrm_aaltttioonnoo ablationnssmmarry summary.json and compo- hyperparam ablation
Important: The experiment results in hyperparam_ablatte suatiioyj_mmmaaooonn and compo- nent ablation sunmary.json are summarized in the tables hyperparam ablation summary table.tex and component ablation summary table.tex.
teaeeet_table.eex. In these ttex tables, the columns ablation_ name and ablation descriptiononn correspond to the name and description of each experiment's ablation idea.
Do not directly copy these descriptions into the .tex tables. However, you should refer to them when writing the ablation study section in the text.
You may modify how the non-numerical content is written in the tex files, but numerical values must remain unchanged and must be used exactly as provided.
(You may, however, change the rounding or percentage formatting of the numerical values.)
## Available Plots (list filenames)
[PLOT LIST}
## Plot Descriptions (from VLM)
<ec
(PLOT DESCRIPTION;
...
To better understand the idea, experiments, and results, please also refer to:
baseline. tteetteeeeee and baseline_ baaselnne.pp the baseline TeX and code, respectively, When writing up the baseline method, you must follow its method section. baseline_ters biweiine_bbbbexxxxtt contains the baseline paper's bibliographic information and title read this file to identify the baseline method being compared against.
proposed method code/improved proposed method.py as the proposed method's code implemen- tation (note that the proposed method should be referenced in improved_proppoeed ppoposed_meethd..pppp rather than proposed method.py)
method section.tex (pro-written Method section with detailed technical analysis)
# Output Generation
Based on your review, generate the following two files.
29

---

Published in Transactions on Machine Learning Research (02/2026)
## # Part 1: Feedback Report (Markdown)
**lile to Generate:** 'reflection feedback.md
Format: ** Markdown
** *Content:** A comprehensive report detailing the findings from all four criteria of the 'Core Task'
### Markdown Report Template
# # 1. Results of Falsification & Inconsistency Check
[] No significant inconsistencies or signs of falsification were found.
[] The following discrepancies or concerns were identified:
(e.g., Table reports an accuracy of 92.5%, but the JSON data shows an average of 91.8%. This suggests a potential cherry-picking of the best seed's result.)
a "significant improvement, " but comparison with the VLM description indicates it is within the margin of error. The claim is overstaled.)
(e.g., The text describes the result in Figure
(e.g., The method section states dropout rate of 0.5, but the implementation in the source code uses 0.3.)
# # # 2. Results of Guideline Adherence Check
The manuscript generally adheres to the provided guidelines.
The following deviations from the guidelines were found:
(e.g., The font used for equations inconsistent. Please follow the instructions in CLAUDE.md.)
(e.g., The rule that figure captions must be placed below the figure has not been followed.)
# ####$$ 3. Results of Missing Citations Check
-
No significant missing citations were identified.
The following sections may require additional citations:
(e.g., The second paragraph the Introduction discusses the basic Transformer architecture but lacks citation the original paper, "Attention All You Need.) (e.g., A citation should be added for the iNaturalist dataset used in the experiments, referencing the paper that introduced it.)
# # 4. Section-by-Sectionn Improvement Suggestions
** Abstract:*** ** (c.g., The background description currently too long. The abstract would be more impactful if it started with a concise one-sentence summary of the contribution and included key quantitative results.)
#Introduction:** (e.g., The problem statement is clear, but the motivation could be strengthened by better explaining why this problem is important and what the broader impact of solving it would be.) **Related Work:** (e.g., Rather than simply listing prior studies, structure the section thematically. Clearly explaining how your work differs from or builds upon key papers will better highlight your unique contribution.)
** Methood*** ** (e.g., The pseudocode in Algorithm 1 is dense. Consider simplifying it to show only the main steps and moving detailed explanations the main text improve readability.)
*Exxerimental Setup:** (e.g., For reproducibility, it would be beneficial to include table summarizing all key hyperparameters. Justifying the choice of evaluation metrics and baselines would also strengthen the paper's credibility.)
** "Experiments:** (c.g., Instead of just listing results, add more analysis immediately after each result to discuss *why* it occurred. This would add significant depth to the paper.)
** **batioo Study:** (e.g., It's unclear which components of your method contribute most to the performance gain. An ablation study that systematically analyzes the impact of each component would provide valuable insights and strengthen your claims.)
** Conclusion:* (o.g., Providing specific details the study's limitations and future directions
30

---

Published in Transactions on Machine Learning Research (02/2026)
would demonstrate scientific rigor and foresight.)
# # Part 2: Required Citations List (JSON)
File to Generate:* 'reflection __roquiredd__e bib.json'
** Format:** JSON (A list of strings)
*Content:** ** A machine-readable list of the descriptive strings for each missing citation identified
CCore Task 3: Each string the list should be a clear instruction for a search tool.
# ## JSON Format Example 'json
[
"The original Transformer paper, 'Attention Is All You Need' by Vaswani et al.", "The paper that introduced the iNaturalist. dataset."
Prompt for Feedback Revision
# Role
You are an expert writing assistant who accurately incorporates leedback to improve a manuscript. Your task to faithfully execute the requested revisions to enhance the overall quality the text.
# Instructions
Revise the ./template.tex strictly according to the contents of the Feedback Report found reflection 1 feedback.md. Implement all the changes and suggestions noted in the feedback and output the complete, revised manuscript. Do not introduce new information interpretations that are not specified in the feedback. When the feedback points out missing citations, you must add appropriate ()itep() commands in the main text. Refer to the Newly Acquired Citations (BibTeX format).
** Requests Beyond the Scope of Existing Data (CRITICAL):**
If foedback requests that is impossible with the provided information, **DO NOT INVENT DATA OR RESULTS. ** * Instead, address the comment *in the manuscript's text* by:
experiment, comparison with data you do not have, analysis
* ** F*rsttt, attempting to reframe the point as a strength or a deliberate choice using the principles Strategy #5. As a last resort, if this is not feasible, proceed by:**
Acknowledging the value and validity of the feedback's suggestion.
Politely stating that it falls outside the scope of the current study's experiments.
# Inputs ## Research Idea:
<< markdown
[RESEARCH IDEA] <<e
Note that idea text represents preliminary hypothesis and may not necessarily align with the experiments that were eventually performed.
## Experiment Summaries (JSON format)
Refer to the experiment summaries contained in baseline summary.json, im- proved_research sutprrryj_mmmarrjoonnooo nent ablation summary.json. Important: The experiment results in hyperparam htwaraaaaootttassssiiiiooo and compo-
hyperparam ablation summary.json and compo-
31

---

Published in Transactions on Machine Learning Research (02/2026)
nent ablltionnssummrrry summary.json are summarized in the tables hyperparam_ablationn and component Sittatiionssmmrryyyyee table.tex.
Sonolationssummaryyyy ablation summary table.tex
In these .tex tables, the columns ablation name and ablation description correspond to the name and description of each experiment's ablation idea.
Do not directly copy these descriptions into the tex tables. However, you should refer to them when writing the ablation study section in the tcxt.
You may modify how the non-numerical content is written in the .lex files, but numerical values must. remain unchanged and must be used exactly as provided.
(You may, however, change the rounding percentage formatting of the numerical values.)
## Available Plots (list of filenames)
rec
(PLOT LIST)
## Plot Descriptions (from VLM)
(PLOT DESCRIPTION;
...
## Newly Acquired Citations (BibTeX format)
To better understand the idca, experiments, and results, please also refer
baseline tex/ and baseline code/bas btee/baselineppy the baseline TeX and code, respectively. When writing up the baseline method, you must follow its method section. baseline tex/ bex/baaeline.... bibtex.txt contains the baseline paper's bibliographic information and title read this file to identify the baseline method being compared against.
code/improvee _roposd__ppp_e________ met as the proposed method's code implemen- tation (note that the proposed method should be referenced in improved proposed method.py rather than proposed proposed_method..p method.py)
method section.tes (pre-written Method section with detailed technical analysis)
Please note: For the bibliography, do not use the Abegin(flleconten)) (rafreetttsrrffeeennes...ii (references.bib) environment. Instead, all citations should refer to an external file named references.bie bib.
CL for the writing agent
You ambitious AT researcher who is looking to publish a paper that will contribute significantly to the field.
Ensure that the paper is scientifically accurate, objective, and truthful. Accurately report the experimental results, even if they are negative or inconclusive.
You are planning to submit. to a top-tier ML conference, which has guidelines:
The main paper is limited to 8 pages, including all figures and tables, but excluding references, the impact stalement, and optional appendices. In general, try to use the available space and include all relevant information.
The main paper should be singlo-column format, while the appendices can be in single-column format. When in single column format, make sure that tables and figures correctly placed. Do not change the overall style which is mandated by the conference. Keep to the current method of including the references.bl file.
Do not remove the
graphicspath directive or no figures will be found.
Do not add 'Acknowledgements" section to the paper.
Use single backslash ()) for LaTcX commands instcad of a double backslash (1)).
32

---

Published in Transactions on Machine Learning Research (02/2026)
Here are some tips for each section of the paper:
** Title***
Title should be catchy and informative. It should give a good idea of what the paper is about. Try to keep it under 2 lines.
"Abstract**,
TL;DR of the paper.
What are we trying to do and why is it relevant?
Make sure the rbstract reads smoothly and is well-motivated. This should be one continuous paragraph.
Introductionn
Longer version of the Abstract, i.e., overview of the entire paper.
Provide context to the study and explain its relevance. Highlight how the approach effectively addresses the research question or problem. Also, provide an intuitive explanation why this method works well.
Summarize your contributions, highlighting pertinent findings, insights, or proposed methods.
**Related *ork**
Academic siblings of our work, i.e., alternative attempts in literature at trying address the similar problems.
Compare and contrast their approach with yours, noting key differences or similarities.
FEnsure proper citations are provided.
**Meehod**
Clearly detail what you propose to do and why. If your study aims to address certain hypotheses, describe them and how your method is constructed to test them.
Fxperimentaal *
Explain how you tested your method or hypothesis.
Describe necessary details such as data, environment, and basclines, but omit hardware details unless explicitly mentioned.
** "Experiments**;
Present the results truthfully according to the data you have. If outcomes not expected, discuss it transparently.
Include comparisons to baselines if available, and only include analyses supported by genuine data. Try to include all relevant plots and tables. Consider combining multiple plots into one figure if they are related.
In tables, please bold the best result among the compared methods. There no need bold the proposed method if it is not the best.
*Coelusion **;
Summarize the entire paper, including key strengths or findings.
Results are strong, highlight how they might address the research problem.
** *Appendix**
Place for supplementary material that did not fit in the main paper.
Add more information and details (hyperparameters, algorithms, etc.) in the supplementary material. Add more plots and tables in the supplementary material. Make sure that this information is not already covered in the main paper.
33

---

Published in Transactions on Machine Learning Research (02/2026)
When checking for duplicate figures, be sure to also review their descriptions catch cases where different figures convey the same information. For example, figure might present aggregated training accuracy gated training_accatta accuracy.png) png)) while another data as three separate line plots.
single line plot with shaded standard deviation (e.g., aggre-
training accuracy.png) shows the same
Ensure you are always writing good compilable La TeX code. Common mislakes that should be fixed include:
LaTeX syntax errors (unenclosed math, unmatched braces, etc.).
Duplicate figure labels references.
Unsccpped special characters: & % $ ~7
Proper table/figure closure.
Do not hallucinate new citations or any results not in the logs.
Ensure proper citation usage:
Al Always include references within Abegin ( (refrecccsb \end(filecontentss) even if
they haven't changed from the previous round.
**Before citing any paper, ALWAYS read the references.bib file first to find the correct citation key.
Do NOT create new fictional bibtex entries.** **
Do not make any changes to reference.bit
** Verify all citation keys match exactly with those in references.bl before using them the text." **
Note that the bascline BibTeX entry located near the bottom of references.be
Each section (especially Related Work) should have multiple citations.
## Notes
Do not include formatted mathematical formulas or equations in the Abstract. However, including key numerical results (e.g., "an accuracy of 92.5%") and mentioning mathematical concepts by name is perfectly acceptable.
During the reflection process, the abstract should be significantly altered.
Do not include results in the Method section.
Do not include Limitation sections and Future Work sections.
Use '%' to display literal percent sign, standard *% will be treated comment.
C Detail of Agents4Science Reviews
This section presents detailed reviews from Agents4Science. In Section we focused on reviews that address the fundamental limitations of the Jr. Al Scientist. In contrast, this section provides detailed reviews to clarify what is required for Al Scientist soientist-generaaeed papers to be accepted at future Agents4Science conferences. (If the goal just acceptance, including human intervention is the fastest way.) As noted in Section 5 the papers submitted were generated by an carlier version of the Jr. AI Scientist and are therefore not identical to the paper introduced in this work.
C.1 LoCoOp Extention Paper
Reviewer 1
Review Score
Score: 3: Borderline reject
34

---

Published in Transactions on Machine Learning Research (02/2026)
Reviewer Comments
The paper proposes Nuisance-Prompl Tuning (NPT) for few-shot OOD detection with CLIP, in- troducing learnable muisance prompt, estention-weighter patch supervision, and a margin loss to separate nuisance and class prompts. The method is simple, well-motivated, and addresses background contamination in few-shot prompt learning, with coherent integration of components. Empirical gains are consistent on three of four OOD datasets, with the largest improvement. on iNaturalist, but overall improvements are modest (AUROC +0.006; FPR95 -0.028 absolute). 'hhe evaluation is limited in breadth (only one backbone, few basclines, multi-shot results in main text), and lacks statistical rigor (no error bars, single-seed reporting). The reliance CLS attention proxy for foreground/ background is plausible but unvalidated. Some ablation results raise questions about training confounds. The inference scheme is sensible but under-explored. The method is clearly described, but some implementation details and sampling protocols are missing. The work is an incremental advance, with moderate novelty, and reproducibility is plausible but not robustly supported. Ethics and limitations are briefly discussod. Actionable suggestions inchude expanding evaluation (more backbones, baselines, multi-shot, error bars), validating the attention assumption, analyzing SUN degradation, clarifying implementation, and exploring alternative inference schemes. Overall, the paper is clear and reasonable contribution with promising intuitions, but the empirical evidence and breadth fall short of top-tier standards. Recommend rejection in current form, with a path to acceptance after expanded, statistically robust evaluation and stronger baselines aaaelines/analysess. aanalyses.
Reviewer 2
Review Score
Score: 5: Accept
Reviewer Comments
Strengths:
Clear motivation and problem formulation, with a precise critique of prior work. Technically sound, novel, and elegant method, with clever use of CLIP's self-attention and well-designed loss function. Comprehensive and rigorous evaluation, including strong baselines and qualitative evidence. Strong ablation study demonstrating the importance of each component. High clarity and readability throughout the paper.
Weaknesses:
Lack of statistical significance testing due to single-run experiments; error bars or confidence intervals are reported. Slight performance degradation on the SUN dataset is acknowledged but not deeply analyzed. Limited discussion of limitations; dedicated section would be beneficial. Overall, this is high-quality paper with solid technical contribution, rigorous validation, and excellent clarity, The weaknesses are minor and do not detract significantly from the overall strength of the work. The paper makes clear and valuable contribution to the field and is well-suited for publication al a top-tier conference. Strongly recommended for acceptance.
Reviewer 3
Review Score
Score: 3: Borderline reject
35

---

Published in Transactions on Machine Learning Research (02/2026)
Reviewer Comments
This paper introduces Nuisance-Promge Tuning (NPT), method for few-shot out-of-disuribution (OOD) detection that addresses background contamination in prompt learning approaches. The paper is technically sound, with a clear motivation and comprehensive experimental validation across four OOD benchmarks. The method uses a dedicated 'muisance' prompt, attention-wwighhte supervision, and margin-based repulsion, and is well-presented with thorough ablation studies. However, the improvements over baselines are modest, with only small gains in FPR95 and AUROC, and there is performance degradation the SUN dataset, raising concerns about robustness and generalizability The approach is incremental, introducing additional hyperparameters despite claims of reduced tuning complexity, and the novelty is mainly in the combination of existing techniques. The paper is well- written and reproducible, but the Al-generated nature of the work raises questions about the validity of experimental claims. Limitations and ethical considerations are discussed, and related work is well-covered. Specific concerns include small improvement margins, unexplained performance drops, questionable claims about threshold brittleness, limited analysis of computational overhead, and delaset-dependent effectiveness. Minor issues include figure readability and theoretical motivation for the margin-based repulsion. Overall, the work is solid but incremental, with limited impact and some concerns about robustness and reproducibility.
C.2
Min-K%++ Extention Paper
Reviewer 1
Review Score
Score: 2: Reject
Reviewer Comments
The paper introduces Multi-Layer Concentration Analysis (MLCA) to improve pre-training data delection by augmenting Min-K%++ with distribution--hapee features computed at multiple layers. While the motivation is clear and the method is described transparently, the review identifies several significant weaknesses: (1) lack of salibratioon/alliattion for per-layer distributions, (2) errors and ambiguities in aggregation and normalization formulas, (3) inconsistent application of the method across architectures (multi-layer only for Mamba, not Pythia), (4) modest and mixed empirical gains with insufficient statistical rigor (no confidence intervals, inconsistent reporting), (5) hand-chosen fusion weights without proper justification or cross-validation, (6) limited basclincs (only Min-K%++), and (7) incomplete reporting of experimental details and reproducibility. The contribution is seen as incremental, with originality limited by the use of standard fcatures and lack of novel multi-layer probing. The review recommends rejection, suggesting that revised version addressing calibration, formula correctness, fairer comparisons, stronger baselines, and more rigorous evaluation could be solid contribution.
Reviewer 2
Review Score
Score: 3: Borderline reject
Reviewer Comments
Strengths:
Novel insight that SSMs like Mamba benefit more from multi-layer analysis than Transformers, suggesting fundamental architectural differences. Well-motivated and technically sound approach,
36

---

Published in Transactions on Machine Learning Research (02/2026)
building on strong baselines and using theoretically grounded features. Strong empirical results for Mamba, with clear visualizations and meaningful AUROC improvements. Clear writing, good structure, and dedicated limitations section. Weaknesses:
Critically flawed evaluation for the Transformer (Pythia): the comparison is invalid due to a "simplified' analysis for Pythia, lacking transparency and rigor, undermining claims about architectural differences. Inconsistent hyperparameter selection: main results use a suboptimal value, weakening confidence the findings. Limited robustness: small sample sizes, single runs, and no stalistical significance tests make it hard assess the reliability of the reported gains. Overall, the paper presents a promising idea and strong results for Mamba, but major experimental flaws especially regarding the Transformer evaluation and hyperparameter inconsistency outweigh the reasons to accept. The paper is not ready for publication in its current form, but could be reconsidered if these issues are addressed.
Reviewer 3
Review Score
Score: 3: Borderline reject
Reviewer Comments
This paper presents Multi-Layer Concentration Analysis for enhancing pre-training data detection in large language models. The work is technically sound, building on the Min-K%++ baseline and introducing concentration features (Shannon entropy, Gini coefficient, top-k concentration, effective vocabulary size) from multiple network layers. The mathematical formulations are correct and the experimental methodology reasonable. However, improvements modest (especially for Pythia, 0-1 percentage points AUROC), there is statistical significance testing due to single runs, and the theoretical justification for multi-layer analysis could be stronger. The paper is generally well-written and organized, with clear methodology and effective figures, though some technical details (like layer selection and fcature aggregation weights) could be clearer. The significance is limited, with incremental improvements and the most notable gains for Mamba (up to 1.9 percentage points). The architectural insights are interesting but not groundbreaking. The originality lies in combining multi- layer analysis with concentration features, but the core idcas not particularly novel individually. The paper provides sufficient detail for reproduction, though the lack of error bars and single runs limit reproducibility assessment. Ethical considerations are adequately discussed, focusing on data privacy and copyright. The related work section is comprehensive and citations are appropriate. Major concerns include modest improvements, single runs without statistical testing, shallow theoretical understanding, and limited exploration of architectural differences. Minor issues include arbitrary hyperparameter choices, limited computational overhead analysis, and some unsupported claims about architectures. Overall, the paper addresses an important problem and shows consistent if modest improvements, representing an incremental advance rather than significant breakthrough.
C.3 GL-MCM Extention Paper
Reviewer 1
Review Score
Score: 3: Borderline reject
37

---

Published in Transactions on Machine Learning Research (02/2026)
Reviewer Comments
Strengths:
The method is simple, clear, and training-free, with minimal overhead. Uses a principled uncertainty signal to address max-pooling weaknesses. Qualitative evidence is provided via score distributions. Weaknesses:
Evaluation is very small subsets (100-500 500 images), lacking error bars and statistical testing, which limits reliability and invites sampling variance. Baselines limited to GL-MCM; comparisons to other zero-shot OOD methods (e-g., CLIPN, ZOC, MCM). The method is sensitive to the parameter, with no automatic selection mechanism, raising deployment risk. Inconsistency in claims: Table claims "consistent improvements, " but SUN AUROC decreases while FPR95 improves. Subset selection details (randomization, seeds) are unspecified, limiting interpretability and reproducibility. The "first" claim of an information--hheortiic framework overstated, as related ideas exist. Recommendattinns Expand evaluation to full ImageNet-OOD benchmarks, use multiple random seeds, and report confidence intervals. Compare against more zero-shot OOD methods under standardized protocol. Provide sensitivily analyses and investigate automatic selection. Clarify subset selection and compute details; include runtime benchmarks. Add qualitative visualizations of entropy weights on images. Correct or qualify the "consistent improvements" claim and discuss the SUN AUROC trade-off. Consider alternative uncertainty signals and report ablations. If claiming theoretical grounding, provide supporting analysis. Overall, the idea neat and potentially useful, but the current empirical evidence is too limited for acceptance at a high-standard venue. Strengthening evaluation and comparisons would improve the paper's credibility and impact.
Reviewer 2
Review Score
Score: 3: Borderline reject
Reviewer Comments
This paper introduces Entropy- Entropy-Weighted Local Concept Matching (ELCM), a novel, theoretically- motivated method for zero-shot oul-of-listribution (OOD) detection using vision-language models. The main contribution is an entropy-basee weighting scheme for aggregating patch-level predictions, addressing the limitations of max-pooling in prior work (GL-MCM). The paper is well-written, clearly molivated, and demonstrates consistent improvements over the GL-MCM baseline several OOD datasets, with notable reductions in FPR95. The authors are transparent about the method's limitations.
However, the experimental validation is insufficient: the evaluation limited to a single baseline (GL-MCM), with no comparison other state-of-the-art methods, making difficult assess the true significance of the approach. The experiments are conducted on small samples without statistical robustness (no error bars multiple runs), and the method is highly sensitive to key typerparameter. Additionally, the impact of several engineering enhancements is not disentangled from the core contribution due to a lack of detailed ablation studies.
Overall, while the idea is promising and the presentation is strong, the paper's empirical evidence is too weak to support acceptance. recommend rejection in its current form, but encourage the authors to address the experimental shortcomings and resubmit, as the core idea could underpin a strong future paper.
Reviewer 3
Review Score
Score: 3: Borderline reject
38

---

Published in Transactions on Machine Learning Research (02/2026)
Reviewer Comments
This paper presents Werrhlee Local Concept Malching (ELCM), an improvement to GL- MCM for zero-shot of-istiiiibttiinn detection in vision-language models. The technical approach is sound and theoretically motivated, using Shannon entropy to weight patch-level contributions, but the improvement is modest (AUROC from 0.9129 to 0.9188, FPR95 from 0.3495 to 0.2975) and introduces heprraarreeter sensitivity. The implementation includes ad-hoc components that undermine the theoretical elegance. The paper is well-written and clearly structured, though some implementation details relegated to the appendix. The impact is limited, improvements are modest and only compared to GL-MCM, not other methods. The originality lies in applying entropy-based weighting, but the work is incremental. Reproducibility is reasonable, with code provided, but evaluation is limited to 100 images per dataset. The authors are transparent about limitations, including hyperparameter sensitivity and limited baseline comparisons. Related work is adequate but could be broader. Major concerns include limited evaluation scope, modest improvements, hyperparameter sensitivity, and ad- hoc enhancements. Strengths are thcorctical motivation, honest ovaluation, consistent improvements, and clear presentation. Overall, the paper solid incremental contribution but does not make significant impact due to modest improvements and practical limitations.
39


Reviews:
[{"review_id": "RBBG61i2W3", "reviewer": "Reviewer_bvD1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript investigates the potential of AI scientists for the task of expanding a baseline paper. First, by analyzing the baseline both in terms of manuscript and code. Then identifying a gap. Latter generating ideas to cross that gap. And finally, by analyzing by reflection the proposed idea and improving them.\n\nStrengths\n- The paper is well written, transmitting clear ideas, following logical sequences, and technically sound.\n- The overall methodology seems appropriate as it follows the human approach to conduct research. Nonetheless, the details of each phase can be better explained.\n- The results show that the proposed method outperforms previous methods.\n- Authors acknowledge the limitations of the current state of the model.\n\nWeaknesses\n- The proposed method is requieres a lot of human interventions, for instance to manually define the abstract, input the code, or define the captions of figures, which makes it difficult to identify whether the method is simply a sequence of calls to LLM's or a real chain of thoughts in itself.\n- There are several figures that are not cited in the main text. They simply appear.\n- There is a passage that reads \"If all nodes fail, the system selects the next nodes...\", which is contradictory in itself. If all nodes have failed, then there is no next node to be selected.\n- The computation details of the scores used for comparison are not provided.", "claims_and_evidence_yn": "No", "claims_explanation": "Must claims about the design and experimentation sound logical and needed. However, there is no evidence about the results, other than Table 2.", "audience_yn": "Yes", "audience_explanation": "The topic is of interest to the TMLR community, and the proposed methodology seems coherent and appropriate. Moreover, the results show that the proposed model outperforms previous ones.", "requested_changes": "- Section 3.3.2 requires a clearer explanation.\n- Section 3.4.1 requires a clearer explanation.\n- Prompts used for each stage must be provided for reproducibility.\n- All acronyms must be defined at their first appearance, e.g., OOD detection.\n- The computation details of the scores used for comparison must be provided.", "broader_impact_concerns": "There is no concern about ethical implications, as authors have covered all requirements.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "957ifK14CS", "reviewer": "Reviewer_mujM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Key Contributions:\n- Development of a complex multi-file AI Scientist system that significantly outperforms existing benchmarks from other recent works (5.75 vs 4.50)\n- Generation of papers using 3 high-quality publications by making the AI agent follow realistic research workflow steps.\n- A thorough risk report on various areas where human intervention and expertise is still required to ensure the integrity of the scientific research process.\n\nWeaknesses:\n- The paper reports scores based on 3 paper generation experiments. The authors claim that they have generated several papers but only evaluating the highest quality ones without providing any details as to what constitutes as \"several\" on Page 9.\n- The system is dependent on human provided baseline paper and code artifacts for idea generation albeit this is of least concern.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Despite of the weaknesses, the paper still advances AI agent driven AI research benchmarks significantly and also incorporates a feedback loop for LLM based review step towards the end of the research workflow. It also outlines various risks associated with their system where human intervention and expertise is required.", "audience_yn": "Yes", "audience_explanation": "The paper targets a problem in the emerging field of agentic AI led research which readers of the below groups would be interested in:\n\n1. Researchers in Agentic AI and Automated Discovery\nThe primary audience would be those tracking the evolution of \"AI Scientists.\" The paper introduces a system that mimics human research workflows, handles multi-file codebases and achieves state-of-the-art review scores. In automated assessments, it earned an average rating of 5.75, substantially outperforming systems like AI Scientist-v1 (3.30) and AI Scientist-v2 (2.75).\n\n2. Ethical AI and Safety Researchers\nThe paper highlights review-score hacking and integrity issues with citation accuracy, hallucination of facts, and the fabrication of descriptions in autonomous research workflows.\n\n3. Peer Review and Meta-Learning Specialists\nResearchers focused on improving AI reviewers or understanding the peer-review process would find the paper's evaluation methodology valuable. It uses DeepReviewer, author-led evaluations, and actual submissions to the Agents4Science conference (where AI acts as both author and reviewer) to validate its findings.", "requested_changes": "- I would like the authors to provide more information around how many papers they generated in order to come up with the three papers provided as supplemental material and if possible also quantify the cost of doing so. This will provide the paper a new axis on which future work can be conducted in spirit of AI efficiency.", "broader_impact_concerns": "There are no Broader Impact Concerns with this paper as the authors are very forthright in reporting AI use and on the clarity of the experimentation conducted. The paper also presents a risk report outlining various different risks of using AI to conduct scientific research which would be very helpful in studying the impact of such work.", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "tQZeCYCqxg", "reviewer": "Reviewer_MZWK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents Jr. AI Scientist, which is an autonomous agent system designed to mimic the research workflow of a student researcher. The authors also include a risk report and evaluations, including AI reviewers, author inspections, and Agents4Science reviews.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, but some claims need a more clearly defined scope, evaluation rigor, and workflow justification.", "audience_yn": "Yes", "audience_explanation": "It might be good to include how TMLR audience can benefit from this Jr. AI scientist work. In the paper, the contribution focuses on Jr. AI scientist development and risk evaluations. It might be good to build a toolbox or website for this type of Jr. AI scientist would potentially increase participation and enhance the work’s practical impact.", "requested_changes": "1. Clarity of Framing:\n\nThe framing often changes between presenting Jr. AI Scientist as a “novice student analog,” a “state-of-the-art autonomous agent,” and an “upper bound demonstration” of current AI scientist capabilities. These three framings may imply different success criteria. What is the clear definition of Jr. AI scientist? The narrative would benefit from explicitly defining the intended scope early. Otherwise, the evaluation becomes difficult to interpret.\n\nIn the introduction section, it will be great to reorganize to explicitly define what “capability,” “upper bound,” and “novice workflow emulation” mean in measurable terms. This will also clarify the evaluation criteria.\n\n2. Clarity of Framework and Pipeline:\nFigure 2 contains two-stage 3 framework. It's a bit confusing for readers to differentiate between these two stage 3 modules, though they are all ablation stages. It might be beneficial to combine them into one big stage 3, and draw out two boxes to show one is about hyperparameter ablation and the other is component ablation. \n\nFor iterations, why does Stage 1 have 12 iterations and Stage 2 have 50 iterations? There is no justification for these numbers. How do authors get to this number of 12 but not bigger?\n\nAdditionally, the experiments rely heavily on a single coding agent (Claude Code) and a single LLM family (o4-mini, GPT-4o reviewers). Cross-model validation is necessary for broader claims. If it's not feasible to run with other agents, perhaps adding a limitation section and stating this limitation would also be sufficient. \n\n3. Evaluations:\nIt's interesting to see the author's evaluation relies heavily on a mix of: AI-based review scoring (DeepReviewer), Author-led curation of “best papers,” and Agents4Science reviews. In my opinion, the use of “best-of-several generated papers” may introduce significant selection bias. \n\n3.1. It will be great to provide the distribution of review scores across all generated papers. For example, create a table to obtain the full distributions of scores (e.g., min / median / max).\n\n3.2. The DeepReviewer system can be unstable and sensitive to formatting, writing style, and non-semantic features. It will be clear to state the DeepReviewer reliability, variance, and known biases. \n\n3.3. For the Agents4Science evaluation, please also provide excerpts, reviewer comments, or score summaries in the appendix or manuscript so that the evaluation criteria are clear and reproducible for future researchers.\n\n4. Some minor comments: The novelty verification via Semantic Scholar remains fragile. In the paper Limitation section, it will be great to mention the limitations of LLM-based literature checking.", "broader_impact_concerns": "This work could become an impactful reference for AI scientists' system design. It's important to explain how generalizable this type of AI scientist system is to the whole ML or general science community.", "additional_comments": "The paper presents a promising and methodologically rich system. The risk reporting perspective is especially valuable.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)