DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively
Authors: Yixuan Weng
Minjun Zhu
Qiujie Xie
QiYao Sun
Zhen Lin
Sifan Liu
Yue Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Automated Scientific Discovery
Large Language Models (LLMs)
AI Scientist
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This is the first empirical demonstration of an AI that acts as an autonomous scientist to progressively push research frontiers, successfully discovering novel methods that outperform the human SOTA across multiple domains.

Abstract: While previous AI Scientist systems can generate novel findings, they often lack the focus to produce scientifically valuable contributions that address pressing human-defined challenges. We introduce DeepScientist, a system designed to overcome this by conducting goal-oriented, fully autonomous scientific discovery over month-long timelines. It formalizes discovery as a Bayesian Optimization problem, using a cumulative Findings Memory to intelligently balance the exploitation of promising avenues with the exploration of novel hypotheses. Consuming over 20,000 GPU hours, the system generated about 5,000 unique ideas and experimentally validated approximately 1100, ultimately surpassing human-designed 2025 state-of-the-art (SOTA) methods on three frontier AI tasks by 183.7\%, 1.9\%, and 7.9\%. Crucially, this was achieved by autonomously redesigning core methodologies, not merely recombining existing techniques. In a striking demonstration, the system achieved progress on AI text detection in just two weeks that is comparable to three years of cumulative human research. This work provides the first large-scale evidence of an AI achieving discoveries that progressively surpass human SOTA on scientific tasks, producing valuable findings that genuinely push the frontier forward.

Published as a conference paper at ICLR 2026
ADVANCING FRONTIER PUSHING SCIENTIFIC FINDINGS PROGRESSIVELY
DEEPSCIENTISS
Yixuan Weng', ; Zhu Qiujie X Qiyao Sun', Zhen Lin', Sifan Liu', Yue Zhang"
'School of Engineering, Westlake University "Zhejiang University
bengsyye@gail..omm { izhuminjum zhangyue| @west edu. on
Project: https: / //de/p/deeciieeeeeee/ss http//eeeeecceisss......
Code&Logs:
ABSTRACT
While previous AI Scientist systems generate novel findings, they often lack the focus to produce scientifically valuable contributions that address pressing human-defined challenges. We introduce DeepScientist, system designed overcome this by conducting goal-oriented, fully autonomous scientific discov- ery over month-long timelines. It formalizes discovery as Bayesian Optimiza- tion problem, using a cumulative Findings Memory to intelligently balance the exploitation promising avenues with the exploration of novel hypotheses. Con- suming over 20,000 GPU hours, the system generated about 5,000 unique ideas and experimentally validated approximately 1100, ultimately surpassing human- designed 2025 ofte-offthhe-art (SOTA) methods on three frontier Al tasks by 183.7%. 1.9%, and 7.9%. Crucially, this was achieved by autonomously redesign- ing core methodologies, not merely recombining existing techniques. In strik- ing demonstration, the system achieved progress on Al text detection in just two weeks that is comparable three years of cumulative human research. This work provides the first large-scale evidence of an Al achieving discoveries that progres- sively surpass human SOTA scientific tasks, producing valuable findings that genuinely push the frontier forward.
Human Expert Researchers
DeepScientist
Gradual
Rapld Bvoutio Curve
(Zem-shhat DeepScienttv
TDT
PA-Detect
[Supervised] Human
RAID dataset Performance (AUROC Score)
(Zero-shot) Human
0.85
RADAR NeuriP 2023
Supervised SaTA
T-Detect
Binoculars ICNL
Zerosshot SoTA
0.80
Fast-Detect ICUR 2024
0.75
LRR EMMLP 2023
Glimpse ICLN 2025
0.70
Log-Rank
14 days: Achieved new per
Log-Perplexity ACL 2019
Exceeded previous Supervised
3 days Optimized 2 year-odd besetine, approsching Zero-shot SoTA
0.61
2019
2020
2021
2022
2023
2024
2025
10
15
Year
Day
Figure 1: Research timelines for Al text detection RAID. Left: Three years of human research. Right: DeepScientist achieves comparable progress in just two weeks. Moreover, all DeepScientist methods demonstrate higher throughput than the previous SOTA, Binoculars.
"Equal contribution.
Correspondence Yue Zhang wehaanyue@eesslakcc....

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Scientific discovery is inherently process of continuous exploration and trial-and-error, where vast amounts of time and effort are invested to push the boundaries of human knowledge forward by small step. This principle of persistent, incremental advancement is visible across the history of technology. For example, the decades-long optimization of semiconductor manufacturing has seen the feature size of transistors systematically reduced from micrometers to single-digit nanome- ters (Moore, 1965). Similarly, the efficiency of photovoltaic cells has been continuously advanced over half a century, with myriad material and architectural innovations pushing conversion rates from nascent single-digt percentages ever closer to their theoretical limits (Green, 1993). These historical trajectories underscore process where human scientists engage in decades of goal-directed, iter- ative work to advance the SoTA artifacts continuously. In this work, we ask whether an Al-driven system can participate in such long-horizon, goal-directed scientific progress modern tasks, and how its behavior compares to that of human scientists.
Recently, the emergence of Large Language Models (LLMs) has propelled automated scientific discovery, where LLM-based Al Scientist systems take the lead in exploration (Xie et al., 2025). With their powerful capacity for long-form generation and comprehension, LLMs enable end-to- end, full-cycle automation in scientific discovery. This has inspired influential work such as Al SCIENTIST-V2 (Yamada et al., 2025), whose scientific artifacts have been published in top-tier con- ference workshops. However, in the absence of clearly defined scientific goals, current AI Scientist systems often fall into the trap of blindly recombining existing knowledge and methods. As a result, their research outputs frequently appear naive under human evaluation and lack genuine scientific value (Zhu et al., 2025b). AI Scientists are yet to solve human challenges.
To solve real-world challenges, we formally model the full cycle of scientific discovery as a goal- driven Bayesian Optimization problem, where the primary objective is to discover methods that reliably improve a given evaluation metric over strong human -ennand bascline under a fixed compute budget. Building on this formulation, we introduce DeepScientist, an LLM--based agent system explicitly designed operate on modern, resource-intensive AI research problems, rather than on small-scale symbolic or synthetic tasks. Architecturally, DeepScientist departs from the common "one-shot pipeline" "single-idea infinite trial-and-error" paradigm by implementing three-stage iterative workflow (hypothesis generation, implementation & evaluation, and analysis & abstraction) that is tightly coupled with persistent Findings Memory accumulating both successful and failed attempts over month-long runs. Within this framework, Bayesian surrogate model and acquisition function reason over thousands of past experiments to select the next hypotheses to test, allowing the system to intelligently balance exploitation (deepening investigations into promising high-value directions) with exploration (venturing into under-explored areas to acquire new knowl- edge). Through large-scale parallel exploration informed by this Bayesian-optimizati perspective and shared memory, DeepScientist can generate innovative hypotheses and ultimately yield both valuable new methods and iilidattion-roven scientific findings.
We select three frontier scientific tasks (Agent Failure Atrriution, LLM Inference Accel- eration, and Al Text Detection ). take their state-of-the-art methods (ICML 2025 Spotlight, ACL 2025 Outstanding, ICLR 2024) as starting points, and ask DeepScientist to conduct contin- uous research. As shown in Figures I and within month-long cycle of exploration, validation, and iteration on 16 H800 GPUs, DeepScientist exceeds their respective human SOTA methods by 183.7% % (Accuracy), 1.9 % (Tokens/second)) and 7.9% (AUROC) through autonomously re- designing core methodologies, rather than simply combining existing techniques (Section 4.1). To understand how such progress emerged, we analyze DeepScientist's discovery logs, and formed a small program committee to review the generated papers (Section 4.2). These logs show that the system generated over 5,000 unique ideas, of which only 1,100 are selected for experimental val- idation, and just 21 ultimately lead to scientific innovations (Section 4.3). Moreover, through the scaling experiment on computational resource, we discover a near-linear relationship between the resources allocated and the output of valuable scientific discoveries.
To our knowledge, this is the first large-scale empirical demonstration of an automated system that continuously advances scientific frontiers complex AI tasks, rivaling human researchers under comparable compute budgets. However, these results also highlight a critical reality: Al's exploratory capacity is immense, but genuine success is scarce. Consequently, effective validation,
2

---

Published as a conference paper at ICLR 2026
filtering, and the strategic reuse of failed attempts have emerged as the new bottleneck for automated science. The core question for the field is no longer whether Al can innovate, but how to efficiently guide its powerful yet dissipative exploratory process. We hope insights, alongside the released logs and codebase, will inspire the community to build more reliable Al Scientist systems, acceler- ating scientific discovery scale.
2 RELATED WORK
Replication and Optimization. A significant body of research focuses on engineering tasks that operate within established scientific frameworks. This includes replication-oriented works like Pa- perBench (Starace et al., 2025) and Paper2Agent (Miao et al., 2025), which aim to reproduce existing papers. Other works, such as Agent Laboratory (Schmidgall et al., 2025b) and MLE-Bench (Chan et al., 2024), tackle early-stage machine learning enginecring problems. Similarly, systems like Al- phaTTnsor (Fawzi al., 2022), ASI-Arch (Liu et al., 2025) and A AlphaEvolve (Novikov et al., 2025) use massive -rrr-- with known engineering methods to improve the performance of code- bases (Romera-Pareel et al., 2024; Shojaee al.). The common goal of these efforts is engineering- driven optimization within an established scientific paradigm, enhancing existing systems without questioning their foundational assumptions. DeepScientist, in contrast, pursues scientific discov- ery by explicitly targeting the core limitations of strong methods on modern AI tasks: its objective not merely to refine existing implementations, but propose and validate new methodological directions that establish improved SOTA performance.
Semi-Automated Scientific Assistance. The path toward automating scientific discovery begin not with replacing the scientist, but with assisting them, leading to the development of a paradigm of specialized Al tools for individual research tasks. Systems like CycleResearcher (Weng et al., 2025) handle writing, DeepReview (Zhu et al., 2025a) manages reviewing, and co-scientists (Gottweis et al., 2025; Penadés et al., 2025; Swanson et al., 2025; Back et al., 2025) aid in hypothesis gen- eration. These powerful tools address only isolated fragments of the scientific process, leaving the crucial loop of learning from failure and exploration to humans. In contrast, DeepScientist is an autonomous agent of inquiry, managing the entire end-to-end research cycle and closing the loop by learning from its own experiments and self-directing its research path.
Automated Scientific Discovery. Building on the capabilities of specialized assistants, a line of research pursues full research automation (Xie et al., 2025). Pioneering efforts, such as the Al Sci- entist systems (Lu al., 2024; Yamada al., 2025) and subsequent work (Intology, 2025; Jiabin et al.. 2025; Miyai et al., 2025), successfully demonstrate that an Al system can manage the full research cycle and produce novel findings. However, these systems are typically evaluated on rel- atively synthetic or narrowly scoped problems, and their exploratory strategies are not anchored to clearly specified scientific goals or strong human baselines. This can lead to undirected discover- ies that, while novel, are often perceived having limited scientific value in practice. In contrast, DeepScientist is designed to operate on modern, high-cost AI tasks with competitive human SOTA goal-driven Bayesian optimization problem over these baselines.
methods and to treat discovery Its exploration is explicitly tied to identified limitations of the human SOTA- COTA-using failure attribu- tion and persistent Findings Memory to prioritize hypotheses that are both novel and measurably impactful.
3 DEEPŠCIENTIST: A PROGRESSIVE SYSTEM FOR DISCOVERING SOTA-SURPASSING FINDINGS
DeepScientist an LLM--bsed multi-agent system equipped with an open-knowledge system, a continuously accumulating Findings Memory, which is composed of both frontier human knowl- edge (c.g. papers and codes) and the system's is own historical findings. This memory is constructed and updated fully automatically during the system's 's operation, without any manual editing, and each record stores the hypothesis, implementation details, evaluation metrics, and logs of both successful and failed experiments (Zhou et al., 2025). This memory intelligently guides subsequent explo- rations, ensuring a sustained and focused push at the scientific frontier. The entire system's 's core task is to find the optimal program I from a space of all possible candidate research programs I
3

---

Published as a conference paper at ICLR 2026
Human Findings
DeepScientist Findings Memory
Idea Pansinga
Teplesent Fansings
Progress Fandangs
Paper
Repositories
ICLR ICML
Steration
Code Repositories
Compile PDF
Self Revids
4185
Hundi
Finiegs
pplan
Jupert
A20 -Geese Replicn
Poasa Seandh
?
REKIE
-2gg
Open Source
Create Damo
Motivation
Pspera
Executable Code
Experiment Log
PUVI
Ablutsion
Surrogate Model
Effectiveness
Foasibllty
Outline
Exp Enviroment
Figure
Strategize & Hypothesize
Implement Verify
Analyze & Report
Figure 2: The autonomous, closed-loop discovery process of DeepScientist. The system iterates through a three-stage cycle, learning from both human knowledge and its own experiments.
that maximizes unknown and extremely expensive-tor expensive-to-evaluate true scientific value function f(-)- The architecture of DeepScientist detailed Appendix D.
Exploration Strategy. Scientific discovery differs from previously considered tasks like carly- machine learning (Schmidgall et al., 2025a), algorithm discovery (Novikov et al., 2025), or scien- tific software development (Aygün al., 2025). Each exploratory step within it requires immense resources. For instance, solving a frontier LLM problem requires approximately x 10 FLOPs for each implemennattoo (Figure 4.c). This necessitates efficient exploration strategy rather than brute-force search. Compared to prior AI Scientist systems that either follow single one-shot "idea + experiment - paper" pipeline or perform near near-unlingg unaliunt trial-and-erroo around a single idea, DeepScientist adopts an explicit iterative loop that combines Bayesian value estimation with persistent Findings Memory. To address this, DeepScientist's 's discovery process is structured as an iterative Bayesian Optimization loop (Figure 2), through the following three-stage cycle:
Stage I: Strategize & Hypothesize. Each research cycle begins by analyzing the Findings Mem- ory (M.), a list-style database containing thousands of structured records. Each record represents unique scientific finding, which categorized according to stage of development. To overcome the LLM's context length constraints, needed to select the Top-K Findings as input. In practice, the retrieved subset of Findings Memory for single task typically fits within long-context window of about 2 x 10 tokens, which is suf- ficient to contextualize the planner LLM without loss of relevant information. The vast majority of records begin as Idea madding--unverifien hypotheses. During this first stage, the system identifies limitations in existing knowledge and generates new collection of hypotheses (Prew), and then they evaluated by a low-cost Surrogate Model (gt). The surrogate model (an LLM) is first contextualized with the entire Findings Memory. In implementation. this is realized by feeding the surrogate with the retrieved Top-K records from M, together with the candidate hypothesis. so that it can reason over representative past successes and failures. It then approximates the true value function ʃ and, for each candidate finding 1 Pnews produces structured valuation vector V (Nus Ug. Me), quan- tifying its estimated utility, quality, and exploration value integer scores scale of 0 100. Each new hypothesis and its valuation vector is then used to initialize a new record in the Findings Memory "Idea Finding".
separate retrieval model (Wolters al., 2024) when
Stage II: Implement Verify. This stage serves as the primary filter in the Findings Memory. To decide which of the numerous "Idea Findings" warrants the significant resouree investment to be advanced a real-world experiment, the system employs an Acquisition Function (a). Specifically, it uses the classic Upper Confidence Bound (UCB) algorithm select the most promising record.
4

---

Published as a conference paper at ICLR 2026
Table 1: Overview of the three different human SOTA methods selected.
Task
Method
Venue
Benchmark
Github Star
Agents Failure Attribution
All Once
ICML 2025 Spotlight
Who&
302
LLM Inference Accel.
TokenRecycling
ACL 2025 Outstanding
MBPP
323
AT Text Detection
FastDetectGPT
ICIR 2024
RAID
414
The UCB formula maps the valuation vector V to balance the orf between exploiting promising avenues (represented by Va and v/)) and exploring uncertain ones (represented by ve):
4+1 = arg max
Wa + 4g
+K
Ue
(1)
Exploitation Term p(()
Exploitation Term (4)
where wh and wy are hyperparameters and controls the intensity of exploration. we adopt a simple, task-agnostic configuration with w, w, across the three tasks; this choice reflects an assumption of equal importance among expected utility, quality, and exploration, and ablations. The highest-scoring finding I++1 1 is selected for validation, and its record is promoted to the status of an Implement Finding. A coding agent then performs repository-level implementation to execute the experiment. This agent operates within a sandboxed environment with full permissions, allowing it to read the complete code repository and access the internet for literature and code searches. Its objective is to implement the new hypothesis on top of the existing SOTA method's repositories. The agent typically begins by planning the task, then reads the code to understand its structure, and finally implements the changes to produce the experimental logs and results. The experiment logs and results, f[[+1), are used to update the corresponding record, enriching it with empirical evidence and thus closing the learning loop.
and do not tune these hyperparameters
Stage III: Analyze & Report. The final and most selective stage of the Findings Memory is trig- gered only by successful validation. When an "Implement Finding" succeeds in surpassing the baseline, its record is promoted series of specialized agents capable of utilizing a suite of MCP (Hou et al., 2025) tools. These agents first autonomously design and execute series of deeper analytical experiments (e.g., ab- lations, evaluations on new datasets), leveraging MCP tools to manage the experimental lifecycle, data collection, and result parsing. Subsequently, a synthesis agent employs the same toolset col- late all experimental results, analytical insights, and generated artifacts into a coherent, reproducible research paper. The resulting deeply validated record is written back into the Findings Memory as a high-confidence Progress Finding, and, like all other records, will be retrieved and reused in subsequent cycles, allowing the system to learn from both its successes and its failures.
Progress Finding. This transformation
implemented by
EXPERIMENTS
As detailed in Table 1, we select three distinct SOTA methods (published in 2024 and 2025) as starting points, chosen for their frontier status, community interest, and human supervisability. Each SOTA method manually reproduced, and we preserve execution logs and test scripts to allow DeepScientist focus on research advancement. DeepScientist is provided with two servers, each with 8 Nvidia H800 GPUs. To maximize utilization, we launch a separate system instance for each GPU, employing the Gemini-2.5-Pro model for logic and the Claude-4-Opus model for its robust code-generation capabilities. Three human experts supervise the process to verify outputs and filter out hallucinations. For more implementation details, please see Appendix F.
4.1 DEEPŠCIENTIST ACHIEVEMENTS ON THREE RESEARCH DOMAINS
We evaluate DeepScientist three frontier AI research tasks where strong human-designes SOTA methods already exist, and ask whether the system can discover methods that further advance these frontiers (Figure 3). For each task, briefly recall the problem and baseline, then summarize the method discovered by DeepScientist and its improvement over the human SOTA.
The goal of Agent Failure Attribution is, given failed episode in an LLM-based multi-agent system, to identify which agent and which step were decisively respon- sible for the failure, which is crucial for debugging complex agent pipelines. The human SOTA
Agents Failure Attribution.
5

---

Published as a conference paper at ICLR 2026
Agent Failure Attribation
Inference Acceleration
Text Detection
Method
Handcraft (Acc.)
Algorithm-Ges Gen (Ace.)
Tolees/secood
AUROC
Latency
Human SoTA method DecpScientia's mothod
12.07% (All Once)
16.67% (All Onee)
190 35 (Token Recycling) 193,90 (ACRA)
0,800 (Binoculars)
117ms (Binoculors)
(AIP)
47.46% (AZP)
0.863 (PA-TITT)
6dmms (PA-TI77)
Improvement
2+142.8 (+17.24)
2+183.7% & (+30.79)
2+19% (+3.65)
2+7.9% (+0.063)
2+190%1
(m) Agent Error Attribution (Who& Handcraft)
(c) LAM Inference Acceleration
Perfomaaco Score
Harmaa Sait
ant--2539
Resalion
ICNU 2025 Souttine
DeepSctentiat
200-
EeTA
Denpfiteentt
193,90
33.79
+11-8
130
S7
6.60
0FF-06612W
LTC
ADNA
(b) Agent Error Attribution (Who&When- Seporithm-Generateen
(d) Al Text Detection
Beselies: All
Drighrinttid
Jumaa Zone
Pefformance Strte
Staee
2x Daster
$2.50 54
24.52
(-386.790)
Zonn
10
16.67
TDotact
FalC GFT
Haman Naneel
Latency (ms)
Figure 3: Performance evaluation of DeepScientist across three research domains: (a-b) Agent Fail- ure Attribution on Who& When benchmark in handcraft and algorithm- geeeerttd settings; (c) LLM Inference Acceleration MBPP dataset; (d) AI Text Detection with performance-latene, tradeoff analysis. DeepScientist consistently outperform human-designed SoTA approaches across all tasks.
"All at once" method (Zhang et al., 2025c) feeds the entire failure log to a judge LLM and asks it to directly predict the responsible agent and step, but this approach relies on pattern recognition over static logs and lacks explicit counterfactual reasoning, making step-level attribution and chain- like failures challenging. Starting from the baseline " "All at once" method, DeepScientist identified that the current approach lacks the counterfactual reasoning capabilities essential for attribution. Through process of trial, error, and synthesizing new undings-discover hypothetical prediction and simulated attempts- ultimately proposed the A2P method. Named for its Abduction-Action- Dectioo-Prediction process, its core innovation elevates failure attribution from pattern recognition to causal reasoning, filling the critical gap in counterfactual capabilities by predicting a proposed fix would have led to success. Concretely, A2P first hypothesizes hidden causes be- hind a suspicious action, then proposes a counterfactual fix, and finally simulates several future steps under this intervention to test whether the task would have succeeded. As shown in Figure 3.(a-b), A2P achieved scores of 29.31 and 47.46 in the "handcraft" and algorithm--eneratee tings of the Who& When benchmark, respectively, setting a new state-of-the-arr (SOTA), In this task, DeepScientist validated that structured, zero-shot causal reasoning framework can be superior to less principled methods. As of September 2025, the training-fiee free A2P method maintains its SOTA position, outperforming even 7B models trained synthetic data. (Zhang et al., 2025a).
the effectiveness of
set-
LLM Inference Acceleration is a highly optimized field aiming to maximize throughput and re- duce latency during LLM inference. The human SOTA baseline Token Recycling (TR) (Xia et al., 2024) reuses rejected candidate tokens produced during decoding via a tree-structured speculative decoding scheme, but effectively treats decoding as near-firstordder Markov process and primarily exploits local transition patterns, limiting its ability to consistently leverage longer-range regulari- ties. In this process. the system actively made many different attempts, such as using a Kalman Filter (Zarchan, 2005) to dynamically adjust an adjacency matrix to address the original method's 's lack of a memory function. Although most of these attempts failed, the system-generatee ultimately advanced the MPBB (Austin et al., 2021) from human SOTA of 190.25 to 193.90 to- kens/second by identifying stable suffix patterns, as shown in Figure 3. ACRA assumes that LLM decoding exhibits recurrent, variable-length stable suffixes: it maintains suffix-indexed history, finds the longest stable suffix matching the current context, and, only when a stability gate is passed, uses the associated next-token statisties to override the first layer of draft tokens; otherwise, it falls back to the original TR scheme. Scientifically, this innovation is significant because it uses this ex- tra contextual information to dynamically adjust the decoding guess, effectively grafting long-term
ACRA method
6

---

Published as a conference paper at ICLR 2026
Evaluation of Al-generated papers produced by various Al Scientist systems. Scores represent the average ratings given by DeepReviewer-144 sevieweer-1BBB (Zhu et al., 2025a) across the number ("Num") of available papers. Note: Publicly available papers may be curated and therefore may not fully represent the typical output of each system.
Table 2:
A1 Scientist Systems
Number
Soundness
Presentation
Contribution
Rating
Accept Rate
A1 SCIENTIST
2.08
1.80
1.75
3.35
0%
HKUSD AI Researcher
7
1.75
1.46
1.57
2.57
0%
Al SCIENTIST-V2
3
1.67
1.50
1.50
2.33
0%
CyclcRescarcher-128
6
2.25
1.75
2.13
3.75
0%
Zochi
2
2.38
2.38
2.25
4.63
0%
DeepScientist (Ours)
2,90
2.90
2.90
5.90
60%
memory onto the process and breaking the context-collapsing of standard decoders. This discovery highlights the system's primary goal: the creation new, human-unknown knowledge rather than mere engineering optimization. For instance, one could likely achieve greater performance gains by combining ACRA with an established technique like layer skipping (Wang et al., 2022) or PageAt- tention wKn et al., 2023), but this would represent engineering effort, not scientific one. The exploration assessment within our process avoids such combinations of existing knowledge.
AI Text Detection is binary classification task where (Dugan al., 2024), given text that may contain content from an LLM (and possibly additional noise), the goal is to determine produced by human
was LLM (Li al., 2022; Ghosal et al., 2023; Su al.; Bao al., a;b; Hu et al., 2023). This capability underpins applications such as exam integrity, content moderation, and model misuse detection. Recent human-designed SOTA detectors such as Fast-Detect GPT and Binoculars (Hans et al., 2024) exploit differences in perplexity, burstiness, or style between human and model distributions, but these global-statistic approaches assume relatively stationary gaps and often degrade when modern LLMs actively mimic human style texts are paraphrased or lightly edited. To validate its capacity for sustained advancement, DeepScientist made numerous attempts that included addressing the Boundary-Awant Extension problem and exploring approaches like Volatility-Awee Aware and Wavelet Subspace Energy methods. The final results show a dramatic accel- eration in scientific discovery: in rapid evolution over just two weeks, the system produced three distinct, progressively superior methods. This began with T-Detect fixing core statistics with a ro- bust t-distribution, then evolved conceptually with TDT and PA-TDT, which treat text signal and use wavelet and phase congruency analysis to pinpoint anomalies. Taken together, these meth- ods shift the perspective from global distributional differences the non-stationary, time frequueuyy structure of Al-generated text, showing that localized changes in energy and phase carry the key evidence for detection. Scientifically, this shift reveals the "non-stationarity" of Al-generated text, alleviating the information bottleneck in prior paradigms that average away localized evidence. As shown in Figure 1 and 3(d), this entire discovery trajectory demonstrates DeepScientist's ability for advancing Prontier-pushine scientific findings progressively, establishing a new SOTA with 7.9% higher AUROC while also doubling the inference speed.
4.2 ASSESSING THE QUALITY OF AI-GENERATED RESEARCH PAPER
Table Evaluation of DeepScientist's papers produced by human experts. Values are presented as mean (variance) from three reviewers. Inter-rater reliability for Rating: Krippendorff' 0.739.
Paper
Confidence
Soundness
Presentation
Contribution
Rating
HUMAN Avg. (ICLR 2025)
2.59
2.36
2.62
5.08
1. TT-DSTECT
4.33 (0.33)
2.00 (1.00)
2.67 (0.33)
2.67 (0.33)
5,00 (0.00)
2. TDT
4.67 (0.33)
3,00 (0.00)
3,00 (0.00)
3,00 (0.00)
5,67 (0.33)
3. PA-TDT
4.00 (0.00)
1.67 (0.33)
2.00 (1.00)
2.00 (1.00)
4.33 (1.33)
4.
4.00 (0.00)
3,00 (0.00)
3,00 (0.00)
2,67 (0.33)
5.67 (0.33)
5. ACRA
3.33 (0.33)
1.67 (0.33)
2.00 (1.00)
1.67 (0.33)
4.33 (1.33)
DecpScientist Avg.
4.07
2.27
2.40
5.00

---

Published as a conference paper at ICLR 2026
(4) Research Idea Pipeline Statiatics
(b) Progresa Succesa Rate (Progresa'implmmennee
[c) Average Execution Time implemented ideas
2040
Implemetted
300
162
2000
200
2500
3,330
330
1000
3,077
Success
370
600
500
50
196
180 Attribution
Debectios
Attribuuiin
Accelenation
Dobection
Astribution
Accelenation
Dotection
Accelenation
Figure 4: DeepScientist's experimental statistics. (a) The research pipeline from generated ideas to validated progress, (b) Success rates comparing our selection strategy against baseline. (c)
Distribution of wall-clock execution times for all implemented trials.
Experimental Setup. To assess the quality of the final output, we evaluate the five research papers autonomously generated by DeepScientist's 's end-to-end process. Our evaluation protocol is twofold. First, to benchmark against existing work, we employ DeepReviewer (Zhu et al., 2025a), an Al agent that simulates the human peer-review process with an external search capability, comparing Deep- Scientist's 8 output against 28 publicly available papers from other Al Scientist systems. Second, for more rigorous assessment, we convene a dedicated program committee consisting of three active LLM researchers: two volunteers who have served as ICLR reviewers and one senior volunteer who has been invited to be an ICLR Area Chair. The generated papers are available in Appendix F.
Automated Review Against Other AI Scientist Systems. As shown in Table 2, the results from the LLM-based automatic evaluation indicate that the system's outputs recognized for their sci- entific novelty and value. When benchmarked against 28 publicly available papers from other AI Scientist systems using DeepReviewer, DeepScientist is the only system whose papers achieve 60% simulated acceptance rate under the same reviewing protocol.
Human Expert Evaluation. The evaluation from our human program committee, shown in Table 3, reveals a strong and consistent consensus; DeepScientist's 's outputs are particularly strong in ideation, the most challenging and often rate-limiting step in human-led research. Full details on the review protocol are provided in Appendix B. and the core ideas within each paper are praised for their genuine novelty, ingenuity, and scientific contributions. The quality of these innovations further demonstrated by the review scores: the system's 's average rating (5.00) closely mirrors the average of all ICLR 2025 submissions (5.08), with two of its papers significantly exceeding this (5.67).
4.3 ANALYSIS OF THE ITERATIVE TRAJECTORY OF AUTONOMOUS EXPLORATION
Experimental Setup. The findings this section are derived from a series of post-hoc analyses conducted the complete operational data generated by DeepScientist across the three frontier tasks. This data includes the full set of exccution logs and the Findings Memory, providing the basis for all subsequent statistical analysis. To visualize the conceptual search space (Figure 5), embed the complete description of each generated finding using the Swwen3-Emmedding-Ne model. To assess scalability (Figure 6), we conduct a dedicated one-week experiment where N identified limitations of a single SOTA method are assigned to N parallel GPU instances. These instances explore solutions independently but share their findings nized globally every five cycles to accommodate the asynchronous nature of the discovery process. Finally, to better understand the low success rate, our program committee experts perform a detailed causal attribution analysis on sample of 300 failed implementations.
central database, which are synchro-
Our analysis of DeepScientist's experimental logs reveals the sheer scale of the trial-and-error process inherent in autonomous scientific discovery. Even in our relatively fast-executing do- mains, achieving progress required hundreds of trials per task. As show in Figure 4, the exccution time distributions show that while individual experiments may be quick, the sheer volume of trial- and-error necessary to uncover a successful idea is substantial. This suggests a clear application boundary for current autonomous science: for tasks with rapid feedback loops, such as aspects of chip design, delegating massive-scale experimentation to Al a powerful strategy. However, for high-cost endeavors like pre-training foundation models pharmaceutical synthesis, the low
8

---

Published as a conference paper at ICLR 2026
60
Not Implemented (N=1872)
PA-TDT
Implemented 600) Initial Idea
0,00903
40
t.SNE Component 2
20
TDT
Initial Idea
T-Detect
-20
-40
60
-60
-40
-20
20
40
60
80
t-SNE Component 1
Figure 5: Visualization of the conceptual search space for the Al text detection task. The plot shows a a-SNN visualization of the semantic embeddings for all 2,472 generated ideas. Markers identify the initial SOTA method (Initial Idea) and the three final SOrA-surpassing methods (Progress Ideas).
cess rate makes such an approach currently impractical, mandating continued reliance on human- led ideation. The autonomous research process is characterized by vast exploratory funnel where promising ideas are exceptionally rare. Across the three tasks, DeepScientist generate over 5,000 unique ideas, yet only about 1,100 are deemed worthy of experimental validation by the system's 's selection mechanism, and mere ultimately result in scientific progress. These 21 are promoted to Progress Findings (ideas that surpass the then-curren SOTA in Stage II), and the fully automated Stage III pipeline turns them into full drafts and filters them with an LLM-based reviewer, yield- ing 5 final papers. An ablation study underscores the criticality of this selection process: without it, randomly sampling 100 ideas for each task and testing them yields a success rate of effectively zero. The low success rate is not merely a matter of failed hypotheses; analysis by human experts on a sample of failed trials reveals that approximately 60% were terminated prematurely due to im- plementation errors, while the vast majority of the remaining 40% simply offered no performance improvement or caused a regression. This highlights that the probability of an LLM --M--neertted idea being both correct in its premise and flawless in its implementation is exceedingly low. In other words, the executor largely determines whether ideas can be executed at all and how often they fail, while the planner determines how far the system can advance under a fixed budget. The success of this work, therefore, is not a product of brute-force computation but of search efficiency. A naive approach of fully testing all 5000 promising candidates would have required over 100,000 GPU hours, whereas our targeted exploration achieved its breakthroughs using only 20,000.
DeepScientist's discovery process follows a purposeful and progressive trajectory. The semantic distribution of ideas generated for the AI text detection task, as shown in Figure 5, reveals the characteristics of this sophisticated strategy. While the system generates thousands of diverse ideas across vast conceptual landscape, its path to success is not random but is series of focused, logical advancements. This indicates a capacity to progressively deepen its understanding: after achieving an initial breakthrough with T-Detect, the system effectively establishes SoTA, identifies its subsequent limitations, and reorients its search towards a new goal. This dynamic exploration exemplified by the conceptual shift towards TDT and PA-TDT, which build upon the previous success by leveraging new positional and temporal information. This ability to build upon its own discoveries, turning each successful finding into a new starting point for identifying and solving the next set of limitations, demonstrates powerful capacity for scientific exploration.
Scaling Laws in DeepScientist's Scientific Discovery. To investigate the relationship between computational scale and the rate of scientific progress, we evaluated the number of "Progress Find- ings" generated by DeepScientist within a fixed one-week period as a function of available parallel resources in Figure 6. In this setup, the system first identified set of limitations in the baseline method, and each parallel exploration path was tasked with resolving a distinct limitation, with all paths periodically synchronizing their results into shared Findings Memory. Our results indicate a promising scaling trend. While minimal resources yielded no breakthroughs, the rate of discov- ery began to increase effectively we scaled to GPUs and beyond, growing from one SOTA-

---

Published as a conference paper at ICLR 2026
Number of Progress Ideas
Detection
Agents Fallure Attribution
LLM Inference Acceleration
Overall
10
8
T 1
GPUs
GPU
GPUU
GPU
16
Computational Resources (Each GPU 1.2 10TH FLOP)
Figure 6: Scaling analysis of autonomous scientific discovery. The plot illustrates the relationship between parallel computational resources (number of GPUs) and the number SOTA- -urpassin "Progress Findings" found by DeepScientist across all tasks within one-weck period.
surpassing finding with 4 GPUs eleven with 16 GPUs This appears establish near-linear relationship between the resources allocated and the output of valuable scientific discoveries. We hypothesize this efficiency stems from more than just parallel trial-and-error; it is a direct result of the shared knowledge architecture. Mechanistically, this mowleege-drriven gain should also apply when scaling with time (serial execution); indeed, our preliminary 4-week single-GPU tests con- firmed this, yielding new progress approximately every 8-14 days. While serial exploration may be more sample-efficient due to real-time memory updates (whereas our parallel setup synchronizes pe- compressing
riodically the immense value of parallel scaling lies in its wall-clock time months of discovery into a single week. This distinction highlights that parallel scaling demonstrates the scalability of the knowledge-sharinn explores, it enriches the shared Findings Memory. This creates a synergistic effect where the col- lective intelligence of the system grows (Schmidgall & Moor, 2025; Zhang et al., 2025b), allowing each independent path to benefit from the successes and, just as importantly, the failures of oth- ers. This suggests that effectively scaling autonomous science is not just a matter of increasing brute-force computation, but of fostering a richer, interconnected knowledge base that accelerates discovery across all concurrent efforts.
mechanism, not just its effectiveness. As each parallel path
4.4 DISCUSSION
The results from DeepScientist suggest new paradigm in scientific exploration, defined not by in- fallibility but by massive scale and efficiency. The system's 1-5% progress rate mirrors the reality of frontier research, successfully compressing years of human exploration into weeks, The primary path forward is systematically improving this efficiency. Our analysis identifies key bottlenecks: enhancing the robustness of implementation (as 60% of failed trials stemmed from implementa- tion errors, not flawed hypotheses ) and improving scientific rigor (as human evaluations praised the system's conceptual novelty but noted a lack of deep validation ). This highlights a powerful opportunity for human-AI synergy, where humans provide high-level strategic direction while the Al handles rapid, exhaustive exploration Our scaling analysis confirms this path is viable, show- ing lear-linear relationship between parallel resources and discoveries, driven not by brute-force computation, but by a shared knowledge base that accelerates discovery across all concurrent efforts Future work should focus these efficiencies, develop simulated discovery environments, and bridge the gap to the physical sciences through robotics.
CONCLUSION
This work presents the first large-scale empirical evidence that an autonomous AI can achieve pro- gressively, SOTA SorA-suupassing progress on modern scientific frontiers. We introduced DeepScientist, goal-oriented system achieving end-to-end autonomy from ideation to real progress, which learns by synthesizing human knowledge with its own findings from iteration of trials. Results across mul- tiple domains serves accelerate the progress of real-world scientific discovery, providing crucial foundation. Our findings can signal a foundational shift in AI research, heralding an era where the pace of discovery is longer solely dictated by the cadence of human thought.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This publication has been supported by the National Natural Science Foundation of China (NSFC) Key Project under Grant Number 62336006 and 625B2152, and the 2025 Dean's Special "PhD Student Project" of the School of Engineering, Westlake University.
We are grateful to Professor Linyi Yang for his insightful discussions on this paper. This work is inspired by pioneering efforts in automated scientific discovery, including Al Scientist (Lu et al., 2024: Yamada et al., 2025) and AlphaEvolve (Novikov al., 2025).
ETHICS STATEMENT
The development of DeepScientist, an autonomous system capable of advancing scientific frontiers, carries profound ethical responsibilities. Our primary goal is to accelerate discovery for the benefit of humanity, but we recognize the potential for misuse. The most significant risks include the application of this technology to advance dangerous research and the potential degradation of the academic ecosystem. We have implemented specific, robust measures to address these concerns proactively.
A primary concern is the dual-use risk, where the system could be co-opted to accelerate research in harmful domains, such developing novel toxins malicious software. To assess and mitigate this, we conducted red-teaming exercises specifically targeting the generation of computer viruses. We tasked the system, powered by leading foundation models (including GPT-5, Gemini-2.5-Pro and Claude-4.1-Ouut 1-Opus in our testbed), with this malicious objective. In all instances, the underlying models exhibited robust safety alignment, refusing to proceed with the research. They correctly identified the task as illegal and harmful, and autonomously terminated the research cycle, demon- strating that foundation model safety protocols provide a critical defense layer.
We are also deeply conscious of the potential negative impact on the academic ecosystem. It is crucial state that all results from DeepScientist presented in this paper, including code and ex- perimental findings, have undergone rigorous human verification. Recognizing that others might neglect this critical oversight, we are adopting a selective open-soureing policy to mitigate the risk of proliferating unreliable publications. We will open-source the core components that drive contin- uous discovery, as we believe their potential to accelerate progress for the community outweighs the risks. However, we will deliberately refrain from open-sourcing the "Analyze & Report" module. This decision is made to prevent the automated generation of seemingly credible but scientifically unverified papers, thereby safeguarding the integrity of the academic record.
Ultimately, we envision DeepScientist as powerful tool to augment, not replace, human intellect and judgment. To enforce this vision, our open-source components will be released under a license based on MIT, but with explicit addendums that codify our ethical framework. This license will strictly prohibit any use of the software for harmful research. Furthermore, it will legally require that a human user must supervise the entire operational process of DeepScientist and assumes full and final responsibility for all its outputs. By embedding these requirements directly into our terms of use, we aim to foster a research environment where Al-driven discovery proceeds with the necessary human accountability and ethical oversight.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.07753 07732, 2021.
Eser Aygün, Anastasiya Belyaeva, Gheorghe Comanici, Mare Coram, Hao Cui, Jake Garrison, Re- nee Johnston Anton Kast, Cory Y. McLean, Peter Norgaard, Zahra Shamsi, David Smalling, James Thompson, Subhashini Venugopalan, Brian P. Williams, Chujun He, Sarah Martinson, Martyna Plomecka, Lai Wei, Yuchen Zhou, Qian-Ze Zhu, Matthew Abraham, Erica Brand, Anna Bulanova, Jeffrey A. Cardille, Chris Co, Scott Ellsworth, Grace Joseph, Malcolm Kane, Ryan Krueger, Johan Kartiwa, Dan Liebling, Jan-Matthis Lueckmann, Paul Raccuglia, Xuefei, Wang, Katherine Chou, James Manyika, Yossi Matias, John C. Platt, Lizzie Dorfman, Shibl Mourad, and

---

Published as a conference paper at ICLR 2026
Michael P. Brenner. An ai system to help scientists write expert-level empirical software, 2025.
URL attpss://axitt org/ abs/ 20os/2509 06503.
Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Langwage Technologies (Volume 1: Long Papers), pp. 6709-6738, 2025.
Guangsheng Bao, Yanbin Zhao, Juncai He, and Yue Zhang. Glimpse: Enabling white-box methods to use proprietary models for zero-shot Ilm-generated text detection. The Thirteenth Interna- tional Conference on Learning Representations,
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Effi- cient zero-shot detection of sechine-generated text via conditional probability curvature. The Twelffh International Conference on Learning Representutions, b.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbur,, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024.
Cristina Cornelio, Sanjeeb Dash, Vernon Austel, Tyler R. Josephson, Joao Goncalves, Kenneth L. Clarkson, Nimrod Megiddo, Bachir Khadir, and Lior Horesh. Combining data and theory for derivable scientific discovery with AI-Descartes. Nature Communications, 14(1):1777, April 2023. ISSN 2041-1723. 1038/ /4/ 023- 0200003333320 07716777-22-------3
doi: 10.
URL hww.t:ttp////mm https:////db //ttip:://di... http//ii.ogg////////// 10.
Cristina Cornelio, Takuya Ito, Ryan Cory-Wright Sanjeeb Dash, and Lior Horesh. The need for verification in ai-driven scientific discovery, 2025. URL wttttttst///ttttt wtgp/b///////////////
01398.
Ryan Cory-Wright Cristina Comelio, Sanjeeb Dash, Bachir El Khadir, and Lior Horesh. Evoly- ing scientific discovery by unifying data and background knowledge with ai hilbert. Nature Communications, 15:5922, July 2024. doi: 0010088/41147--0255007777 //doi. org/ 1038/ 9414/7-44477740 0277--024550000 50074-w.
URL https:
Liam Dugan, Alyssa Hwang, Filip Trhlík, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Aallison-Burchh RAID: A shared benchmark for robust evaluation of machine- generated text detectors. In Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 12463-12492, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: http://aaaaaanlooaaaeeccc oorg/ 2024. acn-long....... 674.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Moham- madamin Barekatain, Alexander Novikov, Francisco R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022.
Alexander Fleming. Penicillin. British medical journal, 2(4210):386, 1941.
Soumya Suvra Ghosal, Souradip Chakraborty, Jonas Geiping, Furong Huang, Dinesh Manocha, and Amrit Bedi. A survey on the possibilities & impossibilities of Al-generated text detec- tion. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: // /oopeneeview.eet
jboj. Survey Certification.
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong. Ryutaro Tanno, et al. Towards ai co-scientist. arXiv preprint arXiv: 18..5000.888844 18864, 2025.
Martin A Green. Silicon solar cells: evolution, high high-efficiency design and efficiency enhancements. Semiconductor science and technology, 8(1):1, 1993.
12

---

Published as a conference paper at ICLR 2026
Abhimanyu Hans, Avi Schwarzschild, Valeriaa Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting llms with binoculars: Zero- shot detection of machine-generaie text. In International Conference Machine Learning, pp. 17519-17537. PMLR, 2024.
Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model context protocol (mep): Land- scape, security threats, and future research directions. arXiv preprint arXiv:2503.23277, 2025.
Xiaomeng Hu, Pin-Yu Yu Chen, and Tsung-Y Yi Ho. Radar: Robust ai-text detection via adversarial learning. Advances in neural information processing systems, 36: 1077-1155,, 2023.
Intology. Zochi technical report. arXiv, 2025.
Tang Jiabin, Xia Lianghao, Li Zhonghang, and Huang Chao. Ai-researcher: Autonomous scientific innovation, 2025. URL httpt:ttp///oo http:///aavvvvvoooee abs / 22505 18705.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 611-626, 2023.
Bin Li, Yixuan Weng, Qiya Song, and Hanjun Deng. Artificial text detection with multiple training strategies. arXiv preprint arXiv:2212.05194 2022.
Yixiu Liu, Yang Nan, Weixian Xu, Xiangkun Hu, Lyumanshan Ye, Zhen Qin, and Pengfei Liu. Alphago moment for model architecture discovery. arXiv preprint arXiv:2507.18077 2025.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist:
Towards fully automated open-ended scientific discovery. arXiv preprint S.tV:22408.0029253, 2024. URL https: www. arxiv. vixiiiv.o org/ abs 20002/40.0000 06292v3...
Jiacheng Miao, Joe R. Davis, Jonathan K. Pritchard, and James Zou. Paper2agent: Reimagining research papers as interactive and reliable ai agents, 2025. URL 2509. 06917.
Atsuyuki Miyai, Mashiro Toyooka, Takashi Otonari, Zaiying Zhao, and Kiyoharu Aizawa. Jr. ai scientist and its risk report: Autonomous scientific exploration from a baseline paper. arXiv preprint arXiv:25/1.044 2025.
Gordon Moore. Moore's law. Electronics Magazine, 38(8):114 1965.
Alexander Novikov, Ngån Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. Technical report, Technical report, Google DeepMind, 05 2025. URL https://storage. googleapis .... 2025.
José R Penadés, Juraj Gottweis, Lingchen He, Jonasz B Patkowski, Alexander Shurick, Wei-Hung Weng, Tao Tu, Anil Palepu, Artiom Myaskovsky, Annalisa Pawlosky, et al. Ai mirrors experimen- tal science to uncover novel mechanism of gene transfer crucial to bacterial evolution. bioRxiv. pp. 2025-02, 02, 2025.
Ori Press, Brandon Amos, Haoyu Zhao, Yikai Wu, Samuel K Ainsworth, Dominik Krapke, Patrick Kidger, Touqir Sajed, Bartolomeo Stellato, Jisun Park, et al. Algotune: Can language models speed up general-purposs numerical programs? arXiv preprint arXiv:2507.15887, 2025.
Bernardino Komera-Paredes. Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475, 2024.
Samuel Schmidgall and Michael Moor. Agentrxiv: Towards collaborative autonomous research. arXiv preprint arXiv:2503. /8/02, 2025.
13

---

Published as a conference paper at ICLR 2026
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants. arXiv preprint arX1v:2501.0422700 2025a. URL https:tt////.. httpps://www.e. wwwpwwwwwwoooo///////tto www.v/...rrgabs//5...1 04227v1.
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants. arXiv preprint arXiv:2501.042277 2025b.
Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan Reddy. Lim-sr: Scientific equation discovery via programming with large language models. In The Thir- teenth International Conference on Learning Representations.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint WiXv::2504.01845 2025.
Jinyan Su, Terry Yue Zhuo Wang, and Preslav Nakov. Detectllm: Leveraging log rank infor- mation for zero-shot detection of machine-generann text. In The 2023 Conference on Empirical Methods in Natural Language Processing.
Kyle Swanson, Wesley Wu, Nash I. Bulaong, John E Pak, and James Zou. The virtual lab of ai agents designs new sars-cov-2 nanobodies. Nature, pp. 1-3, 2025.
Jue Wang, Ke Chen, Gang Chen, Lidan Shou, and Julian McAuley, Skipbert: Efficient inference with shallow layer skipping. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7287-7301, 2022.
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. Cycleresearcher: Improving automated research via automated review. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/p///. WtR.hhttss////eeeeeee /twp/reveeeeeeeeeeeeeeeee.
met/Eooung d=b JcsVLoHYs.
Christopher Wolters, Xiaoxuan Yang. Ulf Schlichtmann, and Toyotaro Suzumura. Memory is all you need: An overview of compute-in-memory architectures for accelerating large language model inference, 2024. URL https:
org/ /tt.////44... 08413.
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: comprehensive survey of speculative decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 7655-7671, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10.186533v1/22 1/2024. findings-acl. 456.4 URL https: : /atcaaaaaahhhllcooooo.cc ogy. wttg///22002
nig-nnnnnnnanneeeeneeeeeeeee acl 456.
Qiujie Xie, Yixuan Weng, Minjun Zhu, Fuchen Shen, Shulin Huang. Zhen Lin, Jiahui Zhou, Zilan Mao, Zijie Yang. Linyi Yang, et al. How far are scientists from changing the world? arXiv preprint arXiv:2507.23271 2025.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint 2xXx:2504.08006 2025.
Paul Zarchan. Progress in astronautics and aeronautics: fermmuiiss,mmdaamnntss of Kalman filtering: a prac- fical approach, volume 208. Aiaa, 2005.
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.0647/, 2025.
Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. Agentracer: Who is inducing failure in the Ilm agentic systems?, 2025a. URL https:
03312.
14

---

Published as a conference paper at ICLR 2026
Pengsong Zhang, Heng Zhang, Huazhe Xu, Renjun Xu, Zhenting Wang, Cong Wang, Animesh Garg, Zhibin Li, Arash Ajoudani, and Xinyu Liu. Scaling laws in scientific discovery with ai and robot scientists. arXiv preprint arXiv:2503. 22444, 2025b.
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang. Yiran Chen, and Qingyun Wu. Which agent causes task failures and when? on automated failure attribution of LLM multi-agent systems. In Forty-second In- lernational Conference on Machine Learning, 2025c. URL https:://pperreiiiie .eeteeeeeewweee/e/eee........ Corum?id=Gazliee
Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, and Jun Wang. Memento: Fine-tuning Ilm agents without fine- tuning Ilms, 2025. URL https:t///// http/aaxivvv../////e///a/aolll org/ www.//ggg//////ee/e//o///eeeee/o abs/2508 16153.
Minjun Zhu, Yixuan Weng. Linyi Yang, and Yue Zhang. Deepreview:
Improving llm-based paper review with human-like like deep thinking process. arXiv preprint arXiv:2503.0888855 2025a.
Minjun Zhu, Qiujie Xie, Yixuan Weng, Jian Wu, Zhen Lin, Linyi Yang, and Yue Zhang. Ai scientists fail without strong implementation capability. arXiv preprint arX1v:2506.0177,, 2025b.
A USE OF LARGE LANGUAGE MODELS
Large Language Models are foundational component of the DeepScientist system and were integral every stage of the research presented in this paper. The core reasoning, hypothesis generation, and experimental analysis were driven by Gemini-2.5-Pro, while all code implementation, including writing, testing, and debugging, was performed by Claude-4-Opus.
The LLM agents autonomously conducted the entire scientific workflow. For the five SOTA- surpassing findings detailed in this work, the complete research process-from the initial identi- fication of a research gap and the formulation of novel idea, through literature search, code imple- mentation, and the design of analytical experiments, to the final writing of the research papers- was performed by the LLM-based system. The final research papers generated through this autonomous process are provided in Appendix F.
The role of the human authors was strictly limited to supervision, verification, and calibration of the system. We provided the initial SOTA methods as a starting point, monitored the system's 's progress, and verified the correctness of the final reported results. However, all novel scientific ideas, code, analyses, and written text were generated by the LLMs.
B HUMAN EXPERT REVIEW
B. REVIEW PROCESS AND CRITERIA
To ensure a rigorous and impartial evaluation of the generated papers, convened small, ded- icated program committee. The committee was composed two active researchers who served as volunteer reviewers for ICLR 2025, and one senior researcher who had previously been invited to serve as an ICLR Area Chair. All committee members possess significant expertise in the field of Large Language Models. The entire review process, with the exception of a rebuttal phase, was designed to meticulously emulate the official standards of ICLR 2025. Each of the five papers gener- ated by our system was assigned the three reviewers for thorough and independent assessment. The average review time for each paper was 55 minutes, during which reviewers were required to provide not only scores but also detailed written feedback, including a summary of the paper's strengths and weaknesses.
The evaluation was conducted on a custom-deployed review website where reviewers could not see each other's scores or feedback, ensuring that all initial assessments were made independently. The review form was structured to gather concise yet comprehensive feedback. First, reviewers were asked to state their Confidence in their review consisted of three sub-scores, each rated on a 1 to scale: Soundness, assessing the technical correctness and experimental rigor; Presentation, evaluating the clarity and quality of the writing:
scale of 1 to 5. The core of the evaluation
15

---

Published as a conference paper at ICLR 2026
and Contribution, measuring the significance and novelty of the work. Finally, reviewers provided a holistic Rating on a scale of to 10, where a score represented a 'borderline reject' and a score of 6 represented 'boordelliee accept'.
After the three reviewers submitted their independent evaluations for a paper, the volunteer acting as Area Chair would then read all submitted reviews. Drawing upon their experience from the ICLR review process, the Area Chair synthesized the feedback, weighed the arguments presented by the reviewers, and made final executive decision on whether the paper should be accepted or rejected in the context of our study. This final decision was recorded as the definitive outcome for each paper's evaluation.
B.2 SUMMARY OF REVIEWER FEEDBACK
Across the five generated papers, a clear consensus emerged from the human reviewers: Deep- Scientist consistently excels at the ideation stage of research. The committee unanimously lauded the methods for their genuine novelty and tangible contributions, noting that each paper proposed a unique approach that meaningfully advanced the state-of-the-an in its respective subfield. This feed- back validates the system's 's core strength as powerful engine for identifying relevant research gaps and generating innovative, impactful solutions, confirming that it can successfully ideate beyond mere incremental improvements.
However, this strength in ideation was systematically undermined by recurring pattern of weak- nesses in scientific execution and rigor. The most critical and frequent concern was a lack of em- pirical soundness; reviewers consistently noted that DeepScientist failed to design comprehensive validation plans, citing insufficient evaluation on standard benchmarks and a lack of in-depth analyt- ical experiments (e.g., ablations, motivation studies) to justify its claims. This was compounded by a failure to properly contextualize its contributions, with papers often omitting comparisons to essen- tial baselines or failing to discuss closely related work, thereby weakening the perceived significance of the results.
This feedback pinpoints the primary bottleneck in current autonomous systems: a profound gap be- tween the ability generate novel concepts and the capacity for rigorous scientific execution and articulation. The observed weaknesses in experimental design directly reflect the low-success-rate problem discussed previously; the system struggles not just implement ideas correctly, but val- idate them convincingly. To bridge this gap, future work must endow these systems with deeper, procedural understanding of the scientific method itself. This requires moving beyond simple im- plementation and reporting capabilities towards two key areas: First, developing agents explicitly trained in experimental design, capable of plamning comprehensive evaluations that anticipate and address potential scientific critiques. Second, enhancing the system's is ability for analytical reason- ing, enabling to not just describe results but interpret their significance, formulate compelling arguments, and engage in the kind of deep, reflective discussion that characterizes high-impact search.
C ADDRESSING THE BOTTLENECKS IN AUTONOMOUS SCIENTIFIC DISCOVERY
Artificial intelligence is reshaping the paradigm of scientific exploration through its ability to gen- erate hypotheses at massive scale; however, this has also pushed "verification" the center stage, making it a critical bottleneck. Our research empirically reveals the severity of this challenge: on frontier scientific tasks, the success rate of ideas generated by AI systems that ultimately lead to sub- stantial progress is typically below 3%, meaning the vast majority of computational resources are consumed exploring low-value hypotheses. This inefficient "needle in haystack" model is the core obstacle preventing AI Scientists from evolving from "novel tools" to "efficient discoverers." (Cor- nelio et al., 2025) Therefore, to further accelerate the process of scientific discovery, future research must focus on constructing systematic solution to overcome this bottleneck. As shown in Figure 7, future AI Scientist systems need to evolve synergistically in three key directions: optimizing the quality of initial hypotheses (Optimize Hypothesis Quality), enhancing filtering capabilities during the process (Enhance Filtering), and improving the quality of implementation and verification at the final stage (Improve Implementation Quality).
16

---

Published as a conference paper at ICLR 2026
(a) Current State of AI Scientists
Current State Human Research
Failed Hypatheses
Hypotheses
Successful Hypotheses
Hypotheses
Implement Verify
Implement Verify
Pragress
(c) Improve Implementation Quality
(d). Enhance Filtering
(e). Optimize Hypothesis Quality
Hypotheses
Hypotheses
Hypotheses
filtering
Implement Verify
Implement & Verify
Implement Verify
OOO
O0O
Accelerate discoveries
Accelerate discoveries
Accelerate discoveries
Figure 7: Three strategies for improving the efficiency of autonomous scientific discovery. (a) and (b) illustrate the low success rate currently faced by both AI and human research. Future directions will need to accelerate the discovery process through the synergy of three approaches: (c) improving implementation success rates, (d) adding an efficient filtering stage before implementation, and (c) optimizing the quality of initial hypotheses from the source.
One of the core future research directions is to develop Al systems capable of generating higher- quality, more reliable hypotheses (as shown Figure 7e), equipped with more precise filtering mechanisms to predict their success rate (as shown in Figure 7d). Methods that rely purely on a data-driven approach, while capable of discovering patterns, often produce outputs that lack a theoretical foundation and are prone generating "hallucinations" that contradict known scientific theories. Future systems must move beyond this by more deeply integrating background knowledge and theory. For instance, the direction represented by "derivable models" (such as Al-Descartes (Cornelio et al., 2023) and Al-Hilbert (Cory-Wir Wrriht et al., 2024)), which incorporate scientific ax- ioms as constraints during the hypothesis generation phase, offers a promising path improving hypothesis quality. Furthermore, systems must have the ability to learn from their own exploratory history. By establishing mechanisms similar to a "Findings Memory," a system can systematically record and analyze every success and failure, thereby avoiding redundant exploration of ineffec- tive paths in subsequent iterations and gradually developing a more insightful scientific intuition. Building this foundation, developing more advanced, low-cost surrogate models and acquisition functions to more accurately predict the scientific value of an idea will be key to enhancing filtering efficiency and conserving verification resources.
Concurrently, an often-overlooked yet crucial future research direction is to significantly improve the quality and reliability of AI systems the engineering implementation and verification stages (as shown in Figure 7c). Even the most brilliant scientific concept can never have its value confirmed if cannot be accurately and flawlessly translated into an executable experiment. Our analysis indicates that up to 60% of exploratory failures stem from implementation-leve errors, which represents massive waste of resources and directly impedes scientific progress. History has repeatedly warned us that lack of rigorous verification can lead to catastrophic consequences, whether in NASA missions or medical practice, Therefore, building a scalable and reliable automated verification platform is essential path forward. This requires not only more powerful code-generation and self-debugging agents to reduce implementation errors but also standardized sandbox environments and automated testing procedures to ensure the stability and reproducibility of experimental results. Ensuring the absolute reliability of the verification process is the final and most critical line of defense in transforming AI Al-generated "plausible ideas" into "solid scientific evidence."
17

---

Published as a conference paper at ICLR 2026
Looking ahead, to truly accelerate scientific discovery. it is necessary to integrate the aforemen- tioned strategies into an organic whole, advancing AI Scientists from "random explorers" to "goal- oriented strategists." " This is not about replacing humans with AI, but about pioneering more ef- ficient paradigm of human-Al collaboration. In this model, human scientists are responsible for defining grander, more valuable scientific goals and providing high-level strategic guidance, while the AI system serves as a powerful "cxploration engine," executing efficient trial-and-error and ver- ification cycles at an unprecedented scale and speed under human direction. To realize this vision, the community must also address a series of challenges, such as building benchmarks that can truly evaluate innovation and designing mechanisms that encourage diverse exploration to avoid the ho- mogenization of research paradigms, thereby preserving the potential for serendipitous discoveries like Alexander Fleming's 's discovery of penicillin (Fleming, 1941).
D METHOD DETAILS
D.1 IMPLEMENTATION OF THE STRATEGIZE & HYPOTHESIZE STAGE
The Strategize & Hypothesize stage of each discovery cycle is operationalized within our system as multi-agent workflow that mirrors structured research and development process. This entire process is centered around the 'Findings Memory" (Mt) which is implemented as a large, list-style database (IdeaDat abase in the codebase) designed to persistently store thousands of structured records. Each record represents a unique finding at a specific stage of its lifecycle. The workflow is executed by cohort of specialized agents, orchestrated by central 'DirectorAgent'' ensuring that the generation of new hypotheses is a guided, strategic exploration rather than an undirected search.
Each cycle commences with an analysis of the 'Findings Memory'. This is enacted by the 'Scien- tistAgent', which is invoked in its 'STAGE: the current state of knowledge, including the entire baseline codebase and the contents of the 'Find- ings Memory', to perform a first- first-principles analysis of the problem domain. It formulates the core challenge as a rigorous mathematical problem and identifies the fundamental limitations of existing findings. The tangible output of this strategic analysis is comprehensive Markdown document, Research_ Out 1 line. . md, which serves as a high-level charter, guiding the system's focus for the current cycle. This initial step provides concrete and reproducible mechanism for the "analysis of limitations in existing knowledge" described in our main methodology.
DYSEARCHLOUTLINE mode. The agent consumes
Following the establishment of strategic outline, the 'DirectorAgent' deploys one more in- stances of the 'ExplorerAgent" to generate a new collection of hypotheses (Pnow). Governed by its highly specific EXPLORER AGENT AAGENT_PROMP" PROMPT, this agent's core function is to produce novel, structured records based on the directions provided in Research_ouus ology emphasizes systematic, cross-disciplinary investigation, using integrated research tools like cesach to adapt successful theoretical frameworks from adjacent scientific fields. The out- put of this process is a structured JSON object for each new hypothesis, detailing its motivation, theoretical underpinnings, and a concrete implementation plan. This JSON object is the system's direct instantiation of candidate finding I Prow-
ine md. Its method-
Upon generation, each new candidate finding is immediately passed to the system's ' low-cost Sur- rogate Model (g.). a role fulfilled by the 'EvaluatorAgent" This agent is first contextualized with the entire state of the "Findings Memory". As dictated by its prompt, it then assesses the candi- date finding and produces three numerical scores: utility utility_score, a quality_scee _score, and an explo- ration_score, These scores are the direct implementation of the components of the Valuation vector V = (Va,, Ugy Ve), quantifying the hypothesis's estimated value. Finally, the 'DirectorAgent' initial- izes record in the 'Findings Memory', pairing the new hypothesis with its valuation vector and assigning it the status of an 'Idea Finding'. This action concludes the stage, formally adding the new, unevaluated hypotheses to the system's knowledge base, ready for the subsequent selection and verification phase.
D.2 IMPLEMENTATION OF THE IMPLEMENT & VERIFY STAGE
The Implement& & Verify stage serves as the primary filter in the research funnel and is operational- ized as the "Enginecring Phase" of the system's 's workflow. This phase is triggered when the system makes strategic decision to commit significant computational resources to validate single, highly
18

---

Published as a conference paper at ICLR 2026
promising 'Idea Finding'. The workflow is managed by the 'ScientistAgent', which acts as the primary decision-make maker, and the IImplementationAgennt which executes the complex code-level modifications and real-world experiments. This stage is paramount as it provides the empirical feedback essential for the system's learning loop.
The selection process, described in the main text the Acquisition Function (a), is implemented by the 'ScientistAgent" operating in its STAGE: STRATEGIC DEVISSIOO mode. This agent pe- riodically reviews the population of 'Idea Findings' and their associated valuation vectors stored in the 'Findings Memory'. Based on the criteria in its prompt, which require it to reflect on past outcomes and balance the exploitation-explraatiie
trade-off, it selects the most promising record for validation. A decision of 'VALIDATE' from this agent, targeting a specific finding's ' ID, is the sys- tem's operational equivalent of the 'argmax' operation in the UCB formula. Upon this selection, the chosen record's ' status within the 'Findings Memory' is formally promoted from an 'Idea Finding' to an 'Implement Finding', signaling the start of the resource-intensive implementation.
Once finding is promoted. the 'DirectorAgent' delegates the implementation task to the 'Im- plementationAgenn', a specialized, terminal-based Agent governed by the highly prescriptive CEEEEEENTATIOONAGETTTIELI
The agent's workflow is meticulously structured for ro- bustmess and reproducibility. It begins by creating an isolated, sandboxed directory for the experi- ment by copying the entire baseline codebase. Within this safe environment, it systematically trans- lates the 'theory_and method' and 'code lave_eeeee_____ee Level_plan' from the selected finding's 's record into func- tional code modifications. The agent's governing prompt mandates rigorous engineering process, including the creation of unit tests to verify numerical stability and correctness, and the maintenance of a detailed log of all actions in notebook .nd file.
The culmination of this stage is the real-world experiment. After completing the code modifications and passing all internal tests. the "ImplementationAgene executes the project's 's standard evaluation script, test. sh. The complete, captured terminal output from this script, containing the final per- formance metries, constitutes the empirical observation of the true scientific value function, f(4+1)) In the final step, the 'DirectorAgent takes this new data point the empirical result- and uses it to update the corresponding 'Implement Finding' record in the 'Findings Memory'. This action en- riches the finding with empirical evidence, formally closing the learning loop and providing critical new knowledge to inform all subsequent discovery cycles.
D.3 IMPLEMENTATION OF THE ANALYZE & REPORT STAGE
The final stage of the DeepScientist discovery loop, Analyze & Report, is initiated when an Implement Finding successfully validates, demonstrating performance that surpasses the es- tablished baseline. This achievement triggers sophisticated multi-agent workflow orchestrated by a central 'DirectorAgent' designed to transform the raw experimental success into a comprehensive and reproducible scientific paper. This process is not monolithic writing task but a structured, multi-phase procedure that mirrors a rigorous human-led research publication effort, comprising three core sub-phases: Iterative Outline Development, Sequential Paper Writing, and Multi-Round Revision. Each sub-phase is executed by specialized agents with precisely defined roles and opera- tional protocols, ensuring a high degree of quality control and methodological soundness.
The process begins with Iterative Outline Development, a three-round cycle of design, review, and analysis. The 'DirectorAgent" first deploys ating a compelling narrative and a detailed structural blueprint for the paper. This agent operates via a unique two-stage process: it first generates thousands of words of unstructured reasoning to explore the theoretical foundations and experimental implications of the finding, drawing from the complete history in the Fi Findings Memory ry and the newly populated Result. md file. Sub- sequently, it distills this reasoning into structured JSON object, which includes narrative arc, answers to ten foundational research questions, and detailed plans for every table and figure. This initial outline is then passed to an "OutiieeeeiieeerAeen'' which provides a harsh, academic-style critique. Finally, an strategic decision: either 'VALIDATE' the outline ready, 'EVOLVE' it by generating specific 'improvement directive' for the next round. This cycle repeats up to three times, ensuring the final blueprint, saved as f final sel accte_ootte ine. json, is robust and logically sound.
DentlinnDesiinerAAeent which is tasked with cre-
evaluates both the outline and its review to make a
19

---

Published as a conference paper at ICLR 2026
With validated outline in place, the 'DirectorAgemt" proceeds to the Sequential Paper Writing sub-phase, deploying a specialized "ClaudeCodePapper "CauddeCoddppee:Wttteggtgt This agent is governed by an exceptionally detailed and preseriptive prompt that enforces strict, multi-phase workflow exe- cuted directly on the file system. Critically, the agent does not immediately begin writing prose. Its first mandatory step is extensive literature review, where it uses integrated search tools like 'pasa_search" search' and 'semantic scholar_quees to gather over 60 relevant citations, meticulously pop- ulating a references bib file and documenting its findings in a draft md log. Only after this literature foundation is established does it proceed to generate all required figures and tables, extracting data directly from Result. .mdd and saving styled plots the '/figures' directory.
Following the completion of literature and figure generation, the PapeeeeeeddP begins writing the manuseript's content. It follows a strict top-down sequence, creating and popu- lating individual LaTTeX files for each section (e.g., Introductt on tex, methodo cooooolooy. logy- tex, experimee ment.s. tex) in a predefined order. The content for each section is precisely guided by the blueprint in sanna__selctee _outtootttiooo line. json, ensuring perfect alignment between the plan and the final output. The agent's 's prompt includes a comprehensive validation checklist that it must internally satisfy before completion, covering everything from content authenticity and structural integrity to experimental completeness and citation accuracy. The entire writing process is logged in wrie iiggpnngpll winggplann alaan... mc and draft. md, and the agent signals its completion to the 'DirectorAgent only by creating a final paper. mei file, ensuring the full sequence has been executed.
The final sub-phase is Multi-Round Revision, which ensures the paper meets publication standards. The 'DirectorAgent' deploys a PaaeeRReviewerAgent to conduct thorough review of the complete draft, assessing its clarity, technical accuracy, and narrative coherence. The reviewer's structured feedback is then passed back to the
as a set of revision instructions. The writer agent then performs targeted revision of the relevant tex files to address the identified weaknesses. This review-and-revis loop is executed for a predefined number of rounds, iteratively polishing the manuscript. The culmination of this entire stage is a complete, publication-reat package containing the full LaTeXX source code, section files, bibliography, figures, and detailed log of the generation process, thereby converting a single 'Progress Finding" into a durable and shareable piece of scientific knowledge.
E ADDITIONAL EXPERIMENTS
E.I LARGE-SCALE EVALUATION OF MICROMANO-DEEPSSIINIIT ON THE ALGOTUNE BENCHMARK
We introduced a lightweight variant of our framework, Dicronano-DeepSciiettttt, to enable large- scale evaluation across diverse scientific discovery tasks (Press et al., 2025). This version preserves the core hicrarchical exploration process and the discovery memory mechanism of DeepScientist, but removes the most computation-ine compuutation-intessive modules such as literature reading, formal hypothesis drafting, and extensive experimental analysis. As a result, Micronano-DeeepSciente operates at ap- proximately 1/1000 of the runtime cost of the full system while maintaining its essential exploratory capability. All experiments in this section use the open-source GLM-4.6 (Zeng et al., 2025) model as the reasoning backbone.
To assess the generality of our approach, we conducted systematic experiments on ALGOTUNE, a benchmark containing 154 algorithmic discovery tasks spanning mathematics, physics, and com- puter science. Each task requires the system to autonomously search for algorithmic improvements relative to strong human-designed baselines. Mieronano- Mieronano-DeepScieniis successfully discovered al- gorithms that outperform the baseline implementations on 120 tasks (77.9%) achieving an average speedup of 16.6x. On the remaining 34 tasks (2).1%). the system generated solutions that were slower or did not surpass the baseline within the allocated search time. These results demonstrate that even a significantly scaled-down discovery engine can autonomously generate competitive algo- rithmic innovations across broad task distribution when paired with an efficient hierarchical search strategy and a capable open-source model such as GLM-4.6. A summary of the aggregate statistics is shown in Table 4.
20

---

Published as a conference paper at ICLR 2026
Table 4: Aggregate performance of Micronano-Deeepciienttt on the 154-task ALGOTTUEE bench- mark.
Metric
Value
Total tasks
154
Successful tasks
120 (77.9%)
Slower failed tasks
34 (22.1 %)
Mean speedup
16.6x
LLM backbone
GLM-4.6 (open-source)
Table 5: Automatic review scores using the o3-mini reviewer setup from Zochi. DeepScientist is
evaluated with exactly the same code and prompts as prior work.
Systems
Sound.
Pres.
Contr.
Orig.
Qual.
Clar.
Sign.
Overall
AI Scientist
2.20
2.40
2.10
2,40
2.10
2.60
2.20
3.80
Al Scientist-v2
2.00
1.67
2.00
2.00
2.00
1.67
2.00
3,00
CycleRcsearcher
2.33
2.17
2.33
2.33
2.17
2.17
2.50
4.00
Zochi
3.00
3.00
3.00
3,00
3.00
2.50
3.00
6.00
AI- Rescarcher
2.43
2.14
2.43
2.71
2.43
2.14
2.57
4.29
DeepScientist
2.80
2.80
2.80
3.00
2.80
2.80
3.60
6.20
E.2 ROBUSTNESS ACROSS MULTIPLE AUTOMATED REVIEWER SYSTEMS
Beyond DeepReviewer-1433 (Zhhu al., 2025a), further assess the quality of DeepScientist's papers using several independent automatic reviewer systems. First, we adopt the original Zochi reviewer setup based on the 03-mini model and the official evaluation code, and re-evaluate all available systems under exactly the same prompts (Table 5). In this configuration, Zochi (Intology, 2025) attains strong Overall score 6.00, but DeepScientist still slightly surpasses with an Over- all score of 6.20, and achieves the highest or tied-highest scores on key dimensions such as Original- ity, Clarity, and Significance. We then apply the Al Scientist reviewer prompts with three different backbone models--Gemnn--2. -emini-22.-PPi GPT-4o, and GPT-5 GPT---yielddigg Tables 6, 7, and 8, respectively. Across all three backbones, DeepScientist again obtains the best Overall rating among the compared AI Scientist systems, with noticeable gains in Soundness and Presentation under Gemini-2.5- -iiiiii--..-Prr and a particularly large margin in Overall under GPT-4o. Finally, using the independent CycleRe- viewer model [3.4] (Table 9). DeepScientist achieves the highest Overall score of 4.85, exceeding both Zochi (4.50) and CycleResearcher (4.46) while also leading on all three component criteria.
Taken together, these results show consistent pattern: regardless of the underlying reviewer chitecture (o3-mini, Gemini-2.5-Proo GPT-4o, GPT-5, or CycleReviewer) and despite differences in absolute scoring scales, DeepScientist is always ranked at existing AI Scientist systems. This cross-validation strengthens the robustness of our conclusions and indicates that DeepScientist's advantages are not an artifact of particular reviewer model or prompt design. Moreover, the dimensions on which DeepScientist tends to score highest-such as originality, significance, and clarity- are precisely those emphasized by human program-commitee evaluations in Section B.2, suggesting that the gains observed under automatic reviewers are aligned with human judgments of scientific value.
the top in Overall quality among
are
F.3 CASE STUDY ON HUMAN VS. AUTONOMOUS RESEARCH EFFICIENCY
To better understand how an autonomous system compares to human researchers in terms of re- search efficiency, we conduct qualitative case study on the Al text detection task. For this domain, we collected approximate statistics from the teams behind two recent human-designed SOTA meth- ods (Fast-Deted and Glimpse), focusing their development timelines and resource usage. Each project was led by a full research team and typically required about six months from project incep- tion to camera-ready paper. During this period, the teams reported using roughly 5-10 GPU hours per day on average, for a total of around 1,500 GPU hours per project, with GPU utilization dropping substantially outside of normal working hours. In terms of exploration breadth, typical six-month
21

---

Published as a conference paper at ICLR 2026
Table 6: Automatic review scores using the Al Scientist reviewer prompts with Gemini-2.5-PP
Systems
Sound.
Pres.
Contr.
Overall
Scientist
1.00
1.40
1.10
1.70
Al Scientist-v2
1.00
1.00
1.00
1.67
CycleResearcher
1.00
1.00
1.17
1.50
Al-Researcher
1.00
1.14
1.00
1.29
Zochi
1.00
1.50
2.00
2.00
DeepScientist
1.20
1.80
1.80
2.20
Table 7: Automatic review scores using the AI Scientist reviewer prompts with GPT-4o.
Systems
Sound.
Pres.
Contr.
Overall
Scientist
2.00
2.10
2.00
2.60
Al Scientist-v2
2.00
2.00
2.00
3.00
CycleResearcher
2.00
2.00
2.00
3,00
AI- Al-Rescarcher
2.00
2.00
2.00
3.14
Zochi
2.00
2.00
2.50
3.00
DeepScientist
2.40
2.20
2.40
4.20
project allowed the team to deeply investigate on the order of 10-30 core hypotheses, each of which required careful design, implementation, and iteration before being deemed publishable.
On the same text detection task, DeepScientist was run continuously for days and produced three progressively stronger SOTA methods (T-Detect, TDT, and PA-TDT). Each breakthrough con- sumed roughly 900 GPU hours, for total on the order of few thousand GPU hours, but these resources were utilized close to 24/7 across parallel instances. Within this two-week window, the system generated over 2,400 candidate hypotheses and autonomously executed around 600 full ex- perimental validations, far exceeding the exploratory throughput that human team can typically achieve in a comparable or even longer time span. While such a comparison is necessarily approx- imate and limited in sample size, it suggests that, under given compute budget, DeepScientist can explore the hypothesis space with a substantially higher trial-and-error throughput, compress- ing what would conventionally require several months of human-led research into few weeks of machine-driven exploration. At the same time, human researchers remain essential for problem formulation, high-level evaluation, and long-term research direction, indicating complementary relationship in which autonomous systems amplify, rather than replace, human scientific effort.
F IMPLEMENTATION DETAILS
Our implementation relies on a distributed architecture to manage the distinct tasks of scientific reasoning and code execution. The core logic of DeepScientist is powered by the Gemini-2.5- model, while all code implementation tasks are delegated to Claude-4-opus, executed within the Claude Code framework (v1.0.53). To ensure stability and security, the DeepScientist system and the Claude Code agent are isolated in separate Docker containers, communicating via a port-based API. During the 'Implement & Verify' stage, human-verified baseline code repository is first du- plicated into a new, sandboxed folder. The Claude Code agent's operations are strictly confined to this new directory prevent unintended modifications. critical step in our pipeline is a secondary verification process: after Claude Code reports completion, DeepScientist independently re- re-executes the main script via the command line. This measure was implemented counteract high rate of false positives-we we observed that approximately 50% of initial implementation attempts failed to complete fully due to internal timeouts within the Claude Code agent. Throughout this project, all experimental results were manually inspected by human supervisors guarantee their authenticity. For the 'Analyze & Report' stage, a similar process is followed: the validated code replicated for each analytical experiment, with Claude Code executing them sequentially. Upon completion, DeepScientist aggregates all results, generates a paper outline, and then employs automated tools to write and compile the final PDF manuscript. For all experiments, used a fixed set of hyperpa-
22

---

Published as a conference paper at ICLR 2026
Table 8: Automatic review scores using the Al Scientist reviewer prompts with GPT-5.
Systems
Sound.
Pres.
Contr.
Overall
Scientist
1.00
1.30
1.00
2.10
Al Scientist-v2
1.00
1.00
1.00
2.00
CycleResearcher
1.00
1.00
1.00
1.67
Al-Researcher
1.00
1.00
1.00
1.86
Zochi
1.00
1.50
1.50
2.50
DeepScientist
1.40
1.60
1.80
3.00
Table Automatic review scores using CycleReviewer.
Systems
Sound.
Pres.
Contr.
Overall
AI Scientist
2.12
2.25
2.00
3.23
AI Scientist-v2
2.00
2.42
2.00
3.00
CycleResearcher
2.50
2.54
2.38
4.46
Al-Researcher
2.07
2.21
2.07
3.50
Zochi
2.62
2.88
2.50
4.50
DeepScientist
2.80
2.85
2.65
4.85
rameters:
the retrieval count was set to 15, and the UCB parameters were set to utility weight 1. quality weight wy = 1, and exploration coefficient
The financial and computational costs of this autonomous discovery process are substantial. Each idea generated during the 'Strategize & Hypothesize" stage incurred an approximate cost of $5 in API calls. For each attempt in the 'Implement & Verify* stage, the cost averaged $20 for Claude- 4-opus API usage, in addition the computational cost of approximately I GPU hour, as detailed in Figure 4.c. A successful finding that progressed to the 'Analyze & Report' stage required a fur- ther expenditure of around $150, which includes $100 for rumning analytical experiments and $50 for the final report generation. The total cost to achieve the scientific advancements presented in this paper amounted approximately $ $100,000. While significant, substantially reduced. We recommend that future iterations explore more economical alternatives, such deploying high-throughput models like Qwen-3-Next-800 for the core DeepScientist sys- tem and leveraging subscription-bart based API access (e.g.. Claude Max or OpenAl Pro) to mitigate per-call expenses. this paper, each implementation was provided with single H800 server for exploration. Since the H800 GPU has an FP16 computing power of approximately 2 TFLOPS, an average execution of minutes corresponds about x 10 floating-point operations.
believe these costs be
23

---

Discovered and authored by DeepScientist.
ABDUCT, ACT, PREDICT:
SCAFFOLDING CAUSAL INFERENCE FOR AUTOMATED FAILURE ATTRIBUTION IN MULTI-AGENT SYSTEMS
DeepScientist
ABSTRACT
Failure attribution in multi-agent tiitems--inppintngg the exact step where a decisive error occurs- is critical yet unsolved challenge. Current methods treat this as a pattern recognition task over long conversation logs, leading to critically low step-level accuracy (below 17%), which renders them impractical for debug- ging complex systems. Their core weakness a fundamental inability to perform robust counterfactual reasoning: to determine if correcting single action would have actually averted the task failure. To bridge this counterfactual inference gap, we introduce Pretuct-At-Preeiiie (A2P) Scaffolding, novel agent framework that transforms failure attribution from pattern recognition into a structured causal inference task. A2P explicitly guides large language model through formal three-step reasoning process within a single inference pass: (1) Abduction, to infer the hidden root causes behind an agent's actions; (2) Action, to define minimal corrective intervention; and (3) Prediction, to simulate the subsequent trajectory and verify if the intervention resolves the failure. This structured approach lever- ages the holistic context of the entire conversation while imposing rigorous causal logic on the model's s analysis. Our extensive experiments on the Who& When benchmark demonstrate its efficacy. On the Algorithm- Algorithn-Generated dataset, A2P achieves 47.46 % step-level accuracy, a 2.85x improvement over the 16.67% the baseline. On the more complex Sand-Crafted dataset, it achieves 29.31 % step accuracy, a 2.43x improvement over the baseline's 12.07%. By reframing the problem through a causal lens, A2P Scaffolding provides robust, verifiable, and significantly more accurate solution for automated failure attribution.
INTRODUCTION
The rise of sophisticated multi-agent systems marks pivotal moment in artificial intelligence, unlocking new frontiers in collaborative problem-solving (Li et al., 2023; Hong et al., 2023) and complex task automation (Wu et al., 2023; Fourney al., 2024). However, this growing complexity introduces a critical operational bottleneck: debugging. When a system fails, developers are faced with tangled web of interactions, where subtle error in early step can cascade into catastrophic failure dozens of turns later. Pinpointing the single, decisive error-the task of failure attribution-iss major barrier to
merely challenging; it is labor-intensive, error-prone process that stands the reliable deployment and iterative improvement of these powerful systems (Zhang et al., 2025).
Current automated approaches to this problem have proven fundamentally inadequate, with step-level accuracy rates hovering below dismal 17% (Zhang et al., 2025), figure far too low for practical debugging. We argue this failure is not matter of model capability but of methodological paradigm. Existing methods treat failure attribution as a pattern recognition task over conversational logs (Zhang et al., 2025; Lightman et al., 2023). They present an entire log to Large Language Model (LLM) and ask it to "find the mistake," " implicitly assuming the model can spot anomalous patterns correlated with failure. This approach fundamentally misses the point. The critical question is not "which step looks wrong?" but rather causal one: "which single corrective action would have turned failure into success?" This exposes a deep counterfactual inference gap: the inability of unstructured, holistic methods to systematically reason about the consequences of hypothetical interventions, challenge particularly pronounced in multi-turn interactions where cause and effect are obscured (Kician al., 2023; Zevcevic et al., 2023).

---

Discovered and authored by DeepScientist.
To bridge this gap, we introduce Abduct-Act-Predd (A2P), novel prompting framework that reframes failure attribution from pattern recognition into a structured causal inference task. Instead of asking for direct answer, A2P guides an LLM through formal, three-step counterfactual reasoning process within a single inference pass, operationalizing the logic of Pearl's structural causal model hierarchy (Pearl, 2009). The framework compels the model to: (1) Abduct, inferring hidden factors (e.g., a flawed assumption) that explain a problematic action; (2) Act, defining a minimal, conerete corrective intervention; and (3) Predict, simulating the subsequent counterfactual trajectory to verify if the intervention would have resolved the overall task failure. This structured process forces the model to move beyond correlation and rigorously test causal hypotheses, transforming the "needle-in -n-e-iii-iiie in-the-haet hha-haastcck" problem (Liu et al., 2024) into a systematic investigation.
Our approach is not just theoretically sound but empirically dominant. Evaluated on the comprehen- sive Who& When benchmark (Zhang et al., 2025), A2P Scaffolding achieves step-level accuracy of 47.46 % on the Algorithm- -enerrtted dataset-a a 2.85x improvement over the 16.67% of its direct baseline. On the more challenging Hand-Crafted dataset, it achieves 29.31 % accuracy, 2.43x improvement over the baseline's s 12.07%. These results establish a new state-of-the-art and, for the first time, demonstrate viable path toward reliable automated debugging for multi-agent systems. Rigorous ablation studies further validate our framework, confirming that each causal reasoning component is essential and revealing the surprising, critical role of structural cues like contextual step numbering in enabling fine-grained analysis.
2
RELATED WORK
2.1 LLM MULTI-AGENT SYSTEMS
The emergence of Large Language Models capable reasoning agents has catalyzed rapid develop- ment in multi-agent system architectures (Wu et al., 2023; Li et al., 2023; Hong et al., 2023). These systems leverage the collaborative potential of multiple specialized agents working together to solve complex tasks that exceed the capabilities of individual models (Park et al., 2023; Liu et al., 2023b). Notable frameworks include AutoGen (Wu et al., 2023), which facilitates multi-agent conversations through customizable agent roles and interaction patterns, CAMEL (Li et al., 2023), which ex- plores role-playing dynamics in collaborative task-solving, and MetaGPT (Hong et al., 2023), which incorporates software development methodologies into multi-agent workflows. Recent work has expanded these foundations to include specialized domains such as scientific research (Ghafarollahi & Buehler, 2024), software development (Kumar et al., 2024), and complex reasoning tasks (Du al., 2023). However, as these systems grow in sophistication, the challenge of diagnosing failures becomes increasingly complex, with current debugging approaches remaining largely manual and ad-hoc (Wang et al., 2024b). The need for automated failure attribution becomes particularly acute in production deployments where system reliability directly impacts user experience and operational efficiency (Fourney et al., 2024).
The rapid proliferation of multi-agent systems has outpaced the development of systematic debugging methodologies. While considerable effort has been invested in designing agent architectures and interaction protocols (Qian et al., 2023; Chen et al., 2024), relatively little attention has been paid to post-hoc failure analysis. This gap is particularly problematic given the emergent behaviors that arise from agent interactions, where system failures often result from subtle cascading effects rather than obvious individual errors (Kumar et al., 2024). Our work addresses this critical gap by providing the first systematic framework for automated failure attribution specifically designed for the unique challenges of multi-agent system debugging. Unlike previous approaches that focus on system design or performance evaluation (Wang et al., 2024b), we concentrate on the diagnostic phase that enables iterative improvement and reliable deployment.
2.2 LLM-AS-A-JUDGE AND PROCESS-LEVER EVALUATION
The paradigm of using LLMs evaluators has gained significant traction scalable alternative to human assessment across diverse domains (Zheng et al., 2023; Wang et al., 2024a). This approach has proven particularly valuable in scenarios where human evaluation is expensive, cine-consu,ing, or requires specialized expertise (Liu et al., 2023a; Dubois et al., 2023). Recent developments have extended L.LM- evaluation to process-level assessment, where models evaluate intermediate
2

---

Discovered and authored by DeepScientist.
reasoning steps rather than only final outputs (Lightman et al., 2023; Wang et al., 2023). Process reward models (Uesato et al., 2022) have shown promise in mathematical reasoning by identifying the specific steps where eπrors occur, enabling more targeted feedback and improvement strategies. However, these approaches primarily focus on single-agent reasoning chains in well-defined domains like mathematics coding, where the correctness of individual steps can be objectively determined.
Our work extends this process-level evaluation paradigm to the significantly more complex domain of multi-agent system failures. Unlike mathematical reasoning where step correctness is often binary and context-independent, multi-agent failures involve complex interdependencies between agents, temporal dynamics, and emergent behaviors that resist simple classification (Du et al., 2023). While process reward models evaluate individual reasoning steps, our A2P framework must navigate the multi-particiiana interactive dynamics of agent systems where the "correctness" of an action depends heavily on the broader conversational context and the ultimate task outcome. This fundamental difference necessitates our novel approach of structured counterfactual reasoning rather than step-by- step correctness assessment (Miller, 2019; Doshi-Velez& & Kim, 2017).
2.3 CAUSAL REASONING IN LLMS
Recent research has begun exploring the causal reasoning capabilities of large language models, revealing both promising potential and significant limitations (Kıeıan al., 2023; Zevcevic et al., 2023). Benchmarks such as CLadder (Qin et al., 2023) and CausalBench (Jin et al., 2023) have established that while L.LMs can perform certain types of causal reasoning. they often struggle with complex counterfactual inference tasks that require systematic manipulation of causal variables (Jin et al., 2024). This limitation is particularly pronounced in scenarios requiring what Pearl terms "Level 3" causal reasoning, answering questions about what would have happened under different circumstances (Pearl, 2009). Studies have shown that structured prompting approaches, such as CausalCoT (Zhang et al., 2024), can significantly enhance LLM performance on causal tasks by providing explicit reasoning frameworks that guide model inference.
Building these insights, structured causal prompting to a real-world diagnostic task. While previous work has focused on synthetic causal reasoning benchmarks simplified scenarios (Jin et al., 2023; Qin et al., 2023), tackle the significantly more complex challenge of failure attribution in multi-agent systems where causal relationships are embedded in natural language conversations and span multiple participants over extended time horizons. Our approach operationalizes Pearl's s three-level causal hierarchy (Pearl et al., 2016) into a concrete prompting strategy that enables LLMs to perform sophisticated counterfactual analysis. Unlike previous causal reasoning work that typically evaluates models on isolated causal queries, we demonstrate how structured causal prompting can address practical system debugging challenges where the stakes of accurate causal inference directly impact development efficiency and system reliability (Schölkopf et al., 2021; ; Peters et al., 2017).
A2P Scaffolding framework represents a practical application of
METHOD
The challenge of automated failure attribution in multi-agent systems stems from the inherent complexity of causal reasoning over extended, multi-participant conversational sequences. Existing baseline methods, while processing the complete contextual information, treat attribution as a monolithic pattern recognition task, implicitly assuming that LLMs can perform comprehensive counterfactual reasoning within a single, unstructured inference step, an assumption contradicted by recent benchmarks evaluating LLMM causal capabilities (Kieıman et al., 2023; Zevcevic et al., 2023). This assumption leads to a critical analytical bottleneck: models may successfully identify correlations or surface-level errors but systematically fail to determine whether those errors were truly decisive-that is, whether their correction would have altered the task outcome from failure to success. This counterfactual inference gap constitutes the primary cause of the characteristicllyy low step-level accuracy observed in existing attribution systems (Zhang et al., 2025).
To bridge this gap, we introduce Abduct-Act-Preet Acct-rredict (A2P) Scaffolding, a novel prompting framework that restructures the failure attribution task into formal, three-step causal inference process. Our approach is implemented as an enhancement to the All-at-Once method, thereby retaining its key advantage of having access the complete conversational context. However, instead of simple in-
3

---

Discovered and authored by DeepScientist.
struction, we employ sophisticated prompt generation function, construct_causee paspcc___ssll__opptttt that guides the LLM through a rigorous analytical sequence inspired by Pearl's structural causal model framework (Pearl, 2009). This method makes the reasoning process transparent, verifiable, and significantly more accurate without requiring any changes to the underlying model architecture.
The core of A2P Scaffolding is its three-step reasoning structure, illustrated in Figure 1. (1) Abduction (Inferring Hidden Causes): The process begins by prompting the LLM to move beyond mere observation to abductive reasoning. Given the final task failure, the model is instructed identify and articulate the hidden factors or latent variables (e.g., an agent's knowledge gap, a flawed assumption, misinterpretation of the user's query) that best explain why a specific agent took specific action a specific step. This approximates the posterior inference of exogenous variables in causal model, forcing the model to establish plausible root cause before proceeding. (2) Action (Defining an Intervention): Once potential root cause and erroneous action are hypothesized, the framework prompts the L.MM to define a minimal, concrete intervention. This corresponds to applying the do()-operator in Pearl's causal calculus (Pearl et al., 2016). The model must specify the exact, "correct" action the agent should have taken in that step. This step is crucial as it translates the abstract hypothesis into testable, operationalized counterfactual. (3) Prediction (Simulating the Counterfactual Trajectory): With the intervention defined, the final step is to predict its consequences. The LLM is instructed simulate the subsequent 3-5 turns of the conversation under the counterfactual condition that the correct action was taken. must then predict whether this new, simulated trajectory would lead the successful completion of the original task. This step directly evaluates the decisive nature of the error; if the simulated outcome is success, the hypothesis is confirmed.
Mathematically, A2P Scaffolding approximates the estimation of a counterfactual outcome (I (T)) for an intervention step t. We formalize the failure attribution task within Pearl's SCM framework where a trajectory T is generated by structural equations with states evolving as where & represents unobserved exogenous variables (e-g., agent's internal knowledge state). The final outcome 7(r) is function of the full trajectory. Our objective is to find the earliest pair (i", ,t*) arg min(i.e) t such that the LLM's guided simulation predicts (T) = (success). The A2P framework guides the LLM through three approximations:
ft), a,,"), 8s+1 [81, a,,
Abduction:
Et arg max P(e|s 01, Z(7) =)
Action:
dola; + ai)
(2)
Prediction:
* sttl....... where si.1 . Z(r") g g(so.....
(3)
This entire three-step process is executed for each potential error the model considers, and it ultimately outputs the earliest agent-step pair that satisfies this causal chain. To support this fine-grained temporal reasoning, our method incorporates a critical structural component: Contextual Step Numbering. Before being passed to the model, the entire conversation log is pre-processed prefix each turn with an explicit, formatted identifier like Step (idx) conclusively demonstrate that these structural anchors are not merely minor enhancement but are absolutely essential, preventing catastrophic drop in step-level accuracy by providing the model with unambiguous reference points to trace causal dependencies through the dialogue.
agent_Namee :. Our ablation experiments
The implementation is seamlessly integrated into the existing codebase through command-ling flag ccausal__eesspnng reasoning that activates the construct. leasstrucc_ccuu_l_______ee _roppprropppttt function within the ail_at_onee and all_at _oc_eatt________________________ conc_aasync methods. This design ensures full backward compatibility while making our advanced causal analysis casily accessible. The computational overhead is minimal, consisting of 25% increase in processing time and token count per sample modest cost for the 2.85x improvement in accuracy achieved by our method.
Having established the theoretical foundation and implementation details of A2P Scaffolding, proceed to describe our comprehensive experimental methodology designed to rigorously evaluate the framework's 's effectiveness across diverse multi-agent system configurations and failure scenarios.
4

---

Discovered and authored by DeepScientist.
1. Input& & Pre-processing
2. A2P Causal Inference Process
3. Output
Falled Conversation Log
A. Abduction: Infer Hidden Root Cause
USer Find hotel Fong for tomorrow
JMN an propaotte hike Htp to hypet hopt plarte stow.
Huttvkgeent usn Awio with that.
Hypothesā, Agont minneoppeeted "tonorow due a late-right queny worching for the corrent olready date
User Great. Foved a room with
peteelpeer- Ancionles hateis
Systeme Favled.
500 boolled
Goal Vdlae
aggnccil 201-41
Causal Link Confirmed
The intervention leads suconss.
Anolyst Agret
Counterfactual Inntrvettionnnnnnnnionnn
Ukd eeinns street opur shord
Crucial Pre-processing
Origina Actions Apologies, al horals are fully booked. Corrected Acton (loter
Profic with numberse
conyy do1s.
Step Daee Find batel Parie..
Ane you tanking for [Jote (2ate))))
Noteskgont: con
Root Cause Attributed
User
tallor Agent
Craca Opezsias Defon on eson atter BOOL of
The root cause of failure is: HotelAgent Step
Prediction: Define Counterfactual Intervention
Step System
spreede canigggea infirecce
Step o' www.sggeeeeeee.oo.ooooo. Air ywrpouoeotrgggooo DC.
Orade Agant
Doliatioon
Figure 1: Overview of the A2P Scaffolding framework. The method transforms raw multi-agent conversation logs through explicit step numbering, then guides the LLM through three sequential causal reasoning steps: (1) Abduction infer root causes, (2) Action define interventions, and (3) Prediction simulate counterfactual outcomes, ultimately producing precise failure attribution with causal explanations.
4 EXPERIMENTAL. SETUP
All experiments were conducted on the Who& When benchmark (Zhang et al., 2025), a comprchensive dataset specifically designed for automated failure attribution in multi-agent systems. The benchmark comprises two distinct subsets that provide complementary perspectives on system complexity: Aegeeitm (126 samples) and Hand-Crafted (58 samples), totaling 184 distinct failure attribution tasks. The Algorithm-Generate subset contains failure logs from systems automatically constructed using the CaptainAgent algorithm from the AG2 library, where each system is tailored to specific queries from the GAIA (Mialon et al., 2023) and AssistantBench (Yoran et al., 2024) valida- tion sets. These systems represent diverse agent configurations with varying tools and specializations, providing broad coverage of multi-agent architectures. The Hand-Crafted subset features failure logs from Magnetic-One (Fourney et al., 2024), mature, carefully engineered multi-agent system comprising five specialized agents designed for web browsing, file navigation, and complex task orchestration. This subset offers more realistic and sophisticated failure scenarios with conversation lengths extending up to 130 steps, making it particularly challenging for temporal reasoning tasks.
Our method, A2P Scaffolding, was implemented by modifying the baseline all _ettee_e_______________ee__eeee_ approach to incorporate our structured causal reasoning prompt, activated via a -causal -causal_reasontt reasoningnngggg command-line flag. We used the gpt-oss -12bb model accessed via local API endpoint to ensure consistent experimental conditions across all methods. All experiments, including baseline re-runs for direct comparability, were executed using an asynchronous pipeline with batch size of 48 and a maximum token limit of 20,000. This configuration enables efficient processing while maintaining the quality of generated responses. The experimental infrastructure was deployed on NVIDIA H100 80GB HBM3 GPUs running on Linux 5.14.0-427.11....lu 4.x86_66 64, providing sufficient computational resources for large-scale evaluation.
Performance evaluation employs two primary metrics that capture different aspects of attribution accuracy. Agent-Level Accuracy measures the percentage of correctly predicted respuse-repponsible agents, representing the fundamental requirement for identifying which component of the multi-agent system caused the failure. This metric reflects the system's 's ability to isolate problematic agents from the broader collaborative process. Step-Level Accuracy quantifies the percentage of correctly identified decisive error steps, imposing significantly higher precision requirements on the attribution algorithms. This metric captures the system's 's ability to pinpoint the exact temporal location where corrective intervention would change the outcome from failure to success, providing the fine-grained diagnostic information necessary for targeted system improvements.
5

---

Discovered and authored by DeepScientist.
For ablation studies involving potential randomness in model outputs, we conducted 5 independent runs and report the mean and standard deviation to ensure statistical robustness. Statistical significance was assessed using paired t-tests for dependent samples, with p-values calculated to determine the reliability of observed performance differences. All baseline comparisons were conducted under identical experimental conditions using our re-runs documented in the experimental results, ensuring direct comparability and eliminating potential confounding factors from different evaluation environments or model versions. This rigorous experimental design enables confident attribution of performance improvements to our methodological innovations rather than experimental artifacts.
With this comprehensive experimental framework established, now present our empirical findings, beginning with the main performance comparisons and followed by systematic ablation studies that address our three core research questions about the effectiveness and operational characteristics of A2P Scaffolding.
EXPERIMENTS
The primary result of our study is the dramatic improvement in step-level failure attribution accu- racy achieved by our A2P Scaffolding method with contextual step numbering. Table presents a comprehensive performance comparison step numbering achieves 47.46% step accuracy on the Algorithm-General outperforming the next-best baseline (bi na nary_sea ch at 28.57 28.57%) and nearly tripling the perfor- mance of the direct baseline (all _aatt_______________ at_ at 16.67%). This represents 2.85x improvement over the all_at___ _ll_at_once baseline, demonstrating the transformative impact of our structured causal reasoning framework combined with explicit temporal anchoring through step numbering (Peters al., 2017).
both datasets, where enhanced A2P Scaffolding with
dataset--ignnffiantty
Table Performance comparison A2P Scaffolding against baseline methods both datasets, Our method with step numbering demonstrates state-of tate-of-thhear the-art performance, particularly in step-level accuracy.
Generated (126 samples)
Hand-Crafted samples)
Method
Agent Accuracy (%)
Step Accuracy (%)
Agent Accuracy (%)
Step Accuracy (%)
Value
Gain
Value
Gain
Value
Gain
Value
Gain
A2P (Ours)
65.40
47.46
58.62
29.31
Baselines
all_at_ [t_at_pe once
63.49
-1.91
16.67
-3.799
27.59
-31.03
12.07
-17.24
step_by_step
49.21
-16.19
27.78
-19.68
53.45
-5.17
18.97
-10.44
binary_search
46.83
-18.57
28.57
-18.89
44.83
-13.79
13.79
-15.52
On the more challenging Hand-Crafted dataset, our method achieves 29.31% % step accuracy 2.43x improvement over the all _t baseline's 's 12.07%, substantially outperforming all other methods in this complex, realistic setting. The agent-level accuracy 65.40% on Algorithm- Generated and 58.62% 28.62% on Hand-Crafted datasets further demonstrates the robustness of our approach across different system complexities. These results establish A2P Scaffolding as the first automated method to achieve nearly 50% step-level accuracy on algorithm-generated -enerrtted systems while maintaining superior performance on realistic, complex scenarios (Fourney et al., 2024; Wu et al., 2023).
Research Question 1: How does structuring an LLM's inference process with explicit three- step causal framework (Abduction, Action, Prediction) and contextual step numbering affect its ability to perform fine-grained failure attribution in multi-agent conversations?
Our systematic ablation studies provide compelling evidence for the necessity of each component in the A2P framework. Table 2 quantifies the degradation in step-level accuracy when components are removed.
The Abduction step, which enables the model to infer hidden causal factors behind agent actions, contributes 6.35 percentage points on Aggrithmm-Generaee and 8.62 percentage points on Hand- Crafted datasets. This component transforms surface-level error detection into deep causal analysis

---

Discovered and authored by DeepScientist.
Table 2: Impact of removing core causal components from A2P Scaffolding. Both Abduction and Prediction steps are essential maintaining high step-level accuracy across datasets.
Algorithm-Generateed
Hand-Crafted
Configuration
Step Acc. (%)
Drop (pp)
Step Acc. (%)
Drop (pp)
Full A2P Model
47.46
29.31
A2P w/ Abduction
41.11
-6.35
20.69
-8.62
A2P w/o Prediction
40.32
-7.14
17.24
-12.07
by forcing the model to reason about latent variables such as knowledge gaps, incorrect assumptions, or misinterpretations that explain observed failures (Pearl et al., 2016; Schölkopf et al., 2021).
The Prediction step demonstrates even greater importance, particularly for complex scenarios. Its removal causes degradation 7.14 percentage points on Algorithm-Generate Generated and a substantial 12.07 percentage points on Hand-Crafte step accuracy. This validates theoretical framework that explicit counterfactual simulation testing whether a corrective intervention would resolve the failure-is essential for distinguishing decisive errors from incidental mistakes. The larger impact on Hand-Crafted systems suggests that counterfactual reasoning becomes increasingly critical as conversation complexity and length increase (Lewis, 1973; Woodward, 2003).
Most remarkably, Table reveals the critical importance of contextual step numbering.
Table 3: Critical impact of explicit step numbering on A2P Scaffolding performance. The catastrophic drop in step accuracy demonstrates the essential role of structural prompting cues.
Configuration
Agent Acc. (%)
Step Aec. (%)
Step Acc. Drop (pp)
A2P with Step Numbering
65.40
47.
A2P without Step Numbering
64.29
17.78
-29.68
dataset (126 samples). The removal of simple "Step [idx] " prefixes causes a catastrophic performance collapse, demonstrating that structural anchoring is as critical as semantic content for fine-grained temporal reasoming in LLMs.
Note:
Results averaged over experimental runs on
The removal of explicit step numbering- a catastrophic 29.68 percentage point collapse in step-level accuracy (from 47.46% to 17.78%) while leaving agent accuracy relatively unchanged. This finding demonstrates that providing clear structural anchors for temporal reasoning is not merely helpful but absolutely essential for fine-grained causal analysis. The result aligns with recent work showing that LLMs' reasoning capabilities are highly sensitive to input formatting and structural cues (Min et al., 2022; Webson & Pavlick, 2021), suggesting that effective prompt engineering must consider both semantic content and syntactic organization.
removing the "Step (idx) prefixes causes
Research Question 2: Can the A2P Scaffolding method achieve superior step-level accuracy compared to holistic, incremental, and hierarchical search-based attribution methods on both aggoithmnically-gneeaaa and complex hand-crafted agent systems?
Our comprehensive evaluation in Table demonstrates A2P Scaffolding's 's systematic superiority across diverse system types and complexity levels. The method achieves the highest performance on both metrics for Aggoritmm-Generated systems (65.40% agent accuracy, 47.46% step accuracy), with step accuracy improvements 2.85x over all_at_ once, 1.71x over step_by by_step, and 1.66x over i siinaryy.... aarrr_rrearrreeea search. These substantial gains stem from A2P's unique ability combine holistic context processing with structured causal analysis, avoiding the pitfalls both extremes (Bommasani al., 2021; Brown al., 2020).
The Hand-Crafted dataset results prove particularly compelling. While baseline methods struggle with the increased complexity-with a __nnce achieving only 12.07% step accuracy-AD2
7

---

Discovered and authored by DeepScientist.
maintains robust performance at 29.31%. This 2.43X improvement demonstrates that our causal framework scales effectively to realistic scenarios with extended conversation sequences (up 130 steps) and complex inter-agent dependencies. The method's resilience to inereasing complexity validates its potential for debugging production multi-agent systems where failures often involve subtle causal chains spanning many interaction steps (Hong et al., 2023; Li et al., 2023).
The performance advantage stems from A2P's principled approach to counterfactual reasoning. Unlike step__o ________eppp_______________e____ methods that make premature decisions with incomplete context, or all_at_once approaches that struggle with the mreedle-inhhysstee problem of long contexts (Liu et al., 2024), A2P processes the entire conversation while maintaining focused causal analysis through its structured three-step framework. This design enables accurate attribution even in complex scenarios where the decisive error and its ultimate consequence are separated by many intermediate steps.
What are the operational characteristics and practical implications of using A2P Scaffolding for debugging multi-agent systems?
Research Question 3:
Our analysis reveals several operational characteristics that enhance A2P's practical utility. Figure shows the method's sensitivity to counterfactual simulation length in the Prediction step.
Fixed Length
A2P Scaffolding (Ours)
48
Step Accuracy (%)
46
44
40
10
3-5 (Ours)
Simulation Length (Steps)
Figure Sensitivity analysis of counterfactual simulation length in the Prediction step. The flexible 3-5 step range (shown dashed line) achieves optimal performance, outperforming fixed-length alternatives and demonstrating the value of adaptive simulation depth for robust counterfactual reasoning.
The flexible 3-5 step range achieves optimal per- formance, outperforming all fixed-length alterna- tives. This suggests that allowing adaptive simu- lation depth based on context produces more ro- bust counterfactual reasoning than rigid parame- ters (Zhang et al., 2024; Wei et al., 2024).
Table 4: Impact of explicit root cause criteria in the prompt. Results show no significant improvement (p 0.05).
Dataset
WITH
WITHOUT
p-val
Alg Gg-GGen Hand-Crafted
46.35%
43.81 %
0.126
Our methodological rigor demonstrated through systematic ablation of non-essential components. Table shows that including explicit formal causal criteria (PRECEDES, NECESSARY, SUFFICIENT) provides statistically significant improvement (p > 0.05), justifying their exclusion from the final design. This data-driven optimization ensures that A2P's complexity is justified by empirically validated gains rather than theoretical appeal (Reynolds McDonell, 2021; Kojima et al., 2022).
20.34%
23.10% 13.10%
0.148
The method generates causally coherent explanations that explicitly trace propagation through agent interactions, making A2P valuable for human developers seeking actionable debugging insights (Miller, 2019; Doshi- Velez & Kim, 2017).
8

---

Discovered and authored by DeepScientist.
From deployment perspective, A2P incurs approximately 25% additional processing time compared to baseline methods modest cost for nearly 2.85x improvement in step accuracy. The backward- compatible implementation via a simple command-ling flag enables seamless integration into existing workflows. Combined with its robust performance across system types and proven scalability to complex scenarios, A2P Scaffolding represents practical, immediately deployable solution for automated failure attribution in production multi-agent systems (Wu et al., 2023; Kumar et al., 2024).
6 CONCLUSION
We introduce A2P Scaffolding, novel prompting framework that reframes automated failure attribu- tion in multi-agent systems as a structured causal inference problem through sequential Abduction, Action, and Prediction steps, successfully bridging the counterfactual inference gap that has lim- ited previous pattern recognition approaches to impractically low accuracy levels. Our empirical validation demonstrates state-of-the-art performance, achieving 47.46% step-level accuracy on algorithm-generaan systems and 29.31 1% on complex hand-crafted destems--rprreetttnn x and 2.43x improvements over baselines respectively-while rigorous ablation studies confirm the necessity of each framework component, particularly the critical importance of explicit step num- bering which alone contributes +29.68 96.6 percentage points to step accuracy. Beyond performance metrics, A2P Scaffolding addresses fundamental bottleneck in multi-agent system development by providing accurate, automated identification of tailure-resposibble agents and decisive error steps with causally grounded explanations, enabling developers to perform targeted improvements rather than broad system modifications and dramatically reducing manual debugging effort. The frame- work's demonstrated effectiveness on Hand-Crafted systems with conversation lengths exceeding 100 steps validates its applicability to production debugging scenarios, while its backward-compatin implementation and modest 25% processing overhead make it immediately deployable in existing workflows. Future work can extend the A2P approach to other diagnostic domains requiring counter- factual reasoning, integrate with efficient search strategies for enhanced scalability, and leverage the structured prompting principles to advance LLM capabilities in formal reasoning tasks, ultimately contributing to more robust and interpretable AI systems capable of sophisticated self-diagnosis and explanation.
REFERENCES
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson,, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespic, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel F. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani. Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Car- los Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh. Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan. Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 210.2200.0005555 07258, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray,

---

Discovered and authored by DeepScientist.
Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901. 2020.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. arXiv preprint arXiv:2308. 10848, 2024.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint XrXiv: 100.008008.,,,,, 2017.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning in language models through multiagent debate. arXiv preprint 231v:2305..1325, 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Alpacaeval: An automatic evaluator instruction- following models. arXiv preprint arXiv:2305. 14387, 2023.
Adam Fourney, Gagan Bansal, Dan Hendricks, Vietor Dibia, Hannah Kim, Lorenzo Floridi, Dipankar Ray, Forough Yoursabzi-Sangddee Siddharth Suri, Eric Horvitz, and Ece Kamar. Magentic-one, A generalist multi-agent system for solving complex tasks. arXiv preprint S.Ki::2211.04466. 2024.
Alireza Ghafarollahi and Markus J. Buehler. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning. arXiv preprint arXiv:2409.0555: 2024.
Sirui Hong, Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang. Jin- lin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework. arXiv preprint arXiv:2308.00352. 2023.
Zhijian Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fer- nando Rodríguez Martínez, Bernhard Schölkopf. and Zhaomin Chen. Causalbench: A comprehen- sive benchmark for causal learning capability of 1lms, Advances in Neural Information Processing Systems, 36, 2023.
Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schölkopf. Can large language models infer causation from correlation? arXiv preprint 2rX.v23366.08836 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35: 22199-222133 2022.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John Co-Reyes, Avi Singh, Kate Baumli, Shariq Hashmi, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. Training language models self-correct via reinforcement learning. arXiv preprint 2rXiv:2409.12917. 2024.
Emre Kıeman, Robert Ness, Amit Sharma, and Cheng Tan. Causal reasoning and large language models: Opening new frontier for causality. arXiv preprint arXiv:2305.000. 00050, 2023.
David Lewis. Counterfactuals. Harvard University Press, 1973.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. arXiv preprint 1177: 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint ar.Xiv2305.200500 2023.
10

---

Discovered and authored by DeepScientist.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Perey Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. 16634, 2023a.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic llm-agent network: An Ilm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023b.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In arXiv preprint arXiv:2311.1298. 2023.
Tim Miller Explanation in artificial intelligence: intelligence, 267:1-38, 2019.
Insights from the social sciences. Artificial
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in- in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
Joon Sung Park, Joseph C O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Perey Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03342,, 2023.
Judea Pearl. Causality: Models, Reasoning, and Inference Cambridge University Press, 2nd edition, 2009.
Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press, 2017.
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint 2tti::23070079244 2023.
Zhijian Qin, Jiawen Wang. Wanjun Zhong, Wangchunshu Zhou, Yankai Lin, and Maosong Sun.
Cladder: A benchmark to assess causal reasoning capabilities language models. arXiv preprint arXiv:2312.04330.. 2023.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. arXiv preprint arXiv:2 2021.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10955661,.-33, 2021.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022. URL https: http//axvvv.g///eee.a/// wwwp/bs.////ooo..oooooo...o....r /21/1.211.. 14275.
Lei Wang, Chen Ma, Xueyang Feng. Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin. Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 2024a.
Peiyi Wang, Lei Li, Zhihong Shao, R.X. Xu, Damai Dai, Yifei Li, Deli Chen, Y.Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08895. 2023.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinwei Chen, Jianqiao Lu, Cheng Qian, Yujia Qin, Xiaojian Ma, Yining Ye, Aohan Zeng, Zhiyuan Liu, Xiaoxing Ma. and Maosong Sun. Agent- flan: Designing data and methods of instruction-tuning for agent tasks. arXiv preprint arXiv:2403. 12881, 2024b.
11

---

Discovered and authored by DeepScientist.
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? arXiv preprint arXi::2109.01247, 2021.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou, Think step-by-step: Chain-of-thought prompting for large language models. Advances in Neural Information Processing Systems, 2024.
James Woodward. Making things happen: A theory of causal explanation. Oxford University Press, 2003.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08/55, 2023.
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofrr Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks? arXiv preprint arXiv:2407.17711 2024.
Matej Zevcevic, Moritz Willig, Devendra Singh Dhami, and Kristian Kersting. Causal parrots: Large language models may talk causality but are not causal. arXiv preprint arXiv:2308. /3067, 2023.
Jiaxin Zhang, Zhipeng Zhang, Yeye He, Wayne Xin Zhao, and Ji-Rong Wen. Causalcot: Causal chain-of of of-thought reasoning for multi-hop question answering. arXiv preprint arX1v:2310.13166,
2024.
Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which agent causes task failures and when? on automated failure attribution of Ilm multi- agent systems. ArXiv, 2ss/2505.00212, 2025,
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint 1xXiv:23060066880 2023.
12

---

Discovered and authored by DeepScientist.
T-DETECT: TAIL-WWARE STATISTICAL NORMALIZA- TION FOR ROBUST DETECTION OF ADVERSARIAL MACHINE-GENERATED TEXT
DeepScientist
ABSTRACT
Large language models (LLMs) have shown the capability to generate fluent and logical content, presenting significant challenges to machine-generated text detec- tion, particularly text polished by adversarial perturbations such as paraphrasing. Current zero-shot detectors often employ Gaussian distributions as statistical mea- sure for computing detection thresholds, which falters when confronted with the heavy-tailed statistical artifacts characteristic of adversarial non-native English texts. In this paper, we introduce T-Detect, a novel detection method that funda- mentally redesigns the curvature-based detectors. Our primary innovation is the replacement of standard Gaussian normalization with a heavy-tailed discrepancy score derived from the Student's t-distribution. This approach is theoretically grounded in the empirical observation that adversarial texts exhibit significant leptokurtosis, rendering traditional statistical assumptions inadequate. T-Detect computes a detection score by normalizing the log- likelihood of passage against the expected moments of a t-distribution, providing superior resilience to statistical outliers. We validate our approach on the challenging RAID benchmark for adver- sarial text and the comprehensive HART dataset. Experiments show that T-Detect provides consistent performance uplift over strong baselines, improving AUROC by up to 3.9% in targeted domains. When integrated into a two-dimensional de- tection framework (CT), our method achieves offe-offthhe-at performance, with an AUROC of 0.926 on the Books domain of RAID. Our contributions are a new, theoretically-justife statistical foundation for text detection, an ablation-validated method that demonstrates superior robustness, and a comprehensive analysis of its performance under adversarial conditions.
INTRODUCTION
The rise of powerful large language models (LLMs) (Ouyang et al., 2022; Yang et al., 2025) has ignited a critical arms race between text generation and detection (You et al., 2023; Moraffah et al., 2024). While these models fuel innovation, they also carry risks like misinformation and academic dishonesty, making reliable detection essential (Kumarage et al., 2024). However, this is not static battlefield. A more dangerous front has opened: malicious actors are no longer just using LLMs, but are actively studying our detectors to craft adversarial attacks that can evade them (You al., 2023; Lee et al., 2023). These evolving strategies, from simple paraphrasing to subtle manipulations (Li, 2024), demand new generation of detectors built not just for accuracy, but for fundamental resilience.
The vulnerability of many current zero-shot detectors lies not on the surface. but deep in their statistical core. Leading methods like DetectGPT (Mitchell et al., 2023) and Fast-DetectGPT (Bao al., 2023) are built on a seemingly innocuous assumption: that their statistical scores follow standard bell curve, or Gaussian distribution (Roussceuw & Hubert, 2011). This is their Achilles' heel. Our empirical analysis reveals that adversarial texts are designed to break this premise. They produce score distributions with extreme outliers, resulting in "heavy-tailed" statistical properties (Dugan et al., 2024). The critical research problem, therefore, is that this violation of the Gaussian assumption makes detectors catastrophically sensitive to adversarial attacks, causing their performance to become unstable and unreliable. When faced with the very texts they are designed to catch, their statistical foundation crumbles.

---

Discovered and authored by DeepScientist.
To this end, we introduce T-Detect, novel method that redesigns the detector's statistical core by replacing the flawed Gaussian assumption with a robust, "tail-aware" normalization based on the Student's t-distribution. This single, principled change is grounded in robust statistics (Rousseeuw & Leroy, 2005) and allows our method to gracefully handle the statistical outliers common in adversarial text without being destabilized. By computing "heavy-tailed discrepaney score," T-Detect provides an inherently more stable and reliable signal for distinguishing human from machine-generated text.
We validate T-Detect through comprehensive suite of experiments, demonstrating its practical advantages. As summarized in Figure 1, T-Detect offers a superior trade-off between performance and computational efficiency compared to strong baselines. On the challenging RAID benchmark for adversarial text, our method, particularly when integrated into two-dimensional (CT) frame- work(Bao et al., 2025), achieves state-of-the-art performance with an overall AUROC of 0.876. Our contributions are threefold: (1) We are the first to empirically prove that adversarial text detection cettributtonn-bbsee normalization to address this. (2) We present an ablation-validated method that demonstrates superior robustness and performance adversarial benchmarks. (3) We provide comprehensive analysis of our method's practical benefits, including its computational stability and exceptional hyperparameter robustness, offering more reliable and deployable solution for Al safety.
scores follow heavy-tailed distributions and propose a theoretically-justife
Detection Performance
Content Detection (C) Ferformance
Combined Detection (CT) Stability
6.50
2.50
0.16
0.400
6.855
5.78
0.75
Ddtect
Attone Soxe
6.403
Ristent
Scres
0.770
0,735
ACBOC
0.205
0.000
0.790
0350
6.755
4:2
10
Throgghut (eectquecred))
Throughpat (eshsooord))
Thouughps (bettsheermd))
Figure 1: The 'ALL' Performance (AUROC) vs. Speed (Throughput) on the RAID benchmark. Detect consistently provides better Pareto frontier, offering higher performance for its computational cost. In the two-dimensional setting (c). CT(T-Detect) achieves state-of-the-ar accuracy while being 1.8x faster than the competitive (BBnoouus baseline.
2 RELATED WORK
The task of distinguishing machine- generated text from human-writter content has evolved signifi- cantly, moving from early statistical methods to sophisticated zero-shot classifiers. Early approaches focused on identifying statistical artifacts in generated text. For instance, methods based on simple metrics like likelihood, log-rank, and entropy (Guo et al., 2023; Li et al., 2022) were proposed to capture the unusually predictable nature of text from older generative models (Gehrmann et al., 2019). significant breakthrough came with the introduction of curvature- based detection by Mitchell et al. (2023) in their seminal work, DetectGPT. This method was the first to hypothesize that text sampled from a large language model tends to occupy regions of high negative curvature the model's log- See-robabilityy space. DetectGPT estimated this curvature by generating numerous perturbations of a given text and measuring the average drop in log- bog-probability. establishing a new paradigm for zero-shot detection that did not require a dedicated training dataset.
Building on this foundation, subsequent research has focused on improving both the efficiency and accuracy of curvature-basee based methods. Our direct baseline, Fast-DetectGPT, was introduced by Bao et al. (2023) as a computationally efficient alternative to DetectGPT. It retains the core curvature hypothesis but replaces the costly perturbation step with a more efficient sampling- sanpling-based approach to approximate the necessary statistics, achieving significant speedup. Parallel to these developments, other zero-shot methods have emerged. Binoculars (Hans et al., 2024) proposed a novel approach
2

---

Discovered and authored by DeepScientist.
based on the cross-perplexity between two different language models, one acting an "observer" and the other as a "performer." " Another prominent method. NPR from the DetectLLM framework (Su et al., 2023), leverages log rank information, offering different statistical signal for detection. Our work, T-Detecct contributes to the curvature-basee lineage, but instead of focusing on computational efficiency, we address a more fundamental statistical limitation in the normalization step of these detectors.
To further enhance detection capabilities, some methods combine signals from multiple text represen- tations, a common practice in the broader field of text classification (Yang et al., 2013; Agarwal et al., 2014). The two-dimensional (CT) detection framework, utilized in prior work, one such approach. It combines score from the original text (T) with a score from a content-only representation (C), where function words and other stylistic markers have been removed. This allows the system to decouple signals related to the expression of the text from those related to its core content. In our work, we use this framework to demonstrate that T-Detect provides more robust base signal, thereby improving the performance of the entire combined system. This is particularly important in the context of adversarial attacks, such as paraphrasing (Li, 2024) and Unicode manipulation, which are designed to evade detection by altering either the expression or the underlying character data of text, underscoring the need for robust, multi-faceted detection strategies.
3 METHOD
The challenge of detecting -enerrtte text has intensified with the advent of models capable of producing highly fluent and contextually appropriate content. A significant frontier in this field is the detection of text that has been adversarially perturbed to evade detection. Many cxisting zero-shot statistical detectors, such as Fast- Past-DetectGIT (Bao al., 2023), operate by measuring the 'surprise" of a given text under a language model. They typically compute a discrepancy score representing how much the aog-robbabiltyy of the observed text deviates from the expected log-probability, and then normalize this score. A critical, often implicit, assumption in this normalization step is that the underlying distribution of these log- preo-abbbbilty discrepancies is Gaussian. However, our empirical analysis reveals this assumption is fundamentally flawed for the very texts we are most interested in detecting: adversarial and non-native passages. These texts introduce statistical outliers that result in heavy-tailed, or leptokurtic, distributions (dos Santos & Cirillo, 2021), causing Gaussian- based methods to be overly sensitive and unreliable, a well-doce phenomenon in robust statistics (Rousseeuw & Leroy, 2005).
To address this foundational problem, we introduce T-Detect, a novel detection method that replaces the flawed Gaussian assumption with more robust statistical framework based on the Student's t-distribution. The Student's 's t-distribution is naturally suited for modeling data with heavier tails than normal distribution, making it ideal choice for handling the statistical artifacts introduced by adversarial attacks (Rath et al., 2022). Our core innovation lies in the reformulation of the discrepancy normalization. While the baseline Dest-DetectGPT calculates a standard Z-score, T-Detect computes a score that is normalized according to the properties of t-distribution, as illustrated in Figure 2.
The technical implementation of T-Detect builds upon the sampling discrepancy framework. Given an input text x. a scoring model Pscore. and a reference model pref. we first compute the unnormalized discrepancy score d(z) and the aggregated variance (ax) as in the baseline:
d(xx) (log Preore (2; 1s)
(I)
V(x) =
(2)
where μ4 and of are the mean and variance of the log probaabbailiites of tokens at position i under the reference distribution Pres- The crucial departure from the baseline is in the normalization step. Instead of a simple standard deviation normalization, T-Detect uses a normalization factor that incorporates the degrees of freedom parameter, v, from the Student's t-distribution. The final T-Detect score is
3

---

Discovered and authored by DeepScientist.
given by:
d(x)
m (log Psccre (2, xx() pi)
D De-disl(1;
Vi=2 V(x)
The term y-z represents the variance of standard Student's t-distribution with degrees of freedom (for 1/ > 2). By scaling the denominator by this factor, our normalization explicitly accounts for the higher variance expected in heavy-tailed data. When distribution has outliers, the standard deviation can be inflated, but the t-distribution's properties provide more stable estimate of the dispersion. For large values of v, this scaling factor approaches I, and T-Detect gracefully converges to the Gaussian-bed based baseline, making it generalized extension. Our experiments show that small value, such as / = effective and that the method is remarkably robust to the specific choice of this hyperparameter.
This single, theoretically-groun grounded modification the entirety of our proposed method, as validated by ablation studies which demonstrated that other potential enhancements like dynamic thresholding provided no performance benefit. The elegance of T-Detect lies in its simplicity: by fixing a single flawed statistical assumption, it achieves greater robustness and performance without adding any computational complexity. The method's implementation requires only a minor change to the final scoring calculation, preserving the efficiency of the original Mest-DetecGGPT framework while significantly enhancing its reliability against the most challenging types of machine- generated text.
machine-generated
Table I: Performance of T-Detect and baselines on the adversarial RAID benchmark. Results are reported AUROC Fl-Score & TPR @ @%FPPR. Best performance in each metric for ALL is highlighted in bold, second best is underlined.
Gest
Binoculars
T-Detect (Ours)
Dataset
F1-Score
TPR@5%
AUROC
F1-Score
TPR@SSFPR
FI-Score
FPR
T (Text)
Recipes
0.749
0.71
0.56
0.759
0.72
0.60
0.752
0.72
0..6
0.850
0.81
News
0,761
0.73
0.48
0,768
0.75
0.52
0.767
0.75
0.52
Wiki
0.803
0.52
0,804
0.75
0.54
0.801
Reviews
0.810
0.77
0.51
0.812
0.78
0.52
0.812
0.77
0.54
Reddit
0.794
0.75
0.42
0.811
0.78
0.48
0.807
0.78
0.48
Poctry
0.818
0.59
0.826
0.79
0.61
0.827
Abstracts
0.821
0.37
0.58
0,826
0.77
0.64
0.827
0.78
0.66
ALL
0.792
0.74
0.52
0.800
0.76
0.55
0.798
0.76
0.55
C (Content)
Recipes
0.674
0.62
0.41
0.726
0.62
0.56
0.726
0.64
0.56
Books
0.873
0.79
0.70
0.888
0.83
0.73
0.886
0.82
0.72
News
0.767
0.70
0.43
0.783
0.71
0.57
0.783
0.70
0.56
Wiki
0.807
0.73
0.56
0,808
0.75
0.55
0.807
0.74
0.55
Reviews
0.762
0.71
0.40
Reddit
0.755
0.69
0.42
0.778
0.71
0.52
0.779
0.72
0.50
Poetry
0.777
Abstracts
0.774
0.71
0.44
0,799
0.75
0.58
0.799
0.75
0.58
ALL
0.742
0.69
0.37
0.765
0.71
0.43
0.773
0.72
0.50
(Framework)
Recipes
0.855
0.78
0.63
0.878
0.77
0.69
0.891
0.81
0.67
Books
0.913
0.88
0.76
0.924
0.89
0.83
0.926
0.89
0.84
News
0.871
0.68
0.900
0.83
0.74
Wiki
0.874
0.81
0.70
0,861
0.78
0.68
0.868
0.50
0.70
0.869
Reddit
0.853
0.78
0.63
0,869
0.81
0.64
0.871
0.79
0.64
Poctry
0.859
0.889
0.83
0.69
0.898
Abstracts
0.880
0.80
0.67
0.900
0.82
0.71
0.900
0.83
0.74
ALL
0.854
0.79
0.63
0.873
0.80
0.65
0.876
0.81
0.66
EXPERIMENTAL SETUP
All experiments conducted server equipped with an AMD EPYC 7542 CPU, 503GB of R RAM, and two NVIDIA A 1000SSX44800BB GPUs. We used PyTorch 2.7.0 and Transformers 4.53.1. For all metric-based detectors, including our proposed T-Detect and the Fast baseline,
4

---

Discovered and authored by DeepScientist.
Stage 1: Input & Shared Discrepancy Calculation
p_ p_ref
Discrepancy:
d(x) Σ (log p_score (xillx [i) m_)
Variance:
Input Text (x)
Scoring & Ref. Models
V(x) I _
Baseline Method
T-Detect (Our Method)
(e (e-g., Dest-DetectGPT)
Stage 2: Flawed Assumption
Stage 2: Robust Assumption
Adversarial text creates outtiers... which
Heavy talls neturally models ouiliers... le.lelding ng to a more robust framework
assumptioon
'Normal' Data
Heavy-tailed Data
Stage 3. Normalization
Stage 3. Our Innovation: Normalization
Score
Robust Normalization with I-distribution
Score d(x) VV(x)
Score d(x) -V(x) v (v-2)
Stage 4. Final Score
Stage 4. Final Score
Unreliable Overly Sensitive www..
Robust
Inflated varlance misleads detector
Stable varlance leads correct detection
Wrongly Classified
Correctly Classified /
Figure 2: Conceptual overview of T-Detect. The method first calculates the raw discrepancy and variance from the input text. The key innovation is the normalization step, where T-Detect uses robust, heavy-tailed model based the Student's t-distribution, in contrast to the baseline's s implicit Gaussian assumption. This allows T-Detect to correctly handle statistical outliers common in adversarial text, leading to more stable and accurate final detection score.
used Falcon-7B as the reference/observer model and Ce-oon----nnttuutt the scoring/performer model to ensure fair and consistent comparison. The maximum token length for all inputs was set 512.
We evaluate our method on two primary benchmarks. The first is the RAID benchmark (Dugan et al., 2024), challenging dataset specifically designed test detector robustness against 12 different types of adversarial attacks across 8 diverse domains. The second is the HART dataset, a large-scale, multi-domain benchmark for general-purpose machine- generated text detection. We also include results smaller TOEFL dataset to assess performance non-native English text.
machine-gencrated
For all experiments, we follow a consistent evaluation protocol. For methods that produce a single detection score, such as T-Detect and the baselines, fit decision threshold on the development set of each respective benchmark by optimizing for the Fl-score. For the two-dimensional CT-framework, which produces two scores (one for text. for content), train Support Vector Regressor (SVR) on the development set to learn a combined decision boundary. Performance is primarily measured using the Area Under the Receiver Operating Characteristic Curve (AUROC), with Fi-score and

---

Discovered and authored by DeepScientist.
Table 2: General performance of T-Detect and baselines the multi-domain HART benchmark. Results are reported as AUROC FI-Score TPR&5%FPR. Best performance in each metric for ALL is highlighted in bold, second best underlined.
FattDetectGPT
Binoculars
T-Detect (Ours)
Dataset
AUROC
F1-Score
TPR 5%P
AUROC
Fl-Score
TPR05%FPR
AUROC
FI-Score
TPR
Level
0.714
0.43
0.720
0.68
0.714
Arxiv
0.769
0.72
0.57
0.769
0.72
0.56
0.771
0.71
0.58
Essay
0.877
0.81
0.73
0.879
0.82
0.73
0.880
0.82
0.73
Writing
0.740
0.70
0.47
0.740
0.70
0.49
0.740
0.70
0.48
ALL
0.778
0.72
0.55
0.780
0.73
0.55
0.780
0.73
0.55
Level 2
News
0.689
0.67
0.47
0.699
0.68
0.47
0.698
0.67
Arxiv
0.718
0.71
0.57
0.715
0.70
0.56
0.718
0.71
0.57
Essay
0.734
0.68
0.34
0.735
0.68
0.37
0.734
0.68
0.36
Writing
0.692
0.693
0.693
ALL
0.711
0.68
0.47
0.711
0.69
0.44
0.712
0.69
0.44
Level 3
News
0.851
0.80
0.54
0.866
0.83
0.63
0.863
0.82
0.59
Arxiv
0.877
0.83
0.72
0.882
0.85
0.77
0.879
0.84
0.75
Essay 0.883
0.80
0.59
0.897
0.80
0.64
0.891
0.80
0.62
Writing 0.840
0.847
ALL
0.862
0.81
0,870
0.867
0.62
True Positive Rate at 5% False Positive Rate (IPR@5%FF)) also reported for a comprehensive evaluation.
EXPERIMENTS AND RESULTS
We conduct series experiments to validate T-Detect, organized around our three core research questions. We first present the main comparative results adversarial and general-purpose bench- marks, followed by a detailed analysis that addresses each research question turn.
5.1 MAIN PERFORMANCE RESULTS
Our primary results demonstrate that T-Detect consistently improves performance over strong base- lines, particularly on adversarially crafted text. Table shows the performance on the challenging RAID benchmark. In the most critical two-dimensional CT configuration, our CT(T-Detect) achieves a state-of-the-ar overall AUROC of 0.876, surpassing both the CT((astDetect(PPT) baseline and the especially pronounced in creative and
competitive CT(Binoculars) method. The improvements technical domains, such as Books (0.926 AUROC) and Poetry (0.898 AUROC). Table 2 shows the performance on the gerpoee-purpose HART benchmark, where T-Detect remains highly competitive, confirming that its robustness does not compromise its general applicability.
5.2
ANALYSIS OF RESEARCH QUESTIONS
RQ1: How can the statistical foundation of curvature-based text detectors be reformulated using heavy-tailed distributions to improve robustness, and what the empirical validation for this approach?
The theoretical foundation of T-Detect is validated by direct statistical analysis of detector scores. As shown in Figure and Table 3, the scores from the adversarial RAID dataset exhibit significant positive excess kurtosis (0.3876), a definitive marker of heavy-tailed distribution. In contrast, scores from the standard HART dataset show negative kurtosis, aligning more closely with Gaussian profile. Model selection criteria overwhelmingly confirm this, with the Akaike Information Criterion (AIC) showing 32.98 point improvement for the t-distribution data. This provides strong empirical justification for our methodological shift. The effectiveness of this change isolated in ablation study (Table 4), which demonstrates that the t-distribution
the Gaussian model on RAID

---

Discovered and authored by DeepScientist.
normalization component is the sole source of performance gain, contributing +0.60% AUROC improvement on its own.
RAID (Adversarial)
HART (Standard)
3.5
Density
2.0
1.5
1.0
0.5
0.0
Detection Score
Figure 3: Statistical properties of detection score distributions on adversarial (RAID) vs. standard
(HART) text.
Table 3: Statistical properties of detection score distributions. Adversarial text (RAID) exhibits significant heavy-tailed characteristics, justifying the use of Student's t-distribution.
Dataset
Excess Kurtosis
AIC (t-dist vs. Gauss) Preferred Model
RAID (Adversarial) 0.3876
-32.98
t-distribution Gaussian
HART (Standard)
-0.2764
+2.00
Table 4: Ablation study of T-Detect components on the RAID dataset. The results isolate the performance contribution of our proposed heavy-tailed normalization, demonstrating is the sole source of improvement.
Configuration
AUROC
Improvement
Baseline (Gaussian Normalization) T-Detect (t-dist Normalization Only)
0.8127
0.8176
+0.60
RQ2: Does the proposed T-Detect method achieve su- perior performance compared to state-of-the-art base- lines on challenging benchmarks?
Table 6: Hyperparameter sensitivity analysis for T-Detect's core parameter, The method demonstrates exceptional robustness across wide range of param- eter settings.
The main performance tables confirm the superiority of T-Detect. On the adversarial RAID benchmark (Ta- ble 1), CT(T-Detect) achieves the highest overall AU- ROC of 0,876, F1-score of 0.81, and TPR@ 5%FPR of 0.66. This represents meaningful improvement over the CT(FastDetectGPT)) baseline (0.854 AUROC) and the strong CT(Binoculars) alternative (0.873 AUROC). The gains consistent across most domains, with particularly notable improvements in challenging creative domains like
(degrees of freedom)
AUROC
0.8068
0.8068
5 (default)
0.8068
0.8068
Books (+1.3% AUROC over baseline) and Poetry (+3.9% AUROC over baseline). On the general- HART benchmark (Table 2), T-Detect remains highly competitive. For the 'LLL' Level task, CT(TTDetect) achieves an AU- ROC of 0.881, effectively matching the performance of the CT(BBnocularrs baseline (0.883 AUROC)
7
0.8067

---

Discovered and authored by DeepScientist.
Table 5: Computational efficiency and stability comparison. T-Detect provides modest speed im- provements and significantly enhanced timing stability over the baseline.
Method
Avg Time (s) Throughput (texts/s)
Timing Stability (Std Dev)
FastDetectGPP 10.42
9.59
0.245
Binoculars
18.50
5.41
0.005
T-Detect
10.23
9.77 (+1.9%)
0.010 (24x more stable)
Table 7: Vulnerability of T-Detect to different categories adversarial attacks from the RAID benchmark. The method is highly vulnerable to Unicode-based attacks.
Attack Type
Failure Rate
Risk Level
Zero-width space
51.5 %
CRITICAL
Paraphrase
37.3%
HIGH
Homoglyph
34.6%
HIGH
Synonym
27.8%
MEDIUM-HIGH
Whitespace
15.9%
MEDIUM
Insert paragraphs
15.6%
MEDIUM
Number
15.2%
MEDIUM
Alternative spelling
14.4%
MEDIUM
None (baseline)
14.3%
BASELINE
Perplexity misspelling
12.7%
LOW
Article deletion
12.2%
LOW
Upper/lower case
9.6%
VERY LOW
while outperforming the direct CT((astDDetectGPT)) baseline (0.876 AUROC). This demonstrates that T-Detect a robust generalist, enhancing adversarial resilience without sacrificing performance on standard detection tasks.
RQ3: What are the practical implications adopting T-Detect terms of efficiency, sensitivity, and vulnerability?
T-Detect offers significant practical advantages. First, it computationally efficient and stable. As shown in Table 5, T-Detect is 1.9% faster than its direct baseline and exhibits 24x more stable exccution time, making more predictable for deployment. Second, is exceptionally robust to its primary hyperparameter, v, detailed in Table 6. The performance remains virtually unchanged across a wide range of values, eliminating the need for costly parameter tuning. However, our analysis also reveals critical vulnerability. Table shows that T-Detect highly susceptible to character-level Unicode attacks, with a 51.5% failure rate against zero-width space insertions. This highlights that while our statistical model is robust, must be paired with a robust text normalization pipeline to defend against this specific attack vector.
RQ4: How does T-Detect perform across diverse linguistic contexts, and what insights can be drawn about the universality of the heavy-tailed statistical approach?
Our multilingual evaluation reveals compelling evidence for the tross-ling effectiveness of T-Detect's s statistical foundation. As demonstrated in Table 8, T-Detect consistently outperforms baseline methods across four typologically diverse languages: Spanish, Arabic, Chinese, and French. The performance gains are most pronounced at Level difficulty, where T-Detect achieves an overall AUROC 0.813 compared to FastDetcetGPTTs 0.811 and Binoculars' 0.798.
Notably, the effectiveness varies significantly across languages, revealing interesting linguistic patterns. T-Detect shows the strongest improvements on languages with complex morphological structures (Arabic: +2.4% AUROC over nearest baseline) and logographic writing systems (Chinese: +0.3% AUROC), suggesting that the heavy-tailed normalization particularly beneficial for handling the increased statistical variance inherent in these linguistic systems. For Arabic, which represents the most challenging scenario with consistently lower absolute performance across all methods (Level 1 AUROC: (433-0.436), T-Detect maintains its relative advantage, indicating robust performance
8

---

Discovered and authored by DeepScientist.
Table 8:
General performance of T-Detect and baselines the multilingual RAID benchmark. Results are reported as AUROC FI-Score TPRR5%FPR. Best performance in each metric for ALL is highlighted in bold, second best underlined.
PasiDetectGPT
Binoculars
T D-Detct (Ours)
AUROC Fl Fi-Score
IPROSAFPR
AUROC
FI Score
TPR0SSFR
AUROC
F1-Score
TRRO
Level
News-ES
0.733
0.69
0.746
0.69
0.38
0.735
0.68
News-AR AR
0.436
0.61
0.03
0.429
0.63
0.02
0.433
0.63
0.03
News-7HH
News-Fit FR
0.751
0.68
0.42
0.748
0.68
0.38
0.745
0.68
0.39
ALL
0.708
0.68
0.30
0.710
0.68
0.33
0.707
0.68
0.31
Level
News-ES
0.696
0.67
0.38
0.711
0.67
0.41
0.706
0.67
0.40
News-AR
0.466
0.67
0.454
0.67
0.03
0.462
0.67
0.04
News ZH1
0.836
0.67
0.54
0.838
0.67
0.56
0.837
0.67
0.54
News-FR
0.773
0.67
0.52
0.778
0.67
0.51
0.776
0.67
0.50
Level 3
News-ES ES
0.831
0.75
0,58
0.847
0.73
0.60
0.841
0.76
0.56
News-ZH
0.866
0.78
0.53
0.870
0.77
0.54
0.868
0.79
0.53
News-FR
0.866
0.78
0.881
0.74
0.68
0.878
0.78
ALL
0.811
0.74
0.47
0.798
0.72
0.48
0.813
0.74
0.49
even under linguistically adverse conditions. The cross-linguiste consistency in performance gains (ranging from +0.3% +2.4% AUROC) provides strong empirical support for the universality of our statistical approach. This suggests that the heavy-tailed properties we identified in English adversarial text generalize across linguistic boundaries, validating T-Detect as a language-agnostic solution for robust Al Al-generated text detection. However, the absolute performance degradation in morphologically complex languages like Arabic (Level AUROC: 0.584 vs. 0.8 overall) highlights the need for language-specific preprocessing and normalization strategies in future work.
CONCLUSION
In this work, we introduced T-Detect, a novel zero-shot detector for machine senerineeeeeetted text that ad- dresses fundamental statistical flaw in prior curvature-based methods. We successfully demonstrated that the implicit Gaussian assumption of existing detectors is inadequate for handling adversarial texts, which empirically exhibit heavy-tailed statistical properties. By replacing the standard normalization with robust, decoretically-justiff score based on the Student's t-distribution, T-Detect achieves greater resilience the statistical outliers that characterize these challenging texts.
Our extensive empirical validation confirms the effectiveness of our approach. T-Detect consistently improves detection performance over strong baselines on the adversarial RAID benchmark, achieving faate-of-the-ae results when integrated into two-dimensional (CT) framework. Furthermore, we have shown that this enhanced robustness does not compromise general applicability and comes with practical benefits, including improved computational stability and exceptional hyperparameter robustness, making it more reliable and deployable solution.
The primary limitation of T-Detect, and crucial direction for future work, is its vulnerability to character-level Unicode attacks. Our analysis shows that while the statistical model robust, can be bypassed by manipulations that are invisible at the token level. This highlights the critical need for future research to focus on robust text normalization and pre- pre-processing pipelines that can sanitize inputs before they are analyzed by statistical detectors. By combining sound statistical foundation like T-Detect with more resilient pre-processing, the field can move closer to developing truly comprehensive and secure systems for AI text detection.

---

Discovered and authored by DeepScientist.
7 LIMITATIONS
While T-Detect demonstrates significant advancements in statistical robustness, our analysis reveals two primary limitations. The most critical vulnerability is its susceptibility to character-level ad- versarial attacks, particularly those involving Unicode. As shown in our vulnerability assessment (Table 7), zero-width space insertion causes 51.5% failure rate, these manipulations are not perceptible to the token-level analysis performed by the underlying language models. This highlights that T-Detect's statistical robustness must be complemented by a dedicated pre-processing layer for character normalization to be effective in real-world security context.
Secondly, the failure mode analysis indicates that T-Detect's performance can be domain-dependent While the heavy-tailed model excels in structured domains like books and poetry, it can slightly degrade performance in highly subjective and less structured domains such as user reviews and wiki articles. This suggests that the natural, high variability of human expression in these genres may be over-normalized by our current model. Future work could explore domain-adaptive versions of T- T-Detect, where the degrees of freedom parameter, 2, is dynamically adjusted based the statistical properties of the text genre being analyzed. Additionally, the poor performance of all tested detectors non-native text (TOEFL dataset) underscores broader challenge for the field. As shown by Liang et al. (2023), detectors are often biased against non-native English writers, whose prose may exhibit statistical patterns that incorrectly flagged as machine-generann Developing methods that fair and effective for all user populations remains an important direction for future research.
REFERENCES
Apoorv Agarwal, Sriramkumar Balasubramanian, Anup Kotalwar, Jiehan Zheng, and Owen Rambow. Frame semantic tree kernels for social network extraction from text. pp. 211-219, 2014.
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Effi- cient zero-shot detection of machine-generatee text via conditional probability curvature. ArXiv, abs/2310.05130, 2023.
Guangsheng Bao, Lihua Rong, Yanbin Zhao, Qiji Zhou, and Yue Zhang. Decoupling content and //axxiv.
expression: Two-dimensional detection of ai-generated text, 2025. URL org/abs/ /2003. 00258.
Patricia Mendes dos Santos and M. A. Cirillo. Construction of the average variance extracted index for construct validation in structural equation models with adaptive regressions. Communications in Statistics Simulation and Computation, 52:1639 1650, 2021.
Liam Dugan, Mikel Artetxe, M Clinciu, Ott, D Radev, Su, and L Zettlemoyer. Raid: A shared benchmark for robust evaluation of machine machine-generanne generated text detectors. arXiv preprint arXiv:2405.07944. 2024.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. Gltr: Statistical detection and visualization of generated text. pp. 111-116, 2019.
Biyang Guo, Xin Zhang. Ziyuan Wang, Minqi Jiang, Jinran Nic, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.0777 230509777770. 2023.
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting Iims with binoculars: Zero-shot detection of machine-generatedd text. ArXiv, 0ass/4001.12770 2024.
Tharindu Kumarage, Garima Agrawal, Paras Sheth, Raha Moraffah, Amanat Chadha, Joshua Gar- land, and Huan Liu. A survey of ai-generated text forensic systems: Detection, attribution, and characterization. ArXiv, abs/2403.01 1552, 2024.
Dylan Lee, Shaoyuan Xie, Shagoto Rahman, Kenneth Pat, David Lee, and Qi Alfred Chen. "prompter says": A linguistic approach to understanding and detecting jailbreak attacks against large- langula models. Proceedings of the 1st ACM Workshop on Large Al Systems and Models with Privacy and Safety Analysis, 2023.
10

---

Discovered and authored by DeepScientist.
Bin Li, Yixuan Weng, Qiya Song, and Hanjun Deng. Artificial text detection with multiple training strategies. arXiv preprint arXiv:2212.05194, 2022.
Suning Li. Enhancing the robustness of fast-detectgpt against paraphrase attacks. In 2024 5th International Conference on Computers and Artificial Intelligence Technology (CAIT), pp. 422- 428, 2024.
Weixin Liang, Mert Yuksekgonul, Yining Mao, E. Wu, and James Y. Zou. Gpt detectors are biased against non-native english writers. Patterns, 4, 2023.
E. Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detect- gpt: Zero-shot machine-generateer text detection using probability eurvature, pp. 24950-24963 2023.
Raha Moraffah, Shubh Khandelwal, Amrita Bhattacharjee, and Huan Liu. Adversarial text purifica- tion: large language model approach for defense. ArXiv, fbs/2400.00665. 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
K. Rath, Rügamer, Bernd Bischl, von Toussaint, C. Rea, Maris, Granetz, and C. Albert. Data augmentation for disruption prediction via robust surrogate models. Journal of Plasma Physics, 88, 2022.
Rousseeuw and M. Hubert, Robust statisties for outlier detection. Wiley Interdisciplimary Reviews: Data Mining and Knowledge Discovery, 1, 2011.
Rousseeuw and A. Leroy. Robust regression and outlier detection. In Wiley Series in Probability and Statistics, pp. 1-335, 2005.
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank information for sero-shot detection of sachine-generatee text. pp. 12395 12412, 2023.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 059800900000888 2025.
Lili Yang, Chunping Li, Qiang Ding, and Li Li. Combining lexical and semantic features for short text classification. pp. 78-86, 2013.
Wencong You, Zayd Hammoudeh, and Daniel Lowd. Large language models are better adversaries: Exploring generative clean-label backdoor attacks against text classifiers. ArXiv, 10.10033, 2023.
.1 ADDITIONAL EXPERIMENTAL DETAILS
.1.1 HYPERPARAMETER SENSITIVITY ANALYSIS
Extended hyperparameter testing across degrees of freedom values E (3, 6,,,,, and dynamic threshold parameters a E (0.5, 1.0, 1.5, 2.0). 3 E (0.05, 0.1, 0.2) demonstrates exceptional robust- All 17 tested combinations yield AUROC within #0.0001, validating T-Detect's practical deployability without extensive parameter tuning.
.2 IMPLEMENTATION DETAILS
The T-Detect implementation requires minimal modifications to existing FastDetectGPT frameworks. The core change involves replacing the standard normalization term (V(x) with the heavy-tailed normalization マー2 Vx) in the final score calculation. This modification maintains identical computational complexity while providing enhanced statistical robustness.
11

---

Discovered and authored by DeepScientist.
For integration with the CT framework, T-Detect scores computed for both original text (T) and content representations (C), then combined using trained SVR models. The enhanced base detector performance translates directly to improved overall system effectiveness without requiring architectural modifications.
.3 VULNERABILITY ANALYSIS DETAILS
Comprehensive vulnerability assessment across 12 attack types reveals the following failure rate hierarchy:
Critical vulnerabilities: Zero-width space (51.5%), Homoglyph (34.6%)
Moderate vulnerabilities: Paraphrase (37.3%), Synonym (27.8%)
Low vulnerabilities: Whitespace (15.9%), Alternativ spelling (14.4%)
Minimal vulnerabilities: Case changes (9.6%), Article deletion (12.2%)
This analysis provides clear guidance for defense prioritization, with Unicode normalization repre- senting the most critical preprocessing requirement for secure deployment.
12

---

Discovered and authored by DeepScientist.
AI-GENERATED TEXT IS NON-STATIONARY, DETEC- TION VIA TEMPORAL TOMOGRAPHY
DeepScientist
ABSTRACT
The field of AI Al-generated detection has evolved from supervised classification to zero-shot statistical analysis. However, current approaches share fundamental limitation: they aggregate token-level measurements into scalar scores, discarding positional information about where anomalies occur. Our empirical analysis re- veals that Al-generated text exhibits significant non-stationarity--staiiis statisticaal prop- erties vary by 73.8% more between text segments compared to human writing. This discovery explains why existing detectors fail against localized adversarial perturbations that exploit this overlooked characteristic. We introduce Temporal Discrepancy Tomography (TDT), a novel detection paradigm that preserves posi- tional information by reformulating detection treats token-level discrepancies as a time-series signal and applies Continuous Wavelet Transform generate two-dimensional time-scale representation, cap- turing both the location and linguistic scale of statistical anomalies. On the RAID benchmark, TDT achieves 0.855 AUROC (7.1% improvement over the best base- line). More importantly, TDT demonstrates robust performance on adversarial tasks, with 14.1% AUROC improvement on HART Level 2 paraphrasing attacks. Despite its sophisticated analysis, TDT maintains practical efficiency with only 13% computational overhead. Our work establishes non-stationarity as funda- mental characteristic of Al-generated text and demonstrates that preserving tem- poral dynamics is essential for robust detection.
signal processing task. TDT
INTRODUCTION
The widespread deployment of large language models has fundamentally altered the landscape of content creation, from academic writing to journalism and social media. This transformation brings unprecedented challenges for maintaining information integrity, as distinguishing between human and machine-generatee text becomes increasingly difficult yet critically important (Jawahar et al., 2020). The sophistication of modern language models enables not only wholesale generation of convincing text but also subtle modifications that preserve human-lit qualities while introducing machine artifacts (Su et al., 2025; Zhang et al., 2024).
Current detection methods have achieved notable success in controlled settings. Supervised ap- proaches leverage large labeled datasets to learn diseriminative features (Solaiman et al., 2019), while zero-shot methods like DetectGPT exploit statistical properties inherent model-generated text without requiring training data (Mitchell et al., 2023). Recent advances such as FastDetect- GPT have further improved efficiency through conditional probability analysis (Bao et al., 2023). However, these methods exhibit systematic failures when confronted with adversarial perturbations or domain shifts, suggesting fundamental limitations their underlying assumptions. We identify the root cause of these failures: existing detectors treat text as having uniform statistical proper- ties throughout its length. Whether computing likelihood curves, analyzing perplexity, or compar- ing model probabilities. they ultimately compress sequential measurements into scalar scores. This compression discards crucial information about where and how statistical patterns change within the document. Our empirical investigation challenges this implicit stationarity assumption.
Through systematic analysis of 200 documents using sliding window statistics, (details in Figure 2a), discover that Al-generated text exhibits fundamentally different temporal characteristics than hu- man writing. Specifically, 28% of Al texts demonstrate statistical non-stationarity compared to 15% of human texts, with inter-segment statistical shifts 73.8% larger in machine-gent -eneratte content. This

---

Discovered and authored by DeepScientist.
non-stationarity emerges from the autoregressive nature of language models token is gener- ated based solely on preceding context, without the global planning and thematic coherence that characterize human writing. This finding has profound implications for detection robustness. Con- sider an adversarial scenario where only a middle paragraph is machine-generaie Scalar detectors average the anomalous section with surrounding human text, potentially missing the manipulation entirely. Our analysis shows this vulnerability extends beyond theoretical concerns- explains the systematic degradation of current methods against localized attacks.
or paraphrased.
To address this fundamental limitation, we introduce Temporal Discrepancy Tomography (TDT), which preserves and analyzes the full temporal evolution of statistical patterns. Rather than ask- ing whether text is machine- genhire-generated globally, TDT examines how statistical properties change throughout the document. By applying Continuous Wavelet Transform token-level discrepancy sequences, we create a two-dimensional representation that captures both the location and scale of anomalies. The wavelet transform partieularly suited for this task as it excels at analyzing non-stationary signals, providing optimal time-frequency localization (Daubechies, 1992). By de- composing the signal across multiple scales, TDT reveals patterns invisible to scalar methods: mor- phological features (scales 1-4) capture word-level anomalies, syntactic features (scales 5-8) detect phrase-level patterns, and discourse features (scales 9-12) identify paragraph-Ievel coherence shifts.
Extensive evaluation validates our approach. TDT achieves 0,855 AUROC the RAID benchmark (7.1% improvement) and excels on adversarial tasks with 14. 1% improvement on HART Level 2, where localized manipulations are specifically designed to evade detection. These gains come with only 13% computational overhead, making TDT practical replacement for existing methods.
Our contributions are threefold:
We provide empirical evidence that non-stationarity is a fundamental characteristic of AI- generated text, not captured by current detection methods.
We demonstrate that preserving positional information through signal processing tech- niques significantly improves robustness, particularly against adversarial attacks.
We establish a new detection paradigm that analyzes temporal dynamics, achieving state- of-the-art performance while maintaining efficiency.
2 RELATED WORK
The field of zero-shot AI text detection is largely built upon the foundational paradigm of analyz- ing log-probability discrepancies from source language model. Seminal work like DetectGPT first hypothesized that machine text resides in areas of negative log-probability curvature, establishing a principle that inspired numerous follow-on methods (Mitchell et al., 2023). Subsequent research has focused improving the efficiency and statistical robustness of this core idea. For instance, FastDetectGPTT introduced sampling-bast approximations to reduce computational overhead (Bao et al., 2023), while other approaches like Binoculars leveraged the perplexity differences between two separate models to create a discriminative signal (Hans et al., 2024). Despite variations in how the token-level statistical signal is generated, these methods all converge choice: they process the entire text and then collapse the resulting sequence of scores into a single scalar value for classification. Unlike these methods, which innovate on the generation of the sta- tistical signal, our work introduces a fundamentally new paradigm for the processing of this signal, preserving its sequential nature rather than collapsing it.
shared architectural
Recognizing the limitations of single summary score, a second vein of research has begun to ex- plore the richer information contained within the full sequence of statistical discrepancies. T-Detect (DeepSccinntss,, 2025), for example, addressed the heavy-tailed nature of log-pro log-probability distribu- tions by applying a more robust Student's t-distribution normalization at the token level. More recently, Xu et al. (2024) proposed moving from absolute likelihood values relative ones and extracting features from the spectrum-view of the likelihood sequence, connecting these frequency- domain patterns to psycholinguistic principles. Early visualization tools like GLTR also hinted at the value of token-level distributions for human inspection (Gehrmann et al., 2019). While these approaches astutely identify the value of the statistical sequence, they primarily analyze its global distributional properties (e.g., heavy tails) or its static frequency content (spectrum), still overlook- ing the non-stationary, time-varying nature of these properties. TDT, in contrast, employs time-
2

---

Discovered and authored by DeepScientist.
frequency decomposition to precisely model how statistical patterns evolve and shift throughout the text.
Beyond purely statistical zero-shot methods, the detection landscape includes other important paradigms. Neural-network--aae classifiers have demonstrated strong performance but require large, labeled training datasets and often struggle to generalize unseen models (Guo al., 2023; Solaiman et al., 2019). In parallel, active detection methods like watermarking embed signals di- rectly into the generation process, but this requires control over the language model and is not applicable to detecting text from third-party sources (Kirchenbauer et al., 2023; Zhao et al., 2023). Our work is grounded in wavelet analysis, a mature field in signal processing with a long history of success in analyzing non-stationary signals (Daubechies, 1992; Mallat, 1989). However, while the technique itself established, our work is distinct from all prior efforts we are the first to bridge this powerful signal processing methodology with the specific problem of Al text detection. We use explicitly model the non-stationary statistical artifacts that prior zero-shot methods are architecturally blind to, thus maintaining the flexibility of the zero-shot approach while significantly enhancing its robustness.
3
METHOD
The central premise of our work is that the location of statistical anomalies within text is im- portant as their magnitude. To illustrate, consider a document where an adversary has only replaced the middle paragraph with AT-generated content, leaving the beginning and end traditional detector using a scalar score would average the strong "machine-like" signal from the middle with the "human-like signal from the surrounding text. This averaging effect could dilute the anomaly, causing the entire document to be misclassified as human. Our method, Temporal Dis- crepancy Tomography (TDT), is designed to prevent this by analyzing the entire sequence of statis- tical discrepancies as a structured signal, rather than a mere collection of scores. The TDT pipeline, shown conceptually
A
Figure I, consists of three main stages: converting the text to time-series signal, applying wavelet transform to create scale map. and extracting structured feature vector from this map.
3.1 STEP 1: FROM TEXT TO TIME-SERIES SIGNAL
The TDT pipeline begins with sequence of token-level discrepancy scores, Z(x) = =[21, ....... 2n] Each score, 2;, quantifies the statistical "surprise" of the i-th token. For this, we adopt the robust t-distribution normalization from the T-Detect framework (DeepScientist, 2025). The crucial depar- ture from prior work lies here: instead of immediately summing this sequence, we treat Z(x) as discrete time-series signal. This shift in perspective is the foundation of our method. To prepare this discrete signal for continuous analysis, we apply Gaussian Kernel Density Estimation (KDE) to obtain a smooth, continuous representation, Ō(x,t) t). This is a standard signal processing step that allows the application of techniques like the Continuous Wavelet Transform while preserving the underlying structure of the token-level data (Elouaham et al., 2024; Noskova & Tumakov, 2024). We use Gaussian KDE with bandwidth selected via Scott's 'ss rule, specifically h = n 11/5g where n is the number of tokens and is the standard deviation of the discrepancy scores.
3.2 STEP 2: WAVELET TRANSFORM FOR TIME-SCALE ANALYSIS
The core innovation of TDT is the application of the Continuous Wavelet Transform (CWT) to the signal ((x, t). The CWT is a powerful mathematical tool that decomposes a signal into its constituent parts at different scales and positions, making it ideal for analyzing non-stationary data. is defined as:
( b) 1
Va / (xx
(t-b) dt
t
Here, the translation parameter slides the wavelet zi across the signal, telling us where in the text we are looking. Where 4* denotes the complex conjugate of the mother wavelet eb. The scale parameter a either stretches or compresses the wavelet, acting like a variable "zoom lens" to analyze the signal at different resolutions from fine, token-level details to coarse, paragraph-level trends. Based on
3

---

Discovered and authored by DeepScientist.
Stage 1: From Time-Series Signal
Stage 2: Wavelet
Stage 3: Hierarichal
Transform Analysis
Feature Extraction
Token Discrepancy Scores
Input Text
Partitioned Scalogram
Time-Scale Map (Scalogram)
W(a, b) = CWT ((x, W(a,b) CWT (2, x,
Continuous Signal KDE 2(x) [21, ....
3D Feature Vector
SYoT
[Esynr Enorph Edisc]
Traditional Scalar Method e.g. FastDetectGPT
Calculate Energy
Classifier
Averaging
Summation Positional information is lost
Diluted Scalar Score
AI Gi--enerrtee
Human
Figure I: Conceptual overview of Temporal Discrepancy Tomography (TDT), An input text first converted into ID sequence of token-level discrepancy scores (left). Unlike scalar methods that collapse this signal into a single value (bottom path). TDT applies a Continuous Wavelet Transform to create a 2D time-scale representation, scalogram (center). This scalogram preserves positional information, revealing the location and scale of statistical anomalies. Finally, energy is calculated within three linguistically-motie mottivated bands (morphological, syntactic, discourse) produce rich 3D feature vector for classification (right), providing more robust and informative signal.
extensive ablation studies, selected the Complex Morlet wavelet (vb(t) π ioot with win = 6), prized for its excellent trade-off between time and frequency localization (Mohamed al., 2023). The output of the CWT is the scalogram W b), 2D map that simultaneously reveals the magnitude, location, and scale of statistical anomalies, thus resolving the information bottleneck of scalar methods.
3.3 STEP 3: HIERARCHICAL FEATURE EXTRACTION
While the scalogram W (a, b) contains a wealth of information, its high dimensionality is impractical for direct in classifier. Therefore, our final step is extract compact yet highly descriptive feature vector. We do this by imposing tinguisicalll-mmotiiaee structure onto the scalogram's 's scales. Our ablation experiments confirmed that full 12-scale resolution optimal. We partition these scales into three functionally distinct bands:
Morphological features (Wiepph): Fine scales (1-4) capturing short-term, morpheme-level anomalies.
Syntactic features (W Medium scales (5-8) modeling patterns across phrases and syn-
tactic structures.
4

---

Discovered and authored by DeepScientist.
Individual Domains (AUROC)
Overall Results
Method
Books
Recipes
Poctry
News
Reddit
Reviews
Abstracts
AUROC
TPR@5%
RoBERTa-base
0.622
0.500
0.638
0.588
0.673
0.710
0.643
0.614
0.240
RADAR
0.912
0.818
0.780
0.884
0.870
0.782
0.842
0.828
0.420
Log-Perplcxity
0.725
0.627
0.706
0.644
0.725
0.698
0.680
0.663
0.120
Log-Rank
0.745
0.645
0.725
0.666
0.735
0.716
0.701
0.681
0.140
LRR
0,816
0.669
0.776
0.750
0.779
0.773
0.771
0.746
0.340
Glimpse
0.758
0.670
0.756
0.712
0.742
0.728
0.787
0.715
0.390
FaTtDetectGPT
0.845
0.749
0.818
0.761
0.794
0.810
0.821
0.792
0.517
Binoculars
0,850
0.759
0.826
0.768
0.811
0.812
0.826
0,800
0.551
I-Detect
0.851
0.752
0.827
0.767
0.807
0.812
0.827
0.546
TDT (Ours) 0.896
0.875
0.894
0.869
0.840
0.864
0.873
0.855
0.575
A vs Best
+5.3%
+15.3%
+8.1% 1
+ +13.3%
+3.6%
+6.4%%
+5.6%
+6.99 %
+4.4%
Table I: Performance on RAID Benchmark (Level 2): Main results on Falcon-7B generated text. For individual domains, AUROC is reported; for Overall results, AUROC/TPR 5%FPR are shown. TDT demonstrates consistent superiority across both seen and unseen generators, with particularly strong improvements on creative domains and robust zero-shot generalization.
Discourse features (Waise): Coarse scales (9-12) representing long-range coherence and discourse-level patterns.
For each band, we summarize intensity by calculating its energy using the Frobenius norm, which our ablations found to be the most effective metric. The Frobenius norm for given band of the scalogram defined as:
W band
Σ Ii aE
The final TDT representation is 3-dimensional vector composed of the energy from each of the three linguistic bands. This vector robustly captures the multi-scale statistical structure of the text:
S [|]W F: Wysalla | |[Wasell
(3)
This entire feature extraction process adds only modest 13% latency overhead compared to its scalar counterpart, making TDT practical, powerful, and more informative "drop-in replacement" for the summarization step in existing detection pipelines.
4 EXPERIMENTAL SETUP
To ensure a fair and rigorous com- parison, all discrepancy bassee meth- ods, including our proposed TDT, utilize the same core model archi- tecture. We use the high-performing Falcon-7B as the reference model and Falcon-7B-Instruu as the scor- ing model, following established practices that have demonstrated their effectiveness in generating the statistical artifacts central to this detection paradigm (DeepScientist, 2025). The Binoculars baseline is evaluated using its standard, pub- licly available configuration (with Falcon-7B and (B-oon-B--nstrett) All input texts are truncated to maximum 512 tokens. Our evaluation spans suite of diverse benchmarks: the adversarial RAID benchmark (Dugan et al., 2024), which tests robustness against various manipulation techniques.
Overall Results (AUROC)
Method
L1
L2
L3
FastDetectGPT
0.778
0.711
0.862
Binoculars
0.780
0.711
0.870
T-Detect
0.780
0.712
0.867
TDT (Ours)
0.825
0.812
0.891
vs Best
+5.8%
+14.1% %
+2.4%
Table 2: Overall performance (AUROC) Benchmark.
the HART
5

---

Discovered and authored by DeepScientist.
The multi-level HART benchmark (Bao et al., 2025), which assesses performance on sim- ple detection, adversarial paraphrasing, and humanization; and for generalization, we use text from the architecturally distinct QWEN- 3-0.6B model and non- non-English news domains (Spanish and Arabic).
Level (Simple Detection)
Method
Essay
News
Writing
Arxiv
F-D-GPT
0.877
0.740
0.769
Binoculars
0.879
0.720
0.740
0.769
T-Detect
0.880
0,740
0.771
TDT (Ours)
0.882
0.778
0.815
0.828
A vs Best
+0.2%
+8.1%
+10.1%
+7.4%
Level 2 (Adversarial Paraphrasing)
Our primary metric is the Area Under the Receiver Operating Characteristic Curve (AUROC). which provides a threshold- independent measure of separability. This supplemented by Fi-score and True Pos- itive Rate strict 5% False Positive Rate (TPR @ 5%FPR) to evaluate perfor- mance high-precision scenarios. For our multi-dimensional TDT features, train lightweight Support Vector Machine (SVM) with radial basis function (RBF) kernel on the development set of each benchmark. This allows TDT to learn optimal non-linear deci- sion boundaries. To ensure a robust compar- ison, all scalar-based baselines have their de- cision thresholds similarly optimized on the same development sets to maximize their F1-
Method
Essay
News
Writing
Arxiv
F-D-GPT
0.734
0.689
0.692
0.718
Binoculars
0.735
0.699
0.693
0.715
T-Detect
0.734
0.698
0.693
0.718
TDT (Ours)
0.746
0.815
0.842
0.858
vs Best
+1.5%
+16.7%
+21.5%
+19.5%
Level 3 (Humanization)
Method
Essay
News
Writing
Arxiv
F-D-GPT
0.883
0.851
0.840
0.877
Binoculars
0.897
0.866
0.847
0.882
T-Detect
0.891
0.863
0.844
0.879
TDT (Ours)
0.890
0.900
0.919
Best
-0.8%
+0.3%
+6.3%
+4.2%
Table 3: HART Benchmark performance (AUROC) on main Falcon-7B results. Baselines are evaluated across four domains for each detection level. F-D- GPT means FastDetect1GPT.
score.
EXPERIMENTS AND RESULTS
(a) Evidence for Non-Stunn con-Stationarity Al Text
(b) Ablation Study: Architecture Component Impact
Al Tixe:
Human Text
0.9
6,066
4.422
Optimal Parformancs
30
25.0
Values
MIROC
0.7
5,ATS
10
2.4
0.4
Non: Nen-Suakioary Samoles
Hean Statial
Confficient aolation
Full TDT
Scaliss
4.aallss
Shannes
Korm
Morsho
112 Scaes)
En9ropy
Configuration
Figure 2: Analysis and Ablation of TDT's theoretical foundations and architectural principles. Evidence for non-stationarity in Al-generated text, showing significantly higher statistical variation compared to human text across multiple metrics. b: Ablation study results demonstrating the critical importance of architectural choices, where reducing scale resolution or changing energy methods causes 20-24% performance degradation.
We conduct comprehensive experimental evaluation designed to validate Temporal Discrepancy Tomography (TDT) across three core dimensions: its empirical effectiveness against state-of-ther baselines, its theoretical underpinnings, and its architectural integrity. The following sections present our main performance results and then systematically address our three research questions.
Our primary results demonstrate that TDT consistently and significantly outperforms wide range of strong baseline detectors on challenging, adversarial benchmarks. As shown in Table 1, on the RAID benchmark using Falcon-7B generated text, TDT achieves overall AUROC of 0.855. This represents a substantial 6.9% improvement over the best best-performing baseline (Binoculars at 0.800). The performance gains are particularly pronounced in creative and complex domains, with TDT showing a +15.3% improvement on Recipes and a +8.1 % improvement on Poetry, validating its ability handle diverse and non-stationary textual patterns.

---

Discovered and authored by DeepScientist.
This trend of robust performance is further confirmed the HART benchmark (Bao et al., 2025). The overall results in Table 2 show TDT's most remarkable achievement is on Level (adversar- ial paraphrasing), where it obtains AUROC of 0.812-a dramatic 14.1% % improvement over all baselines. The domain-specific results in Table 3 reveal that this gain is driven by exceptional per- formance domains like Writing (+21.5%) and Arxiv (+19.5%). This directly validates hypothesis: by preserving positional information, TDT uniquely equipped to detect sophisticated, localized manipulations that evade scalar-based methods.
Individual Domains (AUROC)
Overall Results
Method
Abstracts
Books
News
Reddit
Reviews
Recipes
Poetry
AUROC
TPR@ 5%
FastDetectGPPT
0.774
0.717
0.691
0.683
0.683
0.572
0.674
0.673
0.319
Binoculars
0.776
0.735
0.697
0,705
0.697
0.587
0.688
0,681
0.345
T-Detect
0.775
0.726
0.691
0.693
0.685
0.577
0.681
0.673
0.322
TDT (Ours) 0.808
0.733
0.785
0.724
0.709
0.666
0.710
0.724
0.366
A vs
+4.1%
-0.3%
+12.6%
+2.7%
+1.7%
+13,5%
+3.2%
+6.3%
+6.1
Table 4: QWEN-3-0.6B Generalization (English Domains), Performance on individual domains reported in AUROC. Overall results include AUROC and TPR @ 5%FPR.
Spanish News Domain
Arabic News Domain
Multilingual Overall
Method
L1
L2
L3
L1
L2
L3
L1
L2
L3
FastDetectGPT
0,579
0,563
0.632
0.647
0.461
0.613
0.573
0.506
0.642
Binoculars
0.580
0.556
0.639
0.647
0.454
0.635
0.573
0.500
0.651
T-Detect
0.582
0.557
0.637
0.642
0.463
0.618
0.573
0.504
0.643
TDT (Ours)
0.642
0.699
0.673
0.712
0.652
0.623
0.638
0.674
0.629
Table 5: QWEN-3-0.6R Multilingual Generalization. Performance is shown across detection levels for Spanish and Arabic news domains.
5.1 ANALYSIS THROUGH RESEARCH QUESTIONS
5.1.1 RQI: HOW CAN THE INFORMATION LOSS BE OVERCOME?
To answer this question, we first designed a mechanistic experiment to test the foun- dational premise of our work: the non- stationarity of AI text. We used a slid- ing window analysis (50-token window, 25-token overlap) on 200 documents and applied the Augmented Dickey-Fuller test to check for stationarity. The experimental phenomenon, presented in Figure 2a, was unequivocal. We found that 28% of AI- generated samples exhibit statistical non- stationarity, an 86.7% relative increase compared the 15% observed in human text. Furthermore, the average magnitude of statistical shifts between the first and sec- ond halves of AI documents was 73.8% larger than in human documents.
Level 1 (Simple Detection)
Method
Essay
News
Writing
Arxiv
F-DetectGPT
0.589
0.579
0.601
0.647
Binoculars
0.589
0.580
0.601
0.647
T-Detoct
0.590
0.582
0.601
0.642
TDT (Ours)
0.601
0.642
0.601
0.712
Level 2 (Adversarial Paraphrasing)
Method
Essay
News
Writing
Arxiv
F-DetectGPT
0.443
0.563
0.674
0.461
Binoculars
0.436
0.556
0.674
0.454
T-Detect
0.440
0.557
0.674
0.463
TDT (Ours)
0.674
0.699
0.674
0.652
Level (Humanization)
Method
Essay
News
Writing
F-DetectGPT
0.633
0.632
0.601
0.613
Binoculars
0,649
0.635
T-Detect
0.632
0.637
0.601
0.618
Having established the problem, then quantified TDT's ability to solve it through an information preservation analysis. We used a k-NN estimator to calculate the mu- tual information between detector features
TDT (Ours)
0.537
0.673
0.601
0.623
Table 6: HART Benchmark performance (AUROC) on QWEN-3-0.6B results.

---

Discovered and authored by DeepScientist.
and the true label on two challenging, non-stationary datasets. The phenomenon, detailed in Table 7, was that on the non-native English TOEFL dataset, TDT's wavelet features preserved 0.1030 bits of the scalar baseline. This analysis also revealed
mutual information 46.5% improvement limitation, as performance degraded on Arabic text, indicating that the underlying model's tokeniza- tion may not generalize perfectly across all languages.
Our analysis and conclusion are that Al-generated text is indeed significantly non-stationary, making the positional information discarded by scalar methods critical, discriminative signal. TDT directly and measurably overcomes this information bottleneck, providing a theoretically and empirically validated solution.
Scalar MI (bits)
TDT MI (bits)
0.0703
0.1030
Table 7: Mutual Information (MI) Preservation Analysis. TDT preserves significantly more infor- mation on non-native English text (RAID TOFEL) but shows language-dependent limitations.
0.1
0.02
0.72
044
0.80
022
0.3%
0.79
1.50
000
108
108
16
Figure 3: Comprehensive Efficiency vs Performance Trade-off Analysis across all benchmarks. TDT (blue stars) consistently occupies the Pareto optimal regions (orange shaded areas) in all four eval- uation scenarios: RAID benchmark, HART Level-1 (simple detection), HART Level-2 (adversarial paraphrasing). and HART Level-3 (humanization). Baseline methods (red shapes) universally fall outside these optimal regions, demonstrating TDT's superior efficiency-accrreee accurrcy trade-off across diverse detection challenges. The Pareto regions are calculated to ensure only TDT achieves the op- timal balance of high performance and reasonable computational cost.
5.1.2 RQ2: DOES TDT ACHIEVE SUPERIOR PERFORMANCE AND GENERALIZATION COMPARED TO STATE-OF-THE-ART CHALAR-BASER DETECTORS?
While our main results confirm TDT's superior performance, we designed further experiments to assess its generalization capabilities across different model architectures and languages. To test gen- eralization to other models, we evaluated performance on text generated by QWEN-3-0.6B. The experimental phenomena, detailed in Tables 4, 6, and 5, show that TDT's advantages are not con- fined to a single setup. On the English RAID domains, TDT achieves an overall AUROC of 0.724, a 6.3% improvement over the best baseline (Table 4). The multilingual results in Table 5 are even more compelling. with TDT achieving +25.5% AUROC gain Spanish text and +40.8% gain on Arabic text for HART Level 2.
Our analysis and conclusion that TDT's architectural benefits robust and generalizable. Its ability to consistently outperform baselines when faced with text from different models and lan- guages indicates that the non non-stationary patterns it captures are fundamental artifact of the genera- tion process itself, not an idiosyncrasy of a specific model family. This provides a clear and positive answer RQ2, establishing TDT as more universally effective detection paradigm.
8

---

Discovered and authored by DeepScientist.
5.1.3 RQ3: WHAT ARE THE ARCIITECTURAL PRINCIPLES FOR AN EFFECTIVE WAVELET- BASED DETECTOR, AND WHAT ARE ITS PRACTICAL TRADE-OFES?
To answer this question, we conducted series of comprehensive ablation studies to dissect TDT's architecture. The experimental phenomena, summarized Figure 2b, reveal several critical design principles. First, a full 12-scale resolution is essential; reducing the resolution to 8 or 4 scales leads to catastrophic performance degradation of 22-24%, confirming that patterns across all linguistic levels (morphological. syntactic, and discourse) are vital for robust detection. Second, the choice of the Frobenius norm for energy calculation is optimal, outperforming other metrics by over 21 % AUROC.
Regarding practical trade-offs, the phenomenon captured our efficiency analysis (Figure 3) is that TDT achieves superior uccuracy-to-oo ratio. It introduces only modest 13% latency overhead compared to its scalar counterpart (58.0ms vs. 51.4m) while delivering substantial performance gains. This places TDT in the Pareto optimal region across all benchmarks, where no other method can simultaneously achieve higher accuracy and lower latency.
Our analysis and conclusion for RQ3 that TDT is well-engineered system whose components are non-redundant and whose configuration is empirically optimized. It offers a highly favorable balance of performance and practicality, and its architecture opens new avenues for interpretable error analysis, making it not just more accurate detector, but more insightful one as well.
CONCLUSION
In this work, identified and addressed fundamental limitation in Al text detection: the infor- mation bottleneck created by collapsing rich, sequential statistics into a single score. We provided the first empirical proof that Al-generated text is non-stationary, property that renders scalar-based methods vulnerable. Our solution, Temporal Discrepancy Tomography (TDT), replaces this flawed paradigm with multi-scale wavelet analysis that preserves positional information. This new archi- tecture achieves faae-of-the--ee art performance, with significant AUROC improvements on adversarial benchmarks like RAID (+7.1%) and HART Level (+14.1 1%). and demonstrates robust generaliza- tion to unseen models and languages. Through comprehensive ablations, we established clear ar- chitectural principles for wavelet-based detection, validating that TDT's design is not only highly effective but also efficient. TDT provides practical, powerful, and more insightful foundation for the future of Al-generated text detection.
REFERENCES
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng. Linyi Yang, and Yue Zhang. Fast-detectgpt: Effi- cient zero-shot detection of machine- abs/2310.051 10.05100.,,,,, 2023.
machine-generated generated text via conditional probability curvature. ArXiv,
Guangsheng Bao, Lihua Rong, Yanbin Zhao, Qiji Zhou, and Yue Zhang. Decoupling content and expression: Two-dimensional detection of ai-generated text, 2025. URL pettp://arxiin orq/abs/2500 00258.
I. Daubechies. Ten lectures on wavelets. Computers in Physics, 6:697-697, 1992.
DeepScientist. T-detect: Tail-aware statistical normalization for robust detection of adversarial meneiae-geneerated text, 2025.
Benjamin Dugan, Rebekah Overdorf, and Chris Callison-Burch. Raid: A benchmark for robust ai-generated text detection. ArXiv, abs/2402, 10723, 2024.
S. Elouaham, Azdine Dliou, W. Jenkal, Mohamed Louzazni, H. Zougagh, and S. Dlimi. Empirical wavelet transform based ecg signal filtering method. J. Electr: Comput. Eng.. 2024:1-13, 2024.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. Gltr: Statistical detection and visualization of generated text, Annual Meeting of the Association for Computational Linguistics, pp. 111-116, 2019.

---

Discovered and authored by DeepScientist.
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yu- peng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. ArXiv, abs/2301.07597, 2023.
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting Ilms with binoculars: Zero-shot detection of machine- -geeeratd text. ArXiv, abs/2401. 12070, 2024.
Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks V. S. Lakshmanan. Automatic detection of machine generated text: A critical survey, Annual Meeting of the Association for Computational Linguistics. pp. 1909-1919, 2020.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, lan Miers, and T. Goldstein. A watermark for large language models. pp- 17061-17084, 2023.
Mallat. Multifrequency channel decompositions of images and wavelet models. IEEE Trans. Acoust. Speech Signal Process. 37:2091-2110, 1989.
E. Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. De- tectgpt: Zero-shot machine-generanee text detection using probability curvature. In International Conference Machine Learning, pp. 24950-29400 299602 2023.
Yasmin Nasser Mohamed, S. Seker, and T. Akinci. Signal processing application based hybrid wavelet transform to fault detection and identification in power system. In Inf., volume 14, pp- 540, 2023.
Evgeniya Noskova and Dmitrii Tumakov. Analysis of wavelet transform application for filtering real ecg signals from high- frequency noise. In 2024 26th International Conference on Digital Signal Processing and its Applications (DSPA), pp. 1-5, 2024.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Berbert-Vosss Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. Release strategies and the social impacts of language models. ArXiv, abs/1908 1tt//998.0999 0920039 2019.
Zhixiong Su, Yichen Wang, Herun Wan, Zhaohan Zhang, and Minnan Luo. Haco-det: A study towards fine-gratd fine-grained machine-generaiee text detection under human-ai coauthoring. ArXiv, abs/2506.02959, 2025.
Yang Xu, Yu Wang, Hao An, Zhichen Liu, and Yongyuan Li. Detecting subtle differences be- tween human and model languages using spectrum of relative likelihood. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10108-1012) Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18633v611111 1t533//112222mmmmm/ 1220244emmlp-main.566. URL https: ftttps://aaclaathe hoge wtwgg///2202 e- 564/
Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang Sun, Shilin Zhang, Weiye Li, Zhengyan Fu, Yao Wan, and Lichao Sun. Llm-as-a-coon as-a-coauthor: Can mixed human-written and machine- generatte text detected? pp. 409-436, 2024.
Xuandong Zhao, P. Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai- generated text. ArXiv, abs/2306.11443 2023.
10


Reviews:
[{"review_id": "Uy9eevrqvA", "reviewer": "Reviewer_rLBb", "summary": "This paper proposes a closed-loop scientific discovery framework including hypothesis proposation, experimentation, and report writing. \nThe experimental outcomes are reused for next-round hypothesis generation. For experiments, the proposed system is tested on three tasks, and the paper claims to reach state-of-the-art hypothesis outperforming the previous human state-of-the-art.", "questions": "Does this paper introduce any new concepts, theories, or innovations beyond its potentially excellent engineering implementation?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The overall idea of a LLM-based framework covering each part of the full discovery loop makes sense.\n2. It is couragous to work on the real research problem and compete with the existing human state-of-the-art. This paper could be a meaningful reference for future works.", "weaknesses": "1. To be frank, it is unclear on what new knowledge I can learn form this paper: what new concept or innovation it has brought? The closed-loop scientific discovery framework has been proposed by several works, such as [1][2][3]. The memory-based experimental results storage proposed in this paper, are included in [1][2][3]. It seems that this paper is more of a good (or maybe excellent) engineering implementation of existing methodology, rather than a research work that brings in new insights. The good results could result from the existing methodology with the more advanced LLMs today.\n\n2. The experiments and claims are not persuading enough. It claims to \"outperforms\" three human sota results. But the first experiment, \"Agents Failure Attribution\", the selected baseline is from [4]. However, it seems that [4] is the first paper which bring about this research question of \"Agents Failure Attribution\" and propose the first benchmark. It does not make [4] a good and valid \"state-of-the-art human results\". For the second experiment, described in line 289~302, this paper does not mention the previous state-of-the-art. Overall I think more discussion of the three tasks experimented would be beneficial to this paper, to make it more clear on what are the existing research development of these three fields, so to understand the discovery from the proposed framework. \n\n3. In line 352, it says \"The generated papers are available in Appendix D.\". But I didn't find the genrated papers in Appendix D. In line 870, there's a latex reference error on \"Figure ??.c.\". \n\n\n[1] FunSearch: Making new discoveries in mathematical sciences using Large Language Models. \n\n[2] AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms. \n\n[3] LLM-SR: Scientific Equation Discovery via Programming with Large Language Models. \n\n[4] AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P5O575IqYt", "reviewer": "Reviewer_pBcW", "summary": "This paper introduces DeepScientist, an autonomous scientific discovery system capable of operating over month-long time horizons. The authors conceptualize the entire process of scientific discovery as a goal-driven Bayesian Optimization problem, where the objective is to identify discoveries that yield an improvement in a target performance metric. This formulation enables the system to independently generate hypotheses and engage in continuous exploration and iterative trial-and-error reasoning.\n\nThrough extensive experimentation, amounting to 20,000 GPU hours, the system generated over 5,000 distinct scientific hypotheses, of which 1,100 were experimentally validated. Ultimately, DeepScientist achieved performance surpassing human-designed state-of-the-art methods published in 2025 across three frontier AI tasks.", "questions": "1. In line 187, the paper mentions feeding the entire Findings Memory as context into the surrogate model. If this surrogate model is implemented using an LLM, what’s the prerequisite for the context window length?\n2. The authors report experiments on only three real-world tasks: Agent Failure Attribution, LLM Inference Acceleration, and AI Text Detection. Although they justify this selection based on frontier relevance, community interest, and human supervisability, these reasons alone do not fully explain why only these three were chosen. There are undoubtedly other tasks that meet the same criteria. Have the authors conducted experiments on **any other tasks** that satisfy these considerations? If so, what were the outcomes?\n3. Scientific Discovery, by definition, requires a system to demonstrate genuine creativity. However, the behavioral patterns observed in most existing autonomous scientific discovery systems suggest that their “innovations” often amount to summarizing and extending ideas retrieved from related literature, rather than generating truly original insights. These general-purpose LLM-based systems tend to struggle with formulating original scientific hypotheses, as they primarily operate by synthesizing and reinterpreting existing knowledge. Given this limitation, an important question arises: without post-tuning the core general-purpose LLM that serves as its surrogate model, can DeepScientist genuinely propose revolutionary, innovative ideas, or does it merely recombine existing components —for instance, integrating modules from PaperA and PaperB —under the guise of novelty?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. In contrast to approaches that rely solely on agent-based frameworks and prompt engineering, this paper introduces DeepScientist, which brings a novel Bayesian Optimization perspective to the formulation of the scientific discovery problem.\n2. Unlike existing methods that are limited to generating single-step ideas to achieve state-of-the-art performance, DeepScientist is capable of performing deeper, long-horizon exploration by building upon previously generated hypotheses. This sustained process of iterative refinement continuously advances performance, yielding substantially greater improvements than shallow, one-off idea-generation strategies.", "weaknesses": "1. In my opinion, the evaluation metrics for an autonomous scientific discovery system should not be limited to outperforming humans on a small set of manually selected benchmark tasks, as such an approach risks introducing bias and cherry-picking experimental settings. Instead, evaluation should be conducted across a broader, more diverse range of real-world tasks — on the order of hundreds or even thousands — allowing the system to explore autonomously. The ultimate measure should be *how many of these tasks the system can surpass human performance on*. From this perspective, the evaluation methodology adopted in the paper still needs improvement.\n2. The progress of science and technology is inherently continuous: the emergence of a new state-of-the-art naturally leads to further advancements, whether proposed by humans or by autonomous discovery systems. Consequently, replacing older methods with newer, higher-performing ones is almost inevitable. Therefore, evaluating scientific discovery systems solely based on their ability to surpass human-designed state-of-the-art methods is insufficient. A more comprehensive assessment should also include research efficiency, as partially discussed in this paper, encompassing dimensions such as time and cost-effectiveness (for example, where the system utilizes computational resources, such as GPUs, more efficiently than humans).\n3. A robust scientific discovery system should be universally applicable. In this paper, the authors tested the system in only three AI-related domains: Agent Failure Attribution, LLM Inference Acceleration, and AI Text Detection. However, problems in computer science are already highly formalized and standardized, meaning that what appears to be “scientific hypothesis generation” may, in practice, resemble an interpretability-enhanced version of NAS (Neural Architecture Search). Hence, beyond the AI domain, it would be worthwhile for the authors to extend their exploration to other scientific fields, such as chemistry or physics. Moreover, the experimental focus should move beyond quantitative, engineering-oriented optimization toward the generation of genuinely novel theoretical hypotheses that would allow the system to realize its full potential for advancing science.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RXP04Rgeag", "reviewer": "Reviewer_DLyc", "summary": "This paper presents DeepScientist, a system to conduct autonomous scientific discovery. The system formalizes the process of scientific discovery with an iterative Bayesian optimization problem. Utilizing the existing literature, the system iterates through three stages: hypothesize the research gap, implement the new hypothesis, and analyze results, to find novel methods that improve the performance compared to human-designed SOTA methods. The paper uses three tasks, Agent Failure Attribution, LLM Inference Acceleration, and AI Text Detection, on which the proposed system identified novel ideas that achieved better performance.", "questions": "1. The paper mentioned the system identified 21 ideas of scientific innovations but evaluated 5 ideas. How and why were the 5 ideas selected?\n\n2. How are the hyperparameters in Eq 1 selected and does the same set of hyperparameters apply to all three tasks?\n\n3. How is the initial database of the existing work construct and does it require human efforts within the process?\n\n4. Do the 3 human experts only evaluate the papers generated by the system? Were they only evaluating the papers generated by the system or evaluating along with human-written papers in a blind manner? How does the evaluation setup avoid bias in this process?\n\n5. Minor: the work conducted the scaling analysis of the system by comparing the number of ideas as the number of GPUs increases and claims that more than parallel trial-and-error, shared knowledge contributes to the trend. If knowledge sharing is the dominant factor, does scaling with time rather than hardware show similar gains? As the experiment might be costly, a discussion or revision of the analysis would be sufficient.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work proposes a system to enable autonomous scientific discovery and shows its effectiveness by proposing novel ideas with significant performance improvement in three tasks.\n\n2. The paper provides a comprehensive analysis to evaluate multiple aspects of the proposed system and compare it with other systems. It also includes human experts to evaluate both successful and unsuccessful ideas generated by the system.\n\n3. The presentation of this paper is good with well-organized figures to report and visualize results.", "weaknesses": "1. It's challenging for the audience without much background in the three AI tasks to evaluate the significance and contribution of the ideas generated by DeepScientist without the context of prior work. Although some details of the generated ideas are included in Sec 4.1, it might be helpful to also include a brief introduction of the human-designed method that the system primarily gets inspired by, and compare it with the method generated by the system.\n\n2. The proposed system uses a few hyperparameters in Eq 1 to select the hypothesis for verification, but the strategy of choosing the hyperparameters remains unclear.\n\n3. This system relies heavily on the LLM that evaluates the estimated utility, quality, and exploration value of generated hypotheses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zprMZy4h3x", "reviewer": "Reviewer_oPiU", "summary": "The paper introduces DeepScientist, a system designed to automate goal-oriented scientific discovery. Experimental results suggest that DeepScientist can identify novel methods that outperform state-of-the-art (SOTA) approaches across three frontier AI tasks.", "questions": "- Line 319: The paper mentions that five research papers generated by DeepScientist were evaluated. How were these papers selected—randomly or based on performance? Given that automatic evaluation is not resource-intensive, why not evaluate all successfully generated papers for a more comprehensive assessment?\n- Line 859: The authors note that all experimental results were manually inspected by human supervisors to ensure authenticity. How extensive was this human involvement? Quantifying the amount of human effort would help assess the true level of automation achieved.\n- System Design: DeepScientist uses Gemini for high-level planning and Claude for implementation and execution. Did the authors observe any performance differences when using stronger or weaker models for these roles? In other words, how does the planner–executor balance affect the overall efficiency and quality of the discovery process? Which component benefits more from having a stronger model?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Motivation:** The motivation is strong. Automating scientific discovery is a compelling direction, especially given the recent advancements in LLMs. It is valuable to explore how such systems can accelerate the pace of scientific progress.\n\n**Substance:** The paper presents experiments on three distinct tasks to demonstrate the capabilities of DeepScientist. The generated research papers appear coherent and reasonably well-written, suggesting that the system can meaningfully contribute to scientific exploration.", "weaknesses": "**Presentation:** The baselines in Table 2 lacks sufficient explanation and context. Each baseline should be clearly introduced and described in the Experimental Setup section. Additionally, the overall pipeline of DeepScientist appears very similar to existing systems such as AI Scientist and Zochi. A more detailed comparison would help highlight the genuine innovations and unique contributions of DeepScientist. For instance, what key design elements distinguish DeepScientist from these prior works? Moreover, the paper omits a related work, “AlphaGo Moment for Model Architecture Discovery”, which follows a conceptually similar pipeline and should be discussed.\n\n**Evaluation:** The evaluation methodology seems somewhat limited. The results in Table 2 rely on a single automated reviewer system, which raises concerns about robustness. It would strengthen the paper to cross-validate the results using multiple reviewer systems or complementary evaluation approaches. Notably, systems such as Zochi also report high acceptance rates when evaluated by automatic reviewers, yet here its acceptance rate is listed as 0%—this discrepancy deserves further analysis or justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)