ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problems
Authors: Xin Gui
King Zhu
JinCheng Ren
Qianben Chen
Zekun Moore Wang
Yizhi LI
Xinpeng Liu
REN WENLI
Linyu Miao
Tianrui Qin
Ziqi Shu
He Zhu
Dingfeng Shi
Jiaheng Liu
Yuchen Eleanor Jiang
Minghao Liu
Ge Zhang
Wangchunshu Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: benchmark
LLM
Agent
Primary area: datasets and benchmarks

Abstract: In recent years, the research focus of large language models (LLMs) and agents has shifted increasingly from demonstrating novel capabilities to complex reasoning and tackling challenging tasks. However, existing evaluations focus mainly on math/code contests or general tasks, while existing multi-domain academic benchmarks lack sufficient reasoning depth, leaving the field without a rigorous benchmark for high-level reasoning. To fill this gap, we introduce the ACADREASON benchmark, designed to evaluate the ability of LLMs and agents to acquire and reason over academic knowledge. 
It consists of 50 expert-annotated academic problems across five high-reasoning domains, including computer science, economics, law, mathematics, and philosophy. All questions are sourced from top-tier publications in recent years and undergo rigorous annotation and quality control to ensure they are both challenging and answerable. We conduct systematic evaluations over 10 mainstream LLMs and agents. The results show that most LLMs scored below 20 points, with even the cutting-edge GPT-5 achieving only 16 points. While agents achieved higher scores, none exceeded 40 points. This demonstrates the current capability gap between LLMs and agents in super-intelligent academic research tasks and highlights the challenges of ACADREASON. The code and data for the ACADREASON benchmark are available at https://github.com/OPPO-PersonalAI/Acadreason-benchmark.

Published as a conference paper at ICLR 2026
ACADREASON: EXPLORING THE LIMITS OF REA- SONING MODELS WITH ACADEMIC RESEARCH PROB- LEMS
Xin Gui
King Zhu
JinCheng Ren
Qianhen Chen
Zekun Moore Wang"
Yizhi Li
Xinpeng Liu 10
Ren Wenli
Linyu Miao
Tianrui Qin
Ziqi Shu
He Zhu
Dingfeng Shi
Jiaheng Liu"
Yuchen Eleanor Jiang
Minghao Liu
Ge Zhangg.91
Wangchunshu Zhou
BUPT "MAP 'OPPO "Hohai University *Kuaishou
Thhe University of Manchester
"Nanjing University $2077AI BytteDance
PePeiing University
wcszhou@outlookk.com com, gezhang@umich. edu, dreamforever rreaforrever.iuu@gaee www@ggaill..omm
ABSTRACT
In recent years, the research focus of large language models (LLMs) and agents has shifted increasingly from demonstrating novel capabilities to complex reason- ing and tackling challenging tasks. However, existing evaluations focus mainly on math/code contests or general tasks, while existing multi-domain academic benchmarks lack sufficient reasoning depth, leaving the field without a rigorous benchmark for high-level reasoning. To fill this gap, introduce the ACADREA- SON benchmark, designed to evaluate the ability of LLMs and agents to acquire consists of 50 expert-annotater aca-
and reason over academic knowledge. demic problems across five high-reasoning domains, including computer science, economics, law, mathematics, and philosophy. All questions are sourced from top-tier publications in recent years and undergo rigorous annotation and quality control to ensure they are both challenging and answerable. We conduct system- atic evaluations over 10 mainstream LLMs and agents. The results show that most LLMs scored below 20 points, with even the cutting-edge GPT-5 achieving only 16 points. While agents achieved higher scores, none exceeded 40 points. This demonstrates the current capability gap between LLMS and agents in super- intelligent academic research tasks and highlights the challenges of ACADREA- SON. The code and data for the ACADREASON benchmark are available at bencharrr
INTRODUCTION
The ability to reason effectively cornerstone of advanced artificial intelligence, enabling systems to tackle complex problems across diverse domains. Recent advancements in large language models (LLMs), exemplified by models such as OpenAl's o3 (OpenAI (2024b), have demonstrated significant strides in reasoning capabilities. These models leverage techniques like inference-time scaling and learning-to-reason, showcasing robust performance across reasoning tasks (Ke et al. 2025).
However, reasoning LLMs continue to evolve, limitations in existing reasoning benchmarks- -uch as MMLU-Pro (Wang et al., 2024), GPQA (Rein et al. 2023) and SuperGPQA (Team et al. 2025>havee have become apparent. These benchmarks, designed for simpler tasks like arithmetic, algebra, grade-school knowledge, or commonsense reasoning, are becoming outdated and saturated, failing to capture the trends of advanced reasoning.
For example, GALA Mialon et al. 2024) assesses LLMs' general agentic abilities through real- world questions, while PaperBench (Starace et al. 2025) challenges LLMs replicate 20 ICML machine
Equal contribution.
t Corresponding Author.

---

Published as a conference paper at ICLR 2026
learning papers, testing their abilities in coding, debugging, paper comprehension. and scientific reasoning. A more detailed comparison to other benchmarks can be found in E.4
Righ-Reasoning Research Question Extraction
High Quallityaaltyyyay Academic Papers Collection
Checklists and Hints Extraction
Domain
background hints definition hints
Ph
Read paper
tier publications
Hints 3. methodology hints
Recent publications
1. Refine the research, questions questions
3. Theoretical research
Research Question
Formulate a golden answer that includes sufficient reasoning details
verifiable independent
10 experts
Golden Answer
Checklist
x 50 high-level theoretical papers
Paper
Research Question
Condidate
Response
Answer & checklist
GPT-5 mini
Scores
Evaluation
Figure 1: Overview of the ACADREASON benchmark construction and evaluation pipeline. It consists of three stages: (1) High-Quality Academic Papers Collection - experts filter 430 papers across domains into 50 top-tier theoretical works; (2) fighhRResonnng Research Question Extraction research questions are refined into formal queries with golden answers containing sufficient reasoning: (3) Checklists and Hints Extraction background, definition, and methodology hints are provided together with verifiable, independent checklists. For evaluation, candidate responses are compared against golden answers and checklists, and GPT-5 mini assigns final scores.
Despite these advancements, these henchmarks often lack either domain breadth failing to compre- hensively cover fields like science and humanities or depth of difficulty missing the professional rigor, timeliness, and complexity required for cutting-edge reasoning tasks. To address these short- comings, we propose ACADREASON, a benchmark designed evaluate the academic-level reasoning abilities of LLM and agent.
As shown in Figure I our approach involves extracting knowledge and synthesizing high-quality reasoning data from diverse, authoritative, and timely academic literature spanning domains such as philosophy, statistics, mathematics, economics, computer science, among others. Specifically, based on publication date and top-tier journal status, we select 430 papers from leading journals. From each paper, we extract only one research question, with the corresponding golden answer designed to cover the full scope of the paper's contributions, thereby making each task demanding in workload and reasoning depth. Building on the extracted information, we further develop a scoring checklist and hints, providing more detailed evaluation rules and experiments. Ultimately, compile total of 50 high-quality research question, forming the ACADREASON benchmark. For evaluation, we adopt Juae- as our evaluation method and utilize GPT-5-mini the judge model, which conducts assessments based on detailed scoring criteria and a checklist.
Our experimental results demonstrate that ACADREASON provides challenging tasks for LLMs and agents. Even the latest and most powerful model, GPT-5, achieves only 16 points in pass rate and 40.6 points in checklist score. Furthermore, we find that reasoning models outperformed general models, with DeepSeek-Ri R1 attaining a checklist score of 23.8, higher than DeepSeek- -33's' 15.9 points. We also test cutting-edge agents, OAgents achieve the highest score of 34 points among all models and frameworks, demonstrating agents' strong capability in solving research problems, though significant room for improvement remains. Additionally, we introduce hints to investigate the impact of different types of knowledge during problem-solving. The experimental results indicate that the incorporation of hints, as supplementary information, positively contributes to model performance, with methodology hints yielding the most significant gains. This suggests that, compared to simple and easily accessible background information, the ACADREASON benchmark places greater emphasis on evaluating LLMs' mastery of methods.
Our contributions are as follows:
We introduce ACADREASON benchmark, which provides multi-domoa evaluation of LLMs' high-level reasoning abilities, and introduces challenges to existing models in terms of both knowledge and reasoning capability.
2

---

Published as a conference paper at ICLR 2026
We evaluate SOTA LLMs and Agents, Our testing experiments demonstrate that general models underperform on ACADREASON while reasoning models and agents exhibit stronger but still improvable performance, validating the dataset's challenge level and reasoning- centric design.
We provide comprehensive and detailed evaluation metrics, along with different types of knowledge hints. This offers insights for uncovering the potential of LLMs and Agents, as well as guiding future improvement directions.
2 RELATED WORK
Large Reasoning Models and Agent With the release of Deepseek-RI (Guo et al. 2025b and OpenAl's ol (OpenAl al. 2024) model, LRMs(Large Reasoning Models) have demonstrated remarkable performance in areas such as inference and academic competitions. Deepseek-Ri (Guo at. 2025b) extends the model's reasoning chain through reinforcement learning approach, achieving impressive results. Qwen3 (Yang et al. 2025) offers hybrid reasoning mode alongside default mode, providing more flexible thinking strategies. Although L.MSs possess exceptional capabilities in reasoning, they are constrained by their limited internal knowledge. The Agent Framework (Zhu ctt al. 2025a b. Team 2025a; Fang et al. 2025: Qin et al. 2025) builds upon the foundational abilities of LRMs and extends them with corresponding tools, enhancing the model's capacity to acquire external knowledge. OAgents (Zhu et al. 2025a) conduct a systematic empirical study on the GAIA benchmark and BrowseComp. achieving outstanding performance. MiroFlow (Team] 2025a) constructs its agent framework based on MCP and has achieved state-of-the-ar results on multiple leaderboards.
Reasoning Benchmark. Evaluating advanced reasoning capabilities remains a central challenge in the development of language models. Benchmarks such as arXivBench (Li et al. 2025a) and Paper- Bench (Starace et al. 2025) have been designed assess the research-related reasoning abilities of LLMs. arXivBench requires LLMs to generate accurate paper names and corresponding links, while PaperBench evaluates models' ability to reproduce ICML papers. DeepResearch Bench (Du et al. 2025) assembles multi-domain tasks evaluate LLMs' research-orientes reasoning. GAIA et al. 2024) presents real-world challenges that require models to demonstrate proficient tool usage, web search capabilities, and reasoning. BrowseComp (Wei al. 2025) places greater emphasis on web search and the ability to synthesize information from multiple web pages. However, existing benchmarks are limited in two key aspects: some lack breadth of coverage, being overly focused on math and coding at the expense of fields like science and humanities, while others lack depth of reasoning, testing only superficial information integration rather than advanced, professional knowledge. In contrast, our work bridges this gap by integrating both dimensions, presenting a novel and comprehensive challenge to evaluate the ability of LLMs and Agents to tackle cutting- academic research questions.
3 ACADREASON BENCHMARK
In this section, we introduce the ACADREASON benchmark, which focuses on measuring the cutting- edge reasoning capabilities of LLMs. Our human annotation process encompasses multiple stages, including data collection, question extraction, and quality assurance, to ensure the quality and challenge level the questions. To establish a comprehensive evaluation framework, we incorporate hints and checklists based questions and answers, thereby constructing robust evaluation methodology with corresponding metrics (specific data can be found in Appendix G
3.1 TASK SPECIFICATION
In ACADREASON, LLMs and agents serve as candidates and are tasked in the role a researcher. They are required to solve complex research questions extracted from high-level theoretical articles without access to the original text, relying either on internal knowledge or utilizing search tools to obtain additional information. Unlike simple information retrieval and integration, ACADREASON simulates real-world research scenarios, demanding that the models not only possess cutting-edge academic knowledge but also demonstrate deep reasoning capabilities.
3

---

Published as a conference paper at ICLR 2026
Formally, each task in ACADREASON benchmark contains such atomic fields:
Each question is research question constructed from the selected paper. which is self-contained,, comprising (a) a specific problem from the puper and (b) the minimal background necessary for comprehension.
Question:
Hints: Supporting information provided to the candidate model. To analyze the impact of different information types, hints are divided into three categories:
Background Hints: background knowledge and related work.
Definition Hints: key concepts and terminology introduced the paper.
Methodology Hints: theoretical tools required for reasoning and proof.
Checklist: Expert-designed checkpoints that capture key milestones the reasoning process (e.g., logical steps or critical facts). Unlike static checklists in prior work, ours are dynamic, tailored to each question, and adapt length to problem complexity.
Golden Answer: complete solution trajectory that fully satisfies all checklist requirements, covering background, definitions, derivations, and conclusions.
3.2 DATA ANNOTATION
Task construction in the ACADREASON benchmark follows strict principles to ensure quality, clarity, and alignment with high-information, high-reasoning challenges. Our data annotation pipeline consists of three components: Collection of high-quality academic papers as raw material. 2. Extraction of high-reasoning question- answer pairs. 3. Development of checklists and hints based golden answers. The annotation guideline can be found in Appendix
High-Quulityy Academic Papers Collection To ensure the challenging nature of the questions in ACADREASON, we design a meticulous data selection protocol. First, based on criteria including publication date and top-tier journal status, we collect 430 eligible papers from various leading journal websites. These papers cover a wide range of domains and exhibit diverse domain-specinn logics, though not are necessarily suitable for conversion into question-answe answer format. Annotation experts are instructed to carefully review and filter these articles according to the following principles: whether they contain challenging reasoning questions, 2. whether they consist of purely theoretical content.
Righ-RRessonin Research Questions Extraction Based on the collected high-quality papers, annotators are required to extract high-reasoning questions and golden answers from them. First, annotators read the entire paper and identify its main contributions and core research questions. Then, they refine the research questions into formal questions that must meet the requirements of being Comprehensive and Challenging. Finally, based on the question and the full content of the paper, the annotators formulate golden answer that includes sufficient reasoning details such as definitions, formulas, key concepts, and derivations whill Comprehensive.
while also satisfying the criteria of being Independent and
Checklist and Hints Extraction Based the extracted questions, golden answers, and the full paper content, annotators further derive and organize hints and checklists. For hints, there are three types: background hints compiled from the introduction section of the paper, definition hints derived from core formulas and definitions in the paper, and methodology hints summarized from the main methodology section. These hints represent critical prompt information from the paper. For the checklist, annotators distill key scoring points from the golden answer, ensuring these points are verifiable and independent.
3.3
VALIDATION PROCESS
To ensure that each question in the benchmark strictly adheres to the design principles and expecta- tions, and to address the issues encountered in the annotation process, we implement a multi-stage data validation pipeline. Only after successfully passing through all filtering stages and the final iterative validation loop will a task be included the final benchmark. The validation process guideline is shown in Figure 10
4

---

Published as a conference paper at ICLR 2026
Data Screening Principles The ACADREASON benchmark is built upon 50 high-level theoretical papers as targeted papers, which are selected by a panel of 10 experts specializing in five distinct fields: computer science, economics, law, mathematics, and philosophy. Annotation is performed by experts with master's 's degree or higher, or those pursuing a Ph.D. or master's at leading universities, Papers chosen according to three criteria: 1) publication in top-tier journals conferences within their respective domains; 2) publication between 2023 and 2025; 3) purely theoretical content, excluding empirical research, reviews, and supplementary materials. These Screening principles ensure the difficulty and quality of ACADREASON, In Table 3 we present the sources of the 20 representative papers.
Question Answerability Verification Since the ACADREASON benchmark requires models to conduct detailed research and demonstration, to prevent questions from being answered too broadly or evaluated ineffectively, we implement Question Answerability Verification. For each annotated question, is assigned to three domain experts for quality inspection, the experts evaluate the questions based three principles: clear boundaries of the question, completeness of information elements, and compliance with domain-specific logic. Only questions that meet all these criteria are retained.
3.4 EVALUATION METHON AND METRICS
Evaluation Prompt Previous work Yue et al. 2024 Zhang et al. 2024a Zhu et al. 2024
Rein et al. 2023; Ma et al., 2025) often use exact match as the evaluation metric. To provide comprehensive evaluation framework, we select GPT-5-mini as the judge model, validated through an inter-annotator agreement (IAA) study with three independent domain experts achieving Cohen's 's 0.861, and multi-judge comparison showing GPT-5-mini attains 89. 89.55% accuracy on Checklist Score against human consensus, outperforming alternatives including GPT-5, and DeepSeck- RI in cost-efficiency (details in Appendix E.3) Given question, the golden answer, and the corresponding checklist, the judge model evaluates the candidate's response in two aspects: (i) exact correspondence the golden answer (1 if all required information is present and non-contradictory; 0 otherwise); (ii) independent satisfaction of each checklist item (1 if fully satisfied; for partial, missing, or conflicting content). The prompt can be found in Appendix
We use the following two metrics as our evaluation criteria: the pass rate, measuring exact agreement with the golden answer, and the checklist score, capturing the proportion of satisfied checklist items.
Evaluation Metrics
Pass Rate (Rp): Probability of full match with standard answers.
Scoring: 8g {0, 1) per question.
Total: Rp = 36 8g 100 (max =100).
.
Checklist Score (Rz)) Probability of meeting checklist criteria.
Scoring: Cg.i E {0, 1} per checklist item.
300 Ei= Se.t x 100 (max 100).
Total: Rg
4 EXPERIMENTS
4.1 EXPERIMENTS SETTINGS
To comprehensively evaluate the performance of the ACADREASON benchmark, we conduct exper- iments from the following four perspectives: the performance of mainstream advanced reasoning models and general models on the benchmark, the performance of leading agent frameworks on the benchmark, the model performance with critical hint prompts, and detailed analysis of failure cases. For mainstream general models and reasoning models, we directly require the models to answer the corresponding questions. For agentic frameworks, we maintain their basic tool configurations.
To further analyze the models' mastery of knowledge across different dimensions, we design detailed ablation experiments evaluate three distinct types of hints. Finally, also provide analysis of
5

---

Published as a conference paper at ICLR 2026
the failure reasons for current advanced models and agents, along with potential directions for future development.
General Model & Reasoning Model For general models and reasoning models, the acareason
benchmark focuses on evaluating their knowledge reserves and reasoning capabilities. We se-
lect general models such as GPT-oss (OpenAl 2025), GPT-4.1 (OpenAl 2024a). GPT-5 (openai
2025a), DeepSeek-V3 (Liu et al. 2024). DeepSeek-Vi V3.1 (DeepSeek-Al 2024 Claude-Sonne
4 (anthropic 2025p. as well as powerful reasoning models including Qwen3 (Yang et al. 20225).
DeepSeek-R R1 (Guo et al. 2025a), Kimi-k2 (Kimi Team et al. 2025). Gemini-2.5-4 (Comanici al. 2025). and (OpenAl 2024b) as our baseline models.
Agent Framework& Agent Model Compared to LLMs, the agent can actively gather necessary information using tools like web search and database queries, giving enhanced retrieval capabilities.
We select current state-of-the-art OAgents (GPT-5 basic model) (Zhu et al. (025a), Gemini-2.55
Pro-Decpe Pro-DeepResearch (google 2025). and o3-ecepResearch (openai 2025b) as our agent framework baselines, and Tongyi DeepResearch [Team 2025b), AFM (Zhang et al. 2024b). MiroThinker (Team)
2025a), WebDancer (Wu et al. 2025) and WebTThinker (Li et al. 2025b) as our Agent baseline.
Ablation Experiment with Hints To provide more comprehensive experimental analysis and insights, we conduct an ablation study to systematically investigate the effectiveness of the multi- hint mechanism. The hints, meticulously curated by hand, encapsulate high-quality background information, methodologies, and key definitions extracted from relevant research. In this experiment, we compare baseline models without hints against ablated models integrated with hints, evaluating their performance across GPT-5, 1.T-4.11 o3, and etc.
Detailed Failure Case Analysis The ACADREASON benchmark assesses the graduate-level rea- soning abilities of LLMs, which typically require models to engage in deep thinking and generate multi-step reasoning chains. To thoroughly investigate the multi-step reasoning process and analyze failure patterns, we conduct a detailed Failure Case Analysis. We select representative models GPT-5 and OAgents, presenting their reasoning chains and logic pathways.
4.2 MAIN EXPERIMENT RESULT
As shown in Table T we present the results of over 10 LLMs and Agents. In terms of pass rate, even the most powerful models on the market exhibit subpar performance. For example, the latest and most powerful model, GPT-5, achieved only a 16 pass rate and 40.6 checklist score. Most general models scored below 10 points in total. It is worth mentioning that powerful models such as GPT-4.1 and Mlaude-Sonnete receive score of O, indicating that ACADREASON is highly challenging.
Compared to the stringent pass rate score, the checklist score effectively assesses how well models meet the tablished criteria and provides a more detailed evaluation framework. We investigate the variation in checklist scores across different academic disciplines. As shown in Figure 2, the results indicate that Computer Science and Eco- nomics exhibit relatively lower score distributions, while Law and Philosophy demonstrate higher scores. This sug- gests that CS and Econ present greater challenges in the ACADREASON benchmark.
80
Checkhist Score
60
40
20
CS
Econ
Law
Math
Phi
Figure 2: General performance on differ- ent domains in Checklist Score
When comparing general models and reasoning models, the latter generally exhibit superior and more balanced performance. We compare general models and reasoning models from the same series. For example, compared to DeepSeck-V3 V3 (2.0/15.9), DeepSeek-RI achieves higher score (2.0/23.8). Similarly, o3 also outperforms GPT-4. demonstrating that reasoning models exhibit stronger performance within their respective series. ACADREASON focuses more on assessing the reasoning capabilities of models.
Within the same model families, newer versions consistently outperform their older counter- parts. For example, GPT-5 outperforms GPT-4.1 in both pass rate and checklist score across multiple subjects. GPT-5 achieves overall score of 16 and 40.5 for the pass rate and checklist score,

---

Published as a conference paper at ICLR 2026
Table I: Performance of various Models and Agents on ACADREASON benchmark. Each entry shows Pass Rate Rp on the left and Checklist Score R, on the right. Note that the best results are in bold.
Model
Overall
CS
Econ
Law
Math
Phi
General Model
GPT-5 GPT-OSS
16/40.5
0/13.5
20/46.1
40/52.1
0/51.4
20/56.6
4/32.2
0/12.6
0/34.2
10/41.7
10/38.3
0/49.1
DeepSeek- V3.1
2/24.8
0/9.0
0/27.6
10/45.8
0/22.4
0/39.6
DeepSeek- V3
2/15.9
0/5.4
10/15.8
0/10.4
0/20.6
0/34.0
Claude-Sonnet-4
0/24.7
0/4.5
0/23.7
0/33.3
0/29.5
0/47.2
GPT-4.1
0/21.0
0/0.0
0/18.4
0/31.2
0/31.8
0/37.7
Reasoning Model
Qwen3
6/20.3
0/6.3
0/21.1
20/45.8
0/12.1
10/41.5
Kimi-K2
6/20.3
0/6.3
0/21.1
20/45.8
0/12.1
10/41.5
03
4/33,4
0/8.1
0/38.2
10/50.0
0/40.2
10/50.9
DeepSeek-RI
2/23.8
0/0.0
0/22.4
0/41.7
0/30.8
10/45.3
Gemini-2.5-PH
2/22.3
0/2.7
0/15.8
0/41.7
0/25.2
10/49.1
Agent
OAgents
34/65.1
30/55.0
30/63.2
50/68.8
50/75.7
10/64.2
28/53.4
40/45.0
20/56.6
40/66.7
10/44.9
30/71.7
Tongyi DeepResearch
20/30.9
0/5.4
10/34.2
60/62.5
0/32.7
30/47.2
o3-Deepresearch
14/47.1
20/36.0
0/38.2
30/52.1
0/54.2
20/64.2
AFM
14/40.5
10/46.5
0/15.8
40/58.3
10/32.7
10/62.3
WebThinker
8/36.4
22/50.0
0/18.4
10/54.2
0/19.4
11/51.1
MiroThinker
0/26.5
0/26.3
0/10.5
0/25.6
0/29.0
0/45.3
WebDancer
0/16.4
0/14.0
0/6.6
0/18.8
0/15.0
0/35.8
respectively, while GPT-4. only manages 21.0. Similarly, DeepSeek-V3. shows notable improvements DeepSeek- V3, with overall scores of 2.0/24.8 and 2.0/15.9. These comparisons clearly demonstrate the positive impact of model updates and iterations on performance enhancement, newer models generally have enhanced knowledge and reasoning ability.
Agent frameworks outperform LLMs. OAgents achieves the best overall results among all evaluated models, with an overall pass rate of 34.0 and checklist score of 65.1, which consistently outperform both general and reasoning models across most domains, achieving top scores in Econ, Law, Math. This is because ACADREASON contains the most challenging knowledge sections currently available as the evaluation set, which places extremely high demands on both reasoning ability and knowledge mastery. For LLMs, even though they possess strong reasoning capabilities, they lack cutting-edge academic knowledge reserve. In contrast, the agent framework can compensate for knowledge gaps through autonomous information retrieval. To verify that agent performance reflects genuine reasoning rather than source-paper retrieval, we conduct URL-masking experiments showing negligible score differences when original paper URLs are blocked (see Appendix [..I) However, the significant gap from the full score of 100 indicates that there is still room for improvement in the academic research tasks.
4.3 ABLATION STUDY
The multi-hint mechanism effectively bolsters the reasoning capabilities of large language models by supplying critical contextual. As shown in Table 2 the model's performance ACADREASON benchmark significantly improves when hints are provided, reaching the highest score when all hints given. Taking GPT-5 example, without hints, the model only achieves score (16.0/40.5). However, with all hints, it attains score of (40.0/67.8) 8),, surpassing the current state-of-the-ar ahrt agent framework, OAgents.
7

---

ALL Hints Published as a conference paper at ICLR 2026
Table 2: Ablation experiment results across different hint settings. Each entry shows Pass Rate Rp on the left and Checklist Score R, on the right. Note that best results are in bold.
Models
No Hint
background
definition
methodology
General Model
GPT-5
16/40.5
16/42.5
24/50.9
34/64.3
40/67.8
GPT-OSS
4/32.2
14/40.5
10/42.3
16/52.2
22/58.5
DeepSeek- V3.1
2/24.8
2/30.9
8/37.2
12/45.3
20/54.7
DeepSeek-V3
2/15.9
4/25.1
4/26.1
4/38.5
6/44.1
GPT-4.1
0/21.0
2/26.3
0/29.9
8/42.8
20/51.6
Saude-Sonnete-
0/24.7
2/24.6
2/30.6
14/40.8
11.3/49.3
Reasoning Model
Qwen3
6/30.4
14/35.7
10/40.5
20/49.1
22/52.7
Kimi-K2
6/20.3
2/32.9
10/36.5
16/46.8
16/51.6
o3
4/33.4
12/38.0
10/48.9
28/56.2
26/60.8
DeepSeck-RI
2/23.8
4/30.6
6/35.7
8/45.3
20/50.4
Different hint types provide varying benefits, with methodology hints yielding the most signifi- cant gains. We further compare the impact of different hint types on models. As shown the Figure 3a we calculate the absolute gain in model accuracy for each hint type. We find that for the vast majority of models, methodology hints provide the highest gain, while background hints provide the smallest relative gain. This suggests that in ACADREASON benchmark, the focus is more on testing model's 's mastery of deep methods, rather than its ability to process simple, easily accessible background information.
50
Backgroumd
Aver Performance Gain
50
Background
Performance Gain
40
Definition
40
Definition
Methodology
Methodology
All Hints
30
Hints
20
10
10
GPT-5
63
DoepSeek-RI RI
DeepSeek 1 Gemini-2.5-Pro
CS
Econ
Law
Math
Phi
a The performance gain of various models across different hint types.
The average performance gain across different hint types and disciplinary categories.
Figure 3: Ablation study results. (a) shows the performance gain per model, while (b) presents the average gain across disciplines.
The benefits of different types of hints vary across different disciplines. As shown in Figure 3b we present the impact of different types of hints across various academic disciplines. We calculate the average improvement for models, with additional results available in Appendix F.2 The experimental results indicate that compared to humanities subjects (Econ, Law, Phi). STEM sub- jects (CS, Math) achieve less improvement. This suggests that humanities disciplines place greater emphasis on the acquisition of external knowledge, while STEM fields require deeper reasoning. Furthermore, each discipline exhibits distinct focuses. For Law and Phi, hints related to methodology and background information sized. This reflects the unique characteristies different academic domains and demonstrates the comprehensiveness of the ACADREA CASOR evaluation.
important, whereas for Econ, definitions more empha-
8

---

Published as a conference paper at ICLR 2026
Title: The Counterfeit Sham
Category LLaw
Research Question
Why do policymakers, courts and academia in the field of design "counterfeit", which has specific legal definition, to describe infringement of design patents? How does this rhetorical strategy distort legal logic, evade due process protection, and ultimately lead to an excessive reinforcement of legal remedies design patent holders. thereby harming public interests?
trequently the
Golden Answer
Revealing why the term "counterfeit" used: Firstly, emotional agitation, "counterfeit" specific ally refers to the intentional act of counterleiting registered...
Checklist
Point the legal fallacy: design patent infringement does require trademark use, confusion, or safety risks, and is not equivalent to counterfelting,
Point1:
Point2: Refute the false safety proposition: design patents are not quality cer tifications, and product safety is irrelevant to infringement.
Point3: Analyze root cause: the "fake" narrative lobbying tactic to enforcement costs and weaken procedural protections.
Point4: Identify judicial impact: this fallacy led mass unilateral relief in "Schedule A' cases, depriving defendants of due process rights.
0,2
OAgents
GPT-5
Scores:(1/1 4/4)
Scores:(0/1 2/4)
Response:
Response:
[Match Point 1]: EU law explicitly distinguishes counterfeit goods"
[Match Point 1]: Trademark
counterfeiting protects nource-ieentilying symbols; liability turns on use of mark identical substantially indistingulshable from the registered mark and likely to confuse...
(trademarks/Gia) from "pirated goods."
lahel design infr infrememnnts "counterfeits" in the legal sense.
(Match Point Rhettorli power and fear appeal. "Counterfeiting" connotes criminality...
[Match Point 4): parte relief without claim construction: Courts grant TROs, asset freezes, and domain seizures "Schedule A' style cases premised on "counterfeiting" When design patents are included, courts...
[Match Point 3]:the record shows that "counterfeit" rhetoric in design-patent disputes is strategic framing that blurs doctrinal lines
[Match Point 4]: parte seizures secrecy borrowed from counterfeiting. Section 1116(d) authorizes
seizures
for "counterfeit marks" but imposes stringent, itemized findings, bands...
Figure 4: Side-by-side comparison of OAgents and GPT-5 the legal reasoning task.
CASE STUDY
To provide a comparison of the leading technical paradigms, conduct case study featuring the top-scoring agent, OAgents (Zhu et al. 2025a). and the top-scoring single model, GPT-5 (openai 2025a). We select a representative case from the ACADREASON benchmark where models are required to analyze the misuse of the term "counterfeit" in design patent law, shown in Figure We evaluate the models' responses against checklist of four required actions: to point out the legal fallacy, refute the false safety proposition, analyze the root cause, and identify the judicial impact. The comparison reveals a difference: OAgents successfully address all four points for a perfect score, while GPT-5 only addresses two. OAgents provide complete analysis, covering all required dimensions, whereas GPT-5 only succeeds in identifying the direct legal fallacy and the judicial impacts (Points and 4).
The evaluation data indicates this performance gap is due to difference in reasoning depth, not a simple lack of knowledge. GPT-5's failure the "false safety proposition" (Point stemmed from

---

Published as a conference paper at ICLR 2026
inability to move beyond surface-level association of "counterfeit" with "consumer harm". It did not perform the deeper reasoning required to explicitly refute the narrative by stating that design patents not quality certifications. Similarly, for the "root cause" (Point 3), GPT-5 identified general "procedural leverage" but failed to synthesize this with political and economic context to identify the specific "coordinated lobbying strategy" required by the checklist. This case demonstrates that while a top-tier single model can handle direct legal analysis, the agentic framework of OAgents enables higher-order, critical synthesis necessary deconstruct the underlying rhetorical and political motives of a complex issue. We also provide failure attribution analysis for HlaudeSSoneet, in ??.
CONCLUSION
In this work, we introduce the ACADREASON benchmark, which comprehensively evaluates the ability of LLMs and agents to acquire and reason over advanced knowledge. The ACADREASON benchmark includes 50 evaluation items across five domains, providing a comprehensive assessment of models' research capabilities. Our experimental results show that even the most advanced model, GPT-5, achieves only 16.0 points, while an advanced agent framework scores 34.0 points. These results demonstrate the difficulty and challenging nature of ACADREASON, indicating that current models still have considerable room for improvement. By releasing the entire annotated data and preliminary benchmarking results, aim to empower the research community to better evaluate and enhance LLMs reasoning capabilities. Our approach represents a significant step towards diversifying LLM reasoning benchmarks and utilizing the vast potential of academic research artifacts in advancing LLM research.
7 ETHICS STATEMENT
In the development of the ACADREASON benchmark, we have rigorously considered several ethical aspects to ensure the responsible construction and deployment of this resource.
Data Sourcing and Intellectual Property. All academic papers used in this benchmark are sourced from publicly available, top-tier journals and conferences. We strictly adhere copyright laws and fair use principles for academic research. Our usage is limited to extracting research questions and creating derived reasoning tasks, without reproducing substantial copyrighted content. Each benchmark item is transformed into novel reasoning task through significant intellectual effort, and is intended solcly for academic rescarch purposes, specifically for evaluating and advancing reasoning capabilities in AI systems.
Expert Involvement and Compensation. The curation process involved domain experts in select- ing papers and formulating research questions. All experts were fairly compensated for their time and expertise according to academic standards, and their contributions properly acknowledged.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our research and facilitate future work, we have open-sourced all code and data. The project resources are available benchmark.
Furthermore, in the Experiment section and corresponding appendices of the paper, provide detailed descriptions of the experimental settings and the full prompts used for reasoning and evaluation, which will fully support the replication of this study.
REFERENCES
anthropic. Claude 4 sonnet, 2025. http://tss////wwaattttrrrcc/ooee
Gheorghe Comanici. Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier
10

---

Published as a conference paper at ICLR 2026
with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
DeepSeek-AI. Deepseek-v3 technical report, 2024, URL httpss//aarriv..or///////2211111
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint, 2025.
Tianqing Fang, Zhisong Zhang, Xiaoyang Wang, Rui Wang, Can Qin, Yuxuan Wan, Jun- Yu Ma, Ce Zhang, Jiaqi Chen, Xiyun Li, et al. Cognitive kernel-pro: A framework for deep research agents and agent foundation models training. arXiv preprint ar.iv:25080004111 2025.
google. Gemini deep-research,
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprins arXiv:2501.12944, 2025a.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948 2025b.
Zixuan Ke, Fangkai Jiao, Yifei Ming. Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, Caiming Xiong, and Shafiq Joty. A survey of frontiers in Ilm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arviv: 2504 3504.09037, 2025.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang, Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li. Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu. Zeyu Shang. Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung. Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang, Dinglu Wang, Feng Wang. Haiming Wang, Jianzhou Wang, Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang, Yao Wang, Yejie Wang, Yigin Wang, Yuxin Wang. Yuzhi Wang, Zhaoji Wang, Zhengtao Wang, Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang. Ying Yang. Zhen Yang, Zhilin Yang. Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang. Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang. Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence, 2025. URL 20534
Ning Li, Jingran Zhang, and Justin Cui. Arxivbench: Can llms assist researchers in conducting research? arXiv preprint arXiv: 1049., 2025a. URL https://arxiv.org/as/// 10496
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. CoRR, abs/2504.21776. 2025b. doi: 10.8850/ARXX2225422176 URL whtthttt:/ //do:. org/10
48550/arXiv 2504 21776

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseck-v2: A strong, economical, and efficient mixture-of- experts language model. CoRR, 2024.
David Ma, Yuanxing Zhang, Jincheng Ren, Jarvis Guo, Yifan Yao, Zhenlin Wei, Zhenzhu Yang, Zhongyuan Peng, Boyu Feng, Jun Ma, et al. Iv-bench: A benchmark for image-grounder video perception and reasoning in multimodal Ilms. arXiv preprint arXiv:2504. 15415, 2025.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general assistants. International Conference on Learning Representations, 2024.
OpenAL. Introducing gpt-4.1, 2024a.
OpenAL. Introducing 03 and intrroducing-o--aad------- 2024b. Blog post. Accessed: October 6, 2024.
04 mini.
https://opent htt.aaii.co///iee///eeee/a
openai. spt-5-system-card,
openai. openai deep-research,
2025b.
URL
https://opemaa..om//nne//
untroduciing-dee--reeeae
OpenAL. gpt-oss-1200 & gpt-oss-20b model card, 2025. URL https://arxxivorrraa////3
10925
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler. Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao. Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang. Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay. Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted
12

---

Published as a conference paper at ICLR 2026
Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang. Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai ol system card. arXiv preprint arXiv: 2412.16720, 2024.
Tianrui Qin, Qianben Chen, Sinuo Wang. He Xing, King Zhu, He Zhu, Dingfeng Shi, Xinxin Liu, Ge Zhang, Jiaheng Liu, al. Flash-searcher: Fast and effective web agents via dag-based parallel execution. arXiv preprint (xXiv:5509225000 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google proof q&a benchmark. arXiv preprint arXiv: 2311.12022, 2023.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai's 's ability to replicate ai research. arXiv preprint arXiv: 2504.01848, 2025. URL https:://axix....t.. org/ abs /aess//50...00 01848
M-A-P Team, Xinrun Du, Yifan Yao. Kaijing Ma. Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhen-Nan Wei, Chujie Zheng, Kaixin Deng, Shuyue Guo, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, Dehua Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Si-Xuan Wu, Tianshun Xing, Ming Xu, Zhenzhu Yang. Ze Wang, Junting Zhou, Yu Bai, Xingyuan Bu, Chenglin Cai, Liang Chen, Yifan Chen, Chengtuo Cheng, Tianhao Cheng, Keyi Ding, Siming Huang, Yun-Jing Huang, Yaoru Li, Yizhe Li, Zhaoqun Li, Tianhao Liang, Chengdong Lin, Hongquan Lin, Yi-Hui Ma, Zhongyuan Peng, Zifan Peng, Qige Qi, Shi Qiu, Xingwei Qu Yizhou Tan, Zili Wang. Chenqing Wang, Hao Wang. Yiya Wang. Yubo Wang, Jiajun Xu, Kexin Yang, Ru- Ru-Qing Yuan, Yuan hao Yue, Tianyang Zhan, Chun Zhang, Jing- Yun Zhang, Xiyue Zhang. Xing Zhang, Yue Zhang, Yongchi Zhao, Xiangyu Zheng. Chenghua Zhong, Yang Gao, Zhoujun Li, Dayiheng Liu, Qian Liu, Tianyu Liu, Shiwen Ni, Junran Peng, Yujia Qin, Wenbo Su, Guoyin Wang. Shi Wang, Jian Yang, Min Yang, Meng Cao, Xiang Yue, Zhaoxiang Zhang, Wangchunshu Zhou, Jiaheng Liu, Qunshu Lin, Wenhao Huang, and Ge Zhang. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. 2025.
MiroMind AI Team. Miroflow: A high-performance open-source research agent framework. https low, 2025a
Tongyi DeepResearch Team. dengeicdcepreseurch.
DeepResearch] 2025b.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max W.F. Ku, Kai Wang, Alex Zhuang, Rongqi "Richard" Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Neural Information Processing Systems, 2024. doi: 10.48550VarXxi.2206.0555
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12511 2025.
Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Gang Fu, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. WehDancer: Towards autonomous information seeking agency, arXiv preprint arXiv:2505.222 226488 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338. 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
13

---

Published as a conference paper at ICLR 2026
Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng. Chumpu Xu, Shuyue Guo, et al. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. arXiv preprint arXhh:2401.11944 2024a.
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ó. Arik. Chain of agents: Large language models collaborating on long-context tasks. arXiv preprint arXiv: 2406.028 02818, 2024b.
He Zhu, Tianrui Qin, King Zhu, Heyuan Huang. Yeyi Guan, Jinxiang Xia, Yi Yao, Hanhao Li, Ningning Wang. Pai Liu, et al. Oagents: An empirical study of building effective agents. arXiv preprint arXiv:2506. 15741, 2025a.
King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li, Shawn Gavin, Tuney Zheng, Jiawei Guo, Bo Li, et al. Lime: Less is more for milm evaluation. arXiv preprint ar arXiv:2409.06851, 2024.
King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, al. Scaling test-time compute for Ilm agents. arXiv preprint arXiv:2506. /2928, 2025b.
14

---

Published as a conference paper at ICLR 2026
Phl (10)
Law (10)
A
DATA STATISTICS
The rigorous curation pipeline culminates in the final ACADREASON benchmark, which comprises 50 high- reasoning academic questions. In Figure 5 we present the category distribution of the dataset. Each category in the ACADREASON benchmark includes samples.
Econ (10)
CS (10)
Table 3 presents 20 representative papers included in the
(10)
ACADREASON benchmark. All papers were selected from publicly available top-tier journals or conferences, cura- tion strategy that ensures the academic rigor and quality of the benchmark dataset originate from its source.
Figure 5: Category Distribution
Table 3: Representative List of 20 Papers in ACADREASON benchmark.
Paper
Category
Source
Reliability and Latency Analysis for Wireless Communication Systems with Secret-Key Budget
Math
IEEE Transactions on Communications, 2024, 72(2): 1033-1044
On the Popov-Belevicch Belevitch Sitevitch-HHuuuss tests for functional observability and output controllability
Math
Automatica, 2025, 174(1): 112122
Algebraic Geometry codes in the sum-rank metric
Math
IEEE Transactions on Information Theory, 2024, 70(5): 3345-3355
Variance Decay Property for Filter Stability
Math
IEEE Transactions on Automatic Control,
2024, online first
Once more, without feeling
Philosophy
Philosophy & Phenomenological
Research, 2025, 111(1): 343-365
Patchwork ethnography
Philosophy
American Ethnologist, 2024, 51(1):
131-139
Pig-feast democracy... in West Papua
Philosophy
American Ethnologist. 2024, 51(2):
193-206
Moral Understanding Between You and Me
Philosophy
Philosophy & Public Affairs, 2024, 52(3):
327-357
Women who pay their own brideprice...
Philosophy
Journal of the Royal Anthropological
Institute, 2025, 31(2): 493-512
A Lower Bound for Light Spanners in General Graphs
Computer Science
Proceedings SODA 2025: 4327 4337
Tight Streaming Lower Bounds for
Computer Science
Proceedings SODA 2025 (Best Student Paper)
Deterministic Approximate Counting
A Refutation of the Pach-Tardos
Computer Science
Proceedings of SODA 2025
Conjecture for Matrices
Universal Perfect Samplers for Incremental Streams
Computer Science
Proceedings of SODA 2025: 3409 3429
Quasi-Monte Carlo Beyond Hardy- Krause Waste, Property, and Useless Things
Computer Science
Proceedings of SODA 2025: 2051-2075 Harvard Law Review, 2025, Vol. 138 (accepted)
Law
The Law and Lawlessness U.S. Immigration Detention Human Rights Obligations in Maritime Search and Rescue
Law
Harvard Law Review, 2025, 138(5);
1186-
Law
International & Comparative Law
Quarterly, 2025, 74(1): 33-60
State Immunity from Non-Judicial
Law
International & Comparative Law Quarterly, 2025, 74(1): 179-204 Journal of Finance, 2023, 78(6):
Measures of Constraint
Informational Black Holes in Financial
Economy
Markets
3099-3140
A Theory of Dynamic Inflation Targets
Economy
American Economic Review, 2025, 115(2): 448 490
15

---

Published as a conference paper at ICLR 2026
B PROMPT FOR INFER AND EVALUATION
The prompts we used for ACADREASON benchmark are shown below.
PROMPT FOR INFER
Please answer the following question:
Question: (query)
Provide precise and detailed response.
Figure 6: Prompt infer
PROMPT
Task:
Judge the following attempt answer to an academic question based on the provided question, checklist criteria, and golden answer reference.
Judgement Criteria
Aspect I: Answer Correspondence
Judge if the answer corresponds to the golden answer:
point: The answer contains all the information of the golden answer
0 point: The answer completely fails meet or only partially the key information required by the golden answer, or there are contradictions
Aspect 2: Checklist Requirements
For every item on the checklist, judge independently whether the answer meets the requirement. The hints provided help you judge:
point: The reasoning and answer meet the requirement
0 point: The reasoning and answer do not meet the requirement, or only partially meet the requirement
Data Information
Inputs
Question: (query)
Checklist: [checklist
Answer to judge
Attempted Answer: [response]
Golden Output
Golden Answer: (golden_answer)
Output Format
Please respond strictly in the JSON format provided below. Note that the number of items in the checklist should be equal the number items the justifications and scores aspect 2. The number of checklist items can vary.
Example Output
[1
"aspect_1 1. aanalysis": "Give the reason for how score the aspect 1";
"aspect_1 _scoree: 0,
"aspect_ 2 L_aaalllss__1 1=: "Give the reason for how score the first item in the checklist in aspect 2",
"aspect score 1": 0.
"aspect analysis 2": "Give the reason for how score the second the checklist aspect 2".
"aspect score 2": 0.
Figure 7: Prompt for eval
16

---

Published as a conference paper at ICLR 2026
C LIMITATIONS
The current benchmark comprises 50 expert-curated problems, a deliberate design choice given the substantial computational cost deep research wvaluation-where single query often necessitates tens to hundreds of reasoning steps consistent with similar benchmarks in this domain (e.g., XBench- deepsearch: 100, PaperBench: 20). As peer-reviewed publications continuously emerge across all academic domains, ACADREASON can be incrementally expanded while maintaining the annotation rigor essential for research-level evaluation. We plan to explore semi-automatic scaling approaches leveraging LLM-assisted pipelines to facilitate sustainable growth of the benchmark.
D LLM USAGE
Large language models (LLMs) used in this work exclusively for text polishing and language refinement during the writing process. Specifically, LLMs assist in improving the fluency, clarity, and conciseness of the writing.
I.MMs are not used for any aspects experimental design, methodological development or scientific interpretation. All scientific contributions and innovations presented in this work are entirely human- originated.
E MORE EXPERIMENT RESULT
E.1 INFORMATION LEAKAGE DETECTION EXPERIMENT
To address concerns about potential data contamination through web search capabilities, we conducted controlled experiment by masking URLs of the original source papers. Specifically, blacklisted all URLs containing the original paper content, preventing agents from directly accessing them during evaluation.
We evaluated two representative agent systems-OAgent and Tonggi-DeepReeeacch--umeee both masked and unmasked conditions. Table presents the results.
Table Performance Comparison with URL Masking
Agent
Pass Rate (%)
Checklist Score (%)
OAgents (w/o mask) OAgents (w/ mask)
34
65.1
32
65.8
Denggi-DeepResearchh (w/o mask)
16
49.2
Tongyi-DecpResearch (w/ mask)
16
47.0
The results show minimal performance differences between masked and unmasked settings for both agents (differences within 2 percentage points). This robustly demonstrates that direct access to source papers does significantly inflate agent performance, confirming that ACADREASON primarily tests reasoning capabilities rather than information retrieval. This finding can be attributed to our careful benchmark design: each research question was deliberately crafted to be highly autonomous and independent from the source paper content, requiring substantial reasoning even when the original paper accessible.
E.2 DETAILED TABLE ABOUT HINTS ABLATION EXPERIMENTS
17

---

Published as a conference paper at ICLR 2026
Table 5: Performance of various Models ACADREASON benchmark, providing with background hint. Each entry shows Pass Rate Rp on the left and Checklist Score R, on the right. Note that best results in bold.
Model
Overall
CS
Econ
Law
Math
Phi
General Model
GPT-5
16.0/42.5
0.0/8.1
0.0/47.4
50.0/60.4
0.0/49.5
30.0/77.4
GPT-OSS
14.0/40.5
0.0/19.8
0.0/32.9
40.0/56.2
10.0/42.1
20.0/77.4
DeepSeek-V3
4.0/25.1
0.0/2.7
0.0/17.1
10.0/47.9
0.0/28.0
10.0/56.6
GPT-4.1
2.0/26.3
0.0/2.7
0,0/21.1
0.0/41.7
0,0/30.8
10.0/60.4
4
2.0/24.6
0.0/0.0
0.0/17.1
10.0/45.8
0.0/29.0
0.0/58.5
DeepSeek-Var V3.1
2.0/30.9
0.0/13.5
0.0/19.7
0.0/39.6
0.0/33.6
10.0/69.8
Reasoning Model
Qwen3
14.0/35.7
0.0/6.3
10.0/34.2
40.0/68.8
0.0/31.8
20.0/77.4
03
12.0/38.0
0.0/7.2
10.0/31.6
20.0/56.2
0.0/46.7
30.0/77.4
DeepSeek-R
4.0/30.6
0.0/5.4
0.0/38.2
10.0/41.7
0.0/29.9
10.0/64.2
Gemini-2.5-Pre
4.0/26.6
0.0/6.3
0.0/23.7
10.0/50.0
0.0/20.6
10.0/64.2
Kimi-k2
2.0/32.9
0.0/8.1
0.0/26.3
0.0/50.0
0.0/35.5
10.0/73.6
Table 6: Performance of various Models ACADREASON benchmark, providing with Definition Hint. Each entry shows Pass Rate Rp on the left and Checklist Score Ry on the right. Note that best results are in bold.
Model
Overall
CS
Econ
Law
Math
Phi
General Model
GPT-5
24.0/50.9
0.0/18.9
40.0/72.4
60.0/64.6
10.0/55.1
10.0/66.0
GPT-OSS
10.0/42.3
0.0/17.1
10.0/59.2
20.0/62.5
0.0/37.4
20.0/62.3
DeepSeek-Ve V3.1
8.0/37.2
0.0/10.8
10.0/51.3
10.0/52.1
20.0/41.1
0.0/50.9
DeepSeek-V3
4.0/26.1
0.0/3.6
10.0/34.2
10.0/43.8
0.0/24.3
0.0/49.1
Claude-Sonnet-4
2.0/30.6
0.0/2.7
0.0/38.2
10.0/47.9
0.0/35.5
0.0/52.8
GPT-4.1
0.0/29.9
0.0/8.1
0.0/42.1
0.0/41.7
0.0/29.9
0.0/47.2
Reasoning Model
o3 Qwen3
10.0/48.9
10.0/35.1
0.0/61.8
20.0/56.2
0.0/42.1
20.0/66.0
10.0/40.5
0.0/16.2
20.0/57.9
10.0/60.4
0.0/33.6
20.0/62.3
Kimi-K2
10.0/36.5
0.0/20.7
10.0/44.7
20.0/54.2
0.0/27.1
20.0/60.4
DeepSeck-RI
6.0/35.7
0.0/4.5
20.0/63.2
0.0/43.8
0.0/33.6
10.0/58.5
Gemini-2.5-Pro
4.0/38.5
0.0/8.1
10.0/63.2
0.0/41.7
0.0/43.0
10.0/54.7
18

---

Published as a conference paper at ICLR 2026
Table 7: Performance of various Models ACADREASON benchmark, providing with Methodology Hints. Each entry shows Pass Rate Rp on the left and Checklist Score RG on the right. Note that the best results in bold.
Model
Overall
CS
Econ
Law
Math
Phi
General Model
GPT-5
34.0/64.3
0.0/37.8
20.0/69.7
70.0/75.0
40.0/70.1
40.0/90.6
GPT-OSS
16.0/52.2
0.0/27.0
0.0/60.5
30.0/54.2
20.0/57.0
30.0/81.1
Claude-Sonnet-4
14.0/40.8
0.0/13.5
0.0/34.2
50.0/62.5
0.0/43.0
20.0/83.0
DeepSeek-V3..
12.0/45.3
0.0/19.8
20.0/47.4
20.0/62.5
10.0/46.7
10.0/77.4
GPT-4.1
8.0/42.8
0.0/18.9
0.0/43.4
30.0/56.2
10.0/46.7
0.0/71.7
DeepSeck-V? V3
4.0/38.5
0.0/14.4
0.0/32.9
10.0/58.3
0.0/43.0
10.0/69.8
Reasoning Model
o3
28.0/56.2
0.0/31.5
40.0/73.7
50.0/58.3
10.0/57.0
40.0/79.2
Qwen3
20.0/49.1
0.0/18.9
10.0/42.1
60.0/70.8
10.0/56.1
20.0/88.7
Kimi-k2
16.0/46.8
0.0/22.5
0.0/53.9
40.0/64.6
10.0/39.3
30.0/86.8
Gemini-2.5-PP
10.0/48.6
0.0/25.2
0.0/50.0
20.0/56.2
10.0/48.6
20.0/88.7
DeepSeek-PP R1
8.0/45.3
0.0/16.2
0.0/48.7
20.0/50.0
0.0/48.6
20.0/90.6
19

---

Published as a conference paper at ICLR 2026
E.3 VALIDATING THE LLM JUDGE: A SIMAAN-LIGGMENT STUDY
To ensure the validity and reliability of our automated evaluation approach, we conducted a systematic comparison between multiple LLM judges and human expert evaluations. This section details our methodology and findings that led to the selection of GPT-5 Mini as our primary judge model.
E.3.1 INTER-ANNOTATE ANNOTATOR AGREEMENT EXPERIMENT
We sampled 10 instances from the full set of 50 data points to inference results, generated by OAgents (backbone GPT-5), were selected across a range from high low scores mitigate selection bias. We then invited experts for each domain to independently judge these samples based on two metrics: Pass Rate and Checklist Score.
validation set. The instances'
Table 8 presents the detailed comparison between GPT-5 Mini judge and expert human judges across different domains and samples.
Table 8: Comparison of GPT-5 Mini Judge and Human Expert Evaluations
Pass Rate
Checklist Score
Domain
CPT-5 Mini
Humanl
Human2
Human3
GPT-5 Mini
Human1
Human2
Human3
3
Philosophy
1/1
1/1
1/1
1/1
4/4
4/4
4/4
3/4
Philosophy
1/6
2/6
2/6
11
Computer Science
0/1
0/1
0/1
0/1
4/6
0/6
1/6
1/6
12
Computer Science
0/1
0/1
0/1
2/8
2/8
2/8
2/8
26
Law
1/1
6/6
6/6
6/6
6/6
30
Law
0/1
0/1
0/1
0/1
0/5
0/5
0/5
0/5
34
Economics
0/1
4/7
3/7
5/7
40
Economics
Q11
8/8
8/8
6/8
44
Muth
1/1
0/1
0/1
0/1
9/9
5/9
5/9
5/9
4/8
For each domain, we engaged three independent human annotators to label the samples. The inter- annotator agreement was measured using Cohen's for each annotator pair, yielding an average k of 0.861 (range: 0.843-0.870) 0.870.. Ground truth labels were established through majority voting among the three annotators, which we then used to calculate the consistency score with series of candidate LLM judge models.
E.3.2 MODEL COMPARISON AND CONCLUSION
We evaluated multiple candidate models against the established ground truth. Tables P and the consistency metries for Pass Rate and Checklist Score, respectively.
10 present
Table Consistency Metrics for Pass Rate Evaluation
Model
Acc (%)
Prec (%)
Recall (%)
F1 (%)
Cost (S/IM-tokens)
Random
50
GPT-5
90
66.67
100
80
1.25/10
GPT-5 mini
90
66.67
100
80
0.25/2
80
50
100
66.67
3/15
DeepSeek-V. V3
70
40
100
57.14
0.28/0.42
DeepSeek-R1
80
50
50
50
0.55/2.19
Table 10: Consistency Metrics for Checklist Score Evaluation
Model
Acc (%)
Prec (%)
Recall %
F1 (%)
Cost (S/IM-tokens)
Random
49.75
48.85
45.45
47.09
GPT-5
86.57
92.86
78.79
85.25
1.25/10
GPT-5 mini
89.55
86.11
93.94
89.86
0,25/2
-laude-Sonnet-4.5
85.07
89.66
78.79
83.87
3/15
DeepSeek-V3
82,09
74.42
96.97
84.21
0.28/0.42
DeepSeck-RR RI
85.07
89.66
78.79
83.87
0.55/2.19
20

---

Published as a conference paper at ICLR 2026
As demonstrated by the results, GPT-5 Mini achieves high overall consistency across both metrics with human expert evaluations, robustly validating its strong alignment with human judgment. Notably, GPT-5 Mini reduces the evaluation cost by 80% compared to GPT-5, while also offering significantly faster inference speed.
Specifically, for Pass Rate evaluation, GPT-5 Mini achieves 90% accuracy with perfect recall (100%), matching the performance GPT-5 while being substantially more cost-cffective. For Checklist Score evaluation, GPT-5 Mini demonstrates superior performance with 89.55% accuracy and 89.86% Fl score. Considering the essential trade-off between performance, cost, and efficiency for large-scale evaluation, we ultimately selected GPT-5 Mini as our primary judge model. This combination of high performance, low cost, and fast inference makes GPT-5 Mini the optimal choice for our large-scale evaluation framework.
E.4 COMPARISON WITH OTHER BENCHMARKS
To intuitively illustrate the differences between ACADREASON and existing benchmarks, we pro- vide systematic comparison with related mainstream benchmarks, including PaperBench, HLE, BrowseComp.
GAIA, and DeepResearchBench.
E.4.1 QUALITATIVE ANALYSIS
These benchmarks generally cover three types of tasks: (1) Code reproduction (e.g.. PaperBench): given a paper as input, the goal is to reproduce the corresponding repository; (2) Search/QA (c.g., HLE, GAIA, BrowseComp. XBeech-DeeppSarch) the core capability tested is information retrieval and short-form question answering: (3) Open-ended research/report (e.g., DeepResearchBench)) given semi-open-ended questions such "How enhance classroom participation for students with autism?", the model performs broad research and provides a report. Table 11 summarizes the key characteristics of these benchmarks in terms of domain coverage, task type, and output format.
Table 11: Comparison of ACADREASON with Related Benchmarks
Benchmark
Domain Numbers
Task Type
Output Format
PaperBench
Code reproduction
Repo/Code
HLE
8
Expert-level reasoning QA
Short QA
BrowseComp
Search-based QA
Short QA
XBench-DeepSearch
Scarch-based QA
Short QA
GAIA
5
Assistant-style QA (weh, code, multimodal)
Short QA
DeepResearchBench
22
Research-style information gathering Research-level multi-step reasoning
Long-form report
ACADREASON (Ours)
5
Long-form report
Our benchmark is specifically designed for the "research-level long report" scenario on academic research problems. Given a specific research question, the model must summarize the status quo, perform multi-step reasoning, and provide a solution-oriented research report-mimicking the workflow of human researcher. This task setting currently absent in existing benchmarks.
Compared with "Bench-DeepSeee retrieval capability itself. Instead, we focus on whether the model can complete research-level comprehensive analysis and compose a complete long-form report under the premise of having obtained relevant evidence.
and BrowseComp, we do not primarily evaluate long-chai
Unlike HLE and GAIA, which also target academic domains but adopt QA format focusing retrieval correctness and short answers, our task requires the model to conduct systematic research and output structured, long-form report.
PaperBench centers code reproduction with repository outputs the goal. While DeepResearch- Bench also requires report generation, it uses semi-open-ended questions from public domains (e.g., "How enhance classroom participation for students with autism?"). In contrast, ACADREASON focuses on specific academic research problems, emphasizing problem decomposition, literature review, and solution reasoning manner consistent with human researchers' methodologies.
E.4.2 QUANTITATIVE DIFFICULTY ANALYSIS
To quantitatively demonstrate ACADREASONT challenging nature, we compared the performance of state-of- the-an art models and agents across multiple benchmarks. Tables [2ales[[2amn and [3 pressentt performance comparisons across different evaluation frameworks.
21

---

Published as a conference paper at ICLR 2026
Table 12: Cross-Benchmark Performance Comparison for Agent Systems
Model/Agent
HLE
GAIA
BrowseComp
ACADREASON
Vonggy-DeepResearce
32,9
70.9
43.4
16.0
AFM
18.0
55.3
11.1
14.0
WebThinker
15.8
48.5
8.0
Table 13: Model Performance Across Academic Reasoning Benchmarks
Model
HLE (Academic)
GPQA Diamond (Scientific)
ACADREASON
GPT-5
25.32
84.2
16.0
Gemini-2.55PPr
18.08
84.0
2.0
Kimi-K2
75.1
4.7
6.0
Qwen3
71.1
6.0
DeepSeek-Ri
71.5
2.0
The results reveal consistent pattern: models and agents achieve substantially lower scores on ACADREASON compared to other benchmarks, For instance, Tongyi-DeepResearde scores 70.9% on GAIA but only 16.0% on ACADREASON-4 54.9 percentage point drop. Similarly, GPT-5 and 2emini-2.5 Pro achieve over 84% GPQA Diamond yet score only 16.0% and 2.0% respectively ACADREASON. These dramatic performance gaps suggest that ACADREASON measures distinct capabilities- hecellliees--sspificcllll the ability conduct deep, multi-step research-leve reasoning- that not adequately captured by existing benchmarks focused on factual recall or search-based question answering.
To provide a comprehensive quantification of ACADREASON's difficulty, we computed aggregate statisties across all 19 evaluated models and agents. The remarkably low average Pass Rate (8.53%, std: 9.97%) and moderate Checklist Score (31.34%, std: 13.20%) demonstrate that ACADREASON poses substantial challenges even frontier systems. The moderate standard deviation indicates that while the benchmark is difficult, it maintains sufficient discriminative power across different capability levels. The gap between Pass Rate and Checklist Score suggests that models can partially complete reasoning steps but struggle to produce fully correct, comprehensive solutions pattern consistent with the research-level nature of our tasks.
F ANNOTATION AND VALIDATION GUIDELINE
22

---

Published as a conference paper at ICLR 2026
RESEARCH QUESTIONS ANNOTATION GUIDELINE
Your task is to extract one high-quality research question from provided academic paper and then construct a comprehensive golden answer for The final question-answer pair should be self-contained, accurately reflecting the paper's core theoretical contribution, and must be solvable without requiring access to the original text. The primary goal is to create a challenging benchmark item that tests advanced reasoning.
Research Question
Questions should have well-defined boundaries and include minimal necessary background, focusing specific theoretical problems.
Clarity and Self-consisteney:
Alignment and Independence: Questions must align with the paper's core contribution and be answerable without requiring access the full text.
Structural Constraints: Avoid open-ended formulations, composite questions requiring decomposition, or references to the original paper's 's structure.
Golden Answer
Answers should cover background, definitions, derivation- s/proofs, and conclusions, satisfying all checklist requirements.
Comprehensive Coverage:
Verifiability: Provide key intermediate steps and essential formulas to ensure repro- ducibility and self-contained reasoning.
Content Integrity: Maintain logical continuity without skipping critical steps, introducing external information, or violating domain- speccfccc conventions.
Figure Guideline For Research Question Annotation
HINTS AND CHECKLIST ANNOTATION GUIDELINE
Your task is to create Hints and Checklist based the provided Research Question, Golden Answer, and the original paper. The Hints should provide necessary but incomplete support for reasoning, while the Checklist must enable clear and objective verification of complete answer. Hints Annotaion
Background: Context from the introduction and related work necessary to understand the problem.
Definitions: Standardized statements of core concepts and terminology.
Methods: Essential theoretical tools, methodological frameworks, and key technical tips.
Selection Principle: Include only information necessary to facilitate reasoning, avoiding final conclusions direct answers.
Checklists Annotaion
Atomicity: Each item contains a single step or fact.
Decidability: Criteria for fulfilling each item are clear and binary.
Independenee: Minimizing dependencies between different items.
Source: Key steps and evidential facts are extracted from the Golden Answer.
Explicit Referencing: Phrasing items checks for specific statements (e.g., "Did it prove that [statement]?") instead of referencing internal labels.
Figure Guideline For hints and checklist Annotation
23

---

Published as a conference paper at ICLR 2026
VALIDATION GUIDELINE
Your task is to ensure the creation of high-quality dataset for complex reasoning. You will be responsible for reviewing and refining data items, each consisting of a Research Question, Hints, Checklist, and a Golden Answer.
Data Screening
Source Verification: Confirm the academic authority and timeliness of data sources.
Content Qualification: Ensure the content is purely theoretical, excluding applied and empirical materials.
Difficulty Assessment: Filter for problems with high reasoning complexity and intellec- tual challenge.
Question Answerability Verification The core principle is to ensure the question itself is well- defined and answerable.
The input conditions, solution scope, and final objectives of the question must be unambiguous.
Clear Boundaries:
Complete Information: Provide the minimal necessary background knowledge and key information points required for understanding and solving the problem.
Logical Compliance: The problem statement and reasoning process must strictly adhere the norms and theoretical framework of the respective discipline.
Consistency Check Conduct a systematic verification four core components that constitute complete data item:
Overall Consistency:
The Question, Hints, Checklist, and Golden Answer must be logically self-consistent, mutually supportive, and free of contradictions.
Verifiability of Checklist Items: Each item the checklist must correspond to explicit evidence in the Golden Answer, with verification criteria that are clear and actionable.
Figure 10: Guideline For Quality Validation
24

---

Published as a conference paper at ICLR 2026
G SPECIFIC CASE OF ACADREASON BENCHMARK
PHIL PHILOSOPHY
Title:
Moral Understanding Between You and Me
Category: Philosophy
Research Question: Why shared moral understanding is important?
Golden Answer:
Moral understanding is an epistemic achievement about moral matters. To have it, your conception of moral issue must be accurate (the objective dimension) and also "make sense you" (the subjective dimension); your moral beliefs must be based directly in the reasons that make them true. You can know more than you understand; for example...
Shared Understanding as the Aim of Interpersonal Justification, Within the central moral practices of interpersonal justification..
Shared Understanding and the Norms of Apology.An apology should aim to reflect shared moral understanding of the wrong...
Reasoning with the Unreasonable.A worry about the shared understanding requirement is...
Checklist:
1. Does the answer include content related sharing moral understanding?
2. Does the answer include the view that the constitutive aim interpersonal justification is shared moral understanding?
3. Does the answer include content shared understanding in the context of interpersonal justification?
4. Does the answer include the view that an apology should aim to reflect shared moral understanding of the wrong done to recipient?
5. Does the answer include content on the Shared Understanding Condition of apology?
Hints:
1. Background: Much attention has been paid to moral understanding as an individual achievement, when a single agent gains insight into distinctly moral matters. But the importance of moral understanding cannot be fully explained by merely focusing on individuals' moral understanding...
2. Definition:
understanding: The capacity to grasp the moral significance of actions, principles, situations. It involves only knowing moral facts or rules but also appreciating the reasons behind them, recognizing the perspectives and experiences of others, and being able make sense of moral demands in context...
3. Methodology: giving account of what it takes for you and me share moral 2ew.rsttaningg..TTrruuu
comparison of the Delivery Model, the moral address view, and shared moral understanding, the constitutive aim of interpersonal justification is clarified as shared moral...
Figure The sample of Philosophy domain
25

---

Published as a conference paper at ICLR 2026
MATH
Title:
Sorting permutations using a pop stack with bypass
Category: Math
Research Question:
How can permutations be characterized and enumerated under sorting by pop stack equipped with a bypass operation? In particular, which forbidden patterns give necessary and sufficient criteria for sortability, how can a bijec- tion with suitably restricted Motzkin paths be constructed so that the counting sequence is the odd-indexed Fibonacci numbers, and how can one design and an- alyze algorithm compute
especcaall for permutations with few preimages and for principal classes-wind with a structural description of these sets? Furthermore, how do these results extend several pop stacks in parallel with bypass, yielding explicit bases for the sortable permutations, rational generating functions, and connections to classical sorting algorithms, with rigorous proofs throughout?
Golden Answer:
Pattern Characterization and Algorithm Optimality: Permutations sortable by the pop stack with bypass (PSB) are precisely those that avoid the patterns 231 and 4213...
Enumeration via Motzkin Paths and Fibonacci Numbers: Sortable permutations be encoded ternary words built from the PSB operations (PUSH BYPASS 1, POP+PUSH- = 2)...
Preimages under PSB: Every permutation has a well-defined of preimages under PSB. The algorithm for constructing preimages relies on decomposing permutation by its left-to-right maxima and...
Preimages of Permutation Classes: For certain principal classes, preimages under PSB remain classes. If the basis permutation begins with its maximum (na) or begins with the second maximum...
Checklist:
Defines fundamental concepts: permutation म, pop stack operations (PUSH, POP, BYPASS), and the pattern avoidance framework.
2. Characterizes PSB-sortable permutations by avoidance of patterns 231 and 4213, showing necessity and sufficiency.
3. Establishes bijection between sortable permutations and restricted Motzkin paths, proving the enumeration equals odd-indexed Fibonacci numbers.
4. Provides an algorithm for computing preimages under PSB and analyzes its correctness.
5.
Hints:
1. Background: 1. Sorting permutations combinatorics Central research topic, studied through containers like stacks, queues, and pop stacks. ! Pattern...
2. Definition: 1. Permutation basics + A permutation и of size is a bijection from n] to [1, n)]. written as π = πi ... T. Identity permutation:
id,, 12 . I Sets: S, = all permutations of size n, S
3. Methodology: 1. Sortability characterization - Goal: determine necessary and sufficient conditions for PSB sortability....
Figure 12: The sample Math domain
26

---

Published as a conference paper at ICLR 2026
LAW
Title: The Dilemmas of Schrödinger's Citizenship
Category: Law
Research Question: Exploring the contradiction of whether individual simultaneously hold the citizenship of one or more countries and be stateless (the "Schrödinger's citizen" dilemma), analyzes its legal roots and the impact the international human rights system.
Golden Answer:
The Essence of the Theoretical Dilemma. There is fundamental split in citizenship: The declarative nature asserts that nationality declaration of natural rights (such the principle of bloodline), and requires that the effect of nationality be retroactive to birth...
Reconstruction plans for the third-level systems. Level One: Amendment to international law. Amend Article IA(2) of the Refugee Convention: A. An explanatory clause has been added to clarify that the determination of "multiple nationalities" requires meeting three conditions:
Level Two: Reform domestic laws. Establish a special tribunal for judicial review of nationality: A. In the initial trial stage, two types of cases distinguished: for those involving the determination foreign nationality...
Checklist:
1. Point out the core issue: three major problems in determining citizenship (retroactivity, foreign courts' power, and mismatch rights with operabil- ity).
2. Conflict of laws path: prohibit foreign courts from interpreting nationality laws, require nationality review courts within the sovereign state, and ensure independent review of domestic cases.
3. Human rights path: establish graded responses to statelessness, redefine refugee standards, and change "potential nationality" to "actual administra- tive feasibility"".
4. Institutional design: eliminate obstacles to naturalization by setting a max- imum processing time and creating cross-border nationality verification centers.
Hints:
1. Background: 1. T.Theoretical Background (1'The declaratory declaratory.constiuuure constitutive dichotomy (Ross, Austin): Legal acts are divided into declaring natural facts (such birth) and creating new rights (such as naturalization). (2The theory of exclusive state sovereignty over nationality (Article 1 of the Hague Convention on Nationality)...
2. Definition:
1.Schrödinger citizenship: legal status where an individual is entitled the nationality of certain country under law, but in practice, it is not recognized by that country and is forcibly attributed by a third country. Declarartt citizenship...
3. Methodology: !.Normative Analysis research method: Deconstructing the Semantic Ambiguity of the "Multiple Nationality" Clause in the Refugee Convention. Empirical research method: Citing the naturalization rate of the United Arab Emirates in 2010...
Figure 13: The sample of Law domain
27

---

Published as a conference paper at ICLR 2026
COMPUTER SCIENCE
Title:
Tight Bounds and Phase Transitions for Incremental and Dynamic Retrieval
Category: Computer Science
Research Question: Determining the optimal redundancy R for retrieval data structures in the incremental and dynamic settings when the universe size is polynomial, i.e., U = (n).
Golden Answer:
For a polynomial universe [2]] poly(n), the optimal redundancy R = S - nw for retrieval data structures is: Incremental setting (insert-only). The optimal redundancy is:
Rine e(n + max log( (b)n)))
Equivalently:
If 2 clog (for constant > 0), then Rine (n).
If lo log log then Rine (n log log log n).
If log 0.99 n, then Rime = O log log
These bounds are tight: there an incremental structure with
S s no + 0(n) 0 log
log a
and matching lower bound
$ z no + ((n) + 22 log log
(for [M]] 2 n²),
giving the phase transition around x log n. Timewise...
Checklist:
1. Setup States |U] poly(n). Defines redundancy R = nw.
2. Incremental formula Gives Rinc O(n + max lo, log )))
Mentions phase transition at 0 x log n.
3. Incremental cases v 2 clog = Rine O(n). v log log log 0.99
log
Rice (n log log log n))
Ranc = On log logn). n).
4. Incremental bounds - Upper bound: S < no + O(n) + On log( log ")).
Lower bound: S no + Mm)) Ω(n log( logn )) for |0| n² o
5.
Hints:
1. Background: Retrieval data structures designed answer key-value
queries without explicitly storing the keys...
2. Definition:
Retrieval Data Structure: A data structure that answers key-value queries without storing keys explicitly. Given set of keys K C 2 and a v-bit value fkk for each key & E K, it supports queries of the form:
[((k),
if E K, anythinn, otherwise.
Query(k: =
Figure 14: The sample of Computer Science domain
28

---

Published as a conference paper at ICLR 2026
ECONOMICS
Title: Informational Black Holes in Financial Markets
Category: Economics
Research Question:
Suppose project be either benign type G inferior type B, with investors receiving independent private signals that satisfy the strict monotone likelihood ratio property (MLRP). The project is undertaken if and only if at least one investor participates. Please address the following: In a competitive market with N investors, demonstrate rigorously that a robust symmetric equilibrium is characterized by unique participation threshold, a participates if and only their signal 8; 2 &N. Explain the economic intuition behind the existence and uniqueness of this threshold. In the limit as N - the number of participants, k, converges to a Poisson distribution. Derive the limiting distributions for к conditional project types G and B. Furthermore, provide a complete derivation for the closed-fore expression of the parameter
(0, 1), where an investor
lim Pr(S; 2 SN A B), N+000
expressing in terms of the signal's is top likelihood ratio (A), the prior probability (TO). and the project's break-even posterior probability (T*).
Golden Answer:
Existence and Uniqueness of the Participation Threshold (sN): ): A robust symmetric equilibrium is one that holds even with small, non-zero participation cost. The equilibrium characterized by unique cutoff s satisfying the marginal investor's 's indifference condition-i.e. the investor with signal s is exactly indifferent between participating or not.
Asymptotic Analysis: As N > 30, the threshold an + 1. The probability of any single investor participating. Pr(S, 2 sN), approaches zero. The number of participants, KN,, which follows binomial distribution, therefore converges Poisson distribution under these conditions.
The rate parameter r defined under the bad state (0 B) as:
lim N Pr(S, 2 SN 0 B), N
and can be expressed as a function of (A, Tu. x²) based on the likelihood ratio and posterior thresholds.
Checklist:
1. Define robust equilibrium and explain how the winner's curse creates threshold participation strategy.
2. Prove existence and uniqueness of the participation threshold via the marginal investor's break-even condition.
3. Define parameter 7 as the limit of N Pa. where pa is participation proba- bility in the bad state.
4.
Hints:
1. Background: This problem is rooted in the economic theory of asymmetric information, where different parties in a transaction hold unequal knowl- edge...
2. Definition: Strict Monotone Likelihood Ratio Property (MLRP): The ratio of conditional densities fs(s)/ fs() ) is strictly increasing in the signal s. This ensures higher signal unambiguously "good news."...
3. Methodology: The existence of the threshold xx proven by analyzing the zero-profit condition for the marginal investor....
Figure 15: The sample of Economics domain
29

---

Published as a conference paper at ICLR 2026
30


Reviews:
[{"review_id": "IWHezcY1xt", "reviewer": "Reviewer_k3t6", "summary": "The paper introduces ACADREASON, a multi‑domain benchmark intended to test high‑level academic reasoning of LLMs (and agents). It uses 50 expert‑constructed problems drawn from theoretical papers in computer science, economics, law, mathematics, and philosophy. Construction proceeds via (i) paper selection, (ii) extraction of a formal question with a golden answer, and (iii) derivation of question‑specific checklists and three types of hints (background/definitions/methodology). Evaluation adopts an LLM‑as‑Judge scheme (GPT‑5‑mini) with two metrics: Pass Rate (probability of full match to the golden answer) and Checklist Score (probability of meeting checklist criteria). Experiments show low pass rates for state‑of‑the‑art LLMs, higher but still limited scores for agents, and gains when methodology hints are provided.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is well motivated. It shows a clear gap in current reasoning benchmarks. By collecting research-level problems from recent top venues, the benchmark is both meaningful and timely.\n* The task design encourages reasoning. Each question is self-contained, with a golden answer, a short checklist, and simple hints (background, definition, methodology). This supports step-by-step solutions and makes error analysis easier.\n* The evaluation has broad coverage. It tests many state-of-the-art models and several agent systems across five domains. Domain-level results and hint ablations make the findings clear and easy to compare.", "weaknesses": "- The dataset is small. While the examples are high quality, the limited size reduces representativeness. A semi-automatic or LLM agent system might help scale up questions.\n- The evaluation relies on a single judge (GPT-5-mini). There is no human calibration or test of consistency. Adding multiple judges (from different models, or more advanced LLM-as-Judge methods) and a small human study would make the results more reliable. \n\nOverall, the work is timely and useful as a benchmark. But it reads more like a benchmark release (more suitable for DMLR). For ICLR, I expect stronger methodological innovation—such as a calibrated multi-judge evaluation or a scalable automated pipeline—would make the contribution more suitable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1TZrWGmfz1", "reviewer": "Reviewer_Zbd9", "summary": "This paper introduces ACADREASON, a new benchmark designed to address shortcomings in current evaluations of Large Language Model (LLM) reasoning. The authors argue that existing benchmarks (e.g., math/code contests, general tasks) lack sufficient academic reasoning depth. To fill this gap, ACADREASON aims to evaluate the ability of LLMs and Agents to acquire and reason with specialized academic knowledge.\nThe benchmark consists of 50 expert-annotated academic problems across five high-reasoning domains: Computer Science, Economics, Law, Mathematics, and Philosophy. All questions are sourced from top-tier publications from 2023-2025 and underwent rigorous quality control to ensure they are both challenging and answerable.\nKey contributions include:\n1. The ACADREASON Benchmark: A challenging, cross-disciplinary benchmark focused on frontier academic reasoning, complete with golden answers, verifiable checklists, and three types of hints (background, definition, methodology).\n2. SOTA Model Evaluation: A systematic evaluation of over 10 state-of-the-art LLMs and Agents (e.g., GPT-5, o3, DeepSeek-R1, OAgents).\n3. Revealed Capability Gap: The results show that even the most advanced LLMs (GPT-5) score poorly (16% pass rate, 40.5% checklist score). Agents perform better (OAgents at 34% / 65.1%) but still show significant room for improvement.\n4. Hint Analysis: An ablation study demonstrating that providing hints (especially \"methodology hints\") significantly improves model performance, suggesting models struggle more with complex methods than with background knowledge.", "questions": "Based on the methodological analysis of the paper, we kindly request clarification and supplementary evidence regarding the following two core issues:\n1. Regarding the Benchmark's Construct Validity\nThe paper claims that ACADREASON is designed to evaluate a model's \"deep reasoning ability.\" However, the experimental results (for example, Claude-4-sonnet, which is widely regarded as having strong reasoning abilities, scoring 0) strongly suggest that the benchmark may be testing memory of specific, narrow, frontier knowledge (for LLMs) or search-and-extraction capabilities (for Agents), rather than transferable, general logical reasoning.\n- Question: Can the authors provide additional evidence or control experiments to demonstrate that ACADREASON genuinely measures the core variable of \"deep reasoning\" and has effectively isolated it from potential confounding variables such as \"specific knowledge-base\" and \"information retrieval ability\"?\n2. Regarding the Evaluation's Reliability\nThe paper's conclusions (particularly the extremely low model scores) are entirely dependent on the results from using GPT-5-mini as an \"LLM-as-Judge.\" Given that these tasks span five highly specialized and complex domains (e.g., Law, Mathematics, Philosophy), the evaluation difficulty far exceeds that of common tasks.\n- Question: Can the authors provide reliability validation for GPT-5-mini's role as the judge? For example, was an Inter-Annotator Agreement (IAA) analysis conducted between GPT-5-mini's scores and the scores from human domain experts (such as law professors or mathematics PhDs)? If such evidence is lacking, how can we trust the accuracy of these automated evaluation results?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "1. Originality & Significance:\n  - The paper addresses a clear and important problem: how to evaluate the deep, domain-specific reasoning capabilities of LLMs.\n  - ACADREASON's uniqueness lies in its combination of breadth (spanning both STEM and humanities) and depth (focusing on recent, theoretical problems from top-tier journals). This design tests reasoning on novel knowledge, not just retrieval of pre-existing, commonly known information.\n  - The benchmark's high difficulty (evidenced by low SOTA scores) confirms its utility and significance as an evaluation tool that is not easily saturated.\n2. Quality:\n  - The benchmark's construction methodology is rigorous, involving domain experts (Master's or PhD level) for data curation and annotation.\n  - A detailed, multi-stage validation process (shown in Figure 10) was used to ensure data quality, theoretical focus, and question answerability.\n  - The evaluation framework is rich. Beyond binary pass/fail ($R_p$), the \"Checklist Score\" ($R_j$) allows for a more granular analysis of the model's reasoning process. The inclusion of three hint types is a significant strength, enabling analysis of why a model fails (e.g., lack of background knowledge vs. lack of methodological understanding).\n3. Clarity:\n  - The paper is well-organized and easy to follow. Figure 1 provides a clear overview of the benchmark construction and evaluation pipeline.\n  - Task specifications, evaluation metrics (Sec 3.4), and experimental setups (Sec 4.1) are clearly articulated.\n  - Results are presented clearly (Tables 1 & 2), and the Case Study (Fig. 4) offers a concrete, intuitive example of the task's difficulty and the difference between Agent and LLM performance.", "weaknesses": "Based on an in-depth analysis, the paper suffers from three major and interrelated methodological flaws. These flaws severely undermine the validity of the benchmark and the reliability of its conclusions.\n1. Unverified Evaluation Reliability\nThe paper's core weakness lies in its evaluation method. The authors use GPT-5-mini as an \"LLM-as-Judge\" to automatically score model outputs.\n- Problem: This is a highly complex, expert-level reasoning task across five specialized domains (law, math, philosophy, etc.). The paper provides no evidence or validation study to prove that GPT-5-mini's scoring aligns with the judgment of human domain experts (e.g., a law professor or a mathematics PhD).\n2. Agent Data Contamination Vulnerability\nThe paper claims Agents (like OAgents) outperform LLMs, attributing this to their capabilities. However, the experimental design has a fatal \"open-book exam\" flaw.\n- Problem: 100% of the evaluation questions are sourced from publicly available, top-tier journal articles from 2023-2025. The tested Agents are permitted to use web search tools. This means an Agent can almost certainly find and \"read\" the original source paper for the question.\n- Impact: The paper claims to test the \"ability to acquire and reason over academic knowledge.\" However, this design cannot distinguish between \"logical reasoning from scratch\" and \"information retrieval + answer extraction + paraphrasing.\" The high Agent scores (up to 65.1%) are likely inflated and may test search ability, not the \"high-level reasoning\" the paper purports to measure.\n3. Flawed Construct Validity: Conflating \"Reasoning\" with \"Knowledge\"\nThis is the most fundamental issue: the benchmark likely fails to test \"general reasoning ability\" and instead tests \"memory of specific, narrow knowledge.\"\n- Problem: The experimental results show that Claude-4-sonnet, a model widely recognized for strong reasoning in other domains (like math and coding), scores a 0 on this benchmark.\n- Analysis: This contradictory finding strongly suggests that ACADREASON does not test general, transferable logical reasoning, but rather whether a model happens to \"know\" the frontier theories from these 50 specific papers.\n- Impact: The benchmark's construct validity is highly questionable.\n  - For LLMs (like GPT-5): The test is more of a \"memory test\" (i.e., were these 2023-2025 papers in its training data?).\n  - For Agents (like OAgents): The test is a \"search test\" (i.e., can it find the paper? See Flaw 2).\n\nOverall Conclusion: The benchmark fails to successfully isolate the variable of \"reasoning ability\" from \"specific knowledge-base.\" Therefore, the paper's conclusion that models like GPT-5 \"lack reasoning ability\" is unfounded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bOOkQkwy76", "reviewer": "Reviewer_RYgK", "summary": "The paper presents ACADREASON, a new benchmark designed to evaluate academic-level reasoning of large language models (LLMs). Unlike existing reasoning benchmarks such as MMLU-Pro, GPQA, or PaperBench, which focus on either broad factual understanding or narrow scientific tasks, ACADREASON targets deep, research-oriented reasoning across five academic domains: Computer Science, Mathematics, Economics, Law, and Philosophy. The authors curate questions from over 400 research papers and design multi-step annotations. They evaluate several frontier models (GPT-5, DeepSeek-R1, Claude 3.7, Gemini 2.0, etc.) and find that all perform far below human experts, especially on methodological and conceptual reasoning. The study also analyzes the role of hints and long-form responses, showing that factual hints help more than methodological ones.", "questions": "- Benchmark uniqueness. How does ACADREASON differ conceptually from PaperBench and GAIA beyond domain coverage? Would the authors consider positioning it as a complement rather than a replacement benchmark?\n- Human validation. How many samples were manually verified by human experts? Is there any inter-annotator agreement score (e.g., Cohen’s κ) for the judgment process?\n- Evaluator bias. Since GPT-5-mini is used as the evaluator, did the authors test whether results are consistent when switching to another LLM judge (e.g., Claude 3 Opus)? Are ranking trends preserved?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- High-quality benchmark design. The dataset is small but carefully curated. Questions are derived from genuine academic contexts rather than textbook or competition problems, giving ACADREASON a strong realism advantage. The multi-domain design broadens evaluation coverage beyond STEM, incorporating social science and philosophy.\n- Transparent and rigorous annotation process. The paper clearly documents every stage: data sourcing, question formulation, verification, and evaluation. The inclusion of structured checklists and fine-grained rubrics for reasoning quality (clarity, coherence, accuracy) improves reproducibility and reliability compared with prior subjective benchmarks.\n- Insightful empirical findings. The experiments reveal important trends: reasoning models still struggle on conceptual abstraction and logical grounding even when they perform well on applied math or coding tasks. The “hint effect” analysis (Table 2, Fig. 3) provides valuable insight into which types of contextual scaffolds actually help reasoning models.\n- Readable and well-structured paper. The narrative flows logically, figures are informative, and the motivation for each step is well explained. The benchmark and evaluation protocol could be easily adopted by others studying academic reasoning.\n- Contribution significance. Although not methodologically groundbreaking, ACADREASON fills a practical and conceptual gap between task-level reasoning (e.g., GSM8K, MATH) and domain-level scholarly reasoning. It contributes a valuable lens for assessing whether modern LLMs can reason like researchers rather than students.", "weaknesses": "- Limited novelty relative to existing benchmarks. While the dataset is well executed, the idea of academic or research-style reasoning has been partially explored in GAIA, PaperBench, and DeepResearchBench. ACADREASON’s main differentiator is diversity and annotation rigor rather than a fundamentally new evaluation paradigm. A clearer comparative discussion would strengthen its originality claim.\n- Scale and statistical power. The dataset contains only about 50 finalized questions, which limits robustness and makes performance variance hard to interpret. It’s uncertain whether differences across models (often within 1–2 points) are statistically meaningful.\n- Evaluation subjectivity. Despite the structured rubric, the “LLM-as-a-judge” setup remains vulnerable to bias and consistency issues. The authors mention human spot-checks but do not quantify inter-rater agreement or cross-model evaluation consistency. A partial human-judged subset would greatly improve reliability.\n- Limited actionable insight for model design. The results primarily reaffirm known findings that reasoning LLMs remain weak in multi-step conceptual reasoning, but offer little guidance on how to improve them. The benchmark thus functions more as a diagnostic dataset than a research breakthrough.\n- Scalability and sustainability. Because the pipeline relies heavily on expert curation and manual validation, it is unclear how ACADREASON could be expanded to larger scales or adapted to new domains without significant effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8fp3mMUrdB", "reviewer": "Reviewer_Gkt7", "summary": "This paper presents a new benchmark ACADREASON. \n\n- This benchmark is designed to be a challenging, reasoning-intensive benchmark for academic domains, including CS, Law, Econ, Math and Philosophy. Each domain contains 10 questions. \n\n- The benchmark is heavily human expert curated. The questions and answers are extracted and formulated by human experts from latest publications, along with hints and scoring checklist.\n\n- The paper also benchmarked the performance of latest Large Reasoning Model and Tool-used agents on ACADREASON. The results show the benchmark is very challenging.", "questions": "- As one aspect of the contributions is the difficulty, would it be possible to quantify the difficulty and compare with other benchmark in a more explicit way?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed benchmarks is a good contribution to the community. It has detailed and careful human expert annotations, which is a good effort. \n\n- The benchmark is well-positioned, as it focuses on challenging reasoning and academic domains.\n\n- The paper has benchmarked a wide range of latest LLMs and reasoning paradigms.", "weaknesses": "- The benchmark is relatively small, 50 questions in total. Although the scoring hints can give a bit more fine-grained signals, but in general the size is limited.\n\n- It would be better to include a comparison section against other relevant benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)