ICML 2025

AAAR-1.0: Assessing AI’s Potential to Assist Research
Renze Lou 1 Hanzi Xu 2 Sijia Wang 3 Jiangshu Du 4 Ryo Kamoi 1 Xiaoxin Lu 1 Jian Xie 5 Yuxuan Sun 5
Yusen Zhang 1 Jihyun Janice Ahn 1 Hongchao Fang 1 Zhuoyang Zou 1 Wenchao Ma 1 Xi Li 6 Kai Zhang 7
Congying Xia 5 Lifu Huang 3 Wenpeng Yin 1
Abstract
Numerous studies have assessed the proficiency
of AI systems, particularly large language mod-
els (LLMs), in facilitating everyday tasks such
as email writing, question answering, and cre-
ative content generation. However, researchers
face unique challenges and opportunities in lever-
aging LLMs for their own work, such as brain-
storming research ideas, designing experiments,
and writing or reviewing papers. In this study,
we introduce AAAR-1.0, a benchmark dataset
designed to evaluate LLM performance in three
fundamental, expertise-intensive research tasks:
(i) EQUATIONINFERENCE, assessing the correct-
ness of equations based on the contextual infor-
mation in paper submissions; (ii) EXPERIMENT-
DESIGN, designing experiments to validate re-
search ideas and solutions; and (iii) PAPERWEAK-
NESS, identifying weaknesses in paper submis-
sions. AAAR-1.0 differs from prior benchmarks
in two key ways: first, it is explicitly research-
oriented, with tasks requiring deep domain exper-
tise; second, it is researcher-oriented, mirroring
the primary activities that researchers engage in
on a daily basis. An evaluation of both open-
source and closed-source LLMs reveals their po-
tential as well as limitations in conducting so-
phisticated research tasks. We will keep iterating
AAAR-1.0 to new versions. Project Webpage:
https://renzelou.github.io/AAAR-1.0/
Work done prior to Jiangshu joining Amazon. 1Pennsylvania
State University;
2Netflix;
3University of California, Davis;
4University
of
Illinois
Chicago;
5Individual
Researcher;
6University of Alabama at Birmingham; 7Ohio State University.
Correspondence to: Renze Lou <renze.lou@psu.edu>, Wenpeng
Yin <wenpeng@psu.edu>.
Procee...