AAAR-1.0: Assessing AI's Potential to Assist Research
Authors: name: Renze Lou, name: Hanzi Xu, name: Sijia Wang, name: Jiangshu Du, name: Ryo Kamoi, name: Xiaoxin Lu, name: Jian Xie, name: Yuxuan Sun, name: Yusen Zhang, name: Jihyun Janice Ahn, name: Hongchao Fang, name: Zhuoyang Zou, name: Wenchao Ma, name: Xi Li, name: Kai Zhang, name: Congying Xia, name: Lifu Huang, name: Wenpeng Yin

Numerous studies have assessed the proficiency of AI systems, particularly
large language models (LLMs), in facilitating everyday tasks such as email
writing, question answering, and creative content generation. However,
researchers face unique challenges and opportunities in leveraging LLMs for
their own work, such as brainstorming research ideas, designing experiments,
and writing or reviewing papers. In this study, we introduce AAAR-1.0, a
benchmark dataset designed to evaluate LLM performance in three fundamental,
expertise-intensive research tasks: (i) EquationInference, assessing the
correctness of equations based on the contextual information in paper
submissions; (ii) ExperimentDesign, designing experiments to validate research
ideas and solutions; (iii) PaperWeakness, identifying weaknesses in paper
submissions; and (iv) REVIEWCRITIQUE, identifying each segment in human reviews
is deficient or not. AAAR-1.0 differs from prior benchmarks in two key ways:
first, it is explicitly research-oriented, with tasks requiring deep domain
expertise; second, it is researcher-oriented, mirroring the primary activities
that researchers engage in on a daily basis. An evaluation of both open-source
and proprietary LLMs reveals their potential as well as limitations in
conducting sophisticated research tasks. We will keep iterating AAAR-1.0 to new
versions.