AAAR-1.0: Assessing AI’s Potential to Assist Research
Authors: Renze Lou
Hanzi Xu
Sijia Wang
Jiangshu Du
Ryo Kamoi
Xiaoxin Lu
Jian Xie
Yuxuan Sun
Yusen Zhang
Jihyun Janice Ahn
Hongchao Fang
Zhuoyang Zou
Wenchao Ma
Xi Li
Kai Zhang
Congying Xia
Lifu Huang
Wenpeng Yin
Venue: ICML 2025 poster (2025)
Keywords: LLMs
Benchmark
AI4Research
Primary area: general_machine_learning->evaluation

Abstract: Numerous studies have assessed the proficiency of AI systems, particularly large language models (LLMs), in facilitating everyday tasks such as email writing, question answering, and creative content generation. However, researchers face unique challenges and opportunities in leveraging LLMs for their own work, such as brainstorming research ideas, designing experiments, and writing or reviewing papers. In this study, we introduce AAAR-1.0, a benchmark dataset designed to evaluate LLM performance in three fundamental, expertise-intensive research tasks: (i) EquationInference, assessing the correctness of equations based on the contextual information in paper submissions; (ii) ExperimentDesign, designing experiments to validate research ideas and solutions; and (iii) PaperWeakness, identifying weaknesses in paper submissions. AAAR-1.0 differs from prior benchmarks in two key ways: first, it is explicitly research-oriented, with tasks requiring deep domain expertise; second, it is researcher-oriented, mirroring the primary activities that researchers engage in on a daily basis. An evaluation of both open-source and proprietary LLMs reveals their potential as well as limitations in conducting sophisticated research tasks. We will release the AAAR-1.0 and keep iterating it to new versions.

AAAR-1.0: Assessing Al's Potential to Assist Research
Renze Lou't
Hanzi Xu² Sijia Wang
Jiangshu Du Ryo Kamoi Xiaoxin Lu Jian Xie Yuxuan Sun
Yusen Zhang'
Jihyun Janice Ahn
Hongchao Fang'
Zhuoyang Zou' Wenchao Ma Xi Li Kai Zhang?
Congying Xia Lifu Huang? Wenpeng Yin'
Abstract
Trxt Serructinn
conteutt paper prondied
Numerous studies have assessed the proficiency of AI systems, particularly large language mod- els (LLMs), in facilitating everyday tasks such as email writing, question answering, and cre- ative content generation. However, researchers face unique challenges and opportunities in lever- aging LLMs for their own work, such as brain- storming research ideas, designing experiments, and writing reviewing papers. In this study, we introduce AAAR-1.0, 1.0, benchmark dataset designed to evaluate LLM performance in three fundamental, expertise-intennive research tasks: (i) EQUATIONINFERENCE, assessing the correct- ness of equations based on the contextual infor- mation in paper submissions: (ii) EXPERIMENT- DESIGN, designing experiments validate re- search ideas and solutions; and (iii) PAPERWEAK, EAK- NESS, identifying weaknesses in paper submis- sions. AAAR-1.0 1.0 differs from prior benchmarks in two key ways: first, it is explicitly research- oriented, with tasks requiring deep domain exper- tise; second, it is essearcher-oriente mirroring the primary activities that researchers engage in daily basis. An evaluation of both open- source and closed-source LLMs reveals their po- tential as well as limitations in conducting so- phisticated research tasks. We will keep iterating AAAR-1. to new versions. Project Webpage: https://renzelou.c
Yowr
[-2 the weak z dejived beloat
This paper propores on digonttee
2- W"bbo
z- Whond
Intodduttion for propose !
where a the parameten sard preateei..!
Masci reforssossssooooooo [.]]
(A) / x
12/
Müsofrastin deru's
Wealmsssss
Task Equalion Inference
Task W3: Paper Weakness
Giaes a portisi paper meste oppriment plan explarations Careet
Adtesion mull with I-]
Commuore porforovce
Sigalfennce aoi L-J
Expetimen Pin
poper proposer alpootmm the
To prove the effertioeeess
2a study the impast of]
condurt poodd- www.oooooooooooooooooooooooo
To pvoii randernnus in.!! Mothation Explaxation
the experments
Task #2: Experiment Design
Figure 1: The input-output illustration of three tasks in the proposed AAAR-1.0 benchmark.
1. Introduction
Although AI has brought transformative changes vari- ous aspects of life, its impact researchers unfolds in nuanced manner. On the one hand, AI assists in various research disciplines, such as Social Science (Neuman et al., 2023), Finance (Gu et al., 2024), Medicine (Rakhimov et al., 2022), GeoScience (Praskievicz, 2018), etc., significantly expediting academic processes. However, many of these applications are superficial, often limited to data-driven clus- tering or classification. On the flip side, the Al era poses challenges for researchers. Despite its ability to streamline some activities, researchers still face demanding, cognitively intensive tasks such as staying current through extensive paper reading, rapidly generating ideas in response to fast- paced advancements, conducting rigorous experiments to substantiate claims, and managing an increasing volume of peer reviews. Then a question looms: How effectively can Al assisi reseurchers in lasks that are domain-specifee and geasoning-intensien
Work done prior to Jiangshu joining Amazon. 'Pennsylvania State University; "Netllix: Uniiersiity of California, Davis: University of Illinois Chicago: 'Individual Researcher: Universsity of Alabama Birmingham: 'Ohio State University. Correspondence to: Renze Lou <renze.lou@ @ psu.edu>, Wenpeng Yin ewenpeng@ @ psu.edu>.
Existing works proved the promising potential for using LLMs in assisting AI research. Si et al. (2024) conducted large-scale human study and found that LLMs can gen-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

AAAR-1.0: Assessing AT's Potential to Assist Research
erate creative research ideas. Lu al. (2024) proposed autonomous agent handle complicated research workflow and write whole research paper. However, most of these works focus on addressing highly subjective problems that require a high degree of expertise, making evaluation labori- ous and hard to reproduce. This underscores the need for a comprehensive benchmark that rigorously assesses LLMs' capabilities in expertise-intensti intensive research activities.
2024), particularly in Al research, such as generating novel research ideas (Kumar et al., 2024; Yu et al., 2024b), re- viewing research draft (Gao et al., 2024; Du et al., 2024; Liang et al., 2024; Zhu et al., 2025), and writing scientific papers (Chamoun et al., 2024; Lu et al., 2024; Weng et al., 2024). For example, Si et al. (2024) conducted a large-scale human investigation on LLM--gnerated research ideas and found that LLMs can generate novel ideas compared with humans while lacking feasibility. Du et al. (2024) found that while LLMs are effective at summarizing papers, they tend to overly trust the authors' claimed strengths and struggle to identify weaknesses specific to the paper. Furthermore, some works try to employ LLMs to solve more complicated research tasks that are composed of multiple steps (Li et al., 2024; 2023; Tang et al., 2023). Notably, Lu al. (2024) proposed AI-SCIENTIST, an autonomous agent framework that can handle series of challenging research tasks consec- utively, including generating research ideas, coming up with the corresponding experiments along with the implementa- tions, and then writing the final research paper - exactly how human conduct a whole research pipeline. However, there is still a lack of systematic evaluations and quantita- tive analyses on the LLMs' (intermediate) output of each single-step research task. Accordingly, our work focuses on building a benchmark consisting of individual research steps with clear input-output expectations, making it suitable for comprehensive L.MM evaluation. Moreover, we emphasize that relying on LLMs to fully replace human effort might compromise academic integrity. While our benchmark primarily serves an educational purpose LLMs assist ju- nior researchers by providing imperfect but insightful ideas, rather than by governing the entire research process.
To this end, in this work, we introduce AAAR-1.0 110, novel benchmark that aims to comprehensively assess the LLMs capacity on expert-level research tasks. As illustrated in Fig- ure 1, AAAR-1.0 1.0 decomposes three distinct expert-level AI research tasks from the researcher's 's daily activities, in- cluding i) EQUATIONINFERENNCE investigating whether the LLMS can infer the equation correctness based on the paper context; ii) EXPERIMENTDESIGN, validating LLMs' ability on designing reliable experiments for a research idea; and iii) SAPERWEAKNE discovered by LLMs from paper drafts. To ensure data qual- ity, senior Al researchers with extensive domain expertise perform data annotation for AAAR-1 1.. .0, followed by rig- orous multi-round data examination and filtering. All three tasks require models possess strong domain knowledge covering various cutting-edge research findings, as well as expert-level research experience, to the extent that even hu- mans need substantial research accumulation to tackle the tasks we designed. Crucially, tasks here are singular, stand- alone challenges (with clear input and output expectations) rather than a complicated task chain (Li al., 2024: Lu al., 2024), providing more transparent assessment of the model's intermediate output. Benefiting from the proposed automatic metrics, we conduct extensive experiments across numerous mainstream LLMs, where we find that:
testing the quality of weaknesses
Benchmarks for AI Research Tasks. Existing "LLM as- sists research" benchmarks mainly focus on the implemen- tation and execution part of the research pipeline (Lu et al., 2024; Chen et al., 2024a; Li et al., 2024; Chan et al., 2024). For instance, Huang et al. (2024) proposed MLAgentBench to test the LLMs' capacity for writing project code and train- ing the ML models, where the evaluation metric is the test performance of the models trained by LLMs. However, real- world AI research activities are diverse and some of them are hard to assess for quality, such as generating research ideas, which requires intensive manual assessment (Si et al., 2024; Liang et al., 2024). Our work centers on tasks that em- phasize a comprehensive mastery of the scientific research field and core elements of researcher's 's daily workload, and we try to build curated task-specific metrics for every single task for a more efficient and accurate LLMs appraisal.
With a random guess baseline of 40% Fi, the perfor- mance of most LLMs on EQINFER hovers just slightly above chance, with the top models reaching around 46%. This highlights the difficulty of the task, despite its reliance primarily on local context reasoning.
In EXPDESIGN, LLM-designed experiments are inno- vative and more diverse than those by humans; how- ever, many are trivial, lack feasibility, and stray from the original research objectives.
In PAPERWEAK WEAKNSS, LLM-identified weaknesses of- ten lack depth and specificity, making them broadly applicable and less useful for providing feedback on paper drafts.
2. Related Work
3. AAAR-1.0
LLMs for AI Research. With the rapid evolution of per- taining techniques, LLMs are found to be useful in assisting various research disciplines (Yu et al., 2024a; Labrak et al.,
Figure 2 provides a data construction overview. In the fol- lowing sections, we elaborate on the data collection de-
2

---

AAAR-1.0: Assessing AI's Potential to Assist Research
Task #1: EQINFER
Task #2: EXPDESIGN
Task #3: WEAKNESS
Rediewett proporess
contrct betore x W a contact after
TEX
irvive
PDF
PNG
Open
PDF
nove retre
Lo
Shurrelnlet
Paper POE
Bion Serresann
Cleansc Paper
①
Data Crawling and Cleaning
i Data Crawilng
Data Crawling
The
Gy: this experiment di?
Reviowart papp
fffer
Putiraction conduct aladen shoxthe
Nltued rnternnee
Undaar tasle L-)
Clasned Paper
GPT4
2 W
Ogi Why is bitnncessarr?
Arnotator
fita Comeenan
Sansbracces
LLM--used Equation Syathesi
L.M--based Fittering
Dumain-expert. Annotation
C LLM--based Weakness Extraction
2- W aab
ViLA
JSON
Paelow Option
W b bb-a
PDF
Exports
PDFFigures
z= W "b
Sestecee
Annetater
bagatiw Options
① Expert-based Examination
N Multi-ronn Peer Discusion
a Input Data Prucessing
Figure 2: Data construction workflows of the three tasks in
AAAR-1.0.
tails, including $ 3.1 IQUCATIONI (EQINFER) $ 3.2 EXPERIMENTDESIGN EXPDESIGN ), and 3.3 PA- PERWEAKNEST ( WEAKNESS), ).
@ LLM-based equation synthesis. As EQINFER assess- ing whether the LLMs can infer the correctness of equation positive
(i.e., binary classification), for each equation, we have to craft counterpart negative equations. To this end, for each positive equation, we prompt GPT-4 to synthesize a negative equation based on the paper context. We repeat this prompt (with a high decoding temperature) until three different negative equations are synthesized."
3.1. EQUATIONINFERENCE
Crafting correct scientific equation in paper writing or validating an equation in paper reviewing is challenging, as it requires a thorough understanding of an algorithm or the intricate relationships among numerous variables. Di- rectly prompting LLMs to generate equations proves overly demanding. Therefore, this work formulates EQINFER (Figure 1) binary inference task.!
@ LLM-based filtering. However, the L.M-syythetti equations can be context-unaligned, i.c., some synthesized equations contain notation that is never defined in the pa- per context, which becomes a superficial shortcut and too effortless for LLMs to identify. To improve data quality, we prompt GPT-4 to identify context-unaligned negative equations. We then eliminate the positive equation and its negative counterparts. where all three negative counterparts are unaligned. This filtering leads to final of 1,449 posi- tive equations and 4,347 negative equations (each positive equation has three negative counterparts, and at least one negative counterpart is "challenging").
① Data crawling and cleaning. For the data source, we adopt the pre-compilation LaTeX code for two reasons: i) existing PDF parsing tools, such as PyMuPDF and Paper- Mage (Lo et al., 2023), can introduce considerable noise to the parsed equation text; ii) considering most of exit- ing LLMs are capable with processing LaTeX code, using LaTeX source instead of parsed text can be more accurate and provide LLMs with richer information. Meanwhile, we only crawl those peer-reviewed papers accepted by top-tier conferences avoid using low-quality human-written equa- tions. Accordingly, we first obtain the accepted paper list from ACL Anthology, from year 2019 to 2023. Next, we search each paper on arXiv to crawl its LaTeX source (if exists). Finally, we get a total of 1,762 papers' source LaTeX packages. We then clean the LaTeX sources by delet- ing all the comments and combining multiple cross-referred tex files into main file. Afterward, we use regex to ran- domly extract (at most) 3 equations' code snippets per paper, resulting in 3,877 human-writted equations.
① Expert-based examination. Furthermore, it's also pos- sible that synthesized negative equations are actually correct (i.e., false negative) - even if the negative and positive equations are written differently, the final compiled results might be the same. We then employ human experts to re- view the data further and filter out false negative equations, checking the classification instances for accuracy.
We asked senior PhD students who are experienced in AI research to check all instances. We ask human experts to consider the following criteria for each positive equation and its negative counterparts (each pair): i) Are all equations
EQINFER also facilitates multiple-choice QA setting; while we find binary inference is more challenging for LLMs.
Thhe number of negative equations is empirically decided.
3

---

AAAR-1.0: Assessing AT's Potential to Assist Research
grammatically correct? ii) After compilation, are all negative equations different from the positive ones? We ask every human expert to use external LaTeX compilation tools (e.g., TeXlive), and identify the pairs that cannot meet the criteria. Each pair is examined by at least two experts, and we only keep pairs that all experts decide to keep. After this strict examination, total of 1,049 pairs are eventually kept (27.6% pairs are filtered)
annotator has concisely answer two questions: i) What did this experiment do? ii) Why did the paper authors conduct this experiment? In other words, we ask the an- notator to summarize all the key experiments in this paper and explain the underlying motivations hased on their rich domain experience.
@ Multi-round peer discussion. Intuitively, different ex- perts might have different opinions on the same research topic. Particularly, when explaining the underlying moti- vation of experiment, adopting only single expert's s opinion might introduce bias to our annotation. Hence, we conduct a further multi-round peer discussion. For each paper, where all the key experiments are identified, summa- rized, and explained, we ask a different expert (reviewer) to review the annotation by considering the following three criteria: i) Are the identified experiments all the key ex- periments? ii) Does each experiment summarization covers all key information? iii) Does each explanation sound reasonable and reliable? Each reviewer must leave comments on the online PDF regarding the above criteria, and then the annotator must respond to each comment either accept the suggestion and revise the previous anno- tation or provide a "rebuttal" the reviewer to uphold the annotation. This discussion is iterative until both opinions align. Eventually, for each paper, collect two lists: i) the experiment list, summarizing each experiment step of the paper: ii) the explanation list, the underlying motivations that are one-one corresponding to the experiment.
Final data. We finally obtain 1,049 positive equations (each has three negative counterparts). We show data statis- ties of EQINFER in Table 7 and data examples in Figure 8.
3.2. EXPERIMENTDESICA
Given a research topic, such as a novel ML algorithm, a qualified researcher can design a solid experiment plan for it, and clarify underlying motivation to ensure the reliability of the designed experiment. Unlike the concurrent works that focus on the experiment implementation (Lu et al., 2024; Huang et al., 2024), emphasize the importance of assessing the high-level experiment design of L.MMs be- fore the subsequent implementation to avoid any expensive execution iteration. Therefore, as shown in Figure 1, we formulate EXPDESIGN as a gext-generation task that takes pre-experiment paper context as input, and then generates the experiment and explanation list.
@ Data crawling. As for the data source, we first collect > 10k papers' data from arXiv, including LaTeX sources and PDFs, which cover broad Al categories, including cs.AI, AI, cs. cs,CL, and cs.CV, from year 2018 to 2023. Sim- ilarly, to ensure the source data quality, we only use papers that have appeared well-known conferences.
Final data. After annotation, we use the pre-experiment context of each paper (according to the first-experiment lo- cation identified by the annotator) as the input. Furthermore, we use GPT-4 to delete any sentence that potentially leaks the experiment from the input. 3 Similar to the EQINFER, we utilize the source LaTeeX as the input text to avoid PDF paring noise. As for the image input, we collect those fig- ures within each paper's 's source LaTeX package and only keep figures that are used in the pre-experiment context. Overall, a total of 100 instances are collected. As shown in Figure 1, the input of each instance is the pre-experiment context (including the figures), and the ground- -ruth output is the expert-annotated experiment plan and the explana- tions. Table 8 shows data statistics and Figure 9 illustrates the sample case in EXPDESIGN.
@ Domain-expert annotation. Making reliable and ex- ecutable experiment plan requires solid foundation knowl- edge of a specific research area. Consequently, we set a high standard for choosing annotators: i) be a senior Ph.D. stu- dent with at least one peer- peee-reviewed publication in leading Al venues; ii) have more than 4 years of Al research experi- ence; iii) frequently serve conference reviewers. Finally, we invite total of 10 qualified experts to participate in our data collection procedure. Given the 10k crawled papers, we first ask every annotator to bid on the papers that they are interested in. After bidding, each of them is assigned 10 papers, i.c., a total of 100 papers to be annotated. Dur- ing annotation, we post each paper PDF on online Google Drive and ask the annotator first carefully read the whole paper. Then, we ask them to identify and locate the key experiments in each paper (i.e., highlighting the relevant paragraphs of each experiment). We don't 't consider some trivial experiments, such as those supplemental analyses in the appendix section. For each identified experiment, the
3.3. PAPER WEAKNESS
Another critical research task is paper review. Previous works have demonstrated the usefulness of the LLM--ased review feedback (Gao et al., 2024; Jin et al., 2024; Lu et al., 2024). However, as indicated by Du et al. (2024); Liang et al. (2024), LLMs only excel at summarizing the research
'About 9.8% sentences are deleted.

---

AAAR-1.0: Assessing AT's Potential to Assist Research
strengths while falling significantly short on weakness criti- cism. Hence, we build WEAKNESS for particularly inves- tigating the LLM-generated weaknesses.
Final data. Our final data is composed of 993 instances, each input is paper text along with figure/table images, and each output is peer reviewers' weakness lists. Table 9 shows data statistics; Figure 10 presents an example of the data instances. We show the data diversity (score and track dis- tribution) in Figure 3.
① Data crawling. We first crawl total of 3.779 anony- mous submissions of ICLR 2023 from OpenReview. includ- ing PDF and other meta information (e.g., scores, decisions, and tracks). As the ICLR 2023 has 13 distinct tracks while the paper distribution across different tracks is highly biased, we then uniformly sample papers from different research tracks to improve the domain diversity. Meanwhile, during sampling, we also keep the accept/reject papers distributed equally to avoid data bias. In word, we finally collect to- tal of 1,000 papers (500 accepted; 500 rejected), uniformly covering all 13 tracks. Please refer to Figure 3 for the track and score distribution of the 1,000 papers.
4. Evaluation Criteria
For EQINFER, we adopt F1 as the classification criterion. For EXPDESIGN and WEAKNESS, since both tasks have free-form outputs, we develop several novel task-specific metrics in addition to the conventional ROUGE (Lin, 2004).
We use LLMs to evaluate the experiment list of EXPDESIGN, Specifically, given a model-predicted ex- periment list p. and the ground-truth list g. we calculate:
@ Extraction of bumtnewritten weaknesses. Since the raw comments crawled from ICLR 2023 are mixed with both strengths and weaknesses, we further employ GPT-4 to extract all the weaknesses from each reviewer's comments and compose multiple weaknesses into a list. Notably, we force GPT-4 to keep the original text of the reviewer, i.e., all weaknesses in our dataset are those original sentences written by the reviewer without any modifications." 5 What's more, sometimes one reviewer might repeatedly mention the same weakness throughout the comment. In this case, we simply keep all the repeated weaknesses because, if one weakness is repeatedly mentioned by the reviewer, it's intuitively an important weakness that the reviewer wants to emphasise; accordingly, keeping the repeat items can penalize LLMs more on missing this weakness.
En En-Precision =
f(x. g)
(1)
En-Recall Σ [(g p)
(2)
where the m and n are the list length of p and g; f()) represents the LLM prompting, where we prompt LLM to decide whether each predicted experiment item (p:) is en- tailed by the whole ground ground-truth list (g), proceeding with binary output, and vice versa. Intuitively, En-Precision re- flects how many prediction experiments match ground-truth experiments. In this work, we used GPT-4o as evaluator.
While for the explanation generation of EXPDESIGN, as the prediction experiments are one-on-one corresponding to the ground truth, we adopt a semantic-based metric:
For each paper, can finally get multiple weakness lists (one weakness list per reviewer, one paper can have multiple reviewers). We further delete a few papers without any weaknesses found in the raw comments, resulting in a total of 993 instances, i.e., 993 (paper, weakness lists) pairs.
m S-Match Σ
sim(ps, pp...)
where we use SentenceBERT (Reimers, 2019) to measure the semantic similarity between p; and gj-
Unlike EXPDESIGN, the ground truth of WEAKNESS is multiple reviewers' weakness lists. Instead of merely merg- ing the opinions of various reviewers into one flattened list and keeping LLM-as-judge as the metric (which is not only costly but also loses the structural information of diverse research perspectives), we employ the following semantic- based metric to efficiently evaluate predicted weaknesses:
Input data processing. As we mentioned before, we crawl papers from OpenReview instead of arviv because the under-review paper draft is required for this task. However, not every paper from Reen can be found on arXiv, i... the source LaTeX code and figures of most under-review papers are unavailable. Therefore, we utilize VILA (Lin et al., 2023) to parse text data out from the PDF; we also employ PDFFigures-2.0 (Clark & Divvala, 2016) to extract all the figures and tables (in image) from the paper, as Vila not good at processing the table data.
m2 Σ
S-Precision
max sim 99,
(4)
3
1=1
S-Recall
1
Σ
max sim gf. pt)
(5)
*We adopt ICLR because releases full submissions, while some other conferences only release accepted papers.
'We manually checked GPT-4's extraction results of 200 cases GPT-4 only missed < 1 % of reviewer-written weaknesses and maintained almost all the original text.
where r is the number of reviewers of the given paper, Mi means the length of k-th reviewer's weakness list, and g)
5

---

AAAR-1.0: Assessing AT's Potential to Assist Research
indicates the j-th item in k-th reviewer's weakness list.
Table I: Various LLMs' performances on EQINFER task
(1,049 positive and 3,147 negative samples). "All- Aol-possiive" indicates a baseline that predicts all equations as positive.
Additionally, in the real world, we would think a review weakness is reliable if it is specific to paper. Mcanwhile, we also hope the review is informative, I.e., no excessive similar weaknesses in one review. Inspired by the classic TF-IDE, we propose a novel review diversity metric:
Methods
F1
Prec.
Rec.
All-Positive
40.00
25.00
100.00
ITF-IDF
1
log
m/ 02
x log
W
(6)
Open-source LLMs
W
m
RNC
OLMo-7B (Groeneveld et al., 2024)
13.64
11.93
15.91
Mistral-7B (Tiang et al., 2023)
28.45
19.28
54.24
Matral-8x22B-Mou (Jiang al., 2024)
40,90
26.15
93.80
0
sim (p² (p/.pt) m()
Qwen 2.5-72B (Qwen Team, 2024)
31.22
26.28
57.40
Llama 3.1-70B (MetaAl, 2024)
33.08
22.14
65.39
Closed-source LLMs
R²- max sim(p!
(8)
Gemini 1.5 Pro (Anil ct al., 2023)
46.74
32.05
86.27
Claude 3.5 sonnet (Anthropic, 2024)
45.13
29.48
96,18
GPT-4o (UpenAl. 2024a)
40.35
30.79
58.53
where the w is the total number of papers in the dataset, p' is j-th paper's prediction weakness list, y is the i-th weakness in p². Moreover, O2 calculates the intra-paper occurrence frequency of pç: R. is the "soft" number of papers that also contain the pt. which is computed by summing the maximum similarity scores between pa and other paper's 's weaknesses. In a word, O measures informativeness, and Ri measures specificity. The complete ITF-IDF consider both aspects and reflects the overall weakness diversity.
ol preeeve (OpenAT, 2024b)
46.35
31.43
88.27
03-mini (OpenAI, 2025)
47.98
34.34
79.59
et al., 2023). The generally high recall with low precision of all LLMs also indicates real-world risks, e-g., relying on LLMs to check the validity of equations in paper review.
Do more contexts boost performance? EQIN- FER places high demands on reasoning within the scientific context. To quantify the impact of input context length, we scale the input length (per side) from 100 to 1,500 words. As shown in Figure 4, for the open-source L.LMS (Ilaama and Qwen), an appropriate context length can boost the per- formance; while for GPT-4o, scaling up the context length doesn't 't contribute much to the F1. However, during the scaling, we find that the precision of GPT-4o is gradually in- creased, and the recall is decreased accordingly; considering the label distribution of EQINFER, we believe precision can better reflect the model's true capacities on this task. Thus, we anticipate that scaling up context shall be beneficial to those strong close-source LLMs such as GPT-4o.
Q:
5. Experiments and Analyses
In this section, we conduct extensive experiments on AAAR-1.0, across various mainstream LLMs, to quan- tify the current LLMs' capacity to tackle high-level re- search tasks. Specifically, $ 5.1 for EQINFER, $ 5.2 for EXPDESIGN, and $ 5.3 for WEAKNESS Please refer to Appendix B.2 for running details of the LLMs.
5.1. EOUATIONINFERENCE
Settings. As different LLMs have distinct context win- dows, to ensure a fair comparison, we fix the maximum input length for all models. According to Table 7, we em- pirically use 1,000 words for both contexts before and after equations, i.e., 2,000 surrounding words.
5.2. EXPERIMENT
Settings. Similarly, we unify the input context length of different LLMs to ensure fair comparison. According to Table 8, we set 2,000 and 3,000 input words for open- and closed-source LLMs, respectively. Meanwhile, as ex- periment explanation is the subsequent task of experiment design, using -ener-egeee experiments can propagate errors in explanation, leading to inferior results for most LLMs. To this end, we provide LLMs with the oracle exper- iments when generating explanations.
Main results. Table 1 shows the main results. Firstly, simple baseline that predicts all equations as positive achieves 40% F, (due to the 1:3 of positive and negative equations), while nearly all open-source LLMs even cannot beat this naive baseline. Notably, though the performance of Mixtral is slightly superior to the baseline, the extremely biased precision and recall scores imply that Mixtral is also simply predicting almost all samples as positive instead of truly inferring. Meanwhile, compared to the All-Positive bascline, the performance superiority of the strong close- source LLMSs is not significant, the best LLM on this task only obtains 47.98%, which demonstrates the challenge of EQINFER compared with other similar benchmarks (Song
Main results. Table 2 shows the main results. For the experiment design, the closed-source LLMS generally out- perform open-source LLMs. However, the score values of all LLMs are relatively low (20%~~0%) implying the LLMs consistently miss ground -rouh experiments from the origin paper (low recall), and they tend to generate
6

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Table 2: Various LLMs' performances on the 100 instances of EXPDESIGN, The explanation generation is based on the
oracle experiments to prevent error propagation. "Copy Input" directly copies each experiment idea as the explanation.
Experiment Design
Experiment Explanation
Methods
En-Fi
En-Preccsson
En-Recall
S-Match
ROUGE-I.
ROUGE-1
Copy Input
40.32
22.06
25.28
Open-source LLMs
OLMo-7B (Groeneveld et al., 2024)
14,80
17.50
19,80
45.78
26.30
30.38
Mistral-7B (Jiang al., 2023)
18.96
24.83
21.38
50.18
30.20
34.69
Seeeal-x22B-MMMM (Jiang al., 2024)
23.16
24.45
30.57
49.07
29.96
34.53
I.lama 3.1-70B (MetaAI, 2024)
22.92
23.10
29.76
50.05
29.33
34.11
Qwen 2.5-72B (Qwen Team, 20124)
24.28
22.48
34.44
51.12
29.46
34.68
Closed-source LLMss
Gemini 1.5 Pro (Anil et al., 2023)
27.25
28,
34.92
52.87
28.52
33,80
Claude 3.5 sonnet (Anthropic, 2024)
27.99
24.48
42.09
53,03
18.75
26.15
GPT-4o (OpenAI, 2024a)
25.03
22.25
36.59
54.79
27.54
34.31
ol-preview (OpenAI, 2024b)
30.13
28.13
38.59
58.55
29.11
36.70
03-mini (OpenAI, 2025)
30.17
28.70
37.67
54.01
20.71
29.14
more novel experiments that didn't 't show in the origin pa- per (low precision). As for the experiment explanation, the
Table 3: The human evaluation results on the novel experi- ments suggested by LLMs. "A", "B", and "C" represent the different quality level (i.e., necessity); "A" is the best level.
S-Match scores of closed-source LLMs still surpass the open-source LLMs. Furthermore. there is a negative cor- relation between S-Match and ROUGE score, where the ROUGE scores of closed-source LLMs are broadly inferior. We find that the open-source I.LMs often try to copy the terms or phrases from the given experiment, or even sim- ply paraphrase the experiment instead of explaining, which results in a high superficial overlap with the ground-truth explanation. This observation highlights the importance of adopting the proposed S-Match to avoid evaluation bias of traditional generation metrics.
Necessity (%)
Models
of novel EXP
Gemini 1.5 Pro
59
30.59
45.76
Claude 3.5 sonnet ol ol-preview
112
21.78
50.00
71
35.84
36.61
Table 4: The impact on S-Match scores of maintaining the experiment's self-containment for EXPDESIGN
Q1: What is the quality of the model-generated novel experiments? The low En-Precision of I.LMs in Table 2 indicates the creativity of LLMs in generating novel experi- ments. We then randomly sample 15 papers from the Ex- PDESIGN and ask 3 experts to manually review the model- generated novel experiments. Specifically, we ask the ex- perts to judge necessity of the novel experiments, where we set three necessity levels: "A" indicates the experiment necessary/mandaoory support the main claim, "B" rep- resents optional//supllementar experiments, and "C" for those unrelated experiments (see Appendix C.2 for evalua- tion details). Table 3 shows the necessity scores of the three strongest LLMs. We find that LLMs consistently generate a lot of novel experiments, especially the Claude; though most of them are optional, even fancy/unrelated experiments, there are still considerable amount of necessary experi- ments generated, e.g., the results of ol. We further find that some novel experiments can be regarded as useful supple- mentary analyses w.r.t. the human experiments. Table 11 shows examples of model-suggested experiments.
Models
One-by-One
Whole-List
Llama 3.1-70B
50.05
49.36 (4 0.7)
Qwen 2.5-72B
51.12
48.56 (1 2.6)
Gemini 1.5 Pro
52.87
57.48 (1 4.6)
Claude 3.5 sonnet
53.03
59.11 (1 6.1)
GPT-4
55.03
56.95 (1 1.9)
GPT-4o ol-preview
54.79
58.54 (1 3.8)
58.55
61.58 (1 3.0)
2z Can self-contained experiment design enhance the experiment explanation? When generating the explana- tion in Table 2, we provide LLMs with each individual ex- periment and let them explain by one, because find that, when providing the whole experiment list, those open- source models only explain partial experiments because of their poor instruction- following capacity. However, there are intuitively some semantic or logical relations between different experiments, e.g., some experiments are prerequi-
7

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Table 5: The human evaluation results on LLMs' output explanations of EXPDESIGN: "Ace. ratio" means how many model outputs are accepted by the annotator.
understanding of the paper. just like how human experts do. Hence, there is still considerable gap between the LLMss and humans in terms of grasping research motivations.
Models
Acc. ratio
2s: Does multi-modal input boost performance? Intu- itively, besides the text, when designing experiments for a given research topic, the figures can provide rich supplemen- tary information, such as an algorithm illustration that can help better understand this research topic and underlying motivations. Hence, we test the performance of different LMMs (Large Multimodal Models), including GPT4-o and Intern VL.2 (Chen et al., 2024b). Table 12 shows the abla- tion results on the figure data. To our surprise, the figure data doesn't 't improve the LMMs' results in this task, even harming the performances. This might be due to the low informativeness of the figures, as figures usually consume more input tokens but act only as supplementary informa- tion to the text, indicating future work on developing LMMs that can effectively leverage the scientific figures.
Llama 3.1-70B
22.93
Gemini 1.5 Pro
55.07
Claude 3.5 sonnet
61.46
GPT-4o
69.72
ol-preview
76.14
sites to others. Therefore, this one-by-one prompting might break the self-containment of experiment plan. Conse- quently, we test with the "whole-list" prompting, where the LLMs are given the complete experiment list and are asked to explain all experiment steps together.
As shown in Table 4, unlike the open-source LLMs, the explanation performances of those closed-source LLMs are generally improved after adopting whole-list prompting. Ac- cording to further manual checking, after maintaining the self-containment of the experiments, the LLMs can refer to other experiments and better grasp the underlying motiva- tion of the current experiment.
5.3. PAPERWEAKNESS
Settings. Intuitively, full paper content is necessary for paper reviewing. Therefore, instead of setting a maximum input length, in WEAKNESS, we try to utilize the whole paper. As the input length of WEAKNESS is extremely long (see Table 9), we adopt a "split-combine" method we first split the whole paper into smaller pieces and let LLMs predict the weaknesses of each piece separately; after that, we merge all pieces' weaknesses as a final prediction. For the length of each small piece, we set 2,000 and 3,000 words for open- and -ouure LLMs, respectively. Ad- ditionally, in this task, we also examine the performance of AI-SCI (Lu et al., 2024), which enhances LLMs' paper re- view ability by leveraging advanced prompting techniques, e.g., self-reflection (Shinn et al., 2024) and response ensem- bling (Wang et al., 2023). 6
23: Do human evaluation results align with automatic metrics for explanation? As the explanation can be open- ended, in this paragraph, we provide the human evaluation results on different LLMs' experiment explanation outputs. In detail, we randomly select 20 out of 100 papers and ask 5 annotators to read the experiments along with each model's I's explanations; we then let the annotator decide whether each model's explanation is acceptable (see Appendix C.3 for more details). Table 5 illustrates the results, where the score variance is higher than Table 2. However, the performance ranking of both tables is perfectly correlated with each other (Spearman's 's rank correlation coefficient 1), demonstrat- ing the effecttveness of S --tathh.
Main results. Table 6 shows the main results, where the closed-source LLMs' overall performances are generally su- perior to the results of open-source LLMs. Similarly, closed- source LLMs are particularly excellent in S-Recall because of more generated weaknesses. However, there is still a considerable gap in the weakness diversity between the LLMs and human experts.? Compared with human review, most LLM-generated weaknesses are vague and lack the necessary knowledge about some frontier research works. Surprisingly, AI-SCI performs worse than backbone GPT-
21: Do more contexts boost performance? We also in- vestigate the impact of input context length for EXPDESIGN. As shown in Figure 5, we scale up the input pre-experiment context length from 0. to 10k tokens (10k is the length of the longest paper). For the experiment design, more input context does improve the performance of different LLMs, while this benefit stops after exceeding 8k tokens, which means that after the necessary information has been cov- ered, scaling context becomes inefficient. Meanwhile, the explanation generation results reveal that LLMs primarily depend on given experiments rather than paper context to explain motivations. However, we do not expect this as we hope I.LMs can explain the motivation based on a thorough
'We don't 't run AI-SCI on EXPDESIGN, because AI-SCI takes model-generated ideas as the inputs, which are incompatible with our task setting.
'The human's ITF-IDF score in Table 6 can be slightly under- estimated. This is hecause we keep the repeated weaknesses in the human review, which affects the human review's informativeness (lower ITF) but is useful when calculating the S-Recall for L.M.s.
8

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Table 6: Various LLMs' performances on the 993 instances of WEAKNESS
Weakness Diversity
Methods
S-F 1 (%)
S-Precision (%)
S- S-Recall (%)
ITF-IDF (t)
Human Review
7.69
Open-source LLMs
OLMo-7B (Groeneveld et al., 2024)
43.25
40.38
47.04
2.45
Mistral-7B (Jiang et al., 2023)
42.03
43.80
40.77
1.17
Mixtral-8x222 -oarrl--222BMMoo (Jiang et al., 2024)
43.23
44.59
42.23
0.98
Llama 3. 1-70B (MelaAI, 2024)
42.78
43.19
42.70
2.60
Owen 2.5-72B (Qwen Team, 2024)
42.74
43.80
42.05
1.21
Closed-source LLMs
Gemini 1.5 Pro (Anil al., 2023)
48.75
43.97
55.08
5.88
Claude 3.5 sonnel (Anthropic, 2024)
47.85
41.97
56.00
3.91
GPT-4o (OpenAI, 2024a)
47.73
42.09
55.48
5,95
ol op-preveew (OpenAl, 2024h)
48.62
42.54
57.08
5.63
03-mini (OpenAI, 2025)
46.33
42.00
51.99
5.85
LLM Agent Framework
AI-SCI (GPT-4o) (Lu et al., 2024)
45.05
40.02
51.91
2.23
4o, especially on ITF-IDF, which suggests the challenge of WEAKNESS, i.e., simply adopting popular prompting techniques cannot well address this task.
two LMMs investigate the effectiveness of image inputs. Overall, image information, including both figures and ta- bles, doesn't 't bring significant performance improvement, i.e., only Intern VI.2 gains a performance boost after incor- porating figures; while tables slightly drop both models' results. This is probably because the LMMs cannot reason well over the coemmatton-intenivve images, especially the table images.
Q1: Is the split-combine effective? Ideally, if the LLM has sufficient context window size, splitting the input pa- pers for separate processing is unnecessary. Consequently, in this paragraph, we utilize the LLMs accepting long con- text input to compare "split-combine" with "no-split", i.e., letting LLMs write weaknesses by giving the full paper. In practice, we set the maximum number of input words to 20k, be fully
6. Conclusion
which ensures 295% papers in the WEAKNESS processed. As shown in Table 10, compared with giving the full paper contexts, split-combine generally brings about su- perior performances. During manual checking, we find that, when full paper is available, LLMs frequently neglect some important sections and omit weaknesses accordingly, while split-combine ensures that the LLMs can carefully brain- storm weaknesses within each smaller piece. Surprisingly, the LLMs' performances with full paper context be even worse than just remaining the first 3,000 words. This implies that even the current powerful long-contedt LLMs still fall short when processing long scientific documents.
In this work, we propose AAAR-1.0 0, novel benchmark targeting a comprehensive evaluation of the current LLMs' capacity in assisting Al research. AAAR-1 1.0 consists of distinct expertise-intensin tasks along with the curated eval- uation metrics. We collect high-quality data by employing senior Al researchers and conducting strict data examina- tions. Extensive experiments highlight the challenges and values of AAAR-1.0.
Acknowledgments
The authors would like thank Ibraheem Moosa and Sarkar Snigdha Sarathi Das for assisting the data collection.
22: Does multi-modal input boost performance? Our dataset covers both tables and figure illustrations extracted from the paper PDF as inputs. Intuitively, when reviewing paper, both figures and tables are critical, not only for better understanding, but also because some weaknesses related to tables/figures." Therefore, in Table 13, we adopt
Impact Statement
Our study explores whether LLMs can assist human re- searchers in AI research. We do not advocate for Al replac- ing human researchers. Instead, we stress that the primary responsibility for scientific research should remain with hu- mans prevent societal risks, with LLMs serving tools
*We find that there is approximately one human-written weak- ness related to figures or tables in each paper.

---

AAAR-1.0: Assessing AT's Potential to Assist Research
enhance research efficiency. Specifically, our work analyzes the strengths and weaknesses of LLMs to ensure researchers remain judicious in their use of these tools. Our goal is to mitigate risks while maximizing the benefits offered by L.Mss. We are committed to the careful distribution of data collected in our research, ensuring it is used solely for research purposes.
Z., Cheng, L.,, Ranathunga, S., Fang, M., Fu, J., Liu, F, Huang, R., Blanco, E.,, Cao, Y., Zhang, R., Yu, P. S., and Yin, W. Lims assist NLP researchers: Critique paper (meta- breeta-grvie In The 2024 Conference on Empirical Methods in Natural Language Processing, 2024. doi: 10.48550/ARXIV224066 16253. URL https:///oi..rgg 10. 48550/ arw.v.. 2406. 16253.
Gao, Z., Brantley, K., and Joachims, T. Reviewer2: Op- timizing review generation through prompt generation. arXiv preprint arXiv:2402. 10886, 2024.
References
Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, E.,, Heslow, D., Lau- nay, J., Malartic, Q.. Noune, B., Pannier, B., and Penedo, G. Falcon-4OB: an open large language model with state- of-the-art performance, 2023.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E.,, Pyatkin, V., Ravichan- der, A., Schwenk, D., Shah, S., Smith, W. Subramani, N., Wortsman, M., Dasigi. P., Lambert, N., Richardson, K., Dodge, J., Lo. K., Soldaini, L.,, Smith, N. A., and Ha- jishirzi, H. Olmo: Accelerating the science of language models. Preprint, 2024.
Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Team, G., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXi::2312. 2023.
Anthropic. Introducing claude 3.5 sonnet. https:// sonnet, June
www. http://ppp....omm////////aeaeeee 2024.
Gu, J., Ye, J., Yin, W., and Wang, G. Adaptive and ex- plainable margin trading via large language models on portfolio management. In Proceedings of the 5th ACM International Conference on Al in Finance (CCAI)24)), 2024.
Chamoun, E., Schlichktrull, M., and Vlachos, A. Auto- mated focused feedback generation for scientific writing assistance. arXiv preprint arXiv:2405.20477, 2024.
Chan, J. S., Chowdhury, N., Jaffe, O., Aung. J., Sherbur,, D., Mays, E.,, Starace, G., Liu, K., Maksin, L., Patward- han, T., et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.070 2024.
Huang, Q., Vora, J., Liang, P., and Leskovec, J. Mlagent- bench: Evaluating language agents on machine learning experimentation. In Forty-first International Conference on Machine Learning, 2024.
Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z. Wei, C., Lu, Z., et al. Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:24 10.04000000000 2024a.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXi:22310.06225 2023.
Jiang. A. Q. Sablayrolles, A.,, Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. 2024.
Chen, 7., Wang, W., Tian, H., Ye, S., Gao, 7., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024b.
Jin, Y., Zhao, Q. Wang, Y., Chen, H., Zhu, K., Xiao, Y., and Wang. J. Agentreview: Exploring peer review dynamics with Ilm agents. arXiv preprint arXiv:2406. 12708, 2024.
Clark, C. and Divvala, S. Poffigures 2.0: Mining fig- ures from research papers. In Proceedings of the 16th ACM/IEEE-C on Joint Conference on Digital Libraries, pp. 143-152, 2016.
Kumar, S., Ghosal, T., Goyal, V., and Ekbal, A. Can large language models unlock novel scientific research ideas? arXiv preprint arXiv:2409. 06/80001888. 885, 2024.
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Venkit, P. N., Zhang, N., Srinath, M., Zhang, H. R., Gupta, V., Li, Y., Li, T., Wang, F., Liu, Q. Liu, T. Gao, P., Xia, C., Xing, C., Cheng. J., Wang. Z., Su, Y., Shah, R. S., Guo, R., Gu, J., Li, H., Wei, K., Wang,
Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rou- vier, M., and Dufour, R. Biomistral: A collection of open-source pretrained large language models for medi- cal domains. arXiv preprint arXiv:2402. 10373, 2024.
10

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Li, Jiang, H., Zhang, T., Yu, Z., Yin, A., Cheng, H., Fu, S., Zhang, Y., and He, W. Traineragent: Customizable and efficient model training through llm-powered multi- agent system. arXiv preprint arXiv:231 1.06622, 2023.
OpenAL. Openai 03-mini. https: / /ttsss://pppaaa..m com/ laaoomiiin index/ openai-o3-mini/, January 2025.
Praskievicz, S. River classification a geographic tool in the age of big data and global change. Geographical Review, 108( I): (108(1):120-11)) 120--337, 2018.
Li, R., Patel, T., Wang, Q., and Du, X. Mir-copilot: Au- tonomous machine learning research based on large lan- guage models agents. arXiv preprint arXiv:2408. 14033, 2024.
Rakhimov, M., Akhmadjonov, R., and Javliev, S. Artificial intelligence in medicine for chronic disease classification using machine learning. In 2022 IEEE 16th International Conference on Application of Information and Communi- cation Technologies (AICT), pp. 1-6. IEEE, 2022.
Liang, W. Zhang, Y., Cao, H., Wang, B., Ding, D. Y., Yang, X., Vodrahalli, K., He, S., Smith, D. S., Yin, Y., et al. Can large language models provide useful feedback on research papers? large-scale empirical analysis. NEJM AI, 1(8):Alo224001% 2024.
Reimers, N. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Lin, C.-Y. Rouge: A Package for Automatic Evaluation of Summaries. In Text summarization branches out, pp. 74-8 81, 2004.
Shinn, N.. Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models, 2023.
Si, C., Yang, D., and Hashimoto, T. Can Iims generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint Sttt:::4090041009 2024.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Song, L., Zhang. J.. Cheng, L.,, Zhou, P., Zhou, T., and Li, I. Niphench: Evaluating large language models solving nip problems. arXiv preprint wxXiv:230.15660 2023.
Lo, K., Shen, Z., Newman, B., Chang, Z., Authur, R., Bransom, E., Candra, S., Chandrasekhar, Y., Huff, R., Kuchl, B., et al. Papermage: A unified toolkit for process- ing, representing, and manipulating visually-rich scien- tific documents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 495-507, 2023.
Tang, X., Liu, Y., Cai, Z., Shao, Y., Lu, J., Zhang, Y., Deng, Z., Hu, H., An, K., Huang, R., et al. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. arXiv e-prints, pp. arXiv
2311, 1, 2023.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The AI Scientist: Towards fully automated onen-ended scientific discovery. arXiv preprint arXiv:2408.0620 2024.
Team, G. Google launches gemma 2, its next generation of open models. https://blog http://lllloggggle/eoooooooo google/technologe technology/ developers/ wwwggoogggmaaa----oo Jun 2024a.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024b. URL https:://wwelll gi thub. io/blog/ qwen2.5/.
MetaAI. Introducing llama 3.1: Our most capable models to date. https://at //ai. meta. wwt.a.coo/...a..e www.//oo//////oeeceeeeeoooeoeeoe wwlpo//mmtaaaaaaaa2-2- July 2024.
Wang, M., Chen, I.., Fu, C., Liao, S., Zhang, X., Wu, B., Yu, H., Xu, N., Zhang, L.., Luo, R., et al. Leave no document behind: Benchmarking long-context 1lms with extended multi-doc qa. arXiv preprint arXiv:2406.1 17419, 2024.
Neuman, Y., Cohen, Y., and Yin, W. Identifying social norm violation in movie plots: from borat to american pie. Digit. Scholarsh. Humanit., 3864)1633-1665, 2023. doi: 10. 1L110///L//OAA00022222 URL htRps:::s///oioorr 10. 1t..///1/c/faa005.222
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL http:///eereviiieeeee/////dddddd
OpenAl. Hello gpt-4o. https://openai..om//dddee hello-gpt-40/, May 2024a.
OpenAl. Introducing openai ol. https://openaal com/ index/intrrdducnngoppene prpenai-ol-previwww/ Septem- ber 2024b.

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Weng, Y., Zhu, M., Bao, G., Zhang. H., Wang, J., Zhang, Y., and Yang. L.. Cycleresearcher: Improving auto- mated research via automated review. arXiv preprint arXiv:2411.0086, 2024.
Yu, B., Baker, F. N., Chen, Z., Ning, X., and Sun, H. Llas- mol: Advancing large language models for chemistry with a scale, comprehensive, high-quality instruc- tion tuning dataset. arXiv preprint 1r2999240,.0939,, 2024a.
Yu, H., Hong, Z., Cheng, Z., Zhu, K., Xuan, K., Yao, J., Feng, T., and You, J. Researchtown: Simulator of human research community. arXiv preprint 17X772241..17767, 2024b.
Zhu, M., Weng, Y., Yang, L.,, and Zhang, Y. Deepreview: Improving llm-based paper review with human Mike deep thinking process. arXiv preprint 2rXiv25503.08569, 2025.
12

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Appendices
Within this supplementary material, we elaborate on the following aspects:
Appendix A: Data Statistics and Diversity
Appendix B: Implementation Details
Appendix C: More Experiment Results and Details
Appendix D: Data Cases and Annotation Platform Illustration
Appendix E: Prompt Templates
A. Data Statistics and Diversity
We provide the detailed data statistics of three datasets in our benchmark, as shown in Table 7, 8, and 9, We use the NLTK package" to tokenize words and count the length. When calculating the length of equations, we use the pylatexenc tool to simplify the equations first.
Meanwhile, for the WEAKNESS, we also plot the review scores distribution of the papers used in the dataset, as well as the track distribution. As be found in Figure 3, our dataset has decent distribution, where the papers are uniformly distributed across 13 tracks, and most papers' scores ranged from 5 to (i.e., most papers weakly rejected or accepted).
Table 7:
The statistics of EQINFER Here, the "left" and "right" input context indicates the paper contexts before and after the missed equation; "pos." " means the ground-truth equations (written by the source paper authors), while "neg." the GPT4-synthetic wrong equations.
# of positive equations
1,049
# of negative equations
3,147
# of source papers
869
ave. "left" input context length (in words) ave. "right" input context length (in words) max "left" input context length (in words) max "right" input context length (in words) min "left" input context length (in words) min "right" input context length (in words)
4,377
6,362
24,849
32,948
711
8
ave. "pos." " output equation length (in character) ave. "neg." " output equation length (in character) max "pos." output equation length (in character)
55
48
1,039
max "neg." output equation length (in character)
306
min "pos." " output equation length (in character)
min "neg." " output equation length (in character)
4
B. Implementation Details
B.I. Metric Details
When calculating the metrics, specifically for the similarity-based scores, we utilize SentenceBERT (Reimers, 2019) to encode each segment (e.g., each experiment idea in the list) into a dense vector, and then calculate the cosine similarity," !I which takes about IGB of memory when running on single A100 GPU.
hhttps://ww/ org/ "hhttps:/github. wwtpp//aaast/pyllateeeeeoe
13

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Table 8: The statistics of EXPDESIGN
# of instances
100
# of source papers
100
ave. input context length (in words)
4,288
max input context length (in words) min input context length (in words)
9,799
698
ave. # of input figures
2.6
max # of input figures min # of input figures
16.0
0.0
ave. length of Experineen&Explaaaiinn list
5.7
ave. length per experiment (in words)
34.3
ave. length per explanation (in words)
27.1
max length of Experimen&&Explaaatton list max length per experiment (in words)
13
135
max length per explanation (in words)
89
min length of Eeteriiment&Epplaaatttii list min length per experiment (in words)
2
min length per explanation (in words)
9
B.2. LLMs Running Details
In our experiments, we utilize various LLMs, including both closed and open-sourced. We list the model weight sources for the open-source LLMs:
OLMo-7B (Groeneveld et al., http:://s//uuuneeeeeeeee////////
Falcon-40B (Almazrouei et al., 2023): httpss//wgginnffee..oootteeeeeee
Gemma 2-27B (Gemma Team, 2024): http:://hhggiifffooeoooooo////22
Mistral-7B (Jiang et al., 2023): http:://wggaaaae...cesrrrrrrssss
Mixtral-8x22B-Mon (Jiang et al., 2024) http:://hhgggiggg.e.ccrssrrrrsrs
Llama 3.1-70B (MetaAI, 2024); httpp:///nggiiifeeeccoeeeeenle/d
Qwen 2.5-72B (Qwen Team, 2024): https://hugginggaace http://ggaaaa..ee//ee///eeeeee//
We use VLLM to unify the inference endpoints all the above models." We use Pytorch 2.4.0 with CUDA 12.1, and use NVIDIA A100 GPUs for the LLMs inference.
Meanwhile, we use the gpt-40-2024408-06 gpt-4-1 106-preview, for the closed-source LLMs. We use LiteLLM to unify the API calling for all these LLMs. 13
10-err------- ol --ppriiew-2224-09--- gemini- 1.5 -.5-pro-002, and claude-3-
Given the unstable performance of L.Mss, particularly closed-source ones, we run each model thrice during our experiments, selecting the median result from these repeated runs.
http:ttutb..mmmmmmmmopeet/t/ll "httpps///gts thub. com/BerriAT/litelll
14

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Table 9: The statistics of WEAKNESS
# of instances
993
# of source papers
993
ave. input context length (in words)
9,811
max input context length (in words)
49,195
min input context length (in words)
24
ave. of input figures
7.0
max # of input figures
37.0
min # of input figures
0.0
ave. of input tables max # of input tables min # of input tables
4.3
53.0
0.0
ave. of reviewers per paper max # of reviewers per paper min # of reviewers per paper ave. of weaknesses per reviewer max # of weaknesses per reviewer min # of weaknesses per reviewer ave. length of weakness (in words) max length of weakness (in words)
3.8
9.0
3.0
4.8
39.0
1.0
39.1
371.0
min length of weakness (in words)
2.0
C. More Experiment Results and Details
C.1. Input Context Scaling Investigation
Figure 4, Figure 5, and Table 10 show the context scaling results of EQINFER, EXPDESIGN, and WEAKNESS.
Table 10: The performance comparison of different input processing methods for WEAKNESS. We use GPT-4o and GPT-4-Turbo because both accept a maximum of 128k tokens input. We also put the results of AI-SCI in the table for reference. Here, "split-combine" splits the input paper into several pieces, where each piece's length is denoted as "window size"; "no-split" means the conventional input cutting, for example, if the window size is 3,000, then only the first 3,000 words in the paper are used. According to the data statistics, 20,000 words can cover maximum lengths of more than 95% of the papers in our datasct.
Input Context
Window Size (in words)
Models
S-F1
S-Precision
S-Recall
ITF-IDE
Processing
split-combine
3,000
47.73
42.09
55,48
5.95
GPT-4o
no-split no-split
3,000
45.74
43.45
48.54
5.92
20,000
45.47
42.97
48.51
6.02
split-combine
3,000
45.05
40.02
51.91
2.23
AI-SCI
no-split
3,000
42.56
40.90
44.65
2.53
no-split
20,000
42.53
40.75
44.78
2.58
15

---

AAAR-1.0: Assessing AT's Potential to Assist Research
C.2. Human Evaluation LLM--Genrraedd Novel Experiments
Figure 6 illustrates the evaluation guideline for novel experiments generated by LLMs. We ask 3 senior PhD students evaluate each paper; that is, if the first two annotators disagree with each other, a third annotator will make final decision. Table 11 presents several human evaluation cases.
C.3. Human Evaluation LLM-Generated Explanation
We ask 5 annotators to evaluate the LLM-generated explanations. Specifically, each of them is assigned 4 or 5 papers, along with the corresponding experiment lists. For each paper, the annotator is given 5 different models' outputs (model names anonymized), and the annotator has to decide if each L.LM-generaed explanation is acceptable according to the experiment. We show the human evaluation results in Table 5.
C.4. Multi-Modal Input Ablation
We post the multi-modal ablation study of EXPDESIGN and WEAKNESS in Table 12 and Table 13.
D. Data cases and Annotation Platform Illustration
As shown in Figure 9, and 10, we show the sample cases of the three tasks AAAR-1.0. Meanwhile, we illustrate the screenshot of our annotation platform in Figure
E. Prompt Templates
In this appendix, we attach all the prompts used in this work, including prompts in data collection and model prediction, as shown in Figure 11, 12, and 13.
16

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Overall Score Distribution
Score Range
[1. 3)
5)
[5. 6)
[6, 8)
10)
40.1%
2.9% 1.4%
7.8%
47.8%
(a) The review score distribution the papers used in WEAKNESS
82
82
82
92
62
82
82
81
81
80
33
0
Optimzztoon
Theory
Genstratii model
General
Neurocciaace Cagsltiee
Intraatruuuurrs
Meti Wathods
Scierraz
Adpects
oppicatnnatiinalll learnng
and adl-supervinnd
(b) The track distribution of the papers used in WEAKNESS
Figure 3: The data diversity illustration of WEAKNESS including the score distribution and track distribution of the papers used in our dataset.
17

---

AAAR-1.0: Assessing AT's Potential to Assist Research
GPC-4c
25
46-
Llam331
26-
32.
100
300
500
900
11000
1.300
1.38
Iaput Context Length (# of Words)
Figure 4: The input context length scaling trend on the EQINFER task.
GP
Qwen22
Experiment Design (Eacail-F1)
Input Context Longth (# of Words)
Qum-21 Lima- J.1
Experiment Ixpllmation (s-Macch)
30
0.5x
Input Context Length of (ords)
Figure 5: The input context length scaling trend of different LLMs the EXPDESIGN task.
18

---

AAAR-1.0: Assessing AT's Potential to Assist Research
For each paper, you are given this papers human annelated exgarmonts (Column CJ along with three difiorent moce prod ction experiments (Columns D,G. Those model modei generated expertments are novel experments shat the original humon annotated experiments (Column C) diant mention. And your tack is evoluate whother these novel experiments are good not Baoed on the orginal paper and nss expermenty ments, pls rate the quality of each model-generated experiment.
(nacessary expariment): Label an axperiment
you think axperiment necessary for paper:
"necessary" experiment mean authors induda this experiment paper, paper highty likety rejected by the redenver exampa paper proposes noval neural adaptor model, ablation study recuieed
having the proposed adaptor can contributa performance.
loptional experimenti: For example, paper proposes new mobro leaming algor conducting represeraatinn space Vulalzzatoon required but useful enhanding the explainabiliy of aggorthmm
experiment with you
experiment optional choice this paper.
experiments experiment you think experiment unrelated core motivation paper. these fancy experiments can just without any Impact. experiment.
model-generaled experiments general, simply suggening
study" any then you categorise
"Your Assisamant column, For there any you confidont ralings.
the model ganerated experiments,
wric longth
Figure The human guideline for evaluating the LLM L.M-geeerated novel experiments.
Hare, provitee suggstad unnotaiin Fieliio
you
you palpppp
For each experiment within the "Experiene" section by tre fillwing tio aedices: What expertesets de you auppest dbong? (estume thes sheed
two queesions making commentts boter undersandi-)
diverty Highiihhing the reicked peragaghhutseeeeeeeooo fguess
partof your
eaey your annstations inte Orgamize expeitmnt suggest Into the for cokorna
should wrihe sometiing
AAA
aon....
if experieent the
itens sheet
you go to see 20.05t sheet and use mootees forrat as nene (hg.. how make comments PDF).
Other noter
wockute fre th author oxsedily clarced Bhe enportance of Nequenty exniiored a expermaor es the saper's mann cody.
socusder experiructe the
the soiical
Paper 113e
POF Lass
Wehal eqaeinent auggent thing?
Why de you magaed theme scaetr and
Few-ehot instruction
comparionn wcross difi tnventgate whath the cument LMfe tuty se-atand the
rodleb/ fow inatrauss 1. Pror go Busod Moddl Roaly Un Stw.. awu. woo. 2 The effoct of the langet words. The auttrr should to invellguts whorece To seself the model can buly follw insructionn to solve the uas
Fowashut trsiruction duthowing perfrmmane areny efferent indrousidd Teseoil alffrrrt intrratioo a van
Crese-tash iratructin--olteee following performance evaturtion the suthonu 1. To prove that the lask essructioe in the proposed dataast dn both trai
Ablation studi different components task instraction: 2e2 Since suthor prosoced marous comconanta tank inatactiin
Figure 7: The annotation platform for collecting the annotation of EXPDESIGN We ask annotators to first make comments on the Google Drive PDF, then move all the annotations the online Google Doc (for further verification and discussion).
Contest Befere
Context After
Equation
Answer
tn tiis paper, we inesstigate what types of starestspica: Langange the first deteset compeintng stenectypical attrosutes of a rang of entrratred larguge scealc to in untapervisee tactise. dareouer, ve evergent stereotypes
mma (eno)
group stark L.v.)
cootises as heans stuts their ewotioaa effects in sore gonera rarner le.el
Io tiis paper, Breestigste what types of sterestypical coptured
the drsset corpeteing steretrspical aetrinutes of a range elleit
pretraiied lanuugg sccets in an ussuperviset fastiae. Hareower, ve group stats l..e! sonifestatinn emotiees is seans to stuty their enotional effects in lore
then seftie ecctioon #tsertcpsttiyyaateettaaaaes each
(IN, CT6T3 Vicu Dv Win N,STOT3 Scorels, cno) sncerrect
"Viteatnores de3 (ww.eaaateeereeettt Wifracd1)
Figure 8: Two sample cases of EQINFER
19

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Table 11: Examples of human evaluation on the model-generated novel experiments.
Puper Title
Original Experiments (by bumun)
Novel Experiment (by LLMs)
Rating
Analysis Verification Problem Distribution: paper should provide detailed analysis and statisties about the verification problems the proposed datasct,
2. Off-the-shelf entailment classification perfornance: The authors should provide catailment classification performance of existing models on the proposed dataset without fine- mne-tuning,
Human Performance: The authors should show hu- man performance on the proposed datasct.
Performance of fine-tuned models: The authors should provide the performance models fine-tuned the propesed dataset.
WICE: Real- World Entailment for
Assess model performance WiCE without Raning to test domsin gencralization from tradi- tional NI datasets.
Claims in Wilipedlia
A
Performance on evidence retrieval task: au- thors should show the performance the evidenee retricval task, which is a sub-tack the propoeed dataset.
6. Performance of LLMs: The authors should provide the performance LLMs on the proposed dataset.
Retrievale Entailment: Authors should provide ex- periments framework retrieving evidence and evaluate entailment by using the retrieved sentences.
Analysis of Claim-Spli Downstream Tacks: The authors should analyze how claim-split, the proposed method, is effective on other the proposed dataset.
Results multiple LLMs popular math datasets: The authors should show the performance of multiple LLMss fine-tuned their daraset on popalar math datasets.
Performance on open-source models with different sizes: authors should show the performanes of models with different sizes trained on the proposed dataset.
Comparison SOTA closed-souree models: The authors show compare the performance of open-souree mod- els trained on the proposed datasct strong close-souree models.
MetaMathe Bontstrop Your Own Muthematicul Questions for Large Language
Evaluate the effect of iugmentations: The authors need to perform an ablation study to compare the different argumentation methods they proposed.
Prompe Sensitivity Analysis: Evaluate the sensi- tivity of MctaMath to different prompt formats or phrasings mathematical questions.
B
Analyze Training on Incorrect Answers: The au- thors should analyze whether wrong data augmeentation can harm the performance.
generated in
increase the size of training data: The authors should evaluate other ways to increase the training data and compare performance with models trained on their proposed train data.
6. Evaluate other
Ererr Analysis: The authors should analyze the formance their models in different conditions (e.g., lengths questions).
The authors should evaluate self-corection performance with oracle labels. Intrinsic Self Correction:
Deff-Correction with Ornele Labels:
authors should perfor-
mance without using the oracle labels.
Analysis of Mistakes in Self-Corection The authors should analyze the properties of mistakes made self-correztion
Visualization of leamed representations or at- tcution mochanisms to provide insights into the model's inner workings.
Large Language Models Cannot
Correct Reasoning Yer
framewerk.
Multi-Agent Debate:
evaluate self-
correction with multi-ageot debate.
5. Prompt Design Analysis: The authors should analyze the influence prompt design for the initial responses self- correction performance.
20

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Table 12: The figure inputs ablation of EXPDESIGN For the maximum text imput length, same the setting in Table 2, we use 2,000 and 3,000 words for open- and closed-source models, respectively. For the closed-source GPT-4o and GPT-4, as they have long context window sizes, we use all the figures of each paper. While for Intern VL2, we randomly select two figures per input paper.
Experiment Design
Experiment Explanation
Models
En-F1
En-Precision
En-Recall
S-Match
ROUGE-L
ROUGE-1
GPT-4o
25.03
22.25
36.59
58.54
29.25
35.50
w/ figures
25.39
24.35
32.80
58.53
27.87
34.30
IntemVLL-226
24.26
39.50
14.91
50.03
29.13
34.26
w/ figures
15.04
38.50
8.64
50.29
29.29
34.06
The ablation study about the paper tables and figures WEAKNESS Based on the conclusion in Table 10, we use the "split-combine" to process the text input here (2,000 and 3,000 words context window size for open- and closed-source models). For GPT-4o, we use all the table/figure images; while for IntemVL2,, we randomly select two images random figure table.
Table 13:
per paper, i.e., two random figures, random tables,
Models
S-F1
S-Precision
S-Recall
ITF-IDE
GPT-4o
47.73
42.09
55.48
5.95
w/ tables
46.76
41.32
54.17
5.53
w/ figures
46.62
41,20
54.04
5.48
w/ tables & figures
46.58
41.17
53.98
5.36
Intern VL2-26B
41.91
41.02
43.28
1.48
w/ tables
40.55
40.37
42.91
1.46
w/ figures
42.88
42.10
43.76
1.46
w/ tables & figures
42.44
42.00
43.31
1.44
Pre-Experiment Context (Input)
Experiment Design (Output)
Motivation Explanation (Output)
1. To check whether non-entity everlap between two languages also contribute to better per formance on recognizing entities. recognize additionally, successfully recogniring that word entity also contributes the performance on the NER task.
may information
Expand the definition of overlap. authors should calculate overlap based on all the words shared between tuo languages, instaad of just shared vecabelary on just the entities.
understand which features language sodel can inghtsts model typological sinitarity sultilinguat pretraining.
In this paper, we show that Multilingual BERT (\Anbert()), released Nccttet(de\LL2218eeerr
gains using popular language similarity criterion, e.g-, WALS.
Report
exploit cress- Lingual transfer. This give
a single
language nodel pre-trained from languages,
Effect of tokens per word. The authors should perfor experiments on nore scripts, specifically looking being split
3. To understand the effect of POS label frequency. The idea that twe tangusges with sinilar taken to word ro result better cross-Lingual transfer. The reason
monolingual сorpora surprisingly geod at zero-shot cross- lingual model transfer, in which task- specific annotations used to fine-tune the model for evaluation another language [...]
efrect of nultiple
continustion properly continuation perfomance.
should corpus differen: will result
language
Control for vocabulary overlap languages, Choose languages that have large vocabulary overlap and different word order feature, Train on one set of languages and perform zero shot evaluation on
properly control for the effect of vacabulary overlap. Since large overlap in vocabulary can lead to performance gain, reported resuits daes the
Ablate the effect of connon word pieces using non-over non-overlapping tokenizer for different languages.
5. To understand the effect of structure of sentences different languages for Lragu--Lngua understanding of innuul language nodels. Since there will be 8o everlap between different languages the model nust learn on syntactic and cross- Linuuii representationn languages. properties
Figure 9: A sample case of EXPDESIGN
21

---

AAAR-1.0: Assessing AT's Potential to Assist Research
Paper Context (Input)
Weaknesses (Output)
Revievent1:
1. The writing is not on per with the idea.
Reviever92:
1. would he inforsative to see haw MPNPs scale with higher dinensionality. for exarple, enpirical comparisons on high-D regression task comp conplementing the 10 one.
2. The results of the Lotsa-Votterra task would deserve further analysis: Why is ENP/BAMP seamingly rore apt at dealing with nisspecification than HPMPs My under rstanding is that rodel data-niswatch is prooles general to Bayesian inference, s.c., should also affect BIAINP.
Neural
IGarnelo at., 2018a;bl
stochastic process describing relationship between inputs outputs in a given data streal, where each task the data strean consists of a neta-training set of input-output pairs rs and also neta-ve neta-velidation process whose functional fom deternined by neural network input [...]
Reviewerd3:
set, The NP then defines an implicit stochestic
1. The consistent outgertoraanee of BNP/BANP over MAUP/MPANP weakens the central hypethesis of the paper.
taking the meta-training
2. The comparisans appear to be against relatively old versions of NPs. wonder haw the proposed method compares against nore recent versions of NPs than WNPs (2018) and INPs (2020), for instance Evidential Turing Processes (2822).
3. I find that the adaptation of the NPMP idea to CNP bit di dilutes the msin ressage the paper. It is after all heavy pipeline with neny components.
great that the paper points out the Linitations of the presented nethod, but would be even better if it also gave an educated guess on which properties of the method ceuse then.
Figure 10: A sample case of WEAKNESS
LLM- LLM--base based Equation Syuthests
LLM M--aaedd Equation Fiiteingg
Model Prediction
she Taak:
are complete
poper. the
nee equation in NLP paper, whill thi equation meskod. Your task to idently comedinees the glvnn candidtt oqual
before and after
where equation hela me recovert that ecuation.
det Task
Glea me the tates sourcs cods of the ii mined the equation
Only give me squation, avokd any other lanatione.
Requitemens::
Leaining ring and NUP shoul help me iyuntlly l't equalion has covious
regpeding
2Mt Context Before:
Vinnngg
eee Contect fefore:
squation fee sigiikannt flawos audh Istuoa ploose mork os Whong
The conkedt belorw the sspaelion))
3. Peean witer "Comect Whony: Ancid other seplaradionee
AAM Contect Alter:
Equition:
equation)
The contnot after the equatiion)
dequelior
Equator::
Equation
Ask Your Arsnnet
degasdorg
Lof of the ground suth scuation)
MM Your Anower
Figure 11: The prompts used in EQINFER including both data collection and model prediction.
22

---

AAAR-1.0: Assessing AT's Potential to Assist Research
LL-i-based Leaking Seatence Deletion
Model Prediction (Experimet Dedign)
Model Prodiction (Mediration Explamation)
N.P ceeki seciaree tegg, sttthec and Intnsduckin)) hewng aome bosic in to rescurch,
loppiiment
You ane parbally glven an Mi. paper inn atwol, inch ding some useful sectinn ja o, heurtec and (nsroduction) heving somn ieroduation research paper, "anpurirant" sections
nehil you gleen experimant that exgermentts this
with a int of the sepertments from the paper help me dedii whither thes
diccuaea any segeemmnna
Phoxee me carvillly
by underabenc mot medivlloone the pposseniinortttuc?
imagine the sopartmment tuat you crested you sepdain
Let' cay, Fone sentence inclues slues for coming up with any
noneeiennoss of this paper fro others?
axpertmnnt
sartence loaking atheruies,
ar accemple experimant
autiors
nion lesk centence".
papec what esperiments you tave so condust to prove your research? Namelyy you have to "reetoerr the espartmerss proviting
you swamples)
repertmntt where
super
commapanndnng
Here excmmple:
Powe chot ewarpples)
yoow shhtt cosmmplss))
The expermmntt lo1)
Now: help me look the folloing paper:
Here barget MLL papprr (perkel contart):
Berlonce:
(Th confoct Input]]
(The Input]
The s
3bd Expor inno
Now besed paper, give experiment only give other
deciaon atherr cr
w thout any sogra aepaaratmend
Experimnt
Please longth un the Thaparmman Laf: the iame of the lwo lirbs corneagaone and one lo-ooo Only give without any other
explanation should
e*e Exptaration Lse
Figure 12: The prompts used in EXPDESIGN including both data collection and model prediction.
Model Prediction (Weaknesses)
You are given an NLP paper, along with its figure illustrations. Imagine you machine learning expert with rich research experience. Please carefully review this paper and identify the weaknesses of this research.
Here is the paper (it might in partial content):
...
The context input.
Now, based on the provided context, give me list weaknesses of this research paper (such as '1. XXXIn2. XXX', one point per line).
Note that if the given context irrelevant to research, such it talking about 'acknowledgement, just generate 'No research content'.
Please either give me the weakness list of this research paper generate 'No research content' to clarify this is not research paper, without any other words.
### Your Answer:
Figure 13: The prompts used in WEAKNESS
23


Reviews:
[{"review_id": "BShBnTaafE", "reviewer": "Reviewer_qTbh", "summary": "This paper aims to measure the capability of Large Language Models (LLMs) in research-relevant tasks. Specifically, those tasks include 1) Equation Inference, which measures whether the equation is relevant to the given context of the paper, 2) Experiment Design, which measures whether the experimental designs generated by LLMs align with the designs generated by humans, and 3) Paper Weakness, which measures whether the weaknesses of the paper identified by LLMs align with humans. Through extensive experiments, this paper shows that even state-of-the-art LLMs are not sufficient for those advanced research-relevant tasks, and points out that there is room for improvement.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your detailed and comprehensive review. We're pleased you found our evaluation criterion reasonable and the literature topic important. As shown below, we address your main concerns one by one:\n\n---\n> Q1:  Unclear evaluation setup for the Equation Inference.\n\nWe apologize for the confusion. Although `EQINFER` was originally designed for a MCQA setting (which explains the 1:3 positive-to-negative distribution), we ultimately adopted a **binary inference setup**, as shown in Figure 1 / Footnote 1. This change was made because of two reasons: \n- i) We found that the MCQA setup contains shortcuts --- LLMs may focus on superficial character differences among the options rather than reasoning through the paper context. In contrast, binary inference requires LLMs to evaluate the correctness of **each option separately**, eliminating any straightforward shortcuts. \n- ii) In real-world equation reviewing during paper submissions, the task also involves determining whether the author-written equation is valid based on the context. We are making our 'AI assists researcher' **setting more realistic**.\n\nIf you're interested, you can find the MCQA setup performances at this **[link](https://anonymous.4open.science/r/ICML2025-rebuttal-A9BF/EQINFER_MCQA_performance.png)**. As shown, LLMs generally achieve higher scores due to the MCQA shortcut. We apologize again for the misleading MCQA prompt template in the appendix and will update it in our next manuscript.\n\n---\n\n> Q2. The reliability of the employed annotators.\n\nThank you for pointing out this critical argument. We will include the following recruitment details in our next version:\n- **Recruiting**: we posted an online recruiting form with strict qualification requirements: i) more than 4 years of AI research experience; ii) more than 5 publications, with at least one first-authored publication in leading AI venues; iii) reviewed 10+ peer publications. The form was then shared through social media.\n- **Annotator Profile**: over 10 annotators were selected, all from renowned academic institutions with strong ML/AI research backgrounds (**not just five students**), including some professors.\n- **Domains**: we aimed to include annotators with diverse AI expertise, including NLP, CV, and ML theory.\n- **Reward**: given the challenge of collecting expertise-level data, we offer a high payment of \\\\$70 per hour to all annotators. Additionally, they receive a \\\\$5 bonus for each low-quality sample identified or valid response made during peer discussions.\n\nWe will follow your suggestion to tone down the term 'senior researcher' in the next version. It is important to emphasize that we have made every effort to ensure all annotators have a strong background and are fully engaged in rigorous data collection and examination, which was also acknowledged by Reviewer `WuHG`.\n\n---\n\n> Q3. Performance of adding few-shot examples.\n\nThis is a good suggestion. Here, we provide the performance of adopting the few-shot examples.\n\n| Model         | 0-shot | 2-shot |\n|---------------|--------|--------|\n| Mistral-7B    | 28.45  | 30.45  |\n| Qwen-72B      | 31.22  | 33.09  |\n| GPT-4o        | 40.35  | 40.61  |\n| o1            | 46.35  | 46.28  |\n\nThe above table illustrates the results of `EQINFER`, we found that adopting few-shot examples only improves the F1 score of smaller open-source LLMs. We believe the examples mainly serve as format guidance for classification in this case, while the reasoning ability of LLMs plays a more critical role in EQINFER (see our Q1 discussion with Reviewer `xpyH`).\n\n| Model       | 0-shot  | 2-shot  |\n|-------------|-----|-----|\n| Mistral-7B  |1.17 |0.89 |\n| Qwen-72B    |1.21 |0.94 |\n| GPT-4o      |5.95 |5.02 |\n| o1          |5.63 |4.78 |\n\nThe above table illustrates the results of `WEAKNESS`, we found that few-shot examples even negatively impact the ITF-IDF scores of various LLMs. Our observation suggests that adding examples restricts LLMs' creativity in generating novel weaknesses. For instance, if we include 'lack of novelty' as an example weakness, LLMs tend to repeat this weakness across different papers.\n\nFor `EXPDESIGN`, due to the list format generation, this is the only task where we observed significant and consistent performance improvement after adopting few-shot examples (which we’ve already used in our paper, as seen in Figure 12). \n\n---\n\n> Q4. Missing references.\n\nThanks for suggesting highly relevant references. We will include a discussion on them in our next version. Additionally, we have conducted further baseline experiments to make our evaluation more comprehensive (please see our Q1 discussion with Reviewer `aPKE`).\n\n---", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are mostly reasonable. However, one critical concern about the overall quality and reliability of the proposed dataset collection process and evaluation setup is that the authors seem to be annotating data with five (or a few) PhD students. While the authors claim that they are senior researchers, I view them as still students, and it may be beneficial to check the quality of the collected data and the annotated experimental results with more seasoned researchers. Also, according to this, I think the authors may need to tone down their claim of performing annotations with senior researchers. Lastly, in addition to them, it is questionable how they were recruited, how much they were compensated, and how diverse they are across domains. Providing this information seems conventional for benchmark work.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The evaluation setup for the Equation Inference is not clear. For the given context of the paper, there is one positive equation and three negative equations. As described in Figure 11 (the prompt template), the authors provide those four options as well as the given context, and prompt the model to predict one that is the relevant equation for the given context. If so (i.e., if the setting is the multiple-choice question answering), it is not clear how to formulate the \"All-Positive baseline\" that predicts all equations as positive. Also, as shown in the results of Table 1, most LLMs tend to predict the equation as positive, and there are no substantial improvements over the \"All-Positive baseline\". In this regard, I am wondering whether the LLMs still do not show substantial performance improvement over the \"All-Positive baseline\" if the authors use the multiple-choice question-answering setup.", "relation_to_broader_scientific_literature": "There is a growing body of literature on using AI for science (to accelerate it), and this paper is relevant to this topic, which is very important and timely.", "essential_references_not_discussed": "For the proposed Experiment Design task, indeed there are few recent studies that evaluate the capability of LLMs in generating experiment designs [A, B]. The authors may cite them, and potentially include their approaches in their benchmark evaluation.\n\n[A] ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models\n\n[B] Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents", "other_strengths_and_weaknesses": "Please see my previous comments.", "other_comments_or_suggestions": "It would be interesting to see the performance where LLMs are prompted with the few-shot examples to perform the given task. The LLMs might not be familiar with the given tasks (as they are a bit different from a typical suite of evaluation benchmarks); however, by learning from examples in-context, they can capture the core principle of the given task and then achieve significantly higher performance.", "questions_for_authors": "I feel like this paper is borderline, and I would like to increase my score if the authors would address all my comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEGLnUbIuI", "reviewer": "Reviewer_WuHG", "summary": "The paper introduces AAAR-1.0, a benchmark for measuring the ability of LLMs to perform 3 key research tasks: mathematical equation understanding, designing experiments, and identifying weaknesses in paper submissions.\n\nThe authors curate datasets for each of their chosen research tasks by scraping public research papers and reviews, transforming it, and using expert human annotation to filter and assure quality.\n\nFinally, they evaluate both open and closed-source LLMs on their constructed benchmark and report results along with ablations and analyses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing our manuscript! We're glad you found our dataset interesting, the proposed evaluation metrics reasonable, and the experimental results useful. Below, we address your concerns in detail.\n\n---\n\n>Q1. Problem for setting the 'ground truth' for `EXPDESIGN` and `WEAKNESS`.\n\nThis is a great point. While research is inherently 'open-ended,' human annotations still meet the regular acceptance standards of research — **not a gold standard, but at least a reasonable one**. Therefore, despite its limitations, using human annotations as ground truth remains a practical approach for benchmark construction, as evidenced by recent works that continue to adopt this methodology [1][2].\n\nIn this work, to mitigate potential bias from setting human annotations as the ground truth, we establish the evaluation framework that integrates both **automatic metrics** and **human assessment**. For example, in `EXPDESIGN`:\n- First, we use an automatic metric to compute the En-F1 score (Table 2) and identify 'negative' predictions.\n- Then, to *quantify false negative judgments*, experts manually assess the true correctness of these ‘negative’ predictions (Table 3).\n\nTo our knowledge, combining automatic metrics with further human assessment represents the 'best' current practice for evaluating research outputs.\n\n---\n\n>Q2. Analysis for the false positive / false negative cases w.r.t. the 'ground truth'.\n\nThanks for your helpful suggestion. We have indeed made some analyses in our paper. Specifically, as shown in Table 3, for those model-predicted experiment ideas that do not match the human ground truth, we asked the human experts to evaluate manually. \n\nWe found a few model-generated experiment ideas that deviate from the ground truth but are deemed reasonable in manual checks (implying **some false negatives**). Meanwhile, our manual examination confirms that our rigorous peer review process ensures the ground truth's correctness and objectivity (indicating **no notable false positives**).\n\nWe hope these results can inspire future work on fairly evaluating research outputs without reliance on ground truth while maintaining efficiency and reproducibility.\n\n---\n\n>Q3. Concerns on data contamination.\n\nThis is an important point to discuss. First, we believe that `EQINFER` and `EXPDESIGN` are less likely to be affected by data contamination, as the ground truth outputs for both tasks have been **reformulated** or **rewritten** from the original source. Output rewriting or distribution perturbation is a widely adopted method to mitigate data contamination [3][4].\n\nHowever, we acknowledge the potential data leakage issue in the `WEAKNESS` task, as all outputs are directly taken from OpenReview (though we believe this is a common challenge faced by most current benchmark datasets [5]). We maintain that our experimental results still offer valuable insights and can **serve as an upper bound for certain LLMs**, especially if they were pretrained on papers from OpenReview.\n\nNotably, this work introduced AAAR-1.0. Our ongoing efforts involve collecting confidential data that no LLM has touched; we will include that as blind test sets in AAAR-2.0, our next version.\n\n---\n\n>Q4. Concerns on unifying input length.\n\nSince not all LLMs support long-context reasoning and different models have varying maximum context sizes, we standardized the input length for a fair comparison. We agree that it is crucial to ensure all input information is provided to the LLMs.\n\nTo address this, we conducted extensive context length analyses in our manuscript (pls refer to Appendix D1 --- 'Input Context Scaling Investigation'). For example, Figure 4 in the Appendix demonstrates that not all long-context LLMs benefit from the full input information.\n\n---\n\n### References:\n\n[1]. [ResearchTown: Simulator of Human Research Community.](https://arxiv.org/abs/2412.17767) (*arxiv 2024*)\n\n[2]. [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing.](https://arxiv.org/pdf/2406.16253) (*EMNLP 2024*)\n\n[3]. [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery.](https://arxiv.org/abs/2410.05080) (*ICLR 2025*)\n\n[4]. [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code.](https://arxiv.org/abs/2311.09835) (*ICLR 2025*)\n\n[5]. [CycleResearcher: Improving Automated Research via Automated Review.](https://arxiv.org/abs/2411.00816) (*ICLR 2025*)", "claims_and_evidence": "The main contribution of the paper is their dataset and benchmark, which is well-supported by construction.\n\nOtherwise, there are no significant claims besides their main results which are directly supported by their experiments. I leave specific criticisms of the benchmark and experiments to later sections.", "methods_and_evaluation_criteria": "The overall benchmark is built around a good choice of 3 main tasks, and the authors have done a great job creating interesting datasets for all 3.\n- EquationInference - no complaints here, this appears to be a solid and useful dataset for mathematical reasoning with straightforward metrics.\n- ExperimentDesign - The dataset is well-constructed. Precision/Recall are suitable metrics. One issue is that you’re making an assumption that the ground truth experiments are exactly what are needed - nothing more, nothing less. If a paper includes an extraneous experiment, or misses a useful experiment that the LLM thinks of, the LLM gets penalized unfairly.\n- PaperWeakness - Dataset is well-constructed, and metrics are fine. However, this has the same issue of unfairly penalizing LLMs as I mentioned for ExperimentDesign.\n\nGiven the subjective “correctness” of real-world papers/reviews, treating these datasets as ground truths to exactly match, it’s likely that this benchmark is unfair / impossible in some ways. It would be good to have more careful analysis / discussion of what the extent of false positives / false negatives might be in the dataset.", "theoretical_claims": "No key theoretical claims.", "experimental_designs_or_analyses": "- The choice of unifying context lengths is an unusual one. This arbitrarily limits the performance of models, and adds a lot of complexity to the paper’s discussion and results\n  - The paper dedicates many experiments to this topic, which would be relevant for a long-context-focused benchmark, but is only a practical consideration for the AI-research benchmark. (The results are surprising in some cases but it feels mostly like performing hyperparameter sweeps which minimally change the overall conclusions)\n  - In my opinion, the benchmark should always directly provide all the necessary information - if models can’t handle it, the models may do worse, but future models will soon have more context length given the quick rate of progress in AI.", "relation_to_broader_scientific_literature": "This work contributes valuable benchmarks and metrics for understanding the ability of LLMs to perform 3 key research tasks that are not currently well-covered by existing datasets. This will be a valuable dataset for future work to build upon. The Related Work in Appendix A does a good job laying out the relevant literature.", "essential_references_not_discussed": "None that come to mind.", "other_strengths_and_weaknesses": "Strengths\n- All 3 tasks give really interesting datasets! Even leaving the main benchmark aside, I think they are all really interesting datasets in their own right.\n- Very valuable data collection and annotation of papers with expensive experts. Multi-stage annotation with multi-annotation and peer review shows a great care for data quality.\n- Interesting results! I was surprised to see the overall weak performance of models on this benchmark, suggesting that it might reveal useful new information about our understanding of models' capabilities.\n\nWeaknesses\n- The main concern I have is of data contamination. All the datasets and answers are scraped from publicly available papers on the internet. Before long, models will train on the data and then the benchmark results will no longer be trustworthy. This doesn't seem to be an issue now given the low results, but this will likely limit the lifespan of the benchmark.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "jfuUENOkiv", "reviewer": "Reviewer_aPKE", "summary": "This paper introduces AAAR-1.0, a novel benchmark designed to evaluate the ability of Large Language Models (LLMs) to assist researchers in expert-level tasks. The benchmark comprises three distinct tasks: EquationInference (EQINFER), which assesses the LLM's ability to validate the correctness of equations within a given context; ExperimentDesign (EXPDESIGN), which evaluates the LLM's capacity to design reliable experiments based on a research idea; and PaperWeakness (PAPERWEAKNESS), which tests the LLM's ability to identify weaknesses in research paper drafts. The authors meticulously curated datasets for each task, employing expert annotators to ensure high-quality data. The evaluation process involved a range of both open-source and closed-source LLMs, and the results were analyzed using a combination of quantitative metrics and qualitative assessments. The study found that while LLMs demonstrate some capability in these tasks, their performance is often only slightly above chance, particularly in the EquationInference task, highlighting limitations in their practical utility for advanced research assistance. The paper's core contribution lies in the creation of a benchmark that targets complex, reasoning-intensive tasks that are highly relevant to the research process, moving beyond superficial applications of LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your comments are very much appreciated! We took your comments carefully and tried to address them one by one.\n\n---\n\n>Q1. The critical references that were missed (i.e., CycleResearch and DeepReview).\n\nThanks for highlighting these important concurrent works. Specifically, we ran CycleResearch on our benchmark during the rebuttal.\n\n| Method         | S-F1 | ITF-IDF |\n|----------------|------|--------|\n| Llama3.1-70B     |42.78 |2.60     |\n| GPT-4o         |**47.73** |**5.95**    |\n| AI-SCI         |45.05 |2.23    |\n| CycleReview-70B|46.68 |2.65    |\n\nThe table above presents the `WEAKNESS` results, where the 70B CycleReview model (based on LLaMA-3.1) achieves an S-F1 score nearly comparable to GPT-4o, highlighting the benefits of post-training for open-source LLMs on this task. However, CycleResearch's ITF-IDF score (the proposed diversity metric) remains similar to LLaMA-3.1 due to a lack of specificity—a common error pattern of LLMs discussed in our paper.\n\n| Method          | En-F1 |\n|-----------------|-------|\n| Llama3.1-70B      | 22.92 |\n| GPT-4o          | **25.03** |\n| CycleResearch-72B| 21.16 |\n\nThe table above presents the `EXPDESIGN` results. In our view, CycleResearch may not be a suitable baseline for this task, as it is a policy model designed for whole-paper writing rather than specializing in experiment design. Consequently, applying it to our task requires modifying its original system prompt and task objective, which could make the comparison unfair.\n\nWe will provide further observations on CycleResearch in our next manuscript.\n\n---\n\n>Q2. The definition of “assisting research”.\n\nIn terms of 'AI for research', our benchmark differs from existing works in two key aspects.\n- i) **The scope of 'research'**: Since research activities are broad and diverse, we focus on domain-specific, expertise-demanding, and reasoning-intensive tasks that highlight the **irreplaceability of researchers**. For example, writing experimental code is a reasoning-light task, often done by students, whereas determining 'what experiments are necessary' to support a paper’s primary claim is an expertise-demanding task, typically decided by senior advisors. The latter is clearly more challenging and demonstrates that AI models cannot easily replace senior researchers.\n- ii) **'Assisting researchers' rather than 'replacing researchers'**: For high-level research tasks, our benchmark primarily serves an educational purpose — LLMs assist junior researchers by offering imperfect yet insightful ideas rather than governing the entire research process [1]. Relying on LLMs to oversee research and replace human effort compromises academic integrity. For example, we can use LLMs to suggest weaknesses as feedback to help us refine our own manuscripts, rather than directly using model-generated comments for peer review. \n\n---\n\n>Q3. Reason for choosing the three tasks and their connection with the real-world scenario.\n\nAs addressed in Q2, our benchmark focuses on expertise-demanding research tasks that highlight the irreplaceability of researchers. Though more tasks could be included, we prioritize those that are **widely underestimated** in existing works. For example, while writing a paper review is well-studied, identifying a paper’s weaknesses is significantly more challenging than writing a paper summary/strengths [2][3].\n\n`EQINFER` relates to scenarios where LLMs assist in double-checking the correctness of our own equation writing. `EXPDESIGN` mirrors a PhD student seeking a professor’s advice before writing experimental code. `WEAKNESS` represents using LLM feedback to refine our entire research project. Each corresponds to a real-world scenario and strictly aligns with our 'assisting researcher rather than replacing researcher' perspective.\n\n---\n\n>Q4. The limitation of the benchmark.\n\nThanks for your suggestion. We agree that discussing limitations would benefit readers and future research. In fact, we initially included a limitation discussion section in our manuscript but removed it to comply with the ICML submission format. Due to the rebuttal word limit, we have provided that section in **[this link](https://anonymous.4open.science/r/ICML2025-rebuttal-A9BF/limitation%20section.png)** and will aim to reintegrate it in future versions.\n\n---\n\n>Q5. Details about random baseline and how we ensure the data collection is not biased.\n\nFor the PaperWeakness task (Table 5), we reported only human performance and did not include a 'random baseline'. Please refer to the Q2 of Reviewer `qTbh` for more details on how we ensured unbiased data collection.\n\n---\n\n### References:\n\n[1]. [Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration.](https://arxiv.org/abs/2412.15701) (*arxiv 2025*)\n\n[2]. [Can large language models provide useful feedback on research papers? A large-scale empirical analysis.]() (*NEJM AI 2024*)\n\n[3]. [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing.]() (*EMNLP 2024*)", "claims_and_evidence": "1. The paper lacks a clear and operational definition of what it means for an LLM to \"assist researchers.\" This ambiguity makes it difficult to interpret the results and understand the practical implications of the benchmark. While the motivation section outlines the challenges researchers face, and the introduction specifies the three tasks, a broader definition is missing. The notion of assistance is too broad and could encompass a wide range of activities, from generating research ideas to writing code, and the paper does not specify which of these are targeted. This lack of clarity makes it challenging to assess the real-world utility of the benchmark.\n\n2. The paper does not provide a clear explanation of how the performance of LLMs on the benchmark translates to their ability to assist researchers in practical scenarios. The connection between the benchmark tasks and real-world research assistance is not well-established, leaving the reader to speculate about the practical utility of the results. For example, it is unclear how well performance on the EquationInference task correlates with the ability of an LLM to help a researcher identify errors in their own equations. While the paper attempts to connect the tasks to real-world scenarios, the explanation is not always explicit and lacks strong supporting evidence or citations.", "methods_and_evaluation_criteria": "Convincing enough.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "It does not explicitly discuss the limitations of the proposed benchmark. The benchmark not cover all aspects of research assistance, and the tasks biased towards certain types of research or domains. The tasks may also be too narrow, focusing on specific sub-tasks rather than the broader context of research.", "relation_to_broader_scientific_literature": "See in Essential References Not Discussed.", "essential_references_not_discussed": "I believe the authors have overlooked discussing a very closely related work: CycleResearcher: Improving Automated Research via Automated Review. In fact, when we compare the data construction of the two papers, it's readily apparent that both AAAR and CycleResearcher's collected Review-5K and Research-14K are oriented towards assisting researchers. For AAAR's ExperimentDesign task, it is actually similar to the \"Idea\" and \"Experiment\" sections of Research-14K (and I think they are very similar). For AAAR's PaperWeakness task, it is almost identical to the content of the Review-5K dataset. I understand the authors of AAAR want to focus on benchmarking and evaluation, but the lack of discussion of CycleResearcher, and the absence of using CycleResearcher and CycleReviewer as baseline methods, is difficult to accept.", "other_strengths_and_weaknesses": "In my opinion, the true brilliance of this paper absolutely shines through in its introduction of the AAAR-1.0 benchmark. This isn't just another benchmark; it's genuinely novel and undeniably relevant to the booming interest in leveraging LLMs for research support. What I particularly appreciate is how the benchmark tackles expert-level tasks – equation validation, experiment design, and paper weakness identification – this is a remarkably significant contribution. It's refreshing to see a benchmark that moves beyond the shallow, commonplace uses of LLMs and dives into something truly meaningful. Frankly, the authors have brilliantly identified a real gap in existing benchmarks, and they've masterfully filled it with a resource that is precisely what the research community desperately needs.", "other_comments_or_suggestions": "First and foremost, please, please take the time to clearly define what you mean by \"assisting researchers.\" This isn't just a minor detail; it's fundamental. Make this definition specific and operational – something that truly guides the scope of your benchmark. For instance, you could explicitly state that \"assistance\" encompasses tasks like identifying errors in equations, suggesting relevant experimental designs, or pinpointing weaknesses in research papers. Ground this definition in the real needs of researchers – what do we actually struggle with? – and let that guide the very design of your benchmark tasks.\n\nIt's crucial that you provide a clear and compelling rationale for why you chose these specific tasks. Don't just assume their relevance is obvious. Walk us through your thinking. Discuss the common hurdles researchers face, and explain precisely how your benchmark tasks directly address these challenges. Think about the broader research landscape. Consider how this benchmark could be used across different research domains, and honestly discuss its limitations in those diverse contexts. Transparency here is key.\n\nYou need explicitly and honestly discuss the limitations of your proposed benchmark. Don't shy away from this. Analyze potential biases in the tasks. Acknowledge domains not covered. Be upfront about potential biases – is it skewed towards certain research types or domains? Does it truly capture all aspects of research assistance? Discuss the very real possibility of \"gaming\" – could LLMs be specifically trained to excel on your benchmark without truly understanding the underlying research tasks? This discussion needs to be honest, transparent, and insightful.", "questions_for_authors": "What is the random guess baseline for the PaperWeakness task? how to set?\n\nHow do you ensure that the experts involved in the data collection and evaluation process are not biased towards certain LLMs or research domains?\n\n---\n\nDo you know \"DeepReview\"? I believe this work may be helpful to you.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b7A07KCqLD", "reviewer": "Reviewer_xpyH", "summary": "This paper introduces AAAR-1.0, a benchmark designed to assess the capabilities of Large Language Models (LLMs) in assisting with research-specific tasks. While most existing benchmarks focus on general-purpose tasks, AAAR-1.0 specifically targets high-level academic reasoning and research assistance, addressing three core challenges:\n\nEquation Inference (EQINFER): Evaluates an LLM’s ability to verify the correctness of equations within research papers, a critical aspect of scientific validation.\nExperiment Design (EXPDESIGN): Tests whether LLMs can propose well-structured experimental plans aligned with research objectives.\nPaper Weakness Identification (WEAKNESS): Assesses an LLM’s capacity to critically analyze research methodologies and identify key weaknesses.\nThe dataset is constructed using expert-annotated research examples, ensuring that evaluation aligns with real-world academic challenges. The authors conduct extensive comparative evaluations across a range of LLMs, including GPT-4o, Claude 3.5, Gemini 1.5, Mistral, and Mixtral, revealing:\n\nLLMs struggle significantly with equation reasoning, with performance barely exceeding random baselines.\nWhile LLMs can generate diverse experimental plans, these plans are often misaligned or infeasible for real-world research applications.\nIn research critique tasks, LLMs can identify broad weaknesses but frequently fail to provide specific, actionable insights comparable to expert reviews.\nThe paper positions AAAR-1.0 as a necessary step toward evaluating and improving AI’s role in research, highlighting the current limitations and potential future directions for AI-assisted research workflows.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed review and comments. Below, we provide our comprehensive responses to your questions.\n\n---\n\n >Q1. Why LLMs fail on `EQINFER` task & provide the failure patterns discussion.\n\n`EQINFER` leverages a challenging binary inference setting, where LLMs are forced to examine each option separately rather than relying on superficial shortcuts from multiple-choice QA (please refer to the Q1 of Reviewer `qTbh` for more detailed discussion). According to our observations, most LLMs tend to predict any given equation as correct (**a common error pattern for most LLMs**) without performing in-depth reasoning over the paper context. Thus, we assume that the inferior performances of most LLMs result from their **limited long-context reasoning capacity**.\n\nTo verify our assumption, we ran the OpenAI o1 model with varying levels of reasoning effort.\n\n|Model   |F1   |\n|--------|-----|\n|o1-low     |42.98|\n|o1-medium  |46.35|\n|o1-high    |**47.12**|\n\nThe o1’s performance consistently improves as we increase the reasoning effort. Although this is a simple empirical verification, we believe it highlights the importance of reasoning capacity for this task, particularly for open-source LLMs.\n\n---\n\n>Q2. Heavy reliance on automatic metrics for evaluation.\n\nThanks for your suggestions. We proposed task-specific automatic metrics for all the tasks in AAAR to ensure that the public can easily and efficiently reproduce the experimental results reported in our paper. \n\nAt the same time, we agree that human judgment is also valuable. As shown in Tables 3 and 4, we conducted a small-scale human evaluation for the model-generated experiment ideas, where we found that despite a few unexpected evaluation results (i.e., false negatives), the results from the automatic metrics generally align well with human judgments, especially when compared with conventional generation metrics like ROUGE. This suggests the reliability of the proposed automatic metrics.\n\n---\n\n>Q3. Discussion on computational efficiency.\n\nApologies for missing this detail. When running the open-source LLMs on our local machine, we used [vllm](https://docs.vllm.ai/) to accelerate computational efficiency. Given 4 NVIDIA A100 GPUs for LLM inference, the largest model we utilized in our experiment, Qwen-72B, took approximately 1.25 hours, 0.4 hours, and 1.75 hours for `EQINFER`, `EXPDESIGN`, and `WEAKNESS`, respectively. All the running hyperparameters, such as the maximum model length, can be found in our paper.\n\nIn our next version, we will include more details about the computational costs of various LLMs on AAAR.\n\n---\n\n>Q4. Would AAAR generalize to other fields?\n\nYes. Our proposed data collection method can ideally be generalized to other disciplines, with AAAR serving as a representative benchmark in the AI/ML field. However, the main constraint is still the requirement for domain experts, as recruiting a reliable and large annotation team is extremely expensive for this kind of research benchmark (see the Q2 discussion of Reviewer `qTbh`).\n\n---\n\n>Q5. Could incorporating multi-turn dialogue improve the performance of LLMs in research critique tasks?\n\nWe assume so. To our knowledge, multi-turn dialogue context can be seen as a structured reasoning path, potentially benefiting LLMs in complex tasks (based on our experience). Meanwhile, some studies have explored dialogue-based collaboration between humans and LLMs or among different LLMs, showing that intra-perspective message sharing can enhance performance in reasoning-intensive tasks [1][2].\n\n---\n\n### References:\n\n[1]. [Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration.](https://arxiv.org/abs/2412.15701) (*arxiv 2025*)\n\n[2]. [Chain of Agents: Large Language Models Collaborating on Long-Context Tasks.](https://arxiv.org/abs/2406.02818) (*NeurIPS 2024*)", "claims_and_evidence": "Claim 1: AAAR-1.0 is the first benchmark specifically designed to evaluate LLMs in research-oriented tasks.\n\nWhile prior work has explored AI-driven research assistance, most datasets focus on code generation, summarization, or retrieval, rather than high-level research tasks.\nThe authors present AAAR-1.0 as a unique benchmark addressing critical reasoning tasks encountered by academic researchers.\nClaim 2: LLMs perform poorly in verifying equations and mathematical reasoning.\n\nExperimental results from the Equation Inference (EQINFER) task show that even the best-performing models (GPT-4o, Claude 3.5) achieve only 46% F1-score, indicating a fundamental gap in symbolic reasoning capabilities.\nOpen-source models like Mistral and Mixtral perform close to random guessing, reinforcing the difficulty of formal equation verification for LLMs.\nClaim 3: LLM-generated experimental designs are often misaligned with real-world research constraints.\n\nIn the Experiment Design (EXPDESIGN) task, LLMs produce syntactically correct but practically infeasible experiment proposals.\nHuman evaluations of 15 model-generated plans show that many experiments were either unnecessary, redundant, or impossible to implement with available resources.\nClaim 4: LLMs provide general but shallow critiques in peer review tasks.\n\nIn the Paper Weakness Identification (WEAKNESS) task, LLM-generated critiques highlight surface-level issues but often lack depth and specificity compared to human reviewers.\nUsing an ITF-IDF-based informativeness metric, the paper shows that human-written critiques outperform LLM-generated ones in specificity and actionable feedback.", "methods_and_evaluation_criteria": "The evaluation framework is well-structured, incorporating:\n\nThree benchmark tasks covering distinct aspects of AI-assisted research.\nComparisons across multiple models, including both open-source (Mistral, Mixtral, Qwen, LLaMA) and closed-source (GPT-4o, Claude, Gemini) LLMs.\nTask-specific metrics:\nF1 Score for equation inference.\nSemantic Precision, Recall, and ITF-IDF Informativeness for weakness identification.\nHuman evaluation for experiment design.", "theoretical_claims": "The paper does not propose new theoretical models but provides empirical insights into LLM limitations in symbolic reasoning, research critique, and experiment design.\nA discussion on why LLMs fail in equation inference (e.g., limitations in token-level vs. structural mathematical reasoning) would strengthen the study.\nA deeper exploration of LLMs’ long-context reasoning capabilities in academic texts could provide additional insights.", "experimental_designs_or_analyses": "The experiments are controlled and reproducible, ensuring fair model comparisons.\nAblation studies explore the impact of context length, multimodal inputs (figures, tables), and response verification methods.\nThe finding that multimodal inputs (figures, tables) do not significantly improve performance is valuable, suggesting that LLMs struggle with processing visual research data effectively.", "relation_to_broader_scientific_literature": "This work contributes to ongoing research in AI for research automation, building on:\n\nAI-powered academic assistants (Lu et al., 2024; Si et al., 2024).\nMathematical reasoning in LLMs (Song et al., 2023).\nAutomated peer review tools (Gao et al., 2024; Liang et al., 2024).", "essential_references_not_discussed": "Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models", "other_strengths_and_weaknesses": "Strengths:\n\nWell-structured benchmark tailored to research tasks.\nExpert-validated dataset, ensuring high annotation quality.\nProvides critical insights into LLMs' limitations in academic reasoning.\nWeaknesses:\n\nLimited failure case analysis.\nNo discussion of computational efficiency.\nEvaluation relies heavily on automated metrics instead of human judgments.", "other_comments_or_suggestions": "No", "questions_for_authors": "What are the primary failure patterns in equation inference? Are LLMs failing due to lack of symbolic reasoning or misunderstanding notation?\nWould AAAR-1.0 generalize to non-STEM fields like law, medicine, or social sciences?\nCould incorporating multi-turn dialogue improve the performance of LLMs in research critique tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces AAAR-1.0, a benchmark designed to evaluate LLMs' capabilities in assisting with research-oriented tasks across three dimensions: validating equation correctness, designing experiments, and identifying paper weaknesses. All four reviewers recommended acceptance, appreciating the paper's novel contribution in addressing expertise-intensive research tasks, high-quality dataset with rigorous expert annotation, comprehensive evaluation across various LLMs, and valuable insights into current limitations. During the rebuttal, the authors effectively addressed concerns about the evaluation setup, annotator reliability, baseline comparisons, and provided a clearer definition of "assisting research." The authors' responses were thorough and satisfactory, leading some reviewers to increase their ratings. Given the paper's strong contributions, thorough evaluation, and comprehensive responses to reviewer feedback, I recommend accepting this paper for ICML 2025 as it provides valuable insights into LLMs' capabilities in supporting research activities.
Decision: Accept (poster)