Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review
Authors: Kai Lu
Shixiong Xu
Jinqiu Li
Kun Ding
Gaofeng Meng
Venue: ICML 2025 poster (2025)
Keywords: Peer Review Automation; Multimodal Review System; Multi-agent System; Shared Memory Pool;
Primary area: applications->language_speech_and_dialog
TL;DR: We present Agent Reviewers, an LLM-based multi-agent system for peer review, enhanced by multimodal feedback and a shared memory of prior papers.

Abstract: Feedback from peer review is essential to improve the quality of scientific articles. However, at present, many manuscripts do not receive sufficient external feedback for refinement before or during submission. Therefore, a system capable of providing detailed and professional feedback is crucial for enhancing research efficiency. In this paper, we have compiled the largest dataset of paper reviews to date by collecting historical open-access papers and their corresponding review comments and standardizing them using LLM. We then developed a multi-agent system that mimics real human review processes, based on LLMs. This system, named Agent Reviewers, includes the innovative introduction of multimodal reviewers to provide feedback on the visual elements of papers. Additionally, a shared memory pool that stores historical papers' metadata is preserved, which supplies reviewer agents with background knowledge from different fields. Our system is evaluated using ICLR 2024 papers and achieves superior performance compared to existing AI-based review systems. Comprehensive ablation studies further demonstrate the effectiveness of each module and agent in this system.

Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review
Kai 1 Shixiong Xu 12 Jinqiu Li 12 Kun Ding
12
Gaofeng Meng
2.3
Abstract
Raw Faper
TEXT Aostroct: This paper proposes Introdaction: Lorge longugge...
Strong ancept
Feedback from peer review is essential to im- prove the quality of scientific articles. However, at present, many manuscripts do not receive suffi- cient external feedback for refinement before or during submission. Therefore, a system capable of providing detailed and professional feedback crucial for enhancing research efficiency. In this paper, we have compiled the largest dataset of pa- per reviews to date by collecting historical open- access papers and their corresponding review com- ments and standardizing them using LLM. We then developed a multi-agent system that mim- ics real human review processes, based on LLMs. This system, named Agent Reviewers, includes the innovative introduction of multimodal review- ers to provide feedback on the visual elements of papers. Additionally, a shared memory pool that stores historical papers' metadata is preserved, which supplies reviewer agents with background knowledge from different fields. Our system is evaluated using ICLR 2024 papers and achieves superior performance compared to existing AI- based review systems. Comprehensive ablation studies further demonstrate the effectiveness of each module and agent in this system, Our code and data are available at https: // / thub. com/ /ww.AAeeeeeerrseeeeeeeeeee.eeeee
8, Accept
Donvverter
Conclusion: paper
Weak reject
Weak acropt
Rayect
Agent Reviewers
1. Strong reject
Thumbrai
HA
S1: Ar interasting
+
Meroory Retriesal using Keywords
Domain- spadiic
S2: Good expression
difference
W2 The cquation
Memory
W3 In line 124-128-
Shared Memory Pool
Figure I. Agent Reviewers: a paper review system comprises domain-specine multimodal agents with shared memory. It lever- ages both paper text and thumbnails for evaluation, retrieving domain-specifee memory from shared memory pool to deliver decisions, scores, and insightful comments.
the growing number of researchers has led to an explosive rise in annual paper submissions. For example, in artificial intelligence, ICLR submissions surged from 490 in 2017 to 11,672 in 2025. However, the quantity and quality of peer reviews have failed to keep pace, leaving many young researchers, especially students, without the high-quality feedback needed to refine their work (Stelmakh et al., 2021; Zhang et al., 2022; Fox et al., 2023). As result, many newcomers find themselves caught in an inefficient arms race, prioritizing submission volume over research quality.
Fortunately, the emergence of Large Language Models (LLMSs) offers a promising solution to this prevalent is- sue. Many researchers have explored the use of LLMs to provide feedback on academic papers. Existing researches typically formulate automated peer review as Supervised Fine-Tuning (SFT) task (Gao et al., 2024) or a zero-shot task based on LLMs (Yuan et al., 2022; Liu & Shah, 2023; Liang et al., 2024). The former relies on the collection and cleaning of real review data, while the latter depends on the adjustment of model prompts.
1. Introduction
In academia, peer review is essential for validating research findings and improving manuscript quality. In recent years,
"Equal contribution 'ttate Key Laboratory of Multimodal Ar- tificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing. China "School of Artificial In- telligence, University of Chinese Academy of Sciences. Beijing, China 'The Centre for Artificial Intelligence and Robotics, HK Institute of Science & Innovation, Chinese Academy of Sciences, Correspondence to: Gaofeng Meng <gfmmng @ iipr.ia.ac.cn>>.
However, review comments generated by these methods often suffer from severe homogenization due to multiple factors. First, the same paper may receive diverse opinions from different reviewers, but a model fine-tuned on such data risks collapsing into uniform review comments. Sec-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Agent Reviewers
ond, current approaches heavily rely on the model's internal domain knowledge, which is often insufficient for generat- ing constructive and distinctive feedback. Lastly, these meth- ods typically assess papers based solely on text, whereas real peer review also considers figures, layout, and other visual elements. The absence of visual modalities further reduces the diversity of review comments (Von Bearnen- squash, 2010: Huang, 2018). As a result, Al-generated ho- mogenized comments could not provide insightful feedback before submission. Moreover, their increasing presence in academic journal and conference reviews significantly diminishes the submission experience for authors.
with state-of-the-art (SOTA) methods, in terms of decision prediction and review quality scoring. Finally, extensive ablation experiments are conducted to validate the effective- ness of each module within the system.
Our contributions are summarized as follows: (1) We con- structed a multi-agent review system, named Agent Review- ers, which consists of shared memory pool for domain- specific reviewers initialization and a multimodal agent to enhance the diversity of the generated comments. (2) We proposed meticulously standardized large-scale paper re- view dataset, Reviewer-STD, along with a benchmark from ICLR 2024. (3) Agent Reviewers significantly outperforms existing methods in terms of paper acceptance prediction and the quality of paper review comments.
Inspired by the aforementioned analysis, we propose Agent Reviewers for automated paper review, as shown in Fig- ure 1. This framework fundamentally emulates real-world peer reviewers' expertise acquisition patterns: By construct- ing multiple reviewers with distinct roles and modalities, it mirrors how actual reviewers leverage Nomain-specifer knowledge related to paper keywords, ultimately generating high-quality, insightful evaluations. The system architecture embodies three core innovations: Multi-agent Interaction (MI), Shared Memory Pool (SMP) designed mimic re- viewers' background knowledge repository, and Multimodal Agent (MA). Specifically, the SMP and domain- specific agent design originate from simulating human reviewers' practice: the Meta-reviewer agent extracts paper keywords to initialize specialized reviewers through SMP, which sys- tematically organizes domain knowledge akin to real review- ers' interests. Concurrently, the MA processes PDF/PNG files to critique visual elements (Von Bearnensquash, 2010; Huang, 2018). These domain-specific agents then conduct multi-perspective assessments, generating preliminary re- views. After iterative discussion and comments refinement, a chair reviewer agent summarizes the entire review process to provide the final review comments and ratings. These review comments will serve as meta-information for the paper and update the shared memory pool.
2. Related work
2.1. AT for Review
Recent advances in AI, particularly LLMs, have created new opportunities for scientific research (Back et al., 2024; Wang et al., 2024b; Huang et al., 2024; Wang et al., 2024c; D' Arry et al., 2024; Wei et al., 2023). Peer review, a critical research component, has gained attention, with several studies lever- aging LLMs for this task. (Yuan al., 2022) explores using NLP models for initial peer peer reviews and discusses evaluation metrics. (Liu & Shah, 2023; Liang et al., 2024) assess the reliability of LLM-generated reviews through large-scale analyses with carefully designed prompts. The Al Scientist framework (Lu et al., 2024) introduces a fully automated sci- entific pipeline, covering idea generation, paper writing, and simulated review process with self-reflection and aggrega- tion. (Du et al., 2024) compares human-written written and LLM- generated reviews at the sentence level, analyzing LLMs' potential as reviewers and meta-reviewers. (Zhou et al., 2024) presents the RR-MCQ dataset for review-revision tasks, evaluating GPT-3.5 and GPT-4 in score prediction and review generation. REVIEWER2 (Gao et al., 2024) proposes a two-stage review generation framework using prompt guidance and fine-tunes LongLoRA on large peer review dataset. SEA (Yu et al., 2024) standardizes reviews into a unified format, fine-tuning Mistral-7B and employ- ing a self-correcting strategy to enhance quality. (Weng et al., 2024) introduces an autonomous research and review framework, where CycleResearcher conducts research while CycleReviewer simulates peer review, providing iterative feedback via reinforcement learning. Unlike prior works focusing on text, (Von Bearnensquash, 2010; Huang, 2018) employ AdaBoost and deep convolutional networks to pre- dict paper acceptance or rejection based on visual features, underscoring the importance of visual information.
To facilitate this study, a dataset with standard reviews is pro- posed in Section 4, termed Reviews-STD. It is derived from Reviewer2 (Gao et al., 2024) and SEA (Yu et al., 2024), in- cluding papers from the International Conference on Learn- ing Representations (ICLR) spanning 2017 to 2024 and the Conference on Neural Information Processing Systems (NeurIPS) covering 2016 to 2024. To enhance consistency and accessibility, we apply the ERNIE- Speeed- 128K model to transform disparate reviews into a unified format, which consists of two lists (strengths and weaknesses) and the de- cision. For the test set, we use 100/300 evenly randomly selected papers from ICLR 2024.
Once the shared memory pool is initialized using Reviewer- STD, we conducte evaluations on the test set, where Agent Reviewers demonstrates superior performance compared
Collectively, these efforts highlight the potential of AI, in- cluding LLM-based and vision-based approaches, assist
2

---

Agent Reviewers
PHASE 1: Domain- Domain-Specific Reviewers Initialization
SMP
Shared Memory Pool
Meta Reviewer
Keyword 1
Keyword 1
Description XXXX
Keyword 2
Memory Retrieval
Prompt
-- Poper 1: summory, decision,
:Related Papers
Template
yyerr strengths, weoknesses.
:
Poper 2.:
Keyword
Keyword 3
Domain-Specific Memory
Reviewers
Keyword
PHASE 2: Visual-enhanced Review Process
Update Memory
Multimodal Reviewer
Comments & Decision
Revised Review 1
Visual Comments
Review
Area Chair
Score: 8
Decision-Accept
Revised
Strengths: [XXX xOX...
:10
Review
Review 2
Weaknessess(xxx..pr xX...
:
Paper info
Pure Text of Paper
Revised Review 3
Summary-30000
10
Converter
Review
(xyyordss[xxxxxxx.
1::M
Discussion Period
Figure 2. Overview of the Agent Reviewers system. In PHASE the meta reviewer extracts keywords from the paper text and retrieves memory from the SMP to initialize domain-specific reviewers. In PHASE 2, multimodal reviewer provides visual comments, and sonain--peeific reviewers integrate these with the paper text initial review and discussion refine their assessments. Finally, the AC consolidates all revised reviews to produce the final comments and decision, updating the SMP with the paper information and review.
and enhance the traditional scientific review process. How- ever, prior research often relies single-agent frameworks and treats textual and visual data in isolation. In our study, we advance this by integrating multi-agent collaboration and multimodal capabilities to leverage both textual and visual information effectively.
scenario simulation focuses on using multi-un multi-agent systems to model real-world scenarios. For example, (Park et al., 2023) created small town with 25 autonomous agents to simulate human behavior, while (Chang et al., 2024) developed simulated hospital system with patient, nurse, and doctor agents, where the treatment performance of the doctor agent gradually improves as the simulation evolves.
2.2. Multi-agent System
Recently, multi-agent systems have been increasingly ap- plied to scientific research, with studies such as (Ghafarol- lahi & Buehler, 2024) integrating them with ontological knowledge graphs to support materials development. An- other example is (Jin et al., 2024b), which developed a multi-agent peer review system to simulate and study psy- chological phenomena peer review. In contrast, our work focuses on enhancing the design of multi-agent systems by introducing shared memory pool and multimodal capa- bilities, to leverage multi-agent collaboration to generate high-quality review feedback.
In recent years, multi-agent systems based on large language models (LLMs) have demonstrated exceptional capabili- ties and promising applications. By leveraging specialized agents with distinct functions to collaborate and interact, multi-agent systems have found broad applications in fields such as software engineering (Huang et al., 2023; Qian et al., 2023; Jin et al., 2024a), society simulation (Park et al., 2023; Li et al., 2024), and embodied intelligence (Mandi et al., 2024; Chang et al., 2024). Currently, multi-agent systems can be roughly categorized into two types from an application perspective: collaborative task execution and scenario simulation. Collaborative task execution aims to solve complex problems through cooperation among spe- cialized agents. For example, (Khan et al., 2024) uses two expert models to engage in debate, with a non-expert model acting as a judge to select a side, improving the ac- curacy of the non-expert model on reading comprehension tasks. (Huang et al., 2023) designed a multi-agent code generation framework consisting of programmer agent, test designer agent, and test executor agent, achieving efficient and low-cost code generation. On the other hand,
3. Agent Reviewers
3.1. System Overview
We overview our proposed system in Figure 2. Agent Re- viewers comprise four categories of agent roles and two main phases: Domain-Specinn Reviewers Initialization and Visual-enhanced Review Process.
Domain-Specific Reviewers Initialization. During the Domain-Specific Reviewers Initialization phase, the meta re-
3

---

Agent Reviewers
Meto Revlewer
Multimodal Reviewer
Domain-spectt Reviewers Input: Text of paper, comments from multi- modal reviewer, reviews from other reviewers
Area Chair Reviewer Input: The reviews from different reviewers Output: Final reviews, score, and decision
Input: Text of paper
Input: Thumbnail PNG the paper
H
Output: Several keywords of this paper
Output: Comments about the layout.
Responsibility; Extracting a carefully selected set of keywords from the paper's content, which should over core topics expertise required to evaluate the submitted paper.
Responsibility: Evalluates the paper's visuall appeal, formatting, and quality of visual elements such as charts and images, providing feedbbcck to assist other agents the review process.
Output: Comments, score, and decision Responsibiiity: Generating the (Attention/NLL P/O///////tiiiiion related reviews, discussing with other reviewers,
Responsibility: Synthesizing final comments from domain -speifffc reviewers to in two mode: single- paper and batch review.
modifying the reviews accordingly
Figure 3. Key roles in Agent Reviewers system.
viewer agent extracts & keywords from the manuscript. For each keyword, historical paper information (e.g., summaries, decisions) is retrieved from a shared memory pool and in- corporated into a prompt template to initialize a domain- specific reviewer with expertise in the corresponding field. This process generates i specialized reviewers.
specialized multimodal reviewer agent to extract layout and other visual cues relevant to the review. Specifically, we represent the document's main text as a 3x3 thumbnail, serv- ing as input to the multimodal reviewer. We find that this approach effectively balances minimal visual token usage with accurate layout assessment and content understanding. The agent evaluates visual appeal, formatting, and the qual- ity of visual elements such as charts and images, providing feedback to support other agents in the review process.
Visual-enhanced Review Process. In this phase, the multi- modal reviewer first provides comments on the visual con- tent of the manuscript based on the thumbnail of the paper. A PDF converter is utilized to obtain a plain-text version of the manuscript. The visual comments and the plain text manuscript are then fed into the domain-specifer reviewers to elicit variety of review opinions. This is followed by the discussion phase, where each reviewer reads and considers the feedback from others to revise their own reviews. The final revised reviews are submitted to the Area Chair (AC) to compile the final comments on the manuscript and to gather information for updating the shared memory pool.
Domain specific reviewer agent. Peer review significantly benefits from diverse expertise, as reviewers typically have varied backgrounds. To mimic this diversity, we employ multiple domain- specific reviewer agents for each paper. These agents specialized in distinct subfields, assigned by the meta reviewer agent based on keywords extracted from the paper. Each agent is initialized with domain- specific memory from a shared memory pool to assess the paper's 's strengths, weaknesses, and overall quality, providing scores and acceptance recommendations.
This structured system ensures that the paper undergoes comprehensive evaluation, incorporating both textual and vi- sual elements, leading well-rounded and informed review outcomes. The introductions of the different roles and their interaction, together with the design of the shared memory pool are detailed below.
Chair reviewer agent. The chair reviewer agent combines paper information and domain-specifie agents' reviews to deliver the final score, decision and comments, and it also updates the relevant keyword entries in the shared memory pool, enabling the dynamic expansion of the pool.
3.2. Agent Roles
3.3. Agent Interactions
Meta reviewer agent. The meta reviewer agent is responsi- ble for extracting a set of keywords from the paper's 's content, which are then used to initialize group of domain-specific reviewer agents. These keywords are thoughtfully chosen to cover the core topics and expertise required to evaluate the submitted paper, ensuring that each domain-specifee re- viewer agent possesses the necessary domain knowledge to provide thorough and insightful feedback.
Inspired by the peer review process, we have subdivided the two-phase review workflow into the following 5 interaction periods. The domain-specinf reviewers initialization phase corresponds to period I, while the visual-enhanced review process encompasses periods II through V.
I. Domain Specific Reviewer Agent Initialization. The meta reviewer agent extracts A keywords from the paper text to initialize & domain-speciff reviewer agents, each specializing in a distinct field, using relevant knowledge retrieved from the shared memory pool.
Mulit-modal reviewer agent. Existing LLM review meth- ods typically convert papers into plain text, ignoring crucial visual information such as formatting and figures (Huang, 2018). Recent multimodal models can leverage visual in- formation, but high-resolution document images require extensive token usage, making visual token integration com- putationally expensive. To address this, we developed
II Visual Information Extraction. viewer agent evaluates the paper's visual aspects (e.g., lay- out, charts, and figures) based on its thumbnail. This feed- back serves as supplementary information for the subse-
The multimodal re-
4

---

Agent Reviewers
quent review phases.
Table 1. Dataset Comparison
III. Initial Review. Each domain-specific reviewer agent generates an initial review by integrating their area-specific knowledge with the paper's text and visual information, assessing both strengths and weaknesses.
standardized review format
papers
# reviews
PeerRead
3,006
10,770
X
(Kang et al., 2018)
ASAP-Revias
IV. Discussion and Revision. Initial reviews are shared with all agents, allowing them to revise their feedback based on peer insights. This process refines the initial reviews.
8,877
28,119
X
(Yuan et al., 2022)
MReD
7,894
30,764
X
(Shen et al., 2022)
V. Final Review and Memory Update. The chair reviewer agent synthesizes paper information and domain-specift reviews to generate the final comments and decision, while also updating relevant keyword entries in the SMP.
NLPeer
5,672
11,515
(Dycke et al., 2023)
Reviewer2
27,805
99,727
(Gao et al., 2024)
SEA
12,296
47,602
(Yu et al., 2024)
3.4. Shared Memory Pool
Reviews-STD
38,312
144,027
To equip the dopaif-specific reviewer agent with special- ized knowledge, we have designed precise RAG-based Shared Memory Pool (SMP).For each keyword, the pool stores information on related papers, including their sum- maries, strengths and weaknesses, submission years, deci- sions, and other pertinent details. When the meta reviewer agent assigns a keyword to a domain -spaai-specific agent, rele- vant paper knowledge can be retrieved from the pool and provided to the domainn-peciinn agent.
memory about related papers from the Shared Memory Pool (SMP) to initialize the domain-specifi reviewer. We design a two-stage process for fine-grained retrieval: (1) Keywords Retrieval. The meta-reviewer uses keyword description embeddings to search the SMP and retrieves the top-k) most similar keywords and their associated papers as candidates. (2) Paper Reranking. Candidate papers are reranked based on the similarity between their summary embeddings and that of the paper under review, with the top-k2 most similar papers selected as supplementary knowledge for the domain- specific reviewer. After the review, the reviewed paper information is updated to the Shared Memory Pool.
Initialization. Before evaluating Agent Reviewers on the ICLR 2024 test set, we initialize the shared memory pool with papers from Reviews-STD before ICLR 2024 to sure comprehensive knowledge coverage. Using the free GLM-4-Flash API, we generate a summary and several key- words for each paper, along with a description for each keyword. To integrate the keywords, we design a two-stage aggregation pipeline inspired by Instag (Lu et al., 2023). (1) Rule-based Aggregation. Keywords are normalized (lower- cased and cleaned of special characters) and processed with NLTK's stemming to merge synonyms like "conditional gen- eration" and "conditional generator." (2) Semantics-bar based Aggregation. Keywords are encoded into semantic embed- dings, and DBSCAN is used to cluster similar keywords. We found that existing embedding models struggle with Al-specific terms (e.g., overemphasizing surface similarity, linking "recurrent neural network" and "graph neural net- work" while separating "diffusion model" and "VAE"), so we use keyword descriptions for embedding, encoded by all-mpnet- -baoonettbaseee2 model. Finally, GLM-4-Flash creates a new keyword and it's corresponding description for each cluster. The semantics-based aggregation can be iterated for results at different granularities.
4. Reviews-STD
We incorporate selected data from the Reviewer2 (Gao et al., 2024) and SEA (Yu et al., 2024) datasets, which include pa- pers from the International Conference on Learning Repre- sentations (ICLR) (2017-2024) and the Conference on Neu- ral Information Processing Systems (NeurIPS) (2016-2024). For each paper, we retain metadata such the full content, reviews, meta-reviews, and final decisions. Notably, these datasets are sufficiently dated to have potentially been in- cluded in the training datasets of foundational LLMs. Al- though prior research (Lu et al., 2024) suggests LLMs have not memorized this data, the lack of transparency in pub- licly available training datasets prevents confirming this. To ensure the robustness of our results, we also collect ICLR 2024 submissions from OpenReview?, retaining relevant metadata for each paper. Additionally, we employ MinerU (Wang et al., 2024a) to convert PDFs into JSON files.
Standardization. Each paper in the dataset is typically accompanied by 3-5 reviews. Given the variability in re- view formats and evaluation criteria across conferences and years, directly using this data without preprocessing could
Retrieval. During the review process, the meta-reviewer extracts keywords from the paper under review and retrieves
htttps:: hhttps::/hugginggface
sentence-traansfomee watpa//saaaammmaaaseeeeeea base-v2
tettpa://openreee eww..weeeee..e....eeeeeee.
5

---

Agent Reviewers
introduce biases into evaluation outcomes. To enhance con- sistency and accessibility, we apply the ERNIE-Speed-1 128K model to unify diverse reviews into a standardized format. This standardized format comprises two lists: strengths and weaknesses. Strengths highlight features supporting a paper's is acceptance, while weaknesses identify points that could lead to rejection. Each list item is accompanied by a distinct comment to avoid redundancy, As shown in Table I, our dataset provides more consistent format and contains a larger, more comprehensive dataset compared to other publicly available datasets. The prompts used for review summarization are detailed in Appendix D, and the statistics of the proposed Reviews-STD are in Appendix A.
reviews, which are then used the strengths and weak- nesses for these two methods.
5.2. Evaluation Metries
We evaluate the performance of our method in two key aspects: the quality of generated strengths and weaknesses and the accuracy of decisions.
Strengths- Seangtts-Weaaknsssee Weaknesses Analysis. Strengths weaknesses quality analysis focuses on how well the strengths and weak- nesses identified by the LLM align with those of the human reviewers. Evaluating the quality of LLM-generated feed- back the review level is particularly challenging, strong LLM review may represent a synthesis of perspec- tives from multiple reviewers, yet exhibit low similarity to any single one of them. To address this, we focus on fine-grained evaluation by extracting the strengths and weak- nesses from the ground truth review comments in the dataset. This allows us to break down the review-level assessment into a more detailed analysis at the strengths and weaknesses level. Specifically, we use the all-mpnes -base-vvb model to encode the strengths and weaknesses into embed- dings, and then calculate the cosine similarity. We evaluate the extent to which LLM-generated strengths and weak- nesses align with those identified by human reviewers, en- suring high-quality feedback while minimizing irrelevant or inaccurate additions. To achieve this, we design and employ Fi-score and Jaccard to balance coverage and accuracy, sup- plemented by Recall and MaxSim to further assess coverage. Detailed metric definitions are provided in Appendix B.
5. Experiments
5.1. Experimental Setup
We use GPT-4o-mini as the core LLM unless otherwise specified, due to its balanced knowledge base, multimodal capabilities, and cost-effectiveness. By default, we use "title + abstract + introduction" as the paper content, employ 3 domain-specific reviewer agents, and keep the shared memory pool frozen during test time.
Evaluation Dataset. We conducted our evaluation on ICLR 2024. The dataset offers relatively balanced distribution between accepted and rejected papers, and its release date postdates the knowledge cutoff of the core LLM, eliminating the risk of data leakage. We randomly selected 300 papers from ICLR 2024 for the main experiments, and 100 papers for ablation studies, balancing evaluation sufficiency with experimental cost. To ensure consistency, accepted and rejected papers were sampled in accordance with ICLR 2024's overall acceptance rate of approximately 40%.
Decisions Analysis. We approach this as a binary classi- fication problem. However, the inherent class imbalance due to the low acceptance rate may result in biased evalua- tions. To address this, we employ robust metrics designed for imbalanced datasets, including F1-score, Matthews Cor- relation Coefficient (MCC), Balanced Accuracy (Bal. Acc), and G- G-mm ensuring more reliable assessment. Detailed definitions are provided in Appendix B.1.
Shared Memory Pool. We initialized the shared mem- ory pool with all 28372 papers from ICLR 2017-2023 and NeurlPS 2016-2023 in Reviewer-STD, ensuring the SMP provides domain speciffc agents with comprehensive knowl- edge. We used the free Gl.M-4-llash API, extracting key- words per paper, resulting in total of 43954 keywords after 3 rounds of merging(DBSCANN e 0.25, min nampses 2), with an average of 1.78 papers per keyword. During retrieval, the parameters top-kı for keywords and top-k2 for papers as final memory are both set to 5.
5.3. Main Results
Table 2 compares the performance of Agent Reviewers with existing methods across multiple metrics. More detailed comparison results, including additional evaluation metrics, are provided in Appendix B.3.
Compared Methods. We compare our method with AI- Scientist (Lu et al., 2024), AgentReview (Jin et al., 2024b), and LLM Review (Liang al., 2024), using the same LLM (GPT-40-mini) for all methods to ensure fair com- parison. Other settings follow their respective defaults (see Appendix B.2 for details). Neither LLM Review nor Agen- tReview explicitly lists their strengths and weaknesses. To enable a standardized comparison with our method, we use GPT-40-mini extract the strengths and weaknesses of the
Strengths- Weakgtssss Analysis. Using GPT-4o-mini, Agent Reviewers outperforms the current best approach across multiple metrics, achieving a 10.5% improvement in total Jaccard and an 8.5% increase in F1-score, demon- strating a stronger alignment between the strengths and weaknesses generated by our method and those identified
Phttpes//huggiigffac..o
se sentence-traasforreers htw.://rrrnnrreaaaaae-mee--aaaaa -mpnet-base-e
6

---

Agent Reviewers
Table Main results for trengths-weakneese and decisions analysis. Str. and Wk. denote strengths and weaknesses, respectively. F1- score and Jaccard in strengths weaknesses analysis are calculated with a similarity threshold of 0.5, Bold indicates the best performance, underline the second-best. AgentReview(Top-)) accepts the top-k ranked papers per batch of 10 papers. All methods use GPT--4o-iin
tiengthhsweekknessss
Strengths- Strenghss-eaakessess Weaknesses Analysis
Decisions Analysis
Cost(S)
Method
Fl-scoret
Jaccardt
Decision
Fl-scoret
MCC1 Bal. Acet
G-meant
/100 Papers
Total
Str.
Wk.
Total
Str.
Wk.
Offered?
AgentReview((op-33
0.340
0.442
0.245
0.215
0.252
0.194
0.410
0.104
0.549
0.515
7.53
(Jin et al., 2024b)
(gen))eveew(oop4)
0.340
0.442
0.245
0.215
0.252
0.194
0.417
0.028
0.514
0.505
7.53
(Jin et al. 2024b)
AI Scientist
0.426
0.523
0.345
0.285
0.313
0.272
0.049
0.123
0.513
0.158
3.59
(Lu et 2024)
LLM Review
0.420
0.560
0.328
0.275
0.243
0.336
0.13
(Liang al. 2024)
Agent Reviewers
0.462
0.577
0.357
0.315
0.333
0.310
0.566
0.220
0.613
0.612
0.65
0.475
Table 3. Ablation study for strengths-weaknesse analysis. Each
metric is comprehensive measure both strengths and weak- nesses, correpooning to the "Total" column in Table 2.
0.30
0.450
Method
Recallf Fl-scoret
MaxSimf
Jaccard
Total Jaccard
Baseline
0.347
0,416
0.442
0.270
Total F1
0.425
0.29
w/o Multi-Agent
0.335
0.404
0.435
0.260
w/o SMP
0.364
0,412
0.456
0.275
0.400
w/o Multimodality
0.394
0.436
0.463
0.296
0.28
w/o Discussion
0.389
0.424
0.472
0,282
0.375
Agent Reviewers
0.409
0.453
0.476
0.307
-0.27
0.350
2020
2021
2022
2023
Table 4. Ablation study for decisions amalysis.
Year
Method
Accf
Fl-scoret
MCC1
Bal. Acet
G-meant
Baseline
0.66
0.469
0.219
0.609
0.593
Figure 4. Impact of the cutoff' year of shared memory PSoI(SMP) on strengths and weaknesses quality.
w/o Multi-Agent
0.63
0.413
0.143
0.571
0,547
w/o SMP
0.66
0.292
0.110
0.543
0.436
w/o Multimodality
0.65
0.568
0.314
0.668
0.666
5.4. Ablation Studies
w/o Discussion
0.51
0.424
0.045
0.524
0.522
Agent Reviewers
0.70
0.605
0.385
0.705
0.705
We conducted ablation experiments on the key modules of Agent Reviewers to systematically assess their indi- vidual contributions to review quality and decision accu- racy. Tables 3 and 4 present the results for strengths- weaknesses analysis and acceptance decision metrics, re- spectively. "Baseline" refers to a text-only single agent directly generating final reviews. "w/o Multimodality" ex- cludes visual information, relying solely on textual data to review. "w/o Multi-Agent" removes multi-agent interac- tions, using single multimodal reviewer to process both text and vision tokens, then directly generate the final com- ments and decision. "w/o SMP" eliminates the shared mem- ory pool for domain-specific agents. "w/o Discussion" re- moves the discussion period, forcing domain-speciff agents to submit initial review comments directly to the AC without refinement. Furthermore, we assess the generalizability of our method across different LLM APIs, shown in Table 5, demonstrating its adaptability across varying LLMs. Ap- pendix B.6 presents the evaluation results across different conferences and years.
by human reviewers. Our Agent Reviewers strike well- balanced trade-off between coverage and precision, deliv- ering consistently strong performance across all metrics. Appendix B.7 provides a further analysis of strengths and weaknesses, discussing their distribution across categories as well as differences in alignment with human reviewers.
Decisions Analysis. Using GPT-4o-mini, our method ranks first across all four metrics, significantly outperforming isting approaches. Fl-score, MCC, Bal. Acc, and G-mean achieve relative improvements of 35.7%. 78.9%, 11 11.7% and 18.8% over the second-best method, respectively, demon- strating superior alignment with human reviewers' decision patterns. These substantial gains suggest that Agent Review- ers better capture human reviewers' preferences, leading to more consistent and reliable decisions.
Cost Comparison. Table 2 demonstrates that Agent Re- viewers not only achieves strong performance, but also main- tains low costs. It results from our efficient process design, which avoids excessive iterations and delivers excellent re- sults even on cost-effective models like GPT-4o-mini.
Effects of Multi-Agent Interaction. As shown in Tables and 4, the removal of multi-agent interaction results in
7

---

Agent Reviewers
Table Comparison between proprietary and open-source LLMs. 'i text-only LLM, we GPT-44omini for multimodal agent.
Strengths Weaknesses Analysis
Decisions Analysis
LLM
F1-scoref
Jaccard
F1-scoret
MCC1
Bal. Acct
G-meant
Total
Str.
Wk.
Total
Str.
Wk.
GPT-4o-mini
0,453
0.558
0.361
0.307
0.333
0.296
0.605
0.385
0.705
0.705
Proprietary
Gemini-exp- 1206
0.430
0.561
0.327
0.289
0.283
0.304
0.536
0.367
0.668
0.638
Deepseek-V3-1226 V3-1226*
0.461
0.584
0.358
0.310
0.275
0.356
0.531
0.223
0.594
0.484
Deepseek- V3-0324*
0.458
0.564
0.370
0,306
0.288
0.332
0,552
0.299
0.618
0.485
Open-source
DWven.55-VL-72-IImstrre
0.450
0,554
0.362
0,303
0.290
0.327
0.532
0.222
0.600
0.516
InterVL--2..
0,450
0.540
0.373
0.301
0.303
0.309
0.526
0.210
0.586
0.470
ing the multimodal reviewer's importance. We attribute this to its ability to assess document formatting and evalu- ate images and tables, leading to more relevant strengths, weaknesses, and decisions.
Table 6. Decision Analysis Using AC aggregation and Majority Voting VM. Initial: M.V. based on initial reviews before discus- sion; Revised: M.V. based on revised reviews after discussion.
Acct
Fl-score?
MCC1
Bal. Acct G-mcant
M.V. (Initial)
0.53
0.484
0.136
0.572
0.560
Ablation Study on Different LLMs. Table presents the performance of our method across various open-source and proprietary LLMs, including GPT-4o-mini, Gemini- exp-1206, D33 Dwwen255vLL-22B-Insttee
V.V. (Revised)
0.57
0.517
0.206
0.609
0.599
AC Aggregation
0.70
0.605
0.385
0.705
0.705
Table 7. Ablation study on memory retrieval methods. Keyword refers to retrieval based on keywords, while Random refers to random sampling. Both methods utilize 5 papers as memory.
and Intern VL-2.----5--8B Our approach demonstrates strong gen- cralization across different LLMs: Deepseck-V3-1226 achieves the highest quality in analyzing strengths and weak- nesses, while GPT-4o-mini shows the closest alignment with human reviewers in decision decision-making. Appendix B.4 com- pares Agent Reviewers and the Single Agent baseline under different LLMs, while Appendix B.5 explores the use of different LLMs for domain-specific reviewer roles.
V3-1226
Str. & Wk.t
Decisionst
Retrieval
Fl-score Jaccard
Fi-score MCC
Bal. Acc G-mean
Keyword 0.453
0.307
0.605
0.385
0.705
0.705
Randam
0.417
0.278
0.400
0.054
0.528
0.525
an average relative decline of 13% in stranesss metrics and 29% in acceptance decision metries compared to the full method, emphasizing its pivotal role in enhanc- ing review quality and decision accuracy. Furthermore, eliminating the discussion period significantly degrades both strengths- weaknesses evaluation and acceptance deci- sion consistency, underscoring the importance of collabo- rative discussions in refining reviews. To study the role of the AC reviewer in final decisions, Table 6 compares AC- aggregated decisions with majority voting among domain- specific AC-ccssse-- decisions show higher consistency with human reviewers than majority voting.
5.5. Case Study
Figure 5 presents selected weaknesses for KAN: Kol- mogorov-Arnold Networks. By citing specific prior works (e.g., ICLR 2017, 2021, 2022) and identifying overlooked research, our system leverages the Shared Memory Pool (SMP) to produce contextually rich and well-informed cri- tiques. Further details and additional case studies on LoRA and this paper are provided in Appendix C.
1.. The paper does not sufficiently leverage discuss existing res- earch on learnable activation functions (e.g., ICLR 2017, ICLR 2021, ICLR 2022) or related concepts, limiting the context and perceived novelty of the proposed approach.
Effects of Shared Memory Pool. As shown in Tables 3 and 4, removing the SMP results in an average 9% rela- tive decrease in trengthss-eaknessse metrics and 38% relative decrease in acceptance decision metrics compared to our full method. To further investigate the role of SMP, Table 7 shows that retrieving the top-5 papers by keywords significantly outperforms randomly sampling 5 papers as memory, highlighting the importance of precise domain- specific knowledge retrieval. Figure 4 illustrates the impact of initializing SMP with papers from different cutoff years on the quality of strengths and weaknesses. Later cutoff years result in more comprehensive and up-to-date SMP, enhancing the overall quality of review comments.
2. The paper does not leverage the findings from previous research papers about 'Kolmogorov-Arooll Networks (KANs), such as 'ICLR 2023 3364' which proposed novel theoretical architecture, or 'ICLR_208 2___2022__44000 1438' which applied Koopman operator theory to neural sequential models...
Figure 5. Weaknesses Identified for KAN: Kolmogorow-moold Networks Using Gemini-Fxp-11200
6. Conclusion
We constructed multi-agent review system called Agent Reviewers. It is built on LLMs to mimic the human peer re- view process by creating multiple agents with different roles. It includes a meta-reviewer agent for initializing reviewers, multimodal reviewer agent for perceiving visual content in
Effects of Multimodality. Tables 3 and 4 show that re- moving multimodality reduces both strengths-weaknessses weaknesses review quality and acceptance decision accuracy, underscor-
8

---

Agent Reviewers
papers, domain-specine agents responsible for paper review, and chair agent for summarizing review comments. To address the common issues of homogenization and un- professional review comments in Al-based peer review, we designed a shared memory to initialize reviewers with differ- ent backgrounds and introduced a reviewer communication phase to revise review opinions. Additionally, we intro- duced& a multimodal reviewer agent to provide qualitative comments on the visual elements of the papers. Meanwhile, we collected large-scale and standardized dataset called Reviews-STD to facilitate our study and built benchmark using 300 randomly selected ICLR 2024 papers to evaluate and compare different methods. Agent Reviewers demon- strated excellent performance across multiple metrics. This system can serve as an important tool to provide insightful feedback for researchers to enhance paper quality.
ethical safeguards and best practices for its integration.
A Acknowledgements
This work was supported partially by the National Nat- ural Science Foundation of China (Grant No.62376267), the 2035 project of CASIA, the innoHK project, and the Strategic Priority Research Program of Chinese Academy of Sciences under Grant XDA0480200.
References
Back, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07733 2024.
I Limitations. While our system achieves promising results, it has some limitations. First, we focus only on papers from two major Al conferences, limiting the scope of the memory. This training-fis free method can generalize to other domains by expanding the memory accordingly. Second, the current memory design is relatively simple and could be enhanced with techniques from retrieval-angmeentd generation (RAG) or integrating online search. Finally, the system has not undergone any fine-tuning; theoretically, its performance could be further improved by fine-tuning specific agents, such as the Meta-reviewer or Area Chair reviewer. We hope that this work can gain the attention of the community and collaboratively address these limitations in the future.
Chang, M., Chhablani, G., Clegg, A., Cote, M. D., Desai, R., Hlavac, M., Karashchuk, V., Krantz, J., Mottaghi, R., Parashar, P., et al. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. arXiv preprint arXiv:2411.00033 2024.
D' Arcy, M., Ross, A., Bransom, E.,, Kuehl, B., Bragg, J., Hope, T., and Downey, D. ARIES: A corpus of scientific paper edits made in response to peer reviews. In Asso- ciation Computational Linguistics (Volume 1: Long Papers). pp. 6985-7001, 2024.
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Narayanan Venkit, P., Zhang, N., Srinath, M., Zhang, H. R., Gupta, V., Li, Y., Li, T., Wang, F,, Liu, Q., Liu, T., Gao, P.. Xia, C., Xing, C., Jiayang, C., Wang, Z., Su, Y., Shah, R. S., Guo, R., Gu, J., Li, H.,, Wei, K., Wang, Z. Cheng, L.,, Ranathunga, S., Fang. M., Fu, J., Liu, F,, Huang, R., Blanco, E., Cao, Y., Zhang, R., Yu, P. S., and Yin, W. LLMs assist NLP researchers: Critique paper (meta-)reviewins Jreviewing. In Conference on Empirical Methods in Natural Language Processing, pp. 5081 1-5999 2024.
Impact Statement
Peer review is essential for improving the quality of sci- entific research, yet many papers, especially those from early-career researchers, lack sufficient feedback before or during submission. To address this, we introduce Agent Reviewers, a multi-agent LLM-based system that emulates real peer review dynamics. By incorporating multimodal reviewer and a shared memory pool, our multi-agent system delivers detailed and insightful feedback, aiding researchers in refining their work.
Dycke, N., Kuznetsov, I., and Gurevych, I. NLPeer: A uni- fied resource for the computational study of peer review. In Association for Computational Linguistics (Volume I: Long Papers), pp. 5049 1449-5073, 2023.
However, Al-assisted peer review comes with ethical re- sponsibilities. The integrity of peer review depends on fairness, expertise, and human judgment. While our sys- tem enhance and support the review process, it is not designed to replace human reviewers, as LLMS can still produce biased, inconsistent, or overly generic feedback. Human oversight remains essential to ensure fairness and reliability. Additionally, peer review data is inherently sensi- tive, and any deployment of AI-driven review systems must carefully address data privacy and confidentiality concerns. We strongly advocate for the responsible use of Al in peer review and emphasize the need for continued discussion
Fox, C. W., Meyer, J., and Aimé, E. Double-blind peer review affects reviewer ratings and editor decisions at an ecology journal. Functional Ecology, 37(5): 10e11114-111 145-1157, 2023.
Gao, Z., Brantley, K., and Joachims, T. Reviewer2: Op- timizing review generation through prompt generation. arXiv preprint arXiv:2402.10886 2024.
Ghafarollahi, A. and Buehler, M. J. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning. arXiv preprint wwxiv:2409.055555 2024.

---

Agent Reviewers
Huang, D., Bu, Q.. Zhang, J. M., Luck, M., and Cui, H. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2023.
Lu, K., Yuan, H., Yuan, Z., Lin, R., Lin, J., Tan, C., Zhou, C., and Zhou, J. instag: Instruction tagging for ana- lyzing supervised fine-tuning of large language models. In The Tweifth International Conference on Learning Representations, 2023.
Huang, J.-B. Deep paper gestalt. arXiv preprint arXiv:1812.08775. 2018.
Mandi, Jain, S., and Song, S. Roco: Dialectic multi-robot Huang, Q. Vora, J., Liang, P., and Leskovec, J. MLAgeet- Bench: Evaluating language agents on machine learn- ing experimentation. In Proceedings of the 41st Inter- national Conference on Machine Learning, pp. 20271- 20309, 2024.
collaboration with large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 286-299. IEEE, 2024.
Park, J. S., O' O'Bien,, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium user interface software and technology, pp. 1-22, 2023.
Jin, H., Huang, L., Cai, H., Yan, J., Li, B., and Chen, H. From Ilms to llm-based agents for software engineering: survey of current, challenges and future. arXiv preprint arXiv:2408. 02479, 2024a.
Qian, C.,, Dang, Y., Li, Liu, W., Xie, Z., Wang, Y., Chen, W., Yang, C., Cong, X., Che, X., et al. Experiential co- learning of software-developing agents. arXiv preprint arXiv:2312.17023 2023.
Jin, Y., Zhao, Q. Wang. Y. Chen, H., Zhu, K., Xiao, Y., and Wang, Agentreview: Exploring peer review dynamics with Ilm agents. arXiv preprint arXiv:2406. 12708, 2024b.
Shen, C., Cheng. L.., Zhou, R., Bing, l.., You, Y., and Si, L. MReD: A meta-review dataset for sinucture-controllde text generation. In Findings of the Association for Com- putational Linguistics: ACL 2022, pp. 2521-2533 2022.
Kang, D., Ammar, W., Dalvi, B., van Zuylen, M., Kohlmeier, S., Hovy, E., and Schwartz, R. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp- 1647-1661, 2018.
Stelmakh, I., Shah, N. B., Singh, A., and Daumé III, Prior and prejudice: The novice reviewers' bias against resubmissions in conference peer review. Proceedings of the ACM on Hluman-Compunn Interaction, 5(CSCW1); 1-17, 2021.
Khan, A., Hughes, J., Valentine, D., Ruis, L.., Sachan, K., Radhakrishnan, A., Grefenstette, E., Bowman, S. R., Rocktäschel, T., and Perez, Dehating with more persua- sive Ilms leads to more truthful answers. arXiv preprint arXiv:2402.06782, 2024.
Von Bearnensquash, C. Paper gestalt. Secret Proceedings of Computer Vision and Pattern Recognition (CVPR), 2010.
Wang. B., Xu, C.,, Zhao, X., Ouyang, L., Wu, F., Zhao, Z., Xu, R., Liu, K., Qu, Y., Shang, F,, et al. Mineru: An open- source solution for precise document content extraction. arXiv preprint arXiv:2409. 18839, 2024a.
Li, J., Wang, S., Zhang, M., Li, W., Lai, Y., Kang, X., Ma, W., and Liu, Y. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint 20X:022405.02957 2024.
Wang, Q., Downey, D., Ji, H., and Hope, T. SciMON:
Liang, W. Zhang, Y., Cao, H., Wang, B., Ding, D. Y., Yang, X., Vodrahalli, K., He, S., Smith, D. S., Yin, Y., et al. Can large language models provide useful feedback on research papers? large-scale empirical analysis. NEJM AI, Aloa241420 2024.
Scientific inspiration machines optimized for novelty. In Association for Computational Linguistics (Volume 1: Long Papers), pp. 279-299, 2024b.
Wang. Y., Guo, Q. Yao, W., Zhang, H., Zhang, X., Wu, Z.. Zhang, M., Dai, X., zhang, M., Wen, Q. Ye, W., Zhang, S. and Zhang, Y. Autosurvey: Large language models can automatically write surveys. In Neural Information Processing Systems, pp. 1-10, 2024c.
Liu, R. and Shah, N. B. Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306. 00622, 2023.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The AI Scientist: Towards fully automated open-ended scientific discovery. arXiv preprint 2XXv:24088.0629, 2024.
Wei, S., Xu. X., Qi, X. Yin, X., Xia. J., Ren, J., Tang, P., Zhong, Y., Chen, Y., Ren, X., et al. Aca- demicgpt: Empowering academic research. arXiv preprint arXiv:2311. 2023.
10

---

Agent Reviewers
Weng, Y., Zhu, M., Bao, G., Zhang. H., Wang, J., Zhang, Y., and Yang, L.. Cycleresearcher: Improving automated research via automated review, 2024. URL https:// arxiv. org/ abs/24333 00816.
Yu, J., Ding, Z., Tan, J., Luo, K., Weng, Z., Gong, C.,, Zeng, L.., Cui, R., Han, C., Sun, Q- Wu, Z.., Lan, Y., and Li, X. Automated peer reviewing in paper SEA: Standardization, evaluation, and analysis. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10164- 10184, 2024.
Yuan, W., Liu, P., and Neubig, G. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75, 2022.
Zhang, J., Zhang, H., Deng, Z., and Roth, D. Investigating faimeess disparities in peer review: A language model enhanced approach. arXiv preprint arXiv:2211.0m 1.06698, 2022.
Zhou, R., Chen, L., and Yu, K. Is LLM reliable reviewer? comprehensive evaluation of LLM automatic paper viewing tasks. In Joint International Conference on Com- putational Linguistics, Language Resources and Evalua- tion (LREC-COLING 2024), pp. 9340-9351, 2024.

---

Agent Reviewers
A. Dataset Statistics
Table Dataset Statistics for ICLR
ICLR
Total
2017
2018
2019
2020
2021
2022
2023
2024
# papers
487
907
1,550
2,210
2,590
2,659
3,821
5,703
19,927
tokens per paper
7,666
8,762
9,553
10,197
11,354
12,877
12,974
21,830
12,206
# reviews
1,489
2,739
4,719
6,712
10,010
10,401
14,480
22,064
72,614
# tokens per review
411
514
557
541
639
812
793
686
635
# weaknesses
3,024
6,009
10,406
14,099
17,206
16,151
23,435
35,169
125,499
# weaknesses per paper
6.21
6.63
6.71
6.38
6.64
6.07
6.13
6.17
6.30
# tokens per weakness
20.42
21.09
21.02
21.41
21.51
21.04
21.01
21.82
21.02
# strengths
2,050
3,825
6,510
9,739
12,229
11,883
17,692
27,244
91,172
# strengths per paper
4.21
4.22
4.20
4.41
4.72
4.47
4.63
4.78
4.58
# tokens per strength
17.42
17.87
17.87
18.60
18.73
18.34
17.96
19.17
18.57
accepted 41%
37%
32%
31%
33%
41%
41%
40%
37%
Table 9. Dataset Statistics for NewrIPS
NeurIPS
Total
2016
2017
2018
2019
2020
2021
2022
2023
2024
# papers
554
666
986
1,386
1,895
2,449
2,817
3,395
4,237
18,385
tokens per paper
6,941
7,237
7,266
7,405
7,733
8,347
8,372
24,428
28,643
15,723
# reviews
3,159
1,938
3,008
4,243
7,261
9,589
10,393
15,175
16,647
71,413
# tokens per review
454
438
542
455
699
766
733
736
629
665
weaknesses
3,367
4,002
6,137
8,758
11,027
14,125
16,982
19,239
26,079
109,716
# weaknesses per paper
6.08
6.01
6.22
6.32
5.82
5.77
6.03
5.67
6.16
5.97
# tokens per weakness
22.14
22.80
23.04
22.96
22.63
22.24
21.20
22.08
21.78
21.79
# strengths
2,871
3,367
5,314
7,429
10,123
12,148
13,975
17,093
22,199
94,519
# strengths per paper
5.18
5.06
5.39
5.36
5.34
4.96
4.96
5.03
5.24
5.14
# tokens per strength
19.17
19.61
19.88
18.99
18.88
19.24
18.40
18.99
19.35
18.90
accepted
100%
100%
100%
100%
100%
95%
95%
95%
95%
96%
B. Experiments Details
B.1. Evaluation Metrics
B.1.1. STRENGTHS- WEAKNESSES ANALYSIS METRICS
To evaluate the similarity between strengths and weaknesses generated by LLMs and those from human reviews, we compute pairwise similarity scores and use a similarity threshold T' determine matches. Based on this, we define the following metrics(total refers to the combined strengths and weaknesses):
Recall: Measures the proportion of human review strengths weaknesses that captured by the LLM-ggnerated reviews. A strength or weakness is considered "recalled" if at least one LLM-generated strength or weakness has a similarity score 2 T with it.
Recalled Human Strengths
Recalled Human Weaknesses
Recallsı.
Recallws.
(1)
Total Human Strengths
Total Human Weaknesses
Total Recalled
RecallToul
(2)
Total Human Strengths + Weaknesses
12

---

Agent Reviewers
Precision: Measures the proportion of L.M--generated strengths weaknesses that align with human-reviewed strengths weaknesses. A generated strength weakness is considered "precise" if at least one human-reviewed strength weakness has similarity score 2 ' with it.
Precise LLM Strengths
Precise LLM Weaknesses
Precisionga
Precisionws. =
(3)
Total LLM Strengths
Total LLM Weaknesses
Total Precise
Precisioniaal
(4)
Total LLM Strengths + Weaknesses
F1-Score: Balances Recall and Precision by computing their harmonic mean:
F1-Scoresu. =2 Precisionsi. x Recallsir
Precisionws x Recallws.
F1-Scorews. 2
(5)
Precisionsu. Recallsu.
Precisionwk Recallwx.
Precisionyaal X Recall
Fl-Score = x
(6)
Precisioniaall + Recall
Measures the average maximum similarity between numan-reviewed strengths weaknesses and LLM-generated strengths weaknesses. For each human strength or weakness, the highest similarity with any LLM-ggeerated item is computed, and the average across all human-reviewed items is taken:
MaxSim:
MaxSim
1 N
Σ
max Sim Sim G₃)
where NN is the total number of strengths weaknesses in the human review, H, represents the i-th human-reviewed strength weakness, Gy represents the j-th LLM-generater strength or weakness, and Sim H,, Gy) is their similarity score.
Jaccard Index: Measures the overlap between human-reviewed and LLM-generated strengths or weaknesses(items can refer to strengths, weaknesses, or the total):
Intersection Recalled LLM Items + Precise Human Items
(8)
Intersection
Jaccard
Total Human Items Total LLM Items Intersection
These metrics comprehensively evaluate the alignment between L.LM-generated and human-reviewed strengths and weak- nesses, capturing both coverage and accuracy, well their overall overlap.
1.1.2. DECISIONS ANALYSIS METRICS
To address potential bias caused by class imbalance, we supplemented Accuracy with the Fl-score, Matthews Correlation Coefficient (MCC), Balanced Accuracy (Bal. Acc), and G-mean. These metrics provide greater robustness imbalanced scenarios by accounting for both positive and negative class distributions.
Definitions:
TP (True Positives):
Cases where the LLM correctly predicts acceptance, matching the human reviewer's decision. TN (True Negatives): Cases where the LLM correctly predicts rejection, matching the human reviewer's decision.
FP (False Positives): Cases where the LLM predicts acceptance, but the human reviewer rejects.
FN (False Negatives): Cases where the LLM predicts rejection, but the human reviewer accepts.
Based on these definitions, the metrics are calculated as follows:
13

---

Agent Reviewers
TP TN
Accuracy
(10)
TP+TN F
Precision X Recall
TP TP + FP
TP
FI-Score = 2
where
Precision
Recall
(11)
Precision + Recall
TP + FN
TPXTN - FP EN FN)(TN +N)]TT FP)(N FNN
MCC
(12)
TP
TN
Bal. Acc
+
(13)
TP+ EN TN FP 2
TP V TP+ FN
TN TN FP
G-mean
(14)
These metries collectively evaluate the model's performance under imbalanced conditions by considering both the correctness of predictions and the balance between positive and negative classes.
B.2. Compared Methods Configurations
We followed the default settings for each method and extracted strengths and weaknesses from their review comments for evaluation. For AgentReview, adopted the baseline settings of the method. To enable strengths-weaknesse evaluation, used GPT-4o-mini to extract strengths and weaknesses from the AC's final review. For AI- Al-Scientist, followed its default settings, including num_ref lect ions=5, nuw_fuu sam_fs_exxmpless-..... num_reviews set.ieewwsssemmbe=55.5ee and t remperature=0 1. Since its reviews contain formatted strengths and weaknesses, we directly used them for evaluation. For L.M Review, we parsed the paper content, including the title, abstract, and captions of tables and figures, to construct the standardized imput as specified by the method. We then used GPT-4o-mini to extract strengths and weaknesses from its complete review for evaluation.
B.3. Detailed Main Results
Tables 10 and 1l present detailed strengths-weale weaknesses and decision analysis results. We compare Agent Reviewers with existing methods and Single Agent method, which uses a single text-only agent to directly generate the final comments and decision. Table 10 shows that Agent Reviewers perform well across all metrics, particularly balancing coverage and accuracy (see Fl-score and Jaccard). Table 11 shows that, compared to other methods, Agent Reviewers achieve the highest alignment with human reviewers in decision-making.
Table 10, Detailed strengths-weatd weaknesses analysis main results. Str. denotes strengths, and Wk. denotes weaknesses. Recall and F1-score calculated using similarity threshold of 0.5, chosen for its alignment with human perception of similarity.
Recallf
Fl-scoref
MaxSimt
Jaccard
Method
LLM
Total
Str.
Wk.
Total
Str.
Wk.
Total
Str.
Wk.
Total
Str.
Wk.
AgentReview
GPT-4o-mini
0.314
0.445
0.209
0.340
0.442
0.245
0.438
0.495
0.391
0.215
0.252
0.194
(Jin al., 2024b)
AL.Scientist
GPT-4o-mini 0.361
0.453
0.287
0.426
0.523
0.345
0.444
0.477
0.418
0.285
0.313
0.272
(Lu et al., 2024)
LLM Review
GPT-4o-mini 0.439
0.520
0.373
0.420
0.560
0.328
0.487
0.515
0.466
0.275
0.243
0.336
(Liang et al., 2024)
Single Agent
GPT-4o-mini
0.357
0.496
0.245
0.430
0.575
0.302
0.444
0.488
0.410
0.284
0.317
0.268
Agent Reviewers
GPT-4o-mini
0.418
0.541
0.319
0.462
0.577
0.357
0.477
0.526
0.438
0.315
0.333
0.310
14

---

Agent Reviewers
Table 11. Detailed decisions analysis main results. (gentRevviowIopp-k) accepts the top-k ranked papers per batch of papers.
Decision Offered?
Method
LLM
Acct
Fl-scoref
MCC1
Bal. Acct
G-meant
AgentRevieww(oo-3)((inn et al., 2024b)
GPT-4o-mini
0.587
0.410
0.104
0.549
0.515
AgennRevieww(To--)((in et al., 2024b)
GPT-4o-mini
0.533
0.417
0.028
0.514
0.505
AI AL et al., 2024)
GPT-4o-mini
0.610
0.049
0.123
0.513
0.158
LLM Review(Liang al., 2024)
GPT-4o-mini
Single Agent
GPT-4o-mini
0.610
0.489
0.175
0.586
0.574
Agent Reviewers
GPT-4o-mini
0.607
0.566
0.220
0.613
0.612
B.4. Detailed Ablation Results
Table 12 presents the performance of Agent Reviewers across different LLMs, including GPT-Ao-mini, Gemini-exp- and Reepscek-V3-12286 The experiments were conducted on the same set of 300 ICLR 2024 papers used in the main study. We compare it with single agent method, which uses single text-based agent directly generate the final review. Notably, the knowledge cutoff dates for GPT-4o-mini and Gemini-exp-1201 120- precede the public release of ICLR 2024, ensuring data leakage. However, DeepSeek-Vi V3 has a later knowledge cutoff, posing potential risk of knowledge leakage. Nevertheless, our experiments did not reveal any anomalously high performance suggestive of data leakage, nor any clear signs memorization of paper content or reviews. Therefore, we included it in our evaluation. The results indicate that Agent Reviewers achieve consistently strong performance across different LLMss, showing improvements over the single-agent method.
Table Detailed experiments on different LLMs. Single Agent means using single text-only agent to generate review comments
directly. Using GPT-4o-mini for multimodal ability, DeepSeck-V3. V3 is text-only the time of writing. Acceptance Decisions Analysis
Fine-grained Review Analysis
LLM
Method
Fl-scoret
Jaccard
F1-scoret
MCC1
Bal. Acct
G-mean
Total
Str.
Wk.
Total
Str.
Wk.
Single Agent
0,430
0.575
0.302
0.284
0.317
0.268
0.489
0.175
0.586
0.574
GPT-4o-mini
Agent Reviewers
0.462
0.577
0.357
0.315
0.333
0.310
0.566
0.220
0.613
0,612
Improvement
+0.032
+0.002
+0.055
+0.031
+0.016
+0,042
+0.077
+0.045
+0.027
+0.038
Single Agent
0.467
0.606
0.345
0.325
0.321
0.340
0.308
0.092
0.542
0.493
Gemini-exp sitiiiexpp-2000 1206
Agent Reviewers
0.465
0.603
0.355
0.316
0.284
0.364
0.452
0.113
0.556
0.542
Improvement
-0,002
-0.003
+0.010
-0.007
-0.037
+0.024
+0.144
+0.021
+0.014
+0.049
Single Agent
0.427
0.583
0.2889
0.288
0.300
0.288
0.578
0.097
0.519
0.234
Deepseck-V3-1226 V3-1226*
Agent Reviewers
0.485
0.602
0.383
0.329
0.293
0.379
0.589
0.155
0.551
0.390
Improvement
+0.057
+0.019
+0.094
+0.041
-0.007
+0.091
+0.010
+0.058
+0.032
+0.156
B.5. Experiments on Using Different LLMs for Domain-Specific Reviewers
We consider the diversity of domain-specific reviewers to be an important factor in the review process. To further explore this, we experimented with assigning different large language models (LLMs) to different domain-specific reviewers, as shown in Table 13. Specifically, we used GPT-4o-mini, Gemini-exp-I200 and Deepseck-V3--e V3-0324 the LLMs for three domain-specific reviewers, and evaluated them on the same set of 100 randomly sampled papers. The results show that the multi-LLM setup did not yield performance improvements. We attribute this to the fact that system already encourages diverse perspectives by equipping each reviewer with distinct domain knowledge, leaving limited room for further gains through LLM heterogeneity.
B.6. Evaluation across Conferences and Years
Table 14 explores the review performance across conferences(ICI.R 2024 and NeurlPS 2024). We evaluate the proposed Agent Reviewers against Single Agent baseline, where a single text-only agent directly generates the final review. For each conference, 100 papers were randomly sampled for evaluation using the default language model, 4PT-4o-mini The results across both conferences are consistent: Agent Reviewers consistently outperform the Single Agent baseline, with an average relative improvement of 23.5% on ICLR and 44.7% on NeurIPS.
15

---

Agent Reviewers
Table 13. Comparison between using single LLM and multiple LLMs. text-only at the time of writing.
'Using GPT-4o-mini for multimodal ability, DeepSeek-V.
Strengths- Weaknesses Analysis
Decisions Analysis
LLM
Fl-scoret
Jaccard
Fl-scoret
MCC1
Bal. Acct G-meant
Total
Str.
Wk.
Total
Str.
Wk.
GPT-4o-mini
0.453
0.558
0.361 0.307
0.333
0.296
0.605
0.385
0.705
0.705
Gemini-exp- 1206 1206
0.430
0.561
0.327
0.289
0.283
0.304
0,536
0.367
0.668
0.638
Deepseek-V3-0222
0.458
0.564
0.370
0.306
0.288
0.332
0.552
0,299
0.618
0,485
Multi-LLMs
0.454
0.568
0.356 0.304
0.303
0.313
0.514
0.241
0.628
0.627
Table 14. Comparison of review performance across conferences (ICLR 2024 and NeurIPS 2024). Single Agent means using a single
text-only agent to generate review comments directly.
Strengths-Weaknessee Weaknesses Analysis
Decisions Analysis
Conference
Method
Recall
F1
MaxSim
Jaccard
F1
MCC
Bal.A
G-mean
Single Agent
0.347
0,416
0,442
0.270
0.469
0.219
0.609
0.593
ICLR 2024
Agent Reviewers
0.409
0.453
0.476
0.307
0.605
0.385
0.705
0.705
Improvement
0.062
0.037
0.034
0.037
0.136
0.166
0.096
0.112
Single Agent
0.357
0.440
0,444
0.299
0.349
0,039
0.525
0.424
Ne 2024
Agent Reviewers
0.418
0.461
0.476
0.319
0.589
0.120
0.591
0.569
Improvement
0.061
0.021
0.032
0.020
0.240
0.081
0.066
0.145
Table 15 explores the review performance across years(ICLR 2024 and ICLR 2023). We evaluate the proposed Agent Reviewers against a Single Agent baseline, where a single text-only agent directly generates the final review. For each conference, 100 papers were randomly sampled for evaluation using the default language model, GPT-4o-mini. The cutoff' year for the shared memory pool (SMP) was set one year prior to the test data year, to avoid data leakage from SMP. It is worth noting that, in the ICLR 2023 evaluation, GPT-4o-mini's knowledge cutoff postdates the paper release, posing potential risk of data leakage though no clear signs of memorization were observed. Whether in 2023 or 2024, Agent Reviewers have shown better results than the single-agent baseline, demonstrating the consistency of the gains of our method. For ICLR 2023, the size of the shared memory pool (SMP)-an an essential component of our method-war reduced from 28,372 to 21,156 papers (a 25% decrease) due to the earlier cutoff year. This reduction may have limited the effectiveness of decision analysis compared to ICLR 2024.
Table 15. Comparison of Review Performance across Years(ICLR 2024 and ICLR 2023). Single Agent means using a single text-only agent generate review comments directly.
BrrengtssWeaakesse Analysis
Decisions Analysis
Conference Method
Recall
F1
MaxSim
Jaccard
F1
MCC
Bal.A
G-mean
Single Agent
0.347
0.416
0.442
0.270
0.469
0.219
0.609
0.593
ICLR 2024
Agent Reviewers
0.409
0.453
0.476
0.307
0.605
0.385
0.705
0.705
Improvement
0.062
0.037
0.034
0.037
0.136
0.166
0.096
0.112
Single Agent
0.334
0.408
0.432
0.271
0.474
0.169
0.580
0.554
ICLR 2023
Agent Reviewers
0.390
0.439
0.467
0.298
0.538
0.139
0.570
0.570
Improvement
0.056
0.031
0.035
0.027
0.064
-0.030
-0.010
0.016
B.7. Detailed Strengths-Weid Weaknesses Analysis
We observe in Table 2 that nearly all methods are less effective at identifying paper weaknesses compared to strengths. We hypothesize that this discrepancy arises because the strengths proposed by reviewers tend to be relatively consistent, e.g., similar to claimed contributions, whereas weaknesses are more diverse and often depend on the reviewers' individual background and subjective judgment. This diversity makes it more difficult for LLM-generated weaknesses to match humans'. To further investigate this, analyzed the semantic similarity of strengths and weaknesses generated by Agent Reviewers using GPT-4o-mini, Gemini, and Deepseek- V3 for the same set of papers. On average, the similarity scores between GPT-4o-mini and Gemini were 0.664 for strengths and 0.484 for weaknesses, while the similarity between GPT-4o-mini and Deepseek-V3 V3 was 0.738 for strengths and 0.663 for weaknesses.
We present a classification of the strengths and weaknesses identified by Agent Reviewers (using GPT-4o-mini) and human reviewers on the subset of 300 papers from ICLR 2024 in Figure 6. We used GPT-4o-mini to classify the strengths and
16

---

Agent Reviewers
weaknesses, with each strength weakness being classified into up categories. In terms of strengths, both Agent Reviewers and human reviewers placed significant emphasis on "Implications of the research" and "Novelty," with high degree of alignment between the two. However, Agent Reviewers mentioned "Reproducibility" far less frequently compared to human reviewers. Regarding weaknesses, Agent Reviewers mentioned "Clarity and presentation" more frequently than human reviewers, but they fell short addressing aspects such as "Comparison with related work," "Theoretical soundness," " and "Novelty." We believe that this detailed analysis can help us understand which areas the evaluations of Agent Reviewers may align more closely with those human reviewers, thereby indicating areas where their assessments may hold greater value.
Implications of the research
LLM Matched
Add experiments on more datasets
LLM Unmatched
Human Matched
Clarity and presentation Ethical aspects
Human Unmatched
Algorithm efficiency
Reproducibility
Comparison with related work
Theoretical soundness
Missing citations
Add ablation experiments
Novelty
Scalability
Generalization
0.1
0.2
0.4
Proportion
(a) Proportion of different opinion categorics LLM/Human strengths(stacked matched/unmatched)
Implications of the research
LLM Matched
Add experiments more datasets
LLM Unmatched
Human Matched
Clarity and presentation Ethical aspects Algorithm efficiency
Human Unmatched
Reproducibility
Comparison with related work
Theoretical soundness
Missing citations
Add ablation experiments
Novelty
Scalability
Generalization
0.1
0.2
0.3
0.4
Proportion
(b) Proportion of different opinion categories in LLM/Human weaknesses(stackkdd matched/unmatched)
Figure Proportion of different opinion categories in LLM//uman strengths and weaknesses (stacked matched/unmatched) Proportion refers to the ratio of the number of strongths/weckknessss to the total number of strengths/weakknessss
17

---

Agent Reviewers
C. Case Study
Tables 16 and 17 present the strengths and weaknesses generated by Agent Reviewers for KAN; Kolmogorov-Arood Networks and LoRA: Low-Rank Adaptation of Large Language Models, using Gemini-exp-1206 Agent Reviewers produce insightful, detailed, and comprehensive critiques, incorporating discussions on the specific previous works and analyses of figures. Tables 18 present study own paper, where Agent Reviewers provide constructive and insightful feedback.
Table 16. A case study KAN: Kolmogorov Arnold Networks, demonstrating weaknesses and strengths generated by Agent Reviewers.
KAN: Kolmogorov-Amoll Networks
Weaknesses
The paper does not sufficiently leverage or discuss existing research on learnable activation functions (e.g., ICLR 2017, ICLR 2021, ICLR 2022) or related concepts, limiting the context and perceived novelty of the proposed approach.
The paper does not leverage the findings from previous research papers about "Kolmogorov-Amold Networks (KANs)', such as 'ICLR.203..3 2022_3364' which proposed a novel theoretical architecture, or ICCR_2022_144 which applied Koopman operator theory to neural sequential models.
While KANs are shown to outperform MLPs in specific tasks, the paper does not adequately address potential limitations in scalability or handling complex. high-dimensional data, despite acknowledging that KANs typically require smaller computation graphs than MLPs.
The empirical validation is primarily limited to small-scale AI + Science tasks, and the paper lacks wider range of datasets to demonstrate the generalizability of KANs. The paper does not provide a comprehensive comparison with other state-of-the-ar methods beyond MLPs, making it difficult to assess the relative performance of KANs in a broader context, especially regarding computational cost and efficiency.
extensive experimental results
The practical implementation details, potential challenges, and computational overhead of using KANs in real-world applications not extensively discussed.
Strengths
The paper introduces Kolmogorov-Amold Networks (KANs) as novel alternative to Multi-Layer Perceptrons (MLPs), based on the Kolmogorov representation theorem, representing a significant shift in neural network architecture.
KANs utilize learnable activation functions edges, parameterized splines, instead of fixed activation functions on nodes, offering enhanced flexibility and expressiveness function approximation.
KANs are presented as more interpretable than MLPs, with visualizations (Figures 0.1, 2.1, 2.2) illustrating their structure and potential for use in scientific discovery, aiding in the (re)discovery of mathematical and physical laws.
The paper demonstrates that smaller KANs achieve comparable or better accuracy than larger MLPs in function fitting tasks, as supported by results in mathematics and physics examples.
18

---

Agent Reviewers
Table 17. case study on LoRA: Now-Rank Adaptation of Large Language Models, demonstrating weaknesses and strengths generated by Agent Reviewers.
LoRA: Low-Rank Adaptation Large Language Models
Weaknesses
The paper's exploration the relationship between rank and performance is limited, particularly in how the optimal rank is determined for different tasks and models. A more systematic study rank selection criteria would be beneficial.
The paper notes that the performance of adaptation methods does not always monotonically improve with more trainable parameters, shown in Figure 2, which could be seen as a limitation area for further investigation. A deeper exploration of this phenomenon would strengthen the paper.
The paper does not explore the potential synergies or trade-off's with methods like those proposed in other papers, which focuses on memory footprint reduction. Such an analysis would provide a more comprehensive understanding of LoRA's capabilities.
The paper does provide detailed comparison with other parameter-efficient methods like Compacter, which could highlight the relative advantages and disadvantages of LoRA, especially in terms of performance and computational efficiency.
The paper acknowledges that it's not straightforward to batch inputs to different tasks when using LoRA, which could limit its applicability in multi-task scenarios. A more thorough discussion and potential solutions to this limitation are needed.
Strengths
Introduces LoRA, novel parameter-efficient adaptation method that significantly reduces the number of trainable parameters by freezing pre-trained weights and injecting trainable rank decomposition matrices.
LoRA orthogonal to many prior methods and can be combined with them, enhancing its flexibility and potential for further improvements, demonstrated in Appendix E.
The linear design of LoRA allows merging the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to a fully fine-tuned model.
Provides empirical evidence that LoRA can reduce the number of trainable parameters by up to 10,000 times and GPU memory requirements by 3 times compared to fine-tuning GPT-3 175B with Adam.
Demonstrates that LoRA performs on par with or better than full fine-tuning on various models (RoBERTa, DeBERTa, GPT-2, GPT-3) while having fewer trainable parameters and higher training throughput.
19

---

Agent Reviewers
Table 18. A study on this paper, using Deepseek-V..
Agent Reviewers: Domain-specifer Multimodal Agents with Shared Memory for Paper Review
Keywords and memory
Agent Reviewers
You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism
Group Fairness in Peer Review
Sine-Size-Fits-All Approach Improving
Randomness in Paper Assignment
Counterfactual Evaluation of Peer-Review Assignment Policies
Shared Memory Pool
Mention Memory: incorporating textual knowledge into Transformers through entity mention attention
An Investigation into the Role of Author Demographics in ICLR Participation and Review
What's new? Summarizing Contributions in Scientific Literature
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Sparse is Enough in Scaling Transformers
Multimodal Agent
You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism
Perceptual Score: What Data Modalities Does Your Model Perceive?
Humanly Certifying Superhuman Classifiers
Teaching Machines Describe Images with Natural Language Feedback
Paraphrasing Is All You Need for Novel Object Captioning
Comments
Weaknesses
The reliance on pre-trained LLMs and multimodal models may limit the generalizability of the system, especially domains with limited or no pre-existing data.
There may be potential gaps in the experimental design, particularly in how the performance of the proposed system is benchmarked against existing methods. The paper lacks a detailed discussion the evaluation metrics and datasets used.
The paper does not fully address potential scalability issues, especially given the large volume of submissions in conferences like ICLR. It is unclear how the system would perform under high-load conditions.
The generalizability of the proposed system to other domains conferences has not been thoroughly explored, raising concerns about its applicability beyond the tested scenarios.
Strengths
The system leverages both text and visual elements (e.g.. figures, layout) provide more comprehensive feedback, addressing significant limitation of existing Al-based review systems.
The shared memory pool (SMP) is an innovative feature that mimics real reviewers' back- ground knowledge, enhancing the system's ability to generate domain-specift and insightful feedback.
The results indicate significant performance improvements in paper review tasks, shown in the comparative analysis with existing systems, demonstrating the practical applicability of the proposed approach.
The system architecture, including Multi-agent Interaction (MI), Shared Memory Pool (SMP), and Multimodal Agent (MA), well-designed and aligns with real-world peer review practices.
The paper presents well-structured layout with clear sections, effective use of headings, and high-qualin figures that enhance understanding of the content.
Decision
score
6
recommendation
Accept
20

---

Agent Reviewers
D. Prompts
D.1. Prompts for dataset standardization
Prompt for Review Summarization
are an expert computer science and artificial intelligence. will provi de you with multiple reviews of a machine learning conference paper. Each review begins with "vveeesssstrr and ends with "<]reviex_end)>> REVIEWS: (reviews)
Your task is to generate a final summary in JSON format based on these reviews. The JSON object should contain two keys: 'strengths' and 'wcaknesses', each mapping to 1ist. The 'strengths' list should capture the positive aspects of the paper as potential reasons for acceptance, while the 'weaknesses" list should capture the negative aspects, questions, or suggestions as potential reasens for rejection.
Please ensure that:
Each item in the lists is specific, detailed, and avoids generalization.
The items are not duplicated.
All key points from the original reviews are covered.
The summary dces not introduce any content that is not present in the original reviews.
Only output the JSON content, and do net include any additienal text or commentary, as it will not be parsed correctly. Your output should be formatted as a markdown code snippet according to the follewing pattern:
''json
rsnghhhs"" [String], "weaknesses": [String]
21

---

Agent Reviewers
D.2. Prompts for Agent Reviewers
Prompt for Mulitmodal reviewer agent
As multimodal agent, your task is to act as AI expert, and provide valuable information and evaluations to assist in reviewing an Al conference paper, based solely v*thumbnaila4 of the paper. Please adhere to the following requirements; **Avoid Repeating the Abstractes; Dc not include information that overlaps with the paper's abstract. Focus content **Base on Visible Informatione:: All content information visible in the thumbnai1. Aveid any form of speculation +*Carefully Aveid Hallucinations= the thumbnail. Do not add uncertain details.
covered in the abstract.
strictly derive from the assumption on. Only describe what you can directly observe in
**Use Structured Output**: Organize your response in a clear and orderly manner to facilitate understanding and reference by other agent.
*+Clear and Concise L Use precise and straightforwarr language to ensure accurate information delivery.
Please provice your output in the following JSON format, starting with Y and ending with }) followed by "<|END__FF _F__OON
"strengths";
"innovation"; "Describe the innovative aspects research methods, theoretical Frameworks, or experimental results.", wwieual_presentatiin "Evaluate the paper's formatting, quality figures and tables, and overall visual appeal."
"weaknesses";
cethhhcoooogica_ssssesse "Point cut any potential shortcomings in exporimental design, data analysis, or thecretical reasoning.", seriting_qualitee "Assess the language expression, logical structure, and readability of the paper."
"additiona__Kyy_idddngg
"Provide interpretations of key results based on the charts and data in the thumbnail, avoiding repetition of the abstract.", "Discuss the impact app ication value of these results in the relevant field."
rrcsults_intorprettttee
" "structure_ organization"
srection_layout. "Describe the paper's s section arrangement and content distribution help the B-Group understand the overall framework.", "tigures_and. appendiicess "Mentien important tigures, tables, and appendices, and their roles the paper."
"overall lewpalatiiaalaiiinnneoooe and_ and_suggestions""
"Provide an objective assessment of the paper's overall quality based on the above information.", "publication_sugee" suggessiigsss:: "Suggest whether the paper is suitable for publication or what modifications are needed,
*mpoortant Notes**;
**Strictly Based the Thumbnail All information must originate from the content visible in the thumbnail. Avoid subjective guesses, **Avoid Abstract Repetitione:: Focus en providing details and insights not mentioned in the abstract.
**Maintain Objectivitys:: Provide impartial evaluations without personal bias. **Clear and Specific information*:: Ensure each piece of information clear and verifiable.
Please follow the above guidelines to complete your task and assist the other agents in conducting high-quality review without. access to the paper's visual content.
22

---

Agent Reviewers
Prompt for Meet-Reeiewwer to generate keywords
You are an expert in the field of machine learning. Based en the provided paper content and the following thumbnail information from visien agent, extract expected_count) key topics keywords that best represent the core ideas of the paper. Ensure that these keywords are distinct, relevant, and represent the most critical aspects of the work.
Thumbnai1 Information: (json. dumps (thumbnai 1_info)
If keyword is synenymous with an existing keyword listed below, use the existing one instead of creating a new one: Existing Keywords: (existing keywords_see |
Please provide the keywords in JSON format as list strings, without any additional text or code block indicaters. Example: ("keywordl", "keyword2", "keyword3"]
Here is the paper content rextracted part, may incomplete) <PAPER_BEGIN>
(PAPER_END>
Output the keywords below:
23

---

Agent Reviewers
Prompt for domain-specinf agent(initial review)
You are an AI researcher tasked with reviewing a paper submitted to prestigious machine learning conference. Your objective is to critically evaluate the paper according to the provided guidelines. Be thorough and cauticus in your review. Pay special attentien content and the thumbnail information from a vision agent. Additionally, here is some informatioo ion from previous research papers related to the keyword: { nemory_insertion]
aspects related "[keyword)", Based en the provided paper
Here is the paper you are asked to review (extracted part, may be incomplete) <PAPER_ BEGIN>
(pape__content) <PAPER_END>
Here is the thumbnail information from a vision agent: (json. dumps (thumbnall 1tmbibnall__ffoo info)
Provide only the JSON object with extra formatting (e.g., no ""*"" "s''s""", similar), starting with 'H and ending with }) followed by [[IEND_OF_JJON
II "strengths";
"Clearly list and describe the key contributions innevations made by the paper, with each strength on separate line.", "Provide concrete examples or evidence that highlight the effectiveness of the approach."
"weaknesses"; [
"Identify and any significant limitations or flaws, ensuring that each weakness is articulated on separate linc.", "Mention any concerns regarding scalability, robustness, generalizability, and suggest areas for improvement."
},
@overral_evaluttinn {{
"justification": "Summarize your overall assessment, focusing on the balance between the strengths and weaknesses.", "score": "0-10", "recommendation": "Accept/Reject", }}
}}
**Imoortant:** Your recommendation be either "Accept" "Reject." Options like "Accept with Revisions" are not allowed! Ensure your review reflects a balanced evaluation of strengths and weaknesses.
*otte*** The conference acceptance is 40% please review carefully.
Begin your JSON response here and end with "<IRND_OF_ [
24

---

Agent Reviewers
Prompt for domain-speciin
period)
As an AI reviewer, your task is to revise your reviex focused on the keyword '{ keyword)', You will be provided with your initial review, feedback from other reviewers, relevant memory information about [keyword)', the paper content, and thumbnail information from vision agent, Use these resources to refine and improve your review.
Be1 is your initial review:
jjson. dumps secideeer__eedbacee indent=2) }
Consider the following review from ether reviewers on different aspects of the paper:
[other_foocback)
(memoryinnsertion
Here is the paper text (extracted part, may incomplete) BPAPER_BEGIN>>
Seape__contene <PAPER_END>>
Here is the thumbnail information from a vision agent: (jscn. durps (thumbnai1_inff)
Please revise review taking into account the feedback from other reviewers, the information about [keyword)', and the original paper. Maintain your independent perspective while carefully evaluating and improving your review, addressing any significant issucs.
Provide only the JSON object with extra formatting (e.g., ..t "'''sson", similar), starting with 'H and ending with "))' followed by "<|END JSON|>"
I
"strengths";
"Clearly 1ist and descripe the key contributions innovations made by the paper, with each strength on separate line.", "Provide concrete examples or evidence that highlight the effectiveness of the approach."
"peaknesses";
"Identify and list any significant. limitations or flaws, ensuring that each weakness is articulated on a separate line.", "Mention any concerns regarding scalability, robustness, generallzability. and suggest areas for improvement."
"bverall_evaluation" {{
" "justi fuustification' "Summarize your everall assessment, fecusing on the balance between the strengths and weaknesses.",
"score": "0-10",
"recomeeddation"; "Accept or Reject"
11
}}
*IIporttat::** Your recommendation must be either "Accept" "Reject." " Options like "Accept with Revisions" are not allowed! Ensure your review reflects a balanced evaluation of strengths and weaknesses.
**Note:** The conference acceptance rate is 408, please review carefully.
Begin your JSON response here and end with "<|END_OW OF JSONI>".
25

---

Agent Reviewers
Prompt for AC(final review)
You are an Area Chair (AC) at prestigicus AI conference. Your responsibility is to provide a meta-review for paper that has already been reviewed by multiple AI reviewers with expertise in different fields. Each reviewer has provided their feedback in the form of a structured JSON object. You will receive the paper content and thumbnail information from a visien agent. Your task is to **consider and integrate the insights from these reviews to generate a comprehensive final review**. Knsure that **all reasonable strengths and weaknesses from each reviewer are. covered**!
Here is the paper text (extracted part, may be incomplete) <PAPER_BEGLE BEGIN> (paper_content) <PAPER_ENDS
Here is the thumbnail information from a vision agent: (json. durps (thumbnaii i (amambbaii_info))
Ensure that you consider each reviewer's strengths, weaknesses, and overall evaluation, and make a vell-justified final recommendation. Your final review should be structurea in the same JSON format provided belox.
Here the individual reviews: (review_promptss
Provide only the JSON object with no extra formatting (e.g., no ".**" "ss's""", or similar), starting with 'I' and ending with "))* followed by "<|END OF JSON >";
"strengths";
"Clearly list and describe the key contributions or innevations made by the paper, with each strength on separate line.", "Provide concrete examples evidence that highlight the effectivenee of the approach."
".vEnsure that all reasonable points from each reviewer are covered!
"weaknesses";
" "Identify and list any significant limitations or flaws, ensuring that each weakness is articulated on a separate line.",
"Mention any concerns regarding scalability, robustness, or generalizabilityy and suggest areas for improvement."
"..Ensure that all reasonable points from each reviewer covered! 1,
Doverall_evaluationn !f
" "justification:: "Summarize your everall assessment, focusing the balance between the strengths and weaknesses.", "score": "0-10", "recommendation". "Acoept/Reject" # choose one from them ))
}}
**Important:** Please make sure:
1. Ensure that strengths and weaknesses should cover all reasonable points from each reviewer! Coverage is very important!
2. Again: Ensure that strengths and weaknesses should cover all reasonable points
from each reviewer! Coverage very important!
3. Your recommendation must be either "Accept" or "Reject." Options like "Accept with Revisions" are not alloved!
4. The conference acceptance rate is 40%, please review carefully.
Begin your JSON response here and end with "<|END_OW >F
26

---

Agent Reviewers
D.3. Prompts for SMP initialization
Prompt for keywords extraction
You are expert in artificial intelligence. Based the provided paper content, extract [keywords_num) speci fic and distinct. keywords that best represent the ideas of the paper-avoiding broad terma like "Artificial Intelligence", "Neural Networks", or "Deep Learning"-and provide concise se summary of the paper.
Please format the output in JSON with the following structure with "< END OF ; <|START_OF_JE0N]>
"keywords";
"keyword"; "keyword_
"description" "keyword_1 is xxxxxx."
1),
I "keyword"; "Keyword
"descriptian"; "keyword_2 is xxxxxx." 1),
"summary": "A concise summary the paper's content."
1]
[ENN_OF__SON]>>
Instructions:**
Do not include any additional text cutside the JSON structure.
Ensure that the JSON is properly formatted without any syntax errors.
The descriptions should follow the fermat "keyword is xxx. " and provide an overall intrucuction each keyword from the perspective of the entire field, not limited to the paper.
The summary should be a brief overview capturing the main points of the paper.
Below the article's title, abstract, and introduction: <PAPER_BEGIN> !
[PAPER_ENDS
Output the keywords and summary below (end with LOND__F_JJO0 >>);
27

---

Agent Reviewers
Prompt for keywords aggregation
You are provided with list of artificial intelligence keywords and their descriptions. Determine whether subset of these keywords can be merged into single keyword based their affiliaticn with the same subfield of artificial intelligence.
**Merge Criteria:**
If at least two keywords belong to the same speci fic Al subfield, set 'can_merge" it to 'no'.
to 'yes'; otherwise, "keywords to merge not need to include all keywords.
a list of subsets of keywords that can be merged and does
The 'merged. keyword" must represent the most specific common concept and should not be overly broad (e.g., not "artificial intelligence, "neural networks, or deep learning").
The 'merged_kcyword' should be concise. should fecus solely on the merged_ keyword itself, and does
The parged_descrigg not reference the keywords being merged.
**Input: keywords_ane lacis__nnn__dddedeeee descriptions.. dnserrroannn.sssriitoooo... deeywords_cescripiion
**Ottput:**
structured object with the follewing fields:
If can_merge is "yes":
"can Sean_merge"! "yes",
"merge_infor info": {I
secywords_to_meree [list keywords to mergel, snergged_kywordd "new_keyword",
"a brief intreduction to the merged keyword" F Use the sentence pattern "( (nnerge__kywword) is xxxx"
}}
II san_merge is "no":
"can_merge"; "no", "merge_info"; mull
Begin your output here in json format:
28


Reviews:
[{"review_id": "2iDQqHphhZ", "reviewer": "Reviewer_GyWW", "summary": "This paper introduces Agent Reviewers, a multi-agent system designed to enhance peer review processes using Large Language Models (LLMs). The system comprises domain-specific agents with a shared memory pool (SMP) that enables them to incorporate historical knowledge and multimodal reviewers that assess visual elements of a paper. The authors construct the largest standardized dataset of paper reviews, Reviews-STD, and evaluate their system on ICLR 2024 submissions. The results indicate superior performance compared to existing AI-based review methods, demonstrating improved quality in review comments and acceptance predictions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments.\n\n**Q1**: How is the NeurIPS dataset obtained, given that only the reviews for accepted papers are released?\n\n**A1**: All our data(papers and reviews) stems from the public data on OpenReview, including the NeurIPS dataset. As you noted, 96% of the NeurIPS papers (2016 - 2024) released on OpenReview were accepted. We describe this statistic in Appendix A. Due to the severe imbalance in the NeurIPS data, we do not use it as test data. Instead, we use ICLR 2024 for evaluation.\n\nWe use NeurIPS 2016 - 2023 and ICLR 2017 - 2023 data to initialize the shared memory pool, providing domain-specific reviewers with sufficient  background knowledge for review. Although nearly all NeurIPS dataset papers are accepted, the ICLR dataset has an acceptance rate of about 37%. Consequently, papers in the shared memory pool are relatively evenly balanced in acceptance and rejection.", "claims_and_evidence": "Claims\n\n- Agent Reviewers generates more insightful and diverse reviews compared to existing AI-based review systems.\n\n- The shared memory pool enhances domain-specific knowledge retrieval, leading to improved review accuracy.\n\n- The multimodal reviewer contributes to better assessments by incorporating visual information.\n\nEvidence\n- Quantitative Evaluation: The system is benchmarked against AI Scientist, AgentReview, and LLM Review, demonstrating an 8.5% improvement in F1-score and a 10.5% increase in Jaccard index for strengths-weaknesses alignment.\n- Decision Analysis: The system achieves a 35.7% improvement in decision F1-score and a 78.9% increase in MCC over existing methods.\n- Ablation Studies: Removing the multi-agent discussion or shared memory pool leads to a decline in review accuracy, confirming their effectiveness.\n- Case Studies: Examples show how the system provides richer, context-aware critiques by referencing prior research.", "methods_and_evaluation_criteria": "System Architecture:\n- Multi-Agent Interaction (MI): Different agents collaborate in reviewing, discussion, and final decision-making.\n- Shared Memory Pool (SMP): Enables domain-specific knowledge retrieval from prior papers.\n- Multimodal Reviewer (MA): Evaluates visual aspects like figures and layout.", "theoretical_claims": "This is an application paper.", "experimental_designs_or_analyses": "- Benchmarking Against Existing Systems: Compared to AI Scientist, AgentReview, and LLM Review on ICLR 2024 papers.\n- Ablation Studies: Tested the impact of removing multi-agent collaboration, shared memory, and multimodal components.\n- Impact of Shared Memory Initialization: Evaluated how the cutoff year of included papers affects review quality.", "relation_to_broader_scientific_literature": "This work builds on / is relevant to AI-assisted peer review, Multi-agent systems for research automation (AI Scientist, CycleReviewer). Multimodal AI models incorporating textual and visual data.", "essential_references_not_discussed": "The related work is comprehensive", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "How is the NeurIPS dataset obtained, given that only the reviews for accepted papers are released?", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hoDLj7T72q", "reviewer": "Reviewer_pd6u", "summary": "This paper introduces a multi-agent system that enhances automated peer review. It mimics human review processes by employing domain-specific agents, a multimodal reviewer for visual analysis, and a shared memory pool (SMP) that retrieves past paper reviews for informed evaluation. The system also introduces *Reviews-STD*, a standardized dataset of paper reviews from ICLR and NeurIPS, formatted into strengths, weaknesses, and decisions. Tested on 300 ICLR 2024 papers, *Agent Reviewers* outperforms existing AI-based review systems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and insightful comments.\n\n**Q1**:What kind of LLMs are used in the baseline methods in Table 2? \n\n**A1**:For the main experiment, all baselines and our Agent Reviewers use GPT-4o-mini for fair comparison (see Table 2 header). Other baseline settings like hyperparameters follow their defaults (Line 307, and detailed in Appendix B.2.). Thanks for your question; we'll clarify in the main text.\n\n**Q2**:Concerns about potential data leakage of Gemini and Deepseek v3.\n\n**A2**:We selected the latest ICLR 2024 data as the test set, taking measures to avoid data leakage. In main experiments, we used GPT-4o-mini (knowledge cutoff: Oct. 2023, no leakage). Gemini-exp-1206 (cutoff: Dec. 2023, no leakage) and Deepseek-V3 (cutoff: Jun. 2024, leakage not guaranteed despite efforts) were used only for cross-LLM generalization verification. The risk is explained in Appendix B.4.\n\nWe contend Deepseek-V3 has no serious evaluation-affecting leakage due to:1)Overall performance is inferior to GPT-4o-mini(no leakage).2)When asked to detail papers by title, hallucinations are severe.3)Table 12 shows Agent Reviewers outperforms Single Agent with Deepseek-V3, indicating our method value.\n\n**Q3**:Background knowledge domains and coverage.\n\n**A3**:We currently focus on review in AI. Background knowledge comes from LLMs' inherent knowledge and the shared memory pool(SMP), which has 283728 AI papers and reviews from ICLR 2017-2023 and NeurIPS 2016-2023 on OpenReview, spanning nearly all AI fields. Notably, our training-free method can be applied to other fields by expanding SMP.\n\n**Q4**:Open-source model's performance.\n\n**A4**:Thanks for your suggestion. We've added comparisons with more open-source models (Deepseek-V3-0324, Qwen2.5-VL-72B-Instruct, InternVL-2.5-78B) in the [link](https://imgdrop.io/image/2iYyt).\n\nOverall, GPT-4o-mini performs best, Deepseek-V3 close behind. Largest open-source model approaches or surpasses some proprietary ones. Little difference exists between 2 Deepseek-V3 versions.\n\n**Q5**:Diversity concern & Proposal for different LLMs in different role experiments.\n\n**A5**:We agree on the importance of diversity in review, tried your suggestion with GPT-4o-mini, Gemini-exp-1206, and Deepseek-V3-0324 as LLMs for 3 domain-specific reviewers. Results at [link](https://imgdrop.io/image/2iI9B).\n\nWe've seen no better results with a multi-LLM system. We analyze that our system has already enhanced review perspective diversity by endowing domain-specific reviewers with different domain knowledge, so multi-LLMs have limited room for improvement.\n\n**Q6**:Concerns about low-quality review impact in training data.\n\n**A6**:Our method is training-free. We initialized the SMP with ICLR 17-23 and NeurIPS 16-23 papers, and retrieved paper info(summaries, processed reviews) as domain-specific background knowledge for reviewers.\nWe agree low-quality reviews can affect performance and mitigate it as follows: 1) Processed reviews in SMP, retaining AC decisions and LLM-aggregated  pros&cons to reduce individual low-quality review impact.2) Retrieved 5 domain-related papers for each reviewers as memory to ensure sufficient high-quality reviews.\n\n**Q7**:Why focus on title+abstr.+intro.& Plan to extend to full paper content.\n\n**A7**:We use only title+abstr.+intro. in our current method for two reasons: 1) The full text being about 12 times longer, using it in a multi-agent system with interactions incurs great overhead.2)They're crucial in peer review, even for humans. We agree full text can provide richer info and potentially better performance, but challenges like long context need to be overcome. To extend efficiently, we suggest adding an in-depth reading feature for reviewers. After reading first three parts, they can raise questions and retrieve relevant full-text parts for further analysis before generating comments.\n\n**Q8**:Insights into why system identifies paper weaknesses less effectively.\n\n**A8**:Thanks for noting this profound observation. We observed it in all tested  methods. We posit it's because reviewers' proposed strengths(str.) maybe similar, e.g., similar to claimed contributions, while weaknesses(wk.) vary greatly, making it harder for LLM-generated weaknesses to match humans'.\n\nWe studied the similarity of Agent Reviewers' str. and wk. generated by GPT-4o-mini, Gemini, and Deepseek-V3 for the same article. On average, GPT-4o-mini vs. Gemini: str. 0.664/wk. 0.484; GPT-4o-mini vs. Deepseek-V3: str. 0.738/wk. 0.663. Figure 6 shows most matching str. in \"Novelty\" & \"Research Implications\", wk. dispersed.\n\n**Q9**:Suggestion about  real-time online search and memory pool expanding.\n\n**A9**:Thanks for your comments! We agree and will try these two approaches in future work to offer reviewers more comprehensive knowledge.\n\n**Q10**:Plans for code and data availability.\n\n**A10**:We'll open-source all code and data (include memory pool) upon acceptance for reproducibility and are preparing for it.", "claims_and_evidence": "The proposed system shows better results compared to baselines but the setting of the experiments (see below) and some questions (see below) need to be clarified to better review this paper.", "methods_and_evaluation_criteria": "Please see questions below.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "I have thoroughly reviewed the experimental designs and identified several critical questions that need to be addressed:\n\nFirst, there are concerns about the open-source model's performance metrics and how they compare to proprietary models. The paper would benefit from a more detailed analysis of these performance comparisons.\n\nSecond, the authors should clarify their plans for code and data availability. Making these resources open-source would greatly enhance reproducibility and allow the broader research community to build upon this work.\n\nThird, there are potential data leakage concerns regarding the use of Gemini and Deepseek v3 models. The authors should address how they prevented any training data overlap between these models and their evaluation dataset.", "relation_to_broader_scientific_literature": "This paper is related to the agent system for automatic paper reviews. The general idea is to develop automatic agent systems to review papers.", "essential_references_not_discussed": "The related work has discussed essential references", "other_strengths_and_weaknesses": "The motivation and research question is important for this field and the proposed system seems promising but there are still quite a few questions need to be addressed for further review.", "other_comments_or_suggestions": "To strengthen the system's capabilities, I suggest implementing real-time online search functionality rather than relying exclusively on retrieval-augmented generation (RAG). This would allow the system to access the most current research and developments in the field.\n\nAdditionally, the system would benefit from expanding its memory pool to include papers from a broader range of conferences and journals beyond ICLR and NeurIPS. This expansion, combined with real-time search capabilities, would provide more comprehensive and up-to-date context for paper evaluations.", "questions_for_authors": "I have several important questions for the authors that would help clarify key aspects of the system and potentially influence my evaluation:\n\nFirst, I would like to understand the scope of background knowledge incorporated into the system. Which specific academic fields and domains are currently covered, and how comprehensive is this coverage?\n\nSecond, regarding data quality considerations, could you elaborate on how the presence of low-quality reviews in the training dataset affects the system's overall performance and reliability?\n\nThird, I have concerns about the current architecture using a single LLM for multiple agent roles. This approach may limit the diversity of perspectives and capabilities. Have you considered evaluating the system's performance with different specialized LLMs assigned to different reviewer roles?\n\nFourth, the disparity between Strength and Weakness scores is notable. Could you provide insights into why the system appears to be less effective at identifying weaknesses compared to strengths in papers?\n\nFifth, what kind of LLMs are used in the baseline methods in Table 2? Line 307 indicates that these baselines follow their default configurations but it is unclear what LLM they are using. \n\nFinally, the current analysis seems to focus primarily on titles, abstracts, and introductions (line 281). Could you explain this limitation and discuss any plans to extend the analysis to include the full content of papers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TZi3zM9bbu", "reviewer": "Reviewer_FTae", "summary": "This paper proposes a multi-agent system to simulate the review process of research papers, called Agent Reviewers. It is equipped with multi-agent interaction, shared memory pool, and multimodal agent. It empowers agent reviewers with observations not only on textual content but also on visual content. The experiments have shown an improvement compared with previous works. This work can benefit the research community, especially by providing more chances to polish their work for all researchers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments.\n\n**Q1**: Suggestion of put the explanation of metrics in the main text of this paper.\n\n**A1**: Thank you for your suggestion. We agree that a clear explanation of metrics is crucial for understanding the paper. However, due to the page limit of the main text in ICML, we opted for a trade-off by providing qualitative descriptions in the main text and detailed explanations in Appendix B.1. We plan to provide a more detailed explanation of the metrics in the main text in a future version, which will be updated on arXiv after acceptance.\n\n**Q2**: Are the results different among various conferences? For example, ICLR and NeurIPS. \n\n**A2**:\nWe have supplemented the results for ICLR 2024 and NeurIPS 2024. For each conference, 100 papers were randomly sampled for evaluation, and the default LLM GPT-4o-mini was used. The results are shown in the table below. Columns from Recall(S&W) to Jaccard(S&W) represent the analysis of strengths and weaknesses, while columns from F1(Dec.) to G-mean(Dec.) represent the analysis of decisions. \"S&W\" refers to strengths and weaknesses, and \"Dec.\" stands for decisions. All metrics are better when larger.\n|Conference|Method|Recall(S&W)|F1(S&W)|MaxSim(S&W)|Jaccard(S&W)|F1(Dec.)|MCC(Dec.)|Bal.A(Dec.)|G-mean(Dec.)|\n|:-|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ICLR 2024|Single Agent|0.347|0.416|0.442|0.270|0.469|0.219|0.609|0.593|\n|ICLR 2024|Agent Reviewers|0.409↑|0.453↑|0.476↑|0.307↑|0.605↑|0.385↑|0.705↑|0.705↑|\n|NeurIPS 2024|Single Agent|0.357|0.440|0.444|0.299|0.349|0.039|0.525|0.424|\n|NeurIPS 2024|Agent Reviewers|0.418↑|0.461↑|0.476↑|0.319↑|0.589↑|0.120↑|0.591↑|0.569↑|\n\nICLR 2024 and NeurIPS 2024 produce similar outcomes, where Agent Reviewers outperform the Single Agent baseline, achieving an average relative improvement of  **23.5%**  on ICLR and  **44.7%**  on NeurIPS, thereby demonstrating consistency.\n\n**Q3**:Are the results different among various years? For example, ICLR 2024 and ICLR 2023.\n\n**A3**:\nWe have supplemented the results for ICLR 2023, and ICLR 2024. For each conference, 100 papers were randomly sampled for evaluation, and the default LLM GPT-4o-mini was used. The cutoff year for the shared memory pool (SMP) was set to one year prior to the test data year, to avoid data leakage from SMP. The results are shown in the table below. Columns from Recall(S&W) to Jaccard(S&W) represent the analysis of strengths and weaknesses, while columns from F1(Dec.) to G-mean(Dec.) represent the analysis of decisions. \"S&W\" refers to strengths and weaknesses, and \"Dec.\" stands for decisions. All metrics are better when larger.\n|Conference|Method|Recall(S&W)|F1(S&W)|MaxSim(S&W)|Jaccard(S&W)|F1(Dec.)|MCC(Dec.)|Bal.A(Dec.)|G-mean(Dec.)|\n|:-|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ICLR 2024|Single Agent|0.347|0.416|0.442|0.270|0.469|0.219|0.609|0.593|\n|ICLR 2024|Agent Reviewers|0.409↑|0.453↑|0.476↑|0.307↑|0.605↑|0.385↑|0.705↑|0.705↑|\n|ICLR 2023|Single Agent|0.334|0.408|0.432|0.271|0.474|0.169|0.580|0.554|\n|ICLR 2023|Agent Rreviewers|0.390↑|0.439↑|0.467↑|0.298↑|0.538↑|0.139↓|0.570↓|0.570↑|\n\nWhether in 2023 or 2024, Agent Reviewers have shown better results than the single-agent baseline, demonstrating the consistency of the gains of our method. It should be noted that for ICLR 2023, due to the earlier cutoff year of the SMP, the number of papers in the SMP decreased from 28,372 to 21,156 (a 25% relative decrease), which may cause the performance decline in 2023.", "claims_and_evidence": "In the experiment section, the author presents abundant results to show the effectiveness of their works. I think they can verify the claims that the author has proposed.", "methods_and_evaluation_criteria": "The demonstration of the proposed method is clear. It designs multiple roles to simulate different reviewers during various stages. The review process is reasonable as well. The Share Memory Pool can further enhance the capability to provide responses of higher quality. The evaluation metrics are reasonable but should be further discussed in the main part of this paper.", "theoretical_claims": "There seem to be no theoretical claims.", "experimental_designs_or_analyses": "The experimental designs are reasonable, with major experimental results and ablation studies. These experiments verify the claims proposed in Section 1, such as the capabilities of multimodality and SMP.", "relation_to_broader_scientific_literature": "Yes, this paper can contribute to the research community.", "essential_references_not_discussed": "There seem to be no essential references that are not discussed.", "other_strengths_and_weaknesses": "Please see above.", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "- Are the results different among various conferences? For example, ICLR and NeurIPS.\n- Are the results different among various years? For example, ICLR 2024 and ICLR 2023.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Agent Reviewers, a multimodal multi‑agent framework that aims to automates scholarly peer review. A large corpus of historical reviews populates a shared memory pool that agents query while collaboratively critiquing both the textual and visual content of a manuscript. The system outperforms prior AI review tools on strengths/weakness alignment and decision prediction; ablation studies confirm the importance of multi‑agent discussion, the memory pool, and the multimodal agent.

The reviewers agree the paper tackles an important practical problem. The constructed peer-review dataset can be useful to the research community. The proposed method is clear. The experiments are comprehensive: comparisons with three baselines (AI Scientist, AgentReview, and LLM Review), ablations, cross‑LLM generalization, and new results added during rebuttal for venues of NeurIPS 2024 and ICLR 2023/24.

The original submission lacked clarity regarding evaluation metrics, dataset sourcing, data leakage safeguards, and the choice of baseline LLM. It seems the rebuttal addressed most of these points satisfactorily.
One major issue is the analysis currently focuses only on titles, abstracts, and introductions sections of the paper, which does not fully reflect the real-world peer-review process. The authors explain this limitation is due to challenges in handling long contexts, and defer the full-paper analysis and real-time retrieval to future work. The system's weaker coverage of weaknesses versus strengths remains an open issue – the authors should discuss this in the future revisions.
Decision: Accept (poster)