From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review
Authors: Yaohui Zhang
Haijing ZHANG
Wenlong Ji
Tianyu Hua
Nick Haber
Hancheng Cao
Weixin Liang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Peer Review Redesign
Comparative Paper Evaluation
Primary area: other
TL;DR: We propose a novel LLM-based peer review system that evaluates academic papers through pairwise comparisons.

Abstract: The advent of large language models (LLMs) offers unprecedented opportunities to reimagine peer review beyond the constraints of traditional workflows.
Despite these opportunities, prior efforts have largely focused on replicating traditional review workflows with LLMs serving as direct substitutes for human reviewers, while limited attention has been given to exploring new paradigms that fundamentally rethink how LLMs can participate in the academic review process.
In this paper, we introduce and explore a novel mechanism that employs LLM agents to perform pairwise comparisons among manuscripts instead of individual scoring. By aggregating outcomes from substantial pairwise evaluations, this approach enables a more accurate and robust measure of relative manuscript quality.
Our experiments demonstrate that this comparative approach significantly outperforms traditional rating-based methods in identifying high-impact papers. However, our analysis also reveals emergent biases in the selection process, notably a reduced novelty in research topics and an increased institutional imbalance. These findings highlight both the transformative potential of rethinking peer review with LLMs and critical challenges that future systems must address to ensure equity and diversity.

From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review
Yaohui Zhang'
Haijing Zhang'
Wenlong Ji'
Tianyu Hua Nick Haber!
Hancheng Cao² Emmor University
Weixin Liang?
1 Stanford University
Abstract
The advent of large language models (LLMs) offers unprecedented opportunities to reimagine peer review beyond the constraints of traditional workflows. Despite these opportunities, prior efforts have largely focused replicating traditional review workflows with LLMs serving as direct substitutes for human reviewers, while limited attention has been given exploring new paradigms that fundamen- tally rethink how LLMs can participate in the academic review process. In this paper, we introduce and explore a novel mechanism that employs LLM agents perform pairwise comparisons among manuscripts instead of individual scoring. By aggregating outcomes from substantial pairwise evaluations, this approach enables more accurate and robust measure of relative manuscript quality. Our experiments demonstrate that this comparative approach significantly outperforms traditional rating- based methods in identifying high-impact papers. However, our analysis also reveals emergent biases in the selection process, notably reduced novelty research topics and an increased institutional imbalance. These findings highlight both the transformative potential of rethinking peer review with LLMs and critical challenges that future systems must address to ensure equity and diversity.
Introduction
The peer review system serves as a cornerstone for validating and disseminating scientific ideas [2]. Yet, despite its widespread adoption, the peer review system has become increasingly strained because of the rapid growth of research output and a persistent shortage of qualified reviewers [13, 30, 40].
Recent advances in large language models (LLMs) [1, 16, 37] have sparked growing interest in using Al assist even automate parts the review process [17, 22, 39]. Unlike human reviewers, whose availability and productivity are inherently constrained, LLM- based agents offer the potential for greater scalability and consistency in academic evaluation.
However, existing LLM-based paper review efforts [18, 27, 49, 55, 61] largely mirror traditional workflows: For each paper, multiple L.MM agents first independently generate structured reviews (e.g., summary, strengths, weaknesses, suggestions for improvement, numeric ratings). Subsequently. an LLM meta-reviewer synthesizes these individual LLM reviews and composes cohesive meta-review summarizing the collective assessment. Finally, a decision (e.g., accept or reject) is generated either directly by this LLM meta-reviewer agent based its synthesized ratings.
While LLMs offer new possibilities for peer review, simply replicating existing workflows may overlook deeper opportunity. Historical analysis suggests that the conventional structure of peer review was not carefully engineered for optimal evaluation, but rather emerged as a pragmatic response to resource constraints. Studying the development of peer review in sociology, Merriman [32] observes that "the evolution of peer review is best understood as the product of continuous efforts steward editors' scarce attention while preserving open submission policy that favors authors' interests."
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

This perspective highlights that many features of today's 's review systems reflect historical compro- mises rather than principled design choices. As scholarly publishing expands globally and across disciplines [10], these inherited structures have come under increasing scrutiny. Issues like bias [20, 44, 46] and noisy ratings [28] have spurred debates on how to improve or reform peer review [14, 45, 48, 52].
Given their exceptional scalability, LLMs present an opportunity merely to automate existing practices, but to fundamentally rethink the design of manuscript evaluation systems. Rather than replicating workflows optimized for human reviewers, it is crucial to rethink the foundations of effective evaluation and to design Al systems that complement and address the persistent limitations of traditional peer review.
Traditional Peer Review Process
Submissions
Reviews
Meta Review
Average Rating
Summary
Decision
In thüs papper..
Paper
Review
Review
Review
Average Rating
Summary
Decision
In t&á papper..
Paper
Review
Review
Average Rating
Summary
Decision
popwer...
REJECT
Review
Review
Review
Ours Submissions
Pairwise Comparison
Leaderboard
Pepar
Paper N-2
Peper
Paper N
Paper
Decision
Paper
Paper
Paper #D57
4.85
Accept
Poper #1124
4,23
Accept
Paperi
2236
Paper ND45
-2.18
Reject
Paper N-2
2
Paper
N-1
7150
Paper 8362
-3445
Reject
Paper N!
1: Win
D: Lose
7: Not Compared
Figure 1: Comparison of the conventional peer review process (top) and the proposed pairwise evaluation ranking system. The traditional paradigm (followed by both human reviewers and existing LLM-based review systems) works on paper-level assessment, where each manuscript is first evaluated independently by multiple reviewers who provide detailed comments and scores. These evaluations are integrated by a meta-reviewer to recommend a final decision. Our method adopts pairwise comparisons between papers: each agent is randomly assigned pair of submissions to evaluate. These comparative judgments are then aggregated using the Bradley-Terie -erry model, where the resulting BT coefficients serve scoring functions produce global ranking of all submissions, Final decisions are derived from the aggregated rankings, enabling more objective assessment of the relative quality across the entire submission pool.
In this work, we propose shift from replication to redesign. We focus on the decision-making layer of peer review-speciffially mechanisms that can rank submissions and support acceptance decisions. Our approach is intended to complement, not replace, the current peer review process. Specifically, we introduced and explored a novel mechanism that engages L.IMM agents in pairwise comparisons for more effective manuscript assessment. Rather than assigning absolute scores to

---

individual papers, our approach directly contrasts pairs of submissions to determine which one better meets the multidimensional evaluation criteria. Each LLM agent is tasked with evaluating two manuscripts at a time, and by integrating the results from numerous pairwise comparisons, we can construct a robust overall assessment that more accurately reflects relative quality.
Our results indicate that with proper scaling, this pairwise mechanism demonstrates potential to identify papers of higher academic impact (as measured by future citation count), significantly outperforming paper-level rating approach. However, our analysis also reveals important limitations: Papers preferred by our LLM- based pairwise ranking system tend exhibit lower topic novelty and greater concentration among limited number of high-prestige institutions. While these biases highlight critical challenges for the deployment of LLM-based review systems, we view this work as a first step toward exploring fundamentally new paradigms for scalable, equitable, and robust research evaluation.
is important to note that while we use proxy metrics-such as future citation counts evaluate our proposed mechanism, this represents an exploratory and pragmatic step rather than an ideal. The gold standard would be direct human evaluation of how well a mechanism supports the selection and improvement of high-quality research, but such evaluations are costly and difficult scale [43]. Proxy metrics offer a tractable, though imperfect, way to approximate long-term impact and anticipate how different mechanisms might perform. Ideally, a strong altemnatvee review mechanism should correlate with but not fully replicate- the outcomes of the current system, offering complementary perspectives and surfacing different strengths. As we show, our proposed pairwise comparison mechanism achieves this balance, aligning with current human judgments while introducing useful differentiation. Crucially, our goal is not to mimic existing outcomes, but to design mechanisms that add value.
Moreover, we believe the strength of human-led peer review lies not only in final decisions, but also in the human interactions it enables-feedback, discussion, and iterative refinement- help authors strengthen their work. Through these interactions, reviewers and authors also engage in shared evaluative practice that reinforces norms, builds trust, and helps individuals identify with the scholarly community [63, 19]. Our intention not to automate or replace this process, but to broaden how we understand and support scholarly evaluation.
2 Related Work
Challenges in Peer Review Peer review, while essential to scientific progress, faces numerous challenges in today's s academic landscape. Reviewers often struggle with overwhelming workloads, further compounded by the absence of meaningful incentives and the inequitable distribution of review requests [7, 31]. In addition, implicit biases can influence evaluation [46, 47, 50), with studies showing that factors like author affiliation or gender may affect acceptance rates [59]. As research output continues to accelerate globally [5]. these systemic challenges demand innovative solutions to preserve the integrity and effecttiveness of scientific evaluation. Earlier works [48, 57, 56] introduce pairwise comparisons by incorporating author-provided rankings to calibrate noisy reviewer scores. In contrast, we fully replace paper-level scores with large-scal pairwise comparisons performed by LLM agents, without relying on human ratings or author input.
LLMs for Peer Review Recent advancements in Large Language Models (LLMs) have sparked considerable interest in their potential to transform academic peer review processes [21, 62, 24, 23). While existing studies [9, 22, 38) have demonstrated that L.MMs can provide valuable feedback for research papers, exhibiting substantial overlap with reviews written by human reviewers, they often struggle with assigning reasonable ratings and making sound decisions, which raises concerns about their reliability for evaluative tasks [26, 58]. For instance, Niu al. [35] showed that ChatGPT tends to grant acceptances: out of 1558 full paper evaluations, ChatGPT suggested 1243 acceptances compared to only 315 rejections, resulting in an acceptance rate of approximately 79.8%. Similarly, Zhou et al. [60] highlighted that LLMs could generate reasonable aspect scores (e.g. recommendation, soundness, originality) from human reviews but failed when given only the research paper. Lu et al. [27] improved the base LLM's decision-making process through prompting techniques such as self-reflection [42], few-shot examples [54] and response ensembling [53]. CycleReviewer [55] and DeepReview [61] enhanced LLM-based paper review via fine-tuning on high-quality review dataset.
3

---

In contrast, Less effort is made to explore new paradigms that fundamentally rethink how LLMs can participate in the academic review process.
3 Method
3.1 Framework Overview
We now describe our peer review mechanism (Figure 1) that transforms the traditional paper-level assessment process into a comparative ranking system through pairwise judgments. This approach builds on key insight that evaluators are generally more reliable when deciding which of two items is better than when assigning each an absolute score [8]. By integrating large number of pairwise judgments into a global ranking with the Bradley-Terry model, we could potentially mitigate the calibration inconsistencies and personal rating biases that plague conventional peer review systems.
The mechanism operates in three key stages: First, we collect pairwise paper comparisons from multiple LLM agents, where each agent evaluates a randomly sampled pair of papers and produces binary preference judgment. Second, we apply the Bradley-Terry model quantify each paper's relative quality through BT coefficients. Finally, we determine the optimal coefficients through maximum likelihood estimation and derive complete ranking of all papers.
3.2 From Pairwise Comparisons to Rankings
Suppose we have paper candidates indexed by E [N] and M LLM agents participating in the evaluation process. We aim to establish quality-based ranking through pairwise comparisons. The framework consists of the following steps:
1. Pairwise Comparison: Let S {(i,)) j and i,j [N]] be all possible pairs of papers to compare. For each of M agents, we randomly assign one pair (v,j) E S to evaluate. Each agent then analyzes both papers and produces binary preference judgment Yy {0, 1), where Iky = 1 indicates that the agent prefers paper i over paper j. and Yı = 0 indicates the opposite preference. The full prompt used for pairwise judgment is provided in Supp Figure 7. Denote the collection of all chosen pairs as A, where |.A| = M. We collect all comparison results into observation set O {(i, J. Biy) (i E A). which contains triplets of the compared papers and their corresponding preference judgments from all agents. Since the size of [S] grows quadratically with
only afford to assess a small fraction of all possible pairs, and hence it is challenging to recover the ranking under this scenario. Later in Section 4.2, we empirically examined how ranking quality scales with the number of LLM agents. The results reveal a clear scaling law, and we are able to recover a high-quality ranking with less than 2% samples of all possible pairs."
2. Bradley-Ters Terry Model: We employ the Bradley-Tery model [6] to recover the paper ranking from pairwise comparison results. The model provides stable statistical estimation from sparse comparisons, which is particularly suitable for our scenario. Specifically, it assumes each paper has an underlying quality score B3 IR and the outcome of comparing papers and j is Bernoulli random variable, where the probability of paper beating is determined by logistic function of the score difference B, - By. i.e.
e
Pij = P(y; = ((( e + e 1 1
(I)
estimate these scores based on the outcome of observed pairwise comparisons, we utilize the maximum likelihood estimator on the Bradley-Terry model, which maximizes the log- likeeiid function below:
CBB
Σ
(uey log(p;)) + yıÿ) log(1 -Pt))]
(K.-.))E
3. Ranking Inference: Once the scores are estimated, they can be used to rank all candidates in descending order, with a higher score indicating "stronger" candidates with higher quality.
'Theorem 4 in [34] shows that m > 12n log n pairs are sampled uniformly at random, then with high
probability the estimate â satisfies ||0 0" O
log

---

Since all our experiments were conducted using GPT-4o mini [36] for its balance of performance and computational cost, we will refer our proposed mechanism as the GPT ranking system in the following sections.
Experiments
We now empirically validate whether proposed pairwise-based ranking framework better identifies high-impact papers compared to conventional rating-basse approaches.
4.1 Dataset Desciption
We collected papers from major ML conferences publicly available OpenReview, including ICLR, NeurlPS, CoRL, and EMNLP. Papers were categorized into different decision outcomes (e.g., accepted vs. rejected, main findings track, oral poster, etc.), reflecting the quality assessments by the peer review process. The paper PDFs and corresponding decisions were retrieved using the OpenReview API (https: //t:://doccc openreview net/). We used ScienceBeam [12] extract the title, abstract, figure and table captions, and the main text. For each conference, we applied our method the submitted papers and maintained the original distribution of papers across decision categories. Experiment setup details are available in the Appendix B.
4.2 Agent Scaling Boosts the Performance of GPT Ranking System
25
Average Number Citations for accepted papers
GPT Ranking System without Area Control
GPT Ranking System with Area Control
Human Accept Baseline
Randomly Accept Baseline
GPT Rating System Baseline
10
10
10
10a
10a
Number of Agents
Figure 2: Scaling of average citation counts for accepted papers by GPT ranking system with increasing pairwise comparisons. We introduce two variants of our GPT ranking system: a basic version without area control, and one with area control, which ensures that the distribution of accepted papers across primary research areas matches that of human peer review. These two variants are compared against three baselines: (1) human acceptance decisions, (2) random acceptance based on the conference acceptance rate, and (3) GPT rating system (acceptance depends on the rating synthesized by a meta-reviewer from three independent GPT reviewers). As the number of comparisons increases, both variants steadily improve and approach human-leve performance. At maximum scale, the GPT ranking system without area control achieves 20.00 average citations, while the version with area control reaches 18.97 average citations. These compare favorably to human acceptance decisions (19.36 average citations), and significantly outperform both random acceptance (8.56 average citations) and the GPT rating system baseline (11.41 average citations).

---

We investigate the scaling performance of our GPT ranking system with the number of agents, where each agent conducts a single pairwise comparison between randomly sampled papers from ICLR 2024. Figure 2 shows the overall academic impact (measured by their average citation counts) of the accepted papers as the number of agents increases. In particular, the system exhibits a steady increasing trend in average citations as the number of agents scales from around 109 to 10 At insufficient sample sizes (e.g., 10 agents), the system performance approaches that of random acceptance. Since each paper receives too few comparisons, it fails to establish reliable relative rankings across the entire pool of 7158 papers. As the scale increases beyond 10 the growth rate gradually slows down, with the average citation count eventually plateauing at approximately 20.
Given that the GPT ranking system demonstrates different acceptance patterns across research areas compared to humans (Section 4.5), we include two variants in our evaluation: one with area control, which maintains the same distribution of accepted papers across primary research areas as the human peer review system, and one without such controls. Nevertheless, they all yield comparable results to human-accepted papers at the convergence point. In the subsequent analysis, we focus on the GPT ranking system without area control it better reflects the true properties of our system without deliberate controls.
We also compare our GPT ranking system with another baseline that follows a similar setup as prior works [27, 18]. This baseline, called the GPT rating system, involves three GPT reviewers independently generating reviews for each paper. Another agent then acts as meta-reviewer to synthesize these individual reviews and provide a final rating (on a scale of 10). Acceptance decisions are made based on the rating provided by the meta-reviewer. However, it revealed a critical limitation: GPT--generated ratings exhibited less diversity, with most ratings concentrated around and 7. Consequently, the system's 's ability to differentiate papers of varying quality was significantly constrained, leading to only minor improvements over the random baseline.
4.3 Discriminative Capability of GPT ranking system in Research Evaluation
With sufficient scale, our GPT ranking system approaches human-level performance in terms of average citation counts for accepted papers. However, beyond raw citation averages, a key question remains: does the GPT ranking system effectively distinguish between highly influential and less impactful papers in a manner comparable to human reviewers? To address this question, we analyzed the system's ability discriminate papers across the entire citation distribution spectrum.
Across multiple top AI conferences (ICLR 2024, EMNLP 2023, ICLR 2023, and CoRL 2023) and under various decision conditions (accepted vs. rejected, main vs. findings track, oral vs. poster, etc.), papers ranked highly by our system consistently received more citations than those ranked lower (Figure 3, Supp Figures 8, 9). This pattern closely mirrors the citation advantages observed in human- selected papers across the same decision categories. The statistical significance these differences (indicated by asterisks) further validates that our GPT ranking system proxies for human peer review when identifying work likely to generate greater impact in the research community. The consistency of this pattern across different conferences and years suggests that as the system employs sufficient agents to conduct comprehensive pairwise comparisons, it could also capture the subtle quality signals that correlate with future scholarly influence.
as reliable
4.4 Consistency Analysis between GPT Ranking System and Human Peer Review
We further examined the decision consistency between the GPT ranking system and conventional human peer review in ICLR 2024 and ICLR 2023. As shown in Table I, each paper was independently categorized by human reviewers and by the GPT ranking system into one of four decision categories Oral, Spotlight, Poster, or Reject. Papers withdrawn after review releases were considered as rejected for this analysis.
Notably, 41.0% of the papers accepted by human reviewers in ICLR 2024 were also accepted by the GPT ranking system (42.2% for papers accepted in ICLR 2023; Supp Table 4). The result aligns with findings from the NewrIPS 202/ consistency experiment [3], where 48.0% of papers accepted by the first committee were also accepted by the second committee. This level of agreement between Al and human reviewers roughly on par with the consistency observed between independent human review committees, suggesting that the GPT ranking system could offer valuable insights for identifying

---

50
50
50
Meen Citation
Mean Citation
Meant Citation
40
Citation
20
20
10
10-
T-acooppled
GPT-rejacted
Human-accsspped
Humen-re/ucodd
GPT-main
GPT-inding
Human-main
Human-findinge
ICLR 2023
ICLR 2023
CoRL 2023
CoRL 2023
Mean Citalion
Su
Mean Citation
Mean Citation
Mean Citation
10-
92
125-
30
32
100
2n
75
13
26-
%
OPT-sctapled
@PT-rejeaed
Humann Humon-accoppa accopted
Humen-rejjcci rejocld
GFT-oral
GPT-poder
Human-oral orsi
Human-padher
Figure 3: Comparison of mean citation counts across multiple AI conferences under two- fold decisions (e.g., accepted vs. rejected, main vs. findings track, oral vs. poster). The results consistently show that higher-tier papers (both GPT-selected and human-selected) receive substantially higher citation counts than lower-tier papers. This implies that our GPT ranking system effectively distinguish more influential papers, performing comparably to human peer review system in identifying works likely to generate greater impact in the academic community. Error bars represent 95% confidence intervals. #p < 0.05, **p < 0.01, ***p < 0.001, and s***p < 0.0001.
Table I: Summary of recommendations for ICLR 2024 papers by two review systems: the human peer review (Human) and the GPT ranking system (GPT), Each row represents humans' decision, while each column shows how the GPT ranking system categorized the same papers.
Human\GPT
Oral
Spotlight
Poster
Reject
Oral
11
28
41
Spotlight
10
32
130
191
Poster
29
116
556
1,085
Reject
41
204
1,072
3,606
potentially impactful papers that might otherwise be rejected due to the inherent randomness in the traditional peer review process.
4.5 Different Acceptance Patterns across Research Areas
We found notable disparities between the GPT ranking system and human peer review in their acceptance rates of several research areas (Figure 4, Supp Figure 10): GPT ranking system shows significantly higher acceptance rates for applicd research areas such as robotics/autonomy applications (0.56 vs. 0.32) and societal considerations (0.51 vs. 0.30). In contrast, theoretical areas that received relatively high acceptance rates from human reviewers, such as learning theory (0.50 vs. 0.12) and optimization (0.31 vs. 0.12), had much lower acceptance rates from the GPT ranking system.
There are several ways to interpret these findings. First, LLMs are trained on massive web-crawled corpora rich practical, application-oriented content. Consequently, these models may exhibit an inherent preference for studies that demonstrate immediate real-world applicability. Alternatively, large language models often struggle with complex mathematical reasoning [33], which could hinder their understanding of papers that require deep theoretical and mathematical rigor beyond memorized
7

---

applications robotics.
neurosymbelic
societal considerations Including fairness, safety, privacy
635
0.31
causal reasoning
generative
0.4
representation learning
language,
reinforcement learning
transfer learning, learning, and lifelong learning
0.9
0.32
6.33
applications to physical sciences
infrastructure,
(physics, chemistry, biology,
07
01
02
00
01
02
probabilistic methods
0.28
applications to neuroacience & cognitive
general machine learning (i.., nome the above)
visualization interpretation of learned representations
138
929
learning on grapha, other geometrias
1:22
optimization
6.2
033
datasets and benchmarks
6.30
learning theory
2.9
unsupervised, sulf-supervised, semi supervised, & supervised representation
9:00
metric learning, kernel learning, sparse coding
6.12
0
11
02
Accepiance 08
$4
60
02
0.4
0.
Figure Comparative acceptance rates of ICLR' 24 papers by humans and GPT ranking system across research areas. We sort areas by the GPT ranking system's acceptance rate from highest to lowest. The GPT ranking system exhibits noticeably higher acceptance rates application- ooriiattoo--oieneedd fields compared to human reviewers, showing the most striking disparities in robotics (0.56 vs. 0.32) and societal considerations (0.51 vs. 0.30). In contrast, for more theoretical or methodologically focused areas, it assigns significantly lower acceptance rates than human reviewers. Learning theory demonstrates the largest gap, with acceptance rate at 0.12 versus humans at 0.50. The categorization based the 20 primary areas by ICLR" 24.
patterns. This limitation can result in preference for research with more concrete implementations than abstract theoretical work. Future interdisciplinary research could explore these hypotheses.
4.6 Potential Bias against Novel Research Topics
We explored how GPT ranking system influences the novelty of topics in selected papers. To measure the novelty of the topies, we embed each paper's abstract with OpenAl's text-embedding- 3-small model, generating vector representation for each abstract. We then compute the distance between each paper's 's vector and the closest neighbor within the abstracts from the same conference. A smaller distance indicates greater similarity between abstracts, thus lower novelty of the topic.
Through examination of papers categorized into top decision tiers by human reviewers and our GPT ranking system, our analysis revealed a consistent pattern (Figure 5): papers receiving higher rankings from the GPT system exhibited substantially smaller average distances to their nearest neighbors compared to papers selected by human reviewers. This might indicate that GPT exhibit bias toward papers that cover topics similar to those in the existing literature, potentially undervaluing work that delves into novel research areas.
4.7 Impact on Academic Inequality
We also observed that our GPT ranking system tend to favor papers from established research institutions, potentially amplifying existing imbalances in the academic ecosystem. To quantify this imbalance, we track the first authors' affiliation distribution across papers presented in higher decision tiers. Using the Gini coefficient, statistical measure commonly applied to income inequality, quantified the degree publication concentration across institutional affiliations.
We found concerning pattern of institutional bias: the GPT ranking system consistently exhibited higher institutional inequality compared to humans, with significantly higher Gini coefficients across all conferences studied. The most significant disparity was observed in ICLR 2023 and ICLR 2024.

---

GPT Selected
Human Selected
1.0
Nearest Neighbar Distance
0.7
0.6
0.5
ICLA2023
ICLR2024
EMNLP2023 Main Track
CoRL2023
NeurIPS2023 Oral
Notable-top-5%
Oral
Oral
Figure 5: Nearest neighbor distances of papers selected for top-tier acceptance by GPT ranking system versus human reviewers. Higher distances indicate greater novelty. Human-selected papers (blue) consistently show higher nearest neighbor distances than GPT-selected papers (orange) across all conferences. The differences statistically significant 0.05) for ICLR 2023 Notable-top-5%% ICLR 2024 Oral, and EMNLP 2023 Main Track. For CoRI. 2023 Oral and NeurIPS 2023 Oral, we did observe statistically significant differences, which may be due smaller sample sizes.
ICLR 2023, 43.8% of GPT ranking system's top-tier papers came from 10 institutions compared to only 27.0% in human decisions, while in ICLR 2024, the gap persisted with 37.2% of GPT ranking system's 's top selections coming from 10 institutions versus 26.7% in human evaluations.
0.7
Human Selected
GPT Selected
0.6
Gini Coefficient
0.5
0.4
0.3
0.2
0.1
ICLR2023 Notable-top-5%
ICLR2024 Oral
EMNLP2023 Main Track
CoRL2023 Oral
NeuriPS2023 Oral
Figure 6: Comparison of Gini coefficients for papers selected by humans system across conferences. Larger Gini coefficients represent greater inequality or imbalance in the distribution. The consistently higher Gini coefficients for GPT-selected papers (purple bars) compared to human sellnte papers (green bars) indicate that GPT ranking system exhibit greater institutional concentration, potentially favoring papers from established research institutions.
GPT ranking

---

Conclusion
Existing LLM-assisted peer review efforts largely replicate traditional workflows, framing evaluation as absolute scoring followed by aggregation [18, 27, 49, 55, 61]. However, we argue that the scalability of LLMs open new opportunities to redesign scholarly evaluation structures rather than merely automate human processes. In this work, we introduce and explore novel mechanism using LLM agents to evaluate academic papers through pairwise comparisons. Rather than assigning isolated scores, the system constructs a global ranking by aggregating local relative judgments between submissions. Through empirical experiments, we find that with sufficient scale, our system effectively identifies high-impact papers across multiple conferences, significantly exceeding traditional rating- based methods. The overlap between papers accepted by our system and those accepted by human reviewers aligns with human-humal agreement levels observed in previous studies, suggesting potential value as complementary tool in the review process.
At the same time, our analysis reveals important challenges. The system exhibits area-specific preferences that diverge from human reviewers, shows measurable decline in topic novelty among selected papers, and concentrates acceptances among a smaller set of prestigious institutions than human-selected papers. These patterns suggest that while scalable LLM-driven evaluation systems hold promise, careful design will be critical to ensure they promote diversity, equity, and innovation rather than reinforcing existing hierarchies.
10

---

References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[]] B. Alberts, B. Hanson, and K. L. Kelner. Reviewing peer review, 2008
[3] A. Beygelzimer, N. Dauphin, P. Liang, and W. Vaughan. Has the machine learning review process become more arbitrary as the field has grown? the neurips 2021 consistency experiment. arXiv preprint arxivv2306.00 03262, 2023.
[4] L. Bormmann and H.-D. Daniel. What do citation counts measure? review of studies on citing behavior. Journal of documentation, 64(1):45-80, 2008.
[5] I.. Bornmann and R. Mutz. Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the association for information science and technology, 66(1 1:22252- 2015.
[6] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[7] M. Breuning. J. Backstrom, J. Brannon, B. I. Gross, and M. Widmeier. Reviewer fatigue? why scholars decline to review their peers' work. PS: Political Science Politics, 48(4):595-600, 2015.
[8] B. Carterette, P. Bennett, M. Chickering, and S. T. Dumais. Here there: Preference judgments for relevance. In European Conference on Information Retrieval, pages 16-27. Springer, 2008
[9] M. D'Arcy T. Hope, L. Birnbaum, and D. Downey. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04253 2024.
[10] M. Demeter, A. Jele, and Z. B. Major. The international development of open access publishing: comparative empirical analysis over seven world regions and nine academic disciplines. Publishing Research Quarterly, 37(3):364-383, 2021.
[11] M. S. Di Bitetti and J. A. Ferreras. Publish (in english) or perish: The effect citation rate of using languages other than english in scientific publications. Ambio, 16:1211 2017.
[12] D. Eeer and G. Maciocci. Sciencebeam using computer vision extract pdf data. Elife Blog Post, https: /aap:////eleescenceee. /teaafeecieeeee.o///aaa///////e org/labs/SbS6afff. sciencebeam- uwwnuu ceimppter-vision--ee vision tettt--ettttttttto pdf data, 2017. Accessed:
26 March 2025.
[13] C. W. Fox, A. Y. Albert, and T. H. Vines. Recruitment of reviewers is becoming harder at some journals: a test the influence of reviewer fatigue at six journals in ecology and evolution. Research Integrity and Peer Review, 2:1-6, 6,, 2017.
[14] C. W. Fox, J. Meyer, and F. Aimé. Double-blind peer review affects reviewer ratings and editor decisions at an ecology journal. Functional Ecology, 37(5):1144-1155) 2023.
[15] M. L. Gordon. M. S. Lam, J. S. Park, K. Patel, J. Hancock, T. Hashimoto, and M. S. Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference Human Factors in Computing Systems, pages 1-19, 2022.
[16] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The Ilama herd of models. arXiv preprint 277.::2407.21783, 2024.
[17] M. Hosseini and S. P. Horbach. Fighting reviewer fatigue amplifying bias? considerations and recommendations for use of chatgpt and other large language models scholarly peer review. Research integrity and peer review, 8(1):4, 2023.
11

---

[18] Y. Jin, Q. Zhao, Y. Wang, H. Chen, K. Zhu, Y. Xiao, and J. Wang. Agentreview: Exploring peer review dynamics with Ilm agents. arXiv preprint arXiv:2406. 1224406.177000 12708, 2024.
[19] M. Lamont. How professors think: Inside the curious world of academic judgment. Harvard University Press, 2009.
[20] C. J. Lee, C. R. Sugimoto, G. Zhang, and B. Cronin. Bias in peer review. Journal of the American Society for information Science and Technology, 64(1):2-17, 2013.
[21] W. Liang, Z. Izzo, Y. Zhang, H. Lepp, H. Cao, X. Zhao, L. Chen, H. Ye, S. Liu, Z. Huang, et al. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on conference peer reviews. In International Conference on Machine Learning, pages 29575-29620. PMLR, 2024.
[22] W. Liang, Y. Zhang, H. Cao, Wang, D. Y. Ding, Yang, K. Vodrahalli, He, S. Smith, Y. Yin, D. A. McFarland, and J. Zou. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NE/M AI, 10)::A1o2440019, 2024, doi: (0.106//Alo24401996 URL bottps:/aal .nemm http:////fuu11111.00////0000
[23] J. Lin, J. Song, Z. Zhou, Y. Chen, and X. Shi. Moprd: A multidisciplinary open peer review dataset. Neural Comput. Appl., 35(34):24191-2422) Sept. 2023. ISSN 0941-0643, doi:
00.1007/00552--023-0833. 02--0099-----5.5.. URL htt.hhttp:///eee httpcpp::/ddii.. 1 wttp:///....0///000000000000-000
[24] Lin, Song. Z. Zhou. Y. Chen, and X. Shi. Automated scholarly paper review: Con- cepts, technologies, and challenges. Information Fusion. 98:101830, 2023, ISSN 1566-2535. doi: https://doi.oor/1OO.o http:////............/.........0 inffus. URL http: /www wcwencedreeccc..c.c.o...m....... com/ scie
[25] D. Lindsey. Using citation counts as measure of quality in science measuring what's measur- able rather than what's valid. Scientometrics, 15:189-2033 1989.
[26] R. Liu and N. B. Shah. Reviewergpt? an exploratory study on using large language models for paper reviewing, 2023. URL https:///azii..tttttt/
abs/2306. 00622.
[27] C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408 06292, 2024.
[28] Y. Lu and Y. Kong. Calibrating "cheap signals" in peer review without prior. Advances in Neural Information Processing Systems, 36:20640-20666 2023.
[29] S. Ma. Q. Chen, X. Wang. C. Zheng, Z. Peng. M. Yin, and X. Ma. Towards human-a ai deliberation: Design and evaluation of lIm-empowered deliberative ai for ai-assisted decision- making. arXiv preprint arXiv:2403. 2024.
[30] A. McCook. Is peer review broken? submissions are up. reviewers are overtaxed, and authors lodging complaint after complaint about the process top-tier journals. what's wrong with peer review? The scientist, 20(2):26-35, 2006.
[31] B. Mehmani and A. Ghildiyal. Rethinking Reviewer Fatigue, https://eonn.ubbub.oor ort..p.../aaaagyyy......
EON, nov 15 2024.
[32] B. Merriman. Peer review as an evolving response to organizational constraint: Evidence from sociology journals, 1952-2018. The American Sociologist, 52:341-366, 2021.
[33] I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024. URL https://arxiv..
org/ abs/2410 05229.
[34] S. Negahban, S. Oh, and D. Shah. Rank centrality: Ranking from pairwise comparisons. Oper. Res., 65(1)2266-287 Feb. 2017. ISSN 0030-364X. doi: 10.1287/0pre.2116.5522 URL https://doi. http:///ooooorr/////11 12w. /prre. 2016. 1534.
[35] I.. Niu, N. Xue, and C. Pöpper. Unveiling the sentinels: Assessing ai performance in eybersecu- rity peer review. arXiv preprint arXhv:2309.054577 2023.
12

---

[36] OpenAl. Gpt-4o mini: advancing cost-efficient intelligence, 2024. URL https:// openai apenai.com/iille htt/inddxxx/gg/ sptpx/gpt-444 cppt-4-miinn co--mini-oe coo-mini-advaaeee -avaaniinn cost efficient- i Ac- cessed: 2025-04-02.
[37] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[38] Z. Robertson. Gpt4 is slightly helpful for peer-review assistance: pilot study, arXiv preprint ar arXiv:2307.05499, 2023.
[39] D. Scherbakov, N. Hubig, Jansari, A. Bakumenko, and L.. A. Lenert. The emergence of large language models (Ilm) as tool in literature reviews: an Ilm automated systematic review. arXiv preprint arXiv:2409.04600, 2024.
[40] N. B. Shah. Challenges, experiments, and computational solutions in peer review. Commun. ACM, 65(6):76-87, May 2022. ISSN 0001-0782.
doi: 10.1145/3528086. URL https:
//doi. orgoi.org/1n 10. 1 1145/111550 3528086.
[41] Y. Shao, V. Samuel, Y. Jiang. J. Yang, and D. Yang. Collaborative gym: A framework for enabling and evaluating human-agent collaboration. arXiv preprint arXiv:2412.17011, 2024.
[42] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36: 8634-8652, 2023.
[43] C. Si, D. Yang, and T. Hashimoto. Can ilms generate novel research ideas? large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.044 1099, 2024.
[44] F. Squazzoni, G. Bravo, M. Farjam, A. Marusic, B. Mehmani, M. Willis, A. Birukou, P. Dondio, and F. Grimaldo. Peer review and gender bias: A study on 145 scholarly journals. Science Advances, 7(2):eabd0299, 2021. doi: 00.1126/sciddv.abdoe org/doi/abs/ 1126/sciadv abd0299.
URL
[45] S. Srinivasan and J. Morgenstem. Auctions and peer prediction for academic peer review, 2023. 00923.
URL http::::///xxxii.r////bbb///////
[46] I. Stelmakh, N. B. Shah, A. Singh, and H. Daumé III. Prior and prejudice: The novice reviewers' bias against resubmissions in conference peer review. Proceedings of the ACM on Commut-Computer Interaction, 5(CSCWI) 1 2021.
[47] I. Stelmakh, C. Rastogi, R. Liu, S. Chawla, F. Echenique, and N. B. Shah. Cite-seeing and reviewing: A study on citation bias in peer review. PLOS ONE, 18(7):c0283980, July 2023. ISSN 1932-6203. doi: wooema/laaall.poe..... pone. ponee
URL http: / /tx. doi org/10. 1371/
journal.
0283980.
[48] W. J. Su. You are the best reviewer of your own papers: an owner-assisted scoring mechanism. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393,
[49] C. Tan, D. Lyu, Li, Gao, J. Wei, S. Ma. Z. Liu, and Z. Li. Peer review as multi-turn and long-comtes dialogue with role-based interactions. arXiv preprint arXiv:2406.05683 2024.
[50] Verharen. Chatgpt identifies gender disparities in scientific peer review. Elife, 12:RP90230. 2023.
[51] Wang. Unpacking the matthew effect in citations. Journal of Informetrics, 8(2):329-339 2014.
[52] Wang and N. B. Shah. Your 2 is my 1, your my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '1,, page 864-872, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 9781450363099.
13

---

[53] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou. self-consisteney improves chain of thought reasoning language models. arXiv preprint arXiv:2203. / 171, 2022.
[54] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. tfain-of-thoouht prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
[55] Y. Weng, M. Zhu, G. Bao, H. Zhang, J. Wang, Y. Zhang, and L. Yang. Cycleresearcher: Improving automated research via automated review. arXiv preprint M.Ki::211..008//1 2024.
[56] Wu, H. Xu, Y. Guo, and W. Su. An isotonic mechanism for overlapping ownership, 2025. URL https://arxiii org/ attp///.../....//// 11154.
[57] Y. Yan, W. Su, and J. Fan. Isotonic mechanism for exponential family estimation in machine learning peer review. Journal of the Royal Statistical Society Series Statistical Methodology, page qkaf025, 2025.
[58] R. Ye, X. Pang. J. Chai, J. Chen, Z. Yin, Z. Xiang, X. Dong. J. Shao, and S. Chen. Are there yet? revealing the risks of utilizing large language models in scholarly peer review, 2024. URL https:/ / //rxivix.v.v/////////////////av/ org/ abs/ /412. 01708.
[59] Zhang, H. Zhang. Z. Deng, and D. Roth. Investigating fairness disparities in peer review: A language model enhanced approach. arXiv preprint vrXiv2221..06330 2022.
[60] R. Zhou, L. Chen, and K. Yu. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC--CLJIN) 2024), pages 9340-9351, Torino, Italia, May 2024. ELRA and ICCL. URL httphtt:/acclaatttoooooo 2024. 1rec-main.ee...mmm 816/.
org/
[61] M. Zhu, Y. Weng, L. Yang, and Y. Zhang. Deepreview: Improving Ilm- based paper review with human-like deep thinking process. arXiv preprint arXiv:2503 08569, 2025.
[62] Z. Zhuang, Chen, H. Xu, Y. Jiang, and J. Lin. Large language models for automated scholarly paper review: A survey, Information Fusion, 124:103332, 2025. ISSN 1566-2535. doi: sci cccaac///rrricce////////BBEEEES
URL htt.htppss//ww.sccieeeeedlle http://www.iiieeeeee.t..... com/
[63] H. Zuckerman and R. K. Merton. Patterns of evaluation in science: Institutionalisatoon, structure and functions of the referee system. Minerva, pages 66-100, 1971.
14

---

A Discussion
A.1 Limitations
In this study, we rely on citation counts proxy for academic impact. Although citation metrics are widely used and provide a quantifiable measure of influence, they are affected by numerous factors beyond quality, such as the visibility of research communities, prevailing research trends, and established reputations [25, 4, 51, 11]. More critically, papers selected by human reviewers inherently receive more visibility through conference presentations and proceedings. The fact that our GPT ranking system identified papers with comparable citation impact despite this disadvantage demonstrates a certain level of effectiveness in our approach. Nevertheless, this circularity problem makes it difficult to establish truly independent measure of quality. To further verify the robustness of our proposed method, we conducted experiments using alternative measures. Specifically, we measured the Spearman correlation between our system's 's output and human review scores. In both ICLR 2023 and ICLR 2024, the BT scores produced by our mechanism show moderate correlation with the average human ratings- with signals in the current review process, while not fully replicating its outcomes. We believe this complementary nature is essential for identifying potential improvements to existing peer review systems.
and 24%, respectively. This indicates that our system aligns
Furthermore, while we exclusively used GPT-4o mini for our large-scale experiments due to its balance of performance and computational cost, we acknowledge that there are other diverse LLMs capable of academic assessment, and they might yield substantially different results. A more comprehensive evaluation across multiple models would provide better insight into the generalizability of our method and findings. In Appendix C.4, we tested two additional models beyond GPT--4o mini: Gemini 2.0 Flash and Saaidd---Haiuu2222200 and found that, in both cases, our pairwise comparison framework performs significantly better than the GPT rating system baseline. Specifically, papers selected by Gemini 2.0 Flash receive an average of 18.3 citations (vs. 11.4), while those chosen by Claude-3-Haiku average 16.8 (vs. 11.4). These results suggest that the advantage of our approach robust across different LLMs.
We scaled the number of comparisons to over 106
Finally, while our current work constructs single global ranking, future extensions could explore personalized or multi-objective evaluation systems that explicitly account for epistemic diversity and evolving community goals.
A.2 Expanding Peer Review through Pairwise Evaluation
Beyond the immediate results, our framework and explorations open broader directions for redesign- ing scholarly review systems.
First, because pairwise comparisons produce local relative judgments that can be incrementally aggregated, our approach naturally lends itself to continuous evaluation. Rather than operating within fixed submission and decision timeline, conferences could maintain ongoing review pipeline, where new papers are progressively compared against the existing submission pool. Such a dynamic process could allow for rolling acceptances, faster feedback loops, and better accommodation of late-breaking research.
Second, pairwise evaluation offers distinct advantages for emerging or interdisciplinary fields where traditional scoring rubries are poorly defined or difficult to establish. Absolute scoring requires consensus evaluation criteria and careful calibration across reviewers, which can be challenging in fast-moving or nascent research areas. Comparative judgments, by focusing on relative rather than absolute assessments, can surface high-potential work even when shared evaluation norms are still evolving, making the system more adaptable to domains where innovation resists rigid checklist-bane quantification.
Third, the flexibility of the aggregation process suggests opportunities for personalized and diversified peer review. Different weighting schemes could prioritize novelty, interdisciplinarit, methodological rigor, or other dimensions based on the goals of specific conferences. tracks, or even reviewer com- munities. In principle, such mechanisms could allow peer review to better reflect the heterogeneous values of different research communities, moving beyond the sine-size-fit--a model [15] currently dominant in scientific publishing.
15

---

Finally, an important direction for future exploration lies in integrating human expertise and LLM evaluations [41, 29]. Rather than viewing LLM-based and human-based assessments competing alternatives, hybrid models could combine the strengths of both: leveraging L.MMs for large-scale, consistent pairwise comparisons while relying on human reviewers to provide deeper qualitative insights, assess boundary cases, and adjudicate particularly novel or interdisciplinary submissions. Designing effective protocols for human-Al collaboration in peer review could further enhance both the scalability and the fairness of the evaluation process.
Together, these directions highlight how reframing peer review around relative comparisons, en- riched by human-Al collaboration, could support more scalable, inclusive, and adaptive scholarly communication ecosystem.
16

---

B Experimental Details
B.1 Dataset Details
Here we include additional details on the datasets used for our experiments.
Table 2: Academic paper data from major ML conferences.
Conference
Decision Types
# of Papers
ICLR 2023
Notable-top-5%
89
Notable-top-25%
281
Poster
1,193
Reject (Withdrawn submissions included)
3,303
ICLR 2024
Accept (Oral)
86
Accept (Spotlight)
363
Accept (Poster)
1,786
Reject (Withdrawn submissions included)
4,923
NeurIPS 2023
Accept (Oral)
67
Accept (Spotlight)
374
Accept (Poster)
2,748
EMNLP 2023
Accept-Maij
975
Accept-Findings
993
CoRL 2023
Accept (Oral)
32
Accept (Poster)
166
B.2 Implementation Details
We use the official OpenAl's Batch API for GPT-4o mini and set temperature as during pairwise comparison. The number of API calls and estimated cost for each simulated conference is follows:
Table 3: API usage and estimated costs for pairwise comparison across each conference. EMNLP 2023 and CoRL 2023 used all available pairs, while others were randomly downsampled to 3M pairs.
Conference
# of API Calls
Estimated Cost (USD)
ICLR 2023
3,000,000
1, 350
ICLR 2024
3,000,000
1, 350
NeurIPS 2023
3,000,000
1, 350
EMNLP 2023
3,871,056
1, 700
CoRL 2023
39,006
17
17

---

B.3 LLM prompts used in the study
Please act as an impartial judge and evaluate the quality of the following two papers. As the area chair for top ML conference, you can only select reasoning of the pros and cons for each paper (two sentences), then provide your choice in a binary format. Start with a brief meta ttaarevvievrresoonin of the pros and cons for each paper, focusing novelty, significance, clarity, methodology. and very critical
paper. Start with brief meta-revier/
not biased by what the author claimed. Finally, provide your choice in a binary format.
practical implications.
Please provide your analysis in format.
*## Paper 1: Submission Title: (title)
...
Abstract: (abstract)
Figures Captions:
Main: (main_content)
### Paper 2:
Submission Title: (title)
..e
Abstract: (abstract)
Figures Captions: (figure_ and ttlpttaale__cp_ppe captionss
Main: (main_content)
Your JSON output should look like this:
{{
"Your meta-revies and reasoning for paper 1", "paper_____t "aper___reviev". "Your geta-reviee and reasoning for paper 2", "chosen_paper", "paper_1 or paper_2"
}
Figure 7: Example prompt for pairwise comparison.
18

---

Additional Results
C.1 Discriminative Capability of GPT ranking system in Research Evaluation
ICLR 2024
ICLR 2024
NeuriPS 2023
NewriPS 2023
Mean Citstton
80
Mean Citation
Mean Citation
Citation
200
00
200
60
150
GPT--rral
GPT-spatigh OPT-postor
Humen-spotio spoltight
onal
Homen poater
GPT-oral
GPT-spoligh OPT-postor
Human
Humon-sppttigh
Human poster
JOLR
ICLR 2023
Mean Citation
Mean Citation
230 175
176-
190-
150
100
GPT-tp-51s
OPT-ap-25M Gaaale
Humen-lop-SWi
-00-25% Human-top-22 Human-pestar
Figure Comparison mean citation counts across multiple AI conferences under fine-grained decisions in accepted papers. The results consistently show that higher-tier papers (both GPT- selected and human-selected, receive overall higher influence than lower-tier papers. Error bars represent 95% confidence intervals. *P < 0.05, **p < 0.01, ***p < 0.001, and s***p < 0.0001.
JCLR 2024
ICLR 2024
EMNLP 2023
EMNLP 2023
18
10 16 t2
18
Median Citation
15
Median Citation
Median Citation
Modian Citation
12
10
to or a
SPT-accepted
GPT-rejected
Human accepted Human-acrepted
Human-rejjcted
GPT-main
GPT-findings
Human-main
Hunaan-indings
JOLR 2023
ICLR 2023
CoRL 2023
CoRL 2023
Median Citstton
15.
Median Citation
ta
Median Citation
40
Media Citetion
12
20
2J 6 10.
10
40
2
GPT-acoepted
GPT-rejacted
Human- accepted
Human-rejscted
GPT-oral
GPT-poster
Human-oral
Hlmaan-paster
Figure 9: Comparison of median citation counts across multiple AI conferences under two-fold decisions (e.g., accepted vs. rejected, main vs. findings track, oral vs. poster). The results consistently show that higher-tier papers (both GPT--elected and human-selected) receive overall higher influence than lower-tier papers.
19

---

C.2 Consistency Analysis between GPT Ranking System and Human Peer Review
Table 4: Summary of recommendations for ICLR 2023 papers by two review systems: the human peer review (Human) and the GPT ranking system (GPT), Each row represents humans' decision, while each column shows how the GPT ranking system categorized the same papers.
Human\GPT
Notable top 5%
Notable top 25 %
Poster
Reject
Notable top 5% Notable top 25%
10
9
26
44
27
95
148
Poster
29
93
360
711
Reject
39
152
712
2,400
C.3 Different Acceptance Patterns across Research Areas
applications to robotics, autonomy, planning
0.36
causal reasoning
2.26
0:40
including fairness, safety, privacy
societal considerations
semi-supervised, & supervised representation learning
unsupervised, superviedd,
applications physical sciences (physics, biology,
0.48
neurosymbolic systems
0.54
generative models
transfer learning, learning, and lifelong learning
8.32
3:11
reinforcement tearning
ropresentation learning for vision, audio, language, etc.
6.32
Acceptance Nate
Acceptance Rate
probabilistic methods
2:2
4.42
(la,, none the above)
02
datasets and benchmarks
p32
optimization
0:25
infrastructure,
visualization or interpretation representations
software Biraries, hardware,
learning graphs, geometries& topologies
6.36
learning, learning,
512
applications neuroscience & cognitive science
learning theory
Acceplance Rate
Acceptance Rate
Figure 10: Comparative acceptance rates of ICLR' 23 papers by humans and GPT ranking Systems across research areas. We sort areas by the GPT ranking system's acceptance rate from highest lowest. The GPT ranking system exhibits noticeably higher acceptance rates in application- oriented fields compared to human reviewers, showing the most striking disparities in robotics (0.53 vs. 0.36) and societal considerations (0.51 vs. 0.25). In contrast, for more theoretical or methodologically focused areas, it assigns significantly lower acceptance rates than human reviewers. Learning theory demonstrates the largest gap, with acceptance rate at 0. versus humans 0.43.
20

---

C.4
Using other LLMs as agents
25
Average Number Citations for accepted papers
Claude Ranking System without Area Control
Claude Ranking System with Area Control
Human Accept Baseline
Randomly Accept Baseline
GPT Rating System Baseline
15
10a
10a Number of Agents
10
10
Figure 11: Scaling of average citation counts for accepted papers by Claude ranking system with increasing pairwise comparisons. We observe similar temporal scaling pattern in the GPT ranking system, with citation counts increasing as the number of pairwise comparisons grows.
25
Average Number Citations for accepted papers
Gemini Ranking System without Area Control
Gemini Ranking System with Area Control
Human Accept Baseline
Randomly Accept Baseline
GPT Rating System Baseline
15
10a
10a
10
10a
Number of Agents
Figure 12: Scaling of average citation counts for accepted papers by Gemini ranking system with increasing pairwise comparisons. We observe similar temporal scaling pattern in the GPT ranking system, with citation counts increasing as the number of pairwise comparisons grows.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction state the primary contributions of the paper, in- cluding the introduction of LLM-based pairwise ranking system, the significant performance improvements and emergent biases in the selection process through experimental results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix A.1.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
There are no theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We explained comprehensive details on dataset (Section 4.1, Appendix B. experimental setups (Appendix B), and methods used (Section 3), ensuring that the results can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: All numerical experiments performed are simple enough to reproduce without access to code. Code and data are available upon request.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / ctttpp..cc/ubbbceec gww/gguiessss CoodeSubmesscc ssi assionPolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings explained in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports error bars and includes information on the statistical signifi- cance of the experimental results, ensuring the reliability of the findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The experiments mainly involved inference using API calls to GPT-4o mini The type of compute workers, memory and time of execution are managed by the API provider. Details about the number of API calls and configurations are provided in Appendix B.2.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /te/rreursp..// httpppplicc/hhcccciddeeeeeee
Answer: [Yes]
Justification: We confirm that the research conducted in the paper conforms, every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses potential societal impacts in Section 5, Appendix A.2.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No such models or datasets are involved.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper properly credits the creators of existing assets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Our LLM-based pairwise ranking system employs LLMs as evaluators to produce binary preference judgments between paper candidates. Please refer to Section 3.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy pectp::/neeurjje http:///urrrpps.ccoooeeeeeeeeeee c for what should or should not be described.
28


Reviews:
[{"review_id": "Y3VbjFQVox", "reviewer": "Reviewer_vHzj", "summary": "This paper proposes a novel redesign of LLM-based peer review by shifting from conventional per-paper scoring to pairwise comparison of manuscripts. Instead of assigning absolute scores, LLM agents compare two papers at a time and vote for the better one. These results are aggregated using the Bradley–Terry model to produce a global ranking of all submissions. Experiments on major ML conference papers (e.g., ICLR, NeurIPS, EMNLP) demonstrate that this method better identifies high-impact papers (measured by citation count) than traditional rating-based GPT review systems. However, the method also exhibits biases: a preference for less novel topics and overrepresentation of papers from elite institutions.", "questions": "see weakness", "limitations": "no", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clear and make sense. The authors argue convincingly that the current peer review system is a historically contingent artifact, and LLMs allow rethinking the entire structure.\n2. The paper is well-written. \n3. The idea is simple but novel in review paradigm: The paper proposes a genuinely different peer review workflow using pairwise comparisons, diverging from the current mainstream of LLM-based review replication.\n\nWeaknesses:\n1. Bias Amplification: The approach tends to favor papers from top institutions and penalize novelty, raising serious fairness and diversity concerns.\n\n2. Limited Model Diversity: Experiments rely solely on GPT-4o mini. It's unclear whether the findings generalize across other LLMs.\n\n3. Citation as Sole Impact Metric: The paper assumes that citation count is an accurate proxy for quality, which can be problematic due to citation inflation, visibility bias, and Matthew effects.", "was_revised": true, "final_justification": "I  maintain my original overall score 4 ( boardline accept).", "rebuttal": "We thank Reviewer vHzj for their positive comments and helpful feedback on our work.\n\n> Bias Amplification: The approach tends to favor papers from top institutions and penalize novelty, raising serious fairness and diversity concerns.\n\nThank you for the comment. As noted in the discussion, while our LLM-based peer review approach shows promise, it also raises concerns around fairness and diversity, such as institutional concentration and reduced topic novelty. We see these not as definitive flaws, but as opportunities to inform more equitable and inclusive review practices. Our system is intended to augment—not replace—the human review process, and we hope this analysis inspires further research on mitigation strategies and alternative designs aligned with academic values.\n\n> Limited Model Diversity: Experiments rely solely on GPT-4o mini. It's unclear whether the findings generalize across other LLMs.\n\nThank you for the suggestion. We have conducted experiments to test two additional models other than GPT-4o mini: Gemini 2.0 Flash and Claude-3-haiku-20240307. We scaled the number of comparisons to over $10^6$ and found that, in both cases, our pairwise comparison framework performs significantly better than the GPT rating system baseline. Specifically, papers selected by Gemini 2.0  Flash receive an average of 18.3 citations (vs 11.4), while those chosen by Claude‑3‑Haiku average 16.8 (vs 11.4). These results suggest that the advantage of our approach is robust across different LLMs.\n\n> Citation as Sole Impact Metric: The paper assumes that citation count is an accurate proxy for quality, which can be problematic due to citation inflation, visibility bias, and Matthew effects.\n\nThank you for raising this important point. We acknowledge that citation counts, while widely used, are an imperfect reflection of research quality. As described in Appendix A.1, they are affected by numerous factors beyond quality, such as the visibility of research communities, prevailing research trends, and established reputations. \n\n\nTo further verify the robustness of our proposed method, we conducted experiments using alternative measures. Specifically, we measured the Spearman correlation between our system’s output and human review scores. In both ICLR 2023 and ICLR 2024, the BT scores produced by our mechanism show moderate correlation with the average human ratings—27% and 24%, respectively. This indicates that our system aligns with signals in the current review process, while not fully replicating its outcomes. We believe this complementary nature is essential for identifying potential improvements to existing peer review systems.\n\nMoreover, we  added explicit clarification to emphasize that these metrics are used as a practical means of evaluation: “ It is important to note that while we use proxy metrics—such as future citation counts—to evaluate our proposed mechanism, this represents an exploratory and pragmatic step rather than an ideal. The gold standard would be direct human evaluation of how well a mechanism supports the selection and improvement of high-quality research, but such evaluations are costly and difficult to scale [1]. Proxy metrics offer a tractable, though imperfect, way to approximate long-term impact and anticipate how different mechanisms might perform. Ideally, a strong alternative review mechanism should correlate with—but not fully replicate—the outcomes of the current system, offering complementary perspectives and surfacing different strengths. As we show, our proposed pairwise comparison mechanism achieves this balance, aligning with current human judgments while introducing useful differentiation. Crucially, our goal is not to mimic existing outcomes, but to design mechanisms that add value.” \n\n[1] Si, Chenglei, Diyi Yang, and Tatsunori Hashimoto. \"Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers.\" The Thirteenth International Conference on Learning Representations."}, {"review_id": "05cWER0UrO", "reviewer": "Reviewer_rL5P", "summary": "This paper proposes a new LLM-based pairwise comparison mechanism to rank manuscripts. The LLMs are prompted to give a meta-review + reasoning for each paper, followed by a binary indicator to show the preferred manuscript. Based on these scores, the authors estimate a Bradley-Terry model to recover the manuscript ranking from these pairwise comparisons. In their experiments, the authors use OpenReview data, including ICLR, NeurIPS, CoRL, and EMNLP, and GPT-4o mini (context window: 128K tokens). They empirically find that the theoretical burden of the O(n^2) scaling does not apply as with less than 2% of the samples a high-quality ranking is recovered. The authors validate the LLM-based ranking via citation metrics and human reviewer decisions (e.g., accepted vs. rejected, main vs. findings track, oral vs. poster, etc.). Finally, the authors also show that this mechanism tends to reduce novelty (measured via distance in word embeddings) and increases institutional imbalance.", "questions": "- Are the differences in the novel research topics statistically significant? There seems to be a large overlap between the distributions.\n- Do the authors have any suggestions on how to incorporate responses to the meta-reviews? The back and forth (e.g., clarifications, additional experiments) between reviewers and authors is an important part of the review process which is omitted using this approach.", "limitations": "The authors discuss the limitations in Appendix Section A1.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- I like how the authors do not try to replicate the existing review process, but argue that LLMs may offer the opportunity to redesign the entire process, which they argue mostly reflects historical compromises rather than principled design choices.\n- The authors propose a clear methodology and empirically validate it. For example, showing that the theoretical burden of the O(n^2) scaling does not apply in practice.\n- I like Appendix Section A2. While more speculative (and therefore probably part of the appendix), I like that the authors try to assess the real-world implications of this pairing scheme (which at first might seem limited because this is not how we currently conduct peer review).\n\nWeaknesses:\n- Since we have no access to the ground truth when implementing this system in real time, it would be helpful if the authors could offer either theoretical bounds or heuristics on how many pairs should be evaluated before we converge to some high-quality ranking.\n- The authors acknowledge this limitation already partially in Appendix Section A1, but still the endogeneity effect of citations (being accepted to the conference may bump citations) and using human reviewer decisions (e.g., accept vs reject) as a validation criterion seems to contradict the statement that current peer review is failing us as we try to replicate its outcomes. Maybe the authors should clarify the stance that the mechanism how we conduct peer review could be improved, but the outcomes themselves we're trying to replicate.", "was_revised": true, "final_justification": "Update after rebuttal:\nThe overall score has been increased to 5 and quality to 3. The authors have assessed statistical significance + provided some theoretical bounds on the number of necessary pairs to obtain a qualitative ranking.", "rebuttal": "Thank you very much for your helpful feedback and support for the paper! We have carefully updated the paper to incorporate your suggestions. \n\n> Since we have no access to the ground truth when implementing this system in real time, it would be helpful if the authors could offer either theoretical bounds or heuristics on how many pairs should be evaluated before we converge to some high-quality ranking.\n\nThank you for the suggestion. Following your guidance, We have added a theoretical bound (Theorem 4 in [1]) to clarify how many pairwise comparisons are needed to recover a high-quality global ranking. Specifically, it shows that if $ m > 12n \\log n $ pairs are sampled uniformly at random, then with high probability, the estimate $ \\hat{\\theta} $ will satisfy\n\n$$\n\\Vert \\hat{\\theta} - \\theta^* \\Vert = O\\left(n \\sqrt{\\frac{\\log n}{m}}\\right)\n$$\n\nThis provides a theoretical guideline for how many comparisons are required to achieve reliable ranking performance, even when the true scores are not accessible during real-time setting.\n\n> Maybe the authors should clarify the stance that the mechanism how we conduct peer review could be improved, but the outcomes themselves we're trying to replicate.\n\nThank you for the suggestion. We have added discussion clarifying our stance on improving the peer review process versus replicating its outcomes as follows:“ Ideally, a strong alternative review mechanism should correlate with—but not fully replicate—the outcomes of the current system, offering complementary perspectives and surfacing different strengths. As we show, our proposed pairwise comparison mechanism achieves this balance, aligning with current human judgments while introducing useful differentiation. Crucially, our goal is not to mimic existing outcomes, but to design mechanisms that add value. ” \n\n> Are the differences in the novel research topics statistically significant? There seems to be a large overlap between the distributions.\n\nThank you for your question. Although there is a large overlap between the distributions, the differences are statistically significant (p < 0.05) for ICLR 2023 Notable-top-5%, ICLR 2024 Oral, and EMNLP 2023 Main Track. For CoRL 2023 Oral and NeurIPS 2023 Oral, we did not observe statistically significant differences, which may be due to smaller sample sizes. We have added a clarification in the corresponding experiments section of the paper.\n\n> Do the authors have any suggestions on how to incorporate responses to the meta-reviews? The back and forth (e.g., clarifications, additional experiments) between reviewers and authors is an important part of the review process which is omitted using this approach.\n\nThank you for raising the important question of how to incorporate responses to the meta-reviews. In our study, we focus on the **decision-making layer** of peer review—specifically, mechanisms that can rank submissions and support acceptance decisions. While our current framework does not explicitly incorporate such back-and-forth exchanges, we agree that doing so could enrich the review process by capturing meaningful clarifications and author revisions.\n\nOne natural extension would be to introduce a reranking phase after the initial comparisons. Specifically, once authors have responded to meta-reviews, all submissions could be updated and then re-included in a second round of pairwise polling. To better capture the effect of these interactions, both original and revised versions could be included in the reranking phase. Comparing their relative positions allows us to study how author responses influence reviewer preferences.\n\nWe have added the following paragraph to clarify that our framework is intended to broaden, not replace, the collaborative dynamics that characterize high-quality peer review: “ Moreover, we believe the strength of human-led peer review lies not only in final decisions, but also in the human interactions it enables—feedback, discussion, and iterative refinement—that help authors strengthen their work. Through these interactions, reviewers and authors also engage in a shared evaluative practice that reinforces norms, builds trust, and helps individuals identify with the scholarly community [2, 3]. Our intention is not to automate or replace this process, but to broaden how we understand and support scholarly evaluation.” \n\n**References**\n\n[1] Sahand Negahban, Sewoong Oh, and Devavrat Shah. (2017). Rank Centrality: Ranking from Pairwise Comparisons. Oper. Res. 65, 1 (January-February 2017), 266–287. https://doi.org/10.1287/opre.2016.1534\n\n[2] Zuckerman, H., & Merton, R. K. (1971). Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, 66-100.\n\n[3] Lamont, M. (2009). How professors think: Inside the curious world of academic judgment. Harvard University Press."}, {"review_id": "3T9lmHktvJ", "reviewer": "Reviewer_tDGv", "summary": "## Update after rebuttal\n\nGiven that the authors’ responses have addressed some of my concerns, I raise the Quality score from 2 to 3; the Significance score from 1 to 2; and the overall Rating from 2 to 3.\n\nI still hold a negative view on the novelty, but it is subjective -- defer to the Area Chair for the final judgment.\n\n## Summary\n\nThis paper explores the use of LLMs to automate the peer review process. Instead of following the traditional approach of having an LLM evaluate each paper individually, the authors use LLMs to perform **pairwise comparisons between papers**. The comparison results are then aggregated using the Bradley-Terry model to recover a global ranking of the papers. This approach is similar to [1] and its subsequent works, with the key difference being that LLMs, rather than authors, perform the pairwise comparisons.\n\nIn the experimental section, the authors use the number of citations a of paper as the proxy for its quality and evaluate the performance of the proposed system (referred to as the \"GPT ranking system\") against human reviewers. The results show that the average citation count of papers accepted by the GPT ranking system is comparable to that of papers accepted by human reviewers.\n\nIn addition to the main experimental results, the authors also conducted further analyses, such as the discriminative capability of the GPT ranking system and the consistency analysis between the GPT ranking system and human peer review.\n\n[1] Su W. *You are the best reviewer of your own papers: An owner-assisted scoring mechanism*. Advances in Neural Information Processing Systems, 2021, 34: 27929–27939.", "questions": "See Questions in the \"Weaknesses\" paragraph.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Overall Merit\n\nI like this work, especially its experiments. However, the idea presented is not quite novel, and the performance of the proposed method does not significantly exceed the baseline -- human reviewers. I would classify this paper as a solid but ordinary piece of work, and I believe it does not meet the bar for acceptance at NeurIPS.\n\n\n## Strengths\n\n- The experimental section is very thorough and covers most of the key questions that readers may have.\n- The organization and writing of the paper are excellent and very satisfying.\n\n\n## Weaknesses\n\n- The idea of using pairwise comparison to improve the peer review process is not entirely novel—see [1] and its follow-up works.\n- Whether LLMs are truly capable of performing academic paper pairwise comparisons is not very intuitive. Although the authors provide the prompt in the appendix, I would prefer to see a more detailed explanation and analysis in the main text.\n- There are many methods available for recovering rankings from pairwise comparisons. The authors might consider going beyond the Bradley-Terry model.\n- I would like the authors to add a paragraph analyzing how much noise LLM agents introduce when making pairwise comparisons. Since multiple LLM agents were used for comparing each pair of papers (I'm not sure the source of randomness), this analysis likely does not require additional experiments. I would be very interested to see a comparison of the noise from LLM agents vs. human reviewers on the same set of papers.\n- The paper only uses GPT as the LLM agent. I would like to see experimental results using other LLMs as agents, not just GPT.\n- I disagree with using absolute citation count as the evaluation metric for paper quality. Citation counts are heavily influenced by research domain, so I believe that the comparison between \"GPT Ranking System without Area Control\" and human reviewers does not provide meaningful insight.\n- The proposed method, \"GPT Ranking System,\" achieves performance comparable to human reviewers in the experiments. However, this result is obtained by using a large number of LLM agents, whereas human reviewers typically rely on just 3 to 5 reviewers per paper. This suggests that the actual performance of the proposed method may not be as strong as it appears.", "was_revised": true, "final_justification": "## Update after rebuttal\n\nGiven that the authors’ responses have addressed some of my concerns, I raise the Quality score from 2 to 3; the Significance score from 1 to 2; and the overall Rating from 2 to 3.\n\nI still hold a negative view on the novelty, but it is subjective -- defer to the Area Chair for the final judgment.", "rebuttal": "Thank you very much for your thoughtful feedback! We have added several detailed explanations and new experiments to address each of your comments. Please let us know if you have any further questions.\n\n> The idea of using pairwise comparison to improve the peer review process is not entirely novel.\n\nThank you for the comment. While we acknowledge the contributions of prior work on using pairwise comparison in peer review, our method differs in several key ways:\n\n**Shift from Human Ratings to LLM-Driven Comparative Judgments**: Earlier works [1-3] introduce pairwise comparisons by incorporating author-provided rankings to calibrate noisy reviewer scores. In contrast, we fully replace paper-level scores with large-scale pairwise comparisons performed by LLM agents, without relying on human ratings or author input.\n\n**Rethinking the Review Process Design**: While previous studies often aim to improve score accuracy within the traditional workflow (both human and LLM-based peer review), our approach reimagines the decision-making layer itself. We propose a fully comparative mechanism where ranking is derived entirely from relative assessments via the Bradley-Terry model.\n\nWe believe this transition, from replication to redesign, is a meaningful and novel contribution to the literature on LLM-based peer review. We have modified the manuscript to highlight these differences.\n\n> Whether LLMs are truly capable of performing academic paper pairwise comparisons is not very intuitive. Although the authors provide the prompt in the appendix, I would prefer to see a more detailed explanation and analysis in the main text.\n\nThank you for the suggestion. Our approach builds on recent findings that strong LLM judges can reliably perform pairwise comparisons that closely align with human preferences [4]. In addition, prior studies have shown that evaluators are generally more reliable when deciding which of two items is better than when assigning each an absolute score [5]. We also present extensive empirical results in the Experiments Section demonstrating that the system consistently identifies higher-impact papers across multiple conferences and decision settings.\n\nIn the revised manuscript, we have expanded the explanation of our pairwise evaluation setup and discussed the need for future human studies to assess whether LLMs can reliably compare academic papers.\n\n> There are many methods available for recovering rankings from pairwise comparisons. The authors might consider going beyond the Bradley-Terry model.\n\nThank you for the suggestion. While we agree that there exist many approaches for recovering rankings from pairwise comparisons, we chose the Bradley–Terry model because it offers a simple, interpretable, and statistically grounded method, and has been widely adopted in the machine learning community. For example, Chatbot Arena uses the BT model to aggregate pairwise judgments from human voters, and Direct Preference Optimization (DPO) builds on a similar formulation to learn from binary preferences. We believe it is a strong and practical choice, and leave exploration of alternative aggregation methods to future work.\n\n> I would like the authors to add a paragraph analyzing how much noise LLM agents introduce when making pairwise comparisons. Since multiple LLM agents were used for comparing each pair of papers (I'm not sure the source of randomness), this analysis likely does not require additional experiments. I would be very interested to see a comparison of the noise from LLM agents vs. human reviewers on the same set of papers.\n\nThank you for the opportunity to clarify. We clarified that in our method, we uniformly random-sample from all pairs and then use the Bradley–Terry model to recover global rankings from sparse but representative observations. We empirically show that the required sample size for a high-quality ranking is significantly smaller than the total number of possible combinations (Figure 2). The scaling performance of our framework also allows us to study how noise in individual pairwise judgments affects the overall ranking quality. As a result, each paper pair is evaluated only once with high probability. This sampling process helps minimize the potential for bias or noise that could arise from repeated evaluations of the same paper pairs. \n\nWhile we have not directly compared the noise introduced by LLM agents and human reviewers on the same set of papers, we agree that such a comparison would provide valuable insights into the relative consistency of both methods. We suggest that future work explore this comparison to better understand how noise in LLM-based pairwise comparisons compares to human evaluation.\n\n> The paper only uses GPT as the LLM agent. I would like to see experimental results using other LLMs as agents, not just GPT.\n\nThank you for the suggestion. We have conducted experiments to test two additional models other than GPT-4o mini: Gemini 2.0 Flash and Claude-3-haiku-20240307. We scaled the number of comparisons to over $10^6$ and found that, in both cases, our pairwise comparison framework performs significantly better than the GPT rating system baseline. Specifically, papers selected by Gemini 2.0  Flash receive an average of 18.3 citations (vs 11.4), while those chosen by Claude‑3‑Haiku average 16.8 (vs 11.4). These results suggest that the advantage of our approach is robust across different LLMs.\n\n> I disagree with using absolute citation count as the evaluation metric for paper quality. Citation counts are heavily influenced by research domain, so I believe that the comparison between \"GPT Ranking System without Area Control\" and human reviewers does not provide meaningful insight.\n\nThank you for raising this important point. We acknowledge that citation counts, while widely used, are an imperfect reflection of research quality. As described in Appendix A.1, they are affected by numerous factors beyond quality, such as the visibility of research communities, research domain, etc. \n\nTo further verify the robustness of our proposed method, we conducted experiments using alternative measures. Specifically, we measured the Spearman correlation between our system’s output and human review scores. In both ICLR 2023 and ICLR 2024, the BT scores produced by our mechanism show moderate correlation with the average human ratings—27% and 24%, respectively. This indicates that our system aligns with signals in the current review process, while not fully replicating its outcomes. We believe this complementary nature is essential for identifying potential improvements to existing peer review systems.\n\nMoreover, we  added explicit clarification to emphasize that these metrics are used as a practical means of evaluation: “ It is important to note that while we use proxy metrics—such as future citation counts—to evaluate our proposed mechanism, this represents an exploratory and pragmatic step rather than an ideal. The gold standard would be direct human evaluation of how well a mechanism supports the selection and improvement of high-quality research, but such evaluations are costly and difficult to scale . Proxy metrics offer a tractable, though imperfect, way to approximate long-term impact and anticipate how different mechanisms might perform. Ideally, a strong alternative review mechanism should correlate with—but not fully replicate—the outcomes of the current system, offering complementary perspectives and surfacing different strengths. As we show, our proposed pairwise comparison mechanism achieves this balance, aligning with current human judgments while introducing useful differentiation.” \n\n> The proposed method, \"GPT Ranking System,\" achieves performance comparable to human reviewers in the experiments. However, this result is obtained by using a large number of LLM agents, whereas human reviewers typically rely on just 3 to 5 reviewers per paper. This suggests that the actual performance of the proposed method may not be as strong as it appears.\n\nThank you for the comment. We agree that our system utilizes a larger number of LLM agents than the typical number of human reviewers per paper. However, we believe this reflects a fundamental trade-off between time, cost, and scalability. Conventional peer review is an expensive and time-consuming process—each human review takes several hours on average and incurs substantial labor and opportunity costs [6-7]. This challenge is becoming increasingly pronounced due to the rapid growth of research output and a persistent shortage of qualified reviewers. \n\nIn contrast, our GPT ranking system leverages the scalability of LLM agents to perform thousands of pairwise comparisons, enabling the construction of a global ranking that would be impractical for human committees. This demonstrates the potential for scalable, low-cost review mechanisms, which may complement traditional peer review in settings where efficiency and volume are prioritized. Rather than replicating the conventional peer review process, our goal is to explore how LLM-based systems can be designed to generate effective, reliable assessments.\n\n[1] Su, Weijie. \"You are the best reviewer of your own papers: An owner-assisted scoring mechanism.\" NeurIPS 2021\n\n[2] Yan et al. \"Isotonic mechanism for exponential family estimation in machine learning peer review.\" J. R. Stat. Soc. Ser. B Methodol 2025\n\n[3] Wu et al. \"A truth serum for eliciting self-evaluations in scientific reviews.\" arXiv:2306.11154 (2023)\n\n[4] Zheng et al. \"Judging llm-as-a-judge with mt-bench and chatbot arena.\" NeurIPS 2023\n\n[5] Carterette et al. \"Here or there: Preference judgments for relevance.\" ECIR 2008, Springer\n\n[6] LeBlanc et al. \"Scientific sinkhole: estimating the cost of peer review based on survey data with snowball sampling.\" Res. Integr. Peer Rev. 2023\n\n[7] Aczel et al. \"A billion-dollar donation: estimating the cost of researchers’ time spent on peer review.\" Res. Integr. Peer Rev. 2021"}, {"review_id": "AQMDizZYOI", "reviewer": "Reviewer_gH5a", "summary": "In this paper, the authors propose a novel peer review mechanism that replaces traditional per-paper scoring with pairwise comparisons conducted by LLM agents. Instead of assigning absolute ratings, each agent compares two papers and outputs a binary preference. These comparisons are aggregated using the Bradley-Terry model to generate a global ranking of submissions. The approach is empirically validated on papers from major ML and NLP conferences, showing improved alignment with citation-based impact metrics compared to standard rating-based LLM review pipelines. Notably, the authors identify tradeoffs in the system: while it scales well and improves consistency, it tends to favor applied research over theoretical work, reduces topic novelty, and amplifies institutional inequality—highlighting both the promise and limitations of LLM-based evaluation systems.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well written, with clear pointers to mathematical details where appropriate.\n\nThe paper proposes a fundamental shift from traditional rating-based evaluation to a pairwise ranking system using LLM agents.\n\nThe authors conduct comprehensive experiments across multiple top-tier conference datasets, showing that the proposed method achieves acceptance patterns and impact levels comparable to those of human reviewers.\n\nBeyond reporting performance metrics, the paper analyzes biases in topic novelty, institutional concentration, and research area preferences, contributing to the ongoing discussion of fairness and diversity in AI-assisted evaluation systems.", "was_revised": true, "final_justification": "I have read the authors' rebuttal. My initial review was highly positive, and I am maintaining my recommendation to accept. The paper's core strength lies not only in its novel and well-validated pairwise comparison method but also in its crucial and transparent analysis of the system's inherent biases. As I did not raise any major concerns, my positive assessment stands unchanged.", "rebuttal": "Thank you for your feedback and the time you invested in reviewing our manuscript. We are delighted that you found our paper to be well-written, clear, and compelling. If there are specific concerns or areas that were unclear, we would be happy to address them in detail."}]

Author rebuttal:

Meta-review: This paper proposes using LLM agents to perform pairwise comparisons of manuscripts rather than individual scoring. This new approach to peer review demonstrates improved identification of high-impact papers across multiple conferences compared to rating-based methods, while the author(s) analyzed concerning biases including reduced novelty and institutional imbalance. Despite reviewer concerns about limited novelty and evaluation metrics, the thorough experimental validation across different LLMs and the comprehensive examination of system limitations are solid contributions. Neverthess, the author(s) should address concerns raised by the reviewers in the final version.
Decision: Accept (poster)