Title: From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Peer Review Redesign

Comparative Paper Evaluation
Abstract: The advent of large language models (LLMs) offers unprecedented opportunities to reimagine peer review beyond the constraints of traditional workflows.
Despite these opportunities, prior efforts have largely focused on replicating traditional review workflows with LLMs serving as direct substitutes for human reviewers, while limited attention has been given to exploring new paradigms that fundamentally rethink how LLMs can participate in the academic review process.
In this paper, we introduce and explore a novel mechanism that employs LLM agents to perform pairwise comparisons among manuscripts instead of individual scoring. By aggregating outcomes from substantial pairwise evaluations, this approach enables a more accurate and robust measure of relative manuscript quality.
Our experiments demonstrate that this comparative approach significantly outperforms traditional rating-based methods in identifying high-impact papers. However, our analysis also reveals emergent biases in the selection process, notably a reduced novelty in research topics and an increased institutional imbalance. These findings highlight both the transformative potential of rethinking peer review with LLMs and critical challenges that future systems must address to ensure equity and diversity.


Peer reviews:
comments: SUMMARY:
In this paper, the authors propose a novel peer review mechanism that replaces traditional per-paper scoring with pairwise comparisons conducted by LLM agents. Instead of assigning absolute ratings, each agent compares two papers and outputs a binary preference. These comparisons are aggregated using the Bradley-Terry model to generate a global ranking of submissions. The approach is empirically validated on papers from major ML and NLP conferences, showing improved alignment with citation-based impact metrics compared to standard rating-based LLM review pipelines. Notably, the authors identify tradeoffs in the system: while it scales well and improves consistency, it tends to favor applied research over theoretical work, reduces topic novelty, and amplifies institutional inequality—highlighting both the promise and limitations of LLM-based evaluation systems.

STRENGTHS AND WEAKNESSES:
The paper is well written, with clear pointers to mathematical details where appropriate.

The paper proposes a fundamental shift from traditional rating-based evaluation to a pairwise ranking system using LLM agents.

The authors conduct comprehensive experiments across multiple top-tier conference datasets, showing that the proposed method achieves acceptance patterns and impact levels comparable to those of human reviewers.

Beyond reporting performance metrics, the paper analyzes biases in topic novelty, institutional concentration, and research area preferences, contributing to the ongoing discussion of fairness and diversity in AI-assisted evaluation systems.

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Update after rebuttal\n\nGiven that the authors’ responses have addressed some of my concerns, I raise the Quality score from 2 to 3; the Significance score from 1 to 2; and the overall Rating from 2 to 3.\n\nI still hold a negative view on the novelty, but it is subjective -- defer to the Area Chair for the final judgment.\n\n## Summary\n\nThis paper explores the use of LLMs to automate the peer review process. Instead of following the traditional approach of having an LLM evaluate each paper individually, the authors use LLMs to perform **pairwise comparisons between papers**. The comparison results are then aggregated using the Bradley-Terry model to recover a global ranking of the papers. This approach is similar to [1] and its subsequent works, with the key difference being that LLMs, rather than authors, perform the pairwise comparisons.\n\nIn the experimental section, the authors use the number of citations a of paper as the proxy for its quality and evaluate the performance of the proposed system (referred to as the \"GPT ranking system\") against human reviewers. The results show that the average citation count of papers accepted by the GPT ranking system is comparable to that of papers accepted by human reviewers.\n\nIn addition to the main experimental results, the authors also conducted further analyses, such as the discriminative capability of the GPT ranking system and the consistency analysis between the GPT ranking system and human peer review.\n\n[1] Su W. *You are the best reviewer of your own papers: An owner-assisted scoring mechanism*. Advances in Neural Information Processing Systems, 2021, 34: 27929–27939.\n\nSTRENGTHS AND WEAKNESSES:\n## Overall Merit\n\nI like this work, especially its experiments. However, the idea presented is not quite novel, and the performance of the proposed method does not significantly exceed the baseline -- human reviewers. I would classify this paper as a solid but ordinary piece of work, and I believe it does not meet the bar for acceptance at NeurIPS.\n\n\n## Strengths\n\n- The experimental section is very thorough and covers most of the key questions that readers may have.\n- The organization and writing of the paper are excellent and very satisfying.\n\n\n## Weaknesses\n\n- The idea of using pairwise comparison to improve the peer review process is not entirely novel—see [1] and its follow-up works.\n- Whether LLMs are truly capable of performing academic paper pairwise comparisons is not very intuitive. Although the authors provide the prompt in the appendix, I would prefer to see a more detailed explanation and analysis in the main text.\n- There are many methods available for recovering rankings from pairwise comparisons. The authors might consider going beyond the Bradley-Terry model.\n- I would like the authors to add a paragraph analyzing how much noise LLM agents introduce when making pairwise comparisons. Since multiple LLM agents were used for comparing each pair of papers (I'm not sure the source of randomness), this analysis likely does not require additional experiments. I would be very interested to see a comparison of the noise from LLM agents vs. human reviewers on the same set of papers.\n- The paper only uses GPT as the LLM agent. I would like to see experimental results using other LLMs as agents, not just GPT.\n- I disagree with using absolute citation count as the evaluation metric for paper quality. Citation counts are heavily influenced by research domain, so I believe that the comparison between \"GPT Ranking System without Area Control\" and human reviewers does not provide meaningful insight.\n- The proposed method, \"GPT Ranking System,\" achieves performance comparable to human reviewers in the experiments. However, this result is obtained by using a large number of LLM agents, whereas human reviewers typically rely on just 3 to 5 reviewers per paper. This suggests that the actual performance of the proposed method may not be as strong as it appears.\n\nQUESTIONS:\nSee Questions in the \"Weaknesses\" paragraph.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new LLM-based pairwise comparison mechanism to rank manuscripts. The LLMs are prompted to give a meta-review + reasoning for each paper, followed by a binary indicator to show the preferred manuscript. Based on these scores, the authors estimate a Bradley-Terry model to recover the manuscript ranking from these pairwise comparisons. In their experiments, the authors use OpenReview data, including ICLR, NeurIPS, CoRL, and EMNLP, and GPT-4o mini (context window: 128K tokens). They empirically find that the theoretical burden of the O(n^2) scaling does not apply as with less than 2% of the samples a high-quality ranking is recovered. The authors validate the LLM-based ranking via citation metrics and human reviewer decisions (e.g., accepted vs. rejected, main vs. findings track, oral vs. poster, etc.). Finally, the authors also show that this mechanism tends to reduce novelty (measured via distance in word embeddings) and increases institutional imbalance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- I like how the authors do not try to replicate the existing review process, but argue that LLMs may offer the opportunity to redesign the entire process, which they argue mostly reflects historical compromises rather than principled design choices.\n- The authors propose a clear methodology and empirically validate it. For example, showing that the theoretical burden of the O(n^2) scaling does not apply in practice.\n- I like Appendix Section A2. While more speculative (and therefore probably part of the appendix), I like that the authors try to assess the real-world implications of this pairing scheme (which at first might seem limited because this is not how we currently conduct peer review).\n\nWeaknesses:\n- Since we have no access to the ground truth when implementing this system in real time, it would be helpful if the authors could offer either theoretical bounds or heuristics on how many pairs should be evaluated before we converge to some high-quality ranking.\n- The authors acknowledge this limitation already partially in Appendix Section A1, but still the endogeneity effect of citations (being accepted to the conference may bump citations) and using human reviewer decisions (e.g., accept vs reject) as a validation criterion seems to contradict the statement that current peer review is failing us as we try to replicate its outcomes. Maybe the authors should clarify the stance that the mechanism how we conduct peer review could be improved, but the outcomes themselves we're trying to replicate.\n\nQUESTIONS:\n- Are the differences in the novel research topics statistically significant? There seems to be a large overlap between the distributions.\n- Do the authors have any suggestions on how to incorporate responses to the meta-reviews? The back and forth (e.g., clarifications, additional experiments) between reviewers and authors is an important part of the review process which is omitted using this approach.\n\nLIMITATIONS:\nThe authors discuss the limitations in Appendix Section A1.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a novel redesign of LLM-based peer review by shifting from conventional per-paper scoring to pairwise comparison of manuscripts. Instead of assigning absolute scores, LLM agents compare two papers at a time and vote for the better one. These results are aggregated using the Bradley–Terry model to produce a global ranking of all submissions. Experiments on major ML conference papers (e.g., ICLR, NeurIPS, EMNLP) demonstrate that this method better identifies high-impact papers (measured by citation count) than traditional rating-based GPT review systems. However, the method also exhibits biases: a preference for less novel topics and overrepresentation of papers from elite institutions.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The motivation is clear and make sense. The authors argue convincingly that the current peer review system is a historically contingent artifact, and LLMs allow rethinking the entire structure.
2. The paper is well-written. 
3. The idea is simple but novel in review paradigm: The paper proposes a genuinely different peer review workflow using pairwise comparisons, diverging from the current mainstream of LLM-based review replication.

Weaknesses:
1. Bias Amplification: The approach tends to favor papers from top institutions and penalize novelty, raising serious fairness and diversity concerns.

2. Limited Model Diversity: Experiments rely solely on GPT-4o mini. It's unclear whether the findings generalize across other LLMs.

3. Citation as Sole Impact Metric: The paper assumes that citation count is an accurate proxy for quality, which can be problematic due to citation inflation, visibility bias, and Matthew effects.

QUESTIONS:
see weakness

LIMITATIONS:
no
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-11