ReviewerToo: Should AI Join The Program Committee? A Look At The Future
  of Peer Review
Authors: name: Gaurav Sahu; name: Hugo Larochelle; name: Laurent Charlin; name: Christopher Pal
Published: 2025-10-09 23:53:19

Abstract: Peer review is the cornerstone of scientific publishing, yet it suffers from
inconsistencies, reviewer subjectivity, and scalability challenges. We
introduce ReviewerToo, a modular framework for studying and deploying
AI-assisted peer review to complement human judgment with systematic and
consistent assessments. ReviewerToo supports systematic experiments with
specialized reviewer personas and structured evaluation criteria, and can be
partially or fully integrated into real conference workflows. We validate
ReviewerToo on a carefully curated dataset of 1,963 paper submissions from ICLR
2025, where our experiments with the gpt-oss-120b model achieves 81.8% accuracy
for the task of categorizing a paper as accept/reject compared to 83.9% for the
average human reviewer. Additionally, ReviewerToo-generated reviews are rated
as higher quality than the human average by an LLM judge, though still trailing
the strongest expert contributions. Our analysis highlights domains where AI
reviewers excel (e.g., fact-checking, literature coverage) and where they
struggle (e.g., assessing methodological novelty and theoretical
contributions), underscoring the continued need for human expertise. Based on
these findings, we propose guidelines for integrating AI into peer-review
pipelines, showing how AI can enhance consistency, coverage, and fairness while
leaving complex evaluative judgments to domain experts. Our work provides a
foundation for systematic, hybrid peer-review systems that scale with the
growth of scientific publishing.