ReviewScore: Misinformed Peer Review Detection with Large Language
  Models
Authors: name: Hyun Ryu; name: Doohyuk Jang; name: Hyemin S. Lee; name: Joonhyun Jeong; name: Gyeongman Kim; name: Donghyeon Cho; name: Gyouk Chu; name: Minyeong Hwang; name: Hyeongwon Jang; name: Changhun Kim; name: Haechan Kim; name: Jina Kim; name: Joowon Kim; name: Yoonjeon Kim; name: Kwanhyung Lee; name: Chanjae Park; name: Heecheol Yun; name: Gregor Betz; name: Eunho Yang
Published: 2025-09-25 22:55:05

Abstract: Peer review serves as a backbone of academic research, but in most AI
conferences, the review quality is degrading as the number of submissions
explodes. To reliably detect low-quality reviews, we define misinformed review
points as either "weaknesses" in a review that contain incorrect premises, or
"questions" in a review that can be already answered by the paper. We verify
that 15.2% of weaknesses and 26.4% of questions are misinformed and introduce
ReviewScore indicating if a review point is misinformed. To evaluate the
factuality of each premise of weaknesses, we propose an automated engine that
reconstructs every explicit and implicit premise from a weakness. We build a
human expert-annotated ReviewScore dataset to check the ability of LLMs to
automate ReviewScore evaluation. Then, we measure human-model agreements on
ReviewScore using eight current state-of-the-art LLMs and verify moderate
agreements. We also prove that evaluating premise-level factuality shows
significantly higher agreements than evaluating weakness-level factuality. A
thorough disagreement analysis further supports a potential of fully automated
ReviewScore evaluation.