ReviewScore: Misinformed Peer Review Detection with Large Language
  Models
Authors: name: Hyun Ryu, name: Doohyuk Jang, name: Hyemin S. Lee, name: Joonhyun Jeong, name: Gyeongman Kim, name: Donghyeon Cho, name: Gyouk Chu, name: Minyeong Hwang, name: Hyeongwon Jang, name: Changhun Kim, name: Haechan Kim, name: Jina Kim, name: Joowon Kim, name: Yoonjeon Kim, name: Kwanhyung Lee, name: Chanjae Park, name: Heecheol Yun, name: Gregor Betz, name: Eunho Yang
Organization: KAIST AI

Peer review serves as a backbone of academic research, but in most AI
conferences, the review quality is degrading as the number of submissions
explodes. To reliably detect low-quality reviews, we define misinformed review
points as either "weaknesses" in a review that contain incorrect premises, or
"questions" in a review that can be already answered by the paper. We verify
that 15.2% of weaknesses and 26.4% of questions are misinformed and introduce
ReviewScore indicating if a review point is misinformed. To evaluate the
factuality of each premise of weaknesses, we propose an automated engine that
reconstructs every explicit and implicit premise from a weakness. We build a
human expert-annotated ReviewScore dataset to check the ability of LLMs to
automate ReviewScore evaluation. Then, we measure human-model agreements on
ReviewScore using eight current state-of-the-art LLMs and verify moderate
agreements. We also prove that evaluating premise-level factuality shows
significantly higher agreements than evaluating weakness-level factuality. A
thorough disagreement analysis further supports a potential of fully automated
ReviewScore evaluation.