ReviewScore: Misinformed Peer Review Detection with Large Language Models

## Abstract

Abstract Peer review serves as a backbone of academic research, but in most AI conferences, the review quality is degrading as the number of submissions explodes. To reliably detect low-quality reviews, we define misinformed review points as either “weaknesses” in a review that contain incorrect premises, or “questions” in a review that can be already answered by the paper. We verify that 15.2% of weaknesses and 26.4% of questions are misinformed and introduce ReviewScore indicating if a review point is misinformed . To evaluate the factuality of each premise of weaknesses, we propose an automated engine that reconstructs every explicit and implicit premise from a weakness. We build a human expert-annotated ReviewScore dataset to check the ability of LLMs to automate ReviewScore evaluation. Then, we measure human-model agreements on ReviewScore using eight current state-of-the-art LLMs and verify moderate agreements. We also prove that evaluating premise-level factuality shows significantly higher agreements than evaluating weakness-level factuality. A thorough disagreement analysis further supports a potential of fully automated ReviewScore evaluation. 1 1 1 We will make the source code and dataset publicly available.

## 1 Introduction

Peer review is an evaluation of academic work by experts to assess its originality, significance, and validity before publication (Kelly et al., 2014; Taylor & Francis, 2025).
In AI conferences, as the number of submissions is exponentially increasing, required number of reviewers is also exploding.
As a result, review quality is degraded, which undermines the integrity and reliability of a peer-review system (Stelmakh et al., 2021; Cortes & Lawrence, 2021; Shah, 2022; Kim et al., 2025).
Due to the importance of the issue, previous works propose criteria for evaluating review quality. However, we observe a trade-off between applicability to reviews and specificity of rubrics. Goldberg et al. (2025) and Du et al. (2024) proposes criteria that could apply to nearly every review but those are quite vague and subjective. Guo et al. (2023), Sadallah et al. (2025), Purkayastha et al. (2025), and Ou et al. (2025) introduce specific and objective criteria but those target narrow scope of reviews.
If we could reliably measure review quality based on specific yet applicable criteria, we could filter out misinformed reviews or give feedback to reviewers before the reviews are released.

To resolve this issue, we introduce two specific yet applicable criteria of a review quality: unanswerability of questions and factuality of weaknesses.
To select the criteria, we recruited a group of human experts and let them independently analyze a small subset of ICLR reviews(^2^22A group of three graduate students studying AI analyzed reviews of ICLR submissions available in OpenReview. Detailed process of this group work is described in Appendix [B](https://arxiv.org/html/2509.21679v1#A2).).
Specifically, each of them decomposed a review into review points, which are formally defined as follows.

###### Definition 1 (Review Point) .

A review point is a single, self-contained unit of evaluation or inquiry in a review–either a weakness or a question–that stands on its own semantically.

Each human evaluated quality of review points, and they discussed trustworthy criteria to detect low-quality.
Based on the discussion, we formally define a misinformed review point as follows.

###### Definition 2 (Misinformed Review Point) .

A review point is *misinformed* if and only if

- •
a question stated in a review can be already answered by the paper, or
- •
a weakness stated in a review is incorrect or contains incorrect premises regarding the paper.

In this work, the human annotation shows that 26.4% of questions and 15.2% of weaknesses are misinformed, which means that the current AI conference reviews contain considerable amount of misinformed review points.(^3^33A group of 15 graduate students annotate reviews of 40 works submitted to ICLR 2021-2023. Detailed process of this human annotation is described in Section [3.3](https://arxiv.org/html/2509.21679v1#S3.SS3).)
We also note that these criteria aligns with reviewer guidelines of major AI conferences. For instance, ACL 2023 Peer Review Policies indicate that “before writing a negative review, check whether your questions are already answered.” (Boyd-Graber et al., 2023), and NeurIPS 2025 reviewer guidelines indicate that “reviewers should minimize the chance of misunderstandings during the reviewing process” (NeurIPS 2025 Program Chairs, 2025).

Figure: Figure 1: Overview of ReviewScore. Each review point in a review is categorized into question and weakness. We further categorize weakness into claim and argument by the presence of supporting reasons. Based on an appropriate knowledge base, if a question is answerable by the paper, a claim is factually incorrect, or an argument contains factually incorrect premises, then the review point is misinformed. For arguments, to extract all explicit and implicit premises, we also introduce an automatic argument reconstruction engine.
Refer to caption: x1.png

Based on these observations, we define ReviewScore that indicates if a review point is misinformed (Section [3.1](https://arxiv.org/html/2509.21679v1#S3.SS1)).
First, we define Base ReviewScore by directly applying the definition of misinformed review points in a 5-point scale.
However, we discovered human-annotated factuality often diverges. This is because a weakness often contains both correct and incorrect explicit premises or incorrect implicit premises, which hinders humans to reliably annotate factuality.
To resolve this issue, we further define Advanced ReviewScore as an aggregation of premise factuality scores.
If a weakness has no supporting reason, we call it as a claim, and if a weakness consists of more than one premises, we call it as an argument.
We adopt two aggregation methods, logical conjunction, following the literature of logic (Beall et al., 2024), and weighted average, maintaining a 5-point scale (Figure [1](https://arxiv.org/html/2509.21679v1#S1.F1)).
Before evaluating Advanced ReviewScore, we have to extract all explicit and implicit premises from an argument. This process is called argument reconstruction in logic and critical thinking (Brun & Betz, 2016; Groarke, 2024; Dowden, 2024), and we construct an engine that automatically does this process (Section [3.2](https://arxiv.org/html/2509.21679v1#S3.SS2)).
To fully automate misinformed review detection with LLMs, we should verify if model-estimated ReviewScore align with human-annotated ReviewScore. To do that, we construct a human expert-annotated ReviewScore dataset based on ICLR reviews (Section [3.3](https://arxiv.org/html/2509.21679v1#S3.SS3)). We recruited 15 experienced graduate students and they dedicated total 244 hours for trustworthy human annotation.

We verify the reliability of automatic evaluation of ReviewScore by measuring human-model agreements (Section [4](https://arxiv.org/html/2509.21679v1#S4)). We use eight current state-of-the-art models, including five proprietary and three open-sourced models, and confirm moderate agreements with the human experts. Advanced ReviewScore clearly outperforms Base ReviewScore, which proves the effectiveness of premise-level factuality scoring. We also conduct a thorough human-model disagreement analysis and observe that models sometimes misunderstand the meaning of review points or predict scores that minority human annotators give.

To summarize our contributions:

- 1.
We introduce ReviewScore, a novel evaluation criteria that detects misinformed review points (i.e., questions that can be answered by the paper or weaknesses with incorrect premises).
- 2.
To evaluate the factuality of premises, we propose an automatic argument reconstruction engine that generates a valid and faithful set of premises and conclusion.
- 3.
We construct a trustworthy human expert-annotated dataset to measure the reliability of automatic evaluation of ReviewScore.
- 4.
We validate the reliability of automatic ReviewScore evaluation by using eight current state-of-the-art LLMs and conducting a comprehensive human-model disagreement analysis.

###### Definition 1 (Review Point) .

###### Definition 2 (Misinformed Review Point) .

## 2 Related Works

#### Peer review evaluation.

Previous works studied how to evaluate the quality of peer reviews.
Goldberg et al. (2025) introduces four criteria, understanding, substantiation, coverage, and constructiveness. However, they focus on evaluating review as a whole, not each individual review point, which makes difficult to clarify which specific points are misinformed. They also do not use LLMs to automate evaluation.
Guo et al. (2023) specifically focuses on evaluating substantiation. However, they evaluate whether each review point has supporting evidence, not whether the supporting pieces of evidence are factually correct, leaving it as a future work. Our work’s factuality criteria exactly covers this part.
Sadallah et al. (2025) proposes four criteria, actionability, grounding & specificity, verifiability, and helpfulness. However, they only focus on evaluating weaknesses, excluding questions, and as in Guo et al. (2023), they only evaluate whether weaknesses are grounded and supported, not whether those are factually correct.
Purkayastha et al. (2025) defines lazy thinking in the context of NLP paper reviews and introduces heuristic-based rubrics. However, their criteria is only applicable to NLP domain which is hard to be generalized into other domains.

#### Argument evaluation.

In logic and critical thinking, an argument is a list of statements, one of which is the conclusion and the others are the premises (Dutilh Novaes, 2022; Lau & Chan, 2025).
To evaluate an argument, we need to follow two steps. First, we have to identify and reconstruct the argument into a set of premises and conclusion, which is called an argument reconstruction (Brun & Betz, 2016; Groarke, 2024; Dowden, 2024). Then, we evaluate whether each premise is factually correct.
An argument reconstruction should both be valid, which means premises deductively imply a conclusion, and faithful, which means premises and a conclusion accurately and completely represents an original argument (Brun & Betz, 2016; Betz & Richardson, 2021).
Previously, Betz & Richardson (2021) trains a T5 model for argument reconstruction. However, the training datasets are either synthetic or polished and the reconstruction do not require any additional context information. In contrast, our work targets peer reviews, which include real-world unpolished arguments, and the reconstruction requires an entire paper to fully understand the context of arguments.

## 3 ReviewScore

We newly define ReviewScore that measures how misinformed a review point is (Section [3.1](https://arxiv.org/html/2509.21679v1#S3.SS1)). To evaluate ReviewScore, we also introduce an engine that automatically extracts every explicit and implicit premises from a weakness (Section [3.2](https://arxiv.org/html/2509.21679v1#S3.SS2)). Lastly, we construct a human expert-annotated dataset that evaluates LLM’s ability to evaluate ReviewScore (Section [3.3](https://arxiv.org/html/2509.21679v1#S3.SS3)).

### 3.1 Definition

Our goal of defining ReviewScore is to detect misinformed review points.
Following this goal and the review quality criteria discussion in Section [1](https://arxiv.org/html/2509.21679v1#S1), we first define Base ReviewScore as factuality of weaknesses and unanswerability of questions. The following definition formally describes it.

###### Definition 3 ( Base ReviewScore ) .

Let $x$ be a review point (either a weakness or a question) about a submitted paper $S$. Define

$$ $\operatorname{Factuality}_{S}:\mathcal{W}\to\{1,2,3,4,5\},\qquad\operatorname{Unanswerability}_{S}:\mathcal{Q}\to\{1,2,3,4,5\},$ $$

where $\mathcal{W}$ and $\mathcal{Q}$ are, respectively, the sets of weaknesses and questions appearing in a review of $S$(^4^44Detailed rubric is described in Appendix [H](https://arxiv.org/html/2509.21679v1#A8).).
We considered score 1–2 as Misinformed and score 3–5 as Not misinformed for binary classification setup.
The Base ReviewScore of $x$ is:

$$ $\operatorname{ReviewScore}_{\text{base}}(x)=\begin{cases}\operatorname{WScore}(x)=\operatorname{Factuality}_{S}(x),&\text{if }x\in\mathcal{P}\ \text{(weakness)},\[6.0pt] \operatorname{QScore}(x)=\operatorname{Unanswerability}_{S}(x),&\text{if }x\in\mathcal{Q}\ \text{(question)}.\end{cases}$ $$

However, during the group discussion in Section [1](https://arxiv.org/html/2509.21679v1#S1), we discovered that human annotators’ evaluations on factuality diverge if a weakness contains both factual and nonfactual premises or a nonfactual premise is implicitly presumed.
It happens since the human annotators implicitly weigh the importance of underlying premises of a weakness and then decide the final factuality score.

To resolve this issue, we further define Advanced ReviewScore.
We categorize weaknesses into arguments and claims based on whether there are supporting reasons or not. We keep the definition of WScore to evaluate claims, but further develops a finer-grained score to evaluate arguments. Following the literature of critical thinking (Brun & Betz, 2016; Groarke, 2024; Dowden, 2024), we reconstruct an argument into a premise-conclusion structure and then define Advanced ReviewScore for arguments as (an aggregation of) factuality of premises. The following definition formally describes it.

###### Definition 4 ( Advanced ReviewScore ) .

Let $x$ be a review point about a submitted paper $S$. Let $\mathcal{C}$, $\mathcal{A}$, and $\mathcal{Q}$ denote, respectively, the sets of claims, arguments, and questions in a review of $S$.
For $x\in\mathcal{A}$, let its (explicit and implicit) premises be
$\mathcal{P}(x)=\{p_{1},\dots,p_{k}\}$ with $\{p_{1},\dots,p_{k}\}\vdash C$ for the conclusion $C$ of $x$.
Let $\mathcal{K}$ be the set of knowledge bases available for factuality judgments (i.e., $S$, annotator knowledge, referred papers), and let

$$ $\operatorname{Factuality}:\ \mathcal{U}\times\mathcal{K}\to\{1,2,3,4,5\},\qquad\mathcal{U}\coloneqq\mathcal{C}\ \cup\ \bigcup_{x\in\mathcal{A}}\mathcal{P}(x),$ $$

be a 5-point scoring function for claims and premises given a knowledge base.
Define a selector $\mathrm{KB}:\mathcal{U}\to\mathcal{K}$ that chooses the knowledge base used
for each item (for claims $x\in\mathcal{C}$, $\mathrm{KB}(x)=S$; for premises $p_{i}$,
$\mathrm{KB}(p_{i})=\mathrm{KB}_{i}\in\mathcal{K}$).
For a given $\mathrm{KB}(x)\in\mathcal{K}$, we notate the factuality function as $\operatorname{Factuality}_{\mathrm{KB}(x)}(\cdot)$.
$\mathrm{Agg}$ is an operator that aggregates a list of scores to a single score. The Advanced ReviewScore of $x$ is:

$$ $\operatorname{ReviewScore}_{\mathrm{adv}}(x)=\begin{cases}\operatorname{ClaimScore}(x)=\operatorname{Factuality}_{S}(x),&\text{if }x\in\mathcal{C}\ \text{(claim)},\[8.0pt] \operatorname{ArgScore}(x)=\mathrm{Agg}\!\left(\operatorname{Factuality}_{\mathrm{KB}_{i}}(p_{i})_{p_{i}\in\mathcal{P}(x)}\right),&\text{if }x\in\mathcal{A}\ \text{(argument)},\[12.0pt] \operatorname{QScore}(x)=\operatorname{Unanswerability}_{S}(x),&\text{if }x\in\mathcal{Q}\ \text{(question)}.\end{cases}$ $$

###### Definition 3 ( Base ReviewScore ) .

###### Definition 4 ( Advanced ReviewScore ) .

#### Aggregation methods.

To aggregate premise factuality scores as a single ArgScore, we introduce two aggregation methods: logical conjunction and weighted average.
Following the literature of logic, an argument is true if and only if all premises are true.
We define a premise is true if and only if it has factuality score 3–5, and otherwise, it is false.
We dubbed this aggregation as logical conjunction, which follows the binary classification setup.
However, if an annotator mistakenly evaluates one of premises, then the error propagates to the entire argument.
To alleviate this issue, we also aggregate by a weighted average. Since it is difficult to measure the importance of premises, we instead weighted scores by untrivialness of premises (with a scale of 0–2). This is intended to simply filter out trivially true premises by measuring their importance as 0.(^5^55Since we reconstruct every argument as valid (i.e., a set of premises deductively implies a conclusion), there are often conditional premises that make an argument valid but are trivially true. Detailed rubric of untrivialness is described in Appendix [H](https://arxiv.org/html/2509.21679v1#A8).)

### 3.2 Automatic Argument Reconstruction

Figure: (a)
Refer to caption: x2.png

To evaluate ArgScore, we have to extract (explicit and implicit) premises $\mathcal{P}(x)$ from an argument $x\in\mathcal{A}$. Since human experts require significant amount of time and costs to do this, we automate it using LLMs.
First, we check if a model could directly reconstruct an argument by giving detailed instructions. To preserve the context of an argument, we also give the model a submitted paper $S$. However, it mostly fails to generate valid and faithful reconstructions(^6^66Detailed numerical results and qualitative analysis are reported in Appendix [C](https://arxiv.org/html/2509.21679v1#A3).).
To resolve this issue, we add two feedback loops to ensure validity and faithfulness of an argument reconstruction (Figure [2(a)](https://arxiv.org/html/2509.21679v1#S3.F2.sf1)).

An LLM alone often falls short in ensuring validity, so we include a SAT solver which could automatically judges the validity of a set of premises and a conclusion without any logical errors.
To do that, an LLM translates a set of natural language (NL) premises and conclusion into first-order logic (FOL) formulas. Then, a SAT solver decides whether the premises deductively implies the conclusion.
If the reconstruction is valid, then an LLM translates formalized premises and conclusion back into NL domain. This process is called logical streamlining, which means to rephrase NL premises or conclusion in order to make their logico-semantic structure more transparent (Bowell & Kemp, 2014; Brun & Betz, 2016; Betz & Richardson, 2021). We then pass these streamlined NL premises and conclusion to the subsequent faithfulness feedback loop.
However, if the reconstruction is invalid, then we feed a rule-based reward signal to the argument reconstructor to regenerate the reconstruction. There are two types of reward signals, one is a naive signal that says the formalized premises do not imply the conclusion, and the other one tells that the proof is circular.

Although the reconstruction is now valid, it should also faithfully represents the original argument.
To ensure that, we prompt an LLM to judge whether the reconstruction is faithful or not and justify its decision.
If the reconstruction is faithful, then we stop iterating the loop.
If the reconstruction is unfaithful, then we feed LLM judge’s justification to the argument reconstructor to regenerate the reconstruction.
To minimize model calls in practice, we connect these two loops in series so that only valid reconstructions are judged for their faithfulness.
Also, we limit the total number of loop iteration to 10 and return the last reconstructed argument if the loop fails to generate valid and faithful reconstruction.
We provide an example reconstruction in Figure [2(b)](https://arxiv.org/html/2509.21679v1#S3.F2.sf2), and details on feedback loops and model prompts are described in Appendix [C](https://arxiv.org/html/2509.21679v1#A3) and [H](https://arxiv.org/html/2509.21679v1#A8), respectively.

#### Quality of argument reconstruction.

We measure validity and faithfulness of reconstructed arguments using a SAT solver and human annotators, respectively. With claude-sonnet-3.7 as a base LLM, every reconstruction is valid, and average faithfulness score is 4.5 / 5.
Detailed analysis including comparison with direct reconstruction is described in Appendix [C](https://arxiv.org/html/2509.21679v1#A3).

### 3.3 Dataset Construction

Our final destination of proposing ReviewScore is to automatically filter out misinformed review points using LLMs. To verify LLM’s ability to do that, we build a human expert-annotated dataset to measure an agreement between humans and LLMs on ReviewScore evaluation.
Our dataset contains total 657 annotated review points, consisting of 143 questions, 92 claims, and 422 arguments. Specifically, 1,748 premises of the arguments are manually annotated
(^7^77Details of the dataset construction and statistics are described in Appendix [D](https://arxiv.org/html/2509.21679v1#A4).).

#### Human-review matching.

We recruit 15 graduate students studying AI as human annotators, and they annotate total 40 papers submitted to ICLR 2021–2023(^8^88Since ICLR 2024–2025 submission drafts are not opened to public, we exclude these years.).
Specifically, we first make five groups by their research interests. For each group, three human annotators discuss which papers to annotate and select eight papers that are relevant with all three. Then, each human annotates selected eight papers’ review points which are preprocessed from OpenReview.

#### Data curation process.

We collaboratively use an LLM and humans for ReviewScore data curation, where an LLM preprocesses reviews and then humans annotate those.
Given a review, an LLM extracts independent review points.
For each review point, an LLM automatically annotates the type (i.e., claim, argument, or question) and human verifies it.
If the review point is a question, then human scores if the question is (un)answerable by the paper in a 5-point scale and justifies it if needed.
If the review point is a claim, then human scores if the claim is true in a 5-point scale and justifies it if needed.
If the review point is an argument, then human scores the argument’s factuality same as in claims (i.e., Base ReviewScore).
To annotate Advanced ReviewScore for arguments, we first run automatic argument reconstruction engine (Section [3.2](https://arxiv.org/html/2509.21679v1#S3.SS2)) to extract underlying premises of the argument.
After that, human scores if the reconstruction is faithful in a 5-point scale. If the faithfulness score is less than 4 (i.e., faithful, but one or two minor changes recommended), then they skip the subsequent annotations. Otherwise, they judge the factuality of premises.
For each premise, they first select a knowledge base (i.e., submitted paper, annotator knowledge, or referred papers), score if the premise is true based on the knowledge base in a 5-point scale, and score if the premise is (un)trivial in a 3-point scale. They justify any of three decisions if needed.

#### Trustworthiness of human annotation.

Three humans independently annotate reviews (i.e., we take a median value as a final human label), and they are allowed to use any related materials or tools (e.g., discussion between authors and reviewers in OpenReview, web search, etc.).
We also provide annotators detailed guidelines and an hour-long online orientation session. Furthermore, we highly encourage them to communicate with us through a group chat so that we could give them instant responses to their questions and share with all, which builds a global consensus among human annotators.
Thanks to these efforts, despite the difficulty of the work, we obtain median 0.301 and highest 0.428 inter-annotator agreement in Krippendorff’s Alpha (Hughes, 2021) across different annotator groups. Detailed results are described in Appendix [E](https://arxiv.org/html/2509.21679v1#A5).

## 4 Reliability of Automatic Evaluation of ReviewScore

To evaluate LLM’s ability to evaluate ReviewScore, we describe an experimental setup (Section [4.1](https://arxiv.org/html/2509.21679v1#S4.SS1)) and show human-model agreement results (Section [4.2](https://arxiv.org/html/2509.21679v1#S4.SS2)). We also compare the effectiveness of Base and Advanced ReviewScore (Section [4.3](https://arxiv.org/html/2509.21679v1#S4.SS3)). We further analyze human-model disagreements and the effect of providing authors response to models (Section [4.4](https://arxiv.org/html/2509.21679v1#S4.SS4)).

### 4.1 Setup

Given score rubrics in a 5-point scale, an LLM evaluates ReviewScore according to a knowledge base it selects. We only provide a submitted paper to a model since we assume that the model has a pretrained human-level internal knowledge and has a general understanding of referred papers. Detailed model prompts are described in Appendix [H](https://arxiv.org/html/2509.21679v1#A8).

#### Language models.

To measure LLM’s ability to automatically evaluate ReviewScore, we perform experiments on eight current state-of-the-art LLMs that achieve significantly high alignments with humans. We include five proprietary models, claude-sonnet-3.7 (Anthropic, 2025a), claude-sonnet-4 (Anthropic, 2025b), gpt-4o (OpenAI, 2024), gpt-5 (OpenAI, 2025), and gemini-2.5-flash (Gemini Team, Google, 2025), and three open-sourced models, qwen3-235b-a22b (Yang et al., 2025), deepseek-v3 (DeepSeek-AI et al., 2024), and llama-3.3 (AI@Meta, 2024)(^9^99Details of model specifications are described in Appendix [G](https://arxiv.org/html/2509.21679v1#A7)).
To get consistent and reliable scores from LLM judges, we set a low temperature (i.e., 0) and select the highest probability response from the model (Liang et al., 2022; Liu et al., 2023; Gu et al., 2024).
We exclude reasoning models as LLM judges since ReviewScore mostly depends on grounding and evidence, not longer or smarter chains of thought.

#### Evaluation metrics.

We use different sets of metrics for two types of problem formulations, binary classification and 5-point scale scoring.
For the binary classification, since majority of human-annotated labels are Not misinformed, we mainly use F1 score which is robust to class imbalance.
For the 5-point scale scoring, since majority of human-annotated scores are 4 and 5, we mainly use Quadratic Weighted Kappa (Warrens, 2012), a variant of Cohen’s Kappa (Cohen, 1960), that is robust to the skewed data distribution. Hereinafter, we call this metric Kappa for simplicity.
To provide more comprehensive results, we additionally use Precision and Recall for the binary classification and Pearson rank correlation and Gwet’s AC2 (Gwet, 2001) for the 5-point scale scoring.

### 4.2 Main Results

**Table 1: Human-model agreement on ReviewScore evaluation.**
|  | ClaimScore | ArgScore | QScore | ReviewScore |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | F1 | Kappa | F1 | Kappa | F1 | Kappa | F1 | Kappa |
| Proprietary models |  |  |  |  |  |  |  |  |
| claude-sonnet-3.7 | 0.160 | 0.137 | 0.458 | 0.380 | 0.542 | 0.387 | 0.448 | 0.355 |
| claude-sonnet-4 | 0.125 | 0.130 | 0.329 | 0.266 | 0.507 | 0.357 | 0.367 | 0.333 |
| gpt-4o | 0.000 | 0.119 | 0.329 | 0.239 | 0.538 | 0.340 | 0.405 | 0.362 |
| gpt-5 | 0.125 | -0.004 | 0.426 | 0.334 | 0.543 | 0.300 | 0.441 | 0.310 |
| gemini-2.5-flash | 0.091 | 0.106 | 0.447 | 0.366 | 0.512 | 0.225 | 0.439 | 0.318 |
| Open-sourced models |  |  |  |  |  |  |  |  |
| qwen3-235b-a22b | 0.231 | 0.094 | 0.403 | 0.149 | 0.531 | 0.234 | 0.436 | 0.243 |
| llama-3.3 | 0.133 | 0.153 | 0.250 | 0.107 | 0.551 | 0.284 | 0.409 | 0.355 |
| deepseek-v3 | 0.000 | 0.165 | 0.276 | 0.196 | 0.597 | 0.228 | 0.382 | 0.310 |

We empirically validate the alignment of human-annotated and model-estimated ReviewScore using different models and evaluation metrics in Table [1](https://arxiv.org/html/2509.21679v1#S4.T1).
Most models show 0.4–0.5 F1 score and 0.3–0.4 Kappa score, which indicates moderate agreement between humans and models on ReviewScore evaluation.
However, there are differences in human-model agreements for three types of review points. Regardless of the models, questions show the highest agreement, arguments follow subsequently, and claims show the lowest agreement. Specifically, for claims, some models show zero F1 score or negative Kappa score. We analyze human-model disagreements thoroughly in Section [4.4](https://arxiv.org/html/2509.21679v1#S4.SS4) and conclude that since claims lack supporting evidence and are often value-laden, models often misinterpret the intended meaning of the claims or judge differently than humans.

For proprietary models, claude-sonnet-3.7 shows the highest human-model agreements across different types of review points.
One thing to note is claude-sonnet-4 consistently shows lower agreement than its preceding model. We find out that it’s because claude-sonnet-4 sometimes overthink and more frequently predicts the minority human-annotated scores than claude-sonnet-3.7 does.
In contrast, comparing F1 scores of gpt-4o and gpt-5, the advanced model shows higher agreement than the preceding model.

For open-sourced models, qwen3-235b-a22b shows best human-model agreements in terms of F1 score.
Open-sourced models show lower human-model agreements than proprietary models, but the gap is not significant.
Open-sourced models sometimes show better agreements than proprietary models, for instance, llama-3.3 outperforms claude-sonnet-4 on ReviewScore evaluation.
However, open-sourced models consistently show lower agreements on evaluating ArgScore.
More results and qualitative analysis of model evaluation are described in Appendix [F](https://arxiv.org/html/2509.21679v1#A6).

### 4.3 Base vs. Advanced ReviewScore

**Table 2: Comparison of human-model agreement of Base vs. Advanced ReviewScore.**
| Model | Metric | Base | Advanced w/o Agg | Advanced |
| --- | --- | --- | --- | --- |
| claude-sonnet-3.7 | F1 | 0.185 | 0.262 | 0.458 |
| Kappa | 0.153 | 0.280 | 0.380 |  |
| gpt-5 | F1 | 0.243 | 0.316 | 0.426 |
| Kappa | 0.223 | 0.315 | 0.334 |  |
| gemini-2.5-flash | F1 | 0.167 | 0.349 | 0.447 |
| Kappa | 0.138 | 0.330 | 0.366 |  |
| deepseek-v3 | F1 | 0.203 | 0.163 | 0.276 |
| Kappa | 0.120 | 0.202 | 0.196 |  |

Figure: (a)
Refer to caption: x4.png

We compare human-model agreements of Base and Advanced ReviewScore using different models and evaluation metrics. Since the only difference is in defining scores for arguments, we compare ReviewScore for arguments in Table [2](https://arxiv.org/html/2509.21679v1#S4.T2). To verify the effectiveness of aggregation methods, we additionally include Advanced ReviewScore for arguments without aggregation (i.e., factuality of premises).
Regardless of models, Advanced ReviewScore clearly shows higher agreements than Base ReviewScore. For claude-sonnet-3.7, Advanced ReviewScore performs 2.48$\times$ higher F1 score and 2.48$\times$ higher Kappa score than the Base ReviewScore.
The aggregation methods also contribute to higher agreements.
For claude-sonnet-3.7, Advanced ReviewScore performs 1.75$\times$ higher F1 score and 1.36$\times$ higher Kappa score than the one without aggregation.

### 4.4 Analysis

#### ReviewScore Distribution.

**Table 3: Proportion of misinformed review points estimated by models and humans.**
|  | Model | Human |
| --- | --- | --- |
| Claims | 22.8 | 5.9 |
| Arguments | 30.1 | 17.7 |
| Questions | 40.0 | 26.4 |
| Review points | 31.2 | 17.8 |

We present model-estimated ReviewScore distribution and compare with human-annotated ones. Specifically, following the binary classification setup, we report the proportion of misinformed review points in Table [3](https://arxiv.org/html/2509.21679v1#S4.T3).
There is a clear trend that models more frequently decide review points as Misinformed than humans.
Models decide 31.2% of reviews points as Misinformed in average, where humans decide only 17.8% as Misinformed.
The discrepancy is the largest in claims where models decide 22.8% as Misinformed and humans decide only 5.9% as Misinformed.
To understand these results more deeply, we continue our analysis on the following human-model disagreements.

#### Human-model disagreements.

We analyze types of human-model disagreements in Figure [3(a)](https://arxiv.org/html/2509.21679v1#S4.F3.sf1).
Across all review points, there are two common types of disagreements, which are models misunderstand or incorrectly reason and models predict scores that are minority of human-annotated scores. These two types comprise nearly all disagreements on arguments and questions, and about half of disagreements on claims.
For claims, we observe that 36% of disagreements are caused by either claims are value-laden or lack supporting reason. These make the factuality judgment of claims subjective, leading to a low human-model agreement.
Practical limitations such as incorrect pdf parsing and not providing appendix to models cause nontrivial portion of disagreements.
We also analyze the reason why claude-sonnet-3.7 shows higher human-model agreements than claude-sonnet-4 in Figure [3(b)](https://arxiv.org/html/2509.21679v1#S4.F3.sf2). The advanced model shows smaller number of misunderstanding or incorrect reasoning, however, it shows higher number of overthinking(^10^1010We judge a model overthinks if its justification of a score is significantly longer than another model. We also manually verify if a model overanalyzes a review point.) and minority human-annotated score predictions, leading to a lower human-model agreement.

#### Helpfulness of authors response.

We study whether providing authors responses of reviews to a model benefits the automatic ReviewScore evaluation using claude-sonnet-4 in Table [4](https://arxiv.org/html/2509.21679v1#S4.T4).
Providing authors response clearly shows higher human-model agreement than not providing it. Specifically, the agreement of ReviewScore increases by 22.3% in F1 score and 13.2% in Kappa score by incorporating authors response.
Note that claude-sonnet-4 with authors response achieves the highest agreement compared to all models without authors response. Furthermore, the agreement of WScore and ArgScore without aggregation (i.e., premise factuality) also increases significantly by incorporating authors response.
These observations show that authors response largely benefits detecting misinformed reviews by providing additional cues for model judgments.

**Table 4: Effect of providing Authors Response (AR) to a model for ReviewScore evaluation.**
|  | F1 | Kappa |  |  |
| --- | --- | --- | --- | --- |
|  | w/o AR | w/ AR | w/o AR | w/ AR |
| WScore | 0.211 | 0.286 | 0.178 | 0.212 |
| ArgScore w/o Agg | 0.230 | 0.283 | 0.271 | 0.312 |
| ArgScore | 0.329 | 0.447 | 0.266 | 0.340 |
| QScore | 0.507 | 0.513 | 0.357 | 0.324 |
| ClaimScore | 0.125 | 0.133 | 0.130 | 0.225 |
| ReviewScore | 0.367 | 0.449 | 0.333 | 0.377 |

## 5 Limitations

We acknowledge three types of limitations of our work.
First, there are technical and practical limitations in the automated ReviewScore evaluation.
Throughout Section [4](https://arxiv.org/html/2509.21679v1#S4), human-model agreement seems low even with authors response.
However, compared to Du et al. (2024) which measures human-model agreement on Deficient review detection and reports the best F1 score as 0.220, our work reports the best F1 score as 0.449, indicating more than $2\times$ agreement.
To save API calling costs, when the paper is given to LLMs, we only provide text and tables of the main paper, where any figure or appendix is excluded.
We also observe nontrivial amount of incorrect pdf parsing.
Second, there are technical limitations in the automatic argument reconstruction.
The reconstruction output is not always perfect since it depends on the capability of the base model. However, we observe a considerable output quality improvement when we upgrade the base model to claude-sonnet-4, indicating that the reconstruction engine would perform better as the model improves in general.
Lastly, there are practical limitations in the dataset construction.
Since human annotators are graduate students with varying skillfulness and the annotation requires significant amount of cognitive load, there are unavoidable noise in human annotation.
Furthermore, a manual selection of papers by human annotators might introduce any kind of unintended biases. However, to collect the most reliable annotation under our limited budget, we inevitably choose this method.

## 6 Conclusion

We introduce ReviewScore, a new evaluation of peer review quality, focusing on detecting misinformed review points. To automate ReviewScore evaluation using LLMs and validate it, we also construct a trustworthy human-annotated dataset. The results show a moderate human-model agreement, and further comprehensive disagreement analysis supports the potential of a fully automated ReviewScore evaluation with current state-of-the-art LLMs. We hope our work could contribute to mitigate the issue of low-quality reviews in AI conferences.

## References

- AI@Meta (2024)
AI@Meta.
The llama 3 herd of models.
07 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
Family report for Llama 3; used to cite Llama 3.3 70B Instruct.
- Anthropic (2025a)
Anthropic.
Claude 3.7 sonnet system card.
Technical report, Anthropic PBC, 02 2025a.
URL [https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf](https://assets.anthropic.com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf).
System card / technical report.
- Anthropic (2025b)
Anthropic.
System card: Claude opus 4 & claude sonnet 4.
Technical report, Anthropic PBC, 05 2025b.
URL [https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf](https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf).
System card / technical report.
- Arora et al. (2023)
Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re.
Ask me anything: A simple strategy for prompting language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=bhUPJnS2g0X](https://openreview.net/forum?id=bhUPJnS2g0X).
- Beall et al. (2024)
Jc Beall, Greg Restall, and Gil Sagi.
Logical Consequence.
In Edward N. Zalta and Uri Nodelman (eds.), *The Stanford Encyclopedia of Philosophy*. Metaphysics Research Lab, Stanford University, Summer 2024 edition, 2024.
- Betz & Richardson (2021)
Gregor Betz and Kyle Richardson.
Deepa2: A modular framework for deep argument analysis with pretrained neural text2text language models.
*arXiv preprint arXiv:2110.01509*, 2021.
- Borisov et al. (2023)
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci.
Language models are realistic tabular data generators.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=cEygmQNOeI](https://openreview.net/forum?id=cEygmQNOeI).
- Bowell & Kemp (2014)
Tracy Bowell and Gary Kemp.
*Critical Thinking: A Concise Guide*.
Routledge, 2014.
- Boyd-Graber et al. (2023)
Jordan Boyd-Graber, Naoaki Okazaki, and Anna Rogers.
Acl’23 peer review policies.
ACL 2023 Blog, February 2023.
URL [https://2023.aclweb.org/blog/review-acl23/](https://2023.aclweb.org/blog/review-acl23/).
Program Chairs blog post.
- Brun & Betz (2016)
Georg Brun and Gregor Betz.
Analysing practical argumentation.
In *The argumentative turn in policy analysis: Reasoning about uncertainty*, pp. 39–77. Springer, 2016.
- Cohen (1960)
Jacob Cohen.
A coefficient of agreement for nominal scales.
*Educational and psychological measurement*, 20(1):37–46, 1960.
- Cortes & Lawrence (2021)
Corinna Cortes and Neil D. Lawrence.
Inconsistency in conference peer review: Revisiting the 2014 neurips experiment.
*arXiv preprint arXiv:2109.09774*, 2021.
doi: 10.48550/arXiv.2109.09774.
URL [https://arxiv.org/abs/2109.09774](https://arxiv.org/abs/2109.09774).
- Couairon et al. (2023)
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.
Diffedit: Diffusion-based semantic image editing with mask guidance.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=3lge0p5o-M-](https://openreview.net/forum?id=3lge0p5o-M-).
- Creswell et al. (2023)
Antonia Creswell, Murray Shanahan, and Irina Higgins.
Selection-inference: Exploiting large language models for interpretable logical reasoning.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=3Pf3Wg6o-A4](https://openreview.net/forum?id=3Pf3Wg6o-A4).
- De Moura & Bjørner (2008)
Leonardo De Moura and Nikolaj Bjørner.
Z3: An efficient smt solver.
In *International conference on Tools and Algorithms for the Construction and Analysis of Systems*, pp. 337–340. Springer, 2008.
- DeepSeek-AI et al. (2024)
DeepSeek-AI, Aixin Liu, Bei Feng, et al.
Deepseek-v3 technical report.
12 2024.
URL [https://arxiv.org/abs/2412.19437](https://arxiv.org/abs/2412.19437).
Technical report; applicable to the V3-0324 checkpoint lineage.
- Dosovitskiy et al. (2020)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
*arXiv preprint arXiv:2010.11929*, 2020.
- Dowden (2024)
Bradley H Dowden.
Logical reasoning (dowden), 2024.
URL [https://human.libretexts.org/Bookshelves/Philosophy/Logic_and_Reasoning/Logical_Reasoning_(Dowden)](https://human.libretexts.org/Bookshelves/Philosophy/Logic_and_Reasoning/Logical_Reasoning_(Dowden)).
- Du et al. (2024)
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, et al.
Llms assist nlp researchers: Critique paper (meta-) reviewing.
*arXiv preprint arXiv:2406.16253*, 2024.
- Dutilh Novaes (2022)
Catarina Dutilh Novaes.
Argument and Argumentation.
In Edward N. Zalta and Uri Nodelman (eds.), *The Stanford Encyclopedia of Philosophy*. Metaphysics Research Lab, Stanford University, Fall 2022 edition, 2022.
- Fu et al. (2023)
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot.
Complexity-based prompting for multi-step reasoning.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=yf1icZHC-l9](https://openreview.net/forum?id=yf1icZHC-l9).
- Gal et al. (2023)
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or.
An image is worth one word: Personalizing text-to-image generation using textual inversion.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=NAQvF08TcyG](https://openreview.net/forum?id=NAQvF08TcyG).
- Gemini Team, Google (2025)
Gemini Team, Google.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
Technical report, Google DeepMind, 06 2025.
URL [https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf](https://storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf).
Technical report; covers Gemini 2.5 Pro and 2.5 Flash.
- Goldberg et al. (2025)
Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave, and Nihar B Shah.
Peer reviews of peer reviews: A randomized controlled trial and other experiments.
*PloS one*, 20(4):e0320444, 2025.
- Groarke (2024)
Leo Groarke.
Informal Logic.
In Edward N. Zalta and Uri Nodelman (eds.), *The Stanford Encyclopedia of Philosophy*. Metaphysics Research Lab, Stanford University, Spring 2024 edition, 2024.
- Gu et al. (2024)
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo.
A survey on llm-as-a-judge.
*arXiv preprint arXiv:2411.15594*, 2024.
URL [https://arxiv.org/abs/2411.15594](https://arxiv.org/abs/2411.15594).
Recommends temperature=0 to reduce randomness and stabilize judge outputs.
- Guo et al. (2023)
Yanzhu Guo, Guokan Shang, Virgile Rennard, Michalis Vazirgiannis, and Chloé Clavel.
Automatic analysis of substantiation in scientific peer reviews.
*arXiv preprint arXiv:2311.11967*, 2023.
- Gwet (2001)
Kilem Gwet.
Handbook of inter-rater reliability.
*Gaithersburg, MD: STATAXIS Publishing Company*, pp. 223–246, 2001.
- Hertz et al. (2023)
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or.
Prompt-to-prompt image editing with cross-attention control.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=_CDixzkzeyb](https://openreview.net/forum?id=_CDixzkzeyb).
- Hsu et al. (2022)
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin.
Language model compression with weighted low-rank factorization.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=uPv9Y3gmAI5](https://openreview.net/forum?id=uPv9Y3gmAI5).
- Hu et al. (2022)
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
LoRA: Low-rank adaptation of large language models.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9).
- Huang et al. (2023)
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han.
Large language models can self-improve, 2023.
URL [https://openreview.net/forum?id=NiEtU7blzN](https://openreview.net/forum?id=NiEtU7blzN).
- Hughes (2021)
John Hughes.
krippendorffsalpha: An r package for measuring agreement using krippendorff’s alpha coefficient.
*arXiv preprint arXiv:2103.12170*, 2021.
- Jang et al. (2022)
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi, and Minjoon Seo.
Towards continual knowledge learning of language models.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=vfsRB5MImo9](https://openreview.net/forum?id=vfsRB5MImo9).
- Ji et al. (2023)
Haozhe Ji, Pei Ke, Zhipeng Hu, Rongsheng Zhang, and Minlie Huang.
Tailoring language generation models under total variation distance.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=VELL0PlWfc](https://openreview.net/forum?id=VELL0PlWfc).
- Kelly et al. (2014)
Jacalyn Kelly, Tara Sadeghieh, and Khosrow Adeli.
Peer review in scientific publications: benefits, critiques, & a survival guide.
*Ejifcc*, 25(3):227, 2014.
- Kim et al. (2025)
Jaeho Kim, Yunseok Lee, and Seulki Lee.
Position: The ai conference peer review crisis demands author feedback and reviewer rewards.
*arXiv preprint arXiv:2505.04966*, 2025.
- Kim et al. (2023)
Yoonjeon Kim, Hyunsu Kim, Junho Kim, Yunjey Choi, and Eunho Yang.
Learning input-agnostic manipulation directions in styleGAN with text guidance.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=47B_ctC4pJ](https://openreview.net/forum?id=47B_ctC4pJ).
- Kukleva et al. (2023)
Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht.
Temperature schedules for self-supervised contrastive methods on long-tail data.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=ejHUr4nfHhD](https://openreview.net/forum?id=ejHUr4nfHhD).
- Lau & Chan (2025)
Joe Lau and Jonathan Chan.
What is an argument?
Critical Thinking Web, 2025.
URL [https://philosophy.hku.hk/think/arg/arg.php](https://philosophy.hku.hk/think/arg/arg.php).
Module A01: Argument analysis.
- Li et al. (2022)
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto.
Large language models can be strong differentially private learners.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=bVuP3ltATMz](https://openreview.net/forum?id=bVuP3ltATMz).
- Liang et al. (2021)
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin.
Mix{kd}: Towards efficient distillation of large-scale language models.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=UFGEelJkLu5](https://openreview.net/forum?id=UFGEelJkLu5).
- Liang et al. (2022)
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al.
Holistic evaluation of language models.
*arXiv preprint arXiv:2211.09110*, 2022.
URL [https://arxiv.org/abs/2211.09110](https://arxiv.org/abs/2211.09110).
Standardizes eval settings; many HELM runs use temperature=0 for reproducibility.
- Lipman et al. (2023)
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le.
Flow matching for generative modeling.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=PqvMRDCJT9t](https://openreview.net/forum?id=PqvMRDCJT9t).
- Liu et al. (2023)
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.
G-eval: Nlg evaluation using GPT-4 with better human alignment.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 2511–2522. Association for Computational Linguistics, 2023.
URL [https://aclanthology.org/2023.emnlp-main.153/](https://aclanthology.org/2023.emnlp-main.153/).
Uses low temperature / deterministic decoding for stable evaluator outputs.
- Lu et al. (2023)
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan.
Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=DHyHRBwJUTN](https://openreview.net/forum?id=DHyHRBwJUTN).
- Meng et al. (2022)
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
SDEdit: Guided image synthesis and editing with stochastic differential equations.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=aBsCjcPu_tE](https://openreview.net/forum?id=aBsCjcPu_tE).
- Nam et al. (2023)
Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin.
STUNT: Few-shot tabular learning with self-generated tasks from unlabeled tables.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=_xlsjehDvlY](https://openreview.net/forum?id=_xlsjehDvlY).
- NeurIPS 2025 Program Chairs (2025)
NeurIPS 2025 Program Chairs.
Neurips 2025 reviewer guidelines.
NeurIPS 2025 website, 2025.
URL [https://neurips.cc/Conferences/2025/ReviewerGuidelines](https://neurips.cc/Conferences/2025/ReviewerGuidelines).
Guidelines page.
- Ni et al. (2023)
Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao.
Learning math reasoning from self-sampled correct and partially-correct solutions.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=4D4TSJE6-K](https://openreview.net/forum?id=4D4TSJE6-K).
- Nie et al. (2023)
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam.
A time series is worth 64 words: Long-term forecasting with transformers.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=Jbdc0vTOcol](https://openreview.net/forum?id=Jbdc0vTOcol).
- OpenAI (2024)
OpenAI.
Gpt-4o system card.
Technical report, OpenAI, 08 2024.
URL [https://cdn.openai.com/gpt-4o-system-card.pdf](https://cdn.openai.com/gpt-4o-system-card.pdf).
System card / technical report.
- OpenAI (2025)
OpenAI.
Gpt-5 system card.
Technical report, OpenAI, 08 2025.
URL [https://cdn.openai.com/gpt-5-system-card.pdf](https://cdn.openai.com/gpt-5-system-card.pdf).
System card / technical report.
- Ou et al. (2025)
Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, et al.
Claimcheck: How grounded are llm critiques of scientific papers?
*arXiv preprint arXiv:2503.21717*, 2025.
- Poole et al. (2023)
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.
Dreamfusion: Text-to-3d using 2d diffusion.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=FjNys5c7VyY](https://openreview.net/forum?id=FjNys5c7VyY).
- Purkayastha et al. (2025)
Sukannya Purkayastha, Zhuang Li, Anne Lauscher, Lizhen Qu, and Iryna Gurevych.
Lazyreview a dataset for uncovering lazy thinking in nlp peer reviews.
*arXiv preprint arXiv:2504.11042*, 2025.
- Razdaibiedina et al. (2023)
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi.
Progressive prompts: Continual learning for language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=UJTgQBc91_](https://openreview.net/forum?id=UJTgQBc91_).
- Ruis et al. (2023)
Laura Eline Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette.
Large language models are not zero-shot communicators, 2023.
URL [https://openreview.net/forum?id=WgbcOQMNXB](https://openreview.net/forum?id=WgbcOQMNXB).
- Sadallah et al. (2025)
Abdelrahman Sadallah, Tim BaumgÃĪrtner, Iryna Gurevych, and Ted Briscoe.
The good, the bad and the constructive: Automatically measuring peer review’s utility for authors.
*arXiv preprint arXiv:2509.04484*, 2025.
- Sanh et al. (2022)
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush.
Multitask prompted training enables zero-shot task generalization.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=9Vrb9D0WI4](https://openreview.net/forum?id=9Vrb9D0WI4).
- Shah (2022)
Nihar B. Shah.
Challenges, experiments, and computational solutions in peer review.
*Communications of the ACM*, 65(6):76–87, June 2022.
doi: 10.1145/3528086.
URL [https://cacm.acm.org/research/challenges-experiments-and-computational-solutions-in-peer-review/](https://cacm.acm.org/research/challenges-experiments-and-computational-solutions-in-peer-review/).
- Shen et al. (2022)
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer.
How much can CLIP benefit vision-and-language tasks?
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=zf_Ll3HZWgy](https://openreview.net/forum?id=zf_Ll3HZWgy).
- Shi et al. (2023)
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei.
Language models are multilingual chain-of-thought reasoners.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=fR3wGCk-IXp](https://openreview.net/forum?id=fR3wGCk-IXp).
- Shukla & Marlin (2021)
Satya Narayan Shukla and Benjamin Marlin.
Multi-time attention networks for irregularly sampled time series.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=4c0J6lwQ4_](https://openreview.net/forum?id=4c0J6lwQ4_).
- Shukla & Marlin (2022)
Satya Narayan Shukla and Benjamin Marlin.
Heteroscedastic temporal variational autoencoder for irregularly sampled time series.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=Az7opqbQE-3](https://openreview.net/forum?id=Az7opqbQE-3).
- Si et al. (2023)
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang.
Prompting GPT-3 to be reliable.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=98p5x51L5af](https://openreview.net/forum?id=98p5x51L5af).
- Singer et al. (2023)
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman.
Make-a-video: Text-to-video generation without text-video data.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=nJfylDvgzlq](https://openreview.net/forum?id=nJfylDvgzlq).
- Stelmakh et al. (2021)
Ivan Stelmakh, Nihar B. Shah, Aarti Singh, and Hal Daumé III.
A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 35, pp. 4785–4793, 2021.
doi: 10.1609/aaai.v35i6.16610.
URL [https://ojs.aaai.org/index.php/AAAI/article/view/16610](https://ojs.aaai.org/index.php/AAAI/article/view/16610).
- Taylor & Francis (2025)
Taylor and Francis.
Understanding the peer review process. what is peer review? a guide for authors, 2025.
URL [https://authorservices.taylorandfrancis.com/publishing-your-research/peer-review](https://authorservices.taylorandfrancis.com/publishing-your-research/peer-review).
- Wang et al. (2023)
Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao.
MICN: Multi-scale local and global context modeling for long-term series forecasting.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=zt53IDUR1U](https://openreview.net/forum?id=zt53IDUR1U).
- Warrens (2012)
Matthijs J Warrens.
Some paradoxical results for the quadratically weighted kappa.
*Psychometrika*, 77(2):315–323, 2012.
- Wei et al. (2022)
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le.
Finetuned language models are zero-shot learners.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=gEZrGCozdqR](https://openreview.net/forum?id=gEZrGCozdqR).
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, et al.
Qwen3 technical report.
05 2025.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
Covers the Qwen3 family including the 235B A22B series (e.g., 2507 checkpoint).
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=WE_vluYUL-X](https://openreview.net/forum?id=WE_vluYUL-X).
- Zhang et al. (2022)
Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik.
Graph-guided network for irregularly sampled multivariate time series.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=Kwm8I7dU-l5](https://openreview.net/forum?id=Kwm8I7dU-l5).
- Zhang & Yan (2023)
Yunhao Zhang and Junchi Yan.
Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=vSVLM2j9eie](https://openreview.net/forum?id=vSVLM2j9eie).
- Zhang et al. (2023)
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola.
Automatic chain of thought prompting in large language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=5NTt8GFjUHkr](https://openreview.net/forum?id=5NTt8GFjUHkr).
- Zhou et al. (2023a)
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi.
Least-to-most prompting enables complex reasoning in large language models.
In *The Eleventh International Conference on Learning Representations*, 2023a.
URL [https://openreview.net/forum?id=WZH7099tgfM](https://openreview.net/forum?id=WZH7099tgfM).
- Zhou et al. (2023b)
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.
Large language models are human-level prompt engineers.
In *The Eleventh International Conference on Learning Representations*, 2023b.
URL [https://openreview.net/forum?id=92gvk82DE-](https://openreview.net/forum?id=92gvk82DE-).

## Appendix A Future Impact

We expect automatic ReviewScore evaluation could greatly benefit different roles of the current peer reviewing system.
For authors, by providing reconstruction of argumentative review points, it helps to understand or clarify the reviewer’s intention and to formulate a rebuttal.
For reviewers, by providing ReviewScore of their review points, it allows them to verify the review quality by themselves and helps reviewers to better understand the paper.
For metareviewers, by providing ReviewScore of each reviewer, it could assist their final decisions.
To summarize, it could serve as an automated system for managing the review quality.

## Appendix B Review Quality Criteria Discussion

First, we recruit a group of three graduate students studying AI and NLP and let them independently analyze reviews of ten common manually selected submitted papers from ICLR 2021–2023(^11^1111Since those papers include papers written by authors of our work, we do not share the full list of target papers to keep anonymity. We will uncover the full list after the paper is published.). Specifically, we guided the group to: decompose a review into several independent review points, evaluate quality of the review points by a 5-point scale with their own criteria, and justify their scores that ends with a meta-sentence that does not involve paper’s context which is used for our further analysis. During human evaluation, we minimize our effort to provide detailed guidelines to facilitate bias-free human analysis.

Then, we categorize these justifications in order to find common features of low-quality review points.
There are five common types: (1) questions that can be already addressed by the paper, (2) comments that reflect a misunderstanding of the paper, (3) out-of-scope remarks, (4) observations pointing out minor details, and (5) unclear points.
However, for the last three types, we observe that majority of review points are agreed or argued by a single human annotator. In other words, given a review point, only a single human argues it is out-of-scope, whereas other two humans argue it is within scope and could be a potential drawback of the submitted paper. Following this pattern, a single human argues a review point is addressing minor details but others argue it is a major point, and a single human argues a review point is unclear whereas others do not agree with this.
In contrast, the first two types of review points (i.e., answerable questions and misunderstood comments) are mostly agreed by two or more human annotators, indicating that these two types have more objective and trustworthy criteria for detecting low-quality reviews. Based on these observations, we focus on evaluating review points based on the first two criteria.

Lastly, we share some meaningful insights from this group discussion that strongly motivates our work.
Based on the analysis of score justifications, given a review point, some (sub)sentences are high-quality or factually correct but others could be misinformed or factually incorrect. This later motivates our premise-level factuality evaluation (Section [3.1](https://arxiv.org/html/2509.21679v1#S3.SS1)).
Furthermore, during the group discussion, we observe that human annotators sometimes struggle which parts in a review point they should weigh more to evaluate the review point. This later motivates our aggregation methods which include logical conjunction and weighted average by untrivialness (Section [3.1](https://arxiv.org/html/2509.21679v1#S3.SS1)).
We also discover that human evaluations are sometimes incorrect. We leave this as a limitation of our work.
However, to alleviate this issue, we recruit three human annotators for every instance and further ensure the annotation quality by providing them careful guidelines and actively communicating with them to build a global consensus on the evaluation criteria (Section [3.3](https://arxiv.org/html/2509.21679v1#S3.SS3)).

## Appendix C Automatic Argument Reconstruction Details and Results

### C.1 Implementation details

We elaborate components of automatic argument reconstruction engine (Section [3.2](https://arxiv.org/html/2509.21679v1#S3.SS2), Figure [2](https://arxiv.org/html/2509.21679v1#S3.F2)) in the following. We also refer the corresponding prompt used in each step if it exists.

- 1.
Given an argumentative review point, an LLM extracts a verbatim conjecture and its verbatim reason statements (Figure [23](https://arxiv.org/html/2509.21679v1#A8.F23)).
- 2.
Given a verbatim conclusion and reasons of the argumentative review point and a corresponding paper parsed from a pdf file, an LLM reconstructs the argument into a premise-conclusion structure. At the same inference, the model also translates (or formalizes) NL premises and conclusion into corresponding FOL formulas and generate keys which assign NL meaning to variables and predicates. To facilitate the model to generate a valid formalization, the model generates a deductive proof using formalized premises and conclusion at the end (Figure [24](https://arxiv.org/html/2509.21679v1#A8.F24)).
- 3.
Given FOL premises and conclusion with keys and a deductive proof, an LLM extracts necessary FOL premises for the deductive proof, write a python program using Z3 (De Moura & Bjørner, 2008) that automatically checks the validity of the necessary FOL premises and conclusion, extracts a final FOL conclusion that is used in the python program, and judges whether the proof is circular (i.e., whether the final FOL conclusion is included in one of necessary FOL premises.) (Figure [25](https://arxiv.org/html/2509.21679v1#A8.F25), Figure [26](https://arxiv.org/html/2509.21679v1#A8.F26)).
- 4.
If the proof is circular, then a NL feedback indicating circularity of the proof is sent to Step 2, and the model re-generates an argument reconstruction. Otherwise, we run the python program that checks the validity of the reconstruction. However, if the program returns an error, the model takes this error message and re-generate the python program that fixes the error (Figure [27](https://arxiv.org/html/2509.21679v1#A8.F27)).
If the reconstruction is invalid, then a NL feedback indicating invalidity of the reconstruction is sent to Step 2, and the model re-generates an argument reconstruction. Otherwise (i.e., if the reconstruction is valid), we proceed to the next step.
- 5.
To check faithfulness of the reconstruction, the model first translates FOL premises and conclusion (one of the outputs of the model in Step 2) with keys (one of the outputs of the model in Step 1) into NL premises and conclusion (Figure [28](https://arxiv.org/html/2509.21679v1#A8.F28)). This process is called logical streamlining in logic and critical thinking (Bowell & Kemp, 2014; Brun & Betz, 2016; Betz & Richardson, 2021).
- 6.
Lastly, given an original argumentative review point (or an argument) and the streamlined NL premises and conclusion, the model judges whether the reconstruction is faithful with justifications (Figure [29](https://arxiv.org/html/2509.21679v1#A8.F29)). If the reconstruction is unfaithful, then a NL feedback including the model’s justifications is sent to Step 2, and the model re-generates an argument reconstruction accordingly. Otherwise, since the reconstruction is valid yet faithful, the feedback loop is finished and the streamlined NL premises and conclusion become a final argument reconstruction.

### C.2 Quantitative Results

To verify effectiveness of the feedback loop, we report the argument reconstruction performance with and without feedback in Table [5](https://arxiv.org/html/2509.21679v1#A3.T5).
We mainly measure average validity and faithfulness of the reconstructed arguments using a SAT solver and human annotators, respectively. We provide score rubric for evaluating faithfulness in Figure [4](https://arxiv.org/html/2509.21679v1#A3.F4).
We also include the pass rate, which indicates whether the (last) reconstructed argument fulfill the validity and faithfulness criteria in the feedback loop.
Furthermore, we report the average number of loop iterations to check if the feedback loops are actively used.

In Table [5](https://arxiv.org/html/2509.21679v1#A3.T5), we verify that the proposed method (i.e., w/ feedback) achieves a perfect validity and nearly perfect faithfulness and pass rate, whereas the performance of the direct reconstruction (i.e., w/o feedback) largely lags behind that.
We observe that the average number of loop iterations is 3.09, indicating active usage of feedback signals, but the number varies a lot depending on the clarity of logical structures of the arguments.

**Table 5: Performance of Automatic Argument Reconstruction.**
|  | w/o feedback | w/ feedback |
| --- | --- | --- |
| Validity | 0.895 | 1.00 |
| Faithfulness | 2.91 | 4.47 |
| Pass Rate | 0.369 | 0.935 |
| # of loops | N/A | 3.09 $\pm$ 2.73 |

Figure: Figure 4: Score rubric for evaluating faithfulness of argument reconstruction by human annotators.

### C.3 Qualitative Results

We present three representative examples of model-generated argument reconstruction and compare outputs with and without the feedback loop.
We observe the direct reconstruction (i.e., w/o feedback) often generates either invalid or unfaithful reconstructed arguments, however, the proposed method (i.e., w/ feedback) generates valid yet faithful reconstructed arguments.
We provide detailed analysis in Figure [5](https://arxiv.org/html/2509.21679v1#A3.F5)–[7](https://arxiv.org/html/2509.21679v1#A3.F7).

Figure: Figure 5: Example #1 of automatic argument reconstruction.

Figure: Figure 6: Example #2 of automatic argument reconstruction.

Figure: Figure 7: Example #3 of automatic argument reconstruction.

## Appendix D Dataset Details

### D.1 Terms of Use and License

All papers and corresponding reviews used in our work are crawled from OpenReview(^12^1212[https://openreview.net](https://openreview.net)), and our work is consistent with the Openreview terms of use(^13^1313[https://openreview.net/legal/terms](https://openreview.net/legal/terms)).
Following these terms, we will release our work with a CC-BY 4.0 license.

### D.2 Pilot Dataset

Before we construct the main dataset in Section [3.3](https://arxiv.org/html/2509.21679v1#S3.SS3), to ensure a trustworthiness and reliability of human annotation, we conduct a pilot study.
We recruit three graduate students studying AI as human annotators, and let them choose total five papers submitted to ICLR 2021–2025(^14^1414Unlike the main dataset, we include ICLR 2024–2025 since authors can access to their own submissions.).
Specifically, each human annotates reviews of three papers in OpenReview, where only one paper is authored by themselves and the other two non-authored papers are assigned in common.
To ensure trustworthiness of non-authors’ annotation, three humans annotate reviews and then we take a median value as a final human label.

### D.3 Dataset Statistics

We present the ReviewScore dataset statistics including the pilot and main subsets in Table [6](https://arxiv.org/html/2509.21679v1#A4.T6). We include number of instances and percentage of misinformed labels.

**Table 6: ReviewScore dataset statistics.**
|  | Pilot | Main | Total |
| --- | --- | --- | --- |
| ICLR Years | 2021–2025 | 2021–2023 | - |
| # Papers | 5 | 40 | 45 |
| # Review(er)s | 19 | 155 | 174 |
| Number of instances |  |  |  |
| Review Points | 84 | 573 | 657 |
| Questions | 22 | 121 | 143 |
| Claims | 5 | 87 | 92 |
| Arguments | 57 | 365 | 422 |
| Premises | 227 | 1,521 | 1,748 |

### D.4 Full List of Papers

We provide a full list of papers used in the main subset of ReviewScore dataset in Table [7](https://arxiv.org/html/2509.21679v1#A4.T7). As explained in Section [3.3](https://arxiv.org/html/2509.21679v1#S3.SS3), eight papers are selected by each human annotator group with a common research interest.

**Table 7: Full list of ICLR submitted papers used in the ReviewScore dataset.**
| Group | Topic | Paper Title |
| --- | --- | --- |
| 1 | Image Generation | Lipman et al. (2023), Hertz et al. (2023), Couairon et al. (2023), |
| Singer et al. (2023), Meng et al. (2022), Gal et al. (2023), Poole et al. (2023), |  |  |
| Kim et al. (2023) |  |  |
| 2 | Time Series ML | Shukla & Marlin (2021), Zhang et al. (2022), Borisov et al. (2023), |
| Nie et al. (2023), Nam et al. (2023), Zhang & Yan (2023), Wang et al. (2023), |  |  |
| Shukla & Marlin (2022) |  |  |
| 3 | LLM Reasoning / | Hsu et al. (2022), Ji et al. (2023), Lu et al. (2023), Ni et al. (2023), |
| Compression | Fu et al. (2023), Liang et al. (2021), Ruis et al. (2023), Arora et al. (2023) |  |
| 4 | LLM / VLM | Hu et al. (2022), Kukleva et al. (2023), Wei et al. (2022), Sanh et al. (2022), |
| Shen et al. (2022), Zhou et al. (2023b), Si et al. (2023), Shi et al. (2023) |  |  |
| 5 | LLM Prompting | Zhang et al. (2023), Yao et al. (2023), Zhou et al. (2023a), |
| Razdaibiedina et al. (2023), Li et al. (2022), Jang et al. (2022), |  |  |
| Creswell et al. (2023), Huang et al. (2023) |  |  |

## Appendix E Analysis of Human Annotators

### E.1 Human Annotator Expertise

We report human annotator’s expertise on ReviewScore evaluation in Table [8](https://arxiv.org/html/2509.21679v1#A5.T8). Specifically, for each annotator, we indicate a number of publications in AI/ML (including arXived works) and an averaged paper relevance score across eight assigned papers. We also indicate averaged numbers and scores for each group.
The results show that an average number of publication is 3.93 and average paper relevance score is 4.06 / 5, indicating highly-experienced and relevant experts conduct a human annotation process.
However, we observe that there are inter-group gaps in human expertise. Specifically, Group 2 and 4 shows significantly higher number of publications and paper relevance than other groups.
Detailed score rubric for paper relevance is described in Figure [8](https://arxiv.org/html/2509.21679v1#A5.F8), and we note that there is no instance where paper relevance of any human annotator is less than 3 (i.e., Moderate relevance).

**Table 8: Human annotator’s expertise on ReviewScore evaluation. A bold indicates the highest number/score across different groups, and an underline indicates the second highest.**
| Group | Annotator ID | # AI/ML Pub | Paper Relevance | Avg # Pub | Avg Paper Relevance |
| --- | --- | --- | --- | --- | --- |
| 1 | anno_11 | 2 | 3.50 | 3.00 | 4.00 |
| anno_12 | 3 | 4.13 |  |  |  |
| anno_13 | 4 | 4.38 |  |  |  |
| 2 | anno_21 | 6 | 4.50 | 5.33 | 4.38 |
| anno_22 | 3 | 4.38 |  |  |  |
| anno_23 | 7 | 4.25 |  |  |  |
| 3 | anno_31 | 0 | 4.00 | 1.33 | 3.96 |
| anno_32 | 1 | 4.00 |  |  |  |
| anno_33 | 3 | 3.88 |  |  |  |
| 4 | anno_41 | 1 | 4.00 | 6.67 | 4.04 |
| anno_42 | 6 | 3.88 |  |  |  |
| anno_43 | 13 | 4.25 |  |  |  |
| 5 | anno_51 | 3 | 3.13 | 3.33 | 3.92 |
| anno_52 | 1 | 4.13 |  |  |  |
| anno_53 | 6 | 4.50 |  |  |  |
| Total | - | - | - | 3.93 | 4.06 |

Figure: Figure 8: Score rubric for evaluating paper relevance of human annotators.

### E.2 Inter-Annotator Agreement

To ensure trustworthiness of human annotation, we report inter-annotator agreement in Krippendorff’s Alpha (Hughes, 2021) on ReviewScore evaluation in Table [9](https://arxiv.org/html/2509.21679v1#A5.T9).
Overall, ReviewScore shows 0.301 Krippendorff’s Alpha, indicating a moderate inter-annotator agreement. Specifically, ClaimScore shows the highest agreement, QScore follows subsequently, and ArgScore shows the lowest agreement.

By comparing inter-group agreements, Group 2 and 4 show significantly higher ReviewScore agreement than other groups.
In contrast, Group 1 and 5 include negative inter-annotator agreement, which is a critical concern for reliable human annotation. By a manual disagreement analysis, we confirm that most disagreements come from human annotators with low paper relevance (i.e., anno_11 and anno_51 in Table [8](https://arxiv.org/html/2509.21679v1#A5.T8)).
To summarize, we observe that the inter-annotator agreement aligns with the annotator expertise (Table [8](https://arxiv.org/html/2509.21679v1#A5.T8)), and we (unsurprisingly) confirm that human annotations could be more reliable if their research interests become more relevant to the assigned papers.

**Table 9: Inter-annotator agreement (Krippendorff’s Alpha) on ReviewScore evaluation. A bold indicates the highest agreement across different groups, and an underline indicates the second highest.**
| Group | ClaimScore | ArgScore | QScore | ReviewScore |
| --- | --- | --- | --- | --- |
| 1 | 0.399 | -0.023 | 0.245 | 0.125 |
| 2 | 0.328 | 0.329 | 0.350 | 0.428 |
| 3 | 0.400 | 0.146 | 0.436 | 0.301 |
| 4 | 0.375 | 0.361 | 0.465 | 0.385 |
| 5 | 0.203 | -0.023 | -0.078 | -0.041 |
| Median | 0.375 | 0.146 | 0.350 | 0.301 |

## Appendix F Additional Results of Automatic ReviewScore Evaluation

### F.1 Quantitative Results

We additionally report human-model agreement on ReviewScore evaluation using different evaluation metrics in Table [10](https://arxiv.org/html/2509.21679v1#A6.T10)–[13](https://arxiv.org/html/2509.21679v1#A6.T13). Specifically, we report agreement on ClaimScore evaluation in Table [10](https://arxiv.org/html/2509.21679v1#A6.T10), agreement on ArgScore evaluation in Table [11](https://arxiv.org/html/2509.21679v1#A6.T11), agreement on QScore evaluation in Table [12](https://arxiv.org/html/2509.21679v1#A6.T12), and agreement on ReviewScore evaluation in Table [13](https://arxiv.org/html/2509.21679v1#A6.T13).
For the binary classification setup, we use Precision, Recall, and F1 Score, and for the 5-point scale setup, we use Pearson rank correlation, Gwet’s AC2 (Gwet, 2001), and Quadratic Weighted Kappa (Warrens, 2012).

**Table 10: Human-model agreement on ClaimScore evaluation.**
|  | Binary | 5-point Scale |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Precision | Recall | F1 | Pearson | AC2 | Kappa |
| Proprietary models |  |  |  |  |  |  |
| claude-sonnet-3.7 | 0.091 | 0.667 | 0.160 | 0.192 | 0.035 | 0.137 |
| claude-sonnet-4 | 0.091 | 0.200 | 0.125 | 0.172 | 0.048 | 0.130 |
| gpt-4o | 0.000 | 0.000 | 0.000 | 0.130 | 0.064 | 0.119 |
| gpt-5 | 0.083 | 0.250 | 0.125 | -0.014 | -0.057 | -0.004 |
| gemini-2.5-flash | 0.059 | 0.200 | 0.091 | 0.146 | 0.054 | 0.106 |
| Open-sourced models |  |  |  |  |  |  |
| qwen3-235b-a22b | 0.136 | 0.750 | 0.231 | 0.135 | 0.045 | 0.094 |
| llama-3.3 | 0.091 | 0.250 | 0.133 | 0.144 | 0.122 | 0.153 |
| deepseek-v3 | 0.000 | 0.000 | 0.000 | 0.195 | 0.086 | 0.165 |

**Table 11: Human-model agreement on ArgScore evaluation.**
|  | Binary | 5-point Scale |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Precision | Recall | F1 | Pearson | AC2 | Kappa |
| Proprietary models |  |  |  |  |  |  |
| claude-sonnet-3.7 | 0.345 | 0.679 | 0.458 | 0.425 | 0.369 | 0.380 |
| claude-sonnet-4 | 0.250 | 0.482 | 0.329 | 0.258 | 0.239 | 0.266 |
| gpt-4o | 0.483 | 0.250 | 0.329 | 0.280 | 0.221 | 0.239 |
| gpt-5 | 0.353 | 0.536 | 0.426 | 0.368 | 0.329 | 0.334 |
| gemini-2.5-flash | 0.333 | 0.679 | 0.447 | 0.419 | 0.358 | 0.366 |
| Open-sourced models |  |  |  |  |  |  |
| qwen3-235b-a22b | 0.381 | 0.429 | 0.403 | 0.281 | 0.012 | 0.149 |
| llama-3.3 | 0.344 | 0.196 | 0.250 | -0.014 | 0.124 | 0.107 |
| deepseek-v3 | 0.267 | 0.286 | 0.276 | 0.238 | 0.188 | 0.196 |

**Table 12: Human-model agreement on QScore evaluation.**
|  | Binary | 5-point Scale |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Precision | Recall | F1 | Pearson | AC2 | Kappa |
| Proprietary models |  |  |  |  |  |  |
| claude-sonnet-3.7 | 0.571 | 0.516 | 0.542 | 0.394 | 0.380 | 0.387 |
| claude-sonnet-4 | 0.462 | 0.563 | 0.507 | 0.387 | 0.335 | 0.357 |
| gpt-4o | 0.447 | 0.677 | 0.538 | 0.395 | 0.298 | 0.340 |
| gpt-5 | 0.500 | 0.594 | 0.543 | 0.320 | 0.285 | 0.300 |
| gemini-2.5-flash | 0.386 | 0.759 | 0.512 | 0.283 | 0.154 | 0.225 |
| Open-sourced models |  |  |  |  |  |  |
| qwen3-235b-a22b | 0.394 | 0.813 | 0.531 | 0.325 | 0.117 | 0.234 |
| llama-3.3 | 0.409 | 0.844 | 0.551 | 0.375 | 0.196 | 0.284 |
| deepseek-v3 | 0.469 | 0.821 | 0.597 | 0.325 | 0.101 | 0.228 |

**Table 13: Human-model agreement on ReviewScore evaluation.**
|  | Binary | 5-point Scale |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Precision | Recall | F1 | Pearson | AC2 | Kappa |
| Proprietary models |  |  |  |  |  |  |
| claude-sonnet-3.7 | 0.350 | 0.622 | 0.448 | 0.380 | 0.339 | 0.355 |
| claude-sonnet-4 | 0.291 | 0.495 | 0.367 | 0.360 | 0.308 | 0.333 |
| gpt-4o | 0.417 | 0.393 | 0.405 | 0.389 | 0.359 | 0.362 |
| gpt-5 | 0.370 | 0.543 | 0.441 | 0.327 | 0.300 | 0.310 |
| gemini-2.5-flash | 0.324 | 0.678 | 0.439 | 0.369 | 0.296 | 0.318 |
| Open-sourced models |  |  |  |  |  |  |
| qwen3-235b-a22b | 0.351 | 0.576 | 0.436 | 0.336 | 0.161 | 0.243 |
| llama-3.3 | 0.358 | 0.424 | 0.388 | 0.004 | 0.322 | 0.316 |
| deepseek-v3 | 0.328 | 0.459 | 0.382 | 0.361 | 0.278 | 0.310 |

### F.2 Qualitative Results

We present a thorough qualitative human-model disagreement analysis on ReviewScore evaluation in Figure [9](https://arxiv.org/html/2509.21679v1#A6.F9)–[14](https://arxiv.org/html/2509.21679v1#A6.F14).
For each review point type, we demonstrate two representative examples which include model-based scores and corresponding justifications using claude-sonnet-3.7 and claude-sonnet-4.

Figure: Figure 9: Example #1 of ClaimScore evaluation.

Figure: Figure 10: Example #2 of ClaimScore evaluation.

Figure: Figure 11: Example #1 of ArgScore evaluation.

Figure: Figure 12: Example #2 of ArgScore evaluation.

Figure: Figure 13: Example #1 of QScore evaluation.

Figure: Figure 14: Example #2 of QScore evaluation.

## Appendix G Model Details

We measure reliability of automatic ReviewScore evaluation with eight current state-of-the-art LLMs, including five proprietary models and three open-sourced models.
For proprietary models, we use
Anthropic’s claude-3-7-sonnet-20250219 (Anthropic, 2025a) and claude-sonnet-4-20250514 (Anthropic, 2025b),
OpenAI’s gpt-4o-2024-08-06 (OpenAI, 2024) and gpt-5-2025-08-07 (OpenAI, 2025),
and Google’s gemini-2.5-flash (Gemini Team, Google, 2025).
Since the ReviewScore evaluation does not require longer chain of thought, we exclude reasoning models. Following this rule, we use Anthropic’s claude models without thinking modes and OpenAI’s gpt-5 model with minimal reasoning effort.
For open-sourced models, we use
Alibaba’s qwen3-235b-a22b-2507 (Yang et al., 2025) with 235B model parameters,
DeepSeek’s deepseek-v3-0324 (DeepSeek-AI et al., 2024) with 671B model parameters
and Meta’s llama-3.3-70b-instruct (AI@Meta, 2024) with 70B model parameters.
We call the models by their APIs.

## Appendix H Prompts

We list every prompt used for automatic ReviewScore evaluation (Section [4](https://arxiv.org/html/2509.21679v1#S4)), automatic review point type classification (used in preprocessing stage of the dataset construction in Section [3.3](https://arxiv.org/html/2509.21679v1#S3.SS3)), and automatic argument reconstruction (Section [3.2](https://arxiv.org/html/2509.21679v1#S3.SS2)).
Figure [15](https://arxiv.org/html/2509.21679v1#A8.F15) – Figure [20](https://arxiv.org/html/2509.21679v1#A8.F20) indicate prompts for automatic ReviewScore evaluation and related score rubrics.
Figure [21](https://arxiv.org/html/2509.21679v1#A8.F21) and Figure [22](https://arxiv.org/html/2509.21679v1#A8.F22) indicate prompts for automatic review point type classification, where the type is one of claim, argument, and question.
Figure [23](https://arxiv.org/html/2509.21679v1#A8.F23) – Figure [29](https://arxiv.org/html/2509.21679v1#A8.F29) indicate prompts for automatic argument reconstruction.
In the prompts, capital letters with double square brackets (i.e., [[XYZ]]) would be replaced by the corresponding material, and words with a curly bracket (i.e., {abc}) would be replaced by the corresponding score rubric (Figure [15](https://arxiv.org/html/2509.21679v1#A8.F15) – Figure [17](https://arxiv.org/html/2509.21679v1#A8.F17)) or a python code snippet in Figure [26](https://arxiv.org/html/2509.21679v1#A8.F26).

Figure: Figure 15: Score rubric for evaluating unswerability.

Figure: Figure 16: Score rubric for evaluating factuality.

Figure: Figure 17: Score rubric for evaluating untrivialness.

Figure: Figure 18: Prompt used for evaluating QScore in a 5-point scale.

Figure: Figure 19: Prompt used for evaluating ClaimScore or WScore in a 5-point scale.

Figure: Figure 20: Prompt used for evaluating ArgScore in a 5-point scale.

Figure: Figure 21: Prompt used for deciding if a review point is an argument or not (is_argument).

Figure: Figure 22: Prompt used for deciding if a review point is a question or a claim (is_question).

Figure: Figure 23: Prompt used for extracting verbatim conjecture and reason statements in an argument (extract_verbatim_conclusion_reason).

Figure: Figure 24: Prompt used for reconstructing an argument (argument_reconstruction).

Figure: Figure 25: Prompt used for streamlining formalized reconstruction (streamlining).

Figure: Figure 26: Python code snippet for evaluating validity of reconstruction.

Figure: Figure 27: Prompt used for debugging python programs that evaluate validity of reconstruction (program_debugging).

Figure: Figure 28: Prompt used for translating FOL formulas with keys (i.e., defined variables/predicates) to NL reconstructed arguments (deformalization).

Figure: Figure 29: Prompt used for evaluating faithfulness of reconstruction (check_faithfulness).

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How consistent are human labels for the same review points across groups (e.g., claim vs argument disagreements)?
2. Could the reconstruction engine generalize to other reasoning tasks (e.g., debate analysis, scientific claim verification)?
3. Could incorporating citation graphs or author responses systematically (rather than as an optional experiment) raise reliability?
soundness: 3
strengths: 1. Novel Problem Definition: the paper identifies misinformed reviews as a measurable and impactful aspect of review quality — a previously underexplored issue. The operational definitions (answered question / incorrect premise) are specific yet widely applicable.

2. Methodological Innovation: The automatic argument reconstruction engine combining LLMs, formal logic translation, and a SAT solver is a creative and rigorous integration of symbolic reasoning and LLM capabilities. The validity–faithfulness feedback loops are well-motivated and thoughtfully designed.

3. High-quality Dataset: The authors build a trustworthy expert-annotated dataset (657 review points, 1,748 premises) with strong documentation and process controls (cross-annotation, consensus, training sessions).
summary: The paper introduces ReviewScore, a novel framework and dataset for detecting misinformed peer review points in academic paper reviews. They are defined as (1) questions already answered in the paper or (2) weaknesses based on incorrect premises. 

The authors verify that a significant fraction of review points (15.2% of weaknesses, 26.4% of questions) are misinformed. 

To automate this detection, they proposes a two-stage evaluation approach:
1. Base ReviewScore, measuring unanswerability and factuality on a 5-point scale.
2. Advanced ReviewScoreE, which decomposes review points into arguments and reconstructs their premises via an automatic argument reconstruction engine, checked for validity (using a SAT solver) and faithfulness (via LLM feedback loops).

A human-annotated dataset of 657 review points (ICLR 2021–2023) is created to benchmark human–model agreement across eight modern LLMs (Claude, GPT-4/5, Gemini, LLaMA-3, etc.). Experiments show moderate agreement (F1≈0.4–0.5, κ≈0.3–0.4), and the paper demonstrates that premise-level analysis significantly improves over direct weakness-level evaluation.
weaknesses: 1. Limited Agreement and Practical Readiness: Despite the substantial improvement over the baseline, the overall human-model agreement remains moderate (F1 ~0.45, Kappa ~0.35). This level of reliability is likely insufficient for fully autonomous deployment and would require human oversight, potentially reducing the net efficiency gains.

2. Subjectivity in Annotations: Inter-annotator α=0.30–0.43 is low, suggesting high ambiguity in human judgments of factuality. The paper’s conclusions about “moderate agreement” might partly reflect human inconsistency rather than LLM capability.

3. Dataset Scope and Diversity: Only ICLR reviews are used; the framework’s generality to NLP, CVPR, or smaller venues is unclear. Annotators are graduate students rather than senior researchers — potentially limiting domain depth.

4. Evaluation Simplifications Models were provided only the main paper text, not figures or appendices, which could substantially affect factual verification. The binary misinformed/not-misinformed threshold is somewhat arbitrary; alternative calibration curves could provide richer insight.

5. Argument Reconstruction Validation: While impressive, the argument reconstruction quality is mainly assessed via one model (Claude 3.7). Broader ablations (e.g., GPT-5 vs open models) or human inspection metrics could strengthen confidence.

---

confidence: 3
contribution: 1
initial_rating: 2
presentation: 3
questions: See the weaknesses.
soundness: 2
strengths: 1. The problem studied in this paper is valuable. 
2. The paper is well-organized.
summary: This paper attempts to detect low-quality reviews by defining a misinformed review point, i.e., ReviewScore. Specifically, the misinformed review point includes a question stated in a review that can already be answered by the paper, or a weakness stated in a review is incorrect or contains incorrect premises regarding the paper. After that, the authors build a human expert-annotated benchmark to evaluate the ability of current SOTA LLMs.
weaknesses: 1. My main concern is the proposed definition 2 (Misinformed Review Point), which is the core assumption of this paper. The claim "a question stated in a review can already be answered by the paper" may be caused by many other cases, not just the misinformed point. For example, maybe the paper is poorly written, the clarity is misleading, or the reviewer is not familiar with this area, and so on. If this paper is poorly written
2. We believe that, in the peer-review system, scholars evaluate their academic level rankings based on the "consistency assumption", i.e., scholars with stronger abilities usually have stronger persuasiveness for evaluating others, and these scholars can also obtain higher achievements [1]. Therefore, we believe the "review quality" is mainly determined by the human/expert preference. And the "low-quality review problem" is a system problem.  


[1] PICO: PEER REVIEW IN LLMS BASED ON CONSISTENCY OPTIMIZATION. ICLR'2025.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: * How well does your argument reconstruction engine perform on an external dataset or human-reconstructed arguments?
* Have you tested REVIEWSCORE on reviews from other venues to see how it generalizes beyond the training data?
* Could REVIEWSCORE be used as part of a rebuttal assistant or reviewer feedback system? If yes, what would that pipeline look like?
soundness: 2
strengths: * Well-defined problem: The concept of “misinformed review points” is clearly formalized and reflects real concerns authors face, such as reviewers asking questions that are already answered in the paper.
* Argument reconstruction engine: The paper presents a novel approach that extracts both explicit and implicit premises from weaknesses, enabling premise-level factuality evaluation—a step beyond most prior work.
* Empirical study with LLMs: Eight LLMs are evaluated on the REVIEWSCORE dataset, with quantitative results showing moderate human-model agreement, and ADVANCED scoring improves over BASE.
summary: This paper introduces REVIEWSCORE, a framework to automatically detect misinformed review points in peer reviews from AI conferences. A misinformed review point is defined as either (1) a question already answered by the paper or (2) a weakness based on incorrect or unsupported claims. The authors develop two scoring systems: BASE REVIEWSCORE and ADVANCED REVIEWSCORE

The authors construct a human-annotated dataset of review points from 40 ICLR 2021–2023 papers, labeled by 15 graduate students. They then evaluate agreement between these human annotations and predictions from 8 SOTA LLMs (GPT-4, Claude 2, PaLM 2, Gemini, etc.), finding that ADVANCED REVIEWSCORE achieves higher alignment than BASE, especially in weakness detection.
weaknesses: * Insufficient dataset size: The dataset only covers 40 ICLR submissions across three years (~0.4% of the total), which is far too small to support general conclusions or train reliable automated systems. Most findings should be considered preliminary.
* Limited diversity and experience of annotators: All annotations are from graduate students. There’s no evidence that experienced reviewers, ACs, or SACs were consulted—this may bias the labeling toward an author-centric view of review quality.
* No practical deployment pathway: The paper does not explore how REVIEWSCORE could be integrated into real conference workflows (e.g., as part of rebuttal, AC dashboards, or review filtering), nor does it simulate such use cases.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. What is the rationale behind limiting the classification to just these two categories? Are there other types of misinformed review points, such as misunderstandings of methodologies, misinterpretation of results, or reviewer biases, that could be considered?
2. Could you provide a more in-depth comparative analysis to highlight the strengths and limitations of REVIEWSCORE in relation to other state-of-the-art review quality assessment tools? How does REVIEWSCORE perform in comparison to other AI-based approaches for identifying misinformed review points?
3. How do you plan to address the generalizability of your findings given the limited scope of the dataset? Additionally, there could be biases introduced during the human annotation process. Could you expand the dataset to include a wider range of conferences and academic domains to ensure robustness? Also, could the inclusion of external data sources (e.g., citation counts, feedback from future users) strengthen the accuracy and completeness of the dataset?
soundness: 3
strengths: * This paper introduces a refined approach to reviewing AI conference papers by providing a more detailed examination of each review point. 
* The paper proposes the ADVANCED REVIEWSCORE, which evaluates the factuality of individual premises within arguments, proving to be more effective than simply evaluating weaknesses at a high level.
summary: This paper addresses the degrading quality of peer review in AI conferences, a problem driven by the explosive growth in paper submissions. The authors aim to reliably detect low-quality review by defining "misinformed review points" as either reviewer "questions" that are already answerable by the paper or "weaknesses" that are incorrect or based on incorrect premises. The primary contribution is ReviewScore, a new evaluation criterion to automatically identify these misinformed points. The paper introduces base ReviewScore for foundational scoring and a more effective advanced ReviewScore, which breaks arguments down to evaluate the factuality of each individual premise. To enable this, the authors also propose an automatic argument reconstruction engine that uses LLMs to extract all explicit and implicit premises from a review. To validate this automated system, the authors built a human expert-annotated ReviewScore dataset and measured the performance of eight state-of-the-art LLMs against it. The results showed moderate human-model agreement and confirmed that evaluating premise-level factuality is significantly more reliable than evaluating weakness-level factuality.
weaknesses: 1. While the paper introduces the REVIEWSCORE system to detect misinformed review points, its classification of errors into two categories: questions and weaknesses, appears too simplistic. There are likely more nuanced types of errors in peer reviews that this system does not currently address. 
2. Although the paper presents REVIEWSCORE to detecting misinformed review points, it fails to adequately compare its method with existing peer review evaluation systems.
3. The paper’s experimental evaluation relies on a small and homogeneous dataset consisting of 40 ICLR submissions. While this serves as a proof of concept, the limited size and scope of the dataset restrict its generalizability. The dataset's lack of diversity in terms of conferences and academic domains may not reflect the wide range of review styles and subjects encountered across various fields.


Decision: Reject