NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimation

## Abstract

Abstract The ability to estimate the quality of scientific papers is central to how both humans and AI systems will advance scientific knowledge in the future. However, existing LLM-based estimation methods suffer from high inference cost, whereas the faster direct score regression approach is limited by scale inconsistencies. We present NAIPv2, a debiased and efficient framework for paper quality estimation. NAIPv2 employs pairwise learning within domain-year groups to reduce inconsistencies in reviewer ratings and introduces the Review Tendency Signal (RTS) as a probabilistic integration of reviewer scores and confidences. To support training and evaluation, we further construct NAIDv2, a large-scale dataset of 24,276 ICLR submissions enriched with metadata and detailed structured content. Trained on pairwise comparisons but enabling efficient pointwise prediction at deployment, NAIPv2 achieves state-of-the-art performance (78.2% AUC, 0.432 Spearman), while maintaining scalable, linear-time efficiency at inference. Notably, on unseen NeurIPS submissions, it further demonstrates strong generalization, with predicted scores increasing consistently across decision categories from Rejected to Oral. These findings establish NAIPv2 as a debiased and scalable framework for automated paper quality estimation, marking a step toward future scientific intelligence systems.

## 1 Introduction

The advent of large language models (LLMs) has propelled scientific research into an unprecedented phase of acceleration. One of the most immediate manifestations is the explosive growth of scholarly publications, with about 242,740 AI-related publications in 2023 (Maslej et al., 2025). Such an overwhelming influx of literature not only far exceeds the limits of human reading capacity but also creates significant challenges for automated scientific systems (Lu et al., 2024; Si et al., 2024; Baek et al., 2024; Wang et al., 2024; Zhao et al., 2024). To discern valuable insights and maintain a leading academic perspective, the necessity of focusing on high-quality work in the most recent publications has become increasingly paramount. However, assessing the quality of early-stage articles poses particular challenges, as the absence of citation histories and other bibliometric indicators provides little evidence of their quality. At the same time, both DoRA (F1000Research et al., 2012) and the Leiden Manifesto (Hicks et al., 2015) strongly discourage the use of journal or conference prestige as a proxy for paper quality. Consequently, growing attention is directed toward approaches that evaluate research quality based on content-related features rather than external attributes.

For decades, expert peer review has served as a reliable standard for evaluating research quality . While rigorous, this process is costly, time-consuming, and inherently difficult to scale. In response, recent studies (Yu et al., 2024; Zhu et al., 2025; D’Arcy et al., 2024; Idahl & Ahmadi, 2024) have investigated the use of LLMs to assist in the review process (e.g., generating review comments, predicting review scores, or inferring acceptance decisions directly from paper content). As shown in Fig. [1](https://arxiv.org/html/2509.25179v2#S1.F1) left panel, while these approaches demonstrate promising potential, their reliance on autoregressive reasoning over long contexts introduces substantial latency and computational overhead, thereby limiting deployment in automated scientific systems (e.g., literature intelligence systems).

Figure: Figure 1: Comparison of various frameworks. AutoRegressive approaches (Zhu et al., 2025; Weng et al., 2025) rely on sequential generation, resulting in substantial inference latency; NAIPv1 (Zhao et al., 2025) enables fast pointwise regression but suffers from scale inconsistency; NAIPv2 leverages debiased pairwise training with confidence-aware signals and operates as an efficient pointwise regressor with linear-time complexity at inference.
Refer to caption: x1.png

To mitigate the inherent latency of autoregressive models, we re-examine regression approaches that target citation-based indicators (middle part of Fig. [1](https://arxiv.org/html/2509.25179v2#S1.F1)). Despite recent research exploring the direct prediction of various citation-based indicators from textual content (Zhao et al., 2025; de Winter, 2024), such approaches primarily emphasize forecasting research impact rather than assessing the intrinsic quality of papers. As reported by  Cortes & Lawrence (2021) and  Dougherty & Horne (2022), citation counts are considered unreliable proxies of article quality. This underscores the need to shift quality estimation toward more intrinsically reliable signals (i.e., peer review score). However, our experiments reveal that directly regressing on review scores remains unsatisfactory. We attribute this shortcoming to two fundamental factors: First, review scores may lack a consistent and stable measurement scale. The distribution of review scores differs across research domains and publication years. In addition, conference evaluation standards and invited expert reviewers may shift over time. Second, reviewer confidence is largely neglected in existing formulations. Each review score is typically accompanied by a confidence rating, reflecting the reviewer’s self-assessed reliability of judgment. Nevertheless, most approaches ignore this signal, treating all scores as equally trustworthy. This neglect allows low-confidence assessments to exert the same influence as high-confidence ones, introducing additional label noise and weakening the supervisory signal.

Building on these observations, we propose NAIPv2 (originating from Newborn Article Impact Prediction), a fast and debiased framework for paper quality assessment. NAIPv2 learns the relative ordering of submissions within the same domain–year group in a pairwise manner, thereby mitigating biases introduced by cross-domain, temporal variations. During inference, the framework is decoupled into a simplified single-branch pointwise regressor, enabling robust score prediction while preserving linear-time complexity. In addition, we introduce a probabilistic treatment of review scores and reviewer confidences, where each score is viewed as a noisy observation of the latent paper quality and confidence determines its uncertainty, yielding a principled aggregation signal. Experimental results demonstrate that our method attains state-of-the-art performance (78.2% AUC and 0.432 Spearman) while offering substantially improved inference efficiency. Notably, on unseen NeurIPS submissions, the predicted scores align precisely with decision categories, consistently assigning the lowest values to rejected papers and the highest values to oral presentations.

In summary, our main contributions are as follows:

- 1.
A large-scale domain-labeled dataset for quality estimation. We release NAIDv2, a large-scale dataset of 24,276 ICLR submissions (2021–2025) that combines metadata, parsed PDF content, and clustered labels to enable cross-domain debiasing and foster research on automated peer review and scholarly evaluation.
- 2.
A scalable and debiased framework for scientific article quality estimation. We propose a scalable and debiased framework NAIPv2 for early-stage paper quality prediction. Beyond achieving state-of-the-art performance, NAIPv2 exhibits strong generalization and consistency on unseen NeurIPS review data.
- 3.
Confidence-aware probabilistic signal. We introduce the RTS, a probabilistic formulation that models review scores as Gaussian observations of latent paper quality with reviewer confidence shaping uncertainty, providing a principled aggregation that yields a reliable comparative signal and clarifies the role of confidence in evaluation outcomes.

## 2 Related Work

Peer Review Dataset.
With its open-access policy, ICLR has become a widely used primary data source in contemporary research Staudinger et al. (2024). However, such reliance also introduces considerable risks of information leakage. The first type of leakage is induced by large language models. For example, PeerRead (Kang et al., 2018) is an early and influential dataset, containing more than 14K paper drafts with accept/reject decisions from leading venues. Given its early release, the dataset has likely been incorporated into LLM training corpora, thereby raising concerns about unintended information leakage. The second type of leakage results from overlaps between training and test sets. Recently, Zhang et al. (2025a) introduced the Re^2 dataset. However, some of its benchmark data may overlap with training sets employed by other methods, which can lead to deviations between reported results and those presented in the original papers. Following rigorous academic practice, we construct the NAIDv2 test set exclusively from the latest ICLR 2025 data, thereby minimizing the risk of potential information leakage.

Automated Peer Review.
To optimize performance on review-related tasks, most automated peer-review methods construct dedicated datasets and perform task-specific fine-tuning.  Yu et al. (2024) introduces a modular framework for standardization, evaluation, and analysis, and further proposes a mismatch score to measure paper–review alignment. OpenReviewer (Idahl & Ahmadi, 2024) fine-tunes an 8B-parameter LLM on expert reviews to produce structured, guideline-compliant feedback. DeepReview (Zhu et al., 2025) employs a multi-stage pipeline integrating structured analysis, literature retrieval, and evidence-based reasoning. CycleReviewer Weng et al. (2025) goes beyond single-pass reviewing by coupling manuscript generation with iterative peer review through reinforcement learning, aiming to automate the entire research-review cycle. Together, these approaches exploit LLMs’ CoT reasoning, boosting performance but incurring substantial computational cost. Instead, NAIPv2 adopts pairwise learning to address scale inconsistency and decouples into a pointwise regressor at inference, maintaining competitive accuracy with markedly improved speed.

Numerical Impact and Quality Estimation.
Predicting the future impact of scientific papers has long been an active area of study (Zhao & Feng, 2022; Xia et al., 2023; Thelwall, 2025), typically leveraging early signals such as citation counts, author profiles, venue prestige, and linguistic features. More recently, LLM-based approaches have been explored:  de Winter (2024) investigated whether ChatGPT can estimate bibliometric indicators including citation counts, readership, and social media interactions, finding stronger correlations than traditional readability metrics, while  Zhao et al. (2025) employed LLMs to directly predict a field- and time-normalized impact score ($TNCSI_{sp}$) from titles and abstracts, achieving an MAE of 0.216.

A number of methods have also been proposed to estimate research quality. Prior research (Ghosal et al., 2019; Bharti et al., 2021) has primarily focused on using machine learning algorithms to predict final recommendation scores or classify papers into accept/reject outcomes. Such practices, however, tend to overlook the role of confidence annotations. In contrast, our approach jointly models scores and confidence in a probabilistic framework, thereby preserving the semantics of uncertainty and enabling principled aggregation across reviewers.

Pairwise Ranking.
Pairwise ranking methods have long been established as effective approaches in information retrieval and recommendation (Liu et al., 2009; Cao et al., 2007), yet their adoption in peer review and scientometric prediction remains limited. Höpner et al. (2025) apply pairwise ranking for score prediction and find it feasible for citation counts but unreliable for review scores. Zhang et al. (2025d) investigate pairwise manuscript evaluation using off-the-shelf LLMs. Owing to the quadratic growth of pairwise comparisons, they demonstrate that aggregating fewer than 2% of all possible pairs is sufficient to recover a robust global ranking. Distinct from prior studies, this work addresses debiasing and efficiency by training in a pairwise manner while performing pointwise inference, thereby achieving inference with linear complexity.

## 3 Methods

### 3.1 Review Tendency Signal (RTS)

In practice, reviewers are not always domain experts in the exact area of a submission, which may prevent them from providing accurate evaluations. To mitigate this source of uncertainty, most conferences and journals require reviewers to report not only an overall score but also an accompanying confidence level, reflecting the degree of trust they place in their own assessment.

Our proposed solution is to interpret each review not as a deterministic score but as a noisy observation of the latent “true quality” of the paper. In this view, the reported overall score $s_{i}$ is treated as the mean of a probability distribution, while the confidence level $c_{i}$ determines the variance (or equivalently, the precision) of that distribution. High-confidence reviews correspond to narrow distributions centered around their scores, whereas low-confidence reviews correspond to wider distributions, reflecting greater uncertainty.

Formally, we assume each review $i$ provides an observation modeled as a Gaussian likelihood. Here, both the reported score and the confidence level are normalized to the unit interval $[0,1]$.

$$ $p(s_{i}\mid x,c_{i})\;=\;\mathcal{N}\!\left(s_{i}\mid x,\sigma(c_{i})^{2}\right),$ (1) $$

where $x\in[0,1]$ denotes the latent true quality, and $\sigma(c_{i})$ is a decreasing function of confidence, ensuring that higher confidence yields lower variance.

Given $n$ independent reviews, the posterior distribution over $x$ can be written as:

$$ $p(x\mid s_{1:n},c_{1:n})\;\propto\;\prod_{i=1}^{n}p(s_{i}\mid x,c_{i}).$ (2) $$

We then define the Review Tendency Signal (RTS) of a submission as the posterior mean:

$$ $\mathrm{RTS}\;=\;\mathbb{E}[x\mid s_{1:n},c_{1:n}].$ (3) $$

This probabilistic formulation naturally balances reviewer scores according to their associated uncertainties. Unlike weighted averaging, which collapses confidence to a fixed coefficient, $RTS$ preserves the semantics of uncertainty, enables principled aggregation across reviewers, and guarantees that the resulting estimate lies within the valid range $[0,1]$. Defination of $\sigma(c_{i})$ and more details are provided in the Appendix [F](https://arxiv.org/html/2509.25179v2#A6).

### 3.2 Construction of the NAIDv2 Dataset

To facilitate efficient debiasing in training and evaluation, we construct the NAIDv2 (where D denotes Dataset). Following established community practices, we build it from ICLR conference review data provided by OpenReview.net, ensuring an acceptance ratio that closely reflects real-world conditions. Given that review data is continually updated, earlier studies either failed to incorporate the latest ICLR reviews or covered a narrower time span (Zhu et al., 2025; Weng et al., 2025). To address this, we developed a dedicated data engine and used it to compile NAIDv2, encompassing 24,276 samples spanning 2021–2025. Although our experiments relied only on review scores, confidence ratings, and related fields, the underlying SQL-based dataset also retains additional information, including the full textual review comments. Furthermore, we downloaded the corresponding original PDF submissions and applied MinerU for structured parsing.

In addition to its scale and coverage, NAIDv2 is distinguished by its explicit handling of domain bias (as depicted in Fig. [2](https://arxiv.org/html/2509.25179v2#S3.F2)). As discussed earlier, raw review scores exhibit inconsistencies across domains and across time. To correct for these distortions, we move beyond noisy keyword-based domain labeling and instead adopt a clustering-driven strategy. Specifically, we encode paper titles and abstracts using Qwen3-Embedding-4B (Zhang et al., 2025c) and apply hierarchical clustering (Bar-Joseph et al., 2001; Müllner, 2011) to identify latent domains based on semantic relatedness. Accordingly, we also compute the RTS from the raw scores, providing a robust signal for debiased training. This pipeline produces domain-normalized labels that more faithfully capture expert consensus and enable debiased pairwise training.

Figure: Figure 2: Debiased data construction in NAIDv2. Titles and abstracts are embedded and hierarchically clustered into latent domains to mitigate domain bias, while raw scores with reviewer confidences are modeled via Gaussian likelihoods to calculate RTS, yielding a normalized dataset for robust debiased pairwise training.
Refer to caption: x2.png

For additional information on dataset construction, please refer to the Appendix [I](https://arxiv.org/html/2509.25179v2#A9).

### 3.3 Pairwise Training and Pointwise Scoring with NAIPv2

Pairwise Training Stage.
In the training phase, NAIPv2 adopts a pairwise learning paradigm, where the model is optimized to capture relative quality differences between two papers rather than directly regressing absolute scores (depicted in the rightmost panel of Fig. [1](https://arxiv.org/html/2509.25179v2#S1.F1)).
Given a pair of submissions $(a,b)$ with ground-truth review tendency signal ${RTS}_{a}$ and ${RTS}_{b}$, we define a binary label ${RTS}_{ab}=\mathbb{I}[\,{RTS}_{a}>{RTS}_{b}\,]$ and obtain predictions $\hat{y}_{a},\;\hat{y}_{b}$ from a shared LLaMA-3 (Dubey et al., 2024) backbone. $\mathbb{I}[\cdot]$ denotes the indicator function, which equals $1$ if the condition holds and 0 otherwise. The pairwise preference probability is:

$$ $\hat{z}_{ab}=sigmoid(\hat{y}_{a}-\hat{y}_{b}).$ (4) $$

The training objective is the binary cross-entropy loss.

$$ $\mathcal{L}(a,b)=-\Big({RTS}_{ab}\,\log\hat{z}_{ab}+(1-{RTS}_{ab})\,\log(1-\hat{z}_{ab})\Big).$ (5) $$

To mitigate distributional biases across domains and years, training pairs $(a,b)$ are restricted to submissions within the same cluster and the same publication year. This design prevents spurious comparisons caused by cross-field or temporal inconsistencies.

To further enhance robustness and stability, we employ a curriculum learning mechanism based on the gap in $RTS$. Specifically, we partition training pairs into difficulty buckets according to:

$$ $\Delta_{ab}=|\,{RTS}_{a}-{RTS}_{b}\,|,$ (6) $$

where pairs with larger $\Delta_{ab}$ are considered easier, while those with smaller $\Delta_{ab}$ are treated as harder. During the early stages of training, easier pairs dominate the sampling distribution, guiding the network to quickly capture coarse quality distinctions. As training progresses, harder pairs are gradually upsampled, enabling the model to learn fine-grained quality differences.

Pointwise Inference Stage.
Although training supervision is provided only on submission pairs $(a,b)$, the shared backbone guarantees that each input is independently mapped to a scalar score:

$$ $\hat{y}_{a}=f(x_{a};\theta),\quad\hat{y}_{b}=f(x_{b};\theta),$ (7) $$

where $x_{a}$ and $x_{b}$ denote the textual representations of submissions $a$ and $b$, respectively.
In other words, the pairwise objective implicitly induces a decoupled pointwise scoring function under shared parameters $\theta$.

At inference time, this function naturally generalizes to single-paper evaluation:

$$ $\hat{y}=f(x;\theta),$ (8) $$

which provides a pointwise estimate of paper quality that is directly comparable across submissions within the same domain-year subgroup.
Although the pairwise loss constrains only relative differences, the shared backbone forces all submissions into a common representation space, thereby yielding in practice a globally consistent scoring scale (up to an additive constant).

## 4 Experiments

### 4.1 Metrics

We evaluate the performance of our method from two different viewpoints.
The first viewpoint focuses on predicting whether a paper is accepted or rejected, an interesting and practical task evaluated by classification accuracy.
The second viewpoint addresses the requirements of literature intelligence systems (e.g., recommendation systems, automated research assistants, or automatic literature review generation system), where the emphasis is on the quality of score-based ranking and recommendation.

For classification evaluation, we report Accuracy (Acc), F1-score (F1), ROC-AUC (AUC), and Pairwise Accuracy (P. Acc). AUC is defined as $\mathrm{AUC}=1/(|P||N|)\sum_{i\in P}\sum_{j\in N}\mathbb{I}[\hat{y}_{i}>\hat{y}_{j}]$, where $P$ and $N$ are the positive and negative sets. PairAcc measures pairwise directional correctness using thresholded predictions as $\mathrm{PairAcc}=1/|\Omega|\sum_{(i,j)\in\Omega}\mathbb{I}[(\tilde{y}_{i}-\tilde{y}_{j})(y_{i}-y_{j})>0]$, where $\Omega=\{(i,j):i<j,\;y_{i}\neq y_{j}\}$. For regression-based methods, classification results are reported at the threshold yielding the best F1 score.

For ranking evaluation, we use Spearman’s $\rho$ and NDCG@$k$. Spearman’s $\rho$ is $\rho=1-6\sum_{i=1}^{n}d_{i}^{2}/(n(n^{2}-1))$ where $d_{i}$ is the rank difference. NDCG@$k$ is $\mathrm{NDCG@}k=(1/\mathrm{IDCG@}k)\sum_{i=1}^{k}(2^{s_{i}}-1)/\log_{2}(i+1)$ where $s_{i}$ is the ground-truth relevance and $\mathrm{IDCG@}k$ is the maximum DCG. We set $k=20$ in experiments.

### 4.2 Implementation Details

We divide NAIDv2 into training, validation, and test subsets.
The validation set is sampled from the training data with an internal 9:1 split and is used for hyperparameter tuning.
To avoid potential information leakage (e.g., earlier ICLR submissions that might have been included in the LLaMA-3 pretraining corpus), the test set is restricted to submissions from the year 2025, comprising 1,029 samples. NAIPv2 is fine-tuned on 4$\times$A40 GPUs (48 GB each) using 8-bit quantization and LoRA adaptation (Hu et al., 2022), with a training time of roughly 1 hour for 10k pairs (constructed from 23,247 samples). On a more common consumer GPU (RTX 3090, 24 GB), the same configuration is conservatively estimated to complete within 6 hours, making the approach accessible to most academic labs. The main hyperparameters include a learning rate of $1\times 10^{-4}$, a batch size of 8, and training for one epoch.

### 4.3 Comparison with Previous Approaches

As shown in Tab. [1](https://arxiv.org/html/2509.25179v2#S4.T1), we compare our proposed method against various approaches. Existing methods can be broadly divided into three categories. Category I consists of prompting LLMs to predict review scores or acceptance decisions based on paper titles, abstracts, or full texts. Owing to the training-free nature of API calls, this paradigm has gained popularity, though its long-term usage costs remain high. Category II encompasses approaches that fine-tune LLMs for automatic reviewing tasks or develop specialized frameworks. The goal of these methods is to maximize the reasoning and analytical strengths of LLMs, enabling them to produce comprehensive review texts. Category III fine-tunes LLMs to regress specific numerical values. Since these approaches avoid the autoregressive generation process, they achieve much faster inference speeds. In addition, we also report the empirical upper and lower bounds on the NAIDv2 Test Set. The upper bound is obtained by training an MLP with cross-entropy loss, where the inputs are reviewer scores and the ground truth is the binary acceptance decision (0/1). The lower bound is determined by a stochastic model.

**Table 1: Performance comparison on the ICLR review prediction task. $\uparrow$ denotes the higher the better. Results marked with * are taken from Zhu et al. (2025); Yu et al. (2024). “-” means the metric is not reported. Bold numbers denote the best results.**
| Category | Method | Acc $\uparrow$ | F1 $\uparrow$ | AUC $\uparrow$ | NDCG $\uparrow$ | $\bm{\rho}$ |
| --- | --- | --- | --- | --- | --- | --- |
| - | Lower Bound (Random) | 0.514 | 0.410 | 0.527 | 0.525 | 0.002 |
|  | Upper Bound (Info. Leak) | 0.819 | 0.757 | 0.894 | 0.995 | 0.984 |
| API | API-pointwise (ChatGPT) | 0.644 | 0.427 | 0.654 | 0.7024 | 0.315 |
|  | API-pairwise (ChatGPT) | 0.658 | 0.448 | 0.655 | 0.686 | 0.297 |
|  | AgentReview* (Gemini) | 0.678 | 0.559 | - | - | 0.279 |
|  | AI Scientist* (Gemini) | 0.678 | 0.573 | - | - | 0.267 |
| AutoRegressive | SEA-EA* (7B) | 0.582 | 0.563 | - | - | - |
|  | CycleReviewer* (8B) | 0.678 | 0.559 | - | - | 0.279 |
|  | CycleReviewer* (70B) | 0.678 | 0.573 | - | - | 0.267 |
|  | DeepReviewer* (14B) | 0.689 | 0.623 | - | - | 0.408 |
| Regress Only | NAIPv1 (8B) | 0.545 | 0.472 | 0.605 | 0.629 | 0.183 |
|  | NAIPv2 (ours, 8B) | 0.706 | 0.609 | 0.782 | 0.771 | 0.432 |

For the first category, we implement the “API (pointwise)” and “API (pairwise)” approaches with reference to  Höpner et al. (2025) (see Appendix [G](https://arxiv.org/html/2509.25179v2#A7) for details). This category has emerged as the most common choice among researchers due to its ease of use; however, neither the pointwise nor the pairwise variant has demonstrated satisfactory performance. The second research direction explores autoregressive inference using LLMs. Due to supervised fine-tuning on specialized data, these methods typically demonstrate stronger performance. DeepReviewer (Zhu et al., 2025) reaches an F1 score of 62.3 and a Spearman correlation of 0.408 on their test set, demonstrating strong capability on both acceptance decision prediction and score ranking tasks. Nevertheless, since the method generates tokens autoregressively, inference time scales with output length. On mainstream consumer GPUs (e.g., RTX 3090), inference for a single paper can take up to three minutes, which limits its applicability in literature intelligence systems. Regression-only based approaches seek to bypass autoregression by directly predicting a single numerical output. NAIPv1 (Zhao et al., 2025), for example, achieves up to ten predictions per second on the RTX 3090. However, since its training objective is aligned with influence metrics rather than paper quality, its predictive quality is only marginally better than random guessing. By contrast, our proposed approach leverages a debiased pairwise learning design, matching NAIPv1 in inference efficiency (over a thousand times faster than typical autoregressive approaches) while attaining performance on par with DeepReviewer.

### 4.4 Comparison of Learning Paradigms and Inference Complexity

**Table 2: Comparison of alternative paradigms. The last column reports the theoretical lower-bound inference complexity of each paradigm.**
| Paradigm | AUC | $\bm{\rho}$ | Th. Compl. |
| --- | --- | --- | --- |
| Pointwise | 0.633 | 0.237 | $O(C)$ |
| Pairwise Concat | 0.720 | 0.351 | $O(C\log C)$ |
| NAIPv2 (ours) | 0.782 | 0.432 | $O(C)$ |

p
We further analyze the paradigm differences to underscore the advantages of the NAIPv2 design. Table [2](https://arxiv.org/html/2509.25179v2#S4.T2) compares three variants: (i) a pointwise method based on the NAIPv1 architecture but using RTS labels in place of the originals, (ii) a pairwise variant that adopts the same architecture as the pointwise model but concatenates two papers into a single input sequence and predicts a binary 0/1 label, and (iii) our proposed NAIPv2. The pointwise approach suffers from inconsistencies, resulting in suboptimal performance. The concatenation-based variant achieves higher accuracy, but its inference requires explicit pairwise comparisons, leading to a complexity of $O(C\log C)$, where $C$ denotes the number of candidate papers. In contrast, NAIPv2 incorporates debiased pairwise learning during training while retaining pointwise inference, thus preserving the linear complexity $O(C)$ of pointwise methods and achieving superior performance.

### 4.5 Effect of Clustering Strategies on Performance

The strategy chosen for pairwise grouping plays a crucial role in the model’s capacity to distinguish between papers within the same set. Table [4](https://arxiv.org/html/2509.25179v2#S4.T4) presents the performance differences across grouping methods. When no grouping is applied, or when grouping is based solely on temporal information, the model achieves only marginal improvements over direct prediction, and the results remain below optimal. Grouping via GPT-generated keywords performs worst, with accuracy approaching random guessing. This outcome is mainly attributed to the overabundance of candidate keywords, which produces sparse valid pairs and insufficient training data. In contrast, hierarchical clustering outperforms temporal grouping alone, and the combination of temporal and hierarchical grouping yields the strongest overall results. These findings highlight the critical role of debiasing in pairwise learning for paper quality assessment.

Figure [4](https://arxiv.org/html/2509.25179v2#S4.F4) further illustrates the effect of the maximum distance parameter (max distance) in hierarchical clustering. A small max distance produces numerous small clusters with limited samples, thereby suffering from sparsity similar to keyword-based grouping. Conversely, a large max distance leads to a few large clusters, essentially collapsing the structure into a setting akin to no grouping. The results indicate that performance is compromised at both extremes, with the optimal outcome attained when the max distance parameter is set to 1.0.

**Table 3: Debias Strategy. P. Acc stands for Pairwise Accuracy**
| Group By | AUC | P. Acc | $\bm{\rho}$ |
| --- | --- | --- | --- |
| None | 0.739 | 0.630 | 0.400 |
| Pub. Time | 0.736 | 0.638 | 0.392 |
| Keyword | 0.556 | 0.620 | 0.007 |
| Hier. Cluster | 0.753 | 0.639 | 0.401 |
| Time+Hier. | 0.782 | 0.649 | 0.432 |

### 4.6 Benefit of the proposed RTS

Table [4](https://arxiv.org/html/2509.25179v2#S4.T4) compares RTS with common aggregation strategies. Simple averaging (Mean) and its weighted variant show only marginal differences, indicating that naïve incorporation of confidence contributes little. Median aggregation performs slightly better, but the gain remains limited. Mode aggregation, on the other hand, leads to the weakest performance, as it discards the distributional nuances of reviewer scores. In comparison, RTS achieves consistently stronger results. Its probabilistic treatment of reviewer confidence allows more informative integration of scores, resulting in a more discriminative and reliable quality signal than all traditional baselines.

Figure: Figure 3: Effect of clustering granularity.
Refer to caption: x3.png

### 4.7 Learning with Difficulty-Aware Supervision

**Table 5: Impact of curriculum learning (“w/o” denotes without, “w/” denotes with).**
| Setting | AUC | $\rho$ |
| --- | --- | --- |
| w/o Curriculum | 0.770 | 0.418 |
| w/ Curriculum | 0.782 | 0.432 |

To study the effect of pairwise difficulty, we group training pairs into buckets by the $RTS$ gap $\Delta_{ab}$ (detailed in Appendix [H](https://arxiv.org/html/2509.25179v2#A8)), where larger gaps indicate easier pairs and smaller gaps correspond to harder ones. Figure [4](https://arxiv.org/html/2509.25179v2#S4.F4) shows that performance consistently improves as the proportion of easy pairs increases. This finding suggests that simple comparisons provide more stable and reliable learning signals, whereas emphasizing hard pairs tends to introduce noise and degrade performance. Building on this observation, we further adopt a curriculum learning strategy. In the early training stages, sampling is biased toward easy pairs, allowing the model to first capture coarse distinctions. Over time, harder pairs are gradually introduced, enabling the model to refine its judgment on subtle quality differences. The results in Tab. [5](https://arxiv.org/html/2509.25179v2#S4.T5) verify that curriculum learning provides measurable improvements, supporting it as a useful complement to difficulty-aware supervision.

## 5 From ICLR to NeurIPS: A Generalization Case Study

To further examine the generalization ability of our model, we conduct a cross-venue evaluation on NeurIPS. Following the same protocol as for NAIDv2, we construct a NeurIPS test set comprising 13,223 samples, covering submissions from 2021 to 2024. Unlike ICLR, NeurIPS exhibits an imbalance in review availability: reviews of accepted papers always appear in public, whereas reviews of rejected papers emerge only when authors choose to disclose them. This practice inflates the proportion of accepted papers in the accessible record relative to the true acceptance rate of about 20%, thereby producing a strongly skewed distribution. This imbalance renders previous tasks on the NeurIPS dataset less meaningful.

Considering that NeurIPS organizes submissions into a hierarchy, with rejected papers at the lowest tier and accepted papers further divided into poster, spotlight, and oral presentations of increasing contribution and quality, it is natural to expect stronger papers to receive higher scores and be placed into more prestigious categories. We examine whether our predicted evaluation scores (sigmoid normalized) align with this hierarchy, and as shown in Fig. [5](https://arxiv.org/html/2509.25179v2#S5.F5) (a), the model outputs follow a clear upward trend from rejected to poster, spotlight, and oral papers compared to NAIPv1 and the pointwise API-based method, demonstrating strong consistency with the actual conference decisions.

Figure: Figure 5: Cross-venue evaluation on NeurIPS. (a) Predicted scores follow the hierarchical structure of conference decisions. (b) Distribution of rejected papers across score ranges.
Refer to caption: x5.png

In Fig. [5](https://arxiv.org/html/2509.25179v2#S5.F5) (b), we show the distribution of rejected papers across score ranges. Each row corresponds to the min-to-max assigned scores. Light green denotes accepted papers, while deep red denotes rejections. The reviewer row can be regarded as the ground-truth distribution. As expected, most rejections cluster in the lower-score region, yet some appear in higher ranges, reflecting the valuable role of expert preferences in calibrating the review process. NAIPv2 exhibits a trend similar to reviewer assessments, with lower-score regions containing more rejected papers, while the high-score regions contain fewer rejections than the reviewer-assigned scores. The API-based approach shows a weaker tendency, yet still outperforms stochastic prediction. Owing to inconsistencies, NAIPv1 and direct RTS regression perform poorly, as rejected papers are distributed almost randomly across score ranges. This case study shows that NAIPv2 achieves stable generalization and consistent alignment with conference outcomes, suggesting promising directions for broader applications.

## 6 Conclusion

This work presents NAIPv2, a debiased and scalable framework for automated paper quality estimation. By employing pairwise learning within domain-year groups and introducing a confidence-aware probabilistic signal, NAIPv2 mitigates score inconsistencies while enabling efficient linear-time inference. On the curated ICLR test set, the model achieves state-of-the-art performance, demonstrating both accuracy and efficiency compared to prior approaches. Moreover, in a cross-venue evaluation on NeurIPS, NAIPv2 predicts a clear monotonic trend from rejected to oral papers and accurately places less rejections in the high-score regions, demonstrating consistency with human judgments and robustness under distributional shifts. Taken together, these results highlight NAIPv2 as an exploratory step toward robust scalable scientific intelligence systems, while leaving open important avenues for future research on quality prediction and broader academic applications.

#### Acknowledgments

This research was supported by the Fund of the National Natural Science Foundation of China (Grant No.62576177, 62206134), the Fundamental Research Funds for the Central Universities 070-63233084, and the Tianjin Key Laboratory of Visual Computing and Intelligent Perception (VCIP). Computation is supported by the Supercomputing Center of Nankai University (NKSC). This work was supported by the National Science Fund of China under Grant No. 62361166670.

## References

- Ali et al. (2025)
Awais Ali, Muhammad Yousuf Jat Baloch, Muhammad Naveed, Anam Nigar, Abdulrahman Seraj Almalki, Ayesha Ghulam Rasool, Meseret Abeje Gedfew, and Ahmed A Arafat.
Advanced satellite-based remote sensing and data analytics for precision water resource management and agricultural optimization.
*Scientific Reports*, 15(1):27527, 2025.
- Baek et al. (2024)
Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang.
Researchagent: Iterative research idea generation over scientific literature with large language models.
*arXiv preprint arXiv:2404.07738*, 2024.
- Bar-Joseph et al. (2001)
Ziv Bar-Joseph, David K Gifford, and Tommi S Jaakkola.
Fast optimal leaf ordering for hierarchical clustering.
*Bioinformatics*, 17(suppl_1):S22–S29, 2001.
- Bharti et al. (2021)
Prabhat Kumar Bharti, Shashi Ranjan, Tirthankar Ghosal, Mayank Agrawal, and Asif Ekbal.
Peerassist: leveraging on paper-review interactions to predict peer review decisions.
In *International Conference on Asian Digital Libraries*, pp. 421–435. Springer, 2021.
- Cao et al. (2007)
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li.
Learning to rank: from pairwise approach to listwise approach.
In *Proceedings of the 24th international conference on Machine learning*, pp. 129–136, 2007.
- Chen et al. (2024a)
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu.
Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.
*arXiv preprint arXiv:2402.03216*, 2024a.
- Chen et al. (2024b)
Zhenyuan Chen, Lingfeng Yang, Shuo Chen, Zhaowei Chen, Jiajun Liang, and Xiang Li.
Revisiting prompt pretraining of vision-language models.
*arXiv preprint arXiv:2409.06166*, 2024b.
- Cortes & Lawrence (2021)
Corinna Cortes and Neil D Lawrence.
Inconsistency in conference peer review: Revisiting the 2014 neurips experiment.
*arXiv preprint arXiv:2109.09774*, 2021.
- D’Arcy et al. (2024)
Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey.
Marg: Multi-agent review generation for scientific papers.
*arXiv preprint arXiv:2401.04259*, 2024.
- de Winter (2024)
Joost de Winter.
Can chatgpt be used to predict citation counts, readership, and social media interaction? an exploration among 2222 scientific abstracts.
*Scientometrics*, 129(4):2469–2487, 2024.
- Dougherty & Horne (2022)
Michael R Dougherty and Zachary Horne.
Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences.
*Royal Society Open Science*, 9(8):220334, 2022.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv e-prints*, pp. arXiv–2407, 2024.
- F1000Research et al. (2012)
F1000Research, Iowa State University Library, Research Ireland, and University of Calgary.
San francisco declaration on research assessment (dora).
2012.
- Ghosal et al. (2019)
Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, and Pushpak Bhattacharyya.
Deepsentipeer: Harnessing sentiment in review texts to recommend peer review decisions.
In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pp. 1120–1130, 2019.
- Girshick (2015)
Ross Girshick.
Fast r-cnn.
In *Proceedings of the IEEE international conference on computer vision*, pp. 1440–1448, 2015.
- Hicks et al. (2015)
Diana Hicks, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols.
Bibliometrics: the leiden manifesto for research metrics.
*Nature*, 520(7548):429–431, 2015.
- Hinton et al. (2015)
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
Distilling the knowledge in a neural network.
*arXiv preprint arXiv:1503.02531*, 2015.
- Höpner et al. (2025)
Niklas Höpner, Leon Eshuijs, Dimitrios Alivanistos, Giacomo Zamprogno, and Ilaria Tiddi.
Automatic evaluation metrics for artificially generated scientific research.
*arXiv preprint arXiv:2503.05712*, 2025.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
*ICLR*, 1(2):3, 2022.
- Idahl & Ahmadi (2024)
Maximilian Idahl and Zahra Ahmadi.
Openreviewer: A specialized large language model for generating critical scientific paper reviews.
*arXiv preprint arXiv:2412.11948*, 2024.
- Kang et al. (2018)
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine Van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz.
A dataset of peer reviews (peerread): Collection, insights and nlp applications.
*arXiv preprint arXiv:1804.09635*, 2018.
- Kuckreja et al. (2024)
Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan.
Geochat: Grounded large vision-language model for remote sensing.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 27831–27840, 2024.
- Li et al. (2024a)
Yuxuan Li, Xiang Li, Yunheng Li, Yicheng Zhang, Yimian Dai, Qibin Hou, Ming-Ming Cheng, and Jian Yang.
Sm3det: A unified model for multi-modal remote sensing object detection.
*arXiv preprint arXiv:2412.20665*, 2024a.
- Li et al. (2025)
Yuxuan Li, Yicheng Zhang, Wenhao Tang, Yimian Dai, Ming-Ming Cheng, Xiang Li, and Jian Yang.
Visual instruction pretraining for domain-specific foundation models.
*arXiv preprint arXiv:2509.17562*, 2025.
- Li et al. (2023)
Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang.
Curriculum temperature for knowledge distillation.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, pp. 1504–1512, 2023.
- Li et al. (2024b)
Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang.
Promptkd: Unsupervised prompt distillation for vision-language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 26617–26626, 2024b.
- Li et al. (2024c)
Zheng Li, Yibing Song, Ming-Ming Cheng, Xiang Li, and Jian Yang.
Advancing textual prompt learning with anchored attributes.
*arXiv preprint arXiv:2412.09442*, 1, 2024c.
- Liu et al. (2009)
Tie-Yan Liu et al.
Learning to rank for information retrieval.
*Foundations and Trends® in Information Retrieval*, 3(3):225–331, 2009.
- Lu et al. (2024)
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha.
The ai scientist: Towards fully automated open-ended scientific discovery.
*arXiv preprint arXiv:2408.06292*, 2024.
- Maslej et al. (2025)
Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, et al.
Artificial intelligence index report 2025.
*arXiv preprint arXiv:2504.07139*, 2025.
- McInnes et al. (2017)
Leland McInnes, John Healy, Steve Astels, et al.
hdbscan: Hierarchical density based clustering.
*J. Open Source Softw.*, 2(11):205, 2017.
- Müllner (2011)
Daniel Müllner.
Modern hierarchical, agglomerative clustering algorithms.
*arXiv preprint arXiv:1109.2378*, 2011.
- Si et al. (2024)
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto.
Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.
*arXiv preprint arXiv:2409.04109*, 2024.
- Staudinger et al. (2024)
Moritz Staudinger, Wojciech Kusa, Florina Piroi, and Allan Hanbury.
An analysis of tasks and datasets in peer reviewing.
In *Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024)*, pp. 257–268, 2024.
- Thelwall (2025)
Mike Thelwall.
Research quality evaluation by ai in the era of large language models: advantages, disadvantages, and systemic effects–an opinion paper.
*Scientometrics*, pp. 1–13, 2025.
- Wang et al. (2024)
Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, et al.
Autosurvey: Large language models can automatically write surveys.
*Advances in neural information processing systems*, 37:115119–115145, 2024.
- Weng et al. (2025)
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang.
Cycleresearcher: Improving automated research via automated review.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=bjcsVLoHYs](https://openreview.net/forum?id=bjcsVLoHYs).
- Wu et al. (2024)
Ge Wu, Xin Zhang, Zheng Li, Zhaowei Chen, Jiajun Liang, Jian Yang, and Xiang Li.
Cascade prompt learning for vision-language model adaptation.
In *European Conference on Computer Vision*, pp. 304–321. Springer, 2024.
- Xia et al. (2023)
Wanjun Xia, Tianrui Li, and Chongshou Li.
A review of scientific impact prediction: tasks, features and methods.
*Scientometrics*, 128(1):543–585, 2023.
- Yu et al. (2024)
Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Renjing Cui, Chengcheng Han, Qiushi Sun, et al.
Automated peer reviewing in paper sea: Standardization, evaluation, and analysis.
*arXiv preprint arXiv:2407.12857*, 2024.
- Zhang et al. (2025a)
Daoze Zhang, Zhijian Bao, Sihang Du, Zhiyi Zhao, Kuangling Zhang, Dezheng Bao, and Yang Yang.
Re2: A consistency-ensured dataset for full-stage peer review and multi-turn rebuttal discussions.
*arXiv preprint arXiv:2505.07920*, 2025a.
- Zhang et al. (2025b)
Xin Zhang, Xue Yang, Yuxuan Li, Jian Yang, Ming-Ming Cheng, and Xiang Li.
Rsar: Restricted state angle resolver and rotated sar benchmark.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 7416–7426, 2025b.
- Zhang et al. (2025c)
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al.
Qwen3 embedding: Advancing text embedding and reranking through foundation models.
*arXiv preprint arXiv:2506.05176*, 2025c.
- Zhang et al. (2025d)
Yaohui Zhang, Haijing Zhang, Wenlong Ji, Tianyu Hua, Nick Haber, Hancheng Cao, and Weixin Liang.
From replication to redesign: Exploring pairwise comparisons for llm-based peer review.
*arXiv preprint arXiv:2506.11343*, 2025d.
- Zhao et al. (2024)
Penghai Zhao, Xin Zhang, Jiayue Cao, Ming-Ming Cheng, Jian Yang, and Xiang Li.
A literature review of literature reviews in pattern analysis and machine intelligence.
*arXiv preprint arXiv:2402.12928*, 2024.
- Zhao et al. (2025)
Penghai Zhao, Qinghua Xing, Kairan Dou, Jinyu Tian, Ying Tai, Jian Yang, Ming-Ming Cheng, and Xiang Li.
From words to worth: Newborn article impact prediction with llm.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pp. 1183–1191, 2025.
- Zhao & Feng (2022)
Qihang Zhao and Xiaodong Feng.
Utilizing citation network structure to predict paper citation counts: A deep learning approach.
*Journal of Informetrics*, 16(1):101235, 2022.
- Zhou et al. (2024)
Ruiyang Zhou, Lu Chen, and Kai Yu.
Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks.
In *Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (LREC-COLING 2024)*, pp. 9340–9351, 2024.
- Zhu et al. (2025)
Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang.
Deepreview: Improving llm-based paper review with human-like deep thinking process.
In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 29330–29355, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.1420.
URL [https://aclanthology.org/2025.acl-long.1420/](https://aclanthology.org/2025.acl-long.1420/).

## Appendix A Ethical Statement

The scores presented in this work are generated by machine learning models trained on historical peer-review data and should be understood as black-box predictions, not as objective or authoritative measures of scientific quality. These scores are not intended, nor should they be used, to replace human peer review or editorial decision-making. Our contribution is solely for research purposes, with the aim of studying peer-review dynamics and developing methods for literature intelligence systems. The authors bear no responsibility for any use of the predicted scores in real-world decision processes.

## Appendix B Reproducibility Statement

We fix the random seed to $42$ that controls all randomness in random, NumPy, and PyTorch, including CUDA backends.
This minimizes nondeterminism during training and evaluation.

## Appendix C Disclosure of LLM Usage

ChatGPT was employed during the experimental stage as an auxiliary tool for code debugging, code development, and data visualization. All code was thoroughly reviewed by the authors to ensure accuracy and reliability. ChatGPT also assisted in polishing the language of the manuscript, but was not involved in substantive writing. The authors take full responsibility for any issues arising from the use of ChatGPT.

## Appendix D Debiased Pairwise Learning Objectives

In this section we broaden our investigation beyond the main loss and systematically examine
alternative objectives for pairwise learning. The motivation is to understand the design space
of loss functions and the trade-offs they entail, rather than to commit to a single formulation.
Formally, given a pair $(a,b)$ with binary label $z_{ab}=\mathbb{I}[{RTS}a>{RTS}b]$, outputs
$y_{a},y_{b}$, and difference $d_{ab}=y_{a}-y_{b}$, we consider the following representative families.

##### (A) Likelihood-based objectives.

These methods interpret the pairwise preference as a probability that $a$ is better than $b$,
and optimize a proper scoring rule between predicted and observed preference.

*Thurstone–Probit.*
Replacing the sigmoid with a probit link yields

$$ $\hat{z}_{ab}^{\textsf{probit}}=\Phi\!\left(\tfrac{d_{ab}}{\tau}\right),\quad\mathcal{L}_{\textsf{Probit}}(a,b)=-\Big(z_{ab}\log\hat{z}_{ab}^{\textsf{probit}}+(1-z_{ab})\log(1-\hat{z}_{ab}^{\textsf{probit}})\Big),$ (9) $$

where $\Phi(\cdot)$ is the standard normal CDF and $\tau$ controls the slope.
This treats preference as arising from a latent Gaussian noise model, offering smoother gradients when comparisons are uncertain.

*Pairwise Brier.*
Instead of a log loss, we minimize squared error between predicted probability and label:

$$ $\mathcal{L}_{\textsf{Brier}}(a,b)=\big(\sigma(d_{ab})-z_{ab}\big)^{2}.$ (10) $$

This penalizes deviations quadratically and is less harsh than cross-entropy when predictions are confident but wrong.

##### (B) Margin- and regression-style objectives.

These objectives work directly on the score difference $d_{ab}$, either enforcing a margin
or softly aligning it with the review gap $\Delta_{ab}$.

*Hinge margin.*
With labels $y_{ab}=2z_{ab}-1\in\{-1,+1\}$, we require $d_{ab}$ to exceed a margin $m$:

$$ $\mathcal{L}_{\textsf{Hinge}}(a,b)=\big[m-y_{ab}\,d_{ab}\big]_{+},\quad[x]_{+}=\max(x,0).$ (11) $$

This loss is zero if the preferred item is already ahead by at least $m$,
and otherwise grows linearly—encouraging confident separation.

*Huber-on-difference.*
Here we compare the predicted difference to the observed review score gap $\Delta_{ab}$.
Residual $r_{ab}=d_{ab}-\kappa\,\Delta_{ab}$ is penalized with a robust Huber function:

$$ $\mathcal{L}_{\textsf{HuberDiff}}(a,b)=\begin{cases}\tfrac{1}{2}r_{ab}^{2},&|r_{ab}|\leq\delta,\[3.0pt] \delta\!\left(|r_{ab}|-\tfrac{\delta}{2}\right),&|r_{ab}|>\delta.\end{cases}$ (12) $$

This encourages alignment with the magnitude of human score gaps,
while limiting the influence of outliers beyond threshold $\delta$.

##### (C) Hybrid objectives.

These combine a directional likelihood with a light regression regularizer, aiming to
preserve ordering while stabilizing score magnitudes.

*Rank-then-Regress.*
We augment RankNet with an $L_{2}$ penalty on the difference–gap residual:

$$ $\mathcal{L}_{\textsf{RtR}}(a,b)=-\Big(z_{ab}\log\sigma(d_{ab})+(1-z_{ab})\log(1-\sigma(d_{ab}))\Big)+\lambda\,(d_{ab}-\kappa\,\Delta_{ab})^{2},$ (13) $$

where $\lambda$ controls the regression weight.
This formulation preserves pairwise preference consistency while encouraging smoother calibration within domain–year subgroups.

**Table 6: Comparison of different loss functions for pairwise learning.**
| Category | Loss | Acc | F1 | AUC | NDCG | $\bm{\rho}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Likelihood-based | BCE (adopted) | 0.706 | 0.609 | 0.782 | 0.771 | 0.432 |
|  | Thurstone–Probit | 0.689 | 0.594 | 0.756 | 0.762 | 0.393 |
|  | Pairwise Brier | 0.606 | 0.541 | 0.685 | 0.750 | 0.293 |
| Margin / Regression | Hinge Margin | 0.711 | 0.606 | 0.772 | 0.783 | 0.420 |
|  | Huber-on-difference | 0.531 | 0.548 | 0.699 | 0.758 | 0.288 |
| Hybrid | Rank-then-Regress | 0.681 | 0.591 | 0.765 | 0.789 | 0.396 |

Overall, results in Tab. [6](https://arxiv.org/html/2509.25179v2#A4.T6) support our choice of a purely pairwise likelihood objective as the most stable formulation. Importantly, once regression-style supervision is introduced, performance degrades across metrics, suggesting that absolute review scores suffer from scale inconsistencies across domains and years. This reinforces the view that enforcing numerical alignment is detrimental, whereas ordinal-only objectives provide more robust and transferable signals.

## Appendix E Data Efficiency Analysis

Figure: Figure 6: Effect of data size on model performance. (a) Performance comparison of NAIPv1 and NAIPv2 with different numbers of samples, measured by AUC and Spearman’s $\rho$. (b) Performance of NAIPv2 with different numbers of training pairs.
Refer to caption: x6.png

We analyze the effect of data size on NAIPv2 performance from two perspectives.
Figure [6](https://arxiv.org/html/2509.25179v2#A5.F6)(a) compares NAIPv1 and NAIPv2 under varying sample sizes. Both models benefit from more data, and NAIPv2 consistently outperforms NAIPv1. However, due to the scarcity of reviewed data, we could not scale sample size further. Figure [6](https://arxiv.org/html/2509.25179v2#A5.F6)(b) examines NAIPv2 trained with different numbers of pairs. The model shows clear improvements as the number of pairs increases up to about 10,000. Beyond this point, performance saturates, and additional pairs do not provide significant gains; instead, they may even degrade performance due to overfitting when the task difficulty becomes too low. These results suggest that while more data is generally beneficial, NAIPv2 requires only about 10,000 pairs to reach optimal performance, and further increasing the pair data size is unnecessary and potentially harmful.

## Appendix F Normalization in the Computation of RTS

In computing the RTS, we do not directly rely on the raw review scores; instead, both the scores and the confidence levels are normalized. The normalization of scores is straightforward and follows a standard min–max scaling scheme:

$$ $\tilde{s}_{i}\;=\;\frac{s_{i}-\min_{j}s_{j}}{\max_{j}s_{j}-\min_{j}s_{j}},$ (14) $$

where $s_{i}$ denotes the raw score given by reviewer $i$, and $\tilde{s}_{i}\in[0,1]$ is the normalized score.

For confidence levels, however, the choice of normalization strategy is less obvious. A central question is how the lowest reported confidence should be mapped on the normalized scale—e.g., whether it should correspond to 0.2, 0.5, 0.8, or other values. To allow for this flexibility, we define the normalization as:

$$ $\tilde{c}_{i}\;=\;\alpha\;+\;(1-\alpha)\,\frac{c_{i}-\min_{j}c_{j}}{\max_{j}c_{j}-\min_{j}c_{j}},$ (15) $$

where $c_{i}$ denotes the raw confidence reported by reviewer $i$, and $\tilde{c}_{i}\in[\alpha,1]$ is the normalized confidence. Here, $\alpha\in[0,1]$ is a tunable parameter that controls the mapping of the lowest confidence level to the normalized scale. To address this question, we conducted experiments to evaluate which normalization strategies for confidence yield the most effective results.

As shown in Tab. [7](https://arxiv.org/html/2509.25179v2#A6.T7), the smooth formulation of $\sigma(c)$ yields superior performance, indicating that gradual scaling of confidence is more effective than aggressive penalization. For the lower bound $\alpha$, the best results occur at $\alpha=0.2$, while both smaller and larger values reduce performance, showing that a moderate range is preferable.

**Table 7: Ablation study on RTS normalization strategies. (a) Different formulations of $\sigma(c)$. (b) Effect of the lower bound $\alpha$ for confidence normalization.**
| $\bm{\sigma(c)}$ | AUC | NDCG | $\bm{\rho}$ |
| --- | --- | --- | --- |
| $\sigma_{\text{aggressive}}$ | 0.766 | 0.775 | 0.391 |
| $\sigma_{\text{smooth}}$ | 0.782 | 0.771 | 0.432 |

## Appendix G Implementation of API-Only Baselines

To ensure fair comparison and to examine the limitations of prompt-only approaches, we re-implemented baseline experiments using GPT-5-mini as an AI reviewer.
In the pointwise setting, the model was provided with the title and abstract of a single paper and instructed to output a normalized quality score in the range $[0,1]$.
In the pairwise setting, two papers were presented simultaneously, and the model was required to judge which paper exhibited higher overall quality.
To further evaluate the robustness of prompt design, we constructed two distinct prompts for each setting, as reported in Tab. [8](https://arxiv.org/html/2509.25179v2#A7.T8).
We then evaluated the outputs using both ranking-based and classification-based metrics.
For classification, the F1 score was computed under a decision threshold corresponding to the top-30% predicted scores, following standard practice in acceptance prediction.
This setup allows us to quantify the effectiveness of prompt-only methods and highlight their inherent weaknesses when compared with our proposed debiased pairwise learning framework.

**Table 8: Performance comparison.**
| Category | Prompt | F1 | AUC | NDCG | $\bm{\rho}$ |
| --- | --- | --- | --- | --- | --- |
| Pointwise | “You are an expert academic paper reviewer.\nEvaluate the quality of a single paper based on title and abstract only.\n\nConsider only:\n- Novelty and significance\n- Clarity of writing\n- Technical soundness (as inferred from abstract)\n\nYou MUST output a STRICT JSON object ONLY with this exact schema:\n{"score": 0.00}\n\nThe score must be a normalized quality between 0 and 1 (inclusive),\nformatted with exactly two decimal places. Do NOT include any extra keys or text.” | 0.427 | 0.654 | 0.702 | 0.315 |
|  | “Review the paper from its title and abstract.\nJudge quality by originality, clarity, and soundness.\n\nReturn ONLY a JSON object:\n{"score": 0.00}\n\nScore $\in[0,1]$, exactly two decimals. No other text.” | 0.422 | 0.637 | 0.667 | 0.278 |
| Pairwise | “You are an expert academic paper reviewer.\nYour job is to decide which of two papers is of higher overall quality\nbased on title and abstract only.\n\nConsider only:\n- Novelty and significance\n- Clarity of writing\n- Technical soundness (as inferred from abstract)\n\nYou MUST choose exactly one better paper (no ties).\nOutput a STRICT JSON object ONLY, with this exact schema:\n{"better": a} OR {"better": b}\n\nDo NOT include any explanations, notes, or extra fields.\nDo NOT include reasoning.\nDo NOT include any text outside the JSON object.” | 0.448 | 0.655 | 0.686 | 0.297 |
|  | “Decide which paper (a or b) is of higher quality\nbased on title and abstract only. Consider novelty, clarity, and\ntechnical soundness. Pick exactly one.\n\nOutput STRICT JSON:\n{"better": a} OR {"better": b}\n\nNo extra text.” | 0.439 | 0.659 | 0.690 | 0.316 |

## Appendix H Bucket Distribution Definition

As noted in Section [4.7](https://arxiv.org/html/2509.25179v2#S4.SS7), we discretize the pairwise $RTS$ gap $\Delta_{ab}$ into buckets
with edges $[0,0.1,0.2,\dots,1.0]$, which correspond to 9 intervals. We define five representative ratio distributions over these buckets to characterize different difficulty settings,
ranging from Easiest to Hardest.
Table [9](https://arxiv.org/html/2509.25179v2#A8.T9) lists the exact proportions.
In particular, easier distributions allocate more weight to large gaps ($>0.8$),
whereas harder ones emphasize small gaps ($<0.1$).
The moderate case is close to uniform and serves as a neutral baseline.

**Table 9: Bucket ratio definitions for different difficulty levels.**
| Difficulty | $<0.1$ | 0.1–0.2 | 0.2–0.3 | 0.3–0.4 | 0.4–0.5 | 0.5–0.6 | 0.6–0.7 | 0.7–0.8 | $>0.8$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Easiest | 0.03 | 0.04 | 0.06 | 0.08 | 0.10 | 0.12 | 0.14 | 0.16 | 0.27 |
| Easier | 0.05 | 0.06 | 0.07 | 0.08 | 0.10 | 0.12 | 0.15 | 0.17 | 0.20 |
| Moderate | 0.111 | 0.111 | 0.111 | 0.111 | 0.111 | 0.111 | 0.111 | 0.111 | 0.112 |
| Harder | 0.20 | 0.17 | 0.15 | 0.13 | 0.10 | 0.08 | 0.07 | 0.06 | 0.04 |
| Hardest | 0.27 | 0.16 | 0.14 | 0.12 | 0.10 | 0.08 | 0.06 | 0.04 | 0.03 |

## Appendix I Data Engine for Constructing NAIDv2

To enable scalable data collection and management, we implemented a dedicated data engine for NAIDv2 (Tab. [10](https://arxiv.org/html/2509.25179v2#A9.T10)).
The engine adopts a three-layer architecture: the Controller layer interfaces with external APIs (e.g., OpenReview) and orchestrates data flow;
the Service layer parses raw submissions and reviews, normalizes metadata (such as confidence scores), and exports processed datasets;
and the DAO layer provides object-oriented access to the underlying MySQL database.

The database consists of two core tables, ‘Papers’ and ‘Reviews’, linked by the paper identifier.
The ‘Papers’ table stores submission-level metadata (title, abstract, venue, derived GT labels, etc.),
while the ‘Reviews’ table stores reviewer comments, ratings, and rebuttals.
Automatic crawlers retrieve all submissions from OpenReview, insert missing entries, and populate reviews through structured parsing.

This modular design ensures reproducibility, facilitates large-scale updates across multiple years of OpenReview data,
and supports downstream model training and evaluation.

**Table 10: Schema of the Papers and Reviews tables in NAIDv2 (partial view). PK = Primary Key, FK = Foreign Key.**
| Field | Type | Desc. |
| --- | --- | --- |
| id | BIGINT | PK |
| title | VARCHAR(255) | Title |
| abstract | TEXT | Abstract |
| external_id | VARCHAR(255) | Openreview ID |
| openreview_venue | VARCHAR(255) | Venue |
| scores | JSON | Review scores |
| confs | JSON | Confidences |
| pub_year | INT | Year |
| embedding | JSON | Embedding vec |
| cluster_cat | VARCHAR(64) | Cluster label |

## Appendix J Effect of Additional Information

As noted in NAIPv1 (Zhao et al., 2025), incorporating additional information can sometimes enhance model performance, though such gains are not always consistent. In this study, adhering to the double-blind requirement, we extracted auxiliary information directly from the papers and incorporated it into network training. Specifically, we employed a computer vision-based approach to extract key figures and key tables: the scores for the key figures were derived through network inference, and the table descriptions were generated using analysis by GPT-5-mini. However, our experiments show that adding such auxiliary content does not improve performance across the tested settings. In fact, models augmented with introductions, conclusions, figure scores, or table descriptions all perform worse than those relying solely on titles and abstracts. These results are broadly consistent with prior findings (Zhou et al., 2024), indicating that not all types of supplementary content provide useful signals for review score prediction, further confirming that abstracts already contain sufficient information.

**Table 11: Comparison of different model architectures for review score prediction.**
| Method | AUC | NDCG | $\bm{\rho}$ |
| --- | --- | --- | --- |
| Title & Abstract (ours) | 0.782 | 0.771 | 0.432 |
| +Introdution | 0.716 | 0.760 | 0.321 |
| +Conclusion | 0.750 | 0.768 | 0.395 |
| +Key Figure Score | 0.769 | 0.798 | 0.417 |
| +Key Table Description | 0.758 | 0.753 | 0.381 |

## Appendix K Additional Remarks on Experimental Comparisons

Figure: (a) NAIDv2 (Ours)
Refer to caption: figs/NAID_data_dist.png

In Tab. [1](https://arxiv.org/html/2509.25179v2#S4.T1), we compare the performance of existing methods, reporting some results directly from the original publications. We were unable to reproduce SEA-E (Yu et al., 2024), OpenReviewer (Idahl & Ahmadi, 2024), DeepReviewer Zhu et al. (2025), and CycleReviewer (Weng et al., 2025) for two primary reasons. First, the training and test sets used in these works differ and sometimes overlap, raising the possibility that our test data appear in their training sets (and vice versa). Second, retraining these models is computationally infeasible. For example, CycleReviewer requires training a 123B-parameter model, which far exceeds our available resources. Moreover, we note that instances from benchmark datasets may be embedded in the training data of these methods, suggesting that their reported gains could partly reflect data leakage rather than genuine improvements in generalization.

Nonetheless, most prior studies follow a relatively consistent dataset construction protocol, typically sampling randomly from the ICLR dataset. As shown in Fig. [7](https://arxiv.org/html/2509.25179v2#A11.F7), the distribution of our test set is well aligned with that of DeepReviewer, with a Jensen–Shannon of 0.079. This alignment indicates that, despite the aforementioned limitations, the comparisons in Tab. [1](https://arxiv.org/html/2509.25179v2#S4.T1) remain reasonably meaningful across methods.

## Appendix L Further Details on Embedding and Clustering

A key motivation for our embedding–clustering design arises from the observation that review scores can vary substantially across domains. However, directly applying keyword-based normalization often suffers from issues of uncontrollable granularity and drifting category boundaries, which weaken its reliability. For instance, research on remote sensing (Li et al., 2024a) may be regarded either as satellite-based sensing (Ali et al., 2025; Kuckreja et al., 2024) or as visual perception (Girshick, 2015; Li et al., 2025; Zhang et al., 2025b). “PromptKD” (Li et al., 2024b) might be situated within prompt learning (Li et al., 2024c; Wu et al., 2024; Chen et al., 2024b) or knowledge distillation (Hinton et al., 2015; Li et al., 2023). To address this challenge, we employ Qwen3-Embedding in combination with hierarchical clustering, which enables semantically grounded grouping of papers beyond brittle keyword matching.

As shown in Table [12](https://arxiv.org/html/2509.25179v2#A12.T12) and Table [13](https://arxiv.org/html/2509.25179v2#A12.T13), this setup allows us to systematically study the impact of embedding and clustering choices. Both BGE-M3 (Chen et al., 2024a) and Qwen3-Embedding provide strong baselines, yet Qwen3-Embedding paired with hierarchical clustering achieves the most consistent gains. HDBSCAN McInnes et al. (2017) remains competitive but trails slightly in AUC and $\rho$. Importantly, embeddings enriched with explicit clustering-oriented instructions outperform those without, and incorporating both titles and abstracts yields better representations than titles alone. Among the prompts we explored, the one explicitly directing the model to group semantically similar papers provides the most balanced improvements across all evaluation metrics.

**Table 12: Impact of embedding models and clustering algorithms on retrieval performance. (a) Comparison of embedding models. (b) Comparison of clustering algorithms with confidence normalization.**
| Model | AUC↑ | NDCG↑ | $\bm{\rho}$ |
| --- | --- | --- | --- |
| BGE-M3 | 0.774 | 0.769 | 0.417 |
| Qwen3-emb | 0.782 | 0.771 | 0.432 |

**Table 13: Comparison of different instruction prompt designs**
| Instruction | AUC↑ | NDCG↑ | $\bm{\rho}$ |
| --- | --- | --- | --- |
| None (no instruction) | 0.759 | 0.775 | 0.266 |
| Identify the main category of scholar papers based on the titles and abstracts | 0.767 | 0.831 | 0.416 |
| Given a title of a scientific paper, retrieve the titles of other relevant papers | 0.768 | 0.784 | 0.412 |
| Cluster similar papers as close as possible based on title and abstract (ours) | 0.782 | 0.771 | 0.432 |

## Appendix M Ablation on Pairwise Sampling Strategies

To further investigate the robustness of the proposed pairwise learning strategy, we conduct ablation studies by introducing additional constraints during pair construction. The first experiment (Table [14](https://arxiv.org/html/2509.25179v2#A13.T14) Panel (a)) examines the effect of limiting the maximum number of times a sample can appear in training pairs. The results show that imposing such restrictions does not provide clear benefits. In fact, the best overall performance is achieved when no limitation is enforced, suggesting that allowing samples to be paired freely provides greater diversity and stronger supervision signals for training.

The second experiment (Table [14](https://arxiv.org/html/2509.25179v2#A13.T14) Panel (b)) analyzes the role of enforcing a minimum score difference between paired samples. Without such a margin, performance degrades, indicating that constructing pairs from nearly indistinguishable items introduces noise into the training signal. A margin of 0.05 provides the best overall balance, improving both AUC and Spearman correlation, while larger margins ($\geq 0.1$) result in performance drops due to reduced pair availability. These findings highlight the importance of carefully controlling pair selection to enhance both the reliability and effectiveness of pairwise learning.

**Table 14: Effect of pairwise learning constraints. (a) Influence of limiting the maximum occurrence times of each sample in pairwise training. (b) Influence of enforcing a minimum score difference when constructing pairs.**
| Setting | AUC | NDCG | $\bm{\rho}$ |
| --- | --- | --- | --- |
| 2 | 0.776 | 0.773 | 0.426 |
| 4 | 0.766 | 0.791 | 0.408 |
| 8 | 0.767 | 0.774 | 0.391 |
| 16 | 0.782 | 0.771 | 0.432 |
| 32 | 0.782 | 0.771 | 0.432 |

## Appendix N Discussion and Practical Considerations

**Table 15: Performance of NAIPv1/v2 on article impact prediction task.**
| Model | NDCG | $\rho$ |
| --- | --- | --- |
| NAIPv1 | 0.901 | 0.459 |
| NAIPv2 | 0.761 | 0.279 |

As shown in Tab. [15](https://arxiv.org/html/2509.25179v2#A14.T15), NAIPv1 performs strongly in predicting the academic impact of newly published articles, whereas NAIPv2 underperforms on the NAIDv1 benchmark. This result appears consistent with the observation of  Cortes & Lawrence (2021) that impact and quality are often only weakly correlated.

These findings suggest that there is no single “universal key” for all tasks, and the choice of method should depend on the specific application. API-based solutions are easy to use and broadly accessible, but their performance is limited (Tab. [1](https://arxiv.org/html/2509.25179v2#S4.T1)); fine-tuned autoregressive models benefit from domain-specific optimization and achieve stronger results, though at the cost of additional deployment requirements; NAIPv2 provides performance comparable to these autoregressive approaches with much faster inference, but lacks the interpretive outputs that autoregressive models offer.

Overall, these comparisons indicate that no single method prevails across all scenarios, underscoring the importance for both research and industry communities to recognize and leverage the complementary strengths of these approaches.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: What are some good application scenarios for NAIPv2?
soundness: 2
strengths: 1. This paper tackles an important problem to collect a massive dataset of paper review and propose a data-driven solution to automate the paper review process.
2. This paper introduces several methods to make its framework efficient and practical such as pairwise learning, pointwise inference; Review Tendency Signal etc.
3. The performance and generalizability of the framework appears to be good.
summary: This paper constructs a dataset (NAIDv2) along with a framework (NAIPv2) for automated paper quality estimation. In particular, the framework adopts pairwise learning (comparing two papers at a time) but is deployed as an efficient pointwise predictor (judging a single paper's quality). The framework also uses a gaussian model, named Review Tendency Signal, to aggregate multiple reviews into a single ground-truth quality signal. Tested on unseen NeurIPS submissions, NAIPv2 is able to achieve better performance over the previous NAIPv1 framework.
weaknesses: 1. The relationship between NAIDv2 and NAIPv2 is confusing due to the naming. It appears that NAIDv2 is just the dataset used for learning the NAIPv2 framework.
2. The paper introduces models such as Review Tendency Signal, but it is unclear how good this probabilistic model is, compared to other alternatives, e.g., the review score might follow a Bradley–Terry model.  
3. The experiment results on the neurips dataset is better than the previous method, but it is still unclear how it is useful for practical applications, e.g., can it guide the AC decision in assessing the paper quality in addition to human reviewers.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: Related Work and Fairness of Discussion
(124)
The criticism of Zhang et al. (2025a) on the Re² dataset seems unfair and unsubstantiated. The statement “deviations between reported results and those presented in the original papers” needs concrete evidence or examples. If the concern is data leakage or overlap, that should be clearly documented and cited. Otherwise, this reads as an unsupported claim, can the authors clarify?


(154)
The summary of Höpner et al. (2025) lacks essential details. The claim that pairwise ranking is “unreliable” for review scores is unconvincing.


If regression on review scores is feasible, pairwise regression should also be; at worst it merely doubles the noise variance of independent samples.
The critical question is how pairs are selected and what is predicted (rank order, score difference, or ordinal classification). It seems to me that pairwise comparisons are most interesting for ranking, rather than cardinal scores, as you have the additional benefit of simplifying the problem domain to a binary prediction. In cardinal (regression) settings, all that's happening is canceling out translational offsets, which can be interesting when you're controlling for specific effects, but seems unimportant for random pairings. It's not clear from the descriptions in related work which of these methods are being employed, and this seems extremely important. Pairwise methods aren't monolithic: the advantage or lack thereof in using them comes from how pears are chosen and what function of the pair one tries to predict, and I would like to see this explored or discussed.

Broader Modeling Questions:

In automated paper review methods discussed, it seems like there are at least two axes of comparison here. 

1: The methods used to make predictions, and 2: what is being predicted. Why not treat citation counts, reviewers scores, and pairwise predictions as separate target, try to predict them all with one model, and select some combination of them for the actual quality estimate? Moreover, these various targets don't seem incompatible. Some concepts are better predictors of quality than others, but likely in complementary ways. It seems a joint predictive model would benefit from shared structure, and this multimodal learning would also make better use of available data.

It would be valuable to separate and jointly analyze multiple predictive targets (e.g., citation counts, reviewer scores, and pairwise preferences) within a unified model. These signals are not incompatible and could benefit from shared structure via multitask learning.



Bias:
It seems to me that, under a Bradley Terry or Thurstone-Mosteller model, you are controlling for bias by subtracting out factors. You subtract out year-dependent and domain dependent factors with this strategy. 

The “de-biasing” strategy seems roughly equivalent to including additive terms for each year-domain combination in a regression model, since you're using pairwise comparisons to control for these effects. It would help to compare these approaches empirically or at least discuss their equivalence.

In the same vein, why not restrict pairwise comparisons to papers reviewed by the same reviewers? Although this reduces the number of pairs, it would directly control for reviewer-level bias and may yield cleaner comparisons. It also may alleviate or reduce the need to normalize for each reviewer.
soundness: 2
strengths: This is an interesting and ambitious paper tackling a key problem in automated peer review. The dataset and efficiency contributions are also substantial.
Equation (1) is well-motivated and welcome to see; it provides a principled probabilistic link between confidence and uncertainty rather than relying on ad hoc weighting.


The proposal to integrate reviewer confidence directly into the score aggregation model addresses a genuine issue in automated peer review and quality estimation.


The focus on efficiency and domain-year stratified training is well thought out and practically motivated.
summary: This paper proposes NAIPv2, a framework for estimating scientific paper quality by combining pairwise learning with a probabilistic Review Tendency Signal (RTS) that integrates reviewer scores and confidences. The authors introduce a new large-scale dataset (NAIDv2) derived from ICLR submissions and report strong results in both within-domain and cross-domain evaluation. The central claim is that NAIPv2 provides debiased, scalable, and efficient quality estimation compared to prior LLM-based or regression-based approaches.
The paper is clearly written and tackles an important and timely problem in automated peer review and scientometric prediction. However, the theoretical framing and probabilistic modeling choices raise several concerns, especially regarding the internal consistency of the modeling assumptions and the clarity of related work.
weaknesses: (Main critique)
You introduce and motivate a Gaussian probabilistic model (RTS) but then discard it during pairwise ranking. When training with a logistic loss, you implicitly assume Gumbel noise, not Gaussian, effectively switching from a Thurstone–Mosteller to a Bradley–Terry model.


This inconsistency weakens the theoretical story: RTS models uncertainty via Gaussian variance, yet the pairwise comparisons ignore that structure and the confidence information is lost.


For example, if paper A’s reviewers are all high-confidence and paper B’s are all low-confidence, this distinction disappears once RTS is computed. A unified probabilistic model that preserves confidence across comparisons would be both cleaner and more principled.

For the avoidance of doubt, the model to which I refer is essentially a Thurstone-Mosteller model, where we take the probability of one or more reviewers scoring paper A over paper B, given the observed scores we have from reviewers for these papers, using your theoretical model of latent-factor based reviewer-paper scoring, and confidence values as additional inputs.

See e.g., 
[Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignment
Cyrus Cousins, Vijay Keswani, Vincent Conitzer, Hoda Heidari, Jana Schaich Borg, Walter Sinnott-Armstrong] for discussion of probabilistic pairwise prediction models in the broader ML context.

The Thurstone-probit of Appendix D approaches this idea, but variance information has already been removed by this point. I’d like to see this further explored.

Conceptual and Modeling Issues
(186–213)
Many technical aspects are vague in their description, hampering understanding and readability. This is likely a writing quality issue, rather than a criticism of the work behind it.
The introduction of the Review Tendency Signal (RTS) as a posterior mean implicitly assumes a Bayesian framework, but the paper never defines a prior or explains how the posterior is derived. As written, the likelihood $p(s_i \mid x, c_i)$ is defined, but this does not imply a likelihood on $x$. Please clarify whether a uniform prior is assumed, and present the closed-form expression for the posterior mean.


Given your confidence mapping functions, the posterior mean likely admits a clean closed form, which would help motivate the simplicity of your model.


(Appendix F; σ(cᵢ))
The mapping from reviewer confidence to variance is extremely limited. Although you normalize per reviewer, a learned nonlinear function over the entire reviewer population would be both straightforward and far more flexible.


(269)
Clarify that the additive offset correction may depend on year and domain; a global offset is only justified if all pairwise comparisons form a connected component.


(242, 682–696)
The term sigmoid is too vague — you almost certainly mean the logistic function. This matters because later in the appendix you replace it with a Gaussian CDF, which is also a sigmoid; the text should explicitly reference this distinction.


The model should be described more clearly in the language of Bradley–Terry and Thurstone–Mosteller frameworks, which would provide a unifying probabilistic interpretation and clarify the noise assumptions.


The appendix (lines 682–696) is interesting but should be integrated into the main text. The theoretical implications of alternative link functions (logistic vs. Gaussian CDF) should be discussed explicitly, as should the choice of Brier loss. A connection to strictly proper scoring rules would strengthen this section.


Terminology and Framing
The term scale inconsistency appears repeatedly but seems misapplied. The issue appears to be translational inconsistency (offsets), not multiplicative scale differences. The current phrasing is consistent but confusing.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - Regarding Section 3.2, it’s not very intuitive to me whether NAIDv2 really handles domain bias. NAIDv2 only contains ICLR submissions, which seems to be a relatively homogenous set given that ICLR is a ML conference. Are the authors suggesting that two papers of similar content and quality could have significantly different scores just because the authors chose different submission categories?
soundness: 4
strengths: - Clear benefits of using confidence information and RTS over raw scores, as seen in Table 4
- Computational efficiency of NAIPv2 with pointwise inference
- NAIDv2, a collection of reviews and confidence scores from ICLR, appears to be thoroughly constructed.
- Performance gains over more computationally expensive API and autoregressive baselines
- Extensive evaluation of NAIPv2/NAIDv2, including interesting demonstration of model generalizability based on NeurIPS decision category hierarachy
summary: This paper introduces NAIPv2, a fast regression model for paper review score estimation. The key idea behind this model is Review Tendency Signal (RTS), which uses not only the raw scores of each paper review but also the specified confidence level to model uncertainty behind individual scores. Alongside the model, the authors also introduce NAIDv2, a collection of ICLR review data compiled from OpenReview.
weaknesses: - It looks like the paper does not exactly mention how regression head was implemented on top of LLaMA 3 for NAIPv2. I didn’t have time to check out NAIPv1 paper so I’m not sure if the architecture was discussed there, but still I think it would be nice to have the architecture description somewhere in the paper.
- It would have been nice if NAIPv2 was also tested on the LLMs other than LLaMA 3 to see if the benefits of NAIPv2 still holds.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Line 247: 248: One of the claims in the paper is that reviewing style varies by domain and year, and clustering is used to create domain buckets. Can you show that review scores indeed follow a different distribution in each cluster-year group? Maybe a plot with cluster-year on the x-axis and some statistic quantifying the distribution on y-axis? 

2. Line 213-214 -- “pipeline produces domain-normalized labels”. How is RTS normalised using the domain information?  In eqn 14 and 15, what is j iterating over? Scores of the same paper, or all the scores in the dataset?

3. Given that papers from the same year are used to select the pairs during training, does it mean that we can only compare scores of the papers in the same domain and year? Can we use it to meaningfully compare papers across years?

4. Line 314: 316: Could you please explain the upper bound? Is the input to the MLP just a scalar or a vector of all the review scores for a paper? What if a paper has 3 reviews and another has 4 reviews? 

5. To compute F1 and accuracy, you need to select a threshold for the score based on validation data. Is the threshold selection dependent on domain / year?

6. Table 4 / Section 4.6: When using a different statistic (e.g., mean), I hope that it is used both for training and while testing and reporting the numbers in Table 4 - i.e. please clarify that you are not training using mean but then using  RTS to compute ground-truth in pairwise accuracy and ground-truth rank in correlation. 

7. Section 4.7: In Figure 4, it is not clear what you are comparing.  You have pairwise difficulty on the x-axis. For a given bucket, say the “easiest” bucket, are you reporting the performance obtained when you train your model using only the “easiest” pairs?  If yes, then how come it matches exactly with the performance in  ‘w/ Curriculum’ setting in Table 5 where you are using all pairs but in a curriculum.



**Comments:**

1. Given that the paper claims RTS as one of its contributions, its details should be moved from the appendix to the main paper.

2. Table 1: Isn’t a higher correlation better? There is no upward arrow sign beside it.
soundness: 3
strengths: 1. The paper is clearly written, with most of the details present either in the main paper or the appendix.

2. The paper has thorough ablations to justify all the design choices.

3. The paper releases a new dataset that may help the research community.
summary: As the number of submissions to peer-reviewed conferences increases exponentially each year, systems that automatically assign a review score to submissions may become increasingly important. This paper proposes to train an LLM to assign a score to a research article solely based on its title and abstract. 
It first introduces a new dataset, NAIPD2, curated using  ICLR submissions from 2021 to 2025. 
To account for reviewer confidence, it proposes a confidence-normalised target score, RTS, which is used as the ground-truth label for each paper. The paper claims that reviewing style (and hence review scores) varies by the paper’s domain and year. To account for this bias, instead of directly regressing on the RTS score, they propose a pairwise loss computed using a siamese-style network over the pairs selected from the same domain and year. The network is trained via BCE loss to predict which paper in a training pair has a higher RTS score.  To create domain buckets, instead of using the given keywords, the paper proposes to use hierarchical clustering of the paper’s title and abstract embeddings. To ensure that the LLMs being trained haven’t seen the papers/reviews before, it proposes to use only 2025 submissions in the test set.
Thorough ablations justify the design choices.
weaknesses: 1. While I understand that regression-based methods are obviously much more efficient, the lack of any reasoning provided by the model significantly hampers their applicability. On the other hand, methods like DeepReview provided a detailed output that can assist in actually reviewing a paper and perform as good as the proposed method. 


2. Another cited contribution is release of Naidv2 dataset. However, similar datasets ( and recipes to create them) already exist, e.g. the dataset used in DeepReview.

3. Another issue is a fair comparison with other baselines. All the reported experiments are ablations for the design choices made by the paper, e.g. pairwise loss vs pointwise loss, curriculum vs no curriculum etc., I don’t see an apples-to-apples comparison with other trained auto-regressive models that may be more inefficient but more accurate and useful.  

4. A simple baseline would be an LLM finetuned to generate the review along with the score. This way, all the reviews from the paper can be considered as a separate training sample. 

5. Line 322:323: Table 1 reports numbers directly from DeepReview and CycleReview papers, which are computed on different test data (though from ICLR 2025 only). Typically, numbers reported in the same table should be computed on exactly the same dataset.

6. OpenReviewer is mentioned in the related works, but not used as a baseline. 

7. The designed system takes only the title and abstract into account, whereas human reviewers would seldom base their decision solely on these two attributes. Does it mean that the system is capturing spurious correlations? 

8. While the application of a pairwise loss function to the problem at hand may be novel, the loss function used is the standard one employed to train reward models under Bradley-Terry (BT) reward model assumptions.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: * Many studies in this area use citation count as a proxy for paper quality because (1) their methods are unsupervised, and (2) human reviews are highly noisy, making it hard to align with them. I suggest adding experiments that use citation-based proxies and comparing them with existing methods, although they are not reliable. Such results could reveal deeper insights: if NAIPv2 performs worse on predicting citations but aligns much better with human reviews, it may suggest that human judgments rely heavily on information from titles and abstracts.
* This is more of a curiosity than a critique: since judging paper quality solely from titles and abstracts is unrealistic, I believe that NAIPv2’s strong alignment with human reviewers may come from certain “cheap signals” in abstract / title. I wonder what those signals might be; perhaps the authors could offer some hypotheses or discussion.
soundness: 3
strengths: * The paper proposes a paper quality predictor NAIPv2 that achieves state-of-the-art performance using only titles and abstracts, showing a high rank correlation with human reviews.
* NAIPv2 is efficient (linear time), generalizes well, and does not require full text, which makes it practically useful. I look forward to seeing downstream applications, such as helping authors improve their titles and abstracts.
* The paper combines existing tools into a clear and effective workflow. Although it uses Llama 3 as a backbone, given that paper quality estimation is still a new topic, I find the overall novelty acceptable.
* The experiments are well done. Some additional comments are listed in the “Questions” section.
summary: LLM-assisted peer review has become a popular topic in recent years, and Paper Quality Estimation is an important part of it. This paper proposes NAIPv2, a model that takes the titles and abstracts of two papers as input and predicts their pairwise quality. The model is based on the Llama 3 backbone, trained on the ICLR dataset with human reviewer’s feedback as ground truth. NAIPv2 achieves state-of-the-art performance (AUC and rank correlation) than several baselines, including autoregressive models. In addition, the method runs in linear time, generalizes well, and works when only abstracts are available instead of full papers.
weaknesses: * I would like to see experiments using more proxies for paper quality (see “Questions” section).
* Section 3.3 is a bit confusing. the training details could be explained more clearly in the main text.


Decision: Accept (Poster)