NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimation
Authors: Penghai Zhao
Jinyu Tian
Qinghua Xing
Xin Zhang
Zheng Li
Jianjun Qian
Ming-Ming Cheng
Xiang Li
Venue: ICLR 2026 Poster (2026)
Keywords: AI for Research
Article Quality Estimation
Literature Intelligence Systems
Automated Peer Review
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: NAIPv2 leverages pairwise learning on debiased data to predict article quality in a pointwise manner, achieving SoTA results and generalizing consistently with human decisions on unseen NeurIPS data.

Abstract: The ability to estimate the quality of scientific papers is central to how both humans and AI systems will advance scientific knowledge in the future. However, existing LLM-based estimation methods suffer from high inference cost, whereas the faster direct score regression approach is limited by scale inconsistencies. We present NAIPv2, a debiased and efficient framework for paper quality estimation. NAIPv2 employs pairwise learning within domain-year groups to reduce inconsistencies in reviewer ratings and introduces the Review Tendency Signal (RTS) as a probabilistic integration of reviewer scores and confidences. To support training and evaluation, we further construct NAIDv2, a large-scale dataset of 24,276 ICLR submissions enriched with metadata and detailed structured content. Trained on pairwise comparisons but enabling efficient pointwise prediction at deployment, NAIPv2 achieves state-of-the-art performance (78.2\% AUC, 0.432 Spearman), while maintaining scalable, linear-time efficiency at inference. Notably, on unseen NeurIPS submissions, it further demonstrates strong generalization, with predicted scores increasing consistently across decision categories from Rejected to Oral. These findings establish NAIPv2 as a debiased and scalable framework for automated paper quality estimation, marking a step toward future scientific intelligence systems.

Published as a conference paper at ICLR 2026
NAIPV2: DEBIASED PAIRWISE LEARNING FOR EFFICIENT PAPER QUALITY ESTIMATION
Penghai Zhao Jinyu 2i.52.1.3.1, Qinghua Xing2.1.311 Xin Zhang Zheng Li
Jianjun Qian", Ming-Ming Cheng Xiang Li2.2,3,
ENKIARI, Shenzhen Futian
VCII, CS, Nankai University
3 AAIS, Nankai University
PCCALab, School of Computer Science and Engineering,
Nanjing University of Science and Technology
2 (zhaopenghai, ttaihai,,iinnn tianj inyu, xingqinghua, zhasion, zhengli97, cmm, xi xiang. 1i. imi.implii Lus fênankai. edu.cn
4 ccs jngrjssnnsst.... edu. on
| Equal contribution
"Corresponding author
ABSTRACT
The ability estimate the quality of scientific papers is central to how both hu- mans and Al systems will advance scientific knowledge in the future. However, existing LLMM-bsee estimation methods suffer from high inference cost, whereas the faster direct score regression approach is limited by translational inconsisten- cies. We present NAIPv2, debiased and efficient framework for paper quality estimation. NAIPv2 employs pairwise learning within domain-year groups reduce inconsistencies in reviewer ratings and introduces the Review Tendency Signal (RTS) probabilistic integration of reviewer scores and confidences. To support training and evaluation, we further construct NAIDy2, a large-scale dataset of 24,276 ICLR submissions enriched with metadata and detailed struc- tured content. Trained on pairwise comparisons but enabling efficient pointwise prediction at deployment, NAIPv2 achieves state-of-the-ard performance (78.2% AUC, 0.432 Spearman), while maintaining scalable, linear-time efficiency at in- ference. Notably, on unseen NeurIPS submissions, further demonstrates strong generalization, with predicted scores increasing consistently across decision cat- egories from Rejected to Oral. These findings establish NAIPy2 as a debi- ased and scalable framework for automated paper quality estimation, marking a step toward future scientific intelligence systems. Project page is available at https://sway cloud.
INTRODUCTION
The rapid progress of large language models (LLMs) has enabled a growing ecosystem of AI-for- Science systems, such as autonomous research agents (Lu al. 2024 Liu
2025 Back al.
2024). automated survey-generee generrennn pipelines (Wang et al.) 2024b] Yan et al., 2025 Liang et al.,
2025 and emerging peeraatur-iitelligennee tools (He al. [225]] Chen al. 2025 Li et al. (0256), etc. A central bottleneck shared by these systems is the need to quickly identify high-quality and early-stage research from the continuous stream of newborn papers. However, such papers lack ci- tation histories, reliable bibliometric signals, and meaningful venue-based indicators. Furthermore, both DoRA (Fs00Oeesearrh et al. (0122) and the Leiden Manifesto (Hicks et al.) 2015)) caution against using prestige-baat based metrics proxies for scientific quality. Consequently, growing atten- tion directed toward approaches that evaluate research quality based on content-related features rather than external attributes.
Recent work has explored predicting citation-based indicators directly from textual content (Zhao al. 2025 de Winter 2024). While these approaches provide useful signals for modeling aspects

---

Published as a conference paper at ICLR 2026
of research impact, they are generally oriented toward predicting future citation behavior rather than
assessing a paper's s intrinsic quality. Prior analyses (Cortes Lawrence 2021 Dougherty Hore, 2022) have noted that citation counts may not rehably track scientific meni. cautioning against their use as a sole indicator of quality. These considerations motivate the development of quality estimation methods that rely more directly intrinsic, content-grounded signals.
For decades, expert peer review has served as a reliable standard for evaluating research's 's intrinsic quality. While rigorous, this manual process is costly, time-consuming and inherently difficult to scale. In response, recent studies (Yu et al. 2024 Zhu et al. 2025 D' Arcy et al. 2024; Idahl & Ah- madi 2024) have exploited LLMs to assist in the review process (e.g., generating review comments, predicting review scores, or inferring acceptance decisions directly from paper content). As shown Fig.[i lig- panel, although these approaches yield more interpretable textual explanations, their re- liance on autoregressive reasoning over long contexts incurs substantial latency and computational overhead. certain settings, the absence of accessible PDFs renders them entirely inoperative, thereby constraining their applicability in many academic search and recommendation systems.
AutoRegressive
NAIP₂1
NAIPv2
Tokenize P.E.
Tokenize P.E..
Tokenize P.E.
Tokenize P.E.
LLM
LLM
LLM
LLM
%
overall score is
La LiBCE g (R m)
Inference
Inference
Training
Inference
Figure 1: Comparison of various frameworks. AutoRegressive approaches (Zhu et al. 2025 Weng
et al. 2025) rely on sequential generation, resulting in substantial inference latency; NAIPv1 (Zhao ct al. 2025) enables fast pointwise regression but suffers from translational inconsistency; NAIPv2 leverages debiased pairwise training with confidence-awarc signals and operates pointwise regressor with linear-time complexity at inference.
efficient
To reduce the inference latency inherent in autoregressive approaches, we revisit regression-based methods, i.c., NAIPv1, which is shown in the middle panel of Fig. [ Our experiments, however, indicate that directly regressing on reviewer scores provides limited effecctiveness. We identify two underlying challenges. First, reviewer confidence tends be systematically overlooked. Although reviewers vary in their expertise and confidence when evaluating a submission, existing formulations typically treat all scores as equally informative, which reduces the robustness of the supervision sig- nal. Second, review scores lack a unified and stable evaluation standard. Scoring criteria differ across research domains, publication years, and even individual reviewers, introducing inconsisten- cies that pose challenges for calibrating direct regression.
Building on these observations, we propose NAIPv2 (originating from Newborn Article Impact Prediction), a fast and debiased framework for paper quality assessment. NAIPv2 learns the relative ordering of submissions within the same domain-yeat group in a pairwise manner, thereby miti- gating biases introduced by cross-domain, temporal variations. During inference, the framework decoupled into a simplified single-branch pointwise regressor, enabling robust score prediction while preserving finear-ime complexity. In addition, we introduce probabilistic treatment of re- view scores and reviewer confidences, where each score is viewed as noisy observation of the latent paper quality and confidence determines its uncertainty. yielding principled comparative signal that guides pairwise ranking. Experimental results demonstrate that our method attains state- of-the-art performance (78.2% AUC and 0..42 Spearman) while offering substantially improved
2

---

Published as a conference paper at ICLR 2026
inference efficiency. Notably, on unseen NeurIPS submissions, the predicted scores align precisely with decision categories, consistently assigning the lowest values to rejected papers and the highest values to oral presentations.
In summary, our main contributions are as follows:
1. A large-scale domain-labeled dataset. We release NAIDv2, dataset of 24,276 ICLR submissions (2021-2025) with parsed PDF content and clustered domain labels, enabling large-scale cross-domain debiasing and research on scholarly evaluation.
2. A scalable and debiased quality estimation framework. We propose NAIP^2, which combines pairwise training with efficient pointwise inference for early-stage paper qual- ity prediction, achieving state-of-the-art performance and strong generalization on unseen NeurIPS review data.
3. A confidence-aware probabilistic signal. We introduce RTS, a confidence-aware proba- bilistic formulation that improves the stability of quality estimation.
2 RELATED WORK
Peer Review Dataset. The blind nature of the peer-review process makes publicly accessible review data extremely limited. Among major conferences, ICLR is particularly notable for its open policy, providing comprehensive review information such as reviewer comments, rating scores, and area- chair decisions, and thus has been used by most existing studies (Staudinger et al. (024). However, reliance on ICLR also carries substantial risks of information leakage. A common and almost in- evitable issue the inadvertent overlap between training and test sets, where the training data of one method may appear in another method's test set (Zhang et al. 2025a), motivating many works to construct their datasets from scratch. Another source of leakage arises from large language models; for instance, PeerRead (Kang et al., 2018), released in 2018, has likely been incorporated into LLM training corpora, raising concerns about unintended memorization. The training, validation, and test sets of NAIDv2 are constructed following rigorous academic practice. Building
random split, we ensure that the test set exclusively contains the latest ICLR 2025 submissions, thereby preventing any potential information leakage. In addition, compared with existing datasets, NAIDv2 offers distinctive advantage by providing multi- multi-granular topic labels along with multimodal PDF-parsed representations.
Automated Peer Review. Automated pecr-review systems aim to approximate the human review- ing process by taking the full manuscript as input and producing corresponding review reports. To optimize performance on review-related tasks, most automated peer-review methods perform task-specific fine-tuning. YYu et al.] (2024) introduces a modular framework for standardization, evaluation, and analysis, and further proposes mismatch score to measure paper-review align- ment. OpenReviewer (Tdahl & Ahmadi (024) fine-tunes an 8B-parameter LLM on expert reviews to produce structured, guideline-comee compliant feedback. DeepReview (Zhu et al. 2025) employs multi-stage pipeline integrating structured analysis, literature retrieval, and evidence-basse reason- ing. CycleReviewer WWng al. (2025) goes beyond single-pass reviewing by coupling manuscript generation with iterative peer review through reinforcement learning, aiming to automate the entire research-review cycle. Together, these approaches exploit LLMs' CoT reasoning, boosting perfor- mance but incurring substantial computational cost. In contrast to automated peer-review systems, NAIPv2 provides numerical estimates of paper quality for a wide range of Al-for-science applica- tions, enabling fast ranking of candidate papers even when only the title and abstract are available.
Numerical Quality Estimation. Predicting the quality of scientific papers has long been an ac- tive research area (Zhao & Feng] [2022, Xia et al., 2023) Thelwall, 2025). Traditional approaches typically rely on early impact signals, such as citation counts, author profiles, venue prestige, and linguistic indicators extracted from the manuscript. More recently. researchers have begun to ex- plore LLM-based methods. For example, De et al. (de Winter 2024) examined whether Chat- GPT can estimate bibliometric indicators (e.g., citation counts), and found that its estimates cor- relate more strongly with ground-trute -trth impact metrics than traditional readability-basee measures. NAIP (Zhao et al. (025) further used LLMs to directly predict a field- and time-normalized im- pact score (TNCSI,, from titles and abstracts, achieving an MAE of 0.216. However, prior work (Cortes & Lawrence, 2021 Dougherty & Horne 2022) has shown that reviewer-annotated
3

---

Published as a conference paper at ICLR 2026
paper quality exhibits little correlation with citation- impact. Although several studies have attempted to predict metrics derived from peer review (Ghosal et al. 2019 Bharti et al. (021), they have neither adequately addressed translational inconsistency nor systematically examined the role of reviewer confidence, instead largely overlooking it. contrast, NAIPv2 learns from the proba- bilistic aggregation signal RTS and adopts pairwise learning framework to mitigate translational inconsistency, achieving competitive performance with substantially improved speed.
Pairwise ranking methods have long been established effective approaches in information retrieval and recommendation (Liu et al. 2009; Cao et al., 2007), yet their adoption peer review and quality estimation remains limited. Cousins et al.J (2025) show that classical pairwise models like Bradley-Terry (Bradley & Terry (1522) and Thurstone Monteelllll (Thurstone) 1994) can be derived from basic axioms of human comparative judgment. Zhang et al. (2025e) investigate pairwise manuscript evaluation using off-the-shelf LLMs. Owing to the quadratic growth of pairwise comparisons, they demonstrate that aggregating fewer than 2% of all possible pairs is sufficient to recover robust global ranking. Distinct from prior studies, this work addresses debiasing and efficiency by training in a pairwise manner while performing pointwise inference, thereby achieving inference with linear complexity.
Pairwise Ranking.
3 METHODS
3.1 REVIEW TENDENCY SIGNAL (RTS)
In practice, reviewers not always domain experts in the exact area of submission, which may prevent them from providing accurate evaluations. To mitigate this source of uncertainty, most con- ferences and journals require reviewers report not only an overall score but also accompanying confidence level, reflecting the degree of trust they place in their own assessment.
Our proposed solution is to treat each review not as a fixed score but as a confidence-weightee estimate of the latent "true quality" of the paper. In this view, the reported overall score s, reflects a reviewer's 's central judgment, while the confidence level c; controls how reliable that judgment is. High-confidence reviews contribute more sharply to the aggregated signal, whereas low-confidence reviews contribute more diffusely, reflecting greater uncertainty.
Formally, we aggregate multiple reviewer scores by treating each score as confidence-modulltee estimate of the latent true quality æ [0, 1]. Rather than relying on fixed weighting heuristics, we employ smooth Gaussian-shaped scheme in which reviewer confidence directly controls the sharpness of its contribution. After normalizing both scores and confidence levels to [0, 1] (using a lower bound of 0.2 for confidence normalization), each review i contributes term of the form
p((: x. x Nas æ (c:)²),
where (c,) 0.2(1 c.) + 0.05 is smooth linear mapping that decreases with confidence, assigning greater influence to high-confidence reviews while down-weighting more uncertain ones (see Appendix E for the detailed specification of (c:)).
Combining n such terms yields a Gaussian-shaped aggregate over a::
p( $1m; Cr:n)
II p(s; x, x.);).
(2)
i=1
whose closed-form mean given by a confidence-adjuster precision weighting:
Σ 8; o(c))
B[x $1m C1:n]
((;))-
Because the latent quality is constrained to [0, 1], we take the RTS to be the mean of this aggregate truncated and renormalized over the interval:
RTS
E|2 S1m, Cla,, x E [0, 1]]
(4)
RTS provides smooth and interpretable mechanism for incorporating reviewer confidence: high- confidence assessments exert greater influence, while low- confidence assessments remain included without dominating. Unlike simple weighted averaging, RTS preserves uncertainty structure in principled way and produces a final score that is guaranteed to within jo, 1]. Additional details of RTS are provided in Appendix B
4

---

Published as a conference paper at ICLR 2026
3.2 CONSTRUCTION OF THE NAIDV2 DATASET
To facilitate efficient debiasing in training and evaluation, we construct the NAIDv2 (where D de- notes Dataset). Following established community practices, we build from ICLR conference re- view data provided by OpenReview.net, ensuring acceptance ratio that closely reflects real-world conditions. Given that review data continually updated, earlier studies either failed to incorporate the latest ICLR reviews covered narrower time span (Zhu al. [025 Weng al. 2025). To address this, we developed a dedicated data engine and used it to compile NAIDv2, encompass- ing 24,276 samples spanning 2021-2025. Although our experiments relied only review scores, confidence ratings, and related fields, the underlying SQL-based dataset also retains additional infor- mation, including the full textual review comments. Furthermore, downloaded the corresponding original PDF submissions and applied MinerU (Wang et al. 2024a) for structured parsing (additional multimodal experiments
provided in Appendix
In addition to its scale and coverage, NAIDv2 is distinguished by its explicit handling of domain bias (as depicted in Fig. 2 As discussed earlier, raw review scores exhibit inconsistencies across domains and across time. To correct for these distortions, we move beyond noisy keyword-basee domain labeling and instead adopt clustering-driven strategy. Specifically, we encode paper titles and abstracts using
-4B (Zhang et al. (025d) and apply hierarchical cluster- ing (Bar-Joseph et al. (00T;; Müllner (20T)) identify latent domains based on semantic related- ness. Accordingly, we also compute the RTS from the raw scores, providing a robust signal for debiased training. This pipeline produces Nomain--ormallle labels that more faithfully capture expert consensus and enable debiased pairwise training.
NAIDv2 Dataset
Score
0,72
Conf.
3
Score
6
0,41
Conf.
Qwen
Emb.
Hier. Cluster
0,53
Gauss. Modeling
Score
5
Conf.
0,27
Conf.
Title Abstract
Labeled Samples
Embedding
RTS
Raw Score
Figure 2: Debiased data construction in NAIDv2. Titles and abstracts are embedded and hier- archically clustered into latent domains to mitigate domain bias, while raw scores with reviewer confidences are modeled via Gaussian likelihoods to calculate RTS, yielding normalized dataset robust debiased pairwise training.
For additional information on dataset construction, please refer to the Appendix H
3.3 PAIRWISE TRAINING AND POINTWISE SCORING WITH NAIPv2
Pairwise Training Stage. NAIPv2 adopts a Bradley-Terry style pairwise learning paradigm, in which the model trained to capture relative differences in RTS between two papers rather than regressing their absolute values (as illustrated in the rightmost panel of Fig. [ For pair of sub- missions (a, b) with ground -ruth signals RTS, and RTS, we construct a binary preference label RTS..b RT. RTS,), where T[-] denotes the indicator function (equal to 1 if the condition holds and otherwise). Each paper encoded independently by a shared LLaMA-3 (Dubey al., 2024) backbone followed by lightweight MLP head, producing scalar predictions and yo.
The Brarley-Terry preference probability is then defined as:
Eab = sigmoid -gb),
(5)
where sigmoid(-) denotes the logistic function. The model optimized using the standard binary cross-entropy loss shown in Eq. 6. with alternative pairwise objectives discussed in Appendix
5

---

Published as a conference paper at ICLR 2026
C(a,b) b) =-(RTS. log Eab + (1- RT log(1 - -Rab))
(6)
To mitigate distributional biases across domains and years, training pairs (a, b) restricted to submissions within the same cluster and the same publication year. This design prevents spurious comparisons caused by cross-field temporal inconsistencies.
To further enhance robustness and stability, we employ a curriculum learning mechanism based on the gap in RTS. Specifically, we partition training pairs into difficulty buckets according
aa = RTS. RT
where pairs with larger Aes are considered easier, while those with smaller Aab are treated as harder. During the early stages of training, easier pairs dominate the sampling distribution, guiding the network to quickly capture coarse quality distinctions. As training progresses, harder pairs are gradually upsampled, enabling the model to learn fine-grained quality differences.
Pointwise Inference Stage. Although training supervision is provided only submission pairs (a, b). the shared backbone guarantees that each input independently mapped scalar score:
J. = (xa; 0), ÿb [ 0),
where za and 2 denote the textual representations of submissions and 6, respectively. In other words, the pairwise objective implicitly induces a decoupled pointwise scoring function under shared parameters 0.
At inference time, this function naturally generalizes to single-paper evaluation:
ý fx;;
which provides pointwise estimate of paper quality that is directly comparable across submissions within the same domain-year subgroup. Although the pairwise loss constrains only relative dif- ferences, the shared backbone forces all submissions into common representation space, thereby yielding in practice a globally consistent scoring scale.
4 EXPERIMENTS
4.1 METRICS
As discussed in the introduction, NAIPv2 designed to support variety of application scenarios. To reflect these use cases, we evaluate the model using two complementary families of metrics. The classification metrics assess performance in settings related to automated decision prediction, allowing direct comparison with related work. The ranking metrics, on the other hand, align with the intended downstream applications of NAIPy2 and focus on performance tasks such as literature retrieval and recommendation.
For classification, we report Accuracy, FI, ROC-AUC, and Pairwise Accuracy (P. Ace). For ranking- oriented evaluation, report Spearman's 'ss p and NDCG @20. Formal definitions of all metrics provided in Appendix
4.2 IMPLEMENTATION DETAILS
We divide NAIDv2 into training, validation, and subsets. The validation is sampled from the training data with an internal 9:1 split and is used for hyperparameter tuning. To avoid potential information leakage (e.g., earlier ICLR submissions that might have been included the LLaMA-3 pretraining corpus), the test set is restricted to submissions from the year 2025, comprising 1,029 samples. NAIPv2 fine-tuned on A40 GPUs (48 GB each) using 8-bit quantization and LoRA adaptation (Hu et al. 2022), with a training time of roughly hour for 10k pairs (constructed from 23,247 samples). On a more common consumer GPU (RTX 3090, 24 GB), the same configuration conservatively estimated academic labs. The main hyperparameters include a learning rate of 1 10 a batch size 8, and training for epoch (more implementation details provided in the Supplementary Material).
complete within 6 hours, making the approach accessible to most

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on the ICLR review prediction task.
denotes the higher the
better. Results marked with are taken from Zhu et al. (2025)
Yu et al. 2024
means the
metric reported. Bold numbers denote the best results.
Category
Method
Acc 1
FI
AUC
NDCG f
p
Lower Bound (Random)
0.514
0,410
0.527
0.525
0.002
Upper Bound (Info. Leak)
0.819
0.757
0.894
0.995
0.984
API
APl-pointwise (ChatGiP)
0.644
0.427
0.654
0.702
0.315
APl-pairwise (ChatGPP)
0.658
0,448
0.655
0.686
0.297
AgentReview (Gemini)
0.424
0.424
0.097
AI Scientist* (Gemini)
0.614
0,481
0,120
AutoRegressive
SEA-EA (7B)
0,582
0.563
CycleReviewer (8B)
0.678
0.559
0.279
CycleReviewer (70B)
0.678
0.574
0.267
DeepReview* (14B)
0.689
0.623
0,405
Regress Only
N (8B)
0.545
0,472
0,605
0.629
0.183
NAIPv2 (ours, 8B)
0.706
0.609
0.782
0.771
0.432
4.3 COMPARISON WITH PREVIOUS APPROACHES
As shown in Tab. I we compare our proposed method against various approaches. Existing meth- ods can broadly divided into three categories. Category consists of prompting LLMs to predict review scores acceptance decisions based on paper titles, abstracts, full texts. Owing to the training-fie free nature of API calls, this paradigm has gained popularity, though its long-term usage costs remain high. Category II encompasses approaches that fine-tune LLMs for automatic viewing tasks or develop specialized frameworks. The goal of these methods is to maximize the reasoning and analytical strengths of LLMs,, enabling them produce comprehensive review texts. Category III fine-tunes LLMs to regress specific numerical values. Since these approaches avoid the autoregressive generation process, they achieve much faster inference speeds. In addition, we also report the empirical upper and lower bounds on the NAIDv2 Test Set. The upper bound is obtained by training an MLP with cross-entropy loss, where the inputs are reviewer-scor statisties (an 11 -ddienniional vector including the number of reviews and summary statistics of both scores and confidences), and the ground truth the binary acceptance decision (0/1). The lower bound determined by stochastic model.
For the first category, we implement the "API (pointwise)" and "API (pairwise)" approaches with reference to Höpner et al. (2025) (see Appendix F for details). This category has emerged as the most common choice among researchers due to its ease of use; however, neither the pointwise nor the pairwise variant has demonstrated satisfactory performance. The second research direction explores autoregressive inference using LLMs. Due to supervised fine-tuning on specialized data, these methods typically demonstrate stronger performance. DeepReview (Zhu al. 2025) reaches an F1 score of 62.3 and Spearman correlation of 0.,08 on their test set, demonstrating strong capability both acceptance decision prediction and score ranking tasks. Nevertheless, since the method generates tokens autoregressively. inference time scales with output length. On mainstream consumer GPUs (c.g., RTX 3090), inference for single paper can take up to three minutes, which limits its applicability in literature intelligence systems. Regression- only based approaches seek to bypass autoregression by directly predicting a single numerical output. NAIPv1 (Zhao al.) (025). for example, achieves up to ten predictions per second on the RTX 3090. However, since its training objective aligned with influence metrics rather than paper quality, its predictive quality only marginally better than random guessing. By contrast, pairwise learning design, matching NAIPvl in inference efficiency (over a thousand times faster than typical autoregressive approaches) while attaining performance par with DeepReview.
proposed approach leverages debiased
4.4 COMPARISON LEARNING PARADIGMS AND INFERENCE COMPLEXITY
We further analyze the paradigm differences to underscore the advantages of the NAIPv2 design. Table 2 compares three variants: (i) a pointwise method based on the NAIPv1 architecture but using RTS labels in place of the originals, (ii) pairwise variant that adopts the same architec-

---

Published as a conference paper at ICLR 2026
ture as the pointwise model but concatenates two papers into single input sequence and pre- dicts a binary 0/1 label, and (ii)) our proposed NAIPv2. The pointwise approach suffers from inconsistencies, resulting in suboptimal performance. The concatenation- variant achieves higher accuracy, but its inference requires explicit pairwise comparisons, leading to a complex- ity of OCC log C)), where C' denotes the number of candidate papers. In contrast, NAIPv2 incor- porates debiased pairwise learning during training while retaining pointwise inference, thus pre- serving the linear complexity O(C) of pointwise methods and achieving superior performance.
Table 2:
Comparison of alternative paradigms. The last column reports the theoretical lower-bound in- ference complexity of each paradigm.
4.5 EFFECT OF CLUSTERING STRATEGIES ON PERFORMANCE
The strategy chosen for pairwise grouping plays a crucial role in the model's capacity to distinguish between papers within the same set. Table 3 presents the performance dif- ferences across grouping methods. When no grouping is applied, or when grouping is based solely on temporal information, the model achieves only marginal improvements over direct prediction, and the results remain below optimal. Grouping via GPT-generated keywords performs worst, with accuracy approaching random guessing. This outcome mainly attributed to the overabundance of candidate keywords, which produces sparse valid pairs and insufficient training data. In contrast, hierarchical clustering outperforms tempo- ral grouping alone, and the combination of temporal and hierarchical grouping yields the strongest overall results. These findings highlight the critical role of debiasing in pairwise learning for paper quality assessment.
Paradigm
AUC
p
Th. Compl. O(C)
Pointwise
0,633
0.237
Pairwise Concat
0.720
0.351
OCClog C)
NAIPv2 (ours)
0.782
0.432
O(C)
Figure 3 further examines how the maximum distance parameter (max distance) in hierarchical clus- tering affects performance. As this parameter becomes smaller, the clustering grows increasingly fine-grained, eventually producing excessive number of small groups with very few samples, which introduces sparsity and weakens the reliability pairwise supervision. As the parameter becomes larger, the clustering becomes overly coarse, merging most papers into only few broad groups and diminishing the benefits of structured comparison. The results show that both extremes detrimental, and that intermediate setting provides more stable and informative partitioning. In particular, the strongest performance is achieved when the max distance parameter is set to 1.0.
Table 3: Impact of Pair Grouping Strategies. Ace stands for Pairwise Accuracy.
Table 4: Superiority of RTS.
Signal
AUC
P. Ace
p
Group By
AUC
P. Acc
p
Cites
0.583
0.556
0.358
None
0.739
0.630
0.400
Mean
0.753
0.586
0.385
Pub. Time
0.736
0.638
0.392
Weighted
0.754
0.598
0.402
Keyword
0.556
0.620
0.007
Median
0.757
0.599
0.398
Hier. Cluster
0.753
0.639
0.401
Mode
0.730
0.564
0.343
Time+Hier.
0.782
0.649
0.432
RTS (ours)
0.782
0.609
0.432
4.6 BENEFIT OF THE PROPOSED RTS
Table compares RTS with several commonly used quality signals. When citation count (Cites) is adopted as the signal for pairwise training, its AUC performance is substantially lower, partially supporting prior observations by Cortes Lawrence (202): y; Dougherty & Horne (2022). Simple averaging (Mean) and its confidence-weigheee variant differ only marginally, indicating that naïvely incorporating confidence provides limited value. Median aggregation performs slightly better, al- though the overall improvement remains modest. Mode aggregation yields weaker results, as it discards important distributional structure in reviewer scores. In contrast, RTS delivers consistently stronger performance: its probabilistic treatment of reviewer confidence enables more informative and principled integration of scores, resulting in a more discriminative and reliable quality signal than all traditional signals.
8

---

Published as a conference paper at ICLR 2026
AUC
0.80
0782
0:31
5.766
2.755 0.540
0.50
0.738
0.745
0.702
758
0.782
0.432
0.50
AUC
0779
0.404
0.70
2201
0.40
0.70
0.389
0.40
3.58
0,300
0.367
0.65
3334
0033
0.65
0.35
0.60
0.6
0.7
0.8
0.9
1.0
1.2
1.3
1.4
0.30
hardest
harder
moderate easier
easiest
Painvise Diffiuully Ase
Max Distance
Figure 3: Effect of clustering granularity.
Figure Effect of difficulty bucketing.
4.7 LEARNING WITH DIFFICULTY-AWARE SUPERVISION
To study the effect pairwise difficulty, we group training pairs into buckets by the RTS gap Aas (de- tailed in Appendix G). where larger gaps indicate easier pairs and smaller gaps correspond to harder ones. Figure 4 shows that model performance im- proves the proportion of easy pairs within the training mixture increases. This finding suggests that simple comparisons provide more stable and re- liable learning signals, whereas emphasizing hard pairs tends to introduce noise and degrade per- formance. Building on this observation, we further adopt a curriculum learning strategy. In the early training stages, sampling is biased toward easy pairs, allowing the model to first capture coarse distinctions. Over time, harder pairs are gradually introduced, enabling the model to refine its judg- ment subtle quality differences. The results in Tab. 5 verify that curriculum learning provides measurable improvements, supporting it as a useful complement to difficulty-awar supervision.
Table 5: Impact of curriculum learning
("w'o" denotes without, "w/" denotes with).
Setting
AUC
p
w/o Curriculum
0.770
0.418
w/ Curriculum
0.782
0.432
4.8
SCALING BEHAVIOR OF NAIPV2
NAPIZALC
NEVITUIC
MAPPT
HA ALC WPNP
0772
0.8
D,77
154
3551
3425
AUC TD
AUC
2050
0.4
6M
0.72
0.2
3.346
1000
2030
4200
5000
8000
10000
20000
25267
6000
8000
10000 20000 40000 60000
Number Samples
Number of Pairs
(a)
(b)
Figure 5: Effect of data size on model performance. (a) Performance comparison of NAIPvI and NAIPy2 with different numbers of samples, measured by AUC and Spearman's p. (b) Performance of NAIPv2 with different numbers of training pairs.
We analyze the data efficiency of NAIPv2 by varying both the number of supervised samples and the number of constructed pairs. As shown in Fig. []] increasing the amount of training samples consistently improves performance, and NAIPv2 maintains an advantage over NAIPy] across all scales. When varying the number of pairs, NAIPv2 shows steady gains up to roughly 10,000 pairs, after which performance plateaus and may decline slightly, suggesting diminishing returns from adding redundant constraints. Taken together, these trends indicate that the pairwise formulation enables NAIPv2 to extract more effective supervision from a limited set of examples, achieving strong performance even when trained with relatively few samples.

---

Published as a conference paper at ICLR 2026
5 FROM ICLR TO NEURIPS: A GENERALIZATION CASE STUDY
To further assess the generalization of our model, we conduct a cross-venue evaluation on NeurIPS. Following the same protocol as for NAIDv2, construct test set of 13,223 submissions from 2021 to 2024. Unlike ICLR, NeurlPS exhibits an imbalance in review availability; reviews of accepted papers are always public, whereas reviews of rejected papers appear only when authors choose to disclose them. This practice inflates the fraction of accepted papers in the accessible record relative to the true acceptance rate of about 20%, producing strongly skewed distribution. Such imbalance limits the meaningfulness of prior tasks built on this dataset.
NAIP22 *
NAPP1
API
GT
Accepted Paper
Reject Subribaiioo
0.8
0738
0,750
0.9
0.4 Avg. GT (Reviewen) Score
Reviewer
Avg. Predictive Score
0.7
0.8
0.6
0.714
0.488
0.7
0.5
API Ponttise
0:344
378
08
0.3
0.485
0:572
Pointwise RTS
0.233
NAIPv1
Rejected
Spotight
Oral
Highest
Decision Categories
Score
(b)
Figure 6: Cross-venus evaluation on NeurIPS, (a) Predicted scores follow the hierarchical structure of conference decisions. (b) Distribution of rejected papers across score ranges.
Given these limitations, we examine a more reliable property of the NeurlPS process: its hierarchical decision structure. NeurIPS organizes accepted papers into posters, spotlights, and orals, forming a quality-ordens sequence above rejected submissions. Under this structure, stronger papers should receive higher evaluation scores and appear more selective categories. We therefore test whether our predicted scores (logistic-simoid normalized) align with this hierarchy. As shown in Fig. (a), our model produces a clear monotonic increase from rejected to poster, spotlight, and oral papers, with a more consistent trend than NAIPv] and the pointwise API-based method. This alignment indicates that our approach is well calibrated with respect to the observed conference decisions.
In Fig. 6 (b), we further show the distribution of rejected papers across score ranges. Each row corresponds to the min-to-max assigned scores (Green denotes accepted papers, while red denotes rejections). The reviewer rejections cluster in the lower-score region, yet some appear in higher ranges, reflecting the valuable role of expert preferences in calibrating the review process. NAIPy2 exhibits trend similar viewer assessments, with lower-score regions containing more rejected papers, while the high-score regions contain fewer rejections than the reviewer-assigned scores. The APl-based approach shows weaker tendency, yet still outperforms stochastic prediction. Owing to inconsistencies, NAIPv] and direct RTS regression perform poorly, as rejected papers are distributed almost randomly across score ranges. This case study shows that NAIPv2 achieves stable generalization and consistent alignment with conference outcomes, suggesting promising directions for broader applications.
be regarded as the ground-truth distribution. As expected, most
CONCLUSION
This work presents NAIPv2, debiased and scalable framework for automated paper quality esti- mation. By employing pairwise learning within domain- year groups and introducing a confidence- aware probabilistic signal, NAIPy2 mitigates score inconsistencies while enabling efficient linear- time inference. On the curated ICLR test set, the model achieves state-of-the-art performance, demonstrating both accuracy and efficiency compared to prior approaches. Moreover, in a cross- venue evaluation NeurlPS, NAIPy2 predicts clear monotonic trend from rejected to oral papers and accurately places less rejections the high-score regions, demonstrating consistency with hu- man judgments and robustness under distributional shifts. Taken together, these results highlight NAIPy2 as an exploratory step toward robust scalable scientific intelligence systems, while leaving open important avenues for future research quality prediction and broader academic applications.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was supported by the Fund of the National Natural Science Foundation of China
(Grant No. 625761 77), Shenzhen Science and Technology Program (QNXMB3050701090000
JCYJ20250604 84027034, JCY1202408133 14237048), Guangdong Basic and Applied Basic Re- search Foundation (2066A151501145) the Fundamental Research Funds for the Central Universi- ties 070-63253222, and the Tianjin Key Laboratory of Visual Computing and Intelligent Perception (VCIP). Computation is supported by the Supercomputing Center of Nankai University (NKSC).
"Science and Technology Yongjiang 2035" key technology breakthrough plan project (2025Z053), Chinese government- guided local science and technology development fund projects (scientific and technological achievement transfer and transformation projects) (25420102G), National Science Fund of China under Grant No. 62361 166670. This work was also supported by the Academy for Advanced Interdisciplinary Studies, Nankai University (AAIS, NKU).
REFERENCES
Awais Ali, Muhammad Yousuf Jat Baloch, Muhammad Naveed, Anam Nigar, Abdulrahman Seraj Almalki, Ayesha Ghulam Rasool, Meseret Abeje Gedfew, and Ahmed A Arafat. Advanced satellite-based remote sensing and data analytics for precision water resource management and agricultural optimization. Scientific Reports, 15(1)2 2025.
Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Ziv Bar-Joseph, David K Gifford, and Tommi S Jaakkola. Fast optimal leaf ordering for hierarchical clustering. Bioinformatics, 17(uppl 1):S22 S)9, 2001.
Prabhat Kumar Bharti, Shashi Ranjan, Tirthankar Ghosal, Mayank Agrawal, and Asif Ekbal. Peeras- sist: leveraging paper-review interactions predict peer review decisions. In International Conference on Asian Digital Libraries, pp. 421-435. Springer, 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(334))324-45, 1952.
Zhe Cao, Tao Qin, Tie Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach listwise approach. In Proceedings of the 24th international conference on Machine learning, pp. 129-136 2007.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, fultioiunccionaality.. multi multi-granularity text embeddings through self-knowledge dis- tillation. arXiv preprint ar 242.:244..032221.. 2024a.
Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, et al. Ai4research: A survey of artificial intelligence for scientific research. arXiv preprint arXiv:2507.01903, 2025.
Zhenyuan Chen, Lingfeng Yang, Shuo Chen, Zhaowei Chen, Jiajun Liang, and Xiang Li. Revisiting prompt pretraining of vision-language models. arXiv preprint arXiv:2409.0611 2024b.
Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment. arXiv preprint arXiv:2109.09774, 2021.
Cyrus Cousins, Vijay Keswani, Vincent Conitzer, Hoda Heidari, Jana Schaich Borg, and Walter Sinnott- Armstrong. Towards cognitively-faithuu decision decision-making ment. arXiv preprint arXiv:2509, 04445, 2025.
making models to improve ai align-
Mike D' Arey, Tom Hope, Larry Birnbaum, and Doug Downey, Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.0422 2024.

---

Published as a conference paper at ICLR 2026
Joost de Winter. Can chatgpt be used to predict citation counts, readership, and social media in- teraction? an exploration among 2222 scientific abstracts. Scientometrics, 129(4):2469 2487, 2024.
Michael R Dougherty and Zachary Horne. Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences, Royal Society Open Science, 98))220334, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
F1000Research, lowa State University Library, Research Ireland, and University of Calgary. San francisco declaration on research assessment (dora). 2012.
Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, and Pushpak Bhattacharyya. Deepsentipeer: Har- nessing sentiment review texts to recommend peer review decisions. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 1 1120-11330 1130, 2019.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 1440-1448, 2015.
Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. Pasa: An Ilm agent for comprehensive academic paper search. arXiv preprint 20X1v:2501.10120, 2025.
Diana Hicks, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. Bibliometries: the leiden manifesto for research metrics. Nature, 520(7548):429 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint dr310 2015.
Niklas Höpner, Leon Eshuijs, Dimitrios Alivanistos, Giacomo Zamprogno, and llaria Tiddi. Automatic evaluation metrics for artificially generated scientific research. arXiv preprint arXiv:2503.005122.000 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Maximilian Idahl and Zahra Ahmadi. Openreviewer: A specialized large language model for gen- erating critical scientific paper reviews. arXiv preprint arXiv:2412.11945 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//axxiv.org/abs//22.. 2310//aaaa//3...... 06825
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine Van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A dataset of peer reviews (peerread): Collection, insights and nlp applications. arXiv preprint 3XXv:1804.09635 2018.
Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fahad Shahbaz Khan. Geochat: Grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 27831 2024.
Danyang Li, Tianhao Wu, Bin Lin, Zhenyuan Chen, Yang Zhang, Yuxuan Li, Wing-Ming Cheng, and Xiang Li. WOW-seg: A word-free open world segmentation model. In The Fourteenth International Conference on Learning Representations, 2026. URL paattss://ppeneeviee
net htte//orummiccc Ay
Yuxuan Li, Xiang Li, Yunheng Li, Yicheng Zhang, Yimian Dai, Qibin Hou, Ming-Ming Cheng, and Jian Yang. Sm3det: A unified model for multi-modal remote sensing object detection. arXiv preprint arXiv:2412.2066 2024a.
12

---

Published as a conference paper at ICLR 2026
Yuxuan Li, Yicheng Zhang, Wenhao Tang, Yimian Dai, Ming-Ming Cheng, Xiang Li, and Jian Yang. Visual instruction pretraining for domain- -ppain-ppecific foundation models. arXiv preprint 11X1::2509.17562, 2025a.
Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1504-1512, 2023.
Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26617-26626, 2024b.
Zheng Li, Yibing Song, Ming-Ming Cheng, Xiang Li, and Jian Yang. Advancing textual prompt learning with anchored attributes. arXiv preprint arXiv:4412.09442, 1, 2024c.
Zhuoqun Li, Xuanang Chen, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. Paperregis- ter: Boosting flexible-graie grained paper search via hierarchical register indexing. arXiv preprint arXiv:2508. 11116, 2025b.
Xun Liang, Jiawei Yang, Yezhaohui Wang, Chen Tang, Zifan Zheng, Shichao Song, Zehao Lin, Yebin Yang, Simin Niu, Hanyu Wang, et al. Surveyx: Academic survey automation via large language models. arXiv preprint arXiv:2502. 14776 2025.
Chengwei Liu, Chong Wang, Jiayue Cao, Jingquan Ge, Kun Wang, Lyuye Zhang, Ming-Ming Cheng, Penghai Zhao, Tianlin Li, Xiaojun Jia, et al. A vision for auto research with 1lm agents. arXiv preprint arXiv:2504. 18765, 2025.
Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends@ in Informa- fion Retrieval, 3(3::225-311, 2009.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien- tist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
Leland Mclnnes, John Healy, Steve Astels, et al. hdbscan: Hicrarchical density based clustering, J. Open Source Softw, 2(11 2(11):205, 2017.
Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378, 2011.
Moritz Staudinger, Wojeiech Kusa, Florina Piroi, and Allan Hanbury. An analysis of tasks and datasets in peer reviewing. In Proceedings of the Fourth Workshop on Scholarly Document Pro- cessing (SDP 2024), pp. 257-268, 2024.
Mike Thelwall. Research quality evaluation by ai in the era of large language models: advantages, disadvantages, and systemic effects-an an opinion paper. Scientometrics, pp. 1-13, 2025.
Louis L Thurstone. A law of comparative judgment. Psychological review, 101 1994.
Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv: 18839, 2024a.
Yidong Wang. Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, al. Autosurvey: Large language models can automatically write surveys. Advances in neural information processing systems, 37:115119-115144 2024b.
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang. Jindong Wang, Yue Zhang, and Linyi Yang. Cyclenesearcher: Improving automated research via automated review. In The Thirteenth International Conference on Learning Representations, 2025. URRlhhhtts:://peeeeeeee
net/ wwwporuummidibisssoooo j=b
Ge Wu, Xin Zhang, Zheng Li, Zhaowei Chen, Jiajun Liang, Jian Yang, and Xiang Li. Cascade prompt learning for languael model adaptation. In European Conference on Computer Vision, pp. 304-321. Springer, 2024.
13

---

Published as a conference paper at ICLR 2026
Wanjun Xia, Tianrui Li, and Chongshou Li. A review of scientific impact prediction: tasks, features and methods. Scientometrics, 128(1):543-555 585, 2023.
Xiangehao Yan, Shiyang Feng, Jiakang Yuan, Renqiu Xia, Bin Wang, Lei Bai, and Bo Zhang. Sur- veyforge: On the outline heuristics, memory-driven generation, and multi-dimensional evaluation for automated survey writing. In Proceedings of the 63rd Annual Meeting of the Association for 2025.
Computational Linguisties (Volume Long Papers), pp.
Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, Renjing Cui, Chengcheng Han, Qiushi Sun, et al. Automated peer reviewing in paper sea: Stan- dardization, evaluation, and analysis. arXiv preprini urXiv:2407. 12857, 2024.
Daoze Zhang, Zhijian Bao, Sihang Du Zhiyi Zhao, Kuangling Zhang, Dezheng Bao, and Yang Yang. Re2: A consistency-ensuree dataset for full-stage peer review and multi-turn rebuttal dis- cussions. arXiv preprint arXiv:2505. 07920, 2025a.
Xin Zhang, Xue Yang, Yuxuan Li, Jian Yang, Ming-Ming Cheng, and Xiang Li. Rsar: Restricted state angle resolver and rotated benchmark. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7416-7426, 2025b.
Xu Zhang. Danyang Li, Xiaohang Dong, Tianhao Wu, Hualong Yu, Jianye Wang, Qicheng Li, and Xiang Li. Unichange: Unifying change detection with multimodal large language model. arXiv preprint arXiv:2511.026000 2025c.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint vrXiv:2506.05176 2025d.
Yaohui Zhang, Haijing Zhang, Wenlong Ji, Tianyu Hua, Nick Haber, Hancheng Cao, and Weixin Liang. From replication to redesign: Exploring pairwise comparisons for Ilm-based peer review. arXiv preprint arXiv:2506. 10506111.3333 11343, 2025c.
Penghai Zhao, Qinghua Xing, Kairan Dou, Jinyu Tian, Ying Tai, Jian Yang, Ming-Mine Cheng, and Xiang Li. From words worth: Newborn article impact prediction with 11m. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 1183-1191, 2025.
Qihang Zhao and Xiaodong Feng. Utilizing citation network structure to predict paper citation counts: A deep learning approach. Journal of Informetrics, 16(1):101235, 2022.
Ruiyang Zhou, Lu Chen, and Kai Yu. Is llm a reliable reviewer? a comprehensive evaluation of 11m on automatic paper reviewing tasks. In Proceedings of the 2024 joint international conference computational linguistics, language resources and evaluation (LREC-COLING 2024), pp. 9340- 9351, 2024.
Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. Deepreview: Improving Ilm-based pa- per review with human-like deep thinking process. In Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 99330-29355, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89178 251-0. doi: 10.18653/v 1tt33/l/1222...oo acl- wow.::/...ll--o.......m.o 1420.
ac mel-long.eo 100g.14200 1420/]
14

---

Published as a conference paper at ICLR 2026
A ETHICAL STATEMENT
The scores presented in this work are generated by machine learning models trained on historical peer-review data and should be understood as black-box predictions, measures of scientific quality. These scores are not intended, nor should they be used, to replace human peer review editorial decision-making. Our contribution is solely for research purposes, with the aim of studying peer-review dynamics and developing methods for literature intelligence systems. The authors bear no responsibility for any use of the predicted scores real-world decision processes.
objective authoritative
B DISCLOSURE OF LLM USAGE
ChatGPT was employed during the experimental stage as an auxiliary tool for code debugging, code development, and data visualization. All code was thoroughly reviewed by the authors ensure accuracy and reliability. ChatGPT also assisted in polishing the language of the manuscript, but was involved in substantive writing. The authors take full responsibility for any issues arising from the use of ChatGPT.
C EVALUATION METRICS
This appendix provides the formal definitions of the metrics used our experiments. We group them into two categories: classification metrics and ranking metrics.
C.I CLASSIFICATION METRICS
ROC-AUC. Let P and N denote the sets of positive and negative samples. The ROC-AUC for model predictions ÿ is defined
AUC
ΣΣ JEPJEN
10000;1
(10)
|P|N
Pairwise Accuracy (PairAce). Let ÿ the thresholded predictions and {(i,g) y4 # v₁} be the set of cross-label pairs. PairAcc is computed as
PairAcc
1
Σ
I( o,) 1s) 0].
(11)
Ω
(K.))
Accuracy and F1. We follow standard definitions of Accuracy and F1-score.
C.2 RANKING METRICS
Spearman's Rank Correlation (p). Let d, denote the difference between the predicted and ground-truth ranks of item i. Spearman's p is defined as
6 3 d? 1
(12)
n n(n² 1)
NDCG @k
Let denote the ground-truth relevance of the item at position i. The Discounted
Cumulative Gain is
2
DCG
(13)
log, (i +1)
1=1
Let IDCG@k the maximum possible DCG@k. Then
NDCG@k DCG@k
(14)
IDCG@E
In experiments, use k = 20.
15

---

Published as a conference paper at ICLR 2026
D DEBIASED PAIRWISE LEARNING OBJECTIVES
In this section we broaden our investigation beyond the main loss and systematically examine alter- native objectives for pairwise learning. The motivation tions and the trade-offs they entail, rather than to commit to single formulation. Formally, given pair (a, b) with binary label 2nb = [[T Sa > Sb]. outputs Ya, yh, and difference das = ya yb- consider the following representative families.
understand the design space of loss func-
(A) Likelihood- objectives. These methods interpret the pairwise preference as a proba- bility that a is better than b. and optimize a proper scoring rule between predicted and observed preference.
Thurstone-Protee Replacing the logistic sigmoid with probit link yields
:prbbit ob
=(((ee)) dde)) [a, b) = =- log aab
sprobit
+ - (1-zab) log(1 (prob))
sprobir) ))
(15)
where (--) is the standard normal CDF and controls the slope. This treats preference as arising from latent Gaussian noise model, offering smoother gradients when comparisons are uncertain.
Thurstone-Mon MMosetene-Moselllr (TM). The TM model assumes that each paper has a latent Gaussian utility with mean μu and variance o². The preference probability is
TM
Ha
146
(16)
V03
where (()) is the standard normal CDF. The loss is the same likelihood-hant -hased objective,
TM LaM(a, b) - (Zab log Fb + (1 Zab) log(1
TTMM) ))-
(17)
Pairwise Brier. Instead of log loss, minimize squared error between predicted probability and label:
CBrir (a, b) (ddas) - zab))
(18)
This penalizes deviations quadratically and less harsh than cross-entropy when predictions are confident but wrong.
(B) Margin- and regression-stye objectives. These objectives work directly on the score differ- ence dah. either enforcing a margin or softly aligning with the review gap Agh-
Hinge margin. With labels
= 22ab - E 11,, +1}, we require dub to exceed a margin m:
Chinge (a, b) mm Yab dat] + [x]+ max(x, 0).
(19)
This loss is zero if the preferred item is already ahead by at least m, and otherwise grows lin- early-encouraggng confident separation.
Huber-on-ddiferencc Here we compare the predicted difference to the observed review score gap Aab. Residual Fab = dab Aub is penalized with robust Huber function:
12 ab>
|rabl S 8,
LuuberDift (a, (a,b)
(20)
5 ((rabll 2) |Pail > 8.
This encourages alignment with the magnitude of human score gaps, while limiting the influence of outliers beyond threshold 8.
(C) Hybrid objectives. aiming preserve ordering while stabilizing score magnitudes.
These combine a directional likelihood with a light regression regularizer,
Rank-then- Regress. We augment RankNet with an L2 penalty on the difference-gap residual:
CRI( (a, b) = =-(zat (Zab log (dat) + (1 - zas.) log(1 - a (dat)) + A (dar - k Anb)
(21)
16

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of different loss functions for pairwise learning.
Category Likelihood-basse
Loss
Acc
F1
AUC
NDCG
p
BCE (adopted)
0.706
0.609
0.782
0.771
0,432
Thurstone-
0.689
0.594
0.756
0.762
0.393
Thusslone-Mostelleer
0.717
0.592
0.685
0.759
0.395
Pairwise Brier
0.606
0.541
0.685
0.750
0.293
Margin Regression
Hinge Margin
0.711
0.606
0.772
0.783
0.420
Huber-on-differencc
0.531
0.548
0.699
0.758
0.288
Hybrid
Rank-then then-Regress
0.681
0.591
0.765
0.789
0.396
where \ controls the regression weight. This formulation preserves pairwise preference consistency while encouraging smoother calibration within domain-yeat subgroups.
Overall, results Tab. [ support our choice of a purely pairwise likelihood objective as the most stable formulation. Importantly once regression-style supervision is introduced, performance de- grades across metries, suggesting that absolute review scores suffer from translational inconsis- tencies across domains and years. This reinforces the view that enforcing numerical alignment is detrimental, whereas ordinal-only objectives provide more robust and transferable signals.
E NORMALIZATION IN THE COMPUTATION OF RTS
In computing RTS, we do not use the raw review scores directly. Instead, both the scores and the associated confidence values are normalized. Score normalization follows a standard min-max transformation:
minjeR 8
$ max 8j minjeR
(22)
where 8; denotes the score assigned by reviewer and is the set of reviewers for the submission. This ensures &; E [0. 1].
Confidence values are normalized with an adjustable lower bound a to control the effective range. The normalized confidence is defined as
minzeR Cj
ج = = maxjeR Cj minjeR Cj'
(23)
where & is the reported confidence of reviewer and ē (a, 1]. The parameter a [0, deter- mines the relative position of the lowest observed confidence in the mapped interval.
We conduct an ablation study to evaluate different meafidence-normalizann forms of (c), which governs the variance used in the RTS computation. As summarized in Tab. 7] smoother confidence-to-variann
schemes and alternative
mappings yield consistently better performance. moderate lower bound, specifically a 0.2, provides the strongest overall results. Both substantially smaller and larger values lead to performance degradation, indicating that restricting the effective confidence range to balanced level is beneficial for stability.
Table 7: Ablation study RTS normalization strategies. (a) Different formulations of (c).. (b)
Effect the lower bound a for confidence normalization.
(a) Formulations of ((c)
(b) Confidence normalization (lower bound
(c)
AUC
NDCG
p
a
AUC
NDCG
p
uaggressive
0.766
0.775
0.391
0.0
0.756
0.796
0.391
smooth
0.782
0.771
0.432
0.2
0.782
0.771
0.432
Rageressive (c) = 0.4(1
0.1.
0.5
0.764
0.772
0.405
mmooobl
0.2(1
0.05.
0.8
0.769
0.771
0.395
17

---

Published as a conference paper at ICLR 2026
F IMPLEMENTATION OF API-ONLY BASELINES
To ensure fair comparison and to examine the limitations of prompt-only approaches, implemented baseline experiments using GPT-5-mini as an AI reviewer. In the pointwise setting, the model was provided with the title and abstract of single paper and instructed to output normalized quality score in the range [0, 1]. In the pairwise setting, two papers were presented simultaneously, and the model was required to judge which paper exhibited higher overall quality. To further evaluate the robustness of prompt design, we constructed two distinct prompts for each setting, as reported Tab. [8] We then evaluated the outputs using both ranking-based and classification-bald based metrics. For classification, the Fl score was computed under decision threshold corresponding to the top- 30% predicted scores, following standard practice in acceptance prediction. This setup allows us to quantify the effectiveness of prompt-only methods and highlight their inherent weaknesses when compared with our proposed debiased pairwise learning framework.
Table 8: Performance comparison.
Category
Prompt
F1
AUC
NDCG
P
Pointwise
"You are an expert academic paper re- viewer. (iger.(EEvvaa nEvallute the quality of a single paper based on title and abstract only. \,n nCnnsider only:),n- Novelty and significance nn- Clarity of writing',n- Technical soundness (as inferred from abstract) ('' You MUST output STRICT JSON object ONLY with this exact schema:' n sscoree: 0.00).n score must be a normalized quality between and (inclusive). |nfommtted with exactly two decimal places. Do NOT include any extra keys or text."
0.427
0.654
0.702
0.315
"Review the paper from its title and ab- stract. stract.)nJudge JJJuge quality by originality, clarity, and soundness. (n RRetur ONLY a JSON object:), (f sccocr"" O. 000 J'n))nn)n (Score E [0, 1]. exactly two decimals. No other text."
0.422
0.637
0.667
0.278
Pairwise
"You are an expert academic paper re- viewer. Your job is to decide which of two papers is of higher overall quality',nbasse on title and abstract only. In' |n,,nnnsidee only:) Novelty and significance n Clarity of writing',n n- Technical soundness (as in- ferred from abstract)) nou MUST choose exactly one better paper (no ties). (nOutput a STRICT JSON object ONLY, with this exact schema: (n (("better": ("better": b} nDo NOT include any explanations, notes, extra fields. \nDo NOT include reasoning.. nDo NOT include any text outside the ISON object."
0.448
0.655
0.686
0.297
a) OR
"Decide which paper (a or b) is of higher qaality',
0.439
0.659
0.690
0.316
title and abstract only. Consider novelty, clarity, and') soundness. Pick exactly one. (n'utpu STRICT JSON:) "itttteee"e""" a} OR ("better": b) DnnnnnN nnNnNo extra text."
18

---

Published as a conference paper at ICLR 2026
G BUCKET DISTRIBUTION DEFINITION
we discretize the pairwise RT'S gap Aab into buckets with edges [0, 1.0.1..... 0.2....... 1.0), which correspond intervals. We define five representative ratio distribu- tions over these buckets to characterize different difficulty settings, ranging from Easiest to Hardest. Table 9 lists the exact proportions. In particular, easier distributions allocate more weight large gaps 0.8), whereas harder ones emphasize small gaps (< 0.1). The moderate case is close to uniform and serves as a neutral baseline.
As noted in Section
Table 9: Bucket ratio definitions for different difficulty levels.
Difficulty
0.1
0.1-0.2
0.2-0.3
0.3-0.4
0.4-0.5
0.5-0.6
0.6-0.7
0.7-0.8
0.8
Easiest
0.03
0.04
0.06
0.08
0.10
0.12
0.14
0.16
0.27
Easier
0.05
0.06
0.07
0.08
0.10
0.12
0.15
0.17
0.20
Moderate
0.111
0.111
0.111
0.111
0.111
0.111
0.111
0.111
0.112
Harder
0.20
0.17
0.15
0.13
0.10
0.08
0.07
0.06
0.04
Hardest
0.27
0.16
0.14
0.12
0.10
0.08
0.06
0.04
0.03
H DATA ENGINE FOR CONSTRUCTING NAIDV2
To enable scalable data collection and management, we implemented a dedicated data engine for NAIDv2 (Tab. 10 The engine adopts three-layer architecture: the Controller layer interfaces with external APIs (e.g.. OpenReview) and orchestrates data flow; the Service layer parses raw submissions and reviews, normalizes metadata (such confidence scores), and exports processed datasets; and the DAO layer provides object-oriented access to the underlying MySQL database.
The database consists of two core tables, 'Papers' and 'Reviews', linked by the paper identifier. The 'Papers' table stores submission-leve metadata (title, abstract, venue, derived GT labels, etc.), while the 'Reviews' table stores reviewer comments, ratings, and rebuttals. Automatic crawlers retrieve all submissions from OpenReview, insert missing entries, and populate reviews through structured parsing.
This modular design ensures reproducibility, facilitates large-scale updates across multiple years of OpenReview data, and supports downstream model training and evaluation.
EFFECT OF ADDITIONAL INFORMATION
As noted in NAIPvl (hao al. 2025). incorporating additional information can sometimes en- hance model performance, though such gains are not always consistent. In this study, adhering to the double-bling requirement, we extracted auxiliary information directly from the papers and in- corporated it into network training. Specifically, we employed computer vision-based approach to extract key figures and key tables: the scores for the key figures were derived through network inference, and the table descriptions were generated using analysis by GPT-5-mini. However, our experiments show that adding such auxiliary content does not improve performance across the tested settings. In fact, models augmented with introductions, conclusions, figure scores, table descrip- tions all perform worse than those relying solely on titles and abstracts. These results are broadly consistent with prior findings (Zhou al. 2024). indicating that not all types of supplementary content provide useful signals for review score prediction, further confirming that abstracts already contain sufficient information.
J ADDITIONAL. REMARKS ON EXPERIMENTAL COMPARISONS
In Tab. [ we compare the performance of existing methods, reporting some results directly from the original publications. We were unable to reproduce SEA-E Yu et al. 2024). OpenReviewer [Iahll] & Ahmadi, 2024), DeepReview Zhu et al. 22025), and CycleReviewer (Weng al. 2025) for two primary reasons. First, the training and test sets used in these works differ and sometimes over- lap, raising the possibility that test data appear in their training sets (and vice versa). Second,
19

---

Published as a conference paper at ICLR 2026
Table 10: Schema of the Papers and Reviews tables in NAIDv2 (partial view). PK Primary Key, FK = Foreign Key.
(a) Papers Table
(b) Reviews Table
Field
Type
Desc.
Field
Type
Desc.
BIGINT
RIGINT
title
VARCHAR(255)
Title
paper-id
BIGINT
FK + Papers.id
abstract
TEXT
Ahstract
belong_to
VARCHAR(128)
Group ID
external
VARCHAR(255)
Openreview
seply
VARCHAR(128)
Reply target
openreview venue
VARCHAR(255)
Venue
conf score
FLOAT
Confidence
scores
ISON
Review scores
rec score
FLOAT
Recommendation
confs
ISON
Confidences
reply.title
TEXT
Reply title
pub year
INT
Year
review summary strengths
TEXT
Summary
embedding
ISON
Embedding vee
TEXT
Strengths
cluster.cut
VARCHAR164)
Cluster label
weakness
TEXT
Weaknesses
Table 11: Effect of additional information on paper quality estimation.
Method
AUC
NDCG
p
Title & Abstract (ours)
0.782
0.771
0.432
+ Introdution
0.716
0.760
0.321
+Conclusion
0.750
0.768
0.395
+ Key Figure Score
0.769
0.798
0.417
+Key Table Description
0.758
0.753
0.381
retraining these models is computationally infeasible. For example, CycleReviewer requires train- ing a 123B-paraa -as--marameter model, which far exceeds our available resources. Moreover, we note that instances from benchmark datasets may be embedded in the training data of these methods, suggest- ing that their reported gains could partly reflect data leakage rather than genuine improvements in generalization.
Nonetheless, most prior studies follow a relatively consistent dataset construction protocol, typically sampling randomly from the ICLR dataset. As shown in Fig. 7 the distribution of our test set is well aligned with that of DeepReview, with Jensen-Shannon of 0.079. This alignment indicates that, despite the aforementioned limitations, the comparisons in Tab. remain reasonably meaningful across methods.
To further validate the effectiveness of NAIPv2, we retrain our model on the DeepReview-13K training set and evaluate it on the corresponding test set. Despite relying on a smaller backbone (8B vs. 14B), NAIPv2 delivers competitive, and in several cases superior, performance, underscoring the strength of our debiased pairwise learning framework. We additionally advocate for title- based data-splitting protocol, which enables researchers to determine whether a paper belongs the training or test set using only its title.
Method
2024 FI
2024 P.Ace
2024
2025 F11
2025 P.Ace
2025 p
DeepReview (14B)
63.07
62.42
35.59
62.27
64.02
40.47
NAIPv2 (ours, 8B)
66.13
63.37
33.22
58.94
60.87
40.82
Table 12: Results the DeepReview- -13K dataset (Zhu al. identical to those used in the main paper.
2025 with all hyperparameters kept
K FURTHER DETAILS ON EMBEDDING AND CLUSTERING
A key motivation for our embedding wwalldiiiggluutiiinnggn design arises from the observation that review scores can vary substantially across domains. However, directly applying keyword-based normalization of- ten suffers from issues of uncontrollable granularity and drifting category boundaries, which weaken its reliability. For instance, research on remote sensing (Li et al. 2024a) may be regarded either as visual per-
satellite-bal baseed sensing (Ali et al. 2025 Kuckreja et al.
2024 Zhang et al. 2025c)
ception (Girshick/ 2015 Li et al. 2025a Zhang et al.
2025b Li et al. 2026; ?). "PromptKD" (Li
et al. 2024b) might be situated within prompt learning (Li et al. 2024c Wu al. 2024 Chen al.
20

---

Published as a conference paper at ICLR 2026
100
125
100
Count
Count
60
75
50
25
0.0
0.6
0.4
0.6
1.0
(a) NAIDv2 (Ours)
(b) DeepReview
Figure 7: Data distribution comparison between NAIDv2 (ours) and DeepReview Test 2025 datasets. The two distributions are well aligned, with Jensen-Shannon of 0.079, indicating com- parability of evaluation results across methods.
To address this challenge, employ Qwenž-Embedding in combination with hierarchical cl clustering, which enables semantically grounded grouping of papers beyond brittle keyword matching.
2024b)
knowledge distillation (Hinton et al. 2015 Li et al. 2023
As shown in Table 13 and Table 14 this setup allows us to systematically study the impact of em- bedding and clustering choices. Both BGE-M3 (Chen et al. 2024a) and Qwen3-Emmeedene provide strong baselines, yet Qwen3-Embedding paired with hierarchical clustering achieves the most con- sistent gains. HDBSCAN Mclnnes al. (2017) remains competitive but trails slightly in AUC and Importantly, embeddings enriched with explicit cousterigg-orintte instructions outperform those without, and incorporating both titles and abstracts yields better representations than titles alone. Among the prompts we explored, the one explicitly directing the model to group semantically simi- lar papers provides the most balanced improvements across all evaluation metrics.
Table 13: Impact of embedding models and clustering algorithms on retrieval performance. (a) Comparison of embedding models. (b) Comparison of clustering algorithms with confidence nor- malization.
(a) Embedding models
(b) Clustering algorithms
Model
AUCT
NDCG
Method
AUC1
NDCG1
p
BGE-M3
0.774
0.769
0.417
HDBSCAN
0.765
0.803
0.422
Qwen3-emb
0.782
0.771
0.432
Hier. Cluster
0.782
0.771
0.432
Table 14: Comparison of different instruction prompt designs
Instruction
AUCT
NDCGT
None (no instruction)
0.759
0.775
0.266
Identify the main category of scholar pa-
0.767
0.831
0.416
pers based on the titles and abstracts
Given title of scientific paper, retrieve the titles of other relevant papers Cluster similar papers close as possible 0.782
0.768
0.784
0.412
0.771
0.432
based on title and abstract (ours)
L ABLATION ON PAIRWISE SAMPLING STRATEGIES
To further investigate the robustness of the proposed pairwise learning strategy, conduct ablation studies by introducing additional constraints during pair construction. The first experiment (Table [5] Panel (a)) examines the effect of limiting the maximum number of times sample can appear in training pairs. The results show that imposing such restrictions does not provide clear benefits. In fact, the best overall performance is achieved when limitation is enforced, suggesting that
21

---

Published as a conference paper at ICLR 2026
allowing samples to be paired freely provides greater diversity and stronger supervision signals for training.
The second experiment (Table 15 Panel (b)) analyzes the role of enforcing minimum score dif- ference between paired samples. Without such a margin, performance degrades, indicating that constructing pairs from nearly indistinguishable items introduces noise into the training signal. A margin of 0.05 provides the best overall balance, improving both AUC and Spearman correlation, while larger margins (2 0.1) result in performance drops due reduced pair availability. These find- ings highlight the importance of carefully controlling pair selection to enhance both the reliability and effectiveness of pairwise learning.
Table 15: Effect of pairwise learning constraints. (a) Influence of limiting the maximum occurrence times of each sample pairwise training. (b) Influence of enforcing minimum score difference when constructing pairs.
(a) Maximum occurrence times
(b) Minimum score difference
Setting
AUC
NDCG
p
Setting
AUC
NDCG
p
2
0.776
0.773
0.426
0
0.752
0.778
0.395
4
0.766
0.791
0.408
0.05
0.782
0.771
0.432
8
0.767
0.774
0.391
0.1
0.753
0.785
0.397
16
0.782
0.771
0.432
0.15
0.770
0.778
0.402
32
0.782
0.771
0.432
0.2
0.761
0.774
0.404
M
CROSS-ARCHITECTURE EVALUATION OF NAIPV2
We further evaluate the transferability of NAIPv2 on additional LLM families: Qwen (Bai al., 2023) and Mistral (Jiang et al.) 2023). These experiments aim to verify whether the gains brought by our debiased pairwise learning framework are specific single backbone or generalize across architectures.
The results in Table 16 demonstrate that the core idea of NAIPv2 continues yield consistent improvements both Qwen and Mistral models. In both cases, replacing pointwise regression with our pairwise training strategy leads to substantial gains across all evaluation metrics, including accuracy, F1, AUC, NDCG, and Spearman correlation. These findings indicate that NAIPv2 is not tied to a particular LLM family and transfers well across different backbone architectures.
Table 16: Performance comparison Qwen and Mistral backbones. Pairwise training consistently improves evaluation metrics across both model families.
Approach
Ace f
F1
AUC
NDCG 1
Spearman
Q Pointwise
0.555
0.521
0.656
0.682
0.232
Qwen-7B Pairwise
0.653
0.576
0.738
0.786
0.357
Mistral-8B Pointwise
0.567
0.536
0.675
0.769
0.285
Mistral-8B Pairwise
0.621
0.553
0.711
0.827
0.342
N DISCUSSION AND PRACTICAL CONSIDERATIONS
As shown Tab. 17 NAIPv1 performs strongly in predicting the academic impact newly pub- lished articles, whereas NAIPv2 underperforms on the NAIDv1 benchmark. This result appears consistent with the observation of Cortes Lawrence (20TT) that impact and quality are often only weakly correlated.
These findings suggest that there is no single "univer- sal key" for all tasks, and the choice of method should depend on the specific application. API-based solutions easy use and broadly accessible, but their perfor- mance is limited (Tab. []: fine-tuned autoregressive mod- els benefit from
Table 17: on article impact prediction task.
Performance of NAIPv 1/v2
Model NAIPv1 NAIPv2
NDCG
p
optimization and achieve
0.901
0.459
0.761
0.279
22

---

Published as a conference paper at ICLR 2026
stronger results, though the cost of additional deploy- ment requirements; NAIPv2 provides performance com- parable to these autoregressive approaches with much faster inference, but lacks the interpretive outputs that autoregressive models offer.
Overall, these comparisons indicate that single method prevails across all scenarios, underscoring the importance for both research and industry communities to recognize and leverage the comple- mentary strengths of these approaches.
23


Reviews:
[{"review_id": "i6eqlLwFhC", "reviewer": "Reviewer_hKKi", "summary": "This paper constructs a dataset (NAIDv2) along with a framework (NAIPv2) for automated paper quality estimation. In particular, the framework adopts pairwise learning (comparing two papers at a time) but is deployed as an efficient pointwise predictor (judging a single paper's quality). The framework also uses a gaussian model, named Review Tendency Signal, to aggregate multiple reviews into a single ground-truth quality signal. Tested on unseen NeurIPS submissions, NAIPv2 is able to achieve better performance over the previous NAIPv1 framework.", "questions": "What are some good application scenarios for NAIPv2?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper tackles an important problem to collect a massive dataset of paper review and propose a data-driven solution to automate the paper review process.\n2. This paper introduces several methods to make its framework efficient and practical such as pairwise learning, pointwise inference; Review Tendency Signal etc.\n3. The performance and generalizability of the framework appears to be good.", "weaknesses": "1. The relationship between NAIDv2 and NAIPv2 is confusing due to the naming. It appears that NAIDv2 is just the dataset used for learning the NAIPv2 framework.\n2. The paper introduces models such as Review Tendency Signal, but it is unclear how good this probabilistic model is, compared to other alternatives, e.g., the review score might follow a Bradley–Terry model.  \n3. The experiment results on the neurips dataset is better than the previous method, but it is still unclear how it is useful for practical applications, e.g., can it guide the AC decision in assessing the paper quality in addition to human reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dLapbwz9SU", "reviewer": "Reviewer_g7oZ", "summary": "This paper proposes NAIPv2, a framework for estimating scientific paper quality by combining pairwise learning with a probabilistic Review Tendency Signal (RTS) that integrates reviewer scores and confidences. The authors introduce a new large-scale dataset (NAIDv2) derived from ICLR submissions and report strong results in both within-domain and cross-domain evaluation. The central claim is that NAIPv2 provides debiased, scalable, and efficient quality estimation compared to prior LLM-based or regression-based approaches.\nThe paper is clearly written and tackles an important and timely problem in automated peer review and scientometric prediction. However, the theoretical framing and probabilistic modeling choices raise several concerns, especially regarding the internal consistency of the modeling assumptions and the clarity of related work.", "questions": "Related Work and Fairness of Discussion\n(124)\nThe criticism of Zhang et al. (2025a) on the Re² dataset seems unfair and unsubstantiated. The statement “deviations between reported results and those presented in the original papers” needs concrete evidence or examples. If the concern is data leakage or overlap, that should be clearly documented and cited. Otherwise, this reads as an unsupported claim, can the authors clarify?\n\n\n(154)\nThe summary of Höpner et al. (2025) lacks essential details. The claim that pairwise ranking is “unreliable” for review scores is unconvincing.\n\n\nIf regression on review scores is feasible, pairwise regression should also be; at worst it merely doubles the noise variance of independent samples.\nThe critical question is how pairs are selected and what is predicted (rank order, score difference, or ordinal classification). It seems to me that pairwise comparisons are most interesting for ranking, rather than cardinal scores, as you have the additional benefit of simplifying the problem domain to a binary prediction. In cardinal (regression) settings, all that's happening is canceling out translational offsets, which can be interesting when you're controlling for specific effects, but seems unimportant for random pairings. It's not clear from the descriptions in related work which of these methods are being employed, and this seems extremely important. Pairwise methods aren't monolithic: the advantage or lack thereof in using them comes from how pears are chosen and what function of the pair one tries to predict, and I would like to see this explored or discussed.\n\nBroader Modeling Questions:\n\nIn automated paper review methods discussed, it seems like there are at least two axes of comparison here. \n\n1: The methods used to make predictions, and 2: what is being predicted. Why not treat citation counts, reviewers scores, and pairwise predictions as separate target, try to predict them all with one model, and select some combination of them for the actual quality estimate? Moreover, these various targets don't seem incompatible. Some concepts are better predictors of quality than others, but likely in complementary ways. It seems a joint predictive model would benefit from shared structure, and this multimodal learning would also make better use of available data.\n\nIt would be valuable to separate and jointly analyze multiple predictive targets (e.g., citation counts, reviewer scores, and pairwise preferences) within a unified model. These signals are not incompatible and could benefit from shared structure via multitask learning.\n\n\n\nBias:\nIt seems to me that, under a Bradley Terry or Thurstone-Mosteller model, you are controlling for bias by subtracting out factors. You subtract out year-dependent and domain dependent factors with this strategy. \n\nThe “de-biasing” strategy seems roughly equivalent to including additive terms for each year-domain combination in a regression model, since you're using pairwise comparisons to control for these effects. It would help to compare these approaches empirically or at least discuss their equivalence.\n\nIn the same vein, why not restrict pairwise comparisons to papers reviewed by the same reviewers? Although this reduces the number of pairs, it would directly control for reviewer-level bias and may yield cleaner comparisons. It also may alleviate or reduce the need to normalize for each reviewer.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This is an interesting and ambitious paper tackling a key problem in automated peer review. The dataset and efficiency contributions are also substantial.\nEquation (1) is well-motivated and welcome to see; it provides a principled probabilistic link between confidence and uncertainty rather than relying on ad hoc weighting.\n\n\nThe proposal to integrate reviewer confidence directly into the score aggregation model addresses a genuine issue in automated peer review and quality estimation.\n\n\nThe focus on efficiency and domain-year stratified training is well thought out and practically motivated.", "weaknesses": "(Main critique)\nYou introduce and motivate a Gaussian probabilistic model (RTS) but then discard it during pairwise ranking. When training with a logistic loss, you implicitly assume Gumbel noise, not Gaussian, effectively switching from a Thurstone–Mosteller to a Bradley–Terry model.\n\n\nThis inconsistency weakens the theoretical story: RTS models uncertainty via Gaussian variance, yet the pairwise comparisons ignore that structure and the confidence information is lost.\n\n\nFor example, if paper A’s reviewers are all high-confidence and paper B’s are all low-confidence, this distinction disappears once RTS is computed. A unified probabilistic model that preserves confidence across comparisons would be both cleaner and more principled.\n\nFor the avoidance of doubt, the model to which I refer is essentially a Thurstone-Mosteller model, where we take the probability of one or more reviewers scoring paper A over paper B, given the observed scores we have from reviewers for these papers, using your theoretical model of latent-factor based reviewer-paper scoring, and confidence values as additional inputs.\n\nSee e.g., \n[Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignment\nCyrus Cousins, Vijay Keswani, Vincent Conitzer, Hoda Heidari, Jana Schaich Borg, Walter Sinnott-Armstrong] for discussion of probabilistic pairwise prediction models in the broader ML context.\n\nThe Thurstone-probit of Appendix D approaches this idea, but variance information has already been removed by this point. I’d like to see this further explored.\n\nConceptual and Modeling Issues\n(186–213)\nMany technical aspects are vague in their description, hampering understanding and readability. This is likely a writing quality issue, rather than a criticism of the work behind it.\nThe introduction of the Review Tendency Signal (RTS) as a posterior mean implicitly assumes a Bayesian framework, but the paper never defines a prior or explains how the posterior is derived. As written, the likelihood $p(s_i \\mid x, c_i)$ is defined, but this does not imply a likelihood on $x$. Please clarify whether a uniform prior is assumed, and present the closed-form expression for the posterior mean.\n\n\nGiven your confidence mapping functions, the posterior mean likely admits a clean closed form, which would help motivate the simplicity of your model.\n\n\n(Appendix F; σ(cᵢ))\nThe mapping from reviewer confidence to variance is extremely limited. Although you normalize per reviewer, a learned nonlinear function over the entire reviewer population would be both straightforward and far more flexible.\n\n\n(269)\nClarify that the additive offset correction may depend on year and domain; a global offset is only justified if all pairwise comparisons form a connected component.\n\n\n(242, 682–696)\nThe term sigmoid is too vague — you almost certainly mean the logistic function. This matters because later in the appendix you replace it with a Gaussian CDF, which is also a sigmoid; the text should explicitly reference this distinction.\n\n\nThe model should be described more clearly in the language of Bradley–Terry and Thurstone–Mosteller frameworks, which would provide a unifying probabilistic interpretation and clarify the noise assumptions.\n\n\nThe appendix (lines 682–696) is interesting but should be integrated into the main text. The theoretical implications of alternative link functions (logistic vs. Gaussian CDF) should be discussed explicitly, as should the choice of Brier loss. A connection to strictly proper scoring rules would strengthen this section.\n\n\nTerminology and Framing\nThe term scale inconsistency appears repeatedly but seems misapplied. The issue appears to be translational inconsistency (offsets), not multiplicative scale differences. The current phrasing is consistent but confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9gRpFwf7oX", "reviewer": "Reviewer_f3tL", "summary": "This paper introduces NAIPv2, a fast regression model for paper review score estimation. The key idea behind this model is Review Tendency Signal (RTS), which uses not only the raw scores of each paper review but also the specified confidence level to model uncertainty behind individual scores. Alongside the model, the authors also introduce NAIDv2, a collection of ICLR review data compiled from OpenReview.", "questions": "- Regarding Section 3.2, it’s not very intuitive to me whether NAIDv2 really handles domain bias. NAIDv2 only contains ICLR submissions, which seems to be a relatively homogenous set given that ICLR is a ML conference. Are the authors suggesting that two papers of similar content and quality could have significantly different scores just because the authors chose different submission categories?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Clear benefits of using confidence information and RTS over raw scores, as seen in Table 4\n- Computational efficiency of NAIPv2 with pointwise inference\n- NAIDv2, a collection of reviews and confidence scores from ICLR, appears to be thoroughly constructed.\n- Performance gains over more computationally expensive API and autoregressive baselines\n- Extensive evaluation of NAIPv2/NAIDv2, including interesting demonstration of model generalizability based on NeurIPS decision category hierarachy", "weaknesses": "- It looks like the paper does not exactly mention how regression head was implemented on top of LLaMA 3 for NAIPv2. I didn’t have time to check out NAIPv1 paper so I’m not sure if the architecture was discussed there, but still I think it would be nice to have the architecture description somewhere in the paper.\n- It would have been nice if NAIPv2 was also tested on the LLMs other than LLaMA 3 to see if the benefits of NAIPv2 still holds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lL0bSj1yTk", "reviewer": "Reviewer_64Pf", "summary": "As the number of submissions to peer-reviewed conferences increases exponentially each year, systems that automatically assign a review score to submissions may become increasingly important. This paper proposes to train an LLM to assign a score to a research article solely based on its title and abstract. \nIt first introduces a new dataset, NAIPD2, curated using  ICLR submissions from 2021 to 2025. \nTo account for reviewer confidence, it proposes a confidence-normalised target score, RTS, which is used as the ground-truth label for each paper. The paper claims that reviewing style (and hence review scores) varies by the paper’s domain and year. To account for this bias, instead of directly regressing on the RTS score, they propose a pairwise loss computed using a siamese-style network over the pairs selected from the same domain and year. The network is trained via BCE loss to predict which paper in a training pair has a higher RTS score.  To create domain buckets, instead of using the given keywords, the paper proposes to use hierarchical clustering of the paper’s title and abstract embeddings. To ensure that the LLMs being trained haven’t seen the papers/reviews before, it proposes to use only 2025 submissions in the test set.\nThorough ablations justify the design choices.", "questions": "1. Line 247: 248: One of the claims in the paper is that reviewing style varies by domain and year, and clustering is used to create domain buckets. Can you show that review scores indeed follow a different distribution in each cluster-year group? Maybe a plot with cluster-year on the x-axis and some statistic quantifying the distribution on y-axis? \n\n2. Line 213-214 -- “pipeline produces domain-normalized labels”. How is RTS normalised using the domain information?  In eqn 14 and 15, what is j iterating over? Scores of the same paper, or all the scores in the dataset?\n\n3. Given that papers from the same year are used to select the pairs during training, does it mean that we can only compare scores of the papers in the same domain and year? Can we use it to meaningfully compare papers across years?\n\n4. Line 314: 316: Could you please explain the upper bound? Is the input to the MLP just a scalar or a vector of all the review scores for a paper? What if a paper has 3 reviews and another has 4 reviews? \n\n5. To compute F1 and accuracy, you need to select a threshold for the score based on validation data. Is the threshold selection dependent on domain / year?\n\n6. Table 4 / Section 4.6: When using a different statistic (e.g., mean), I hope that it is used both for training and while testing and reporting the numbers in Table 4 - i.e. please clarify that you are not training using mean but then using  RTS to compute ground-truth in pairwise accuracy and ground-truth rank in correlation. \n\n7. Section 4.7: In Figure 4, it is not clear what you are comparing.  You have pairwise difficulty on the x-axis. For a given bucket, say the “easiest” bucket, are you reporting the performance obtained when you train your model using only the “easiest” pairs?  If yes, then how come it matches exactly with the performance in  ‘w/ Curriculum’ setting in Table 5 where you are using all pairs but in a curriculum.\n\n\n\n**Comments:**\n\n1. Given that the paper claims RTS as one of its contributions, its details should be moved from the appendix to the main paper.\n\n2. Table 1: Isn’t a higher correlation better? There is no upward arrow sign beside it.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written, with most of the details present either in the main paper or the appendix.\n\n2. The paper has thorough ablations to justify all the design choices.\n\n3. The paper releases a new dataset that may help the research community.", "weaknesses": "1. While I understand that regression-based methods are obviously much more efficient, the lack of any reasoning provided by the model significantly hampers their applicability. On the other hand, methods like DeepReview provided a detailed output that can assist in actually reviewing a paper and perform as good as the proposed method. \n\n\n2. Another cited contribution is release of Naidv2 dataset. However, similar datasets ( and recipes to create them) already exist, e.g. the dataset used in DeepReview.\n\n3. Another issue is a fair comparison with other baselines. All the reported experiments are ablations for the design choices made by the paper, e.g. pairwise loss vs pointwise loss, curriculum vs no curriculum etc., I don’t see an apples-to-apples comparison with other trained auto-regressive models that may be more inefficient but more accurate and useful.  \n\n4. A simple baseline would be an LLM finetuned to generate the review along with the score. This way, all the reviews from the paper can be considered as a separate training sample. \n\n5. Line 322:323: Table 1 reports numbers directly from DeepReview and CycleReview papers, which are computed on different test data (though from ICLR 2025 only). Typically, numbers reported in the same table should be computed on exactly the same dataset.\n\n6. OpenReviewer is mentioned in the related works, but not used as a baseline. \n\n7. The designed system takes only the title and abstract into account, whereas human reviewers would seldom base their decision solely on these two attributes. Does it mean that the system is capturing spurious correlations? \n\n8. While the application of a pairwise loss function to the problem at hand may be novel, the loss function used is the standard one employed to train reward models under Bradley-Terry (BT) reward model assumptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WAmKMXYGP0", "reviewer": "Reviewer_qFdP", "summary": "LLM-assisted peer review has become a popular topic in recent years, and Paper Quality Estimation is an important part of it. This paper proposes NAIPv2, a model that takes the titles and abstracts of two papers as input and predicts their pairwise quality. The model is based on the Llama 3 backbone, trained on the ICLR dataset with human reviewer’s feedback as ground truth. NAIPv2 achieves state-of-the-art performance (AUC and rank correlation) than several baselines, including autoregressive models. In addition, the method runs in linear time, generalizes well, and works when only abstracts are available instead of full papers.", "questions": "* Many studies in this area use citation count as a proxy for paper quality because (1) their methods are unsupervised, and (2) human reviews are highly noisy, making it hard to align with them. I suggest adding experiments that use citation-based proxies and comparing them with existing methods, although they are not reliable. Such results could reveal deeper insights: if NAIPv2 performs worse on predicting citations but aligns much better with human reviews, it may suggest that human judgments rely heavily on information from titles and abstracts.\n* This is more of a curiosity than a critique: since judging paper quality solely from titles and abstracts is unrealistic, I believe that NAIPv2’s strong alignment with human reviewers may come from certain “cheap signals” in abstract / title. I wonder what those signals might be; perhaps the authors could offer some hypotheses or discussion.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper proposes a paper quality predictor NAIPv2 that achieves state-of-the-art performance using only titles and abstracts, showing a high rank correlation with human reviews.\n* NAIPv2 is efficient (linear time), generalizes well, and does not require full text, which makes it practically useful. I look forward to seeing downstream applications, such as helping authors improve their titles and abstracts.\n* The paper combines existing tools into a clear and effective workflow. Although it uses Llama 3 as a backbone, given that paper quality estimation is still a new topic, I find the overall novelty acceptable.\n* The experiments are well done. Some additional comments are listed in the “Questions” section.", "weaknesses": "* I would like to see experiments using more proxies for paper quality (see “Questions” section).\n* Section 3.3 is a bit confusing. the training details could be explained more clearly in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)