Post Hoc Regression Refinement via Pairwise Rankings
Authors: Kevin Tirta Wijaya
Michael Sun
Minghao Guo
Hans-peter Seidel
Wojciech Matusik
Vahid Babaei
Venue: NeurIPS 2025 poster ()
Keywords: prediction refinement
pairwise ranking
molecular property prediction
Primary area: general_machine_learning

Abstract: Accurate prediction of continuous properties is essential to many scientific and engineering tasks. Although deep-learning regressors excel with abundant labels, their accuracy deteriorates in data-scarce regimes. We introduce RankRefine, a model-agnostic, plug-and-play post-hoc refinement technique that injects expert knowledge through pairwise rankings. Given a query item and a small reference set with known properties, RankRefine combines the base regressor’s output with a rank-based estimate via inverse-variance weighting, requiring no retraining. In molecular property prediction task, RankRefine achieves up to 10\% relative reduction in mean absolute error using only 20 pairwise comparisons obtained through a general-purpose large language model (LLM) with no finetuning. As rankings provided by human experts or general-purpose LLMs are sufficient for improving regression across diverse domains, RankRefine offers practicality and broad applicability, especially in low-data settings.

Post Hoc Regression Refinement via Pairwise Rankings
Kevin Tirta Wijaya
Michael Sun MIT
Minghao Guo MIT
MPI-INF kevintirta. wwgmail.cmm.oooommmmmooooo.
maun4155csail.miie mit.edu
guomh2014@@gail.co com
Hans-Peter Seidel
Wojciech Matusik MIT wojcisch@csail mit. edu
Vahid Babaci MPI-INF
MPI-INF
hpseidel@mpiiinn mpg. de
@babaei@mpi-ii mpg.pp.....
Abstract
Accurate prediction of continuous properties is essential to many scientific and engineering tasks. Although deep-learning regressors excel with abundant labels, their accuracy deteriorates in data-scarce regimes. We introduce RankRefine, a model-agnostic, plug-and-play post hoc method that refines regression with expert knowledge coming from pairwise rankings. Given query item and small reference set with known properties, RankRef ine combines the base regressor's output with a rank-based estimate via inverse-variance weighting, requiring no retraining. In molecular property prediction task, RankRef ine achieves up to 10% relative reduction in mean absolute error using only 20 pairwise comparisons obtained through
large language model (LLM) with no finetuning. As rankings provided by human experts or general-purposs LLMs are sufficient for improving regression across diverse domains, RankRef f ine offers practicality and broad applicability, especially low-data settings.
Introduction
Accurate prediction of continuous properties is crucial across scientific and engineering disciplines. In poolecula--rroperyy prediction (MPP), for instance, reliable estimates physical or chemical attributes can accelerate drug discovery, materials design, and catalyst development. Recent advances deep learning have enhanced regression models by leveraging large datasets uncover complex, nonlinear relationships between structured inputs (e.g.. molecular graphs, crystal structures) and their associated properties. Yet, unlike computer vision or langraeee processing, where large labeled corpora can be mined or crowd-sourced, many specialized fields face fundamental bottleneck: acquiring ground-truth labels requires expert-led experiments that are both costly and slow. Consequently, real-world tasks often operate in data-scarce regimes where even 50 labeled samples may considered plentiful (Sun et al. 2024].
When large-scale labels are unavailable, expert knowledge becomes valuable but under-utilized asset. One accessible form of such knowledge relative comparison: pairwise judgments about which of two samples exhibits higher (or lower) property value. Pairwise rankings convey substantial information, are often easier for human experts provide, and can even be generated by general- purpose large language models (LLMs), which have shown surprising ranking skills in chemistry and related domains [Sun et al. 2025
Guo et al. 2023
We introduce RankRef ine, a post hoc refinement framework that improves regression predictions by incorporating pairwise rankings. Given query sample and small of labeled references, which can conveniently be drawn from training data, ranker infers the relative ordering between the query and each reference; these comparisons define likelihood whose minimization yields rank-based
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

property estimate. RankRefine then fuses this estimate with the base regressor's prediction via inverse-variance weighting, all without retraining.
Because RankRef ine is largely model-agnostic and requires no architectural changes, suits low- data, few-shot, and meta-learning scenarios. Leveraging ranking signals from publicly hosted LLMs, it provides substantial improvements in regression accuracy with minimal labeled data and negligible client-side computation. Experiments on synthetic and real-world benchmarks, including multiple MPP tasks, demonstrate that RankRef ine consistently boosts predictive performance. Further analysis links its effectiveness to ranking quality, underscoring the promise of LLMs and other expert-informed rankers as practical complements to data-scarce regression models. The source code available at
2 Background
2.1 Combining Regression and Ranking
Huang et al. 2024] reformulate regression as joint pegression-pairwise--ee pairpise-rankingge objectives and show
that the combination improves predictive accuracy. [Tynes et al. [2021], |Falish et al. 2023] instead train a regressor to predict pairwise differences in molecular properties; at inference, query molecule is compared with reference molecules of known labels, and the mean of the predicted differences yields the final estimate. The post hoc method proposed by Gonçalyes et al. [2023) refines regression outputs by computing weighted average of the original prediction and those of its top-k nearest neighbors.
A closely related approach is the consolidation method for document relevance proposed by Yan ctt al. 2024]. Given a pointwise rating and a set of pairwise comparisons, their post-processor makes minimal adjustments to preserve the ranker-derived inequalities. For one-dimensional rating á this projection simplifies to
max(min(.. U). L),
where L and U are the lower and upper bounds of the feasible interval.
Preference learning with Gaussian processes [Chu and Ghahramani] 2005 Houlsby et al. 2012 Chau al. 2022 utilize pairwise data but differ fundamentally from our setting. Preference learning assumes latent score fzz) whose absolute scale is irrelevant; the aim is to preserve ordering and learn-to-rank, not to predict calibrated values. In contrast, we target real-world quantities. e.g., solubility or toxicity, and treat pairwise comparisons as auxiliary signals that refine supervised regression into meaningful, calibrated estimates. Similarly, many learning-to-ran approaches (Burges et al. 2005, Yildiz et al. 2020 Tom et al., 2024) may seem related to our work, but they actually address different problem, i.e., ranking rather than regression.
2.2 LLM as Pairwise Ranker
Large language models (LLMs) have recently been explored in molecular- prediction and generation [Sun et al.)
2025 Several efforts fine-tune GPT-33 LLaMA-based models with
chemistry-specifie prompts Jablonka et al.] 2024, Xie et al., 2024 Jacobs et al.) [2024]. while MolecularGPT (Liu et al. 2004
instruction- tunned I 000 property-predicee tasks and achieves few-shot performance competitive with graph neural networks.
Fine-tuning, however, is resource-intensive and incurs substantial computational overhead. We therefore ask: can general-purpose, publicly hosted LLMs be leveraged directly, without domain- specific adaptation? We hypothesize that such models can supply useful signals when deployed as pairwise rankers.
Human cognition offers an analogue. People often judge quantities either by estimating absolute magnitudes (regression) or by deciding which of two items greater (ranking). A long-standing hypothesis holds that comparative judgments are easier and less biased [Thurstone] 1927 Miller 1956 Stewart et al., 2005). Pairwise tasks impose lower cognitive load (Routh et al. 2023 and avoid scale-interpretation bias HHoejmakers et al. 2024). Because LLMS are pretrained on corpora rich
in relative statements and later aligned with human feedback (Zhang et al., 2023 Bai al. 2022
Rafailov et al., 2023 they may inherit similar advantage. As real example, Guo et al. 2023

---

training phase (only regressor)
inference...
external ranker
query
reference
rankings
general dataset
pretrain
regressor
(optional)
uncertainties
domain-specific training set
regressor
predictions
rank-based estimator
train
regressor
inverse variance weighting
refined prediction
Overview of the RankRefine framework. During training, regressor is trained using labeled data. At inference, a query sample is paired with reference samples with known properties and compared by an external ranker produce pairwise rankings. These rankings are used estimate the query's ss property value via rank-based estimator, The final prediction is obtained by fusing the regressor's output and the rank-based estimate using inverse variance weighting.
Figure I:
show that off-the-shelf LLMS rival domain domain-specific models on chemistry classification and ranking benchmarks, underscoring the promise of pairwise prompting in specialized tasks.
3 Method
Our goal is reduce the test-time error of an existing regressor by fusing its prediction with pairwise-ranking information in plug-and-play fashion. We present RankRef ine, framework that augments base regressor with rankings from an external source (e.g., an LLM or domain expert). RankRefimg ine is largely model-agnostic and applies to any regressor that returns both a point estimate and an uncertainty measure, for example Gaussian processes or random forests.
We assume access to a reference set D = {(I,, == with known property values. This reference can be drawn from the training set. At inference, the regressor predicts the property of query
sample Tu, giving reg while the ranker supplies pairwise comparisons between and each x; e ID.
3.1 RankRefine
Let Bo Nyo, Fieg) be the regressor's unbiased Gaussian estimate of the true property yo- A second, rank-based estimate 96nk is derived from the comparisons. Under these assumptions, the optimal fusion of the two estimates reduces to the inverse-variance weighting (Cochran and Carroll 1953], formalized below.
Theorem 3.1 (RankRefine Fusion Theorem) If to and grank are independent, unbiased Gaussian estimators of yo with variances Pryy and o2 Panko the minimum-variance unbiased estimator
pank 3n
(1 VeR
90
post
02
post
(1)
rig
Yank
See supplementary materials for the proof.
Rank- based estimate. Pairwise probabilities follow the Bradley-Terry model [Bradley and Terry 1952 P(x; > x.2) = s(yi M;) with sigmoid function s(2) = (1 - Indexing the query as
i = 0 and references as i > 0. we obtain grank by minimizing the negative log- ligeeiill
raak arg min Greak
Σ log (1ng
vi) - log(1 < uu))}
Uo
ZAEA
EB
where & contains references ranked below zo and B those ranked above.

---

Lemma 3.2 (Variance of the rank-based estimate) Let io minimize equation approximated by the inverse observed Fisher information (Ly et al. 2017/.
Its variance is
o
Σ
with A, 90
nank
(3)
BEAUR
Applying Theorem
3.1 with mank from Lemma 3.2
yields 06-
3.2 Analysis of RankRefine
measure performance via the mean absolute error (MAE), For folded Gaussian derived from zero-mean Gaussian, MAF 22/T nt 0, so
MAEpost post < MAEge
post
<
oreg
Corollary 3.2.1 Any informative ranker with finite variance (o 20) lowers the expected MAE after fusion.
More generally, letting a E [0, 1] be the desired ratio between post-refinement and the original MAEs,
a 0
MA S a MAE
rank
1 - a²
Regularization. the ranker is biased yet over-confident (o < Pegg) we temper its variance via
rank max o co
with user-chosen constant c > 0.
Experimental Results
We evaluate RankRef ine synthetic and real-world tasks. After outlining datasets, metrics, and implementation details, we report results in synthetic settings, where ranking oracle is available, and practical settings across multiple domains.
4.1 Experimental Setup
We use nine molecular datasets from the TDC ADME benchmark (Huang et al. 202 |: Caco-2
Waan al. 2016), Clearance Microsome and Clearance Hepatocyte Di et al. 2012 log Half-
Life [Obach et al.) 2008), FreeSolv MMobley and Guthrie 2014). Lipophilicity Wu et al. 2018
PPBR, Solubility (Sorkun et al. 2019]. and VDss Lombardo and Jing 2016 We additionally
test three tabular regressions: crop-yield prediction from sensor data [Soundankar 2025 student-
performance prediction [Cortez 2014), internationallezucatiion cost estimation (Shamim) 2025 In
the human-as-ranker experiment, we use UTTKFace [Zhang et al. 2017 for age estimation.
To emulate low-data regimes, sample 50 training points from above datasets uniformly at random and merge the remainder with the original test split, repeating this re-split over five random seeds.
The primary metric is the mean absolute error (MAE). We report the normalized error
MAE post
8
MAF
where 3 < 1 indicates improvement.
Suppose that a pairwise ranker R evaluates a pair of inputs 2,)) The ranker returns R(x,, 22y) if it predicts I; Ijj and R(2;, 2zy) otherwise. The pairwise ranker quality can be measured via pairwise ranking accuracy (PRA), defined as
PRA
Σ (KJ))P
T(R(,,))))) = T(ys < <3))),

---

Observed
MiEpot PAAF.
0.8
Ideal
MaEjor
0.6
0.4
0.2
Figure 2: Empirical validation of the theoretical MAE reduction bound under ideal conditions. For each target improvement factor 3, we compute the required ranker variance o using the right-side inequality in Implication 4 We simulate predictions from an oracle regressor (o70g (yo, 1)) and an oracle ranker (77 ~ NNoo. )ann)) and compute the fused estimate. The observed post- refinement MAE ratios match the ideal values, demonstrating the correctness of the fusion rule under Gaussian distribution assumptions.
where P= ((i,)) is the set of comparable pairs with ground-truth labels yis Mj. and T is the indicator function which return 1 if the condition is true, and () otherwise. Unless stated otherwise, the base model is random-forest regressor from scikit-learn [Pedregosa al. 2011] with default hyper-parameters, executed on single CPU.
4.2 Verifying Theoretical MAE Reduction Under Ideal Conditions
To validate Implication we simulate unbiased oracle predictions with known variances. For target factors 8 {0.10, 2.0,0.02...... 0.90, 0.99), the required ranker variance Tank is obtained from <feg the right-side inequality of Implication 4 Synthetic estimates are drawn as ~ N(yo 1) and
ignnk ~ (30, ). The fused prediction ø follows Equation Figure confirms that empirical MAE ratios match the prescribed targets.
4.3 Effect of Ranker Accuracy and Reference Sample Size on Molecular-Property Prediction with RankRefine
Using the nine TDC datasets, we vary oracle ranker accuracy and the number k of reference com- parisons per query. plots 3, the post refinement MAE error divided by the regression error, versus pairwise ranker accuracy. RankRef ine consistently lowers error, even with accuracy 0.55 and A: 10. Increasing the number of pairwise comparisons & tend to further lower 3, We observe that k = 20 optimal, as the difference between & = 20 and 30 is often minimal.
For very accurate rankers (>> 0.95) a slight uptick in 3 arises from over-confident curvature estimates and extreme solutions when the query is an outlier. For the first source, recall that of approximated by the second derivative of the negative log- likellodd This curvature becomes smaller as the ranker accuracy approaches perfection, leading over-confident estimates. For the second source, consider the case when the query is truly the smallest or largest value among the reference set. This is especially common when is small and ranker is accurate. The optimizer may push gaank toward extreme values that still satisfy pairwise ranking constraints, but overshoot the true property value yo-
4.4 Cross-Domain Generalization
Applying the same protocol to the three tabular datasets yields Figure whose reductions mirror those in chemistry datasets. The results show that RankRefine ine generalizes beyond molecular tasks.
5

---

non-efined
k-10
11
1.15
K-20
km5
km30
1.10
1.10
0.9
1.05
1.05
0.88
1.00
1.00
0.95
0.95
0.6
0.90
0.90
0.85
0.85
0.5
0.6
0.8
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
Banker Accuracy
Rariker Accuracy
Ranker Accuracy
(a) Caco2
(b) Clearance Microsome
Clearance Hepatocyte
1.2
0.9
0.8
0.9
0.7
0.6
1.0
0.5
Ranker Accuracy
Ranker
Ranier
(d) Half Life
(c) FreeSolv
(f) Lipophilicity
1.20
1.20
1.15
1.15
1.1
1.10
1.30
1.05
1.0
1.05
1.00
1.00
0.95
0.9
0.95
0.90
0.90
0.8
0.85
0.85
0.80
0.7
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.8 Ranker Accuracy
0.7
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Ranker Accuracy
Ranker Accuracy
(g) PPBR
(h) Solubility
(i) VDss
Figure 3: Performance of RankRefine molecular property prediction datasets under varying oracle ranker accuracy and number of reference comparisons. Each plot shows the normalized
eиror 8 MAE MAF as function of ranker accuracy, averaged over 5 random splits. B < 1 indicates the base regressor. is the number of pairwise comparisons for each test molecule. Shaded regions indicate standard deviation. Dashed gray line shows baseline MAE with no refinement. Across configurations, RankRefine improves regression performance when using a ranker with accuracy as low as 0.55. Increasing l tends to lower 3, but typically the benefits start diminish beyond 20.
improvements in regression performance

---

1.2
1.2
1.20
ron-efined
k-10
1.1
1.15
k=20
1.1
1.10
k=5
k=30
km
1.05
1.00
0.7
0.95
0.90
0.85
0.80
0.7
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.8 Reniker Accuracy
0.7
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Panker Accuracy
Rankeer Accuracy
(a) International Education Cost
(b) Smart Farming
(c) Student Performance
Figure 4: Performance of RankRefine tabular datasets under varying oracle ranker accuracy and number of reference comparisons. Across most configurations, RankRef ine improves regression performance when using ranker with accuracy low 0.55.
0.6
PPBR
Half Life
0.0
H.
FreeSolv.
Clr.
Lipo. Solubility
Paurs Pprojecionn
0.4
Caco2
VDss
0.2
0.2
PPBR
Half Life
0.0
Clr. H.
FreeSolv.
-0.4
Clr. M.
Lipo.
Caco2
Solubility
VDss
0..4
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Ranker Accuracy
Ranker Accuracy
(a)
(b)
Figure (a) Comparison between RankRefine the constrained optimization (projection)- refinement method of Yan et al. [024] molecular property prediction tasks. We report the difference in normalized error, Bours projection negative value indicates better performance by RankRefine. Each curve corresponds when the ranker accuracy is between 0.5 and 0.95. (b) We also compare our method a related post
dataset. RankRefine generally outperforms the bascline
hoc regression improvement method, regression re-ranking [Gonçalves et al. 2023)
4.5 Comparison with Other Baselines
We compare against two post hoc refinement methods: constrained optimization Yan et 2024 and regression by re-ranking (RbR) [Gonçalves et al. 2023]. We reimplemented the two baselines from scratch, as no publicly available code was provided. We use it 30 comparisons. Figure displays Bours Pprojection and Pours Preranking; negative values favour RankRef ine. Our approach generally outperforms RbR all datasets. Against the projection-based baseline, RankRefine excels when ranker accuracy lies between 0.50 and 0.95; projection dominates only when the ranker is nearly perfect. This because, when all pairwise rankings are correct, the projection method strictly enforces the constraints without increasing the absolute error. In such cases, the MAE will either remain unchanged or improved, with the improvements depend primarily on the distribution and resolution of the known labels. Yet, such near-perfect rankers are rare in real-world scenarios. Thus, robust performance of RankRefine with moderately-accuraa rankers offers more practicality, More comparisons with projection-based refinement can be found in the supplementary materials.
4.6 Few-Shot Molecular-Property Prediction with LLMs
To evaluate the practical applicability RankRefine realistic few-shot settings, we replace the oracle ranker with ChatGPT-4o [OpenAl 2025), large language model (LLM). Note that we can

---

Table 1: PRA and resulting RankRefine performance using ChatGPT-44 as the ranker on six datasets from the TDC ADME benchmark. For each test molecule, 20 training molecules are randomly sam- pled to form pairwise comparisons. Values are averaged over random splits. Despite moderate pairwise accuracy (~0.600-069)) RankRefing consistently improves regression performance, demon- strating the viability of LLMM-based rankers in low-data domains.
Dataset
Lipophilicity
Solubility
VDss
Pairwise Ranking Accuracy
0.622 = 0.008 0.957 + 0.012
0.693 = 0,035
0,605 = 0.010
0.934 + 0.048
0.895 0.053
Dataset
Caco2
Half Life
FrecSolv
Pairwise Ranking Accuracy
0.660 = 0.013
0.602 = 0.014
0.681 0.050
3
0.970 = 0.027
0.971 = 0.005
0.937 + 0.012
Table 2: Performance BankRefine when using human participants as both direct estimators and pairwise rankers a facial age estimation task. The base emror (MAE,cg) corresponds to the raw human estimate error. Each test face is compared to 15 reference faces via human-judged pairwise comparisons. RankRef ine improves prediction accuracy by refining each estimate based on human- provided rankings.
MAEreg
Pairwise Ranking Acc.
8
6.343 0.610
0.759 = 0.052
0.954 + 0.046
still evaluate the ranking accuracy of the LLM through oracle for the purpose of analysis. For each test molecule, we randomly sample 20 reference molecules from the training set and ask ChatGPT-40 to perform pairwise comparisons (e.g.. "Which molecule is likely to have higher solubility?"). The model is queried using textual molecular descriptions, SMILES [Weininger 1988. The details the prompt is provided in the supplementary materials.
Table I summarizes ChatGPT-4o ranking accuracy and the resulting on six TDC datasets. Even moderate accuracies (x=0.62-0.99) yield tangible MAE gains, underscoring both the robustness and practicality of our method.
One natural concern when employing pretrained LLM as a ranker is whether it simply memorizes from training data which item (e.g., a molecule) has higher lower property value (e.g.. solubility) compared to others. To test this, we conducted an additional evaluation on a private compound- activity dataset consisting of 75 compounds with IC50 labels measured against specific target protein complex. This dataset is unpublished, and therefore, not part of any L.MM training corpora, ruling out direct memorization. On this dataset, ChaiGPT-4o achieves 60% pairwise ranking accuracy, indicating generalization capability and not simple memorization.
4.7 Human Age Estimation with Pairwise Self-Correction
BankRefine can potentially be used in interactive, human-in-the-loop scenarios, where domain experts act as the pairwise rankers. As a proof of concept, we conduct a user study on the age estimation task. Details about the user study is available in the supplementary materials.
Table reports a UTKFace user study where participants are asked to: (1) estimate the age of 17 individuals from their corresponding pictures, and (2) perform pairwise judgments to 15 individuals in the reference set. Pairwise judgments raise ranking accuracy to 0.76 and cut MAE by R55 %, illustrating RankRefine" value for expert self-correction.
8

---

4.8 Effects of Noisy Variance Estimates
We study the effect of noise in the ranker variance estimate the Solubility dataset using A: = 30 and k == 40. We add uniform perturbations sampled from [-b, +b] to the estimated ranker variance before fusion. As shown in Figure 6 performance degradation becomes noticeable when b z which is more than three times the standard deviation of the ranker variance estimates. For reference, the mean and standard deviation of the estimated ranker variance are 2.358 1.079.
1.25
0.01
1.0
b 0.05
b 5.0
1.30
b= 0.1
b 10.0 non-perturbed
b 0.5
0.35
0.3
83 0.90
80
0.90
0.35
0.93
0.99
0.80
0.72
0.25
2.00
Ranker Accuracy
Ranker Accuracy
(a) k 30
(b) k = 40
Figure 6: Impact of noisy ranker variance estimate on the regression performance in Solubility dataset. Uniform noise from [-b, +b] is added to the estimated ranker variance. Performance starts to degrade when b 5, more than times the standard deviation of the ranker variance estimate.
4.9 Effects of Biased Regressor
We systematically inject constant bias into an otherwise unbiased regressor. Specifically, we uniformly draw ground-truth labels yo (O, 1), and set predictions by injecting zero-mean Gaussian noise Vo = yo + such that MAE 2 ISD where SD is the standard deviation of the ground-truth labels. We then add constant offset, B, increasing in magnitude from 0% to 70% of the label standard deviation (SD). We report the refined-to to-original MAE ratio, 3, for varying ranker accuracies. Table 3 in Appendix D shows that 3 worsens as bias increases, but remains < up B 60% SD, indicating that RankRefine remains effective even under substantial regressor bias.
4.10 Effects of Biased Sampling
We study the effects of non-unin non-uniform sampling of references, particular, when the references are clustered. Similar to the biased regressor experiment, we generate synthetic predictions with ground-truth label range [0, 1]. We then restrict the reference labels to increasingly narrow subranges of the ground-truth label range, centered around the midpoint of the range. The reference set bias, RB = =%, means that the reference labels cover only (100 x) of the full ground-truth range. For example, an RB 10% means the reference labels range is [0.05, 0.95). Table in Appendix suggests that RankRefine performance best when the reference set spans the full ground-trute labels' range (i.c., RB 0%). but clustered sets still yield substantial gains. For example, at RB 90% and ranker accuracy = 60%, 3 0.8842, an 11.58% improvement.
Furthermore, we evaluate biased sampling in two challenging settings: (i) disjoint: reference labels are [-100] or [1,2] while queries are in [0. and (ii) partial overlap: reference labels are [-0.5, 0.5] or [0.5, 1.5], while queries are in [0,1]. This simulates distribution shift, where the distribution of the reference labels is different from the distribution the query labels. Despite the challenging
setups, RankRefine improves the base regressor when the ranker is moderately accurate (i.e., 265% in the disjoint case, 255% the partial overlap case). More details are available Table 5 in Appendix E.

---

5 Limitations
While RankRef ine demonstrates strong empirical performance, it has number of limitations:
Our theoretical analysis assumes that both the regressor and rank-based estimates have unbiased, Gauusian-distribune errors and are independent. Despite the empirical robustness of our method, its assumptions may not hold in practice, particularly in the presence of heavy-tailed or skewed noise.
RankRef ine depends on well-calibrated uncertainty estimates from the regressor and a trustworthy estimate of ranker variance (via Equation Miscalibration in either component can reduce or even negate the performance gains.
Our oracle experiments assume uniformly random ranking errors; however, real-world rankers may exhibit systematic biases, for example consistently failing on extreme values which can skew the fusion process.
RankRef ine uses the Bradley-Terry model by treating true property values proxies for latent scores, which differs from its original use where latent scores are learned to explain probabilistic pairwise outcomes. While practical, this introduces modeling mismatch. A possible solution is to learn a mapping from standard Bradley- Terry latent scores to labels holdout set and apply it during inference
6
Future Work
Future work can extend RankRefine in several promising directions. One avenue is to replace the Bradley-Terry likelihood with richer Aoohastii-transsitvvie models that better capture structured or systematic errors in the ranker. Another important direction involves incorporating Bayesian or conformal calibration layers to automatically correct misestimated variances and improve reliability, Beyond scalar predictions, RankRefine could potentially be extended to handle multivariate or structured targets, such as full pharmacokinetic profiles. Finally, leveraging rationale from language models could improve the interpretability the ranking process and facilitate richer forms expert feedback in decision-critical domains.
7 Conclusion
introduced RankRefine, plug-and-play, post hoc framework that injects pairwise-ranking signals into any uncertainty-aware regressor. Theoretically, we proved that svvers-vvrriace fusion with a rank-based estimator lowers the expected mean absolute eгrоr (MAE) whenever the ranker variance is finite. Empirically, oracle simulations generally follows the theory, and experiments benchmarks plus three diverse tabular tasks showed consistent improvements
nine molccular- with ranker accuracies of around 55% and few 10 comparisons. ChatGPT-44 rankings yielded measurable MAE drops on six ADME datasets, and user study demonstrated similar self- correction for human age estimation. Compared with a projection baseline, RankRefine prevailed whenever ranker accuracy lay in the realistic 0.50-0.95 range, underscoring its robustness and broad applicability.
References
Michael Sun, Minghao Guo, Weize Yuan, Veronika Thost, Crystal Elaine Owens, Aristotle Franklin Grosz, Sharvaa Selvan, Katelyn Zhou, Hassan Mohiuddin, Benjamin Pedretti, al. Representing molecules as random walks over interpretable grammars. In International Conference on Machine Learning, pages 46988 47016. PMLR, 2024.
Michael Sun, Gang Liu, Weize Yuan, Wojeiech Matusik, and Jie Chen. Foundation molecular grammar: Multi-modal foundation models induce interpretable molecular graph languages. In International Conference Machine Learning. PMLR, 2025.
Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36:59662-59688 2023.
10

---

Pin- Yen Huang, Szu-We Fu, and Yu Tsao. Rankup: Boosting semi semi-sipe regression with an auxiliary ranking classifier. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL wtt.tttt:s//oprreiiiiiiiii wttp://wwwwe///ooummddd/d-2----e
Michael Tynes, Wenhao Gao, Daniel J Burrill, Enrique R Batista, Danny Perez, Ping Yang, and Nicholas Lubbers. Pairwise difference regression: a machine learning meta- aegga--lgrithmmm for improved prediction and uncertainty quantification in chemical search. Journal of chemical information and modeling, 61(8):3846 -8588466-877,, 2021.
Zachary Fralish, Ashley Chen, Paul Skaluba, and Daniel Reker. Deepdelta: predicting admet improvements of molecular derivatives with deep learning. Journal of cheminformatics, 15(1):101, 2023.
Filipe Marcel Fernandes Gonçalves, Daniel Carlos Guimarães Pedronette, and Ricardo da Silva Torres. Regression by re-ranking, Pattern Recognition, 140: 109577, 2023.
Le Yan, Zhen Qin, Honglei Zhuang, Rolf Jagerman, Xuanhui Wang, Michael Bendersky, and Harrie Oosterhuis. Consolidating ranking and relevance predictions of large language models through post- processing. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 410-423, 2024.
Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the 22nd international conference on Muchine learning, pages 137-144, 2005.
Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Jose Hernández-lobato. Collaborative gaussian processes for preference learning. Advances in neural information processing systems, 25, 2012.
Siu Lun Chau, Javier Gonzalez, and Dino Sejdinovic. Learning inconsistent preferences with gaussian processes. In International Conference Artificial Intelligence and Statistics, pages 2266-2281. PMLR, 2022.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89-96, 2005.
Ilkay Yildiz, Jennifer Dy, Deniz Erdogmus, Jayashree Kalpathy-Cramer, Susan Ostmo, J Peter Campbell, Michael F Chiang, and Stratis Ioannidis. Fast and accurate ranking regression. In International Conference on Artificial Intelligence and Statistics, pages 77-88. PMLR, 2020.
Gary Tom, Stanley Lo, Samantha Corapi, Alan Aspuru-Guzik, Guzik, and Benjamin Sanchez-Lengelingg Ranking over regression for bayesian optimization and molecule selection. arXiv preprint arXiv:2410.09290, 2024.
Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrere and Berend Smit. Leveraging large language models for predictive chemistry. Nature Machine Intelligence, 6(2):161-169, 2024.
Zikai Xie, Xenophon Evangelopoulos, Ömer H Omar, Alessandro Troisi, Andrew I Cooper, and Linjiang Chen. Fine-tuning gpt-3 for machine learning electronic and functional properties of organic molecules. Chemical science, 15(2):500-55 51000, 2024.
Ryan Jacobs, Maciej P Polak, Lane E Schultz, Hamed Mahdavi, Vasant Honavar, and Dane Morgan. Regression with large language models for materials and molecular property prediction. arXiv preprint arXiv:2409. 06080, 2024.
Yuyan Liu, Sirui Ding, Sheng Zhou, Wenqi Fan, and Qiaoyu Tan. Moleculargpt: Open large language model (Ilm) for few-shot molecular property prediction. arXiv preprint arXiv:2406. 12950, 2024.
LL Thurstone. A law of comparative judgment. Psychological Review, 34(4), 1927.
George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2)811 1956.
Neil Stewart, Gordon DA Brown, and Nick Chater. Absolute identification by relative judgment. Psychological review, 118( 2005.

---

Jennifer Routh, Sharmini Julita Paramasivam, Peter Cockcroft, Sarah Wood, John Remnant, Cornélie Westermann, Alison Reid, Patricia Pawson, Sheena Warman, Vishna Devi Nadarajah, et al. Rating and ranking preparedness characteristics important for veterinary workplace clinical training: novel application of pairwise comparisons and the elo algorithm. Frontiers in Medicine, 10: 1128058, 2023.
Eva Hoeijmakers, Bibi Martens, Babs MF Hendriks, Casper Mihl, Razvan L Miclea, Walter H Backes, Joachim E Wildberger, Frank M Zijta, Hester A Gietema, Patricia J Nelemans, et al. How subjective ct image quality assessment becomes surprisingly reliable: pairwise comparisons instead of likert scale. European Radiology, 34(7):494-4503. 2024.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308. 10792, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint N.Kiv:2204.058662 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 01553788-537444 2023.
William G Cochran and Sarah Porter Carroll. A sampling investigation of the efficiency of weighting inversely as the estimated variance. Biometrics, 9(4::447-459 1953.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4))324-345, 1952.
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagenmakers. A tutorial on fisher information. Journal of Mathematical Psychology, 80:40 55, 2017.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Advances in neural information processing systems, 2021.
Wing-Ning Wang, Jie Dong, Yin-Hua Deng, Min Min-Feng Zhu, Ming Wen, Zhi-Jiang Yao, Ai-Ping Lu, Jian-Bing Wang. and Dong-Sheng Cao. Adme properties evaluation drug discovery: prediction of caco-2 cell permeability using combination of nsga-ii and boosting. Journal of chemical information and modeling, 56(4):763-773 2016.
Li Di, Christopher Keefer, Dennis O Scott, Timothy J Strelevitz, George Chang, Yi-An Bi, Yurong Lai, Jonathon Duckworth, Katherine Fenner, Matthew D Troutman, et al. Mechanistic insights from comparing intrinsic clearance values between human liver microsomes and hepatocytes to guide drug design. European journal of medicinal chemistry, 57:441 448, 2012.
Scott Obach, Franco Lombardo, and Nigel J Waters. Trend analysis of database of intravenous pharmacokinetic parameters in humans for 670 drug compounds. Drug Metabolism and Disposi- fion, 36(7))1385-1 1405, 2008
David L Mobley and Peter Guthrie. Freesolv: database of experimental and calculated hydration free energies, with input files. Journal of computer-aided molecular design, 28:711-720, 2014.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9(2):513-534 530, 2018
Murat Cihan Sorkun, Abhishek Khetan, and Süleyman Er. Aqsoldb, a curated reference set of aqueous solubility and 2d descriptors for a diverse set of compounds. Scientific data, 6(1):143, 2019.
12

---

Franco Lombardo and Yankang Jing. In silico prediction of volume of distribution in humans. extensive data set and the exploration of linear and nonlinear methods coupled with molecular interaction fields descriptors. Journal of chemical information and modeling, 56(10):004222552 2016.
Atharva 2025.
Soundankar.
Smart farming
sensor data for yield prediction,
URL htttps:/www.kagggt wwtp///attsttteeaaaaaaaaaaaaaaaa
farrtf-frrnnn---ee ceeesor-dtta-aae data-for-yiieeee ptweeididrriiiiiionn
Paulo Cortez.
Student Performance. UCI Machine Learning Repository, 2014.
Adil Shamim. Cost of international education, 2025.
wtw.g//ww.....aaaeeeeeeeeeeeeeee htt.tttss.//w....aggeeooo....a adilshamim//com wwwpa///ooooooeeeeeeeeeeeeeeeee of- international intermatiionl--eduaaaatttt -duuaain
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/reeression by conditional adversarial autoencoder. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5810-5818, 2017.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit-learn Machine learning in python. the Journal of machine Learning research, 12:2825-2830 2011.
OpenAI. Chatgpt-4o, 2025. URL.attps:/Cchatpt..cm
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflects the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in separate section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
13

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions and proofs provided in the Method section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Implementation details are provided in the Experimental Results section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
14

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code will be shared to the public upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/
http://dddee//deemmiiiiiia CCodeSoodieeeeeee ssi mollicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
15

---

Answer: [Yes]
Justification: Training and test details are described in the Experimental Results section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The experimental results are supplemented with appropriate statistical infor-
mation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources information described the Experimental Results section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
16

---

9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work studies a generic post-hoc method for improving the performance of a regressor using pairwise ranking predictions.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: There is no high risk for misuse associated with this method.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
17

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Original owners of assets are property credited in the reference.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, papprssiiptheeeoooc thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The public code will be accompanied by documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Full text of instructions given participants and screenshots are provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There are no potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The usage of LLMs is described on multiple occasions in the manuscript and supplementary materials.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy htap:p////errppsccc/oooofoooeee for what should or should not be described.
19

---

A Extended Proof
A.1 RankRefing ine is minimum-variance unbiased estimator
Suppose that we have independent unbiased estimates of the same parameter you
reg
(Nu, Teg)):
yo
rank
(Bo, Tank).
and we want to form linear combination of the two estimates,
y = wij rey + (1 w w))00m..
(5)
goal is to choose E [0, 1] such that the variance is minimized.
For independent random variables X and Y, Var(aX + 6Y) a² + bey. The post-fusion variance of the unbiased estimates is then.
p w2o Feg + (1 o)
(6)
To minimize opout with respect to w, we take the derivative and solve for,
d (w Frg + (1 (1-w) ((ak)) O,
dw
2wo r0g - 2(1 w 0 rank
2wo 2
2(1 - w) o
2
wo rog
wo reg
o -w
wo reg
+ wo rank
2
o
w( w rog
+ o
rank-
Therefore,
w)
reg
(8)
2 Tank
7
Substituting the weights in Equation 6 with Equation variance in Equation
we obtain the RankRefing post-fusion
2
2
org
reg
2 rank
post
2 + 2
2
o rank
rank
(o 200
( Po
(o + o )2
02 oh (0 + ogg
(9)
(02 + o
02 oeg
2 + 0
2
cank
20

---

Similarly, substituting the weights in Equation 5 with Equation g we get the RankRef ine estimate Equation!
I
96
rank
yo
reg
reg
y0
grank
0 reg
+ o mink
Frg
o
rank
rank reg
o reg rank
2
(o + o rankk )o
Mo
reg
+
(a
0
rank Jo ) mank
mrank
(10)
0 post
reg
post
crank
2
Uo
o rank
o
1/0
reg
20
Grank
post
0
rank
A.2 Analysis of RankRefine
Corollary
3.2.
is obtained from:
posa
ong
reg!
2
rank
2
0
2
(11)
rank
reg
O,
02
rank
o 00.
For unbiased Gaussian estimates 9o N NNoo, 2²), the expected error is,
E(loo yol) E(Bo + vol), with e N((,
(12)
= E(le))
Since is Gaussian distribution, then |<| follows the haff-Gaussian distribution with an expected value equals to 22/TO. The subsequent bound in Equation is obtained from:
MAE Aps MAE
post I 3 Oreg
pma S 3 orgg
B2 o
02 rgg
Frank
1
(13)
02
0
rank
g2
o2
Teg
B²
g2
0 16g
rank
82
reg
rank
1- 32
B Additional Experimental Details
B.1 Prompt for obtaining pairwise rankings from ChatGPT-40
We use the following prompt to query ChatGPT-4o to predict the pairwise rankings for the Lipophilic-
ity dataset.
# Identity
21

---

You are an expert chemistry and biology. Given short description molecular property and two molecular SMILES, you can determine if Molecule A has greater property value than Molecule or not.
Instructions
The list of pairwise molecules is given in two CSV files: "test_A csv" and different lists that you need to perform rank prediction on. The first line is the header.
Bessst..csv' These
You cannot use external cheminformatics library directly predict the property. You can design your own heuristics, comparing atom types, bonds,
and other important information to make your predictions.
Before you start designing the heuristics, briefly explain the property
and what influence its values. Incorporate this prior knowledge into your heuristic.
You are given 5 examples in the prompt. Make sure your heuristics in general is aligned with the given examples. Pay attention to the examples, especially because some properties have greater effects when its values are lower. You should output CSV files titled "test_pred_A.ce csv" and "test_pred_B".
The header is "molecule_a, molecule_b, is_a_greater".
Only output 0 or for "is_a_greater", molecule is less than that of Molecule B. means that the property value of molecule is greater than that of Molecule In addition to the output CSV file, you should output the first and last molecule pairs, in both answer for these three molecule pairs.
means
the property value
and CSV B, the chat. Briefly explain your
# Examples
<user_query>
The property of interest is lipophilicity, the ability of a drug to dissolve in lipid (e.g. fats, oils) environment.
Examples:
( art (03ccc)c)c CC(C) CCC (c3ccccc3)) C3ccccc))CC))ccc.. olt.a//cccccccccoooooo.oooooooor
0=C (cCCCcccccccc11))cccccll]]]
O=C(Nc1ccc httpa//cweccccccaaoccceeeenoocan 0
out.ccccccceecccconnoooccccoocc, Ccicc
Ccceeceeeeccctcoooo-oooooooooooo c1,
Ce1ce
0=C htt.:
(C(cc [
</user_query>
22

---

inggr
Mecbeecke
O
(a)
Wes
O
O O
Figure 7: Partial interface used in the study. Participants first estimate the age of each test. individual, then make pairwise judgments on who appears older between the test individual and reference individuals of the same UTKeaaee gender label.
B.2
User Study
The user study uses age estimation on the UTKEaaee dataset as regression task, Six participants estimate the ages of 17 individuals from facial images. Each test individual is then paired with 15 disjoint reference individuals, and participants judge who appears older in each pair based solely on facial appearance. To prevent fatigue, the total number of comparisons is limited to 17 x 15 255. screenshot of the Streamlit app used in the study is shown in Figure 7
C More Comparisons with the Projection-based Approach
how performance varies between RankRefing ine and the projection- baased approach as the number of reference samples increases. While the projection-basee bbssed method benefits from high-accuracy rankers or minimal references, RankRef ine demonstrates stronger robustness in lower-accuracy regimes, particularly the number of reference samples k increases.
Figure highlights higghihhshhlights
D Ablation: Biased Regressor
We inject constant bias B into an otherwise unbiased regressor, where the magnitude of the offset measured as percentage the ground truth labels standard deviation. RankRefine offers improvement even when the regressor is biased (e.g., 60% of the SD of the ground truth labels)
Table 3: RankRefine performance, measured in 8 (lower is better), biased regressor.
Ranker Accuracy
50%
55%
60%
65%
70%
Bias (B) 0% SD
0.8911
0.8436
0.8694
0.8668
0.8949
10% SD
0.8995
0.8800
0.8623
0.8602
0.8969
20% SD
0.9084
0.8869
0.8704
0.8704
0.9047
30% SD
0.9224
0.8997
0.8844
0.8861
0.9192
B 40% SD
0.9409
0.9172
0.9059
0.9071
0.9395
50% SD
0.9645
0.9403
0.9329
0.9343
0.9654
B 60% SD
0.9942
0.9702
0.9649
0.9670
0.9967
70% SD
1.0291
1,0062
1.0022
1.0049
1.0326
23

---

0.25
PABR
Half Life
Prossote
0.5
PPBR
Ck.
Hor
FrocSol.
-0.25
0.00
0.3
Cano2
M
Lipo.
Solubiitty
0.4
Cl.
Carn2
Lipo.
Sclubiiity
0.50
VPss
VDee
-0.75
0.1
1.00
Lio.
0.1
1.25
ViDa
Salubiity
-0.1
-150
Parker Accunity
Banker Accunity
Ranker Accunccy
(b) k=
(c)
PPBR
Half
Half
PPBR
0.5
FreeSolv
FreeSatk
FreeSohv
0.4
0.2
Bun
0.
-0.1
-0.2
0.5
0.5
0.5
0.0
0.5
0.6
0.5 1.0 0.5
0.6
0.7
0.8
0.5
(d) k
(c)
(f) = 30
Figure 8: Comparison of RankRef ine and the projection-basee based approach across varying reference sample sizes The projection- references, while RankRef ine excels when accuracy is below 70% for A 7, 80% for A = 10, 90% for k 20, and 95% for k 30. Since RankRef ine can leverage general-purpont LLMs, collecting more pairwise rankings can be done efficiently as long as the reference set is sufficiently large.
method performs better with high ranker accuracy or very few
E Ablation: Biased Sampling
We restrict the reference set range to increasingly narrow subsets of the ground truth label range ([O, I]), simulating clustered or biased samples. The reference set bias, RB = x%, means that the reference labels cover only (100 x²) of the full ground-truth range. For example, an RB = 10% means the reference labels range is [0.05, 0.95]. Best performance is achieved when the reference set spans the entire range of the ground truth labels, but clustered reference sets still provide substantial regression improvements.
Table 4: RankRefine performance, measured in 3, with non-uniform sampling of the reference set.
Ranker Accuracy
50%
55%
60 %
65%
70%
Reference Set Bias (RB) 0% range
0.8828
0.8706
0.8343
0.8443
0.8598
RB 10% range
0.9034
0.8731
0.8493
0.8529
0.8768
RB = 20% range
0.9078
0,8753
0.8366
0,8694
0.9002
30% range
0.9076
0.8701
0.8439
0.8733
0.9280
RB 40% range
0.9143
0.8767
0.8618
0.8824
0.9414
RB 50% range
0.9253
0.8786
0.8640
0.8769
0.9480
RB 60% range
0.9349
0.8858
0.8712
0.8929
0.9546
70% range
0,9385
0.8919
0.8770
0.8957
0.9597
RB 80% range
0.9551
0.8984
0.8798
0.8943
0.9558
RB 90% range
0.9664
0.9042
0.8842
0.9035
0.9631
To test the impact of distribution shift between reference and query, we created two challenging scenarios where the reference labels and query labels are drawn uniformly from different range values:
Disjoint distributions: reference labels in [-1, 0] or [1, 2), query labels in [0, 1]. We averaged the results of the two cases, that is, between ref [-1, 0), query = [0, 1] and ref = [1, 2]. query = [0. 1].
24

---

Partial overlap: reference labels in [-0.5, 0.5] or [0.5, 1.5], query labels in [0, 1]. We averaged the results of the two cases, that is, between ref = [-0.5, 0.5], query = [0, and ref = [0.5, 1.5], query = [0. l].
The results in shows that RankRefine still improves the base regressor when the ranker moderately accurate, even when the distribution of the reference labels is different from the query.
Table 5: RankRefine performance, measured in 8, for disjoint and partially overlapping reference set.
Ranker Accuracy
50%
55% %
60%
65%
70%
Disjoint
1.2906
1.1629
1.0170
0.9437
0.9219
Partial Overlap
1.0160
0.9465
0.8700
0.8607
0.8898
25


Reviews:
[{"review_id": "zYEUu20eFT", "reviewer": "Reviewer_Wxfo", "summary": "This paper introduces RankRefine, a novel post hoc method for refining regression predictions by incorporating pairwise rankings. The approach is model-agnostic and does not require retraining the base regressor, making it highly versatile and practical, especially in low-data scenarios. Extensive experiments on multiple molecular property prediction benchmarks and diverse tabular datasets demonstrate consistent improvements in predictive performance across different settings.", "questions": "- 1.What is the optimal size of the reference set for different types of regression tasks and ranker accuracies? \n- 2.How does RankRefine compare with other ranking methods or models that could potentially be used as rankers in different domains?", "limitations": "The authors have already articulated several limitations within the paper. However, there are additional concerns that merit further consideration.\n\n- 1.The method involves comparing the query sample with a reference set to obtain pairwise rankings. As the size of the reference set increases, the computational cost of generating and processing these rankings may become a concern.\n- 2.The oracle experiments assume uniformly random ranking errors, but real-world rankers may exhibit systematic biases.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- 1.The paper introduces RankRefine, a novel post hoc method that refines regression predictions by incorporating pairwise rankings. This approach is model-agnostic and does not require retraining the base regressor, making it highly versatile and practical for various applications, especially in low-data scenarios.\n- 2.Extensive experiments on multiple molecular property prediction benchmarks and diverse tabular datasets demonstrate that RankRefine consistently improves predictive performance across different settings. \n\n**Weaknesses**\n- 1.The theoretical analysis relies on several assumptions, such as the regressor and rank-based estimates having unbiased, Gaussian-distributed errors and being independent. These assumptions may not hold in real-world scenarios, especially when dealing with heavy-tailed or skewed noise.\n- 2.The paper mainly focuses on using LLMs as rankers but does not explore other potential types of rankers in detail. There could be other ranking approaches or models.", "was_revised": true, "final_justification": "The authors provide detailed rebuttal and additional experiments. My concerns have been addressed. I will keep my positive score.", "rebuttal": "We thank the reviewer for their thoughtful assessment and constructive feedback. We appreciate the recognition of RankRefine's novelty, versatility, and empirical strength across diverse benchmarks. Below, we address each of the reviewer’s concerns in detail and hope our clarifications and new experimental results strengthen your confidence in our work. We believe RankRefine offers a practical and broadly applicable contribution, especially in settings where ranking information is more accessible than precise regression labels.\n\n**Q1: The method relies on some assumptions such as unbiased regressor and may not hold in real-world scenarios.**\n\nWe agree that this is an important, practical concern. Our experiments on the TDC ADMET datasets (Figure 3 in the main paper) demonstrate that RankRefine works in real-world scenarios. We have also conducted new experiments where we systematically inject **constant bias** into an otherwise **unbiased regressor**. Specifically:\n\n1. We start from synthetic predictions (arbitrary labels injected with random noise) with zero-mean Gaussian noise, yielding a mean absolute error (MAE) close to one standard deviation of the ground truth labels.\n2. We then add a constant offset, increasing in magnitude from **0% to 70% of the label standard deviation (SD)**.\n3. We report the **refined-to-original MAE ratio** (β = $\\frac{MAE_\\text{refined}}{MAE_\\text{non-refined}}$, lower is better) for varying ranker accuracies.\n\nThe results (see Table 1) demonstrate that **RankRefine remains effective even under substantial regressor bias**. For example, with a 60% bias and only 60% ranker accuracy, RankRefine achieves β = 0.9649, i.e., a 3.51% gain.\n\nTable 1. RankRefine performance, measured in β, on a biased regressor. We inject constant bias into an otherwise unbiased regressor, where the magnitude of the offset is measured as a percentage of the ground truth labels standard deviation. RankRefine offers improvement even when the regressor is significantly biased (e.g., 60% of the SD of the ground truth labels)\n\n| **Ranker Accuracy**   | **50%** | **55%** | **60%** | **65%** | **70%** |\n|------------------------|---------|---------|---------|---------|---------|\n| **_Bias (B) = 0% SD_** | 0.8911  | 0.8436  | 0.8694  | 0.8668  | 0.8949  |\n| **_B = 10% SD_**       | 0.8995  | 0.8800  | 0.8623  | 0.8602  | 0.8969  |\n| **_B = 20% SD_**       | 0.9084  | 0.8869  | 0.8704  | 0.8704  | 0.9047  |\n| **_B = 30% SD_**       | 0.9224  | 0.8997  | 0.8844  | 0.8861  | 0.9192  |\n| **_B = 40% SD_**       | 0.9409  | 0.9172  | 0.9059  | 0.9071  | 0.9395  |\n| **_B = 50% SD_**       | 0.9645  | 0.9403  | 0.9329  | 0.9343  | 0.9654  |\n| **_B = 60% SD_**       | 0.9942  | 0.9702  | 0.9649  | 0.9670  | 0.9967  |\n| **_B = 70% SD_**       | 1.0291  | 1.0062  | 1.0022  | 1.0049  | 1.0326  |\n\n**Q2: How does RankRefine compare with other ranking methods or models that could potentially be used as rankers in different domains?**\n\nThank you for this suggestion. While RankRefine is agnostic to the choice of ranker, we recognize the value of exploring different types. To that end, we experimented with a Random Forest (RF) classifier as the ranker on the Caco2 dataset (TDC ADMET benchmark). Despite being a simple model, it achieves **75.78% pairwise ranking accuracy**.\n\nThis highlights the **flexibility of RankRefine**, it can leverage any ranker (few-shot LLMs, learnt classifiers, domain-specific heuristics), making it adaptable to different domains. \n\n**Q3: What is the optimal size of the reference set?**\n\nThank you for raising this point. As shown in Figure 3 (main paper), RankRefine’s performance improves as reference set size increases, up to a point. We observe that performance saturates around 20 samples, with diminishing gains beyond that (e.g., 30 samples).\nThis suggests that RankRefine is not only effective but also **efficient**, requiring only a small number of reference samples to yield performance improvements.\n\n**Additional Comments**\n\nWe also note the reviewer’s concern about theoretical assumptions (e.g., Gaussian errors, independence). While these simplify analysis, we validate RankRefine extensively in real-world settings (e.g., molecular property prediction, tabular data) where such assumptions do not strictly hold. The consistent empirical improvements across diverse tasks suggest that RankRefine is **robust beyond the theory’s assumptions**.\n\nOn the concern of ranking cost, we note that:\n- Pairwise comparisons can be parallelized.\n- Smaller reference sets (≤ 20) are sufficient for substantial improvements in most tasks.\n- RankRefine only requires inference-time computation, no retraining is involved, preserving efficiency."}, {"review_id": "RR8OMx3eC6", "reviewer": "Reviewer_UF75", "summary": "This paper proposes RankRefine: a method for improving the estimates of uncertainty-aware regressors, for scientific tasks for example, by utilizing pairwise rankings generated either by LLMs or humans in the loop. Given a query, RankRefine first gets an estimate of the mean and variance using a regressor. Then, the query is ranked against other known references by asking either a human expert or LLM. These rankings are then used to generate a rank based estimate of the mean and variance. Finally, these two estimates are combined with inverse variance weighting to produce an improved estimate with lower variance. Under some assumptions, the authors show that this improved estimate will have lower mean absolute error. Experiments show that these improvements hold up in practice.", "questions": "1. How bad is a biased regressor?\n\nIn many practical applications I can imagine that the regressor might systematically over or under predict by a constant factor. If this was the case, how would the performance of the proposed estimator deteriorate? Could potentially make a table with the performance as a function of different degrees of over or under prediction.\n\n2. Does the LLM simply memorize some of the ranking answers?\n\nIt seems plausible that the LLM might simply memorize whether a particular molecule is more or less soluble than another, as an example. Is there evidence that the LLM can successfully reason through the question to come to the right conclusion? I'll leave my recommendation at borderline pending the authors response to these first 2 questions.\n\n3. Have you considered non-uniform sampling of the reference points?\n\nIt seems the reference points are sampled uniformly randomly from the labeled data. Could the variance of the rank-based estimate be reduced by a smarter choice of references? Intuitively I would imagine that references that aren't clustered together would be better.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The quality of this work is excellent. The figures are clear, all claims are justified with assumptions stated, and the authors are honest about the limitations of their work. That said, I do feel that there are a lot of limitations, in particular the assumption that the regressor is unbiased. It would be nice to see a bit more detail on this.\n\nThe clarity is also excellent. I had no issues understanding the method and feel like enough detail was provided to implement it myself.\n\nI would rate the significance as fair. The experimental results show improvements when using RankRefine, however this involves 20 pairwise rankings per query, which could be expensive for a human in practice. The idea of using an LLM is promising and shows some good results, although its not clear how much of this is due to memorization of the LLM's training data.\n\nThe originality is good. Previous work has explored using pairwise rankings to generate predictions, and LLMs have shown some promise in answering scientific ranking questions. Combining these two ideas to create a more accurate estimate is fairly novel.", "was_revised": true, "final_justification": "The authors have addressed my main concerns in their rebuttal. The additional ablation demonstrates that their method maintains performance improvements even with a biased estimator. The use of a proprietary dataset mitigates concerns about GPT memorization, and the authors have also provided satisfactory responses to my minor questions. For these reasons, I am increasing my recommendation to accept.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We are encouraged by the positive remarks on the quality, clarity, and originality of our submission. Below, we address each of the reviewer’s concerns in detail and provide new empirical results to support our claims.\n\n**Q1: How bad is a biased regressor? In many practical applications, the regressor may systematically under or overestimate.**\n\nWe agree that this is an important, practical concern. Our experiments on the TDC ADMET datasets (Figure 3 in the main paper) demonstrate that RankRefine works in real-world scenarios. We have also conducted new experiments where we systematically inject **constant bias** into an otherwise **unbiased regressor**. Specifically:\n\n1. We start from synthetic predictions (arbitrary labels injected with random noise) with zero-mean Gaussian noise, yielding a mean absolute error (MAE) close to one standard deviation of the ground truth labels.\n2. We then add a constant offset, increasing in magnitude from **0% to 70% of the label standard deviation (SD)**.\n3. We report the refined-to-original MAE ratio (β = $\\frac{MAE_\\text{refined}}{MAE_\\text{non-refined}}$, lower is better) for varying ranker accuracies.\n\nThe results (see Table 1) demonstrate that RankRefine **remains effective even under substantial regressor bias**. For example, with a 60% bias and only 60% ranker accuracy, RankRefine achieves β = 0.9649, i.e., a 3.51% gain.\n\n**Table 1**. RankRefine performance, measured in β, on a biased regressor. We inject constant bias into an otherwise unbiased regressor, where the magnitude of the offset is measured as a percentage of the ground truth labels standard deviation. RankRefine offers improvement even when the regressor is significantly biased (e.g., 60% of the SD of the ground truth labels)\n\n| **Ranker Accuracy**   | **50%** | **55%** | **60%** | **65%** | **70%** |\n|------------------------|---------|---------|---------|---------|---------|\n| **_Bias (B) = 0% SD_** | 0.8911  | 0.8436  | 0.8694  | 0.8668  | 0.8949  |\n| **_B = 10% SD_**       | 0.8995  | 0.8800  | 0.8623  | 0.8602  | 0.8969  |\n| **_B = 20% SD_**       | 0.9084  | 0.8869  | 0.8704  | 0.8704  | 0.9047  |\n| **_B = 30% SD_**       | 0.9224  | 0.8997  | 0.8844  | 0.8861  | 0.9192  |\n| **_B = 40% SD_**       | 0.9409  | 0.9172  | 0.9059  | 0.9071  | 0.9395  |\n| **_B = 50% SD_**       | 0.9645  | 0.9403  | 0.9329  | 0.9343  | 0.9654  |\n| **_B = 60% SD_**       | 0.9942  | 0.9702  | 0.9649  | 0.9670  | 0.9967  |\n| **_B = 70% SD_**       | 1.0291  | 1.0062  | 1.0022  | 1.0049  | 1.0326  |\n\n**Q2: Does the LLM simply memorize some of the ranking answers?**\n\nThis is a valid concern. To test for potential memorization, we used a **proprietary compound activity dataset** (75 compounds; IC50 labels) collected by our collaborators (experimental biochemists). This dataset is **unpublished and therefore not part of any public training corpora**, ruling out direct memorization. ChatGPT-4o achieved 60.14% pairwise ranking accuracy on this private dataset. This result **indicates generalization** and not memorization.\n\n**Q3: Have you considered non-uniform sampling of the reference point? Intuitively, references that aren’t clustered together should be better.**\n\nWe agree with the reviewer’s intuition that a clustered reference set would reduce the performance gains. We evaluated the effect of non-uniformity on synthetic data (similar to Q1) by **restricting the reference set range** to increasingly narrow subsets of the ground truth label range, simulating clustered or biased samples.\n\nAs shown in Table 2, RankRefine’s performance remains stable as long as the **reference set overlaps with the query label range**. Even with severe bias (e.g., 60% range), the method still improves the regressor estimate. For instance, with 60% ranker accuracy and 70% reference bias, RankRefine achieves a 12.30% MAE reduction (β = 0.8770). We also confirm, as mentioned by the reviewer, that RankRefine performs best when the reference set labels are spread across the full range of the ground truth labels and not clustered.\n\n**Table 2**. RankRefine performance, measured in β, with non-uniform sampling of the reference set. We restrict the reference set range to increasingly narrow subsets of the ground truth label range, simulating clustered or biased samples. Best performance is achieved when the reference set spans the entire range of the ground truth labels, but clustered reference sets still provide substantial regression improvements.\n\n| **Ranker Accuracy**      | **50%** | **55%** | **60%** | **65%** | **70%** |\n|---------------------------|---------|---------|---------|---------|---------|\n| **_Reference Set Bias (RB) = 0% range_** | 0.8828  | 0.8706  | 0.8343  | 0.8443  | 0.8598  |\n| **_RB = 10% range_**             | 0.9034  | 0.8731  | 0.8493  | 0.8529  | 0.8768  |\n| **_RB = 20% range_**             | 0.9078  | 0.8753  | 0.8366  | 0.8694  | 0.9002  |\n| **_RB = 30% range_**             | 0.9076  | 0.8701  | 0.8439  | 0.8733  | 0.9280  |\n| **_RB = 40% range_**             | 0.9143  | 0.8767  | 0.8618  | 0.8824  | 0.9414  |\n| **_RB = 50% range_**             | 0.9253  | 0.8786  | 0.8640  | 0.8769  | 0.9480  |\n| **_RB = 60% range_**             | 0.9349  | 0.8858  | 0.8712  | 0.8929  | 0.9546  |\n| **_RB = 70% range_**             | 0.9385  | 0.8919  | 0.8770  | 0.8957  | 0.9597  |\n| **_RB = 80% range_**             | 0.9551  | 0.8984  | 0.8798  | 0.8943  | 0.9558  |\n| **_RB = 90% range_**             | 0.9664  | 0.9042  | 0.8842  | 0.9035  | 0.9631  |\n\n**Q4: Pairwise rankings could be expensive for a human in practice.**\n\nWhile 20 comparisons per query may seem expensive, we note that many scientific domains already rely on human expert ranking (e.g., drug design). Our method is also compatible with low-cost and parallelizable rankers, like LLMs (which we have shown is not simply memorizing answers) or crowd workers, offering a promising tradeoff between accuracy and human effort."}, {"review_id": "JQFTw4qh8y", "reviewer": "Reviewer_8t5p", "summary": "This work introduces RankRefine, a model-agnostic, post-hoc method to refine regression predictions by incorporating pairwise ranking information from human experts or general-purpose LLMs, to overcome the limitations of deep-learning regressors when data is scarce. Experiments on synthetic and real-world benchmarks demonstrate the effectiveness of the proposed method.", "questions": "I have listed three questions I'm interested in the **weakness** part.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**:\n- This method is model-agnostic and plug-and-play. This means there is no need to post-train the model, which reduces implementation efforts.\n- The paper is well-motivated and the studied scenario is meaningful: There does exist label scarcity issues from real-world experiments across various scientific domains and tasks.\n- The experiment part demonstrates the effectiveness and robustness of the proposed method, and provides some insights, eg, moderate accuracies yield non-marginal performance gains.\n\n**Weakness**:\n\nThe reference set is a core component in the effectiveness of the method, but the paper does not provide very sufficient discussions  (The paper only analyzes the effect of the reference set size). The paper leverages random sampling for simplicity which is good, but a deeper analysis of the reference set’s quality and sampling strategy could empirically strengthen the work. Here are some discussion points that might be interesting:\n- The sampling strategy: Would a more advanced sampling strategy improve the results?\n- The distribution misalignment between reference and test: Would performance degrade if the reference and test distributions differ largely? The training-set (label, molecular structure, etc.) distribution could differ largely from the test distribution. This is common in practical scientific scenarios. Considering this in this paper could improve practicality.\n- The diversity of the selected reference data points: How about the significance of ensuring\n  - 1) label diversity, which means it spans the full range of property values (avoiding bias toward extreme or median values), or \n  - 2) the molecular scaffold structure diversity, which means the reference set covers various molecular scaffolds (not a single structure type)", "was_revised": true, "final_justification": "During the rebuttal, the authors have addressed several concerns I raised via the added experiments about alternative sampling strategies and distribution shift issues. I will keep my score.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We address the concerns regarding the reference set’s sampling strategy, distribution alignment, and diversity with new empirical analyses and clarifications below.\n\n**Q1: Will a more advanced sampling strategy improve the results?**\n\nWe agree that the reference set is central to RankRefine's effectiveness. While we used random sampling for simplicity, we now include additional experiments to study the impact of biased sampling. We use synthetic data where the reference and query labels are drawn uniformly within some ranges. We simulate predictions by injecting zero-mean Gaussian noises to the query labels.\n\nAs shown in Table 1, we systematically restrict the reference set to increasingly narrow subsets of the query labels' range to simulate non-uniform sampling (e.g., bias toward certain property values). Even under severe reference bias (e.g., 70% range coverage centered away from query), RankRefine maintains strong performance. For example, with 60% ranker accuracy and 70% reference set bias, we still achieve a **12.3% reduction in MAE** over the base regressor. \n\nThe fact that 0% reference bias achieves the best performance indicates that sampling from the full label range (uniform, unbiased sampling) yields the best performance. However, even clustered references retain strong utility. While more advanced strategies such as adaptive, query-aware sampling (e.g., using the base regressor’s prediction to guide selection) could further improve performance, we leave this as a promising direction for future work.\n\n\n**Table 1**. RankRefine performance, measured in β (β = $\\frac{MAE_\\text{refined}}{MAE_\\text{non-refined}}$, lower is better), with non-uniform sampling of the reference set. We restrict the reference set range to increasingly narrow subsets of the ground truth label range, simulating clustered or biased samples. Best performance is achieved when the reference set spans the entire range of the ground truth labels, but clustered reference sets still provide substantial regression improvements.\n\n| **Ranker Accuracy**      | **50%** | **55%** | **60%** | **65%** | **70%** |\n|---------------------------|---------|---------|---------|---------|---------|\n| **_Reference Set Bias (RB) = 0% range_** | 0.8828  | 0.8706  | 0.8343  | 0.8443  | 0.8598  |\n| **_RB = 10% range_**             | 0.9034  | 0.8731  | 0.8493  | 0.8529  | 0.8768  |\n| **_RB = 20% range_**             | 0.9078  | 0.8753  | 0.8366  | 0.8694  | 0.9002  |\n| **_RB = 30% range_**             | 0.9076  | 0.8701  | 0.8439  | 0.8733  | 0.9280  |\n| **_RB = 40% range_**             | 0.9143  | 0.8767  | 0.8618  | 0.8824  | 0.9414  |\n| **_RB = 50% range_**             | 0.9253  | 0.8786  | 0.8640  | 0.8769  | 0.9480  |\n| **_RB = 60% range_**             | 0.9349  | 0.8858  | 0.8712  | 0.8929  | 0.9546  |\n| **_RB = 70% range_**             | 0.9385  | 0.8919  | 0.8770  | 0.8957  | 0.9597  |\n| **_RB = 80% range_**             | 0.9551  | 0.8984  | 0.8798  | 0.8943  | 0.9558  |\n| **_RB = 90% range_**             | 0.9664  | 0.9042  | 0.8842  | 0.9035  | 0.9631  |\n\n**Q2: Would performance degrade if the reference and test distribution differ largely?**\n\nTo test the impact of distribution shift between reference and query, we created two challenging scenarios where the reference labels and query labels are drawn uniformly from different range values:\n\n- **Disjoint distributions**: reference labels in [-1, 0] or [1, 2], query labels in [0, 1]. We averaged the results of the two cases, that is, between {ref =  [-1, 0], query =  [0, 1]} and {ref =  [1, 2], query =  [0, 1]}.\n- **Partial overlap**: 50% overlap between reference and query label range\n\nIn both cases, we fix the reference label range width and vary the ranker accuracy. Despite the mismatch, RankRefine still improves the base regressor when the ranker is moderately accurate (≥65% in the disjoint case, ≥55% in the partial overlap case). These findings suggest that **RankRefine is robust to distribution shift** that is common in real-world settings with scarce or heterogeneous data.\n\n**Table 2**. β for disjoint and partially overlapping reference set.\n\n| **Ranker Accuracy**   | **50%** | **55%** | **60%** | **65%** | **70%** |\n|-----------------------|---------|---------|---------|---------|---------|\n| **_Disjoint_**        | 1.2906  | 1.1629  | 1.0170  | 0.9437  | 0.9219  |\n| **_Partial Overlap_** | 1.0160  | 0.9465  | 0.8700  | 0.8607  | 0.8898  |\n\n**Q3: How does label and scaffold diversity in the reference set affect performance?**\n\nLabel diversity plays a key role. Table 1 demonstrates that reference sets spanning the full range (0% reference bias) of ground-truth labels consistently achieve the best results, underscoring the importance of covering the ground truth labels spectrum.\n\nWe agree that scaffold diversity is also an important factor in molecular applications. However, we did not explicitly control for it in our experiments, as our goal is to present RankRefine as a general-purpose method applicable across domains. Measuring or enforcing scaffold (i.e., design) diversity is specific to molecular data and does not easily transfer to other domains where structural diversity is harder to define. We view scaffold-aware reference selection as an important, **domain-specific extension** of our work and leave it for future."}]

Author rebuttal:

Meta-review: The paper proposes a conceptually simple and general method for tuning regression models given preference ranking. The core experiments are done using LLMs to elicit rankings that are used to improve regression models. 

Reviewers voted for accepting the work. They appreciated that (a) the method is model agnostic (as shown by results that use RF), (b) shows consistent empirical gains, (c) the paper is clearly written.

Memorization was among raised concerns: it is not clear to what extent the prior exposure of LLMs to large sways of data allows them to just leak test information. This was partially addressed during the rebuttal phase but adding a private test set.

The novelty of the work is limited. First, approaches such as GRPO are routinely used to adjust (generative) models responses based on preference data, also stemming from the Bradley-Terry Model. Second, test-time adaptations are common, including various prompting techniques. Third, there are existing post-hoc adjustment methods (included in the baseline comparison). The AC differed in the assessment novelty from Reviewers and it was discussed during the private discussion phase.

Relatedly, the paper misses an important baseline: adjusting an LLM directly based on the preference data (vs adapting a regressor based on the LLM). This naturally would in the default setting require retraining but could be also done in-context with prompting.

However, given that the paper compares to two prior methods for adjusting regression models based on ranking information, the paper does show to make an advance in a timely application.

All in all, I am recommending acceptance of the paper.
Decision: Accept (poster)