Unveiling Extraneous Sampling Bias with Data Missing-Not-At-Random
Authors: Chunyuan Zheng
Haocheng Yang
Haoxuan Li
Mengyue Yang
Venue: NeurIPS 2025 poster ()
Keywords: Missing-Not-At-Random
Sampling Bias
Primary area: probabilistic_methods

Abstract: Selection bias poses a widely recognized challenge for unbiased evaluation and learning in many industrial scenarios. For example, in recommender systems, it arises from the users' selective interactions with items. Recently, doubly robust and its variants have been widely studied to achieve debiased learning of prediction models, however, all of them consider a simple exact matching scenario, i.e., the units (such as user-item pairs in a recommender system) are the same between the training and test sets. In practice, there may be limited or even no overlap in units between the training and test. In this paper, we consider a more practical scenario: the joint distribution of the feature and rating is the same in the training and test sets. Theoretical analysis shows that the previous DR estimator is biased even if the imputed errors and learned propensities are correct in this scenario. In addition, we propose a novel super-population doubly robust estimator (SuperDR), which can achieve a more accurate estimation and desirable generalization error bound compared to the existing DR estimators, and extend the joint learning algorithm for training the prediction and imputation models. We conduct extensive experiments on three real-world datasets, including a large-scale industrial dataset, to show the effectiveness of our method. The code is available at
https://github.com/ChunyuanZheng/neurips-25-SuperDR.

Unveiling Extraneous Sampling Bias with Data Missing-Not-At-Randomm
Chunyuan Zheng' Peeking University
Haocheng Yang²
Haoxuan Li Mengyue Yang*
Natioonal University of Singapore
University of Bristol
cyzhengestu, pku. edu. cn
Abstract
Selection bias poses a widely recognized challenge for unbiased evaluation and learning in many industrial scenarios. For example, in recommender systems, it arises from the users' selective interactions with items. Recently, doubly robust and its variants have been widely studied achieve debiased learning of prediction models, however, all of them consider a simple exact matching scenario, i.e., the units (such user-item pairs in recommender system) are the same between the training and test sets. In practice, there may be limited or even no overlap prac-
in units between the training and test. In this paper, consider a tical scenario: the joint distribution the feature and rating is the same the training and test sets. Theoretical analysis shows that the previous DR estimator is biased even if the imputed errors and learned propensities are correct in this scenario. In addition, we propose a novel super-population doubly robust esti- mator (S)perDR), which can achieve a more accurate estimation and desirable generalization error bound compared to the existing DR estimators, and extend the joint learning algorithm for training the prediction and imputation models. We conduct extensive experiments on three real-world datasets, including large-scale industrial dataset, to show the effectiveness of our method. The code is available at https//gghhha.oooommuuayya////gu 25-Sper
Introduction
Selection bias means the distribution of collected data differs from that in the target population. It ubiquitous and occurs when data are missing-not-at- -to--- (MNAR). For example, in the recommender system (RS), due to the subjective preferences of users and the data collection process itself, selection bias always exists in the collected data 2. Thus, selection bias poses widely- recognized challenge 3 4 3 Ø.. Ignoring selection bias makes machine learning methods difficult to achieve unbiased predictions and reducing reliability 8
Many methods have been proposed to address selection bias. The error imputation-based (EIB) method [9 10 utilizes imputation model impute the missing relevance. The inverse propensity score (IPS) method uses inverse propensity to reweight the observed events to achieve unbiased- ness
The doubly robust (DR) method combines the error imputation model and the propensity model [5] [7 19), which is unbiased if either the imputed errors or the learned propensities are accurate, and also proved to have lower variance compared IPS [0)]
12 13..
Although previous DR-based methods have demonstrated promising performance debiasing tasks, all of them consider the exact matching scenario, i.e., the units are the same between training and test set. In RS, it means the users and items are the same between training and test set, as illustrated in the left part Figure However, there may be limited even overlap in units between training and
Conresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

HOt
Trairing
Exacely matchinn
Poold
Perato (st, r)
Press (at, (r),
Gold
Test
Traditional debias
Cold start
Ours
Figure I: (1) Traditional debiasing methods consider an exact matching scenario, where the users and items are the same between training and test set, the non-random missingness (missing two negative ratings) indicates selection bias; (2) The cold-start problem refers to the training data containing only hot users/items, while the test data has only cold users/items, which differs from the debiasing scenario intrinsically; (3) This paper extends the exact matching scenario, considering more general scenario that the joint distribution of the feature and rating Pz in training and test set is the same.
test sets. For example, in many industrial scenarios, the user-item pairs in offline training data cannot be exactly the same as those in online test data [2] 22 Therefore, instead of exact matching, a more practical scenario is that the joint distribution of the feature and outcome P(xr, r) is the same in training and test set, as illustrated in the right part in Figure Note that this scenario is intrinsically different with the cold-start problem, where the training data distribution Phot differs from the test data distribution Foold, as shown in the middle part in Figure | In addition, selection bias is rarely considered in the cold-start problem.
To this end, in this paper, we first derive the bias of the DR estimator in super-population scenario, which contains additional covariance term besides the term that measures the accuracy of imputed errors and learned propensities. Surprisingly, the DR estimator is biased even if the imputed errors and learned propensities scenario, and propose the SuperDR estimator with the corrected imputation model, which can effectively control the additional covariance term with many desirable theoretical properties such as bias and variance reduction, leading to a more accurate estimation under the super-population scenario. In addition, we extend the previous joint learning algorithm based the proposed corrected imputation loss and further derive the generalization error bound for the proposed SuperDR, and show that the proposed learning approach can effectively control it. Extensive experiments are conducted on three real-world datasets to show the effectiveness of our SuperDR method.
correct. Then we provide unbiased conditions under the super-population
Our main contributions be summarized follows:
the best of our knowledge, this is the first paper that extends the exact matching scenario to the super-population scenario. In this scenario, we derive the bias of the DR estimator and show DR estimator is biased even the imputed errors and learned propensities are correct. We propose the SuperDR method based on the corrected imputation model, which can effectively control the additional covariance term. In addition, the corrected imputation model not only benefits unbiased estimation but also reduces the generalization bound to enhance prediction performance.
We conduct extensive experiments three real-worth datasets, including large industrial dataset, to demonstrate the effectiveness of our proposed method.
2
Related Work
There are various biases in the collected data 23
which have been of increasing concern in 31. Selection bias is one of the most common biases and a lot of research has been done aiming to eliminate this kind of bias
24
recent years 25 26
27
28
29
30
35 36 Based on the causal 42 43 the error imputation based method (EIB) 44 45
inference techniques 37
38
39
first imputes pseudo-labels for missing events from the observed events, and then leverages these labudo-labels to train the prediction model [46). The propensity-basee based approaches weight the inverse

---

propensity score (IPS) on the observed data to eliminate bias [2 from large variance when the extreme values exist in the estimated propensities 47
13 However, IPS will suffer
The doubly robust (DR) method improves the weakness of EIB and IPS methods and becomes the mainstream model due to the weaker unbiased conditions (unbiased when either imputed errors
or learned propensities are correct) and smaller variance than the IPS method [48 49
50 51 52].
In particular, the DR estimator is unbiased when either learned propensities or imputed errors are accurate. Many augmented DR methods are developed further enhance the previous DR method performance by modifying the propensity model and imputation model or the form of the DR estimator, such as MRDR [33], BRD-DR [53] StableDR [54], TDR [55]. DR-MSE [56], MR [57],
CDR (58), AKBDR [59] DCE-TDR (60), and D-DR [61]. In addition, there are methods leveraging few unbiased ratings to mitigate hidden confounding and improve DR debiasing efficacy [5] 62 63
In this paper, we consider more general apper-population scenario and propose SuperDR with the corrected imputation model to achieve a more accurate estimation.
3 Preliminaries
We start with the classic debiasing scenario and take RS as an example. Note that the selection bias also exists in other scenarios such pattern recognition and causal effects estimation, and our proposed method is also applicable for debiasing in these scenarios. Suppose the training user set
Uirain = [11, Ug ....... Um contains m users, the item set Iuain {ii, i2. ................................ 2n.} contains n items. The purpose of RS is to train a prediction model to accurately predict the ratings of all user-item pairs, thus the target population is defined as all user-item pairs Duures Uraan Turin- Let R E R=*x" be the ground truth rating matrix of all user-item pairs in Duugett where ru.; is the rating of user on item i. Let Fu the feature of user u and item and fu,s f () is the predicted rating by prediction model, 0 is the corresponding parameter. Denote R E mRm as the matrix containing all the predicted ratings. Let O (0, jm be the binary observation indicator matrix for all user-item pairs, ou indicates the rating of user u on item is observed, otherwise missing Ou,i = 0. All previous methods implicitly assume Dirrget Diest D with fixed user-item pairs, thus the only randomness comes from the missing mechanism. If all the ratings are observed, the prediction model can be trained directly by minimizing the following ideal loss
1 D
Lideal (0)
Cu,i
(u,D)E
where eu,i C(fu,i Tw.s) is the loss between the predicted rating f..; and the true rating ru, and -(.,)) is an arbitrary loss function. However, the ideal loss is not available in most cases because we can only observe part of the data with selection bias. Thus, for user-item pair with ou,i 0, the Tu, is missing. To tackle this issue, the DR estimator has been proposed:
1
Ou.ileus @u,k)
ED (0)
( +
D
Pu,i
(D,i)ED
where Pu,i (I :t v)) is the propensity model to estimate Pu,i P(ou, 1 Fu,i), and e., = m(Fa,ti d) is the imputation model to impute the missing eu,i-
4 Proposed Method
4.1 From Finite Population to Super-population
Before introducing our method, we first focus on the theoretical properties of the DR estimator and start from the bias form of DR estimator.
Lemma 4.1 (Bias of DR Estimator [14)). Given imputed errors e and learned propensities Pu,s O, when considering only the randomness of missing indicators, the bias of DR estimator is
{Pu.s Pu.i) {eu,s ēu,}
Biaso ED (0)
Σ
D
Pu.
(M.))D
3

---

We find that either éu.s Cu,i or Pu.i Pu.i is sufficient to eliminate bias, which inspires the double robustness condition for the DR method.
Corollary 4.2 (Double Robustness 14). The DR estimator is unbiased when either imputed errors &u,i or learned propensities Pu.a are accurate for all user-item pairs, i.e, either εu,i Cu,i or Pu,i Pu,, for all u and i.
The above Lemma 4.1 shows the bias form of the DR estimator when users and items in the training and the users and items the test are exactly the same. However, scenario is too simple in many real-world scenarios. Thus, we consider a more general scenario, also known super-populatee populltioo,,. with U {u1, U2 ... .. I {ir. d2.... Diareet {ui, U2,,.. Um x {i1, ......, J and Dest {uz, Uf2...... "i x {ik, dky...... Akk,' J are sampled from the whole user set and item set, respectively. Without loss of generality, we consider the sampling strategy to be the same for both Parget and Dieu datasets (otherwise, we can adjust the sampling strategy by reweighting). Therefore, instead of Duunge = Des D, we consider a more practical scenario P(Durge)) = ((Diea) = P(D), that is, the joint distribution P(z, r) in the target and the test population are the same. We can regard the ground-truti ratings and covariate values drawing (Dargetl times from the P(z, r) in the super-population and are therefore they stochastic. Furthermore, the randomness of ratings and covariates leads to the randomness of other variables such as 6x,, and εu,i. For unbiased prediction in this scenario, we need to estimate the expected ideal loss below:
discussed earlier, this
Lideal (0) E| Cideal (0) = Elen,a],
where the expectation is taken on the super-populattos distribution P(z, r). Unless otherwise stated, all expectations are taken on the P(x,1 r) later. With the additional randomness caused by super- population, the theoretical results the DR estimator change. The following theorem and corollary show the bias and the double robustness property under super-population for the DR estimator.
Theorem 4.3 (Bias of DR Estimator under Super-population). Given error imputation model ē.,i and propensity model pa.s. then the bias of the DR estimator for estimating the expected ideal loss under super-pony super-population is
Biasp (0) = Cov Puus Ous
Ousi Pu.i
Eugi Enp
+E
(uni sunal - Elen, an
Pu,
expuila of independent
equals O either Blo..:/B..:: xu.e) Lor eu.e Rn.il
Corollary 4.4 (Double Robustness under Super-population). Under super-population, the DR estimator is unbiased when both the following conditions hold:
(i) Either learned propensities satisfy Blou.s/Pu.i Fuu,s) 1, or imputed errors have the same conditional expectation with true prediction errors B(êu,i Juull = Eleu,i Tugil;
(Bun-oau Puni
(ii) The covariance term vanishes, that is, Cov
GCafi Cu.i))
Remark: Previous DR estimators are biased even if Eu,i = eu, or Pu,i = Pu.s for all (u, i) E Diarget-
Compared with the existing theoretical results in Lemma 4.1 it is obvious that condition (i) is necessary to achieve unbiasedness, which directly extends the conditions of accurate imputed errors and learned propensities in Lemma 4.1 the expectation form. However, note that the condition (ii) that covariance vanishes is also needed for the unbiasedness under super-population scenario. Intuitively, if ignoring the randomness caused by sampling process, then eu, êu,i is constant given Fu,i- By the double expectation formula, the covariance term vanishes automatically. The detailed proofs are in the Appendix Therefore, it is necessary modify the previous DR learning approach to control the covariance while learning accurate propensity and imputation models under super-population scenario.
4.2 The SuperDR Estimator
It is important to note that the true covariance is unknown because we cannot access the true data distribution. However, we can use the empirical covariance over user-item pairs as an approximation of the true covariance. We first give the definition of empirical covariance.

---

Definition 4.5 (Empirical Covariance). The empirical expected conditional covariance between
(Pu,i Pu. and Eu,i Gu,i
Pu,s - Ou.
Pu.i
Oui (eu,s êu,i)-
Cov
Cui ev, =
Pu,i
Pu,i
(u,))D
When the learned propensities imputed errors are accurate, i.e., satisfying condition (i) Corollary 4.4. the empirical covariance will converge to Cov method to control the empirical covariance is to regard it as a regularization term. However, since the data are partially observed, cannot obtain the value of the empirical covariance all user-item pairs. In addition, the large penalty term may hurt the prediction performance. Interestingly, motivated by targeted maximum likelihood estimation [55 64], we found that the empirical covariance can be controlled with targeting correction step based on the DR estimator, Specifically, we designed imputation correction follows:
Pu.i-ouu Cu.i - Cu.s |D] A direct Pu.:
e.j = Êu,s + c(Ou,) pu,i).
where द.,; m () the imputed errors in previous DR estimators, Pa.i = T (Tu, (b) the learned propensity, and is learnable parameter. We optimize and in Êu,s by minimizing the loss based on imputation correction:
Ou,i - ēnni)
(Ø E²) = arg min C. Sup (Ø,
Σ
d.c
[
Du,5
(a,i)ED
Specifically, the added correction term c(Ou,) Pu,i) has several desired properties. First, the correction term enlarges the hypothesis space ê compared to éu,i and does not bring extra concerns to the double robustness property due to it has zero mean under accurate P... Second, the derivatives on the proposed loss with respect to are shown below:
① Sup (Ø,c)
Du,s Ou,
Σ (u.J)EO
(Cu,s Ex,i)-
[D]
Pu,i
It has the same form the empirical covariance for user-item pairs with Ou, 1, which means that can make the empirical covariance for observed user-item pairs directly. Note that adding the correction term on either εu,i or Cu,i will not affect the gradient above, thus add such covariance can also be bounded by minimizing LSuP using the concentration inequality. To proceed, we first define the empirical Rademacher complexity as follows.
by minimizing the Luup
éu,' for illustration. In the step, show that the unobserved empirical
Definition 4.6 (Empirical Rademacher Complexity (651). Let F be family of prediction models mapping from E X' to (a, b). and {ær,; (u, i) E D) fixed sample of size [D] with elements in X. Then, the empirical Rademacher complexity of F with respect to the sample S is defined as:
R(F) = B0 (- sup feeF
Σ
Cu.
D
(a,i)ED
where o (oa.. (u, E D), and ou,i are independent uniform random variables taking values in +1]. The random variables ou, are called Rademacher variables.
Then derive the controllability of empirical covariance for all user-item pairs in Theorem 4.7 Refer to Appendix [ for the complete proof for this theorem.
Theorem 4.7 (Controllability of Empirical Covariance). The corrected imputation model trained by LSuP is sufficient for controlling the empirical covariance.
(i) For user-item pairs with observed outcomes, the empirical covariance is O. Formally, we have
L dc
SSp
(0,c)
= 0, which is equivalent to
Σ
Pu,i On. (Bus - Eu.s) = 0:
D
Pus
c==e
(u.i):
on
5

---

Algorithm 1: The Proposed Doubly Robust Joint Learning Algorithm under Super-population
Input:
observed ratings R and pre-trained propensity model π ((m,):)).
while stopping criteria is not satisfied do for number of steps for training the corrected imputation model do
Sample batch of user-item pairs [ from O: (Ø, E);
Update & by descending along the gradient V &&Le Sup (Ø, c);
Update e by descending along the gradient V.L. end
for number of steps for training the debiased prediction model do ) Sample batch of user-item pairs {(ug, &x) 1k=1 from D;
Update 0 by descending along the gradient V VaL (0; Φ, 1/);
10
end
end
(ii) For user-item pairs with missing outcomes, suppose that Pu,s 2 K and lea, - Euul S K then with probability at least ?). have
Σ
Pu,t Ouk
(ee, Cuyi)
D
Puii
(u,
L. Sup (o,,)) +
Ouei Tuud
Kg 1+ K
2R(F) (2K + 1)
2 2lo((4/1)
D
P
B.ED
propored loas
eupiriical bios from propensity
controlled empiricct Rudemacher complecit and scmple vion
where the Kig, Kg 1) constants.
Note the proposed imputation correction has no harm property theoretically, shown in Corollary 4.8
Corollary 4.8 (Relation to previous imputed errors). The learned coefficient c" will converge to when the imputation model ê has zero empirical covariance, making ε., degenerates to é
In addition, the proposed imputation correction can not only control the empirical covariance effec- tively but also be helpful for learning more accurate imputed errors.
Corollary 4.9 (Bias reduction property). The proposed corrected impulation loss leads the smaller bias of imputed errors ε.,á, when êy,s are inaccurate. Formally, we have
min C
SuP (0,E)
1 D
Ou.ileu, - ēur)? I min Co(o) =
1
Σ
Ou.ileu,s - Cu,i
Du,i
D
Du,i
(a.))D
(a.jeD
Moreover, while reducing bias, the proposed method also reduces the variance compared to the previous imputed errors under moderate condition, as shown below.
Corollary 4.10 (Variance reduction property). The proposed corrected imputation loss leads to smaller variance of Enss when the optimal e" lies in certain range. Formally, have
v(eu.s) V(eu, + (ou.s - Pu,i)) < V(e.)), if E" (o, 2 Cov(éu, Pau. Ov.i)
V(D.. Ou.s)
See Appendix for the proof for the above three corollaries. Finally, the proposed SuperDR estimator given below based on the corrected imputation:
(eni Eu.i)
ESuperDR (0)
D
Pu,i
(a.

---

Table I: Performance AUC, NDCG @K K and Recall @ K the Coat, Yahoo! R3 and KuaiRec datasets. The best result is bolded and the best baseline result is underlined, where means statistically significant results (p-value I 0.05) using the paired-t-test.
Metbods
Coot
Yahoo! R3
Koasifee
AUIC
NDCG
Recalld:
AEC
NDDG65
Recalli
AUC
NDCG@550
Ro
MLP
0.729
0,003
0,535
0,0006
0,614 1. 0.0077
1.064 0.002
[U540 0.002
0.442 22 0.004
0.808
0,005
0.610
0.007
0.643
0.010
DAMP
0.720
1005
0,5322 1:0 0.00007
0.628 1. 0008
0.064
0.0012
[1642
0.000
0.438
0.002
0.811
1 1.0000
0.009
000011
0.623
00005
CVIB
0:720
0030
0.547 1:0 18/5
0.625 1 0.000
0,670 31 0.0014
0.656 0.003
0.422 000001
0.816
0.577
00:05
01651
(00P)
0.731
0.625
0.667 0.0013
0.647
0.006
0.415 0.007
0.000 0.936
0.006
0.000
0.643
0.005
0.732
0.0005
0.529
0.607 01.013
0.641
0.412 0.033
0.004
0.000
0.699
0.005
0.730
0.520
0.508 01,012
0.452 0.005
0.000
0.000
0.032
0.055
0.628
0.502 010003
0.001 0.651
0.445 0.002
0,830
0.000
0.008
0.005
0.625
0.607 0.0015
0.449 0.008
0,839
00001
0007
0.448 0.005
0.834
0.116:1
0003
0.000
SDR
0.748
01625
B.005 0.005
0.204
0.451 0.004
0.005
0.632 0.005
TDR
0.744
0,6511 0.0005
0.064 0.001
0.207
0.453 0.003
0.005
0.009
0.656 0.010
0,550
0.672 0,005
0.005
0.454
0.002
0,823 0.000
0.004
0.633 0.005
AKBOR
0.748
0,536
0,630 0.007
0,676 0,004
0.004
0.061
0.00
0.821
0.0
0.005
0.087 0.055
DCE-TDR
0. T46
0.0.0
0,622 0.000
0.679
0.004
01005
0.650 0.003
0,521
1 0:03
0
0.001
0.611 0.005
D-DR
0.
0.033
0.654 0.004
0.630 0.008
0.675
0.004
0.650
0.001
0. 05656 0.003
0.522 0.001
0.650
0.005
(1672
+ 0.005
SuperRR
0.757 0J004
(L667' + IUMS
0,637 0U007
0.686
0.003
0,667
0.004
0.463
0.083
0.828
0,004
0.640
0,005
0.680
0.005
4.3 The Extend Joint Learning Algorithm
We optimize the prediction model and the imputation model of the SuperDR method by a widely used joint learning framework (14), which alternatively optimizes two models to achieve unbiased learning. Specifically, we train the prediction model by minimizing SuperDR loss:
Du (0)
1
Σ (a,i)ED
ē..i + Ous(eu,i - Eu.i)
D
Pu.i
We update the imputation model parameters and simultaneously by minimizing the CEMP(o e) in Section 4.2 and we train the propensity model by minimizing the following cross cross-entropy loss.
Cp(x)=
1
Σ (u,))ED
i- ip.() - (1 ou.i) log(1 pu,)))
D
The propensity model is pre-trained, and the parameters of the prediction and imputation model are updated alternatively via SGD. The joint learning process is summarized in Algorithm Note that the complexity will not increase due to we only additionally update one single parameter compared to the traditional joint learning algorithm.
4.4 The Generalization Bound
Next, we analyze the generalization error bound of the DR methods using the models for estimating eu,i and Pa,á, and show that controlling empirical covariance leads to a tighter bound. Specifically, the generalization eтror theories for the previous DR estimators relied mainly on the boundedness of the loss to each user-item pair in the DR estimators from the binary indicator Ou,i, i.e., for the DR estimator, the bound for DR loss on (u, i) is (eu., - éu.i)/fu..- However, these analyses no longer hold under super-population scenario. In the following theorem, we provide the generalization error bound of SuperDR, which includes four terms: the SuperDR loss, the empirical covariance, the bias of the SuperDR estimator, and the tail bound. Compared to previous DR methods, the proposed method can further control the covariance term, leading to a more desirable generalization bound thus improving debiasing performance. See Appendix A for the proof.
Theorem 4.11 (Generalization Bound under Suppr-population). Suppose that pu,s 2 K and min (Cu, leu, } < Kg. then with probability at least 17. we have
Lideat(0) (0) S DR (0)
1
Σ
1
E
Ou.i
Iu,i
Eleu, [u,i] - Eleu,i Fa.a]
D
Pr.i
(M.))DD
vanilla DR only controls the empirical DR loss, and empirical risks of imputation and propensity models
+
1
Cov
Ou,s
Pa.s
Buyi Euai)
+
(1+ K
2R(F) + Køy
18 log(4/n)
(D)
Pu.i
(D)
corrected loss further controls the independence
tail bound controlled by empirical Rademacher complexity and sample size
7

---

0.835
0.64
0.68
0.830
NDCG@50
Recalk@50
0.825
0.63
0..67
0.820
M3DR
AXBOR
AXBOR
0.62
DCETDR
0.66
DCETDR
0.815
0:08
D-DR
D- SuporDR
SuperDR
SuperDR
810
0.61
0.65
10% 30%
Sample
Sample Ratio
Sample Ratio
(a) AUC
(b) NDCG@50
(c) Recall
Figure 2: Effects of varying sample ratios 6% on debiasing performance on the KuaiRec dataset.
0.69
0.670
0.47
0.68
0.6665
RecallgSS
0.46
0.660
0.67
ACOOR OCETOR
AKSOR
0.45
0.655
Supert4
0.56
5% 10% 50% 50%
70% 90%
0.650
5%
10%
30%
505
70%
90%
0.44
90% 5%
10%
30%
505s
70%
Sample
Sample
Sample
(a) AUC
(b) NDCG@50
(c) Recall@500
Figure 3: Effects of varying sample ratios 5% on debiasing performance on the Yahoo! R3 dataset.
5 Experiments
5.1 Experimental Setup
Dataset Selection and Preprocessing. To verify the effectiveness of the proposed method in the real-world dataset, the dataset that contains both biased and unbiased data is required. Following the
66]. the following three widely used real-world datasets are adopted to conduct our experiments: Coat contains ratings from 290 users to 300 items with 6,960 biased ratings and 4,640 unbiased ratings. Yahoo! R3 contains ratings from 15,400 users 1,000 items with 311,704 biased ratings and 54,000 unbiased ratings. We binarize the ratings to 0 for ratings less than three, otherwise to 1. We further use fully exposed industrial dataset KuaiRec [67] with 4.676.570 video watching ratio records from 1,411 users to 3,327 videos. Following previous studies [59 60). we biasedly select 201,171 samples according the watch ratio the training set and randomly select 117,113 samples as the unbiased test set. For this dataset, we binarize the records to 0 for records less than two, otherwise
previous studies [1 15 32
Baselines. In our experiments, as there are only very few features or no features for users and items in all three datasets, we first use the matrix factorization (MF) 3 method to generate the embedding for each user and item, and then fix such embedding as the user-item features. Then we take the MLP the backbone model and compared the proposed method with the following debiasing baselines including DAMF [68], the information bottleneck based method: CVIB (69,, the propensity based methods: IPS 13, SNIPS 70,, ASIPS (35), and IPS-V2 [71]. and the DR-based
methods: DR [14], MRDR B3 DR-MSE [56), DR-V2 (71) TDR 55 SDR 54) MR [57]
AKBDR DCE-TDR 60 and D-DR 61.
Experimental Protocols and Details. The following three metrics are used to measure the debiasing performance: AUC, NDCG@K K, and Recall@K K, where we set K 5 for Coat and Yahoo! R3, while set K = 50 for KuaiRec. All the experiments are implemented on PyTorch with the GeForce RTX 3090 as the computational resource. Adam is utilized as the optimizer in all experiments. To simulate the super-population scenario, we first randomly sample 6% users and items (unless otherwise stated, 6 is set 50% in our experiments) from the training set and then use the whole unbiased test set to evaluate the debiasing performance, Note that this intervention will not affect the data sparsity, it will only affect the number of observed users and items and will ensure PD = ((Dua) with limited overlapped users and items. In addition, the dimension of user and item embedding are fixed as 32. We tune learning rate in {0.001, 0.005, 0.01, 0.02, 0.05) } for parameters in prediction, imputation, and propensity model, and in (0.01, 0.05, 0.1. 05. 0.2) } for e, batch size in (128, 256, 512} for Coat and (1024, 2048, 4096) for Yahoo! R3 and KuaiRee. The weight decay is tuned in {le- 6, 5e
the logistic regression model 6...., 5e 3, le 2}. In addition,

---

Base TDR
DCE-TDR
Base TDR
DCE-TDR SuperDR
Base TDR
DCE-TDR
SuperDR
SuperDR
RI (%5) of NDCG@K
EC Reduction (N)
RI (% AUC
25
0-
-55
Cost
KuniRec
Coat
Yahoo Dataset
KualRec
Coat
Yahoo Dataset
KualRec
(a) EC Reduction
(b) of AUC
(c) RI of NDCG@K K
Figure 4: Effects of empirical covariance (EC) reduction (%) on relative improvement (RI) (%).
SuperOR
DCE-DRR
SuperDR
OCETDR
SuperDR
DCE-TOR
SuperDR
DCE-TDR
0.645
0.685
NDCG@8
0.668
0,810
NDCG@50
0.640
AUC
0.680
0.060
0.820
0.630
0.01
0.85
Learning rate of
Learning rate of
Learning rate of
Learning rate of
(a) AUC on Yahoo! R3 (b) NDCG@S on Yahoo! R3
(c) AUC on KuaiRec
(d) NDCG@50 on KuaiRec
Figure 5: Effects of learning rate of the learnable imputation correction parameter
as the propensity model, which means that there is no unbiased data requirement. To prevent the propensity small, tune the propensity clip threshold in [0.005, 0.05). For simplicity, fix the step inner loop for updating prediction and imputation models Algorithm aas 1.
5.2 Performance Comparison
Table summarizes the debiasing performance various methods on three benchmark datasets Coat, Yahoo! R3, and KuaiRec, and have the following findings. First, most debiased methods outperform the base model naive MLP, which shows the necessity for debiasing. Second, overall speaking, DR-based methods such D-DR and DCE-TDR demonstrate the
competitive performance, indicating the superiority of DR methods over other baselines. Third, the proposed SuperDR method achieves the best performance in terms of all evaluation metrics. This indicates that the SuperDR method can effectively reduce the additional bias introduced by sampling through controlling empirical covariance, thus achieving an unbiased estimate of the ideal loss in scenarios where users and items in the training set are not exactly the same as those in the test set.
5.3 In-Depth Analysis
Effeets of Varying Bias Level. Figures investigates the impact of different levels of bias introduced by sampling on prediction performance on the KuaiRec dataset. We change the sample ratios to control the degree of overlap between users and items in the training and test sets. A higher sample ratio indicates greater proportion the same users and items in both sets, resulting in less bias introduced by sampling. When the sample ratio is large (e.g., 0.7 or 0.9), our method slightly outperforms recently proposed state-of-the-ard methods such as DCE-TDR and D-DR. When the sample ratio is 0.05 or 0.1, there are few overlapping users and items between the training and test sets, resulting in significant bias introduced by sampling. The performance of previous methods noticeably declines, while the SuperDR method effectively addresses this bias, achieving significant performance improvements. We also conduct experiments on Yahoo! R3 dataset. The experiment results are in Figure with similar phenomenons.
Effects Empirical Covariance Control. We explore the effects of Empirical Covariance (EC) Reduction on the prediction performance in Figure We find that SuperDR achieves the most significant empirical covariance decreases and the most competitive performance in AUC and NDCG@K K, which empirically demonstrates the effectiveness of the targeting correction step and the ECC reduction benefit the prediction performance. Note that DCE-TDR and TDR method obtains some performance improvement compared to base model naive MLP, this is because they

---

add o 1_ - 1) as the correction term to the imputed errors to control the covariance on observed Pu.
samples. Unfortunately, DCE-TDR and TDR unable to control the covariance missing outcomes, resulting in sub-optimal performance.
5.4 Sensitivity Analysis
We conduct sensitivity analysis on the Yahoo! R3 and KuaiRec datasets to explore the relationship between the learning rate of learnable parameter and the debiasing performance, with AUC and NDCG@K as the evaluation metries, where K-5 on Yahoo! R3 and K=50 on KuaiRec. As shown in Figure the proposed SuperDR stably outperforms the DCE-TDR, the most competitive baseline on these two datasets, under varying learning rates of c, demonstrating that the enhanced imputation model with target learning mitigates the additional bias introduced by sampling and exhibits no- harm property. Meanwhile, under relatively moderate learning rates (0.05, 0.15), the SuperDR demonstrates competitive prediction performance, which further indicates the robustness.
6 Conclusion
In this paper, we extend the previous exact matching scenario, i.e., the units are the same between training and test set, and consider more general scenario that the joint distribution of the feature and rating P(z,77 r) in the training and test set to be the same. Then show the DR estimator is biased even if the imputed errors and learned propensities are correct in this scenario and provide the explicit bias form, which has two terms: the term that measures the accuracy of imputed errors and learned propensities and an additional covariance term. To achieve a more accurate estimation, we propose the SuperDR estimator with the corrected imputation model, which can effectively control the additional covariance term with many desirable theoretical properties such as bias and variance reduction. In addition, extend the previous joint learning algorithm based on the proposed corrected imputation loss and further derive the generalization error bound for the proposed SuperDR, and show that the proposed learning approach can effectively control it. Extensive experiments real-world datasets to show the effectiveness our SuperDR method. One of the potential limitations and research directions is how develop a tighter bound for controlling the empirical covariance and to develop a more efficient algorithm for alternatively updating the prediction model, the imputation model, and the target learning parameter.
conducted three
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (623B2002).
References
[1] Bruno Pradel, Nicolas Usunier, and Patrick Gallinari. Ranking with non-random missing ratings: influence of popularity and positivity evaluation metrics. In RecSys, 2012.
[2] Wenbo Hu, Xin Sun, Qiang Liu, Wu, and Liang Wang. Uncertainty calibration for counter- factual propensity estimation in recommendation. IEEE Transactions on Knowledge and Data Engineering. 2025.
[3] Benjamin M Marlin and Richard $ Zemel. Collaborative prediction and ranking with non- random missing data. RecSys, 2009.
[4] Chunyuan Zheng, Hang Pan, Yang Zhang, and Haoxuan Li. Adaptive structure learning with partial parameter sharing for post-click conversion rate prediction. In SIGIR, 2025.
[5] Haoxuan Li, Kunhan Wu, Chunyuan Zheng, Yanghao Xiao, Hao Wang, Zhi Geng, Fuli Feng, Xiangnan He, and Peng Wu. Removing hidden confounding in recommendation: unified multi-task learning approach. In NewrIPS, 2023.
[6] Chuan Zhou, Haoxuan Li, Lina Yao, and Mingming Gong. Counterfactual implicit feedback modeling. In NewrIPS, 2025.
10

---

[7] Wenjie Wang, Yang Zhang, Haoxuan Li, Peng Wu, Fuli Feng, and Xiangnan He. Causal recommendation: Progresses and future directions. Tutorial on SIGIR, 2023.
[8] Honglei Zhang, Shuyi Wang, Haoxuan Li, Chunyuan Zheng, Xu Chen, Li Liu, Shanshan Luo, and Peng Wu. Uncovering the propensity identification problem debiased recommendations. In ICDE, 2024.
[9] Yin- Wen Chang, Cho-Jui Hsich, Kai- Wei Chang, Michael Ringgaard, and Chih-Jen Lin. Train- ing and testing low-degree polynomial data mappings via linear svm. Journal of Machine Learning Research, 2010.
[10] Harald Steck. Evaluation of recommendations: rating- precing-pedittionn and ranking. In RecSys, 2013.
[11] Guido W. Imbens and Donald B. Rubin. Causal Inference For Statistics Social and Biomedical Science. Cambridge University Press, 2015.
[12] Yuta Saito, Suguru Yaginuma. Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing-not-at-randd implicit feedback. In WSDM, 2020.
[13] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations treatments: Debiasing learning and evaluation. In ICMI., 2016.
[14] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recom- mendation on data missing not at random. In JCML, 2019.
[15] Yuta Saito. Doubly robust estimator for ranking metrics with post-click conversions. In RecSys, 2020.
[16] Hao Wang. Wei Chang, Tianqiao Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. ESCM Entire space counterfactual multi-task model for post-click conversion rate estimation. In SIGIR, 2022.
[17] Harrie Oosterhuis. Doubly robust estimation for correcting position bias in click feedback for unbiased learning rank. ACM Transactions on Information Systems, 2023.
[18] Haoxuan Li, Chunyuan Zheng, Wenjie Wang, Hao Wang, Fuli Feng, and Xiao-Hua Zhou. Debiased recommendation with noisy feedback. In SIGKDD, 2024.
[19] Haoxuan Li, Chunyuan Zheng, Sihao Ding, Fuli Feng, Xiangnan He, Zhi Geng, and Peng Wu. Be aware of the neighborhood effect: Modeling selection bias under interference for recommendation. In ICLR, 2024.
[20] Harrie Oosterhuis. Reaching the end of unbiasedness: Uncovering implicit limitations of click-based learning rank. In SIGIR, 2022.
[21] Xin Zhang, Kai Wang, Zengmao Wang, Bo Du, Shiwei Zhao, Runze Wu, Xudong Shen, Tangjie Lv, and Changjie Fan. Temporal uplift modeling for online marketing. In SIGKDD, 2024.
[22] Hao Zhou, Kun Sun, Shaoming Li, Yangfeng Fan, Guibin Jiang, Jiaqi Zheng, and Tao Li. State: A robust estimator of heavy-tailed metrics for variance reduction in online controlled experiments. In SIGKDD, 2024.
[23] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 2023.
[24] Peng Wu, Haoxuan Li. Yuhao Deng, Wenjie Hu, Quanyu Dai, Zhenhua Dong, Jie Sun, Rui Zhang, and Xiao-Hua Zhou. On the opportunity of causal learning in recommendation systems: Foundation, estimation, prediction and challenges. In IJCAI, 2022.
[25] Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. Unbiased learning to rank with unbiased propensity estimation. In SIGIR, 2018.
11

---

[26] Dugang Liu, Pengxiang Cheng, Hong Zhu, Zhenhua Dong, Xiugiang He. Weike Pan, and Zhong Ming. Mitigating confounding bias in recommendation information bottleneck. In RecSys, 2021.
[27] Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. Causal intervention for leveraging popularity bias in recommendation. In SIGIR, 2021.
[28] Jinwei Luo, Dugang Liu, Weike Pan, and Zhong Ming. Unbiased recommendation model based on improved propensity score estimation. Journal of Computer Applications, 2021.
[29] Dugang Liu, Pengxiang Cheng, Zinan Lin, Jinwei Luo, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. KDCRec: Knowledge distillation for counterfactual recommendation via uniform data. IEEE Transactions on Knowledge and Data Engineering, 2022.
[30] Zinan Lin, Dugang Liu, Weike Pan, Qiang Yang, and Zhong Ming. Transfer learning for collaborative recommendation with biased and unbiased data. Artificial Intelligence, 2023.
[31] Shufeng Zhang and Tianyu Xia. CBPL: A unified calibration and balancing propensity learning framework in causal recommendation for debiasing. In IJCAI Workshop, 2025.
[32] Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. Autodebias: Learning to debias for recommendation. In SIGIR, 2021.
[33] Siyuan Guo, Lixin Zou, Yiding Liu, Wenwen Ye, Suqi Cheng, Shuaiqiang Wang, Hechang Chen, Dawei Yin, and Yi Chang. Enhanced doubly robust learning for debiasing post-click conversion rate estimation. SIGIR, 2021.
[34] Dugang Liu, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. A general knowledge distillation framework for counterfactual recommendation via uniform data. In SIGIR, 2020.
[35] Yuta Saito. Asymmetric tri-training for debiasing missing- cessinngnot-at-reee not-aat explicit feedback. In SIGIR, 2020.
[36] Shuqiang Zhang, Yuchao Zhang, Jinkun Chen, and Haochen Sui. Addressing correlated latent exogenous variables debiased recommender systems. SIGKDD, 2025.
[37] Shanshan Huang, Haoxuan Li, Chunyuan Zheng, Mingyuan Ge. Wei Gao, Lei Wang, and Li Liu. Text-driven fashion image editing with compositional concept learning and counterfactual abduction. In CVPR, 2025.
[38] Shanshan Huang, Haoxuan Li, Chunyuan Zheng, Lei Wang, Guorui Liao, Zhili Gong, Huayi Yang, and Li Liu. Visual representation learning through causal intervention for controllable image editing. In CVPR, 2025.
[39] Huayi Yang, Chunyuan Zheng, Guorui Liao, Shanshan Huang, Jun Liao, Zhili Gong, Haoxuan Li, and Li Liu. CAP: Causal air quality index prediction under interference with unmeasured confounding. In WWW, 2025.
[40] Chuan Zhou, Yaxuan Li, Chunyuan Zheng, Haiteng Zhang, Min Zhang, Haoxuan Li, and Mingming Gong. two-stage cerraining-fiettniigg framework for treatment effect estimation with unmeasured confounding. In SIGKDD, 2025.
[41] Peng Wu, Haoxuan Li, Chunyuan Zheng, Yan Zeng, Jiawei Chen, Yang Liu, Ruocheng Guo, and Kun Zhang, Learning counterfactual outcomes under rank preservation. In NeurIPS, 2025.
[42] Hao Wang, Jiajun Fan, Zhichao Chen, Haoxuan Li, Weiming Liu, Tianqiao Liu, Quanyu Dai, Yichao Wang, Zhenhua Dong, and Ruiming Tang. Optimal transport for treatment effect estimation. In NewrlPS, 2023.
[43] Hao Wang, Zhichao Chen, Zhaoran Liu, Xu Chen, Haoxuan Li, and Zhouchen Lin. Proximity matters: Local proximity enhanced balancing for treatment effect estimation. In SIGKDD, 2025.
12

---

[44] Benjamin Marlin, Richard S Zemel, Sam Roweis, and Malcolm Slaney, Collaborative filtering and the missing at random assumption. In UAI, 2007.
[45] Harald Steck. Training and testing of recommender systems on data missing not at random. In SIGKDD, 2010.
[46] Miroslav Dudik, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In ICML, 2011.
[47] Philip Thomas and Emma Brunskill. Data-efficient off- off-policy policy evaluation for reinforce- ment learning. In ICML, 2016.
[48] David Benkeser, Marco Carone, MJ Van Der Laan, and Peter B Gilbert. Doubly robust nonparametric inference on the average treatment effect. Biometrika, 2017.
[49] Stephen L Morgan and Christopher Winship. Counterfuctuals and causal inference. Cambridge University Press, 2015.
[50] Haoxuan Li, Chunyuan Zheng, Shuyi Wang, Kunhan Wu, Bric Wang, Peng Wu, Zhi Geng, Xu Chen, and Xiao-Hua Zhou. Relaxing the accurate imputation assumption in doubly robust learning for debiased collaborative filtering. In ICML, 2024.
[51] Hao Wang, Zhichao Chen, Zhaoran Liu, Haozhe Li, Degui Yang, Xinggao Liu, and Haoxuan Li. Entire space counterfactual learning for reliable content recommendations, IEEE Transactions on Information Forensics and Security, 2025.
[52] Hao Wang. Improving neural network generalization on data-limited regression with doubly- robust boosting. AAAI, 2024.
[53] Sihao Ding, Peng Wu, Fuli Feng, Xiangnan He, Yitong Wang, Yong Liao, and Yongdong Zhang Addressing unmeasured confounder for recommendation with sensitivity analysis. In SIGKDD, 2022.
[54] Haoxuan Li, Chunyuan Zheng, and Peng Wu. StableDR: Stabilized doubly robust learning for recommendation data missing not at random. In ICLR, 2023.
[55] Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. TDR-CL: Targeted doubly robust collaborative learning for debiased recommendations, In ICLR, 2023.
[56] Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang. Xiuqiang He, Rui Zhang, and Jie Sun. A generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In SIGKDD, 2022.
[57] Haoxuan Li, Quanyu Dai, Yuru Li, Yan Lyu, Zhenhua Dong, Xiao-Hua Zhou, and Peng Wu. Multiple robust learning for recommendation. In AAAI, 2023.
[58] Zijie Song, Jiawei Chen, Sheng Zhou, QiHao Shi, Yan Feng, Chun Chen, and Can Wang. Cdr: Conservative doubly robust learning for debiased recommendation. In CIKM, 2023.
[59] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, Zhi Geng, Xu Chen, and Peng Cui. Debiased collaborative filtering with kernel-based causal balancing. In ICLR, 2024.
[60] Wonbin Kweon and Hwanjo Yu. Doubly calibrated estimator for recommendation on data missing not random. In WwW, 2024.
[61] Mingming Ha, Xuewen Tao, Wenfang Lin, Qionxu Ma, Wujiang Xu, and Linxun Chen. Fine- grained dynamic framework for bias-variance joint optimization on data missing not at random. In NewrIPS, 2024.
[62] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confound- ing with few unbiased ratings in debiased recommendations, In WWW. 2023.
[63] Yanghao Xiao, Haoxuan Li, Yongqiang Tang, and Wensheng Zhang. Addressing hidden confounding with heterogeneous observational datasets for recommendation. In NeurIPS, 2024.
13

---

[64] Mark J. van der Laan and Sherri Rose. Targeted Learning: Causal Inference for Observational and Experimental Data. Springer, 2011.
[65] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[66] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Combating selection biases in recom- mender systems with a few unbiased ratings. WSDM, 2021.
[67] Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairee: A fully-observed dataset and insights for evaluating recommender systems. In CIKM, 2022.
[68] Yuta Saito and Masahiro Nomura. Towards resolving propensity contradiction in offline recommender learning. In IJCAI, 2022.
[69] Zifeng Wang, Xi Chen, Rui Wen, Shao-Lun Huang, Erean E. Kuruoglu, and Yefeng Zheng. Information theoretic counterfactual learning from missing-not-at-reann not-at-random feedback. In NeurIPS, 2020.
[70] Adith Swaminathan and Thorsten Joachims. The nelf-normalized estimator for counterfactual learning. In NeurIPS, 2015.
[71] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Measuring and enhancing balancing for recommendation. In ICML, 2023.
14

---

A Proofs
Lemma 4.1 (Bias of DR Estimator 14).
Given imputed errors ēu,, and learned propensities Pu, when considering only the randomness of missing indicators, the bias of DR estimator is
(pus Pu.i) {ens En,i)
Biaso (nD = D
Σ (u,J)DD
Pu.i
Proof of Lemma 4.1. The proof can be found in Lemma 3.1 of [14). However, should note that, as stated in the proof, "the prediction and imputed errors are treated as constants when taking the expectation, since ou does not result from any prediction imputation models [T3... The DR
estimator [4] is given as
(eus + Ou Ou.ileu. Eu.s)
(D (0)
Σ (u,))ED
D
Pu,i
By considering only the randomness Ou,i+ have
Eo (0)] Eo
1
Σ
(éus
Ouileu, Evii)) Duse
DD]
(u.j)D
1
Buus + Pu,ileu, Baid)
D
Du.i
(u.))DD
By definition, the bias of the DR estimator is
Biaso (0)] Eideat (0) Eo (ED
Pu,sleui - Ev,i))
[D]
Cu,i
[D]
Σ
Cu.i
Du,i
(u,))D
(u,i)DD
(Pu.i Pu,s) feu,i Cn,ih
D
Puçi
(u,i)ED
which yields the stated results.
The DR estimator is unbiased when either imputed errors Eu,i or learned propensities Pa.i are accurate for all user-item pairs, i.e., either lugi Cu,i or Pu.i for all and
Corollary 4.2 (Double Robustness 14]
Proof of Corvllary 4.2. The proof can found Corollary 3.1 Appendix 14] However, one should note that, stated in the proof. "the prediction and imputed errors treated as constants when taking the expectation, since ou,i does not result from any prediction or imputation models [13]
Let oy,; Cu,i u and A.,s Pu.e-Pu..s On the hand, when imputed errors are accurate, we have Futi
du,i for (u, i) D. In such case, we can compute the bias of the DR estimator by
(EDSO(ERR(O)))
1
Σ
1
Σ A =0 0.
D
D
u.IDD
U,EDD
On the other hand, when the learned propensities are accurate, we have Au,5 = 0 for (u,1) E D. In this case, we compute the bias of the DR estimator by
Bias (EDR) =
Σ
Σ 0. 0.
D
[D]
UNEDD
In both cases, the bias of the DR estimator is zero, which means that the expectation of the DR
estimator over all the possible instances of o,, is exactly the same as the prediction inaccuracy. This completes the proof.
D
15

---

Theorem 4.3 (Bias of DR Estimator under super-populations Given enror imputation model é and propensity model Pu.i then the bias of the DR estimator for estimating the expected ideal loss under super-population is
Biasp Biasp[EDR(0)
Cov
Pu.i
Ous
Cu.i Eup
Ou.i
Fu..3
[Ble Fu,al - Eleu, sudl
Pu.é
Pusi
eguuala if independent
equalr O eiiher /llo../Bu..: (Tu.)) Brr..... Ra.il =
Proof of Theorem 1. Instead of considering only the randomness of the rating missing indicator, in the following, we treat all variables, including imputed errors and learned propensities, as random variables. Formally, we have
{0u.5
Pu,i) fens (.ri) pu,
Bias (0) Lidtea (0)] [[))n(()] El
E
Cu,e
Pu,i Out
E
E
{en.s Enei) Fiuz
(by the double expectation formula)
Pugk Out
Pu.i Outi
Puii ouii } {(ea, Ev,i) Bleun eu,il + Bleut ewiil Fu,k
E
E
Dusi
Dus
Dus
Pusi Ouii
Du.i Ouii
} fleus Cna) Eleee Eugel Fn.s)
E
E
Pu.i
Pus
+
-
Ou,i
[Blea
Lusl Bleun munil]]
Pu,i
Cov
Paui
Ou,
Cuis
Cu,
+
E
Ovei Pu,t
[Bleu,s Lu.i Bieu, [usll]
Pu,i
which yields the stated results.
Corollary 4.4 (Double Robustness under Super-population. Under super-population, the DR estimator unbiased when both the following conditions hold:
(i) Either learned propensities satisfy Blou.s/Pu. Fu,s] 1, or imputed errors have the same conditional expectation with true prediction errors Blêu,i Fu.il B[eu,i zu,i);
(ii) The covariance term vanishes, that is, Cov (Buss (Dug-Pad Cuue Cu.i) 0.
Proof of Corollary 4.4. First, when condition (ii) holds, i.e., Cov((pu, - On.:) Puras Buei - Eu.s) =
follows from the results in Theorem 1 that
{ E Ouai
}
Bias|EDR (0)
E
Bleun Fu.il - BB(eu, tu
Pu,i
On the hand, when the learned propensities satisfy Blou, Purt Fuua) = 1. In such case, we can compute the bias the DR estimator by
Bias[EDR (0)) E [0- {E(ew,a [nna)) E (en
[w,]]]] 0.
On the other hand, when imputed errors have the same conditional expectation with true prediction eтrors, we have E(êu, xa.i] = Bley,: Lu,]]. In this case, we can compute the bias of the DR estimator by
Bias[EDR (0) E {1
Ou.
}
Tu,s
0.
Pu,i In both cases, the bias of the DR estimator is zero, which completes the proof.
Definition 4.5 (Empirical Covariance). The empirical expected conditional covariance between
(fu,i Pu., and En,i - Ên,i is
Puss Ou,k
Pu.i Ou, (eu, b.,i)-
Cov
Cu, Buyi
Pu.i
D
Pu,i
(u,))D
16

---

Definition 4.6 (Empirical Rademacher Complexity [65]). Let F be family of prediction models mapping from E X' to bl. and {au,a (u, E D fixed sample of size [D] with elements in X. Then, the empirical Rademacher complexity of F with respect to the sample S is defined as:
R(F) 10 (-1,t))))
sup feeT
Σ
On,iEu.
D
(N.J)D
where fo (u, 8) E D), and uuii independent uniform random variables taking values in {-1, +1}. The random variables ou,i are called Rademacher variables.
Lemma A.1 (Rademacher Comparison Lemma [65]
Let F be family of real-valued functions
to (a, b), and (Iu, (u, i) E D) a fixed sample of size (D] with elements in X. Then
]
sup
B1((2)] 2N0
$ (Zmaa) (u,J)C
Σ
13 Siupini
sup SEF
Σ
owif (Fui)
Swa
D
D
(u.j)CD
where σ = -ouu;: (u, E D), and σu are independent uniform random variables laking values in [-1, The random variables ou, are called Rademacher variables.
Proof of Lemma A.I. The proof can be found in Lemma 26.2 of [65
Lemma A.2 (MeDiarmid's Inequality (65)) Let V be some set and let f V + be a function of m variables such that for some >> 0, for all E [m]] for all 21...., Im x²₂ E V we have
If (I1,..., Im) f (Z1,.... I1-1, 2/, Iithy... Imm S
Let XI...., Xm be m independent random variables taking values in V. Then, with probability of at least 1 8 we have
[ Xm) E (X1
Xm s of log 8
m/2
Proof of Lemma A.2. The proof can be found in Lemma 26.4 of (65).
Lemma A.3 (Rademacher Calculus 65 For any A C I",, scalar E R, and vector an E Rm we have
R (ffaa + a a E A}) S [cA)
Proof of Lemma A.3. The proof can be found in Lemma 26.6 of 65
Theorem 4.7 (Controllability Empirical Covariance). The corrected imputation model trained by Luu is sufficient for controlling the empirical covariance.
(i) For user-item pairs with observed outcomes, the empirical covariance is O. Formally, we have
0 L de
Pu - Ou.i (Cu,s Cu,i) = 0:
(b,c)
0, which is equivalent to
Σ
|D]
Pu,
t==e
(u,f):
on.==1
(ii) For user-item pairs with missing outcomes, suppose that pu,s 2 Ki and Jen, - Eusll I Ko, then with probability at least 1 77, have
1
Σ
Pu,i Ou.i Pu,s
(eu,s Tu.s)
D
(a,d):
=0
2 log( (4/m)
K 1 Kip 2R(F)+ (2K, + +1)
D
D
propored loue
eupprrical biss from propensity model
bound controlled ampiricat Rademacher complestty somple
where the K K o7 constants.

---

Proof. For the proof of Theorem 4.7(), first recap that the proposed boosted imputation model is
E.. ( x) e(Ou.5 - ( v)))
and the proposed corrected imputation loss function for training the boosted imputation model is
Ou,ileus E,,a)
( -
arg min L. (0)
P.c
D
Pu,
(a,i)ED
By taking the partial derivative with respective to the above formula and setting it to zero, we
have
Du.s -ut
L.
L.uu
(0,
0, which is equivalent
Σ
(eu,i En,i) O,
0e
D
Pus
lc=e
(u,4); ou
which proves the empirical convariance on the observed outcomes is O. For the proof of Theorem 4.7(ii), by noting that
Pa.s Ou, (En,i En,i)
1
Σ
(en,i-Eu,i)
(eng - En.i)
D
Pu.i
[D]
[D]
(u,,): =0
(u,i): ou.;=D
(u.i)ED
we now focus on bounding the last term of the above equation with the least probability.
Suppose that Pu.i 2 K and leu,i Eu,il I K2, then
Σ
(Cu,i Eu,i)²
1
Ou.ileus Eus))
+
Σ
(Cu,t Eu,i)
(D]
[D]
Pa.s
[D]
(u,D)D
(M,D)D
On,ilen,t Emui)
Ou,ileu,i - Eur,)²
Ou,ilen,t - Eni)?
+ E
D
Puji
D
Pu,i
D
Duçi
(M,))DD
(a)))D
(M.))D
Ou.ilCa,i (un))
Ou,ileus Cu.i)
+
(eu,i E0,s) 22
D
Puçk
D
D
Pu,
(u,))ED
(u,))D
(u,i)ED
Ou,i - eu
Oujileus Enne)2
+
E
D
Paji
[D]
Du,
(M.
(a.i)DD
(su + K3
1
Ou!i
D
Du.i
(u,))E
ou, (Euri Ex.i)²
1
Oureleu, Eni))2
+ sup foEJ
E
Priji
D
Puii
(u.))D
(u,))ED
For simplicity, we denote the last term in the above formula as
Ou,i (en,i Eu
Ouslenii - Eus)2
B(F) = sup
E
Σ
Σ (u.))D
foe
D
Pu.s
D
Priis
(u,iED
we then aim to bound B(F) the following.
Note that
B(F)
E B(F)
B(F)
S-PIR [B(F)
E
|B
S-P
where the first term is
[FF),, and by Lemma A. we have
SNPID
1 [ (u.d)ED
ou.i(ey, - )
E
[BF]]
2
E
EE Egn (-1+111)) sup FocF
ou,
S~PID
Duo
18

---

By the assumptions that Pu,i 2 K and lu, - Eu, S Kg we have
E [BF]]
& 2K
1+
1
E Supiii
(-1
sup DoE
Σ
ous(eut - Ev,))
S-PiPi
Ky
D
(u,i)ED
1
=2Ks
11
E Supiz1
(R(F))
K
where the last equation directly from Lemma A.3, and R(F) is the empirical Rademacher com- plexity
R(F) (-1,--1)
sup feEF
Σ
Ju,iCus
D
(N.))D
where a ao (u, E D), and du,i are independent uniform random variables taking values in +1). The random variables oa,i are called Rademacher variables.
2 By applying McDiarmid's inequality in Lemma A.2, and let TDj with probability at least 2.
log(4/n) 2|D]
2
Blg((4/))
R(F)
E S-pizl
[R(F)}
s 2
K
D
For the rest term B(F)
E [B(F)], by applying McDiarmid's inequality in Lemma A.2 and the
S-P ml
2K3 1+ K
assumptions that Pu,i 2 K and leu,s Eu,ll S Ko let c=
then with probability at
least 1
B(F)
E SwPiP
[BF)]
< 223
1+
[ Ky
log log(4/n) 2
K3
1
1
1
2
log(44/))
K
[D]
now bound B(F) combining the above results. Formally, we have
B(F) E SNPIDI
[BF) +
B(F)
E
[B(F))
S~elD
2 1+
(R(F)} + ((F))
E S~EiD
[B(F))
K
S-F
With probability at least 1 - 7). we have
B(F) < 2Kg (1+ K
1+ 1) R(F) + +R(F)+KO)
1og(4/1))
+ K2
1
2og(44m) 4/n)
D
K
[D]
= =Ko (1 [x/
18 log (4/v)
2R(F) + Koy
Kyp
D
19

---

We now bound the empirical convariance on the missing outcomes combining the above results. Formally, have
Pu.i Ouii
(euii - Eu.i) s
1
Σ
(Ca.s - Pu,i) (Bas)
(D]
Puçi
DD]
(u,,);: Pu
(a.0)D
Le
(b,
SSup (),c)++
+
K2
1
E
Ou,i Pu,e
+ Ks
1+ Re)
2R(F) + k
18 18log(4/1)
(
[D]
M,iED
< C.S (0, + Ks.
Σ
1
Ouii
Fu.s
+
D
Pusi
K (1 x,x
18 log (4/n)
2R(F) + Koy
Kgp
which yields the stated results.
Corollary 4.8 (Relation to previous imputed errors), The learned coefficient e² will converge to when the imputation model εu,i has zero empirical covariance, making εu, degenerates to e.,i.
Proof of Corollary 4.8. Note that e" is solved by minimizing
Ou,(
tu,i c(Ou,i - D),
Σ
D
Pu,i
(u,j)ED
Taking the first derivative of the above loss with respect to and setting it to zero yields
Σ
Out
(eus Cu,i
c(ou, Puril } (Ou,i Pu.i) O,
Pusi
(u.))D
which implies that
Ouii feu,i Eu,i. (on. Pu,i)) = =0,
Pu.i
(u.i)ED
from which implies the uniqueness Formally, if ê.,s already satisfies zero empirical covariance on the observed outcomes, then = 0 is solution of the above equation. Let é be another solution of the above equation. Since the solution of equation is unique, then will converage making εu,i degenerates to Ê.,i.
Corollary 4.9 (Bias reduction property). The proposed corrected imputation loss leads the smaller bias of imputed errors En,i. when Ên,i inaccurate. Formally, we have
Ou.ileus - ēu,)) < min Co(0)
1
Ou.ileu,i - Eu,i)
min C
Sup (0, E)
D
Du.i
D
Du.i
(a.))D
(a.jED
Proof of Corollary 4.9. The result holds by noting that
min
Sup (Ø. & min L (Spp c c 0) = min (c =
1
Σ
Ou,ileu, - Cur)? Da.s
6,6
|D]
(u.i)D
Corollary 4.10 (Variance reduction property). The proposed corrected imputation loss leads the smaller variance of ēy when the optimal e* lies in a certain range. Formally, we have
v(e..)) V(ens + (0u,s - Pu,))) S V(e,,)), if e 0, Cov(êu, Pu,s Ou.i)
V(pn. - Ov,i)
20

---

Proof of Corollary 4.10. First, we note that V( equals to
V(éu.)) 2e Cov(éu,i,Pur ou,s) + (c")" - Pu,i).
which serves as a quadratic function with respect e". By taking the partial derivative respective c² of the above formula and setting it to zero, the optimal c* with the minimal variance is given as Cov(Eu, Pu,i Ou.i)
V(pu, - Ou,i)
By exploiting the symmetry of the quadratic function, we have V(Gu,)) =((en, + (0u.: (0u,i-Pu,s)) (uu,)) S V(eu,)) if c E o, 2- Cov(en,, Pu.i Ons)) V(pu,i Ou,z)
Theorem 4.11 (Generalization Bound under Superpopulation). Suppose that Pu, K and C leus Eu,il} I Ka then with probability at least 1 77, have
Lideat (0) & LSuperDR (0)
Σ
-
Ou,i
Eleu, Fu,il Eleu, Fa,el
D
Pu,i
(a.i)ED
vanilla DR only controls the empirical DR loss, and empirical risks of imputation and propensity models
Eaiit) )
1
Cov
Ou,
Park
(1+
1 K (2((F) (2R(5)+ Kay
18 log 44/))
D
Pu,s
D
(u,i)ED
corrected loss further controls the independence
tail bound controlied by empirical Rademacher complexity and sample size
Proof of Theorem 4.1
First we decompose the ideal loss as follows. Lideat (0) LDn (0) (Lideat (0) [[[DR(())) + (E[Lon (0) (DR(
= LD (8) + Bias[LDR (f)] + (E[[DR(O)] - (D)(
$ (0)
[ias (D)
Cuss + Ou.aleui bu,i)
Oajileui Cuii) Du,
Σ
Σ (u,i)ED
sup foe
E
ên,i
[D]
Du,
|D]
(u.i)ED
For simplicity, we denote the last term in the above formula as
e + Oualeuni - (
OuilCun - Cuyi)
B(F) = sup feeT
E
D
Pu,
D
Dus
(a.i)ED
(a,i)ED
we then aim to bound B(F) in the following.
Note that
B(F)
E [(F)] B(F)
S~PiR [B(F)
E
[B
S-P
where the first term is
E SAPID
[BF)], and by Lemma A.1 we have
Cu,ibu, + Gu.Duu,(eus - Eu.s)
E. [BFF) <
E a~ (-1(-)))
sup FOCF
Σ
SupiP)
BAPIDI
D
Du,i
(a,i)D
By the assumptions that Pu,i 2 K2 and min minfdu,ir Ceui Eu,il) < Kg, we have
Ga. Bu.a)
E B(F)
sup 2 E (-- SwPIDi
SapiD
foeT
Puçi
(u,1)E
2 1 K2
E Swaloy
(R(F))
21

---

where the first equation is from Lemma A.3, and R(F) is the empirical Rademacher complexity
R(F) (-
sup FeEF
D
(a.J)ED
where a {oun.s (u, i) E D), and ou,i are independent uniform random variables taking values in {-1,+1) +1). The random variables uu,i are called Rademacher variables.
2 with probability at least y. By applying McDiarmid's inequality in Lemma A.2, and let
[D]
1og(4/n)
2 llg((4/n)
R(F)
E S-piz)
[R(F)}
S 2
K
2|D]
D
For the rest term B(F)
E S~piDl
[BF)), by applying McDiarmid's inequality in Lemma A.2 and
2K, 1+ Kg
the assumptions that Pu,s Ky and min lea,i En.il3 S Ko, let probability at least 1 - 4.
then with
S 2Ko (1+ 1
log(4/n) 2|D]
2 log (4/n)
B(F)
E S-FID
[B(F]
Ko
1 +
K
K
[D]
We now bound B(F) combining the above results. Formally, we have
B(F)
E S~PiDI
[B(F) + B(F)
E
[B(F]]
SNpiPi
2 1 K
E S-Pim
(R(F)) ((F))
[BFF)
S-P
With probability least 1 17, we have
1 1
2 2lo((4/n)
1
2 2log(4/n)
B(F) 2 1
R(F) + Koy
Kss
11
K
D
K
[
18 log( (4/g)4))
2R(F) + Koy
K
D
We now bound the ideal loss combining the above results. Formally, we have Lideat(0) S (DR( + [ias|[DR + B(F)
+ 1+ - (2((F) + Kay
18 log (log(44))
< (0) +
K
[D]
In Theorem 4.3, we have already prove that
(uut Out
Bias (EDE(())))))))
Cov
Cuai Ba.s
D
Pusi
(u,i)eD
1
E
Ou.
Lue } {Eleuu xa,eel Elêu, æu.il
D
Dens
(u,t)D
Cov
Ou,i
Durk
Cu,i Euis)
D
Prii
(
- E
Oud Pu,i
B Bleaa, (uu.)) Be(Cu, su.el
D
(u,i)ED
22

---

therefore with probability at least 1 - 77, we have
Ouai
Lideat (0) S (0) +
E
Tuii
B Bleu, Tu,al - Blea, Tu,il
D
Putt
(u.))ED
Cov
Ou,i
Pul
Ens - Enss)
+ +(+ 1 Rn)) 2R(F) + Koy
18 log 4/n) [D]
P
Duji
[u,
which yields the stated results.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide main claims and contributions in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
We provide a complete proof in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed description of the experimental process in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Data and code are open-source in GitHub. Link is provided in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttpcccpppbccccceee guwde//ddeeeeee Cooeeeoodeesssseeeeeee /tte://bb///uudddd//eeeeeeiessss swiiisisisiyyccyyccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide the experimental setting and details. See details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report standard deviations and statistical significance in the main compara- tive experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics and paper conforms with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research does not have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The assets used have been properly noted and credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are being released.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We not have any studies or results regarding crowdsourcing experiments and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do have any studies or results including study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core methodological development in this research does not involve large language models (LLMs) as essential, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standarr components.
Please refer to our LLM policy battps://neune / for what should or should be described.
30


Reviews:
[{"review_id": "B3Sav4fd8Z", "reviewer": "Reviewer_UUJ9", "summary": "The paper proposes a novel doubly robust estimator with a joint learning algorithm to address the limitation of selection bias in designing the recommender systems.", "questions": "Can this method be applied in a real-time setting where data arrives sequentially? In other words, is the proposed approach capable of efficiently adapting or updating its predictions as new data becomes available, without requiring full retraining?", "limitations": "See above.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-organized and easy to follow. It presents strong results on large-scale datasets, which highlight the effectiveness and practical relevance of the proposed method. Additionally, the inclusion of a generalization bound provides valuable theoretical support, offering insights into the method's expected performance beyond the training data. However, further clarification or empirical validation of the bound would strengthen the contribution even more.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for efforts in evaluating our work and helpful suggestions. Below, we will try our best to address your concerns and question.\n\n> **W1:** Further clarification or empirical validation of the bound would strengthen the contribution even more.\n\n**Response to W1:** We thank the reviewer for the useful suggestions.\n\n- Further clarification of error bounds:\nGiven the propensity model $\\hat p_{u, i}$ and imputation model $\\hat e_{u, i}$, satisfying $\\hat p_{u, i}\\geq K_{\\psi}$ and $|e_{u, i}-\\hat e_{u, i}|\\leq K_{\\phi}$, then with probability at least $1-\\eta$, we have\n\n$$L_{\\text{ideal}}(\\theta) \\leq \\underbrace{{\\frac{1}{|\\mathcal{D}|} \\sum_{(u, i) \\in \\mathcal{D}}\\big[\\hat e_{u, i}+\\frac{o_{u, i}(e_{u, i}-\\hat e_{u, i})}{\\hat{p}_{u, i}}\\big]}} _{\\text{empirical DR loss}} $$\n\n$$ +  ~\\underbrace{\\frac{1}{|\\mathcal{D}|} \\sum_{(u, i) \\in \\mathcal{D}}\\left|1-\\mathbb{E}\\left[\\frac{o_{u, i}}{\\hat p_{u, i}} | x_{u, i}\\right]\\right| \\cdot\\left|\\mathbb{E}\\left[e_{u, i} \\mid x_{u, i}\\right]-\\mathbb{E}\\left[\\hat e_{u, i} \\mid x_{u, i}\\right]\\right|} _{\\text{empirical risks of imputation and propensity models}}$$\n\n$$ + ~\\underbrace{\\left|\\frac{1}{|\\mathcal{D}|} \\sum_{(u, i) \\in \\mathcal{D}} \\operatorname{Cov}\\left(\\frac{o_{u, i}-\\hat p_{u, i}}{\\hat p_{u, i}}, e_{u, i}-\\hat e_{u, i}\\right)\\right|} _{\\text{the empirical covariance}}$$\n\n$$ + \\underbrace{\\left(1+\\frac{1}{K_\\psi}\\right)\\left(2 \\mathcal{R}(\\mathcal{F})+K_\\phi \\sqrt{\\frac{18 \\log (4 / \\eta)}{|\\mathcal{D}|}}\\right)} _{\\text{tail bound controlled by empirical Rademacher complexity and sample size}},$$\n\nwhich includes four terms: the empirical DR loss, the empirical risks of imputation and propensity models, the empirical covariance, and the tail bound. Even if the $\\hat p_{u,i} = p_{u,i}$ or $\\hat e_{u,i} = e_{u,i}$, the empirical covariance does not vanish in the super-population scenario. **The previous DR-based method can only control the first two terms, while our method can further control the empirical covariance by subtly modifying the imputation model (introducing one learnable parameter), without adding new modules and increasing time complexity.**\n\n- Further empirical validation of error bounds:\n  - In our manuscript, we have demonstrated that **our method can control empirical covariance on the observed samples on three widely used real-world datasets.** \n  - Furthermore, we validate **empirical covariance control on the whole dataset by conducting a semi-synthetic experiment on the Movielens 100k dataset.** Following previous studies [1-4], we use matrix factorization to generate the full binary rating matrix as the ground truth, the propensities for generating MNAR observed data, as well as the embedding for each user and item. Then we randomly select 50% / 70% users and items into our training data and train DR-JL, DCE-DR, and our method on the training data, finally calculating the empirical covariance (EC) on the whole dataset.\n\nThe results are shown below:\n||Ratio = 50%||Ratio = 70%||\n|-|-|-|-|-|\n|Method|AUC|EC on Whole Data|AUC|EC on Whole Data|\n|DR-JL|0.8855|-3.3429|0.9104|-3.0932|\n|DCE-DR|0.8941|-1.9507|0.9169|-2.0315|\n|Ours|**0.9017**|**-0.9428**|**0.9214**|**-1.5318**|\n\n**The results show that our method can control EC the most, which verifies the validity of the Controllability of Empirical Covariance (Thm 4.7), as well as we may a tighter error bounds.**\n\n> **Q1:** Can this method be applied in a real-time setting where data arrives sequentially? In other words, is the proposed approach capable of efficiently adapting or updating its predictions as new data becomes available, without requiring full retraining?\n\n**Response to Q1:** We thank the reviewer for the insightful question. This is a very common scenario in practice. For example, in the CTCVR scenario, the new MNAR data comes daily. **To evaluate whether our method can be efficiently applied in this scenario, we propose two ways to efficiently update our predictions:**\n- We store the parameters in models and optimizers at the end of training, and fine-tune them when new data comes (named \"continue training\").\n- We freeze the parameters in prediction, imputation, and propensity model, and add a learnable shallow residual structure, which is zero-initialized when new data comes (named \"adding residual\").\n- Also, we consider the full retraining of all parameters on both old and new data as the oracle baselines.\n\n**We add experiments on the industrial dataset KuaiRec.** To process the data, we first randomly split 80% users and items to construct the training set to make sure we do not see all users and items during training. Then we consider the following two scenarios:\n- **The new data has the same distribution compared to the training data (Consistent in distribution).** We randomly split the user set and item set into three parts with a ratio of 50/25/25. First, we train our method on the data with 50% users and 50% items, and then retrain the model on the remaining data sequentially, i.e., twice in total.\n\n- **The new data has a different distribution compared to the training data (Inconsistent in distribution).** We randomly split the user set and item set into two parts with a ratio of 50/50, and train our method on the first dataset. Then we assign a propensity according to the item frequency and the user's active level to sample the first half of users and items into the first time retraining, and the remaining into the second time retraining. We control the sampling process to ensure the sample number is the same in the two retrainings.\n\nFor all methods, we test on the MAR dataset. The results on the **KuaiRec** dataset are shown below. We bold the better results between ours and DR-JL.\n\n|Data||Consistent in distribution|||Inconsistent in distribution|||\n|-|-|-|-|-|-|-|-|\n|||AUC|NDCG@50|Recall@50|AUC|NDCG@50|Recall@50|\n|Train Set1 (50%)|Ours|**0.825**|**0.636**|**0.678**|**0.825**|**0.636**|**0.678**|\n||DR-JL|0.817|0.623|0.663|0.817|0.623|0.663|\n|Train Set2 (25%)|Continue training on ours|**0.819**|**0.628**|**0.664**|**0.821**|**0.633**|**0.669**|\n||Continue training on DR-JL|0.808|0.617|0.647|0.811|0.631|0.666|\n||Adding residual on ours|**0.811**|**0.614**|**0.648**|**0.821**|**0.640**|**0.680**|\n||Adding residual on DR-JL|0.805|0.599|0.639|0.816|0.620|0.653|\n||Full training on ours|**0.826**|**0.638**|**0.679**|**0.827**|**0.640**|**0.679**|\n||Full training on DR-JL|0.819|0.625|0.665|0.821|0.626|0.665|\n|Train Set3 (25%)|Continue training on ours|**0.819**|**0.629**|**0.660**|**0.729**|**0.492**|**0.504**|\n||Continue training on DR-JL|0.817|0.617|0.654|0.712|0.457|0.449|\n||Adding residual on ours|**0.824**|**0.627**|**0.658**|**0.725**|**0.479**|**0.506**|\n||Adding residual on DR-JL|0.823|0.614|0.653|0.715|0.450|0.449|\n||Full training on ours|**0.828**|**0.640**|**0.679**|**0.828**|**0.640**|**0.679**|\n||Full training on DR-JL|0.822|0.627|0.668|0.822|0.627|0.668|\n|\n\nWe find that our methods can efficiently adapt our predictions as new data becomes available. **An interesting finding is that when there is a large gap between the new data and the original training data (such as cold users and items, as shown in the \"Train Set3\" in the \"Inconsistent in distribution\" scenario), though the performance of our method will drop, our method significantly outperforms DR-JL. One possible reason is that our method deals with a super-population scenario, which is more robust toward a huge distribution shift.**\n\nWe sincerely thank you for your feedback, and welcome any further technical advice or questions on this work. We will do our best to address your concerns.\n\n---\nReferences\n\n[1] Recommendations as Treatments: Debiasing Learning and Evaluation. ICML 2016.\n\n[2] Doubly Robust Joint Learning for Recommendation on Data Missing Not at Random. ICML 2019.\n\n[3] Enhanced Doubly Robust Learning for Debiasing Post-Click Conversion Rate Estimation. SIGIR 2021.\n\n[4] Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference for Recommendation. ICLR 2024."}, {"review_id": "afgzYBfBSn", "reviewer": "Reviewer_g41m", "summary": "This paper introduces SuperDR, a novel doubly robust estimator for debiasing recommendation models under a more realistic super-population scenario, where the joint distribution of features and ratings remains the same across training and test sets, but the specific user-item pairs may not overlap. This setting contrasts with the conventional exact matching assumption commonly made in prior work. The key insight is that, under the super-population setting, existing DR estimators remain biased due to an uncontrolled covariance term, even when the imputation and propensity models are correctly specified. To address this, the authors propose a correction mechanism through a learnable parameter $\\epsilon$, which is added to the imputation model. This adjustment aims to reduce the empirical covariance $\\hat{\\text{Cov}}\\Big({\\hat{p}_{u, i} - o_{u, i} \\over \\hat{p}_{u, i}}, e_{u, i} - \\hat{e}_{u, i}\\Big)$. The paper presents supporting theoretical results—including a generalization error bound—and demonstrates, through experiments on three datasets (Coat, Yahoo! R3, and KuaiRec), that SuperDR outperforms existing debiasing methods.", "questions": "Nothing", "limitations": "The authors properly address the limitations in Section 6.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "- **Strengths**\n    - The shift from exact matching to super-population is realistic and meaningful, especially for large-scale industrial recommendation systems, where user-item overlap between training and test sets is uncommon.\n    - The paper establishes a strong theoretical foundation, with clearly articulated lemmas, corollaries, and proofs. Notably, the identification and treatment of the covariance term addresses an important and underexplored aspect of doubly robust (DR) estimation.\n    - The correction to the imputation model using $\\epsilon (o_{u, i} - \\hat{p}_{u, i})$ is elegant and well-motivated by targeted maximum likelihood estimation (TMLE).\n    - The derivation of the generalization error bound incorporating the new covariance term and empirical Rademacher complexity is both novel and informative.\n    - Experiments on multiple datasets consistently demonstrate performance improvements over a broad set of DR- and IPS-based baselines.\n    - The sensitivity study on the learning rate for $\\epsilon$ and sample ratio shows the method’s robustness under different bias severities and hyperparameter configurations.\n    \n- **Weaknesses**\n    \n    Nothing", "was_revised": true, "final_justification": "The paper is well-organized, and the proposed method shows strong performance compared to the baselines. The authors also responded sincerely and appropriately during the discussion period to the weaknesses and related questions raised by other reviewers. In particular, I believe the authors provided a sufficiently convincing response to the concern raised by another reviewer regarding the Assumption of Equal Joint Distributions. However, the concern about Dependence on Propensity Pre-training still appears to remain partially unaddressed. That said, the motivation behind the problem, the proposed solution, and the logical progression presented by the authors offer a meaningful contribution to the field.", "rebuttal": "Response: Thank you for your time and dedication in reviewing our manuscript! We are really glad that you are enjoying our paper and give our paper very positive comments! We believe that our study makes an important contribution to the field of debiasing in RecSys and causal inference, and we hope that you will support our research in subsequent discussions."}, {"review_id": "SX7KLMGblN", "reviewer": "Reviewer_gNuC", "summary": "This paper investigates a practical and underexplored challenge in unbiased learning: selection bias in scenarios where training and test sets do not share identical units (e.g., different users/items in recommender systems), though the joint distribution of features and outcomes remains the same. The authors argue that existing doubly robust (DR) estimators, which assume exact matching, are insufficient in this \"super-population\" setting. They derive the theoretical bias of DR under this new scenario and identify a non-trivial covariance term that induces additional bias. To address this, the paper proposes SuperDR, a novel estimator with a corrected imputation model designed to control this covariance term. They provide theoretical guarantees on bias, variance, and generalization error, and validate their method on three real-world datasets, demonstrating superior performance over existing approaches.", "questions": "Please refer to the Weaknesses.", "limitations": "Yes.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n**S1. Principled Solution**: The proposed SuperDR method is well-motivated and theoretically grounded. The use of imputation correction inspired by targeted learning is elegant and practical.\n\n**S2. Strong Empirical Results**: Experiments across three datasets (Coat, Yahoo! R3, KuaiRec) demonstrate that SuperDR consistently outperforms strong DR baselines, particularly in low-overlap scenarios between train and test data.\n\n**Weaknesses**\n\n**W1. Assumption of Equal Joint Distributions:** The super-population scenario assumes $P_{\\rm train}(x,r)=P_{\\rm test}(x,r)$; in practice, covariate drift or nonstationarity could violate this and undermine unbiasedness.\n\n**W2. Dependence on Propensity Pre-training:** Requires a pre-trained propensity model (e.g., logistic regression) whose miscalibration could propagate through the correction step, yet sensitivity to propensity errors is not deeply explored.\n\n**W3. Computational Overhead:** Jointly learning three components (prediction, imputation, and ε) may increase training time and tuning complexity compared to standard DR methods; scalability to very large item × user spaces warrants further study.\n\n**W4. Limited Domain Scope:** Experiments focus on recommender-system datasets; applicability to other MNAR settings (e.g., causal inference, medical missingness) remains to be demonstrated.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for efforts in evaluating our work and helpful suggestions. Below, we will try our best to address your concerns and questions.\n\n> **W1. Assumption of Equal Joint Distributions:**   The super-population scenario assumes $P_{\\rm train}(x,r)=P_{\\rm test}(x,r)$; in practice, covariate drift or nonstationarity could violate this and undermine unbiasedness.\n\n**Response to W1:** We thank the reviewer for the question. **However, we think there is a huge misunderstanding.** \n- Note that we assume $P_{\\rm train}(x,r)=P_{\\rm test}(x,r)$ means that the target data distribution (not the observed data distribution) in training is the same as the distribution in test data. Then, due to the MNAR issue, the observed data distribution $P_{\\rm obs}(x,r)$ is different from the test data.\n-  For example, the Coat dataset includes 290 users and 300 items with 6,960 MNAR observations, then **we only assume target data distribution in training $P_{\\rm train}(x,r)$ (the distribution of 290 * 300 = 87,000 user-item pairs) the same as the test distribution $P_{\\rm test}(x,r)$, and we do not assume distribution on 6,960 observations $P_{\\rm obs}(x,r)$ the same as the test distribution.** We will make it clearer in our revised manuscript.\n\n> **W2. Dependence on Propensity Pre-training:** Requires a pre-trained propensity model (e.g., logistic regression) whose miscalibration could propagate through the correction step, yet sensitivity to propensity errors is not deeply explored.\n\n**Response to W2:** We thank the reviewer for the question. The existing DR-based training algorithms are mainly divided into two types:\n\n- The first is based on the joint training algorithm, which pretrains the propensity model, and then trains the imputation model and the prediction model, such as DR-JL [1], MRDR-JL [2], DCE-DR [3], etc. \n- The second is based on the multi-task learning algorithm, which learns the propensity model, the prediction model, and the imputation model together, such as ESCM2-DR [4], SDR [5], DR-V2 [6], etc.\n\nAs suggested by the reviewer, **we added experiments on all three datasets, Coat, Yahoo, and Kuairec, where the propensity model is 1 layer (i.e., logistic regression), 2 layers, and 3 layers neural networks, respectively, to explore the sensitivity to propensity errors of our method. In addition, we also implemented both joint learning and multi-task learning as the learning algorithm to investigate whether our method needs a pre-trained propensity model.** The results are as follows:\n\n|Num of layers in propensity model|**Coat**|||**Yahoo! R3**|||**KuaiRec**|||\n|-|-|-|-|-|-|-|-|-|-|\n||AUC|NDCG@5|Recall@5|AUC|NDCG@5|Recall@5|AUC|NDCG@50|Recall@50|\n|Layer1_MTL|0.754|0.663|0.638|**0.688**|**0.669**|**0.467**|0.829|0.633|0.670|\n|Layer1_JL|0.753|0.661|**0.649**|0.674|0.654|0.450|0.826|0.631|0.667|\n|Layer2_MTL|0.752|0.663|0.636|**0.688**|0.668|0.464|0.822|0.637|0.674|\n|Layer2_JL|**0.757**|0.667|0.637|0.686|0.667|0.463|0.828|0.640|0.680|\n|Layer3_MTL|0.754|**0.669**|0.638|0.678|0.659|0.455|**0.830**|**0.641**|**0.681**|\n|Layer3_JL|0.740|0.655|0.624|0.684|0.665|0.464|0.829|0.636|0.675|\n|Best baseline|0.750|0.654|0.630|0.679|0.662|0.459|0.824|0.632|0.671|\n|\n\nThe results show that our method is stable to the choice of propensity model and learning algorithm.\n\n> **W3. Computational Overhead:** Jointly learning three components (prediction, imputation, and $\\epsilon$) may increase training time and tuning complexity compared to standard DR methods; scalability to very large item × user spaces warrants further study.\n\n**Response to W3:** We thank the reviewer for raising this meaningful concern. \n- Note that the proposed method subtly modifies the imputation loss $\\hat e$ by adding $\\epsilon (o-p)$, which **theoretically does not increase complexity compared to the original joint learning method, due to we only require one more gradient calculation and backward propagation for a single learnable parameter $\\epsilon$.**\n- Empirically, the model structurally is also not more complex. Besides a single learnable parameter $\\epsilon$, no new learnable parameters are introduced.\n- **We also add training time experiments on all three datasets under varying dataset scales (controlled by the ratio parameter $b$, varying in {0.1, 0.3, 0.5, 0.7, 0.9}, i.e., we randomly sample 100b% of users and items).** We take DR-JL as the baseline. Thus, all results are 1 for DR-JL. The results show that the time complexity of our method is close to the doubly robust joint learning method, and much less than the previous SOTA DR-based methods, demonstrating the efficiency of our method.\n\n|**Coat**|b=0.1|b=0.3|b=0.5|b=0.7|b=0.9|\n|-|-|-|-|-|-|\n|DR-JL|1.000|1.000|1.000|1.000|1.000|\n|DR-MSE|1.435|1.956|2.008|1.619|1.183|\n|MR|1.588|2.206|2.365|1.524|1.296|\n|DR-V2|1.298|2.093|1.850|1.405|1.028|\n|DCE-TDR|2.235|2.906|3.252|1.912|1.596|\n|Ours|**1.078**|**1.122**|**1.273**|**1.185**|**0.956**|\n|**Yahoo! R3**||||||\n|DR-JL|1.000|1.000|1.000|1.000|1.000|\n|DR-MSE|1.831|2.583|2.866|3.401|2.808|\n|MR|**1.253**|2.458|2.907|2.689|2.862|\n|DR-V2|1.429|1.958|**1.320**|1.921|1.891|\n|DCE-TDR|2.014|3.271|3.247|2.864|3.075|\n|Ours|1.255|**1.401**|1.536|**1.424**|**1.356**|\n|**KuaiRec**||||||\n|DR-JL|1.000|1.000|1.000|1.000|1.000|\n|DR-MSE|1.371|1.706|2.871|1.867|2.714|\n|MR|1.604|2.294|3.355|2.036|3.036|\n|DR-V2|1.266|1.647|1.677|1.012|2.631|\n|DCE-TDR|2.734|4.000|5.065|2.831|3.607|\n|Ours|**1.354**|**1.412**|**1.129**|**0.827**|**1.726**|\n|\n\n> **W4. Limited Domain Scope:** Experiments focus on recommender-system datasets; applicability to other MNAR settings (e.g., causal inference, medical missingness) remains to be demonstrated.\n\n**Response to W4:** We thank the reviewer for the question. **As suggested by the reviewer, we further explore the effectiveness of our method on two scenarios, the medical field and business field, with two datasets, IHDP and Jobs.**\n- The semi-synthetic IHDP dataset [7] is constructed from the Infant Health and Development Program (IHDP) with 747 individuals and 25 covariates.\n- The real-world Jobs dataset [8] is based on the National Supported Work program with 3,212 individuals and 17 covariates.\n\nIn this scenario, the DR estimator has the following form:\n\n$$\\tau(x)=E[\\frac{T(Y-m_1(X))}{\\pi(X)}-\\frac{(1-T)(Y-m_0(X))}{1-\\pi(X)}+m_1(X)-m_0(X) | X=x],$$\n\nwhere $T \\in $ {0,1} is the binary treatment indicator, $Y$ is the outcome, $\\pi(X) = P(T=1|X)$ is the propensity model, and $m_0(X), m_1(X)$ are the imputation models for $T = 0$ and $T = 1$, respectively.\n\nFollowing the previous study [9], we split the data into training/validation/testing sets with ratios 63/27/10 and 56/24/20, with 10 repeated times on the IHDP and the Jobs datasets, respectively. **Note that each sample is already disjoint, and the training distribution is the same as the test distribution due to we randomly split the data.** For evaluation, we use  $\\sqrt{\\epsilon_{\\text{PEHE}}}$ and  $\\epsilon_{\\text{ATE}}$ for IHDP, and  $R_{\\text{pol}}$ and  $\\epsilon_{\\text{ATT}}$ for Jobs. The definitions are shown below: \n\n- $\\epsilon_{\\text{PEHE}} =\n{\\frac{1}{N} \\sum_{i=1}^N [(\\hat {Y}_i(1) - \\hat {Y}_i(0)) - (Y_i(1) - Y_i(0))]^2}$, where $\\hat {Y}_i(1)$ and $\\hat {Y}_i(0)$ are the predicted values for the corresponding true potential outcomes of unit $i$. \n- $\\epsilon_{\\text{ATE}}=\\frac{1}{N}| \\sum_{i=1}^N ((\\hat {Y}_i(1) - \\hat {Y}_i(0)) - (Y_i(1) - Y_i(0)))|.$\n- ${R}_{\\text{Pol}}= 1-(\\mathbb{E}[Y(1) \\mid \\hat {Y}(1) - \\hat {Y}(0) >0, X=1] \\cdot \\mathbb{P}(\\hat {Y}(1) - \\hat {Y}(0)>0)+\\mathbb{E}[Y(0) \\mid \\hat {Y}(1) - \\hat {Y}(0) \\leq 0, X=0] \\cdot \\mathbb{P}(\\hat {Y}(1) - \\hat {Y}(0) \\leq 0))$, where $T, C, E$ are the indexes of treatment, control, and randomized sample set, respectively.\n-  $\\epsilon_{\\text{ATT}} = |\\text{ATT} - \\frac{1}{|T|} \\sum_{i \\in T} (\\hat {Y}_i(1) - \\hat {Y}_i(0)|$\n\nwhere $\\text{ATT} = |\\frac{1}{|T|} \\sum_{i \\in T}{Y_i} - \\frac{1}{|C \\cap E|} \\sum_{i \\in (C \\cap E)}{Y_i}|$.\n\nThe lower the metrics, the better the performance. See [9] for a more detailed definition of evaluation metrics. The performance on the test set is shown below:\n\nResults on IHDP:\n|Methods|$\\sqrt{\\epsilon_{\\text{PEHE}}}$↓|$\\epsilon_{\\text{ATE}}$↓|\n|-|-|-|\n|DR-JL|1.37±0.30|0.22±0.15|\n|Ours|**0.95±0.11**|**0.14±0.09**|\n|\n\nResults on Jobs:\n|Methods|${R}_{\\text{Pol}}$↓|$\\epsilon_{\\text{ATT}}$↓|\n|-|-|-|\n|DR-JL|0.27±0.06|**0.05±0.03**|\n|Ours|**0.23±0.07**|**0.05±0.02**|\n|\n\nThe results show that our method can be applied to those datasets, and the performance is better than the DR-JL method.\n\n**We believe the above discussion has thoroughly addressed your concerns regarding our work. We would be grateful if you could kindly consider raising your rating.** Thank you!\n\n---\nReferences\n\n[1] Doubly Robust Joint Learning for Recommendation on Data Missing Not at Random. ICML 2019.\n\n[2] Enhanced Doubly Robust Learning for Debiasing Post-Click Conversion Rate Estimation. SIGIR 2021.\n\n[3] Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random. WWW 2024.\n\n[4] ESCM$^2$: Entire Space Counterfactual Multi-Task Model for Post-Click Conversion Rate Estimation. SIGIR 2022.\n\n[5] StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not at Random. ICLR 2023.\n\n[6] Propensity Matters: Measuring and Enhancing Balancing for Recommendation. ICML 2023.\n\n[7] Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics 2011.\n\n[8] Evaluating the Econometric Evaluations of Training Programs with Experimental Data. The American Economic Review 1986.\n\n[9] Estimating Individual Treatment Effect: Generalization Bounds and Algorithms. ICML 2017."}, {"review_id": "8z53Al28QB", "reviewer": "Reviewer_FSqV", "summary": "The paper identifies a critical limitation in existing debiasing methods, which assume \"exact matching\" between training and test sets. It introduces a more realistic \"super-population\" setting, where samples are drawn independently from the same underlying distribution. Under this setting, even perfectly specified DR estimators are biased due to a new covariance term. To address this, the authors propose SuperDR, which corrects the imputation model by introducing a learnable parameter ε to minimize the empirical covariance, inspired by TMLE.", "questions": "* Was any propensity score stabilization used?\n* How were the inner loop steps and learning rates chosen in Algorithm 1?\n* Why apply correction to the imputation model rather than the prediction output?\n* Are MF-based features sufficient for generalization across disjoint user-item samples?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "#### **Strengths:**\n\n* The clear distinction between exact matching, cold-start, and the proposed super-population setting fills an important gap and better reflects real-world applications.\n\n* The paper formally derives the bias and shows how the correction term effectively targets the covariance source. Theoretical results (e.g., Theorem 4.3, 4.7) are elegant and insightful.\n\n*  Experiments on multiple datasets, including the large-scale KuaiRec, demonstrate significant and consistent improvements. Analyses directly validate theoretical claims (e.g., performance vs. empirical covariance).\n\n**Weaknesses:**\n\n*  Lack of detail on whether stabilization techniques (e.g., propensity clipping) were used, which affects reproducibility and variance control explanation.\n*  The alternating optimization scheme increases overhead but is not analyzed or discussed in terms of cost-performance trade-offs.\n*  Lack of some important related work:\n[1] Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation\n[2] Cdr:Conservative doubly robust learning for debiased recommendation", "was_revised": true, "final_justification": "My concerns have been adequately addressed, and I have decided to maintain my positive rating.", "rebuttal": "We thank the reviewer for efforts in evaluating our work and helpful suggestions. Below, we will try our best to address your concerns and questions.\n\n> **W1 & Q1:** Lack of detail on whether stabilization techniques (e.g., propensity clipping) were used, which affects reproducibility and variance control explanation.\n\n**Response to W1 & Q1:** We thank the reviewer for pointing out these issues. Yes, we use a propensity clipping technique, and the clipping threshold is a hyperparameter tuned in [0.005, 0.05] on all three datasets. The best threshold is 0.0186 for **Coat**, 0.0127 for **Yahoo! R3**, and 0.0112 for the **KuaiRec** dataset. We will add these details in our revised version.\n\n> **W2:** The alternating optimization scheme increases overhead but is not analyzed or discussed in terms of cost-performance trade-offs.\n\n**Response to W2:** We thank the reviewer for raising this meaningful concern. \n- Note that the proposed method subtly modifies the imputation loss $\\hat e$ by adding $\\epsilon (o-p)$, which **theoretically does not increase complexity compared to the original joint learning method, due to we only require one more gradient calculation and backward propagation for a single learnable parameter $\\epsilon$.**\n- Empirically, the model structurally is also not more complex. Besides a single learnable parameter $\\epsilon$, no new learnable parameters are introduced.\n- **We also add training time experiments on all three datasets under varying dataset scales (controlled by the ratio parameter $b$, varying in {0.1, 0.3, 0.5, 0.7, 0.9}, i.e., we randomly sample 100b% of users and items).** We take DR-JL as the baseline. Thus, all results are 1 for DR-JL. The results show that the time complexity of our method is close to the doubly robust joint learning method, and much less than the previous SOTA DR-based methods, demonstrating the efficiency of our method.\n\n|**Coat**|b=0.1|b=0.3|b=0.5|b=0.7|b=0.9|\n|-|-|-|-|-|-|\n|DR-JL|1.000|1.000|1.000|1.000|1.000|\n|DR-MSE|1.435|1.956|2.008|1.619|1.183|\n|MR|1.588|2.206|2.365|1.524|1.296|\n|DR-V2|1.298|2.093|1.850|1.405|1.028|\n|DCE-TDR|2.235|2.906|3.252|1.912|1.596|\n|Ours|**1.078**|**1.122**|**1.273**|**1.185**|**0.956**|\n|**Yahoo! R3**||||||\n|DR-JL|1.000|1.000|1.000|1.000|1.000|\n|DR-MSE|1.831|2.583|2.866|3.401|2.808|\n|MR|**1.253**|2.458|2.907|2.689|2.862|\n|DR-V2|1.429|1.958|**1.320**|1.921|1.891|\n|DCE-TDR|2.014|3.271|3.247|2.864|3.075|\n|Ours|1.255|**1.401**|1.536|**1.424**|**1.356**|\n|**KuaiRec**||||||\n|DR-JL|1.000|1.000|1.000|1.000|1.000|\n|DR-MSE|1.371|1.706|2.871|1.867|2.714|\n|MR|1.604|2.294|3.355|2.036|3.036|\n|DR-V2|1.266|1.647|1.677|1.012|2.631|\n|DCE-TDR|2.734|4.000|5.065|2.831|3.607|\n|Ours|**1.354**|**1.412**|**1.129**|**0.827**|**1.726**|\n|\n\n> **W3:** Lack of some important related work: [1] Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation [2] Cdr:Conservative doubly robust learning for debiased recommendation.\n\n**Response to W3:** We thank the reviewer for the useful suggestions. We will modify the related work section to include these two important works. In addition, we add experiments to compare our method to these two methods. The results are shown below:\n\n|Methods|**Coat**|||**Yahoo! R3**|||**KuaiRec**|||\n|-|-|-|-|-|-|-|-|-|-|\n||AUC|NDCG@5|Recall@5|AUC|NDCG@5|Recall@5|AUC|NDCG@50|Recall@50|\n|DR-JL|0.733±0.003|0.650±0.005|0.625±0.007|0.667±0.005|0.655±0.004|0.449±0.008|0.818±0.003|0.620±0.004|0.655±0.007|\n|CDR [2] |0.743±0.004|0.652±0.005|0.627±0.005|0.672±0.004|0.658±0.003|0.454±0.005|0.823±0.005|0.623±0.004|0.660±0.005|\n|MRDR-CAL [1] |0.744±0.003|0.651±0.004|0.630±0.004|0.675±0.003|0.660±0.003|0.454±0.005|0.822±0.006|0.625±0.005|0.665±0.006|\n|DCE-TDR|0.746±0.005|0.654±0.005|0.629±0.006|0.679±0.004|0.662±0.005|0.459±0.004|0.824±0.003|0.632±0.004|0.659±0.004|\n|D-DR|0.750±0.004|0.654±0.004|0.630±0.005|0.678±0.004|0.659±0.004|0.456±0.003|0.822±0.004|0.630±0.005|0.672±0.005|\n|Ours|**0.757±0.004**|**0.667±0.005**|**0.637±0.007**|**0.686±0.003**|**0.667±0.004**|**0.463±0.003**|**0.828±0.004**|**0.640±0.005**|**0.680±0.005**|\n\nWe find that our method still outperforms the baselines.\n\n> **Q2:** How were the inner loop steps and learning rates chosen in Algorithm 1?\n\n**Response to Q2:**  We thank the reviewer for the question. Following the previous implementation in [1-3], the step is fixed to 1. In addition, as shown in lines 250-251 in the manuscript, the learning rate is tuned in {0.001, 0.005, 0.01, 0.02, 0.05} for parameters in prediction, imputation, and propensity model, and in {0.01, 0.05, 0.1, 0.15, 0.2} for $\\epsilon$. The best choice is {0.01, 0.15} for **Coat**, {0.005, 0.15} for **Yahoo! R3**, and {0.005, 0.1} for **KuaiRec**.\n\n> **Q3:** Why apply correction to the imputation model rather than the prediction output?\n\n**Response to Q3:**   We thank the reviewer for the interesting question. **The conclusion is, applying the correction to either the imputation model or the prediction model will lead to exactly the same results!** Here are the reasons.\n- For updating the prediction model, we first recap the vanilla DR loss below: \n\n    $$\\mathcal L_{\\text{DR}}(\\theta)=\\frac{1}{|\\mathcal{D}|} \\sum_{(u, i) \\in \\mathcal{D}}\\left[\\hat e_{u, i}+\\frac{o_{u, i}\\left(e_{u, i}-\\hat e_{u, i}\\right)}{\\hat p_{u, i}}\\right].$$\n**For updating the prediction model, note that whether adding the correction to the imputation model or the prediction model based on this DR loss will not affect the updating of the prediction model, because there is no gradient to the prediction model.**\n\n- For updating the imputation model, we recap the vanilla imputation loss function below\n    $$\\mathcal L_e = \\frac{1}{|\\mathcal{D}|} \\sum_{(u, i) \\in \\mathcal{D}} \\frac{o_{u, i}\\left(e_{u, i}-\\hat e_{u, i}\\right)^2}{\\hat p_{u, i}}.$$\n\n**Note that adding the correction to the imputation model or the prediction model based on this imputation loss also makes no difference when updating the imputation model and the parameter $\\epsilon$.**\n\n> **Q4:** Are MF-based features sufficient for generalization across disjoint user-item samples?\n\n**Response to Q4:**\n\nWe thank the reviewer for raising this concern. \n\n- **We add experiments on all three datasets, demonstrating that MF-based features are sufficient for generalization.** Specifically, for all three datasets, we randomly split the training data (i.e., MNAR data) into training / validation / test data, with a ratio 0.8 / 0.1 / 0.1. Then we use an MLP as the backbone to train a prediction model using the naive loss, i.e., loss on the observed user-item pairs. \n$$\\mathcal L_{\\text{naive}}(\\theta)=\\frac{1}{|\\mathcal{O}|} \\sum_{(u, i) \\in \\mathcal{O}}e_{u,i}.$$\n\nThe test performance on all three datasets corresponding to MSE and AUC is shown below:\n|Datasets|AUC|MSE|\n|-|-|-|\n|**Coat**|0.952|0.092|\n|**Yahoo! R3**|0.946|0.096|\n|**KuaiRec**|0.889|0.117|\n\n- The results show that simple naive loss-based neural networks demonstrate excellent performance without data MNAR on all three datasets, proving MF-based features are sufficient for generalization. \n- Furthermore, since **KuaiRec** is a more complicated industrial dataset, its lower results compared to **Coat** and **Yahoo! R3** are in line with expectations. \n- Finally, naive methods perform poorly with data MNAR (**Coat** and **Yahoo! R3** have around 20% lower AUC, and KuaiRec around 10% lower AUC), highlighting the need for debiasing in real-world datasets.\n\n**We believe the above discussion has thoroughly addressed your concerns regarding our work. We would be grateful if you could kindly consider raising your rating.** Thank you!"}]

Author rebuttal:

Meta-review: This paper considers the problem of selection bias. They go beyond the setting of exact matching and instead assume only that the joint distribution of the feature and the rating is the same in the training and test set. They show that in this broader setting, the previous DR estimator is biased due to an additional covariance term. They propose SuperDR which adds a learnable “correction” parameter. They provide sound and nontrivial theoretical guarantees and demonstrate improved performance on a variety of datasets. This paper is novel, well-written, and a nice contribution.
Decision: Accept (poster)