Removing Hidden Confounding in Recommendation: A Unified Multi-Task Learning Approach
Authors: Haoxuan Li
Kunhan Wu
Chunyuan Zheng
Yanghao Xiao
Hao Wang
Zhi Geng
Fuli Feng
Xiangnan He
Peng Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: Debiased recommender system
Multi-task learning
Causal inference

Abstract: In recommender systems, the collected data used for training is always subject to selection bias, which poses a great challenge for unbiased learning. Previous studies proposed various debiasing methods based on observed user and item features, but ignored the effect of hidden confounding. To address this problem, recent works suggest the use of sensitivity analysis for worst-case control of the unknown true propensity, but only valid when the true propensity is near to the nominal propensity within a finite bound. In this paper, we first perform theoretical analysis to reveal the possible failure of previous approaches, including propensity-based, multi-task learning, and bi-level optimization methods, in achieving unbiased learning when hidden confounding is present. Then, we propose a unified multi-task learning approach to remove hidden confounding, which uses a few unbiased ratings to calibrate the learned nominal propensities and nominal error imputations from biased data. We conduct extensive experiments on three publicly available benchmark datasets containing a fully exposed large-scale industrial dataset, validating the effectiveness of the proposed methods in removing hidden confounding.

Removing Hidden Confounding in Recommendation: A Unified Multi-Task Learning Approach
Haoxuan Li', Kunhan Wu², Chunyuan Zheng', Yanghao Xiao*,
Hao WangS, Zhi Geng", Fuli Feng', Xiangnan He" Peng Su
Peking University CCrregie Mellon University University of California, San Diego
University of Chinese Academy of Sciences *Zhejiang University "Bijjing Technology and Business University "University of Science and Technology of China
Abstract
In recommender systems, the collected data used for training is always subject to se- lection bias, which poses great challenge for unbiased learning. Previous studies proposed various debiasing methods based on observed user and item features, but ignored the effect of hidden confounding. To address this problem, recent works suggest the use of sensitivity analysis for worst-case control of the unknown true propensity, but only valid when the true propensity is near to the nominal propensity within a finite bound. In this paper, we first perform theoretical analysis to reveal the possible failure of previous approaches, including propensity-baseee multi-task learning, and bi-level optimization methods, in achieving unbiased learning when hidden confounding is present. Then, we propose unified multi-tasi learning approach to remove hidden confounding, which uses a few unbiased ratings to cali- brate the learned nominal propensities and nominal error imputations from biased data. We conduct extensive experiments on three publicly available benchmark datasets containing fully exposed large-scale industrial dataset, validating the effectiveness of the proposed methods in removing hidden confounding.
Introduction
Recommender systems (RS) play key role information retrieval by filtering out items that may
be of interest to users [16] 24 In general, the training process of recommendation uses historical user interaction data. However, challenge in using interactions to make predictions selection bias
3 32 40, 57]. i.e., users always choose preferred items to interact [5, 511 resulting in
difference in the distribution between data with and without interactions 14, 45 54 [66] 60),
which poses great challenge for unbiased evaluation and learning of the prediction models.
Many methods have been proposed tackle the selection bias problem, such the error imputation- based (EIB) [2]. inverse propensity scoring (IPS) [33 41 42). and doubly robust (DR) methods 7 [1] 29 (2). Based on these, recent works further incorporate multi-task learning alleviate the data sparsity issue, such as entire space multi-task model (ESMM) [34], multi-task IPS and multi-task DR (Multi-IPS and Multi-DR) [59 and entire space counterfactual multi-task models (ESCM?) 44
However, these studies only consider the selection bias induced by measured confounders, but ignore the presence of hidden (or unmeasured) confounders, which considerably influences the application of these advanced methods in real-world recommendation scenarios. Hidden confounders ubiquitous and inevitable in RS due information limitations (e.g., friend useful suggestions) or In this paper, we perform theoretical analysis
privacy restrictions (e.g., user salary) g
35 50
*Corresponding author: pengwu@btbu.edu...
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

for the existing propensity- baste and multi-task learning methods, showing that all of them will lead to biased evaluation and learning in the presence of hidden confounding.
To remove hidden confounding, previous causal inference literature suggested the use of instrumental variables 12 or front door adjustment [36.. However, in real-world application scenarios, these approaches require strong assumptions that are difficult to verify in practice. On the other hand, recent recommendation study proposed robust deconfounder (RD) to adopt sensitivity analysis using a min-max optimization for worst-case control of the unknown true propensity [8. Nevertheless, they
assume the true propensity is near to the nominal one within a bound, which decided by the strength of the unmeasured confounder, posing another challenge in case these assumptions are violated.
In contrast, unbiased datasets are regarded as the gold standard for unbiased evaluations and can be collected from A/B tests or randomized controlled trials [10 13 21 55 This provides an alternative solution remove hidden confounding [15] 20 26 58 Nevertheless, training recommendation
models directly on unbiased datasets can suffer from severe overfitiigg due to the limited sample size restricted by the collection cost of random exposure 44. Despite few works leverage the unbiased data to combat selection bias 4 53 by adopting the bi-level optimization, this paper shows that they still lead biased learning, due the biased hypothesis space of the prediction model.
To tackle the above problems, we propose a unified multi-task learning approach to remove hidden confounding by leveraging few unbiased ratings. Interestingly, we show that the unbiased data can help calibrate the learned nominal propensities and nominal error imputations, which differs from RD using sensitivity analysis for nominal propensities only [8]. Specifically, the proposed multi-task learning builds residual networks for learned propensities and imputed errors from biased data, respectively. Next, consistency loss of IPS (or DR) estimation on the biased dataset and empirical average prediction errors on the unbiased dataset are developed to help the training of the two residual networks, for calibrating the learned nominal propensities and nominal error imputations, respectively. The prediction model is then trained with the calibrated IPS (or calibrated DR) loss and unbiased data to achieve unbiased learning in the presence of hidden confounding.
The main contributions of this paper are summarized as follows:
We theoretically reveal limitations of the existing multi-task learning and bi-level optimiza- tion methods for achieving the unbiasedness in the presence of hidden confounding.
We provide a unified multi-task learning approach to remove hidden confounding by com- bining few unbiased ratings, in which the learned nominal propensities and nominal error imputations can be calibrated by the residual networks with the proposed consistency loss.
We conduct extensive experiments on three publicly available benchmark datasets, including large-scale industrial dataset, to validate the effectiveness of the proposed methods.
2 Problem Setup
We formulate the selection bias in RS using post-click conversion rate (pCVR) prediction task for illustration purpose, which can be naturally generalized to other recommendation tasks with explicit feedback, c.g., rating prediction. Suppose that the entire space has m users and n items, let D be the set of all user-item pairs. Denote R (0, 1) m the true post-click conversion label matrix of user-item pairs, where each entry Tu, indicates whether a conversion occurs after user u clicks on item i. Let Tn, be the feature of user-item pair (u, i), and π. E m be the prediction matrix for PCVR, where fu.: = I ( (cvr) E [o, 1] is the predicted pCVR obtained by a model f with parameter OCVR. In RS, users always select the preferred items to click on, leading to a significant difference in the distribution between clicked and unclicked events thus causing selection bias.
If R is fully observed, then pCVR model flu, OCVR) can be trained by minimizing the ideal loss
1
Lideal (OevR)
8
[
(a.i)ED
where du A 8(rm,5; Pass) and 6(.,)) ) is pre-specified loss, e.g., the cross-entropy loss, 8(ru,s, Pr,s) = -rn,i log Fu,i (1 Tu.i) log (1 f..s). However, the post-click conversion feedback of a user-item pair (u, i) can be observed only when user u clicks on item i, making the ideal loss not computable. Let ou,s be the indicator of user clicking on item i, and B {(u, i) (u, e D, ou, 1} be
2

---

the set of clicked events, where B means that the clicked events is biased sample of the entire space D. In this paper, we further consider the presence of hidden confounding. Without loss of generality, we assume that all confounders consist of measured part Ja,, and hidden (unmeasured) part hy.i. where the latter arises from issues such as information limitations (e-g., friend suggestions) and privacy restrictions (e.g., user salary), which cannot be observed explicitly and used for training.
3 Previous Methods Lead to Biased Learning under Hidden Confounding
3.1 Multi-Task Learning
The pCVR prediction task is closely related to click-through rate (CTR) and post-view click-through & conversion rate (CTCVR) prediction tasks, as the formula CTCVR CTR pCVR holds, where the CTR is Pa,: - (o l)),,)), also known as propensity score in the causal inference literature, represents the probability of a user clicking on an item i. The CTCVR is P(ru, = 1, Ou, = 1/2,,s). means the probability that item i is clicked and converted by user
Let Pu.s ( OCTR) be CTR prediction model with parameter OCTR. The ESMM method [34] learns pCVR by joint- cent-training CTR and CTCVR losses
(CTR(OCTR)
8 (oss)sPus), LeTevR (OCTR, bCvR) [D]
Σ
8 Pu
[D]
(u,i)ED
(a.j)ED
However, the ESMM loss (CTR(OCTR) + EerevR(Oem) tCvR) is a biased estimator of the ideal loss [59]. To achieve unbiased learning, the MTL-IPS and MTL-DR methods [59] use the losses
Ou.suua Pu,
Ba.s + Ou.iloui 8
Liss (OCTRR (CvR) =
CDR (OCTR, OcvR. (IMP) = [D]
D
Pud
(u.))DD
(u,))ED
where S,,; A ( 01MP) is the imputation model that predicts ōu,i using Fu,,i i.e., estimates Iu.i A /f Without hidden confounders, Lips is an unbiased estimator of the ideal loss
when the learned propensities are accurate, i.e., P... Pu.i [1 (2). and LDR is unbiased if either
Pu.: Pu,, or 8 = 9u.: 33 52]. However, both IPS and DR are biased under hidden confounders.
Lemma (Theorem 3.1 in R) In the presence of hidden confounders ha,i, both Lirs and LDR are biased estimators of the ideal loss, even if Pu.s = Pu.s and 8u,5 = 9u,i-
The above result also holds in the follow-up DR studies 23
and be naturally extended to conclude that the MTL-IPS and MTL-DR [59] are biased under hidden confounding. Recently, 2SCM²-IP -PS and ESCM2-DS showed state-of-the- the-aar performance in pCVR prediction by incorporat- ing the ESMM loss LerevR as the global risk of CTCVR, and MTL-IPS and MTL-DR losses L as counterfactual risk of PCVR [4). Formally, the ESCM loss is
25
CESCM = LCTR + CIMP + de + A3
where A for t = 1,2,3 3 are ayper-parameters, LEve is either LIPS or CDR. The imputation loss
(5a.i 8,,,)²
LIMr(OCTR, OcvR. 0IMP)
B
Pu,s
(u,))
However, since both LerevR and LêvR are biased under hidden confounding, ESCM is also biased.
3.2 Debiasing with Few Unbiased Ratings
Instead of only using the biased dataset B, many methods are proposed to improve the debiasing performance by combining a small unbiased dataset and a large biased dataset B, such as bi- level optimization approaches, including learning to debias (LTD) [53] and AutoDebias (4, causal embedding method (CausF) [I] knowledge distillation framework for counterfactual recommendation via uniform data (KDCRec) 30 32], and causal balancing methods (26).

---

Specifically, learning to debias (LTD) [33 and AutoDebias 4 adopt bi-level optimization 17 37 learn a CTR model (a, rCTR) such that the pCVR prediction model fu., =f f CVR performs well on the small unbiased dataset Formally, the bi-level optimization in LTD is
OCTR arg min LCvv (0EvR(OCTR)
OCTR
s.t. (OCTR) = arg min Lêva (fCTR: BCvR),
BCvR
where the upper loss is as an average of prediction errors on the unbiased dataset
CeR ((cER(((TTR)
Σ
8 (Fu,i Pu, (fCTr)))(TT)))))))))))))))))))))
=
(u.J)E
and AutoDebias further develop a solution for universal debiasing. However, despite the of unbiased ratings, we show that they still lead to biased estimates under hidden confounding. Proposition In the presence of hidden confounders, both LTD and AutoDebias are biased.
Proof. Let HCTR = ( ACTR) OCTR E OCTR] } be the hypothesis space of propensity model. Then lyTR defined above is the parameter in OCTR such that the pCVR prediction model performs optimally U. The bi-level optimization takes CVR estimate the ideal loss using a selected propensity ( (CTR)) E HCTR, however, L could be biased estimator of the ideal loss for all Pu. E HCTR (due to Pu.i could deviates from the true one by an arbitrary distance, as formally stated in Theorem 2). therefore both LTD and AutoDebias using bi-level optimization are biased. D
Proposition formally reveals the limitations directly adopting bi-level optimization for addressing hidden confounding. Both LTD and AutoDebias essentially use unbiased data for model selection among all possible IPS or DR losses, but in fact, without correcting the IPS or estimators themselves for predicting the ideal loss, those IPS or DR-based methods will not be able to tackle unobserved confounding, because of the intrinsic biasedness the estimators to the ideal loss.
An alternative class of methods that use unbiased data for tuning is the causal embedding method Specifically, they both consider building a connection between model trained with biased data and another model trained with unbiased data. CausE [1] designs an alignment term as the pairwise difference between the parameters of the two models, which is then included in the object function to be minimized. KDCRec 30 32 proposes a general knowledge distillation framework for counterfactual recommendation via uniform data, including label-based, feature-based, sample-based, and model structure-based distillations. The above methods empirically shown impressive performance by distilling the shared information on both biased and unbiased data. However, theoretical guarantees under hidden confounding are lacking, and it would be interesting to investigate the conditions under which these methods can achieve unbiased learning.
(CausE) M and KDCRec 30 32
3.3 Mitigating Hidden Confounding with Sensitivity Analysis
To tackle the problem of hidden confounding, robust deconfounder (RD) (8] proposes to adopt sensitivity analysis from the causal inference literature 38 to minimize the worst-case prediction loss. Formally, the unknown true propensity Pu,i P(ou,: 1| bu, is assumed to near the nominal propensity Pa.: = Poo,, = 1|zu,) within a bound that
(1 - Pu.i) Pu.s ST I Po.i (1 po,s)
1 (1/Pu,: - 1) au,i Wu,i S bu,s A 1 (1/Pu,i I)T,
where hyper-parametee I' corresponds to the strength of unmeasured confounding, Wa,i = 1/Pa, the inverse of the true propensity, and au,; and bu, are the lower and upper bounds of w,,5. respectively. Let W = [ bi.1) x ..... .... x [ām, bm.n] be the possible inverse propensities on all user-item pairs, where a and b are the estimates of an, and b respectively, then RD-IPS method trains the prediction model by minimizing the worst-case IPS loss
LRDAPS (OCTR, OCvR) = max WEW
Ou, (a.i)D
Σ
D
and the RD-DR method can be developed by similar argument controlling the worst-case DR loss.

---

To summarize, the RD methods first obtains the bounds of true propensities around the nominal propensities, then minimize the upper bound of the IPS (or DR) loss to control the worst-case caused by unmeasured confounders. However, on the one hand, it is not clear how to set I correctly since both the true propensities and the strength of hidden confounding are unknown. On the other hand, the effectiveness of sensitivity analysis for controlling the hidden confounding requires that the true propensity is around the nominal propensity for all user-item pairs. but such (strong) assumptions cannot be verified from the data and raises another concern in case the assumptions are violated.
Debiasing Residual Networks under Hidden Confounding
4.1 Methodology Overview
Different from the previous methods that use sensitivity analysis to control the worst-case IPS or DR loss caused by hidden confounding, we propose a unified multi-task learning approach with residual networks in Figure I with the motivation of using small unbiased data to calibrate the learned propensities and imputed errors in IPS or DR loss for training the unbiased prediction model.
Following causal inference literature 18 19, we define the true propensity and true imputation as Pa.s - Po,,, 1 hu.i): Ia.s - E( (Fu,,, hu,s). both of them are functions of (Zu,z, hu,s), with Pa,i and 8 as their estimates. To distinguish, we call Pu,i Po (z and Ju. El the nominal propensity and nominal imputation, with Pu.i and 8,,s as their estimates.
Next, we show the necessity of calibrations on both Pu,i and dusi estimated from the baised data B. Theorem 2 (Necessity of Calibration). Suppose the partial derivative of Pus and Du.i with respect to hidden confounders h are not always equal to O, and Puti and du, are consistent estimators of Pa, and Su then there exists rj > O, such that
lim P(Pputt - Pu,al >0 1D0+000
lim P(J.., Ju.al 17) 0. DD|+00
Proof. Given the partial derivative of Pu.i with respect to hidden confounders hy, is always equal to O, that is, h has non-zero effect on Ou.i, so we have Pu, * Pu, according to their definitions. Thus, for some >> 0, there exist positive constants 8 Ni (e) > such that for all [D] > Ne).).
P(P - Paoal e) 5, > O.
Since iu, is a consistent estimator of Pa.i. there exists some N2(c) > 0, such that for all [D] > N2(c),
6
P(PP.. -Pu.il 2 c/2) <
Thus, if [D] > (e). we have
PP(P,., Pu,il > e, Iiu,i - Pu.il < <l/2)
P(Ppu,. Pu,il > x) P(Pu, - Pu.il > e, (uu. - Pa.l p 2
> 5 8 3
Let = e/2 and note that {lpu,i - Pu.il > e, Iuu, Pu.il < </22} C (Puu. Pa.il > m1. we have
PP(p.. Pu,il > 11) 2 P( Pu,il > e, lin Pu.il < e/2) > 35./ / > 0.
which leads to 1i PP Pu,s Pu,el n) 0. Similarly, it can be shown that lim pi- P(IS., - Ju.il > 1)) > 0.
Theorem shows that in the presence of hidden confounding, the estimated nominal propensities and nominal imputed emror deviate from the true one, even with the infinite sample size. To address this problem, as shown in Figure T we propose a novel consistency loss that utilizes unbiased data to calibrate the learned nominal propensities and imputed errors from the biased data.
5

---

B
Consistency
B
B
i
Condistency
B
Unbiased
IPS Loss Aive
Unéiased
4ove
Imputaaiin
Gem
22gg
Propensity
Res Propensity
Pru O Estimate
Aui
Bus Faas-inn peseiimpuned
P
Estimate
Enror
fui
Fut
MLP
MLP
MLP
MLP
MLP
Pes-Propensity (CTR) Network
Res-Propensity
Prediction
CVR) Network
(CTR) Network
[CVR] Network
Imputation Network
Shared Network (MLP)
Shared Network (MLP)
Input Covariates
Input Covariates
(a) Res-IPS
(b) Res-DR
Figure I: Proposed debiasing residual networks for removing hidden confounding.
Specifically, we define the calibrated propensity model p = =5(OCTR, ØCTR) and the calibrated imputation model 5(0IMP. <IMP) as follows
Pu:i (o (Pu, + '(Apu,
Ru,i = (0 (6., + -
where is the sigmoid function, and the transformations are designed for numerical stability, e.g., to control range from to Compared with Pu,s and 6u, adopted in the previous multi-tass task learning approaches [34] 44 59], the residual terms Apu,i and Adu,t are further added to Pu,i and o,, to capture the effect of hidden confounding. The loss function of the proposed approach defined
CRes (F) + (8 +B (CEvR (8. + LETEVR (P) + L
B&cld CVR (p, 1o (p.p,0.0 0)
+y
Initialization using biased data
Prediction model training with calihruted losses
Calibration and 8
where a; B and y
for trade-off. The following states for each loss.
4.2 Nominal Propensities and Imputations Initialization
Similar the previous IPS and DR methods [39 42 52], as well as the multi-task learning approaches [34]]] 59) we use the losses LCTr and LIMP in Section for training the estimated nominal propensities Pu,s and nominal imputed errors dw, from the biased dataset. However, since Pu.i and ò,,; can only capture the effect of measured confounding, we do directly use them for training the prediction model under hidden confounding, Instead, we train the prediction model using calibrated propensities and imputed errors as illustrated Section
4.3 Debiased Prediction Model Training
Training on Biased Data. To train an unbiased prediction model, the losses need to be unbiased with respect to the ideal loss, thus the direct use of nominal Pu.s and 8 under hidden confounding would lead to biased predictions. In contrast previous studies, we propose use both the initialized p..; and 6,,; in Section 4.2 and the corresponding residual terms Apu.i and A (see Section 4.4 for more details) using IPS (or DR, CTCVR loss) achieve unbiased prediction model learning.
Specifically, using the calibrated propensity model, the proposed residual-IB -PS (Res-IPS) loss
1
Ou,i 6
C (f,T()) (
D
Pu,i

---

Similarly, using both the calibrated propensity model and the calibrated imputation model, the proposed residual-DR (Res-DR) loss
8u. + Ou,i Suns)
LöR (fevR (u,s(OcTR), Ap. Su, (0IMP), ( =
Σ (u,i)DD
[D]
Pu,,
Although CTCVR loss is not unbiased to the ideal loss [59]. as previous studies have shown, using both IPS (or DR) loss and CTCVR loss empirically can lead to better debiasing performance by mitigating the data sparsity issue 44 Similarly, using the calibrated propensity model, the proposed calibrated CTCVR loss
LerevR(OevR Pu,,(OeTR), (
(o Σ (u.S)D
D
Empirically, one can choose from the three calibrated losses for training debiased prediction model.
Training Unbiased Data. Since the unbiased data do encounter any confounding problems, provides a golden standard to evaluate and train the pCVR prediction model. The unbiased loss is
Ceve(Oeva
Σ (N.J)Eu
(runis Puck (fcvR)).
U
Different from the previous studies or imputation models, our approach is not necessary for training the prediction model using the unbiased loss (since one can use the aforementioned calibrated losses). addition, the direct use of the unbiased loss can lead to severe overfitting, which once again demonstrates the importance of calibrating the propensity and imputation models the IPS or DR loss.
in which V used select the optimal propensity
4.4 Residual Networks Training
Collected through a carefully designed experiment, the unbiased data can be regarded as repre- sentative sample of the entire space [53). Thus, we always have L Ludea (#CvR), regardless of hidden confounding in biased data, which motivates us to propose a consistency loss
CESA (OCTR, ØCTR, OCvR, 0IMP, ΦIMP) S(CEvR C((vR)
which measures the discrepancy between LV and CevvR. which equivalently provides with optimization direction for removing hidden confounding.
Proposition 3. If [BSd then LêVR is an unbiased estimator of the ideal loss, regardless of whether hidden confounders exist or not.
Essentially, the consistency loss uses unbiased data to calibrate the debiasing loss LVvR based on the biased data, thereby guaranteeing the debiasing ability of the proposed methods in the presence of hidden confounding. As discussed in Sections 4.33 we utilize the two residual terms Apu.: and Ad,,; to capture the effect of hidden confounding, which are trained by mimimizing L
World Experiments
three real-world datasets: COAT YAНOO!! R.F. and KUAIREC [9] for evaluating the debiasing performance of the proposed methods, where KUAIREC is public large-scale industrial dataset. COAT contains 6,960 biased ratings from 290 users to 300 items, where each user picks 24 items to rate based on their personal preferences. Meanwhile, also contains 4,640 unbiased ratings, where each user is asked to rate 16 randomly selected items. YAHOO! R3 contains 311.704 biased ratings and 54,000 unbiased ratings, where the unbiased ratings are from the first 5,400 users for 10 random selected items. We binarize ratings less than four to 0 and other ratings to for the above two five-scale
Dataset and Pre-processing.
Following the previous studies 4
42 52
Phttss//www.es.com/ wwwpw//.....cscccaaeesseeeeeceee
hhut://wwehscope hhans://gthuu.coommcooonnnnggggu
http:///wehssopp.sadddoxo sandhox yammo.com/

---

Table Performance in terms of AUC, NDCG@K, K. and Recall @ on the unbiased dataset of COAT, YAHOO! R3 and KUAIREC. The best two results are bolded, and the best baseline is underlined.
COAT
YAHOO! R3
KUAIREC
Method
AUC
N@5
R@5
AUC
N@5
R@5
AUC
N@50
R@ 50
MF
22
(Bias)
0.747
0.500
0.546
0.721
0.553
0.716
0.820
0.561
0.816
MF
22
(Uniform)
0.580
0.363
0.386
0.574
0.455
0.611
0.664
0.491
0,816
MF
22
(Combine)
0.751
0.504
0.546
0.724
0.558
0.717
0.822
0,566
0.812
CausE
1
0.763
0.512
0.575
0.730
0.555
0.736
0.819
0.581
0.856
ESMM
34
0.745
0.506
0.525
0.708
0.545
0.693
0.823
0.563
0,852
KD-Label
30
0.760
0.509
0.562
0.726
0.583
0.752
0.815
0.570
0,858
AutoDebias
0.762
0,540
0,580
0.735
0.632
0.785
0.818
0.584
0.866
KD-Fcature
30
0.766
0.522
0.584
0.717
0.557
0.736
0.809
0.588
0.873
IPS 42
0.761
0,513
0.566
0.722
0.555
0.733
0.826
0.574
0.849
Multi-IPS 59
0.758
0.514
0.531
0.719
0.546
0.710
0.810
0.554
0,875
ESCM- IPS 44
0.757
0.514
0.558
0.729
0.559
0.714
0.815
0.577
0,860
RD-IPS g
0.764
0,514
0,566
0.730
0,571
0.735
0.832
0,585
0.873
BRD-IPS 8
0.763
0.511
0,564
0.735
0.582
0.743
0.834
0.588
0.877
Res-IPS (ours)
0.777
0.575
0.601
0.759
0.639
0.785
0.849
0.601
0.885
DR [2
0.766
0.525
0.552
0.725
0.553
0.727
0.824
0.567
0.838
Multi-DR 59
0.759
0.527
0.565
0.719
0.553
0.712
0.829
0.562
0.859
ESCM2-DR 44
0.760
0,553
0,568
0.715
0,566
0.722
0.827
0,569
0.830
RD-DR g
0.768
0.539
0.571
0.732
0.569
0.738
0.833
0.585
0.884
BRD-DR g
0.770
0.546
0.577
0.735
0.576
0.737
0.831
0.585
0.883
Res-DR (ours)
0.793
0.588
0.607
0.750
0.654
0.803
0.854
0.595
0.860
Table 2: Effects of varying unbiased data ratio on KUAIREC in terms of AUC and NDCG@50. The best two results are bolded, and the best baseline is underlined.
AUC
NDCG@50
Unhiased data ratio
2%
4%
6%
8%
10%
2%
4%
6%
8%
10%
CausR
0.818
0.818
0.819
0.819
0.819
0.579
0.580
0.584
0.586
0.587
KD-Label
0.815
0.815
0.815
0.816
0.816
0.582
0.584
0,588
0.588
0.589
AutoDebias
0.810
0.815
0.818
0.826
0.832
0.569
0.580
0,587
0.589
0.590
Res-IPS (ours)
0.845
0.848
0.850
0.850
0.852
0.595
0.596
0.602
0.602
0.603
Res-DR (ours)
0.850
0.851
0.854
0.855
0.855
0.592
0.593
0.597
0.605
0.606
datasets. KUAIREC is fully exposed dataset containing 4,676,570 video watching ratio records from 1,411 users for 3,327 videos. The records less than two are binarized to 0 and other records to 1.
Baselines. In our experiment, we take the widely-used Matrix Factorization (MF)
[22
as the base
model. We compare our methods with the debiasing methods: IPS
41 42
DR
39 52]. RD-IPS 8
RD-DR 8 and multi-task learning approaches: ESMM [34], Multi-IPS (59] Multi-DR [59]
ESCM2-IPS 44 and ESCM [44. We also compared the methods using both biased data and
unbiased data: CausE 1. KD-Label
300, KD-Feature 300 and AutoDebias [4.
Experimental Protocols and Details. We adopt three widely-used evaluation metrics: AUC, Recall @K (R @ K), and NDCG@* @K K (N@K) K) for debiasing performance evaluation. We set 5 for COAT and YAHOO! R3, and K 50 for KUAIREC. All the experiments are implemented on Pytorch with Adam as the optimizer. We tune learning rate in {0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05]. weight decay in {0, {0.1, 0.5, 1, 5, 10). and in {0.001, 0.005, 0.01, 0.05, 0.1}. In addition, we randomly split 5% of unbiased data from the test set to train models for all methods that require unbiased data
..... .,. 1e 1, 1}. For our methods, we tune a in (0.1, 0.5, 1]. 8 in
Performance Comparison. Table T shows the real-world debiasing performance for varying meth- ods three datasets. First, most debiasing methods outperform the base model, i.e., MF (bias), demonstrating the necessity for debiasing in the presence of selection bias. Second, methods using both biased and unbiased data outperform the methods using only one of them, which indicates that there exists some non-overlap information that can benefit for debiasing between both biased and
$ For all experiments, Tesla T4 GPU as the computational resource.
8

---

Table 3: Ablation study residual networks in Res-DR method, with AUC, NDCG@K and Recall @K as evaluation metrics. The best result is bolded and the second is underlined.
Training loss
COAT
YAHOO! R3
KUAIREC
Method
A8
Ap
AUC
R@5
N@5
AUC
N@5
R@5
AUC
N@:0
R@
ESCM'-DR
0.760
0.553
0.568
0.715
0.566
0.722
0.827
0.569
0.830
Res-DR w/o Ap A8
0.763
0.544
0.568
0.716
0.560
0.715
0.831
0.570
0.836
Res-DR w/o Ap
V
0.783
0.561
0.573
0.734
0.630
0.781
0.833
0.570
0.849
Res-DR w/o A6
X
0.768
0.555
0.581
0.721
0.645
0.791
0.841
0.579
0.836
Res-DR
0.793
0.588
0.607
0.750
0.654
0.803
0.854
0.595
0.860
Table 4: Ablation study on loss components in Res-IPS and Res-DR methods, with AUC, NDCG@K and Recall @ K evaluation metrics. The best result is bolded and the second is underlined.
Training loss
COAT
YAHOO! R3
KUAIREC
Method
CCvR
AUC
N@5
R@5
AUC
N@5
R@5
AUC
N@:0
R@
ESCM
X
X
0.757
0,514
0.558
0.729
0.559
0.714
0.815
0.577
0.860
IPSnone
X
0.755
0.522
0.546
0.722
0.552
0.707
0.825
0,580
0.853
Res-IPS-U
0.770
0.562
0.570
0.718
0.587
0.741
0.833
0.583
0.849
Res-IPS-B&U
X
0.784
0.573
0.592
0.756
0.635
0.778
0.845
0.592
0.880
Res-IPS
0.777
0.575
0.601
0.759
0.639
0.785
0.849
0.601
0.885
ESCM
X
0.760
0.553
0.568
0.715
0.566
0.722
0.827
0.569
0.830
Res-DR-None
X
0.765
0.544
0.550
0.714
0.575
0.735
0,824
0.562
0.823
Res-DR-U
0.770
0.565
0.577
0.722
0.604
0.756
0.836
0.562
0.842
Res-DR-B&U
0.790
0.574
0.601
0.744
0.640
0.782
0.848
0.586
0.848
Res-DR
0.793
0.588
0.607
0.750
0.654
0.803
0.854
0.595
0.860
unbiased data, and highlights the importance of leveraging both kinds of data. Meanwhile, the direct use of multi-task learning approaches to IPS and DR estimators cannot benefit the debiasing perfor- mance under hidden confounding. The proposed Res-IPS and Res-DR methods stably outperform previous methods on all three datasets, which provides empirical evidence of the existence of hidden confounding in the real-world recommendations, as well as the effectiveness of our methods for removing hidden confounding. Ishoes the results of AUC and NDCG@ 50 KUAIREC with varying unbiased data ratios. The performance of all methods improves with increasing unbiased data ratio, and our method stably outperforms the bascline methods by large margin.
A Ablation Studies. The two losses C'ÓvR and LVV as well as the two residual terms Ad and Ap are crucial in the proposed multi-task learning approach. We further conduct ablation studies with respect to the residual networks and the training loss components, respectively. From Table Res-DR using either the propensity residual network the imputation residual network can stably outperform ESCM2-DR, and Res-DR achieves the best performance when two residual networks are adopted together. When both propensity and imputation residual networks are removed, although Res-DR includes ClLvR and 2 losses, it has similar performance to ESCM2-DR, which further indicates that the performance improvement of Res- DR can be attributed to the effectiveness of the residual networks. From Table 4 both Res-IPS and Res-DR methods without CoTVR loss or without Leve loss outperform ESCM*= and ESCM²-DR. Similarly, when both two losses are preserved. Note that the model with LEVR loss performs better than the model with CLvvR loss, which is attributed to LEVA corrects for biased learned propensities and biased imputed errors under the presence of hidden confounding. In addition, unbiased data does not significantly improve the debiasing performance when it is directly used to train the prediction model through L' due the overfitting problem caused by the limited unbiased data size, which is consistent to the poor performance of MF (uniform) in Table Meanwhile, minimizing C does not provide any residual information for biased learned propensities and biased imputed errors.
methods achieve the best performance
In-Depth Analysis. The proposed methods contain the prediction model, imputation model, propen- sity model, and residual models, thus is meaningful to investigate the effect of different optimization algorithms among these models on the debiasing performance. Specifically, we implement the fol- lowing learning approaches on the Res-DR: (1) Joint Learning (JI.) (52). which joint optimizes the prediction model and the imputation model. (2) Double Learning (DL) μ. which adds parameter

---

0.80
065
080
0.75
0.70
0.75
S
8el
9.50
0.45
0.5
0.65
0.60
0.70
CR
0.5
0:50
0.35
0.40
No: 20-TTL
00.00
0.40
(ecCk]
0.25
Rec. 10c
0.50
10
20
30
40
0.55
5
10
20
40
0.25
0
10
au
30
40
0.20
0
30
20
40
Number of Iterations
Number of Iterations
Number of iteration
Number of Iterations
(a) R R@S on YAHOO! R3
(b) R@ on KUAIREC
(c) N@S on YAHOO! R3
(d) N@ on KUAIREC
Figure Joint learning (JL), double learning (DL), and task learning (MTL) on Res-DR.
Res 08
Red IPS
Ros DR
Res ESCX2-PS
Ros BR ESCMQ DR
Res ESCN2-1P
Res CR
Res PS
ESCN2-0R
ESCM2-1P5
ESCM2 DR
ESCM2 OR
ESCH2 PS
0.78
0.05
0.03
0.00
0.70
0.33
HDCG@S
0.62
NDCG@33
0.74
0.50
0.83
0.10
0.50
0.70
0.03 0,003 0883 Hyper-parametery Y
0.81
0.001 GUL
0.501
0.00 D65
0.55
0.001
G.0: 0.6
Hyper parameter
Hyper parampaarrttryyy
Hyper
(a) AUC on YAHOO! R3
(b) AUC KUAIREC
(c) N@5 YAHOO! R3
(d) N@ on KUAIREC
Figure 3: Effects of varying weights of the consistency loss
CB&ll CVR on Res-IPS and Res-DR.
sharing mechanism between the prediction model and the imputation model based on JL. (3) Multi- Task Learning (MTL) adopted in our methods. Figure Tigure[Jshows shows the experiment results. Remarkably. MTL significantly outperforms JL and DL, whereas in Table DDR and MTL-based ESCM perform similarly. This is because DR only has three training models (namely the prediction model, imputation model, and propensity model), whereas the proposed Res-DR has two additional residual models. As the number of models increases, JL can no longer bridge the models efficiently, which leads to a slow convergence and limited performance. Instead, DL increases the connection between models via timely parameter sharing. Finally, MTL trade-off's the different tasks in Res-DR to train all the models, leading to the optimal debiasing performance.
Sensitivity Analysis. We perform the sensitivity analysis of on Res-IPS and Res-DR, as shown in Figure 3 Our methods achieve the optimal performance when is moderate (0.005-0.01) This because when 7 is too large, hurts the performance of other tasks (e.g., CVR model training). and when is too small, it makes the consistency loss be paid with less attention, so that the hidden confounding cannot be effectively removed. Res-IPS and Res-DR stably outperform ESCM²-IPS and ESCMIDR under varying y. This further illustrates the effectiveness the consistency loss.
6 Conclusion
This paper investigates the use of a few unbiased ratings to calibrate the learned propensities and imputed errors for removing hidden confounding, First, we theoretically reveal the biasedness of previous debiasing methods in the presence of hidden confounding. Next, we propose a multi-task debiasing residual networks learning approach for training the debiased prediction model. By building residual networks and calibrating the biased learned propensities and biased imputed errors, the prediction model is trained on both the calibrated IPS or DR losses and the unbiased dataset to ensure the unbiasedness. Extensive experiments on two benchmark datasets and a large-scale industrial dataset validate the effectiveness of our proposal. A limitation of this work is the use of slightly more model parameters due to the need to address hidden confounding with residual networks.
Acknowledgements
This work is supported by National Key R&D Program of China (No. 2020YFE0204200 and National Natural Science Foundation of China (No. 62272437 and 61973329). Peng Wu is partially supported by the National Natural Science Foundation of China (No. 12301370) and the Disciplinary Funding of Beijing Technology and Business University (No. STKY202301). Zhi Geng is partially supported by the National Nature Science Foundation of China (No. 12071015).
10

---

References
[1] Stephen Bonner and Flavian Vasile. Causal embeddings for recommendation. In RecSys, 2018.
[2] Yin-Wen Chang, Cho-Jui Hsieh, Kai-Wei Chang, Michael Ringgaard, and Chih-Jen Lin. Train- ing and testing low-degree polynomial data mappings via linear svm. Journal of Machine Learning Research, (44), 2010.
[3] Jiawei Chen, Can Wang, Martin Ester, Qihao Shi, Yan Feng, and Chun Chen. Social recommen- dation with missing not random data. In ICDM, 2018.
[4] Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. AutoDebias: Learning to debias for recommendation. In SIGIR, 2021.
[5] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3):1-39 2023.
[6] H Christopher Frey and Sumeet R Patil. Identification and review of sensitivity analysis methods. Risk analysis, 22(3)-553-578 2002.
[7] Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Xiuqiang He, Rui Zhang, and Jie Sun. generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In KDD, 2022.
[8] Sihao Ding, Peng Wu, Fuli Feng, Xiangnan He, Yitong Wang, Yong Liao, and Yongdong Zhang. Addressing unmeasured confounder for recommendation with sensitivity analysis. In KDD, 2022.
[9] Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recommender systems. In CIKM, 2022.
[10] Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. Offline a/b testing for recommender systems. In WSDM, 2018.
[11) Siyuan Guo, Lixin Zou, Yiding Liu, Wenwen Ye, Suqi Cheng, Shuaiqiang Wang, Hechang Chen, Dawei Yin, and Yi Chang. Enhanced doubly robust learning for debiasing post-click conversion rate estimation. In SIGIR, 2021.
[12] Miguel A. Hernán and James M. Robins. Causal Inference: What If. Boca Raton: Chapman and Hall/CRC, 2020.
[13] Wenjie Hu, Kiao-Hua Zhou, and Peng Wu. Identification and estimation of treatment effects on long-term outcomes in clinical trials with external observational data. Statistica Sinica, 2023.
[14] Shanshan Huang, Haoxuan Li, Qingsong Li, Chunyuan Zheng, and Li Liu. Pareto invariant representation learning for multimedia recommendation. In ACM-MM, 2023.
[15] Paul Hünermund and Elias Bareinboim. Causal inference and data fusion in econometrics. 2019.
[16] Dietmar Jannach, Lukas Lerche, and Markus Zanker. Recommending based on implicit feedback. In Social Information Access, pages 510-569, Springer, 2018.
[17] Simon Jenni and Paolo Favaro. Deep bilevel learning. In ECCV, 2018.
[18] Nathan Kallus and Angela Zhou. Confounding-robuss policy improvement. In NeurIPS, pages 9289-9299, 2018.
[19] Nathan Kallus and Angela Zhou. Minimax-optimal policy learning under unobserved confound- ing. Management Science, 67(5):2870-2888 2890, 2021.
[20] Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experi- mental grounding. In NewrIPS, 2018.

---

[21] Ron Kohavi and Roger Longbotham. Online controlled experiments and a/b testing, Encyclope- dia of Machine Learning and Data Mining. 2017.
[22] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.
[23] Haoxuan Li, Quanyu Dai, Yuru Li, Yan Lyu, Zhenhua Dong, Xiao-Hua Zhou, and Peng Wu. Multiple robust learning for recommendation. In AAAI, 2023.
[24] Haoxuan Li, Taojun Hu, Zetong Xiong. Chumyuan Zheng, Fuli Feng. Xiangnan Hc, and Xiao- Hua Zhou. ADRNet: A generalized collaborative filtering framework combining clinical and non-clinical data for adverse drug reaction prediction. In RecSys, 2023.
[25] Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. IDR-CL: Targeted doubly robust collaborative learning for debiased recommendations, In ICLR, 2023.
[26] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confound- ing with a few unbiased ratings in debiased recommendations. In WWW, 2023.
[27] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Measuring and enhancing balancing for recommendation. In ICML., 2023.
[28] Haoxuan Li, Chunyuan Zheng, and Peng Wu. StableDR: Stabilized doubly robust learning for recommendation on data missing not at random. In ICLR, 2023.
[29] Haoxuan Li, Chunyuan Zheng, Peng Wu, Kun Kuang, Yue Liu, and Peng Cui. Who should be given incentives? counterfactual optimal treatment regimes learning for recommendation. In KDD, 2023.
[30] Dugang Liu, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. A general knowledge distillation framework for counterfactual recommendation via uniform data. In SIGIR, 2020.
[31] Dugang Liu, Pengxiang Cheng, Hong Zhu, Zhenhua Dong, Xiugiang He, Weike Pan, and Zhong Ming. Mitigating confounding bias in recommendation via information bottleneck. In RecSys, 2021.
[32] Dugang Liu, Pengxiang Cheng, Zinan Lin, Jinwei Luo, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. KDCRec: Knowledge distillation for counterfactual recommendation via uniform data. IEEE Transactions Knowledge and Data Engineering, 2022.
[33] Jinwei Luo, Dugang Liu, Weike Pan, and Zhong Ming. Unbiased recommendation model based on improved propensity score estimation. Journal of Computer Applications, 2021.
[34] Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In SIGIR, 2018.
[35] Xuying Meng, Suhang Wang, Kai Shu, Jundong Li, Bo Chen, Huan Liu, and Yujun Zhang Personalized privacy-preservve social recommendation. In AAAI, 2018.
[36] Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, second edition, 2009.
[37] Mengye Ren, Wenyuan Zeng. Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. ICML, 2018.
[38] Paul R Rosenbaum, Rosenbaum, and Briskman. Design of observational studies, volume 10. Springer, 2010.
[39] Yuta Saito. Doubly robust estimator for ranking metrics with post-click conversions. In RecSys, pages 92-100, 2020.
[40] Yuta Saito and Masahiro Nomura. Towards resolving propensity contradiction in offline recommender learning. In IJCAI, 2022.
12

---

[41] Yuta Saito, Suguru Yaginuma. Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing- not-----ot-t-t---nioo implicit feedback. In WSDM, 2020.
[42] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In ICML., 2016.
[43] Zijie Song, Jiawei Chen, Sheng Zhou, Qihao Shi, Yan Feng, Chun Chen, and Can Wang. CDR: Conservative doubly robust learning for debiased recommendation. In CIKM, 2023.
[44] Hao Wang, Tai- Wei Chang, Tianiio Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. ESCM2: Entire space counterfactual multi-task model for post-click conversion rate estimation. In SIGIR, 2022.
[45] Hao Wang. Zhichao Chen, Jiajun Fan, Haoxuan Li, Weiming Liu, Tianqiao Liu, Quanyu Dai, Yichao Wang, Zhenhua Dong, and Ruiming Tang. Optimal transport for treatment effect estimation. In NeurIPS, 2023.
[46] Haotian Wang, Wenjing Yang, Longqi Yang, Anpeng Wu, Liyang Xu, Jing Ren. Fei Wu, and Kun Kuang. Estimating individualized causal effect with confounded instruments. In KDD, 2022.
[47] Haotian Wang, Kun Kuang, Haoang Chi, Longgi Yang. Mingyang Geng, Wanrong Huang, and Wenjing Yang. Treatment effect estimation with adjustment feature selection. In KDD, 2023.
[48] Haotian Wang, Kun Kuang. Long Lan, Zige Wang, Wanrong Huang, Fei Wu, and Wenjing Yang. Out-of-distributien generalization with causal feature separation. IEEE Transactions on Knowledge and Data Engineering, 2023.
[49] Jun Wang, Haoxuan Li, Chi Zhang, Dongxu Liang, Enyun Yu, Wenwu Ou, and Wenjia Wang. Counterelr: Counterfactual contrastive learning with non-random missing data in recommenda- tion. In ICDM, 2023.
[50] Qinyong Wang, Hongzhi Yin, Tong Chen, Junliang Yu, Alexander Zhou, and Xiangliang Zhang. Fast-adapting and privacyppeeeervig federated recommender system. The VLDB Journal, pages
1 1-20, 2021.
[51] Wenjie Wang, Yang Zhang, Haoxuan Li, Peng Wu, Fuli Feng, and Xiangnan He.. Causal recommendation: Progresses and future directions. In Tutorial on SIGIR, 2023.
[52] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recom- mendation on data missing not at random. In ICML, 2019.
[53] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Combating selection biases in recom- mender systems with a few unbiased ratings. WSDM, 2021.
[54] Zimu Wang, Yue He, Jiashuo Liu, Wenchao Zou, Philip S Yu, and Peng Cui. Invariant preference learning for general debiasing in recommendation. In KDD, 2022.
[55] Peng Wu, Shasha Han, Xingwei Tong, and Runze Li. Propensity score regression for causal inference with treatment heterogeneity. Statistica Sinica, 2022.
[56) Peng Wu, Haoxuan Li, Yuhao Deng, Wenjie Hu, Quanyu Dai, Zhenhua Dong, Jie Sun, Rui Zhang, and Xiao-Hua Zhou. On the opportunity of causal learning in recommendation systems: Foundation, estimation, prediction and challenges. In IJCAI, 2022.
[57] Peng Wu, Zhiqiang Tan, Wenjie Hu, and Xiao- Zhou. Model-assisted inference for covariate- specific treatment effects with high-dimensional data. Statistica Sinica, 2022.
[58] Shu Yang and Peng Ding. Combining multiple observational data sources to estimate causal effects. Journal of the American Statistical Association, 1 115:1540-1554 1554,, 2020.
[59] Wenhao Zhang, Wentian Bao, Xiao Yang Liu, Keping Yang, Quan Lin, Hong Wen, and Ramin Ramezani. Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning. In WWW, 2020.
[60] Hao Zou, Haotian Wang, Renzhe Xu, Bo Li, Jian Pei, Ye Jun Jian, and Peng Cui. Factual observation based heterogeneity learning for counterfactual prediction. In 2nd Conference on Causal Learning and Reasoning, 2023.
13


Reviews:
[{"review_id": "sTzzHwoM4F", "reviewer": "Reviewer_y77Q", "summary": "The paper shows that existing approaches that are based on multi-task learning or take advantage of unbiased data have theoretical limitations in the problem of recommendation debiasing when hidden confounding is present. The paper proposes to address these limitations by a unified multi-task learning approach whose key idea is to remove hidden confounding by calibrating learned nominal propensities and nominal error imputations by a consistency loss on unbiased data. The paper conducts extensive experiments to confirm the effectiveness of the proposed approach on 3 widely used recommendation datasets.", "questions": "Q1: Do you have any explanations why the proposed approach underperforms RD-DR [7] in terms of R@50 on the KUAIREC dataset?\n\nQ2: Is N@K in Tables 1 and 3 NDCG@K? If so, it is important to clarify the meaning of N@5 (and R@5) in the main text as it is not common to abbreviate NDCG@K as N@5. \n\n[7] Addressing unmeasured confounder for recommendation with sensitivity analysis", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1: The idea of tackling hidden confounding by a multi-task learning approach using unbiased data to calibrate learned nominal propensities and nominal error imputations is novel to me.\n\nS2: The paper compares the proposed approach against a wide range of existing representative approaches on 3 benchmark recommendation datasets under commonly used evaluation metrics.\n\nS3: The paper does a great job in reviewing major research directions in the problem of recommendation debiasing and the references cover a full spectrum of related works.\n", "weaknesses": "W1: It is not fair to claim that the paper is the first to \"perform theoretical analysis to reveal the possible failure of previous approaches\" as Ding et al. has already done a majority of such theoretical analysis [7], which is cited by the paper.\n\nW2: The proofs for the theoretical results presented in the paper are a bit hand waved. Take Theorem 2 as an example. It is not unclear what is the space spanned by x and why the fact that \\bar{p} will not degenerate a point in the space spanned by x leads to the existence of a positive constant \\eta that satisfies the condition.\n\n[7] Addressing unmeasured confounder for recommendation with sensitivity analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the helpful suggestions. **Below, we hope to address your concerns and questions to improve the clarity and quality of our paper.** \n  \n  > **W1:** It is not fair to claim that the paper is the first to \"perform theoretical analysis to reveal the possible failure of previous approaches\" as Ding et al. has already done a majority of such theoretical analysis [7], which is cited by the paper.\n\n**Response to W1:** The reviewer might have misunderstood. We do not claim that we are the first to do this. In lines 7-13, we simply say \"We **first** perform theoretical analysis ...... **Then**, we propose ....\".  : ) Please kindly note that we explicitly cite such theoretical analysis in [7] in line 101 and line 104. We fully agree that this part is not our theoretical contribution. Instead, this paper provides a novel proof to shown the biasedness of AutoDebias and LTD under unobserved confounding in line 122.\n\nBelow, we kindly provide a detailed comparison table to illustrate the difference between [7] and our work for enhancing clarity of our work.\n\n|**Paper**      | **[7] Addressing unmeasured confounder for recommendation with sensitivity analysis** | **Ours** | \n|  :---        |    :----   |     :--- |\n|Dataset(s)| **Only** biased data| Biased data **with a few unbiased data**|\n|Method|**Sensitivity analysis,** which is a **Minimax** approach   | Using **unbiased data** to **calibrate** learned nominal propensities and nominal error imputations |\n|Theoretical result |Only **mitigate the unobserved confounding**, see the last paragraph of section 3.2 in [7] :\" **Instead of aiming to eliminate the unmeasured confounding thoroughly,** the proposed RD framework ..., which provides a flexible way to **mitigate the unmeasured confounding**\"| **Entirely eliminate the unobserved confounding** with a few unbiased ratings|\n|Assumption| **Assumption** that \"assume the nominal propensity score can be expressed as ...\" in Section 3.2 in [7] |**No additional assumptions,** except the need to use unbiased data|\n|Unbiasedness|**No**| **Yes (with the help of the unbiased data)**|\n|Learning Approach|**Joint learning,** i.e., alternative update (See Alg. 1 in [7])|**Multi-task learning**|\n\n> **W2:** The proofs for the theoretical results presented in the paper are a bit hand waved. Take Theorem 2 as an example. It is not unclear what is the space spanned by x and why the fact that \\bar{p} will not degenerate a point in the space spanned by x leads to the existence of a positive constant \\eta that satisfies the condition.\n\n**Response to W2:**  We apologize for the lack of clarity.  **Next, we give a more detailed proof of Theorem 2**. \n\nIn the setting of MNAR, $r_{u,i}$ has a non-zero effect on $o_{u,i}$, so $p_{u,i} \\neq \\bar p_{u,i}$ according to the definition of MNAR.  Thus, for some $\\epsilon>0$,   there exist  positive constants  $\\delta_\\epsilon,N_1(\\epsilon)>0$,  such that for all $|\\mathcal{D}|>N_1(\\epsilon)$, \n$$  \\mathbb{P}(  | \\bar p_{u,i} -  p_{u,i} | \\geq \\epsilon )\\geq \\delta_\\epsilon > 0.   $$\n  Without loss of generality, if $\\hat p_{u,i}$ is the learned propensity by fitting $o_{u,i}$ with $x_{u,i}$, then $\\hat p_{u,i}$ essentially estimates $\\mathbb{E}[o_{u,i}=1|x_{u,i}] =\\mathbb{P}(o_{u,i}=1 | x_{u,i}) = p_{u,i}$. Then, there exists some  $N_2(\\epsilon)>0$,  such that   for all $|\\mathcal{D}|>N_2(\\epsilon)$,\n$$   \\mathbb{P}(  | \\hat p_{u,i} -  p_{u,i} | \\geq \\frac{ \\epsilon}{2} )<\\frac{\\delta_\\epsilon}{4}.  $$\n  Thus, if $|\\mathcal{D}|> \\max \\\\{ N_1(\\epsilon), N_2(\\epsilon) \\\\}$, we have \n$$\n  \\mathbb{P}( | \\bar p_{u,i} -  p_{u,i} | \\geq \\epsilon, | \\hat p_{u,i} -  p_{u,i} | < \\frac{ \\epsilon}{2} ) \n= \\mathbb{P}(  | \\bar p_{u,i} -  p_{u,i} | \\geq \\epsilon )-\\mathbb{P}( | \\bar p_{u,i} -  p_{u,i} | \\geq \\epsilon, | \\hat p_{u,i} -  p_{u,i} |  \\geq \\frac{ \\epsilon}{2} )\n\\geq {\\delta_\\epsilon} -\\frac{\\delta_\\epsilon}{4} =\\frac{3}{4} \\delta_\\epsilon.\n$$ \n  Let $\\eta=\\epsilon/2$. Since $\\\\{| \\bar p_{u,i} -  p_{u,i} |\\geq \\epsilon, | \\hat p_{u,i} -  p_{u,i} | < { \\epsilon}/{2}\\\\} \\subset \\\\{| \\hat p_{u,i} - \\bar p_{u,i} | \\geq \\eta\\\\}$, we have\n$$  \\mathbb{P}(  | \\hat p_{u,i} - \\bar p_{u,i} | \\geq \\eta )\\geq \\mathbb{P}( | \\bar p_{u,i} -  p_{u,i} |\\geq \\epsilon  ,| \\hat p_{u,i} -  p_{u,i} | <\\frac{ \\epsilon}{2} )> \\frac{3}{4}\\delta_\\epsilon.  $$ \n  Thus, \n$$ \\lim_{|\\mathcal{D}| \\to\\infty} \\mathbb{P}(  | \\hat p_{u,i} - \\bar p_{u,i} | \\geq \\eta) \\geq\\frac{3}{4}\\delta_\\epsilon>0.  $$\n  Similarly, it can be shown that \n$$ \\lim_{|\\mathcal{D}| \\to\\infty} \\mathbb{P}(  | \\hat \\delta_{u,i} - \\bar g_{u,i} | \\geq \\eta) >0.  $$\n\n> **Q1:** Do you have any explanations why the proposed approach underperforms RD-DR [7] in terms of R@50 on the KUAIREC dataset?\n\n**Response to Q1:**   Thanks for your comments. We would like to emphasize that the **overall performance** of the proposed methods is the best among all the methods on all three datasets. In addition, we do carefully tune the parameters of all baseline models, so that some of them will have a competitive result. \n \n> **Q2:** Is N@K in Tables 1 and 3 NDCG@K? If so, it is important to clarify the meaning of N@5 (and R@5) in the main text as it is not common to abbreviate NDCG@K as N@5. \n\n**Response to Q2:** We thank the reviewer for pointing out this issue. Yes, in Tables 1 and 3, N@K means NDCG@K and R@K means Recall@K. We will clarify this in our revised manuscript. \n\n***\n\n**We sincerely thank you for your feedback and will provide more clarifications and explanations in the revised version, and welcome any further technical advice or questions on this work and we will make our best to address your concerns.**\n "}, {"review_id": "7dhjc9vQtA", "reviewer": "Reviewer_Uwjf", "summary": "This paper studies unbiased learning in recommendation systems in the presence of hidden confounding. The authors first theoretically analyze the limitations of previous MTL methods and those combine some unbiased data and then design a unified MTL debiasing method by calibrating the learned nominal propensities and error imputations using a novel consistency loss. Extensive experiments on benchmark datasets have shown the effectiveness of their proposed method.\n\n", "questions": "How do you tune the hyperparameters $\\alpha, \\beta, \\gamma$? Do you use a validation set? If so, how do other baselines use the validation set?", "limitations": "The authors have a discussion of the limitations in the Conclusion section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work is well-motivated with the analysis of the limitations of existing work in the presence of hidden confounding. \n2. The presentation of the method design is clear and vivid. Each part of the optimization goal is stated in a good structure.\n3. Extensive experiments of comparison with various baselines and in-depth analysis are persuasive, which shows the effectiveness of the designed method and the function of each component.", "weaknesses": "The novelty of this work lies in the proposed consistency loss that utilizes unbiased data to calibrate the learned nominal propensities and imputed errors from the biased data. However, the theoretical analysis of this term seems trivial (Proposition 3). Maybe the trade-off between different terms in $\\mathcal{L}_{Res}$ can be discussed.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the helpful suggestions. **Below, we hope to address your concerns and questions to improve the clarity and quality of our paper.** \n\n> **W1:** The novelty of this work lies in the proposed consistency loss that utilizes unbiased data to calibrate the learned nominal propensities and imputed errors from the biased data. However, the theoretical analysis of this term $\\mathcal{L}_{res}$ seems trivial (Proposition 3).\n\n**Response to W1:** We agree with the reviewer that it is not hard to derive Proposition 3, thus it should not be regarded as the core contribution of this paper. However, **it provides us a very intuitive way to understand the design of $\\mathcal{L}_{CVR}^{B\\\\&U}$, which is important to make the presentation clearer.** \n\n> **W2:**  Maybe the trade-off between different terms in $\\mathcal{L}_{res}$  can be discussed.\n\n**Response to W2:** We thank the reviewer for pointing out this issue. Please kindly refer to the discussion of the trade-off between different terms in $\\mathcal{L}_{res}$ in below.\n\n- First, **we perform the sensitivity analysis of $\\gamma$, which is the weight for $\\mathcal{L}_{CVR}^{B\\\\&U}$ in Res-IPS and Res-DR.**  The associated results are displayed in Figure 3, which indicates that our methods perform best when $\\gamma$ is moderate.\n- This is because when $\\gamma$ is too large, it hurts the performance of other tasks (e.g., debiased CVR model training), and when $\\gamma$ is too small, it makes this balance loss be paid with less attention, so that the residuals are no longer sufficiently updated. \n- Meanwhile, **we conduct ablation studies for Res-IPS and Res-DR, with respect to the residual components and the training losses, respectively.** The results are shown in Table 3 and Table 4, from which one can see that our methods reach the best performance when both two losses are preserved. \n\n> **Q1:** How do you tune the hyperparameters $\\alpha, \\beta, \\gamma$?  **Q2:** Do you use a validation set? If so, how do other baselines use the validation set? \n\n**Response to Q1 and Q2:**  We thank the reviewer for raising this question. **Yes, we use a validation set to tune the hyper-parameters** and to decide the stop criterion **for all baseline methods.** Specifically, **for the stop criterion,** the training process is finished **when the predicted AUC value on the validation set no longer increases.** Also, **for tuning the hyper-parameters,** we tune $\\alpha$ in $\\\\{0.1, 0.5, 1\\\\}$, $\\beta$ in $\\\\{0.1, 0.5, 1, 5, 10\\\\}$, and $\\gamma$ in $\\\\{0.001, 0.005, 0.01, 0.05, 0.1\\\\}$ and use **grid search** to choose the hyper-parameters **which can achieve the highest AUC value on the validation set.**  \n\n***\n\n**We sincerely thank you for your feedback and will provide more clarifications and explanations in the revised version, and welcome any further technical advice or questions on this work and we will make our best to address your concerns.**\n"}, {"review_id": "ej4H9aZaIC", "reviewer": "Reviewer_J6YV", "summary": "This paper highlights the prevalent issue of selection bias in recommender systems, emphasizing the often-overlooked aspect of hidden confounding. Existing approaches and their limitations are discussed, with a special focus on hidden confounders. The authors then introduce a unified multi-task learning approach which utilizes a small set of unbiased ratings to calibrate nominal propensities and error imputations. This approach aims to handle hidden confounding, thereby achieving unbiased learning.", "questions": "In Figure 2, the higher weight of the consistency loss actually reduces the performance of the model which is not very intuitive. Based on the propositions, reduced consistency loss leads to unbiased calibration loss. With higher weight, we expect smaller consistency loss and in general a better calibration loss. Why do we observe a regression in the AUC?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper does an excellent job of dissecting existing propensity-based and multi-task learning methods theoretically to demonstrate how they can lead to biased learning in the presence of hidden confounding.\n\n- Although the multi-task learning approach for debiasing is not novel, the specific setting of calibrations on unbiased data is novel. It uniquely uses a few unbiased ratings to calibrate learned propensities and imputed errors from biased data, aiming to eliminate the biases caused by hidden confounding. The unified loss is also justified by theoretical analysis. \n\n- The proposed methods are extensively validated on three publicly available benchmark datasets, including a large-scale industrial dataset. This broad empirical evidence strengthens the claims made in the paper.", "weaknesses": "- The theoretical result is weak. It states that if the consistency loss is zero then the calibrated loss is unbiased. While the result does help justify the design of the loss function, it does not provide any guarantee regarding the unbiasedness of the learned model. The consistency loss might not reach zero or even be small enough to reduce the bias of the calibrated loss.\n\n- The overall model contains multiple loss components and multiple parameters. While the authors point out the potential issues of the existing minimax framework, the proposed solution requires more parameter tuning.\n\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the helpful suggestions. **Below, we hope to address your concerns and questions to improve the clarity and quality of our paper.**\n\n> **W1:** The theoretical result is weak. It states that if the consistency loss is zero then the calibrated loss is unbiased. While the result does help justify the design of the loss function, it does not provide any guarantee regarding the unbiasedness of the learned model. The consistency loss might not reach zero or even be small enough to reduce the bias of the calibrated loss.\n\n**Response to W1:** Thanks for your comments.  We are glad that the reviewer agree that the theoretical results do help justify the design of the loss function. Actually, all the theoretical results are intended to provide us with a very intuitive way to understand the design of each component of the proposed method, which is important to make the presentation clearer. \n\nWe fully agree with the reviewer that the consistency loss might not reach zero or even be small enough to reduce the bias of the calibrated loss. However, we would like to clarify that **this is a general problem with multi-tasking learning and is not unique to our method**, see references [1, 2, 3].  Actually, **all the methods cannot guarantee the exact zero for the loss functions.** For example,  propensity-based methods are unbiased when the propensity model is correctly specified, but no methods can guarantee and verify they obtain 100\\% accurate estimation of propensities. \n\nIn summary, the proposed method provides a framework to combat the risk of hidden/unmeasured confounding. In particular, **the consistency loss provides a ideal and natural optimization direction to debias in the presence of hidden/unmeasured confounding, and with theoretical guarantees when its value is zero.**  \n\n\n> **W2:** The overall model contains multiple loss components and multiple parameters. While the authors point out the potential issues of the existing minimax framework, the proposed solution requires more parameter tuning.\n\n**Response to W2:**  Thanks for your comments. Indeed, we agree with the reviewer that the proposed methods requires more parameter tuning. However, we would like to clarity that due to the usage of the unbiased dataset and the carefully designed model structure in the proposed method, it is easy to obtain several hyper-parameter combination of $\\alpha, \\beta, \\gamma$ with promising performance in practice.  \n\nActually, the proposed method mainly adds only an additional hyper-parameter $\\gamma$ empirically. This is because we can first implement ESCM$^2$ method (the state-of-the-art multi-task learning method for combating the selection bias) and then take it as a pre-trained model. Thus, **we essentially have only one hyper-parameter ($\\gamma$) instead of three, which is easy to implement (Please kindly refer to our codes in the Supplementary Material).**\n\n> **Q1:** In Figure 2, the higher weight of the consistency loss actually reduces the performance of the model which is not very intuitive. Based on the propositions, reduced consistency loss leads to unbiased calibration loss. With higher weight, we expect smaller consistency loss and in general a better calibration loss. Why do we observe a regression in the AUC? \n\n**Response to Q1:**  We thank the reviewer for pointing out this issue and apologize for the lack of clarity. As shown in Figure 2, our method performs best when $\\gamma$ is moderate. Here are the reasons. \n\n- **When $\\gamma$ is too large**, **though a higher weight can mitigate the shift between training set and testing set caused by unobserved confounding, it hurts the performance of other tasks** according the loss function $L_{Res}$ in lines 179-180. This is becase much attention is paid to $L_{CVR}^{B\\\\&U}$ (which strictly relies on $L_{CVR}^{B}$ and $L_{CVR}^{U}$), whereas the other loss functions (including $L_{CTR}$, $L_{IMP}$, $L_{CVR}^{B}$, $L_{CTCVR}^{B}$, and $L_{CVR}^{U}$) cannot be sufficiently optimized, which prevents the debiased learning of the CVR model (via $L_{CVR}^{B}$, $L_{CTCVR}^{B}$, and $L_{CVR}^{U}$), accurate learned propensities (via $L_{CTR}$), and accurate imputed errors (via $L_{IMP}$) from being realized, leading to sub-optimal prediction performance.\n\n-  **When $\\gamma$ is too small**, it makes this balance loss $L_{CVR}^{B\\\\&U}$ be paid with less attention, so that **the residuals are no longer sufficiently updated to combat the unobserved confounding.** \n\nTherefore, our methods perform best when $\\gamma$ is moderate. \n\n***\n\n**We sincerely thank you for your feedback and will provide more clarifications and explanations in the revised version, and welcome any further technical advice or questions on this work and we will make our best to address your concerns.**\n\n___\n**References**\n\n[1] Ma, Xiao, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In SIGIR, 2018.\n\n[2] Zhang, Wenhao, Wentian Bao, Xiao-Yang Liu, Keping Yang, Quan Lin, Hong Wen, and Ramin Ramezani. Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning. In WWW, 2020.\n\n[3] Wang, Hao, Tai-Wei Chang, Tianqiao Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. ESCM2: Entire Space Counterfactual Multi-Task Model for Post-Click Conversion Rate Estimation. In SIGIR, 2022.\n\n"}, {"review_id": "K2g8NVKl70", "reviewer": "Reviewer_csfC", "summary": "This paper presents a critical examination of prevalent debiasing methods in recommendation systems and their limitations in addressing hidden confounding factors. The authors underline that current methods—propensity-based, multi-task learning, and bi-level optimization—fail to mitigate selection bias when unobserved confounding variables exist. In response, they propose a unified multi-task learning approach that incorporates a small set of unbiased ratings to calibrate nominal propensities and error imputations, thereby reducing the influence of hidden confounding factors. The approach utilizes a newly introduced consistency loss for calibration, advancing the field's theoretical understanding of bias and confounding in recommendation systems. The paper further validates its method through comprehensive experiments on three public benchmark datasets, including a large-scale industrial dataset. The outcomes affirm the efficacy of their approach in countering selection bias and hidden confounding. The study hence significantly contributes to the realm of recommender systems, revealing theoretical shortcomings in current debiasing approaches, and offering a novel, robust method for reducing bias in practice.", "questions": "1. How good will the method be useful on real-world products? Assuming it is possible to get unbiased labels from evaluators, but usually at a much much smaller scale (<0.1%), and sparser (no consistent history of a user).", "limitations": "1. What are the assumptions about the hidden confounding? The main new term introduced L_{CVR}^{B&U} is not sufficiently discussed. If I understand correctly, the main assumptions are that the unbiased set has no hidden confounding and aligning the distribution of prediction on the biased set to that of the unbiased set should calibrate / de-confound the predictions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposed methods show quite significant improvements over the strong baselines.\n2. Good to interpret each term in the equations. Very helpful for readers to intuitively understand the equations.\n3. Code is attached, to be published with the paper.\n4. Extended and complete experiments conducted on real-world datasets including a large-scale industrial dataset with many baselines and ablation studies.", "weaknesses": "1. Reference numbers for equations are missing. Please add the reference numbers to make referring more easily.\n2. Where is Theorem 1? Or why not Lemma 1, Proposition 2, and Theorem 3?\n3. Please refer to the proofs in the Appendix in the main text when the corresponding theorem or lemma are given.\n4. Equation of L_{CVR}^{B&U} in Sect 4.4 is not very well-defined. The cross-entropy loss \\delta(., .) is defined for binary random variables, however, neither L_{CVR}^B or L_{CVR}^U is a binary random variable. Please spell out the specific definition, as it’s also one of the main contribution.\n5. Statistical significance analysis is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the helpful suggestions. **Below, we hope to address your concerns and questions to improve the clarity and quality of our paper.** Below we categorized the reviewers' concerns into **Methodology**, **Experiments**, and **Clarity**.\n\n### **Methodology**\n> **W4:** Equation of L_{CVR}^{B&U} in Sect 4.4 is not very well-defined. The cross-entropy loss \\delta(., .) is defined for binary random variables, however, neither L_{CVR}^B or L_{CVR}^U is a binary random variable. Please spell out the specific definition, as it’s also one of the main contribution.\n\n**Response to W4:** We apologize for the lack of clarity and its caused misunderstanding by the reviewer. **In fact, we use \\delta(. , .) to denote generic loss functions** in our manuscript and **are not limited to cross-entropy losses**. The reviewer is correct that **for L_{CVR}^{B&U}, we use the square of the difference between L_{CVR}^B and L_{CVR}^U.** This can also be verified in our released codes in the supplementary material. Instead, when defining the ideal loss, since this work considers binary ratings, we use cross-entropy loss in our experiments. We will clarify it in our revised version.\n\n> **Limitations:** What are the assumptions about the hidden confounding? The main new term introduced L_{CVR}^{B&U} is not sufficiently discussed. If I understand correctly, the main assumptions are that the unbiased set has no hidden confounding and aligning the distribution of prediction on the biased set to that of the unbiased set should calibrate / de-confound the predictions.\n\n**Response to Limitations:** With respect to the assumptions about hidden confounding, **the reviewer is correct: the main assumptions are that the unbiased set has no hidden confounding.**  Nevertheless, we would like to clarify that this is a truth for unbiased datasets, instead of an assumption about hidden confounding [1, 2]. To gather unbiased ratings, we may ask users to rate randomly selected\nitems. This way, the propensities of observing different ratings are the same and the observed ratings are thus unbiased.\n\nFor the term L_{CVR}^{B&U}, **the reviewer is also correct: we align the distribution of prediction on the biased set to that of the unbiased set to calibrate / de-confound the predictions.** The core idea to tackle the hidden confounding by a multi-task learning approach is **the using of unbiased data to calibrate learned nominal propensities and nominal error imputations,** which motivates our design for the consistency loss L_{CVR}^{B&U}.\n\n### **Experiments**\n\n> **W4:** Statistical significance analysis is missing.\n\n**Response to W4:** As suggested by the reviewer, we add the statistical significance test in the attachment PDF. The result shows that Res-IPS and Res-DR outperform other baseline methods.\n\n> **Questions:** How good will the method be useful on real-world products? Assuming it is possible to get unbiased labels from evaluators, but usually at a much much smaller scale (<0.1%), and sparser (no consistent history of a user).\n\n**Response to Questions:** Thanks for the question. Actually, we conduct the unbiased data ratio experiments from **2% to 10%** in the submitted paper. As suggested by the reviewer, we add a unbiased data ratio experiments from **0.05% to 10% with more subtle step size** in the attachment PDF. It shows that some of the baseline methods (like autodebias on KuaiRec) performance can drop off dramatically with very little unbias data. However, Res-IPS and Res-DR still outperform the baseline methods with varying unbiased data ratio. Please kindly refer to the attachment PDF for more details.\n\n\n### **Clarity**\n> **W1:** Reference numbers for equations are missing. Please add the reference numbers to make referring more easily.\n\n**Response to W1:** Thanks for your helpful suggestion, and we will add reference numbers for equations in our revised manuscript.\n\n> **W2:** Where is Theorem 1? Or why not Lemma 1, Proposition 2, and Theorem 3?\n\n**Response to W2:** We thank the reviewer for pointing out this issue, it seems our template generates the wrong number of lemmas, propositions, and theorems, and we apologize to this expected issue and will adjust the numbering of lemmas, propositions, and theorems in the revised version.\n\n> **W3:** Please refer to the proofs in the Appendix in the main text when the corresponding theorem or lemma are given.\n\n**Response to W3:** We agree with you and will add the \"please refer to the proofs in the Appendix\" in our main text when the corresponding theorem or lemma are given. Thanks again for helping us to improve the clarity of our paper.\n\n***\n**We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score.** We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!\n***\n**References**\n\n[1] Wang, Xiaojie, et al. \"Combating selection biases in recommender systems with a few unbiased ratings.\" Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 2021.\n\n[2] Chen, Jiawei, et al. \"AutoDebias: Learning to debias for recommendation.\" Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021."}, {"review_id": "JjpRPu1Its", "reviewer": "Reviewer_KtZ4", "summary": "The paper addresses that there are hidden confounders in recommendations and existing methods cannot handle them well. \\\nThe paper proposes a unified multi-task learning approach to tackle that problem. \\\nSpecifically, they devise a residual network to calibrate the propensity and the imputed error by using unbiased data.\n", "questions": "Please refer to Weaknesses.", "limitations": "There are many hyperparameters for the loss functions.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and presented.\n\n2. Each loss function is technically solid.\n- The proposed method is end-to-end multi-task learning with a debiasing residual network that simultaneously deals with selection bias, data sparsity, and hidden confounding.\n\n3. The proposed method is validated through experiments.\n- They adopt three real-world datasets, three metrics, and extensive baselines.\n", "weaknesses": "1. Motivation is weak.\n- Hidden confounders are assumed to exist. There is no theoretical or experimental evidence for that.\n- Proofs (Proposition 1, Theorem 2) are naive. They are proved on the assumption that is already the conclusion of the proofs.\n\n2. Proposed end-to-end framework is not novel. \n- There is already an end-to-end multi-task learning framework [7] removing hidden confounders with biased and unbiased data.\n- Loss functions are from the existing methods. Most of them are already adopted in a multi-task manner in ESCM^2 [33].\n- The unique contribution of this paper relies on unbiased data, which is hard to obtain in real-world applications.\n\n3. Missing related work\n- Balancing Unobserved Confounding with a Few Unbiased Rating in Debiased Recommendations, WWW 2023.\n- StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not At Random. ICLR 2023.\n- Multiple Robust Learning for Recommendation. AAAI 2023.\n- This work seems to tackle the same problem as the proposed work and needs to be included in the manuscript.\n\n4. Notations are confusing\n- There are no explicit loss forms. For example, what is $L^{B}_{CVR}$?\n- The argument of loss functions is not organized. I cannot distinguish which is the training parameter and which is the fixed parameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the helpful suggestions. **Below, we hope to address your concerns and questions to improve the clarity and quality of our paper.**\n\n> **W1:** Motivation is weak.\n\n- Hidden confounders are assumed to exist. There is no theoretical or experimental evidence for that. \n\n**Response to W1:** We thank the reviewer for pointing out this issue. Theoretically, the assumption of \"unconfoundedness\", i.e., the conditional independence of treatment and outcome given the covariates, cannot be tested with observational data, and thus we cannot theoretically prove the existence of hidden confounders [1]. However, experimentally, as summarized in a recent survey paper [2] on causal recommendation, a large amount of empirical evidence for the existence of hidden confounders has been explored.\n\n- Proofs (Proposition 1, Theorem 2) are naive. They are proved on the assumption that is already the conclusion of the proofs.\n\n**Response to W1:** We agree with the reviewer that it is not hard to derive Proposition 1 and Theorem 2, thus it should not be regarded as the core contribution of this paper. However, it provides us a very intuitive way to understand the motivation of calibrating the learned nominal propensities and nominal error imputations by a multi-task learning approach using unbiased data.\n\n> **W2:** Proposed end-to-end framework is not novel.\n\n- There is already an end-to-end multi-task learning framework [7] removing hidden confounders with biased and unbiased data.\n\n**Response to W2:** The reviewer raises an interesting concern. However, we have reflected this comment in Section 3.3, where we discuss the difference between our work and [7]. In summarize, [7] proposed to adopt sensitivity analysis from the causal inference literature to minimize the worst-case prediction loss. Whereas our work propose to tackle hidden confounding by a multi-task learning approach using unbiased data to calibrate learned nominal propensities and nominal error imputations.\n\n- Loss functions are from the existing methods. Most of them are already adopted in a multi-task manner in ESCM^2 [33].\n\n**Response to W2:** We thank the reviewer for pointing out this issue. However, we would like to emphasize that (a) the motivation and problem set are different: the main purpose of our work is to address unobserved confounding using unbiased data, whereas ESCM^2 only uses biased data to address selection bias; (b) the model setup is different: our work novelly introduces a residual imputation model and a residual propensity model to calibrate the learned biased nominal propensities and biased nominal error imputations due to unobserved confounding.\n\n- The unique contribution of this paper relies on unbiased data, which is hard to obtain in real-world applications.\n\n**Response to W2:** Thank you for the comment. We note that much recent work proposes to use a small amount of unbiased ratings to address selection bias, but fails to address unobserved confounding [3, 4]. Empirically, we add experiments in a supplemental one-page pdf that verify that even if the unbiased scoring has much much smaller scale (<0.1\\%), the proposed method still has promising performance. To gather unbiased ratings, we may ask users to rate randomly selected items. This way, the propensities of observing different ratings are the same and the observed ratings are thus unbiased.\n\n\n> **W3:** Missing related work.\n\n- Balancing Unobserved Confounding with a Few Unbiased Rating in Debiased Recommendations, WWW 2023.\n\n**Response to W3:** First, we would like to kindly remind the reviewers that the work mentioned was published online in 30 April 2023, see: https://dl.acm.org/doi/abs/10.1145/3543507.3583495, while our submission was in May, which is in line with NeurIPS submission policy for baseline comparisons (due to the very close time). **We add comparisons that use this work as a baseline, and the experimental results validate the superiority of our proposal (please kindly refer to the one-page supplementary pdf for the detailed experimental results).**\n\n- StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not At Random. ICLR 2023.\n\n- Multiple Robust Learning for Recommendation. AAAI 2023.\n\n**Response to W3:** We thank the reviewers for the useful suggestions. We kindly remind the reviewer that though both of them aim to combat the selection bias, neither of them tacking the effect of unobserved confounding. And as suggested by the reviewer, we will include them in our manuscript. **We also add comparisons that use these works as baselines in our one-page supplementary pdf.**\n\n> **W4:** Notations are confusing.\n\n- There are no explicit loss forms. For example, what is $L_{CVR}^B$?\n\n**Response to W4:** Please kindly refer to line 108.5 in our original manuscript, $L_{CVR}^B$ is either $L_{IPS}$ or $L_{DR}$.\n\n- The argument of loss functions is not organized. I cannot distinguish which is the training parameter and which is the fixed parameter.\n\n**Response to W4:** The reviewer raises an interesting concern. Following ESCM^2, our work uses multi-task learning to simultaneously train the propensity model, the CVR model, and the imputation model.\n\n***\n**We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score.** We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!\n***\n**References**\n\n[1] Imbens, Guido W., and Donald B. Rubin. Causal inference in statistics, social, and biomedical sciences. 2015.\n\n[2] Luo, Huishi, et al. \"A Survey on Causal Inference for Recommendation.\" 2023.\n\n[3] Wang, Xiaojie, et al. \"Combating selection biases in recommender systems with a few unbiased ratings.\" WSDM, 2021.\n\n[4] Chen, Jiawei, et al. \"AutoDebias: Learning to debias for recommendation.\" SIGIR, 2021."}]

Author rebuttal:
We sincerely thank you for all helpful suggestions. We add statistical significant results and more detailed results of the unbiased data ratio in the attached PDF. We welcome any further technical advice or questions on this work and we will make our best to address your concerns.

Meta-review: All the reviewers agree that the paper studies a well-motivated and significant problem of debiased learning of recommender systems, and proposes an effective multi-task learning approach that is robust in the presence of hidden confounders. The experiments are extensive and the reviewers appreciated the authors' several clarifications in their responses. The authors performed additional sensitivity analysis experiments, analyzed the performance of their method as more unbiased data becomes available, and statistical significance results. All of these additions substantially strengthen the paper and should be included in the revision.
Decision: Accept (poster)