Addressing Hidden Confounding with Heterogeneous Observational Datasets for Recommendation
Authors: Yanghao Xiao
Haoxuan Li
Yongqiang Tang
Wensheng Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Recommender System
Causal Inference
Bias
Hidden Confounding
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: The collected data in recommender systems generally suffers selection bias. Considerable works are proposed to address selection bias induced by observed user and item features, but they fail when hidden features (e.g., user age or salary) that affect both user selection mechanism and feedback exist, which is called hidden confounding. To tackle this issue, methods based on sensitivity analysis and leveraging a few randomized controlled trial (RCT) data for model calibration are proposed. However, the former relies on strong assumptions of hidden confounding strength, whereas the latter relies on the expensive RCT data, thereby limiting their applicability in real-world scenarios. In this paper, we propose to employ heterogeneous observational data to address hidden confounding, wherein some data is subject to hidden confounding while the remaining is not. We argue that such setup is more aligned with practical scenarios, especially when some users do not have complete personal information (thus assumed with hidden confounding), while others do have (thus assumed without hidden confounding). To achieve unbiased learning, we propose a novel meta-learning based debiasing method called MetaDebias. This method explicitly models oracle error imputation and hidden confounding bias, and utilizes bi-level optimization for model training. Extensive experiments on three public datasets validate our method achieves state-of-the-art performance in the presence of hidden confounding, regardless of RCT data availability.

Addressing Hidden Confounding with Heterogeneous Observational Datasets for Recommendation
Yanghao Xiao 1.3
Haoxuan Li Yongqiang Tang
Wensheng Zhang***
University of Chinese Academy of Sciences "Peking University
Institute of Automation, Chinese Academy of Sciences *Guangzhou University
xi xiaoyangha ucasuacaa.... ac. cn, hxli@stu, pku. edu. cn, yonggiang. tang@ia. ac. en, Zlangsenshhengiachhommal... com
Abstract
The collected data in recommender systems generally suffers selection bias. Con- siderable works proposed to address selection bias induced by observed user and item features, but they fail when hidden features (e.g., user age or salary) that affect both user selection mechanism and feedback exist, which called hidden con- founding. To tackle this issue, methods based on sensitivity analysis and leveraging few randomized controlled trial (RCT) data for model calibration are proposed. However, the former relies strong assumptions of hidden confounding strength, whereas the latter relies on the expensive RCT data, thereby limiting their applica- bility in real-world scenarios. In this paper, we propose to employ heterogeneous observational data address hidden confounding, wherein some data subject hidden confounding while the remaining is not. We argue that such setup is more aligned with practical scenarios, especially when some users do not have complete personal information (thus assumed with hidden confounding), while others do have (thus assumed without hidden confounding). To achieve unbiased learning, we propose novel meta-learning based debiasing method called MetaDebias. This method explicitly models oracle error imputation and hidden confounding bias, and utilizes bi-level optimization for model training. Extensive experiments on three public datasets validate our method achieves state-of-the-as performance the presence of hidden confounding, regardless of RCT data availability.
Introduction
Recommender systems have been developed with the purpose of providing users with personalized recommendations, serving potent tool in capturing users' true preferences [22]. In recent years, deep learning algorithms are proposed to train recommendation models with collected historical data. However, the selection bias introduced by the users' sclective interactions with items poses a challenge to the unbiased learning the prediction model 44 [44 59]. For instance, in explicit feedback data, users are free to rate items they prefer, thereby engendering the distribution discrepancy between interacted and non- non-interacted data [35 (5).
To alleviate selection bias, line of works are proposed, including eтror imputation 2 16
48).
inverse propensity weighting [2] 36 45 46), and doubly robust 43 55
Moreover, recent works have sought to enhance these foundational methods from diverse perspectives, including analysis
on both bias and variance 7 12 considerations for training stability 31[49, and integration
with multi-task learning [37 52 64
However, they all fail to achieve unbiased when some features
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

User Profile
User Profile
Dui
Du.i
Usernome
Username
Alice
Bab
Gender
Gender
Oul
Tu,i
Ou,
Tu,
Male
Age (Opmional)
Age (Optional)
hui
Cuay
Salory (Optional)
Salory (Optional)
2000
MNAR without hidden confounder
MNAR with hidden confounder
Ou.
Iu.I
9 u 1
Ju.i
(a) Toy examples of user information collection process. (b) The causal graphs of two types of MNAR data.
Figure I:
Toy examples and causal graphs of heterogeneous observational data, both which are missing not at random (MNAR) due to the selection bias. In causal graphs. bu,is Ou,i Tu, and hy, denote basic mandatory features, observation, rating and optional features, respectively, where observed and unobserved variables are represented by solid-line and dashed-lin line circles.
that simultaneously affect both user selection mechanism and feedback remain unobserved, which is called hidden confounding and is widely prevalent in real-world scenarios [10 28 29
To address hidden confounding, sensitivity analysis based methods assume the hidden confounding strength can be bounded such that the inverse of true propensity scores are near and bounded by the estimated values, and further adopt worst-case optimization [10 [6]. More recently, data fusion based methods leverage few RCT data collected from randomized controlled trials or A/B tests to calibrate propensity and imputation models to achieve unbiasedness [28 29]. Unfortunately. both of them prove challenging to implement in real-world scenarios, the former relies strong assumption hidden confounding strength, whereas the latter relies on the significantly expensive RCT data.
To fill this gap, this paper proposes to employ heterogeneous observational data to address hidden confounding, wherein some data affected by hidden confounding while the remaining is not. We argue that such setup is more aligned with practical recommendation scenarios, especially when some users/items do not have complete information (thus assumed with hidden confounding), while others do have (thus assumed without hidden confounding). For example, in Figure Ia during user registration process, some users such as Alice may complete both mandatory fields and optional fields like age and salary, while others such as Bob may only complete mandatory fields. Similarly, this also holds on the item side. We depict the corresponding causal graphs in Figure Tb where du,5 and ha, denote basic mandatory and optional features that affect both observation and feedback, respectively. To achieve unbiased learning with above heterogeneous observational data, we propose novel meta- learning based debiasing method called MetaDebias, to explicitly model the oracle prediction errors and the bias introduced by hidden confounders. We further adopt bi-level optimization technique to train the prediction model. The contributions of this work summarized as follows.
To the best of our knowledge, this is the first work to employ heterogeneous observational data to address hidden confounding in debiased recommendation, relaxing the reliance on RCT data in previous data fusion methods.
We propose a meta-learning based debiasing method called MetaDebias to explicitly model the oracle error imputation, and employ bi-level optimization for model training.
We conduct extensive experiments on three public datasets, and our method achieves state-of-the-art performance in the presence of hidden confounding, regardless of the availability of RCT data.
2 Preliminaries and Related Work
Let 4 = {u). I (i). and D = {(,, | u U, E Σ} be the set of users, the set of items, and the target population consisting of all user-item pairs, respectively. Define Tu, and Ta.s be the observed features and rating of user-item pair (u, i), and suppose observation indicator ou, {0, 1}

---

where cu,i = L(fu, Tu,t) is the prediction error and L(:,) .) is a pre-specified loss function such as the squared loss Cu,i = (Pu, r. However, rating ru,i is observed only when Ou,i = and naively minimizing prediction loss the observed population O will suffer from bias and lead to be binary treatment variable, where ou,i = indicates Tu,i is observed, otherwise is not. Let O {(u,)) (u,i) E D, Ou.i 1] be the observed population consisting of user-item pairs with observed ratings. Denote and E be the distribution and expectation on the target population DD
We denote prediction model with parameter 0 fu.; ( which aims to predict true ratings. If all the ratings {ru. (u, 2) E D} are observed, i.c., O = D, the parameter 0 can be trained by minimizing the ideal loss:
Cideal (0)
Cu,i
D
(u,i)ED
sub-optimal performance (55). This attributed to the disparities between the observed population
O and the target population D, which is called selection bias.
2.1 Addressing Selection Bias without Hidden Confounding
To address selection bias, prior works preliminarily assume the absence of hidden confounding in the collected data, implying that the observed features zu,i include all the possible confounders. Without loss generality, we define a binary variable Ju,i E {0, 1 as the data source indicator which measures the completeness of observed features Tu, As shown in the toy example in Figure [ Ju.i = indicates that all confounders are completely observed, while Ju.; = 0 indicates that some hidden confounders exist. To ensure consistency in the input, fill the missing values in Z with 0, so that the observed features Zu,, have the same dimensions when Ja,i = and 9u = 1.
Assumed no hidden confounding (fa.i = 1, (u, i) E DD, it holds that Gu,i 1l Ou.i Tu,i, Ju.i the widely adopted propensity based methods are proposed. Specifically, the Inverse Propensity Scoring (IPS) estimator [46] reweights each sample based on the probability of being observed and is given as
1
Ou,iCa.i
LiP
D
Pu,i
(u,))ED
where Pu.; is the estimation of propensity Pa,i - P(ou,i 1/ Bu. 1). Furthermore, the Doubly Robust (DR) estimator [4] further extends IPS with error imputation and is presented as
[Rn,s + Ou,i êu,i)
LD (0)
Σ (u,1)ED
DD
Pu,i
where ē,, is the imputed error. Based IPS and DR estimators, many variants have been proposed, and please refer to Appendix A. for more details.
2.2 Addressing Selection Bias with Hidden Confounding
However, in some real-world scenarios, the observed features do not include all confounders and hidden confounding exists. Based on the incomplete observed features Lu,i with Ju, = O, it holds that ou, IL Cu.i Lu,is In.i O, which prevents previous methods such as DR from achieving unbiasedness. To further mitigate the hidden confounding, methods based on sensitivity analysis and model calibration using RCT data have been proposed.
Sensitivity Analysis. Inspired from causal inference literature, sensitivity analysis based approach assumes the hidden confounding strength can be bounded such that the inverse of true propensity scores are near and bounded by the estimated values, and adopt worst-case optimization to mitigate hidden confounding [10, 66) However, above strong assumption hidden confounding strength is hard to be satisfied in real-world scenarios and such method fails when the assumption is violated.
Model Calibration. Recent works propose to leverage few unbiased RCT data collected from randomized controlled trials or A/B tests for model calibration 28 29]. Collecting RCT data requires users to rate items randomly, hence RCT samples are representatives of the target population, and the prediction loss on these samples serves as an unbiased estimator of the ideal loss. Thus, the biased propensity and imputation models learned from the incomplete observed features Zu,i can be

---

corrected using such unbiased loss, for instance, with the help of additive residual models 22 or multiplicative reweighting models [29]. However, the acquisition cost of RCT data is prohibitively high, posing challenges to the practical implementation of such methods in real-world settings.
Apart from debiased recommendation, more related works about causal inference under hidden confounding can be found in Appendix A.I A.I which is widely studied topic.
3 MetaDebias: Meta Learning Based Debiased Recommendation Approach
3.1 Problem Formulation
We first present the problem formulation, that is to provide unbiased estimator for the ideal loss Lideal in the presence hidden confounding, given the collected heterogeneous observational data with known data source indicator gu.i. Unlike existing works that assume training data originates from a single dataset without or with hidden confounding, we argue that the training data is more likely composed of two heterogeneous datasets, one of which has sufficient feature collection and is assumed without hidden confounding (Mu.: = 1), while the other has insufficient feature collection and is assumed with hidden confounding (9u.: = with hidden confounders denoted as ha.
Here, based on the given Ju,i E {0, 1] for distinguishing between two types of observational data, partition the observed population O into
subpopulations Do {(u, i) (u, i) E DD Iu.; 0, Ou,i = 1} and D2 = [u, i) (u, i) E DD Du.u 1, ou, = D. ((u, (u, E D, ou,i = 0}. A naive approximation method for ideal loss is minimizing the prediction loss over the observed data. Specifically, given Do and D,, the respective losses over each subgroup are follows:
=1} 1}, while unobserved population is
LD.
Σ (u,i)EDo
Cu,i
LD.
Cu.i.
Dul
D1
(u,1)
However, due to the selection bias, neither LD., LD, nor their combination LNavve = LD. + LD, is unbiased estimator for the ideal loss. Furthermore, discussed in the Preliminaries previous methods cannot be applied to achieve the goal without additional hidden confounding strength or RCT data assumptions.
2
3.2 Methodology
To achieve the goal, we propose to explicitly estimate the prediction error eu, on all user-item pairs D using observed features Tu,i as the ideal loss is defined as the average prediction error over D, i.e., Cideal [D] E(u.)eD Cu,s. In other words, our goal is transformed into accurately estimating the oracle error imputation E [ea,, zu,il, which can inherently derive an unbiased estimator.
The challenges in achieving accurate oracle error imputation estimation lie in the fact that the prediction error Eu,; is only partially observable, and the missing mechanisms differ between the two subgroups D, and Dj due to the influence of hidden confounding. To address these challenges, define the identifiable propensity scores aimed at modeling the two types missing mechanisms, imputations defined over the entire space D, to assist in estimating the
along with the naive target oracle error imputation. Next, we will introduce the details.
To start with, different from the previously widely used propensity score PP (raas) which is applicable only for samples in subgroup D, and the generalized formulation P( = 1 a by,s) designed for subgroup Do which is unidentifiable, we define the identifiable propensity score π g) which models the probability of being observed for any user-item pair on the entire space D:
x (x. g) P (ou,s 1 Fn,. = x. Gu.i g)
The proposed Txx g) models the propensity confounding, based on the input data source indicator 9n.i-
for both the absence and presence of hidden
Similarly, propose to use the observed features In,; and data source indicator Iu,i to estimate the naive prediction error as Ou,i Ca,is which computable on the target population D and expected to be the prediction error Eu,i when sample (u, i) is observed ou, 1, or to be zero when not

---

observed ou, = O. Note that the zero value in the defined naive prediction enror Ou,i Cu, contains the information about data missing mechanism, thus gu used to capture the differences in the missing mechanism when hidden confounding is present or absent. Formally, to achieve this target, we define the naive error imputation m(x.)) g) as:
ma 9) - Eloo,t Cu,i Fu,i gu,i - g].
Based on the proposed propensity score π (2, g) and naive error imputation m g). the oracle error imputation E (e Tu,zl can be derived, we summarize their relationship in Lemma below.
Lemma 1. The relationship between the proposed propensity score ι (F, naive error imputation m(x, g) and the oracle error imputation E [eu, (u,i is as follows:
m(x, g) = { [eu,i Tu, = x]+ (1 g)n(1)} (xx,)).
where n(x) =E [eu, Fu, = 2, Fu.t 0, Ou, 1] EE [eu, Fu,i = x].
The above result delineates the conditional independence relationship between observation and prediction error. Specifically, on subgroup D where no hidden confounding exists, the prediction enror G is independent of the observation Ou,i given the observed features Fu,i i.., Cu,, Il Ou,i Tu.i Iu.i = I, and the expectation of Ou,i EGui naturally equals the product of their respective expectations shown in Equation 3
However, on subgroup Do where hidden confounders hy,5 exist, above conditional independence does not hold, thus we introduce an additional residual module 17(z) which describes the expectation difference between the mean prediction eгrоr over target population D and the biased subpopulation Do. to account for the bias introduced by hidden confounders.
Lemma 1 provides an approach for estimating the oracle error imputation [eu, [u..)) and the estimation depends solely on propensity score π (x. g) and naive error imputation m(x, g). Note that both न (xx,)) g) and m(x, g) are typically estimated from the heterogeneous observational datasets, which implies that when the estimation of (2, g) and m(x,9) g) are inaccurate, the resultant oracle error imputation estimation will further deteriorate in accuracy.
To address this limitation, we propose further incorporating additional data information including the observation ou, and the naive prediction eиror Ou,i Cu, achieve more robust estimation of the oracle error imputation. The details are shown in Lemma] 2 below.
Lemma 2. The relationship between the propensity score x<,, g). naive error imputation m(fa, g). observation Ougi naive prediction error Ou,i Cu, and oracle error imputation EE [ex,, Tuae is:
Ou,i - Cu,i - m((x g) = {E (eu, Tu,i = x + (1 g)m(m}} } fou,i <(I, + E,
where & = =u, {eu, {E(eu,t Da.s = x] + (1 - g))(m)}}}} with B[g x, r. =0 O.
The findings in Lemma 2 provide an alternative and robust approach for estimating the oracle error imputation E (eu,i J...]. wherein the estimation depends not only on the learned propensity score (I. g). naive error imputation m( g), but also on the observation 0g along with the naive prediction error ou, Cu,i, where with zero-mean property can be regarded as a noise.
3.3 Training Objectives
For the method implementation, employ deep learning models to estimate the aforementioned proposed modules, and name this approach as MetaDehias. We present the architecture of MetaDebias in Figure 2 and introduce the training objective of each model as follows.
Initially, we employ the commonly used cross-entropy loss to train the propensity model (<x, g: dx) with parameters σxx using heterogeneous data, denoted as L. (0xx shown in Equation 5 below:
L. ( =
Σ (u,i)DD
[-om, log ग (Ka, 9u.,) - Ou.i) log(1 - x (Xu (u.:))]
D
log at (Nu,is Iu,i)
-lo((1 n (u,.)) (5)
D
(
(u,i)ED.
5

---

Propensity Loss Ly
Meta Loss meeta
Prediction Loss pred
Naive Imputation Loss Lm
e(tua) + (1 Fudd
Bas ml mm Fu.)
Oui
Euj
1f 0
Cxc
Fud
Our
Cus
TKKuu Fuj)
xxx
e(xui)
f(xu,)
m(Xup Iud
Oracle Imputation Model
Naive Imputation Model
Propensity Model
Residual Model
Prediction Model
O imput features Xy1
OOO
data type Ow
input features Xas
imput features Xyi
input features Xyi
imput features data type
Xay
fu.
Figure Architecture of MctaDebias to address selection bias in the presence of hidden confounding.
According the definition of m(#; g) in Equation adopt the square loss to train the naive imputation model m(x, Jú Pm) with parameters Φm. denoted as Lm (6m, 0) shown Equation 6
1
L. (mmmm 0)
(m (Inis Ouii eu,x)
D
(D)))D
Σ
(Ma,i Das) eu,)²
+
Σ
(m (Fa,)
(6)
D
D
(D
(a.i)ED.
where tu,i = L(s(sn,;; 0), ru,a) .) is the prediction error and only available when Ou,i =
Next, based Lemma we employ the square loss train the oracle imputation model e(2; de) with parameters de and residual model ny(:; on)) with parameters Φ, given learned propensity model 6n and imputation model @m. denoted Lmeta (Øes Pn Oms &x, shown in Equation 7 below:
Liera Dny Pns On, 0) - D
Σ (u,))D
fouss - Cu,i - m (Zu,in9a.))
-e((.,,)) + (1 - Iu.)) m((,,)) (ou, - n (Fuss: 9n,
Furthermore, we adopt the oracle imputation model e(æ; Φe) to generate prediction errors on the target population D as the training objective for prediction model f( 0) with parameters 0. The training objective (0r (,,00,0 (b) is demonstrated in the following Equation
(0,
Σ (a,i)ED
e(Fu,s3 &e (),
(8)
D
where e(xy,s: de () is the learned oracle imputation model training from Cmeta loss in Equation
7
3.4 Learning Algorithm
we propose a bi-level optimization based learning algorithm for model training. Specifically, first train the propensity model < g) independently minimizing the Lg loss shown Equation as training objective orthogonal to those of other models. Next, use bi-level optimization update the remaining models.
Following previous works 3 56
In the bi-level optimization framework, we first assumed update of the naive imputation model m(x, (x gi dm) and the oracle imputation model e(æ; )e). ensure that the gradient of (o 0) can

---

Algorithm 1: The Proposed MetaDebias Learning Algorithm
Input: observed ratings R
Pretrain propensity model x <(2,9. Φz)))
while stopping criteria is not satisfied do for number of training steps do
Sample batch of user-item pairs {(u,, [ from D;
Assumed update naive imputation model: 6m (0) - Pm am Van fm (ma(sm)) 10):
Assumed update oracle imputation model:
6.(6m, 0) = Φe - ae Vo Emtaa (e(δe) 6'm 0):
Update prediction model: 0 + a aa (fo2(0).)))
end
for number of training steps do
Sample batch of user-item pairs {(u, 2,) B2 from D;
10
Update naive imputation model: oin Dm Oim V om for (m((m)) 0): end
11
12
13
for number of training steps do
Sample batch of user-item pairs {(u,, d;)] from D; Update oracle imputation model: &e de ae Ve, Emeta (e(Φe) ) om O):
14
15
16
Update residual model: on + on an Cmeta (n(0,) pm. 8)
17
end
end
Output: A
be back propagated through the chain rule to update the prediction model parameter 0. Subseguently, the prediction model f( 0) is updated by minimizing (ered de) defined in Equation g
After updating the prediction model, adopt joint learning to update the naive imputation model m(x, g; Φm) by minimizing Lm in Equation Simultaneously, the oracle imputation model e(æ; Φ.) and residual model ()2;; (by) updated by minimizing Liaata in Equation The whole procedure of the learning algorithm is outlined in Algorithm [ where a represents the learning rate and we use distinct subscripts correspond to different models.
Experiments
4.1 Experimental Setup
Dataset and Experimental Details. Following previous studies 3 43 55 56
conduct extensive
experiments on three public datasets, COAT YAHOO! R and KUAIREC 11] COAT contains 6,960 biased ratings and 4,640 unbiased ratings derived from 290 users evaluating 300 items. YAНOO!! R3 contains 31 1,704 biased ratings and 54,000 unbiased ratings derived from 15,400 users evaluating 1,000 items. Both datasets employ five-point rating scale, and we binarize the ratings greater than as 1 and others as O. KUAIREC contains 4,676,570 video watching ratios derived from 1,411 users evaluating 3.327 videos. The ratios that greater than are binarized as 1, otherwise O. We adopt feature masking to simulate 9u.s which measures feature completeness, and Ja.: only when the features of both users and items are fully preserved. otherwise 9u,; O, and see Appendix A.3 for more details. Following previous works 3 the test set as validation set, and for all methods requiring RCT data, we employ observational data without hidden confounding to pretend RCT data. For evaluation, we employ three widely adopted metrics AUC, Recall K, and NDCG@ K to measure debiasing performance, where we set K = on COAT and YAНОO! R3, and K = 50 on KUAIREC. All the methods are implemented on PyTorch with Adam as the optimizer and NVIDIA A40 as the computing resource, and we tune learning rate in f0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05) and weight decay in [e 7, 10].
28 29
we randomly split 5% unbiased data from
phtps://www.s..om wow../..tttis/scchaaa/mmaaaaaaa
httpss//weesccopessnmddee sandbox, wwwtppp....ooooo.oooooooooooo https///gitubboomm//oognnnggg//

---

Table 1: Performance on AUC, Recall K and NDCG@ K on the COAT, YAHOO! R3 and KUAIREC datasets. The best result is bolded and the best results of both types of baseline methods are underlined, where * means statistically significant results (p-value I 0.05) using the paired-t-test.
COAT
YAHIO! R3
KUAIREC
Method
AUC
R 5
NDCG@5
AUC
Recall
NDCG@s
AUC
Recall 55
NDCG@50
Naive
0,698
0.478 10.0s
0.44
0.705
0.638 1o.0
0.489 10.00s
0.817
0,830 Lo000
IPS
2..17
2.083
0.446-0020
0.642 20.00
o1s
0.492
0.814_0.0mr
0.834_0000
0.548
DR
0.725+0.012
0.485
0.44820026
0.70940.004
0.6432.002
2.498
0.824-00000
0.836-20010
0.55020.010
Stable-DR
1.754
0,486
0.452.0m
1.015
1.65610.03
0.515 10.01s
0.826 .0.007
0,839 10.006
0,560
TDR
0.736.0.00s
0.492
0,458_00m
0.712
0.669
20.015
0.525 20.014
0.830 2.00
0.846 20000
ESMM
0.00020.014
0.48020.008
0.45220025
0.710
20.07s
2.51540.014
0.826
0.838
0.838=0.006
Multi-IPS
0.736.a..us
0,482
0,456_0011
2..09
2.64720.07
0.820 1..02
0,830 200006
0,548 c0.00
Multi-Do
10.00s
0.485_0.000
0.447_000
2.6091
0.50610.010
0.832 10.00
20000
g.562
ESCM--IPS
20.000
0.484200054
0,451
0.713
0.666
0.520
00.30
0.825
0.844_0.000
0,567 10.00s
BSCMPDDR
40.00
0,492-0.011
0.08_00006
4.670ss.0s
0.521
20.00s
0.85220.002
0.841
=00002
BRD-IPS
0.733
10.013
0.490_0.000 10.03
0.490
0.462
1.012
0.659
10.00s
0.515 10.015
0.833 10.000
0.846
00000
0.566 10.005
BRD-DR
0.739
0.494.0.015
0.464-0012
0.7 0.7
0.66220.015
0.316g0.01
0.834..
0.848
0.572
KD-Label
0.488 2.735
2,461
0.012
0.664 000
20.17
g0.00
200005
g0.66
AutolDehins
0.736-20.010
0.501
20:012
0.465=0000
0.210ga.0s
20.044
0.520.0.013
0.833=00000
0.843
20.022
0 2458+00000
LTD-IPS
0.752..0.ou
0,483
0,458_0m11
0. 20.087
0 2..60
0.514
0.834
0,847
0.572
LTD-DR
1.734
0.015
0,460_0010
goo0s
2..62
0.5161o.m15
g0.35_o.0m
0.848 000
0,570
Ral-IPS
0.733+0.010
0.486=20.18
0.46220011
400022
2666
0.51520.012
0.833=0.000
0.846-0.005
0.573
Bal-DR
0.735
0.490a0.011
0,464-0010
0. =0.08
0
0.517 26.8
0.834=0.00
0,847
0,568
Res-IPS
10.000
0.495 1000
_o
1.534 1o00ss
0.836 10.0en
0.850 _o00
1,578
Res-DR
0 0.740s
0,49820010
0,467=0000
0. 0.720
0.678
006
0.53840.005
0.838=0000
0.852
0.580
MetaDebias
0.74620.com
0.510 ooce
0.473 20010
0.722
0.688 go no5
0.544 20.c0s
0.840 20.00
0.857
000
g..03 2o.0c
MetaDabias
BRD-DR
MetaDabbias
BRD-DR
Metaaabias
BRD-DR
Res- DR
DR
Res-DR
DR
Res-DR
DR
0.59
0.86
NDCG0050
Recallaa50
0.57
0.83
0.56
0.83
0.1
0.3
0.5
0.7
0.9
0.54
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
Hidden Confounding Strength
Hidden Contounding ding Strength
Hidden Contounding Strength
AUC
(b) NDCG@ 50
(c) Recall @50
Figure 3: Effects of hidden confounding strength on the KUAIREC dataset.
Baselines. Two-layer multi-layer perceptron are used as the base model, and we compare proposed methods with both RCT data-free and RCT data-based methods.
RCT data-free Methods: Without RCT data, the baselines includes IPS-based methods: IPS [46].
ESMM 37 57]. Multi-IPS [64], ESCM2-IPS [52), BRD-IPS 100 and DR-based methods:
DR 55 Stable-DR (31) TDR (27. Multi-DR [64). ESCM2-DR (52). BRD-DR [10]
RCT data-based Methods: Based on RCT data, the baselines include model selection based
methods: KD-Label (34), AutoDebias (3), LTD-IPS LTD-DR (56), and model calibration
based methods: Bal-IPS 29. Bal-DR 29 Res-IPS 28). Re 28
4.2 Experimental Results
Performance Comparison. We compare our proposed MetaDebias with existing methods and the results are shown in Table DI First, all debiasing models outperform the base model which adopts naive empirical risk minimization, indicating the necessity for debiasing. Second, BRD- DR and Res-DR which are two representative methods for eliminating hidden confounding serve as the most competitive baselines, implying that ignoring hidden confounding leads to inevitable performance degradation. Third, compared to RCT data-free methods, RCT data-based methods do not demonstrate significant improvements across the three datasets, indicating that effective model selection or model calibration cannot be achieved using MNAR data without hidden confounding. Furthermore, MetaDebias exhibits significantly superior overall performance on all three datasets, which validates that our method achieves the performance in the presence of hidden confounding leveraging heterogeneous observational data without the utilization of RCT data.
8

---

DR
Res-DR
DR
Res
DR
Res
BRO-DR
0.59
MetaDebias
0.86
MetaDebias
0.58
NDCG@50
Recalli
0.85
0.57
0.56
0.55
0.820
3:1 2:1 3:2 1:1
2:3
1:2
1:3
0.54
3:1
2:1
3:2
1:1
2:3
1:2
1:3
3:1
2:1
3/2
1:1 2:3
1:3
#VIG 0) #NIG =1
#N
#NIG
#NGG
#NIG
(a) AUC
(b) NDCG@50
(c) Recall @550
Figure 4: Effects of varying proportions of heterogeneous observational data on the KUAIREC dataset.
Table 2: Effects of training dataset size on NDCG@ K on the KUAIREC and YAHOO! R3 datasets.
NDCG@50
Training Set Size (%)
NDCG@s
Training Set Size (%)
Method
10
20
40
60
80
100
Method
10
20
40
60
80
100
0.542
0.548
0.550
0.490
0.492
0.494
ESCM -DR
0.556
0.560
0.563
0,565
0.568
0.570
ESCM--DR
0,506
0.510
0.513
0.517
0.519
0.521
BRD-DR
0.567
0,570
0.571
0.572
BRD-DR
0,500
0.503
0.506
0.509
Bal-DR
0.552
0.554
0.558
0,562
0.565
0.568
Bal-DR
0,502
0.505
0.506
0.509
0.513
0.517
Res-DR
0.569
0.574
0.577
0.580
Res-DR
0,523
0.527
0.528
MetaDebias
0.576
0.577
0.578
0.579
0.581
0.584
MetaDehias
0.534
0.536
0.538
0.540
0.541
0.544
(a) NDCG@50 on the KUAIREC dataset
(b) NDCG@5 on the YAHOO! R3 dataset
In-depth Analysis. We conduct in-depth analysis to further explore the effect of hidden confounding strength, the proportions of heterogeneous observational data, and training data size on performance. Moreover, further explore the performance under conditions where RCT data is available, which is consistent with the problem setup in prior works. See Appendix [A.4 for more experimental results.
The Hidden Confounding Strength. Figure 3 shows the impact of hidden confounding strength, where we simulate higher hidden confounding strength by masking more features. We observe all methods exhibit performance degradation as the hidden confounding gets stronger, with vanilla DR exhibiting the poorest performance, highlighting the necessity of removing hidden confounding. The performance of BRD-DR deteriorates significantly, primarily due to the violation of imposed strong model assumptions, while Res-DR demonstrates competitive performance. In addition, the proposed MetaDebias stably outperforms the baselines across varying hidden confounding strength, implying our method is able to effectively address strong hidden confounding using only observational data.
The Proportion of Heterogeneous Observational Data. We explore the impact of varying propor- tions of heterogeneous observational data on the KUAIREC dataset in Figure where #N(G 0) and (N = 1) denote the quantity of training data with and without hidden confounding. First, we observe the performance of DR varies significantly under different proportions, indicating the pronounced disparities between the heterogeneous observational data. Besides, performances of all methods are enhanced when the proportion data without hidden confounding increases, this is because more feature information aids in accurate propensity and imputation model learning. Furthermore, MetaDebias demonstrates the best performance, indicating our method can achieve effective debiasing performance across various observational data proportions, and the potential for application in a range real-world scenarios.
The Training Dataset Size. Table explores the impact of training set size on NDCG@ K on both KUAIREC and YAHOO! R3 datasets. We find that the performance of all methods declines as the size of training set decreases, highlighting the importance of training data size on model training. Besides, proposed MetaDebias consistently outperforms
baselines across varying training data size especially when the size is extremely small such as only 10%, which indicates our method that explicitly models oracle prediction errors on the entire space fully exploits the heterogeneous observational data and is still effective with small training set size.

---

KD-Label
AutoDebias
Bal-DR
LTD-DR
Res-DR
Meta Debias
Coat
Yahoo! R3
KuaiRec
0.720
0.74
0.830
0.715
AUC
0.73
0.710
0.825
0.72
0.705
0.71
0.815
5
10
20
30
40
50
5
10
20
30
40
50
5
10
20
30
40
50
RCT Training Set Size (%)
Figure 5: Effects of varying RCT training set size on AUC on three benchmark datasets.
The Impact of RCT Training Data. In Figure 5 we explore the debiasing performance when RCT data is available, which is consistent with the problem setup in prior works (28 [9] Specifically, the training dataset is composed two parts: an observational dataset with hidden confounding and small RCT dataset. For implementing our proposed MetaDebias,
label the RCT sample as Gu.i = 1, as the RCT data can be regarded special case of MNAR data without hidden confounding. We find that the performances of all methods exhibit an increasing trend the size of RCT training set increases, which consistent with previous research findings. Moreover, the proposed MetaDebias stably outperforms all the baseline methods with varying RCT training set size, which indicates our method remains superior to the existing approaches under the previous problem setup, further validating the effectiveness of our method.
Conclusion
In this study, we investigate the problem of selection bias the presence hidden confounding. First, argue that existing methods are challenging to be applied in real-world scenarios, as they either rely on strong assumptions on hidden confounding strength or depend on the costly RCT data. To tackle this issue, propose to adopt heterogeneous observational datasets which are more likely to be collected to address hidden confounding and claim that such setup is more aligned with practical scenarios. Then, propose meta-learning based debiasing method called MetaDebias, which explicitly models the oracle error imputation and additional bias induced by hidden confounders, and we adopt bi-level optimization with assumed update for model training. Extensive experiments conducted on three public datasets validate our method achieves state-of- the-art performance, regardless of the availability RCT data. A limitation of this work lies in the assumption that all the confounders can be included through sufficient feature collection. Even though it is possible to collect hundreds of features industrial scenarios, and some features like consumption records may potentially represent hard-to-obtain features like salary, it is still challenging to guarantee that all confounders for partial samples have been included. In future work, will explore how to further relax this theoretically feasible assumption.
Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their valuable comments. This work supported by the National Natural Science Foundation China under Grant 623B2002, Grant 621 62106266 and Grant U22B2048.
References
[1] Matteo Bonvini and Edward Kennedy. Sensitivity analysis via the proportion of unmeasured confounding. Journal of the American Statistical Association, (541539)::540-5555 2022.
[2] Yin- Chang, Cho-Jui Hsieh, Kai-Wei Chang, Michael Ringgaard, and Chih-Jen Lin. Train- ing and testing low-degree polynomial data mappings via linear svm. Journal of Machine Learning Research, 11(4), 2010.
10

---

[3] Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. Autodebias: Learning to debias for recommendation, SIGIR, 2021.
[4] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 2022.
[5] David Cheng and Tianxi Cai. Adaptive combination of randomized and observational data. arXiv preprint arXiv:2111. 2021.
[6] H Christopher Frey and Sumeet R Patil. Identification and review of sensitivity analysis methods. Risk analysis, pages 553-578, 2002.
[7] Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Xiugiang He, Rui Zhang, and Jie Sun. A generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In KDD, 2022.
[8] Piersilvio De Bartolomeis, Javier Abad Martinez, Konstantin Donhauser, and Fanny Yang. Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. In AISTATS, 2024.
[9] Ilker Demirel, Edward De Brouwer, Zeshan M Hussain, Michael Oberst, Amthony A Philippakis, and David Sontag. Benchmarking observational studies with experimental data under right- censoring. In AISTATS, 2024.
[10] Sihao Ding, Peng Wu, Fuli Feng, Xiangnan He, Yitong Wang, Yong Liao, and Yongdong Zhang. Addressing unmeasured confounder for recommendation with sensitivity analysis. KDD, 2022.
[11] Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recommender systems. In CIKM, 2022.
[12) Siyuan Guo, Lixin Zou, Yiding Liu, Wenwen Ye, Suqi Cheng, Shuaiqiang Wang, Hechang Chen, Dawei Yin, and Yi Chang. Enhanced doubly robust learning for debiasing post-click conversion rate estimation. In SIGIR, 2021.
[13] Tobias Hatt, Jeroen Berrevocts, Alicia Curth, Stefan Feuerriegel, and Mihaela van der Schaar. Combining observational and randomized data for estimating heterogeneous treatment effects. arXiv preprint a2Xi:: /2891, 2022.
[14] Xiangnan He, Yang Zhang, Fuli Feng, Chonggang Song, Lingling Yi, Guohui Ling, and Yongdong Zhang. Addressing confounding feature issue for causal recommendation. ACM Transactions on Information Systems, 41(3):1-23, 2023.
[15] Miguel A. Hernán and James M. Robins. Causal Inference: What If. Boca Raton: Chapman and Hall/CRC, 2020.
[16] José Miguel -oanández-Lobatoo Neil Houlsby, and Zoubin Ghahramani. Probabilistic matrix factorization with non-random missing data. In JCML, 2014.
[17] Jin Huang, Harrie Oosterhuis, and Maarten de Rijke. It is different when items are older: Debiasing recommendations when selection bias and user preferences dynamic. In WSDM, 2022.
[18] Zeshan Hussain, Ming-Chieh Shih, Michael Oberst, Ilker Demirel, and David Sontag, Falsifica- tion of internal and external validity in observational studies via conditional moment restrictions. In AISTATS, 2023.
[19] Zeshan M Hussain, Michael Oberst, Ming-Chied Shih, and David Sontag. Falsification before extrapolation in causal effect estimation. In NeurIPS, 2022.
[20] Guido W Imbens. Potential outcome and directed acyelic graph approaches to causality: Relevance for empirical practice in economies. Journal of Economic Literature, pages 1129-79, 2020.
11

---

[21] Guido W. Imbens and Donald B. Rubin, Causal Inference For Statistics Social and Biomedical Science. Cambridge University Press, 2015.
[22] Dietmar Jannach, Lukas Lerche, and Markus Zanker. Recommending based on implicit feedback. In Social information access: systems and technologies, pages 510-569, Springer, 2018.
[23] Nathan Kallus and Angela Zhou. Confounding-robuss policy improvement. In NewrIPS, 2018. 2018.
[24] Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experi- mental grounding. In NewrlPS, 2018.
[25] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.
[26] Haoxuan Li, Quanyu Dai, Yuru Li, Yan Lyu, Zhenhua Dong. Xiao-Hua Zhou, and Peng Wu. Multiple robust learning for recommendation. In AAAI, 2023.
[27] Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. TDR-CL: Targeted doubly robust collaborative learning for debiased recommendations. In ICLR, 2023.
[28] Haoxuan Li, Kunhan Wu, Chunyuan Zheng, Yanghao Xiao, Hao Wang, Zhi Geng, Fuli Feng, Xiangnan He, and Peng Wu. Removing hidden confounding in recommendation: A unified multi-task learning approach. In NeurIPS, 2023.
[29] Haoxuan Li. Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confound- ing with few unbiased ratings in debiased recommendations. In WWW, 2023.
[30] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Measuring and enhancing balancing for recommendation. In ICML, 2023.
[31] Haoxuan Li, Chunyuan Zheng, and Peng Wu. StableDR: Stabilized doubly robust learning for recommendation on data missing not at random. In ICLR, 2023.
[32] Haoxuan Li, Chunyuan Zheng, Yanghao Xiao, Peng Wu, Zhi Geng, Xu Chen, and Peng Cui. Debiased collaborative filtering with kernel-based causal balancing. In ICLR, 2024.
[33] Mare Lipsitch, Eric Tchetgen Tchetgen, and Ted Cohen. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology, 2010.
[34] Dugang Liu, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. A general knowledge distillation framework for counterfactual recommendation via uniform data. In SIGIR, 2020.
[35] Dugang Liu, Pengxiang Cheng, Zinan Lin, Jinwei Luo, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. Kderec: Knowledge distillation for counterfactual recommendation via uniform data. IEEE Transactions on Knowledge and Data Engineering, 2022.
[36] Jinwei Luo, Dugang Liu, Weike Pan, and Zhong Ming. Unbiased recommendation model based on improved propensity score estimation. Journal of Computer Applications, 2021.
[37] Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In SIGIR, 2018
[38] Benjamin M Marlin and Richard S Zemel. Collaborative prediction and ranking with non- random missing data. In RecSys, 2009.
[39] Stephen L. Morgan and Christopher Winship. Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge University Press, second edition, 2015.
[40] Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, second edition, 2009.
[41] Paul R. Rosenbaum. Design of Observational Studies. Springer Nature Switzerland AG, second edition, 2020.
12

---

[42] Evan TR Rosenman, Guillaume Basse, Art B Owen, and Mike Baiocchi. Combining obser- vational and experimental datasets using shrinkage estimators. Biometrics, 79(4):2961-297; 2023.
[43] Yuta Saito. Doubly robust estimator for ranking metrics with post-click conversions. In RecSys, 2020.
[44] Yuta Saito and Masahiro Nomura. Towards resolving propensity contradiction in offline recommender learning. In IJCAI, 2022.
[45] Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing miising-not--t--ng -tt-aa- implicit feedback. In WSDM, 2020.
[46] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In ICML, 2016.
[47] Zihua Si, Xueran Han, Xiao Zhang, Jun Xu, Yue Yin, Yang Song, and Ji-Rong Wen. A model-agnostic causal learning framework for recommendation using search data, wWW, 2022.
[48] Harald Steck. Training and testing of recommender systems on data missing not at random. In KDD, 2010.
[49] Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. In NeurlPS, 2015.
[50] Eric Tchetgen Tchetgen. The control outcome calibration approach for causal inference with unobserved confounding. American journal of epidemiology, 179(5)x33--640, 2014.
[51] Victor Veitch and Anisha Zaveri. Sense and sensitivity analysis: Simple post-hoc analysis of bias due to unobserved confounding. In NeurlPS, 2020.
[52] Hao Wang, Tai- Wei Chang, Tianqiao Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. ESCM Entire space counterfactual multi-task model for post-click conversion rate estimation. In SIGIR, 2022.
[53] Haotian Wang, Wenjing Yang, Longqi Yang, Anpeng Wu, Liyang Xu, Jing Ren, Fei Wu, and Kun Kuang. Estimating individualized causal effect with confounded instruments. In KDD, 2022.
[54] Wenjie Wang, Yang Zhang, Haoxuan Li, Peng Wu, Fuli Feng, and Xiangnan He. Causal recommendation: Progresses and future directions. In Tutorial SIGIR, 2023.
[55] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recom- mendation data missing not at random. In ICML, 2019.
[56] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Combating selection biases in recom- mender systems with a few unbiased ratings. In WSDM, 2021.
[57] Hong Wen, Jing Zhang, Yuan Wang, Fuyu Lv, Wentian Bao, Quan Lin, and Keping Yang. Entire space multi-task modeling via post-click behavior decomposition for conversion rate prediction. In SIGIR, 2020.
[58] Lili Wu and Shu Yang. Integrative r-learner of heterogeneous treatment effects combining experimental and observational studies. In CLeaR, 2022.
[59] Peng Wu, Haoxuan Li. Yuhao Deng, Wenjie Hu, Quanyu Dai, Zhenhua Dong, Jie Sun, Rui Zhang. and Xiao-Hua Zhou On the opportunity of causal learning in recommendation systems: Foundation, estimation, prediction and challenges. In IJCAI, 2022.
[60] Mengyue Yang. Guohao Cai, Furui Liu, Jiarui Jin, Zhenhua Dong, Xiuqiang He, Jianye Hao, Weiqi Shao, Jun Wang, and Xu Chen. Debiased recommendation with user feature balancing. ACM Transactions Information Systems, 41(4):1-25, 2023.
13

---

[61] Shu Yang and Peng Ding. Combining multiple observational data sources to estimate causal effects. Journal of the American Statistical Association, 115:1540-1554 2020.
[62] Shu Yang, Donglin Zeng, and Xiaofei Wang. Improved inference for heterogeneeous treatment effects using real-world data subject to hidden confounding. hutps:/aaxiiiorr///////0......1 2020.
[63] Shu Yang, Chenyin Gao, Donglin Zeng, and Xiaofei Wang. Elastic integrative analysis of randomised trial and real-world data for treatment heterogeneity estimation. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(3):575-596, 2023.
[64] Wenhao Zhang, Wentian Bao, Xiao-Yang Liu, Keping Yang, Quan Lin, Hong Wen, and Ramin Ramezani. Large-scale causal approaches debiasing post-click conversion rate estimation with multi-task learning. In WWW, 2020.
[65] Xiang Zhang, Douglas E Faries, Hu Li, James D Stamey, and Guido W Imbens. Addressing unmeasured confounding in comparative observational research. Pharnacoepidemiology and drug safety, 27(4):373-382, 2018.
[66] Zhiheng Zhang, Quanyu Dai, Xu Chen, Zhenhua Dong, and Ruiming Tang. Robust causal inference for recommender system to overcome noisy confounders. In SIGIR, 2023.
[67] Hao Zou, Haotian Wang, Renzhe Xu, Bo Li, Jian Pei, Ye Jun Jian, and Peng Cui. Factual observation based heterogeneity learning for counterfactual prediction. In CLeaR, 2023.
14

---

A Appendix/ supplemental material
A.1 Related Work
Debiased Recommendation. Recommender systems play an important role in mitigating information overload which are trained on users' historical feedback, but selection bias inherent in the collected data impede the algorithms from accurately capturing users' true preferences [17 53 67]. If we naively adopt empirical risk minimization to train prediction model without debiasing, it will achieve sub-optimal prediction performance [46 55 60]. To address the selection bias and achieve unbiased
learning, the error imputation based (EIB) method [3, 48) treats selection bias missing data problem, imputes the missing data, and subsequently trains prediction model combining with the imputed data. The inverse propensity scoring (IPS) method [36 45 46] estimates the probability of an user-item pair being observed which is called propensity and performs inverse propensity weighting for the observed samples. The doubly robust (DR) method 39
43 55 combines both error imputation and inverse propensity weighting, achieving superior performance while exhibiting enhanced robustness. Owing the extensive adoption of IPS and DR, numerous variants have been proposed [26] 31 [4]. Specifically, some approaches focus on bias and variance trade-off 12, 227,
while others integrate with multi-task learning to address data sparsity [52
64 Additionally, some
works explore how to learn more appropriate propensity scores [3
However, the aforeeeentoned methods fail achieve unbiased when some features that simultaneous affect both user selection mechanism and the feedback remain unobserved, which is called hidden confounding.
30
56
Despite recent works proposed address observed confounding effects [4 477, tackling hidden confounding remains a challenging task. To fill this gap, sensitivity analysis based methods are proposed, where worst case optimization is used to mitigate the hidden confounding [O 66]. However, the validity of such methods relies on the assumption on hidden confounding effect that true propensities are near and can be bounded by the estimated values, and such approaches become ineffective when this assumption is violated. Motivated by this, recent works propose to leverage small amount of RCT data for model calibration, where the calibrated estimator serves an unbiased estimator of the ideal loss [28] [9]. Observing that the acquisition cost of RCT data is prohibitively high, it is equally challenging to implement such calibration based methods in real-world applications. In this paper, we propose more practical solution to address hidden confounding using heterogencous observational data for explicitly modeling the oracle prediction errors.
Causal Inference under Hidden Confounding, Hidden confounding is a widely studied topic in causal inference literature 1 50, 65]. and there are two primary approaches to addressing this issue. One approach involves leveraging additional information to achieve unbiased estimation of the target causal quantity, for example the instrumental variables [15,, front door adjustment (40), negative control [3 and data fusion 24 58). The second approach based on sensitive analysis, on the other hand, aims to estimate appropriate bounds for the target causal effects, rather than pursuing precise [1]. However, in practical applications. it is hard to identify suitable instruments or mediators that meet the required criteria [15] 20), while the sensitive analysis based method heavily relies on strong assumption of the hidden confounding effect, which is also hard to satisfy. The data fusion approach aims to combine data from observational studies and randomized trials, and can be broadly classified into three categories: statistical test based, correction based, and weighted based. Statistical test based methods introduce statistical tests to compare the causal effects estimated from observational studies and randomized trials, thereby detecting and mitigating hidden confounding [
point estimation 6 23
41
18 19. Correction based methods are proposed to correct the biased causal effect estimation derived from observational data using the unbiased RCT data [3
9
and an efficient integrative estimator is established based on semi-parametric theory [62 and further integrated with machine learning models (58). Weighted methods propose to train a biased estimator using observational data, train an unbiased estimator using RCT data, and take the weighted average of these two estimators as the final result [5] 42 61). A limitation of data fusion methods is the availability of RCT data, as the cost of obtaining RCT data is prohibitively high. Moreover, for the correction based method, the randomized trial and observational study should share the same support sets. When the support sets differ and only a partial overlap exists, additional strong parametric assumptions are required for extrapolation, for instance, the hidden confounding effect is assumed to be a linear function [24]. In contrast to the aforementioned methods, our proposed approach is more readily feasible in real- wollw scenarios, as it does not impose assumption on hidden confounding effect and the required heterogeneous observational data is relatively more accessible.
15

---

A.2 Proofs
Lemma The relationship between the proposed propensity score x x(x,)) 9). naive evvor imputation m(x, 9) and the oracle error imputation E [eu,s
| æu,i is as followed:
m(z, g) = {E [eu,i Tuss = x + (1 g)n(z)} T (x,
where n(x)
[eu, Fu,a = z. Bu.i O, Ou,i
1] EE [eu,s Fu,i z².
Proof. Recall that:
m(x 9) = E Elou, Gu,i Fu,s = I. Ia,i = g), π (x,9)) P (Ou,) Fa,i I. Gusi = 9)
Then, when 9 equals 1, we have:
m(x, g = 1) = [Ou,i Cu,i Tu,i = T. gu.i = =1]
E (eu,i i
Luti 2. Iu.i Ou,ii 1]] -P Pou.i Fu,i I. Iu.i 1)
[Cu,i Lu,i = v. Ju.i = 1] P Pou, = Fu,i = 2, Iu.i = 1)
E [Cu,i Zu,i z] (Ou.: = 1 Tu. 2, Iu.u. 1)
[eu,i Zu,i = x т (
The first equation the definition of mmxx.) g). the second equation the law of total probability, and the third equation holds as the conditional independence Cu,i IL Ou,i Zu,i Pa. = 1. The fourth equation holds because cu.i = (fr,ss ru,s) independent of the data type variable Ju and the last equation is the definition of क (2, g)-
Similarly, when g equals O, the equivalent equation holds:
mx., g 0) E [ou,i Cu,i Fu,i = Ju.i = 0]
[eu,i Tuzi = F. Ju.i 0, Ou,i = 11. (ou.i Tuni = IT = 0)
E [Cu,i Tuzi 2. Gu.i 0, Ou,i = =1] T(I, 0)
{E [Ca,e Tuus = x²) (x)) (x, 0).
The second equation the law of total probability, the third equation is the definition of <<. g), and the last equation is the definition of v)(3)) which is equivalently expressed as:
E
Jea,s
Fu,i = v. Ju. = 0, Ou,i 1] = 17(x)) + EE [eu, Fa,s x²]
To sum up, the equation holds:
m(x.g) = {E (ew, Fu.j x] + (1 - (1-1))(()) (x.
Lemma 2 The relationship between the propensity score π x(I (I),g), naive error imputation m(f, g), observation Ou,i, naive prediction error Ou,i Gu, and oracle error imputation IE [eu, Faael is:
Ou Cu,i m(x 9) {E (ew, Fu,s x]] (1 g)m(+}} {oo,s (x, & where & Ou, {Eu,t [E[euu, Fu,i æ] + (1 9)m(+)}} with EKg x.9] = 0.
Proof. We prove this Lemma in two steps. Initially, we establish the equality relation in the Lemma Ou,i" Cu,i m(x 9) {E [eu.t Fu,s x]] + (1 g)n(2)} fou,, x(2, g)} &. Subsequently, prove that the conditional expectation of the noise term is zero i... E[g x. gl =
To start with, based on Lemma 1, the equation holds:
m(x, (r. g) = { [eu,i Fu,s = x] + (1 (1-9)(()) x(x.g).
According to the definition of have:
Ou,i Cu,i = E Ou,i {[Elens Lu,i = 2] + (1 -0)(())))
16

---

MetaDebias
BRD-DR
MetaDebias
BRD-DR
MetaDebias
BRD-DR
Res DR
DR
Res-OR
DR
Res-OR
DR
0.73
0.54
NDCG85
Recalles
0.52
3.66
0.71
0.70
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
Hidden Confounding Strength
Hidden Confounding Strength
Hidden Confounding Strength
(a) AUC
(b) NDCG@5
(c) Recall @
Figure 6: Effects of hidden confounding strength on the YAHOO! R3 dataset.
The subtraction of the above two equations yields the following:
Ou Cu,i - m(x 9) { [eu, Tu, = xi (1 9)m(x)} fou,s - x(x,9)} g.
Next we demonstrate that & possesses the zero mean property. Recall that:
= Ou.i {eu,i - (Elea, Luti = = (1 g)m()}}
Ou,i Ca,i Ou,i {[Elle,, Luui = (1 (1=9)n(=)3)
Then the conditional expectation of g is: E L
x: g] = E [Ou,s Cu.i x. g] - EF [ou.i [e Tu,i = x] + (1 - g)n(x))) gl
m(x,9) g) E (o..i (eu,i Fu.i = 2| + +1 g)n(1>} 1
m(x.9) g) { [eu,s Fugi = x] + (1 (1))((=)) E Jou,s Duut = x, gu,i = gl
m(r, g) E (eu, Fu,t x] + (1 -9)m(=)) P POo.. Fu,i Ir, = g)
= m(x.9) g) { Je,,. Dy,y = z] + (1 - 9)m(2)} ((x.g)
0.
The first equation is the definition of ç, the second equation is the the definition of mxx g), and the third equation holds because E [eu,i Fu. æ| + (1 ())(2;) is constant given and g. The fourth equation arises from the fact that the treatment variable ou, is a binary variable taking values of 0 or 1, and the fifth equation is the definition of л (ar, g)- The last equation holds based on Lemma 1.
A.3 More Experimental Details
We introduce more experimental details about feature preparations here. In particular, we first employ Matrix Factorization (MF) (25 to transform the ID>based data format into the feature-basee format and regard the derived features as complete features. Next, we simulate the hidden confounding effect induced by insufficient feature collection process through feature masking. The condition Ja.i satisfied only when both user and item features are sufficiently preserved, i.e., there are no masked values their respective feature vectors; otherwise, it equals O. At the outset, we set the proportion of two types of heterogeneous observational data at 1:1, and the impact of varying proportions is explored in the following in-depth analysis.
A.4 More Experimental Results
In this subsection, we will present more experimental results that could not be accommodated within the main text due space limitations. The main text primarily elucidates the results obtained on the KUAIREC dataset. Supplementary to this, the appendix predominantly supplements experimental findings on the YAHOO! R3 dataset, which similarly explores the impact of hidden confounding strength, the proportions of the heterogeneous observational data, and the data sparsity. Apart from these, we conduct an additional investigation into the algorithm runtime on three datasets to explore the computational resource demands of different methods. The findings are follows.
17

---

DR
Res-DR
DR
Res
DR
Res
MetaDebias
MetaDebias
0.725
0.70
0.54
0.720
NDCG@S
Recall@s
0.68
AUC
0.715
0.52
0.66
0.710
0.50
0.64
3:1 2:1 3:2 1:1
2:3
1:2
1:3
3:1
2:1
3:2
1:1
2:3
1:2
1:3
3:1
2:1 3/2
1:1
2:3
1:
#NIG= 0) #NIG
#N(G=
0)
#NIG
#NGG = #NIG=11
(a) AUC
(b) NDCG@5
(c) Recall
Figure 7: Effects of varying proportions heterogeneous data on the YAHOO! R3 dataset.
Table 3: Effects of training data size on AUC on the KUAIREC and YAHOO! R3 datasets.
AUC
Training Set Size (%)
AUC
Training Set Size (%)
Method
10
60
80
100
Method
20
40
60
80
100
DR
0.812
0.815
0.818
0.820
0.821
0.824
DR
0,700
0.702
0.704
0.705
0.707
0.709
ESCM-DDR
0.816
0.819
0.827
0.830
0.832
ESCM
0.705
0.708
0.710
0,711
0.713
0.715
BRD-DR
0.827
0.830
0.830
0.832
0.833
0.834
BRD-DR
0.704
0.706
0.708
0.710
0.712
0.714
Bal-DR
0.825
0.827
0.828
0.829
0.830
0.834
Bal-DR
0.702
0.704
0.705
0,705
0.706
0.708
Res-DR
0.830
0.832
0.834
0.836
0.837
0.838
Rec-DR
0,712
0.714
0.714
0.716
0.718
0.720
MetaDebias
0.839
0.840
0.721
0.722
0.721
0.722
(a) AUC on the KUAIREC dataset
(b) AUC on the YAHOO! R3 dataset
The Hidden Confounding Strength. digontaede the debiasing performance with varying hidden confounding strength on the YAHOO! R3 dataset. Similar to the findings observed in the KUAIREC dataset shown in the main text, on the YAHOO! R3 dataset, an increase in hidden confounding strength leads to decrease in performance across all methods. Mcanwhile, the performance deterioration of the BRD-DR more pronounced, particularly in terms AUC metric, attributed to the violation of the required assumptions. Furthermore, the proposed MetaDebias demonstrates superior performance across all three evaluation metrics, and its performance shows minimal degradation the hidden confounding strength increases. This indicates the superiority of our approach over existing methods, and robustness to the hidden confounding strength.
The Proportion of Heterogeneous Observational Data. Figure 7 shows the prediction performance with varying proportions of heterogeneous observational data on the YAHOO! R3 dataset, where varying data proportions simulate potential data collection scenarios under different real-world conditions. From the figure, we observe an upward trend in the performance of all methods with an increase in the proportion of data without hidden confounding, indicating that the selection bias inherent in such observational data without hidden confounding is more amenable to be addressed. Moreover, MetaDebias outperforms other baselines and exhibits insensitivity to data proportions, which underscores that our method is capable of handling hidden confounding in various possible data combinations and can be broadly applied
wide range of real-world scenarios.
The Training Dataset Size. We explore the impact of training set size on prediction performance in Tables and which report the AUC and Recall @ K with varying training set size on both KUAIREC and YAHOO! R3 dataset. The presented data represents the mean values obtained from 10 random replicate experiments. We find that MetaDebias demonstrates superior performance across varying training data size, highlighting the efficacy of our method. Meanwhile, MetaDebias exhibits relatively minor fluctuations with changes in dataset volume, particularly evident in the YAHOO! R3 dataset, where the AUC of our method consistently maintains superior performance even amidst substantial variations in training data volume, such as change from 100% to 10%. This further demonstrates the stability of our approach even with small the training data size.
The Computational Resource Demands. We investigate the algorithm training and inference time on three datasets in Table 5 Despite the involvement of five models in the training process, the comparison with other baseline methods reveals that the training time of the proposed approach is
18

---

Table 4: Effects of training dataset size on Recall K on the KUAIREC and YAHOO! R3 datasets.
50
Training Set Size
Recall@
Training Set Size (%)
Method
10
20
40
60
80
100
Method
10
20
40
60
80
100
DR
0.824
0.826
0.828
0.829
0.833
0.836
0.634
0.635
0.636
0.637
0.640
0.643
ESCM-DR
0.829
0.832
0.835
0.837
0.839
0.841
ESCMP-DR
U.652
0.636
0.660
0.663
0.666
0.670
BRD DR
0.843
0.844
0.845
0.845
0.847
0.848
BRD DR
0.648
0.653
0.656
0.658
0.660
0.663
Bal-DR
0.838
0.839
0.840
0.842
0.844
0.847
Bal-DR
0.652
0.655
0.657
0.660
0.665
0.668
Res DR
0.844
0.845
0.846
0.846
0.849
0.852
Res DR
0.664
0.668
0.670
0.673
0.675
0.678
MetaDebias
0.846
0.847
0.849
0.851
0.854
0.857
MetaDebias
0.673
0.674
0.677
0.681
0.683
0.688
(a) Recall@50 50 the KUAIREC dataset
(b) Recall@5 on the YAHOO! R3 dataset
Table Comparison of training time (minutes) and inference time (milliseconds per sample) on the COAT, YAHOO! R3 and KUAIREC datasets.
COAT
YAHOO! R3
KUAIREC
Method
Training
Inference
Training
Inference
Training
Inference
DR
0.793
0.498
6.599
0.258
219.948
0.274
TDR
0.693
0.472
6.227
0.265
135.778
0.285
Multi-DR
0.552
0.515
5,906
0.283
238.706
0.265
ESCM²-DR
0,462
0.578
1.098
0.262
104.668
0.782
BRD-DR
0,682
0.347
2.973
0.261
198.872
0.249
KD-Label
1.124
0.519
2.449
0.358
163.358
0.239
AutoDebias
1.182
0.395
1,662
0.243
204.555
0.632
LTD-DR
2.351
0.459
1.832
0.262
253.277
0.267
Bal-DR
2,912
0.513
1.702
0.236
157.784
0.257
Res-DR
0.674
0.387
3.927
0,288
165.444
0,244
MetaDebias
1.916
0.521
6.375
0.263
174.866
0.254
acceptable, particularly on the large-scale dataset KuaiRec. A potential reason for the relatively long training time required by proposed method is that the bi-level optimization process with assumed updates results multiple gradient computations throughout the training procedure.
A.5 Boarder Impacts
Recommender systems (RS) serve as an effective tool for mitigating information overload, yielding significant economic benefits by accurately recommending items of interest to individual user. However, in the data collection process of RS, there inevitably exists significant amount of bias, rendering prediction models trained on the collected historical feedback incapable of capturing users' true preferences, where selection bias is particularly common and extensively studied issue. Recently, an increasing number of studies have further focused on and emphasized the impact of hidden confounding, which aims to achieve better prediction performance. In this paper, we propose more practical solution to effectively address selection bias in the presence hidden confounding. Specifically, we propose to leverage heterogeneous observational datasets, which is more readily available in real-world scenarios, while not imposing additional assumptions or requiring RCT data. This indicates that our method has the potential to be applied in real-world recommendation scenarios, offering opportunities to enhance economic benefits for businesses.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction include the claims made in the paper, and the contributions are summarized in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the paper in the conclusion in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

The paper provides the full set of assumptions and complete proofs for the theoretical results in Appendix AA.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results of the paper in the Dataset and Experimental Details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The paper provides open access to data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ prijps.cc/pbbliccguu.eee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test information including datasets, preprocessing, experimental protocols and details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports the experimental results accompanied by error bars and conducts statistical significance test using the paired-t-test in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper provides sufficient information on the computer resources (type of compute workers, memory) in Section 4
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://eeurpps http://uuuripsscccppuplllluteeet httpp//c.....p........cccuuaada.
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts of the work in Appendix A.5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creator or original owner of the assets (e.g., code, data, models) used in the paper is properly credited, and the license and terms of use are explicitly mentioned and appropriately respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "6xu1Vj9sFe", "reviewer": "Reviewer_TTAY", "summary": "This paper proposes to use heterogeneous observational data to address hidden confounding in recommender system.", "questions": "Please refer to my summary of weakness.", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Addressing selection bias in recommender system is very important.\n+ If the assumption holds, i.e., confounder missing mechanism follows the user attribute missing mechanism, I would say the method makes sense to me, though it is complicated.\n+ Experiments seem to demonstrate the effectiveness of the proposed method.", "weaknesses": "- It seems to me that using missing feature mechanism to estimate the missing confounder mechanism is a over simplification of the problem, where the author provide no strong empirical evidence to demonstrate the reliability of this simplification.\n- If confounders are just missing features, why don't we just infer them from the data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. Below, we hope to address your concerns and questions to improve the clarity and readability of our paper.\n\n> **[W1] It seems to me that using missing feature mechanism to estimate the missing confounder mechanism is a over simplification of the problem, where the author provide no strong empirical evidence to demonstrate the reliability of this simplification.**\n\n**Response:** Thank you for the comments. Below, we will demonstrate why hidden confounders are equivalent to missing features within the potential outcome framework, and introduce the motivation of our work.\n- In fact, we follow previous works and adopt the potential outcome framework to formalize the debiasing problem in the presence of hidden confounding. Specifically, we define $X$ as the features of user-item pair, such as user gender and item color, which influence both click and purchase behaviors. Here, click $T$ and purchase $Y$ are respectively defined as the treatment and outcome. \n- Notably, previous works rely on the unconfoundedness assumption (also known as ignorability), i.e., $T \\perp (Y_{0}, Y_{1}) \\mid X$, where $Y_{0}$ and $Y_{1}$ are potential outcomes. In causal inference, this indicates that given the observed features, the purchase outcome if the user clicks is independent of the click behavior itself. In recommendations, this indicates the observed features are sufficient to model both click and purchase behaviors.\n- However, the assumption may sometimes be violated. For example, user income is the feature that may simultaneously influence both clicks and purchases, but if it is missing, it leads to a violation of the unconfoundedness assumption. Therefore, hidden confounders are equivalent to missing features. In fact, it is a widely adopted setup in many previous studies.\n- Note that, when only observational data with hidden confounding is available, even with strong assumptions, it remains difficult to eliminate the hidden confounding bias. This illustrates why recent works propose incorporating RCT data for calibration. However, the cost of acquiring RCT data is exceptionally high due to the requirement for random assignment of treatment. In recommendation scenarios, this requires users to randomly click and rate items. This motivates us to utilize observational data with fully observed features rather than RCT data to address hidden confounding.\n\n> **[W2] If confounders are just missing features, why don't we just infer them from the data?**\n\n**Response:** Thank you for the question. Below, we will discuss the feasibility of feature imputation methods from both theoretical and experimental perspectives. In this study, we categorize the data into two groups, $g=1$ and $g=0$, based on whether they satisfy the unconfoundedness assumption. In fact, the samples with $g=1$ can be further categorized into two types. \n- As shown in the motivation graph in the paper, the first case where $g=1$ corresponds to complete feature collection. With fully observed features, it is possible to estimate the joint distribution of all features, and allows the missing feature inference as the reviewer noted. However, there may exist selection bias, meaning that the features are missing not at random, which hinders the accurate missing feature imputation. \n- Below are the results of three different feature imputation methods on three benchmark datasets. The Sample-Imp method imputes missing values by sampling from a Gaussian distribution, where the mean and variance of the distribution are estimated using the features with $g=1$. The Naive-Imp method aims to learn a model for imputing missing features from observed features by training on the naive loss on the samples with $g=1$, while IPW-Imp further incorporates propensity scores to account for selection bias.\n\n|NDCG@K|Coat|Yahoo!R3|KuaiRec|\n|:--:|:--:|:--:|:--:|\n|Naive|0.444±0.014|0.489±0.009|0.540±0.009|\n|Sample-Imp|0.449±0.012|0.498±0.012|0.545±0.006|\n|Naive-Imp|0.458±0.013|0.496±0.013|0.556±0.008|\n|IPW-Imp|0.462±0.011|0.500±0.009|0.563±0.006|\n|MetaDebias|**0.473*±0.010**|**0.544*±0.005**|**0.584*±0.003**|\n||\n\nThe results show that the feature imputation methods achieve performance improvements compared to the naive approach, but the proposed MetaDebias method still significantly outperforms all baselines. Moreover, IPW-Imp outperforms Naive-Imp, indicating that the features are missing not at random and imputing missing values remains a challenge.\n\n- RCT data satisfy the independence condition of unconfoundedness assumption, and thus is also labeled as $g=1$. Such data is obtained through the random assignment of treatment, and does not require the collection of complete features. In this case where complete features are absent as inferring labels, feature imputation methods often struggle to achieve good performance. \n- Below, we compare the feature imputation method with proposed MetaDebias, where samples with $g=1$ are all derived from randomized trials, and the Random-Imp method employs random values from [-0.5, 0.5] to impute missing feature values.\n\n|NDCG@K|Coat|Yahoo!R3|KuaiRec|\n|:--:|:--:|:--:|:--:|\n|Naive|0.450±0.012|0.498±0.015|0.545±0.006|\n|Random-Imp|0.452±0.013|0.492±0.013|0.542±0.006|\n|MetaDebias|**0.468*±0.012** |**0.536*±0.008**|**0.581*±0.003**|\n||\n\nExperimental results indicate that when samples with $g=1$ are from randomized trials, the performance of feature imputation method is even inferior than the naive approach that does not involve imputation, and significantly worse than the proposed MetaDebias. In this case, feature imputation fails.\n\n***\n**We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score.** We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BFFvrgi4VI", "reviewer": "Reviewer_X846", "summary": "The paper addresses the issue of selection bias in recommender systems, particularly when hidden confounding factors are present. The authors propose a new approach using heterogeneous observational data, where some data is affected by hidden confounding and some is not. The proposed MetaDebias is a meta-learning based debiasing method that models oracle prediction errors and bias from hidden confounders using bi-level optimization for training. Experiments on three public datasets demonstrate that MetaDebias achieves the best performance despite the presence of hidden confounding.", "questions": "How does the proposed MetaDebias algorithm perform in terms of time and space efficiency compared to the baseline?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Addressing hidden confounders in recommender systems is a significant issue.\n\nThe approach is quite novel.\n\nThe experimental evaluation is well done, with a clear explanation of most aspects of the experimental evaluation.\n\nThe related work is broadly covered and compared to.", "weaknesses": "In the comparative experiments shown in Table 1, the improvement of the MetaDebias algorithm over other algorithms is relatively small.\n\nAlthough the paper is well-organized, the presentation quality needs improvement. For example, some abbreviations should be spelled out when first mentioned, such as Random Controlled Trial (RCT) in the abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your approval of the idea and the novelty of this work, and thank you for the helpful suggestions. Below, we hope to address your concerns and questions to improve the clarity and readability of our paper.\n\n> **[W1] In the comparative experiments shown in Table 1, the improvement of the MetaDebias algorithm over other algorithms is relatively small.**\n\n**Response:** Thank you for the comments. In fact, we carefully tune the parameters of all baselines, so that some baseline results seem competitive. Even though, MetaDebias significantly outperforms these baselines under p-value less than 0.05. To further address your concern, we add experiments on PCIC and Ali-CCP datasets to validate the effectiveness of our method. \n\n- PCIC dataset contains 19420 biased ratings and 2040 unbiased ratings derived from 1000 users evaluating 1720 movies. Ali-CCP is a large-scale public dataset for click and conversion prediction in industrial scenarios, where the training, validation, and test data consist of 38 million, 4.2 million, and 43 million records, respectively. \n- The AUC performance on both datasets are shown below.\n\n|AUC Metric|PCIC|Ali-CCP|\n|:--:|:--:|:--:|\n|Naive|0.694±0.005|0.590±0.013|\n|IPS|0.696±0.004|0.602±0.004|\n|DR|0.701±0.003|0.608±0.005|\n|ESMM|0.695±0.005|0.592±0.006|\n|ESCM2-IPS|0.705±0.003|0.608±0.009|\n|ESCM2-DR|0.706±0.004|0.611±0.008|\n|Res-IPS|0.706±0.003|0.611±0.007|\n|Res-DR|0.709±0.004|0.614±0.005|\n|MetaDebias|**0.715*±0.005**|**0.626*±0.006**|\n||\n\nThe results show that the proposed MetaDebias stably outperforms the baseline methods across both datasets, and achieves a significant performance improvement on the large-scale industrial dataset Ali-CCP. Please refer to the attached PDF for more results.\n\n> **[W2] Although the paper is well-organized, the presentation quality needs improvement. For example, some abbreviations should be spelled out when first mentioned, such as Random Controlled Trial (RCT) in the abstract.**\n\n**Response:** We thank the reviewer for pointing out this issue and apologize for the lack of clarity. We will carefully polish the revised manuscript to improve readability.\n\n> **[Q1] How does the proposed MetaDebias algorithm perform in terms of time and space efficiency compared to the baseline?**\n\n**Response:** Thanks for the question. As suggested by the reviewer, we compare the parameter size, training time (minutes) and inference time (milliseconds per sample) of different methods on the Coat, Yahoo! R3 and KuaiRec datasets. The results are shown below.\n\n|KuaiRec Dataset|Parameters|Training|Inference|\n|:--:|:--:|:--:|:--:|\n|Naive|$1 \\times$|91.093|0.258|\n|DR|$3 \\times$|219.948|0.274|\n|TDR|$3 \\times$|135.778|0.285|\n|Multi-DR|$3 \\times$|238.706|0.265|\n|ESCM2-DR|$3 \\times$|104.668|0.782|\n|BRD-DR|$5 \\times$|198.872|0.249|\n|KD-Label|$2 \\times$|163.358|0.239|\n|Autodebias|$4 \\times$|204.555|0.632|\n|LTD-DR|$3 \\times$| 253.277|0.267|\n|Bal-DR|$5 \\times$|157.784|0.257| \n|Res-DR|$5 \\times$|165.444|0.244| \n|MetaDebias|$5 \\times$|174.866|0.254| \n|| \n\n|Yahoo!R3 Dataset|Parameters|Training|Inference|\n|:--:|:--:|:--:|:--:|\n|Naive|$1 \\times$|0.574|0.230|\n|DR|$3 \\times$|6.599|0.258|\n|TDR|$3 \\times$|6.227|0.265|\n|Multi-DR|$3 \\times$|5.906|0.283|\n|ESCM2-DR|$3 \\times$|1.098|0.262|\n|BRD-DR|$5 \\times$|2.973|0.261|\n|KD-Label|$2 \\times$|2.449|0.358|\n|Autodebias|$4 \\times$|1.662|0.243|\n|LTD-DR|$3 \\times$| 1.832|0.262|\n|Bal-DR|$5 \\times$|1.702|0.236| \n|Res-DR|$5 \\times$|3.927|0.288| \n|MetaDebias|$5 \\times$|6.375|0.263| \n|| \n\n|Coat Dataset|Parameters|Training|Inference|\n|:--:|:--:|:--:|:--:|\n|Naive|$1 \\times$|0.206|0.250|\n|DR|$3 \\times$|0.793|0.498|\n|TDR|$3 \\times$|0.693|0.472|\n|Multi-DR|$3 \\times$|0.552|0.515|\n|ESCM2-DR|$3 \\times$|0.462|0.578|\n|BRD-DR|$5 \\times$|0.682|0.347|\n|KD-Label|$2 \\times$|1.124|0.519|\n|Autodebias|$4 \\times$|1.182|0.395|\n|LTD-DR|$3 \\times$| 2.351|0.459|\n|Bal-DR|$5 \\times$|2.912|0.513| \n|Res-DR|$5 \\times$|0.674|0.387| \n|MetaDebias|$5 \\times$|1.916|0.521| \n|| \n\n- **Space efficiency:** For all methods, we employ a multi-layer perceptron network to model the prediction model, while the same architecture is also used for the propensity and imputation model. As shown in tables above, the Naive method employs only a single prediction model to fit the training data, with the parameter size denoted as $1 \\times$, while the Doubly Robust (DR) method further incorporates both propensity and imputation model to achieve double robustness, with the parameter size denoted as $3 \\times$. The parameter size of the proposed MetaDebias method is comparable to that of some existing methods such as Res-DR, indicating that the proposed method outperforms the competitive baselines under the same parameter size.\n\n- **Time efficiency:** Despite the involvement of five models in the training process, the comparison with other baseline methods reveals that the training time of the proposed approach is acceptable, particularly on the large-scale dataset KuaiRec. A potential reason for the relatively long training time required by proposed method is that the bi-level optimization process with assumed updates results in multiple gradient computations throughout the training procedure.\n\n- In summary, the computational resource demands of proposed method are acceptable. The experimental results can also be found in the one-page attached PDF.\n\n***\n**We sincerely thank you for your feedback and will provide more clarifications and explanations in the revised version, and welcome any further technical advice or questions on this work and we will make our best to address your concerns.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6TNDkI5e7", "reviewer": "Reviewer_Yf6Q", "summary": "The paper studies the problem of selection bias due to hidden confounding in recommendation systems. Previous methods struggle with real-world application due to reliance on strong assumptions or unavailable RCT data. The proposed solution, MetaDebias, leverages heterogeneous observational data, which is more practical and readily available. The approach involves a meta-learning framework that uses both confounded observational data and unconfounded observational data for model training. Experiments show that MetaDebias consistently outperforms baseline methods across various metrics and conditions.", "questions": "(Q1) Could you explain with more details how are you assessing the statistical significance of the performance improvement over the previous methods?", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(S1) The paper compares the proposed approach against many existing methods on 3 benchmark recommendation datasets under several commonly used evaluation metrics.", "weaknesses": "(W1) I found the use of the term heterogenous to describe the observational data a bit vague and misleading. I think the authors should make very clear from the beginning that they are working with one dataset where they can observe everything (no latents) and one dataset where some variables are unobserved. \n\n(W2) I think the related work section on causal inference (Appendix A.1) could be improved. In particular, it is not clear what is the weakness of the data fusion methods since they are clustered with IVs and negative controls. Moreover, I think some relevant works that use randomized trials to address hidden confounding are missing, for example [1] and [2]. \n\n(W3) The (empirical) performance improvement over previous methods is not so strong.\n\n(Minor) Typo in Lemma 1, should be \"as follows\" not \"as followed\". \n\n[1] Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. De Bartolomeis et al. AISTATS '24\n\n\n[2] Falsification before Extrapolation in Causal Effect Estimation. Hussain et al. NeurIPS '22", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s great efforts and insightful comments to improve our manuscript. Below, we hope to address your concerns and questions to improve the clarity and readability of our paper.\n\n> **[W1] I found the use of the term heterogeneous to describe the observational data a bit vague and misleading.**\n\n**Response:** We thank the reviewer for pointing out this issue. In fact, we use the term \"heterogeneous\" to distinguish whether the data meets the unconfoundedness assumption, i.e., $o \\perp (Y_{0}, Y_{1}) \\mid x$. Samples with fully observed features or from randomized controlled trial (RCT) meet such independence assumption and are denoted as $g=1$, while samples with missing features that do not satisfy the assumption are denoted as $g=0$. As suggested by the reviewer, we will clearly specify the dataset information and above difference in the revised manuscript's introduction. \n\n> **[W2] I think the related work section on causal inference (Appendix A.1) could be improved.**\n\n**Response:** We thank the reviewer for pointing out this issue and apologize for the lack of clarity. As suggested by the reviewer, below we will introduce more relevant works and discuss their limitations. \n- **Statistical test based methods** introduce statistical tests to compare the causal effects estimated from observational studies and randomized trials, thereby detecting and mitigating hidden confounding [1, 2, 3, 4]. **Correction based methods** are proposed to correct the biased causal effect estimation derived from observational data using the unbiased RCT data [5, 6, 7], and an efficient integrative estimator is established based on semi-parametric theory [8], and further integrated with machine learning models [9]. **Weighted methods** propose to train a biased estimator using observational data, train an unbiased estimator using RCT data, and take the weighted average of these two estimators as the final result [10, 11, 12].\n- A **limitation** of data fusion methods is the availability of RCT data, as the cost of obtaining RCT data is prohibitively high. Moreover, for the correction based method, the randomized trial and observational study should share the same support sets. When the support sets differ and only a partial overlap exists, additional strong parametric assumptions are required for extrapolation, for instance, the hidden confounding effect is assumed to be a linear function [5].\n\n> **[W3] The performance improvement over previous methods is not so strong.**\n\n**Response:** Thank you for the comments. In fact, we carefully tune the parameters of all baselines, so that some baseline results seem competitive. Even though, MetaDebias significantly outperforms these baselines under p-value less than 0.05. To further address your concern, we add experiments on PCIC and Ali-CCP datasets. Due to the space limit, some results are shown below, and please refer to the PDF for more details.\n\n|AUC Metric|PCIC|Ali-CCP|\n|:--:|:--:|:--:|\n|Naive|0.694±0.005|0.590±0.013|\n|DR|0.701±0.003|0.608±0.005|\n|ESCM2-DR|0.706±0.004|0.611±0.008|\n|Res-DR|0.709±0.004|0.614±0.005|\n|MetaDebias|**0.715*±0.005**|**0.626*±0.006**|\n||\n\nThe results of the statistical significance test indicate that MetaDebias significantly outperforms the baselines.\n\n> **[Minor] Typo in Lemma 1.**\n\n**Response:** We thank the reviewer for pointing out this issue, and we will carefully polish the revised paper to avoid typos and improve readability. \n\n> **[Q1] Could you explain with more details how are you assessing the statistical significance of the performance improvement over the previous methods?**\n\n**Response:** Thanks for the question. Actually, we use a paired t-test to examine whether the proposed method significantly outperform the **optimal** baseline under different metrics across various datasets. The specific testing procedure is as follows:\n- First, we conduct independent repeated experiments to calculate the mean and variance of all methods on the specified metrics. \n- Next, the t-statistic is calculated using the mean and variance from both the proposed MetaDebias and the optimal baseline results. \n- Finally, we identify the critical value based on the pre-specified degrees of freedom and significance level, and compare it with the absolute value of the t-statistic. If the absolute value of t-statistic exceeds the critical value, the difference is considered significant. \n\nFor the empirical implementation, we use the ‘stats.ttest_rel’ function from the SciPy library in Python.\n\n***\n**We sincerely thank you for your feedback and will provide more clarifications and explanations in the revised version, and welcome any further technical advice or questions on this work and we will make our best to address your concerns.**\n\n***\n**References**\n\n[1] Hidden yet quantifiable: A lower bound for confounding strength using randomized trials\n\n[2] Falsification before Extrapolation in Causal Effect Estimation\n\n[3] Falsification of internal and external validity in observational studies via conditional moment restrictions\n\n[4] Benchmarking Observational Studies with Experimental Data under Right-Censoring\n\n[5] Removing hidden confounding by experimental grounding\n\n[6] Combining observational and randomized data for estimating heterogeneous treatment effects\n\n[7] Elastic integrative analysis of randomised trial and realworld data for treatment heterogeneity estimation\n\n[8] Improved Inference for Heterogeneous Treatment Effects Using Real-World Data Subject to Hidden Confounding\n\n[9] Integrative R-learner of heterogeneous treatment effects combining experimental and observational studies\n\n[10] Adaptive combination of randomized and observational data\n\n[11] Combining observational and experimental datasets using shrinkage estimators\n\n[12] Combining multiple observational data sources to estimate causal effects", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and AC, 

We sincerely thank all reviewers and AC for your great effort and constructive comments on our manuscript. During the rebuttal period, we have been focusing on these beneficial suggestions from the reviewers and doing our best to add several experiments.

As reviewers highlighted, we believe our paper tackles an important and relevant problem (**Reviewer TTAY**, **Reviewer X846**), and introduces a novel and interesting idea (**Reviewer X846**). We also appreciate that the reviewers found our paper well-organized (**Reviewer X846**) and offers solid and convincing experiments (**Reviewer TTAY**, **Reviewer X846**, **Reviewer Yf6Q**).

Moreover, we thank the reviewers for the suggestions for incorporating feature imputation methods (**Reviewer TTAY**), as well as pointing out the concerns regarding the empirical performance improvement (**Reviewer X846**, **Reviewer Yf6Q**), and time and space efficiency (**Reviewer X846**). In response to these comments, we have added the following experiments:

- [Reviewer TTAY] We **add experiments to compare the feature imputation methods with proposed MetaDebias** (in Table 1 and Table 2).
- [Reviewer X846 and Reviewer Yf6Q] We **add experiments on additional PCIC and Ali-CCP datasets to further validate the effectiveness** of MetaDebias (in Table 3).
- [Reviewer X846] We **add experiments to investigate the time and space efficiency** on the **Coat**, **Yahoo! R3**, and **KuaiRec** datasets (in Table 4).

We hope our response could address all the reviewers' concerns, and are more than eager to have further discussions with the reviewers in response to these revisions.

Thanks,

Submission9441 Authors.

Meta-review: The manuscript tackles the hidden confounder issue in offline learning from observational datasets. The key idea is to leverage heterogenous observational data (i.e., all features are completely observable in some instances) to perform error imputation, which leads to improved learning of prediction models. 

The reviewers generally appreciate the idea proposed in this paper, but also expressed serious concerns regarding the clarity about the problem setup and proposed methodology. First of all, even during the rebuttal period, the authors kept emphasizing “missing features” is equivalent to “hidden/missing confounders”. This is only true under the key assumption made in this paper, which they dodged, is that there are training samples where all confounders are observed. This is clearly a very strong assumption and can hardly hold in the targeted application of this paper, i.e., recommender systems. For example, a shopping website might never observe their users’ latest salary figures or medical history. 

Second, the notion of $x_{u,i}$ is quite confusing: it actually has different dimensions when $g=1$ vs $g=0$. Then it is a bit hard to imagine how to implement those prediction models, e.g., $f(x_{u,i})$ and $\eta(x_{u,i})$. We urge the authors to better explain this in the revision. 

Last but not least, although the authors kept emphasizing the goal is to perform unbiased learning from observational data, there is no discussion about how or whether the proposed method could be unbiased. 

To summarize, we believe this manuscript has its unique value and would be a good add to the research in unbiased learning from observational data. We urge the authors improve the clarity of the manuscript and include the discussions suggested by the reviewers.
Decision: Accept (poster)