Addressing Hidden Confounding with Heterogeneous Observational Datasets for Recommendation

Authors: yanghao  xiao
haoxuan  li
yongqiang  tang
wensheng  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:43:56

Abstract: The collected data in recommender systems generally suffers selection bias. Considerable works are proposed to address selection bias induced by observed user and item features, but they fail when hidden features (e.g., user age or salary) that affect both user selection mechanism and feedback exist, which is called hidden confounding. To tackle this issue, methods based on sensitivity analysis and leveraging a few randomized controlled trial (RCT) data for model calibration are proposed. However, the former relies on strong assumptions of hidden confounding strength, whereas the latter relies on the expensive RCT data, thereby limiting their applicability in real-world scenarios. In this paper, we propose to employ heterogeneous observational data to address hidden confounding, wherein some data is subject to hidden confounding while the remaining is not. We argue that such setup is more aligned with practical scenarios, especially when some users do not have complete personal information (thus assumed with hidden confounding), while others do have (thus assumed without hidden confounding). To achieve unbiased learning, we propose a novel meta-learning based debiasing method called MetaDebias. This method explicitly models oracle error imputation and hidden confounding bias, and utilizes bi-level optimization for training. Extensive experiments on three public datasets validate our method achieves state-of-the-art performance in the presence of hidden confounding, regardless of RCT data availability.

Main paper: 
 introduction: 
 Recommender systems have been developed with the purpose of providing users with personalized recommendations, serving as a potent tool in capturing users' true preferences [22] . In recent years, deep learning algorithms are proposed to train recommendation models with collected historical data. However, the selection bias introduced by the users' selective interactions with items poses a challenge to the unbiased learning of the prediction model [4, 54, 59] . For instance, in explicit feedback data, users are free to rate items they prefer, thereby engendering the distribution discrepancy between interacted and non-interacted data [35, 44, 55] .To alleviate selection bias, a line of works are proposed, including error imputation [2, 16, 48] , inverse propensity weighting [21, 36, 45, 46] , and doubly robust [39, 43, 55] . Moreover, recent works have sought to enhance these foundational methods from diverse perspectives, including analysis on both bias and variance [7, 12, 27] , considerations for training stability [31, 49] , and integration with multi-task learning [37, 52, 64] . However, they all fail to achieve unbiased when some features Figure 1 : Toy examples and causal graphs of heterogeneous observational data, both of which are missing not at random (MNAR) due to the selection bias. In causal graphs, b u,i , o u,i , r u,i and h u,i denote basic mandatory features, observation, rating and optional features, respectively, where observed and unobserved variables are represented by solid-line and dashed-line circles.that simultaneously affect both user selection mechanism and feedback remain unobserved, which is called hidden confounding and is widely prevalent in real-world scenarios [10, 28, 29] .To tackle hidden confounding, sensitivity analysis based methods inspired by causal inference are proposed, which assume the true propensities are near and bounded by the estimated values, and further adopt worst-case optimization [10, 66] . More recently, data fusion based methods leverage a few RCT data collected from randomized controlled trials or A/B tests to calibrate propensity and imputation models to achieve unbiasedness [28, 29] . Unfortunately, both of them prove challenging to implement in real-world scenarios, as the former relies on strong assumption on hidden confounding strength, whereas the latter relies on the significantly expensive RCT data.To fill this gap, this paper proposes to employ heterogeneous observational data to address hidden confounding, wherein some data is affected by hidden confounding while the remaining is not. We argue that such setup is more aligned with practical recommendation scenarios, especially when some users/items do not have complete information (thus assumed with hidden confounding), while others do have (thus assumed without hidden confounding). For example, in Figure 1a , during user registration process, some users such as Alice may complete both mandatory fields and optional fields like age and salary, while others such as Bob may only complete mandatory fields. Similarly, this also holds on the item side. We depict the corresponding causal graphs in Figure 1b , where b u,i and h u,i denote basic mandatory and optional features that affect both observation and feedback, respectively. To achieve unbiased learning with above heterogeneous observational data, we propose a novel metalearning based debiasing method called MetaDebias, to explicitly estimate the oracle prediction errors and the bias introduced by hidden confounders. We further adopt bi-level optimization technique for training the prediction model. The contributions of this work are summarized as follows.• To the best of our knowledge, this is the first work to employ heterogeneous observational data to address hidden confounding in debiased recommendation, relaxing the reliance on RCT data in previous data fusion methods.• We propose a meta-learning based debiasing method called MetaDebias to explicitly estimate the oracle error imputation, and employ bi-level optimization for model training.• We conduct extensive experiments on three public datasets, and our method achieves state-of-the-art performance in the presence of hidden confounding, regardless of the availability of RCT data. 
preliminaries and related work: 
 Let U = {u}, I = {i}, and D = {(u, i) | u ∈ U, i ∈ I} be the set of users, the set of items, and the target population consisting of all user-item pairs, respectively. Define x u,i and r u,i be the observed features and rating of user-item pair (u, i), and suppose observation indicator o u,i ∈ {0, 1} Sensitivity Analysis. Inspired from causal inference literature, sensitivity analysis based approach assumes the true propensity p u,i is near and can be bounded by the estimated propensity pu,i = P(o u,i = 1|x u,i , g u,i = 0), and adopt worst-case optimization to mitigate hidden confounding [10, 66] . However, above strong assumption on hidden confounding strength is hard to be satisfied in real-world scenarios and such method fails when the assumption is violated.Model Calibration. Recent works propose to leverage a few unbiased RCT data collected from randomized controlled trials or A/B tests for model calibration [28, 29] . Collecting RCT data requires users to rate items randomly, hence RCT samples are representatives of the target population, and the prediction loss on these samples serves as an unbiased estimator of the ideal loss. Thus, the biased propensity and imputation models learned from the incomplete observed features x u,i can be corrected using such unbiased loss, for instance, with the help of additive residual models [28] or multiplicative reweighting models [29] . However, the acquisition cost of RCT data is prohibitively high, posing challenges to the practical implementation of such methods in real-world settings.Apart from debiased recommendation, more related works about causal inference under hidden confounding can be found in Appendix A.1, which is a widely studied topic.3 MetaDebias: Meta Learning Based Debiased Recommendation Approach 
problem formulation: 
 We first present the problem formulation, that is to provide an unbiased estimator for the ideal loss L ideal in the presence of hidden confounding, given the collected heterogeneous observational data with known data source indicator g u,i . Unlike existing works that assume training data originates from a single dataset without or with hidden confounding, we argue that the training data is more likely composed of two heterogeneous datasets, one of which has sufficient feature collection and is assumed without hidden confounding (g u,i = 1), while the other has insufficient feature collection and is assumed with hidden confounding (g u,i = 0) with hidden confounders denoted as h u,i .Here, based on the given g u,i ∈ {0, 1} for distinguishing between two types of observational data, we partition the observed population O into two subpopulationsD 0 = {(u, i) | (u, i) ∈ D, g u,i = 0, o u,i = 1} and D 1 = {(u, i) | (u, i) ∈ D, g u,i = 1, o u,i = 1}, while unobserved population is D u = {(u, i) | (u, i) ∈ D, o u,i = 0}.A naive approximation method for ideal loss is minimizing the prediction loss over the observed data. Specifically, given D 0 and D 1 , the respective losses over each subgroup are as follows:L D0 = 1 |D 0 | (u,i)∈D0 e u,i , L D1 = 1 |D 1 | (u,i)∈D1e u,i .However, due to selection bias, neither L D0 , L D1 nor their combination L Naive = L D0 + L D1 is an unbiased estimator for the ideal loss. Furthermore, as discussed in the Preliminaries 2, prior methods cannot be applied to achieve the goal without additional model or data assumptions.To achieve the goal, we propose to explicitly estimate the prediction error e u,i on all user-item pairs D using observed features x u,i , as the ideal loss is defined as the average prediction error over D, i.e., L ideal = 1 |D| (u,i)∈D e u,i . In other words, our goal is transformed into accurately estimating the oracle error imputation E [e u,i | x u,i ], which is inherently an unbiased estimator of the ideal loss.The challenges in achieving accurate oracle error imputation estimation lie in the fact that the prediction error e u,i is only partially observable, and the missing mechanisms differ between the two subgroups D 0 and D 1 due to the influence of hidden confounding. To address these challenges, we define the identifiable propensity scores aimed at modeling the two types of missing mechanisms, along with the naive error imputations defined over the entire space D, to assist in estimating the target oracle error imputation. Next, we will introduce the details.To start with, different from the previously widely used propensity score P(o u,i = 1|x u,i ) which is applicable only for samples in subgroup D 1 and the generalized formulation P(o u,i = 1|x u,i , h u,i ) designed for subgroup D 0 which is unidentifiable, we define the identifiable propensity score π(x, g) which models the probability of being observed for any user-item pair on the entire space D:EQUATIONThe proposed π(x, g) models the propensity score for both the absence and presence of hidden confounding, based on the input data source indicator g u,i .Similarly, we propose to use the observed features x u,i and data source indicator g u,i to estimate the naive prediction error as o u,i • e u,i , which is computable on the target population D and expected to be the prediction error e u,i when sample (u, i) is observed o u,i = 1, or to be zero when not observed o u,i = 0. Note that the zero value in the defined naive prediction error o u,i • e u,i contains the information about data missing mechanism, thus g u,i is used to capture the differences in the missing mechanism when hidden confounding is present or absent. Formally, to achieve this target, we define the naive error imputation m(x, g) as:EQUATIONBased on the proposed propensity score π(x, g) and naive error imputation m(x, g), the oracle error imputation E [e u,i | x u,i ] can be derived, we summarize their relationship in Lemma 1 below. EQUATIONwhereη(x) = E [e u,i | x u,i = x, g u,i = 0, o u,i = 1] -E [e u,i | x u,i = x].The above result delineates the conditional independence relationship between observation and prediction error. Specifically, on subgroup D 1 where no hidden confounding exists, the prediction error e u,i is independent of the observation o u,i given the observed features x u,i , i.e., e u,i ⊥ ⊥ o u,i | x u,i , g u,i = 1, and the expectation of o u,i • e u,i naturally equals the product of their respective expectations as shown in Equation 3.However, on subgroup D 0 where hidden confounders h u,i exist, above conditional independence does not hold, thus we introduce an additional residual module η(x) which describes the expectation difference between the mean prediction error over target population D and the biased subpopulation D 0 , to account for the bias introduced by hidden confounding.Lemma 1 provides an approach for estimating the oracle error imputation E [e u,i | x u,i ], and the estimation depends solely on propensity score π(x, g) and naive error imputation m(x, g). Note that both π(x, g) and m(x, g) are typically estimated from the heterogeneous observational datasets, which implies that when the estimation of π(x, g) and m(x, g) are inaccurate, the resultant oracle error imputation estimation will further deteriorate in accuracy.To EQUATIONwhereξ = o u,i • {e u,i -{E[e u,i | x u,i = x] + (1 -g)η(x)}} with E[ξ | x, g] = 0.The findings in Lemma 2 provide an alternative and robust approach for estimating the oracle error imputation E [e u,i | x u,i ], wherein the estimation depends not only on the learned propensity score π(x, g), naive error imputation m(x, g), but also on the observation o u,i , along with the naive prediction error o u,i • e u,i , where ξ with zero-mean property can be regarded as a noise.For the method implementation, we employ deep learning models to estimate the aforementioned proposed modules, and name this approach as MetaDebias. To ensure consistency in the model input, we fill the missing feature values with 0, so that the observed features x u,i have the same dimensions when g u,i = 0 and g u,i = 1. We present the architecture of MetaDebias in Figure 2 , and introduce the training objective of each model as follows.Initially, we employ the commonly used cross-entropy loss to train the propensity model π(x, g; ϕ π ) with parameters ϕ π using heterogeneous data, denoted as L π (ϕ π ) shown in Equation 5 below:EQUATION)According to the definition of m(x, g) in Equation 2, we adopt the square loss to train the naive imputation model m(x, g; ϕ m ) with parameters ϕ m , denoted as L m (ϕ m , θ) shown in Equation 6:EQUATION)where e u,i = L(f (x u,i ; θ), r u,i ) is the prediction error and only available when o u,i = 1.Next, based on Lemma 2, we employ the square loss to train the oracle imputation model e(x; ϕ e ) with parameters ϕ e and residual model η(x; ϕ η ) with parameters ϕ η given learned propensity model ϕ π and imputation model ϕ m , denoted as L meta (ϕ e , ϕ η , ϕ m , ϕ π , θ) shown in Equation 7below:EQUATIONFurthermore, we adopt the oracle imputation model e(x; ϕ e ) to generate prediction errors on the target population D as the training objective for prediction model f (x u,i ; θ) with parameters θ. The training objective L pred (θ, ϕ e ) is demonstrated in the following Equation 8:EQUATIONwhere e(x u,i ; ϕ e | θ) is the learned oracle imputation model training from L meta loss in Equation 7.Following previous works [3, 56] , we propose a bi-level optimization based learning algorithm for model training. Specifically, we first train the propensity model π(x, g) independently by minimizing the L π loss shown in Equation 5, as its training objective is orthogonal to those of other models. Next, we use bi-level optimization to update the remaining models.In the bi-level optimization, we initially assumed update of the naive imputation model m(x, g; ϕ m ) and the oracle imputation model e(x; ϕ e ) to ensure that the parameter ϕ ′ e (ϕ ′ m , θ) is differentiable with respect to the parameter θ. Then we update the prediction model f (x u,i ; θ) by L pred (θ, ϕ e ) in Equation 8, where the gradient of θ is accessible to be calculated and back propagated.After obtaining the updated prediction model, we adopt joint learning to update the naive imputation model m(x, g; ϕ m ) by L m in Equation 6and oracle imputation model e(x; ϕ e ) and residual model η(x; ϕ η ) by L mata in Equation 7. We summarize the whole procedure of the learning algorithm in Algorithm 1, where α is denoted as the learning rate and we use distinct subscripts to correspond to different models. Assumed update naive imputation model:ϕ ′ m (θ) = ϕ m -α m ∇ ϕm L m (m(ϕ m ) | θ) ;Assumed update oracle imputation model:ϕ ′ e (ϕ ′ m , θ) = ϕ e -α e ∇ ϕe L meta (e(ϕ e ) | ϕ ′ m , θ); Update prediction model: θ ← θ -α∇ θ L pred f ϕ ′ e (ϕ ′ m ,θ) ; end for number of training steps do Sample a batch of user-item pairs {(u i , i i )} B2 i=1 from D;Update naive imputation model:ϕ m ← ϕ m -α m ∇ ϕm L m (m(ϕ m ) | θ) ; end for number of training steps do Sample a batch of user-item pairs {(u i , i i )} B3 i=1 from D;Update oracle imputation model:ϕ e ← ϕ e -α e ∇ ϕe L meta (e(ϕ e ) | ϕ m , θ) ; Update residual model: ϕ η ← ϕ η -α η ∇ ϕη L meta (η(ϕ η ) | ϕ m , θ) ; end end Output: θ 4 Experiments 4.1 Experimental SetupDataset and Experimental Details. Following previous studies [3, 43, 55, 56] , we conduct extensive experiments on three public datasets, COAT2 , YAHOO! R33 , and KUAIREC4 [11] . COAT contains 6,960 biased ratings and 4,640 unbiased ratings derived from 290 users evaluating 300 items. YAHOO! R3 contains 311,704 biased ratings and 54,000 unbiased ratings derived from 15,400 users evaluating 1,000 items. Both datasets employ a five-point rating scale, and we binarize the ratings greater than 3 as 1 and others as 0. KUAIREC contains 4,676,570 video watching ratios derived from 1,411 users evaluating 3,327 videos. The ratios that greater than 2 are binarized as 1, otherwise as 0. We adopt feature masking to simulate g u,i which measures feature completeness, and g u,i = 1 only when the features of both users and items are fully preserved, otherwise g u,i = 0, and see Appendix A.3 for more details. Following previous works [3, 28, 29, 34] , we randomly split 5% unbiased data from the test set as validation set, and for all methods requiring RCT data, we employ observational data without hidden confounding to pretend RCT data. For evaluation, we employ three widely adopted metrics AUC, Recall@K, and NDCG@K to measure debiasing performance, where we set K = 5 on COAT and YAHOO! R3, and K = 50 on KUAIREC. All the methods are implemented on PyTorch with Adam as the optimizer and NVIDIA A40 as the computing resource, and we tune learning rate in {0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05} and weight decay in [1e -7, 10].Baselines. Two-layer multi-layer perceptron are used as the base model, and we compare proposed methods with both RCT data-free and RCT data-based methods.• RCT data-free Methods: Without RCT data, the baselines includes IPS-based methods: IPS [46] , ESMM [37, 57] , Multi-IPS [64] , ESCM 2 -IPS [52] , BRD-IPS [10] , and DR-based methods: DR [55] , Stable-DR [31] , TDR [27] , Multi-DR [64] , ESCM 2 -DR [52] , , BRD-DR [10] .• RCT data-based Methods: Based on RCT data, the baselines include model selection based methods: KD-Label [34] , AutoDebias [3] , LTD-IPS [56] ,LTD-DR [56] , and model calibration based methods: Bal-IPS [29] , Bal-DR [29] , Res-IPS [28] , Res-DR [28] .Table 1 : Performance on AUC, Recall@K and NDCG@K on the COAT, YAHOO! R3 and KUAIREC datasets. The best result is bolded and the best results of both types of baseline methods are underlined, where * means statistically significant results (p-value ≤ 0.05) using the paired-t-test.  
experimental results: 
 Performance Comparison. We compare our proposed MetaDebias with existing methods and the results are shown in Table 1 . First, all debiasing models outperform the base model which adopts naive empirical risk minimization, indicating the necessity for debiasing. Second, BRD-DR and Res-DR which are two representative methods for eliminating hidden confounding serve as the most competitive baselines, implying that ignoring hidden confounding leads to inevitable performance degradation. Third, compared to RCT data-free methods, RCT data-based methods do not demonstrate significant improvements across the three datasets, indicating that effective model selection or model calibration cannot be achieved using MNAR data without hidden confounding. Furthermore, MetaDebias exhibits significantly superior overall performance on all three datasets, which validates that our method achieves state-of-the-art performance in the presence of hidden confounding leveraging heterogeneous observational data without the utilization of RCT data.In-depth Analysis. We conduct in-depth analysis to further explore the effect of hidden confounding strength, the proportions of heterogeneous observational data, and training data size on performance. Moreover, we further explore the performance under conditions where RCT data is available, which is consistent with the problem setup in prior works. See Appendix A.4 for more experimental results.The Hidden Confounding Strength. Figure 3 shows the impact of hidden confounding strength, where we simulate higher hidden confounding strength by masking more features. We observe all methods exhibit performance degradation as the hidden confounding gets stronger, with vanilla DR exhibiting the poorest performance, highlighting the necessity of removing hidden confounding. The performance of BRD-DR deteriorates significantly, primarily due to the violation of imposed strong model assumptions, while Res-DR demonstrates competitive performance. In addition, the proposed (b) NDCG@5 on the YAHOO! R3 dataset MetaDebias stably outperforms the baselines across varying hidden confounding strength, implying our method is able to effectively address strong hidden confounding using only observational data.The Proportion of Heterogeneous Observational Data. We explore the impact of varying proportions of heterogeneous observational data on the KUAIREC dataset in Figure 4 , where #N (G = 0) and #N (G = 1) denote the quantity of training data with and without hidden confounding. First, we observe the performance of DR varies significantly under different proportions, indicating the pronounced disparities between the heterogeneous observational data. Besides, performances of all methods are enhanced when the proportion of data without hidden confounding increases, this is because more feature information aids in accurate propensity and imputation model learning. Furthermore, MetaDebias demonstrates the best performance, indicating our method can achieve effective debiasing performance across various observational data proportions, and the potential for application in a range of real-world scenarios.The Training Dataset Size. Table 2 explores the impact of training set size on NDCG@K on both KUAIREC and YAHOO! R3 datasets. We find that the performance of all methods declines as the size of training set decreases, highlighting the importance of training data size on model training. Besides, the proposed MetaDebias consistently outperforms the baselines across varying training data size especially when the size is extremely small such as only 10%, which indicates our method that explicitly models oracle prediction errors on the entire space fully exploits the heterogeneous observational data and is still effective even with small training set size.The Impact of RCT Training Data. In Figure 5 , we explore the debiasing performance when RCT data is available, which is consistent with the problem setup in prior works [28, 29] . Specifically, the training dataset is composed of two parts: an observational dataset with hidden confounding and a small RCT dataset. For implementing our proposed MetaDebias, we label the RCT sample as g u,i = 1, as the RCT data represents a special case of MNAR data without hidden confounding. We find that the performances of all methods exhibit an increasing trend as the size of RCT training set increases, which is consistent with previous research findings. Moreover, the proposed MetaDebias stably outperforms all the baseline methods with varying RCT training set size, which indicates our method remains superior to the existing approaches under the previous problem setup, further validating the effectiveness of our method.  
conclusion: 
 In this study, we investigate the problem of selection bias in the presence of hidden confounding. First, we argue that existing methods are challenging to be applied in real-world scenarios, as they either rely on strong assumptions on hidden confounding strength or depend on the costly RCT data. To tackle this issue, we propose to adopt heterogeneous observational datasets which are more likely to be collected to address hidden confounding and claim that such setup is more aligned with practical scenarios. Then, we propose a meta-learning based debiasing method called MetaDebias, which explicitly models the oracle error imputation and additional bias induced by hidden confounders, and we adopt bi-level optimization with assumed update for model training. Extensive experiments conducted on three public datasets validate our method achieves state-ofthe-art performance, regardless of the availability of RCT data. A limitation of this work lies in the assumption that all the confounders can be included through sufficient feature collection. Even though it is possible to collect hundreds of features in industrial scenarios, and some features like consumption records may potentially represent hard-to-obtain features like salary, it is still challenging to guarantee that all confounders for partial samples have been included. In future work, we will explore how to further relax this theoretically feasible assumption.A Appendix / supplemental material A.1 Related Work Debiased Recommendation. Recommender systems play an important role in mitigating information overload which are trained on users' historical feedback, but selection bias inherent in the collected data impede the algorithms from accurately capturing users' true preferences [17, 53, 67] . If we naively adopt empirical risk minimization to train prediction model without debiasing, it will achieve sub-optimal prediction performance [46, 55, 60] . To address the selection bias and achieve unbiased learning, the error imputation based (EIB) method [38, 48] treats selection bias as a missing data problem, imputes the missing data, and subsequently trains prediction model combining with the imputed data. The inverse propensity scoring (IPS) method [36, 45, 46] estimates the probability of an user-item pair being observed which is called propensity and performs inverse propensity weighting for the observed samples. The doubly robust (DR) method [39, 43, 55] combines both error imputation and inverse propensity weighting, achieving superior performance while exhibiting enhanced robustness. Owing to the extensive adoption of IPS and DR, numerous variants have been proposed [26, 31, 49] . Specifically, some approaches focus on bias and variance trade-off [7, 12, 27] , while others integrate with multi-task learning to address data sparsity [52, 64] . Additionally, some works explore how to learn more appropriate propensity scores [3, 30, 32, 56] . However, the aforementioned methods fail to achieve unbiased when some features that simultaneous affect both user selection mechanism and the feedback remain unobserved, which is called hidden confounding. Despite recent works proposed to address observed confounding effects [14, 47] , tackling hidden confounding remains a challenging task. To fill this gap, sensitivity analysis based methods are proposed, where worst case optimization is used to mitigate the hidden confounding [10, 66] . However, the validity of such methods relies on the assumption on hidden confounding effect that true propensities are near and can be bounded by the estimated values, and such approaches become ineffective when this assumption is violated. Motivated by this, recent works propose to leverage a small amount of RCT data for model calibration, where the calibrated estimator serves an unbiased estimator of the ideal loss [28, 29] . Observing that the acquisition cost of RCT data is prohibitively high, it is equally challenging to implement such calibration based methods in real-world applications. In this paper, we propose a more practical solution to address hidden confounding using heterogeneous observational data for explicitly modeling the oracle prediction errors.Causal Inference under Hidden Confounding. Hidden confounding is a widely studied topic in causal inference literature [1, 50, 65] , and there are two primary approaches to addressing this issue. One approach involves leveraging additional information to achieve unbiased estimation of the target causal quantity, for example the instrumental variables [15] , front door adjustment [40] , negative control [33] and data fusion [24, 58] . The second approach based on sensitive analysis, on the other hand, aims to estimate appropriate bounds for the target causal effects, rather than pursuing precise point estimation [6, 23, 41, 51] . However, in practical applications, it is hard to identify suitable instruments or mediators that meet the required criteria [15, 20] , while the sensitive analysis based method heavily relies on strong assumption of the hidden confounding effect, which is also hard to satisfy. The data fusion approach aims to combine data from observational studies and randomized trials, and can be broadly classified into three categories: statistical test based, correction based, and weighted based. Statistical test based methods introduce statistical tests to compare the causal effects estimated from observational studies and randomized trials, thereby detecting and mitigating hidden confounding [8, 9, 18, 19] . Correction based methods are proposed to correct the biased causal effect estimation derived from observational data using the unbiased RCT data [13, 24, 63] , and an efficient integrative estimator is established based on semi-parametric theory [62] , and further integrated with machine learning models [58] . Weighted methods propose to train a biased estimator using observational data, train an unbiased estimator using RCT data, and take the weighted average of these two estimators as the final result [5, 42, 61] . A limitation of data fusion methods is the availability of RCT data, as the cost of obtaining RCT data is prohibitively high. Moreover, for the correction based method, the randomized trial and observational study should share the same support sets. When the support sets differ and only a partial overlap exists, additional strong parametric assumptions are required for extrapolation, for instance, the hidden confounding effect is assumed to be a linear function [24] . In contrast to the aforementioned methods, our proposed approach is more readily feasible in real-world scenarios, as it does not impose assumption on hidden confounding effect and the required heterogeneous observational data is relatively more accessible. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss the potential societal impacts of the work in Appendix A.5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creator or original owner of the assets (e.g., code, data, models) used in the paper is properly credited, and the license and terms of use are explicitly mentioned and appropriately respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Lemma 1 The relationship between the proposed propensity score π(x, g), naive error imputation m(x, g) and the oracle error imputation E [e u,i | x u,i ] is as followed:m(x, g) = {E [e u,i | x u,i = x] + (1 -g)η(x)} • π(x, g), where η(x) = E [e u,i | x u,i = x, g u,i = 0, o u,i = 1] -E [e u,i | x u,i = x].Proof. Recall that:m(x, g) = E[o u,i • e u,i | x u,i = x, g u,i = g], π(x, g) = P (o u,i = 1 | x u,i = x, g u,i = g) .Then, when g equals 1, we have:m(x, g = 1) = E [o u,i • e u,i | x u,i = x, g u.i = 1] = E [e u,i | x u,i = x, g u.i = 1, o u,i = 1] • P (o u,i = 1 | x u,i = x, g u.i = 1) = E [e u,i | x u,i = x, g u.i = 1] • P (o u,i = 1 | x u,i = x, g u.i = 1) = E [e u,i | x u,i = x] • P (o u,i = 1 | x u,i = x, g u.i = 1) = E [e u,i | x u,i = x] • π(x, g = 1).The first equation is the definition of m(x, g), the second equation is the law of total probability, and the third equation holds as the conditional independence e u,i ⊥ ⊥ o u,i | x u,i , g u,i = 1. The fourth equation holds because e u,i = L(f (x u,i ), r u,i ) is independent of the data type variable g u,i , and the last equation is the definition of π(x, g).Similarly, when g equals 0, the equivalent equation holds:m(x, g = 0) = E [o u,i • e u,i | x u,i = x, g u.i = 0] = E [e u,i | x u,i = x, g u.i = 0, o u,i = 1] • P (o u,i = 1 | x u,i = x, g u.i = 0) = E [e u,i | x u,i = x, g u.i = 0, o u,i = 1] • π(x, g = 0) = {E [e u,i | x u,i = x] + η(x)} • π(x, g = 0).The second equation is the law of total probability, the third equation is the definition of π(x, g), and the last equation is the definition of η(x) which is equivalently expressed as:E [e u,i | x u,i = x, g u,i = 0, o u,i = 1] = η(x) + E [e u,i | x u,i = x] .To sum up, the equation holds:m(x, g) = {E [e u,i | x u,i = x] + (1 -g)η(x)} • π(x, g).The relationship between the propensity score π(x, g), naive error imputation m(x, g), observation o u,i , naive prediction error o u,i • e u,i and oracle error imputationE [e u,i | x u,i ] is: o u,i • e u,i -m(x, g) = {E [e u,i | x u,i = x] + (1 -g)η(x)} • {o u,i -π(x, g)} + ξ, where ξ = o u,i • {e u,i -{E[e u,i | x u,i = x] + (1 -g)η(x)}} with E[ξ | x, g] = 0.Proof. We prove this Lemma in two steps. Initially, we establish the equality relation in the Lemmao u,i • e u,i -m(x, g) = {E [e u,i | x u,i = x] + (1 -g)η(x)} • {o u,i -π(x, g)} + ξ.Subsequently, we prove that the conditional expectation of the noise term is zero, i.e., E[ξ | x, g] = 0.To start with, based on Lemma 1, the equation holds:m(x, g) = {E [e u,i | x u,i = x] + (1 -g)η(x)} • π(x, g).According to the definition of ξ, we have: The subtraction of the above two equations yields the following:o u,i • e u,i = ξ + o u,i • {{E[e u,i | x u,i = x] + (1 -g)η(x)}} .o u,i • e u,i -m(x, g) = {E [e u,i | x u,i = x] + (1 -g)η(x)} • {o u,i -π(x, g)} + ξ.Next we demonstrate that ξ possesses the zero mean property. Recall that:ξ = o u,i • {e u,i -{E[e u,i | x u,i = x] + (1 -g)η(x)}} = o u,i • e u,i -o u,i • {{E[e u,i | x u,i = x] + (1 -g)η(x)}} .Then the conditional expectation of ξ is:E [ξ | x, g] = E [o u,i • e u,i | x, g] -E [o u,i • {(E [e u,i | x u,i = x] + (1 -g)η(x))} | x, g] = m(x, g) -E [o u,i • {E [e u,i | x u,i = x] + (1 -g)η(x)} | x, g] = m(x, g) -{E [e u,i | x u,i = x] + (1 -g)η(x)} • E [o u,i | x u,i = x, g u,i = g] = m(x, g) -{E [e u,i | x u,i = x] + (1 -g)η(x)} • P (o u,i = 1 | x u,i = x, g u,i = g) = m(x, g) -{E [e u,i | x u,i = x] + (1 -g)η(x)} • π(x, g) = 0.The first equation is the definition of ξ, the second equation is the the definition of m(x, g), and the third equation holds because E [e u,i | x u,i = x] + (1 -g)η(x) is a constant given x and g. The fourth equation arises from the fact that the treatment variable o u,i is a binary variable taking values of 0 or 1, and the fifth equation is the definition of π(x, g). The last equation holds based on Lemma 1.We introduce more experimental details about feature preparations here. In particular, we first employ Matrix Factorization (MF) [25] to transform the ID-based data format into the feature-based format and regard the derived features as complete features. Next, we simulate the hidden confounding effect induced by insufficient feature collection process through feature masking. The condition g u,i = 1 is satisfied only when both user and item features are sufficiently preserved, i.e., there are no masked values in their respective feature vectors; otherwise, it equals 0. At the outset, we set the proportion of two types of heterogeneous observational data at 1:1, and the impact of varying proportions is explored in the following in-depth analysis.In this subsection, we will present more experimental results that could not be accommodated within the main text due to space limitations. The main text primarily elucidates the results obtained on the KUAIREC dataset. Supplementary to this, the appendix predominantly supplements experimental findings on the YAHOO! R3 dataset, which similarly explores the impact of hidden confounding strength, the proportions of the heterogeneous observational data, and the data sparsity. Apart from these, we conduct an additional investigation into the algorithm runtime on three datasets to explore the computational resource demands of different methods. The findings are as follows. The Hidden Confounding Strength. Figure 6 demonstrates the debiasing performance with varying hidden confounding strength on the YAHOO! R3 dataset. Similar to the findings observed in the KUAIREC dataset shown in the main text, on the YAHOO! R3 dataset, an increase in hidden confounding strength leads to a decrease in performance across all methods. Meanwhile, the performance deterioration of the BRD-DR is more pronounced, particularly in terms of AUC metric, attributed to the violation of the required assumptions. Furthermore, the proposed MetaDebias demonstrates superior performance across all three evaluation metrics, and its performance shows minimal degradation as the hidden confounding strength increases. This indicates the superiority of our approach over existing methods, and robustness to the hidden confounding strength.The Proportion of Heterogeneous Observational Data. Figure 7 shows the prediction performance with varying proportions of heterogeneous observational data on the YAHOO! R3 dataset, where varying data proportions simulate potential data collection scenarios under different real-world conditions. From the figure, we observe an upward trend in the performance of all methods with an increase in the proportion of data without hidden confounding, indicating that the selection bias inherent in such observational data without hidden confounding is more amenable to be addressed. Moreover, MetaDebias outperforms other baselines and exhibits insensitivity to data proportions, which underscores that our method is capable of handling hidden confounding in various possible data combinations and can be broadly applied to a wide range of real-world scenarios.The Training Dataset Size. We explore the impact of training set size on prediction performance in Tables 3 and 4 , which report the AUC and Recall@K with varying training set size on both KUAIREC and YAHOO! R3 dataset. The presented data represents the mean values obtained from 10 random replicate experiments. We find that MetaDebias demonstrates superior performance across varying training data size, highlighting the efficacy of our method. Meanwhile, MetaDebias exhibits relatively minor fluctuations with changes in dataset volume, particularly evident in the YAHOO! R3 dataset, where the AUC of our method consistently maintains superior performance even amidst substantial variations in training data volume, such as a change from 100% to 10%. This further demonstrates the stability of our approach even with small the training data size.The Computational Resource Demands. We investigate the algorithm training and inference time on three datasets in Table 5 . Despite the involvement of five models in the training process, the comparison with other baseline methods reveals that the training time of the proposed approach is Recommender systems (RS) serve as an effective tool for mitigating information overload, yielding significant economic benefits by accurately recommending items of interest to individual user. However, in the data collection process of RS, there inevitably exists a significant amount of bias, rendering prediction models trained on the collected historical feedback incapable of capturing users' true preferences, where selection bias is a particularly common and extensively studied issue.Recently, an increasing number of studies have further focused on and emphasized the impact of hidden confounding, which aims to achieve better prediction performance. In this paper, we propose a more practical solution to effectively address selection bias in the presence of hidden confounding. Specifically, we propose to leverage heterogeneous observational datasets, which is more readily available in real-world scenarios, while not imposing additional assumptions or requiring RCT data. This indicates that our method has the potential to be applied in real-world recommendation scenarios, offering opportunities to enhance economic benefits for businesses. Justification: The paper provides sufficient information on the computer resources (type of compute workers, memory) in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://www.cs.cornell.edu/~schnabts/mnar/https://webscope.sandbox.yahoo.comhttps://github.com/chongminggao/KuaiRec 


Decision: The manuscript tackles the hidden confounder issue in offline learning from observational datasets. The key idea is to leverage heterogenous observational data (i.e., all features are completely observable in some instances) to perform error imputation, which leads to improved learning of prediction models. 

The reviewers generally appreciate the idea proposed in this paper, but also expressed serious concerns regarding the clarity about the problem setup and proposed methodology. First of all, even during the rebuttal period, the authors kept emphasizing “missing features” is equivalent to “hidden/missing confounders”. This is only true under the key assumption made in this paper, which they dodged, is that there are training samples where all confounders are observed. This is clearly a very strong assumption and can hardly hold in the targeted application of this paper, i.e., recommender systems. For example, a shopping website might never observe their users’ latest salary figures or medical history. 

Second, the notion of $x_{u,i}$ is quite confusing: it actually has different dimensions when $g=1$ vs $g=0$. Then it is a bit hard to imagine how to implement those prediction models, e.g., $f(x_{u,i})$ and $\eta(x_{u,i})$. We urge the authors to better explain this in the revision. 

Last but not least, although the authors kept emphasizing the goal is to perform unbiased learning from observational data, there is no discussion about how or whether the proposed method could be unbiased. 

To summarize, we believe this manuscript has its unique value and would be a good add to the research in unbiased learning from observational data. We urge the authors improve the clarity of the manuscript and include the discussions suggested by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper proposes to use heterogeneous observational data to address hidden confounding in recommender system.
strength_weakness: strength_weakness: - It seems to me that using missing feature mechanism to estimate the missing confounder mechanism is a over simplification of the problem, where the author provide no strong empirical evidence to demonstrate the reliability of this simplification.
- If confounders are just missing features, why don't we just infer them from the data?
questions: questions: Please refer to my summary of weakness.
limitations: limitations: N/A
paper_summary: paper_summary: The paper addresses the issue of selection bias in recommender systems, particularly when hidden confounding factors are present. The authors propose a new approach using heterogeneous observational data, where some data is affected by hidden confounding and some is not. The proposed MetaDebias is a meta-learning based debiasing method that models oracle prediction errors and bias from hidden confounders using bi-level optimization for training. Experiments on three public datasets demonstrate that MetaDebias achieves the best performance despite the presence of hidden confounding.
strength_weakness: strength_weakness: In the comparative experiments shown in Table 1, the improvement of the MetaDebias algorithm over other algorithms is relatively small.

Although the paper is well-organized, the presentation quality needs improvement. For example, some abbreviations should be spelled out when first mentioned, such as Random Controlled Trial (RCT) in the abstract.
questions: questions: How does the proposed MetaDebias algorithm perform in terms of time and space efficiency compared to the baseline?
limitations: limitations: Yes
paper_summary: paper_summary: The paper studies the problem of selection bias due to hidden confounding in recommendation systems. Previous methods struggle with real-world application due to reliance on strong assumptions or unavailable RCT data. The proposed solution, MetaDebias, leverages heterogeneous observational data, which is more practical and readily available. The approach involves a meta-learning framework that uses both confounded observational data and unconfounded observational data for model training. Experiments show that MetaDebias consistently outperforms baseline methods across various metrics and conditions.
strength_weakness: strength_weakness: (W1) I found the use of the term heterogenous to describe the observational data a bit vague and misleading. I think the authors should make very clear from the beginning that they are working with one dataset where they can observe everything (no latents) and one dataset where some variables are unobserved. 

(W2) I think the related work section on causal inference (Appendix A.1) could be improved. In particular, it is not clear what is the weakness of the data fusion methods since they are clustered with IVs and negative controls. Moreover, I think some relevant works that use randomized trials to address hidden confounding are missing, for example [1] and [2]. 

(W3) The (empirical) performance improvement over previous methods is not so strong.

(Minor) Typo in Lemma 1, should be "as follows" not "as followed". 

[1] Hidden yet quantifiable: A lower bound for confounding strength using randomized trials. De Bartolomeis et al. AISTATS '24


[2] Falsification before Extrapolation in Causal Effect Estimation. Hussain et al. NeurIPS '22
questions: questions: (Q1) Could you explain with more details how are you assessing the statistical significance of the performance improvement over the previous methods?
limitations: limitations: Yes

Comments: title: We would like to supplement further clarification for the equivalence of "Hidden Confounding" and "Missing Features (Covariates)".
We are happy to provide more details about the PCIC and Ali-CCP datasets!
Thank you for raising your score!
comment: Dear Reviewer TTAY,

Since the discussion period will end in a few hours, we will be online waiting for your feedback on our rebuttal, which we believe has fully addressed your concerns.

We would highly appreciate it if you could take into account our response when updating the rating and having discussions with AC and other reviewers.

Thank you so much for your time and efforts. Your feedback would be extremely helpful to us. If you have further comments or questions, we hope for the opportunity to respond to them.

Many thanks,

Submission9441 Authors
Thanks for your response. I will keep my score "7: Accept".
Dear Reviewer TTAY,

Thank you again for your time to review our paper and thoughtful feedback. In below, we would like to further supplement two main claims to help readers understand our problem setup.

**Main Claim 1: For the potential outcome framework [1, 2] in causal inference, ```hidden confounding``` and ```missing features (covariates)``` are  equivalent, supported by the relevant literature published in important venues [3, 4, 5, 6].**

- In [3], Section 2 (Binary Outcome) on page 3, the ```unmeasured confounder``` is defined as $U$, whereas in Section 3 (Survival Time) on page 8, $U$ is referred to as a ```covariate```.

- In [4], on page 1, the authors explicitly wrote that _“throughout the article we use the term ```‘unmeasured confounder’``` rather than using terms such as ```‘omitted variable’``` or ```‘unobserved covariate’``` for the sake of consistency and clarity”_.

- In [5], the ```unobserved covariate``` $U$ is first defined in the unconfoundedness assumption in Section 2 (Problem Statement and Preliminaries) on page 2. In the following Section 2.1 (Related Work), $U$ is explicitly referred to as the ```unobserved confounder```.

- In [6], Section 2 (Preliminaries and Challenges to Identification) on page 5, the problem formulation is explicitly stated as follows, _"we are interested in the effects of $X$ on $Y$ , which may be ```confounded by a vector of $q$ unobserved covariates $U$```”_, which indicates the equivalence of confounders and covariates.

**Main Claim 2: In recommendation systems, prior works addressing hidden confounding similarly builds on the equivalence between ```hidden confounding``` and ```missing features``` [7, 8, 9].**

- In [7], Section 2 (Problem Formulation) on page 2, $x_{u,i}$ is defined as the ```observed features``` of user-item pair $(u, i)$, and is considered to be the ```measured confounders```. In addition, ```unmeasured confounders``` $h_{u,i}$ refer to the ```unobserved features```, as shown in Figure 2.

- In [8], the authors claim that “we assume that ```all confounders consist of a measured part $x_{u,i}$ and a hidden (unmeasured) part $h_{u,i}$```, where the latter arises from issues such as information limitations (e.g., friend suggestions) and privacy restrictions (e.g., user salary)” in the Section 2 (Problem Setup) on page 3.

- In [9], the authors state that “The ```observed feature/confounder``` $x_{u,i}$ refers to the ```observed feature``` vector from the user $u$, item $i$”, which indicates the consistency between the confounders and the features.

From above, **we conclude that the terminologies ```hidden confounders``` and ```unobserved features``` are exactly equivalent, not over-simplification, in the potential outcome framework of causal inference and debiased recommendation literature. In addition, we also added extensive experiments to validate the superiority of our method compared to the "simple" feature imputation methods** (please kindly refer to our rebuttal). We will definitely put the above discussions and added experiments into our final version to fully address your concern!

***

Could you please check whether they properly addressed your concern? If there are more remaining issues, we would appreciate the chance to address them and work towards achieving a higher score. We deeply appreciate all the insightful comments you have posted, as they have greatly enhanced our paper!

With thanks and warm wishes,

Submission9441 Authors

***
**References**

[1] Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational psychology 1974.

[2] Neyman et al. On the application of probability theory to agricultural experiments. Statistical Science 1990.

[3] Danyu Lin et al. Assessing the sensitivity of regression results to unmeasured confounders in observational studies. Biometrics 1998.

[4] Nicole Bohme Carnegie et al. Assessing sensitivity to unmeasured confounding using a simulated potential confounder. Journal of Research on Educational Effectiveness 2016.

[5] Nathan Kallus et al. Confounding-robust policy improvement. NeurIPS 2018.

[6] Wang Miao et al. Identifying effects of multiple treatments in the presence of unmeasured confounding. Journal of the American Statistical Association 2023.

[7] Sihao Ding et al. Addressing Unmeasured Confounder for Recommendation with Sensitivity Analysis. KDD 2022.

[8] Haoxuan Li et al. Removing Hidden Confounding in Recommendation: A Unified Multi-Task Learning Approach. NeurIPS 2023. 

[9] Zhiheng Zhang et al. Robust causal inference for recommender system to overcome noisy confounders. SIGIR 2023.
Thank you for engaging with our responses and we are highly encouraged to know that "you really like our work". In below, we are happy to provide more details about the **PCIC** and **Ali-CCP** datasets.

- **PCIC is a public dataset** for evaluating debiasing algorithms in recommendations [1, 2]. In the training set, users are free to choose items to rate, resulting in 19,420 biased ratings. While in the test set, users are required to rate randomly exposed items, resulting to 2,040 unbiased ratings. **FYI, the PCIC dataset is public available: https://competition.huaweicloud.com/information/1000041488/introduction.**

- **Ali-CCP is a public dataset** gathered from real-world traffic logs of the recommender system in an e-commerce platform [3]. The training and test set are split along the time sequence of traffic logs, which is a traditional industrial setting. Specifically, the latter 1/2 data in the time sequence are split to be test set, thus **the test data is not exactly unbiased, but with different training the inference space, which is similar to our debiased recommendation problem setup**. In summary, Ali-CCP is also a widely adopted debiased recommendation dataset especially used for **post-click conversion rate (pCVR) estimation** task in recommendation systems [3, 4, 5, 6, 7, 8]. **The Ali-CCP dataset is public available: https://tianchi.aliyun.com/datalab/dataSet.html?dataId=408.**

***

We will definitely put the above experimental details and results into our revised manuscript  -- thank you so much!

***
**References**

[1] Mengyue Yang et al. Debiased Recommendation with User Feature Balancing. Transactions on Information Systems 2023.

[2] Mengyue Yang et al. Generalizable Information Theoretic Causal Representation. arXiv.

[3] Xiao Ma et al. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. SIGIR 2018.

[4] Wenhao Zhang et al. Large-scale Causal Approaches to Debiasing Post-click Conversion Rate Estimation with Multi-task Learning. WWW 2020.

[5] Dongbo Xi et al. Modeling the Sequential Dependence among Audience Multi-step Conversions with Multi-task Learning in Targeted Display Advertising. KDD 2021.

[6] Hao Wang et al. ESCM2: entire space counterfactual multi-task model for post-click conversion rate estimation. SIGIR 2022.

[7] Xiaofan Liu et al. Task Adaptive Multi-learner Network for Joint CTR and CVR Estimation. WWW 2023.

[8] Xinyue Zhang et al. Adversarial-Enhanced Causal Multi-Task Framework for Debiasing Post-Click Conversion Rate Estimation. WWW 2024.
Thank you for your response. I really like the work. 
It will be better if the authors could provide more details about the PCIC and Ali-CCP datasets. Is PCIC public access?  Are the ratings in test set of Ali-CCP unbiased?
We are happy that our clarifications and additional experiments addressed your current concerns. We look forward to your continued support during the follow-up discussion period -- thank you so much!
I thank the authors for their response. In light of the clarifications and additional experiments I will raise my score.