Title: Removing Hidden Confounding in Recommendation: A Unified Multi-Task Learning Approach
Venue: NeurIPS 2023
Accepted: yes
Keywords: Debiased recommender system

Multi-task learning

Causal inference
Abstract: In recommender systems, the collected data used for training is always subject to selection bias, which poses a great challenge for unbiased learning. Previous studies proposed various debiasing methods based on observed user and item features, but ignored the effect of hidden confounding. To address this problem, recent works suggest the use of sensitivity analysis for worst-case control of the unknown true propensity, but only valid when the true propensity is near to the nominal propensity within a finite bound. In this paper, we first perform theoretical analysis to reveal the possible failure of previous approaches, including propensity-based, multi-task learning, and bi-level optimization methods, in achieving unbiased learning when hidden confounding is present. Then, we propose a unified multi-task learning approach to remove hidden confounding, which uses a few unbiased ratings to calibrate the learned nominal propensities and nominal error imputations from biased data. We conduct extensive experiments on three publicly available benchmark datasets containing a fully exposed large-scale industrial dataset, validating the effectiveness of the proposed methods in removing hidden confounding.

Removing Hidden Confounding in Recommendation: A Unified Multi-Task Learning Approach

Abstract

In recommender systems, the collected data used for training is always subject to selection bias, which poses a great challenge for unbiased learning. Previous studies proposed various debiasing methods based on observed user and item features, but ignored the effect of hidden confounding. To address this problem, recent works suggest the use of sensitivity analysis for worst-case control of the unknown true propensity, but only valid when the true propensity is near to the nominal propensity within a finite bound. In this paper, we first perform theoretical analysis to reveal the possible failure of previous approaches, including propensity-based, multi-task learning, and bi-level optimization methods, in achieving unbiased learning when hidden confounding is present. Then, we propose a unified multi-task learning approach to remove hidden confounding, which uses a few unbiased ratings to calibrate the learned nominal propensities and nominal error imputations from biased data. We conduct extensive experiments on three publicly available benchmark datasets containing a fully exposed large-scale industrial dataset, validating the effectiveness of the proposed methods in removing hidden confounding.

1 Introduction

Recommender systems (RS) play a key role in information retrieval by filtering out items that may be of interest to users [16, 24]. In general, the training process of recommendation uses historical user interaction data. However, a challenge in using interactions to make predictions is selection bias [3, 32, 40, 57], i.e., users always choose preferred items to interact [5, 51], resulting in a difference in the distribution between data with and without interactions [14, 45-49, 54, 56, 60], which poses a great challenge for unbiased evaluation and learning of the prediction models.

Many methods have been proposed to tackle the selection bias problem, such as the error imputationbased (EIB) [2], inverse propensity scoring (IPS) [33, 41, 42], and doubly robust (DR) methods [7, 11, 27-29, 52]. Based on these, recent works further incorporate multi-task learning to alleviate the data sparsity issue, such as entire space multi-task model (ESMM) [34], multi-task IPS and multi-task DR (Multi-IPS and Multi-DR) [59], and entire space counterfactual multi-task models (ESCM 2 ) [44].

However, these studies only consider the selection bias induced by measured confounders, but ignore the presence of hidden (or unmeasured) confounders, which considerably influences the application of these advanced methods in real-world recommendation scenarios. Hidden confounders are ubiquitous and inevitable in RS due to information limitations (e.g., friend useful suggestions) or privacy restrictions (e.g., user salary) [8, 31, 35, 50]. In this paper, we perform theoretical analysis

for the existing propensity-based and multi-task learning methods, showing that all of them will lead to biased evaluation and learning in the presence of hidden confounding.

To remove hidden confounding, previous causal inference literature suggested the use of instrumental variables [12] or front door adjustment [36]. However, in real-world application scenarios, these approaches require strong assumptions that are difficult to verify in practice. On the other hand, a recent recommendation study proposed robust deconfounder (RD) to adopt sensitivity analysis using a min-max optimization for worst-case control of the unknown true propensity [8]. Nevertheless, they assume the true propensity is near to the nominal one within a bound, which is decided by the strength of the unmeasured confounder, posing another challenge in case these assumptions are violated.

In contrast, unbiased datasets are regarded as the gold standard for unbiased evaluations and can be collected from A/B tests or randomized controlled trials [10, 13, 21, 55]. This provides an alternative solution to remove hidden confounding [15, 20, 26, 58]. Nevertheless, training recommendation models directly on unbiased datasets can suffer from severe overfitting due to the limited sample size restricted by the collection cost of random exposure [4]. Despite few works leverage the unbiased data to combat selection bias [4, 53] by adopting the bi-level optimization, this paper shows that they still lead to biased learning, due to the biased hypothesis space of the prediction model.

To tackle the above problems, we propose a unified multi-task learning approach to remove hidden confounding by leveraging a few unbiased ratings. Interestingly, we show that the unbiased data can help calibrate the learned nominal propensities and nominal error imputations, which differs from RD using sensitivity analysis for nominal propensities only [8]. Specifically, the proposed multi-task learning builds residual networks for learned propensities and imputed errors from biased data, respectively. Next, a consistency loss of IPS (or DR) estimation on the biased dataset and empirical average prediction errors on the unbiased dataset are developed to help the training of the two residual networks, for calibrating the learned nominal propensities and nominal error imputations, respectively. The prediction model is then trained with the calibrated IPS (or calibrated DR) loss and unbiased data to achieve unbiased learning in the presence of hidden confounding.

The main contributions of this paper are summarized as follows:

We theoretically reveal limitations of the existing multi-task learning and bi-level optimization methods for achieving the unbiasedness in the presence of hidden confounding.

We provide a unified multi-task learning approach to remove hidden confounding by combining a few unbiased ratings, in which the learned nominal propensities and nominal error imputations can be calibrated by the residual networks with the proposed consistency loss.

We conduct extensive experiments on three publicly available benchmark datasets, including a large-scale industrial dataset, to validate the effectiveness of the proposed methods.

2 Problem Setup

We formulate the selection bias in RS using post-click conversion rate (pCVR) prediction task for illustration purpose, which can be naturally generalized to other recommendation tasks with explicit feedback, e.g., rating prediction. Suppose that the entire space has m users and n items, let D be the set of all user-item pairs. Denote R ∈ { 0 , 1 } m × n as the true post-click conversion label matrix of user-item pairs, where each entry r u,i indicates whether a conversion occurs after user u clicks on item i . Let x u,i be the feature of user-item pair ( u, i ) , and ˆ R ∈ R m × n be the prediction matrix for pCVR, where ˆ r u,i = f ( x u,i , θ CVR ) ∈ [0 , 1] is the predicted pCVR obtained by a model f with parameter θ CVR. In RS, users always select the preferred items to click on, leading to a significant difference in the distribution between clicked and unclicked events thus causing selection bias.

If R is fully observed, then a pCVR model f ( x u,i ; θ CVR ) can be trained by minimizing the ideal loss

where δ u,i ≜ δ ( r u,i , ˆ r u,i ) and δ ( · , · ) is a pre-specified loss, e.g., the cross-entropy loss, δ ( r u,i , ˆ r u,i ) = -r u,i log ˆ r u,i -(1 -r u,i ) log (1 -ˆ r u,i ) . However, the post-click conversion feedback of a user-item pair ( u, i ) can be observed only when user u clicks on item i , making the ideal loss not computable. Let o u,i be the indicator of user u clicking on item i , and B = { ( u, i ) | ( u, i ) ∈ D , o u,i = 1 } be

the set of clicked events, where B means that the clicked events is a biased sample of the entire space D . In this paper, we further consider the presence of hidden confounding. Without loss of generality, we assume that all confounders consist of a measured part x u,i and a hidden (unmeasured) part h u,i , where the latter arises from issues such as information limitations (e.g., friend suggestions) and privacy restrictions (e.g., user salary), which cannot be observed explicitly and used for training.

3 Previous Methods Lead to Biased Learning under Hidden Confounding

3.1 Multi-Task Learning

The pCVR prediction task is closely related to click-through rate (CTR) and post-view click-through &conversion rate (CTCVR) prediction tasks, as the formula CTCVR = CTR ∗ pCVR holds, where the CTR is p u,i ≜ P ( o u,i = 1 | x u,i ) , also known as propensity score in the causal inference literature, represents the probability of a user u clicking on an item i . The CTCVR is P ( r u,i = 1 , o u,i = 1 | x u,i ) , means the probability that item i is clicked and converted by user u .

Let ˆ p u,i ≜ ˆ p u,i ( x u,i , θ CTR ) be CTR prediction model with parameter θ CTR. The ESMM method [34] learns pCVR by joint-training CTR and CTCVR losses

However, the ESMM loss L CTR ( θ CTR ) + L CTCVR ( θ CTR , θ CVR ) is a biased estimator of the ideal loss [59]. To achieve unbiased learning, the MTL-IPS and MTL-DR methods [59] use the losses

where ˆ δ u,i ≜ ˆ δ u,i ( x u,i , θ IMP ) is the imputation model that predicts δ u,i using x u,i , i.e., it estimates g u,i ≜ E [ δ u,i | x u,i ] . Without hidden confounders, L IPS is an unbiased estimator of the ideal loss when the learned propensities are accurate, i.e., ˆ p u,i = p u,i [41, 42], and L DR is unbiased if either ˆ p u,i = p u,i or ˆ δ u,i = g u,i [39, 52]. However, both IPS and DR are biased under hidden confounders.

Lemma 1 (Theorem 3.1 in [8]) . In the presence of hidden confounders h u,i , both L IPS and L DR are biased estimators of the ideal loss, even if ˆ p u,i = p u,i and ˆ δ u,i = g u,i .

The above result also holds in the follow-up DR studies [23, 25, 43], and can be naturally extended to conclude that the MTL-IPS and MTL-DR [59] are biased under hidden confounding. Recently, ESCM 2 -IPS and ESCM 2 -DR showed state-of-the-art performance in pCVR prediction by incorporating the ESMM loss L CTCVR as the global risk of CTCVR, and MTL-IPS and MTL-DR losses L B CVR as counterfactual risk of pCVR [44]. Formally, the ESCM 2 loss is

where λ t for t = 1 , 2 , 3 are hyper-parameters, L B CVR is either L IPS or L DR. The imputation loss is

However, since both L CTCVR and L B CVR are biased under hidden confounding, ESCM 2 is also biased.

3.2 Debiasing with a Few Unbiased Ratings

Instead of only using the biased dataset B , many methods are proposed to improve the debiasing performance by combining a small unbiased dataset U and a large biased dataset B , such as bilevel optimization approaches, including learning to debias (LTD) [53] and AutoDebias [4], causal embedding method (CausE) [1], knowledge distillation framework for counterfactual recommendation via uniform data (KDCRec) [30, 32], and causal balancing methods [26].

Specifically, learning to debias (LTD) [53] and AutoDebias [4] adopt bi-level optimization [17, 37] to learn a CTR model ˆ p u,i ( x u,i , θ CTR ) such that the pCVR prediction model ˆ r u,i = f ( x u,i , θ CVR ) performs well on the small unbiased dataset U . Formally, the bi-level optimization in LTD is

where the upper loss is as an average of prediction errors on the unbiased dataset

and AutoDebias further develop a solution for universal debiasing. However, despite the use of unbiased ratings, we show that they still lead to biased estimates under hidden confounding.

Proposition 1. In the presence of hidden confounders, both LTD and AutoDebias are biased.

Proof. Let H CTR = { ˆ p u,i ( x u,i , θ CTR ) : θ CTR ∈ Θ CTR } be the hypothesis space of propensity model. Then θ ∗ CTR defined above is the parameter in Θ CTR such that the pCVR prediction model performs optimally on U . The bi-level optimization takes L B CVR to estimate the ideal loss using a selected propensity ˆ p u,i ( x u,i , θ ∗ CTR ) ∈ H CTR, however, L B CVR could be a biased estimator of the ideal loss for all ˆ p u,i ∈ H CTR (due to ˆ p u,i could deviates from the true one by an arbitrary distance, as formally stated in Theorem 2), therefore both LTD and AutoDebias using bi-level optimization are biased.

Proposition 1 formally reveals the limitations of directly adopting bi-level optimization for addressing hidden confounding. Both LTD and AutoDebias essentially use unbiased data for model selection among all possible IPS or DR losses, but in fact, without correcting the IPS or DR estimators themselves for predicting the ideal loss, those IPS or DR-based methods will not be able to tackle unobserved confounding, because of the intrinsic biasedness of the estimators to the ideal loss.

An alternative class of methods that use unbiased data for tuning is the causal embedding method (CausE) [1] and KDCRec [30, 32]. Specifically, they both consider building a connection between a model trained with biased data and another model trained with unbiased data. CausE [1] designs an alignment term as the pairwise difference between the parameters of the two models, which is then included in the object function to be minimized. KDCRec [30, 32] proposes a general knowledge distillation framework for counterfactual recommendation via uniform data, including label-based, feature-based, sample-based, and model structure-based distillations. The above methods empirically shown impressive performance by distilling the shared information on both biased and unbiased data. However, theoretical guarantees under hidden confounding are lacking, and it would be interesting to investigate the conditions under which these methods can achieve unbiased learning.

3.3 Mitigating Hidden Confounding with Sensitivity Analysis

To tackle the problem of hidden confounding, robust deconfounder (RD) [8] proposes to adopt sensitivity analysis from the causal inference literature [6, 38] to minimize the worst-case prediction loss. Formally, the unknown true propensity ¯ p u,i ≜ P ( o u,i = 1 | x u,i , h u,i ) is assumed to near the nominal propensity p u,i = P ( o u,i = 1 | x u,i ) within a bound that

where hyper-parameter Γ corresponds to the strength of unmeasured confounding, w u,i = 1 / ¯ p u,i is the inverse of the true propensity, and a u,i and b u,i are the lower and upper bounds of w u,i , respectively. Let W = [ˆ a 1 , 1 , ˆ b 1 , 1 ] ×··· × [ˆ a m,n , ˆ b m,n ] be the possible inverse propensities on all user-item pairs, where ˆ a u,i and ˆ b u,i are the estimates of a u,i and b u,i , respectively, then RD-IPS method trains the prediction model by minimizing the worst-case IPS loss

and the RD-DR method can be developed by a similar argument controlling the worst-case DR loss.

To summarize, the RD methods first obtains the bounds of true propensities around the nominal propensities, then minimize the upper bound of the IPS (or DR) loss to control the worst-case caused by unmeasured confounders. However, on the one hand, it is not clear how to set Γ correctly since both the true propensities and the strength of hidden confounding are unknown. On the other hand, the effectiveness of sensitivity analysis for controlling the hidden confounding requires that the true propensity is around the nominal propensity for all user-item pairs, but such (strong) assumptions cannot be verified from the data and raises another concern in case the assumptions are violated.

4 Debiasing Residual Networks under Hidden Confounding

4.1 Methodology Overview

Different from the previous methods that use sensitivity analysis to control the worst-case IPS or DR loss caused by hidden confounding, we propose a unified multi-task learning approach with residual networks as in Figure 1, with the motivation of using a small unbiased data to calibrate the learned propensities and imputed errors in IPS or DR loss for training the unbiased prediction model.

Following causal inference literature [18, 19], we define the true propensity and true imputation as ¯ p u,i ≜ P ( o u,i = 1 | x u,i , h u,i ) , ¯ g u,i ≜ E ( δ u,i | x u,i , h u,i ) , both of them are functions of ( x u,i , h u,i ) , with ˜ p u,i and ˜ δ u,i as their estimates. To distinguish, we call p u,i = P ( o u,i = 1 | x u,i ) and g u,i = E [ δ u,i | x u,i ] the nominal propensity and nominal imputation , with ˆ p u,i and ˆ δ u,i as their estimates.

Next, we show the necessity of calibrations on both ˆ p u,i and ˆ δ u,i estimated from the baised data B .

Theorem 2 (Necessity of Calibration) . Suppose the partial derivative of ¯ p u,i and ¯ g u,i with respect to hidden confounders h u,i are not always equal to 0, and ˆ p u,i and ˆ δ u,i are consistent estimators of p u,i and g u,i , then there exists η > 0 , such that

̸

Proof. Given the partial derivative of ¯ p u,i with respect to hidden confounders h u,i is not always equal to 0, that is, h u,i has a non-zero effect on o u,i , so we have ¯ p u,i = p u,i according to their definitions. Thus, for some ϵ > 0 , there exist positive constants δ ϵ , N 1 ( ϵ ) > 0 , such that for all |D| > N 1 ( ϵ ) ,

Since ˆ p u,i is a consistent estimator of p u,i , there exists some N 2 ( ϵ ) > 0 , such that for all |D| > N 2 ( ϵ ) ,

Thus, if |D| > max { N 1 ( ϵ ) , N 2 ( ϵ ) } , we have

Let η = ϵ/ 2 and note that {| ¯ p u,i -p u,i | > ϵ, | ˆ p u,i -p u,i | < ϵ/ 2 } ⊂ {| ˆ p u,i -¯ p u,i | > η } , we have

which leads to lim |D|→∞ P ( | ˆ p u,i -¯ p u,i | > η ) > 0 . Similarly, it can be shown that lim |D|→∞ P ( | ˆ δ u,i -¯ g u,i | > η ) > 0 .

Theorem 2 shows that in the presence of hidden confounding, the estimated nominal propensities and nominal imputed errors deviate from the true one, even with the infinite sample size. To address this problem, as shown in Figure 1, we propose a novel consistency loss that utilizes unbiased data to calibrate the learned nominal propensities and imputed errors from the biased data.

(a) Res-IPS

(b) Res-DR

B

u

ConsistencyLoss

B

B

n

Consistency Loss

B

B

CE

CE

CTRLoSS

LCTR

Unbiased

IPS LoSS

CEVR

CTR LOSS

LCTR

Unbiased

DR LOoss LEVR

Imputation

LOSSLIMP

Res-Propensity

Res-Propensity

PuiEstimate

Pui Estimate

Qoui

pui

CVR

pui

CVR

Res-lmputed

Estimate

Estimate

Error

fui

fui

i△pui

i△pui

Adui

MLP

MLP

MLP

MLP

MLP

MLP

MLP

MLP

Res-Propensity

Prediction

Res-Propensity

Prediction

Res-Error

(CTR)Network

(CVR)Network

(CTR)Network

(CVR)Network

ImputationNetwork

Shared Network (MLP)

Shared Network (MLP)

↑

Input Covariates

Input Covariates

Specifically, we define the calibrated propensity model ˜ p = ˜ p ( θ CTR , ϕ CTR ) and the calibrated imputation model ˜ δ = ˜ δ ( θ IMP , ϕ IMP ) as follows

where σ is the sigmoid function, and the transformations are designed for numerical stability, e.g., to control range from 0 to 1. Compared with ˆ p u,i and ˆ δ u,i adopted in the previous multi-task learning approaches [34, 44, 59], the residual terms ∆ p u,i and ∆ δ u,i are further added to ˜ p u,i and ˜ δ u,i to capture the effect of hidden confounding. The loss function of the proposed approach is defined as

where α , β and γ are hyper-parameters for trade-off. The following states for each loss.

4.2 Nominal Propensities and Imputations Initialization

Similar to the previous IPS and DR methods [39, 41, 42, 52], as well as the multi-task learning approaches [34, 44, 59], we use the losses L CTR and L IMP in Section 3.1 for training the estimated nominal propensities ˆ p u,i and nominal imputed errors ˆ δ u,i from the biased dataset. However, since ˆ p u,i and ˆ δ u,i can only capture the effect of measured confounding, we do not directly use them for training the prediction model under hidden confounding. Instead, we train the prediction model using the calibrated propensities and imputed errors as illustrated in Section 4.3.

4.3 Debiased Prediction Model Training

Training on Biased Data. To train an unbiased prediction model, the losses need to be unbiased with respect to the ideal loss, thus the direct use of nominal ˆ p u,i and ˆ δ u,i under hidden confounding would lead to biased predictions. In contrast to previous studies, we propose to use both the initialized ˆ p u,i and ˆ δ u,i in Section 4.2, and the corresponding residual terms ∆ p u,i and ∆ δ u,i (see Section 4.4 for more details) using IPS (or DR, CTCVR loss) to achieve unbiased prediction model learning.

Specifically, using the calibrated propensity model, the proposed residual-IPS (Res-IPS) loss is

Similarly, using both the calibrated propensity model and the calibrated imputation model, the proposed residual-DR (Res-DR) loss is

Although CTCVR loss is not unbiased to the ideal loss [59], as previous studies have shown, using both IPS (or DR) loss and CTCVR loss empirically can lead to better debiasing performance by mitigating the data sparsity issue [44]. Similarly, using the calibrated propensity model, the proposed calibrated CTCVR loss is

Empirically, one can choose from the three calibrated losses for training a debiased prediction model.

Training on Unbiased Data. Since the unbiased data do not encounter any confounding problems, it provides a golden standard to evaluate and train the pCVR prediction model. The unbiased loss is

Different from the previous studies [4, 53], in which L U CVR is used to select the optimal propensity or imputation models, our approach is not necessary for training the prediction model using the unbiased loss (since one can use the aforementioned calibrated losses). In addition, the direct use of the unbiased loss can lead to severe overfitting, which once again demonstrates the importance of calibrating the propensity and imputation models in the IPS or DR loss.

4.4 Residual Networks Training

Collected through a carefully designed experiment, the unbiased data can be regarded as a representative sample of the entire space [53]. Thus, we always have L U CVR ≈ L ideal ( θ CVR ) , regardless of hidden confounding in biased data, which motivates us to propose a consistency loss

which measures the discrepancy between L B CVR and L U CVR , which equivalently provides us with an optimization direction for removing hidden confounding.

Proposition 3. If L B & U CVR = 0 , then L B CVR is an unbiased estimator of the ideal loss, regardless of whether hidden confounders exist or not.

Essentially, the consistency loss uses unbiased data to calibrate the debiasing loss L B CVR based on the biased data, thereby guaranteeing the debiasing ability of the proposed methods in the presence of hidden confounding. As discussed in Sections 4.1-4.3, we utilize the two residual terms ∆ p u,i and ∆ δ u,i to capture the effect of hidden confounding, which are trained by mimimizing L B & U CVR .

5 Real-World Experiments

Dataset and Pre-processing. Following the previous studies [4, 42, 52], we use three real-world datasets: COAT 2 , YAHOO! R3 3 , and KUAIREC 4 [9], for evaluating the debiasing performance of the proposed methods, where KUAIREC is a public large-scale industrial dataset. COAT contains 6,960 biased ratings from 290 users to 300 items, where each user picks 24 items to rate based on their personal preferences. Meanwhile, it also contains 4,640 unbiased ratings, where each user is asked to rate 16 randomly selected items. YAHOO! R3 contains 311,704 biased ratings and 54,000 unbiased ratings, where the unbiased ratings are from the first 5,400 users for 10 random selected items. We binarize ratings less than four to 0 and other ratings to 1 for the above two five-scale

datasets. KUAIREC is a fully exposed dataset containing 4,676,570 video watching ratio records from 1,411 users for 3,327 videos. The records less than two are binarized to 0 and other records to 1.

Baselines. In our experiment, we take the widely-used Matrix Factorization (MF) [22] as the base model. We compare our methods with the debiasing methods: IPS [41, 42], DR [39, 52], RD-IPS [8], RD-DR [8], and multi-task learning approaches: ESMM [34], Multi-IPS [59], Multi-DR [59], ESCM 2 -IPS [44] and ESCM 2 -DR [44]. We also compared the methods using both biased data and unbiased data: CausE [1], KD-Label [30], KD-Feature [30] and AutoDebias [4].

Experimental Protocols and Details. We adopt three widely-used evaluation metrics: AUC, Recall@K (R@K), and NDCG@K (N@K) for debiasing performance evaluation. We set K = 5 for COAT and YAHOO! R3, and K = 50 for KUAIREC. All the experiments are implemented on Pytorch with Adam as the optimizer. We tune learning rate in { 0 . 0001 , 0 . 0005 , 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 } , weight decay in { 0 , 1 e -6 , . . . , 1 e -1 , 1 } . For our methods, we tune α in { 0 . 1 , 0 . 5 , 1 } , β in { 0 . 1 , 0 . 5 , 1 , 5 , 10 } , and γ in { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 } . In addition, we randomly split 5% of unbiased data from the test set to train models for all methods that require unbiased data 5 .

Performance Comparison. Table 1 shows the real-world debiasing performance for varying methods on three datasets. First, most debiasing methods outperform the base model, i.e., MF (bias), demonstrating the necessity for debiasing in the presence of selection bias. Second, methods using both biased and unbiased data outperform the methods using only one of them, which indicates that there exists some non-overlap information that can benefit for debiasing between both biased and

unbiased data, and highlights the importance of leveraging both kinds of data. Meanwhile, the direct use of multi-task learning approaches to IPS and DR estimators cannot benefit the debiasing performance under hidden confounding. The proposed Res-IPS and Res-DR methods stably outperform previous methods on all three datasets, which provides empirical evidence of the existence of hidden confounding in the real-world recommendations, as well as the effectiveness of our methods for removing hidden confounding. Table 2 shows the results of AUC and NDCG@50 on KUAIREC with varying unbiased data ratios. The performance of all methods improves with increasing unbiased data ratio, and our method stably outperforms the baseline methods by a large margin.

Ablation Studies. The two losses L U CVR and L B & U CVR as well as the two residual terms ∆ δ and ∆ p are crucial in the proposed multi-task learning approach. We further conduct ablation studies with respect to the residual networks and the training loss components, respectively. From Table 3, Res-DR using either the propensity residual network or the imputation residual network can stably outperform ESCM 2 -DR, and Res-DR achieves the best performance when two residual networks are adopted together. When both propensity and imputation residual networks are removed, although Res-DR includes L U CVR and L B & U CVR losses, it has similar performance to ESCM 2 -DR, which further indicates that the performance improvement of Res-DR can be attributed to the effectiveness of the residual networks. From Table 4, both Res-IPS and Res-DR methods without L U CVR loss or without L B & U CVR loss outperform ESCM 2 -IPS and ESCM 2 -DR. Similarly, our methods achieve the best performance when both two losses are preserved. Note that the model with L B & U CVR loss performs better than the model with L U CVR loss, which is attributed to L B & U CVR corrects for biased learned propensities and biased imputed errors under the presence of hidden confounding. In addition, unbiased data does not significantly improve the debiasing performance when it is directly used to train the prediction model through L U CVR , due to the overfitting problem caused by the limited unbiased data size, which is consistent to the poor performance of MF (uniform) in Table 1. Meanwhile, minimizing L U CVR does not provide any residual information for biased learned propensities and biased imputed errors.

In-Depth Analysis. The proposed methods contain the prediction model, imputation model, propensity model, and residual models, thus it is meaningful to investigate the effect of different optimization algorithms among these models on the debiasing performance. Specifically, we implement the following learning approaches on the Res-DR: (1) Joint Learning (JL) [52], which joint optimizes the prediction model and the imputation model. (2) Double Learning (DL) [11], which adds a parameter

(a) R@5 on YAHOO! R3

(b) R@50 on KUAIREC

(c) N@5 on YAHOO! R3

(d) N@50 on KUAIREC

(a) AUC on YAHOO! R3

(b) AUC on KUAIREC

(c) N@5 on YAHOO! R3

(d) N@50 on KUAIREC

0.80

0.90

0.65

0.60

5

三

R

U

0.55

0.75

0.70

0.65

0.60

Res-DR-MTL

Res-DR-DL

Res-DR-JL

5

R

0

0.85

0.75

0.65

0.60

0.80

0.70

Res-DR-JL

Res-DR-MTL

Res-DR-DL

5

?

G

U

0.55

0.45

0.60

0.50

0.40

Res-DR-JL

Res-DR-MTL

Res-DR-DL

2

0

5

G

0.35

0.30

0.55

0.50

0.45

0.40

0.25

Res-DR-MTL

Res-DR-DL

Res-DR-JL

0.50

0

10

20

30

40

0.55

0

10

20

30

40

0.35

0

10

20

30

40

0.20

0

10

20

30

40

NumberofIterations

NumberofIterations

NumberofIterations

NumberofIterations

Res-DR

Res-IPS

Res-DR

Res-IPS

Res-DR

Res-IPS

Res-DR

Res-IPS

ESCM2-DR

ESCM2-IPS

ESCM2-DR

ESCM2-1PS

ESCM2-DR

ESCM2-1PS

ESCM2-DR

ESCM2-IPS

0.78

0.86

0.61

0.66

C

A

0.74

0.76

0.72

C

0.84

0.85

0.82

0.83

0.62

G

C

5

D

Z

0.58

0.59

0.58

?

C

0

Z

D

0.57

0.56

0.60

0.54

0.70

0

0.001 0.005

0.01

0.05

0.1

0.81

0

0.0010.005

0.01

0.05

0.1

0

0.0010.005

0.01

0.05

0.1

0.55

0

0.0010.005

0.01

0.05

0.1

Hyper-parametery

Hyper-parametery

Hyper-parametery

Hyper-parametery

sharing mechanism between the prediction model and the imputation model based on JL. (3) MultiTask Learning (MTL) adopted in our methods. Figure 2 shows the experiment results. Remarkably, MTL significantly outperforms JL and DL, whereas in Table 1 DR and MTL-based ESCM 2 -DR perform similarly. This is because DR only has three training models (namely the prediction model, imputation model, and propensity model), whereas the proposed Res-DR has two additional residual models. As the number of models increases, JL can no longer bridge the models efficiently, which leads to a slow convergence and limited performance. Instead, DL increases the connection between models via timely parameter sharing. Finally, MTL trade-offs the different tasks in Res-DR to train all the models, leading to the optimal debiasing performance.

Sensitivity Analysis. We perform the sensitivity analysis of γ on Res-IPS and Res-DR, as shown in Figure 3. Our methods achieve the optimal performance when γ is moderate (0.005-0.01). This is because when γ is too large, it hurts the performance of other tasks (e.g., CVR model training), and when γ is too small, it makes the consistency loss be paid with less attention, so that the hidden confounding cannot be effectively removed. Res-IPS and Res-DR stably outperform ESCM 2 -IPS and ESCM 2 -DR under varying γ . This further illustrates the effectiveness the consistency loss.

6 Conclusion

This paper investigates the use of a few unbiased ratings to calibrate the learned propensities and imputed errors for removing hidden confounding. First, we theoretically reveal the biasedness of previous debiasing methods in the presence of hidden confounding. Next, we propose a multi-task debiasing residual networks learning approach for training the debiased prediction model. By building residual networks and calibrating the biased learned propensities and biased imputed errors, the prediction model is trained on both the calibrated IPS or DR losses and the unbiased dataset to ensure the unbiasedness. Extensive experiments on two benchmark datasets and a large-scale industrial dataset validate the effectiveness of our proposal. A limitation of this work is the use of slightly more model parameters due to the need to address hidden confounding with residual networks.

References

Stephen Bonner and Flavian Vasile. Causal embeddings for recommendation. In RecSys , 2018.

Yin-Wen Chang, Cho-Jui Hsieh, Kai-Wei Chang, Michael Ringgaard, and Chih-Jen Lin. Training and testing low-degree polynomial data mappings via linear svm. Journal of Machine Learning Research , 11(4), 2010.

Jiawei Chen, Can Wang, Martin Ester, Qihao Shi, Yan Feng, and Chun Chen. Social recommendation with missing not at random data. In ICDM , 2018.

Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. AutoDebias: Learning to debias for recommendation. In SIGIR , 2021.

Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems , 41(3):1-39, 2023.

HChristopher Frey and Sumeet R Patil. Identification and review of sensitivity analysis methods. Risk analysis , 22(3):553-578, 2002.

Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Xiuqiang He, Rui Zhang, and Jie Sun. A generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In KDD , 2022.

Sihao Ding, Peng Wu, Fuli Feng, Xiangnan He, Yitong Wang, Yong Liao, and Yongdong Zhang. Addressing unmeasured confounder for recommendation with sensitivity analysis. In KDD , 2022.

Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recommender systems. In CIKM , 2022.

Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. Offline a/b testing for recommender systems. In WSDM , 2018.

Siyuan Guo, Lixin Zou, Yiding Liu, Wenwen Ye, Suqi Cheng, Shuaiqiang Wang, Hechang Chen, Dawei Yin, and Yi Chang. Enhanced doubly robust learning for debiasing post-click conversion rate estimation. In SIGIR , 2021.

Miguel A. Hernán and James M. Robins. Causal Inference: What If . Boca Raton: Chapman and Hall/CRC, 2020.

Wenjie Hu, Xiao-Hua Zhou, and Peng Wu. Identification and estimation of treatment effects on long-term outcomes in clinical trials with external observational data. Statistica Sinica , 2023.

Shanshan Huang, Haoxuan Li, Qingsong Li, Chunyuan Zheng, and Li Liu. Pareto invariant representation learning for multimedia recommendation. In ACM-MM , 2023.

Paul Hünermund and Elias Bareinboim. Causal inference and data fusion in econometrics. arXiv:1912.09104 , 2019.

Dietmar Jannach, Lukas Lerche, and Markus Zanker. Recommending based on implicit feedback. In Social Information Access , pages 510-569. Springer, 2018.

Simon Jenni and Paolo Favaro. Deep bilevel learning. In ECCV , 2018.

Nathan Kallus and Angela Zhou. Confounding-robust policy improvement. In NeurIPS , pages 9289-9299, 2018.

Nathan Kallus and Angela Zhou. Minimax-optimal policy learning under unobserved confounding. Management Science , 67(5):2870-2890, 2021.

Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experimental grounding. In NeurIPS , 2018.

Ron Kohavi and Roger Longbotham. Online controlled experiments and a/b testing. Encyclopedia of Machine Learning and Data Mining , 2017.

Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer , 42(8):30-37, 2009.

Haoxuan Li, Quanyu Dai, Yuru Li, Yan Lyu, Zhenhua Dong, Xiao-Hua Zhou, and Peng Wu. Multiple robust learning for recommendation. In AAAI , 2023.

Haoxuan Li, Taojun Hu, Zetong Xiong, Chunyuan Zheng, Fuli Feng, Xiangnan He, and XiaoHua Zhou. ADRNet: A generalized collaborative filtering framework combining clinical and non-clinical data for adverse drug reaction prediction. In RecSys , 2023.

Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. TDR-CL: Targeted doubly robust collaborative learning for debiased recommendations. In ICLR , 2023.

Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confounding with a few unbiased ratings in debiased recommendations. In WWW , 2023.

Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Measuring and enhancing balancing for recommendation. In ICML , 2023.

Haoxuan Li, Chunyuan Zheng, and Peng Wu. StableDR: Stabilized doubly robust learning for recommendation on data missing not at random. In ICLR , 2023.

Haoxuan Li, Chunyuan Zheng, Peng Wu, Kun Kuang, Yue Liu, and Peng Cui. Who should be given incentives? counterfactual optimal treatment regimes learning for recommendation. In KDD , 2023.

Dugang Liu, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. A general knowledge distillation framework for counterfactual recommendation via uniform data. In SIGIR , 2020.

Dugang Liu, Pengxiang Cheng, Hong Zhu, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. Mitigating confounding bias in recommendation via information bottleneck. In RecSys , 2021.

Dugang Liu, Pengxiang Cheng, Zinan Lin, Jinwei Luo, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. KDCRec: Knowledge distillation for counterfactual recommendation via uniform data. IEEE Transactions on Knowledge and Data Engineering , 2022.

Jinwei Luo, Dugang Liu, Weike Pan, and Zhong Ming. Unbiased recommendation model based on improved propensity score estimation. Journal of Computer Applications , 2021.

Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire space multi-task model: An effective approach for estimating post-click conversion rate. In SIGIR , 2018.

Xuying Meng, Suhang Wang, Kai Shu, Jundong Li, Bo Chen, Huan Liu, and Yujun Zhang. Personalized privacy-preserving social recommendation. In AAAI , 2018.

Judea Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, second edition, 2009.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML , 2018.

Paul R Rosenbaum, P Rosenbaum, and Briskman. Design of observational studies , volume 10. Springer, 2010.

Yuta Saito. Doubly robust estimator for ranking metrics with post-click conversions. In RecSys , pages 92-100, 2020.

Yuta Saito and Masahiro Nomura. Towards resolving propensity contradiction in offline recommender learning. In IJCAI , 2022.

Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing-not-at-random implicit feedback. In WSDM , 2020.

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In ICML , 2016.

Zijie Song, Jiawei Chen, Sheng Zhou, Qihao Shi, Yan Feng, Chun Chen, and Can Wang. CDR: Conservative doubly robust learning for debiased recommendation. In CIKM , 2023.

Hao Wang, Tai-Wei Chang, Tianqiao Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. ESCM2: Entire space counterfactual multi-task model for post-click conversion rate estimation. In SIGIR , 2022.

Hao Wang, Zhichao Chen, Jiajun Fan, Haoxuan Li, Weiming Liu, Tianqiao Liu, Quanyu Dai, Yichao Wang, Zhenhua Dong, and Ruiming Tang. Optimal transport for treatment effect estimation. In NeurIPS , 2023.

Haotian Wang, Wenjing Yang, Longqi Yang, Anpeng Wu, Liyang Xu, Jing Ren, Fei Wu, and Kun Kuang. Estimating individualized causal effect with confounded instruments. In KDD , 2022.

Haotian Wang, Kun Kuang, Haoang Chi, Longqi Yang, Mingyang Geng, Wanrong Huang, and Wenjing Yang. Treatment effect estimation with adjustment feature selection. In KDD , 2023.

Haotian Wang, Kun Kuang, Long Lan, Zige Wang, Wanrong Huang, Fei Wu, and Wenjing Yang. Out-of-distribution generalization with causal feature separation. IEEE Transactions on Knowledge and Data Engineering , 2023.

Jun Wang, Haoxuan Li, Chi Zhang, Dongxu Liang, Enyun Yu, Wenwu Ou, and Wenjia Wang. Counterclr: Counterfactual contrastive learning with non-random missing data in recommendation. In ICDM , 2023.

Qinyong Wang, Hongzhi Yin, Tong Chen, Junliang Yu, Alexander Zhou, and Xiangliang Zhang. Fast-adapting and privacy-preserving federated recommender system. The VLDB Journal , pages 1-20, 2021.

Wenjie Wang, Yang Zhang, Haoxuan Li, Peng Wu, Fuli Feng, and Xiangnan He. Causal recommendation: Progresses and future directions. In Tutorial on SIGIR , 2023.

Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recommendation on data missing not at random. In ICML , 2019.

Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Combating selection biases in recommender systems with a few unbiased ratings. In WSDM , 2021.

Zimu Wang, Yue He, Jiashuo Liu, Wenchao Zou, Philip S Yu, and Peng Cui. Invariant preference learning for general debiasing in recommendation. In KDD , 2022.

Peng Wu, Shasha Han, Xingwei Tong, and Runze Li. Propensity score regression for causal inference with treatment heterogeneity. Statistica Sinica , 2022.

Peng Wu, Haoxuan Li, Yuhao Deng, Wenjie Hu, Quanyu Dai, Zhenhua Dong, Jie Sun, Rui Zhang, and Xiao-Hua Zhou. On the opportunity of causal learning in recommendation systems: Foundation, estimation, prediction and challenges. In IJCAI , 2022.

Peng Wu, Zhiqiang Tan, Wenjie Hu, and Xiao-Hua Zhou. Model-assisted inference for covariatespecific treatment effects with high-dimensional data. Statistica Sinica , 2022.

Shu Yang and Peng Ding. Combining multiple observational data sources to estimate causal effects. Journal of the American Statistical Association , 115:1540-1554, 2020.

Wenhao Zhang, Wentian Bao, Xiao-Yang Liu, Keping Yang, Quan Lin, Hong Wen, and Ramin Ramezani. Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning. In WWW , 2020.

Hao Zou, Haotian Wang, Renzhe Xu, Bo Li, Jian Pei, Ye Jun Jian, and Peng Cui. Factual observation based heterogeneity learning for counterfactual prediction. In 2nd Conference on Causal Learning and Reasoning , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses that there are hidden confounders in recommendations and existing methods cannot handle them well. \\\nThe paper proposes a unified multi-task learning approach to tackle that problem. \\\nSpecifically, they devise a residual network to calibrate the propensity and the imputed error by using unbiased data.\n\nSTRENGTHS:\n1. The paper is well-organized and presented.\n\n2. Each loss function is technically solid.\n- The proposed method is end-to-end multi-task learning with a debiasing residual network that simultaneously deals with selection bias, data sparsity, and hidden confounding.\n\n3. The proposed method is validated through experiments.\n- They adopt three real-world datasets, three metrics, and extensive baselines.\n\nWEAKNESSES:\n1. Motivation is weak.\n- Hidden confounders are assumed to exist. There is no theoretical or experimental evidence for that.\n- Proofs (Proposition 1, Theorem 2) are naive. They are proved on the assumption that is already the conclusion of the proofs.\n\n2. Proposed end-to-end framework is not novel. \n- There is already an end-to-end multi-task learning framework [7] removing hidden confounders with biased and unbiased data.\n- Loss functions are from the existing methods. Most of them are already adopted in a multi-task manner in ESCM^2 [33].\n- The unique contribution of this paper relies on unbiased data, which is hard to obtain in real-world applications.\n\n3. Missing related work\n- Balancing Unobserved Confounding with a Few Unbiased Rating in Debiased Recommendations, WWW 2023.\n- StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not At Random. ICLR 2023.\n- Multiple Robust Learning for Recommendation. AAAI 2023.\n- This work seems to tackle the same problem as the proposed work and needs to be included in the manuscript.\n\n4. Notations are confusing\n- There are no explicit loss forms. For example, what is $L^{B}_{CVR}$?\n- The argument of loss functions is not organized. I cannot distinguish which is the training parameter and which is the fixed parameter.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nThere are many hyperparameters for the loss functions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a critical examination of prevalent debiasing methods in recommendation systems and their limitations in addressing hidden confounding factors. The authors underline that current methods—propensity-based, multi-task learning, and bi-level optimization—fail to mitigate selection bias when unobserved confounding variables exist. In response, they propose a unified multi-task learning approach that incorporates a small set of unbiased ratings to calibrate nominal propensities and error imputations, thereby reducing the influence of hidden confounding factors. The approach utilizes a newly introduced consistency loss for calibration, advancing the field's theoretical understanding of bias and confounding in recommendation systems. The paper further validates its method through comprehensive experiments on three public benchmark datasets, including a large-scale industrial dataset. The outcomes affirm the efficacy of their approach in countering selection bias and hidden confounding. The study hence significantly contributes to the realm of recommender systems, revealing theoretical shortcomings in current debiasing approaches, and offering a novel, robust method for reducing bias in practice.\n\nSTRENGTHS:\n1. Proposed methods show quite significant improvements over the strong baselines.\n2. Good to interpret each term in the equations. Very helpful for readers to intuitively understand the equations.\n3. Code is attached, to be published with the paper.\n4. Extended and complete experiments conducted on real-world datasets including a large-scale industrial dataset with many baselines and ablation studies.\n\nWEAKNESSES:\n1. Reference numbers for equations are missing. Please add the reference numbers to make referring more easily.\n2. Where is Theorem 1? Or why not Lemma 1, Proposition 2, and Theorem 3?\n3. Please refer to the proofs in the Appendix in the main text when the corresponding theorem or lemma are given.\n4. Equation of L_{CVR}^{B&U} in Sect 4.4 is not very well-defined. The cross-entropy loss \\delta(., .) is defined for binary random variables, however, neither L_{CVR}^B or L_{CVR}^U is a binary random variable. Please spell out the specific definition, as it’s also one of the main contribution.\n5. Statistical significance analysis is missing.\n\nQUESTIONS:\n1. How good will the method be useful on real-world products? Assuming it is possible to get unbiased labels from evaluators, but usually at a much much smaller scale (<0.1%), and sparser (no consistent history of a user).\n\nLIMITATIONS:\n1. What are the assumptions about the hidden confounding? The main new term introduced L_{CVR}^{B&U} is not sufficiently discussed. If I understand correctly, the main assumptions are that the unbiased set has no hidden confounding and aligning the distribution of prediction on the biased set to that of the unbiased set should calibrate / de-confound the predictions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper highlights the prevalent issue of selection bias in recommender systems, emphasizing the often-overlooked aspect of hidden confounding. Existing approaches and their limitations are discussed, with a special focus on hidden confounders. The authors then introduce a unified multi-task learning approach which utilizes a small set of unbiased ratings to calibrate nominal propensities and error imputations. This approach aims to handle hidden confounding, thereby achieving unbiased learning.\n\nSTRENGTHS:\n- The paper does an excellent job of dissecting existing propensity-based and multi-task learning methods theoretically to demonstrate how they can lead to biased learning in the presence of hidden confounding.\n\n- Although the multi-task learning approach for debiasing is not novel, the specific setting of calibrations on unbiased data is novel. It uniquely uses a few unbiased ratings to calibrate learned propensities and imputed errors from biased data, aiming to eliminate the biases caused by hidden confounding. The unified loss is also justified by theoretical analysis. \n\n- The proposed methods are extensively validated on three publicly available benchmark datasets, including a large-scale industrial dataset. This broad empirical evidence strengthens the claims made in the paper.\n\nWEAKNESSES:\n- The theoretical result is weak. It states that if the consistency loss is zero then the calibrated loss is unbiased. While the result does help justify the design of the loss function, it does not provide any guarantee regarding the unbiasedness of the learned model. The consistency loss might not reach zero or even be small enough to reduce the bias of the calibrated loss.\n\n- The overall model contains multiple loss components and multiple parameters. While the authors point out the potential issues of the existing minimax framework, the proposed solution requires more parameter tuning.\n\nQUESTIONS:\nIn Figure 2, the higher weight of the consistency loss actually reduces the performance of the model which is not very intuitive. Based on the propositions, reduced consistency loss leads to unbiased calibration loss. With higher weight, we expect smaller consistency loss and in general a better calibration loss. Why do we observe a regression in the AUC?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies unbiased learning in recommendation systems in the presence of hidden confounding. The authors first theoretically analyze the limitations of previous MTL methods and those combine some unbiased data and then design a unified MTL debiasing method by calibrating the learned nominal propensities and error imputations using a novel consistency loss. Extensive experiments on benchmark datasets have shown the effectiveness of their proposed method.

STRENGTHS:
1. The work is well-motivated with the analysis of the limitations of existing work in the presence of hidden confounding. 
2. The presentation of the method design is clear and vivid. Each part of the optimization goal is stated in a good structure.
3. Extensive experiments of comparison with various baselines and in-depth analysis are persuasive, which shows the effectiveness of the designed method and the function of each component.

WEAKNESSES:
The novelty of this work lies in the proposed consistency loss that utilizes unbiased data to calibrate the learned nominal propensities and imputed errors from the biased data. However, the theoretical analysis of this term seems trivial (Proposition 3). Maybe the trade-off between different terms in $\mathcal{L}_{Res}$ can be discussed.

QUESTIONS:
How do you tune the hyperparameters $\alpha, \beta, \gamma$? Do you use a validation set? If so, how do other baselines use the validation set?

LIMITATIONS:
The authors have a discussion of the limitations in the Conclusion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that existing approaches that are based on multi-task learning or take advantage of unbiased data have theoretical limitations in the problem of recommendation debiasing when hidden confounding is present. The paper proposes to address these limitations by a unified multi-task learning approach whose key idea is to remove hidden confounding by calibrating learned nominal propensities and nominal error imputations by a consistency loss on unbiased data. The paper conducts extensive experiments to confirm the effectiveness of the proposed approach on 3 widely used recommendation datasets.\n\nSTRENGTHS:\nS1: The idea of tackling hidden confounding by a multi-task learning approach using unbiased data to calibrate learned nominal propensities and nominal error imputations is novel to me.\n\nS2: The paper compares the proposed approach against a wide range of existing representative approaches on 3 benchmark recommendation datasets under commonly used evaluation metrics.\n\nS3: The paper does a great job in reviewing major research directions in the problem of recommendation debiasing and the references cover a full spectrum of related works.\n\nWEAKNESSES:\nW1: It is not fair to claim that the paper is the first to \"perform theoretical analysis to reveal the possible failure of previous approaches\" as Ding et al. has already done a majority of such theoretical analysis [7], which is cited by the paper.\n\nW2: The proofs for the theoretical results presented in the paper are a bit hand waved. Take Theorem 2 as an example. It is not unclear what is the space spanned by x and why the fact that \\bar{p} will not degenerate a point in the space spanned by x leads to the existence of a positive constant \\eta that satisfies the condition.\n\n[7] Addressing unmeasured confounder for recommendation with sensitivity analysis\n\nQUESTIONS:\nQ1: Do you have any explanations why the proposed approach underperforms RD-DR [7] in terms of R@50 on the KUAIREC dataset?\n\nQ2: Is N@K in Tables 1 and 3 NDCG@K? If so, it is important to clarify the meaning of N@5 (and R@5) in the main text as it is not common to abbreviate NDCG@K as N@5. \n\n[7] Addressing unmeasured confounder for recommendation with sensitivity analysis\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-15"}