SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, we define the calibrated propensity model  $\tilde{p} = \tilde{p} (\theta_{\mathrm{CTR}},\phi_{\mathrm{CTR}})$  and the calibrated imputation model  $\tilde{\delta} = \tilde{\delta} (\theta_{\mathrm{IMP}},\phi_{\mathrm{IMP}})$  as follows
$$
\tilde {p} _ {u, i} = \sigma \left(\sigma^ {- 1} (\hat {p} _ {u, i} (\theta_ {\mathrm {C T R}})) + \sigma^ {- 1} (\Delta p _ {u, i} (\phi_ {\mathrm {C T R}}))\right),
$$
$$
\tilde {\delta} _ {u, i} = \sigma \left(\sigma^ {- 1} \left(\hat {\delta} _ {u, i} \left(\theta_ {\mathrm {I M P}}\right)\right) + \sigma^ {- 1} \left(\Delta \delta_ {u, i} \left(\phi_ {\mathrm {I M P}}\right)\right)\right),
$$
where  $\sigma$  is the sigmoid function, and the transformations are designed for numerical stability, e.g., to control range from 0 to 1. Compared with  $\hat{p}_{u,i}$  and  $\hat{\delta}_{u,i}$  adopted in the previous multi-task learning approaches [34, 44, 59], the residual terms  $\Delta p_{u,i}$  and  $\Delta \delta_{u,i}$  are further added to  $\tilde{p}_{u,i}$  and  $\tilde{\delta}_{u,i}$  to capture the effect of hidden confounding. The loss function of the proposed approach is defined as
$$
\mathcal {L} _ {\mathrm {R e s}} = \underbrace {\mathcal {L} _ {\mathrm {C T R}} (\hat {p}) + \alpha \cdot \mathcal {L} _ {\mathrm {I M P}} (\hat {\delta})} _ {\text {I n i t i a l i z a t i o n u s i n g b i a s e d d a t a}} + \beta \cdot \underbrace {\left(\mathcal {L} _ {\mathrm {C V R}} ^ {\mathcal {B}} (\tilde {p} , \tilde {\delta}) + \mathcal {L} _ {\mathrm {C T C V R}} ^ {\mathcal {B}} (\tilde {p}) + \mathcal {L} _ {\mathrm {C V R}} ^ {\mathcal {U}}\right)} _ {\text {P r e d i c t i o n m o d e l t r a i n i n g w i t h c a l i b r a t e d l o s s e s}} + \gamma \cdot \underbrace {\mathcal {L} _ {\mathrm {C V R}} ^ {\mathcal {B} \& \mathcal {U}} (\hat {p} , \tilde {p} , \hat {\delta} , \hat {\delta})} _ {\text {C a l i b r a t i o n o n} \hat {p} \text {a n d} \hat {\delta}},
$$
where  $\alpha, \beta$  and  $\gamma$  are hyper-parameters for trade-off. The following states for each loss.
4.2 Nominal Propensities and Imputations Initialization
Similar to the previous IPS and DR methods [39, 41, 42, 52], as well as the multi-task learning approaches [34, 44, 59], we use the losses  $\mathcal{L}_{\mathrm{CTR}}$  and  $\mathcal{L}_{\mathrm{IMP}}$  in Section 3.1 for training the estimated nominal propensities  $\hat{p}_{u,i}$  and nominal imputed errors  $\hat{\delta}_{u,i}$  from the biased dataset. However, since  $\hat{p}_{u,i}$  and  $\hat{\delta}_{u,i}$  can only capture the effect of measured confounding, we do not directly use them for training the prediction model under hidden confounding. Instead, we train the prediction model using the calibrated propensities and imputed errors as illustrated in Section 4.3.
4.3 Debiased Prediction Model Training
Training on Biased Data. To train an unbiased prediction model, the losses need to be unbiased with respect to the ideal loss, thus the direct use of nominal  $\hat{p}_{u,i}$  and  $\hat{\delta}_{u,i}$  under hidden confounding would lead to biased predictions. In contrast to previous studies, we propose to use both the initialized  $\hat{p}_{u,i}$  and  $\hat{\delta}_{u,i}$  in Section 4.2, and the corresponding residual terms  $\Delta p_{u,i}$  and  $\Delta \delta_{u,i}$  (see Section 4.4 for more details) using IPS (or DR, CTCVR loss) to achieve unbiased prediction model learning.
Specifically, using the calibrated propensity model, the proposed residual-IPS (Res-IPS) loss is
$$
\mathcal {L} _ {\mathrm {I P S}} ^ {\mathcal {B}} \left(\theta_ {\mathrm {C V R}} \mid \hat {p} _ {u, i} \left(\theta_ {\mathrm {C T R}}\right), \Delta p _ {u, i} \left(\phi_ {\mathrm {C T R}}\right)\right) = \frac {1}{| \mathcal {D} |} \sum_ {(u, i) \in \mathcal {D}} \frac {o _ {u , i} \cdot \delta_ {u , i}}{\tilde {p} _ {u , i}}.
$$
Similarly, using both the calibrated propensity model and the calibrated imputation model, the proposed residual-DR (Res-DR) loss is
$$
\mathcal {L} _ {\mathrm {D R}} ^ {\mathcal {B}} \left(\theta_ {\mathrm {C V R}} \mid \hat {p} _ {u, i} \left(\theta_ {\mathrm {C T R}}\right), \Delta p _ {u, i} \left(\phi_ {\mathrm {C T R}}\right), \hat {\delta} _ {u, i} \left(\theta_ {\mathrm {I M P}}\right), \Delta \delta_ {u, i} \left(\phi_ {\mathrm {I M P}}\right)\right) = \frac {1}{| \mathcal {D} |} \sum_ {(u, i) \in \mathcal {D}} \left[ \tilde {\delta} _ {u, i} + \frac {o _ {u , i} \left(\delta_ {u , i} - \tilde {\delta} _ {u , i}\right)}{\tilde {p} _ {u , i}} \right].
$$
Although CTCVR loss is not unbiased to the ideal loss [59], as previous studies have shown, using both IPS (or DR) loss and CTCVR loss empirically can lead to better debiasing performance by mitigating the data sparsity issue [44]. Similarly, using the calibrated propensity model, the proposed calibrated CTCVR loss is
$$
\mathcal {L} _ {\mathrm {C T C V R}} (\theta_ {\mathrm {C V R}} \mid \hat {p} _ {u, i} (\theta_ {\mathrm {C T R}}), \Delta p _ {u, i} (\phi_ {\mathrm {C T R}})) = \frac {1}{| \mathcal {D} |} \sum_ {(u, i) \in \mathcal {D}} \delta (o _ {u, i} r _ {u, i}, \tilde {p} _ {u, i} \hat {r} _ {u, i}).
$$
Empirically, one can choose from the three calibrated losses for training a debiased prediction model.
Training on Unbiased Data. Since the unbiased data do not encounter any confounding problems, it provides a golden standard to evaluate and train the pCVR prediction model. The unbiased loss is
$$
\mathcal {L} _ {\mathrm {C V R}} ^ {\mathcal {U}} \left(\theta_ {\mathrm {C V R}}\right) = \frac {1}{| \mathcal {U} |} \sum_ {(u, i) \in \mathcal {U}} \delta \left(r _ {u, i}, \hat {r} _ {u, i} \left(\theta_ {\mathrm {C V R}}\right)\right).
$$
Different from the previous studies [4, 53], in which  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}}$  is used to select the optimal propensity or imputation models, our approach is not necessary for training the prediction model using the unbiased loss (since one can use the aforementioned calibrated losses). In addition, the direct use of the unbiased loss can lead to severe overfitting, which once again demonstrates the importance of calibrating the propensity and imputation models in the IPS or DR loss.
4.4 Residual Networks Training
Collected through a carefully designed experiment, the unbiased data can be regarded as a representative sample of the entire space [53]. Thus, we always have  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}} \approx \mathcal{L}_{\mathrm{iddeal}}(\theta_{\mathrm{CVR}})$ , regardless of hidden confounding in biased data, which motivates us to propose a consistency loss
$$
\mathcal {L} _ {\mathrm {C V R}} ^ {\mathcal {B} \& \mathcal {U}} \left(\theta_ {\mathrm {C T R}}, \phi_ {\mathrm {C T R}}, \theta_ {\mathrm {C V R}}, \theta_ {\mathrm {I M P}}, \phi_ {\mathrm {I M P}}\right) = \delta \left(\mathcal {L} _ {\mathrm {C V R}} ^ {\mathcal {B}}, \mathcal {L} _ {\mathrm {C V R}} ^ {\mathcal {U}}\right),
$$
which measures the discrepancy between  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{B}}$  and  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}}$ , which equivalently provides us with an optimization direction for removing hidden confounding.
Proposition 3. If  $\mathcal{L}_{CVR}^{\mathcal{B}\& \mathcal{U}} = 0$ , then  $\mathcal{L}_{CVR}^{\mathcal{B}}$  is an unbiased estimator of the ideal loss, regardless of whether hidden confounders exist or not.
Essentially, the consistency loss uses unbiased data to calibrate the debiasing loss  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{B}}$  based on the biased data, thereby guaranteeing the debiasing ability of the proposed methods in the presence of hidden confounding. As discussed in Sections 4.1-4.3, we utilize the two residual terms  $\Delta p_{u,i}$  and  $\Delta \delta_{u,i}$  to capture the effect of hidden confounding, which are trained by minimizing  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{B}\& \mathcal{U}}$ .
5 Real-World Experiments
Dataset and Pre-processing. Following the previous studies [4, 42, 52], we use three real-world datasets:  $\mathrm{COAT}^2$ , YAHOO!  $\mathbb{R}3^3$ , and KUAIREC[9], for evaluating the debiasing performance of the proposed methods, where KUAIREC is a public large-scale industrial dataset. COAT contains 6,960 biased ratings from 290 users to 300 items, where each user picks 24 items to rate based on their personal preferences. Meanwhile, it also contains 4,640 unbiased ratings, where each user is asked to rate 16 randomly selected items. YAHOO! R3 contains 311,704 biased ratings and 54,000 unbiased ratings, where the unbiased ratings are from the first 5,400 users for 10 random selected items. We binarize ratings less than four to 0 and other ratings to 1 for the above two five-scale
$^{2}$ https://www.cs.cornell.edu/~schnabts/mnar/
<sup>3</sup>http://webscope.sandbox.yahoo.com/
<sup>4</sup>https://github.com/chongminggao/KuaiRec
Table 1: Performance in terms of AUC, NDCG@K, and Recall@K on the unbiased dataset of COAT, YAHOO! R3 and KUAIREC. The best two results are bolded, and the best baseline is underlined.
COAT	YAHOO! R3	KUAIREC
Method	AUC	N@5	R@5	AUC	N@5	R@5	AUC	N@50	R@50
MF [22] (Bias)	0.747	0.500	0.546	0.721	0.553	0.716	0.820	0.561	0.816
MF [22] (Uniform)	0.580	0.363	0.386	0.574	0.455	0.611	0.664	0.491	0.816
MF [22] (Combine)	0.751	0.504	0.546	0.724	0.558	0.717	0.822	0.566	0.812
CausE [1]	0.763	0.512	0.575	0.730	0.555	0.736	0.819	0.581	0.856
ESMM [34]	0.745	0.506	0.525	0.708	0.545	0.693	0.823	0.563	0.852
KD-Label [30]	0.760	0.509	0.562	0.726	0.583	0.752	0.815	0.570	0.858
AutoDebias [4]	0.762	0.540	0.580	0.735	0.632	0.785	0.818	0.584	0.866
KD-Feature [30]	0.766	0.522	0.584	0.717	0.557	0.736	0.809	0.588	0.873
IPS [42]	0.761	0.513	0.566	0.722	0.555	0.733	0.826	0.574	0.849
Multi-IPS [59]	0.758	0.514	0.531	0.719	0.546	0.710	0.810	0.554	0.875
ESCM2-IPS [44]	0.757	0.514	0.558	0.729	0.559	0.714	0.815	0.577	0.860
RD-IPS [8]	0.764	0.514	0.566	0.730	0.571	0.735	0.832	0.585	0.873
BRD-IPS [8]	0.763	0.511	0.564	0.735	0.582	0.743	0.834	0.588	0.877
Res-IPS (ours)	0.777	0.575	0.601	0.759	0.639	0.785	0.849	0.601	0.885
DR [52]	0.766	0.525	0.552	0.725	0.553	0.727	0.824	0.567	0.838
Multi-DR [59]	0.759	0.527	0.565	0.719	0.553	0.712	0.829	0.562	0.859
ESCM2-DR [44]	0.760	0.553	0.568	0.715	0.566	0.722	0.827	0.569	0.830
RD-DR [8]	0.768	0.539	0.571	0.732	0.569	0.738	0.833	0.585	0.884
BRD-DR [8]	0.770	0.546	0.577	0.735	0.576	0.737	0.831	0.585	0.883
Res-DR (ours)	0.793	0.588	0.607	0.750	0.654	0.803	0.854	0.595	0.860
Table 2: Effects of varying unbiased data ratio on KUAIREC in terms of AUC and NDCG@50. The best two results are bolded, and the best baseline is underlined.
AUC ↑	NDCG@50 ↑
Unbiased data ratio	2%	4%	6%	8%	10%	2%	4%	6%	8%	10%
CausE	0.818	0.818	0.819	0.819	0.819	0.579	0.580	0.584	0.586	0.587
KD-Label	0.815	0.815	0.815	0.816	0.816	0.582	0.584	0.588	0.588	0.589
AutoDebias	0.810	0.815	0.818	0.826	0.832	0.569	0.580	0.587	0.589	0.590
Res-IPS (ours)	0.845	0.848	0.850	0.850	0.852	0.595	0.596	0.602	0.602	0.603
Res-DR (ours)	0.850	0.851	0.854	0.855	0.855	0.592	0.593	0.597	0.605	0.606
datasets. KUAIREC is a fully exposed dataset containing 4,676,570 video watching ratio records from 1,411 users for 3,327 videos. The records less than two are binarized to 0 and other records to 1.
Baselines. In our experiment, we take the widely-used Matrix Factorization (MF) [22] as the base model. We compare our methods with the debiasing methods: IPS [41, 42], DR [39, 52], RD-IPS [8], RD-DR [8], and multi-task learning approaches: ESMM [34], Multi-IPS [59], Multi-DR [59],  $\mathbf{ESCM}^2$ -IPS [44] and  $\mathbf{ESCM}^2$ -DR [44]. We also compared the methods using both biased data and unbiased data: CausE [1], KD-Label [30], KD-Feature [30] and AutoDebias [4].
Experimental Protocols and Details. We adopt three widely-used evaluation metrics: AUC, Recall@K (R@K), and NDCG@K (N@K) for debiasing performance evaluation. We set K = 5 for COAT and YAHOO! R3, and K = 50 for KUAIREC. All the experiments are implemented on Pytorch with Adam as the optimizer. We tune learning rate in {0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05}, weight decay in {0, 1e - 6, ..., 1e - 1, 1}. For our methods, we tune  $\alpha$  in {0.1, 0.5, 1},  $\beta$  in {0.1, 0.5, 1, 5, 10}, and  $\gamma$  in {0.001, 0.005, 0.01, 0.05, 0.1}. In addition, we randomly split  $5\%$  of unbiased data from the test set to train models for all methods that require unbiased data<sup>5</sup>.
Performance Comparison. Table 1 shows the real-world debiasing performance for varying methods on three datasets. First, most debiasing methods outperform the base model, i.e., MF (bias), demonstrating the necessity for debiasing in the presence of selection bias. Second, methods using both biased and unbiased data outperform the methods using only one of them, which indicates that there exists some non-overlap information that can benefit for debiasing between both biased and
For all experiments, we use Tesla T4 GPU as the computational resource.
Table 3: Ablation study on residual networks in Res-DR method, with AUC, NDCG@K and Recall@K as evaluation metrics. The best result is bolded and the second is underlined.
Training loss	COAT	YAHOO! R3	KUAIREC
Method	Δδ	Δp	AUC	R@5	N@5	AUC	N@5	R@5	AUC	N@50	R@50
ESCM2-DR	×	×	0.760	0.553	0.568	0.715	0.566	0.722	0.827	0.569	0.830
Res-DR w/o Δp Δδ	×	×	0.763	0.544	0.568	0.716	0.560	0.715	0.831	0.570	0.836
Res-DR w/o Δp	✓	×	0.783	0.561	0.573	0.734	0.630	0.781	0.833	0.570	0.849
Res-DR w/o Δδ	×	✓	0.768	0.555	0.581	0.721	0.645	0.791	0.841	0.579	0.836
Res-DR	✓	✓	0.793	0.588	0.607	0.750	0.654	0.803	0.854	0.595	0.860
Table 4: Ablation study on loss components in Res-IPS and Res-DR methods, with AUC, NDCG@K and Recall@K as evaluation metrics. The best result is bolded and the second is underlined.
Training loss	COAT	YAHOO! R3	KUAIREC
Method	\( \mathcal{L}_{\text{CVR}}^{\mathcal{U}} \)	\( \mathcal{L}_{\text{CVR}}^{B\&U} \)	AUC	N@5	R@5	AUC	N@5	R@5	AUC	N@50	R@50
\( ESCM^2-IPS \)	×	×	0.757	0.514	0.558	0.729	0.559	0.714	0.815	0.577	0.860
Res-IPS-None	×	×	0.755	0.522	0.546	0.722	0.552	0.707	0.825	0.580	0.853
Res-IPS-U	✓	×	0.770	0.562	0.570	0.718	0.587	0.741	0.833	0.583	0.849
Res-IPS-B&U	×	✓	0.784	0.573	0.592	0.756	0.635	0.778	0.845	0.592	0.880
Res-IPS	✓	✓	0.777	0.575	0.601	0.759	0.639	0.785	0.849	0.601	0.885
\( ESCM^2-DR \)	×	×	0.760	0.553	0.568	0.715	0.566	0.722	0.827	0.569	0.830
Res-DR-None	×	×	0.765	0.544	0.550	0.714	0.575	0.735	0.824	0.562	0.823
Res-DR-U	✓	×	0.770	0.565	0.577	0.722	0.604	0.756	0.836	0.562	0.842
Res-DR-B&U	×	✓	0.790	0.574	0.601	0.744	0.640	0.782	0.848	0.586	0.848
Res-DR	✓	✓	0.793	0.588	0.607	0.750	0.654	0.803	0.854	0.595	0.860
unbiased data, and highlights the importance of leveraging both kinds of data. Meanwhile, the direct use of multi-task learning approaches to IPS and DR estimators cannot benefit the debiasing performance under hidden confounding. The proposed Res-IPS and Res-DR methods stably outperform previous methods on all three datasets, which provides empirical evidence of the existence of hidden confounding in the real-world recommendations, as well as the effectiveness of our methods for removing hidden confounding. Table 2 shows the results of AUC and NDCG@50 on KUAIREC with varying unbiased data ratios. The performance of all methods improves with increasing unbiased data ratio, and our method stably outperforms the baseline methods by a large margin.
Ablation Studies. The two losses  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}}$  and  $\mathcal{L}_{\mathrm{CVR}}^{B\& \mathcal{U}}$  as well as the two residual terms  $\Delta \delta$  and  $\Delta p$  are crucial in the proposed multi-task learning approach. We further conduct ablation studies with respect to the residual networks and the training loss components, respectively. From Table 3, Res-DR using either the propensity residual network or the imputation residual network can stably outperform  $\mathrm{ESCM}^2$ -DR, and Res-DR achieves the best performance when two residual networks are adopted together. When both propensity and imputation residual networks are removed, although Res-DR includes  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}}$  and  $\mathcal{L}_{\mathrm{CVR}}^{B\& \mathcal{U}}$  losses, it has similar performance to  $\mathrm{ESCM}^2$ -DR, which further indicates that the performance improvement of Res-DR can be attributed to the effectiveness of the residual networks. From Table 4, both Res-IPS and Res-DR methods without  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}}$  loss or without  $\mathcal{L}_{\mathrm{CVR}}^{B\& \mathcal{U}}$  loss outperform  $\mathrm{ESCM}^2$ -IPS and  $\mathrm{ESCM}^2$ -DR. Similarly, our methods achieve the best performance when both two losses are preserved. Note that the model with  $\mathcal{L}_{\mathrm{CVR}}^{B\& \mathcal{U}}$  loss performs better than the model with  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}}$  loss, which is attributed to  $\mathcal{L}_{\mathrm{CVR}}^{B\& \mathcal{U}}$  corrects for biased learned propensities and biased imputed errors under the presence of hidden confounding. In addition, unbiased data does not significantly improve the debiasing performance when it is directly used to train the prediction model through  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}}$ , due to the overfitting problem caused by the limited unbiased data size, which is consistent to the poor performance of MF (uniform) in Table 1. Meanwhile, minimizing  $\mathcal{L}_{\mathrm{CVR}}^{\mathcal{U}}$  does not provide any residual information for biased learned propensities and biased imputed errors.
In-Depth Analysis. The proposed methods contain the prediction model, imputation model, propensity model, and residual models, thus it is meaningful to investigate the effect of different optimization algorithms among these models on the debiasing performance. Specifically, we implement the following learning approaches on the Res-DR: (1) Joint Learning (JL) [52], which joint optimizes the prediction model and the imputation model. (2) Double Learning (DL) [11], which adds a parameter
(a) R@5 on YAHOO! R3
(b) R@50 on KUAIREC
(c) N@5 on YAHOO! R3
(d) N@50 on KUAIREC
Figure 2: Joint learning (JL), double learning (DL), and multi-task learning (MTL) on Res-DR.
(a) AUC on YAHOO! R3
Figure 3: Effects of varying weights  $\gamma$  of the consistency loss  $\mathcal{L}_{\mathrm{CVR}}^{BS\&U}$  on Res-IPS and Res-DR.
(b) AUC on KUAIREC
sharing mechanism between the prediction model and the imputation model based on JL. (3) Multi-Task Learning (MTL) adopted in our methods. Figure 2 shows the experiment results. Remarkably, MTL significantly outperforms JL and DL, whereas in Table 1 DR and MTL-based  $\mathrm{ESCM^2}$ -DR perform similarly. This is because DR only has three training models (namely the prediction model, imputation model, and propensity model), whereas the proposed Res-DR has two additional residual models. As the number of models increases, JL can no longer bridge the models efficiently, which leads to a slow convergence and limited performance. Instead, DL increases the connection between models via timely parameter sharing. Finally, MTL trade-offs the different tasks in Res-DR to train all the models, leading to the optimal debiasing performance.
Sensitivity Analysis. We perform the sensitivity analysis of  $\gamma$  on Res-IPS and Res-DR, as shown in Figure 3. Our methods achieve the optimal performance when  $\gamma$  is moderate (0.005-0.01). This is because when  $\gamma$  is too large, it hurts the performance of other tasks (e.g., CVR model training), and when  $\gamma$  is too small, it makes the consistency loss be paid with less attention, so that the hidden confounding cannot be effectively removed. Res-IPS and Res-DR stably outperform  $\mathrm{ESCM}^2$ -IPS and  $\mathrm{ESCM}^2$ -DR under varying  $\gamma$ . This further illustrates the effectiveness the consistency loss.
This paper investigates the use of a few unbiased ratings to calibrate the learned propensities and imputed errors for removing hidden confounding. First, we theoretically reveal the biasedness of previous debiasing methods in the presence of hidden confounding. Next, we propose a multi-task debiasing residual networks learning approach for training the debiased prediction model. By building residual networks and calibrating the biased learned propensities and biased imputed errors, the prediction model is trained on both the calibrated IPS or DR losses and the unbiased dataset to ensure the unbiasedness. Extensive experiments on two benchmark datasets and a large-scale industrial dataset validate the effectiveness of our proposal. A limitation of this work is the use of slightly more model parameters due to the need to address hidden confounding with residual networks.
This work is supported by National Key R&D Program of China (No. 2020YFE0204200) and National Natural Science Foundation of China (No. 62272437 and 61973329). Peng Wu is partially supported by the National Natural Science Foundation of China (No. 12301370) and the Disciplinary Funding of Beijing Technology and Business University (No. STKY202301). Zhi Geng is partially supported by the National Nature Science Foundation of China (No. 12071015).
[1] Stephen Bonner and Flavian Vasile. Causal embeddings for recommendation. In RecSys, 2018.
[2] Yin-Wen Chang, Cho-Jui Hsieh, Kai-Wei Chang, Michael Ringgaard, and Chih-Jen Lin. Training and testing low-degree polynomial data mappings via linearsvm. Journal of Machine Learning Research, 11(4), 2010.
[3] Jiawei Chen, Can Wang, Martin Ester, Qihao Shi, Yan Feng, and Chun Chen. Social recommendation with missing not at random data. In ICDM, 2018.
[4] Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. AutoDebias: Learning to debias for recommendation. In SIGIR, 2021.
[5] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3):1-39, 2023.
[6] H Christopher Frey and Sumeet R Patil. Identification and review of sensitivity analysis methods. Risk analysis, 22(3):553-578, 2002.
[7] Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Xiuqiang He, Rui Zhang, and Jie Sun. A generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In KDD, 2022.
[8] Sihao Ding, Peng Wu, Fuli Feng, Xiangnan He, Yitong Wang, Yong Liao, and Yongdong Zhang. Addressing unmeasured confounder for recommendation with sensitivity analysis. In KDD, 2022.
[9] Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recommender systems. In CIKM, 2022.
[10] Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dolle. Offline a/b testing for recommender systems. In WSDM, 2018.
[11] Siyuan Guo, Lixin Zou, Yiding Liu, Wenwen Ye, Suqi Cheng, Shuaiqiang Wang, Hechang Chen, Dawei Yin, and Yi Chang. Enhanced doubly robust learning for debiasing post-click conversion rate estimation. In SIGIR, 2021.
[12] Miguel A. Hernán and James M. Robins. Causal Inference: What If. Boca Raton: Chapman and Hall/CRC, 2020.
[13] Wenjie Hu, Xiao-Hua Zhou, and Peng Wu. Identification and estimation of treatment effects on long-term outcomes in clinical trials with external observational data. Statistica Sinica, 2023.
[14] Shanshan Huang, Haoxuan Li, Qingsong Li, Chunyuan Zheng, and Li Liu. Pareto invariant representation learning for multimedia recommendation. In ACM-MM, 2023.
[15] Paul Hünermund and Elias Bareinboim. Causal inference and data fusion in econometrics. arXiv:1912.09104, 2019.
[16] Dietmar Jannach, Lukas Lerche, and Markus Zanker. Recommending based on implicit feedback. In Social Information Access, pages 510-569. Springer, 2018.
[17] Simon Jenni and Paolo Favaro. Deep bilevel learning. In ECCV, 2018.
[18] Nathan Kallus and Angela Zhou. Confounding-robust policy improvement. In NeurIPS, pages 9289–9299, 2018.
[19] Nathan Kallus and Angela Zhou. Minimax-optimal policy learning under unobserved confounding. Management Science, 67(5):2870-2890, 2021.
[20] Nathan Kallus, Aahlad Manas Puli, and Uri Shalit. Removing hidden confounding by experimental grounding. In NeurIPS, 2018.