SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that in Eqs. (a.1) and (a.2), the  $Y_{u,i}$ 's on the right-hand side can be replaced with  $Y_{u,i}^{1}$  and  $Y_{u,i}^{0}$ , respectively. This substitution is valid because when  $Z_{u,i} = 1$ , the  $Y_{u,i}$  term in Eq. (a.1) is equivalent to  $Y_{u,i}^{1}$  by definition. In this scenario,  $\hat{Y}_{u,i}^{0}$  is always 0, regardless of the value of  $Y_{u,i}^{0}$ . Similarly, when  $Z_{u,i} = 0$ , the  $Y_{u,i}$  term in Eq. (a.2) is equivalent to  $Y_{u,i}^{0}$ . Therefore, the IPS estimator for the causal effect, in Eq. (1) of the main text, can be rewritten as
$$
\hat {\tau} _ {u, i} = \frac {Z _ {u , i} Y _ {u , i} ^ {1}}{p _ {u , i}} - \frac {(1 - Z _ {u , i}) Y _ {u , i} ^ {0}}{1 - p _ {u , i}}. \tag {a.3}
$$
As established earlier [30], if the propensity score  $p_{u,i}$  and exposure  $Z_{u,i}$  are correctly assigned, the expectation of the estimated causal effect from the IPS estimator is
$$
\mathbb {E} \left[ \hat {\tau} _ {u, i} \right] = Y _ {u, i} ^ {1} - Y _ {u, i} ^ {0} = \tau_ {u, i}. \tag {a.4}
$$
That is to say, if we have ground-truth propensity score as well as exposure, the estimated causal effect is unbiased. If we have only ground-truth exposure but have to estimate propensity scores by substituting  $p_{u,i}$  with  $\hat{p}_{u,i}$  in Eq. (a.3), denote the resulting estimator for the casual effect as  $\hat{\tau}_{u,i}'$ . Then, the expectation and bias of  $\hat{\tau}_{u,i}'$  are
$$
\mathbb {E} \left[ \hat {\tau} _ {u, i} ^ {\prime} \right] = \frac {p _ {u , i} Y _ {u , i} ^ {1}}{\hat {p} _ {u , i}} - \frac {(1 - p _ {u , i}) Y _ {u , i} ^ {0}}{1 - \hat {p} _ {u , i}}, \tag {a.5}
$$
$$
\operatorname {B i a s} \left(\hat {\tau} _ {u, i} ^ {\prime}\right) = \mathbb {E} \left[ \hat {\tau} _ {u, i} ^ {\prime} \right] - \tau_ {u, i} = \left(\frac {p _ {u , i}}{\hat {p} _ {u , i}} - 1\right) Y _ {u, i} ^ {1} - \left(\frac {1 - p _ {u , i}}{1 - \hat {p} _ {u , i}} - 1\right) Y _ {u, i} ^ {0}. \tag {a.6}
$$
Finally, in our setup, both  $Z_{u,i}$  and  $p_{u,i}$  are estimated. Hence, denote the resulting estimator based on  $\hat{Z}_{u,i}$  and  $\hat{p}_{u,i}$  as  $\hat{\tau}_{u,i}^{\prime \prime}$ . We can obtain its bias relative to  $\hat{\tau}_{u,i}^{\prime}$  as follows.
$$
\begin{array}{l} \mathrm {B i a s} (\hat {\tau} _ {u, i} ^ {\prime \prime}) - \mathrm {B i a s} (\hat {\tau} _ {u, i} ^ {\prime}) = \mathbb {E} \left[ \hat {\tau} _ {u, i} ^ {\prime \prime} \right] - \mathbb {E} \left[ \hat {\tau} _ {u, i} ^ {\prime} \right] \\ = \mathbb {E} \left[ \frac {\hat {Z} _ {u , i} Y _ {u , i} ^ {1}}{\hat {p} _ {u , i}} - \frac {(1 - \hat {Z} _ {u , i}) Y _ {u , i} ^ {0}}{1 - \hat {p} _ {u , i}} - \frac {Z _ {u , i} Y _ {u , i} ^ {1}}{\hat {p} _ {u , i}} + \frac {(1 - Z _ {u , i}) Y _ {u , i} ^ {0}}{1 - \hat {p} _ {u , i}} \right] \\ = \mathbb {E} \left[ \frac {(\hat {Z} _ {u , i} - Z _ {u , i}) Y _ {u , i} ^ {1}}{\hat {p} _ {u , i}} - \frac {(Z _ {u , i} - \hat {Z} _ {u , i}) Y _ {u , i} ^ {0}}{1 - \hat {p} _ {u , i}} \right] \\ = \frac {\mathbb {E} \left[ \hat {Z} _ {u , i} - Z _ {u , i} \right]}{\hat {p} _ {u , i}} Y _ {u, i} ^ {1} + \frac {\mathbb {E} \left[ \hat {Z} _ {u , i} - Z _ {u , i} \right]}{1 - \hat {p} _ {u , i}} Y _ {u, i} ^ {0}. \tag {a.7} \\ \end{array}
$$
By adding Eqs. (a.6) and (a.7), we are able to obtain the bias of  $\hat{\tau}_{u,i}^{\prime \prime}$
$$
\begin{array}{l} \operatorname {B i a s} \left(\hat {\tau} _ {u, i} ^ {\prime \prime}\right) = \operatorname {B i a s} \left(\hat {\tau} _ {u, i} ^ {\prime}\right) + \operatorname {B i a s} \left(\hat {\tau} _ {u, i} ^ {\prime \prime}\right) - \operatorname {B i a s} \left(\hat {\tau} _ {u, i} ^ {\prime}\right) \\ = \left(\frac {p _ {u , i} + \mathbb {E} \left[ \hat {Z} _ {u , i} - Z _ {u , i} \right]}{\hat {p} _ {u , i}} - 1\right) Y _ {u, i} ^ {1} - \left(\frac {1 - p _ {u , i} - \mathbb {E} \left[ \hat {Z} _ {u , i} - Z _ {u , i} \right]}{1 - \hat {p} _ {u , i}} - 1\right) Y _ {u, i} ^ {0}. \tag {a.8} \\ \end{array}
$$
This concludes the proof of Proposition 1.
C Additional experimental settings
We describe more details on the datasets, implementation and evaluation metrics.
C.1 Descriptions of datasets
We introduce additional details on data generation and splitting.
Data processing and generation We perform data processing and generation steps per the earlier studies on the DunnHumby $^2$  (DH) [30] and MovieLens $^3$  [29] (ML) datasets.
Specifically, for DH, the items that appear in the weekly mailer are deemed as recommended (i.e., exposed) items. The empirical distribution of  $Y_{u,i}^{1}$  can be found by tallying the weeks in which item  $i$
2The raw data are available at https://www.dunnhumby.com/careers/engineering/sourcefiles.
3The raw data are available at https://grouplens.org/datasets/movielens.
was both recommended to and purchased by user  $u$  (or purchased but not recommended when dealing with  $Y_{u,i}^{0}$ ). The ground-truth values of  $Y_{u,i}^{1}$  and  $Y_{u,i}^{0}$  are then sampled from their respective empirical distributions, which are used to calculate the ground-truth causal effect, as follows.
$$
\tau_ {u, i} = Y _ {u, i} ^ {1} - Y _ {u, i} ^ {0}. \tag {a.9}
$$
Subsequently, two different ways of simulating the ground-truth propensity scores have been attempted. In DH ORIGINAL, the propensity score  $p_{u,i}$  is defined based on the number of weeks in which the item was recommended while the user visited the retailer during the same week. For DH_persoalized, the propensity score  $p_{u,i}$  is established based on the position of item  $i$  in a simulated ranking based on user  $u$ 's probability of interaction with  $i$ . The ground-truth exposure  $Z_{u,i}$  is then sampled from a Bernoulli distribution whose parameter is set to the propensity score  $p_{u,i}$ . Finally, the ground-truth interaction can be computed as follows.
$$
Y _ {u, i} = Z _ {u, i} Y _ {u, i} ^ {1} + \left(1 - Z _ {u, i}\right) Y _ {u, i} ^ {0}. \tag {a.10}
$$
For ML, the empirical distributions of  $Y_{u,i}^{1}$  and  $Y_{u,i}^{0}$  are derived from the interaction log using matrix factorization-based techniques. The propensity score is determined by a simulated ranking for each user, similar to DH和个人ized. Subsequently, the ground-truth values of the exposure, causal effect and interaction is sampled and established following the same process in DH.
Data splitting The above data generation steps are applied to both DH and ML datasets to generate training, validation and testing sets. For the DH datasets, the data generation process is repeated 10 times to simulate the 10-week training data, once more to simulate the 1-week validation data, and 10 more times to simulate the 10-week testing data. For the ML dataset, the generation is repeated once each to generate the training, validation, and testing data, respectively.
It is worth noting that the data generation and splitting processes are dependent on some form of simulation. We utilize such "semi-simulated" data for a number of reasons. First, the true causal effects are not observable due to their counterfactual nature. Second, ground-truth propensity scores and exposure are often not available in public datasets, which nonetheless are essential for model evaluation (even though they are not required for our model training). Third, while some datasets [20, 36, 6] do include information on item impressions or exposure statuses, they often provide a one-sided view of the situation. Specifically, the vast majority, if not all, of the item interactions are preceded by prior impressions. This leaves limited scope for investigating item interactions that occur without prior impressions.
Let us first present key implementation choices regarding propensity and exposure modeling in PROPCare. On all datasets, we set  $\alpha = 0.2$  and  $\beta = 1.0$  for the Beta distribution used in the regularization term. To derive  $\hat{Z}_{u,i}$  based on  $\hat{p}_{u,i}$ , we implement Norm as a Z-score normalization function, such that  $\hat{Z}_{u,i} = 1$  if  $\mathrm{Norm}(\hat{p}_{u,i})\geq \epsilon$ , where the threshold  $\epsilon$  is set to 0.2 for DH-original and DH和个人ized, and 0.15 for ML. To address the bias introduced by  $\hat{p}_{u,i}$ , we employ a practical trick to scale the estimated propensity by a constant factor  $c$ , i.e.,  $\hat{p}_{u,i}\gets c\times \hat{p}_{u,i}$ , prior to training DLCE. This technique aims to further minimize the disparity in the scale between  $p_{u,i}$  and  $\hat{p}_{u,i}$  in order to improve the accuracy of causal effect estimation, according to the theoretical analysis in Sect. 4.5. For DH ORIGINAL and DH個人ized, the scaling factor  $c$  is set to 0.8, while for ML it is set to 0.2. The hyperparameters  $\epsilon$  and  $c$  are tuned using validation data. In particular, we perform a grid search where  $\epsilon$  is searched over the range (0,1) in steps of 0.05 and  $c$  is searched over the range (0,1] in steps of 0.1.
With regard to the model architecture for PROPCARE, we randomly initialize  $\mathbf{x}_u$  and  $\mathbf{x}_i \in \mathbb{R}^{128}$  as user and item ID features. The embedding model  $f_{e}$  takes  $(\mathbf{x}_u||\mathbf{x}_i)$  as input and is implemented as an MLP with 256, 128 and 64 neurons for its layers.  $f_{p}$  and  $f_{r}$  are both implemented as MLPs with 64, 32, 16, 8 neurons for the hidden layers and an output layer activated by the sigmoid function. Except the output layer, all layers of  $f_{e}$  and hidden layers of  $f_{p}$  and  $f_{r}$  are activated by the LeakyReLU function. Finally, PROPCARE is trained with a stochastic gradient descent optimizer using mini-batches, with a batch size set to 5096.
Table a.1: Performance comparison with conventional interaction-based recommenders.
Methods	DH-original	DH和个人ized	ML
CP@10↑	CP@100↑	CDCG↑	CP@10↑	CP@100↑	CDCG↑	CP@10↑	CP@100↑	CDCG↑
MF	.0206±.001	.0118±.000	.8569±.002	.0433±.001	.0196±.000	.9699±.002	-.460±.004	-.205±.002	9.421±.023
BPR	.0301±.000	.0138±.000	.8983±.002	.0476±.001	.0245±.000	1.018±.001	-.408±.002	-.197±.001	9.898±.028
LightGCN	.0309±.001	.0149±.001	.9113±.004	.0821±.001	.0263±.001	1.095±.002	-.342±.006	-.177±.002	10.16±.050
PROPCARE	.0351±.002	.0156±.001	.9268±.005	.1270±.001	.0381±.000	1.426±.001	.0182±.002	.0337±.002	13.80±.011
Results are reported as the average of 5 runs (mean±std). Best results are bolded.
For the baseline CJBPR, we have used its authors' implementation<sup>4</sup> and modified its mini-batch setting and optimizer to be identical to PROPCare. For the baseline EM, we have implemented their proposed propensity model in Python. To conduct downstream causality-based recommendation, we have used the authors' implementation of DLCE<sup>5</sup> following the settings in their work [30]. For a fair comparison, the normalization function Norm and constant scaling factor  $c$  are identically applied to all baselines, which generally improve the performance across the board.
We implement PROPCare using TensorFlow 2.11 in Python 3.10. All experiments were conducted on a Linux server with a AMD EPYC 7742 64-Core CPU, 512 GB DDR4 memory and four RTX 3090 GPUs.
C.3 Evaluation metrics
Unlike commonly used metrics like precision and NDCG that reward all positive interactions regardless of whether the item is exposed or not, we use a variant of them that only reward positive interactions resulting from item exposure. Concretely, we use the Causal effect-based Precision (CP) and Discounted Cumulative Gain (CDCG) as defined in existing work [30].
$$
\mathrm {C P} @ K = \frac {1}{U} \sum_ {u = 1} ^ {U} \sum_ {i = 1} ^ {I} \frac {\mathbf {1} \left(\operatorname {r a n k} _ {u} \left(\hat {s} _ {u , i}\right) \leq K\right) \tau_ {u , i}}{K}, \tag {a.11}
$$
$$
\mathrm {C D C G} = \frac {1}{U} \sum_ {u = 1} ^ {U} \sum_ {i = 1} ^ {I} \frac {\tau_ {u , i}}{\log_ {2} \left(1 + \operatorname {r a n k} _ {u} \left(\hat {s} _ {u , i}\right)\right)}, \tag {a.12}
$$
where  $\mathbf{1}(\cdot)$  is an indicator function, and  $\mathrm{rank}_u(\hat{s}_{u,i})$  returns the position of item  $i$  in the ranking list for user  $u$  as determined by the ranking score  $\hat{s}_{u,i}$ . In our paper, we report CP@10, CP@100 and CDCG. Note that since  $\tau_{u,i}$  can be  $-1$ , these metrics can be negative.
We present additional empirical results on the comparison to conventional recommender systems, as well as the robustness and scalability of PROPCare.
D.1 Comparison to interaction-based recommenders
To demonstrate the advantage of PROPCare over conventional interaction-based recommender systems, we compare with three well-known models, namely Matrix Factorization $^6$  (MF) [13], Bayesian Personalized Ranking $^6$  (BPR) [23] and LightGCN $^7$  [7]. Note that we use only interaction data  $Y_{u,i}$  as training labels for these methods and rank the items for each user based on the predicted  $\hat{y}_{u,i}$ , as in their classical paradigm. We evaluate the causal performance with CP@10, CP@100 and CDCG in Tab. a.1. From the results we can observe that the conventional interaction-based methods do not demonstrate strong causal performance.
A fundamental reason is that the conventional models indiscriminately reward all positive interactions, regardless of whether the candidate items have been exposed or not. To delve deeper into the impact
<sup>4</sup>Available at https://github.com/Zziwei/Unbiased-Propensity-and-Recommendation.
Available in ancillary files at https://arxiv.org/abs/2008.04563.
<sup>6</sup>Implementation available in ancillary files at https://arxiv.org/abs/2008.04563.
$^{7}$ Implementation available at https://github.com/kuandeng/LightGCN.
Table a.2: Influence of exposure status on causal effect.
E[τu,i|Yu,i=1,Zu,i=1]	E[τu,i|Yu,i=1,Zu,i=0]
DH-original	.8882	-.901
DH和个人ized	.8882	-.985
ML	.8623	-.818
Table a.3: Employing CausE as the causality-based recommendation backbone.
Methods	DH ORIGINAL	DH和个人ized	ML
CP@10↑	CP@100↑	CDCG↑	CP@10↑	CP@100↑	CDCG↑	CP@10↑	CP@100↑	CDCG↑
Ground-truth	.0495±.002	.0216±.002	1.020±.004	.0663±.003	.0241±.002	1.105±.003	.1849±.002	.1438±.004	15.25±.007
POP	.0059±.001	.0097±.002	.7759±.002	.0427±.002	.0159±.001	.9616±.002	-.191±.001	-.036±.002	12.27±.006
CJBPR	.0073±.001	.0100±.005	.7809±.003	.0451±.002	.0165±.003	.9621±.005	-.217±.001	-.044±.001	12.05±.006
EM	.0065±.000	.0103±.001	.7802±.002	.0478±.001	.0166±.001	.9819±.006	-.197±.002	-.041±.002	12.24±.009
PROPARE	.0123±.001	.0114±.001	.8084±.001	.0580±.005	.0201±.001	1.052±.002	-.138±.001	-.035±.003	12.40±.009
Results are reported as the average of 5 runs (mean±std). Except Ground-truth, best results are bolded and runners-up are underlined.
of different exposure statuses on the causal effect of user-item pairs with positive interactions, we calculate  $\mathbb{E}[\tau_{u,i}|Y_{u,i} = 1,Z_{u,i} = 1]$  and  $\mathbb{E}[\tau_{u,i}|Y_{u,i} = 1,Z_{u,i} = 0]$  for each dataset, as presented in Tab. a.2. The divergent results given different exposure statuses imply that exposure plays a significant role in the causal effect involving positive interactions. Conventional interaction-based models ignore exposure and may rank items in a way that negatively impacts the causal effect. Between the two DH datasets, DH和个人ized exhibits a lower value of  $\mathbb{E}[\tau_{u,i}|Y_{u,i} = 1,Z_{u,i} = 0]$ , which explains the greater causal performance gap between the conventional models and PROPCare than the gap on DH-original. This greater gap arises because recommending items with  $Y_{u,i} = 1,Z_{u,i} = 0$  lowers the causal effect more on DH和个人ized than on DH ORIGINAL.
D.2 Robustness to alternative backbone and scalability analysis
To show the robustness of PROPCARE, we opt for  $\mathrm{CausE}^6$  [2], another causality-based recommender, as an alternative backbone to DLCE. Like DLCE, CausE makes causality-based recommendation given the estimated propensity and exposure data from PROPCARE or the baselines. The results in Tab. a.3 show a similar pattern as using DLCE in the main text. That is, our PROPCARE consistently outperforms other baselines even with a different causality-based model as the backbone.
We further examine the scalability of our proposed PROPCARE on increasingly larger datasets, namely, MovieLens  $1\mathrm{M}^8$ , MovieLens  $10\mathrm{M}^8$  and MovieLens  $20\mathrm{M}^8$ . In Tab a.4, we report the training times of both PROPCARE and DLCE, in hours. We observe that the training of PROPCARE generally follows a linear growth, in line with the time complexity analysis in Appendix A. Moreover, PROPCARE only presents a marginal overhead on top of the backbone DLCE, showing its feasibility in working with existing backbones.
Table a.4: Training times for PROPCare and DLCE, in hours.
MovieLens 1M	MovieLens 10M	MovieLens 20M
PROPCARE	.0814	1.494	3.170
DLCE	2.1759	22.658	40.658
Available at https://grouplens.org/datasets/movielens/.