SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(\frac {p _ {u , i} + \mathbb {E} \left[ \hat {Z} _ {u , i} - Z _ {u , i} \right]}{\hat {p} _ {u , i}} - 1\right) Y _ {u, i} ^ {1} - \left(\frac {1 - p _ {u , i} - \mathbb {E} \left[ \hat {Z} _ {u , i} - Z _ {u , i} \right]}{1 - \hat {p} _ {u , i}} - 1\right) Y _ {u, i} ^ {0}. \tag {9}
$$
We defer the proof to Appendix B. From the bias stated in Proposition 1, we make two further remarks to guide the learning and evaluation of propensity scores and exposure.
Remark 1 The bias is influenced by three major factors:  $p_{u,i} / \hat{p}_{u,i}$ ,  $(1 - p_{u,i}) / (1 - \hat{p}_{u,i})$  and  $\mathbb{E}\left[\hat{Z}_{u,i} - Z_{u,i}\right]$ . Note that if  $\hat{p}_{u,i} = p_{u,i}$  and  $\hat{Z}_{u,i} = Z_{u,i}$ , the bias would be zero which is consistent with earlier findings [30].
Remark 2 If the estimated  $\hat{p}_{u,i}$  is extremely close to 0 or 1, the bias can be potentially very large.
The above proposition and remarks shed some light on what we should focus on when estimating or evaluate exposure and propensity score. On the one hand, since exposure is a binary variable and the bias is influenced by  $\mathbb{E}\left[\hat{Z}_{u,i} - Z_{u,i}\right]$ , we may evaluate it with binary classification metrics such as F1 score. On the other hand, since propensity is a continuous variable and estimations extremely close to zero or one should be avoided, regularizing the global distribution in Eq. (7) and ensuring a proper scale of the propensity scores would be useful.
Table 1: Statistics of datasets.
Dataset	#users	#items	\( {\bar{Y}}_{u,i} \)	\( {\bar{Z}}_{u,i} \)	\( {\bar{\tau }}_{u,i} \)	\( {\bar{p}}_{u,i} \)
DH-original	2,309	1,372	.0438	.6064	.0175	.2894
DH和个人ized	2,309	1,372	.0503	.6265	.0178	.4589
ML	943	1,682	.0676	.0593	.0733	.0594
In this section, we comprehensively evaluate the effectiveness of the proposed PROPCARE through both quantitative and qualitative experiments.
Datasets We employ three standard causality-based recommendation benchmarks. Among them, DH ORIGINAL and DH和个人ized are two versions of the DunnHumby dataset [30], which includes purchase and promotion logs at a physical retailer over a 93-week period. The difference in the two versions mainly lies in the derivation of ground-truth propensity scores as stated by Sato et al. [30], which are based on items featured in the weekly mailer in DH ORIGINAL, and with a simulated personalization factor in DH personalized. The third dataset is MovieLens 100K (ML) [29], which includes users' ratings on movies and simulated propensity scores based on the ratings and user behaviors. Note that PROPCARE do not require any propensity or exposure data at all. The ground-truth values are only used to evaluate model output. On each dataset, we generate the training/validation/test sets following their original work [30, 29], respectively. We summarize each dataset in Tab. 1, listing the number of users (#users) and items (#items), as well as the average value of several key variables including the observed interaction  $(\bar{Y}_{u,i})$ , exposure  $(\bar{Z}_{u,i})$ , causal effect  $(\bar{\tau}_{u,i})$  and propensity  $(\bar{p}_{u,i})$ . Further details can be found in Appendix C.1.
Baselines We compare PROPCare with the following propensity estimators: (1) Ground-truth: Propensity score and exposure values are directly taken from the datasets. (2) Random: Propensity scores are assigned randomly between 0 and 1. (3) Item popularity (POP): Propensity scores are assigned as item popularity normalized to  $(0,1)$ . (4) CJBPR [42]: An unbiased recommendation model that optimizes propensity and relevance alternately in a point-wise manner. (5) EM [21]: An recommendation model that learns propensity scores in a point-wise manner using an expectation-maximization algorithm.
Note that Ground-truth uses the ground-truth values of propensity  $p_{u,i}$  and exposure  $Z_{u,i}$  directly as input to train DLCE [30]. All other baselines do not need such ground-truth values in any stage just as PROPCARE. In these methods, the estimated propensity  $\hat{p}_{u,i}$  is used to further derive the exposure  $\hat{Z}_{u,i}$ , in the same way as PROPCARE (see Sect. 4.3). Finally, we utilize the estimated values to train DLCE (see Sect. 4.4).
Parameter settings We tune the hyperparameters based on the validation data, following guidance in the literature. Specifically, in PROPCare, the trade-off parameter  $\lambda$  and  $\mu$  are set to 10 and 0.4, respectively, on all datasets. For the downstream causal recommender DLCE, we follow the earlier settings [30]. For other settings and implementation details, refer to Appendix C.2.
Evaluation metrics We evaluate the performance of causality-based recommendation with CP@10, CP@100 and CDCG, whose definitions can be found in Appendix C.3. Additionally, we measure the accuracy of estimated propensity scores w.r.t. the ground-truth values using Kullback-Leibler divergence (KLD) and Kendall's Tau (Tau) [12], and that of estimated exposure using F1 score. Note that all metrics, except KLD, indicate better performance with a larger value.
5.2 Results and discussions
We first compare the performance of PROPCare and the baselines, followed by analyses of model ablation, the regularization term, and various influencing factors. Additional experiments including
Table 2: Performance comparison on downstream causality-based recommendation.
Methods	DH-original	DH和个人ized	ML
CP@10↑	CP@100↑	CDCG↑	CP@10↑	CP@100↑	CDCG↑	CP@10↑	CP@100↑	CDCG↑
Ground-truth	.0658±.001	.0215±.001	1.068±.000	.1304±.001	.0445±.001	1.469±.003	.2471±.001	.1887±.000	16.29±.006
Random	.0154±.001	.0071±.002	.7390±.004	.0479±.004	.0107±.005	.8316±.039	.0124±.002	.0135±.005	13.16±.076
POP	.0200±.000	.0113±.000	.7877±.001	.0457±.000	.0096±.001	.8491±.002	-.142±.001	-.092±.001	11.43±.005
CJBPR	.0263±.001	.0087±.001	.7769±.002	.0564±.008	.0106±.005	.8528±.032	-.410±.002	-.187±.001	9.953±.006
EM	.0118±.001	.0067±.001	.7247±.001	.0507±.002	.0121±.001	.8779±.003	-.437±.002	-.194±.002	10.21±.011
PROPCare	.0351±.002	.0156±.001	.9268±.005	.1270±.001	.0381±.000	1.426±.001	.0182±.002	.0337±.002	13.80±.011
Results are reported as the average of 5 runs (mean±std). Except Ground-truth, best results are bolded and runners-up are underlined.
Table 3: Performance comparison on propensity score (KLD, Tau) and exposure (F1 score) estimation.
Methods	DH ORIGINAL	DH_personalized	ML
KLD↓	Tau↑	F1 score↑	KLD↓	Tau↑	F1 score↑	KLD↓	Tau↑	F1 score↑
Random	.5141±.001	.0002±.000	.4524±.013	3.008±.002	.0001±.000	.4463±.021	.0363±.002	.0002±.000	.4511±.022
POP	.5430±.000	.4726±.000	.2851±.000	4.728±.000	.6646±.000	.2772±.000	.0615±.000	.4979±.000	.5050±.000
CJBPR	.3987±.008	.3279±.011	.2853±.005	2.650±.022	.6477±.013	.2825±.005	.0230±.006	.4956±.045	.5189±.020
EM	.6380±.002	.0834±.000	.4974±.001	2.385±.001	.0934±.002	.4954±.009	.0517±.001	.1321±.002	.3653±.005
PROPARE	.3851±.023	.3331±.065	.5846±.006	1.732±.038	.4706±.072	.6059±.017	.0204±.005	.3889±.034	.4847±.020
Results are styled in the same way as in Tab. 2.
comparison to conventional recommendation methods, evaluation on an alternative backbone, and a scalability study are presented in Appendix D.
Performance comparison We evaluate PROPCare against the baselines in two aspects: (1) The downstream causality-based recommendation using the estimated propensity and exposure; (2) The accuracy of the estimated propensity and exposure.
We first illustrate the performance of causality-based recommendation in Tab. 2. It is not surprising that Ground-truth achieves the best causal effect by incorporating actual propensity and exposure values in DLCE. However, since ground-truth values are often unavailable, we rely on estimations. Among all baselines, PROPCARE most closely approaches Ground-truth's performance. Notably, in the DH和个人ized dataset, PROPCARE exhibits only a  $6.6\%$  average decrease from Ground-truth across three metrics, significantly outperforming the second-best EM which suffers a  $56.6\%$  drop. Furthermore, PROPCARE surpasses the point-wise CJBPR and EM, implying the advantage of our pairwise formulation based on Assumption 1.
Next, we analyze the accuracy of propensity and exposure estimation in Tab. 3. Among the baselines, POP performs the best in Kendall's Tau. However, the causality metrics of POP is poor (see Tab. 2) due to the ill-fit propensity distribution, reflected in its large KLD from the ground-truth distribution. The estimation of exposure is also challenging for POP in most cases. In contrast, PROPARE demonstrates outstanding performance in F1 score and KLD, leading to effective causal metrics. Although its Tau scores lag behind some baselines, a robust distribution on propensity and accurate binary predictions of exposure still contribute to good causal performance. The results in Tab. 3 highlight that causality-based recommendation is influenced by multiple factors, rather than relying solely on a single aspect of estimation. We will discuss these influencing factors further toward the end of this part.
Ablation study To evaluate the impact of our key design motivated by Assumption 1, we derive five variants from Eq. (6): (1) NO_P: removing the constraint on estimated  $\hat{p}_{u,i}$  by deleting the term with  $f_{p}(\mathbf{x}_{u,i}) - f_{p}(\mathbf{x}_{u,j})$ ; (2) NO_R: removing the constraint on estimated  $\hat{r}_{u,i}$  by deleting the term with  $f_{r}(\mathbf{x}_{u,j}) - f_{r}(\mathbf{x}_{u,i})$ ; (3) NO_P_R: removing  $\mathcal{L}_{\mathrm{pop}}$  entirely from the overall loss to eliminate Assumption 1 altogether; (4) NEG: reversing Assumption 1 by replacing  $\mathrm{Sgn}_{i,j}$  with  $-\mathrm{Sgn}_{i,j}$  to assume that more popular items have smaller propensity scores; (5)  $\kappa = 1$ : setting all  $\kappa_{u,i,j}$ 's to a constant 1, resulting in equal weighting of all training triplets. Their causal performances are illustrated in Fig. 3. Comparing to the full version of PROPCare, NO_R and NO_P show a small drop in performance due to the absence of additional constraints on propensity or relevance, indicating that the pairwise loss is still partially effective. The drop in  $\kappa = 1$  highlights the need for controlling the similarity between interaction probabilities. The further drop observed in NO_P_R
Figure 3: Ablation study on PROPCARE.
Figure 4: Effect of regularization term.
after removing the entire assumption further demonstrates the validity of our assumption and the effectiveness of  $\mathcal{L}$  pop. The worst performance of NEG indicates that contradicting Assumption 1 greatly affects the results of causality-based recommendation.
Effect of regularization We examine how effective the regularization term is by varying the trade-off hyperparameter  $\mu$ , as used in Eq. (7), over the set  $\{0, 0.01, 0.1, 0.2, 0.4, 0.8, 1, 2, 4, 8, 10\}$ . The results, shown in Fig. 4, indicate that when  $\mu$  is set to 0, which means the regularization term is completely removed, the causal performance is significantly compromised. This demonstrates the advantage of regularizing the propensity with a beta distribution during the training of PROPCare. As  $\mu$  increases, the causal performance gradually improves and reaches its peak within the range of  $[0.2, 0.8]$ . We suggest further exploration by fine-tuning the hyperparameter within this interval.
Factors influencing causality-based recommendation We further investigate the factors crucial to causality-based recommendation, by injecting noises into ground-truth propensity or exposure values. In Fig. 5(a), we randomly flip a fraction of the exposure values while using the ground-truth propensity scores for DLCE training. The flipping ratios range from  $\{0, 0.01, 0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5\}$ . In Fig. 5(b), we add Gaussian noises to the propensity scores, where the variances of the noise range from  $\{0, 0.1, \dots, 0.5\}$  while using the ground-truth exposure. The results in Fig. 5 consistently show a decrease in performance due to misspecified exposure or propensity. To further examine the correlation between estimation accuracy and recommendation performance, we create scatter plots in Fig. 6 for the DH ORIGINAL dataset. Each data point represents a baseline labeled with its name. The plots reveal a general trend where the CDCG is correlated with the accuracy of estimations.
Figure 5: Analysis of factors that influence causality-based recommendation performance.
Figure 6: Correlation analysis on factors influencing recommendation.
Table 4: Case study of an anonymous user.
Ground-truth	POP	CJBPR	PROPCare
garlic bread (1176)	bananas (1310)	‡ fluid milk (1169)	‡ infant soy (1232)
‡ cleansing wipes (737)	toilet tissue (742)	bananas (1310)	‡ fluid milk (1169)
‡ fluid milk (1169)	‡ fluid milk (1169)	cereal (1090)	bananas (1310)
‡ primal (807)	white bread (675)	strawberries (834)	pure juice (1277)
alkaline batteries (754)	tortilla chips (634)	margarine tubs/bowls (1245)	coffee creamers (1169)
Each column represents the recommendation list output by DLCE trained with the estimated propensity and exposure by the corresponding baseline. The purchased items are highlighted in bold. Items with positive causal effect  $(\tau_{u,i} = 1)$  and negative causal effect  $(\tau_{u,i} = -1)$  are marked by  $\ddagger$  and  $\mathbb{I}$ , respectively, and unmarked items have zero causal effect  $(\tau_{u,i} = 0)$ . Numbers in brackets are the popularity ranks in the training set.
5.3 Case study
We conduct a case study to demonstrate the advantages of PROPCARE in a practical ranking-based recommendation scenario. In Tab. 4, we analyze the top-5 recommended items for an anonymous user with ID 2308 in the DH和个人ized dataset. In the first column, by utilizing ground-truth propensity scores and exposure, DLCE effectively generates a ranking list where most items have a positive causal effect. All items with a positive causal effect were eventually purchased, achieving the goal of causality-based recommendation. Comparing the lists generated by CJBPR and PROPCARE, it is evident that the associated causal effects of the purchased items differ. For example, in the CJBPR list, recommending "strawberries" has zero causal effect, indicating that the user could have still purchased it even without recommendation. In contrast, PROPCARE recommends "infant soy", which has a positive causal effect, making it a more ideal choice. Overall, given the list recommended by CJBPR, the user would only purchase "strawberries" and "fluid milk". However, given the list from PROPCARE, in addition to "infant soy" and "fluid milk", the user may still purchase "strawberries" even without being recommended due to its zero causal effect. Besides, POP tends to recommend popular items but with a lower causal effect, even including an item with a negative causal effect. The results suggest that POP is not an appropriate tool for estimating propensity scores in the context of causality-based recommendation.
In this paper, we introduced PROPCARE, a propensity estimation model for causality-based recommendation systems without the need to access ground-truth propensity and exposure data. Leveraging our observation on the pairwise characteristics between propensity scores and item popularity, we formulated a key assumption and incorporated it as prior information to enhance our estimation, thereby improving causality-based recommendation. A theoretical analysis was presented to understand the factors influencing the bias in estimated causal effects, thereby informing model design and evaluation. Empirical studies demonstrated the superiority of PROPCARE over the baselines. Future research avenues include exploring direct exposure estimation without propensity scores, and investigating parametric causal effect estimators that are potentially more powerful.
This research is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funds (Grant No. A20H6b0151). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the A*STAR. Dr. Yuan Fang also acknowledges the Lee Kong Chian Fellowship awarded by Singapore Management University for the support of this work.
[1] Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. Unbiased learning to rank with unbiased propensity estimation. In ACM SIGIR Conference on Research and Development in Information Retrieval, pages 385-394, 2018.
[2] Stephen Bonner and Flavian Vasile. Causal embeddings for recommendation. In ACM Conference on Recommender Systems, pages 104-112, 2018.
[3] Alexey Borisov, Ilya Markov, Maarten De Rijke, and Pavel Serdyukov. A neural click model for web search. In ACM International Conference on World Wide Web, pages 531-541, 2016.
[4] Richard K Crump, V Joseph Hotz, Guido W Imbens, and Oscar A Mitnik. Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1):187-199, 2009.
[5] James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, et al. The YouTube video recommendation system. In ACM Conference on Recommender Systems, pages 293-296, 2010.
[6] Zhengxiao Du, Xiaowei Wang, Hongxia Yang, Jingren Zhou, and Jie Tang. Sequential scenario-specific meta learner for online recommendation. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2895-2904, 2019.
[7] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 639-648, 2020.
[8] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In ACM International Conference on World Wide Web, pages 173-182, 2017.
[9] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[10] Dietmar Jannach and Michael Jugovac. Measuring the business value of recommender systems. ACM Transactions on Management Information Systems, 10(4):1-23, 2019.
[11] Christopher C Johnson. Logistic matrix factorization for implicit feedback data. Advances in Neural Information Processing Systems, 27(78):1-9, 2014.
[12] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
[13] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
[14] Haoxuan Li, Chunyuan Zheng, and Peng Wu. StableDR: Stabilized doubly robust learning for recommendation on data missing not at random. In International Conference on Learning Representations, 2023.
[15] Xueqi Li, Guoqing Xiao, Yuedan Chen, Zhuo Tang, Wenjun Jiang, and Kenli Li. An explicitly weighted gcn aggregator based on temporal and popularity features for recommendation. ACM Transactions on Recommender Systems, 2023.
[34] Aaron Van den Oord, Sander Dieleman, and Benjamin Schrauwen. Deep content-based music recommendation. Advances in Neural Information Processing Systems, 26, 2013.
[35] Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. KGAT: Knowledge graph attention network for recommendation. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 950-958, 2019.
[36] Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. MIND: A large-scale dataset for news recommendation. In Annual Meeting of the Association for Computational Linguistics, pages 3597-3606, 2020.
[37] Teng Xiao and Suhang Wang. Towards unbiased and robust causal ranking for recommender systems. In ACM International Conference on Web Search and Data Mining, pages 1158-1167, 2022.
[38] Xu Xie, Zhaoyang Liu, Shiwen Wu, Fei Sun, Cihang Liu, Jiawei Chen, Jinyang Gao, Bin Cui, and Bolin Ding. CausCF: Causal collaborative filtering for recommendation effect estimation. In ACM International Conference on Information and Knowledge Management, pages 4253-4263, 2021.
[39] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Adversarial counterfactual learning and evaluation for recommender system. Advances in Neural Information Processing Systems, 33:13515-13526, 2020.
[40] Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In ACM conference on Recommender Systems, pages 279-287, 2018.
[41] Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. Causal intervention for leveraging popularity bias in recommendation. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 11–20, 2021.
[42] Ziwei Zhu, Yun He, Yin Zhang, and James Caverlee. Unbiased implicit recommendation and propensity estimation via combinational joint learning. In ACM Conference on Recommender Systems, pages 551-556, 2020.
A Training procedure of PROPCARE
We present the pseudocode for training our proposed PROPCARE in Algorithm 1. The training steps involve calculating various loss terms including  $\mathcal{L}_{\mathrm{naive}}$ ,  $\mathcal{L}_{\mathrm{pop}}$  and the regularization term. We update all learnable parameters based on the total loss defined in Eq. (7) of the main text.
Algorithm 1: Training PROPCARE
Input: Observed training interaction data  $D$    
Output: Model parameters  $\Theta$    
Initialize model parameters  $\Theta$    
while not converged do   
foreach user-item pair  $(u,i)$  in  $D$  do Compute  $\mathcal{L}_{\mathrm{naive}}$  by Eq. (3) of the main text; Sample an item  $j$  for each pair  $(u,i)$  from  $\{1,2,\dots,I\} \backslash i$  Compute  $\mathcal{L}_{\mathrm{pop}}$  by Eq. (6) of the main text; Compute the total loss in Eq. (7) of the main text;   
end Update  $\Theta$  by backpropagation of the loss in Eq. (7) of the main text;   
end return  $\Theta$
We then analyze the time complexity of the training procedure. We first consider the computation of  $\mathcal{L}_{\mathrm{naive}}$ . This involves three MLP models:  $f_{e}$ ,  $f_{p}$ , and  $f_{r}$ . As we have to compute  $\mathbf{x}_{u,i}$ ,  $\hat{p}_{u,i}$  and  $\hat{r}_{u,i}$  for each user-item pair, each MLP incurs a time complexity of  $\mathcal{O}(n)$ , where  $n = |D|$ , the number of user-item pairs in the training data. To further find the interactions  $\{(y_{u,i})\}$ , we need to compute the product between  $\hat{p}_{u,i}$  and  $\hat{r}_{u,i}$ , which also has a linear time complexity of  $\mathcal{O}(n)$ . Next, to compute  $\mathcal{L}_{\mathrm{pop}}$ , we need to sample another item  $j$  for each user-item pair in the training data, which has a time complexity of  $\mathcal{O}(n)$ . For each tuple  $(u,i,j)$ , we need to further compute  $\hat{p}_{u,j}$  and  $\hat{r}_{u,j}$ , each taking  $\mathcal{O}(n)$  time. Finally, the time complexity to compute the regularization term is also  $\mathcal{O}(n)$ , as the empirical distribution is based on all estimated propensity scores for training pairs. In summary, a series of  $O(n)$  procedures are carried out for the user-item pairs in the training set  $D$ , resulting in an overall linear time complexity of  $O(n)$ . This demonstrates the scalability of our proposed PROPCARE, which we further analyze empirically in Appendix D.2.
B Bias of the estimated causal effect
Here, we present the proof of Proposition 1, which have appeared in the main text, Sect. 4.5.
Proof 1 As defined earlier [30], we can model the potential outcomes as follows,
$$
\hat {Y} _ {u, i} ^ {1} = \frac {Z _ {u , i} Y _ {u , i}}{p _ {u , i}}, \tag {a.1}
$$
$$
\hat {Y} _ {u, i} ^ {0} = \frac {\left(1 - Z _ {u , i}\right) Y _ {u , i}}{1 - p _ {u , i}}. \tag {a.2}
$$