Estimating Propensity for Causality-based Recommendation without Exposure Data
Authors: Zhongzhou Liu
Yuan Fang
Min Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: recommendation systems
causal effect
propensity score
propensity estimation

Abstract: Causality-based recommendation systems focus on the causal effects of user-item interactions resulting from item exposure (i.e., which items are recommended or exposed to the user), as opposed to conventional correlation-based recommendation. They are gaining popularity due to their multi-sided benefits to users, sellers and platforms alike. However, existing causality-based recommendation methods require additional input in the form of exposure data and/or propensity scores (i.e., the probability of exposure) for training. Such data, crucial for modeling causality in recommendation, are often not available in real-world situations due to technical or privacy constraints. In this paper, we bridge the gap by proposing a new framework, called Propensity Estimation for Causality-based Recommendation (PropCare). It can estimate the propensity and exposure from a more practical setup, where only interaction data are available *without* any ground truth on exposure or propensity in training and inference. We demonstrate that, by relating the pairwise characteristics between propensity and item popularity, PropCare enables competitive causality-based recommendation given only the conventional interaction data. We further present a theoretical analysis on the bias of the causal effect under our model estimation.  Finally, we empirically evaluate PropCare through both quantitative and qualitative experiments.

Estimating Propensity for Causality-based Recommendation without Exposure Data
Zhongzhou Liu
School of Computing and Information Systems
Singapore Management University
Singapore, 178902 zzliu 2020@phdsssss smu. edu..g
Yuan Fang*
Min Wu
School of Computing and Information Systems Singapore Management University
Institute for Infocomm Research STTAR Singapore, 138632
Singapore, 178902 y yfangesmu. edu. sg
wumin0i2r.. a-star edu. sg
Abstract
Causality-basee recommendation systems focus on the causal effects of user-item interactions resulting from item exposure (i.e., which items are recommended exposed to the user), as opposed to conventional correlation- tion. They are gaining popularity due to their multi-sided benefits users, sellers and platforms alike. However, existing causality-based recommendation methods require additional input in the form of exposure data and/or propensity scores (i.e., the probability of exposure) for training. Such data, crucial for modeling causality in recommendation, are often not available in real-world situations due to techni- cal privacy constraints. In this paper, we bridge the gap by proposing new framework, called Propensity Estimation for Causality-basee bbsed Recommendation (PRoPCARE). It can estimate the propensity and exposure from more practical setup, where only interaction data are available without any ground truth on ex- posure or propensity in training and inference. We demonstrate that, by relating the pairwise characteristics between propensity and item popularity, PROPCARE enables competitive causality-based recommendation given only the conventional interaction data. We further present a theoretical analysis on the bias of the causal effect under model estimation. Finally, we empirically evaluate PROPCARE through both quantitative and qualitative experiments.
recommenda-
1 Introduction
Recommendation systems have been widely deployed in many real-world applications, such as streaming services 34 5) online shopping [17] and job searching (19]. The primary aim of recommendation systems, such as boosting sales and user engagement [10], depends heavily on user interactions, such as clicking on or purchasing items. Hence, classical paradigm is predict user-item interactions, and accordingly, recommend items with the highest probability of being interacted (e.g., clicked or purchased) to users 22 33 B5). This paradigm ignores the causal impact behind recommendation 31: : If an item already has a high probability of being interacted by a user without being recommended, there really need to recommend the this user?
Corresponniing author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Zuf
Zus
Unobserved variable
pop
pop
Observed variable
Yuui
Yus
(e) Previous framework where exposure observable
(b) Our framework PropCare where exposure is unobservable and papularity incorporated as prior
Figure 1: Causal diagrams under different frameworks. pop; is the popularity (prior) of item i. Ya, indicates if user interacts with item i. Zu, indicates if item is exposed to user u.
37 have shifted the focus to this question. They aim to recommend item based on the uplift, also called the causal effect, in the user's behavior (e.g.. clicks or purchases) caused by different treatments (i.e., reconmending/expsiig the item or not) 9]. Such causality-based recommendation systems posit that recommending items with higher causal effect carries greater merit than those with a higher interaction probability. Typical approaches involve quantifying the causal effect the user's behavior, based on the observed data and the counterfactual treatment [37]. Existing works assume that the exposure data (i.e.. whether an item has been recommended a user not), or the propensity scores [25] (i.e., the probability of an item to user), are observable at least during the training stage. However, in real-world scenarios, those data are often unavailable. For instance, while it is feasible log each user who purchased an item in e-commerce platform, it may be difficult to distinguish between purchases made with or without exposure due to technical and privacy constraints determining if a user has been exposed to the item a priori. Without the exposure data and/or propensity scores provided during training, existing causality-baan based recommenders cannot be deployed.
Recently, a few studies 38
30
28
recommending/expee
Toward practical causality-based recommendation, we consider a more relaxed and realistic setup where exposure and propensity scores are not observable. Although some previous works 38 42
14 21 have attempted to estimate propensity scores in a different context (e.g., addressing biases in recommendation), they further suffer two key limitations. First, most state-of- the ahr methods still require exposure data to train the propensity estimator (38 I 14] Second, they fail to integrate prior knowledge into the propensity estimator, resulting in less robust estimation. To address these challenges and bridge the data gap in many recommendation scenarios and benchmarks, we propose a novel framework of Propensity Estimation for Causality- to estimate the propensity score and exposure of each item for each user. Specifically, observe a pairwise characteristic that relates propensity scores and item popularity when the probability of user-item interaction is well controlled. (The observation is formalized as Assumption | and empirically validated in Sect. (42)) Based on the observation, we incorporate item popularity as prior knowledge to guide propensity estimation. Furthermore, bias of the estimated causal effect. The analysis enables us to investigate the factors that influence estimation and subsequently guide our model and experiment design.
Recommendation (PROPCARE),
present theoretical analysis on the
In summary, we compare previous propensity estimation and PROPCARe Fig. highlighting our key advantages: PROPCARE does not need propensity or exposure data at all, and incorporates prior information for robust estimation. The contributions of this paper include the following. (1) Our proposed framework bridges the gap in existing causality-based recommendation systems, where propensity score and/or exposure data are often unavailable but required for model training or inference. (2) We incorporate the pairwise relationship between propensity and item popularity as prior knowledge for more robust propensity estimation. We present a further analysis on the factors that influence model. (3) We conduct extensive experiments to validate the effectiveness of PRopCaRE through both quantitative and qualitative results.
Related Work
Causal effect estimation in recommendation While typical recommendation systems consider positive feedback or interactions like clicks and purchases as successful, it may be more beneficial to optimize the uplift interactions, also called the causal effect, solely caused by recommendations [18. However, obtaining the causal effect in real-world scenarios is challenging because of its

---

counterfactual nature [9]. Conducting online A/B tests to compare exposure strategies may be feasible but expensive and susceptible selection bias [77. To address these issues, several causal effect estimators have been proposed. The naïve estimator 30) assumes random exposure assignment to all user-item pairs, which is inconsistent with most recommendation scenarios. The inverse propensity score (IPS) estimator [30] incorporates the propensity score, defined as the probability of exposure [25], to overcome this limitation. Direct model estimators like CausCF [38] directly predict outcomes using parametric models based on different exposure statuses. A recently proposed doubly robust estimator 337 integrates a parametric model with the non-parametric IPS estimator for reduced bias and variance. However, these estimators require access to input data containing propensity scores and/or exposure data, at least during the training stage, which are often unavailable due to technical and privacy limitations.
Propensity estimation in recommendation Existing causal effect estimation approaches require exposure data and/or propensity scores at least in training, which are frequently unavailable subject the missing-not--n not-at-random (MNAR) issue [32]. Hence, we have to rely on their estimations. Some methods estimate propensity in heuristic way. such as using item popularity [30] or other side information (e.g., items participating
promotional campaigns) 28 However, these estimations lack personalization and may result in noisy results. Other approaches utilize interaction models (also called click models) [24 3 42 21 to relate propensity scores, relevance and interactions. However, without additional constraints, the interaction model alone can be difficult to optimize
will elaborate in Sect. 4.1 Besides, matrix factorization [16 38), linear regression [28), dual learning 21 and doubly robust learning [4] can also learn propensity scores, but they assume exposure data as training labels or known variables, which is incompatible with our setup without any observable propensity exposure data.
3 Preliminaries
Data notations Consider a typical recommendation dataset that contains only interactions between users and items, such purchases clicks. Let Yu,, E (0. denote the observed interaction between user u E {1,2.... U} and item i E (1.2...., 1). observed training user-item interaction data. Note that framework does not assume the availability of any additional data except the interaction data. Moreover, let Zu,: E {0, 1} denote an unobservable indicator variable for exposure, i.e., Zu iff item is exposed/recommmnded to user u. We use Pu.i to represent the propensity score, which is defined as the probability of exposure, i.e., Pa,i P(Zu, =
= {(Yu,)} denotes the collection of
Causal effect modelling Let Yo and Y E {0, be the potential outcomes for different exposure statuses. Specifically, Yl,, is defined the interaction between user and item i when has been exposed u. Accordingly, yo is the interaction when i has not been exposed to w. This setup assumes a counterfactual model: In the real world only one of the scenarios can happen, but not both. Subsequently, the causal effect Tu,s (-1,0, 1} defined as the difference between the two potential outcomes
(26). i.e., Tu,i YI, u In other words, Tu,i i to user u will increase the interaction between u and i and Tu, -1 means the opposite. Tu,i = 0 means recommending or not will not change the user's interaction behavior. Naturally, users, sellers and platforms could all benefit from recommendations that result in positive causal effects.
means recommending item
Causal effect estimation The causal effect cannot be directly computed based on observed data due to its counterfactual nature. Among the various estimators introduced in Sect. 2 direct parametric models [38] 37 are sensitive to the prediction eтror of potential outcomes [37]. Hence, high-quality labeled exposure data are required parametric models, which is not the setup of this work. To avoid this issue, adopt nonparametric approach, known as the inverse propensity score (IPS) estimator 30), for causal effect estimation as follows.
Zm Vu.i
(1 Zu,s)Y
fa.
(1)
Pu.i
1 Pu,i
3

---

Interaction model In line with prior works [2 42 39, we adopt interaction model that assumes the following relationship between interactions, propensity and relevance:
24
ya..
Pa.iTu,ãs
where yu, P(Y..; 1) is the probability of interaction between u and item and Tu, represents the probability that item is relevant to user u.
4 Proposed Approach: PROPCARE
In this section, we introduce our propensity estimation approach PROPCARE. We start with a naive approach, followed by our observation on prior knowledge, before presenting the overall loss for propensity learning and how the learned propensity can be used for causality- recommendation. We end the section by discussing theoretical property of our estimation.
4.1 Naïve propensity estimator
The overall objective is to estimate propensity scores and exposure from a more practical setup where only interaction data are observable. Since the propensity score Pu,: is the probability of exposure P(Zu, = 1), we focus on the estimation of propensity scores, whereas the corresponding exposure can be readily sampled based on the propensity. The interaction model in Eq. intuitively leads us to the naïve loss function below.
Luuive = Yu,i log (x..s (x.;; +,) (1 Yu,s) log(1 - fx( Op)s (x,,i 0,)), (3)
where Xu.i = fe [u, i; θe) is a joint user-item embedding output by a learnable embedding function fei fp and fr are learnable propensity and relevance functions to produce the estimated propensity score Pu,: and relevance probability fu,ts respectively. Note that each learnable function f. is parameterized by O., and implement each multi-layer perceptron (MLP)..
However, through the na've loss we cannot learn meaningful propensity and relevance functions (Sp and fr), since they are always coupled in a product and can be collapsed into one function. It is equivalent to learning a single interaction function, instead of learning each individual factor.
4.2 Incorporating prior knowledge
To avoid the above issue, solution is introduce prior knowledge further constrain the propensity or relevance function. In particular, it has been observed that more popular items will have a higher chance be exposed [1]. The popularity of item pop,, is defined based on the total number of observed interactions in the dataset, i.e., pop; Eunn Yu.s/ E
However, this observation [41]. while intuitive, not adequate in explaining the relationship between popularity and exposure. In particular, items with a higher interaction probability also tend to have higher chance be exposed, especially when prior exposure was decided by recommenders in the classical paradigm. To incorporate popularity prior toward propensity/exposuee estimation, we propose to introduce a control on the interaction probability, and formulate the following assumption.
Assumption (Pairwise Relationship on Popularity and Propensity) Consider a user and pair of items (i.j). Suppose the popularity of item is greater than that of j. and their interaction probabilities with user u are similar. Then it follows that item i more likely to be exposed to user than item j is.
The intuition is that, when user's interaction probabilities are similar toward two items and j. but item is more likely to be exposed to the user, the reason could be item i more popular than j. Our assumption essentially places control on the interaction probability to eliminate its influence on the exposure, and simultaneously isolate the effect of popularity on the exposure.
Empirical validation of Assumption 1
In the following, we examine our assumption by calcu- lating the fraction of item pairs that satisfy this assumption in three datasets, namely, DH_ _Hooriginal,
Also called the "click" model when the interaction refers click in literature.

---

ratio
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.1
0.5
0.1
0.5
0.1
0.5
inverse similarity DH_ original
inverse similarity DH_ personalized
Inverse similarity
ML
Figure 2: Histogram of item pairs (i.j) that satisfy Assumption The bins are based on the inverse similarity in interaction probabilities, Mu..j - Mu.al divided by (0. 10.......... 0.09, 0.1, 0.2,.... 0.5). That is, the first 10 bins have an equal width of 0.01 and the last 4 bins have an equal width of 0.1.
DH personalized and ML (see Sect. 5.1 for dataset descriptions). Specifically, first estimate the probability Vu,i of each interaction Yu,; using logistic matrix factorization [[T]. We also obtain the propensity score Pu, from ground truth values provided by the datasets (note that we only use the ground truth for evaluation purposes, not in model training or inference). Then, for each user u, we place item pair (i,j), where randomly sampled is paired with each of the remaining items, into several bins based on and j's similarity in their interaction probability with u. More specifically, each bin b contains (i, pairs such that ya.j Y/ falls into b's boundaries. Finally, we compute the ratio of (i,j) pairs consistent with Assumption
1 to the total pairs in each bin b, as follows.
U
item pairs (i:j) for user a bin b.. (pois - Pu,,) - pop,) > (4)
ratio;- U
# item pairs (i.j) sampled for user in bin b
We report the ratios in Fig. 2 It can be observed that when Wy., - yusl is smaller (i.e., and j's interaction probabilities with u are more similar), higher fraction of items pairs in the bin satisfy our assumption. In contrast, when Wy..j yu.al grows larger (i.e., the interaction probabilities are not well controlled and become less similar), the validity of the original observation [41) becomes weaker. In summary, the results on the three datasets demonstrate the validity of Assumption []]
Integrating prior knowledge Based on Assumption| 1
we utilize item popularity to inject prior knowledge on the probability of exposure (i.e., propensity score) through the following loss.
log (o( (x(x))) S. t. pop; > poDy, Vns R yn.j
While Eq. B models propensity in point-wise manner, Eq. (5) incorporates popularity as prior knowledge in a pairwise manner. The advantage is twofold. First, it decouples the propensity and relevance functions, using only item popularity which can be readily computed from the interaction data shown carlier without the need for external information. Second, by separating the estimated propensity of less popular items and more popular items, it prevents all predicted values from clustering in narrow range near 1 or O. This is beneficial in mitigating the issue of high variance caused by extreme values [37].
where a is the sigmoid activation and y is computed as ff
To materialize the control yu, & Dugj on the interaction probabilities in Eq. (5) we adopt the following loss that involves a soft version of Yus R Yo.j-
Sppp = -Ku,i. log [o(sg (fp(Xus) - (x(x,,))) + (sgn (fr(xu.)) - fx(x,)))))] (6)
where sgn;,; E {1, is the sign of (pop; - pop,) and Ku,i.j is a weighting function such that it will assign higher weight if yuu and Vn.5 are closer. Specifically, we choose Ku,i.f = -v.,), where 77 < 0 is a learnable parameter. Moreover, according to the interaction model in Eq. 2 for fixed yu a higher Pu. implies a lower Tu, This explains the additional constraint on the relevance function fr in Eq. (6). which will further improve model training.
4.3 Propensity learning
4.2 the naive loss essentially optimizes the interaction model, whereas the pairwise loss utilizes popularity as prior information for propensity learning. For more robust learning, we further take a global view on the distribution of propensity scores, which usually follow a long-tailed distribution 44] 42 In particular, we employ beta distribution to regularize the
Based on the discussions Sect. 4.1
5

---

propensity scores, as has been done in literature in modeling propensity other long-tailes quantities [5]. Overall, we minimize the following loss toward propensity learning:
min L = Em,. (Caavee + popp + μKL(QQ) (Beaaa, B)).
Here Q is the empirical distribution of all estimated propensity scores Pa. Beta 33) is a refer- ence beta distribution with parameters and 3 which are selected simulate a long-tailed shape. KL(-II-) computes the Kullback-Leible divergence between two distributions. À and μ are trade-off hyperparameters to balance different terms.
Finally, we the estimated propensity score P,,, to predict the exposure variable Zu Zu, 1 if Norm(fa.s) 2 e, and ( otherwise, where a threshold hyperparameter and Norm is normalization function such Z-score normalization. The overall training steps sketched in Algorithm 1 in Appendix A.
4.4
Causality- -based recommendation
We resort to DLCE 30 a state-of-the-ar causality- -baed recommender equipped with an IPS estimator. It takes interaction Ya. exposure Zu,i and propensity Pu,s input, and outputs a ranking score su.s for each user-item pair. Given a triplet (u, i,j) such that u is user and i + are randomly sampled from the item set, the loss DL.CE is defined as follows (30]
max(pu,i.xt) log (1 [1+e- ( +
Zu,Y
max(1 Pu,i, x0) log Bald)) (8)
(1 Zz.i)
where x², 2x and w are hyperparameters. We follow the standard training procedure of DLCE, except that we substitute the ground-truth exposure and propensity score with our estimated values Zu,i and the above loss. Hence, the entire training process for our propensity learning and DLCE do not require any ground-trum exposure propensity data. After DLCE is trained, for each user u, we generate a ranked list of all items based on the optimized S,,i.
pu.s. respectively,
4.5 Theoretical property
The performance of causality-based recommendation depends on how accurate we can model the causal effect in the user-item interactions. Although it has been established elsewhere [30] that the IPS estimator defined in Eq. ! is unbiased as long as exposure Zu and propensity score Pu, are correctly assigned, in setup only estimated propensity scores and exposure available. Thus, we characterize the bias of the IPS estimator when estimations are used instead.
Proposition 1 Suppose we replace the ground truth values of Zu,i and Pu. with the estimated Zu,s and Pu,i in Eq. ① respectively. Then, the bias of the estimated causal effect Pu,s i is
Pu.i + E 2n,s Zuni
Pa.i
E 2u
Zu,é
-1 Yi!,
-1) yo
Pa,i
Pu,s
We defer the proof to Appendix B. From the bias stated Proposition we make two further remarks to guide the learning and evaluation of propensity scores and exposure.
Remark The bias influenced by three major factors: Pu,s/ (u,p),)/( fu.s) and
E Zu, Zn.] Note that Pu,, Pu,i and Zu,i Zu the bias would be zero which is consistent with earlier findings 30
Remark 2 If the estimated Pu.s is extremely close to 0 or 1, the bias can be potentially very large.
The above proposition and remarks shed some light on what we should focus when estimating evaluate exposure and propensity score. On the one hand, since exposure is binary variable and the bias is influenced by E 2n,5 Zu,, we may evaluate with binary classification metries such as F1 score. On the other hand, since propensity is a continuous variable and estimations extremely close to zero or one should be avoided, regularizing the global distribution in Eq. (7) and ensuring a proper scale of the propensity scores would be useful.

---

Table 1: Statistics of datasets.
Dataset
#users
#items
Yu.4
Zu.i
Fu.i
Pu.i
DH_origh [H_oogiinal
2,309
1,372
.0438
.6064
.0175
.2894
DHI personalized
2,309
1,372
.0503
.6265
.0178
.4589
ML
943
1,682
.0676
.0593
.0733
.0594
5
Experiment
In this section, comprehensively evaluate the effectiveness of the proposed PROPCARE through both quantitative and qualitative experiments.
5.1 Experiment setup
Datasets We employ three standard causality- based recommendation benchmarks. Among them, DH original and DH peessonalezedd are two versions of the DunnHumby dataset [30,, which includes purchase and promotion logs at physical retailer over 93-week period. The difference in the two versions mainly lies in the derivation of ground-truth propensity scores as stated by Sato et al. [00) which are based on items featured in the weekly mailer in DH @H_original, and with simulated personalization factor in DH_ _H__ppsnnaalzed The third dataset MovicLens 100K (ML) movies and simulated propensity scores based the ratings
[29], which includes users' ratings and user behaviors. Note that PROPCARE do not require any propensity or exposure data at all. The ground-truth values are only used to evaluate model output. On each dataset, we generate the saaniing/vaiddatinnneess sets following their original work [30 29], respectively. We summarize each dataset in Tab. [[] listing the number of users (#users) and items (#items), well the average value of several key variables including the observed interaction (Y,,i), exposure (Zu,)), causal effect (Fw,i) and propensity (Pu,i). Further details can be found in Appendix C.I.
Baselines We compare PROPCARE with the following propensity estimators: (1) Ground-truth: Propensity score and exposure values are directly taken from the datasets. (2) Random: Propensity scores are assigned randomly between 0 and (3) Item popularity (POP): Propensity scores are assigned as item popularity normalized to (0, 1). (4) CJBPR [4]: An unbiased recommendation model that optimizes propensity and relevance alternately in point-wise manner. (5) EM (21): An recommendation model that learns propensity scores in a point-wise manner using an expectation- maximization algorithm.
Note that Ground- truth uses the ground-trutt values of propensity Pu,i and exposure Zu, directly as input to train DLCE [30]. All other baselines do not need such ground-truth values in any stage just as PROPCARE. In these methods, the estimated propensity Pa.; is used to further derive the exposure Zu, in the same way PROPCARE (see Sect. 4.3). Finally, we utilize the estimated values to train DLCE (see Sect. 4.4).
Parameter settings We tune the hyperparameters based on the validation data, following guidance in the literature. Specifically, in PROPCARE, the trade-off parameter À and μ set to 10 and 0.4, respectively, on all datasets, For the downstream causal recommender DLCE, we follow the earlier settings [30]. For other settings and implementation details, refer to Appendix C.2.
Evaluation metrics We evaluate the performance of causality- -bsed recommendation with CP@ 10, CP@ 100 and CDCG, whose definitions can be found in Appendix C.3. Additionally, we measure the accuracy of estimated propensity scores w.r.t. the ground-truth values using Kullback-Leibler divergence (KLD) and Kendall's Tau (Tau) 12), and that of estimated exposure using Fl score. Note that all metrics, except KLD, indicate better performance with a larger value.
5.2 Results and discussions
We first compare the performance of PROPCARE and the baselines, followed by analyses of model ablation, the regularization term, and various influencing factors. Additional experiments including
7

---

Table 2: Performance comparison on downstream causality- based recommendation.
DH OHiigini
DH personalized
ML
Methods
CP 101
CP
CDCG
CP 107
CP 1001
CDCG
CP 107
CP@ CP@1000
CDCG
Ground-truth
0658+.001
.0215+0011
1.088+.000
13044.001
0445#.001
1,49940000
22771
.1887.000
16.29+.006
Random
01544+001
4001
.7390-
044944004
0107= 005
83164 039
011244002
.0135+0005
13.16+t.076
POP
0200:1.000
.01134.000
76774.001 001
.457:1.000
00962.001
.84911.002
..1424.001
.0924.001
11.434.005
CJRPR
02634.001
00877+000
.77694.00 002
05644+008
0106-+005
88284-005
033
.420++0002
-18744001
9.953+4006
EM 0118:10011
.00671.001
.72471.000
..507...02
01214.001
87794
003
..4374.002
-114::1002
10.21::0111
PROPCARE
.03514.002
+016++0011
-9268+.00
.12704-001
0081 000
1.426 001
.018244.02
.0337+.002
13.80-+.011
Results are reported as the average of 5 runs (mean:tstd). Except Ground-truth, best results are bolded and runners-up are underlined.
Table 3: Performance comparison on propensity score (KLD, Tau) and exposure (F1 score) estimation.
Oriiginnal
DH personalized
ML
Methods
KLDJ
Tau?
Fl score?
KLD1
Taut
Fl scoret
KLD1
Taut
Fl score
Random
5141::001
.0002+.000
4524=
3.008=t002
0001
4463:t.021
11863:-002
.0002+0000
_4$11=.022
POP
5430:1.00
.472:1.000
-2851=000
4.728:.000
.66462.000
277244.000
.0615=0000
.4979=.000
.5050=t.00
CJBPR
3987..008 3379:1.011
2253
005
2.650+.022
64774.013
.28254.005
0230+ 006
.49562.045
.20
EM
6300+.002
.0834+
-4974
001
2.385+
.0934+-002
4954+-0009
001
.1321 0.21+.00
.3653++000
PROPCARE
.851+.023
.33314.065
.5846
006
0338
.470611.022
.609-1.017
.02044.005
005
38899.003
.88474.020
Results styled in the same way as Tab.
comparison to conventional recommendation methods, evaluation on an alternative backbone, and a scalability study are presented in Appendix D.
Performance comparison We evaluate PROPCARE against the baselines in two aspects: (1) The downstream causality-based recommendation using the estimated propensity and exposure; (2) The accuracy of the estimated propensity and exposure.
We first illustrate the performance of causality- based recommendation in Tab. 2 It is not surprising that Ground-truth achieves the best causal effect by incorporating actual propensity and exposure values in DLCF. However, since ground-truth values are often unavailable, we rely on estimations. Among all baselines, PROPCARE most closely approaches Ground-truth's performance. Notably, in the DH _HH_peronnllzeed dataset, PROPCARE exhibits only a 6.6% average decrease from Ground-truth across three metries, significantly outperforming the aecond-bess EM which suffers a 56.6% drop. Furthermore, PROPCARE surpasses the point-wise CJBPR and EM, implying the advantage of our pairwise formulation based on Assumption I
Next, we analyze the accuracy of propensity and exposure estimation in Tab. 3 Among the baselines, POP performs the best in Kendall's Tau. However, the causality metries of POP is poor (see Tab. 2 due to the ill-fit propensity distribution, reflected in its large KLD from the ground truth distribution. The estimation of exposure is also challenging for POP in most cases. In contrast, PROPCARE demonstrates outstanding performance in Fl score and KLD, leading to effective causal metrics. Although its Tau scores lag behind some baselines, robust distribution on propensity and accurate binary predictions of exposure still contribute to good causal performance. The results in Tab, B highlight that causality- recommendation influenced by multiple factors, rather than relying solely on a single aspect of estimation. We will discuss these influencing factors further toward the end of this part.
Ablation study To evaluate the impact of our key design motivated by Assumption I we derive five variants from Eq.
(6): (1) NO P: removing the constraint on estimated pu.s by deleting the term with (x,,)) fx( )i (2) NO R: removing the constraint on estimated f.,i by deleting the term with fx fo (xu.):: (3) NO P_R: removing Cpop entirely from the overall loss to eliminate Assumption T altogether; (4) NEG: reversing Assumption [ by replacing Sgn,,, with -Sgn,,, to assume that more popular items have smaller propensity scores; (5) K 1: setting all Ka,i.'s i to constant 1, resulting in equal weighting of all training triplets. Their causal performances are illustrated in Fig. 3 Comparing to the full version of PROPCARE, NO_R and NO_P show a small drop in performance due to the absence of additional constraints on propensity relevance, indicating that the pairwise loss is still partially effective. The drop in x = highlights the need for controlling the similarity between interaction probabilities. The further drop observed in NO PO_P.A R
8

---

NEG
NO 0.0
NO.
60.
PropCare
CoCo
13.5
1.2
13
0.7
12.5
DH Loooginnall
DH. jiw_personalzed
ML
Figure 3: Ablation study on PROPCARE.
1.6
15
0.95
14
CDCG
1.4
0.9
12
1.2
0.85
11
0.8
10
10
DH.original
pe.possonliized
ML
Figure 4: Effect of regularization term.
after removing the entire assumption further demonstrates the validity of our assumption and the effectiveness of Cpop. The worst performance of NEG indicates that contradicting Assumption | greatly affects the results of causality- -baee recommendation.
Effeet regularization We examine how effective the regularization term by varying the trade-off hyperparameter μ,, used in Eq. Z over the set {0, 0.01, 0.1, 0.2, 0.4, 0.8, 2, 4, 10). The results, shown Fig. 4 indicate that when μ is set to O. which means the regularization term is completely removed, the causal performance is significantly compromised. This demonstrates the advantage of regularizing the propensity with a beta distribution during the training PROPCARE. As y increases, the causal performance gradually improves and reaches its peak within the range of [0.2, 0.8). We suggest further exploration by fine-tuning the hyperparameter within this interval.
Factors influencing causality- -based recommendation We further investigate the factors crucial to causality-based recommendation, by injecting noises into ground-truth propensity or exposure values. In Fig.5lai a). we randomly flip fraction of the exposure values while using the tround-truth propensity scores for DLCE training. The flipping ratios range from {0, 0.01, 0.05, 0.1, 0.15, 0.2, 0.3. 0.4, 0.5). In Fig. b), we add Gaussian noises to the propensity scores, where the variances of the noise range from (0,0.1,... 0.5) while using the ground-truth exposure. The results Fig. 5 consistently show decrease in performance due to misspecified exposure or propensity. To further examine the correlation between estimation accuracy and recommendation performance, we create seatter plots in Fig. for the DH original dataset. Each data point represents baseline labeled with its name. The plots reveal a general trend where the CDCG is correlated with the accuracy estimations.
1.2
1.5
18
1.2
1.5
16.6
EDCG
0.9
1
15
1
1.4
16.3
12
0.6
0.5
0.8
1.3
16
0.3
ratio
noise
DH original
DH_ personalized
ML
DH original
DH perpoonooaleedd
ML
(a) Ground-tuth propensity score randomly flipped exposure
(b) Noisy propensity score ground-truth exposure
Figure 5; Analysis factors that influence causality-bait recommendation performance.

---

PropC are
PrgoCare
ProaCare
CDCG
0.9
0.9
0.9
0.8
CJ@PR
POP
0.8
CJ CI
POP
0.8
POP
Random
Random MEM
BPR
Random
EM
0.7
0.7
0.7
0.3
0.5
0.7
0.4
0.2
0.4
0.6
KLD
Kendall's Tau
F1
Figure 6: Correlation analysis on factors influencing recommendation.
Table 4: Case study of an anonymous user.
Ground-truth
POP
CJBPR
PROPCARE
garlic bread (1176)
bananas (1310)
fluid milk (1169)
infant soy (1232)
cleansing wipes (737)
toilet tissue (742)
bananas (1310)
fluid milk (1169)
fluid milk (1 (1169)
fluid milk (1 (116
cereal (1090)
bananas (1310)
primal (807)
white bread (675)
strawberries (834)
pure juice (1277)
alkaline batteries (754)
tortilla chips (634) margarine tuhs/howls (1245)
coffee creamers (1169)
Hach column represents the recommendation list output by DLCH trained with the estimated propensity and exposure by the corresponding baseline. The purchased items are highlighted in bold. Items with positive causal effect (Tu.s 1) and negative causal effect (To., 1) are marked by and 4, respectively, and unmarked items have causal effect (Tu.. 0). Numberr brackets the popularity ranks in the training set.
5.3 Case study
We conduct case study to demonstrate the advantages of PROPCARE in practical ranking- based recommendation scenario. In Tab. we analyze the top-5 recommended items for an anonymous user with ID 2308 in the DH_ personalized dataset. In the first column, by utilizing ground-truth propensity scores and exposure, DLCE effectively generates a ranking list where most items have positive causal effect. All items with a positive causal effect were eventually purchased, achieving the goal of causality-basse recommendation. Comparing the lists generated by CJBPR and PROPCARE, it is evident that the associated causal effects of the purchased items differ. For example, in the CJBPR list, recommending "strawberries" has zero causal effect, indicating that the user could have still purchased it even without recommendation. In contrast, PROPCARE recommends "infant soy", which has positive causal effect, making it more ideal choice. Overall, given the list recommended by CJBPR, the user would only purchase "strawberries" and "fluid milk". However, given the list from PROPCARE, in addition to "infant soy" and "fluid milk", the user may still purchase "strawberries" even without being recommended due its zero causal effect. Besides, POP tends to recommend popular items but with lower causal effect, even including an item with negative causal effect. The results suggest that POP not an appropriate tool for estimating propensity scores the context of causality- recommendation.
6 Conclusion
In this paper, we introduced PROPCARE, a propensity estimation model for causality-basae recom- mendation systems without the need to access ground- truth propensity and exposure data. Leveraging our observation on the pairwise characteristics between propensity scores and item popularity, we formulated key assumption and incorporated it prior information to enhance our estimation, thereby improving causality-based recommendation. theoretical analysis was presented to under- stand the factors influencing the bias in estimated causal effects, thereby informing model design and evaluation. Empirical studies demonstrated the superiority of PROPCARE over the baselines. Future research avenues include exploring direct exposure estimation without propensity scores, and investigating parametric causal effect estimators that are potentially more powerful.
10

---

Acknowledgments
This research is supported by the Agency for Science, Technology and Research (A STAR) under its AME Programmatic Funds (Grant No. A20H6b0151). Any opinions, findings and conclusions or recommendations expressed this material are those of the author(s) and do not reflect the views of the S*TTR Dr. Yuan Fang also acknowledges the Lee Kong Chian Fellowship awarded by Singapore Management University for the support of this work.
References
[1] Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. Unbiased learning to rank with unbiased propensity estimation. In ACM SIGIR Conference on Research and Development in Information Retrieval, pages 385-394, 2018. 8.
[2] Stephen Bonner and Flavian Vasile. Causal embeddings for recommendation. ACM Conference on Recommender Systems, pages 104-112, 2018.
[3] Alexey Borisov, Ilya Markov, Maarten De Rijke, and Pavel Serdyukov. A neural click model for web search. In ACM International Conference on World Wide Web, pages 531-5411 2016.
[4] Richard K Crump. Joseph Hotz, Guido W Imbens, and Oscar A Mitnik. Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1):187-199 2009.
[5] James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, et al. The YouTTube video recommendation system. In ACM Conference on Recommender Systems, pages 293-296, 2010.
[6] Zhengxiao Du, Xiaowei Wang, Hongxia Yang, Jingren Zhou, and Jie Tang. Sequential scenario- specific meta learner for online recommendation. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2895-2904, 2019.
[7] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 639-648, 2020.
[8] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In ACM International Conference on World Wide Web, pages 173-182, 2017.
[9] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[10] Dietmar Jannach and Michael Jugovac. Measuring the business value of recommender systems.
ACM Transactions on Management Information Systems, 10(4):1-23, 2019.
[11] Christopher C Johnson. Logistic matrix factorization for implicit feedback data. Advances in Neural Information P Processing Systems, 27(78):1-9) 2014.
[12] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/)):811999 1938.
[13] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-31) 010:303-7,00 2009.
[14] Haoxuan Li, Chunyuan Zheng, and Peng Wu. StableDR: Stabilized doubly robust learning for recommendation on data missing not at random. In International Conference on Learning Representations, 2023.
[15] Xueqi Li, Guoqing Xiao, Yuedan Chen, Zhuo Tang, Wenjun Jiang, and Kenli Li. An explicitly weighted gen aggregator based on temporal and popularity features for recommendation. ACM Transactions Recommender Systems, 2023.

---

[16] Dawen Liang, Laurent Charlin, James McInemny, and David M Blei. Modeling user exposure in recommendation. In ACM International Conference on World Wide Web, pages 951-961, 2016.
[17] Greg Linden, Brent Smith, and Jeremy York. Amazon. com recommendations: Item-to-item collaborative filtering. IEEE Internet computing, 7(1):76 80, 2003.
[18] Huishi Luo, Fuzhen Zhuang, Ruobing Xie, Hengshu Zhu, and Deqing Wang. A survey on causal inference for recommendation. arXiv preprint arXiv:2303. 2023.
[19] Ioannis Paparrizos, Barla Cambazoglu, and Aristides Gionis. Machine learned job recom- mendation. In ACM Conference on Recommender Systems, pages 325-328, 2011.
[20] Greg Pass, Abdur Chowdhury, and Cayley Torgeson. A picture of search. In International Conference on Scalable Information Systems, pages 1-es, 2006.
[21] Zhen Qin, Suming Chen, Donald Metzler, Yongwoo Noh, Jingzheng Qin, and Xuanhui Wang. Attribute-based propensity for unbiased learning in recommender systems: Algorithm and case studies. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2359-2367, 2020.
[22] Steffen Rendle. Factorization machines. In IEEE International conference on Data Mining, pages 995-1000, 2010.
[23] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thies BPR: Bayesian personalized ranking from implicit feedback. In Conference on Uncertainty in Artificial Intelligence, pages 452-461, 2009.
[24] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: estimating the click-through rate for new ads. In ACM International Conference on World Wide Web, pages 521-530, 2007.
[25] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
[26] Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688, 1974.
[27] Masahiro Sato. Online evaluation methods for the causal effect of recommendations. ACM Conference on Recommender Systems, pages 96-101, 2021.
[28] Masahiro Sato, Janmajay Singh, Sho Takemori, Takashi Sonoda, Qian Zhang, and Tomoko Ohkuma. Uplift- based evaluation and optimization of recommenders. In ACM Conference on Recommender Systems, pages 296-304, 2019.
[29] Masahiro Sato, Janmajay Singh, Sho Takemori, and Qian Zhang. Causality-aware neighborhood methods for recommender systems. In Advances in Information Retrieval: European Conference on IR Research, pages 603-618. 2021.
[30] Masahiro Sato, Sho Takemori, Janmajay Singh, and Tomoko Ohkuma. Unbiased learning for the causal effect of recommendation. In ACM Conference Recommender Systems, pages 378-387, 2020.
[31] Amit Sharma, Jake M Hofman, and Dunean J Watts. Estimating the causal impact of recommen- dation systems from observational data. In ACM Conference Economics and Computation, pages 453-470, 2015.
[32] Harald Steck. Training and testing of recommender systems on data missing not at random. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 713-722, 2010.
[33] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In ACM International Conference Information and Knowledge Management, pages 1441-1450, 2019.
12

---

[34] Aaron Van den Oord, Sander Dieleman, and Benjamin Schrauwen. Deep content-based music recommendation. Advances in Neural Information Processing Systems, 26, 2013.
[35] Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. KGAT: Knowledge graph attention network for recommendation. In ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 950-958, 2019.
[36] Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. MIND: A large-scale dataset for news recommendation. In Annual Meeting of the Association for Computational Linguistics, pages 3597-3606, 2020.
[37] Teng Xiao and Suhang Wang. Towards unbiased and robust causal ranking for recommender systems. In ACM International Conference on Web Search and Data Mining, pages 1158-1167, 2022.
(38) Xu Xie, Zhaoyang Liu, Shiwen Wu, Fei Sun, Cihang Liu, Jiawei Chen, Jinyang Gao, Bin Cui, and Bolin Ding. CausCF: Causal collaborative filtering for recommendation effect estimation. In ACM International Conference on Information and Knowledge Management, pages 4253-4263, 2021.
[39] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Adversarial counterfactual learning and evaluation for recommender system. Advances in Neural Information Processing Systems, 13:15515-33526, 2020.
[40] Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for nissing-not-at-randoos implicit feedback. ACM conference on Recommender Systems, pages 279-287, 2018.
[41] Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. Causal intervention for leveraging popularity bias in recommendation. In International ACM SIGIR Conference Research and Development in Information Retrieval, pages 11-20, 2021.
[42] Ziwei Zhu, Yun He, Yin Zhang, and James Caverlee. Unbiased implicit recommendation and propensity estimation via combinational joint learning, In ACM Conference Recommender Systems, pages 551-556, 2020.
13

---

Appendices
A Training procedure of PROPCARE
We present the pseudocode for training our proposed PROPCARE in Algorithm | The training steps involve calculating various loss terms including Laavve pup and the regularization term. We update all learnable parameters based on the total loss defined in Eq. (7) of the main text.
Algorithm 1: Training PROPCARE
Input: Observed training interaction data D.
Output: Model parameters θ
Initialize model parameters O
while not converged do
foreach user-item pair (u, i) in D do
Compute Cuave by Eq. (3) of the main text; Sample item j for each pair (u, from {1,2..... ,..
Compute Cppp by Eq. (6) of the main text;
Compute the total loss in Eq. (7) of the main text;
end
Update by backpropagation the loss Eq. (7) of the main text; end
return θ
then analyze the time complexity of the training procedure. We first consider the computation of Cuavve. This involves three MLP models: fe. fp. and f,. As we have to compute Xu, Pu,i and fu,i for each user-item pair, each MLP incurs a time complexity of Onn). where n = [D], the number of user-item pairs in the training data. To further find the interactions [Vu,))) we need to compute the product between Pu,s and Fu, which also has linear time complexity of (n)). Next, to compute fppp- we need sample another item for each user-item pair in the training data, which has time complexity of (n)) For each tuple (w, i, we need to further compute Pu.j and fugj. each taking ((n) time. Finally, the time complexity the empirical distribution is based on all estimated propensity scores for training pairs. In summary, series of (n) procedures are carried out for the user-item pairs in the training set D, resulting in an overall linear time complexity of (n). This demonstrates the scalability of our proposed PROPCARE, which we further analyze empirically in Appendix D.2
compute the regularization term is also O(n), as
B
Bias of the estimated causal effect
Here, present the the proof of Proposition 1, which have appeared in the main text, Sect. 4.5.
Proof As defined earlier (30), can model the potential oulcomes as follows,
y Zu.iYu.
(a.1)
Pu.i
yo: (1 Zu.s) 1 Pu.i
(a.2)
Note that in Eqs. a. and (a.2), the Yu,á 's on the right-hand side can be replaced with Y.,, and Y respectively. This substitution valid because when Zu 1, the Ya, term Eq. (a.Iy equivalent to u.i by definition. In this scenario, 8n Zun. = O, the Yus term in Eq. (a.2) is equivalent to YO Therefore, the IPS estimator for the causal effect, in Eq. (1) of the main texi, can rewritten as
Yu is always regardless of the value of u.x Similarly, when
Zu Y
(1 (1-Zu,i) Yo
f
(a.3)
Pu.i
1 Pu,i
14

---

As established earlier (30]. if the propensity score Pa.: and exposure Zu are correctly assigned, the expectation of the estimated causal effect from the IPS estimator is
E [Fu,]] Y
Y Tu,i-
(a.4)
That is say, if we have ground-truth propensity score well as exposure, the estimated causal effect is unbiased. If we have only ground-truth exposure but have to estimate propensity scores by substituting Pa.: with Pu in Eq. da.3). denote the resulting estimator for the casual effect as fa,s Then, the expectation and bias of Fu,i are
Pu,iXu uii
(1 -u.P,)
E P
(a.5)
Du.i
pu,i
Bias(Fu.)) E [Fun]] Tu,i
Puçi
-1 Yuu: -
Pu.i
yo
(a.6)
Pug
Pu,i
Finally, in our setup, both Zu and Puii are estimated. Hence, denote the resulting estimator based on Zug and S u.. We obtain its bias relative Rygs as follows.
Bias (aa,) Bias(Fl,
E [Pa.a]
Vu.
(1
Zu.
Zu,iVo
(1 Zu, Y
Puli
1 -uu
Pu.i
1 Pu.i
(Zu,i Zui
(Zu,i Zu, Me.
Du,i
- Pags
E
Zuss Zu,
E
Zu,s Ze,
Y1
+
(a.7)
Pu.i
Pu,i
By adding Eqs. a.6 and pa.7
we are able to obtain the bias of 9'.
Bias( (FW,))
Bias(
+ Bias (F",,) - Biiss(Fu.))
Puçi + E Zuni Zuri]
1 - Puái
E Zu.i
Zu.i
-1 Yil
Pu,s
puti
(a.8)
This concludes the proof of Proposition I.
C Additional experimental settings
We describe more details on the datasets, implementation and evaluation metrics.
C.1 Descriptions of datasets
We introduce additional details on data generation and splitting.
Data processing and generation We perform data processing and generation steps per the earlier
studies the DunnHumby? (DH) 30 and MovieLens (29 (ML) datasets.
Specifically, for DH, the items that appear in the weekly mailer are deemed as recommended (i.e.. exposed) items. The empirical distribution of Y can be found by tallying the weeks in which item i
2The raw data are available http://wwwww..aaaaaa.....//////a
Thhe raw data are available at httpss http:p///ggooooooo.oooo/aaaaaaaa
15

---

was both recommended to and purchased by user u (or purchased but not recommended when dealing with ). The ground -ruth values of Y. and yu, are then sampled from their respective empirical distributions, which are used to calculate the ground-truth causal effect, as follows.
Tu,i Y 2, Y
(a.9)
Subsequently, two different ways of simulating the ground-truth propensity scores have been at- tempted. In DH original.. the propensity score Pu, is defined based on the number of weeks in which the item was recommended while the user visited the retailer during the same week. For DH peLpessonllized, the propensity score Pu,: is established based on the position of item i in sim- ulated ranking based on user u's probability of interaction with i. The ground-trui exposure Zu is then sampled from a Bernoulli distribution whose parameter is set to the propensity score Pu,i. Finally, the ground-trume -ruth interaction can be computed as follows.
Yu,= Zu,aYu, u,s + (1 -Z,,,,)
(a. (a.
For MI., the empirical distributions of YI, and yo, are derived from the interaction log using matrix factorization- based techniques. The propensity score is determined by simulated ranking for each user, similar to DH personalized. Subsequently, the ground-truth values of the exposure, causal effect and interaction is sampled and established following the same process in DH.
Data splitting The above data generation steps are applied to both DH and ML datasets to generate training, validation and testing sets. For the DH datasets, the data generation process is repeated 10 times to simulate the 10-week training data, once more simulate the 1-week validation data, and 10 more times to simulate the 10-week testing data. For the ML dataset, the generation is repeated once each generate the training, validation, and testing data, respectively.
is worth noting that the data generation and splitting processes are dependent on some form of simulation. We utilize such "semi-simulated"" data for number of reasons. First, the true causal effects are not observable due to their counterfactual nature. Second, ground-truth propensity scores and exposure are often not available in public datasets, which nonetheless essential for model evaluation (even though they are not required for our model training). Third, while some datasets 20 33 6 do include information on item impressions or exposure statuses, they often provide one-sided view of the situation. Specifically, the vast majority, if not all, of the item interactions are preceded by prior impressions. This leaves limited scope for investigating item interactions that occur without prior impressions.
C.2 Implementation details
Let us first present key implementation choices regarding propensity and exposure modeling in PROPCARE. On all datasets, we set a 0.2 and / = 1.0 for the Beta distribution used in the regularization term. To derive Zu,i based pu, we implement Norm as Z-score normalization function, such that Zu,s = 1 if Norm (p..:) 2 c, where the threshold c is set to 0.2 for DH_ _H_original and DH personalized, and 0.15 for ML. To address the bias introduced by Pu.s. we employ a practical trick scale the estimated propensity by constant factor c, i.e., Pu.s t c X Pu.i. prior to training DLCE. This technique aims to further minimize the disparity in the scale between Pa, and pu,, in order to improve the accuracy of causal effect estimation, according to the theoretical analysis in Sect. 4.5. For DH_ _ooigiinall and DH_ "tlpersonalized, the scaling factor is set to 0.8, while for MI. it is set to 0.2. The hyperparameters e and c are tuned using validation data. In particular, we perform a grid search where is searched over the range (0,1) in steps of 0.05 and is searched over the range (0,1) in steps of 0.1.
With regard to the model architecture for PROPCARE, we randomly initialize x,, and X; E IR 128 as user and item ID features. The embedding model fa takes (x,l(x;) as input and is implemented as an MLP with 256, 128 and 64 neurons for its layers. fp and fr are both implemented as MLPs with 64, 32, 16, neurons for the hidden layers and an output layer activated by the sigmoid function. Except the output layer, all layers of le and hidden layers of f, and f, are activated by the Lea function. Finally, PROPCARE is trained with a stochastic gradient descent optimizer using mini-batches, with batch size set to 5096.
16

---

Table a.l: Performance comparison with conventional interaction- based recommenders.
DH OHiiina
[HLpersonalized
ML
Methods
CP@
CP@ 100
CDCG
CP@101
CP@ 100)
CDCGT
CP@
CP@ 1001
CDCGT
MF
.0206-1.001
.0118 000
88569_002
04334-001
.019-+-000
.699-4.002
.,460+-0044
..20-+-0022
9.421+-023
BPR
030111000
0138:0000 000
89831.002
04761.001
02455..00
1.0141.011
.408:.002
-199:4.001
9.8981.028
LightGCN
0209:4.001
0149=000) 001
-1133
.0263+.001 0821
1.0954.002
-.34245006
-.177#0002
10.16=.050
PROPCARE
.031110002
.0156=0001
_268=.005
.12701.001
.03811.000
1.4261.001
.01824.002
.0337:.002
13.80:.011
Results are reported as the average of 5 runs (mean:tstd). Best results are bolded.
For the baseline CJBPR, we have used its authors' implementation and modified its mini-batch setting and optimizer to be identical to PROPCARE. For the baseline EM, we have implemented their proposed propensity model in Python. To conduct downstream causality- -baed recommendation, we have used the authors' implementation of DLCE following the settings in their work [30]. For fair comparison, the normalization function Norm and constant scaling factor are identically applied to all baselines, which generally improve the performance across the board.
We implement PROPCARE using TensorFlow 2. in Python 30. All experiments were conducted on a Linux server with AMD EPYC 7742 64-Core CPU, 512 GB DDR4 memory and four RTX 3090 GPUs.
C.3 Evaluation metrics
Unlike commonly used metrics like precision and NDCG that reward all positive interactions re- gardless whether the item is exposed not, we use variant of them that only reward positive interactions resulting from item exposure. Concretely, we use the Causal effect-based Precision (CP) and Discounted Cumulative Gain (CDCG) as defined in existing work 30
1 CP@K
Σ
1 (rank, (8w.i) S K) Tu,
(a.
U
K
i=1
CDCG 1 U
ΣV u=1
Tu.
(2.12)
log2 (1 (au.s((s.s)).
where (-) is indicator function, and ranka($w.t) returns the position of item in the ranking list for user u as determined by the ranking score 8u.i. In our paper, we report CP@ 10, CP@ 100 and CDCG. Note that since Tu,i can be 1, these metrics be negative.
D Additional experiments
We present additional empirical results on the comparison to conventional recommender systems, as well as the robustness and scalability of PROPCARE.
D.1 Comparison to interaction- recommenders
To demonstrate the advantage of PROPCARE over conventional interaction- recommender systems, we compare with three well-known models, namely Matrix Factorizatior (MF) [3] Bayesian Personalized Ranking (BPR) 23 and LightGCN M Note that we use only interaction data Yu, as training labels for these methods and rank the items for each user based the predicted In.i. as in their classical paradigm. We evaluate the causal performance with CP@ 10, CP@ 100 and observe that the conventional interaction-base methods
CDCG in Tab. a.1 From the results do not demonstrate strong causal performance.
A fundamental reason is that the conventional models indiscriminately reward all positive interactions, regardless whether the candidate items have been exposed or not. To delve deeper into the impact
Available at http:://wthttttooeeeeee//a/aaata
Available in ancillary files "Implementation in htaplaayllleaaalpppp///aaaaaoarr wttp:o/r/g/abb/0..0005555555 045633
http://aaaaa.rrrrrr///////////// 04563
files
Implementation available at http::///gstuub.ooomuuuaeeedeeee
17

---

Table a.2: Influence of exposure status on causal effect.
B T Yu, 1,
=
E Tu .u., Zu.;
0)
DH _riigiiaaa DH persooonnalize
.8882
-901
.8882
-9855
MI.
.8623
-8818
Table a.3: Employing CausE as the causality- recommendation backbone.
DH origiiia
DH personalized
ML
Methods
CP@ 101
CP@ 1001
CDCG
CP@ 101
CP@ 1001
CDCGt
CP@ 101
CP@ 1001
CDCG
Ground-truh
0095:.00000 002
021662.002
1.020=.004
.0663.0003
.02412.002
1.105=.003
.1849..002
.1438..004
15.25:.007
.059
.09974-000
04274 002
0159+.001
9616+.002
-.191+001
-036-002
12.274-006
CIBPR
0073+-001
01000+000 005
78899 003
0451
0165=.003
9621=0005
-217=
12.05:t0066 -044+.001
EM
0065:.000
0103=+001
-7802=002 002
0478+.001
0166==001
9819=.006
-197=.002
-0412.002
12.24::009
PROPCARE
0123:1.011
.0114:.001
8084=.001
0050
.0201:.001
1,052 0.00
-.138=0001
-0352.003
12.40:1.099
Results are reported as the average runs (mean.istd), Except Ground truth, best results are bolded and runners up are underlined.
of different exposure statuses on the causal effect of user-item pairs with positive interactions, we calculate [T (Y., = 1, Zu.: 1] and E[Tu, Ya. : 1, Zu,s = 0] for each dataset, as presented in Tab. a.2] The divergent results given different exposure statuses imply that exposure plays significant role the causal effect involving positive interactions. Conventional interaction based models ignore exposure and may rank items in a way that negatively impacts the causal effect. Between the two DH datasets, DH 2HH_peroonlized exhibits lower value of [B(TT.. V = 1, Zu.i = 0), which explains the greater causal performance gap between the conventional models and PROPCARE than the gap on DH DHooriginal. This greater gap arises because recommending items with Y,,; 1, Zu,i lowers the causal effect more on peppessonllized than on DH oorigiaa...
D.2 Robustness to alternative backbone and scalability analysis
To show the robustness of PROPCARE, opt for Caus
c 2 another causality- basedd recommender,
as an alternative backbone to DLCE. Like DLCE, CausE makes causality-basee recommendation given the estimated propensity and exposure data from PROPCARE or the baselines. The results in Tab. show similar pattern as using DLCE in the main text. That is, our PROPCARE consistently outperforms other baselines even with a different causality- based model as the backbone.
We further examine the scalability of our proposed PROPCARE on increasingly larger datasets, namely, MovieLens IM MovieLens 10M and MovieLens 20M In Tab we report the training times of both PROPCARE and DLCE, in hours. We observe that the training of PROPCARE generally follows linear growth, line with the time complexity anlysis in Appendix A Moreover, PROPCARE only presents marginal overhead on top of the backbone DLCE, showing its feasibility in working with existing backbones.
Table a.4: Training times for PROPCARE and DLCE, in hours.
MovieLens IM
MovieLens 10M
MovieLens 20M
PROPCARE
.0814 2.1759
1.494
3.170
DLCE
22.658
40.658
Available aulC:p/r//goooooaa...ore/e/ttttt htt./daaaasstts/ovvvee..s.s
18


Reviews:
[{"review_id": "5cUMhSnUSq", "reviewer": "Reviewer_KaLM", "summary": "This paper focuses on causality-based recommendation system, by proposing a PROPCARE method that estimates the propensity score by using its correlation with popularity. The motivation is well stated and related work is well discussed. Through experiments, the proposed method outperforms the baselines.", "questions": "1. How should the author distinguish the relationship between popularity and propensity as correlation or causality?\n2. What's the recommendation accuracy comparison between  PROPCARE and state-of-the-art recommendation models?\n3. How is the model's performance and scalability to large-scale datasets?", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to follow. The motivation is clearly stated and the authors did a good survey around related work.\n2. The proposed method is sound, with some theoretical analysis.\n3. Experiments are conducted carefully with case studies to demonstrate the effectiveness of the proposed method.", "weaknesses": "1. In order to model causality, the authors should make clear statement about whether the relationship between popularity and propensity is correlation or causality, which significantly affects the modeling foundation of this work.\n2. Technical foundation is limited. From ML perspective, the technical contribution in this work is to combine point-wise and pair-wise modeling by combining the popularity with propensity estimation, which are all well-developed methods.\n3. The baselines used in this method are not state-of-the-art, which lacks of confidence to evaluate the contribution of this work. Causality-based recommendation should share the same goal of general recommender systems, which should include the state-of-the-art methods to compare the recommendation accuracy.\n4. The datasets used in this work is quite small, it's worthy to check the performance and scalability of the proposed method in large-scale datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging our motivation, writing and experiment. we will give answers to each question asked by this reviewer below.\n\n**Q1:** Distinguish popularity and propensity as correlation or causality.\n\n**A1:** Thanks for the question. To model propensity (the probability of exposure), we first identify that item popularity directly affects the exposure status of the item, which has been also established by previous works [a,b]. Based on that fact, we propose Assumption 1 to estimate propensity using popularity as a proxy. Though the propensity is highly correlated to the popularity, it is hard to say that propensity and popularity have a direct causal relationship. First, according to interaction model stated in Eq.(2), the propensity score is also affected by interaction and relevance probabilities. Second, there might be other confounder such as item stocks that affects both propensity and popularity. Therefore, **the relationship between popularity and propensity is generally correlation, but not always be causality.** In our model (especially Assumption 1), we require the propensity and popularity to have at least correlation relationship. While two variables in a causality relationship are also correlated, it will not affect our modelling foundation whether it is causality or correlation.\n\n**Q2:** About the recommendation accuracy comparison between PROPCARE and state-of-the-art recommendation models.\n**Weakness 3:** About the baselines used in this method are not state-of-the-art. And about the same goals between causal recommendation and conventional recommendation systems.\n\n**A2 and response to weakness 3:** We have already compared the causal performance of traditional recommendation models (including **MF**, **BPR** and **LightGCN**) with PropCare (using DLCE as the causal backbone) in **Appendix D.2**, due to page limit. Here we report the comparison of traditional recommendation accuracy with LightGCN, a state-of-the-art conventional recommender below. \n\nLightGCN:\n|        | Prec@10 | Prec@100 | DCG   |\n|-----------------|---------|----------|-------|\n| DH_original     | .1029   | .0475    | 2.475 |\n| DH_personalized | .0737   | .0390    | 2.505 |\n| ML              | .3220   | .2151    | 14.78 |\n\nPropCare:\n|      | Prec@10 | Prec@100 | DCG   |\n|-----------------|---------|----------|-------|\n| DH_original     | .1967   | .0764    | 3.097 |\n| DH_personalized | .2680   | .0939    | 3.615 |\n| ML              | .6168   | .4109    | 18.21 |\n\nFrom the results it can be found that **our model consistently outperforms LightGCN in terms of accuracy metrics.** It is because by definition, items with high causal effect will likely be interacted when they are recommended. In contrast, items with high interaction probabilities do not necessarily have high causal effect. In other word, causality-based recommender has a similar but more focused optimization objective than conventional recommenders like LightGCN. The results also support your comment ``Causality-based recommendation should share the same goal of general recommender systems''.\n\nWe clarify that as an emerging topic, there are not many approaches for estimating propensity or exposure without additional information. Previously, one popular way is to directly use item popularity (named POP in our experiment as a baseline) to simulate propensity. Besides, we also compare our method with EM and CJBPR, which are both published in recent premier conferences. There are some other works for propensity/exposure estimation, but as we have stated in the related works in the original paper, they assume observable exposure data as training labels [f,g]. For the backbone model, DLCE is also a SOTA open-sourced causal recommendation model. \n\n**Q3:** Model's performance and scalability to large-scale datasets.\n\n**A3:** It is a great suggestion to test our model on larger dataset. However, there is no other publicly available datasets for evaluation of propensity estimation or downstream causal recommendation. It is because to evaluate them we must have ground-truth propensity and causal-effect, which are usually unavailable due to privacy or technical constraints. Currently, only DH\\_original and DH\\_personalized provided by [c] and ML provided by [d] provide such ground-truth information. Those three datasets are widely used benchmarks in the evaluation of causality-based recommendation systems, like in [c,d,e]. \n\nNevertheless, to show the scalability of our method, we train our model on 3 popular larger-scale datasets, namely MovieLens-1M, MovieLens-10M and MovieLens-20M. We report the overhead incurred by training PropCare for propensity estimation against DLCE, the backbone causal recommender below.\n\n|    Training time (hours)      | 1M     | 10M    | 20M    |\n|----------|--------|--------|--------|\n| PropCare | .0814  | 1.494  | 3.170  |\n| DLCE     | 2.1759 | 22.658 | 40.658 |\n\n\nNote that due to the missing of ground-truth propensity/exposure and causal-effect, we are unable to evaluate the model performance. Yet, **the results show our PropCare is scalable to larger datasets and incurs only a marginal overhead on top of the backbone DLCE.**\n\n**References**\n\n[a] Zhang, Yang, et al. \"Causal intervention for leveraging popularity bias in recommendation.\" SIGIR. 2021.\n\n[b] Tianxin Wei, et al. \"Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System\". KDD. 2021.\n\n[c] Masahiro Sato, et al. 2020. \"Unbiased Learning for the Causal Effect of Recommendation\". RecSys. 2020.\n\n[d] Masahiro Sato, et al. \"Causality-aware neighborhood methods for recommender systems\". ECIR. 2021.\n\n[e] Teng Xiao, et al.\"Towards Unbiased and Robust Causal Ranking for Recommender Systems\". WSDM. 2022.\n\n[f] Dawen Liang, et al. \"Modeling user exposure in recommendation\". WWW. 2016.\n\n[g] Masahiro Sato, et al. \"Uplift-based evaluation and optimization of recommenders\". RecSys, 2019\n\n\n\n"}, {"review_id": "Spd0bGybL2", "reviewer": "Reviewer_Vbya", "summary": "The authors propose a propensity estimation/learning method based for unbiased recommendations. The method assumes no external data and only uses the user interaction data for learning. The main idea is well explained in Assumption 1 in the paper, which states that for two items with similar click/interaction probability, the more popular item is likely to be recommended to the user over the other. Authors incorporate this assumption in Eq. 6. The propensities learned via the proposed method perform better as compared to the baselines on several benchmark datasets. ", "questions": "- What is the effect of the regularization term in propensity training? An ablation experiment could help, additionally, an experiment with varying $\\mu$ values could also give some insight into this. \n- If authors could elaborate more on the ML dataset setup (ground-truth propensity, binary feedback from ratings, etc)? (see point 2 in the weakness section)", "limitations": "Limitations are correctly addressed.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is very well-written and very easy to follow. The math is also intuitive to understand, overall a good job by the authors in writing. \n- The proposed method (Eq 6) follows intuitively from the main assumption (Assumption 1). \n- Experiments are performed on multiple benchmark datasets, and the proposed method outperforms the baselines.\n- Authors use the state-of-the-art causal recommendation method DLCE.", "weaknesses": "- The choice of the KL-divergence-based regularization is not super clear. The GNN paper citation (11) does not use beta distribution as a regularizer, but rather as a weight in the GNN aggregation. Also since $Q$ is the empirical distribution of the estimated propensity scores, how is the regularizer used in the training (could be explained via the gradient equation)?\n- It is not clear how the ground-truth propensities in the ML dataset are used. In the appendix authors very briefly touch upon that (lines 58, 59), but it's not clear how that relates to the ground-truth propensities. MovieLens has explicit ratings, which users self-select, and the bias in the original dataset reflects the user's self-selection bias, not the recommendation bias. Saito et al [1] transform the ratings into binary feedback to model the bias used in the current paper.\n-  The main metrics used for the evaluation (CP, CDCG) are not explained in the main section, but rather in the appendix. Since they are not very well-known metrics, it would be better if the authors include them in the main experimental section. \n\nReference:\n- [1] Saito, Yuta, et al. \"Unbiased recommender learning from missing-not-at-random implicit feedback.\" Proceedings of the 13th International Conference on Web Search and Data Mining. 2020. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging the strengths of our paper. We will give answers to each question asked by this reviewer below.\n\n**Q1:** What is the effect of the regularization term in propensity training? An ablation experiment could help, additionally, an experiment with varying $\\mu$ values could also give some insight into this.\n\n**A1:** The intuition of regularization term is to control the distribution of learned propensity to prevent them from clustering near extreme values like 0 and 1. In other words, to **avoid the violation of positivity assumption,** as pointed out by Reviewer 5c9m. Although the citation (cited as [11] in the main paper) does not directly use Beta distribution as a regularizer, it still uses it to control the distribution of weights. As that paper finds Beta distribution is good at modelling popularity characteristics, we chose it to model the propensity distribution which is correlated to popularity in our model. \n\nBesides, we have conducted experiments to analyze $\\mu$ for the regularization term in **Appendix D.1**, due to page limitation. We found that when $\\mu =0$ (no regularization at all), the causal performance is significantly compromised. And the performance reaches the peak when $\\mu$ is in the range of [0.2, 0.8], showing the advantage of regularizing the propensity with beta distribution.\n\n**Q2:** If authors could elaborate more on the ML dataset setup? \n\n**A2:** Sure, let us explain it with more details. The authors of [a] also described how they pre-processed ML dataset.\n\nGiven an user-item pair, first, they predicted the rating $\\hat{R}$ and probabilities of observing the rating $\\hat{O}$ using matrix factorization methods.\n\nThen, they estimate the interaction probability with recommendation ($\\mu^{\\mathrm{T}}$) and without recommendation ($\\mu^{\\mathrm{C}}$) using\n\n$\\mu^{\\mathrm{T}}=\\sigma\\left(\\hat{R}-\\epsilon\\right), \\quad \\mu^{\\mathrm{C}}=\\hat{O},$\n\n where $\\sigma$ is sigmoid function and $\\epsilon$ is hyper-parameter set to 5. \n\n Next, the propensity for this user-item pair is estimated by \n\n$P=\\min \\left(1, a\\left(1 / \\operatorname{rank}\\right)^b\\right),$\n \nwhere $\\operatorname{rank}$ is the rank of items for the user according to $\\mu^{\\mathrm{T}}+\\mu^{\\mathrm{C}}$. $a$ and $b$ are hyper-parameters which are set to 100 and 1 separately.\n\n The potential outcome with and without recommendations and exposure status are sampled from their corresponding probabilities. \n\n$Y^{\\mathrm{T}} \\sim \\operatorname{Bernoulli}\\left(\\mu^{\\mathrm{T}}\\right), \\quad Y^{\\mathrm{C}} \\sim \\operatorname{Bernoulli}\\left(\\mu^{\\mathrm{C}}\\right), \\quad Z \\sim \\operatorname{Bernoulli}\\left(P\\right) .$\n\n Finally, the causal effect and interaction are obtained as \n\n$\\tau=Y^{\\mathrm{T}}-Y^{\\mathrm{C}}, \\quad Y=Z Y^{\\mathrm{T}}+\\left(1-Z\\right) Y^{\\mathrm{C}}.$\n\nNote that in our experiment we use the obtained interaction $Y$ as ground-truth for training PropCare and causal effect $\\tau$, propensity $P$, exposure status $Z$ as ground-truth for evaluation.\n\n**reference**\n\n[a] Masahiro Sato, et al. \"Causality-aware neighborhood methods for recommender systems.\" In Advances in Information Retrieval: 43rd European Conference on IR Research, pages 603–618. Springer, 2021."}, {"review_id": "xyLVl2GauN", "reviewer": "Reviewer_xH3J", "summary": "This paper proposes a framework for causality-based recommendation system. Different from traditional correlation-based recsys (e.g. collaborative filtering), causality-based recsys makes recommendations based on the causal \"uplift\". While there are several causal recsys models in the literature, they rely on exposure data and/or propensity scores to be given. In real-world scenarios, exposure data are often unavailable, difficult to obtain or noisy. Hence, in this paper, the authors proposed a propensity estimation framework, in the absence of exposure data which is a more practical setup, to ultimately allow causality-based recommendation. Experimental analysis are conducted with both quantitative results and a case study.", "questions": "See Weaknesses.", "limitations": "NA", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The problem studied is of significant research and practical value. In particular, the setup without assuming any exposure data is practical and can be easily deployed on most recsys platforms. Overall, the motivation and challenges are well articulated and convincing.\n\nS2. The key assumption in 4.2 is well argued and presented. The empirical validation of the assumption convincingly support the assumption.\n\nS3. Experiments are comprehensive with evaluations on causal performance as well as the quality of estimated propensity and exposure. Detailed analysis of results reveal deeper insights, such as the factors influencing casual recommendation. Moreover, the case study is interesting and provide intuitive evidence to the benefit of a causal recsys.\n\nS4. Overall the paper is well executed with a well motivated and effective solution.", "weaknesses": "W1.  Below the theoretical property in 4.5, the authors mentioned that the proposition guides certain design choices, such as regularization of the global distribution in Eq (7). While this is a theoretical implication of the proposition, is there any empirical evidence? The current ablation study does  not seem to test the usefulness of the KL regularizer.\n\nW2. Data set differences： it is not clear to me the difference between DH_orig and DH_personalized. It was stated DH_personalized has a simulated \"personalization factor\". What is this factor? how does it work exactly?\n\nMinor comments:\nline 233: \"should be avoid\" -> avoided\nTable 1: the last 4 column names, are not clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for acknowledging the contribution of our work and paper presentation. we will give answers to each question asked by this reviewer below.\n\n**Q1:** Below the theoretical property in 4.5, the authors mentioned that the proposition guides certain design choices, such as regularization of the global distribution in Eq (7). While this is a theoretical implication of the proposition, is there any empirical evidence? The current ablation study does not seem to test the usefulness of the KL regularizer.\n\n**A1:** Sure, we have already provided additional ablation study and parameter test for KL regularizer in **Appendix D.1**, due to page limitation. We found that when $\\mu =0$ (no regularization at all), the causal performance is significantly compromised. And the performance reaches the peak when $\\mu$ is in the range of [0.2, 0.8], showing the advantage of regularizing the propensity with beta distribution.\n\n**Q2:** Data set differences: it is not clear to me the difference between DH\\_orig and DH\\_personalized. It was stated DH\\_personalized has a simulated \"personalization factor\". What is this factor? how does it work exactly?\n\n**A2:** The difference in DH\\_original and DH\\_personalized is how they simulate the propensity. For DH\\_original, the propensity of an item is basically computed as the ratio between how many weeks the item is exposed to the user and how many weeks the user visits the retailer. For DH\\_personalized, the items are first ranked by the computed user's interaction probabilities, then the propensity is estimated by \n\n$P_{u,i}=\\min \\left(1, a\\left(1 / \\operatorname{rank}\\right)^b\\right),$\n\nwhere $\\operatorname{rank}$ is the item's rank for that user $u$, $a=100$, $b=1$ are hyper-parameters. In the original paper, we refer ``personalization factor'' to the term $a(1 / \\operatorname{rank})^b$. More detailed simulation steps can be found in [a].\n\n**Minor comments:** line 233: \"should be avoid\" - avoided Table 1: the last 4 column names, are not clearly explained.\n\n**Response to minor comments:** Thanks for pointing out these. We will revise the typo and add clear explanations for the last 4 columns of Table 1 in the latest manuscript.\n\n**reference**\n\n[a] Masahiro Sato, et al. 2020. ”Unbiased Learning for the Causal Effect of Recommendation”. RecSys. 2020."}, {"review_id": "uTVZbXDyoH", "reviewer": "Reviewer_Pdiw", "summary": "This paper proposes a propensity estimation model for causality-based recommendation without accessing the ground-truth propensity score or exposure data. Prior knowledge about item popularity is utilized to estimate the propensity score. A theoretical analysis is provided to understand the proposed model.", "questions": "1. The authors only experiment with one backbone model, namely DLCE. I suggest the authors to compare with more state-of-the-art causal recommendation approaches.\n2. What are the main contributions of the proposed model? What are the differences between the proposed pairwise relationship and existing works such as [1] and [2].\n3. I suggest the authors to conduct experiments on large-scale datasets.\n4. Please provide more explanations on the case study.", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper investigates an interesting problem in causality-based recommendation, propensity score estimation, and proposes a model to get rid of the requirements for ground-truth exposure data.\n- A theoretical analysis is provided to understand the critical factors of the proposed model.\n- Ablation study is conducted.", "weaknesses": "- I have serious concerns about the evaluation framework in this paper. All the recommendation experiments are based on one single DLCE model, and the adopted baselines for comparison are rather weak. I suggest the authors to include more advanced baselines, and more importantly, to compare with a wider range of backbone models. Just to list a few widely acknowledged causal recommendation approaches [1-3] which all do not require exposure data.\n- The technical contributions of the paper is limited. The proposed pairwise relationship between item popularity and propensity score is similar to the design in [1,2], which also leverage popularity to serve as a *soft* proxy for propensity score.\n- The adopted three datasets are very small, considering modern recommendation platforms.\n- In Table 3, the proposed PropCare model does not achieve the best performance with respect to Tau and F1 score, outperformed by both POP and CJBPR.\n- The case study is hard to follow and not convincing to me.  \n\n[1] Bonner, Stephen, and Flavian Vasile. \"Causal embeddings for recommendation.\" Proceedings of the 12th ACM conference on recommender systems. 2018.\n[2] Zheng, Yu, et al. \"Disentangling user interest and conformity for recommendation with causal embedding.\" Proceedings of the Web Conference 2021. 2021.\n[3] Zhang, Yang, et al. \"Causal intervention for leveraging popularity bias in recommendation.\" Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments from this reviewer. we will give answers to each question asked by this reviewer below.\n\n**Q1:** Suggestion to compare with SOTA causal recommendation approaches. \n\n**A1:** Thanks for the suggestion. However, this review contains **factual errors** which we'd like to highlight to help reviewers better understand the goal of our paper and experiment settings.\n\nFirst, we clarify that our paper is **NOT** to propose a new causal recommendation approach. Instead, the proposed PropCare is a **propensity estimation** method for downstream causal recommendation, aiming to *bridge the gap in existing causality-based recommendation systems, where the propensity score and/or exposure data are often unavailable but required for model training or inference.*\n\nSecond, we clarify that the term \"causality-based recommendation systems\" used in our paper refers to the models that recommend items with a higher causal effect (line 28-32). However, the last two papers suggested by the reviewer [b,c] are neither for propensity estimation nor for causality-based recommendation. In particular, [b] aims to tackle conformity bias by decomposing the observed ratings into factors of interest and conformity for click estimation. [c] aims to utilize desired popularity bias by adjusting them in inference for click estimation too. Only [a] is a causal recommendation model, which jointly learn two models on data with and without recommendations. Hence,  exposure data are still needed for training. The reviewer said it does not require exposure data, which is wrong. **We further report the causal recommendation results of PropCare, using CausE [a] as an alternative backbone in Table 1 of the global rebuttal PDF.** It shows a similar pattern as using DLCE backbone in the main paper, i.e., our PropCare consistently outperforms other baselines even with different causality-based models as the backbone.\n\nThird, as a popular causality-based recommendation system, DLCE [d] (RecSys'20) is still considered a SOTA causal recommendation backbone, which is also agreed by Reviewer Vbya.\n\n**Q2:**  Main contributions and comparisons with [a] and [b].\n\n**A2:** For the first contribution, please refer to the first point in our A1 to this reviewer. Our second main contribution is to *incorporate prior knowledge for robust propensity estimation* by proposing assumption 1 to model the pairwise relationship on popularity and propensity. This contribution is acknowledged by Reviewer xH3J. Moreover, the novelty of our work is also acknowledged by Reviewer 5c9m.\n\nAs we have explained in the second point in A1, paper [b] is totally unrelated to our work; paper [a] is a typical causality-based recommendation system which requires exposure status to be observed for training. Our work is proposed for the gap in models like [a] where such data is usually unavailable.\n\n**Q3:** Suggestion for large-scale datasets.\n\n**A3:** Thanks for the suggestion, but other large datasets are not available for evaluation. To evaluate our model we must have ground-truth propensity and causal-effect, which are usually unavailable. Currently, only DH\\_original and DH\\_personalized [d] and ML [e] provide such ground-truth, which are widely used in evaluation of causality-based recommenders [d,e,f]. Nevertheless, we still conduct efficiency experiments on larger datasets, see A3 to Reviewer KaLM.\n\n**Q4:** Explanations on the case study.\n\n**A4:** The case study aims to show how the propensity scores from different approaches affect the causal recommendations. In Table 4, we investigate the top-5 recommended items with each baseline and draw the following conclusions: (1) With ground-truth propensity score and exposure, most recommended items have a positive causal effect ($\\tau=1$, denoted by \\ddag). (2) Comparing the lists between CJBPR and PropCare, they both hit two purchased items ($y=1$, denoted with bolded text), but CJBPR hits only one item with positive causal effect and PropCare hits two. It means although they have equal performance in conventional recommendation, PropCare outperforms CJBPR in causal recommendation. (3) Though item popularity was used to estimate propensity in some previous works [g,h,i], they are not suitable in causal recommendation. \n\n**Weakness 3:** About Tau and F1 score in Table 3.\n\n**Response**: In Table 3, we use multiple metrics to measure the accuracy of propensity estimation from different aspects. Tau only measures similarity between two rankings by counting the number of concordant and discordant pairs, which is not enough to judge whether it is a good estimation or not. Hence, we also use KLD and F1. Our PropCare performs best in KLD and F1 in most cases. Even in ML, the F1 is only 0.03 lower than the best. Moreover, using propensity and exposure inferenced by PropCare, we can achieve the best on downstream causal recommendation, which shows the advantage of PropCare over other baselines.\n\n**References**\n\n[a] Bonner, Stephen et al. \"Causal embeddings for recommendation.\" RecSys. 2018. \n\n[b] Zheng, Yu, et al. \"Disentangling user interest and conformity for recommendation with causal embedding.\" WWW. 2021. \n\n[c] Zhang, Yang, et al. \"Causal intervention for leveraging popularity bias in recommendation.\" SIGIR 2021.\n\n[d] Masahiro Sato, et al. 2020. \"Unbiased Learning for the Causal Effect of Recommendation\". RecSys. 2020.\n\n[e] Masahiro Sato, et al. \"Causality-aware neighborhood methods for recommender systems\". ECIR. 2021.\n\n[f] Teng Xiao and Suhang Wang.\"Towards Unbiased and Robust Causal Ranking for Recommender Systems\". WSDM. 2022.\n\n[g] Yuta Saito. \"Unbiased Pairwise Learning from Implicit Feedback\". NeurIPS 2019 Workshop on Causal Machine Learning\n\n[h] Yuta Saito, et al. \"Unbiased Recommender Learning from Missing-Not-AtRandom Implicit Feedback:. WSDM. 2020.\n\n[i] Longqi Yang, et al. \"Unbiased offline recommender evaluation for missing-not-at-random implicit feedback\". Recsys. 2018."}, {"review_id": "6RLnDjhTrx", "reviewer": "Reviewer_5c9m", "summary": "This paper presents a review of a novel method proposed for estimating causal effects in situations where no observation of the treatment variable is available. The authors introduce an innovative approach that utilizes interactions to approximate missing exposure or propensity data. The method relies on several key assumptions, including the significance of \"popularity\" as a strong confounder of both treatment and outcome variables, as well as the proposition that the outcome variable is a product of the unobserved propensity and relevance (which is a function of historical popularity). Additionally, the paper assumes a monotonic relationship between popularity and exposure (Assumption 1) and utilizes a parametric assumption based on the Beta distribution.\n\nThe paper does a great job reviewing existing literature. And the simulation section provides a thorough comparison with baselines and oracles that have access to the treatment variable. The ablation study of its own method is also very comprehensive which provided intuition on the most important factors of the proposed model.", "questions": "Question 1: PropCare is a solid method when all the assumptions are met: The method assumes `popularity` as a strong confounder of treatment and of Y. \nI wonder how general the assumptions are when the treatment changes or when applied to other datasets when the treatment does not have a single strong-enough confounder. \n\nQuestion 2: about line 291 (simulation section): Can authors share the distribution of the fitted propensity scores using all the compared methods? Wonder if POP's bad performance is due to a practical violation of the positivity assumption, plus not mitigating it through bounding the fitted value between e.g. [0.02, 0.98]. Since the PropCare method handles the positivity assumption violation in its regularization term, it should be a fair comparison to perform comparable mitigation for the other baselines.\n\n", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed method is very novel by leveraging strong intuitions in the domain to find strong confounders that can reliably impute the exposure variable through a learnable objective. The paper presents a fresh new perspective into the causal inference application to Recommendation systems when privacy limits the access to treatment assignments in production applications.\n", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging our novelty and literature review. we will give answers to each question asked by this reviewer below.\n\n**Q1:** PropCare is a solid method when all the assumptions are met: The method assumes popularity as a strong confounder of treatment and of Y. I wonder how general the assumptions are when the treatment changes or when applied to other datasets when the treatment does not have a single strong-enough confounder.\n\n**A1:** Assuming the casual relationship between popularity and treatment is quite general in many previous works. For example, [a,b] explicitly state the confounder role of popularity in the causal graph, [c,d] also empirically investigate how popularity affects the item exposure or predicted ratings. Hence, we believe the assumptions are reasonably general in most of the situations. \n\n**Q2:** about line 291 (simulation section): Can authors share the distribution of the fitted propensity scores using all the compared methods? Wonder if POP's bad performance is due to a practical violation of the positivity assumption, plus not mitigating it through bounding the fitted value between e.g. [0.02, 0.98]. Since the PropCare method handles the positivity assumption violation in its regularization term, it should be a fair comparison to perform comparable mitigation for the other baselines.\n\n**A2:** We show the distribution of fitted propensity scores by baselines for DH\\_original dataset in **Figure 1 of global rebuttal PDF**. From the results it can be seen that the long tailed distribution of POP where most of the values are close to 0 might be a reason for its bad performance, as the violation of positivity assumption may lead to large variance in the outputs of causal recommendation [e,f]. However, to avoid that, our backbone model DLCE [e] has already deployed a **capping threshold** $\\chi$ to reduce variance of the prediction. Besides, in data generation steps of [e], the simulated propensity are **clipped** in the range of [$10^{-6}$, $1-10^{-6}$] as a mitigation. In our experiment, we follow the same clipping step after we obtained the estimated propensity of each baseline by default. Hence, *give these mitigation steps, it is a fair comparison*. \n\nAdditionally, please note that we have also done an ablation study and parameter analysis of the regularization term in **Appendix D.1**, and we found that even without the regularizer (i.e., $\\mu$=0), PropCare still outperformed POP, e.g., 0.87 vs. 0.79 in CDCG on DH_original. This implies that the better performance of PropCare is not just due to the regularizer. \n\n**References**\n\n[a] Zhang, Yang, et al. \"Causal intervention for leveraging popularity bias in recommendation.\" SIGIR 2021.\n\n[b] Tianxin Wei, et al. \"Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System.\" KDD. 2021\n\n[c] Zhongzhou Liu, et al. \"Mitigating Popularity Bias for Users and Items with Fairness-centric Adaptive Recommendation.\" ACM Trans. Inf. Syst. 41, 3, Article 55. 2023\n\n[d] Ziwei Zhu, et al. \"Measuring and Mitigating Item Under-Recommendation Bias in Personalized Ranking Systems.\" SIGIR. 2020.\n\n[e] Masahiro Sato, et al. 2020. ”Unbiased Learning for the Causal Effect of Recommendation”. RecSys. 2020.\n\n[f] Teng Xiao and Suhang Wang.\"Towards Unbiased and Robust Causal Ranking for Recommender Systems\". WSDM. 2022."}]

Author rebuttal:
We express our sincere gratitude to all the reviewers for their valuable feedback and insightful comments on our paper. We are humbled by the positive reception and are encouraged by the recognition of the efforts we put into this research. We acknowledge the time and expertise each reviewer has invested in evaluating our work.

Especially, we want to thank Reviewer KaLM for acknowledging our **presentation, soundness and experiments.** we thank Reviewer Vbya for  also acknowledge our **presentation, experiments and intuition.** We thank Reviewer xH3J for praising the **practical value, presentation and experiments.** We thank Reviewer Pdiw for agreeing the **proposed question, theoretical analysis and ablation study.** Finally, we thank Reviewer 5c9m for strongly appreciating the **novelty and literature review** of our work.

Please kindly find our responses to individual reviewers in the corresponding rebuttals, and some of the figures in the rebuttals can be found in the attached PDF below.

Sincerely,

The authors of "Estimating Propensity for Causality-based Recommendation without Exposure Data"


Meta-review: This paper tackles the problem of propensity estimation when exposure data is not available. The main idea is by making the observation that propensity is often correlated with popularity. The estimated propensity can then be used in a downstream task, e.g., many causal-based recommender systems where propensity is required. The scores are very divided ranging from 3 to 8. Most of the reviewers agree that the problem this paper considers is very important and under-explored. The paper is clearly written and easy to understand. The reviewers also praised the paper for the theoretical analysis that is grounded with demonstratable assumptions. On the negative side, the empirical studies have to partially rely on simulated data, which hinders the credibility. I also read the paper myself and I think the reviewers are spot on with both the positive and negative sides of the paper. Overall, I still think the pros over-weigh the cons -- especially for considering a relatively underexplored (yet quite important) problem. Therefore I am voting for accept. 

(For more context: the reviewer who gave the lowest score seems to misunderstand the main contribution of the paper and didn't respond to authors' rebuttal as well as the discussions among reviewers. Therefore, I didn't weigh this review as much as the others.)
Decision: Accept (poster)