Counterfactual Implicit Feedback Modeling
Authors: Chuan Zhou
Lina Yao
Haoxuan Li
Mingming Gong
Venue: NeurIPS 2025 poster ()
Keywords: Implicit  Feedback
Recommendation System
Counterfactual Estimation
Missing Treatment
Primary area: probabilistic_methods

Abstract: In recommendation systems, implicit feedback data can be automatically recorded and is more common than explicit feedback data. However, implicit feedback poses two challenges for relevance prediction, namely (a) positive-unlabeled (PU): negative feedback does not necessarily imply low relevance and (b) missing not at random (MNAR): items that are popular or frequently recommended tend to receive more clicks than other items, even if the user does not have a significant interest in them. Existing methods either overlook the MNAR issue or fail to account for the inherent mechanism of the PU issue. As a result, they may lead to inaccurate relevance predictions or inflated biases and variances. In this paper, we formulate the implicit feedback problem as a counterfactual estimation problem with missing treatment variables. Prediction of the relevance in implicit feedback is equivalent to answering the counterfactual question that ``whether a user would click a specific item if exposed to it?". To solve the counterfactual question, we propose the Counterfactual Implicit Feedback (Counter-IF) prediction approach that divides the user-item pairs into four disjoint groups, namely definitely positive (DP), highly exposed (HE), highly unexposed (HU), and unknown (UN) groups. Specifically, Counter-IF first performs missing treatment imputation with different confidence levels from raw implicit feedback, then estimates the counterfactual outcomes via causal representation learning that combines pointwise loss and pairwise loss based on the user-item pairs stratification. Theoretically the generalization bound of the learned model is derived. Extensive experiments are conducted on publicly available datasets to demonstrate the effectiveness of our approach. The code is available at https://github.com/zhouchuanCN/NeurIPS25-Counter-IF.

Counterfactual Implicit Feedback Modeling
Chuan Zhou 1.2 Lina Y Haoxuan Li
Mingming Gongl:2.
'The University Melbourne MMohamed bin Zayed University Artificial Intelligence "The University New South Wales *CSIRO's Data61 "Peking University chuan. zhou@student unimelb. edu. au, lina. yaodaaao@ csiro.au au, hxli@stu. pku, edu. en, mingming gongguniignggl edu. au
Abstract
In recommendation systems, implicit feedback data can be automatically recorded and is more common than explicit feedback data. However, implicit feedback poses two challenges for relevance prediction, namely (a) positive-unlabeled (PU): negative feedback does not necessarily imply low relevance and (b) missing at random (MNAR): items that are popular or frequently recommended tend to receive more clicks than other items, even if the user does not have significant interest in them. Existing methods either overlook the MNAR issue or fail to account for the inherent mechanism of the PU issue. As a result, they may lead to inaccurate relevance predictions inflated biases and variances. In this paper, we formulate the implicit feedback problem as a counterfactual estimation problem with missing treatment variables. Prediction of the relevance in implicit feedback is equivalent to answering the counterfactual question that "whether a user would click specific item if exposed to it?". To solve the counterfactual question, we propose the Counterfactual Implicit Feedback (Counter-IF) prediction approach that divides the user-item pairs into four disjoint groups, namely definitely positive (DP), highly exposed (HE), highly unexposed (HU), and unknown (UN) groups. Specifically, Counter-IF IF first performs missing treatment imputation with different confidence levels from raw implicit feedback, then estimates the counterfactual outcomes via causal representation learning that combines pointwise loss and pairwise loss based on the user-item pairs stratification. Theoretically the generalization bound of the learned model derived. Extensive experiments conducted on publicly available datasets to demonstrate the effectiveness of our approach. The code is available fattps://gtt giggh...b.... com http://chccuaa///uuulllllaaa INe.nn/ Counter IF
Introduction
Recommender systems are technical tools that analyze historical user behavioral data to predict their preferences and actively provide personalized recommendations [I 2 B which have been widely applied to various fields, such as e-commerce, streaming video, and social media 4 6 7 There are two types of feedback mechanisms user behavioral data, including explicit feedback g 10] which refers to the explicit signals that users directly express their preferences, such as product ratings. Implicit feedback [1], on the other hand, comes from the indirect interactions between users and the system, including unstructured behavioral trajectories such the length of time spent on the page, clicks, and so on. Implicit feedback has great advantage over explicit feedback terms of data acquisition [12 5). most user behaviors are implicit [13), which can be collected continuously on a large scale and can reflect the potential user demand more promptly.
In implicit feedback recommendation, the goal is to infer the user's true relevance or preference for an item from their behavior data, i.e., determine whether the user is likely to be interested an item.
Haoxuan Li and Mingming Gong are the corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, this process faces two key challenges. The first one is the positive-unlabeld (PU) learning problem [14 The system only observes a binary signal of whether the user clicks the item, but the clicking behavior does not fully reflect user preferences. Specifically, the items not clicked on may be attributed to user disinterest, or they may not even be exposed the user. The second challenge is that the feedback data is missing not at random (MNAR) [16] [7]. For example, items frequently recommended tend to attract more clicks, even if the user does not have a substantial interest in them. In recent years, studies have begun to notice that MNAR in implicit feedback data can introduce bias into predictions, thus damaging the performance of the recommendation system [88 19 20).
In the development of implicit feedback recommendation, early approaches such as weighted matrix factorization (WMF) [21], exposure perception matrix factorization (ExpoMF) [8] and Bayesian personalized ranking (BPR) [13] are based on the key assumption that the observed feedbacks directly reflect the true preferences, whereas unobserved feedbacks are uniformly treated as negative samples or given low weights. However, early methods assume that the missing mechanism is missing completely at random (MCAR), ignoring the MNAR nature of implicit data, leading to biased relevance estimation [16]. Recent studies attempt to introduce causal techniques into implicit feedback modeling, e.g., through counterfactual bias correction using propensity score 122 23 However, these causal learning methods still have some drawbacks. The first one is that propensity models are prone to be overly confident, generating extremely inaccurate propensity score estimation [24 25 26]. And like any propensity- brassd method, the bias and variance of the estimator can be extremely large with small propensity [27 28), thus affecting the effectiveness of relevance prediction. The second one is that existing causal recommendation methods for implicit feedback data usually use EM algorithms to estimate the exposure propensity, but the intrinsic positive-unlaheled mechanism of implicit feedback is neglected, which may hinder the model from accurately estimating the propensity.
In this paper we propose that implicit feedback recommendation can be addressed by answering the counterfactual question: whether user would click a specific item if exposed to it? To tackle this problem, we formulate it as a counterfactual outcome prediction problem with missing treatment. Let the feedback label be the outcome, and the exposure be the treatment. To answer the counterfactual question, we actually need to estimate the potential outcome corresponding to the exposed treatment group, with treatment of only the exposed group observed. Although there has been work dealing with the MNAR problem with explicit feedback data [27 22 30, 31]. and remarkably much more work in the area of statistics estimating causal effects 32
33 none of them could be applied the counterfactual problem formulated in this paper. That is because all previous causal methods require the treatment to be observed for every sample in the dataset. However, this counterfactual problem brings unique challenges with missing treatment.
To overcome these challenges, we propose the Counterfactual Implicit Feedback (Counter-IF) method for estimating counterfactual outcomes in the presence of missing treatment variables in implicit feedback scenarios. To uncover more information from negative samples based on the dusitive-unlabele nature, the Counter-IF consists of stratifying user-item pairs in implicit feedback. Specifically, using the estimated confidence, we impute the exposure only for samples with high confidence. Apart from the definitely positive (DP) samples group, we classify the negative samples into three groups based on the different reasons leading unclick: those with high probability of being exposed (HE) tend to have low relevance and those with high probability of being unexposed (HU) tend to have higher relevance than other unknown (UN) negative samples. Counter-II also includes a causal representation learning framework that combines pointwise and pairwise losses based on the imputed treatments, leading to accurate counterfactual outcomes estimation.
The main contributions can be summarized as follows:
We are the first to formalize the relevance prediction problem under implicit feedback scenarios as a counterfactual outcome estimation problem with missing treatments.
We propose a sample stratification algorithm in Counter-IF for implicit feedback using treatment variable imputation method with confidence, reflecting different mechanisms of negative sample generation.
We propose a causal representation learning framework in Counter-IF to answer the for- malized counterfactual questions. We theoretically derive the generalization bound of our causal learning model.
We conduct extensive experiments on publicly available real-world datasets, demonstrating the proposed Counter-IF significatlyy outperforms thth--ffthhe-rt methods.
2

---

2 Problem Setup of Implicit Feedback
Let u U denote user, i I be an item and D U x I be the set of all user-item pairs. The complete set consists of [2]] x [I] user-item pairs. The recommender system observes only the implicit feedback Y,,; E {0,1}. which passively captures whether the user u clicks the item i. The feature of (u, 8) is denoted as Xu.i- We divide all user-item pairs according to Yu,i, i.e., Di ((, (u, E D, Yu, 1} and Do ((u,i) i) (u, i) E D, Yu, 0). Due the sparsity of positive samples in implicit feedback data, we have [Di < [Pol_ A positive feedback Yu,i = directly implies that u and i are relevant. However, we are not certain whether negative feedback Yu, = O indicates the item is irrelevant to the user since it depends on whether u is exposed to i.
We consider the exposure Ou, the item and vice versa. Note that we cannot observe O.,, for all the user-item pairs in D. Instead, we can merely infer that for those (u. E D1. the user must be exposed to item i, thus Ou,i 1. However, for those (u, i) E Do. Ou, is unknown. To more clearly formulate the implicit feedback problem, we introduce the true relevance score Su,i (0, 1}, with S.,; = 1 indicating that u and i are relevant and vice versa. Following previous work 16 we assume in this paper that:
treatment, with Ou,i meaning that the user is exposed to
Yu,= = Ou,i Su,s.
(1)
Given Yu,i Ou,; for (u, f) E DD and Yu,i 0 for (u, i) E Doo we would like to predict Sa, for all (u, E 'D. Note that Equation (1) implies Yu,; = =1 - Ou,i = 1 and S,,i = 1.
3 Methodology
3.1 Causal Formulation Implicit Feedback
The question we aim to answer in implicit feed- back recommendation is "whether user would click a specific item exposed to it", a typical counterfactual question. Therefore, it is logi- cal to express it in causal terminology. In this paper, we employ the potential outcome frame- work. In our causal formulation, we let Ou.; be the treatment, and (1) be the potential outcome of feedback if forcing Ou,i = 1, and (0) be the potential outcome of feedback if we force O,., = O. We require the consis- tency assumption, which means that if u ex- posed to i, the observed outcome of feedback is the potential outcome we aim to estimate, i.c.. Yu,: Ou.: (L) + (1 - Oa.;) Y... We
Table 1: The user-item pairs are divided into four subgroups from a principal stratification perspec- tive, named Definitely Positive group, Highly Ex- posed group, Highly Unexposed group, and Un- known group. The red font indicates that the value imputed.
Group
0
R
Y
Y(1)
Definitely Positive (DP)
1
1
1
1
Highly Exposed (HE)
1
1
0
0
Highly Unexposed (HU)
0
1
0
Unknown (UN)
?
0
0
?
also assume that the stable unit treatment value assumption (SUTVA) holds, i.e., there should be only one form of exposure between u and i, and there is no interference between user-item pairs. In addition, we assume that the unconfoundedness assumption holds, i.e., (1), 1L /0.,/Xa... In the recommendation scenario, this assumption implies that the intrinsic relevance between and i is independent of whether to expose u to i. Then on the one hand, by the consistency assumption we have O..; = 0= Yu,: Yo.:()). Meanwhile, by Equation ( we have Ou,; = 0 Y,,, = 0. As consequence. Yu, (0) = for (u, i) E D. On the other hand, we have Ou,i = 1 + Y,,, = Ya,: and Ou,; 1 = Yu,d Su, Combining the two equations derived under the condition Ou, 1. we have (1) = Sa,; for (u, E D.
Therefore, we transform the estimation of Sy.; into the estimation of (1). In other words, we formulate the implicit feedback problem as missing treatment counterfactual problem, where the treatment assignments to the samples with negative feedback are unknown, and we aim to estimate the potential outcomes (1). With the consistency assumption, we can infer that the potential outcome Yu,; = 1 for (u, i) E Di, since for these samples Yu, = 1, we can infer On,; 1, then i) E Do. we are not sure whether O.,i = or Ou,i = 0, so we cannot directly infer the value of Y (1) for this group of user-item pairs.
(1) = Yu.i 1. However, for (u,1)
3

---

A
A
Treatment
O
Imputation
4
O
A
A
A
Original Feature Space
Kernel Space
Positive Sample
Definitely Positive
A
Negative Sample
Highly
A
Unknown
Highly Unexposed
Exposed
Figure I: Missing treatment imputation with different confidence level from implicit feedback.
3.2
Stratification of User-Item Pairs
For the purpose of counterfactual estimation, it is almost impossible to solely rely on the outcome variables and small portion of the treatment variables. So we, therefore, wanted dig deeper into the intrinsic mechanisms of the implicit feedback data to get more information about exposure. Unlike previous methods that impute treatments using consistent scheme for all samples in D, based on the diversity of meanings of negative samples in implicit feedback, we consider imputation only for those items the system is highly confident recommend a user. Specifically, we define a binary indicator R E {0, 1}. which reflects the confidence of the recommender system in assigning the treatment. Ru,i = means that the system is with high confidence, and R,, means that the system is not confident about treatment assignment. We denote the estimated confidence as Ru. for treatment imputation model. For those with high confidence, impute the treatment as the predicted Ô Then based on (Ou,i, Ru, i), we divide all user-item pairs into four strata:
Definitely Positive (DP) Group Pp: This group consists of user-item pairs with (L)
1, the imputation method should be definitely confident about the corresponding treatment
Ou = 1, meaning u has definitely been exposed to i. We have DDP = Dj.
For user-item pairs in Do. we further classify them into three categories, according to the estimated confidence indicator Ru,; and imputed treatment Ón.;:
Highly Exposed (HE) Group: The imputation method has high confidence that user is exposed to the item (Ru,, = and Õu,, = 1), thus impute the treatment as meaning the user tend to be exposed the item (O,.i 1). However, the observed
Yu, = 0, which means that (1,((1) is likely to O.
Highly Unexposed (HU) Group: The imputation method has high confidence that the user is unexposed to the item (Ru,, and Ô.,; = 0), thus impute the treatment as O. (1) more likely be than other negative samples.
Unknown (UN) Group: The imputation method has low confidence to assign treatment (Ru,i = 0), so we do not impute for the sample (Ou,; =?), keeping its multiple interpretation nature.
As Table I shows, can infer the value of (1) for the DP and HE groups, but not for the HU and UN groups. For the DP group (1) = Y2,i = 1. For the HE group, given the imputation is accurate, we have (1) Y.,, O.
3.3 Treatment Imputation with Confidence
Intuitively, we expect the DP group, consisting user-item pairs known to exposed, and the HE group, which includes user-item pairs for whom the system has high confidence that they are exposed, to be close to each other in a specific feature space. the other hand, the HU group, composed of user-item pairs for whom the system is confident that are unexposed, should be far

---

away from the exposed groups (HE & DP). The remaining users, whose exposure status remains uncertain, form the Unknown group (UN), positioned between the positive and negative groups. To divide users into the four subgroups, we propose novel exposure estimation approach to impute the treatments for solving the counterfactual problem, as shown in Figure[T Inspired by support vector domain description [34,, we propose novel treatment imputation method that utilizes the distance requirement between the four groups to assign samples to groups. Specifically, with feature Fn.s E X as the input, we enclose proportion of samples of DP within hypersphere, where E (0, 1) is a hyperparameter. Then we assign the negative samples within the hypersphere into the HE group. Then the farthest 3 proportion of the negative samples is assigned to the HU group, where 3 E (0, another hyperparameter. The remaining negative samples go to the UN group.
Our treatment assignment method aims to encapsulate proportion of positive samples within hypersphere defined by its center a and radius The optimization problem can be formulated as:
min F(r, a) = = 7,a,e
Cu,i
(u,i)EDor
s.t. | all2 S r² + Ea, Eu, 2 0, ((u E DDP,
where is the radius of the hypersphere, is its center, and Cu,i are slack variables that allow certain user-item samples to lie outside the hypersphere.
To solve this optimization problem, we incorporate the Lagrange multipliers Au,i and Tu,i for each constraint:
L 3-r+
Σ
{Co.i
Au,i (1² Ca.i (o2.s 2ar + a²)) -uncau.)
(u.))EDoP
Then, the dual optimization problem can be represented as:
Σ
Σ
max (u.
s.t. 0S Au,i S.
To handle non-linearity in the data, we apply a kernel function K (Zu,is Fu(s)) which implicitly maps the input data into higher-dimensional feature space. A commonly used kernel is the Gaussian radial basis function (RBF) kernel, defined as:
K (3i,,i Fud F.) exp(-q]laa
where q controls the width of the kernel and thus the smoothness of the decision boundary. The squared distance of sample from the center of the hypersphere is computed as:
d (x) = <(x,))) 2
Σ
(Kw,ir x) +
Σ
da,i (KIn,s Dw)..
(u.i)EDDP
(w
And the squared radius of the hypersphere is d (xx)) where v, a support vector.
Then, we find the smallest a and the largest proportion of (d (u, E Do), and denote the threshold ri and r2 respectively to classify Do into the following groups:
(u, E DHE if d (Ius) ST1
(u, E DuN
if < (d(ss)) S sm,
(u, E DHU
if d r2.
3.4 Counterfactual Representations
After obtaining a stratification for the samples, we propose a causal representation learning method to predict the relevance between users and items across D. When Ou 1, indicating that the relevance scores for samples in HE and DP groups are observed, we can use pointwise loss to train the model based on the outcomes (1). This ensures that the model effectively fits the potential outcome. In contrast, when the samples are not assigned the treatment group (i.e., On O, corresponding to the UN and HU groups), treat these unobserved interactions counterfactual data and employ
5

---

DP
DP
poiut ( (x(x))))
HE
+
Caa (7) ( X (
A
UN
A HE
point (h X
HU
h
Cruis (h( (((()(((())))))
A UN
IPMa(Ps .P8-1)
( (oo)))((X))))))) A(b(X
HU
IPM 10-1 1-1--11
PP
Input
Network
Prediction
O
Definitely Positive
Highly Exposed
Unknown
Highly Unexposed
Figure 2: Representation learning framework through a counterfactual lens: exploring four strata of implicit feedback.
pairwise loss to model the relative rankings. This approach enhances the capability of our model to capture user-item relevance across all samples.
The user-item feature Zu,; is transformed into the representation ( The prediction model is then applied to the representation s to output the predicted interaction outcome h($(ru,)))
For DP and HE samples, we apply a pointwise cross-entropy loss. This ensures that the model accurately predicts the relevance based on the observed data. Specifically, for the DP samples, where Y(1) 1, we have:
p (aa@n((((((P))))
Σ
log h(D(zu,))
(DDll (u.i)EPpr
While for the HE samples, where Y(1) 0, the loss is defined as:
Spiin (((((())))))))))
Σ
log(1 - 11(1h(((r ((((Fu))))
|DHE
(M,))DHH
Therefore, the whole point loss is defined below:
L1 paii ( (x
For the unobserved interactions (i.c., O.,: O. corresponding to Xpr and X we use the pairwise loss to model the ranking of the interaction outcomes. Specifically, the following pairwise loss is applied to optimize the relative ranking between strata:
(M)((((()))) = |+1 D__
Σ
(u.i)ED -.A)DD
where D+ and D_ the sets of positive samples and negative samples. The pairwise ranking loss encourages the model to assign higher scores to positive samples over negative samples.
Specifically, observe that the interaction probability for the HE group is significantly greater than that for UN group, while the interaction probability for HU group is markedly lower than that for the UN group. We apply the interaction probabilities among various user-item groups establish positive and negative sample pairs and obtain the counterfactual loss:
L2 = Cppi (a.p))))))))(())) (((())))) +
h ((((Nn)) + (ppie ((0((NN)))
To mitigate distributional shifts between different strata of samples, introduce Integral Probability Metric (IPM) regularization. IPM ensures that the representations samples with confidence (R and samples that not sure (R 0) are aligned in the representation space. Furthermore, also

---

introduce regularization for R = 1,0 = O and R = 1,0 = samples to ensure consistency. The IPM regularization terms are defined as:
LIPM- (P
R-0 R-1 PP
IPMa R=1,0=0
:P
R=1.0=1).
By introducing IPM, we aim to ensure that the model learns representations that generalize well across different sample distributions. The IPM is defined as:
IPMc == sup
((s)(p()))))))))))))){{)))))(( ((((p((s))(())(())) da
gEG
where p and y two probability distributions, and G' is class of functions for which seek to optimize the difference in expectations.
The total loss for the model incorporates both pointwise and pairwise losses, addressing the observed and unobserved interactions across different strata. The final relevance prediction model is trained by minimizing the following loss:
Cioca Apoint L1 + Apait C2 + LIPM.
3.5 Theoretical Analysis
We theoretically derive the generalization bound under our framework and show that minimizing the proposed pointwise loss L1, pairwise loss L2, and IPM loss LIPM can effectively control the bound. First, we introduce the following assumption ensure the existence of the inverse representation:
Assumption 3.1 (Inverse Representation and Function Class [35]). The representation 4 X A a one-to-one function, with inverse V. Let be a family of functions g : - - y. Assume there exists constant B > 0, such that 1 4b v (a) Y()) e G. B.
Based Assumption 3.1 we then derive the following generalization bound: Theorem 3.2 (Generalization Bound). Under Assumption 3.7 the deviation between the estimated
relevance h( (x)) and expected relevance mi (x) = E. 11) X = x) averaging on all user-item pairs has the upper bound:
Eg [h(((())) ml(x))² Extr, (1(((())))))))) - Y(1))² - 12-1
factual of the and HE group
+ +P(0 =P R= Ba IPM R=1,0=0 R=1,0=1
Pb
P(R=0) 0) Ba IPMa
R=0 R=1)
distribation shift O given Rui
UN group
distribution shift on R
m
variance of potential outcome
In the above bound, the first term is the factual loss based on the true value of ( of the DP and HE groups, which can be controlled by minimizing the Lpcin and Lpiir The second and third terms are the IPM distance measuring the distribution shift on distribution shift on R = 1 and R = 0 weighted by the proportion of HU group given R = 1 and proportion UN group P R 0), respectively. These IPM distance terms can also be effectively controlled by minimizing the proposed LIPM- Intuitively, if (PRR 0) 0, there is no need to control the distribution shift on R. The last term measures the minimal variance of potential outcomes, which is independent of the model selection. See Appendix A for the detailed proof.
and O given
group and
4 Experiments
4.1 Experimental Setup
Datasets. evaluate the performance of unbiased recommendations, we utilize two real-world datasets: Coat and Yahoo! R3. Each dataset includes both biased training data and an unbiased test set. The Coat dataset contains 6,960 biased ratings and 4,640 unbiased ratings provided by 290 users for 300 items, where each user rates 16 randomly selected items. The Yahoo! R3 dataset includes

---

Table 2: Ranking performance Yahoo and Coat. We bold the best results and underline the best baseline. The results with * indicate < 0.05 using the pairwise t-test with the best competitor.
Methods
NDCGOK
Recallid
MAPO'
NDCG@K
Recall K
MAP
NDCG@K
Recalli
MAPΩK
ExpoMi
0.524
0.008
0,581
0.461
00008
0.583 0,007
0.736 0.010
0,509
0.007
0,652
00.005
0512
0.000
0.548 1: 0.006
WMP
0.538 0.005
0.596
0.005
0.470
00.05
0.600 0,000
0.755 1: 0.0011
0.529
0.004
0,665
00004
0.18 1 0.000
0.561
0.001
Rel MT
0.554 0.008
0,559
0.099
0,965
00006
0.593 0.007
0.369 1 1.00000
0.525 1: 0.077
0,655
00004
0.11
0.000
0.555
1 0.005
Rel-MFFdn
0,540 0.008
0.596
0008
0,478
0.009
0611 0.0007
0.756:: 1 0.0.00
0,530
0.009
0,668
0.007
0,915
0.000
0.557 1: 0.008
BPR
0.517 1: 0.001
0.574
0005
0.455
= 0.005
(138) 1 0:00
0.752--0.001 0.011
0.502
0.005
0.654
00004
0.905 0.000
0,542 + 0.004
URPR
0.592 0.005
0.502
0/005
0.470
- 0.005
0.506
+ 0.000
0.246
0.000
0.517 0.001
0.57
0.002
0,913 +0000
0.555 + 0.004
UIPR--cclip
0..56 + 0.008
0.577
0008
0.474
- 0.009
0.597
+ 0.006
0.010 0.746
0.522 + 0.009
0.59
0.007
0,914
+-0000
0.557 + 0.008
UPL
0.546
0.005
0,605
0011
0,483
0.00s
0.610
0,008
0.709
0.552
0.005
0,668
0.004
0.007
0.568 1 0.004
RecRee
0.545
0107
0,002
0006
0,684
0,006
0.607
0,004
0757 0.004
0,533
0.006
0,669
0.004
0.923
0.006
0,570
0.006
Ours
0,562
0.007
0.624
0.009
0.499
0.007
0.628'
0,005
0.776'
0.006
0,547
0.006
0.681
0.004
0,930
1 0.0.0
6,582
0.005
ExpoME
01:324 #: 0.0077
0,340
0,010
0,256
= 12007
0.372 =: 0.006
0.459 =: 0.012
0.287
0U807
I.425
0.005
0.601
2: 0.003
0.315 = D.0US
WMF
11:335 0.016
0.332
0.019
0.254
= 0.015
0.369
=: 0.016
0.412
=: 0.120
11.244
0U015
1.426
0.016
0.610
0.125
0.325
= 0.015
ReIMF
11.388 0.013
0,344
0,015
0.267
= 0,010
0,385 2: 0,005
0.462 +: 0,012
1.2064
0U006
0.435
0.008
0.614
0,022
0,338 2: 0.005
ReIMP du
0,240 2: 0.016
0,532
0,025
0,275
2 0.013
0,374
1: 0,010
0.420 2 0.014
0.207
0U009
0.455
0.015
0.640
1 0.021
0,355
1 0.012
BPK
0.324 1 0.011
0.325
0,018
0.266
1 0,010
0,370
1 0,010
0.433 1 0.017
0.290
0.008
0,445
1 0.309
0,60 1 0.017
0,335
1 0.007
UDPR
0.345 1: 0.012
0.342
0018
0.269
1 0,012
0.384 1 0.009
0:451 1 0.017
0.206
0.009
0.449
0.009
0,642 1 0.023
0,339
1 0.008
UBPPR-nclip
0.355 1: 0.007
0,345
0.013
0.261
: 0.006
0.368
3: 0.011
0.430 11 0.022
0.290
0.009
0.445
0.007
0.640 1 0.012
0,335
2 0.005
UPL PPL-BPR
113455
0.009
0.343
0.014
0.273
:: 0,009
0,377
1 0.009
0.427
0,025
1.2022
0U807
0
0.009
0.615
2 0.014
0,340
:: 0.005
RecRoe
0.360 2: 0.008
0.365
0001
0.264
: 0.005
0.392
4 0.009
0.452
0.019
0.314
0.006
0,454
0.007
0.629
0,018
0,356
: 0.001
Ours
0,368 1. 0.011
0,383
1 0.012
0,296
0,414
0,012
0,478'
1 0,014
0,332
1 0.00
0.473'
1 0.012
0
1 0.021
0,365
0.009
Table 3: Ablation study on the Yahoo and Coat datasets.
Methods
K=
K=
K=
NDCGO
R
MAPG3K
NDCGo
Recall
MAPSK
NDCG
RecalliKK
MAP
0.00%
0:009
0.009
1,0009
0.668
0.923
0.556
0.007
0.66
0.008
00,00
0.536
0.63
0,927
0.572
Pair
8.009
0:000
0.005
0.16
0.006
0.770
0.004
0.536
0.877
0.73
0.927
0.572
Point
0.005
0:007
0.006
0.19
0.007
0.774
0.004
0.540
0.887
0.75
0.927
0.575
wio
0.008
0:010
0.009
0.600
1 0.009
0,009
0.540
0.008
0.76
0.927
0.576
0.007
0.034
0.009
0,499
0.007
I1625
IUMI
0.776
0.808
0.547 : 0.006
0.681
0.004
0.930
0.004
0.582
0.010
0.376
0:010
0.250
0.099
0.298
0.011
0,012
0.314
0,038
0:450
0,618
0.347
Pount
0.009
10:010
0.009
0,010
0.477
0.013
0.319
0,009
0:450
0,609
0.352
Pair
0.011
0:012
0.008
0.412
0,010
0,015
1.0011
0.422
0.616
Point
0.009
0.363
0:010
0.00P
0,407
0.010
0.484
0.013
0.325
0.039
0.555
0.614
0.356
0.364
0.019
0:011
0.010
0.402
:: 0,012
0.478
0,014
0.317
0.010
0.46
0.666
0.020
0.357
0,011
0.382
0.012
0.296
0.009
0,414
0,813
0.478
0,014
0.332
0.009
0.433
0,660
0.821
0.369
311,704 biased ratings and 54,000 unbiased ratings from 15,400 users interacting with 1,000 items. The positive samples are sparse, which is consistent with the real-world situation. We employed the preprocessing steps following previous studies [23 36). which can be seen in the Appendix [B
Evaluation Metrics and Details. We use three common metrics to evaluate implicit recommendation systems: NDCG @k (Normalized Discounted Cumulative Gain), Recall @k, and MAP@K (Mean Average Precision). DCG@k evaluates the ranking quality by giving more weight to relevant items appearing earlier in the list. Recall@k k measures how many relevant items are retrieved within the top k recommendations. MAP@1 k calculates the mean precision across users, considering both relevance and order. Results are presented for k- = 3, k = 5, and k = 8.
Hyperparamter Tuning, For each dataset, the data was divided into training and test sets. A portion of 10% from the training set was randomly selected to serve as the validation set for hyperparameter tuning. Several key parameters were adjusted during this phase. The latent factor dimensions, representing user-item interactions, were explored within the range of 100 to 300, while the L2 regularization term was fine-tuned between [10-T, 10-3] for all models, and the Apoint as well as Apair are tuned in (0.01, 0.1, 1, 10, 100).
Baselines. To achieve a comprehensive comparison, we consider the following methods as baselines: WMF (1. ExpoMF [8. Rel-MF [16). Rel-MF-du [16], BPR 13) UBPR 23), UBPR-nclip 233
UPL 37,. and RecRec (36].
4.2 Performance Comparison
We evaluate the performance of our proposed method against several baseline approaches on multiple datasets, as shown in Table The results highlight several key findings. Traditional methods like BPR and WMF show moderate performance but struggle with the PU and MNAR challenges in implicit feedback. BPR misclassifies potential positives as negatives by assuming unobserved interactions are negative, while WMF treats unobserved interactions as low-weight positives, failing to fully address false negatives. Methods like ExpoMF and Rel-MF improve by modeling exposure and item popularity, reducing bias. ExpoMF incorporates exposure variables but remains limited by traditional matrix factorization, while Rel-MF leverages item popularity to estimate propensities, though it still
8

---

0.58
0.00
0.37
0.99
0.56
0.62
0.36
0.38
0.62
0.54
0.6
0.34
0.35
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.3
0.1
0.2
0.2
0.3
0.1
0.2
0.2
0.1
0.2
0.2
0.4
0.1
0.4
0.1
0.4
0.1
0.4
0.1
(a) 3 on Yahoo
(b) Recall@ Yahoo
(c) NDCG@3 on Coat
(d) Recall @3 on Coat
Figure 3: Sensitivity analysis on the proportion of the HE group a and the proportion of the HU
group 3 for samples in Do-
0.367
M
0.380
0.365
0.380
0.560
0,375
0.333
0.822
0.623
0.378
0.550
0.620
0.535
NDCG
0.620
0.360
0.357
NDCGB3
0.370
0.365
0.355
0.375
0.545
Recaligu
0.1
Recalig3
10.0
100.0
8 815
0.355
0.1
10.0
100.000.0
00360
0.350
0.02
Recaliga
10.0
100.0
(a) Apair on Yahoo
(b) Apoint on Yahoo
(c) Apair Coat
(d) Apoint on Coat
Figure 4: The sensitivity of balancing weight between pointwise and pairwise losses.
struggles with unobserved items. UBPR and UBPR-nclip extend BPR to address PU and MNAR problems, with UBPR-nelip further reducing bias through non-clipping estimator. UPL simplifies pairwise learning, performing well in high- high-variance scenarios. Finally, ReCRec shows strong results but computationally intensive. limiting scalability. Despite improvements over simpler models, these baselines still fail to fully address the challenges of implicit feedhack. Our method outperforms all baselines by explicitly modeling exposure and preference to address PU and MNAR issues. By combining the proposed treatment imputation and balanced representation learning, our method leads to more accurate predictions.
4.3 Ablation Study
We conduct ablation studies to validate the effectiveness of our model by examining the impact of the Wasserstein distance (Wass), pairwise loss (Pair), and pointwise loss (Point). Table shows performance metrics (NDCG@K K, Recall @ K, MAP@ K) on Yahoo and Coat datasets. Removing any component degrades performance, as each addresses positive-unlabeled and MNAR challenges in implicit feedback. Removing both Wass and Pair (w/o Wass w/o Pair) yields the lowest performance, as the model cannot stratify negative samples or learn robust representations. Similarly, removing Wass and Point (w/o Wass w/o Point) compromises the causal learning framework. Removing only Pair (w/o Pair) or Point (w/o Point) also reduces performance, though less severely. Both losses contribute uniquely: Pair improves ranking, while Point predicts click likelihood under exposure. Removing only Wass (w/o Wass) degrades performance, as the model loses the ability to stratify negative samples by exposure probability, crucial for distinguishing unclicked items due to low relevance versus lack exposure. In summary, the proposed model with all components (Wass, Pair, Point) achieves the best performance, highlighting the importance of each in addressing implicit feedback challenges and estimating counterfactual outcomes with missing treatments.
4.4 Sensitivity Analysis
Threshold and proportion. Figure 3 shows the sensitivity of the model's performance on the proportion of the HE group and the proportion of the HU group for samples in Dg. With very low or very high a, performance tends to degrade because the model potentially excludes relevant samples or includes irrelevant samples. Similarly, extreme 3 degrades the performance.
Coefficients the loss function. As shown Figure 4 we investigate the impact of the coefficients Apair and Apoint for the pairwise and pointwise losses the proposed model. We evaluate the effects of varying these coefficients on NDCG@K, K, Recall K, and MAP@ across the Yahoo! and Coat datascts. The results show that setting Apalr and Apoint within moderate range (c.g., 0.1, 1, or 10)

---

leads to significant improvements in ranking accuracy and click prediction. Excessively large values (e.g,, 100) overemphasize either ranking or click prediction, degrading performance. Similarly, very small values (e.g., 0.01) ) fail to leverage both losses effectively, resulting in poor performance. The optimal performance achieved when these coefficients are set or 10, striking a balance between ranking accuracy and click likelihood prediction.
5 Related Work
5.1 Implicit Feedback
Early work solving the positive-umlabeled problems in implicit recommender systems includes weighted matrix factorization (WMF), and some approaches like MF [[] downweight negative samples uniformly, some reweight samples via user activities [8]. and others use item popularity
to adjust weights [39]. Exposure models, such ExpoMF g models exposure probabilities
based on item popularity and text topics, while other methods incorporate social or community data [12 40). Most methods rely on pointwise loss, but pairwise loss (e.g., Bayesian personalized ranking, BPR [3] is better suited for ranking tasks by learning relative preferences. However, these methods do not consider the implicit feedback in MNAR, thus may get biased relevance prediction.
Recently, there are some propensity-based methods proposed to address positive-unlabeled and MNAR issues. Rel-MF [22] leverages item popularity for propensity estimation and solves MNAR problem in implicit feedback, while joint learning approaches [41] 42) infer both propensity and recommendation models. Methods using small unbiased datasets employ embedding alignment (6), knowledge distillation 43,. or meta-learning [44 to Iearn exposure/propensity models. For pairwise loss, UBPR (33 proposed debiased loss. However, the propensity models are prone be overly confident, generating extremely inaccurate propensity score estimation [25] 26 45
And the bias and
variance of the estimator can be extremely large with small propensity 27 28
5.2 Causal Recommendation
Causal recommendations refer to applying the causal frameworks 44 47 48 to predict the conversion rates [49] and provide personalized recommendations [1). Compared with previous recommen- dation methods, the causal recommendations can address various biases, including the confounding bias 55, 53 54], popularity bias [55], selection bias [18] 56 57]. and exposure bias [58]. which previous methods fail address because they highly rely on the associations. Most of the methods are based on rnverse-propensee same training and inference space with entire-space multi-task learning approaches [62 63 28 64]
or doubly robust weighting [59 60 61 Some studies propose to use the
They jointly train the parameters to achieve better recommendation performance. More recent works focus on counterfactual learning, which predicts the counterfactual outcomes for the user-item pairs 665 66 67 However, most of them cannot achieve individual-level counterfactual predictions. [68] extends the above methods to perform individual counterfactual predictions. However, the above methods are mainly applied in explicit feedback recommendations. It is impossible to directly apply the above methods to the implicit feedback because is challenging to determine whether an observed negative user-item pair is irrelevant in implicit feedback recommender systems.
6 Conclusion
In this paper, we focus the problem of inferring the true relevance preference of user in the implicit feedback recommendation scenarios. Specifically, to the best of our knowledge, we are the first paper to formalize the relevance prediction problem as a counterfactual outcome estimation problem with missing treatments, which provides a novel approach to tackle this problem. Corre- spondingly, we propose a sample stratification method, which uses treatment variable imputation method with feature similarity -base confidence, reflecting different mechanisms of negative sample generation. addition, we propose balanced representation- based causal learning framework to answer the formalized counterfactual questions and theoretically derive the generalization bound of our causal learning model, showing that minimizing the proposed loss functions can effectively control the bound. Extensive experiments our method. One potential limitation that the method requires pre- pre-specification of the distance thresholds. Changing the framework to an end-to-end manner may further help improve performance.
public benchmark datasets show the effectiveness of
10

---

Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their valuable comments. HL was supported by the Na- tional Natural Science Foundation of China (623B2002). MG was supported by ARC DP240102088 and WIS-MBZUAI 142571.
References
[1] Yifan Hu, Yehuda Koren, and Chris Volinsky, Collaborative filtering for implicit feedback datasets. In ICDM, 2008.
[2] Dawen Liang, Jaan Altosaar, Laurent Charlin, and David Blei. Factorization meets the item embedding: Regularizing matrix factorization with item co-occurrence. In RecSys, 2016.
[3] Chunyuan Zheng, Hang Pan, Yang Zhang, and Haoxuan Li. Adaptive structure learning with partial parameter sharing for post-click conversion rate prediction. In SIGIR, 2025.
[4] Jiahui Liu, Peter Dolan, and Elin Rønby Pedersen. Personalized news recommendation based on click behavior. In IUI, 2010.
[5] Dietmar Jannach, Lukas Lerche, and Markus Zanker. Recommending based on implicit feedback. In Social information access: systems and technologies, pages 510-569, Springer. 2018.
[6] Stephen Bonner and Flavian Vasile. Causal embeddings for recommendation. In RecSys, pages 104-112, 2018
[7] Beibei Li, Beihong Jin, Jiageng Song, Yisong Yu, Yiyuan Zheng, and Wei Zhou, Improving micro-video recommendation via contrastive multiple interests. In SIGIR, 2022.
[8] Dawen Liang, Laurent Charlin, James Melnemny, and David M Blei. Modeling exposure in recommendation. In WWW, 2016.
[9] Haoxuan Li, Chunyuan Zheng, Wenjie Wang. Hao Wang. Fuli Feng, and Xiao-Hua Zhou. Debiased recommendation with noisy feedback. In SIGKDD, 2024.
[10] Shuqiang Zhang, Yuchao Zhang, Jinkun Chen, and Haochen Sui. Addressing correlated latent exogenous variables debiased recommender systems. In SIGKDD, 2025.
[11] Hao Wang, Zhichao Chen, Haotian Wang, Yanchao Tan, Licheng Pan, Tianqiao Liu, Xu Chen, Haoxuan Li, and Zhouchen Lin. Unbiased recommender learning from implicit feedback via weakly supervised learning. In ICML., 2025.
[12] Menghan Wang, Xiaolin Zheng, Yang Yang, and Kun Zhang. Collaborative filtering with social exposure: A modular approach to social recommendation. In AAAI, 2018.
[13] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thiemas Bpr: Bayesian personalized ranking from implicit feedback. UAI, 2009.
[14] Fengxiang He, Tongliang Liu, Geoffrey I Webb, and Dacheng Tao. Instance-dependent pu learning by bayesian optimal relabeling. arXiv preprint
2018.
[15] Jessa Bekker, Pieter Robberechts, and Jesse Davis. Beyond the selected completely at random assumption for learning from positive and unlabeled data. In ECML/KKDD, 2019.
[16] Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing-not-at-raadoo implicit feedback. In WSDM 2020.
[17] Chunyuan Zheng. Haocheng Yang. Haoxuan Li, and Mengyue Yang. Unveiling extrancous sampling bias with data missing-not-a aaassinn-no--t---anddde In NeurlPS, 2025.
[18] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In ICML, 2016.
[19] Harald Steck. Training and testing of recommender systems on data missing not at random. In SIGKDD, 2010.

---

[20] Longqi Yang. Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for -tta--ooom implicit feedback. In RecSys, 2018.
[21] Yehuda Koren, Robert Bell, and Chris Volinsky, Matrix factorization techniques for recom- mender systems. Computer, 42(8))30-37, 2009. doi: 10.1 10.10009002202
[22] Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing-noo--t--andoon implicit feedback. In WSDM, 2020.
[23] Yuta Saito. Unbiased pairwise learning from biased implicit feedback. In ICTIR, 2020.
[24] Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: well-founded and easily implemented improvement logistic calibration for binary classifiers. In AISTATS, 2017.
[25] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. In /CML, 2017.
[26] Yu Bai, Song Mei, Huan Wang, and Caiming Xiong. Don't just blame over-parametrization for over-confidence: Theoretical analysis of calibration in binary classification. In ICML, 2021.
[27] Haoxuan Li, Chunyuan Zheng, and Peng Wu. StableDR: Stabilized doubly robust learning for recommendation on data missing not at random. In ICLR, 2023.
[28] Hao Wang, Tai-Wei Chang, Tianqiao Liu, Jianmin Huang, Zhichao Chen, Chao Yu, Ruopeng Li, and Wei Chu. Esem2: entire space counterfactual multi-task model for post-click conversion rate estimation. In SIGIR, 2022.
[29] Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. TDR-CL: Targeted doubly robust collaborative learning for debiased recommendations In ICLR, 2023.
[30] Jun Wang, Haoxuan Li, Chi Zhang, Dongxu Liang, Enyun Yu, Wenwu Ou, and Wenjia Wang. Counterelr: Counterfactual contrastive learning with non-random missing data in recommenda- tion. In ICDM, 2023.
[31] Yanghao Xiao, Haoxuan Li, Yongqiang Tang, and Wensheng Zhang. Addressing hidden confounding with heterogeneous observational datasets for recommendation. In NeurlPS, 2024.
[32] Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Representation learning for treatment effect estimation from observational data. NeurIPS, 2018
[33] Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. In ICML, 2016.
[34] Bernhard Schölkopf, John C Platt, John Shawe-Taylee Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of a high nigh-dimensiooal distribution. Neural Computation, 13(7):1443- 1471, 2001.
[35] Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In ICML, 2017.
[36] Siyi Lin, Sheng Zhou, Jiawei Chen, Yan Feng, Qihao Shi, Chun Chen, Ying Li, and Can Wang. ReCRec: Reasoning the causes of implicit feedback for debiased recommendation. ACM Transactions on Information Systems, 2024.
[37] Yi Ren, Hongyan Tang, Jiangpeng Rong, and Siwen Zhu. Unbiased pairwise learning from implicit feedback for recommender systems without biased variance control. In SIGIR, 2023.
[38] Rong Pan and Martin Scholz. Mind the gaps: weighting the unknown in large-scale one-class collaborative filtering. In SIGKDD, 2009.
[39] Xiangnan He, Hanwang Zhang, Min- Yen Kan, and Tat-Seng Chua. Fast matrix factorization for online recommendation with implicit feedback. In SIGIR, 2016.
12

---

[40] Jiawei Chen, Can Wang, Sheng Zhou, Qihao Shi, Jingbang Chen, Yan Feng, and Chun Chen. Fast adaptively weighted matrix factorization for recommendation with implicit feedback. In AAAI, 2020.
[41] Ziwei Zhu, Yun He, Yin Zhang, and James Caverlee. Unbiased implicit recommendation and propensity estimation via combinational joint learning. In RecSys, 2020.
[42] Jae-woong Lee, Seongmin Park, Joonseok Lee, and Jongwuk Lee. Bilateral self-unbiased learning from biased implicit feedback. In SIGIR, 2022.
[43] Dugang Liu, Pengxiang Cheng, Zhenhua Dong, Xiugiang He, Weike Pan, and Zhong Ming. A general knowledge distillation framework for counterfactual recommendation via uniform data. In SIGIR, 2020.
[44] Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. Autodebias: Learning to debias for recommendation. In SIGIR, 2021.
[45] Honglei Zhang, Shuyi Wang, Haoxuan Li, Chunyuan Zheng, Xu Chen, Li Liu, Shanshan Luo, and Peng Wu. Uncovering the propensity identification problem in debiased recommendations. In ICDE, 2024.
[46] Hao Wang, Jiajun Fan, Zhichao Chen, Haoxuan Li, Weiming Liu, Tianqiao Liu, Quanyu Dai, Yichao Wang, Zhenhua Dong, and Ruiming Tang. Optimal transport for treatment effect estimation. In NeurlPS, 2023.
[47] Peng Wu, Haoxuan Li, Chunyuan Zheng, Yan Zeng, Jiawei Chen, Yang Liu, Ruocheng Guo, and Kun Zhang. Learning counterfactual outcomes under rank preservation. In NeurIPS, 2025.
[48] Hao Wang, Zhichao Chen, Zhaoran Liu, Xu Chen, Haoxuan Li, and Zhouchen Lin. Proximity matters: Local proximity enhanced balancing for treatment effect estimation. In SIGKDD, 2025.
[49] Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong, Xiao-Hua Zhou, Rui Zhang, Xiuqiang He, Rui Zhang, and Jie Sun. A generalized doubly robust learning framework for debiasing post-click conversion rate prediction. In SIGKDD, 2022.
[50] Peng Wu, Haoxuan Li, Yuhao Deng, Wenjie Hu, Quanyu Dai, Zhenhua Dong, Jie Sun, Rui Zhang, and Xiao-Hua Zhou. On the opportunity of causal learning in recommendation systems: Foundation, estimation, prediction and challenges. In ISCAI, 2022.
[51] Haoxuan Li, Chunyuan Zheng, Sihao Ding, Fuli Feng, Xiangnan He, Zhi Geng, and Peng Wu. Be aware of the neighborhood effect: Modeling selection bias under interference for recommendation. In ICLR, 2024.
[52] Wenjie Wang. Yang Zhang, Haoxuan Li, Peng Wu, Fuli Feng, and Xiangnan He. Causal recommendation: Progresses and future directions. In Tutorial on SIGIR, 2023.
[53] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confound- ing with few unbiased ratings in debiased recommendations. In WWW, 2023.
[54] Chuan Zhou, Yaxuan Li, Chunyuan Zheng, Haiteng Zhang, Min Zhang, Haoxuan Li, and Mingming Gong. A two-stage pretraining-fieetunnng framework for treatment effect estimation with unmeasured confounding. In SIGKDD, 2025.
[55] Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. Causal intervention for leveraging popularity bias in recommendation. In SIGIR, 2021.
[56] Haoxuan Li, Kunhan Wu, Chunyuan Zheng, Yanghao Xiao, Hao Wang, Zhi Geng, Fuli Feng, Xiangnan He, and Peng Wu. Removing hidden confounding in recommendation: unified multi-task learning approach. In NewrIPS, 2023.
[57] Haoxuan Li, Chunyuan Zheng, Shuyi Wang, Kunhan Wa, Eric Wang, Peng Wu, Zhi Geng, Xu Chen, and Xiao-Hua Zhou. Relaxing the accurate imputation assumption doubly robust learning for debiased collaborative filterin. In ICML, 2024.
13

---

[58] Sami Khenissi and Olfa Nasraoui. Modeling and counteracting exposure bias in recommender systems. arXiv preprint arX1v:2001.08832, 2020.
[59] Haoxuan Li, Quanyu Dai, Yuru Li, Yan Lyu, Zhenhua Dong, Xiao-Hua Zhou, and Peng Wu. Multiple robust learning for recommendation. In AAAI, 2023.
[60] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, and Peng Cui. Propensity matters: Measuring and enhancing balancing for recommendation. In ICML, 2023.
[61] Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, Peng Wu, Zhi Geng, Xu Chen, and Peng Cui. Debiased collaborative filtering with kernel-based causal balancing. In ICLR, 2024.
[62] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture- offture-of-experts.. In SIGKDD, 2018.
[63] Wenhao Zhang, Wentian Bao, Xiao- Yang Liu, Keping Yang, Quan Lin, Hong Wen, and Ramin Ramezani. Large-scale causal approaches to debiasing post-click conversion rate estimation with multi-task learning. In WWW, 2020.
[64] Hao Wang, Zhichao Chen, Zhaoran Liu, Haozhe Li, Degui Yang. Xinggao Liu, and Haoxuan Li. Entire space counterfactual learning for reliable content recommendations. IEEE Transactions on Information Forensics and Security, 2025.
[65] Tiankai Gu, Kun Kuang, Hong Zhu, Jingjie Li, Zhenhua Dong, Wenjie Hu, Zhenguo Li, Xi- uqiang He, and Yue Liu. Estimating true post-click conversion via group-stratified counterfactual inference. In SIGKDD, 2021.
[66] Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, and Xiangnan He. Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system. In SIGKDD, 2021.
[67] Chongming Gao, Shiqi Wang, Shijun Li, Jiawei Chen, Xiangnan Hc, Wenqiang Lei, Biao Li, Yuan Zhang, and Peng Jiang. CIRS: Bursting filter bubbles by counterfactual interactive recommender system. ACM Transactions on Information Systems, 42(1):1-27, 2023.
[68] Haoxuan Li, Chunyuan Zheng, Peng Wu, Kun Kuang, Yue Liu, and Peng Cui. Who should be given incentives? counterfactual optimal treatment regimes learning for recommendation. In SIGKDD, 2023.
[69] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NeurIPS, 2007.
[70] Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In NewrIPS, 2000.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We list in the abstract and introduction the contributions of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We will provide proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have stated the experimental details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[Yes]
The data is publicly available and we will provide all code if the paper is accepted.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/pil wwwppbblccgguiicuceccccccoe.oopp wet.ggddddee/eeebbiniinn
ssiiiioiiccyyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided all the details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include the error bars in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include the computer resources and approximate time of execution in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwwwp..cccaaaa//ccccaaaa
Answer: [Yes]
Justification:
This paper has no ethics problem.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is about counterfactual estimation under missing treatment, posing no direct societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no such risks in our paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original papers of the used assets.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The method in this paper does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Proof of Theorem
Assumption A.1 (Inverse Representation and Function Class 35].. The representation + X - A a one-to-one function, with inverse V. Let be a family of functions 9 A - y. Assume there exists constant B& 0, such that H.
B4 (h o t (a) Y(1)² E G.
Based on Assumption A.I we then derive the following generalization bound:
Theorem A.2 (Generalization Bound), Under Assumplion A.I the deviation between the estimated
relevance h( @ (x)) and expected relevance mi (1) E. 11 X x averaging on all user-item
pairs has the upper bound:
m1(x)) S Exgrro((((((a))) - Y(1)² R R= 1
factual loss of the DP and HE groups
+P(O= R 1) By IPMc R=1,0=0 R=1.0=1 + P(R = 0)
=0) BB (PMc .p _
R=0 R=1)
P4
distribution shift given
UN group
distribution shift
-((
variance of patential outcome
Proof. For simplicity, we define <(x) [x))((x))) Y(1)²² x 2] and Crotal = Boyta=BBl([aat Bz(c(x)] = [(()(((z)) (1))²²]. The target error that we would like bound cTarget := Ez (x(()((()) mi (.. We will first derive the connection between the target error éTarget and the total error Tooaat Consider the following conditional expectation:
-((h(((z))- - ( X x]
=E(((((((=)) (x)) (m)(x) Y(1))) =
E[ -E((h(((z))- - -m1(=))² | =| (x))) | x
+=)( (m) (m))z)))(m((z)) Y()) ==
=E((h(((z) -m,(=))² ( X
(z) Y(1))² x
Taking expectation w.r.t the distribution of X, we have:
ETotal
(F((((((())) Y(1)² X =x]}
-E,(B(((((z)) m1(x))² X x]} (m(()
x= = x}}
=E((n(((x))) - my (x))² + Y(1))
=Erarget + E[ (x) (1))
(2)
Denoting L(z) = (h()(z)) - (1)) v = P(R = 0), !== Ezle(x) | R 1] and R== Ex(e(x) R = 0), we can decompose éTotal follows:
éTotal
= [c(x)(((>) R= - E.p R= 0] R 0)
R=1 (1 vn) R=0
R=1 R= vo + RRDD
R= + vo R=I)
R=1 to (ʃ (x))(x)))))))))))))) R= 00dr ((x))(x) R 1)dx
R=1 Uo
(<)(p(x) R 0) ppxx R= 1))dr
R=1 Uo
- x =>]pp< R = 0) p(x R = 1))
R=1 U0
((=)(o())R p(x R- 1))dz.
22

---

Based on Assumption A.1
we have L(x)/BB = Ba (h v V(a) - - E G, and thus
/
(xx)((x) R 0) p(= R 1))de
L(x) (p(x)R = 0) - p(x = 1))dr
= -4
BR
L(V(a)) (p(af = 0) p(a R 1))da
= =B4
/
SB
sup
((a)(rt=0(0) p4 1= (a))dal
gEG
= BBg IPMc R
R=0
.p 1
(4)
Combining Eq. 33 and Eq.
we have
eootal S e R=1
Un Bs IPMo(P6 pb 1)-1).
R=0
(5)
Next, we denote uo -P(O- 0 R 1), R=10== = Estrol((x) | R 1,0
1] and
R==10=0 [x)nol((2) R = 1, = 0) and then decompose R=1 as follows:
R==
f==100= (1 - uo) + R=1,0=0 40
=6 R=10=1 + un R=1,00= - 16R=10000-
R=110=1)
= R=1 + on ( ((0)p(() R 1,0 0)dr - =,)dx
<(z)p(x R= == = 1)dr
= R=1,0=1 + u0
( 0) p(u R= 1,=1,0 = 1))dr
R=1,0=1
+ uo / B[(b(((z))- Y())² X =](p(< R = 1, 0) p(x R 1,0 1))de
-= + up [(x)(p(x) R= 1 0) - p(z R = 1,0 = 1))dz.
(6)
Analogous to the derivation of Eq.
we have
/ (xx 1=1,0= p(x R= 1,0 1))dr
/ L(x) (p(x R= 1,0 0) p(x R= 1,0 1))dr
= Bx
By
(()((a)
=B+
BR (p(a) R= 0) p(a R = 1,0 = 1))da
/ g(a)(pp
((a)(p6-1.00
(=100-0(a))))))))))))))))))))))) (a) Pv
R=1,0-1 (a))
<Bp sup
9EG
=B IPMc R=1,0=0 Pp R=1,0==).
(7)
Combining Eq. 66 and Eq.
we have
R=1 R=1,0=1 + uo Bo 1PMc Pp
R=1,0=0 R=1,0==),
(8)
Note that R=1,0=1 (x)role(x) and combining the results of Eq. 2 Eq. 55) and Fq. (8. we complete the proof of the theorem.
===1 R=1.0-
(
1,,00
23

---

B Data Preprocess
All of the datasets used in this paper contain user ratings on an item as explicit feedback. So we simulate the implicit feedback mechanism using the following data preprocessing pipeline.
1. Transform all ratings into relevance scores using the following formula:
2 1
Tu.i + (1
2r max
where Tu, denotes the rating for each user-item pair the observed set O2 and r'ax the maximum rating. The parameter E [0, 1] controls the noise level. Following the previous studies [23 36], we set e = 0.1 for the training datasets and e = 0 for the test datasets to ensure unbiased evaluation.
2. Sample the binary relevance Su,i using Bernoulli sampling:
Sa, ~ Bern(ru,)) v(u,i) E O,
where Bern()) denotes the Bernoulli distribution.
3. Define the exposure variable for all user-item pair (u, U x I:
Ou = {1
if item is rated by user u,
if item i is not rated by user u.
4. Finally, we sample the binary outcome as the implicit feedback:
Sa,
if Ou,i = 1,
Yu,i t0
if O 1.
Note that S and Ou,i unobservable in our setting and the training data is {(u, i, Yu.)) (u, i) E 2 x Z}.
C Efficient Alternatives of Kernel-Based Hypersphere Model
Although the kernel-based hypersphere model in Section 3.3 introduces (m²) complexity with m positive samples, we would like to emphasize that the calculation of the kernel function is only one possible option; the core lies in the framework of treatment imputation with confidence. Here, we provide two more efficient and scalable kernel function approximation methods, i.e., Random Fourier Features (RFF) 69 and Nyström approximation (70 replace the method Section 3.3 Experimental results on the Coat and Yahoo datasets are presented in E] showing that regardless of the kernel function calculation method adopted, the ranking performance is guaranteed.
Table 4: Ranking performance with efficient alternatives kernel-based hypersphere model.
Methods
K=8
NDCG
Recalloc
MAP
NDKGG
Recalli
MAP
NDCGØK
Recall
MAP
(SVDID)
0.562
0.00
0.624
0.009
0.007
0.625
0.006
0.547
0.006
0.681
0.004
0.004
0.582
(RFF)
0,578
6.002
0.585
(Nystttm)
0.572
0.587 0.007
(SVDD)
0,368
0.009
0.099
0.389
(RPP)
0.015
0,387
0.468
0.036
0.360
(Nysuttm)
0,389 8.011
0.017
0.314
0.427
0.023
0.038
0.347
0.486
0.015
0.030
0.384
D Experiments Compute Resources
We conduct all experiments server with 1 112ccore Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz.
The server is equipped with a 512GB random access memory (RAM). To reproduce all the experi- mental results including the baselines takes few hours.
24


Reviews:
[{"review_id": "NcOWJg6Frb", "reviewer": "Reviewer_dfyx", "summary": "This paper proposes Counterfactual Implicit Feedback (Counter-IF), a novel approach for modeling implicit feedback in recommender systems by framing relevance prediction as a counterfactual estimation problem with missing treatments. Traditional implicit feedback methods suffer from two main issues: (1) the positive-unlabeled (PU) problem, where non-clicks may be due to non-exposure rather than disinterest, and (2) missing-not-at-random (MNAR) biases, where popular items are more likely to be clicked simply due to increased exposure. Counter-IF addresses these challenges by stratifying user-item pairs into four groups, definitely positive, highly exposed, highly unexposed, and unknown, using a novel confidence-based treatment imputation method. The model then estimates counterfactual outcomes using a causal representation learning framework that integrates pointwise and pairwise losses across strata, and regularizes with Integral Probability Metrics (IPM) to align distributions.", "questions": "As mentioned in the Strengths And Weaknesses,  treatment imputation step relies on a kernel-based hypersphere model using the Gaussian kernel, which introduces significant computational overhead due to the quadratic complexity in the number of positive samples. \n\nHave the authors considered using more scalable alternatives, such as random Fourier features, Nystrom approximation, or neural embedding-based methods, to reduce the computational cost while retaining the benefits of exposure confidence estimation? Is it possible to include this extension?", "limitations": "Yes, the author(s) mentioned the limitation of pre-specifying the distance threshold. They could consider discussing one more limitation regarding scalability to large-scale scenarios, as mentioned in the \"Strengths and Weaknesses\" section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- To my best knowledge, this paper proposes a novel problem formulation. It is the first to formulate relevance prediction under implicit feedback as a counterfactual outcome estimation problem with missing treatments, offering a fresh causal perspective on a longstanding challenge in recommendation systems.\n\n- The paper introduces an exposure estimation method based on hypersphere distance using kernel methods, enabling imputation only for high-confidence samples and reducing risk from incorrect imputations.\n\n- The experimental settings are clear, and experiments on public datasets show consistent and statistically significant improvements over competitive baselines across multiple metrics.\n\n- The paper includes ablation studies, sensitivity analyses, and hyperparameter tuning, providing clear insight into which components contribute most to performance and demonstrating model robustness.\n\nOverall, the problem formulation is interesting to me, and the paper presentation is easy to follow.\n\n\nWeaknesses:\n\n- One limitation of the paper lies in the treatment imputation step, which relies on a kernel-based hypersphere model using the Gaussian kernel. While this approach provides a principled way to estimate exposure confidence and stratify negative samples, it introduces significant computational overhead due to the quadratic complexity in the number of positive samples. Specifically, the need to compute and store the full kernel matrix and solve a constrained optimization problem limits the scalability of the method to large-scale recommendation scenarios.\n\n- Suggestion: To improve scalability, the kernel-based imputation can be replaced or approximated using random Fourier features, Nystrom approximation, or deep neural representations. These methods significantly reduce computational complexity while preserving the core idea of stratifying user-item pairs by exposure confidence.", "was_revised": true, "final_justification": "The authors have thoroughly addressed my questions, so I will maintain my positive score.", "rebuttal": "> Gaussian kernel introduces significant computational overhead\n\nThank you for highlighting these critical points. We acknowledge that the kernel-based hypersphere model introduces $O(m^2)$ complexity with $m$ positive samples, which poses challenges for large-scale data. We have explored scalable alternatives as you suggested:\n\n**Random Fourier Features (RFF)** and **Nyström approximation**  reduce kernel matrix computations from $O(m^2)$  to $O(md)$, where $d$ is the dimension of features and is significantly smaller than $m$. Preliminary tests on the Coat and Yahoo dataset show that RFF performs even better than original SVDD while cutting runtime by xx%. Nyström retain xx% of the original performance while cutting runtime by xx%. We plan to include this extension in the revised manuscript.\n\n We can now achieve to address the scalability concern for this paper by replacing the kernel-based method. Sincerely thank you again for your valuable suggestions, which make our work much more solid.\n\n\n||RFF|Nystroem|Ours|\n|-|-|-|-|\n||**Yahoo**|||\n|NDCG@3|0.578|0.571|0.562|\n|Recall@3|0.645|0.637|0.624|\n|MAP@3|0.513|0.508|0.499|\n|NDCG@5|0.638|0.634|0.625|\n|Recall@5|0.790|0.786|0.776|\n|MAP@5|0.560|0.555|0.547|\n|NDCG@8|0.693|0.685|0.681|\n|Recall@8|0.939|0.925|0.930|\n|MAP@8|0.595|0.587|0.582|\n||**Coat**|||\n|NDCG@3|0.361|0.389|0.368|\n|Recall@3|0.384|0.390|0.382|\n|MAP@3|0.282|0.314|0.296|\n|NDCG@5|0.387|0.427|0.414|\n|Recall@5|0.454|0.495|0.478|\n|MAP@5|0.309|0.347|0.332|\n|NDCG@8|0.468|0.486|0.473|\n|Recall@8|0.671|0.653|0.660|\n|MAP@8|0.360|0.384|0.369|\n|"}, {"review_id": "Y9rEC4o2p6", "reviewer": "Reviewer_x5VN", "summary": "This paper addresses the challenges of PU and MNAR of learning from implicit feedback in recommender systems. The authors propose to formalize this problem as one of counterfactual estimation, with the goal to predict whether a user would click an item if they were exposed to it. Specifically, the proposed method will firstly use a support vector-based method to stratify user-item pairs into four groups visa missing treatment imputation. Then it employs a causal representation learning framework with a hybrid loss function to combine pointwise loss for samples with observed/imputed outcomes and pairwise loss for samples with unobserved outcomes. Experiments are conducted on several public datasets, showing improved performance over existing methods.", "questions": "1. Given that unconfoundedness is a strong assumption likely violated in real-world recommender systems, could you provide further justification for its use? More importantly, how robust is your method to potential violations from unobserved confounders, which are common in practice?\n\n2. The SVDD-based stratification appears to be a computational bottleneck, especially with kernel methods. Could you please discuss the time complexity of this step and outline a practical strategy for scaling your approach to industrial-sized datasets?", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of stratifying negative samples based on a confidence measure of exposure is a clear improvement. This allows the model to differentiate between various reasons for non-interaction, moving beyond the simplistic assumption that all unobserved items are equivalent.\n\n2. The experimental results demonstrate the proposed method improve the ranking performance in recommendation systems. \n\n3. The paper's primary strength is its insightful reframing of the implicit feedback problem as a counterfactual estimation task. \n\nWeaknesses:\n1. The paper relies on the unconfoundedness assumption (Y_u,i(1) ⊥ O_u,i | X_u,i). This assumption is very strong and highly likely to be violated in real-world recommender systems, where the exposure mechanism (the previous model) is a powerful, partially unobserved confounder. The paper fails to provide any discussion of the potential impact of violating this assumption or any sensitivity analysis to demonstrate the model's robustness to unobserved confounding.\n\n2. The two-stage process, requiring a potentially O(N^2) kernel-based SVDD step, is not scalable to large, industrial datasets. The paper misses discussion of computational complexity or strategies for making the approach practical.\n\n3. The framework introduces a large number of sensitive hyperparameters (α, β for stratification, kernel parameters q, loss weights λ_point, λ_pair). This complexity presents a significant tuning challenge, potentially limiting the model's practical utility and ease of deployment.", "was_revised": true, "final_justification": "The authors provide additional information during the rebuttal which adjust my concern. I will keep my original score.", "rebuttal": "We appreciate your insights and address each point below.\n\n > Unconfoundedness Assumption\nAlthough the violation of unconfoundedness assumption is arthogonal to our contribution, \nand the unconfoundedness assumption is common in implicit feedback recommendation litertures [1,2], we agree with the reviewer that it is meaningful to validate the robustness of our proposed method against its violation.  To test robustness, we conduct experiments on a synthetic dataset, introducing varying levels of unmeasured confounding. The data generating process is as follows.\n- $U\\sim \\mathcal{N}(\\mu,1)^{n_U\\times n_I}$,\n- $\\tilde r_{u,i}=U_{u,i}+\\epsilon_{u,i}$,\n- $r_{u,i}=round(\\min(0,\\max(\\tilde r_{u,i},5)))$,\n- $O_{u,i}\\sim Bernoulli(\\sigma(U_{u,i}-\\mu))$\n\nHere the potential outcome and the missing mechanism is counfounded by an unmeasured variable $U$. We take $\\mu=1,2,3$ for different strength of unmeasured confounding. The results show that our proposed method consistently outperforms the baselines and indicates reasonable resilience against stronger confounding. We will include these results in the revised version of the manuscript.\n\n\n||K=3|||K=5|||K=8|||\n|-|-|-|-|-|-|-|-|-|-|\n||NDCG@K|Recal@K|MAP@K|NDCG@K|Recal@K|MAP@K|NDCG@K|Recal@K|MAP@K|\n|||||$\\mu$=1||||||\n|WMF|0.179|0.224|0.154|0.201|0.282|0.168|0.263|0.472|0.192|\n|BPR|0.182|0.232|0.159|0.210|0.300|0.175|0.272|0.488|0.203|\n|Ours|**0.195**|**0.250**|**0.172**|**0.225**|**0.32**|**0.188**|**0.29**|**0.510**|**0.218**|\n|||||$\\mu$=2||||||\n|WMF|0.167|0.215|0.148|0.189|0.282|0.159|0.249|0.448|0.186|\n|BPR|0.174|0.221|0.152|0.201|0.287|0.167|0.260|0.458|0.193|\n|Ours|**0.188**|**0.238**|**0.165**|**0.218**|**0.305**|**0.180**|**0.280**|**0.490**|**0.208**|\n|||||$\\mu$=3||||||\n|WMF|0.154|0.193|0.137|0.173|0.258|0.151|0.24|0.427|0.176|\n|BPR|0.165|0.210|0.145|0.195|0.274|0.158|0.248|0.435|0.185|\n|Ours|**0.178**|**0.225**|**0.155**|**0.205**|**0.299**|**0.173**|**0.265**|**0.465**|**0.200**|\n|\n > SVDD Complexity and Scalability\n\n The original SVDD step has $O(N^2)$ complexity, but we mitigate this by implementing the Random Fourier Features (RFF)** and **Nyström approximation**  reduce kernel matrix computations from $O(N^2)$  to $O(Nd)$, where $d$ is the dimension of features and is significantly smaller than $N$. Surprisingly, we find that they even outperforms the original SVDD while reduces the runtime by 75%. We will include these results in the revised version of our manuscript, and hope this can address your concern about scalability.\n\n# Compare with RFF and Nystroem\n\n||RFF|Nystroem|Ours|\n|-|-|-|-|\n||**Yahoo**|||\n|NDCG@3|0.578|0.571|0.562|\n|Recall@3|0.645|0.637|0.624|\n|MAP@3|0.513|0.508|0.499|\n|NDCG@5|0.638|0.634|0.625|\n|Recall@5|0.790|0.786|0.776|\n|MAP@5|0.560|0.555|0.547|\n|NDCG@8|0.693|0.685|0.681|\n|Recall@8|0.939|0.925|0.930|\n|MAP@8|0.595|0.587|0.582|\n||**Coat**|||\n|NDCG@3|0.361|0.389|0.368|\n|Recall@3|0.384|0.390|0.382|\n|MAP@3|0.282|0.314|0.296|\n|NDCG@5|0.387|0.427|0.414|\n|Recall@5|0.454|0.495|0.478|\n|MAP@5|0.309|0.347|0.332|\n|NDCG@8|0.468|0.486|0.473|\n|Recall@8|0.671|0.653|0.660|\n|MAP@8|0.360|0.384|0.369|\n|\n\n> Hyperparameter Tuning\n\nDespite our method has 5  hyperparameters in total, but are used in two separate stages, i.e., the percentiles for stratification $\\alpha, \\beta$ and kernel parameter $q$ at the imputation and stratification stage, while loss weights $\\lambda_{point}$ and $\\lambda_{pair}$ in the counterfactual representation model training stage. In this way, the searching space is much smaller."}, {"review_id": "s9SAVRGOq8", "reviewer": "Reviewer_goth", "summary": "The counterfactual implicit feedback model formulates the implicit feedback problem in recommender systems as a counterfactual estimation problem with missing treatment variables. Based on the implicit feedback positive and negative samples, user–item pairs are divided into four mutually exclusive groups, namely definitely positive (DP), highly exposed (HE), highly unexposed (HU), and unknown (UN) groups. Then, by stratifying user–item pairs and combining pointwise loss and pairwise loss, the model learns causal representations to estimate counterfactual outcomes. Extensive experiments on public datasets demonstrate the effectiveness of Counter-IF.", "questions": "1. Since SVDD is essentially based on geometric distances in the feature space to perform grouping, is this approach suitable for cold-start users or items?\n2. In Section 2, the paper mentions “We divide all user-item pairs according to Yu,i, i.e.,” and in Section 3.4 it mentions “where and are the set of positive samples and negative samples.” Are the sets of positive and negative samples referred to in these two places consistent?\n3. Since Counter-IF heavily relies on the accuracy of treatment imputation, would inaccuracies or biases in the imputation process have any impact on the subsequent results?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. From a new perspective, this work formalizes the relevance prediction under implicit feedback scenarios as a counterfactual outcome estimation problem with missing treatment variables.\n2. Proposes a novel sample stratification method combined with confidence treatment variable imputation to address the issue of missing treatment variables.\n3. Propose a causal representation learning method to predict the relevance between users and items across the set of all user-item pairs.\n4. Theoretically derives the generalization bound of the Counter-IF framework and proves that minimizing the proposed loss functions can effectively control this bound.\n5. Conducts extensive experiments on publicly available real-world datasets, demonstrating that the proposed Counter-IF method significantly outperforms existing state-of-the-art approaches.\n\nWeakness:\n1. Is there any hyperparameter analysis for the DP sample ratio hyperparameter  and the negative sample ratio hyperparameter  mentioned in Section 3.3?\n2. In Section 3.4, the sentence “In contrast, when the samples are not assigned the treatment (i.e., Ou,i = 0, corresponding to UN and HU groups” has a formatting issue, as it only includes half of the closing parenthesis.\n3. Figure 2 is presented in the paper, but the text does not explain which part of the content it is associated with.\n4. Since the paper adopts a counterfactual approach, it is recommended to include baseline comparisons with existing counterfactual recommendation methods in the experimental section.\n5. The theoretical analysis of the generalization bound is relatively abstract, it is suggested to illustrate the terms in the bound with practical examples.\n6. The paper does not specify which parts of the main text correspond to Appendices A, B, C, and D, it is recommended to clarify this relationship.\n7. It is suggested to add visualizations (e.g., t-SNE plots) of the distributions of the DP, HE, HU, and UN groups in the experiments to verify the reasonableness of the stratification.", "was_revised": true, "final_justification": "This paper presents a novel approach to implicit feedback in recommender systems by framing the task as a counterfactual estimation problem. The authors provide compelling theoretical justifications and extensive experiments that validate the superiority of the proposed method over existing baselines. However, some issues remain. While hyperparameter sensitivity and imputation errors have been clarified, the manuscript still has areas needing further refinement. For instance, parts of the text and figures lack clear associations between the theory and implementation. The paper suffers from some structural clarity issues, such as inconsistent appendix references and notations, which have now been clarified. \n\nIn summary, I recommend a weak accept, as the contributions outweigh the residual shortcomings, especially if the presentation is further refined in the final version.", "rebuttal": "Thank you for these detailed and constructive suggestions. Here are our responses.\n\n> W1 Hyperparameter Analysis\n\nWe have actually conducted the sensitivity analysis for the DP sample ratio hyperparameter and the negative sample ratio\nhyperparameter as Figure 3 of the submitted manuscript shows. Unfortunately, we made a typo in Fig 3, which may have misled you. The $D$ in Fig 3 should be the corresponding $\\alpha$ and $P_{HU}$ should be $\\beta$ defined in Section 3.3.\n\n> W2 Formatting Issue\n\nThanks to point out, we have made a typo here, and the correct sentence should be *In contrast, when the samples are not assigned the treatment (i.e., $O_{u,i} = 0$, corresponding to UN and HU groups), we treat these unobserved interactions as counterfactual data and employ pairwise loss to model the relative rankings.*\n\n> W3 Figure 2 Explanation\n\nThanks for your suggestion. Figure 2 illustrates our causal representation learning framework: input features are mapped via $\\Phi$ to a latent space, where pointwise losses (DP/HE) and pairwise losses (UN vs. HE, HU vs. UN) guide training. IPM regularization aligns distributions of $R=0$ and $R=1$ samples, as well as the $R=1, O=0$ and $R=1, O=1$ groups. We will modify the manuscript so that each part corresponding to the figure is refered.\n\n> W4 Counterfactual Baselines\n\nThanks for your suggestion, we added comparisons with IPS and DR on Coat and Yahoo datasets. Our method outperforms them significantly. We will include these baselines in the final version of this paper.\n\n|Methods||K=3|||K=5|||K=8||\n|-|-|-|-|-|-|-|-|-|-|\n||NDCG@K|Recall@K|MAP@K|NDCG@K|Recall@K|MAP@K|NDCG@K|Recall@K|MAP@K|\n||||||**Yahoo**|||||\n|IPS|0.541±0.006|0.599±0.007|0.474±0.005|0.604±0.006|0.755±0.008|0.530±0.005|0.664±0.005|0.919±0.005|0.563±0.005|\n|DR|0.544±0.005|0.598±0.007|0.477±0.004|0.607±0.005|0.757±0.009|0.531±0.004|0.666±0.004|0.919±0.006|0.566±0.004|\n|Ours|**0.562±0.007**|**0.624±0.009**|**0.499±0.007**|**0.625±0.005**|**0.776±0.008**|**0.547±0.006**|**0.681±0.004**|**0.930±0.004**|**0.582±0.005**|\n||||||**Coat**|||||\n|IPS|0.337±0.015|0.327±0.018|0.267±0.014|0.372±0.015|0.416±0.019|0.297±0.014|0.429±0.015|0.613±0.024|0.328±0.014|\n|DR|0.341±0.012|0.331±0.016|0.270±0.012|0.375±0.013|0.420±0.021|0.299±0.012|0.433±0.012|0.614±0.021|0.332±0.012|\n|Ours|**0.368±0.011**|**0.382±0.012**|**0.296±0.009**|**0.414±0.012**|**0.478±0.014**|**0.332±0.009**|**0.473±0.012**|**0.660±0.021**|**0.369±0.009**|\n|\n\n> W5 Generalization Bound Explanation\n\n- The first term is the factual loss based on the true value of $Y(1)$ of the DP and HE groups. It measures the error of potential outcome estimation for those in DP and HE groups.\n\n- The second and third terms\nare the IPM distance measuring the distribution shift on $O = 1$ and $O = 0$ given $R = 1$ group and\ndistribution shift on $R = 1$ and $R = 0$ groups weighted by the proportion of UN group $P(R = 0)$. \n\n- The last term measures the minimal variance of potential outcomes on $O=0$ and $O=1$ groups, which is independent of the model selection. \n\n- This bound actually guides the loss design for our method. Intuitively, minimizing the L_point and L_pair can control the factual loss, while minimizing the L_IPM is an effective approach to handle the distribution shifts.\n\n> W6 Appendix Clarification\n\nWe appreciate the reviewer for pointing out this flaws. We should clarify that the Related Work is in Appendix A, the proof for our theory is in Appendix B, the data preprocessing protocal is in Appendix C, the experimental results on KuaiRec dataset is in Appendix D, while Appendix E provides the compute resources we used to conduct all the experiments.\n\n> W7 t-SNE Visualization\n\nAs suggested by the reviewer, we conduct experiments on a synthetic dataset and visualize the distributions of the DP, HE, HU, and UN groups with t-SNE plots. As expected, we see the distribution of HE and HU groups aligns well with the exposed and unexposed populations, and the UN groups lie in the boundry of the two populations. Unfortunately, due to the rule of NeurIPS 2025, we cannot share the visualization results by images at this stage. We are delighted to include the figures in the final version of this paper.\n\n> Q1 Cold-Start Suitability\n\nThank you for providing another potential benefits of our method. We conduct experiments for cold-start users and compare with existing baselines. We investigate the users with the lowest 5%, 10%, 15% or 20% of interactions, and analysis the drop against normal setting. Performance on cold-start dataset is slightly lower than normal setting but still outperforms baselines. \n\n\n|Method|5-quantile|Drop|10-quantile|Drop|15-quantile|Drop|20-quantile|Drop|Normal|\n|-|-|-|-|-|-|-|-|-|-|\n|Yahoo||||||||||\n|BPR|0.465|10.03%|0.473|8.52%|0.485|6.21%|0.501|3.02%|0.517|\n|WMF|0.489|9.12%|0.499|7.31%|0.509|5.42%|0.523|2.81%|0.538|\n|Ours|**0.520**|**7.51%**|**0.528**|**6.12%**|**0.538**|**4.31%**|**0.550**|**2.11%**|**0.562**|\n|Coat||||||||||\n|BPR|0.292|10.02%|0.297|8.31%|0.305|5.91%|0.314|3.01%|0.324|\n|WMF|0.304|8.71%|0.310|6.92%|0.317|4.81%|0.325|2.42%|0.333|\n|Ours|**0.343**|**6.82%**|**0.349**|**5.21%**|**0.355**|**3.51%**|**0.361**|**1.91%**|**0.368**|\n|\n\n> Q2 Notations\n\nNo they are not. In Section 2, we defined $D_1=\\{(u,i)|(u,i)\\in D,Y_{u,i}=1\\}$ and $D_0={(u,i)\\mid(u,i)\\in D,Y_{u,i}=0}$, which is only based on the observed outcome to devided the dataset. While in Section 3.4, the positive and negative samples refer to two groups present in the pairwise loss, which is based on our stratified dataset. We have included 3 pairwise losses, so the positive and negative samples may refer to (DP, HE), (UN, HE) and (HU, UN) respectively.\n\n> Q3 Imputation Errors\n\nThanks for providing another in-depth analysis perspective for our method. We compare the proposed imputation method with random imputation and K Nearest Neighbor (KNN)-based imputation on Coat and Yahoo dataset. Here we choose K=3 and impute a unlabeled sample as positive if all 3 neighbors are positive, and negative if all 3 neighbors are negative. The experimental results shows that our kernel-based imputation significantly outperforms the other imputation methods. This indicate that the imputation quality has a significant impact on the subsequent prediction results, and our kernel-based imputation is effective.\n\n|Metrics||K=3|||K=5|||K=8|||\n|-|-|-|-|-|-|-|-|-|-|-|\n||NDCG@K|Recall@K|MAP@K|NDCG@K|Recall@K|MAP@K|NDCG@K|Recall@K|MAP@K|\n||||||**Yahoo**|||||\n|random|0.472±0.006|0.543±0.009|0.407±0.005|0.543±0.004|0.713±0.007|0.456±0.004|0.611±0.003|0.898±0.007|0.497±0.004|\n|knn|0.471±0.004|0.539±0.006|0.406±0.003|0.544±0.004|0.714±0.008|0.457±0.004|0.610±0.003|0.896±0.004|0.496±0.003|\n|Ours|**0.562±0.007**|**0.624±0.009**|**0.499±0.007**|**0.625±0.005**|**0.776±0.008**|**0.547±0.006**|**0.681±0.004**|**0.930±0.004**|**0.582±0.005**|\n||||||**Coat**|||||\n|random|0.257±0.022|0.261±0.027|0.187±0.019|0.312±0.024|0.395±0.035|0.229±0.021|0.384±0.028|0.590±0.044|0.274±0.025|\n|knn|0.247±0.038|0.256±0.041|0.180±0.033|0.310±0.040|0.402±0.047|0.226±0.037|0.376±0.042|0.579±0.050|0.268±0.040|\n|Ours|**0.368±0.011**|**0.382±0.012**|**0.296±0.009**|**0.414±0.012**|**0.478±0.014**|**0.332±0.009**|**0.473±0.012**|**0.660±0.021**|**0.369±0.009**|\n|"}, {"review_id": "NJyki8bfV8", "reviewer": "Reviewer_SdkD", "summary": "This paper addresses the problem of relevance prediction in recommendation systems using implicit feedback, where observed interactions (e.g., clicks) are sparse and may not fully reflect user preferences. Two key challenges are presented: the positive-unlabeled (PU) nature of the data—since unclicked items may be either irrelevant or unexposed to the user—and the missing-not-at-random (MNAR) bias, where exposure is influenced by item popularity or recommendation frequency.\n\nThe authors formalize the problem with causal language, where exposure is a binary treatment and the relevance is a potential outcome under positive treatment. Then to tackle the presented challenges, the authors propose a method called Counterfactual Implicit Feedback (Counter-IF). Counter-IF stratifies user-item pairs into four groups based on estimated exposure confidence: definitely positive (DP), highly exposed (HE), highly unexposed (HU), and unknown (UN). Treatment status for negative feedback is imputed using a feature-based confidence model.\n\nThe method combines pointwise and pairwise losses across strata, alongside representation learning with distribution alignment (via IPM regularization), to improve relevance estimation. Theoretical analysis provides a generalization bound under this framework, and experiments on real-world datasets demonstrate improved performance over existing baselines.", "questions": "1) What is the difference, if there is any, from the formalism of a PU-learning problem under the SAR assumption? I.e. not under the Selected-Completely-At-Random assumption (SCAR). If the problem are mathematically equivalent, then how do generalization bound connect to existing ones, and how is the method related to existing methods PU-learning?\n2) How are hyperparameters like $\\alpha, \\beta$ are chosen? Can you also better clarify the pair loss and complete all gaps in notation that is not fullly defined?", "limitations": "The authors mention the in one line the potential limitation of pre-specifying the distance threshold in their method. I think this is a rather significant limitation as the authors do not offer any principled way to choose it. I also think there are other limitations, as discussed earlier in the review.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The method that combines representation alignment (via Wasserstein loss) with the point-wise and pair losses is novel as far as I know, and the empirical performance is better than the studied baselines. The empirical study is well scoped and gives informative ablations over several datasets.\n2. I appreciated the finite sample generalization bound given in the paper.\n\nWeaknesses:\n1. I found the paper quite difficult to follow. The counterfactual notation seems like an overly complicated way to formalize a problem that is formally equivalent to PU-learning under the Selected-At-Random assumption, see e.g. [1, 2]. Since the exposed labels are all positive, and the authors assume unconfoundedness in section 3.1, I am pretty sure that the problem turns out equivalent, but if it's not then I'd be happy to understand why. Also, it seems like there are papers studying recommendation problems under MNAR using causal notation, e.g. [3], so the first contribution in the authors' contribution list seems like a very small increment w.r.t to prior work.\n2. Considering the equivalence to PU-learning under SAR, there are several relevant works that should be discussed. For instance, how does the generalization bound given by the authors relate to those in the PU-learning literature under SAR? Like the one in [4] which uses distributional divergences that seem qualitatively different from IPMs and standard divergences like TV/KL etc, or the bounds given in Bekker et al [1]. Also, the algorithmic ideas around stratification seem very closely related to the propensity weighting ideas in [1] and other follow up work on these problems, or to methods in PU-learning that identify reliable negatives as a first stage (see e.g. section 5.1 in [2]).\n3. Beyond unclarity about the required notation and connection to prior work, some algorithmic choices are also weakly motivated. For instance the inclusion of pairwise loss seems like it could use motivation. The authors just say that it helps minimize the \"factual loss\" in their bound, but there is very little detail about it. For instance, I couldn't find the definition of $\\mathcal{D}_{+}$, I do not understand why is there a sigmoid on one of the terms in that loss, and generally I didn't understand why this ranking loss is used and not simply prediction of the assumed label. Finally, how are the hyerparameters $\\alpha, \\beta$ that the authors present in section 3.3 tuned in practice? and how is $D$ selected? Some hyper parameter selection is described in section 4, but I didn't understand if there are any additional hyper parameters that need to be tuned (like $\\alpha, \\beta$). If there are, then the authors should state how they are tuned, if there aren't any additional parameters then the authors should specify a least of the hyper parameters included in the method and remove the mentions of any redundant parameters from the paper. Fig. 3 shows that the choice of $D$ for example, has a large effect on performance, so this is a crucial point.\n4. The overall method is very complex and there are choice like using the RBF kernel to measure similarity, that may not transfer well to other types of datasets.\n\n\n[1] Jessa Bekker, Pieter Robberechts, and Jesse Davis. Beyond the selected completely at random\nassumption for learning from positive and unlabeled data. In Joint European conference on\nmachine learning and knowledge discovery in databases, pages 71–85. Springer, 2019.\n\n[2] Bekker, Jessa, and Jesse Davis. \"Learning from positive and unlabeled data: A survey.\" Machine Learning 109.4 (2020): 719-760.\n\n[3] Haoxuan Li, Chunyuan Zheng, and Peng Wu. Stabledr: Stabilized doubly robust learning for\nrecommendation on data missing not at random. In The Eleventh International Conference on\nLearning Representations, 2023.\n\n[4] Wald, Yoav, and Suchi Saria. \"Birds of an odd feather: guaranteed out-of-distribution (OOD) novel category detection.\" Uncertainty in Artificial Intelligence. PMLR, 2023.", "was_revised": true, "final_justification": "The authors responded in great length to comments raised in the review.\nWe seemed to arrive at some partial agreement, which is sufficient on my behalf to give a positive rating.", "rebuttal": "We appreciate the opportunity to clarify your concerns about our work.\n> W1&Q1 Relationship to PU-Learning\n1. **Setup: Shared Foundations with PU-Learning**\n- We acknowledge that our problem shares a foundational setup with PU-learning under the SAR assumption [1,2]. The **core challenge** of distinguishing \"true negatives\" from \"unobserved positives\" is **equivalent**.\n- However, the work in [3] (StableDR) focuses on explicit feedback (e.g., ratings), where \"missing data\" is less ambiguous. In contrast, our setting involves implicit feedback, where unobserved interactions (no click) are inherently ambiguous—they could stem from non-exposure or disinterest. This ambiguity makes **our problem more complex**.\n\n2. **Method: Divergences from Traditional PU-Learning**\n- First, our method diverges from PU-learning from the very beginning of **problem formulation**.\n    - Traditional PU-learning [1,2] treats unlabeled samples as a single \"unlabeled set\" and focuses on weighting or thresholding to identify reliable negatives.\n    - We introduce a causal formulation with \"exposure\" as a treatment variable, stratifying unlabeled samples into three groups (HE, HU, UN) based on estimated exposure confidence. This stratification explicitly models why an interaction is unobserved (high exposure → likely disinterest; low exposure → likely unobserved), which is absent in [1,2].\n- Second, our method has a completely novel **loss function** design.\n    - Traditional PU-learning methods such as [1,2] primarily use pointwise losses (e.g., cross-entropy) with reweighting to handle class imbalance.\n    - We combine pointwise losses for absolute relavance prediction and pairwise losses to model relative relevance, critical for recommendation ranking tasks. Additionally, we introduce IPM regularization which is  to align distributions across exposure states, addressing MNAR-induced shifts ignored in [1,2].\n3. **Uniqueness of Implicit Feedback in Recommendation Systems**\n- Recommendation systems aim to rank items by relevance, not just classify them. Thus, pairwise losses (modeling relative relevance between HE, UN, HU) are more aligned with task objectives than the classification-focused losses in [1,2].\n> W2&Q1 Comparing Generalization Bound of PU learning\n## Comparison with generalization bound in [4]\n- First, we have carefully read the two PU learning-related literatures cited by the reviewers and found that [4] focuses on novel class detection under distribution shift, with its generalization bound constraining the error rate of novel class detection on the target distribution.\n- Meanwhile, our work is dedicated to implicit feedback prediction in recommendation systems, and our generalization bound constrains the deviation between the estimated relevance and the true relevance on the whole population. \n- The target distribution of [4] is equivalent to the distribution of **unlabeled group** in our setting.   However, the error targets we consider includes **both labeled or unlabeled groups**, making the two generalization bounds quantitatively incomparable.\n## Comparison with Generalization Bound in [1]\n- The generalization bound in [1] is as follows:\n$$R(\\hat y_{\\hat R}\\mid y)\\leq \\hat R(\\hat y_{\\hat R}\\mid e,s)+\\sqrt{\\frac{\\delta^2_{\\max}\\ln \\frac{|\\mathcal{H}|}{\\eta}}{2n}}.$$\nAs a classical generalization bound for empirical risk minimization (ERM), it only depends on the size of the hypothesis space $|\\mathcal{H}|$ and the total sample size $n$, which measn that it includes all the error of MNAR and positive unlabelled machenism in the single term.\n- Meanwhile, our generalization bound explicitly model four types of error sources:\n1. Empirical risk for DP and HE groups,\n2. Distribution shift between exposed ($O=1$) and unexposed ($O=0$) groups with high confidence ($R=1$),\n3. Distribution shift between high confidence ($R=1$) and low confidence ($R=0$) groups,\n4. Irreducible variance of potential outcome in the exposed ($O=1$) and unexposed ($O=0$) groups.\n- Our generalization bounds provide a more detailed analysis of the sources of error, while [1] lumps all sources of error together and bounds them using the Hoeffding inequality.\n- Intuitively, in the following situations, our bound will have a more significant advantage compared to [1]:\n1. When the **MNAR problem is severe**, the IPM term in our bound directly quantifies the distribution difference between the exposed and unexposed groups, making it more suitable for the popularity bias scenarios commonly seen in recommendation systems.\n2. When the **proportion of unexposed samples is high**, our bound can sense this proportion by weighting the distribution shift of low-confidence samples (UN group), which is more in line with the cold start scenario in the recommendation system.\n3. When the variance of potential outcomes is small, the counterfactual prediction has high certainty, and the upper bound of our bound will be significantly reduced, which is more in line with the scenario in the recommendation system where users have similar strong preferences for specific items.\nIn addition, **empirically** we also compared our method with some PU learning methods under the SAR assumption on the Coat and Yahoo datasets. The following experimental results demonstrate the effectiveness of our method.\n## Bekker [1]\n\n|**Yahoo**| | | | |\n|-|-|-|-|-|\n|NDCG@3|0.448|0.457|0.456|0.462|0.451|\n|Recall@3|0.519|0.511|0.523|0.533|0.516|\n|MAP@3|0.383|0.398|0.392|0.397|0.387|\n|NDCG@5|0.523|0.538|0.533|0.535|0.526|\n|Recall@5|0.696|0.703|0.710|0.707|0.696|\n|MAP@5|0.434|0.454|0.445|0.449|0.440|\n|NDCG@8|0.599|0.610|0.606|0.607|0.600|\n|Recall@8|0.901|0.896|0.907|0.903|0.896|\n|MAP@8|0.481|0.497|0.489|0.491|0.484|\n|\n\n## PGPU\n|**Yahoo**|||||\n|-|-|-|-|-|\n|NDCG@3|0.484|0.470|0.487|0.477|0.483|\n|Recall@3|0.549|0.541|0.551|0.545|0.541|\n|MAP@3|0.420|0.404|0.424|0.413|0.421|\n|NDCG@5|0.551|0.544|0.558|0.547|0.551|\n|Recall@5|0.711|0.719|0.720|0.711|0.703|\n|MAP@5|0.469|0.456|0.475|0.463|0.471|\n|NDCG@8|0.622|0.609|0.624|0.618|0.622|\n|Recall@8|0.906|0.898|0.902|0.903|0.901|\n|MAP@8|0.511|0.496|0.515|0.505|0.512|\n|\n\n## PUe\n|**Yahoo**|||||\n|-|-|-|-|-|\n|NDCG@3|0.296|0.310|0.281|0.289|0.284|\n|Recall@3|0.357|0.369|0.332|0.345|0.341|\n|MAP@3|0.239|0.253|0.229|0.236|0.229|\n|NDCG@5|0.377|0.381|0.359|0.367|0.366|\n|Recall@5|0.546|0.537|0.521|0.530|0.533|\n|MAP@5|0.292|0.301|0.279|0.286|0.283|\n|NDCG@8|0.473|0.473|0.463|0.461|0.465|\n|Recall@8|0.812|0.795|0.809|0.787|0.810|\n|MAP@8|0.345|0.350|0.333|0.338|0.336|\n|\n\n> W3&Q2 Motivation of Algorithmic Choices\n\n- Inclusion of pairwise losses. We are motivated by the Bayesian Personalized Ranking (BPR) [5] from implicit feedback community. When the label for each sample is not exactly accurate, while we are more confident about the relationship between two samples, a pair-wise loss is suitable. The effectiveness of inclusion of pairwise losses can also be validated by the following ablation study.\n\n- Definition of D_+. We actually have included the definition in line 219 of the original manuscript. We use D_+ to represent the positive group in each of our three pairwise losses.\n\n- Sigmoid in Pair-wise Losses. We follow the convention of BPR loss [5] to use a sigmoid function to normalize the difference between pairs. Intuitively, if the sigmoid was omitted, some pairs with significant difference would dominate the pairwise losses. \n\n- Hyperparameters Tuning. It is easy to find that for a given dataset, there is a 1-1 correspondence between $\\alpha$ and $D$, and also for $\\beta$ and $D'$. Besides these two hyperparameters, there is also a kernel parameter $q$ and the loss weights $\\lambda_{point},\\lambda_{pair}$. The first three hyperparameters are tuned in the first stage (stratification), while the last two are tuned in the second stage (training), which will significantly reduce the searching space.\n\n> W4 RBF Kernel and Other Type of Datasets\n\nWe acknowledge that the RBF kernel in the paper is a straightforward and naive approach. However, in fact, our method can incorporate more efficient kernel estimation methods easily. We added the experiments using random Fourier feature and Nystrom approximation, which shows that the performance is retained while the runtime reduces to one-fourth of the original SVDD.\n\n||RFF|Nystroem|Ours|\n|-|-|-|-|\n||**Yahoo**|||\n|NDCG@3|0.578|0.571|0.562|\n|Recall@3|0.645|0.637|0.624|\n|MAP@3|0.513|0.508|0.499|\n|NDCG@5|0.638|0.634|0.625|\n|Recall@5|0.790|0.786|0.776|\n|MAP@5|0.560|0.555|0.547|\n|NDCG@8|0.693|0.685|0.681|\n|Recall@8|0.939|0.925|0.930|\n|MAP@8|0.595|0.587|0.582|\n||**Coat**|||\n|NDCG@3|0.361|0.389|0.368|\n|Recall@3|0.384|0.390|0.382|\n|MAP@3|0.282|0.314|0.296|\n|NDCG@5|0.387|0.427|0.414|\n|Recall@5|0.454|0.495|0.478|\n|MAP@5|0.309|0.347|0.332|\n|NDCG@8|0.468|0.486|0.473|\n|Recall@8|0.671|0.653|0.660|\n|MAP@8|0.360|0.384|0.369|\n|\n\n\nIn addition, based on this, we transfer to an image-type dataset and compared the breast cancer detection performance with some PU learning methods. The experimental results are as follows.\n\n\n|Metrics|Precision|Recall|Acc|\n|-|-|-|-|\n|Naive Resnet|0.4269|0.8438|0.5089|\n|Bekker+Resnet|0.4404|0.3751|0.5804|\n|Ours+Resnet|0.4608|0.5087|0.6793|\n\n[5] Rendle, S., et al. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09). \n\n[6] He, F., et al. \"Instance-Dependent PU Learning by Bayesian Optimal Relabeling. arXiv, Aug 2018.\" URL https://arxiv. org/abs (1808).\n\n[7] Wang, Xutao, et al. \"Pue: Biased positive-unlabeled learning enhancement by causal inference.\" Advances in Neural Information Processing Systems 36 (2023): 19783-19798."}]

Author rebuttal:

Meta-review: This paper formulate recommendation with implicit feedback data as a counterfactual estimation problem with missing treatments and proposes a novel estimation algorithm with theoretical guarantee. The paper received positive scores after the rebuttal period. While counterfactual formulation of implicit feedback is not necessarily novel, the proposed approach is indeed a fairly significant contribution and both theoretically and empirically sound. One potential weakness is the non-trivial computational cost, which thanks to a reviewer's suggestion also got significantly reduced. There were still some amount of disagreement with one reviewer on whether some of the contributions can be considered novel/significant in the context of PU-learning (which is unfortunately not something that i am super familiar with), but eventually the reviewer acknowledged that at least a partial agreement has been reached and raised the score to positive. Overall, I think this is a solid piece of work that I am happy to vote accept.
Decision: Accept (poster)