Title: Counterfactual Implicit Feedback Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Implicit  Feedback

Recommendation System

Counterfactual Estimation

Missing Treatment
Abstract: In recommendation systems, implicit feedback data can be automatically recorded and is more common than explicit feedback data. However, implicit feedback poses two challenges for relevance prediction, namely (a) positive-unlabeled (PU): negative feedback does not necessarily imply low relevance and (b) missing not at random (MNAR): items that are popular or frequently recommended tend to receive more clicks than other items, even if the user does not have a significant interest in them. Existing methods either overlook the MNAR issue or fail to account for the inherent mechanism of the PU issue. As a result, they may lead to inaccurate relevance predictions or inflated biases and variances. In this paper, we formulate the implicit feedback problem as a counterfactual estimation problem with missing treatment variables. Prediction of the relevance in implicit feedback is equivalent to answering the counterfactual question that ``whether a user would click a specific item if exposed to it?". To solve the counterfactual question, we propose the Counterfactual Implicit Feedback (Counter-IF) prediction approach that divides the user-item pairs into four disjoint groups, namely definitely positive (DP), highly exposed (HE), highly unexposed (HU), and unknown (UN) groups. Specifically, Counter-IF first performs missing treatment imputation with different confidence levels from raw implicit feedback, then estimates the counterfactual outcomes via causal representation learning that combines pointwise loss and pairwise loss based on the user-item pairs stratification. Theoretically the generalization bound of the learned model is derived. Extensive experiments are conducted on publicly available datasets to demonstrate the effectiveness of our approach. The code is available at https://github.com/zhouchuanCN/NeurIPS25-Counter-IF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of relevance prediction in recommendation systems using implicit feedback, where observed interactions (e.g., clicks) are sparse and may not fully reflect user preferences. Two key challenges are presented: the positive-unlabeled (PU) nature of the data—since unclicked items may be either irrelevant or unexposed to the user—and the missing-not-at-random (MNAR) bias, where exposure is influenced by item popularity or recommendation frequency.\n\nThe authors formalize the problem with causal language, where exposure is a binary treatment and the relevance is a potential outcome under positive treatment. Then to tackle the presented challenges, the authors propose a method called Counterfactual Implicit Feedback (Counter-IF). Counter-IF stratifies user-item pairs into four groups based on estimated exposure confidence: definitely positive (DP), highly exposed (HE), highly unexposed (HU), and unknown (UN). Treatment status for negative feedback is imputed using a feature-based confidence model.\n\nThe method combines pointwise and pairwise losses across strata, alongside representation learning with distribution alignment (via IPM regularization), to improve relevance estimation. Theoretical analysis provides a generalization bound under this framework, and experiments on real-world datasets demonstrate improved performance over existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method that combines representation alignment (via Wasserstein loss) with the point-wise and pair losses is novel as far as I know, and the empirical performance is better than the studied baselines. The empirical study is well scoped and gives informative ablations over several datasets.\n2. I appreciated the finite sample generalization bound given in the paper.\n\nWeaknesses:\n1. I found the paper quite difficult to follow. The counterfactual notation seems like an overly complicated way to formalize a problem that is formally equivalent to PU-learning under the Selected-At-Random assumption, see e.g. [1, 2]. Since the exposed labels are all positive, and the authors assume unconfoundedness in section 3.1, I am pretty sure that the problem turns out equivalent, but if it's not then I'd be happy to understand why. Also, it seems like there are papers studying recommendation problems under MNAR using causal notation, e.g. [3], so the first contribution in the authors' contribution list seems like a very small increment w.r.t to prior work.\n2. Considering the equivalence to PU-learning under SAR, there are several relevant works that should be discussed. For instance, how does the generalization bound given by the authors relate to those in the PU-learning literature under SAR? Like the one in [4] which uses distributional divergences that seem qualitatively different from IPMs and standard divergences like TV/KL etc, or the bounds given in Bekker et al [1]. Also, the algorithmic ideas around stratification seem very closely related to the propensity weighting ideas in [1] and other follow up work on these problems, or to methods in PU-learning that identify reliable negatives as a first stage (see e.g. section 5.1 in [2]).\n3. Beyond unclarity about the required notation and connection to prior work, some algorithmic choices are also weakly motivated. For instance the inclusion of pairwise loss seems like it could use motivation. The authors just say that it helps minimize the \"factual loss\" in their bound, but there is very little detail about it. For instance, I couldn't find the definition of $\\mathcal{D}_{+}$, I do not understand why is there a sigmoid on one of the terms in that loss, and generally I didn't understand why this ranking loss is used and not simply prediction of the assumed label. Finally, how are the hyerparameters $\\alpha, \\beta$ that the authors present in section 3.3 tuned in practice? and how is $D$ selected? Some hyper parameter selection is described in section 4, but I didn't understand if there are any additional hyper parameters that need to be tuned (like $\\alpha, \\beta$). If there are, then the authors should state how they are tuned, if there aren't any additional parameters then the authors should specify a least of the hyper parameters included in the method and remove the mentions of any redundant parameters from the paper. Fig. 3 shows that the choice of $D$ for example, has a large effect on performance, so this is a crucial point.\n4. The overall method is very complex and there are choice like using the RBF kernel to measure similarity, that may not transfer well to other types of datasets.\n\n\n[1] Jessa Bekker, Pieter Robberechts, and Jesse Davis. Beyond the selected completely at random\nassumption for learning from positive and unlabeled data. In Joint European conference on\nmachine learning and knowledge discovery in databases, pages 71–85. Springer, 2019.\n\n[2] Bekker, Jessa, and Jesse Davis. \"Learning from positive and unlabeled data: A survey.\" Machine Learning 109.4 (2020): 719-760.\n\n[3] Haoxuan Li, Chunyuan Zheng, and Peng Wu. Stabledr: Stabilized doubly robust learning for\nrecommendation on data missing not at random. In The Eleventh International Conference on\nLearning Representations, 2023.\n\n[4] Wald, Yoav, and Suchi Saria. \"Birds of an odd feather: guaranteed out-of-distribution (OOD) novel category detection.\" Uncertainty in Artificial Intelligence. PMLR, 2023.\n\nQUESTIONS:\n1) What is the difference, if there is any, from the formalism of a PU-learning problem under the SAR assumption? I.e. not under the Selected-Completely-At-Random assumption (SCAR). If the problem are mathematically equivalent, then how do generalization bound connect to existing ones, and how is the method related to existing methods PU-learning?\n2) How are hyperparameters like $\\alpha, \\beta$ are chosen? Can you also better clarify the pair loss and complete all gaps in notation that is not fullly defined?\n\nLIMITATIONS:\nThe authors mention the in one line the potential limitation of pre-specifying the distance threshold in their method. I think this is a rather significant limitation as the authors do not offer any principled way to choose it. I also think there are other limitations, as discussed earlier in the review.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe counterfactual implicit feedback model formulates the implicit feedback problem in recommender systems as a counterfactual estimation problem with missing treatment variables. Based on the implicit feedback positive and negative samples, user–item pairs are divided into four mutually exclusive groups, namely definitely positive (DP), highly exposed (HE), highly unexposed (HU), and unknown (UN) groups. Then, by stratifying user–item pairs and combining pointwise loss and pairwise loss, the model learns causal representations to estimate counterfactual outcomes. Extensive experiments on public datasets demonstrate the effectiveness of Counter-IF.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. From a new perspective, this work formalizes the relevance prediction under implicit feedback scenarios as a counterfactual outcome estimation problem with missing treatment variables.\n2. Proposes a novel sample stratification method combined with confidence treatment variable imputation to address the issue of missing treatment variables.\n3. Propose a causal representation learning method to predict the relevance between users and items across the set of all user-item pairs.\n4. Theoretically derives the generalization bound of the Counter-IF framework and proves that minimizing the proposed loss functions can effectively control this bound.\n5. Conducts extensive experiments on publicly available real-world datasets, demonstrating that the proposed Counter-IF method significantly outperforms existing state-of-the-art approaches.\n\nWeakness:\n1. Is there any hyperparameter analysis for the DP sample ratio hyperparameter  and the negative sample ratio hyperparameter  mentioned in Section 3.3?\n2. In Section 3.4, the sentence “In contrast, when the samples are not assigned the treatment (i.e., Ou,i = 0, corresponding to UN and HU groups” has a formatting issue, as it only includes half of the closing parenthesis.\n3. Figure 2 is presented in the paper, but the text does not explain which part of the content it is associated with.\n4. Since the paper adopts a counterfactual approach, it is recommended to include baseline comparisons with existing counterfactual recommendation methods in the experimental section.\n5. The theoretical analysis of the generalization bound is relatively abstract, it is suggested to illustrate the terms in the bound with practical examples.\n6. The paper does not specify which parts of the main text correspond to Appendices A, B, C, and D, it is recommended to clarify this relationship.\n7. It is suggested to add visualizations (e.g., t-SNE plots) of the distributions of the DP, HE, HU, and UN groups in the experiments to verify the reasonableness of the stratification.\n\nQUESTIONS:\n1. Since SVDD is essentially based on geometric distances in the feature space to perform grouping, is this approach suitable for cold-start users or items?\n2. In Section 2, the paper mentions “We divide all user-item pairs according to Yu,i, i.e.,” and in Section 3.4 it mentions “where and are the set of positive samples and negative samples.” Are the sets of positive and negative samples referred to in these two places consistent?\n3. Since Counter-IF heavily relies on the accuracy of treatment imputation, would inaccuracies or biases in the imputation process have any impact on the subsequent results?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenges of PU and MNAR of learning from implicit feedback in recommender systems. The authors propose to formalize this problem as one of counterfactual estimation, with the goal to predict whether a user would click an item if they were exposed to it. Specifically, the proposed method will firstly use a support vector-based method to stratify user-item pairs into four groups visa missing treatment imputation. Then it employs a causal representation learning framework with a hybrid loss function to combine pointwise loss for samples with observed/imputed outcomes and pairwise loss for samples with unobserved outcomes. Experiments are conducted on several public datasets, showing improved performance over existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea of stratifying negative samples based on a confidence measure of exposure is a clear improvement. This allows the model to differentiate between various reasons for non-interaction, moving beyond the simplistic assumption that all unobserved items are equivalent.\n\n2. The experimental results demonstrate the proposed method improve the ranking performance in recommendation systems. \n\n3. The paper's primary strength is its insightful reframing of the implicit feedback problem as a counterfactual estimation task. \n\nWeaknesses:\n1. The paper relies on the unconfoundedness assumption (Y_u,i(1) ⊥ O_u,i | X_u,i). This assumption is very strong and highly likely to be violated in real-world recommender systems, where the exposure mechanism (the previous model) is a powerful, partially unobserved confounder. The paper fails to provide any discussion of the potential impact of violating this assumption or any sensitivity analysis to demonstrate the model's robustness to unobserved confounding.\n\n2. The two-stage process, requiring a potentially O(N^2) kernel-based SVDD step, is not scalable to large, industrial datasets. The paper misses discussion of computational complexity or strategies for making the approach practical.\n\n3. The framework introduces a large number of sensitive hyperparameters (α, β for stratification, kernel parameters q, loss weights λ_point, λ_pair). This complexity presents a significant tuning challenge, potentially limiting the model's practical utility and ease of deployment.\n\nQUESTIONS:\n1. Given that unconfoundedness is a strong assumption likely violated in real-world recommender systems, could you provide further justification for its use? More importantly, how robust is your method to potential violations from unobserved confounders, which are common in practice?\n\n2. The SVDD-based stratification appears to be a computational bottleneck, especially with kernel methods. Could you please discuss the time complexity of this step and outline a practical strategy for scaling your approach to industrial-sized datasets?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Counterfactual Implicit Feedback (Counter-IF), a novel approach for modeling implicit feedback in recommender systems by framing relevance prediction as a counterfactual estimation problem with missing treatments. Traditional implicit feedback methods suffer from two main issues: (1) the positive-unlabeled (PU) problem, where non-clicks may be due to non-exposure rather than disinterest, and (2) missing-not-at-random (MNAR) biases, where popular items are more likely to be clicked simply due to increased exposure. Counter-IF addresses these challenges by stratifying user-item pairs into four groups, definitely positive, highly exposed, highly unexposed, and unknown, using a novel confidence-based treatment imputation method. The model then estimates counterfactual outcomes using a causal representation learning framework that integrates pointwise and pairwise losses across strata, and regularizes with Integral Probability Metrics (IPM) to align distributions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- To my best knowledge, this paper proposes a novel problem formulation. It is the first to formulate relevance prediction under implicit feedback as a counterfactual outcome estimation problem with missing treatments, offering a fresh causal perspective on a longstanding challenge in recommendation systems.\n\n- The paper introduces an exposure estimation method based on hypersphere distance using kernel methods, enabling imputation only for high-confidence samples and reducing risk from incorrect imputations.\n\n- The experimental settings are clear, and experiments on public datasets show consistent and statistically significant improvements over competitive baselines across multiple metrics.\n\n- The paper includes ablation studies, sensitivity analyses, and hyperparameter tuning, providing clear insight into which components contribute most to performance and demonstrating model robustness.\n\nOverall, the problem formulation is interesting to me, and the paper presentation is easy to follow.\n\n\nWeaknesses:\n\n- One limitation of the paper lies in the treatment imputation step, which relies on a kernel-based hypersphere model using the Gaussian kernel. While this approach provides a principled way to estimate exposure confidence and stratify negative samples, it introduces significant computational overhead due to the quadratic complexity in the number of positive samples. Specifically, the need to compute and store the full kernel matrix and solve a constrained optimization problem limits the scalability of the method to large-scale recommendation scenarios.\n\n- Suggestion: To improve scalability, the kernel-based imputation can be replaced or approximated using random Fourier features, Nystrom approximation, or deep neural representations. These methods significantly reduce computational complexity while preserving the core idea of stratifying user-item pairs by exposure confidence.\n\nQUESTIONS:\nAs mentioned in the Strengths And Weaknesses,  treatment imputation step relies on a kernel-based hypersphere model using the Gaussian kernel, which introduces significant computational overhead due to the quadratic complexity in the number of positive samples. \n\nHave the authors considered using more scalable alternatives, such as random Fourier features, Nystrom approximation, or neural embedding-based methods, to reduce the computational cost while retaining the benefits of exposure confidence estimation? Is it possible to include this extension?\n\nLIMITATIONS:\nYes, the author(s) mentioned the limitation of pre-specifying the distance threshold. They could consider discussing one more limitation regarding scalability to large-scale scenarios, as mentioned in the \"Strengths and Weaknesses\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}