Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a loss function to mitigate the popularity bias in recommendation. It argues that existing debiasing methods poses two limitations: i) focusing on the trade-off between evaluations for popular and unpopular users/items (e.g. re-weighting popular and unpopular samples), ii) holding infeasible assumptions (e.g. accessibility to unbiased data). The proposed method can be divided into two parts: i) Evaluating the bias degree of each user-item pair, using the prediction error of a popularity-based interaction prediction network. ii) The proposed BC Loss, which is basically the softmax cross-entropy loss with the bias-specific margin based on the bias degree. The paper also gives geometric illustrations and some theoretical analysis on the strength of the proposed BC Loss.
strengths_and_weaknesses: Strength:
1.	The proposed model is a simple plugin suitable for various kind of recommendation methods.
2.	According to the experimental evaluations, the proposed BC loss indeed results in overall performance improvements for both popular and unpopular items.
Weakness:
1.	The technical contribution of this paper is limited. Comparing to existing CF methods, it only proposes to employ an extra popularity-based predictor and combine the results with an existing CF model.
2.	The paper overclaims the strength of the proposed BC loss in theoretical analysis. The geometric interpretability, theorem 1, the high/low entropy representations, and the hard-negative mining ability, are actually the same thing (i.e., applying stronger constrains for samples with higher popularity) from different viewpoints.

questions: The essence of the proposed BC loss is to add popularity-related bias to the prediction scores of normal recommenders, which share the same debiasing paradigm with some existing debiasing works. One difference is that this paper measure the popularity bias using radius, which yields intuitive geometric interpretation.
The idea of employing an extra predictor based on popularity features to extract popularity degree is novel and promising. But the technical solution of this paper is rather simple, and the theoretical analysis does not give deep enough investigation on the re. 

To evaluate the bias degree, if the popularity-based predictor achieves strong performance for a user-item pair, this pair is strongly influenced by the popularity.
limitations: Please refer to the identified weakness.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a simple-yet-effective loss, called Bias-aware margins into Contrastive (BC) loss, to mitigate the popularity bias. Specifically, it introduces a popularity bias extract to estimate the popularity bias scores from a pair of user and item popularity. Then it incorporates it into the contrastive loss as the marginal information. Despite the simplicity of the proposed loss, experimental results show that the BC loss can be effective for tail/head evaluation and non-debiasing methods. Overall, the idea of using BC loss is interesting to me, and it also shows the theoretical analysis.
strengths_and_weaknesses: Strengths
- (Originality) The idea of using the popularity bias extractor is quite interesting.
- (Quality) The proposed loss outperforms existing methods in various evaluation protocols.
- (Clarity) It is well-written and easy to understand. Also, the theoretical proof is quite solid.


Weaknesses
- (Originality) Although the idea of incorporating the popularity bias into the contrastive loss is interesting, some recent studies also tried similar ideas. It is necessary that the proposed loss is compared with the following references.

[1] Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, Xiuqiang He, SimpleX: A Simple and Strong Baseline for Collaborative Filtering, CIKM 2021

[2] Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, Tianyu Qiu, Xiangnan He, On the Effectiveness of Sampled Softmax Loss for Item Recommendation, 2022

- (Quality) Although the proposed model is compared with various debasing strategies, the models using inverse propensity scores (IPS) are not compared. It would be better to show that the proposed model is still better than some existing studies using IPS.
- (Quality) Although the proposed model shows the results on the KuaiRec dataset, most existing studies show two datasets for unbiased evaluation, i.e., Coat and Yahoo! R3. For a fair comparison, it would be better to show additional experimental results on these datasets.

questions: - Q1) It is wondering whether the popularity bias effectively captures the popularity bias. Could you show some empirical or quantitative results?
- Q2) The source code does not work well. It may be related to pre-processing. Please check your code.

limitations: This paper does not address the negative societal impact. However, this paper seems not to have any negative impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work tries to reduce the negative impact of popularity bias on CF models. The paper incorporates Bias-aware margins into Contrastive loss and proposes a simple yet effective BC Loss, where the margin tailors quantitatively to the bias degree of each user-item interaction. Extensive experiments on six datasets demonstrate the effectiveness of the proposed algorithm.
strengths_and_weaknesses: For the strengths, following the idea of MACR, this paper also designs a bias extractor and aims to remove the popularity bias in practice. Interestingly, this work explored the debias problem from a geometric view and unified the bias term into a marginal contrastive learning framework elegantly. The geometric interpretation and theoretical analysis also show the good properties of the proposed BC loss. Extensive experiments on six datasets with various settings verify the effectiveness of BC loss on both debiasing and general recommendation. Overall, I agree this is a good paper with clear novelty.

Meanwhile, I also have some questions. For the Section Popularity Bias Extractor, I wonder if it'll lose part of useful bias information if just inputting the user/item popularity score. Have you tried other inputs? It's also unclear what kinds of interactions will be learned well according to equation (4). It'll be better if you can provide some visualizations of the learned angles of different types of interactions in the experiments. For the experiments, I'd like to know if the balanced test setting uses an imbalanced or balanced validation set. Though current experiments demonstrate the ability of BC loss on debiasing. however, to verify its effectiveness as a standard loss, more comparisons with some general loss functions are required.
questions: Please refer to my questions above.
limitations: The limitations are discussed in the Section 7. I’m not aware of any potential negative societal impact.
review_title: null

Meta-review: metareview: This paper studies the popularity bias of collaborative filtering-based recommendation systems. Specifically, this paper proposes a bias-aware margin into the contrastive loss, resulting in a modified BC loss to remedy the problem. Geometric interpretation and experimental results are provided to validate the effectiveness of the proposed method. This paper received mixed review comments. A review raises concerns about the technical novelty and contribution of the proposed method mainly due to the simplicity of the proposed solution. Considering that the simple solution is well supported by theoretical justification and empirical results, I lean to accept this paper.

confidence: Less certain
Decision: Accept