Incorporating Bias-aware Margins into Contrastive Loss for Collaborative Filtering
Authors: An Zhang
Wenchang Ma
Xiang Wang
Tat-Seng Chua
Venue: NeurIPS 2022 Accept (2022)
Keywords: Recommendation
Collaborative Filtering
Popularity Bias
Contrastive Loss
Popularity Debiasing

Abstract: Collaborative ﬁltering (CF) models easily suffer from popularity bias, which makes recommendation deviate from users’ actual preferences. However, most current debiasing strategies are prone to playing a trade-off game between head and tail performance, thus inevitably degrading the overall recommendation accuracy. To reduce the negative impact of popularity bias on CF models, we incorporate Bias-aware margins into Contrastive loss and propose a simple yet effective BC Loss, where the margin tailors quantitatively to the bias degree of each user-item interaction. We investigate the geometric interpretation of BC loss, then further visualize and theoretically prove that it simultaneously learns better head and tail representations by encouraging the compactness of similar users/items and enlarging the dispersion of dissimilar users/items. Over six benchmark datasets, we use BC loss to optimize two high-performing CF models. In various evaluation settings (i.e., imbalanced/balanced, temporal split, fully-observed unbiased, tail/head test evaluations), BC loss outperforms the state-of-the-art debiasing and non-debiasing methods with remarkable improvements. Considering the theoretical guarantee and empirical success of BC loss, we advocate using it not just as a debiasing strategy, but also as a standard loss in recommender models. Codes are available at https://github.com/anzhang314/BC-Loss.

Incorporating Bias-aware Margins into Contrastive Loss for Collaborative Filtering
An Zhangft Wenchang Ma Xiang Wang Tat-Seng Chua
Sea-NExT Joint Lab
National University of Singapore
University of Science and Technology of China
anzhang@u. nus edu, e07242900u. nus edu, xiangwang12230gman com dcscts@nus.edus edu.sg
Abstract
Collaborative filtering (CF) models easily suffer from popularity bias, which makes recommendation deviate from users' actual preferences. However, most current debiasing strategies are prone to playing trade-off game between head and tail performance, thus inevitably degrading the overall recommendation accuracy. To reduce the negative impact of popularity bias CF models, we incorporate Bias- aware margins into Contrastive loss and propose a simple yet effective BC Loss, where the margin tailors quantitatively to the bias degree of each user-item interac- tion. We investigate the geometric interpretation of BC loss, then further visualize and theoretically prove that it simultaneously learns better head and tail represen- tations by encouraging the compactness similar users/items and enlarging the dispersion of dissimilar users/items. Over eight benchmark datasets, loss to optimize two high-performing CF models. On various evaluation settings (i.e., imbalanced/balancee temporal split, fully-observed unbiased, tail/head test evaluations), BC loss outperforms the state-of-the-art debiasing and non-debiasing methods with remarkable improvements. Considering the theoretical guarantee and empirical success of BC loss, we advocate using it not just as debiasing strategy, but also standard loss in recommender models. Codes are available https: htt///ggttttt github....... com http//nnaannn1..10 4
BC
Introduction
At the core of leading collaborative filtering (CF) models is the learning of high-quality representations of users and items from historical interactions. However, most CF models easily suffer from the popularity bias issue in the interaction data [1, 2, 3, 4]. Specifically, the training data distribution is typically long-tailed, e.g., few head items occupy most of the interactions, whereas the majority of tail items are unpopular and receive little attention. The CF models built upon the imbalanced data prone learn the popularity bias and even amplify by over-recommending head items and underr-rcoommndinn tail items. As a result, the popularity bias causes the biased representations with poor generalization ability, making recommendations deviate from users' actual preferences.
Motivated by concerns popularity bias, studies on debiasing have been conducted to lift the tail performance. Unfortunately, most prevalent debiasing strategies focus the trade-off between head and tail evaluations (see Table 3), including post-processing re-ranking [5, 6, 7, 8, 9], balanced training loss [10, 11, 12, 9], sample re-weighting [13, 14, 15. 16, 17, 18), and head bias removal by causal inference [19, 20, 21, 22]. Worse still, many of them hold some assumptions that are
Xiang Wang is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) BPR loss (head)
(b) Softmax loss (head)
(c) IPS-CN [15] (head)
(d) BC loss (head)
(e) BPR loss (tail)
(f) Softmax loss (tail)
(g) IPS-CN [15] (tail)
(h) BC loss (tail)
Figure 1: Visualizations of item representations learned by LightGCN [25] on Yelp2018 [25], where subfigures (a-d)((e-h) depict the identical head/tail user as a green star, while the red and blue points denote positive and negative items, respectively. In each subfigure, the first row presents the 3D item representations projected on the unit sphere, while the second row shows the angle distribution of items w.r.f. the specific user and the statistics of mean angles. Compared other losses, BC loss learns better head representations (cf. with the smallest mean positive angle, the vast majority of positive items fall into the group closest to the user) and tail representations (cf. a clear margin exists between positive and negative items for the tail user). BC loss learns a more reasonable representation distribution that is locally clustered and globally separated. See more details in Appendix A.
infeasible in practice, such as the balanced test distribution is known in advance to guide the hyperparameters' adjustment [23, 22], or a small unbiased data is present to train the unbiased model [24, 19]. Consequently, they pursue improvements on tail items but exacerbate the performance sacrifice of head items, leading to severe overall performance drop. The trade-off between the head and tail evaluations results in suboptimal representations, which derails the generalization ability.
In this paper, we conjecture that an ideal debiasing strategy should learn high-quality head and tail representations with powerful discrimination and generalization abilities, rather than playing a trade- off game between the head and tail performance. Here we follow the prior studies [15, 23, 13, 14] to focus on one key ingredient in representation learning: the loss function. Figure/ depicts the item representations, which is optimized via two non non-debiasing losses (BPR [26] and Softmax [27]) and one debiasing loss (IPS-CN [15]). Wherein, representation discrimination is reflected in how well the positive items of user are apart from the negatives. Our insights are: (1) For user, the non-debiasing losses are inadequate to discriminate his/her positive and negative items well, since their representations are largely overlapped as Figures 4a and 4b show; (2) Although IPS-CN achieves better discrimination power in the tail group than BPR (cf. positive items get smaller angles to the ego user in Figure 4g, as compared to Figure 4c), it gets worse discrimination ability in the head (cf. positive items hold larger angles to the ego user in Figure 4c, as compared to Figure 4a).
Towards this end, we incorporate Bias-aware margins into Contrastive Loss and devise simple yet effective BC Loss to guide the head and tail representation learning of CF models. Specifically, we first employ a bias degree extractor to quantify the influence of interaction-wise popularity that is, how well an interaction is predicted, when only popularity information of the target user and item is used. Interactions involving inactive users and unpopular items often align with lower bias degrees, indicating that popularity fails to reflect user preference faithfully. In contrast, interactions with active users and popular items are spurred by the popularity information, thus casily inclining to
2

---

high bias degrees. We then move on to train the CF model by converting the bias degrees into the angular margins between user and item representations. the bias degree is low, we impose larger margin to strongly squeeze the tightness of representations. In contrast, if the bias degree is large, we exert small or vanishing margin to reduce the influences of biased representations. Through this way, for each ego user's 's representation, BC quantitatively controls its bias-aware margins with item representations- - adaptively intensifying the representation similarity among positive items, while diluting that among negative items. Benefiting from stringent and discriminative representations, BC loss significantly improves both head and tail performance.
Furthermore, BC loss has three desirable advantages. First, it has clear geometric interpretation, as illustrated in Figure Second, it brings forth a simple but effective mechanism of hard example mining (See Appendix A.2). Third, we theoretically reveal that BC loss tends to learn low-entropy cluster for positive pairs (e.g., compactness of matched users and items) and a high-entropy space for negative pairs (e.g,, dispersion of unmatched users and items) (See Theorem 1). Considering the theoretical guarantee and empirical effectiveness, we argue that BC loss is not only promising to alleviate popularity bias, but also suitable as a standard learning strategy in CF.
Preliminary of Collaborative Filtering (CF)
Task Formulation. Personalized recommendation is retrieving a subset of items from a large catalog to match user preference. Here we consider a typical scenario, collaborative filtering (CF) with implicit feedback [28], which can be framed as a top-N recommendation problem. Let O+ {(u, ()lve = 1} be the historical interactions between users V/ and items I, where yuu indicates that user E l/ has adopted item i I before. Our goal is to optimize a CF model ÿ U x - IR that latches on user preference towards items.
Modeling Scheme. Scrutinizing leading CF models [26, 25, 29, 30], we systematize the common paradigm as a combination of three modules: user encoder (+). item encoder Φ(-), and similarity function s-)). Formally, we depict one CF model as 5(u, i) = ((b(),, d6()). where & : U + Bd and o I Rd encode the identity (ID) information of user and item into d-dimensional representations, respectively; s :: Rd x Rd + R measures the similarity between user and item representations. In literature, there are various choices of encoders and similarity functions:
Common encoders roughly fall into three groups: ID-based (e.g. MF [26, 29], NMF [31], CMN [32]), history-based -based (e.g.. SVD++ [29], FISM [33], MultVAE [30]), and graph based (e.g., GCMC [34], PinSage [35], LightGCN [25]) fashions. Here we select two high performiffo encoders, MF
and LightGCN, as the backbone models being optimized.
The uidely-used similarity functions include dot product [26], cosine similarity [36], and neural networks [31]. As suggested in the recent study [36], cosine similarity is a simple yet effective and efficient similarity function in CF models, having achieved strong performance. For better interpretation, we take geometric view and denote by:
(u) 6(1) = cos (Bus). [v("]] [o(i)]
(V(())
in which ous is the angle between the user representation (u) and item representation ((1).
Learning Strategy. To optimize the model parameters, CF models mostly frame the top-N rec- ommendation problem into supervised learning task, and resort to one of three classical learning strategies: pointwise loss (e.g., binary cross- cross-entropy [37], mcan square error [29]), pairwise loss (e.g., BPR [26], WARP [38]), and softmax loss [28]. Among them, pointwise and pairwise losses are long-standing and widely-
objective functions in CF. However, extensive studies [9, 1, 39] have analytically and empirically confirmed that using pointwise pairwise loss is prune to propagate more information towards the head user-item pairs, which amplifies popularity bias.
Softmax loss is much less explored in CF than its application in other domains like CV [40, 41]. Recent studies [36, 42, 43, 44, 45] find that it inherently conducts hard example mining over multiple negatives and aligns well with the ranking metric, thus attracting a surge of interest in recommendation.
3

---

Hence, we cast the minimization of softmax loss [27] as the representative learning strategy:
exp (cos(fus)/T)
Σ log exp (cos((0.))/7) + ΣjEx. exp (cos(0ug)/T)
Lo =-
(uN)EO-
where (u, O+ is observed interaction of user u, while NN {ilMuy = 0} is the set of sampled unobserved items that u did not interact with before; T is the hyper-parameter known as the temperature in softmax [46]. Nonetheless, modifying softmax loss to enhance the discriminative power of representations and alleviate the popularity bias remains largely unexplored. Therefore, work aims to devise more generic and broadly-applicable variant of softmax loss for CF tasks,
which can improve the long-tail performance fundamentally.
3 Methodology of BC Loss
On the basis of softmax loss, we devise our BC loss and present its desirable characteristics.
3.1 Popularity Bias Extractor
Before mitigating popularity bias, need to quantify the influence of popularity bias a single user-item pair. One straightforward solution is to compare the performance difference between the biased and unbiased evaluations. However, this is not feasible the unbiased data is usually unavailable in practice. Statistical metrics of popularity could be a reasonable proxy of the biased information, such as user popularity statistics pv, E IP (i.e., the number of historical items that user has interacted with before) and item popularity statistics interactions that item is involved in). If the impact of the interaction between and i can be captured well based solely on such statistics, the model is susceptible to exploiting popularity bias for prediction. Hence, we argue that the popularity-onle prediction will delineate the influence of bias.
E IP (i.e., the number of observed
Towards this end, we first train an additional module, termed popularity bias extractor, which only takes the popularity statistics input to make prediction. Similar to the modeling of CF (ef. Section 2), the bias extractor is formulated as a function is IP x P R:
(Pa. Pt) = s(i (Pu), (p:)) cos((us).
(3)
where the user popularity encoder th : IP - Rd and the item popularity encoder di IP + Rd map the popularity statistics of user u and item into d-dimensional popularity embeddings vh (p.) and (p:), respectively; 8 : IRd x Rd - R is the cosine similarity function between popularity embeddings (ef. Equation (1)). Eui is the angle between tho (Pa) and (p.)..
We then minimize the following softmax loss to optimize the popularity bias extractor:
exp (cos(É.:)/T)
Lo Σ log
+ EjEN. (c)(
exp (cos(Eug)/T)
(N.1)EO-
This optimization enforces the extractor to reconstruct the historical interactions using only biased information (i.e., popularity statistics) and makes the reconstruction reflect the interaction-win wise bias degree. As shown in Appendix B.5, interactions with active users and popular items are inclining to learn well via Equation (4). Furthermore, we can distinguish hard interactions based on the bias degree, i.e., the interactions that can be hardly predicted by popularity statistics ought to be more informative for representation learning in the target CF model. nutshell, the popularity bias extractor underscores the bias degree of each user-item interaction, which substantively reflects how hard it is to be predicted.
3.2
BC Loss
We move on to devise a new BC loss for the target CF model. Our BC loss stems from softmax loss but converts the interaction- bias degrees into the bias-aware angular margins among the representations to enhance the discriminative power of representations. Our BC loss is:
exp (Q + Mus)/T)
LBC Σ
log exp (cos(Our + Ma.)/T) EJEN, exp (cos(Ou))/TT
(5)

---

k
k
k
(a) Softmax Loss (2D)
(b) BC Loss (2D)
(c) Softmax Loss (3D)
(d) BC Loss (3D)
Figure 2: Geometric Interpretation of softmax loss and BC loss in 2D and 3D hypersphere. The dark red region indicates the discriminative user constraint, while the light red region is for comparison.
where Mus is the bias-aware angular margin for the interaction (u, i) defined as:
Mut min π Ous
where Éus is derived from the popularity bias extractor (cf. Equation (3)), and - O.i is the upper bound to restrict cos(- + Mus) to be monotonically decreasing function. Intuitively, if user-item pair (u, i) is the hard interaction that hardly be reconstructed by its popularity statistics, it holds high value of Eus and leads to high value of Mus; henceforward, BC loss imposes the large angular margin Mui between the negative item j and positive item and optimizes the representations of user and item i to lower E... See more details and analyses in Section 4.
is noted that BC loss is extremely easy to implement in recommendation tasks, which only needs to revise several lines of code. Moreover, compared with softmax loss, BC loss only adds negligible computational complexity during training (ef. Table 5) but achieves more diseriminative representations. Hence, we recommend to use BC loss not only as a debiasing strategy to alleviate the popularity bias, but also as standard loss in recommender models enhance the discriminative power. Note that the modeling of M.i is worth exploring, such as the more complex version Mn. min{ A Eus, 0.i} where À controls the strength of the bias-margin. Meanwhile, carefully designing a monotonically decreasing function helps to get rid of the upper bound restriction. We will leave the exploration of bias-margin in future work.
4 Analyses of BC Loss
We analyze desirable characteristics of BC loss. Specifically, we start by presenting its geometric inter- pretation, and then show its theoretical properties w.r.f. compactness and dispersion of representations. The hard mining mechanism of BC loss is discussed in Appendix A.2.
4.1 Geometric Interpretation
Here we probe into the ranking criteria of softmax loss and BC loss, from the geometric perspective. To simplify the geometric interpretation, we analyze one user u with one observed item i and only unobserved items and k: Then the posterior probabilities obtained by softmax loss
exp (xpp((((0)..)r
exp (co)((_../r))4exp (/))/))++ (cos(fux)/r)) During training, softmax loss encourages the ranking
criteria ous < Ouy and 0ut < 6nk to model the basic assumption that the observed interaction (u, i) indicates more positive cues of user preference than the unobserved interactions (u.j) and (u, k).
Intuitively, to make the ranking criteria more stringent, we can impose an angular margin Mar on and establish a new criteria Ō.,- + Mns O.j and O.. + Mai < Ō.. Directly formulating this idea exp (cos((.,.ttM)))
arrives at the posterior probabilities of BC loss:
exp (cos[d.. Mu:lr) (0)r(000))))))er (P) Obviously, BC loss is more rigorous about the ranking assumption compared with softmax loss. See Appendix A.2 for more detailed explanations.
We then depict the geometric interpretation and comparison of softmax loss and BC loss in Figure 2. Assume the learned representations of i, j. and are given, and softmax and BC losses are optimized to the same value. In softmax loss, the constraint boundaries for correctly ranking user u's preference
5

---

are fus Ous and ous = uux: whereas, in BC loss, the constraint boundaries are dus + Mui = Ouj and o.. + Mui Ous- Geometrically, from softmax loss (cf. Figure 2c) to BC loss (cf. Figure 2d), it is a more stringent circle-like region on the unit sphere in the 3D case. Further enlarging the margin M.. will lead to smaller hyperspherical cap, which an explicit discriminative constraint on manifold. As result, limited constraint regions squeeze the tightness of similar items and encourages the separation of dissimilar items. Moreover, with the iccrasse of representation dimension, BC loss has more restricted learning requirements, exponentially decreasing the area of constraint regions for correct ranking, and becomes progressively powerful learn discriminative representations.
4.2 Theoretical Properties
BC loss improves head and tail representation learning by enforcing the compactness of matched users and items, while imposing the dispersion of unmatched users and items. See detailed proof in Appendix A.3.
Theorem 1. Let Vu vb().. V; = =(1), and c. = [Pul Eiep. Vi C2 = [P EuEp, Vu, where Pu {ilyus = 1} and No { yus = 0} are the sets of user u's positive and negative items, respectively: P: = (ulyus = 1} is the set of item i's positive users. Assuming the representations of users and items normalized, the minimization of BC loss is equivalent minimizing compactness part and a dispersion part simultaneously:
LBC 2
Σ NEW
vv cull 112+
Σ VET
1I,
c,ll=
ΣΣ Iv vyll H(VY)) HEUSEN
H(V). (7)
Compactesss
Dispersion
Compactesss part
Dispersion part
Discussion, Cu the averaged representations of all items that u has interacted with, which describes w's interest; similarly, c; profiles item i's user group. For the compactness part, BC loss forces the user's positive items to be user-centric and vice versa. From the entropy perspective, compactness part tends to learn low-entropy cluster for positive interactions, i.e., high compactness for similar users and items. For the dispersion part, for users and items from unobserved interactions, BC loss maximizes the pairwise euclidean distance between their representations and encourages them to be distant from each other; Hence, from the entropy viewpoint, dispersion part levers the spread of representations learn high-entropy representation space, i.e., large separation degree for dissimilar users and items.
Experiments
We aim to answer the following research questions:
RQI: How does BC Loss perform compared with debiasing strategies in various evaluations?
RQ2: Does BC loss cause the trade-off between head and tail performance?
RQ3: What are the impacts of the components (e.g., temperature, margin) on BC Loss?
Baselines & Datasets. SOTA debiasing strategies in various research lines are compared: sample re-weighting (IPS-CN [15]), bias removal by causal inference (MACR [22], CausE: [19]), and regularization-basee framework (sam+reg [12]). Extensive experiments are conducted on eight real- world benchmark datasets: Tencent [47], Amazon-Book [48], Alibaba-iFashion [49], Yelp2018 [25]. Douban Movie [50], Yahoo!R3 [5]], Coat [13] and KuaiRec [52]. For comprehensive comparisons, almost all standard test distributions in CF are covered in the experiments: balanced test set [22, 23, 24], randomly selected imbalanced test set [10, 53], temporal split test set [20, 21, 12], and unbiased test set [13, 52, 51]. See more experiments on KuaiRec, Yahoo!R3 and Coat for unbiased test evaluation in Appendix B.3 and more comparison results between BC loss and other standard losses (most widely used BPR [26], newest proposed CCL [54] and SSM [55]) in Appendix B.4.
5.1 Performance Comparison (RQ1)
5.1.1 Evaluations on Imbalanced and Balanced Test Sets
Motivation. Many prevalent debiasing methods assume that test distribution is known advance [22, 23, 10], iee. the validation set has similar distribution with the test set. Moreover, only an

---

Table 1: Overall debiasing performance comparison in balanced and imbalanced test sets.
Tencent
Amazon-book
Alibaha-IFashio
Balanced
Imbulanced
Balanced
Imbulanced
Balanced
Imbulanced
Rocall
NDCG
Rocall
NDCG
Recall
NDCG
Recall
NDOG
Recall
NDCG
Recall
NDCG
MostPop
0.0002
0.0002
0.0384
0.0208
0.0001
0.0001
0.0102
0.0063
0.0003
0.0001
0.0212
0.0084
MF
0,0052
0.0040
0.0982
0.0643
0.0109
0.0103
0.0656
0.0638
0.0056
0.0028
0,0843
0.0441
+ IPS-CN
0,0075
0.0058
0.0686
0.0421
0.0152
0.0123
0.0765
0.0554
0.0050
0.0027
0.0551
0.0255
+ CausE
0,0086
0.0043
0,0687
0.0468
0,0115
0.0105
0,0720
0.0551
0,0005
0.0003
0.0185
0.0086
samttigg
0,0070
0.0054
0.0406
0.0266
0.0141
0.0132
0.0143
0.0067
0.0032
0.0305
0.0146
+ MACR
0,0067
0.0046
0.0326
0.0241
0.0181
0.0146
0.0292
0.0229
0.0086
0.0041
0.0650
0.0331
BC Loss
0.0087
0.0068
0.1298
0.0904"
0.0221
0.0202
0.1198
0.09488
0.0095
0.00488
0.0967
0
Tmp:
16.0%
17.2%
32.2%
40.1%
22.1%
38.4%
40.0%
49.6%
10.5%
17.1%
14.7%
18.5%
LightGCN
0,0055
0.0042
0,1065
0.0712
0.0123
0.0116
0.0941
0.0724
0.0036
0.0017
0.0660
0.0322
+ IPS-CN
0.0072
0.0054
0.0900
0.0599
0.0148
0.0136
0,0836
0.0639
0.0038
0.0017
0.0658
0.0317
+ CausE
0.0055
0.0040
0.0966
0.0134
0.0121
0.0717
0.0029
0.0013
0,0149
0.0221
+ summtreg
0.0076
0.0056
0.0653
0.0436
0.0157
0.0149
0.0773
0.0600
0.0056
0.0027
0.0502
0.0252
+ MACR
0,0078
0.0055
0.0331
0.0532
0.0183
0.0153
0.0767
0.0600
0.0033
0.0015
0,0475
0.02338
0.0095*
0.0073
0.1194
0.0832
0.0257
0.0227
0.1123
0.0903
0.0077
0.0037
0.0992
0
Imp. %
25.0%
30.1%
12.1%
16.9%
40.45
48.4%
19.3%
24.7%
37,5%
37.0%
50,3%
58.4%
imbalanced or balanced test set is evaluated. However, in real-world applications, the test distributions are usually unavailable and can even reverse the prior in the training distribution. We conjecture that good debiasing recommender is required to perform well on both imbalanced and balanced test distributions. In our settings, no information about the balanced test is provided in advance.
Data Splits. The models are identical across both imbalanced and balanced evaluations. The test distribution in the balanced evaluation uniform, i.e., randomly sample 15% of interactions with equal probability w.r.f. items. Besides, the test splits for the imbalanced test are similarly long- tailed like the train and validation sets, i.e., randomly split the remaining interactions into training, validation, and imbalanced test sets (60%: : 10%: 15%).
Results. Table reports the comparison of performance in imbalanced and balanced test evaluations. The best performing methods are bold and starred, while the strongest baselines are underlined; Imp.% % measures the relative improvements of BC loss over the strongest baselines. We observe that:
BC loss significantly outperforms the state-of-the-art baselines in both balanced and imbal- anced evaluations across all datasets. In particular, it achieves consistent improvements over the best debiasing baselines and original CF models by 12.1% ~ 58.4%. This clearly demonstrates that BC loss not only effectively alleviates the amplification of popularity bias but also improves the discriminative power of representations. Moreover, Table 5 shows the computational costs of all methods. Compared to the backbone models, BC loss only adds negligible time complexity.
Debiasing baselines sacrifice the imbalanced performance and perform inconsistently across datasets. Debiasing strategies generally achieve higher balanced performance at the expense of a large imbalanced performance drop. Specifically, the strongest basclines over all imbalanced test sets are the original CF models. Worse still, as the degree of data sparsity increases, some debiasing methods fail to quantify the popularity bias and limit their bias removal ability. For example, in the sparest Aibaba-iFashion dataset, the results of MF+IPS-CH MF+CausE, LightGCN+MACR, and LightGCN+Causs on the balanced evaluation are lower than original CF models (MF or Light- GCN). In contrast, benefiting from popularity bias-aware margin, BC loss can learn discriminative representations that accomplish more profound user and item understanding, leading to higher head and tail recommendation quality.
5.1.2 Evaluations on Temporal Split Test Set
Motivation.
Table 2: The performance comparison on Douban dataset.
In real applications, popularity bias dynamically changes over time. Here we consider temporal split test eval- uation on Douban Movie where the historical interactions are sliced into the training, validation, and test sets (7:1:2) according the timestamps.
MF
LightGCN Recall
HR
Recall
NDCG
HR
NDCG
Backhone
0.2924
0.0294
0.0472
0.3543
0.0313
0.0602
+ IPS-CN
0.2514
0.0174
0.0324
0.3212
0.0261
0.0502
CausE
0.2725
0.0203
0.0376
0.3403
0.0275
0.0514
sam+reg
0.2826
0.0191
0.0390
0,2944
0.0252
0.0488
MACR
0.1084
0.0087
0.0163
0.3127
0.0271
0.0519
BC loss
0.3742
0.0324*
0.0601
0.3562
0.0346
0.0652
Imp. %
28.0%
10.2%
27.3%
0.5%
10.4%
8.3%
Results. As Table shows, BC loss
is stcadily superior to all baselines w.n.t. all metrics on Douban Movic. For instance, it achieves
7

---

Table 3: The performance evaluations of head, mid, and tail Tencent dataset.
Balanced NDCG6
Imbaalaccdd NDCG Mid
Tail
Mid
Head
Overall
Tail
Head
Overall
MF
0,00094
0.00097
0.02500
0,00402
0.00021
0.00197
0,06837
0.06431
IPS-NN
0.00000" 2555
0.00212 1199
0.01684 6555
0,00775 H-T3
0.00056 100%
0.00401 1000
0,04439 453
0.04205 -155
CausE
0.00008
0.00L491608
0,01188
0,00630+78
0.00028
0.002X3
0,04876
0,04580
0.00000
0.00135 190g
0.01575
0,00535 HARE
0.00011
0.00281
0.02850
0.02661
MACR
100188 --000055
00021 1275
0.00553 1530
0.00456 +198
0,00370 +1G600S
0.00613
0.02748
0.02413
BC loss
0.00024 50005
0.00355
0.01831
0. 0.00680
0.00142
0.00712
0 11095522
0.090440
LightGCN
0.00025
0.00193
0.01336
0.00417
0.00094
0.00291
0,07561
0,07121
IPS-
0,001 1005S
0,00241-
0.01580-377
0,00544 -00T
0.00108 -005
0.00522 336
0
0.05993 -105
CausE
0.00000 -785
0.0013S
0.01177
0.00:003 3X
0.000:0 5076
0.00279 20
0,08996 TM
0,06650
0.00006 7856
0.00120
0.01727
0,005600 H540
0.00024
0.00255
0.04647
0.04335
MACR
0.00287 1104895
0. 0.00-461 --3005
0,004554
19035
0,007011 1000
0.00389 4-133
0.006335
-22%
0,04058
0.05323 -105
BCC loss
0.00057
0.0052]
0.01943
0. 0,00730
0.00125
0.00516
0 080882
0. 0.06320
significant improvements over the MF and LightGCN backbones w.r. Recall @ 20 by 10.2% and 10.4%, respectively. This validates that BC loss endows the backbone models with better robustness against the popularity distribution shift and alleviates the negative influence of popularity bias. Surprisingly, none of the debiasing baselines could maintain comparable performance to the backbones. We ascribe the failure to their preconceived idea of tail items, which possibly change over time.
5.2 Head, Mid, & Tail Performance (RQ2)
Motivation. To further evaluate whether BC loss lifts the tail performance by inevitably sacrificing the head performance, we divide the test set of Tencent into three subgroups, according to the interaction number of each item: head (popular items that are in the top third), mid (normal items in the middle), and tail (unpopular items in the bottom third). Most previous studies focus on average NDCG@ @ 20 for evaluation, especially balanced test evaluations [22, 23J. However, average metrics could be insufficient to reflect the performance of each subgroup. A trivial solution to achieve high performance is promoting the rankings of low-popularity items in the recommendations. In this case, only the average metries are not reliable on the balanced test. Therefore, we report the performance of individual subgroups both balanced and imbalanced test sets for more comprehensive comparison.
Results. Table 3 shows the evaluations of the head, mid, and tail subgroups. The red and blue numbers in percentage separately refer to the improvement and decline of each method relative the original CF model (MF or LightGCN). We find that:
BC loss is the only method that consistently yields remarkable improvements in every sub- group. With a closer look at the head evaluation, BC loss shows its ability to learn more discrimi- native representations for popular items across imbalanced and balanced settings. In particular, it achieves significant improvements over MF and LightGCN w.t.t head NDCG by 40% and 17% in the imbalanced test evaluation, respectively. We attribute improvements the usage of bias-aware margin, which boosts the recommendation quality for the tail and head items.
As the performance comparison among subgraphs in the imbalanced scenario shows, the baselines enhance the tail performance but sacrifice the head performance. Specifically, these baselines hardly maintain the head performance and show a clear trade-off trend between the head and tail performance. Taking MACR as an example, although the great improvement (+1662%) over MF is achieved in the tail subgraph, it brings in the dramatic drop (-62%) in the head subgraph, which lowers the overall performance by big drop (-60%). Here we ascribe the trade-off blindly promoting the rankings of tail items for matched and unmatched, rather than improving the discriminative power of representations.
5.3 Study on BC Loss (RQ3)
Effeet of Bias-aware Margin. Figure 3a displays the performance on balanced and imbalanced test sets on Tencent among softmax loss, BC loss with constant margin M [40], and BC loss with adaptive bias-aware margin. BC loss achieves the best performance, illustrating that bias-aware margin indeed is effective at reducing popularity bias and learning high-quality representations.
8

---

Balenced NDCO-32
8.008
0.08
Softmax
Balanced NDCG92
0.048
2.09
0.006
0.06
Sslanced
0.304
6.004
0.04
Imbo
0.000
Balanced
balanced
0.00
(c) LighiGCN+BC loss Figure 3: (a) Comparisons with varying margin; (b-c) Temperature sensitivity analysis Tencent.
(a) Varying margins.
(b) MF+BC loss
Effect of Temperature т. BC loss has one hyperparameter tune temperature in Equation (5). In Figure 3b and 3c, both balanced and imbalanced evaluations exhibit the concave unimodal functions of where the curves reach the peak almost synchronously in small range of r. For example, MF+BC loss gets the best performance when = 0.05 and r = 0.07 in balanced and imbalanced settings, respectively; We observe similar trends on other datasets and skip them due to the space limit. This justifies that BC loss does not suffer from the trade-off between the balanced and imbalanced evaluations and improves the generalization without sacrificing the head performance.
6
Related Work
Prevalent popularity debiasing strategies in CF roughly fall into four research lines.
Post-processing re-ranking methods [5, 6, 8, 9] applied to the output of the recommender system without changing the representations of users and items. The purposes of modifying the
ranking of models be various: Calibration [5] ensures that the past interests proportions of users are expected to maintain at the same level; RankALS [6] aims to increase the diversification of recommendation; FPC [8] investigates the popularity bias in the dynamic recommendation by rescaling the predicted scores.
Regularization-basee frameworks [10, 11, 12, 9] explore the of regularization that provides tunable mechanism for controlling the trade-off between recommendation accuracy and coverage. The difference among these methods is the design of penalty terms: ALS+Reg [11] defines intra-list distance as the penalty to achieve the fair recommendation; ESAM [10] introduces the attribute correlation alignment, center-clustering, and self self-training regularization to learn good feature rep- resentations; sam-reg [12] regularizes the biased correlation between user-item relevance and item popularity; Reg [9] decouples the item popularity with the model preference predictions.
Sample re-weighting methods [13, 14, 15, 16, 17, 18], also known as Inverse Propensity Score (IPS), view the item popularity in the training set the propensity score and exploit its inverse to re-weight loss each instance. To address the high variance of re-whighted loss, many them [15, 14] further employ normalization smoothing penalty to attain more stable output. However, the unreliability of methods is due to their measurement of the propensity score, leveraging the item frequency but failing consider interaction- wise popularity bias.
Bias removal by causal inference methods [19, 24, 23, 20, 21, 22], getting inspiration from the recent success of counterfactual inference, specify the role of popularity bias in assumed causal graphs and mitigate the bias effect on the prediction. However, the causal structure heuristically assumed based on the author's understanding, without any theoretical guarantee.
BC loss opens up possibility conventional debiasing methods in CF that mitigate the popularity bias by enhancing the discriminative power. Recent studies, boosting the discriminative feature spaces by modified softmax loss are mainly discussed face recognition, where a constant margin is added [40] to better classify. We transfer it in CF and compare it with BC loss in Figure 3a.
Conclusion
Despite the great success in collaborative filtering, today's popularity debiasing methods are still far from being able to improve the recommendation quality. In this work, we proposed simple yet effective BC loss, utilizing popularity bias-aware margin to eliminate the popularity bias. Grounded by theoretical proof, clear geometric interpretation and real-world visualization study, BC loss boosts

---

the head and tail performance by learning more discriminative representation space. Extensive experiments verify that the remarkable improvement head and tail evaluations on various test sets indeed comes from the better representation rather than simply catering to the tail.
The limitations of BC loss are in three respects, which will be addressed in future work: 1) the model- ing of bias-aware margin is worth exploring, which could significantly influence the performance of BC loss, 2) multiple important biases, such as exposure and selection bias, are not considered, and 3) more experiments comparing BC loss to standard CF losses (e.g., cross-entropy, WARP) are needed to further demonstrate the power of BC loss in regular recommendation tasks (See comparison to BPR, CCL and SSM in Appendix B.4). We believe that this work provides a potential research direction to diagnose the debiasing long-tail tail ranking and will inspire more works.
Acknowledgments and Disclosure of Funding
This research is supported by the Sea-NExT Joint Lab, and CCCD Key Lab of Ministry of Culture and Tourism. Assistance provided by Jingnan Zheng (e0718957@u. nus. edu) is greatly appreciated.
References
[1] Himan Abdollahpouri. Masoud Mansoury, Robin Burke. and Bamshad Mobasher. The un- fairness of popularity bias in recommendation. In RMSE@ @ RecSys, volume 2440 of CEUR Workshop Proceedings, 2019.
[2] Rocío Cañamares and Pablo Castells. Should follow the crowd?: A probabilistic analysis of the effectiveness of popularity in recommender systems. In SIGIR, 2018
[3] Yitong Ji, Aixin Sun, Jie Zhang, and Chenliang Li. A re-visit of the popularity baseline in recommender systems. SIGIR, 2020.
[4] Sirui Yao and Bert Huang. Beyond parity: Fairness objectives for collaborative filtering. In NIPS, 2017.
[5] Harald Steck. Calibrated recommendations. RecSys, 2018
[6] Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. Managing popularity bias in recommender systems with personalized re-ranking. In FLAIRS Conference, 2019.
[7] Harald Steck. Collaborative filtering via high-dimensional regression. CoRR, 1et/1904.13333 2019.
[8] Ziwei Zhu, Yun He, Xing Zhao, and James Caverlee. Popularity bias in dynamic recommenda- tion. In KDD, 2021.
[9] Ziwei Zhu, Yun He, Xing Zhao, Yin Zhang, Jianling Wang, and James Caverlee. Popularity- opportunity bias collaborative filtering. WSDM, 2021.
[10] Zhihong Chen, Rong Xiao, Chenliang Li, Gangfeng Ye, Haochuan Sun, and Hongbo Deng. ESAM: discriminative domain adaptation with non-displayed items to improve long-tail perfor- mance. In SIGIR, 2020.
[11] Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. Controlling popularity bias in learning-to-rank recommendation. In RecSys, 2017.
[12] Ludovico Boratto, Gianni Fenu, and Mirko Marras. Connecting user and item perspectives in popularity debiasing for collaborative recommendation. Inf. Process. Manag., 58(1):1023)) 2021.
[13] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In ICML, 2016.
[14] Léon Bottou, Jonas Peters, Joaquin Quiñonneo Candela, Denis Xavier Charles, Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Y. Simard, and Ed Snelson. Counterfactual reasoning and learning systems: the example of computational advertising. J. Mach. Learn, Res., 14(1):3207- 3260, 2013.
[15] Alois Gruson, Praveen Chandar, Christophe Charbuillet, James McInerney, Samantha Hansen, Damien Tardieu, and Ben Carterette. Offline evaluation to make decisions about playlist recommendation algorithms. In WSDM, 2019.
10

---

[16]
Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. Unbiased learning- coominggto--ralle to-rank with biased feedback. In IJCAI, 2018.
[17] Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge J. Belongie, and Deborah Estrin. Unbiased offiin recommender evaluation for missing-not--t-raddoon implicit feedback. In RecSys, 2018.
[18] Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata. Unbiased recommender learning from missing-not not-at-random implicit feedback. In WSDM, 2020.
[19] Stephen Bonner and Flavian Vasile. Causal embeddings for recommendation, In RecSys, 2018.
[20] Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. Causal intervention for leveraging popularity bias in recommendation. In SIGIR, 2021.
[21] Wenjie Wang, Fuli Feng. Xiangnan He, Xiang Wang, and Tat-Seng Chua. Deconfounded recommendation for alleviating bias amplification. In KDD, 2021.
[22] Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, and Xiangnan He. Model-agnostic counterfactual reasoning for climinating popularity bias in recommender system. In KDD, 2021.
[23] Yu Zheng, Chen Gao, Xiang Li. Xiangnan He. Yong Li, and Depeng Jin. Disentangling user interest and conformity for recommendation with causal embedding. In WWW, 2021.
[24] Dugang Liu, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. A general knowledge distillation framework for counterfactual recommendation via uniform data. In SIGIR, 2020.
[25] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong- Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In SIGIR, 2020.
[26] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thienee BPR: bayesian personalized ranking from implicit feedback. CoRR, abs/1205.261 2618, 2012.
[27] Yoshua Bengio and Jean-Sébastien Senecal. Quick training of probabilistic neural nets by importance sampling. In AISTATS, 2003.
[28] Steffen Rendle. Item recommendation from implicit feedback. CoRR, abs/2101.08766 2021.
[29] Yehuda Koren. Factorization meets the neighborhood: multifaceted collaborative filtering model. In KDD, 2008.
[30] Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffmman, and Tony Jebara. Variational autoen- coders for collaborative filtering. In WWW, 2018.
[31] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In WWW, 2017.
[32] Travis Ebesu, Bin Shen, and Yi Fang. Collaborative memory network for recommendation systems. In SIGIR, 2018
[33] Santosh Kabbur, Xia Ning, and George Karypis. FISM: factored item similarity models for top-n recommender systems, In KDD, 2013.
[34] Rianne van den Berg, Thomas N. Kipf, and Max Welling. Graph convolutional matrix comple- tion. In KDD, 2018.
[35] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In KDD, 2018.
[36] Steffen Rendle, Walid Krichene, Li Zhang, and John R. Anderson. Neural collaborative filtering vs. matrix factorization revisited. In RecSys, 2020.
[37] Ying Shan, T. Ryan Hoens, Jian Jiao, Haijing Wang, Dong Yu, and J. C. Mao. Deep crossing: Web-scale modeling without manually crafted combinatorial features. In KDD, 2016.
[38] Jason Weston, Samy Bengio, and Nicolas Usunier. WSABIE: scaling up to large vocabulary image annotation. In IJCAI, 2011.
[39] Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Yiqi Wang, Jiliang Tang, Charu Aggarwal, Prasenjit Mi- tra, and Suhang Wang. Investigating and mitigating degree-related biases graph convoltuional networks. In CIKM, 2020.
11

---

[40] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. CVPR, 2019.
[41] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR. 2017.
[42] Chang Zhou, Jianxin Ma, Jianwei Zhang, Jingren Zhou, and Hongxia Yang. Contrastive learning for debiased candidate generation in large-scale recommender systems. In KDD, 2021.
[43] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang. Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In CIKM, 2020.
[44] Defu Lian, Qi Liu, and Enhong Chen. Personalized ranking with importance sampling. In WWW, 2020.
[45] Xinyang Yi, Ji Yang. Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed H. Chi. Sampling-bias-correed neural modeling for large corpus item recommendations. In RecSys, 2019.
[46] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR, 2021.
[47]
Fajie Yuan, Xiangnan He, Haochuan Jiang, Guibing Guo, Jian Xiong, Zhezhao Xu, and Yilin Xiong. Future data helps training: Modeling future contexts for bessonn-based recommendation. In WWW, 2020.
[48] Ruining He and Julian J. McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In www, 2016.
[49] Wen Chen, Pipei Huang, Jiaming Xu, Xin Guo, Cheng Guo, Fei Sun, Chao Li, Andreas Pfadler, Huan Zhao, and Bingiang Zhao. POG: personalized outfit generation for fashion recommendation at alibaba ifashion. In KDD, 2019.
[50] Weiping Song, Zhiping Xiao, Yifan Wang, Laurent Charlin, Ming Zhang, and Jian Tang. Session-based social recommendation via dynamic graph attention networks. In WSDM, 2019.
[51] Benjamin M. Marlin and Richard S. Zemel. Collaborative prediction and ranking with non- random missing data. RecSys, 2009.
[52] Chongming Gao, Shijun Li, Wenqiang Lei, Biao Li, Peng Jiang, Jiawei Chen, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset for recommender systems. CoRR, abs/2202.10842, 2022.
[53] Ziwei Zhu, Jianling Wang, and James Caverlee. Measuring and mitigating item under- recommendation bias in personalized ranking systems. In SIGIR, 2020.
[54] Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, and Xiuqiang
He. Simplex: A simple and strong baseline for collaborative filtering. In CIKM, 2021.
[55] Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, Tianyu Qiu, and Xiangnan
He. On the effectiveness of sampled softmax loss for item recommendation. CoRR, 2022.
[56) Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised graph learning for recommendation. In SIGIR, 2021.
[57] Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. unifying mutual information view of metric learning: Cross-entropy vs. pairwise losses. In ECCV, 2020.
[58] Meihong Wang and Fei Sha. Information theoretical clustering via semidefinite programming.
In AISTATS, 2011.
[59] Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. KGAT: knowledge graph attention network for recommendation. In KDD, 2019.
[60] Dawen Liang, Laurent Charlin, and David M Blei. Causal inference for recommendation. In Causation: Foundation Application, Workshop at UAL. AUAI, 2016.
[61] Zaiqiao Meng. Richard McCreadie, Craig Macdonald. and Iadh Ounis. Exploring data splitting strategies for the evaluation of recommendation models. In RecSys, 2020.
[62] Walid Krichene and Steffen Rendle. On sampled metrics for item recommendation. In KDD,
2020.
12

---

[63] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
13


Reviews:
[{"review_id": "xRDCFts1Mpz", "reviewer": "Reviewer_aXKZ", "summary": "This work tries to reduce the negative impact of popularity bias on CF models. The paper incorporates Bias-aware margins into Contrastive loss and proposes a simple yet effective BC Loss, where the margin tailors quantitatively to the bias degree of each user-item interaction. Extensive experiments on six datasets demonstrate the effectiveness of the proposed algorithm.", "questions": "Please refer to my questions above.", "limitations": "The limitations are discussed in the Section 7. I’m not aware of any potential negative societal impact.\n\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "For the strengths, following the idea of MACR, this paper also designs a bias extractor and aims to remove the popularity bias in practice. Interestingly, this work explored the debias problem from a geometric view and unified the bias term into a marginal contrastive learning framework elegantly. The geometric interpretation and theoretical analysis also show the good properties of the proposed BC loss. Extensive experiments on six datasets with various settings verify the effectiveness of BC loss on both debiasing and general recommendation. Overall, I agree this is a good paper with clear novelty.\n\nMeanwhile, I also have some questions. For the Section Popularity Bias Extractor, I wonder if it'll lose part of useful bias information if just inputting the user/item popularity score. Have you tried other inputs? It's also unclear what kinds of interactions will be learned well according to equation (4). It'll be better if you can provide some visualizations of the learned angles of different types of interactions in the experiments. For the experiments, I'd like to know if the balanced test setting uses an imbalanced or balanced validation set. Though current experiments demonstrate the ability of BC loss on debiasing. however, to verify its effectiveness as a standard loss, more comparisons with some general loss functions are required.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "vWNgPhYjoWq", "reviewer": "Reviewer_jXGF", "summary": "This paper proposes a simple-yet-effective loss, called Bias-aware margins into Contrastive (BC) loss, to mitigate the popularity bias. Specifically, it introduces a popularity bias extract to estimate the popularity bias scores from a pair of user and item popularity. Then it incorporates it into the contrastive loss as the marginal information. Despite the simplicity of the proposed loss, experimental results show that the BC loss can be effective for tail/head evaluation and non-debiasing methods. Overall, the idea of using BC loss is interesting to me, and it also shows the theoretical analysis.", "questions": "- Q1) It is wondering whether the popularity bias effectively captures the popularity bias. Could you show some empirical or quantitative results?\n- Q2) The source code does not work well. It may be related to pre-processing. Please check your code.\n", "limitations": "This paper does not address the negative societal impact. However, this paper seems not to have any negative impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- (Originality) The idea of using the popularity bias extractor is quite interesting.\n- (Quality) The proposed loss outperforms existing methods in various evaluation protocols.\n- (Clarity) It is well-written and easy to understand. Also, the theoretical proof is quite solid.\n\n\nWeaknesses\n- (Originality) Although the idea of incorporating the popularity bias into the contrastive loss is interesting, some recent studies also tried similar ideas. It is necessary that the proposed loss is compared with the following references.\n\n[1] Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, Xiuqiang He, SimpleX: A Simple and Strong Baseline for Collaborative Filtering, CIKM 2021\n\n[2] Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, Tianyu Qiu, Xiangnan He, On the Effectiveness of Sampled Softmax Loss for Item Recommendation, 2022\n\n- (Quality) Although the proposed model is compared with various debasing strategies, the models using inverse propensity scores (IPS) are not compared. It would be better to show that the proposed model is still better than some existing studies using IPS.\n- (Quality) Although the proposed model shows the results on the KuaiRec dataset, most existing studies show two datasets for unbiased evaluation, i.e., Coat and Yahoo! R3. For a fair comparison, it would be better to show additional experimental results on these datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QmsBJonInY5", "reviewer": "Reviewer_BZTL", "summary": "This paper proposes a loss function to mitigate the popularity bias in recommendation. It argues that existing debiasing methods poses two limitations: i) focusing on the trade-off between evaluations for popular and unpopular users/items (e.g. re-weighting popular and unpopular samples), ii) holding infeasible assumptions (e.g. accessibility to unbiased data). The proposed method can be divided into two parts: i) Evaluating the bias degree of each user-item pair, using the prediction error of a popularity-based interaction prediction network. ii) The proposed BC Loss, which is basically the softmax cross-entropy loss with the bias-specific margin based on the bias degree. The paper also gives geometric illustrations and some theoretical analysis on the strength of the proposed BC Loss.", "questions": "The essence of the proposed BC loss is to add popularity-related bias to the prediction scores of normal recommenders, which share the same debiasing paradigm with some existing debiasing works. One difference is that this paper measure the popularity bias using radius, which yields intuitive geometric interpretation.\nThe idea of employing an extra predictor based on popularity features to extract popularity degree is novel and promising. But the technical solution of this paper is rather simple, and the theoretical analysis does not give deep enough investigation on the re. \n\nTo evaluate the bias degree, if the popularity-based predictor achieves strong performance for a user-item pair, this pair is strongly influenced by the popularity.", "limitations": "Please refer to the identified weakness.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1.\tThe proposed model is a simple plugin suitable for various kind of recommendation methods.\n2.\tAccording to the experimental evaluations, the proposed BC loss indeed results in overall performance improvements for both popular and unpopular items.\nWeakness:\n1.\tThe technical contribution of this paper is limited. Comparing to existing CF methods, it only proposes to employ an extra popularity-based predictor and combine the results with an existing CF model.\n2.\tThe paper overclaims the strength of the proposed BC loss in theoretical analysis. The geometric interpretability, theorem 1, the high/low entropy representations, and the hard-negative mining ability, are actually the same thing (i.e., applying stronger constrains for samples with higher popularity) from different viewpoints.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept