PSL: Rethinking and Improving Softmax Loss from Pairwise Perspective for Recommendation
Authors: Weiqin Yang
Jiawei Chen
Xin Xin
Sheng Zhou
Binbin Hu
Yan Feng
Chun Chen
Can Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Recommender Systems
Ranking Metrics Optimization
Surrogate Loss
Distributionally Robust Optimization
Primary area: other
TL;DR: This paper rethinks Softmax Loss (SL) from a pairwise perspective, introducing a novel family of robust DCG surrogate losses to address the limitations of SL, termed Pairwise Softmax Loss (PSL).

Abstract: Softmax Loss (SL) is widely applied in recommender systems (RS) and has demonstrated effectiveness. This work analyzes SL from a pairwise perspective, revealing two significant limitations: 1) the relationship between SL and conventional ranking metrics like DCG is not sufficiently tight; 2) SL is highly sensitive to false negative instances. Our analysis indicates that these limitations are primarily due to the use of the exponential function. To address these issues, this work extends SL to a new family of loss functions, termed Pairwise Softmax Loss (PSL), which replaces the exponential function in SL with other appropriate activation functions. While the revision is minimal, we highlight three merits of PSL: 1) it serves as a tighter surrogate for DCG with suitable activation functions; 2) it better balances data contributions; and 3) it acts as a specific BPR loss enhanced by Distributionally Robust Optimization (DRO). We further validate the effectiveness and robustness of PSL through empirical experiments. The code is available at https://github.com/Tiny-Snow/IR-Benchmark.

PSL: Rethinking and Improving Softmax Loss from Pairwise Perspective for Recommendation
Weiqin Yang ti Zhejiang University
Jiawei Chen +f18 Zhejiang University s sleepyhunt@zju edu. cn
Xin Xin Shandong University xinxin@sdu. edu. en
tinysnow0zju. edu.
Sheng Zhou
Binbin Hu
Yan Feng 13
Zhejiang University zhousheng_2juūzju.u edu. on
Ant Group bin www.hhbbett.....m com
Zhejiang University
f fengyan0zju edu. cn
Chun Chen i! Zhejiang University chenc&zju. edu. cn
Can Wang 18
Zhejiang University wcan@zju. edu. cn
Abstract
Softmax Loss (SL) is widely applied in recommender systems (RS) and has demon- strated effectiveness. This work analyzes SL from pairwise perspective, revealing two significant limitations: 1) the relationship between SL and conventional rank- ing metries like DCG is not sufficiently tight; 2) SL. is highly sensitive false negative instances. Our analysis indicates that these limitations are primarily due to the use of the exponential function. To address these issues, this work extends SL to new family of loss functions, termed Pairwise Softmax Loss (PSL), which replaces the exponential function in SL with other appropriate activation functions. While the revision is minimal, we highlight three merits of PSL: I) it serves as a tighter surrogate for DCG with suitable activation functions; 2) it better balances data contributions; and 3) it acts specific BPR loss enhanced by Distributionally Robust Optimization (DRO). We further validate the effectiveness and robustness of PSL through empirical experiments. The code is available at https: / //g thub. com/Tiny- Snow/ IR/ RRBenchmark.
Introduction
Nowadays, recommender systems (RS) have permeated various personalized services [1-4]. What sets recommendation apart from other machine learning tasks is its distinctive emphasis on ranking [5]. Specifically, RS aims to retrieve positive items in higher ranking positions (i.e., giving larger prediction scores) over others and adopts specific ranking metrics (e.g., DCG [6] and MRR [7]) to evaluate its performance.
The emphasis ranking inspires surge of research on loss functions in RS. Initial studies treated recommendation primarily as a classification problem, utilizing pointwise loss functions (c.g., BCE [8], MSE [9]) to optimize models. Recognizing the inherent ranking nature of RS, pairwise loss
Corresponding author.
State Key Laboratory of Blockchain and Data Security. Zhejiang University.
'College Computer Science Technology. Zhejiang University.
'Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

functions (e.g.. BPR [10]) were introduced to learn partial ordering among items. More recently. Softmax Loss (SL) [11] has integrated contrastive learning paradigms [12, 13], augmenting positive items as compared with negative ones, achieving state-of-the-arr (SOTA) performance.
While SL. has proven effective, it still suffers from two limitations: 1) SL can be used to approximate ranking metrics, c.g., DCG and MRR [11, 14], but their relationships are not sufficiently tight. Specifically, SL uses the exponential function exp(-) as the surrogate activation approximate the Heaviside step function in DCG, resulting in a notable gap, especially when the surrogate activation takes larger values. 2) SL sensitive to noise (e.g., false negatives [15]). Gradient analysis reveals that SL assigns higher weights to negative instances with large prediction scores, while the weights are rather skewed and governed by the exponential function. This characteristic renders the model highly sensitive to false negative noise. Specifically, false negative instances are common in RS, as user's s lack of interaction with an item might stem from unawareness rather than disinterest [16-18]. These instances would receive disproportionate emphasis, potentially dominating the training direction, leading to performance degradation and training instability.
To address these challenges, we propose a new family of loss functions, termed Pairwise Softmax Loss (PSL). PSL first reformulates SL in pairwise manner, where the loss is applied to the score gap between positive-negative pairs. Such pairwise perspective is more fundamental to recommendation as the ranking metrics are also pairwise dependent. Recognizing that the primary weakness of SL lies in its use of the exponential function, PSL replaces this with other surrogate activations. While this extension straightforward, it brings sigmificant theoretical merits:
Tighter surrogate for ranking metrics. We establish theoretical connections between PSL and conventional ranking metrics, e.g., DCG. By choosing appropriate surrogate activations, such as ReLU or Tanh, we demonstrate that PSL achieves tighter DCG surrogate loss than SL.
Control over the weight distribution. PSL provides flexibility in choosing surrogate activations that control the weight distribution of training instances. By substituting the exponential function with an appropriate surrogate activation, e.g., ReLU or Tanh, PSL can mitigate the excessive impact of false negatives, thus enhancing robustness to noise.
Theoretical connections with BPR loss. Our analyses reveal that optimizing PSL is equivalent to performing Distributionally Robust Optimization (DRO) [19] over the conventional pairwise loss BPR [10]. DRO is a theoretically sound framework where the optimization is not only on a fixed empirical distribution but also across a set of distributions with adversarial perturbations. This DRO characteristic endows PSL with stronger generalization and robustness against out-of- distribution (OOD), especially given that such distribution shifts are common in RS, e.g., shifts in user preference and item popularity [16, 20, 21].
Our analyses underscore the theoretical effectiveness and robustness of PSL. To empirically validate these advantages, we implement PSL with typical surrogate activations (Tanh, Atan, ReLU) and conduct extensive experiments on four real-world datasets across three experimental settings: I) IID setting [22] where training and test distributions are identically distributed [23]: 2) OOD setting [24] with distribution shifts in item popularity; 3) Noise setting [15] with a certain ratio of false negatives. Experimental results demonstrate the superiority of PSI. over existing losses terms of recommendation accuracy, OOD robustness, and noise resistance.
2 Preliminaries
Task formulation. We will conduct our discussion in the scope of collaborative filtering (CF) [25], widely-used recommendation scenario. Given the user set 2/ and item set I, CF dataset DD C 2 x is a collection of observed interactions, where each instance (u, i) E 'D means that user u has interacted with item i (c.g., clicks, reviews, etc). For each user u, we denote P., = {i E I: (u, i) E D) as the set of positive items of u. while I P., represents the negative items.
The goal of recommendation is to learn a recommendation model, or essentially a scoring function fu, i) 24 x I - R. that quantifies the preference of user u on item i accurately. Modern RS often adopts an sabeddinng-bsse paradigm [26]. Specifically, the model maps user u and item i into d-dim embeddings u. v.v E Rd, and predicts their preference score f u, i) based on embedding similarity. The cosine similarity is commonly utilized in RS and has demonstrated particular effectiveness [27]. Here we set (u, i) = uv
[iv 2. where the scaling factor 2 is introduced for faciliating analyses

---

and can be absorbed into the temperature hyperparameter (T). The scores fu, i) are subsequently utilized to rank items for generating recommendations.
Ranking metrics. The Discounted Cumulative Gain (DCG) [6] is a prominent ranking metric for evaluating the recommendation quality. Formally, for each user u, DCG is calculated as follows:
DCG( = Σ EP
(2.1)
logz (1 + Tu (i))
where (1) is the ranking position of item in the ranking list sorted by the scores f(u, i). DCG quantifies the cumulative gain of positive items, discounted by their ranking positions. Similarly, the Mean Reciprocal Rank (MRR) [7. 28] is another popular ranking metric using the reciprocal of the ranking position as the gain, i.e. MRR(u) = Liep, (1) Additionally, other metries such as Recall [29], Precision [29], and AUC [30] are also utilized in RS [29]. Compared to these metrics, DCG and MRR focus more on the top-ranked recommendations, thus attracting increasing attention in RS [11, 31J. In this work, aim to explore the surrogate loss for DCG and MRR.
Recommendation losses. train recommendation models effectively, a series of recommendation losses has been developed. Recent work on loss functions can mainly be classified into three types:
Pointwise loss (e.g., BCE [8], MSE [9], etc.) formulates recommendation specific classifica- tion or regression task, and the loss is applied to each positive and negative instance separately. Specifically, for each user u, the pointwise loss is defined
pointwise (u) = Σ
log(s* (f(u, xEP
Σ JEZYP..
log(s> (f(u,
(2.2)
where (-) and ( (.) are the activation functions adapted for different loss choices.
Pairwise loss (e.g., BPR [10], etc.) optimizes partial ordering among items, which is applied to the score gap between negative-positive pairs. BPR [10] is representative pairwise loss, which is defined as
LBpr ( = - Σ log (f(,j) (000:)))(_u11))))) fu,
(2.3)
(EP.JEIVP.
where denotes the activation function that approximates the Hcaviside step function. The basic intuition behind BPR loss is to let the positive instances have higher scores than negative instances. In practice, there are various choices of the activation function. For instance, Rendle et al. [10]
originally uses the sigmoid function, and the resultant BPR loss can approximate AUC metric. Softmax Loss (i.e., SI. [11]) normalizes the predicted scores into a multinomial distribution [32] and optimizes the probability positive instances over negative ones [33], which is defined as
exp(f(u, i)/T) Ejerexp( exp(f(u,.))7)
Cs((u) = Σ log PEP.
(2.4)
where is the temperature hyperparameter. SL can also be understood specific contrastive loss, which draws positive instances (u, i) closer and pushes negative instances (u,j) away (13).
3 Analyses on Softmax Loss from Pairwise Perspective
In this section, aim to first represent the Softmax Loss (SL) in pairwise form, followed by an analysis of its relationship with the DCG metric, where two limitations of SL are exposed.
Pairwise form of SL. To facilitate the analysis of SL and to build its relationship with the DCG metric, we rewrite SL (cf. Equation (2.4)) in the following pairwise form:
(sl(u) Σ log Σ
exp((duss/T) iEP JEI
where duij = f(,,j) ff(,
(3.1)
Equation (3.1) indicates that SL is penalized based on the score gap between negative-positive pairs, i.e., dwip = [u, j) - ʃ(u, i). This concise expression fundamental for ranking, as it optimizes the relative order of instances rather than their absolute values.

---

Connections between SL and DCG. We now analyze the connections between SL and the DCG metric (cf. Equations (2.1) and (3.1)), which could enhance our understanding of the advantages and disadvantages of SI. Our analysis follows previous work [11, 14], which begins by relaxing the negative logarithm of DCG with
log (C) + log [Pul - - log
1
Σ (1))
S
Σ iEP
log (u
(3.2)
|Pal
|Pal
where the first inequality holds due to log2(1 + Ru (1)) I Tu (d), and the second inequality holds due to Jensen's 's inequality [34]. Note that the ranking position Tu (i) of item i can be expressed as
Tu (1) EI(f(u.)) f(u, JET
Σ 8(dwe)) JET
(3.3)
where 6(-) denotes the Heaviside step function, with 0(x) for x 2 0 and 6(x) = 0 for æ < 0. Since 5(dui;) S exp(duis/7 holds for all T > we deduce that SL is a smooth upper bound of Equation (3.2), and thus serves reasonable surrogate loss for DCG and MRR metrics!.
However, our analysis also reveals two limitations of SL:
Limitation 1: SL is not tight enough as a DCG surrogate loss. There remains significant gap between the Heaviside step function 8(.) and the exponential function exp(-), especially when duij reaches a relatively large value, where cxp(.) becomes substantially larger than 8(-). This gap is further exacerbated by the temperature т. Practically, we find that the optimal T is usually chosen to be less than 0.2 (cf. Appendix B.5.2). Given the explosive nature of exp(-), the gap becomes extremely large, potentially leading to suboptimal performance of SI. in optimizing DCG.
Limitation 2: SL is highly sensitive to noise (e.g., false negative instances). False negative instances [15] are common in the typical RS. This often due to the exposure bias [16]. where user's lack of interaction with an item might stem from unawareness rather than disinterest. Unfortunately, SL is highly sensitive to these false negative instances. On one hand, these instances (u. which may exhibit patterns similar to true positive ones, are difficult for the model to differentiate and often receive larger predicted scores, thus bringing potentially larger duiz for positive items i. As analyzed in Limitation 1, these instances can significantly enlarge the gap between SL and DCG due to the exponential function, causing the optimization to deviate from the DCG metric.
Gradient analysis of SL. Another perspective to support the view of Limitation comes from the gradient analysis. Specifically, the gradient of SL w.r.t. duij is
@Ls((w) adui
exp(duis/T1
<eep(dus//T//
(3.4)
IExyz/exp(ddas////
As can be seen, SL implicitly assigns a weight to the gradient of each negative- pegative-positive pair, where the weight is proportional to (xp(dw///)) This suggests that instances with larger duij will receive larger weights. While this property may be desirable for hard mining [11], which can accelerate convergence, it also means that false negative instances, which typically have larger duij. will obtain disproportionately large weights, as shown in the weight distribution of SL in Figure 1b. Therefore, the optimization of SL can be easily dominated by false negative instances, leading to performance drops and training instability.
Discussions on DRO robustness and noise sensitivity. Recent work [15] claims that SL exhibits robustness to noisy data through Distributionally Robust Optimization (DRO) [19]. However, we argue that this is not the case. DRO indeed can enhance model robustness to distribution shifts, but also increases the risk of noise sensitivity, as demonstrated by many studies on DRO [35, 36). Intuitively, DRO emphasizes hard instances with larger losses, making noisy data contribute more rather than less to the optimization. This is also demonstrated from the experiments with false negative instances (cf. Figure 8 in [15]), where the improvements of SL over other baselines in Noise setting do not increase significantly but sometimes decay.
Note that the middle term in Equation (3.2), i.e., - log P.. ΣiEP, 1/Tu(i).. is exactly - log MRR(u). Therefore, SL serves as an upper bound of the negative logarithm of DCG and MRR, and minimizing SL leads the improvement of these ranking metries.

---

Bidwy)
175
exp(d.a)
150-
Munh
atan
adda)
6san
125
PSL-relu
dicla
100
50-
0.5
0.0
1.00
0.75 0.50 0.25 0.00 0.25 0.50
1.00
0
1.000
0.75
0.50
0.25
0.00
0.25 dut
0.50
0.75
1.00
dni
(a) Surrogate activations.
(b) Weight distributions.
Figure (a) Illustration different surrogate activations. compared with PSL using three different surrogate activations. Here we set = 0.2, which typically achieves optimal results in practice.
(b) The weight distribution SL as
Methodology
4.1 Pairwise Softmax Loss
Recognizing the limitations SI., particularly its reliance on the unsatisfactory exponential function, we propose to extend SL with more general family of losses, termed Pairwise Softmax Loss (PSL). In PSI., the exponential function exp(-) is replaced by other surrogate activations a--) approximating the Heaviside step function d(-). For each user the PSL is defined as
(u) Σ log
(dddnss) JEI
(4.1)
One might wonder why apply the temperature outside the activation function (i.c., extending exp(dwi)) ((duii))//T ) rather than within it (i.e., extending exp(duij/T) to a (duss/T)). This subtlety will be elucidated later as we demonstrate that the form in Equation (4. I) offers superior properties over the alternative.
Our PSL provides a flexible framework for selecting better activation functions, allowing the loss to exhibit improved properties compared to SL. We advocate for three activations. including PSL-tanh: and PSL-relu: T (dws 1). In the following, we will discuss the advantages of PSL. and provide evidence for the selection of these surrogate activations.
Junh tanh tann)d
PSL-atan: aaaan arctan(dd,,.
Advantage 1: PSL is a better surrogate for ranking metrics. To highlight the advantages of replacing exp(-) with alternative surrogate activations, we present the following lemma: Lemma 4.1. If the condition
5(duiz) S (duss) & exp(duss)
(4.2)
is satisfied for any duis E [-1,1) 1), then PSL. serves as tighter DCG surrogate loss compared to SL.
The proof is presented in Appendix A. This lemma reveals that PSI. could be tighter surrogate loss for DCG compared to SI.. Additionally, it provides guidance on the selection of proper surrogate activation may choose the activation that lies between exp(.) and d(-). As demonstrated in Figure la, our chosen surrogate activations Ganh Jatan and reeu adhere to this principle.
Advantage 2: PSL controls the weight distribution. The gradient of PSL w.r.t. duij is
o' (dwij) (ddi))) /T
x ('(us))) o(dus)) //-- 1/T-1 /T
(4.3)
Oduis
Note that the equation exp(dauj /7) exp(duis) 1/7 holds.
5

---

This implies that the shape of the weight distribution is determined by the choice of surrogate activation. By selecting appropriate activations, PSL can better balance the contributions of instances during training. For example, the three activations advocated before can explicitly mitigate the explosive issue on larger duis (ef. Figure lb), bringing better robustness to false negative instances.
One might argue that adjusting r in SL could improve noise resistance. However, such adjustments do alter the fundamental shape of the weight distribution, which remains exponential. Furthermore, as we discuss subsequently, 7 plays a crucial role in controlling robustness against distribution shifts. Thus, indiscriminate adjustments to J may compromise outtt--distibuution (OOD) robustness.
Advantage 3: PSL is a DRO-empowered BPR loss. We establish a connection between PSL and BPR [10] based Distributionally Robust Optimization (DRO) [19, 37J. Specifically, optimizing PSL. is equivalent to applying a KI. divergence DRO on negative item distribution over BPR loss (cf. Equation (2.3)), demonstrated in the following theorem²:
Theorem 4.2. For each user u and its positive item i, let P P(jlu, (u, i) be the uniform distribution over I. Given a robustness radius rj > 0, consider the uncertainty set Q consisting of all perturbed distributions Q = Q(Iuu, i) satisfying: (i) Q is absolutely continuous w.r.l. P, i.e. Q < P; (ii) the KL divergence between Q and P is constrained by t, i.e., (KI(Q S v). Then, optimizing PSL is equivalent to performing DRO over BPR loss, i.e.,
min
Ei-Pe
log Eس~I ( (bbgl((dm)))/
}} + min
EiwPa
sup Q)-QU) [log o llog (duss)) QEEQ
(x)
CRPR (M)
(4.4)
where: r = T(1) is temperature parameter controlled by n.
The proof is presented in Appendix A.2. Theorem 4.2 demonstrates how PSL, based on the DRO framework, is inherently robust to distribution shifts. This robustness particularly valuable in RS, where user preference and item popularity may shift significantly. Therefore, PSL can be regarded as a robust generalization of BPR loss, offering better performance in OOD scenarios.
In addition, Theorem 4.2 also gives insights into the rationality of PSL that differs from serving as a DCG surrogate loss, but rather as a eem-wweddered BPR loss:
Rationality of surrogate activations:
The activation function in BPR is originally chosen as an approximation to the Heaviside step function [10]. Since PSL is a generalization of BPR as stated in Theorem 4.2, it is reasonable to select the activations in PSL that aligns with the ones in BPR. Interestingly, this principle coincides with our analysis from the perspective of DCG surrogate loss.
Rationality of the position of temperature: Theorem 4.2 also rationalizes the extension form that places the temperature on the outside rather than inside. For the outside form (i.c., or (duis) 1/T), Theorem 4.2 holds, and the temperature 7 can be interpreted as a Lagrange multiplier in DRO optimization, which controls the extent of distribution perturbation. However, for the inside form (i.e., aduss /T) Theorem 4.2 no longer holds, and it would be challenging to establish the relationship between PSL and BPR.
Rationality of pairwise perspective: Recent work such BSL [15] also reveals the DRO property of SL (cf. Lemma in [15]). However, we wish to highlight the distinctions between Theorem 4.2 and Wu al. [15]'s analyses: 1) Wu et al. [15] views SL from pointwise perspective and associates it with a specific, less commonly used pointwise loss. In contrast, our analyses adopt a pairwise perspective and establish relationship between PSL and the widely used BPR loss. 2) We construct link between two families of losses with flexible activation selections, and Wu et al. [15]'s analyses can be regarded as a special case within our broader framework.
The above analyses underscore the advantages of PSL and provide the principles to select surrogate activations. Remarkably, PSL is easily implemented and can be integrated into various recommenda- tion scenarios. This can be achieved by merely replacing the exponential function exp(-) in SL with another activation (-) surrogating the Heaviside step function, requiring minimal code modifications.
'Note that (= = (dues)) 1/7 holds, thus the PSL in Equation (4.4) is identical to the one in Equation (4.

---

4.2 Discussions
Comparisons of two extension forms. In previous discussions, we highlight the advantages of the form that positions the temperature outside (i.e., o(dna;) 1/) over the inside (i.e., ('duij (T). As discussed in the analyses of Theorem 4.2, the outside form can be regarded as a DRO-empowered BPR, while the inside form cannot, which ensures the robustness of PSL against distribution shifts.
Here we provide an additional perspective on the advantages of the outside form. In fact, the outside form facilitates the selection of surrogate activations. For instance, ensure that PSL serves tighter DCG surrogate loss compared to SL (i.c., ensure Lemma 4.1 holds), the outside form only need to consider the condition (4.2) on the range of duij E [-1, 1]. However, for the inside form, this condition should be satisfied on the entire domain of the activation a a--), which complicates the selection of activation functions. Therefore, the outside form is more flexible and easier to implement.
We further provide empirical evidence in Appendix C.3, demonstrating that the inside form will lose the advantages of achieving tighter DCG surrogate loss, leading compromised performance.
Connections with other losses. We further discuss the connections between PSL and other losses:
Connection with AdvInfoNCE [38]: According to Theorem 3.1 in Zhang et al. [38], AdvInfoNCE
can indeed be considered as a special case of PSL with (-)) = exp(exp(-)). ) We argue that this activation is not a good choice as it would enlarge the gap between the loss and DCG. In fact, we have log DCG S LESl I CSL S CAdvInINNCE (cf. Appendix A.3 for proof). While AdvInfoNCE may achieve good performance in some specific OOD scenarios as tested in Zhang et al. [38], we argue that AdvInfoNCE is a looser DCG surrogate loss and would be highly sensitive to noise (cf. Table and Figure 2 in Section 5.2 for empirical validation).
Connection with BPR [10]: Besides the DRO relation stated in Theorem 4.2, we also derive the bound relation between BPR and PSL with the same activation, i.e., - log DCG S Lpst. S log LAPR (cf. Appendix A.3 for proof). This relation clearly demonstrates the effectiveness of PSL over BPR - performing DRO over BPR results robustness to distribution shifts, while also achieving a tighter surrogate of DCG, which is interesting (cf. Tables 1 and 2 in Section 5.2 for empirical validation). An intuitive explanation is that DCG focuses more on the higher-ranke ranked items. Given that DRO would give more weight to the hard negative instances with larger prediction scores and higher positions, it would naturally narrow the gap between BPR and DCG.
Experiments
5.1 Experimental Setup
Testing scenarios. We adopt three representative testing scenarios to comprchensively evaluate model accuracy and robustness, including: 1) IID setting: the conventional testing scenario where training and test data are randomly split and identically distributed; 2) OOD setting: to assess the model's 'ss robustness on the dutoof-distrbuution (OOD) data, we adopt a debiasing testing paradigm where the item popularity distribution shifts. We closely refer to Zhang al. [20], Wang et al. [24), and Wei et al. [39], sampling a test set where items are uniformly distributed while maintaining the tail nature of the training dataset; 3) Noise setting: to evaluate the model's sensitivity to noise, following Wu et al. [15], we manually impute a certain proportion of false negative items in the training data. The details of the above testing scenarios are provided in Appendix B.
Amazon- Datasets. Four widely-used datasets including Amazon-Bood Amazon-Electronic Electronic, Amazon-Movis [40, 41], and Gowalla [42] are used in our experiments. Considering the item popularity is not heavily skewed in the Amazon- Book and Amazon- Movie datasets, we turn to other conventional datasets, Amazon-CD [40, 41] and Yelp2018 [43], as replacements for OOD testing. All datasets are split into 80% training set and 20% test set, with 10% of the training set further treated as the validation set. The details of the above datasets are summarized in Appendix B.
Metrics. We closely refer to Wu et al. [15] and Zhang et al. [38], adopting Top-K K metrics including NDCG@K [6] and Recall K [29] for performance evaluation, where NDCG is the normalized DCG, i.e., dividing DCG by the ideal value. Here we simply set = 20 in recent work [15, 38] while observing similar results with other choices. For more details, please refer to Appendix B.2.
7

---

Table I: Performance comparison in terms of Recall@20 and NDCG@20 under the IID setting. The best result bolded, and the blue-colored zone indicates that PSL is better than SL. Imp. denotes the NDCG@ 20 improvement of PSL over SL. The marker "*" indicates that the improvement is statistically significant (p-value < 0.05).
Amazon-Boos
Amazon- Electrone
Amazon-Movie
Gowalla
Model
Loss
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
BPR [101
0,0665
0.0453
0,0816
0.0527
0.0916
0.0608
0.1355
0.1111
LLPALCC [44]
0.1150
0.0811
0.0821
0.0499
0.1271
0.0883
0.1610
0.1189
[11]
0.1559
0.1210
0.0821
0.0529
0.1286
0.0929
0,2064
0.1624
AdvInfoNCE [38]
0.1557
0.1172
0.0829
0.0527
0.1293
0.0934
0.2067
0. 0.1627
[15]
0.1563
0.2071
0.1630
MF [26]
PSL-tanh
0.1567
0.1225
0.0832
0,0535
0.1297
0.094]
0,2088
0.1646
PSL-atan
0.1567
0.1226
0.0832
0.0535
0.1296
0.0941
0.2087
0.1646
relu
0.1569
0.1227
0.0838
0.0541
0.1299
0.0945
0.2089
0.1647
Imp. Sc
+1.40 %
+2.31 %
+1.72%
1..2%
BPR [10]
0.0984
0.0678
0,0813
0.0524
0.1006
0.0681
0.1745
0.1402
LLPAUC [44]
0.1147
0.0810
0.0831
0.0507
0.1272
0.0886
0.1616
0.1192
SL
0.1567
0.1220
0.0526
0.0941
0,2068
0.1628
AdvInfoNCF: [38]
0.1568
0.1177
0.0823
0.0528
0. 0.12
0.0936
0,2066
0. 1625
BSL [15]
0.1568
0.1220
0.0823
0,0526
0.1306
0.0943
0,2009
0. 0.1628
LightGCN [22]
PSL-tanh
0,1575
0.1233
0,0825
0,0532
0,1300
0.0947
0,2091
0,1648
PSL-ataa
0.1575
0.1233
0.0825
0,0532
0.1300
0.0948
0.2091
0.1648
PST
0.1575
0.1233
0.0830
0.053
0. 1300
0.0953
0.2086
0.1648
Imp.% %
+1.12%* *
+1.98%
+1.22%
+1.24%
BPR [10]
0.1269
0.0905
0.0777
0.0508
0.1236
0.0857
0,1966
0,1570
LLPAUC [44]
0.1363
0.1008
0.0781
0.0481
0.1184
0.0828
0.1632
0.1200
SI.[1I]
AdvInfoNCE [38]
0.1568
0.1
0.0776
0,0489
0.1252
0.0906
0,2010
0.1564
BSL [15]
0.1550
0.1207
0.0800
0.0507
0.1267
0.0918
0.2037
0.1597
XSanGCL
PSL-tanh
0.1567
0.1225
0.0790
0.0501
0.1308
0.0926
0.2034
0.1591
PSI.
0.1565
0.1225
0.0792
0.0502
0.1 0.1253
0.0917
0.2035
0. 1591
PSL-relu
0.1571
0.1228
0.0801
0.0507
0.1313
0.0935
0,2037
0.1593
Imp.%
+1.72%
+3.39%
+3.42% *
+1.48%
Compared methods. Five representative loss functions are compared in our experiments, including the representative pairwise loss BPR (UAI'09 [10]); 2) the SOTA recommendation loss Softmax Loss (SL) (40IS'24 [11]) and its two DRO-ennancsmmnts Adv AdvInfoNCE (NIPS' 23 [38]) and BSL (ICDE'24 [15]); 3) another SOTA loss LLPAUC (wWW'2 24 [44]) that optimizes the Lower-Left Partial AUC. Refer to Appendix B.3 for more details about these baselines.
Backbones. We also adopt three representative backbone models to evaluate the effectiveness of loss, including MF [26], LightGCN [22], and XSimGCL [45], see Appendix B.4 for more details.
Hyperparameter settings. A grid search is utilized to find the optimal hyperparameters. For all compared methods, we closely refer to the configurations provided their respective publications to ensure their optimal performance. As also carefully finetune SL, the improvements of existing methods over it are not as significant as those presented in their papers. The hyperparameter settings provided in Appendix B.5, where the detailed optimal hyperparameters for each method on each dataset and backbone are reported.
5.2 Performance Comparisons
Results under IID setting. Table presents the performance of our PSL compared with baselines.
PSL outperforms SL and other baselines. Experimental results demonstrate that PSL, with three carefully selected surrogate activations, consistently outperforms SL across all datasets and backbones, with only few exceptions. For instance, on the MF backbone, compared to the marginal improvements or sometimes even degradation of AdvInfoNCE (+3%-0.5%) and BSL (0.0%-0.5%). PSL shows significant enhancement over SL (1)..)). Morcover, our PSL surpasses all compared basclines in most cases, clearly demonstrating its effectiveness.
PSL achieves tighter connections with ranking metries. We observe that the results align well with our theoretical analyses of PSL's Advantage Section By replacing the exponential function with other suitable surrogate activations, PSL establishes tighter relationship with ranking

---

Table 2: Performance comparison in terms of Recall @ 20 and NDCG@20 under the OOD setting with popularity shift (on MP backbone). The best result is bolded, and the blue-colored zone indicates that PSL is better than SL. Imp. % denotes the NDCG@ 20 improvement of PSL over SL. The marker "** indicates that the improvement is statistically significant (p-value < 0.05).
Amazon- CD
Amazon- Electoonic
Gowalla
Yelp2018
Lows
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
BPR 1101
0,0518
0.0318
0,0132
0.0069
0.0382
0.0273
0,0118
0.0072
LLPAUCC [44]
0.1103
0.0764
0.0225
0.0134
0.0729
0.0522
0.0324
0.0210
SL [1]]
0,1184
0.0815
0.0230
0.0142
0.1006
0.0737
0.0349
0.0224
AdvInfoNCE [38]
0.1189
0.0818
0.0228
0,0139
0.0927
0.0676
0,0348
0.0223
BSL [15]
0.1184
0.0142
0,0351
PSL-tonh
0.1202
0.0834
0,0239
0.0146
0.1013
0.0748
0.0357
0.0228
PSL-atan
0.1202
0.0835
0.0239
0.0144
0.1013
0.0748
0.0358
0.0228
PSL-relu
0.1203
0.0839
0.0241
0.0149
0.1014
0,0752
0.0358
0,0229
Imp.5 %
+3.01% *
+5.02%
+2.02% *
+2.05% *
Amazon-Book NDCG6620
Amazon Amazon-Flectreni NDCG6000
Amazon-Mavie NDCG620
0.090
18%
0.340-
14%
0.070
13%
NDCG820
0.085
RX
NDCG802
NDCO820
0.080-
Im
0.080
4%
0,300
0.075
Ing 56
PSI-cvli
7N4
6035.
Inp
PStredla
-05
2%
0.000
PSL-m/
Ing
2%
0.070
005
0.1
0.2
0.3
0.5
0.05
0.1
022
0.3
0.5
0005
011
0.2
05
0.5
Noise Ratio p
Nosse Ratio
Noise Ratio P
(a) Amazon-Boos
(b) Electroncccc
(c) Amazon- Movic
Figure 2: Performance comparison of SL and PSL in terms of NDCG@20 with different false negative noise ratio (on MF backbone). We also present the relative improvements (i.e., Imp. %) achieved by PSL over SL. The complete results of other baselines are provided in Appendix C.I.
metrics, thus achieving better NDCG performance (cf. Lemma 4.1). This is also empirically evident from the larger improvements in NDCG compared to Recall. In contrast, as discussed in Section 4.2, other baselines like AdvInfoNCE: and BSL, either widen the gap or fail to connect with the ranking metrics, resulting in slight improvements or even performance drops.
Results under OOD setting. Table 2 presents the results in OOD scenarios with popularity shift. Given the consistent behavior across the three backbones, here we only report the results on MF.
PSL is robust distribution shifts. Experimental results indicate that PSL has strong robustness against distribution shifts, which is consistent with PSL's Advantage 3 in Section 4. As can be seen, PSL not only outperforms all baselines (2%-5%), but also achieves more pronounced improvements than in IID setting, like on Amazon-Electrome (2.31 1.31% % - 5.02%) and Gowalla (1.42% - 2.02%). This demonstrates the superior robustness of PSI. distribution shifts, as shown in Theorem 4.2.
PSL is a DRO-enhancement of more reasonable loss. Although both PSL and SL can be considered as DRO-enhanced losses (cf. Theorem 4.2), the original loss of our three PSLs before DRO-enhancement is more reasonable than that of SL., which degenerates from BPR loss linear triplet loss [46). Therefore, we observe significant improvements of PSL. over SI..
Results under Noise setting. Figure 2 and Appendix C. presents the results with certain ratio of im- puted false negative noise. Specifically, we regard 10% of the positive items in the training set as false negative noise and allow the negative sampling procedure to have certain probability p of sampling those items. We test the model performance with varying noise ratios p E (0.05, 0.1, 0.2, 0.3, 0.5).
PSL has strong noise resistance. Experimental results demonstrate that as the noise ratio p increases, both the performance of SL and PSL decline. The performance decline rate of PSL. is significantly smaller than that of other baselines, resulting in higher performance enhancement(> 10% when p = 0.5). These results indicate that PSI. possesses stronger noise resistance than SI., which stems from our rational activation design, discussed in PSL's Advantage 2 in Section 4.

---

However, for DRO-enhanced losses such as AdvInfoNCE, the performance declines similarly to or even more quickly than SL (ef. Appendix C.I), which coincides with our theoretical analyses.
6 Related Work
Model-related recommendation research. Recent years have witnessed flourishing publications on collaborative filtering (CF) models. The earliest works are mainly extensions of Matrix Factorization [26], building more complex interactions between embeddings [47], such as MF [26], LRML [48], SVD [49, 50]. SVD++ [51], NCF [8], etc. In recent years, given the effectiveness of Graph Neural Networks (GNNs) [52-58] in capturing high-order relations, which align well with CF assumptions, GNN-based models have emerged and achieved great success, such as LightGCN [22], NGCF [55], ICF [59], APDA [60], etc. Building upon LightGCN, some works attempt to introduce contrastive learning [12, 61] for graph data augmentation, such as SGL. [62] and XSimGCI, [45], achieving SOTA performance in recommendation.
Loss-related recommendation research. Existing recommendation losses can be primarily cat- egorized into pointwise loss [8, 9]. pairwise loss [10], and Softmax Loss (SL) [11], discussed in Section 2. Given the effectiveness of SL, recently some researchers have proposed enhance SL from different perspectives. For instance, BSL [15] aims to enhance the positive distribution robustness by leveraging Distributionally Robust Optimization (DRO); AdvInfoNCE [38] employs
adversarial learning to enhance SL's robustness; Zhang et al. [20] suggests incorporating bias-aware margins in SI. to tackle popularity bias. Beyond these three types of losses, other approaches have also been explored in recent years. For example, Zhao et al. [63] introduces auto-loss, which utilizes automated machine learning techniques to search the optimal loss; Shi et al. [44] proposes LLPAUC to approximate Recall @ K metric. The main concerns with these losses are their lack of theoretical connections to ranking metrics like DCG, which may result in them not consistently outperforming the basic SL. Moreover, both auto-loss and LLPAUC require iterative learning, leading additional computational time and increased instability.
7 Conclusion and Limitations
In this work, introduce new family of loss functions, termed Pairwise Softmax Loss (PSL). PSL theoretically offers three advantages: 1) it serves as a better surrogate for ranking metrics with appropriate surrogate activations; 2) it allows flexible control over the distribution of the data contribution; 3) it can be interpreted as a specific BPR loss enhanced by Distributionally Robust Optimization (DRO). These properties demonstrate that PSL has greater effectiveness and robustness compared to Softmax Loss. Our extensive experiments across three testing scenarios validate the superiority of PSL over existing methods.
One limitation of both PSL and SL is inefficiency, as they require sampling relatively large number of negative instances per iteration. How to address this issue and improve the efficiency of these losses is an interesting direction for future research.
Acknowledgments and Disclosure of Funding
This work is supported by the Zhejiang Province JJanBingLLinggan+X Research and Development Plan (202
References
[1] Hycyoung Ko, Suyeon Lee, Yoonsco Park, and Anna Choi. A survey recommendation systems: recommendation models, techniques, and application fields. Electronics, 11(1):141, 2022.
[2] Shuai Zhang. Lina Yao, Aixin Sun, and Tay. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), 52(1):1-38, 2019.
[3] Feiran Huang, Zefan Wang, Xiao Huang. Yufeng Qian, Zhetao Li. and Hao Chen. Aligning distillation for coad-start item recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1147-1157, 2023.
10

---

[4] Feiran Huang, Zhenghang Yang, Junyi Jiang, Yuanchen Bei, Yijie Zhang, and Hao Chen. Large language model interaction simulator for cold-start item recommendation. arXiv preprini arXiv:2402.09176 2024.
[5] Tie- Yan Liu et al. Learning to rank for information retrieval. Foundations and Trends@ in Information Retrieval, 3(3):225-333 2009.
[6] Kalervo Järvelin and Jaana Kekäläänen. Ir evaluation methods for retrieving highly relevant documents. In ACM SIGIR Forum, volume 51, pages 243-250 ACM New York, NY. USA, 2017.
[7] Xiangkui Lu, Jun Wu, and Jianbo Yuan. Optimizing reciprocal rank with bayesian average for improved next item recommendation. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2236 2240, 2023.
[8] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering In Proceedings of the 26th international conference on world wide web, pages 173-182, 2017.
[9] Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR conference Research and Development in Information Retrieval, pages 355-364, 2017.
[10] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pages 452-461, 2009.
[11] Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, and Tianyu Qiu. On the effectiveness of sampled softmax loss for item recommendation. ACM Transuctions on Information Systems, 42(4): 1-26. 2024.
[12] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative or contrastive. IEEE transactions on knowledge and data engineering, 35(1):857-877 2021.
[13] Junkang Wu, Jiawei Chen, Jiancan Wu, Wentao Shi. Xiang Wang, and Xiangnan He. Understanding contrastive learning via distributionally robust optimization. Advances in Neural Information Processing Systems, 36, 2024.
[14] Sebastian Bruch, Xuanhui Wang, Michael Bendersky, and Marc Najork. An analysis of the softmax cross entropy loss for seaminngtto-rant with binary relevance. In Proceedings of the 2019 ACM SIGIR international conference theory of information retrieval, pages 75-78, 2019.
[15] Junkang Wu, Jiawei Chen, Jiancan Wu, Wentao Shi, Jizhi Zhang, and Xiang Wang. Bsl: Understanding and improving softmax loss for recommendation. In 2024 IEEE 40th International Conference on Data Engineering (ICDE). pages 816-830. IEEE, 2024.
[16] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41()):
1-39, 2023.
[17] Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin Xin, Liang Chen, Guli Lin, and Keping Yang. Autodebias: Learning to debias for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 21-30, 2021.
[18] Bohao Wang, Feng Liu. Jiawei Chen, Yudi Wu, Xingyu Lou, Jun Wang. Yan Feng, Chun Chen, and Can Wang. LIm4dsr: Leveraing large language model for denoising sequential recommendation. arXiv preprint 2xXv:2448,,82000 2024.
[19] Alexander Shapiro. Distributionally robust stochastic programming. SIAM Journal on Optimization, 27(4): 2258-2275, 2017.
[20] An Zhang, Jingnan Zheng, Xiang Wang, Yancheng Yuan, and Tat-Seng Chua. Invariant collaborative filtering popularity distribution shift. In Proceedings of the ACM Web Conference 2023, pages 1 1240-1251, 2023.
[21] Zihao Zhao, Jiawei Chen, Sheng Zhou, Xiangnan He, Xuezhi Cao, Fuzheng Zhang, and Wei Wu. Popularity bias is always evil: Disentangling benign and harmful bias for recommendation. IEEE Transactions on Knowledge and Data Engineering, 35(10):99009993. 2022.
[22] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong 7hang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development Information Retrieval, pages 639-648, 2020.
[23] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018
[24] Bohao Wang. Jiawei Chen, Changdong Li, Sheng Zhou, Qihao Shi, Yang Gao, Yan Feng, Chun Chen, and Can Wang. Distributionally robust graph based recommendation system. arXiv preprint 2xXiv:2402.1299/4 2024.
11

---

[25] Xiaoyuan Su and Taghi M Khoshgoftaar. A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009, 2009.
[26] Ychuda Koren, Robert Bell, and Chris Volinsky, Matrix factorization techniques for recommender systems. Computer, 42(8):30 37, 2009.
[27] Jiawei Chen, Junkang Wu, Jiancan Wu, Xuezhi Cao, Sheng Zhou, and Xiangnan He. Adap-T: Adaptively modulating emhedding magnitude for recommendation. In Proceedings of the ACM Web Conference 2023, pages 1085-1096 2023.
[28] Andreas Argyriou, Miguel González-Fierro, and Le Zhang. Microsoft recommenders: best practices for production-ready recommendation systems. In Companion Proceedings of the Web Conference 2020, pages 50-51, 2020.
[29] Zeshan Fayyaz, Mahsa Ebrahimian, Dina Nawara, Ahmed Ibrahim, and Rasha Kashef. Recommendation systems: Algorithms, challenges, metrics, and business opportunities. applied sciences, 10(21):7748, 2020.
[30] Thiago Silveira, Min Zhang, Xiao Lin, Yiqun Liu, and Shaoping Ma. How good your recommender system is? survey on evaluations in recommendation. International Journal of Machine Learning and Cybermetics, 10:813-38 831, 2019.
[31] Ahmed Rashed, Josif Grabocka, and Lars Sichmidt-Thienee A guided learning approach for item recom- mendation via surrogate loss learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 605-613, 2021.
[32] George Casella and Roger Berger. Statistical inference. CRC Press, 2024.
[33] Zhe Cao, Tao Qin, Tie- Yan Liu, Ming-Fenn Mong-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference Machine learning, pages
129-136, 2007.
[34] Johan I Ludwig William Valdemar Jensen. Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta mathematica, 30(1):175-193, 1906.
[35] Runtian Zhai, Chen Dan. Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In International Conference on Machine Learning, pages 12345-12355. PMLR, 2021.
[36] Sloan Nietert, Ziv Goldfeld, and Soroosh Shafice. Outlier-robust wasserstein dro. Advances in Neural Information Processing Syxtems, 36, 2024.
[37] Zhaolin Hu and Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online. 1(2)99 2013.
[38] An Zhang, Leheng Sheng, Zhibo Cai, Xiang Wang, and Tat-Seng Chua. Empowering collaborative filtering with principled adversarial contrastive loss. Advances in Neural Information Processing Systems, 36, 2024.
[39] Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, and Xiangnan Hc. Model-agnostic counterfac- tual reasoning for eliminating popularity bias in recommender system. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 1791-1800, 2021.
[40] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507 517,517,750 2016.
[41] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-hased recommen- dations styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on rexearch and development in information retrieval, pages 43-52, 2015.
[42] Eunjoon Cho, Seth A Myers, and Jure Leskovec. Friendship and mobility: user movement in location- based social networks. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1082-1090, 2011.
[43] Yelp. Yelp dataset. https: / //www//w. yelp. com dataset, 2018
[44] Wentao Shi, Chenxu Wang, Fuli Feng, Yang Zhang, Wenjie Wang, Junkang Wu, and Xiangnan He. Lower-left partial auc: An effective and efficient optimization metric for recommendation. arXiv preprint
arXiv:2403.0m 308444. 2024.
[45] Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui, Nguyen Quoe Viet Hung, and Hongzhi Yin. Xsimgel: To- wards extremely simple graph contrastive lcarning for recommendation. IEEE Transactions Knowledge and Data Engineering, 2023.
[46] Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: deep quadruplet network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 403 412, 2017.
[47] Emile Fiesler and Russell Beale. Handbook of neural computation. CRC Press, 2020.
12

---

[48] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. Latent relational metric learning via memory-based attention for collaborative ranking. In Proceedings of the 2018 world wide web conference. pages 729-739, 2018.
[49] Scott Deerwester, Susan T Dumais, George W Furnas. Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6); 391-407, 1990.
[50] Robert Bell, Yehuda Koren, and Chris Volinsky. Modeling relationships at multiple scales to improve accuracy of large recommender systems. In Proceedings the 13th ACM SIGKDD international conference Knowledge discovery and data mining, pages 95-104, 2007.
[51] Ychuda Koren. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 426-434, 2008
[52] Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: survey. ACM Computing Surveys, 1-37,, 2022.
[53] Chen Gao, Xiang Wang, Xiangnan He, and Yong Li. Graph neural networks for recommender system. In Proceedings of the Fifteenth ACM International Conference on Web Search and Duta Mining, pages 1623-1625, 2022.
[54] Thomas N Kipf and Max Welling. Semi-superiised classification with graph convolutional networks. arXiv preprint arXiv:1609.022 02900,0227.,0000
[55] Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng. and Tat-Seng Chua. Neural graph collaborative filtering In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, pages
2019.
[56] Hande Dong, Jiawei Chen, Fuli Feng, Xiangnan He, Shuxian Bi, Zhaolin Ding, and Peng Cui. On the equivalence of decoupled graph convolution network and label propagation. In Proceedings of the Web Conference 2021, pages 3651-3662, 2021.
[57] Jiancan Wu, Xiangnan Hc, Xiang Wang, Qifan Wang, Weijian Chen, Jianxun Lian, and Xing Xie. Graph convolution machine for context-aware recommender system. Frontiers of Computer Science, 16(6): 166614, 2022.
[58] Hao Chen, Yuanchen Bei, Qijie Shen, Yue Xu, Sheng Zhou, Wenbing Huang, Feiran Huang, Senzhang Wang, and Xiao Huang. Macro graph neural networks for online billion-scale recommender systems. In Proceedings of the ACM on Web Conference 2024, pages 3598 3608, 2024.
[59] Wenhui Yu and Zheng Qin. Graph convolutional network for recommendation with low-pass collaborative filters. In International Conference on Machine Learning, pages (0936-1094) PMLR, 2020.
[60] Huachi Zhou, Hao Chen, Junnan Dong, Dauchen Zha, Chuang Zhou, and Xiao Huang. Adaptive popularity debiasing aggregator for graph collaborative filtering. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 7-17, 2023.
[61] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive 2018.
coding, arXiv preprint 1807.00807.00
[62] Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan Hc, Liang Chen, Jianxun Lian, and Xing Xic. Self- supervised graph learning for recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 726-735, 2021.
[63] Xiangyu Zhao, Haochen Liu, Wenqi Fan, Hui Liu, Jiliang Tang, and Chong Wang. Autoloss: Automated loss function search in recommendations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3959-3967, 2021.
[64] R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009.
[65] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[66] Ruining He and Julian McAuley. Vbpr: visual bayesian personalized ranking from implicit feedback. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[67] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive sule-super learning. Technologies, 91)::, 2020.
[68] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint MnXiv:1412.6940 2014.
[69] Charles Dugas, Yoshua Bengio, François Belisle, Claude Nadeau, and René Garcia. Incorporating second- order functional knowledge for hetter option pricing. Advances in neural information processing systems, 13, 2000.
13

---

A Theoretical Proofs
A.1 Proof of Lemma 4.1
Lemma A.1 (Lemma4.) 4.1). If the condition
8(dui)) a (dwis) I exp(duaj)
(4.2)
satisfied for any dusz E [-1,1]] then PSL serves as a tighter DCG surrogate loss compared to SL.
Proof of Lemma 4.1. For any T > 0, Equation (4.2) indicates that
(dwws)) S oddus)) S exp(dwag)
(A.1)
which means of ( is tighter than exp(-) ( approximating 8(.). According to Equations (3.2) and (3.3) in Section 3, we conclude that PSL a tighter surrogate loss for DCG compared to SL. 0
A.2 Proof of Theorem 4.2
Theorem A.2 (Theorem 4.2). For each user and its positive item i, let P P
P(juu, u, i) be the uniform
distribution over I. Given robustness radius r O, consider the uncertainty set Q consisting of all perturbed distributions Q = Qiiu,i) i) satisfying: (i) Q is absolutely continuous w.r.t. P. Le., Q P: (ii) the KI. divergence between Q and is constrained by n, i.e,, (xI(Q1)) S n. Then, optimizing PSL is equivalent to performing DRO over BPR loss, i.e.,
min Ej-p.
log Ez~z [ (n)l
[]}
+ min EiNPa
sup By-QUla,) Dlog (ddss)) QEE
(x)(
CRPR.DRO (v)
(4.4)
where T(1) is temperature parameter controlled by n.
To prove Theorem 4.2, suffices to prove the following lemma:
Lemma A.3 (DRO under KI. divergence). Given the loss term E(x; 0) of input x and parameters O, for any robustness radius > 0, DRO under KL divergence is equivalent to optimizing a loss in the form of log ([(ex(()) )), i.e..
min sup Ez-olf(z; 0)] <> min {r (e-p-eee((x((((00)) +T) 8,r20
(A.2)
0 QEo
where the uncertainty set Q consists of all perturbed distributions Q constrained by KL divergence w.r.t. the original distribution P, i.e., Q IQ P: Dxl(Q]IP) s n)
Lemma A.3, which was first proposed by Hu and Hong [37] with a complex proof, gives a closed-fomm solution for DRO under KL divergence. Here we provide an elegant proof based the following general result about the ş-divergence DRO, which was first proposed by Shapiro [19].
Theorem A.4 (DRO under d-divergence, [19]). Consider the DRO problem d-divergence
D((QIIP) = I
dQ dp
dp
(A.3)
where <b IR + = } is a conver function such that ((11 = and 6()) = too
for any 1 < Then the inner maximization problem DRO, i.e. supqeq (e(=) 0)) with the
uncertainty set Q (Q P: D2(Q1)P) S n) is equivalent to the following optimization problem:
inf {Ez~p ([(x) - 14] + +m} T>0,μ1
(A.4)
where (1) = sup, fyr - [(x) is the Fenchel conjugate [64] for any convex function f: - TR.
Proof of Theorem A.4. Let the likelihood ratio L(x) - (d( then the inner maximization problem in DRO can be reformulated as
sup (Ba-r (Txx)((x;;)) (.-r(((((())))))) < [x-r( =1 L20
(A.5)
14

---

The Lagrangian of Equation (A.5) is
T, E
pL(x)] + + P
(A.6)
where r 2 0 and 1l are the Lagrange multipliers. Problem (A.5) a convex optimization problem.
One can easily check the Slater's condition [65] by choosing L(x) = thus the strong duality [65]
holds, and problem (A.5) is equivalent to the dual problem (A.7) of the Lagrangian (A.6):
inf sup T.) +20,0 1
(A.7)
Consider the inner maximization problem supı>0 CLL, T, in Equation (A.7), mn + μl is constant and can be ignored. By the theorem of interchange minimization and integration [64], we can interchange sup and expectation in Equation (A.7).Then supizo CLL, T, u)) can be reformulated
E
EgsP
sup ((<))(((x;))) ()) -=o((()))) LZD
(A.8)
The above problem can be rewritten by the Fenchel conjugate as
Eg (((=)) 41]
(A.9)
Thus, problem (A.7) is equivalent to
inf ((o)) 720,j
n)] + rn m1)
(A.10)
Finally, note that the condition in problem (A. can be relaxed to 0 without affecting the optimal value, thus problem (A.10) is equivalent to problem (A.4), which completes the proof. D
Lemma A.3 can be directly derived from Theorem A.4 as follows:
Proof of Lemma A.3. KL divergence is special case d-divergence with ((x) = x log x, and the Fenchel conjugate of TΦ is
(T))" sup {yr Txr log Tel/TT1
(A.11)
By Theorem A.4, the DRO problem under KL divergence equivalent
inf { (EEnP (Tel((=0) + Tn + 1} +>0,11
(A.12)
{ (EewP [ Te Te- n/T-1 + + + 11
inf ~>0,
We fix and solve the optimal value as
log Eno (
1
(A.13)
Therefore, by substituting the optimal fz in Equation (A.13) back to Equation (A.12), the original DRO problem equivalent
{ log Be-p [c(c:)11]]] + mn)
inf 0.520
(A.14)
This completes the proof.
Theorem 4.2 a direct consequence of Lemma A.3, when setting the loss term E(x; 0) as log a (dwss)
(i.e., the pairwise loss term in BPR loss), P as the uniform distribution over Z, Q as the perturbed distribution constrained by KI. divergence w.r.t. P. and T = T(1) as the optimal value of Lagrange multiplier in Equation (A.2). This completes the proof of Theorem 4.2.
15

---

A.3
Proof of the Bound Connections between PSL and Other Losses in Section 4.2
Proof of the Bound Connections in Section 4.2. We have proved in Lemma 4.1 that
log (C) + log Puul
1
log
Σ [d(dis)) JEL
(A.15)
Pu
iEPe
with any surrogate activation satisfying 8(duij) (ddus)) Furthermore, if two surrogate acti- vations 01, 02 satisfy a (dwi)) & 02 (dhus) for any duij E [-1, 1]. then the corresponding DCG surrogate losses satisfy the same inequality. Therefore, have
log DCG S LPSl S EsL. LAdvInfONCE
(A.16)
where the constant term is omitted for simplicity.
Finally, prove that BPR serves surrogate loss for DCG. Apply Jensen's inequality the RHS of Equation (A.15), we have
Σ iEP
log
Ed(dusi)i/+ JEI
log
Σ Σ (duis)) iEP
(A.17)
Pu
Pu
JET
The RHS of Equation (A.17) is just log (up - log [Pu| with the same surrogate activation o in BPR. Equation (A.17) indicates that for any surrogate activation o, the general PSL (including SL, BSL., and AdvInfoNCE) always better than BPR with the same a, i.e.,
log DCG S LPst. log LBpr
(A.18)
where the constant term is omitted for simplicity. This completes the proof.
16

---

B Experimental Details
B.1
Datasets
The six benchmark datasets used in our experiments are summarized in Table B.1. In dataset preprocessing, following the standard practice in Wang al. [55], we use 10-core setting [66], i.c., all users and items have at least 10 interactions, We also remove the low-quality interactions, such as those with ratings (if available) lower than 3. After preprocessing, we split the datasets into 80% training and 20% test sets. In IID and Noise settings, we further randomly split a 10% validation set from training set for hyperparameter tuning.
Table B.I: Statistics of datasets, All datasets cleaned by 10-core setting. If the dataset is used in both IID and OOD settings, the statistics below are provided for the IID setting.
Dataset
#Users
#Items
#Interactions
Density
Amazon-Electronie [40, 41]
13,455
8,360
234,521
0.00208
Amazon-CD [40, 41]
12,784
13,874
360,763
0.00203
Amazon-Movie [40, 41]
26,968
18,563
762,957
0.00152
Gowalla [42]
29,858
40,988
1,027,464
0.00084
Yelp2018 [43]
55,616
34,945
1,506,777
0.00078
Amazon-Boo [40, 41]
135,109
115.172
4,042,382
0,00026
The details of datasets are as follows:
Amazon [40, 41]: The Amazon dataset is large crawl of product reviews from Amazon'. The 2014 version of Amazon dataset contains 142.8 million reviews spanning May 1996 Jully 2014. We process four widely-used categories: Electronic, CD, Movie, and Book, with interactions ranging from 200K to 4M.
Gowalla [42]: The Gowalla dataset is a check-in dataset collected from the location-based social network Gowalla including IM users, 1M locations, and 6M check-ins.
Yelp2018 [43]: The Yelp dataset is a subset of Yelp's businesses, reviews, and user data, which was originally used in the Yelp Dataset Challenge. The 2018 version of Yelp dataset contains 5M reviews.
The detailed dataset constructions in IID, OOD and Noise settings follows:
randomly split from the original dataset. Specifically, the positive items of each user are split into 80% training and 20% test sets. Moreover, the training set is further split into 90% training and 10% validation sets for hyperparameter tuning. In the IID setting, the training and test sets are both long-tail.
IID setting [22]: In the IID setting, the test
OOD setting [20, 24, 39]: In the OOD setting. 20% test set is uniformly sampled (w.r.t. items) from the original dataset, while the 80% training set remains long-tail. The OOD setting used to simulate real-world online recommender systems. In order to avoid leaking information about the test set distribution, we do not introduce the validation set.
Noise setting [15]: In the Noise setting, the validation and test sets are split in the same way as the IID setting. However, we randomly sample 10% of the training set as the false negatives. In Noise training, the negative items will be sampled from the false negatives with a probability of as the negative noise, where p E {0.05, 0, 0.2, 0.3, 0.5} is a.k.a. the noise ratio.
All experiments are conducted on one NVIDIA GeForce RTX 4090 GPU and one AMD EPYC 7763 64- 64-Core Processor.
B.2 Metrics
This section provides a detailed explanation of the recommendation metrics used or mentioned in our experiments.
'nttps://www.n httpw//mazon.coom com/
sestpa://en.vikkieele http://wwiiipeiia...oo/iiii///// org/vikis/ooalla
'https://wwwyyll] com/
17

---

As stated in Section 5.1, we use Top- K recommendation [5]. It should be noted that for each user, the positive items in the training set will be masked and not included the Top-K recommendations when evaluating, and the ground-truth positive items Pa only consist of those in the test set. For convenience, we denote the set of hit items in the Top- K recommendations for user u as Hy { E
Pu. ( Σ } The recommendation metrics are defined as follows:
Recall K [29]: The proportion of hit items among Pu in the Top-K recommendations,
i.e., Recall @ K( = [H //Pul and the overall Recall K = Eu-ue (Recalleellllla (w)).
NDCG@ K [6]: The Discounted Cumulative Gain in the Top- recommendations (DCG@K) is defined as DCG@K (u) = Σve 1/ logg (1 + Tu (f)). Since the range of DCG@K will vary with the number of positive items IPal, we should consider to normalize DCG@K to (0. 1]. The Normalized DCG in the Top-K recommendations (NDCG@K) (cG@K())/DDCOK)) where DDCG&K K is the ideal DCG@K, i.e., IDCG@K (u) = E [K.,(K 1/ logg + i). The overall NDCG@ K (=-u(NDDC@K(())
_i=1
MRR@K K [7. 28]: The Mean Reciprocal Rank (MRR) is originally defined as the reciprocal of the rank of the first hit item. Here we follow the dcfinition of Argyriou et al. [28]'s to meet the requirements of multi-hit scenarios, i.c., MRR@K(u) = H. [1/Tu (i)], and the overall MRR@K Eu-u MRR@@ (u)).
B.3 Baselines
We reproduced the following losses as baselines in our experiments:
BPR [10]: A pairwise loss based on the Bayesian Maximum Likelihood Estimation (MLE). The objective of BPR is to learn a partial order among items, i.e., positive items should be ranked higher than negative items. Furthermore, BPR surrogate loss for AUC metric [10, 30). In our implementation, we follow He et al. [22]'s setting and use the inner product as the similarity function for user and item embeddings.
LLPAUC [44]: surrogate loss for Recall and Precision. In fact, LLPAUC is a surrogate loss for the lower-left part of AUC. In practice, LLPAUC is a min-max loss.
Softmax Loss (SL. [11]: A SOTA recommendation loss derived from the listwise MLE, i.e., maximizing the probability of the positive items among all items. The effectiveness of SL has been thoroughly reviewed in Sections 2 and 3. In fact, SL is special case of PSL with
surrogate activation a = exp(.).
AdvInfoNCE [38]: A DRO-based modification of SL. AdvInfoNCE tries to introduce adap- tive negative hardness to pairwise score duuj in SL (ef. Equation (3.1)). Zhang al. [38]'s original design, AdvInfoNCE, can be seen as a failure case of PSL with surrogate activation
a = exp( (ex(())) , as discussed in Section 4.2. In practice, AdvInfoNCE is a min-max loss.
BSL [15]: A DRO-based modification of SL. BSL applies additional DRO on the positive term in the pointwise form of SL.
The hyperparameter settings of each method are detailed in Appendix B.5.
B.4 Backbones
We implemented three popular recommendation backbones in our experiments, including
MF [26]: MF is the most basic but still effective recommendation model, which factorizes the user- item interaction matrix into user and item embeddings. All the embedding- based recommendation
models use MF as the first layer. Specifically, we set the embedding size d = 64 for all settings, following the setting in Wang et al. [55].
LightGCN [22]: LightGCN is an effective GNN-based recommendation model. LightGCN performs graph convolution on the user-item interaction graph, as aggregate the high-order interactions. Specifically, LightGIN simplifies NGCF [55] and only retains the non-parameterized graph convolution operator. In our experiments, we set the number of layers as which aligns with the original setting in He et al. [22].
XSmGCL [45]: XSimGCL is a novel recommendation model based on contrastive learning [12, 67]. Based on 3-layers LightGCN, XSimGCI. adds random noise to the output embeddings of cach layer, and introduces the contrastive learning between the final layer and the (" layer,
18

---

i.e., adding an auxiliary InfoNCE Joss [61] between these two layers. Following the original Yu et al. [45]'s setting, the modulus of random noise between each layer is set as 0.1, the contrastive layer 1" (where the embedding layer is 0-th layer), the temperature of InfoNCE is set as 0.1, and the weight of the auxiliary InfoNCE loss is set as 0.2 (except for the Amazon-Flectrone dataset, where the weight is set as 0.05).
B.5 Hyperparameters
B.5.1
Hyperparameter Settings
Optimizer, We use Adam [68] optimizer for training. The learning rate (Ir) searched in
{10-1.10 10 1003), except for BPR, where the Ir is searched in f10 10-3, 2,13,,, 3.10-1] The weight decay (wd) searched in {0, 10- 10-5, 10-0). The batch size is set 1024, and the number of epochs is set as 200. Following the negative sampling strategy in Wu et al. [15], we uniformly sample 1000 negative items for each positive instance training.
Loss. The hyperparameters of each loss are detailed as follows:
BPR: No other hyperparameters.
LLPAUC: Following Shi al. [44]'s setting, the hyperparameters and 3 E {0.01, 3.13 are searched.
0.3, 0.5,
Softmax Loss (SL): The temperature (0.005, 0.025, 0.05, 0.1, 0.25) is searched.
AdvInfoNCE: The temperature is searched in the space SL. The other hyperparameters are fixed as the original setting in Zhang et al. [38]. Specifically, the negative weight is set as 64, the adversarial learning will be performed every 5 epochs, with the adversarial learning rate as
x 10-5
BSL:
The temperatures 71: for positive and negative terms are searched in the same space as SL,
respectively.
PSL: The temperature r is searched in the same space as SL.
B.5.2 Optimal Hyperparameters
The hyperparameters we search include the learning rate (Ir), weight decay (wd), and other hyperpa- rameters: {a. 3) for LLPAUC, {r} for SL, AdvInfoNCE, and PSL, (Ti, )2) for BSL.
IID optimal hyperparameters. Table B.2 shows the optimal hyperparameters of IID setting, including four datasets (Amazon-Bone Book,, Enazzon-Electroiic, Amazon-Movie Gowalla) and three backbones (MF, LightGCN, XSimGCL).
OOD optimal hyperparameters. Table B.3 shows the optimal hyperparameters of OOD setting MF backbone, including four datasets (Amazon-CI, Amazon-Flectrrom Gowalla, Yelp2018).
Noise optimal hyperparameters. The Noise setting uses the optimal hyperparameters of III setting, as listed Table B.2. We compare the performance of each method under different noise ratios p E {0.05, 0.1, 0.2, 0.3, 0.5) J on MF backbone and four IID datasets (Amazon-Book Amazon- Electronic, Amazon-Movie Gowalla).
19

---

Table B.2: Optimal hyperparameters of IID setting.
Amazon-Book
Amazon-Electroon
Model
Loss
Ir
others
nd
others
BPR
10-4
10
10
LLPAUC
10
[0.7, 0.D1]
10-1
[0.5, 0.01]
AdvInfoNCE
10-2
[0.05]
10-1
[0.1]
{0.025)
10
MF
BSL
10-1
(0.25, 0.025)
10-1
0
(0.25. 0.1]
PSL tanh
10
[0,025)
10
0
[0.1]
PSI. alan
10
[0.025]
10 -2
[0.1]
PSL-relu
[0.025)
10
[0.1]
BPR
10
10
10
LLPAUC
1
[0.7, 0.01)
10 -1
10.5. 0.011
AdvinioNCE
10
[0.05]
10-2
[0.1]
SL.
10
[0.025]
10
[0.1]
LighIGCN
BSI.
10
(0.25, 0.025)
10
10.1, 0.1)
PSL-tank
10
(0.025)
10
0
[0.1]
PSL. alan
10
[0.025]
10
[0.1]
PSI-rrlu
{0.025)
10 22
[0.1]
BPR
10
10
[0.7, 0.01]
10-1
[0.3, 0.11]
AdvinioNCE
[0.05]
(0.1)
SL
10
[0,025]
10
[0.1]
XSimGCL
BSL
10
(0.025, 0,025]
10
0
[o.05. 0.1]
PSI.
10
[0.025]
10
[0.1]
(0.025)
[0.1]
PSL-relu
(0.025)
10
[0.1]
Amazon-Mowt
Gowalla
Model
Loss
BPR
10-3
10-3
10-6
[0.7, 0.01)
10
[0.7, 0.011
AdvInfoNCE
(0.05)
10
{0.05)
SL
10-1
[0.05]
10-1
O
(0.05]
BSL
10-2
(0.25, 0.05)
10
O
[0.1. 0.05)
PSL. lanh
10
(0.05)
10
[0.05]
PSL-atan
(0.05)
10
(0.05)
PSL-relu
10-1
[0.05]
10-1
(0.05)
BPR
10
10
10-1
[0.7, 0.01)
10
[0.7, 0.01
AdvInfoNCE
10-1
[0.05)
10-1
[0.05]
SL
10
(0.05)
10
0
(0.05)
LightGCN
BSL
10
(0.025. 0.05)
10
0
(0.025. 0.05)
PSL lank
10
[0.05]
10-1
(0.05]
[0.05]
10
[0.05]
PSI-relu
[0.05)
10
[0.05]
10
10-4
LLPAUC
(0.3, 0.013
10 1
[0.7. 0.01
AdvinIoNCE:
10
[0.05]
10
(0.05)
SL.
10-2
(0.05)
10
0
[o.05]
XSimCCL
BSI.
10
(0.025, 0.05)
10
(0.025, 0.05)
[0.1]
[0.05]
PSL-atan
(0.05)
10
(0.05]
PSI-relu
10-2
[0.1]
10
[0.05]
20

---

Table B.3: Optimal hyperparameters of OOD setting.
Amazon-CU CD
Electronie
Model
Loss
BPR
10-2
10-2
10-6
10"
0.01)
10-1
[0.7, 0.17
AdvInfoNCE
[0.05)
[0.05)
SL
[0.05]
10
(0.05)
MF
BSL
10-1
(0.05, 0.05)
10-1
(0.1. 0.05
PSI-tanh
10 1
(0.05)
10
(0.05)
PSt-atan
(0.05)
(0.05)
rel-
[0.05)
(0.05)
Gowalla
Yelp2018
Model
Loss
BPR
10-3
10-3
LLPAUC
10"
(0.7, 0.01)
(0.7, 0.01)
(0.7.0.01)
AdvInfoNCE
(0.05)
10
(0.05)
SL
10-1
[0.025
10-1
(0.05]
MF
BSL
10-1
{0.25, 0.025]
10-1
(0.1. 0.05
tanh
10
[0.025]
10
[0.025]
PSL-atan
[0.025]
[0.025]
PSL-telu
10
[0.025]
10-1
[0.025]
21

---

C Supplementary Experiments
C.1
Noise Results
The Recall 20 and NDCG@ 20 results under Noise setting on four datasets Hamazon-Bookk Amazon- Electronic, Amazon-Movis Gowalla) are shown in Figures C. to C.4.
Amazon-Bal BoozonnBBookkkoo 2ecall@:20
Amazon-Boo NDCG@20
10%
0.09
10%
0.11
8%
Recalk
0.10
NDCG@20
696
Imp. %
Imp. %
6%
0.09
AdviafaNCE
Imip
0.07
AdvinIoNCE
4%6
BSL
BSL
PSL-atam
0.06
PSL-utan
PSI-relu
PSI-relu
2%
0.07
SL
09
SL.
0%
0.05
0.1
0.2
0.3
0.5
0.05
0.1
0.2
0.3
0.5
Noise Ratio p
Noise Ratio p
Amazon-Book (Recall@20)
(b) amazon-Book (NDCG@20)
Figure C.I: Noise results on Amazon- Book dataset.
Amazon-Elees Electronic Recall@20
Ammazon-Electroii NDCG@20
0.065
12%
0.040
14%
0.060
12%
Recall::20
0.055
8%
NDCG@20
0.035
10%
Imp.%
Imp %
8%
0.050-
AdviafaNCE
6%
Imp.
AdvinfoNCI
6%
BSL
4%
0.030
BSL
0.045
PSL-atan
PSL-olan
4%
PSL-cclu
2%6
PSL-colu
0.040
SL
0.025
SL
0.05
0.1
0.2
0.3
0.5
0.05
0.2
0.5
Noise Ratio p
Noise Ratio p
(a) Amazoo-Ellectonic (Recall (0)
(b)
(NDCG@20)
Figure C.2: Noise results on
dataset.
22

---

Amazon-Movie Recall@820
Amazon-Movie NDCG@20
9%
0.070
0.095
8%
Recall@:20
0.090
6%
NDCG
896
0.085
Imp.%
5%
0.060
Imp.
6%
%
0.080-
AdviafoNCE
4%6
3%
0.055
AdvlofoNCE
BSL
PSL-atan
PSL-relu
2%
2%6
0.070
1%
0.050
0%
0%
0.05
0.1
0.2
0.3
0.5
0.05
0.1
0.2
0.3
0.5
Noise Ratio
Noise Rutio
(a) Amazon-Movie (Recall (0)
(b) Amazon-Movie (NDCG@20)
Figure C.3: Noise results on Amazon-Movie dataset.
Gowalla Recall@220
Gowalla NDCG@20
0.11
12%6
10%
0.14
10%
Recall@:20
NDCG
0.10
0.13
Imp %
8%
%
0.12
AdviafoNCE
Imp.
0.09
AdvlafoNCE:
6%
4%6
BSL
4%
0.11
PSL-celu
296
0.08
PSL-celu
0%
0%
0.05
0.1
0.2
0.3
0.5
0.05
0.1
0.2
0.3
0.5
Noise Ratio
Noise Ratio
(a) Gowalla (Recall 20)
(b) Gowalla (NDCG@20)
Figure C.4: Noise results Gowalla dataset.
23

---

C.2 PSL-softplus Results
BPR uses Softplus [69] as logo, i... (dui)) ) exp(duaj) + 1, which is looser than SL. That this surrogate activation is not a suitable choice for PSL. We call this PSL variant as PSL-softplus.
In this section, we conduct experiments evaluate the performance of PSL-softplus with surrogate activation a (dmis) = exp(dui)) + 1. The IID, OOD, and Noise results of PSI--oftplus are shown in Tables C.4 and C.5, Figures C.5 C.8, respectively. Results demonstrate that PSL.-softplus is inferior to SL. and three PSLs in all settings. This confirms our claim - the choice of surrogate activation a is crucial, and an unreasonable or intuitive design will decrease the accuracy.
Table C.4: IID results of PSL-softplus. The results of SL, PSL-tanh, PSL-atan, and PSL-relu have been listed in Table 1. The blue-colored results are better than PSL-softplus.
Amazon-Book
Amazon-Flectronic
Amazon-Move
Gowalla
Model
Loss
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
SL
0.1559
0.1210
0,0821
0,0529
0.1286
0.0929
0.2064
0,1624
PSL-tanb
0.1567
0.1225
0.0832
0.0535
0.1297
0.0941
0.2088
0.1646
PSL-atan
0.1226
0.0832
0.1296
0.0941
0.1646
MF
rel-
0.1669
0.1227
0.0838
0.0541
0.1299
0.0945
0.2089
0.1647
PSL-softplus
0.1636
0.1149
0.0826
0.0522
0.1280
0.0919
0.2053
0,1613
St.
0.1567
0.1220
0.0821
0,0526
0.1304
0.0941
0.2068
0.2628
PSL-tanh
0.1575
0.1233
0.0825
0.0532
0.1300
0.0947
0.2091
0. 0.1688
LightGCN
ata
0.1575
0.1233
0.0825
0,0532
0.1300
0.0948
0.2091
0. 1648
PSL-relu
0.1575
0.1233
0.0830
0.0536
0.1300
0.0953
0,2086
0. 1648
PSL-softplus
0.1536
0,1152
0,0814
0,0514
0.1296
0,0932
0.2053
0.1613
SL
0. 0.1449
0.1207
0.0772
0.0490
0.1255
0.0905
0,2005
0,1570
PSL-tanb
0.1567
0.1225
0.1308
0.1591
XSInGCL.
PSL-atan
0.1565
0.1225
0.0792
0,0502
0.1253
0.0917
0.2035
0.1591
rSt--relu
0.1228
0.0801
0.1313
0.0935
PSL-soliplus
0.1161
0.1242
0.0894
0.1557
Table C.5: OOD results of PSL-softplus. The results of SL, PSL-tanh, PSL-atan, and PSL-relu have been listed in Table 2. The blue-colored results are better than PSL-softplus.
Amazon-CD
Amazon- Amazon-Flectronte
Gowalla
Yelp2018
Loss
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
S1.
0.1
0.0815
0.0230
0.0142
0.1006
0.0737
0.0349
0.0224
PSL-tanh
0,1202
0.0834
0.0239
0.0146
0,1013
0.0748
0.0357
0.0228
PSL-stan
0,1202
0.0835
0.1013
0.0228
rel.
0,1203
0.0839
0.0241
0.0149
0,1014
0.0752
0.0358
0.0229
Pot.ssofplls
0.9
0.0799
0.0232
0.0139
0.0909
0.0665
0.0346
0.0222
Amazon Book Rccall@20
Amazon-Bood NDCG@20
0.09
0.11
0.08
0.10
NDCG@20
Recall@:20
0.09
0.07
0.08
0.06
PSL-relu PSL-sofiplus
PSL-relu PSL-sofiplus
0.07
0.05
SL
SL.
0.06
0.05
0.1
0.3
0.5
0.05
0.2
0.3
0.5
Noise Ratio p
Noise Ratio p
(a) Amazon-Book Recalll@200
(b) Amazon-Book (NDCG@ 20)
Figure C.5: Noise results of PSL-softplus on Amazon-Boo BBok dataset.
24

---

Aimazon-llectroni Recall@:20
Ainazon-Electroon NDCG@20
0.065
0.060
Recall@200
0.055
NDCG@220
0.035
0.050
0.030
0.045
PSL-telu
PSL-relu PSL-softplus
0.040
PSL-sofiplus
0.025
0.035
0.05
0.1
0.2
0.5
0.05
0.1
0.2
0.3
0.5
Noise Ratio p
Noisc Ratio p
(a) Elletr--leccronic (Recall (0)
(b) Amazon- Eleecronie (NDCG@20
Figure C.6: Noise results of PSL-softplus on Amazon-Electroni Electronic dataset.
Amazon-Movie Recall@220
Amazon-Movie NDCG@20
0.070
0.09
0.065
Recall@:20
NDCG@20
0.08
0.055
PSL-relu
PSL-relu PSL-softplus
0.07
PSL-softplus
0.050
SL
SL
0.045
0.05
0.1
0.3
0.5
0.05
0.1
0.2
0.3
0.5
Noise Ratio p
Noise Ratio p
(a) Amazon-Movie (Recall@20))
(b) Amazon-Movie (NDCG@20)
Figure C.7: Noise results of PSL-sofiplus on Amazon-Movie dataset.
Gowalla Recall@:20
Gowalla NDCG@20
0.15
0.11
0.14
Recall@200
0.13
NDCG@:20 @2:20
0.10
0.12
0.09
0.11
PSl-
0.08
PSL-relu
PSL-sofiplus
PSL-softplus
0.10
SL.
SL.
0.07
0.05
0.1
0.2
0.3
0.5
0.05
0.1
0.2
0.3
0.5
Noise Ratio
Noise Ratio
(a) Gowalla (Recall 20)
(b) Gowalla (NDCG@ 20)
Figure C.8: Noise results of PSL-softplus on Gowalla dataset.
25

---

C.3 Comparisons of Two Extension Forms
In this section, we compare the two different extension forms from SL PSL, i... outside form (duas) ) and inside form (due) /T). As discussed in Section 4.2, the outside form scales in the value domain, while the inside form scales in the definition domain. Therefore, the inside form will lead certain drawbacks: the condition (4.2) must be satisfied over the entire duij R to ensure tighter DCG surrogate loss (cf. Lemma 4.1), which is hard to achieve; 2) the value of a (dwig /T) and its gradient may be quickly exploded when T O, the range of duij /T is hard control, which may cause numerical instability.
To empirically compare the above two extension forms, conduct experiments MF backbone and four IID datasets. Specifically, since there exists serious numerical instability, we expand the range of to (0.005, 0.025, 0.05, 0, 0.25, 0.5, 1.0) for the inside form, where the outside form remains the same search space - E {0.005, 0.025, 0.05, 0.1, 1,1, 0.25). The results are shown Table C.6, demonstrating that the outside form is superior the inside form in all cases.
Table C.6: Extension forms comparisons on MF under IID setting. The blue-colored results better than the counterpart.
Amazon-Boon
ComazonEElectrang
Amazon-Movie
Gowalla
Form
Loss
Recall
NDCG
Recall
NDCG
Recall
NDCG
Recall
NDCG
PSL-tamh
0.1567
0.1225
0.0832
0.0535
0.1297
0.0941
0.2088
0.1646
(duss)) //r
PSI Jatan
0.1567
0.1226
0.0832
0.0535
0.1296
0.0941
0.2087
0. 16466
0.1569
0.0541
0.2089
0.1647
PSL-tanh
0.1415
0.0767
0.0494
0.0590
0.1956
0.1507
o(dui/7)
PSL-atam
0.0307
0.0213
0.0453
0.0268
0.0363
0.0247
0.0982
0.0727
0.0452
0.1732
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract accurately conclude this paper's contri- bution, including theoretical findings and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work is discussed in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Full theory assumptions and proofs are provided in Section to Section 4, and Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the experi- mental results in Section 5, Appendices B and C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The data and code are provided in IR- Benchmark.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ guuidee/Coeeuubmnii
olicyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ ctc//ublic/gguui Seeiidde//oodiiiii ss hearbbimsssoooooiiiiiicco onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the experimental details are included in Section 5, Appendices and C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The p-value of the main performance is reported in Tables and 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sufficient information on the computer resources is provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ /tte:r///eeu...//eee ct/ cec/puulcc//eeeeeeeeeee ctc csGuidel ines?
Answer: [Yes]
Justification: The research conducted in the paper conform in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work improves the effect of recommendation loss and has no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The original paper that produced the codes and datasets cited in the paper without any omissions.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The assets introduced in the paper well documented and the details of the code are included.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "jwSNuak50X", "reviewer": "Reviewer_VFmg", "summary": "This paper aims to investigate the effectiveness of softmax loss in the recommendation model. To overcome the limitation of SL, the authors propose a new pairwise softmax loss (PSL). Based on the analysis, the authors argue that replacing exponential function with other active functions can benefit the ranking performance. Experiments on both IID, OOD, and noise datasets prove the effectiveness of the proposed PSL.", "questions": "1. How the SPL affects other metrics, for example, AUC is important in pointwise click scenarios.\n\n2. In experiments, only results of NDCG@20 are given. Results on more cutoffs should be given in the recommendation.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The analysis of softmax loss on the DCG metric is impressive and reasonable. The pair-wise softmax combines BPR loss and activation operation, which is easy to incorporate into the current model.\n2. The technique is sound based on well-motivated according to the analysis.\n3. The experiment design is reasonable and follows the related work, which verifies the effectiveness.", "weaknesses": "1. Overall, the softmax loss is utilized in training and the test stage, which means we can utilize the model's output as the probability. The probability can be used in downstream tasks, such as ECPM calculation in online advertisements. So, replace the operation related to ranking metrics and practical use.\n2. The distinction between AUC, Recall, and other metrics should be paid more attention. The motivation behind SPL is prioritized to focus on the  DCG metrics. \n3. The experiment design is reasonable, but the analysis needs to be further enhanced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `VFmg`\n\nDear Reviewer  `VFmg`,\n\nWe sincerely appreciate your recognition of our work. Your detailed comments are highly valued, and the questions you raised are both interesting and practical. Below are our detailed responses to the Weaknesses (**W**) and Questions (**Q**):\n\n> **[W1] Overall, the softmax loss is utilized in training and the test stage, which means we can utilize the model's output as the probability. The probability can be used in downstream tasks, such as ECPM calculation in online advertisements. So, replace the operation related to ranking metrics (may harm the) practical use.**\n\n**[Response]** Thank you for the insightful comments. It is not straightforward to derive the probability from PSL, but we can construct it by leveraging classic *Plackett-Luce ranking model* [a1]. The Plackett-Luce ranking model establishes relationships between permutation probabilities with the prediction scores, and SL was firstly introduced in the recommendation community from this perspective [a2].\n\nTo elaborate, in the Plackett-Luce ranking model, the probability that an item occupies the top position (i.e., the probability that it is the item the user likes most) can be expressed as:\n\n$\nP_s(i)=\\dfrac{\\phi\\left(s_i\\right)}{\\sum_{j \\in \\mathcal I} \\phi\\left(s_j\\right)}\n$\n\nwhere $s_i$ denotes the *Plackett-Luce score* of an item $i$. Function $\\phi(\\cdot)$ can be any increasing and strictly positive function. In our PSL, we express $s_i=f(u,i)-b_{ref}$ and $\\phi(\\cdot)=\\sigma(\\cdot)^{1/\\tau}$. Here a reference term $b_{ref}$ is introduced, which can be set as the prediction value $f(u,j)$ of a specific item $j$ (e.g., the top-$K$ th item, which serves as a threshold score for an item being considered positive). The introduction of $b_{ref}$ is important as PSL optimizes the relative magnitudes of $f(u,i)$ from a pairwise perspective, rather than their absolute values. Introducing a reference term can enhance the stability of the results.\n\nSuch Plackett-Luce model possesses some interesting properties:\n\n- When we set $\\phi(\\cdot)=\\exp(\\cdot)$, the probability $P_s(i)$ degenerates to the Softmax form.\n- The objective of PSL can be viewed as optimizing the log-likelihood of the Plackett-Luce model with specific reference terms. That is, for each positive user-item pair $(u,i)$, we have:\n  $\\mathcal{L} _ {\\text{PSL}}(u,i) = \\log\\left(\\sum _ {j \\in \\mathcal{I}} \\sigma(d_{uij})^{1/\\tau} \\right) = -\\log\\left( \\dfrac{\\sigma{(f(u,i)-b_{ref}})^{1/\\tau}} {\\sum _ {j \\in \\mathcal{I}}{\\sigma(f(u,j)-b_{ref})^{1/\\tau}}} \\right) = -\\log P_s(i)$\n  , where we set $b_{ref}=f(u,i)$.\n\n**[Reference]**\n\n[a1] The analysis of permutations, Journal of the Royal Statistical Society Series C: Applied Statistics, 1975.\n\n[a2] Learning to rank: from pairwise approach to listwise approach, ICML'07\n\n> **[W2] The distinction between AUC, Recall, and other metrics should be paid more attention. The motivation behind PSL is prioritized to focus on the DCG metrics.**\n>\n> **[Q1] How the PSL affects other metrics, for example, AUC is important in pointwise click scenarios.**\n\n**[Response]** Thank you for the constructive comments. This work prioritizes DCG metric due to its widespread application in evaluating model performance within industrial RS. Furthermore, optimizing DCG remains a challenging problem. We appreciate the reviewers' suggestions to emphasize other metrics. Consequently, we have conducted further experiments to assess the performance of our PSL model using other metrics, including Precision, MRR, and AUC, detailed in the subsequent Table A1, while Recall has been reported in the Table 1 in our manuscript. From these tables, we observe that PSL consistently outperforms other baselines across these varied metrics.\n\n**Table A1.** Precision@20, AUC, and MRR@20 on IID datasets and MF backbone.\n|Gowalla|Precision@20|AUC|MRR@20|Electronic|Precision@20|AUC|MRR@20|\n|-|:-:|:-:|:-:|-|:-:|:-:|:-:|\n|SL|0.0637|0.9666|0.1870|SL|0.0156|0.8004|0.0702|\n|AdvInfoNCE|0.0638|0.9667|0.1872|AdvInfoNCE|0.0156|0.8004|0.0685|\n|BSL|0.0638|0.9635|0.1881|BSL|0.0156|0.8003|0.0686|\n|PSL-tanh|0.0646|**0.9683**|**0.1886**|PSL-tanh|0.0158|**0.8074**|0.0704|\n|PSL-relu|**0.0647**|0.9674|0.1885|PSL-relu|**0.0159**|0.8059|**0.0718**|\n|PSL-atan|0.0646|**0.9683**|**0.1886**|PSL-atan|0.0158|0.8071|0.0704|\n\n> **[W3] The experiment design is reasonable, but the analysis needs to be further enhanced.**\n\n**[Response]** Thank you for highlighting this aspect. We will augment our experiment section with more comprehensive analyses, including more detailed comparisons of baseline performances, more analyses across three distinct scenarios, and more analyses of the impact of noise on different losses, etc.\n\n> **[Q2] In experiments, only results of NDCG@20 are given. Results on more cutoffs should be given in the recommendation.**\n\n**[Response]** Thank you for your constructive suggestions. Table A2 present results of our PSL in comparison with other baselines across various NDCG@$K$ metrics. These results consistently demonstrate that our PSL model outperforms the baselines, thereby validating its effectiveness and robustness.\n\n**Table A2.** NDCG@$K$ on IID datasets and MF backbone, $K \\in [5, 10, 20, 50]$.\n|Gowalla|NDCG@5|NDCG@10|NDCG@20|NDCG@50|Electronic|NDCG@5|NDCG@10|NDCG@20|NDCG@50|\n|-|:-:|:-:|:-:|:-:|-|:-:|:-:|:-:|:-:|\n|SL|0.1323|0.1417|0.1624|0.1994|SL|0.0352|0.0429|0.0529|0.0696|\n|AdvInfoNCE|0.1325|0.1420|0.1627|0.1997|AdvInfoNCE|0.0343|0.0422|0.0527|0.0699|\n|BSL|0.1329|0.1423|0.1630|0.2001|BSL|0.0345|0.0425|0.0530|0.0694|\n|PSL-tanh|0.1343|0.1439|0.1646|**0.2013**|PSL-tanh|0.0357|0.0434|0.0535|0.0702|\n|PSL-relu|**0.1345**|**0.1440**|**0.1647**|0.2012|PSL-relu|**0.0362**|**0.0439**|**0.0541**|**0.0705**|\n|PSL-atan|0.1343|0.1439|0.1646|**0.2013**|PSL-atan|0.0357|0.0434|0.0535|0.0702|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eRVywdIAgp", "reviewer": "Reviewer_kuk6", "summary": "The authors re-examine the connection between the Softmax Loss (SL) and the evaluation metric Discounted Cumulative Gain (DCG), \nhighlighting the inadequate tightness of SL as a surrogate loss for DCG. \nThey propose minimal yet effective modifications (PSL-tanh/relu/atan) based on the pair-wise formulation of SL,\nshowing provable tightness and controllable weight distributions.", "questions": "Why does BSL only keep on par with SL in both IID and OOD settings? \nIt appears that the connection to Distributional Robust Optimization (DRO) determines the performance in OOD settings, \nbut it is not the case for BSL?", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors provide a unified framework that helps to design a tighter and stronger DCG surrogate loss.\n2. Extensive experiments have been conducted in IID, OOD, and Noise settings. \nSpecifically, the proposed PSL-relu loss shows superior performance in both OOD and Noise settings.", "weaknesses": "1. In the IID setting, PSL demonstrates only marginal improvement even when compared to the standard Softmax Loss (SL).\n2. The authors argue that SL is highly sensitive to noise, while PSL with an appropriate activation function can mitigate this sensitivity. \nI would expect that as the noise ratio increases, an activation function with a moderate weight distribution should perform better. \nIn other words, PSL-tanh/atan shall be a better choice to SL/PSL-relu if the noise ratio is high. \nHowever, this is not the case according to Figures C.1-4, in which PSL-relu consistently outperforms the other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `kuk6`\n\nDear Reviewer `kuk6`,\n\nWe sincerely appreciate your recognition of our theoretical contributions and experimental setup. Below are our detailed responses to the Weaknesses (**W**) and Questions (**Q**) you have raised:\n\n> **[W1] In the IID setting, PSL demonstrates only marginal improvement even when compared to the standard Softmax Loss (SL).**\n\n**[Response]** Thank you for raising this concern. Admittedly, the improvements of PSL over SL in IID setting are not particularly large, typically ranging from 1% to 3.5%. But we would like to highlight the following aspects:\n\n- The improvements of PSL are **consistent** across multiply datasets, backbones and metrics. This consistency demonstrates the effectiveness and generalizability of PSL, while some existing advanced losses like AdvInfoNCE and LLPAUC may performs worse than SL in some cases. \n- The improvements of PSL are **significant** -- we have conduct statistically tests with $p < 0.05$ to validate these improvements are statistically significant.\n- PSL exhibits much stronger **robustness** than SL with the distribution shifts or noises. From Table 2 and Figure 2, we observe that PSL shows notable improvements over SL, sometimes reaching improvement levels of 5% and 10% in OOD and noisy settings, respectively. Given that distribution shifts and noises are common in practical RS, we anticipate that PSL could achieve impressive performance in practical.\n- The modification required to implement PSL over SL is **minimal and practical**. PSL only necessitates a simple replacement of the activation function without introducing additional parameters or hyperparameters, making it highly practical for industrial applications. In contrast, losses like AdvInfoNCE and BSL require more parameters and hyperparameters, necessitating higher training costs and extensive tuning efforts.\n\nOverall, we believe that **PSL can serve as a superior alternative to SL in practical settings, given its greater effectiveness and robustness, while not incurring extra implementation complexity or hyperparameter tuning efforts.**\n\n> **[W2] The authors argue that SL is highly sensitive to noise, while PSL with an appropriate activation function can mitigate this sensitivity. I would expect that as the noise ratio increases, an activation function with a moderate weight distribution should perform better. In other words, PSL-tanh/atan shall be a better choice to SL/PSL-relu if the noise ratio is high. However, this is not the case according to Figures C.1-4, in which PSL-relu consistently outperforms the other methods.**\n\n**[Response]** Thanks for your insightful question. We have indeed observed that PSL-ReLU achieves slightly better performance than PSL-Tanh/Atan in high-noise scenarios. The reasons may stem from the following two aspects:\n\n- The disparity in the weights of noisy data between PSL-ReLU and PSL-Tanh/Atan is not as pronounced as the disparity between PSL-ReLU and SL. The following Table A1 presents the weights ($\\nabla_d \\mathcal{L}$) of those extremely noisy instances where $d_{uij}=1$ across different losses. The absolute values between PSL-ReLU and PSL-Tanh/Atan are not so large as those between PSL-ReLU with SL.\n- PSL-ReLU employs the widely-adopted linear activation function, ReLU, which has been shown to be more easily optimized and exhibits greater numerical stability than the non-linear Tanh/Atan activation functions.\n\nBoth of these aspects contribute to the results. it appears that the second aspect outweighs the first, and PSL-ReLU exhibits slightly better performance even in highly noisy settings.\n\n**Table A1.** Weights ($\\nabla_d \\mathcal{L}$) of extremely noisy instances with $d_{uij}=1$ across different losses.\n\n| Loss                   |  SL  | PSL-ReLU | PSL-Tanh | PSL-Atan |\n| ---------------------- | :--: | :------: | :------: | :------: |\n| $\\nabla_d \\mathcal{L}$ | 742  |    80    |    20    |    25    |\n\n> **[Q1] Why does BSL only keep on par with SL in both IID and OOD settings? It appears that the connection to Distributional Robust Optimization (DRO) determines the performance in OOD settings, but it is not the case for BSL?**\n\n**[Response]** Thanks for raising this concern. BSL enhances SL by incorporating Distributionally Robust Optimization (DRO) on positive samples. However, this approach exhibits certain limitations:\n\n- BSL adopts a **pointwise** rather than a pairwise perspective, treating positive and negative instances separately. Given that ranking metrics depend on pairwise comparisons (as discussed in Section 3), this approach disrupts the theoretical alignment with ranking metrics, potentially rendering BSL an unsuitable surrogate for the DCG loss.\n- In typical RS, positive samples are typically **sparse**, with a user often having fewer than 15 positive items. BSL applies KL-divergence-based DRO on these positive instances. Consequently, the support of the perturbed distributions within DRO is confined to these limited positive items. This means the set of uncertainty distributions used in DRO lacks flexibility and may not adequately cover the test distribution, thus compromising the robustness.\n\nGiven these two drawbacks, BSL might outperform SL, but the improvement is not substantial. In fact, based on our experience on hyperparameter tuning, tuning the hyperparameters to align BSL closely with SL often yield optimal performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HsuLqGMQ73", "reviewer": "Reviewer_ufck", "summary": "From the unique characteristics of recommendation systems (RS), this paper highlights the incompatibility of the exponential function in the softmax loss when applied to RS. To address these issues, the authors propose the PSL loss, introducing two main modifications: the replacement of the activation function and the adjustment of the temperature factor's position. These changes result in multiple benefits, which are thoroughly discussed in the paper. Compared to several SOTA loss functions in RS, the proposed method demonstrates substantial improvements in user-item recommendation experiments conducted across three different scenarios.", "questions": "1.\tRecommendation systems often suffer from data sparsity issues. How does PSL perform under conditions of extreme data sparsity? Have the authors conducted experiments to evaluate the effectiveness of PSL in such scenarios, and what mitigation strategies, if any, are suggested? \n2.\tGenerally, contrastive loss is sensitive to noise (false negative instances). Is it feasible to apply PSL to general contrastive learning?\n3.\tIn the experiments, PSL-ReLU consistently achieves sound performance across all settings. Can authors provide any rudimentary insights or intuitive understandings for the selection of activation function for broader application? \n4.\tThe introduction of new activation functions in PSL may alter the training dynamics of the backbone models. Apart from performance metrics, are there differences in the objective function values or convergence rates between PSL and SL?", "limitations": "See weaknesses.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe paper is well-written and accessible to readers, with the clearly-stated and intriguing motivation. The topic is fundamental to the field of recommendation systems. The proposed method effectively alleviates existing problems, despite the minimal changes.\n2.\tThe proposed PSL loss could serve as a tighter surrogate of DCG compared to SL and could be considered as a DRO-empowered BPR loss. The bound relations with AdvInfoNCE and BPR underscore the sound merits of PSL. The logical coherence of several theorems and lemmas contributes to the comprehensiveness of this work.\n3.\tIn the experiments, the detailed parameter settings for all compared methods are provided. The inclusion IID setting, OOD setting and noisy setting are thoughtfully designed to evaluate the applicability of PSL.\n4.\tThough the straightforward implementation, it’s encouraging that the proposed method demonstrates substantial improvements on widely-used datasets compared to several SOTA performances in RS.", "weaknesses": "1.\tThe proposed PSL loss is derived from the pair-wise form of softmax loss; however, the role of the original softmax loss in the subsequent discussion is not thoroughly addressed, leading to a minor gap in the exposition.\n2.\tWhile the paper provides some justification for the chosen activation functions, the theoretical rationale for selecting ReLU, Tanh, and Atan over others is not fully developed. The impact of the chosen activation functions on the stability of the training process is not sufficiently discussed. \n3.\tThe paper mentions the use of grid search for hyperparameter optimization but does not provide a detailed analysis of the sensitivity of PSL to the temperature factor.\n4.\tThere are a few typos in the paper: such as ‘faciliating’ in line73, ‘Scenerio‘ in line 256, ‘outperforms’ in line 314, ‘furture’ in line 325, , mismatch in line 157 158.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `ufck` (1/2)\n\nDear Reviewer `ufck`,\n\nWe greatly appreciate your acknowledgement of our contributions and your insightful comments. In what follows, we provide responses to the Weaknesses (**W**) and Questions (**Q**) you have raised:\n\n> **[W1] The proposed PSL loss is derived from the pair-wise form of softmax loss; however, the role of the original softmax loss in the subsequent discussion is not thoroughly addressed, leading to a minor gap in the exposition.**\n\n**[Response]** Thank you for underlining this issue. We will offer more  explanations in the next version.\n\nNote that the **original form** (cf. Eq.(2.4)) is derived by Maximum Likelihood Estimation (MLE), where SL maximizes the probability of positive items:\n\n$\n\\mathcal{L}_{\\text{SL}}(u) = -\\sum _ {i\\in\\mathcal{P}_u}\\log\\left(\\dfrac{\\exp(f(u, i)/\\tau)}{\\sum _ {j\\in\\mathcal{I}}\\exp(f(u, j)/\\tau)}\\right)\n$\n\nRecent works (e.g., BSL) have rewritten SL into **pointwise form** for analyses, where positive and negative instances are treated separately:\n\n$\n\\mathcal{L}_{\\text{SL}}(u) = -\\sum _ {i\\in\\mathcal{P}_u}f(u,i)/\\tau+\\sum _ {i\\in\\mathcal{P}_u}\\log\\left(\\sum _ {j\\in\\mathcal{I}}\\exp(f(u,j)/\\tau)\\right)\n$\n\nAlthough the original and pointwise forms are equivalent, improving SL from pointwise perspective may disrupt the *pairwise nature*  -- i.e., the loss can not be expressed by a function of  $d_{uij}=f(u,j)-f(u,i)$ . As discussed in Section 3, ranking metrics are pairwise -- it optimizes the relative  order of instances rather than their absolute values. Once the *pairwise nature* is disrupted, the theoretical relation with ranking metrics is no longer maintained. That's why we derive PSL from **pairwise form** (cf. Eq.(3.1), Eq.(4.1)).\n\n> **[W2] While the paper provides some justification for the chosen activation functions, the theoretical rationale for selecting ReLU, Tanh, and Atan over others is not fully developed. The impact of the chosen activation functions on the stability of the training process is not sufficiently discussed.**\n\n**[Response]** Thank you for the constructive comments.  According to our Lemma 3, the selection of these three activations satisfies the condition $\\delta(d_{uij}) \\le \\sigma(d_{uij}) \\le \\exp(d_{uij})$. Admittedly, while the selected activations may not be optimal,  they are simple and  are sufficiently better than the $\\exp(\\cdot)$ function adopted in SL, serving as a closer approximation to the ideal $\\delta(\\cdot)$ function. Our experiments further validate the effectiveness of the proposed PSL.\n\nActivations can affect training stability, specifically impacting the gradient distribution (cf. Eq.(4.3)). In fact, PSL exhibits more moderate gradient distributions compared to SL (cf. Fig.1(b)), thereby enhancing training stability.\n\n> **[W3] The paper mentions the use of grid search for hyperparameter optimization but does not provide a detailed analysis of the sensitivity of PSL to the temperature factor.**\n\n**[Response]** Thank you for the constructive suggestions. In Table A1, we have conducted **sensitivity analysis of temperature $\\tau$** on four IID datasets and MF backbone (metric: NDCG@20). Results show that PSL and SL exhibit similar temperature sensitivity, and PSL consistently outperforms SL under the same $\\tau$.\n\n**Table A1.** Sensitivity analysis of temperature $\\tau$ (metric: NDCG@20).\n\n|Gowalla|$\\tau$ = 0.025|$\\tau$ = 0.05|$\\tau$ = 0.1|$\\tau$ = 0.15|\n|-|:-:|:-:|:-:|:-:|\n|SL|0.1543|0.1624|0.1248|0.0980|\n|PSL-tanh|0.1543|0.1646|0.1308|0.1014|\n|PSL-relu|0.1541|**0.1647**|**0.1323**|**0.1032**|\n|PSL-atan|**0.1547**|0.1646|0.1308|0.1013|\n\n|Movie|$\\tau$ = 0.025|$\\tau$ = 0.05|$\\tau$ = 0.1|$\\tau$ = 0.15|\n|-|:-:|:-:|:-:|:-:|\n|SL|0.0755|0.0929|0.0885 |0.0755|\n|PSL-tanh|0.0750|0.0941|0.0910 |0.0771|\n|PSL-relu|**0.0755**|**0.0945**|**0.0920**|**0.0783**|\n|PSL-atan|0.0750|0.0941|0.0910 |0.0771|\n\n|Electronic|$\\tau$ = 0.025|$\\tau$ = 0.05|$\\tau$ = 0.1|$\\tau$ = 0.15|\n|-|:-:|:-:|:-:|:-:|\n|SL|0.0274|0.0360|0.0529|0.0516|\n|PSL-tanh|0.0292|**0.0417**|0.0535|0.0521|\n|PSL-relu|0.0292|0.0416|**0.0541**|**0.0526**|\n|PSL-atan|**0.0293**|**0.0417**|0.0535|0.0522|\n\n|Book|$\\tau$ = 0.025|$\\tau$ = 0.05|$\\tau$ = 0.1|$\\tau$ = 0.15|\n|-|:-:|:-:|:-:|:-:|\n|SL|0.1210|0.1174|0.0807|0.0611|\n|PSL-tanh|0.1225|0.1207|0.0856|0.0641|\n|PSL-relu|**0.1227**|**0.1214**|**0.0876**|**0.0658**|\n|PSL-atan|0.1226|0.1207|0.0856|0.0642|\n\n> **[W4] There are a few typos in the paper...**\n\n**[Response]** Thanks for pointing them out. We have made corrections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gtnG01zwKJ", "reviewer": "Reviewer_m2Qo", "summary": "This paper examines the limitations of softmax loss in recommender systems, focusing on its weak correlation with ranking metrics like DCG and sensitivity to false negatives. The authors propose pairwise softmax loss (PSL), which replaces the exponential function in softmax loss with activation functions such as ReLU, Tanh, and Atan to improve alignment with ranking metrics and enhance robustness. PSL optimizes the score gap between positive and negative item pairs, aiming for a tighter correlation with ranking metrics and improved noise resistance. Empirical evaluations on various datasets under different conditions (IID, OOD, and noisy) indicate that PSL generally outperforms SL and other baseline loss functions in recommendation accuracy and robustness.", "questions": "* The proposed activation functions (ReLU, Tanh, Atan) may have been tested on models like MF, LightGCN, and XSimGCL, but these models do not cover the entire spectrum of recommendation systems. Different models may have unique characteristics and architectural dependencies that interact differently with these activation functions. As a result, the performance improvements observed in the tested models may not generalize to other recommendation models or architectures. \n  - Have the authors considered testing the activation functions on a broader range of recommendation models? How do the activation functions perform on models with different embedding sizes or layer structures? \n  - The authors could extend their experiments to include a wider variety of recommendation models to validate the generalizability of the proposed activation functions. Additionally, conducting ablation studies to understand the interaction between the activation functions and various model components could provide deeper insights.\n\n* The method might introduce biases because the chosen activation functions could favor specific types of data distributions or interaction patterns. For instance, ReLU may perform better on sparse data, while Tanh might be more effective in scenarios with more uniformly distributed interactions. This bias can limit the applicability of the method to a narrower range of scenarios.\n  - What criteria were used to select the specific activation functions (ReLU, Tanh, etc.)? Providing a rationale for the selection of each activation function based on empirical evidence would also strengthen the validity of their choices.\n\n* The method relies on temperature scaling to control the influence of different data points, which introduces parameter sensitivity. Finding the optimal temperature setting is crucial for achieving the desired robustness and performance, but this process can be labor-intensive and impractical in real-world scenarios. Moreover, the optimal temperature may vary across different datasets, requiring frequent re-tuning. \n  - I wonder how sensitive are the results to variations in the temperature parameter? The authors should provide guidelines or automated methods for tuning the temperature parameter effectively?", "limitations": "* The proposed activation functions have been tested on a limited number of recommendation models, which may not represent the full diversity of models used in practice. This limits the generalizability of the findings.\n\n* The specific choice of activation functions may introduce biases, favoring certain data distributions or interaction patterns, which could reduce the method's applicability across diverse datasets.\n\n* The reliance on temperature scaling introduces sensitivity to the temperature parameter, requiring careful and potentially impractical tuning for different datasets and scenarios.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-organized with clear explanations and presentation of theoretical and empirical results.\n* The authors provide strong theoretical analysis and comprehensive empirical evaluations on multiple datasets and testing conditions.\n* The proposed PSL method addresses key limitations of softmax loss, potentially leading to more accurate and robust recommender systems.", "weaknesses": "* The proposed activation functions may interact differently with various underlying recommendation models. While the paper tests on a few models (MF, LightGCN, XSimGCL), the results might not hold for other models or architectures, limiting the generalizability of the method.\n* The method may introduce biases due to the specific choice of activation functions, potentially favoring certain types of data distributions or interaction patterns.\n* The reliance on temperature scaling for robustness could lead to parameter sensitivity, requiring careful tuning that might not be practical in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `m2Qo` (1/2)\n\nDear Reviewer `m2Qo`,\n\nMuch thanks for your detailed comments. In the following, we provide responses to the concerns you have raised:\n\n> **[C1] While the paper tests on a few models, the results might not hold for other models or architectures, limiting the generalizability of the method.**\n\n**[Response]** Thank you for the constructive suggestions. Our experimental setup, which follows previous works (SL, BSL, etc.), tests on three typical backbones including basic `MF`, graph-based `LightGCN` and SOTA `XsimGCL`. However, we understand your concerns about generalizability and have subsequently tested our loss on a broader range of recommendation models, different embedding sizes, and different neural layers:\n\n- We conduct additional experiments on various model architectures, including neural-based `NCF` [a1], neighbor-based `Simplex` [a2],  VAE-based `Mult-VAE` [a3], and causality-based `PDA` [a4].\n- We conducted sensitivity analysis on embedding sizes.\n- We test the performance with different layers in LightGCN.\n\nThe results are presented in the following tables in terms of NDCG@20 under IID setting. **Our PSL consistently achieved the best performance across all tested scenarios.** These results demonstrate the generalizability of PSL, aligning with our theoretical analyses -- **the theoretical merits of PSL do not depend on any specific assumptions about model architecture, making it broadly applicable across diverse backbones.**\n\n**Table A1.** Recommendation performance (NDCG@20) with various backbones (NCF, SimpleX, Mult-VAE, PDA).\n\n|`NCF`|Gowalla|Movie|Electronic|`SimpleX`|Gowalla|Movie|Electronic|`PDA`|Gowalla|Movie|Electronic|`Mult-VAE`|Gowalla|Movie|Electronic|\n|-|:-:|:-:|:-:|-|:-:|:-:|:-:|-|:-:|:-:|:-:|-|:-:|:-:|:-:|\n|SL|0.1469|0.0787|0.0475|SL|0.1620|0.0920|0.0493|SL|0.1630|0.0927|0.0529|SL|0.1626|0.0930|0.0528|\n|AdvInfoNCE|0.1443|0.0780|0.0467|AdvInfoNCE|0.1613|0.0919|0.0495|AdvInfoNCE|0.1626|0.0934|0.0526|AdvInfoNCE|0.1624|0.0932|0.0519|\n|BSL|0.1475|0.0787|0.0476|BSL|0.1620|0.0921|0.0493|BSL|0.1630|0.0928|0.0529|BSL|0.1626|0.0931|0.0528|\n|PSL-tanh|0.1515|0.0806|0.0482|PSL-tanh|0.1637|0.0932|0.0504|PSL-tanh|0.1647|0.0941|0.0533|PSL-tanh|0.1645|0.0944|0.0533|\n|PSL-relu|**0.1520**|**0.0817**|0.0492|PSL-relu|**0.1641**|**0.0936**|**0.0506**|PSL-relu|**0.1647**|**0.0944**|**0.0535**|PSL-relu|**0.1648**|**0.0947**|**0.0537**|\n|PSL-atan|0.1508|0.0814|**0.0493**|PSL-atan|0.1638|0.0932|0.0503|PSL-atan|0.1646|0.0941|0.0533|PSL-atan|0.1645|0.0945|0.0533|\n\n**Table A2.**  Recommendation performance (NDCG@20) with varying embedding size $d$ in MF.\n\n|Gowalla|$d$ = 16|$d$ = 32|$d$ = 64|$d$ = 128|Electronic|$d$ = 16|$d$ = 32|$d$ = 64|$d$ = 128|\n|-|:-:|:-:|:-:|:-:|-|:-:|:-:|:-:|:-:|\n|SL|0.1343|0.1496|0.1624|0.1689|SL|0.0378|0.0449|0.0529|0.0566|\n|AdvInfoNCE|0.1346|0.1492|0.1624|0.1696|AdvInfoNCE|0.0384|0.0454|0.0527|0.0566|\n|BSL|0.1346|0.1495|0.1627|0.1688|BSL|0.0386|0.0459|0.0530|0.0564|\n|PSL-tanh|0.1398|0.1524|0.1646|**0.1708**|PSL-tanh|0.0405|0.0464|0.0535|0.0570|\n|PSL-relu|**0.1407**|**0.1534**|**0.1647**|0.1706|PSL-relu|**0.0411**|**0.0469**|**0.0541**|**0.0572**|\n|PSL-atan|0.1401|0.1524|0.1646|**0.1708**|PSL-atan|0.0401|0.0464|0.0535|0.0569|\n\n**Table A3.** Recommendation performance (NDCG@20) with varying number of layers $L$ in LightGCN.\n\n|Gowalla|$L$ = 1|$L$ = 2|$L$ = 3|Electronic|$L$ = 1|$L$ = 2|$L$ = 3|\n|-|:-:|:-:|:-:|-|:-:|:-:|:-:|\n|SL|0.1547|0.1628|0.1551|SL|0.0482|0.0526|0.0486|\n|AdvInfoNCE|0.1542|0.1625|0.1548|AdvInfoNCE|0.0483|0.0528|0.0485|\n|BSL|0.1547|0.1628|0.1551|BSL|0.0482|0.0526|0.0486|\n|PSL-tanh|**0.1567**|0.1648|**0.1572**|PSL-tanh|0.0501|0.0532|**0.0504**|\n|PSL-relu|0.1566|**0.1648**|0.1570|PSL-relu|**0.0504**|**0.0536**|0.0503|\n|PSL-atan|**0.1567**|0.1648|0.1571|PSL-atan|0.0501|0.0532|**0.0504**|\n\n**[References]**\n\n[a1] Neural collaborative filtering, WWW'17\n\n[a2] SimpleX: A simple and strong baseline for collaborative filtering, CIKM'21\n\n[a3] Variational Autoencoders for Collaborative Filtering, WWW'18\n\n[a4] Causal intervention for leveraging popularity bias in recommendation, SIGIR'21", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Overall Rebuttal

Dear Reviewers `VFmg`, `kuk6`, `ufck`, and `m2Qo`,

We thank all reviewers for taking the time to review our paper and for providing valuable and insightful feedback. We are delighted to see that our work has been recognized for its contributions and inspiration to the recommendation loss community, as mentioned by Reviewers `VFmg`, `kuk6`, `ufck`, and `m2Qo`. We also note that some reviewers raised in-depth questions, and we have endeavored to provide comprehensive responses.

We would like to express our gratitude to the reviewers for affirming the reasonable motivation, solid theoretical foundation, and the simplicity and effectiveness of our PSL loss, as highlighted by Reviewers `VFmg`, `kuk6`, `ufck`, and `m2Qo`. We are pleased to see that the reviewers found the experiment design and results to be comprehensive and convincing, as mentioned by Reviewers `VFmg`, `kuk6`, and `ufck`. We also appreciate the insightful comments and suggestions for our experiments, including more recommendation metrics (Reviewer `VFmg`), temperature sensitivity analysis (Reviewers `ufck` and `m2Qo`), more backbones and model structures (Reviewer `m2Qo`). We have tried our best to conduct these additional experiments, and we believe that the results are consistent with our claims.

In our rebuttal, we carefully considered the comments and suggestions provided by the reviewers, and we have addressed them point by point. We believe and hope that our responses adequately address the reviewers' concerns.

Once again, we sincerely thank the reviewers for their valuable feedback, which help us improve the quality of our work.

Meta-review: This paper reveals the limitations of the commonly used Softmax Loss (SL) for ranking tasks in recommendation systems and introduces an improved method, Pairwise Softmax Loss (PSL), to address these shortcomings. The proposed method is both well-motivated and supported by theoretical guarantees. Reviewers found the paper to be novel, with a simple and practical approach, and the analysis and results were effective. Most of the reviewers' concerns were satisfactorily addressed in the rebuttal. However, the authors are encouraged to incorporate the reviewers' suggestions in the final version, such as including discussions on additional metrics like AUC and Recall.
Decision: Accept (poster)