ITEM: Improving Training and Evaluation of Message-Passing based GNNs for top-k recommendation
Authors: Yannis Karmim
Elias Ramzi
Raphael Fournier-S'niehotta
Nicolas THOME
Venue: Accepted by TMLR ()

Abstract: Graph Neural Networks (GNNs), especially message-passing-based models, have become prominent in top-k recommendation tasks, outperforming matrix factorization models due to their ability to efficiently aggregate information from a broader context. 
Although GNNs are evaluated with ranking-based metrics, e.g NDCG@k and Recall@k, they remain largely trained with proxy losses, e.g the BPR loss. In this work we explore the use of ranking loss functions to directly optimize the evaluation metrics, an area not extensively investigated in the GNN community for collaborative filtering.
We take advantage of smooth approximations of the rank to facilitate end-to-end training of GNNs and propose a Personalized PageRank-based negative sampling strategy tailored for ranking loss functions. Moreover, we extend the evaluation of GNN models for top-k recommendation tasks with an inductive user-centric protocol, providing a more accurate reflection of real-world applications.
Our proposed method significantly outperforms the standard BPR loss and more advanced losses across four datasets and four recent GNN architectures while also exhibiting faster training. Demonstrating the potential of ranking loss functions in improving GNN training for collaborative filtering tasks.

Published in Transactions on Machine Learning Research (07/2024)
ITEM: Improving Training and Evaluation of Message-Pasiing based GNNs for top-k recommendation
Yannis Karmim
yannis.karmim karminaaaammmmmm.......m
Conservatoire national des arts métiers, CEDRIC, Paris, France
Elias Ramzi
elios. manissaamci@ullee.com com.
Conservatoire national des arts métiers, CEDRIC, Paris, France
Raphaël Fournier S'niehotta
fournier@enam.fr
Conservatoire national des arts métiers, CEDRIC, Paris, France
Nicolas Thome
nicolas. thome upme.ff
Sorbonme Université, CNRS, ISIR, Paris, France
Reviewed on OpenReview: https: // openreview. forum? id= M2uos
Abstract
Graph Neural Networks (GNNs), especially geessagepassing-bee based models, have become prominent in top-k recommendation tasks, outperforming matrix factorization models due to their ability to efficiently aggregate information from broader context. Although GNNs evaluated with ranking-besse based metrics, e.g. NDCG@k and Recall@k, they remain largely trained with proxy losses, e.g. the BPR loss. In this work we explore the use of ranking loss functions to directly optimize the evaluation metrics, an area not extensively investigated in the GNN community for collaborative filtering. We take advantage of smooth approximations of the rank to facilitate end-to-end training of GNNs and propose a Personalized PageRank based negative sampling strategy tailored for ranking loss functions. Morcover, we extend the evaluation of GNN models for top-k recommendation tasks with an inductive user-centric protocol, providing a more accurate reflection of real-world applications. Our proposed method significantly outperforms the standard BPR loss and more advanced losses across four datasets and four recent GNN architectures while also exhibiting faster training. Demonstrating the potential of ranking loss functions in improving GNN training for collaborative filtering tasks.
1 Introduction
Recommender systems have become essential component in many online applications, helping users discover relevant and personalized content amid the overwhelming abundance of information available on the internet. Collab- orative filtering is one of the most popular and widely adopted techniques for building recommender systems, which operates by leveraging the past behavior of users and the relationships among items to generate recommendations.
Graph Neural Network architectures have emergod as powerful methods for learning and representing complex data structures, particularly those that exhibit non-Fuclidean properties ies such graphs (Veličković et al. (017):

---

Published in Transactions on Machine Learning Research (07/2024)
rank
rank 2
rank 3
rank 4
rank
143
Mnyy m
User query
ITEM
01 8pu
GNN
8p4
BREAK ING AWAY
real
1hg
top-k recommendation
Figure 1: The goal in top-k recommendation is to recommend to user, e.g. My (purple), relevant items such as ip. (in green), based on its interaction history, i.e. items in blue such as ip-- ITEM directly optimizes the evaluation metric, Le. NDCG, during training using a smooth approximation of the rank and Personalized PageRank (Page [1998] based necative sampling Best seen in color,
Hamii ton al. (2017): Kipl & Welling (2016); Rossi et (2020); Xu et al. (2018)). In the context of recommender systems, GNNs offer a natural way to model user-item interactions, leveraging the inherent graph structure to
capture higher-order relationships among users and items. Among the various GNN architecturos, messago-passe passing
GNNs (MP-GNNs) have demonstrated impressive performance on top-k recommendation tasks (He al. (2020);
Wang al. (2019); (Sn al. (2020); Chen et al. (2020b)), consistently surpassing traditional matrix factorization methods (Rendle et al. (2012 Hsieh al. (2017): Chen al. (2020a)).
Top-k recommendation is standard task in recommender systems, which involves generating ordered list of k items for a user. This process necessitates calculating similarities between elements before ranking them. Evaluation of top-k recornendattions typically employs ranking-based metrics such as Average Precision (AP), Normalized Discounted Cummlaativ Gain (NDCG), or Recall at k (R@k). However, these metrics are non-differentiable and cannot be directly used to train neural networks like GNNs with Stochastic Gradient Descent (SGD).
Instead, GNNs are commonly trained using a pairwise loss, the well-known Bayesian Personalized Ranking (BPR) Rendle et al. (2012) which serves as coarse approximation of the ranking metric. The BPR is not explicitly designed to optimize standard non-differentiable evaluation rank-based metrics, such NDCG@K and Recall@K: there are some criticisms about the gap between the evaluation objective and the training objective of the BPR (Wul el. al. 2021 ). We investigale alternative loss functions that more closely aligu the training and evaluation objectives.
The second issue arises from another misalignment, between batch learning and rank- based evaluation. During train- ing, one does not have access to the true rank and must resort to hard negatives, that are more challenging to distin- guish from positive itemss, for better performance. Some methods, such as MixGCF (Huang al. (001)), ), generate ar- tificial hard negatives for the BPR loss. While this approach is suitable for pairwise loss like BPR, its online process is computationally expensive. Given that we employ listwise loss function, need to sample large number of infor- mative negative items per user, the hard negative generation of MixGCF is not tractable. We thus propose to sample negative items offline, based on their Personalized PageRank (PPR) score (Page et al. (1998)): a high PPR score indicates proximity in the graph, making them more challenging to distinguish from a positive item for MP-GNNs.
Another impediment in the GNN literature for top-k recommendation is the limited and unrealistic evaluation protocol. Recent works predominantly use transductive approach, evaluating on the same users used in training. This not only deviates from real-world recommendation scenarios but also fails to consider GNN models' generalization capacity. We propose to enhance evaluation by incorporating an inductive user-pplit protocol, evaluating models on users not seen during training.
In this paper, we introduce our framework ITEM (Improving Training and Evaluation of Messago-passinggg baassigg-bssd based GNNs). Our framework is designed to provide training and evaluation for Graph Neural Networks tailored to the top-k recommendation task.
Specifically, our list-wise loss LITEM first leverages smooth rank approximations, which have recently been revisitod image retrieval and machine learning Bruch al. (2019): Brown et al. (2020); Ramzi al. (2021). leading good approximations of the evaluation metrics, such as the NDCG AP.
2

---

Published in Transactions on Machine Learning Research (07/2024)
Additionally, we enhance our loss LITEM by incorporating negative sampling strategy tailored for rank approxination losses and leveraging the graph data structure. This stralegy is based on the Personalized PageRank (Page et al. (1998) (PPR) score. We show that this sampling is particularly well suited to our loss CITEM, since it allows for a fast sampling of many informative negative items. This sampling helps to build large efficient batches to better approximale the true ranking.
Finally, we propose to evaluate and benchmark GNNs for top-k recommendation in an inductive user-split. protocol. While it is known in the field of recommendation and used by some traditional models (Meng ct al. (2020); Liang et al.] (2018) this setting has not been used to evaluate GNN architectures. This user-split protocol is more realistic because it introduces new users in testing, thus better evaluating the generalization capacity of recommender systems.
Wc carry out extensive experimental validations in both transductive and inductive scttings. Our results highlight the benefits of ITEM over the standard training with BPR loss in terms of time and performance across multiple GNN architectures. Moreover, demonstrate that ITEM outperforms more advanced state-of-the-art loss functions, showcasing its effectiveness.
2 Related Work
2.1 Graph Neural Networks for Collaborative Filtering
We locus on collaborative filtering (CF) models applied to data with implicit feedback, where only connections between users and items are considered, without incorporating other informations such as rating. A common variant of CF is the top-k recommendation lask, where the goal is to identify a small set of items that are most relevant to user's interests. In this context, Graph Neural Networks have shown impressive results, indeed they are directly suitable to this task since user-item interactions can be modeled by bipartite graphs. Within the family of graph neural network models, message- message-pasinng-hee passing-bassed [Gilmer et al. (2017) methods have demonstrated superior performance compared to traditional CF models. These include mairix factorization (MF) (Rendle et al. (2012); Hsieh et al. (0177); Chen et al. (2020a)) ), auto-encoders (Liang et al. (2018)), and node embedding models that rely on random walks for gencrating reprosentations &Perozzi al. (2014); Grover Leskovoe (2016)). ). MP-GNNs for CF learn user and item represenlations by propagaling and updating their embedddngs through the bipartite graph. The BPR loss is employed to optimize the user-item representations by ensuring that user's embedding is closer to a positive item than to negative one. In this sense several models of MP-GNNs have been designed such as NGCF (Wang et al. (201)). LR-GCF (Chen et al. (2020b)), or DGCF (Wang et al. (2020)).
He et al. introduce LCg (He et al. 2020) as a simplified version of NGCF, achieved by eliminating weight. matrices and non-linear activation layers. Although LightGCN is less expressive, it proves to be highly effective and more efficient, with a considerably simplified training process. Subsequent research has focused on enhancing the training of these models: SGL-ED (Wu et al. (2021)) proposes to combine the standard BPR loss with a self supervised loss, while MixGCF (Huang et al. (2021) artificially generates hard negatives embeddings for negative sampling to replace the random negative sampling in the standard BPR loss.
2.2 Evaluation protocol for top-k recommendation
All these GNN models competed with matrix factorization methods thus, their learning and evaluation setups are transductive.
However, other evaluations and training protocols for recommendation exist (Liang et al. (018)). ). In (Meng et al. (2020)), authors show that the evaluations of GNNs-based models are limited compared to all those existing in the literature. Instcad of splitting the data based on interactions, one may split the data by user, with some users in the training set, and new users for testing (see Fig. 3 This setup is closer to real applications, and allows to produce recommendation to new users without re-training the model. It is also more challenging, requiring to construct representations for a new user without learning.
We propose here to evaluate our ITEM model and benchmark MP-GNNs baselines in this realistic setting which not used before for MP-GNNs, the best of our knowledge.
3

---

Published in Transactions on Machine Learning Research (07/2024)
Message passing
user-item similarity
ITEM
u
V
rank, DjEvo 8pm)
14
User queez
Ug
GNN
IITEM = 1 DCG,
items to rank
iDCG
DCG,
log,(1 + ra)k,
Backpropagation
pEV+
Figure 2: Using message passing, a GNN creates embeddings for every node of the graph. For each user we first construct batch of randomly sampled positive itemss and negative items selected with our Personalized PageRank PPge et al. (1988) based negative sampling Eq. 4 We then compute the score of the user wrt. the batch of items and calculate the loss using the approximation of the rank of Eq. (2). Finally the loss is backpropagated to update the parameters of the GNN, and update the embeddings for the items and users in the transductive set ting. Best seen in color.
2.3 Lonking-based loss function
The Learning to Rank problem (Cao et al. (0077) is evaluated using ranking based metrics, e.g. NDCG (Järvelin & Kekäläinen (2002)), ) R&K, or AP (Croft et al. (2010)). As these metrics are not differentiable (because of the ranking operator), their optimization has been abundantly studied. Different proxy methods have been built in Information Retricval, with for instance pairwise loss (Burgos et all (2005); Hadsoll et al. (2006)), or triplct losses Rendle al. (2012): [i et al. (2017)). To train GNNs, the most well-know loss is the BPR loss (Rendle et al. (2012), smoothed triplet loss. However, it was shown that triplet losses tend to put more emphasis
correcting errors at the bottom of the ranked list, rather than at the top, which would have the most impact to maximize the metric (Brown et al. (0200). The direct optimization of ranking- based metrics has long been studied for Information
Retrieval using structured SVM (Yue et al. (2007)), or rank approximations (Burges et al. (2006); TTylor et al. (2008)
Qin ct al. (2009) ). Soveral direct optimization mothods have gained traction in deep learning using for instance soft
binning approaches (Revaud et al. (2019)) or rank approximations (Bruch et al. (2019); Poorotyn Bialobrzeski
(2021): Brown et al. (2020); Ramzi et al. (2021 2022)). NeuraINDCG Pobrotyy & Bialobrzeski (2021) proposed differentiable approximation to NDCG by using NeuralSort, differentiable relaxation of the sorting operator,
resulting in smooth variant of the metric and new ranking loss function. ROADMAP et al. 2021 introduce robust and decomposable average precision (AP) loss for image retrieval, addressing
and non-
decomposability issues with a new rank approximation and calibration loss. |Ranzi et al. (2022) use a hierarchical AP training mothod for pertinent image retrieval, leveraging a concept hicrarchy to refine AP by intograting errors' importance and better evaluate rankings. In our work, we introduce the use of the sigmoid function for approximating the NDCG rank Brown al. (2020): Qin et al. (2009) on GNNs in a top-k recommendation context.
3 ITEM framework
In this section, we present the ITEM framework. We first define our ranking- loss using a smooth approximation of the rank in Section 3.. well as our adapted negative sampling strategy in Section 3.2 We then introduce the protocol used to train and evaluate the GNNs performances in Section 3.3
Training context We consider an undirected bipartite graph G= (,, with [4] users, [V] items and |g| edges.
We assign to each node in W/VV an embedding h. We use a GNN that re-embeds the node embeddings to another space of the same dimension using message passing.
The task is to construct an embedding space such that, after message passing, the embedding of a user closer to the embeddings of its positive items ()+) than to its negative items (V-), i.e. for user u, its embedding h,;
4

---

Published in Transactions on Machine Learning Research (07/2024)
T and 8, s,=hu hj 2 positive item p. its embedding hp, and a negative item j. we want Sp>s,, sp sp 8p and &j are respectively the similarity score between the positive and the negative item. To evaluate the performances of a GNN, we use ranking-bases metrics, NDCG Eq. (1), that measures the quality of a ranking.
DCG
[DCG=E EpEV! Togg (1-rank(p)) iDCG mDCGkmaX DCG
NDCG= iDGG
with
(1)
3.1 Direct lanked-based optimization
GNNs are evaluated using standard ranking based metrics, egg. NDCG, R@k, AP. We propose to train GNNs by optimizing directly smooth approximations of those metrics. Specifically, operator to yield losses amenable to gradient descent.
approximation of the ranking
The ranking operator can be defined as: E QQnn et al. (2009): Brown et al. (2020) ). The intuition behind this definition is that in order to have the rank of a (positive) item p. we must "count" " the number of items j that have a similarity to the query user sj greater than p's, Sp, Lee ((8jj=8)== 1. During training, we aim to minimize rank = 11 Eje (s, of "negative" items that have higher score than positive items. Lonking-baseds metrics optimize this objective. This writing of the rank shows why the ranking operator is not differentiable, i.e. becanse the Hcaviside function is not, specifically its gradients are null or undefined.
with H the Heaviside (step) function
; i.e. the number
We propose to use the sigmoid function as an approximation of the Heaviside function (Qin et al. 20009 Brown ct al. (020 ): (2; 1+exp = with T F a temperature scaling parameter. T controls the slop of the sigmoid,
gets smaller the slope is greater, and the sigmoid saturates faster.
Using this approximation we can define a smooth version of the rank:
rank,(p.))=14 Σ
(s Sp:T)
(2)
JEV
rank, is differentiable and is thus amenable to gradient descent. It has a single hyper-parameter, т, we study its impact our experimental validation (Section 4.5g.
Application to ranked-based metrics This plug-and-play rank approximation can be used to get a smooth version of ranking metrics, e.g. NDCG, R@k and AP.
The approximation of the Normalized Discounted Cumulative Gain (NDCG) is defined as follows:
CITEM = DCG, iDCG with DCG, = Σ
(3)
loga (rank,(p)))
PED+
We use this approximation of rank, to approximate the DCG (see Eq. [1), and use the exact iDCG (see Eq. |
In Sec. A of supplementary material we show how other ranking- based metrics can be approximated and show their effect on the evaluation metrics Sec. B.5.
End-to-end manked-based training of GNN GNNs, e.g. GCN (Kipf & Welling 2016)), jointly learn embeddings for the items and users of graph, and through parameterized (or not for LightGCN He et al. (2020)) message passing they create representation for items and users. Using a user a query (purple embedding in Fig. D. as in Learning to Rank (Cao et al.] (2007)), ), we aim to make the distance between a user's embeddings and the ones of its positive items, t.e. with implicit feedback (in green on Fig, 2 closer than the negatives ones, Le. no implicit feedback (in red on Fig. 2). ITEM Eq. (3) is directly applied on the similarities, to produce better ranking. As ITEM differentiable, after computing the loss we can backpropagate gradients through the network (see Fig. 2 to update the potential weights of the GNNs, and update the users and items embeddings.
3.2 Negative Sampling
The objective of ITEM is to rank, for user query, its positive items (high rating or implicit feedback) before the negative ones. To do so, for each user we have construct, batch of positive items and negative items
5

---

Published in Transactions on Machine Learning Research (07/2024)
Train items
Test items
Train users
u1
i
i
i
i
i
U1
2
22
i
i
i
i
i
22
3
i
i
i
i
i
43
Test users
Fold-in
Fold-out
Figure 3: Transductive interaction-split (left) vs Inductive user-split (right) (Meng et al. (2020))- )- In the first case, the same users are train and test, their learned embeddings can directly used in test. In the second case, a part of the users and 100% of their interactions are used in train. During the evaluation, the model infers a representation of test. user from some interactions (fold-in), in order to predict the fold-out items where apply the ranking metries.
cannot use all items. For a given user the number of negative items is much larger than that of the positive items. In order to better approximate the global ranking in mini-batch learning, we have to sample significant number of informative negative items. Uniform random sampling is adopted as a solution in many GNN models for recommendation (He et al. (2020): Wang et al. (2019)), however the sampled negatives are often not very informative which can limit the performances of the model. We propose use the Personalized PageRank (Page) et all (1998)) (PPR) weigh the sampling of negative examples. Specifically we normalize the PPR score using the softmax function and sample negative items for high PPR score will be harder to rank, they are in closer proximity to the user query.
query Ug according to Eq. (4.. Indoed items with
pprny (3,
p(1jlug) (ug)
Σkev p (5)
Unlike PinSage Yiin et al. 2018)). , our sampling strategy evaluates negative items by weighing them differently. Specifically, the PPR score calculated for each target user, asscssing its proximity every item in the bipartite graph. These scores are normalized with the softmax function create a probability distribution for sampling. contrast to existing sampling strategies such MixGCF (Huang al. (2021) which generate hard negatives from embeddings "online" (i.e., during training), our PPR score be computed the bipartite graph "offline", before training. The offline computation makes hard negative sampling much faster, allowing us to build informative and large batches well-suited for our fITEM loss. We perform ablative experiments in Table 5 to show the benefit. of this sampling strategy. Our strategy efficiently selects informative hard negative items, but runs the risk of including false negatives. Alternative strategies may avoid false negatives entirely, but result in non-informativv samples. Our results in Table 5 demonstrate that our trade-off, which samples hard negatives, in spite of the risk of false negatives, improves model performances.
3.3 Protocol for collaborative filtering
3.3.1 Inductive protocol
We propose to evaluate GNNs in the top-k recommending task using inductive uscr-split protocol (Meng et al. (2020): Liang et al. (008)) (illustrated on the right side of Fig, 3). During training, we have access to a fraction # (i.e,, U1 1/2 on Fig. 3 of the user and their entire click history, for evaluation we consider the 1- - (i.e. us on Fig. 3) unseen users and, with access to fraction y7 of their click history (fold-in in blue on Fig. 3). we must recommend the other 1-n fraction (fold-out in green on Fig. 3 of their click history. Despite the benefits of this inductive protocol known in top-k recommendation, to knowledge GNN models measured in this

---

Published in Transactions on Machine Learning Research (07/2024)
Table Statistics of the four datasets used for our experimental validation.
Dataset
# Users
#Items
# Links
1000 (Harper & Konstan 2015
610
8957
100k
MovieLens-IMV 1M (Harper & Konstan 2015)
6022
3043
1m
Yelp-2018 Asgharr (2016])
31668
38048
1.5m
Amazon-book (Wang ct al. (2019)
52643
91599
2.9m
user-split protocol in this task. This protocol measures the capacity of a GNN to make accurate recommendation for new user (not seen during training), based on its first interactions.
3.3.2 Transductive protocol
The standard protocol for evaluating GNNs in a recommendation task (Wang (2019 He al. (2020); Wu
et. al. (2021]) is lo evaluate them in transductive manner, de. items and users are the same during training and evaluation (Jeft of Fig. 3). The training and evaluation splits are defined using an interaction-split specifically for given user we use fraction (orange on Fig. 3 of its interaction to perform message passing during training and for evaluation the GNN must give a higher score to the 1-p (green on Fig. 3 fraction of its other interactions before the items to which it has no interaction with. For this protocol, the nodes in the graph remain the same for training and evaluation: the system cannot accommodate new users, it is limited presenting new content to existing users.
4 Experiments
In our experiments and ablation studies, aim is demonstrate the effectiveness of our ranking loss function well as our negative sampling setup applied to MP-GNNs in comparison to the standard BPR loss and other training enhancement strategies such as self-super regularization loss or alternative negative sampling techniques. Additionally, we propose to refine the evaluation of GNNs for CF by benchmarking them using an inductive, user-centric protocol.
4.1
Datasets and evaluation protocol
4.1.1
Evaluation metrics
We follow (Huang et al. (2021 He et al. (2020) Wang et al. (0119) evaluation protocol, computing the NDCG@2
and Recall@20 with the all-ranking protocol (ee et al. (2017)
4.1.2 Datasets
We validate our method on four recommendation datasets. We use the MovieLLnns-100k and MovieLens-1N Harper & Konstan| (2015)) dataset, the 2018 edition of the dataset from the Yelp challenge (Asghar) (016 and
Amazon-book /Wang al. (2019)) We detail the preprocessing of the datasets in Sec. B.1 supplementary.
4.1.3 Data splitting strategies
Transductive sctting: For the transductive sctup we use the exact same data preproccssing and split as the LightGCN (He et al. (2020)) and MixGCF (Huang et al. (2021)) models. In the transductive split, for each user 80% of these interactions have been randomly selected for the training set and the 20% for the test and validation set. We followed this same ratio across all datascts in our experiments. This data split is illustrated on the left side of Fig. 3
Inductive setting: The inductive setup is based on the user-split data protocol, detailed in Meng et al. 2020 and used by the variational encoder model Mult-VAE (Liang et al. (018)) ). We used the same ratio as in the
Mult-VAB paper, Le.. In this protocol, we first separate users into training.vllidation validation and test sets. 80% of the users are kept for training, 10% for validation and 10% for testing. Unlike the test and validation sets, the training users keep 100% of their interaction history for learning our models. The test and validation user interactions
7

---

Published in Transactions on Machine Learning Research (07/2024)
are separated into fold-in and fold-out set. The fold-in in set is used to build representation of the user in inference from partial history, predictions and evaluation are performed on the fold-oul set. In our experiments, the fold-in is composed of 80% of the randomly sampled interactions of a user, the rest is for the fold-out. This data split is illustrated on the right side Fig.
4.2
Baselines and implementation details
4.2.1
Baselines
We use several GNN architectures GCN
Kipl & Welling (2016)), GAT (Velčković et al. (011)) ). GIN (Xu et al. (2018) and LightGCN (He et al. 2020 and compare their original training, i.e. with the BPR (Rendle et al.
2012 ), to ITEM training. To assess our model against the state of the art, we compare it with various CF model families, such as MF (MF-BPR (Rendle et al. (012)), ENMF (Chen et al. (2020a) 1), metric-learning based (CML
Hsich et al. (2017)) )) graph embedding (DeepWalk (Perozzi et al. (2014)). LINE (Tang al.] (2015) ). Node2Vee
Grover & Leskovec (2016))), VAE models (Mult-VAE (Liang et al. (2018)) (), MP-GNNS (NGCF WWang et al.
2019 ), DGCF (Wang et al. (2020)), LightGCN (He et al. (2020)), NIA-GCN (Sun et al. (2020)) LR-GCCF
Chen et al. (2020))) ))- We also compare ITEM to a recent embedding enhancement method Chen et al. (2023).
F inally, we compare ITEM to other losses and negative sampling methods such NeuralNDCG (Pobrotyn &
Bialobrzeski ( another ranking loss); SGL-ED (Wu et al. (2021) ) which combines BPR and a self-supervised
loss: XSImGCL and SimGCL (Yu et al. (2022) two contrastive based models for CF; MixGCF (Huang et al.
2021 which employs a hard negative sampling method for BPR. For a fair comparison of the various loss function, we employ the same LightGCN backbone across all models.
4.2.2 Implementation details
We give all details on byper-paramcters and optimization procodure in Sec. B.2 of supplementary. For the inductive setting, we found experimentally that learning user embeddings during training is harmful for the generalization of the GNNs to new users, so for training and evaluation users' emboddings are inferred using message passing only.
4.3
Main results
In this section we present our main results, we compare ITEM vs state-of-the-art methods. We first compare ITEEM on the standard transductive protocol in Table 3 we then compare ITEM on the indcutive protocol Table 2 For
both protocols we use LightGCN (He et al. (2020)) backbone following (Huang et al. (2021); Wu al. 2021
4.3.1 Transductive state of the art comparison
In Table 3 we compare, in the transductive setting, ITEM using LightGCN (He et al. (020 backbone state-of-the-art methods. We show that across all datasets ITEM outperforms all the MP-GNNs state-of- the-th- methods NGCF, LR-GCCF, NIA-GCCN, LightGCN and DGCF. It surpasees non-graph methods on the three datasets with relative improvements of +18% NDCG@20 on Yelp-2018 or 21% NDCG@20 on Amazon-book vs Mult- VAE, which shows the interest of using dedicated architectures for graph learning. Furthermore, ITEM outperfomms the recent NeuralNDGG (Pobrotyn & Bialobrzeski (2021) ), e.g. +0.89 R&20 and +0.65 NDCG@2 on Yelp-2018, well SGL-ED, e.g. +0.55 R and +0.23 NDCG@20 on manzzon-Booke and the advanced MovieLens- 1M. Despite the limitations of MP-GNNs (Alon & Yahav (200); Chen al. 2019)), ITEM shows competitive results against the hybrid model SimpleX (Ma et al. (2021)) with better results on two of three datasets with +1.78 R@20 on MovieLens- 1M and +0.38 R@20 on Yelp-2018 with the best state of the arts results.
sampling method MixGCF (Huang et al. (2021)) ), e.g. +2.45 R@20 and +2.64 NDCG@20
4.3.2 Inductive state-of-the-ar comparison:
MP-GNNs can be easily evaluated in an inductive setup, unlike matrix factorization models or random walk embodding methods. MP-GNNs have a strong ability generalize to users not seen during the training phase with the propagation and update process. We show that our ITEM method significantly boosts the performance of MP-GNNs in the inductive setup. We compare ITEM using LightGCN (He et al. (2020)) to state-of-the-art.
8

---

Published in Transactions on Machine Learning Research (07/2024)
Table 2: Comparison of ITEM vs state-of-the-art methods on three inductive benchmarks, Best results in bold.
MovieLens 100K
Yelp-2018
Amazon- Book
Method
R@20
N@20
R@20
N@20
R@20
N@20
Mult-VAE [Liang el al.
2018
30.14
28.28
10.15
8.18
10.86
9.2
GCN
Kipf & Welling
2016
28.74 74
27.68
7.34
5.76
8.85
7.61
GAT
Velicković et al.
2017
31.01
28.92
9.04
7.32
9.88
8.17
GIN
Xu et al. (0018)
29.71
27.58
7.34
5.76
9.62
8.05
LightGCN ee et al.
(0200)
30.79
29.73
7.88
6.34
9.56
8.02
NeuraINDCG (Pobrotyn & Bialobrzeski MixGCF (Huang et al. (2021)
2021
31.12
30.07
9.14
7.62
9.54
8.19
32.07
30.62
9.85
8.21
10.11
9.63
ITEM (ours)
33.84
32.63
10.54
8.70
11.03
9.89
Table 3: Comparison of ITEM vs state-of-the-art methods on three transductive benchmarks. Best results in bold.
MovieLens- 1M
Yelp-2018
Amazon-Bood
Method
R@20
N@20
R@20
N@20
R@20
N@20
MF-BPR (Rendle et al.
2012
21.53
21.75
5.49
4.45
3.38
2.61
CML (Hsich et al. 2017
17.30
15.63
6.22
5.36
5.22
4.28
ENMF (Chn et (2020a]
23.15
20.69
6.24
5.15
3.59
2.81
Walk (Perozzi et al. (0144)
13.48
10.57
4.76
3.78
3.46
2.64
LINE Tang et al.
2015
23.36
22.26
5.49
4.46
4.10
3.18
Node2 Vec
Grover & Leskovee
2016
14.75
11.86
4.52
3.60
4.02
3.09
Mult-VAE [Liang et al.
2018
29.23
23.84
6.41
4.97
4.46
3.33
NGCF (Wang et al. (2019))
25.13
25.11
5.79
4.77
3.44
2.63
LR-GCCE (Chen et al.
(2020b)
22.31
21.24
5.61
3.43
3.35
2.65
NIA-GCN (Sun al. 2020)
23.59
22.42
5.99
4.91
3.69
2.87
LightGCN He et al. 2020
25.76
24.27
6.28
5.15
4.23
3.17
DGCF (Wang et al.
20201)
26.40
25.04
6.54
5.34
4.22
3.24
SGL-ED Wu
2021
26.34
24.87
6.75
5.55
4.78
3.79
SimGCT.
Yu et al.
2022
27.55
25.01
7.21
6.01
4.90
3.98
XSimGCL YYu al.
(2022)
27.94
25.36
7.33
6.06
5.02
4.11
Adap-r Chen et al.
(2033
27.87
26.15
7.33
6.12
6.12
4.90
NeuraINDDG (Pobrolyn & Bialobrzeski 2021
29.45
25.13
6.50
5.23
4.38
3.27
MixGCF (Huang et al. (2021)
27.35
24.56
7.17
5.84
4.51
3.41
ITEM (ours)
29.80
27.20
7.39
5.88
5.23
4.02
methods, and show that the three datasets, ITEM sets new state-of-the-ars performances for inductive recommendation. It outperforms Mult- VAE (Liang et al. (2018)) which is designed for the inductive setting, by +3.79 NDCG@20 on MovieLenss +1.4 NDCG@20 on Yelp-2018 and +0.69 NDCG @20 on Amazon-book. We compare ITEM to different GNNs that were optimized using the BPR loss, this is further studied in Tab. of supplementarr where we show the interest of the well designed loss of ITEM vs standard BPR loss. Finally ITEM outperforms MixGCF (Huang et al. (2021)) on the three datasets, e.g. on MovieLens 100k with +1.77 R:020 +1.58 R@200 Yelp-2018 well the ranking loss function NeuraINDCG (Pobrotyn & Bialobrzeski (2021)), e.g- +1.54 R@20 and +1.08 NDCG@20. Although mussage-passingg models face certain challenges (Alon & Yahav (2020); Chen et al. (2019)), when evaluated with the proposed inductive protocol they outperform state-of-the-arl methods and exhibit fast training and strong generalization capabilities for unseen users during the training phase.

---

Published in Transactions on Machine Learning Research (07/2024)
4.4 Ablation studies
In this section we perform ablation studies for the two elements of ITEM. We first study the impact of using CITEM vs the BPR loss Table 4
We then study the impact the sampling in Table
4.4.1 Ranking loss vs BPR
In Table 4 compare in the settings the BPR loss (Rendle et al. (2012)) and our proposed ranking-based loss CITEM three transductive benchmarks, and show that for all four considered architectures, 1TEM outperforms the BPR loss. Using our loss in combination with LightGCN (He et al. (020 the state-of-tho-art GNN for transductive datasets, increases relative performances from +10.05% NDCG@20 on MovieLens to +22.1% NDCG@20 on Amazon-book. We can also note that when using ITEM more expressive GNNs, e.g- GAT Veličković et al. (2017)) ) perform better than LightGCN on MovieLens- 1M and Amazon-book. With GAT the relative improvements range from +18.55% NDCG@20 on MovieLens- 1M up to +54.9% NDCG@20 on Amazon-book. Overall, Table shows the huge benefits of optimizing a ranking- based loss rather than prozy loss. Table of supplementary shows similar results for the inductive benchmarks. In Table 1 of supplementary show that the best performing architectures the inductive selling are not the same in the transductive setting.
Table 4: Comparison of our ranking-besse based loss ITEM vs the BPR loss (Rendle et al. (011 using different GNN architectures on transductive benchmarks. For each architecture best results bold, best overall results is underlined.
MovieLens-IMM
Yelp-2018
Amazon-book
Method
R@20
NDCG@20
R@20
NDCG@20
R@20
NDCG@20
BPR
27.26
22.75
5.21
4.04
3.48
2.55
GCN
CITEM
29.96
25.62
6.81
5.33
4.89
3.70
%1mprov.
+9.90%
+12.62%
+30.7%
+31.9%
+40.5%
+45.1%
BPR
26.99
22.28
5.52
4.32
3.42
2.54
GIN
CITEM
29.94
25.60
6.87
5.38
5.24
3.94
%Improv.
+10.93%
+14.90%
+21.5%
+24.5%
+53.2%
+55.1%
BPR
27.50
22.96
5.08
3.89
3.41
2.53
GAT
CITEM
30.43
27.22
7.03
5.47
5.20
3.92
XImprov.
+10.65%
+18.55%
+-8..%
+40.6%
+52.5%
+54.9%
BPR
25.76
24.27
6.26
5.14
4.23
3.17
LGGN
CITEM
29.69
26.71
7.25
5.70
5.09
3.87
%Improv.
+15.27%
+10.05%
+15.81%
+10.89%
+20.3%
+22.1%
4.4.2
Negative sampling
We show in Table 5 ablation studies for the impact of our PPR sampling strategy. On both datasets our sampling strategy boosts the performances ITEM, e.g. +3.2% NDCG@20 relative improvement on Yelp-2018 and +3.9% NDCG@20 on Amazon-book. We also use the MixGCF (Huang al. (2021) negative sampling with ITEM, however due its high computation overhead, cannot sample many negatives. This leads to MixGCF negatively affecting performances Yelp-2018 and Amazon-book, e.g. -0.34 NDCG@20 Amazon-book.
10

---

Published in Transactions on Machine Learning Research (07/2024)
Table 5: Ablation study of the components of ITEM. The loss: / for LITEM, BPR otherwise (X). And sampling: random nogative sampling vs MixGCF (Huang et al. (021)) the ITEM sampling Eq. (PPR).With LightGCN
He al. (0200) on transductive benchmarks.
Yelp-2018
Amazon-book
LITEM
MixGCF
PPR (ours)
R@20
NDCG@20
R@20
NDCG@20
X
X
6.26
5.14
4.23
3.17
X
7.25
5.70
5.09
3.87
V
V
X
7.21
5.71
4.86
3.53
/
X
V
7.39
5.88
5.23
4.02
4.5
Model analysis
4.5.1
Efficiency comparison
In Table 6 compare the training times of LightGCN He
Table 6: Computational cost on Yelp-2018 with Light-
ct al. (2020)) using the BPR loss, MixGCF and ITEM on
GCN backbone.
the Yelp-2018 dataset until convergence. We train these 3 variants with a Quadro RTX 5000 GPU. LightGCN-BPR takes 10h26 before convergence on the Yelp-2018 dataset, which contains 1.5 million interactions. With MixGCF, the convergence time is reduced to approximately 6 hours, hard negative sampling accelerates convergence, despite the time per epoch being more than twice as long due to the online generation of the hard negatives examples. Finally, ITEM loads to model convergence in just hour and 43 minutes. ITEM performs far fewer iterations per
Method
Time epoch
Training Time
BPR
53s
10h26
MixGCF
145s
6h07
ITEM (ours)
12.3s
1h43
epoch by building batches of negative and positive items per user to approximate the NDCG metric.
4.5.2 Impact of
We show in Figs. 4a and 4b the impact of in Eq. 2 on the R@20 and NDCG@20 when optimizing ITEM on MovieLens for the inductive protocol. We can see on both figures that our method is robust for wide range of т. Specifically, for values of r in range 0.1 to 2.0, ITEM outperforms the BPR loss. Also note that using a finer selection of t could lead to better results than reported in Table 2 e.g. using 7 = 0.3 leads to a R&200 of 33.52 in Fig. 4a against 33.13 for the value of 7 used in Table (note that in Table 2 ITEM additionally uses the PPR sampling).
4.6 Qualitative results
In the qualitative results presented in Fig. 5 a comparative analysis between our approach and the BPR loss baseline reveals the superior ranking achieved by ITEM, This enhanced ranking is not solely attributable optimizing the target metric directly during training: it is also result of the efficacy of our hard negative sampling strategy. This stralegy contributes (o more effective differentiation between negalive and posilive instances, further refining the ranking quality. Supplementary B.6 includes a detailed listing of the top-20 results for comprehensive examination.
5 Conclusion
In this study, we introduce the ITEM framework to optimize GNNs for the top-k recommendation task. By incorporating ranking losses used in image retrieval and directly optimizing item ranking for a given user, we demonstrate the limitations of the BPR loss, a widely-used pairwise loss within the GNN community.
Our proposed list-wise loss requires sampling a substantial number of hard negative items for each user, which challenging task. Unable to employ online methods such MixGCF (Huang et al. (021)). devised an offline
11

---

Published in Transactions on Machine Learning Research (07/2024)
32
32
NDCG@20
30
28
R@20
28 *
26
24
26
ITEM
ITEM
BPR
22
BPR
24
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
(a) R@20 vs T in Eq.
2
(b) NDCG@20 in Eq.
Figure 4: 7 in Eq.
vs R@20, NDCG@20 on MovieLens-10M6 100k (inductive) with LightGCN
He et al. 2020
rank 1
rank 2
rank 3
rank 4
rank 5
JINE MANI
THE WOMAN GREEN
User query
ITEM
Ballets
Brooddary
BASIL RATHBONE NIGEL BRUCE
PEREECT
MANI
SCISSORS
RON
BPR
MELIDAA
Figure 5: Qualitative results on 1o0keLens-100k. We compare the ranking obtained using LightGCN trainod with the baseline BPR loss (bottor row) to the ranking obtained using ITEM (top row). Positive elements are highlighted in groen, while negative elements are indicated in red.
strategy to sample negatives, relying on the Personalized PageRank score. It leverages the graph structure of a local neighborhood around user: item with high PPR score for user will be more challenging distinguish for an MP-GNN, and enhance the learning performance.
Furthermore, we expanded the evaluation of GNNs by benchmarking on an inductive user-pplit protocol, which gauges the generalization capacity of GNNs. In both inductive and transductive protocols, we showcased the effectiveness our ITEM learning framework, exhibiting improvements over the conventional BPR loss function, even compared standard GNN rrining-enhanccmmen methods such as SGL-ED, SIMGCL, or MixGCF. Our findings demonstrate that using ranking losses to enhance the training of MP-GNNs is very promising research direction.
References
Uri Alon and Eran Yahav. On the Bottleneck of Graph Neural Networks and its Practical Implications. 2020. doi: 10.48550/arxiv 00055000/xxv22200000520000
Nabiha Asghar. Yelp dataset challenge: Review rating prediction. 2016.
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-AP: Smoothing the Path Towards Targo-Scale Image Retrieval. 2020.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Sebastian Bruch, Masrour Zoghi, Michael Bendersky, and Mare Najork. Revisiting approximate metric optimization in the age of deep neural networks. In ACM SIGIR, 2019.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deods, Nicole Hamilton, and Greg Hullender, Learning to rank using gradient descent. In ACM TCML, 2005.
Christopher Burges, Robert Ragno, and Quoc Le. Learning to rank with nonsmooth cost functions. In NeurIPS, 2006.
Zhe Cao, Tao Qin, Tie Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: From pairwise approach to listwisc approach. In ACM ICML, 2007.
Chong Chen, Min Zhang, Yongfeng Zhang, Yiqun Liu, and Shaoping Ma. Efficient neural matrix factorization without sampling for recommendation. ACM Transactions Information Systems, 38(2):14, 1 2020a. ISSN 15582868. doi: 10.1145/3373807 URL https://doi...... 10tr///..114//3338077
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and Relieving the Over-smoolhing Problem for Graph Neural Networks from the Topological View. AAAI 2020 34th AAAI Conference Artificial Intelligence, pp. 3445, 9 2019. ISSN 2159-5399. doi: 10.1609/aaai v3400..444. URL
https: fitps://arxivg wwaaxii.org//aee abw//9/9............. 03211122
Jiawei Chen, Junkang Wu, Jiancan Wu, Sheng Zhou, Xuezhi Cao, and Xiangnan He. Adap-Sktaal Adaptively Modulating Embedding Magnitude for Recommendation. ACM Web Conference 2023 Proceedings of the
World Wide Web Conference, WWW 2023, 111085-1096, 2023. doi: 10.1145/3543507. 45115/33353337733333333 URL http:
//arxiv. wwrpiiv.org/eee abs babs/23300 2302 www///4t::///.....oo. wtt.://////..1///555555. 1141//144///57.... 355457733350773333333
Lei Chen, Le Wu, Richang Hong, Kun Zhang, and Meng Wang. Revisiting Graph based Collaborative Filtering: A Linear Residual Graph Convolutional Network Approach. AAAI 2020 34th AAAI Conference on Artificial Intelligence, pp. 27- 34, 1 2020b. ISSN 2159-5399. doi: (0.1609/aaat) v34001 15330. URL htttps://arrttt /rg/
Abs/2001.101611 10167v1
Bruce Croft, Donald Metzler, and Trevor Strohman. Search engines: Information retrieval in practice, volume 520. Addison- Addison-Wesley Reading, 2010.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Justin Gilmer, Samuel S. Schocnholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Mcssage Passing for Quantum Chemistry. 2017. URL http://arxiv http::://aavv..rr/////////00000
Aditya Grover and Jure Leskovec. node2vec: Scalable Feature Learning for Networks. 7 2016. URL https
7/arxiv. wwwp/o//w///....o///////// httpa///100.00000 00653
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In IEEE CVPR, 2006.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs. 6 2017.
URL httts:///axiv org/ abs/////....//.//////1////////1/ 1706.006000 02216
F Maxwell Harper and Josoph A Konstan. The Moviclens datasots: History and context. ACM TIIS, 2015.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural Collaborative Filtering. 8 2017. URL httt:://aaxv..orrgee// abs/ httpb///77800033333
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. 2 2020. URL
Cheng Kang Hsich, Longqi Yang, Yin Cui, Tsung Yi Lin, Sorge Belongic, and Deborah Estrin. Collaborative metric learning. 26th International World Wide Web Conference, WWW 2017, pp. 193-201, 2017. doi: 10. N0.1145/3033912330522233
13

---

Published in Transactions on Machine Learning Research (07/2024)
Tinglin Huang, Yuxiao Dong, Ming Ding, Zhen Yang, Wenzheng Feng, Xinyu Wang, and Jie Tang. MixGCF: An Improved Training Method for Graph Neural Network-based Recommender Systems. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 665 674, 8 2021. doi: 14.1445/3447544.34644400
Kalervo Järvelin and Jaana Kekäläinen. Cumullated gain-based evaluation of IR lechniques. ACM TOIS, 2002.
Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks, 2016.
URL http:://arxiv.or//bbaa. 12090
Mu Li, Qilong Wang, David Zhang, Peihua Li, and Wangmeng Zuo. Joint distance and similarity measure learning based on triplct-based constraints. IS, 2017.
Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffmman, and Tony Jebara. Variational autoencoders for collaborative filtering. The Web Conference 2018 Proceedings of the World Wide Web Conference, WWW 2018, pp. 689 698, 2 2018. doi: 10.1145/317888763116133 URL lattpe://axxxit http:///aaaawwww.aaaa//a/aaa///e
05814
Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, and Xiugiang He. SimpleX: A Simple and Strong Baseline for Collaborative Filtering. International Conference on Information and Knowledge Management, Proceedings, pp. 1243 1252, 9 2021. doi: 00.145/3459933.3482225 URL https:
abs/ /aabs//110 2ttps/.........11222221
Zaiqiao Meng, Richard McCreadie, Craig Macdonald, and ladh Ounis, Exploring Data Splitting Strategies for the Evaluation of Recommendation Models. RecSys 2020, 7 2020. doi: 10.485500/rxxv.20000 200/aaaxxx.000.32220 URL
http:://axxvv.orgaaaaee a0tg//bbs2007..... 13237
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The PageRank Citation Ranking: Bringing Order to the Web. World Wide Web Internet And Web Information Systems, 54(1999-66), 1998. ISSN 1752-0509.
doi: 10.1.1.31.1688.
Yash Patel, Giorgos Tolias, and Jirí Malas. Recall@ k surrogale loss with large batches and similarity mixup. In IEEE/CVF CVPR, 2022.
Bryan Perozzi, Rami AI-Rfo,, and Steven Skiena. WeepWalk: Online Learning of Social Representations. 3 2014. doi: 00.1145/22233002222777 URL http:://arr...ooo//aa////3266612 httt:////bbbb///o..o/ttttt////t doi 0923330.2623733
www.:dwwoooooootooooooooooooooo. orgo 10///00.1111 1145/
Przemyslaw Pobrolyn and Radoslaw Bialobrzeski. NeuralNDCG: Direct optimisation of ranking metric via differentiable relaxation of sorting, 2021.
Tao Qin, Tic-Yan Liu, and Hang Li. A general approximation framework for direct optimization of information retrieval measures. Information Retrieval, 2009.
Eliss Ramzi, Nicolas Thome, Clément Rambour, Nicolas Audebert, and Xavier Bitot. Robust and decomposable average precision for image retrieval. NewrIPS, 2021.
Elias Ramzi, Nicolas Audebert, Nicolas Thome, Clément Rambour, and Xavier Bitot. Hierarchical average precision training for pertinent image retrieval. In ECCV, 2022.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. BPR: Bayesian Personalized 2218.2618
Ranking from Implicit Feedback. 5 2012. URL http:::/aawt...rr///////
Jerome Revand, Jon Almazán, Rafael Rezende, and Cesar Roberto de Souza. Learning with average precision: Training image retrieval with a listwise loss. In CVPR, 2019.
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. TGN: Temporal Graph Networks for Deep Learning on Dynamic Graphs. Ar 2020, 6 2020. URL httpps //aaxiv.org/aas////
14

---

101///..114///21811... Published in Transactions on Machine Learning Research (07/2024)
Jianing Sun, Yingxue Zhang, Wei Guo, Huifeng Guo, Ruiming Tang, Xiuqiang He, Chen Ma, and Mark Coates. Neighbor Interaction Aware Graph Convolution Networks for Recommendation. SIGIR 2020 Proceedlings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1298, 7 2020. doi: /1114453397271.3444
URL https:://dd :/dl. ////l/acc.....///////// acm. www.aac...oor........ org/ h/t///di/1....o 1145/3397211111
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. LINE: Large-scale Information
Network Embedding. WWW 2015 Proceedings of the 24th International Conference World Wide Web, pp. 1067-1077, 3 2015. doi: 10. 20.1144/2332277.77110555 URL httpp://arraa.orrr///////....... /dii htg/10..145/2736277 27411
Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. Softrank: Optimizing non-smooth rank metries. In WSDM, 2008.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph Attention Networks. 10 2017. URL http:://arxiv.org/ab//////
Xiang Wang, Xianguan He, Meng Wang, Fuli Feng, and Tal-Seng Chua. Neural Graph Collaborative Filtering, 2019. dot:
URL
Xiang Wang, Hongye Jin, An Zhang, Xiangnan He, Tong Xu, and Tat-Seng Chua. Disentangled Graph Collaborative Filtering. SIGIR 2020, 7 2020. doi: 00.145/3399771.3401111 URL http://aaraav...org//b///0/0//// 01764
Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised Graph Learning for Recommendation. SIGIR 2021 Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 726 735, 10 2021. doi: 10.1145/34048355446288800 URL /ttps://arxiv, org/ abt///a...220... 10783
Keyulu Xu, Waihua Hu, Jure Leskovee, and Stefanic Jegelka. How Powerful are Graph Neural Networks? 10 2018.
URL https: http://axxvvvvo wttp://rr/aab//1...000e533
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph Convolutional Neural Networks for Web-Scale Recommender Systems (PinSage). 6 2018, doi: 10.1145/3219819, 3219890. URL http:://aaall..ogg/a//////0///01
Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui, Nguyen Quoc, Viet Hung, and Hongzhi Yin. XSimGCL: Towards Extremely Simple Graph Contrastive Learning for Recommendation. 9 2022. URL https: abs// 2209. 0254444
Yisong Yuc, Thomas Finlcy, Filip Radlinski, and Thorsten Joachims. A support vector method for optimizing average precision. In ACM SIGIR, 2007.
Bangzuo Zhang, Yu Guan, Haichao Sun, Qingchao Liu, and Jun Kong. Survey of user behaviors as implicit feodback. In 2010 International Conference on Computer, Mechatronics, Control and Electronic Engineering, volume 6, pp. 345-22 348, 2010. doi: N0.1109/MMCE.200005000
A
Method
In this scction we define several ranking- motrics usod evaluate recommender systems Appendix A. Appendix A.22 we show how use the rank approximation of Eq. (1) of main paper define other differentiable ranking-based losses.
15

---

Published in Transactions on Machine Learning Research (07/2024)
A.1 Metric definition
We remind here the definition of other well-known ranking-based metrics, used to evaluate recommender systems:
# of positive in the top-k min(D>+{{.))
Rok=
Σ
H(k ((kkrrnk(p))
(5)
min([V+1,))
PeVt
rank + (p)
AP
with vank+(0)=
Σ
M(s)-18)
(6)
D+
rank(p)
PEV+
JEV+
A.2 Other ranking based losses
In this section using the differentiable rank approximation of Eq. (1) of the main paper we define other differentiable
ranking-basee losses. In Appendix B.4 we show the choice of the metric used during training impacts the evaluation metric.
The differentiable Average Precision loss defined as follows QQn al. (
Brown al. 2020
rank, (p)
CP
with rank (p,T) ra)k,(p)
=+
Σ (18--88p - 8p.T)
|+1
DEV+
jeV+
Note that we use the approximation of Eq. (1) of the main paper to approximate both rank and rank" + using the same temperature 7 parameter.
Finally, Patel et al. (2022 defines an approximation the Rãk as follows:
Bbikk TTEM
Σ ((kk-rank(())7"") DEVI
KEK min((++
In (Patel et al. (2022) the loss uses different level of recalls, ie. for in KC, it is necessary to provide enough gradient signal to all positive items. To train Chaa is also necessary to approximate a second time the Heaviside function, using a sigmoid with temperature factor r* Note that one other interesting property of both CITEM and LITEM is that they only require select hyper-parameter, T.
B
Experiments
B.1
Datasets
We evaluate our approach in the implicit feedback recommendation task Zhang et. al. (2010), simulating real-world scenarios where explicit feedback is too costly. The relationships are binary: if user interacts with item, it indicates their appreciation, represented by a link in the graph. We validate our method on four recommendation datasets. We use the latest small MovieLens- dataset (Harper & Konstan (2015)) in the inductive settings. It features relations between users and movies, where users give ratings to different movies. For implicit foedback, we keep relations with ratings
or higher. We use the 2018 edition the dataset from the Yelp challenge [Asghar (2016)). Rated items are bars and restaurants, we follow (Wang et. al. (2019); He et al. (2020)) to get implicit feedback. We also evaluate ITEM on the Amazon-Book dataset from Amazon-revieww, we use the same pre-processing NGCF (Wang et al. (2099) and LightGCN [He et al. (2020)). For the three datascts apply pre-processing to make sure that we have the same items for the training and for the evaluation. We follow Wang et al. (2019) and only keep users that have at least. 10 interactions. Tab. in main paper presents the datasets statistics.
16

---

Published in Transactions on Machine Learning Research (07/2024)
B.2 Implementation details
All models trained using the Adam optimizer, with learning rate in {0.01, 0.001]. On the inductive setting, use 7=10, and 1.5 on the transductive setting. We sum-pooling and embeddings of dimension 200 for the inductive setting on Yelp and MovieLens- 100k and embeddings of dimension 64 for Amazon-Boook sum-pooling except for LightGCN (He et al. (2200) with mean-pooling and embeddings of dimension 64 for the transductive setting. On both settings, we sample positives and 200 negatives for ITEM for Yelp2018, MovieLens-Iiee and MovieLens-Iie 1M and 600 negatives for AmazonBook. We use, on both protocols, batch sizes of 512 for MovieLens- 100k, MovieLens- 1M, and 2048 for Yelp-2018 and Amazon-book. For the inductive setting, we found experimentally that learning user embeddings during training is harmful for the generalization of the GNNs to new users, for training and evaluation users' embeddiings are inferred using message passing only. For GIN (uu et al. (0118) GAT (Veličković et al. (2017)) and GCN (Kipl & Welling (2016)) we use the framework Pytorch Geometric
Fey & Lenssen 20199) to implement the models.
B.3 Ablation studies
Table 7: Comparison of our ranking-bess based loss ITEM vs the BPR loss (Rendle al. (012)), using different GNN architecture on induclive benchmarks. For each architecture best results is bold, best. overall results underlined.
MovieLens-1000k
Yclp-2018
Amazon-book
Method
R@20
NDCG@20
R@20
NDCG@20
R@20
NDCG@20
BPR
28.74
27.68
7.34
5.76
8.85
7.61
GCN
CITEM
32.24
31.44
10.06
8.35
10.23
8.82
%1mprov.
+12.2%
+13.5%
+37.1%
+-45%
+15.6%
+15.9%
BPR
29.71
27.58
8.25
6.74
9.62
8.05
GIN
CITEM
32.00
29.87
10.57
8.79
13.05
11.59
%Improv.
+7.7%
+8.3%
+28.5%
+30.4%
+35.7%
+ 44.0%
BPR
31.01
28.92
9.04
7.32
9.88
8.17
GAT
CITEM
32.04
30.54
11.43
9.58
11.70
10.00
%1mprov.
+3.3%
+5.6%
+26.4%
+30.9%
+18.4%
+22.4%
BPR
30.79
29.73
7.88
6.34
9.56
8.02
LGCN
CITEM 33.13
32.07
9.88
8.32
10.66
9.52
%Improv.
+7.6%
+7.9%
+25.1%
+31.2%
+11.5%
+18.7%
B.3.1
Inductive loss comparison
In Table we compare in the same settings the BPR loss (Rendle (2012 and our proposed ranking-based loss ITEM. We show that on the three inductive benchmarks, and across all four considered architect ITEM outperforms the BPR loss. On MovieLens 100k and with the best performing architecture, LightGCN (He et al. (2020)), ), our loss outperforms the BPR loss with +2.34 R@20 and +2.34 NDCG@20. The relative improvements are also significant, ranging from +3.3% NDCG@20 for GCN, up to 12.2% NDCG@20 for GCN. On Yclp-2018 which is a large scale dataset, GAT is the best performing architecture. ITEM outperforms the BPR loss by +2.39 R@20 and +2.26 NDCG@20, We can point out the considerable relative improvements on kelp-2018, which are always larger than 25% and reaching 45% over GCN for NDCG@20. Finally on Amazon-book the best overall results are obtained with GIN using ITEM. It outperforms BPR and GIN by +3.4 R@20 and 3.5 NDCG@20 which huge relative improvements Amazon-book our most large scale dataset. Note that accros the three
17

---

Published in Transactions on Machine Learning Research (07/2024)
considered datasets different GNN architectures work best, LightGCN (ee et al. (2020) on MovieLens-1000: GAT Veličković et al. (2017) on Yelp-2018 and GIN (Xu et al. (2018)) on Amazon-book. Overall, Table 7 similarly Tab. 4 in the main paper shows the interest of optimizing a ranking- based loss, LITEM, rather than the BPR loss, i.c. prozy loss.
B.3.2 Negative sampling
In Table S show the impact of our PPR negative sampling strategy in the inductive setting. We show that the three inductive benchmarks using our sampling strategy boost the performances of our trained GNN model over the random negative sampling (RNS), e.g. +0.66 R@20 and +0.4 NDCG@20 on Yelp-2018.
Table 8: Comparison of Random Negative Sampling (RNS) vs the sampling method in ITEM (PPR). We use a LightGCN (He et al. 2020 backbone for the three inductive benchmarks.
MovieLens-
Yelp-2018
Amazon-book
Loss
Sampling
R@20
NDCG@20
R@20
NDCG@20
R@20
NDCG@20
RNS
33.13
32.07
9.88
8.32
10.66
9.52
LITEM
PPR (ours)
33.84
32.63
10.54
8.70
11.03
9.89
B.3.3 Cold-start evaluation
we evaluate our model cold-start sctting with only a small proportion training data available for each user (10%, 20%, 50%). Our method, which uses an approximation of NDCG and PPR-based negative sampling, shows significant. robustness compared to the baseline LightGCN with BPR loss.
In Table D
Table 9: Cold-start evaluation in transductive settings on MovieLens-Ii 1M with different amount of training data (%).
10%
20%
50%
Method
R:020
NDCG@20
R@20
NDCG@20
R@20
NDCG@20
LGCN-BPR
15.88
15.43
20.05
19.18
24.19
23.36
ITEM
17.11
16.89
22.37
21.68
29.03
28.20
B.4
Model analysis
B.4.1
Metric optimization
compare the benefits of optimizing different ranking- based losses MovieLens-100k (inductive) using LightGCN model. Specifically, BPR loss as a baseline. First we can note that on each metric all our ranking- based losses outperform the BPR loss. We can see that, for each metric, using its smooth approximation to optimize a GNN during training yields a higher score on this target metric. CAP, yields the best score of 19.05 AP, CRÆK, yields the best score of 33.37 R@20 (outperforming the results reported in Table 7 Finally LNDGG, yiclds the best score for both NDCG and NDCG@20 of 53.55 and 32.07 respectively.
In Table 10
compare the optinization of AP, Rãk and NDCG, and include the
B.5 Qualitative Results
We display on Fig. D the top-20 items retrieved when using ITEM on MovieLLens-100k and on Fig. the top-20 items retrieved when using the baseline BPR loss (Rendle et al] (2012)). Both models are LightGCN (He et al. (2020)) ). We can observe qualitatively that ITEM brings more positive results, and leads to a better ranking than the BPR loss.
18

---

Published in Transactions on Machine Learning Research (07/2024)
Table 10:
Performances different ranking-basse losses on the MovieLens- 100k inductive benchmark, and the
BPR loss
Rendle al. (2012)
baseline. The model used is LightGCN He et al. 2020
Loss
AP
NDCG
R@20
NDCG@20
BPR
16.56
50.68
29.59
28.24
ITEM
CAP
19.05
53.48
32.61
31.91
18.57
52.93
33.37
31.82
CITEM
18.94
53.55
33.13
32.07
ITEM
User query
rank 1
rank
rank 3
rank 4
rank 5
CINEMANIA
Ki
CONED
THE WOMAN IN GREEN
Bullets Broodwayy
BASIL RATHBONE NIGEL BRUCE
SEREECT
rank 6
rank
rank 8
rank 9
rank 10
BRut
t
Nice
m mina
HOTFUZI
RUSH YOUR
MAT FAP PwA.CRAVV....
rank 11
rank
rank 13
rank
rank 15
Andre
KNOW IT
AS WE
Taie Nia
JOHN WOO LAST HURRAH for CHIVALRY
Frank Sinetra Harvey
Betty Fisher
The Manchurian Candidate
rank 16
rank
rank 18
rank
rank 20
CAPTAIN RON
AORCO DECRUR MONTY DVTHON
DOROTHY McGUIRE JOHN GARFIELD
Gentleman's Aóreement
WOMEN
PYTHON
Figure 6: Top-20 results MovieLens-1000. (inductive) when using LightGCN and ITEM.
19

---

Published in Transactions on Machine Learning Research (07/2024)
BPR
User query
rank 1
rank 2
rank 3
rank 4
rank 5
INEMANIA
SCISSORS
CAPTAIN RON
MELINDA
MILNNM
rank 6
rank 7
rank 8
rank 9
rank 10
B
BACK
THE WOMAN IN GREEN
BASIL RATHBONE NIGEL BRUCE
Betty Fisher
dkg Wiluble Files
rank 11
rank 12
rank 13
rank 14
rank 15
KNOW
AS WE
JONO
CONVERSATION WOMEN
rank 16
rank 17
rank 18
rank 19
rank 20
SAMUE BRNNSNN CHAILTIO DESOON SOPELA LOBEE
dior WEDDING
ICK
CAD
PERFECT
Figure 7: Top-20 results on MovieLens-100 (inductive) when using LightGCN and BPR loss.
20


Reviews:
[{"review_id": "EP7uWhXOUE", "reviewer": "Reviewer_iMQg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The idea of using list-wise ranking loss function to directly optimize the evaluation metric makes a lot of sense. The authors successfully proposed a loss function and proved its usefulness.\n2. The authors conducted extensive experiments and analysis to demonstrate the effectiveness of their framework, which makes this paper  empirically sound.\n3. The authors adapted inductive user split protocol in their evaluation, which is important but has been overlooked a lot.\n\nWeakness:\n1. It would be great to have more discussions of the connection and comparison of the proposed idea v.s. existing work. For example, BPR is used as an important baseline in this work. In some sense, the proposed idea is similar to a transformation of BPR. What are the connections/similarities between them, what makes the proposed ideas more effective than BPR? Meanwhile, I did not see much discussions of existing ranking-based loss function and their differences to the proposed one. These discussion can help readers better understand the landscape of related research and better appreciate the proposed loss function.\n2. It needs some justification about why the authors only consider 0/1 feedback in their loss function. In some cases, e.g. product/movie recommendation, the users may provide rating feedback other than 0/1.\n3. The inductive user-split protocol is a great extension to evaluation. However, it does not cover the cold-start cases, where a user only have very few items. However does the framework work for cold-start users?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors made the following major contributions in this paper: \n1. They proposed a ranking loss function to directly optimize NDCG in learning MP-GNN for recommendation while most existing work use a proxy (e.g. BPR).\n2. They proposed to use Personalized PageRank for negative sampling, which is computationally more efficient and generate negative samples of higher quality, which eventually boosted the performance of their model.\n3. They extended evaluation by benchmarking on an inductive user-split protocol, which better gauges the generalization ability of the model.\n4. They conducted extensive experiments and compared with multiple existing baselines to prove the effectiveness of their framework. They also conducted thoughtful ablation experiments to understand the impact of each of their proposed ideas.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This work is solid and it just needs some minor changes to get even better. I suggested the author considering making the following changes\n1. Add some discussion of similarity/connections and differences v.s. BPR and other exiting ranking-based loss function.\n2. Add more details of Personalized PageRank for negative sampling for replication purpose. There is also one interesting thing worth discussing, as we all know, negative samples are not strictly negative as users may not be exposed to them yet. Does PPR increase the risks of picking items users interested to but not exposed to yet? Or does it decrease the risk? Why?\n3. Add more details of the used dataset. What type of user feedback does each of them have? 0/1 or ratings? If feedback are ratings, how to transform them to 0/1 feedback?\n4. Add some evaluation & analysis for cold-start users if possible?\n5. Correct the citation error at the end of Paragrah 1, Page 8.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RBuDxkqjTk", "reviewer": "Reviewer_2cdS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Recommender systems, as one of the most important applications of machine learning, well matches the scope of TMLR.\n- The authors consider four widely-used datasets for evaluation.\n- The proposed method achieves the best performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of recommender systems with graph neural networks. Specifically, the authors consider that the evaluation metrics can be directly optimized as loss functions in graph neural networks. The authors conduct experiments on four datasets, showing the best performance of the proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- The motivation is not enough. Optimizing the loss function directly is not a good choice in real-world recommender systems since metrics are too many. Furthermore, there are more metrics than NDCG. What about the performance when using other metrics?\n- The baselines are out-of-date. There is no baseline published in 2023. As we know, the recommendation models are developing very fast.\n- The method is relatively straightforward. There is no tailored solution for recommender systems, while it is only an adaption from general graph neural network-based approaches.", "broader_impact_concerns": "There is no concern on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xhhobo6ZsW", "reviewer": "Reviewer_jWmJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* Multiple recommendation with implicit feedback is a relevant problem for many applications.\n* Rank function approximation enables direct optimization of the target utility function, and works over a wide range of temperature values.\n* Experiments are very advanced.\n\n### Notes\n* The effect of selecting negative examples through PPR could be both beneficial and detrimental. On the one hand, we're highlighting items that are close to the decision frontier, and which therefore help the learning process. But, on the other hand, it forces the learned model to rank these items as negatively as possible, even though they are potentially the most relevant items for the user among those he hasn't considered so far. This aspect is all the more important because, at the same time, items that are much further away from the user are rarely selected by PPR, so the learned model is less encouraged to reject them.\nHowever, experiments show that the beneficial aspect prevails, and the fact that negative examples are randomly drawn means that distant examples can be selected from time to time.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of multiple recommendations with implicit feedback. It improves the learning phase in two ways. First, it uses a smooth approximation of the object rank computation to optimize a cost function that directly approximates the target ranking metrics. Secondly, the paper uses Personalized PageRank (PPR) to randomly draw negative examples that are as informative as possible. The corresponding approach is extensively evaluated experimentally, both in a transductive context (the training base contains part of the returns for each user), and in an inductive context (some users are not part of the training base).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In Section 3, paragraph \"Training Context\", I guess the embedding $h$ is for any node in $\\cal{U} \\cup \\cal{V}$ instead of $\\cal{U} \\times \\cal{V}$.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The comments by the reviewers were generally positive, reporting negative aspects mainly concerning presentation, motivation, and some additional justification. One reviewer asked for more experimental evidence involving more recent approaches, which authors provided, although the reviewer seems not to become aware of this addition to the experimental assessment. In the updated version, authors also responded in a satisfactory way to the comments by the other two reviewers. Overall, the contribution of the paper is relevant, timely, and solid.
Decision: Accept (poster)