Graph-enhanced Optimizers for Structure-aware Recommendation Embedding Evolution
Authors: Cong Xu
Jun Wang
Jianyong Wang
Wei Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Embedding optimization
Recommender systems
Graph
Optimizers
Primary area: optimization_for_deep_networks

Abstract: Embedding plays a key role in modern recommender systems because they are virtual representations of real-world entities and the foundation for subsequent decision-making models.  In this paper, we propose a novel embedding update mechanism, Structure-aware Embedding Evolution (SEvo for short), to encourage related nodes to evolve similarly at each step. Unlike GNN (Graph Neural Network) that typically serves as an intermediate module, SEvo is able to directly inject graph structural information into embedding with minimal computational overhead during training.  The convergence properties of SEvo along with its potential variants are theoretically analyzed to justify the validity of the designs.  Moreover, SEvo can be seamlessly integrated into existing optimizers for state-of-the-art performance. Particularly SEvo-enhanced AdamW with moment estimate correction demonstrates consistent improvements across a spectrum of models and datasets, suggesting a novel technical route to effectively utilize graph structural information beyond explicit GNN modules.

Graph-enhanced Optimizers for Structure-aware Recommendation Embedding Evolution
Cong Xu Jun Wang Jianyong Wang
Wei Zhang"*
*East China Normal University TTinghu University
[congrueric, wongjun, zhangwei.. thu2011/0gmail
"jianyon@tssinhua.c edu.
Abstract
Embedding plays key role modern recommender systems because they are virtual representations of real-world entities and the foundation for subsequent decision-making models. In this paper, we propose a novel embedding update mechanism, Structure-aware Embedding Evolution (SEvo for short), to encour- age related nodes to evolve similarly at each step. Unlike GNN (Graph Neural Network) that typically serves rectly inject graph structural information into embedding with minimal compu- tational overhead during training. The convergence properties of SEvo along with its potential variants are theoretically analyzed to justify the validity of the de- signs. Moreover, SEvo be seamlessly integrated into existing optimizers for state-of-the-ar performance. Particularly SEvo-enhanced Adam with moment estimate correction demonstrates consistent improvements across spectrum of models and datasets, suggesting novel technical route to effectively utilize graph structural information beyond explicit GNN modules. Our code is available https:/ htpps://gitaan http//ggtttttbccce com http///aaanHHSStt
intermediate module, SEvo is able to di-
Introduction
Surfing Internet leaves footprints such as clicks [15], browsing [5], and shopping histories [58]. For modern recommender system [6, 12], the entities involved (e.g., goods, movies) are typi- cally embedded into a latent space based on these interaction data. As the embedding takes the most to construct and is the foundation to subsequent decision- making models, its modeling qual- ity directly determines the final performance assumption [31, 59], it is natural to expect that related entities have closer representations in the latent space. Note that the similarity between two entities refers specifically to those extracted from interaction data [47] prior knowledge [37]. For example, goods selected consecutively by the same user or movies of the same genre are often perceived as more relevant. Graph neural networks (GNNs) [2, 10, 14] are a widely adopted technique exploit such structural information, in concert with a weighted adjacency matrix wherein each entry characterizes how closely two nodes are re- lated, Rather than directly injecting structural information into embedding, GNN typically serves as an intermediate module in the recommender system. However, designing versatile GNN module suitable for various recommendation scenarios challenging. This is especially true for sequential recommendation [8, 51], which needs to take into account both structural and sequential informa- tion the same time. Moreover, the post-processing fashion inevitably increases the overhead of training and inference, limiting the scalability for real-time recommendation.
the entire system. According to the homophily
In this work, we aim directly inject graph structural information into embedding through novel embedding update mechanism. Figure (a) illustrates a normal embedding evolution process, in
*Corresonndig author. This work was supported in part by National Natural Science Foundation of China (No. 92270119 and No. 22072182) and Shanghai Institute of Artificial Intelligence for Education.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Embadding Evolution
((E)) 1-8 3
E
Es
A'AEEE
(d)
g
E AE
(AR) urgnin (1 D]1l AEB + LA
E
AE
(a)
Structure-and aware Embedding Evolution
AE
E
Eg
(AE)
E + -)
VgC
AE
(b)
Figure/ I: Overview of SEvo. (a) Normal embedding evolution. (b) (Section 2) Structure- aware em- bedding evolution. (c) (Section 2.2) Geometric visualization of the variation from AE to e* (AE). The gray ellipse represents the region with proper smoothness. (d) (Section 2.3) The L-layer ap- proximation with faster convergence guarantee.
which the embedding E is updated at step / as follows:
E, E.-1 + AE,11
Note that the variation AE is primarily determined by the (anti-)gradien) It points to region able to decrease loss function concerning recommendation performance [35], but lacks an explicit mechanism to ensure that the variations between related nodes are similar. The embeddings thus cannot be expected to capture model pairwise relations while minimizing the recommendation loss.
Conversely, structural information can be effectively learned if related nodes evolve similarly at each update. The structure-aware embedding evolution (SEvo), depicted in Figure I (b), is developed for this goal. A special transformation is applied so as to meet both smoothness and convergence [57]. Given that these two criteria inherently conflict to some extent, resort to graph regularization framework [57, 7] to balance them. While analogous frameworks have been used to understand feature smoothing and modify message passing mechanisms for GNNs [28, 60], applying this to variations is not as straightforward as to features [24] or labels [19]. Previous efforts are capable of smoothing, but cannot account for strict convergence. The specific transformation form must be chosen carefully; subtle differences may slow down or even kill training. Through comprehensive theoretical analysis, we develop an applicable solution with provable convergence guarantee.
Apart from the anti- mati-gradient, the variation AE can also be derived from the moment estimates. Therefore, existing optimizers, such as SGD [41] and Adam [23], can benefit from SEvo readily. In contrast to Adam, Adam [27] decouples the weight decay from the optimization step. making it more compatible with SEvo as it is unnecessary to smooth the weight decay well. Furthermore, we recorrect the moment estimates of SEvo-enhanced AdamW when encountering sparse gradients. This modification enhances its robustness across variety of recommendation models and scenar- ios. Extensive experiments over six public datasets have demonstrated that it can effectively inject structural information to boost recommendation performance. important to note that SEvo does not alter the inference logic of the model, so the inference time is exactly the same and very little computational overhead is required during training.
The main contributions of this paper can be summarized as follows. 1) The graph regularization framework [57, 7] has been widely used for feature/label smoothing. To the best of our knowledge, we are the first to apply it to variations as an alternative to explicit GNN modules for recommenda- tion. 2) The final formulation of SEvo non-trivial (previous iterative [57] or Neumann series [39] approximation methods proved to be incompatible in this case) and comes from comprehensive theo- retical analyses. 3) We further present SEvv-enhanced AdamW by integrating SEvo and recorrecting the original moment estimates. These modifications demonstrate consistent performance, yielding average 9%-23% improvements across spectrum of models. For larger-scale datasets contain- ing millions of nodes, the performance gains can be as high as 40%~~ 13%%. 4) Beyond interaction

---

data, we preliminarily explore the pairwise similarity estimation based on other prior knowledge: node categories to promote intra-class representation proximity, and knowledge distillation [18] to encourage a light-weight student to mimic the embedding behaviors of a teacher model.
Structure-aware Embedding Evolution
In this section, we first introduce some necessary terminology and concepts, in particular smooth- ness. SEvo and its theoretical analyses will be detailed in Section 2.2 and 2.3. The proofs hereafter are deferred to Appendix A.
2.1 Preliminaries
Notations and terminology. Let V (Ui,,.., Un denote a set nodes and A = [way] E Raxx symmetric adjacency matrix, where each entry Waj Wij characterizes how closely e; and oy are related. They jointly constitute the graph 9 = (V. A). For example, could be set of movies, and Wy is the frequency of tú and vy being liked by the same user. Denoted by D E TRax the diagonal degree matrix of A, the normalized adjacency matrix and the corresponding Laplacian matrix are defined as A = D 1/2 AD 1/2 and L = I Á, respectively, For case of notation, we use (s.-) to denote the inner product, (x,
x² y for vectors and (X, Y) Tr(X" Y) for matrices. Here, Tr(-) denotes the trace of a given matrix.
Smoothness. Before delving into the details of SEvo. it is necessary to present a metric to measure the smoothness [57, 7] of node features X as a whole. Denoted by x,, Xj the row vectors of and d; Ej Waj. we have
Ismothhuess (:9) Tr(X = Eijjev Way
X
x;
This term is often used as graph regularization for feature/label smoothing [24, 19]. A lower I Jemoothness indicates smaller difference between closely related pairs nodes, and this case X considered smoother. However, smoothness alone is not sufficient from performance perspec- tive. Over-emphasizing this indicator instead leads to the well-known over-smoothing issue [26]. How to balance variation smoothness and convergence is the main challenge to be addressed below.
2.2 Methodology
The entities involved in recommender system are typically embedded into a latent space [6, 12], and the embeddings in E E Rnxd are expected to be smooth so that related nodes are close to each other. As discussed above, E is learnable and updated at step by Eq. (1), where the variation AE is mainly determined by the (anti- ggrddettr For example, AE - V W when gradient descent with a learning rate of 1y is used to minimize a loss function However, the final embeddings based on this evolution process may be far from sufficient smoothness because: 1) The variation AE points the region able to decrease the loss function concerning recommendation performance, but lacks an explicit smoothness guarantee. 2) As numerous item embeddings (millions of nodes in practice) be trained together for recommender system, the variations two related nodes may quite different due to the randomness from initialization and mini-batch sampling.
We design special transformation vi(+) to smooth the variation so that the evolution deduced from the following update formula is structure-aware,
E, < E, wh (AE.1).
(3)
Recall that, in this paper, the similarity is confined to quantifiable values in the adjacency matrix A, in which more related pairs are weighted higher. Therefore, this transformation should encourage pairs of nodes connected with higher weights evolve more similarly than those connected with lower weights. This can be boiled down to structure-aware transformation as defined below.
Definition (Structure-awarr transformation). The transformation (-)) is structure-aware if
T Tamoothneas (1b AE)) S T Jemoothness AE).
(4)
3

---

On the other hand, the transformation must ensure convergence throughout the evolution process, which means that the transformed variation should not differ too much from the original. For the sake of theoretical analysis, the ability to maintain the update direction will be used to qualitatively depict this desirable property below, though quantitative squared error will be employed later.
Definition "Direction-awarr transformation).
The transformation 15(-) direction-aware if
(b(EE) AE > 0, AAE #
(5)
These two criteria inherently conflict to extent. We resort to hyperparameter 3 E [0, 1) to make a trade-off and the desired transformation the corresponding minimum; that is,
(AE; 3) argmin (1 3) || AEI (T<AA
(6)
larger 3 indicates a stronger smoothness constraint and 4 (AE) reduces to AE when / -> 0. Geometrically, as shown in Figure (c), (AE) can be interpreted the region with proper smoothness. Taking the gradient to zero could give a closed-form solution, but it requires prohibitive arithmetic operations and memory overhead, which is particularly time- consuming in recommendation due to the large number of nodes. Zhou et al. [57] suggested L-layer iterative approximation to circumvent this problem (with do (AE) AE):
projection of ΔE onto
Water (AE) == db.. (AE), (AE) = 3A (AE) + (1 B)
The resulting transformation is essentially a momentum update that aggregates higher-order infor- mation layer by layer. Analogous message-passing mechanisms have been used in previous GNNs such as APPNP [24] and C&S [19]. However, this commonly used approximate solution is incom- patible with SEvo; sometimes, variations after the transformation may be opposite to the original direction, resulting in failure to converge.
Theorem 1. The iterative approximation is direction-awart for all possible normalized adjacency matrices and T. 2 0, if and only if B < 1/2. In contrast, the Neumann series approximation Gna (AE) = (1 - 3) 1 A'A is prructure-awar and direction aware for any E [0, 1).
As suggested in Theorem 1, a feasible compromise for Viter restrict 8 to [0, 1/2), but this may cause a lack of smoothness. The Neumann series approximation [39] appears to be a viable alternative as it qualitatively satisfies both desirable properties. Nonetheless, this transformation can be further improved for a faster convergence rate based on the analysis presented next.
2.3 Convergence Analysis for Further Modification
In general, the recommender system has some additional parameters 0 E RT except for embedding be trained. Therefore, analyze the convergence rate of the following gradient descent strategy:
E + E. nrunno (VRC(E,, O.)). O + O, 1/- v'Vo,(E,, 0.),
wherein SEvo is performed the embedding and normal gradient descent is applied to 0. To make the analysis feasible, some mild assumptions on the loss function should be given: L Ra x mmm + R is twice continuously differentiable function whose first derivative is Lipschitz continuous for some constant C. Then, we obtain the following properties.
Theorem 2 (Informal). If n = r/ = 1/C, the convergence rate after T updates is OCC/((1- 3)T))) we adopt modified learning rate of 1 (1-81+13C for embedding, the convergence rate could be improved to OCC/((1- 1001--311 --T)))
Remark 1. Our main interest is to justify the designs of SEvo rather than to pursue a particular
convergence rate, so some mild assumptions suggested in [32] are adopted here. By introducing the steepest descent for quadratic norms [1], better convergence can be obtained with stronger assumptions.
Two conclusions can be drawn from Theorem 2. 1) The theoretical upper bound becomes worse when 3 + 1. This makes sense since Ensa (V C) is getting smoother and further away from the

---

original descent direction. 2) A modified learning rate for embedding can significantly improve the convergence rate. This phenomenon can be understood readily if we notice the fact that
ne (AE) S (1 -1-28 (1-
Thus, the modified learning rate is indeed to offset the scaling effect induced by SEvo. In view of this, we directly incorporate this factor into SEvo to avoid getting stuck in the learning rate search, yielding the final desired transformation:
1- (AE; B) = T BL+1
Σ 8 A'AE
It can be shown that & is structure-aware and direction direction-aware, aware, and converges to v as L increases.
2.4 Integrating SEvo into Existing Optimizers
SEvo-enhanced Adam W. Differences from the original AdamW are colored in blue. The matrix operation below are element-wise Input: embedding matrix E, learning rate momentum factors B1, 11,82, 3 E [0, 1), weight decay A.
Algorithm 1:
foreach step t do
G. < VgC;
// Get gradients w.r.t
Update first/second moment estimates for each node à:
M,M, [i] + (1 81 [
if [ otherwise
M. <
M.M.
[
+ 1-
(1-81
M,-1
82 V - +
(1 32 G
if [. otherwise
V
V
1- 1
(1-8)
Compute bias-corrected first/second moment estimates:
M, +
V, + ve/(1 83);
Update via SEvo:
E, E 1 n
+ (
M
- -1/AE,1-
Output: optimized embeddings E.
SEvo can be seamlessly integrated into existing optimizers since the variation involved in Eq. (3) can be extended beyond the (ardie)gr. For SGD with momentum, the variation becomes the first moment estimate, and for Adam this is jointly determined by the first/second moment estimates. AdamW is also widely adopted for training recommenders. Unlike Adam whose moment estimate a mixture of gradient and weight decay, AdamW decouples the weight decay from the optimiza- tion step, which is preferable since it makes sense to require the weight decay to be smooth as well. However, very rare cases, ascertain the causes and then improve the robustness of SEvo-enhanced AdamW.
Adam W fails to work very well. We next try to
Denoted by g v.L Rd the gradient for a node embedding and g2 == g @ g the element-wise square, Adam estimates the first and second moments at step t using the following formulas
m, 3 m; + (1 )g,-1, V. = 822 + (1 - 32)g2-1
where $1, 32 are two momentum factors. Then the original AdamW updates embeddings by
e, e 7 Ae,-1, Ae, m
Note that the bias-corrected estimates m, = m./(1 34) and v, = v, 3)) are employed for numerical stability [23]. In practice, only fraction of nodes are sampled for training in a mini- batch, so the remaining embeddings have zero gradients. In this case, the sparse gradient problem may introduce some unexpected 'biases' as depicted below.
5

---

Proposition 1. If node is no longer sampled in subsequent batches after step t, we have A0t+p-1 = R PT A and the coefficient of is mainly determined by
Considering a common case 82 R 81, the right-hand side approaches (( The step size for inactive nodes then gets slower and slower during idle periods. This seems reasonable as their moment estimates are becoming outdated; however, this effect sometimes prevents the variation from being smoothed by SEvo. We hypothesize that this is because SEvo itself tends to assign more energy to active nodes and less energy to inactive nodes. So this auto-attenuation effect of the original A is somewhat redundant. Fortunately, there is a feasible modification to make SEvo-enhanced A more robust:
Theorem 3. Under the same assumptions as in Proposition I, Acttp-1 Aci- if the moment estimates are updated in the following manner when g: = 0,
m = - + +(1 (1) 1- 89-1 m Vi = 32 (1 82)
Vt-1-
(8)
1 B32
As can be seen, when sparse gradients are encountered, the approach in Theorem is actually to estimate the current gradient from previous moments. The coefficients 1/(1 (111-511 ) and 1/(1
( are used here for unbiasedness (refer Appendix A.3 for detailed discussion and proofs). We summarize the SEvo-enhanced AdamW in Algorithm 1 and the modifications for Adam and SGD in Appendix B.1, with an empirical comparison presented in Section 3.3.
The previous discussion lays the technical basis for injecting graph structural information, but the final recommendation performance is determined by how 'accurate' the similarity estimation is. Fol- lowing other GNN-based sequence models [48, 49]. the number of consecutive occurrences across all sequences will be used as the pairwise similarity way. In other words, items i and " that appear consecutively more frequently are assumed more related. Notably, we would like to emphasize that SEvo can readily inject other types of knowledge beyond interaction data. We have made some preliminary efforts in Appendix C and observed some promising results.
3 Experiments
In this section, we comprehensively verify the superiority of SEvo. We focus sequential rec- ommendation for two reasons: 1) This is the most common scenario in practice; 2) Utilizing both sequential and structural information is beneficial yet challenging. We showcase that SEvo is promising way to achieve this goal. It is worth noting that although technically SEvo can be ap- plied to general graph embedding learning [4], we contend SEvo-Adam is especially useful for mitigating the inconsistent embedding evolution caused by data sparsity, while effectively injecting structural information in conjunction with other types of information.
Due to space constraints, this section presents only the primary results concerning accuracy, effi- ciency, and some empirical evidence that supports the aforementioned claims. We begin by intro- ducing the datasets, evaluation metrics, baselines, and implementation details.
Datasets. Six benchmark datasets are considered in this paper. The first four datasets including Beauty, Toys, Tools, and MovieLens-IM 1M are commonly employed in previous studies for empirical com- parisons. Additionally, two larger-scale datasets, Clothing and Electronics, are used to assess SEvo's scalability in scenarios involving millions of nodes. Following [22, 13], we filter out users and items with less than 5 interactions, and the validation set and test set are split in a leave-one-out fashion, namely the last interaction for testing and the penultimate one for validation. This splitting allows for fair comparisons, either for sequential recommendation or collaborative filtering. The dataset statisties presented in Table 1.
Table 1: Dataset statistics
Dataset
#Users
#Items
#Intcractions
Avg.
Beauty
22,363
12,101
198,502
8,9
Toys
19,412
11,924
167.597
8.6
Tools
16,638
10,217
134,476
8.1
Moviel ens- IM
6,040
3,416
999.611
165.5
Electronics
728,489
159.729
6,737.580
9.24
Clothing
1,219,337
376,378
11,282,445
9.25
Evaluation metrics. For each user, the predicted scores over all items will sorted in descending order to generate top-N candidate lists. We consider two widely-used evaluation metrics, HR @

---

Table 2: Overall performance comparison. The best baselines and ours marked in underline and bold, respectively. Symbol A % stands for the relative gap between them. Paired t-test is performed over 5 independent runs for evaluating p-value (s 0.05 indicates statistical significance). 'Avg. Improv.' for each backbone depicts average relative improvements against the baseline.
LightGCN SR-GNNN LESSR MAFRee
GNN-
MF-BPR +SE
GRUGRa
MF
RNN//hanstormer-e
SASRe:
BFRT-RRo
STOSA
p-tulse
+SEve
-SEve
15E1e
+5850
HR
0.0074
0.088
0.013
00.001
0.0076
(0.005
0.009
(0.0120
0.0154
(10157
0.0172
0.0066
0.0266
30.2%
690F-05
1RR85
0.0299
0.0257
0.0222
0.0124
0.0272
0.0295
0.0233
0.0326
0.0404
0.0499
0.05%
0.0522
0.11
0.0884
13.5%
7 1.99C-0 04
HR 10
019472
0.0416
0.506
0.862
0.9454
0.3480
0.1395
0.0524
0.1634
0.0759
00776
0.0722
00680
0J0774
8.25
2.66E-01
NDKXGAA5
00181
0.0132
0,0205
0.0209
0.0070
0,0184
0.0146
0.0309
0.1262
0.0328
011319
0.0330
00032
0J0383
17.2%
9.70E-40
NEXXI 10
0.0240
0.0003
0.0264
0.0146
0.0228
0.0344
0.0108
0.0273
0.0336
0.04 1
0.8905
0.0430
0.0201
0.9457
15.5%
8 5.02F-055
HR
0.0677
0.0100
0.0126
0.0.1
0.0009
0.0099
0.0059
0.0080
0.0172
0.0192
0.0600
0.0175
0.020
0.0267
15.3%
1 1-46--03
Togs
HRS
I 10
0.0:56
0.220
0.0294
0.0439
0.511
00352
00099
0.0532
0.0027
0.0067
0.0477
0.0306
0.0345
0.0759
0.0446
0.0276
0,0727
0.0506
0,084:
0.0584
010400
0.0645
0.0723
0.0492
0037
0.776
0.0625
0.0872
9.5%
9.65%
3.07C 04
5.70F-03
NDOG@55
0.0183
0.0198
0.0240
00035
0.0074
0,0203
0.0134
0.017
0,0144
0.0392
0.0297
0.0336
0.0005
0.0453
11.6%
2.48C- 01
NDCX30900 T0
0.1240
0.0245
0,0292
034400
0300225
0.0055
0.0177
0.0234
0.0413
0.0475
011366
0.0430
0.0472
0J0532
12.7%
3.12E-05
HR
0/8667
0.0066
0.0045
0,033
0.0005
0.0071
0.0053
0.005
0.0099
010108
QUB0744
0.0187
0.0015
0J0133
33.8%
1.238-02
Toxls
HRSS
HR to
0.0325
0.0212
0.0260
0.0162
0.0263
0.0177
0,0311
00033
0.0087
0.0383
0.0148
0.0225
0.0174
0.0272
0.03%
0.0308
0.0317
11AAP
0.0337
0.0497
0.0244
0.0405
0.044
0.0279
0.0276
0.3417
0JA582
0.0350
10.5%
7.7%
8 6.94--03
8 6.25--01
NDCG@55
0.1400
0.0103
00101
0,0177
0.0123
0.0147
0.0114
0.0135
0.02:0
0.0225
0.0159
0,0183
0.0186
0.0244
15.9%
5.570 03
NDCG@:10
0.0176
0.0135
0.0135
0.0226
0.0157
0.0187
0.0145
0.0174
0.0238
0.0274
0.0211
0.0235
0.0251
0.0293
13.4%
3,790 03
0.0244
0,0083
0.8133
0,0429
0.0117
0.0133
0.0487
0.0487
0.0490
0.0517
0.0681
0.0733
0.0457
0.0510
7.6%
3.810
Moviel
013495
0.1665
0.1563
0.0470
0.3509
0.1625
0.
0.159
0.200
0.1419
4.175-15
0.0866
0.3001
0.2539
0.0836
0.2522
0.2568
0.2567
0.3018
0.3075
0.2185
9.328-42
NDO08-5
0.0307
0.1092
0.0791
1.10
164666
0.1437
0.0922
NDCG@110
0.0427
0.1071
0.1373
0.1292
0,0408
0.9435
0.1350
0.1366
0.1333
0.1385
0.1003
0.1743
0.118]
0.1295
2.9%
1.30E
Avg. Improo.
+13.1%%
+33.4%
-1..3%
19,6%
Azg. Train. Time
2,50ss
62,457s
31,674
2,863s
+173%
3.582s
+1246
137%
-25%
2,057s
1137%
Avg. luf. Time
1.19%
11,18s
9.346
2.75%
1.13%
-0x
-0x
1.388
+0s
1.01s
6.72
+8x
(Hit Rate) and NDCG@N (Normalized Discounted Cumulative Gain). The former measures the rate of successful hits among the top-N recommended candidates, while the latter takes into account the ranking positions and assigns higher scores in order of priority.
Baselines. We select four GNN-based models (LightGCN [16], SR-GNN [48), LESSR [8], and
MAERec [51]) as performance and efficiency benchmarks. Since this study is not to develop a
new model, four classic sequence models (GRU4Rec [17], SASRec [22], BERT4Rec [40], and
STOSA [13]) are utilized as backbones to validate the effectiveness of SEvo. Besides, MF-BPR [35] also considered here as a backbone without sequence modeling. We carefully tune the hyperpa- rameters according to their open-source code and experimental settings.
Implementation details. Since the purpose is to study the effectiveness of SEvo, only hyperparam- eters concerning optimization are retuned, including learning rate ([1e-4, 5e-3)), weight decay ((O, 0.II) and dropout rate (IO, O.7D. Other hyperparameters in terms of the architecture are consistent with the corresponding baseline. For a fair comparison, the number of layers L is fixed to 3 as in other GNN-based recommenders. As for the hyperparameter in terms of the degree of smoothness, we found 3 = 0.99 performs quite well in practice. The loss functions follow the suggestions in the respective papers, given that SEvo can be applied to any of them.
3.1 Overall Comparison
In this section, we are to verify the effectiveness of SEvo in boosting recommendation performance. Table 2 compares the overall performance and efficiency over four widely used datasets, and Table further provides the results on two large-scale datasets with millions of nodes.
Firstly, GNN-based models seem to over-emphasize structural information but lack full exploitation of sequential information. Their performance is only on par with GRU4Rec. On the Tools dataset, SR-GNN and LESSR are even inferior to LightGCN, collaborative filtering model with access to sequential information. MAERec makes a slightly better attempt to combine the two by learning structural information through graph-bases reconstruction tasks. It employs SASRec backbone for recommendation to allow for a better utilization of sequential information. Despite the identi- cal recommendation backbone, SASRec trained with SEvo-enhanced AdamW enjoys significantly better performance. Not to mention the consistent gains on other sate--of-the-an methods such as BERT4Rec and STOSA, from 2% 30% according to the last two columns of Table 2. Overall, the promising improvements from the SEvo enhancement suggest a different route to exploit graph structural information, especially in conjunction with sequence modeling.
7

---

Beanty
X11
X300 V/AR]
X10
10-
1.0
4.-
Loss
3
0.5
0.6
0.5
2000
1000
6000
8000
Ortginai
Sbve wo socalag
MovieLens-IMM IM
1.5
0.4
2.-
1.5
Los
0.5
0.0
1.0
0.0
2000
1000
6000
8000
(10)
(m)
(a) Loss curves
(b) Smoothness
Figure 2: Empirical illustrations of convergence and smoothness. The top and bottom panels
spectively depict the results for Beauty and Moviel (a) SASRec enhanced by SEvo with or without rescaling, (b) Smoothness of (I) the original variation; (II) the smoothed variation; (III) the optimized embedding. A lower I Jemoothness indicates stronger smoothness.
Secondly, either dynamic graph construction in SR-GNN and LESSR, or path sampling in MAERec, requires heavy computational overhead, which directly causes the train- ing failures on large-scale datasets like Electronics and Clothing, Even worse, these high costs associated with SR-GNN and LESSR are inevitable during inference. contrast, SEvo does not alter the model inference logic at all, thereby maintaining consistent inference time. The computational overhead required in training is also negli- gible compared to previous graph-enhanced models that employ GNNs intermediate modules. For example, SASRec with SEvo consumes only 10 minutes compared the hours of training time required for MABRec. When millions of nodes are encountered in Table 3, each epoch demands just few more seconds. SEvo is arguably superior to these cumbersome GNN modules in real-world applications. Combining Table 2 and Table 3, it can be inferred that the performance gain increases as the number of items increases. This can be explained by the fact that the randomness of sampling leads to much more inconsistent evolution when more and more nodes are encountered [56]. SEvo thus plays an increasingly important role is capable of imposing direct consistency constraints on embeddings.
Table SEvo on large-scale datasets.
SASRec +SEvo
Electronics
HR@@
0.0033
0.0063
+92.599
HR
0.0208
0.0293
+40.6%
NDCG@ 10
0.0103
0.0159
+53.9%
Time/Epooc
19.94s
+2.22s
Epochs
100
150
HR
0.0071
0.0171
+139.1%
Clothing
HR 10
0.0360
0.0626
+73,9%
NDCG@ 10
0.0199
0.0377
+89.1%
Time/Epoch
25.20s
+8.27s
Epochs
300
Since SASRec is pioneer in the field of sequential recommendation, it will serve as the default backbone for subsequent studies.
3.2 Empirical Analysis
Convergence comparison. In Section 2.3, we theoretically verified the necessity of rescaling the original Neumann series approximation for faster convergence. Figure 2a shows the loss curves of SASRec trained with AdamW under identical settings other than the form of SEvo. Without resealing, SASRec exhibits significantly slower convergence, consistent with the conclusion The- orem 2. While the theoretical worst-case convergence rate of the corrected SEvo is only 1% of the normal gradient descent when 3 0.99, its practical performance much better. SASRec trained with SEvo-enhanced AdamW W initially converges marginally slower and catches up in the final stage.

---

Beunty
Beanty
Uscomaried
Comeesed
27
Original +SExo
0.01
MF-3PR
NDCG010
0,035
NDCG010
GRUARe
0.03
0.030
SASRce
Raweiiee Neumann
0.025
0.02
BEKI4Ro:
-9.3%
Iterative
STOSA
13.5%
0.020
SGD
Adam
Adam
10
15
Optimizer
Avg. Improw, (5)
(a) Optimizers
(b) Neumann versus Iterative
(c) Moment estimate correction
Figure 3: SEvo ablation experiments.
Smoothness evaluation. Figure 2b demonstrates the variation's smoothness throughout the evolu- tion process and the eventual embedding differences from / = 0 to B 0.999. (I) -> (II: The original variations exhibit a similar degree of smoothness, but after transformation, they are quite as 3 increases.
(II) - (III): Consequently, the embedding trained with stronger smoothness constraint becomes smoother as well. The structure-aware embedding evolu- tion successfully makes related nodes closer in the latent space. Although smoothness is not the sole quality measure of embedding, combined with the analyses above, we can conclude that SEvo injects appropriate structural information under the default setting of B3 0.99.
different-smmoneee
3.3 Ablation Study
SEvo for various optimizers. It is of interest to study whether SEvo can be extended other com- monly used optimizers such as SGD and Adam. Figure 3a compares NDCG@ 10 performance on Beauty and MovieLens-IM. For fair comparison, the hyperparameters are tuned independently. is evident that the performance of SGD, Adam, and Adam W improves significantly after integrating SEvo, with AdamW achieving the best it does not need to smooth the weight decay.
Neumann
series approximation versus iterative approximation. Theorem suggests that the Neumann series approximation is preferable to the commonly used iterative approximation because the latter is not always direction-aware and thus a conservative hyperparameter sure convergence. This conclusion also be drawn from Figure 3b. When only little smoothness required, their performance is comparable as both approximations differ only at the last term. The iterative approximation however fails to ensure convergence once 3 0.7 on the Beauty dataset, potentially resulting in a lack of smoothness.
is needed to en-
Moment estimate correction for W. We compare SEvo-enhanced Adam with or with- out moment estimate correction in Figure 3c, in which average relative improvements against the baseline presented for each recommender. Overall, the two variants of SEvo-enhanced Adam W perform comparably, significantly surpassing the baseline. However, in some cases (e.g., GRU4Rec and STOSA), the moment estimate correction as suggested in Theorem 3 is particularly useful to improve performance. Recall that BERT4Rec trained using the output softmax from a separate colneccon layer that is fully updated at each step. This may explain why the correction has little effect on BERTRec. In conclusion, the results underscore the importance of the proposed modification in alleviating bias in moment estimates.
3.4 Applications of SEvo Beyond Interaction Data
We further explore the potential of applying SEvo other types of prior knowledge. On the one hand, the category smoothness constraint can also be fulfilled through SEvo (see Appendix C.2), leading to progressively stronger clustering effects as / increases. This provides compelling visual evidence of why SEvo is inherently structure-aware. On the other hand, SEvo is arguably an effi- cient tool for transferring embedding knowledge (see Appendix C.3). Notice that the learning of other modules cannot be guided in the same way, so SEvo alone is still inferior to oftheatt knowledge distillation methods [21, 55]. Fortunately, SEvo and other methods can work together to further boost the recommendation performance.

---

4 Related Work
Recommender systems are developed to enable users to quickly and accurately find relevant items in diverse applications, such as e-commerce [58], online news [15] and social media [5]. Typically, the entities involved are embedded into a latent space [6, 12, 54], and then decision models are built on top of the embedding for tasks like collaborative filtering [16] and context/knowledge-ae
rec- ommendation [46, 44]. Sequential recommendation [36, 25] focuses on capturing users' dynamic interests from their historical interaction sequences. Early approaches adapted recurrent neural net- works (RNNs) [17] and convolutional filters [42] for sequence modeling. Recently, Transformer [45, 11] has become popular architecture for sequence modeling due to its parallel efficiency and superior performance. SASRec [22] and BERT4Rec [40] use unidirectional and bidirectional self- attention, respectively. Fan et al. [13] proposed a novel stochastic self-attention (STOSA) to model the uncertainty of sequential behaviors.
Graph neural networks [2. 10] are type of neural network designed to operate on graph-structured data, in concert with a weighted adjacency matrix to characterize the pairwise relations between nodes. GNN equipped with this adjacency matrix can be used for message passing between nodes. The most relevant work is the optimization framework proposed in [57] for solving semi-supervised learning problems via smoothness constraint. This graph regularization approach has recently inspired a series of work [7, 28, 60]. As opposed to applying it to smooth node representations [24] or labels [19], it is employed here primarily to balance smoothness and convergence the variation.
Structural information recommendation is typically learned through GNN as well, with spe- cific modifications made to cope with like data sparsity [52, 29]. LightGCN [16] is a pioneering collaborative filtering work on modeling user-item relations, which removes nonlinearities for eas- ier training. To further utilize sequential information, previous efforts focus equipping sequence models with complex GNN modules, but this inevitably increases the computational cost of training and inference, making it unappealing for practical recommendation. For example, SR-GNN [48] and LESSR [8] need dynamically construct adjacency matrices for each batch of sequences. Differently, MAERec [51 proposes adaptive data augmentation to boost novel graph masked autoencoder, which learns to sample less noisy paths from semantic similarity graph for subsequent reconstruction tasks. The resulting strong supervvunoi signals help the model capture more useful information.
5 Broader Impact and Limitations
Utilizing both sequential and structural information is beneficial yet challenging, and SEvo pro- posed in this paper suggests novel and effective technical route for this purpose. Compared to other explicit GNN modules, SEvo is light- and easy-to-use in practice. These insights may inspire future research efforts regarding structure-aware optimization. However, there are still some limitations. Firstly, the training-free nature of SEvo makes it versatile, but also limits the expres- sive power. For specific task, sophisticated GNN module may be more desirable for achieving higher recommendation accuracy. Secondly, it might not be so straightforward to apply SEvo to the scenario involving multiple types of prior knowledge. Some efforts [43, 33] in the field of multiple graph learning have proposed some technically feasible solutions. However, these approaches still encounter challenges terms of efficiency, particularly in the context of recommendation systems.
6 Conclusion and Future Work
In this work, we have proposed a novel update mechanism for injecting graph structural information into embedding. Theoretical analyses of the convergence properties motivate some necessary mod- ifications to the proposed method. SEvo can be seamlessly integrated into existing optimizers. For AdamW, we recorrect the moment estimates to make it more robust. Besides, an interesting direc- tion for future work is extending SEvo to multiplex heterogeneous graphs [3], as real-world entities often participate in various relation networks. Furthermore, we believe that SEvo holds potential for application to dynamic graph structures and incremental updates [53]. Two challenges may be encountered in practice: the computational overhead associated with the ongoing adjacency matrix normalization, and how to adaptively weaken the outdated historical information.
10

---

References
[1] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[2] Joan Bruma, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and lo- cally connected networks on graphs. In International Conference on Learning Representations (ICLR), 2014.
[3] Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. Representa- tion learning for attributed multiplex heterogeneous network. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pages 1358-1368, 2019.
[4] Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low- dimensional hyperbolic knowledge graph embeddings. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 6901-6914. ACL, 2020.
[5] Jiajia Chen. Xin Xin, Xianfeng Liang, Xiangnan He, and Jun Liu. Gdsrec: Graph- basph de- centralizedeollaboraiie filtering for socialrecommendation. JEEE Transactions Knowledge and Data Engineering (TKDE), 2022.
[6] Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. Behavior sequence transformer for e-commerce recommendation in alibaba. In Proceedings of the Ist International Workshop on Deep Learning Practice for High-dinmessonal Sparse Data, pages 1-4, 2019.
[7] Siheng Chen, Aliaksei Sandryhaila, José MF Moura, and Jelena Kovacevic. Signal denoising on graphs via graph filtering. In /EEE Global Conference on Signal and Information Process- ing (GlobalSIP), pages 872-876. IEEE, 2014.
[8] Tianwen Chen and Raymond Chi-Wing Wong. Handling information loss of graph neural networks for session-bass based recommendation. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pages 1172- 2020.
[9] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST@ EMNLP 2014, Eighth Workshop Syntax, Semantics and Structure in Statistical Translation, pages 101-111. ACL, 2014.
[10] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, 2016.
[11] Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLL?), pages 4171-41888 ACI., 2019.
[12] Ahmed El-Kishky, Thomas Markovich, Serim Park, Chetan Verma, Backjin Kim, Ramy Es- kander, Yury Malkov, Frank Portman, Sofía Samanicgo, Ying Xiao, et Twhin: Embedding the twitter heterogencous information network for personalized recommendation. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pages 2842-2850, 2022.
[13] Ziwei Fan, Zhiwei Liu, Yu Wang, Alice Wang, Zahra Nazari, Lei Zheng, Hao Peng, and Philip S Yu. Sequential recommendation via stochastic self-attention. In ACM Web Conference (WWW), pages 2036-2047, 2022.
[14] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neu- ral message passing for quantum chemistry. In International Conference Machine Learning pages 1263-1272. PMLR, 2017."
pages
[15] Shansan Gong and Kenny Q Zhu. Positive, negative and neutral: Modeling implicit feedback in session-based news recommendation. In International ACM Conference on Research and Development in Information Retrieval (SIGIR), 2022.
[16) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In International ACM Conference on Research and Development Information Retrieval (SIGIR), pages 639-
648, 2020.
11

---

[17] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session- based recommendations with recurrent neural networks. In International Conference on Learning Representations (ICIR), 2016.
[18]
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, Abs/1503.02531. 2015.
[19] Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R. Benson, Combining label propagation and simple models out-perfonns graph neural networks. In International Conference on Learning Representations (ICLR), 2021.
[20] Tony Jebara, Jun Wang, and Shih-Fu Chang. Graph construction and b-matching for semi- supervised learning. In International Conference on Machine Learning (ICML), pages 441- 448, 2009.
[21] SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu. Topology distillation for recommender system. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 829-839. ACM, 2021.
[22] Wang-Chene Kang and Julian McAuley. Self-attentive sequential recommendation. In IEEE International Conference on Data Mining (ICDM), pages 197-206. IEEE, 2018.
[23] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna-
tional Conference on Learning Representations (ICLR), 2015.
[24] Johannes Klicpera, Aleksandar Bojehevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR), 2019.
[25] Jiacheng Li, Yujie Wang, and Julian McAuley.
Time interval aware self-attention for sequential recommendation. In International Conference on Web Search and Data Mining (WSDM), pages 322-330, 2020.
[26] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional net- works for superiuper learning. In Conference on Artificial Intelligence (AAAT), 2018.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2018
[28] Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In ACM International Conference on Information and Knowledge Management (CIKM), pages 1202-1211, 2021.
[29] Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, and Xiuqiang He. UlraGGN: Ultra simplification of graph convolutional networks for recommendation. In International Conference on Information and Knowledge Management (CIKM), 2021.
[30] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint /802.00
2018.
[31] Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of a feather: Homophily in social networks. Annual Review of Sociology, pages 415-444, 2001.
[32] Yurii Nesterov. Introductory lectures on convex programming volume i: Basic course. Lecture notes, 3(4):5, 1998.
[33] Feiping Nie, Jing Li, and Xuelong Li. Self Self-weighted multiview clustering with multiple graphs. In International Joint Conference on Artificial Intelligence (IJCAI), pages 2564-2570, 2017.
[34] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3967-3976. IEEE, 2019.
[35] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Schmidt-Thiee Lars. Bpr: Bayesian personalized ranking from implicit feedback. In Conference on Uncertainty in Arti- ficial Intelligence (UAI), 2009.
[36] Guy Shani, David Heckerman, Ronen I Brafman, and Craig Boutilier. An mdp-based recom- mender system. Journal of Machine Learning Research (JMLR), 6(9). 2005.
[37] Kartik Sharma, Yeon-Chang Lee, Sivagami Nambi, Aditya Salian, Shiok Shah, Sang- Sang-Wook Kim, and Srijan Kumar. A survey of graph neural networks for social recommender systems. arXiv preprini arXiv:2212.0444 2022.
12

---

[38] Daniel A Spielman. Spectral graph theory and its applications. In Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) 077), pages 29-38. IEEE, 2007.
[39] Gilbert W Stewart. Matrix algorithms: volume I: basic decompositions. SIAM, 1998.
[40] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In ACM International Conference on Information and Knowledge Management (CIKM), pages 1441-1450, 2019.
[41] Ily Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of ini- tialization and momentum in deep learning. In International Conference on Machine Learning (ICML), pages 1139-1147. PMLR, 2013.
[42] Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In ACM International Conference on Web Search and Data Mining (WSDM), pages 565-573. ACM, 2018.
[43] Wei Tang, Zhengdong Lu, and Inderjit S Dhillon. Clustering with multiple graphs. In IEEE International Conference on Data Mining (ICDM), pages 1016-1021. IEEE, 2009.
[44] Zhen Tian, Ting Bai, Wayne Xin Zhao, Ji-Rong Wen, and Zhao Cao. Eulernet: Adaptive fea- ture interaction learning via euler's s formula for CTR prediction. In International ACM SIGIR Conference Research and Development in Information Retrieval (SIGIR), pages 1385-
ACM, 2023.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), volume 30, 2017.
[46] Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. Kgat: Knowledge graph attention network for recommendation. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pages 950-958, 2019.
[47] Yu Wang, Yuying Zhao, Yi Zhang, and Tyler Dem. Collaboration-awar graph convolutional network for recommender systems. ACM Web Conference (WWW). pages 91-101, 2023.
[48] Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-bass recommendation with graph neural networks. In Conference on Artificial Intelligence (AAAI), volume 33, pages 346-353, 2019.
[49] Xin Xia, Hongzhi Yin, Junliang Yu, Qinyong Wang, Lizhen Cui, and Xiangliang Zhang. Self- supervised hypergraph convolutional networks for session-based recommendation. In Confer- ence on Artificial Intelligence (AAAI), volume 35, pages 4503-4511, 2021.
[50] Cong Xu, Jun Wang, and Wei Zhang. Stablegen: Decoupling and reconciling information propagation for collaborative filtering. JEEE Transactions on Knowledge and Data Engineer- ing (TKDE), 2023.
[51] Yaowen Ye, Lianghao Xia, and Chao Huang, Graph masked autoencoder for sequential recom- mendation. In International ACM Conference on Research and Development in Information Retrieval (SIGIR), pages 321 321-330. ACM, 2023.
[52] Rex Ying, Ruining He. Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pages 974 983, 2018
[53] Jiaxuan You, Tianyu Du, and Jure Leskovec. Roland: graph learning framework for dynamic graphs. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pages 2358-2366, 2022.
[54] Shengyu Zhang, Fuli Feng, Kun Kuang. Wenqiao Zhang, Zhou Zhao, Hongxia Yang, Tat- Seng Chua, and Fei Wu. Personalized latent structure learning for recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023.
[55] Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge dis- tillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11942-11522. IEEE, 2022.
13

---

[56] Wayne Xin Zhao, Zihan Lin, Zhichao Feng, Pengfei Wang. and Ji-Rong Wen. A revisiting study of appropriate offline evaluation for top-n recommendation algorithms. ACM Transac- lions on Information Systems (TOIS), 41(2);1-41, 2022.
[57] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in Neural Information Processing Sys- tems (NeurIPS), 16, 2003.
[58] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In ACM SIGKDI) International Conference on Knowledge Discovery & Data Mining (KDD), pages 1059-1068, 2018.
[59] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Be- yond homophily in graph neural networks: Current limitations and effective designs. 33:7793- 7804, 2020.
[60] Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In ACM Web Conference (WWW), pages 2215-1226, 2021.
14

---

Contents
Introduction
2
Structure-aware Embedding Evolution
2.1
Preliminaries
2.2
Methodology
2.3
Convergence Analysis for Further Modification
2.4
Integrating SEvo into Existing Optimizers
3
Experiments
3.1
Overall Comparison
3.2
Empirical Analysis
3.3
Ablation Study
3.4
Applications of SEvo Beyond Interaction Data
4
Related Work
10
5 Broader Impact and Limitations
10
Conclusion and Future Work
10
Proofs
16
A.1 Proof of Theorem
16
A.2 Proof of Theorem 2
19
A.3 Proofs of Proposition and Theorem 3
A.4 Connection between LightGCN and SEvo-enhanced MF-BPR
23
Detailed Settings
23
B.1 Algorithms
23
B.2 Datasets
B.3 Baselines
24
Applications of SEvo beyond Interaction Data
25
C.I Pairwise Similarity Estimation Factors
25
C.2 SEvo for Intra-class Representation Proximity
26
C,3 SEvo for Knowledge Distillation
27
Additional Experimental Results
28
D.1 SEvo for CNNNbased models
28
D.2 Jemoothness as Regularization Term
29
D.3 L-layer Approximation
29
D.4 Training and Inference Times
29
15

---

A Proofs
A.1 Proof of Theorem
In this part, prove the struccure-waamedddccttoo-ava properties of the L-layer iterative approximation [57]:
Viter (E)) dn (AE)
(1 B) Σ
3 +
BLA
AE
(9)
1=0
=
and the L-layer Neumann series approximation [39]:
nsa AE) = (1 3) Σ
AA
(10)
1=0
=
Fact If T. is odd, the geometric series S(x) = Ek= xt is monotonically increasing when x: S 0.
Proof. is easy show that
S(x) = 1 1- x-
1- x
and the derivative w.r. x + is
S'(x) La (L + 1)xt +1
(1 - x²2
If T. is odd, (x) is positive when and in this case S(x) is monotonically increasing.
Lemma Given a normalized adjacency matrix À R let the symmetric matrix deduced from the Neumann series approximation be
P P= (1
Σ A
(11)
Denoted by Amin(P). Amax (P) the smallest and largest eigenvalues of P, respectively, then we have 43 [0,1)
2 (1- (1-B") S Amin (P) Amax (P) = -Bt+11
(12)
+ 3
Proof. It is easy shown that the eigenvalues of P are in the form of
A = (1 - 3)
3 X 1.2....,
(13)
1=0
where d1 < ..... < An denote the eigenvalues of Á. Recall that these eigenvalues all fall into [-1,]] and An can be achieved exactly [38]. Hence,
in = (1 - 3) Σ
g4+1
(14)
1=0
the largest eigenvalue of P.
16

---

In addition, notice that the last term (1 3)8" A is non non-negative when L is even. Then, we can get lower bound no matter L odd even:
2[(L-11/2J4
(1- 3)
Σ B (1-
Σ
(15)
1=0
1
=:S(A)
The minimum of S()) must be achieved in [-1, 0] because S(-X) S(A) for any À 0. In fact, in view of Fact 1, we know that S(A) 2 S(-1). Hence, we have
21 1)/111
x (1 (1- -11-)))(- B)
Σ
(-11)
(L-1)/2]
= -1-83 - 3)
Σ
(22 g21+1
1=0
((1-1))2 1)/2|
B = -11 B3)² (1 (2((4-1)/2)+
= (1 - 8)²
1 - B2
1=0
= 1 8
(1 82l(1- (21(1-1//2)42
2 1 8 =1 8h). (1-
1 1-
(16)
The last inequality holds because 2[( - 1)/2] + 2 L. Therefore, the smallest eigenvalue of P must be greater than
1-
1 1 1g((1 -hh)
Proposition 2. The Neumann series approximation is structure-aware and direction-aware for any E (
Proof. In view of Lemma 1, we know
3 Amin (P) 2 1 -1- 0, AB (0,10,1).
so P is positive definite and thus (nsa(-) is direction-aware. Also, notice that has the same eigenvectors as A, and so does Hence, it also structure-aware:
(Unsa (x), L(
(Unsa (x), Le = (Px, LPx)
(x, L S (x. Lx).
The last inequality follows from the fact Amax (P) 1.
Lemma 2. Given a normalized adjacency matrix À E Ruxn, let the symmetric matrix deduced from the iterative approximation be
L-1
p' (1 (1-13 3)
Σ
B' + A
(17)
1=0
We have Nunin (P') > 0, 43 < 1/2.
Proof. This conclusion trivial for the case of L 1. Let us assume that L 2 Firstly, rewrite Eq. (17) as
P'=P+ P'=P+
(18)
17

---

where P : -1 Σizo S'A'. P positive definite in view of Lemma and 3 A is positive semidefinite when L is even. Therefore, only the case of needs be proved. For vector x, we have
x x
Px + 3
Amin (P)[ B+ Amin (A)
1 1- 3
31-1) + + f Amin (A) |
(19)
1 +
1 3 (1 B1- - 1 |xx1
1 - 3 - B--1 gL+1
|
1 +
1 1+1 5
3 82 gr
|x
(20)
1+ 8
The first two inequalities follow from Lemma The last inequality holds by noting the fact that, for L 2 3 and B 1/2,
3 + B1 + gL+t 2 B² 34 <
13/16.
Therefore,
x T
P'x
1 -
3 6 8
Amin (P") min
0. 1 B
(21)
|
Proposition 3. The iterative approximation is direction-awaree all possible normalized adjacency matrices and L 2 and only if 3 < 1/2.
Proof. If 3 < 1/2, we have Amin (P') > 0 according to Lemma 2, and thus v is direction-aware.
Conversely, if B 2 1/2, we can construct an adjacency matrix Á such that Amin < 0 for some L. Let us assume that L and
(22)
1 0
In this case, we have
1 1-8 3
P' (1 B 3A
(23)
3
1-
whose eigenvalues are 1 and 233 The latter is non-positive for any 3 2 1/2.
L
Remark 2. The construction of A Eq. (22) is not unique. In fact, any bipartite graph can be used as counterexample.
Corollary (The proof of Theorem 1). The iterative approximation is direction-aware for all pos-
sible normalized adjacency matrices and 2 0, if and only if 3 1/2. In contrast, the Neumann series approximation
Snea AE = (1-B) Σ
8 A'A'EE
(24)
1=0
is structure-aware and direction-awart for any 3 E [0, 1).
Proof. This corollary of Proposition 2 and Proposition 3.
Proposition The rescaled Neumann series approximation & is structure-aware and direction- aware, and converges to the optimal solution L increases.
Proof. The convergence to the optimal solution obvious by noting that
lim (AE)
lim I-32 lim Unea (AE)
(25)
. 10 (AE) = (AE).
Similar to Proposition 2, can be proved that 1- gL+1 < 1. Therefore, the rescaled transformation also structure-aware and direction-aware.
P is positive definite with a largest eigenvalue
18

---

A.2 Proof of Theorem 2
Before delving into the proof of the convergence, we would like claim that the lemmas below are well known and can be found in most textbooks [32, 1] on convex optimization. For the sake of completeness, we provide here these proofs. Hereinafter, we use |||| denote the spectral norm which returns the largest singular value of the matrix X.
Lemma For a twice continuously differentiable function f R" + R with fxx)(( C, Vx dom(J) have
I|2 S
f(y) S f(x) + (Vf(x), - x) + 2 y - -x
(26)
Proof. For a Taylor expansion of [xx), there exists a 2 = TX + (1 - r)y for some 7 E [0, 1] such that
f(y)= - + (Vf((), (y x)) + (y f(z)(y x) 5((x) (VI(x) (y - x))
C
lly x12-
2
Lemma 4. For positive definite matrix P, let ||x||p = px) be the quadratic norm induced
from P. If the eigenvalues of fall into [a, b], then
|Px| S b|
a||x||3 |x
(27)
Proof. Firstly,
Px|12 1/1/2p1/2x/// |||/||/|||||P S vb||x|Ip.
Secondly,
|x|| |P 1p-1/3p1/2x// S |P-1/2 S
1 Va
|x||p.
Theorem 4. Let 12 x R + H. be twice continuously differentiable function bounded below, and its Hessian matrix satisfies
[(x))x S, V(x.y) dom(J)
(28)
for some constant C. The following gradient descent scheme is used to train x, x,y:
X++1 + X₁ x Yt-1 t y: - (x,
(29)
where P is deduced from the L-layer Neumann series approximation. Then, after T updates, have,
20 7( 7( + +1
min K
(30)
where e = f(x,y) f(x",y .Y) and
34) + BL+1),
in = 1/ - otherwise
9-
I-BLII
Proof. The update formula (29) can be unified into
2++1 =
X(+1 Y(+1
X;
pp
0 I
(x(.
Yt
P
19

---

It is easy to show that
a= m
1-8 1+8 (1 - m7) I Amin (P)
- mmx (P) = max((1 - (B+1111, 11) =: b.
Specifically,
1-8 (1 34) ) 0
if -/1= = t
I+3
1-8 1-g1 1
otherwise
T+, T-BU-T 2
(1 11+t1)
n- = c otherwise
E1
In view of Lemma 3 and Lemma 4, we have
f(z(+1) S f(z)) (VI(z)), PV C ((V
2
C
f(z) [f(i))
PVf(()))
2
S f(z)) [2))) + bc
V [2)))))
(31)
bc
f(z)) (1
|| [2+))
2
&C
S f(z)) - a(1
||V
(32)
2
Denoted by
7 == aC(2 - 6C)
I48
1-8 31) + B(t1),
1 =
(33)
1-8 g-
otherwise
I+B I-BL+T
we have
min 0SIST
[V/(x.))))1
Σ
V yx).
T+I
T
2C
(s(x), - ((x)1))))))
T 11 1=0
20
(f( ((XT+1.8T41)
7(T + 1)
20
7(T 11)
Theorem (The proof of Theorem 2). If = =/ 1/C. after T updates, we have
min KTT
|VL 0.) = -
(34)
If we adopt modified learning rate for embedding:
(35)
(1- (1-
the convergence rate could be improved to O(C/((1-37T)
Proof. This is true for = 0, since in this case the update mechanism becomes a normal gradient
descent regardless of n = 1/C or 77 = -T- Let us prove general case for L, 2 1 next.
20

---

According to Theorem 4, we have
20 min VL[ 0.) =
7(I + 1)
where L)E,0) L(",, 0*) and
I+f (1 31 34t1),
1
7=
1- 1-8L
otherwise
I+Ø I-3LTI,
Notice that, for 1) =/ =/C,
1/7 lim 17(1-9)2
I-3 limg-+1
(36)
81 1-8 -- I
lim
and for 1) (1-311160 1/ c
[1-ple
limg-+1 1/(1-3)
1/7
lim
1+7 (1-31+1)
(37)
1- 1 2(L+1)
2 lim
1-B"
Therefore,
1 001(1-10))))011000001111 8)),
if n/=
7 001(1111))))001 B)),
if = [I-B+FIJC. 1/
The remainder of the proof is straightforward.
A.3 Proofs of Proposition 1 and Theorem 3
Adam( W) [23] uses the bias-corrected moment estimates for updating because they are unbiased when the actual moments stationary throughout the training. Below, Lemma formally elabo- rates on this, and Theorem extends Theorem 3 with proof of unbiasedness.
Lemma ([23)). Denoted by m, = m1/(1-31) and V, v1/(1-8t) the bias-corrected estimates, the first and second moments are stationary, i.e.
Egg Egg)
[gg]] Elg"),
1I = 1,2, 2,
then these bias-corrected estimates unbiased:
[mm = [gg]
= E[g
Vt 1,2,....
Proposition 5. If node is no longer sampled in subsequent batches after step t, we have
Dettp-1 = K.
Ac.-1:
(38)
B
where the coefficient of k is mainly determined by t.
Proof. In this case, the iterative formula becomes
m,+j 3mm + +, Vetj = By, + 0.
Vj 10,1,.....
21

---

Therefore,
V V1- B2-P
ma+p
mitp
Acttp-1-
Vi+p
1 1-
VVL+p
82
gh+p
89
m
89
12 m
1 8
13
Vv
B (1
V
BP 31) 1 32-2 m
VB 81
V 32 VV.
(1 11- fr
Ac1-1.
v g (1-
8
It is easy to show that
lim k(t,p)= =1 V3, 10. ++000
(39)
Theorem 6 (The proof of Theorem 3). Under the same assumptions as in Lemma 5 and Propo- sition 1, the bias-corrected estimates are unbiased and Aottp-1 -De-1 if the estimates are updated in the following manner when g: = 0,
1
m, - + (1 -1)
V. B2V1-1 (1 (2)
13 ---1 1-1- (40)
Proof. We first show the unbiasedness of m, and the proof for is completely analogous. It remains only to show that
E[my] (1 E(g)
This is trivial for = Assuming that this also holds in the case of - 1, it can be proved by induction.
g: # 0,
E mm = (1-
=1 BE + (1)
(1 {(g))
g: 0,
E[m,] = 3,E[m,-1) + (1-33)
1 3 m
1 1 8
= (1 9 E(g)
1 3
the node is no longer sampled in subsequent p batches after step t, we have
Ma+j
3 -.m.++j1++ (1 - 31)
8 14411 mi+j-1
8
1 1- -/ - 3 g
8 a(+s-1 M
- 8 gi+j-1
3 sl+s-2 m
1 gt+j
Vj = 1,2,....
1 - 3
22

---

Analogously, we have,
1- 1-2 8
Vi+j 1 BB Vts
Vj = 1,,......
Therefore, the conclusion can be deduced from
ma+p 1 01th)
m++p
m.
m
1 8
1
Vi+D 1 $2+2
Vttp
Vt
V..
33
A.4 Connection between LightGCN and SEvo-enhanced MF-BPR
For L-layer LightGCN, it can be formulated follows
F = dh E
A'E
1=0
where aq, I = O,..., represent the layer weights. According to the linear nature of the gradient operator, it can be obtained that
VEL = (VFL).
Hence, denoted by (() the gradient processing procedure of optimizer, establish that LightGIN is identical to the following system:
F(t) (E((t)
(E((t 1) - nAE(t 1)) (E((( 1)) - (E 1)) F( 17 vVV)...
When ((-) is identity mapping (i.c., standard gradient descent), LightGCN is equivalent MF- BPR with SEvo being applied twice at each update. However, when ((-) is not an identity mapping (e.g., optimizer with momentum single system. Compared to explicit GNNs, SEvo is easy-to-use and has minimal impact on the for- ward pass, making it more suitable for assisting recommenders in simultaneously utilizing multiple types information. These connections in part justify why SEvo can inject structural information directly.
weight decay is integrated), they cannot be unified into
B Detailed Settings
B.1 Algorithms
We present the algorithms of SEvo-enhanced Adam and SGD in 2 and Algorithm 3, respectively.
B.2
Datasets
In this study, we perform experiments on six public datasets. Specifically, the Beauty, Toys, and Tools datasets are extracted from Amazon reviews published in 20142, while Electronics and Cloth- ing are sourced from Amazon reviews published 20183. Additionally, the Moviel.ens dataset made available by GroupLens".
>nteps://cseubb uesd. hta.a/////jjaalllaaaaaaaaaaaoooa html
>nttpe:/cceeub. ueed. htt.aeerwb..ceeeeeeee//aaaaaaaaa htth://ncealle///attssss/aaaa
hhttps://groupleet org/ wtgg/daataaees/mmmieeee////////
23

---

Algorithm 2: Adam enhanced by SEvo. Differences from the original Adam are colored in
blue. The matrix operation below are element-wise.
Input: embedding matrix E, learning rate momentum factors B1, 82, E [0, 1), weight decay X.
foreach step t do
G. f VVC+
// Get gradients
Update first/second moment estimates:
M2 + M + (1 (1- Gt)
V, + 81 V + (1 G
Compute bias-corrected first/second moment estimates:
M. - M./(1 ())
V, - v,/(1 :35):
Update via SEvo:
E, e E.- 17 2 M,/ Ve//VV++eBB)
Output: optimized embeddings E.
Algorithm 3: SGD with momentum enhanced by SEvo. Differences from the original SGD colored in blue. The matrix operation below are clement-wise.
Input: embedding matrix E, learning rate 7). momentum factors μ, B E [0, 1), weight decay A.
foreach step t do
G. t VgC+ A
// Get gradients
// Moment update
M, + + G
Update via SEvo:
E, E, - n (Mri8)
Output: optimized embeddings E.
B.3 Baselines
Four GNN-based baselines for performance and efficiency benchmarks:
LightGCN [16] is a pioneering collaborative filtering model that simplifies graph convolutional networks (GCNs) by removing nonlinearities for easier training. It uses only graph structural information and has access to sequential information.
SR-GNN [48] and LESSR [8] are two baselines dynamically constructing session graph. The for- mer employs gated graph neural network to obtain the final node vectors, while the latter utilizes edge-order preserving multigraph and shorteut graph to address the lossy session encoding and ineffective long-range dependency capturing problems, respectively.
MAERec [51] learns to sample less noisy paths from a semantic similarity graph for subsequent reconstruction tasks. However, we found that the official implementation treats the recommenda- tion loss and reconstruction loss equally, leading to poor performance here. Therefore, an addi- tional weight is attached to the reconstruction loss and grid search is performed in the range of [0, 1]. Almost all hyperparameters are tuned for competitive performance.
Four sequence backbones validate the effectiveness of SEvo:

---

GRU4Rec [17] applies RNN [9] to recommendation with specific modifications made to cope with data sparsity. In addition the learning rate in fle-4, 5e-4, le-3, Se-3) and weight decay in [0, 0.1], we also tune the dropout rate for node features in the range of [0, 0.7].
SASRec [22] and BERT4Rec [40] are pioneering works sequential recommendation equipped with unidirectional and bidirectional self-attention, respectively. For BERT4Rec which employs separate fully- fully-connected layer for scoring, the weight matrix therein will also be smoothed by SEvo. In addition to some basic hyperparameters, the mask ratio is also researched for BERT4Rec.
STOSA [13] is one the state-of-the-art models. aims to capture the uncertainty of sequential behaviors by modeling each item as a Gaussian distribution. The hyperparameters involved are tuned similarly SASRec.
Four knowledge distillation methods used in Appendix C.3:
KD [18] and DKD [55] are two logits-based upproaches to transfer knowledge. DKD decomposes the classical KD loss into target class knowledge distillation loss and non-target class knowledge distillation loss.
RKD [34] and HTD [21] are two ranking- based approaches. The former focuses the distillation of relational knowledge through distance- bistance-wiis and angle-wise alignments, while the latter empha- sizes the distillation of hierarchical topology by dividing nodes into multiple groups and requiring intra-group and inter-group alignments.
C Applications of SEvo beyond Interaction Data
Here, we preliminarily explore the exploitation of more types of knowledge besides consecutive occurrences. We first investigate some elementary factors for interaction data, and then introduce the applications of SEvo to node categories and knowledge distillation.
C.1 Pairwise Similarity Estimation Factors
B
G X $
D D
D
6 O
D
Deiault
Last
Walk length 2m=2
Prequency Disttance
Figure 4: Illustrations of different pairwise similarity estimation methods based on interaction data. (a) The default is to adopt the co-occurrence frequency within the last K items. (b) Using only the first K items. (c) Allowing a maximum walk length of H beyond (d) Frequency-baa similarity versus distance-based similarity.
Recent GNN-based sequence models [48, 49], as well as the SEvo-enhanced models reported in Sec- tion 3, estimate the pairwise similarity wy between items and uy based on their co-occurrence fre- assumed
quency in sequences. In other words, items that appear consecutively more frequently more related. Yet there are some factors that deserve closer look: (1) The maximum sequence length K for construction investigate the number of interactions required for accurate estimation; (2) Using only the first K versus last K interactions in each sequence to compare the utility of early and recent preferences; (3) Allowing related items to be connected by walk length < rather than strict consecutive occurrences; (4) Frequency-base similarity versus distance-based similar- ity. The former weights all related pairs equally, while the latter weights inversely to their walk length. For example, given sequence 02 U1 03 with maximum walk length of H 2,
25

---

Algorithm 4: Python-style algorithm for similarity estimation based on interaction data.
for seq seqs:
if first:
seq seq[:K] First K items
else:
seq seq[-K:] Last K items
for range (se))) 1):
Maximum walk length for h, 1 in enumerate
range (i 1, min(i + 1, len ()eq))) start=1
if frequency: Frequency [[seq[i], seq += 1
A[seq[j], [i]l +=
else: # Distance
A[seq[il, seq [j]]
h
A [[seq[j] [i]] += 1 h
Deanty
MorteLous 1M
Beanty
Movislan-IM
0.138
XD03800
0,0033
Powelite
Roweline
0.0373
Baadin
Busiae
Fint
Fint
Projoncc
Proppery
0.0300
D.134
0.0300
0.0325
0.182
0.0325
0.139
50
100
150
300
100
200
500
100
(a) Sequence length
(b) Frequency- bbased versus Distance-based
Figure 5: Comparison of similarity estimation across four potential factors. 'x' indicates the default way applied to SEvo-enhanced sequence models Section 3. (a) Using only the first/last K items for pairwise similarity estimation. (b) Frequency- and distance-based similarity with maximum walk length H.
the frequency-based similarity of (w2, 03) gives while the distance- similarity is 1/2 (as the walk length from v2 to Vy is 2).
Figure 4 illustrates these four variants and Algorithm 4 details step-by-step process. We further compare these four potential factors in Figure 5:
Figure 5a shows the effect of confining the maximum sequence length to the first/last K items, so only the early/recent preferences will be considered. In constrast to early interactions, recent ones imply more precise pairwise relations for future prediction, even for small K. With the increase of the maximum sequence length, the recommendation performance on Beauty improves steadily, but not the case for MovieLens- M.. This suggests that shopping relations may be more consistent than movie preferences.
Figure 5b explores the relations beyond strict consecutive occurrences; that is, two items are considered related once they co-occur within a path length & H. For the shopping and movie datasets, estimating similarity beyond co-occurrence frequency appears less reliable overall. We also compare frequency-based similarity with distance-based similarity that decreases weights for more distant pairs. It is clear that the distance-based approach performs more stably as the maximum walk length increases.
C.2 SEvo for Intra-class Representation Proximity
Sometimes embeddings are expected be smooth w.r.t. a prior knowledge. example, in addition to the interaction data, each movie in the MovieLens- M dataset is associated with at least one genre.
26

---

Epodh:
Epodi
Epoch
Epocii DO
Epochh 100
Figure 6: UMAP [30] visualization of movies based their embeddings. For case of differentiation, we group the 18 genres into 6 categories and colored them individually: ManillerCCrime/Attioo/AAee Horro//yystey//FlnnNNNoo Nat/Drama/Romanee; Comedy/Mu- sical/Children s/Animation; Fantasy/Sci-FFi Western/Documentary;
Table 4: Pairwise similarity estimation based interaction data versus node categories (movie genres).
MovieLens- HR@ 10
HR @
NDCG@ 10
Baseline (SASRec)
0.0457
0.2482
0.1315
0.5
0.0494
0.2538
0.1362
Interaction data
0.99
0.0517
0.2567
0.1385
0.5
0.0492
0.2527
0.1352
Movie genres
0.99
0.0508
0.2549
0.1371
It is natural to assume that movies of the same genre are related to each other. Heuristically, we can define the similarity Wi be if U; and vy belong to the same genre and otherwise.
As can be seen in Figure 6, such smoothness constraint can also fulfilled through SEvo, leading to progressively stronger clustering effects / increases. However, the resulting performance gains are slightly less than those based on interaction data (see Table 4). One possible reason is that the movie genres are too coarse to provide particularly useful information. In conclusion, while smoothness is an appealing inductive bias, its utility depends on how well the imposed structural information agrees with the performance metrics of interest.
C.3 SEvo for Knowledge Distillation
In addition the affinity matrix extracted from interaction data or relation data, the pairwise simi- larity can also be estimated from a heavy-weight teacher model. Recall that Knowledge Distillation (KD) [18] encourages light-weight student model mimic the behaviors (e,g., output distribution) of the teacher model, so the learned student model achieves both accuracy and efficiency. In general, nigher-dimensional embeddings are capable of better fitting the underlying distribution between en- tities. The pairwise similarities extracted from a teacher model, needless to say, can be used to guide embedding evolution of student model. Unlike interaction or relation data, the deduced graph is dense if only the distance function is applied. Therefore, some graph construction steps including sparsification and reweighting should be involved as well. We attempt to use the widely used KNN graph here, and leave more comprehensive study of graph construction [20] as a future work.
27

---

Table 5: Knowedge distillation from Teacher (SASRec with a embedding size of 200) to Student (SASRec with a embedding size of 20). The results are averaged over 5 independent runs on the Beauty dataset. ne-nearest neighbors (iee. K 10) are selected for each node.
HR@1
HR@5
HR@10
NDCG@ NDCG@5
Teacher
0.0198
0.0544
0.0786
0.0374
0.0452
Student
0.0094
0.0327
0.0526
0.0210
0,0275
+KD [18]
0.0105
0.0352
0.0552
0.0229
0,0294
+RKD [34]
0.0082
0.0311
0.0515
0.0196
0.0262
+HTD
0.0085
0.0344
0.0549
0.0215
0.0281
+DKD [55]
0.0138
0.0389
0.0577
0.0265
0,0325
Student
0.0094
0.0327
0.0526
0.0210
0.0275
+SEvo
0.0107
0.0364
0.0576
0.0236
0.0304
+DKD
0,0166
0.0407
0.0568
0.0289
0.0341
Specifically, the distance between each pair Uú, By is estimated using a cosine similarity distance function:
ej
d = 2
(41)
e,, e
Then, K-nearest neighbors are selected for each node; that is
(i, E i i j iff + i duk ddjisss
S K.
(42)
This sparsification is neccessary for several reasons: 1) SEvo with a dense adjacency matrix is computationally prohibitive to conduct; 2) Generally, only the top-ranked neighbors reliable for next distillation. Finally, the adjacency matrix obtained through reweighting and symmetrizing:
We Wij Wge
Wiy = exp(-di//7),
where = >> O is the kernel bandwidth parameter.
Table 5 reports the results of the SASRec backbone with different embedding sizes (200 versus 20). Although a student equipped with SEvo can only derive guidance from the teacher in terms of em- bedding modeling, has surpassed RKD and HTD that focus feature/output alignments. Recall that SEvo only needs to access the teacher model once for adjacency matrix construction, whereas other knowledge distillation approaches require accessing the teacher model for each update. SEvo arguably an efficient tool for transferring embedding knowledge. Nevertheless, SEvo alone can- not be expected to facilitate the learning of the other modules, which consequently is still inferior to state-of-the-an methods such as DKD. Fortunately, SEvo and DKD can work together to further boost the recommendation performance.
D Additional Experimental Results
D.1 SEvo for GNN-based models
Table 6: Beauty recommendation performance comparison. SEvo-enhanced AdamW applicd to
LESSR and MAERec.
HR @ 1
HR@5
HR @ 10
NDCG@5
NDCG@ 10
LESSR
0.0088
0.0322
0.0506
0.0205
0.0264
+SEvo
0.0126
0.0405
0.0625
0.0267
0.0338
Improv.
43.2%
26.0%
23.5%
30.4%
27.9%
MAERec 0.0113
0.0424
0.0662
0.0269
0.0346
+SEvo
0.0120
0.0441
0.0677
0.0283
0.0358
Improv. 6.7%
4.0%
2.3%
4.9%
3.6%
In Section we have comprehensively validated the effectiveness of SEvo for classic sequence models. It is also of interest to explore the impact GNN-based models that have Iearned certain
28

---

structural information. Table 6 reports the results on LESSR and MAERec: SEvo not only facilitates the learning of LESSR, but also helps MAERec that already utilizes global graph information. This implies that previous efforts fail to fully exploit structural information, while SEvo demonstrates superior performance in this regard.
D.2
I smmoothnes as Regularization Term
Beauty
MovieLens- IM
0.14
0.040
0.13
NDCGL10
0.035
0.12
0.030
0.11
0.025
SASRec
0.10
+SEvo
0.020
10
10-3
10-4
3
10-2
10-6
10-5
10
10-3
10
A
Figure 7: Smoothness constraints through a regularization term.
Structural information may be injected by imposing Jsmonthness as a regularization term; that is,
min E,0
C(E, 0) smmotthnesd 9),
(43)
where À 2 O is a hyperparameter governing the degree of smoothness. We conduct this ablation study Figure with from 10-6 to 0.01. As can be seen, incorporating a smoothness regular- ization term could slightly improve the recommendation performance, but it is not optimal. SEvo performs better because the gradient of the regularization term may be in conflict with the primary loss function.
D.3 L-layer Approximation
Table SEvo using different approximation layers L.
Beauty
MovieLens-IM
HR@
HR@
NDCG@ 10
HR@1
HR@ 10
NDCG@ 10
L=0
0.0124
0.0664
0.0353
0.0465
0.2487
0,1321
0.0140
0.0717
0.0388
0.0498
0.2562
0,1372
0.0152
0.0740
0.0403
0.0511
0.2589
0,1389
L-3
0.0154
0.0759
0.0411
0.0517
0.2567
0.1385
L=1
0.0153
0.0755
0.0408
0.0510
0.2576
0,1383
1=5
0.0150
0.0750
0.0403
0.0492
0.2581
0.1382
As T. increases, SEvo gets closer to the exact solution while accessing higher-order neighborhood information. Table 7 lists the performance of different layers, which reaches its peak around and starts to decrease then. A possible reason is that the higher-order information is over-smoothed and thus not as reliable and easy to use as the lower order information. Similar phenomena have been found in previous works [50, 8] applying GNNs to recommendation.
D.4 Training and Inference Times
The time complexity of SEvo is mainly determined by the arithmetic operations of À A'A 1,2,..., L. Assuming that the number of non-zero entries of Á is S. the complexity required is about (LSd). Because the recommendation datasets are known for high sparsity (i.e, very small). the actual computational overhead can be reduced to very low level, almost negligible. Table 8 provides the actual training and inference times.
29

---

Table 8: Training and inference times. The wall time (seconds) here is evaluated on an Intel Xeon E5-26220 v4 platform and a single GTX 1080Ti GPU, while the results in Table 3 are tested on an Intel Xeon CPU ES- 26-2680 v4 platform and a single RTX 3090 GPU.
Remuty Inference
Toys Inference
Tools Inference
Moviel ens- IM Inference
Training
Epochs
Training
Epochs
Training
Epochs
Training
Epochs
LightGCC
2000 50
1.07
1000
1461.14
0.97
900
922.66
0.78
600
6898 42
1.95
600
GNN
SR-GNN
25837.60
13711.61
9455.18
150
126129.93
LESSR
19686 60
14.35
300
15923.59 59
9.67
300
10994,38
8,30
300
203226.08
5.02
300
MAERec
13956.43
3.60
100
43233,90
2.60
200
16920.16
2.31
100
42586.58
2.41
290
MF RVVV/Tann Transformer
MF-BPR
178165
0.96
1000
1508.73
0.91
1000
1326.93
0.71
1000
6837.01
1,95
600
+SEvo
1937.32
0.96
1000
1572.91
0.91
1000
1510.23
0.71
1000
7128.
1.95
GRU44ee
927.90 98
1.98
300
646.51 51
1,77
300
638.57
1,65
300
12116.04 04
1,78
300
+SEvo
987.47
300
791.19
300
12387.25
SASRec
445 145.68
2.16
200
413.35
2.30
200
353.10
181
200
919.27
1.24
200
-SEvvo
469.27
2.16
200
480.37
2.30
200
378.25
1.81
200
954.77
1.24
200
BERT-R20
1470.76
2.03
500
1330.58
1.71
500
1092.10
1,51
500
1131.66 62
1.18
500
+SPvo
1965.90 97
2.03
500
1.595.66 66
1.71
500
1374,72
1,51
500
1243.08 08
1.18
500
STOSA
2253.98
500
2049.00
1827.60 60
6.65
500
2220.18
1.98
-SEvo
2491.54
9.84
500
2231.11
8.42
300
1879.62
6.65
500
2259.66
1.98
500
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions are summarized in the introduction point by point.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might
30

---

not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
All assumptions and proofs detailed in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to the settings introduced in Section 3.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
31

---

(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All experiments are conducted on public datasets and the code can be found in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: /ttpas////ippp.ee cc/
htllii/gguides//oeeeubbbba htt.uubmissiiooooooooo.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. cec/public/guideer http:/guidee//eeeeeieeeeeeeeesee /CodeSubmii saionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The implementation details of baselines and ours can be found in Ap- pendix B.3 and Section 3, respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
32

---

7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: 1) Paired t-test is performed over independent runs (see Table 2). 2) 1 1-sigma error bars are marked most figures and tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix D.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerrpp.eeeee httpe/epp.p.p.aape.eeeeeeeeeeeee http:publlclceeiee i
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
33

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impact has been discussed in Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work poses no such risks.
Guidelines:
The answer means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
34

---

The source of datasets are credited in Appendix B.2, and all baselines have been introduced in Appendix B.3.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
35

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research
with human subjects.
Depending on the country in which research is conducted, IRB approval (or equíva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "8Qsjc0a2GQ", "reviewer": "Reviewer_Um97", "summary": "This paper proposes SEvo, an embedding updating mechanism that directly injects the graph information into the optimization process. This paper points out two critical criteria for directly injecting graph structure information into the embedding updating process for recommendation. Based on the proposed two criteria, this paper makes efforts to derive a solution named SEvo for injecting the graph structure information directly. SEvo is model-agnostic and can be implemented in various optimizers. The experiments are detailed, and the algorithm is theoretically guaranteed.", "questions": "I am curious about the differences in gradient descent between SEvo and methods that explicitly model the neighborhood relationship with graph structure (e.g., LightGCN). The modified embedding updating mechanism in SEvo also includes components of the adjacent matrix, which looks similar to the graph propagation mechanism in GNN-based methods. Could you please give an example showcasing the differences in gradient descent (e.g., comparing the gradient descent processes of LightGCN and MF-BPR+SEvo)?", "limitations": "See weaknesses and questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-motivated and well-organized, making this paper easy to understand. This paper first proposes two criteria and derives the final form of SEvo. I appreciate the efforts of the authors to make this process so clear.\n2. The experiments are detailed. Experiments on various datasets showcase the effectiveness of SEvo, with detailed ablation studies.\n3. The proposed method is easy to implement. SEvo can be integrated into various optimizers without complex modification.", "weaknesses": "The reason behind the success of SEvo on large-scale datasets remains unclear. I am extremely curious about this. The improvement is unbelievably huge, making me doubt the reported results. A level of 5% in practice industrial application is extremely huge. However, the experiment results show that SASRec equipped with SEvo performs twice as well as the vanilla SASRec. I believe that if the reported results are true, this performance even exceeds the SOTA method by a large margin since SASRec is still a strong baseline in practice. Has the author carefully tuned the base model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work well-motivated and well-organized.\nEach comment (presented in *italics*) is followed by its corresponding response.\n\n> *W1: The reason behind the success of SEvo on large-scale datasets remains unclear. I am extremely curious about this. The improvement is unbelievably huge, making me doubt the reported results. A level of 5% in practice industrial application is extremely huge. However, the experiment results show that SASRec equipped with SEvo performs twice as well as the vanilla SASRec. I believe that if the reported results are true, this performance even exceeds the SOTA method by a large margin since SASRec is still a strong baseline in practice. Has the author carefully tuned the base model?*\n\nThank you for your constructive feedback.\nWe did carefully tune the base model and the following hyper-parameters were **grid-searched**:\n\n| Parameter | Range |\n|:-: |:-: |\n|Learning rate | \\{1e-4, 5e-4, 1e-3, 5e-3\\} |\n| Weight decay | [0, 1e-4] |\n| Dropout rate | [0, 0.4] |\n| Batch size | \\{1024, 2048, 4096\\} |\n\nThe **best checkpoint**, determined by the validation metric, was used for the final comparison.\n**In the attached PDF within the global response, we illustrate partial results in terms of the Electronics dataset.**\nWe were as surprised as the reviewers by SEvo's success with these large-scale datasets.\nAt this point, the following observations may provide valuable insights.\n\n- The training process of the base model exhibits increased instability on larger-scale datasets. This can be attributed to the sampling randomness: only a small fraction of items are sampled for training within a mini-batch, resulting in the remaining embeddings receiving zero gradients and being updated along the outdated and inconsistent directions. This problem gets worse as the dataset grows larger and sparser. SEvo and the specific modifications to AdamW (Section 2.4) have a positive impact on this issue. In this scenario, even the embeddings of highly inactive items will be updated appropriately throughout the training process.\n\n- In real industrial applications, this performance gap may not be as large because the embeddings are typically trained more adequately beyond a link prediction task [R2]. Hence, the data sparsity problem can be greatly alleviated. Moreover, the rich side information (e.g., attributes [R1], behaviors [R3]) can further minimize the impact of poor embedding quality.\n\n> *Q1: Could you please give an example showcasing the differences in gradient descent (e.g., comparing the gradient descent processes of LightGCN and MF-BPR+SEvo)?*\n\nThank you for your insightful question.\nWe investigate the gradient descent process of LightGCN and find an interesting connection to SEvo.\nFor a $L$-layer LightGCN, it can be formulated as follows\n$$\n\\mathbf{F} = \\psi (\\mathbf{E}) := \\sum_{l=0}^L \\alpha_l \\mathbf{\\tilde{A}}^l \\mathbf{E},\n$$\nwhere $\\alpha_l (l=0, \\ldots, L)$ represent the layer weights.\nAccording to the linear nature of the gradient operator, it can be obtained that\n$$\n\\nabla_{\\mathbf{E}} \\mathcal{L} = \\psi (\\nabla_{\\mathbf{F}} \\mathcal{L}).\n$$\nHence, denoted by $\\zeta(\\cdot)$ the gradient processing procedure of an optimizer, \nwe can establish that LightGCN is identical to the following system ($\\mathbf{F}(t) := \\mathbf{F}_t$ due to OpenReview's inability to recognize '\\_' in very long formulas):\n\n$$\n\\mathbf{F}(t) = \\psi( \\mathbf{E}(t) ) = \\psi( \\mathbf{E}(t-1) - \\eta \\Delta \\mathbf{E}(t-1) ) = \\psi( \\mathbf{E}(t-1) ) - \\eta \\psi( \\Delta \\mathbf{E}(t-1) ) = \\mathbf{F}(t-1) - \\eta \\psi \\circ \\zeta \\circ \\psi ( \\nabla_{\\mathbf{F}} \\mathcal{L} ).\n$$\n\nWhen $\\zeta(\\cdot)$ is an identity mapping (i.e., standard gradient descent), LightGCN is equivalent to MF-BPR with SEvo being applied twice at each update.\nHowever, when $\\zeta(\\cdot)$ is not an identity mapping (e.g., an optimizer with momentum or weight decay is integrated), they cannot be unified into a single system.\nCompared to explicit GNNs, \nSEvo is easy-to-use and has minimal impact on the forward pass, making it more suitable for assisting recommenders in simultaneously utilizing multiple types of information.\nThese connections in part justify why SEvo can inject structural information directly. We will emphasize this point in the revised manuscript.\n\n[R1] Chen Q., et al. Behavior sequence transformer for e-commerce recommendation in Alibaba. DLP-KDD, 2019.\n\n[R2] Bai T., et al. A contrastive sharing model for multi-task recommendation. WWW, 2022.\n\n[R3] Yang Y., et al. Multi behavior hypergraph-enhanced transformer for sequential recommendation. KDD, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "51pf7PiLNh", "reviewer": "Reviewer_DLFs", "summary": "The paper introduces Structure-aware Embedding Evolution (SEvo), a novel embedding update mechanism for recommender systems. SEvo directly integrates graph structural information into embeddings, ensuring that related nodes evolve similarly with minimal computational overhead. This approach differs from traditional Graph Neural Networks (GNNs), which typically serve as intermediate modules. SEvo is designed to enhance existing optimizers, particularly AdamW, to improve recommendation performance by incorporating moment estimate corrections. Theoretical analysis confirms the convergence properties of SEvo, and experiments demonstrate consistent improvements across various models and datasets.", "questions": "1. I do not see the relationship between your method and recommendation task. Your method aims to change gradient direction based on graph topology. It is more suitable to study it under more general graph datasets. Recommendation task has no relationship with your method.\n\n2. In line 116, the author mentioned “These two criteria inherently conflict to some extent”. Why are structure-aware and direction-aware inherently in conflict? Do you have any explanation on this point?\n\n3. We have already enhanced smoothness during the forward-pass during neighborhood aggregation. Why do authors think it is necessary to further enhance it during the backward pass? Will it lead to more severe over smoothing problem?", "limitations": "See weakness and questions", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a new method to enhance over smoothing during the backward pass.\n\n2. The method can be naturally integrated with momentum-based optimizers", "weaknesses": "1. Uncleared relationship related to recommender system\n\n2. Key points need further explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work novel.\nEach comment (presented in *italics*) is followed by its corresponding response.\n\n> *W1/Q1: Uncleared relationship related to recommender system. It is more suitable to study it under more general graph datasets. Recommendation task has no relationship with your method.*\n\nThank you for pointing out the lack of a clear statement of the motivation in this manuscript.\nWe agree that **technically** SEvo can be applied to a wider range of graph datasets; however, it is more appropriate to discuss its application in a recommendation scenario for two practical considerations.\n\n1. Embedding is particularly important in modern recommender systems and its quality directly affects the subsequent decisions. However, due to **data sparsity** [R2], **millions of item embeddings** cannot be consistently updated through a simple recommendation-driven loss function. To this end, SEvo-enhanced AdamW introduces a graph regularization framework for consistent embedding evolution while simultaneously modifying AdamW to effectively address the challenges posed by extremely sparse gradients.\n\n2. Compared to embedding learning for general graph datasets [R1], a major challenge for recommendation is **how to effectively injecting structural information while leveraging other types of information** (e.g., sequential information). SEvo excels in this aspect as it has minimal impact on the forward process (this can be empirically demonstrated through comparisons with other GNN-based sequence recommenders).\n\nIn summary, SEvo-enhanced AdamW is specifically designed to address the challenges of data sparsity and injecting multiple types of information, which rarely co-exist in general graph datasets. We will further emphasize these challenges in the revised manuscript.\n\n> *Q2: Why are structure-aware and direction-aware inherently in conflict? Do you have any explanation on this point?*\n\nGiven an adjacency matrix $\\mathbf{\\tilde{A}}$, the smoothest direction is along its principal eigenvector $\\mathbf{D}^{1/2} \\mathbf{1}$, ensuring that $\\mathcal{J}_{smoothness}(\\mathbf{D}^{1/2}) = 0$. However, the region around this smoothest direction tends to be an infeasible descent direction. Therefore, we have to resort to Eq. (6) for a trade-off.\n\n> *Q3: We have already enhanced smoothness during the forward-pass during neighborhood aggregation. Why do authors think it is necessary to further enhance it during the backward pass? Will it lead to more severe over smoothing problem?*\n\nThank you for your insightful comment.\nWe agree that if the forward process already involves neighborhood aggregation, re-enhancing smoothness during the backward pass may exacerbate the over-smoothing issue. However, as stated in the introduction, SEvo is not intended to replace those sophisticated GNNs but rather to offer an easy-to-use and plug-and-play alternative for structural information learning.\nIn addition, if the base model fails to exploit the structural information **adequately** during the forward pass (e.g., LESSR and MAERec), SEvo can still facilitate the learning (the comparison on Beauty as an example):\n\n| | HR@5 | HR@10 | NDCG@5 | NDCG@10 |\n|:-: |:-: |:-: |:-: |:-: |\n| LESSR | 0.0322 | 0.0506 | 0.0205 | 0.0264 |\n| +SEvo | **0.0405** | **0.0625** | **0.0267** | **0.0338** |\n| Improv. | 26.0% | 23.5% | 30.4%  | 27.9%  |\n| MAERec| 0.0424 | 0.0662 | 0.0269 | 0.0346 |\n| +SEvo | **0.0441** | **0.0677** | **0.0283** | **0.0358** |\n| Improv. | 4.0% | 2.3% | 4.9% | 3.6% |\n\n[R1] Chami I., et al. Low-dimensional hyperbolic knowledge graph embedding. arXiv preprint, 2020.\n\n[R2] Chen Z., et al. A systematic literature review of sparsity issues in recommender systems. TORS, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0d3sUGz71Q", "reviewer": "Reviewer_feMV", "summary": "This paper proposes Structure-aware Embedding Evolution (SEvo) to improve recommender systems by directly integrating graph structural information into embeddings. Unlike traditional methods, authors propose guide embedding update momentum with graph smoothing regularization. The proposed method can be integrated with a wide range of optimizers for neural networks, e.g., AdamW. The proposed method significantly increases performance metrics on several recommender datasets for standard models and outperforms GNNs.", "questions": "* How does SEvo work in mini-batch fashion? Does it require specific batch preparations, or should it be smaller on average?\n* How does the model training time scale with the size of the graph? Could you provide a graph epoch time vs. graph size e.g. for a SASRec?\n* Can high-order proximity / long-range dependencies be incorporated using SEvo?", "limitations": "* The time and space complexity analyses of the method are required to understand its scalability.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novel intriguing view on the problem of preserving structural information for recommender systems\n* Plug-n-play design of the method so that it can be easily transferred to any recommendation architecture\n* Solid performance gains\n* Faster than training GNNs", "weaknesses": "* The paper is difficult to follow. It would be nice to see a simple summary of the SEvo pipeline in the form of a scheme or algorithm. \n* The SEvo formulae use the normalized adjacency matrix, so propagation of the gradients over the sampled node neighborhood is required. Suppose we have a large graph with a relatively high degree of each node. Mini-batch may have poorly correlated nodes, so many node embeddings should be updated simultaneously. This can lead to memory consumption issues and a notable increase in training time.\n* The tables report only the average time across different datasets. The detailed computational (or non-aggregated graphs with time)  and space complexity is required to understand the ability of the method to scale\n* SEvo accelerates momentum only over first-order neighbors. However, for some graph-related tasks, it is critical to handle long-range dependencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work intriguing.\nEach comment (presented in *italics*) is followed by its corresponding response.\n\n> *W1: The paper is difficult to follow. It would be nice to see a simple summary of the SEvo pipeline in the form of a scheme or algorithm.*\n\nThank you for your constructive feedback.\nThe SEvo pipeline can be summarized as follows (the algorithms for SEvo-enhanced SGD/Adam/AdamW are detailed in Appendix B.1):\n1. Compute gradients for embeddings;\n2. Update moment estimates (with certain modifications discussed in Section 2.4);\n3. Smooth the variations according to Eq. (7);\n4. Update the embeddings.\n\n> *W2: The SEvo formulae use the normalized adjacency matrix, so propagation of the gradients over the sampled node neighborhood is required. Suppose we have a large graph with a relatively high degree of each node. Mini-batch may have poorly correlated nodes, so many node embeddings should be updated simultaneously. This can lead to memory consumption issues and a notable increase in training time.*\n\nAs discussed in Appendix D.4, the complexity required for SEvo is comparable to that of the simplest GCN solely with neighborhood aggregation. Therefore, SEvo can be readily used for large graphs to which other GCNs can be applied.\n\n> *W3/Q2: The tables report only the average time across different datasets. The detailed computational (or non-aggregated graphs with time) and space complexity is required to understand the ability of the method to scale. How does the model training time scale with the size of the graph? Could you provide a graph epoch time vs. graph size e.g. for a SASRec?*\n\nWe present the computational and memory costs below (more training and inference times have been detailed in Appendix D.4). The graph size increases from Tools to Clothing.\n\n| | Tools | Beauty | Electronics | Clothing |\n|:-: |:-: |:-: |:-: |:-: |\n| #Users | 16,638 | 22,363 | 728,489 | 1,219,337 |\n| #Items | 10,217 | 12,101 | 159,729 | 376,378 |\n| #Edges | 134,476 | 198,502 | 6,737,580 | 11,282,445 |\n\n| Method (second/epoch) | Tools | Beauty | Electronics | Clothing |\n|:-: |:-: |:-: |:-: |:-: |\n| SR-GNN | 63.03s | 86.13s | 1,653.53s | 2,909.38s |\n| LESSR | 36.65s | 65.62s | 1,846.30s | 3,615.53s |\n| MAERec | 169.20s | 239.56s | 15,464.64s | 14,017.92s |\n| SASRec | 1.76s | 2.23s | 19.94s | 25.20s |\n| SASRec+SEvo | 1.89s | 2.35s | 22.16s | 33.47s |\n\n| Method (GPU memory) | Tools | Beauty | Electronics | Clothing |\n|:-: |:-: |:-: |:-: |:-: |\n| SR-GNN | 1,214M | 1,212M | 2,056M | 2,328M |\n| LESSR | 1,618M | 1,660M | 19,868M | 20,772M |\n| MAERec | 1,952M | 1,972M | 4,664M | 7,478M |\n| SASRec | 2,064M | 2,036M | 2,510M | 3,282M |\n| SASRec+SEvo | 2,074M | 2,046M  | 2,908M | 4,080M |\n\n\nWe have the following observations:\n1) Compared to vanilla SASRec, the implementation of SEvo incurs minimal computational and memory costs.\n2) Due to the high sparsity of recommendation datasets, the additional cost increases acceptably as the graph size increases.\n3) The costs associated with SEvo are negligible compared to other GNN-based sequence models, including SR-GNN, LESSR, and MAERec. Surprisingly, the training time for these models on Tools (the smallest dataset) greatly exceeds the time required for SEvo on Clothing (the largest dataset). This limitation hinders their application in real recommendation scenarios.\n\n> *W4/Q3: SEvo accelerates momentum only over first-order neighbors. However, for some graph-related tasks, it is critical to handle long-range dependencies. Can high-order proximity / long-range dependencies be incorporated using SEvo?*\n\nThanks for the comment but there is a misunderstanding about this.\nSEvo can intrinsically handle long-range dependencies as reflected in Eq. (7).\nMoreover, as investigated in Appendix D.3 and Table 7, SEvo can benefit from long-range dependencies when $L > 1$, with better results when $L=$ 2 or 3.\n\n\n> *Q1: How does SEvo work in mini-batch fashion?*\n\n**Graph sampling** methods [R1, R2] can be applied in a manner similar to that employed for other GCNs. Furthermore, it is worth noting that **graph partition** approaches [R3, R4] are more appropriate here, as the items/entities in a mini-batch are not necessarily included in the graph utilized for SEvo. When the original graph is too large for practical applications, it can be **pre-sliced** into multiple subgraphs for subsequent training. Moreover, if these smaller subgraphs are mutually exclusive, parallel updates can be performed for better acceleration and accuracy.\n\n> *Q1.5:  Does it require specific batch preparations, or should it be smaller on average?*\n\nDo you mean the sampled/sliced sub-graphs should be smaller on average?\nWe think the bigger the better, if possible.\n\n\n> *L1: The time and space complexity analyses of the method are required to understand its scalability.*\n\nAs discussed in Appendix D.4, the **time complexity** of SEvo is mainly determined by the arithmetic operations of $\\mathbf{\\tilde{A}}^l \\Delta \\mathbf{E}, l=1,2, \\ldots, L$.\nAssuming that the number of non-zero entries of $\\mathbf{\\tilde{A}}$ is $S$, the complexity required is about $\\mathcal{O}(LSd)$. On the other hand, the **additional space complexity** of SEvo is $\\mathcal{O}(S)$ for the storage of the normalized adjacency matrix.\nBecause the recommendation datasets are known for high sparsity (i.e., $S$ is very small), the actual overhead can be reduced to a very low level.\n\n\n[R1] Hamilton W. L., et al. Inductive representation learning on large graphs. NeurIPS, 2017.\n\n[R2] Zou D., et al. Layer-dependent importance sampling for training deep and large graph convolutional networks. NeurIPS, 2019.\n\n[R3] Chiang W., et al. Cluster-GCN: An efficient algorithm for training deep and large graph convolutional networks. KDD, 2019.\n\n[R4] Liu X., et al. Survey on graph neural network acceleration: An algorithmic perspective. IJCAI, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hx2sAaeBSV", "reviewer": "Reviewer_9GVe", "summary": "In this paper, the authors propose a novel optimization algorithm that is talored for recommender systems. It incorporates graph structural information into the optimization process, aleviating the burden of performing GNN for RS. The convergence of the algorithm is theoretically demonstrated. Besides, it could be incorporated into existing well-performed optimizers like AdamW. The experiments are conducted to test its effectiveness on different types of recommendation models and consistent performance improvements are observed.", "questions": "In Table 6, the proposed optimization algorithm even improves the GNN-based recommendation models. What is the reason for this phenomenon?", "limitations": "The authors have addressed the limitations in Section 5 and Appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is innovative in its consideration of utilizing graph information the algorithm level, not model level, for recommender systems. The proposed new algorithm is theoretically guaranteed to converge.\n2. The application of the proposed algorithm is widely discussed, including its incorporation into existing popular optimizers and its combination with knowledge distillation for recommender systems.\n3. The experiments are extensively conducted, including different size of datasets, different types of recommendation backbones and baselines. The results demonstrate both the effectiveness and efficiency of the proposed algorithm for improving existing recommendation models.", "weaknesses": "1. The technique in Section 2.4 is a little too specific for AdamW.\n2. Table 5 seems to have an error. In the last row, 0.568 is not the best result. Is it just a typo? Or does +DKD not further improve the performance in this case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work innovative.\nEach comment (presented in *italics*) is followed by its corresponding response.\n\n> *W1: The technique in Section 2.4 is a little too specific for AdamW.*\n\nThank you for your feedback.\nAdamW was chosen as an example because it is the most widely used optimizer in recommender systems other than Adam.\nSimilar modifications can also be implemented for other optimizers if they incorporate an **exponential moving average momentum term** (e.g., Adamax [R1], AMSGrad [R2], AdaBelif [R4]) and can decouple the weight decay regularization [R3].\n\n> *W2: Table 5 seems to have an error. In the last row, 0.568 is not the best result. Is it just a typo? Or does +DKD not further improve the performance in this case?*\n\n|Seed| HR@1 | HR@5| HR@10| NDCG@5| NDCG@10|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|0\t|0.01744\t|0.04109\t|0.05755\t|0.02956\t|0.03489|\n|1\t|0.01646\t|0.04087\t|0.05643\t|0.02907\t|0.03404|\n|2\t|0.01655\t|0.04092\t|0.05666\t|0.02900\t|0.03406|\n|3\t|0.01641\t|0.04092\t|0.05612\t|0.02878\t|0.03370|\n|4\t|0.01619\t|0.03971\t|0.05746\t|0.02811\t|0.03381|\n\nThank you for pointing out this mistake. \n**We reviewed the results (for the 5 seeds shown above) and found that '+DKD' indeed does not improve the HR@10 performance.**\nFurthermore, the overall comparison in Table 2 indicates the common conclusion: the application of SEvo is primarily beneficial for the top-ranked targets. \nThis is understandable as SEvo encourages the related nodes to be closer; conversely, it may adversely affect the targets being less related to the historical items.\nTo address this problem, graphs describing multiplex relations should be introduced, which is left as future work as discussed in Section 5.\nWe will further analyze this interesting observation in the revised manuscript.\nThank you once again for your meticulous review.\n\n\n> *Q1: In Table 6, the proposed optimization algorithm even improves the GNN-based recommendation models. What is the reason for this phenomenon?*\n\n\nWe believe that it is challenging to exploit both structural and sequential information for these GNN-based recommenders. LESSR and MAERec have developed sophisticated architectures for this purpose; however, they remain suboptimal in effectively utilizing either structural or sequential information alone. To be specific, the local session graph used by LESSR is insufficient to model the necessary structural and sequential information. In contrast, MAERec has improved its performance in this regard due to the use of two separate modules; however, effectively fusing these modules remains a challenging task.\n\n\n[R1] Kingma D. P., et al. Adam: A method for stochastic optimization. ICLR, 2015.\n\n[R2] Reddi S. J., et al. On the convergence of adam and beyond. ICLR, 2018.\n\n[R3] Loshchilov I., et al. Decoupled weight decay regularization. ICLR, 2019.\n\n[R4] Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. NeurIPS, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for reviewing our paper and providing us with their insightful comments.
We are excited that the reviewers found our work novel and well-written, and pleased that they were satisfied with both our theoretical analysis and experimental results.
We do our best to respond to reviewers' comments so that the work is of a higher standard.
Some of the major critiques are listed below; each reviewer's comments will be addressed point by point.

1. Reviewer Um97 expressed concerns regarding the substantial success of SEvo on large-scale datasets. We have thoroughly examined our reproducibility results and ensured that the comparisons are conducted fairly.
2. Reviewer DLFs thought that SEvo may apply to general graph datasets. We believe that both the motivation and design are specifically tailored to address the two major challenges of recommendation.
3. Reviewer feMV had concerns about the computational and space complexity of SEvo. We have carried out a complexity analysis supported by experimental evidence, which shows that SEvo is scalable and efficient compared to the simplest GCN.
4. Reviewer 9GVe carefully reviewed the paper and found a few typos. We have made modifications in the revised manuscript.

Please check our rebuttal and let us know if you have any further questions.

Meta-review: Post rebuttal, all reviewers have agreed to accept the paper. I concur that the paper offers a valuable graph embedding update mechanism for recommender models. Therefore, I recommend acceptance. Please ensure that the necessary revisions are incorporated into the final version.
Decision: Accept (poster)