Title: Graph-enhanced Optimizers for Structure-aware Recommendation Embedding Evolution
Venue: NeurIPS 2024
Accepted: yes
Keywords: Embedding optimization

Recommender systems

Graph

Optimizers
Abstract: Embedding plays a key role in modern recommender systems because they are virtual representations of real-world entities and the foundation for subsequent decision-making models.  In this paper, we propose a novel embedding update mechanism, Structure-aware Embedding Evolution (SEvo for short), to encourage related nodes to evolve similarly at each step. Unlike GNN (Graph Neural Network) that typically serves as an intermediate module, SEvo is able to directly inject graph structural information into embedding with minimal computational overhead during training.  The convergence properties of SEvo along with its potential variants are theoretically analyzed to justify the validity of the designs.  Moreover, SEvo can be seamlessly integrated into existing optimizers for state-of-the-art performance. Particularly SEvo-enhanced AdamW with moment estimate correction demonstrates consistent improvements across a spectrum of models and datasets, suggesting a novel technical route to effectively utilize graph structural information beyond explicit GNN modules.

Graph-enhanced Optimizers for Structure-aware Recommendation Embedding Evolution

Abstract

Embedding plays a key role in modern recommender systems because they are virtual representations of real-world entities and the foundation for subsequent decision-making models. In this paper, we propose a novel embedding update mechanism, Structure-aware Embedding Evolution (SEvo for short), to encourage related nodes to evolve similarly at each step. Unlike GNN (Graph Neural Network) that typically serves as an intermediate module, SEvo is able to directly inject graph structural information into embedding with minimal computational overhead during training. The convergence properties of SEvo along with its potential variants are theoretically analyzed to justify the validity of the designs. Moreover, SEvo can be seamlessly integrated into existing optimizers for state-of-the-art performance. Particularly SEvo-enhanced AdamW with moment estimate correction demonstrates consistent improvements across a spectrum of models and datasets, suggesting a novel technical route to effectively utilize graph structural information beyond explicit GNN modules. Our code is available at ❤/a116/a116♣/a115✿✴✴❣✐/a116❤✉❜✳❝♦♠✴▼❚❛♥❞❍❏✴❙❊✈♦ .

1 Introduction

Surfing Internet leaves footprints such as clicks [15], browsing [5], and shopping histories [58]. For a modern recommender system [6, 12], the entities involved ( e.g. , goods, movies) are typically embedded into a latent space based on these interaction data. As the embedding takes the most to construct and is the foundation to subsequent decision-making models, its modeling quality directly determines the final performance of the entire system. According to the homophily assumption [31, 59], it is natural to expect that related entities have closer representations in the latent space. Note that the similarity between two entities refers specifically to those extracted from interaction data [47] or prior knowledge [37]. For example, goods selected consecutively by the same user or movies of the same genre are often perceived as more relevant. Graph neural networks (GNNs) [2, 10, 14] are a widely adopted technique to exploit such structural information, in concert with a weighted adjacency matrix wherein each entry characterizes how closely two nodes are related. Rather than directly injecting structural information into embedding, GNN typically serves as an intermediate module in the recommender system. However, designing a versatile GNN module suitable for various recommendation scenarios is challenging. This is especially true for sequential recommendation [8, 51], which needs to take into account both structural and sequential information at the same time. Moreover, the post-processing fashion inevitably increases the overhead of training and inference, limiting the scalability for real-time recommendation.

In this work, we aim to directly inject graph structural information into embedding through a novel embedding update mechanism. Figure 1 (a) illustrates a normal embedding evolution process, in

(a)

≈

Structure-aware Embedding Evolution

(b)

Structure-aware Embedding Evolution

(b)

(d)

(c)

which the embedding E is updated at step t as follows:

Note that the variation ∆ E is primarily determined by the (anti-)gradient. It points to a region able to decrease a loss function concerning recommendation performance [35], but lacks an explicit mechanism to ensure that the variations between related nodes are similar. The embeddings thus cannot be expected to capture model pairwise relations while minimizing the recommendation loss.

Conversely, structural information can be effectively learned if related nodes evolve similarly at each update. The structure-aware embedding evolution (SEvo), depicted in Figure 1 (b), is developed for this goal. A special transformation is applied so as to meet both smoothness and convergence [57]. Given that these two criteria inherently conflict to some extent, we resort to a graph regularization framework [57, 7] to balance them. While analogous frameworks have been used to understand feature smoothing and modify message passing mechanisms for GNNs [28, 60], applying this to variations is not as straightforward as to features [24] or labels [19]. Previous efforts are capable of smoothing, but cannot account for strict convergence. The specific transformation form must be chosen carefully; subtle differences may slow down or even kill training. Through comprehensive theoretical analysis, we develop an applicable solution with a provable convergence guarantee.

Apart from the anti-gradient, the variation ∆ E can also be derived from the moment estimates. Therefore, existing optimizers, such as SGD [41] and Adam [23], can benefit from SEvo readily. In contrast to Adam, AdamW [27] decouples the weight decay from the optimization step, making it more compatible with SEvo as it is unnecessary to smooth the weight decay as well. Furthermore, we recorrect the moment estimates of SEvo-enhanced AdamW when encountering sparse gradients. This modification enhances its robustness across a variety of recommendation models and scenarios. Extensive experiments over six public datasets have demonstrated that it can effectively inject structural information to boost recommendation performance. It is important to note that SEvo does not alter the inference logic of the model, so the inference time is exactly the same and very little computational overhead is required during training.

The main contributions of this paper can be summarized as follows. 1) The graph regularization framework [57, 7] has been widely used for feature/label smoothing. To the best of our knowledge, we are the first to apply it to variations as an alternative to explicit GNN modules for recommendation. 2) The final formulation of SEvo is non-trivial (previous iterative [57] or Neumann series [39] approximation methods proved to be incompatible in this case) and comes from comprehensive theoretical analyses. 3) Wefurther present SEvo-enhanced AdamW by integrating SEvo and recorrecting the original moment estimates. These modifications demonstrate consistent performance, yielding average 9% ∼ 23% improvements across a spectrum of models. For larger-scale datasets containing millions of nodes, the performance gains can be as high as 40% ∼ 139%. 4) Beyond interaction

data, we preliminarily explore the pairwise similarity estimation based on other prior knowledge: node categories to promote intra-class representation proximity, and knowledge distillation [18] to encourage a light-weight student to mimic the embedding behaviors of a teacher model.

2 Structure-aware Embedding Evolution

In this section, we first introduce some necessary terminology and concepts, in particular smoothness. SEvo and its theoretical analyses will be detailed in Section 2.2 and 2.3. The proofs hereafter are deferred to Appendix A.

2.1 Preliminaries

Notations and terminology. Let V = { v 1 , . . . , v n } denote a set of nodes and A = [ w ij ] ∈ R n × n a symmetric adjacency matrix, where each entry w ij = w ji characterizes how closely v i and v j are related. They jointly constitute the graph G = ( V , A ) . For example, V could be a set of movies, and w ij is the frequency of v i and v j being liked by the same user. Denoted by D ∈ R n × n the diagonal degree matrix of A , the normalized adjacency matrix and the corresponding Laplacian matrix are defined as ˜ A = D -1 / 2 AD -1 / 2 and ˜ L = I -˜ A , respectively. For ease of notation, we use ⟨· , ·⟩ to denote the inner product, ⟨ x , y ⟩ = x T y for vectors and ⟨ X , Y ⟩ = Tr ( X T Y ) for matrices. Here, Tr ( · ) denotes the trace of a given matrix.

Smoothness. Before delving into the details of SEvo, it is necessary to present a metric to measure the smoothness [57, 7] of node features X as a whole. Denoted by x i , x j the row vectors of X and d i = ∑ j w ij , we have

This term is often used as graph regularization for feature/label smoothing [24, 19]. A lower J smoothness indicates smaller difference between closely related pairs of nodes, and in this case X is considered smoother. However, smoothness alone is not sufficient from a performance perspective. Over-emphasizing this indicator instead leads to the well-known over-smoothing issue [26]. How to balance variation smoothness and convergence is the main challenge to be addressed below.

2.2 Methodology

The entities involved in a recommender system are typically embedded into a latent space [6, 12], and the embeddings in E ∈ R n × d are expected to be smooth so that related nodes are close to each other. As discussed above, E is learnable and updated at step t by Eq. (1), where the variation ∆ E is mainly determined by the (anti-)gradient. For example, ∆ E = -η ∇ E L when gradient descent with a learning rate of η is used to minimize a loss function L . However, the final embeddings based on this evolution process may be far from sufficient smoothness because: 1) The variation ∆ E points to the region able to decrease the loss function concerning recommendation performance, but lacks an explicit smoothness guarantee. 2) As numerous item embeddings (millions of nodes in practice) to be trained together for a recommender system, the variations of two related nodes may be quite different due to the randomness from initialization and mini-batch sampling.

We are to design a special transformation ψ ( · ) to smooth the variation so that the evolution deduced from the following update formula is structure-aware,

Recall that, in this paper, the similarity is confined to quantifiable values in the adjacency matrix A , in which more related pairs are weighted higher. Therefore, this transformation should encourage pairs of nodes connected with higher weights to evolve more similarly than those connected with lower weights. This can be boiled down to structure-aware transformation as defined below.

Definition 1 (Structure-aware transformation) . The transformation ψ ( · ) is structure-aware if

On the other hand, the transformation must ensure convergence throughout the evolution process, which means that the transformed variation should not differ too much from the original. For the sake of theoretical analysis, the ability to maintain the update direction will be used to qualitatively depict this desirable property below, though a quantitative squared error will be employed later.

Definition 2 (Direction-aware transformation) . The transformation ψ ( · ) is direction-aware if

̸

These two criteria inherently conflict to some extent. We resort to a hyperparameter β ∈ [0 , 1) to make a trade-off and the desired transformation is the corresponding minimum; that is,

A larger β indicates a stronger smoothness constraint and ψ ∗ (∆ E ) reduces to ∆ E when β → 0 . Geometrically, as shown in Figure 1 (c), ψ ∗ (∆ E ) can be interpreted as a projection of ∆ E onto the region with proper smoothness. Taking the gradient to zero could give a closed-form solution, but it requires prohibitive arithmetic operations and memory overhead, which is particularly timeconsuming in recommendation due to the large number of nodes. Zhou et al. [57] suggested a L -layer iterative approximation to circumvent this problem (with ˆ ψ 0 (∆ E ) = ∆ E ):

The resulting transformation is essentially a momentum update that aggregates higher-order information layer by layer. Analogous message-passing mechanisms have been used in previous GNNs such as APPNP [24] and C&S [19]. However, this commonly used approximate solution is incompatible with SEvo; sometimes, variations after the transformation may be opposite to the original direction, resulting in a failure to converge.

Theorem 1. The iterative approximation is direction-aware for all possible normalized adjacency matrices and L ≥ 0 , if and only if β < 1 / 2 . In contrast, the Neumann series approximation ˆ ψ nsa (∆ E ) = (1 -β ) ∑ L l =0 β l ˜ A l ∆ E is structure-aware and direction-aware for any β ∈ [0 , 1) .

As suggested in Theorem 1, a feasible compromise for ˆ ψ iter is to restrict β to [0 , 1 / 2) , but this may cause a lack of smoothness. The Neumann series approximation [39] appears to be a viable alternative as it qualitatively satisfies both desirable properties. Nonetheless, this transformation can be further improved for a faster convergence rate based on the analysis presented next.

2.3 Convergence Analysis for Further Modification

In general, the recommender system has some additional parameters θ ∈ R m except for embedding to be trained. Therefore, we analyze the convergence rate of the following gradient descent strategy:

wherein SEvo is performed on the embedding and a normal gradient descent is applied to θ . To make the analysis feasible, some mild assumptions on the loss function should be given: L : R n × d × R m → R is a twice continuously differentiable function whose first derivative is Lipschitz continuous for some constant C . Then, we obtain the following properties.

Theorem 2 (Informal) . If η = η ′ = 1 /C , the convergence rate after T updates is O ( C/ ((1 -β ) 2 T )) . If we adopt a modified learning rate of η = 1 (1 -β L +1 ) C for embedding, the convergence rate could be improved to O ( C/ ((1 -β ) T )) .

Remark 1. Our main interest is to justify the designs of SEvo rather than to pursue a particular convergence rate, so some mild assumptions suggested in [32] are adopted here. By introducing the steepest descent for quadratic norms [1], better convergence can be obtained with stronger assumptions.

Two conclusions can be drawn from Theorem 2. 1) The theoretical upper bound becomes worse when β → 1 . This makes sense since ˆ ψ nsa ( ∇ E L ) is getting smoother and further away from the

original descent direction. 2) A modified learning rate for embedding can significantly improve the convergence rate. This phenomenon can be understood readily if we notice the fact that

Thus, the modified learning rate is indeed to offset the scaling effect induced by SEvo. In view of this, we directly incorporate this factor into SEvo to avoid getting stuck in the learning rate search, yielding the final desired transformation:

It can be shown that ˆ ψ is structure-aware and direction-aware, and converges to ψ as L increases.

2.4 Integrating SEvo into Existing Optimizers

Algorithm 1: SEvo-enhanced AdamW. Differences from the original AdamW are colored in blue. The matrix operation below are element-wise.

Input: embedding matrix E , learning rate η , momentum factors β 1 , β 2 , β ∈ [0 , 1) , weight decay λ .

foreach step t do

Update first/second moment estimates for each node i :

●❡/a116

̸

̸

Compute bias-corrected first/second moment estimates:

Output: optimized embeddings E .

SEvo can be seamlessly integrated into existing optimizers since the variation involved in Eq. (3) can be extended beyond the (anti-)gradient. For SGD with momentum, the variation becomes the first moment estimate, and for Adam this is jointly determined by the first/second moment estimates. AdamW is also widely adopted for training recommenders. Unlike Adam whose moment estimate is a mixture of gradient and weight decay, AdamW decouples the weight decay from the optimization step, which is preferable since it makes no sense to require the weight decay to be smooth as well. However, in very rare cases, SEvo-enhanced AdamW fails to work very well. We next try to ascertain the causes and then improve the robustness of SEvo-enhanced AdamW.

Denoted by g := ∇ e L ∈ R d the gradient for a node embedding e and g 2 := g ⊙ g the element-wise square, AdamW estimates the first and second moments at step t using the following formulas

where β 1 , β 2 are two momentum factors. Then the original AdamW updates embeddings by

Note that the bias-corrected estimates ˆ m t = m t / (1 -β t 1 ) and ˆ v t = v t / (1 -β t 2 ) are employed for numerical stability [23]. In practice, only a fraction of nodes are sampled for training in a minibatch, so the remaining embeddings have zero gradients. In this case, the sparse gradient problem may introduce some unexpected 'biases' as depicted below.

Update via SEvo:

✴✴

❣/a114❛❞✐❡♥/a116/a115

✇✳/a114✳/a116

E

Proposition 1. If a node is no longer sampled in subsequent p batches after step t , we have ∆ e t + p -1 = κ · β p 1 √ β p 2 ∆ e t -1 , and the coefficient of κ is mainly determined by t .

Considering a common case β 2 ≈ β 1 , the right-hand side approaches O ( β p/ 2 1 ) . The step size for inactive nodes then gets slower and slower during idle periods. This seems reasonable as their moment estimates are becoming outdated; however, this effect sometimes prevents the variation from being smoothed by SEvo. We hypothesize that this is because SEvo itself tends to assign more energy to active nodes and less energy to inactive nodes. So this auto-attenuation effect of the original AdamW is somewhat redundant. Fortunately, there is a feasible modification to make SEvo-enhanced AdamW more robust:

Theorem 3. Under the same assumptions as in Proposition 1, ∆ e t + p -1 = ∆ e t -1 if the moment estimates are updated in the following manner when g t = 0 ,

As can be seen, when sparse gradients are encountered, the approach in Theorem 3 is actually to estimate the current gradient from previous moments. The coefficients 1 / (1 -β t -1 1 ) and 1 / (1 -β t -1 1 ) are used here for unbiasedness (refer to Appendix A.3 for detailed discussion and proofs). We summarize the SEvo-enhanced AdamW in Algorithm 1 and the modifications for Adam and SGD in Appendix B.1, with an empirical comparison presented in Section 3.3.

The previous discussion lays the technical basis for injecting graph structural information, but the final recommendation performance is determined by how 'accurate' the similarity estimation is. Following other GNN-based sequence models [48, 49], the number of consecutive occurrences across all sequences will be used as the pairwise similarity w ij . In other words, items v i and v j that appear consecutively more frequently are assumed more related. Notably, we would like to emphasize that SEvo can readily inject other types of knowledge beyond interaction data. We have made some preliminary efforts in Appendix C and observed some promising results.

3 Experiments

In this section, we comprehensively verify the superiority of SEvo. We focus on sequential recommendation for two reasons: 1) This is the most common scenario in practice; 2) Utilizing both sequential and structural information is beneficial yet challenging. We showcase that SEvo is a promising way to achieve this goal. It is worth noting that although technically SEvo can be applied to general graph embedding learning [4], we contend SEvo-AdamW is especially useful for mitigating the inconsistent embedding evolution caused by data sparsity, while effectively injecting structural information in conjunction with other types of information.

Due to space constraints, this section presents only the primary results concerning accuracy, efficiency, and some empirical evidence that supports the aforementioned claims. We begin by introducing the datasets, evaluation metrics, baselines, and implementation details.

Datasets. Six benchmark datasets are considered in this paper. The first four datasets including Beauty, Toys, Tools, and MovieLens-1M are commonly employed in previous studies for empirical comparisons. Additionally, two larger-scale datasets, Clothing and Electronics, are used to assess SEvo's scalability in scenarios involving millions of nodes. Following [22, 13], we filter out users and items with less than 5 interactions, and the validation set and test set are split in a leave-one-out fashion,

namely the last interaction for testing and the penultimate one for validation. This splitting allows for fair comparisons, either for sequential recommendation or collaborative filtering. The dataset statistics are presented in Table 1.

Evaluation metrics. For each user, the predicted scores over all items will be sorted in descending order to generate top-N candidate lists. We consider two widely-used evaluation metrics, HR@N

GNN-based

MF or RNN/Transformer-based

▲

%

p

-value

LightGCN SR-GNN

LESSR

MAERec

MF-BPR

GRU4Rec

SASRec

BERT4Rec

STOSA

+SEvo

+SEvo

+SEvo

+SEvo

+SEvo

Beauty

HR@1

0.0074

0.0059

0.0088

0.0113

0.0071

0.0076

0.0061

0.0094

0.0120

0.0154

0.0157

0.0172

0.0166

0.0216

30.2%

6.90E-05

HR@5

0.0289

0.0247

0.0322

0.0424

0.0272

0.0293

0.0233

0.0326

0.0404

0.0499

0.0479

0.0522

0.0479

0.0544

13.5%

7.69E-04

HR@10

0.0472

0.0406

0.0506

0.0662

0.0454

0.0480

0.0395

0.0524

0.0634

0.0759

0.0716

0.0772

0.0680

0.0774

8.2%

2.66E-03

NDCG@5

0.0181

0.0152

0.0205

0.0269

0.0170

0.0184

0.0146

0.0209

0.0262

0.0328

0.0319

0.0350

0.0327

0.0383

17.2%

9.70E-05

NDCG@10

0.0240

0.0203

0.0264

0.0346

0.0228

0.0244

0.0198

0.0273

0.0336

0.0411

0.0395

0.0430

0.0391

0.0457

15.5%

8.02E-05

Toys

HR@1

0.0087

0.0100

0.0126

0.0171

0.0079

0.0099

0.0059

0.0080

0.0172

0.0192

0.0160

0.0175

0.0232

0.0267

15.3%

1.46E-03

HR@5

0.0279

0.0294

0.0352

0.0532

0.0267

0.0306

0.0209

0.0276

0.0506

0.0584

0.0430

0.0492

0.0571

0.0625

9.6%

5.70E-03

HR@10

0.0456

0.0439

0.0513

0.0796

0.0427

0.0477

0.0345

0.0446

0.0727

0.0844

0.0645

0.0723

0.0776

0.0872

9.5%

3.07E-04

NDCG@5

0.0183

0.0198

0.0240

0.0355

0.0174

0.0203

0.0134

0.0179

0.0342

0.0392

0.0297

0.0336

0.0406

0.0453

11.6%

2.48E-03

NDCG@10

0.0240

0.0245

0.0292

0.0440

0.0225

0.0258

0.0177

0.0234

0.0413

0.0475

0.0366

0.0410

0.0472

0.0532

12.7%

3.12E-05

Tools

HR@1

0.0067

0.0046

0.0045

0.0083

0.0058

0.0071

0.0053

0.0058

0.0099

0.0108

0.0074

0.0087

0.0095

0.0133

33.8%

1.23E-02

HR@5

0.0212

0.0162

0.0157

0.0271

0.0187

0.0225

0.0174

0.0208

0.0317

0.0337

0.0244

0.0279

0.0276

0.0350

10.5%

8.94E-03

HR@10

0.0326

0.0260

0.0263

0.0423

0.0293

0.0348

0.0272

0.0336

0.0466

0.0497

0.0405

0.0441

0.0417

0.0502

7.7%

8.25E-03

NDCG@5

0.0140

0.0103

0.0101

0.0177

0.0123

0.0147

0.0114

0.0133

0.0210

0.0223

0.0159

0.0183

0.0186

0.0244

15.9%

5.57E-03

NDCG@10

0.0176

0.0135

0.0135

0.0226

0.0157

0.0187

0.0145

0.0174

0.0258

0.0274

0.0211

0.0235

0.0231

0.0293

13.4%

3.79E-03

MovieLens

HR@1

0.0124

0.0383

0.0513

0.0439

0.0117

0.0133

0.0487

0.0487

0.0490

0.0517

0.0681

0.0733

0.0457

0.0510

7.6%

3.81E-02

HR@5

0.0495

0.1297

0.1665

0.1563

0.0470

0.0509

0.1625

0.1663

0.1599

0.1670

0.2069

0.2127

0.1409

0.1569

2.8%

4.17E-03

HR@10

0.0866

0.2009

0.2539

0.2462

0.0836

0.0876

0.2522

0.2568

0.2492

0.2567

0.3018

0.3075

0.2185

0.2356

1.9%

9.32E-02

NDCG@5

0.0307

0.0842

0.1092

0.1003

0.0291

0.0319

0.1061

0.1075

0.1046

0.1096

0.1387

0.1437

0.0932

0.1041

3.6%

4.19E-04

NDCG@10

0.0427

0.1071

0.1373

0.1292

0.0408

0.0436

0.1350

0.1366

0.1333

0.1385

0.1693

0.1743

0.1181

0.1295

2.9%

1.30E-02

Avg. Improv.

+13.1%

+23.4%

+12.3%

+9.6%

+17.5%

Avg. Train. Time

2,820s

43,783s

62,457s

31,674

2,863s

+173s

3,582s

+124s

532s

+37s

1,256s

+288s

2,087s

+127s

Avg. Inf. Time

1.19s

11.18s

9.34s

2.73s

1.13s

+0s

1.80s

+0s

1.88s

+0s

1.61s

+0s

6.72

+0s

(Hit Rate) and NDCG@N (Normalized Discounted Cumulative Gain). The former measures the rate of successful hits among the top-N recommended candidates, while the latter takes into account the ranking positions and assigns higher scores in order of priority.

Baselines. We select four GNN-based models (LightGCN [16], SR-GNN [48], LESSR [8], and MAERec [51]) as performance and efficiency benchmarks. Since this study is not to develop a new model, four classic sequence models (GRU4Rec [17], SASRec [22], BERT4Rec [40], and STOSA [13]) are utilized as backbones to validate the effectiveness of SEvo. Besides, MF-BPR [35] is also considered here as a backbone without sequence modeling. We carefully tune the hyperparameters according to their open-source code and experimental settings.

Implementation details. Since the purpose is to study the effectiveness of SEvo, only hyperparameters concerning optimization are retuned, including learning rate ([1e-4, 5e-3]), weight decay ([0, 0.1]) and dropout rate ([0, 0.7]). Other hyperparameters in terms of the architecture are consistent with the corresponding baseline. For a fair comparison, the number of layers L is fixed to 3 as in other GNN-based recommenders. As for the hyperparameter in terms of the degree of smoothness, we found β = 0 . 99 performs quite well in practice. The loss functions follow the suggestions in the respective papers, given that SEvo can be applied to any of them.

3.1 Overall Comparison

In this section, we are to verify the effectiveness of SEvo in boosting recommendation performance. Table 2 compares the overall performance and efficiency over four widely used datasets, and Table 3 further provides the results on two large-scale datasets with millions of nodes.

Firstly, GNN-based models seem to over-emphasize structural information but lack full exploitation of sequential information. Their performance is only on par with GRU4Rec. On the Tools dataset, SR-GNN and LESSR are even inferior to LightGCN, a collaborative filtering model with no access to sequential information. MAERec makes a slightly better attempt to combine the two by learning structural information through graph-based reconstruction tasks. It employs a SASRec backbone for recommendation to allow for a better utilization of sequential information. Despite the identical recommendation backbone, SASRec trained with SEvo-enhanced AdamW enjoys significantly better performance. Not to mention the consistent gains on other state-of-the-art methods such as BERT4Rec and STOSA, from 2% to 30% according to the last two columns of Table 2. Overall, the promising improvements from the SEvo enhancement suggest a different route to exploit graph structural information, especially in conjunction with sequence modeling.

0

2000

4000

6000

8000

0

.

5

1

.

0

Loss

Beauty

Original

SEvo

SEvo w/o rescaling

0

2000

4000

6000

8000

t

0

.

5

1

.

0

Loss

MovieLens-1M

(a) Loss curves

3

.

0

3

.

5

4

.

0

4

.

5

5

.

0

J

smoothness

×

10

4

∆

E

1

2

3

4

5

×

10

4

ˆ

ψ

(∆

E

)

0

.

4

0

.

6

0

.

8

1

.

0

×

10

4

smoother

E

t

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

2

.

0

J

smoothness

×

10

4

(I)

t

0

.

5

1

.

0

1

.

5

2

.

0

×

10

4

(II)

β

1

.

0

1

.

5

2

.

0

2

.

5

×

10

3

smoother

(III)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

β

(b) Smoothness

Secondly, either dynamic graph construction in SR-GNN and LESSR, or path sampling in MAERec, requires heavy computational overhead, which directly causes the training failures on large-scale datasets like Electronics and Clothing. Even worse, these high costs associated with SR-GNN and LESSR are inevitable during inference. In contrast, SEvo does not alter the model inference logic at all, thereby maintaining consistent inference time. The computational overhead required in training is also negligible compared to previous graph-enhanced models that employ GNNs as intermediate modules. For example, SASRec with SEvo consumes only 10 minutes compared to the hours of training time required for MAERec. When millions of nodes are encountered in Table 3, each epoch

demands just a few more seconds. SEvo is arguably superior to these cumbersome GNN modules in real-world applications. Combining Table 2 and Table 3, it can be inferred that the performance gain increases as the number of items increases. This can be explained by the fact that the randomness of sampling leads to a much more inconsistent evolution when more and more nodes are encountered [56]. SEvo thus plays an increasingly important role as it is capable of imposing direct consistency constraints on embeddings.

Since SASRec is a pioneer in the field of sequential recommendation, it will serve as the default backbone for subsequent studies.

3.2 Empirical Analysis

Convergence comparison. In Section 2.3, we theoretically verified the necessity of rescaling the original Neumann series approximation for faster convergence. Figure 2a shows the loss curves of SASRec trained with AdamW under identical settings other than the form of SEvo. Without rescaling, SASRec exhibits significantly slower convergence, consistent with the conclusion in Theorem 2. While the theoretical worst-case convergence rate of the corrected SEvo is only 1% of the normal gradient descent when β = 0 . 99 , its practical performance is much better. SASRec trained with SEvo-enhanced AdamW initially converges marginally slower and catches up in the final stage.

SGD

Adam

AdamW

Optimizer

0

.

020

0

.

025

0

.

030

0

.

035

0

.

040

NDCG@10

Beauty

Original

+SEvo

(a) Optimizers

0

.

0

0

.

5

1

.

0

β

0

.

02

0

.

03

0

.

04

NDCG@10

Beauty

Baseline

Neumann

Iterative

(b) Neumann versus Iterative















































(c) Moment estimate correction

Smoothness evaluation. Figure 2b demonstrates the variation's smoothness throughout the evolution process and the eventual embedding differences from β = 0 to β = 0 . 999 . (I) → (II): The original variations exhibit a similar degree of smoothness, but after transformation, they are quite different--smoother as β increases. (II) → (III): Consequently, the embedding trained with a stronger smoothness constraint becomes smoother as well. The structure-aware embedding evolution successfully makes related nodes closer in the latent space. Although smoothness is not the sole quality measure of embedding, combined with the analyses above, we can conclude that SEvo injects appropriate structural information under the default setting of β = 0 . 99 .

3.3 Ablation Study

SEvo for various optimizers. It is of interest to study whether SEvo can be extended to other commonly used optimizers such as SGD and Adam. Figure 3a compares NDCG@10 performance on Beauty and MovieLens-1M. For a fair comparison, the hyperparameters are tuned independently. It is evident that the performance of SGD, Adam, and AdamW improves significantly after integrating SEvo, with AdamW achieving the best as it does not need to smooth the weight decay.

Neumann series approximation versus iterative approximation. Theorem 1 suggests that the Neumann series approximation is preferable to the commonly used iterative approximation because the latter is not always direction-aware and thus a conservative hyperparameter of β is needed to ensure convergence. This conclusion can also be drawn from Figure 3b. When only a little smoothness is required, their performance is comparable as both approximations differ only at the last term. The iterative approximation however fails to ensure convergence once β > 0 . 7 on the Beauty dataset, potentially resulting in a lack of smoothness.

Moment estimate correction for AdamW. We compare SEvo-enhanced AdamW with or without moment estimate correction in Figure 3c, in which average relative improvements against the baseline are presented for each recommender. Overall, the two variants of SEvo-enhanced AdamW perform comparably, significantly surpassing the baseline. However, in some cases ( e.g. , GRU4Rec and STOSA), the moment estimate correction as suggested in Theorem 3 is particularly useful to improve performance. Recall that BERT4Rec is trained using the output softmax from a separate fully-connected layer that is fully updated at each step. This may explain why the correction has little effect on BERTRec. In conclusion, the results underscore the importance of the proposed modification in alleviating bias in moment estimates.

3.4 Applications of SEvo Beyond Interaction Data

We further explore the potential of applying SEvo to other types of prior knowledge. On the one hand, the category smoothness constraint can also be fulfilled through SEvo (see Appendix C.2), leading to progressively stronger clustering effects as β increases. This provides compelling visual evidence of why SEvo is inherently structure-aware. On the other hand, SEvo is arguably an efficient tool for transferring embedding knowledge (see Appendix C.3). Notice that the learning of other modules cannot be guided in the same way, so SEvo alone is still inferior to state-of-the-art knowledge distillation methods [21, 55]. Fortunately, SEvo and other methods can work together to further boost the recommendation performance.

4 Related Work

Recommender systems are developed to enable users to quickly and accurately find relevant items in diverse applications, such as e-commerce [58], online news [15] and social media [5]. Typically, the entities involved are embedded into a latent space [6, 12, 54], and then decision models are built on top of the embedding for tasks like collaborative filtering [16] and context/knowledge-aware recommendation [46, 44]. Sequential recommendation [36, 25] focuses on capturing users' dynamic interests from their historical interaction sequences. Early approaches adapted recurrent neural networks (RNNs) [17] and convolutional filters [42] for sequence modeling. Recently, Transformer [45, 11] has become a popular architecture for sequence modeling due to its parallel efficiency and superior performance. SASRec [22] and BERT4Rec [40] use unidirectional and bidirectional selfattention, respectively. Fan et al. [13] proposed a novel stochastic self-attention (STOSA) to model the uncertainty of sequential behaviors.

Graph neural networks [2, 10] are a type of neural network designed to operate on graph-structured data, in concert with a weighted adjacency matrix to characterize the pairwise relations between nodes. GNN equipped with this adjacency matrix can be used for message passing between nodes. The most relevant work is the optimization framework proposed in [57] for solving semi-supervised learning problems via a smoothness constraint. This graph regularization approach has recently inspired a series of work [7, 28, 60]. As opposed to applying it to smooth node representations [24] or labels [19], it is employed here primarily to balance smoothness and convergence on the variation.

Structural information in recommendation is typically learned through GNN as well, with specific modifications made to cope with like data sparsity [52, 29]. LightGCN [16] is a pioneering collaborative filtering work on modeling user-item relations, which removes nonlinearities for easier training. To further utilize sequential information, previous efforts focus on equipping sequence models with complex GNN modules, but this inevitably increases the computational cost of training and inference, making it unappealing for practical recommendation. For example, SR-GNN [48] and LESSR [8] need dynamically construct adjacency matrices for each batch of sequences. Differently, MAERec [51] proposes an adaptive data augmentation to boost a novel graph masked autoencoder, which learns to sample less noisy paths from semantic similarity graph for subsequent reconstruction tasks. The resulting strong self-supervision signals help the model capture more useful information.

5 Broader Impact and Limitations

Utilizing both sequential and structural information is beneficial yet challenging, and SEvo proposed in this paper suggests a novel and effective technical route for this purpose. Compared to other explicit GNN modules, SEvo is light-weight and easy-to-use in practice. These insights may inspire future research efforts regarding structure-aware optimization. However, there are still some limitations. Firstly, the training-free nature of SEvo makes it versatile, but also limits the expressive power. For a specific task, a sophisticated GNN module may be more desirable for achieving higher recommendation accuracy. Secondly, it might not be so straightforward to apply SEvo to the scenario involving multiple types of prior knowledge. Some efforts [43, 33] in the field of multiple graph learning have proposed some technically feasible solutions. However, these approaches still encounter challenges in terms of efficiency, particularly in the context of recommendation systems.

6 Conclusion and Future Work

In this work, we have proposed a novel update mechanism for injecting graph structural information into embedding. Theoretical analyses of the convergence properties motivate some necessary modifications to the proposed method. SEvo can be seamlessly integrated into existing optimizers. For AdamW, we recorrect the moment estimates to make it more robust. Besides, an interesting direction for future work is extending SEvo to multiplex heterogeneous graphs [3], as real-world entities often participate in various relation networks. Furthermore, we believe that SEvo holds potential for application to dynamic graph structures and incremental updates [53]. Two challenges may be encountered in practice: the computational overhead associated with the ongoing adjacency matrix normalization, and how to adaptively weaken the outdated historical information.

References

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. In International Conference on Learning Representations (ICLR) , 2014.

Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. Representation learning for attributed multiplex heterogeneous network. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 1358-1368, 2019.

Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Lowdimensional hyperbolic knowledge graph embeddings. In Annual Meeting of the Association for Computational Linguistics (ACL) , pages 6901-6914. ACL, 2020.

Jiajia Chen, Xin Xin, Xianfeng Liang, Xiangnan He, and Jun Liu. Gdsrec: Graph-based decentralizedcollaborative filtering for socialrecommendation. IEEE Transactions on Knowledge and Data Engineering (TKDE) , 2022.

Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. Behavior sequence transformer for e-commerce recommendation in alibaba. In Proceedings of the 1st International Workshop on Deep Learning Practice for High-dimensional Sparse Data , pages 1-4, 2019.

Siheng Chen, Aliaksei Sandryhaila, José MF Moura, and Jelena Kovacevic. Signal denoising on graphs via graph filtering. In IEEE Global Conference on Signal and Information Processing (GlobalSIP) , pages 872-876. IEEE, 2014.

Tianwen Chen and Raymond Chi-Wing Wong. Handling information loss of graph neural networks for session-based recommendation. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 1172-1180, 2020.

Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation , pages 103-111. ACL, 2014.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems (NeurIPS) , volume 29, 2016.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages 4171-4186. ACL, 2019.

Ahmed El-Kishky, Thomas Markovich, Serim Park, Chetan Verma, Baekjin Kim, Ramy Eskander, Yury Malkov, Frank Portman, Sofía Samaniego, Ying Xiao, et al. Twhin: Embedding the twitter heterogeneous information network for personalized recommendation. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 2842-2850, 2022.

Ziwei Fan, Zhiwei Liu, Yu Wang, Alice Wang, Zahra Nazari, Lei Zheng, Hao Peng, and Philip S Yu. Sequential recommendation via stochastic self-attention. In ACMWebConference (WWW) , pages 2036-2047, 2022.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML) , pages 1263-1272. PMLR, 2017.

Shansan Gong and Kenny Q Zhu. Positive, negative and neutral: Modeling implicit feedback in session-based news recommendation. In International ACM Conference on Research and Development in Information Retrieval (SIGIR) , 2022.

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In International ACM Conference on Research and Development in Information Retrieval (SIGIR) , pages 639648, 2020.

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. In International Conference on Learning Representations (ICLR) , 2016.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR , abs/1503.02531, 2015.

Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R. Benson. Combining label propagation and simple models out-performs graph neural networks. In International Conference on Learning Representations (ICLR) , 2021.

Tony Jebara, Jun Wang, and Shih-Fu Chang. Graph construction and b-matching for semisupervised learning. In International Conference on Machine Learning (ICML) , pages 441448, 2009.

SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu. Topology distillation for recommender system. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 829-839. ACM, 2021.

Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In IEEE International Conference on Data Mining (ICDM) , pages 197-206. IEEE, 2018.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR) , 2019.

Jiacheng Li, Yujie Wang, and Julian McAuley. Time interval aware self-attention for sequential recommendation. In International Conference on Web Search and Data Mining (WSDM) , pages 322-330, 2020.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Conference on Artificial Intelligence (AAAI) , 2018.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR) , 2018.

Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In ACM International Conference on Information and Knowledge Management (CIKM) , pages 1202-1211, 2021.

Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, and Xiuqiang He. UltraGCN: Ultra simplification of graph convolutional networks for recommendation. In International Conference on Information and Knowledge Management (CIKM) , 2021.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of a feather: Homophily in social networks. Annual Review of Sociology , pages 415-444, 2001.

Yurii Nesterov. Introductory lectures on convex programming volume i: Basic course. Lecture notes , 3(4):5, 1998.

Feiping Nie, Jing Li, and Xuelong Li. Self-weighted multiview clustering with multiple graphs. In International Joint Conference on Artificial Intelligence (IJCAI) , pages 2564-2570, 2017.

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3967-3976. IEEE, 2019.

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Schmidt-Thieme Lars. Bpr: Bayesian personalized ranking from implicit feedback. In Conference on Uncertainty in Artificial Intelligence (UAI) , 2009.

Guy Shani, David Heckerman, Ronen I Brafman, and Craig Boutilier. An mdp-based recommender system. Journal of Machine Learning Research (JMLR) , 6(9), 2005.

Kartik Sharma, Yeon-Chang Lee, Sivagami Nambi, Aditya Salian, Shlok Shah, Sang-Wook Kim, and Srijan Kumar. A survey of graph neural networks for social recommender systems. arXiv preprint arXiv:2212.04481 , 2022.

Daniel A Spielman. Spectral graph theory and its applications. In Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) , pages 29-38. IEEE, 2007.

Gilbert W Stewart. Matrix algorithms: volume 1: basic decompositions . SIAM, 1998.

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In ACM International Conference on Information and Knowledge Management (CIKM) , pages 1441-1450, 2019.

Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning (ICML) , pages 1139-1147. PMLR, 2013.

Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In ACM International Conference on Web Search and Data Mining (WSDM) , pages 565-573. ACM, 2018.

Wei Tang, Zhengdong Lu, and Inderjit S Dhillon. Clustering with multiple graphs. In IEEE International Conference on Data Mining (ICDM) , pages 1016-1021. IEEE, 2009.

Zhen Tian, Ting Bai, Wayne Xin Zhao, Ji-Rong Wen, and Zhao Cao. Eulernet: Adaptive feature interaction learning via euler's formula for CTR prediction. In International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) , pages 1376-1385. ACM, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS) , volume 30, 2017.

Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. Kgat: Knowledge graph attention network for recommendation. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 950-958, 2019.

Yu Wang, Yuying Zhao, Yi Zhang, and Tyler Derr. Collaboration-aware graph convolutional network for recommender systems. In ACM Web Conference (WWW) , pages 91-101, 2023.

Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-based recommendation with graph neural networks. In Conference on Artificial Intelligence (AAAI) , volume 33, pages 346-353, 2019.

Xin Xia, Hongzhi Yin, Junliang Yu, Qinyong Wang, Lizhen Cui, and Xiangliang Zhang. Selfsupervised hypergraph convolutional networks for session-based recommendation. In Conference on Artificial Intelligence (AAAI) , volume 35, pages 4503-4511, 2021.

Cong Xu, Jun Wang, and Wei Zhang. Stablegcn: Decoupling and reconciling information propagation for collaborative filtering. IEEE Transactions on Knowledge and Data Engineering (TKDE) , 2023.

Yaowen Ye, Lianghao Xia, and Chao Huang. Graph masked autoencoder for sequential recommendation. In International ACM Conference on Research and Development in Information Retrieval (SIGIR) , pages 321-330. ACM, 2023.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 974-983, 2018.

Jiaxuan You, Tianyu Du, and Jure Leskovec. Roland: graph learning framework for dynamic graphs. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 2358-2366, 2022.

Shengyu Zhang, Fuli Feng, Kun Kuang, Wenqiao Zhang, Zhou Zhao, Hongxia Yang, TatSeng Chua, and Fei Wu. Personalized latent structure learning for recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2023.

Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11943-11952. IEEE, 2022.

Wayne Xin Zhao, Zihan Lin, Zhichao Feng, Pengfei Wang, and Ji-Rong Wen. A revisiting study of appropriate offline evaluation for top-n recommendation algorithms. ACM Transactions on Information Systems (TOIS) , 41(2):1-41, 2022.

Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances in Neural Information Processing Systems (NeurIPS) , 16, 2003.

Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) , pages 1059-1068, 2018.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. 33:77937804, 2020.

Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In ACM Web Conference (WWW) , pages 1215-1226, 2021.

Contents

A Proofs

A.1 Proof of Theorem 1

In this part, we are to prove the structure-aware/direction-aware properties of the L -layer iterative approximation [57]:

and the L -layer Neumann series approximation [39]:

Fact 1. If L is odd, the geometric series S ( x ) = ∑ L k =0 x l is monotonically increasing when x ≤ 0 .

Proof. It is easy to show that and the derivative w.r.t x = 1 is

̸

is the largest eigenvalue of P .

If L is odd, S ′ ( x ) is positive when x ≤ 0 and in this case S ( x ) is monotonically increasing.

Lemma 1. Given a normalized adjacency matrix ˜ A ∈ R n × n , let the symmetric matrix deduced from the Neumann series approximation be

Denoted by λ min ( P ) , λ max ( P ) the smallest and largest eigenvalues of P , respectively, then we have ∀ β ∈ [0 , 1)

Proof. It is easy to shown that the eigenvalues of P are in the form of

where λ 1 ≤ λ 2 · · · ≤ λ n denote the eigenvalues of ˜ A . Recall that these eigenvalues all fall into [ -1 , 1] and λ n = 1 can be achieved exactly [38]. Hence,

In addition, notice that the last term (1 -β ) β L λ L is non-negative when L is even. Then, we can get a lower bound no matter L is odd or even:

The minimum of S ( λ ) must be achieved in [ -1 , 0] because S ( -λ ) ≤ S ( λ ) for any λ > 0 . In fact, in view of Fact 1, we know that S ( λ ) ≥ S ( -1) . Hence, we have

The last inequality holds because 2 ⌊ ( L -1) / 2 ⌋ +2 ≥ L . Therefore, the smallest eigenvalue of P must be greater than

Proposition 2. The Neumann series approximation is structure-aware and direction-aware for any β ∈ [0 , 1) .

Proof. In view of Lemma 1, we know

so P is positive definite and thus ˆ ψ nsa ( · ) is direction-aware. Also, notice that P has the same eigenvectors as ˜ A , and so does ˜ L . Hence, it is also structure-aware:

The last inequality follows from the fact λ max ( P ) ≤ 1 .

Lemma 2. Given a normalized adjacency matrix ˜ A ∈ R n × n , let the symmetric matrix deduced from the iterative approximation be

We have λ min ( P ′ ) > 0 , ∀ β < 1 / 2 .

Proof. This conclusion is trivial for the case of L ≤ 1 . Let us assume that L ≥ 2 . Firstly, rewrite Eq. (17) as

where P := (1 -β ) ∑ L -1 l =0 β l ˜ A l . P is positive definite in view of Lemma 1 and β L ˜ A L is positive semidefinite when L is even. Therefore, only the case of L ≥ 3 needs to be proved. For a vector x , we have

The first two inequalities follow from Lemma 1. The last inequality holds by noting the fact that, for L ≥ 3 and β < 1 / 2 ,

Therefore,

Proposition 3. The iterative approximation is direction-aware for all possible normalized adjacency matrices and L ≥ 0 , if and only if β < 1 / 2 .

Proof. If β < 1 / 2 , we have λ min ( P ′ ) > 0 according to Lemma 2, and thus ˆ ψ is direction-aware. Conversely, if β ≥ 1 / 2 , we can construct an adjacency matrix ˜ A such that λ min ( P ′ ) ≤ 0 for some L . Let us assume that L = 1 and

Remark 2. The construction of ˜ A in Eq. (22) is not unique. In fact, any bipartite graph can be used as a counterexample.

In this case, we have whose eigenvalues are 1 and 1 -2 β . The latter is non-positive for any β ≥ 1 / 2 .

Corollary 1 (The proof of Theorem 1) . The iterative approximation is direction-aware for all possible normalized adjacency matrices and L ≥ 0 , if and only if β < 1 / 2 . In contrast, the Neumann series approximation

is structure-aware and direction-aware for any β ∈ [0 , 1) .

Proof. This is a corollary of Proposition 2 and Proposition 3.

Proposition 4. The rescaled Neumann series approximation ˆ ψ is structure-aware and directionaware, and converges to the optimal solution as L increases.

Proof. The convergence to the optimal solution is obvious by noting that

Similar to Proposition 2, it can be proved that 1 1 -β L +1 P is positive definite with a largest eigenvalue ≤ 1 . Therefore, the rescaled transformation is also structure-aware and direction-aware.

A.2 Proof of Theorem 2

Before delving into the proof of the convergence, we would like to claim that the lemmas below are well known and can be found in most textbooks [32, 1] on convex optimization. For the sake of completeness, we provide here these proofs. Hereinafter, we use ∥ X ∥ 2 to denote the spectral norm which returns the largest singular value of the matrix X .

Lemma 3. For a twice continuously differentiable function f : R n → R with ∥∇ 2 f ( x ) ∥ 2 ≤ C, ∀ x ∈ dom ( f ) , we have

Proof. For a Taylor expansion of f ( x ) , there exists a z = τ x +(1 -τ ) y for some τ ∈ [0 , 1] such that

Lemma4. For a positive definite matrix P , let ∥ x ∥ P := ( x T Px ) 1 / 2 be the quadratic norm induced from P . If the eigenvalues of P fall into [ a, b ] , then

Proof. Firstly,

Secondly,

Theorem 4. Let f : R n × R m → R be a twice continuously differentiable function bounded below, and its Hessian matrix satisfies

for some constant C . The following gradient descent scheme is used to train x , y :

where P is deduced from the L -layer Neumann series approximation. Then, after T updates, we have,

where ϵ = f ( x , y ) -f ( x ∗ , y ∗ ) and

Proof. The update formula (29) can be unified into

.

It is easy to show that

Specifically,

In view of Lemma 3 and Lemma 4, we have

Denoted by we have

Theorem 5 (The proof of Theorem 2) . If η = η ′ = 1 /C , after T updates, we have

If we adopt a modified learning rate for embedding:

the convergence rate could be improved to O ( C/ ((1 -β ) T ) ) .

Proof. This is true for L = 0 , since in this case the update mechanism becomes a normal gradient descent regardless of η = 1 /C or η = 1 (1 -β ) C . Let us prove a general case for L ≥ 1 next.

According to Theorem 4, we have

where ϵ = L ( E , θ ) -L ( E ∗ , θ ∗ ) and

Notice that, for η = η ′ = 1 /C ,

and for η = 1 (1 -β L +1 ) C , η ′ = 1 C ,

Therefore,

The remainder of the proof is straightforward.

A.3 Proofs of Proposition 1 and Theorem 3

Adam(W) [23] uses the bias-corrected moment estimates for updating because they are unbiased when the actual moments are stationary throughout the training. Below, Lemma 5 formally elaborates on this, and Theorem 6 extends Theorem 3 with a proof of unbiasedness.

Lemma5 ([23]) . Denoted by ˆ m t = m t / (1 -β t 1 ) and ˆ v t = v t / (1 -β t 2 ) the bias-corrected estimates, if the first and second moments are stationary, i.e.,

then these bias-corrected estimates are unbiased:

Proposition 5. If a node is no longer sampled in subsequent p batches after step t , we have

where the coefficient of κ is mainly determined by t .

Proof. In this case, the iterative formula becomes

.

Therefore,

It is easy to show that

Theorem 6 (The proof of Theorem 3) . Under the same assumptions as in Lemma 5 and Proposition 1, the bias-corrected estimates are unbiased and ∆ e t + p -1 = ∆ e t -1 if the estimates are updated in the following manner when g t = 0 ,

Proof. We first show the unbiasedness of m t and the proof for v t is completely analogous. It remains only to show that

If the node is no longer sampled in subsequent p batches after step t , we have

This is trivial for t = 0 . Assuming that this also holds in the case of t -1 , it can be proved by induction.

If g t = 0 ,

̸

If g t = 0 ,

Analogously, we have,

Therefore, the conclusion can be deduced from

A.4 Connection between LightGCN and SEvo-enhanced MF-BPR

For a L -layer LightGCN, it can be formulated as follows

where α l , l = 0 , . . . , L represent the layer weights. According to the linear nature of the gradient operator, it can be obtained that

Hence, denoted by ζ ( · ) the gradient processing procedure of an optimizer, we can establish that LightGCN is identical to the following system:

When ζ ( · ) is an identity mapping (i.e., standard gradient descent), LightGCN is equivalent to MFBPR with SEvo being applied twice at each update. However, when ζ ( · ) is not an identity mapping (e.g., an optimizer with momentum or weight decay is integrated), they cannot be unified into a single system. Compared to explicit GNNs, SEvo is easy-to-use and has minimal impact on the forward pass, making it more suitable for assisting recommenders in simultaneously utilizing multiple types of information. These connections in part justify why SEvo can inject structural information directly.

B Detailed Settings

B.1 Algorithms

We present the algorithms of SEvo-enhanced Adam and SGD in 2 and Algorithm 3, respectively.

B.2 Datasets

In this study, we perform experiments on six public datasets. Specifically, the Beauty, Toys, and Tools datasets are extracted from Amazon reviews published in 2014 2 , while Electronics and Clothing are sourced from Amazon reviews published in 2018 3 . Additionally, the MovieLens-1M dataset is made available by GroupLens 4 .

Algorithm 2: Adam enhanced by SEvo. Differences from the original Adam are colored in blue. The matrix operation below are element-wise.

Input: embedding matrix E , learning rate η , momentum factors β , β , β [0 , 1) , weight

1 2 ∈ decay λ .

foreach step t do

G t ←∇ E L + λ E t -1 Update first/second moment estimates:

Compute bias-corrected first/second moment estimates:

Output: optimized embeddings E .

Algorithm 3: SGD with momentum enhanced by SEvo. Differences from the original SGD are colored in blue. The matrix operation below are element-wise.

Output: optimized embeddings E .

B.3 Baselines

Four GNN-based baselines for performance and efficiency benchmarks:

LightGCN [16] is a pioneering collaborative filtering model that simplifies graph convolutional networks (GCNs) by removing nonlinearities for easier training. It uses only graph structural information and has no access to sequential information.

SR-GNN [48] and LESSR [8] are two baselines dynamically constructing session graph. The former employs a gated graph neural network to obtain the final node vectors, while the latter utilizes edge-order preserving multigraph and a shortcut graph to address the lossy session encoding and ineffective long-range dependency capturing problems, respectively.

MAERec [51] learns to sample less noisy paths from a semantic similarity graph for subsequent reconstruction tasks. However, we found that the official implementation treats the recommendation loss and reconstruction loss equally, leading to poor performance here. Therefore, an additional weight is attached to the reconstruction loss and a grid search is performed in the range of [0, 1]. Almost all hyperparameters are tuned for competitive performance.

Four sequence backbones to validate the effectiveness of SEvo:

Update via SEvo:

;

✴✴

●❡/a116

❣/a114❛❞✐❡♥/a116/a115

GRU4Rec [17] applies RNN [9] to recommendation with specific modifications made to cope with data sparsity. In addition to the learning rate in {1e-4, 5e-4, 1e-3, 5e-3} and weight decay in [0, 0.1], we also tune the dropout rate for node features in the range of [0, 0.7].

SASRec [22] and BERT4Rec [40] are two pioneering works on sequential recommendation equipped with unidirectional and bidirectional self-attention, respectively. For BERT4Rec which employs a separate fully-connected layer for scoring, the weight matrix therein will also be smoothed by SEvo. In addition to some basic hyperparameters, the mask ratio is also researched for BERT4Rec.

STOSA [13] is one of the state-of-the-art models. It aims to capture the uncertainty of sequential behaviors by modeling each item as a Gaussian distribution. The hyperparameters involved are tuned similarly to SASRec.

Four knowledge distillation methods used in Appendix C.3:

KD [18] and DKD [55] are two logits-based approaches to transfer knowledge. DKD decomposes the classical KD loss into target class knowledge distillation loss and non-target class knowledge distillation loss.

RKD [34] and HTD [21] are two ranking-based approaches. The former focuses the distillation of relational knowledge through distance-wise and angle-wise alignments, while the latter emphasizes the distillation of hierarchical topology by dividing nodes into multiple groups and requiring intra-group and inter-group alignments.

C Applications of SEvo beyond Interaction Data

Here, we preliminarily explore the exploitation of more types of knowledge besides consecutive occurrences. We first investigate some elementary factors for interaction data, and then introduce the applications of SEvo to node categories and knowledge distillation.

C.1 Pairwise Similarity Estimation Factors

(a) Default

(b) Last

→

First

(c) Walk length

H

=1

→

H

=2

(d) Frequency

→

Distance

v

1

v

3

v

2

0

0

0

0

0

2

0

2

0

0

0

0

0

0

2

0

2

0

0

1

1

1

0

0

1

0

0

0

1

1

1

0

0

1

0

0

+

…

+

…

=

A

Last

K

=3,

H

=1, Frequency

v

1

v

2

v

3

v

2

v

1

v

3

v

2

0

0

0

0

0

2

0

2

0

0

1

1

1

0

0

1

0

0

+

…

+

…

=

A

Last

K

=3,

H

=1, Frequency

v

1

v

2

v

3

v

2

v

1

v

3

v

2

0

0

0

0

0

2

0

2

0

0

1

1

1

0

0

1

0

0

+

…

+

…

=

A

Last

K

=3,

H

=1, Frequency

v

1

v

2

v

3

v

2

0

1

0

1

0

1

0

1

0

0

1

0

1

0

1

0

1

0

0

1

1

1

0

0

1

0

0

0

1

1

1

0

0

1

0

0

+

…

+

…

=

A

v

2

First

K

=3,

H

=1, Frequency

v

1

v

2

v

3

v

1

v

3

v

2

0

1

0

1

0

1

0

1

0

0

1

1

1

0

0

1

0

0

+

…

+

…

=

A

v

2

First

K

=3,

H

=1, Frequency

v

1

v

2

v

3

v

1

v

3

v

2

0

1

0

1

0

1

0

1

0

0

1

1

1

0

0

1

0

0

+

…

+

…

=

A

v

2

First

K

=3,

H

=1, Frequency

v

1

v

2

v

3

v

1

v

3

v

2

v

1

v

3

v

2

0

0

0

0

1

2

0

2

0

0

0

0

0

1

2

0

2

0

0

1

1

1

0

1

1

1

0

0

1

1

1

0

1

1

1

0

+

…

+

…

=

A

Last

K

=3,

H

=2, Frequency

v

1

v

2

v

3

v

2

v

1

v

3

v

2

0

0

0

0

1

2

0

2

0

0

1

1

1

0

1

1

1

0

+

…

+

…

=

A

Last

K

=3,

H

=2, Frequency

v

1

v

2

v

3

v

2

v

1

v

3

v

2

0

0

0

0

1

2

0

2

0

0

1

1

1

0

1

1

1

0

+

…

+

…

=

A

Last

K

=3,

H

=2, Frequency

v

1

v

2

v

3

v

2

v

1

v

3

v

2

0

0

0

0

.5

2

0

2

0

0

0

0

0

.5

2

0

2

0

0

1

1

1

0

.5

1

.5

0

0

1

1

1

0

.5

1

.5

0

+

…

+

…

=

A

Last

K

=3,

H

=2, Distance

v

1

v

2

v

3

v

2

v

1

v

3

v

2

0

0

0

0

.5

2

0

2

0

0

1

1

1

0

.5

1

.5

0

+

…

+

…

=

A

Last

K

=3,

H

=2, Distance

v

1

v

2

v

3

v

2

v

1

v

3

v

2

0

0

0

0

.5

2

0

2

0

0

1

1

1

0

.5

1

.5

0

+

…

+

…

=

A

Last

K

=3,

H

=2, Distance

v

1

v

2

v

3

v

2

Recent GNN-based sequence models [48, 49], as well as the SEvo-enhanced models reported in Section 3, estimate the pairwise similarity w ij between items v i and v j based on their co-occurrence frequency in sequences. In other words, items that appear consecutively more frequently are assumed more related. Yet there are some factors that deserve a closer look: (1) The maximum sequence length K for construction to investigate the number of interactions required for accurate estimation; (2) Using only the first K versus last K interactions in each sequence to compare the utility of early and recent preferences; (3) Allowing related items to be connected by a walk of length ≤ H rather than strict consecutive occurrences; (4) Frequency-based similarity versus distance-based similarity. The former weights all related pairs equally, while the latter weights inversely to their walk length. For example, given a sequence v 2 → v 1 → v 3 with a maximum walk length of H = 2 ,

5

50

100 150

300

K

0

.

0325

0

.

0350

0

.

0375

0

.

0400

NDCG@10

Beauty

Baseline

First

K

Last

K

5

100

200

300

500

K

0

.

132

0

.

134

0

.

136

0

.

138

MovieLens-1M

Baseline

First

K

Last

K

1

2

3

4

5

H

0

.

0325

0

.

0350

0

.

0375

0

.

0400

NDCG@10

Beauty

Baseline

Frequency

Distance

1

2

3

4

5

H

0

.

132

0

.

134

0

.

136

0

.

138

MovieLens-1M

Baseline

Frequency

Distance

(b) Frequency-based versus Distance-based

the frequency-based similarity of ( v 2 , v 3 ) gives 1, while the distance-based similarity is 1 / 2 (as the walk length from v 2 to v 3 is 2 ).

Figure 4 illustrates these four variants and Algorithm 4 details a step-by-step process. We further compare these four potential factors in Figure 5:

Figure 5a shows the effect of confining the maximum sequence length to the first/last K items, so only the early/recent preferences will be considered. In constrast to early interactions, recent ones imply more precise pairwise relations for future prediction, even for small K . With the increase of the maximum sequence length, the recommendation performance on Beauty improves steadily, but not the case for MovieLens-1M. This suggests that shopping relations may be more consistent than movie preferences.

Figure 5b explores the relations beyond strict consecutive occurrences; that is, two items are considered related once they co-occur within a path of length ≤ H . For the shopping and movie datasets, estimating similarity beyond co-occurrence frequency appears less reliable overall. We also compare frequency-based similarity with distance-based similarity that decreases weights for more distant pairs. It is clear that the distance-based approach performs more stably as the maximum walk length H increases.

C.2 SEvo for Intra-class Representation Proximity

Sometimes embeddings are expected to be smooth w.r.t. a prior knowledge. For example, in addition to the interaction data, each movie in the MovieLens-1M dataset is associated with at least one genre.

0.5

=

=0.99

Epoch: 0

Epoch: 10

Epoch: 100

It is natural to assume that movies of the same genre are related to each other. Heuristically, we can define the similarity w ij to be 1 if v i and v j belong to the same genre and 0 otherwise.

As can be seen in Figure 6, such smoothness constraint can also be fulfilled through SEvo, leading to progressively stronger clustering effects as β increases. However, the resulting performance gains are slightly less than those based on interaction data (see Table 4). One possible reason is that the movie genres are too coarse to provide particularly useful information. In conclusion, while smoothness is an appealing inductive bias, its utility depends on how well the imposed structural information agrees with the performance metrics of interest.

C.3 SEvo for Knowledge Distillation

In addition to the affinity matrix extracted from interaction data or relation data, the pairwise similarity can also be estimated from a heavy-weight teacher model. Recall that Knowledge Distillation (KD) [18] encourages a light-weight student model to mimic the behaviors ( e.g. , output distribution) of the teacher model, so the learned student model achieves both accuracy and efficiency. In general, higher-dimensional embeddings are capable of better fitting the underlying distribution between entities. The pairwise similarities extracted from a teacher model, needless to say, can be used to guide the embedding evolution of a student model. Unlike interaction or relation data, the deduced graph is dense if only the distance function is applied. Therefore, some graph construction steps including sparsification and reweighting should be involved as well. We attempt to use the widely used KNN graph here, and leave a more comprehensive study of graph construction [20] as a future work.

Specifically, the distance between each pair v i , v j is estimated using a cosine similarity distance function:

Then, K -nearest neighbors are selected for each node; that is

̸

̸

This sparsification is neccessary for several reasons: 1) SEvo with a dense adjacency matrix is computationally prohibitive to conduct; 2) Generally, only the top-ranked neighbors are reliable for next distillation. Finally, the adjacency matrix is obtained through reweighting and symmetrizing:

where τ > 0 is the kernel bandwidth parameter.

Table 5 reports the results of the SASRec backbone with different embedding sizes (200 versus 20). Although a student equipped with SEvo can only derive guidance from the teacher in terms of embedding modeling, it has surpassed RKD and HTD that focus on feature/output alignments. Recall that SEvo only needs to access the teacher model once for adjacency matrix construction, whereas other knowledge distillation approaches require accessing the teacher model for each update. SEvo is arguably an efficient tool for transferring embedding knowledge. Nevertheless, SEvo alone cannot be expected to facilitate the learning of the other modules, which consequently is still inferior to state-of-the-art methods such as DKD. Fortunately, SEvo and DKD can work together to further boost the recommendation performance.

D Additional Experimental Results

D.1 SEvo for GNN-based models

In Section 3 we have comprehensively validated the effectiveness of SEvo for classic sequence models. It is also of interest to explore the impact on GNN-based models that have learned certain

structural information. Table 6 reports the results on LESSR and MAERec: SEvo not only facilitates the learning of LESSR, but also helps MAERec that already utilizes global graph information. This implies that previous efforts fail to fully exploit structural information, while SEvo demonstrates superior performance in this regard.

D.2 J smoothness as a Regularization Term

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

0

.

020

0

.

025

0

.

030

0

.

035

0

.

040

NDCG@10

Beauty

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

0

.

10

0

.

11

0

.

12

0

.

13

0

.

14

MovieLens-1M

SASRec

+SEvo

λ

λ

Structural information may be injected by imposing J smoothness as a regularization term; that is,

where λ ≥ 0 is a hyperparameter governing the degree of smoothness. We conduct this ablation study in Figure 7 with a λ from 10 -6 to 0 . 01 . As can be seen, incorporating a smoothness regularization term could slightly improve the recommendation performance, but it is not optimal. SEvo performs better because the gradient of the regularization term may be in conflict with the primary loss function.

D.3 L -layer Approximation

As L increases, SEvo gets closer to the exact solution while accessing higher-order neighborhood information. Table 7 lists the performance of different layers, which reaches its peak around L = 3 and starts to decrease then. A possible reason is that the higher-order information is over-smoothed and thus not as reliable and easy to use as the lower order information. Similar phenomena have been found in previous works [50, 8] on applying GNNs to recommendation.

D.4 Training and Inference Times

The time complexity of SEvo is mainly determined by the arithmetic operations of ˜ A l ∆ E , l = 1 , 2 , . . . , L . Assuming that the number of non-zero entries of ˜ A is S , the complexity required is about O ( LSd ) . Because the recommendation datasets are known for high sparsity ( i.e. , S is very small), the actual computational overhead can be reduced to a very low level, almost negligible. Table 8 provides the actual training and inference times.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's contributions are summarized in the introduction point by point.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might

not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions and proofs are detailed in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to the settings introduced in Section 3.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All experiments are conducted on public datasets and the code can be found in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴ ♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿ ✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The implementation details of baselines and ours can be found in Appendix B.3 and Section 3, respectively.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: 1) Paired t-test is performed over 5 independent runs (see Table 2). 2) 1-sigma error bars are marked in most figures and tables.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to Appendix D.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impact has been discussed in Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The source of datasets are credited in Appendix B.2, and all baselines have been introduced in Appendix B.3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
In this paper, the authors propose a novel optimization algorithm that is talored for recommender systems. It incorporates graph structural information into the optimization process, aleviating the burden of performing GNN for RS. The convergence of the algorithm is theoretically demonstrated. Besides, it could be incorporated into existing well-performed optimizers like AdamW. The experiments are conducted to test its effectiveness on different types of recommendation models and consistent performance improvements are observed.

STRENGTHS:
1. This paper is innovative in its consideration of utilizing graph information the algorithm level, not model level, for recommender systems. The proposed new algorithm is theoretically guaranteed to converge.
2. The application of the proposed algorithm is widely discussed, including its incorporation into existing popular optimizers and its combination with knowledge distillation for recommender systems.
3. The experiments are extensively conducted, including different size of datasets, different types of recommendation backbones and baselines. The results demonstrate both the effectiveness and efficiency of the proposed algorithm for improving existing recommendation models.

WEAKNESSES:
1. The technique in Section 2.4 is a little too specific for AdamW.
2. Table 5 seems to have an error. In the last row, 0.568 is not the best result. Is it just a typo? Or does +DKD not further improve the performance in this case?

QUESTIONS:
In Table 6, the proposed optimization algorithm even improves the GNN-based recommendation models. What is the reason for this phenomenon?

LIMITATIONS:
The authors have addressed the limitations in Section 5 and Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Structure-aware Embedding Evolution (SEvo) to improve recommender systems by directly integrating graph structural information into embeddings. Unlike traditional methods, authors propose guide embedding update momentum with graph smoothing regularization. The proposed method can be integrated with a wide range of optimizers for neural networks, e.g., AdamW. The proposed method significantly increases performance metrics on several recommender datasets for standard models and outperforms GNNs.\n\nSTRENGTHS:\n* Novel intriguing view on the problem of preserving structural information for recommender systems\n* Plug-n-play design of the method so that it can be easily transferred to any recommendation architecture\n* Solid performance gains\n* Faster than training GNNs\n\nWEAKNESSES:\n* The paper is difficult to follow. It would be nice to see a simple summary of the SEvo pipeline in the form of a scheme or algorithm. \n* The SEvo formulae use the normalized adjacency matrix, so propagation of the gradients over the sampled node neighborhood is required. Suppose we have a large graph with a relatively high degree of each node. Mini-batch may have poorly correlated nodes, so many node embeddings should be updated simultaneously. This can lead to memory consumption issues and a notable increase in training time.\n* The tables report only the average time across different datasets. The detailed computational (or non-aggregated graphs with time)  and space complexity is required to understand the ability of the method to scale\n* SEvo accelerates momentum only over first-order neighbors. However, for some graph-related tasks, it is critical to handle long-range dependencies.\n\nQUESTIONS:\n* How does SEvo work in mini-batch fashion? Does it require specific batch preparations, or should it be smaller on average?\n* How does the model training time scale with the size of the graph? Could you provide a graph epoch time vs. graph size e.g. for a SASRec?\n* Can high-order proximity / long-range dependencies be incorporated using SEvo?\n\nLIMITATIONS:\n* The time and space complexity analyses of the method are required to understand its scalability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper introduces Structure-aware Embedding Evolution (SEvo), a novel embedding update mechanism for recommender systems. SEvo directly integrates graph structural information into embeddings, ensuring that related nodes evolve similarly with minimal computational overhead. This approach differs from traditional Graph Neural Networks (GNNs), which typically serve as intermediate modules. SEvo is designed to enhance existing optimizers, particularly AdamW, to improve recommendation performance by incorporating moment estimate corrections. Theoretical analysis confirms the convergence properties of SEvo, and experiments demonstrate consistent improvements across various models and datasets.

STRENGTHS:
1. The paper proposes a new method to enhance over smoothing during the backward pass.

2. The method can be naturally integrated with momentum-based optimizers

WEAKNESSES:
1. Uncleared relationship related to recommender system

2. Key points need further explanation.

QUESTIONS:
1. I do not see the relationship between your method and recommendation task. Your method aims to change gradient direction based on graph topology. It is more suitable to study it under more general graph datasets. Recommendation task has no relationship with your method.

2. In line 116, the author mentioned “These two criteria inherently conflict to some extent”. Why are structure-aware and direction-aware inherently in conflict? Do you have any explanation on this point?

3. We have already enhanced smoothness during the forward-pass during neighborhood aggregation. Why do authors think it is necessary to further enhance it during the backward pass? Will it lead to more severe over smoothing problem?

LIMITATIONS:
See weakness and questions
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SEvo, an embedding updating mechanism that directly injects the graph information into the optimization process. This paper points out two critical criteria for directly injecting graph structure information into the embedding updating process for recommendation. Based on the proposed two criteria, this paper makes efforts to derive a solution named SEvo for injecting the graph structure information directly. SEvo is model-agnostic and can be implemented in various optimizers. The experiments are detailed, and the algorithm is theoretically guaranteed.\n\nSTRENGTHS:\n1. This paper is well-motivated and well-organized, making this paper easy to understand. This paper first proposes two criteria and derives the final form of SEvo. I appreciate the efforts of the authors to make this process so clear.\n2. The experiments are detailed. Experiments on various datasets showcase the effectiveness of SEvo, with detailed ablation studies.\n3. The proposed method is easy to implement. SEvo can be integrated into various optimizers without complex modification.\n\nWEAKNESSES:\nThe reason behind the success of SEvo on large-scale datasets remains unclear. I am extremely curious about this. The improvement is unbelievably huge, making me doubt the reported results. A level of 5% in practice industrial application is extremely huge. However, the experiment results show that SASRec equipped with SEvo performs twice as well as the vanilla SASRec. I believe that if the reported results are true, this performance even exceeds the SOTA method by a large margin since SASRec is still a strong baseline in practice. Has the author carefully tuned the base model?\n\nQUESTIONS:\nI am curious about the differences in gradient descent between SEvo and methods that explicitly model the neighborhood relationship with graph structure (e.g., LightGCN). The modified embedding updating mechanism in SEvo also includes components of the adjacent matrix, which looks similar to the graph propagation mechanism in GNN-based methods. Could you please give an example showcasing the differences in gradient descent (e.g., comparing the gradient descent processes of LightGCN and MF-BPR+SEvo)?\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}