Recommender Transformers with Behavior Pathways
Authors: Zhiyu Yao
Xinyang Chen
Sinan Wang
Qinyan Dai
Yumeng Li
Tanchao Zhu
Mingsheng Long
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Recommendation
Deep Learning
Transformers

Abstract: Sequential recommendation requires the recommender to capture the evolving behavior characteristics from logged user behavior data for accurate recommendations. However, user behavior sequences are viewed as a script with multiple ongoing threads intertwined. We find that only a small set of pivotal behaviors can be evolved into the user's future action. As a result, the future behavior of the user is hard to predict. We conclude this characteristic for sequential behaviors of each user as the \textit{Behavior Pathway}. Different users have their unique behavior pathways. Among existing sequential models, transformers have shown great capacity in capturing global-dependent characteristics. However, these models mainly provide a dense distribution over all previous behaviors using the self-attention mechanism, making the final predictions overwhelmed by the trivial behaviors not adjusted to each user. In this paper, we build the \textit{Recommender Transformer} (RETR) with a novel \textit{Pathway Attention} mechanism. RETR can dynamically plan the behavior pathway specified for each user, and sparingly activate the network through this behavior pathway to effectively capture evolving patterns useful for recommendation. The key design is a learned binary route to prevent the behavior pathway from being overwhelmed by trivial behaviors. We empirically verify the effectiveness of RETR on seven real-world datasets and RETR yields state-of-the-art performance.


The human being has many different parts of the brain specialized for various tasks, yet the brain only 34 calls upon the relevant pieces for given situation [40]. To some extent, user behavior sequences can Recommender Transformers with Behavior Pathways
Anonymous Author(s) Affiliation Address
email
Abstract
2 Sequential recommendation requires the recommender capture the evolving behavior characteristics from logged user behavior data for accurate recommen- dations. However, user behavior sequences are viewed ongoing threads intertwined. We find that only a small set of pivotal behaviors can be evolved into the user's future action. As result, the future behavior of the user is hard to predict. We conclude this characteristic for sequential behaviors of each user as the Behavior Pathway. Different users have their unique behavior pathways. Among existing sequential models, transformers have shown great capacity in capturing global-dependent characteristics. However, these models mainly provide dense distribution over all previous behaviors using the self-attention mechanism, making the final predictions overwhelmed by the trivial behaviors not adjusted each user. In this paper, we build the Recommender Transformer (RETR) with a novel Pathway Attention mechanism. RETR can dynamically plan the behavior pathway specified for each user, and sparingly activate the network through this behavior pathway to effectively capture evolving patterns useful for recommenda- tion. The key design is a learned binary route to prevent the behavior pathway from being overwhelmed by trivial behaviors. We empirically verify the effectiveness RETR seven real-world datasets and RETR yields state-of-the-a performance.
script with multiple
11
13
15
17
Introduction
19
have been widely adopted in real-world industrial applications 21 such as E-commerce and social media. Benefiting from the increase computing power and model
20 Recommender systems
16
24 42
22 capacity, some recent efforts formulate recommendation as time-series forecasting problem, known The core idea of this field to infer upcoming actions 24 based on user's historical behaviors, which are reorganized time-ordered sequences. This intuitive modeling of recommendation is proved time-tensitive and context-aware make precise predictions.
23 as sequential recommendation [18
25
28
Recent advanced sequential recommendation models, such as SASRec [8], Bert4Rec [31] and SMRec (6) have achieved significant improvements. Transformers enable these models to recognize global-range sequential patterns, and to model how future behaviors are anchored in historical ones.
27
28
29 The self-attention mechanism does make possible to explore previous behaviors of each user,
30 with the whole neural network activated. However, misuse of user information, regardless of whether
31 they are informative or not, floods models with trivial ones, makes models dense and inefficient, and results in key behaviors losing voice. And this clearly contradicts with the way our brain works.
35 be viewed as a script with multiple ongoing threads intertwined. And only key clues suggest what will
38 happen next. In sequential recommendation, we find that only a small part of pivotal behaviors can
37 be evolved into the user's future action. And conclude this characteristics of sequential behaviors
38 as the Behavior Pathway.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Correlated Behavior Pathway
Predict Item
Casual Behavior Pathway
Drifted Behavior Pathway
Different Users
User Behavior Trace Axis
Figure 1: Three main characteristics of the behavior pathway for different users, making sequential
recommendation extremely hard. The behavior pathway outlined by the red boxes.
39
Different users have their unique behavior pathways and they can be grouped into three categories:
40
Correlated Behavior Pathway: A user's behavior pathway is closely associated with behaviors at certain period. As shown in the first line of Figure I the mouse is clicked many times recently, leading to the final decision to buy mouse.
41
42
43
Casual Behavior Pathway: A user's behavior pathway is interested in a specific item at casual times. As shown in the second line Figure I the backpack randomly clicked sequentially.
44
45
48
Drifted Behavior Pathway: A user's behavior pathway in a particular brand might drift over time. As shown in the third line of Figure I the user was initially interested keyboard, but suddenly became interested in buying phone at last.
47
48
49
It's challenging to capture these aspects dynamically for each user to make precise recommendations.
50
Motivated by the Pathways [8]. a new way of thinking about AI, which builds a single model that is sparsely activated for all tasks with small pathways through the network called into action needed, we propose a novel Recommender Transformer (RETR) with a Pathway Attention mechanism. RETR and then captures evolving patterns through these pathways effectively. To be specific, the user-dependent pathway attention, which incorporates a pathway router, determines whether not a behavior token will be maintained the behavior pathway. Technically, the pathway router generates a customized binary route for each token based on their information redundancy. Recommender Transformers are stacked, and successive pathway routers constitute hierarchical evolution pathway of user behaviors. To enable the pathway router modules be end-to-end optimized, we adopt the Guffel-Softmax [17] sampling strategy to overcome the non-differentiable problem of sampling from a Bernoulli distribution.
51
52
dynamically explores behavior pathways for different
53
54
55
56
57
58
59
61
To effectively capture the evolving patterns via the behavior pathway, our pathway attention mecha- nism makes our RETR mainly attend to the obtained pathway. We force the model to focus on the most informative behaviors by using the query routed through the behavior pathway. We cut off the interaction from the off- pathway behaviors of the query. Compared with using all previous behaviors, our pathway attention mechanism is obviously more effective and can avoid the most informative tokens being overwhelmed by trivial behaviors. To validate the effectiveness of our approach, we conduct experiments on seven real-world competitive datasets for sequential recommendations and RETR achieves state-of of -fte----thhhtrtn--e the- performance. Our main contributions can be summarized as follows:
52
63
84
85
66
67
68
89
Our work is the first to propose the concept of behavior pathway for sequential recommen- dation. We find the key to the recommender is to dynamically capture the behavior pathway for each user.
71
72
We propose the novel recommender transformer (RETR) with a novel pathway attention mechanism, which can generate the behavior pathway hierarchically and capture the evolving patterns dynamically through the pathway.
73
74
75
We validate the effectiveness RETR on seven real-world datasets of different scales across different scenarios for sequential recommendations and achieve of-te-o-thhear performance.

---

77 2
Related Work
78
Traditional recommendation approaches. Capturing evolving behavior characteristics is crucial
79 for many online applications, such as advertising, social media and E-commerce, and it is the key
80 challenge for sequential recommendation [ 39 26
5 45 23). Traditional recommendation
81 approach, such as the collaborative filtering (CF) [15] based on matrix approximation [20 21]. always
assumes that the user's behavior is static. However, in practice, user behaviors often change over time due to various reasons, making the CF deteriorate in a real-world application.
83
84
Sequential recommendation approaches. To overcome this challenge, some methods, such as FPMC [14 and HRM [34, use Markov chains to capture sequential patterns by learning user- specific transition matrices. Higher-order Markov Chains assume the next action is related to several previous actions. Benefit from this strong inductive bias, MC-based methods [14] [3] show superior performance in capturing short-term patterns. At the same time, there is potential state space explosion problem when these approaches are faced with different possible sequences (55 recent years, many works have been using the deep neural network for sequential recommendation. The GRU4Ree [16] and the RepeatNet [27] adopt the recurrent network to capture dynamic patterns from the user behaviors dependent on sequence positions. The RNN-based models achieve competitive performance in capturing short-term behavior patterns but cannot capture long-term behavior patterns effectively. The CNN-based model, such as Caser [3], applies convolutional operations extract transitions while tending to overlook the intrinsic relationship across user behaviors. The GNN- based methods, such as SRGNN (37]. GCSAN [38. Jodie [22] and TGN 28 model behavior sequences as graph-structured data and incorporate an attention mechanism for a session-based recommendation. In addition, DIN 4X uses the gate mechanism to weight different user behaviors. However, concatenating all behaviors makes these models overlook the sequential characteristics.
85
86
87
88
89
91
99
95
97
98
89
intention-based models for Sequential Recommendation. The attention- praention-bbsed models like SINE 32 have the strong capacity to capture behavior patterns via the attention mechanism. achieving sfate-of-the-art performance while involving many parameters. Especially, SASRec [18) BertRec
101
103
BI, S3-Rec [44], TGSRee [0). LightSANs 9 and SSE-PT 36) introduce the transformer archi- tecture into sequential recommendation, which might lead to the over over-ooranann pree-parameterized architecture of Transformer- based methods. These models capture the evolving patterns by the self-attention mechanism, interacting with all previous behaviors. However, dense interactions will make the model not adapt to different users and overwhelm behavior pathways. To tackle this challenge, our paper builds the Recommender Transformer (RETR) with a new Pathway Attention mechanism that is dynamically activated for the behavior pathway of all users. Distinct from the previous routing architecture like Switch Transformer [12] using the MoE (30] structure for natural language tasks, our RETR is designed explicitly for sequential recommendation. Our RETR uses the pathway router to adaptively route the sequential behavior of each user rather than routing the experts of feed-forward networks in switch transformer.
105
106
107
108
109
110
111
112
113
3 Method
Suppose that have set of users and items, denoted by U and I respectively. In the task of sequential recommendation, chronologically-ordeee behaviors of a user u E U could be represented by user-interacted item sequence: (i,,... ,..]. Formally, given a user u with her his behavior sequence (i1, ) the goal of sequential recommendation is to predict the next item the user u would interact with at the (n + 1)-th step, denoted (fx+1 (1m)
116
117
118
119
120 As aforementioned, we highlight the key to sequential recommendation as the exploration of user- tailored behavior pathways. through which evolving characteristics could be learned. Motivated by this, we propose a novel Recommender Transformer (RETR) with a new Pathway Attention, the core subassembly of which is pathway router. Besides the modification of architecture, additionally introduce& a hierarchical update strategy for the behavior pathway in the feed-forward procedure.
122
128
124
125
3.1
Recommender Transformer
126
Considering the limitation of Transformers 4 for sequential recommendation, we renovate the vanilla architecture to the Recommender Transformer (Figure 2 with Pathway Attention mechanism.
3

---

Predict Behavior
Pathway Attention
Iearr
Pradiction
Pathway Router
Multi
LN
MatMul
A Gumbel-Softmaz Sampling
Matidul
LX
FFN
Iieerr
Thear
LN
MEP
K
Patmway Allention
Embedding
Behaviors
Figure Recommender Transformer architecture (right). Pathway Attention (left) explores the behavior pathway by the pathway router (orange module) and captures the evolving sequential characteristics by the multi-head attention.
Model inputs. To obtain the model inputs, we follow the sliding window practice and transform the user's behavior sequence into a fixed-length-N sequence & = (81, , & Then we produce an item embedding matrix &x E x]xd where is the embedding dimensionality. We perform look-up operation from & retrieve the input embedding matrix & RNxx for sequence s. Besides, we also add learnable position embedding P, E R for sequence s. Finally, we can generate the input embedding of each behavior sequence s X, E. + P. IR Nxd.
129
130
132
133
134
Overall architecture. Recommender Transformer is characterized by stacking the Pathway Attention blocks and mred-forward layers alternately, containing L blocks. This stacking structure conducive to learning behavior representations hierarchically. The overall equations block I are formalized as:
136
RR 22 = LN 2
(Z!-1,RR!
22111)
(1)
z' LN (FFN (2)) 21))
138
L}} denotes the output of the I-th block. The initial input 2 X' e N represents the raw behavior embedding. R!-1 is the previous route from the block / 1 and initialize all elements in the route RD 1. Path-MSA()) is to conduct the pathway attention. LN(.) to conduct layer normalization and FFN represents the point-wise feed-forward network 4
where Z' E BR {1,....
139
141
3.1.1
Pathway Attention
Note that the single- single-branch self-attention mechanism 4 in vanilla transformer cannot model the behavior pathway dynamically, resulting in key behaviors being overwhelmed by these non- non-pivotal ones. To solve this problem, propose the Pathway Attention mechanism, as shown in Figure 2 which can dynamically attend to the behavior pathway of pivotal behavior tokens.
144
146
Pathway router. The pathway attention employs sequence-adaptive pathway router to custom-tailor
behavior pathway routes for users. The router generates a binary route R' E {0, 1] to determine whether behavior token would be part of the behavior pathway not. Each router takes the pre-order route R'-1 and user behavior tokens z & of the block 1 as its inputs. All elements in the route are initialized by and are updated progressively in training.
148
150
152 Foremost, to suppress the potential disturbance the model caused by the local drifted interest (Figure I it is crucial to incorporate the global information in the route generation. We apply average pooling to the preserved behavior tokens routed by R!-1, and produce the global sequential representation via multilayer perceptrons (MLP) module. Then, we combine this global representation with the inputs and employ residual connection to maintain the original input information. Finally, feed them to another MLP layer to predict the probabilities of keeping
153
154
156

---

158
dropping the behavior tokens. The above procedure can be formulated as follows:
Σ
R3
2
Lmm 2'-1 + 2 + MLP
Ai=l
R2
(2)
ii=1
π Softmax (MLP(Ze.h)) E R2
159
where @ is the Hadamard product. For & (1.2. ,.). we let :. = =
04 ar) where the logit
180
O₂ denotes the probability that the f-th behavior token is kept for the behavior pathway.
Gumbel-Softmax sampling from for router. Our goal is generate the binary route from
162
ग. However, sampling from π directly is non-differentiable, and it will impede the gradient-hased training. Thus, we apply the Gumbel-Softmanl (17) technique such sample. Gumbel-Softma is an effective way to approximate the original non-differentiable sample from a discrete distribution with a differentiable sample from Gumbel-Softmax distribution. Instead of directly sampling a
163
184
165
keep-or-drop decision R2 for the I-th behavior token from the distribution Te we generate as:
186
R
arg max (log (T( G.(i)), jE(0.1)
where G, = - -oo(-- log U2) is standard Gumbel distribution, and U, is sampled ii.d. from uniform distribution Uniform(0, 1). To remove the non-differentiable argmax operation in Eq the
168
Humbel-Softmat uses the reparameterization trick [17] differentiable approximation to relax
one-hot Rd E 10 to U2 R2:
170
exp((log + G.G))/T)
vy ( = Σ Eigoo,1 exp((log (2) + G.(I)/T)) J E 10,
(4)
where 7 is the temperature parameter of the Softmax, which is commonly set to in deep networks.
Hierarchical update strategy for router. The preliminary route R², sampled from л. is not a final decision. In our design, once a token fails to be routed in a certain block, it would permanently lose the privilege to be part of the behavior pathway in the following feed-forward procedure, constituting a hierarchical pathway router strategy. Thus finally we formulate the route R' as the Hadamard
172
173
175
product of R' and the pre-order route R 1 in the block I 1-1- 1:
178
R'. R'= R R
(5)
177
Multi-head pathway attention. The standard self-attention mechanism retrieves sequential characteristics exploiting all behavior tokens, making the behavior pathway overwhelmed by the trivial behaviors. In the new pathway attention, the pathway router would be firstly applied to the input behavior tokens to route information. The pathway router would not pare down the number of tokens, but only the interactions between the off off-pathway and on-pathway tokens, as these off oaf-hpttway tokens may also convey contextual information.
178
179
181
182
183
Specifically, for the query, key, and value in the pathway attention: the query is routed by the pathway router, to prevent the pathway from being overwhelmed and to force the pathway attention to attend to the behavior pathway; the key and value are the original input behavior tokens, to ensure that the contextual information from off- aff-pathway behavior tokens can be captured as well:
184
185
186
2m, Kem; Vm (z!-1
R')W' em.
2''''Z W.... z'-'W/
@ Va/h
(6)
2
Softmax
Di
m3
187
where mn E (1,2,-.. /1 is the index of head in the multi-head self-attention; W 2... WL W
K,,,
V., E
pdxd/h are transformation matrices learned from data, Finally, the outputs (2), R. Nxd/h IS Sh
188
of multiple heads are concatenated into 2 E R Nxd, We use 21, R' = Path-MSA (2"-1, R ) to
189
summarize the above pathway attention. Its output is further transformed by Eq. to form the final output of the 1-th block z' E IR N
191
192
Causality. In the prediction of the (L 1)-th behavior, only the first / observable behaviors should be taken into account, To avoid a future information leak and ensure causality, a loea- mask employed and all links between Q, and K,, (j >>> i) are removed.
193
5

---

195
3.2
Prediction Layer and Training Objective
196
Prediction layer. In the final layer of our RETR, we calculate the user's preference score for the
item & in the step (L + 1) in the context of user behavior history p ( = k. (1:t) ex 2 where ej: is the representation
item & from item embedding matrix &T, and 2 is the output of the
196
T- RETR blocks at step & (L is the number of RETR blocks).
200
Training objective. We adopt the pairwise ranking loss to optimize the model parameters as:
C C= Σ dEN 1=1
log op d14) p(0 (1=)),
201
where we pair each ground-truth item 4441 with randomly sampled negative item "1+1- In each
202 epoch, we randomly generate one negative item for each time step in each sequence. This pairwise
ranking loss is widely adopted in previous literature of sequential recommendation [18 45
4 Experiments
204
We extensively evaluate the proposed Recommender Transformer on seven real-world benchmarks.
208
Datasets.
Here are descriptions of the seven
207
datasets: (1) Netflix: Netflix dataset is a large-
208
scale movie rating dataset released by Netflix.
Table 1: Statistics of the datasets.
209
(2) MSD: The Million Song Dataset (MSD) is
Dataset
Users
Items
Actions
210
a large-scale, metadata-rich and open-source dataset on Kaggle. (3) Taobao: Taobao dataset
Netflix
463,435
17,769
57,000,000
211
MSD
571,355
41,140
34,000,000
212
32
contains user behaviors in Taobao's 's recom-
Taobao
987, 19.994
4.162.024
100,150,807
mender system. In experiments, we only use the
Yelp
30,431
20,033
316,354
214
click behaviors. (4) Yelp (2]: Yelp is a dataset
MovicLens-IM
6,040
3,416
1,000,000
for business recommendation. We only use the transaction records after January 1st, 2019. (5)
Tmall
66,909
37,367
427,797
Steam
334,730
13,047
3,700,000
216
Tmall: Tmall contains users' shopping logs Tmall online shopping platform, which is from the IJCAI-15 competition. (6) Steam 18] Steam
218
dataset is collected from a large online video game distribution platform. This dataset includes
220
2,567,538 users, 15,474 games and 7,793,069 English reviews from October 2010 to January 2018. (7) MovieLens: this is widely used benchmark dataset for evaluating collaborative filtering algo- rithms. The version we use is MovieLens- IM, which includes million user ratings. The statistics of
222
223
the seven datasets summarized in Table D All datasets are widely used in the recommendation task. It is notable that Netflix, MSD, Taobao and Steam are large-scale recommendation datasets.
224
We group the interaction records by users or sessions for all datasets and sort them by the timestamps in ascending order. We follow the operation in SASRec [88 and split the historical sequence for each user into three parts: (1) the most recent behavior for testing, (2) the second most recent behavior for validation, and (3) all remaining behaviors for training. During testing, the input sequences contain training behaviors and validation behaviors. We filter less popular items and inactive users with fewer than five interaction records.
226
228
230
we apply top-k Hit Ratio (HR @k), top-k Normalized Discounted Cumulative Gain (NDCG @ k) and Mean Reciprocal Rank (MRR) for evaluation. We report HR @ 10, NDCG@ 10 and MRR of the results. Besides, following the standard strategy in SASRee [88, we pair the ground -ruth item with 100 randomly sampled negative items that the user has not interacted with. All metrics are calculated according to the ranking of the items and we report the average score.
Evaluation metrics. Following the previous literature 41
45
18
232
234
236
237
Baseline methods. We compare our RETR with GRU4Rec 16 a simple baseline that applies
238
GRU to model item sequences, and -fate-of-the-ar sequential recommendation models: SASRec BertRec [1]], SMRec [6] S3-Rec
239
44 SINE F2), TGSRec (10] and LightSANs ( These methods adopt the attention mechanism to make precise recommendations. Besides, we also compare
18
240
our RETR with state-of-the-ard graph-based sequential recommendation methods: Jodie 22 and
242
TGN 28
All baseline methods are configured using default parameters of the original paper or optimal parameters which can produce their best results through grid search.

---

244
Implementation details. Our model is supervised by the pairwise rank loss in Equ 7 using the ADAM [9] optimizer with an initial learning rate of 0.001. Batch size is set to 512. The maximum number of training epochs for all methods is set to 200. All hyperparameterss are tuned on the validation set. The training process early stopped within 10 epochs. Our RETR has I= layers, and each layer has h- = 4 heads (the ablation study of multi-head attention can be found in the appendix A) and d is set to be 256. The maximum sequence length N is set to 200 for MovieLens- Im and 100 for the other six datasets. All experiments are repeated three times, implemented in Py PyTorch 25 and conducted on a single NVIDIA 3090 GPU.
245
246
247
248
240
251
Table 2: Performance comparison to state-of-the-art models: GRU4Rec
16 BERTRec 3
SASRec
SMRec 6, S3-Rec 44. SINE 32) TGSRec 10, LightSANs
Jodie 22
TGN
28
Datasets
Meric
GRU4Rec
BERT4Rec
SASRcc
SMRcc S3-Rec SINE
TGSRe:
LightSAN
Jodie
TGN
RETR
HR@ 10
0.4358
0.4792
0.4622
0.4848
0.4917
0.4902
0.4887
0.4852
0.4813
0.4802
0.5142
Netflix
NDCG@10 10
0.2912
0.3330
0.3202
0.3492
0.3571
0.3601
0.3512
0.3441
0.3368
0,3318
0.3725
MRR
0.2431
0.2652
0.2519
0.2725
0.2819
0.2796
0.2778
0.2785
0.2687
0.2612
0.3134
HR 1O
0.3546
0.4819
0.4766
0.5083
0.5315
0,5264
0.5137
0.4994
0.4825
0,4782
0.5912
MSD
NDCG@ 10
0.3772
0.4891
0.4831
0.5112
0,5381
0,5304
0.5279
0.5163
0.4872
0,4832 0,5981
MRR
0,2503
0.3120
0.3079
0.3302
0.3494
0.3667
0.3612
0.3451
0.3224
0.3102 0.3901
HR@ IO
0.0788
0.1261
0,1182
0.1272
0,1336
0,1580
0.1537
0.1590
0.1447
0.1421
0.1768
Taobao
NDCG@ 10
0,0182
0.0425
0.0391
0.0631
0,0827
0,0873
0.0745
0.0694
0,0582
0.0571
0.1195
MRR
0.0273
0.0489
0.0436
0.0721
0.0919
0.0934
0.0802
0.0741
0.0628
0.0603
0.1117
HR@ 10
0.7265
0.7597
0.7373
0.7548
0.7597
0.7564
0.7533
0.7552
0.7492
0.7473
0,7730
Yelp
NDCG@ IO
0.4375
0.4778
0.4642
0.4789
0.4937
0,4902
0,4887
0,4863
0,4792
0.4784
0,5136
MRR
0,3630
0.4026
0.3927
0.4023
0.4107
0.4093
0.4072
0,4086
0.3997
0.3985
0.4354
HR@ 1O
0,5581
0.8269
0.8233
0.8302
0,8352
0,8311
0.8303
0,8294
0.8277
0.8259
0.8467
MovieLens
NDCG@ 10
0.3381
0.5965
0.5936
0.6079
0.6172
0,6134
0.6081
0.6119
0.6009
0,5998
0.6351
MRR
0.3002
0.5614
0.5573
0.5703
0.5812
0,5801
0.5734
0.5791
0.5651
0.5627
0.5921
HR@ IO
0.6432
0.6196
0.6275
0.6476
0.6687
0,6512
0.6506
0.6399
0.6384
0.6362
0.7138
Tmall
NDCG@@ 10
0,5169
0,5025
0,5049
0.5192
0,5423
0,5411
0,5372
0,5415
0.5307
0,5198
0.6103
MRR
0.4975
0.4026
0,4804
0.4934
0.5194
0,5147
0.5121
0,5119
0.5003
0.4997
0.5822
6
0.4190
0.8656
0.8729
0.8792
0,8813
0.8765
0,8773
0,8832
0.8780
0.8731
0.9001
Steam
NDCG@ 10
0,2691
0.6283
0.6306
0.64088
0.6573
0.6502
0.6491
0.6519
0.6451
0.6399
0.6795
MRR
0,2402
0.5883
0.5925
0.6011
0.6135
0.5972
0.6003
0.6104
0.5873
0.5798
0.6326
252
4.1
Main Results
The results of different methods on seven datasets are shown in Table 2 We can easily find that 254
attention- based models, SASRec [18), BertRec [31], SMRec 6). Re-Rec 144, SINE (2), TGSRec
10 and LightSANs [9I achieve better performance than RNN-based model GRU4Rec [16] on most datasets. It indicates that the attention mechanism is crucial for sequential recommendation, making
256
257
the model have better capacity to capture sequential characteristics. These models can capture the interaction information between all previous user behaviors via the attention mechanism. Besides, the graph-based models like Jodie 22 and TGN (8 also achieve competitive performance. Besides, our RETR can achieve statc-of-the performance by a large margin on most datasets compared with all bascline models.
258
259
280
261
262
Results on Yelp, MovieLensIM and Tmall. Specifically, our RETR achieves competitive perfor- mance the Yelp and Tmall. These datasets are sparse, containing less action information. Thus they have lots of noisy logged information. By effectively capturing the behavior pathway, our RETR is not affected by this trivial behavior information and captures the most informative behavior
263
264
265
266
representation to achieve better performance. Note that under the Tmall benchmark, RETR gains 7 % HR @ 10, 12% NDCG@ @1 10 and 14% MRR against the strongest baseline SMRec (6. Besides, for the MoveLens I M, our RETR also achieves the best performance among all competing baselines.
267
268
Results on large-scale datasets. Our RETR consistently achieve state-of-the-an results on large-scale datasets (Netflix, MSD, Taobao, and Steam). These datasets are challenging and difficult to capture pivotal behavior pathway useful for precise recommendation from the rich but noisy user's behaviors. Especially for the Taobao dataset, our RETR gains relative improvements of 12% % HR@ 10,
270
272
273
37% % NDCG@ @ 10 and 20% MRR against the strongest baseline SINE 32). It provides evidence that
274
our RETR can achieve competitive performance in both small- and large-scale datasets.
275
The substantial performance gains of our RETR indicate that focusing more on the behavior pathway enables RETR to capture sequential characteristics more efficiently and effectively than the vanilla self-attention mechanism, which considers all previous user behaviors.
276
7

---

278
4.2
Ablation Study
279
Effectiveness of each model component. In the left column of Table we analyze the efficacy of each component in RETR on the Yelp dataset and have the following observations. First, we remove the pathway router module and randomly choose whether it can maintained or dropped for each input behavior token. Removing the pathway router decreases the prediction performance lot (MRR: 0.4354 -> 0.3887), showing the necessity of learning behavior pathway effectively based on data-dependent module. Second, discarding the hierarchical update strategy for the behavior pathway also decreases the prediction performance, suggesting that this strategy is crucial for RETR
280
281
283
285
to get more accurate behavior pathway.
287
we adjust the number of blocks for RETR on Yelp. We find that the performance first increases rapidly with the growth of the block number and achieves the best performance at L 2. We perform a similar grid search on other datasets.
Number of blocks. In the right column of Table 3
288
289
Table 3: Ablation study of (Left) the effectiveness of each model component and (Right) the number of blocks for each RETR block. Experiments are conducted on the Yelp Dataset.
Model
MRR
Model (# number of blocks)
MRR
RETR
0.4354
RETR (I. = 1)
0.4197
RETR w/o Pathway Router
0.3887
RETR (L = 2)
0.4354
RETR w/o hierarchical update
0.4234
RETR (L = 33
0,4342
SASRec
0.3927
RETR (L
0.4340
Table 4: Ablation study of (Left) the effectiveness of different temperatures; Comparison Parameters and GFLOPss (Right). All ablation study experiments are conducted on the Yelp Dataset.
Model (temperature)
MRR
Model
Parameters (M)
GFLOPs
MRR
RETR (T = 0.4)
0.4312
RETR
5.021
9.558
0.4354
RETR (T 0.8)
0.4354
SASRec 18
4.916
9.552
0.3927
RETR (T 1)
0.4292
SINE B2
5.112
9.741
0.4011
RETR (T = 2)
0.4183
SMRec 6
5.173
9.864
0.4023
290
Effectiveness of temperature. In the left column of Table 4 we analyze the efficacy of different temperatures for Gumbel-Softmax sampling in RETR on the Yelp dataset. We observe that the perfor- mance first increases rapidly with the growth of the temperature and achieves the best performance when т 0.8, while the performance degenerates a lot when т > 1. The temperature т softens the softmax with 5 > However, when 7 - oo, the Gumbel-Softmax distribution p- (y,) - 0.5 becomes more smooth, leading to the maximum uncertainty. To make the sampling results more convineing, we apply the temperature calibration < during training to avoid overconfident predictions. These results show that Gumble-Softmax sampling with lower temperature (r < 1) avoids overconfident predictions, leading to better performance.
291
292
293
294
295
296
298
299
Evaluation on efficiency. The efficiency is compared between SASRec 18] SINE 32 and SMRec
300
6 on the Yelp dataset. The computation cost measured with gigabit floating-pott point operations (GLLOPs) on the self-attention module with position encoding. Meanwhile, the model scale measured
301
302
with parameters is also presented. As shown in Table our RETR has almost the same number of parameters or GFLOPs, compared with SASRec, indicating that our pathway router is weight module. Our pathway attention does not bring more costs. It's worth noticing that the parameter scales and GLOPs of other competing transformers (apart from SASRec) are larger than RETR, but our RETR achieves higher performance. This result shows that our RETR is more efficient and effective than other competing attention-base models.
303
304
305
306
307
4.3
Case study
309
Technically, we Setups. We also provide qualitative visualizations for our RETR, and SASRec 18]
use the GradCAM (29 to generate behavior heatmaps of the output of the last layer in each model.
311 Three random examples of users' historical behaviors in the Steam dataset are shown in sequential
order through subplots (a)-(c in Figure 3 We provide attention heatmaps of each example at the last
8

---

313
ten time steps. We can observe three main behavior pathway characteristies corresponding to three behavior sequences respectively: (a) Casual behavior pathway: RPG games are randomly clicked by the user, while the user has continuing interest in RPGss. (b) Correlated behavior pathway: The user has recently been interested indie games. (c) Drifted behavior pathway: The user has recently been interested in simulation games but chooses an indie game at last.
314
316
317
318
Visualization results. We elaborate the three representative categories of behavior pathway in recommender systems with model-learned attention heatmaps. (1) Casual behavior pathway: As shown in Figure 3 a)), the RGB game is randomly clicked at casual times. Our RETR can capture all the RPG casual behavior pathways, while the SASRec focuses on the incorrect recent adventure games. The SASRec cannot capture the early clicked RPG game. This phenomenon proves that our RETR can deal with the casual behavior pathway effectively. (2) Correlated behavior pathway: For the correlated behavior pathway, we also provide an example which is shown in Figure bb). The indie game is clicked many times recently, leading to the final decision to an indie game. Our
319
320
321
322
323
324
326
RETR can effectively capture the correlated behavior pathway. However, the SASRee provides higher attention scores the recent RPG games. On the contrary, our RETR pays no attention to these
328
wrong results, showing that it has a greater ability to cope with the correlated behavior pathway.
(3) Drifted behavior pathway: As shown in Fig c). The user was initially interested in the indie game but suddenly became interested simulation games recently and chose an indie game at last.
330
Our RETR captures the drifted behavior pathway for the indic game and has not concentrated on the old drifted pathway simulation games, while the SASRec is affected by the trivial behaviors
332
of simulation games. These visualization results strongly show that our RETR capture various behavior pathways dynamically for each user.
334
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Adventure
VR
RPG
Action
RPG
Action
RPG
Adventure Adventure RPG
Ground
Target: RPG
RETR
SASRec
(8) Casual Behavior Pathway
RPG Strategy Adventure
Action
Action
RPG
Indie
Indie
Indie
Indie
Ground Truth
Target: Indie
RETR
SASRec
(b) Correlated Behavior Pathway
Action
Indie
RPG
Casual
Indie
Simulate
Simulate
Simulate
Indie
Indie
Ground
Truth
Target: Indle
RETR
SASRec
(c) Drifted Behavior Pathway
Figure Visualizations of behavior heatmaps for RETR and SASRec of three random users Steam dataset. They are corresponding to casual, correlated and drifted behavior pathways respectively.
5
Conclusion
336
sequential recommender is designed to make accurate recommendations based on users' historical behaviors. The sequential recommendation system has benefited many practical applications such as online advertising. However, the users' behaviors are dynamic and come in a continually evolving manner. user's current decision may only call upon the interest from the certain relevant behaviors
337
338
339
of the past. We conclude these sequential characteristics the behavior pathway. Previous models cannot capture the behavior pathway dynamically. We propose the Recommender Transformer
340
341
342
(RETR) with novel pathway attention mechanism to tackle these challenges. The pathway attention develops pathway router to dynamically get the behavior pathway for each user and capture the evolving patterns. Our RETR achieves fate--of-thearr performance on seven real-world datasets for sequential recommendation. The visualization results also show that our RETR can dynamically capture the behavior pathway for each user.
343
345

---

References
347
348
[1] Gediminas Adomavicius and Alexander Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE transactions on knowledge and data engineering, 17(6):7347749, 2005.
349
350
351
[2] Nabiha Asghar. Yelp dataset challenge: Review rating prediction. arXiv preprint arXiv: 2016.
352
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.066 06450, 2016.
354
[4] Prateep Bhattacharjee and Sukhendu Das. Temporal coherency based criteria for predicting video frames using deep multi-stage generative adversarial networks. In NeurlPS, pages 4268 4277, 2017.
355
356
[5] Shuqing Bian, Wayne Xin Zhao, Kun Zhou, Jing Cai, Yancheng He, Cunxiang Yin, and Ji-Rong Wen. Con- trastive curriculum learning for sequential user behavior modeling via data augmentation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 3737-3746.
357
359
2021.
360
[6] Chao Chen, Haoyu Geng, Nianzu Yang, Junchi Yan, Daiyue Xue, Jianping Yu, and Xiaokang Yang. Learn- ing sell-modulaiing attention in continuous time space with applications to sequential recommendation. In International Conference on Machine Learning, pages 1606-1616. PMLR, 2021.
361
362
[7] Qiang Cui, Shu Wu, Qiang Liu, Wen Zhong. and Liang Wang. Mv-mn: A multi-view recurrent neural network for sequential recommendation. IEEE Transactions on Knowledge and Data Engineering, 32(3)3317-31, 2018.
384
365
386
[8] Jeff Dean. Introducing pathways: A nextgeneration ai architecture. Google Blog, 2021.
387
19] Xinyan Fan, Zheng Liu, Jianxun Lian, Wayne Xin Zhao, Xing Xie, and Ji-Rong Wen. Lighter and better: low-rank decomposed self-attention networks for next-item recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1733-1737, 2021.
370
371
[10] Ziwei Fan, Zhiwei Liu, Jiawei Zhang. Yun Xiong, Lei Zheng, and Philip S Yu. Continuous timm sequen- tial recommendation with temporal graph collaborative transformer. In Proceedings of the 30th ACM International Conference Information & Knowledge Management, pages 433 333-442, 2021.
372
[11] Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Transactions on Information Systems (TOIS), 39(1):1-42, 2020.
375
376
377
[12] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint 2 039611 2021.
[13] Ruining He, Chen Fang, Zhaowen Wang, and Julian McAuley. Vista: A visually, socially, and temporally- aware model artistic recommendation. In Proceedings of the 10th aem conference on recommender systems, pages 309-316, 2016.
380
381
382
[14] Ruining He and Julian McAuley. Fusing similarity models with markov chains for sparse sequential recommendation. In 2016 IEFE 16th International Conference on Data Mining (ICDM), pages 191-200. IEFF, 2016.
384
385
[15] Jonathan L Herlocker, Joseph A Konstan, Al Borchers, and John Riedl. An algorithmic framework for performing collaborative filtering. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 230-237, 1999.
386
387
388
[16] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommen- dations with recurrent neural networks. arXiv preprint 1rXiv:1511.06939, 2015.
380
[17] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, arXiv preprint ar Xrv: 1.01 144, 2016.
391
392
[18] Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM), pages 197-206. IEBE, 2018
383
[19] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
10

---

395
[20] Yehuda Koren. Factorization meets the neighborhood: multifaceted collaborative filteinng model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 426-434, 2008
397
388
[21] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
389
400
[22] Srijan Kumar, Xikun Zhang, and Jure Leskovee. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages
401
402
2019.
403
[23] Yuli Liu, Christian Walder, and Lexing Xie. Determinantal point process likelihoods for sequential recommendation. arXiv preprint arXiv:2204.1 11562, 2022.
404
405
[24] Jie Lu, Dianshuang Wu, Mingsong Mao, Wei Wang, and Guangquan Zhang. application developments: survey. Decision Support Systems, 74:12-32, 2015.
Recommender system
406
407
[25] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high perfforrrre deep learning library. Advances neural information processing systems, 32, 2019.
409
[26]
Qi Pi. Guorui Zhou, Yujing Zhang, Zhe Wang. Lejian Ren, Ying Fan. Xiaogiang Zhu, and Kun Gai. Search-based user interest modeling with lifelong sequential behavior data for click through rate prediction. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. pages 2685-2692, 2020.
411
412
413
414
[27] Pengjie Ren, Zhumin Chen, Jing Li, Zhaochun Ren, Jun Ma, and Maarten De Rijke. Repeatnet: A repeat aware neural recommendation machine for session-based recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4806 4813, 2019.
415
417
[28] Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bron- stein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006. 10637, 2020.
418
419
420
[29] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 618-626, 2017.
421
422
423
[30] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of oo--eeertstsssseseeeeeeeeee layer. arXiv preprint 17X1v:1701.06538, 2017.
424
426
[31] Fei Sun, Jun Liu, Jian Wa, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bertárec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1441-1450, 2019.
428
[32] Qiaoyu Tan, Jianwei Zhang, Jiangehao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, and Xia Hu. Sparse-interest network for sequential recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 598-606, 2021.
430
431
432
[33] Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference on web search and data mining, pages 665-573. 2018.
433
435
[34] Pengfei Wang, Jiafeng Guo. Yanyan Lan, Jun Xu, Shengxian Wan, and Xueqi Cheng. Learning hierarchical representation model for nextbasket recommendation. In Proceedings the 38th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 403-412, 2015.
437
[35] Chao- networks. In Proceedings of the tenth ACM international conference on web search and data mining, pages 0355503,222 2017.
Chao-Yuan Yuan Wu, Amr Ahmed, Alex Beutel, Alexander Smola, and How Jing. Recurrent recommender
439
440
441
[36] Liwei Wu, Shuqing Li, Cho-Jui Hsich, and James Sharpnack. Ssc-pt: Sequential recommendation via personalized transformer. In Fourteenth ACM Conference on Recommender Systems, pages 928-337, 2020.
442
443
[37] Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-base recom- mendation with graph neural networks. In Proceedings of the AAAI conference on artificial intelligence. volume 33, pages 346 353,,,33220 2019.
444
11

---

448
[38] Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu. Fuzhen Zhuang. Junhua Fang. and Xiaofang Zhou. Graph contextualized self-attention network for session-based recommendation. In IJCAI, pages 3940-3946, 2019.
448
[39] An Yan, Shuo Cheng, Wang-Cheng Kang, Mengting Wan, and Julian McAuley. Cosrec: 2d convolutional neural networks for sequential recommendation. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 2173-2176, 2019.
450
451
452
[40]
Zeenat F Zaidi. Gender differences in human brain: a review. The open anatomy journal, 2(1), 2010.
453
[41] Wayne Xin Zhao, Junhua Chen, Pengfei Wang. Qi Gu, and Ji-Rong Wen. Revisiting alternative experi- mental settings for evaluating top-n item recommendation al algorithms. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages
454
2020.
456
[42] Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. Recbole: Towards unified, comprehensive and efficient framework for recommendation algorithms. In Proceedings of the 30th ACM International Conference on Information Knowledge Management, pages 4653 4664, 2021.
458
459
460
[43] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li. and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1059-1068. 2018.
481
462
463
[44] Kun Zhou, Hui Wang. Wayne Xin Zhao, Yutao Zhu, Sirui Wang. Fuzheng Zhang, Zhongyuan Wang, and Ji- Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 1893-1902, 2020.
485
486
467
[45] Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. Filler-enhanced mlp all you need sequential recommendation. In Proceedings of the ACM Web Conference 2022, pages 2388-2399, 2022.
468
Checklist
469
1. For all authors...
471
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
472
473
(b) Did you describe the limitations of your work? [Yes] Broader impact in the supplemental material
475
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Broader impact in the supplemental material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
478
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
481
3. If you ran experiments..
482
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] The code and the data are proprietary.
483
484
485
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
486
[Yes] Section in the main text.
487
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] Section in the main text.
488
489
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Section in the main text.
491
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Section in the main text.
(b) Did you mention the license of the assets? [N/A]
494
(c) Did you include any new assets either in the supplemental material or as URL? [No]
485
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're us- ing/curating? [Yes] Section in the main text.
12

---

497
(e) Did discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [Yes] Broader impact in the supplemental material.
499
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
503
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
505
13


Reviews:
[{"review_id": "yyw3FJodPHI", "reviewer": "Reviewer_sJpS", "summary": "This paper proposes a new sequential recommendation model with behavior pathways, effectively capturing specific evolving item patterns for each user. The pathway attention using learned binary routes can effectively remove unnecessary items for a given user sequence. Experimental results show that the proposed model significantly outperforms existing models on seven benchmark datasets and achieves state-of-the-art performance. Overall, the idea of using behavior pathways is interesting to me, but it has several weak points, especially in evaluating the proposed model. ", "questions": "- Q1) Is V^t in Eq. (4) the same as hat(R^l) in Eq. (5)? If so, please clarify this equation.\n- Q2) The proposed idea seems to be model-agnostic. Could you apply the pathway module for other sequential recommender models, e.g., BERT4Rec?\n- Q3) In Figure 1, the authors mention various behavior pathways. Does the proposed model capture various pathways or focus on capturing drifted behavior pathways?\n", "limitations": "This paper does not address the negative societal impact. However, this paper seems not to have any negative impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- (Originality) It is interesting to introduce behavior pathways in long user sequences. Besides, this paper utilizes a router to choose item patterns selectively.\n- (Originality) Because the idea of using the router is a model-agnostic property, it can be effectively applied to various sequential recommender models.\n- (Clarity) It is well-written and easy to understand.\n\n\nWeaknesses\n- (Quality) Although the evaluation is extensive, the proposed model has not been compared with recent sequential recommendation models. Please refer to the following references.\n[1] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, Ji-Rong Wen, “S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization,” CIKM 2020\n[2] Qiaoyu Tan, Jianwei Zhang, Jiangchao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, Xia Hu, “Sparse-Interest Network for Sequential Recommendation,” WSDM 2021\n[3] Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, Philip S. Yu, “Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer,” CIKM 2021\n- (Quality) In Section 4.3, the visualization result shows that the proposed model effectively captures the pathways in user sequences. However, it is wondering if this result is generalized for other cases. It is necessary to show a quantitative result on whether the proposed model effectively captures a useful pathway. One possible evaluation is that the proposed model also shows a comparable result using a subset of sequences derived from a pathway.\n- (Quality) In Table 3, RETR(L=1) is similar to SASREC(L=1). However, RETR(L=1) shows a better performance than SASRec(L=1). It is wondering if the proposed model is fairly compared with the existing model.\n- (Clarify) There are some minor typos.\n- 1p 37line: use’s -> user’s\n- 2p 41line: second -> first\n- 2p 44line: first -> second\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kGNUEpkZKjL", "reviewer": "Reviewer_bU2p", "summary": "In this paper, the authors propose the Recommender Transformer (RETR) with a novel Pathway Attention mechanism. RETR can dynamically plan the behavior pathway specified for each user, and sparingly activate the network through this behavior pathway to effectively capture evolving patterns useful for recommendation.", "questions": " It is unclear why need use this swith rounter in sequential recomemndation. Usually, the length of users' behaviors is very short, like less than 25 for most of transaction. Do we really need this router in the Transformer? \n\nActually, recent work verify that a simle MLP can outperform the Transformer in the sequential recommendation.\n\nZhou, Kun, et al. \"Filter-enhanced MLP is all you need for sequential recommendation.\" Proceedings of the ACM Web Conference 2022. 2022.\n\n\n3. In the experiment, \"we pair the ground-truth item with 100 randomly sampled negative items that the user has not interacted with.\" Does this raise a smapling bias?", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The paper is well written and easy to follow. Basically, the authors try to use the behavior pathway in the Transformer.\n\n2. The experimental results are good compared with the baselines.\n\n\nWeakness:\n1. The novelty of this work is not very high. The mechanism of router is wildey used in the MoE-style Transformer. It seems that this work applied it to the sequential recommendation. The Gumbel-softmax is also widely used to optimize the discrete binary variable.\n\n2. It is unclear why need use this swith rounter in sequential recomemndation. Usually, the length of users' behaviors is very short, like less than 25 for most of transaction. Do we really need this router in the Transformer? \n\nActually, recent work verify that a simle MLP can outperform the Transformer in the sequential recommendation.\n\nZhou, Kun, et al. \"Filter-enhanced MLP is all you need for sequential recommendation.\" Proceedings of the ACM Web Conference 2022. 2022.\n\nThe motivation of this work is thus not very strong. \n\n3. In the experiment, \"we pair the ground-truth item with 100 randomly sampled negative items that the user has not interacted with.\" Does this raise a smapling bias?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "VfnnsSrHmB", "reviewer": "Reviewer_uSox", "summary": "This paper proposes a recommender transformer with a pathway attention mechanism. It is characterized by its ability to capture three types of user pathways and predict user action sequences with high accuracy. The paper demonstrates the usefulness of the proposed method in comparison with several state-of-the-art methods using several types of real data.", "questions": "The authors state that user pathways fall into three categories (correlated, casual, and drifted behavior pathways), but on what evidence? Justification is needed.", "limitations": "There is no mention in the paper of negative impacts on society. Also, I can't think of any.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n  - Starting with the actual example in Figure 1, the motivation for proposing the method is well explained, making it easy to understand the content of the proposed technique.\n  - The authors have conducted prediction experiments using seven different behavioral log datasets from various sites. They compared the accuracy of the proposed method with seven existing methods and confirmed that the proposed method outperforms them.\n  - Experiments are conducted using real data, not artificial data.\n- Weaknesses\n  - The meaning and boundaries of the three types of pathways are vague. The definition of each should be clearly stated. Also, are these three types sufficient?\nFor example, what is the difference between the Correlated behavior pathway and the Drifted behavior pathway? They seem to have similar properties in the local and short-term. The types and definitions of pathways should be better justified, such as providing references that support the authors' definitions.\n  - Although the experimental results quantitatively demonstrate the effectiveness of the proposed method, the architecture in Figure 2 is straightforward and somewhat lacking in technical novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OoQ970eZ6CA", "reviewer": "Reviewer_zis8", "summary": "The authors propose Recommender Transformer (RETR) with a Pathway Attention mechanism which can generate the behavior pathway hierarchically and capture the evolving patterns dynamically through the pathway. The key design is a learned binary route to prevent the behavior pathway from being overwhelmed by trivial behaviors. The authors also show RETR has high accuracy and efficiency compared with other self-attention or transformer based sequential recommendation methods through experiments.", "questions": "- What’s the essential difference between RETR and sequential model with attention mechanism when putting aside the concept of pathway?  \n- In the introduction, the authors list three kinds of behavior pathways, so how can RETR capture them to make precise recommendations in each case? \n- In Line 172, what’s the meaning of  “lose the privilege to be part of the behavior pathway”? Since the off-pathway tokens are also considered, it seems that all items will always be considered whether they are in the pathway or not. \n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The paper is generally easy to follow.\n2. The idea of using pathway in recommendation algorithms seems to be new. \n3. The authors conducted extensive experiments on seven datasets to prove RETR can make accurate recommendations.\n\nCons:\n1. The essential difference between RETR and other sequential recommendation methods with attention mechanism is not clear. It seems to me that RETR not only utilizes the “on-pathway tokens” but also leverages the “off-pathway tokens” as they “also convey contextual information” and the difference between the two kinds of tokens is their weight. However, in SASRec and all the other self-attention-based methods, different tokens already have varying attention weights, so that more important historical items may have higher attention weights and less important historical items may have lower attention weights. I don't understand why is it necessary to design a pathway.\n2. The targeted problem is not new, and several recent works have been proposed to address the same issue of self-attention. Apart from reference [6] in the paper, there are several others trying to improve self-attention-based recommendation methods. For example, the LightSANs work published in SIGIR ‘21. It would be useful to also conduct experimental comparisons with these more recent baselines in addition to [6]. Besides, there are many sparse attention works in the literature and it is interesting to replace the proposed pathway-based method with other sparse attention methods to see if pathway-based method is superior.\n3. Besides self-attention-based methods, other types of sequential recommendation methods, e.g., temporal graph-based sequential recommendation methods, have also achieved state-of-the-art results. Considering that attention weights can be regarded as edge weight in a graph, it might be useful to compare with some recent temporal graph based sequential recommendation methods, e.g., Jodie in KDD 19 and TGN in ICML 20. Especially, TGN also adopted attention in its model.\n4. There is no evidence/experiment to show that RETR will not be overwhelmed by trivial behaviors. As this is the main claim of the paper, it should be necessary to have more justifications. The case study in Section 4.3 seems to be a cherry-pick result.\n5. All the datasets that used in this paper are relatively small. Larger datasets such as MSD and Netflix should be more desirable.\n\nMinor issues:\n1. “in the second line” in Line 41 and “in the first line” in Line 44 should be exchanged. \n2. In column “Actions” of Table 1, the commas are not consistent. \n3. The authors should use “GRURec” or “GRU4Rec” consistently in the paper to avoid misunderstandings.\n4. The ablation study of multi-head attention is missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject