Recommender Transformers with Behavior Pathways
Authors: Zhiyu Yao
Xinyang Chen
Sinan Wang
Qinyan Dai
Yumeng Li
Tanchao Zhu
Mingsheng Long
Venue: Submitted to ICLR 2023 (2023)
Keywords: Recommendation
Transformers
Behavior Pathway

Abstract: Sequential recommendation requires the recommender to capture the evolving behavior characteristics from logged user behavior data for accurate recommendations. Nevertheless, user behavior sequences are viewed as a script with multiple ongoing threads intertwined. We find that only a small set of pivotal behaviors can be evolved into the user's future action. As a result, the future behavior of the user is hard to predict. We conclude this characteristic for sequential behaviors of each user as the \textit{behavior pathway}. Different users have their unique behavior pathways. Among existing sequential models, transformers have shown great capacity in capturing global-dependent characteristics. However, these models mainly provide a dense distribution over all previous behaviors using the self-attention mechanism, making the final predictions overwhelmed by the trivial behaviors not adjusted to each user. In this paper, we build the Recommender Transformer (RETR) with a novel Pathway Attention mechanism. RETR can dynamically plan the behavior pathway specified for each user, and sparingly activate the network through this behavior pathway to effectively capture evolving patterns useful for recommendation. The key design is a learned binary route to prevent the behavior pathway from being overwhelmed by trivial behaviors. Pathway attention is model-agnostic and can be applied to a series of transformer-based models for sequential recommendation. We empirically evaluate RETR on seven intra-domain benchmarks and RETR yields state-of-the-art performance. On another five cross-domain benchmarks, RETR can capture more domain-invariant representations for sequential recommendation.

Under review as a conference paper at ICLR 2023
RECOMMENDER TRANSFORMERS WITH BEHAVIOR PATHWAYS
Anonymous authors Paper under double-blind review
ABSTRACT
Sequential recommendation requires the recommender to capture the evolving behavior characteristics from logged user behavior data for accurate recommenda- tions. Nevertheless, user behavior sequences are viewed as a script with multiple ongoing threads intertwined. We find that only a small set of pivotal behaviors can be evolved into the user's future action. As result, the future behavior of the user is hard to predict. We conclude this characteristic for sequential behaviors of each user the behavior pathway. Different users have their unique behavior pathways. Among existing sequential models, transformers have shown great capacity in capturing global-dependent characteristics. However, these models mainly provide a dense distribution over all previous behaviors using the self-attention mechanism, making the final predictions overwhelmed by the trivial behaviors not adjusted each user. In this paper, we build the Recommender Transformer (RETR) with a novel Pathway Attention mechanism. RETR can dynamically plan the behavior pathway specified for each user, and sparingly activate the network through this be- havior pathway effectively capture evolving patterns useful for recommendation. The key design is a learned binary route prevent the behavior pathway from being overwhelmed by trivial behaviors. Pathway attention is model-agnostic and can be applied to a series of transformer-based models for sequential recommendation. We empirically evaluate RETR on seven intra-domain benchmarks and RETR yields state-of- the-ar performance. On another five cross-domain benchmarks, RETR can capture more ionaaininvariiat representations for sequential recommendation.
INTRODUCTION
Recommender systems (Hidasi et al., 2015 Lu et al. 2015 Zhao et al. 2021 have been widely adopted in real-world industrial applications such E-commerce and social media. Benefiting from the increase in computing power and model capacity, some recent efforts formulate recommendation time-series forecasting problem, known sequential recommendation (Kang & McAuley] 20018] Sun et al. 2019 Chen et al.] (021). The core idea of this field is to infer upcoming actions based on user's historical behaviors, which are reorganized as time-ordered sequences. This intuitive modeling of recommendation is proved time-sensitive and context-aware to make precise predictions.
Recent advanced sequential recommendation models, such as SASRec (Kang & McAuley 2018). Bert4Rec (Sun et al. 2019) and S3-Rec (Zhou et al. 2020), have achieved significant improvements. Transformers enable these models recognize global-range sequential patterns, and to model how future behaviors are anchored in historical ones. The self attention mechanism does make it possible to explore all previous behaviors of each user, with the whole neural network activated. However, misuse of all user information, regardless of whether it is informative or not, floods models with trivial ones, makes models dense in neuron connections and inefficient in computation, and results in key behaviors losing voice. And this clearly contradicts with the way our brain works.
The human being has many different parts of the brain specialized for various tasks, yet the brain only calls upon the relevant pieces for given situation (Zaidi 2010). To some extent, user behavior sequences can be viewed as a script with multiple ongoing threads intertwined. And only key clues suggest what will happen next. In sequential recommendation, pivotal behaviors can evolved into the user's s future action. And we conclude this characteristics of sequential behaviors as the behavior pathway.
find that only small part of

---

Under review as a conference paper at ICLR 2023
Correlated Behavior Pathway
Predict Item
Casual Behavior Pathway
Drifted Behavior Pathway
Different Users
User Behavior Trace Axis
Figure I: Three typical examples of the behavior pathway for different users: correlated, causal, and drifted. The behavior pathway is outlined by the red boxes.
Different users have their unique behavior pathways, and we have provided three typical examples: (a) Correlated behavior pathway: A user's behavior pathway closely associated with behaviors in certain period. As shown in the first line of Figure the mouse is clicked many times recently, leading to the final decision to buy a mouse. (b) Casual behavior pathway; A user's behavior pathway interested in specific item casual times. In the second line of Figure [ the backpack is randomly clicked sequentially in a multi-hop manner. (c) Drifted behavior pathway: A user's behavior pathway particular brand might drift time. In the third line of Figure the user initially interested in keyboard, but suddenly became interested in buying a phone at last. It's challenging to capture these potential behaviors dynamically for each user make precise recommendations.
Motivated by the Pathways (Dean) 2021). a new way of thinking about AI, which builds a single model that sparsely activated for all tasks with small pathways through the network called into action needed, we propose novel Recommender Transformer (RETR) with a Pathway Attention mechanism. RETR dynamically explores behavior pathways for different users and then captures evolving patterns through these pathways effectively. Specifically, the user-dependent pathway attention, which incorporates a pathway router, determines whether or not a behavior token will be maintained in the behavior pathway. Technically, the pathway router generates customized binary route for each token based on their information redundancy. RETR has a stacked structure, and successive pathway routers constitute a hierarchical evolution of user behaviors. To enable the pathway router to be end-to-end optimized, we propose an adaptive Gumbel-Softman sampling strategy overcome the non-differentiable problem of sampling from Bernoull distribution.
To effectively capture the evolving patterns via the behavior pathway, our pathway attention mech- anism makes RETR mainly attend the obtained pathway. We force the model to focus the most informative behaviors by using the query routed through the behavior pathway. We cut off the interaction from the off-pathway behaviors of the query. Compared with using all previous behaviors, our pathway attention mechanism is obviously more effective and can avoid the most in- formative tokens being overwhelmed by trivial behaviors. Besides, our pathway attention mechanism is model-agnostic and can be easily applied to the existing transformer-basee models. To validate the effectiveness of our approach, we conduct experiments on seven intra-domain competitive datasets for sequential recommendations and RETR achieves state-of-the-art performance; Furthermore, our RETR also achieves consistent performance improvements under the cross-domain setting, indicating RETR capture more domain- uniierrall representation for sequential recommendation. Our main contributions can be summarized as follows:
We first propose the concept behavior pathway for sequential recommendation, and find the key to the recommender is to dynamically capture the behavior pathway for each user. We propose the novel Recommender Transformer (RETR) with a novel pathway attention mechanism, which can generate the behavior pathway hicrarchically and capture the evolving patterns dynamically through the pathway.
We validate the effectiveness of RETR on seven intra antra-domain benchmarks and five cross- domain benchmarks, both achieving state-of-the-ee performance. RETR capture more domain-invariant representations and our pathway attention can be applied together with a rich family of transformer-bas models to yield consistent performance improvements.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Traditional recommendation approaches. Capturing evolving behavior characteristics is crucial
for many online applications, such as advertising, social media and E-commerce, and it is the key
challenge for sequential recommendation (Adomavicius & Tuzhilin 2005| Kang & McAuley| 2018 Cui et al. 2018 Yan et al., 2019; Fang et al. [020] Pi et ] [2020] Bian et al. [021] Zhou et al. 2022). Traditional recommendation approach, such as the collaborative filtering (CF) (Herlocker al. 1999) based on matrix approximation (Koren 2008 Koren et al. 2009). always
2022|
Liu et al.
assumes that the user's behavior is static. However, in practice, user behaviors often change over time due to various reasons, making the CF deteriorate in real-world application.
Sequential recommendation approaches. To overcome this challenge, some methods, such as FPMC (He& & McAuley 2016)) and HRM (Wang et al. (2015), use Markov chains to capture sequential patterns by learning user-specific transition matrices. Higher-order Markov Chains assume the next action is related to several previous actions. Benefit from this strong inductive bias, MC-based methods (He & McAuley 2016 He et al.] (016) show superior performance in capturing short- term patterns. At the same time, there potential state space explosion problem when these approaches are faced with different possible sequences (Wu et al., 2017). In recent years, many works have been using the deep neural network for sequential recommendation. The GRU4Rec (Hidasi et al. 2015) and the RepeatNet (Ren et al., 2019) adopt the recurrent network to capture dynamic patterns from the user behaviors dependent on sequence positions. The RNN-based models achieve competitive performance in capturing short-term behavior patterns but cannot capture long-term behavior patterns effectively. The CNN-based model, such as Caser (Tng & Wang 2018), applics convolutional operations to extract transitions while tending to overlook the intrinsic relationship across user behaviors. The GNN-based methods, such as SRGNN (Wu et al. 2019), GCSAN (Xu et al. 2019), Jodie (Kumar et al. 2019 and TGN (Rossi al. 2020) model behavior sequences as recommendation.
graph-structured data and incorporate an attention mechanism for session- In addition, DIN (Zhou et al., 2018) uses the gate mechanism weight different user behaviors. However, concatenating all behaviors makes these models overlook the sequential characteristics. Recently, the MLP-based model like FMLP-Rec (Zhou al.) 2022) uses the MLP as the backbone for sequential recommendation. However, these methods are still overwhelmed by the trivial behaviors.
Transformer-basee models for Sequential Recommendation. SASRec (Kang & McAuley 2018). BertRec (Sun et al.; 2019), S3-Rec (Zhou et al., 2020), TGSRee (Fan et al. 2021b). LightSANs (Fan jt aal.,22021: and SSE-PT (Wu et 2002)) introduce the transformer architecture into sequential recommendation, which might lead to the over parameterized architecture of Transformer based methods. These models capture the evolving patterns by the self-attention mechanism, interacting with all previous behaviors. However, dense interactions will make the model not adapt to different users and overwhelm behavior pathways. Some methods like Locker (He et al. 2021) and Rec- denoiser (Chen et al. 2022) propose the sparse attention mechanism with learned mask, while they may overlook the ability of capturing the behavior pathway in the token level. To tackle this challenge, our paper builds the Recommender Transformer (RETR) with new Pathway Attention mechanism that is dynamically activated for the behavior pathway of all users. Distinct from the previous routing architecture like Switch Transformer (Fedus et al. (22T)) using the MoE (Shazeer et al. 2017) structure for natural language tasks or TRAR {Zhou et al. 202TT) using the learned sparse attention for visual question answering, our RETR is designed explicitly for sequential recommendation. Our RETR uses the pathway router to adaptively route the sequential behavior of each user rather than routing the experts of nerdforwwrd networks in switch transformer.
METHOD
Suppose that have set of users and items, denoted by 24 and I respectively. In the task of sequential recommendation, chronologically-oddeee behaviors of a user u É U could be represented by user-interacted item sequence: (11, (, Formally, given user u with her his behavior sequence {ir,... in), the goal of sequential recommendation is to predict the next item the user u would interact with at the (n + 1)-th step, denoted (f,41 i...)
As aforementioned,, we highlight the key to sequential recommendation as the exploration of user- tailored behavior pathways, through which evolving characteristics could be learned. Motivated by
3

---

Under review as a conference paper at ICLR 2023
Predict Behavior
Pathway Attention
huats-head Attention
Adaptivo Gumbel- Softmmx
Sampling
Lrear
LN
MEP
Pathway Attention
Aug
Embeciiing
Input User Behaviors
Figure 2: The architecture of Recommender Transformer (RETR) on the right subfigure. Pathway Attention (left) explores the behavior pathway by the pathway router (orange module) and captures the evolving sequential characteristics of the use behaviors by the multi-head attention.
this, we propose novel Recommender Transformer (RETR) with a new Pathway Attention, the core subassembly of which is pathway router. Besides the modification of architecture, we additionally introduce a hierarchical update strategy for the behavior pathway in the fera-fd procedure.
3.1 RECOMMENDER TRANSFORMER
Considering the limitation of overwhelming attention in Transformers (Bhattacharjee & Das 2017) for sequential recommendation, we renovate the vanilla architecture to the Recommender Transformer (RETR) with Pathway Attention mechanism, shown in Figure
Model inputs. To obtain the model inputs, we follow the sliding window practice and transform the user's behavior sequence into a fixed-length-N sequence ($1, , Then we produce an item embedding matrix Ez E [ where d is the embedding dimensionality. We perform a look-up operation from Ez to retrieve the input embedding matrix E, E R xd for sequence s. Besides, we also add learnable position embedding P. R N for sequence Finally, we can generate the input embedding of each behavior sequence s as X, E, P, E Nxd.
Overall architecture. Recommender Transformer is characterized by stacking the Pathway Attention blocks and feed-forward layers alternately, containing L blocks. This stacking structure conducive to learning behavior representations hierarchically. The overall equations of block 7 are formalized as:
ž' R' Path-MSA (Z'-1, (
7
LN
7
Z (1-)
LN (FFN (Z') + 2').
where Z' E Nxx {1,.... L}} denotes the output of the 1-th block. The initial input T X. E R represents the raw behavior embedding. RL-1 E R is the previous route from the 1)-th block and initialize all elements in the route RD to Path-MSA()) is to conduct the pathway multi-hee head self-atter self-attention. LN(-) is to conduct layer normalization (Ba et al. 2016) and FFN represents the point-wiee wise reew-fowward network (Bhattacharjee & Das 2017).
3.1.1 PATHWAY ATTENTION
Note that the single-brand branch self-attention mechanism (Bhattacharjec & Das 2017) in vanilla trans- former cannot model the behavior pathway dynamically, resulting in key behaviors being over- whelmed by those non-pivotal trivial ones. To solve this problem, propose the Pathway Attention mechanism, as shown Figure which can dynamically attend to the behavior pathway of pivotal behavior tokens.
Pathway router. The pathway attention employs sequence-adaptive pathway router to custom-tailor behavior pathway routes for The router generates binary route R' E {0, 1] to determine
4

---

Under review as a conference paper at ICLR 2023
whether behavior token would be part of the behavior pathway not. Each router takes the pre-order route R'-1 and user behavior tokens Z'-1 E RNNx xx of the (1 - 1) block as its inputs. All elements in the route are initialized by 1 and are updated progressively in training.
Foremost, to suppress the potential disturbance to the model caused by the local drifted interest (Figure [ it is crucial to incorporate the global information in the route generation. We apply the average pooling to all the preserved behavior tokens routed by R'-1, and produce the global sequential representation via a multilayer perceptron (MLP) module. Then, we combine this global representation with the inputs and employ a residual connection to maintain the original input information. Finally, we feed them to another MLP layer to predict the probabilities of keeping or dropping the behavior tokens. All MLP layers are column- bolumn-wise and operate on the embedding dimensionality. The above procedure can be formulated as follows:
Ei=1 R-'ZZ!'
Z Z Z O MLP
E R
π = Softmax (MLP(Zemb))
Nx2
where @ is the Hadamard product. For t E (1.2, ... N),, we let n. = [1 - at ag] where the logit Oz denotes the probability that the (-th behavior token is kept alive for the behavior pathway.
Adaptive Gumbel-Softmax sampling from π for router. Our goal is to generate the binary route from However, sampling from directly is non-differentiable, and it will impede the gradient-based training. Gumbel-Softmax (Jang et al. (2016) is an effective way to approximate the original non-differentiable sample from a discrete distribution with a differentiable sample from Gumbel-Softmal distribution. Thus, we adapt the Humbel-Softmal technique to achieve such a sampling procedure. Instead of directly sampling keep-or-drop decision R2 for the /-th behavior token from the distribution Tr, we generate it as:
R arg max (f) + G.(i)),
}E(0,1}
where G, = - log( log U2) is a standard Gumbel distribution, and U, is sampled i.i.d. from a uniform distribution Uniform (0, 1). To remove the non-differentiable argmax operation in equation 3 the standard Gumbel-Softmal uses the reparameterization trick (Jang et al. 2016) as a differentiable approximation to relax the one-hot Ri E 1) to v E R2::
exp((logm.()) + G.()))T)
v) Eie(o.)) exp((log T (1) + G((I)/T) J E E (0,1),
where T is the temperature parameter of the Softmax. However, it remains a well-known challenge to tune the temperature in Humbel-Softmas since a low temperature will cause high variance in gradient magnitude and a high temperature will lead to an over-smoothing probability. Furthermore, a fixed temperature is not adaptive across different datasets or behaviors of each user, which incurs huge tweaking cost. Motivated by these difficulties, our propose an adaptive variant of Gumbel-Softmax that introduces the token-specific weight mechanism into the Gumbel--Sotmaxs
w ReLUU (MLP (Zmmb) E Nx1
exp(wr log Te (i) + G.
v (o,)) exp(wi log (1) + (())) E..0.
where Wt is the weight specific for each token / of each sequence. For different user behaviors, we use an MLP module to dynamically introduce the weight from the inputs Zemb: avoiding the high variance in the gradient and mitigating the over-smoothing phenomenon. Our adaptive Gumbel-Softmax can make RETR dynamically adapt to diverse datasets and user behaviors without tuning the temperature.
Hierarchical update strategy for router. The preliminary route R', sampled from π, is not a final decision. In our design, once token fails to be routed in certain block, it would permanently lose the privilege to be part of the behavior pathway in the following feed-forward procedure. This constitutes more efficient hierarchical pathway router strategy. Thus finally formulate the route R! R Nxl as the Hadamard product of R' and the pre-order route R!-1 in the (1 1)-th block:
R². -d= R' ② R1
(6)
5

---

Under review as a conference paper at ICLR 2023
Multi-head pathway attention. The standard multi-head self-attention mechanism retrieves sequen- tial characteristics by exploiting all behavior tokens, making the behavior pathway overwhelmed by the trivial behaviors. In the proposed pathway attention, the pathway router would be firstly applied to the input behavior tokens to route information. The pathway router would not pare down the number of tokens, but only the interactions between the off-pathway and on-pathway tokens, these off- off-pathway tokens may also convey contextual information.
Specifically,, for the query Q, key K, and value V in the pathway attention: the query is routed by the pathway router through token-wise multiplication between R2 and Z prevent the pathway from being overwhelmed and to force the pathway attention to attend to the behavior pathway. The key and value are the original input behavior tokens, to ensure that the contextual information from off- off-pathway behavior tokens can be captured as well:
Qm Km V
a... Z'
W Km.
Z Z'
Vm
Qm KT Vd/h
Softmax
V
where m E 21, , is the head index the multi-head self-attention; WQ.... Wk... W.... _ax are transformation matrices learned from data. Finally, the outputs {Z,, R. Sis of multiple heads are concatenated into Ž' E xxd We use Z', R' = Path-MSA (Z'-1, R) to summarize the above pathway attention. Its output is further transformed by equation I to form the final output of the I-th block Z' E R Nxd
In the prediction of the (t 1)-th behavior, only the first observable behaviors should be taken into account. To avoid a future information leak and ensure causality, we apply a causal pathway attention in that look-ahead mask is employed and all links between Q and K, (j i) are removed.
Model-agnostic pathway mechanism. is worth noting that our pathway attention is a lightweight module readily pluggable into any cansformer-based model by replacing the self-attention mechanism with our pathway attention while remaining the architecture unchanged. To verify the effectiveness of our model-agnostic pathway mechanism, we apply our pathway attention to mainstream sequential recommendation transformers: BERTRec (Sun et al. 2019), SASRec (Kang & McAuley 2018). SMRec (Chen al., 2021). S3-Rec (Zhou et al. 2020), TGSRec (Fan et al. (00Tb), and LightSANs (Fan et al. 2021a), which further enhances the performance and generalization of the these models.
3.2 PREDICTION LAYER AND TRAINING OBJECTIVE
Prediction layer. In the final layer of RETR, we calculate the user's preference score for the item in the step (L + 1) in the context of user behavior history (i++1 k ii:e) = Z where ex is the representation of item & from item embedding matrix Ez, and Z is the output of the L-th block in RETR at step t, with L being the number of RETR blocks.
Training objective. We adopt the pairwise ranking loss to optimize the RETR model parameters as:
L -= Σ log a (plie+1 (1a) p(1741 1 d1:))),
(8)
where we pair each ground-truth item i++1 with randomly sampled negative item i In each epoch, we randomly generate one negative item for each time step in each sequence. This pairwise ranking loss is widely adopted in previous literature of sequential recommendation (Kang & McAuley 2018 Zhou et al. 2022).
EXPERIMENTS
We extensively evaluate the proposed Recommender Transformer (RETR) seven intra-domain real-world benchmarks and five cross-domain benchmarks. Due to page limitation, we also include further ablation study results and visualization examples in Appendix B and Appendix C respectivvely
Intra-domain setting. We evaluate RETR on seven intra-domain datasets: Netflix, MSD, Taobao, Yelp, Tmall, Steam, and Mo All methods are trained from scratch these datasets. The

---

Under review as a conference paper at ICLR 2023
statistics of the seven datasets are summarized in Table of Appendix and the description for these datasets can be found therein. All datasets are widely used for sequential recommendation task. is notable that Netflix, MSD, Taobao and Steam are large-scale datasets.
Cross-domain setting. We evaluate the ability of RETR to capture the domain-invaricn represen- tation for sequential recommendation under the cross-domain setting. Specifically, we follow the training strategy in UniSRec (Hou et al.) 2022) to pre-train our RETR on multiple datasets "Grocery and Gourmet Food", "Home and Kitchen", "CDs and Vinyl". "Kindle Store" and "Movies and TV"", and then finetune the pre-trained RETR respectively on different target datasets "Prime Pantry", "Industrial and Scientific". "Musical Instruments", "Arts, Crafts and Sewing" and "Office Products". These datasets are all sub-categories in the Amazon review datasets (Ni et al., 2019). The detailed descriptions of source and target datasets are deferred to Appendix A
Following previous works (Kang& & McAuley, 2018), we group the interaction records by users or sessions for all datasets and sort them by the timestapss in ascending order. We split the historical sequence for each user into three parts: (1) the most recent behavior for testing, (2) the second most recent behavior for validation, and (3) all remaining behaviors for training. During testing, the input sequences contain training behaviors and validation behaviors. We filter less popular items and inactive users with fewer than five interaction records. Note that we also provide the rolling prediction results following the protocol from previous works (Chen et al.
202T in Appendix
Evaluation metrics. Following the previous literature Zhou et al. 2022 Kang & McAuley 2018) we apply top-k Hit Ratio (HR @k), top-k Normalized Discounted Cumulative Gain (NDCG@k) and Mean Reciprocal Rank (MRR) for evaluation. We report HR @ 10, NDCG@ 10 and MRR of the results. Besides, following the standard strategy in SASRec (Kang & McAuley 2018), we pair the ground-truth item with 100 randomly sampled negative items that the user has not interacted with. All metries are calculated according to the ranking of the items and we report the average score.
Baseline methods. We compare our RETR with several state-of-the-ate sequence recommendation models. Specificallyy we compare our RETR with state-of-the-arr transformer-basse
sequential recommendation models: Rec-denoiser (Chen et al. 2022). Locker (He et al. (02T). SASRec (Kang & McAuley 2018), BertRec (Sun al. | 2019). SMRec (Chen et al. [2021] p. S3-Rec (Zhou al. 2020), TGSRec (Fan et al.) 2021b) and LightSANs (Fan et al.) (021a). These methods adopt the attention mechanism to make precise recommendations. Note that Rec-denoiser (Chen et al. 2022) and Locker (He et al. [02T]] are novel transformer-basee models with learnable sparse attention. Besides, we also compare our RETR with sfate-of-the-ar art graph-based sequential recommendation methods: Jodie (Kumar et al., 2019) and TGN (Rossi et al.) 2020). We further compare our approach with some cross-domain recommendation models RecGURU (Li et al. 2022) and UniSRec (Hou) et al. 2022). All baseline methods are configured using default parameters of the original paper or optimal parameters which can produce their best results through grid search.
Implementation details. Our model is supervised by the pairwise rank loss in equation X using the ADAM (Kingma & Bal 2015) optimizer with an initial learning rate of 0.001. Batch size is set to 512. The maximum number of training epochs for all methods is set 300. All hyperparameters are tuned on the validation set. The training process is early stopped within 10 epochs. Our RETR has L = 2 layers, and each layer has h heads (the ablation study of multi-head attention can be found in Appendix [)) and dimension d is set to be 256. The maximum sequence length N is set to 200 for MovieLens IM and 100 for the other intra-domain and cross-domain datasets.
In the cross-domain setting, we further pre-train the proposed approach and baseline methods from multiple datasets following the training strategy in UniSRec (Hou et al. 2022) for 300 epochs using the default parameters from UniSRec (Hou et al. 2022). All models are pretrained without using the item ID embeddings. In the phase of finetuning on the target domain, we use the item ID embeddings and fix the backbone for all competing pretrained models. All experiments repeated three times, implemented in Py PyTorch (Paszke et al. 2019), and conducted on a single NVIDIA 3090 GPU.
4.1 INTRA-DOMAIN RESULTS
The results of different methods seven intra-domain datasets are shown in Table [ We can easily find that transformer-baut based models, SASRec (Kang & McAuley 2018), BertRee (Sun et al. 2019), SMRec (Cen et al.] 2021), S3-Rec (Zhou et al. 2020). TGSRec (Fan et al. (02 and

---

Under review as a conference paper at ICLR 2023
Table 1: Performance comparison to state-of-the-art models under the intra-domain setting. Rec- denoiser (Chen et al. 2022) uses BertRec as the backbone.
Datasets
Meric
BERT4Rec
SASRe:
SMRec
S1- Rec
SINE
TGSRec
LighSAN
Locker
Rec-denoiser
Jodie
TGN
RETR
HR
0,4792
0.4622
0,4848
0.4917
0.4902
0,4887
0.4852
0.4897
0.4913
0.4813
0.4802
0.5184
Nottix
NDCG@10 10
0,3350
0.3202
0.3492
0.3571
0.3601
0,3512
0,3441
0.3557
0,3582
0.3368
0.3318
0.3795
MRR
0.2652
0.2519
0.2725
0.2819
0.2796
0.2778
0.2785
0.2873
0.2886
0.2687
0.2612
0.3175
HR 10
0,4819
0.4766
0.5083
0.5315
0.5264
0.5137
0.4994
0.5495
0.5581
0,4825
0.4782
0.5963
MSD
NDCG@ 1D
0.4891
0.4831
0.5112
0.5381
0.5304
0.5279
0.5163
0.5495
0.5471
0.4872
0,4832
0.6012
MRR
0.3120
0.3079
0.3302
0.3494
0.3667
0,3612
0.3451
0,3686
0.3723
0.3224
0.3102
0.4025
HR 10
0.1261
0.1182
0.1272
0.1336
0.1580
0,1537
0.1590
0.1584
0.1603
0.1447
0.1421
0,1803
Taobao
NDCG@
0.0391
0.0531
0.0873
0,0745
0.0794
0,0922
0.0952
0.0582
0.0571
0.1218
MRR
0,0489
0.0436
0.0721
0.0788
0.0934
0.0802
0.0841
0.0928
0.0967
0.0628
0.0603
0.1149
10
0.7597
0.7373
0.7548
0.7597
0.7564
0,7533
0.7552
0.7492
0,7473
0.7775
Yelp
NDCG@10 10
0.4778
0.4642
0.4789
0.4937
0.4902
0.4887
0.4863
0.4935
0.4973
0.4792
0.4784
0.5169
MRR
0.4026
0.3927
0.4023
0.4107
0.4093
0,4072
0.4086
0.4189
0.4214
0.3997
0.3985
0.4378
HR 10 NDCG@10
0.8269
0.8233
0.8302
0.8352
0.8311
0.8303
0.8294
0.8349
0.8368
0.8277
0.8259
0.8513
MovieLens
0.5965
0.5936
0.6079
0.6172
0,6134
0.6081
0.6119
0.6003
0.6128
0.6009
0,5998
0.6397
MRR
0,5614
0,5573
0,5703
0,5812
0.5801
0.5734
0.5791
0,5787
0.5812
0.5651
0.5627
0.5978
HR 8 10 NDCG@
0.6196
0,6275
0.6476
0.6687
0,6512
0,6506
0.6399
0.6703
0.6729
0.6384
0.6362
0.7214
Tmall
0,5049
0.5192
0.5411
0,5372
0.5415
0,5792
0.5830
0.5307
0.5198
0.6197
MRR
0.4026
0.4804
0.4934
0.5194
0.5147
0,5121
0.5119
0.5373
0.5426
0.5003
0.4997
0.5903
HR 10
0.8792
0.8765
0,8773
0,8831
0.8892
0.8731
0.9079
Steam
NDCG@ 10
0.6283
0.6306
0.6408
0.6573
0.6502
0.6491
0.6519
0.6497
0.6523
0.6451
0.6399
0.6835
MRR
0,5883
0.5925
0.6011
0.6135
0.5972
0.6003
0.6104
0.6114
0.6159
0.5873
0.5798 0.6383
LightSANs (Fan et al. the transformer-based models have a better capacity to capture sequential behaviors of complex characteristics. These models can capture the interaction information between all previous user behaviors via the attention mechanism. Besides, the graph-based models like Jodie (Kumar et al. 2019) and TGN (Rossi et al. 2020) also achieve competitive performance. Rec-denoiser (Chen et al. 2022 and Locker (He et al. 2021 introduce novel sparse attention mechanisms, thereby achieving better performance compared with other baselines. This validates the effectiveness of the learned mask attention. While these baselines are strong competitors, our RETR can achieve state-of-the-an performance by a large margin on most datasets compared with the Rec-denoiser and Locker.
202
achieve competitive performance on most datasets, indicating that
Results on Yelp, M and Tmall. Our RETR achieves competitive performance on Yelp and Tmall. These datasets are sparse, containing less action information. Thus they have lots of noisy logged information. By effectively capturing the behavior pathway, RETR is not affected by this trivial behavior information and captures the most informative behavior representation to achieve better performance. Note that under the Tmall benchmark, RETR gains 7 7% HR @ 10, 12% NDCG@ @ 10 and 14% MRR against the strongest baseline SMRec (Chen et al. 2021). Besides, for the MoveLens IM benchmark, RETR also achieves the best performance among all competing baselines.
Results on large-scale datasets. Our RETR can consistently achieve state-of-the-art results on large-scale datasets (Netflix, MSD, Taobao, and Steam). These datasets are challenging and difficult to capture pivotal behavior pathway useful for precise recommendation from the rich but noisy user's s behaviors. Especially for the Taobao dataset, RETR gains relative improvements of 12% HR @ 10, 37 % NDCG@ @ 10 and 20% % MRR against the strongest baseline SINE (Tan al. T02T It provides evidence that RETR can achieve competitive performance in both small- and large-scale datasets. The substantial performance gains of our RETR indicate that focusing more on the behavior pathway enables RETR to capture sequential characteristics more efficiently and effectively than the vanilla self-attentic mechanism, which considers all previous user behaviors and is easily overwhelmed.
4.2 CROSS-DOMAIN RESULTT
To verify the ability of RETR to capture domain-invariant representations, we evaluate pre-trained RETR on five target datasets under the cross-domain setting. The multi-domain pre-training version of RETR, denoted as X-RETR, can be effectively transferred to new recommendation domains. We also provide the results for multi-domain pre-training version of Rec-denoiser, SASRee, SMRee, and LightSAN, denoted as X-Rec-denoiser, UniSRec, X-SMRec, and X-LightSAN respectively. Techni- cally, we follow the pretraining strategy of UniSRec (Hou et al. 2022) to train all models. As shown in Table 2, the X-RETR already achieves competitive cross domain performance, outperforming the tfate-of-the-art
method UniSRee by a large margin on most target datasets. Compared with other multi-domain pre-trained backbones, X-RETR achieves the highest performance and em-
8

---

Under review as a conference paper at ICLR 2023
powers better transferability among different backbones. Specially, X-RETR gains 12% HR @ 10 and 22.5% NDCG@ 10 compared with X-SMRec, on the Scientific benchmark. These results indicate that representations for sequential recommendation, indicating that stronger backhone is crucial in parallel with transfer-learning method for enhancing the transferabilityy. RETR can be regarded as general backbone to capture more domain-invariant representations.
RETR extract
Table Performance comparison of different competitive methods under the cross-domain setting, X-* indicates the model pre-trained on multiple datasets and finetuned on target dataset ("X" stands
for "cross-domain" ") following the training procedure of UniSRec (ouu et al. 2022
Target Datasets Meric
X-SMRec
X-LightSAN X-Locker X-Rec-denoiser
UniSRec
RecGURU
X-RETR
Scientific
HR@ 10
0.1304
0.1315
0,1312
0.1329
0.1235
0.1023
0,1459
NDCG@10
0.0706
0,0725
0,0744
0.0752
0.0634
0.0572
0,0865
HR@ 10
0.0713
0.0725
0.0741
0.0734
0.0693
0.0469
0.0847
Pantry
NDCG@10 0.0327
0.0321
0.0346
0,0338
0.0311
0.0209
0.0425
HR@ 10
0,1293
0.1278
0,1316
0,1301
0,1267
0.1113
0,1353
Instruments
NDCG@10 10
0.0792
0.0778
0,0830
0,0813
0.0748
0.0681
0.0893
HR@ 10
0.1281
0.1275
0.1283
0.1293
0,1239
0. 1084
0,1378
Arts
NDCG@10 10
0.0749
0.0738
0.0749
0.0763
0,0712
0.0651
0,0853
HR
0.1319
0.1321
0.1336
0.1343
0.1280
0.1145
0,1426
Office
NDCG@1
0.0842
0.0858
0.0857
0.0869
0.0831
0.0768
0.0998
4.3
ABLATION STUDY
Here we provide ablation study of RETR. For more ablation results, please refer to Appendix B including the ablation study of different hyperparameters and effectiveness of each model component. Further visual examples are provided in Appendix
Pathway attention towards different transformer- based models. As described before, our RETR yields ofte-off-hhearr performance all datasets. We further apply our pathway mechanism towards different transformer- based models like BERTRec (Sun et al. 2019). SMRee (Chen et al. 2021 I.
S3-Rec (Zhou et al. 2020)) TGSRec (Fan et al. (0211)), and LightSANs (Fan et al.) 2021a). In Table 3
we observe that our pathway attention can improve the performance of all baseline transformer bassdd
models substantially. RETR can be further enhanced using advanced backbones alternative the
vanilla transformers and achieve the best results among all competing methods. These results provide
strong evidences that our proposed pathway attention is modellaggosste to wansformer-based methods
and not limited to a particular architectural choice.
Table 3: Ablation study of model-agnostic pathway attention on Moviel ens. Results in each column are obtained without/with pathway attention. (f: positive improvement using pathway attention.)
Meric
BRRT4Rec
SMRec
S3-Rec
TGSRec
LightSAN
RETR
HR@ 10 0.8269/ / 0.8506 t 0.8302 008555
0,8352/ / 0.8554
0.8303 0.8497 t 0.8294 0.8529
0.8513
NDCG@ 10
0.5965 / 0.6389 1 0.6079 0.6486
0.6172/ / 0.6487
0.6081 / 0.6325 1 0.6119 1119//0..47 0.6475
0.6397
MRR
0.5614 / 0.5998 t 0.5703 0.6031
0.5812/ / 0.6052
0.5734 / 0.5973 f 0.5791 / 0.5993
0.5978
CONCLUSION
sequential recommender designed make accurate recommendations based on users' historical behaviors. However, the users' behaviors are dynamic and come in a continually evolving manner. user's current decision may only call upon the interest from the certain relevant behaviors of the past. We conclude these sequential characteristics as the behavior pathway. We propose the Recommender Transformer (RETR) with a novel pathway attention mechanism to tackle these challenges. The pathway attention mechanism develops pathway router to dynamically allocate the behavior pathway for each user and capture the evolving patterns. RETR can capture more domain invariantt representations and the pathway attention is model-agnostic and can be easily applied to a series of cransformer-basee methods. RETR achieves state-of-the-ar performance on seven intra-domain datasets and five cross-domain benchmarks for sequential recommendation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gediminas Adomavicius and Alexander Tuzhilin. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. TKDE, 2005.
Nabiha Asghar. Yelp dataset challenge: Review rating prediction. arXiv preprint arXiv:6605.0536,,
2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 1607.06400 2016.
Prateep Bhattacharjee and Sukhendu Das. Temporal coherency based criteria for predicting video frames using deep multi-stage generative adversarial networks. In NeurlPS, pp. 4268-4277 2017.
Shuging Bian, Wayne Xin Zhao, Kun Zhou, Jing Cai, Yancheng He, Cunxiang Yin, and Ji-Rong Wen. Contrastive curriculum learning for sequential user behavior modeling via data augmentation. In CIKM, 2021.
Chao Chen, Haoyu Geng, Nianzu Yang. Junchi Yan, Daiyue Xue, Jianping Yu, and Xiaokang Yang. Learning self madulladug attention in continuous time space with applications to sequential recommendation. ICML, 2021.
Huiyuan Chen, Yusan Lin, Menghai Pan, Lan Wang, Chin-Chia Michael Yeh, Xiaoting Li, Yan Zheng, Fei Wang, and Hao Yang. Denoising self-attentive sequential recommendation. In Proceedings of the 16th ACM Conference on Recommender Systems, 2022.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arxiv. preprint, 2019.
Qiang Cui, Shu Wu, Qiang Liu, Wen Zhong, and Liang Wang. Mv-rnn: A multi-view recurrent neural network for sequential recommendation. TKDE, 2018.
Jeff Dean. Introducing pathways: A nextgeneration ai architecture. Google Blog, 2021.
Xinyan Fan, Zheng Liu, Jianxun Lian, Wayne Xin Zhao, Xing Xie, and Ji-Rong Wen. Lighter and better: low-rank decomposed self-attention networks for next-item recommendation. SIGIR, 2021a.
Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip Yu. Continuous-time sequential recommendation with temporal graph collaborative transformer. In CIKM, 2021b.
Hui Fang. Danning Zhang, Yiheng Shu, and Guibing Guo. Deep learning for sequential recommen- dation: Algorithms, influential factors, and evaluations. TOIS, 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arX1v:2001.0366,, 2021.
Ruining He and Julian McAuley. Fusing similarity models with markov chains for sparse sequential recommendation. In ICDM, 2016.
Ruining He, Chen Fang, Zhaowen Wang. and Julian McAuley. Vista: A visually, socially, and temporally-aware model for artistic recommendation. In RecSys, 2016.
Zhankui He, Handong Zhao, Zhe Lin, Zhaowen Wang, Ajinkya Kale, and Julian McAuley. Locker:
Locally constrained self-attentive sequential recommendation. In CIKM, 2021.
Jonathan L Herlocker, Joseph A Konstan, Al Borchers, and John Riedl. An algorithmic framework for performing collaborative filtering. SIGIR, 1999.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session- recommendations with recurrent neural networks. arXiv preprint 1rXiv:/5/1.069:: 2015.
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji- Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In KDD, 2022.
10

---

Under review as a conference paper at ICLR 2023
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 1rXiv:1611.01 144, 2016.
Wang-Cheng Kang and Julian McAuley, Self-attentive sequential recommendation. In ICDM, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Yehuda Koren. Factorization meets the neighborhood: multifaceted collaborative filtering model. In KDD, 2008.
Yehuda Koren, Robert Bell, and Chris Volinsky, Matrix factorization techniques for recommender systems. Computer, 2009.
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In KDD, 2019.
Chenglin Li, Mingjun Zhao, Huanming Zhang, Chenyun Yu, Lei Cheng, Guoqiang Shu, Beibei Kong, and Di Niu. Recguru: Adversarial learning of generalized user representations for cross-domain recommendation. WSDM, 2022.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. NeurIPS, 2019.
Yuli Liu, Christian Walder, and Lexing Xie. Determinantal point process likelihoods for sequential recommendation. arXiv preprint arXiv:2204. / /562, 2022.
Jie Lu, Dianshuang Wu, Mingsong Mao, Wei Wang. and Guangquan Zhang. Recommender system application developments: survey. Decision Support Systems, 2015.
Jianmo Ni, Jiacheng Li, and Julian McAuley, Justifying recommendations using distantly-labele reviews and fine-grained aspects. In EMNIP-IJCNIPP, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. Search- based user interest modeling with lifelong sequential behavior data for ttccktthrouhh rate prediction. In CIKM, 2020.
Pengjie Ren, Zhumin Chen, Jing Li, Zhaochun Ren, Jun Ma, and Maarten De Rijke. Repeatnet: A repeat aware neural recommendation machine for session-based recommendation. In AAAI, 2019.
Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006. 10637, 2020.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In ICCV, 2017.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gand gated -otture-of-experts layer. arXiv preprint 1oXiv:7701006533 2017.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In CIKM, 2019.
Qiaoyu Tan, Jianwei Zhang, Jiangehao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, and Xia Hu. Sparse-interest network for sequential recommendation. In WSDM, 2021.
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In WSDM, 2018.
11

---

Under review as a conference paper at ICLR 2023
Pengfei Wang, Jiafeng Guo, Yanyan Lan, Jun Xu, Shengxian Wan, and Xueqi Cheng, Learning hierarchical representation model for nextbasket recommendation. SIGIR, 2015.
Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander Smola, and How Jing. Recurrent recom- mender networks. In WSDM, 2017.
Liwei Wu, Shuqing Li, Cho-Jui Hsieh, and James Sharpnack. Sse-pt: Sequential recommendation via personalized transformer. In Fourteenth ACM Conference on Recommender Systems, pp. 328-337, 2020.
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xic, and Tieniu Tan. recommendation with graph neural networks. In AAAI, 2019.
Session-based
Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou, Graph contextualized self-attention network for session-basee recommenda- tion. In IJCAI, 2019.
Yan, Shuo Cheng, Wang-Cheng Kang, Mengting Wan, and Julian McAuley. Cosrec: 2d convolu- tional neural networks for sequential recommendation. In CIKM, 2019.
Zeenat F Zaidi. Gender differences in human brain: review. The open anatomy journal, 2010.
Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. C/KM, 2021.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for tlick-through rate prediction. In KDD, 2018.
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang. Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Sclf-supervised learning for sequential recommendation with mutual information maximization. In CIKM, 2020.
Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. Filter-enhanced mlp is all you need for sequential recommendation. In WWW, 2022.
Yiyi Zhou, Tianhe Ren, Chaoyang Zhu, Xiaoshuai Sun, Jianzhuang Liu, Xinghao Ding, Mingliang Xu, and Rongrong Ji. Trar: Routing the attention spans in transformer for visual question answering. In ICCV, 2021.
12

---

Under review as a conference paper at ICLR 2023
A DESCRIPTIONS OF THE DATASETS
Intra-domain datasets. Here are descriptions of the seven datasets: (1) Netflix: Netflix dataset a large-scale movie rating dataset released by Netflix. (2) MSD: The Million Song Dataset
Table 4: Statistics of the intra-domain datasets.
(MSD) is large-scale, metadata-rich and open- source dataset on Kaggle. (3) Taobao: Taobao dataset (Tan al., 2021) contains user behav- iors in Taobao's 's recommender system. In ex- periments, we only use the click behaviors. (4) Yelp (Asghar 2016): Yelp is dataset for busi- ness recommendation. We only use the transac- tion records after January 1st, 2019. (5) Tmall: Tmall contains users' shopping logs on Tmall
Dataset
Users
Items
Actions
Netflix
463,435
17,769
57, $7,000,000
MSD
571,355
41,140
34,000,000
Taobao
987,994
4,162,024
100,150,807
Yelp
30,431
20,033
316,354
Moviel.ens/ln
6,040
3,416
1,000,000
Tmall
66,909
37,367
427,797
Steam
334,730
13,047
3,700,000
online shopping platform, which is from the IICAI-15 competition. (6) Steam (Kang & McAuley
2018): Steam dataset collected from large online video game distribution platform. This dataset includes 2,567,5 3,38 users, 15,474 games and 7,793,069 English reviews from October 2010 to January 2018. (7) IM this is a widely used benchmark dataset for evaluating collaborative filtering algorithms. The version we is MovieLens-In IM, which includes million user ratings.
Cross-domain datasets. We choose five categories from Amazon review datasets (Hou et al. 2022): Grocery and Gourmet Food, Home and Kitchen, CDs and Vinyl. Kindle Store, and Movies and TVV, as the source domain datasets for pre-training. For the target datasets, we choose another five categories from Amazon review datasets (Hou al., 2022): Prime Pantry, Industrial and Scientific, Musical Instruments, Arts, Crafts and Sewing, and Office Products, as target domain datasets to evaluate the proposed approach under the cross-domain setting. The detail statisties are shown in Table
Table 5: Statistics of the cross-domain datasets.
Source Dataset
Users
Items
Actions
Target Dataset
Users
Items
Actions
Food
115,349
39,670
1,027.4137
Scientific
8,442
4,385
59,427
CDs
94,010
64,439
1,118,563
Pantry
13,101
4,898
126,9626
Kindle
138,436
98,111
2,204,596
Instruments
24,962
9,964
208,926
Movies
281,700
59,203
3,226.731
Arts
45,486
21,019
395,150
Home
731,913
185,552
6,451,926
Office
87,436
25,986
684,837
B FURTHER ABLATION STUDY
Number of heads and maximum sequence length. In the left column of Table we adjust the number of heads for RETR on Yelp. We find that the performance first increases rapidly with the growth of the head number and achieves the best performance at h = We perform a similar grid search on other datasets. In the right column of Table for RETR on Yelp. As shown in T we find that the performance of our RETR first increases rapidly with the growth of the block number and achieves the best performance at N 100. We perform a similar grid search on other datasets.
adjust the maximum sequence length N
Table 6: Ablation study of the head number (Left) and the maximum sequence length for RETR (Right). Experiments are conducted on the Yelp Dataset.
Model h)
MRR
Model (# maximum sequence length
MRR
RETR (h = 1)
0.4317
RETR (N = 25)
0.4237
RETR
0.4345
RETR (N 50)
0.4294
RETR (h- 4)
0.4378
RETR (N 100)
0.4378
RETR 8)
0.4369
RETR (N -200)
0.4362
13

---

Under review as a conference paper at ICLR 2023
Effectiveness of each model component and number of blocks. In the left column of Table 7 we analyze the efficacy of each component in RETR on the Yelp dataset and have the following observations. First, we remove the pathway router module and randomly choose whether it can be maintained or dropped for each input behavior token. Removing the pathway router decreases the prediction performance lot (MRR: 0.4354 -> 0.3887), showing the necessity of learning behavior pathway effectively based on data-dependent module. Second, discarding the hierarchical update strategy for the behavior pathway also decreases the prediction performance, suggesting that this strategy is crucial for RETR to get more accurate behavior pathway. In the right column of Table 7 we adjust the number of blocks for RETR on Yelp. We find that the performance first increases rapidly with the growth of the block number and achieves the best performance at L 2. We perform a similar grid search on other datasets.
Table 7: Ablation study of (Left) the effectiienesss of each model component and (Right) the number of blocks for each RETR block. Experiments are conducted on the Yelp Dataset.
Model
MRR
Model (# number of blocks)
MRR
RETR
0.4378
RETR (L 1)
0.4197
RETR w/o Pathway Router
0.3887
RETR (L. 2)
0.4378
RETR w/o hierarchical update
0.4234
RETR (T. =
0.4342
SASRec
0.3927
RETR (L 4)
0.4340
Evaluation on efficiency. The efficiency is compared between SASRec (Kagg & McAuley, 201 8).
SINE (Tan et al. 202T) and SMRec (Chen et al. 202T) on the Yelp dataset. The computation cost measured with gigabit floating-point operations (GFLOPs) on the self-attention module with position encoding. Meanwhile, the model scale measured with parameters is also presented. As shown in Table our RETR has almost the same number of parameters or GFLOPs, compared with SASRec, indicating that our pathway router is a light-weight module. Our pathway attention does not bring more costs. It's worth noticing that the parameter scales and GLOPs of other competing transformers (apart from SASRec) are larger than RETR, but our RETR achieves higher performance. This result shows that our RETR is more efficient and effective than other competing attention-based models.
Table 8: Ablation study of (Left) the effectiveness of different temperatures; Comparison Parameters and GFLOPs (Right). All ablation study experiments are conducted the Yelp Dataset.
Model
Parameters (M)
GFLOPs
MRR
RETR
5.0
9.6
0.4378
SASRec 2018
5.0
9.6
0.3927
SINE (02T)
5.1
9.7
0.4011
SMRec 2021
5.2
9.9
0.4023
Why we need to use this pathway router in sequential recommendation? Previous sequential recommendation methods have proved that the recommender can be benefited a lot from the user's historical behaviors, even though the behavior sequence may be short. However, when meeting with the short behavior sequence, the recommender still needs to deal with various behavior pathways and can be overwhelmed by the trivial behaviors. As shown in Figure 4 we show the last 10 behaviors from random user in the Steam dataset. Further, in Figure 4 we show that the
MLP-based model, FMI--Rec (Zhou et al. 2022). is still overwhelmed by the old drifted behaviors (simulation games). To avoid the recommender being overwhelmed by the trivial behaviors, we design the pathway router to capture the pivotal behavior pathway that explains the user's preferences, whenever the behavior sequence is short or long. It's crucial to develop the pathway router to capture the behavior pathway for making precise recommendations. The pathway router is important for sequential recommendation models to avoid being overwhelmed by trivial user behaviors.
Replace the proposed pathway-based method with other sparse attention methods. We replace the proposed pathway-bas based method with two sparse attention methods: LogSparse (Li et al. 2019) and sparse attention (Child et al. 2019). As shown in Table 9 our RETR using the pathway attention
14

---

Under review as a conference paper at ICLR 2023
Action
Indie
RPG
Casual
Indie
Simulate
Simulate
Simulate
Indie
Indie
Ground Truth
Target: Indie
RETR
SASRed
FMLP-Red
Drifted Behavior Pathway
Figure 3: Illustrations of how RETR, SASRec and FMLP-Rec (Zhou et al. 2022) differs utilizing the historical behaviors of a random user in Steam Dataset. We provide the visualizations of behavior heatmaps for RETR, SASRec and FMLP-Rec of random user in Steam dataset.
remarkably outperforms other competing methods with two sparse attention methods on Tmall. These experiment results show that sparse attention methods cannot capture the exact behavior pathway and show worse performance than RETR.
Table 9: Ablation study of sparse attention methods on the Tmall dataset.
Model
NDCG@1 10
HR の 10
MRR
RETR w/ LogSparse
0.4923
0.6015 0.5873
0.4735
RETR w/ Sparse attention
0.4871
0.4620
RETR
0.6197
0.7214
0.5903
Quantitative results on whether the proposed model effectively captures a useful pathway. We give quantitive results to validate that our RETR can effectively capture various behavior pathways. Specifically, we evaluate our RETR using subset of sequences derived from the obtained behavior pathway on Tmall. Technically, we first train RETR on Tmall. For each user, we take the captured behavior pathway from our RETR the inputs to retrain RETR rather than using the whole user's behaviors. As shown Table 10 we find that using the behavior pathway as the inputs can achieve comparable results as the original RETR which uses complete user behaviors. It provides the evidence that our RETR can aptly capture the useful pathway for each user.
Table 10: Quantitative results on the Tmall dataset.
Model
NDCG@1 10
HR@ 10
MRR
RETR w/ pathway inputs RETR
0.6112 0.6197
0.7142
0.5831
0.7214
0.5903
SASRec
0.5049
0.6275
0,4804
SASRec w/ pathway inputs
0.5778
0.6812
0.5425
SASRec off-pathway inputs
0.4383
0.5697
0.4215
What's the essential difference between RETR and sequential model with attention mechanism? Our RETR has essential differences compared with sequential model with attention mechanism: (1) Our RETR designs the pathway router to capture the behavior pathway, while the other sequential models have considered it before. As shown Figure the previous selffattention mechanism mainly focuses on the recent behaviors, and cannot capture the accurate behavior pathway. Only our RETR can capture the precise behavior pathway. (2) The pathway attention for RETR the cross- attention between the pathway behavior tokens and off- pathway tokens. Our pathway cross-attention mechanism can avoid the trivial interaction between the off fff-pathwa tokens.
Why use the cross- cross-attention mechanism? We choose the cross-attention mechanism for three main reasons: (1) The cross- attenntn mechanism can force the pathway attention to attend to the behavior pathway; (2) It ensure that the contextual information from off off-pathway behavior tokens can be captured, using the original input behavior tokens as the key and value; (3) Our pathway cross- attention mechanism avoids the trivial interaction between the off-pathway tokens, while the previous
15

---

Under review as a conference paper at ICLR 2023
self-attention mechanism for sequential models be overwhelmed by the trivial information in the off- off-paawway behavior tokens. To verify our explanation, we further conduct evaluation experiments on Tmall. Specifically, train RETR Tmall, and then use the trained RETR on Tmall to capture the behavior pathway for each user in Tmall. We use the pathway behaviors and off-pathway behaviors the inputs train SASRec respectively. As shown in Table
that SASRec achieves better performance using the behavior pathway as the inputs compared with the original SASRec using the whole user's behavior as the inputs. On the contrary, the oft-pathway inputs hurt SASRec's performance seriously. Finally, our RETR achieves the best performance, indicating that the aathway-offpathwe. cross- rros-aattention is more effective than the pathway self ceff-attention.
C VISUAL EXAMPLES
Setups. We also provide qualitative visualizations for RETR, and SASRec SASRec|Kang & McAuley (2018). Technically, we use the GradCAM Selvaraju et al. (2017) to generate behavior heatmaps of the output of the last layer in each model. Three random examples of users' historical behaviors in the Steam dataset are shown in sequential order through subplots (a)-(C in Figure ??. We provide attention heatmaps of each example at the last ten time steps. We can observe three main behavior pathway characteristies corresponding to three behavior sequences respectively: (a) Casual behavior pathway: RPG games are randomly clicked by the user, while the user has a continuing interest in RPGs. (b) Correlated behavior pathway: The user has recently been interested in indie games. (c) Drifted behavior pathway: The user has recently been interested simulation games but chooses an indie game at last.
0.3
0.4
0.5
0.6
0.9
Adventure
RPG
Action
RPG
Action
Adventure Adventure
RPG
Ground Truth
Target: RPG
RETR
SASRec
DUREWES
(a) Casual Behavior Pathway
RPG
Strategy Adventure
Action
Action
RPG
Indle
Indie
Indle
Indie
Ground Truth
MENETOL
Target: Indle
RETR
SASRec
(b) Correlated Behavior Pathway
Action
Indie
RPG
Casual
Simulate
Simulate
Simulate
Indie
Indie
Ground
Truth
Target: Indie
SASRec
(c) Drifted Behavior Pathway
Figure 4: Visualizations of behavior heatmaps for RETR and SASRec of three random users in Steam dataset. They are corresponding to casual, correlated and drifted behavior pathways respectively.
Visualization results. We elaborate the three representative categories of behavior pathway in recommender systems with model-learne attention heatmaps. (1) Casual behavior pathway: As shown in Figure a)), the RGB game is randomly clicked at casual times. Our RETR can capture all the RPG casual behavior pathways, while the SASRee focuses on the incorrect recent adventure games. The SASRec cannot capture the early clicked RPG game. This phenomenon proves that our RETR can deal with the casual behavior pathway effectively. (2) Correlated behavior pathway: For the correlated behavior pathway, we also provide example which is shown in Figure 4().. The indie game is clicked many times recently, leading to the final decision to an indie game. Our RETR effectively capture the correlated behavior pathway. However, the SASRec provides higher attention scores on the recent RPG games. On the contrary, our RETR pays no attention to these wrong results, showing that has greater ability cope with the correlated behavior pathway. (3)
16

---

Under review as a conference paper at ICLR 2023
Drifted behavior pathway: As shown in Figure 4 c). The user initially interested in the indie game, but suddenly became interested in simulation games recently and chose an indie game at last. Our RETR captures the drifted behavior pathway for the indic game and has not concentrated on the old drifted pathway simulation games, while the SASRec is affected by the trivial behaviors of simulation games. These visualization results strongly show that our RETR capture various behavior pathways dynamically for each user.
D ROLLING PREDICTION ON HELD-OUT TEST USER SEQUENCES
Setup. We build a rolling prediction setup held-out test user sequences. Specifically, we split the users into train/val/test sets with the ratio 8 : 1: 1. For training users, we use the last record for prediction, while all remaining behaviors are for training. For validation and test users, the last records of each user are for the rolling prediction, and we tune hyper-parameters using the validation users. We report the average results of the last 5 records on test users for the model that achieves the best results on the validation users.
Table 11: Rolling prediction performance comparison to state-of-the-art models under the intra- domain setting on Tmall. Rec-denoiser (Chen et al. 2022) uses BertRec as the backbone.
Meric
BERTARec
SASRec
SMRee S3 S3-Rec
SINE
TGSRee
LightSAN
Rec-denoiser Locker
Jodie
TGN
RETR
HR@
0.2437
0,2415
0.2639
0.2751
0,2682
0.2617
0,2639
0.2872
0,2830
0,2622 0,2628 0,3051
NDCG@10 10
0.3873
0.3776
0.4039
0.4115
0.4073
0.4036
0.4082
0.4176
0.4144
0.4578 0.2635 0.2649 0.2985
0.3971
0.3905
MRR
0,2584
0.2553
0,2671
0,2716
0,2683
0,2675
0,2690
0.4176
0.4144
Rolling prediction results. We conduct rolling prediction experiments on the Tmall dataset. Note that we evaluate the models on the users that are not present training time. This setup more complicated than the case where test and training users overlap. As shown in Table I our RETR achieves state-of-the-art performance among all baselines. Note that Rec-denoiser and Locker also achieve competitive performance with the help of learned sparse attention, but our RETR outperforms these two methods substantially. These results show that our RETR develops an effective pathway- attention mechanism with strong capacity to make better rolling predictions on held-out test user sequences.
E COMPARISON WITH TRAR
TRAR Zhou et al. 2021 uses an efficient sparse attention mechanism for visual question answering by routing the attention span, while our RETR designed explicitly for sequential recommendation. Besides the incomparable application domains, the architecture of RETR is substantially different from TRAR, especially three main aspects:
TRAR routes the choice of adjacency masks, automatically selecting the attention span. The sparse attention mechanism in TRAR is effective for visual question answering, but it is not designed specifically to be effective for sequential recommendation. As shown in Table 12, we replace the pathway attention in RETR with the sparse attention in TRAR and conduct experiments on Tmall, MovieLens MM, and Yelp. We find that unsurprisingly TRAR achieves worse performance than SASRec, while our RETR outperforms TRAR and SASRec considerably. These results are sufficient evidence that capturing the behavior pathway is crucial for sequential recommendation and our pathway attention mechanism is more effective for sequential recommendation.
RETR is aimed at capturing the behavior pathway. It develops the pathway router to capture the behavior pathway dynamically for each user. Technically, the pathway router can embed global information from the whole behavior sequence and keep the original information from the input representation via the residual connection, while the pathway controller in TRAR only captures the global information via attention pooling. As shown in Table [3 we replace the pathway router in RETR with the pathway controller in TRAR. Our RETR
17

---

Under review as a conference paper at ICLR 2023
performs worse using the pathway controller. These results show that the pathway controller cannot capture the behavior pathway effectively and achieve worse performance.
RETR develops a hierarchical update strategy for router as described in Equ 6 As shown in Table 7 discarding the hierarchical update procedure will lead worse performance, indicating that this strategy is crucial to capture the behavior pathway effectively from the intrinsically hierarchical representations. However, the pathway controller TRAR cannot update the choice of mask hierarchically.
Table 12: Quantitative results on the Tmall, MovieLens IM, and Yelp dataset.
Dataset
Model
NDCG@ 10
HR 10
MRR
TRAR Zhou ct al.
2021
0.5007
0.6139
0,4659
Tmall
SASRec Kang & McAuley 2018
0.5049
0.6275
0.4804
RETR
0.6197
0.7214
0.5903
TRAR Zhou et al.
2021
0.5759
0.8119
0.5384
MovieLensIM
SASRec Kang & McAuley
2018
0.5936
0,8233
0,5573
RETR
0.6397
0.8513
0,5952
TRAR Zhou et al.
2021
0.4439
0.7186
0,3883
Yelp
SASRec KKng & McAuley
2018
0.4642
0.7373
0,3927
RETR
0.5169
0.7775
0,4378
Table 13: Ablation on the pathway attention mechanism on the Tmall dataset.
Model
NDCG@ 10
HR 10
MRR
RETR pathway controller RETR
0.5613
0.6526 0.7214
0.5408
0.6197
0,5903
F ABLATION STUDY FOR ADAPTIVE GUMBEL-SOFTMAX
The temperature parameter T a crucial hyperparameter for the standard Gumbel-Softmax. A fixed temperature cannot be adaptive across different datasets users. It is widely-knows to be uncasy to tune the temperature parameter, in that a lower value may lead to high variances in gradients and a higher value may lead to over-smoothing probabilities. To mitigate these technical issues, propose a novel adaptive Gumbel-Softmal mechanism to eliminate the need of temperature tuning, which can produce token-specific weights automatically adjusted varying behaviors of each user.
Table 14: Quantitative results on the Tmall dataset. 4_" indicates failure case of model training.
Model
NDCG@1 10
HR @ 10
MRR
RETR w/ standard Gumbel-Softuan (r = 2)
0.6049
0.7084
0.5787
RETR w/ standard Gumbel-Softuaal (T 1)
0.6084
0.7105
0,5803
RETR w/ standard Sumbel-Softmax (T -0.8
0.6103
0.7138
0,5822
RETR w/ standard Humbel-Softman (T =0.6)
0.6095
0.7129
0,5817
RETR w/ standard Humbel-Softuaa (T =0..4)
RETR w/ standard Gunbel-Softmax (T =0.22)
RETR w/ Gumbel-Softmax (lemperature annealing)
0.6093
0.7133
0.5814
RETR w/ adaptive Gumbel-Softmax
0.6197
0.7214
0.5903
As shown in Tables 14 and 15
we find that RETR with the standard Gumbel-Softmax achieves the highest performance in different datasets at different temperatures (r = 0.8 for Tmall and 0.6 for Yelp). These results show that fixed temperature is not adaptive across diverse datasets. However, lower temperatures (T = 0.2, 0.4) cause the failure model training for RETR due to the high variance in gradients. Higher temperatures 1,2) may perform worse because of the
18

---

Under review as a conference paper at ICLR 2023
Table 15: Quantitative results the Yelp dataset. s_a indicates failure case of model training.
Model
NDCG@10 10
HR@ 10
MRR
RETR w/ standard Gumbel-Softmaal (r = 2)
0,5113
0.7695
0.4328
RETR w/ standard Gumbel-Softnan (T 1)
0,5124
0.7719
0,4345
RETR w/ standard Gumhel-Softma =0.8)
0.5136
0.7730
0.4354
RETR w/ standard Gumhel-Softmar (r =00.6)
0.5152
0.7749
0.4360
RETR w/ standard Gumhel-Softma (T =0..4)
RETR w/ standard Gumbel-Softuae (T = -.22)
RETR w/ Gumbel-Softmax (temperature annealing)
0.5134
0.7727
0,4350
RETR w/ adaptive Gumbel-Softmax
0.5169
0.7775
0.4378
over-smoothing probabilities in Gumbel-Softmax. It proves difficult to tune the temperature for the standard
Previous work like TRAR (Zhou al. 202I develops schedule that starts with high temperature and gradually anneals to small but non value, This schedule can make the training more stable, but it cannot achieve the best performance adaptively across different datasets.
In contrast, RETR with the proposed adaptive Gumbel-Softmax featured by the token-specific weight mechanism achieves the best performance compared with the standard Gumbel-Softmal under different temperatures. These results indicate that the adaptive Gumbel-Softmax is much more effective in capturing the behavior pathway and can be dynamically adapted to different datasets without tuning the temperature. This adaptive mechanism can simultaneously overcome the over-smoothing phenomenon and dynamically avoid the high variances in gradients.
G VISUALIZATION FOR HIERARCHICAL UPDATE
0,6
Adventure
VR
RPG
Action
RPG
Action
RPG
Adventure Adventure
RPG
Ground Truth
Target: RPG
RETR (L=1)
RETR (L=2)
DUIGEW'S
Visualization for hierarchical update of RETR. We provide the visualization of behavior heat maps RETR for different blocks of random user in Steam dataset.
Figure 5:
We further explore how the behavior pathway updates hierarchically via the visualization in Figure 5 Technically, we use the GradCAM Selvaraju et al. (2017) to generate behavior heat maps of the output of the first and last layer in RETR, respectively. As shown in the Figure we observe that our RETR can update the captured behavior pathway hierarchically, dropping the drifted behavior pathway Adventure in the last layer (row 2) from the first layer (row 1). This visualization shows that our RETR can determine the practical behavior pathway via the hierarchical update procedure.
19


Reviews:
[{"review_id": "ZSwKAvbkVm", "reviewer": "Reviewer_k31G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new Transformer-based RecSys model with learnable attention masks (namely behavior pathway) based on global information of the input to each layer, which limit input queries to only certain tokens. It is claimed that through this routing of attention the model can focus more on behaviors that are relevant for next item prediction. It can be used on many existing Transformer models. Empirical results on 7 recsys datasets show outperformance of the model compared to several full-attention Transformer models and graph-based models. They also experiment with cross-domain setting where RETR shows improvement with pretraining-finetuning regime.", "summary_of_the_review": "The paper proposes a new way of learning attention masks which seems to achieve consistent outperformance against several standard baselines on multiple datasets. However the lack of discussion and comparison to related work on sparse attention/learnable mask makes the conclusion incomplete, and some of the claims lack concrete support. Therefore, it may benefit from a significant revision.", "strength_and_weaknesses": "Strength:\n\n- The authors propose a novel modification of vanilla Transformer for RecSys by introducing learnable ‘behavior pathways’ that masks the attention adaptively to current user sequence history.\n- It compares with multiple strong baselines for sequential recommendation and show outperformance against Graph based models and several Transformer baselines. The experiments are extensive in terms of dataset coverage. \n\nWeakness:\n\n- The method is very closely related to the study of sparse attention for Transformer where many prior work exists (e.g. Informer, Denoising self-attention).However literature review and discussion in this area is missing. Similarly for the benchmarking experiments, comparison to other Transformer based RecSys with learnable sparse attention / constrained attention masks, such as LOCKER (https://cseweb.ucsd.edu/~jmcauley/pdfs/cikm21.pdf) and Denoising self-attention (https://dl.acm.org/doi/abs/10.1145/3523227.3546788) is lacking as well. \n- There is some concern regarding the computational efficiency, as it seems that only one item prediction can be made per forward pass (as the attention mask will change with `t`). This makes it less efficient in training compared to SASRec/BERT4REC, in which losses w.r.t multiple positions can be computed in one forward pass. Any discussion on the efficiency/memory cost of training?\n- The train/test/valid setup in SASRec is not very convincing, as it only test on one last interaction per user. It might be more natural to split by users and evaluate rolling prediction performance on held-out test user sequences. Also using negative sampling of size 1 is very inefficient. One could achieve much better results by increasing negative sampling size.\n- The Cross-domain experiments is hard to interpret. As the pretraining and finetuning method uses descriptive text information instead of item ID embedding, which is not used for the other non-pretrained models. It is difficult to judge whether the boost is due to the use of textual information.\n- It is not intuitive to this reviewer why attention mask learned from global average pooling can captures the three behavior pathways listed in figure 1, as it is most likely to pick up popular items in the given sequence. The case study in figure 3 is rather cherry picked and lack generalizability as RPG is the most frequently appeared items in this example, and almost all attentions are given to RPG. From this example it seems RETR fails to capture temporal locality and ignores the most recent items, which is a bit contradictory to the claims that the model captures sequential information.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Lej_tf-dHG", "reviewer": "Reviewer_pNaA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents behavior pathway identification approach that can distinguish a certain user behavior thread from other random actions. Specifically, by applying Gumbel-Softmax on Pathway prediction functions, the proposed approach is able to augment query representations of the multi-head attention model such that prediction could focus on a long-lasting user preference. Experiments were conducted on multiple public datasets and showing promising performance improvement.", "summary_of_the_review": "Overall, I suggest the authors should make some improvement on the algorithm description and clarify some confusing points listed above. I believe the paper is interesting in general and I like the high level idea of this work. \n\nQuestions:\n1. the paper seems implicitly assuming binary pathway identification (useful or not). Is it possible to extend this idea into tracking multiple user preference threads?\n", "strength_and_weaknesses": "1. Interesting and novel idea on identifying behavior pathway.\n2. Poor descriptions and demonstrations that causing some confusions in the writeup.\n3. Comprehensive experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ERs_gEslOnY", "reviewer": "Reviewer_oeeP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel pathway attention for sequential recommendation, which can prevent the final predictions from being overwhelmed by trivial behaviors in the user behavior sequence, and instead makes the model focus on the pivotal behaviors. The key part in the pathway attention is the router which relies on the Gumbel-Softmax trick to fulfill the binary routing. The authors have conducted extensive and comprehensive experiments to support their claims. The performance gains brought by the proposed attention are significant. \n", "summary_of_the_review": "The motivation of this paper is interesting and rational. The proposed pathway attention is easy to implement. Extensive experiments have been conducted to support the claims. However, my main concern about this paper is its novelty and technical contribution. Though the proposed pathway attention looks effective, the used techniques (i.e., Gumbel-Softmax) are not exciting. Particularly, a similar technique known as Transformer Routing has been introduced in [1]. I suppose the authors might be more-or-less inspired by it but the paper is not cited. \n\nreference\n[1]. TRAR: Routing the Attention Spans in Transformer for Visual Question Answering\n", "strength_and_weaknesses": "Strength:\n(1). This paper has a good and interesting motivation. It is rational to borrow ideas from the pathway architecture.\n(2). This paper is well-organized and easy to read.\n(3). The authors have conducted extensive and comprehensive experiments to support their claims.\n\nWeakness:\nOverall, this is a good paper that should not receive too much criticism. Everything looks OK. However, it is not a paper that can surprise the readers. The main issue is the technical novelty. Using Gumbel-Softmax to implement pathway routing is not new or exciting. A previous work known as Transformer Routing has a very similar solution though it is developed for the field of computer vision. I am also wondering if the training is stable since Gumbel-Softmax often causes high variances in gradients.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject