Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new sequential recommendation model with behavior pathways, effectively capturing specific evolving item patterns for each user. The pathway attention using learned binary routes can effectively remove unnecessary items for a given user sequence. Experimental results show that the proposed model significantly outperforms existing models on seven benchmark datasets and achieves state-of-the-art performance. Overall, the idea of using behavior pathways is interesting to me, but it has several weak points, especially in evaluating the proposed model. 
strengths_and_weaknesses: Strengths
- (Originality) It is interesting to introduce behavior pathways in long user sequences. Besides, this paper utilizes a router to choose item patterns selectively.
- (Originality) Because the idea of using the router is a model-agnostic property, it can be effectively applied to various sequential recommender models.
- (Clarity) It is well-written and easy to understand.


Weaknesses
- (Quality) Although the evaluation is extensive, the proposed model has not been compared with recent sequential recommendation models. Please refer to the following references.
[1] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, Ji-Rong Wen, “S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization,” CIKM 2020
[2] Qiaoyu Tan, Jianwei Zhang, Jiangchao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, Xia Hu, “Sparse-Interest Network for Sequential Recommendation,” WSDM 2021
[3] Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, Philip S. Yu, “Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer,” CIKM 2021
- (Quality) In Section 4.3, the visualization result shows that the proposed model effectively captures the pathways in user sequences. However, it is wondering if this result is generalized for other cases. It is necessary to show a quantitative result on whether the proposed model effectively captures a useful pathway. One possible evaluation is that the proposed model also shows a comparable result using a subset of sequences derived from a pathway.
- (Quality) In Table 3, RETR(L=1) is similar to SASREC(L=1). However, RETR(L=1) shows a better performance than SASRec(L=1). It is wondering if the proposed model is fairly compared with the existing model.
- (Clarify) There are some minor typos.
- 1p 37line: use’s -> user’s
- 2p 41line: second -> first
- 2p 44line: first -> second

questions: - Q1) Is V^t in Eq. (4) the same as hat(R^l) in Eq. (5)? If so, please clarify this equation.
- Q2) The proposed idea seems to be model-agnostic. Could you apply the pathway module for other sequential recommender models, e.g., BERT4Rec?
- Q3) In Figure 1, the authors mention various behavior pathways. Does the proposed model capture various pathways or focus on capturing drifted behavior pathways?

limitations: This paper does not address the negative societal impact. However, this paper seems not to have any negative impact.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper, the authors propose the Recommender Transformer (RETR) with a novel Pathway Attention mechanism. RETR can dynamically plan the behavior pathway specified for each user, and sparingly activate the network through this behavior pathway to effectively capture evolving patterns useful for recommendation.
strengths_and_weaknesses: Strength:

1. The paper is well written and easy to follow. Basically, the authors try to use the behavior pathway in the Transformer.

2. The experimental results are good compared with the baselines.


Weakness:
1. The novelty of this work is not very high. The mechanism of router is wildey used in the MoE-style Transformer. It seems that this work applied it to the sequential recommendation. The Gumbel-softmax is also widely used to optimize the discrete binary variable.

2. It is unclear why need use this swith rounter in sequential recomemndation. Usually, the length of users' behaviors is very short, like less than 25 for most of transaction. Do we really need this router in the Transformer? 

Actually, recent work verify that a simle MLP can outperform the Transformer in the sequential recommendation.

Zhou, Kun, et al. "Filter-enhanced MLP is all you need for sequential recommendation." Proceedings of the ACM Web Conference 2022. 2022.

The motivation of this work is thus not very strong. 

3. In the experiment, "we pair the ground-truth item with 100 randomly sampled negative items that the user has not interacted with." Does this raise a smapling bias?
questions:  It is unclear why need use this swith rounter in sequential recomemndation. Usually, the length of users' behaviors is very short, like less than 25 for most of transaction. Do we really need this router in the Transformer? 

Actually, recent work verify that a simle MLP can outperform the Transformer in the sequential recommendation.

Zhou, Kun, et al. "Filter-enhanced MLP is all you need for sequential recommendation." Proceedings of the ACM Web Conference 2022. 2022.


3. In the experiment, "we pair the ground-truth item with 100 randomly sampled negative items that the user has not interacted with." Does this raise a smapling bias?
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a recommender transformer with a pathway attention mechanism. It is characterized by its ability to capture three types of user pathways and predict user action sequences with high accuracy. The paper demonstrates the usefulness of the proposed method in comparison with several state-of-the-art methods using several types of real data.
strengths_and_weaknesses: - Strengths
  - Starting with the actual example in Figure 1, the motivation for proposing the method is well explained, making it easy to understand the content of the proposed technique.
  - The authors have conducted prediction experiments using seven different behavioral log datasets from various sites. They compared the accuracy of the proposed method with seven existing methods and confirmed that the proposed method outperforms them.
  - Experiments are conducted using real data, not artificial data.
- Weaknesses
  - The meaning and boundaries of the three types of pathways are vague. The definition of each should be clearly stated. Also, are these three types sufficient?
For example, what is the difference between the Correlated behavior pathway and the Drifted behavior pathway? They seem to have similar properties in the local and short-term. The types and definitions of pathways should be better justified, such as providing references that support the authors' definitions.
  - Although the experimental results quantitatively demonstrate the effectiveness of the proposed method, the architecture in Figure 2 is straightforward and somewhat lacking in technical novelty.
questions: The authors state that user pathways fall into three categories (correlated, casual, and drifted behavior pathways), but on what evidence? Justification is needed.
limitations: There is no mention in the paper of negative impacts on society. Also, I can't think of any.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose Recommender Transformer (RETR) with a Pathway Attention mechanism which can generate the behavior pathway hierarchically and capture the evolving patterns dynamically through the pathway. The key design is a learned binary route to prevent the behavior pathway from being overwhelmed by trivial behaviors. The authors also show RETR has high accuracy and efficiency compared with other self-attention or transformer based sequential recommendation methods through experiments.
strengths_and_weaknesses: Pros:
1. The paper is generally easy to follow.
2. The idea of using pathway in recommendation algorithms seems to be new. 
3. The authors conducted extensive experiments on seven datasets to prove RETR can make accurate recommendations.

Cons:
1. The essential difference between RETR and other sequential recommendation methods with attention mechanism is not clear. It seems to me that RETR not only utilizes the “on-pathway tokens” but also leverages the “off-pathway tokens” as they “also convey contextual information” and the difference between the two kinds of tokens is their weight. However, in SASRec and all the other self-attention-based methods, different tokens already have varying attention weights, so that more important historical items may have higher attention weights and less important historical items may have lower attention weights. I don't understand why is it necessary to design a pathway.
2. The targeted problem is not new, and several recent works have been proposed to address the same issue of self-attention. Apart from reference [6] in the paper, there are several others trying to improve self-attention-based recommendation methods. For example, the LightSANs work published in SIGIR ‘21. It would be useful to also conduct experimental comparisons with these more recent baselines in addition to [6]. Besides, there are many sparse attention works in the literature and it is interesting to replace the proposed pathway-based method with other sparse attention methods to see if pathway-based method is superior.
3. Besides self-attention-based methods, other types of sequential recommendation methods, e.g., temporal graph-based sequential recommendation methods, have also achieved state-of-the-art results. Considering that attention weights can be regarded as edge weight in a graph, it might be useful to compare with some recent temporal graph based sequential recommendation methods, e.g., Jodie in KDD 19 and TGN in ICML 20. Especially, TGN also adopted attention in its model.
4. There is no evidence/experiment to show that RETR will not be overwhelmed by trivial behaviors. As this is the main claim of the paper, it should be necessary to have more justifications. The case study in Section 4.3 seems to be a cherry-pick result.
5. All the datasets that used in this paper are relatively small. Larger datasets such as MSD and Netflix should be more desirable.

Minor issues:
1. “in the second line” in Line 41 and “in the first line” in Line 44 should be exchanged. 
2. In column “Actions” of Table 1, the commas are not consistent. 
3. The authors should use “GRURec” or “GRU4Rec” consistently in the paper to avoid misunderstandings.
4. The ablation study of multi-head attention is missing.

questions: - What’s the essential difference between RETR and sequential model with attention mechanism when putting aside the concept of pathway?  
- In the introduction, the authors list three kinds of behavior pathways, so how can RETR capture them to make precise recommendations in each case? 
- In Line 172, what’s the meaning of  “lose the privilege to be part of the behavior pathway”? Since the off-pathway tokens are also considered, it seems that all items will always be considered whether they are in the pathway or not. 

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: metareview: This paper presents Recommender Transformer (RETR) with a pathway attention mechanism that can dynamically zeroing-out the interactions (e.g., the trivial/noisy ones) in transformer-based sequential recommender systems. Extensive experimental results demonstrate the effectiveness of the proposed architecture. 

Overall this paper received mixed reviews with borderline scores. The reviewers raised concerns around baselines and evaluations, some of which the authors promptly addressed in the revision during the rebuttal period. I also read the paper in details myself. I do agree with some of the concerns from the reviewers but I don't think a method needs to beat every other published papers to be published (and I think the current baselines are more than thorough enough). My biggest complaint about the paper is around the writing, specifically, how the proposed idea is presented. 

This paper tries to tackle an important question, which is that in sequential recommendation, not every interactions are useful in helping predict future interaction. The self-attention mechanism in transformer kind of addresses this problem but in a more "softer" fashion with attention weights. This paper presents a simple yet effective method to introduce a pathway mechanism that adaptively zeroing-out some of the interactions via a binary pathway router. In order to train such a model end-to-end, Gumbel-softmax sampling is utilized.

The most important part of the contribution to me is that this is an improvement to the transformer architecture, as opposed to a new model which is what this paper's writing suggests -- the proposed approach is effectively model-agonistic and doesn't marry to a particular loss function or finer-grained architectural choices (number of layers, etc.). Currently there are many baselines in the paper, but each made some different model/architecture choices, which could contribute to the difference in performance (or not, but we wouldn't know). An ideal evaluation should have been to take all the transformer-based baselines that are currently in the paper, add this pathway mechanism without changing anything else, and show that the results improved over the transformer architecture. In this way, we know the improvements are exactly coming from introducing the pathway. The authors might argue some of the current results are already supporting this argument, but my point is to emphasize this point very explicitly rather than leaving it for the readers to infer. 

From what I read in this paper, I truly believe this pathway idea has its potential. Therefore, I would especially want the authors to further refine the presentation to better convey the idea, which in turn will hopefully increase the impact of this paper once it is eventually published. 

Some minor comments:
* The way the paper is currently written seems to suggest there are only three types of pathways and the network is capable of capturing all of them. I am personally not a big of fan of over-interpreting what a neural net is trying to do. Therefore, I wouldn't overly focus on the characterization of different pathways and only show the qualitative examples at the end as a high-level demonstration.
* In Eq 2 "softmax" should really be "sigmoid" if a 0-1 prediction is made there. Then the following line "logit" is probably not the right word here. 
* The qualitative examples at the end (figure 3) can be more carefully examined/labeled. For example, the current categorization is quite ambiguous -- "Indie" refers to the type of developers while "JPG" refers to the genre of the game, they are certainly not mutually exclusive.  
confidence: Certain
Decision: Reject