Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a new Transformer-based RecSys model with learnable attention masks (namely behavior pathway) based on global information of the input to each layer, which limit input queries to only certain tokens. It is claimed that through this routing of attention the model can focus more on behaviors that are relevant for next item prediction. It can be used on many existing Transformer models. Empirical results on 7 recsys datasets show outperformance of the model compared to several full-attention Transformer models and graph-based models. They also experiment with cross-domain setting where RETR shows improvement with pretraining-finetuning regime.
strength_and_weaknesses: Strength:

- The authors propose a novel modification of vanilla Transformer for RecSys by introducing learnable ‘behavior pathways’ that masks the attention adaptively to current user sequence history.
- It compares with multiple strong baselines for sequential recommendation and show outperformance against Graph based models and several Transformer baselines. The experiments are extensive in terms of dataset coverage. 

Weakness:

- The method is very closely related to the study of sparse attention for Transformer where many prior work exists (e.g. Informer, Denoising self-attention).However literature review and discussion in this area is missing. Similarly for the benchmarking experiments, comparison to other Transformer based RecSys with learnable sparse attention / constrained attention masks, such as LOCKER (https://cseweb.ucsd.edu/~jmcauley/pdfs/cikm21.pdf) and Denoising self-attention (https://dl.acm.org/doi/abs/10.1145/3523227.3546788) is lacking as well. 
- There is some concern regarding the computational efficiency, as it seems that only one item prediction can be made per forward pass (as the attention mask will change with `t`). This makes it less efficient in training compared to SASRec/BERT4REC, in which losses w.r.t multiple positions can be computed in one forward pass. Any discussion on the efficiency/memory cost of training?
- The train/test/valid setup in SASRec is not very convincing, as it only test on one last interaction per user. It might be more natural to split by users and evaluate rolling prediction performance on held-out test user sequences. Also using negative sampling of size 1 is very inefficient. One could achieve much better results by increasing negative sampling size.
- The Cross-domain experiments is hard to interpret. As the pretraining and finetuning method uses descriptive text information instead of item ID embedding, which is not used for the other non-pretrained models. It is difficult to judge whether the boost is due to the use of textual information.
- It is not intuitive to this reviewer why attention mask learned from global average pooling can captures the three behavior pathways listed in figure 1, as it is most likely to pick up popular items in the given sequence. The case study in figure 3 is rather cherry picked and lack generalizability as RPG is the most frequently appeared items in this example, and almost all attentions are given to RPG. From this example it seems RETR fails to capture temporal locality and ignores the most recent items, which is a bit contradictory to the claims that the model captures sequential information.

clarity,_quality,_novelty_and_reproducibility: The paper is overall clear and well-written. It improves existing model by introducing another sparse attention mechanism, which is novel but not completely new.
summary_of_the_review: The paper proposes a new way of learning attention masks which seems to achieve consistent outperformance against several standard baselines on multiple datasets. However the lack of discussion and comparison to related work on sparse attention/learnable mask makes the conclusion incomplete, and some of the claims lack concrete support. Therefore, it may benefit from a significant revision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents behavior pathway identification approach that can distinguish a certain user behavior thread from other random actions. Specifically, by applying Gumbel-Softmax on Pathway prediction functions, the proposed approach is able to augment query representations of the multi-head attention model such that prediction could focus on a long-lasting user preference. Experiments were conducted on multiple public datasets and showing promising performance improvement.
strength_and_weaknesses: 1. Interesting and novel idea on identifying behavior pathway.
2. Poor descriptions and demonstrations that causing some confusions in the writeup.
3. Comprehensive experiments.
clarity,_quality,_novelty_and_reproducibility: The paper is not quite well written unfortunately.
E.g. 
1. The MLP function applied to R and Z, are they same? are inputs all matrixes? is that row or column-wise MLP? is there any flattening or something there to make a 2-d input consumable by the function? 
2. For gambel-softmax sampling, what is the actual temperature parameter used in the paper? 1 seems over smoothing I presume. Figure 2 is not quite understandable. 
3. How to compute Hadamard product between X and R, if R is in shape (2, N), but Z is (d, N)?  something is missing here.

Novelty seems tricky here. In terms of identifying pathway, I think it is novel. But for the architecture used for identifying the pathway, I don't think it is novel as it is simply Gumbel-softmax with multiple layers of repeating.

Due to the poor quality of paper presentation, I have very low confidence to reproduce the paper. I think the authors need to make some polishing here to clarify the details further.

 

summary_of_the_review: Overall, I suggest the authors should make some improvement on the algorithm description and clarify some confusing points listed above. I believe the paper is interesting in general and I like the high level idea of this work. 

Questions:
1. the paper seems implicitly assuming binary pathway identification (useful or not). Is it possible to extend this idea into tracking multiple user preference threads?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel pathway attention for sequential recommendation, which can prevent the final predictions from being overwhelmed by trivial behaviors in the user behavior sequence, and instead makes the model focus on the pivotal behaviors. The key part in the pathway attention is the router which relies on the Gumbel-Softmax trick to fulfill the binary routing. The authors have conducted extensive and comprehensive experiments to support their claims. The performance gains brought by the proposed attention are significant. 

strength_and_weaknesses: Strength:
(1). This paper has a good and interesting motivation. It is rational to borrow ideas from the pathway architecture.
(2). This paper is well-organized and easy to read.
(3). The authors have conducted extensive and comprehensive experiments to support their claims.

Weakness:
Overall, this is a good paper that should not receive too much criticism. Everything looks OK. However, it is not a paper that can surprise the readers. The main issue is the technical novelty. Using Gumbel-Softmax to implement pathway routing is not new or exciting. A previous work known as Transformer Routing has a very similar solution though it is developed for the field of computer vision. I am also wondering if the training is stable since Gumbel-Softmax often causes high variances in gradients.

clarity,_quality,_novelty_and_reproducibility: This paper is of high quality in terms of Clarity and quality. The proposed pathway attention is easy to understand and implement. The experiments are comprehensive and the supplementary material has introduced the experimental settings and investigated all aspects of the model. My main concern is the technical novelty.

summary_of_the_review: The motivation of this paper is interesting and rational. The proposed pathway attention is easy to implement. Extensive experiments have been conducted to support the claims. However, my main concern about this paper is its novelty and technical contribution. Though the proposed pathway attention looks effective, the used techniques (i.e., Gumbel-Softmax) are not exciting. Particularly, a similar technique known as Transformer Routing has been introduced in [1]. I suppose the authors might be more-or-less inspired by it but the paper is not cited. 

reference
[1]. TRAR: Routing the Attention Spans in Transformer for Visual Question Answering
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper presents recommender transformer (RETR) that can dynamically zeroing-out the interactions in a so-called behavior pathway mechanism. The basic intuition is that many of the interactions in a sequential recommender system are noisy/trivial and don't help with making prediction for the next item. Behavior pathway is a way to mask out some of the interactions in the query (as in query-key-value in a standard transformer) before applying the multi-head self-attention. The discrete mask is approximated with an adaptive Gumbel-softmax. Extensive experimental results demonstrate the effectiveness of the proposed model. The authors also show that the proposed behavior pathway can be integrated into other transformer-based models which can lead to better performance. 

Strengths: All the reviewers agree that the proposed pathway attention mechanism on top of the vanilla transformer makes intuitive sense. The experimental results are extensive and show that the introduced pathway is effective compared with other approaches. 

Weaknesses: All the reviewers have concerns regarding the novelty of the proposed approach. It is true that the work in this paper is relatively incremental but I don't think this alone should be the reason to reject a paper. Reviewer k31G raised some interesting questions, majority of which I think the authors have addressed. I have read the paper myself and engaged with the authors on some of the details about the computational efficiency of the proposed pathway router (also one of the concerns from reviewer k31G) and it is clear that there is still work to be done on this front. The authors did provide many additional results (and even introduced a slightly different architecture) over the last few days but I think: 1) given the amount of changes that happened in the last few days (and overall during the rebuttal period), I would feel more confident if it goes through another round of actual review; 2) I don't particularly like how a slightly different architecture needs to be employed to address different types of prediction problem. I encourage the authors to explore ways to unify things rather than making it overly-complicated. Finally, this is not a major issue but I think the paper right now lacks a bit of focus -- it tries to do too many things within the very limited space (especially considering the additional experiments that happened in the last few days) and it can sometimes dilute the actual message. I hope the authors can come up with a coherent narrative when preparing for revision. 


justification_for_why_not_higher_score: I like the idea in this paper even though it's a bit incremental (which is also the major complaints from all the reviewers). The reason that I didn't give it a higher score is because I think there is still work to be done on the computational efficiency side as described in my main review as well as the discussion thread with the authors.
Decision: Reject