Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper talks about a simplification of transformer architectures. The original transformers employ a dense attention between all tokens in a sequence to focus on sequence comprehension. In this work, however, the focus is on the retrieval of the last item. It therefore only evaluates the attention between the proposed candidate item and the preceding items in user history. The other novelty as I see is the replacement of Residual connections between layers with a GRU layer, though the impact is less discussed. The work is successfully deployed in industrial recommender systems, showing significant improvements in the click-through rate.
main_review: The paper is very clear and the proposals are quite practical. However, I worry that there might not be sufficient academic novelties in this work. Particularly, by giving up the self-attentions within the user's own browsing history, the new method would have fundamental limitations in the comprehension of user intents with long histories. The original self-attention layers would summarize exponentially longer sequences as more layers are added, but the target-attention mechanism only does so linearly. Perhaps this is okay for recommender systems, when there are not much complexity in the sequential patterns, but the contribution is rather technical than academic.

Additionally, the proposed method has another limitation that it is only suitable for reranking tasks when there is a short list of candidates, instead of end-to-end retrieval tasks. This is because every candidate item has to be evaluated through a complex neural network, which is in contrast to traditional transformer models such as SASRec, when the decoder would potentially support the retrieval tasks through approximate max-inner-product search. This follows the same limitation as DIN and I thus worry that its cross-disciplinary impact could be limited.
summary_of_the_review: Interesting practical work and great industrial success, but limited academic contributions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on tackle the sequential recommendation task, where the authors proposed Iterative Memory Network (IMN), an end-to- end differentiable framework for long sequential user behaviour modeling. The main contribution of the paper is the IMN framework with efficient in memory and complexity. Specifically, the authors proposed iterative memory updates module with the multi-way attention and memory enhancement module.
main_review: Strength

* S1: The writing is clear, easy to follow. 
* S2: The authors conducted good experiments, where they also used industrial dataset for comparison. (It will be great if the authors could also provide comparison with current production model(s)).


Weakness:

* W1: The paper has very limited contributions. The idea of memory based attention networks is not new [1, 2]. 
* W2: The paper has poor analysis (Section 5.4). The paper needs to have deeper analysis in explanation of model architecture and not only performance. For example, besides the benefits of memory based, which were already stated in few papers such as [1, 2], the authors should provide in details about the advantages of the proposed framework (e.g., from theoretical perspectives) compared to previous baselines
* W3: The results in Figure 2 does not seem to be convincing, since the performance on AmazonBooks and AmazonMovies decrease dramatically after only few iterations. 


[1] Latent Relational Metric Learning via Memory-based Attention for Collaborative Ranking. WWW 2018. 
[2] Signed Distance-based Deep Memory Recommender. WWW 2019.

summary_of_the_review: The paper is good in terms of writing and conducting experiments. However, the contributions are very limited. This paper requires more in-depth analysis about the model performance, ablation studies, run-time/real-time comparison, etc. A huge drop in the AUC metric after only a few iterations in Figure 2 also raise a concern about the performance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an "iterative memory network" for long user sequence modeling such as those for ctr prediction in ads and recommender systems. The basic idea is to use a iteratively updated memory vector to interact with items in the user sequence and target item, so there is higher-order interaction intra-sequence, without the need to use self-attention. Experiments are conducted on two offline datasets comparing with some attention and memory network based methods.


main_review: This paper proposes an "iterative memory network" for long user sequence modeling such as those for ctr prediction in ads and recommender systems. The basic idea is to use a iteratively updated memory vector to interact with items in the user sequence and target item, so there is higher-order interaction intra-sequence, without the need to use self-attention. Experiments are conducted on two offline datasets comparing with some attention and memory network based methods.

Strength
- Modeling long user sequence is an important problem and is a popular topic recently.
- The proposed model is overall sound and seems to be a reasonable way for this problem.

Weakness
- The major weakness of this paper is evaluation. First, the baselines are relatively out dated. The most recent baseline was in 2019. As mentioned above, the problem is an actively researched area recently, and the DIN paper drew ~400 citations in the last 2 years. The experimental sections need more recent and competitive baselines to compare with. Otherwise the experiments look like what a paper would do 2 years go. Some important related work or baselines seem missing, for example, retrieval based methods such as "User Behavior Retrieval for Click-Through Rate Prediction". Work along this line also try to tackle efficiency issues for long-sequence modeling and need to be discussed and compared with. Second, only offline experiments are done on two datasets. The industrial dataset does not have much details and it is not clear what the impact is. Many papers in this area, such as DIN, performed actual online validation so this paper is on or below the borderline in this dimension.
- The proposed method is intuitive and easy to understand, but may fit an application oriented conference better. Using GRU units to update the memory and establish intra-sequence connections is hand-waving and there is no rigorous mathematical analysis on why and how this works. This may be ok for an application paper but seems to be below the bar of ICLR.

Minor comments
- Figure 1 quality should be better.
- Make notation more consistent, eq4 and eq6 the scripts should be both super or sub.

summary_of_the_review: Though the method proposed in this paper is intuitive and sound in general, the lack of rigorous algorithm analysis and significant lack in experiments in terms of setting and baselines may require the paper to be more polished before publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the Iterative Memory Network (IMN), an end-to-end differentiable framework for long sequential user behavior modeling. Moreover, this paper conducts experiments to show its better performance. 
main_review: Strengths 
1. This paper is well written and easy to follow. 
2. The topic is significant. Modeling the long sequence is technically challenging. 
3. This paper proposes the Iterative Memory Network (IMN), an end-to-end differentiable framework for long sequential user behavior modeling, which shows better performance than baselins. The IMN model is technically sound and interesting. 


weaknesses
The compared sequential models lack some STOA, such as [1, 2]. This paper claims its advantages over transformer variants. However, the comparisons ignore the STOA models based on transformers, which does not make sense. 


[1] Sun F, Liu J, Wu J, et al. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer[C]//Proceedings of the 28th ACM international conference on information and knowledge management. 2019: 1441-1450.
[2] Zhou K, Wang H, Zhao W X, et al. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization[C]//Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020: 1893-1902.

summary_of_the_review: Modeling the long sequence is a hard problem in recommendation applications. This paper proposes the IMN model to learn the user preference from a long sequence and performs better than baselines. The technical significance and novelty of the proposed model are good. The reviewer tends to accept the paper.
review_title: null

Meta-review: The paper proposes a new architecture named Iterative Memory Network (IMN) to encode long user behavior sequence for recommendations. Reviewers appreciate the clarity of the writing as well as practicality and the O(L) complexity of the proposed architecture, however do raise questions on novelty. Different design choices employed in the paper are not well explained. The rebuttal was not able to convince the reviewers to accept the work at this venue, but reviewers do feel the paper could fly in an application oriented venue.
Decision: Reject