Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary:

This paper proposed to care about the sequential multi-behavior recommender systems where the multiple behaviors include a subset from {view, favorite, cart, purchase, browse, review}. To capture the multiple user behaviors, a multi-relation deep GNN is adopted to propagate behavior-aware messages and the graph Laplacian normalized function is incorporated into the message passing. In this way, both item representations and user’s short-term interests can be captured. As for the relations among multiple behaviors, The paper is to learn cross-type behavior dependency matrix by a Transformer, or called self-attention-based memory network in the paper. In this way, both sequential/time-evolving and multi-type behavior dependencies are learned. In the end, the paper proposes a contrastive learning to pull the type-specific behavior and multi-behavior representation of the same user to be closer as positive pairs while to push the behavior of different users away as negative pairs. Experiments are conducted on three datasets by comparing with various kinds of 19 baselines. Ablation and visualization are further shown to understand the components of the proposed EGCM model.

strength_and_weaknesses: Strong & Weak:

S1: It seems novel to use contrastive learning framework for the sequential & multi-behavior recommender systems. Although both sequential and multi-behavior ResSys methods are proposed in existing works, this paper is the first to adopt contrastive learning technique to construct the two auxiliary tasks to better learn the main task. Furthermore, the ablation study shows that, the proposed model degrades the performance by a large margin if the contrastive learning component is removed, especially on the Tmall and IJCAI datasets, though the impact of contrastive learning is not so big on the E-commerce dataset.

W1: One possible weakness in my own opinion: While the proposed EGCM is a new method for sequential & multi-behavior recommender systems, each of the individual piece of EGCM (multi-relation GNN, Transformer, contrastive learning) is not new in itself and has been widely adopted in existing works as pointed out in the related works. This may be not a big issue in practice, since EGCM demonstrated a strong empirical result on three datasets.

clarity,_quality,_novelty_and_reproducibility: writing is good; technical originality is somewhat weak.
summary_of_the_review: Comments:

C1: some typos. On page 2, “an” in the statement “we propose a Evolving Graph”. On page 6, the “+” symbol in computing $\tilde E_{t,u}$; the subscript $e_i$ should be $e_u$ in the statement “by pulling the type-specific behavior embedding…”; the dot product in the statement “measured by the doc product…”
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a graph neural network (GNN) based recommender system for datasets featuring heterogeneous interactions (click, purchase, etc). This is done by performing message passing within behavior and time slot-specific subgraphs. Information is fused between behavior types and across time slots using various standard neural network components. Additionally, the paper proposes to use contrastive learning to encourage representations for the same user to be similar while pushing apart representations for different users. Experiments demonstrate the proposed model outperforms baselines on two small benchmark datasets and a proprietary (I think) e-commerce dataset.
strength_and_weaknesses: **Strengths**

- The method performs well in the experiments.
- Some ablation studies are included to validate choices.

**Weaknesses**

- The constituent pieces of the proposed model are known and there is limited technical novelty. 
- Many of the design choices seem arbitrary and are not adequately justified. Over three pages of the paper are spent describing the architecture and providing fairly loose intuitive motivation without actually justifying why the specific choice is obvious or should be preferred, e.g., "both short-term and long-term multi-behavior preferences of users can be preserved by the fused representations," "our EGCM can preserve the dedicated time-evolving behavior dependencies across different types of user interactions." The desirability of these properties may be intuitive, but how this maps to the specific choices made is not. 
- The ablation studies are extremely limited, and the experiments are not designed to justify the design decisions made. For example, without designing experiments that explicitly test the model's ability to "capture such evolving cross-type behavior dependencies across time slots"  how can the reader be sure this is what the proposed mechanism is actually doing? Indeed, in the results section the authors somewhat tautologically explain the observed performance improvements, "EGCM outperforms various baselines in all cases by achieving significant performance improvements." The lack of rigorous experimental design to validate the reason for performance improvements makes me doubtful that the paper provides a solid foundation on which subsequent research can build.
- The paper cites [1], although it does not seem to take any lessons from the work. For example, the arguments that applying non-linearities and transformation matrices to non-feature-based representations (e.g. a user or item id) seem to apply to this work as well.
- The authors use sampled evaluation metrics despite previous work [2] showing this methodology can be misleading. Since the datasets seem relatively small (the largest dataset features 35.5K items), I'm curious if there was a particular reason the paper chose this evaluation.
- The "In-Depth Analysis of ECGM Model" section is only a paragraph, and the conclusions drawn from the figures are not adequately described. I didn't find this section particularly interesting, informative, or in-depth.

**References**
1. [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://arxiv.org/abs/2002.02126)
2. [On sampled metrics for item recommendation](https://dl.acm.org/doi/abs/10.1145/3535335)
3. [A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research](https://dl.acm.org/doi/abs/10.1145/3434185)
clarity,_quality,_novelty_and_reproducibility: Despite the ideas being relatively straight-forward, I found the paper quite difficult to read. I found the mathematical presentation particularly poor. In particular, almost everything is an E with various subscripts, superscripts, and accents. Many terms are not defined after they are first used, and the dimension of numerous variables is not explicitly.

Given the large number of architectural components, I also suspect it would be difficult to reproduce the results in the paper, and the authors do not provide anonymized source code. As this has been a persistent issue in the space [3], I would strongly encourage the authors to consider whether their results would be reproducible.
summary_of_the_review: While the experiments demonstrate performance improvements over baselines, the large number of seemingly arbitrary design decisions are not sufficiently explored. Instead, the paper reads mostly like a description of random components that happen to perform well in the particular benchmarks chosen. As a result, I believe the paper's contribution is quite limited and do not recommend acceptance.

--------

**Edit**

The authors have addressed several of my concerns in their replies and I have raised my score accordingly. Furthermore, it should be noted that my original statement that the authors do not provide source code was incorrect. I apologize for having overlooked it initially.

--------

**Edit 2**

I have re-reviewed the most recent revision of the paper. I would like to thank the authors for engaging with reviewers and for their updates to the paper. I believe the paper has improved as a result of the process, but will not be further raising my score at this time, and cannot argue strongly for acceptance of the paper. 

While I believe there are interesting ideas presented in the paper, I continue to find the disconnect between motivation and experiments the most critical outstanding issue. I would encourage the authors to address this issue in future revisions of the work, as I feel if this issue was addressed, the work has the potential to be a significant contribution. 

To clarify my meaning here, I will reiterate and elaborate on an example I used previously. In Section 3.2 the authors motivate the necessity of the cross-relational memory network component through the need to "reflect a holistic view of diverse preferences of users across different time slots" and then go on to give the following example "customers may first view some products in online retail sites, and make their purchase decisions one day after." While the ablation experiments show average performance increases when this component is included, they do not validate that this performance increase is due to the stated motivation. A more convincing experiment would identify examples of such behavior (a product view temporally followed by a purchase) and validate that including this component improves performance on these specific instances (and that performance degrades on these instances when the component is not included). Without this sort of analysis, it is impossible to disentangle the given motivation from other potential sources of performance improvement (regularizing effect, increased capacity, etc).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an Evolving Graph Contrastive Memory (EGCM) framework which effectively models the multiple behaviors in the recommendation. EGCM is comprised of (1) a multi-behavior graph encoder to model the behavior-aware short-term interests of users, (2) a dynamic cross-relational memory network to model cross-behavior relational transitions, (3) a contrastive learning module to enhance the generalizability and robustness. The experimental results show that the proposed method could effectively outperform SOTA methods.
strength_and_weaknesses: Strengths:

1. The experimental results show that the proposed method could significantly outperform the SOTA baselines.
2. Code is provided. 

Weaknesses:
1. The novelty of the paper is limited. Many modules are simple applications/extensions of existing methods.
2. The proposed model is too complex.
3. The writing is not quite clear, and some parts are difficult to follow. 

For example, Eq. (4) is difficult to understand. Where to use $\mathbf{m}^b_{u_t\leftarrow i_t}$? 

You use $f({\mathbf{e}^b_{i_t}}, {\mathbf{e}^b_{u_t}})$ in Eq. (4), however, ${\mathbf{e}^b_{u_t}}$ does not appear on the right hand side.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper should be further improved.

The overall quality looks good. 

Novelty is limited since many modules are simple applications/extensions of existing methods.

Reproducibility is good. The implementation details are presented and the code is also provided.
summary_of_the_review: The experimental results demonstrate that the proposed method could significantly outperform SOTA methods. However, the novelty is limited.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper studies the problem of capturing the interaction heterogeneity in modeling dynamic user preference for recommendation. A new evolving graph contrastive learning paradigm is designed to preserve both multi-behavior diversity and commonality. Extensive experiments are conducted to demonstrate the effectiveness of the new method compared with a large number of alternative solutions from different perspectives.
strength_and_weaknesses: Pros:
1.The manuscript is well-organized and easy to follow. 
2. The descriptions of model motivation with technical details are very clear. The authors point out key challenges in learning preference dynamics for multi-behavior recommender systems. In the proposed EGCM method, the multi-behavior dynamic dependency challenges are addressed by integrating the contrastive SSL with cross-relation memory network. 
2.Theoretical discussion of the new contrastive learning is offered to analyze the benefits of the introduced behavior-wise contrastive-based SSL.
3.Comprehensive experiments are conducted to justify the effectiveness of the new approach by comparing EGCM with 19 alternative solutions. 
4.Component-wise model evaluation as well as the in-depth analysis of EGCM method are provided to show the rationale of the superior performance.

Cons:
In the model inference phase, the self-supervision loss objectives L_{cl} with both short-term and long-term behavior modeling are incorporated into the original supervision optimized BPR-based loss. Besides the provided pseudo code in the appendix, the details of learning process in EGCM can be elaborated.

For the experiment of alleviating data sparsity issue, while the variants already achieve the obvious performance gain, the recommendation results of the new method can also be added.

clarity,_quality,_novelty_and_reproducibility: This paper solves an under-explored recommendation problem with dynamic diverse behavior patterns. A novel contrastive learning paradigm is designed to tackle the corresponding technical challenges. Sufficient and appropriate experiments are conducted to show the new recommender EGCM’s effectiveness. The source code and evaluation data are released for result reproducibility. The parameter settings are described in the paper.
summary_of_the_review: The proposed new framework would be beneficial for contrastive learning domain in learning the relational dynamics. The idea of dynamic contrastive learning to behavior-aware recommendation is novel and interesting. Good performance with comprehensive experiments on benchmark datasets are given.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: This paper presents Evolving Graph Contrastive Memory Network (EGCM) which aims to incorporate multiple types of behavior signals (click, purchase, review, etc.) in a sequential recommendation setting (i.e., learning the ever-evolving user preference over time). There are a few components in EGCM. First, a graph neural network encoder is used to capture the short-term preference with different behavior types.  Following that, a transformer-style attention mechanism is used to capture the long-term multi-behavior preference from the short-term representations as a memory network. Finally, a contrastive learning objective is adopted to make sure the behavior-specific representation and behavior-aggregated representation from the same user stay close (positive pairs) while pushing the representations from different users further away (negative pairs). Experimental results demonstrate that the proposed EGCM outperforms other baselines. Ablation studies are also conducted to demonstrate the importance of each individual component. 

Strengths: Extensive experimental results show that the proposed EGCM works well over other baselines. The authors did make an effort at demonstrating the effect of various components via ablation studies. 

Weaknesses: There have been quite a few score updates during the discussion phase. I have also read the paper myself in the past week and discussed with some of the reviewers regarding their opinions about the paper. Some of the minor weaknesses are mostly addressed. To me there still seems to be a major weakness:  

* The paper right now reads overly complicated with limited technical novelty (Reviewer rXXA and 1ytF). As the summary above shows, it consists of a few components, each of which involves some modifications to an existing piece of work. Many of the design choices seem rather arbitrary and are not adequately justified -- I am sure these choices all helped but as with many developments with improving the architecture of a neural net, we are practically faced with infinitely many choices. I don't doubt that adding a particular component improves the performance, especially given the relatively smaller datasets used in the experiments (more regularization will help here). But the degree of specificities in this paper at the moment seems too much. It almost reads like a combination of specially-designed components which are put together to do well on the benchmark datasets. If we are one day faced with a similar multi-behavior dynamic recommendation problem, I am not sure how much learning we can get from this paper. (It might not be the most accurate analogy, but this almost feels like the winning solution of a competition which is too complicated that people will end up using a simplified version in practice.) 
* As for in which direction the authors can improvement the paper, I think adequately justifying many of the design decisions is in order (maybe even simplifying things a bit). To quote a reviewer during the discussion phase: "I believe it would be much more valuable if the authors had designed experiments to demonstrate the components improve performance because of the specific motivation given. As an example, in Section 3.2 the authors motivate the necessity of the cross-relational memory network component through the need to "reflect a holistic view of diverse preferences of users across different time slots" and then go on to give the following example "customers may first view some products in online retail sites, and make their purchase decisions one day after." It seems it would have been easy to find examples of such behavior and validate that including this component improves performance there." 
* There are also some concerns around the sampled metrics vs full-catalog metrics. Given the choice the datasets, I don't think it is particularly challenging to compute the full-catalog metrics as the authors claimed. Update: I noticed that the authors just provided the full-catalog metrics. And as some of the reviewers suspected they are indeed much closer than the sampled metrics, which begs the question: do we really need so many over-complicated components to be able to achieve practically (not statistically, since none of the error bars in the experiments actually means much) significant results?

I also think this paper will find a larger audience in a more recommender-systems-focused venue (e.g., KDD, TheWebConf, etc.)
justification_for_why_not_higher_score: The scores the paper currently receives (10, 6, 6, 6) certainly don't suggest a clear rejection. There have been quite a few score updates during the discussion phase where initially two reviewers gave rejections but later changed to marginally accept. However, during the discussion, one reviewer who gave it a 6 still feels that this paper is not good enough to be accepted, which makes me recalibrate the scores. (I myself also read the paper and I think I would give it a 4 or 5 if I were a reviewer.) Furthermore, I don't find the comments from reviewer HypT who gave this paper a 10 very helpful -- this paper clearly has some problems (as pointed out by other reviewers), yet despite us asking for more input, reviewer HypT did not participate in any of the discussions. Therefore, his/her comments did not get much weight in my decision.
Decision: Reject