Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a model to learn disentangled representations for user intentions from complex and noisy multi-feedback. The model can capture complex relations among multi-feedback and various user intentions, and filter out feature-level noise in multi-feedback. Moreover, this paper proposes an adjustable curriculum to alleviate the impact of label-level noise in a more controllable way.


main_review: Originality: This paper proposes a new method to utilize multi-feedback data and a new adjustable curriculum learning method. The contributions are clear and the related work is adequate. The method combines existing components to form a reasonable solution, so the overall technical novelty is not big. 

Quality: The proposed method is clear and the experiment is sufficient.

Clarity: This paper is well written and organized. And most of the proposed method and the experiment are clear. But I have doubts on some details. In second row and fourth row of Eq. (1), why does d_{ci}^{v} use softmax and d_{ui}^{v} use sigmoid? And in fourth row of Eq. (1), should the output of MLP between 0 and 1 like sigmoid? In Eq. (4), why are z_{ci}^{v}-n_{c}^{v} and z_{ci}^{v} \odot n_{c}^{v} also inputs to MLP?

Significance: The idea of filtering the noise in both clicked and unclicked items from disliked items is particular. The proposed curriculum may be applied to other tasks. The extensive experiments shows that the model achieve state-of-the-art performance. 

limitations_and_societal_impact: In Section 3.2.1, you take the time and candidate item factors into consideration, how these factors affect the performance of the model? You need to add an experiment to demonstrate it.

In In Algorithm 1, \mu changes as the training goes on. Can $thre$ also change as the training goes on?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a curriculum disentangled recommendation model to learn disentangled representations from multi-feedback data. Specifically, the authors design a co-filtering routing mechanism to capture the relations among multi-feedbacks and denoise the representations at the feature level. Then they utilize the curriculum learning to evaluate sample difficulties to do denoising at the label level. The extensive experiments show the effectiveness of the proposed method on recommendation tasks.
main_review: Pros:
-	The proposed method tries to learn the disentangled representation for the recommendation problem from multi-feedback with noisy information. The task is well-motivated and important for real-world scenarios.
-	The paper originally considers two-level noisy information. One is feature-level noise, and another is label-level noise.
-	The paper is well-written and easy to follow.

Cons:
-	The authors should give the definitions of feature level noise and label level noise. The label-level noise is easy to understand. But the feature level noise is confusing. For example, what kind of click action would be regarded as noise? The noisy unclicked items might be useless items that are different from the noise in the labels.
-	The hyper-parameters of curriculum learning need to be searched and predefined manually. It is difficult to find the optimal parameters for different datasets, especially in the online scenario. 
-	How to decide the number of k latent intentions is not clear.
-	The code is not provided. It is hard for readers to reproduce the results in the experiments.

limitations_and_societal_impact: No discussions about the potential negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper introduces an approach to take into account different types of positive and negative feedback in a recommendation model, while accounting for the possible noise contained in the feedback. The proposed model relies in particular on three types of feedback (click, lack of click and dislike). The sequences of different types of feedback are first encoded with a Transformer and then aggregated through a series of MLPs and attention mechanisms into vectors supposed to capture and disentangle the intents of the user.

Independently from this module, the paper also proposes a curriculum learning approach to train the model and allocate with flexibility more weight on either the easier samples or the harder samples at the beginning of the training.

The proposed approach is evaluated on 3 public datasets and 1 proprietary dataset, and compared against several baselines.
main_review: ### Strengths

- The proposed curriculum approach seems easy to use and general enough to be applied to a wide range of recommendation approaches.

### Weaknesses

- Many choices in the modeling of the "co-filtering dynamic routing" are overly complex, unclear or lack motivations/justifications (see Section "Model design").
- The results of the paper are not reproducible as the code is proprietary.
- The experiments miss the comparison against appropriate and recent baselines (see Section "Empirical methodology"), and did not report statistical significance or error bars. 
- Given the high complexity of the approach and its large number of parameters, comparing the runtime of the approach against that of baselines would have been needed.

### Model design

- Based on Section 3.1, the similarity function seems to systematically apply LayerNorm before computing the dot product and softmax operation. What is the motivation for using LayerNorm here?
- In Section 3.2.1, the input item features are projected into an embedding space. It seems that features are assumed to be discrete. Is the approach still adapted for continuous features?
- In Section 3.2.1, it is unclear why the sequence of click interactions, the sequence of "unclick" interactions, and the sequence of dislike interactions are all three considered independently. These sequences might have be linked together. For example, the user may first click on an item A, then check a similar item B and dislike it. One could have simply considered all the interactions in sequence, and after that extract the item representations for each interaction type. Additionally, it is not fully clear if the different Transformer encoders (C-Encoder, U-Encoder and D-Encoder) share some parameters or are totally independent.
- What is the motivation for using a residual connection in Equations (1), (2) and (4)? No justification is provided on that, and the experiments do not report results with/without these to show the need for this.
- The scalars d_ci^(v) and d_uj^(v) in Equation (1) represent the contribution of clicked and unclicked items to the user intention. d_ci^(v) is given by a softmax and is thus strictly positive, however d_uj^(v) is based on the output of an MLP and it is therefore unclear if this quantity is positive or not.
- The MLP used in the definition of d_uj^(v) takes as input the concatenation of F^(v), which contains the base embeddings of all g user features, and z_uj^(v) which was output by the U-Encoder. How is the concatenation performed here given that there are g+1 vectors? Do these vectors have the same dimensions, given that some are base embeddings and some are output by the Transformer encoder?
- In Equation (2), it is unclear what the terms time_ci^(v) and time'_cm^(v) are trying to capture. Given that z_ci^(v) and z_cm^(v) are the output of the Transformer encoder, they should already capture temporality/sequentiality through position embeddings. What is the need then to add a fully-connected layer that takes as input the embedding output by the Transformer *and* the positional embedding? This seems redundant and unnecessary.
- In Equation (4), what are zl_ci^(v) and zl_uj^(v) intended to capture? It is unclear why it is needed to aggregate z_ci^(v) and n'_c^(v) rather than directly using z_ci^(v) in the definition of intent_k^(v).
- The quantity inten_k^(v) in Equation (4) is supposed to capture the interest of user v according to the k-th intent. However inten_k^(v) depends through f_ci^(v) and f_uj^(v) on the candidate item considered. It is unclear why the intention of the user would depend on a candidate item, which should be totally independent from the user as it can be any item.
- The multihead attention introduced in Equation (5) lacks motivations. What is the expected role of the different heads in this context?
- If one omits the bias beta_k in Equation 4, which has only a marginal role, then the sum over the different intents in Equation 6 will actually result in ignoring c_ik^(v) and u_jk^(v), and the notion of intents altogether. Indeed, zl_ci^(v) does not depend on k and sum_k c_ik^(v) = 1. A similar observation holds for zl_uj^(v) and sum_k u_jk^(v). Then the sum over intents in Equation 6 removes all dependencies with respect to an intent: sum_k intent_k^(v) = sum_i d_ci^(v) * f_ci^(v) * zl_ci^(v) + lambda * sum_j d_uj^(v) * f_uj^(v) * zl_uj^(v). Therefore, the prediction made by the model mostly ignores intents, despite intent disentanglement being presented as a main feature of the approach.
- In Equation 6, the dot product term is positive but it is unclear if the output of the MLP term is positive or not.

### Empirical methodology

The sequential recommendation models used as baselines (DIN, SASRec, DFN, SDR) were designed for next-item recommendation: based on a past sequence of interactions, predict the next item. However the experimental setup corresponds to a top-k recommendation scenario: given the past history of interactions, predict the set of test items.

The sequential baselines are then not very appropriate for a task where the test set contains more than a positive item per user, because the different items in the test set won't be all correspond to the next item following the training sequence. Therefore, one needed to either consider a next-item recommendation setup with only one test item to predict, or compare instead against other top-k recommendation baselines.

The authors could have considered the state-of-the-art recommendation model LightGCN [1] as a potential, more recent baseline.

[1] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, Meng Wang: LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. SIGIR 2020: 639-648

### Clarity

- It was not fully clear at the beginning of the paper what the "unclick" interaction means. An unclick interaction seems to suggest the user had first clicked on an item and then removed the click. But in fact in this paper an unclick interaction simply corresponds to a lack of click interaction. This should be made more clear and unclick interaction should instead be replaced with lack of click.
- The choice to denote the positive interaction as "click" and the negative interaction has "dislike" creates an asymmetry which can be confusing for the reader. One could question one does not consider the positive interaction "like" which is the positive counterpart of "dislike" and exhibits more confidence than a click, which can be more noisy in its positiveness.
- In Section 4.3, it is not fully clear what the variant c&un of the proposed model consists in.

### Update after author response

I thank the authors for their detailed response to my comments and questions. The extensive efforts that were put in the response is commendable and appreciated. However, there are still several points which, despite the authors' response, remain problematic for me (detailed below), and I will thus keep my initial score.
- **Efficiency & Complexity.** The runtime reported by the authors for the proposed model indeed seems reasonable in comparison to that of DFN. However, the complexity indicated in the response does not seem correct as the reported Transformer complexity ignores the dependence on the length of interaction sequences.
- **Empirical methodology.** The response clarified that the experiments follow a next-item recommendation setting. This will need to be explained in the experimental setup section as in its current version the paper and the supplementary material lack too many details about the evaluation procedure. It is also still not clear whether at inference time, all items are considered as candidates or if a sampling is performed (which should be avoided [2]). With respect to the argument that LightGCN cannot be used as a baseline because of the next-item recommendation setting, I do not agree with the authors. Some of the baselines considered in the paper, such as DeepFM and AutoInt, were not initially designed either for the next-item recommendation task. Therefore, LightGCN could be adapted to the task in the same way as it is done for other baselines. This can be simply done by retraining the model each time new samples are added to the training set.
- **Choice of different behaviors.** The authors claim that "like" behavior is much more rare than "dislike" behavior, however I did not see justifications or references to explain why there would be such an asymmetry.
- **Motivations for using LayerNorm.** The authors mention that using a simple dot product as opposed to a cosine similarity leads to mode collapse, and thus normalization is needed. Then, why not use a simple cosine similarity, by dividing the dot product by the vector norms, instead of using LayerNorm? This argument did not convince me that LayerNorm is necessary or needed in the proposed model. This design choice seems to be guided by the fact that Transformer architectures adopt this, without evidence that this is needed in the proposed model (which vastly differs from a vanilla Transformer). For this choice and others (e.g., use of residual connections) I would have liked to see ablation studies to validate their necessity.
- **General user intentions should be independent of the candidate item.** I still find problematic and unintuitive that user intentions depend on the candidate item. The user-item matching should intuitively be done by comparing a user representation (independent of the specific candidate item) and an item representation (independent of the specific user). Otherwise, the matching process boils down to matching an item with itself, which makes little sense. Having a user representation which depends on the candidate item also seems to lead to inefficient and non-scalable inference, as in the inference phase all items in the datasets need to be considered as candidate items for each user.
- **About the disentangled intentions.** The authors describe beta_k as the center under the k-th intention, and zl_ci and zl_uj as offsets to these intention centers. However, zl_ci and zl_uj are not specific to any intention as they do not depend on intent k; it is then unclear how could they be at the same time the offset from all intent centers beta_k. Moreover, the authors' response did not address the fact that the influence of intents on Eq. 6, with or without beta_k, remain very minor as detailed in my original comment.

[2] Krichene, W., & Rendle, S. (2020). On Sampled Metrics for Item Recommendation. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1748–1757. 

### Update 2

I thank the authors again for the additional clarifications provided in the second round of discussion. In light of the satisfactory response brought by the authors, and although I still consider that some parts of the proposed approach are slightly arbitrary and that significant work will be needed to polish the paper based on our discussion, I will raise my score.
limitations_and_societal_impact: The paper did not discuss the limitations of this work and its potential negative societal impact.

A limitation of the work is in the type of feedback considered. The approach is based on "click", "lack of click" and "dislike" signals. However a platform could have access to other types of signals ("like", "add to basket", "add to wishlist", "remove from wishlist", etc.) and it is unclear if the proposed approach could directly use this feedback.

Additionally, the approach seems to suppose that user/item features are discrete and did not cover how continuous features could be handled.
review_title: null

Meta-review: Overall strong scores, and reviewers are mostly aligned in finding the paper to be above the bar. Scores were initially more mixed, though after the discussion the weakest score was improved (to "borderline"). Reviewers praised the originality of the idea, and found the model performance through the experiments convincing enough. Several issues were raised regarding clarity and other specific details, and an in-depth discussion took place to resolve most of these issues.

Ultimately the scores are strong, and the rebuttal was fairly persuasive. The reviewers do highlight several issues (and in spite of the strong scores, the reviews are not particularly gushing in the text). But the issues raised seem mostly in terms of clarity, or otherwise are adequately  addressed during the rebuttal phase.
Decision: Accept (Poster)