Long-Sequence Recommendation Models Need Decoupled Embeddings

Authors: Ningya Feng, Junwei Pan, Jialong Wu, Baixu Chen, Ximei Wang, QianLi, Xian Hu, Jie Jiang, Mingsheng Long
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Recommender System, User Interest Modeling

Abstract:
Lifelong user behavior sequences are crucial for capturing user interests and predicting user responses in modern recommendation systems. A two-stage paradigm is typically adopted to handle these long sequences: a subset of relevant behaviors is first searched from the original long sequences via an attention mechanism in the first stage and then aggregated with the target item to construct a discriminative representation for prediction in the second stage. In this work, we identify and characterize, for the first time, a neglected deficiency in existing long-sequence recommendation models: a single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes. Initial attempts to address this issue with some common methods (e.g., linear projections---a technique borrowed from language processing) proved ineffective, shedding light on the unique challenges of recommendation models. To overcome this, we propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are initialized and learned separately to fully decouple attention and representation. Extensive experiments and analysis demonstrate that DARE provides more accurate searches of correlated behaviors and outperforms baselines with AUC gains up to 9‰ on public datasets and notable improvements on Tencent's advertising platform. Furthermore, decoupling embedding spaces allows us to reduce the attention embedding dimension and accelerate the search procedure by 50\% without significant performance impact, enabling more efficient, high-performance online serving. Code in PyTorch for experiments, including model analysis, is available at https://github.com/thuml/DARE.

Decision: accept (poster)
Reviewer scores: 5, 6, 3, 8

Reviews and discussion:
<thread>
creation date:
1729916195793

thread_type:
official_review

<thread_object>
creator: reviewer_rd7c

<content>
summary:
This paper has focused on the interference between search and aggregation stages in the long-sequence recommendation. The authors first conducted some preliminary experiments and found the unbalanced and adverse gradients brought by the two stages. Inspired by such findings, they propose to adopt two individual embedding sets for the two stages, respectively, which may alleviate the conflicts between the two stages. The experiments on two public datasets validate that the proposed method can benefit the previous two-stage method, i.e., TWIN, and achieve SOTA performance.

strengths:
- S1. This paper is well-written and -structured, which is easy to follow and understand.
- S2. This paper is well-motivated, where the motivation is from an interesting preliminary study.
- S3. The proposed method is simple but effective.

weaknesses:
- W1.  I can understand the results and conclusion from the preliminary study, but the detailed process of the preliminary study is not clear. For example, I wonder how to calculate the gradients from the attention and representation to the embeddings.
- W2. Only one metric is revealed in the experiments. GAUC and Logloss are often two common metrics in CTR prediction tasks. Thus, I suggest the authors add more metrics.
- W3. The number of filtered behaviors, i.e., K in this paper, is often important to the two-stage method, but no related hyper-parameter experiment is conducted.
- W4. It seems that the proposed method can only fit the two-stage method, such as TWIN. However, there still exist many one-stage works [1], which may illustrate that the two-stage method has not been the optimal solution. If so, the value of this paper is limited.

[1]. Yu, W., Feng, C., Zhang, Y., Hu, L., Jiang, P., & Li, H. (2024). IFA: Interaction Fidelity Attention for Entire Lifelong Behaviour Sequence Modeling. arXiv preprint arXiv:2406.09742.

questions:
- Q1. Why the performance of TWIN-V2 is worse than TWIN, which does not align with the original paper? Are there some differences in experimental settings?

All of my other questions are included in the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730340618804

thread_type:
official_review

<thread_object>
creator: reviewer_zxcu

<content>
summary:
The paper focuses on long-sequence recommendation models. It first identifies a problem in existing models where a single set of embeddings struggles to handle both attention and representation learning, leading to interference. Initial attempts to address this using linear projections from language processing failed. Then it proposes the Decoupled Attention and Representation Embeddings (DARE) model. This model uses two distinct embedding tables for attention and representation, which are initialized and learned separately. Extensive experiments on public datasets and an online system show its effectiveness.

strengths:
1. The paper clearly identifies a crucial problem in long-sequence recommendation models. The analysis of the interference between attention and representation learning due to shared embeddings is comprehensive.

2. The extensive experiments on public datasets (Taobao and Tmall) and an online advertising platform are a major strength. The comparison with a wide range of baselines, including state-of-the-art models like TWIN and its variants, demonstrates the superiority of the DARE model. The evaluation metrics used, such as AUC, NDCG, and representation discriminability measures, provide a comprehensive assessment of the model's performance.

weaknesses:
1. The performance on the short-sequence modeling using the Amazon dataset was marginal. This suggests that the DARE model may not be as effective in short-sequence scenarios, and more research is needed to understand its applicability in different sequence lengths. Additionally, the experimental setup for some baselines, like DIN, required adjustments to fit the long-sequence context, which may introduce some biases in the comparison.

2. There are some results that lack a satisfactory explanation. For instance, on the Tmall dataset with embedding dimension = 16, TWIN (w/o TR) outperforms TWIN, which was unexpected. The authors could not convincingly explain this result, leaving a gap in the understanding of the model behavior under different dataset characteristics.

3. In the exploration experiment, although the data is somewhat persuasive, some special cases or edge cases may not be fully covered. For example, it is uncertain whether different gradient relationships will occur under certain combinations of specific categories or user behavior patterns. Moreover, there is no further in-depth analysis of the reasons behind why it is nearly two-thirds of the cases, which may cause readers to have some doubts about the universality of the conclusion.

questions:
How was the decision made to use two distinct embedding tables for attention and representation? Were there any other alternative architectures considered and why were they rejected?

In the DARE model, how sensitive is the performance to the choice of embedding dimensions for attention and representation? Have different combinations been explored and what are the trade-offs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730343850929

thread_type:
official_review

<thread_object>
creator: reviewer_jvl4

<content>
summary:
The paper addresses a key limitation in existing long-sequence recommendation models, which struggle to balance the learning of attention and representation with a single set of embeddings. To tackle this, the authors introduce the Decoupled Attention and Representation Embeddings (DARE) model, which separates these two processes with distinct embeddings. Experiments show that DARE not only improves the accuracy of correlated behavior searches but also boosts efficiency, reducing embedding dimensions and accelerating the search by 50% without significant performance loss.

strengths:
The definition of the problem is very clear.
The research perspective is quite innovative.

weaknesses:
The definition and explanation of the gradient issue are one-sided, and the motivation and the logic of the method are not clear. For example, why do the impact of different modules on embedding gradients must be ‘equal-magnitude’ and ‘consistent’? How do imbalance-magnitude and inconsistent gradients degrade recommendation performance? Additionally, the connection between the proposed method and the target problem is weak. For example, why does the embedding decoupling approach help mitigate the gradient issues? Overall, the logic and the motivation of the main idea need to be illustrated clearly.

questions:
1.	Why are the gradients from \textbf{attention} significantly smaller than those from \textbf{representation} in Fig. 2? And why is this imbalance undesirable during training (e.g., does it contradict theoretical derivations)? It is necessary to provide a theoretical analysis here.
2.	The definition of conflicts between attention and representation is quite vague. The cancellation of gradients during backpropagation cannot be explained merely by the sign of the angle. Could you supplement this with the merged gradient values or in other ways to illustrate the degradation caused by conflicts to the embeddings?
3.	What are the consequences of these gradient issues? Do they lead to the failure of representation learning or the malfunction of certain modules? Would you add a presentation or explanation of the outcomes of the mentioned problem？
4.	How does DARE solve the aforementioned gradient issues (gradient magnitude imbalance and gradient conflicts)?
5.	The embedding dimension shown in Figure 6 should reach at least 200 to be consistent with the analysis in lines 208-213. Additionally, the analysis here can be summarized as: linear mapping requires a large embedding dimension to be effective, but increasing the embedding dimension leads to the dimensional collapse problem in recommendation systems. How does DARE address this problem? Does it only work effectively in low dimensions, or does it solve the dimensional collapse problem and work under any embedding capacity?
6.	How are the embeddings between the two modules in DARE aligned? Why can the weights used to measure attention embeddings be directly utilized for representation embeddings?
7.	The concept of Mutual Information （MI） is repeatedly mentioned in the experimental section, so it is necessary to introduce its calculation method. Moreover, this concept is already used in Figure 1, but it is not explained there. The connection between MI and the problem analyzed in this paper (Section 2) needs careful and thorough clarification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709783237

thread_type:
official_review

<thread_object>
creator: reviewer_vgn1

<content>
summary:
This paper suggests the inefficiency of using the same embeddings for attention calculation and preference prediction (representations) in lifelong sequence recommendations, where the user behavior history (sequence) is usually very long (more than 10,000). Authors correspondingly propose a decoupling framework, DARE, to boost the recommendation performance by decoupling embeddings. Specifically, DARE sets separate embedding tables for attention calculation and preference prediction. Abundant preliminary studies and experimental evaluation support the effectiveness.

strengths:
1. Preliminary study is available. Authors provide experimental evidence supporting that using a shared embedding for attention calculation and preference prediction can impair performance due to the varying magnitudes and conflicting gradients. Furthermore, the authors examine the linear projection solutions of current methods, noting that the constrained embedding size may hinder the effectiveness of linear projection for decoupling. 

2. Clarity. The paper presents its ideas with remarkable clarity, making it not only easy to follow but also engaging for the reader.

3. Experiments. Comprehensive experiments demonstrate the effectiveness of DARE. Various behavioral sequence models, including DIN, SDIM, ETA, and TWIN, are compared. A thorough analysis of attention and representation capabilities, as well as efficiency tests, is presented.

weaknesses:
1. Applicability. This work aims to decouple attention and representation embeddings for sequence modeling. I wonder if the proposed solution in Section 2 can be extended to more scenarios. When the attention mechanism is not used in certain sequential models, the suggested method does not apply and, therefore, cannot benefit these models.

2. Inconsistent Solution without Novelty. The proposed solution is too straightforward-setting separate embedding tables for attention and representation. This is not novel and does not align with the findings, although the separate embeddings can resolve the conflicting gradient and are not simply a linear projection. In my view, authors should go beyond linear projection. For instance, a linear projection-based decoupling method that addresses the gradient issue can be applied to small embedding sizes in recommendation systems, rather than reverting to separate embeddings, a very basic solution.

3. Storage Complexity. Authors only provide the inference efficiency. However, for large-scale recommender systems, a separate embedding table can occupy exceptionally large storage space.

questions:
1. What are potential application scenarios other than sequential models with attention and representation?

2. To what extent would DARE increase the storage complexity? I believe it would be doubled.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732283523124

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The updates in our paper.

comment:
We sincerely appreciate all reviewers' valuable feedback! Based on your   suggestions and concerns, we have thoroughly revised our paper. **The changes are mainly in appendix** (page 13 to 20), and **all changes have been highlighted in blue** for your convenience. The updates can be summarized as follows:
1. **DARE defeat all other candidates for decoupling**

We have provided **a detailed account of our research process leading to the development of DARE**. Following the principle of decoupling, we explored **eight alternative decoupling methods**, none of which fully resolved the issue of gradient conflict. These results and analyses underscored the need for a stronger decoupling approach, ultimately leading us to the core idea of DARE—back-propagating the gradient to different embeddings. After extensive experimentation, we concluded that **DARE is the only method capable of completely resolving the conflict** while achieving state-of-the-art performance. Details are provided in Appendix C, "THE RESEARCH PROCESS LEADING TO DARE."

2. **Analysis of Retrieval Number (K)**

We analyzed the impact of the number of behaviors retrieved (represented as K in our paper) and demonstrated that DARE can retrieve more important behaviors. We choose K=20 in the light of other models, and **the advantage of DARE will only be more obvious under constrained retrieval numbers.** Details are shown in Appendix D.1 "EFFECTS OF RETRIEVAL NUMBER IN THE SEARCH STAGE".

3. **Sequence Length Analysis**

We examined the effect of sequence length, showing that **DARE consistently benefits from longer sequences under any settings**, whereas TWIN relies on larger embedding dimension to work with longer sequences in some certain setting. This highlights DARE's distinct advantage. Details are provided in Appendix D.2, "EFFECTS OF SEQUENCE LENGTH".

4. **Comprehensive Gradient Conflict Analysis**

We conducted a more in-depth and rigorous analysis of gradient conflict, **demonstrating its prevalence in TWIN and further validating the necessity of decoupling**. Additional findings are discussed in Appendix E.2, "GRADIENT CONFLICT ON TWIN"

5. **Exploration of Attention and Representation Dimensions**

We investigated various combinations of attention and representation dimensions, offering insights that may guide future research inspired by our work. Details are provided in Appendix D.3, "EFFECTS OF ATTENTION AND REPRESENTATION EMBEDDING DIMENSION".

6. **Additional Metrics for Robust Evaluation**

To strengthen the robustness of our experiments, we incorporated two additional metrics, GAUC and Logloss, into our evaluation. Details can be found in Appendix E.1, "GAUC AND LOGLOSS".

Thank you once again for your insightful comments, which have significantly improved the clarity and comprehensiveness of our work. We look forward to any further feedback you may have.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper is well motivated with insights from a well designed preliminary study. The experiments and comprehensive and the writing is generally clear.

There are some major questions on experimental setup, explanation of empirical results, baseline comparisons, and storage complexity, to which the authors have provided detailed responses, as most reviewers acknowledged.

If accepted, the authors are encouraged to further polish the paper based on the reviews.