Long-Sequence Recommendation Models Need Decoupled Embeddings

## Abstract

Abstract Lifelong user behavior sequences, comprising up to tens of thousands of history behaviors, are crucial for capturing user interests and predicting user responses in modern recommendation systems.
A two-stage paradigm is typically adopted to handle these long sequences: a few relevant behaviors are first searched from the original long sequences via an attention mechanism in the first stage and then aggregated with the target item to construct a discriminative representation for prediction in the second stage.
In this work, we identify and characterize, for the first time, a neglected deficiency in existing long-sequence recommendation models: a single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes.
Initial attempts to address this issue using linear projections—a technique borrowed from language processing—proved ineffective, shedding light on the unique challenges of recommendation models.
To overcome this, we propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are initialized and learned separately to fully decouple attention and representation.
Extensive experiments and analysis demonstrate that DARE provides more accurate search of correlated behaviors and outperforms baselines with AUC gains up to 9‰ on public datasets and notable online system improvements.
Furthermore, decoupling embedding spaces allows us to reduce the attention embedding dimension and accelerate the search procedure by 50% without significant performance impact, enabling more efficient, high-performance online serving.

## 1 Introduction

In recommendation systems, content providers must deliver well-suited items to diverse users. To enhance user engagement, the provided items should align with user interests, as evidenced by their clicking behaviors. Thus, the Click-Through Rate (CTR) prediction for target items has become a fundamental task. Accurate predictions rely heavily on effectively capturing user interests as reflected in their history behaviors. Previous research has shown that longer user histories facilitate more accurate predictions (Pi et al., ). Consequently, long-sequence recommendation models have attracted significant research interest in recent years (Chen et al., ; Cao et al., ).

In online services, system response delays can severely disrupt the user experience, making effcient handling of long sequences within a limited time crucial. A general paradigm employs a two-stage process (Pi et al., ): search (a.k.a. General Search Unit) and sequence modeling (a.k.a. Exact Search Unit). This method relies on two core modules: the attention module(^1^11In this paper, “attention” refers to attention scores—the softmax output that weights each behavior.), which measures the target-behavior correlation, and the representation module, which generates a discriminative representation of behaviors. The search stage uses the attention module to retrieve the top-k relevant behaviors, constructing a shorter sub-sequence from the original long behavior sequence(^2^22The search stage can also be “hard” selecting behaviors by category, but we focus on soft search based on learned correlations for better user interest modeling.). The sequence modeling stage then relies on both modules to predict user responses by aggregating behavior representations in the sub-sequence based on their attention, thus extracting a discriminative representation involving both behaviors and the target. Existing works widely adopt this effective paradigm (Pi et al., ; Chang et al., ; Si et al., ).

Attention is critical in the long-sequence recommendation, as it not only models the importance of each behavior for sequence modeling but, more importantly, *determines which behaviors are selected in the search stage*. However, in most existing works, the attention and representation modules share the same embeddings despite serving distinct functions—one learning correlation scores, the other learning discriminative representations. Our analysis reveals that, unfortunately, gradients of these shared embeddings are dominated by representation learning during training, and more concerning, gradient directions from two modules tend to conflict with each other. As a result, attention fails to capture behavior importance accurately, causing *key behaviors to be mistakenly filtered out during the search stage* (as shown in Sec. 4.3). Furthermore, gradient conflicts also degrade the discriminability of the representations (as shown in Sec. 4.4).

Inspired by the use of separate query, key (for attention), and value (for representation) projection matrices in the original self-attention mechanism (Vaswani et al., ), we experimented with attention- and representation-specific projections in recommendation models, aiming to resolve conflicts between these two modules. However, this approach did not yield positive results and led to over-confidence in attention (as shown in Sec. 2.3). Through insightful empirical analysis, we hypothesize that the failure is due to the significantly lower capacity (i.e., fewer parameters) of the projection matrices in recommendation models compared to those in natural language processing (NLP). This limitation is difficult to overcome, as it stems from the low embedding dimension imposed by interaction collapse theory (Guo et al., ).

Figure: Figure 1: Overview of our work. During search, only a limited number of important behaviors are retrieved according to their attention scores. During sequence modeling, the selected behaviors are aggregated into a discriminative representation for prediction. Our DARE model decouples the embeddings used in attention calculation and representation aggregation, effectively resolving their conflict and leading to improved performance and faster inference speed.
Refer to caption: x1.png

To address these issues, we propose the Decoupled Attention and Representation Embeddings (DARE) model, which completely decouples these two modules at the embedding level by using two independent embedding tables—one for attention and the other for representation. This decoupling allows us to fully optimize attention to capture correlation and representation to enhance discriminability. Furthermore, by separating the embeddings, we can accelerate the search stage by 50% by reducing the attention embedding dimension to half, with minimal impact on performance. On the public Taobao and Tmall long-sequence datasets, DARE outperforms the state-of-the-art TWIN model across all embedding dimensions, achieving AUC improvements of up to 9‰. Online evaluation on one of the world’s largest online advertising platforms achieves a 1.47% lift in GMV (Gross Merchandise Value). Our contribution can be summarized as follows:

- •
We identify the issue of interference between attention and representation learning in existing long-sequence recommendation models and demonstrate that linear projections borrowed from NLP fail to decouple these two modules effectively.
- •
We propose the DARE model, which uses module-specific embeddings to fully decouple attention and representation. Our comprehensive analysis shows that our model significantly improves attention accuracy and discriminability of representations.
- •
Our model achieves state-of-the-art on two public datasets and gets a 1.47% GMV lift in one of the world’s largest recommendation systems. Additionally, our method can largely accelerate the search stage by reducing decoupled attention embedding size.

## 2 An In-Depth Analysis into Attention and Representation

In this section, we first review the general formulation for long-sequence recommendation. Then, we analyze the training of shared embeddings, highlighting the domination and conflict of gradients from the attention and representation modules. Finally, we explore why straightforward approaches using module-specific projection matrices fail to address the issue.

### 2.1 Preliminaries

#### Problem formulation.

We consider the fundamental task, Click-Through Rate (CTR) prediction, which aims to predict whether a user will click a specific target item based on the user’s behavior history. This is typically formulated as binary classification, learning a predictor $f:\mathcal{X}\mapsto[0,1]$ given a training dataset $\mathcal{D}=\{(\mathbf{x}_{1},y_{1}),\dots,(\mathbf{x}_{|\mathcal{D}|},y_{|
\mathcal{D}|})\}$, where $\mathbf{x}$ contains a sequence of items representing behavior history and another single item representing the target.

#### Long-sequence recommendation model.

To satisfy the strictly limited inference time in online services, current long-sequence recommendation models generally construct a short sequence first by retrieving top-k correlated behaviors. The attention scores are measured by the scaled dot product of behavior and target embedding. Formally, the $i$-th history behavior and target $t$ is embedded into $\bm{e}_{i}$ and $\bm{v}_{t}\in\mathbb{R}^{d}$, and without loss of generality, $1,2,\dots,K=\mathrm{argsort}(\langle\bm{e}_{i},\bm{v}_{t}\rangle,i\in\left[1,N
\right])$, where $\langle\cdot,\cdot\rangle$ stands for dot product.
Then the weight of each behavior $w_{i}$ is calculated using softmax function: $w_{i}=\frac{e^{\langle\bm{e}_{i},\bm{v}_{t}\rangle/\sqrt{d}}}{\sum_{j=1}^{K}e^
{\langle\bm{e}_{j},\bm{v}_{t}\rangle/\sqrt{d}}}$. Finally, the representations of retrieved behaviors are compressed into $\bm{h}=\sum_{i=1}^{K}w_{i}\cdot\bm{e}_{i}$. TWIN (Chang et al., ) follows this structure and achieves state-of-the-art performance through exquisite industrial optimization.

### 2.2 Gradient Analysis of Domination and Conflict

Figure: Figure 2: The magnitude of embedding gradients from the attention and representation modules.
Refer to caption: x2.png

The attention and representation modules have distinct goals: the former focuses on learning correlation scores for behaviors, while the latter focuses on learning discriminative (i.e., separable) representations in a high-dimensional space. However, current methods use a shared embedding for both tasks, which may prevent either from being fully achieved. To validate this assumption, we analyze the gradients from both modules on the shared embeddings.

#### Experimental validation.

We empirically observe the gradients back propagated to the embeddings from the attention and representation modules. Comparing their norms, we find that gradients from the representation are five times larger, dominating those from attention, as demonstrated in Fig. 3. Then, we further explore whether they can be consistent with each other by analyzing gradient directions. Unfortunately, results show that in nearly two-thirds of cases, the cosine of their angles is negative, indicating the conflict between them, as shown in Fig. 3. In summary, the attention module and representation modules optimize the embedding table towards different directions with varying intensities during training, causing attention to lose correlation accuracy and representation to fail to fully fulfill its discriminability. Notably, due to domination, such influence is more severe to attention, as indicated by the poor learned correlation between categories in Sec. 4.3.

Figure: (a) Attention in TWIN
Refer to caption: x4.png

### 2.3 Recommendation Models Call for More Powerful Decoupling Methods

Figure: Figure 5: The dispersed distribution of attention scores in TWIN w/ proj.
Refer to caption: x6.png

#### Linear projections would cause dispersed distribution of attention score.

To address such conflict, a straightforward approach is to use separate projections for attention and representation, mapping the original embeddings into two new decoupled spaces. This is adopted in the standard self-attention mechanism (Vaswani et al., ), which introduces query, key (for attention), and value projection matrices (for representation). Inspired by this, we propose a variant of TWIN that utilizes linear projections to decouple attention and representation modules, named TWIN w/ proj.. The comparison with the original TWIN structure is shown in Fig. 4a and 4b. Surprisingly, linear projection, which works well in NLP, loses efficacy in recommendation systems, leading to negative performance impact, as shown in Tab. 4c. An analysis of attention logits distribution in Fig. 5 shows that the logits of the TWIN w/ proj have a very dispersed distribution. For more analysis, refer to Sec. 4.3.

Figure: Figure 6: The influence of linear projections with different embedding dimensions in NLP.
Refer to caption: x7.png

#### Larger embedding dimension makes linear projection effective in NLP.

The failure of introducing projection matrices makes us wonder why it works well in NLP but not in recommendation. One possible reason is that the relative capacity of projection matrices regarding the token numbers in NLP is usually large, *e.g.*, with an embedding dimension of 4096 in LLaMA3.1 (Dubey et al., ), there are around 16 million parameters ($4096\times 4096=16,777,216$) in each projection matrix to map only 128,000 tokens in the vocabulary. To validate our hypothesis, we conduct a synthetic experiment in NLP using nanoGPT () with the Shakespeare dataset. In particular, we decrease its embedding dimension from 128 to 2 and check the performance gap between two models with/without projection matrix. As shown in Fig. 6, we observe that when the matrix has enough capacity, *i.e.*, the embedding dimension is larger than 16, projection leads to significantly less loss. However, when the matrix capacity is further reduced, the gap vanishes. Our experiment indicates that projection matrix only works with enough capacity.

#### Limited embedding dimension makes linear projections fail in recommendation.

In contrast, due to the interaction collapse theory (Guo et al., ), the embedding dimension in recommendation is usually no larger than 200, leading to only up to $40000$ parameters for each matrix to map millions to billions of IDs.
Therefore, *the projection matrices in recommendation never get enough capacity, making them unable to decouple attention and representation*.

## 3 DARE: Decoupled Attention and Representation Embeddings

In the above analysis, we find that using projection matrices is insufficient to decouple the attention and representation. To this end, we propose to decouple these two modules at the embedding level; that is, we employ two embedding tables, one for attention ($\bm{E}^{\text{Att}}$) and another for representation ($\bm{E}^{\text{Repr}}$). With gradient back propagated to different embedding tables, our method has the potential to fully resolve the gradient domination and conflict between these two modules. We introduce our model specifically in this section and verify its advantage by experiments in the next section.

### 3.1 Attention Embedding

Attention measures the correlation between history behaviors and the target (Zhou et al., ). Following the common practice, we use the scaled dot-product function (Vaswani et al., ). Mathematically, the $i$-th history behavior $i$ and target $t$, are embedded into $\bm{e}_{i}^{\text{Att}},\bm{v}_{t}^{\text{Att}}\sim\bm{E}^{\text{Att}}$, where $\bm{E}^{\text{Att}}$ is the attention embedding table. After retrieval $1,2,\dots,K=\mathrm{argsort}(\langle\bm{e}_{i},\bm{v}_{t}\rangle,i\in\left[1,N
\right])$ their weight $w_{i}$ is formalized as:

$$ $w_{i}=\frac{e^{\langle\bm{e}_{i}^{\text{Att}},\bm{v}_{t}^{\text{Att}}\rangle/ \sqrt{|\bm{E}^{\text{Att}}|}}}{\sum_{j=1}^{K}e^{\langle\bm{e}_{j}^{\text{Att}} ,\bm{v}_{t}^{\text{Att}}\rangle/\sqrt{|\bm{E}^{\text{Att}}|}}},$ (1) $$

where $\langle\cdot,\cdot\rangle$ stands for dot product and $|\bm{E}^{\text{Att}}|$ stands for the embedding dimension.

Figure: Figure 7: Architecture of the proposed DARE model. One embedding is responsible for attention, learning the correlation between the target and history behaviors, while another is responsible for representation, learning discriminative representations for prediction. Decoupling these two embeddings allows us to resolve the conflict between the two modules.
Refer to caption: x8.png

### 3.2 Representation Embedding

In the representation part, another embedding table $\bm{E}^{\text{Repr}}$ is used, where $i$ and $t$ is embedded into $\bm{e}_{i}^{\text{Repr}},\bm{v}_{t}^{\text{Repr}}\sim\bm{E}^{\text{Repr}}$. Most existing methods multiply the attention weight with the representation of each retrieved behavior and then concatenate it with the embedding of the target as the input of Multi-Layer Perceptron (MLP): $[\sum_{i}w_{i}\bm{e}_{i},\bm{v}_{t}]$.
However, it has been proved that MLP struggle to effectively learn dot product or explicit interactions (Rendle et al., ; Zhai et al., ).
To enhance the discriminability, following TIN (Zhou et al., ), we adopt the target-aware representation $\bm{e}_{i}^{\text{Repr}}\odot\bm{v}_{t}^{\text{Repr}}$, which abbreviate as TR in our following paper (refer to Sec. 4.4 for empirical evaluation of discriminability).

The overall structure of our model is shown in Fig. 7. Formally, user history $h$ is compressed into:

$$ $\bm{h}=\sum_{i=1}^{K}w_{i}\cdot(\bm{e}_{i}^{\text{Repr}}\odot\bm{v}_{t}^{\text {Repr}}).$ (2) $$

### 3.3 Inference Acceleration

By decoupling the attention and representation embedding tables, the dimension of attention embeddings $\bm{E}^{\text{Att}}$ and the dimension of representation embeddings $\bm{E}^{\text{Repr}}$ have more flexibility.
In particular, we can reduce $\bm{E}^{\text{Att}}$ while keeping $\bm{E}^{\text{Repr}}$ to accelerate the searching over the original long sequence whilst not affecting the model’s performance.
Empirical experiments in Sec. 4.5 shows that our model have the potential to speed up searching by 50% with quite little influence on performance and even by 75% with an acceptable performance loss.

### 3.4 Discussion

Figure: Figure 8: Illustration of the TWIN-4E model.
Refer to caption: x9.png

Considering the superiority of decoupling the attention and representation embeddings, one may naturally raise an idea: we can further decouple the embeddings of history and target within the attention (and representation) module, i.e. forming a TWIN with 4 Embeddings method, or TWIN-4E in short, consisting of attention-history (named keys in NLP) $\bm{e}_{i}^{\text{Att}}\in\bm{E}^{\text{Att-h}}$, attention-target (named querys in NLP) $\bm{v}_{t}^{\text{Att}}\in\bm{E}^{\text{Att-t}}$, representation-history (named values in NLP) $\bm{e}_{i}^{\text{Repr}}\in\bm{E}^{\text{Repr-h}}$ and representation-target $\bm{v}_{t}^{\text{Repr}}\in\bm{E}^{\text{Repr-t}}$. The structure of TWIN-4E is shown in Fig. 8.
Compared to our DARE model, TWIN-4E further decouples the behaviors and the target, meaning that the same category or item has two totally independent embeddings as behavior and target.
This is strongly against two prior knowledges in recommendation system. 1. The correlation of two behaviors is similar no matter which is target and which is behavior. 2. Behaviors with the same category should be more correlated, which is natural in DARE since a vector’s dot product with itself tends to be bigger.

## 4 Experiments

### 4.1 Setup

#### Datasets and task.

We use the publicly available Taobao (Zhu et al., ; ; Zhuo et al., ) and Tmall (Tianchi, ) datasets, which provide users’ behavior data over specific time periods on their platforms. Each dataset includes the items users clicked, represented by item IDs and their corresponding category IDs. Thus, a user’s history is modeled as a sequence of item and category IDs. The model’s input consists of a recent, continuous sub-sequence of the user’s lifelong history, along with a target item. For positive samples, the target items are the actual items users clicked next, and the model is expected to output “Yes.” For negative samples, the target items are randomly sampled, and the model should output “No.” In addition to these public datasets, we validated our performance on one of the world’s largest online advertising platforms. More datasets information and details such as training/validation/test splits are shown in Appendix B.1.

#### Baselines.

We compare against a variety of recommendation models, including ETA (Chen et al., ), SDIM (Cao et al., ), DIN (Zhou et al., ), TWIN (Chang et al., ) and its variants, as well as TWIN-V2 (Si et al., ). As discussed in Sec.3.2, the target-aware representation by crossing $\bm{e}_{i}^{\text{Repr}}\odot\bm{v}_{t}^{\text{Repr}}$ significantly improves representation discriminability, so we include it in our baselines for fairness. TWIN-4E refers to the model introduced in Sec. 3.4, while TWIN (w/ proj.) refers to the model described in Sec. 2.3. TWIN (hard) represents a variant using “hard search” in the search stage, meaning it only retrieves behaviors with the same category as the target. TWIN (w/o TR) refers to the original TWIN model without target-aware representation, *i.e.*, representing user history as $\bm{h}=\sum_{i}w_{i}\cdot\bm{e}_{i}$ instead of $\bm{h}=\sum_{i}w_{i}(\bm{e}_{i}\odot\bm{v}_{t})$.

### 4.2 Overall Performance

In recommendation systems, it is well-recognized that even increasing AUC by 1‰ to 2‰ is more than enough to bring online profit. As shown in Tab. 1, our model achieves AUC improvements of 1‰ and 9‰ compared to current state-of-the-art methods across all settings with various embedding sizes. In particular, significant AUC lifts of 9‰ and 6‰ are witnessed with an embedding dimension of 16 on Taobao and Tmall datasets, respectively.

There are also some notable findings. TWIN outperforms TWIN (w/o TR) in most cases, proving that target-aware representation $\bm{e}_{i}^{\text{Repr}}\odot\bm{v}_{t}^{\text{Repr}}$ do help enhance discriminability (further evidence shown in Sec. 4.4). Our DARE model has an obvious advantage over TWIN-4E, confirming that the prior knowledge discussed in Sec. 3.4 is well-suited for the recommendation system. ETA and SDIM, which are based on TWIN and focus on accelerating the search stage at the expense of performance, understandably show lower AUC scores. TWIN-V2, a domain-specific method optimized for video recommendations, is less effective in our settings.

**Table 1: Overall comparison reported by the means and standard deviations of AUC. The best results are highlighted in bold, while the previous best model is underlined. Our model outperforms all existing methods with obvious advantages, especially with small embedding dimensions.**
| Setup | Embedding Dim. = 16 | Embedding Dim. = 64 | Embedding Dim. = 128 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | Taobao | Tmall | Taobao | Tmall | Taobao | Tmall |
| ETA () | $\underset{(0.00338)}{0.91326}$ | $\underset{(0.00108)}{0.95744}$ | $\underset{(0.00079)}{0.92300}$ | $\underset{(0.00042)}{0.96658}$ | $\underset{(0.00032)}{0.92480}$ | $\underset{(0.00039)}{0.96956}$ |
| SDIM () | $\underset{(0.00103)}{0.90430}$ | $\underset{(0.00069)}{0.93516}$ | $\underset{(0.00085)}{0.90854}$ | $\underset{(0.00093)}{0.94110}$ | $\underset{(0.00119)}{0.91108}$ | $\underset{(0.00081)}{0.94298}$ |
| DIN () | $\underset{(0.00060)}{0.90442}$ | $\underset{(0.00037)}{0.95894}$ | $\underset{(0.00092)}{0.90912}$ | $\underset{(0.00033)}{0.96194}$ | $\underset{(0.00054)}{0.91078}$ | $\underset{(0.00013)}{0.96428}$ |
| TWIN () | $\underset{(0.00211)}{\underline{0.91688}}$ | $\underset{(0.00073)}{0.95812}$ | $\underset{(0.00052)}{\underline{0.92636}}$ | $\underset{(0.00039)}{\underline{0.96684}}$ | $\underset{(0.00056)}{\underline{0.93116}}$ | $\underset{(0.00005)}{\underline{0.97060}}$ |
| TWIN (hard) | $\underset{(0.00053)}{0.91002}$ | $\underset{(0.00024)}{0.96026}$ | $\underset{(0.00048)}{0.91984}$ | $\underset{(0.00042)}{0.96448}$ | $\underset{(0.00055)}{0.91446}$ | $\underset{(0.00019)}{0.96712}$ |
| TWIN (w/ proj.) | $\underset{(0.00351)}{0.89642}$ | $\underset{(0.00088)}{0.96152}$ | $\underset{(0.00437)}{0.87176}$ | $\underset{(0.00403)}{0.95570}$ | $\underset{(0.02022)}{0.87990}$ | $\underset{(0.00194)}{0.95724}$ |
| TWIN (w/o TR) | $\underset{(0.00063)}{0.90732}$ | $\underset{(0.00057)}{\underline{0.96170}}$ | $\underset{(0.00083)}{0.91590}$ | $\underset{(0.00032)}{0.96320}$ | $\underset{(0.00084)}{0.92060}$ | $\underset{(0.00103)}{0.96366}$ |
| TWIN-V2 () | $\underset{(0.00077)}{0.89434}$ | $\underset{(0.00110)}{0.94714}$ | $\underset{(0.00063)}{0.90170}$ | $\underset{(0.00037)}{0.95378}$ | $\underset{(0.00059)}{0.90586}$ | $\underset{(0.00045)}{0.95732}$ |
| TWIN-4E | $\underset{(0.01329)}{0.90414}$ | $\underset{(0.00026)}{0.96124}$ | $\underset{(0.01505)}{0.90356}$ | $\underset{(0.0004)}{0.96372}$ | $\underset{(0.01508)}{0.90946}$ | $\underset{(0.01048)}{0.96016}$ |
| DARE (Ours) | $\underset{(0.00025)}{\textbf{0.92568}}$ | $\underset{(0.00024)}{\textbf{0.96800}}$ | $\underset{(0.00046)}{\textbf{0.92992}}$ | $\underset{(0.00012)}{\textbf{0.97074}}$ | $\underset{(0.00045)}{\textbf{0.93242}}$ | $\underset{(0.00016)}{\textbf{0.97254}}$ |

### 4.3 Attention Accuracy

Mutual information, which captures the shared information between two variables, is a powerful tool for understanding relationships in data. We calculate the mutual information between behaviors and the target as the ground truth correlation, following  (Zhou et al., ). The learned attention score reflects model’s measurement of the importance of each behavior. Therefore, we compare the attention distribution with mutual information in Fig. 9.

In particular, Fig. 9a presents the mutual information between a target category and behaviors with top-10 categories and their target-relative positions (i.e., how close to the target is the behavior across time). We observe *a strong semantic-temporal correlation*: behaviors from the same category as the target (5th row) are generally more correlated, with a noticeable temporal decay pattern. Fig. 9b presents TWIN’s learned attention scores, which show a decent temporal decay pattern but *over-estimate the semantic correlation of behaviors across different categories*, making it too sensitive to recent behaviors, even those from unrelated categories. In contrast, *our proposed DARE can effectively capture both the temporal decaying and semantic patterns*.

The retrieval in the search stage relies entirely on attention scores. Thus, we further investigate the retrieval results on the test dataset, which provide a more intuitive reflection of attention quality. Behaviors with top-k mutual information are considered the ground truth for optimal retrieval, and we evaluate model performance using normalized discounted cumulative gain (NDCG) (Järvelin & Kekäläinen, ). The results, along with case studies, are presented in Fig. 10. We find that:

- •
*DARE achieves significantly better retrieval.* As shown in Fig. 10a, the NDCG of our model is substantially higher than all baselines, with a 46.5% increase (0.8124 vs. 0.5545) compared to TWIN and a 27.3% increase (0.8124 vs. 0.6382) compared to DIN.
- •
*TWIN is overly sensitive to temporal information.* As discussed, TWIN tends to select recent behaviors regardless of their categories, against the ground truth, due to overestimated correlations between different categories, as shown in Fig. 10b and 10c.
- •
*TWIN with projection performs unstably.* As shown in Sec. 2.3, the TWIN variant with projection has a highly dispersed attention logits distribution, with maximum logits often exceeding 10. This excessive “confidence” can be a double-edged sword. In some cases, like Fig. 10b, it performs exceptionally well; but in others, like Fig. 10c, it fails due to an “overconfident misjudgment” of an unimportant behavior.

Figure: (a) GT mutual information
Refer to caption: x10.png

Figure: (a) Retrieval on Taobao
Refer to caption: x13.png

### 4.4 Representation Discriminability

We then analyze the discriminability of learned representation. Specifically, on test datasets, we take the compressed representation of user history $\bm{h}=\sum_{i=1}^{K}w_{i}\cdot(\bm{e}_{i}\odot\bm{v}_{t})$, which forms a vector for each test sample. Using K-means, we quantize these vectors, mapping each $\bm{h}$ to a cluster $Q(\bm{h})$. Then, the mutual information (MI) between the discrete variable $Q(\bm{h})$ and label $Y$ (whether the target was clicked or not) reflects the discriminability of the representation. Mathematically, $\text{Discriminability}(\bm{h},Y)=\text{MI}(Q(\bm{h}),Y)$.

As shown in Fig. 11a, across various numbers of clusters, our DARE model outperforms the state-of-the-art TWIN model, demonstrating that decoupling improves representation discriminability.
There are also other notable findings. Although DIN achieves more accurate retrieval in the search stage (as evidenced by a higher NDCG in Fig. 10a), its representation discriminability is obviously lower than TWIN, especially on Taobao dataset, which explains its lower overall performance. TWIN-4E shows comparable discriminability to our DARE model, further confirming that its poorer performance is due to inaccurate attention caused by the lack of recommendation-specific prior knowledge.

To fully demonstrate the effectiveness of $\bm{e}_{i}\odot\bm{v}_{t}$, we compare it with the classical concatenation $[\Sigma_{i}\bm{e}_{i},\bm{v}_{t}]$. As shown in Fig. 11c, a huge gap (in orange) is caused by the target-aware representation, while smaller gaps (in blue and green) result from decoupling. Notably, our DARE model also outperforms TWIN even when using concatenation.

Figure: (a) Discriminability on Taobao
Refer to caption: x16.png

Figure: (a) Training on Taobao
Refer to caption: x19.png

### 4.5 Convergence and Efficiency

#### Faster convergence during training.

In recommendation systems, faster learning speed means the model can achieve strong performance with less training data, which is especially crucial for online services. We track accuracy on the validation dataset during training, shown in Fig. 12a. Our DARE model converges significantly faster. For example, on the Tmall dataset, TWIN reaches 90% accuracy after more than 1300 iterations. In contrast, our DARE model achieves comparable performance in only about 450 iterations—one-third of the time required by TWIN.

#### Efficient search during inference.

By decoupling the attention embedding space $\bm{e}_{i},\bm{v}_{t}\in\mathbb{R}^{K_{A}}$ and representation embedding space $\bm{e}_{i},\bm{v}_{t}\in\mathbb{R}^{K_{R}}$, we can assign different dimensions for these two spaces. Empirically, we find that the attention module performs comparably well with smaller embedding dimensions, allowing us to reduce the size of the attention space ($K_{A}\ll K_{R}$) and significantly accelerate the search stage, as its complexity is $O(K_{A}N)$ where $N$ is the length of the user history. Using $K_{A}=128$ as a baseline (“1”), we normalize the complexity of smaller embedding dimensions. Fig. 12c shows that our model can accelerate the searching speed by 50% with quite little influence on performance and even by 75% with an acceptable performance loss, offering more flexible options for practical use. In contrast, TWIN experiences a significant AUC drop when reducing the embedding dimension.

### 4.6 Online A/B Testing and Deployments

We apply our methods to one of the world’s largest online advertising platforms.
In online advertising, users’ behaviors on ads are sparse, making the sequence length relatively shorter than the content recommendation scenario. To this end, we involve the user’s behavior sequence from our article and the micro-video recommendation scenario.
Specifically, the user’s ad and content behaviors in the last two years are introduced. Before the search, the maximal length of the ads and content sequence is 4000 and 6000, respectively, while the average length is 170 and 1500, respectively. After searching using the proposed DARE, the sequence length is reduced to less than 500.

Regarding sequence features (side info), we choose the category ID, behavior type ID, scenario ID, and two target-aware temporal encodings, *i.e.*, position relative to the target, and time interval relative to the target (with discretization).
There are about 1.0 billion training samples per day.
During the 5-day online A/B test in September 2024, the proposed DARE method achieves 0.57% cost, and 1.47% GMV (Gross Merchandize Value) lift over the production baseline of TWIN.
This would lead to hundreds of millions of dollars in revenue lift per year.

### 4.7 Short-Sequence modeling

We have also tried our method in the short-sequence modeling, using the Amazon dataset (He & McAuley, ; McAuley et al., ) with the same setup as the state-of-the-art TIN model (Zhou et al., ).
However, the performance improvement is marginal (TIN: 0.86291$\pm$0.0015 AUC vs. DARE: 0.86309$\pm$0.0004 AUC).
This is likely because, unlike long-sequence modeling, short-sequence modeling lacks a search stage.
As shown by Zhou et al. (), representation is more critical than attention in short-sequence settings, so *the dominance of representation doesn’t significantly impact performance as long as all behaviors are preserved.*
In contrast, in long-sequence modeling, representation dominance affects retrieval during the search stage, causing some correlated behaviors to be filtered out and their representation lost.

## 5 Related Work

#### Click-through rate prediction and long-sequence modeling.

CTR prediction is fundamental in recommendation systems, as user interest is often reflected in their clicking behaviors. Deep Interest Network (DIN) (Zhou et al., ) introduces target-aware attention, using an MLP to learn attentive weights of each history behavior regarding a specific target. This framework has been extended by models like DIEN (Zhou et al., ), DSIN (Feng et al., ), and BST (Chen et al., ) to capture user interests better. Research has proved that longer user histories lead to more accurate predictions, bringing long-sequence modeling under the spotlight. SIM (Pi et al., ) introduces a search stage (GSU), greatly accelerating the sequence modeling stage (ESU). Models like ETA (Chen et al., ) and SDIM (Cao et al., ) further improve this framework. Notably, TWIN (Chang et al., ) and TWIN-V2 (Si et al., ) unify the target-aware attention metrics used in both stages, significantly improving search quality. However, as pointed out in Sec. 2.2, in all these methods, attention learning is often dominated by representation learning, creating a significant gap between the learned and actual behavior correlations.

#### Attention.

The attention mechanism, first introduced in Transformers (Vaswani et al., ), has proven highly effective and is widely used for correlation measurement. Transformers employ Q, K (attention projection), and V (representation projection) matrices to generate queries, keys, and values for each item. The scaled dot product of query and key serves as the correlation score, while the value serves as the representation. This structure is widely used in many domains, including natural language processing (Brown, ) and computer vision (Dosovitskiy, ). However, in recommendation systems, due to the interaction-collapse theory pointed out by Guo et al. (), the small embedding dimension would make linear projections completely lose effectiveness, as discussed in Sec. 2.3. Thus, proper adjustment is needed in this specific domain.

## 6 Conclusion

This paper focuses on long-sequence recommendation, starting with an analysis of gradient domination and conflict on the embeddings.
We then propose a novel Decoupled Attention and Representation Embeddings (DARE) model, which fully decouples attention and representation using separate embedding tables.
Both offline and online experiments demonstrate DARE’s potential, with comprehensive analysis highlighting its advantages in attention accuracy, representation discriminability, and faster inference speed.

## Appendix A Implementation Details

### A.1 Hyper-parameters and Model Details

The hyper-parameters we use are listed as follows:

| Parameter | Value |
| --- | --- |
| Epoch | 2 |
| Batch size | 2048 |
| Learning rate | 0.01 |
| Weight decay | $10^{-6}$ |

The search stage retrieves 20 behaviors. Besides, we use the Adam optimizer. Layers of the Multi-layer Perceptron (MLP) are set as $200\times 80\times 2$, which is the same as Zhou et al. ().

These settings remain the same in all our experiments.

### A.2 Baseline Implementation

Many current methods are not open-source and may focus on a certain domain. Thus, we followed their idea and implemented their method according to our task setting. Some notable details are shown as follows:

- •
DIN is a short-sequence modeling method, so we introduced the search stage and aligned it with long-sequence models. To be specific, the original DIN will aggregate all history behaviors based on a learned weight, so we let DIN also select top-K behaviors like other methods according to its learned weight. Note that the original DIN is not practical in long-sequence modeling, since aggregating such a long history is unacceptable due to the too high time complexity.
- •
TWIN-V2 is specially designed for Kuaishou, a short video-sharing app, capturing video-specific features to make certain optimization, so we retained their core ideas and adjusted to our scenarios. We made some necessary adjustments, e.g., TWIN-V2 would first group the items based on the proportion a video is played, which does not have a corresponding feature in our datasets. So, we first group user history based on temporal information.

## Appendix B Data Processing

### B.1 Dataset Details

#### Dataset information.

Some detailed information is shown in Table 2. We use Taobao (Zhu et al., ; ; Zhuo et al., ) and Tmall (Tianchi, ) datasets in our experiments. The proportion of active users in these two datasets is more than 60%, which is relatively more satisfying. Note that Taobao is relatively more complex, with more categories and more items, which is a higher challenge for model capacity.

**Table 2: Some basic information of public datasets (active user: User with more than 50 behaviors).**
| Dataset | #Category | #Item | #User | # Active User |
| --- | --- | --- | --- | --- |
| Taobao | 9407 | 4,068,790 | 984,114 | 603,176 |
| Tmall | 1492 | 1,080,666 | 423,862 | 246,477 |

#### Training-validation-test split.

We sequentially number history behaviors from one (the most recent behavior) to T (the most ancient behavior) according to the time step. The test dataset contains predictions of the first behaviors, while the second behaviors are used as the validation dataset. For the training dataset, we use the $(3+5i,0\leq i\leq 18)$th behavior. Models would finish predicting the $j$th behavior based on $j-200$ to $j-1$ behaviors (padding if history length is not long enough). Only active users with behavior sequences longer than 210 will be reserved.

We make such settings to balance the amount and quality of training data. In our setting, each selected user would contribute 20 pieces of data visible to our model in the training process. Besides, we can guarantee that each piece of test data would contain no less than 200 behaviors, making our results more reliable. To some degree, we break the “independent identical distribution” principle because we sample more than one piece of data from one user. However, it’s unavoidable since the dataset is not large enough due to the feature of the recommendation system (item number is usually several times bigger than user number), so we sample with interval 5, using the $((3+5i)\textit{th},0\leq i\leq 18)$ behaviors as the training dataset.

## Appendix C Extended Experimental Results

### C.1 Learned Attention

More cases of comparison between ground truth mutual information and learn attention score are shown below. Each line contains three pictures, where the first picture is the ground truth mutual information, while the second and third line is the learned attention score of TWIN and DARE. Our Dare model is closer to the ground truth in all cases.

Figure: (a) GT mutual information
Refer to caption: x23.png

Figure: (d) GT mutual information
Refer to caption: x26.png

Figure: (g) GT mutual information
Refer to caption: x29.png

Figure: (j) GT mutual information
Refer to caption: x32.png

### C.2 Retrieval Performance during Search

More case studies of the retrieval result in the search stage are shown below:

Figure: (m) Case study 1
Refer to caption: x35.png

Figure: (o) Case study 1
Refer to caption: x37.png

## Appendix D Limitation.

There are also some limitations. We empirically find that linear projection only works with higher embedding dimensions, and small embedding dimensions would cause a dispersed distribution of attention logits. However, we still can’t completely find out how this happen or what the underlying reasons are causing this strange phenomenon, which is left to future work. Besides, our AUC result in Section 4.2 indicates that target-aware representation benefits model performance in most cases, leading to an AUC increase of more than 1% on the Taobao dataset. But on the Tmall dataset with embedding dimension = 16, TWIN (w/o TR) outperforms TWIN, which is out of our expectations. This is possibly due to some features of the Tmall dataset (e.g. fewer items), but we could not explain this result convincingly, which is also left to future work.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. What are potential application scenarios other than sequential models with attention and representation?

2. To what extent would DARE increase the storage complexity? I believe it would be doubled.
soundness: 3
strengths: 1. Preliminary study is available. Authors provide experimental evidence supporting that using a shared embedding for attention calculation and preference prediction can impair performance due to the varying magnitudes and conflicting gradients. Furthermore, the authors examine the linear projection solutions of current methods, noting that the constrained embedding size may hinder the effectiveness of linear projection for decoupling. 

2. Clarity. The paper presents its ideas with remarkable clarity, making it not only easy to follow but also engaging for the reader.

3. Experiments. Comprehensive experiments demonstrate the effectiveness of DARE. Various behavioral sequence models, including DIN, SDIM, ETA, and TWIN, are compared. A thorough analysis of attention and representation capabilities, as well as efficiency tests, is presented.
summary: This paper suggests the inefficiency of using the same embeddings for attention calculation and preference prediction (representations) in lifelong sequence recommendations, where the user behavior history (sequence) is usually very long (more than 10,000). Authors correspondingly propose a decoupling framework, DARE, to boost the recommendation performance by decoupling embeddings. Specifically, DARE sets separate embedding tables for attention calculation and preference prediction. Abundant preliminary studies and experimental evaluation support the effectiveness.
weaknesses: 1. Applicability. This work aims to decouple attention and representation embeddings for sequence modeling. I wonder if the proposed solution in Section 2 can be extended to more scenarios. When the attention mechanism is not used in certain sequential models, the suggested method does not apply and, therefore, cannot benefit these models.

2. Inconsistent Solution without Novelty. The proposed solution is too straightforward-setting separate embedding tables for attention and representation. This is not novel and does not align with the findings, although the separate embeddings can resolve the conflicting gradient and are not simply a linear projection. In my view, authors should go beyond linear projection. For instance, a linear projection-based decoupling method that addresses the gradient issue can be applied to small embedding sizes in recommendation systems, rather than reverting to separate embeddings, a very basic solution.

3. Storage Complexity. Authors only provide the inference efficiency. However, for large-scale recommender systems, a separate embedding table can occupy exceptionally large storage space.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1.	Why are the gradients from \textbf{attention} significantly smaller than those from \textbf{representation} in Fig. 2? And why is this imbalance undesirable during training (e.g., does it contradict theoretical derivations)? It is necessary to provide a theoretical analysis here.
2.	The definition of conflicts between attention and representation is quite vague. The cancellation of gradients during backpropagation cannot be explained merely by the sign of the angle. Could you supplement this with the merged gradient values or in other ways to illustrate the degradation caused by conflicts to the embeddings?
3.	What are the consequences of these gradient issues? Do they lead to the failure of representation learning or the malfunction of certain modules? Would you add a presentation or explanation of the outcomes of the mentioned problem？
4.	How does DARE solve the aforementioned gradient issues (gradient magnitude imbalance and gradient conflicts)?
5.	The embedding dimension shown in Figure 6 should reach at least 200 to be consistent with the analysis in lines 208-213. Additionally, the analysis here can be summarized as: linear mapping requires a large embedding dimension to be effective, but increasing the embedding dimension leads to the dimensional collapse problem in recommendation systems. How does DARE address this problem? Does it only work effectively in low dimensions, or does it solve the dimensional collapse problem and work under any embedding capacity?
6.	How are the embeddings between the two modules in DARE aligned? Why can the weights used to measure attention embeddings be directly utilized for representation embeddings?
7.	The concept of Mutual Information （MI） is repeatedly mentioned in the experimental section, so it is necessary to introduce its calculation method. Moreover, this concept is already used in Figure 1, but it is not explained there. The connection between MI and the problem analyzed in this paper (Section 2) needs careful and thorough clarification.
soundness: 3
strengths: The definition of the problem is very clear.
The research perspective is quite innovative.
summary: The paper addresses a key limitation in existing long-sequence recommendation models, which struggle to balance the learning of attention and representation with a single set of embeddings. To tackle this, the authors introduce the Decoupled Attention and Representation Embeddings (DARE) model, which separates these two processes with distinct embeddings. Experiments show that DARE not only improves the accuracy of correlated behavior searches but also boosts efficiency, reducing embedding dimensions and accelerating the search by 50% without significant performance loss.
weaknesses: The definition and explanation of the gradient issue are one-sided, and the motivation and the logic of the method are not clear. For example, why do the impact of different modules on embedding gradients must be ‘equal-magnitude’ and ‘consistent’? How do imbalance-magnitude and inconsistent gradients degrade recommendation performance? Additionally, the connection between the proposed method and the target problem is weak. For example, why does the embedding decoupling approach help mitigate the gradient issues? Overall, the logic and the motivation of the main idea need to be illustrated clearly.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: How was the decision made to use two distinct embedding tables for attention and representation? Were there any other alternative architectures considered and why were they rejected?

In the DARE model, how sensitive is the performance to the choice of embedding dimensions for attention and representation? Have different combinations been explored and what are the trade-offs?
soundness: 3
strengths: 1. The paper clearly identifies a crucial problem in long-sequence recommendation models. The analysis of the interference between attention and representation learning due to shared embeddings is comprehensive.

2. The extensive experiments on public datasets (Taobao and Tmall) and an online advertising platform are a major strength. The comparison with a wide range of baselines, including state-of-the-art models like TWIN and its variants, demonstrates the superiority of the DARE model. The evaluation metrics used, such as AUC, NDCG, and representation discriminability measures, provide a comprehensive assessment of the model's performance.
summary: The paper focuses on long-sequence recommendation models. It first identifies a problem in existing models where a single set of embeddings struggles to handle both attention and representation learning, leading to interference. Initial attempts to address this using linear projections from language processing failed. Then it proposes the Decoupled Attention and Representation Embeddings (DARE) model. This model uses two distinct embedding tables for attention and representation, which are initialized and learned separately. Extensive experiments on public datasets and an online system show its effectiveness.
weaknesses: 1. The performance on the short-sequence modeling using the Amazon dataset was marginal. This suggests that the DARE model may not be as effective in short-sequence scenarios, and more research is needed to understand its applicability in different sequence lengths. Additionally, the experimental setup for some baselines, like DIN, required adjustments to fit the long-sequence context, which may introduce some biases in the comparison.

2. There are some results that lack a satisfactory explanation. For instance, on the Tmall dataset with embedding dimension = 16, TWIN (w/o TR) outperforms TWIN, which was unexpected. The authors could not convincingly explain this result, leaving a gap in the understanding of the model behavior under different dataset characteristics.

3. In the exploration experiment, although the data is somewhat persuasive, some special cases or edge cases may not be fully covered. For example, it is uncertain whether different gradient relationships will occur under certain combinations of specific categories or user behavior patterns. Moreover, there is no further in-depth analysis of the reasons behind why it is nearly two-thirds of the cases, which may cause readers to have some doubts about the universality of the conclusion.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - Q1. Why the performance of TWIN-V2 is worse than TWIN, which does not align with the original paper? Are there some differences in experimental settings?

All of my other questions are included in the weaknesses.
soundness: 3
strengths: - S1. This paper is well-written and -structured, which is easy to follow and understand.
- S2. This paper is well-motivated, where the motivation is from an interesting preliminary study.
- S3. The proposed method is simple but effective.
summary: This paper has focused on the interference between search and aggregation stages in the long-sequence recommendation. The authors first conducted some preliminary experiments and found the unbalanced and adverse gradients brought by the two stages. Inspired by such findings, they propose to adopt two individual embedding sets for the two stages, respectively, which may alleviate the conflicts between the two stages. The experiments on two public datasets validate that the proposed method can benefit the previous two-stage method, i.e., TWIN, and achieve SOTA performance.
weaknesses: - W1.  I can understand the results and conclusion from the preliminary study, but the detailed process of the preliminary study is not clear. For example, I wonder how to calculate the gradients from the attention and representation to the embeddings.
- W2. Only one metric is revealed in the experiments. GAUC and Logloss are often two common metrics in CTR prediction tasks. Thus, I suggest the authors add more metrics.
- W3. The number of filtered behaviors, i.e., K in this paper, is often important to the two-stage method, but no related hyper-parameter experiment is conducted.
- W4. It seems that the proposed method can only fit the two-stage method, such as TWIN. However, there still exist many one-stage works [1], which may illustrate that the two-stage method has not been the optimal solution. If so, the value of this paper is limited.

[1]. Yu, W., Feng, C., Zhang, Y., Hu, L., Jiang, P., & Li, H. (2024). IFA: Interaction Fidelity Attention for Entire Lifelong Behaviour Sequence Modeling. arXiv preprint arXiv:2406.09742.

Meta-review: The paper is well motivated with insights from a well designed preliminary study. The experiments and comprehensive and the writing is generally clear.

There are some major questions on experimental setup, explanation of empirical results, baseline comparisons, and storage complexity, to which the authors have provided detailed responses, as most reviewers acknowledged.

If accepted, the authors are encouraged to further polish the paper based on the reviews.

Decision: Accept (Poster)