Long-Sequence Recommendation Models Need Decoupled Embeddings
Authors: Ningya Feng
Junwei Pan
Jialong Wu
Baixu Chen
Ximei Wang
QianLi
Xian Hu
Jie Jiang
Mingsheng Long
Venue: ICLR 2025 Poster (2025)
Keywords: Recommender System
User Interest Modeling
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Decoupled embeddings help solve the conflict between attention and representation in recommendation models.

Abstract: Lifelong user behavior sequences are crucial for capturing user interests and predicting user responses in modern recommendation systems. A two-stage paradigm is typically adopted to handle these long sequences: a subset of relevant behaviors is first searched from the original long sequences via an attention mechanism in the first stage and then aggregated with the target item to construct a discriminative representation for prediction in the second stage. In this work, we identify and characterize, for the first time, a neglected deficiency in existing long-sequence recommendation models: a single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes. Initial attempts to address this issue with some common methods (e.g., linear projections---a technique borrowed from language processing) proved ineffective, shedding light on the unique challenges of recommendation models. To overcome this, we propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are initialized and learned separately to fully decouple attention and representation. Extensive experiments and analysis demonstrate that DARE provides more accurate searches of correlated behaviors and outperforms baselines with AUC gains up to 9‰ on public datasets and notable improvements on Tencent's advertising platform. Furthermore, decoupling embedding spaces allows us to reduce the attention embedding dimension and accelerate the search procedure by 50\% without significant performance impact, enabling more efficient, high-performance online serving. Code in PyTorch for experiments, including model analysis, is available at https://github.com/thuml/DARE.

Published as a conference paper at ICLR 2025
LONG-SEQUENCE RECOMMENDATION MODELS NEED DECOUPLED EMBEDDINGS
Ningya Feng', Junwei Pan Jialnng BBixu Chen' Ximei Wang", Qian Li,, Xian Hu? Jie Jiang'. Mingsheng Long 153
1 "School of Software, BNRist, Tsinghua University, China Tencent Inc, China fny21dmails tsinghua edu. en, jonaspan&tencooeeeeo com, wufialong02298 com wtwphhhnnnngsiinngnnnn en
ABSTRACT
Lifelong user behavior sequences crucial for capturing user interests and pre- dicting user responses in modern recommendation systems. A two-stage paradigm is typically adopted to handle these long sequences: subset of relevant behav- iors is first searched from the original long sequences via an attention mecha- nism in the first stage and then aggregated with the target item to construct a discriminative representation for prediction in the second stage. In this work, we identify and characterize, for the first time, a neglected deficiency in exist- ing long- sequence recommendation models: a single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes. Initial attempts to address this issue with some common methods (e.g., linear projections- technique borrowed from language process- ing) proved ineffective, shedding light on the unique challenges of recommenda- tion models. To overcome this, we propose the Decoupled Attention and Repre- sentation Embeddings (DARE) model, where two distinct embedding tables are initialized and learned separately to fully decouple attention and representation. Extensive experiments and analysis demonstrate that DARE provides more accu- rate searches of correlated behaviors and outperforms baselines with AUC gains up 9% on public datasets and notable improvements on Tencent's advertising platform. Furthermore, decoupling embedding spaces allows us to reduce the at- tention embedding dimension and accelerate the search procedure by 50% without significant performance impact, enabling more efficient, high-performance online serving. Code PyTorch for experiments, including model analysis, available at hattps://githul com/ httmt//ummmpaa//// DARE
INTRODUCTION
In recommendation systems, content providers must deliver well-suited items to diverse users. To enhance user engagement, the provided items should align with user interests, as evidenced by their clicking behaviors. Thus, the Click-Through Rate (CTR) prediction for target items has become a fundamental task. Accurate predictions rely heavily on effectively capturing user interests as reflected in their history behaviors. Previous research has shown that longer user histories facilitate more accurate predictions (Pi et al. 2020). Consequently, long- sengseeuennee recommendation models have attracted significant research interest recent years (Chen al. 202
Cao al., 2022).
In online services, system response delays can severely disrupt the user experience, making efficient handling of long sequences within limited time crucial. general paradigm employs two-stage process (Pi et al., 2020): search (a.k.a. General Search Unit) and sequence modeling (a.k.a. Exact Search Unit). This method relies on two core modules: the attention module]] which measures the target-behavior correlation, and the representation module, which generates discriminative represen- tations of behaviors. The search stage uses the attention module to retrieve top-k relevant behaviors,
$ Equul contribution. Work was done while Ningya Feng and Baixu Chen were interns at Tencent. In this paper, "attention" refers attention scores the softmax output that weights each hehavior.

---

Published as a conference paper at ICLR 2025
Nettoe Parforance
Attention
Elfowrce undAA Tolbe
Embedding
Target
Benavar Hestory
1:10
Searching
Irgroved
Decoupling
Retrieval
Sequence Modeling
Embedding
Representation
Targel
83
Retroved Balaviors
Prodictor
Click
Not
Figure| 1: Overview of our work. During search, only a limited number of important behaviors retrieved according to their attention scores. During sequence modeling, the selected behaviors are aggregated into a discriminative representation for prediction. Our DARE model decouples the embeddings used in attention calculation and representation aggregation, effectively resolving their conflict and leading to improved performance and faster inference speed.
constructing a shorter sub-sequence from the original long behavior sequence The sequence mod- eling stage relies on both modules to predict user responses by aggregating behavior representations in the sub-sequence based on their attention, thus extracting discriminative representation. Existing works widely adopt this paradigm (Pi et al., 2020 Chang et al. 2023 Si et al. 2024).
Attention is critical in the long-sequence recommendation, as it not only models the importance of each behavior for sequence modeling but, more importantly, determines which behaviors are selected in the search stage. However, in most existing works, the attention and representation mod- ules share the same embeddings despite serving distinct functions- one learning correlation scores, the other learning discriminative representations. We analyze these two modules, for the first time, in the perspective of Multi- Task Learning (MT'L. (Caruana 1997). Adopting gradient analysis com- monly used in MTL (Yu et al. [020 Liu et al., 2021). ). reveal that, unfortunately, gradients of these shared embeddings are dominated by representation, and more concerning, gradient directions from two modules tend to conflict with each other. Domination and conflict of gradients are two ryp- ical phenomena of interference between tasks, inftuencing the model', 's performance on both tasks. Our experimental results are consistent with the theoretical insight: attention fails to capture be- havior importance accurately, causing key behaviors to be mistakenly filtered out during the search stage (as shown in Sec. 4.3). Furthermore, gradient conflicts also degrade the discriminability of the representations (as shown Sec. 4.4
Inspired by the use of separate query, key (for attention), and value (for representation) projec- tion matrices in the original self-attention mechanism (Vaswani et al., 2017), we experimented with attention- and prteessentationspeccffe projections in recommendation models, aiming to resolve con- flicts between these two modules. However, this approach did not yield positive results. We also tried three other kinds of candidate methods, but unfortunately, none of them worked effectively. Through insightful empirical analysis, we hypothesize that the failure is due to the significantly lower capacity (i.e., fewer parameters) of the projection matrices in recommendation models com- pared to those in natural language processing (NLP). This limitation is difficult overcome, as stems from the low embedding dimension imposed by interaction collapse theory (Guo et al. 2024).
To address these issues, we propose the Decoupled Attention and Representation Embeddings (DARE) model, which completely decouples these two modules at the embedding level by using two independent embedding tables-one one for attention and the other for representation. This de- coupling allows to fully optimize attention to capture correlation and representation to enhance discriminability. Furthermore, by separating the embeddings, we can accelerate the search stage by 50% by reducing the attention embedding dimension to half, with minimal impact on performance. On the public Taobao and Tmall long-sequence datasets, DARE outperforms the state-of-the-art TWIN model across all embedding dimensions, achieving AUC improvements of up to 9%c. Online evaluation on Tencent's 's advertising platform, one of the world's largest platforms, achieves a 1.47% lift in GMV (Gross Merchandise Value). Our contribution can be summarized as follows:
The search stage also be "hard" selecting behaviors by category, hut we focus soft search based on learned correlations for better user interest modeling.
2

---

Published as a conference paper at ICLR 2025
We identify the issue of interference between attention and representation learning in exist- ing long-sequence recommendation models and demonstrate that common methods (e.g., linear projections borrowed from NLP) fail to decouple these two modules effectively.
We propose the DARE model, which uses module-specific embeddings to fully decouple attention and representation. Our comprehensive analysis shows that our model signifi- cantly improves attention accuracy and representation discriminability.
Our model achieves state-of-the-ard on two public datasets and gets 1.47% GMV lift in one of the world's largest recommendation systems. Additionally, method can largely accelerate the search stage by reducing decoupled attention embedding size.
AN IN-DEPTH ANALYSIS INTO ATTENTION AND REPRESENTATION
In this section, we first review the general formulation for long-sequence recommendation. Then, we analyze the training of shared embeddings, highlighting the domination and conflict of gradients from the attention and representation modules. Finally, we explore why straightforward approaches (e.g., using module-specific projection matrices) fail to address the issue.
2.1 PRELIMINARIES
Problem formulation. We consider the fundamental task, Click-Thron Through Rate (CTR) prediction, which aims to predict whether user will click specific target item based on the user's behavior history. This typically formulated as binary classification, learning a predictor given training dataset D {(x1,,1),.. vi,,..., (xyD), ) where x contains sequence of items representing behavior history and another single item representing the target.
X' - [o, 1]
Long-sequence recommendation model. To satisfy the strictly limited inference time in online services, current long-sequence recommendation models generally construct a short sequence first by retrieving top-k correlated behaviors. The attention scores measured by the scaled dot product of behavior and target embedding. Formally, the i-th history behavior and target t is embedded into e, and v. IRd, and without loss of generality, 1......, K Top-K((e;; uz), i [1, NJ), where (,, stands for dot product. Then the weight of each behavior w, is calculated using softmax nD///
function: Wi
- e ("iie Finally, the representations of retrieved behaviors are compressed
K into h 2i-1 W TWIN (Chang et al. 2023 follows this structure and achieves state-of-the-art performance through exquisite industrial optimization.
2.2 GRADIENT ANALYSIS OF DOMINATION AND CONFLICT
The attention and representation modules can be seen as two tasks: the former focuses on learning correlation scores for behaviors, while the latter focuses on learning discriminative (i.e., separable) representations in nigh-dimensional space. However, current methods use shared embedding for both tasks, which may cause a similar phenomenon to "task conflict" in Multi-Task Learning (MTL) (Yu et al. 2020 Liu et al. (2021) and prevent either from being fully achieved. To validate this assumption, analyze the gradients from both modules the shared embeddings.
Experimental validation. Following the methods in MTL, we empirically observe the gradients back prop- agated to the embeddings from the attention and repre- sentation modules. Comparing their gradient norms, we find that gradients from the representation are five times larger, dominating those from attention, demonstrated in Fig. 2 Observing their gradient directions, we fur-
Average Magnituds (x10-3)
2.5
2.0
1.5
1.0
0.5
ther find that nearly two-thirds of cases, the cosine of the gradient angles is negative, indicating the conflict be- tween them, as shown in Fig. Domination and conflict are two typical phenomena of task interference, suggest- ing challenges in learning them well.
0.0
Attention
Representation
Gradient Source
Figure 2: The magnitude of embedding gradients from the attention and repre- sentation modules.

---

Published as a conference paper at ICLR 2025
In summary, the attention module and representation modules optimize the embedding table towards different directions with varying intensities during training, caus- ing attention to lose correlation accuracy and representa- tion lose its discriminability. Notably, due to domina- tion, such influence is more severe to attention, as indi- cated by the poor learned correlation between categories in Sec. 4.3 While some commonly used techniques in
15%
Proportion
10%
Conflicting
Consistent
64.31%
35.69%
0%
1.0
0.5
0.0
0.5
MTL may ease the conflict, we tend to seek an optimized model structure that further resolves the conflict.
Cosine Similarity
Figure
Cosine angles of gradients.
Finding 1. In sequential recommenders, gradients from the representation module tend to con- flict with that from the attention module, and typically dominate the embedding gradients.
Scaled Prouuct
Scaled Dot-Product
Attention
Attention
Taobao
Tmall
Linear
Linear
Linear
TWIN 2023
0.91688 (0.00211)
0.95812 (0.00073)
Input Embedding
Input E mbeddinn
TWIN
(W/ proj.)
0.89642 (0.00351)
0.96152 (0.00088)
Target
History
Target
History
(a) Attention in TWIN
(b) TWIN with projection
AUC results of TWIN variants
Illustration and evaluation for adopting linear projections. (a-b) The attention module in the original TWIN and after adopting linear projections. (c) Performance of TWIN variants. Adopting linear projections causes an AUC drop of nearly 2% on Taobao.
Figure 4:
2.3 RECOMMENDATION MODELS CALL FOR MORE POWERFUL DECOUPLING METHODS
Normal decoupling methods fail resolve conflicts. To address such conflict, a straightfor- ward approach is to use separate projections for attention and representation, mapping the original embeddings into two new decoupled spaces. This adopted in the standard self-attention mech- anism (Vaswani et al.] 2017), which introduces query, key (for attention). and value projection matrices (for representation). Inspired by this, we propose a variant of TWIN that utilizes linear projections to decouple attention and representation modules, named TWIN (w/ proj.). The com- parison with the original TWIN structure is shown in Fig. 4a and 4b) Surprisingly, linear projection, which works well in NLP, loses efficacy in recommendation systems, leading to negative performance impact, as shown in Tab. 4c] We also tried three kinds of other candidate methods (MLP-based pro- jection, strengthening the capacity of linear projection, and gradient normalization), resulting in total of eight models, but none of them resolved the conffict effectively. For the structure of these models and more details, refer Appendix C
Larger embedding dimension makes linear projec- tion effective in NLP. The failure of introducing pro- jection matrices makes us wonder why it works well in NLP but not in recommendation. One possible reason is that the relative capacity of projection matrices regarding the token numbers in NLP is usually strong, e.g., with an embedding dimension of 4096 LLaMA3.1 (Dubey let al.) 2024)). there are around 16 million parameters (4096 x 4096 = 16, 777, 216) in each projection matrix to map only 128,000 tokens in the vocabulary. To vali- date our hypothesis, we conduct a synthetic experiment in NLP using nanoGPT (Andrej) with the Shakespeare dataset. In particular, we decrease its embedding dimen- sion from 128 to 2 and check the performance gap be- tween the two models with/without projection matrices. As shown in Fig. 5 we observe that when the matrix has enough capacity, i.e., the embedding dimension is larger than 16, projection leads to significantly less loss. However, when the matrix capacity is further reduced, the gap vanishes. Our experiment indicates that using projection matrices only works with enough capacity.
w/ Linear Projection
3.0
w/o Linear Projection
Loss
2.5
2.0
1.5-
2
8
16
32
64
128
Embedding Dimension
Figure The influence of linear projec- tions with different embedding dimen- sions in NLP.
4

---

Published as a conference paper at ICLR 2025
Retrivve top K
MLP
Result
cale / Softmax
Weighted Sum
Embedding
Embedding
Item
Item N-1
Item
Item
Item K
Item K-1
Item
Item 0
Lifelong Behavior Sequence
Target
Retrieved S Sequence
Target
Figure 6: Architecture of the proposed DARE model. One embedding is responsible for atten- tion. learning the correlation between the target and history behaviors, while another embedding is responsible for representation, learning discriminative representations for prediction. Decoupling these two embeddings allows us to resolve the conflict between the two modules.
Limited embedding dimension makes linear projections fail in recommendation. In contrast, due to the interaction collapse theory (Guo al., 2024), the embedding dimension recommenda- tion is usually no larger than 200, leading to only up to 40000 parameters for each matrix to map millions to billions of IDs. Therefore, the projection matrices in recommendation never get enough capacity, making them unable decouple attention and representation. In this case, other normal decoupling methods mentioned in Appendix also suffer from weak capacity.
Finding Normal methods like linear projection fail to decouple attention and representation in sequential recommendation models due to limited capacity caused by embedding dimension.
DARE: DECOUPLED ATTENTION AND REPRESENTATION EMBEDDINGS
With all eight normal decoupling models shown in Appendix C failed, based on our analysis, we seek methods with enough capacity, hoping to completely resolve the conflict. To this end, we propose to decouple these two modules at the embedding level. That is, employ two embedding tables, one for attention (EAt)) and another for representation Repr.. With gradient back propagated to different embedding tables, method has the potential to fully resolve the gradient domination and conflict between these two modules. We introduce our model specifically this section and demonstrate its advantage by experiments in the next section.
3.1 ATTENTION EMBEDDING
Attention measures the correlation between history behaviors and the target (Zhou al.
2018
Following the common practice, we use the scaled dot dot-product function (Vaswani et al. Mathematically, the i-th history behavior i and target I, are embedded into e nbtH Am where A is the attention embedding table. After retrieval 1,2...., K Top-Ki vi), [1, NJ) their weight wi is formalized as:
2017
(otlll an >/1 E
(1)
K
E
where ( stands for dot product and E stands for the embedding dimension.
3.2 REPRESENTATION EMBEDDING
In the representation part, another embedding table ERopr used, where and is embedded into Repr
e u Repr Most existing methods multiply the attention weight with the representation of each retrieved behavior and then concatenate it with the embedding of the target as the input of
Multi-Layer Perceptron (MLP): [Σ; w.e,, v.]. However, it has been proved that MLP struggles to
effectively learn explicit interactions (Rendle et al. 2020 Zhai et al. 2023). To enhance the discrim-
5

---

Published as a conference paper at ICLR 2025
inability, following TIN (Zhou et al. 2024), adopt the target-aware representation e; denoted as TR in our following paper (refer to Sec. 4.4 for empirical evaluation of discriminability).
Repr Repr
The overall structure of our model is shown in Fig. Formally, user history h is compressed into:
h
K wi (e, (e,40e C vn R
Repe
3.3 INFERENCE ACCELERATION
By decoupling the attention and representation embedding tables, the dimension of attention em- beddings E and the dimension of representation embeddings RRpr have more flexibility. In particular, we can reduce E while keeping ERppr accelerate the searching over the original long sequence whilst not affecting the model's performance. Empirical experiments in Sec. [455 show that model has the potential speed up searching by 50% with quite little influence performance and even by 75% with an acceptable performance loss.
3.4 DISCUSSION
Considering the superiority of decoupling the attention and representation embeddings, one may naturally raise an idea: we can further decouple the embeddings of history and target within the attention (and representation) mod- ule, Le. forming TWIN with 4 Embeddings method, or TWIN-4E in short, consisting of attention-history (named keys in NLP) e.ttt E EAuh tttentioo-tarret (named querys in NLP)
Attention
Representatinn
Embedding (E-TTh)
Embedding (EATT
Embedding (E*:srh)
E Embedding IE Exppr))
History Item
Targetitem
Figure 7: Illustration of the TWIN-4E model.
oA"" E AARR, representation-histor (named values NLP) E target U, model, TWIN-4E further decouples the behaviors and the target, meaning that the same category or item has two totally independent embeddings as behavior and target. This is strongly against two prior knowledge in recommendation system. The correlation of two behaviors is similar no matter which is the target and which is from history. 2. Behaviors with the same category should be more correlated, which natural DARE since a vector's 's dot product with itself tends to bigger.
Repr
Reep-h and representation-
Repe E ERepe- The structure of TWIN-4E is shown in Fig. Compared to our DARE
EXPERIMENTS
4.1 SETUP
Zhuo et al. 2020) and Tmall (Tianchi 2018) datasets, which provide users' behavior data over specific time periods on their platforms. Each dataset includes the items users clicked, represented by item IDs and their corresponding category IDs. Thus, a user's history is modeled as a sequence of item and category IDs. The model's input consists of a recent, continuous sub-sequence of the user's 's lifelong history, along with target item. For positive samples, the target items are the actual items users clicked next, and the model is expected to output "Yes." For negative samples, the target items randomly sampled, and the model should output "No." In addition to these public datasets, validated our performance on one the world's 'ss largest online advertising platforms. More details on datasets and training/validationn/
Datasets and task. We use the publicly available Taobao (Zhu al., 2018 2019;
splits shown in Appendix B
Baselines. We compare against variety of recommendation models, including ETA (Chen al., 2021]. SDIM (Cao et al. 2022). DIN (Zhou et al. 2018). TWIN (Chang et al., (023) and its variants, as well as V2 (Si al. 2024). As discussed See Sec the target-aware representation by crossing e baselines for fairness. TWIN-4E refers to the model introduced in Sec.3.4| while TWIN (w/ proj.) refers to the model described in Sec. 2.3 TWIN (hard) represents a variant using "hard search" in the search stage, meaning it only retrieves behaviors from the same category the target. TWIN (w/u TR) refers to the original TWIN model without target-aware representation, i.e., representing user history h Σ, wy e; instead of h Σ, wi(e; O v.).
Repr O ve
Repr significantly improves representation discriminability, so we include it in our

---

Published as a conference paper at ICLR 2025
Table 1: Overall comparison reported by the means and standard deviations of AUC. The best
results are highlighted in bold, while the previous best model is underlined. Our model outperforms all existing methods with obvious advantages, especially with small embedding dimensions.
Sctup
Embedding Dim. 16
Embedding Dim. 64
Embedding Dim. 128
Dataset
Taobao
Tmall
Taobao
Tmall
Taobao
Tmall
ETA (2021
0.91326
0,95744
0.92300
0.96658
0,92480
0,96956
(0. (0.00538)
(0.00108)
(0.00079)
(0.00012)
(0.00032)
(0.00039)
SDIM (2022)
0,90430
0,93516
0.90854
0.94110
0,91108
0,94298
(0. (0.00103)
(0. (0.00069)
(0. 00085]
(0. (0.00093)
(0.00119)
(0. (0.00081)
DIN (2018)
0.90442
0.95894
0,90912
0.96194
0.91078
0.96428
(0.00060)
(0.00037)
(0. (0.00092)
(0.00033)
(0.00054)
(0.00013)
TWIN (2023)
0.91688
0.95812
0.92636
D. 1996684
0.93116
0.97060
(0 00.00211)
(0.00073)
00052)
00039)
(0.00056)
00.00005)
TWIN (hard)
0.91002
0.96026
0.9 0.91984
0.96448
0.91446
0.96712
(0.00053)
(0.00024)
(0.00048)
00042)
(0.00055)
(0.00019)
TWIN (w/ proj.)
0.89642
0.96152
0.87176
0.95570
0.87990
0.95721
(0.00351)
(0.00058)
(0.00437)
00403)
(0.02022)
(0.00194)
TWIN (w/o TR)
0.90732
0.961 70
0.91590
0.96320
0.92060
0.96366
(0. (0)00063)
(0.00057)
(0. (0.00083)
(0. (0.00032)
(0.00084)
(0.00103)
TWIN-V2 2024
0. 0.89334
0.94714
0.901
0.95378
0.90586
0.95732
(0.00077)
(0.00110)
(0. (0.00063)
(0 (0.00)37)
(0.00059)
(0.00045)
TWIN-4E
0.90414
0. 0.96124
0. 0.90356
0.96372
0.90946
0,96016
(0.01329)
(0.00028)
(0.01505)
(0.0004)
(0.01508)
(0.01048)
DARE (Ours)
0.92568
0,96800
0,92992
0.97074
0,93242
0.97254
(0.00025)
(0.00024)
(0.00046)
(0.00012)
(0.00045)
(0.00016)
4.2 OVERALL PERFORMANCE
In recommendation systems, is well-recognized that even increasing AUC by 1% to 2% is more than enough to bring online profit. As shown in Tab. [ our model achieves AUC improvements of 1% and 9%c compared to current state-of-the-an methods across all settings with various embedding sizes. In particular, significant AUC lifts of 9%, and 6%c are witnessed with an embedding dimension of 16 on Taobao and Tmall datasets, respectively.
There are also some notable findings. TWIN outperforms TWIN (w/o TR) in most cases, proving that target-aware representation e, is shown in Sec. 4.4). Our DARE model has an obvious advantage over TWIN-4E, confirming that the prior knowledge discussed in Sec. 3.4 is well-suited for the recommendation system. ETA and SDIM, which are based on TWIN and focus on accelerating the search stage at the expense of performance, understandably show lower AUC scores. TWIN-V2, a domain-specifie method optimized for video recommendations, is less effective in our settings.
Repe u.eep do help enhance discriminability (further evidence
4.3 ATTENTION ACCURACY
Mutual information, which captures the shared information between two variables, is a powerful tool for understanding relationships in data. We calculate the mutual information between behaviors and the target as the ground truth correlation, following (Zhou al. 2024). The learned attention score reflects the model's measurement of the importance of each behavior. Therefore, we compare the attention distribution with mutual information in Fig. g
In particular, Fig. 8a presents the mutual information between a target category and behaviors with t0p-10 categories and their target-relative positions (i.e., how close to the target is the behavior across time). We observe strong semantic-tente temporal correlation: behaviors from the same category as the target (5th row) are generally more correlated, with a noticeable temporal decay pattern. Fig. 8b presents TWIN's learned attention scores, which show a decent temporal decay pattern but over- estimate the semantic correlation of behaviors across different categories, making it too sensitive to recent behaviors, even those from unrelated categories. In contrast, our proposed DARE can effectively capture both the temporal decaying and semantic patterns.
The retrieval in the search stage relies entirely on attention scores. Thus, we further investigate the retrieval on the test dataset, which provides a more intuitive reflection of attention quality. Behaviors with top-k mutual information are considered the optimal retrieval, and we evaluate model perfor- mance using normalized discounted cumulative gain (NDCG) (Järvelin & Kekälänen (002). The results, along with case studies, are presented in Fig. (more examples in Appendix E.4) We find:
7

---

Published as a conference paper at ICLR 2025
Category-wwise wise Target-aware Correiation -0000
Attention Target Calegory 15
Targel Calegory ID 15
2.4 can c2 5 oas 01 0:22 cot 0:
J4.i5 0:0 c oar cot 8 oos o oir
Top-1 Appppred Categorte
Bed 0J 0:30:10 81 0/22 855 0J3 E.0 9.1
2.00 0.9 843 0.2 000 8.2 001 0.1 o0t 8:1
2/12
Top- Agpeared Cale
/1.82 cuT 8:1
+0.30 0.24
0:0
0.0 001
0.1
Top-10 Agrpeated
18 *0
0.72 23382300.7700
0.00 0.44
5.35 0.22 8:55 006 0.05 8:65
D
aM
128 0:1
OUT 0.5 0.22 DUE 8.01
0.36 0:24 10 15 2:3 sin
9.8 0:1
nelllvve Postion
Targel-raiatiie Position
Targal-relative Position
(a) GT mutual information
(b) TWIN learned correlation
(c) DARE learned correlation
Figure 8: The ground truth (GT) and learned correlation between history behaviors of top-10 10 fre- quent categories (y-axis) at various positions (x-axis), with category 15 as the target. Our correlation scores are noticeably closer to the ground truth.
Retrieval Performance on Tsdbso
Correlations with Category B7
Correlations Category 19
0.8
Ground Trch
0.05
Ground Truth -5.02
(242 0.31
0.24 0.23 0:27
001
DARE 0.28 0.13 oms
021
0:12
5:70
DAVE -:4
0.28 0.00 0.00
0.01
0.8
NDOG
4WN
0.22
TWN-4E-
2.44 c.t1 0.15
0.24 0,02 0.11 0.2 0.u5
0.3
THIN--
THenl
TWIN
0:55 0:49 04 0.27 0.5 0.12
0.00
0.22
Think an (P1g
TWIN Pry.-
TVAN ue Pinj 0
0
0.00
DNN 0.11 0.c2 0.7 0.00
0.14
0.04
02 0.04
Behaviors Retured by Search
10
20
121 151
e7
11
28
57
310 Behavior Sequence
Behavior Sequenze
(a) Retrieval on Taobao
(b) Case study 1
(c) Case study
Figure 9: Retrieval in the search stage. (a) Our model can retrieve more correlated behaviors. (b-c)
Two showcases where the x-axis is the categories of the recent ten behaviors.
the NDCG of our model is substantially higher than all baselines, with a 46.5% increase (0.8124 vs. 0.5545) compared to TWIN and a 27.3% increase (0.8124 vs. 0.6382) compared to DIN.
DARE achieves significantly better retrieval. As shown in Fig. 9al
TWIN is overly sensitive to temporal information. As discussed, TWIN tends to select re- cent behaviors regardless of their categories, against the ground truth, due to overestimated correlations between different categories, as shown in Fig. ab
Other methods perform unstably. For the other methods, they filter out some important behaviors and retrieve unrelated ones in many cases, which explains their bad performance.
Result I. DARE succeeds in capturing the semantic-temporal correlation between behaviors and the target, retaining more correlated behaviors during the search stage.
4.4 REPRESENTATION DISCRIMINABILITY
We then analyze the discriminability of learned representation. On test datasets, we take the com- pressed representation of user history / = Σ w; (e; C vz), which forms a vector for each test sample. Using K-means, we quantize these vectors, mapping each h to cluster Qh). The mutual information (MI) between the discrete variable Q(h) and label Y (whether the target was clicked) can then reflect the representation's is discriminability: Discriminability(h. Y) = MI))(h) Y).
As shown in Fig. 10a across various numbers of clusters, our DARE model outperforms the state-of- the-art TWIN model, demonstrating that decoupling improves representation discriminability. There are also other notable findings. Although DIN achieves more accurate retrieval in the search stage (as evidenced by a higher NDCG in Fig. (a). its representation discriminability is obviously lower than TWIN, especially on Taobao dataset, which explains its lower overall performance. TWIN-4E shows comparable discriminability to our DARE model, further confirming that its poorer performance is prior knowledge.
due to inaccurate attention caused by the lack of recommendatton-speein
To fully demonstrate the effectiveness of e; O Vú we compare it with the classical concatenation [Ere;, vt). As shown in Fig. 10c huge gap (in orange) is caused by the target-aware representation,
8

---

Published as a conference paper at ICLR 2025
Representation's Discriminability
Representation's Discriminablity
Representation's Discriminabiilt
Mutuel Informatio
Mutual Information
0.09
Mittall Information
0.04
0.05
025
0.03
prg)
0.04
(wipr))
020
Impoxemeet
0.03
0.15
0.02
0.02
0.10
0.01
0.01
0.05
Iyproseet di
0.00
2'
0.00
0.00
Number Clusters
Number Clusters
(a) Discriminability Taobao
(b) Discriminability Tmall
(c) Discriminability of e, O
Figure 10: Representation discriminability of different models, measured by the mutual information between the quantized representations and labels.
Training Performance Taobao
Training Performence on Trail
Dficiency and AUC Taobao
Effickenyy and AUC on Tmail
0.88
0.82
0.87
0.930
0.90
Volleation
( 0.925
0.88
0.95
0.83
TAIN DARE
0.85
TWN DAME
0.900
7W0H DARE
DARE
TATN
0.82
500 1200 1500
2000
0.04
500
100
1500
0.25
0.50
0.75
1,50
0.5
0.50
0.75
Training llrntion
Traiting llenatinn
Normalized Complexity
Normalized Complexity
(a) Training on Taobao
(b) Training on Tmall
(c) Efficiency on Taobao
(d) Efficiency on Tmall
Figure 11: Efficiency during training and inference. (a-b) Our model performs obviously better with fewer training data. (c-d) Reducing the search embedding dimension, a key factor of online inference speed, has little influence our model, while TWIN suffers obvious performance loss.
while smaller gaps (in blue and green) result from decoupling. Notably, our DARE model also outperforms TWIN even when using concatenation.
Result 2. In the DARE model, the form of target-aware representation and embedding decou- pling both improve the discriminability of representation significuntly.
4.5 CONVERGENCE AND EFFICIENCY
Faster convergence during training. In recommendation systems, faster learning speed means the model can achieve strong performance with less training data, which is especially crucial for online services. We track accuracy on the validation dataset during training, shown in Fig. Ia Our DARE model converges significantly faster. For example, on the Tmall dataset, TWIN reaches 90% accuracy after more than 1300 iterations. contrast, our DARE model achieves comparable performance in only about 450 iterations -he of the time required by TWIN.
Efficient search during inference. By decoupling the attention embedding space e,,, U2 E R. and representation embedding space e;, V2 E RRx, we can assign different dimensions for these two spaces. Empirically, we find that the attention module performs comparably well with smaller embedding dimensions, allowing us to reduce the size of the attention space (K,, significantly accelerate the search stage, as its complexity is O(KAN) where N is the length of the user history. Using KA 128 as baseline ("1"", normalize the complexity of smaller embedding dimensions. Fig. IIc shows that our model can accelerate the searching speed by 50% with quite little influence on performance and even by 75% with an acceptable performance loss, offering more flexible options for practical use. In contrast, TWIN experiences a significant AUC drop when reducing the embedding dimension.
Kn) and
Result 3. Embedding decoupling leads to faster model training convergence and at least 50% inference acceleration without significuntly influencing the AUC by reducing the dimension of altention embeddings.

---

Published as a conference paper at ICLR 2025
4.6 ONLINE A/B TESTING AND DEPLOYMENTS
We apply our methods to Tencent's advertising platform. Since users' behaviors on ads are sparse, which makes the sequence length relatively shorter than the content recommendation scenario, we involve the user's behavior sequence from our article and the micro-video recommendation scenario. Specifically, the user's ad and content behaviors in the last two years are introduced. Before the search, the maximal length of the ads and content sequence is 4000 and 6000, respectively, with 170 and 1500 on average. After searching with DARE, the sequence length reduced to less than 500. Regarding sequence features (side info), we choose the category ID, behavior type ID, scenario ID, and two Larget-aware temporal encodings, i.e., position relative to the target, and time interval relative to the target (with discretization). There are about 1.0 billion training samples per day. During the 5-day online A/B test September 2024, the proposed DARE method achieves 0.57% cost, and 1.47% GMV (Gross Merchandize Value) lift over the production baseline of TWIN. This would lead to hundreds of millions of dollars in revenue lift per year.
4.7 SUPPLEMENTARY EXPERIMENT RESULTS IN APPENDIX
Retrieval number in the search stage. DARE's advantage is more obvious with less retrieval num- ber, proving once again that DARE selects important behaviors more accurately (Appendix D.I.
Sequence length and short-sequence modeling. DARE can consistently benefit from longer se- quences, while it delivers marginal advantages in short-sequence modeling (Appendix D.2)
GAUC and Logloss Besides AUC, we also evaluate DARE and all the baselines under GAUC and Logloss. DARE shows consistent superiority, proving the solidity of our results (Appendix E.1
5 RELATED WORK
Click-through rate prediction and long-sequence modeling, CTR prediction is fundamental in recommendation systems, as user interest often reflected in their clicking behaviors. Deep Interest Network (DIN) (Zhou et al. 2018) introduces target-aware attention, using an MLP to learn attentive weights of each history behavior regarding specific target. This framework has been extended by models like DIEN (Zhou et al. 2019), DSIN (Feng et al. 2019). and BST (Chen et al.) 2019) to capture user interests better. Research has proved that longer user histories lead more accurate predictions, bringing long-sequence modeling under the spotlight. SIM (Pi et al.) 2020) introduces a search stage (GSU), greatly accelerating the sequence modeling stage (ESU). Models like ETA (Chen et al.
2021 and SDIM (Cao et al. 2022) further improve this framework. Notably,
TWIN (Chang et al.] 2023) and TWIN-V2 V2 (Si et al. (002)) unify the target-aware attention metrics used in both stages, significantly improving search quality. However, as pointed out in Sec. 2.2 in all these methods, attention learning is often dominated by representation learning, creating a significant gap between the learned and actual behavior correlations.
The attention mechanism, most well-known in Transformers (Vaswani et al. 2017), has proven highly effective and is widely used for correlation measurement. Transformers employ Q. K (attention projection). and V (representation projection) matrices to generate queries, keys, and values for each item. The scaled dot product of query and key serves as the correlation score, while the value serves as the representation. This structure is widely used in many domains, in- cluding natural language processing (Brown et al., 2020) and computer vision (Dosovitskiy et al.) 202T). However, in recommendation systems, due the interaction-collapse theory pointed out by Guo et al. (2024), the small embedding dimension would make linear projections completely lose effectiveness, as discussed Sec. 2.3 Thus, proper adjustment is needed in this specific domain.
Attention.
CONCLUSION
This paper focuses on long- long-sequence recommendation, starting with an analysis of gradient domina- tion and conflict on the embeddings. We then propose a novel Decoupled Attention and Representa- tion Embeddings (DARE) model, which fully decouples attention and representation using separate embedding tables. Both offline and online experiments demonstrate DARE's potential, with com- prehensive analysis highlighting its advantages in attention accuracy, representation discriminability, and faster inference speed.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To ensure reprodueibility, we provide the hyperparameters and baseline implementation details in Appendix A along with dataset details in Appendix We have released the full code, includ- ing dataset processing, model training, and analysis experiments, at Ehuml /DARE
com/
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (62021002), the BNRist Project, the Tencent Innovation Fund, and the National Engineering Research Center for Big Data Software.
REFERENCES
Andrej. karpathy/nanoGGP. URI. httppss//githhb..cm//aaatttttt/t date: 2022-12-28T00:5111222
original-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yue Cao, Xiaojiang Zhou, Jiaqi Feng, Peihao Huang, Yao Xiao, Dayao Chen, and Sheng Chen. Sampling is all you need on modeling long-term user behaviors for ctr prediction. In ACM Inter- national Conference on Information and Knowledge Management (CIKM), 2022.
Rich Caruana. Multitask learning. Machine learning, 1997.
Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, al. Twin: Two-stage interest network for lifelong behavior modeling in etr prediction at kuaishou. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2023.
Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. Behavior sequence transformer for e-commerce recommendation in alibaba. In International Workshop on Deep Learning Practice for High-Dimensional Sparse Data (DLP-KDD), 2019.
Qiwei Chen, Changhua Pei, Shanshan Lv, Chao Li, Junfeng Ge, and Wenwu Ou. End-to-end user behavior retrieval in click-through rateprediction model. In ACM SIGIR Conference Research and Development in Information Retrieval (SIGIR), 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
Yufei Feng, Fuyu Lv, Weichen Shen, Menghan Wang, Fei Sun, Yu Zhu, and Keping Yang. Deep session interest network for tlick-through rate prediction. In International Joint Conference Artificial Intelligence (IJCAI), 2019.
Xingzhuo Guo, Junwei Pan, Ximei Wang. Baixu Chen, Jie Jiang, and Mingsheng Long. On the embedding collapse when scaling up recommendation models. In International Conference on Machine Learning (ICMI), 2024.
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In International Conference on World Wide Web (WWW), pp. 507-517, 2016.
11

---

Published as a conference paper at ICLR 2025
Kalervo Järvelin and Jaana Kekäläänen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 2002.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu, Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recom- mendations on styles and substitutes. In ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2015.
Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang. Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. Search-basee based user interest modeling with lifelong sequential behavior data for flick-through rate prediction. In ACM International Conference Information and Knowledge Management (CIKM), 2020.
Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. Neural collaborative filtering vs. matrix factorization revisited. In ACM Conference on Recommender Systems (RecSys), 2020.
Zihua Si, Lin Guan, ZhongXiang Sun, Xiaoxue Zang, Jing Lu, Yiqun Hui, Xingchao Cao, Zeyu Yang, Yichen Zheng, Dewei Leng, et al. Twin v2: Scaling ultra-long user behavior sequence modeling for enhanced ctr prediction at kuaishou. In ACM International Conference Informa- tion and Knowledge Management (CIKM), 2024.
Tianchi. 1jcai-15 repeat buyers prediction dataset, 2018, URL http:/daaattdddaaacc aadtaaaaaaeeaaaaaaaaaaaaaaaaaa ?dataId=
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurlPS), 2017.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Wenhui Yu, Chao Feng, Yanze Zhang, Lantao Hu, Peng Jiang, and Han Li. Ifa: Interaction fidelity attention for entire lifelong behaviour sequence modeling, 2024. URL htttts:://axxi......t
/o0/2240..om 09742
Jiaqi Zhai, Zhaojie Gong, Yueming Wang, Xiao Sun, Zheng Yan, Fu Li, and Xing Liu. Revisiting neural retrieval on accelerators. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2023.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In ACM SIGIR Conference on Research and Development Information Retrieval (SIGIR), 2018
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click-through rate prediction. In Association for the Advance- ment of Artificial Intelligence (AAAI), 2019.
Haolin Zhou, Junwei Pan, Xinyi Zhou, Xihua Chen, Jie Jiang, Xiaofeng Gao, and Guihai Chen. Temporal interest network for user response prediction. In Companion Proceedings of the ACM on Web Conference (WWW Companion). 2024.
Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai. Learning tree-based deep model for recommender systems. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2018
Han Zhu, Daqing Chang, Ziru Xu, Pengye Zhang, Xiang Li, Jie He, Han Li, Jian Xu, and Kun Gai. Joint optimization of tree-based index and deep model for recommender systems. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Jingwei Zhuo, Ziru Xu. Wei Dai, Han Zhu, Han Li, Jian Xu, and Kun Gai. Learning optimal tree models under beam search. In International Conference on Machine Learning (ICML), 2020.
12

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 PYPER-PARAMETTRS AND MODEL DETAILS
The hyper-parameters we use are listed as follows:
Parameter
Value
Retrieve number
20
Epoch
2
Batch size
2048
Learning rate
0.01
Weight decay
le-6
Besides, we use the Adam optimizer. Layers of the Multi-layer Perceptron (MLP) are set as 200 x 80 2, which is the same as Zhou et al. (2024).
These settings remain the same in all our experiments.
A.2 BASELINE IMPLEMENTATION
Many current methods not open-source and may focus on certain domain. Thus, followed their idea and implemented their method according to our task setting, Some notable details are shown as follows:
DIN is primarily designed for short-sequence modeling, and aligned it with long-sequence models. Specifically, while the original DIN aggregates all historical behaviors using learned weight, our approach enables DIN select the top- K most significant behaviors based on these weights, the same as other long-sequence mod- eling techniques. Note that the original DIN is impractical for long-sequence modeling, as aggregating such extensive history would result in prohibitively high time complexity.
introduced the search stage
TWIN-V2 is specifically designed for Kuaishou, a short video-sharing app. leveraging video-specific features to optimize performance in video recommendations. However, our experiments focus on more general scenario where only item IDs and category IDs are available, Thus, we made some necessary adjustments while retaining the core ideas of TWIN-V2. e.g., TWIN-V2 would first group the videos based on the proportion video is played, which does not have corresponding feature in our datasets. Consequently. we grouped user history using temporal information instead. It's understandable that outside its specific domain, TWIN-V2 cannot fully realize its potential.
B DATA PROCESSING
Dataset information. Some detailed information is shown in Table 2 We use Taobao (Zhu et al., 2018 2019 Zhuo al. 2020) and Tmall (Tianchi 2018) datasets in our experiments. The propor- tion of active users (Users with more than 50 behaviors) in these two datasets is more than 60%, which is relatively satisfying. Note that the Taobao dataset is more complex, with more categories and more items, which is a higher challenge for model capacity.
Table 2: Some basic information of public datasets (active user: User with more than 50 behaviors).
Dataset #Category
#Item
#User
# Active User
Taobao
9407
4,068,790 1,080,666
984, 14
603,176 246,477
Tmall
1492
423,862
split. We sequentially number history behaviors from one (the most re- cent behavior) to T (the most ancient behavior) according the time step. The test dataset contains
13

---

Published as a conference paper at ICLR 2025
predictions of the first behaviors, while the second behaviors used the validation dataset. For the training dataset, we use the (3 + 5i, 0 & & 18)th behavior. Models would finish predicting the jth behavior based on j 200 to j behaviors (padding if history length is not long enough). Only users with behavior sequences longer than 210 will be reserved.
We make such settings balance the amount and quality of training data. In setting, each selected user would contribute 20 pieces of data visible to our model in the training process. Besides, can guarantee that each piece of test data would contain no less than 200 behaviors, making our results more reliable. To some degree, we break the "independent identical distribution" principle because we sample more than one piece of data from one user. However, it's unavoidable since the dataset is not large enough due to the feature of the recommendation system (item number is usually several times bigger than user number), so we finally sample with interval 5, using the ((3 5i)th, O & i & 18) behaviors as the training dataset.
C THE RESEARCH PROCESS LEADING TO DARE
C.I OTHER DECOUPLING METHONS
Besides linear projection, we have tried many other decoupling methods before we came up with the final DARE model. Their structures are illustrated in Figure 12 Specifically:
Linear projection. This is the structure referred as TWIN (w/ proj.) in this paragraph, applying linear projection to address the conflict.
Item/CateorryTTime linear projection. Item, category, and time features exhibit signifi- cant differences (e.g. item number is about 1,000 times larger than category number). So we tested the effectiveness of linear projections when applied to each feature individually.
Cate. and time linear projection. The number of items is too large, making it too chal- lenging for a simple linear projection to project millions item embeddings into another space. So we designed this model and only linear projection on category and time.
Larger embedding. To enhance the capacity of linear projection while maintaining the feature dimension, we used a larger embedding dimension while keeping the output di- mension of linear projection the same other models.
MLP projection. We replace the linear projection with Multilayer Perceptron (MLP), which has much stronger capacity. This experiment aims to figure out the impact of pro- jection capacity on model performance.
Avoid domination. Basing on the original TWIN model, whenever the gradient is back propagated to the embedding (we have demonstrated in Section [.22 that gradient from representation about five times larger than that from attention), we manually scale the gradient from attention to make its 2-norm the same as representation, which can solve the problem of domination.
C.2 AUC RESULT
We evaluated the models on the Taobao and Tmall datasets, with the results presented in Table 3 Among the other eight models except DARE, none of them achieved consistent and significant im- provements across both datasets. The Taobao dataset is notably more complex, containing nearly nine times the number of categories as Tmall. Thus, some decoupling methods showed improve- ments on the simpler Tmall dataset but lost effectiveness on the more complex Taobao dataset. Interestingly, while the "MLP projection" model theoretically offers greater capacity, failed to outperform the simpler linear projection, which captured our attention. To investigate further, examined the gradient behavior of these models.
C.3 GRADIENT CONFLICT IN THESE MODELS
We then observed whether these models have the potential to resolve gradient conflict. For each category, we observed the gradients from attention and representation at every iteration and cal-
14

---

Published as a conference paper at ICLR 2025
Repr
Attin Repr
Repe Behavior Festure
Behavior Feature
Concat/Ade
Add
Concat/Acd
Attin Repr
Attin Repr Linear
Altin/ Repr Linear
Attn/ Linear
Repr Linear
Linear
Item Embeddiig
Category Embedding
Time Embedding
Item E mbbdiing
Gategory Emboading
Time Embedding
Item Embedeing
Category E Embeddding
Time E Embedding
Item
Calegory ID
Time
Item ID
Category ID
Time
Item
Calegory ID
Time
One Behavior
One Behavior
Linear projection
(b) Item linear projection
(c) Category linear projection
Repr Behavior Festure
Attn/ Repr Behavior
Repr
Behavlor Festure
Concat/Ade
Concat/AAl Add
Concat/Acd
Altn/ Repr Linear
Attn Repr Linear
Adtnn Repr Linear
Attn/Rapr
AttryRepe Lingar
AttryRepe
Item Embedding
Category Embedding
Time Embedding
Item E Embedding
Category Embedding
Time Emoedding
Item Embedoing
Category Embedding
Time Embedding 1 Time
f Category ID
Item
Category ID
Time ID
Item ID
Category ID
Time ID
Item ID
One Behavior
Behavior
One Behavior
(d) Time linear projection
(e) Cate. and time linear projection
Larger embedding
Repr Feature
Do Maloo
Attention radiont
Representation Gradient
Concat /Add
Attn/ / Repr
Alth Repr NLL
Attin Repr MLP
MLP
scale
scale
scale
|| mbedding Item ID
Category Embedding
Time Embedding
Embedicng
Category Embedding
1 Time
Embodding
Item ID
Category ID
Category
Time ID
Behavior
(g) MLP projection
Avoid domination
Figure 12: Eight other methods tried before came up with DARE.
Table 3: The performance of other models tried reported by the means and standard deviations of AUC. Only DARE achieved a satisfying result. Each model's comparison with the original TWIN highlighted: improvements are marked in green, while deteriorations are marked in red.
Setup
Taobao
Tmall
TWIN 2021
0.91688 t 0.00211
(baseline)
0.95812 + 0.00073
(baseline)
Linear projection
0.89641 + 0.00351
(-0.02046)
0.96152 + 0.00088
(+00.0988
Item linear projection
0.90886 + 0.00218
(-0.00802)
0.96032 + 0.00119
(+0.00220)
Category linear projection
0.91738 0.00099
(+0.00050)
0.96658
0.00068
(+0.00846)
Time linear projection
0.91354 0.00046
(-0.00334)
0.95758
0.00116
(-0000054)
Cate. and time linear projection
0.91202 + 0.00340
-0.00486)
0.96604
0.00038
(+0.00792)
Larger embedding
0.91348 1 0.00247
0.00040)
0.96348 + 0.00057
(+0000536)
MLP projection
0.86960 + 0.04013
(-004728
0.95678 + 0.00066
-0.00134)
Avoid domination
0.91976 0.00052
(+0.00288)
0.93887 = 0.00127
0..01925
DARE (Ours)
0.92568 + 0.00025 (++000888))
0.96800 + 0.00024 (+0.00998))
culated the percentage iterations in which the gradient for that category exhibited conflict. Results shown in Figure 14] As demonstrated in this figure, find that:
15

---

Published as a conference paper at ICLR 2025
0.5
Category Ratio
Caregory Ratio
0.23
Calagory
0.22
0.22
0.2
0.33
0.65
0.83
100
(atio(%)
Conflict Ratio[%)
Conflict Ratio(%)
(c)
Category
linear
projection
(a) Linear projection (62.34%)
(b) Item linear projection (72.13%)
(78.47%)
0.30
Category Ratio
0.5
Category
0.25
0.20
0.22
0.15
6.35
0.10
633
0.3
Conflict Ratio(%)
0.03
Conflict Ratlo(%)
Conflict Ratio(%)
(e) Cate. and time lincar projection (84.8%)
(d) Time lincar projection (69.26%)
(f) Larger embedding (88.29%)
0.23
0.
0.
Category Raclo
0.33
0.35
0.22
02s
0.23
Category
0.35
Gatagory
0.00
0.0
0.00
0.3
Ratio(%)
Ratio(%)
Conflict Ratio(%)
(g) projection (30.96%)
(h) Avoid domination (71.51%)
TWIN (80.91%)
Analysis of gradient conflict on the original TWIN and eight other models we tried. The number after model name means the ratio of categories falling the right side of the red line (meaning that the category reported gradient conflict in more than half iterations). Most models fail resolve gradient conflict well.
Figure 14:
The conflict of TWIN. the original TWIN, most categories (80.91%) experienced gradient conflict in more than half of the iterations.
The failure of these models. Some methods (like Item linear projection) to some degree, solve the conflict, but that's far from enough. Some methods even worsen the conflict (like Larger embedding).
The challenge of MLP projection. MLP projection
solves the conflict best, although still 30% categories re-
Training Performance
porting conflict in than half of iterations, this model outperforms other projection-based decoupling methods. However, MLP projection performs poorly. To under- stand this discrepancy, we further analyzed its perfor- mance during training, and the results are shown Fig- ure 13 Though resolving conflict better than some other models, MLP projection struggles to optimize the train- ing process due to more parameters and higher complex- ity. For example, after 100 iterations, the accuracy of DARE is 82.44%, while that of MLP projection is only 74.87% (Note that even continually outputting "No" can achieve 66.77 66.7% accuracy).
Validation Accuracy
0.90
0.85
0.80
MLP Projection
0.75
500
1000 1500 2000 Training Iteration
Figure 13: Comparison of MLP projec- tion and DARE models during training.
16

---

Published as a conference paper at ICLR 2025
Retrieval Analysis on Taobao
Retrieval Analysis on Tmall
0.82
0.985
0.980
0.90
AUC
0.955
0.88
0.990
0.945
0.86
TWIN DARE
TWIN
0.940
15
20
Retrieved Number
Retrieved Number
(a) Influence retricval number on Taobao
(b) Influence of retrieval number on Tmall
Figure 15: On both datasets, when the number of retrieved behaviors increases from to 25, models first perform better, then keep the same performance. DARE outperforms TWIN at any settings, achieving even 5.7% higher AUC on Taobao when retrieving behavior.
C.4 CONCLUSION
We explored various decoupling methods, but none could fully resolve gradient conflicts, may introduce optimization issues. All the results call for more effective decoupling method, that is, back-propagating the gradient to different embedding tables, which can completely solve whatever problems like domination and conflict, since attention and representation will each have an exclusive embedding table now. This insight led to the development of DARE.
D INFLUENCE OF PAPPER-ARAMETERS
D.1 EFFECTS OF RETRIEVAL NUMBER IN THE SEARCH STAGE
The number of retrieved behaviors, K in this paper, is crucial hyper-parameter the two-stage method. We modified this parameter, and the results are presented in Figure [5] Key findings include:
On Taobao dataset, TWIN must retrieve more than 15 behaviors to fulfill its potential, while DARE can achieve best performance when retrieving more than 10 behaviors. This indicates that DARE can retrieve those important behaviors more accurately, while TWIN must retrieve more behaviors avoid missing important ones.
DARE consistently outperforms TWIN across all settings, especially with fewer retrieval numbers. On Taobao dataset, when retrieving only behavior, DARE outperform TWIN with an AUC increase of 5.7% (even a 0.1 1% AUC increase is considered significant).
In all our other experiments, the retrieve number is set to 20 to ensure all models perform at their best. Our advantage over TWIN would only be more obvious in some other settings.
D.2 EFFECTS OF SEQUENCE LENGTH
We analyzed the impact of sequence length and identified scenarios where the DARE model exhibits a more significant advantage. Results are shown in Figure [6] Some notable findings are:
Reduced advantage with shorter sequences: DARE's advantage over TWIN diminishes as the sequence length decreases. Shorter sequences make it easier to model user history, reducing the impact of inaccuracies in measuring behavior importance. Under these conditions, TWIN achieves performance comparable to DARE.
17

---

Published as a conference paper at ICLR 2025
Analysis of Sequence Length
Analysis of Sequence Length
0.925
0.9675
DARE
0.920
0.9650
Test AUS
Test Test AUC
0.915
0.9825
0.910
0.9900
0.906
0.9575
TWIN
0.9025
160
150
Sequence Length
Sequence Length
(a) Influence of sequence length on Taohao
(h) Influence of sequence length on Tmall
Figure 16: With shorter sequence length, the advantage of our DARE model over TWIN becomes smaller. DARE can perform better with longer sequence length, indicating its potential select important behaviors in the long user history. However, on Tmall dataset, TWIN works better with sequence length 120 than 160 200, indicating that TWIN relies on larger embedding dimension to become effective.
Superior performance with longer sequences: DARE excels with longer sequences. On the Tmall dataset with embedding dimension=16, however, TWIN performs worse with a sequence length of 200 compared to 120. This suggests that DARE effectively captures the importance of each behavior and leverages long user histories at any setting, while TWIN relies heavily embedding dimension and would struggle with an abundance of historical behaviors when embedding dimension is small.
We also tried method in the short-sequence modeling (removing the search stage and modeling the whole sequence). We use the Amazon dataset (He & McAuley 2016; McAuley et al. 2015)
with the same setup as the state-of-the-art art TIN model (Zhou al. 2024). However, the performance improvement is marginal (TIN: 0.86291 0.0001 AUC vs. DARE: 0.86309 AUC). For the Amazon dataset, the average user history length is no longer than ten. Shorter sequence means fewer candidate behaviors, so it becomes easier to model behavior importance. Removing the search stage means the important behaviors will never be discarded by mistake as long-sequence modeling, so the attention module will not cause too severe result even if it is not capable enough. As shown by TIN Zhou et al. (2024), representation is more critical than attention short-sequence settings, so the dominance of representation doesn't significantly impact performance when the attention task is relatively easier.
Relevance to modern recommendation systems: It is worth noting that modeling longer user his- tories is a growing trend in recommendation systems (Pi et al.) 2020). Contemporary online systems increasingly incorporate extended user histories, making short sequence modeling less important. As this trend continues, the advantages of the DARE model will become more pronounced in today and future online systems.
D.3 EFFECTS OF ATTENTION AND REPRESENTATION EMBEDDING DIMENSION
In general, increasing the embedding dimension improves model performance. However, in practice, limitations such as the interaction collapse theory (Guo et al. 2024) or strict time constraints make it impractical to use arbitrarily large embeddings. To address this, we analyzed various attention- representation dimension combinations, offering insights that could guide future implementations. The results are presented in F7 A key observation is that the representation embedding dimension has a stronger impact on model performance compared to the attention embedding di- mension. This suggests that balanced approach-use uuing smaller attention embedding for faster online processing and a larger representation embedding for enhanced performance-conn could be an optimal strategy.
18

---

Published as a conference paper at ICLR 2025
0.973
0.932
0.972
0 0.971
0.030
0.99 .70
60888
0.927
0.968
0.25
0.967
AltenBon Dirretsion
128
96
loppheentation Dimensiin
Attention Dimension
22 96
Representation Dimensson
64
52
32
32
32
(a) AUC and embedding dim on Taobao
(b) AUC and embedding dim Tmall Figure 17: The influence of attention and representation embeddings on AUC.
Table 4: Overall comparison reported by the means and standard deviations of GAUC (grouped by category). The best results are highlighted in bold, while the previous best model is underlined.
Sctup
Embedding Dim. = 16
Embedding Dim. = 64
Embedding Dim. 128
Dataset
Taobao
Tmall
Taobao
Tmall
Taobao
Tmall
ETA 12021)
0.89900
0.94908
0.90980
0.95918
0.91 144
0.96230
(0.00372)
(0.00121)
(0.00084) 4)
(0.00046)
(0.90050)
(0.00054)
SDIM 2222
0.89128
0.92408
0.89496
0.93320
0.89780
0.93062
(0.00097)
(0.00084)
(0.00088)
(0.00067)
(0.00098)
(0.001 12)
DIN (0188)
0.88748
0.94998
0.89300
0.95350
0.89566
0.95628
(0.00054)
(0.00039)
00000))))
(0.00039)
(0.00049)
(0.00021)
TWIN (2233
0.90364
0.94998
0.91374
0.95952
0.91880
0.96370
(0. (0209)
(0.00088)
C006500)))
(O. 00048)
(0.00068)
(0. 0000))
TWIN (hard)
0. 89386
0. 0.95216
0. 0.90588
0.95 0.95698
0,90008
0, 0,95908
(0. (..0033)
(0.00037)
00040)
00.00063)
[0.00066]
(0. (0.00021)
TWIN (w/ proj.)
0.88038
0.95012
0.85372
0.946188
0.84696
0.94791
(0.00403)
(0.00799)
(0.00427)
(0.00497)
(0.01012)
(0.00235)
TWIN (w/o TR)
0.89190
0.953-10
0.90098
0.95521
0.90628
0.95570
(0. (0.00081)
(0.00053)
(0.00081)
000360
(0.00085)
(0.00115)
TWIN-V2 2024
0.87954
0. 0.93772
0, 888888
0. 0.94510
0.89164
0.94894
(0.00067)
(0.00118)
(0.00050)
(0. (0.00038)
(0.00063)
(0.00056)
TWIN-4E
0.88864
0.95278
O, 0,88810
0.95570
0.89448
0.95148
(0.01565)
(0.00051)
(0.01560)
(0.00051)
(0.01595)
(0.01229)
DARE (Ours)
0.91240
0.96062 00.00021)
0.91712
0.96392
0.91966
0.96582
(0.00036)
(0.00052)
(0.00012)
(0.00033)
(0.00013)
E
EXTENDED EXPERIMENTAL RESULTS
E. GAUC AND LOGLOSS
We also evaluated model performance using additional metrics, including GAUC (group area under the curve, grouped by category in our experiments) and Logloss (test loss). The results are presented in Tables and Our findings reveal that AUC and GAUC trends are consistent across all models. Logloss results largely follow the same trend, with the exception of two models: SDIM and TWIN- V2. Further analysis indicates that these two models tend to be "conservative." Let p+ represent the probability of positive outcome predicted by the model and p. represent the probability of negative outcome. The average value of maal(p+p- p-} 89.55% for DARE, compared to 85.90% for SDIM and 86.78% 78% for TWIN-V2. The prediction-confiiee are similar to DARE, whereas SDIM and TWIN-V V2 appear more conservative. This conservatism may help reduce their loss due the characteristics of cross-entropy loss, but offers tangible benefit for prediction accuracy or practical performance.
levels of the other seven models
E.2 GRADIENT CONFLICT ON TWIN
To better illustrate the universality of gradient conflict, we analyzed conflicts on a carecaty basis. Specifically, each category has its own embedding row in the embedding table), observed the
19

---

Published as a conference paper at ICLR 2025
Table 5: Overall comparison reported by the means and standard deviations of Logloss.
Setup
Embedding Dim. 16
Embedding Dim. = 64
Emhedding Dim. 128
Dataset
Taobao
Tmall
Taobao
Tmall
Taobao
Tmall
ETA (001)
0.69203
0.50732
0.64648
0.441566
0.64268
0.41796
(0.01410)
(0.00724)
(0.00546)
(0. (0.00600)
(0.00642)
(0.00339)
SDIM 2022
0.35402
0.29586
0.34250
0.29016
0.33376
0.29238
(0.00183)
(0.00262)
(0. (0.00288)
(0.00270)
(0.00220)
(0.00447)
DIN (2018)
0.70362
0.46560
0.68712
0.44864
0.68368
0,42956
(0. (0.00321)
(0. 00314)
(0. 00370)
(0. (0.00257)
(0.00852)
(0.00154)
TWIN (2023
0.67436
49222
0.68812
0, 43838
0.61176
0,40152
(0.00596)
(0. (0-00)45)
00370)
(0. (0.00)
(0.00574)
(0.00284)
TWIN (hard)
0.66888
0.47248
644244
0. 44020
0.63956
0,40956
(0.00145)
(0. (0.0033)))
(0. (0330)
(0.001 88)
(0.00344)
(0.00166)
TWIN (w/ proj.)
0.75762
0.48282
0.80758
0.50514
0.82670
0.49166
(0.008864)
(0.00/66))
(0.01286)
(0.02384)
(0.02953)
(0.01193)
TWIN (w/o TR)
0.71484
0.48388
0.67618 8
0.471 48
0.65368
0.45886
(0. (0.D01368)
(0.00785)
(0.00 (0.00223)
(0.00716)
(0.00542)
(0.00226)
TWIN-V2 20024
0.37096
0.27066
0.35412
0.24926
0.34526
0.23646
(0.00198)
(0.00194)
(0.00134)
(0.00167)
(0.00185)
(0.00139)
TWIN-4E
0. 0.71226
0. 0.48144
0. 0.70654
O, 7.46276
0. 0.68412
0,47432
(0.03604)
(0.00254)
(0.04449)
(0.00164)
(0.04719)
(0.05169)
DARE (Ours)
0.61922
0,41826
0.60132
0.39548
0.58960
0.38204
(0.00257)
(0.00181)
(0.00341)
(0.00289)
(0.00411)
(0.00142)
Conflict Ratio (%)
Average Category Frequency (%)
0~10
1.63
0.35
10~20
6.04
Category Ratio
0.30
20~30
24.15
0.25
30~40
45.61
0.20
40~50
46.11
50~60
64.81
0.10-
60~70
62.61
70~80
38.83
2.00
80~
14.52
Conflict Ratio(%)
90~~00
2.54
(a) Category- conflict in TWIN
Divide categories into groups by conflict ratio. This table shows average calegory frequency each group.
Figure 18: Conflict analysis on TWIN. The category frequency measured by the probability that a category appears in a batch.
gradient from attention and representation this category-wise embedding. We calculated the percentage iterations which a conflict was reported for each category, with the results shown in Figure 18 (The same method is used in Appendix C.3). Notably, 80.91 % of categories experienced conflicts in more than half of the iterations.
To explore whether conclusions like "popular categories more likely to experience conffict" exist, we further examined the relationship between category-wise conflict ratio and category frequency. To do this, grouped categories based their conflict ratios and calculated the average category popularity (measured as the probability of a category appearing in a batch) within each group. The results are presented in Table T8b] The differences observed are largely due statistical instability for categories that appear infrequently (for example, those categories appearing only once would have either 0% or 100% conflict ratio). However, there is no clear trend indicating that popular categories are either more or less prone to conflicts. This finding underscores the universality of gradient conflict the TWIN model.
E.3 LEARNED ATTENTION
Mutual Information (MI) is a measure the amount of information that two random variables share. quantifies the reduction in uncertainty about one variable given knowledge of another. In
20

---

Published as a conference paper at ICLR 2025
Correlations with Category 15
Correlations with Category
Ground Truth-
0 0.01
0.21
0.88
0.02
0
0.63
Ground Truth
0:43
0.15 0.15 0.06 0.03 0.02
0.83 0.01
DARE-
0.8E
0.4
0.58
0.97
0.17
DARE
3.92
0.49
0.28 0.18 0.17 0.14
0.47 0.08
TWIN-4E
0.48
0.33
0.51
0.69 0.
58
0.01
0.41
0.27
TWIN4E
0.38
0.1
0.03
0.01 0.01
TWIN
0.66
0.61
TWIN
0.7
10.53
0.44
0.21
0.27
0.1
0.03 0.08
TWIN w/ Proj.-
0
0.08
0.63
0.69
0.36
0.56
0.17
0.57
TWIN Proj.
DIN-0.00
0.1
0.04
DIN
0.45
0.12
0.37
0.15
0.51
0.6
0.16
11
18
i
16
16
16
19
Behavior Sequence
Behavior Sequence
Correlations with Category 16
Correlations with Category 16
Ground Truth
0
0
0.02 0.02 0.02 0.01
DARE
0.58
0.19
0.12 0.07
0.11
0.81
0.76
0.01
0.13
0.25 0.18
0.17
0.03
TWI-4E
0.32
0.11
0.09
0.12
0.21 02 0.15 0.1
TWIN-
0.71
0.6
0.5
0.2
0.6
0.3
0.19
0.02
TWIN
0.64
0.45
0.62
0.14
TWIN w/ Proj.
0.01
0.97
0.94
0.05
0.05
TWIN w/ Proj
0
0
0.02 0,02
0.02
0.02
DIN
0
0.08
0.01
DIN
0.04
0.1
0.01
0.91
0.13
0.38 0.34
202
148 148
19
16
16
16
9
2884 2884
78
10
Behavior Sequence
Behavior Sequence
Figure 19: More case studies of the retrieval performance in search stage.
our paper, we use the standard definition of MI:
p(x)p(v)
I(X;Y) Ep(a, v) log p(r,y)
where p(x),, p(v) and p(ar, y) computed based the statistical result of the training data.
More cases of comparison between ground truth mutual information and learned attention score are shown in Figure 20] Each line contains three pictures, where the first picture is the ground truth mutual information, while the second and third line is the learned attention score of TWIN and DARE. Our DARE model is closer the ground truth in all cases.
E.4 RETRIEVAL PERFORMANCE DURING SEARCH
More case studies of the retrieval result in the search stage are shown in Figure 19
F LIMITATION
There are also some limitations. We empirically find that linear projection only works with higher embedding dimensions, and small embedding dimensions would cause a severe "over confidence" problem. However, we still can't completely find out how this happened or what the underlying reasons are causing this strange phenomenon, which is left to future work. Besides, our AUC result in Section 4.2 indicates that target-aware representation benefits model performance in most cases, leading to an AUC increase of more than 1 1% on the Taobao dataset. However, on the Tmall dataset with embedding dimension 16, TWIN (w/o TR) outperforms TWIN, which is beyond tions. This is possibly due to some features of the Tmall dataset (e.g. fewer items), but we could not explain this result convincingly, which is also left to future work. Finally, although two-stage meth- ods are currently more prevalent, we also notice that there exists some one-stage methods like Yu al. (2024). The future of these one- stage methods remains an open question, which left for our research community.
expecta-
21

---

Published as a conference paper at ICLR 2025
Category-wise Target-aware Correlation
Attention Given Target Category ID 16
Attention Given Target Category ID 16
Top-10 Appeered Categories
x2
p.7 ont ast s.n e c-t 4.22 D.R
Top-10 Appeared Categories
- 19 10 11 +o
8-47 an 0.24 0:21 1: 0:13 0:66 no 03 02
B. 18 0.01 01 c.
Top-10 Appeared Catagories
0.000066600 0.M
D:33 D22
o.2 022 6.1
21 0
0.00 0:21 D0I
-0-0822002 021 0.16 0:25 0.00 0.05 0.11 c.11
-8.1 0.01 0.01
DE OZ
DE 6/22 D2
17
--0.00
0.0
--8.360 24 0.16 0:15 0:57 0.1 0.04 0.02 0.01
-0.01 0.01
0
12
D
D
0
12 22-0.30: 2. 0.19 0.16 0:23 D.1 0.04 0.22
0.11
-0.05 0. 0.00 0.02
0.01 CDI 201 0.11 0.01 CO
0
0
0
0
0
0
-0.31 0.2 0.16 0:18 0:89 0.00 0.69 0:11 6.11
-8007 0.04
0.08
0:02 0:22 0.02 0:1 o. 6.01 on
0
0
0
0
0.04
0.08
081 01 6.01
R.n0
0:1
0
D
0
2
10
Target relative Position
Target-relafive Position
relget-relative Position
Wate Targe Correlation
Attention Given Target Category ID 11
Attention Given Target Category ID 11
2
0:31 024 02 0.1 812 C.03 0:0 0.52 5.11
27
27-.05
005
0.04 D.03 D:22 oz 0222 c.11
10p-10 Appeared Categortes
CAE DAS DAT
Top-10 Appeared Categorias
11
046 0.26 031 0.15 0.11 5.01
Top-10 Appeared Categories
-8.34
0.28
003 0.02
012 0.05
0,22
0
0
0
19
2:45 023 0.2 011 812 0.00 0:82 0.01
8.08 085 0200 2
D
2:48 0.12 025 021 0:11 813 0.000000000000
001 006 0.0
D.B D. 2.7 2 C
22
0
0
0
0
0
@ 2--3.8 0.29 0.18 0:0 0.1 0.84062001 0.01
2
--002 682 0.01
0C1
0.1 0
0
0
0
012 0.000500.2250
606
0.22 an2 nre
0
R:001
8+ R-9.33 0.26 000 0.0022
22 22--11
10
rergel-relalive Position
Target-relative Position
Target-relative Position
Category-wise Target-aware Correlation
Attention Given Target Category ID
Attention Given Target Category
e 10.00
8,26 18 11 nod 07 0:53 nei néi o.
0.0 nmi
T0p-10 Appeared Categories
0.01 D:1
Top- 10 Appeared Categories
VE 87 6 11
Top- 10-110AM Appeared Calegories
0:23 287
231
CAE 41 0J5
0:4 0.32
0:55 0:47 0.27 0.32 0 0:11 0.1
R
023
4 5.000 0:01
0
D
D
O
D
0..5 0.19 0:15 0:12 0:06 8:c8 D:C3 0.22 n:11 0.01
10-0001 0.01
C
O
20.001 0:1
1..52 0.21 0.16 0.14 0.27 0:09 0:04 0.22 0.02 0.01
29-0.02 0.01 0.11
C
-00.01
15 1
0.14 0.07 0.29 0.22 0.02 0.01
2-0.01 0.01
2
0.28 0.19 0:15 0:12 0.28 8:08 0:23 0.22 0.51 0.01
22-001 0.11 D.11
0
Q.14 0.11 0.29 0:04 8:25 0.22 021
R. 0:00 0.00
0:01
0101 D:01
1
d
Target-relative Position
relgeve Posttion
rerget- Position
Category Target-aware Correlation
Attention Given Target Category ID 87
Attention Given Target Categary ID 87
9.400 02 e/a 0.68 ono oca DO do o
-.5 0.23 22 0.13 8:15 o.s c.s 0.04 0.55
2: #-00.7 0.1. carr 0.55 0.0 8:44 0:33 0.0 023 0.22
Top-10 Appeared Calagories
Top-10 Appeared Categorias
0.23 0.19 0:12 0:25 0.33 0.02 0:12
Tog-1 10 Appeared Callegories
2-.0 12 007 0.25 0:00 0.0 0:23 0:2 022 022 0.22
-..088 0.22 6.17
0.01 0.01
87
15
15
0.18015500 0.87 0:25 0:04
0.11
2
0DI DC1
0.1 DEl
0
-041 0.27 6.210.88 0.00 0.11 0.05 0.00 0.02 0.61
04 002 0/22 0.11 0.01 001 D01 0.01 051 O
0
0 0
0
0
-0.3 0.2 6.15 0.15 0.06 0.8 0/3
0.01
2 2.001 001 6:11 0. 0.
0
D
030 02 0:17.00 0.00 0.10 n:1
0.DI 6:02 n0l nni nc1 021
0
0
22
22-2.01
0:1
--0 0.2 0.120.00.0 0.01
R R-001
Target-rolative Position
Target-relative Position
Target-relative Posttion
(a) GT mutual information
(b) TWIN learned correlation
(c) DARE learned correlation
Figure 20: Comparison of learned attention
22


Reviews:
[{"review_id": "n9NkIBWofb", "reviewer": "Reviewer_vgn1", "summary": "This paper suggests the inefficiency of using the same embeddings for attention calculation and preference prediction (representations) in lifelong sequence recommendations, where the user behavior history (sequence) is usually very long (more than 10,000). Authors correspondingly propose a decoupling framework, DARE, to boost the recommendation performance by decoupling embeddings. Specifically, DARE sets separate embedding tables for attention calculation and preference prediction. Abundant preliminary studies and experimental evaluation support the effectiveness.", "questions": "1. What are potential application scenarios other than sequential models with attention and representation?\n\n2. To what extent would DARE increase the storage complexity? I believe it would be doubled.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Preliminary study is available. Authors provide experimental evidence supporting that using a shared embedding for attention calculation and preference prediction can impair performance due to the varying magnitudes and conflicting gradients. Furthermore, the authors examine the linear projection solutions of current methods, noting that the constrained embedding size may hinder the effectiveness of linear projection for decoupling. \n\n2. Clarity. The paper presents its ideas with remarkable clarity, making it not only easy to follow but also engaging for the reader.\n\n3. Experiments. Comprehensive experiments demonstrate the effectiveness of DARE. Various behavioral sequence models, including DIN, SDIM, ETA, and TWIN, are compared. A thorough analysis of attention and representation capabilities, as well as efficiency tests, is presented.", "weaknesses": "1. Applicability. This work aims to decouple attention and representation embeddings for sequence modeling. I wonder if the proposed solution in Section 2 can be extended to more scenarios. When the attention mechanism is not used in certain sequential models, the suggested method does not apply and, therefore, cannot benefit these models.\n\n2. Inconsistent Solution without Novelty. The proposed solution is too straightforward-setting separate embedding tables for attention and representation. This is not novel and does not align with the findings, although the separate embeddings can resolve the conflicting gradient and are not simply a linear projection. In my view, authors should go beyond linear projection. For instance, a linear projection-based decoupling method that addresses the gradient issue can be applied to small embedding sizes in recommendation systems, rather than reverting to separate embeddings, a very basic solution.\n\n3. Storage Complexity. Authors only provide the inference efficiency. However, for large-scale recommender systems, a separate embedding table can occupy exceptionally large storage space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TB2PKdsRcY", "reviewer": "Reviewer_JvL4", "summary": "The paper addresses a key limitation in existing long-sequence recommendation models, which struggle to balance the learning of attention and representation with a single set of embeddings. To tackle this, the authors introduce the Decoupled Attention and Representation Embeddings (DARE) model, which separates these two processes with distinct embeddings. Experiments show that DARE not only improves the accuracy of correlated behavior searches but also boosts efficiency, reducing embedding dimensions and accelerating the search by 50% without significant performance loss.", "questions": "1.\tWhy are the gradients from \\textbf{attention} significantly smaller than those from \\textbf{representation} in Fig. 2? And why is this imbalance undesirable during training (e.g., does it contradict theoretical derivations)? It is necessary to provide a theoretical analysis here.\n2.\tThe definition of conflicts between attention and representation is quite vague. The cancellation of gradients during backpropagation cannot be explained merely by the sign of the angle. Could you supplement this with the merged gradient values or in other ways to illustrate the degradation caused by conflicts to the embeddings?\n3.\tWhat are the consequences of these gradient issues? Do they lead to the failure of representation learning or the malfunction of certain modules? Would you add a presentation or explanation of the outcomes of the mentioned problem？\n4.\tHow does DARE solve the aforementioned gradient issues (gradient magnitude imbalance and gradient conflicts)?\n5.\tThe embedding dimension shown in Figure 6 should reach at least 200 to be consistent with the analysis in lines 208-213. Additionally, the analysis here can be summarized as: linear mapping requires a large embedding dimension to be effective, but increasing the embedding dimension leads to the dimensional collapse problem in recommendation systems. How does DARE address this problem? Does it only work effectively in low dimensions, or does it solve the dimensional collapse problem and work under any embedding capacity?\n6.\tHow are the embeddings between the two modules in DARE aligned? Why can the weights used to measure attention embeddings be directly utilized for representation embeddings?\n7.\tThe concept of Mutual Information （MI） is repeatedly mentioned in the experimental section, so it is necessary to introduce its calculation method. Moreover, this concept is already used in Figure 1, but it is not explained there. The connection between MI and the problem analyzed in this paper (Section 2) needs careful and thorough clarification.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The definition of the problem is very clear.\nThe research perspective is quite innovative.", "weaknesses": "The definition and explanation of the gradient issue are one-sided, and the motivation and the logic of the method are not clear. For example, why do the impact of different modules on embedding gradients must be ‘equal-magnitude’ and ‘consistent’? How do imbalance-magnitude and inconsistent gradients degrade recommendation performance? Additionally, the connection between the proposed method and the target problem is weak. For example, why does the embedding decoupling approach help mitigate the gradient issues? Overall, the logic and the motivation of the main idea need to be illustrated clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IU3jfx3dDJ", "reviewer": "Reviewer_ZXcu", "summary": "The paper focuses on long-sequence recommendation models. It first identifies a problem in existing models where a single set of embeddings struggles to handle both attention and representation learning, leading to interference. Initial attempts to address this using linear projections from language processing failed. Then it proposes the Decoupled Attention and Representation Embeddings (DARE) model. This model uses two distinct embedding tables for attention and representation, which are initialized and learned separately. Extensive experiments on public datasets and an online system show its effectiveness.", "questions": "How was the decision made to use two distinct embedding tables for attention and representation? Were there any other alternative architectures considered and why were they rejected?\n\nIn the DARE model, how sensitive is the performance to the choice of embedding dimensions for attention and representation? Have different combinations been explored and what are the trade-offs?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper clearly identifies a crucial problem in long-sequence recommendation models. The analysis of the interference between attention and representation learning due to shared embeddings is comprehensive.\n\n2. The extensive experiments on public datasets (Taobao and Tmall) and an online advertising platform are a major strength. The comparison with a wide range of baselines, including state-of-the-art models like TWIN and its variants, demonstrates the superiority of the DARE model. The evaluation metrics used, such as AUC, NDCG, and representation discriminability measures, provide a comprehensive assessment of the model's performance.", "weaknesses": "1. The performance on the short-sequence modeling using the Amazon dataset was marginal. This suggests that the DARE model may not be as effective in short-sequence scenarios, and more research is needed to understand its applicability in different sequence lengths. Additionally, the experimental setup for some baselines, like DIN, required adjustments to fit the long-sequence context, which may introduce some biases in the comparison.\n\n2. There are some results that lack a satisfactory explanation. For instance, on the Tmall dataset with embedding dimension = 16, TWIN (w/o TR) outperforms TWIN, which was unexpected. The authors could not convincingly explain this result, leaving a gap in the understanding of the model behavior under different dataset characteristics.\n\n3. In the exploration experiment, although the data is somewhat persuasive, some special cases or edge cases may not be fully covered. For example, it is uncertain whether different gradient relationships will occur under certain combinations of specific categories or user behavior patterns. Moreover, there is no further in-depth analysis of the reasons behind why it is nearly two-thirds of the cases, which may cause readers to have some doubts about the universality of the conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kaqv5D1iT8", "reviewer": "Reviewer_Rd7C", "summary": "This paper has focused on the interference between search and aggregation stages in the long-sequence recommendation. The authors first conducted some preliminary experiments and found the unbalanced and adverse gradients brought by the two stages. Inspired by such findings, they propose to adopt two individual embedding sets for the two stages, respectively, which may alleviate the conflicts between the two stages. The experiments on two public datasets validate that the proposed method can benefit the previous two-stage method, i.e., TWIN, and achieve SOTA performance.", "questions": "- Q1. Why the performance of TWIN-V2 is worse than TWIN, which does not align with the original paper? Are there some differences in experimental settings?\n\nAll of my other questions are included in the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- S1. This paper is well-written and -structured, which is easy to follow and understand.\n- S2. This paper is well-motivated, where the motivation is from an interesting preliminary study.\n- S3. The proposed method is simple but effective.", "weaknesses": "- W1.  I can understand the results and conclusion from the preliminary study, but the detailed process of the preliminary study is not clear. For example, I wonder how to calculate the gradients from the attention and representation to the embeddings.\n- W2. Only one metric is revealed in the experiments. GAUC and Logloss are often two common metrics in CTR prediction tasks. Thus, I suggest the authors add more metrics.\n- W3. The number of filtered behaviors, i.e., K in this paper, is often important to the two-stage method, but no related hyper-parameter experiment is conducted.\n- W4. It seems that the proposed method can only fit the two-stage method, such as TWIN. However, there still exist many one-stage works [1], which may illustrate that the two-stage method has not been the optimal solution. If so, the value of this paper is limited.\n\n[1]. Yu, W., Feng, C., Zhang, Y., Hu, L., Jiang, P., & Li, H. (2024). IFA: Interaction Fidelity Attention for Entire Lifelong Behaviour Sequence Modeling. arXiv preprint arXiv:2406.09742.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)