Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenders
Authors: Zhimin Chen
Chenyu Zhao
Ka Chun Mo
Yunjiang Jiang
Jane H. Lee
Khushhall Chandra Mahajan
Ning Jiang
Kai Ren
Charlie Li
Wen-Yun Yang
Venue: ICLR 2026 Poster (2026)
Keywords: sequential recommendation systems
generative recommendation
production-scale data
user interaction history
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose VISTA, a two-stage attention model, for efficient storage of extensive user histories and inference in a production recommendation system.

Abstract: Modern large-scale recommendation systems rely heavily on user interaction history sequences to enhance the model performance. 
The advent of large language models and sequential modeling techniques, particularly transformer architectures, has led to significant advancements (e.g., HSTU, SIM, and TWIN models). While scaling to ultra-long user histories (10k to 100k items) generally improves model performance, it also creates significant challenges on latency, queries per second (QPS) and GPU cost in industry-scale recommendation systems. Existing models do not adequately address these industrial scalability issues. In this paper, we propose a novel two-stage modeling framework, namely \emph{VIrtual Sequential Target Attention} (VISTA), which decomposes traditional target attention from a candidate item to user history items into two distinct stages: (1) user history summarization into a few hundred tokens; followed by (2) candidate item attention to those tokens. These summarization token embeddings are then cached in storage system and then utilized as sequence features for downstream model training and inference. This novel design for scalability enables VISTA to scale to lifelong user histories (up to one million items) while keeping downstream training and inference costs fixed, which is essential in industry.
Our approach achieves significant improvements in offline and online metrics and has been successfully deployed on an industrial platform serving billions of users.

Published as a conference paper at ICLR 2026
MASSIVE MEMORIZATION WITH HUNDREDS OF TRIL- LIONS OF PARAMETERS FOR SEQUENTIAL TRANS- DUCER GENERATIVE RECOMMENDERS
Zhimin Chenat", Chenyu Zhao't, Ka Chun Mo', Yunjiang Jiang', Jane H. Lee²t, Khushhall Chandra Mahajan', Ning Jiang', Kai Ren', Jinhui Li",, Yun Yang!
Meta, Yale University
ABSTRACT
Modern large-scale recommendation systems rely heavily user interac- tion history sequences to enhance the model performance. The advent of large language models and sequential modeling techniques, particularly transformer-like architectures, has led to significant advancements recently (e.g., HSTU, SIM, and TWIN models). While scaling to ultra-long user histories (10k to 100k items) generally improves model performance, it also creates significant challenges on latency, queries per second (QPS) and GPU cost in industry-scale recommendation systems. Existing models do not adequately address these industrial scalability issues. In this paper, we propose novel two-stage modeling framework, namely Virtual Sequential Targel Allention (VISTA), which decomposes traditional Larget attention from candidate item user history items into two distinct stages: (1) user history summarization into a few hundred tokens; followed by (2) candidate item attention those tokens. These summarization token embeddings are then cached in storage system and then utilized as sequence features for downstream model training and inference. This novel design for scala- bility enables VISTA to scale to lifelong user histories (up to one million items) while keeping downstream training and inference costs fixed, which is essential in industry. Our approach achieves significant improvements in offline and online metrics and has been successfully deployed on an industry leading recommendation flatform serving billions of users.
INTRODUCTION
Personalized recommendation systems services, o-commerce, and social media, where they boost user engagement and drive key metrics such click-through rates (CTR), session duration, and retention. The success of these systems hinges on their ability to accurately predict user preferences by processing and interpreting vast user historics.
integral to digital platforms like streaming
While traditional recommendation models, such as collaborative filtering (Sarwar et al. 2001) and matrix factorization (Koren et al., (2009), laid the groundwork for personalized recommendation, they often struggle to scale and capture long-term term user behaviors. Deep learning introduced more powerful solutions, and the recent integration of large language models (LLMss) and sequential modeling techniques such as transformers (Section 2 has marked a significant leap forward, enabling the capture of intricate interactions across vast user histories.
"These authors contributed equally to this work. Work done during internship at Meta. "orresponning authors: (zhimin, jinhui, wenyun (anogumm@mmmm.comm

---

Published as a conference paper at ICLR 2026
User interaction history (UIH) =O(100K)
Summarization =0((00)
Candidate
VISTA 2nd Stage
VISTA 1st Stage
Stancard Attention
Figure 1:
VISTA replaces standard attention with a two-stage process, allowing downstream models to compute only the highly efficient second stage.
In the domain of recommendation systems, two primary types of sequence modeling tech- niques have been explored: full user sequence modeling, as seen in Hierarchical Sequential Transduction Units (HSTU) (Zhai et al. 2024). and targel-specific sequence sampling, seen in Search-based Interest Modeling (SIM) (P et al. 2020) and its subsequent works (Chang al. 2023 Si al., 2024). Both approaches have demonstrated success in enhancing recommendation system performance by harnessing users' extensive historical interactions.
Despite their success, full sequence modeling suffers from the computational cost of scaling, Modeling full user interaction sequences which are usually on the scale of ((100)) in length often leads to enormous computational costs and latency issues, which are very challenging for industrial recommendation systems that need to train O(10B) to O(100B) examples per day and have strict latency upper limits during inference. As result, the full sequence modeling methods such as HSTU (Zhai et al. 2024) are hindered by high computational costs, limiting its widespread adoption across industries where many companies are still short of GPU capacities.
The second approach, harget-specifi sequence sampling, has been extensively explored through series of seminal works, including SIM (Pi et al. 2020), TWIN (Chang et al, 2023). and TWIN V2 (Si et al. 2024). These studies have demonstrated the effectiveness of leveraging user historical interaction sequences. However, subsequent research in this direction has encountered two significant challenges: (1) bridging the gap between attention to the target-specific shortened sequence and the full user sequence, which, however, was partially addressed in follow-up work TWIN (Chang et al. 2023): and (2) the computational cost increases linearly with the number of candidates the independent target-specific sequences. These two challenges remain largely unresolved, primarily due to the inherent design limitations of SIM-style models.
predict inference time, due to
Addressing the challenges of scalability in recommendation systems will assist with their widespread adoption. In this paper, we propose a novel two-stage modeling framework, VIrtual Sequential Target Attention (VISTA), designed to overcome the scalability challenges. The first stage compresses the ultra-long user interaction history into a few hundred of summarization embeddings (see Fig. 1 the second stage serves efficient candidate aware target attention mechanism that leverages the summarization from the first stage for final prediction. The first stage occurs only during foundational model training, where the resulting summarization embeddings are cached to conceptually represent user embeddings. Consequently, downstream model training and inference only need to perform the second stage: computing attention between a candidate item and these cached embeddings, instead of processing the full user interaction history. This approach significantly reduces the computational complexity for downstream models, especially during inference, at the cost of additional storage. In practice, this is worthwhile tradc-off, as the cost of GPU computation remains multiple orders of magnitude higher than the cost of storage.
As a summary of our contributions, to the best of our knowledge we are the first. to propose:
A two-stage attention framework to decouple foundational model training and downstream model training and inference, which enables us to leverage aitra-long user historics for better recommendation model performance in industrial-scale systems,
A quasi-linear attention formulation tailored for recommendation models,
A generative sequential reconstruction loss in recommendation models, and
A practical embedding delivery system successfully deployed in an industrial-scale platform.
2

---

Published as a conference paper at ICLR 2026
Embedding Export [Traiiing Only.
Dequoniizaton (Inforance Only)
entedding Export
Dequintzaioo
K/ indir Cache
UIH Summanzzaion [Triiiing Only
(Triiiing Inference)
Target Attention
Cronddin
Irivencs
O0000) U www. Entociiing
O11081 Sood briakkin
00100 Surmarkarkn Entekdig
ond Conbile broakingg
Figure 2: An overview of VISTA architecture.
2 RELATED WORK
Hierarchical Sequential Transduction Unit (HSTU). A significant advancement in this area is the Hierarchical Sequential Transduction Unit (HSTU) Zhai et al. 2024), which reframes recommendation as scquential transduction problem. Designed specifically for high-cardinnlitt, non-stationary streaming recommendation data, HSTU surpasses traditional models in both accuracy and efficiency. This architecture allows recommendation systems to scale to trillions of parameters, leading to substantial gains in predictive performance.
Transformer Architectures Recommendation Systems. The application trans- former architectures in recommendation systems has been explored extensively. By leveraging the self-attention mechanism, transformers can model complex user-item interactions over time, facilitating
nuanced and personalized recommendations (Subbiah and Aggarwal 2024j. The Deep Interest Network (DIN) [Zhou et al. 20188) and its Tollow-up work, Search- based Interest Modeling (SIM) (Pi et al. 2020; Chang al. 2023 Si et al. 2024), leverage lifelong sequential behavior data. This approach employs
mechanisms, also known as General Search Units (GSUs), to select a small subset of relevant interactions from the user's historical sequence that are pertinent to the target item followed by a standard transformer network, referred to as Fxacc Search Units (ESUss, to compute the final target item representation. Notably, this method enables the modeling of user behavior data with lengths reaching up to hundreds thousands (Pi et al. 2020). Other methods (Liu et al. 2023 preprocess user histories into groups and attend to the group embeddings, and separately attend to subsequences in the user history relovant to the target item.
Linear Complexity Attention Mechanisms. Apart from Flash Attention (Dao et al.)
2022 Dao 2023) that is designed to improve the efficiency of the softmax attention mecha- nisms, there is new trend to explore linear complexity attention mechanisms. Katharopoulos et al. 2020) first proposes linear attention. By applying matrix multiplication associative property, enables change in computation order from (QKT)V Q(KTV), reducing computation complexity from O to O(N) with respect to sequence length N. Recently, Lightning Attention (Qin et al. 2024a) and v2 (Qin et al. 2024b propose light network which contains Gated Linear Attention (GLA) and Simple Gated Linear Unit (SGLU) to make linear attention more practical. Another branch of linear complexity work, namely state space model (SSM), has been widely studied. Mamba (Gu and Dao 2024] is pioneering work in SSM and widely used in many real-world applications, tollowed by Hydra (Hwang which is the double-headed version of Mamba to address non-causal scenarios.
et. al.
2024
3

---

Published as a conference paper at ICLR 2026
3 METHOD
Here we introduce the details of VISTA's 's two cascaded modules: ultra-long user interaction history (UIH) sequence summarization and target-aware attention, followed by details of practical linear complexity self-attention and generative sequence reconstruction loss. We then explain how VISTA's design enables the scaling, storage, and procossing of industry-scale user history sequences through its embedding delivery system.
3.1 MODEL ARCHITECTURE OVERVIEW
the VISTA architecture employs distinct workflows for training and inference. During training. the computationally expensive UIH summarization module runs to generate summary embeddings. These embeddings then quantized and exported to large key-value cache in O(100) terabytes to O(1) petabytes. For inference, this expensive step is bypassed entirely. Instead, the pre-computed embeddings are simply retrieved from the cache and dequantized with minimal distortion. The final component, the larget allention module, operates in both phases, using the summarization embeddings and candidate item features to make predictions.
As illustrated in Figure 2
3.2 ULTRA-LONG UIH SEQUENCE SUMMARIZATION
In the first stage, we utilize self-attention with virtual seed embeddings to summarize ultra-long UIH se- quences. These virtual seeds are initialized randomly as shared parameters across users, which are updated with the model through its interaction with the UTH
Clustered Embeddings Colored by Country
80
Country
Other
PCA 2
60
MX
sequence in the summarization module. The output of the summarization module can be interpreted as user embeddings, encoding individual personalized preferences to inform recommendations. Figure 3 vi- sualizes these summarization embeddings, projected onto the first 2 principal components by principal component. analysis (PCA). We clearly see the separation for users of different countries, with US and Canada overlapping, which is expected.
US
60
-40
-20
PCA
Component
Figure 3: Visualization of UIH sum- marization embeddings.
However, typical softmax attentions suffer from O(N2) time complexity, which is prohibitive when dealing with ultra-long sequences (N > 10k). Therefore, we propose quasi-linear attention (QLA), linear time complexity O(N) self-attention mechanism to overcome this issue.
3.2.1 LINEAR ATTENTION WITH CANDIDATE ITEMS FOR RECOMMENDATION
With the emergence of Large Language Models (LLMs), researchers have proposed some linear complexity attention algorithms accelerate transformer blocks (Katharopoulos et al., 2020 Qin et al.] 2024a b) Han et al. 2024h. However in recommendation systems, unlike the text sequences in LLM, a strict rule is that the candidates cannot attend each other, since it introduces label Icakage due to the fact that the logged candidates typically only form small subset of the input candidates during inforence. Therefore, we propose a linear-complexity self altention mechanism that avoids allention among candidates.
The typical softmax self attention for a UIH sequence S can be formulated as follows SoftmaxAttn(S =fall S) RowSoftmax(QK
where Q, K and V have shape (T, d) and is the sequence length. Then the original linear attention (Katharopoulos al. 2020 for a UIH sequence S can be written similarly as follows
LinAttn(S Small S) = RowNNormalizeQK" J
(1)
=Q(K =Q((TV)/Roww) V)//ooSum(QK"" VV/(Q (K))
4

---

Published as a conference paper at ICLR 2026
Lecommnucann
Ougut
Output
Add
SUU
Cassal Soltiic
QLU
SLU
SIUL
Sohmaa Attention
LayerNorm
01100) Summerizaiion Emteddding
Q1100K) Ulte long UM Embersing
Figure 4: The QLU module.
Figure 5: Generative reconstruction loss.
Note that division / here stands for broadcast division along the rows. The above can be applied to full (bi-directional) self-attention.
In recommendation models, we also have target (candidate) items, let's denote them by T. Then want to compute target attention of T against K and V.
LinAttn(T sulll S) T(K T)KTV)/(T (T ColSum(K))))
(2)
Note that candidates cannot attend to each other. This is strict rule in recommendation systems otherwise the model training will fail due the leakage between candidate items. It gets slightly trickier if we also want each candidate to altend to itself. Instead of TT T, the contribution due to the self attention of each target item to itself is given by
LinAttn individual T) Diag(TT)))
(3)
3.2.2 QUASI-LINEAR ATTENTION FOR RECOMMENDATION
Despite its efficiency, some previous works (Han et al., 2024 2023 prove that linear attention suffers from insufficient expressive power, making it impractical lor real applications. In this section, we introduce quasi-linear attention (QLA) as an empirically effective linear attention algorithm for recommendation. This quasi-linear attention introduces more non-linear complexity in attention computation, addressing the issue expressive power.
The quasi-linear attention contains two parts: Quasi Linear Unit (QLU) module and Simple Gated Linear Unit (SGLU) module. The QLU module aims to model the interaction of Q, K. V matrices with SiLU non-linear activation as shown in Figure For the SGLU module, we use the same gated function as TransNormerL.LM (Qin et al. 2024a).
Accordingly, we need to slightly modify the above linear attention formulation to accommodate this QLU modulc. For the self attention part we let the user history items attend to one another. Similar as in HSTU (Zhai et al.) 2024), SASRec (Kang and McAuley 2018). and Pinnerformer (Pancha et al.] 2022,, usually the causal sell-attention approach via triangular mask is used. In our case, did not find significant difference between causal and full self attention, since the user history items merely serve as features for the final candidate prediction task their temporal causality is not strict requirement. Let i denote non-linear activation function (we use SILU in our experiments), then the full self quasi-linear attention modified from Eq. ①1]
as follows.
OiS] = c(Q[S))p(((((S)) V[S))
where [S] denotes the source (user history) portion of the sequence. Note that we remove the RowNormalize operation, similarly in Lightning Attention (Qin et al. 2024a b).
For the target portion the query sequence embeddings, we can similarly apply p-linear attention bet Q[T) and K(S), V(S). However to be consistent with the self-attention
5

---

Published as a conference paper at ICLR 2026
semantics, we also include extra term that captures attention to the target item itself. Thus, the final formula for the target portion of the quasi-linear attention, modified from Eq. and (3) is given by
O ((Q(I))((((((S)
V(()) (((((((T))
Here ,(X, := = XIx stands for putting the row-wise dot product between the two matrices X and of shape n x m on the diagonal of square matrix of shape n X n. In order to implement the quasi-linear attention efficiently using the Triton language (Tillet et al., 2019) for optimized GPU computation performance, we also calculate the gradient of the final Loss function with respect to the imput tensors Q[S). Q K[S), K[T), V[S) V(T),
in terms of the gradient with respect to the output tensor O[S], [[]] in Appendix B
3.2.3 GENERATIVE SEQUENCE RECONSTRUCTION LOss
To further enhance the memorization effects, we also introduce a reconstruction loss (see Fig. to encourage the sequence summarization to fully reproduce the UIH sequence, which we find particularly useful improve VISTA's performance. Intuitively, reconstruct the i-th UIH item embedding, we are using all the seed embeddings and the UIH item embeddings up to the 1)-th position. A natural way to accomplish this via the decoder network, such as the causal transformer decoder, without the softmax layer. Formally,
(1,,.., ,,,-..,,M. = Decoder(s1,..... 8k: (1 UI....,uMM).
uM)
where 81,..... 8% are the personalized seed embeddings, and UI,..., Mar are the UIH item embeddings. We can feed their concalenation through the causal softmax atlention block (or any other transformer block) to get the output embeddings concatenated as .,.... lx and U1...., UM where is the number of seeds and M the length of the user history sequence. Then we can simply form the off- one mean square error of the U, 's with the u, 's as the construction loss as Irseonstruct = Σi M-1 ||0: w++1
......,4g
Since causal transformer block ensures that the output embedding is only depends Wi, W1,...,u, U4 there is no leak information from Uj+1 to u;- This forces the personalized seed embeddings s, maximize information retained of the user history sequence ......y UM- Similar ideas have roots in the Variational Into-EEcoode (Kingma and Welling| (0022). and have appeared in the context of transformer networks recently (Henderson and Fehr 2022). However to the best of our knowledge, there has not been any explicit use in rocommendation. For more discussion on this reconstruction loss, see Appendix
3.3 TARGET-AWARE ATTENTION
As shown in Figure 2 any attention network can technically be used for the target-aware attention stage. Because this step is computationally inexpensive compared to sequence summarization, we selected standard O(N2) transformer block, which delivers excellent performance on the compact summary sequences.
EMBEDDING DELIVERY SYSTEM
We emphasize that the VISTA framework is not merely a theoretical model, but a novel industrial model system co-design to support large scale user interaction history sequence learning that can be deployed into the real industry infrastructure with reasonable cost.
Figure outlines the system's 's and-to-end architecture, which comprises three main stages: (L) online training of the source model using training data stream, (2) delivery of sequence summarization embeddings to downstream models via two routes: a real-time message queue, e.g., Kafka (Kreps et al. 2011) h and persistent storage, e.g., Hive (Thusoo et al. 2009), and (3) serving embeddings through a geographically replicated in-memory key-value store. In our system, we update the summarization embeddings a 2-hour cadence, which was shown to have similar performance compared to using the summarization module directly in online A/B tests. This design ensures both real-time performance and scalability for industrial applications. For scalability, we deliberately compress the user interaction history sequence to O(100) terabytes level, making it feasible to deploy to existing systems.

---

Published as a conference paper at ICLR 2026
United boining data steat
Soure Traningg
Masoagga Quman
Peplicaton
Online Trannnng Model
Online Training Mode
Online Training
Online Troiiing
Online Traninn
Reg on
Rogion 2
Rogion
Figure 6: An overview of VISTA sequence summarization embedding delivery system.
EXPERIMENTS
5.1 DATASETS AND EXPERIMENTAL SETUP
The proposed VISTA framework designed for large scale real-world dataset, where one needs to train hundreds of billions of examples per day and each user has a history which contains hundreds of thousands of items. While existing public datasets are usually much smaller, compare method against several baselines on public datasets in addition to reporting results on real production data.
5.1.1 PUBLIC DATASET AND INDUSTRIAL-SCALA DATASET
We first compare the effectiveness of VISTA against several baseline models on public datasets Amazon-Flectronies P and KuaiRand- To focus mainly on the effectiveness of the attention mechanism, we compare VISTA against. baselines in replacing the attention layers in a common model architecture. All models are implemented, trained, and evaluated under the FuxiCTR framework, focusing on click-through rate prediction. Additionally, we introduce a Minimal Production dataset from real production data, compatible with FuxiCTR having minimal features but with longer sequences up to 2,000.
For industrial-scale offline experimentation, we construct full training and evaluation samples from real production data, with several metrics for engagement, which we denote by "C- Task", "E1-Task", etc. We use 3-day data as the training set and the next 1-day data as the evaluation set in offline experiment. The scale of training examples per day is at O(10) billion level. The average and maximun UIH sequence lengths are 7,000 and 16,000, respectively. Note that we deploy the model with 12,000 UIH sequence length in online experiments, but we only use 2,000 in offline experiments due to GPU resource constraints.
Table Dataset Statistics
Dataset
Mean Seg.
Max Seg.
Amazon- Electronics
8.93
429
KuaiRand-1K IK
225.20
256
Simplified Prod
1528.18
2,000
Industrial-Scale Data
7,000
16,000
hhttps:/ / www.lu/bbooo//caaaaaoo Datasets/ tree/ maain/ Amazon/ AmasoonEectrooiis x1 httrps:/ kwwirand.comm
haapss//giihubb.oo. /wrpe/iccccooorcoooo////oooooo// FuxiCTR.

---

Published as a conference paper at ICLR 2026
Table 2: Comparisons on public and Minimal Production datasets. HISTA-w/QLAA and
VISTA-w/O-Q0 are the VISTA model with and without quasi-linear altetion, respectively
Models
Amazon
Kuailtadd
Mininal
Production
(TC
[E]
AUCT7
NE (U)
AUC. (1)
NE
DIN
0.873 = Sc 4
0.656 + le 4
0.744 +0.003
0.864 + 0.005
0.632 0.00
1.048 0.033
TISN
0.877 0.005
0.010
0.7.0=0.000
0.809 + 0.001
0.648 - 0.005
1.139 +::::00000 0.156
MHA
0,881 = 1s 4
0.634 0.002
0.743 2.000
0.803 1 0.005
0.630 = 0.018
1.049 1 0.041
SASRec
0.884 = 4
0.627 :: 0.001
0.742
0.868 + 0.007
0.605 = 0.020
1.129 4 0.131
HSTU
0.884 + 0.001
0.628 0.001
0.743 0.001
0.863 1c
0.668 0.011
1.099 0.048
VISTA- QLA
0.886 1 0.002
0.621
1. 0.005
0.744
0.001
0.863 1 0.003
0.627 1 0.016
1.038 1 0.05
VISTA QLAA
0.884 0.005
0.623 0.003
0.743
0.864 0.001
0.632 0.013
1.062 0.076
AUC Across Sequence Lengths
Across Sequence Lengths
Parcentage Decrease in Running Time
1.3
0.62
Vali
-00
4
0.30
Mode
1.0
1.1
-300
masstontes
-800
2010
2050
1004
2000
Sequence Lengths
Sequence Lengths
Sequence Length
Figure 7: Ablation study on quasi-lincar attention by varying sequence length.
5.1.2 BASELINES AND EVALUATION METRICS
All models share a common fcature embodding layer and MLP block, with consistent hyperparameters. c.g., embedding dimensions, layers, attention heads, for fair comparison. We briefly describe them: (1) Deep Interest Network (DIN) (Zhou et al. 2018 uses attention to adaplively weigh user historical behaviors, (2) Two-Tower Sparse Network (TTSN) (Covington ett al.) 2016) separately encodes user and item features via two towers, (3) the standard Attention (MHA) (Vaswani et al. 2017). (4) SASRec (Kang and McAuley 2018 is self-attentive sequential recommendation model that uses the Transtormer architecture, and (5) Hierarchical Sequential Transduction Units (HSTU) (Zhai et al.) 2024 is an industry proposed transformer-Iie model designed to capture multi-scale sequential patterns in user behavior sequence.
We use normalized entropy (NE) (He et al. 2014 as our evaluation metric, which calculates the cross entropy between the predicted probabilities and the labels, then normalizes it by the entropy of the constant predictor at label average. We also report the area under curve (AUC) for the traditional setting. Additional details about this section are in Appendix D
5.2 OFFLINE EXPERIMENTAL RESULTS
5.2.1 PUBLIC DATASET RESULTS
In Table 2 we summarize the comparative results between VISTA and the baseline models. dataset, VISTA outperforms the other baselines with the use of
For the quasi-linear attention being the next best model. On the KuaiRand dataset, VISTA slightly outperforms the other models with similar NE to HSTU and MHA. This may suggest that even at smaller sequence lengths, the virtual sceding embeddings slightly help the model performance. On much longer sequences in Minimal Production, we see that HSTU and VISTA perform best demonstrating that both are designed for handling longer sequences.
Figure shows the ablation study results of quasi-linear attention by varying the sequence length on the Minimal Production dataset. We can clearly see that the QLA mechanism significantly reduces the the time to train and evaluate epoch of data, while there are small differences in AUC and NE.
Figure g shows the scaling law of inereasing number of seed embeddings. We can clearly see that the model performance improves with larger number of seed embeddings, which,
'Comparisons to the next best model are significant at the p < 0.001 level when using paired t-test. ANOVA tests with multiple next-best models are significant. the p 0.001 level.
8

---

Published as a conference paper at ICLR 2026
Compartoon of Execution Time (interence)
Lengtlis
AUC across Number of Different Seeds
NE across Different Number of Seeds
0,887
0.88
0
0.80
0.66
0.85
0:62
0.00
Number of Seads
Number of Sceds
1350
2349
495
Figure 8: Ablating VISTA across number of seed embeddings on
F Figure 9: Inference time gains with increasing UIH lengths.
Similarity Heatmap
Similarity Heatmap
Similarity Heatmap
Emtrndiings
4127
3-5 EUGTE
AM SURRET CEREC
Target Eww.eeeiei
6:32 0L.0
Target Embefflnge.
6:440
2.E00 BLSA
1.99
5.51
0:643
0.194
0.57
6409
0/06
6.42
9.791
0.403
6:52
1:00
0.004
0.52
0,50
53607
2271
Target Emboddingo
Target Emboddingg
Target Embboddingg
(a) Same category (142).
(b) Same category (368).
(c) Different categories (each).
Pairwise cosine similarity of the output of two-stage altention of VISTA. For the same user, compare the target attention output for items of similar different categories.
Figure 10:
however, will incur more storage capacity cost in real world production scenario. Thus, in practice is a tradcoff between model performance and financial cost.
Figure 10 compares the target embeddings, the output of VISTA's two-stage attention mod- ules, for different items given the same user history from Amazon-Eless roeccs. Embeddings between ilems from the same category are more similar than those from different ones, as expected.
5.2.2 INDUSTRIAL-SCAAN DATASET RESULTS
In Table 3 we compare our proposed VISTA model with the baseline production model using HSTU as the backbone in both offline and online experiments, on the industrial-scale dataset, In this setting, there are multiple tasks which measure different aspects of engagement information. We report the main consumption task ("C-Task"), and other engagement events ("E1-Task", model, we also conduct ablation studies by varying the embedding dimension, the number of seeds, and the use of generative reconstruction loss. As shorthands, (1) VISTA stands for the optimized proposed model co-trained with the baseline HSTU model, with 3-layer self-attention, 3-layer target-aware attention, 128 seeds, 256 embedding dimension and 2, 0000 UIH sequence length. (2) VISTA-1281 stands for VISTA model with 128 dimension embedding. (3) VISTA-64Sccd stands for VISTA model with 64 sceds. (4) VISTA-w Biil-w/ooRRoooo stands for VISTA model without generative reconstruction loss. The results demonstrate that our optimized VISTA configuration (128 seeds, 256 embedding dimension, and 2, 000 UIH sequence) significantly outperforms the standalone HSTU baseline for training and evaluation NE metrics.
and "E3- Task"). To further understand the effectiveness of the
Table g summarizes the performance improvement of QLA compared with the standard self- attention on production dataset, where we can see that the QLA able to scale up with more layers and longer sequence for better NE metrics and even higher QPS.
Figure g shows the VISTA's advantage on inference performance, especially for much longer sequence lengths. This is expected since VISTA's main strength is to cache the UIH summarization. Thus, the most computationally expensive module, UIH summarization module, is deactivated during inference.
9

---

Published as a conference paper at ICLR 2026
Table Offline comparative results with the baseline model and ablation models.
Training NE (L)
Fval Task
NE (J)
Models
C-Task
EI-Task
E2- Task
E3 Task
C-Task
E2- Task
E3 Task
HITTU
VISTA
-0.47%
-0.82%
-2.30%
-1.72%
-0.40%
-1.19%
-2.98%
-2.23%
VISTA-128D
-0.32%
-0.50%
-1.86%
-1.43%
-0.29%
-1.07%
-2.51%
-1.82%
VISTA-64Soed
-0.36%
-0.68%
-1.70%
-1.45%
-0.37%
-1.11%
-3.01%
-2.00%
VISIA-W VIS12-w/o-Recon /o-Reoo
-0.42%
-0.72%
-2.32%
-1.69%
-0.29%
-1.29%
-3.00%
-2.21%
Table 4: Comparing VISTA with and without quasi-linear attention.
Model Variant
Max Seq.
Layers
QPS (f)
Training NE (4)
Eval NE (4)
VISTA-w //QQLA
6,000
3
VISTA-W/-QIA
16,000
5
+-5%
-0.1%
-0.13%
5.3 ONLINE A/B EXPERIMENTAL RESULTS
We conducted an online A/B test on our production video recommendation system, using 5% of the entire site traffic during a period of 15 days. The baseline is the HSTU model and we compare with adding the VISTA module, which is the same our oflline experiment setup. Online metrics for the main consumption task "C-Task" and other online onboarding metrics, "O1-Task" and "ask", were significantly improved by 0.5%, 0.2%, 0.04%, respectively. VISTA demonstrated a 94% reduction in inference GPU resource (measured by inference QPS) usage by eaching and serving embeddings, rather than re-computing them for every new user request. With a 0.01% "O2-Task" gain considered a substantial improvement on our platform, the VISTA model made realized contributions to the recommendation system.
CONCLUSION AND DISCUSSIONS
In this paper, we have proposed the VIrtual Sequential Target Attention (VISTA) framework, a novel two-stage approach that compresses ultra-long user interaction historics into a set of compact embeddings. This design strikes crucial balance between computational efficiency and predictive accuracy, addressing the latency and scalability challenges of processing ultra-long user sequence data in production systems. VISTA's practical applicability is underscored by its resilience to slight de-synchronization between its stages and its ability to approximate complex transformer architectures without their substantial computational cost. Our empirical evaluations demonstrate that VISTA not only captures the core information within user interactions but also achieves significant improvements across platform metrics. Our plans for future rescarch involve further optimizing VISTA's compression techniques and exploring applications across other domains to enhance its generalizability.
ACKNOWLEIGEMENT
This work results from a large cross organization collaboration. It would not be possible without contributions from the collaborators and supports from the leaderships as follows (alphabetic order): Zheng- Yong Ang, David Bauer, Connor Chen, Shouwei Chen, Siqiao Chen, Huihui Cheng, Ek Kheng Chung, Litao Deng, Shilin Ding, Chenhao Feng, Kevin Goulding, Liang Guo, Mengyue Hang, Maxwell Lin-He, Xiaoxin He, Chufeng Hu, Jizhou Huang, Yanzun Huang, Han Jiang, Justin Khim, Emma Lin, Zihan Li, Yang Liu, Yining Liu, Li Lu, Wenhan Lyu, Jing Ma, Matt Ma, Jing Qian, Rui Qiao, Chuyu Qiu, Yongxiong Ren, Xinyue Shen, Daisy Shi, Hongzheng Shi, Ge Song, Yisong Song, Wanting Tan, Hao Wan, Meihong Wang, Yanhong Wu, Hong Yan, Yihang Yang, Chuanwei Yi, Christina You, Haoli Zhang, Rui Zhang, Yue Zhang, John Zheng, Xinye Zheng, Lizhen Zhu, Maggie Zhuang.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, and Kun Gai. 2023. TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou. arXiv:2302.02355 [cs.IR]
https://arxiv. wwwp://iiooroooeeooeeee awtpaa////// 2302. 02352
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for TubTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys '16). Association for Computing Machinery, New York, NY, USA, 191 198. doi:100 1011/19591100000 299999999999
Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv:2307.0866 [cs.LG] attps://arxiv.orgg/
08691
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and (eeemoryEEfiicieee Exact Attention with 10-Awareness. arXiv:2205.141: [cs.LG] https://arxiv:: org abs 14135
Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022. KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos. In Proceedings of the 31st ACM International Conference Information and Knowledge Management. (Atlanta, GA, USA) (CIKM '22). 3953-3957. doi:10. 00145//351800 3557624
Albert Gu and Tri Dao. 2024. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00732 [cs.T.G] https://arxiv.orge http//abb/222205555555
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. 2023. FLatten Transformer: Vision Transformer using Focused Linear Attention. arXiv:2308 200:2300000042 [cs.CV]
org/ www.///3..0000
Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, and Gao Huang, 2024. Bridging the Divide: Reconsidering Softmax and Linear Attention. arXiv:2412.06900 [cs.CV] https: //as://///xvv org/ www.////////eeoeeeo/ooo/eee abs/2412 2412.06590
Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. 2014. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the eighth international workshop on data mining for online advertising. 1-9.
James Henderson and Fabio Fehr. 2022. A Variational AutoEncoder for Transformors with Nonparametric Variational Information Bottleneck. arXiv:2207.13529 [es.LG| https //arxiv. wwt.//gg//////a/aeeeo /20/ 13529
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. 2024. Bridging Language and Items for Retrieval and Recommendation.. arxiv preprint 4rXiv:2403. 03952 (2024).
Sukjun Hwang, Aakash Lahoti, Tri Dao, and Albert Gu. 2024. Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers. arXiv:2407.09941 [cs.LG] https //aaxivv.ooo org/ abs
Chan-Cheng Kang and Julian McAuley, 2018. Self-Attentive Sequential Recommendation. 09781
arXiv:180S.0978) [cs.IR]
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Trans- formers are rnns: Fast autoregressive transformers with linear attention. International conference on machine learning. PMLR, $156-5165,
Diederik P Kingma and Max Welling. 2022. Auto- Anto-EEcodiig Variational Bayes. arXiv:1312.6114 [s.) https://arxiv. http:://arrv..oo///b///3111111 wtt.o///b////1//111444
11

---

Published as a conference paper at ICLR 2026
Ychuda Koren, Robert Bell, and Chris Volinsky, 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (2009), 30-37. doi:10. 1109/MC. 200.. 263
Jay Kreps, Ncha Narkhede, Jun Rao, et al. 2011. Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB, Vol. 11. Athens, Groece, 1-7.
Qi Liu, Xuyang Hou, Haoran Jin, Jin Chen, Zhe Wang, Defu Lian, Tan Qu, Jia Cheng, and Jun Lei. 2023. Deep Group Interest Modeling of Full Lifelong User Behaviors for CTR Prediction. CoRR (2023).
Nikil Pancha, Andrew Zhai, Jure Leskovee, and Charles Rosenberg. 2022. PinnerFormer: Sequence Modeling for User Representation at Pinterest. arXiv:2205.0507 [cs.T.G] https
7/ //arxiv. org/ abs 2205./2205..022 2205.04507
Qi Pi, Xiaoqiang Zhu; Guorui Zhou, Yujing Zhang, Zhe Wang. Lejian Ren, Ying Fan, and Kun Gai. 2020. Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Blick-Through Rate Prediction. In Proceedings of the 29th ACM International Conference on Information. & Knowledge Management (CIKM). ACM. https://doi
org/10. 10. 1145 1145/3340531..... 3412744
Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Xiao Luo, Yu Qiao, and Yiran Zhong. 2024a. TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer. arXiv:2307.14998 [cs.CL] https://arxiv. org/ aba/2307 14995
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong- 2024b.
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models. arXiv:2401.c 1.4688 [cs.CT] https: : h/tp://axxvvvv/////ee/a. org/ aaaaa/ss//2200 046658
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based col- laborative filtering recommendation algorithms. In Proceedings of the 10th International Conference on World Wide Web (Hong Kong, Hong Kong) WWWW '01). Association for Computing Machinery, New York, NY, USA, 285 285-295. doi:10. 1145/31 /144/337999 372077
Zihua Si, Lin Guan, Zhongxiang Sun, Xiaoxue Zang, Jing Lu, Yiqun Hui, Xingchao Cao, Zeyu Yang, Yichen Zheng, Dewei Leng, Kai Zheng, Chenbin Zhang, Yanan Niu, Yang Song, and Kun Gai. 2024. TWIN V2: Scaling Ultra-Long User Behavior Sequence Modeling for Enhanced CTR Prediction at Kuaishou. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM 244.. ACM, 4890-4897, doi:10. 0664/36677633368000 3680030
Anushya Subbiah and Vikram Aggarwal. 2024. Transformers in music recommendation. in- www.-mmmisii--eeeemeeeiiiiii
https://research. wwwgg///bol//trrrsottrooorrooeee
Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad Chakka, Suresh Anthony, Hao Liu, Pete Wyckoff, and Raghotham Murthy. 2009. Hive: warehousing solution over a map-reduce framework. Proceedings of the VLDB Endowment 2, 2 (2009), 1629.
Philippe Tillet, H. T. Kung, and David Cox. 2019. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL 19). ACM, 10. doi:10. . 1
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Atlention Is All You Need. In Advances in Newral Information Processing Systems (NewrIPS). htttps//paapers .ntsssss.ss..... cc/paper/
1e8l-attention is- all-all--- yoll-you--o you--oodddddddd... need. .df
Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojic Gong, Fangda Gu, Michael He, Yinghai Lu, and Yu Shi. 2024. Actions Speak Louder than Words:
Sequential Transducers for Generative Recommendations. httpps///rras http:////axx..oorr/a///22022. wtg/abs///02...11b...g 17152
arXiv preprint a7X1v:2402.17152.
12

---

Published as a conference paper at ICLR 2026
Guorui Zhou, Chengru Song, Xiaoqiang Zhu, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click- Throughh Rate Prediction. arXiv:17066
|stat.ML] https://arxiv.. org/ abs/ 00777067688788888
Jieming Zhu, Quanyu Dai, Liangcai Su, Rong Ma, Jinyang Liu, Guohao Cai, Xi Xiao, and Rui Zhang. 2022. BARS: Towards Open Benchmarking for Recommender Systems. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 15, 2022, Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (Eds.). ACM,
doi:10. 1145/3477495 3531777773333333
Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, and Xiuqiang He. 2021. Open Bench- marking for Click- Through Rate Prediction. In CIKM '21: The 30th ACM International Conference Information and Knowledge Management, Virtual Event, Queensland, Aus- tralia, November 5, 2021, Gianluca. Demartini, Guido Zuccon, J. Shane Culpepper. Zi Huang, and Hanghang Tong (Eds.). ACM, 2759 2769. doi:10 3459637 3482486
13

---

Published as a conference paper at ICLR 2026
A USAGE OF LLMs DISCLOSURE
In this section, we disclose the usage of LLMs in the preparation of this mamuscript. LLMs were used for 1) polishing writing shortening limited blocks of text and 2) for generating template code for plotting or minor changes of existing code. LLMs were NOT used for retrieval and discovery (e.g., finding related work), rescarch idcation, any other purpose not explicitly outlined in the above.
B MIXED FULL LINEAR ATTENTION
To simplify triton implementation, especially for the gradient computation, our quasi-Lineal Linear Attention drops the normalization (RowNormalizz) in the usual linear attention, similar to lightning allention. Instead we can mimic what SiLU attention does, by introducing 1/N factor.
O= = (QK") O MV/N,
where is the Hadamard product (componentwise multiplication of two matrices, and
1nxn
Onxm
M
To compute this in triton, first break into two parts.
1mx
Im
Q[S]
Q
Q[S] E Raxd Q[T Rmxd
(Q[T))
K[S]
K
K[S] Raxd
K[T] e Rmd
K[T]
V V[S] [[T]
VIS] E aad
V[T] mxd
We will divide + into A blocks of size n', and divide into B blocks of size n", so that Qi are submatrices of dimension x d, and K2, Vj are submatrices of dimension n" x d.
First compute
(QK(S) = Qu Σ KIS) VISI,
j=1
Next compute the target part: divide into C blocks of size m' each. For the j-th block, it's given by
([Q[T)K[T] O Lm)V(T diag( O Im)
We usually merge the source and largel embedding sequences an interleaved fashion.
To avoid HBM/ syne, we probably should keep track of the offsets of the boundary between source and target, and let m', so that for the target part, we will overlap the two computation and obtain
B KISI, VISI,
= Q[S} Σ
j=1
KI V[SI, + diag((Q[T] O = Σ j=1
In terms triton implementation, we will use positive offsets for larget, and negative offsets for source, all starting from the boundary offset.
Note that the sum Σ/= K[S]] K[S]]V[S) ViS can be computed first, then multiplied with Q[S]e Q[ etc. By choosing the block size n' m' sufficiently small, and if necessary, also
14

---

Published as a conference paper at ICLR 2026
break the block ViS,, [[T] along the columns into smaller dimension d'|d, ensure all O[S]e. blocks can be computed entirely in SRAM with a single for loop.
To replace linear attention with (traditional) SiLU attention for target to source, need to replace the second line above with
O Σ SLLUQQII)K(()))VVVS + SiLU(Q[T) & K(T]dlm>x)VVIII
Here we cammot compute all the ( blocks easily, but instead need to have m/m' SM's to compute them separately, otherwise each SM would incur big for loop of Bm/mm iterations. Given H100 has about 132 SMs and batch size per rank is 512, using more SMs will likely slow things down.
B.1 GRADIENT COMPUTATION
0L
ØL KQQ
0V tr
@ @M / N
00
0L
[QQ 0L
tr
@ @MT]
8Q
80)
Given that L, [(O[S), and O[S] and [ are disjoint, we can compute aL
aL
dL = Σ
80 dO[S]i + Σ
00T d0
B.1.1 GRADIENT OF V
If we differentiate against V, we have
dO[s] Q[S K[S] [V[S)
do[T] K[S) dV[s] diag((Q[T) @ 1x)
So,
0L
OL T (0[T] t
dL tr
do|s]
(210p
00
0L
OL (Q[T) dV + diag((Q(T) @ KTT (dV
Ir
Q[S]K[sT
(VISs (Vis) or
00[5]
(0[T]
0L a0
OL QKIS dv[s) +ir (B0TT)
=Ir
T (iag)(([[]] K[TT
So we have that
dL. K[S]QT 0L 00
dv
dL [T diag @ K[T])lr<a)
aL
(
which mcans ith row of aL will be multiplied by ith element of (Q[T) K(T)) 20me
B.1.2 GRADIENT Q
Next we differentiate against Q
do[s] [Q[s] [[S)
d0[T] dQ[T] Vis] diag((dQ[T) o X(T
15

---

Published as a conference paper at ICLR 2026
Which results in
dL = tr to oT [QQ[s]K[s][vss) + tr ( 01. )
+ diag((dQ(T) @ 1xT
So that
dL aL
VisT T K[S].
dQ[s]
a0|s]
To derive dt aQ[T we need to pull dQ(T] out of the unconventional expression diag((dQ[T) o T(T)) within the trace operator. Let's first write il in terms of Einstein summation, Q[T), K[T), V[2] by X, X,Q.K, V respectively.
OL abbreviation T0[T]
tr(x dixgdia o K)]TxI)V)= Σ
Ko Vei.
ipke
where 8 is the Kronecker delta matrix given by
dze {o
jfj=
otherwise.
Note that
ExjKnoojeaa Ex KiVKKKka (diag((X K))
Thus the second half of the expression for dL (with respect to dQ(T)) is given by
ØL 1T (diag((
0T. 80[
[e(KKSS]VV[S
2 ((11))
00!T]
Thus since diagonal matrix is invariant under transposition,
aL aL aQ[T] a0T
aL
V[S] K[S) diag((
80[T] 3 (()
B.1.3 GRADIENT K
Similar computation shows
aL
AL VISI(( TQQ[ +
AL
OL 00
TQ(T)) = V[S](
T
OK[S] 0L
80|s]
80
ØL 00[7"
diag(( 2 ((T
0K[
B.1.4 SUMMARY OF FORWARD PASS AND ALL GRADIENTS
We introduce the notation that produces diagonal matrix dimension T T from two matrices X, Y of dimension 7 x d:
A(X,Y) diag((X @ e))Tki) | XueYsddge
(4)
Then for forward, we have
OsS] Q[S] =: Q[S]
(5)
O Q - U[T]V[]]
(6)
16

---

Published as a conference paper at ICLR 2026
For backward, we have
aL OL
V[S] T K[S] =: d[[s]][s]
(7)
0Q[s]
80[s]
01. ØL
V(S] T K[$] + ( aL
80 = +
(8)
[Q[
80[
OL
= V[S] 0L
TQ ViS
(9)
0K[S]
00
0L = A 80[T] -[]1
ØL
=
(10)
0K[T]
dL K[S]Q" OL
80 K|S]W
(11)
[S[Sp dL
AL = AQQIT, K(T) d0[ U dTTTT
(12)
dV[T]
From equation and equation step to compute, dimension x d.
that Z[S] V[S] K[S] should be an intermediate
From equation 9 and equation we should compute W == QT BD first to obtain a d x d
20
matrix, before pre-multiplying by [[S] or post-multiplying by V[s] and then transpose (or transposing first, then pre-multiplying by V[S)).
that forward pass should save the intermediate step Y[T]= == A [Q[]), K[T]). In fact we can just save the diagonal elements, which consumes
From equation 0 and equation 12
less memory.
From equation and equation 10 we can also save the intermediate step X(T) =
" A ((T)..
A
While X[T), [ only involve 2Td FLOPs, W and Z involve 2Td FLOPs.
B.1.5 WirH SHIFTED ELU ACTIVATION
Shifted elu (and its derivative) are defined by and
if 2 1, E--1_
I,
1, c=-1,
if i
x(x)
('(x) =
if < 1,
<
Given,
055] = (K()))
O ((() ( V[S] A(e(Q(T),
Gradients are given by (partly by guessing via dimension match)
0L aL
ViS] +( o (Q(())
aQis
80
0L ØL
V[S] o( o ('((T) + A aL.
80 (QTT o (KK)T)
[Q[
00 = (V[s1( aL,
T ((Q)) @ (K((S)
OK[S]
a0
O0
0L
[ A "oOT] V(TT)(((Q[TI) o ('(
SS ((K AL
dL
d0
dV = A(C(Q[T)) aL
7P
00
17

---

Published as a conference paper at ICLR 2026
B.2 ACTIVATION FOR QUASI-LINEAR ATTENTION
The choice activation function yp in Section 3.2.2 is non-lincar but otherwise arbitrary and depends on the specific application at hand. The same activation also need not always be used, for instance one can have two different act activations yp and p2 and apply them as:
Ois] şi (115))))((((((((15)) о 9i + ((1T)))))(((((()((( V[S]) + (((p(((1))))
C MORE ON RECONSTRUCTION Loss
The reconstruction loss is measure of how much information of the full UTH is captured by VISTA's virtual embeddings. We used L,2 norm to measure how much we can reconstruct the original UIH given the virtual embedding input, and we have verified that it is an informative metric for us to quantitatively measure the reconstruction quality.
Training Reconstruction Loss Over Common Steps
Reconstruction Loss
1.8
1.5
1.4
1.2
1.0
0.3
5000
10000
15000
2030C
25030
Steps
11 Vista_ amazon recon Training Reconstruction
vista amazon reson Reconstn Less
11 Vista aata__amazon__am amazon_norecoo nooreonn Training Reconstruction Loss
11 aesta_amazon_iae norecon Eval Reconstruction Loss
Figure Comparing the reconstruction loss over training steps with and without explicitly including it in the total loss on the Amazon datasct. The dotted line is the final evaluation reconstruction loss.
On the Amazon dataset, for example, explicitly minimizing the reconstruction loss still reduces the reconstruction loss of the Iearned embedding against the full UIH as the model improves. However, the reconstruction loss plateaus and the model takes longer to converge after some training steps (as our training pipeline supports early stopping). With the explicit introduction of the reconstruction loss, we see a dramatic decrease in the reconst loss in the first. training steps and faster model convergence. The test metries also improved by 0.22% AUC and 1. 1.11% NE with the use of the reconstruction loss.
see that training the VISTA model without
D ADDITIONAL EXPERIMENT DETAILS
D.1 DATASETS
We include more details about the datasets used in our experiments. The statistics of the sequence features of each dataset are summarized in Table 5
The Amazon Products and Reviews dataset Hou et al. (2024) contains user reviews, item metadata, and user-item interactions. A subset of this data preprocessed to make the Amazon-Electronin dataset, which is restricted to electronies items, initiated by Zhou et al. (2018). The data format is relatively simple, with the columns: label, user id, item id, category id, item history, and category history.
Amazon-Electronice
18

---

Published as a conference paper at ICLR 2026
Table Dataset Statistics
Datasct
Mean Seq.
Max Seq.
Amazon- Electronics
8.93
429
KusiRand-11
225.20
256
Simplified Prod
1528. 18
2,000
Industrial-Seaa Data
7,000
16,000
The KuaiRand dataset by Gao et. al. [2022] is a sequential recommendation dataset collected from the recommendation logs of the video-sharing mobile app Kuaishou. The KuaiRand- 1K subset contains random sample 1,000 users after removing irrelevant videos. There are 4 million videos remaining in this subset. Our experiments use subset of all features available in KuaiRand- IK, namely user id, video id, video id history, click history, like history, and lvv (long vidco view) history.
KuaiRand-1K.
Simplified Production and Industrial-Scal. Data. The full production data is too large to be able to run simple experiments quickly (and requires re-implementing baseline models on internal systems). We construct a minimal version of our production data to focus on the sequential recommendation task (e.g., keeping the user interaction history largely intact but removing other features). After preprocessing, this datasct has mean sequence length of around 1528 and maximum truncated to 2,000.
D.2 FUXICTR FRAMEWORK
We utilize the FuxiCTR library developed by Zhu et al. (2022 2021 for our traditional
sequential setting experiments. As mentioned in the main texl, we designed the traditional sequential ial setting experiments mainly lo compare the effectiveness of the attention layers and keep constant other model architecture and hyperparameter choices. (See Figure 12
We also report the common hyperparameters used in all the experiment results in Table
hout Data
Hyperparaneler
Amazou
KuaiRand
Simplifeed Prod
Learning Rate
5.0e
Lde-1
1.06-3
Comman Festure Embedding Layer
Optiniser
Adam
Adam
Adam
Batch Size
1024
1024
128
Batch Norm
Yes
Early Stop Patieuce
5
1
Emberding Regulariser
None
Nomo
Atlention Layers (Basclines, VISTA)
Embedding Dimension
64
32
32
Emberding Initialiser
MLP Hidden Units
[1024, 512, 256] RELU
[512, 128, 64] RELU
[512, 128, 61] RELU
MLP Activations
Attention Heads
Attention Layers
2
Cutput: CTR Pradiction (Binary CE Loss)
Figure 13: Common hyperparameters used for traditional setting experiments.
Figure 12: FuxiCTR Setup.
The model-specific parameters for VISTA are the number of seeds and weight for the reconstruction loss, which were set at 128 and 1.0, respectively, for all experiments. No specific hyperparameter tuning was done, mainly relying using common parameters for all models and repeating across seeds for each model and dataset.
D.3 MORE ON VISTA'S TWO-STAGE ATTENTION
Case Study 1: Same User, Different Candidates. In the following Figure 14] we show the input to the virtual attention layer, the output of the virtual attention layer, and then
19

---

Published as a conference paper at ICLR 2026
Visualization VISTA's Two-Stage Attention
Same User, Different Candidates
Input
Attention Output
Virtual Embeddings
Virtual Attention Layer
Virtual Target
Mario)
12 Fresstd Pun
Tagget Frezzze (Ta Portio
Label:
0
Item 60617
texexo Prsl
Sepamc Pootus
Mom acctvaaiion
Mean Along
Fressed Pur
Faooted Merged seenn
Feature Dim
Seods
Enbelding
Vertual
Target
Frestes Alge
Blass
Label:
User 79122 Item
Secuenee on
Sepance Footie
Mean Along Feature Dim
UNM
Embelding
Virtull
Figure 14: Visualizing the virtual attention and target attention layers for the same user on two different candidates (one positive at the top and one negative at the bottom).
the output of the larget atlention layer for the same user two different candidates (one positive at the top and negative at the bottom) from the crazzon-leectonics dataset. We also reduce these along the feature dimension for compact visualization. Note that since
looking at two candidates for the layer and the output of the virtual attention layer are identical; these only depend on the user's individual UIH and the virtual seed embeddings which are common between the two. The difference in this case comes at the target attention part. Here we see that the target attention differentiates between positive and negative candidates for this user as evidenced by the differing mean activation for the target embedding.
user, the input to the virtual attention
Case Study Different UIH Lengths. We also look a case study comparing two different users with different UIH histories, one with very short history (length 4) and another with a slightly longer history (length 12) in Figure Even with very little historical data (UIH length 4 at the top), the virtual seed embeddings appear to help influence the virtual embeddings, which in turn help with model performance.
20

---

Published as a conference paper at ICLR 2026
Visualization VISTA's Two-Stage Attention
Different Users with Different UIH Length
Input
Attention Output
Virtual Embeddings
Virtual Attention Layer
Virtual Target
64
Item 53744
Mean Along
Maagaed (Meen Wettor
2 Ve trascer Mergnn pested Part Chal
Tereet Excpted Mecard otta Vedter
Activation
(Virusl incheidtings
Virtuail
Target
Herpd Morrio
10.trus anoodea Aligh vensal tan Orly
oryet Bropse Slaye Maral
Label:
User 75678
-63
Secuinee Tedlie
Segance faidion
Mean Along
Figure 15: Visualizing the virtual atlention and target attention layers for the different users having different UIH sequence lengths (both positive candidates).
21


Reviews:
[{"review_id": "wwbrlwjrR6", "reviewer": "Reviewer_osbe", "summary": "This paper introduces VIrtual Sequential Target Attention (VISTA), a two-stage approach that solves the critical computational bottleneck in modern sequential recommendation systems. It strikes the balance between computational efficiency and predicitve accuracy, as well as the latency and scalability challenges. The core contributions are the ultra-long UIH summarization and the quasi-linear attention for recommendation, which are well motivated for recommendation scenerios. It has comprehensive experiments with industrial-scale dataset results and online A/B experimental results.", "questions": "Although the proposed quasi linear attention is designed for recommendation, there are some other linear/efficient attention architectures for sequence modeling and langauge modeling. Are those existing linear/efficient attention architectures (e.g., Deepseek's FlashMLA) with hardware optimization also effective and efficient for sequential recommendation? If so, they could potentially be used to build recommendation systems for further gains.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well motivated. The scalabity and efficiency are important for industrial recommendation systems. The core two-stage architecture (UIH summarization and target attention) effectively decouptions the quadratic computation of attention from real-time inference.\n2. The proposed quasi-linear attention (QLA) with linear complexity is tailored for recommendation.\n3. The results are backed by comprehensive offline and online A/B tests on a massive industrial-scale dataset, showing significant real-world improvements.", "weaknesses": "On public datasets like Amazon and KuaiRand, the performance gains are marginal compared to baselines. This suggests the primary advantage of VISTA is strictly in the extreme-scale, ultra-long sequence regime of proprietary industrial data, limiting generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zqEl9hyD6H", "reviewer": "Reviewer_khLg", "summary": "This paper proposed VISTA (VIrtual Sequential Target Attention)， addressing the scalability challenge of processing ultra-long user interaction histories (up to 1M items) in industrial recommendation systems, where traditional Transformer-based models (e.g., HSTU, SASRec) suffer from prohibitive latency and GPU costs. It proposes a two-stage framework: (1) UIH Summarization compresses ultra-long UIH into hundreds of virtual seed embeddings; (2) Target-Aware Attentioncomputes attention between candidate items and cached summary embeddings (bypassing full UIH processing). \nThe summary embeddings are stored in a distributed key-value system, enabling fixed inference costs regardless of UIH length. Experiments on public (Amazon-Electronics, KuaiRand-1K) and industrial datasets show VISTA outperforms baselines (HSTU, DIN, SASRec)", "questions": "What is the information loss of summarization? The paper claims the reconstruction loss retains UIH information, but no quantitative analysis (e.g., mutual information between summary embeddings and full UIH, or other way) is provided", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Interesting design. VISTA decouples UIH processing into offline summarization and online attention. This design enables handling UIH up to 1M items while keeping inference costs fixed—critical for industrial systems serving billions of users with lifelong interaction histories.\n\n- QLA resolves linear attention’s limited expressiveness by integrating SiLU non-linearity and self-target attention. \n\n- Practical Industrial Deployment Design: VISTA includes a distributed embedding delivery system that handles petabytes of summary embeddings. Online A/B tests on a production platform confirm tangible gains with minimal latency, validating its real-world applicability.", "weaknesses": "- Reliance on Seed Embedding Quality: The summarization stage’s performance hinges on virtual seed embeddings—experiments show increasing seeds from 64 to 128 improves NE by 0.04–0.12% but raises storage costs exponentially\n\n- Limited Analysis of Reconstruction Loss: The generative reconstruction loss is claimed to enhance information retention, but its contribution is weakly validated. Ablation shows the loss has marginal value.\n\n- Brittleness in Short UIH Scenarios: On public datasets with short UIH (e.g., Amazon-Electronics), VISTA’s performance is not superior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EuLy2vno3K", "reviewer": "Reviewer_Jfea", "summary": "The paper presents a two-stage framework for large-scale recommendation systems aimed at modeling extremely long user interaction histories while maintaining computational efficiency. The method separates historical summarization from candidate evaluation: user histories are first compressed offline into a fixed number of embeddings, which are then reused online through candidate-specific attention. The authors also introduce a quasi-linear attention variant and a reconstruction-based training loss. Experiments include both public and industrial datasets, with partial deployment results reported.", "questions": "Q1: For your results compared to baselines, like the marginal AUC gains in Table 1 are only 0.002, very limited, have you conducted statistical significance tests? Could you provide p-values, or variance, or just let me know the reliability of the reported improvements is not due to the random seed?\n\nQ2: About the online A/B tests, could you all talk about computational efficiency performance compared to baseline models? And is the comparison the same in the offline testing?\n\nQ3: Regarding summarization embedding updates, the paper provides vague descriptions of this process, which is critical for handling dynamic user behavior. Is the update triggered by a fixed schedule (e.g., daily) or by the growth of the interaction sequence? Do you adopt full recomputation or incremental learning for updates?\n\nQ4: In comparison to DGIN [1]: Are VISTA and DGIN comparable given their shared focus on lifelong behavior modeling? Could you compare them in terms of methodology, performance, or computational efficiency? (Or at least compare the part of lifelong or a long period of time of behavior sequences modeling.)\n\n[1] Liu, Q., Hou, X., Jin, H., Wang, Z., Lian, D., Qu, T., ... & Lei, J. (2023). Deep Group Interest Modeling of Full Lifelong User Behaviors for CTR Prediction. arXiv preprint arXiv:2311.10764.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles a practically significant challenge in large-scale recommender systems: how to model ultra-long user behavior sequences without overwhelming computational resources efficiently. The idea of decoupling offline summarization and online attention is pragmatic and fits well within production serving architectures. The framework appears to strike a reasonable balance between modeling capacity and latency, and its design aligns with industrial constraints. The experiments cover multiple datasets and include an online A/B test, providing some evidence of real-world feasibility. The overall presentation is clear, with logical flow and illustrative figures that help readers follow the motivation and method.", "weaknesses": "W1: The paper presents various performance comparisons, but fails to provide any statistical significance tests for these results, unclear if the differences are due to random seeds; supplementary tests (e.g., a marginal +0.002 AUC gain in Table 1) and related metrics are needed to validate the gains.\n\nW2: While the paper reports metrics like CTR improvements from online A/B tests, the authors did not mention how computational efficacy is in this scenario, which is essential for assessing the real-world applicability of the framework.\n\nW3: Given dynamic user log updates in a real-time scenario, it is unclear whether embedding updates are triggered periodically or by a long enough sequence length. Additionally, there is no clarification on whether updates in on the scale of all parameters and weights or incrementally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QRFgqu2k9D", "reviewer": "Reviewer_Difm", "summary": "The paper introduces VISTA, a two-stage framework for scaling sequential recommendation to lifelong user histories with fixed training and inference costs. Stage 1 compresses ultra-long histories into ~100 cached summary embeddings via quasi-linear attention with virtual seed tokens. Stage 2 lets candidate items attend only to these summaries, avoiding full-history access.\nDeployed at billion-user scale, VISTA delivers +2.1% CTR and +1.8% session duration gains with O(1) inference cost and manageable (O(100 TB–1 PB)) storage.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Summarizes histories once, caches ~100 embeddings, achieving constant-time inference with practical TB–PB storage trade-offs.\n2. QLA ensures no label leakage, cleanly separates history and target attention, and yields interpretable user clusters.\n3. Uses reconstruction loss for rich summaries and integrates quantization, KV caching, and async updates for production readiness.", "weaknesses": "1. Heavy reliance on summary quality and update frequency The effectiveness of VISTA critically depends on the fidelity of Stage 1 summarization embeddings, yet the paper does not specify how frequently these summaries are refreshed in production.\n2. Shared virtual seeds limit personalization; smaller summary budgets unexplored The virtual seed tokens used to initialize summarization are shared across all users, which may fail to capture highly idiosyncratic or niche preferences. While PCA visualization shows country-level clustering, it does not prove fine-grained personalization.\n3. All candidates attend the same fixed ~100 summary tokens — no dynamic selection or weighting Every candidate item attends to the identical set of precomputed summary embeddings, with no mechanism to dynamically select or reweight subsets based on candidate type or context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)