Efficient Inference for Large Language Model-based Generative Recommendation
Authors: Xinyu Lin
Chaoqun Yang
Wenjie Wang
Yongqi Li
Cunxiao Du
Fuli Feng
See-Kiong Ng
Tat-Seng Chua
Venue: ICLR 2025 Poster (2025)
Keywords: LLM-based Generative Recommendation
Speculative Decoding
Decoding Acceleration
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Large Language Model (LLM)-based generative recommendation has achieved notable success, yet its practical deployment is costly particularly due to excessive inference latency caused by autoregressive decoding. For lossless LLM decoding acceleration, Speculative Decoding (SD) has emerged as a promising solution. However, applying SD to generative recommendation presents unique challenges due to the requirement of generating top-K items (i.e., K distinct token sequences) as a recommendation list by beam search. This leads to more stringent verification in SD, where all the top-K sequences from the target LLM must be successfully drafted by the draft model at each decoding step. To alleviate this, we consider 1) boosting top-K sequence alignment between the draft model and the target LLM, and 2) relaxing the verification strategy to reduce trivial LLM calls. To this end, we propose an alignment framework named AtSpeed, which presents the AtSpeed-S optimization objective for top-K alignment under the strict top-K verification. Moreover, we introduce a relaxed sampling verification strategy that allows high-probability non-top-K drafted sequences to be accepted, significantly reducing LLM calls. Correspondingly, we propose AtSpeed-R for top-K alignment under this relaxed sampling verification. Empirical results on two real-world datasets demonstrate that AtSpeed significantly accelerates LLM-based generative recommendation, e.g., near 2x speedup under strict top-K verification and up to 2.5x speedup under relaxed sampling verification. The codes and datasets are available at~\url{https://github.com/Linxyhaha/AtSpeed}.

Published as a conference paper at ICLR 2025
EFFICIENT INFERENCE FOR LARGE LANGUAGE MODEL- BASED GENERATIVE RECOMMENDATION
Xinyu Lin Chaoqun Yang2 Wenjie Wang Yongqi Li
Cunxiao Du
Fuli Feng See-Kiong Ng' Tat-Seng Chua
NNational University of Singapore TTsinghua University
SUniversity of Science and Technology of China
"The Hong Kong Polytechnic University
xylin1022 1p22@m002@@maaiii 1. com, chaoqun@yang. ema cn,
(wenjiewang 1 yoygggi 0, cnsdunm, ful i feng93) @gma 1l com, seekiongênus. edu. sg, dcsct edu..ssedd.sp..
ABSTRACT
Large Language Model (LLM)-based generative recommendation has achieved notable success, yet its practical deployment is costly particularly due to excessive inference latency caused by autoregressive decoding. For lossless LLM decoding acceleration, Speculative Decoding (SD) has emerged However, applying SD) to generative recommendation presents unique challenges due the requirement of generating top-K items (i.e., K distinct token sequences) as a recommendation list by beam search. This leads to more stringent verification in SD, where all the top-K sequences from the target LLM must be successfully drafted by the draft model at each decoding step. To alleviate this, we consider 1) boosting top-K sequence alignment between the draft model and the target LLM. and 2) relaxing the verification strategy to reduce trivial LLM calls. To this end, we propose an alignment framework named AtSpeed, which presents the AtSpeed-S optimization objective for top- K alignment under the strict top- K verification. Moreover, we introduce a relaxed sampling verification strategy that allows bigh-probabilltyy non-top-K drafted sequences to be accepted, significantly reducing LLM calls. Correspondingly, we propose AtSpeed-R R for top-K alignment real-world
promising solution.
under this relaxed sampling verification. Empirical results datasets demonstrate that AtSpeed significantly accelerates LLM- generative recommendation, e.g., near 2x speedup under strict top-K verification and up 2.5x speedup under relaxed sampling verification. The codes and datasets are available https: //gpthub.bb//t// cow http//inxyyaaahaaaeae Speeed
INTRODUCTION
Large Language Model (LLM))based generative recommendation has achieved remarkable perfor- mance, emerging promising avenue and attracting widespread attention (Bao al., 2023; Rajput et al., al,, 20236). Technically speaking, L.MSs encode the user's historical interactions, and then perform multiple LLM calls (i.e., forward processes of LLMs) autoregressively decode top-K ranked items as recommendations (Zheng et al., 2024). Despite the effectiveness, the inference of LLM-based recommender models is unaffordably mone-consuning, hindering real-world deployments (Cui et al., 2024). Such intolerable time consumption primarily arises from the decoding process as shown in Figure 1(a), where multiple serial LLM calls are required for step-by-stey autoregressive genera- tion (Leviathan et al., 2023; Cai et al., 2024). In light of this, it is essential to achieve lossless LLM decoding acceleration for 1.M--baed generative recommendation.
To accelerate the LLM decoding losslessly. Speculative Decoding (SD) (Leviathan et al., 2023; Xia et al., 2023) has been proposed as a promising approach in Natural Language Processing (NLP). SD utilizes a draft model (e.g., a compatible small-sized language model) to reduce the number of target
"qqual contributions.
Corresponding authors.

---

Published as a conference paper at ICLR 2025
Prefilling (9%)
Draft Model
Draft Modiel
bi
Step1
Step2
Step3
Varily
Verity
Veeriyy
Warty
Quaiity
bj
Target LLM
Decoding (91%)
S Step1
Step2 o'
Step3x
Target Model
Step Lx
Stop2
Step3
(a) Inference Time Cost
(b) N-to-1 Verification of Traditional SD (1-3)
(c)N-to-K Verification of with Search (N-K-B)
Figure 1: (a) The inference time costs of IC-Reec (Zheng et al., 2024) with LLaMA-7B on a single A5000 GPU. (b) The illustration of the N-to-1 verification of SD with greedy decoding in NLP tasks.
(c) N-to-K verification of SD with beam search in recommendation tasks, where the drafting length, candidate number N, and beam size K are set at for illustration.
LLM calls in the decoding process (Miao et al., 2023). Technically, SD follows a draft- traft-then-verify paradigm (Xia et al., 2024), which first efficiently drafts multiple subsequent tokens by draft model, and then verifies the drafted tokens in parallel by the target L.M in a single call. As depicted in Figure (b), tokens are either accepted rejected, with the accepted tokens up the first rejection step being utilized for subsequent decoding. As such, the number of L.M calls can be reduced by leveraging the accepted tokens from the draft model, thereby improving the decoding efficiency.
verify
However, is non-trivial to apply SD for LLM-based generative recommendation due to the challenge of more stringent N-to-K verification. Specifically, traditional SD for NLP tasks typically follows an N-to-1 verification to generate only one response, which requires accepting a single token out of N drafted tokens at each step (Figure 1(b)). In contrast, recommendation tasks necessitate generating top- K items (i.e., K distinct token sequences) through beam search, resulting in an N-to-K verification problem (Figure I(c)). For each verification step, one LLM call can be skipped if and only if all the top-K sequences SD fails to reduce target LLM calls in Figure ((c) since dg is not drafted in the first step. As such, the N-to-K K verification poses greater challenges than N-to-1 verification each step requires drafting all the top-K sequences.
successfully drafted from the N candidates. To elaborate,
To achieve effective SD for LLM-based generative recommendation, we formulate the SD task under the N-to-K verification. To reduce the target LLM calls under N-to-K verification, we consider two objectives in the drafting and verification steps of SID: 1) top- alignment, which aims to align the drafted sequences with the top- K sequences generated by the target LLM, thereby maximizing the recall of all top-K sequences; and 2) verification relaxation, which seeks ease the strict matching with the top- sequences from the target LLM, enhancing the acceptance rate of drafted sequences while maintaining the accuracy of top-K recommendations.
To this end, we propose an Alignment framework for Speculative decoding (AtSpeed) tailored for LLM-based generative recommendation. First, under the strict top-K verification, we propose an optimization objective named AtSpeed-S to train the draft model. AtSpeed-S improves the top-K alignment theoretically by minimizing the Reverse Kullback-Leithe 2015) and probability density regularization term (see Section 3.1). Moreover, for verification relaxation, we introduce relaxed sampling verification strategy that allows the non-top-K drafted sequences with high generation probabilities to be accepted. To maintain recommendation accuracy, this verification strategy ensures that the generation distribution of SD is approximately equivalent to that of the target LLM with sampling-bes based beam search (see Section 3.2). Under the relaxed sampling verification, we design AtSpeed-R for top- K alignment, which minimizes the Total Variance Distance (TVD) the generation probabilities of top-K sequences (see Section 3.2).
Divergence (RKLD) (Huszár,
We conduct extensive experiments using both verification strategies on two real-world recommen- dation datasets, demonstrating that AtSpeed significantly accelerates the decoding for LLM-based recommendation (around 2x speedup). Besides, the results confirm that the relaxed sampling verifi- cation strategy substantially improves decoding efficiency without sacrificing much recommendation accuracy. The contributions of this work are summarized as follows:
We are the first to propose the speculative decoding task for LLM- based recommender acceleration, highlighting the significant challenge of shifting from N-to-1 verification to N-to-K verification.
2

---

Published as a conference paper at ICLR 2025
Algorithm 1 SD step with Top-K Strict Verification
Input: Draft model Ma. target LLM Mp. prefix, target beam size K, draft beam size N I: 2 < prefix, y 4 f 0. Vou +- 0
2: for = 1 7
do
9, - M, 1)) }/ ()))(g;)
b Drafting sequences for every beam search step
4: pl, ...P......
+ M) M, MMp.)) 5: for j = 1 do
b Run Mp in parallel
6:
y (pp)( Von 37
7: ir E yリ then
8:
continue
Accept if idcal top- sequenees are fully drafted
else
10:
break
Reject
Output:
Vou
We propose novel aligmment framework named AtSpeed for speculative decoding under N-to- K verification, with a relaxed sampling verification strategy for verification relaxation and two alignment objectives for superior top-K alignment of draft models.
We conduct extensive experiments on datasets, which 1) demonstrate the verification efficiency and accuracy of the relaxed sampling verification strategy; and 2) validate that AtSpeed achieves almost 2 speedup for LLM-based recommender decoding.
TASK FORMULATION
LLM-based Generative Recommendation. In LLM-based generative recommendation, each item is represented by an item identifier, i.e., token sequence such as item title (Bao et al., 2023) or learnable token sequence (Zheng et al., 2024), linking the recommendation items to the language space for LLMs understand user behaviors and recommend items (Lin et al., 2024a). Formally, given the user's historical interactions æ, the well-trained target LLM--ased recommender model Md generates top-K ranked items via beam search, i.e., {yr., K + (a) where yr., (y1, .2...., Mr.); the item identifier of the i-th recommended item of length L'. However, the LLM inference is time-consuming due to the need to perform multiple LLM calls (i.e., forward process) during autoregressive generation. To accelerate the LLM inference, we are motivated to leverage SD for LLM-based generative recommendation for its decoding acceleration without losing accuracy.
SD for LLM-based Recommendation. The challenge of applying SD to LLM-based generative recommendation with beam search lies in the shift from N-to-1 to the harder N-to-K verification. In the following, we detail the draft-then tiet-thhh------yy paradigm under the strict top-K verification and formulate the task of SD for LLM-based generative recommendation under N N-to-K verification.
Drafting. Given the user's historical interactions and the generated sequences in the previous SD step 3/- = { 1= compatible small-sized draft model M, is used to generate the drafted beam sequences for 7 steps via beam search with the beam size of N:
1t+1, Qt+2; ... At+ BeamSearch(z; Ma,, 2 V 02
(1)
V2t, Top.V(ge+1), Top.N(()+2). Top.N
where lt+j with j E (1. the sequence probability distribution at step j obtained by the beam search of draft model; and Yt+, collects the top-N drafted sequences with highest probabilities. The drafted sequences are then fed into the target LLM MM to obtain the target sequence probability P++1, Pt+2..... ... P1+y+1, where Pt+j Mp with j E {o..... Target LLM Mp then select the sequences with top-K probabilities ideal sequences, i.e., Vi+, + TopK (Pt+)). For simplicity, we omit the subscript and use f,. Pý- V/. and ) as shorthand for 4t+j- Pttj. y and y respectively, in contexts where the exact value of f is not essential.
'e follow the widely used codebook based item identifier, due to its promising results and generalization ability on cold-start items (Wang et al., 2024a; Rajput et al., 2023a). The codebook- based identifier will ensure each item has identifier of the same length.
3

---

Published as a conference paper at ICLR 2025
Sen.
((x,y2)
Sample
Exer
9:
Pred
Target LLM
Draft Model
Vorification
Draft Model Beam Search
Stept
y px))
bg
Aigge-
profi
1/2
Stepl
D' (0oyy))
Steph
(a) AtSpeed Training
(b) AtSpoed Inference
(c) Tree-based Attention
Figure 2: Overview of AtSpeed. (a) shows the alignment training of the draft model with an additional
alignment loss tailored for different verification strategies, e.g., CAlign-R- (b) depicts the AtSpeed inference, where the well-trained draft model produces beam search sequences from each step, i.e., Dr for the target LLM to verify. The beam sequences from the last accepted step before encountering the first rejection are utilized in the following SD step, i.e., 3/2 (c) demonstrates the tree-based attention for the drafted beam sequences (N=2 and y=3).
Strict Top-K Verification. For every beam search step j. we have N drafted sequences 3/22 and K' ideal sequences V,, We define the step acceptance as successfully drafting all the top- ideal sequences for that step. Formally, the beam search step is accepted if y yg. Otherwise, reject the step, and discard the whole drafted sequences V/, and correct it with y/p- From beam search step 1 to 7, we sequentially verify Y/ until the first rejection step occurs. The corrected sequences }p at the first rejection step will be the output of the current SD step. If every step is successfully accepted, SD step. The target LLM calls for all previously accepted steps could be reduced as they are successfully skipped through single LLM call via parallel verification. The process of strict top-K verification is presented in Algorithm 1.
select top-K sequences from py+1 the output of the current
It is highlighted that SD for beam search under N-to-K verification is more difficult than N-to-1 verification because an LLM call can be skipped if and only if the top-K sequences are entirely drafted. To improve the acceptance rate of the drafted sequences to reduce the LLM calls, have two key considerations: 1) top-K alignment, which encourages the draft model to generate strongly aligned top-K sequences; and 2) verification relaxation, which seeks relax the strict matching for greater acceleration without much accuracy sacrifice.
ATSPEED
To pursue the two objectives, we propose AtSpeed, an alignment framework for SD under N-to-K verification. AtSpeed designs two effective alignment objectives for the draft model to get higher acceptance rate under the strict top-K verification (Section 3.1) and the proposed novel relaxed verification (Section 3.2). The overview of AtSpeed is presented in Figure 2.
3.1 ALIGNMENT FOR STRICT TOP-K VERIFICATION (ATSPEED-S)
Under strict top-K verification, the draft model is expected to generate top-K sequences that strictly align with the top-ranked sequences from the target LLM. To achieve this, we design an optimization objective named AtSpeed-S that directly optimizes the acceptance rate for the strict top- K verification.
Acceptance Rate. We define the acceptance rate B as the probability of the step acceptance, i.e., accepting all the top-K ideal sequences from the draft sequences yy. With strict top-K verification strategy, for each step, we have 3 if 3V C V, such that p(y) > p(уK) for Vy E ) where DV = K, and yK is the sequence that has the K-th highest probability in p.
A Alignment Objective. Since the acceptance rate directly affects the acceleration performance, we aim to optimize the acceptance rate of the draft model to achieve superior top- K alignment with the target LLM. To elaborate, we consider the following alignment objective for the draft model M,i
p(y) arg max Eng-Yan p(yK) dee
(oy))ogg ge(y)
qo(y) log 46(y) p(yk)
0- = arg max
Σ ME
p(y)
Σ
(2)
DEV
We denote y/5 and 2 with V, and Ud, respectively, whenever step j is clear.
4

---

Published as a conference paper at ICLR 2025
where Do = u ................... ...... ... aims at maximizing acceptance rate for every beam search step (see detailed derivatives in Appendix A.2.2). This alignment objective can be further expanded as:
Lo czt)
qo (v(lec))) qo (0(veleee) p(yx)
== arg min VI 1E0
Σ
Σ
Σ BIEV
qo (yelle))) log p(uslee))
Σ
]]:
y
NeV
(3)
where Gt (x, yce) is the context, p(yk) plyK, st), and D' [(, Top ((1 ())) + xM,)) D)3, and V is the LLM vocabulary.
However, aligning every token over the entire vocabulary with y: E V might lead to suboptimal results due the noises introduced by invalid tokens with high probabilities. L.MM-based recommender models usually employ constrained generation (Hua et al., 2023) to generate valid item identifiers only, Blindly aligning across all tokens in V may cause unnecessary alignment to these -roo-brbbbbllity but invalid tokens that will never be generated by the target LLM. To mitigate this issue, we define Ve = To (q) block out the potential alignment noises from invalid tokens. We then define the alignment loss as:
Sllign
Σ (m.V)~D' yEy
go (cee) log
go (valCe)) p(ah (cee)
qo gu (ce)) log go
D
p(yk)
(RKLD)
(Density Regularization)
(4)
which essentially minimizing the RKLD over the top-K sequence probabilities and a density regularization term. Intuitivell, minimizing RKLD emphasizes aligning the draft model M,g to the target I.LM Mp within the draft model's generation capability (Gu et al., 2024), particularly for the top-K sequence probability distribution. Moreover, the density regularization term encourages the top-K sequence probabilities of the draft model to dominate the whole probability distribution.
Overall Loss. Based on Eq.(4), AtSpeed-S defines the overall training loss for the draft model as: CAiSpodd-S = ACCllign-S + (1 - CRoe
(5)
where Lree is the original loss to train the model for the recommendation tasks (see Appendix 1.3.1) and a is the hyper-parameter to control the alignment strength.
3.2 ALIGNMENT FOR RELAXED SAMPLING VERIFICATION (ATSPEED-B)
In addition the strengthened alignment for the strict top-K verification, we further consider verification relaxation to reduce the trivial LLM calls. From the angle of verification, we introduce a relaxed sampling verification strategy based on the sequence generation probability to improve the acceptance rate of draft sequences. This verification strategy ensures that sequences falling outside the target's 's top- predictions can still be accepted with a certain probability, while preserving the output distribution closely aligned with that of the target LLM to maintain the recommendation accuracy to some extent. Under this relaxed sampling verification strategy, we propose AtSpeed-R for top-K K alignment accordingly.
Relaxed Sampling Verification. We design the verification strategy for each beam search step j E (1,..., } as follows. Given the drafted results V/, ~ ; for each E Vg., we accept if p(y) > q(y). Intuitively, if the target LLM is even more confident in generating a high-probability sequence, the candidate sequence is likely to be generated by the LLMs and should be accepted. Otherwise,
we reject y with probability of 1 p(y) and resample y ~ p' = norm(max(0.. p(y) q()))).
a(w)
Notably, for the non-top-K drafted sequence with high draft probability (i.e., g(y) 2 p(y)), this verification strategy ensures they have certain probability of p(y) to be accepted. We present more details on the relaxed sampling verification strategy in Algorithm 2 in Appendix. Drawing upon the
a(3)
with-replacement sampling approximation' the joint distribution of K output sequences under such
We follow (Gu et al., 2024) to adopt the target LLM-mixed sampling for data construction to improve training efficiency and generated data quality.
"The drafted sequences are sampled from the sequence distribution 75, and superseript j is omitted for clarity.
'When the sample size (i.e.. the number of recommended items K) is much smaller than the total population size (i.e, all sequences for sampling). the sampling without replacement can be effectively approximated to the sampling with replacement. See Appendix A.2.1 for proof.
5

---

Published as a conference paper at ICLR 2025
verification is approximately equivalent to that of the target LLM as proven in Appendix A.2.1. Such equivalence ensures that the relaxed sampling verification does not sacrifice much recommendation accuracy (see Section 4.2 for empirical evidence).
Acceptance Rate. For the relaxed sampling verification strategy, we obtain the probability of step acceptance II under the with-replacement sampling approximation, where
1,
a(y) S p(y)
Eyya(t) min(1, p(y) - TVD(p(y) ((V)).
b E(y)
(6)
((v) > p(y)
((y)
9030
The derivation of (6) be easily extended from (Leviathan et al., 2023) in Appendix A.2.3.
Alignment Objective. Under the relaxed sampling verification, we consider maximizing log B = Σx k log TVD(4, p), which is equivalent minimizing Σx TVD(g, p) improve the acceptance rate. Thereafter, we present the alignment objective for the draft model:
lyl
0 = arg min (x..))-D.I K BEB
TVD(p' wer), (122,,,))))
(7)
Similar to AtSpeed-S, we perform constrained alignment over the top- valid tokens, and therefore p' (- y and p' (1 M)t) is the normalized truncated sequence probability over top- valid tokens. Based on the with-replacement sampling approximation, Eq.(7) minimizes TVD with the strength of K for the same sequence. Nevertheless, considering the beam search with sampling would obtain different sequences, we alternatively leverage the top-K sequences from the target L.IM, i.., D { y ~ (Mp)(M(()))))aa x D) to minimize the TVD. Finally, we define the alignment loss of AtSpeed-R as,
y
Aliign
TVD(p y (')x,
(8)
[D'
(m.D)~~
Overall Loss. The overall loss of AtSpeed-R R is defined as:
LAISpeed-R = Aliga + (1 a) CRee
(9)
where a controls the strength of the alignment loss.
3.3 INFERENCE OF ATSPEED
The inference of AtSpeed follows the draft-then-ver paradigm. To efficiently recommend items during inference, AtSpeed first leverages well-aligned draft model for drafting and then chooses specific strategy for verification. Specifically, 1) to obtain identical recommendation results, AtSpeed utilize the draft model trained by AtSpeed-S and adopt the strict top-K verification strategy for SD. Alternatively, 2) to further improve the inference speedup, AtSpeed can use the draft model trained by AtSpeed-R along with the corresponding relaxed sampling verification strategy. Note that any draft model can be interchangeably applied to the strict top- K and relaxed sampling verification strategies, although they may not be optimized for enhancing the acceptance rate under the corresponding verification strategy.
Tased Attention. challenge of AtSpeed inference the verification inefficiency issue. Using beam search results from every step would largely increase the number of drafted sequences (>N sequences), where some prefix shared by different beam sequences. As such, self-attention repeatedly calculated for the same prefix through single LLM call, thus Icading to verification inefficiency, especially with a larger N (e.g., N 20). In this work, we leverage the tree-based attention (Miao et al., 2024) eliminate the repeated calculations via single flattened sequence, thereby boosting the efficiency of verification. The tree- rree-based attention for the drafted sequences from all steps is illustrated in Figure 2(c) and implementation details presented in Appendix A.
4 EXPERIMENTS
In this section, we conduct extensive experiments to answer the following research questions:
1) RQ1: How does our proposed AtSpeed perform LLM-based recommendation under strict top-K and relaxed sampling verification strategies? 2) RQ2: How do different components of AtSpeed affect the decoding acceleration? 3) RQ3: How do different hypereparameters affect the performance?

---

Published as a conference paper at ICLR 2025
4.1 EXPERIMENTAL SETTINGS
Datasets and Baselines. To evaluate our proposed framework, we instantiate AtSpeed on a SOTA LLM-based generative recommender model LC-Rec (Zheng et al., 2024) and test on two real-world recommendation datasets from the popular benchmark Amazon review datasets'. 1) Beauty contains user interactions with the beauty products and 2) Games collects the user interactions with the video games. For both datasets, each item has rich textual information such as title and description. Based on the item textual information, we follow L.C-Rec to assign each item with an item identifier generated by RQ-VAL VAE (Lee et al., 2022). More details of the datasets can be found in Appendix A.
We compare our proposed alignment methods with Supervised Fine-tuning (SFT) and several repre- sentative Knowledge Distillation (KD) methods as follows: 1) SFT fine-tunes the draft model with the recommendation loss Rec on the recommendation dataset: 2) WordKD (Sanh. 2019; Hinton, 2015) fine-tunes the draft model to align with the token probability of the target LLM by KLD on the recommendation dataset; 3) SD (Kim & Rush, 2016) additionally utilizes the top- K target LLM- generated data fine-tune the draft model with the recommendation loss L 4) TVDKD (Wen et al., 2023) aligns the draft model and the target LLM with the symmetric divergence TVD. We also extend retrieval-based drafting method 5) DARE (Xi et al., 2024) for a comprehensive comparison.
Evaluation Metrics. To measure the decoding efficiency, we use walltime speedup@ K (WS@) K) compared to the standard target LLM inference for evaluation. In addition, report accept step @ K" (AS@ K), which is defined as the average number of accepted steps during the SD for each user. As for the recommendation performance, we adopt the widely used metries Recall @ K and NDCG@ K for evaluation.
Implementation Details. We instantiate AtSpeed on LC-Ree, where the backend LLM LLaMA- 7B (Touvron et al., 2023). For each dataset, we fine-tune the target LLM using the recommendation loss LRe: with the parameter-efficient fine-tuning technique LoRA A (Hu et al., 2021). As for the draft model, we fully fine-tune a compatible small-sized model LLaMA-68M (Miao et al., 2024) for alignment training, and then utilize the fine-tuned model for SD inference. We set draft length y 4. number of recommended items K {1,3, 5, 10, 20). and draft beam size N 40. More implementation details are provided in Appendix A.3.I.
4.2 OVERALL PERFORMANCE (RQ1)
We evaluate our method under both standard beam search (beam size=K) with strict top- K verification and sampling-basee beam search (temperature=1) with relaxed sampling verification. The acceleration results of the baselines and AtSpeed on two datasets are presented Table 1. We can observe that:
The walltime speedup under the relaxed sampling verification is generally better than the strict top-K verification, validating the effectiveness of our proposed relaxed strategy. It boosts the speedup by allowing non-top-K sequences to be accepted with a certain probability.
Compared to SFT, the logit-level KD methods (WordKD and TVDKD) often yield comparable or even inferior performance. However, this is not surprising because the alignment task for the logit-level KD is more difficult due to two possible reasons. 1) While SFT solely focuses on fitting the top-1 probability token, logit-level KD requires alignment across the entire vocabulary, making it more difficult to achieve due to the limited expressiveness of the smaller model (Agarwal et al., 2024). 2) The alignment on the top-K K ranked items might be negatively affected by the invalid tokens with high probabilities. During item generation, certain tokens with high probabilities will never be generated as they are deemed invalid tokens, and thereby will never be accepted by the target LLM (cf. Section 3.1).
Among the baselines, SeqKD yields competitive results in most cases. This is reasonable since SeqKD 1) aligns the draft model with the target LLM at the sequence level, which fosters the generation of sequences more aligned with the target LLM's distribution (superior performance than WordKD); and 2) fine-tunes the draft model with target M-g-eeeerttdd data, which implicitly incorporates the information of valid tokens in item identifiers, thereby alleviating the negative
'We also evaluate our methods on MovieLens- IM and Goodreads datasets (refer to Table 6 in Ap- pendix A.3.2).
httpps:/ wtwwwbewweeeecc/eccootcco......b uesd. besadl.edd//-ffmaaaaeeee/tteeeee
v2..
7

---

Published as a conference paper at ICLR 2025
Table Overall comparison of walltime speedup (WS @ K) and accept steps (AS@ K) between the baselines and AtSpeed instantiated on LC-Ree (LLaMA-7B) across two datasets. "Avg" is calculated across K from 20 (full results are presented in Table 4 in Appendix A.3.2). For each verification strategy, the best results are highlighted in bold and the second-best results are underlined.
Beauty
Verification
Method
Wses
WSe
WSS2
Arg WS
ASGS
AS6F10
AS4 20
Arg AS
Recall
NDCGes
DARF
1.66
1.15
1.48
1.18
0.26
0.01
11.00
0.31
0.015
0.0051
SPT
1.43
1.37
1.55
1.56
1.32
0.96
11.09
1.18
a.0156
0.0051
WordKD
1.58
1.52
1.58
1.68
1.60
1,03
01.36
1.40
0.0051
Striet Top-K
TYDKD
1.44
1.37
1.57
1.55
1.31
0.65
11.0
1.17
0.005)
SeqKD
1.75
1.67
1.58
1.33
1.85
1.27
0.30
1.60
000166
0,0051
AtSpeed-S
1.84
1.87
1.84
1.97
2.00
1.64
0.57
1.80
0.0866
0,0051
ASSperd-R
1.70
1.71
1.74
1.76
1.82
1.33
0.43
1.56
0.0156
0.0051
DARF
1.70
1.53
1.40
1.78
1.97
1.14
1.00
1.62
0,009)
0.0044
SFT
1.50
2,06
2.36
1.95
2.03
1,99
1.48
1.94
0,005
0.001
WerdKD
1.81
1.99
2,05
1.87
2.01
1,67
1.07
1.82
0.0066
09.002
0.0045
Relaxed Sampling
TVDKD
1.81
2,05
1,99
0,0057 51,000
0U8045
(1.5000
1.90
50,000
09.0395
AtSperd-S
1.39
2.12
2.51
0.0060
11:0333
ASSperd-R
1.94
2,16
2.42
2.13
2.01
1.77
2.10
0.0088
0.0049
Games
Verification
Method
WSeE5
WSe 10
WS8020
Arg WS
AS855
AS
AS88 20
Avg. AS
Recall 85
NDCG
DARE
0.99
1.13
1.44
1.09
0.00
0.00
0.00
0.05
0.0074
0,0066
SFT
1.45
1.40
1.58
1.55
1.32
0.91
0.15
1.20
0.0774
0,0066
WordKD
1:31
1.35
1.47
1.48
1.10
0.80
0.14
1.13
0.074
0.00665
Striet Top-K
TVDKD
1.34
1.32
1.50
1.42
0.95
0.96
0.09
0.99
0.074
D.0006
SogKD
1.60
1.46
1.77
1.71
1.67
1.05
0.76
1.55
00074
0.0006
AESperd-S
1.78
1.85
1.76
1.83
1.96
1,69
0.68
1.72
0.074
0,0006
ASSpeed-R
1.26
1.76
1.00
1.74
1.95
1,53
0.32
1.39
0.0044
0,0066
DARE
1.68
1.19
1.42
1.51
1.96
0.57
0.00
1.27
0,0076
0.0055
SFT
1.84
1.97
1.59
1.86
2.05
1.89
0.38
1.78
010073
WerdKD
1.78
1.84
1.68
0.0072 11AM00
0.0058
Relased Sampling
TVDKD
1.90
1,90
21.000
1.90
051000
A1Sperd-S
1.91
2.04
2.13
2.04
2.10
1.98
1.28
2.00
0.0000
0.0068
10.0033
AtSperd-R
2.00
3.05
2.20
2.05
2.17
1.95
1.35
2.02
0,0076
0L0003
effect from the noisy invalid tokens (superior performance than TVDKD). The competitiveness of SeqKD is also consistent with the observations in NLP tasks (Agarwal et al., 2024). The less satisfying performance of DARE is reasonable since the uniformly retrieved valid sequences might not be aligned with the target L.LM specifically on top- sequences.
AtSpeed significantly accelerates the LLM-based recommender inference under both strict top-K and relaxed sampling verifications, showing the superiority of our proposed method for strong top-K K alignment. In particular, AtSpeed-S under strict top- K verification achieves an average speedup of 1.97 on Beauty and 1.83 x on Games while obtaining the identical results of the top-K recommended items. Moreover, AtSpeed-R R further enhances the efficiency with an average speedup of 2. and 2.05x on Beauty and Games, respectively. Furthermore, as K increases, the number of accept steps gradually decreases due to the increased difficulty of the N-to-K verification. Nevertheless, it is highlighted that the speedup for K 20 is still comparable that for K = 5 and outperforms the speedup for K = 10. This is attributed to tree-based attention, which efficiently verifies the drafted sequences (more detailed analysis in Appendix A.3.2).
The ranking performance under strict top-K verification is lossless because we only accept the perfectly matched top-K beam sequences to obtain identical recommendation results. For relaxed sampling verification, the ranking performance across different alignment methods only shows limited performance drops compared to the target LLM's 's top-K K results, i.e., performance under verification ensures
strict top-K verification. This meets our expectations since the sampling-bass the approximately equivalent distribution between the SD output and target LLM output (refer to Appendix A.3.2 for additional accuracy results and analysis).
4.3 IN-DEPTH ANALYSIS
Ablation Study (RQ2). To analyze the effect of AtSpeed-S, we remove the density regularization term ("wwo DR"), constrained alignment ("w/o CA"), and tree-based attention ("w/o TA") for AtSpeed-S. These variants of AtSpeed-S are evaluated under the strict top-K K verification. For AtSpeed-R, -R we replace the top-K K sequences with the top-1 sequence from target LLM (denoted as "w/o top-K"), and separately remove the constrained alignment ("w/o CA") and the tree-based attention ("w/o TA"). These variants evaluated under the relaxed sampling verification.
8

---

Published as a conference paper at ICLR 2025
(WSg2A
Striet Top-K Verification
(AS(aA)
(WSG)))
Sampling Verification
(ASgaA)
2,00
DW
2,00
2.60
WS220
2.20
BA
ASg9.3
1.60
1,80
1.20
2.20
1.60
1.80
1,40
0.80
1.80
1,20
1,00
0.00
1.40
1.40
CIA
DR
AtSpeed-S
Tapk
AtSpeed-R
8.0096
0,0095
0.0088
Recalig::::
IL0094
0,8877
(x)
(b)
Figure 3: Ablation study of AtSpeed-S and AtSpeed-R on Beauty dataset.
(WSS(A)
Top-X Verification (ASQUE)
Sampling Verifivation (AS24)
(WS228)
Striet Top-K (ASRA)
2.2
-#-W5gtt W55 Asg.l #:Agg
24
199
14
1.6
WSS
1.1
1.3
48 88
W88..0
A8
64
18
65
20
50
20
50
0.1
0.5
0.7
Effeet of for AtSpeed-S
(b) Effect of N for AtSpeed-R
(c) Effect for AtSpeed-S
Figure 4: Effect of draft beam size N and the alignment strength a on Beauty dataset.
From the results shown in Figure 3, we have the following observations. 1) the performance decline of "wo DR" indicates that aggregating probability around the top- K tokens can effectively strengthen the alignment. 2) The use of top-1 sequence from the target LLM to execute AtSpeed-R results in a lower acceptance rate. This makes sense since the top-1 sequence lacks diversity, which might overemphasize the alignment constrained alignment or tree attention would lead to performance decline for both AtSpeed-S and AtSpeed-R. This is expected since removing constrained alignment would introduce potential noisy tokens, which have high probability yet will be rejected to ensure in-corpus item recommendation. Furthermore, 4) the removal of tree-based attention would hurt the speedup although does affect the acceptance rate, which validates the efficiency of utilizing tree-based attention for verification.
the top-1 sequence and hurt the speedup. 3) By discarding either the
Effect of Draft Beam Size N (RQ3). study how the draft beam size N affects the acceleration of AtSpeed, we vary N from 20 to 60 and test AtSpeed-S under strict top-K verification (Figure 4) and AtSpeed-R under relaxed sampling verification, respectively. From the figures, we can observe that from N = 20 to N = 60, the acceptance rate constantly increases for both AtSpeed-S and AtSpeed-R. This is reasonable as a larger beam size can produce more drafted sequences for verification, thus enhancing the acceptance rate. Moreover, 2) the walltime speedup continues to improve steadily as gets larger. We attribute this phenomenon to the utilization of tree-based attention, which efficiently verifies drafted sequences with single flattened sequence (refer Appendix A.3 for detailed speedup analysis of tree-based attention). Nevertheless, 3) the improvement of acceptance rate and speedup gradually slows down (i.e., decreasing slope), indicating continuously increasing N is approaching a balance between the enhanced acceptance rate and the increased overhead.
Effect of Alignment Strength a (RQ3). We train the draft model with different alignment strength a and present the results of AtSpeed-S in Figure 4(c)². From the figure, it is observed that the integration of alignment loss (incrementally increased from 0) improves performance, which validates the necessity of alignment to improve acceptance rate. However, 2) blindly emphasizing the alignment loss may ineffective, as the model struggles to capture the output of the target L.M due the limited expressiveness of the small-sized model without the knowledge of recommendation task. Therefore, it is essential to strike a balance between the task-specific objective and the alignment objective. Empirically, we recommend setting 0.5 for alignment training.
RELATED WORK
Speculative Decoding. As efficient L.M decoding technique, SD (Leviathan al., 2023; Fu et al., 2024; Du et al., 2024; Liu et al., 2024c; Chen et al., 2024; Sun et al., 2024) has been proposed
Mooe analysis of AtSpeed presented in Appendix A.3.2 save space.

---

Published as a conference paper at ICLR 2025
to reduce the LLM calls by efficiently drafting multiple future tokens and then verifying them in parallel. Early studies focus on SD in greedy decoding, where the draft model generates one token at each generation step for LLM to verify if they match the LLM's top-1 token. Later on, some studies extend SD to nucleus sampling to accelerate I.L decoding without changing its output distribution (Leviathan et al., 2023). However, the early attempts simply draft one token at each step for verification, which significantly limits the acceptance rate (Xia et al., 2023). To improve the acceptance rate, subsequent work designs various relaxed verification strategies for greedy decoding such as enlarging LLM candidates with a tolerable score away from the top-1 token (Xia et al., 2023) and measuring the distance of the probability between the draft model and the target LLM (Kim et al., 2024). Concurrently, some studies propose different methods to draft multiple candidates for different sampling, as seen in Medusa (Cai et al., 2024), EAGLE (Li et al., 2024a) and so on. While previous work primarily applies SD for NLP tasks and follows the N-to-1 verification, our work extends SD to LLM-based generative recommendation tasks, addressing the unique challenge of N-to-K verification, i.e., perfect matching over the top-K sequences. Closely related to work, DistillSpec (Zhou et al., 2024) investigates the use of various KD techniques to enhance the alignment between the draft model and the target LLM, to achieve high acceptance rate. However, they mainly focus on how KD helps for the N-to-1 verification. In contrast, this work studies the SD under N-to-K verification for recommendation tasks, and proposes a framework to align the draft model with the target LLM on the top- K sequences specifically for recommendation tasks. While another conceptually similar work SpecGR (Ding et al., 2024) drafts unseen items for cold-start recommendations, this work aims at drafting beam sequences at every step for inference acceleration.
Inference Acceleration for LLM-based Recommendation. While LLM-based generative recom- mendations have shown remarkable performance (Liu et al., 2024a; Xu et al., 2024a; Zhang et al., 2024a; Lv et al., 2025; Lin et al., 2024b; Zhao al., 2025) compared to traditional recommender models (Liu et al., 2024b; Gao et al., 2023; Li et al., 2024b; Zhang et al., 2021), their practical implementation is impeded due to the high time latency of LLM decoding. Consequently, acceler- ating LLM decoding imperative to facilitate real-world deployment of the powerful LLM- based recommender models. To tackle this issue, several approaches leverage KD to transfer knowledge from teacher LLM to either a smaller language model (Xu et al., 2024b; Wang al., 2024h) traditional recommender model (Cui et al., 2024). Additionally, some methods focus on designing more lightweight model architectures, such narrow and deep transformer structures (Mei & Zhang, 2023). However, the application of SD is underexplored in LLM-based recommendation. DARE (Xi al., 2024) is relevant study that applies SD in user feature augmentation for LLM-based recom- mendation. However, they focus on the LLM decoding for user feature generation that requires only one response from the target LLM, which is essentially SD under the N-to-1 verification. In contrast, our work explores decoding acceleration for top- K ranked item recommendation. tackling the more difficult N-to-K K verification problem. In addition to these approaches, various other techniques, such as model quantization (Xiao et al., 2023), model pruning (Hassibi & Stork, 1992), and Key- (KV) cache optimization (Zhang et al., 2024b), can also enhance inference acceleration. Nonetheless, these methods are orthogonal to SD and can be effectively integrated with SD to further improve decoding efficiency.
6 CONCLUSION AND FUTURE WORK
In this work, we formulated the speculative decoding task for LLM-based generative recommendation, highlighting the significant challenge of shifting from N-to-1 to N-to-K. To effectively accelerate decoding under -tto-K verification, the key lies in drafting all top-K sequences from the target LLM, which drives two objectives for SD: top-K alignment and verification relaxation. To achieve this, we proposed an alignment framework called AtSpeed, along with relaxed sampling verification strategy. AtSpeed designs two alignment objectives for the strict top-K verification and relaxed sampling verification strategies. Extensive experiments show that AtSpeed achieves around 2x speedup two real-world recommendation datasets. As an initial attempt work, it points out many promising directions for further explorations, such as 1) more efficient drafting methods to reduce the overhead of drafting, such as LLM recommender models for efficient high-qualin drafting that might lead to better recommendation accuracy under the relaxed sampling verification strategy, and 2) integration with other acceleration techniques such memory access optimization.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our research focuses on effectively boosting the inference efficiency of LLM--ased generative recom- mendation with the AtSpeed framework, unlocking the significant practical potential of LLM-based recommender models. This work is technical and practical, with applications to LLM-based genera- tive recommendation. We have fully considered the potential societal impacts and do not foresee any direct, immediate, or negative consequences. We are committed to the ethical dissemination of our findings and encourage their responsible use.
REPRODUCIBILITY STATEMENT
All the results in this work are reproducible. We provide all the necessary code to replicate our results in an GitHub repository. The repository includes environment configurations, run scripts, and other relevant materials. We discuss the experimental settings in Section 4.1, including implementation details such the utilization of parameter-efffcient training. Additionally, detailed experimental settings are provided in Appendix A.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In International Conference on Learning Representations, 2024.
Keqin Bao, Jizhi Zhang, Wenjie Wang, Yang Zhang, Zhengyi Yang. Yancheng Luo, Fuli Feng, Xiang- naan He, and Qi Tian. A bi-step grounding paradigm for large language models in recommendation systems. arXiv:2308 08434, 2023.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. In International Conference Machine Learning. PMLR, 2024.
Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Jie Huang, and Kevin Chen-Chuan Chang. Cascade speculative drafting for even faster Ilm inference. In Conference on Neural Information Processing Systems. Curran Associates, Inc., 2024.
Yu Cui, Feng Liu, Pengbo Wang, Bohao Wang, Heng Tang, Yi Wan, Jun Wang, and Jiawei Chen. Distillation matters: Empowering sequential recommenders to match the performance of large language model. ACM Conference on Recommender Systems. ACM, 2024.
Yijie Ding, Yupeng Hou, Jiacheng Li, and Julian McAuley. Inductive generative recommendation via retrieval-based speculation. 2024.
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. In International Conference on Machine Learning. PMLR. 2024.
Jacques Dutka, The early history the factorial function. Archive for history of exact sciences, pp. 225-249, 1991.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. In International Conference on Machine Learning. PMLR, 2024.
Chongming Gao, Shiqi Wang, Shijun Li, Jiawei Chen, Xiangnan He, Wenqiang Lei, Biao Li, Yuan Zhang, and Peng Jiang. Cirs: Bursting filter bubbles by counterfactual interactive recommender system. TOIS, 42(1), 2023. ISSN 1046-8188
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In International Conference on Learning Representations, 2024.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Conference Neural Information Processing Systems, volume 5. Curran Associates, Inc., 1992.
11

---

Published as a conference paper at ICLR 2025
Geoffrey Hinton. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253)
2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Alue Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv:2106.0900 2021.
Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How to index item ids for recommen- dation foundation models. In ACM Annual International ACM SIGIR Conference on Research and Development in Information Retrieval Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pp. 195-204. ACM, 2023.
Ferene Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv: 151.0001 2015.
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Conference Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2024.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Conference on Empir- ical Methods in Natural Language Processing, pp. 1317-1327. Association for Computational Linguistics, 2016.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 11523-11532, 2022.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077 2024a.
Zhenyang Li, Fan Liu, Yinwei Wei, Zhiyong Cheng, Liqiang Nie, and Mohan Kankanhalli. Attribute-driven disentangled representation learning for multimodal recommendation, In MM, pp. 9660-9669. ACM, 2024b.
Xinyu Lin, Wenjie Wang, Yongqi Li, Fuli Feng, Sce-Kiong Ng, and Tat-Seng Chua. Bridging items and language: A transition paradigm for large language model-based recommendation. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, 2024a.
Xinyu Lin, Wenjie Wang, Li Yongqi, Shuo Yang, Fuli Feng, Yinwei Wei, and Tat-Seng Chua. Data-efficient fine-tuning for ilmebased recommendation. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2024b.
Fan Liu, Yaqi Liu, Huilin Chen, Zhiyong Cheng, Liqiang Nie, and Mohan Kankanballi. Understanding before recommendation: Semantic aspect-awas awwrre review exploitation via large language models. TOIS, pp. 1-26, 2024a.
Fan Liu, Shuai Zhao, Zhiyong Cheng. Liqiang Nie, and Mohan Kankanhalli. Cluster-based graph collaborative filtering. TOIS, 42(6), 2024b.
Xiaoxuan Liu, Lanxiang B7H Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In International Conference on Machine Learning. PMLR, 2024c.
Zheqi Lv, Tianyu Zhan, Wenjie Wang, Xinyu Lin, Shengyu Zhang, Wenqiao Zhang, Jiwei Li, Kun Kuang, and Fei Wu. Collaboration of large language models and small recommendation models for device-cloud recommendation. 2025.
Kai Mei and Yongfeng Zhang. Lightlm: a lightweight deep and narrow language model for generative recommendation. arXiv preprint arXiv: 2023.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.152: 2023.
12

---

Published as a conference paper at ICLR 2025
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 932-949. ACM, 2024.
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghumandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al. Recommender systems with generatie retrieval. In Conference on Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2023a.
Shashank Rajput, Nikhil Mchta, Anima Singh, Raghunandan H Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong. Yi Tay, Vinh Q Tran, Jonah Samost, et al. Recommender systems with generative retrieval. In Conference on Neural Information Processing Systems. Curran Associates, Inc., 2023b.
V Sanh. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01 108, 2019.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. In Conference on Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Bric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 2e.i2222221391,,,
Wenjie Wang. Honghui Bao, Xinyu Lin, Jizhi Zhang, Yongqi Li, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. Learnable tokenizer for Ilm-based generative recommendation. In International Conference on Information and Knowledge Management. ACM, 2024a.
Yuling Wang, Changxin Tian, Binbin Hu, Yanhua Yu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou, Liang Pang, and Xiao Wang. Can small language models be good reasoners for sequential recommenda- tion? In Proceedings of the ACM on Web Conference, pp. 3876-3887 ACM, 2024b.
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation, In Annual Meeting of the Association for Computational Linguistics, pp. 10817-10834. Association for Computational Linguistics, 2023.
Yunjia Xi, Hangyu Wang, Bo Chen, Jianghao Lin, Menghui Zhu, Weiwen Liu, Ruiming Tang, Weinan Zhang, and Yong Yu. A decoding acceleration framework for industrial deployable Ilm-based recommender systems. arXiv preprint 20Xiv:2408.05676, 2024.
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Conference on Empir- ical Methods in Natural Language Processing, pp. 19099-925, Association for Computational Linguisties, 2023.
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Chen Xu, Wenjie Wang, Yuxin Li, Liang Pang, Jun Xu, and Tat-Seng Chua. A study of implicit ranking unfairness in large language models. In EMNLP, pp. 7957-7970. ACI., 2024a.
Wujiang Xu, Zujie Liang, Jiaojiao Han, Xuying Ning, Wenfang Lin, Linxun Chen, Feng Wei, and Yongfeng Zhang. SImrec: Empowering small language models for sequential recommendation. arXiv preprint 2rXiv:2405.1788 2024b.
13

---

Published as a conference paper at ICLR 2025
Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang. and Liang Wang. Mining latent structures for multimedia recommendation. In MM, pp. 3872-3880, 2021.
Jinghao Zhang, Yuting Liu, Qiang Liu, Shu Wu, Guibing Guo, and Liang Wang. Stealthy attack on large language model based recommendation. 2024a.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song. Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter orannual meeting of the association for computational linguisticse for efficient generative inference of large language models. In Conference on Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2024b.
Jujia Zhao, Wenjie Wang, Chen Xu, Zhaochun Ren, See- Bee-Kiong Ng, and Tat-Seng Chua. Llm-based federated recommendation. In NAACL. ACL, 2025.
Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen. Adapting large language models by integrating collaborative semantics for recommendation. In International Conference on Data Engineering. IEEE, 2024.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation, In International Conference on Learning Representations, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ALGORITHM OF RELAXED SAMPLING VERIFICATION
Algorithm 2 SD step with Relaxed Sampling Verification
Input: Draft model My. larget LLM My. prefix, target beam size K, draft beam size N I: 30 + prefix, ) + 0, Your f-
2: for j 1 to do
3: 9 M2 1). ....) 3
Sample K sequences from 9
p2,2,.... ........ P444 f-
D. Run Mp in parallel
5: for j = 1 1 do
6: Vig < 9
7:
for i= 1 K do
Verify each sequenee based independent modeling
8:
7, (0, 1), y: - The i-th sequence from )
9:
if rf I p(v.) then
> Accept the sequence
a(w))
10:
Von + Von U
11:
else
b Reject the sequence
12:
Vnj < Vig U(3}
13:
if [ K then
D. Accept the step
14:
Vou Randomly sample K sequences from V
15:
else
b Reject step and resample the sequences
16:
f- norm (max00, -;))
17:
(3.) + for y E Veur
18:
) Vieeammll Sample B sequences from p/.
B K Voul
19:
Vou Vou U Vresample
Output: Von
In Algorithm 2, norm(max(0, - -9)) =
max(O.p;=0))
max
A.2 PROOF AND DETAILED DERIVATION
A.2.1 WITI-REPPACEMENT SAMPLING APPROXIMATION
We aim to prove that the distribution of sampling without replacement is approximately equivalent to that of sampling with replacement. Our proof is mainly based on Stirling's ' approximation (Dutka, 1991). In the following, distribution and the multinomial distribution, which are used to model the sampling without replace- ment and sampling with replacement, respectively. We then present Stirling's approximation, and show the step-by-step proof.
will first clarify notations, and introduce multivariate hypergeometric
Notations. To model the sampling, we have the total population size N, sample size n, the category size r, the number of items in category in the population K;, and the number of items in category in the samples kg. In the case of sequence sampling in LLM decoding, the population includes every possible sequence. Every possible sequence is unique category, and the sample size n is the beam size, the population size is the total number of all possible sequences at each beam search step. Now we assume the population sizes go to infinity in such a way that p; .. and Ki N.
K. And we have Σ Di k
Multivariate Hypergeometrie Distribution. Formally, when sampling without replacement, the probability drawing k1, .e....,, ky items from each category is given by the multivariate hypergeo- metric distribution
K M
Phyper (ki, .e....,
(10)
kick
N!
nj]
15

---

Published as a conference paper at ICLR 2025
Multinomial Distribution. Formally, when sampling with replacement, the probability follows the multinomial distribution as
m I ky!
Pmuttl (ki,
(11)
Stirling's Approximation. Stirling's approximation (Dutka, 1991) gives us the approximation the logarithm of factorials as:
In 2 nlnn n + In(2Tn). 2
(12)
Theorem 1. When population size N is large and sample size is small compared to N (i... n & N), the multivariate hypergeometric distribution approximates the multinomial distribution:
(ky , ky) se Pmmuti(ke (k2, ,2....,,,))
(13)
Proof.
We can expand the logarithm of multivariate hypergeometric probability in factorials as follows:
I K. K,K,K
In Payper(As (Ai,, k2, .2,...,k,) = In
N
m! -nj!
K₁!
N!
kgl(K, - k;)!
n!(N n))
(14)
Kg!
N!
In
k;,(K; k:))
n!(N - nt)!
In K2! - In kjl In(K, k [In No In In(Nnn n])]..
1=1
Using the Stirling's approximation, have:
In K2! K K; K; + In 2T
kn R3 Ink4 k, + 2TT
In(K;- - k:)! ~ (Ke 1,) In(K; k,) (K; k2) + 2T ks),
(15)
In ~ Nln - N In 2T
In ~ n n
In 2mn
2
n)! ~ (N ---m n) In(N - n) - (N n) + 2T(NN - n).
16

---

Published as a conference paper at ICLR 2025
Then, substitute the logarithm of factorials with the approximation in Eq.(14) as:
In Phyper (k1, ,2,-..,ki kr)
In K2! - kt. - In(K; -a)! [In N - In - In(N n)!
1
K K, - -K++
+ + In 2T., - (k: kn - k2 +
In 2xTks)
2
2
1
- ((K; - kk:) -n:)ln(K - k:) - (K: - k;) + In 2 (K; k:))
2
IN - +
2 In N -n - n マ/2 In 2mn)
1
2
1
-(N In m) (N- - m) + In 2 (N -n))]
2
1
[ K; - lin - (K, - kk,) In(K; - k;) + In k
-
In 2
In 2T( k;)]
2
2
1
N (N n) In(N n) + 2 In 2n N-
In 2mn
In 2T (N n2)l.
2
(16)
Since N is a very large number and n < N, k < Ki we have
In 2 i 2 2 In (m ki) s 0 and 2 In 2T 1 In 2mn 1 In 2T n) xe O. Then, the logarithm of multivariate hypergeometric distribution is approximated as:
In Payper RM
ΣiK, [K, K; k In - -K; - -k) In(, - IN Nn (N-n)
In(N - n)].
(17)
Since ky is small compared to KG, we can expand In(K, k2) using Taylor expansion
k
hs
In(K; ky) In K, where we neglect the high-order terms and obtain
(18)
K;
K;
....
k;
In(K; - ki)) Re K;
(19)
K;
Similarly, for In(N- - 12), we have
In(N n1) R N
(20)
N
We can then substitute In(K; ks) and In(N n) in logarithm of multivariate hypergeometric distribution (17) and obtain
Happer
Ki In K: Inke (K: (oKs-ka)lii In(K, ks)] [N In N n Inn nN) - -n]]
K, In K. kk (K, k2) K; + he )] K.
IN N n - NN ~)
) - nln - n²
1K,n In K. k In (K, In K. - by In K, + h2
>>
K.
ly In K;
h + K2
N
n²
k.]
|n In
ml
K,
n
N
ke In K; 2.2
In In
(we have-
0n last expression)
k; K:
N
(21)
17

---

Published as a conference paper at ICLR 2025
We then relate K; to N and ps. Since K, Nps. have In K; In Np. We also have n Ei=1 his Then, we can express the logarithm of hypergeometric distribution in terms of ps and k; as
K2
k
N
n
In P hypee
2
Rig In
+
n
In
R;
K;
N
Np:
k2
n²
Rbs In
In
+
h;
Np:
n2
N
k-
N
n²
Re (In N + In
In k;)
+
[n
In
+
Np;
n
N
k?
N
n²
N
(1m In k.))
+
In
Nps
72
N
n²
k
= min N - n In +
+
Ik (Inp;
In kn);)
+
(we have Σ k: in last expression)
N
Np:
k?
n²
(Ia - km); +
+ n
Inn
Np;
N
n
h2
lnnn-
Inkk
by lnpp
Np:
(22)
Now the approximation of the logarithm of multivariate hypergeometric distribution has been finished. Similarly. we approximate the multinomial distribution with Stirling's 's approximation. We expand the logarithm of multinomial distribution as
Paunti (k1,
= I
k
1=1
(23)
In in!!
Σ
In knlt
Σ kg In pi-
Using Stirling's approximation, we have Inn! s n lnn n and In ky xo Ky Ink; - k. We then
substitute nn and In k; with the approximation and obtain
In malii R n In - n Σ (ku ki - ki)) + Σ ke Iu
(24)
=nhn
k In
+
k Pr.
1=1
Since have Ei Kr nI, have
In Pmutii & n lu - 1 Σ
ki In ki + n +
k Inpp
1=1
1=1
(25)
=nhn
Σ In
ky Inps.
Now, comparing the approximated logarithm of multinomial distribution with the approximated logarithm of multivariate hypergeometric distribution, we have
n²
k?
In Pmmltii 2 Phyper +
(26)
N
Nps
18

---

Published as a conference paper at ICLR 2025
Note that the term N Ei=1 A is negalectable when is large and k; and n are small compared Np: to N. Therefore, we show that when the population size N is large (e.g., all sequences for sampling) and k;, small (e.g., h less than 20 in LLM-based recommendation) the multivariate hypergeometric distribution can be approximated to the multinomial distribution. That sampling without replacement approximately equivalent to sampling with replacement.
0 since each sequence represents category and is usually
A.2.2 DERIVATION OF ALIGNMENT OBJECTIVES OF ATSPEED-S
To maximize the acceptance rate under the strict top- verification, consider:
p(y)
arg max #EG
Ey-Y's p(yk)
Ey-Y log p(y)
arg RE
p(ux)
(27)
Σ ((1) log ply)(y)
Σ ((y) logr(uk)
Σ ((r) log p(y)
Σ ((v) logp(yx).
We can add (Ey a(y) (wy) Ey (v) log (V)) to the expression since Ey (v)) log ((1)
EΦ (1)) log (z) = 0. After adding the term with some rearrangement, we obtain
Σ 918) log p(y)
Σ (2v) ((y)()
E ((v) log (vv)
Σ ( log
M
Σ ((y) log
Σ (8) (w)) Σ (()) log
Σ
(w) log logp(yk)
p(y)
(28)
Σ
((y) log
((y)
Σ (1)) log ((v)
Σ {()) logp(ık)
p(y)
Σ a(y) log
((v)
Σ y
4()) log
(lv)
p(y)
p(yx)
where p(uk) p(yk, (yk,</) According the derivation, obtain the alignment objective as in (2). We now consider context in LLM-based recommendation. We omit input æ and denote Cit = Yet, referred to as context, then we have ( = (Velcce) and p(y) IIe p(ye Cee). We then can substitute q(y) with II. ((31)ce)) and p(y) with II. P(yelCe)) and rewrite the objective in (28) as:
(1)) p(H)
Σ ((2) log 2(v)
Σ (1) log
p(gk)
II. alan loxe)
II. (ueloc))
Σ ( log
Σ ((v)) log
II, p(y (ce))
II. el
2/
Σ (lg II [(vllee))
- -log II [(1e) + Σ a(v))logg II glve (ec))ce)) - -log II PluK,
Σ (()) Σ log (ogq(velcee)
Σ logp(yrccee)l + Σ (())) <(v) log loga(uilced - Σ log lox
log A(vileee) Σ ((v) Σ
Σ 9(8) log a(salce))
p(urle p(uilcee) cee)
p(y..)
ΣΣ Σ (3)) log g(ullce)) p(uilce))
Σ Σ ((y) log ((sl)ee)
p(yK,NuK,))
(29)
Since y = (211822, ... Un)
(y Ut. y>t) is a sequence, we can decompose Σ, ((v) into nested sum Eye, Eg, Eys, (ly) Eyer Eg, Eys, AWeer. Ve, yst), where the nested sum
19

---

Published as a conference paper at ICLR 2025
three parts of y. f.e. yet. yt, and Y> cover all possible sequence y. Since (V Vt. y>c)
(ove)))(llcc))((y>slll we can rewrite the nested sum of (V yo. y>t) over y:
(1))
Σ E
Σ
a (ener
MCt
N>e
(30)
Σ Σ ((ver))(unlce) Σ ((salee)))
Σ Eq(Vet) Σ alullce)) Σ Lyysslege))
We then can substitute and obtain
En ((y) with Exsc ((ye)) Ey, ((vlyyer) Ens, a(uxelogr) into (q.
ΣΣ { log ((slcce)
ΣΣ 9(v) log q(velce))
plyelcee)
p(yK,1yKK,tt
Σ ((y>elese)) log ((elce))
Σ (yEet) Σ ((velees)
p(yelee))
Mce
U
V>
a(velce)
DIΣ a(cc)) Σ a(velce)) Σ [alisclcs)))og Mice
p(yk, (UK,e))
Usc
ΣΣ ((v<)) Σ ((volce)) log g(velee))
Σ [3(ssllo5))
p(yeecee))
Vet
((volc2)) log q(pllce)) Σ ((3Sele3)).
ΣΣ (3)L)) Σ VCL
plyk,lyk,es
(31)
Since Eys, a(y>elo5)) 1, we remove Ey., ((yreles)) in E3,
ΣΣ (UEL)) Ed(velee) log (vllace) plue cxe) Σ ((3
Σ [(volcee) log alul Σ C((sseles))
((
p(yk: [xK,<t) Mst
vice
D1
ΣΣ (v)))) Σ ((vllee) log a(v1/cet
a(veleee)
Σ Eg(ve) Σ C(lellce))
pluelee (cee)
p(yK.:lyx.e)) (32)
ΣΣ a(V&)) Alulcce) log ((ee)
Σ (oze) log g(veleee)
p(yelexe)
plyk,19K.e1)
Vici
31
= =-
--E
E
((v loea) log (y:lcee) p(vllce))
Σ (xvelc=)) log pluk.1yk.ec)
q(yo(e2))
Now we have a step-wise alignment over all sequence from g, i.e., Eyerry- We aim align the draft model with the target LLM at every beam search step. Notably, at each beam search step T, the sequence lengths are fixed and independent with Therefore, we can rewrite (32) as:
q(ve ((ve)ee)) p(velee))
9(16 (2e)) log a(vilcee) Σ 211
Σ Eyer mig
((v()e2)) log
p(yx..BK.e.)
D4
(33)
Eygr
Ale (2t) log q(vclee))
Σ ((Vllce))
a(v)(c2))
p(an (cze)
p(yx.BKKKe
Since we aim to align at every step T E (1...., [)...,,,,,.. where is the length of item identifier in LLM--base recommendation, further normalize the expression by sequence length prevent
20

---

Published as a conference paper at ICLR 2025
different scales alignment loss across different steps. As such, for every step T {l..... L}, the objective can be rewritten as
Eysr
((v)) log p(y (cee)
((vloa))
Σ ((v)) log
q(vllee)
(34)
BT
plyR, (yk.IyK..t)
However, the expectation over the sequence space, (i.e,, Eygr) is intractable, so we follow previous work (Wen et al., 2023; Kim & Rush, 2016) to approximate it by sampling top-K sequences generated by draft model Mg. Now we can rewrite our alignment objective for strict top-K verification and obtain (3) in our paper:
M
( (oze)
g(sn (ce)) p(yx, (e))
max R 31
(veleee) log
glv (cee) log Σ N
plyh (cce)
DEY
1
avelcce) log (velaee) p(uilce))
9(3 (cze)
arg min B(
Σ Wt
((eelce)) log p(yK, (UK,c))
Dee
KEY
(35)
where Y denotes the top-K' K beam sequences generated from the mixture distribution of draft model and target LLM (see "Alignment Objective" paragraph in Section 3.1).
A.2.3 DERIVATIVES OF ACCHPTANCE RATE FOR RELAXED SAMPLING VERIFICATION.
Based on the with-replacement sampling approximation, for every independent sampling, we have acceptance rate of sequence
((y) S p(y)
p(y) (
b - a(v)
plvl 200)
(3y) p(y)
Ey min(1,
TVD(P()).
(36)
which can be easily extended from (Leviathan et al., 2023) as follows.
Definition 1. TV (VD(()g) = E, (p()) - M(y)) = Ey (q()) M (v)). where M(4) = p(a))+a(v))
Lemma 1. TVD = En mimmmi(())....)). <())).
Proof. TVD(p, g) = E, Ip(2) M(2)l = En 2 E, min(p(y) - -q(V)).
= - E, p(u)+o(u) (p(u) (p(u)-=(z) wlv) -
Theorem 2. b = - DDVDVPPpp g).
(4v) < ply) ( > pvy)
Proof. By
E Byna(a) min(1,
p(y) ((r)
Σy min(1,
p(M) q(vl
p(u) otu)
Based on the with-replacement sampling approximation (see Appendix A.2. for proof), we can obtain the acceptance rate of step with K accepted sequences as G II b. Similarly, we could also extend the distribution equivalence for sampling sequence between the draft model and the target LLM from Appendix A. in (Leviathan et al., 2023). As such, when the sampling-based beam search performed with a beam size K, the joint distribution of the output K sequences from the SD under the relaxed sampling verification is approximately equivalent to the counterparts from the target LLM, according to the approximation of with-replacement sampling, i.e., independent sampling.
A.3 EXPERIMENTAL DETAILS AND ANALYSIS
A.3.1 DETAILED EXPERIMENTAL SETTINGS
Datasets Details, For both Beauty and Games, all interactions are sorted according to the global timestamps, and then split into training. validation, and testing sets with the ratio of 8:1:1. The dataset statistics is presented in Table 2. For the item identifier, we follow LC-Ree (Zheng et al., 2024) to set the length I. 4, i.e., the token sequence length of generated item would be 4.
21

---

Published as a conference paper at ICLR 2025
Table 2: Statistics of the datasets.
Datasets
Users
* Itenss
Internetions
Density
Beauty
19,383
12,035
138,870
0.06%
Games
49,156
17,332
342,329
0.04%
Movieleens-IM
6.038
3,017
60,162
0.33%
Goodreads
2,437
4,667
80,744
0.71%
Implementation Details. For draft model training, use AdamW optimizer with batch size= 64, learning rate=0.001, and cosine scheduler with warmup step of 200 to adjust the learning rate. We train the draft model for 20 epochs on 4 NVIDIA RTX A5000 GPUs. Meanwhile, alignment strength in {0.1, 0.3, 0.5, 0.7} and weight decay in {0.01, 0.1}. For efficient training, we utilize top-K sequences from the last step of beam search to construct the alignment data D', Despite the utilization of beam sequences from different steps of beam search could potentially enhance the acceleration effects further, it exacerbates the training burden regarding time cost and storage cost (see Table 7).
search the
Recommendation Loss. The recommendation loss used for the backbone LLM-bbssed recommender model in our work is defined as
Cree
Σ
Σ
log Mp(yel
(37)
N
(m.D))~
where x is user's historical interactions, y is the user's next interacted item identifier, and D = {(xx, y)} denotes the original recommendation dataset. Nevertheless, our method is model-agnostic, thus the recommendation loss LRCe can be can be substituted by any form of the loss function from the backend LLM-based recommender models.
Implementation of Tree-based Attention. We first compress the N beam sequences into a single flattened sequence, and then construct the sparse tree-based attention mask for efficient target LLM verification. More precisely, given >N drafted beam sequences with different lengths, where N is the draft beam size and y is the number of drafted beam steps, we first 1) flattened the beam sequences by sequentially adding the newly generated tokens from the beam sequences at each step. We denote the length of the flattened sequence as Ly. Then, based on the flattened sequence, we construct an attention mask with the shape of L x Ly. Specifically, each row in attention mask represents specific beam sequence. For each row, we set the corresponding column of the last token as well as the preceding tokens in the beam sequence as I, otherwise O. For example, we set the beam size of and collect the beam sequences of 3 steps as shown in Table 3.
step beam 1: al
step beam 2: a2
step beam 1: al bl
step beam 2: a2
b2
step 3 beam 1: al bl
cl
step beam 2:
bl
c2
Table 3: Beam sequences from 3 beam search steps with beam size 2. The flattened sequence is "al a2 bl b2 cl c2".
The flattened sequence will "al a2 cl c2". The constructed tree-based attention is shown in Figure 2(c) of our manuscript, where each row represents a specific beam sequence. For each row, the tickled cell represents the preceding tokens and the last tokens of cach beam. And the different colors represent the different steps of beam search. This flattened sequence and sparse tree-base based attention enable efficient verification since it saves the repeated calculation of the same prefix across different beam sequences, e.g., "al bl".
A.3.2 ADDITIONAL. EXPERIMENT RESULTS.
Comprehensive Results Beauty and Games Datasets. We report the full acceleration perfor- mance comparison on Beauty and Games datasets in Table and the full recommendation performance comparison on the two datasets in Table 5, where have similar observations in Table 1.
22

---

Published as a conference paper at ICLR 2025
Table 4: Full results of walltime speedup (WS@K) accept steps (AS@ K) between the baselines and AtSpeed instantiated on LC-Ree (LLaMA-7B) across two datasets. For each verification strategy, the best results are highlighted in bold and the second-best results are underlined.
Beauty
Verification
Method
WS@3
WS@S
Ws@10
WS@20
AS@5
AS@
AS@
DARE
1.13
1.07
1.06
1.15
1.48
0.81
0.44
0.26
0.05
0,00
1.55
1.43
1.37
2.19
WordKD
2,02
1.70
1.58
1.52
1.58
2.33
1.89
1.60
1.03
0.16
Strict Top-K
TVDKD
1.37
SeqKD
2.18
188
1.75
1.67
1.88
2.46
2.10
1885
1.27
0.30
AtSpeed-S
2.33
1,97
1.84
1.87
1.84
2.58
2.20
2.00
1.64
0.57
AtSpeed-R
1.95
1.71
1.70
1.71
1.74
2.26
1.96
1.82
1.33
0.43
1.65
1.53
1.97
SFT
1.77
1.76
1.80
2,06
2.36
2.155
2.06
2.03
1.99
1.48
WordKD
1.07
Relaxed Sampling
TVDKD
1.80
1.77
1.81
2.06
2.35
2.17
2.07
2.03
1.99
1.45
SeqKD
1.85
1,86
1,90
2.11
2.31
2.21
2.13
2.10
2.01
1.40
AtSpeed-S
1.94
188
1.89
2.12
2.51
2.26
14
2.09
2.03
1.71
AtSpeed-R
2.05
1.94
1.94
2.16
2.47
2.37
2.19
2.13
2.01
1.77
Verification
Method
WS@1
WS@3
WS65
WS@ 10
WS
A
AS@3
AS@5
AS@ 10
AS@ 20
DARE
0.94
0.95
0.99
1.13
1.44
0.15
0.00
0.00
0.00
0.00
SFT
1.82
1.45
1.43
1.40
1.58
2.13
1.49
1.32
0.91
0,15
WordKD
1.82
1.45
1.31
1.35
1.47
2.13
1.49
1.10
0.80
0.14
Striet Top-K
TVDKD
1.70
1.33
1.24
1.32
1.50
1.99
1.26
0.95
0.66
0.09
Seq
1.72
1.46
1.67
ALSpeed-S
2.01
1.77
1.78
1,85
1.76
2.27
2.02
1.96
1.69
0.68
AtSpeed-R
1.76
DARE
1.59
1.64
1.68
1.19
1.42
2.00
2,00
1.96
0.37
0.00
1.96
1.85
1.84
1.97
2.28
2.12
2.05
0.58
WordKD
1.84
1.77
1.78
1.84
1.56
2.17
2.05
1.99
1.68
0.25
Sampling
TVDKD
SegKD
1.91
1.87
1.90
2,03
2.05
2.22
2.13
2.10
1.98
1.22
AtSpeed-S
2.16
1,28
AtSpeed-R
2.10
1.92
2.00
2.05
2.20
2.40
2.18
2.17
1.98
1.35
Table 5: Full results of ranking performance in terms Recall and NDCG between the baselines and AtSpeed instantiated LC-Rec (LLaMA-7B) on Beauty and Games datasets.
Beauty
Method
Recall@s
Recall@:10 10
NDCG@5
NDCG@ 10
Target (Top))
0.0056
Without SD
Target LLM (Sampling)
0.0056
0.0082
0.0043
0.0066
SFT
0.0057
0.0091
0.0041
0.0063
WordKD
0.0066
0.0105
0.0043
0.0058
Relaxed Sampling Verification
TVDKD
0.0057
0.0083
0.0045
0.0054
SegKD
0.0055
0.0116
0.0045
0.00367
AISpeed-8
0.0060
0.0096
0.0046
0,0060
AISpeed-R
0.0058
0.0092
0.0049
0.0063
Average
0.0059
Games
Method
Recall@5
Recall@:10
NDCG@5
NDCG@10
Without SD
Target LLM (TopK)
Target LLM (Sampling)
0.0075
0.0074
0.0115
0.0125
0.0066
0.0065
0.0083
0.0079
DARE
0.0076
0.0119
0.0065
0.0080
SFT
0.0073
0.0112
0.0060
0.0074
WordKD
0.0072
Relaxed Sampling Verification
TVDKD
0.0069
0.0108
0.0061
0.0074
SceKD
0.0059
AtSpeed
0.0080
0.0131
0.0068
0.0085
A1Speed-R
0.0076
0.0123
0.0063
0,0085
Average
0.0074
0.0117
0.0062
0.0077
Detailed Analysis on Recommendation Accuracy. Based the results in Table 5, we have following observations: 1) The ranking performance under striet top-K verification is lossless. This is expected since strict verification only accepts the drafts that perfectly match the top-K sequence from the target LLM. Therefore, we obtain identical generation results with and without speculative decoding under strict verification. Based lossless results, proposed method AtSpeed-S achieves up to an average of 1.85x speedup. 2) The ranking performance under relaxed sampling verification across different alignment methods only shows limited performance drops
23

---

Published as a conference paper at ICLR 2025
Table 6: Performance comparison between AtSpeed and baselines on MovieLens- 1M dataset. For each verification strategy, the best results are highlighted in bold and the second-best results are underlined.
MovieLens-IM
Verification
Method
Wse3
WSes
WS@10
WS
AS@3
AS@S
ASe 10
AS
DARE
1,00
1.00
SET
1.65
1.35
1.48
1.76
1.99
1.26
1.14
1.03
WordKD
1.29
1.29
1.39
1.69
1.07
1,02
1.00
1.00
Strict Top-XC
TVDKD
1.73
1,72
1.25
1.24
1.98
1.90
1.04
1,00
SeyKD
2.03
1.05
AtSpeed-S
1.86
1.79
1,80
1.75
2,08
2.03
1.98
1,09
AtSpeed-R
1.76
1.78
1.54
1.74
2.01
1.29
1,08
DARE
2.01
1.84
1.35
1.44
2.16
2.02
1.00
0.35
SET
2.08
2,03
2,02
1.61
2.28
2.20
2.06
0.93
WordKD
Relaxed Sampling
TVDKD
2.00
1.98
1.68
1.29
2.30
2.21
1.59
0.00
Se
2.13
2,08
1.93
1.56
2.19
2.14
2.01
0.78
AtSpeed-8
1,02
AtSpeed-R
2.22
2.14
1.64
2,44
2.15
0.95
Goodreads
Verification
Methed
W863
WSG5
WSG 10
WS6/20
AS63
AS@5
ASG 10
AS-020
1.00
SFT
1.83
1.81
2.17
2.46
2.04
1.98
1.72
1.07
WordKD
1.83
1.92
2.07
2.38
2,00
1.96
1,58
1,00
Strict Top-XC
SeqKD
1.82
1.89
2.19
2.48
2,00
1.96
1.73
1.08
AtSpeed-S
2.25
2.26
2.20
2.48
2.32
2.18
1.81
1488
AtSpeed-R
DARE
1.84
1.83
1.38
1.43
2.06
2.02
1.00
0.35
SET
2.15
2,09
1.70
1.91
2.27
2,08
1.01
0.10
WordKD
Relaxed Sampling
TVDKD
2.27
2:22
1.71
2.02
2.36
2.18
1.03
0.25
SegKD
1.96
1.96
1666
1.85
2.08
2.01
1,00
0.02
AtSpeed-8
AtSpeed-R
2.45
2.39
1.77
2.36
2.50
2.32
1.10
0.87
compared to the target LLM's top-K results (comparable performance on AtSpeed-S, AtSpeed-R and "Average" line), which is consistent with the results Table This also meets our expectations since the sampling based verification ensures the approximately equivalent distribution between the SD output and target LLM output under sampling-basee generation. We calculate the average over about how relaxed sampling verification affects the
all methods for comparison because recommendation accuracy. other words, baseline draft models are also expected to show limited if they are less aligned with the target L.LM and have relatively
ranking performance drop low speedup (e.g.. SFT on Beauty). 3) Compared to NDCG, the Recall under relaxed sampling verification usually achieves comparable reasonable since this work aims to align the top-K sequence distribution between the draft model and the target LLM. We emphasize the top-K drafted sequence be accepted with higher acceptance rate (i.e., a high recall of topK sequences), which does not explicitly require the draft model to distinguish the ranking between top-K sequences (potentially lead to relatively limited performances in terms of NDCG). Nonetheless, it is worth pursuing the non-trivial explicit probability ordering during alignment, which we consider leaving for further exploration in future work.
even better values than that of the target LLM. This is
Additional Results on MovieLens-IIM' and Goodreads Datasets 10, To evaluate the generalization ability of AtSpeed across different domains, we compare our proposed method with baselines on the MovieLens-IMM M and Goodreads datasets. The results are presented in Table 6. From the results, we can observe that 1) AtSpeed-S and AtSpeed-R outperforms baselines most cases under strict top-K verification and relaxed sampling verification, respectively. This validate the effectiveness of our proposed method on diverse datasets and is consistent with the observations on Amazon Beauty and Games (see Table 1). The relaxed sampling verification generally shows superior speedup compared to strict top-K K verification when K = 3, 5, while yields inferior speedup when is large (e.g., g.g K 20 on MovieLens- 1M and Goodreads). One possible reason that the item size is relatively small on the two datasets (3, 017 movies and 4, 667 books) compared to Beauty (12, 035 products) and Games (17,332 332 products), which might results in long-tailed draft distribution, where
Phttps://grone wwt.grooopeesooooo/raraaeoeeeeee htttee///./oooooooooo///stttttts "https: / www. wttp:/e..oom/maaeeeeeeee
24

---

Published as a conference paper at ICLR 2025
topK valid sequences have overwhelmingly high probabilities (i.e., q p), thus leading to a high rejection probabilities.
Speedup of Tree-based Attention. To analyze the speedup effect of tree-based attention, instantiate it on the standard LLM decoding with beam search, where each decoding step will utilize the tree-basee brseb attention in the L.M forward process. We compare the beam search with and without tree-based attention and report the results in Figure (a). As shown in the figure, we can find that the utilization of tree-based attention indeed exhibits an acceleration effect, corroborating the efficacy of our method. Furthermore, as the draft beam size N grows larger, the speedup of tree-based attention becomes higher. This enhancement attributed to the substantial reduction in repeated calculations of the same prefix shared across different beam sequences.
Waltinee Speedop Beauty
(WSg10)
Relaxed Sampling Verifivation (AS2) 10)
(WS222) Relaxed Sampling Verilivation (ASS220)
3.10
2.45
1.0
14
2.44
1/2
2.35
2,01
1.90
+ +SSg10 +Agg0
235
+ WWSg20+
1.50
10 20
0.7 0.9 1.0
0 0.1 0.3 0.5
1.0
(a) Effecct of bree- attention with different N
(b) Effect of for AtSpeed-R
(c) Effect of for AtSpred-R
Figure 5: (a) Speedup effect of tree-based attention on Beauty. (b) Performance of AtSpeed on (alignment strength a) on Beauty.
varying
Utilization of Beam Sequences from All Steps. Table 7 shows inference and training efficiency comparison between AtSpeed training with and without additional data generated from beam search results on diverse steps. From the table, it is observed that incorporating data from diverse generating steps usually enhances the speedup and accept steps, which is attributed to the additional alignment of top-K sequences across different steps. Nevertheless, using additional data could incur increased training time and storage cost. Specifically, utilizing a larger dataset necessitates greater storage capacity and extends the training duration. Despite these additional costs, the marginal improvement in performance is not substantial, leading to low cost-effectiveness ratio. Therefore, we should carefully consider the balance between the benefits of enhanced alignment against the increased training resource burden.
Table 7: Comparison between AtSpeed with and without additional training data regarding decoding efficiency, training time efficiency, and storage efficiency. "w/ AD" denotes AtSpeed trains the draft model with additional data generated from different steps.
Verification
Method
Ws@101
WS@20'
AS@101
AS@
|Time Cost (h).
Storagee Cost (GB).
Striet Top-K
AtSpeed-S
AtSpeed-S w/ MD
1.8710.00
1.87
1.66
1.84
1.60
1.64
0.6870008
0.57
4.10
2.56
28.09
14.68
Relaxed Sampling
AtSpeed-R
2.09
2.44
2.01
1.61
2.63
14.68
AtSpeed-R w/ MDD
2. 16700
2.42
2.0870.07
1.657000
3.00 7.00
28.09 718-41
Hyper-parameter Analysis. 1) Effect of alignment strength Figure 5(b-c) presents the results indicating the impact of alignment strength on AtSpeed-R. R. As the value of a increases, the overall trend of accept steps exhibits an upward trajectory from 0.1 0.9, which validates the necessity of alignment to improve the performance. Nevertheless, compared to AtSpeed-S (Figure 4(b-c)), we can find that only extremely large a (e.g., would hurt the performance of AtSpeed-R while setting to 0.7 for AtSpeed-S already cause a performance drop. We suspect the different behaviors are due to the different scales between Lingas and LAiig-RR Specifically, given the sequence distribution from draft model q and that from target LLM p, we have RKLD q log ! in AtSpeed-S and TVD
lg-2l in AtSpeed-R to align the two models. When sequence probability becomes small as the sequence length increases, 1 becomes very large. It is possible for AtSpeed-S to give larger loss value compared to AtSpeed-R (i.e., | Therefore, for the same a < 1, the strength that is too large for AtSpeed-S might still benefit alignment for AtSpeed-R.
25

---

Published as a conference paper at ICLR 2025
Striet Top-d Vertleatiou (AN60 (WAgg)
Striet Tay Verifentiee (S55 (aaas Ridrsed Sumplin Variivatio (our))
(miion Relaced Sampping Vectitration
(ogd)
+W5g
200
Alggs
AS33
3M
579
13
532
180
135
100
185
A
Alipodd-S
AlSpeed-R
Acdpaed-R
Top-K (45030)
(1522
Sumpling (oen (WSg2)) Sampling
0.4
2.55
1.50
319
020
+WNgg +-AggI
E.flet Acipeed--
ofy AcSppedd
Rfffet Allipeed-
Kfffct ofy Atspecd--
Figure 6: Performance of AtSpeed with different draft length y Beauty.
Prefilling
= Decoding
Prefilling Decoding
3.%
67%
91%
9%
Transfrrmers
Transformers
A1Sipeed 20
AtSpeed (1-10)
A1Spted
A15peed
Figure 7: Comparison of the time cost ratio between the prefilling and decoding. The time cost of prefilling is the same across different methods.
2) Effect of draft length 7, Figure 6 presents the impact of varying Y on the performance of AtSpeed. Despite acceptance generally increases with 7, speedup does not consistently improve with 7. The trend of speedup in relation to is influenced by draft beam size parameter N. Specifically, when N is small, speedup tends to increase with ",, conversely, when N large, speedup tends to decrease with y. This variation is primarily attributed the computational overhead during the draft phase, which escalates with larger beam size, thereby diminishing the acceleration benefits. The key determinant of the draft phase's 's computational cost is the constraint search process.
Case Study. As shown in Figure 7, our proposed methodology achieves significant reduction in decoding time. Since the prefilling time is the same between target LLM with and without SD, AtSpeed boosts the LLM decoding by reducing the percentage of decoding by 58% (N=1), 37% (N=10), and 23% (N=20).
26


Reviews:
[{"review_id": "EUpBByknPs", "reviewer": "Reviewer_y4np", "summary": "This paper focuses on accelerating inference in LLM-based generative recommendation using speculative decoding. It highlights the challenges of applying speculative decoding directly to the generative recommendation, due to the N-to-K issue. The authors introduce two improvements in both the drafting and verification stages. Experiments on two public datasets demonstrate the efficiency of the proposed method.", "questions": "Please refer to \"Weaknesses\" for more details.\n\nLastly, I want to share a related paper, \"Inductive Generative Recommendation via Retrieval-based Speculation\". This paper was released after the ICLR submission deadline and is currently available only as a preprint. While this is not a critique or question, it’s relevant as it discusses speculative decoding in generative recommendation. It proposes a dynamic N for the draft model and introduces a technique to perform limited beam search, using prefixes generated by the first few steps of the target model to guide the draft model. I believe this work shares some conceptual similarities with this paper, so I thought it worth mentioning.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. A timely study addressing inference efficiency in generative recommendation.\n2. The ideas of (1) fine-tuning the draft model to align with top-K items of the target model and (2) adding a probability to accept rejected items in a more flexible verification setting are both novel and interesting.\n3. The paper is well-written and easy to follow.\n4. Experiments on two public datasets demonstrate the efficiency of the proposed method.\n5. Code is available during the review phase, enhancing reproducibility.", "weaknesses": "1. Performance metrics (e.g., NDCG and Recall) are not well-presented. Only Table 2 includes some ranking metrics (also only Recall, without NDCG), which may raise doubts about the proposed method's ranking performance. Including these metrics in Table 1 and Figure 3 would strengthen the results. The limited metrics reported make it challenging to fully assess AtSpeed's ranking performance.\n2. Presentation issues. Reporting WS@K and AS@K for all values of K in {1, 3, 5, 10, 20} in Table 1 seems unnecessary, especially since the discussion focuses on average WS and AS. I suggest presenting only representative Ks alongside the averages, freeing space for additional ranking metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6tS8L0hFzv", "reviewer": "Reviewer_WeyY", "summary": "This paper proposes AtSpeed, a framework to accelerate LLM-based generative recommendation systems through speculative decoding. The main challenge addressed is the inherent inefficiency of autoregressive beam search in generating top-K recommendations. The authors identify that traditional SD methods, which focus on N-to-1 verification, are insufficient for recommendation tasks that require N-to-K verification to output a ranked list of K items. To tackle this, the paper introduces two methods: AtSpeed-S for strict top-K alignment and AtSpeed-R for relaxed sampling verification, which aims to reduce the number of LLM calls while preserving accuracy. Experimental results demonstrate that AtSpeed achieves significant speedups (up to 2.5×) with minimal degradation in recommendation accuracy.", "questions": "- [Q1] The paper primarily compares AtSpeed with KD-based baselines. Existing SD-based baselines should also be compared including the paper I mentioned before (A Decoding Acceleration Framework for Industrial Deployable LLM-based Recommender Systems). \n- [Q2] Have the authors tested their framework on any larger and more diverse datasets, such as MovieLens or more complex datasets like Goodreads? If not, can the authors comment on the expected performance and generalization ability of AtSpeed in these contexts?\n- [Q3] The paper mentions the use of LLaMA-7B, but it would be interesting to know how AtSpeed scales with even larger models (e.g., LLaMA-13B or GPT-3). Do the authors anticipate any bottlenecks or limitations when scaling to larger models, especially concerning memory usage and GPU efficiency?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper extends the N-to-1 verification to the N-to-K verification in the context of generative recommendation, which fits the setting of real-world recommender systems.\n- The authors provide a solid theoretical foundation for AtSpeed to align the draft and target models. The optimization objectives are clearly motivated and mathematically sound.\n- Empirical results are compelling, showing that AtSpeed can achieve up to a 2.5× speedup while maintaining competitive recommendation accuracy. The results are well presented and highlight the practical utility of the proposed method.", "weaknesses": "- The paper may overclaim their contribution to be the first to propose the speculative decoding task for LLM-based recommender acceleration. There already exists prior work on speculative decoding for LLM-based recommendation: *A Decoding Acceleration Framework for Industrial Deployable LLM-based Recommender Systems (Xi, Yunjia, et al)*, which needs to be compared and discussed.\n- The experiments are somewhat limited to two datasets (Amazon Beauty and Games). While these datasets are commonly used, the paper would benefit from broader validation across additional domains or larger-scale datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cNh00Ucos7", "reviewer": "Reviewer_Up12", "summary": "This paper introduces a speculative decoding approach for generative recommendation. In traditional speculative decoding, a cheap draft LLM is given a prefix and at each decoding step it suggests $N$ tokens among which the token predicted by the target LLM should be. This enables the target model to verify all the drafted predictions in parallel through a single call, conditioning on the draft model's previous predictions. The sequence of drafted predictions is then accepted up to the decoding step where the draft model and the target model disagree in their predictions.\n\nIn top-$K$ generative recommendation, a beam search is performed to generate the $K$ item identifiers. Applying speculative decoding to this problem then requires the $N$ tokens suggested by the draft model to contain the $K$ tokens predicted by the target model to accept the current decoding step. This condition is hard to fulfill in practice. To address this, the paper defines a framework named AtSpeed, which trains the draft model to be more aligned with the target model and optionally relaxes the acceptance condition to allow the $N$ drafted tokens to contain tokens that are similar (rather than identical) to the $K$ target tokens. AtSpeed was validated through experiments on the Amazon Beauty and Games datasets.", "questions": "- In Section 3.1, paragraph \"Acceptance Rate\", the condition given for having $\\beta = 1$, namely that $p(\\mathbf{y}) \\geq p(\\mathbf{y}_K)$ for all $\\mathbf{y}$ in $\\mathcal{Y}_q$, doesn't seem to be fulfillable. By definition of $\\mathbf{y}_K$, there exists only $K-1$ sequences with greater probability according to $p$, yet the condition requires that all $N$ ($\\geq K$) sequences in $\\mathcal{Y}_q$ have a probability greater than $\\mathbf{y}_K$. The only possibility for this condition to be realized is if $\\mathcal{Y}_q$ contains duplicates, which should not happen in the case of beam search.\n- Derivation of Eq 15 in App A.2 (which leads to the definition of the alignment objective in Eq 3) seems incorrect or at least misses important steps to be understandable for the reader. More steps or explanations are needed.\n- Eq 3 introduces $\\mathcal{D'}$ in which $\\mathcal{Y}$ is the top-$K$ of a mixture of $q$ and $p$. Following Gu et al (2024) is mentioned as the motivation for this, but it would be appreciated to have more intuitions on this choice. Moreover, the $\\mathcal{D'}$ used later in the relaxed alignment objective (Eq 8) is different. What is the rationale to have different $\\mathcal{D'}$ in the strict and relaxed objectives?\n- What loss is used in practice for $\\mathcal{L}_{Rec}$? It is only mentioned to be a recommendation loss but additional details would be appreciated (at least in the appendix).\n- The relaxation sampling verification strategy misses some intuition: why is $p(\\mathbf{y}) \\geq q(\\mathbf{y})$ the right criterion for accepting a sequence $\\mathbf{y}$? Why are such $\\mathbf{y}$'s good candidates? Moreover, in the case of rejection, $\\mathbf{y}$ is drawn from $p' = norm(max(0, p(\\mathbf{y}) - q(\\mathbf{y})))$ but it is unclear whether this is an actual distribution and what the norm operator exactly consists of. Is $p'$ denoting the uniform distribution over the $\\mathbf{y}$'s such that $p(\\mathbf{y}) \\geq q(\\mathbf{y})$? If so, how does one sample from this in practice? The definition of this distribution would deserve more clarifications and details.\n- The \"with-replacement sampling approximation\" paragraph in App A.2 is difficult to follow so it would benefit from being reworked.\n- The proof of Lemma 1 in App A.2, which corresponds to Lemma 3.3 from Leviathan et al (2023), is missing the step with the term $1 - \\sum_{\\mathbf{y}} \\frac{p(\\mathbf{y})+q(\\mathbf{y})-|p(\\mathbf{y})-q(\\mathbf{y})|}{2}$.\n- In Eq 7, what does $\\sum_K$ denote? It seems like there is an index variable missing there.\n- The strategy relying on tree-based attention to speed up inference would benefit from being described in more details (at least in Appendix). Section 3.3 only refers to Figure 2(c) to describe the strategy, but this figure is not self-explanatory.\n-  The WS metric represents the walltime speedup, but with respect to which baseline? I assume this is in comparison to directly running the target model without speculative decoding, but it would be helpful for the reader to mention this when defining the metric.\n- Table 1 reports the results for AtSpeed-S and AtSpeed-R on both the strict and the relaxed settings. How can AtSpeed-S be applied to the relaxed setting and AtSpeed-R to the strict setting?\n- In Figure 5 of the Appendix, it seems that a larger value for $\\alpha$ is always beneficial for AtSpeed-R, whereas Figure 3 (c) showed that $\\alpha$ should neither be too small nor too large for AtSpeed-S. Are there any intuitions on these different behaviors between AtSpeed-R and AtSpeed-S?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The approach proposed for applying speculative decoding to generative recommendation is both novel and intuitive.\n- The experiments are comprehensive and provide convincing evidence of the benefits of AtSpeed from an empirical standpoint.\n- The source code was made available.", "weaknesses": "- The paper suffers from an overall lack of mathematical rigor, both in the proofs and notations, which raises concerns about the theoretical soundness of the approach. In particular, some key formulas, such as the proposed alignment objective, appear to be either incorrect or lack sufficient derivation to validate their correctness (see specific points in the 'Questions' section).\n- The paper is overall lacking polish and contains multiple typos in mathematical formulas, which makes it hard for the reader to follow the details of the approach. Thorough proofreading would be needed to improve the paper's readability. Certain parts of the paper could also benefit from being further clarified, as pointed out in the 'Questions' section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)