SLMRec: Distilling Large Language Models into Small for Sequential Recommendation
Authors: Wujiang Xu
Qitian Wu
Zujie Liang
Jiaojiao Han
Xuying Ning
Yunxiao Shi
Wenfang Lin
Yongfeng Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models; Knowledge Distillation
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: SLMs could perform better equipped with knowledge distillation than LLMs in recommendation task.

Abstract: Sequential Recommendation (SR) task involves predicting the next item a user is likely to interact with, given their past interactions. 
The SR models examine the sequence of a user's actions to discern more complex behavioral patterns and temporal dynamics. 
Recent research demonstrates the great impact of LLMs on sequential recommendation systems, either viewing sequential recommendation as language modeling or serving as the backbone for user representation. Although these methods deliver outstanding performance, there is scant evidence of the necessity of a large language model and how large the language model is needed, especially in the sequential recommendation scene. Meanwhile, due to the huge size of LLMs, it is inefficient and impractical to apply a LLM-based model in real-world platforms that often need to process billions of traffic logs daily. In this paper, we explore the influence of LLMs' depth by conducting extensive experiments on large-scale industry datasets. Surprisingly, our motivational experiments reveal that most intermediate layers of LLMs are redundant, indicating that pruning the remaining layers can still maintain strong performance.
Motivated by this insight, we empower small language models for SR, namely SLMRec, which adopt a simple yet effective knowledge distillation method. Moreover, SLMRec is orthogonal to other post-training efficiency techniques, such as quantization and pruning, so that they can be leveraged in combination. Comprehensive experimental results illustrate that the proposed SLMRec model attains the best performance using only 13\% of the parameters found in LLM-based recommendation models while simultaneously achieving up to 6.6x and 8.0x speedups in training and inference time costs, respectively. Besides, we provide a theoretical justification for why small language models can perform comparably to large language models in SR.

Published as a conference paper at ICLR 2025
SLMREC: DISTILLING LARGE LANGUAGE MODELS INTO SMALL FOR SEQUENTIAL RECOMMENDATION
Wujiang Xu', Qitian Wu?, Zujie Liang', Jiaojiao Han",
Xuying Ning.. Yunxiao Shis, Wenfang Lin", Yongfeng Zhang
Rutgers University 2 Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard
3 Ant Group Dian Diagnostics Group Co.
5 University of Illinois thhama-Champaign
University of Technology Sydney
ABSTRACT
Sequential Recommendation (SR) task involves predicting the next item user is likely to interact with, given their past interactions. The SR models examine the sequence of a user's actions to discern more complex behavioral patterns and temporal dynamics. Recent research demonstrates the great impact of LLMs on sequential recommendation systems, either viewing sequential recommendation as language modeling serving as the backbone for user representation. Although these methods deliver outstanding performance, there is scant evidence of the necessity of a large language model and how large the language model is needed, especially in the sequential recommendation scene. Meanwhile, due to the huge size of LLMs, it is inefficient and impractical apply a LLL--based model in real- world platforms that often need to process billions of traffic logs daily. In this paper, we explore the influence LLMs' depth by conducting extensive experiments on large-scale industry datasets. Surprisingly. our motivational experiments reveal that most intermediate layers of LLMS are redundant, indicating that pruning the remaining layers can still maintain strong performance. Motivated by this insight, we empower small language models for SR, namely SLMREC, which adopt simple yet effective knowledge distillation method. Moreover, SLMREC is orthogonal to other post-training efficiency techniques, such as quantization and pruning, that they can be leveraged in combination. Comprehensive experimental results illustrate that the proposed SLMREC model attains the best performance using only 13% of the parameters found in I.LM-based recommendation models, while simultaneously achieving up to 6.6x and 8.0x speedups in training and inference time costs, respectively. Besides, we provide a theoretical justification for why small language models can perform comparably to large language models in SR. The source code and datasets are available the URL
INTRODUCTION
Learning temporal interest information fundamental for sequential recommendation models. Tra- ditional sequential recommendation (TSR) methods (Wu et al., 2017; Hidasi et al., 2015; Kang & McAuley, 2018; Sun et al., 2019) focus on the development of intricate sequential encoders, evolving from LSTM and GRU architectures to the self-attention layers and Transformer models. However, the state-of-the-an performance in TSR has hit a plateau, limited by model sizes that usually feature fewer than 0.1 billion parameters.
Recently, Large Language Models (LLMs) (Achiam et al., 2023; Touvron et al., 2023; Anil et al., 2023) have made significant advancements in various aspects by scaling the size of the training data or the model's architecture. Building upon the scaling laws delineated in prior research (Kaplan et al., 2020; Hoffmann al., 2022), it endows LLMs with enhanced expressivity, culminating in superior performance benchmarks. Naturally, burgeoning trend among contemporary LLM- based recommendation architectures has raised concerns. The current LLM-based recommender system
"Corresponding author.
'https:/ //gi thub. cwm/ wtw.u/iingnn//uismmm

---

Published as a conference paper at ICLR 2025
g O, h4
F 12O h
+
hy
Next Token Prediction
Adapter2
Seq Encoder
LLM LORA
Adapter1
h2
Prompt
Embedding Encoder
Mhit rocermimeee
ascacaed sula (iilaay 2
Embeddingg Encoder
Input tokens
Candidate item
Input tokens
Candidate item
(a) TSR methods
(b) G-LLMRee, E-LLMRec methods
Figure 1: This overview compares traditional sequential recommendation (TSR) methods with LLM-based recommendation (LLMRec) methods. Here, h2 and h; represent the user and item representations, respectively. In contrast G-LLMRec methods, E-LLMRec approaches adhere to the TSR prediction framework. These methods leverage LLMs as feature extractors in the manner of BERT, diverging from the generative focus of G-LLMRec.
can be classified as 1) generation-based approaches, e.g., (Geng et al., 2022; Xu et al., 2023a), CoLLM (Zhang al., 2023b) and LLaRa (Liao et al., 2023); 2) embedding- approaches such as E4SRec (Li et al., 2023a), CLLM4Rec (Zhu et al., 2023) and Lite-LLM4Rec (Wang et al., 2024). As shown in Fig. 1, innecration-bscle approaches (G-LLMRec) encode an item token and formulate the sequential recommendation as the next token prediction task. By contrast, embedding- based approaches (E-LLMRec) regard the last hidden representation external adapter to compute user-item preference. The adoption of LLMs has vastly driven the development of sequence recommendation tasks, bringing an improvement of nearly 20% against the TSR model on the benchmark (Li et al., 2023a; Liao et al., 2023; Wang et al., 2024). This arouses the following research motivation for this work.
representation and learn an
Some researchers (Ardalani et al., 2022; Zhang et al., 2023a; 2024) have attempted to investigate the scaling laws in the recommendation domain. However, the largest model examined in these studies is less than 1 billion parameters, significantly smaller than the 175 billion parameters of GPT-3 (Brown al., 2020). Additionally, the focus has been primarily on test loss rather than on ranking- evaluation metrics, which limits the practical applicability of their findings. Recent studies (Liang al., 2023; Gromov et al., 2024; Men al., 2024) on the NL. domain suggest a high degree of redundancy in the LLMs model architecture. Since the ID information of the recommendation domain has not been explicitly learned during the LLMs' training process, we also aim to find out whether increasing the model size of LLMs is beneficial for the SR task.
Despite the large performance gain, the LLMRec methods also escalate the model size significantly, e.g., nearly 70 times greater parameters compared with TSR models (from O.IB to 7B+). Even within the parameter-efficient training technique (Hu et al., 2021a), the paradigm still poses a significant challenge for real-world sequential recommendation use cases, where billions of traffic logs every day and potential new items need to be processed constantly. This disparity imposes strict hardware demands and makes it both inefficient and infeasible to deploy the LLMRec model.
Our contributions. This paper presents an initial attempt to reassess the need for LLMs in sequential recommendation. To explore the reasons for the significant improvement of LLMRec methods, we conduct a series of experiments on surge-scale industry datasets to investigate the effects of reducing the number of parameters during the training and inference stages on overall performance. From the empirical results, we found some profound insights that the improvement of the rise of the model parameters is not consistent. Meanwhile, it reveals that some layers of L.MSs are redundant in the recommendation task, similar to findings in NLP domains (Men et al., 2024; Gromov et al., 2024).
Motivated by these findings, we empower small language models for the sequential recommendation. named SLMREC. We adopt the vanilla knowledge distillation approaches align the representation knowledge. Moreover, multiple supervision signals are crafted to steer the student model toward acquiring task- knowledge within its hidden representations. Additionally, our model oper- ates without the need for any supplementary model design elements and compatible with other quantization and pruning techniques utilized within L.Mss.
2

---

Published as a conference paper at ICLR 2025
0.22
Liilen Skgle
0.18
0.176
Euibe 1833
0.30
0.16
0,18
0.150
0.14
0.16
0.125
0.12
0.10
0.100
0.18
0.075
24
30
36
Layer Namber
Nomber
Nummber
(a) Cloth (Infer) - HR @ 10
(b) Cloth (Infer) NDCGG@ 10
(c) Cloth (Infer) MRR
Lodder aduuta
1oik 2012
0.315
0.16
0.210
0,305
18
50
10 20
30
10
20
30
Layer
Layer
Layer Nunther
(d) Cloth (Train)- HR@ 10
(e) Cloth (Train) NDD0@@D 10
(f) Cloth (Train)- MRR
Figure 2: We present the relationship between the number of decoder layers and the final recommen- dation performance, with the performance of SASRec plotted as a baseline. Figures (a)-(c) show the results of directly using representations from the middle layers for inference without training, while (f)-(f) prune the later layers and train model using only the specified number of layers. From the results, we observe that deeper decoder layers introduce redundancy in recommendation tasks, with models utilizing fewer layers (8-layer) achieving performance nearly equivalent to (24-layer) models.
Extensive experiments have revealed that SLMRec, with model size of less than billion parameters, can deliver performance that is remarkably competitive with baselines using LLMs sized over 7 billion parameters. Furthermore, SLMRee achieves up to 6.6x/8.0x speedup in terms of training/in- ference time costs against LLM- based recommendation models. Besides, we present the results of SLMRec employing online knowledge distillation, demonstrating its competitive performance. Beyond empricial experiment results, we provide a theoretical justification for why small language models can perform comparably to large language models in SR.
2 MOTIVATIONAL EXPERIMENTS
As described above, here we try to explore the effectiveness of LLMs in recommendation via decreasing the parameters of popular LLMs (i.e., LLaMa-7B) and observe the change in performance.
Evaluation Protocol. In the motivational experiment, we select SASRec as a traditional sequential recommendation baseline due to its performance (Klenitskiy & Vasilev, 2023). We adopt embedding- based method (Li et al., 2023a) as the baseline, named E4SRec, to easily generate the ranking for the full/sampled list of items. As shown in Fig. 2, pre-trained embedding layer learned from SASRec used to obtain the sequential item embedding. Then we concatenate the item embeddings with the prompt embeddings obtained after the tokenization. After encoding of stacked attention blocks of LLM, we regard the representation
the last layers as the user representation. Then, we follow the TSR methods to calculate the inner product of user embeddings and item embeddings from the pre-trained embedding layer to serve as the score for the user-item pair. Also, cross cross-cntropp entropy loss and fully candidate item are utilized for the optimization achieve best results (Xu et al., 2024a; Petrov & Macdonald, 2023). To reduce both computational demands and processing time, LoRA (Hu et al., 2021a) is used to update a comparatively smaller set of parameters. Besides, generate an unbiased evaluation for fair comparison (Krichene & Rendle, 2020; Zhao et al., 2020), we randomly sampled 999 negative items, which were items not interacted with by the user, along with I positive item that served as the ground-truth interaction. To obtain large-scale industry data, we use the Amazon 18 version' dataset in this paper. More details are shown in Section 5.
2To accelerate and align with the prediction head traditional SR methods, we remove the original softmax layer and instead use the dot product of the user and item representations to compute the prediction score. "htps:/mijianmoo.. wwt.tb..tttthaineeeeee wwwamazon/iddx..nieeo html
3

---

Published as a conference paper at ICLR 2025
Adapter
O
Paue Parcre
Diitilation
LLM (LLaMa)
CLM (LLaMa)
b-th Block $
b-th Block LoRA
1-th Block
/-th Block
LoRA
A5
Isl Block
Duiilbation
1st Block
LoRA A
Adapter
Pronut auto inarcca
Proans nute cevernmmnnt
Adapter
ID Embedding +
Pretrained ID Embedding Layer
S $
I -..
Sequence IDs
Sequence IDs
Candidate item
Teacher Model
Student Model
Figure 3: The overview of SIMREC. layer-wise knowledge distillation approach is applied to align the representation knowledge by grouping the layer into serveral blocks. The teacher and student model share similar E-LLMRec model architecture. Multiple supervision signals are introduced to steer the student model toward acquiring fine- fine-grained task-aware knowledge.
Evaluation Strategy. To examine the connection between the number of parameters and the performance of LLM-based methods (E4SRec), have truncated the original LLM architecture-in this case, 32-layer decoder from the LLaMa 7B model-by by pruning the decoder layers during both the inference and the training stages. As direct inference method, we refrain from additional training using new labels and instead directly employ the output from the final ten layers as user representations gauge recommendation performance. Instead of direct inference, we focus on conserving the initial layers of the decoder and proceed to train more lightweight E4SRec model while adhering to the original training protocol. The models resulting from varying levels of layer retention are designated as E4SRec,, with the variable / indicating the number of layers retained. The chosen values of encompass a spectrum, specifically {1, 2, 4, 8, 16, 24, 32). Results from both experimental approaches graphically depicted in Figure 2, providing insight into how the models' depth influences their recommendation capabilities.
Insights. From Figure 2 (a)-(b), we can observe that directly utilizing the representation of other layers without training cannot obtain a comparative performance. Compared to TSR baseline SASRec, Figure 2 (c)-(d) yield the following insightful findings: (1) As the number of layers increases, the performance of the model also improves. Furthermore, even when the model has the same layer number (i.e., (=2) as SASRee, its performance still superior to that of SASRec. We assume the gains observed in LLM- methods could likely be attributed to the larger hidden representation size ((i.e, 4096 V.S. 128), the initialization from LLMs, and the introduction of PEFT (Hu al., 2021a). (2) When is set ranging from 8-24, the model's 's improvement is slight. It reveals that an 8-layer E4SRecs can obtain nearly as informative user representations as 24-layer E4SRec24- Considering the two findings above, it naturally inspires to explore better training methods to obtain a smaller-size LLM-based SR model that is comparable with large models. If we want to learn E4SRec, that perform similar E4SRec (M < N), we should make sure the intermediate representations in E4SRec to be as closer to those E4SRec as possible. Knowledge distillation (KD) is straightforward idea in this case. Thus, we design simple yet effective knowledge distillation method to train a tiny LL.M-based model with similar performance. For the motivation experiment results in Movie domain, it be found in Appendix B.I. In Section 6, provide theoretical justification that aligns with these empirical insights.
PRELIMINARIES
In this study, rather than constructing complex additional structures, we slightly modify existing E-LLMRec methods for our purposes. Initially, we delineate the E-LLMRec model that we employ for sequential recommendation tasks.
4

---

Published as a conference paper at ICLR 2025
Model structure. The E-LLMRec models capitalize such as BERT4Rec, SASRec, and GRU4Rec, which is pre-trained on a designated dataset (Sun et al., 2019; Kang & McAuley, 2018; Hidasi et al., 2015). The objective of sequential recommendation is to forecast subsequent items utilizing the user action sequence = (11, i2..... ir), a sequence that is either truncated or padded maintain uniform length. Through truncation and padding, we derive the user's s action sequence mask, serving as the attention mask in LLMs (Large Language Models). The fixed-length sequence S IR" translated into sequential representation S E 22'xxo pre-trained ID embedding layer. A linear transformation is then applied to upscale the representation from a lower dimension do to a higher dimension d1 suitable for the hidden layers within the LLMs.
ID embedding layer from TSR models
Xdo via the
Upon defining the prompt template, the tokenization layer within the LLMs processes the natural language input into corresponding text embeddings and their associated attention masks. These embeddings and attention masks, derived from both the II) sequence and the text, are then introduced into the LLM decoder. The final temporal output has from the last layer of the decoder is inferred as the user representation and subsequently mapped through linear layer to condense the dimensionality from d1 back to dg- Finally, user-item interaction predictions p are inferred by executing a dot product between the user and item representations. The learning process of the model is refined through the application of a entss-yntropy loss.
ePi
Loo
=-
Σ WED
e
jEl
where U and denote the whole user set and item set. Yus denotes user-item interaction label.
Knowledge Distillation. Knowledge distillation is a technique aimed transferring knowledge from a sophisticated teacher model a more streamlined student model (Hinton et al., 2015). We represent the teacher by fz f(x)) and the student by fs (O,). We aim to solve the following optimization problem:
min(Cce DRa (O,, 0.)].
Here, (O,, O.) signifies the knowledge distillation loss, which quantifies the discrepancies between the teacher and the student models. prevalent method involves employing the KL divergence to evaluate the divergence between the logits produced by both models. One well- established training schema known as offline distillation, wherein the teacher is fully trained beforehand and remains unchanged. while the student is refined based on the criteria outlined in Eq. the offline knowledge distillation manner, the teacher model O is initially trained in a designated training set by minimizing the cross-entropy loss Lec-
4 SLMREC
In this work, do not adopt logits- based knowledge distillation, our goal is for the student model to learn how to encode hidden representations similar to the teacher model, rather than merely replicating its predictions. To achieve this, we perform feature distillation across multiple layers. Specifically, considering that the teacher model consists of M stacked decoder layers and the student model has N stacked decoder layers, we design several feature regularizers to guide the distillation process at regular intervals between the hidden representations of both models. We divide the layers of the teacher and student models into blocks by grouping every m layers of the teacher and every layers of the student. The number of resulting blocks is B, calculated as B = M = I Let the hidden representations from the teacher model be denoted as: H, {h," ..... h ). where h," represents the final temporal dimension of the hidden representation from the m--th layer of the teacher. Similarly, the hidden representations from the student model are denoted as: H, (h..... h In this study, we use a deeper LLM as the teacher model and a shallower LLM as the student model, both sharing the same hidden dimension d, such that H., H. R Bxd
Feature Similarity, To regulate the alignment of feature directions between the teacher and student models, employ a cosine similarity- based loss term. Formally, it is described by the equation:
5

---

Published as a conference paper at ICLR 2025
1 B
h
(km)
h;
(km)
Doos pr O.)
(km))
||2
(km))
11
Feature Norm Regularization. In addition, introduce straightforward regularization term designed to minimize the L.2 distance between the hidden representations of the teacher and student models. is mathematically formulated as:
norm (0,,
(km)) h h. (km)) 12
Multiple Supervision. Furthermore, we employ multiple supervision strategies to steer the student model toward assimilating specific aspects of recommendation--relttdd knowledge. For each represen- tation, we learn additional adapters Wa to reduce the dimension. The modified prediction ( (f ) can be acquired as described by Eq. 1:
(km)
B-1 Σ
Cms ( W.
(5)
Total Loss. Integrating the aforementioned distillation losses, the composite objective function for training the student model is given by:
min [Cee (9.) + A1 Doos (9,, +,)) + >2 (0,, θ,) + AgLms(9,.) Wa)). @,
(6)
where Å1, A2 and λ are hyperparameters that control the contribution of each term.
EXPERIMENTS
In this section, we present extensive experiments demonstrate the effectiveness SLMREC, aiming to answer the following four research questions (RQs).
RQ1: How does the performance of our proposed SLMREC model compare LLM-based recommendation models when evaluated on large-scale industry dataset?
RQ2: What the comparative efficiency and runtime of our SLMREC model against the G-
LLMRec and E-LLMRec models?
RQ3: Whether the proposed three knowledge regularizers work?
RQ4: Is it feasible to train our model, SLMREC, simultaneously with an untrained teacher model?
5.1
EXPERIMENT SETUP
For our experimental evaluation, we utilize data from the clothing, movies. music, and sports categories within the extensive, industry-scall Amazon18 18 dataset Statistics of the datasets shown in Table 1. In all datasets, we interpret any rating above 3 as positive feedback, indicat- ing user interaction with the item, and employ timestamps establish the chronological order of actions. We eliminate users and items that have fewer than 5 associated actions ensure sufficient data density. The historical sequence of interactions for each user divided into three segments: (1) the most recent interaction is reserved
Table 1: Statistics of the Amazon datasets. U [V), and [E] denote the number of users, items, and ratings, respectively.
Dataset
[[4]
Density
Cloth Movie
1.219,678
376,858
11,285,464
0.002%
297,529
60,175
3,410,019
0.019%
Music
112,395
73,713
1,443,755
0.017%
Sport
332,447
12,314
146,639
0.008%
for testing, (2) the second most recent for validation, and (3) all preceding interactions used
for training. Based on the ranking results, we utilize the typical top-N metrics rate (HR II,
5, 10)), normalized discounted cumulative gain (NDCG@ @ (5,10)) (Järvelin & Kekäläinen, 2002)
hhttps: //ni jianmo. .ero.ggittn git.ub..hh.... / aaaaaa/a//////aaaaa index. htm1

---

Published as a conference paper at ICLR 2025
Table 2: Experimental results (%) on the Cloth and Movie dataset. The missing MRR value of Open-P5 is unavailable due to the time complexity constrictions. The number on the left of the arrow the layers N of the student model. The left number on the right of the arrow is the layers M of the teacher model. For Open-P5, we adopt LLaMa as their backbone. We highlight the methods with the best and second-best average performances. We also give the average ranking of each evaluation metric. Moreover, E4SRec which has the same number of layers our SLMREC, is also marked.
Model
Cloth
Movie
Rank
HR@1
HR@5
NDCG@5
MRR
HR@1
HR@5
NDCG@5
MRR
Caser
9.66
15.18
12.66
13.03
4.27
14.96
9.57
10.36
13.50
GRU4Rec
13.79
15,46
14.64
15.15
10.56
19.47
15.11
15,46
9.25
BERT4Rec
13.60
14,66
14.14
14.59
9.68
14.91
12.40
12.74
11.63
SASRec
13.08
16.94
15.01
15.76
5.57
16.80
11.17
12.08
11.63
HGN
15.96
18.70
17.30
18.27
7,54
19.20
13:42
14.73
6.50
LightSANs
14.12
20.32
17.30
16.86
17.54
11.81
12.82
8.00
-9-Rec
14,10
18.67
16.10
16.95
7.75
20.39
15.69
14.34
7,50
DuoRec
13.06
18.29
15.79
15.42
10.07
20.37
17.96
16.61
7.88
MAERec
13.29
18.35
15.68
16.13
8.89
20.24
16.03
15.28
8.38
Open-P5
14.13
17.68
17.02
12.66
21.98
17.13
5.67
E4SRec
16.71
19,45
18.09
18.77
14.74
23.79
19.45
19.74
1.75
E4SRo0s
15.30
18.54
16.91
17.60
13.32
22.49
17.99
18.46
4.00
E4SRoc4
14.58
18.05
16.32
17.01
11.80
21.54
16.73
17,20
5.75
SLMRect-s
16.69
19.47
18.07
18.74
15.29
24.25
19.90
20.36
1.50
and Mean Reciprocal Rank (MRR) (Sarwar et al., 2001) to evaluate the model performance. For all the metrics, higher values indicate better performance. Models that achieve the highest MRR performance on the validation set, including ours and other baseline models, will be preserved for subsequent performance evaluation on the test set. In order to ensure an unbiased evaluation, we adopt the methodology employed in previous works (Krichene & Rendle, 2020; Zhao et al., 2020), wherein we randomly select 999 negative items (i.e., items that the user has not interacted with) and combine them with positive item (i.e., a ground truth interaction) to form our recommendation candidates for the ranking test. Detailed hyperparameters of our model in each dataset are Appendix B.2.
5.2 PERFORMANCE COMPARISONS
Compared Methods. We compare method with three classes of baselines: (1) Traditional sequential recommendation methods. i.e., GRU4Rec (Hidasi et al., 2015), Caser (Tang & Wang, 2018), HGN (Ma al., 2019). BERT4Ree (Sun et al., 2019), SASRec (Kang & McAuley, 2018) and LightSANs (Fan et al., 2021). (2) Self-supervised sequential recommendation methods, i.e., Ree (Zhou et al., 2020), DuoRec (Qiu al., 2022) and MAERec (Ye al., 2023). (3) G-LLMMec method: Open-P5.LLAMe (Xu et al., 2023a). (4) E-LLMRec method: E4SRec (Li et al., 2023a). A detailed introduction to these baselines can be found in Appendix B.3. It should be noted that we did not select various G-LLMRec methods or E-LLMRec methods as baselines. This is because the differences between each LLM- method are minimal, and our model is a universal approach that is not confined to specific model type. Our primary focus is to improve the efficiency of language model utilization. Hence, we opted to select one G-LLMRec method (Open-P5) and one E-LLMRec method (E4SRec) baselines.
Quantitative Results (RQ1). Tables 2-3 showcase the quantitative comparison of four large-scale sequential recommendation datasets. From our analysis, we have several insightful observations: (1) LLM-based recommendation methods exhibit substantial improvements over traditional sequential recommendation (TSR) methods, primarily due to their enhanced modeling capacity which adeptly extracts informative sequential interest patterns. (2) Our model, SLMRec outperforms the teacher model E4SReca by leveraging knowledge distillation within the hidden layers. By refraining from applying this constraint prior to the prediction phase, we enable the final representation to organically gravitate towards the label leiellyyeliin an approximate 8% enhancement in performance in comparison to the teacher model. (3) Introducing vanilla knowledge distillation techniques into LLMRee, without altering the model structure, allows SLMRocat-s to achieve a marginally superior performance compared to E4SRecy2. This suggests that small language models equipped with
For Open-P5, we adopt the version of L.LaMa as the foundation model in their code repository implementa- tion ensure the best results achieved.
7

---

Published as a conference paper at ICLR 2025
Table 3: Experimental results (%) on the Music and Sport dataset.
Music
Sport
Model
Rank
HR@1
IR@@
NDCG@5
MRR
IRR@1 @1
IR@5
NDCG@s
MRR
Caser
0.71
3.28
1.66
2.29
1.05
3.75
2.39
2.84
13.50
GRU4Rec
1.89
3,22
2.57
3,08
7.75
6.52
7.08
10.13
BERT4Rcc
2,10
3.16
2.64
3.11
4.81
6.70
5.79
6.26
10.63
SASRec
1.82
5.72
3.79
4.51
4.70
8.43
6.59
7.24
8.75
HGN
2.01
5,49
3.82
4.17
3.42
6.24
4.83
5.30
10.50
LightSANs
1.05
4.06
2,54
3.00
5.18
8.94
7.07
7.72
8.25
S RRcRec
2,48
7.37
4.94
4.68
4.14
8.49
6.89
7.35
6.88
DuoRee
1.84
4.50
3.19
3.04
4.13
8.81
7.03
6.64
9.13
MAFRee
2.19
6.35
4.67
3.96
4.01
8.35
6.65
6.98
8.63
Open-P5
4.35
8.12
6,74
5.49
8.50
6.92
5,33
E4SRee
5.62
9.29
7,50
7.98
9.67
8.05
8.70
1,75
E4SRecs
5.46
7.21
7.74
8.63
7.06
7.76
3.63
E4SRec4
5.33
8.75
7.08
7.59
5.41
8.65
7.04
7.72
4.50
SLMRocte-
5.72
9.15
7.48
8.03
6.62
9.83
8.25
8.89
1.25
Table 4: Experiment results (%) of ablation study.
Cloth
Movie
SLMREC
HR@1
HR@5
NDC@@5
MRR
HR@1
HR@5
NDCG@5
MRR
+DDnn
16.10
18.85
17.48
18.17
14.83
23.08
19.08
19.45
+D DeonDDoomm
16.28
19.12
17.69
18.40
14.86
23.89
19.36
19.84
+Dros,Ems 16.85
19.05
17.96
18.59
15.05
23.48
19.40
19.76
4D *DennDDormnnee
16.69
19.47
18.07
18.74
15.29
24.25
19.90
20.36
Table 5: Efficiency comparison of Open-P5, E4SRec, and our SLMREC in terms of epoch-wise training time (hours), inference time (hours), number of training parameters (B) and inference parameters (B). These comparisons were conducted on a machine with an A 100 GPU. The training batch size for all models was standardized 256. During inference, E4SRec and SLMREC utilized batch size of 512, whereas P5's inference was performed with batch size of 1.
Method
Tr time(h)
Inf time(h)
Tr params (B)
Inf params (B)
Open-PPSLaaMe
0.92
4942
0,023
7.237
B4SRec
3.95
0.415
0.023
6.631
SLMREC1-8
0.60
0.052
0.003
0.944
efficacious training strategies can rival, even exceed, larger language models in the sequential recommendation task. This phenomonon is also matched with our therotical justification in Section
Model Efficiency (RQ2). We report the time efficiency and parameters of comparative baselines and our model in Table 5. All time and parameter metrics represent the average across the four datasets reported. Inference time evaluates the prediction ranking among 1, 1,000 candidate items for each user. Detailed training and inference times for each dataset are provided in Appendix B.4. The Open-P5, an LLMRec model based on generative methods, offers reasonable training duration. Yet, during the inference phase, it becomes considerably time-consuming (4942 hours) as it necessitates generating substantial pool of candidate items (for instance, 1000). Owing to the intrinsic workings of generative LLMs, employing generation-bast based LLMRec models for the comprehensive ranking of extensive item sets is advised. Our model outperforms E4SRec with enhanced efficiency, maintaining only 13% and 14% in F4SRec's 's parameters for training and inference, respectively. Moreover, our SL.MRFC demonstrates a remarkable gain in speed, being 6.6 times faster during training and 8.0 times quicker in inference than E4SRec.
Ablation Study (RQ3). As shown in Table 4, SLMREC, when enhanced with various knowledge regularizers (namely D.... Dnorm and Lms), demonstrates improved performance. The regularizers Deos and D Doorm aid SLMREC in aligning its intermediate representations with those of the teacher model, thereby endowing it with
potent representational extraction capabilities. Meanwhile, Lys steers the model to assimilate domain knowledge pertinent to recommendation systems within its preliminary layers. The ablation study in Music and Sport domain be found in Appendix B.5.
8

---

Published as a conference paper at ICLR 2025
70-
20.0
18.0
16-
16.0
HRg
HRgs
KOCOgS
MRA
14
HRgs
Metric
(a) Online (b) Online KD
(c) Block Number
(d) Block Number
Figure Experiment results (%) of online KD and block number B in the Cloth dataset.
5.3 MODEL STUDY
Study of Online KD (RQ4). In our methodology, we first train the teacher model on downstream recommendation tasks and then train the student model through knowledge distillation, which an offline knowledge distillation technology. In this section, demonstrate that we can train both the teacher model and SLMREC together on downstream recommendation tasks, which constitutes an online knowledge distillation. Under this setting, we are able to achieve comparative results.
Study of block number B. We also conducted experiments to investigate the effect of block number B. As shown Figure 4, when B is set 4, our model achieves the best performance. When B set to 1 or 2, the feature constraint imitation for each block within SLMREC is diminished relative to the teacher model, resulting in a decline in performance.
THEORETICAL JUSTIFICATIONS
Beyond empirical experiments, we aim to provide insights into why small language models can perform effectively as large language models learning desirable user representations. Specifically, we focus on the feature propagation process within single layer of an LLM, as outlined below:
H H A (1) (k-1)
where (*) represents the hidden representation of the k-th layer, and A (k) is the attention matrix. In LLaMa, the attention matrix defined as = softmax QQ''T where Q' and K' incorporate rotational encoding (Su al., 2024). Our analysis is hinged interpreting the stack of propagation layers of Transformers as optimization dynamics for minimizing energies of certain forms (Shuman al., 2013; Kalofolias, 2016; Fu et al., 2022; Wu al., 2024).
Proposition 1. Given the updating matrix (k) (k) + L, Eqn. 7 is equivalent gradient descent step with respect to the following optimization problem:
min
H
As A (k) changes across layers, multi-layer attention models can be interpreted as a series of iterative descent steps, each focusing on layer-specific denoising objectives. We will show that this multi multi-layer structure can be simplified into a single-layer model while retaining the same denoising effectiveness.
Proposition 2. For any K-layer attention model (where K is an arbitrary positive integer) with the layer-wise updating rule defined by Eqn. 7, there exists C such that one gradient descent step for the optimization problem (from the initial embeddings H())
min H
H H-CH
where C* associated with A, can yield the output embeddings (K)) of the K-layer model.
These findings indicate that for any multi- stacked decoder, equivalent single-layer decoder can be constructed to encode hidden representations in similar way. Moreover, while the multi-layer model optimizes distinct objectives at each layer, this may introduce redundancy when compared to a single-layer model that achieves its objective a single step. Consistent with the motivation underlying framework design, employ knowledge distillation (KD) to guide the one-layer network, enabling it to streamline the learning process and replicate the feature extraction capabilities of multi- layer network.

---

Published as a conference paper at ICLR 2025
7 RELATED WORK
In this section, we introduce the most related background and scientific investigations to this work, which are roughly divided into five categories, i.e., 1) Sequential Recommendation, 2) Knowledge Distillation (KD), 3) Depth-wise Knowledge of LLMs, 4) Model Pruning, and 5) Parameter-Efficien Fine-Tuning (PEFT). For details on sections three through five, please refer to Appendix C.
Sequential Recommendation. Traditional Sequential Recommendation (TSR) methods (Wu et al., 2017: Hidasi et al., 2015; Kang & McAuley, 2018) primarily focus on developing various temporal encoders to capture short- and long-term user interests. The evolution of temporal sequential encoders has progressed from LSTM units (Wu et al., 2017) and GRU units (Hidasi et al., 2015), to more advanced architectures such as graph neural networks (He et al., 2020; Xu et al., 2023b; 2024c; Zhao et al., 2023), self-attention layers (Kang & McAuley, 2018; Xu et al., 2024b), and Transformer models (Sun et al., 2019). Following the triumph of large language models (LLMs), researchers have begun leveraging open-source LLMs (Touvron et al., 2023) to construct their recommendation systems (Zhao et al., 2024; Bao et al., 2023; Wei et al., 2024). G-LLMRec methods (Geng et al., 2022; Xu et al., 2023a; Zhang et al., 2023b; Liao et al., 2023; Mei & Zhang, 2023) generate the next item based on historical sequences, while E- approaches (Li et al., 2023a; Zhu et al., 2023; Wang et al.. 2024) use LLMs as feature extractors to learn user representations for prediction. More recently, (Zhai et al., 2024) introduces generative sequential framework scalable up to GPT-3 dimensions. LLM-based recommendation systems frequently outperform TSR models by margin of 20% (Li et al., 2023a; Liao et al., 2023; Wang et al., 2024), also increasing the parameters by nearly 100 times compared to TSR models. Therefore, the deployment of LLMRec models in real-world platforms is heavily constrained by computational resources.
Knowledge Distillation (KD). Training a smaller "student" model on the distribution predicted by large "teacher" model is known as a powerful knowledge distillation technique (Hinton et al., 2015). The fundamental insight behind this is to transform the knowledge and capabilities of the teacher into more compact, compressed, and possibly skill-specific representations (Jiao et al., 2020; Gu al., 2024). For those cases when the student only has access to the output tokens generated by the teacher, another way of KD is data distillation (Eldan & Li, 2023; Li et al., 2023b; Fu et al., 2023; Hsieh et al., 2023). This technique first generates high-quality synthetic data by prompting the larger teacher model. The synthetic data are then used to enhance the student's capabilities by fine-tuning.
8 CONCLUSIONS
This paper explores the effectiveness of large language models (LLMs) in sequential recommendation. Our motivational experiments reveal that intermediate layers in L.MSs redundant for achieving optimal recommendation performance. Motivated by empirical insights, we adopt vanilla knowledge distillation methods to improve the performance of small language models. Achieving only 13% of the parameters compared to the LLMRec baseline, our SLMREC model yields an 8x acceleration and slightly better performance. On top of our technical contributions, we believe the results in this paper could shed light on a new promising direction for building effective and efficient recommenders based on LLMs, which is largely under-explored. Additionally, we provide theoretical justifications showing that while multi-layer models optimize distinct objectives at each layer, this can introduce redundancy compared to a single-layer model that achieves its objective in one step. These theoretical insights align with the motivation behind our framework design, where we employ knowledge distillation (KD) to guide the one-layer network, enabling it to streamline the learning process and replicate the feature extraction capabilities of multi-layer network.
Future Work. This work concentrates on enhancing the efficiency of Large Language Model (LLM) utilization in the sequential recommendation. notable limitation is the model's inability to adapt to new scenarios through few-shot learning. When confronted with a fresh dataset or new traffic logs from the platform, the model requires retraining from the entire dataset. In contrast, LLMs have demonstrated promising results in adapting to downstream language tasks using few-shot learning approaches. Looking ahead, we intend investigate the incorporation of incremental learning into LLMe-baeed recommendations to bolster the model's transferability. Additionally, integrating auxiliary linguistic and visual information of users and items into the LLMRec model may offer further improvements in its adaptability to new scenarios.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, arXiv preprint S.Xiv:230330877:2 2023.
Gpt-4 technical report.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm technical report. arXiv preprint arXiv:2305. 10403, 2023.
Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz. Under- standing scaling laws for recommendation models. arXiv preprint arXiv:2208. 08420, 2022.
Saleh Ashkboos, Maximilian Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.1500 2024.
Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pp. 1007-1014, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
2022.
Ronen Eldan and Yuanzhi Li. Tinystories: How small language models be and still speak coherent english? arXiv preprint TrXXiv23305007755 2023.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909. 1556, 2019.
Xinyan Fan, Zheng Liu, Jianxun Lian, Wayne Xin Zhao, Xing Xie, and Ji-Rong Wen. Lighter and better: low-rank decomposed self-attention networks for next-item recommendation. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 1733-1737, 2021.
Guoji Fu, Peilin Zhao and Yatao Bian. p-Laplacian based graph neural networks. In Kama- lika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5878-6917. PMLR, 17-23 Jul 2022. URL https: http://ccceeeeennceeeee mllr wrw.s/ v /2/ fu22e html.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot.
Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, pp-
10421-10430. PMLR, 2023.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference Recommender Systems, pp. 299-315. 2022.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
URL https:ttt//// wwpp/peeeeviieeeeeeeeeeeeee net forum? 5000t/hh0hgiii
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightenn: Simplifying and powering graph convolution network for recommendation. Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp. 639-648, 2020.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.0694 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint 22Xiv:2203,15551 2022.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9782-9793, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference machine learning, pp. 2790-2799. PMLR, 2019.
Cheng- Yu Hsich, Chun-Liang Li, Chih-Kuan Kuan Ych, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Ye Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301. 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09683 2021a.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021b.
Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain- based evaluation of techniques. ACM Transactions Information Systems (TOIS), 20(4):422 446, 2002.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.
2020.
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang. Mengnan Du, and Yongfeng Zhang. Exploring concept depth: How large language models acquire knowledge and concept at different layers? In Proceedings of the 31 International Conference on Computational Linguistics, pp. 558- 573, Abu Dhabi, UAE, January 2025. Association for Computational Linguisties. URL https: //acl antho ogy- o 2.0................ col ing-mat maim. inaiin... 37/.
Vassilis Kalofolias. How to learn graph from smooth signals. In Artificial Intelligence and Statistics, pp. 920-929. PMLR, 2016.
Wang- Cheng Kang and Julian McAuley. Self-attentive seguential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar 2001 003611 2020.
12

---

Published as a conference paper at ICLR 2025
Young Jin Kim and Hany Hassan Awadalla. Fastformers: Highly efficient transformer models for natural language understanding. arXiv preprint arXiv:2010.13382, 2020.
Anton Klenitskiy and Alexey Vasilev. Turning dross into gold loss: is bert4rec really better than sasrec? In Proceedings of the 17th ACM Conference on Recommender Systems, pp. 120-1125, 2023.
Walid Krichene and Steffen Rendle. On sampled metrics for item recommendation. In Proceedings of the 26th ACM SIGKDD international conference knowledge discovery data mining, pp- 1748-177 1757, 2020.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
Xinhang Li, Chong Chen, Xiangyu Zhao, Yong Zhang, and Chunxiao Xing. E4srec: An elegant effective efficient extensible solution of large language models for sequential recommendation. arXiv preprint arXiv:2312.02443, 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05466. 2023b.
Chen Liang. Simiao Zuo. Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pp. 20852-208667. PMLR, 2023.
Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. Llara: Aligning large language models with sequential recommenders. arXiv preprint arXiv:2312.02445 2023.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022.
Chen Ma, Peng Kang, and Xue Liu. Hierarchical gating networks for sequential recommendation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 825-833, 2019.
Kai Mei and Yongfeng Zhang. Lightlm: lightweight deep and narrow language model for generative recommendation. arXiv preprint arXiv:2310. 17488, 2023.
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXhv:2403.03855 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17377 2022.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Aleksandr Vladimirovich Petrov and Craig Macdonald. gsasrec: Reducing overconfidence in sequential recommendation trained with negative sampling. In Proceedings of the 17th ACM Conference on Recommender Systems, pp. 116-128, 2023.
Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. Contrastive learning for representation degen- eration problem in sequential recommendation. Proceedings of the fifteenth ACM international conference on web search and data mining, pp. 813-823, 2022.
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429, 2023.
13

---

Published as a conference paper at ICLR 2025
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pp. 285-295, 2001.
David I Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. The emerging field of signal processing on graphs: Extending high- -ihh-immensioall data analysis to networks and other irregular domains. IEEE signal processing magazine, 30(3):83-98, 2013.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1441-1450, 2019.
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference on web search and data mining. pp. 565-573, 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.0940 09418, 2019.
Hanbing Wang. Xiaorui Liu, Wenqi Fan. Xiangyu Zhao, Venkataramana Kini, Devendra Yadav, Fei Wang, Zhen Wen, Jiliang Tang, and Hui Liu. Rethinking large language model architectures for 2024.
sequential recommendations. arXiv preprint arXiv:2402.094.
Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Limrec: Large language models with graph augmentation for recommendation. In Proceedings of the 17th ACM International Conference on Weh Search and Data Mining, pp- 806-815, 2024.
Chao-Yual Wu, Amr Ahmed, Alex Beutel, Alexander Smola, and How Jing. Recurrent recom- mender networks. Proceedings of the tenth ACM international conference on web search and data mining, pp. 495-503, 2017.
Qitian Wu, Wentao Zhao, Chenxiao Yang, Hengrui Zhang, Fan Nie, Haitian Jiang, Yatao Bian, and Junchi Yan. Simplifying and empowering transformers for large-graph representations. Advances in Neural Information Processing Systems, 36, 2024.
Cong Xu, Zhangchi Zhu, Jun Wang, Jianyong Wang, and Wei Zhang. Fairly evaluating large language model-based recommendation needs revisit the cross-entropy loss. arXiv preprint arXiv:2402.06216. 2024a.
Shuyuan Xu, Wenyue Hua, and Yongfeng Zhang. Openp5: Benchmarking foundation models for recommendation. arXiv preprint 2xXiv23306.11144 2023a.
Wujiang Xu, Shaoshuai Li, Mingming Ha, Xiaobo Guo, Qiongxu Ma, Xiaolei Liu, Linxum Chen, and Zhenfeng Zhu. Neural node matching for targi- cross domain recommendation. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pp. 2154-2166. IEEE, 2023b.
Wujiang Xu, Xuying Ning, Wenfang Lin, Mingming Ha, Qiongxu Ma, Qianqiao Liang, Xuewen Tao, Linxun Chen, Bing Han, and Minnan Luo. Towards open-world cross-domain sequential recommendation: A model-agnostic contrastive denoising approach. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 161-179. Springer, 2024b.
14

---

Published as a conference paper at ICLR 2025
Wujiang Xu, Qitian Wu, Runzhong Wang, Mingming Ha, Qiongxu Ma, Linxun Chen, Bing Han, and Junchi Yan. Rethinking cross-domand sequential recommendation under open-world assumptions. In Proceedings of the ACM on Web Conference 2024, pp- 3173-3184, 2024c.
Yaowen Ye, Lianghao Xia, and Chao Huang. Graph masked autoencoder for sequential recom- mendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 321-330, 2023.
Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda sequential transducers
Gu, Michael He, et al. Actions speak louder than words: for generative recommendations. arXiv preprint arXiv:2402. 17152, 2024.
Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, al. Wukong: Towards scaling law for large-scale recommendation, arXiv preprint ar S4ti:220030025455 2024.
Gaowei Zhang, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, and Ji-Rong Wen, Scaling law of large sequential recommendation models. arXiv preprint arXiv:2311. 2023a.
Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping, Advances in Neural Information Processing Systems, 33: 33:1401 1-14023, 2020.
Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. Collm: Integrat- ing collaborative embeddings into large language models for recommendation. arXiv preprint 23X1::3310.19488 2023b.
Wayne Xin Zhao, Junhua Chen, Pengfei Wang, Qi Gu, and Ji-Rong Wen. Revisiting alternative experimental settings for evaluating top-n item recommendation algorithms. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 23322, 2020.
Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin. Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. Recbole: Towards unified, comprehensive and efficient framework for recommendation algorithms. In proceedings of the 30th acm international conference on information & knowledge management, pp. 4653-4664, 2021.
Ziwei Zhao, Xi Zhu, Tong Xu, Aakas Lizhiyu, Yu Yu, Xueying Li, Zikai Yin, and Enhong Chen. Time-interval aware share recommendation via bi-directional continuous time dynamic graphs. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 822-831, 2023.
Ziwei Zhao, Fake Lin, Xi Zhu, Zhi Zheng, Tong Xu, Shitian Shen, Xueying Li, Zikai Yin, and Enhong Chen. Dynlim: When large language models meet dynamic graph recommendation. arXiv preprint ar arXiv:2405.07555
2024.
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM international conference on information & knowledge management, pp. 1893-1902, 2020.
Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li. Collaborative large language model for recommender systems. arXiv preprint arXiv:2311.01343, 2023.
15

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
Motivational Experiments
3 Preliminaries
4
SLMRec
Experiments
5.1 Experiment Setup
5.2
Performance Comparisons
5.3
Model Study
Theoretical Justifications
7 Related Work
10
8 Conclusions
10
Proof
17
A.1 Proof for Proposition
17
A.2 Proof for Proposition
17
Experiments
18
B.1 Motivation Experiment Results
18
B.2 Training Details
18
B.3 Compared methods
18
B.4 Model Efficiency
20
B.5 Ablation Study
Extended Related Work
22
C.1 Depth-wise Knowledge of LLMs
22
C.2 Model Pruning
22
C.3 Parameter-Efticient Fine Line-Tuning (PEFT)
22
16

---

Published as a conference paper at ICLR 2025
A PROOF
A.1 PROOF FOR PROPOSITION
Proposition 1. Given the matrix A A (k) +L Eqn. is equivalent to a gradient descent step with step size I for the following optimization problem:
min
A H(k
(10)
Proof. The cost function at the k-th layer is denoted by
E(H 1)) min H
A
H (s-
(11)
Then, the gradient of EHH; H - 1)) is computed as
(E H 1)) =2 H - ((
(12)
ÖH
With the step size of the gradient descent, it minimizes the cost function H H 1)) at the current layer is
(k) (k--1) OE(H: H 1))
(-H(E-1)
(13)
ÖH
H(k-1) (H()-1) A (4) (Kk-1)
(14)
((
(15)
H(k-1) A (M)
(16)
L
A.2 PROOF FOR PROPOSITION
Proposition 2. For any K- Kllayer attention model (where K is an arbitrary positive integer) with the layer-wise updating rule defined by Eqn. 7, there exists C such that one gradient descent step for the optimization problem (from the initial embeddings ())
min |
(17)
where C' associated with A, can yield the output embeddings (K) of the K K-layer model.
Proof. Similar to Theorem 1, we define À (k) to simpify the Eqn. 7.
A (k) =I+ A
(18)
Then Eqn. 7 can be equivalently written as
H (k-1),
(19)
By stacking K layers of propagation, we can denote the output embeddings as
H () A (K-I) = Á (K) ( (K- H (K2))
À (K)... ( (O)
(20)
where A defined as multiple matrix production.
We can show that solving the denoising problem with gradient step size 44/ w.r.t. the objective
min H
H
COO
(21)
17

---

Published as a conference paper at ICLR 2025
0.30
0.20
Eeilor 100GON
0.25
0.10
0.10
0.05
24
28
30
32
24
26
28
30
32
24
26
2H
32
Layer Number
Layer Number
Layer Number
(a) Movic (Infer) HR
(b) Movic (Infer) NDCG@10
(c) Movie (Infer) MRR
0.290
-O-
-O- Leabo 140Gg
NA1 -A88. 2014
0.20
0.245
0.18
0.260
0.16
0.275
0.14
0.270
30 0.12
20
26
Layer Number
Layer Number
Layer Number
(d) Movic (Train) HR@ 10
(c) Movic (Train) NDCG@ 10
(f) Movic (Train) MRR
Figure 5: We present the relationship between the number of decoder layers and the final recommen- dation performance, with the performance of SASRec plotted as a baseline. Figures (a)-(c) show the results of directly using representations from the middle layers for inference without training, while (d)-(f) prune the later layers and train model using only the specified number of layers. From the results, we observe that deeper decoder layers introduce redundancy in recommendation tasks, with models utilizing fewer layers (8-layer) achieving performance nearly equivalent to (24-layer) models.
Defining C' a- (A* (1 (1-p")) II) H A' (0) will induce the output embeddings (()) by noticing that
((K) (H) 4
OE(H: H()
(22)
0H
H-H
H
(H)))
(0)
('H())
(23)
2
-H (0)
(A* (1 - H)
(24)
2
OH
1
H (0)
+ A
H (0)
H)) + J (()
(25)
(0
(26)
B EXPERIMENTS
B.1 MOTIVATION EXPERIMENT RESULTS
B.2 TRAINING DETAILS
In Table 6, we provide hyper-paramette in training stage. Our implementation is based on Huggingface Transformers 6 The input and intermediate hidden dimension in the feed-forward network is 4096. We use mixed precision training and train on *800 Nvidia A100 GPU.
B.3 COMPARED METHODS
Tranadtional sequential recommendation methods:
hnttps:/// thub. htwp://www.ooooogggooo///eerra/a
18

---

Published as a conference paper at ICLR 2025
Table 6:
(HP) settings of our method on each dataset.
HP
Cloth
Movie
Music
Sport
adam betal
0.9
0.9
0.9
0.9
adam_betal
0.999
0.999
0.999
9.999
adam_epsiloon
Ic-8
Ic-8
1c-8
1c-8
learning_rad rate
0.003
0.001
0.002
0.002
logging_steed steps Ir_ L_schedduler__ type max grad norm max_steps
1
cosine
cosine
cosine
cosine
1.0
1.0
1.0
1.0
1500
-1
800
2000
optimizer
adamw_ Morch
aalamw_torch
adamw_tords
adamw_toreh
save_strategy
sleps
steps
steps
steps
save_steps
50
100
100
100
eval steps
50
100
100
100
warmup_ster
50
50
100
50
A
1.0
1.0
1.0
1.0
22
0.1
0.1
0.1
0.1
X3
1.0
1.0
0.01
0.1
b
4
4
4
4
Caser (Tang Wang, 2018) introduces a novel approach to sequential recommendation systems by modeling user-item interactions sequences, which is designed to predict the next item may interact with by capturing both short-term and long-term dependencies user behavior.
GRU4Ree (Hidasi et al., 2015) tackles the issue of modeling sparse sequential data while also adapting RNN models to the recommender system. To achieve this, the authors propose a new ranking loss function that is specifically designed for training these models. The implementation of GRU4Rec in PyTorch can be found at the URL."
BERT4Rec (Sun al., 2019) designs bidirectional self-attention network model user behavior sequences. To prevent information leakage and optimize the training of the bidirectional model, Cloze objective used predict the randomly masked items in the sequence by considering both their left and right context. The implementation of BERT4Rec in PyTorch can be found at the URL
SASRec (Kang McAuley, 2018) is self-attention based sequential model that addresses the challenge of balancing model parsimony and complexity in recommendation systems. By using an attention mechanism, SASRec identifies relevant items user's action history and predicts the next item based on relatively few actions, while also capturing long-term semantics like an RNN. This enables SASRec to perform well in both extremely sparse and denser datasets. The implementation of SASRec in PyTorch can be found at the URL
HGN (Ma et al., 2019) propose novel hierarchical gating mechanism effectively capture both short-term and long-term user preferences in sequential recommendation tasks. Their model dynami- cally selects relevant interaction history at multiple temporal levels, improving next-item prediction accuracy. This approach outperforms state-of-the-une methods while maintaining efficiency and scalability for large-scale recommendation systems.
LightSANs (Fan al., 2021) introduces a low-rank decomposition technique for self-attention networks, reducing their computational complexity while maintaining strong performance. This approach makes the model more efficient and scalable for large-scale recommendation tasks without compromising accuracy.
For the code implementation of Caser, HGN and LightSANs, the experiment based on the RecBole (Zhao et al., 2021) 10.
BERT -Re- Pytorch
pytorch /
19

---

Published as a conference paper at ICLR 2025
Self-supervised sequential recommendation methods
S*eRec (Zhou et al., 2020) introduces self-supervised learning into a sequential recommendation by utilizing mutual information maximization (MIM) to learn better representations from user sequence data. The model incorporates four auxiliary self-supervised objectives: item cropping, item masking, item reordering, and segment prediction, to enhance the quality of learned item representations. By pre-training the model with these superviee tasks and then fine-tuning in the recommendation task, it achieves strong performance even with limited training data. The method demonstrates that self-supervised learning can effectively leverage the inherent supervisory signals within sequential data to improve recommendation quality.
DuoRec (Qiu et al.. 2022) addresses the representation degencration problem in sequential recom- mendation where item embeddings tend be similar and occupy narrow cone. Instead traditional data-level augmentation (like masking or cropping), it introduces model-level augmentation using dif- ferent Dropout masks to generate sequence representations. They also propose using sequences with the same target item as positive samples, which provides more meaningful contrasts. Through this contrastive regularization approach, DuoRec encourages a more uniform distribution of embeddings in the representation space, leading to better recommendation performance. The key innovation is tackling representation degeneration through model architecture rather than data augmentation while maintaining semantic consistency in the contrasting process.
MAERec (Ye et al., 2023) addresses limitations in sequential recommendation systems where other methods struggle with limited labels and noisy user behavior. Unlike previous approaches using manual contrastive learning strategies, MAERec implements a graph masked autoencoder framework that automatically identifies and focuses on meaningful item relationships. The model uses adaptive masking and task-specific regularization to ensure the learned representations align with recommendation goals while filtering out noise. By dynamically reconstructing masked item transitions rather than relying on hand -crafted data augmentation, MAERec achieves more robust performance across different recommendation scenarios without requiring manual heuristics. The approach demonstrates strong results in handling both data sparsity and noise while maintaining computational efficiency. The implementation of MAERec in PyTorch can be found the URL 11.
For the code implementation of ReeR and DuoRec, we run the experiment based on the RecBole (Zhao et al., 2021) 12 13,
LLM-based recommendation methods:
Open-P5 (Xu et al., 2023a) is an open-source platform introduced to catalyze research LLM-bbssed generative recommender systems. It supports key model architectures like T5 and Llama-2 across diverse public datasets, focusing on sequential and straightforward recommendation tasks. The platform emphasizes the role of item IDs through various indexing methods and offers a customizable, efficient, and standardized environment for developing and assessing recommender systems. The implementation of Open-P5 -P in PyTorcl can be found at the URL 14
E4SRec (Li et al., 2023a) integrate of Large Language Models (LLMs) into sequential recommen- dation systems, offering significant leap in handling item IDs and personalization. In the original paper, they use Softmax layer to output each user-item prediction score. The implementation of E4SRec in PyTorch can be found at the URL
B.4 MODEL EIFICIENCY
We show the running time of Open-P5, E4SRec, and our SLMREC in each dataset. These compar- isons were conducted on a machine with an A100 GPU. The training batch size for all models was standardized at 256. During inference, E4SRec and SLMREC utilized a batch size of 512, whereas Open-P5's inference was performed with a batch size of 1.
20

---

Published as a conference paper at ICLR 2025
Table Detailed efficiency comparison of Open-P5, E4SRec, and our SLMREC, in terms of training and inference time, on each dataset.
Method
Cloth
Movic
Music
Sport
thy
Open-PSraaate
3692
B4SRoc
5.27
0,578
1.90
0,208
1,88
0.216
6,75
0.660
SLMREC
0.97
0.070
0.15
0.030
0.30
0.030
0.98
0.078
Table 8: Efficiency comparison of SASRec, MAERec and our SLMREC in terms of epoch-wise inference time (hours). These comparisons were conducted on machine with an A100 GPU. During inference, models leverage parallel processing with a batch size of 512.
Method
Inf timeth)
Improw. (%)
SASRec 0.015
0.00
MAERec
0.061
11.96
SLMREC
0.052
45.26
Table 9: Experiment results (%) of ablation study.
SLMREC
Music
Sport
HR@1 1
HR@s
NDCG@5
MRR
HR@
HR@5
NDCG@5
MRR
+D coa
5.62
8.78
7.23
7.81
6.25
9.25
8.41
+D DooiDnorm
5,95
9.26
7.65
8.23
6.61
9.82
8.24
8.87
+D DDenaeona
5.69
8.94
7.36
7.91
6.51
9.39
7.96
8.62
5.72
9.15
7.48
8.03
6.62
9.83
8.25
8.89
To evaluate deployment efficiency in real-world scenarios, we compare the inference time between tra- ditional recommendation approaches and LLM-based methods. Our experimental results demonstrate that SLMRec achieves comparable computational efficiency to traditional methods while delivering substantial performance improvement of nearly 50% over SASRec.
B.5 ABLATION STUDY
We present the remaining ablation study results in Table 9, SLMREC, when enhanced with various knowledge regularizers (namely Deoss Dorm and L.ss), demonstrates improved performance. The regularizers Deo. and Dromm aid SLMREC in aligning its intermediate representations with those of the teacher model, thereby endowing with more potent representational extraction capabilities. Meanwhile, Lms steers the model to assimilate domain knowledge pertinent to recommendation systems within its preliminary layers.
21

---

Published as a conference paper at ICLR 2025
C EXTENDED RELATED WORK
C.I DEPTH-WISE KNOWLEDGE OF LLMS
The recent community interest stems from how linguistic properties and knowledge are encoded in language models. (Meng et al., 2022; Dai et al., 2022; Jin et al., 2025) emphasize that knowledge localizes within the middle or final layers. On the other hand, (Hase et al., 2024) attempts to perform knowledge editing and concludes that information may be stored non-locally across layers. What's s more, (Men et al., 2024; Gromov et al., 2024) share a similar view that current pretraining methods not properly leveraging the parameters in the deeper layers of the network that the shallow layers play a critical role in storing knowledge. By contrast, we are the first to investigate which part of knowledge the LLMs plays key role, especially in the sequential recommendation scene.
C.2 MODEL PRUNING
Model Pruning is fundamental approach for reducing the size of a well-trained large model by removing unimportant parameters (Hassibi & Stork, 1992). Recent work has focused on applying pruning methods to the Transformer architecture (Vaswani et al., 2017). These works have studied different components of the model architecture for pruning, including dropping attention heads (Voita et al., 2019; Michel et al., 2019), dropping layers (Fan al., 2019; Zhang & He, 2020; Kim & Awadalla, 2020; Sajjad et al., 2023), dropping hidden xistates (Hou et al., 2020), replacing sparse weight matrices with smaller dense ones (Ashkboos al., 2024), and combinations of these solutions. By contrast, our work performs layer removal through simple knowledge distillation, rather than more complex pruning techniques.
C.3 PARAMETER-EFFIE EFFICIEET FINE-TUNING (PEFT)
PEFT emerges as novel technique for tailoring Large Language Models (LLMs) to specific tasks while ensuring minimal computational and memory costs (Houlsby al., 2019; Lester al., 2021; Hu et al., 2021bs Liu et al., 2022). In this work, we combine our method with the Low-Rank Adapters (LoRA) (Hu et al., 2021b) to reduce the memory and computation of the knowledge distillation process. Specifically, we freeze the pre-trained model and only tune small set of additional trainable parameters.
22


Reviews:
[{"review_id": "Z8DscO5s05", "reviewer": "Reviewer_HP9b", "summary": "This paper introduces SLMREC, a method that enhances small language models for sequential recommendation tasks using knowledge distillation. The authors conduct extensive experiments on large-scale industry datasets, revealing that most intermediate layers in large language models are redundant. SLMREC achieves competitive performance with only 13% of the parameters of LLMs, making it highly efficient and practical for real-world applications. The model is orthogonal to other post-training efficiency techniques, such as quantization and pruning, allowing for combined use to further improve efficiency. Comprehensive experimental results demonstrate that SLMREC outperforms existing state-of-the-art methods in various evaluation metrics.", "questions": "- The article mentions the use of multiple supervisory signals. What are these signals? How can we combine them in the training process?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper reveals that most intermediate layers in large language models are redundant, providing valuable insights into the architecture of these models. \n\n- The paper introduces a novel method, SLMREC, which effectively enhances small language models for sequential recommendation tasks using knowledge distillation. SLMREC achieves competitive performance using only 13% of the parameters of large language models, making it highly efficient and practical for real-world applications.\n\n- The proposed method is orthogonal to other post-training efficiency techniques like quantization and pruning, allowing for combined use to further enhance efficiency.", "weaknesses": "- The paper does not describe the specific process of knowledge distillation in detail, and lacks a detailed explanation of the key steps in the distillation process (such as temperature adjustment, loss function selection, etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UGly3jByQS", "reviewer": "Reviewer_jtBY", "summary": "This work discusses the impact of LLMs in SR, but questions the necessity of their size. The inefficiency of applying LLMs to real-world platforms due to their size is noted. The experiments show that most intermediate layers of LLMs are redundant, leading to the development of a small language model for SR using knowledge distillation. SLMRec achieves strong performance with significantly fewer parameters than LLM-based models, along with notable speedups in training and inference times. It also provides theoretical reasoning for the effectiveness of small language models in SR.", "questions": "1. What are the results of fine-tuning small LLMs using only CE loss? (i.e., without employing knowledge distillation methods). Additionally, what would the results be if only both CE loss and MS loss were used? (It appears that MS loss is unrelated to knowledge distillation.)\n\n2. Could the authors include comparisons with other traditional sequence recommendation methods in the Efficiency Comparison section to provide a more comprehensive evaluation of their efficiency?\n\n3. Could the authors provide experimental results for the self-supervised and pre-trained sequence recommendation methods mentioned in the fourth point of the Weakness section?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors explored the relationship between the parameter count of pre-trained LLMs and recommendation performance under the E-LLMRec paradigm, discovering a balance point between performance and parameter count that enhances the practicality of deployment in real-world scenarios.\n\n2. The authors conducted thorough theoretical justifications to demonstrate why small language models can possess learning capabilities for user representations that are comparable to those of large language models.\n\n3. The paper is well-written, and the figures clearly illustrate the overall workflow of SLMRec.", "weaknesses": "1. The motivation is not sufficiently clear. The authors claim that LLMs with fewer layers can achieve recommendation performance comparable to those with more layers, and thus they opt to use large LLMs to distill a small LLM. However, it is unclear why small LLMs cannot be fine-tuned directly. The authors also do not present results in the ablation study showing the fine-tuning of a 4-layer small LLM with CE loss without any distillation operation.\n\n2. Generally speaking, distillation is performed to transfer knowledge from one model to another. However, small LLMs possess certain language capabilities on their own. During the distillation process, how can the teacher model be guided to transfer recommendation-related information to the student model? If only general knowledge is distilled, the parameter limitations of small LLMs imply that they may not be able to learn additional knowledge effectively.\n\n3. Although knowledge distillation can help small LLMs acquire certain recommendation capabilities and reduce deployment costs, simply reducing the number of layers from 8 to 4, while being more efficient than Open-P5 and E4SRec, still renders it too inefficient compared to traditional sequence recommendation methods. This creates a significant gap in terms of practical deployment, weakening the motivation claimed by the authors.\n\n4. The experimental section lacks some of the latest traditional sequence recommendation methods, such as self-supervised approaches (e.g., DuoRec [1], MAERec [2]) and pre-trained methods (e.g., S3-Rec [3]).\n\n5. In Table 2, the experimental results on the Cloth dataset show that E4SRec outperforms SLMRec in HR@1, NDCG@5, and MRR. However, it is marked in purple as having \"the second best performance,\" which seems inconsistent.\n\n[1] Qiu, Ruihong, et al. \"Contrastive learning for representation degeneration problem in sequential recommendation.\" Proceedings of the fifteenth ACM international conference on web search and data mining. 2022.\n\n[2] Ye, Yaowen, Lianghao Xia, and Chao Huang. \"Graph masked autoencoder for sequential recommendation.\" Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2023.\n\n[3] Zhou, Kun, et al. \"S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization.\" Proceedings of the 29th ACM international conference on information & knowledge management. 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIUqW6SOe1", "reviewer": "Reviewer_M71Q", "summary": "This work investigates the use of small language models (SLMs) in the realm of sequential recommendation systems, proposing a model called SLMRec. The authors identify that traditional sequential recommendation (TSR) methods have reached a performance plateau and that existing large language model (LLM) approaches often require significant computational resources due to their large size. The proposed SLMRec aims to maintain competitive performance with a much smaller parameter footprint (less than 1 billion parameters) compared to the over 7 billion parameters typical of LLMs used in recommendations. The authors utilize knowledge distillation techniques to align the representation knowledge between a larger teacher model and the smaller student model, achieving an up to 8x acceleration in both training and inference times.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. SLMRec demonstrates significant efficiency improvements, achieving competitive performance while using substantially fewer parameters than larger models.\nS2. The model's design allows it to operate within the resource constraints typical of real-world applications, making it more applicable for widespread use in industry.\nS3. The paper's approach to knowledge distillation, focusing on feature alignment rather than just output prediction, is a noteworthy contribution that can lead to better representation learning in smaller models.\nS4. The authors conduct comprehensive experiments on large-scale datasets, providing empirical evidence of the model's effectiveness across multiple metrics and domains.", "weaknesses": "W1. The implementation of knowledge distillation can introduce complexity in the training process, which may not be straightforward for all practitioners to apply.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NwUMQKF09o", "reviewer": "Reviewer_ZHQw", "summary": "This paper works on improving the efficiency of LLM as a recommender scheme by arguing that the intermediate layer of LLM is redundant. Specifically, the paper suggests a vallina knowledge distillation approach to achieve near LLM-based Recommendations in an efficient way.", "questions": "Could the author justify the claim that “an 8-layer E4SRec can obtain nearly as informative user representation as a 24-layer E4SRec” by answering the following question in detail?\n1. Provide a more detailed analysis of the performance differences between the 8-layer, 24-layer, and 32-layer models.\n2. Explain the apparent performance surge at 32 layers and discuss its implications for your claims.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper addresses an important problem regarding the efficiency of LLM as a recommender scheme and provides a reasonable knowledge distillation approach to solving the problem.\n2.\tThe paper is generally well-written.", "weaknesses": "1.\tThe knowledge distillation approach is widely used in LLM-related work [1], which raises concerns about its novelty when adopting it in the recommendation.\n2.\tAlthough the paper claims that they provide both experimental and theoretical justification for adopting KD to LLMRec. It does not look strong enough. \n    1. In the experimental part, as only 8-layer and 24-layer are discussed, could the author provide a more detailed analysis of the performance differences between the 8-layer, 24-layer, and 32-layer models? \n    2. In terms of the theoretical part, the author suggested that a small model can also learn user representation that is similar to the larger model, while it does not align with the recent work on the emergent ability of LLM that a small model cannot achieve a similar result of the large language model.\nCould the author discuss how your findings relate to existing work on emergent abilities in LLMs, and explain why user representation learning may differ from other tasks where emergent abilities have been observed? Also, some supportive experiment is needed to justify this claim.\n\n\n[1] Xu X, Li M, Tao C, Shen T, Cheng R, Li J, Xu C, Tao D, Zhou T. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116. 2024 Feb 20.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)