Customizing Language Models with Instance-wise LoRA for Sequential Recommendation
Authors: Xiaoyu Kong
Jiancan Wu
An Zhang
Leheng Sheng
Hui Lin
Xiang Wang
Xiangnan He
Venue: NeurIPS 2024 poster (2024)
Keywords: Sequential Recommendation
Large Language Models
Mixture of Experts
Primary area: other

Abstract: Sequential recommendation systems predict the next interaction item based on users' past interactions, aligning recommendations with individual preferences. Leveraging the strengths of Large Language Models (LLMs) in knowledge comprehension and reasoning, recent approaches  are eager to apply LLMs to sequential recommendation. A common  paradigm is converting user behavior sequences into instruction data, and fine-tuning the LLM with parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaption (LoRA). However, the uniform application of LoRA across diverse user behaviors is insufficient to capture individual variability, resulting in negative transfer between disparate sequences.
To address these challenges, we propose Instance-wise LoRA (iLoRA). We innovatively treat the sequential recommendation task as a form of multi-task learning, integrating LoRA with the Mixture of Experts (MoE) framework. This approach encourages different experts to capture various aspects of user behavior. Additionally, we introduce a sequence representation guided gate function that generates customized expert participation weights for each user sequence, which allows dynamic parameter adjustment for instance-wise recommendations. 
In sequential recommendation, iLoRA achieves an average relative improvement of 11.4\% over basic LoRA in the hit ratio metric, with less than a 1\% relative increase in trainable parameters.
Extensive experiments on three benchmark datasets demonstrate the effectiveness of iLoRA, highlighting its superior performance compared to existing methods in mitigating negative transfer and improving recommendation accuracy.
Our data and code are available at https://github.com/AkaliKong/iLoRA.

Customizing Language Models with Instance-wise LoRA for Sequential Recommendation
Xiaoyu Kong' Jiancan Wu An Zhang²
Leheng Sheng² Hui Lin" Xiang Wang' Xiangnan He*
MoE Key Lab of BIPC, University of Science and Technology of China
National University of Singapore
Electroniic Science Research Institute of China Electronics Technology Group Corporation
kongxyčmail .stc.. edu, en, wuj jujcangmmai.. com com
xlangnanhe@gmall.com
Abstract
Sequential recommendation systems predict the next interaction item based on users' past interactions, aligning recommendations with individual preferences. Leveraging the strengths of Large Language Models (LLMs) in knowledge com- prehension and reasoning, recent approaches are cager to apply LLMs to sequential recommendation. A common paradigm is converting user behavior sequences into instruction data, and fine-tuning the LLM with parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaption (LoRA). However, the uniform appli- cation of LoRA across diverse user behaviors is insufficient to capture individual variability, resulting in negative transfer between disparate sequences. To address these challenges, we propose Instance-wise LoRA (iLoRA). We innovatively treat the sequential recommendation task as a form of multi-task task learning, integrating LoRA with the Mixture of Experts (MoE) framework. This approach encourages different experts to capture various aspects of user behavior. Additionally, we introduce sequence representation guided gate function that generates customized expert participation weights for each user sequence, which allows dynamic parame- ter adjustment for mnstance-wise recommendations. In sequential recommendation, iLoRA achieves an average relative improvement of 11.4% over basic LoRA in the hit ratio metric, with less than 1 % relative increase in trainable parameters. Extensive experiments on three benchmark datasets demonstrate the effectiveness of iL.AA, highlighting its superior performance compared to existing methods in mitigating negative transfer and improving recommendation accuracy. Our data and code available https: //t::/gttttt //ggtttttb.///////////// com/ Akal Kooil/ooon/gangg Kooo/ooooo.... iLoRA.
Introduction
Sequential recommendation [1] suggests user's next item of interest by analyzing his/her past interactions, tailoring recommendations to individual preferences. As Large Language Models (LLMs) [2] exhibit impressive proficiency in global knowledge comprehension and reasoning, their potential for application in sequential recommendation is garnering increasing interest [3-14]. Recent efforts [9, 10] approach the sequential recommendation task under language generation paradigm, wherein user behavior sequences are converted into input prompts by either purely textual prompting (ID numbers or descriptions) [3, 5, 15, 6] or hybrid prompting with additional behavioral tokens [9, 10, 16], achieving remarkable success. Upon scrutinizing prior studies on LLM-based sequential recommenders, we can summarize a common fine-tuning pipeline comprising three components:
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Cluster ID
C2
C2
14
Cu
14
-22
32
C
(Color Represents Genre)
same movie
Figure 1: Gradient similarity of LoRA modules across training steps. The sequence dataset is partitioned into 8 clusters using Euclidean distance, with hierarchical clustering applied to reorder clusters, so that clusters closer in the collaborative space are also closer together in the heatmap. Gradient similarity is used assess the geometric characteristics of the loss, with darker cells indicating bigher similarity. In the case study on the right, dashed lines connect similar items, while solid lines link identical items. Users with gradient similarity of 0.86 share strong interest in thriller movies, while those with -0.75 cosine similarity show no clear preference alignment.
(1) convert sequence of historical behaviors into a prompt: (2) pair these prompts with the subse- quent items of interest, to create instruction-tuning datasets; (3) incorporate a trainable Low-Rank Adaptation (LoRA) module [5, 16, 10, 9] into LLMs and fine-tune it such prompts. Existing studies [17, 3, 18, 6, 9, 10, 16] primarily focus on refining high-quality prompts to more effectively incorporate recommendation information, while leaving the choice of LoRA unexplored. Instead, they employ a standard LoRA module for fine-tuning, which freezes the LLM weights and updates the model through two additional low-rank matrices.
Previous work [19, 20] have demonstrated that related tasks tend to develop similar loss geometries, while unrelated tasks exhibit dissimilar ones. Using the same set of parameters in multi-task learning context can lead to conflicts, especially when tasks have low gradient similarity. This can result in negative transfer, which limits further improvement of the model. From this perspective, since user behaviors often exhibit substantial individual variability (e.g., distinct interests, behavior patterns, feedback mechanisms), diverse behaviors may not effectively capture these variabilities. Specifically, the inherent variability in user behaviors naturally inspires to view item sequences with different behavior variables as different tasks. Simply applying singular LoRA module leaves this multitask nature of sequential recommendation untouched, thus potentially overestimating the relationships between sequences. casily causes the negative transfer between significantly discrepant sequences. Take LLaRA [9] as an example, which fine-tunes a LoRA module on top of Llama-2 [21] across all sequences. Following prior studies [19, 20], we use symmetric heatmap to analyze the pair-wise gradients similarities of LLaRA associated with disparate sequences and reveal significant misalignment,
argue that employing standard LoRA module
such
shown in Figure 1. Specifically, we observe strong clustering by membership closeness in the collaborative space, along the diagonal of the gradient similarity matrix. Meanwhile, clusters that are more distant in the collaborative space tend to exhibit dissimilar gradient trajectories. Clearly, such gradients are misaligned, which can result in suboptimal performance. To mitigate this issue, one straightforward solution is to deploy multiple LoRA modules, each fine-tuned for specific sequence, enabling each module to act as lightweight expert tailored to its respective sequence. However, it is impractical in terms of resources and time complexity, as the number of sequences often scales to millions.
To address these challenges, we propose a new fine-tuning framework, Instance- wise LoRA (iLoRA), which adapts the mixture of experts (MoE) concept [22, 23] to tailor the LLM for individual variability sequential recommendation. The key idea to integrate diverse array of experts within the basic LoRA module, each encouraged to capture specific aspect of user behaviors. Specifically, for each instance of item sequence a user adopted before, we feed it into conventional recommender model (e.g., SASRec [24]) to get holistic sequence representation. Consequently, this representation,

---

reflecting personal behavior variability, is used by gating network to customize instance-wise attention scores for the experts, where each score dictates the participation of each expert. With the attention scores, the experts are further assembled as instance-wise LoRA for this item sequence. Hereafter, we follow I.L.RA [9] and fine-tune the L.LM (i.e., Llama-2 [21]) on a hybrid prompt, but instead with the -yatiiaatly LoRA to mitigate the negative transfer between discrepant sequences. Importanly, iLoRA maintains the same total number of parameters as the standard LoRA, thereby avoiding overfitting while dynamically adapting to diverse user behaviors. We assess the effectiveness of iLoRA through extensive experiments on three benchmark sequential- recommendation datasets (i.e., LastFM [25], MovieLens [26], Steam [27]), showcasing its superiority over leading methods (e.g., GRU4Rec [28], Caser [29]. SASRec [24]. MoRec [30], TALLRec [5],
LLaRA [9]).
2 Preliminary
LLM-based Sequential Recommendation. The primary task of sequential recommendation is to predict the next item that aligns with user preference [31, 32]. Formally, consider user with a historical interaction sequence represented as i<n = [81, l2, in-1], where each iz is an item interacted with the j-th step. A sequential recommender. parameterized by inputs this sequence and outputs probability distribution over potential next items in the candidate set. This model is trained maximize the likelihood of the true next item in
max Σ
log Po lien).
(1)
D
In the context of LLM-based sequential recommendation, we employ instruction tuning [33, 34], which fine-tunes LLMS using training data structured into explicit instructional pairs (x,y). Here, comprises detailed textual instruction describing the interaction sequences i<. and recommendation task [17, 5, 15, 9, 16, 10], and y is the textual description of the predictive item in in the user's sequence [35]. The training objective is formulated as an autoregressive model optimization problem:
max
log (
where & denotes the LLM's model parameters, Yt represents the t-th token in the output sequence, and y<t includes all preceding tokens in the sequence. This objective ensures that each prediction informed by both the prior items in the sequence and the detailed instructions describing the sequential recommendation task [32, 31, 9, 16, 10].
Fine-tuning with Low-rank Adaption (LoRA). Fully fine-tuning LLMs (cf. Equation (2)) entails substantial computational resources [36, 35, 37, 21, 38-40]. LoRA emerges as an efficient alternative [41-49], which injects trainable low-rank matrices into transformer layers to approximate the updates of pre-trained weights [46]. At the core, LoRA employs a low-rank decomposition where the update A AW to the pre-trained matrix W the is represented as AW- BA, where B IR and A E dd and are tunrable up- and coon-projection matrices, respectively. The rank r is significantly smaller than both dn and doats enhancing adaptation efficiency.
Typically, LoRA applies such updates to the query and value projection matrices in the multi-head attention sub-layers within transformer layers [41]. Specifically, for an input h to the linear projection the multi-head attention, LoRA results the output h' as:
h' (W + a - Wh+ a
BAh
7
where W remains frozen, and a is introduced scaling factor that adjusts the influence of the updates w.E.f. the original W.
This methodology introduces flexible and efficient means to customize these models new tasks, circumventing the need for extensive retraining of all model parameters.
Fine-tuning with Hybrid Prompting. In the field of LLM-based sequential recommendation, critical challenge the divergence between the natural language space and the "user behavior" space. To bridge this gap, previous research [9, 10, 16] introduces hybrid prompt approach, which
3

---

ut <?
Instruction:
History ID-sequence
This user has watched <item <itema? Predict the next movie the user will watch.
<itema?
<itemyr
SR-EMB X
This
Other Text:
Item Name:
<Titanic>
<item P
V
SR-EMB
X
Seq emb
LLM
V
Tokenizer
Gating Network
Adapter
Contribution Weight
Other Emb
Name Emb
Emb
Name Emb
Emb
Other Emb
LLMs
iLoRA
item.it
AW (i())))
Figure 2: The iLoRA framework, which integrates the idea of MoE with LoRA, to implement sequence-customized activation patterns for various sequences.
incorporates behavioral insights captured by recommendation models into the prompts. This approach combines the textual token representation derived from the LLM's word embedding layer, with behavior token representation learned from the recommender model with a moosll projector. Formally, for item i with associated metadata trt, the LLM tokenizer and word embedding layer LLM-TKZ(--) convert it into token representations s:
I.LM-KZ.. (frtt.
Concurrently, item i's ID embedding z, pretrained using the encoder SR-EMB(.) of sequential recommender (e.g., SASRec [24], GRU4Rec [28]), is represented as:
(R))MBo()).
Then the ID embedding is transferred into a behavior token representation through a trainable projector Proj(-) parameterized by Ti facilitate the alignment between the two modalities. This alignment strategy enables LLMs to interpret and leverage the behavioral knowledge distilled by conventional recommenders [9, 16, 10]. Subsequently, the textual and behavioral token representations are then concatenated to form a comprehensive description of item é:
Concat(s, Proj,,
By converting each item into hybrid token representation (cf. Equation (6)), we can rewrite the input prompt x describing the sequence ięn and target response decipting the item of interest in. Upon such prompts and responses, the objective of applying uniform LoRA module as follows:
lyl
log Po Pol
1=1
3 Methodology
To address the issue of negative transfer associated with conventional LoRA fine-tuning, we introduce the Instance-wise LoRA (iLoRA) fine-tuning framework. This innovative approach adapts the Mixture of Experts (MoE) concept [22, 23] to tailor Large Language Models (LLMs) to individual characteristics sequential recommendation, as illustrated Figure 2. At the core is the integration of multiple experts, each encouraged to capture a specific aspect of user behaviors. Different instances of user behavior (i.e., item sequences) use a gating network to create instance-win wise attention scores over experts. Such attentive experts instantiate the trainable matrices B and A, thus personalizing L. Upon this instance with its individually activated LoRA, A, We fine-tune the LLM to minimize the negative transfer among disparate sequences.

---

3.1 Instance-wise Generation for Sequential Recommendation
Applying a uniform LoRA across the population of sequence instances risks overlooking individual variability and easily causes negative transfer, where distinct sequences might adversely affect each other. This inspires us view the modeling of each individual instance as separate task instead, and customize instance- LoRA module. By doing so, the L.LM-basee recommender is expected align more closely with the behavioral and preference variability of individual users. Formally, for any sequence instance Ixx the autoregressive objective with instance- wise LoRA modules is as:
ly
max AΦ
log (U x,,)),
(8)
(x.y)
where (p(icn) yields ign exxllusive parameters of instance-wise LoRA, as compared to the shared parameters of uniform LoRA (i.e., in Equation (7))
To this end, one straightforward solution is to set up differennt LoRA modules for individual sequences. enabling each module to act the expert tailored to its respective sequence. However, it is impractical terms of resource and time requirements, particularly the number of sequences often reaches the millions. This highlights the need for more scalable solution to address the challenge of sequence-specific customization without excessive computational overhead.
3.2 Instance-wise LoRA with the Mixture of Experts Concept
Instead of establishing various LoRA modules, we implement the mixture-of-expert (MoE) concept [22, 23] to devise our instance-wis LoRA (iLoRA) framework. This framework includes three components: (1) Diverging from the standard LoRA module with up- and down-projectee matrices, we divide each matrix into an array of experts, each encouraged to capture a distinct, hidden aspect of user behavior; (2) For a given sequence instance, use gating network to obtain attention scores across the arrays of up- and down-projection experts, such that distinct sequences are likely to activate different experts; (3) Such an attentive combination of up- and down wown-proecction experts instantiates the weights of LoRA, which are individually customized for the instance of interest. We will elaborate these components one by one.
3.2.1 Splitting Low-Rank Matrices into Experts
Typically, the architectural foundation LoRA is built upon two low-rank matrices: down-projection B E Rd and up-projection A E Rx Here we meticulously divide each projection matrix into an array of experts, as illustrated in Figure 2. Each expert is intended to focus on capturing one specific, hidden aspect of user preference. Formally, splitting the low-rank matrices is as follows:
B B2, .B,],
= (A Akk
where B₁ E &a Xr" and A₁ E IR" xdu are the up- and down-projection pairs for the k-th expert, respectively; r* is the partial rank determined by the total rank of LoRA and a predefined number experts
By dividing individual LoRA modules into specialized experts, we ensure more granular and precise adaptation to user preferences. This segmentation approach allows each expert to focus on specific aspects of user interaction patterns, thereby mitigating the risk of negative transfer that arises from generalized adaptations. We should emphasize that such a segmentation scheme preserves the overall number of parameters equivalent to that of the standard LoRA, therefore preventing the potential overfitting issue.
3.2.2 Generating Instance-wise Attentions over Experts
Having obtained the experts (i.e., up-projection submatrices (B₂ }B down- submatrices (Aa k=1)) construct instance-guided gating function to yield the contribution of each expert tai- lored to a specific sequence. Specifically, for a sequence of historical items izn = [1, f2.... ... in-1),
utilize sequential recommender (e.g.. SASRec [24]) to extract its representation follows:
SR-EMB (icn).
(10)
5

---

Here z E Rd provides a holistic view of the user's behavioral patterns and preferences. Subsequently, ascertain the influence of each expert on distilling behavior patterns from this sequence, we get the contribution scores via a linear transformation with a softmax function:
Softmax (z)),
(11)
where Proj,, (z) W 9 with the trainable transformation matrix W y RR and the softmax function ensures these contributions normalized as the attention scores, thereby preventing any single expert from disproportionately influencing the recommendation; w represents the attention scores over the experts, with the k-th element indicating the contribution of expert K.
By using the sequence representation as the guidance signal, we can get the instance-wise attention scores experts and encourage each expert's contribution closely aligned with the individual variability inherent in the sequence. Moreover, distinct sequences tend to yield different attention scores and activate different experts, while similar sequences incline toward analogous attention scores. By dynamically adapting to wide range of user behaviors and preferences, these experts could specialize diverse aspects of user behaviors and be more adept handling diverse user needs.
3.2.3 Aggregating Mixture of Experts as Instance-wise LoRA
For the instance ien associated with the instance-wise attentions w. we use the mixture-of-experts concept to aggregate the up- and down-projection submatrices from different experts, so as establish the instance-wise LoRA parameters:
AW(zaa)
wi A
(12)
where A (ic encapsulates the adjustments enabled by our iLoAA. The attention score wk- assigned by the gating network (cf. Equation (I1)), reflects the relevance of expert k's contribution to the particular sequence.
We apply such instance-wise LoRA updates the transformer layers of the base LLM, which collectively construct the tunable parameters A Clearly, iLoRA maintains the same total number of parameters the standard LoRA, but dynamically customizes varying LoRA modules for different instances. This dynamic adaptation of parameters ensures that our model remains flexible and responsive the varied preferences and behaviors exhibited by users, effectively managing the complexity inherent in sequential recommendation systems. This approach not only enhances personalization but also improves the predictive accuracy of the recommendation system.
4 Experiments
In this section, we first justify the need to reshape the fine-tuning task with a uniform LoRA module multi-task learning framework for sequential recommendation. We request the dataset from LLaRA [9] and maintain exactly the same experimental settings as described in the original paper. Our study builds upon the main table from the LLaRA paper, using the results reported in the LLaRA paper directly. Here we conduct extensive experiments on various real-world datasets, including LastM [25]. MovieLens [26], and Steam [27], to evaluate the effectiveness of our iLoRA framework. Our analysis includes detailed comparisons of iLoRA against established baseline models, which encompass both traditional sequential recommender models (e g., GRU4Rec [28], Caser [29]. SASRec (24]) and LLM-based recommender models (e.g., Llama2-77 7B [21], GPT-4 [50], MoRec [30], TALLRec [S], LLaRA 19]). ValidRatio [9] and HitRatio@ are used evaluation metrics, to separately quantify the ratios of valid responses over all sequences and relevant items over all candidate items, reflecting the model capability of instruction following and recommendation accuracy. See Appendix A for more details of these baselines, datasets, and metrics. Moreover, we perform a thorough ablation study to identify the key components that enhance iLoRA's performance, focusing particularly on the role of the gating network and expert settings. In nutshell, we would like to answer the following research questions:
RQ1: What is the rationale behind instance-wise LoRA compared to the uniform LoRA module?
RQ2: How does iLoRA perform in comparison traditional sequential recommender systems and LLM-based recommender models?

---

Table 1: The Results of iLoRA compared with traditional sequential recommender models and LLMss-assed methods.
LasiFM
Moviel
Steam HitRatoo@1
ValidRatio
@iRatio@1
Imp.5
ValidRatio
HitRatio@1
Imp.
ValidRatio
Imp.8
Traditional
GRU4Roc
1.0000
0.2616
91.13%
1,0000
0.3750
40.67%
1,0000
0.4168
26.30%
Caser
1.0000
0.2233
123.91%
1,0000
0.3861
36.62%
1,0000
0.4368
20.51%:
SASRec
1.0000
0.2233
123.91%
1,0000
0.3444
53.16%
1,0000
0.4010
31.27%
LL L-M--h
Llama2
0.3443
0,0246
1932.52%
0.4421
0.0421
1152.97%
0.1653
0.0135
3799.261
ChaiRec
1,0000
0.3770
32.63%
0,9895
0.2000
163.75%
0.9798
0.3626
45.17%
MoRec
1.0000
0,1652
202.66%8
1.0000
0.2822
86.92%
1,0000
0.3911
34.59%
TALLRec
0.9836
0,4180
19.62%
0,9263
0.3895
35.43%
0.9840
0.4637
13.52%
LLaRA
1,0000
0,4508
8.51%
0.9684
0.4421
19.32%
0.9975
0.4949
6.36%
Ours
ILoRA
1.0000
0,5000
0.9891
0.5275
0.9981
0.5264
RQ3: What is the impact the designed components (e.g., the gating network, expert settings) on the recommendation performance of iLoRA?
(a) Basic LoRA
(b) iLoRA
(c) Attention Scores
Figure 3: 3a and 3b separately show gradient similarities LLaRA and iLoRA, with sequences partitioned into 8 clusters; 3c exhibits the attention scores over four experts, for ten sequences.
4.1 Investing Rationale of Instance-wise LoRA (RQ1)
We begin by experimenting with LLaRA [9], LLM-basse recommender using uniform LoRA module, to identify a key limitation: negative transfer between significantly different sequences. Next, examine the experts within iL.RA, which employs instance-wise LoRA module, demonstrate the varying attributions of experts when handling different sequences. For more details, see Appendix B.
4.1.1 Negative Transfer in Uniform LoRA & Instance-wise LoRA
Gradient similarity reflects proximity of recommendation sequences [19]. Here we explore whether using LLaRA to perform recommendations conditioned on different sequences exhibits similar loss geometries and vice versa. To achieve this, we use Euclidean distance to control task similarity and gradient similarity measure loss geometry. In Figure 1, symmetric heatmap visually displays the average gradient similarity across LLaRA checkpoints at different training steps. We demonstrate this test in Figure 3. Specifically, observe strong clustering along the diagonal of the gradient similarity matrix for sets recommendation sequences that are closely related in the Euclidean space. Conversely, recommendation sequences that distant in Euclidean space exhibit correspondingly lower gradient similarity. leading to negative transfer.
In contrast, we visualize iLoRA's gradient similarity among the identical clusters. The similarities between some clusters tend to achieve zero scores, indicating iL.oR''s capability to mitigate the negative transfer between significantly different sequences.

---

0.60
0.60
0.55
0.55
HitRatl@81
0.50
0.50
0.15
expert
Moviel ans
0.40
experté
experti
0.40
0.35
Number of Experts
Epoch
(a) Ablation study on the number experts.
(b) Comparison performance over various epochs
Figure 4: 4a illustrates the performance of iLoRA w.r.t. HitRatio @ across different datasets with varying numbers of experts. 4b further demonstrates the HitRatio@) I performance of the model across different epochs during training the Steam dataset with varying numbers of experts.
4.1.2 Expert Showcase in Instance-wise LoRA
In this section, visualize the attention scores of iLRAA's four experts for ten distinct sequences in Figure 3c. Each horizontal bar represents sequence, and the length the segments within each bar indicates the percentage of attention scores assigned each expert. We have several findings:
Sequence Variability:
There is significant variability in expert activation across different sequences. For example, Sequence 4 heavily relies on Expert with a 42.5% activation weight, while Expert only contributes 18.8%, demonstrating distinct preferences for different experts among sequences.
Certain experts have notably high contributions for specific sequences. For instance, in Sequence 2, Expert has a dominant activation weight of 46.7%, indicating that this expert captures the personalized preferences of the user group represented by Sequence
Expert Contribution:
Some sequences exhibit more balanced distribution of activation weights among multiple experts, suggesting collaborative contributions. For example, in Sequence 9, Experts 2 and 3 have similar activation weights of 31.1 % and 31.5%, respectively, indicating their joint influence the recommendations.
Collaborative Contribution:
These observations demonstrate that iLoRA effectively adjusts expert activation based on the charac- teristics of each sequence.
4.2 Performance Comparison (RQ2)
This section comprehensively compares iLoRA against some traditional and LLM-based recom- menders. We conduct a holistic evaluation, consid- ering metrics of both HitRatio@ and ValidRatio across LastFM, MovieLenns, and Steam datasets to demonstrate the effectiveness of iL The results of this comparison are summarized in Table 1.
Randens Eabedding
Stnysler LeRA Exbroling
0.50
0.43
Our findings indicate that iLoRA consistently outper- forms these baseline models across the three datasets. Specifically, iLoRA achieves the highest HitRatio metrics of 0.5000, 0.5275, and 0.5264 on the LastFM, MovieLens, and Steam datasets, respectively. These results demonstrate the efficacy of leveraging sequence representations as guidance signals to fine-tune LoRA parameters, enabling personalized recommendations at the parameter level.
Figure Effects of iLoRA's components
4.3 Ablation Study (RQ3)
In this section, we analyze the effectiveness of the main components of iLoRA in Section 4.3.1. Subsequently, in Section 4.3.2, we conduct an in-depth investigation and analysis of how varying the number of experts affects the performance of iLoRA.

---

4.3.1 Effects of Gating Network
Here we explore the influence of sequence representation on the gating network and MoE. Going beyond the sequence-tailore representation, we test two variants: using aundom-nitiialized and ceken-collppsed embeddings as the guidance. As Figure 5 shows, using sequence representation as the guidance consistently outperforms the other variants across three datasets, illustrating the rationale of our gating network and the benefits for the MoE combination.
4.3.2 Effect of Expert Numbers
In this section, we investigate how iLoRA would react to the number of experts. As depicted in Figure 4a, our model achieves optimal performance when the number of experts set to 4. Increasing the number of task experts does not necessarily correlate with enhanced performance. Specifically, employing only 2 experts does not significantly improve the HitRatio@* metries on Steam and Moviel datasets, while showing a slight decrease on LastFM. However, with the increase to 4 experts, the model exhibits its best performance across all three datasets, notably surpassing the 2-expert variant. To elaborate, on LastFM, MovieLens, and Steam datasets, the performance of the 4-expert variant exceeds that of the 2-expert variant by 5.2%, 6.4% and 2.2%, respectively. When the number of experts is further increased to 8, the performance resembles that of the 4-expert scenario or even shows slight decrease, This suggests that the benefits of increased capacity gradually converge adapt 4 experts as the default setting.
utilize more experts. Consequently,
Furthermore, we analyzed the performance across different numbers of experts various epochs. on the Steam dataset. It is evident that as the number of experts is set to 1, 2, and 4, the overall recommendation performance of the model steadily improves with training progress. Under this configuration, the @itRatio@ values exhibit a positive correlation with the number of experts. However, when the number of experts reaches 8, the data indicate that the model rapidly achieves a decent performance, but subsequent HitRatio @1 values do not show significant improvements with increasing epochs. We speculate that as the number of experts increases to 8, the model may overly focus on personalized user behaviors, leading to decrease in generalization ability and premature overfitting.
5 Conclusion
In this paper, we introduced instance-wise LoRA (iLoRA), a novel fine-tuning framework designed to address the challenges posed by the substantial individual variability user behaviors within sequential recommendation systems. By integrating the mixture of experts (MoE) concept into the basic LoRA module, iLoRA dynamically adjusts diverse user behaviors, thereby mitigating the negative transfer issues observed with standard single-module LoRA approaches. iLoRA represents significant advancement in the application of large language models to sequential recommendation tasks. By incorporating mixture of expert frameworks within the LoRA module, iLoRA provides a more nuanced and effective means of tailoring recommendations to individual user preferences, paving the way for more personalized and accurate recommendation systems.
Limitation
While iLoRA demonstrates promising results, there are several limitations to consider. First, our experiments are constrained by computational resources, limiting the exploration of a larger number of expert combinations and their potential impact on recommendation performance. Second, we do not extensively investigate the effects of using hard routing for recommendations with large number of experts. Finally, our study focused on sequential recommendation tasks, and the applicability of iLoRA to other types of recommendation systems or domains remains to be explored. These limitations suggest that further research is needed to fully understand the scalability and effectiveness of iLoRA with more complex expert configurations.

---

7 Broader Impact
Our proposed method, Instance-wise LoRA (iLoRA), advances sequential recommendation systems by aequence-tailoree recommendations. By leveraging the Mixture of Experts (MoE) framework, iLoRA streamlines the user experience, reduces decision fatigue, and promotes inclusivity in online spaces. Its instance-wise adaptation mechanism ensures diverse content exposure, fostering more enriched online discourse. Beyond recommendations, iLoRA's principles extend to education, healthcare, and e-commerce, offering customized solutions in various domains. Overall, iLoRA represents step forward enhancing user experience and promoting inclusivity the digital landscape.
However, despite the advancements offered by iLoRA, is essential to acknowledge potential drawbacks. Algorithmic biases present in the training data may persist, potentially amplifying existing biases recommendations. Moreover, over-reliance on customization could lead to filter bubbles, limiting exposure to diverse viewpoints and serendipitous discovery. Additionally, concerns regarding privacy arise due the collection and analysis of user data for personalized recommendations, necessitating careful consideration of ethical implications in its deployment.
Acknowledgments and Disclosure of Funding
This research is supported by the National Natural Science Foundation of China (922701 14, 62302321) and the National Science and Technology Major Project (20232D0121102), This research also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.
10

---

References
[1] Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Trans. Inf. Syst. 2020.
[2] Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. A survey on large language models for recommendation. World Wide Web (WWW). 2024.
[3] Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5). In RecSys, 2022.
[4] Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian J. McAuley. Text is all you need: Learning language representations for sequential recommendation, In KDD, 2023.
[5] Kegin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework align large language model with recommendation. In RecSys, 2023.
[6] Jianchao Ji, Zelong Li, Shuyuan Xu, Wenyue Hua, Yingqiang Ge, Juntao Tan, and Yongfeng Zhang. Genrec: Large language model for generative recommendation. In ECIR (3), volume 14610 of Lecture Notes in Computer Science, 2024.
[7] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt's capabilities in recommender systems. In RecSys, 2023.
[8] Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian J. McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In ECIR (2), volume 14609 of Lecture Notes in Computer Science, 2024.
[9] Jiay Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, and Xiang Wang. Llara: Aligning large language models with sequential recommenders. CoRR, abs/23 0ans/2312.02445, 2023.
[10] Xinhang Li, Chong Chen, Xiangyu Zhao, Yong Zhang, and Chunxiao Xing. E4srec: An elegant effective efficient extensible solution of large language models for sequential recommendation. CoRR, 0ts/2312.022433 2023.
[11] Bohao Wang, Feng Liu, Jiawei Chen, Yudi Wu, Xingyu Lou, Jun Wang, Yan Feng, Chun Chen, and Can Wang. LLM4DSR: leveraing large language model for denoising sequential recommendation. CoRR, 2024.
[12] Yu Cui, Feng Liu, Pengbo Wang, Bohao Wang, Heng Tang, Yi Wan, Jun Wang, and Jiawei Chen. Distillation matters: Empowering sequential recommenders to match the performance of large language models. In Proceedings of the 18th ACM Conference Recommender Systems, RecSys 2024, Bari, Italy, October 14-18, 2024, 2024.
[13] Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On softmax direct preference optimization for recommendation. In NeurIPS. 2024.
[14] Leheng Sheng, An Zhang, Yi Zhang, Yuxin Chen, Xiang Wang, and Tat-Seng Chua. Language models encode collaborative signals in recommendation. arXiv preprint 2rX 2024.
[15] Zhenrui Yue, Sara Rabhi, Gabriel de Souza Percira Moreira, Dong Wang. and Even Oldridge. Llamarec: Two-stage recommendation using large language models for ranking. CoRR, Abs/2311.02089 2023.
[16] Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. Collm: Integrating collaborative embeddings into large language models for recommendation. CORR, abs/2310. 19488, 2023.

---

[17] Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt predict paradigm (P5). In RecSys, 2022.
[18] Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Recommendation as instruction following: A large language model empowered recommendation approach. CoRR, abs/2305.07001, 2023.
[19] Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi- task optimization in massively multilingual models. ICLR, 2021.
[20] Haowen Wang, Tao Sun, Congyun Jin, Yingbo Wang, Yibo Fan, Yunqi Xu, Yuliang Du, and Cong Fan. Customizable combination of parameter-efficient modules for multi-task learning. In The Twelfth International Conference on Learning Representations, 2023.
[21]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marein Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and tune- chat models. CoRR, abs/2307.0922 2023.
[22] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In ICLR (Poster), 2017.
[23] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23, 2022.
[24] Wang-Chenn Kang and Julian J. McAuley. Self-attentive sequential recommendation. In ICDM, 2018.
(25) Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik, editors. Proceedings of the 2nd International Workshop on Information Heterogeneity and Fusion in Recommender Systems, HetRec '11, Chicago, Illinois, USA, October 27, 2011, 201 ACM.
[26] F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5, 2016.
[27] Wang-Cheng Kang and Julian J. McAuley. Self-attentive sequential recommendation, In ICDM, pages 197-206. IEFE Computer Society, 2018.
[28] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. In ICLR, 2016.
[29] Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In WSDM, 2018.
[30] Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? ID- vs. modality-based recommender models revisited. In SIGIR, 2023.
[31] Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Trans. Inf. Syst. 39, 2020.
12

---

[32] Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet A. Orgun. Sequential recommender systems: Challenges, progress and prospects. In IJCAI, 2019.
[33] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
[34] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry. Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao. Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022.
[35] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[36] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In ACL, 2019.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
[38] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample. Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/231 10.00225, 2023.
[39] Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An cestucctionfoolloinn llama model. 2023.
[40] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2, 2023.
[41] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrickk and Graham Neubig. To- wards unified view of parameter-effiiient transfer learning. In ICLR. OpenReview.net, 2022.
[42] Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. GPT understands, too. CoRR, abs/2103.10385, 2021.
[43] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACI/UCCNPP (1), 2021.
[44] Nam Hyeon- Woo, Moon Ye-Bin, and Tac-Hyun Oh. Fedpara: Low-rank hadamard product for communication-cfficii federated learning. In ICLR, 2022.
[45] Shin-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard B. W. Yang, Giyeong Oh, and Yanmin Gong. Navigating eext-to-image customization: From lycoris fine-tuning to model evaluation. CoRR, abs/2309, 14859, 2023.
13

---

[46] Edward J. Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. 2022.
[47] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-effiient fine-tuning is better and cheaper than in-context learning. In NearlPS, 2022.
[48] Chongjie Si, Xiaokang Yang, and Wei Shen. See further for parameter efficient fine-tuning by standing on the shoulders of decomposition. CoRR, 2024.
[49] Chongjie Si, Zhiyi Shi, Shifan Zhang, Xiaokang Yang, Hanspeter Pfister, and Wei Shen. Unleashing the power of task-specific directions in parameter efficient fine-tuning, CoRR, 2024.
[50] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint TTXIv:2303.08874, 2023.
[51] Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR, abs/1907 Abs//197,00019, 2019.
[52] Chenwang Wu, Xiting Wang, Defu Lian, Xing Xie, and Enhong Chen. A causality inspired framework for model interpretation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023. ACM, 2023.
[53] Lei Chen, Le Wu, Kun Zhang, Richang Hong, Defu Lian, Zhiqiang Zhang, Jun Zhou, and Meng Wang. Improving recommendation fairness via data augmentation. In Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 4 May 2023. ACM, 2023.
[54] Chongming Gao, Shigi Wang, Shijun Li, Jiawei Chen, Xiangnan He, Wenqiang Lei, Biao Li, Yuan Zhang, and Peng Jiang. CIRS: bursting filter bubbles by counterfactual interactive recommender system. ACM Trans. Inf. Syst., 2024.
[55] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2023.
[56] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David S. Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. CoRR, abs/2303. 175644 2023.
[57] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip An- drew Mansfield, Blaise Agüera Arcas, Dale R. Webster. Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. CORR, abs/2212.13111 2022.
[58] Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. CoRR, abs/2306. 16092, 2023.
[59] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87, 1991.
[60] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, An- dré Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In NeurIPS, 2021.
[61] Zhili Liu, Kai Chen, Jianhua Han, Lanqing Hong, Hang Xu, Zhenguo Li, and James T. Kwok. Task cask-customized masked autoencoder via mixture of cluster-conditional experts. In ICLR, 2023,
14

---

[62] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Mach. Learn. Res., 23, 2022.
[63] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In ICLR, 2021.
[64] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Mul- timodal contrastive learning with limoe: the language-image mixture of experts. In NeurIPS, 2022.
[65] Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. In EMNLP (Findings), 2023.
15

---

A Description of Figure 1
This figure illustrates the gradient similarity of LoRA modules across all training steps. We utilize signals from the collaborative space to partition the sequence dataset into 8 clusters. Euclidean distance is employed to evaluate the proximity between clusters, whereas gradient similarity is measured to assess geometrie loss. Clusters that are closer the collaborative space are depicted closer together in the left side of the figure, with darker-colored cells indicating higher gradient similarity. On the right side the figure, wc conducted a case study on three sets of data with pairwise cosine similarities of 0.86 and -0.75. For sequences containing more than three movies of the same genre, we performed a cross-mas mots-mntthingg analysis. was observed that the two user sequences from the cluster with cosine similarity of 0.86 both exhibited strong interest in thriller movies, sharing two identical items in their interaction histories at the same time. In contrast, the two user sequences from the cluster with cosine similarity of -0.75 did not demonstrate any noticeable preference similarities.
B Experimental Design and Evaluation
Datasets. To validate the generalization ability of iLoRA, we conducted extensive experiments on three datasets derived from real-world recommendation scenarios: LastEM, MovieLens, and Steam:
LastFM A popular music dataset for recommendation research, featuring histories of user-artist interactions, user demographic details, artists' names and associated social tags.
MovieLens A widely used benchmark in recommendation systems, containing user ratings and metadata for movies.
Steam collection of user interaction data from the Steam gaming platform, featuring information on game ownership, playtime, and user reviews.
Baselines.
We compared iLoRA with several models, including traditional sequential recommendation models and those based on Large Language Models (LLMs), such GRU4Rec[28], Caser, SASRec[24]] Llama2[21], GPT-4, MoRec, TallRe[[5], and LIaRA[9]. GRU4Rec, Caser and SASRec utilize recurrent neural networks, convolutional neural networks, and Transformer encoders, respectively, to capture sequential patterns in user behavior. Llama2, Meta's 's open-source language model, building on the original Llama to deliver improved performance. We fine-tune the 7B version the model for our experiments. GPT-4, OpenAl's advanced language model, has held the state-of-the-art position across various tasks for an extended period. We directly utilize API for our experiments. MoRec uses pre-trained modality encoders to capture item-specife features, improving recommendation accuracy. TALLRec guides LLMs through fine-tuning recommendation corpora. LlaRA further improves the recommendation effectiveness of LLMs by aligning collaborative signals to the text space through a hybrid prompt method.
Training Protocol.
In study, fine-tune the Llama2-7B [21] model to validate approach. Experiments for traditional sequential recommendation baseline models are conducted on a single Nvidia A40, while iLoRA framework is implemented on single Nvidia A100. All experiments are carried out using Python 3.8 and sytorchL.ightning 1.8.6.
Evaluation Protocol.
To assess the performance of iLoRA and baseline models, we construct candidate sets for each sequence by randomly selecting 20 non- non-interacted items while ensuring the inclusion of the correct next item. Performance is evaluated using the HitRatio@) metric, wherein models attempt identify the correct item from the candidate set. LLM-based recommenders, when provided with appropriate prompts, generate single candidate item. Conversely, traditional models are adapted to select the item with the highest probability. Additionally, we introduce the ValidRatio metric to quantify the proportion of valid responses (i.e., items within the candidate set) across all sequences. This metric serves evaluate the models' adherence to instructions accurately.
16

---

C Experimental Setup of RQ1
We extend the previous research setup to train models on multi-task scenarios[51]. Specifically, we jointly train recommendation sequences in a basic LoRA training framework. We use an effective batch sizes of 128 sequences. The recommendation sequences are divided into multiple tasks using representations derived from the sequential recommendation model SASRec, with a dimension of 64.
To investigate large-scale multi- multi-tasking in sequential recommendation tasks, sample 40k sequences from the Steam dataset. We clustered these sequences into 8 sub-datasets using Euclidean distance. At checkpoints across 1K training steps, we measured the pairwise cosine similarity of model gradients for all sequences. We averaged the LoRA gradients that were bound to the same modules, such as gateproj.
D Related Work
Large Language Models Sequential recommendation [24, 32, 31, 52-54] aims to predict the user's next likely item of interest by analyzing their past interaction patterns and aligning with their preferences. Recent years have witnessed a surge of activity in language modeling research, establishing it as a cornerstone for both understanding and generating language. This momentum has given rise to new breed of language models (LMs), including notable works such as BERT [36], GPT-3 [35], LLama [37], LLama2 [21], Mistral-7B [38], Alpaca [39], and Vicuna [40]. These LMs. predominantly based on the Transformer architecture, have demonstrated remarkable versatility, exemplified by models like BERT [36] and T5 [34], owing to their extensive training corpus. A significant stride in this domain has been the exploration of scaling effects, with researchers pushing the boundaries by augmenting both the parameter and training corpus scales to unprecedented magnitudes, encompassing billions of parameters and trillions of training tokens [37, 21, 40, 35, 55]. These Large Language Models (LLMs) exhibit substantial performance enhancements and showcase unique capabilities, including but not limited to common sense reasoning and instruction following. Moreover, the development of nomain-specifie LLMs further enriches this landscape. Models tailored to specific domains, such finance [56], medicine [57], and law [58], amalgamate domain expertise with the inherent commonsense knowledge of general LLMs. These advancements not only broaden the scope of LLM applications but also inspire exploration into their potential utility in recommendation systems.
Mixture of Experts Mixture of Experts. MoE models [59, 22, 23] are considered as effective way to increasing the model capacity terms of parameter size. In MoE, certain parts of the model are activated while the computation is kept the same close to its dense counterpart. Recently, it has been thoroughly investigated the field computer vision [60, 61], natural language processing [62, 63] and multi-modal learning [64, 65].
E Statistics
In our experimental settings, we closely follow the setup of prior works [9] meaningful comparison. Specifically, for all conventional sequential recommendation baselines, we employ the Adam optimization algorithm, establishing learning rate of 0.001, an embedding dimension of 64, and batch size of 256, respectively. Furthermore, incorporate L2 regularization, and the regularization coefficient is fine-tuned through a grid search over a specified set of possible values. We report the average results from five different random seeds, specifically selected from the set, To reduce the effect of randomness.
to ensure a fair and
For experiments involving methods based on large language models (LLMs), incorporate warm-up strategy for the learning rate, which begins the training process with an initial rate set at fraction of the maximum learning rate. Specifically, the maximum learning rates are set to 2e for the LastFM dataset and le - for the MovieLens and Steam datasets. Notably, the projector parameters updated using the same learning rate curve integrated with iLoRA during training.
We adopt cosine annealing scheduler to adjust the learning rate across training steps, which facilitates a smooth decrease in the learning rate, thereby improving stability during training. Additionally, half-precision computation umixed-precision training) is used to enhance memory efficiency and accelerate the training process.
17

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
the checklist one of the factors in their evaluation.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the con- tributions and scope of our paper. The abstract succinetly summarizes our approach, the Instance-wise -ankknk Adaptation (iLoRA), and its application in sequential recommenda- tion systems. also highlights the integration with the Mixture of Experts (MoE) framework. providing clear overview of the method's novelty and effectivvness. The introduction elaborates on the motivation, background, and significance of our contributions, ensuring that the claims align with the detailed discussions and results presented in the subsequent sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
18

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work performed limitation 6. By explicitly acknowledging these limitations, we provide balanced view of our work and suggest directions for future research to address these challenges.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: we provide the full set of assumptions and a complete (and correct) proof in our methodology part 3. The assumptions are clearly stated, and the proof' are detailed to ensure correctness and reproducibility.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
19

---

4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose all the information needed in Appendix A and Appendix B. This includes details about the datasets, experimental setups, hyperparameters, and evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access to the data and code. And we explain our set in Appendix and Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Seliii//guides/Codeee iosoessoosssiiooooooocoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
20

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips... /ttpss.ccppulllt.teee.. wwtpccccppppppptec..eecccc hpppui/gguudess..../ppppppopaale gwt.///ie/////e/ SCsdeSubdmmmmiii aassonnPoicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Appendix B and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We do it in the Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
21

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: For each experiment proviede sufficient information in our experiments part, Appendix E, Appendix B, Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http://////eueee /ttp///nurrppppeeee wttpippp..pppppeee cc/public/Ethpi wtt.hhiihiiieeeeeeeeeee
Answer: [Yes]
Justification: Our research fully conforms to the NeurIPS Code of Ethics. We have carefully reviewed the guidelines and ensured that all aspects of our research, from data collection and usage to experimental conduct and reporting, adhere to ethical standards. We maintain transparency in our methodology, respect for privacy. and fairness in our experimental design. ensuring that our work aligns with the ethical principles outlined by NeurIPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses the potential positive and negative societal impacts of work in the "Broader Impacts" section. By acknowledging these impacts, we provide balanced view of our work and suggest mitigation strategies to address potential negative outcomes.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
22

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We conduct experiments on recommendation tasks using publicly available datasets and build upon existing open-source methods. Therefore, the risk of misuse is minimal, and specific safeguards for data or model release are not necessary. Our focus remains on transparency and reproducibility, ensuring that our work can be used responsibly by the research community.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We fully adhere to these requirements by properly crediting the creators or original owners of all assets used in our paper. Each asset, including datasets, code, and models, is accompanied by citation to the original source, and the specific versions used are stated. We explicitly mention the licenses and terms of use for each asset, ensuring compliance with their respective usage guidelines.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We offer code.
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We are not involved in these risks as we are only engaged in recommendation tasks. Our research does not include crowdsourcing experiments or research with human subjects, thus this question is not applicable to our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We are not involved in these risks as we are only engaged in recommendation tasks. Our research does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.


Reviews:
[{"review_id": "iKRrR1Lo6h", "reviewer": "Reviewer_cnYP", "summary": "The paper addresses the challenge of personalizing language models for sequential recommendation tasks, where user behaviors exhibit significant individual variability. The proposed solution, Instance-wise LoRA (iLoRA), adapts the mixture of experts concept to tailor LLMsfor this variability.", "questions": "see above", "limitations": "see above", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The iLoRA framework proposes splitting the standard LoRA module into multiple experts, each capturing specific aspects of user behavior. Key components of this solution include:\n\n- Splitting Low-Rank Matrices into Experts: Dividing the projection matrices into an array of experts to capture different aspects of user preferences.\n- Instance-wise Attentions: Using a gating network to generate attention scores for each expert based on the user's historical item sequence. This customization helps in adapting the model more accurately to individual user behaviors.\n- Fine-tuning with Hybrid Prompting: Applying the personally-activated LoRA to fine-tune the LLM, thus mitigating negative transfer and enhancing the model's adaptability.\n\nThe solution leverages the mixture of experts approach to ensure scalability and efficiency, maintaining the same total number of parameters as the standard LoRA to avoid overfitting.\n\nThe starting point of the article's issue is reasonable, and personalized LoRA is an impressive idea.", "weaknesses": "# \n\n- The technical contribution of the paper is insufficient. For example, simply splitting the low-rank matrices into multiple experts representing different aspects seems inadequate and has room for expansion. It could be more refined by setting the number of experts based on different user clusters and controlling the low-rank size of the corresponding experts according to the number of users in each category.\n\n- The paper proposes a framework rather than a model. However, in the main experiments, the authors compare it with numerous models instead of fine-tuning-based methods, which is puzzling. Additionally, the framework proposed in the paper seems to be a general one, or are there any challenges that limit it to sequential recommendation? It might be worthwhile to extend beyond sequential recommendation tasks to broader scenarios.\n\n- The effectiveness of this method needs further validation in extremely large-scale datasets or user groups, or with more evaluation metrics.\n\n- It is hoped that the code will be made open-source for careful review. I may adjust my score based on this.\n\n- The reproducibility does not seem to be good enough, and no code link was found", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments, which greatly improve our paper. Below we provide the point-to-point responses to address your concerns and clarify the misunderstandings of our proposed method.\n\n---\n\n> **C1: More Refined by Integrating Experts with Clustering**\n\nThanks. Based on your suggestions, we have made clarifications and additional experiments:\n\n- **Clarification of Our Technical Contribution**: We agree that, for instance-wise LoRA, a more refined idea will better show its potential. However, our paper focuses on a **simple yet effective implementation** of this idea.\n\n- **Clarification of User Clustering**: The user clustering (in Sec Introduction & Experiments) operates independently from the expert design (in Sec Method).\n  - The role of clustering is to show whether a uniform LoRA causes negative transfer and  iLoRA can mitigate it, where the number of user clusters is a hyperparameter.\n  - The role of experts is to represent different aspects of users, such that their mixture activated with personal attentions makes the LLM specialized in a user instance, where the number of experts is another hyperparameter.\n\n- **Additional Experiment of More Refined Method**: Based on your comments, we conducted additional experiments where the total rank of all experts is 16, while adjusting each expert's low-rank size, as **Fig 1** in one-page uploaded pdf shows.\n\n  - **Stability and Complexity**: Integrating clustering adds complexity and potential instability due to initialization sensitivity and the dynamic nature of user behavior.\n\n  - **Empirical Observations**: Our findings suggest that uniform expert size distribution in our current model approximates the benefits of clustering but with greater simplicity and stability.\n\n---\n\n> **C2: Missing Comparison with Fine-tuning Methods & Generalization in Broader Scenarios**\n\nThanks. Below we clarify the misunderstanding of compared baselines and show the results of additional experiments.\nThere are various parameter-efficient fine-tuning strategies [1], such as LoRA, adapters, and prefix tuning.\n\n- **Fine-tuning Baselines**: We compared iLoRA with the fine-tuning baselines (TALLRec, LLaRA). Specifically, TALLRec tunes LoRA on text-only prompts, and LLaRA tunes LoRA along with a simplified adapter on hybrid prompts.\n- **Additional Experiments of More Fine-tuning Baselines**: We added an additional baseline, which adopts prefix-tuning for recommendation.\n\nWe visualized the fine-tuning paradigms in **Fig 5** in the one-page uploaded PDF and present their results in the table below:\n\n|           |                    | LastFM     |             | MovieLens  |            | Steam      |            |\n| :-------- | ------------------ | ---------- | ----------- | ---------- | ---------- | ---------- | ---------- |\n|           |                    | ValidRatio | HitRatio @1 | ValidRatio | HitRatio@1 | ValieRatio | HitRatio@1 |\n| TALLRec   | LoRA               | 0.9836     | 0.4180      | 0.9263     | 0.3895     | 0.9840     | 0.463      |\n| PrefixRec | Prefix Tuning      | 0.9627     | 0.3754      | 0.8836     | 0.2973     | 0.9273     | 0.4108     |\n| LlaRA     | LoRA + Adapter     | 1.0000     | 0.4508      | 0.9684     | 0.4421     | 0.9975     | 0.4949     |\n| iLoRA     | instance-wise LoRA | 1.0000     | 0.5000      | 0.9891     | 0.5275     | 0.9981     | 0.5264     |\n\nClearly, over the three datasets, iLoRA outperforms the baselines using various fine-tuning strategies (LoRA, LoRA+adapter, prefix-tuning) in terms of the HitRatio@1 metric.\n\n- **Generalization of iLoRA to Broader Scenarios**: We are extending the testing of iLoRA on the collaborative filtering task. Due to the time constraints of the rebuttal phase, full results are still pending. We plan to present comprehensive findings during the discussion phase.\n\n---\n\n> **C3: Evaluation on Larger Datasets & More Metrics**\n\nThanks. We clarify that the Steam dataset tested in our paper is one of the largest datasets currently used in the LLM-based recommendation domain. Following your suggestions, we are running additional experiments on larger datasets. However, due to the time limit of the rebuttal phase, we haven't collected all the results and will update more results during the discussion phase if you are interested.\n\nWe agree that additional evaluation metrics are needed. Beyond HitRatio@1, we added HitRatio@K and NDCG@K as the metrics, where K=3 and 5. The table below demonstrates that LoRA (iLoRA) outperforms other baselines across these metrics, in the LastFM and MovieLens datasets.\n\n|         | LastFM     |            |        |        | MovieLens  |            |        |        |\n| :------ | ---------- | ---------- | ------ | ------ | ---------- | ---------- | ------ | ------ |\n|         | HitRatio@3 | HitRatio@5 | NDCG@3 | NDCG@5 | HitRatio@3 | HitRatio@5 | NDCG@3 | NDCG@5 |\n| GRU4Rec | 0.4370     | 0.4964     | 0.3544 | 0.4110 | 0.4831     | 0.5584     | 0.4075 | 0.4702 |\n| Caser   | 0.4445     | 0.4918     | 0.3564 | 0.4232 | 0.4892     | 0.5603     | 0.4134 | 0.4724 |\n| SASRec  | 0.4253     | 0.4792     | 0.3382 | 0.4073 | 0.4256     | 0.5132     | 0.3881 | 0.4239 |\n| TALLRec | 0.6814     | 0.7473     | 0.3900 | 0.4650 | 0.4874     | 0.5408     | 0.4290 | 0.4601 |\n| LlaRA   | 0.7223     | 0.7862     | 0.6016 | 0.6972 | 0.5505     | 0.6189     | 0.4752 | 0.5067 |\n| iLoRA   | 0.7873     | 0.8350     | 0.6894 | 0.7530 | 0.6464     | 0.7084     | 0.5603 | 0.5905 |\n\n---\n\n> **C4 & 5:  No Open-Source Code Link Available**\n\nThanks. Please find our codes in the supplementary material. Following your suggestions, we have uploaded the codes, datasets, and checkpoints to an open-source repository accessible via the anonymous link: https://anonymous.4open.science/r/iLoRA-8C57.\n\n[1] Towards a Unified View of Parameter-Efficient Transfer Learning. ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DEcWXTm1H9", "reviewer": "Reviewer_59LV", "summary": "The paper introduces iLoRA, which combines LoRA with user representation-guided mixture of expert architecture. The motivation is clear and the writing is good. Extensive experiments are conducted on three public datasets, demonstrating the performance of the proposed method.", "questions": "Please find the details in \"Weaknesses\".", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Timely study on large language models and sequential recommendation.\n2. The motivation of the proposed method, i.e., should be more personalized LoRA parameters, is clear and convincing.\n3. The writing is good and the paper is easy to follow.\n4. Experiments are conducted on three public datasets, demonstrating the performance of the proposed iLoRA technique.\n5. Code is available during the reviewing phase.\n\n=== Update After the Rebuttal Phase ===\n\nThe rebuttal includes new results and clarifications, which are helpful in addressing my concerns. In recent days, I also conducted experiments using the provided code, and the results further verify the paper’s conclusions. As a result, I will raise my rating and vote for acceptance.", "weaknesses": "1. The evaluation setting is kind of fragile. For each input sequence, only 20 candidate items are randomly selected as negative samples. In this setting, the results may be influenced a lot by randomness. In addition, if the proposed method only applies to a candidate size of 20 items, the value of this work to the real recommender systems can be doubted.\n2. References of MoE are limited. The related works about MoE are quite concise. The author(s) are encouraged to discuss more about how MoE is applied to LLMs and recommendation models.\n3. Presentation issues.\n    1. Lines 554 - 562 and lines 563 - 570 are duplicated.\n    2. Figure 3 (c) is suggested to be replaced by a vector graph to make it more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and valuable comments. Your main suggestions about the evaluation setting help us substantiate wide applicability of our proposed iLoRA. To address your concerns, we have detailed our responses point-to-point below. \n\n---\n\n> **C1: Fragile Setting of Sampled Evaluation**\n\nThanks so much for your great suggestions! The issue you raised is highly critical, which is a common limitation in current LLM-based recommendation research. Moreover, we are actively exploring methods such as codebook approaches to address full-rank recommendation scenarios. As the following table shows, preliminary experimental results are promising in this regard.\n\n|    | Amazon-Sports |      |    |    |\n| :--- | --- | --- | --- | --- |\n|              | HitRatio@5    | HitRatio@10 | NDCG@5 | NDCG@10 |\n| SASRec       | 0.0324        | 0.0547      | 0.0182 | 0.0247  |\n| Our new work | 0.0596        | 0.0784      | 0.0389 | 0.0477  |\n\nWe note your mention of the experimental setting where we select the next item from candidate items. In traditional recommendation research, there are excellent studies that adopt sampling-based evaluation methods. \n\nInspired by your query about whether our iLoRA training framework is limited to tasks involving 20 candidate items for next-item prediction, we have extended our experiments on the LastFM and MovieLens datasets to evaluate results with 30 and 40 candidate items, respectively. We benchmarked our approach against TALLRec and LlaRA, with detailed results presented in the table below. We conducted five repetitions of our experiments using different random seeds and calculated a p-value, which was less than 0.05, demonstrating the stability of our proposed iLoRA framework.\n\n\n|         | MovieLens                 |         | LastFM                    |                           |\n| :--- | --- | --- | --- | --- |\n|         | HitRatio@1(30 candidates) | HitRatio@1(40 candidates) | HitRatio@1(30 candidates) | HitRatio@1(40 candidates) |\n| TALLRec | 0.3277                    | 0.2833                    | 0.3461                    | 0.2972                    |\n| LlaRA   | 0.3513                    | 0.3250                    | 0.3772                    | 0.3250                    |\n| iLoRA   | 0.4066                    | 0.3666                    | 0.4283                    | 0.3735                    |\n\nBesides, for the LastFM and MovieLens datasets, we have expanded the evaluation metrics, including HitRatio@3, HitRatio@5, NDCG@3, and NDCG@5, as the following table shows.\n\n|         | LastFM     |            |        |        | MovieLens  |            |        |        |\n| :--- | --- | --- | --- | --- | --- | --- | --- | --- |\n|         | HitRatio@3 | HitRatio@5 | NDCG@3 | NDCG@5 | HitRatio@3 | HitRatio@5 | NDCG@3 | NDCG@5 |\n| GRU4Rec | 0.4370     | 0.4964     | 0.3544 | 0.4110 | 0.4831     | 0.5584     | 0.4075 | 0.4702 |\n| Caser   | 0.4445     | 0.4918     | 0.3564 | 0.4232 | 0.4892     | 0.5603     | 0.4134 | 0.4724 |\n| SASRec  | 0.4253     | 0.4792     | 0.3382 | 0.4073 | 0.4256     | 0.5132     | 0.3881 | 0.4239 |\n| TALLRec | 0.6814     | 0.7473     | 0.3900 | 0.4650 | 0.4874     | 0.5408     | 0.4290 | 0.4601 |\n| LlaRA   | 0.7223     | 0.7862     | 0.6016 | 0.6972 | 0.5505     | 0.6189     | 0.4752 | 0.5067 |\n| iLoRA   | 0.7873     | 0.8350     | 0.6894 | 0.7530 | 0.6464     | 0.7084     | 0.5603 | 0.5905 |\n\n---\n\n> **C2: Limited Reference of MoE**\n\nThank you for highlighting this valuable point. Following your feedback, we will revise the relevant content in the Related Work section. Due to space constraints, detailed citation information will be included in the References.\n\nOur approach to integrating MoE with LLM considers three perspectives:\n\n- **NLP Perspective**: Mixture-of-Experts modifies feedforward neural network layers into sparse activation experts, significantly expanding model capacity without a substantial increase in computational costs. Recent explorations of MoE have evolved from sample-level to token-level MoE, with most works aiming to scale up model parameters while reducing computational overhead. Our approach differs significantly. We employ a MoE-like structure that nearly preserves total parameter count to address negative transfer in LLM-based recommendation systems.\n- **LoRA+MoE Perspective**: In the era of LLMs, researchers have integrated MoE concepts into PEFT methods to enhance model performance. LoraHub trains multiple LoRA models and selects different LoRA combinations based on data type during inference. MOELoRA improves model efficiency in medical multitasking by incorporating MoE structures. However, these methods require data types as input to the router during training, which necessitates prior knowledge of data types for selecting LoRA combinations during inference. Our approach is distinct as iLoRA is an end-to-end framework that does not require a priori knowledge of data types for inference.\n- **Recommendation Perspective**: Traditional recommender systems, exemplified by MMoE, demonstrate remarkable capabilities in task scenarios. With the advent of large models, methods like OneRec leverage MoE structures to design expert combinations for multi-domain user embedding, integrating collaborative knowledge into LLMs. Our approach innovates by combining MoE principles with PEFT in LLM-based recommendation scenarios, addressing negative transfer effects caused by individual user differences during fine-tuning. We further propose user representation-guided routing for instance-wise recommendations.\n\n---\n\n> **C3: Typo & Presentation issues**\nThank you for reviewing our work so conscientiously, and we sincerely apologize for the oversight. We will carefully review the presentation issues and make corrections in the revised version. Additionally, we will review the presentation of the entire manuscript.\n\nOnce again, we appreciate your thorough review and conscientious feedback on our work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUoVfa96Sz", "reviewer": "Reviewer_7niz", "summary": "The author focus on extend sequential recommendation task with the help of large language models. The author proposed instance-wise LoRA and integrate with mixture of experts framework to capture specific interests of user preferences. Experiments results on two benchmark datasets demonstrate the effectiveness of proposed model.", "questions": ". How do you partition the sequence data into 8 clusters? Clustering by calculating Euclidean distance of what? is the output embeddings of SASRec? Explain why you do that.\n\n2. I think that the proposed mod framework in this manuscript is kind of like the soft-attention mechanism with some parameters having shared weights. What are the in-depth or essential differences between the proposed MoE framework and the soft-attention mechanisms?\n\n3. Lacks further analysis in ablation study section 4.3. The authors just repeat the results that have shown in figures. For example, 4-experts can achieve the optimal performance and 8-experts reduce it. Why? With fewer training epochs, 8-experts outperforms, but it cannot keep it while training epochs increase. Is it because of the hidden dimension r of LoRA that limits the performance of more experts? \n\n4. The writing language can be further checked, since there are minor language mistakes such as nvidia-smi a100.", "limitations": "The author explains the limitations of the model constructed in the article, such as the recommendation strategy and universality that still need to be explored.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors proposed iLoRA for sequential recommendation task, which address previous works failing to capture individual variability. It creates a diverse array of experts and captures specific aspects of user preferences, guiding the gating network to output customized expert participation weights.\n2. Extensive experiments on two benchmark datasets demonstrate the effectiveness of iLoRA. The ablation studies clearly show that every part of the proposed model makes sense.\n3. The paper is well-written. Figures and tables are very clear and easy to read.", "weaknesses": "See Questions to authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We gratefully thank you for your valuable comments! Here we meticulously give point-by-point responses to your comments, and further revise our manuscript following your suggestions. Hope that our responses can address all your concerns. \n\n---\n\n> **C1: Questions about experimental details and motivation**\n\nThank you for your feedback. Existing research assumes that gradient conflicts, defined as negative cosine similarity between gradients, lead to negative transfer [1], where different gradients interfere with each other, affecting model performance [2]. To support our research, we aim to investigate whether training a single model to handle all recommendation sequences could result in negative transfer. Previous research [2] suggests that close tasks enjoy similar loss geometries and vice versa. Therefore, we need to partition the original dataset based on user similarity.\n\nIn recommendation scenarios, user preferences are implicitly reflected in their historical interaction sequences. Therefore, we partition the user population using representations derived from their historical interaction records. **SASRec** stands as a cornerstone in sequential recommendation tasks. As you mentioned, we utilize the model to generate rich collaborative information representations for each user. Based on these representations, we apply the **K-means** clustering algorithm to divide the dataset into clusters, with the number of clusters set as a hyperparameter (in this case, 8). Clustering based on **Euclidean distance** and cosine similarity yields similar results, with comparable sequence counts per cluster.\n\n---\n\n> **C2: Lack of clear distinction from soft attention mechanism** \n\nThank you for your feedback! As you mentioned, our proposed iLoRA framework draws inspiration from the soft-attention concept, utilizing multiple experts weighted and summed to obtain activation weights. However, the innovation lies in the differences between iLoRA and traditional soft attention methods. In the context of recommendation systems, user preferences are reflected in their historical interaction sequences. Soft attention routing methods typically employ token-level routing or use the entire input *x* as the routing input. A single token lacks specific meanings in the recommendation context, whereas using the entire prompt as routing input introduces considerable redundancy and irrelevant information. Unlike single tokens that lack specific meanings in the recommendation context, user representations encompass rich user preferences and can serve as inputs for gating, outputting instance-wise expert activation weights. \n\nThis innovation inspired the development of iLoRA, which innovatively guides routing using representations of user interaction history—a core and critical feature that distinguishes iLoRA from traditional soft-attention mechanisms. By routing with user representations, iLoRA effectively provides instance-wise unique activation weights for each different sequence, enabling recommendations that better capture user differences and fine-grained preferences, thereby mitigating the negative transfer.\n\nTo demonstrate the effectiveness of iLoRA compared to traditional soft attention mechanisms, we have conducted an additional ablation experiment. We compared the performance of iLoRA (user representation routing) against two soft attention routing methods (token-level routing and full input *x* routing) on the LastFM dataset, as depicted in **Figure 2** in one-page uploaded pdf. This experiment underscores our innovative use of user historical interaction sequences for routing.\n\n---\n\n> **C3: Lacks further analysis in ablation study** \n\nWe appreciate your insightful observation, which highlights an intriguing phenomenon we have uncovered in our research. Thank you for pointing out the shortcomings in our analysis in section 4.3; we will revise our paper accordingly.\n\nYour observation aligns with a common finding when applying MoE+LoRA in other domains [4-5]. Specifically, increasing the number of experts does not necessarily improve model performance.\n\nWe propose the iLoRA framework to enhance model expressiveness while maintaining a constant trainable parameter count, mitigating negative transfer. Therefore, we maintain a fixed total parameter count, i.e., expert number * each expert's rank = 16. Increasing the number of experts benefits by allowing the model to focus more on individual differences between users. Conversely, increasing the hidden dimension *r* of each expert can encourage the model to emphasize universal preferences across the user population, but it may simultaneously limit the expressive power of additional experts. Thus, under the constraint of a fixed total parameter count, this becomes a trade-off scenario. The optimal number of experts likely depends on the data's heterogeneity. In our experimental setup, the model indeed achieved optimal performance with `expert num=4`.\n\nRegarding your second point, we agree with your analysis. A smaller number of experts restricts the iLoRA framework's ability to capture fine-grained user preferences. We appreciate your insightful discussion on this issue.\n\n> **Comment 4: Language mistake** \n\nWe sincerely apologize for the oversight and have promptly addressed these issues in the manuscript. We will carefully review the language issues and make corrections in the revised version.\n\nOnce again, we appreciate your thorough review and conscientious feedback on our work!\n\n[1] Characterizing and avoiding negative transfer. \n\n[2] Investigating and Improving Multi-task Optimization in Massively Multilingual Models. \n\n[3] LLaRA: Large Language-Recommendation Assistant. \n\n[4] Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters. \n\n[5] MOELoRA: An MOE-based Parameter Efficient Fine-Tuning Method for Multi-task Medical Applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G5Iki6dMNp", "reviewer": "Reviewer_wwLq", "summary": "Sequential recommendations is well studied problem with impact across industries which try to personalise a users experience based on their past interactions. Recent popularity of LLMs has led to a lot of research into their use for this task through various generation methodologies. One of those methods is the use of LoRA to adapt the LLM to a specific data distribution (i.e. dataset) leading to better recommendations but it has been shown that due to the high variability in the sequences being input it is often hard to capture all kinds of patterns. This leads to negative transfer of information - to solve which the authors propose a method called iLoRA which is really MoE meets LoRA. Instead of a single LoRA module they propose having multiple LoRA \"experts\" with the motivation that they will capture different kinds of patterns and information in them. To combine the output from these experts they use an \"attention\"-like gating mechanism that looks at the input sequence embeddings (created using existing techniques) and computes a softmax to get the weight of each expert to be considered.\nThe paper contains extensive experimentation to prove out the ideas and contributions and validate the motivation behind the idea. The metrics show that this approach is better than the other SoTA comparable methods in industry.", "questions": "1. Did you do any comparisons with the idea of training multiple full LoRA's and then doing a weighted sum of them OR swapping them based on instance embeddings ?\n2. Other than the overall metrics did you do any analysis on performance when the data is sparse , or situations of cold start recommendations ?", "limitations": "The authors have discussed the major limitations. The impact of invalid outputs due to the generative approach was not discussed in detail - this could have potential risks.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is written clearly, is easy to follow and explains the key ideas that inspired and led to the approach described. The experiments conducted (including ablation studies) provide sufficient evidence to prove the claims and contributions.\nThe paper is able to take key concepts (MoE, Attention, LoRA, item recommendations) in research and combine them in a new way to solve a problem like sequential recommendation which is significant across many industries.\nThe results on the metrics being used by the paper outperform other SoTA techniques which proves out the significance for industry use. The performance gain is also demonstrated across 3 popular datasets which gives it authenticity.\nThe idea is original albeit similar techniques are present in research where people have either tried to a weighted sum of multiple LoRA's or swap them out at runtime based on a gating mechanism. Although I haven't seen an application in recommendation systems yet.", "weaknesses": "The authors should have shown the contribution by expanding to more metrics - at the least HitRate@K, but also metrics like NDCG@K.\nThese are very common metrics in the recommendation space and provide a more holistic view of the output quality.\nThe novelty in the paper is limited as it combines common research ideas in a not so unique way.\nAlso, the impact of training data size on the model performance was not studied - recommendation models can be heavily dependent on this due to the generic sparse nature of data so this analysis would give more weight and significance to the paper.\nI would have liked to understand more on the out of domain abilities of such a model and also specific abilities on cold start recommendations - which is a big challenge in this space. This was not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer wwLq\n\n**Comment:**\n\nWe gratefully thank you for your valuable comments! Here we meticulously give point-by-point responses to your comments, and further revise our manuscript following your suggestions. We sincerely appreciate the depth of your insights, which have undoubtedly enriched our work. Here we meticulously give point-by-point responses to your comments.\n\n> **Comment 1: More evaluation metrics** - \"The authors should have shown the contribution by expanding to more metrics - at the least HitRate@K, but also metrics like NDCG@K. These are very common metrics in the recommendation space and provide a more holistic view of the output quality.\"\n\nWe fully agree that additional evaluation metrics are needed. Inspired by your comment，For the LastFM and MovieLens datasets, we have expanded the evaluation metrics, including HitRatio@3, HitRatio@5, NDCG@3, and NDCG@5, as **Table 1** in one-page uploaded pdf shows.\n\n> **Comment 2: Limited novelty** - \"The novelty in the paper is limited as it combines common research ideas in a not so unique way.\"\n\nThank you very much for your feedback! In the context of recommendation systems, user preferences are reflected in their historical interaction sequences. Unlike single tokens that lack specific meanings in the recommendation context, user representations encompass rich user preferences and can serve as inputs for gating, outputting instance-wise expert activation weights. while iLoRA draws inspiration from the soft-attention concept, it innovatively guides routing using user interaction history representations, which is a core and critical innovation of our entire iLoRA framework. By routing with user representations, iLoRA effectively provides instance-wise unique activation weights for each different sequence, enabling recommendations that better capture user differences and fine-grained preferences, mitigating the negative transfer.\n\n> **Comment 3: Lack of data size impact experiments**\n\nWe appreciate your insightful question regarding the impact of training data size on model performance. Inspired by your comment, we have delved into the influence of the generic sparse nature of data on final recommendation effectiveness. Specifically, on the LastFM dataset, we have conducted experiments at varying degrees of data sparsity, dropping sequences randomly by 25%, 50%, 75%, and 90% at the user level. Subsequently, we have evaluated the performance of both traditional sequence recommendation models (SASRec, Caser, GRU4Rec) and LLM-based recommenders (TALLRec, LLaRA, iLoRA) across these datasets, as illustrated in the **Figure 3** in one-page uploaded pdf.\n\nSeveral key observations emerged from our study:\n\n- Traditional sequence models exhibit a relatively gradual improvement in recommendation performance as training data size increases. In contrast, LLM-based recommenders demonstrate rapid performance gains with smaller training sets. This phenomenon may stem from LLMs quickly capturing foundational patterns of the recommendation scenario from limited samples, thereby bridging the gap between training data and real-world recommendation scenarios and manifesting \"emergent\" behavior on unseen data. However, as training data size grows, diminishing marginal returns on model performance become evident, albeit maintaining a positive correlation with data size within our training set.\n- Our proposed iLoRA framework consistently outperforms baseline methods under equivalent training data sizes, underscoring the superior effectiveness of iLoRA in recommendation tasks.\n- Even with only 10% of user training data, all LLM-based recommenders employed in our study outperform traditional sequence recommendation methods in terms of HitRatio@1, highlighting the substantial potential of LLM-based recommendations in advancing traditional recommendation paradigms.\n\nIn conclusion, these findings underscore the nuanced relationship between training data size and model performance in recommendation systems, particularly highlighting the unique advantages of LLM-based approaches in sparse data environments. \n\n> **Comment 4: Discussion about out-of-domain and cold-start recommendation abilities**\n\nWe value your comments. \n\nTo further investigate the out-of-domain capabilities of LLM-based Recommenders, we have conducted an additional experiment, as depicted in the **Figure 4** in one-page uploaded pdf. Specifically, we fine-tuned iLoRA using training data from different domains: 1) iLoRA (LastFM), 2) iLoRA (MovieLens), and 3) iLoRA (Steam). We then evaluated the models on the test sets of LastFM, MovieLens, and Steam. Our results demonstrate iLoRA's ability to generalize across domains. For instance, after fine-tuning exclusively on MovieLens data, 'iLoRA (MovieLens)' exhibited strong performance on the LastFM dataset, even surpassing some traditional sequence models trained and tested directly on LastFM data. This finding is impressive, indicating that iLoRA possesses cross-domain generalization capabilities beyond single-domain adaptation.\"\n\n> **Comment 5: Comparison with ensemble LoRA models and instance-based swapping** - \"Did you do any comparisons with the idea of training multiple full LoRA's and then doing a weighted sum of them OR swapping them based on instance embeddings?\"\n\nWe appreciate your comments. We considered the approach you suggested, which theoretically could lead to better recommendation performance. However, in practice, it would incur greater training resource expenses. Our iLoRA framework aims to release negative transfer while maintaining a consistent trainable parameter count to mitigate negative transfer. Thus, we have opted to maintain a fixed total parameter count, specifically expert number multiplied by each expert's rank equals 16.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Summary of strengths acknowledged by the reviewers and the responses to address their concerns

**Comment:**

Dear ACs/SACs/PCs,

We would like to summarize the strengths of this work acknowledged by the reviewers, and the responses we have made to address all the reviewers’ concerns.

------

**Strengths** acknowledged by the reviewers:

1. Novelty: timely study (Reviewer **59LV**); address previous works failing to capture individual variability (Reviewer **7niz**); take key concepts in research and combine them in a new way to solve a problem (Reviewer **wwLq**);  mitigating negative transfer and enhancing the model's adaptability(Reviewer **cnYP**)
2. Well-written (Reviewer **59LV**,**7niz**, **wwLq**); Clear and convincing motivation (Reviewer **59LV**);
3. Extensive experiments (Reviewer **59LV**, **7niz**, **wwLq**).

------

There are four main concerns raised by reviewers:

1. More evaluation metrics (Reviewer **cnYP**, **59LV**, **wwLq**).
2. More explanations related to experiments (Reviewer **cnYP**, **7niz**).
3. Inclusion of the impact of training data size (Reviewer **cnYP**, **wwLq**).
4. Inclusion of out of domain abilities and specific abilities on cold start recommendations (Reviewer **wwLq**).

All of these main concerns have been successfully addressed during the rebuttal phase, and we hope that the improvements we made during this stage will be taken into consideration.

------

We sincerely appreciate your valuable time!

Thanks and regards,

Authors

Meta-review: All reviews somewhat borderline, though with 3/4 leaning somewhat positive, mostly willing to recommend acceptance. Reviewers praise the motivation and framework, writing, "timeliness" of the work, and the experiments. More critical comments center around the overall novelty of the work (raised by several reviewers), and the "fragility" of the experiments. These are arguably fairly serious considerations, which make the paper *somewhat* more borderline, but still leaning positive on the basis of the scores, and fairly detailed rebuttal.
Decision: Accept (poster)