LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Authors: Yicheng Xiao
Lin Song
Rui Yang
Cheng Cheng
Yixiao Ge
Xiu Li
Ying Shan
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
LoRA
Specializing Model
Primary area: deep_learning->large_language_models

Abstract: Recent advances have highlighted the benefits of scaling language models to enhance performance across a wide range of NLP tasks. However, these approaches still face limitations in effectiveness and efficiency when applied to domain-specific tasks, particularly for small edge-side models. We propose the LoRA-Gen framework, which utilizes a large cloud-side model to generate LoRA parameters for edge-side models based on task descriptions. By employing the reparameterization technique, we merge the LoRA parameters into the edge-side model to achieve flexible specialization. Our method facilitates knowledge transfer between models while significantly improving the inference efficiency of the specialized model by reducing the input context length. Without specialized training, LoRA-Gen outperforms conventional LoRA fine-tuning, which achieves competitive accuracy and a 2.1x speedup with TinyLLaMA-1.1B in reasoning tasks.
Besides, our method delivers a compress ratio of 10.1x with Gemma-2B on intelligent agent tasks.

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Yicheng Xiao Lin Song
Rui Yang?
Cheng Cheng
Yixiao Ge
Xiu Li!t
Ying Shan
Abstract
LoR A-Gen
41
Recent advances have highlighted the benefits of scaling language models to enhance performance across a wide range of NLP tasks. However, these approaches still face limitations effec- tiveness and efficiency when applied to domain- specific tasks, particularly for small edge-side models. We propose the LoRA-Gen frame- work, which utilizes a large cloud-side model to generate LoRA parameters for edge-side mod- els based on task descriptions. By employing the reparameterization technique, we merge the LoRA parameters into the edge-side model to achieve flexible specialization. Our method facil- itates knowledge transfer between models while significantly improving the inference efficiency of the specialized model by reducing the in- put context length. Without specialized train- ing, LoRA-Gen outperforms conventional LoRA fine-tuning, which achieves competitive accuracy and 2.1x speedup with TinyLLaMA-I.I in reasoning tasks. Besides, method delivers compression ratio of 10.1x with Gemma-2B on intelligent agent tasks.
43
Accuracy (%)
42
Qwen-1.5B
41
LoRA
1-shot
MixLoRA
A
3-shot
LoRAMoE
5-shot
LoRA-Gen
80
100
120
140
Latency (ms)
Figure 1. Accuracy-latency curves comparison with various few- shot numbers ARC-c task. Best view in color. Base model Qwen-1.5B.
2023 Grangier et al. 2024 Shen et al. 2024). This emerg- ing approach addresses the critical need for more adaptable and resource-efficiet Al solutions across academic and industrial domains. Many approaches utilize parameter- efficient fine-tning techniques (Houlsby al. 2019 Li
& Liangg 2021 Lester al., 2021; Hu al. 2021] partic- ularly LoRA Hu et al. 2021). train specific datasets for specialization. However, this method may encounter the issue of catastrophic forgetting, which can result in decrease in performance on other unseen tasks (Feng et al. 2024 Huang et al. 2023a).
1. Introduction
The principle of scaling laws (Kaplan et al.) 2020) demon- strates that increasing the size of Large Language Mod- els (LLMs) can significantly improve cross-task general- ization. However, due to the constraints of their enor- mous size, generic LLMs struggle to achieve a good bal- ance between efficiency and effectiveness when address- ing domain-speciffe tasks or preferences. Consequently, research has been shifted towards developing more special- ized, compact language models optimized for specific tasks and capable of local deployment on edge devices (Fu et al.
To alleviate knowledge forgetting in specialized training, recent approaches (Dou al., 2024 Gao et al.] 2024a] Yang et al. 2024b Li et al. 2024a). leverage the flexi- bility of the Mixture of Experts (MoE) for LoRA train- ing. Specifically, as shown in Figure _bb), they integrate group of multiple LoRA components as experts within the language model, allowing the language model con- trol the selection of LoRA components during token gen- eration. However, these methods introduce additional in- ference costs due to the extra experts and control units. LoR (Huang et al. 2023b), the other hand, pre- trains set of task-specific LoRA components and employs a manually designed parameter- free optimization method for selection. Nevertheless, the effectiveness of above men- tioned approaches is limited by their model scale, resulting
'Equal contribution 'Tsinghua University ARCC Lab, Tencent PCG 'The University Hong Kong "Xi'a Xn Tono University. Correspondence to: Xiu Li <li.xiu@ si.iiu@sz.tinnhuuue tsu.suu..au...u.nnn
Proceedings of the 1and International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
System prompt
User
Input
System prompt
User Input
System prompt
LoRA Generator
Task LoRA
General LM
General LM
General LM
Reparamelerize
LoRA-MoE
LoRA
User Input
Specialized LM
Multiple Invocation
Assistant Output
Assistant Output
Assistant Output
(a) Vanilla LoRA Paradigm
(b) LoRA-MoE Paradigm
(c) LoRA- for Specialized LM
Figure 2. Comparison of different LoRA-based tine-tuning strategies.(a) Vanilla LoRA is fine-tuned the target task and then merged into the source model. (b) LoRA-MoE introduces additional LoRA experts to improve the generalization performance. (c) Our LoRA- Gen presents task-specific LoRA generator that customizes specialized LM for edge-side users.
Context Compression for Unseen Tasks (Fast Inference)
Reparameterized Model
Training Free for Unseen Tasks (High Flexibility)
Knowledge Transfer
Method
(wlo Additional Params.)
(High Accuracy)
ICL [Dong et
2022
LoRA (Hu ct al.
1
LoRA-MoE (Dou ct al
2024)
X
X
LoraHub (Huang al.
2023b)
X
AutoCompressors Chevalier et al.
2023
X
LoRA-Gen
Table 1. Characteristics comparison with other counterparts. ICL indicates the in-context learning.
in constrained performance and generalization capabilities on unseen tasks. Therefore, this paper explores a new per- spective: utilizing a large cloud-side model to generate pa- rameters for a smaller edge-side model to achieve better specialization.
task-specific system prompt (e.g., task descriptions, few- shot samples, and chat templates) into the LoRA weights, which significantly reduces the context length for the specialized models. ii) Reparameterized model: Unlike LoRA-MoE (Dou et 2004), our approach employs repa- rameterization techniques
merge the generated LoRA weights into the original parameters, thereby avoiding ad- ditional inference costs. iii) Training free for unseen tasks: Our method does not require any additional training, in- cluding few-shot tuning, when specializing the model for unseen tasks. It only necessitates a single-turn inference on the system prompt to obtain the specialized model pa- rameters, which simplifies model deployment. iv) Knowl- edge Transfer: LoRA-Gen allows the cloud side and edge side utilize different models, enabling the injection of knowledge from the large cloud model into the edge model through reparameterivation, which enhances performance effectively as shown in Figure I
To achieve it, we propose new LoRA generation frame- work, termed LoRA-Gen. As shown in Figure [c.), our method can be divided into two parts: Online LoRA gen- eration and Specialized LM. The former is used to gen- erate LoRA parameters based on the task-defined system prompt, while the latter facilitates efficient batch inference for user input. Specifically, fine-tuned large language model and mixture of LoRA experts are deployed in the cloud. The cloud-side language model generates set of meta tokens based the given system prompt. Each meta token corresponds to transformer layer in the edge-side language model, utilizing these tokens to control the com- position of parameters from the LoRA experts. Similarly to vanilla LoRA, the combined parameters are further merged into the edge-side LM through reparameterization, result- ing in an efficient specialized model.
We conduct extensive experiments validate the effec- tiveness of LoRA-Gen on various commonsense reasoning tasks well as agent benchmark. The results demon- strate that our method balances both performance and ef- ficiency, showing significant advantages across eight lan- guage datasets. For the edge-side model of TinyLLaMA- I.IB, LoRA-Gen outperforms vanilla LoRA fine-tuning
As shown in Table [ our LoRA-Gen offers four ad- vantages over previous methods: i) Context compression for unseen tasks: LoRA-Gen dynamically compresses the
2

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
by remarkable margin with only 16% sequence length, + 1.3% on harmonic-mean of accuracy, and 2x speedup. Moreover, for the Gemma-2B model, LoRA-Gen demon- strates competitive performance on unseen agent tasks. Ad- ditionally, since it does not require the input of agent def- initions during inference, it achieves a remarkable 10. Ix compression ratio.
2.3. Context Compression
With the rise of in-context learning (Wei et al. agentic pipelines (Yang al., 2024a). LLMs often need to process thousands of tokens, potentially exceeding their maximum context length. Unlike methods that extend the context window of LLMs, context compression offers an efficient way to reduce the input prompt length. There are two primary methods of context compression: hard prompt and soft prompt. Selective- Conteet (Li) 2023) and Tiang et al. (0223) exemplify hard prompt methods by removing low-information content at the lexical level (c.g., sentences, words, or tokens) to shorten the prompt. On the other hand, gisting (Mu et al.
2022) and
2. Related Work
2.1. Parameter-Effiiient Fine- Fine-Tunnin
Given the billions of parameters in LLMs and the limita- tions of current hardware, fully fine-tuning LLMs in the tra- ditional manner is often impractical. To address this, sev- eral parameter-efficient fine-tuning (PEFT) methods have been developed. Adapter-based approaches (Mahabadi ct al. 2021 Zhou et al.] 2024b; Zhang et al., 2024) in- volve inserting trainable adapter layers into various blocks of pre-trained models. Soft prompt methods (Li & Liang] 2021 Liu et al., 2022) adjust small trainable prefix vec- tor to adapt LLMs to new tasks. Unlike these methods, LoRA (Hu et al. 202T) minimizes the number of train- able parameters for downstream tasks by freezing the pre- trained models and tuning only additional rank decomposi- tion layers. This method approximates weight adjustments during fine-tuming without incurring extra costs during in- ference. Building on this, AdaLoRA (Zhang et al. 2023 dynamically adjusts the parameter budget among weight matrices, while DoRA (Liu et al.] (024c) fine-tunes both the magnitude and directional components decomposed from pre-trained weights. VeRA (Kopiezko et al., (024) further reduces the number of trainable parameters by uti- lizing shared low-rank layers and learnable scaling vectors.
(023). AutoCompressors (Cheva- lier et al., 2023). ICAE (Ge et al.) 2024), and 500xCom- pressor (Li et al. 2024b) represent soft prompt methods that compress input prompts into small number of special tokens. In contrast to these approaches, we propose com- pressing the context into rank-decomposition layers using LoRA methods.
3. Methodology
In this section, we first review LoRA-based Mixture of Experts fine-tuning paradigm and then elaborate on our LoRA-Gen, which generates task-specific LoRA weights according to the system prompt for edge-side language models.
3.1. Revisiting Mixture of LoRA Experts
LoRA (Hu et al.) 2021) improves the efficiency of fine- tuning by significantly reducing the number of trainable parameters. Formally, it updates the weight matrix W E Rd''d" xd" by using a low-rank approximation via two decom- position matrices E Rdd xr and B E Rexd" with a low rank r (r < min(d' d')) as follow:
2.2. LoRA Meets Mixture of Experts
Leveraging its lightweight nature, LoRA is utilized in Mix- ture of Experts (MoE) architectures to enhance perfor- mance. MoLoRA (Zadouri et al. 2023) incorporates LoRA adapters as experts on top of pre-trained models and uses a router layer to integrate these experts. MOELORA (Liu jet al. 2024b) applics this framework to various medical domain tasks, though requires task type input for the router. LoRAMoE (Dou al.) 2024) introduces multi- ple LoRA experts into the ford-forward block to mitigate knowledge forgetting during the instruction-tuning phase. LoraHub (Huang et al. (023b)) allows a dynamic assem- bling of LoRA modules on various tasks and even unseen tasks by combining adapted LoRA modules. Additionally, MoLA (Gao et al., 2024a) proposes layer-specific experts, allocating a varying number of LoRA experts to different layers to boost performance.
W = W + AB.
(1)
Trainable low-rank decomposition matrices can capture the underlying patterns of downstream tasks under the guid- ance of the task-specific direction (Hu et al. 2021). More- over, another effective approach, the Mixture of Experts (MoF) (Jacobs et al. 1991 Jordan & Jacobs) 1994). treats multiple networks as experts and seeks to take advantage of their strengths in a hybrid framework. This method aims to combine the advantages of different models, result- ing in improved generalization and overall performance. Typically, a MoE layer consists of n experts, denoted as [E.}= with a router R as the gate for expert allocation. Given hidden states {h,} of a sequence with the length of s, the output of the MoE can be formulated as:
h₂ = > Ra(h))
(2)
3

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Considering the efficiency of LoRA and the strong perfor-
Routing Module. To control the composition of experts, we propose a routing module using meta tokens. Unlike the token-wise LoRA-MoE (Dou et al. 2024), our MoE is layer-wise. We apply an individual MoE for each trans- former layer in the edge-side I.M, and all tokens in a se- quence use the same composition. For simplicity, the rout- ing module consists of two linear projections with a Batch Normalization (BN) layer. Incorporating a BN layer can further increase the diversity of router output, promoting the utilization of wider range of experts. In formal, the router R' E R of i-th layer of edge-side I.M can be for- mulated as:
mance of MoE, (Li et al. 2024a Dou al., 2024 Gao et al. 2024a; Yang et al. 2024b) integrate LoRA into the
MoE plugin, boosting the fine-tuning performance by uti- lizing mixture of LoRA experts, effectively blending the strengths of both methods.
3.2. Online LoRA Generation
The mixture of LoRA experts has showcased reasonable performance in fine-tumi fine-tuning for specific tasks. However, there remains a gap in its effectiveness for multi- task learning and the generalization to unseen tasks. Ad- ditionally, most LoRA-MoF (Li et al. 2024a Dou et al.] 2024) methods require calculating the expert routing for each token individually, which significantly increases the computational complexity. To address these challenges, we propose a new framework, termed LoRA-Gen that gener- ates task-aware LoRA via an online large language model with system prompts (including few-shot samples, task de- scription, role specification and the conversation format) as presented in Figure In the following, we elaborate our LoRA generation method and the reparameterization of the edge-side language model.
Overview.
R' = =N=2N f (TT
(3)
where fI, f2 are the linear transform and ş denotes the SiLU (Elfwing et al. (0188) activation function. We attempt to increase selection randomness and balance expert loads, by using Gumbel-Softmn (Jang et al. 2016), which can be formulated as:
Rti+g
Gumbel-Softmax (R)) =
(4)
Σ)=1 e R3 +
where g Gumbel 1). (5)
Cloud-side LM& & Meta Token. In adherence to meta- learning (Hospedales et al. 2021 Finn et al., 2017). we construct a unified representation of the task-related in- formation to achieve generalization capabilities for vari- ous tasks. relying on cloud-side LM facilitate this pro- cess. Specifically, given a series of few-shot samples task-specific system prompts, the cloud-side LM appends L special tokens (melu) behind them and transfers the in- herent knowledge into these tokens with causal masks in a single forward pass. We define these tokens as meta tokens (TTeeta) ) where represents the number of layers of the edge-side language model. Each meta token is associ- ated with a transformer layer in the edge-side LM.
Nevertheless, the Gumbel-softmax strategy shows sig- nificant reduction in generalization performance, which is
reported in experiments of Section 4.4 To this end, fol- lowing (Li et al.) 2024a] Dou et al.) 2024). we adopt a
KeepTOP-K strategy to select experts in deterministic manner:
R E
TOP-K
G
j=1 R
(6)
0
else
e
where TOP-K(R))
K
R
Σド=1 Rj
where G represents the the gate score of 1-th experts for i-th decoder layer of the edge-side language model. Con- sequently, we generate task-specific LoRA weights as:
LoRA Expert Pool. Our initial attempt is to generate LoRA parameters directly through continuous projection on the meta token. However, the expansive parameter space poses optimization challenges, making the model suscepti- ble to overfitting and hindering generalization, whose anal- ysis refers to Table [9] Therefore, similar to the previous works (Dou et al., 2024), we adopt an alternative solution by introducing the discrete MoE mechanism. Specifically, as shown in Figure 3 we construct a LoRA expert pool of n experts, whose weights are defined as {E Each LoRA expert contains three LoRA blocks, corresponding to the gate linear layer, up linear layer and down linear layer in FFN of the edge-side model, respectively. Differ- ent from the LoRAHub (Huang et al. 2023b), these experts are trained in an end-to-end manner.
0¹ Σ G
(8)
where the 0i indicates the generated LoRA weights for i-th decoder layer.
Reparametrization. As the same as LoRA, we use the reparameterization strategy to merge the generated LoRA parameters into the FFN layers of the edge-side model. In contrast to the LoRA-MoE, our method is cost-free dur- ing inference, which needs no additional components in the specialized edge-side LMM
4

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
ssistant Outputs
System Prompt:
LORA Expert Pool x L
X
Few-shot samples
Task Description
Tools Definition
Linear down
In/Out-put Format
Expert
Expert
Expert
Expert
SILU
Task-specific LoRA x
Lineargan gaza
Linea
Cloud-side side LM
A ssemble
Reparameterize
down
gate
up
Norm Layer
Router x
Meta Token L
0.27
0.31
Routing Module
0.00
Multi- head A Atention Layer
Expert 1
Expert
Expert
Expert
Norm Layer
A uxiliary Loss
In Training Stage
*** User Inputs
(a) LoRA Generator
(b) Edge-side Specialized LM
Figure 3. Overview of our proposed LoRA-Gee Given the system prompts by users, a large language model first generates meta tokens autoregressively. With a routing module, we obtain the gates of experts in the online LoRA pool. After assembling, we produce the specialized LoRA in the cloud side and deploy it to the edge-side language model by merging the LoRA weights.
3.3. Training Target
4. Experiments
Auxiliary Loss. Balanced load of MoE structure is es- sential for capability of generalization and stability (Jacobs al. 1991). Without constraints, the routing module tends to select a fixed small set of experts, leaving other experts unused and causing load imbalance. To mitigate this issue, we introduce a soft constraint with the coefficient of vari- ation as the auxiliary loss, encouraging a more balanced usage of the available experts. Formally, the constraint can be formulated as:
We conduct extensive experiments evaluate the effec- tiveness of our LoRA-Gen and compare to the widely adopted LoRA-based fine-tuning method commonsense reasoning tasks in fair experimental setting. Further- more, we assess the generalization capacity and system prompt compression performance of LoRA-Gen on an agent dataset, GPT4Tools Yang et al. 2024a).
4.1. Datasets and Metrics.
(G) (G)))
Reasoning Tasks. Following (Dou et al. 2024 Li et al. 2024a), we select eight widely-used benchmarks to as- sess the reasoning ability LoRA-Gen across various knowledge domains ranging from natural science daily life. One classification task: BoolQ (Clark al. 2019)
Lep a(
where a and μu represent the standard deviation and mean of the gates assigned each expert within batch, separately. The coefficient is to balance the auxiliary objective and the main objective.
Five question-answering tasks: ARC-c (Clark al. 2018)
ARC-e (Clark et al.) 2018) OpenBookQA (Mihaylov et al.
2018
PIQA (Bisk et al. 2020) and SocialQA (Sap et al.
Total Loss.
The total loss is consist of the language mod-
2019
One science completion task: Hellaswag (Zellers
cling loss and auxiliary loss follows:
et al.
2019) and a fill-n-thee-landd task: Winogrande (Sak
aguchi al.
2020).
Lrotat Lev + LLM:
(10)
where LLM is the Cross Entropy loss of language modeling in causal LMs.
Agent Dataset. We utilize the GPT4Tools Yang et al. 2024a) which provides a benchmark to evaluate the ability
5

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
compression ratio (speedup). One key advantage of LoRA- of LLM to use tools, to assess the effectiveness of LoRA- Gen in the deployment of intelligent agents. GPT4Tools constructs a tool-related instructional dataset, including positive samples, negative samples, and context samples. It consists of 71k instruction-response pairs with 21 tools in the training set and 652 items in the test set with novel tools absent from the training set.
Gen is to compress the tools definition within the system prompt into the generated LoRA parameters via a single- turn inference. It significantly reduces the context length with a compression ratio of 10.1x, which maintains com- parable performance of 91.5% average score, On the other hand, our method without training on GPT4Tools boosts original Gemma-2B by 4.9% in average score, which shows the effective generalization of our method. In con- trast, removing the tool definitions in the vanilla LoRA setting leads to a marked reduction in performance (SR: -26.1 36.1%, IoU: -9.7%). Furthermore, benefiting from knowl- edge injection from the cloud-side language model, it sur- passes the baseline by 3.1 points while maintaining a 10. Ix compression ratio. The results highlight the strengths of LoRA-Gen in effectiveness and efficiency, attributed to its inference-ting specialization and generalization ability to unseen tools, making it well-suited for tasks with extensive prefix descriptions.
Metrics. The performance of all commonsense reason- ing benchmarks is measured with the accuracy metric in all datasets. To further evaluate the performance in multi- task learning, we utilize two metrics: the average accu- racy (AVE.) and the harmonic mean (HAR.) of all results. For GPT4Tools, we measure the performance of method from five aspect: successful rate of thought (SR,), suc- cessful rate of action (SRact), successful rate of arguments (SRargs), ) successful Rate (SR) and IoU according (Yang ct al. 2024a).
4.2. Implementation Details
4.4. Ablation study
We deploy LLaMA3-8B (Grattafiori et al., 2024 the cloud-side LM during online task-specific LoRA parame- ters generation. We finetune the and projection layers of the LLM with a LoRA adapter. The number of experts is and we set K in the routing function TOP-K to 2 by default. The coefficient a for auxiliary loss L is set 0.01. All the latencies are measured on the same GPU with 40GB of memory. More details can be viewed in the Appendix.
Number of Experts in Online Expert Pool. As shown in Table 5 we present the performance of different num- bers of experts in the cloud-side LoRA pool. Performance generally improves with an increasing number of experts. With 4 experts, the AVE. is 56.4%, and the HAR. is 52.3 %. Increasing the number of experts 12 yields slight im- provements, with the AVE. rising to 57.3% and the HAR. to 53.1%% However, the best performance is achieved with 8 experts, where both AVE. (58.7%) and HAR. (53.6%) reach their peak values. This may indicate that 8 experts strike the best balance between multi-task task learning and un- seen generalization.
4.3. Main results
Reasoning Tasks. We first evaluate the performance of LoRA-Gen in the reasoning scenario as shown in Ta- We divide eight commonly used datasets into two parts, one as the multi-task learning set, including ARC-c, ARC-e, OpenBookQA, BoolQ, SocialQA and the other as an unseen test set, including Hellaswag, Winogrande and PIQA. We randomly sample to construct multi-shot train- ing data. As shown in Table 2 LoRA-Gen consistently achieves comparable performance while exhibiting lower latency compared to other fine-tuning methods across var- ious backbone models. Additionally, As shown in Ta- 4 based on the same LLM, our method achieves abso- lute gains of 1.5% over AutoCompressors (Chevalier et al.) 2023). while maintaining much higher efficiency. The re- sults underscore the advantage of using LoRA- Gen, which balances effectiveness and efficiency across both seen and unseen tasks.
ble 2
Effectiveness of Balanced Load Strategy. Ensuring a balanced load of experts can significantly improve the ro- bustness and stability of the model. We initially conduct an ablation study to assess the impact of the absence of auxiliary losses on model performance. Without the auxil- iary loss, the AVE. decreases by 1.2 points. Subsequently, we summarize the impact of different values of the coeffi- cient for auxiliary loss as shown in Table 6 As the aux- iliary loss coefficient decreases, significant improvement in both performance metrics is observed. Reducing the co- efficient from 0.1 to 0.01 yields further gains, resulting in an average (AVE) of 58.7% and harmonic mean (HAR) of 53.6%, thereby achieving an optimal balance between the auxiliary strategy and the primary objective function. In addition, we investigate the strategy of the router function. As illustrated in Table we compare two routing strategies employed for online experts within the cloud-side LoRA pool. Compared to Gumbel-softmax, KeepTOP-K strat- egy exhibits notable improvement, attaining an AVE of
Intelligent Agent Scenario. We evaluate the perfor- mance of LoRA-Gen with edge-side model Gemma-2B on the GPT4Tools benchmark (Yang et al.) 2024a). The re- sults in Table 3 present a comparison of successful rates, intersection-over-umee (IoU), average performance, and
6

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Seen Tasks OBQA
Unseen Tasks
Latency
Method
AVE. f HAR.
ARC-c
ARC-e
Bool
SIQA
HcllaS
WinoG PIQA
(ms) 4
TimyLlaMA-I.18
34.2
66.9
27.4
58.8
46.0
45.8
60.7
73.9
51.7
46.7
44.5
+LoRA
33.6
67.6
28.6
71.9
51.5
44.5
61.9
75.1
54.3
48.5
44.5
+LRRAMOE
35.2
68.8
28.6
73.2
52.1
45.4
62.0
74.1
54.9
49.3
55.9
+MixLoRA
33.5
67.7
28.4
73.3
51.4
44.9
62.3
74.6
54.5
48.6
100.1
LoRA-Gen
35.8
30.4
73.6
49.6
45.5
62.6
74.1
55.1
49.8
Qwen-I.5B
41.9
73.1
29.0
73.3
50.6
49.0
65.3
76.2
57.3
51.9
56.3
+LoRA
43.3
73.9
31.2
77.6
54.9
66.5
76.9
59.1
53.9
56.3
+LRAMMo
43.9
73.7
29.8
77.3
53.4
48.7
66.3
76.9
58.8
53.2
65.7
+MiXLoRA
43.4
73.8
31.8
78.2
54.6
48.9
66.4
76.5
59.2
54.2
141.9
+eRRA-Gen
44.3
74.3
33.4
79.6
53.6
49.1
67.4
76.9
59.8
55.0
26.7
Gemma-2B
50.3
81.5
33.8
73.4
49.3
55.6
71.5
78.7
61.8
57.0
87.3
+LoRA
49.9
78.2
36.0
80.9
56.8
55.4
71.7
79.2
63.5
59.2
87.3
+LRAMME
50.9
38.8
78.4
72.9
79.3
60.0
101.8
+MixLoRA
52.3
79.4
38.6
75.6
59.1
54.1
72.7
78.2
63.8
60.2
177.7
+oRRAGGen 51.2
81.9
39.0
76.2
55.6
56.0
71.6
79.5
63,9
60.2
36.1
Table Comparison of the performance with 5-shot samples various the datasets are part of the training set, while unseen tasks are not. AVE denotes the average accuracy of 8 tasks while HAR is the harmonic The latency scores of various methods all calculated on ARC-c. Latency is measured Nvidia A100 GPU.
reasoning benchmarks. Seen tasks indicate that
W/ Tools Definiton
Average Score
Compress Ratio
Method
W/ Training
SR,
SRact
SRargs
SR
loU
Gemma-2B
86.3
77.6
77.7
65.0
89.7
79.3
1x
+LoRA
99.4
79.6
93.8
78.2
91.0
88.4
+LoRA
98.0
60.9
83.2
52.1
81.3
75.1
+LoRA-Gen
X
94.1
86.8
79.7
73.3
86.9
84.2
10.1x
+LoRA-Gen
98.6
88.0
93.4
84.0
93.6
91.5
Table Performance different fine-tuning strategies with Gemma-2B (Team et al.
2024) on test set of GPT4Tools YYnn et al.
2024a). W/ Training denotes Gemma-2B is fine-tuning the training set of GPT4Tools with vanilla LoRA or our LoRA-Gen. Gray rows indicate scenarios where the system prompt does not contain tools definitions, typically constituting
the input context.
Latency (ms)
the seen tasks, while the results on the unseen tasks are sig- nificantly lower than those obtained with meta tokens, trail- ing by 11.1%. Generating LoRA parameters directly leads to pronounced overfitting to the training domain, caused by the large parameter space, thereby limiting its ability to generalize to unseen tasks.
Method
HellaS
WinoG
PIQA
AVE.
AutoCompressors
44.7
62.4
73.3
60.1
11.4ms
LoRA-Gen
46.3
63.7
74.9
61.6
7.54ms
Table Comparison with AutoCompressors 2023 in unseon tasks based on OPT-2.7B.
(Chevalier et al.
Effectiveness of Knowledge Transfer. As depicted in Table / we compare the performance of the base- line model and our LoRA-Gen across different few-shot samples. Remarkably, LoRA-Gen with just 1-shot sample surpasses the baseline with 5-shot samples by 3.5% on HAR. We attribute this to the use of LLaMA33 8B (Grattatiori et al. 2024) as the cloud model, which transfers a portion of its knowledge to the edge-side lan- guage model via reparameterization.
58. 7% and HAR of 53. 6%. We consider that an over- abundance of randomness may affect expert ability to learn specific tasks during the optimization process.
Effectiveness of Meta Token. We attempt to utilize the cloud-side large language model to generate LoRA param- eters in a single forward pass directly instead of meta to- kens. Specifically, we directly transform the output to- kens of LLM the LoRA weights space with feedfor- ward neural network and get the i-th layer generated LoRA weights E p3x2xxxr, where d is the hidden dimension and r denotes the low rank of LoRA. As indicated by the exper- imental results in Table 9 this approach exhibits compara- ble performance to that achieved through meta tokens on
4.5. Qualitative Study in Agent Scenario
We deploy LoRA-Gen within Gemma-2B and conduct case studies and visualizations. As illustrated in Figure 4 LoRA-Gen removes the 26 tools description from input
7

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
LoRAA-Gen
Raseline
Baseline
[Tools Definition]
General LM
Ready
Description]
General L.M
Ready
want you to act poet, crafting verses that resonate deeply with the soul...... Whether the lone joyful, melancholic, or contemplative, your poem should paint vivid images and capture the emotions wish convey.
Specialized
is Ready
[Task Description]
want you to poet, crafting verses that resonate deeply with the soul..... Whether the tone joyful, melancholic, or contemplative, your poem should puint vivid images and capture the emotions wish convey.
smage fehmm, pay
imagel looww fech//pphnggg
Receive
Specialized LM
Rendy
Receive
Receive
[Tools Definition] Generate
Poom Subject: love
[Tank Description)
Generate real image of a carton- style image based pose the girl
real image carton-style
Poem Subject: love
image based the pose of the girl
Processing time: 5.48 whisper the wind, so light spark ignittes, bunishing night. inLove's gentle warm embrace in smile that lights up every space, cherish, forpeemmnn: beacon of hope, guiding Lovee's gumtle light, shining near and far.
Processing time: 9.2s
Love, whisper on wind/'nA melady that softly spins, 'nA tapestry of threads fine, in Woven with dreamss and hopes divine: 'n'nA touch that alight, beacon For love's embrace, timeleas art'nAA masterpiece. that sets heart,
Processing time: 3,84s
Processing time: 12.4s
X
Figure 4. Visualization comparison between LoRA-Gen and baseline, Gemma-2B TTam al. 2024 LoRA-Gen compresses the tools definition and task description into the generated LoRA parameters, effectively specializing the language model to reduce processing times while maintaining comparable performance. The detailed LM outputs and system prompt can be accessed in the Appendix.
Number AVE.
HAR.
Coefficient
AVE.
HAR.
Few-shot AVE.
HAR.
Strategy
AVE.
HAR.
56.4
52.3
0.1
57.1
52.0
3-shotf
55.5
49.3
12
57.3
53.1
0.005
56.8
50.5
5-shotf
56.0
49.9
GumbleTOP-K KeepTOP-K
56.4
52.3
8
58.7
53.6
0.01
58.7
53.6
1-shotž
58.7
53.6
58.7
53.6
Table 5. Number of Experts in
Table 6. Coefficient of auxil- iary loss.
Table 7. baseline and in- dicates LoRA-Gen.
Table 8. Routing strategy for ouline experts.
LoRA pool.
LoRA Generation
Seen AVE.
Unseen AVE.
language model to generate task-specific LoRA parameters for edge-side models. Our strategy offers four advantages over previous methods: context compression for unseen tasks, reparameterized language model, inference-time specialization, and knowledge transfer. Extensive exper- iments show that LoRA-Gen achieves competitive results and an impressive speedup on common-sens reasoning tasks. Additionally, our method achieves compression ra- 10. Ix on zero-shot agent tasks, indicating its potential applicability more scenarios. We believe our method- ological approach can inspire future LLM-based research.
Direct
52.4
61.0
Meta-Token 52.0
72.1
Table 9. Different LoRA generation manner.
of the model, significantly reducing inference time and achieving a 3.2x speedup compared to the baseline. The limited generalization of the baseline model results in in- correct tool selection, thereby highlighting the effective- ness of our method. Additionally, in the open text gen- eration scenario, LoRA-Gen accelerates reasoning time by compressing the task definition while achieving compara- ble results. The corresponding generation results are de- tailed in the appendix.
A Acknowledgement
This work was partly supported by Shenzhen Key Lab- oratory of next generation interactive media innova- tive technology (No:ZDS (N072DSSSS021106222000 and Na- tional Natural Science Foundation of China N4.65423544, 62425117).
5. Conclusion
In this paper, we propose an online LoRA generation framework, called Gen, which utilizes a cloud-side
8

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Impact Statement
language models. arXiv preprint arXiv:2403.03432 2024.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Finn, C.,, Abbeel, and Levine, S. Model-agnostie meta- learning for fast adaptation of deep networks. In Interna- tional conference on machine learning, pp. 1126-1133. PMLR, 2017.
References
Fu, Y., Peng, H., Ou, L., Sabharwal, A., and Khot, T. Specializing smaller language models towards multi- step reasoning. In International Conference on Machine Learning, pp. 10421-10430. PMLR, 2023.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Gao, C.,, Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y., Guo, X., Yang. J., and Subrahmanian, V. Higher layers need more lora experts. arXiv preprint a4Xiv:2402.08562, 2024a.
Bisk, Y., Zellers, R., Le bras, R., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural lan- guage. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1432-7439, Jun 2020.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L.,, Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, I.., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 2024b. URL records/126082
Brooks, T,, Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In Pro- ceedings of the IEEE/CVE conference on computer vi- sion and pattern recognition, pp. 18392-18402, 2023.
Chevalier, A.. Wettig, A., Ajith, A., and Chen, D. Adapting language models to compress contexts. In EMNLP, pp. 3829-3846. Association for Computational Linguistics, 2023.
org/
Ge, T., Hu, J., Wang, L., Wang, X., Chen, S., and Wei, F. In-context autoencoder for context compression in large language model. In ICLR. OpenReview.net, 2024.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M.. and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Pro- ceedings of the 2019 Conference of the North, Jan 2019.
Grangier, D., Katharopoulos, A., Ablin, P., and Hannun, A. Specialized language models with cheap inference from limited domain data. arXiv preprint arXiv:2402.01092. 2024.
Clark, P. Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0000 080.0050. 2018.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Dong, Q. Li, I.., Dai, D., Zheng, C., Wu, 7., Chang, B., Sun, X., J., J., and Sui, Z. A survey on in-context learn- ing. arXiv preprint arXiv:2301.00234 2022.
Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: survey. IEEE trans- actions on pattern analysis and machine intelligence, 44 (9):5149-510) 2021.
Dou, S., Zhou, E., Liu, Y., Gao. S., Shen, W., Xiong, L., Zhou, Y., Wang, X., Xi, Z., Fan, X., et al. Loramoe: Al- leviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 1932-1945, 2024.
Houlsby. N., Giurgiu, A., Jastrzebski. S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighter linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106. 09685, 2021.
Feng, W. Hao, C., Zhang, Y., Han, Y., and Wang. Mixture-of-loras: An efficient multitask tuning for large

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Huang, C., Liu, Lin, B. Y., Pang, T., Du, C., and Lin, M. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXtv:2307.13269, 2023a.
Li, Z.,, Su, Y., and Collier, N. 500xcompressor: Generalized prompt compression for large language models. arXiv preprint arXiv:2408.0 03094, 2024b.
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, Few-shot parameter-efficiet fine-tuning is better and cheaper than in-context learn- ing. In NeurIPS, 2022.
Huang, C.,, Liu, Lin, B. Y., Pang, T., Du, C.,, and Lin, M. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.1324 13269, 2023b.
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q. Tian, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. Au- dioldm Learning holistic audio generation with self- supervised pretraining. IEEE/ACM Transactions Au- dio, Speech, and Language Processing, 2024a.
Jacobs, R. A., Jordan, M. I,, Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural com- putation, 3(1):79-87, 1991.
Jang. E., Gu, S., and Poole, B. Categorical repa- rameterization with gumbel-softmax. arXiv preprint MrXiv:1611. 144, 2016.
Liu, Q. Wu, X., Zhao, X., Zhu, Y., Xu, D., Tian, F., and Zheng, When MOE meets Ilms: Parameter efficient fine-tuning for multi-task medical applications. In SI- GIR, pp. 1114.114. ACM, 2024b.
Jiang. H., Wu, Q. Lin, C.-Y,, Yang, Y., and Qiu, L. Limlingua: Compressing prompts for accelerated in- ference of large language models. arXiv preprint arXiv:2310.05736. 2023.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y-C. F. Cheng, K.-T,, and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint MwXiv:2402.0933. 2024c
Jordan, M. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2): 181-214, 1994.
Mahabadi, R. K., Henderson, J., and Ruder, Com- pacter: Efficient low-rank hypercomplex adapter layers. In NeurlPS, pp. 1022- 1035, 2021.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint ar arXiv:2001.0836/ 2020.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? new dataset for open book question answering. Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Jan 2018.
Kopiezko, D. J., Blankevoort, T., and Asano, M. Vera: Vector-based random matrix adaptation. In ICLR. Open- Review.net, 2024.
Mu, J., Li, X., and Goodman, D. Learning to compress prompts with gist tokens. In NeurIPS, 2023.
Labs, B. F. Flux. https://github.. com/
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 0rX1v.2307.01952, 2023.
black forest- labs / flux 2023.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint
2021.
Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge scale. Proceedings of the AAAI Conference on Artifi- cial Intelligence, pp. 8732 8740, Jun 2020.
Li, D., Ma, Y., Wang. N., Cheng, Z., Duan, L., Zuo, J., Yang. C., and Tang, M. Mixlora: Enhancing large lan- guage models fine-tuning with lora based mixture of ex- perts. arXiv preprint arXiv:2404.15 15159, 2024a.
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social inter- actions. Cornell University arXiv, arXiv,Cornell Cornell University arXiv, Apr 2019.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
Li, Y. Unlocking context constraints of llms: Enhancing context efficiency of lims with self-information- based content filtering. arXiv preprint arXiv:2304.12102. 2023.
Shen, J., Tenenholtz, N., Hall, J. B., Alvarez-Melis. D., and Fusi, N. Tag-Ilm: Repurposing general- llms for specialized domains. arXiv preprint 20X.52240..05140, 2024.
10

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Team, G., Riviere, M., Pathak, S., Sessa, G., Hardin, C., Bhupatiraju, S., Hussenot, L.., Mesmard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.00/ 18, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your tence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, Jan 2019.
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N.,
Wang, J., Pu, J., Qi, Z., Guo, J., Ma, Y., Huang, N., Chen, Y,, Li, X., and Shan, Y. Taming rectified flow for in- version and editing. arXiv preprint arXiv:2411.04746 2024.
He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-cffcient fine- tuning. arXiv preprint arXiv:2303. 10512, 2023.
Zhang, R., Han, J., Liu, C., Zhou, A., Lu, P., Qiao, Y., Li,
H., and Gao, P. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In ICLR. OpenReview.net, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain- of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
Zhou, C., Yu. L.. Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:24086 11099, 2024a.
Xiao, Y., Luo, Z., Liu, Y., Ma, Y., Bian, H., Ji, Y., Yang, Y., and Li, X. Bridging the gap: unified video comprehen- sion framework for moment retrieval and highlight de- tection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18709- 18719, 2024a.
Zhou, H., Tang, L., Yang, R., Qin, G., Zhang, Hu, R., and Li, X. Uniqa: Unified vision-language pre- training for image quality and aesthetic assessment. arXiv preprint arXiv:2406. 01069, 2024b.
Xiao, Y., Song. L., Wang, J., Song, S., Ge, Y., Li, X., Shan, Y., et al. Mambatree: Tree topology is all you need in state space model. Advances Neural Information Pro- cessing Systems, 777753299-73355 2024b.
Xiao, Y., Song, L., Chen, Y., Luo, Y., Chen, Y., Gan, Y., Huang, W.. Li, X., Qi, X., and Shan, Y. Mindomni: Un- leashing reasoning generation in vision language models with rgpo arXiv preprint arXiv:2505. 2025a.
Xiao, Y., Song, L.,, Yang, R., Cheng, C., Xu, Z., Zhang, Z., Ge, Y., Li, X., and Shan, Y. Haploomni: Unified sin- gle transformer for multimodal video understanding and generation. arXiv preprint arXiv:2506. 02975, 2025b.
Yang, R., Song, L., Li, Y., Zhao, S., Ge. Y., Li, X., and Shan, Y. Gpt4tools: Teaching large language model to use tools via self- self-instruction. Advances Neural Infor- mation Processing Systems, 36, 2024a.
Yang, R., Song, L., Xiao, Y., Huang, R., Gc, Y., Shan, Y., and Zhao, H. Haplovl: A tranffrransformer base- line for arXiv:2503. 14694, 2025.
understanding. arXiv preprint
Yang, S., Ali, M. A., Wang, C.-L., Hu, L., and Wang, D. Moral: Moe augmented lora for llms' lifelong learning. arXiv preprint arXiv:2402.1 /260 2024b.
Zadouri, T., Üstün, A., Ahmadian, A., Ermiş, B., Locatelli, A., and Hooker, Pushing mixture experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309. 05444444444444 2023.

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
6. Appendix
6.4. Training Data.
6.1. Training details
Table 12 outlines the data scale for each reasoning task.
Moreover, process the Alpaca dataset through GPT-4, resulting in filtered and abstracted set of 37,658 training samples.
The models are trained with eight NPUs (64GB memory per device) by default. We betas and momentum of the AdamW optimizer with (0.9, 0.999) and 0.9, respectively. During training, we utilize Cosine Scheduler with ini- tial learning rate of 2 x 10-5 and weight decay of 0.1. The details are shown in IO
6.5. Efficiency Comparison
Table t presents the efficiency Comparison among dif- ferent approaches. MixLoRAT indicates the method with- out specific optimization. All metrics are measured on an Nvidia GPU. FLOPs are measured using an input of 100 tokens and an instruction of 200 tokens, while memory and latency are evaluated in training mode with batch size of per GPU.
Hyper-parametter
LoRA-Gen
optimizer
AdamW
learning rate
2e-5
warm steps
50
6.6. More Potential Application
weight decay
0.1
Our current validation focuses on LLMs, and future work will explore its applicability multimodal large mod- els (Yang et al., 2025: Bai et al., 2023) and unified sys- tems (Xiao et al. 2025a Zhou et al. 2024a Xiao et al. 2025b) for understanding and generation. In addition, we plan to enrich more models as tools by incorporating ad- vanced image generation and editing models (Labs) [023] Wang al.) (2024), as well as speech recognition and syn- thesis models (Liu al. 2024a).
optimizer momentum
B1, 32=0,9, 0.999
batch size
64
epoch
4
max length
2048
LoRA attention dimension (r)
16
LoRA scaling alpha (a:)
16
LoRA drop out
0.05
Table 10. Fine-tuning configuration.
6.2. Detailed Assistant Output
The definition of tools follows GPT4Tools (Yang al. 2024a), encompassing vision foundation models (Xiao al. 2024b), generative models (Podell al.] 2023). and application-speeific models (Brooks et al. 2023 Xiao ett al. 2024a). Task description for the role play in the qualitative study of the main text can be seen in Table [4] To strengthen LoRA-Gen's ability to compress and process instructions in the system prompt, we modify the Alpaca dataset, using GPT-4 generalize specific problems into instruction sets, which subsequently used training data.
6.3. Statistical Significance
The standard enrors of different tasks shown in Ta- ble [I all statistics are calculated with the open-sourced im-evaluution-harness project (Gao et al. (224b)). Ad- ditionally, we have re-evaluated our method times on GPT4Tools with a variation of about 0.65% average score.
12

---

LoRA-Gen: Specializing Large Language Model via Online LoRA Generation
Method
ARC-c
ARC-e
OBQA
BoolQ
SIQA
HellaS
WinoG
PIQA
LLMY
0.0146
0,0089
0.0219
0,0076
0,0112
0,0050
0.0134
0.0100
Qwen
0.0145
0.0089
0.0229
0.0071
0.0113
0.0050
0.0132
0.0098
Gemma
0.0146
0.0089
0.0218
0.0075
0.0112
0.0050
0.0135
0.0096
Table 11. Standard error on language model benchmarks..
Method
ARC-e
ARC-e
OBQA
BoolQ
SIQA
HellaS
WinoG
PIQA
Train Test
1120
2250
4957
9427
33410 1954
39905 10042
9248 1267
16100
1171
2380
500
3270
1838
Table 12. The data size of tasks used in our experiments.
Training Mode
Inference Mode
Method
FLOPs
Memory
Latency
FLOPS
Memory
Latency
+LoRA
4.736E+11
37096MiB
0.85s
4.708E+11
11208MiB
0.19s
+LoRAMoE
4.742E+11
26326MiB
1.19s
4.742E+11
11286MiB
0.22s
+MixLoRA
5.061E+11
30844MiB
2.17s
5,048E+11
11828MiB
1.08s
LoRA-Gee
1.667E+12
39603MiB
2.84s
1.552E+11
10932MiB
0.11s
Table 13. Efficiency Comparison.
Task description of role play:
I want you act poet, crafting verses that resonate deeply with the soul. Your task is to create poem that not only evokes emotions but also possesses the power move and inspire those who read it. The subject matter will be provided by the user, and your words should be carefully chosen to express the essence of that theme in way that touches hearts and stirs deep feelings. Aim for beauty in your language-let your choice of words carry both meaning and emotion, so that each line speaks to the reader on a profound level. Whether the tone is joyful, melancholic, or contemplative, your poem should paint vivid images and capture the emotions you wish to convey.
Edge-side model with LoRA-Gen answer:
A whisper in the wind, touch so light, \A spark ignites, banishing the night. \ Loveess gentle hand, a warm embrace, A smile that lights up every space.) \ \wwo souls entwined, a tapestry of dreams,) Where laughter dances and love softly gleams. )A bond unbreakable, a trust so deep, In each other's eyes, secrets they keep. Through storms and sunshine, hand in hand they roam, (Building castles of hope, love that calls them home. AA silent language, spoken in the heart,', (A symphony of feelings, work of art.) /For love is journey, path to explore, ) treasure cherish, forevermore. beacon of hope, a guiding star, Love's gentle light, shining near and far.
The baseline answer:
Love, a whisper on the wind, A melody that softly spins, \A tapestry of threads so fine, Woven with dreams, and hopes divine.\\A touch that sets the soul alight, (A beacon the darkest night, )A warmth that melts the coldest fear,), A solace, ever drawing near.) IIt blooms in laughter, shared and bright, In silent moments, bathed in light,', )A gentle hand, knowing glance, (A shared embrace, sacred dance.) Love's strength endures, steadfast guide. Through storms and calm, it cannot hide, A constant echo, soft and deep, A promise whispered, secrets to keep. / (o let us cherish every breath, This precious gift, beyond all death, For love's 's embrace, a timeless art,) A masterpiece, that sets the heart. L.\Lett me know you'd like me to explore a specific aspect of love, or you have a particular style or tone mind.
Table 14. Detailed supplement to the visualization results in the main text.
13


Reviews:
[{"review_id": "CwtXeSqUZK", "reviewer": "Reviewer_HcLz", "summary": "This paper introduces LoRA-Gen, a framework that enhances domain-specific task performance for small edge-side models by leveraging a large cloud-side model to generate LoRA parameters based on task descriptions. Utilizing reparameterization, LoRA-Gen merges these parameters into edge-side models, enabling flexible specialization without specialized training. This approach improves inference efficiency by reducing input context length and facilitates knowledge transfer.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Do not include reasoning-specific datasets such as mathematical reasoning.**\n\n**Ans:** \nThanks for the great advice, we further evaluate our method on the mathematical reasoning dataset GSM8K. As shown in Table 10, LoRA-Gen brings gains. Due to time constraints, we will collect more types of tasks for evaluation in the revised version.\n\n|                       | **GSM8K (mathematical reasoning)** | **OpenBookQA (multi-hop reasoning)** | **Arc challenge (commonsense reasoning)** |\n|-----------------------|:----------------------------------:|:------------------------------------:|:-----------------------------------------:|\n| Baseline              | 62.4                               | 31.2                                 | 43.3                                      |\n| LoRA-Gen              | 64.2                               | 33.4                                 | 44.3                                      |\n\n\n**Table 10: Diverse Tasks Performance.**\n\n**Q2: The paper lacks experiments on training-free unseen tasks.**\n\n**Ans:** \nIn this work, we mainly explored the generalization capabilities of mainstream reasoning scenarios and agent scenarios. We set Hellaswag, Winogrande, and PIQA as generalization test datasets, which do not appear in the training period, to verify the training-free unseen capabilities of our method as shown in Table 2 of manuscript. At the same time, we further verified the generalization performance of our method by testing the unseen tools in the GPT4Tools dataset as shown in Table 3 of manuscript. We will further explore the performance of LoRA-Gen in a wider range of scenarios in our future work.\n\n**Q3: The paper does not provide a detailed discussion on the specific experts included in the LoRA expert pool.**\n\n**Ans:**\nWe analyzed the weight distribution across different experts when processing various tasks, as shown in Table 11. For instance, expert 5 is frequently activated for agent tasks but rarely for SIQA tasks, demonstrating that the online LoRA Pool implicitly assigns knowledge to different experts. \n\n|                    | **Expert 1** | **Expert 2** | **Expert 3** | **Expert 4** | **Expert 5** | **Expert 6** | **Expert 7** | **Expert 8** |\n|--------------------|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|\n| ARC Easy           |    0.0981    |    0.5938    |    0.3027    |    0.0149    |    0.4082    |    0.3223    |    0.2051    |    0.0518    |\n| SIQA               |    0.0107    |    0.3945    |    0.4277    |    0.4297    |    0.0378    |    0.2715    |    0.3906    |    0.0361    |\n| GPT4Tools          |    0.2041    |    0.6875    |    0.2070    |    0.0026    |    0.5586    |    0.2617    |    0.0415    |    0.0361    |\n\n\n**Table 11: Expert Weight Distribution.**", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The evaluation in this paper is constrained in both scope and settings:  \n1. The study focuses on classification, question answering, sentence completion, and fill-in-the-blank tasks, labeling them as \"Reasoning Tasks.\" However, it does not include reasoning-specific datasets such as mathematical reasoning, multi-hop reasoning, or commonsense reasoning, which are critical for validating such claims.  \n2. The paper lacks experiments on training-free unseen tasks, which are necessary to demonstrate the method's generalizability. Expanding the evaluation with broader experiments or more in-depth analysis would strengthen the validity of the proposed approach.", "theoretical_claims": "I have reviewed the formulas within this paper.", "experimental_designs_or_analyses": "See \"Methods And Evaluation Criteria.\"", "relation_to_broader_scientific_literature": "The paper combines and builds upon concepts from parameter-efficient adaptation, mixture-of-experts, context compression, and knowledge transfer. It introduces an online generation and reparameterization mechanism designed to balance efficiency and effectiveness, offering a solution to the inherent trade-offs between these two aspects.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper does not provide a detailed discussion on the specific experts included in the LoRA expert pool. A more thorough explanation of these experts would help clarify their relevance to the tested datasets and better illustrate the method's generalizability across different tasks.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vRG6JWqdKZ", "reviewer": "Reviewer_TR5Y", "summary": "The authors automate the creation of task specific LoRAs, by leveraging a larger scale LLM finetuned to generate LoRA parameters. The larger-scale llm is prompted with a system prompt specifying the task. The LoRa parameters are applied to a smaller scale edge model. The authors find these generated lora parameters are effective for seen tasks, unseen tasks for common sense resasoing. They also find it to be effective for tool usage relative to LoRA alone.\n\n\n# After the Rebuttal \n\nThank you for providing the detailed answers. I am therefore keeping my score.  I'd strongly recommend incroporating training data details in clear place in the main paper, since this is a key piece of understanding the system.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: It's not clear to me that context is needed for trained (non-base models), especially on seen tasks.**\n\n**Ans:** \nSincerely sorry for the confusion. We aim to utilize the online large language model and the online LoRA Expert Pool to convert user instructions, system prompts, or contextual content into tailored LoRA parameters, improving the performance of the small model. Therefore, during training, we will provide a few-shot instruction to the online LLM to guide the model in acquiring this functionality.\n\n**Q2: It'd be interesting to see how a small amount of training would help.**\n\n**Ans:** \nAs shown in Table 12, a small amount of training can lead to a small increase of some tasks.\n\n|                         | **Hellaswag** | **Winogrande** | **PIQA** | \n|-------------------------|:-------------:|:--------------:|:--------:|\n| Training-free           | 49.1          |67.4            | 76.9     |\n| Training With Few Cases | 48.8          |67.5            | 77.1     |\n\n\n**Table 12: Performance with Few-Shot Training.**\n\n**Q3: Ablate different lora generators.**\n\n**Ans:**\nAs shown in Table 13, we conduct an ablation study on the average accuracy and harmonic mean of the LLM-based LoRA generator and the general LoRA generator across 8 datasets, using TinyLLaMA-1.1B as the small LLM. The results demonstrate that the LLM-based LoRA generator outperforms the general LoRA generator. Additionally, as presented in Table 2 of the manuscript, the performance of LoRA-Gen surpasses that of several other LoRA-based MoE methods.\n\n|                          | **Average Score** | **Harmonic Mean** |\n|--------------------------|:-----------------:|:-----------------:|\n| General LoRA Generator   | 54.0              |48.2               |\n| LLM-based LoRA Generator | 55.1              |49.8               |\n\n**Table 13. Ablation of Different LoRA Generators.**\n\n**Q4: How is training-free setting in GPT4Tools trained?**\n\n**Ans:**\nAs mentioned in Section 1.4 of the supplementary material, we process the Alpaca dataset through GPT-4, resulting in a filtered and abstracted set of 37,658 training samples. Then we use this part of the data for pre-training to obtain the instruction-aware LoRA generator and LoRA expert pool. We directly utilize the checkpoint to perform generalization tests on the GPT4Tools test set, corresponding to the 4th row \"without training setting\".\n\n**Q5: How important is the LoRA generator base LLM?**\n\n**Ans:**\nWe conduct an experiment to verify the impact of the LoRA generator. As shown in Table 14, LLaMA3-8B (with the strongest reasoning ability) does bring better results, and increasing the size of the LoRA generator will also improve to a certain extent.\n\n|             | **ARC Challenge** | **OpenbookQA**    |\n|-------------|:-----------------:|:-----------------:|\n| LLaMA2-7B   | 41.3              |28.6               |\n| LLaMA2-13B  | 42.6              |29.0               |\n| LLaMA3-8B   | 43.2              |31.0               |\n\n**Table 14. Ablation of Different LLM-based LoRA Generators.**\n\n**Q6: Are there scaling properties?**\n\n**Ans:**\nIn this work, we offer an initial verification of the advantages of this paradigm. Moving forward, we will expand the dataset size to explore more effective and resilient training paradigms.", "claims_and_evidence": "* LoRA-Gen is faster than alternatives: \nTiming experiments with shorter context. \nIssue: It's not clear to me that context is needed for trained (non-base models), especially on seen tasks. \n    \n* LoRA-Gen generalizes better to new tasks:\nExperiments it shows some generalization capability to new tasks (possibly due to few-shot examples). \n\n* LoRA-Gen is training free \nIndeed, do not need training for new tasks. However, it'd be intersting to see how a small amount of training would help. \n\n*LoRA-Gen Allows knowledge transfer from large model to small model. \n\nIt is unclear how much of the large llm base model gets transfered into the generate lora params, vs how much is learned during the general lora generator training phase. Maybe ablate different lora generators?", "methods_and_evaluation_criteria": "Proposed methods and evaluation criteria do make sense.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "I checked soundness of desigins. Experiment design is sound.", "relation_to_broader_scientific_literature": "This work builds on the prior ideas of PEFT, Mixture-of-Experts, and context compression. It takes findings that LoRA is a highly effective PEFT methodology and that LLM's can compress prompts, and instead represents prompts as parameters.", "essential_references_not_discussed": "HyperDreambooth[1] is an intersting paper to cite; it generates lora parameters for personilzation of generative models. \n\n[1] Ruiz, Nataniel, et al. \"Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.", "other_strengths_and_weaknesses": "Strenghts: Clarity + writing is high on average (for minor writing weakness see below). \n\nWeakness: The lora generator + lora expert pool needs training, yet testing on GPT4Tools is said to not need training (red x, second column, table 3). Then how is it trained? Does it generalize from commonsense reasoning task training?\n\nHow important is the lora generator base llm? Would a stronger reasoing model be better, since the cost is being amortized anyways?\n\nThis type of meta-training really requires a 'dataset of datasets'. However, the current setup is more of a proof-of-concept becacuse the number of seen task is quite low.  Are there scaling properties?", "other_comments_or_suggestions": "Overall, an interesting direction! I'd like to see this scaled up and an experiment on the importance of the cloud-side LM. It almost feels like an embedding-type model (maybe encoder-decoder?) would make more sense.", "questions_for_authors": "See weakneses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kpcocG5OcF", "reviewer": "Reviewer_7PAF", "summary": "The paper proposes LoRA-Gen, a framework for specializing language models (LMs) on edge devices by generating task-specific LoRA parameters using a cloud-side model. The core idea involves leveraging a large cloud-based LM to generate meta tokens from task descriptions, which dynamically assemble LoRA parameters from a pool of experts. These parameters are merged into the edge-side model via reparameterization, reducing input context length and enabling efficient inference. Key claims include:\n● Training-free specialization for unseen tasks via single-turn inference on system prompts.\n● 2.1x speedup with TinyLLaMA-1.1B and 10.1x compression ratio with Gemma-2B on agent tasks.\n● Superior performance over LoRA, LoRA-MoE, and MixLoRA across eight commonsense reasoning benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The training-free generalization claim requires broader validation across diverse tasks.**\n\n**Ans:** \nAs shown in Table 2, Table 3 in the manuscript and Table 10 in the response to reviewer HcLz, we have mainly validated our method in the fields of mathematics, commonsense reasoning, science, daily life and tool usage. In the future, we will further expand to more fields to verify our method.\n\n**Q2: How cloud-to-edge knowledge injection works mechanistically.**\n\n**Ans:** \nAs shown in Table 12, in the response to reviewer TR5Y, when we do not utilize the online LLM to select experts, the overall performance decreases, further highlighting the importance of the online large model in converting instructions into internal knowledge to guide edge-side models.\n\n**Q3: Hyperparameter sensitivity.**\n\n**Ans:**\nOur exploration of different hyperparameter settings, as shown in Table 5 of the manuscript, revealed that adding more experts to the LoRA Expert Pool does not inherently improve results. We speculate that this may be due to the mismatch between the number of experts and the scale of available training data, resulting in poorly tuned expert classifications, and the auxiliary loss coefficient is the result of a hyperparameter search.\n\n\n**Q4: No analysis of how varying cloud-side models (larger/smaller LMs) affect performance?**\n\n**Ans:**\nWe have conducted an experiment to show that the LLM with stronger reasoning ability brings better results, as shown in Table 13 in the response to reviewer TR5Y.  \n\n**Q5: The paper could benefit from a more detailed discussion on how LoRA-Gen compares to other non-LoRA methods, Prompt tuning?**\n\n**Ans:**\nWe conduct an experiment to show performance comparison between LoRA-Gen and Prompt-tuning as indicated in Table 15.\n\n|                 | **Prompt-tuning** | **LoRA-Gen**    |\n|-----------------|:-----------------:|:---------------:|\n| ARC Challenge   | 43.2              |44.3             |\n\n**Table 15. Performance Comparison.**\n\n**Q6: Reliance on cloud infrastructure and the limitations of the method and potential areas for improvement.**\n\n**Ans:**\nOur method supports diverse application scenarios, such as tool calls, personalized virtual assistants, offline intelligent systems, IoT device control, and tasks necessitating long system prompts. However, the current paradigm needs to predefine a pair of cloud and edge-side LM. The model-agnostic framework leaves an open question for future work.\n\n**Q7: Clarify the computational overhead of cloud-side LoRA generation.**\n\n**Ans:**\nAfter specifying the task description or system prompt, the cloud model only needs to perform one inference to complete the generation of LoRA parameters. For specific FLOPs, Memory, and Latency in the training and inference stages, please refer to Table 4 in the supplementary material.\n\n**Q8: Fix typos: Page 6, \"utilyze\" → \"utilize\"; unify \"LoRAMoE\"/\"LoRA-MoE\".**\n\n**Ans:**\nThanks for your valuable advice. We will refine this in our revision paper.\n\n**Q9: How does the computational cost of cloud-side LoRA generation compare to edge-side efficiency gains?**\n\n**Ans:**\nIn fact, in the deployment and reasoning phase, the cloud model only needs the computational overhead of a single inference to complete the specific task. The main computational overhead comes from the training phase. As shown in Table 15 in response to reviewer TR5Y, the 13B model will bring some gains over the 7B model.\n\n**Q10: What task characteristics make LoRA-Gen most effective (e.g., prompt complexity, task diversity)?**\n\n**Ans:**\nOur method is more beneficial in scenarios that typically feature a fixed specialized system prompt and varying user inputs. In such cases, the cloud-side large model performs the generation of customized LoRA weights through a one-time system prompt inference and supplies these weights to the edge-side small model.\n\n**Q10: What are the primary failure modes of LoRA-Gen, and how might they be addressed?**\n\n**Ans:**\nThe major limitation of the current paradigm needs to predefine a pair of cloud and edge-side LM. If we do not use the same model pair as in the training phase, it will lead to a primary failure mode. A possible solution is to train multiple models in cascade, which we will continue to explore in future work.", "claims_and_evidence": "The claims are largely supported by extensive experiments on commonsense reasoning (e.g., ARC-c, OpenBookQA) and agent benchmarks (GPT4Tools). Results demonstrate clear improvements in accuracy, latency, and compression. However:\n● The training-free generalization claim requires broader validation across diverse tasks.\n● Limited analysis on task-specific strengths/weaknesses (e.g., why LoRA-Gen excels in certain tasks but not others).\n● The claim of knowledge transfer needs deeper exploration (e.g., how cloud-to-edge knowledge injection works mechanistically).", "methods_and_evaluation_criteria": "The methodology is well-designed, combining LoRA, MoE, and reparameterization to address edge-side efficiency. Evaluation metrics (accuracy, latency, compression ratio) are appropriate. The use of harmonic mean accuracy for multi-task evaluation and task-specific benchmarks (e.g., GPT4Tools) strengthens validity.", "theoretical_claims": "No formal theoretical proofs are provided. The work is empirically driven, with ablation studies validating design choices (e.g., meta tokens vs. direct LoRA generation, routing strategies).", "experimental_designs_or_analyses": "Experiments are comprehensive, comparing LoRA-Gen against strong baselines (LoRA, LoRA-MoE) across multiple models (TinyLLaMA, Gemma-2B). \nKey gaps:\n● Hyperparameter sensitivity: Limited discussion on the impact of expert count (n=8) or auxiliary loss coefficient (α=0.01).\n● Cloud model dependency: No analysis of how varying cloud-side models (e.g., larger/smaller LMs) affects performance.", "relation_to_broader_scientific_literature": "The paper is well-situated within the broader literature on parameter-efficient fine-tuning of large language models. It builds upon recent advances in LoRA and MoE techniques and addresses the critical need for efficient and effective model specialization. The authors provide a good overview of related work, including various parameter-efficient fine-tuning methods and context compression techniques. However, the paper could benefit from a more detailed discussion on how LoRA-Gen compares to other non-LoRA methods, such as adapter-based approaches or prompt tuning.", "essential_references_not_discussed": "The work builds on LoRA (Hu et al., 2021), MoE (Jacobs et al., 1991), and context compression (e.g., AutoCompressors). However, it omits critical recent works:\n● VeRA (Kopiczko et al., 2024): A parameter-efficient method using shared low-rank layers.\n● DoRA (Liu et al., 2024b): Decomposes weight updates into magnitude/direction components.", "other_strengths_and_weaknesses": "Strengths:\n● The proposed framework is innovative and addresses a significant challenge in deploying specialized language models on edge devices.\n● The results demonstrate clear improvements in both performance and efficiency, making the method highly practical for real-world applications.\n● The idea of generating task-specific LoRA parameters using a cloud model is novel and leverages the strengths of both cloud and edge computing.\nWeaknesses:\n● The dependency on a cloud-side model for generating LoRA parameters may limit the applicability of the method in scenarios with limited network connectivity or strict privacy requirements.\n● The generalization ability of LoRA-Gen to unseen tasks needs further validation on a broader range of tasks and domains.\n● The paper could benefit from a more detailed discussion on the limitations of the method and potential areas for improvement.", "other_comments_or_suggestions": "● Clarify computational overhead of cloud-side LoRA generation.\n● Discuss limitations (e.g., reliance on cloud infrastructure, task scope).\n● Fix typos: Page 6, \"utilyze\" → \"utilize\"; unify \"LoRAMoE\"/\"LoRA-MoE\".", "questions_for_authors": "1. How does the computational cost of cloud-side LoRA generation compare to edge-side efficiency gains?\n2. What task characteristics make LoRA-Gen most effective (e.g., prompt complexity, task diversity)?\n3. What are the primary failure modes of LoRA-Gen, and how might they be addressed?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes LoRA-Gen, an innovative framework that utilizes a large cloud-based model to generate task-specific LoRA parameters for smaller edge models, aiming to enhance performance and efficiency without requiring task-specific training on the edge device. The method merges generated parameters via reparameterization, reducing context length and enabling faster inference. Experimental results demonstrate promising improvements in accuracy, latency, and compression on commonsense reasoning and agent tasks compared to baselines like standard LoRA. The authors effectively addressed most reviewer concerns during the rebuttal phase with additional experiments and clarifications.

**Strengths:**
* **Novelty and Practicality:** Addresses the significant challenge of deploying specialized LMs on resource-constrained edge devices using a novel cloud-edge synergy (All Reviewers).
* **Efficiency Gains:** Demonstrates clear improvements in inference speed and context compression, crucial for edge applications (7PAF, TR5Y).
* **Strong Empirical Results:** Shows superior performance over relevant baselines (LoRA, LoRA-MoE) on several benchmarks (All Reviewers).
* **Clarity:** The paper is generally well-written and the proposed method is clearly explained (TR5Y).

**Concerns and their address:**
* **Generalization Scope:** While authors provided additional data (GSM8K) and pointed to existing unseen task results, the validation of training-free generalization could still be broader across more diverse task types (HcLz, 7PAF). Reviewers HcLz and 7PAF were mostly satisfied by the rebuttal.
* **Training Data Clarity:** Details regarding the training data composition (e.g., Alpaca pre-training) and its specific use across different experimental settings (commonsense vs. tool use) were clarified in the rebuttal but should be more explicitly integrated into the main paper for better reproducibility and understanding (TR5Y). Reviewer TR5Y maintained their score requesting this improvement.
* **Cloud Dependency & Scalability:** The method relies on a cloud model and specific cloud-edge model pairs, limiting offline use and potentially requiring significant meta-training data ('dataset of datasets') for robust scaling, which remains future work (7PAF, TR5Y). Authors acknowledged this limitation; reviewers accepted the clarification.
* **Comparison Breadth:** While compared effectively to LoRA variants and prompt tuning (added in rebuttal), comparison to other recent PEFT methods (e.g., VeRA, DoRA mentioned by 7PAF) is missing. Reviewer 7PAF accepted the rebuttal despite this omission.
* **Hyperparameter Sensitivity:** Limited exploration of hyperparameters like the number of experts and their interaction with training data scale (7PAF). Authors provided justification based on existing experiments (Table 5); Reviewer 7PAF accepted this.
Decision: Accept (poster)