$\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning
Authors: Runqian Wang
Soumya Ghosh
David Daniel Cox
Diego Antognini
Aude Oliva
Rogerio Feris
Leonid Karlinsky
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter Efficient Finetuning
Knowledge Distillation
Large Language Model
Data Synthesis
Primary area: natural_language_processing
TL;DR: $\textit{Trans-LoRA}$ enables nearly data-free transfer of existing LoRA to different base models.

Abstract: Low-rank adapters (LoRA) and their variants are popular parameter-efficient fine-tuning (PEFT) techniques that closely match full model fine-tune performance while requiring only a small number of additional parameters. These additional LoRA parameters are specific to the base model being adapted. When the base model needs to be deprecated and replaced with a new one, all the associated LoRA modules need to be re-trained. Such re-training requires access to the data used to train the LoRA for the original base model. This is especially problematic for commercial cloud applications where the LoRA modules and the base models are hosted by service providers who may not be allowed to host proprietary client task data. To address this challenge, we propose $\textit{Trans-LoRA}$ --- a novel method for lossless, nearly data-free transfer of LoRAs across base models. Our approach relies on synthetic data to transfer LoRA modules. Using large language models, we design a synthetic data generator to approximate the data-generating process of the $\textit{observed}$ task data subset. Training on the resulting synthetic dataset transfers LoRA modules to new models. We show the effectiveness of our approach using both LLama and Gemma model families. Our approach achieves lossless (mostly improved) LoRA transfer between models within and across different base model families, and even between different PEFT methods, on a wide variety of tasks.

Trans-LoRA: towards
data-free Transferable Parameter Efficient Finetuning
Runqian Wang" i
Soumya Ghosh ghoshso@us 1bm. com
David Coxt
raywang4@mit .ecimmittuduu
david. d. cox@ibm. com
Diego Antognini
Aude Oliva"
Rogerio Ferist rsferisuus -imm com
Leonid Karlinsky? leonidkašibm com
diego. antognini@iiii bim.mm..iimiim.m com
oliva@mit. edu
Abstract
Low-rank adapters (LoRA) and their variants are popular parameter-efficient fine-tuning (PEFT) techniques that closely match full model fine-tune performance while requiring only a small number of additional parameters. These additional LoRA parameters are specific the base model being adapted. When the base model needs to be deprecated and replaced with a new one, all the associated LoRA modules need to be re-trained. Such re-training requires access the data used train the LoRA for the original base model. This is especially problematic for commercial cloud applications where the LoRA modules and the base models hosted by service providers who may not be allowed to host proprietary client task data. To address this challenge, we propose Truns-LoRA- novel method for lossless, nearly data-free transfer of LoRAs across base models. Our approach relies on synthetic data to transfer LoRA A modules. Using large language models, we design synthetic data generator approximate the data- generating process of the observed task data subset. Training on the resulting synthetic dataset transfers LoRA modules to new models. We show the effectiveness our approach using both LLmma and Gemma model families. Our approach achieves lossless (mostly improved) LoRA transfer between models within and across different base model families, and even between different PEFT methods, on a wide variety of tasks.
Introduction
The remarkable progress in language modeling has led to the development of Large Language Models (LLMs) 16 13 [2) achieving high performance on general language tasks via scaling model parameters multi-billion sizes. Despite their great progress, even the largest and strongest LLMs [16] still significantly benefit from fine-tuning to downstream tasks for enhanced specialization and consequent performance improvement (50). However, it commonly difficult gain the computational, memory, and disk resources needed for fine-tuning and later hosting fine-tuned large-scale models, especially when serving model customization APIs numerous clients. Thus, common approach to LLM finetuning is to use ceuumeter-efficiee finetuning (PEFT) methods, the most widespread of which are Low-Rank Adapters (LoRA) 3T 4 which only train a small number of additional parameters while freezing the base pre-trained model. Using PEFT can lead to more efficient and compute- compute-friendly training without sacrificing final performance 31. well as allowing efficient serving of large quantities LoRA models "orbiting" common base model 'core' [56]. However, LoRA model fine-tuned for specific task is tied to base model and cannot be used without it, and also cannot be directly transferred to another base model. This is quite problematic in commercial cloud model serving scenarios, where after the base
"MIT
1 MITTIBBM Watson Lab
Work done while at MIT-IBM Watson AT Lab
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Synthete Data Disoiminator Training
Task Finetuning
MLS Source Modell M,
+
Boolean expression tack esamples. time and nee cen nee telie
expression
nee not Prlae frest co
Discrimator LoFAA Ad Jou
Fitared Syrthetic Data
Plre Syrthetic Data
Trensfer
Tesk
Torget Model M,
Booksan oop coppsssion osar plaa: Tree iecine and trre and False ln
Boolcan expression look ass mass aid rable os tra Lt Feise eot froe talie
Ta Tarpet LORA
Figure 1: Trans-LoRA overview. Examples from 'boolean expressions' BBH task illustrate the lower diversity of raw synthetic samples compared to the original task data, which is fixed by our filtering approach. The source model is used to: 1. train the source LoRA; 2. synthesize data for discriminator training; and 3. train the (LoRA) discriminator. Then, the target model is used to synthesize data for transfer (filtered by discriminator) and train target LoRA using the source LoRA teacher.
model needs to be deprecated and replaced by newer LLM, the (potentially thousands of) clients' LoRA models need be switched the new base model. Naively, one would have to re-train all the LoRA models, which, understandably, is a logistic nightmare given that clients' proprietary task data is commonly confidential and not retained on the servers of the cloud service provider. Naturally, asking all of the clients to re-send the data for re-training or retraining on their own is neither scalable nor practical.
In this work, we propose Trans-LoRA an approach for 'universal' LoRA transfer offering an ability to train LoRA models in a way that allows them to be transferred to new base models, and even to other kinds of PEFT (e.g. LoRA [T ++ DoRA
43 or PT(37), an automatic and centralized manner on the model service provider side, while preserving or improving performance, and without the need to access to the clients' data used to train the original LoRAs. Our Trans-LoRA is based on using the source base model LoRA to teach the target base model LoRA, while the main challenge is in obtaining the training curriculum for such a transfer in a manner that is both data-free and sufficiently effective to guarantee the resulting LoRA performance improvement beyond the maximum of the respective target base model and the source LoRA performances. Surprisingly, ToA we demonstrate is possible to obtain an effective transfer curriculum for achieving these feats using synthetic data generated from the target base model. However, this by itself is insufficient to obtain the specified guarantees. We discover in Trans-LoRA that we need to additionally train discriminator model for synthetic data filtering. Our proposed discriminator is trained on a mix of synthetic and real data alongside the source LoRA model and is optimized to ensure the filtered synthetic data most closely resembles the source LoRA training distribution. We provide extensive evidence, insights, and ablations as to why the proposed Trans-LoRe synthetic transfer curriculum works and is superior to the altemnttve cariculuu-buuilding approaches.
We perform numerous experiments confirming that our Trans-LoRA achieves the above guarantees while transferring within and across the popular I Llama2 [15] and Gemma 14 model families, popular I LoRA (3T] DoRA [43. and Prompt Tuning (37) PEFT variants, and using large variety of about 90 (language, code, and math) tasks contained in popular datasets such as BBH (60, MMLU (28). GSM8K [10) MBPP (5), and MBPP+ [41]. Notably, our Trans-LoRA not only achieves overall lossless transfer, it primarily improves performance (by up to 10% in some cases) over the maximum among the fine-tuned source model and the target base model performances, thus consistently achieving positive transfer! We perform an ablation comparing to transfer using unfiltered synthetic data random data from other sources. We explore transferring between different PEFT variants (e.g., LoRABI] ++ DoR ( or PT(37), as well as multi-step transfer through an intermediate model (simulating multiple transfers due to consecutive model deprecations). in all cases supporting the robustness and merits of our LoA approach. We also show that our Trons-LoRA positively benefits from scaling the synthetic data generation. Finally, provide further error analysis of Trans-Loke LoRA and ways to mitigate some edge-case scenarios.
To the best of our knowledge, Trans-LoRA is the first approach to explore the automatic, nearly data-free, and universal transferability of LoRA (or any other PEFT) models between base (LLM) models. The effectiveness of our approach observed in numerous experiments and ablations strongly suggests that Trans-LoRA can be readily used for the said tasks in the challenging and yet very practical masseve-scale custom models serving cloud applications.
2

---

2 Related Work
Parameter Efficient Finetuning (PEFT) has emerged as an important area of research, particularly in the domain of transfer learning where the adaptation of large pre-trained models to specific tasks without extensive retraining is a significant challenge 17 [39 25 [2] [7].. The literature on PEFT spans various approaches, each characterized by its strategy modify a minimal number of parameters while maintaining competitive performance. Many different PEFT methods have been proposed, spanning
Adapter Modules 30 26 59
Prompt Tuning (37 32 including multi-task variants [66], very popular
L Low-Rank Adaptation techniques including LoRA [3I, DoRA 43 69 [4], NOLA [36] and others
33 7TI. A major challenge with PEFT techniques is that they do not transfer across base models
and our proposed approach addresses this challenge for the first time.
Knowledge Distillation (KD) is a technique where knowledge from a larger, typically more complex model (teacher) is transferred to smaller, more efficient model (student) [29 35 [1 53 45 Additional variants proposed include self-Distillation [73 3 74 47 76 with same model as teacher and student, and Weak to Strong Distillation
6 63 34 that can under some circumstance help the stronger model to avoid
overfitting [9]. While these approaches have shown promise in transferring between models, they still rely on training corpus for the distillation making them challenging to apply in a data-frec scenario. We see from experiments that producing a good set of data for distillation that would guarantee lossless transfer of PEFT models between base models and/or PEFT types is challenging and addressed by our proposed approach.
Synthetic Data is increasingly used to train machine learning models 17
49
I 52]. It has been used 55 and more recently 8
in computer vision (20) [1] 24 42 62). language processing [3
27
64
58 While synthetic data has been researched for general model improvement, to the best of our knowledge we are the first to explore its use for PEFT models transfer between base models and PEFT variants. As we show in our experiments and ablations, lossless transfer can only be achieved with careful curation of synthetic data achieved in our approach in an automatic and nearly nource-data-tee frre way. Additionally, we highlight that the synthetic data filtering approach employed Truns-LoRA can be orthogonally applied on top of any of the more
in instruction tuning and LLM alignment (65 61 67 48
46
38
advanced synthetic data generation methods 72
19 44
68
3 Trans-LoRA
Given pre-trained model M, (dubbed the source model going forward) and task-specific dataset, (= 1N of prompt (zx,) and completion (H,,) pairs, we assume that we have tuned M, on D using a PEFT method (c.g.. LoRA (3I)), obtaining a task-adapted set of additional parameters 0g (e.g.. realized as a set of residual adapters in (3I). Next, given distinct model M, (the target model) and access only a small subset of 'seed' examples, D D. our goal is to learn task-adapted parameters 0, for M2 such that Og bestows similar or better capabilities on M, as those bestowed by 0, on M,, In Trans-LoRA we consider D to be very small set of demonstrations ((D) = in all experiments) explaining the intent of the task and its I/O format. Keeping this tiny set of 5 samples DD does not violate the nearly fata-free property of our Trans-LoRA, as D can be cleaned from proprietary information, retaining only the core expected properties of the task.
3.1 Capabilities transfer through knowledge distillation on synthetic data
While D is unavailable when training 0 for Mr. we do have 6., MM., and D available to us. As such, capabilities can be transferred between 9, and 0, via knowledge distillation, i.e., by tuning 0 to match the completions produced by M2 with the task-adapted parameters 0.. Unfortunately, naively distilling on D performs increasingly poorly with shrinking cardinality of D and is often detrimental to point where the un-adapted MM outperforms O tuned on D. This is particularly so for |DD = 5 (Section 4.3 set by us as a requirement for Trany-LoRA to maintain its appealing nearly data-free aspect.
But if D is insufficient, and the original task data cannot be retained, what should then be used as the necessary input samples (outputs are not required) for the knowledge distillation? One attempt could just be using random pieces of text from the web (e.g. from Wikipedia). However, these samples do not follow the input distribution of the task and result in poor transfer (Section 4.3). A key insight behind our approach is that augmenting D with carefully synthesized data, Dayas allows for effective learning of 0, However, interestingly, naive synthesis (e.g. from M,) using D as demonstrations by itself insufficient (Section to produce the set of inputs for lossless transfer, that is for guaranteeing M, + O, outperforms both the
3

---

Task Finetuning
Task Transfer
Seed Task Data
Source
Target Model Mf
Inputa X No Lubula
Tesk Data
Seedd Prompt
Irputs X
Gcnerate
Gonersto
Lubeis y
Synthetic Data for Discriminator Training
Discriminator
LoRA
Source LoRA
Imputs x' No Labels
Traiiin Ioput
Traanig Inpsl
Tranning Irpur.
Irque
Generuto
Source Model
Source Model
Filtered Bynthotic Data
Fittered Synthetic
Inpuss X No Labels
Irpuit x
Pradicted Labels Bired
Fleas PerT
Resultng FEFT
Traning ipput
Discriminator LoRA
Source LoRA 0,
Target Model
Resuling
Target LORA
M
Mr
8,
Figure Detailed breakdown of Trans-LoRA. Task Finetuning is done beforehand and produces the source LoRA for the source model and the discriminator: Task Transfer utilizes the source LoRA and discriminator transfer the LoRA onto the target model and produce the target LoRA.
non-tuned Mr and the MM, + 0, as desired. We find that in addition to following the task distribution (which can be approximated via synthesizing from D as demonstrations), the synthetic data must also adhere to one additional important requirement it must also follow the distribution used to sample the original training set D out of all possible task data. Clearly, this marginal distribution P of just the inputs { of the training samples in would intuitively correspond to the "comfort zone' of the intended teacher model M.
+,+0, 0. (that learned from observing 'D and not the entire task data). Hence making it more likely for M,+ to produce higher quality outputs for the transfer for inputs sampled from P.
Using above intuitions, we build synthetic data simulator that generates data Deyn that is statistically indistinguishable from the observed task data D and is used for the aforementioned knowledge distillation at the time of transfer. Drawing inspiration from GAN 200, our simulator consists of generator and a discriminator, described in greater detail below. While the generator part of the simulator is achieved by an LLM endowed with our designed prompt and using the tiny D in-context demonstrations, the discriminator is a separate PEFT model trained once alongside the training of 0₂ on D and kept for all future transfers. Hence can safely assume access to D for discriminator training. Discriminator training does not require knowledge of the target model Mr.
Data synthesis via a large language model generator. We use an instruction-tuner LLM Mgen and prompt it to generate prompt and completion pairs similar to those in D. In our experiments, we used the target model M, itself for Maens but any model capable of following detailed instructions can be used in its place. See Appendix A.1 for the prompt we used for data synthesis.
Data filtration via large language model discriminator: To train a discriminator that would be able to effectively filter synthetic data, determining how close a synthetic sample is to the marginal distribution of the inputs in D, we need synthetic sample set. This synthetic sample set is serve 'negatives' for the discriminator training while the 'real' inputs from D serve as positives. As stated above, during subsequent transfers of the PEFT model future models Me we use these M, models themselves for the synthetic data generator. However, we do not have access to them during the discriminator training (as it is trained in paral- lel to the source PEFT model). Hence, we use synthetic data generated from M,, for our discriminator train- ing and surprisingly find that the resulting discriminator generalizes well to filter synthetic data for a variety of unseen downstream generators (M,) as evaluated in our experiments (Section (4). For our discriminator,
we use an LLM, M endowed with small set of learnable parameters, Φ. We learn &i by optimizing,
Φ" argmax Ex- blogg Ma ("yes") [()))]+ +Eo-MM [logp My ("no" [noo)
where, we use t(x) to represent the prompt, "L An Is the above question from NAME dataset?" and replace NAME with a short descriptor identifying the dataset from which D is drawn; and æ~M, represents sampling from our synthetic data generation process for the task as explained above (with M, as the generator LLM in this case). See Appendix A.I for the specific prompts we used. In our experiments, we used the source model M, and LoRA to instantiate M disc.

---

Table 1: BigBench-Hard (BBH) collection averaged zero-shot results.
The accuracies listed are averages
of all 27 tasks from this collection. Evaluated using LM-Eval Hamess [8].
Source
Target
Discriminator
Source Model
Target Model no LoRA Acc.
Ours
Model
Model
Model
LoRA Acc.
Llama-2-7b
Llama-2-133
Llama-2-7b
43.32
37.85
43,41
Gemma-2b
Gemma-7b
Gemma-2b
31.84
37.75
43.61
Llama-2-7b
Gemma-7b
Gemma-2b
43.32
37.75
45.41
Llama-2-7b
Gemma-7b
Llama-2-7b
43.32
37.75
44.12
Curating Dayn- At the time of PEFT transfer, we create Day by filtering generations from M Mgen with the trained discriminator, M Missc We incorporate x~ MMgn into Dayn if MMM unable to recognize æ as synthetic sample, i.e., PM ("yes"[1(<))) > P Ma ("no" ((x)). Otherwise we discard æ. We repeat this rejection sampling procedure till the cardinality of Dayn equals that of D.
We summarize our overall Trans-LoRA algorithm in Algorithm I and Figure
2
4 Experiments
Liama-13b
BBHH Gemme- Cemme
Algorithm 1 Trans-LoRa
Require: 2,0,.. MM, MMMMe M
Mgen+ M-
Dayn Ø
while (Daynl < do s generate( (gen)))
if veriffy(M vdiso then
6BH. Lomma-7 Gemma-7a
Liama-7b Gemma-78 7
Dayn + D Dayn
end if
end while Initialize O,
Traederee
Tordonno
00 T
H + Loss))
while O. converged do
L (O, , (Deym))
Suba State PEFT PEFTAccusaty
Modal PEFT Accunety
0, update ().
Transferred LoRA accuracy VS. source LoRA accuracy on BBH tasks. Details the rows of Table Bottom left: row 3; Bottom right: row 4.
Figure 3:
end while
4.1 Experimental Setup
We have evaluated the effectiveness of our Truns-LoRA on two popular LLM families: Llama-2 [5 and Gemma [4, using 86 tasks from a large variety of topics from the following popular benchmarks:
BigBench-Hard (60 (27 reasoning tasks), Massive Multitask I Language Understanding (MMU)) 28 (57 knowledge tasks), Mostly Basic Python Problems (M (1 code task), and Grade School Math (GSMSK) [10] (1 math task). BBH is collection of 27 tasks where pre-existing LLMs could not outperform human evaluators. The tasks cover many different formats including multiple-choiee question answering. and short response. MMLU consists of 57 multiple-choice QA tasks testing common academic subjects with several difficulty levels. MBPP is a set of Python code generation problems with given problem descriptions and test cases. We also report results MBPP+[41) which is built upon MBPP with more strict evaluations and added test cases. GSM8K dataset consists of large number of grade school math problems. Due the large number of training samples in GSM8K, only pick the first 250 samples for fine-tuning our source LoRA models, and keep the number filtered synthetic samples to 250 as in our other experiments.
More specifically, we attempted 4 groups of experiments of LoRA transfer on each collection of tasks: 1. transfer from Llama2-7b to Llama2- 13b with Llama2-7b based discriminator; 2. transfer from Gemma-2 2b to Gemma-7b with Gemma-2b based discriminator; 3. transfer from Llama2-7b to Gemma-7b with Gemma-2b based discriminator; and 4, transfer from Llama2-7b to Gemma-7b with Llama2-7b based discriminator. We used the chat versions of Llama and the base versions of Gemma, thus exploring both
5

---

Table 2: Massive Multitask Language Understanding (MMLU) collection averaged zero-shot results.
Accuracies are averages of all 57 tasks from this collection. Evaluated using LM-Eval Hamess [18]
Source
Target
Discriminator
Source Model
Target Model
Ours
Model
Model
Model
LoRA Acc.
no LoRA Acc.
Llama-2-7b
Llama-2-130
Llama-2-7b
45,89
53.72
55.09
Gemma-21 2b
Gemma-7b
Gemma-2b
42.34
60.45
61.23
Llama-2-7b
Gemma-78 7b
Gemma-2b
45.89
60.45
61.12
Llama-2-7b
Gemma-7b
Llama-2-7b
45.89
60.45
61.22
MHLU, to Lleme:13b
MMUU, Gemme-22 Gemmee77
BBH, Gemma-2b to Gemma-77 7b
Arsnery
0.44
Transferred Accuracy
Model PETT Accuracy
Base Nocel PEIT Accuracy
NMUU, Ul Gemma-70
DBH, Liema 7b to Gemmes 7b
Mone do 1 Pg
To
041
de Number of Filtered Synthetic Samples
Figure 5: Scaling the number of synthetic samples generated through Trans-LoRA. Total training iterations in each experiment are kept identical for fair comparison. Done BBH with Gemma-20 2b to Gemma-7b transfer and Gemma-26 2b as discriminator.
Base Hodel PERT Accunacy
Dion Mozel PETT Accuracy
Figure 4: Transferred LoRA accuracy source LoRA accuracy on MMLU tasks. Details the rows of Table 2 Bottom left: row 3; Bottom right: row 4.
within and across chat and base models LoRA transfer. We evaluate BBH, MMLU, and GSMSK using the Language Model Evaluation Harness [18], and we evaluate MBPP/MBPP+ using Evalplus [40]. We evaluate all our models under the zero-shot setting.
Hyperparameter-wisn wise, we search the learning rate between 2 10-4 and 2 10-5 on the validation set using the AdamW optimizer with weight decay and linear learning rate scheduler without warmup. We end up adopting 2 * 10 for MMLU and 2 10 for all other tasks. We use a fixed 20 epochs for BBH, MBPP, and GSM8K and 10 epochs for MMLU, We train on the default LoRA configuration (adapters built only on query and value matrices of attention block) with effective batch size 8 (gradient accumulation used for larger models). We on average 10 hours to finish. All tasks can be parallelized. 4
1 V100 40GB GPU per transfer task. Each task takes
4.2 Main Results
In Tables to 4 we summarize the results for each task collection (BBH, MMLU, MBPP, and GSMSK respectively) for each source and target model combination. We test each task individually, and the results each table are obtained by averaging over all the tasks in the respective collection. We observe that the LoRA models transferred by our Trans LoRA consistently outperform both the source LoRAs and the target base models, demonstratigg that our transfer is indeed lossless. Moreover, this suggests that our Trans-LoRA is effective combining the information from LoRAs weaker source base model with the improved capabilities of a stronger target base model to create LoRAs on the target that are more powerful than both of them. And our Trans-LoRA is nearly data-free requiring almost access to original tasks training data (beyond the 5 seed examples). We see that our Trans-LoRA consistently attains successful LoRA transfer independently of specific combination of source, target, discriminator models, the initial relative performance difference between the fine-tuned source LoRAs and the target base-models. We note that the performance increase of our transferred model is relatively smaller on MMLU compared to other tasks. As MMLU tasks are more knowledge-focused, we believe the pretraining is more influential
*Our code is provided in Supplementary and will be released upon acceptance. See Appendix A.3

---

Table 3: Mostly Basic Python Problems (MBPP) zero-shot results. Presented in format of (standard MBPP
evaluation/ / more strict MBPP+ evaluation). Evaluated using Evalplus 44
Source
Target
Discriminator
Source Model
Target Model no LoRA Acc.
Ours
Model
Model
Model
LoRA Ace.
Llama-2-7b
Llama-2-138
Llama-2-7b
27.2/25.0
37.1/31.7
39.7/34,4
Gemma-2b
Gemma-7b
Gemma-2b
41.1/33.9
37.9/32.1
50.0/40.6
Llama-2-7b
Gemma-7b
Gemma-2b
27.2/25.0
37.9/32.1
48.7/42.0
1lama-2-7b
Gemma-7b
Llama-2-7b
27.2/25.0
37.9/32.1
48.7/42.0
Table 4: Grade School Math 8K (GSM8K) no chain-of-thought prompting results.
Source Model
Target Model
Discriminator
Source Model
Target Model
Ours
Model
LoRA Acc.
no LoRA Acc.
Llama-2-7b
Llama-2-11 1amma-2-13b
Llama-2-7b
19.64
28.86
30.70
Gemma-2b
Gemma-70 7b
Gemma-26
14.94
40.64
44.58
Llama-2-7b
Gemma-7b
Gemma-2b
19.64
40.64
42.30
Llama-2-7b
Gemma-7b
Llama-2-7b
19.64
40.64
41.62
Table 5: Distillation curriculum ablations 27 tasks of the BigBench-HHard (BBH) collection.
Source Model PEFT no PEFT
Target Model
Unfiltered Synthetic Data
Random Wikipedia
Seed Samples
Model Config
Ours
Acc.
Acc.
Source: Llama-2-1
Target: Llama-2, 2-13
43.32
37.85
37.32
41.95
39.82
43.41
Discriminator: Llama-2-7b
Table 6: Trans-LoRA for transferring between different base models and different PEFT methods on BigBench-Hard (BBH). Accuracies zero-shot averaged results of all tasks from this collection.
Source
Target
Source
Target
Discriminator Model
Model
Model
Ours
Model
Model
PEFT
no PEFT
Acc.
Acc.
Gemma-2b (LoRA)
Gemma-76 (LoRA)
Gemma-22 2b
31.84
37.75
43.61
Gemma-2b (LoRA)
Gemma-7b (DoRA)
Gemma-2b
31.84
37.75
40.74
Gemma-2h (DoRA)
Gemma-7b (LoRA)
Gemma-2b
33.07
37.75
41.81
Gemma-26 2b (DoRA)
Gemma-7b (DoRA)
Gemma-2h
33.07
37.75
41.40
Gemma-2b (LoRA)
Gemma-7b (PT)
Gemma-2h
31.84
37.75
43.99
Gemma-20 (PT)
Gemma-7b (LoRA)
Gemma-20
33.25
37.75
38.14
Gemma-2b (PT)
Gemma-7b (PT)
Gemma-2b
33.25
37.75
42.90
than the finetuning for MMLU. We also experimentally verified that increasing finetuning epochs (without adding more synthetic data) on MMLU does not lead to further improvements.
For more details, Figures and Sshow detailed distribution of LoRA transfer results for each task from the BBH and MMLU collections. We see that both cases, the majority of data points are near or above the y = line (the dotted line), indicating transfemed target LoRAs match outperform the source ones. These individual task distributions demonstrate the robustness of our Trans-LoRA, We analyze the few outliers in Section 5
4.3 Ablation Experiments
Distillation Data
Here we evaluate the effect of the choice of the input data for distillation. As varying kinds transfer on numerous tasks time and resource-consuming, we run this ablation only on BBH tasks and the "between Llama-2 models transfer' (most challenging, smallest gains) objective. Results are summarized T
We compare distilling the source LoRAs on: (I) random Wikipedia text; (2) raw

---

Table 7: Continuous transfer on several models on BigBench- Hard (BBH). We transfer from source model to intermediate model, then from intermediate model to target model, all using the same discriminator model. Accuracies are zero-shot averaged results of all tasks from this collection.
Source
Intermediate
Our Transferred Intermediate Model
Target Model LoRA Acc.
Our Transfemed
Model
Model LoRA
Model Config
LoRA
Target Model
Acc. Acc.
Source: Llama-2-7b
Intermediate: -lama-2-13h
43.32
37.85
43.41
37.75
45.04
Target: Gemma-7b
Discriminator: Llama-2-7b
Table 8: Experiments on T5 models and 3 additional tasks, where our results are reported on Trans-LoRA transfer from T5-L finetuned LoRA to T5-XL base model.
Dataset
T5-L Finetuned
T5-XL Base
Ours
Coga
32.60
55.84
61.44
Newsroom
85.09
84.19
85.70
Squadv2
95.40
96.32
98.48
synthesized samples without discriminator filtering: (3) only the 5 seed samples used for data synthesis; and (4) our ToRAA-LoRA. From Table we see that our Trans-LoRA outperforms other baselines by a large margin, indicating that: (a) synthetic data designed to mimic task data is highly beneficial, and random or seed data does not suffice; and (b) discriminator filtering is effective providing good gains over raw synthetic data. These results further verify hypothesis the importance of the proximity of distillation inputs to the original training data.
Other PEFT Methods To further illustrate the robustness and wide upplicability of our Trans-LoRa. we test its ability to transfer non-LoRA PEFT models. In particular, we apply our Trans-LoRA to Weight-Decomposed Low-Rank Adaptation (DoRA) 43], and Prompt Tuning (PT) (37]. For DoRA, use the same set of hyperparameters as LoRA, and for Prompt Tuning we use a higher learning rate of 2 10-3 and initialization text provided in Appendix A.2 Table 6 indicates that despite the change of the specific PEFT approach, we can achieve satisfactory results upon transfer.
Continuous Transfer To further verify the practical use-case of using Trans-LoRA for several trans- fers in a row, we evaluate continuous transfer, where the LoRA model is transferred from source to target via an internediate model. The discriminator model is kept the same throughout this process, closely mim- icking real-world application scenarios where the discriminator model needs to be re-used for all subsequent transfers. From Table 7 we see that continuous transfer does lead to degradation in performance. This result proves the robustness and practicality of our Toans-LoRA, where the client only needs to deliver the discriminator and trained PEFT once allow for multiple future transfers to different future base models.
Scaling the amount of Synthetic Samples Another advantage of our Trans-LoRA is the theoretically unlimited data synthesis process. In all previous experiments, we kept the number of filtered synthetic samples to be the same as the number of samples in the original training dataset (set to 250), We show in Figure our Trans-LoRA exhibits good scaling behavior w.r.t. the number of filtered samples generated, which gives the user the freedom to balance the trade-off between final task accuracy and total compute.
Additional experiments To demonstrate the effectiveness of our approach on wider range of tasks and models, we performed additional experiments on T5 series model and 3 additional tasks. The results are shown in Table

---

Table 9: Maximum mean disr)))ancy(MMMD) comparing filtered and unfiltered synthetic data with original dataset using first 4 tasks of BBH. Smaller values indicate smaller distance original dataset.
Task Name
Filtered Data MMD
Unfiltered Data MMD
boolean expressions
0.7155
1.3072
causal judgement
0.2255
0.7714
date understanding
0.2438
0.8282
disambiguation QA
0.2097
0.9231
5
Analysis
5.1 Cost analysis
Our Trans-LoRA relies on training additional discriminator. We summarize the empirical cost associated with this process and demonstrate that it only incurs a negligible overhead. Discriminator training typically takes just I epoch reach over 90% accuracy. LoRAs were trained for 20 epochs in our experiments which was empirically observed to produce the best performance for the source LoRA models, Given that both discriminator training and LoRA training used equal number of samples per epoch (half real half synthetic for discriminator training, full real data for LoRA training), the cost of training discriminator is only around 1/20 of the training cost of LoRA modules. Synthetic data generation for training discriminators took less than 5 minutes for most tasks on a single V100 (for all base models). These costs are almost negligible compared training the source LoRAs.
5.2 Distribution of filtered synthetic data
In order to provide a more direct understanding of the difference between filtered synthetic samples from our Trans-LoRA and unfiltered raw synthetic samples, we encode each sample into vector representation using MPNet encoder [57] and calculate maximum mean discrepancy (22 on the encoded representations. The maximum mean discrepancy can viewed as measure of distance between two high dimensional distributions, or in other words how much of the original distribution can be explained by the given distribution. We run this analysis on the first BBH tasks with synthetic data filtered by their respective Llama2-7b discriminators from the 7bb to Llama2-13b LoRA transfer experiment. From Table 9 we clearly observe lower MMD values for our filtered synthetic data, confirming the utility of the discriminators employed in ToRAs-LoRA
To prove that the data we generate through Trans-LoRA fundamentally different from the original data and the discriminator in Trans-LoRA does not simply memorize original samples, we performed further analysis on the us foreign_policc task under MMLU. We find the closest pair of questions from the real data and our synthesized data under the embedding space of a pretrained MPNet. This closest pair has Fuclidean distance of 0.604, which indicates that there is absolutely no overlap between synthetic samples and real samples. This closest pair consists of: "What were the implications of the Cold War for American exceptionalism?" (real) and "What was the significance of the Cold War to the development of American foreign policy?" (synthesized). These questions are asking for completely different aspects of the subject. We also exhibit the T-SNE plot on the embeddings in Appendix (Figure 77.. Although the distributions of synthetic data and real data similar, they do not share any identical points.
5.3 Error Analysis
We see from Figures B and 4 that for very few of our 86 evaluated tasks, the performance of LoRAs transferred by our LoRA may become lower than the baseline. In this section, we take closer look at one such specific example task: Disambiguation-O2 from BBH to analyze why this occurred.
Insufficient understanding of task Example comparison of problematic synthetic vs. real task data is in Figure [ The generated synthetic question is not valid because none of the answers is correct. In this example, the generator model does not seem have correctly understood the task intent; rather, just mimicked the pattemn of the real samples. We observe similar failed samples for (the few) other tasks residing under the Figures and 4 diagonals.
Solution
We observe that increasing the number of real samples used to prompt the data synthesis (i.e., increasing (D)) can effectively help the generator model to learn the inherent reasoning and structuring

---

the following
explain antecssents pronoum (which thing the pro- refers tol, or state that it ambiguous,
following sent eniced,
explain the antecedent of the
jabich thing the
retere or state that
Sentence: Everyone in the class had bo weer und form except
1= anhi gueus.
Sentence: Alex telly us thes
could
someching
Options:
Options:
Alex could not neet
[6] The unszoen
(B) Ne could not meet Arbiguaus
Somethi
Ambiguous
Figure 6: Comparison of problemtiic synthetic sample (left) and real sample (right) from disambiguation-qa task.
Figure 7: T-SNE plot of MPNet embeddings from us us_ us__foreign_polie (MMLU) dataset; Red points are filtered synthetic data, blue points are real data.
of the task questions. Increasing the number of samples from 5 to 15 on disanbiguation-ga, for example, leads to much more robust and realistic synthetic samples and significantly improved (up by 13%) performance. Thus, we recommend tuning the number of seed samples for synthesis when generated samples are not logically coherent and do not follow the task intent.
For more detailed analysis of our synthetic samples, we include T-SNE plot of our samples under pretrained embedding space Figure 7
Conclusions and Limitations
In this paper, we propose Trans-LoRA, an approach capable nearly data-free LoRA model transfer between different base models (and even supporting transfer between different PEFT configurations) without requiring access original task data. Trans-LoRA achieves equivalent or better performance when compared with the source LoRA and the target base model. To our knowledge, this paper is the first to explore the very practical use case of transferability of PEFT models. We hope that the success of our approach will inspire future explorations in this exciting research direction.
Limitations Our Trans-LoRA requires synthesizing data before the transfer requiring small, yet additional, compute. A promising future direction is to explore ways of direct PEFT transfer, without additional computation. Additionally, we discussed potential limitation task understanding by the synthesizer, observed in few cases, and offered a path to mitigate it. We work with LLMs in our experiments, and although LLMs can sometimes produce harmful content, we rely on their authors for proper alignment.
10

---

7 Acknowledgements
This work was funded by MIT-IBM Watson Al Lab.
References
[1] John M Abowd and Lars Vilhuber. How protective are synthetic data? In International Conference Privacy in Statistical Databases, pages 239-246. Springer, 2008.
[2] Al@Meta. Llama 3 model card. 2024.
[3] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep leaminn. arXiv preprint arXiv:2012.09876, 2020.
[4] Authropic. The claude 3 model family: Opus, sonnet, haiku, 2024.
[5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
[6] Duhyeon Bang. Jongwuk Lee, and Hyunjung Shim. Distilling from professors: Enhancing the knowledge distillation of teachers. Information sciences, 576:743-755, 2021.
Verónica Bolón-Canedo, Noelia Sánchez-Marofio, and Amparo Alonso-Betanzos. A review of feature selection methods on synthetic data. Knowledge and information systems, 34:483-519, 2013.
[8] Päl H Brekke, Taraka Rama, Iidikó Pilán, Øystein Nytro, and Lilja Øvrelid. Synthetic data for annotation and extraction of family history information from clinical text. Journal of Biomedical Semontics, 12:1-11, 2021.
[9] Collin Bums, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.089
2023.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichirr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[11] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53-66 65, 2018.
[12] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5(3):200-33,, 2023.
[13] Gemini Team al. Gemini: family of highly capable multimodal models, 2024.
[14] Gemma Team et al. Gemma: Open models based gemini research and technology. 2024.
[15] Hugo Touvron et al. Llama 2: Open foundation and tneettnedd chat models, 2023.
[16] OpenAl et al. Gpt-4 technical report, 2024,
[17] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai I La,, Lidong Bing, and Nigel Collier. On the effectiveness of parammetr-eeffiient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence pages 12799-12807, 2023.
[18] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023.
[19] Mohsen Gholami, Mohammad Akbari, Cindy Hu, Vaden Masrani, Z Jane Wang, and Yong Zhang. Gold: Generalized knowledge distillation via pul-of-distrrbution--uuee language data generation. arXiv preprint arXiv:2403.11 19754, 2024.
[20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications the ACM, 63911)1199-14, 2020.
[21] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey.
International Journal of Computer Vision. 129(6) 1819, 2021.
11

---

[22] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf. and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773. 2012.
[23] Roman Grundkiewicz, Marcin Lunzzys-Dowmutz, and Kenneth Heafield. Neural grammatical error correction systems with unsupervised pre-training on synthetic data. In 14th Workshop on Innovative Use of NLP for Building Educational Applications, pages 252-263. Association for Computational Linguistics, 2019.
[24] Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. review on generative adversarial networks: Algorithms, theory, and applications. IEEE transactions on knowledge and data engineering, 35(4)33313 3332, 2021.
[25] Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficien fine-tuning for large models: comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
[26] Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, fa- Low, Lidong Bing, and Luo Si. On the effectiveness of adapter- based tuning for pretrained language model adaptation. arXiv preprint arXiv:2106.0311 2021.
[27] Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohamuaad Norouzi, Generate, annotate, and Ieam: NIp with synthetic text. Transactions of the Association for Computational Linguistics, 10:826-844 842, 2022.
[28] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2021.
[29] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1rXiv:1303.0254/ 2015.
[30] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Purameter-efffcient transfer learning for nlp. In International conference machine learning pages 2790 2799, PMLR, 2019.
[31] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[32] Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. Knowledgeable tonntttuunnng: Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint (tt::221800000355 2021.
[33] I-Hong Jhuo, Dong Liu, DT Lee, and Shih-Fu Chang. Robust visual domain adaptation with low-rank recon- struction. In 2012 IEEE conference on computer vision and pattern recognition, pages 2168-2175. IEEE, 2012.
[34] Gal Kaplun, Eran Malach, Preetum Nakkiran, and Shai Shalev-Shwartz Knowledge distillation: Bad models can good role models. Advances in Neural Information Processing Systems. 35:28683-288 2266342 2022.
[35] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint 16X.v.1600.07947, 2016.
[36] Soroush Abbasi Koohpayegani. KL Navaneet, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. Nola: Compressing lora using linear combination of random basis, 2024.
[37] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv22104.0899, 2021.
[38] Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang. Xingxing Zhang, Haoyang Huang, Shaohan Huang. Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Symthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.1 13064, 2024.
[39] Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot efffmeter-efficientt fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
[40] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In seirth- Conference on Neural Information Processing Systems, 2023.
[41] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[42] Ming-Yu Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. Advunces in neural information processing systems, 29, 2016.
[43] Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanow, Yu-Chiang Frank Wang, Kwang-Ting Ting Cheng, and Min-Hung Chen. Dora: deight-deoomposed low-rank adaptation, 2024.
[44] Xinyin Ma, Xinchao Wang, Gongfan Fang, Yongliang Shen, and Weiming Lu. Prompting to distill: Boosting data-free knowledge distillation via reinforced prompt. arXiv preprint urXiv:2205.07523, 2022.
[45] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang 1i, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, pages 5191-5198, 2020.
[46] Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orea Teaching small language models how to reason. arXiv preprint urXiv:2311.1045, 2023.
[47] Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
[48] Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, arXiv preprint arXiv:2306.02707, 2023.
[49] Sergey Nikolenko. Synthetic data for deep learning. Springer, 2021.
[50] OpenAI. https///openai.com//dde///gggg
murbo-fine-tuning--eee upiuttssess. 2024.
[51] Wompyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 3967-3976, 2019.
[52] Neha Patki, Roy Wedge, and Kalyan Voeramachaneni. The synthetic data vault. In 2016 IEEE international conference on data science and advanced analytics (DSAA), pages 399-410. IEEE, 2016.
[53] Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International conference on machine learning, pages 5142-5151. PMIR,, 2019.
[54] Vihari Piratla, Pranceth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via -ommon-specific low-rank decomposition. In International Conference on Machine Learning, pages 7728 7738. PMLR, 2020.
[55] Raul Puri, Ryan Spring. Mostofa Patwary. Mohammad Shoeybi, and Bryan Catanzaro. Training question answering models from synthetic data. arXiv preprint (tt:::2002.099999 2020.
[56] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-lora Serving thousands of concurrent lora adapters, 2023.
[57] Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding, 2020.
[58] Shivchander Sudalairaj. Abhishek Bhandwaldar. Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava. Lab: Large-scale alignment for chatbots. arXiv preprint 20000004,,,,, 2024.
[59] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer leaming for vision-and ando- language tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5227-5237, 2022.
[60] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thougle solve them, 2022.
[61] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiinng lama model. httpa://github.
sttnsardddddddccaaaaa 2023.
[62] Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 969-977, 2018.
13

---

[63] Chaofei Wang. Qisen Yang. Rui Huang, Shiji Song, and Gao Huang. Efficient knowledge distillation from model checkpoints. Advances in Neural Information Processing Systems, 35:607-619, 2022.
[64] Dingquan Wang and Jason Fisner. Syuthetic data made to order: The case of parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
[65] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint aXXv:2212.105660 2022.
[66] Zhen Wang. Rameswar Panda. Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Multitask prompt tuning enables parameter--fficiett transfer learning. 2023.
[67] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng. Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244. 2023.
[68] Jiacheng Ye, Jiahui Gao, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Progen: Progressive zero-shot dataset generation via in-context feedback. arXiv preprint arXiv:2200.13229, 2022.
[69] Xiyu Yu, Tongliang Liu, Xinchao Wang. Dacheng Tao. On compressing deep models by rank and sparse decomposition. In Proceedings of the IEEE conference on compuder vision and pattern recognition, pages 1370-7379, 2017.
[70] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficiet fine-tuning for transformer based masked dunguage-moder models. arXiv preprint arXiv:2106. 10199, 2021.
[71] Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513: 2023.
[72] Binjie Zhang, Yixiao Ge, Xuyuan Xu, Ying Shan, and Mike Zheng Shou. Taca: Upgrading your visual foundation model with task-agnostic compatible adapter, 2023.
[73] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEF/CVF international conference computer vision, pages 3713 3722, 2019.
[74] Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillatin:: Towards efficient and compact neural networks. IEEE Transactions Puttem Analysis and Muchine Intelligence, 44(8)-43884444 0403, 2021.
[75] Rongsheng Zhang, Yinhe Zheng, Xiαoxi Man, and Minlie Huang. Unsupervised domain adaptation with adapter. arXiv preprint arXiv:2111.00667 2021.
[76] Zhilu Zhang and Mert Sabuncu. Self-distilatiin as instance-specific label smoothing. Advances in Neural Information Processing Systems, 33:2184-2199 2020.
(77) Yong Zhao, Jinyu Li, and Yifan Gong. Low-rank plus diagonal adaptation for deep neural networks. In 2016 IEFE Internaionnal Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5005-5009, IEEE, 2016.
A Appendix
A.1 Prompt Examples
Figure 8: Example prompt used in data synthesis for boolean expressions task from BBH.
Here are 10 examples:
1. True and False or ( not True is
2. not not True and not False or True is
3. not False and False
False or False is 4. True or False or not True or False is
not not ( False and not False is
6.
14

---

Figure 9: Example prompt used in data filteration for boolean expressions task from BBH.
Answer in as few words as possible. True and False not True is Is the above question from the boolean expressions dataset?
A.2 Prompt Tuning Initialization
Figure 10: Initialization for prompt tuning.
Answer the following question correctly:
A.3
Code
Our code available at hutps//gtthbb.ooomaaaaaa////rttr
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All of our claims have been verified in Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create a separate "L.imitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to viola- tions of these assumptions (e.g.. independence assumptions, noiseless settings, model well- specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimeess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We do not include any theoretical result in paper, and verify our proposed approach through extensive experiments.
16

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross-refereneed.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include a detailed description of our method Section 3 We have provided the settings of our experiments (including hyperparameters, models, evaluation methods, etc.) in Section 4. and the code in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibilltty can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may he necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. of closed-source models, it may be that access to the model is limited in some way (e-g., to registered users), but should be possible for other researchers to have some path to reproducing or verifying the results.
the case
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
17

---

Justification: Our code is available at Appendix A.3 and would be released upon acceptance. All datasets we use are publicly available with citations included. Our evaluation metrics are standard (open source, properly linked and provided with the code) and properly documented. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines lattpss//nips: cc/ public/ for more details.
guides/Codese wwweeSbmissiinnoonn..
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines
/ttee//ddddeeeeeeeuaaiiiiiii ottee/eaaaaiiissiiooooooiioooo.. soi.liiiiisioooooooiiii.. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide our settings in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We provide plots (Figure 3 and Figure 4 that describe the detailed statistical distribution of all data points to show the statistical significance of our results in full detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the enror bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar the standard deviation or the standard error of the mean.
18

---

It is OK to report I I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer:
[Yes]
Justification: We describe the compute we use (including GPU, memory, and time of execution) in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't tt make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpsh/wwwheee///eiiaa.a...oaaaa http://acc//ttthhcccc.....e...ee
Question:
Answer: [Yes]
Justification: We have carefully reviewed the Code of Ethics and confirm that this research follows it completely.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethies.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make
preserve anonymity (e.g., if there special consideration
due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section and Section 6. we discuss the potential social impact of our research, in particular how it impacts current client-service provider relation.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality generative models could he used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for op- timizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any data models with high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with neces- sary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have authored the code and cited all relevant work used in this paper in our References section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswii thcode. cww. datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [Yes]
Justification: We provide detailed comments in our code and a README file for documentation. See Appendix A.3
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasse/ccdeemmodel as part of their submis- sions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing research with human subjects is performed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No crowdsourcing or research with human subjects is performed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "3jyaawqnz0", "reviewer": "Reviewer_qZWK", "summary": "The paper presents Trans-LoRA, a novel approach for transferring Low-Rank Adapter (LoRA) parameters across different base models without requiring access to the original task data. Trans-LoRA utilizes synthetic data generation and a discriminator model to filter this synthetic data, ensuring that the transferred LoRA parameters maintain or improve performance. The effectiveness of the method is validated through experiments on LLaMA and Gemma model families, demonstrating its utility across various tasks.", "questions": "See weakness", "limitations": "The authors address several limitations, including the need for synthetic data generation and the potential computational overhead. However, the discussion could be expanded to include potential biases in synthetic data, variability of performance across different tasks, and the cost implications of the discriminator overhead in a cloud scenario. Additionally, more detail on the safeguards against data privacy risks would be beneficial.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-The paper introduces a novel solution for transferring LoRA parameters across different base models without needing the original task data.\n-The combination of synthetic data generation and discriminator filtering is innovative and well-suited for Parameter-Efficient Fine-Tuning (PEFT).\n-The paper is well-organized and clearly written, with comprehensive explanations of the problem, proposed solution, and experimental setup.", "weaknesses": "-While the paper demonstrates the effectiveness of Trans-LoRA on LLaMA and Gemma models, evaluations on a wider range of models and tasks would strengthen its generalizability.\n-The reliance on a discriminator introduces an overhead, and the paper does not fully address who will bear this cost in a cloud scenario.\n- The paper relies on synthetic data generation, which can introduce biases and limitations. A more detailed discussion on these aspects would strengthen the paper.\n- Exploring the potential integration of other synthetic data generation methods like ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback and GOLD (Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation) could enhance the data quality and effectiveness.\n- Although the synthetic data generation process and discriminator filtering are designed to minimize risk, the potential for synthetic data to inadvertently reflect proprietary information remains a concern. Further safeguards and monitoring practices should be discussed to ensure data privacy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for valuable feedback and comments on our paper. We appreciate the opportunity to address your concerns and clarify any misunderstandings. Below, we provide detailed responses to each of your comments.\n\n>While the paper demonstrates the effectiveness of Trans-LoRA on LLaMA and Gemma models, evaluations on a wider range of models and tasks would strengthen its generalizability. \n\nWe thank the reviewer for this suggestion. To better illustrate the generality of our Trans-LoRA approach we include the following experiment on the popular T5 model with Coqa, Newsroom, and Squadv2 datasets. The results included in the table below illustrate that our approach can be also effectively applied to more tasks and models. We will include this result in the final manuscript.\n| Dataset | T5-L Finetuned | T5-XL Base | Ours |\n|:-:|:-:|:-:|:-:|\n| Coqa | 32.60 | 55.84 | 61.44|\n| Newsroom |85.09|84.19|85.70|\n| Squadv2 | 95.40|96.32|98.48|\n  \n>The reliance on a discriminator introduces an overhead, and the paper does not fully address who will bear this cost in a cloud scenario.\n\nThank you for this suggestion, we are happy to elaborate on the discriminator training overhead and would include this detail in the final version of the paper. Discriminator training typically takes just 1 epoch to reach over 90% accuracy. LoRAs were trained for 20 epochs in our experiments which was empirically observed to produce best performance for the source LoRA models. Given that both used equal amounts of samples per epoch (half real half synthetic for discriminator training), the cost of training discriminator is only around 1/20 of training cost of LoRA modules. Synthetic data generation for training discriminators took less than 5 minutes for most tasks on a single V100 (for all base models). These costs are almost negligible compared to training the original LoRAs.\n\n> The paper relies on synthetic data generation, which can introduce biases and limitations. A more detailed discussion on these aspects would strengthen the paper.\n\nThanks for this suggestion! However, we have already provided a detailed MMD analysis of the synthetic data distribution in section 5.1, also ablating how the discriminator-based filtering improves on the MMD with the real data distribution. Moreover, we also highlighted some potential failures of synthetic data generation and even provided a potential mitigation addressing such failures in Section 5.2. We will further emphasize the importance of these aspects refering to these sections from the method section.\n\n>Exploring the potential integration of other synthetic data generation methods like ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback and GOLD (Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation) could enhance the data quality and effectiveness.\n   \nOur primary contribution is proposing a methodology (Trans-LoRA) that demonstrates the possibility of using synthetic data for lossless LoRA transfer. Currently our experiments used a simple and straightforward prompting approach, while still demonstrating lossless and even positive transfer in all cases, further highlighting the effectivity of our Trans-LoRA approach. We believe that other synthetic generation methods, such as the great methods referenced by the reviewer, are orthogonal to our approach. Any good generation methods can be applied together with our Trans-LoRA, which is a promising direction left for future work to explore. We will add this to the discussion section of the final version of the paper, also citing the works mentioned by the reviewer (that are certainly applicable for this orthogonal future work extension).\n\n> Although the synthetic data generation process and discriminator filtering are designed to minimize risk, the potential for synthetic data to inadvertently reflect proprietary information remains a concern. Further safeguards and monitoring practices should be discussed to ensure data privacy.\n\nWe performed further analysis on the us_foreign_policy task under MMLU. We find the closest pair of questions from the real data and our synthesized data under embedding space of a pretrained MPNet. This closest pair has a Euclidean distance of 0.604 (so there is absolutely no overlap between synthetic samples and real samples) and consists of: “What were the implications of the Cold War for American exceptionalism?” (real) and “What was the significance of the Cold War to the development of American foreign policy?” (synthesized). Note that these questions are asking for completely different aspects of the subject. We also exhibit the T-SNE plot on the embeddings in the shared response PDF. Although the distributions of synthetic data and real data are similar, they do not share any identical points. We will add this analysis and discussion to the final version of the paper, thanks for suggesting!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqafdUyKIA", "reviewer": "Reviewer_K48P", "summary": "This paper proproses Trans-LoRA, a method that utilize synthetic data to transfer abilities learned using LoRA across different downstream tasks. Trans-LoRA first uses the source model for synthetic data generation. The generated synthetic data is used to train discriminator LoRA for filtering synthetic data for target model. Using filtered synthetic data together with source LoRA, the target model acquire synthetic data for training. \nTrans-LoRA is validated on various benchmarks and models, showing the effacy of the method.", "questions": "See Weaknesses", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is clearly written and easy to follow\n* Experiments are conducted on multiple benchmarks and different variants of LLaMA-2/Gemma\n* Experiments show clear improvements in performance on all benchmarks.", "weaknesses": "* I understand that the current trend is to apply PEFT on decoder-only LLMs like LLaMa, but I'm a bit worried that the method might not be generic enough. I think the method might be suitable to alternative model architectures, for example\n    * Encoder-only LMs like DeBERTa or RoBERTa. Since Trans-LoRA requires generation from the source model, it could be challenging to implement. One could work around this by using another LM with a decoder for generation/discrimination, but this approach seems suboptimal.\n    * Similarly, ViT from the vision community. \n* Also, the method seems to rely heavily on the generative ability of the source model/discriminator mdoel. I wonder if it's feasible to use weaker LMs like T5 or GPT2. The results in Table 5 also show that if we use random wikipedia (similarly to use a weaker LM), the performance drops significantly.\n* One of the nice property of the method is one only needs 5 real samples to conduct PEFT. But computation-wise, one still needs a relatively large LM to generate samples/discriminate. Is that actually favorable in practice?\n* minor: line 198: is it A100 40GB?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for valuable feedback and comments on our paper. We appreciate the opportunity to address your concerns and clarify any misunderstandings. Below, we provide detailed responses to each of your comments.\n\n> I understand that the current trend is to apply PEFT on decoder-only LLMs like LLaMa, but I'm a bit worried that the method might not be generic enough. I think the method might be suitable to alternative model architectures, for example\n    -   Encoder-only LMs like DeBERTa or RoBERTa. Since Trans-LoRA requires generation from the source model, it could be challenging to implement. One could work around this by using another LM with a decoder for generation/discrimination, but this approach seems suboptimal.\n    -   Similarly, ViT from the vision community.\n    \nWe thank the reviewer for this suggestion. To better illustrate the generality of our Trans-LoRA approach we include the following experiment on the popular T5 model with Coqa, Newsroom, and Squadv2 datasets. The results included in the table below illustrate that our approach can be also effectively applied to generative models beyond “decoder only” (encoder-decoder in this case), as well as to weaker models than LLaMA and Gemma. We will include this result in the final manuscript. The paper is initially intended for decoder-only architectures, but exploring encoder-only models could be an interesting future direction. \n| Dataset | T5-L Finetuned | T5-XL Base | Ours |\n|:-:|:-:|:-:|:-:|\n| Coqa | 32.60 | 55.84 | 61.44|\n| Newsroom |85.09|84.19|85.70|\n| Squadv2 | 95.40|96.32|98.48|\n\n>Also, the method seems to rely heavily on the generative ability of the source model/discriminator mdoel. I wonder if it's feasible to use weaker LMs like T5 or GPT2. The results in Table 5 also show that if we use random wikipedia (similarly to use a weaker LM), the performance drops significantly.\n\nThanks for the suggestion! In the above table, we tested with the T5 series model and demonstrated that our approach can be applied to these weaker language models as well on 3 separate tasks. We will include this result in the final manuscript. The experiment with transferring using random wikipedia data in Table 5 is only intended to show that data for the transfer is very important, it is not mimicking data generated from weaker models (as demonstrated by results on T5).\n\n>One of the nice property of the method is one only needs 5 real samples to conduct PEFT. But computation-wise, one still needs a relatively large LM to generate samples/discriminate. Is that actually favorable in practice?\n\nThank you for this suggestion, we are happy to elaborate on the discriminator overhead and would include this detail in the final version of the paper. Discriminator training typically takes just 1 epoch to reach over 90% accuracy. LoRAs were trained for 20 epochs in our experiments which was empirically observed to produce best performance for the source LoRA models. Given that both used equal amounts of samples per epoch (half real half synthetic for discriminator training), the cost of training discriminator is only around 1/20 of training cost of LoRA modules. Synthetic data generation for training discriminators took less than 5 minutes for most tasks on a single V100 for all base models. These costs are almost negligible compared to training the original LoRA.\n\nIn terms of synthesis during LoRA transfer, the generation process with the discriminator is quite fast (typically taking less than an hour for an entire dataset on V100). Moreover, our approach supports: \n\n1. re-use of previously synthesized dataset (see Table 7)\n\n2. parallelization of synthesis by using multiple GPUs. \n\nThis implies that our Trans-LoRA approach is effective even when we only synthesize the dataset ONCE for all future transfers, and this one time synthesis can be done fast. Additionally, we also showed above that it is possible to use a smaller LM (T5) for our approach, which would greatly speed up generation as well.\n\n>-   minor: line 198: is it A100 40GB?\n\nThank you for noting this! It means a V100 with 40GB of machine memory, rather than 40GB of GPU memory. We will make sure to clarify this in our final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WgQZJRMGOi", "reviewer": "Reviewer_7oKi", "summary": "This paper proposes a nearly data-free method for transferring pre-tuned PEFT components (e.g., LoRA) between different models. To address issues of data inaccessibility, the authors propose to generate synthetic data from the target base model. To ensure that this synthetic data is in-distribution, they introduce an additional discriminator, which is trained concurrently with the source PEFT component.", "questions": "See weaknesses.", "limitations": "The authors discuss several limitations:\n(i) increased costs associated with data generation,\n(ii) potential misunderstandings of the task.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is clear and interesting, focusing on compatibility issues between the base model and its PEFT components.\n2. The writing is well-crafted and easy to understand.\n3. The experimental design is robust, encompassing (i) compatibility within and across different base models, (ii) various PEFT methods, and (iii) a broad range of tasks.", "weaknesses": "1. The application scope of the proposed method appears limited due to the added constraints on training PEFT components: it necessitates training these components with an additional discriminator. This requirement is uncommon and incurs extra costs.\n2. Further discussion on scalability is needed. As the number of PEFT components grows, updating all components seems time-consuming. By contrast, the approach in [1] suggests keeping all PEFT components static while only updating the base model in a specific way.\n3. Additional properties of the synthetic data should be considered. While the proposed method focuses on in-distribution generation, the diversity of the synthetic data is also crucial.\n4. More baselines of data-free knowledge distillation are needed, such as [2].\n5. Given that the proposed method requires a small set of descriptive data, it may be more accurately described as \"data-efficient\" rather than \"data-free.\"\n\n[1] TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter, arXiv 2023.\n\n[2] Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt, IJCAI 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for valuable feedback and comments on our paper. We appreciate the opportunity to address your concerns and clarify any misunderstandings. Below, we provide detailed responses to each of your comments.\n\n>The application scope of the proposed method appears limited due to the added constraints on training PEFT components: it necessitates training these components with an additional discriminator. This requirement is uncommon and incurs extra costs.\n    \nThank you for this suggestion, we are happy to elaborate on the discriminator overhead and would include this detail in the final version of the paper. Discriminator training typically takes just 1 epoch to reach over 90% accuracy. LoRAs were trained for 20 epochs in our experiments which was empirically observed to produce best performance for the source LoRA models. Given that both used equal amounts of samples per epoch (half real half synthetic for discriminator training), the cost of training discriminator is only around 1/20 of training cost of LoRA modules. Synthetic data generation for training discriminators took less than 5 minutes for most tasks on a single V100 (for all base models). These costs are almost negligible compared against training the original LoRA.\n\n> Further discussion on scalability is needed. As the number of PEFT components grows, updating all components seems time-consuming. By contrast, the approach in [1] suggests keeping all PEFT components static while only updating the base model in a specific way.\n    \n\nWe appreciate your suggestion. We are glad to cite Taca [1] in the related work, it is similar to our work in terms of the usage of distillation. However, Taca is in fact targetting one (very general) type of downstream task when being trained - the task of Vision-and-Language modeling - making image encoder tokens “readable” by an LLM decoder. If Taca is to be applied in our setting, it also needs to be trained separately for each downstream task. In this sense, our work is as scalable as Taca. Addressing the reviewer's concern on scalability, although each PEFT module needs to be transferred separately, these transfers can be executed independently in parallel to greatly speed up the process.\n\n\n\n>Additional properties of the synthetic data should be considered. While the proposed method focuses on in-distribution generation, the diversity of the synthetic data is also crucial.\n    \n\nThank you for pointing this out. As shown in an example T-SNE plot for the us_foreign_policy task (part of MMLU), the distributions of synthetic data and real data are just as diverse. They have similar coverages and characteristics as is apparent from the T-SNE plot. We will include this plot in our final paper.\n\n\n> More baselines of data-free knowledge distillation are needed, such as [2].\n    \n\nThank you for providing this reference! In fact, the baseline of data-free knowledge distillation is equivalent to the \"unfiltered synthetic data\" column in our ablation (Table 5), where samples are generated by simple prompting. We fixed our synthesis prompt to a simple prompt throughout all of our experiments for consistency and simplicity. Our primary contribution is the methodology allowing the use of synthetic data for lossless LoRA transfer. We believe that our approach’s ability to obtain consistent positive transfer results in all experiments using just the simple prompts underline its effectiveness. We also believe that other synthetic generation methods such as [2] are orthogonal to our approach. Any good data synthesis method can be applied together with our proposed Trans-LoRA approach. Combining these methods is a promising research direction for future work. We are happy to cite this work and include this discussion in the final paper.\n\n>Given that the proposed method requires a small set of descriptive data, it may be more accurately described as \"data-efficient\" rather than \"data-free.\"\n\nWe thank the reviewer for noting this, and we fully agree. We actually used the term “nearly data-free” in all 8 descriptions in our main paper except only 1 place in line 46. We will fix the reference on line 46 in our final version. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for reviewing our paper and providing valuable and constructive feedback.  
We are grateful that the reviewers have highlighted our work as:

- well motivated (7oKi)
- innovative and novel in approach (qZWK)
- well-written and clear to understand (qZWK, K48P, 7oKi)
- experimentally solid and robust (qZWK, K48P, 7oKi)
   
We include supplementary materials in response to some of the reviewers' questions in the attached PDF.  
To summarize the major responses we have made in rebuttal:  

- We report the result of our approach on new models and datasets to demonstrate our applicability in weaker/smaller language models, encoder-decoder language models, and diverse model families.
- We address the questions on discriminator-related overheads, showcasing that our method only requires very little overhead cost.
- We discuss related works listed by reviewers and indicate that they are either orthogonal or complementary to our work.
- We include more evidence on the diversity and secureness of our filtered synthetic data, and provide specific quantitative examples to support the claims.

Again, we genuinely appreciate the input from reviewers and we thank all reviewers for their time and effort.

Meta-review: The rebuttal addressed most of the concerns raised by the reviewers and now all reviewers support accepting the paper. Hence, it will be accepted. The paper presents a novel approach for transferring LoRA parameters across various base models. The idea is empirically validated by several experiments. Please
Decision: Accept (poster)