$\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 19:59:07

Abstract: Low-rank adapters (LoRA) and their variants are popular parameter-efficient fine-tuning (PEFT) techniques that closely match full model fine-tune performance while requiring only a small number of additional parameters. These additional LoRA parameters are specific to the base model being adapted. When the base model needs to be deprecated and replaced with a new one, all the associated LoRA modules need to be re-trained. Such re-training requires access to the data used to train the LoRA for the original base model. This is especially problematic for commercial cloud applications where the LoRA modules and the base models are hosted by service providers who may not be allowed to host proprietary client task data. To address this challenge, we propose $\textit{Trans-LoRA}$ --- a novel method for lossless, nearly data-free transfer of LoRAs across base models. Our approach relies on synthetic data to transfer LoRA modules. Using large language models, we design a synthetic data generator to approximate the data-generating process of the $\textit{observed}$ task data subset. Training on the resulting synthetic dataset transfers LoRA modules to new models. We show the effectiveness of our approach using both LLama and Gemma model families. Our approach achieves lossless (mostly improved) LoRA transfer between models within and across different base model families, and even between different PEFT methods, on a wide variety of tasks.

Main paper: 
  
 Appendix:  


Decision: The rebuttal addressed most of the concerns raised by the reviewers and now all reviewers support accepting the paper. Hence, it will be accepted. The paper presents a novel approach for transferring LoRA parameters across various base models. The idea is empirically validated by several experiments. Please

Reviews: review: paper_summary: paper_summary: The paper presents Trans-LoRA, a novel approach for transferring Low-Rank Adapter (LoRA) parameters across different base models without requiring access to the original task data. Trans-LoRA utilizes synthetic data generation and a discriminator model to filter this synthetic data, ensuring that the transferred LoRA parameters maintain or improve performance. The effectiveness of the method is validated through experiments on LLaMA and Gemma model families, demonstrating its utility across various tasks.
strength_weakness: strength_weakness: -While the paper demonstrates the effectiveness of Trans-LoRA on LLaMA and Gemma models, evaluations on a wider range of models and tasks would strengthen its generalizability.
-The reliance on a discriminator introduces an overhead, and the paper does not fully address who will bear this cost in a cloud scenario.
- The paper relies on synthetic data generation, which can introduce biases and limitations. A more detailed discussion on these aspects would strengthen the paper.
- Exploring the potential integration of other synthetic data generation methods like ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback and GOLD (Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation) could enhance the data quality and effectiveness.
- Although the synthetic data generation process and discriminator filtering are designed to minimize risk, the potential for synthetic data to inadvertently reflect proprietary information remains a concern. Further safeguards and monitoring practices should be discussed to ensure data privacy.
questions: questions: See weakness
limitations: limitations: The authors address several limitations, including the need for synthetic data generation and the potential computational overhead. However, the discussion could be expanded to include potential biases in synthetic data, variability of performance across different tasks, and the cost implications of the discriminator overhead in a cloud scenario. Additionally, more detail on the safeguards against data privacy risks would be beneficial.
paper_summary: paper_summary: This paper proproses Trans-LoRA, a method that utilize synthetic data to transfer abilities learned using LoRA across different downstream tasks. Trans-LoRA first uses the source model for synthetic data generation. The generated synthetic data is used to train discriminator LoRA for filtering synthetic data for target model. Using filtered synthetic data together with source LoRA, the target model acquire synthetic data for training. 
Trans-LoRA is validated on various benchmarks and models, showing the effacy of the method.
strength_weakness: strength_weakness: * I understand that the current trend is to apply PEFT on decoder-only LLMs like LLaMa, but I'm a bit worried that the method might not be generic enough. I think the method might be suitable to alternative model architectures, for example
    * Encoder-only LMs like DeBERTa or RoBERTa. Since Trans-LoRA requires generation from the source model, it could be challenging to implement. One could work around this by using another LM with a decoder for generation/discrimination, but this approach seems suboptimal.
    * Similarly, ViT from the vision community. 
* Also, the method seems to rely heavily on the generative ability of the source model/discriminator mdoel. I wonder if it's feasible to use weaker LMs like T5 or GPT2. The results in Table 5 also show that if we use random wikipedia (similarly to use a weaker LM), the performance drops significantly.
* One of the nice property of the method is one only needs 5 real samples to conduct PEFT. But computation-wise, one still needs a relatively large LM to generate samples/discriminate. Is that actually favorable in practice?
* minor: line 198: is it A100 40GB?
questions: questions: See Weaknesses
limitations: limitations: NA
paper_summary: paper_summary: This paper proposes a nearly data-free method for transferring pre-tuned PEFT components (e.g., LoRA) between different models. To address issues of data inaccessibility, the authors propose to generate synthetic data from the target base model. To ensure that this synthetic data is in-distribution, they introduce an additional discriminator, which is trained concurrently with the source PEFT component.
strength_weakness: strength_weakness: 1. The application scope of the proposed method appears limited due to the added constraints on training PEFT components: it necessitates training these components with an additional discriminator. This requirement is uncommon and incurs extra costs.
2. Further discussion on scalability is needed. As the number of PEFT components grows, updating all components seems time-consuming. By contrast, the approach in [1] suggests keeping all PEFT components static while only updating the base model in a specific way.
3. Additional properties of the synthetic data should be considered. While the proposed method focuses on in-distribution generation, the diversity of the synthetic data is also crucial.
4. More baselines of data-free knowledge distillation are needed, such as [2].
5. Given that the proposed method requires a small set of descriptive data, it may be more accurately described as "data-efficient" rather than "data-free."

[1] TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter, arXiv 2023.

[2] Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt, IJCAI 2022.
questions: questions: See weaknesses.
limitations: limitations: The authors discuss several limitations:
(i) increased costs associated with data generation,
(ii) potential misunderstandings of the task.

Comments: comment: Thank you so much for your continued support and for raising the score! 
We will make sure to incorporate all the points raised in the above discussion into the final version of the paper.
We thank the reviewer for their continued support and for raising the score! 
We will make sure to incorporate all the points raised in the above discussion into the final version of the paper.
Thanks for your continued support and for raising the score! 
We will make sure to incorporate all the points raised in the above discussion into the final version of the paper.
We agree that our approach obtains promising results and opens many interesting future work directions for us and others to explore further.
Thank you again for the response. I will raise my score.
Thank you for your continued engagement and for providing further insights. We appreciate the opportunity to clarify and discuss the points you have raised.

>Even if training a discriminator is not costly, users will not train an additional discriminator, as it does not aid their fine-tuning tasks. However, the method proposed by the author makes an additional assumption about user behavior, assuming that users train a LoRA while also training a discriminator, which is not the case in practice. Therefore, I say the application of this method is limited.

To clarify, our approach in fact simplifies the process for user and enables an effortless update of the user's model from the user's perspective.

One desired application setting of our approach is when a user provides their private data to a service provider for training and hosting a PEFT model. The service provider will train both the LoRA for task data and LoRA for discriminator and then delete the private data. There are no additional effort required on the user's side. When the base model needs to be deprecated and updated by a newer model, the provider can simply use the prepared discriminator for transferring the PEFT model without asking the client again for their private data. Thus, our approach simplifies the process for the user, as they now only need to provide their data once and be settled for all future updates (the discriminator only needs to be trained once even for multiple transfers).

A user could also train a PEFT model and discriminator on their side to avoid providing sensitive data, which, given the low overhead of the discriminator, is quite practical.

We acknowledge that our method is intended at all future users of PEFT models and will not apply to PEFT models prior to our method adoption. However, we believe it is a small limitation given the demonstrated success of our approach. We are happy to mention this in the limitation section. Thanks for suggesting!

>Reference [2] is not merely "unfiltered synthetic data." It employs model inversion techniques to invert a topic prompt, which is then fed into a pre-trained language model to produce in-distribution data, similar to your objective.

We used the term "unfiltered synthetic data" to refer that our approach and the approach in [2] are orthogonal and can be applied together. Our approach focuses on filtering of generated data, while the approach in [2] focuses on the generation itself. A straightforward combined approach is to use [2] to perform synthesis, and then apply our discriminator. We will cite [2] in our final paper and address this combination as a promising future direction.

>Model inversion is also a technique used to generate data by optimizing inputs/prompts so as to produce a target output. Could you compare model inversion and prompting generation?

Thank you for mentioning this topic. Model inversions are typically expensive and their operation requires backward passes for the inversion. For example, [2] trains a prompt generator alongside the student model using RL to guide the training of the prompt generator to prompts where student and teacher disagree. So each transfer would require such training and this might be prohibitively expensive. Our discriminator is one time and low cost, and any synthesis used in our method is forward only thanks to discriminator ability to filter useful samples.

Additionally, model inversion is orthogonal to our discriminator filtering approach as well, similar to [2]. One can easily use model inversion for initial generation, then apply our discriminator filtering on the generated samples. We will address this in the discussion of our final paper.
Thank you for the author's rebuttal, which has addressed most of my concerns. I still have the following points to discuss with the author:

1. Even if training a discriminator is not costly, users will not train an additional discriminator, as it does not aid their fine-tuning tasks. However, the method proposed by the author makes an additional assumption about user behavior, assuming that users train a LoRA while also training a discriminator, which is not the case in practice. Therefore, I say the application of this method is limited.
  
2. Reference [2] is not merely "unfiltered synthetic data." It employs model inversion techniques to invert a topic prompt, which is then fed into a pre-trained language model to produce in-distribution data, similar to your objective.
  
3. Model inversion is also a technique used to generate data by optimizing inputs/prompts so as to produce a target output. Could you compare model inversion and prompting generation?
  
[2] Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt, IJCAI 2022.
Dear Reviewer 7oKi,

The authors have provided a rebuttal. Can you please provide your feedback after reading the rebuttal? The deadline is approaching fast.

Thanks,
AC
Thanks for replying to my questions and concerns. I have increased my score.
Thank you again for the response. After reading the new results and the authors' responses to other reviewers, I find myself between a borderline and weak accept decision. Unfortunately, within the limited timeframe for rebuttal, we didn't see results on more variations in base models and tasks, which would have provided a better understanding of the method's behavior. However, the method appears promising, especially with the new results on T5. I encourage the authors to explore this direction further, and I will increase my score to a weak acceptance.
We thank again the reviewer for the support of our work and the constructive feedback! In case we have addressed all the reviewer’s concerns, we would appreciate if the reviewer would possibly consider improving the initial score. Thanks!
Dear Reviewer qZWK,

We sincerely appreciate your valuable feedback and the time you've taken to review our submission. Please let us know if our response has satisfactorily addressed your concerns, and we are more than willing to address any additional comments you may have.

Thank you!
Dear Reviewer 7oKi,

We sincerely appreciate your valuable feedback and the time you've taken to review our submission. Please let us know if our response has satisfactorily addressed your concerns, and we are more than willing to address any additional comments you may have.

Thank you!
Thank you for the feedback!
> Regarding W1, I may not have made my concern clear initially. My main issue is that the method may not be generic enough to work with encoder-only models. While the method works for decoder-only models, and thus naturally for encoder-decoder models, I can accept that it is specifically designed for LMs with decoders.

Thank you for clarifying this! Since our paper is initially targeted at the most popular decoder-only models, we would consider applying on encoder-only models as a promising potential future direction. Based on the positive results on T5, it is likely that when applied on encoder-only models our approach still exhibits good performance. We will mention this point in our final paper.

>I'm a bit confused about the settings used for T5. Could you please elaborate more on the details, such as the source, target, and discriminator?

We are more than happy to elaborate: we used the most basic setting (row 1 and 2) in the main tables (Tables 1-4), where a T5-large model is used to train source LoRA and discriminator LoRA and the T5-XL model is used to distill the target LoRA. The results we reported are source model finetuned accuracy, target model base accuracy, and target model transferred accuracy (ours).
Thank you for the detailed response. I'm glad to see the results on T5 and the explanation regarding computational cost.

* Regarding W1, I may not have made my concern clear initially. My main issue is that the method may not be generic enough to work with encoder-only models. While the method works for decoder-only models, and thus naturally for encoder-decoder models, I can accept that it is specifically designed for LMs with decoders.
* I'm a bit confused about the settings used for T5. Could you please elaborate more on the details, such as the source, target, and discriminator?