PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches

Authors: Rana Shahroz, Pingzhi Li, Sukwon Yun, Zhenyu Wang, Shahriar Nirjon, Chau-Wai Wong, Tianlong Chen
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Large Language Models, NLP, Efficiency, Fine-tuning, Efficient Fine-tuning, Portability

Abstract:
As large language models (LLMs) increasingly shape the AI landscape, fine-tuning pretrained models has become more popular than in the pre-LLM era for achieving optimal performance in domain-specific tasks. However, pretrained LLMs such as ChatGPT are periodically evolved (i.e., model parameters are frequently updated), making it challenging for downstream users with limited resources to keep up with fine-tuning the newest LLMs for their domain application. Even though fine-tuning costs have nowadays been reduced thanks to the innovations of parameter-efficient fine-tuning such as LoRA, not all downstream users have adequate computing for frequent personalization. Moreover, access to fine-tuning datasets, particularly in sensitive domains such as healthcare, could be time-restrictive, making it crucial to retain the knowledge encoded in earlier fine-tuned rounds for future adaptation. In this paper, we present PORTLLM, a training-free framework that (i) creates an initial lightweight model update patch to capture domain-specific knowledge, and (ii) allows a subsequent seamless plugging for the continual personalization of evolved LLM at minimal cost. Our extensive experiments cover seven representative datasets, from easier question-answering tasks {BoolQ, SST2} to harder reasoning tasks {WinoGrande, GSM8K}, and models including {Mistral-7B,Llama2, Llama3.1, and Gemma2}, validating the portability of our designed model patches and showcasing the effectiveness of our proposed framework. For instance, PORTLLM achieves comparable performance to LoRA fine-tuning with reductions of up to 12.2× in GPU memory usage. Finally, we provide theoretical justifications to understand the portability of our model update patches, which offers new insights into the theoretical dimension of LLMs’ personalization.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730593105777

thread_type:
official_review

<thread_object>
creator: reviewer_yysp

<content>
summary:
This paper presents PORTLLM, a novel training-free framework that enables the transfer of personalized knowledge between different versions of evolving LLMs. The key innovation is the creation of lightweight model patches that can be seamlessly applied to updated LLMs without requiring fine-tuning. The framework is extensively evaluated across seven tasks and multiple model architectures, demonstrating comparable or superior performance to LoRA fine-tuning while significantly reducing computational costs. The authors also provide theoretical justification for the portability of their model patches.

strengths:
- A training-free framework for transferring domain-specific knowledge between evolving LLM versions through portable model patches
Theoretical analysis justifying why model patches work effectively, showing certain terms can be neglected to create simplified, training-free patches
- Comprehensive empirical validation across multiple tasks ranging from QA to complex reasoning and different model architectures
- Significant efficiency gains compared to LoRA fine-tuning

weaknesses:
I don't identify major weakness, but I do have a few concern regarding the experimental setting:
- In the experiments, seems the authors only focus on small-scale open-source LLMs. However, from my understanding, it could also be more beneficial for large-scale commercial LLMs. 
- Following the previous setting, a few related works adaptation related works might be considered as related works besides LoRA, LMaaS (Sun et al., 2022), kNN-Adapter (Huang et al., 2023), CombLM (Ormazabal et al., 2023), IPA (Lu et al., 2023), Proxy-Tuning (Liu et al., 2024), and BBox-Adaptor (Sun et al., 2024).
- It is interesting to observe that in some cases it even performs better than fine-tuned models. More explanations on potential reasons could benefit this work.
- Echo my previous questions on experimental settings, I think a better evaluation benchmarks lie in (1) personalization tasks, like LaMP; (2) domain-specific adaptation tasks.
- Any thoughts on rigorous theoretical bounds on performance guarantees?

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643191832

thread_type:
official_review

<thread_object>
creator: reviewer_94qe

<content>
summary:
This paper hypothesizes that the rank of adapters trained with LoRA for a specific downstream task is significantly lower than that of adapters obtained through continued pre-training of large language models using LoRA. The authors experimentally validate this hypothesis and further demonstrate that for pre-trained large language models, using the original adapter fine-tuned for a specific downstream task with LoRA does not lead to significant performance degradation after continued pre-training; in fact, there may even be an improvement in performance on downstream tasks. The performance is comparable to that achieved by training new adapters on the updated large language model. This conclusion suggests that after continued pre-training, the original adapters can still be effectively utilized, significantly reducing training costs.

strengths:
1. The training-free framework proposes the reuse of existing adapters after updating large language models and validates its feasibility, effectively reducing training costs.

2. Extensive experiments across multiple datasets and architectures provide strong evidence for the method’s effectiveness. The paper encompasses a broad range of tasks and models, demonstrating its generalizability and robustness across diverse scenarios.

weaknesses:
1. The background and related work sections contain excessive irrelevant content.

2. In the experimental setup, the fine-tuning for downstream tasks is limited to a rank of 8 without further experimentation.

3. In the experimental setup, the continued pre-training involves only the use of different pre-training datasets for training, without addressing the scenario claimed by the authors: that large language models are updated periodically. In such updates, there may be conflicts between new knowledge that develops over time and existing knowledge, potentially leading to significant changes in the model. The authors did not consider this situation.

4. In the theoretical analysis section, it is better to check the statement that $ \( \theta' = \theta + \Delta\theta \)$ ensures $ \( \text{rank}(\theta') \geq \text{rank}(\theta) \) $.

questions:
Hope the authors can expand the experiments based on the identified weaknesses and revise the writing and theoretical analysis accordingly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705904099

thread_type:
official_review

<thread_object>
creator: reviewer_glne

<content>
summary:
This paper introduces PORTLLM, a framework designed to maintain the effectiveness of domain-specific adaptations in Large Language Models (LLMs) across updates without requiring further fine-tuning. The framework leverages task-specific model patches, derived as the difference between a fine-tuned model and its base (pre-trained) version, to transfer knowledge from one version of an LLM to the next. Experiments demonstrate the method’s effectiveness in enhancing the performance of updated LLMs on various tasks. The approach is claimed to offer computational efficiency and portability across model architectures.

strengths:
1. The framework proposes a training-free alternative to recurrent fine-tuning, which could be useful for applications with limited resources.
2. The portability across model versions and architectures has potential, especially in scenarios where repeated fine-tuning is computationally prohibitive.

weaknesses:
1. The core idea of leveraging parameter differences between fine-tuned and pre-trained models (task vectors) has been explored in prior work, such as Ilharco et al. (2022) on task arithmetic. PORTLLM builds on this concept, but without substantial theoretical or empirical advancements, the framework’s contribution may be seen as incremental rather than groundbreaking. In light of the established work on task vectors, PORTLLM’s innovation could be questioned.

Ilharco, Gabriel, et al. “Editing models with task arithmetic.” arXiv preprint arXiv:2212.04089 (2022).

2. From the results in Tables 2 and 4, it is evident that the PortLLM method is effective in enhancing the performance of the Updated LLM. However, in most cases, it does not surpass the performance of the Fine-tuned LLM. Moreover, since the PortLLM approach requires access to the parameters of the Fine-tuned LLM to calculate $\Delta \theta$ (task vector), applying PortLLM depends on the existence of a Fine-tuned model. In this setup, if PortLLM cannot consistently outperform the Fine-tuned LLM in terms of performance, its practical application value becomes limited.

questions:
1.	In Table 2, why does your method show such a large improvement on the WNLI task compared to the Fine-tuned LLM and Fine-tuned Updated LLM?

2.	Line 344: There is a notation error for the fine-tuned model, it should be $\theta_i$.

3.	Are the two columns of values in Table 6 reversed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706003167

thread_type:
official_review

<thread_object>
creator: reviewer_ts4m

<content>
summary:
This paper proposes a training-free personalization approach for evolving LLMs. Specifically, the proposed PortLLM utilizes LoRA parameters from finetuning older models to approximate updated models. Extensive experiments validate the effectiveness of the proposed approach.

strengths:
The proposed method is based on the observation that residual parameter updates are negligible compared to the naive update, which is both intriguing and empirically proven effective.

weaknesses:
The paper’s core claim is that the residual matrix $\Delta \theta_i’ - \Delta \theta_i$ is negligible in value compared to the naive update $\theta’ + \Delta \theta_i$. Although the authors provide both theoretical and empirical justifications for this, the Lemma (lines 266-269) only establishes that $\text{rank}(\Delta \theta_i’ - \Delta \theta_i) \ll \text{rank}(\theta’ + \Delta \theta_i)$, which does not directly imply that the former is negligible in value relative to the latter. While this statement is supported by empirical results in Table 1, a theoretical proof or an estimated error bound would strengthen the justification, as this could represent a significant proportion of the error for this method.

questions:
- Theoretically, the proposed method $\theta’ + \Delta \theta_i$ introduces error compared to $\theta’ + \Delta \theta_i’$, potentially leading to worse performance. However, in Table 2, a large proportion of $\theta’ + \Delta \theta_i$ results outperform $\theta’ + \Delta \theta_i’$ (the fine-tuned updated LLM). Could the authors elaborate on potential reasons contributing to this improvement?

- What does “# Trainable Parameters” signify in Table 6? Why does the fine-tuning model have zero trainable parameters, while the proposed training-free approach has a large number of trainable parameters?

- In line with the discussion in Weaknesses, could the authors investigate the impact of the approximation error introduced by the proposed method? For instance, providing metrics similar to those in Table 1 for all datasets, along with the corresponding performance gains, could offer a clearer understanding of how this factor contributes to overall performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732240403053

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Manuscript Revisions

comment:
We extend our sincere gratitude to all the reviewers for their thoughtful and constructive feedback. We are particularly encouraged that the reviewers acknowledge several key strengths of our work, including the novelty of our training-free framework (Reviewers YYSP, 94Qe, GLNE), comprehensive empirical validation across multiple tasks and architectures (Reviewers YYSP, 94Qe), and significant efficiency gains compared to existing approaches (Reviewer YYSP, TS4SM). 

In addition to addressing their comments point-by-point, we have made the following updates in our revised manuscript:
1. Expanded Related Work (Reviewer YYSP): We have enriched our discussion of parameter-efficient fine-tuning methods by incorporating the following relevant works:
   - LMaaS (Sun et al., 2022)
   - kNN-Adapter (Huang et al., 2023)
   - CombLM (Ormazabal et al., 2023)
   - IPA (Lu et al., 2023)
   - Proxy-Tuning (Liu et al., 2024)
   - BBox-Adaptor (Sun et al., 2024)
2. Condensed Introduction (Reviewer 94Qe): Removed tangential discussions about general LLM capabilities from Introduction to emphasize the focus on fine-tuning and evolution, the premise of the paper. 
3. Tightened the conversation on general capabilities of LLMs in Related Works section. (Reviewer 94Qe)
4. Fixed Notation Errors (Reviewer GLNE and TS4M):
   - Line 344: Corrected the notation for the fine-tuned model to $\theta_i$. (Now Line 348)
   - Table 6: Fixed the column headers to correctly reflect the savings.
5. A brand new Appendix was added: 
   - Section A: Continual Model Updates Experiment was added. (Reviewer 94Qe)
   - Section B: LoRA Analysis was added.(Reviewer 94Qe)
   - Section C: Detailed proof of Lemmas and theoretical bound on Error for PortLLM. (Reviewer TS4M, YYSP, 94Qe)

All changes are highlighted in cyan in the revised manuscript for easy reference. We believe that these updates have improved the clarity of our manuscript while maintaining its core contributions to training-free LLM personalization.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents PortLLM, a training-free framework that allows for the personalization of evolving LLMs without the need for repeated fine-tuning. The key innovations are the creation of lightweight model patches that can be applied to updated LLMs to transfer domain-specific knowledge, and the theoretical justification for the effectiveness of these patches across model updates. 

*Key strengths:*
- Novelty of the training-free framework for maintaining personalized performance as LLMs evolve
- Comprehensive empirical validation across multiple tasks, from simple QA to complex reasoning, and diverse model architectures
- Significant efficiency gains compared to fine-tuning approaches like LoRA
- Theoretical analysis justifying the effectiveness of the model patches

*main weaknesses and missing aspects:*
- The relationship between the proposed method and prior work on task vectors/arithmetic could be clarified further
- The performance of the method compared to fine-tuned models is not consistently superior, and the practical value proposition needs to be emphasized
- Evaluation on larger, commercial-scale LLMs would strengthen the impact
- Deeper analysis of the approximation error introduced by the method and its impact on performance would be valuable

The reviewers are generally positive about the paper and believe it makes a meaningful contribution, with the average rating being marginally above the acceptance threshold. The key reasons for this decision are:
- The novelty and practical significance of the training-free personalization framework for evolving LLMs
- The strong empirical results demonstrating the efficacy of the approach
- The theoretical underpinnings provided by the authors
- The potential for the method to have significant impact, especially for scenarios with limited resources for repeated fine-tuning

Overall, the paper appears to be a solid contribution to the field of efficient fine-tuning and personalization of large language models.