Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
Authors: Zeyu Han
Chao Gao
Jinyang Liu
Jeff Zhang
Sai Qian Zhang
Venue: Accepted by TMLR ()

Abstract: Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities.
Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large model to adapt it to a specific task or domain while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large-scale language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design.
In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate PEFT computation costs. In addition to providing an extensive survey from an algorithmic standpoint, we also examine various real-world system designs to investigate the implementation costs associated with different PEFT approaches. This survey serves as a valuable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed insights into recent advancements and practical applications.

Published in Transactions on Machine Learning Research (10/2024)
Parameter-Efficient Fine- Tuning for Large Models: A Comprehensive Survey
Zeyu Han
ben.zeyu.horrthhastemeelll
Northeastern University
Chao Gao
cguo0970uur.com edu
University of California, Riverside
Jinyang Liu Northeastern University
liu ww..iinyon@hhorthhsseeee.lle
Jeff (Jun) Zhang Arizona State University
jeffzhang@asu.cc edu
Sai Qian Zhang* New York University
soi. zhang Gnyu. edu
Reviewed on OpenReview: https: // openreview. net/ forum? id= LICC88662
Abstract
Large models represent groundbreaking advancement in multiple application fields, en- abling remarkable achievements across various tasks. However, their unprocedented scale comes with significant. computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands posc considerable challenges when customizing them for particular downstream tasks, particularly over the hardware plat.forms constrained by computational capabilities.
Parameter Efficient Fine- Thne-Tunngg (PEFT) provides a practical solution by efficiently adjust- ing the large models over the various downstream tasks. In particular, PEFT refers to pre-trained large model to adapt to a spe-
the process of adjusting the parameters cific task or domain while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large-scale language models with high parameter counts, finc-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design.
In this survey, we present comprehensive studies of various PEFT algorithms,, examining their performance and computational overhead. Moreover, provide an overview of ap- plications developed using different PEFT algorithms and discuss common techniques em- ployed to mitigate PEFT computation costs. In addition to providing an extensive survey from an algorithmic standpoint, we also examine various real-world system designs to vestigate the implementation costs associated with different PEFT approaches. This survey serves valuable resource for researchers aiming to understand both the PEFT algo- rithm and its system implementation, offering detailed insights into recent advancements and practical applications.
* Corresponding author

---

Published in Transactions on Machine Learning Research (10/2024)
Section 2
Section 3
Section 4
Section 5 Apply PEFT for other Applications
Section 6
Background
PEFT Taxonomy
C
Efficient PEFT
System Design Challenge
Design
2.1
Computational flow for LLM
3.1
Additive PEFT
4.1
KV-cache Management
5.1
PEFT for LLMs
6.1
System Design for PEFT
PEFT Efficiency
3.2
Selective PEFT
4.2
PEFT Pruning
5.2
PEFT for ViTs
6.2
Centralizee PEFT Serving System
2.2
PEFT overview
3.3
Reparameterized
PEFT
4.3
PEFT Quantization
5.3
PEFT for VLAs
6.3
Distributed PEFT Training System
2.3
Downstream tasks
3.4
Hybrid PEFT
4.4
Memory-efficient PEFT
5.4
PEFT for Diffusion Models
6.4
Parallel PEFT Training System
Figure 1: A content overview covered in the survey.
1
Introduction
Large Models (LMs) have recently captured considerable public interest. Their ability to understand context and nuances enables Ihem to proficiently handle diverse lasks across multiple domains, including nalural language processing (NLP), computer vision (CV), etc. In the field of NLP, Large Language Models (LLMs)
have achieved significant advancements across various tasks including text generation (Brown al., 2020: Zhuang et al.
2023), translation
(Zhu et al., 2023c Hadi et al.) 2023), personalized chat-bots (Xu et al.,
2023a Ti al.
2023a Wu et al.
2023c), and summarization (Zhang et al. 2023a demonstrating remarkable
proficiency.
2020 have suggested that LLMs exhibit high levels of generalization, enabling them to apply their acquired knowledge to new tasks not included in their original training. This capability is commonly known as zero-shot leurning. Nevertheless, fine-tuning remains essential to further enhance LLMs for optimal performance on new user datasets and tasks.
Earlier studies (rrwn et al.
Due to its scale, widely adopted strategy for fine-tuning LLMs involves adjusting a limited number of LLM paramcters while keeping the remainder unchanged. This technique, termed -ofrecttr-Effcieet--FF Twning (PEFT), involves selectively adjusting small proportion of their parameters while keeping the rest unaltered. Furthermore, the application of PEFT extends beyond the realm of NLP and quickly attracts interest in the CV community for handling fine-tuning vision models with large parameters, such as Vision Transformers (ViT) and diffusion models, as well as disciplinary models such as vision-language models.
In this survey, we systematically review and categorize recent advancements in PEFT algorithms as well the system implementation costs associated with various PEFT algorithms across diverse scenarios. Figure [ presents the overview content for this survey. In section and PEFT, including computational flow for LLM, basic knowledge of PEFT, commonly used datasets and tasks, and evaluation benchmarks. We categorize all types of PEFT algorithmss in Section Baccording to their computational flow. In Section 3.1 we detail additive algorithms that either introduce new weight parameters or modify activations. Algorithms that only require fine-tuning of existing parameters are categorized as selective approaches, which are introduced in Scction 3.2 Soction 3.3, we explore reparameterized PEFT, which constructs (low- dimensional) reparameteriiztiion of original model parameters for training while transforming the weights back to maintain the inference speed. Additionally, there exist algorithms that combine the above techniques, and we have classified these as hybrid approaches, claborating on them in Section 3.4 We also investigate strategies for further reducing the computational complexity of different PEFT algorithms, including KV-cache management, pruning, quantization, and memory optimization, in Section !
present some fundamental concepts for LLM
In Scction we expand the scope of this survey beyond the computational perspective involve various potential application scenarios. Specifically, we explore innovations that applying PEFT techniques to dif-
2

---

Published in Transactions on Machine Learning Research (10/2024)
ferent model architecture, including LLMs (Section 5.1 Vision Transformer (Section 5.2 Vision-Language alignment models (Section 5.3). and Diffusion models (Section 5.4) for varied downstreem tasks, under- scoring PEFT's versatility and applicability in a range of scenarios. After that, in Section G we explore the system design challonge for PEFT mothods. The discussion includes three advanced system solutions for practical PEFT deployment: PEFT query serving (Section 6.2 distributed tuning (Section 6.3). and
concurrent PEFT tuning (Section 6.4). Finally, in Section 7 we summarize our survey and propose several potential future directions from both algorithmic and systemic perspectives, aiming offer valuable insights for further research and development in the field.
2 Background
In this section, we first discussed the computation flow of LLM, including its fundamental components, computational complexity, and the flow of computations it involves as a case study. We then provide a brief overview of different PEFT algorithms in section 2.2
2.1 Computation flow for LLaMA
In order to gain a deeper understanding of LLM and other Transformer-based models, we employ LLaMA- 7B, a cutting-edge open-source LLM model, to scrutinize the architecture of LLM as well as Transformer. As shown in Figure 2 (a), LLaMA consists of three major components: an embedding block, a stack of decoder blocks, and a head block which consists of linear and softmax layers. The embedding layer's primary role is to transform unstructured textual information, into chunks of discrete numerical vectors (tokens) facilitate subsequent processing. The embedded Lokens then delivered to the decoder layers for further processing. Each LLaMA decoder is composed of two fundamental components: Multi-head Self-Attention (MSA) and Feedforward Network (FFN). In the MSA module, each of the tokens will be clustered by attention map obtained by a dot production between two linear mappings of the input tokens. Then the grouped tokens will be further processed by a Feedforward Neural network. Additionally, Root Mean Square Layer Normalization (RMSNorm) [haang & Sennrich 2019 adopted in LLaMA as a replacement for Layer Normalization to ensure efficient training.
LLM distinguishes itself from other deep neural network (DNN) models such as convolutional neural net- works (CNN) in two significant ways. Firstly, LLM exhibits an inherent autoregressive nature, necessitating multiple iterations to complete the generation task. Moreover, LLM incorporates an attention mechanism, component with computational complexity that scales quadratically with the length of the inputs. On the other hand, the inherent computation characteristic of LLM lies in the attention blocks inside cach decoder layer. Figure 2 (c) depicts the high-level overview of the computation flow in the attention block.
During the inference process, each decoder takes a dhres-dimensional tensor æ E Bbxttd as the input tokens, The input tokens are first multiplied with three weight matrices Wo, WK. and Wv, producing the output referred to query(Q), key(K) and vVlue(V). Given the MSA module's inability to recognize positional data and the inherent auto-regressive nature of LLMs, the query and key will undergo process using Rotary Positional Embedding (Su et al. 2021a) (RoPE, denoted as R(.) in Eq [ to encode the position information. Subsequently, the key and value will be combined with prior tokens.
After the positional embedding, the intermediate activation will then undergo a series of multiplication, softmax, and residual addition to generate MSA output as described in Eq To be noted here, dę in the equation refers to the number of feature dimensions in the multi-head atlention mechanism.
Q.K.V V = x), R(W,z), W.
(1)
SA(x) = Softmax QKT )v vdhead MSA(z) (z): S..
(2)
(3)
3

---

Published in Transactions on Machine Learning Research (10/2024)
Adapter
FC
Decoder
Linear & Softmax
ReiU
FC
FFN
ML
is
awesome
<EOS>
Decoder
FFN
SA
LLaMA
LLaMA
LLaMA
LLaMA
LoRA
Decoder
W
Wassn
SA
K
V
Decoder
Embedding
<BOS>
ML
awesome
Prompt
Input tokens
(a)
(b)
(c)
Figure 2: (a) LLaMA architecture. (b) LLaMA auto-regrossive pattern. (c) Three common PEFT operations. All the learnable components are highlighted in red, while the frozen components LoRA is applied on all the Query, Key, and Value blocks. The adapter targets the FFN module. Soft- Soft-Prompt focused on tuning the input activation of each decoder. We only show one decoder for illustration simplicity.
highlighted in grey.
The SA output will then be forwarded to the FFN blocks for further processing. The FFN block will have another three matrices Waps W Wowwn: and W gate and the computation can be illustrated by:
FFN) = (., g)te O (W down + x,
(4)
where x denotes the input of the FFN layer, and SiLU is the nonlinear function used in LLaMA. In the original Transformer, the FFN block can be demonstrated by:
PrNTransfooes (x) = Wup (ReL + x.
(5)
The output of the last decoder layer will be sent to a linear layer, which then generates a probability distributoon spanning the complete vocabulary to predict the next. token in the sequence. The produced token will then be concatenated with the previous tokens and used the input for the next round of processing. This generating process repeats in an auto-regressive manner until full sequence of tokens, referred to completion, is produced (Figure 2 (b))- For training, the computation flow is similar to that for inference, except that the generated sentences are directly compared to the ground truth output and generate the training loss. Gradients will then be computed across the LLM weights to minimize this training loss.
To analyze the computation cost and memory overhead in LLM, we also set a series of parameters used later section 3. Table shows the parameter size and computation dimension in the LLaMA-7B model as starting example.
LLM models generate tokens (words) one for each round, depicted in Fig based on the previous prompt (input) and previously generated sequence. This process will be repeated until the model outputs hits and termination token. To accelerate the inference process in LLM models, people take the strategy of storing the previous Keys and Values in the Key-Vate Value cache (K) (KCccche),, so they don't need to recalculate them for each new token. Mathematically, we can represent the total decoders' KV-cache memory cost in equation In the equation, and are the context length and batch size and L refers the number of layers. The dhead is the head dimension and Meed is the number of heads.
Size I 2 x b dhead Rhead
(6)
4

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Configuration parameters and computation operation for LLaMA-7B architecture
Operation
Weights Symbol
Weights Dimension
Input Tensor Dimension
Complexity
Eq.
Wq. WK, Wy
d dxkx
bxlxd
O())
Eq.
2
3
O(1))
Eq.
W
d x
bxlxd
O
Eq.
Wup: down, gute
d bxlxd d OR ORIxbxe 4d
O
2.2 Overview on Parameter Efficient Fine Tuning
Fine-tuning remains issential to enhance LLM performance on unseen user datasets and tasks. With the size of the model growing (e.g. 1.5B in GPT-2 to 175B in GPT-3), standard full tuninuning paradigm requires thousands of GPUs work in parallel, which is highly inefficient and unsustainable. A type of algorithm has been raised namely Parameler-efficinnt fine-tuning (PEFT) which aims to tune minimal parameters to achieve better performance over full tuning downstream tasks.
In parallel developments, large-scale pre-trained models in vision and multimodal domains have also demon- strated their effective representational learning capabilities, enabling adaptation from large datasets to smaller ones or across various data modalities through fine-tuning. Consequently, this capability has made PEFT increasingly altractive to the wider research community.
We categorized the PEFT algorithms into additive, selective, reparameterized, and hybrid fine-tuning based their operations. As Figure 3 depicts, three major additive fine-tuning algorithms normally used: (1) Adapter; (2) Soft Prompl; (3) Others. They differ in terms of the additional tunable modules or parameters. Selective fine-tuning, on the other hand, doesn't require any additional parameters, it selects small subset of parameters from the backbone model and only makes them tunable while keeping the majority of parameters untouched during fine-tuning on downstream tasks. We categorized selective fine- tuning based on the grouping of chosen parameters: (1) Unstructural Masking; and (2) Structural Masking. Reparametrization represents transforming model parameters between two equivalent forms. Specifically, reparametrized fine-tuning introduces additional low-rank trainable parameters during training, which are then integrated with the original model for inference. This approach is categorized into two main strategies: (1) Low-rank Decompasition, and (2) LoRA Derivatives. Hybrid fine-tuning explores the design spaces of different PEFT methods and combines their advantages.
2.3 Downstream Tasks for LLM Evaluation
Two types of tasks have been widely used for LLM evaluation, the first type the General Language Under- standing Evaluation (GLUE) (Wang et al. 2018) benchmark, which integrates nine sentence or sentence-pail language understanding tasks (CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, and WNLI), cho- sen for their diversity in dataset sizes, text genres, and difficulty levels, and is based on established existing datasets. It also includes diagnostic dataset designed to evaluate and analyze model performance across diverse linguistic phenomena in natural language. Additionally, it features public leaderboard to track performance on the benchmark and a dashboard to visualize model performance on the diagnostic set.
The other type of dataset that has been used in recent LLM papers common sense reasoning which integrated into our study caters variety of research facets: (1) OpenBookQ4 (Mihaylov et al. 2018 is curated to foster research in advanced question-ansswn answering, delving into profound understanding of both the subject matter and the language in which it is articulated. (2) PIQA (Bisk et al. 2020 primarily emphasizes everyday scenarios, demonstrating predilection for unconventional solutions.
(3) Social IQA (Sap et al. benchmark lailored for gauging social commonsense intelligence. (4) HellaSwag (Zellers et al. 2019) serves as a dataset, the essence of which to ascertain the capability of machines in aptly concluding sentences. (5) BoolQ (Clark) 2019) a dataset dodicated to question-answering
2019) emerges as a novel question-answer
particularly for binary responses (yes/ma no queries). (6) WinoGrande (Sakaguchi et al. (2021) is introduced fresh compilation, encompassing substantial 44,000 problems. (7) ARC-easy (Clark et al. 2018 presents
5

---

Published in Transactions on Machine Learning Research (10/2024)
itself novel dataset constituting genuine grade-school level multiple-choice science questions, designed to invigorate research in intricate question-answeringg (8) ARC-challenges (Clark et al. 2018). distinctively, encompasses solely those questions that were inaccurately addressed by both a retrieval based algorithm and a word co-occurrence algorithm.
Image recognition serves as key benchmark and application for vision models, illustrated by tasks like fine- grained visual categorization (FGVC) and the visual task adaptation benchmark (VTAB). Beyond image classification, video action recognition is another key application area, involving datasets like Kinetics- 400 (Kay et al. 2017h, SSv2 (Goyal et al., 2017), and HMDB51 (Kuehne et al. 2011). 1.. Additionally, PEFT has been utilized for dense prediction tasks, using datasots like MSCOCO (Lin et al. 2014). ADE20K (Zhou
et al. 2017 and PASCAL VOC (Everingham et al. 2010).
2.4 Evaluation Benchmarks for PEFT
A comprehensive benchmark is essential for readers to evaluate performance differences among various PEFT methods under unified standard. We next discuss several commonly used benchmarks.
From the algorithmic perspective, Ding et al. 2023b) benchmarks the performance of several PEFT al- gorithms across more than 100 NLP tasks and conducts systematic experiments based on criteria such as performance, convergence, efficiency, combinability, scalability, and transferability. Similarly, (Xu et al. 2023b) and (Pu et al. 2023) have also established targeted benchmarks evaluate different PEFT algo- rithms.
From the system perspective, three commonly used benchmarks are outlined below to evaluate system per- formance. The first benchmark is the ShareGPTT dataset (OpenAl 2023a which includes real-world inter- actions with OpenAl's ChatGPT. encompasses a broad spectrum of conversational queries and responses that are representative of typical user interactions with large language models (LLMs). This dataset vital for evaluating the system's 's ability to manage diverse and realistic conversational requirements, focusing on the accuracy of responses and efficiency in handling requests.
The second benchmark involves the Microsoft Azure Function Trace from the years 2019 and 2021 (Microsoft, 2023), containing logs from serverless computing activities via Azure Functions. While these logs are from general serverless computing context rather than LLM-specific applications, they offer insights into the computational demands driven by events. These traces simulate the arrival patterns and workload intensities that LLM systems might face, including irregular and peak demands, thus acting as practical proxies for LLM inference tasks.
The third benchmark is based on the Gamma process (Moreno al. 2014), prevalent approach in simu- lations to model the timing of incoming requests in queueing and service systems. This method facilitates the creation of workloads with varied arrival rates and patterns, producing synthetic, yet realistic request scenarios that system could encounter during actual operations. Such synthetic workloads are crucial for testing system performance under controlled conditions that resemble real-world user activity.
3 PEFT Taxonomy
The PEFT strategies be broadly classified into four categories: additive PEFT (Section 3.1 which modifies the model architecture by injecting new trainable modules or parameters; selective PEF" T (Sec-
tion 3.2 which makes a subset of parameters trainable during fine-tuning; reparameterized PEFT (Sec-
tion 3.3 which constructs a (lowddienssioal) reparameterization of the original model parameters for training. then equivalently transforms it back for inference; and hybrid PEFT (Section 3.4), which com- bines advantages from different PEFT methods build unified PEFT model. An overview of different types of PEFT algorithms is depicted in Figure

---

Published in Transactions on Machine Learning Research (10/2024)
Adapter Design
Sessal Adaziet
Houloby et al. 2019
Pacalki Adopter
[Iee a..
2021
CLNT 22u ot ol. 2021
Adapter-based Fine-tunile tuning
CuDA
Lan x
2003
Multi-rack Adhapaation
Piaii AdaatteeSSop
Pletfer
Corompseto
Hypurmmnar
Preficctuiir [1 Laxur 2721
La
2723
2021a
Additive Fines
Suft Prompt Design
Xaranipr
APT ZZhgg e
IDPG
(Wirot
Lao
202
T
Lau
201aan
Loster
Tan. 2725
APrompp
20220
Prompt-bascd
Ttalning Speeduu
SPol of
2021
TPT
(Suet al
2221b
InfoPromntt
Wi es ol
PTP
2025c
IPT
12th
SMoP
Clive
2023
SET & Lipant 2023
[LA us 2022s
U-Diff
Lino
FishMask
Masking
(Dus
Xu
Selective
Einestuning
Structural Musking
S-Diff praning,
Goo 2020
S-BisPa SPT
Lawston sl.
2025
FAR (Vusttic et al.
2122
Bitfi Zaken
2721
Xatin Tuning
Low-sonk Decomupssiton
Lutrinsi SAID Ashuianne Inlaan et al. 2000
KronA DoltA
Edlalati Bil 2uzz konaspatton
LeBA Ra
[e de ii.
20120 HrW: (Lino et ab 3tadi
2021
Conaunterr Karimi Maimsbali at ald
Valte Roppisaeo of al.
2022
2025
Tal of 2r2t
PEPT
Reparansterized Pine Tineitaning
Dynamic
Zhang
Ding all
Capa
LoltA Derivstive
Lalt Improvemend
Laplace= LottA YYans
LoRA Dromous
In
2024
PeriodieLoRAA
Meng
2024
LoltA Hayon
Wi
Multiple LoltA
LoltAtinh Hunnn et Mon
20233
MORLoRR
Ffuret al 2022h
MoLORA Zaciour es al.
2023
Peng 2r Mt. 2024
Wia al.
knstalex [a at 2021
Hybrid
et AUTOPRPT Aile ot
et 50236)
MAM Aduptes
Zhang et 20226
Finestuning
Amapors (m
2023
SPr (20
2023
Figure 3: Taxonomy of Garameter-ffficeen Fine- Tane-TTning Methods for Large Models.
(a) Additive PEFT (b) Selective PEFT (c) Reparameterization PEFT
Output T Combine
Output
Output
Merge
Input
Input
Input
Input (train)
Frozen
Learnalle
Figure 4: Different types of PEFT algorithms.
3.1 Additive PEFT
Standard full fine-tuning entails substantial computational expenses and could also potentially harm the model's generalization ability. To mitigate this problem, widely employed approach is to maintain the pre-trained backbone unchanged and introduce only a minimal number of trainable parameters that are strategically positioned within the model architecture. While fine-tuning for a specific downstream task, only the weights of these additional modules or parameters are updated, which results in a substantial reduction in storage, memory, and computational resource requirements. Due to their characteristic of adding parameters, these techniques can be termed Additive Tuning, as shown in Figure discuss several popular Additive PEFT algorithms.
(a). Next,
3.1.1 Adapters
Adapter approaches involve the insertion small adapter layers within Transformer blocks. Typically, an adapter layer consists of down-projcction matrix W down Rexd followed by a non-lincar activation function a(-), and an up-projection matrix Wup E Rdxr, In this context, represents the dimension of the hidden layer, and serves the bottleneck dimension, which is hyperparameter used in configuring the adapters.
7

---

Published in Transactions on Machine Learning Research (10/2024)
Adapter
Adapter
Adapter
Transformer
Transformer
Transformer
Module
Module
RELL
Module
tokens
lap-i tolions
(a) Serial Adapter
(b) Parallel Adapter
(c) CoDA
(d) Adapter Layer
Figure 5: Illustration of three representative adapter-basod fine-tuning algorithms. Blue represents frozen, while yellow represents trainable.
Denote hin as the input to the adapter, the computation within the adapter module (with residual) can be summarized as follows:
Adapter(a) = Wap (Waown?) + x.
(7)
The concept of adapters in the field of NLP was initially introduced by Scrial Adapter (Houlsby et al. 2019) as shown in Figure 5 (a). In their approach, each Transformer block is enhanced by adding two adapter modules, with one positioned after the self-attention layer and the other after the FFN layer, respectively. Subsoquent rescarch has aimed to address the additional computational cost associated with adapter layers. A modified framework AdapterFusion (Pfeiller al. 2020 was proposed, where adapter layers are inserted only after the 'Add & Norm' step following the FFN layer to enhance the computational efficiency. The adapters mentioned above follow sequential design, placing adapter layers bottlenecks within the Transformer blocks. This approach may potentially reduce the model's parallelism and require a trade-off ween inference efficiency and accuracy. In contrast, He et al. (2021) introduced a parallel adapter (PA) approach as depicted in Figure 5 (b), which reorganizes the traditionally sequential adapter layers into a parallel nedernetwork that runs alongside each Transformer sublayer. Similarly, CIAT (Zhu et al. 2021). CoDA (Lei al. 2023) and KronA (Edalati et al., 2022) also adopts a parallel adapter design. Except for the parallel design, CoDA employs sparse activation mechanism to improve the inference efficiency as shown in Figure 5 (c). Specifically, CoDA uses soft top-k selection process that identifies k important tokens in each layer, which will be processed by both the frozen pre-trained Transformer layer and the adapter branch to maintain model accuracy. In contrast, those unimportant tokens are only processed by the adapter branch while skipping the heavy pre-trained layer, therefore optimizing for inference efficiency without compromising overall performance.
To enhance the performance and generalization of adapters, various studies have implemented multi- task learning strategics, such AdapterFusion (Pfeiffer et al., 2020). AdaMix (Wang et al. 2022a) PHA (Zaao et al.] 2023b), AdapterSoup (Chronopollou et. al. 2023 MerA (He et al.) 2023b). and Hyperformer MMahabadi et al. 2021). AdapterFusion keeps all pre-trained adapters in the model and employs a fusion module to merge the multi-task information. Unlike AdapterFusion, MerA merges pre- trained adapters into a single one through optimal transport based on weights and activations. This approach avoids intrrouggnng any additional trainable parameters, thereby enhancing computational efficiency. Hyper- former stores the multi-task information in a shared hypernetwork, which generates task and layer-specific adapter parameters conditioned on task and layer ID embeddings. Given a new task, only an additional task embedding needs to be learned, therefore reducing the number of trained parameters.
3.1.2 Soft Prompt
Alternatively, prompt tuning presents an additional approach for refining the model to achieve improved performance through fine-tuning. Instead of optimizing discrete token representations through in-context learning, there is a prevailing belief that the continuous embedding space of soft prompts inherently contains more information (Petrov et al. 2023). Drawing inspiration from this concopt, rescarchers directly prepend adjustable vectors, referred to as soft prompts, the start of the input sequence. This can be represented follows:
8

---

Published in Transactions on Machine Learning Research (10/2024)
x = [
(2) No x1
xx1
(8)
where x is the sequence of imput tokens for layer I, including soft prompt tokens 5.6 followed by the original input tokens x Ns is the number of soft prompt tokens, and Nx is the number of original input tokens.
Prefix-tuning (Li & Liang, 2021) introduces learnable vectors that are prepended to keys i and values o across all Transformer layers. To ensure stability during the oplimization process, Prefix-tuning adopts a reparameterization strategy, which utilizes an MLP layer to generate these prefix vectors rather than optimizing them diroctly. After fine-tuning, only the prefix vectors are saved for inference. This technique has been adapted and improved in several studies (Li et al. 2023b Liu et al.
2021a Zhang et al. 2023h
For instance, p-tuning v2 (Liu et al. 2021a]) removes reparameterization and expands its usage to broader model scales and NLP tasks. APT (Adaptive Prefix Tuning) (Zhang et al. 2023h enhances Prefix-tuning
by introducing an adaptive gate mechanism to control the prefix importance in each layer. Concurrent work p-tuning (Liu et al. 2021b) and prompt-tuning (Lester et al. 2021] apply learnable vectors only at the initial word embedding layer rather than all layers to enhance training and inference efficiency. It's important to highlight that prompt-tuning demonstrates its effectiveness primarily in the context of large models, specifically those with over 11 billion parameters (Lester et al., 2021). Complementing this, Xprompt (Ma et al.] (222) climinates the negative prompt tokens through a hierarchically structured pruning, which closes the performance gap at smaller model scales. WWang et al. (2023c) provides some theoretical analysis towards prompt luning, demonstrating its universality and limitations in limited-depth Transformers. IDPG Llnsancce-Deendeen Prompt Generation) (Wu et al. 2022) improves prompt tuning by generating prompts basod on each input sentence with a lightweight prompt generator. In a relatod approach, LPT (Lale Prompl Tuning) (Liu et al.) 2022b) also leverages a prompl generator to obtain instance-aware prompt. Unlike previous work, LPT adds these prompts only after intermediate layer, rather than at the initial all layers. This strategic placement eliminates the gradient calculation below the intermediate layer, thereby significantly accelerating the training speed. Simultancously, LPT can improve the overall performance due to the shorter backpropagation path preserves more task-related information. Inspired by LPT, SPT (Selective Prompt Tuning) (Zhu & Tan 2023) delves deeper into the importance of prompt inserting strategics. It introduces a Icarnable probabilistic gate in each layer to determine whether to use the prompt propagated from the previous layer or inject a newly generated prompt. APrompt (Wang et al.) 2023a) employs another prompt inserting strategy. In addition to input prompts inserted at the beginning of the input sequence for each Transformer layer, A APrompt also prepends additional learnable prompts to the respective query, key, and value matrices in the self-attentio blocks to learn new attention patterns. Besides, APrompt incorporates the learning of a task-specific head.
The concept of soft prompts has been employed for various downstrrem tasks CChoi & Lee 2022), although their training can be prone to instability and slow convergence. To address this, SPoT (Vul et al., 2021) uses a source prompt learned from one or multiple tasks to initialize prompts for new tasks. Similarly, the transfer of soft prompts from one task to initialize another is proposed in TPT (transferable prompt tuning) (Su et al. 2021b,, which demonstrates that a better prompt initialization results in large training convergence speedup. InfoPrompt (Wu et al. 2023b) develops two mutual information functions, i.e., head loss and representation loss, to find better prompt initialization and learn sufficient task-relevant information, thereby also expediting convergence. PTP (Chen et al. 2023c) delves into the root causes of training instability. It identifies the steep nature of the loss landscape in conventional prompt tuning, where minor variations in input data can lead to significant loss fluctuations. To mitigate this, PTP introduces perturbation- based regularizers to smooth the loss landscape and consequently stabilize the training process. DePT (Shi & Lipani 2023) decomposes the soft prompt into shorter soft prompt with a pair of low-rank matrices, which are optimized with two distinct lcarning rates. This strategy not only improves performance but also enhances training and inference efficiency. SMoP (Sparse Mixture-of- Prompts) (Choi et al.) 2023) reduce the training and inference cost by utilizing short soft prompts. During training. multiple short soft prompts are trained, each tailored to specific subsets of the datasot. During inference, SMoP integrates a gating mechanism that routes each input instance to an appropriate short prompt. This technique not only increases efficiency in both training and inference stages but also retains
2023
Wu Ship
loss
9

---

Published in Transactions on Machine Learning Research (10/2024)
W
up
scale
shift
softmax
Operation
Dpermaiie
C
K
W
(a) (IA)²
(b) SSF
Figure 6: Illustration of (TA)² and SSF. Blue represents frozen, while yellow represents trainable.
performance comparable to those achieved with longer soft prompts. To further cut down the number of soft prompt parameters, IPT (Intrinsic Prompt Tuning) (Qin et al. 2021) identifies an intrinsic task subspace by training an auto-encoder on multiple tasks. Tuning on new tasks then requires adjusting only a few parameters within this subspace, significantly reducing the number of training parameters.
3.1.3 Other Additive Methods
Apart from the methods mentioned above, there appear other approaches that strategically incorporate additional parameters during the fine-tuning process. For example, (IA)² (Liu et al., 2022a) introduces three learnable rescaling vectors: la E Rdx, by Rd, and ls E gdrr, to rescale the key, value, and FFN activations, respectively, as depicted Figure G (a). The operations within the self-attention block can be described follows:
Q(*KK) ((..V).
SA()) Softmar( v dhead
(9)
In FFN, the rescaling can be denoted as:
FF Tranafooeer (x) = Wapp1 O (o (WWdown down))))
(10)
where @ is Hadamard product. Furthermore, the scale vectors 4s and I, can be seamlessly integrated into the weight matrices Aq and Aw. This integration effectively eliminates the extra computational costs during inference. A similar technique SSF (Lian et al.) 2002)] also performs lincar transformation model activations, illustrated Figure (b). Specifically, after each operation (i.e., MSA, FFN, and layer normalization) in the pre-trained model, an SSF-ADA layer is injected, which performs scaling and shifting to the features generated from the operation. During fine-tur fine-tuning, only those SSF-ADA layers can be updated, while during inference, similar to (IA)², these SSF-ADA layers can be merged into model weights, so no additional inference overhead would be incurred. IPA (Inference-Timee Time Policy Adapters) (Lu et al. 20233 offers a novel approach to align LLMs, such as GPT-4, with uscr-specific requirements without modifying the base model's parameters. This is particularly significant when dealing with models whose parameters are extremely large and often not directly accessible. IPA achieves this by combining (through multiplication and normalization) the output distribution of base LLM (base policy) with that of smaller-sized model (adapter policy) during the decoding phase. During training, the policy adapter's parameters are fine-tuned using rein reinmoreemnnt learning, while the base policy's 's parameters remain fixed. During inference, IPA decodes with the combined distribution of the base model and the trained policy adapter, tailoring it to fulfill specific uscr-defined critcria.
the
3.2 Selective PEFT
Rather than additive PEFT, which increases the model complexity by adding more parameters, selective PEFT fine-tunes a subset of the existing parameters to enhance model performance over downstream tasks, as depicted in Figure 4 (b).
Specifically, given a model with parameters 0 {#2, 02..... 0n) where each 0, denotes an individual model parameter and represents the total count of these parameters, the process of selective PEFT is represented
10

---

Published in Transactions on Machine Learning Research (10/2024)
by applying a binary mask M {m1, my..... m.y to these parameters. Each m, in M is either 0 or 1, indicating whether the corresponding parameter 0; is selected (1) or not selected (0) for fine-tuning. The updated parameter set 0' after fine-tuning is given by:
0 0 17- my 20,
(11)
where 1) represents the learning rate, and is the gradient of the loss function with respect to the parameter 0z. In this formulation, only the selected parameters (i.c., m, are updated during backpropagation.
89,
Diff pruming (Guo et al. 2020) is represenlative work that applies learnable binary mask to the model weights during fine-tuning, To achieve parameter efficiency, the mask is regularized by differentiable approximation of the Lo-norm penalty. PaFi (Liao et al., 2023a) simply select model paramcters with the smallest absolute magnitude as trainable. FishMask {Sung et al., 2021 determines parameter importance using the approximate Fisher information. Tt then selects the top k parameters based on this information form the mask M. Similarly, Fish-Dip (Das et al., 2023) also uses Fisher information to calculate M, but the mask will be re-calculated dynamically in each train period. LT-SFT (Ansell et al.) 2021) introduces another technique to determine parameter importance inspired by the Lottery Ticket Hypothesis (Frankle & Carbin 2018 Malach et al., 2020), where the subset of parameters that change the most during an initial fine-tuning stage is selected to form the mask M. SAM (Fu et al. 2023) proposes a second-order
approximation method, which approximates the original problem with an analytically solvable optimization function, to help decide the parameter mask. Child-tuning (Xu et al., 2021 proposes two approaches select child network during each training itcration, where only the parameters within this child network
can be updated.
However, the above unstructured parameter masking results in an uneven distribution of non-zero masks and diminished hardware efficiency when implementing PEFT. As shown in Figure 7 the structured mask orga- nizes parameter masking in regular patterns, unlike unstructured ones that apply it randomly, thus enhancing computational and hardware ef- ficiency during training. Therefore, various structured selective PEFT techniques have undergone extensive investigation. Diff pruning pro- poses a structured pruning strategy by partitioning the weight parameters into local groups and strategically eliminating them together. Similarly, FAR (Vucetic et al. 2022) fine-tunes BERT models by grouping weights of the FFN in Transformer blocks into nodes, then ranking and selecting the learner nodes using L1 norm. To further reduce the memory access frequency, they also reconfigure the FFN by grouping the learner nodes. Bitfii (Zaken et al. 2021 is proposed to only fine-tune the bias parameters of each DNN layer, and achieve competitive results for small models. However, this method fails to handle large models. Lawton et al. (2023) applies NAS to Bitfit, where S-BitFit keeps the structural nature in Bitfit that restricts NAS al- gorithm must choose whether 8b = 0 or not for each bias module. Similar to Bitfit fine-tunes a specific module in Transformer, Xattn Tuning (Gheini et al.) 2021) finc-tunes only the cross-attention layers. SPT (sensitivity-awarc visual parameter-efficinn fine-tuning) (He et al.) 2023ab first identifies the sensitive pa- rameters measured by the loss reduction when being tuned. This sensitivity is calculated using a first-order Taylor expansion, derived from single forward and backward pass before fine-tuning in one shot. Next, SPT finds the weight matrices whose number of sensitive parameters exceeds a pre-defined threshold and then applies selected PEFT technique (e.g,, LoRA and Adapter) to these targeted weights to achieve structural tuning.
(a) Unstructural Masking
(b) Structural
Masking
Frozen
Leamablee
Figure 7: Illustration of two pa- rameter masking methods.
3.3 Reparameterized PEFT
Roparamcterization stands for cquivalently transforming a model's architocture from one to another via transforming its parameters. In the context PEFT, this often means constructing a low-rank parameteri- zation to achieve the goal of parameter efficiency during training. For inference, the model can be converted
11

---

Published in Transactions on Machine Learning Research (10/2024)
d
W. e R up
W
Dess
Pre-trained Weights
Pre-trained Weights
Decompaas
x
W
x
y
Pretried Weggts
W, C
W E g
W, C R
Pt geg
W
W
down
(a) LoRA
(b) DyLoRA
(c) DoRA
Figure 8: Illustration of three representative reparameterized PEFT algorithms. Blue represents frozen,
while yellow represents trainable.
to its original weight parameterization, ensuring unchanged inference speed. This procedure is depicted in Figure | (c).
Earlier research studies (Aghajanyan et al. 2020) have shown that common pre-trained models exhibit an exceptionally low intrinsic dimensionality. In other words, it is possible to find a low-dimensional reparam- eterization that is effective for fine-tuning as the entire parameter space. Intrinsic SAID (Aghajanyan) et al. 2020) is the pioneering work in investigating the intrinsic dimension feature during the fine-tuning of L.Mss However, the most widely recognized reparameterization technique is LoRA (Tow-Rank Adap- 2021 Fomenko et al. 2024), as shown in Figure g (a). For a given pro-trained weight matrix Wo E Raxk, LoRA introduces two trainable weight matrices, Wop E Rgd and Witwn E R where the rank r & min(d, k), operating in parallel to Wo. Let hin represent the input. Under normal conditions, the output through Wo hou Wohnnn Instead, LoRA modifies this output by introducing an incremental update AW that encapsulates task-specific knowledge:
tation) (uu et al.
Nout Woltan + - AWh, Woltin W. W Woownhins
(12)
where denotes scaling factor. At the onset of training, Wdown is initialized using a random Gaussian distribution, while Wup is initialized to zero, ensuring that AW initially holds a value of zero. LoRA is straightforward to implement and has been evaluated on models with up to 175 billion parameters. Fig S (c) used a single decoder as an example, the frozen and learnable components are highlighted in grey and red, respectively. Once fine-tuning complete, LoRA's 's adaptive weights seamlessly integrate with the pre- trained backbone weights. This integration ensures that LoRA maintains the model's elficiency, adding no extra burden during inference.
In LoRA training, selecting an appropriate rank has always been a challenging issuc. To address this, DyLoRA (Valipour el al. 2022), as depicted in Figure g (b), trains the LoRA module on a range of ranks within a predefined training budget, rather than adhering to a single, fixed rank. Specifically, for a given rank range R (rminy Fimn 1..., 1,...,,muxx), DyLoRA dynamically chooses a rank r e at cach itcration of the training process. Consequently, the matrices W Wdown and Wup are tailored for the selected rank r, resulting in truncated versions Wdownir Wdown[ : r.:] and Waper Wapl:: r], and the subsequent forward and backward pass during this iteration will be restricted on Wdownir and W Wuplr instead of Wdown and Wup- With this dynamic and search-free approach, DyLoRA significantly reduces the training time required to find an optimal and fixed LoRA rank for specific tasks. AdaLoRA (Zhang et al., 2023e) reformulates the AW with a singular value decomposition (SVD), denoted as AW = PAQ, where P e Rdd and Q e Rrxk are orthometric, A is a diagonal matrix containing singular values {Ai)isisr. All three weight matrices are made learnable. During training, the singular values are pruned iteratively based on their importance scores, which are constructed from the moving average of the magnitude of the wridittt-weight product. To ensure the orthogonality between P and Q, i.e., pT P QQ" I, additional regularizer term is included in the loss:
m'ax),
R(P,Q) pT P 1 10Q - -1|
(13)
12

---

Published in Transactions on Machine Learning Research (10/2024)
This adaptive approach enables the model to dynamically adjust the rank within each LoRA module, effec- tively managing its parameter counts based on the significance of the weight matrices. However, according to SoRA (Ding et al., 2023a), the importance scores used in AdaLoRA are heuristically constructed, which lacks rigorous thcoretical motivation. Additionally, both moving average operation and calculation of Eq. 13
introduce extra computation costs during training- To address this, SoRA eliminates the orthogonality premise of P and Q. Instead, a gating unit g E R between Wap and down is directly applied and opti- mized:
how W (u (W hew)
(14)
where ① is Hadamard product. The gate g is updated using a variation of proximal gradient itcration for l loss (Beck & Teboulle 2009 Chambolle et al., 1998), which has a clear mathematical meaning and does not need the heuristic premise. After training, the zeroed-out gate units are pruned by removing the corresponding columns and rows in Wdown and Wup.
Several subsequent studies have aimed to improve LoRA's performance in various aspects. For instance, 2023a) notices that fine-tuned LLMs often exhibit overconfidence. To enhance the calibration of fine-tuned LLMs, Laplace-LoRA utilizes a Bayesian approach, specifically a post-hoc
Laplace-LoRA (Yang et al.
Laplace approximation (MacKay
1992: Antorán et al., 2022), to the posterior over the LoRA parameters.
LoRA Dropout (Lin et al. 2024) introduces random noises to the learnable low-rank matrices and increases
parameter sparsity to reduce the risk of overfitting. LoRA+ (Hayou et al. 2024] proposes to set different learning rates for the LoRA matrices Wdown and Wap: such that Mup ArJdown with À > fixed and tune 'ddown- MoSLoRA (Mixture--f-Suuspacees LoRA) (Wu el. al., 2024b) decomposes LoRA into subspaces via structural reparameteriization, then employs a. learnable mixer, trained jointly with the original LoRA weights, to fuse the subspaces. Similarly to LoRA, MoSLoRA can also be merged into the original weights.
Thanks to the modular design of LoRA, many studies incorporate multiple LoRA modules in their frameworks to enhance performance. For example, LoRAH aggregates various LoRA modules trained on different tasks. Given a handful of examples from a new task, LoRAHub can autonomously compose compatible LoRA modules without human intervention via a gradient-fre method Shiwa (Liu al., 2020). MOELoRA employs a Mixture-of-Expert (MOE) approach to train LoRA in a multi-task setting, resulting in multiple expert LoRA modules. To retrieve parameters for certain tasks, MOELoRA utilizes a task-motivated gate function that assigns contribution weights to each expert based on the lask ID, and the final parameters are calculated through a weighted sum of all experts.
In addition to LoRA, several other roparamcterization techniques are emerging with significant potential. For instance, Compacter (Karimi Mahabadi et al., 202Th introduces a light-weight. adapter modules by parameterizing the Wdown and Wup as W = >m_ A, & B,, where A; E IR=x", ; B, E R R, and @ denotes the Kronecker product. They further decrease the parameter count by designating A, as shared parameters and reparameterizing B; using the product of two low-rank matrices, effectively reducing the parameter complexity from O(rd) to O(r + d). Related studics, such as KronA (Edalati et al., 2022) and KAdap- tation (He et al. 2023c), also employ the Kronecker product to reparameterize adapter weights, aiming to achieve parameter reduction. HiWi (Liao et al.) 2023a) proposes an adapter fine-tuning method that applies an adapter directly to pre-trained parameters instcad of hidden representations as:
W' = W + oa WoW Waps
(15)
where W denotes the weights or biases within the Transformer block's leed-forward layer. Notably, during inference, this method computes W' in advance, ensuring that the model's inference latency remains on par with that of traditional full fine-tuning. VeRA (Vector-based Random Matrix Adaptation) (Kopickko et al. 2023) employs a single pair of frozen low-rank matrices Wup and Waown that are shared across all layers, and adapts these matrices by learning small, trainable scaling vectors represented as b and d (formally denoted by diagonal matrices As and Aa). Specifically, the reparameterization is given by:
hout = Wohni + An WgddWWWnw dow:
(16)
where both Wap and W Wowwn are initialized using a random Gaussian distribution. Similar to LoRA, the scaling vector b is initialized to zeros to ensure that the weight matrix is unaffected during the first forward pass. This
13

---

Published in Transactions on Machine Learning Research (10/2024)
method significantly reduces the number of trainable parameters compared to LoRA yet maintains the same performance, enabling the fine-tuming of larger models on single GPU. DoRA (Weight--Deoomosee Low-
Rank Adaptation) (Liu et al, (024b) presents a novel approach as illustrated in Figure 8 (c) by decomposing
model weights Wo e Raxx into magnitude and direction as follows:
V |
| Wo
Wo -
(17)
[Wolee"
where Riik is the magnitude vector, V Rdxk is the directional matrix, with Il being the vector-wise norm of matrix across each column. Subsequently, DoRA adopts unique fine-tuning strategy for and V. While both are lunable, only V undergoes LoRA reparameterization, defined as:
W V AV V AVI.
Wo + W W down
m
W W Wap W down
(18)
where AV is the incremental directional update learned by LoRA, and the underlined parameters denote the trainable parameters. Through this methodology, DoRA consistently outperforms LoRA across various tasks and models, demonstrating its superiority.
3.4 Hybrid PEFT
The efficacy of various PEFT methods can significantly differ across different tasks. As a result, numerous studies aim to either combine the advantages of diverse PEFT approaches seek to establish unified perspective by analyzing the similarities among these methods. For instance, UniPELT (Mao et al., 2021 integrates LoRA, prefix-tuning, and adapters into each Transformer block. To control which PEFT submod- ules should be activated, they also introduce a gating mechanism. This mechanism consists of three small FFNs that each produce a scalar value 9 E (0, 1), which is then applied to the LoRA, prefix, and adapter matrices, respectively. Across various setups, UniPELT has consistently shown improvements in accuracy ranging from 1% to 4%. S4 (Chen et al.] (023a) explores design spaces for several PEFT methods (i.e., Adapter (A), Prefix (P), BitFit (B), and LoRA (T.) to uncover underlying design patterns. After a series of experiments, their findings include: (1) Applying the spindle grouping partitioning for Transformer layers, which results in four layer groups G, for i (1...4). 4J. Layers in one group have similar behaviors together, which means should apply similar PEFT strategies. (2) Allocating the number of trainable parameters to layers uniformly. (3) Tuning all the groups. (4) Assigning different PEFT strategies to different groups. The resulting design space that has the best performance is:
G (,, L), (A, (A. (P,B,L
MAM Adapter(He et al., 2021] explores the intrinsic similarity between three additive PEFT methods: adapters, fufixi and LoRA, which leads to the development of three variants: Parallel Adapter, which places adapter layers alongside specific layers (SA FFN) instead of after them; Multi-head Parallel Adapter, which divides the parallel adapter into multiple heads, each affecting the head attention output in SA; and Scaled Parallel Adapter, which adds a scaling term after the parallel adapter layer, similar to LoRA. Extensive experimentation revealed that the most effective configuration involves using prefix-tuning in the SA layer and the scaled parallel adapter in the FEN layer, which is called the MAM Adapter. LLM- Adapters (Hu et al., 2023a) builds an easy-to-use framework that incorporates various PEFT techniques into LLMs. Through comprchensive benchmarking across multiple datasets, the study reveals several key insights: (1) The most ellective locations for series adapters, parallel adapters, and LoRA are after the MLP layers, alongside the MLP layers, and simultaneously following the Attention layers and MLP layers, respectively. (2) Smaller LLMs utilizing PEFT can achieve competitive or even superior results on certain tasks when compared to their larger counterparts. (3) With appropriate in-distribution fine-tuning data, smaller models are capable of surpassing larger models in task-specific performance.
Several studies leverage neural architecture search (NAS) to find better PEFT combination approaches. For example, NOAH (Zhang et al. 2022b) discovers that different PEFT configurations are specifically tailored for different tasks. To address this issue, NOAH employs NAS to identify the most effective PEFT
14

---

Published in Transactions on Machine Learning Research (10/2024)
Fifheien PEPT Dassigo
PEFT Prusing
ArapterDrop
ProPETL
Rickl et al 2020
Sparse.Maapkr
Be ct al. 20226
SPLoRA (edoguard et al.
2022
LoRAPrasing
Zhang et al. 20234
PEPT
Bt-Adapter
PEQA INT2.1
Sin
2028 GLORA Deituar 2ood
Lo
Guo
Quaniiattio
LA-L
QoyLore Po0)02000c
DitDent
2024a.
Momocy officc- PEFT
Side-Tining.
LaftA-PA
7
Zbang
2710
181
Sung
572h
Res Dining
2023
PRP
Jiang
2023
MRPT
0095
Liee
272b
al. 2025
Galarr
Figure 9: Taxonomy of Efficient PEFT Design.
configurations for each dataset. Specifically, NOAH's searching space encompasses three PEFT methods: Adapter, LoRA, and Visual Prompt Tuning (VPT). It utilizes AutoFormer (Chen al., 2021a), one-shot NAS algorithm, for the efficient discovery of optimal prompt modules. In related vein, AUTOPEFT Zhou et al. 2023) first establishes searching space that includes serial adapters, parallel adapters, and prefix tuning. After that, they propose an effective NAS method based on a high-dimensional multi-dinensional Bayesian optimisation (Frazior 2018). Both NOAH and AUTOPEFT demonstrate the capability NAS in enhancing PEFT configurations across a variety of tasks.
4 Efficient PEFT design
Processing latency and peak memory overhead are pivotal factors consider from a computational stand- point. This section introduces key characteristic in LLMs aimed at balancing between latency and memory usage (Section 4.1b. Following this, explore strategies for developing efficient PEFT methods to address computational challenges, including PEFT pruning (Section 4.2 PEFT quantization (Section 4.3 and memory-efficien PEFT techniques (Section 4.4), each designed to enhance model performance while minimizing resource consumption. It noteworthy that quantization inherently addresses memory over- head concerns. However, given distinct characteristics, we address these quantization methods separately rather than incorporating them under the memooy--fficient PEFT section.
4.1 KV-cache Management for PEFT Efficiency
The core of the LLMs model lies in an auto-regressive Transformer model. When we consider the auto- regression characteristic, it becomes a major challenge in designing an inference system, because every time a new token is generated, the entire LLM model has to transfer all the weights from different memories to the memory of the graphics processor, which is very unfriendly to single-user task scheduling or multi- user workload balance. The challenging part of serving the auto-regressive paradigm is that all previous sequences have to be cached and saved for the next proceeding iteration; the cached activation generated from the previous sequences is stored as the Key- Bey-Value Cache (KV-cache). To effectively manage these challenges, S-LoRA [Sheng et al. (2023a) employs Unified Paging mechanism within unified memory pool that dynamically allocates and manages memory in a paged fashion. This sophisticated approach minimizes memory fragmentation and enhances the efficiency of KV-cache storage by allowing for flexible and efficient memory access patterns. These pages are managed such that the KV-cache associated with each adapter segmented into managcable blocks, strcamlining access and reducing the overhead associated with variable cache sizes. By dynamically adjusting to different KV-cache requirements, S-LoRA maintains high throughput and performance, ensuring that the system remains responsive and efficient even it scales to serve thousands of adapters simultaneously. This efficient handling of K KV-cchhe is crucial for supporting the auto-regressive nature of LLMs in high-demand environments, optimizing both single-user and multi-user workload balancing.
4.2 Pruning Strategies for PEFT
The inclusion of pruning can substantially enhance the efficioncy of PEFT methods. In particular, Adap- terDrop (Rücklé al. 2020) explores the removal of adapters from lower transformer layers and multi-task adapters in AdapterFusion Pfeiifer et al. 2020), which shows that the pruning can improve the training
15

---

Published in Transactions on Machine Learning Research (10/2024)
and inference efficiency with minimal decrease in performance. SparseAdapter (He et al. 2022b investi- gates different pruming methods and finds that high sparsity ratio (80%) can outperform standard adapters. Additionally, the Larye-Sparse configuration, which increases the bottleneck dimension while maintaining a constant parameter budget (c.g., doubling dimensions with a 50% sparsity), substantially enhances the model's capacity, resulting in improved performance. SPLoRA (Hedegaard el al.] [022] adopts channel- basod pruning to the LoRA weights Wdown and Wap- This pruning affects not only the source weights Wo. but also the LoRA parameters Wup and Wdown. Similarly, LoRAPruning (Zhang structured pruning not only to the pre-trained model weights but also to the LoRA weights. In contrast to unstructured LoRA pruning methods, which primarily focus on sparsifying model weights while leaving LoRA weights dense, thus making weight merging challenging to achieve, LoRAPruning enables the weights to be merged easily. Additionally, this work also introduces a novel criterion that utilizes LoRA's 's gradients as an approximation of the gradients for the pre-trained weights, enabling the estimation of weight importance. ProPETLL (Zeng et al. 2023b) constructs a single shared prototype (e.g., adapter, prefix, or LoRA) across layers and lasks. In addition, ProPETL learns binary masks prune different sub-networks in different layers and tasks. As a result, the parameters can be reused across layers and tasks, largely increasing the parameter efficiency.
al. 2023d) adopts
4.3 Quantization Strategies for PEFT
Quantization serves as another popular technique for improving computational efficiency and reducing mem- ory usage. For example, by investigating the loss landscape of adapters, BI-Adapter (Jie el al. 2023) finds that adapters are resistant to noise in parameter space. Building on this insight, the authors intro- duce a clustcring-
quantization approach. Remarkably, they demonstrate that a. 1-bit quantization of adapters not only minimizes storage requirements but also achieves superior performance among all preci- sion settings. PEQA (Parameter-Ffficient and Quantiration-aware Adaptation) (Kim et al.) 2023) uses a two-stage pipeline to achieve parameter-efficiett and quantization-awwre fine-tuning, In the first stage, the pre-trained FFN weight matrix W E R=xm is quantized to W = s W. where s E Rax1 represents per- channel scales and W denotes the quantized weight. In the second stage, W remains fixed, and fine-tuning is only conducted on 8. This approach not only ensures memory efficiency but also facilitates paramcter efficiency. QLoRA (Dettmers et al. 2023) proposes several novel techniques, including a 4-bit NormalFloat, a Double Quantization, and a Paged Optimizers, to backpropagate a 4-bit quantized pretrained language model into LoRA. These techniques enable the fine-tuning for 65B language model on single 48GB GPU while maintaining similar performance to the full 16-bit fine-tuning. Similar to the original imple- mentation (Hu et al.] 2021), QLoRA attaches the fixed vero-initialized LoRA weights to the quantized pre-trained model as the training start point. However, when applying the extreme low-bit (e.g., 2-bit) quantization, the huge quantization error can adversely impact the initialization of LoRA fine-tuning, i.c., )uanlization(Wo) + Wdown Wip * Wo where Wdoonn = 0, which will harm the fine-tuning performance as shown in the work by [iao et al. (2023b). To solve this, several quantization strategies are proposed to elimi- nate the quantization error. For example, LoftQ (LoRA-Fine- SeRRA--ineTTunin--aae Tuning-aware Quantization) (Li et al. (023d) presents an innovative framework that provides superior initialization point of quantized backbonn weights and LoRA weights for subsequent LoRA fine-tuning. This approach addresses the discrepancies caused by quantization through the optimization of Frobenius norm objective during network initialization, which takes both the LoRA weights and the quantized pre-trained backbone into consideration. LoftQ exhibits superior performance in 2-bit quantization over QLoRA, as well as greater generalization for downstream tasks. LQ-LoRA (Guo et al. (0023) uses an iterative algorithm inspired by robust principal components analysis 4Zhou& & Tao 2011; Wright et al., 2209) which decomposes the weight Wo such that Wn x Q+ L L2 to resolve the inaccuracy caused by the quantization error, where Q is the quantized component which re- mains fixed and L2 Le is the trainable low-rank component. Moreover, this approach leverages integer linear programming to determine mixed quantization strategy, enabling dynamic quantization configurations for each weight matrix while adhering to a predetermined total bit rale limit. QA-LORA (Xu et al., 2023d] address another limitation of QLoRA, which struggles to preserve its quantized property post-fine- fotetiinetunig.. In QLORA, the quantized pro-trained weight (NF4) has to be recovered to FP16 to match the LoRA weight pre- cision (FP16) during weight merging. Instead, QA-LoRA uses INT4 quantization and introduces group-wise operators to enable quantization during the inference stage, therefore improving the efficiency and accuracy
16

---

Published in Transactions on Machine Learning Research (10/2024)
compared with QLoRA. BitDelta (Liu et al. 2024a introduces novel 1-bit post-training quantization method that acts on the weight delta between a fine-tuned model and its underlying pre-trained model. Specifically, given the weight matrices Winee and W from the fine-tuned and base models respectively, the weight delta A = Wine - Wase is binarized as A = a O Sign(A). Here, a, a high-precision scalar, is initialized based on the mean absolute delta value a = mi E., WWyl with Sigu(-) indicating the sign of A. BitDelta further calibrates the scaling factors via distillation on a compact calibration dataset, while the binary matrices remain unchanged. This approach notably streamlines the deployment of multiple fine-tuned models on shared servers by utilizing a singular full-precision base model alongside efficiently batched 1-bit deltas.
4.4 Memory-efficient PEFT Methods
Fine-tuning the full LLMs necessitates substantial training memory owing to their considerable size. While most PEFT methods primarily target parameter efficiency, they still incur a significant memory overhead during training because gradient computation and backpropagation are still necessary for these methods. For example, prevalent PEFT techniques such as adapters and LoRA can only reduce memory usage to approximately 70% compared to full model fine-tuning according to some literature (Sung et al., 2022a Jin et al. 2023 From computational perspective, memory efficiency also remains critical factor that cannot be overlooked.
To improve memory efficiency, various techniques have been developed to minimize the need for caching gradients for the entire LLM during fine-tuning, thereby reducing memory usage. For example, both Side- Tuning (Zhang et al.] 2020 and LST (Ladder-Side Tuning) (Sung et al. 2022a) introduce a learnable network branch parallel the backbone model. By channeling the backpropagation exclusively through this parallel branch, it circumvents the need to store gradient information for the main model's weights, thus markedly reducing memory requirements during training. Similarly, Res-Tuning (Jiang et al.) 2023) disentangles the PEFT tuners (e.g., prompt tuning, adapter) from the backbone model. On top of the disentanglement, a memory--fficient fine-tuning framework named Res- Stuning-Bypaa is proposed, which generates a bypass network in parallel with the backbone model by removing the data flow from the de- coupled tuners to the backbone. This eliminates the requirement for gradient caching within the backbone model during backpropagation. MEFT (Liao et al. 2023b) (memory-efficient aue-tuning) is an approach inspired by the reversible model (Gomez et al. 2017). During the training of reversible model, intermediate activations are not required to be cached in the forward pass. During backpropagation, they be recaleu- lated from the final output. To save the memory during fine-tuning, MEFT investigales how to transform an LLM to its reversible counterparts without additional pre- treinrainin...... A critical aspect of this transformation is the careful initialization of newly introduced parameters in the pre-trained models. MEFT demonstrates the importance of parameter initialization and suggests that these parameters must be initialized in a manner that preserves the pre-trained model's starting point, ensuring that the fine-tuning of the modified model achieves performance on par with full fine-tuning methods. With this key consideration, MEFT introduces three distinct methods, each significantly curtailing the memory demands traditionally required for storing activations. LoRA-FA (Zhang al. 2023b addresses a limitation about memory overhead in LoRA fine- tuning. During training, LoRA modules still require high activation memory consumption. This is because, during backpropagation, large input activations must be stored during the forward pass to compute gra- dients. LoRA-FA resolves this issue by freezing both the pre-trained weights Wo and the projection-down weights Wdown, and only updating the projection-up weights Wup- Consequently, the input activation h no longer needs to be stored, as the intermediate activation Waownhin is adoquate for gradient computation for Wup- Given that r < d, the memory requirement for activations in LoRA-FA can be significantly reduced.
To further reduce memory usage during fine-tuning, some methods attempt to circumvent backpropagation within LLMs to address this issuc. HyperTuning (Phang et al. 2023) employs a HyperModel to generate PEFT parameters using only fewshot examples. This approach demonstrates results comparable to those obtained through full model fine-tuning. PEFT Plug-in (Jin et al. 2023) first trains PEFT modules on small language models, which is more memory efficient compared to training on large ones. Subsoquently, the research introduces a suite of techniques for scamlessly integrating these trained PEFT modules into LLMs during inference. This strategy effectively circumvents the necessity of gradient-based optimization
17

---

Published in Transactions on Machine Learning Research (10/2024)
directly on the larger models, resulting in substantial memory savings. However, it is important to note that both HyperModel and PEFT Plug-in still require additional model training, and this training cost cannot be entirely overlooked. MeZO (Malladi et al., 2023) introduces a memory-efficien zeroth-order (20) optimizer for LLMs. Unlike conventional PEFT techniques, which rely on backpropagation to compute gradients for updating model parameters, MeZO fine-tunes LLMs through only forward passes. It. accomplishes this by employing a 70 gradient estimator to calculate the gradient. Notably, McZO implements an in-place solution for the classic ZO gradient estimator, effectively mitigating memory consumption during inference execution. This innovative approach allows for efficient fine-tuning of LLMs containing 30 billion parameters a single GPU with 80GB of memory, all while maintaining performance that is comparable to fine-tuning using backpropagation. Furthermore, it can substantially decrease storage demands in comparison to the traditional PEFT methods such as LoRA and adapters.
5 PEFT for DNNs of Other Applications
In Section we outlined four categories of PEFT methods along with their improvements. Nonetheless, our discussion did not fully extend to the utilization or adaptation of PEFT techniques beyond traditional architectures (e.g., LLMs) or standard benchmarks (c.g., the GLUE dataset), where the majority of the discussed PEFT methods are applied. Therefore, in this section, we will highlight and discuss several most representative works that leverage PEFT strategies for various downstream tasks. In this section, we do not aim to cover all PEFT application scenarios. Our objective is to showcase the significant influence of PEFT within various research domains and demonstrate how to optimize and tailor general-purpor PEFT methods to achieve enhanced performance in specific models or lasks.
Typically, fine-tuning happens when adapting a pre-trained backbone model to specialized downstream tasks. To this end, this section organizes the discussion around various model architcctures, which include: LLM, Vision Transformer (ViT), Vision-Language Aligument. Model (VL.A), and Diffusion model. Within each architectural category, the discussion is further classified based on different downstream tasks.
5.1 PEFT for LLMs Beyond the Basics
Instead of common tasks NLP such as NL. and NL.G, PEFT techniques boast. a wide array applica- tions across diverse scenarios. PEFT has been successfully implemented in commonsense question answer- ing (Huang et al. 2023c; Zhao et al., 2023d), multi-leyel implicit discourse relation recognition (Zhao ct al.
2023ch, out-of-distribution detection OOuang et al. 2023), privacy protection (Ozdayi et al. 2023 Xiao ct al. 2023b), federated learning (Che et al. (023), and social biases mitigation (Li et al. 2023c). In this
section, we pay more focus on three representative downstream tasks: visual instruction following, continual learning, and context window extension.
5.1.1 Visual Instruct Following
Several studies, including VL-BART Cho et al. 2021
MiniGPT-4 (Zuu et al. 2023b), and L. Liu
et al. 2023a), have successfully extended the capabilities of LLMs, initially designed for pure text, to com- prehend and generate responses to visual inputs. These enhanced models, namely visual instruct-following LLMs, can process both images and text to produce textual responses, which can be benchmarked on tasks
such as image captioning (Rennie et al.] |017 You et al.] 2016; Vinyals et al. 2016)] Hossain et al. 2019)
Wu al. [017]] Antol et al. (0015). However, these methods fine-tune the entire LLM to learn the visual representations, which
and visual question answering (VQA) (Wang et al.) 2017
be inefficient in both
time and memory. Therefore, it is natural to apply PEFT techniques in the fine-tuning of visual instruct- following LLMs. An carlier work VL-Adapter (Sung et al. 2022b) directly applics several PEFT methods (Adapler (Houlsby el al. 2019), Hyperformer (Mahabadi el. al.) 2021) and Compacter [Karini Mahabadi et al., 2021)) on VL-BART (Cho et al.) 2021) then benchmarks them on several image-text and video-text tasks. Results show that vanilla adapters are the best among them, which can achieve performance on par with full fino-tuning. However, considering the functionality gap between the encoders and decoders in VL- BART, directly assigning identical modular modifications will lead to suboptimal performance. Therefore,
18

---

Published in Transactions on Machine Learning Research (10/2024)
VL-PET (Hu et al. 2023b selectively integrates PEFT modules into different components of the encoder and decoder. They also introduce a gramularity-controllee mechanism for finer-grained control.
To adapt the recently prevalent LLaMA model, LLaMA-Adapter (Zhang et al. 2023f prepends a set of learnable prompts (similar to prefix tuning) to the input tokens in LLaMA's higher transformer layers. To avoid the unstable fine-tuning with large loss values at early training stages, instead of the randomly initialized weights of other PEFT methods, LLaMA-Adapter adopts zero-initialized attention mechanism, which learns zero-initialized gating factor to adaptívely control the contribution of adaptation prompts to the word tokens. This can maintain the fine-tuning starting point the same the original model and progressively inject new knowledge into the model, where a similar idea can be found in MEFT (Liao et al. 2023b) and LoftQ (Li et al. 2023d) discussed earlier. To represent visual information, LLaMA- Adapter extracts multi-scale global image features using a CLIP image encoder and then projects them to linguistic embedding space. After that, the feature is element- wisely added onto the adaptation prompts at all inscrted transformer layers. LLaMA-Adapter only introduces 1.2M learnable paramcters in LLaMA- 7B and costs less than one hour for fine-tuning on 8 A100 GPUs. A following work LLaMA-Adapter V2 (Gao et al. 2003bb demonstrates that the simple multimodal fusion in LLaMA-Adapter cannot generalize to more challenging open-ended multimodal reasoning tasks, where the visual cues tend to dominate the adaptation prompts than the language instruction data. To address this, LLaMA-Adapter V2 decouples the learning of instruction-following ability (to generate long language responses) and vision-language alignment to avoid interference between visual and language fine-tuning. Specifically, LLaMA- Adapter V2 sets disjoint parameter groups which are respectively learned from image-text pairs and language instruction data. The visual adaptation prompts are inserted in the early stage of LLM, while the language adaptation prompts remain at the higher transformer layers similar to the LLaMA-Adapter. Additionally, LLaMA-Adapter V2 introduces more learnable parameters and several expert systems (e-g., captioning, detection, and OCR) to enhance multimodal performance. LayerNorm Tuning (Zhao et al.] (023a) adjust only the weights of the LayerNorm within each attention block. This straightforward technique can achieve comparable or even better performance than the finetuning, while offering about 10x more parameter efficiency than LoRA.
5.1.2 Continual Learning
Continual Learning (CL) aims to learn sequence of new tasks over time within one single model, which has broad application in scenarios such as dialogue systems (Lee) 2017 information extraction systems (Chang
et al. 2006), and question answering systems (Yang et al., 2019). The main challenge in CL is catastrophic forgetting (Kirkpatrick et al.) 2017). popular practice, called architecture- based methods, tackles the CL by maintaining lask-specifie parameters in the model for each new task. Therefore, it's natural to leverage PEFT methods for CL tasks (Madotto et al. 2020 Zhu et al. 2022 Dai et al. 2022 Liang et al. 2023). For example, AdapterCI, (Madotto et al. 2020) paramcterizes each new task using residual adapters. During testing, since the task-id is not provided, AdapterCL uses an entropy-based classifier to select which adapter to use for accomplishing a specific task. CPT (Continual Prompt Tuning) (Zhu et al. 2022) trains a soft prompt for each task. Instead of training soft prompts from scratch, CPT proposes a series of techniques (continual prompt initialization, query fusion, memory replay, and a memory-guided technique) to achieve knowledge transfer from preceding and subsequent tasks. O-LoRA (orthogonal low-rank adaptation) (Wang) ct al. 2023b employs strategy of learning distinct tasks within separate low-rank vector subspaces that are kept orthogonal to each other in order to minimize interference. This approach can elfectively reduce catastrophic forgetting during the acquisition of new tasks.
5.1.3 Context Window Extension
LLMs are typically trained with pre-defined context size. For example, LLaMA and LLaMA2 have pre- defined context sizes of 2048 and 4096 tokens, respectively. The positional encoding RoPE has weak extrap- olation properties (Chen et al. 2023d], which means the performance drops obviously given an input length exceeds the pre-defined context length. To solve this, a naive solution is to fine-tune a pre-trained LLM to a longer context. However, this escalates computational costs quadratically with context size, straining memory and processing resources. To address this, LongLoRA (Chen et al., (023c) proposes to fine-tune a pre-trained LLM using LoRA to enlarge the context size. To reduce the perplexity gap between LoRA
19

---

Published in Transactions on Machine Learning Research (10/2024)
tuning and full fine-tuning, LongLoRA also opens embedding and normalization layers for training. In or- der to further improve training efficiency in long context scenario, LongToRA further introduces a novel shifted sparse attention (St-Attn) as an efficient substitute for standard self-attention during training, A subsequent study LongQLoRA (Yang) 2023) combines the advantages of LongLoRA with QLoRA and Position Interpolation (Su et al. 2021a to save GPU memory. This work successfully extends the context length of LLaMA2-13B from 4096 to 8192 on a single V100 with 32GB memory. LLoCO (Tan et al. 2024) introduces a pipeline that learns contexts offline through the combination of context compression and LoRA. The process begins by compressing documents into compact contexts, then fine-tuning LLM using LoRA the compacted context to improve the LLM's ability to accurately extract and utilize information from these compressed representations. During model serving, a standard RAG retriever selects both the compressed document and the most relevant LoRA module, and applies them to the LLM for inference. This approach effectively extends the context window of a 4k token LLaMA2-7B model to handle up to 128k tokens.
In addition to limited training-stage sequence length, real-world system memory constraints introduce an- other critical bottleneck to the context. window. Specifically, the capacity of the KV-cache is curtailed by available system memory. For example, 30B parameter LLM operating with imput length of 1024 and a batch size of 128 might necossitate up to 180GB for the KV-cache (Zhang et al.) 2024b), thereby restricting the feasible size of the context window. In response to this, some strategies have resorted to quantizing the KV cache (Sheng et al., 2023b Dettmers et al. 2022). but quantization will certainly compromise perfor- mance. To effectively counteract this issue without significant loss, GEAR (Kang et al. 2024] presents a novel approach by employing a low-rank matrix to capture the majority of coherent bases of quantization error, complemented by sparse matrix that addresses errors from outlier entries, thus efficiently minimizing approximation errors.
5.2 PEFT for ViTs
VIT (Dosovisskiy et al. 2010 has emerged as a powerful backbone model in the recent computer vision community. In the ViT model, images are treated as sequences of fixod-size patches analogous how LLM uses discrete tokens. These patches undergo linear embedding and then receive positional encodings. Subsequently, they are processed through standard Transformer encoders. The training of ViT can be super-
vised (Dosovitskiy et al. 2010 Steiner et al. 2021 or self-supervised (Chen et al.) 2021b He et al. 2022a and ViT can achieve superior performance when training with more data and using larger model size (De- highani et al. 2023). However, such scaling up inevitably escalates training and storage costs. Therefore, similar to LLMs, PEFT is widely implemented in various downstream tasks, such as dense prediction (Chen ct al.) 2022b), continual learning (Wang et al.] 2022b Gao al. 2023c),, deep metric learning (Ren et al. 2024). Here, we focus on two typical tasks to showcase the involvement of PEFT: image classification and video recognition.
5.2.1 Image Classification
Image classification on targeted visual datasets is a very common demand and has extensive applications, while pre-train then fine-tuning paradigm serves as a widespread strategy. A variety of methods leverage PEFT techniques to achieve efficient model tuning (Jia al. 2022: Chen al.] 2022b a Jie & Deng, 2022). For instance, FdaptFormer (Chen al. 2022a) inserts adapter modules in parallel to the FFN of the original ViT model for visual recognition tasks. VPT (Visual Prompt Tuning) (Jia et al., 2022 prepends small amount of task-specific parameters into the input sequence of each Transformer layer. When applying VIT to downstream tasks, only these added paramcters and the classification head are set to trainable. Yoo et al. (2023) notices that compared with supervised ViT, VPT often underperforms with self-supervised VIT. Further analysis demonstrates that different pre-trained methods and downstream tasks have varying degrees of dependency on transformer blocks at different locations. To tackle this issuc, the research introduces adaptable gates for ViT blocks. These gates dynamically modulate the contribution of prompt tokens to ViT blocks, allowing for more targeted adaptation of the model to the task at hand.
20

---

Published in Transactions on Machine Learning Research (10/2024)
5.2.2 Video Recognition
Several works consider the more challenging adaptation problem that transfers ViT to downstream tasks that have a much larger domain gap. For example, ST-Adapter "Spatio-Tempores Adapter) (Pan et al.] 2022) and AIM (Yang et al. 2023c) both insert adapters layers into pre-trained VIT blocks. Their primary goal is to model spatial-temporal information, thereby enabling efficient adaptation of ViTs from image models video tasks. Notably, both methodologies have exhibited performance that surpasses traditional full-model fine-tuning approaches.
5.3 PEFT for VLAs
Vision-language alignment models (VLA), such as CLIP (Radford al. 2021 ALIGN Jia et al. 2021 DeCLIP (Li et all 2021], and FLAVA (Singh et al.] 2022), are designed to learn a good image and text features which can be aligned within unified representation space. Each VLA typically consists of separate image and text encoders that extract respective features. Contrastive learning is leveraged in these models to effectively align the image and text features. Fine-tuning is leveraged to improve the performance of VLA in specific datasets or tasks, but fine-tuning the full model is computationally intensive. For instance, fine- tuning CLIP RN50x64 requires a batch size of 32,768 and 18 days of training on 592 V100 GPUs (Radford 2021 Moreover, full fine-tuning on smaller datasets often leads to catastrophic forgetting (Kirkpatrick ct al.
et al.
2017). In response to these challenges, and drawing inspiration from the success of PEFT techniques in NLP, range of PEFT stralegies have been proposed and implemented in VLA models, such as semantic
segmentation (Xu et al.
2023c; Yu et al. 2023 Xu et al.] 2023e), point cloud understanding (Zhang et al.
2022a; Zhu et al. 2023d
Wang et al., 2022c Huang et al., 2023b), video understanding (Ju et al. 2022 2022), visual reasoning (Han et al. 2023b Doveh et al. 2023 temporal action detection NNg el al.] 2022), to name a few. This section will focus on one common task that uses VT.As:
Ni et al. 2022 Lin et al.
open-vocabulaty image classification.
5.3.1 Open-vocabulary Image Classification
In open-vocabulary image classification, earlier works design class-specific prompts, e.g., a photo of [CLASS), for each category, and rank images based on their similarity to these textual descriptions. CoOp (Context Optimization) (Zhou al.) 2022b) replaces the handcrafted text prompt with learnable vectors, while keeping the entire VLA fixes during training. CoCoOp (Conditional Context Optimization) (Zhou et al. 2022a) builds on this by tackling CoOp's limitations in generalizing to unseen classes. It introduces a lighteightt neural network that generates an input-specific context token, dynamically adapting the prompt based on each image, thereby enhancing generalizability, but at the cost of increased computational demands duc to the instanco-aware operation. ProGrad (Zhu et al. 2023a) addresses the over- fitttiiiing risk in CoOp in a few-shot setting by regularizing the soft. prompt updates whose gradient is aligned to the general knowledge only updates the prompt whose gradient is aligned (or non-conflicting)
the general knowledge offered by al.) 2023) notes that existing methods Iearn prompts either in
the original prompt. MaPLe (Khattak the language in the vision branch of CLIP, which is not efficient in leveraging the multimodal nature of VLAss. To address this, MaPLe proposes branch-aware hierarchical prompts that simultaneously adapt both language and vision branches, and achieves superior performance. TPT (test-time prompt tuning) (Shu et al. 2022) studies prompt tuning on the fly without additional training samples. Specifically, during infer- ence, TPT first augments the input image into various views, which are then utilized to tune the learnable prompts. The primary training objective is to ensure the VLA can generate consistent responses when facod with these differing views. A following work DiffTPT [Feng al. 2023) further enhances the data diversity of test samples through diffusion models.
In another direction, several studics explore the usage of adapters in VLA. For example, CLIP- Adapter (Gao el al., 2023a) integrates residual-style adapters after CLIP's text and visual encoders. Therefore, unlike CoOp and CoCoOp, CLIP-Adapter avoids the gradient backpropagation through CLIP's encodors, leading to reduced computational requirements terms of both training memory and time. Tip- Adapter (Zhang et al., (021) adopts the same design with ALip-Adapter. Different from CLIP-Adapter, the weights of the adapter are obtained in a training-free manner from query-key cache model (Orhan
21

---

Published in Transactions on Machine Learning Research (10/2024)
2018
Grave et al. 2017) constructed from few-shot supervisions in a non-parametric manner. As result,
Tip- Adapter exhibits great efficiency compared to CLIP-Adapter' SGD training process.
5.4 PEFT for Diffusion Models
Diffusion models (Ho et al. 2020: Sohl-Dickstein et al. 2015 are class of generative models that learn to generate data by transforming random noise into structured output by progressive denoising process. During training, diffusion models learn to reverse the noise added to training data using denoising network, while in inference, they start from noise, using a denoising network to iteratively create data that mirrors the same distribution as the training examples. Diffusion models have various applications (Han et al, 2023a; Yang et al. 2023b Croitoru al.] [0023] Dhariwal & Nichol 2021 Ruiz et al., 2023); while the most notable stable diffusion (Rombach et al., 2022), which bridges the gap between text and image with its robust capability to generate coherent and contextually relevant images directly from textual descriptions. Numerous studies leverage PEFT techniques to adapt pre-trained diffusion model for downstream tasks, including accelerating sampling speed (Luo et al., 2023: Chai et al. 2023a), text-to-video adaptation (Wul et al. 2023a Xing et al. 2023), text-to-3D adaptation (Zeng et al. 2023a), etc. This section mainly focuses on two scenarios: integrating additional input modalitics beyond mere text- based conditioning, and customizing content. generation based on pre-trained diffusion model.
5.4.1 Additional Input Control
To incorporate additional input modalities (e.g., layout, keypoints) while retaining the extensive knowledge in the pre-trained model, GLIGEN introduces a novel approach, which maintains the original model's weights intact and integrates new, trainable gated Transformer layers (Alayrac et al. 2022) that take in the new grounding input. The resulting model can not only accurately represent the grounding conditions but also produce high-quality images. Remarkably, the model can also generalize well to unseen objects during inference. ControlNet (Zhang al.) 2023c) fine-tunes a trainable copy of the encoding layers from Stable Diffusion while locking its pre-trained parameter weights. The fixed original model and the trainable copy are bridged through zero convolution layers. These layers, starting with zero-initialized weights, are designed to progressively adapt during training, ensuring that harmful noise does not affect the pre-trained features of Stable Diffusion at the beginning of training. This refined model is capable of conditioning on a variety of inputs such as Canny odges, Hough lines, user scribbles, human key points, segmentation maps, shape normals, depths, etc. Concept Sliders (Gandikota et al., (023) introduces slug-andd-plyy LoRA adaptors to allow precise editing of concepts (e.g., age, smiling) within diffusion model. T2I-Adapter (Mou et al., 2023) introduces lightweight adapter model designed to align external control signals with the internal knowledge of text-to-image diffusion models. This adapter enables precise manipulation through structural control (e.g., sketch, depth map, semantic segmentation map, and keypose), color control (e.g., hue and color distribution), and integrating various controls by composing multiple adapters.
5.4.2 Customized Generation
The effectiveness of text-to-image diffusion models is limited by the user's ability to articulate the desired target through text descriptions, For instance, it is difficult to describe the precise features of an innovative toy car which is not encountered during large-scale model training. Consequently, the objective of customized generation is to enable the model to grasp new concepts from a minimal set of user-supplied images. Textual Inversion (Gal et al. ) addresses this by finding a new pseudo-word S+ (similar to soft prompt discussed in Section 3.1.2) that represents new, specific concepts in the textual embedding space of pre-trained text- to-image diffusion models. The pseudo-word S, is optimized via the original optimization goal in diffusion models given a small image set (typically 3-5 images) depicting the concept, and the pre-trained model is left untouched. During inference, S4 can be treated like any other word and composed with other textual queries (e.g, 'a photo of S, on the beach'). Custom Diffusion (Kunari el al.) (023) tackles a more challenging setting: compositional fine-tuning of multiple concepts. It fine-tunes only the We, W, mapping from text latent fcatures attention layers, which yiclds superior performance in multi-concept lcarning scenarios. Additionally, during fine- fine-tuning, Custom Diffusion prevents model forgetting by introducing a small set of real images with captions akin to the target, alongside employing augmentation for faster
22

---

Published in Transactions on Machine Learning Research (10/2024)
convergence and improved results. IP-Adapter Ye et al. 2023 identifies limitations in current approaches (e.g., ControlNet and T21-Adapter) which project condition signals into the cross-atlention modules. When handling image conditions aiming at controlling content, these methods are unable to generate images faithful to the prompted image. The issuc stems from that merging image features and text features within cross- allention layers loses image-specific information, leading to only coarse-grained controllable generation such as image style rather than image content. To overcome this, IP-Adapter introduces a novel decoupled cross-attention mechanism to distinguish between text and image features. IP-Adapter adds an additional cross-attention layer exclusively for image features in each cross-attention layer, and only the parameters of the new cross-attention layers are trained.
6
System Design Challenge for PEFT
6.1 System design for PEFT
In this section, we begin by providing concise overview of cloud-based PEFT systems and analyzing the de- sign challenges. These include the efficient handling of numerous task-specific queries via centralized PEFT query servicing, the resolution of privacy and data transmission issues through distributed PEFT training, and the complexitics associated with concurrent multi-PEFT training processcs. Centralized systems are required to process substantial volume of queries with minimal latency and maximal throughput. Dis- tributed training frameworks must address privacy concerns and the computational inefficiencies that arise from data exchanges between users and cloud services. Furthermore, multi-PEFT training necessitates the oplimization of memory utilization, the management of simultaneous model training, and the formulation of system architectures capable of supporting multi-tenant workloads effectively. These challenges underscore the imperative for innovative approaches to improve scalability, safeguard privacy, and optimize resource allocation in PEFT system architectures. Following this, we present the corresponding metrics employed for evaluating the system performance. Furthermore, we delve into three prospective utilization scenarios to illustrate the challenges in system design.
6.1.1 Centralized PEFT Query Serving
Cloud providers have recently introduced a range of LLM services aimed providing user applications through application programming interfaces (APIs) (OpenAl 2023b Team al. 2023). These APIs facil- itate the scamless integration of many machine-learning functionalities into applications. When receiving one query for one specific downstream task through API, the cloud-based server processes the query with one featured LLM model. Under this scenario, the importance of PEFT becomes apparent. Cloud providers store only a single copy of the LLM and multiple PEFT modules featuring different downstream tasks. This setup allows the LLM to maintain varions branches of PEFT modules, each linked specific API queries, i.e., PEFT queries.
Centralized PEFT query serving solutions address scenarios where multiple PEFT queries arrive in quick succession. A case study of one stato-of-the-ar system for this purpose is discussed in Section 6.2 Fig- ure 10 (b) illustrates the computation pattern for multi-query PEFT inference, wherein packed PEFT queries are scheduled and exccuted according to their deadlines and current system conditions.
6.1.2 Distributed PEFT Training
In most cases, personalized lasks are not fully supported with pre-trained models, consequently, extra fine- tuning is required to be executed with the methodologies mentioned in the previous sections. However, significant concerns arisc when considering the transfer of datasets cloud providers, given the issues related to data privacy, copyright, proprictary information, and the complexities and inefficiencies involved in data transmission. Section 6.3 gives two approaches that address this concern.
23

---

Published in Transactions on Machine Learning Research (10/2024)
Large
Cloud
I like
Serving System
Query
enjoy
LLMs
Execution Engine
Scheduler
LLM
Response
programming
Edge Device
Modules
Request Pool
Personal
(a)
(b)
Figure 10: (a) Distributed-based system computation pattern; (b) centralizod PEFT Query inference.
6.1.3 Multi-PEFT Training
Different from multiple-PEFT serving, tuning with multiple customized PEFTs always involves different backbone LLMs. Therefore, simultaneously luning multiple PEFTs can pose considerable challenges. Chal- lenges like how to manage memory gradient and model weights storage, and how to design efficient kernel for batching PEFT training remain unsolved. PEFTs will be categorized based their PEFT algorithms and backbone LLM models. The design challenge involves how to consolidate multiple PEFTs with the same LLM backbone and multiple different LLM backbones simultaneously. We present case studies related to this topic Section 6.4
6.1.4 Evaluation Metrics
For the proposed evaluation metrics, without loss of generality, we adopt large language models as the basis for our metric definitions.
To evaluate the system performance of PEFT serving systems, we propose a set of evaluation metrics:
Considering PEFT querics as inter and intra tasks, we use tokens per second lo measure the system throughput.
System throughput:
Memory footprint: Run-time memory consumption during query serving, the memory utilization
comes from both model parameters and KV-cache as mentioned in Section 4.1
Accuracy performance: Real-world queries normally have different context lengths, and perfor- mance with variation length serves as a performance benchmark.
Quality of services: Queries associated with latency requirements and deadline missing rates are considered as another benchmark.
To assess the eflicacy of PEFT braining systems, we also establish set of evaluative metries:
Accuracy performance: Performance of the fine-tuned model over the downstream tasks.
Compute cost: The compute cost during forward and backward propagation operations on cloud servers and edge devices.
Communication cost: Refers the volume of data involved during the transfer of intermediate data between the edge device and the cloud.
6.2 Centralized PEFT Serving Frameworks
The PEFT algorithm is notable for its ability distinguish between modifiable and immutable weights within model. This characteristic inspires developers to amalgamate diverse LLMss with distinct PEFT
24

---

Published in Transactions on Machine Learning Research (10/2024)
Input Querles
Task
Task
Task
Task
Task
Queryy <Task_lth
PET Inference Pipaline
<input
User Inpans
Ratch Schediler
Engine
PET-OPs Profiling
clask_ do <irput Dota>
lipul Anvlpiing
Model
Sehwdider
Sctedaling
FEl Sperator
Model
Task D
Task 1
Step 1: Intra-Tast Task Batching
Task 2
Task 3
Task
Register
Mini
Batch
Tre-Sraied Model Tagl
Task Manager
B-2, S-34
PET Parameters
Task Register
Step 2: Inter-Task Batching
PET Parameters
Shared Model
Model
Batch 0
Batch
Batch
Manager
+ Shared-OPs Profiling
Parameter Repository
Macra Barch
Figure 11: PetS system overview: (1) Tasks register; (2) Task manager (3) Task schodule; (4) Task serving.
Bo4, S-34
Figure 12: Coordinated Batching (CB) Strategy
(Image is laken from PetS [Zhou et al. 2022ch
techniques into collective units. PetS, as introduced in Zhou et al. (2022e), advocates for comprehensive approach to managing multiple PEFT tasks by suggesting unified serving framework. The framework's ' core advancement. lies in the translation of varying PEFT tasks into integrated computation kernels to enhance efficiency. Moreover, PetS pioneers an orchestrated batching approach and a scheduling methodology, aiming to augment system throughput and leverage task parallelism respectively.
As depicted in Figure 11] the Pet.S framework begins with users registering PEFT lasks through a standard- ized Application Programming Interface (API). Upon registration, developers are expected to provide the Pre-Trained Model Tag (c.g., LLaMA), PEFT parameters in a compressed format, and the specific PEFT algorithrms (e.g,, LoRA, Adapter, Bitfit, etc.). These tasks are then endowed with unique identifiers, and the inference engine takes charge of query processing. PetS bifurcates the primary computational workload (e.g.. linear layer computations) into three distinct computational operations: (1) Dense Matrix- Vector Multipli- cation (MVM) leveraging universally accessible, pre-trained weights. (2) Bias vector addition (Vadd), using either common or task-exclusive biases. (3) A combination of Sparse/dense MVM operations employing task-specific PET parameters. A unified pro-trained weight matrix W is employed across PetS, facilitating the batching of initial operations, X, x W. However, subsequent task-specific computations involving PET parameters, despite being relatively minimal in complexity, are processed individually.
-eatrix-Vector
Considering the Adapter and Bitfit tasks as an illustration, both aim at the MLP component of LLMs. The Adapter lask integrates additional weight segments, whereas Bitfit adjusts bias elements. The Adapter operation is modeled Y = Xin (W + Waa) + bo. where Xm1 represents the input for the Adapter task, W and Wa are the original and adapter-specific PEFT weights respectively, and by is the initial bias. The Bitfit operation, bias. These operations are further synthesized as (Yi, Y2) = [Xin1, Xina) ) x W + (Xin1 x Wad, 0} + (bo, bn ) delineating that the pertains to the Vadd operation.
the other hand, is defined as Y = Xina x W W with by symbolizing the adfftt-adjustable
x W part is amenable to batching through MVM, while the {bo, b1} segment
For tasks like Diff-Pruning 3.2 is a little bit different than Bitfit and Adapter. For Diff-Pruning, the computation concerning the shared weight and 'difference" are conducted separately. Then the results are added up, namely
X, (W + 5,) X, W X, 6
here, the W denotes the backbone model weights while d, denotes the pruned weights which can be represented as Sparse MVM.
The other challenge PetS proposed is how to schedule different PEFT requests to achieve high performance. PetS scheduler achieves high parallelism through a two-level scheduling policy: Coordinated Batching (CB) and Macro-batch Streaming (MS) as Figure 12 depicts. Through CB, the input queries will first be clustered based on their input length and then grouped based on their shared operator. This is to make sure the same sequence length of queries will be executed without wasting padding. MS strategy will take the grouped
25

---

Published in Transactions on Machine Learning Research (10/2024)
queries after coordinated batching and the theoretical latency for different operators well as the system modeling parameters to generate the best execution order.
The other example design is DLoRA Wu et al. 2024a), which introduces a system that improves the ef- ficiency of serving low-rank adaptation (LoRA) models for large language models (LLMs) by dynamically managing the merging and unmerging of LoRA adapters and the migration of requests across worker replicas. This dynamic orchestration addresses the challenges of high memory footprints, low GPU utilization, and load imbalance caused by variable input and output lengths in traditional LLM serving systems. dLoRA's novel approaches, including credit-based batching algorithm and request-adapter co- migration algorithm, significantly enhance throughput.
6.3 Distributed PEFT Training Frameworks
We already know that fine-tuning LLM for downstream tasks is challenging for two reasons: dual privacy concerns between cloud server and data owner, and issues with computational resources and efficiency. Firstly, the privacy of both parties is at risk: the weights of large models are often proprictary and not made public. Sharing data with model owners for fine-tuning model weights to data proprietors could compromise the ownership of proprietary models. Secondly, even if downstream users have access to pre-trained weights, the stringent hardware requirements make transfer lcarning impractical for most end users.
lead to data privacy concerns while providing
To resolve these two issues, DLoRA (Gao Zhang. 2024] presents a distributed PEFT framework. During the PEFT process, the backbone LLM is executed in the cloud servers while the PEFT modules are trained entirely within the user devices. DLoRA scheme is depicted in Figure 10 a).
Similarly, Offsite-Tuning Tuning (Xiao et al. 2023a presents privacy-preserving and efficient transfer learning framework that enables foundational models to adapt to downstream tasks without the need to access the complete model weights. The key insight of Offsite-Tuning is the cloud provider sends an adapler and an emulator to the data proprietor. Then, with the assistance of the emulator, the data proprietor fine- tunes the adapter. The fine-tuned adapter then sent back to the cloud side, which integrates into the complete model, creating a fine-tuned foundational model for downstream users. Offsite-TTuning safeguards the privacy of data proprietors since they do not need to share their training data directly. It. also protects the foundational model owners, as the complete model weights are not shared, and the emulator provided is lossy, with significantly degraded performance. Compared to existing finc-tuning methods that require access to the full model weights, Offsite- Tuning is more resource-efficien. because it allows for fine-tuning through a compressed emulator without needing the complete model.
6.4 Parallel PEFT Training Frameworks
Unlike the PEFT query serving system, which aims accommodate flexible multi-PEIT algorithms, Punica (Chen et al. 2023b focuses solely on facilitating multiple-LoRA blocks for various tasks. Designing multiple PEFT training systems presents key challenges in two main aspects:
Efficient concurrent execution of multiple PEFT models with the same LLM backbone.
Designing an efficient system for multi-tenant serving with different LLM backbones.
Efficient kernel design Punica addresses the first challenge by using existing matrix multiplication for the backbone computation and introducing new CUDA kernel, Segmented Gather Matrix- Multi- plication (SGMV), for adding the PEFT add-ons to the backbone computation in batched manner. This kernel parallelizes the weight multiplication for different requests in the batch and groups requests corresponding to the same PEFT model to increase operational intensity and use GPU Tensor Cores for acceleration.
The second challenge is beyond the computational cost, designing an efficient system architecture that can effectively serve multi-tenant PEFT model workloads the smallest set of GPUs possible while occupying
26

---

Published in Transactions on Machine Learning Research (10/2024)
the least amount of GPU resources is another significant challenge. Punica addresses this by scheduling user requests to active GPUs that already serve train PEFT models, thereby improving GPU utilization. For older requests, Punica periodically migrates them to consolidate workloads, thus freeing up GPU resources for new requests.
Multi-Tenami PEFT design Designing efficient system for the multi-tenant PEFT model serving in the Punica framework focuses addressing several key challenges to maximize hardware utilization and minimize resource consumption. The system aims to consolidate multi-tenant LoRA serving workloads onto the smallest set of GPUs possible. This consolidation is achieved through strategic scheduling of user requests to active GPUs that are already serving or training LoRA models, thereby improving GPU utilization. For older roquests, Punica periodically migrates them to consolidate workloads further, thus frecing up GPU resources for new requests. It incorporates on-demand loading of LoRA model weights, which introduces only millisecond-level latency. This feature provides Punica with the flexibility to dynamically consolidate user requests to small set of GPUs, without being constrained by the specific LoRA models already running on those GPUs. Besides that, Punica identifies that the decode stage is predominant factor in the cost of model serving, Punica's design primarily focuses on optimizing decode stage performance. Other aspects of model serving leverage straightforward techniques, such as on-demand loading of LoRA model weights, to efficiently manage resource utilization.
7 Conclusion and Future Directions
In the current era dominated by large models and large datasets, PEFT stands out as a highly attractive method for efficiently adapting models to downstream tasks. This technique gains its appeal by addressing the significant challenges posed by traditional full-model fine-tuning, which often places substantial computa- tional and data demands. This survey offers a comprehensive examination of the most recent advancements in PEFT, including algorithmic design, computational efficiency, application scenarios, and system implemen- tation for PEFT. It offers a comprehensive taxonomy and explanation that serves as an excellent guidance and knowledge base, which enables readers of various levels and disciplines to swiftly grasp the core concepts of PEFT.
For further research on PEFT, we propose series of possible directions from both algorithm and system perspectives, hoping to inspire more researchers to engage in further studies in these areas.
7.1 Simplify hyperparameter tuning
The offectiveness of PEFT is often sensitive to its hyperparameters, such as the bottleneck dimension of the adapter, the rank of LoRA, and the arrangement of various additive PEFT layers. Manually tuning these hyperparameters will cost lots of effort. Therefore, future efforts could focus on developing methods that are less dependent on manual tuning of these parameters, or automatically find the optimal configuration settings. Several studics (Valipour et al., 2022 Zhang et al., 2023c. Ding et al., 2023a Chen et al.) [022a]] Zhang et al. 2022b Zhou et al. (023) have started address this issue, but there's a need for more simple and efficient solutions optimizing these hyperparameters.
7.2 Establish a unified benchmark
Despite the existence of librarics like HuggingFace's PEFT Mangrulkar et al. 2022) and AdapterHub (Poth et al., 2023), a comprehensive benchmark for PEFT is still lacking. This gap hinders the ability to fairly compare the performance and efficiency different PEFT approaches. A well-accepted, up-to-date bench- mark akin to MMDetoction (Chen et al.] 2019) for object detection would enable rescarchers to validato their methods against a standard set of tasks and metrics, fostering innovation and collaboration within the community.
27

---

Published in Transactions on Machine Learning Research (10/2024)
7.3 Enhance training efficiency
The presumed parameter efficiency of TEFT is not always consistent with computational and memory savings during training. Given that trainable parameters are intertwined within the pre-trained model's architecture, computing and storing activations and gradients for the full model often become necessary during stuettunigg This oversight calls for rethinking what constitutes efficiency. As outlined in Section potential solutions in the integration of model compression techniques such as pruning and quantization, alongside innovations specifically designed to optimize memory during PEFT tuning (Zhang al., 2023g). Further research into enhancing the computational efficiency of PEFT methodologies is imperative.
7.4 Explore scaling laws
The design and effectiveness of PEFT methods originally developed for smaller Transformer models do not necessarily scale with larger models. As the size of foundation models increases, identifying and adapting PEFT strategies that remain effective crucial. This investigation will aid in customizing PEFT method- ologies to suit. the evolving landscape of large model architectures.
7.5 Serve more models and tasks
The rise of large foundation models across various domains presents new opportunities for PEFT. Design- ing PEFT methods tailored to the unique characteristics of models, such as Sora (Brooks et al., 2024). Mamba (Gu & Dao 2023), and IVM (Bai et al. 2023), can unlock new application scenarios and opportu- nities.
7.6 Enhancing data privacy
Trusting contralized systems to serve fino-tune personalized PEFT modules is yet another issue for system developers. Multiple types of inversion attacks (Dosovitskiy & Brox 2016 He el al. 2019) have been proposed to reconstruct user's data by hijacking the intermediate results. One perspective of future trust-worthy LLM system design involves developing an encryption protocol for both personal data and intermediate training and inference results.
7.7 PEFT with model compression
Model compression is of the most effective ways to make L.M executable resource-limtted devices. Yet. the impact of model compression techniques on the performance of PEFT algorithms rumning hardware remains another systemic challenge. Common compression techniques such as quantization and pruning necessitate dedicated hardware platforms to expedite the process, and building such hardware platforms for compressed modols is yet another direction for future rescarch.
References
Armen Aghajanyan, Luke Zeltlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint orXiv:2012. 13255, 2020.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Newral Information Processing Systems, 35:23716 23736, 2022.
Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint a:Xiv:2110.075000 2021.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on 2015.
computer vision, pp.
28

---

Published in Transactions on Machine Learning Research (10/2024)
Javier Antorán, David Janz, James U Allingham, Erik Daxberger, Riccardo Rb Barbano, Eric Nalismick, and José Miguel Hernández-Tobbtto Adapting the linearised laplace model evidence for modern deep learning. In International Conference on Machine Learning, pp. 796 821. PMLR, 2022.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexci A Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312.0077 2023.
Amir Beck and Mare Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183 202, 2009.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference Artificial Intelligence, 2020.
Tim Brooks, Bill Peebles, Connor Holmes, Will DcPue, Yufei Guo, Li Jing, David Schmurr, Joe Taylor, Troy Luhman; Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai. https///ppenai..comrreeeeaa/ com/research//
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
Weilong Chai, DanDan Zheng, Jiajiong Cao, Zhiquan Chen, Changbao Wang, and Chenguang Ma. Speedup- net: A plug-and-play hyper-network for accelerating text-to-image diffusion models. arXiv preprint arXiv:2312. 08887, 2023a.
Yuji Chai, John Gkountouras, Glenn G Ko, David Brooks, and Gu- Yeon Wei. Int2. 1: Towards fine- tunable quantized large language models with error correction through low-rank adaptation. arxiv preprint orXiv:230608811 2023b.
Antonin Chambolle, Ronald A De Vore, Nam-Yong Lee, and Bradley J Lucier. Nonlincar wavelet im- age processing: variational problems, compression, and noise removal through wavelet shrinkage. IEEE Transactions on image processing, 7(3):319-38) 1998.
Chia-Hui Chang, Mohammed Kayed, Moheb R Girgis, and Khaled F Shaalan. A survey of web information extraction systems. IEEE transactions knowledge and data engineering, 18(10):1411-122 2006.
Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor S Sheng, Huaiyu Dai, and Dejing Dou. Federated learning of large language models with parameter-efficiet prompt tuning and adaptive optimization. arxiv preprint arXiv:2310.11 15080, 2023.
Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. Pfffmeter-effciient fine-tuning design spaces. arXiv preprint Xr1::2301.01821, 2023a.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.0 151i:1906.7755,, 2019.
Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. Punica: Multi- tenant lora serving. arxiv preprint 0tt:23310118477 2023b.
Lichang Chen, Heng Huang, and Minhao Cheng. Ptp: Boosting stability and performance of prompt tuning with perturbation-based regularizer. arXie preprind arXiv:2305.02/22 02422, 2023c.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Scarching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12270- 12280, 2021a.
29

---

Published in Transactions on Machine Learning Research (10/2024)
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664 16678, 2022a.
Shouyuan Chen, Sherman Wong, Liangjian Chon, and Yuandong Tian. Extonding context window of largo language models via positional interpolation. arXiv preprinl arXiv:2306. 15595, 2023d.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. Tn Proceedings of the TEEE/CVE international conference on computer vision, pp. 9640-9648 2021b.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309. 123007 2023c.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint (rXic:225508553, 2022b.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit. Bansal. Unifying vision-and-languaae tasks via text generation. In International Conference on Machine Learning, pp. 1931 1942. PMLR, 2021.
Young Choi, Junho Kim, Jun-Hyung Park, Wing-Lam Mok, and SangKeun Lec. Smop: Towards efficient and effective prompt tuning with sparse mixxture-o-prroptts In Proceedings of the 2023 Conference on Empirical Methods in Natural Language. Processing, pp. 14306-14316, 2023.
YunSeok Choi and Jee-Hyong Lee. Codeprompt: Task-agnostic prefix tuming for program and language gencration. In Findings of the Association for Computational Lènguistics: ACL 2023, pp. 5282-5297, 2023.
Alexandra Chronopoulon, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint 0720:2302.000027 2023.
Christopher et al. Clark. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the al2 reasoning challenge. WxXiv:1833.0555000 2018
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions Pottern Analysis and Machine Intelligence, 2023.
Yi Dai, Hao Lang, Yinhe Zheng, Fei Huang, Luo Si, and Yongbin Li. Lifelong learning for question answering with hierarchical prompts. arXio preprint 2rXii:2208.146000 2022.
Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Pong Shi, Wenpeng Yin, and Rui Zhang. Unified low- resource sequence labeling by sample-aware dynamic sparse finetuning. ar arXie preprinl arXiv:2311 1.00348, 2023.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Leurning, pp. 7480-7512. PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318 30332, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llss. Xix preprint arXiu:2305. 14314, 2023.
30

---

Published in Transactions on Machine Learning Research (10/2024)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural 2021.
information processing systems,
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arviv preprint or Nx::2311.11609 2023a.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-effcient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023b.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. Tn Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4829 4837, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arxiv 2020. arXiv preprint or orXiv:2010.1190 11929, 2010.
Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vi- sion & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2657-2668, 2023.
Ali Edalati, Marzich Tahaci, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mchdi Rezagholizadch. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprinl or Xrx::2221 10650, 2022.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Wimn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-388 338, 2010.
Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning, In Proceedings of the IEEE/CT International Conference on Computer Vision, pp. 2704-2714, 2714, 2023.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-oflorras: An efficient multi- task tuning for large language models. arXiv preprint arXiv:2403.032 033.0332, 2024.
Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, and Weizhu Chen. A note on lora. orXiv preprint arXiv:244.000886 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arxiv preprint arXiv: 0188803.3633 2018.
Peter I Frazier. A tutorial bayesian optimization. arxiv preprint arXiv:1807.02811 2018
Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effective- ness of parameter-efficien fine-tuning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 12799-12807 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXim:2208.0111 01618, 2022.
Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. orXiv preprint arXin:2311.12009, 2023.
Chao Gao and Sai Qian Zhang. Diora: Distributed parametor-cfficient fino-tuning solution for large language
model. arxiv preprini ar arXiv:2404.055 051.88 2024.
Peng Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongshong Li, and Yu Qiao. Clip-adapter: Better langunee models with feature adapters. International Journal of Computer Vision, pp. 1-15, 2023a.
31

---

Published in Transactions on Machine Learning Research (10/2024)
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojum Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Tlama-adapter v2: Parameter-efficient visual instruction model. or Xin preprinl arXiv:230,.15010, 2023b.
Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficent luning. arXiv preprinl or 23033 10070, 2023c.
Mozhdch Gheini, Xiang Ron, and Jonathan May. Cross-attention is all you need: Adapting protrained transformers for machine translation. Xiv preprint arXiv:2104.0877 08771, 2021.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. Advances in nearal information processing systems, 30, 2017.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, He- Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Muelter-Freitag, al. 'hee something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Edouard Grave, Moustapha M Cisse, and Armand Joulin. Unbounded cache model for online language modeling with open vocabulary, Advances in neural information processing systems, 30, 2017.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint
arXiv:2312.0077 00752, 2023.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficiet transfer learning with diff pruming. arXiv preprint 2xXi::201..07463, 2020.
Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decom- position for elficient language model finetuning. arXiv preprinl urXi::2311. 1.11..222,, 2023.
Muhammad Usman Hadi, R Qureshi, Shah, M Irfan, A Zafar, MB Shaikh, Akhtar, J Wu, and S Mirjalili. A survey on large language models: Applications, challenges, limitations, and practical usage. TechRriv, 2023.
Zeyu Han, Yuhan Wang, Luping Zhou, Peng Wang. Binyu Yan, Jiliu Zhou, Yan Wang, and Dinggang Shen. Contrastive diffusion model with auxiliary guidance for coarse-to-fine pet reconstruction. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pp. 239 249. Springer, 2023a.
Zeyu Han, Fangrui Zhu, Qianru Lao, and Huaizu Jiang. Zero-shot referring expression comprehension via structural similarity between images and captions. arXiv preprint arXiv:2311.17046 2023b.
SONG Haobo, Hao Zhao, Soumajit Majumder, and Tao Lin. Inercasing model capacity for free: A sim- ple strategy for parameter efficient fine-tuning. In The Twelfth International Conference on Learning Representations, 2023.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint 24v: 12354, 2024.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang, Sensitivity-aware visual parameter- efficient fine-tuning. Tn Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11825 11835, 2023a.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Borg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. aixiv preprint 4rXiv:2110.0433 2021.
32

---

Published in Transactions on Machine Learning Research (10/2024)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVT /CV conference compuler vision and pattern recognition, pp. 100000160000 2022a.
Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. SparseAdapter: An easy ap- proach for improving the paramotor-cfficenney of adapters. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pp. 2184-2190, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguisties. doi: findings-emmlp..... 160
URL
http://ccaanttoloo..orrr////////
Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Mera: Merging pretrained adapters for few-shot learning. arXiv preprint XrXi:2308.15988 2023b.
Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-efliciet model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 817-825, 2023c.
Proceedings of the 35th Annual Computer Security Applications Conference, pp. 148 162, 2019. Zecheng He, Tianwei Zhang, and Ruby B Lee. Model inversion attacks against collaborative inference. Tn
Lukas Hedegaard, Aman Alok, Juby Jose, and Alexandros losifidis. Structured pruning adapters. arXiv preprinl trXiv:2211.10155 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840-6855 2020.
MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehensive survey of deep learning for image captioning. ACM Compuling Surveys (CsUR), (6):1-336, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parametor-cfficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790 2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 096855 2021.
Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun. Sparse structure search for parameter-cffiiient tuning. arXiv preprint ar Xr:::2200 07382, 2022.
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Lim-adapters: An adapter family for parameter-efficien fine-tuning of large language models. arXiv 2023a.
preprinl ar Xiv:22004
Zi- Yuan Hu, Yanyang Li, Michael R Lyu, and Liwei Wang. Vi-pet: Vision-aandlanguuage parameler-eflcient tuning via granularity control. In Proceedings of the IEEE/CVE /CF International Conference on Computer Vision, pp. 3010-3020, 2023b.
Chengsong Huang, Qian Lin, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307. 13269, 2023a.
Tianyu Huang, Bowen Dong, Yunhan Yang, Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wang- meng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22157-22167, 2023b.
Yongfeng Huang, Yanyang Li, Yichong Xu, Lin Zhang, Ruyi Gan, Jiaxing Zhang, and Liwei Wang. Mvp- tuning: Multi-view knowledge retricval with prompt tuning for commonsense reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13417-13432 2023c.
33

---

Published in Transactions on Machine Learning Research (10/2024)
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904 4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709 727. Springer, 2022.
Zeyinzi Jiang, Chaojie Mao, Ziyuan Huang, Ao Ma, Yiliang Lv, Yujun Shen, Deli Zhao, and Jingren Zhou. Res-tuning: A flexible and efficient tuning paradigm via unbinding tuner from backbone. orXiv preprint arXiv:2310. 19859, 2023.
Shibo Jie and Zhi- ZhngHong Deng, Convolutional bypasses are better vision transformer adapters. arXiv preprint arXiu:2207.07039, 2022.
Shibo Jie, Haoqing Wang, and Zhi-Hong Deng. Revisiting the parameter efficiency of adapters from the per- spective of precision redundancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17217-17226, 2023.
Feihu Jin, Jiajun Zhang, and Chengqing Zong. Parameter-effiiien tuning for large language model without calculating its gradients. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 321- -300, 2023.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In European Conference on Computer Vision, pp. 105 124. Springer, 2022.
Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Joong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference of I1m. arXio preprinl arXiv:2403.0553 2024.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hyper- complex adapter layers. Advances in Neural Information Processing Systems, 34:1022 1035, 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. anXiv preprint ar xix: 06950, 2017.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognilion, pp. 19113 19122, 2023.
Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv preprint ar Xic:2300. 14152) 2023.
James Kirkpatrick, Razvan Pascamu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barskka-a et al. Overcoming catast forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 3526, 2017.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector- based random matrix adaptation. arXiv preprint (rXf::2810.1144, 2023.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 2011 International conference on computer vision, pp. 2556-2563. IEEE, 2011.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Van Zhu. Multi-concept cus- tomization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 1931-1941, 2023.
34

---

Published in Transactions on Machine Learning Research (10/2024)
Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver Steeg. Neural architec- ture search for parameter-efficient fine-tuning of large pre-trained language models. arXiv:2305.16597 2023.
ar preprinl
Sungjin Lec. Toward continual learning for conversational agents. aixiv preprint arXiv:1712.09943 2017.
Tao Lei, Junwen Bai, Siddhartha Brahma,, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y Zhao, Yuexin Wu, Bo Li, et al. Conditional adapters: Paramcter-efficien transfer learning with fast inference. arXiv preprint arXim:2904.00 049947 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint ar Xrx:21104 086011 2021.
Dengehun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang, Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. arXiv preprint arXiv:2404.11130 2024.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirby-seventh Confer- ence on Neural Information Processing Systems, 2023a.
Jonathan Li, Will Aitken, Rohan Bhambhoria, and Xiaodan Zhu. Prefix propagation: Parameter-efficiet tuning for long sequences, arXiv preprint rrXiu:2305.12060 2023b.
Xiang Lisa Li and Percy Liang. Prefix-tuming: Optimizing contimuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive langunge-image pre-training paradigm. arXiv preprint arXic:2100005: 05.0208, 2021.
Yingji Li, Mengnan Du, Xin Wang, and Ying Wang. Prompt tuning pushes farther, contrastive learning pulls closer: A two-stage approach to miligate social biases. arXiv preprinl axXio:2307.01595, 2023c.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tho Zhao. Loftq: fine-tuning-awar quantization for large language models. arXiv preprint arXiv:2310.08652 2023d.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model luning. Advances in Newral Information Processing Systems, 35:109 123, 2022.
Zujie Liang, Feng Wei, Yin Jie, Yuxi Qian, Zhenghong Hao, and Bing Han. Prompts play lottery lickets well: Achieving lifelong information extraction via lottery prompt tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 277 292, 2023.
Baohao Liao, Yan Meng, and Christof Monz. Parameter-efficient fine-tuning without introducing new latency. arXiv preprint 2rX60:2305.6674, 2023a.
Baohao Liao, Shaomu Tan, and Christof Monz. Make your pre-trained model reversible: From parameter to memory efficient fine-tuning, arXiv preprint ar arXix:2306. 00477, 2023b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, and Hong Mei. Lora dropout as a sparsity regularizer for overfitting control. arXiv preprint ar arXiv:440.009610, 2024.
35

---

Published in Transactions on Machine Learning Research (10/2024)
Ziyi Lin, Shijic Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference on Compuler Vision, pp. 388 404. Springer, 2022.
Haokun Liu, Derek Tam, Mohammed Mugeoth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-effieient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950- 19600-19,,,-01 2022a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. orXiv preprinl arXiv:2304.08450 2023a.
James Liu, Guangxuan Xiao, Kai Li, Jason D Lee, Song Han, Tri Dao, and Tianle Cai. Bitdelta: Your fine-tune may only be worth one bit. arXiv preprint ar Xiv::2022110995 2024a.
Jialin Liu, Antoine Morcau, Mike Preuss, Jeremy Rapin, Baptiste Roziere, Fabicn Teytaud, and Olivier Tey- taud. Versatile black-box optimization. In Proceedings of the 2020 Genetic and Evolutionary Compulation Conference, pp. 620 628, 2020.
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. arXiv preprint arXiv:2310. 18339, 2023b.
Shih- Yang Liu, Chien-Ys Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- -iiang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposeed low-rank adaptation. arXiv preprinl or arXiv:2402.0000 09353, 2024b.
Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, and Xipeng Qiu. Late prompt tuming: A late prompt could be better than many prompts. arXiv preprint ar arXiv:2210. 11292, 2022b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt luning can be comparable to fine-tuning universally across scales and tasks. arXie preprinl 0tlt:2110.076000 2021a.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arxiv preprint arXiv:2103.100 10385, 2021b.
Ximing Lu, Faeze Brahman, Peter West, Jachun Jang, Khyathi Chandu, Abhilasha Ravichander, Lianhui Qin, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, et al. Inference-time policy adapters (ipa): Tailoring extreme-scale lms without fine-tuning. arXiv preprint arXiv:2305.1006 2023.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint arXiv:2311.055 200.55560 2023.
Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song, Xprompt: Exploring the extreme of prompt tuning. arXiv preprint arXio:2210.0443 2022.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4 (3):448-44) 102448-472,, 1992.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eu- njoon Cho, and Zhiguang Wang. Continual learning in task-oriented dialogue systems. arXio preprint ar555 2020.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-officion multi-task fine-tuning for transformers via shared hypernetworks. arxiv preprinl arXiv:2106.044 2021.
Eran Malach, Gilad Ychudai, Shai Shalev-Schwarte and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pp. 6682 2221. PMT.R, 2020.
36

---

Published in Transactions on Machine Learning Research (10/2024)
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. arXiv preprint ar Xiv: 1 17333, 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-tho-art parameter-cffiient finc-tuning methods. https://github. com/ hwtp://gggggggc/////////////// 2022.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-lau Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-effcient language model tuning. orXiv preprint arXiv:211000757) 2021.
Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang, Peiyi Wang, Qingxiu Dong, Liang Chen, and Zhifang Sui. Periodiclora: Breaking the low-rank bottleneck in lora optimization. arXiv preprint arXiv:2402.1614. 2024.
Microsoft. Microsoft azure function trace. http:///hhhhh..oommmmuu/uuuuuuue
2023.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct eloctricity? a new dataset for open book question answering. In EMNLP, 2018.
Ismael Solis Moreno, Peter Garraghan, Paul Townend, and Jie Xu. Analysis, modeling and simulation of workload patterns in large-scale utility cloud. IEEE Transactions Cloud Computing, 2(2):208 221, 2014.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint Xiv:2990 08458, 2023.
Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xiang. Zero-shot temporal action dctection via vision- language prompting. In Europeun Conference on Compuler Vision, pp. 681 697. Springer, 2022.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In European Conference Computer Vision, pp. 1-18. Springer, 2022.
OpenAl. Sharegpt. htlps://sharrepll.com/ 2023a.
OpenAL. Gpt-4. In http::://oppeaa.com//// 4.t-4 2023b.
Emin Orhan. A simple cache model for image recognition. Advances in Newral Information Processing Systems, 31, 2018.
Yawen Ouyang, Yongchang Cao, Yuan Gao, Zhen Wu, Jianbing Zhang, and Xinyu Dai. On prefix-tuning for lightweight out-of ant-of-distribution detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1533 1545, 2023.
Mustafa Safa Ozdayi, Charith Peris, Jack Fitzgerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, and Rahul Gupta. Controlling the extraction of memorized data from large language models via prompt-tuning. arXiv preprint 21 11759, 2023.
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-effcient image- to-video transfer learning. Advances in Neural Information Processing Systems, 35:26462 264477, 2022.
Aleksandar Petrov, Philip HS Torr, and Adel Bibi. When do prompting and prefix-tuning work? a theory of capabilitics and limitations. arXiv preprint 1xXiv:2310.1960 2023.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destruetive task composition for transfer learning. arxiv preprint "Xix:2005.0024"" 2020.
37

---

Published in Transactions on Machine Learning Research (10/2024)
Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. Hypertuning: Toward adapting large language models without back-propaggtion. In International Conference on Machine Learning, pp. 27854-27875 PMLR, 2023.
Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulié, Schastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. Adapters: A unified library for parameter-cffiiint and modular transfer learning, 2023.
George Pu, Anirudh Jain, Jihan Yin, and Russell Kaplan. Empirical analysis of the strengths and weaknesses of peft techniques for llms. arviv preprint arXio:234. 14999, 2023.
Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, al. Exploring universal intrinsic task subspace via prompt tuning. arXiv preprint 2rXiv:111000780 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Hossein Rajabzadeh, Mojtaba Valipour, Tianshu Zhu, Marzieh Tahaei, Hyock Ju Kwon, Ali Ghodsi, Boxing Chen, and Mehdi Rezagholizadeh. Qdylora: Quantized dynamic low-rank adaptation for efficient large language model tuming. arXiv preprint arXiv:2402.104 2024.
Li Ren, Chen Chen, Liqiang Wang, and Kien Hua. Learning semantic proxies from visual prompts for parameter-effiiiet fine-tuning in deep metric learning. arXiv preprint arXiv:2402.02340. 2024.
Steven J Rennie, Ftienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7008-7024, 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-rosolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 10684-10695, 2022.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.1100 119181 2020.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. Dream- booth: Fine tuning text-to-image diffuusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-116 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense
reasoning about social interactions. Xr preprint wxXim:1044009728 2019.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint ar Xix:2231 1..22311.0225, 2023a.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christo- pher Ré, lon Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with single gpu. In International Conference on Machine Learning, pp. 31094 31116. PMLR, 2023b.
Zhengxiang Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-efficient fine-tuning. arXiv preprint ar Xic:2300 05173, 2023.
38

---

Published in Transactions on Machine Learning Research (10/2024)
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274 14289, 2022.
Amanpreet Singh, Ronghang Hu, Vedamuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. orXiv preprinl arXiv:2106. 10270, 2021.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arxic prepriut or arXiv:2104.0090 2021a.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Läu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language processing. arXiv preprint 0iiu:2111.06711 2021b.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24200 2021.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst:: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Newral Information Processing Systems, 25:12991-13001 2022a.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and- language tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogrition, pp. 5227-5237, 2022b.
Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E Gonzalez, and Raluca Ada Popa. Lloco: Learning long contexts offline. arXiv preprint or arxi:22404.000 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family highly capable multimodal models. arXio preprint XrX:::2312. 11805, 2023.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient. tuning of pre-trained models using dynamic scarch-free low-rank adaptation. arXiv preprint XXi:22100007555 2022.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEFE transactions on patlern analysis and machine intelligence, 3955 6630666336220 2016.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint ar arXit:2110.07900 2021.
Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard, James J Clark, Brett H Meyer, and War- ren J Gross. Efficient fine-tnnig of bert models on the edge. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1838-1842. IEEE, 2022.
Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0744) 2018.
39

---

Published in Transactions on Machine Learning Research (10/2024)
Peng Wang, Qi Wu, Chunhua Shen, Authony Dick, and Anton Van Den Hengel. Frqa: Fact-based visual question answering. IEEE transactions on pollern analysis and machine intelligence, 40(10):2413-2422 2017.
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, et al. Aprompt: Attention prompt tuning for efficient adaptation of pre- trained language models. In Proceedings of the 2023 Conference Empirical Melhods in Natural Language Processing, pp. 9147 9160, 2023a.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xu- anjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.1425 14152, 2023b.
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: ada for parameter-cffiient tuning of large language models. arXie preprint arXiv:2205. 12410, 1(2):4, 2022a.
Yihan Wang, Jatin Chauhan, Wei Wang, and Cho-Jui Hsich. Universality and limitations of prompt tuning. arXiv preprint Xrx:22900 18787, 2023c.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 139--49, 2022b.
Ziyi Wang, Xumin Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting. Advances in nearal information processing systems, 35:14358 14402, 2022c.
John Wright, Arvind Ganesh, Shankar Rao, Yigang Peng, and Yi Ma. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. Advances in neural information processing systems, 22, 2009.
Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, and Xin Jin. (dLoRA}: Dynamically orchestrating requests and adapters for (LLM)) serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 911-927, 2024a.
Hui Wu and Xiaodong Shi. Adversarial soft prompt tuning for cross-domain sentiment analysis. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Lánguistics (Volume Long Papers), pp. 2438 2447, 2022.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623-7633, 2023a.
Junda Wu, Tong Yu, Rui Wang, Zhao Song, Ruiyi Zhang, Handong Zhao, Chaochao Lu, Shuai Li, and Ricardo Henao. Infoprompt: infommation-thoordee soft prompt tuning for natural language understanding. arXiv preprint arXiv:2306.04900 2023b.
Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Visual question answering: A survey of methods and datasets. Computer Vision and Image Understanding, 163: 21-40, 2017.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang, Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.0811 2023c.
Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. ofxture-of in low-rank adaptation. arXiv preprint ar X00: 1 1909, 2024b.
40

---

Published in Transactions on Machine Learning Research (10/2024)
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXtt:2404.13028, 2024c.
Zhuoleng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, VG Vydiswaran, and Hao Ma. ldpg: An instance-dependent prompt generation method. arXiv preprint arXi::2204.04497, 2022.
Guangxuan Xiao, Lin, and Song Han. Offsite-tuning: Transfer learning without full model. orXiv preprint arXi::2302.04870, 2023a.
Guangxuan Xiao, Ji Lin, and Song Han. Offsite-tuning: Transfer learning without full model. arXiv preprint arXiu:2302.048770 2023b.
Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu- Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation. arXiv preprint arXiv:2308. 0i0:22008009100, 2023.
Binfeng Xu, Xukun Liu, Hua Shen, Zeyu Han, Yuhan Li, Murong Yue, Zhiyuan Peng, Yuchen Liu, Ziyu Yao, and Dongkuan Xu. Gentopia: collaborative platform for tool-augmented llms. orXiv preprint arXiv:2308.0 0400:2008000000, 2023a.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-effiiiet fine- tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312. 12148, 2023b.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2945 2954, 2023c.
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang, Raise child in large language model: Towards effective and generalizable fine-tuning. or arXiv preprint arXiv:2109 009.002109005687 2021.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-awee low-rank adaptation of large language models. arXiv preprint or 2iv:2309. 14717, 2023d.
Zumnan Xu, Zhihong Chen, Yong Zhang, Yibing Song, Xiang Wan, and Guanbin Li. Bridging vision and language encoders: Parameter-efficient tuning for referring image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17503-17512, 2023c.
Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. arXiv preprint Xrxiv:230.. 18111, 2023a.
Jianxin Yang. Longglora: Efficient and effective method to extend context length of large language models.
arXiv preprint 1tf::2311.0487 2023.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023b.
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. arXie preprint #xXw:2302.03024, 2023c.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertscrini. arXiv preprint Xrx::1192.0011 01718. 2019.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint 2xXii:2308.06721 2023.
Seungryong Yoo, Eunji Kim, Dahuin Jung, Jungbeom Lee, and Sungroh Yoon. Improving visual prompt tuning for self-supervised vision transformers. arXio preprint ar Xiv:2230000056 05067, 2023.
41

---

Published in Transactions on Machine Learning Research (10/2024)
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4651- 4659, 2016.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chich Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. arXiv preprint ar xrx::22308.02. 02488, 2023.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction luning. arXiv preprinl arXiv:2309.0544 05444, 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for Transformerbbssee masked models. arXiv preprint arXiv:2106. 10199, 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Bohan Zeng, Shanglin Li, Yulang Feng, Hong Li, Sicheng Gao, Jiaming Liu, Huaxia Li, Xu Tang, Jianzhuang Liu, and Baochang Zhang. Ipdreamer: aopearance-controlle 3d object generation with image prompts. arXiv preprint xrX::23100005375 05377,, 2023a.
Guangtao Zeng, Peiyuan Zhang, and Wei Lu. One network, many masks: Towards more parameter-efficient transfer learning. arXiv preprinl 1 11029005.76888 17682, 2023b.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. Summit: Iterative text summarization via chatgpt. arXiv preprint ar 21x: 14835, 2023a.
Jeffrey O "Zhnng Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glusyow, UK, August 23 23-28, 2020, Proceedings, Part III 16, pp. 018-711. Springer, 2020.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficienn low-rank adaptation for large language models fine-tuning. arXiv preprint arXi:22308.03.0 033003 2023b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3836-3847, 2023c.
Mingyang Zhang, Chunhua Shen, Zhen Yang. Linlin Ou, Xinyi Yu, Bohan Zhuang, et al. Pruning meets low-rank parameter-efficiet fine-tuning. arXiv preprint or Xin:2905. 18403, 2023d.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao Adaptive budget allocation for parameter-cfficient fine-tuning. arXiv preprint XrXiv:2303. 10512, 2023c.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- shong Li. Tip-adapter: Training-freo clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03990 2021.
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In Proceedings of the IFFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8552-8562, 2022a.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. aix preprint arXiv:2003.10 1619003..1999 99, 2023f.
42

---

Published in Transactions on Machine Learning Research (10/2024)
Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. arXiv preprint arXin:2403.0911 03901133, 2024a.
Sai Qian Zhang, Thierry Tambe, Nestor Cuevas, Gu-Yeon Wei, and David Brooks. Camel: Co-designing ai models and embedded drams for efficient on-device learning. aix preprint 21i:23350001444 2023g.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search, 2022b.
Zhen-Ru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, and Songfang Huang. To- wards adaptive prefix tuning for parameter-effiient language model fine-tuning. arXiv preprint arXiv:2305. 15212, 2023h.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024b.
Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning layernorm in attention: Towards efficient multi-modal IIm finetuning. arXiv preprint 1xXi:2312.114200, 2023a.
Hao Zhao, Jie Fu, and Zhaofeng He. Prototype-based hyperadapter for sample-efficient multi-task tuning. arXiv preprint arXiv:2310. 11670, 2023b.
Haodong Zhao, Ruisang He, Mengnan Xiao, and Jing Xu. Infusing hierarchical guidance into prompt. luning: A parameter-efficint framework for multi-level implicit discourse relation recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6477-6492, 2023c.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficiet Ilm training by gradient low-rank projection. arXiv preprint orXiv:2403.0350) 2024.
Ziwang Zhao, Linmei Hu, Hanyu Zhao, Yingxia Shao, and Yequan Wang. Knowledgeable parameter efficient tuning network for commonsense question answering. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 9051 9051-9063, 2023d.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633 641, 2017.
Han Zhou, Xingchen Wan, Ivan Vulié, and Anna Korhonen. Autopeft: Automatic configuration scarch for paramnteer-lllcient fine-tuning. arXiv preprinl wrXi:2231..2132 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 16816 16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337 2348, 2022b.
Tianyi Zhou and Dacheng Tao. Godec: Randomized low-rank & sparse matrix dccomposition in noisy case. In Proceedings of the 28th International Conference on Machine Learning, ICML 2011, 2011.
Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. (PetS): A unified framework for (Parameter- Effiicintt transformers serving. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 489-504, 2022c.
Beier Zhu, Yulci Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659-15669, 2023a.
43

---

Published in Transactions on Machine Learning Research (10/2024)
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. ar preprint arXim:2304.1668) 2023b.
Qi Zhu, Bing Li, Fei Mi, Xiaoyan Zhu, and Minlie Huang. Continual prompt tuning for dialog state tracking. arXiv preprint ar xrx::2203..000 06654, 2022.
Wei Zhu and Ming Tan. Spt: Learning to selectively insert prompts for better prompt tuning. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11862 11878, 2023.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprinl or Xio:2334. 04675, 2023c.
Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointelip v2: Prompting clip and gpt for powerful 3d open-world learning. In Proceedlings of the IEEE/CVF International Conference on Computer Vision, pp. 2639 2639-2650, 2023d.
Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, and Lei Li. Counter-interferen adapter for multilingual machine translation. arXiv preprint 0rXiv2214400815, 2021.
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools. Xiviv preprint ar 0tl:22300.130,, 2023.
44


Reviews:
[{"review_id": "pl4vU7dDCP", "reviewer": "Reviewer_yYG7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper provides a comprehensive review of Parameter-Efficient Fine-Tuning (PEFT) algorithms, covering additive, selective, reparameterized, and hybrid approaches. Additionally, the authors discuss efficient optimization techniques for deploying PEFT models, as well as some existing system examples. Overall, this survey serves as a valuable resource for understanding and utilizing PEFT techniques in the era of large pre-trained models.\n\nWeakness:\n\n1. Lack of clarity on KV-cache management: In the efficient PEFT design section, it is unclear how to specifically conduct KV-cache management for PEFT tasks. Providing more detailed guidance on this aspect would enhance the practicality of this part.\n2. Limited coverage of recent works: The PEFT serving system parts could be improved by including other recent works, such as dLora (OSDI 2024). This would ensure that the survey remains up-to-date and comprehensive.\n3. Absence of benchmarks: The survey lacks related benchmarks regarding both algorithm and system optimization. Including such benchmarks would provide a more thorough evaluation of the PEFT techniques and systems, allowing readers to better understand their performance and trade-offs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a comprehensive overview of Parameter Efficient Fine-Tuning (PEFT) algorithms for adapting large pre-trained models to specific downstream tasks while minimizing the number of additional parameters and computational resources required.\nThe key contributions of the paper are:\n\n1. It examines the performance and computational overhead of various PEFT algorithms. This helps understand the trade-offs between fine-tuning efficiency and model performance.\n2. It provides an overview of applications developed using different PEFT algorithms. This illustrates the practical utility and versatility of PEFT techniques across domains.\n3. It discusses common techniques employed to mitigate the computational costs of PEFT. \n4. Beyond just the algorithms, it also examines real-world training and serving system designs to investigate the implementation costs of different PEFT approaches, which are crucial for practical deployment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To strengthen the work: \n\n1. Comprehensive benchmarking: To provide a more thorough evaluation, I recommend adding benchmarks for different key algorithms (e.g., various forms of PEFT methods) using diverse metrics. Additionally, benchmarking different training and serving systems for PEFT workloads and comparing their metrics would be beneficial. A more detailed discussion of existing evaluation practices would also enhance the work.\n\n2. Correction in Section 6.4: I noticed that S-Lora and Punica are actually Lora serving frameworks, whereas the section title is \"Parallel PEFT Training Frameworks\". I suggest that the authors correct this inconsistency to ensure clarity and accuracy.\n\n3. Abstraction of challenges and design principles: In the PEFT system parts, I recommend that the authors abstract the challenges and key design principles for PEFT workloads instead of simply listing related works and their methods. This would provide a more insightful and structured discussion, allowing readers to better understand the underlying complexities and design considerations for PEFT systems.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5w6aNzMW2Y", "reviewer": "Reviewer_GqRq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Organization makes sense.\n- Ontology makes sense.\n- Figures are clear and helpful.\n- This is an extremely active area of research, and this manuscript provides a useful high-level overview of the field. \n\nWeaknesses:\n- Grammar is poor. \n- The word choice is poor in some parts. Some sentences are vague.\n- The review skims a lot of papers, and does a reasonable job in organizing the work, but there isn't much in the way of synthesis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript provides a literature review of parameter efficient fine-tuning (PEFT) methods, with a focus on fine tuning large language models (LLMs). It provides an ontology of PEFT methods, summarizes the considerations of PEFT design, provides example applications of PEFT outside of LLMs, summarizes considerations for implementing PEFT in systems, and proposes directions of future research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Run a grammar checker to fix grammar problems.\n2. In abstract, avoid hyperbole: \"This survey serves as an indispensable resource\" \n3. Section 3.1.2: \"researchers directly append\" --> \"researchers directly prepend\"\n4. Section 4.1: \"When we look at autoregressive characteristic, it becomes a major challenge in designing an inference system.\" This is unclear.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fDbjhqrhdx", "reviewer": "Reviewer_H9SJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The structure of the paper is clear and the writing is easy to understand. The review is also comprehensive. \n\nWeaknesses: The authors may double check grammars of this paper, e.g., in page 18, there is a misspelling \"approace\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper conducts a comprehensive review on existing parameter-efficient fine-tuning (PEFT) algorithms for large models. The paper revises basic categories of PEFT algorithms, efficient PEFT designs, and introduces literature of applying PEFT in applications such as LLM, ViT, VLA, and diffusion models. The paper also summarizes existing challenges when designing PEFT.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please proofread the paper to revise the spelling and grammar.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewer GqRq highlighted grammar issues and recommended revising hyperbolic language like "indispensable" to more neutral terms, such as "valuable". Also reviewer H9SJ also mentioned minor spelling and grammar errors that need correction throughout the paper. Please fix these for the final version.
Decision: Accept (poster)