Parameter-efficient Tuning for Large Language Model without Calculating Its Gradients
Authors: Feihu Jin
Jiajun Zhang
Chengqing Zong
Venue: EMNLP 2023 Main (2023)
Keywords: Parameter-efficient Tuning
Large Language Model
Gradient-free

Abstract: Fine-tuning all parameters of large language models (LLMs) requires significant computational resources and is time-consuming. Recent parameter-efficient tuning methods such as Adapter tuning, Prefix tuning, and LoRA allow for updating a small subset of parameters in large language models. However, they can only save approximately 30\% of the training memory requirements, due to the problem that gradient computation and backpropagation are still necessary for these methods. This paper proposes a novel parameter-efficient tuning method for LLMs without calculating their gradients. Leveraging the discernible similarities between the parameter-efficient modules of the same task learned by both large and small language models, we put forward a strategy for transferring the parameter-efficient modules, originally derived from small language models to much larger ones. To ensure a smooth and effective adaptation process, we further introduce a Bridge model to guarantee dimensional consistency while also stimulating a dynamic interaction between the models. We demonstrate the effectiveness of our method using the T5 and GPT-2 series of language models on the SuperGLUE benchmark. Our method achieves comparable performance to both fine-tuning and parameter-efficient tuning on large language models without needing gradient-based optimization. Additionally, our method achieves up to 5.7x memory reduction compared to parameter-efficient tuning.

Parameter-efficien Tuning for Large Language Model without Calculating
Its Gradients
Feihu Jin 1.2 Jiajun Zhangl.2.5 and Chengqing Zong!.
Institute of Automation, Chinese Academy of Sciences, Beijing, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 3Wuhnn Al Research
Shanghai Artificial Intelligence Laboratory, Shanghai, China j jinfeihu20200ia ac. cn {ijzhang, cqzong)enlpr.... ia ac cn
Abstract
75.0
72-
72.5
Fine-tuning all parameters of large language models (LLMs) requires significant computa- tional resources and is time-consuming. Re- cent parameter-efficiet tuning methods such Adapter tuning, Prefix tuning, and LoRA allow updating a small subset of parameters in large language models. However, they can only save approximately 30% of the training memory requirements because gradient compu- still necessary
Accursyy
67.5
65.0
Ran LokN Hes ti
60.0
20
40
60
2.
Hemory Usage
Porameters
(a)
(b)
Figure 1: (a) Comparison of the GPU memory usage during training for fine-tuning, Adapter tuning, LoRA, Prefix tuning, and our method on the RTE task. (b) The cosine similarity of the parameter-efficient modules on the GPT-2 series of language models the RTE task.
tation and backpropagation for these methods. This paper proposes novel parameter-efficien tuning method for L.LMs without calculating their gradients. Leverag- ing the discernible similarities between the parameter-effiiint modules of the same task learned by both large and small language mod- els, we put forward strategy for transferring the parameter-efficient modules derived ini- tially from small language models to much larger ones. To ensure a smooth and effec- tive adaptation process, introduce Bridge model to guarantee dimensional consistency while stimulating dynamic interaction be- tween the models. We demonstrate the effec- tiveness of our method using the T5 and GPT-2 series of language models on the SuperGLUE benchmark. Our method achieves compara- ble performance to fine-tuning and parameter- efficient tuning large language models with- out needing gradient- based optimization. Addi- tionally, our method achieves up to 5.7x mem- ory reduction compared to parameter-efficient tuning.
the sheer number of parameters in these models poses challenges for fine-tuning common hard- ware. Parameter-efficient tuning methods (e.g.. Adapter tuning. Prefix tuning, or LoRA) typically involve adding small number of parameters the language model and only fine-tuning the added subset of parameters, achieving comparable perfor- mance to full fine-tuning. Adapter tuning learns the task-specific information (Houlsby et al., 2019; Mahabadi et al., 2021b,a) by inserting small task- specific modules within layers of the Transformer. Prefix tuning (Li and Liang. 2021; Liu al., 2021b) prepends task-specific trainable prompt tokens the hidden states within every intermediate Trans- former layer. LoRA (Hu et al., 2021) merges the low-rank and trainable matrices with the frozen weights at each layer of the Transformer. In Figure (a), we show the comparison of the GPU memory usage during training for fine-tuning. Adapter tun- ing, LoRA, and Prefix tuning on the Recognizing Textual Entailment (RTE) (Bar-Haim et al., 2014) task. These parameter-efficient tuning methods can save approximately 30% of the GPU memory requirements but still rely on gradient-based opti- mization, resulting in increased memory demands for LLMs.
1 Introduction
Large language models such as GPT3 (Brown et al., 2020), GPT4 (OpenAI, 2023), T5-XXL (Raffel al., 2020), and ILaMA (Touvron al., 2023) have demonstrated remarkable capabilities var- ious natural language processing tasks. However,
Corresponning Author
'The author is currently PhD student in Peking Univer- sity. The new email address is fhjin @stt, puu...uu..dducc

---

To address the above limitations, we propose novel paramelcr-cffiient tuning method for LLMs without calculating their gradients. Intuitively, both large and small language models (SLMs) can learn similar task-specific characteristics when applied to downstream tasks. We conduct experiments using existing parameter-efficient tuning methods on the RTE task in the SuperGLUE benchmark (Wang et al., 2019) validate this hypothesis. As depicted in Figure 1(b), we calculate the similar- ity between the parameter-efficien tuning mod- ules derived from GPT2-XL and GPT2-base or GPT2-medium' Specifically, we apply the LoRA method on GPT2-XL and GPT2-medium to ob- tain the parameter-efficient modules of the two lan- guage models and calculate the cosine similarity of the two serameter-efficicee modules. We find that the cosine similarity can reach up 75%. Our observations indicate that these modules exhibit comparable task-specific characteristics throughout the learning process for specific downstream tasks. Inspired by these findings, if we can successfully transfer the task-specific characteristics learned by the small language model to the large language model, we can enrich the task-specific capabilities into the LLMs without needing gradient-based op- timization.
(Raffel al., 2020) and GPT2 series (Radford al., 2019) of language models assess the effective- ness of our method using the SuperGLUE bench- mark, a widely recognized evaluation benchmark for natural language understanding. The results demonstrate that our method performs par with fine-tuning and parameter-efficient tuning on large language models without needing gradient-based optimization. Additionally, our proposed method achieves up 5.7x memory reduction compared parameter-efficien tuning. Our findings highlight the potential bridging small and large language models, thereby efficiently leveraging expansive large language models. In summary, our key con- tributions can be listed as follows:
Our analysis reveals a substantial task similar- ity when applying parameter-efficient tuning methods to SLMs and LLMs for downstream tasks.
We propose a gradient-free method to adapt the parameter-efficient modules learned in a small language model to large language model.
Extensive experiments on SuperGLUE bench- mark under both T5 series and GPT-2 series of language models verify the effectiveness of our proposed method and achieve up to 5.7x memory reduction compared to parameter- efficient tuning.
In this paper, we first utilize existing parameter- efficient tuning methods in a small language model to learn the task characteristics of downstream tasks. Intuitively, we can directly apply the parameter-efficient module obtained from the small language model to the large language model. How- ever, this would face crucial issues of dimension mismatch and limited interaction with the large language model. To address the issue of dimen- sion mismatch, we employ projection module to align the dimensions of the parameter-efffiieen modules between SLMs and LLMs. Furthermore, to enrich the interaction between the parameter- efficient module and the large language model, we introduce Bridge model that can retain the knowl- edge of the large language model while interacting with the parameter-efficicee module, obtaining parameter-efffiient module with dimensions match- ing the large language model. Finally, we seam- lessly plug the acquired parameter-efficient module into the large language model for inference. We conduct comprchensive experiments on T5 series
2 Method
The proposed method utilizes parameter-efficient tuning modules to effectively bridge small and large language models, enriching the task-specific capabilities into the large language model with- out needing gradient- based optimization. As de- picted in Figure 2, the method consists of training and inference. We first employ parameter-efficient tuning methods during the training stage to learn task-specific characteristics the small language model. Then, we fine-tune the Bridge model and the acquired errrmmttr--fficient module to enhance the knowledge of the parameter-efficieet module. Finally, we directly plug the parameter-efficien module into the large language model during the inference stage for efficient predictions.
Plug-in and Bridge Model Fine-tuning: We first utilize existing parameter-effie efficieen tuning methods such as Adapter (Houlsby et al., 2019), LoRA (Hu
GGPT2-base contains 117M parameters, GPT2-medium contains 345M parameters, and GPT2-XI. contains 1542M parameters.

---

Output
Adapter
SLM
LoRA
W piue
W down
1/r
Bridge Model
1/r
LLM
Prefix
Input
(a) Training
Adapter
LoRA
W
LLM
Prediction
Prefix
Input
(b) Inference
Figure 2: (a) In the training stage, we parameter-efffcient tuning methods learn task-specific characteristics in a smaller model and fine-tune the Bridge model with the acquired parameter-efficien modules. (b) In the inference stage, we directly plug the parameter-efficient modules into the large language model for efficient predictions.
et al., 2021), or Prefix tuning (Li and Liang, 2021) in small language model to learn the task char- acteristics of downstream tasks. However, directly applying the parameter-efficient modules obtained from the small language model to the large lan- guage model would face two issues: dimension mismatch and limited interaction with the large lan- guage model. To address the issue of dimension mismatch, we employ linear projection module W Wiig as the Plug-in model to align the dimen- sions of the parameter-efficient module with the large language model. Furthermore, considering that small language models usually have fewer lay- ers than large language models, we address the layer mismatch by duplicating the layers of the parameter-efficien modules. This duplication en- ables us to achieve layer alignment with the large language model.
tions, including word embeddings, from the large language model f to a lower-dimensions. Bridge model g. with reduction factor of r, where be 8, 16, 32, 64, etc. To retain crucial informa- tion from the large language model, we leverage Fisher information (Liu et al., 2021a: Sung et al., 2021) to prune the parameters of the large language model and obtain the initial Bridge model Fisher information could effectively evaluate the impor- tance of parameters in the large language model. Given the W E Rdd of the backbone network that maps the di-dim vectors the dy-dim space, calculate the importance of each weight vector through
D V log p(u()x)))
W
D
Intuitively, successfully adapting the parameter- efficient modules to large language models re- quires a substantial interaction between them and the large language models. To enrich the inter- action, we introduce a Bridge model that can re- tain the knowledge of the large language model while interacting with the parameter-efficien mod- ule. We employ the pruning method from Ladder- side-tuning (Sung et al., 2022) obtain such Bridge model, which involves pruning each layer the large language model f. We use lincar pro- jections to downsample the intermediate activa-
where (æis Mi) are samples from data D. Then, we keep the rows and columns of the W, which have the de and di importance scores. Through itera- tions of this process in each layer of the Trans- former, we obtain set of weight matrices W E IR x that have undergone pruning 1/r times from the backbone network, and utilize them to initialize the Bridge model.
Subsequently, in order to make the parameter- efficient modules (e.g., Adapter, LoRA, or Prefix tuning) learned in the small language model in- teract with the Bridge model, we apply the linear

---

Algorithm Adaptation of Parameter-efficient Modules for Large Language Model
Require: Large language model M, a small language model S, the training set D { y1).-
: (Tr, }. linear projection modules Wping and Wooms and parameter-effiee tuning methods (PEFT) (e.g., Adapter tuning, Prefix tuning, and LoRA)
1: Apply the PEFT on and fine-tuning on D to obtain parameter-efficien modules
2: Employ the Wing to align the dimensions of the parameter-effiiiet modules with M
3: Prune the parameters of the M and get Bridge model 9
4: Apply the linear projection module W Woown on parameter-efficien modules
5: for each instance (21, y1) in D do
6: Fine-tune the parameter-efficient modules together with the Bridge model g
7: end for
8: Plug the parameter-efficient modules and the linear projection modules Wpiug into the M
projection module W Woown on parameter-efficient modules and fine-tune the parameter-efficient mod- ules together with the Bridge model g. This fine- tuning process enables us to achieve two objectives: obtaining parameter-efficient modules that match the dimensions of the large language model and enriching these modules with knowledge from the large language model.
model and the GPT2-XL as the large model. For the T5 series of models, we classify the T5-base and T5-large as small language models, while the T5-3B and T5-XXL are considered large language models for our experiments. To obtain the Bridge model, we set the reduction factor r = 16. In Table 2, we provide the model parameters of the small language models, large language models, and Bridge models. It is worth noting that the model parameters of the Bridge models significantly smaller than those of the large language models. The training process our proposed method is conducted on an NVIDIA A100 GPU with 80GB of memory.
Inference: Once the training of the parameter- efficient modules and the Bridge model g is com- plete, we integrate the trained parameter-efficient modules, enriched with knowledge from the large language model, into the large language model. This integration empowers the large language model to leverage the task-specific knowledge cap- tured by the purameter-efficient modules during the inference process without requiring gradient-based optimization. The complete algorithm is depicted in Algorithm
demonstrate that the parameter-efficient modules learned on the small language models can be effectively adapted to the large language models, achieving comparable per- formance to full fine-tuning and crrameter-efffcien tuning without needing gradient-based optimiza- tion. We list the baselines as follows:
Our objective
3 Experiments
3.1 Experimental Settings
Fine-tuning:
The vanilla Transformer fine-
tuning.
We conduct extensive experiments eight natural language understanding tasks from the SuperGLUE benchmark, including BoolQ (Clark et al., 2019), CB (De Marneffe al., 2019), COPA (Roemmele et al., 2011), MultiRC (Khashabi et al., 2018), RTE (Bar-Haim et al., 2014) 4), WiC (Pilehvar and Camacho-Collados, 2019), WSC (Levesque et al., 2012), and ReCoRD (Zhang et al., 2018). For each task, we report the accuracy or F1-score. In experiments,
Adapter tuning: Inserting a small task-specific module between the self-attention module (and the MLP module) and the subsequent residual con- nection at each Transformer layer (Houlsby et al., 2019).
Prefix tuning: Adding trainable continuous prompt vectors to the Key and Value components of the attention layer at each layer of the Transformer model (Li and Liang, 2021).
evaluate the effectiveness of our method using both the GPT2 series (Radford al., 2019) of autoregressive language models and the T5 series (Raffel et al., 2020) of sequence-to- sequence language models. In the GPT2 series of models, we designate the GPT2-base as the small
LoRA: Merging the low-rank and trainable ma- trices with the frozen weights at each layer of the Transformer (Hu et al., 2021).
Adapter tuning Plug-in: By applying the Adapter tuning method to small language models,

---

BoolQ
CB
COPA
MultRC
RTE
WIC
WSC
ReCoRD
Method
#Params
Acc.
Acc./F1
Acc.
EM/Fla
Acc.
Acc.
Acc.
Acc./F1
GPT2-buse (Radford al., 2019)
100%
71.2
78.6/55.8
64.4
65.8/17.4
67.8
65.5
72.1/7144
GPT2-base Adapter
71.5
79.3/56.3
65.8
65. 2/122/1.2
67.5
65.7
62.9
72.8/71.9
GPT2 base Prefix tuning
70.6
80.3/58.2
65.1
64.7/16.9
67.3
64.9
63.1
71,4/70.5
GPT2-base LoRA
71.4
79.6/57.8
65.7
66.2/18.6
67.2
65.3
62.5
71.8/70.9
GPT2-XL (Radford al., 2019)
100%
82.4
87.4/90.6
76.4/37.6
79.6
74.9
81.8
84,4/83.8
GPT2-XL Adapter
1.98%
81.8
86.9/90.8
76.2
75.4/36.2
78.9
74.2
81.5
84.2/83.1
GPT2-XL Prefix tuning
1.76%
81.0
86.7/89.
74.9
76.7/36.9
79.2
74.3
81.4
83.2/83.1
GPT2-XL LoRA
1.55%
82.1
87.1/90.3
76.2
76.2/37.1
79.4
75.1
81.1
84,4/83.3
GPT2-XL Adapter Plug-in
0%
81.6
86. 86.7/90.4
75.6
75.8/36.7
78.7
74.3
80.8
83.7/82.5
GPT2-XL Prefix tuning Plug-in
0%
81.2
86.3/89.4
75.1
76.1/36.4
78.5
73.8
80.1
83.5/82.6
GPT2-XL LoRA Plug-in
0%
81.5
86.2/89.3
75.3
75.6/36.8
78.3
74.1
80.3
83,9/82.8
Table I: Results on the SuperGLUUF benchmark. The Adapter Plug-in, Prefix tuning Plug-in, and LoRA Plug-in are the parameter-effictt modules learned in the small language models adapted to the large language models. 0% means we do not update any parameter within the large language models.
we obtain parameter-efficient modules that can be adapted large language models.
parameter-efficient tuning methods applied the large language model, our method demonstrates comparable performance without the need to fine- tune any parameters of the large language model. Specifically, it exhibits slight improvement com- pared to the prefix-tuning method on the BoolQ and COPA tasks, demonstrating the effectiveness of our method.
Prefix tuning Plug-in: By applying the Prefix tuning method to small language models, we obtain parfmeter-efficienl modules that can be adapted to large language models.
LoRA Plug-in: By applying the LoRA method small language models, obtain parameter- efficient modules that can be adapted to large lan- guage models.
3.2.2 Experiments on T5 Series of Models
Table 3 and Table display the performance of our proposed method using the T5 series of mod- els (Raffel et al., 2020). In Table 3, we present the results obtained with T5-base as the small lan- guage model and T5-3B as the large language model. The Adapter Plug-in, Prefix tuning Plug-in, and LoRA Plug-in are parameter-efficien modules of our method used to adapt T5-3B. Our method achieves comparable results on all eight Super- GLUE tasks without the need to fine-tune any pa- rameters of the T5-3B, demonstrating the effective- of our method. Particularly in BoolQ, CB, RTE, and ReCoRD, our method performs level similar full fine-tuning. Similarly, Table shows the results using T5-large as the small language model and T5-XXL as the large language model, with the same Plug-ins employed to adapt T5-XXL. Our method achieves comparable results on all eight SuperGLUE tasks without the need to fine- tune any parameters of the T5-XXL. Furthermore, compared parameter-efficient tuning methods applied to the large language model, our method demonstrates comparable performance without the need to fine-tune any parameters of the large lan- guage model. Especially when compared to the
Models
SLMs
LLMs
BMs
GPT2-base and GPT2-XL
117M
1542M
96M
T5-hase and T5-3B
220M
2800M
175M
T5-large and T5-XXL.
770M
11000M
688M
Table 2: Comparison of the model parameters on small language models (SLMs), large language models (LLMs), and the Bridge models (BMs). To obtain the Bridge model, we set the reduction factor 16.
3.2 Main Results
3.2.1 Experiments on GPT-2 Series of Models
Table shows the performance of our proposed method using the GPT2 series of models (Radford et al., 2019). In Table 1, we present the results obtained with GPT2-base as the small language model and GPT2-XL as the large language model. The Adapter Plug-in, Prefix tuning Plug-in, and LoRA Plug-in are the parffitte-efffccientt modules learned in the GPT2-base model adapted to the GPT2-XL model. As can be seen, in compari- son to directly conducting vanilla fine-tuning on the large language model, our method achieves comparable results. Furthermore, compared to

---

BoolQ
CB
COPA
MultRCC
RTE
WIC
WSC
ReCoRD
Method
#Params
Acc.
Acc./F1
Acc.
EM/F1a
Acc.
Acc.
Acc.
Acc./F1
15-base FT (Raffel et al., 2020)
100%
81.4
81,4/91.0
71.2
79.7/43.1
81.5
68.3
80.8
75.0/74.2
TS-base Adapter
81.5
82.6/93.5
71.5
79.3/43.2
81.2
68.8
80.3
75.2/74.6
T5-base Prefix tuning
81.1
81.9/93.2
70.1
79,2/42.1
80.8
67.9
79.9
74.6/73.5
T5-base LoRA
81.7
82.1/94.3
70.4
79.5/42.6
81.2
69.1
80.5
74.8/74.1
15-3B FT (Raffel al., 2020)
100%
89.9
90.3/94.4
86.8/58.3
90.7
72.1
90.4
91.2/90.4
T5-3B Adapter
2.73%
89.7
90.1/93.2
91.5
86,9/59.1
91.1
71.3
90.5
91.1/9003
T5-3B Prefix tuning
2.12%
89.4
89.0/93.4
91.1
86.4/57.1
90.5
70.9
89.1
90.6/89.2
T5-3B LoRA
2.45%
90.4
90.8/95.0
92.1
86.2/57.6
91.0
71.8
90.6
91.3/91.1
T5-3B Adapter Plug-in
0%
89.5
89.6/94.7
91.7
86.5/58.2
90.8
70.9
90.2
91.6/90.7
T5-3B Prefix tuning Plug-in
0%
89.1
89.4/93.8
91.2
86.1/57.4
90.4
70.7
89.5
90.3/89.9
T5-3B LoRA Plug-in
0%
89.6
90.0/94,9
91.8
86.9/58.5
90.1
71.0
89.8
90.9/90.1
Table 3: Results on the SuperGLUEE benchmark. The Adapter Plug-in, Prefix tuning Plug-in, and LoRA Plug-in are the parameter-efficient modules learned in the small language models adapted to the large language models. The full fine-tume fine-tuning performance of T5-base and T5-3B models is based on the research conducted by Raffel et al. (2020). 0% means we do not update any parameter within the large language models.
BoolQ
CB
COPA
MultiCC
RTE
WiC
WSC
ReCoRD
Method
#Params
Acc.
Acc./F1
Acc.
EM/Fla
Acc.
Acc.
Acc.
Acc./F1
T5-large FT (Raffel al., 2020)
100%
85.4
91.6/94.8
83.4
83.3/50.7
87.8
69.3
86.3
86.8/85.9
T5-large Adapter
85.3
90.9/93.7
83.2
83.1/50.1
87.3
69.8
86.7
86.2/85.1
TS-Jarge Prefix tuning
84.9
91 91/4/55.3
82.9
82.7/49.6
87.5
68.9
85.5
86. 1/84.9
T5-Iarge LoRA
85.1
90.9/94.3
84.2
83.6/51.2
87.3
70.2
85.9
7/857
T5-XXL FT (Raffel et al., 2020)
100%
91.2
93.9/96.8
94.8
88.1/63.3
92.5
76.9
93.8
94.1/93.4
TS-XXL Adapter
6,42%
90.8
93.6/95.9
93.7
87.9/63.0
92.8
92.8/91.7
T5-XXL Prefix tuning
6.22%
89.3
93.5/94.4
92.7
86.9/63.1
92.3
74.9
92.8
93.1/91.8
T5-XXI. LoRA
6.15%
91.5
93,4//95.8
93.6
87.8/63.5
92.7
75.7
93.6
93.7/92.8
T5-XXL Adapter Plug-in
0%
90.4
93.1/95.6
94.1
87.2/62.8
92.3
75.2
93.2
93,4/92.2
T5-XXI. Prefix tuning Plug-in
0%
89.9
93.2/94.6
92.3
86.3/62.1
91.8
74.7
92.4
93.5/92.5
T5-XXL LoRA Plug-in
0%
90.5
92.4//95.7
93.2
86.9/62.5
92.1
74.9
92.9
93.8/92.7
Table Results on the SuperGL benchmark. The Adapter Plug-in, Prefix tuning Plug-in, and LoRA Plug-in are the parameter-efficien modules learned in the small language models adapted the large language models. The full fine-tuning performance of TS-large and T5-XXL models is based on the research conducted by Raffel et al. (2020). 0% means we do not update any parameter within the large language models.
prefix-tuning method, our proposed method shows a slight improvement in the CB, COPA, and WSC tasks when applicd to the T5-3B model. Similarly, when applied to the T5-XXI model, our method slightly improves the BoolQ, CB, and ReCoRD tasks.
rameters of large language models, which suggests the potential for application to even larger language models.
3.2.3 Importance of the Reduction Factor r
Considering the impact of the reduction factor r on the amount knowledge retained in the Bridge model, we conduct experiments to analyze its im- portance. For verification, we select CB, RTE, and WiC in the SuperGLUE benchmark. As indi- cated in Table 5, we observe gradual decrease in model performance
is evident that as the parameter size of pre- trained language models grows, existing parameter- efficient tuning methods necessitate the addition of more parameters to the large language model. In contrast, method does not entail any parameter augmentation but instead optimally harnesses the capabilities of the large language model through a plug-in approach. These findings indicate that our method can effectively utilize the knowledge large language models without updating the pa-
increases. This is because higher value of reduces the retained knowl- edge from the large language model to the Bridge model. According to Table 5, we can observe that when the model parameters of the Bridge model

---

Dataset
r=2
r=4
r=16
r=32
1-6
applied existing large language models without compromising performance. Furthermore, our pro- posed method does not slow down the inference speed of the model. By utilizing the plug-in ap- proach, we can directly leverage the knowledge the large language model during inference without compromising the speed.
CB
91.2
90.6
90.2
90.0
88.2
85.4
RTE
90.8
90.5
90.5
90.1
89.1
84.3
WiC
71.3
71.5
71.1
71.0
69.7
67.4
Table 5: The accuracy on CB, RTE, and WiC with dif- ferent reduction factors We perform the verification on T5-3B using the LoRA Plug-in approach.
Models
MU
MR
are significantly smaller than that of the small lan- guage model, our method exhibits a noticeable per- formance decline during the entire inference pro- cess. However, when the model parameters of the Bridge model comparable that of the small language model, our method maintains the perfor- mance of the model without substantial degrada- tion. To strike a balance between model perfor- mance and the parameter of the Bridge model, we believe that selecting = 16 is suitable choice.
GPT2-XI FT
73746
1.0x
GPT2-XL Adapter
57903
1.3
GPT2-XL Prefix wning
56051
1.3 x
GPT2-XL LoRA
54593
1,4x
Ladder-side Tuning
17652
4.2x
GPT2-XL Adapter Plug-in
11203
6.6x
GPT2-XL Preńix tuning Plug-in
10745
6.9
GPT2-XL LoRA Plug-in
10443
7.1x
Table 6: Memory Usage (MU) Memory Reduction (MR) compared to vanilla fine-tuning of our proposed method on a single NVIDIA A A100 GPU with 80GB of memory. Batch sizes are and sequence lengths are 512.
3.2.4 Memory Usage
Our method not only achieves comparable perfor- mance to full fine-tuning and parameter-efficient tuning methods without updating any parameters of the large language models but also significantly achieves an impressive reduction in memory us-
Models
MU
MR
TS-3B FT
77465
1.0 x
T5-3B Adapter
54324
1.4x
age. As shown in Table 6 and Table we compare the memory usage between our proposed method and the baseline models. When using the GPT2-XL model, we conduct experiments with a batch size of 8 and sequence length 512. Our method evidently achieves up 7.1x memory savings compared to vanilla fine-tuning. Similarly, when utilizing the T5-3B model with batch size of and a sequence length 512. our method achieves up to 5.1 mem- ory savings compared to vanilla fine-tuning.
T5-3B Prefix tuning
53324
1.5x
TS-3B LoRA
52134
1,5x
Ladder-side Tuning
20385
3.8x
T5-3B Adapter Plug-in
15432
5.0x
TS-3B Prefix tuning Plug-in
15643
5.0x
TS-3B LoRA Plug-in
15323
5.1x
Table 7: Memory Usage (MU) and Memory Reduction (MR) compared to vanilla fine-tuning of our proposed method on a single NVIDIA A A100 GPU with 80GB of memory. Batch sizes are 1 and sequence lengths are 512.
In particular, we compare the existing parameter- efficient tuning methods, and in GPT2-XL, we can that proposed method achieve signif- icant memory savings. For example, comparing the three parameter-efficient tuning methods of Adapter-tuning, Prefix tuning, and LoRA, our pro- posed method can achieve 5.3x, 5.6x, and 5.7x memory reduction, respectively. Our method also achieves 2.9x more memory savings compared to Ladder-side Tuning. In T5-3B, our proposed method can achieve 3.6x, 3.5x, and 3.6x memory reduction, respectively, compared to the Adapter- tuning, Prefix tuning, and LoRA. Similarly, our method also achieves 1.3x more memory savings compared to Ladder-side Tuning. This demon- strates that our method can be more effectively
3.3 Utilize Bridge Model Directly?
A natural question arises: Why learn the parameter-efficient modules directly on the Bridge model instead of using a smaller language model and then applying the learned efficient modules to the larger language model? Intuitively, given the learned parameter-efffcient modules with the Bridge model, need projection model to project the dimensions match the large language model. However, the projection model can only be well learned with another interaction model. In this section, we train the Bridge model by di- rectly initializing the parameter efffiiien and lin-

---

Method
COPA
RTE
WiC
WSC
such as those proposed by Houlsby al. (2019) and Mahabadi et al. (2021b,a), aim to learn task- specific information by incorporating small-scale task-specific modules into the layers of the Trans- former. Prefix tuning methods, as introduced by Li and Liang (2021) and discussed in the work by Liu et al. (2021b), also introduce additional parameters within the Transformer layers. LoRA, proposed by Hu et al. (2021), merges low-rank and trainable matrices with the frozen weights at each layer of the Transformer. BitFit (Ben Zaken et al., 2022) is a simple yet effective method that optimizes the bias terms within a model while keeping the other parameters frozen. However, these existing parameter-efficient tuming methods typically rely on pradientbbased optimization and still involve substantial memory usage. Our proposed method, on the other hand, enables substantial memory sav- ings while maintaining comparable performance. Gradient-free Optimization: Recently, Sung et al. (2022) introduced method that eliminates the need for gradient updates by directly applying pruned model to downstream tasks, while the method does not fully exploit the knowledge of the large language model. Xiao et al. (2023) propose an efficient transfer learning framework that can adapt large language models to downstream tasks without access to full model parameters, while the method through compute intensive distillation tech- niques may be cost-prohibitive for larger models. In contrast, our proposed method enables further utilization of the knowledge contained in the large language model and enables significant memory savings through simple operation while preserv- ing the performance of large language models.
Adapter Plug-in (BM)
90.3
89.9
69.4
89.3
LoRA Plug-in (BM)
90.7
89.8
70.1
89,4
Adapter Plug-in (SLM)
91.7
90.8
70.9
90.2
LoRA Plug-in (SLM)
91.8
90.1
71.0
89.8
Table 8: The Accuracy on COPA, RTE, WiC, and WSC tasks with T5-3B. BM means the Bridge model obtained with T5-3B. SLM means the small language model T5- base.
ear projection modules and investigate whether a Bridge model is enough to learn the projection module. In Table 8, conduct experiments using the T5-3B model the COPA, RTE, WiC, and WSC tasks. By plugging the parameter-officient modules learned from the Bridge model and the small language model into the T5-3B model, re- spectively, and comparing their performance, we find that directly using the Bridge model consis- tently performs worse than utilizing the small lan- guage model enriched by Bridge model across all tasks. The results indicate that the Bridge model cannot be utilized alone.
4 Related Work
Fine-tuning and Parameter-efficient Tuning: Large language models leverage parameterized Transformers as foundational framework and train them extensive unsupervised language cor- pora (Brown et al., 2020; OpenAl, 2023; Touvron al., 2023; Raffel et al., 2020). Subsequently, specific task objectives are introduced for down- stream tasks to perform full fine-tuning on the pre- trained language models (Jin et al., 2022, 2023; Muennighoff et al., 2022). As the primary method for optimizing pre-trained language models, full fine-tuning involves initializing the model with pre- trained weights, updating all model parameters, and storing a separate fully optimized model for each downstream task. However, the model parame- ter grows, conducting full fine-tuning on existing computational devices becomes increasingly chal- lenging. In light of this, researchers have started exploring efficient methods for effectively hamness- ing the power of large language models.
5 Conclusion
This paper proposes novel parameter-efficient tuning method for large language models with- out calculating their gradients. We first learn the parameter-efficient tuning module for small language models. Then, the learned parameter- efficient tuning module is adapted into large lan- guage models with a bridge model that handles the dimensionality mismatch and enables interaction between the parameter-efficiee tuning module and the large language model. Extensive experiments on the SuperGLUE benchmark demonstrate that our method achieves comparable performance vanilla fine-tuning and parameter-efficceet tuning on large language models without needing gradient-
As an efficient alternative, pfrfiiter-efffciennt tuning a promising way of stimulating large lan- guage models. Compared vanilla fine-tuning, parameter-efficient tuning methods only tune a small portion of the model parameters while keep- ing the rest frozen. Adapter tuning methods,

---

based optimization. We believe that our method offers a potential direction utilize large language models efficiently and economically.
Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NewrIPS.
Christopher Clark, Kenton Lee, Ming-Wo Miing-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-29522 Minneapolis, Min- nesota. Association for Computational Linguistics.
Limitations
For large language models for which weights can- not be obtained, the proposed methods cannot be directly applicd, and there may be certain limi- tations when applying the proposed methods to language models with different architectures. Con- tinuous exploration and research will be conducted to determine how to apply the proposed methods different architecture language models, aiming improve their compatibility and effectiveness across different architectures and language model characteristics.
Marie-Catherinn De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung. 2372::07-124.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, 2019. Parameter-effiiient transfer learning for nlp. In ICML.
The proposed method still updates the parame- ters of PEFT modules based on the gradient on a small language model, and the whole process is pipeline process, which requires practical training in the early stage before it can be applied to the large model.
Edward Hu. Yelong Shen, Phil Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Feihu Jin, Jinliang Lu, and Jiajun Zhang. 2023. Unified prompt learning makes pre-trained language models better few-shot learners. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
Acknowledgement
This work is supported by National Key R&D Pro- gram of China 20222D0160602 and the Natural Science Foundation of China 62122088.
Feihu Jin, Jinliang Lu, Jiajun Zhang, and Chengqing Zong. 2022. Instance-awan aware prompt learning for language understanding and generation. ArXiv, abs/2201.07122
References
Roy Bar-Haim, Ido Dagan, and Idan Szpektor. 2014. Benchmarking applied semantic inference: The PAS-
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018 Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I [Long Pa- pers), pages 252-262, New Orleans, Louisiana. As- sociation for Computational Linguistics.
CAL recognising textual entailment challenges. In Language, Culture, Computation. Computing- The- ory and Technology Essays Dedicated to Yaacov Choueka the Occasion of His 75th Birthday, Part I, volume 8001 of Lecture Notey in Computer Science, pages 409-424. Springer.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple efffiiter-efficientt fine-tuning for transformer-based masked language-models In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Popers), pages 1-9, Dublin, Ireland. Associa- tion for Computational Linguistics.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Interna- tional Conference on the Principles of Knowledge Representation and Reasoning.
Xiang Lisa Li and Percy Liang, 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jingliang Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. 2021a. Group fisher pruming for practical network compression. International Conference on Ma- chine Learning

---

dataset for evalu- Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021b. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/21 10.07602.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971.
Rabeeh Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 202 Compacter: Efficcentt low-rank hypercomplex adapter layers. In NeurIPS.
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In NeurIPS.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021b. Parameter- efficient multi-tass task fine-tuning for transformers via shared hypernetworks. In ACL.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Rose Biderman, Teven Le Scao, Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir R. Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff. and Colin Raffel. 2022. Crosslingual gen- eralization through multitask finetuning. ArXiv, abs/2211.01786.
Guangxuan Xiao, Ji Lin, and Song Han. 2023. Offsite- tuning: Transfer learning without full model. CoRR, abs/2302.04870.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. CoRR. abs/ 10. 1885..
OpenAL. 2023. Gpt-4 technical report. ArXiv. abs/2303.08774.
Mohammad Taher Pilchvar and Jose Camacho-Collados, 2019. WiC: the ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 1267-1273, Minncapolis, Minnesota, Association for Computa- tional Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, (88).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-texx trans- former. J. Mach. Learn. Res., 21(1).
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In Logical Formalizations of Commonsense Reasoning, Pupers from the 2011 AAAI Spring Sym- posium, Technical Report SS-1 1-06, Stanford, Cali- fornia, USA, March 21-23, 2011. AAAI.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. LST: Ladder side-tuning for parameter and memory efficient transfer learning. In Advances in Neural Information Processing Systems.
Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Train- ing neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems.


Reviews:
[{"review_id": "M6vwi9gHgk", "reviewer": "Reviewer_djcv", "summary": "", "questions": "1. When do you train W_{plug}  and W_{down}? Also in Line 6 of Algorithm 1?\n\n2. Line 294~297: This claim is not true. Out of 24 experiments (3 PEFT methods for 8 tasks), only two experiments are better. \n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We address your concerns one by one as follows.  \nQ1: The title and the way to show results are misleading...，  \n(1) About the title, the fact is that our proposed method fine-tunes the PEFT modules on a much smaller model before applying it to the larger model, without further training the large model. That is we did not update any parameters on the large model. Hence, the title is expected to convey the meaning that no gradient updates are conducted on the large model.  \n(2) The utilization of 0% for #Params indeed seems misleading. We just want to say that the incorporation of PEFT modules into the larger model does not introduce any additional trainable parameters for the larger model. We will follow your suggestion to use another term.  \n(3) As the purpose of the small model is to learn parameter-efficient modules tailored for the large model, our primary concern centers around optimizing memory efficiency while maintaining strong performance in the large model. Since our method aims to enhance the utilization of the large model, we focus more on comparing the memory usage of the large model.  \nQ2: All results are shown without an error bar. SuperGLUE tasks are very sensitive to hyper-parameter settings. It's better to show the error bar (like variance) for the main results...  \nThanks for your reminding. We notice similar variance and we will supplement the information in the revised version.  \nQ3: The contribution is limited, mainly combining PEFT and pruning methods together to reduce the memory footprint for training...  \nAs for the contributions, we believe our method will much benefit the parameter-efficient tuning for very large models. Specifically, we first reveal a substantial task similarity when applying parameter-efficient tuning methods to both SLMs and LLMs for downstream tasks. Then we train the PEFT module with the bridge model and employ a linear projection layer to overcome the dimensionality mismatch between the SLM and LLM. Finally, our proposed method achieves up to 5.7× memory reduction compared to parameter-efficient tuning. We believe that the speedup is promising.  \nQ4: When do you train W_{plug} and W_{down}? Also in Line 6 of Algorithm 1?  \nAs shown in Algorithm 1, we train W_{plug} and W_{down} with the bridge model together.  \nQ5: Grammar Style, Presentation Improvements, and References.  \nWe will revise any inappropriate phrasing and address usage issues accordingly. Additionally, we will incorporate the appropriate references as you suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method could achieve comparable results as the one directly fine-tuning the larger language model, saving a large amount of memory footprint.\n\n2. Extensive experiments are conducted to support the main claim.", "reasons_to_reject": "1. The title and the way to show results are misleading: \n(1) The proposed method still updates the parameters of PEFT modules based on the gradient. Instead of fine-tuning the PEFT module directly on the larger language model, the proposed method fine-tunes the PEFT module on a smaller model and applies it directly to the larger language model without further training. But it doesn't mean that it's gradient-free.\n(2) In all tables with #Params, the authors use 0% for the proposed methods. In all related works, #Params means the number of trainable/new parameters. It's not 0% for your work in this case. If you want to highlight that you don't need to further train the PEFT modules when plugging into the larger model, you need to specify it rather than use it in this way.\n(3) In all performance result tables (Table 1, 3, 4), the authors include the results from the smaller language model. However, when comparing the memory footprint for training, all baselines for the smaller language model are excluded. \n\n2. Experimental settings are missing, like the grid search space for learning rate, batch size and etc. All results are shown without an error bar. SuperGLUE tasks are very sensitive to hyper-parameter settings. It's better to show the error bar (like variance) for the main results. \n\n3. The writing is not easy to follow. And the contribution is limited, mainly combining PEFT and pruning methods together to reduce the memory footprint for training.", "questions_to_authors": "1. When do you train W_{plug}  and W_{down}? Also in Line 6 of Algorithm 1?\n\n2. Line 294~297: This claim is not true. Out of 24 experiments (3 PEFT methods for 8 tasks), only two experiments are better. \n", "paper_topic_and_main_contributions": "This paper is about reducing the memory footprint when fine-tuning a large language model. The authors propose to fine-tune the parameter-efficient fine-tuning (PEFT) modules on a smaller pre-trained model, then project these modules to the same dimension of the larger language model by further training the projection matrix on a pruned variant of the larger language model and finally plug these leaned modules to the larger language model.\n\nThe main contributions of this paper are:\n(1) The proposed method reduces the memory footprint significantly, while slightly sacrificing some scores.\n(2) Extensive experiments are conducted on two series of models (GPT and T5) and SuperGLUE tasks.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Alrogrithm 1: PETM -> PEFT\n\n2. Better to add a column about the average score to all tables. "}, "extra_text": {"Missing_References": "In related work, it's better to summarize some related works to rescue training memory, like:\n\n[1] Dettmers, Tim, et al. \"Qlora: Efficient finetuning of quantized llms.\"\n\n[2] Liu, Yitao, Chenxin An, and Xipeng Qiu. \"$\\mathcal {Y} $-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning.\n\n[3] Liao, Baohao, Shaomu Tan, and Christof Monz. \"Make Your Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning.\""}}, {"review_id": "YEOUdo7AQb", "reviewer": "Reviewer_wTP7", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging comments. We address your concerns as follows.  \nQ1: The pictures and tables in the article can be further refined.  \nThanks for your suggestion. In the revision, we will make the pictures and tables much more clear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Excessive memory usage is a typical problem during fine-tuning or parameter-efficient tuning. It is an interesting idea to combine both SLM and LLM, and just calculate the gradient of SLM during training. \n\n2. The proposed method achieves comparable performance while significantly reduces memory usage, and may promote the application of large language models in downstream tasks.\n\n3. Solid experiments and detailed analysis are carried out in the paper. ", "reasons_to_reject": "The pictures and tables in the article can be further refined. \n\n1. The legend in Figure 1 could be reordered so that it appears in the same order as it appears in the figure.\n\n2. The experimental results of the proposed method in Tables 1, 3, and 4 could be highlighted, for example, the results that are less than 1 point lower than the corresponding parameter-efficient methods could be underlined.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides a novel parameter efficient tuning method for large language models.\nThis method learns a parameter efficient module of a small language model, and then adapt this module to the large language model using a linear projection module and a bridge model.\nThe experiment shows that this method achieves significant memory reduction during training, and achieves comparable performances in various tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "duZ1vMRBon", "reviewer": "Reviewer_ZcTW", "summary": "", "questions": "1. Can you provide the mentioned baseline performance in the weakness section?\n\n2. Please explain why '0% #params' is indicated here. \n\nIf the author could address my concerns, I will increase my scores.\n\n### After Rebuttal \n\nThanks the author for providing additional evidence for a comparison to the LST baseline. I have increased the soundness score.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We address your concerns one by one as follows.  \nQ1: Can you provide the mentioned baseline performance in the weakness section?  \nIn the T5-3B model, we observe that LST achieves 2.4× more memory savings compared to Adapter and LoRA, in line with the findings of the original paper. Moreover, our method outperforms LST with a substantial memory saving of 3.5× and 3.6× over Adapter and LoRA, respectively. When analyzing the BBT and BBTv2 methods, their implementation like prefix-tuning on GPT2-XL and T5-3B models introduces additional parameters, leading to heightened memory consumption and diminished inference speed. For instance, in the T5-3B model, BBT incurs a memory usage of approximately 19473MB, whereas BBTv2 utilizes around 22749MB. In contrast, our proposed method showcases a memory requirement of approximately 15643MB, affirming its efficiency in memory management.  \nQ2: Please explain why '0% #params' is indicated here.    \n'0% #params' is expected to show that we do not update any parameter within the larger model. We will also provide accurate and transparent information about the total trainable parameters and the specific parameters that are preserved during the pipeline, rather than using only '0%'.   \nQ3: The description in Lines 396-398 is incorrect. Only the LoRA Plug-in does not slow the inference speed.   \nThanks for your reminding. We will refine the corresponding wording in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Transferring the PEFT learned from the SLM to LLM is a great idea due to the cost of FT LLM.\n2. This work shows significantly reduced memory usage and reasonable performance degradation. \n3. It conducts experiments on two types of backbones with convincing results.", "reasons_to_reject": "1. Important baselines are missing. The author did cite LST but there is no comparison between the performance of LST and the proposed method, where LST also shows significantly reduced memory but competitive performance to FFT. It is important that other black-box forward-only (without calculating gradients) baselines that also reduce memory costs are missing, including but not limited to BBT, BBTv2, MeZo, PromptBoosting.\n2. '0% #params' descriptions in Table 1, 3, 4 are misleading. The author should clearly state the total trainable parameters and the parameter required to be saved during the pipeline instead of stating 0%.\n3. The description in Lines 396-398 is incorrect. Only LoRA Plug-in does not slow the inference speed.", "questions_to_authors": "1. Can you provide the mentioned baseline performance in the weakness section?\n\n2. Please explain why '0% #params' is indicated here. \n\nIf the author could address my concerns, I will increase my scores.\n\n### After Rebuttal \n\nThanks the author for providing additional evidence for a comparison to the LST baseline. I have increased the soundness score.", "paper_topic_and_main_contributions": "This work presents a memory-efficient tuning pipeline by first training the PEFT module on SML. With an interaction of a Bridge model that is pruned from the LLM, this work further trains the PEFT module with the bridge model. It also employs a linear projection layer to overcome the dimensionality mismatch between the SLM and LLM. It shows consistent results with up to 5 times memory footprint reduction on T5 and GPT-2 models. However, this work only considers baselines with naive PEFT modules, where the comparison to important baselines like LST, BBT, BBTv2 is missing. This raises my concerns about the solidness of this work.\n\nSung, Y.-L., Cho, J., & Bansal, M. (2022). LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Sun, T., Shao, Y., Qian, H., Huang, X., & Qiu, X. (2022). Black-Box Tuning for Language-Model-as-a-Service (arXiv:2201.03514).\n\nSun, T., He, Z., Qian, H., Zhou, Y., Huang, X., & Qiu, X. (2022). BBTv2: Towards a Gradient-Free Future with Large Language Models \n\nHou, B., O’Connor, J., Andreas, J., Chang, S., & Zhang, Y. (2022). PromptBoosting: Black-Box Text Classification with Ten Forward Passes (arXiv:2212.09257).\n\nMalladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., & Arora, S. (2023). Fine-Tuning Language Models with Just Forward Passes (arXiv:2305.17333)."}}]

Author rebuttal:

Meta-review: This paper proposes a new parameter- and memory-efficient fine-tuning method that shows significant memory saving compared to other PEFT methods while achieving competitive performance. Concretely, the paper proposes first fine-tuning PEFT modules in a small language model learning task characteristics, then fine-tuning the resulting PEFT models with up/down projections (plug-in module) and a bridge model that is pruned from the large language model. For the inference, the final PEFT modules are inserted into large LM with plug-in up projection without further fine-tuning. The authors presented extensive experiments that include 2 model families (GPT, T5) at different sizes that are evaluated on SuperGLUE benchmark. 

As the reviewer mentioned, the proposed method consistently achieves comparable performance with full fine-tuning large model saving a significant amount of memory footprint. Compared to standard PEFTs, the proposed method achieves up to 5.7x memory reduction. The experimental setup is sound and the results are convincing. 

However, also the reviewers noted, that phrasing and the number of updated parameters in tables (as 0%) may mislead the conclusion. PEFT modules are updated but using small LMs (and the bridge model) which leads to significant memory reduction. I believe rephrasing or clarification highlighting this point will improve the paper. Connected to this point, the authors acknowledged the issue with the updated parameter number in the tables.

Furthermore, the authors provided an additional comparison with another memory-efficient baseline based on the reviewer's suggestion during the discussion period which fill the corresponding gap.
Decision: Accept-Main