LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Authors: Yi-Lin Sung
Jaemin Cho
Mohit Bansal
Venue: NeurIPS 2022 Accept (2022)
Keywords: parameter-efficient transfer learning
memory-efficient learning
transfer learning

Abstract: Fine-tuning large pre-trained models on downstream tasks has been adopted in a variety of domains recently. However, it is costly to update the entire parameter set of large pre-trained models. Although recently proposed parameter-efficient transfer learning (PETL) techniques allow updating a small subset of parameters (e.g. only using 2% of parameters) inside a pre-trained backbone network for a new task, they only reduce the training memory requirement by up to 30%. This is because the gradient computation for the trainable parameters still requires back-propagation through the large pre-trained backbone model. To address this, we propose Ladder Side-Tuning (LST), a new PETL technique that can reduce training memory requirements by more substantial amounts. Unlike existing parameter-efficient methods that insert additional parameters inside backbone networks, we train a ladder side network, a small and separate network that takes intermediate activations as input via shortcut connections (ladders) from backbone networks and makes predictions. LST has significantly lower memory requirements than previous methods, because it does not require back-propagation through the backbone network, but instead only through the side network and ladder connections. We evaluate our method with various models (T5 and CLIP-T5) on both natural language processing (GLUE) and vision-and-language (VQA, GQA, NLVR2, MSCOCO) tasks. LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter usages (hence, 2.7x more memory savings). Moreover, LST achieves higher accuracy than Adapter and LoRA in a low-memory regime. To further show the advantage of this better memory efficiency, we also apply LST to larger T5 models (T5-large, T5-3B), attaining better GLUE performance than full fine-tuning and other PETL methods. The trend also holds in the experiments on vision-and-language tasks, where LST achieves similar accuracy to other PETL methods when training a similar number of parameters while also having 2.7x more memory savings. Our code is available at: https://github.com/ylsung/Ladder-Side-Tuning.

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
Yi-Lin Sung
Jaemin Cho UNC Chapel Hill
Mohit Bansal
fylsung, jmincho, mbansal)0ca unc. edu
Abstract
tinettuning large pre-trained models on downstream tasks has been adopted in variety of domains recently. However, it is costly to update the entire parameter set of large pre-trained models. Although recently proposed parameter-efficien transfer learning (PETL) techniques allow updating small subset of parameters (e.g. only using 2% of parameters) inside pre-trained backbone network for a new task, they only reduce the training memory requirement by up to 30%. This is because the gradient computation for the trainable parameters still requires backpropagation through the large pre-trained backbone model. To address this, we propose Ladder Side-Tuning (LST), a new PETL technique that can reduce training memory requirements by more substantial amounts. Unlike existing parameter-efficient methods that insert additional parameters inside backbone networks, train ladder side network, small and separate network that takes intermediate activations as input via shortcut connections (called ladders) from backbone networks and makes predictions. LST has significantly lower memory requirements than previous methods, because it does not require backpropagation through the backbone network, but instead only through the side network and ladder connections. We evaluate our method with various models (TS and CLIP-T5) on (VQA, GQA,
both natural language processing (GLUE) and NLVR², MSCOCO) tasks. LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter us- ages (hence, 2.7x more memory savings). Moreover, LST achieves higher accuracy than Adapter and LoRA in a low-memory regime. To further show the advantage of this better memory efficiency, we also apply LST to larger T5 models (T5 (T5-large, T5-3B), attaining better GLUE performance than full fine-tuning and other PETL methods. The trend also holds in the experiments where LST achieves similar accuracy other PETL methods when training a similar number of parameters while also having 2.7x more memory savings.'
assionaadd-languae tasks,
Introduction
Recently, large-scale pre-training and fine-tuning of transformers [54] have been successful in various domains [8, 36, 44, 38, 53, 7, 20, 1, 10]. As the model size grows rapidly, fine-tuning the entire parameter of the large pre-trained model has become very costly. Parameter-efffiient transfer learning (PETL) [31, 23, 39, 40, 34, 51, 58, 17, 24, 18, 42, 60, 14, 52, 59] is recent research direction for online or multi-task learning. The goal is to build system that performs well on all tasks without training entire new model for every new task. Concretely, PETL methods select small subset of pre-trained parameters and/or insert few parameters pre-trained network and update those parameters for new tasks, while freezing most of the original parameters. In the
Our code is available https://githmn wwt.bl..oo...aa/..oee wtwp://www..ooolong/l/eaeeeeeeee Side-Tuniin
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(8) Adapters
(b) Prompt Tuning
Side-Tuning (Ours)
87.5
Output
Output
Output
Aversge Accuracy 19
85.0
12.5
Backbone (Frozen)
00.0
Rinetuning
Adapters
77.5
LoRA
Birii
75.0-
Prompt tuning
New Param (Updated)
Ladder Side-tuning
72.5
Ladser Side-tuning (T5-4arge)
Ladter Site-tniing (T5-3R)
70.0
16 Memory Usage (GB)
12
Forward
Figure Comparison between full fine- tuning, Adapter, LoRA, BitFit, and Ladder Side-Tuning over GLUE tasks. The y-axis is the average accuracy of 8 GLUE tasks, while the x-axis is the GPU memory usage during training. Unless specially stated, we use the T5-base in the figure.
Input
Input
Backward
Figure 2: Comparison between transfer learning with (a) Adapters, (b) Prompt Tuning, and our (b) Ladder Side- Tuning (LST). LST reduces memory usage by removing the need of backpropgation through backbone networks.
natural language processing (NLP), computer vision (CV), and vision-and -nd--anndd-gleeee (VL) domains, two types of parameters have been commonly updated for ceaneeer-efffcien transfer learning: (a) Adapters [23, 40, 39]: small modules inserted into transformer blocks; (b) Prompt [34, 31]: small parameters concatenated with input embeddings (see Figure 2).
However, while parameter-efficient techniques reduce the number of parameters to update, they do reduce the memory requirement during training by much (up to 30%). In other words, if large pre-trained language model does not fit on a GPU, these techniques usually do not help to fit the model on the GPU. In other words, these techniques usually do help fit a large pre-trained language model on a GPU if the model cannot be trained on the GPU with standard fine-tuning, Since the updated parameters are inside the backbone language models, to calculate gradients for these parameters for backpropagation, we still need to run the backward pass through the large pre-trained language models. This prevents PETL methods from being applied to many real-world applications with limited computational resources.
To address this issue, propose Ladder Side-Tuning (LST), a memory-efficient PETL method. LST separates trainable parameters from the backbone model to construct a side network, which is responsible for adapting the entire model to new tasks. Concretely, we train ladder side network, a lightweight network that takes intermediate activations via shortcut connections (ladders) from the backbone networks as input and makes predictions. As shown in Figure 2. unlike previous (a) adapters and (b) prompt tuning methods, (c) our LST does not add trainable parameters inside the pre-trained model, and this completely eliminates the need for expensive backpropagation of large backbone network and saves substantial memory during transfer learning. Instead of initializing the side network's 's weights randomly, we utilize network pruning technique to retrieve a smaller pruned network and use it as the side network. In addition to our standard design of the side network, we also boost the efficiency of LST by dropping layers of the side network. We empirically demonstrate that the layer dropping can significantly improve both memory and parameter efficiency without sacrificing performance. Furthermore, during inference, even though we have to propagate forward through two distinct networks, LST does not necessarily use more inference time because the same level of the backbone network and the side network can be computed in parallel.
We conduct comprehensive studies on LST using diverse NLP and VL tasks, namely, GLUE [55], VQA [16], GQA [25], NLVR² [50] and MSCOCO [6]. Overall, in GLUE experiments, LST saves 69% of the GPU memory that is needed for fine-tuning the entire backbone model, saving 2.7x memory compared against Adapter and LoRA. Also, LST achieves higher accuracy than other PETL methods in a low-memory regime. To take advantage of this better memory efficiency, we also apply LST to larger language models (T5-large and T5-3B) and find that it achieves higher accuracy than other PETL techniques when GPU memory utilization is similar. The findings still hold in the Vision- Lision-Langguge (VL.) experiments; LST not only a method that can fit 16GB GPU with 7.5% trainable parameters, but it also has similar or better accuracy than other PETL methods (and

---

again with 2.7x memory savings). To justify our design of LST, we conduct ablation studies on initialization strategies and alternatives to add shortcut connections. The results reveal that these components considerably help performance with minor computation overhead.
2 Related Work
2.1 Parameter-cfficient Transfer Learning (PETL)
PETL for NLP. In the past few years, large pre-trained language models have made huge success in NLP. Parameter-efficient transfer learning (PETL) is a research direction that reduces computational cost of adapting large pre-trained models to new tasks, by avoiding updates of the entire parameters. A popular line of work on PETL is add few trainable parameters and only tune them. For example, Adapters [48, 23] are small bottleneck modules that are inserted into transformer layers, and experiments have shown that training adapters with layer normalization layers is sufficient to achieve full fine-tuning performance. In similar trend, LoRA [24] injects trainable rank decomposition matrices into a frozen pre-trained model. Instead of inserting new parameters into pre-trained models, prompt-based [31, 34] methods add trainable parameters to the input and keep the entire pre- preitri model unchanged during training. The inserted prompts learn to make use of the knowledge of the pre-trame pre-trained model to solve new tasks. Although the concept of adapter-based and prompt- based approaches different, He et al. [18] unify the two lines of approaches (including LoRA) into adapter-based methods. In addition to approaches that introduce new parameters, there are also various methods [51, 58, 17] that select a sparse subset of parameters from the pre-trained model to update, without adding any new parameters. One of such representative methods is BitFit [58], which updates every bias term in the model.
PETL for CV/VL. While most of the progress in PETL is made in NLP domain, researchers have also applied this technique to the CV [47, 46, 27, 59, 62, 28, 60, 21] and VL [52, 61, 62, 28, 60] domains. VL-Adapter [52] benchmarks adapter- based and prompt-based methods on multiple image- text and video-text tasks, and shows adapters enable us to efficiently learn fusion information of vision and language. On the CV side, benefitting from the parameter efficiency of adapters and prompt-tuning, some works [62, 28, 60] apply these approaches to CLIP [43] to achieve strong few-shot performance in image classification tasks. Side-Tuning [59] uses an additive side network, which sums its representation with the backbone network in the last layer, to solve various tasks with ResNet [19] and BERT [8]. Although LST takes inspiration and has similarities to Side-Tuning we argue that there are major differences in motivations, architecture designs, and applied tasks between the two methods. LST aims to reduce the memory requirement of current PETL methods, whereas Side Made-Tuning does not focus on memory reduction (sometimes their side network is even as big the backbone network), but instead their motivation is to ease the forgetfulness in incremental learning. Our ladder side network is more robust than their design because the shortcuts fuse the intermediate information from the backbone network, and also layer dropping and network pruning techniques to make LST more efficient and stronger. Lastly, we further extend LST in VI. architecture and demonstrate its usefulness on multiple VI. tasks.
Current PETL approaches explore how to achieve competitive results using as few parameters as possible. However, parameter efficiency does not necessarily mean memory efficiency. In this work, we propose LST that has these two benefits simultaneously. Concurrently, Liu et al. [37] also propose Y-tuning to address similar issue; it exhausts all possible labels and feeds them into a model to select the best answer from the input. However, it is intractable oftentimes to list all answers in some tasks, for example, regression and open-ended generation tasks. On the other hand, LST is more flexible in applying to different architectures and tasks. We show that LST can outperform Y-tuning with fewer parameter updates in Table 2.
2.2 Memory-Efficient Training
Memory-efficien training aims to reduce the memory cost when training neural networks. Some approaches achieve this goal by cutting down the storage of intermediate activations, which dominate the training cost, to release a large amount of memory. The design of reversible neural networks [15, 29, 41] allows the model not to save intermediate activations because each layer's activations can be reconstructed from the next layer's. Gradient checkpointing [5] proposes to trade computation for memory by dropping some intermediate activations and recovering them from extra forward
3

---

pass. LST uses a different approach to cut the storage of activations; it keeps the backbone model frozen and constructs side network for cheaper training. Since the backbone model is not updated, LST is not only memory-efficies but also parameter-efficient Furthermore, memory saving from reversible neural networks and checkpointing is agnostic to memory saving from L.ST. Researchers can combine those methods with LST if they pursue a higher level of memory efficiency.
Another line of memory-efficient methods network compression, which compresses backbone network to a smaller one, and the generated network is cheaper for both training and inference. Two popular approaches for compression are network pruning and distillation. Network distillation [22, 30] constructs a student network and force it to have same output distribution of the teacher network over chosen dataset. Network pruning [12, 13] makes models lighter by learning importance scores of parameters and trimming unimportant parameters or neurons. While PETL still uses those untrained parameters in the forward pass, network compression entirely discards them or sets them to zero. As a result, network compression can generate models for faster inference speed while PETL. can achieve better performance by updating fewer parameters.
In this paper, we explore using network pruning or network distillation [22] (used in Side-tuning) to extract a sub-network that contains critical information of the backbone model and use it for initializing our side network. For distillation, we did not follow the original Side-tuning to apply distillation with large-scale pre-training datasets (e.g.. C4 [44]) because it makes distillation hard to conduct with limited resources and ultimately violates our goal of "efficient training." Also, using an extra pre-traie pre-training dataset during tine-tuning is unfair other approaches. We use the standard distillation procedure with the T5 [44] pre-training objective to train the side network. That is, the student (side) network learns to predict the masked spans and match the output distribution of the teacher (backbone) network simultaneously. We show the comparison between distillation-ban and pruning-based initializations in Figure 8.
We primarily use the network pruning method proposed by Li et al. [33] to initialize the side network because of its efficiency, and we describe the approach in detail in Section 3.3. The standard procedure of network pruning is (1) learn [12, 13] or heuristically define [33] an "importance measure" identify the importance of parameters, (2) prune p% of parameters with lower importance scores, (3) repeat the first and second steps until reaching the target sparsity. The rewinding procedure enables pruning techniques to find a more sparse sub-network. In this paper, to keep the whole pruning process efficient, we either use weights magnitude [33] or Fisher Information [51, 35] as importance measures, and reach the target sparsity one shot. As PETL method, LST makes use of intermediate information from the backbone model as the inputs, and we empirically demonstrate those additional inputs significantly improve performance in Figure 8.
3 Ladder Side-Tuning (LST)
We introduce Ladder Side-Tuning (LST), new PETL technique that can also reduce training memory requirements by substantial amounts than previous methods. In Section 3.1, we analyze the computational cost for fine-tuning with trainable modules in backbone models. Then we explain the architectural details (Section 3.2), structural weight initialization based on network pruning (Section 3.3), and dropping side network layers for more efficiency (Section 3.4).
3.1 Dependency on Backpropagation through Large Backbone Model
We consider a N multilayer perceptron (MLP): fx fı(x)) = a; (W,r + b,) consists of weight W., bias by and nonlinear function as, We denote the output of jt layer as 0;+1 and the pre-activation as 2411, where 0;+1 = oz(2+1) a,WWa 6.).
where the 8th layer
In backpropagation with loss L, the gradient with respect to W; and b,:
àL = Jazt 0zi+1 OW; dW.
aL dait1 0zi+1
aL Da,41
aL db,
aL
oaa
d.
(I)
00;+1
where of is the derivative of az dotts the gradient with respect to a,, can be calculated with the gradients with respect to a,+2, using the chain rule:
aL aL_

---

(a) Ladder Side Network
(b) Ladder Side Network for Encoder-Decoder
Backbone f
Side Network
Output
Side Enc g
y
Output
LM head
Backbone (Frozen)
Backbone f
Backbone fo
d2
New Param (Updated)
9
Forward
Backward
Forward (Low-dim)
Input
Inpul
Inpul
Figure 3:
Illustration of Ladder Side-Tuning (LST) with transformers described in Section 3.2. (a) shows a high-level overview of LST, and (b) shows LST with an encoder-decoder architecture.
0L 0L 0a:42 025+2
AL
d' Wi+1 0a;-1 da;+2 0zit2 0a:+1 da;+2
(2)
As shown in Equations (1) and (2), during backpropagation, there are two terms dominating the memory footprint: 1) {a} corresponding updated parameters {W} and 2) } that must be cached for the chain rule. Note that we use {-} to denote set of activations, parameters, or gradients. Existing PETL methods, such as Adapters [23]. LoRA [24]. Prompt-tuning [31], and BitFit [58, 3] could reduce the memory footprint by making |a| smaller, they have fewer {W} to update, but do reduce Jo'), where
means the size of set {.). Since most activation functions do not change dimensions (i.e., |a| = (o')), the memory footprint for backpropagation |a| + |a'l can be reduced by up to 50% by the PETL methods when they reduce the entire memory footprint for (a], By making the updated parameter do not require backpropagation through the backbone network, our LSTT can achieve better memory efficiency beyond 50%, and we explain it below.
3.2 Ladder Side Network for Transformers
Unlike existing transfer learning methods that insert additional parameters inside transformer net- work, we propose training ladder side network, small and separate network that takes intermediate activations from the backbone transformer as input and makes predictions. As illustrated in Figure (a), since the ladder side network parameters Φ are not used during the forward pass of the backbone transformer with parameters 6, the update of the ladder side network does not require expensive backpropagation of the large backbone transformer. Note that our LST method is limited to specific architecture. We provide simplified overview of LST with an encoder architecture in Figure (a) and an illustration of LST with encoder-decoder architecture Figure (b).
Lightweight architecture. Our side network g is lightweight version of the backbone transformer f. where all weights and hidden state dimensions of in g are 1 times of the original weights and hidden states of /, where r is reduction factor (e.g. = 2, 4, 8, 16). For example, if the backbone f has 768- 168-dimenssioal hidden state, then the side network with r 16 has hidden state of 48 dimensions (= (66816). The side network reuses frozen word embeddings ('Emb' in Figure (a)) and the language model head ('LM head' in Figure (a)) of the backbone f. Following the analysis in Section 3. we also examine the memory cost of LST. Recall that original memory footprint for backpropagation |a| + lo'|. Because we do not have to run backward pass through the backbone network, we can only consider the gradients for the side network, whose memory footprint is +l+lol Therefore, L.ST has better memory efficiency than other PETL methods (saving up 50%) as long is greater than 2 (we find 8 works well in most experiments).
5

---

(a) Structural Weight Initialization
(b) LayerDrop in Side Network
1. Select backbone weights importance score
2. Initialize weights Side Network
Output
Output
Input
Input
Figure 4: Illustration of (a) Structural Weight Initialization (Section 3.3) and (b) Layer Dropping (Section 3,4). In our experiments, we find that initialization of side network parameters from backbone network parameters improves performance, and dropping some shortcut connections improves efficiency without hurting performance.
Gated ladder connections. Although Zhang et al. [59] found that late fusion to combine the representations of the backbone and the side network works well with convolutional networks for CV tasks, in our experiments, we find that late fusion hurts the performance of the transformer architecture NL.P tasks (see Figure connection (called ladder, due to the overall shape created from the multiple shortcut connections) from intermediate activations from the backbone f to the side network g and find it helpful. We learn linear projections to downsample (x () the intermediate activations (including word embeddings) of to low-dimensional attention blocks in g. Then, we learn linear projection to upsample (xr) the side network output to the dimension of the original language model head. The linear projections are illustrated as green trapezoids in Figure 3 (a). The 8h transformer layer of the side network combines the activation of the backbone bf and the activation of the previous layer of the side network
Section for details). To address this, we use shorteut
hi-1 with learned gating: μ4; h +1 (44 with a learnable zero- iro-iinitil scalar a; and temperature T (= 0.1). We have also tried to use Adapter blocks to build the side network and replace the gating mechanism with noss-attentions, but we find the current design works the best (see ??).
* 1. where 14; sigmoid T is gate parameterized
3.3 Structural Weight Initialization for Ladder Side Network
We find it helpful to initialize the weights of the side network from the weight of the backbone network 0 based on structural pruning [33], as shown in Figure (a). Concretely, given a weight matrix W E Rdout Xden of the backbone network that maps the din-dim vectors to the doum-dim space, and the importance matrix of the weight I E gdousxdss we first calculate the importance score of each
Σ, | denoting the importance of each weight vector. Note that the importance matrix I used in this work either weight magnitude [33] (I W) empirical Fisher Information [51] Fw + EN (Vw log p(y: [z:;)) (xi, 31:,,... Nx yN) are samples from data). Then, we choose the rows of W which have the top Lumi importance scores and prune the
remaining rows to obtain new weight matrix W.P E RR dont xd... The columns of the weights and the importance matrix in the next layer corresponding to the pruned feature map are also pruned. By iterating this process, we obtain the set of weight matrices whose rows and columns are pruned times from the backbone network and shown in Figure 7, we find that using Fisher information as an importance score metric generally performs well, and therefore we use in our structural weight initialization.
them to initialize the side network. In experiments
3.4 Layer Dropping in the Ladder Side Network
explore to increase efficiency LST even further by making side network more compact, by
dropping its intermediate transformer layers, as illustrated in Figure (b). Similar to LayerDrop [1 1]. we drop layers the side network, and this can linearly reduce the memory and parameter require- ments of LST. For instance, side network with N layers will only have 2nd thh 6th layers left,

---

after we drop half of the layers. Refer to Section 4 for more details on applying layer dropping on an encoder-decoder architecture. In Figure we show that layer dropping can greatly boost the model's efficiency without sacrificing performance.
4 Experiment Setup
Datasets. We evaluate LST on NLP and VI. tasks. For tasks, we use the GLUE [55] bench- mark, which consists of seven classification and one regression task. The benchmark evaluate models on multiple diverse tasks over linguistic acceptability (CoLA [56]), sentiment analysis (SST- 2 [49]), similarity and paraphrase (MRPC [9], QQP [26]. STS-B [4]) and natural language inference (MNLI [57], QNLI [45], RTE [2]). For VL tasks, we experiment with visual question answering (VQA [16], GQA [25]), visual reasoning (NLVR² [50]) and image captioning (MSCOCO [6]) tasks.
Baselines. We compare LST against the full fine-tuning and several popular PETL approaches on both NLP and VL setups. In Full fine-tuning, all the parameters are updated for a downstream task. Full finettuning is not parammter--fffcieen nor memory- the fine-tuning performance. To compare to other PETL methods, we reproduce Adapters, where we inject small trainable modules after every attention and feed-ford beed-forward layer, and we solely train those modules and layer normalization layers while keeping the rest of the model frozen. We also reproduce LoRA, which inserts trainable low-rank matrices into the model to parameterize the weights' changes. In BitFit, we only update the bias terms over the course of training. Lastly, we compare our method to Prompt-tuning, where trainable prompt vectors are prepended to the input. We initialize the prompt vectors with the embedding of the pre-trained model's vocabularies.
but it serves as the upper bound of
Training and Evaluation Setup. For NLP tasks, we use T5 [44], a pre-trained encoder-decoder language model as our backbone. We use T5-base in most experiments, except that we scale up LST on Targe and T5-3B to demonstrate its memory efficiency. The training and evaluation process follows the setup used by Mahabadi et al. [40]. Since there is no local test set, we split 1K samples from the training set as the new validation set and use the original validation set as the test set. For datasets whose samples are less than 10k (RTE, MRPC, STS-B, CoLA), we split the validation set into two cqual-sized subsets and treat them as a new validation and test set. For MNLI, we use the mismatched set as the validation set and matched set the test set. We train every approach with 10 epochs on large datasets and 20 epochs on small ones (RTE, MRPC, STS-B, CoLA) for complete convergence. We search for learning rates over {3 x 10-4, 1 10-3 x 10 3) for LST and LoR4[24], and we use the optimal learning rates that are used by Mahabadi et al. [40] for other methods. The reduction factor used in LST is set to 8 if not additionally specified. T5-base has 12 layers each in encoder and decoder, while Ta-large and T5-3B have 24 layers each. In our experiments, we do not drop layers Ta-base unless we specially mention it. For T5-large and T5-3B, we drop 24 layers (12 layers each in encoder and decoder) and 46 layers (23 each) of the side network to make the memory usage close to our basclines. The experiments on T5 take around 12 hours to train with one A6000 GPU (48GB).
For VL tasks, we experiment with CLIP-T5 [52], which is VL architecture combining CLIP [43] and T5 [44]. We always freeze the CLIP and only train the T5 for new tasks. The CLIP visual representation is concatenated with the text embedding. and the combined input is fed to T5 make predictions. A visual projection layer is added between CLIP and T5 to let the visual representation have the same dimension as the text embedding. To avoid updating the visual projection layer by the gradients from the backbone model, we do not feed combined inputs to the backbone model, but only text inputs. The combined inputs are fed the side network, so we can achieve efficient training by only computing the gradients from the side network. Because the backbone network only uses texts as the input, the information from the backbone network via shorteut connections is only summed to the text part of the side network's 's combined inputs. We follow the multi-tasking setting for training and evaluation used in VL-Adapter [52]. We report the performance on Karpathy hee/testtdee/testtP/Eee PaKaestttt/aaaaahhhyyh test split for VQA/GQA//LVV /MSCOCO, and train models for 20 epochs. We search learning rates over (3 x 10-1, x 10-3, 3 x 10-3) 3} for PETL methods, and use x 10-1 used by Sung et al. [52] for full fine-tuning. We set the reduction factor for the side network to We train CLIP-T5 for 16 hours on one A6000 GPU. ??, we comprehensively list hyper-parameters for NLP and VL experiments in ?? and ??, respectively.
7

---

Table 1: Comparison between multiple parameter-efficient training methods on GLUE benchmark. We use T5-base if we don't additionally specify. We report accuracy for SST-2, MNLI, QNLI and RTE. For CoLA and STS-B, we use Matthew's 's Correlation and Pearson-Spearman Correlation as the metrics, respectively. For MRPC and QQP, we report the average of F1 score and accuracy. Each number in the table is the average result over three seeds, and the subscripts are standard deviations. For the results with I, we report the best performance out three seeds due to the instability of the method. We report the maximum memory usage training and evaluating RTE for each method.
Method
Upèine Partes per Tisk (%)
Monory Usaee (GR)
CoLA
SST-2
MRPC
QUP
MNLI
QNLI
RTE
STS-B
Arg.
Trin
Inference
Full fine-teming
100
17.6
0.86
62,8
93.9pa
91.9,
89.9u
86 86.26.
92.5a
74.1
50.3a
85.2u
Adapters
1,63
13.0
0.87
64.4g.1
94.2p.
S8.9t.
88.96.
M6-46.1
93.1e2
75.1ez
91.1g2
85 85.3a3
LoRA
1.71
12.6
0.86
63.3p
94-3a.
89.06
86-
93.2g
75.5ax
90.9aa
85.3a5
BiFit
1:13
10.7
0.86
61.8,
94-3pc
91.00g
88. o
85.4g
93.1g.1
07.6us
90.8au
84-1ua
Prompt-tuning
0.03
22.2
0%.
90.3
6ccc
88.5
82.5uu
92.5p
99.5..
90.1a
72.3
Ludder Side- Tuning
1.74
0.88
58.1s.2
90:4.c
88 8.
85 66.1
93.3a
71.8.
84.los
Ladder Side Luning (15 luge)
1,23
12.2
2.88
65.35,
95.76.1
91.66.8
89.7um
88 6o.s
94.102
79.9u
92. 4g.1
87.1o2
Ladder Side-Tuning (TS-3B)
0.08
22.4
11.01
66.41.7
96.5
92.9ka
89.7a
90.7a.
95.1a9
80.1.
93.0ga
88.164
Table 3: Comparison between multiple parameter-efficient training methods on VQA, GQA, NLVR?, and MSCOCO. We use T5-base for all approaches. We report accuracy for VQA, GQA and NLVR while we use CIDEr to evaluate MSCOCO. Each number in the table is the average result over three seeds, and the subscripts are standard deviations.
Update Param. (%)
Memory Usage (GB)
Method
VQA
GQA
NLVR
MSCOCO
Avg.
Train
Inference
Full fine fine-tuning
100
36.2
0.86
67.lu.1
56.3a.3
74.3p.4
112.2u.3
77.5p.a
Adapters
7.98
28.4
0.93
67.1g.1
56.0g.4
72.70.3
111.8p.1
76.9p.2
LoRA
7.54
27.9
0.86
63.7a.2
53.30.1
70.0g.3
110.30.4
74.30.1
BitFit
0.83
22.7
0.86
55.1g.2
45.5o.2
51.71.1
101.20.2
63.4p.1
Prompt-tuning
1.26
38.7
0.87
47.4u.7
40.6u.4
51.0g.4
96.1a9
58.8p.a
Ladder Side- Tuning
7.46
15.3
0.93
66.5g.1
55.90.1
71.60.3
113.5p.a
76.90.1
5 Experimental Results
In this section, show experiments to justify our design of LST and demonstrate that LST performs the best among all approaches in the scenario with limited memory. As the result, LST is the most efficient tool to fine-tune large-scale pre- pre-trained models for real-world applications.
LST outperforms other methods under similar memory usage. Figure and Table show the results on GLUE of different approaches applying on ba-e We drop layers (3 layers each in side encoder and decoder) for LST to match the parameter usage of the Adapter and LoRA. Under the same parameter usage, LST can save 69% of memory cost to fully fine-tune the model, while Adapter and LoRA only save 26% of that, leading to LST having a 2.7x more memory saving. Compared to BitFit, LST achieves the same average performance but costs 5GB less GPU memory. LST also surpasses Prompt-tuninn -tuiing in terms of both performance and memory cost. To further take advantage of the memory efficiency of LST, we also train T5-large and T5-3B with LST. We find that with a similar budget of memory usage in Adapter and LoRA, LST with T5-large can surpass the performance of other methods by large margin. The result on T5-3B also outperforms the result on la-ger,, demonstrating the scalability of our memory-efficiency method on large language models. Furthermore, even though LST increases the model size, its additional inference memory usage is negligible as LST uses almost the same inference memory (0.88 GB) as the full fine-tuning (0.86 GB).
In Table 2, we also compare LST with a concurrent work, Y-tuning [37] on GLUE tasks (except for STS-B) with BART- large [32] encoder as backbone. Following their experimental setup, we use a different learning rate and report the best accu- racy out of three seeds for each task. Overall, LST outperforms Y-tuning by a large margin with fewer updated parameters.
Table 2: LST vs. Y-tuning.
Update Param. per Task (%)
Avg. GLUB
Method
Y-tuning LST
7.7
76.9
2.6
82.1
LST competitive on VI. tasks. As we have mentioned beforehand, we also extend L.ST on mudai-modal architecture, CLIP-T5, multiple VL tasks, and we demonstrate the outcome in Table 3.
8

---

Adapters
Adapters
LoRA
Average Accuracy (1)
67
Ladder Side tuning
86-
Bkfit
Ladder Site-turing
84
15
02
80
83
14
Memory Usage
Merory Usage (GB)
The cecuracy-memory trade-off for Adapter, LoRA, and Ladder Mide-Tuning over GLUE tasks. We vary the reduction factor in Ladder Side- Tuning hidden dimension in Adapter, rank in LoRA to get the architectures with different training costs.
Figure 5:
The Adapter, LoRA, BitFit, LST over GLUE tasks. drop N {0, 6, 12, 18) layers in inter- leaving manner for LST while we gradually freeze the first N {0. 6, 12, 18] layers in other methods (also remove inserted parame- ters those layers).
Figure 6:
trade-off for
With similar parameter usage, LST is the only method that can fit into single 16GB GPU. Besides the efficiency, it is as competitive as full fine-tuning and Adapter, outperforming other PETL approaches.
LST performs the best in low- low-memory regime. To have a better understand of the memory advantage of LST, adjust the prerr-paraeeters in our method (reduction factor E {32, 16, 8, 4)), Adapter (hidden dimension E (6. 12, 24, 48]) and LoRA (rank E {4,8, 8, 16, (2]) ) to create multiple architectures with different memory costs. Figure 5 shows the performance and memory efficiency trade-off for all methods. We find the memory saving is not obvious for Adapter and LoRA, because the gradients of the backbone model's intermediate outputs are still computed (see Section 3.1 for details). Even though the Adapter and LoRA can get slightly better memory efficiency by reducing the hidden dimension and the rank, find that the performance drops significantly. On the other hand, LST is quite robust across a wide range of side network sizes.
We also consider another way to compare LoRA, Adapter and BitFit to LST in different memory budgets. While we drop N E {0, 6, 12, 18} layers in an interleaving manner to improve memory efficiency, we freeze the first N layers and remove the corresponding inserted modules in other approaches. With this, other methods can achieve better memory efficiency because gradients do not propagate those earlier frozen layers. We discuss the layer dropping and layer freezing with details in the following. In LST, we drop π layers in both side encoder and side decoder. However, in other PETL approaches, we start from freezing layers in the encoder and then in the decoder (e.g. freezing 18 layers means freezing all encoder layers and first 6 decoder layers). We display the comparison in Figure 6, showing that LST has better performance and memory rrfe--off and outperforms other methods in the fow-memory regime. We also find that layer dropping generally reduces the training cost without hurting performance.
N
freeze layers
The ablation of weight initialization on the side network. We compare the different initialization strategies for the side network and demonstrate the results Figure 7. "Random" denotes we randomly initialize the network while we use network pruning to select initial weights for the side network based on two importance measures, "Weight Magnitude" and "Fisher Information." " In general, the initialization from the pruned network helps no matter the size of the side network, showing the effectiveness of our network pruning strategy.
Comparison of LST to network compression methods and Side-tuning. In Section 3.2, we mention that shortcut connections are added to every layer of the side network. We justify this design by comparing LST to two types of approaches: (1) network compression, which discards all shortcut connections and the entire backbone model; (2) Side-tuning, which only adds one shorteut connection to merge representations right before the output layer. Note that we do not drop any layer in the side network but only remove the shortcuts in this experiment. We also compare both distillation-bau based and maming-basce initialization methods describe in Section 2.2. We set the

---

reduction facttor=32
83.8
00-
83.6
Accuracy
73
12.3
Accuracy (%)
70
83.25
61.5
83.00
82.9
55
82.75
50
82.50
Network compression
(notwork proning) Sidee Sale-tiusing (clansic dini..
Side-tuntas tuntuning (nelwork pruning) LST (classic disil.)
fretwork praning)
Natwork (clasiic dlatel.)
82.25
82.00
Random
Weight Magnitude Fisher Information
Initializattion Strategy
Methods
Figure 7: The ablation study on different initialization strategies. Y-axis denotes the average score over GLUE tasks.
Figure 8: The comparison between net- work compression, Side-Tuning, and LST on GLUE tasks.
reduction factor for all approaches. Figure shows the comparison and LST outperforms the other two types of methods significantly. We conclude that PETL methods are stronger than network compression as they use the information from the backbone model. This also suggests that network compression approaches need to train more parameters to achieve the same level of performance as PETL methods. We also demonstrate the usefulness of intermediate shortcuts since LST surpasses Side-tuning by large amount. Furthermore, we find that using distillation- network pruning-bant based initialization provides similar accuracy in setups. Note that our network pruning-base initialization method is more efficient since it does not involve training.
initialization or
Conclusion
We propose Ladder Side-Tuning (LST), a parameter- and memory-efficient method to adapt large backbone network. LST does not require backpropagation through the backbone network, which allows for significantly lower memory requirement during training than recently proposed parameter- efficient training techniques. We demonstrate that LST allows users to adapt larger and more powerful backbone network to target tasks with a limited memory requirement, which cannot be achieved with recent parameter-efficienl techniques. We also show that LST achieves more efficient eecuracy-memane trade-off than recent baselines, the impact of weight initialization side networks, and the usefulness of intermediate shorteut connections. Finally, show that the LST can be also extended beyond NLP tasks, with strong results on VL tasks. We hope that LST helps users with limited computational resources tune larger models in diverse domains.
Acknowledgments
We thank the reviewers, Mugeeth Mohammed, Derek Tam, Prateck Yadav, and Gedas Bertasius for their helpful discussions. This work supported by ARO Award W91 10122100220 ONR Grant N000141812871 and NSF-AI Engage Institute DRL-21 1263. The views, opinions, and/or findings contained in this article those of the authors and not of the funding agency.
References
Hangbo Bao, Li Dong, and Furu Wei. Beit: BERT pre-training of image transformers. CoRR, abs/2106.08254, 2021. URL
http//aaxvv...raaraaaa orgg/ba////// 08254.
[2] Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bemardo Magnini. The fifth pascal recognizing textual entailment challenge. In In Proc Tex Analysis Conference (TA 2009.
[3] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and
10

---

H. Lin. editors, Advances in Neurul Information Processing Systems, volume 33, pages 11285- 11297. Curran Associates, Inc., 2020. URL https://proceedings http:///pccceeeennnnneeepep..c www.:/ednnnnnnuuur...c.o.o httprr/pp.ccppppp/////////. L.ffaaab44117744cc6622270o6666
pdf.
[4] Daniel Cer, Mona Diah, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia, Semeval-2017 task 1: Semantic textual similarty-mulriinnual and cross-lingual focused evaluation. In SemEival, 2017.
[5] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory ArXiv, abs/1604.06172 2016.
[6] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325, 2015. URL http: /tappp:///axv....//// http://aaxv...////ee.// www..orrg/bbb.....eeee 00325.
[7] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying iasson-and-language tasks via text generation. In Proceedings of the 38th International Conference Machine Learning, volume 139, pages 1931-1942. PMLR, 18-24 Jul 2021. URL https:/ htttps://proceedingg mlr press/v139 press/v139/choll... htt99//hh2aaahaaa.
[8] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. NAACL, 2019.
[9] Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005). Asia Federation of Natural Language Processing, January 2005. URL https: http://wwwwwiic.oooooooooo com/ www.eoooeeeeeeeeeeeeeeeeeeeeeeee wtw///nuurrreeeeeeeeeeee ptt.earrcc/pblictioooo// htwpaaattrttnnnaaaarrr--r------- -orpa-a---o-op-------tttttaaaaaa paarppararrpprppppaeeeeeae
[10] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, In International Conference Learning Representations, 2021. URL http:///////peeeeeeeeeeene//eeee id=YicbFaNTTy.
[11] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference Learning Representations, 2020. URL https:///ppenrevvew net/forum htte/forumiddSyy100 Syy=St.
[12] Jonathan Frankle and Michael Carhin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2019.
[13] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connec- tivity and the lottery ticket hypothesis. JCML., 2020.
[14] Peng Gao. Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li. and Yu Jiao Qiao. Clip-adapter: Better vision-language models with feature adapters. ArXiv, abs/21 10.04544, 2021.
[15] Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger Baker Grosse. The reversible residual network: Backpropagation without storing activations. In NIPS, 2017.
[16] Yash Goyal, Tejas Khot, Douglas Summers Stanner--Sta,..... Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 6325-632222 6334,, 2017.
[17] Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4884-489,, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 acl acl- URL h / /aap/aaaaaaannncyocaeaoao org/2021 acl acl-long. 1o11111...77..... .788.
[18] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards a unified view of parfmeter-cfffccient transfer leamnig. In International Conference on Learning Representations, 2022. URL https: http://://///eee/eeeeeeee wwt./eeeennnninne www.mmmmddddddddddeee OdDoodAxokk.
[19] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[20] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. CoRR, abs/2111.0637/ 2021. URL https://arxiv..rr org/
7.777.11.063777.

---

[21] Xuehai He, Chengkun Li, for vision transformers. ArXiv, 16329, 2022.
Pengchuan Zhang, Jianwei Yang. and Xin Wang. Parameter-efflcient fine-tuning
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015. URL http://arxis http//awwwv....aaaaaeaeee www.oo///////o/oooooo...o..e ha//abaa/1100 aataa/153..000 02531. cite arxiv: 11555503.23311mmmei Comment: NIPS 2014 Deep Learning Workshop.
[23] Ncil Houlsby, Andrei Giurgiu, Stanislaw Jastrzehski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Purameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-27999 PMLR, 2019.
[24] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. CoRR, abs/2106.09685, 2021. URL attps://arre http:///aaxi..rrraa///////ss hts//bbs/21000 09685.
[25] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference Computer Vision and Puttern Recognition (CVPR). 2019.
[26] Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs. 2017. URL aattps://daat quora. wwweeiistt/ooorreeeeeaaeoee Daapast----ora--aatt--------ee Rateor---aaateee-eeeas----- Release-Question Question- Panir
[27] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongic, Bharath Hariharan, and Ser Nam Lim. Visual prompt tuning. ArXiv, abs/2203.122119 2022.
[28] Konwoo Kim, Michael Laskin, Igor Mordatch, and Deepak Pathak, How to adapt your large-scale vision-and-languuge model, 2022. URL https: /ttp///eepeeeeeeee/ee/eeeeeeeeee wwwp:r/rwwweeeeettte////ddteaaa
[29] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In Interna- tional Conference on Learning Representations, 2020. URL
http:///wnneeeeeeeeeee/oorrreiia net/ffrummid
rkg rkg.KkHttvB.
[30] Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. LIT: Learned intermediate representation training for model compression. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3509-3518. PMLR, 09-15 Jun 2019. URL htttss//proceediiggg mlr. Salr..presa/v91 koratana19a html.
[31] Brian Lester, Rami Al-Rfou, and Noah Constant, The power of scale for parameter-officient prompt tuning. In EMNLP, 2021.
[32] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-segee to-seuennce pre-training for natural language generation, translation, and comprehension. In ACI, 2020.
[33] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2017.
[34] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan Language Processing (Volume 1: Long Papers), pages 4582- 4597, Online, August 2021. Association for Computational I Linguistics. doi: (0.18653/v1/2022 accl-oo.. 353. URL h/tt:ps://acaaaaa htapp///////caaaooooooooo or 2021. acl--ong. 353.
[35] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jingliang Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In ICML, 2021.
[36] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A rohustly optimized BERT pretraining approach. CoRR, abs/1907. 11692, 2019. URL. http://arxii http:::///wxvv.o///.///./ abs/ /9tp// 11692.
[37] Yitao Liu, Chen An, and Xipeng Qiu. Y-tuning: An efficient tuning paradigm for large-scale pre-trained models via label representation learning. ArXiv, abs/2202.09811,, 2022.
[38] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnostic visiolinguistic representations for vision-and language tasks. In 2019. 2019.
12

---

[39] Rabeeh Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efffcienl multi- task fine-tuning for transformers via shared hypernetworks. In Annual Meeting of the Association for Computational Linguistics, 2021.
[40] Rabeeh Karimi Mahabadi, James Henderson, and Sehastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In Thirry-Fitth Conference on Neural Information Processing Systems, 2021.
URL https: oppnnevieeeeeeeeeeeeeeeeeeee net/ www/eeeettuureee ftrmmm:rmm? wttp:oodddddddddppppp d=bq(K6PyI6-1.
[41] Karttikeya Mangalam, Haoqi Fan. Yanghao Li, Chao- Yuan Wu, Bo Xiong, Christoph Feichtenhofeer, and Jitendra Malik. Reversible vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10830-10840, 2022.
[42] Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen-tau Yih, and Madian Khabsa. Unipelt: unified framework for parameter-efficient language model tuning. CoRR, abs/2110.07577, 2021. URL https / /awxiv.vvv./ wwtplii..rr/b//////eeee/e 2110. .07577.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[44] Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67 2020. URL http://jmlr: wwgggg/prs///aeeeee/ee/eeegseere 20-074.. html.
[45] Pranav Rajpurkar, Jian Zhang, Konstantia Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
[46] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NIPS, 2017.
[47] Sylvestre-Alvis Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of mulli-domaal deep neural networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2018.
[48] Andrei Rusu, Neil C. Rabinowitz. Guillaume Desjardins, Hubert Soyer. James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, abs/ Abs//600.004671, 2016.
[49] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. October 2013. Association for Computational Linguistics. URL attps://aclanthon
org/D13-1100
[50] Alane Suhr, Stephanie Zhou, Ally Zhang, Tris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistiess pages 6418-6428, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10.11653//v1/19-- 1644. URL https: / aalaaaaaaaanto...... arg/P19-44 1644.
[51] Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems, 2021. URL htt.tt::/tpoeeerrereee net/forumm viHSvvx
[52] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficien transfer learning for vision- and-language tasks. CVPR, 2022.
[53] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP, 2019.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg. S. Bengio. H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: : httpe///po//peggeeeeeeee neuriinnnnnnnnni....e...m.r httpipppp.pppppppeeeepepe cc/ppapr//017 file
Paper pdf.
[55] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A tulti-task benchmark and analysis platform for natural language understanding. In International Confer- Learning Representations, 2019. URL https://opeerrviiee net/forumm wowoorrdmdrreeeer -
13

---

[56] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 805. 12471, 2018.
[57] Adina Williams, Nikita Nangia, and Samuel R Bowman, A broad-coverage challenge corpus for sentence understanding through inference. In NAACI,, 2018.
[58] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameler--fficient fine-tuning for transformer- based masked kanguage-models. CoRR, abs/2106. 2021. URL https://arxiv.. org/ abs/2106. 10199.
[59] Jeffrey Zhang, Alexander Sax, Amir Roshan Zamir, Leonidas J. Guibas, and Jitendra Malik. Side-tuning: A baseline for network adaptation via additive side networks. ECCV, 2020.
[60] Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai. Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. ArXiv, abs/21 11.0930,, 2021.
[61] Zhengkun Zhang, Wenya Guo, Xiaojun Meng, Yasheng Wang, Yadao Wang, Xin Jiang, Qun Liu, and Zhenglu Yang. Hyperpelt: Unified parameter-effciien language model tuning for both language and vision-and lnd-uage tasks. ArXiv, abs/2203.033 008878. 2022.
[62] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. ArXiv, abs/2109.01134, 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions nceded to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "y2bmju4-4S-", "reviewer": "Reviewer_qN8G", "summary": "This paper proposes a new parameter-efficient transfer learning (PETL) technique called Ladder Side-Tuning (LST). The proposed method trains a small and separate ladder side network that takes intermediate activations as input and output predictions by short-cut connections. During finetuning, the paper only requires backpropagation through the extra small ladder side network, and thus improves parameter and memory efficient transfer learning. The experiments on various models and learning tasks show the efficiency of the proposed method.", "questions": "(1) More discussions/analyses about the key differences between side-tuning and LST.\n(2) More experiments show the claims/advantages that LST differs from side-tuning.\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) The proposed LST only needs to compute backpropagation on small ladder side networks, which greatly improves the efficiency of PETL models.\n\n(2) The experiments on various models and tasks (vision and language) show the efficiency of LST.\n\nWeaknesses:\n\n(1) The proposed LST shares a similar idea with side-tuning [53]. As claimed in side-tuning [53], “we propose a straightforward alternative: side-tuning. Side-tuning adapts a pre-trained network by training a lightweight “side” network that is fused with the (unchanged) pre-trained network via summation”. In Line 99-107, this paper claims LST there are differences in motivations, architecture designs, and applied tasks between [53] and LST. I summarize these three aspects as follows.\n\n-Motivation. Side-tuning [53] focuses on incremental learning, an adaption task that extends n-category classification to (n+m)-category classification. The proposed LST aims to fine-tune large pre-trained models on downstream tasks, which is a general adaption task. Although the tasks are not extra the same, they are both adaptation tasks. Side-tuning [53] can be directly used to fine-tune large pre-trained models without major adjustment.\n\n-Architecture designs. Side-tuning claimed that places no constraints on the structure of the base model or side network, allowing for the architecture and sizes to vary independently. It applied a residual on the top of the backbone in experiments. LST uses multiple layers as a side network of the backbone. I think the modification mainly depends on how large the gap between a source task and a target task. The main reason could be incremental learning has to maintain the source classification and only require a minor update. [53] used knowledge distillation for initialization and LST uses network pruning. I think both are available.\n\n-Applied tasks. Side-tuning [53] focused on incremental learning and LST focuses on adaptation from a large pre-trained model to downstream tasks.\n\nTherefore, compared with [53], the contribution of this paper is weak.\n\n(2) In Line 295, the paper conducts some studies on w/ random and w/ network pruning. I wonder if knowledge distillation side-tuning also works, which is proposed by side-tuning[53]. As far as I know, both knowledge distillation and network pruning work in network compression. It would be nice if the paper studies why not directly use knowledge distillation as side-tuning does.\n\n(3) In Line 301-309, the paper conducts some studies on side-tuning. I am interested in the analysis that we directly use side-tuning [53] for this task and simply use more intermediate layers.  Using more intermediate layers is widely used in deep neural networks, at least dating back to segmentation/classification/detection tasks in 2016.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nRd5s9X_Xs", "reviewer": "Reviewer_yNyy", "summary": "This paper proposes an efficient fine-tuning method for pre-trained language models. The main idea is to fix the models' weights,  calculate the layer-wise activation, project the activation to lower dimension, and feed the downsampled activation into a smaller side network. During training, only the side network's parameters are updated so that the memory cost is small. The authors propose an initialization method for the side network that directly takes weights from the pre-trained models. Experiments show that the proposed method can memory-efficiently train large language models while preserving decent accuracy. ", "questions": "- I am a little bit confused why the initialization method could help. For example, the activation after layer 1 will be fed into the first layer of the side network, which is also initialized by the partial weights of layer 1 in the full model. With that being said, the activations seem to be processed by the same layer twice (or partially twice), and I am not sure whether this is a good choice if I had understood correctly. Although the ablation experiments show the comparison between different pruning criteria, I think it would be better to compare methods like \"offsetting\" or \"random initialization\". \n\n- This paper emphasizes on training memory but I am curious about how would the inference memory of LST compare with other methods since inference memory may also be bottlenecks. For example, a device may be able to fit T5-base but not able to fit T5-large or T5-3B for inference. So even if we are able to train a T5-3B with smaller memory, it maybe also limit the practical usage. This is somehow related to the comparison in Table 1 where the authors train a larger network for comparison. \n\n- The experiments on LoRA seem to use a larger rank that most works suggested (for example most works use rank smaller than 16 which will lead to smaller than 1% trainable parameters), and original LoRA paper stated that they can save more than 2/3 VRAM on GPT-3 while in this paper the memory saving is fewer. ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The memory saving is significant compared to other methods due to the extracted layer-wise representation and smaller side network. \n- The number of parameters is smaller than full finetuning which saves the cost of storing parameters. \n- Propose an initialization method for side networks that uses Fisher information and structured pruning. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NvRSDjtYT7", "reviewer": "Reviewer_L9hL", "summary": "This paper proposes Ladder Side-Tuning (LST), a method for parameter-efficient transfer learning with pre-trained neural networks. LST is an extension of the concept of Side-Tuning [1], which is a previous work for avoiding full parameter updates by adding small target side networks. In contrast to Side-Tuning, LST fuses the source and target models in the intermediate layer outputs by weighted sum and trains the model with several engineering techniques such as initialization of the weight by structural pruning and layer-wise dropout. Experiments confirm that LST can achieve competitive performances to several previous parameter-efficient transfer learning methods while largely reducing memory usage during the training time.\n\n```\n[1] Zhang, J. O., Sax, A., Zamir, A., Guibas, L., & Malik, J. (2020, August). Side-tuning: a baseline for network adaptation via additive side networks. In European Conference on Computer Vision (pp. 698-714). Springer, Cham.\n```", "questions": "### Q1. Why is memory usage during training important?\nThe motivation explained in the Introduction is not clear to us. As far as we know, parameter-efficient transfer learning methods using large pre-trained models are motivated not only by the reduction of update parameters and memory usage but also by the prevention of overfitting [1,3,4]. There is also a focus on speeding up inference time for classical parameter reduction methods such as knowledge distillation [5] and pruning [6,7]. On the other hand, this paper focuses only on the limited issue of memory usage during training and does not consider the other important motivations enumerated above. We also consider that even the problem of memory usage has not been completely solved, since the experiment in Table 1 when using a huge pre-training network (T5-3B), ended up exceeding 16 GB, which is the standard for memory-saving environments in other evaluations. More to the point, in business, the training phase has relatively usable resources compared to inference, so providing an A6000 with the 48GB of memory used in the experiments in this paper for training is not much of a cost. Rather, LST or other methods adding parameters increase the cost of inference and cannot be used in resource-sensitive environments during inference. In summary, the research question for this paper is unclear and thus we are unable to determine the impact or accomplishment of the reported results. Please give us a specific practical scenario where LST would be useful.\n\n```\n[3] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177–2190, Online. Association for Computational Linguistics.\n[4] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 565–576, Online. Association for Computational Linguistics.\n[5] Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. \"Distilling the knowledge in a neural network.\" arXiv preprint arXiv:1503.02531 2.7 (2015).\n[6] Li, Hao, et al. \"Pruning filters for efficient convnets.\" International Conference on Learning Representation (2017).\n[7] Liu, Liyang, et al. \"Group fisher pruning for practical network compression.\" International Conference on Machine Learning. PMLR, 2021.\n```\n\n### Q2. Why not compare LST to other parameter-efficient methods?\nThis is a related question to Q1.\nOne of the most popular methods for transfer learning with reduction of training parameters and amount of memory is distillation. A method particularly related to LST is FitNets[8], which minimizes the Kullback-Leibler divergence between the intermediate layer outputs of source and target models. The difference between LST and FitNets is whether the output of the source model is directly used as the output of the target model. In contrast to LST, the trained models with FitNets do not require the source model during inference time. Is LST more effective than simple and lightweight transfer learning methods like FitNets?\nFurthermore, the more related study is LIT [2], a prior study that shares a source-target intermediate layer but also distillation. Is there any technical difference between LIT and LST other than the detailed learning techniques (initialization and Layer Dropout)?\n\n```\n[8] Romero, Adriana, et al. \"Fitnets: Hints for thin deep nets.\" arXiv preprint arXiv:1412.6550 (2014).\n```\n### Q3. What is the definition of \"network-pruning\" in Fig. 8?\nThe definition of \"network-pruning\" in L301-L303 is ambiguous. In which strategy is this model pruned in Fig. 7? What is the reduction factor for pruning? Is the pruned model being fine-tuned?\n", "limitations": "Nothing to report.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n  * Reducing memory usage during training time in comparison to Adapters and Prompt Tuning.\n\n### Weakness\n  * Novelty is limited since the idea of fusing intermediate layers is overlapped with an existing work (LIT, [2])\n  * The motivation and research questions are unclear.\n  * Lack of comparisons with baselines such as knowledge distillation, which are important for assessing the practicality of the proposed method.\n  * Rather increasing memory usage during inference time.\n  * Unclearness of experimental settings in particular the experiments of Fig.8.\n\n\nLST is certainly effective in reducing training parameters and memory usage, but its practicality can not be judged based on the current contents because there is no comparison with more simple and lightweight methods such as linear probing (a transfer learning method updating only the target-task-specific layers) and knowledge distillation. Furthermore, the main idea of LST which is to fuse intermediate layers of source and target models is not novel since LIT [2] has already proposed it at ICML2019. Considering the above, we believe that this paper is not yet ready for publication because the weaknesses outweigh the strengths.\n\n```\n[2] Koratana, A., Kang, D., Bailis, P. &amp; Zaharia, M.. (2019). LIT: Learned Intermediate Representation Training for Model Compression. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:3509-3518 Available from https://proceedings.mlr.press/v97/koratana19a.html.\n```", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "13h2MYaNu1M", "reviewer": "Reviewer_tNTu", "summary": "This paper proposes Ladder Side-Tuning to improve the memory efficiency of transfer learning. The core idea is to freeze the backbone while adding an extra trainable side network. The side network takes multiple features extracted by the backbone and produces the output. This paper also shows that using the weight from the backbone to initialize the side network and layer dropping are helpful for memory-efficient transfer learning. ", "questions": "See weaknesses. ", "limitations": "Limitations are discussed in the appendix. Discussions about negative social impact are missing.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Improving the memory efficiency of transfer learning is critical especially when we want to use large pre-trained models. \n2. The proposed method is simple and can be applied to different model architectures.\n\nWeaknesses:\n1. The overall idea of the proposed method is very similar to Side-Tuning [1]. While the authors argue that the proposed method has a different goal and focuses on different tasks, I still find the overall technical novelty of the proposed methods is a bit limited. \n2. In the related work section, I think the authors should add a section discussing the differences between the proposed method and previous memory-efficient transfer learning methods (e.g., [2,3]). In the experiment section, I think it is necessary to add direct comparisons between the proposed method and previous memory-efficient transfer learning methods. \n3. There are some general memory-saving techniques for deep learning, such as reversible neural networks [4,5,6], and checkpointing [7]. I think it is better to add discussions about these techniques.\n4. For the experiment section, I think the pre-trained model should be the same for all compared methods to ensure fair comparisons. To get results under larger memory constraints, the authors can increase the size of side networks. \n\n[1] Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks, ECCV 2020\n\n[2] Tinytl: Reduce memory, not parameters for efficient on-device learning, NeurIPS 2020\n\n[3] Rep-Net: Efficient On-Device Learning via Feature Reprogramming, CVPR 2022\n\n[4] The Reversible Residual Network: Backpropagation Without Storing Activations, NeurIPS 2017\n\n[5] Reformer: The Efficient Transformer, ICLR 2020\n\n[6] Reversible Vision Transformers, CVPR 2022\n\n[7] Training Deep Nets with Sublinear Memory Cost", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept