Consolidator: Mergable Adapter with Group Connections for Visual Adaptation
Authors: Tianxiang Hao
Hui Chen
Yuchen Guo
Guiguang Ding
Venue: ICLR 2023 poster (2023)
Keywords: Efficient Transfer Learning
Groups Connections
Vision Transformer

Abstract: Recently, transformers have shown strong ability as visual feature extractors, surpassing traditional convolution-based models in various scenarios. However, the success of vision transformers largely owes to their capacity to accommodate numerous parameters. As a result, new challenges for adapting a well-trained transformer to downstream tasks arise. On the one hand, classic fine-tuning tunes all parameters in a huge model for every downstream task and thus easily falls into an overfitting situation, leading to inferior performance. On the other hand, on resource-limited devices, fine-tuning stores a full copy of all parameters and thus is usually impracticable for the shortage of storage space. However, few works have focused on how to efficiently and effectively transfer knowledge in a vision transformer. Existing methods did not dive into the properties of visual features, leading to inferior performance. Moreover, some of them bring heavy inference cost though benefiting storage. To tackle these problems, we propose consolidator to achieve efficient transfer learning for large vision models. Our consolidator modifies the pre-trained model with the addition of a small set of tunable parameters to temporarily store the task-specific knowledge while freezing the backbone model during adaptation. Motivated by the success of group-wise convolution, we adopt grouped connections across the features extracted by fully connected layers to construct tunable parts in a consolidator. To further enhance the model's capacity to transfer knowledge under a constrained storage budget and keep inference efficient, we consolidate the parameters in two stages: 1. between adaptation and storage, and 2. between loading and inference. On a series of downstream visual tasks, our consolidator can reach up to 7.56 better accuracy than full fine-tuning with merely 0.35% parameters, and outperform state-of-the-art parameter-efficient tuning methods by a clear margin. Code is available at github.

Published as a conference paper at ICLR 2023
CONSOLIDATOR: MERGEABLE ADAPTER WITH GROUPED CONNECTIONS FOR VISUAL ADAPTATION
Tianxiang Hao Hui Chen Yuchen Guo Guiguang Ding!.3t
Schooo of Software, Tsinghua University *Department of Automation, Tsinghua University BBejjing National Research Center for Information Science and Technology (BNRist)
(beyondhtx, j Sheenhui2011 2, yuchen guo gmail com, dinggg@tsinghua.ccm
ABSTRACT
Recently, transformers have shown strong ability as visual feature extractors, sur- passing traditional convolution-base models in various scenarios. However, the success of vision transformers largely owes to their capacity to accommodate merous parameters. As result, new challenges for adapting a well-trained trans- former to downstream tasks arise. On the one hand, classic fine-tuning tunes all parameters in a huge model for every downstream task and thus easily falls into overfitting situation, leading inferior performance. On the other hand, on resource-limine limited devices, fine-tuning stores a full copy of all parameters and thus is usually impracticable for the shortage of storage space. However, few works have focused on how to efficiently and effectively transfer knowledge in vision transformer. Existing methods did not dive into the properties of visual features, leading to inferior performance. Moreover, some of them bring heavy inference cost though benefiting storage. To tackle these problems, we propose consolida- tor to achieve efficient transfer learning for large vision models. Our consolidator modifies the pre-trained model with the addition of a small set of tunable parame- ters to temporarily store the task-specific knowledge while freezing the backbone model during adaptation. Motivated by the success of group-wiss convolution, we adopt grouped connections across the features extracted by fully connected lay- ers to construct tunable parts in consolidator. To further enhance the model's capacity to transfer knowledge under a constrained storage budget and keep infer- ence efficient, we consolidate the parameters in two stages: 1. between adaptation and storage, and 2. between loading and inference. On series of downstream visual tasks, our consolidator can reach up to 7.56 better accuracy than full fine- tuning with merely 0.35% parameters, and outperform state-of-te of-the--rrt parameter- efficient tuning methods by a clear margin. Code is available at gigghbuub
INTRODUCTION
Recently, transformer architectures originated from natural language processing (NLP) Vaawanni et al., 2017) demonstrate considerable capacity in computer vision (Dosovitskiy et al. 2020 Tou- vron al. 2021 Liu et al., (021b). Vision transformers, along with traditional convolutional neural networks (CNNs) (Krizhevsky et al., 2012, He et al., 2016, Simonyan & Zisserman, 2014), are widely used as feature extractors to generate strong and general visual representations via deriving knowledge from massive images. Thanks to the abundant information in such representations, we can adapt the pre-trained models to downstream tasks by a simple fine-tuning strategy.
However, fine-tuning is not a good solution for adaptation. As is well known, the scale of vision models grows faster and faster in recent years. On the one hand, fine-tuning which tunes all param- eters in such huge model casily falls into overfitting situation. leading to inferior performance. On the other hand, fine-tuning inflicts heavy storage burdens. Since fine-tuning intensively tunes all parameters, it maintains a full copy of the model's 's parameters for each task. Therefore, fine-tuning can cause a huge storage burden when there are many tasks to be adapted, resulting impracticality in real-world scenarios, especially in resource-constee constrae- situations, e.g., embedded systems.
Corresponding authors.

---

Published as a conference paper at ICLR 2023
Efforts have been made to improve the performance as well as reduce the storage overhead of fine- tuning. For example, adapter (Houlsby et al., 2019, Karimi Mahabadi et al. 202I). prompt tun- ing (Li & Liang 2021 Lester al.] 2021; Zhou et al. (021) and LoRA (Hu et al. 2021) inject tunable parameters and freeze the backhone during adaptation. In the vision field, VPT (Jia et al.) 2022) directly leverage learnable prompts, AdaptFormer (Chen al. (022) adopts parallel adapter, NOAH (Zhang et al. (022) searches for the optimal combinations of the three representative mod- ules, i.e. adapter. LoRA, and VPT, and SSF (Lian et al. (022b) use additional scaling and shifting parameters for adaptation. Despite their acceptable performance, existing methods suffer from two common conflicts: trade-off between the inference efficiency and the adaptation performance, and 2. trade-off between the adaptation performance and the number of stored parameters. Previous works (Houlsby et al., 2019) show that introducing more tunable parameters can achieve more fruit- ful results. However, extra parameters can bring significantly larger computation and storage cost, resulting in low inference efficiency and more storage space. Therefore, one essential question is raised: can design module that can share the same inference cost an ordinary model while enjoying superior capacity against existing methods?
In this paper, propose generic module, dubbed consolidator, to tackle the aforementioned issues. The proposed consolidator is designed as a mergeable adapter that accompanies the fully connected (FC) layer in the vision models. Specifically, enrich the model capacity under limited parameter budget, we take inspiration from the success of group-wise convolution (Howard et al. 2017 Ma et al.] 2018 Liu et al., 2022) and build our consolidator as grouped connected (GC) layers. To enhance the flexibility, we further reorder channels for each group connection. followed by a droppath regularizer. Benefiting from the inference- linearity of GC, channel reorder, and droppath operations, the proposed consolidator can be perfectly consolidated into the original FC layer of a vision model, leading to no extra inference cost.
Our consolidator can be easily expanded as a buatii topology without breaking the linearity. Practically, we can simultaneously equip several GC layers with channel reordering for communi- cations between different groups of feature channels. After adaptation, we can first consolidate the multi-branch GC layers into one single sparse parameter matrix and store the sparse matrix for each task. Such property can enhance the model's transfcrability and achieve considerable storage duction when the number of tasks scales up. During inference, such a sparse parameter matrix can be merged into the backbone model well, resulting in inference cost. Thanks to the twice con- solidation, the proposed consolidator can greatly promote efficient and effective visual adaptation.
To verify the superiority of consolidator, we conduct extensive experiments and analysis of downstream recognition tasks. Experimental results show that our consolidator can surpass full fine-tuning by 7.56 top-1 accuracy with merely 0.35% parameters per task. Compared state-of- the-art methods, such as NOAH, AdaptFormer and SSF our method can consistently reach better performance while enjoying no inference cost, On other fundamental visual tasks, i.e., object detec- tion and semantic segmentation, our consolidator shows great power as well.
series
Overall, we summarize our contributions as follows. (i) We propose a basic module, dubbed consol- idator, for effective and efficient visual transfer learning. To enhance the transferability under limited tunable parameters, our consolidator is designed as mergeable grouped connected (GC) layer with channel reorder layer and a droppath regularizer. We extend the single branch to a multi-brand topology for better ficxibility and transferability.
(ii) We design a two-stage consolidation scheme by merging corresponding parameters in the training-storage phase and loading-inferenee phase. In this way, we can maximally dig the adaptation capacity of the model under a constrained storage budget, with extra inference cost. (iii) We conduct extensive experiments and analysis various downstream tasks. Results show that the proposed consolidator method can consistently outperform late-of-theeart methods with fewer stored parameters but superior performance.
2 RELATED WORKS
Parameter-efficien transfer learning. In the language field, works (Houlsby et al. 2019: Pieiffer
et al. 2021 Li & Liang 2021 Lester et al. 2021 Zaken et al. [021 Hu et al. [021] || Karimi Ma- habadi et al. 2021 Liu et al. 202 Ding et al. 2022a) have been done efficiently transfer the knowledge of pre-trained transformers to downstream language tasks. In the field of visual adap-
tation, several explorations have also been made to adapt vision transformers efficiently. Dia et al.
2

---

Published as a conference paper at ICLR 2023
Full Fine-buiing
Consolidator Tuning
Tooloagrosic di fixe penenstoes
liger
speiffic undd
No
Chazze Recodaigi
Chazad Raandelgg
Ieput
Adaptation
FC Layer
Layer
Cocsolidator
Doophath
Ouigst
Outpal
spentic parseeters
Merged
Charal Bouske lpa)
Storage
Baoker
Thilk speeticc infereece paranders
Inferenee
xe1
Figure 1: Consolidator tuning versus full fine-tuning. Consolidator adds tunable multi-branch grouped connected layers to the original fully connected layers. The tunable parameters are merged via addition into one single sparse matrix before storage to reduce the needed storage space. Between loading and inference, the parameters in the sparse matrix will be merged back into the original fully connected layer. Consolidator greatly enlarges the model's adaptation capacity under a constrained storage budget with no extra inference cost. Best viewed in color.
integrates addi- tional tunable convolution layers. NOAH (Zhang al. 2022) first trains a large supernet with three modules, VPT, LoRA, and adapter, and then searches for the optimal configurations of each module for every transformer block using evolution algorithm (Chen et 2021). AdaptFormer (Chen et al.) 2022) adds parallel adapters instead of serial ones. SSF (Lian al. 2022b) tunes additional scal- ing and shifting parameters for adaptation. is also shown that classic methods such as LoRA (Hu al. [021] and adapter (Houlsby al. 2019) can lead good performance for vision transformers. However, existing methods suffer from the two trade-off's as we discussed in Section resulting in difficulties in fully digging the adaptation capacity of vision models efficiently. To solve the prob- lems, we present mergeable adapter, named consolidator, and introduce a two-stage consolidation design to perfectly balance the trade-offs. leading to efficient and effective visual adaptation.
2022) and [Bahng al. (022) directly apply prompt-tuning, Jie & Deng 2022
Inference-efficient structures. Many works (Ding et al. 2019 Guo et al. 2020: Ding et al. 2 2022b) strive to design generic convolution architecture to realize superior capacity while enjoying inference cost. For example, RepVGG (Ding et al.) 2021c) integrates an extra Ix convolution to strengthen the main 3 3 convolution. However, existing methods are typically designed for CNNs. As for the popular vision transformer architectures, rare works investigate how to effectively strengthen their capacity while introducing no extra inference cost. LoRA (Hu et al, 2021) and SSF (Lian al., 2022b) offer possible solutions, but they do not explore the consoli- dation process between training and storage, leading to inferior performance under given storage budget. In this paper, adopt parallel GC layers replace the functionality of the original FC

---

Published as a conference paper at ICLR 2023
layers in vision models, which shows strong abilities for visual adaptation. Furthermore, expand the existing one-stage
inference consolidation to a two-stage process: 1. training-storn storage consolidation, and 2. loading-inferenee inference consolidation. Such two-stage design maximally dig the adaptation capacity of the pre-trained model under a constrained storage budget, with no extra inference cost. Extensive experiments show that consolidator outperform state-of-the-an methods in both the number of tunable parameters and the adaptation performance.
METHODOLOGY
3.1 PRELIMINARIES
In this paper, we mainly focus on the adaptation for vision transformers (Dosovitskiy et al. 2020 Liu et al., 2021b). A typical vision transformer (Dosovitskiy al. 2020) consists of L serial blocks. In each encoder, there are a multi-head self-atiention module (MHSA) and multi-layer percep- tron (MLP). Formally, batch of imput images Xinput E R BBx3xxxx HxW will be first reshaped into a sequence of flattened 2D patches Xp E IR BB where C' is the number of channels and (P. P) is the resolution of each patch, and N = NW/P22 the number of patches. Then the patches are mapped to D channel dimensions with linear projection. Next, a classification X₁ Bx(NN+)) to de-
ken is appended and we get X1 E IR ())+)))) Here note the input of 1-th (1 < S L) block. Its output X(+1 = x + MLP(LayerNorm(x)) where For MHSA, the input features first processed by three FC layers to generate matrices Q, K, V. and the output is calculated by Softmax QK* V and then pro- vá
x/ = x MHSA(Laye/Norm(x))
jected by another FC layer. Therefore, the parametric components of MHSA four FC layers. The parametric components of MLP are two FC layers as well. Therefore, we formulate our consolidator for the FC layers (see Fig. D covering all the parametric components in each MHSA and MLP. We will show that such a design can realize both efficiency and effectiveness in Section Notably, our method is also applicable for MLP (Lian al., 2022a) and CNN (Liu al., 2022 and can reach good results as in Tab.
3.2 CONSOLIDATOR
For efficient transfer learning, we merely tune and store the parameters in consolidators while freez- ing other parameters in the pre-trained model. In this subsection, we will introduce our design of consolidator, an efficient and effective module for adapting vision transformers, in detail.
Grouped connections. Inspired by the success of group convolution in extracting visual features, we hence assume that the cross-channel information exchange is redundant in visual adaptation, and aim to design consolidator by reducing the cross- cross-channel connections between sequential features to minimize the number of stored parameters for downstream tasks while keeping maximum capacity. Therefore, for each FC layer, we add concurrent module consisting of grouped connected layer. Formally, for an input x E R,P, the output x' E R of a GC layer with group g. weight W E ROx Ex and bias b R.F. is formulated by x' GC(x) = ビ₁ Pad X DD 11,,., + b.
Here ja prepends (1-1)D zeros and appends (g-j)D zeros for Z according another input j. In this way, the output channels the j-th group only interact with the input channels in the j-th group, and thus we reduce the cross-channel connections as expected.
9
To flexibly reach different ratios of stored parameters, we adopt a multi-branch topology in our x 25 and bias b, E E for consolidator. There GC layer with weight w E TR. " x 200
th branch with group (f)), During adaptation, consolidator and the original FC layer take the same input and their outputs are summed up to produce the new output y. Formally, for each FC layer with weight W
R ExD and bias E the output of the whole layer modified by m GC branches is
y Wx + b D gd Pad (ad( "x-Do 1D j) (()).
Channel reorder. To flexibly tune the total number of parameters and enrich the exchange of information flow, prepend a "ChannelReorder" operation every branch in consolidator by
4

---

Published as a conference paper at ICLR 2023
manually adjusting the permutation of input features along the channel dimension. In general, adopt shuffle operation (Zhang et al. 2018 Ma et al. 2018) to accomplish such purpose.
Formally, given imput E R* where "*" means any number of dimensions including none, shuffle it into 9 groups and perform recombination across the last dimension. Formally, we first reshape x into x' E Ix**xx 9, and then transpose the last two dimension and get x" I xg, and then reshape x" into x" E R'XD, which the final output. A pythonic style formulation is ChamneReeorder(g.. x) (x.reshape(+, 9. )) transpose - -.-ross---- D)
We set shuffle groups g 9i in the i-th branch, where 9i is the group of the corresponding GC layer. In this way, there are few overlaps between the weight matrices of distinct branches, and the model capacity is greatly expanded each time new branch is contained.
Stochastic depth of pre-trained weight. To further enlarge the model's adaptation capacity, we ap- pend droppath (Huang al., 2016) layer each branch. For small downstream datasets, dropping the consolidator path with a higher ratio p can help reduce overfitting and catastrophic forgetting, which may be beneficial the performance. Empirically shown in Section 4.8, droppath more effective than standard dropout (Srivastava et al. 2014) in the current situation probably for the fol- lowing reasons. The parameters of the whole layer degrade into the pre-trained weight parameters with probability The frozen pre-trained parameters contain domain domain-generie knowledge, which may help the adaptation. Overall, model modified by consolidator will have stochastic depth of pre-trained state of parameters during each forward pass, and different consolidators will be acti- vated for different training samples.
Two-stage consolidation. Now we have constructed all the elements of consolidator: For- mally, the output of the whole layer after being modified by consolidator is y = Wx + + Droppath E (E)r [ ChannelReorder((
(9)
U-10 JD () () bb))).
9(
Since all the operations in a consolidator are inference-time linear, we can easily consolidate the domain-specific knowledge into the domain-agnostie knowledge in the pre-trained backbone model, in both the training-storag phase and the loading-infe -infeennn phase.
1. Training storage consolidation. All we need to store in a consolidator are w( and b (0) How- ever, there are some parameters corresponding to the same entry, and thus they can be merged into single one. As shown in Fig. [ we also tune the bias of the original FC layer in addi- tion to the parameters in consolidator. It is easy to find that the duplicate biases in all branches and the original bias can be merged into single one. And there are some duplicate entries in the weight matrix as well, so we can merge all weight matrices into one single sparse ma- trix. Consolidating such duplicate entries can largely benefit storage. Formally, we use W and b to denote the matrix that we need to store on the disk. Since channel reorder is linear op- eration, we can apply a reverse operation to w to simulate the effect of the reorder applied Compact(W(()))
to And we have W Σ m shapes the input matrix for the preparation of reordering its channels. It is easy to verify that ChannelReorder" (g(()
Channe
Here Compact re-
ChannelReorder((.
y(+)
2. Loading-inference consolidation. After loading the merged sparse weight matrix and merged bias matrix to memory, we can directly add them back to the weight matrix and bias matrix of the original FC layer. Formally, use W and b to denote the final weight and bias of the FC layer for inference. Then W = W W, b b. In this way, no additional inference cost is brought.
Overall, our consolidator reduces the storage space by using grouped connected layers and consol- idating some duplicate entries. The training time non-linearity, e.g., droppath, which turns out to be linear in inference time, effectively enriches model capacity under constrained storage budget. Finally, we can consolidate the task-specific knowledge into the backbone model by merging the inference time linear components to enjoy free, efficient, and effective transfer learning.
5

---

Published as a conference paper at ICLR 2023
Table I:
Full results on the VTAB-Ik Zhai et al. 2019) benchmark. The bold font denotes the best accuracy and the underline font denotes the second best accuracy in each column. Consolidator gives the strongest results, surpasses full fine-tuning by 7.56 accuracy on average, and outperforms the state-of-the-ar methods with low storage overhead and inference cost.
Natural
Specialized
Structured
SNORB Azim
purams
Cifar 100
Caltech10l
Flowend H12
SVHN
Sun397
Camelyon
EuroSAT
Rosiee4
Retinopatht
Cleve-Count
ClewsDist
DMLAB
KITTEDM
dSpr-Loc
JSpr -0i
&NORD
Avergo
DTD
Pets
Full
100%
66899
87.7
64.3
97.2
86.9
87.4
38.8
79.7
95.7
84.2
73.9
556.3
58.6
41.7
65.5
57.5
46.7
25.7
29.1
68.97
Head
0.04%
63.4
63.2
36.6
78.5
87.5
74.0
34.3
33,2
55.4
20.0
57.64
Bias
0.10%
72.8
87.0
59.2
97.5
85.3
59.9
51.4
78.7
91.6
72.9
69.8
61.5
55.6
32.4
55.9
66.6
40.0
15.7
25.1
65.22
VPT
0.75%
78.8
90.8
65.8
98.0
88.3
78.1
49.6
81.8
96.1
83.4
68.4
68.5
60.0
46.5
72.8
73.6
47.9
32,9
37.8
71.97
Adapter
68.8
85.7
95.5
76.1
49.7
LoRA
0.34%
68.1 90.1
69.8
98.9
90.8
84.8
54.0
83.2
95.6
84.2
73.9
82.4
68.7
49.4
80.0
81.7
46.1
31.4
41.8
74,64
AdaptFormer
0.36%
71.0 91.1 69.9
99.3
90.5
87.4
54.8
84,1
95.9
85.9
75.8
83.1
63.8
49.6
79.6
76.5
45.1
30.9
39.2
74.82
NOAH
0.52%
69.6
92.7
70.2
99.1
90.4
86
53.7
84.4
95.4
83.9
75.8
82
68.9
49.9
81.7
81.8
48.3
32.8
44.2
75.48
SSF
0.29%
69.0
92.6
75.1
99.4
91.8
90.2
52.9
87.4
95.9
87.4
75.5
75.9
62.3
$3.3
80.6
77.3
$4.9
29.5
37,9
75.69
Ours
0.35%
74.2
73.9
91.5
95.7
51.6
83.5
52.3
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Baselines. We select several state-of-the-an parameter-efficient methods as our baselines, includ- ing Full, Head, Bias (Zaken et al. 2021). Adapter (Houlsby et al. 2019), VPT (Jia et al. 2022). LoRA (Hu et al. 2021). AdaptFormer (Chen et al. 2022). NOAH (Zhang et al. 2022) and SSF (Lian et al. 2022b). Note that Adapter, FdaptFormer, VPT, and NOAH will bring heavy inference cost to the pre-trained model which may cause troubles
sesource-limitens devices, while LoRA, SSF and our consolidator more friendly for deployment with no extra inference cost brought.
VTAB-lK. We first run experiments on VTAB-1k (Zhai et al., 2019) benchmark, which covers a wide range of visual domains in 19 datasets. Each dataset contains 1000 images picked from the original dataset for training, and the size of test set stays unchanged, varying from 711 to 73728.
Full data setting. VTAB-IK merely has small number of training images, and thus the capacity of a huge transformer is redundant to some extent. Therefore, to check the performance of parameter- efficient methods in a data- sufficient situation, we select 10 widely-used datasets for visual recog- nition in various domains and use the original training, validation, and test split for experiments. On average, dataset in this setting contains lot more images for training. leaving huge space for the model to adapt. The chosen datasets include natural pictures (Caltech101 (Fei-Fei et al. 2004), Cifar10 (Krizhevsky al.) 2009). Cifar100 (Krizhevsky et al., (2099). fine-grained classification (CUB200 (Wah al. 2011), OxfordFlower (et al. 2012), StanfordDogs (Khosla et al. 2011)), textures (DTD (Cimpoi et al. (0144), scene clas- sification (SUN397 (Xiao al. 2010)) and satellite images (EuroSAT (Helber al. 2019)).
(Nilsback & Zisserman, 2008), OxfordPets (Parkhi
4.2 MAIN RESULTS
We first choose a VIT-B (Dosovitskiy et al., 2020) with 86M parameters as a base model.
VTAB-1k Tab. presents the full results on VTAB- benchmark. Overall, our consolidator the best parameter-efficient method. On 12 of 19 datasets, consolidator achieves the best or second best top-1 accuracy. Notably, consolidator surpasses the state-of-the-art methods NOAH and SSF by clear margin, with low storage overhead and no inference cost.
Full data setting Tab. presents the full results on full data setting. Overall, our consolidator still performs best. An interesting observation is that the rank of full fine-tuning rises as the training data increase. None of the full tine-tuning other than consolidator within 0.5% parameter storage overhead. In contrast, the efffccieneeficientt methods can reach at least 5% higher accuracy on VTAB- 1k than full fine-tuning under comparable even lower storage budget (around 0.5%), shown in Tab.
methods can reach comparable performance with

---

Published as a conference paper at ICLR 2023
Table 2: Full results on data-sufficieet scenarios. It is more challenging to take full advantage of large amount of data within a fairly small number of stored parameters. Consolida- tor earns the best second best in all 10 datasets.
Table 3: Adaptation results for a self-supervised model, MoCo v3 ViT-B, Consolidator earns the best average accuracy
VTAB-IK
Full data
* params
Caltech111
Cifart0
Clar100
Flowers 102
Sun397
EuroSAT
Average
params
Average
params
Average
CUB
D71D
Pets
Dogs
Full
100%
86,24
Full
100%
90.9
99.1
92.6
87.7
75.6
98.9
93.9
84.8
70.9
98.6
89.30
Head
0.04%
59.62
0.10%
75.42
Head
0.10%
71.3
90.5
67.0
Bias
0.10%
69.15
0.22%
80,96
Bias
0.22%
90.2
99.0
91.9
86.4
73.5
98.5
92.6
85.4
71.0
97.7
88.62
Adapter
0.36%
73.22
0.55%
86.15
Adapter
0.55%
91.0
99.0
92.2
86.3
72.6
98.7
92.5
85.9
71.9
97.4
88.75
LoRA
0.34%
0.53%
79.70
LoRA
0.53%
99.0
71,9
92.1
71.3
AdaptFormer
0.36%
74.03
0.55%
86.26
Formerrrmer
0.55%
90.9
99.0
92.3
86.5
73.4
98.9
92.9
85.2
71.4
97.1
88.76
NOAII
0.42%
73.55
SSF
0.30%
90.9
99.0
92.1
85.7
72.6
98.3
92.3
85.7
71.6
97.8
88.60
SSF
0.29%
51.41
0.30%
80.73
Ours
0.50%
91.4
99.1
92.5
87.0
24.5
99.0
93.2
86.4
72.1
97.9
89.31
Ours
0.35%
74.71
0.50%
86.41
Table 4: Adaptation performance for more models in full data setting. Consolidator consistently reaches a better result than full fine-tuning within a very small set of parameters which is needed to be stored. Generally, it is more difficult to reach better result than full fine-tuning when the model capacity is insufficient, ie.. the model does not accommodate enough parameters in total.
Supervised Learning
MAE
Architecture
VIT-S
VYT-I.
Swin-B
AS A--MI.PBB ConvNeXI-B
VIT-B
Total Parameters
22M
303M
87M
87M
88M
86M
Averuge
89.10
90.07
91.00
87.75
91.92
82.77
Full
parums
100%
100%
100%
100%
100%
100%
Average
81.87
87.92
90.45
83.49
90.62
58.10
Head
parums
0.20%
0.04%
0.13%
0.13%
0.13%
0.10%
Average
87.86
89.81
21.00
86.26
91.61
79.80
params
0.44%
0.13%
0.36%
0.32%
0.28%
0.22%
LoRA
Average
87.55
89.86
90.95
82.22
params
5.12%
0.33%
0.80%
1. 1.82%
Average
88.28
89.91
90.98
82.89
Adapter
params
5.09%
0.35%
0.78%
1.80%
Average params
89.12
90.52
91.28
86.71
91.79
83.32
Ours
5.06%
0.33%
0.77%
1.13%
1.04%
1.78%
4.3 RESULTS FOR SELF-SUPERVISED VISION TRANSFORMER
Then we use a self-supervised trained transformer by MoCov3 (He et al. 2020) as our target model. Seen from Tab. 3] both VTAB-1k and full data setting, consolidator consistently reaches the highest average top-1 accuracy. AdaptFormer gives the second highest accuracy. SSF significantly falls behind others (51.41 on VTAB and 80.73 on full data setting) when dealing with MoCo v3 ViT-B, showing limited generalization ability for self-supervised visual models.
4.4 RESULTS FOR MORE PRE-TRAINED MODELS
To further verify the generalization ability of consolidator, we conduct extensive experiments in Tab. based on full data setting. First, we apply consolidator to supervised learned models with larger (ViT-L) or smaller (VTT-S) size than standard ViT-B. Compared with full fine-tuning, we achieve a comparable performance for VIT-S with 5.06% parameters. For ViT-L, merely 0.33% parameters can lead to 0.45 higher than full fine-tuning, Then we experiment Swin-B, a hierar- chical architecture using shifted windows to introduce locality for better recognition. We observe a 0.28 improvement while storing only 0.77% parameters. Next, we further verify the effectiveness of consolidator on other vision architectures other than transformers, e.g. AS-MLP (Lian et al.) (202a)) and Co (Liu et al. 2022). Finally, we experiment VIT-B pre-trained by generative SSL
7

---

Right: adapta- Published as a conference paper at ICLR 2023
Addaptatoon result vacying
Addaptaton results varying data sumpling
-o-Pai
-Q- Lo-A
LeMA
1.%
10.0%
130%
4.5% 12.59% 2540%
50
100.0%
Sampling
Figure 2: Left: the Adaptation results corresponding to varying ratios of stored parameters. Clearly, consolidator consistently outperforms LoRA, adapter, and full fine-tuning by significant margin across a wide range of parameter scales from 0.5% to 10%. And each of the three methods will reach higher accuracy if we increase its storage budget and tune more parameters. tion results corresponding to varying sampling rates of data. Consolidator performs best across all sampling rates. As the sampling rate decreases, full fine-tuning slightly falls off its advantage over adapter and LoRA, which is consistent with our previous observations.
method, MAE, as a comparison with the contrastive SSL method MoCo v3. Our method stores 1.78% parameters onto the disk while enjoying 0.55 higher accuracy.
Generally, it is more difficult to perfom better than full fine-tuning when the model does not have enough parameters to fully leverage the information from massive training data.
4.5 ADAPTATION ACROSS VARYING SCALES OF STORED PARAMETERS
Next, we seek to find the principle between downstream accuracy and the number of stored param- eters for all parameter-efficient methods with flexible parameter scales, based full data setting. Results are shown in Fig. left. LoRA, adapter, and consolidator reach better performance as the number of parameters increases. In various parameter scales (from 0.5% 10%), our consolidator consistently outperforms all competitors by a clear margin.
4.6 ADAPTATION ACROSS VARYING DATA SAMPLING RATIOS
Fig. right shows adaptation results corresponding to varying sampling rates of datasets, based on full data setting. For all sampling rates, consolidator keeps the best adaptation ability. In addition, as the sampling rate decreases, full fine-tuning slightly falls off its advantage over Adapter and LoRA. which is consistent with previous observations on VTAB-IK Ik and full data setting.
4.7 RESULTS ON OBJECT DETECTION AND SEMANTIC SEGMENTATION.
We further verify method downstream object detection and semantic segmentation tasks. We adopt Swin-Base as the backbone which can provide hierarchical features. Experiments are done on PASCAL VOC 07+12 (Everingham et al., 2010) for detection and PASCAL VOC 12 (Everingham 2010) for segmentation. We adopt Faster R-CNN (Ren et al. 2015) and UperNet (Xiao et al.) 2018 separately for detection and segmentation framework. Seen from Tab. 5 our consolidator sigmificanlly outperforms full training and detection/segmmettale head training with small number of parameters stored, showing great potential for broader usage.
et al.
4.8 ABLATION STUDIES
We do controlled experiments to identify the effect of individual components our module design. We report tuned parameters, stored parameters, and accuracy in Tab. 6 We experiment 5 datasets with different domains: Caltech101, DTD, OxfordFlowers, StanfordDogs, and EuroSAT.
Droppath F.s. Dropout. We first investigate our choice of Droppath. Droppath with a consolidator of (g) 96, (2 192) the base model. As shown in Tab. 6 compared with dropout and no drop-type layer, droppath can obtain 0.44 and 0.48 performance improvement, respectively, well demonstrating the effectiveness of encouraging stochastic depth for consolidator.
8

---

Published as a conference paper at ICLR 2023
Table 5: Performance on downstream object detection and semantic segmentation tasks. Compared with tuning detection/'semmenttionn head only, consolidator reaches much better mAP/mIoU with negligible parameters increase and surpasses the performance of tuning all parameters in both back- bone and task head as well.
Object Delection
Semantic Segmentation
Method
Backhone
Framework
paramns
mAP
Framework
params
mloU
Full
100%
85.85
100%
83.24
Head
Swin-B
Faster-RCNN
16.74%
84,89
UperNet
28 .8%
82.31
Consolidator
86.64
28.89%
83.96
Table 6: Effect of individual designs for consolidator module.
Original Bias
Extra
Channel Reorder
Tuned Params
stored
Branches
Dropout
Droppath
Accuracy
Accuracy
Bias
Params
96, 192
X
1.95%
1,75%
87.36
-0.00
1,95%
1.75%
192
1.95%
1.75%
86.88 86
-0.48
96, 192
/
X
X
1.95%
1.24%
86.98
00.38
96, 192
185%%
1.75%
87.26
-0.10
1.76%
1.75%
86.92
-0.44
96, 192
V
1.66%
1.65%
86.82
-0.54
384
0.56%
0.47%
87.08
-0.00
0.56%
0.47%
86.98
384,38
0.92%
384
0.37%
0.37%
86.84
-0.00
384 (onst)
X N/A
X
0,37%
0.37%
86.70
-0.14
ChannelReorder. The effect of ChannelReorder operation mainly lies in separating the entries into
different branches to reduce the repetitive ones. In mullt-bbanch case like (g( 96, g(2) 192), ChannelReorder brings 0.38 accuracy improvement. Furthermore, it is helpful even there is only one branch like (g 384), where ChannelReorder still slightly raises the accuracy by 0.1.
Duplication of bias and weight. Based the consolidator with (g 96, 192), can see duplicating bias is relatively effective. Compared with tuning the original bias and only tuning two extra biases, tuning all three biases can lead 0.44 and 0.1 performance improvement, respectively, with the same storage cost. Additionally. we also compare tuning original bias v.s. not tuning bias, and the former only has a slight 0.1 accuracy advantage, which further verifies the effectiveness owes mostly to the delicate consolidation design instead of simple bias tuning. Besides bias, we test on a consolidator with (g) 96) to investigate the effect of integrating duplicate weights. However, this kind of consolidation does not bring notable improvement.
Structured Unstructured. One potential limitation of consolidator is that g()) not be selected arbitrarily for it must be a factor of the channels. This may cause trouble when fairly few parameters other than head parameters, e.g. 0.0001% % are required be tuned. A solution is adopt unstructured sparsity instead of structured block-wise sparsity in consolidator branches to flexibly control the parameter number. We simulate the situation with (1) = 384) for comparison with a unstructured implementation. Seen from Tab. 6 the unstructured branch whose tunable parameter number equals that of branch with = 384 faces slight accuracy drop, 04.. In summary, the unstructured consolidator can be sub-optimal choice when needed, with a slight performance drop and more training cost due to the unstructured sparse matrix being unfriendly to the hardware.
CONCLUSIONS
We propose consolidator, a novel method achieve both parameter- and inference-efficint visual adaptation. Consolidator adds few tunable, mergeable modules along each fully connected layer the pre-trained model and keeps most of the original parameters frozen during adaptation. We design two-stage consolidation The duplicate entries in consolidator will be merged into a single matrix and stored on disk. Finally, consolidate the task task-specific parameters in consolidator into the tasks-agnostic parameters in the pre-trained model. bringing no extra inference cost. On various tasks, consolidator outperforms all state-of-the-. oftthe-artttttte competitors significantly, showing strong scalability and generalization ability.
dramatically boost performance under given storage budget.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by National Key R&D Program of China (No. 0021ZD0114703) National
Natural Science Foundation of China (Nos. 31925107, 6227 U1936202, 61571269), Beijing Natural Science Foundation (No. L223023), China Postdoctoral Science Foundation (BX2021161) and Zhejiang Lab Open Research Project (NO. K2022K10AB01).
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: crXiv:160.00645 2016.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 2022.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:I906.0711 07155, 2019.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling, Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12270-12280, October 2021.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. / htt.ttttttha..g./ wwwp:o/woooooooooooooooooeeoe mmm ab//mmsegmentttioo. 2020.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin
Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.0m6 066904 2022a.
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kemel skeletons for powerful cnn via asymmetric convolution blocks. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 1911-1920, 2019.
Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Resrep: Lossless enn pruning via decoupling remembering and forgetting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4510-4520, 2021a.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Diverse branch block: Building convolution as an inception-like unit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10886-10895, 2021b.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma. Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vtg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp.
2021c
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11963-11975, 2022b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2023
M. Everingham, L. Van Gool, C. Williams, J. Winn, and A. Zisserman. The paseal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303 June 2010.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Shuxuan Guo, Jose M Alvarez, and Mathieu Salzmann. Expandnets: Linear over-parameterrizaton to train compact convolutional networks. Advances in Neural Information Processing Systems, 33, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-97229 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks and Kevin Gimpel. Gaussian arXiv:1606.084155 2016.
error linear units (gelus). arXiv preprint
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 174..4174 2017.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.0900 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203./2119 2022.
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey. arXiv preprint arXiv:2201. 055677, 2022.
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses better vision transformer adapters. arXiv preprint arXiv:2207.07033, 2022.
Bo Fu Junguang Jiang, Baixu Chen and Mingsheng Long. Transfer leateelg--nniibbien https: 2020.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34, 2021.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In First Workshop on Fime-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition. Citeseer, June 2011.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
11

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky, Ily Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effiiiet prompt tuning. arXiv preprint arXiv:2 104006691 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint
101. 000190, 2021.
Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architecture for vision. In International Conference Learning Representations, 2022a.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A baseline for efficient model tuning. Advances in Neural Information Processing Systems, 2022b.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 202
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021b.
Zhuang Liu, Hanzi Mao, Chao-Yure Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.03544 2022.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131,00 2018.
Maria- Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1988, 2014.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 JEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings the 418 434, 2018.
the European conference on computer vision (ECCV), pp-
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint 1rXiv:2106.10199 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv 2019.
preprint
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6848-6856, 2018.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv:2206. 220..00637, 2022.
arXiv preprint
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. arXiv preprint arXiv:2109.01/344 2021.
A EXPERIMENT SETTINGS
A.1 IMAGE RECOGNITION
A.1.1 DATASETS
VTAB-IK. VTAB-1K 1k (Zhai et al. 2019 consists of 19 visual datasets in three groups: Natural, Specialized, and Structured, containing images collected from wide range of visual domains. Each dataset is divided into the training set (800 images), validation set (200 images), and test set (the original set). The final adaptation accuracy reported on the test set. All models are fine-tuned on the full 1000 labeled images, i.e., train+val set. The validation set is used for tuning some hyperparameters.
Full data. We select 10 widely-used visual datasets with sufficient data to further verify the effec- tiveness of the parameter-efficien methods. The datasets are shown the following part:
Caltech101 (Fei-Fei al.) 2004) contains pictures of natural objects belonging 101 classes, with 3060 images for training and 6084 images for testing.
CIFAR-10/100 (Krizhevsky al. 2009) contains pictures of natural objects belonging to 10/100 classes, with 50000 images for training and 10000 images for testing.
Caltech- UCSD Birds-200-201 (CUB) (Wah al. 2011) contains pictures of birds belonging 200 fine-grained bird classes, with 5994 images for training and 5794 images for testing.
Describable Textures Dataset (DTD) (Cimpoi et al. 2014) contains texture images in the wild be- longing to 47 human cenaric classes, with 5640 images in total which are equally split into the training, validation, and test
13

---

Published as a conference paper at ICLR 2023
OxfordFlowers/02 (Nilsback & Zisserman 2008 contains pictures of flowers belonging to 102 fine- grained flower classes, with 1020 images for training. 1020 images for validating, and 6149 images for testing.
Oxford-IIITPE Iltts (Parkhi et al.
2012) contains pictures of pets belonging to 37 fine-grained pet classes, with 3680 images for training and 3669 images for testing.
StanfordDogs (Khosla et al. 2011 contains pictures of dogs belonging to 120 fine-grained dog classes, with 12000 images for training and 8580 images for testing.
Scene UNderstanding (SUN397) (Xiao et al. 2010 contains pictures of scenes belonging to 397 classes, with 76128 images for training, 10875 images for validating, and 21750 images for testing.
EuroSAT (Helber et al. 2019) contains pictures based on Sentinel-2 satellite images belonging to 10 classes, with 27000 images in total.
Some of the datasets do not contain a validation split, and we will manually select 10% random images from the training set the validation set.
To better utilize the massive data various domains, we follow the practice in (Junguang Jiang & Long 2020 Jiang al.] 2022) to make preparation and divide data splits. For data augmentation, adopt a standard pipeline. In training, we do a random resize crop to 224 x 224, random horizontal flip, and normalization for each input image. In test, we do resize x56, center crop to 224 and normalization for each input image.
A.1.2 TRAINING HYPERPARAMETERS
On VTAB-1K, follow the hyperparameters in VPT (Jia et al. 2022) for full fine-tuning, Head, Bias, and VPT, and mainly follow the hyperparameters in NOAH (Zhang et al. (022) and SSF (Lian al. 20226) for adapter, LoRA, NOAH, and consolidator. The detailed hyperparameters for each tuning method can be found in Tab.
On full data sefting, we do quick grid search choose proper set of training hyperparame- ters based on the performance of full fine-tuning for every well-trained visual representation. All training hyperparameters are shown in Tabs. a Following the practice in (Jiang et al., 2022 Junguang Jiang & Long 2020), we reduce the learning rate of backbone parameters to 0.1x that of head parameters. We then adopt the same hyperparameters for all parameter-efficient variants as well as our consolidator after the hyperparameters have been chosen according to full fine-tuning results.
A.1.3 METHODS.
Full: Ordinary fine-tuning. It tunes all parameters and stores all parameters to disk for every down- stream task, leading to heavy storage cost.
Head: Also known as linear probing. It only tunes the classification head and freezes other parame- ters.
Bias Zaken et al. ) Bias only tunes the biases and freezes all the weights in pre-trained model.
Adapter (Houlsby al. 2019): Adapter inserts sequence of tunable parameters, including down projection, a non-linearity (here we use GELU (Hendrycks & Gimpel 2016)) and an up projection layer, into each encoder block. The adapters are serially connected with the backbone layers.
LoRA (Hu et al. 202T]: : LoRA adds a concurrent branch containing low-rank weight matrices for
efficient parameter update. A LoRA module contains serial down projection and up projection layers without non- non-lineante con-lincrrtty, and thus it can be merged into the backbone parameters before inference.
VPT appends tunable virtual tokens to the inputs of transformer blocks, which will participate in the calculation of the subsequent blocks along with the actual tokens.
VPT Li & Liang 2021:
Jia al. 2022):
NOAH (Zhang et al. 2022): NOAH first trains a large supernet with all three modules, VPT, LoRA, and adapter, and then searches for the optimal configurations of each module for each layer using
14

---

Published as a conference paper at ICLR 2023
the NAS algorithm introduced by AutoFormer (Chen et al., 2021). In the end, NOAH retrains the best subnet candidates to produce the final result.
AdaptFormer (Chen al. 2022):
AdaptFormer adopts parallel adapters (Houlsby al. 2019) and a scale operation for each encoder block.
SSF (Lian et al. 2022b): SSF adds tunable scale and shift parameters for each operation of the backbone model. The added parameters can be merged into the original model and thus SSF brings no inference cost.
Implementation details, On VTAB-1k, we follow the same implementations for LoRA and adapter with NOAH (Zhang et al. 2022).
On Full data setting, the detailed implementations are shown as follows. For LoRA (Hu et al, 2021]. we follow its original implementation to do low-rank p-paammttrrization and tune merely Two weight matrices W and W., which generate the attention matrices Q and V, in every trans- former encoder block. For bias,
all the biases (including the parameters outside encoder blocks) in the model. For adapter, we follow its original implementation to additionally tune the parameters of LayerNorm (Ba al., 2016) and add the tunable adapter modules the end of each MHSA and MI.P before calculating the residual connections. For consolidator, we follow the prac- tice of adapter tune the LayerNorm as well and only add consolidator to the linear layers in MHSA and MLP of each transformer encoder block. We sweep the drop ratio in {0.0, 0.2, 0.5, 0.8). We choose training hyperparameters according to the performance full-finetuning for each visual rep- resentation. The hyperparameters are finally configured in Tab. 7 When applying consolidator to Swin-B, we skip the first stage of transformer encoder blocks that contain a relatively small part of parameters (0.6% of total parameters) compared with others and merely insert consolidator for linear layers in MHSA and MLP of blocks of last 3 stages.
A.2 OBJECT DETECTION
On the downstream object detection task, we adopt Faster R-CNN (Ren al. 2015) framework with 2017) to verify the effectiveness of consolidator on Pascal VOC 07+12 dataset (Ev- eringham et al. 2010). We use a Swin-B pre-trained on IN-2 Ik as the backbone model. The consol- idtoo setting is the same the setting of Swin-B in Tab. For hyperparameters, we adopt AdamW as the optimizer with a learning rate of le-4 and weight decay of 5e-2, and train for 8 epochs in to- tal. The learning rate is decayed by factor of 10 after 6 epochs. The first 300 iterations trained with a warmup ratio of 1e-3 for the learning rate. The data augmentation is the same as the default strategy in mmdetection (Chen et al. 2019)
FPN (Lin et al.
A.3 SEMANTIC SEGMENTATION
On the downstream semantic segmentation task, adopt UperNet (Xiao al. 2018) to verify the effectiveness of consolidator on Pascal VOC 12 dataset (Everingham et a 201O). use a Swin-B pre-trained on IN-21k as the backbone model. The consolidator setting is the same the setting of Swin-B in Tab. 7 For byperparameterss, we adopt Adam as the optimizer with a learning rate of 6e-5 and weight decay of 1e-2, and train for 20000 iterations in total. The learning rate is decayed by a polynomial scheduler with a power of 1.0. The first 200 iterations are trained with a warmup ratio of 1e-6 for the learning rate. The data augmentation is the same the default strategy in mmsegmentation (Contributors) 2020
B TRAINING AND INFERENCE COST
Many the classic parameter-efficien tuning methods, e.g. adapter (Houlsby et al., 2019) and Adapt- former (Chen et al., 2022) introduce non-negligible extra cost in inference period and thus slow down the processing speed. In contrast, consolidator tuning shares identical structure with the orig- inal model and bring no extra inference cost. We quantitatively show the training cost and inference cost across different parameter scales in Fig. 3] Here do not show the cost of SSF for it can not be adapted to different scales of parameter budget. In addition, VPT and NOAH both search for an optimal structure from large searching space and thus it is hard to fairly measure their cost as well.
15

---

Published as a conference paper at ICLR 2023
Table 7: The method-related hyperparameters.
model
Setting
LoRA
Adapter
Consolidator
VTAB-IK 1k
rank=8
hidden=16
(384)
Supervised, MoCo v3
VIT-B
Full data
rank=10
hidden=9
(314))
Supervised
VII-S
Full data
rank-58
hidden=56
[g (3)
= (9/=6))
Supervised
VIT-L
Full data rank=9
hidden=8
(g(*)=112
Supervised
Swin-B
Full data
rank-12 12
hidden=10
(x(*)=256)
Supervised
AS-MLP-B
Full data
(112)=18)
Supervised
ConvNeXi-B
Full data
(y(8)=18)
MAB
VTT-B
Full rank-40)
hidden
(g()) m3))
Table 8: The training hyperparameters full data setting.
Supervised
MAE
MoCo
model
ViT-S
VIT-L
Swin-B
AS-MLP-B
ConvNeXI-B
VIT-B
ViT-B
pretraining dataset
IN-21K
IN-21K
IN-21K
IN-Ik
IN-21k
IN-IK
IN-IK
optimizer
sgd
sgd
sgd
sgd
sgd
sgd
sgd
warmup epochs
5
5
5
5
5
epochs
40
40
40
40
40
40
40
batch size
128
32
64
48
64
64
64
Ir
1e:3
1e:3
le:3
1e:3
1e:3
1e:2
1e:2
wd
le-4
le-4
le-4
le-4
le-4
1c-4
le-4
139
230
780
-o-Pat
-Ow Pal
270
-O-1eRA
-O-1sBA
Adgear
-o- Adgter
300
Adepthrrrer
289
Canoliaae
370
230
300
199
230
140
parmuns
Figure 3: Left: comparison of the inference speed (images/second), Right: comparison of the train- ing speed (images/second), We can conclude that consolidator tuning maintains good throughput during training across various storage budgets, and bring extra cost compared with normal fine- tuning in inference period.
Table 9: Sensitivity test on consolidator's hyperparameters. Given particular target storage budget, we may have several choices in selecting the branches and groups As seen below, such choices make little influence on the final results. The performance of consolidator is relatively stable under a given storage budget.
# params
0.75%
1.01%
4.88%
6.42%
branchesmore
(256,768)
(256,384,768)
(48,64.6)
(24,48)
aCCmore
87.02
87.04
87.18
87.28
branchestees
(192)
(128)
(32,64)
(16)
aCCiess
86.98
87.00
87.14
87.36
Aacc
0.04
0.04
0.04
-0.08
16

---

Published as a conference paper at ICLR 2023
C SENSITIVITY OF GROUPS AND BRANCHES IN CONSOLIDATOR
Given a particular target storage budget, we may have several choices in selecting the branches and groups. As seen in Tab. 9 such choices make little influence on the final results. The performance of consolidator is relatively stable under a given storage budget. When the budget increases, the performance of consolidator increases as well. Such monotonical property helpful for real-world applications, making it easy to tune hyperparameters and rapidly find an optimal candidate under given storage budget.
17


Reviews:
[{"review_id": "xjOydozyBjE", "reviewer": "Reviewer_ULMu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a consolidator that modifies the pretrained model with the addition of a small set of tunable parameters to store the task-specific knowledge while freezing the backbone model during adaptation. A two-stage consolidation process is designed by merging corresponding parameters in the training-storage phase and loading-inference phase. Extensive experiments are conducted on various downstream tasks and the results outperform state-of-the-art methods with fewer stored parameters but superior performance. ", "summary_of_the_review": "This work proposes a two-stage consolidation process for efficient transfer learning. It outperforms state-of-the-art methods with fewer stored parameters but superior performance. I tend to vote marginally above the acceptance threshold. Some additional experiments should be conducted to further improve the quality of this manuscript.", "strength_and_weaknesses": "Strength:\n\n(1) The authors propose a basic module, a consolidator for efficient transfer learning.\n\n(2) A two-stage consolidation process is designed. \n\n(3) Extensive experiments on various downstream tasks show state-of-the-art performance.\n\nWeaknesses:\nThis work aims to design a mergeable adapter for efficient transfer learning and proposes a two-stage consolidation process by merging corresponding parameters. I have some questions as follows. \n\n(1)  In Figure 1, the authors split channels and use zero padding, which might cause a heavy computational cost. Thus, the training time/speed should be shown for comparison with other tuning methods. Besides, I notice that the authors insert a consolidator in the FC layer in Figure 1. Will performance be further enhanced if the consolidator is inserted in other locations? Some recent tuning methods [1, 2] should also be discussed and compared in Table 1. \n\n(2)  Some other tuning baselines are missing in Table 4. The authors should conduct more experiments with other tuning methods for fair comparisons. \n\n(3)  The consolidator is designed for a linear layer. Can it be applied to other operational structures, such as self attention. Further, the current consolidator module is designed mainly in the vision transformer. In Table 1, Table 4 and Table 5, the proposed method obtains promising performance compared to the Full tuning. Push it forward, can it be extended to other model families such as CNNs or MLP architectures [3,4,5]? These will be interesting experiments. Due to the rebuttal time constraints, the authors do not need to add these extension experiments, but feel free to discuss them. \n\n(4)  There exists a large number of typos in the current version, the authors are able to polish it further. \n\n[1] ‘AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition’, in NeurIPS2022. \n\n[2] ‘Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning’, in NeurIPS2022. \n\n[3] ‘A ConvNet for the 2020s’, in CVPR2022.\n\n[4] ‘AS-MLP: An Axial Shifted MLP Architecture for Vision’, in ICLR2022.\n\n[5] ‘RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality’, in CVPR2022. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pbHQRZNx_B", "reviewer": "Reviewer_2eph", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed **consolidator** to efficiently fine-tune transformers. Consolidator has multi-branches structures, which can be merged into a single matrix and saved on the disk. Extensive experiments demonstrate the efficiency and effectiveness of consolidator on various vision tasks.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "#### Strength\n\n1. The design of the consolidator is parameter-efficient since its mergeable property. By extending a single branch to multiple branches, the consolidator has better flexibility and transferability.\n\n2.  The two-stage process maximizes the adaptation capacity without extra inference costs.\n\n3. The paper is well-written and easy to follow.\n\n#### Weaknesses\n\n1. The novelty of this work is limited. The mergeable design is proposed by LoRA [1] for efficient parameter finetuing. Besides, what's the main difference between the **Channel Reorder** and **shuffle** operator in ShuffleNet[2]?\n\n2. Some recent related works are not discussed, e.g. [3][4][5].\n\n3. The improvement is marginal compared with NOAH in Table 1.\n\n\n[1] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv 2021.\n\n[2] Zhang, Xiangyu, et al. \"Shufflenet: An extremely efficient convolutional neural network for mobile devices.\" CVPR 2018.\n\n[3] Bahng, Hyojin, et al. \"Exploring visual prompts for adapting large-scale models.\" arXiv 2022.\n\n[4] Chen, Shoufa, et al. \"AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition.\" NeurIPS 2022.\n\n[5] Jie, Shibo, and Zhi-Hong Deng. \"Convolutional bypasses are better vision transformer adapters.\" arXiv 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "INbJAssxSUP", "reviewer": "Reviewer_HhUN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the adaptation of the well-trained transformer to downstream tasks on resource-limited devices. The authors introduced grouped connections with re-parameterization technology into the training process. The experimental results on serval vision tasks demonstrate the effectiveness of the proposed method.\n", "summary_of_the_review": "As written in the Strength And Weaknesses, the strictly restricted application scenario lowers the practical value of the proposed method.", "strength_and_weaknesses": "+ Compared to the baselines and other methods, the proposed methods could bring some improvements in accuracy and computation cost.\n+ Some ablation experiments are introduced to prove the effectiveness of the proposed method.\n\nThe main concerns are listed below. \n- Strictly restricted application scenario. The author introduces the application scenario in the abstract where the resource-limited devices cannot store a full copy of parameters but could provide computation to train/fine-tune the large models. It is a bit strange the devices with the ability to train DL models do not have sufficient storage.\n- The proposed method is irrelevant to Transformer. It is closer to the re-parameterization technology. \n- Some recent literature on the adaptation of Transformers is missing.\n\n[a] AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition, NeurIPS 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-2ry0EpHKN", "reviewer": "Reviewer_zLNL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new fine-tuning method for vision transformers that only trains and stores very few parameters. The fine-tuning method achieves both efficiency and effectiveness through its design of mergeable multiple branches and parameter shifting. Extensive experiments show the method surpasses other adapter methods and full fine-tuning on several transfer learning tasks. ", "summary_of_the_review": "See Strength And Weaknesses", "strength_and_weaknesses": "Strength:\n1. Consolidator is a very effective method. Compared with other adapter methods and full parameter fine-tuning methods, Consolidator can achieve an almost optimal transfer learning effect.\n2. Compared to the adapter methods, the design in this paper can be merged with the original model so that the inference phase does not increase the cost of memory and computation time.\n\nWeakness:\n1. The prior of ChannelReorder operation is strong but the explanation of its effectiveness is weak. \n2. The parameters in all tables are stored parameters for Consolidator but not trainable parameters.\n3. It should be possible to save time compared to the full parameter finetune during training, I wonder why this is not reported, nor is it compared to other baseline training time.\n4. The value of saving storage space is not so great, especially for the base-level model, and there are few experiments on the Large model.\n5. There are slightly more spelling errors and grammatical errors. P3 workss; P4 ti, thw; P5 the all the; P7 accomodate.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster