Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone
Authors: Zeyinzi Jiang
Chaojie Mao
Ziyuan Huang
Ao Ma
Yiliang Lv
Yujun Shen
Deli Zhao
Jingren Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Parameter-efficient Transfer Learning
Memory-efficient Transfer Learning
Residual Network
Vision Transformer
Foundation Model
TL;DR: A parameter and memory efficient tuning paradigm is called Res-Tuning.

Abstract: Parameter-efficient tuning has become a trend in transferring large-scale foundation models to downstream applications. Existing methods typically embed some light-weight tuners into the backbone, where both the design and the learning of the tuners are highly dependent on the base model. This work offers a new tuning paradigm, dubbed Res-Tuning, which intentionally unbinds tuners from the backbone. With both theoretical and empirical evidence, we show that popular tuning approaches have their equivalent counterparts under our unbinding formulation, and hence can be integrated into our framework effortlessly. Thanks to the structural disentanglement, we manage to free the design of tuners from the network architecture, facilitating flexible combination of various tuning strategies. We further propose a memory-efficient variant of Res-Tuning, where the bypass i.e., formed by a sequence of tuners) is effectively detached from the main branch, such that the gradients are back-propagated only to the tuners but not to the backbone. Such a detachment also allows one-time backbone forward for multi-task inference. Extensive experiments on both discriminative and generative tasks demonstrate the superiority of our method over existing alternatives from the perspectives of efficacy and efficiency. Project page: https://res-tuning.github.io/.

Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone
Zeyinzi Jiang' Yiliang Lv
Chaojie Mao
Ziyuan Huang² Ao Ma¹
Yujun Shen
Deli Zhao'
Jingren Zhou'
Alibaba Group
National University of Singapore Ant Group
{zeyinzi jzyz, chaojie chaojie.mcj, mcj, {ziyuan. huang) @u nus, edu (shenyujun0302, zhaodeli/@gmaill com
dave.ma ma, yiliang. .ly1, jingren zhou) Saaeeaalibbbbbaa saalllibbb-inne com
Abstract
Parameter-efficient tuning has become trend in transferring large-scale foundation models to downstream applications. Existing methods typically embed some light- weight tuners into the backbone, where both the design and the learning of the tuners are highly dependent on the base model. This work offers a new tuning paradigm, dubbed Res- Tuning which intentionally unbinds tuners from the backbone. With both theoretical and empirical evidence, we show that popular tuning approaches have their equivalent counterparts under unbinding formulation, and hence can be integrated into our framework effortlessly. Thanks to the structural disentanglement, we manage free the design of tuners from the network architecture, facilitating flexible combination of various tuning strategies. We further propose a memory-efficient variant of Res-Tuning, where the bypass (i.e., formed by sequence of tuners) is effectively detached from the main branch, such that the gradients are back propaaate only the tuners but not to the backbone. Such detachment also allows one-time time backbone forward for multi-task inference. Extensive experiments on both discriminative and generative tasks demonstrate the superiority of our method over existing alternatives from the perspectives of efficacy and efficiency. Project page: https: //res//resseeee tuning. ouning-github. it/h
Introduction
Recently, foundation models have demonstrated strong generalization capability across numerous visual [21, 2]. language [47, 60] and multi-modal tasks [40, 1]. Pre-trained on large corpus of data, foundation model offers good initialization for downstream adaptation. Unfortunately, the increasing model scale makes it expensive and almost infeasible to fully fine-tune such a model for every task. Hence, parameter-efficiem transfer learning (PETL) [37, 41, 27] often resorted to as an efficient approach for downstream adaptation without incurring unaffordable computation burden.
Popular existing approaches for parameter-efficien tuning introduce additional tunable structures (which we term as tuners) to the pre-trained base model [37, 41, 27]. Compared to the fully-fine- tuned counterparts, the light-weight tuners significantly reduce the training cost while maintaining competitive performance [32, 28]. However, the current designs of tuners are deeply coupled with their base structures. as shown in Fig. 1a, thus restricting the design flexibility and impeding the extension to new approaches. For example, prefix tuning [41] is embedded into the self-attention operation, and prompts [28, 85] could only be introduced at the beginning or between layers, etc.
In this work, we introduce Res-Tuning, new tuning paradigm for flexible and efficient transfer learning. As in Fig. 1b, our Res Res-Tuning framework unbinds the tuners from the base model, such that it is possible to decouple both the design and the learning of the tuners from the base structure. Since
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Res- Tuner
FFN
FFN
Tuner
Res-
Block
Tuncr
Res-
Tuner
Res-
MHA
MHA
Res-
Block
Res-
Tuner
Tuner
(a)
(b)
(c)
Figure 1: Concept comparison between existing methods and Res-Tuning framework. (a)
Existing PETL methods are deeply embedded into original structures.
(b) Our Res-Tuning
framework can decouple PETL methods from the backbone. (c) Backpropagation only through a bypass consisting
can be achieved by detaching the connections.
the design of the tuners is no longer dependent on the base structure in our Res-Tuning framework, we explore various possibilities. With both theoretical and empirical evidence, we first show that our framework can seamlessly encompass popular tuning approaches such as prefix-tuning [41], prompt-tuning [28], and adapters [25]. Further, it is demonstrated that the structural disentanglement also allows for flexible combination of various tuners, leading to the discovery of stronger tuning strategies. On top of that, we show that such an unbinding formulation also allows for the detachment of the tuners from the backbone in Fig. 1c, which further improves the memory efficiency. In this memory-efficient version, i.e., Res- Te-TTunnng-yyaaae the gradient computation through the massive parameters is avoided in the base model, but it also reduces the number of forward passes of the backbone model to only once during multi-task inference.
not only is the training cost reduced because
We evaluate Res-Tuni Tuming framework on both discriminative and generative tasks. On discriminative tasks, we show that our unbinding formulation leads tuning strategy that achieves ofte-offthee-rt performance on VTAB-1K with similar learnable parameters. Our Res- Tew-unnnnnnnaasssseeee also performs favorably against the fully-fine-tuned variant, while reducing the training memory consumption by 49.7% and the multi-task inference time by 80.9%. In addition, it also obtains better performance in few-shot learning and domain generalization scenarios. On generative tasks, apart from the strong performance achieved by Res Res-Tuning framework in terms of both FID scores and qualitative results, we further show that our Res- Tons-Tnnng- -eriinggyppass can reduce the memory consumption by 70.7%, training time by 58.6%, and multi-tass task inference time by 83.6% when maintaining a competitive FID score and generation quality compared to the fully-fine-tune variant.
Bypass framework
2 Unbinding parameter-efffcient tuning
In order reduce the entanglement between the base model and the tuners as well increase the flexibility of the tuning strategies, we set out to unbind the tuners from the pre-trained structures. In this section, provide strategies. We start by revisiting the basic building blocks of the foundation models, before diving into unbinding existing tuners from the backbone. In the last part of this section, further provide empirical proof that our unbinding formulation could seamlessly encompass existing PETL methods like prompt tuning [28], prefix tuning [41], and adapters [25].
unbinding formulation existing parfmeter-efficient transfer learning
2.1 Basic building blocks of foundation models
Existing foundation models in both natural language processing, vision, and vision-language applications mostly adopt Transformers [70] as the backbone. The major building blocks of the Transformers that one usually adapts for the downstream tasks are the multi-head attention (MHA) and the feed-forward network (FFN). Formally, the standard MHA and FFN could be expressed as:
QKT T
AMn(Q, K, V) softmax
Vd
(I)
FFN = (i Wi + bb)) ba

---

Original
Parallel
Original
Parallel
Original
Parallel
Adapter
Kars
Q K
pre
Q Y MHN
Leni MHA
MHA
Vere MHA
MHA
MITAL
FFN
FFN
Adapter
Input
Input
Input
Pro.
Input
Pro.
Input
Input
(a) Profix Toming
(b) Prompi Tuning
(6) Adopter Tuning.
Figure 2: The original and the unbinding form of (a) prefix tuning [41], (b) prompt tuning [28],
and (c) adapter tuning [25] for parameter-cfficient transfer learning.
where Q. K and V denote the query, key and value, respectively. W1 and W2 are the projection weights, b, and by are the bias terms, and Φ is the non-linear activation between fully-connected layers. Usually, given input tokens the query, key, and value are obtained through linear projection Q wW K = x WR. and V w,, where W,, WE and W,, are learnable projection weights.
2.2 Unbinding tuners from foundation models
For the adaptation of the foundation models to downstream applications, the existing PETL approaches mostly resort to adjusting the output of MHA, FFN, or the Transformer block composed of MHA and FFN in various ways. We choose popular and exemplary approaches and unbind their structures from foundation models. Here, provide the unbinding formulations of prefix tuning [41] ] and prompt tuning [28] for MHA adaptation, as well as adapter tuning [25] for FFN adaptation.
Prefix tuning [41] prepends learnable parameters, i.e., prefix tokens, to the projected keys and values:
MHAppe Attn(x [K Kprei We), (V prrei x W.l),
where Kr and Vpre are prefix tokens. Essentially, we view this as performing MHA separately between (Q, K, V) and between (Q, K Krre; Vpre), we unbind prefix tuning as follows:
MHAppe = (1 -1-22 2) Attn (Q, K.V) + Attn (Q, Kpre, V
(3)
original attemtion
prefix purallel
where À weighs between the original and prefix attention. Detailed value for À and the derivation process included in appendix A. In this way. the original MHA in the foundation model Attn(Q, K, V) and the prefix attention Attn(Q, Kpre, Vpre) can be computed independently. The unbinding formulation of prefix tuning can be seen in Fig. 2a.
Prompt tuning [28] appends latent tokens the input token before performing MHA the backbone:
MH pro Attn ([æ; pro Wa. [m; Trro We. [x:; mrro W.):
(4)
where Epro are prompt tokens concatenated to the input token æ in the first layer or between multiple layers. Similar prefix tuning, the unbinding formulation of prompt tuning is as follows:
MHApo [[1 A) Attn (Q,KV K.K. V) +A Attn (Q, K Vpro): ); D),
(5)
original attention
prompt atliction in pomillel
where K Epro WA and Vpro Epro W.. D denotes disposable parts that would not affect the output of MHA, where D (1- 3) Attn pro K proo Vpro) + 3 Attn (Q pros K, V). X and 3 are individual weights. More details can be seen in appendix A. The unbinding formulation of prompt tuning can be seen in Fig. 2b.
Adapter tuning [25] typically inserts a multi-layer perceptron (MLP) after FFN. Since the MLP could be performed independently, we simply re-route the adapter and connect it in parallel to the FFN as in Fig. 2c. The resultant unbinding formulation of the adapters is as follows:
FFN adapter
FFN(x)
+ (FFN((z W Woown) W up-
(6)
original modiul
adapter module paralle]
where W down and W., up denote the weights for the down- and up-projection layers, respectively.
3

---

Residual
Residual Profix
Endlaye
Sad-
Soali
Vver
Daea Turres
ham
Vertical
Prompt
Bas Tunez
GP josk Jm
Frozen
Horizontal
OF
Tunee
Tuser
USA
Block
(a)
(b)
(c)
Figure 3: Structure illustration of (a) various bes-Tuners in unbinding form, (b) Rea-Tuning framework that is flexible and efficient, and (c) Hes-Tuning-Bypanl the memory- efficient version of our framework.
Table I: Empirical equivalence of PETL methods and their counterparts in our unbinding form.
VIT/B-16 16 (IN-21K)
ViT/L- 14 (CLIP)
Method
Original
Unbinding form
A
Original
Unbinding form
Adapter [25]
92.34
92.34
0.00
92.43
92.44
+0.01
Prefix [41]
91.90
91.88
-0.02
90.99
91.01
+0.02
Prompt (28)
92.21
92.24
+0.03
91.86
91.83
-0.03
2.3
Empirical equivalency of the unbinding formulation
After we have obtained the unbinding formulation of popular PETL methods with theoretical derivation, Tab. shows empirical evidence of the equivalency between the original formulation and unbinding formulation. We carry out the comparison between two formulations using ViT [13] pre-trained on ImageNet-21K and by CLIP [59]. For all cases, we observe that the performance discrepancy between the original and unbinding form is within reasonable range (+: 0.03), which shows that our formulation encompasses existing PETL methods effortlessly.
CIFAR-100
3 Res-Tuning
3.1 Unified formulation of Res-Tuning
Given the unbinding formulation of the existing PETL methods, derive unified formulation as the combination of the frozen pre-trained operation and the tuner with learnable parameters:
OP((s) xx) + Res-Tuner(z):
where OP denotes existing operations in the pre-trained backbone such as MHA and FFN, while the Res-Tuner represents the learnable structures that are connected in parallel to the existing operations.
With this unified unbinding formulation, the design of the tuner structure can now be independent of the original operation in the base model. This leads to unprecedented flexibility in parameter-efficien tuning, which enables us explore various instantiations of the tuner (as Fig. 3a) for the base structures. For example, we found in Tab. 2a that, instantiating the Res-Tuner with prompt tuning for FFN results in a better performance compared to adapters.
Further, the structural disentanglement also allows for the flexible combination of various tuning strategies. In Fig. 3b, we instantiate our Res- Tes-Tuning framework by associating one res-Tuner with every operation in the base model, including MHA, FFN, and the whole Transformer block.
3.2 Res-Tuning-Bypass Towards memory-efficient PETI
The unified formulation of Res-Tunnn Tusiinnn provides a viable solution for flexible and parameter-efficient transfer learning. However, since it is directly derived from existing PETL methods, it shares the

---

same vulnerability as existing PETL solutions. Despite the parameter efficiency, they require back- propagation through the massive parameters the pre-trained backbone, which leads to unnecessary extra consumption of memory and computation.
To avoid this, further present a memory-efficient version of our Res-Tuning framework, dubbed as Res TTuning-Bppaas as in Fig. 3c. Specifically, we remove the data flow from the Res- the backbone, such that the Res Res-Tuner is detached from the pre-trained architectures. In this way, we form a bypass network constructed by Bes-Tunery in parallel with the backbone. Formally, given the tokenized feature Ty and the output feature of the [-th layer æy from the pre- pre-trained model, our bypass network is formulated as:
bypass 20; 0 bypass ARes- (Res-Tuner(z)) + (1 X)Res - (ypass) ),
(8)
where À is a Iearnable parameter followed by sigmoid function, which is initialized to 0.5. As demonstrated in Fig. 3c, we group the Res-Tuners in the bypass network into horizontal ones and vertical ones, respectively processing the output feature of the I-th layer from the backbone and the (1 - 1) feature from the bypass network. Within each group, we keep the structure identical. Thanks to the flexibility of our unbinding formulation, we can also explore different instantiations of the Res-Tuner and various combinations of the existing tuners.
4 Empirical evaluations on discriminative tasks
4.1 Experimental setups
Evaluation scenarios. We mainly analyze the flexibility and efficiency of our proposed Res-Tuning framework and evaluate the discriminative capabilities on three different scenarios: transfer learning, few-shot learning, and domain generalization.
Baselines. Apart from the traditional downstream adaptation methods like fully fine-tuning and
linear probing, we divide existing tuning approaches into two categories: (i) methods focusing on parameter--fficieenee including adapter tuning [25]. prefix tuning [41]. VPT [28], LoRA [27], AdaptFormer [7], SSF [42]. and NOAH[83]: (ii) methods focusing on memory-efficieney. including [82], and LST [68]. Since these categories have distinct characteristics with respect to the parameters, memory, and performance, we mainly compare our Res-Tun framework within the former category, while the
TTuning-yppass is mainly compared in the latter one.
Implementation details. For most experiments, we adopt ViT-B/16 [13] pre-trained on ImageNet- 21K [11] as the backbone model, following VPT [28]. Unless otherwise specified, the middle of the adapter, as well as the number of prefix and prompt tokens in our bes-Tuning are set to 10 for parameter efficiency. We include the training details in appendix C. For all the tasks, we use top-1 1 accuracy as the main evaluation metric.
4.2 Analysis on flexibility
Flexible combination between backbone structures and tuners. Since our unbinding formulation allows for the structural disentanglement between the frozen structure in the backbone and the tuner with learnable parameters, it enables us to explore various combinations of the operation and the tuners. Here, we experiment with various instantiations of OP and Res-Tuner in our Res-Tuning framework, and the number of tuners is limited to one tuner per block Maangle-Res-- Tuner). The results are presented in Tab. 2a. We found that the default combination in cxisting approaches (prefix and prompt for MHA adaptation and adapter for FFN adaptation) is far from optimal, and connecting prompt tuning to FFN results in the best performance.
Flexible combination of multiple tuning strategies. Next, we show that the flexibility brought by our unbinding formulation could also effortlessly lead to stronger tuning strategies by combining various tuners in our unbinding formulation. Here, we consider two tuners per block respectively connected to MHA and FFN. As in Tab. 2b, employing two tuners for each block brings notable improvements over the Single -uuner variants, with employing two adapters respectively for MHA and FFN achieving the strongest performance of 93.25. On top of the best performing Dual Res- Res--Tner Tuner model, we further attach tuners at the block level in Tab. 2c. With
5

---

Table 2: Exploration of various combinations of operations in the pre- traieed backbone and various achieves a stronger performance compared to the existing tuning strategies on CIFAR- 100. Adapter, prefix, and prompt are abbeviated as Ada., Pre. and Pro., respectively.
(a) Single-Rea-Tume
(b) Dual -Res Rea-Tuner.
(c) Tri Fri-Rest Runer.
Tuner' OP MHA FFN Block
MHA WHHAFFFNJRR RReN Ada.Ada. Res-Pre. Pre. Res-Pro.
Block
Dual Qual-Res- Rese-runer
Res-Ada.
92.46 92.34 92.49
Res-Ada.
93.25
92.95
92.62
Res-Ada.
93.28
Res-Pre. 91.88 92.33 92.39
Res-Pre.
93.22
92.38
92.87
Res-Pre.
93.20
Res-Pro.
92.24 92.68 92.16
Res-Pro.
93.03
92.92
92.91
Res-Pro. 93.16
Table 3: In-depth analysis of our Res-Tuning and Tee--TTunnnnapaaeen in terms of performance,
parameter--fficiency and memory efficiency on CIFAR-100.
(a) Parameter efficeency of Res- Tuning on CIFAR-100. (c) existing tuning strategies CIFAR-1O 100.
Performance and efficiency comparison with
Method
Full
Linear Probing
Single-
Dual-
Tri-
Acc.
89.12
85.95
92.68
93.25
93.28
Method
Acc.
Param. (M)
Mem.
Parum.
85,9M
0.07M
0.17M
0.48M
0.67M
Full
89.12
85.9 (100%)
9.02G
(b) Parameter and memory efficiency of Rea Rea-Tuning-
Linear
85.95
0.07 (0.08%)
2.72G
Bypass on CIFAR-100.
Purameter-effeientt tuning methods
MAM-Adapter [19]
91.70
10.08 (11.72%)
9.57G
Method
Bypass Tuner Acc.
Param, Mem.
AdaptFormer [7]
91.86
1.26 (1.46%)
6.32G
Res Bes-Tuning
93.25
0.48 (0.55%)
6.85G
Lincar probing
x
85.95 0,07M 2.72G
None
86.34 0.07M 3.48G
Memory-efficintt tuning methods
Hori.
88.08 0.27M 3.66G
Side-Tuning [82]
87.16
9.62 (11.18%)
3.48G
Vert.
87,26 0.27M 3.64G
LST' [68]
88.77
0.93 (1 (8%)
5.26G
Rea Rea-Tuning- -Byaas
Both
89.33 0.46M 4.72G
Das-Tuning-Bypare
89.33
0.46 (0.53%)
4.72G
Fully fine-tuning
x
89.12 85.9M 9.02G
T denotes our own implementation since the original
approach presented natural langauge processing.
adapters on top of the classification performance. Compared to existing tuning strategies of underlining in Tab. 2a, without bells and whistles, the Tri-Res-Tune -unner version of our Res- TuniTun framework achieves at least 0.94% performance improvement.
model, we observe further slight improvements on the
4.3 Analysis on parameter-, memory- and multi-task inference-efffciency
Parameete-efficiencyy We analyze the parameter efficiency of our Res-Tunteee Tuning framework in Tab. 3a. Our Single-Rest Res-eeeeReTTueee version surpasses the performance of the fully-fine-tune variant with less than 0.2% learnable parameters. Combining all three tuners on MHA, FFN, and block-level, we manage to outperform the fully-fine-tuned and linear evaluated variants by respectively 4.16% and 7.33%, while only using 0.67M parameters.
-eliry-efliciency. Here, we present the evolution from linear probing to our Res- -eninT Tuning-Bypass in Tab. 3b. It is observed that introducing a bypass network without any tuners can help ensemble the original features obtained from different layers, thereby mildly improving the classification accuracy as compared to the linear probing approach where only the classifiers are trained. On top of that, both horizontal and vertical des-Tunes bring notable performance improvement with limited parameter and memory overhead. With both horizontal and vertical our Res-Tuming BBpass framework achieves stronger performance with only 52% of the memory consumption when compared with the fully-fine-tuned variant.
in place,
Multi-task inference-efficiency. In Fig. 4, our Beas-Tuning-Bpaass demonstrates superior multi- task inference-efffciencyy on both discriminative and generative tasks. For discriminative multi-task inference, we combine the validation set of 19 tasks in VTAB-IK, perform 19 tasks on every image, and obtain the overall process time for the whole validation set. For generation multi-task inference, we take one image and provide the model with 10 fixed-length prompts for generation and record the overall generation time for the 10 prompts, For existing parameter-efin erffiittr-effccennt methods, the inference time grows linearly when the number of tasks grows. Compared to the fully fine-tuned variant, all existing parameter-efficient tuning strategies increase the inference time to various extents. In contrast, our -wwningnnnnuppee discriminative tasks and 10 generative tasks by respectively 80.9% and 83, 83.6% when compared with the silly--fin-tunned variant.
framework significantly reduces the inference time on 19

---

2030
Ree-tynin
Ree-Toiing-ao Shpppr
3..0
NCM
10 1115
LSST
Sie lanio
Number
Nam.
Figure 4:
Comparisons of the parameter, memory,, and multi-task inference-efficiencyy evaluate Res-Tuning- Bypass on both discriminative and
For multi-task inference-efficiency generative tasks. For parameter- and memory-effiienny, here, we show the comparisons on VTAB-IK between our approach and existing tuning strategies.
Tluning-Bypass
Table 4: Performance and efficiency comparison on the VTAB-IK benchmark with ViT-B/16 models pre-trained on ImageNet-2IK. "Group Mean" denotes the average accuracy of the three subgroups. "All Mean" denotes the average accuracy of 19 downstream tasks.
Natural
Specialized
Structured
SNORB- Azim
CIFAR- 100
Caheeshol
Flowers 102
SVIN
Sum399
Canelyon
EuroSAT
Resisc4S
Retinoputhy
Cleve- Count
Clev--iit
DMLab
KITTI-Dist
dSprLee
dSpr
-NORB-Eles
Group Mean
All Mean
Param (M)
Mem, (GB)
DDD
Traditional methods
Full
68.9 87.7 64.3 97,2 86.9 87.4 38.8
79.7 95.7 84.2 73.9) 56.3 58.6 41.7 65.5 57.5
46.726.777
25.7
29.1
66.96 65.57
85,84 9.40
85.0
97.0
86.3
51.0 78.5 87.5 68.6 34.3 30.6
55.4
20.0
19.2
57.64
52.94
3.09
Parameter-effiient tuming methods
Adapter [25]
85.7
62.7
97.8
87.2
36.4 50.7 76.9 89.2 73.5 71.6 45.2 41.8 31.1 56.4 30.4
24.6
13.2
22.0
60.52
56
1.82 6.53
LoRA [27]
67.1 91.4
69.4
98.8
90.4
85.3
54.0
84.9 95.3 84.4 73.6 82,9 69.2
49.8 78.5
75.7
47.1
31.0
44.0
74.60
72.30
0.29
6.88
VPT-Deep [28]
90.8
98.0
49.6
96.1
83.4 68.4
72.8
47.9
37
69.43
8.13
SSF [42]
69.0
75.1 99.4
91.8
90.2
52.9
87.4
95.9 87.4 75.5
75.9
62.3
53.3 80.6
77.3 54.9
29.5 37.9
75.69
73.10
0.24
7.47
NOAH [83]
69.6
70.2
99.1 90.4
86.1
5
95.4 83.9
82.8
68.9 49.9
81.7
81
48.3
32.8
75.48
73.25
0.42
7.27
Res-Toning
92.7
71.9
99.3
91.9
86.7
58.5 86.7 95.6 85.0 74.6 80,2 63.6 50.6
80,2
85,4
55.7
31.9
42.0
76.32
74.10
0.55
8.95
M Memory- efficicent tuning methods
Side Tuning [82]
60.7 60.8 53.6 95.5 66.7 34.9 35.3 58.5 87.7 65. 61.0) 27.6 22.6 31.3 51.7 8.2 14.4 9.8 21.8
21.8
45.65
9.59 3.48
LST' [68]
58.0 87.1 66.2 99.1 89.7 63.2 52.6 81.9 92.2 5, 69.4 68.6 56. 38.8 73.4 72.9 30.5 16.6 31.0
67.067.56
64.52
0.89
5.13
Res- TesinTnnng--ua Bypass
88.8
70.4
52.6
72.32
0.42
4.73
T denotes our own implementation since the original appronch is proposed for natural language processing.
4.4
Comparisons with existing tuning strategies on different scenarios
Transfer Learning. We mainly evaluate our approach on the basic transfer learning scenario, where pre-trained models fine-tuned different downstream tasks. CIFAR-I00 [35] is standard general-purpose image classification dataset. VTAB-1K [80] is composed of 19 various visual classification tasks falling into three categorizations, i.e., natural, specialized, and structured. We compare the performance of our approach and other baseline methods on the following:
CIFAR-100. We present the comparisons to existing tuning strategies in Tab. 3c. For parameter- efficient methods, our Res- Tuningg framework notably improves over AdaptFormer [7] by 1.39%, using only 0.55% extra parameters, which is 0.91% % less than AdaptFormer [7]. For memory-efficien methods, we outperform LST [68] by 0.61% % with memory reduced by 0.54G (10%).
VTAB-IK. In Tab. 4, we present comprehensive evaluation the 19 datasets on the VTAB-1K benchmark. Both our Res-Tuning and Res-Tuning- -eyiingg outperform existing approaches respectively within the group of parameter- and memory-efficient tuning methods. Specifically, our Res- Res-Tuning framework achieves 0.85% improvement in terms of the average performance on 19 datasets, compared to the previous best performance. For Res- Tuning-Bypaan outperform LST [68] in 18 out of 19 tasks and overall by 4.99% in terms of average performance. This is achieved with even fewer parameters and memory consumption. We further visualize the parameter vs. performance curve and memory vs. performance curve in Fig. 4 to show the advantage of our Res Res-Tuning and Res-Tuning-Bypen
framework on VTAB-1K.
Fhw-Shot Learning. To evaluate the ability our approach to adapt with only few training samples, we follow the few-shot evaluation protocol in NOAH [83], using (1, 2, 4, 8, 16)- for training and full test data. We conduct experiments on five fine-grained datasets, including FGVC-Aireraft [51], Food-101 [4], Oxford Flowers [55], Oxford Pets [56], Stanford Cars [15].
7

---

Avsnage 5 detiauts
FBVC Alcrnit
Food 10h
1g- Apt (ND
8
Nambor of abiled Haining exarpple palepppo 006
Number Libeed soinig soancles oles
Number siseeed toain ng oxtimples per dloss
Oxford Flowers
Ordord
Starford Cars
80
Top- (84
100 PC (M
50
Figure 5: Results of fhw-shot learning on five fine-grained visual recognition datasets. The solid line represents the comparison of parameter-efficien tuning methods, and the dashed line represents the comparison memory-efficient tuning methods. All results are averaged over random seeds.
Table 5: Results on domain generalization. "Mean" denotes the average accuracy of four variants
of ImageNet. results are averaged over random seeds.
Source ImageNet
Target IN-A
IN-V2
IN-Sketch
IN-R
Mean
Parameter tuning methods
Adapter [25]
70.5
59.1
16.4
5.5
22.1
25.8
VPT [28]
70.5
58.0
18.3
4.6
23.2
26.0
LoRA [27]
70.8
59.3
20.0
6.9
23.3
27.4
NOAH [83]
71.5
66.1
24.8
11.9
28.5
32.8
Res- Des-Tuning
78.04
66.58
29.23
13.15
29.01
34.50
(eeoory-efficiete tuning methods
Side-Tuning [82]
74.57
62.52
23.55
10.37
25.06
30.38
LST [68]
70.00
57.04
14.39
7.21
17.02
23.92
Res- Moss-Tnnng-BB Tuning-Bypas
77.30
65.23
27.39
10.66
26.45
32.43
As the results shown in Fig. 5, in terms of the overall performance (top-left), both Res-Tuning and Res-Tuning -Byaass demonstrate clear advantages over other corresponding parameter-efficient and memory-efficient tuning strategies of few-shot learning five FGVC datasets. We also observe that Res Res-Tuning- --Tuuinnn-BByasss performs as well as or even better than the nonfmemory-efficien methods on one or two shots with low training samples on serveral datasets.
evaluate robustness of our approach to distribution shift, we train model the source domain using 16 shots per category and test it both the source and target domain. The source domain uses ImageNet-IK [1 IJ) and the target domains use four other variants of ImageNet, including ImageNet- 2 [62], ImageNet-Sketch [73], ImageNet-A [24], ImageNet-R [23]. The results in Tab. 5 prove that our approach is robust under domain shift. Our Res Res-Tuning goes beyond NOAH [83] by 6.54% ImageNet and 1.7% the average accuracy of four variants of ImageNet. Furthermore, Res-Tuning -en--unnnnnaaasssss--------- also demonstrates stronger robustness than the memory-efficient baselines and outperforms most existing parameter-efficient tuning methods.
Domain Generalization.
Empirical evaluations on generative task
5.1 Experimental setups
Downstream tasks. To provide more comprehensive evaluation of our Res-Tuning framework, further apply it to the text-to-image generation task. Following [63], we evaluate the text-to-image

---

Table Comparison of FID and efficiency on COC02017. Following the default settings of stable diffusion we sample 10k captions from the validation set for generating images of size 512² using 50 PLMS steps with classifier-free guidance scale 3.0 and compare against the full validation set.
Method
FID
Param. (M)
Mem. (GB)
Train (Hour/Epoch)
SD v1.5
15.48
Full
14.85
862 (100%)
72.77
1.98
LoRA
14.50
9.96 (1.15%)
61.03
1.42
Adapter
14.73
2.51 (0.29%)
54.30
1.30
Prefix
15.36
4.99 (0.58%)
64.91
2.20
Prompt
14.90
1.25 (0. 14%)
63.70
2.17
Res-Tuning
13.96
2.54 (0.29%)
54.49
1.38
Res-Tuning Bypass
14.89
3.76 (0.44%)
21.35
0.82
Text
SD v1.5
Full
LoRA
Adapter
Prefix
Procupt
Res- TesiiT
Bypass
peopped up in the sand on beech.
black bird on of herry bounches.
Figure 6: Qualitative results of SD v1.5, existing tuning strategies and Res-Tuning COCO2017 validation set [43]. We frame our results in green and others in red.
generation performance COC02017 dataset [43]. The main quantitative evaluation metric is the FID score and we also perform instance-level fine-tuning transfer on the fine-grained datasets [4, 55].
Baselines. We experiment with the version 1.5 of stable diffusion [63] (SD) model. On COC02017,
we compare our bes-Tuning framework with zero-shot SD, SD with fully fine-tuning as well as SD with other existing tuning methods. On the fine-grained datasets, we employ DreamBooth [65] as our baseline and employ various tuning methods including our own for comparison.
5.2 Main results
Text-fo-image generation on COCO. We show the comparison between our Res- des-Tunnng framework and other approaches both quantitatively and qualitatively. The quantitative comparison presented in Tab. 6. Compared with the fully fine-tuned baseline, our Res- moos-Tuning framework improves the FID score by 0.89, using only 0.29% of the parameters. framework is the only approach that reaches FID score below 14, while the best existing tuning strategy on the task is LoRA [27] achieving 14.50 with 4x the number of parameters in Res Res-Tuning, Hence, employing Res Res-Tuning could greatly facilitate the adaptation of pre- pre-trained text-to-image generation models to downstream tasks.
is noteworthy that our Res-Tuning
A highlight is observed that our Res Bes-Tuning- -ynaan-BBaa-B framework reduces the memory consumption for tuning the SD v1.5 model to only 29% while maintaining similar performance (14.89 vs 14.85). Meanwhile, the time consumption for training the model is reduced to 41% In terms of the reduction framework outperforms the best
in the memory and time consumption, our Res-Tue Tuning-Bypea tuning strategy by 3.3x (70.7% memory reduction of Res Res-Tuning- Beaiiinn--Bpasssse vs. 25.3% that of adapter) and 1.7x (58.65 6% reduction time consumption Res- Seee-Tunnng-Byae Bypass vs. 34.3% that adapter), respectively.
The qualitative results are presented in Fig. 6. Both Res -ennin and Res Mee-TTning-Byy Bypass show a better alignment between the text and the generated image, where the surfboard is propped in our generated images. Res-Tunt messTunigg also demonstrates better fidelity where the feather texture realistic in the generated black bird.

---

Balloon Flower
SD
DreamRooth
LoRA
+ Adapter
Prefix
Prompe
Res-Tuning
+ Rypass
potll
Hotdog
Figure Qualitative results of SD v1.5, DreamBooth, existing tuning strategies, and our Res Res-Tuning on Oxford Flowers and Food-101 fine- fine-grained dataset with the same generated seed. We frame our results in green and others red.
Text-to-image generation Oxford Flowers and Food-101. Here, evaluate the transfer ability of existing tuning strategies on specific fine-grained categories. During every evaluation process, we select data from specific category to train the model. The qualitative results are demonstrated in Fig. 7. It is observed that Res-Tuning presents a better view in terms of fidelity and the correct understanding of ambiguous categories. For example, the balloon flower is bell-shaped and has five petals, while existing approaches generate flowers with the wrong number of petals or even balloons rather than real flowers. Instead, both our Res-Tuning and our Bes-Tuning- Bypass retain correct semantics and ane-grrined features.
luming-Bypass
Related work
Transformers in computer vision. Transformers [70] have demonstrated strong capabilities in various fields [5, 61, 12, 59, 2. 17, 48, 53]. In computer vision, Vision Transformers (ViT) [13] are widely applied in various applications, such as visual classification [48, 391, object detection [67, 6], segmentation [84, 74] and generation [63, 57]. Owing to the strong scalability of the Transformer backbone [31, 10], recent endeavors either focus on expanding the size of the ViT [81] or training on larger corpus of data in an unsupervised manner [75, 14].
Parameter-efficient tuning. Despite the strong performance and generalization ability brought by scaling up the Transformers, it also makes the adaptation to the downstream tasks expensive and almost infeasible. Hence, parfmeteer-fffcieett transfer learning (PETL) emerged [26, 25, 76, 41, 85, 28]. Existing PETL methods could be generally categorized into three types: (i) MHA-based
tuning embeds tunable parameters in the multi-head self-attention layers [27, 76, 28, 41, 45, 46). (ii)
FFN-based tuning methods represented by adapters [25] and its generalized versions [58, 33, 32, 19]
introduce a multi-layer perceptron to the FFN layer. (iii) Other tuning methods adapt certain existing parameters [79, 42]. Closely related to our work, some recent efforts are devoted to finding out the optimal design paradigm of the tuning modules by neural architecture search [83]. Instead, we show that through our Res Tes-Tuning framework, a stronger tuning strategy can be easily found by the flexible combination of several existing tuning strategies in our unbinding form.
Memory-efficien tuning. Since the structures of the existing PETL methods are deeply embedded in the backbone structure, back-propagation required through the massive parameters of the pre- trained models, leading to unnecessary extra consumption of memory and computation. Hence, Side- Side-Tuning [82] and LST [68] in natural language processing connect side network in parallel with the pre-trained model to avoid data flow from the trainable parameters to the frozen ones. Our Res- ras-Tuning--.. Tuning-Bypaas is inspired by the conceptual idea of these approuches. Compared to Side- Tuning and LST, we show that our Res-Tuning -Byass is more flexible and memory-efficient.
Conclusion
In this work, unbind the tuners from the backbone and form flexible and efficient tuning paradigm Res-Tuning. With Res-Tuning, we are able to find stronger tuning strategies compared existing ones. On top Res- Res-Tuning, we further extend a memory-efficiee significantly reduces the memory consumption and multi-task inference cost. We hope our discoveries can facilitate further research in the flexible and efficient tuning of large foundation models.
-ess-Tunnnn-Byaa Bypass, which
10

---

References
[1] J.BB. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: A visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[2] H. Bao, L.. Dong. S. Piao, and F. Wei. BETT: BERT pre-training of image Transformers, arXiv preprint 2021.
arXiv:
[3] C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Kütter, A. Lefraneg, Green, V. Valdés,
A. Sadik, al. Deepmind lab, arXiv preprint arXivv:612.03380, 2016.
[4] L. Bossard, M. Guillaumin, and L. Van Gool. Food-101 mining discriminative components with random forests. Eur. Conf. Comput. Vis., 2014.
[5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, Language models are few-shot learners. Adv. Neural Inform. Process. Syst., 2020.
A. Askell,
[6] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with Transformers. In Eur. Conf. Comput. Vis., pages 213-22112222222222222 229, 2020.
[7] S. Chen, C. Ge, Z. Tong. J. Wang, Y. Song, J. Wang, and P. Luo. AdaptFormer: Adapting vision Transformers for scalable visual recognition. In Adv. Neural Inform. Process. Syst., 2022.
[8] G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification; Benchmark and state of the art. Proceedings of the IEEE, 2017.
[9] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In IEEE Conf. Comput. Vis. Pattern Recog., 2014.
[10] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer. A. Steiner, M. Caron, R. Geirhos. I. Alabdulmohsin, et al. Scaling Vision Transformers 22 billion parameters. arXiv preprint 4rXiv:3302.05442, 2023.
[11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large- scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., pages 248 255,, 2009.
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding. North Am. Chap, Assoc. Comput. Linguist., 2018
[13] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. Ini. Conf. Learn. Represent. 2020.
[14] Y. Fang, W. Wang, Xie, Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits masked visual representation learning at scale. arXiv preprint arXiv:2211.07636, 2022.
[15] T. Gebru, J. Krause, Y. Wang, D. Chen, J. Deng, and L. Fei-Fei. Fine-grained car detection for visual consus estimation. Assoc. Adv. Artif. Intell., 2017.
[16] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The KITTI dataset. Int. J. Robotics Research, 2013.
[17] Y. Gong, Y.-A. Chung, and J. Glass, Ast: Audio spectrogram transformer. arXiv preprint 0rXv::2000700 2021.
[18] B. Graham. Kaggle diabetic retinopathy detection competition report., 2015.
[19] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrrik, and G. Neuhig. Towards unified view of parameter-efffiient transfer learning. In Inf. Conf. Learn. Represent. 2022.
[20] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In JEEE Conf. Comput. Vis. Pattern Recog., pages 770-778, 2016.
[21] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In IEEE Conf. Comput. Vis. Pattern Recug., pages 16000-16009, 2022.
[22] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.

---

[23] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, Steinhardt, and Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Int. Conf. Comput. Vis., 2021.
[24] D. Hendrycks, Zhao, S. Basart, Steinhardt, and Song. Natural adversarial examples. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[25] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-effiiient transfer learning for NLP. In Int. Conf. Mach. Learn., pages 2790-2799, 2019.
[26] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nip. In Int. Conf. Mach, Learn., pages 2790-2799, 2019.
[27] E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In Inf. Conf. Learn. Represent. 2021.
[28] Jia, L. Tang. B.-C. Chen, C. Cardie, Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning. In Eur. Conf. Comput. Vis., 2022.
[29] S. Jie and Z.-H. Deng. Convolutional bypasses are better vision Transformer adapters. arXiv preprint arXiv:2207.07039 2022.
[30] J. Johnson, B. Hariharan, I. van der Maaten, I.. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In IEFE Conf. Comput. Vis. Pattern Recog. 2017.
[31] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
[32] R. Karimi Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Adv. Neural Inform. Process. Syst., volume 34, pages 1022-1035,200 2021.
[33] R. Karimi Mahabadi, S. Ruder, M. Dehghani, and J. Henderson. Parameter-efficient multi task fine-tuning for Transformers via shared hypemetworks. In Assoc. Comput. Linguist., pages 365-576, 2021.
[34] A. Khosla, N. Jayadevaprakash, B. Yao, and F.F. Li. Novel dataset for fine-grat tne-grained image categorization: Stanford dogs. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., 2011.
[35] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[36] Y. LeCun, F. J. Huang, and I.. Bottou. Learning methods for generic object recognition with invariance pose and lighting. In IEEE Conf. Comput. Vis. Pattern Recog., 2004.
[37] B. Lester, R. Al-Rfou, and N. Constant, The power of scale for parameter-effiiiee prompt tuning. In Conf. Empirical Methods NLP, 2021.
[38] F.F. Li, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE Trans. Pattern Anal. Mach. Intell., 2006.
[39] K. Li, Y. Wang, P. Gao, G. Song, Y. Liu, H. Li, and Y. Qiao. UniFormer: Unified Transformer for efficient spatiotemporal representation learning. In Ins. Conf. Learn. Represent, 2022.
[40] I.. H. Li, M. Yatskar, D. Yin, C.-. Hsich, and K.-W. Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint arXiv: 1908.0355, 2019.
[41] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Assoc. Comput. Linguist. pages 4582-4597, 2021.
[42] D. Lian, D. Zhou, Feng, and X. Wang. Scaling & shifting your features: A new baseline for efficient model tuning. In Adv. Neural Inform. Process. Syst., 2022.
[43] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. In Eur: Conf. Comput. Vis., pages 740 -45, 2014.
[44] L. Liu, Y. Ren, Z. Lin, and Z. Zhao. Pseudo numerical methods for diffusion models on manifolds. In Int. Conf. Learn. Represent., 2022.
12

---

[45] X. Liu, K. Ji. Y. Fu, Z. Du, Z. Yang. and J. Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks, arXiv preprint arXiv:2110.0702,, 2021.
[46] X. Liu, Y. Zheng. Z. Du, M. Ding. Y. Qian, Z. Yang. and J. Tang. GPT understands, too, arXiv preprint arXiv:2103. 10385. 2021.
[47] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, Levy, Lewis, L. Zettlemoycr, and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907 2019.
[48] Z. Liu, Y. Lin, Y. Cao, H. Hu. Y. Wei, Z. Zhang. S. Lin, and B. Guo. Swin Transformer: Hierarchical Vision Transformer using shifted windows. In Int. Conf. Comput. Vis., 2021.
[49] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In JEEE Conf. Comput. Vis. Pattern Recog,, pages 1 11976-11986, 2022.
[50] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Int. Conf. Learn. Represent, 2019.
[51] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
[52] L. Matthey, I. Higgins, D. Hassabis, and A. Lerchner. dSprites: Disentanglement testing sprites dataset, 2017.
[53] T. Meinhardt, A. Kirillov, L. Leal-Taixe and C. Feichtenhofer. Trackformer: Multi-object tracking with transformers. JEEE Conf. Comput. Vis. Pattern Recog., pages 8844-8854, 2022.
[54] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In Ady. Neural Inform. Process. Syst. Worksh. 2011.
[55] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Ind. Conf. Comput. Vis. Gruph. Image Process. pages 722-729, 2008.
[56] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In IEEE Conf. Comput. Vis. Pattern Recog,, 2012.
[57] W. Peebles and S. Xie. Scalable diffusion models with Transformers. arXiv preprint arX.422112.09748. 2022.
[58] Pfeiffer, A. Kamath, A. Rücklé, Cho, and I. Gurevych. AdapterFusion: Non-destructive task composition for transfer leamnnn. In Eur. Chap. Assoc. Compua. Linguist. pages 487-503, 2020.
[59] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., pages 1748-8765, 2021.
[60] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and J. Liu. Exploring the limits of transfer learning with unified text-to-text Transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
[61] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06123 2022.
[62] B. Recht, R. Roclofs, L. Schmidt, and V. Shankar, Do imagenet classifiers generalize to imagenet? In Int. Conf. Mach. Learn., 2019.
[63] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., pages 10684-10695, 2022.
[64] O. Ronneherger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In Med. Image Comput. Computer-Assisted Interv., pages 234-241, 2015.
[65] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text- to-image diffusson models for subject-driven generation. IEEE Conf. Comput. Vis. Pattern Recog.. 2022.
[66] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Conf. Empirical Methods NLP, pages 1631-1642. 2013.
13

---

[67] H. Song, D. Sun, S. Chun, V. Jampani. D. Han, B. Heo, W. Kim, and M.-H. Yang. ViDT: An efficient and effective fully Transformer-based object detector. In Int. Conf. Learn. Represent, 2022.
[68] Y.-L. Sung, J. Cho, and M. Bansal. LST: Ladder side-tuning for parameter and memory efficient transfer learning. In Adv. Neural Inform. Process. Syst., 2022.
[69] G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 595 604, 2015.
[70] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need, In Adv. Neural Inform. Process. Syst. volume 30, 2017.
[71] B. S. Veeling, J. Limmans, J. Winkens, T. Cohen, and M. Welling. Rotation equivariant cnns for digital pathology. In Med. Image Comput. Computer-Asiisted Interv. 2018
[72] C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 dataset. California Institute of Technology. 2011.
[73] H. Wang, S. Ge, Z. C. Lipton, and F. Xing. Learning robust global representations by penalizing local predictive power. In Adv. Neural Inform. Prucess. Syst., 2019.
[74] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision Transformer: A versatile backbone for dense prediction without convolutions. Int. Conf. Comput. Vis., pages 568-578, 2021.
[75] Wang. H. Bao. L. Dong. Bjorck, Peng. Liu, Aggarwal, O. K. Mohammed. S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[76] 7. Wang, 7. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. DualPrompt: Complementary prompting for rehearsal-free continual learning. In Eur Conf. Comput. Vis.. 2022.
[77] A. Williams, N. Nangia, and S. Bowman, A broad-coverage challenge corpus for sentence understanding through inference. In North Am. Chap, Assoc. Comput. Linguist., pages 1112 1122, 2018
[78] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3485-3492, 2010.
[79] F. B. Zaken, S. Ravfogel, and Y. Goldberg. BitFit: Simple parameter-efficent fine-tuning for Transformer- based masked language- models.. Assoc. Comput. Linguist., 2021.
[80] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, Dosovitskiy, al. large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1910.04677 2019.
[81] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling Vision Transformers. In IEEE Conf. Comput.
Vis. Puttern Recog,, pages 12104-12113, 2022.
[82] J. O. Zhang, A. Sax, A. Zamir, L. Guibas, and Malik. Side-Tuning: Network adaptation via additive side networks. In Eur. Conf. Comput. Vis., 2019.
[83] Y. Zhang, K. Zhou, and Z. Liu. Neural prompt search. arXiv preprint arXiv:2206.04672 2022
[84] S. Zheng, J. Lu, H. Zhao, X. 7hu, 7. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, and I.. 7hang. Rethinking semantic segmentation from sequence-to-ssqqeeme perspective with Transformers. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[85] K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In IEEE Conf. Comput. Vis. Pattern Recog. pages 16816-16825, 2022.
14

---

In the appendix, provide a detailed derivation process for turning existing PETL methods into our unbinding formulation (appendix A), demonstration of our unbinding formulation as unified formulation for existing PETI. methods (appendix B), more implementation details (appendix C) including the dataset used, architectures for discriminative and generative tasks, and hyperparameters used in training, additional results on discriminative tasks (appendix D) and generative tasks (appendix E).
A Detailed derivations
In this section, we provide the detailed deriving process for the existing prefix [41] and prompt [28] tuning to be turned into our unbinding formulation.
Prefix tuning [41]: The following the detailed derivation of Eq. (3).
MHApp = Attn(æ prrss x Wk], k), [V
pres x W.))
Vire
softmax (æW (K pre; [W:]
xW.
(1 X(x)) softmax (ar W W W. ((rr) (aWW, W,KK pre) Vpre
(1 X(=)) Attn (2W, 2 wW k. x W.) X(x) Attn (æ q: K xpre Vpre)
(1 XQ K, K pre)) Attn (Q, K, V) + +X(Q, K, pre) Attn (Q, Krees Vpre)
standard atlention
independent of Kyrei Vpre
where Q. K denote the original query and original key, K and Vpre are prefix key tokens and prefix value tokens, and
exp exp (QK pre)) X(Q, K, K pre) -E exp (QKT), + Ej exp QQTTe))
(10)
Prompt tuning [28]: The following is the detailed derivation of Eq. (5).
MHApp pro = Attn ([x; proo Wg. x xrro Wk. [ xprol We) )..)
= concat softmaa (#W
W Wal
aW " Epro W.
xWW, Epro W.
1)
softmax (xpro Wg Epo Wx]
concat - (11-AQ,,K) K, K pro)) Attn (Q, K, V) + + K, K Kpro) Attn (Q, Krro Vpro),
- ([_ppop K pros K)) Attn (Qpror K Krro, Vpro)
+3(Q pro: K Kpro, K) Attn (Q Qrro K, V))
(11)
where K and Vpro are prompt key tokens and prompt value tokens, and
Σ, exp (QKT.))
X(Q, K, proo)= = Σ, exp exp (QK )) + +, exp (QKT))
(12)
E, exp (Q Qpro
3(Q K Karas K) Σ, exp (Qpro pro); + E, exp (Q (proK())
(13)
15

---

B Unified formulation
From Eq. (7), we derive unified formulation for existing PETL methods, which is the combination of the frozen pre-trained operation (OP) and the tuner with learnable parameters (Res- Tunerr... The following the detailed instantiation of this unified formulation (as in Tab. 7). Thanks to the form of unbinding, tuners can be treated as individual and flexibly combined. Our framework can effectively encompass existing tuning methods residual form and is not limited to the OP and mentioned in our work. For example, for MAM-Adapter, we are free to attach Res-Prefit and Res-Adapter to MHA and FFN modules, respectively. particular, when new PETL methods are proposed, they can be quickly applied (abbreviated as New- New-Tuning), and different methods can be arbitrarily combined (abbreviated as Min-nn.
Table 7: The combination of Res des-Tuning.
Method
OP
Res-Tuner
Adapter [25]
FFN
Res-Adapter
Prefix [41]
MHA
Res-Prefix
Prompt [28]
MHA
Res Res-Prompt
LoRA [27]
Weight
MLP
BitFit [79]
Bias
Parameter
AdaptFormer [7]
FFN
Res-Adapter
MAM-Adapter [19]
MHA FFN
Res-Prefix Res-Adapter
Side-Tuning [82]
All blocks
Side model
LST [68]
Block
ViT
New- New-Tuning
Any where
New-Tuner
Mix- Mix-Tuning
Any combination
Any combination
Implementation details
C.1 Dataset description
In Tab. and Tab. 9, we list the description of each dataset in our experiments, which includes the number of classes and the amount of images in training and test set for discriminative and generative tasks, respectively.
Table 8: Datasets used for generative tasks. denotes only part of the data is used.
Train prompt
Test prompt
Dataset
Description
Classes
image
image
Common Objects in Context (COCO)
COCO2017 Captions
common objects
118287
591753
5000
25014
Fine-grained Image Generation
Aireraft [51]*
Fine-grained aireraft
100
3334
3333
Food-101 [4]*
Fine-grained food
101
75750
25250
NABirds [69]]
Fine-grained bird
555
23929
24633
Stanford Cars [15]*
Fine-grained car
196
8144
8144
Stanford Dogs [34]*
Fine-grained dog
120
12000
8580
SUN397 [78]*
Fine- Fine-graine grained scene
397
76044
16295
16

---

Table 9: Datasets used for discriminative tasks.
Dataset
Description
Classes
Train
Test
General Image Classification
CIFAR-100 [35]
General
100
50000
10000
Fine-grained Visual Classification (FGVC)
CUB-200-2011 [72]
Bird
200
5994
5794
NABirds [69]
Bird
555
23929
24633
Oxford Flowers [55]
Flower
102
1020
6149
Stanford Cars [15]
Car
196
8144
8041
Stanford Dogs [34]
Dog
120
12000
8580
Visual Task Adaptation Benchmark (VTAB-IK)
CIFAR- [35]
100
10000
Caltech101 [38]
102
6084
DTD [9]
47
1880
Flower102 [55]
Natural
102
1000
6149
Pets [56]
37
3669
SVHN [54]
10
26032
SUN397 [78]
397
21750
Camelyon [71]
2
32768
EuroSAT [22]
10
5400
Resisc45 [8]
45
6300
Specialized
1000
Retinopathy [18]
5
42670
Clevr-Count [30]
8
15000
Clevr-Dist [30]
6
15000
DMLab [3]
6
22735
KITTI-Dist [16]
4
711
dSpr-Loc [52]
Structured
16
1000
73728
dSpr-Ori [52]
16
73728
SNORB-Azim [36]
18
12150
SNORB-Ele [36]
9
12150
Few-Shot Learning
FGVC-Aircratt [51]
Aircraft
100
(1/2/4/8/16)
class
3333
Food-101 [4]
Food
101
(1/2/4/8/16 16)
class
30300
Oxford Flowers [55]
Flower
102
(1/2/4/8/16
class
2463
Oxford Pets [56]
Pet
37
(1/2/4/8/16 16) class
3669
Stanford Cars [15]
Car
196
(1/2/4/8/16) class
8041
Domain Generalization
ImageNet-IK IK [11]
General
1000
16 class
50000
ImageNet-V V2 [62]
General
1000
10000
ImageNet-Sketch [73]
General
1000
/
50889
ImageNet-A [24]
General
200
/
7500
ImageNet-R [23]
General
200
/
30000
17

---

C.2 Architectures design
We show the complete -eningn Tuning-Bypass structural design according the different task frameworks. For discriminative tasks, we design based on the architecture of Vision Transformers [13], mainly based on main blocks and unbound bypass (as in Fig. 8). For generative tasks, we design based on the stable diffusion [63] framework and innovatively apply Res-Tuner to U-Net [64] structure (as in Fig. 9). We attach Tuner to the U-Net intermediate module and decoder module for more efficient training.
Head
Block_N N
Tuner
Res
+
Res
Tuner
Vertical
Res-
Tuner
Block 2
Tuner
Horizontal
Res- Tuner
Froward
Backward
Frazen
Block_1 1
Trainable
VIT
Dus-Tuninng-Bpaas
Figure 8: Architecture design for discriminative tasks.
18

---

Decoder
Decoder 64x64
Res- Tuner
Tuner
Decoder 32x32
Res- Tuner
Res- Tuner
Vertical
Decoder 16x16
Res- Tuner
Res-
Tuner
Decoder 8x8
Res- Tuner
Horizontal
Res-
Tuner
Middle 8x8
Encoder 8x8
Res Res-Tuning-Bypass
Encoder 16x16
Froward
Backward
Encoder 32x32
Detach
Frozen
Encoder
Trainable
64x64
Conditional 2D U-Net
Figure 9: Architecture design for generative tasks.
19

---

C.3 Hyperparameters
We list all the hyperparameters for discriminative (see in Tab. 10) and generative (see in Tab. 11) tasks.
Table 10: Hyperparameter selection for discriminative tasks.
Config
Value
Batch size
32
Optimizer
AdamW [50]
Weight decay
0.05
Base learning rate range
{0.05, 0.01, 0.005, 0.001
Learning rate schedule
Cosine decay
Training epochs range
[50, 100)
Warmup epochs
10
Augmentation
RandomResizzedrop. RandomHorizonaalliiip
MHA [70]
OP
FFN [70]
Block [70]
Res-Adapter
Res-Tuner
Res-Prefix
Res-Prompt
ViT/B-16 [13]
Architecture
VIT/L-14 [13]
ImageNet-21K: [11]
Pre-trained
CLIP [59]
Device
A100 x
Table 11: Hyperparameter selection for generative tasks.
Config
Value
Batch size
Optimizer
AdamW [50]
Base learning rate
lc-4
Learning rate schedule
Constant
Training epochs
Augmentation
CenterCrop, RandomHorizontllFiig
Resolution
512 512
Sampler
PLMS [44]
Guidance
3.0
MHA [70]
FFN [70]
OP
Block [70]
Conditional 2D U-Net [64]
Res-Adapter
Res-Tuner
Res-Prefix
Res-Prompi
Architecture
Stable Diffusion [63]
Stable Diffusion v1.5 [63]
Pre-trained
CL.IP [59]
Device
A100
Library
Diffusers
htunss/gghhbb.oom/nnggggghcchcct
20

---

D Additional experiments on discriminative tasks
D.1 Comparisons on FGVC datasets
We compare the transfer ability of our Res- Tuninng framework with different existing approaches for parameter- and memory-efficient transfer learning on FGVC datasets. As shown in Tab. 12. our Res-Tuning outperforms other tuning methods on the average accuracy of five FGVC datasets. For memory-efficient methods, Res- -aa-innn -arianggg achieves 2.89% improvement compared to LST [68] with less memory consumption.
Table 12: Performance and efficiency comparison on FGVC. denotes our own implementation.
Datasets
CUB- 200-2011
Oxford Flowers
Stanford Cars
Stanford Dogs
Param.
Mem.
NABirds
Mean
Method
(M)
(GB)
Full
87.3
82.7
98.8
84.5
89.4
88.54
85,98
9,40
Linear
85.3
75.9
97.9
51.3
86.2
79.32
0.18
3.09
Parameter- efficient tuning methods
Adapter [25]
87.3
84.3
98.4
68.4
88.8
85.46
1.96
6.55
Prefix [41]
85,4
78.8
99.2
76.4
89.5
85.86
0.36
6.60
VPT-Shallow [28]
86.7
78.8
98.4
68.7
90.7
84.62
0.25
8.14
VPT-Deep [28]
88.5
84.2
99.0
83.6
90.2
89.11
0.85
8.16
LoRAi [27]
86.0
80.2
99.2
85.2
88.6
87.84
0.55
6.78
Convpass [29]
86.9
81.4
99.3
85.7
89.9
88.66
0.51
7.44
SSF [42]
89.5
85.7
99.6
89.2
89.6
90.72
0.39
7.47
Res ResiTuning
89.66
85.87
99.45
87.58
92.21
90.95
0.68
8.98
Memory-efficient tuning methods
Side- Side-Tuning [82]
84.7
75.8
96.9
48.6
85.8
78.35
9.73
3.48
LST [68]
82.98
76.11
98.83
78.46
87.89
84.85
1.04
5.28
Res Res-Tuning Baaa-uning-Bypass
88.75
83.00
99.61
75.41
92.40
87.83
0.56
4.73
D.2
More ablation studies
Varying the length of dimensions. The length of dimensions represents the hidden dimension for Res- Adapter and the number of tokens of Res-Prompt and Res-Prefix. Altering the length of dimensions affects performances, the number of parameters, and memory consumption simultaneously (see in Tab. 13). By default, we use the length that balances the number parameters and memory, although there is slight increase the length grows to a certain extent.
Table 13: Ablation studies on the length of dimension for Res des-Tuning on CIFAR-100. Default
setting is marked in gray.
Res- Tes-Tuning
Res Tuninn- Byynng
Dim
Acc.
Param.
Mem.
Acc.
Param.
Mem.
93,03
0.30M
6.84G
89.16
0.37M
4.65G
93.25
0.48M
6.85G
89.33
0.46M
4.72G
20
92.98
0,85M
6.87G
89.28
0.64M
4.83G
50
92.69
1.96M
6.90G
89.22
1.19M
5.08G
100
92,48
3.80M
6.96G
89.52
2.11M
5.68G
21

---

Different number of block layers. Based on the ViT/B- 16 structure, we change the number of block layers in two ways: (i) increase from head toe gradually (see in Tab. 14), (ii) discard several intermediate layers (see in Tab. 15), and evaluate the impact of using different number of layers. can be seen that the best performance can be achieved by using the full number of layers, but requires the use of relatively more parameters and memory.
Table 14: Ablation studies on the tuning block layers for Res-Tuning on CIFAR-100. The right arrow denotes use the number of all layers from left to right. Default setting is marked gray.
Res-Tuning Param.
Blocks
Acc.
Mem.
Acc.
Param.
Mem.
1 1
92.23
0.11M
6,43G
88.50
0.11M
3.58M
1 +
92.78
0.18M
6.52G
88.60
0.17M
3.73G
1 +
92.93
0.28M
6.64G
89.05
0.27M
4.09G
1 1
93.17
0.38M
6,84G
89.32
0.36M
4.53G
1 -
12
93.25
0.48M
6.85G
89.33
0,46M
4.72G
Table 15: Ablation studies on the drop block layers for Res-Tuning on CIFAR-100. Inside the braces indicate the layer number that is discarded. The default setting marked in gray.
Men-Tuning Param.
Res-
Truning-Byppr
Drop block layers
Acc.
Mem.
Acc.
Param.
Mem.
93.25
0.48M
6.85G
89.33
0.46M
4.72G
[3, 6, 9)
92.88
0.38M
6.73G
89.30
0.36M
4.36G
(2, 3, 5, 7,,,,99117
92.96
0.28M
6.68G
88.89
0.27M
3.96G
[2, 3, 4, 5, 6, 8, 9, 10 11)
92.71
0.18M
6.56G
88.79
0.17M
3.65G
11, 3. 4, 5, 6, 7, 8, 9, 10, 11)
87.77
0.11M
3.31G
88.49
0.1
3.48G
Different CNN backbones. We present the results for ConvNeXt [49] and ResNet-101 [20] pre- trained on ImageNet-21K in Tab. 16. We observe that convolutional models have different characteristics, where the performance variation of ConvNeXt is small and that of ResNet is large. In terms of the effectiveness of Ser--Tunnnn-yyyass it observed that it outperforms the fully- finetuned version of ConvNeXt and notably improves the performance of linear probing for ResNet.
Table 16: Ablation studies on the CNN-based backbones on CIFAR-100.
ConvNext
ResNet-101
Method
Acc.
Param.
Mem.
Acc.
Param.
Mem.
Full
90.15
87.67
11.16
77.40
42.70
7.30
Linear
90.06
0.11
3.36
54.96
0.20
2.83
Res Res-Tuning
90.86
0.87
9.45
86,80
0.92
7.20
Res Mla-Tuning-ao -uniing-Bppasss
90.51
1.13
3.63
72.27
3.63
4.05
Experiments on NLP downstream task. We also conduct experiments on downstream tasks beyond vision. For NLP, we perform experiments on the SST2 [66] and MNLI datasets for text classification tasks. observed that the performance of our Res-Tuming framework is on par with or better than MAM-Adapter [19] in text classification,, with slightly longer training time but lower memory consumption. Our Res-Tuning- -Tuning-Bypass Bypass significantly reduces the training time and memory consumption and achieves mildly lower performance.
22

---

Table 17: Performance comparison on text classification task.
SST2
MNLI
Train Param.
Test Param.
Mem.
Method
Acc.
Train Time
Acc.
Train Time
MAM Adapter [19]
94.2
7.2
87.4
41.4
|66.78 (37.4%)
0.61 (0.5%) 22.4G
Res- -es-Tuning
94.56
7.9
87.45
47.3
0.97 (0.77%) 0.97 (0.77%) 19.3G
Res Tuuning-yppaan
92.94
4.2
82.01
24.2
0.98 (0.78%)
0.98 (0.78%) 4.3G
E
Additional Experiments on generative tasks
E.1
More visualizations on COCO
We present more visualization results, comparing real images, stable diffusion [63] (SD) v1.5, fully fine-tuning, LoRA [27]. Res-Tuning and Res-Tuning- BeriinngBypasssse in Fig. 10 and more detailed generated image in Fig. 11. Specifically, the text conditions of text-to-image task are sampled from COCO Captions. better demonstrate the advantages of our approach in understanding the concepts of text, the main elements are marked in blue, and the main modifiers or prepositions are marked in green.
E.2 More visualizations on fine-grained datasets
We present more visualization results on fine-grained datasets (see in Fig. 12, Fig. 13, and Fig. 14), including NABirds [69]. Stanford Dogs [34], Stanford Cars [15], Aircraft [51], SUN397 [78] and Food-101 [4].
F Limitations and Societal Impacts
This work a tuning paradigm that fine-tuned based on pre-trained foundation models while freezing the backbonn network, so its transfer ability depends to large extent the performance of the upstream model. However, when the upstream gre-training model contains illegal content training. it will also lead to the illegal use of tuning methods.
23

---

Real
vD
Full
LoRA
Res Bes--Pune Puning
Res Pwr-nuunnngggg Bypass
A teddy bear is sitting in front group of boxes.
A very cute looking dog with disc on the beach.
A hotel room with chandelier and glass fixtures. display near a wall.
A vase with flowers on a
Figure 10: Qualitative results of existing tuning strategies and our Res- Res-Tuning on COC02017 validation set.
24

---

A landscape with water, a boat, trees and mountains.
Two elephants standing next to each other in field.
Bananas, marshmellows, chocolate chips and sprinkles in bowl.
A single white rose in a glass vase.
The rain is pouring on the white car on the street.
A close up of glazed donuts that are plain or with chocolate.
Figure 11: Visualization of our Res- Tuning on COCO2017 validation set.
25

---

NABirds
Stanford Dogs
aa Res-Tuning
(b) Res Bysp
Figure 12: Visualization of our Res-Tuning on NABirds and Stanford Dogs.
26

---

Stanford Cars
Aircraft
.IIO TEERE
aa Res-Tuning
CTU
(b) Bas-Tuning-Bypass
Figure 13: Visualization of Res-Tuning on Stanford Cars and Aircraft.
27

---

SUN397
Food-101
(a) Res-Tuning
(b) Res Bus-n
Figure 14: Visualization of Res-Tuning on SUN397 and Food-10) 101.
28


Reviews:
[{"review_id": "Qqp2EFRZ8Q", "reviewer": "Reviewer_mYTv", "summary": "This paper pays attention to Parameter Efficient Tuning and propose a unified framework namely Res-tuning. More importantly, based on the proposed unified framework, this paper constructs a memory optimization scheme similar to the LST in the language model. Several experiments are performed to validate the effectiveness , including over visual recognition tasks including VTAB-1k, Cifar-100 and text to image generation tasks on COCO and Oxford Flowers and Food-101.\n", "questions": "1. This approach has only been evaluated on ViTs, and I'm curious if Res-tuning-bypass will work on CNNs.\n\n2. The performance of Side Tuning is quite poor, and VPT proves that this memory-saving design is not suitable for visual tasks. This paper replicates the success of LST for the first time in a visual task, and I am very curious about the implementation process. However, the source code is not submitted in the supplementary material. If the authors can provide anonymous github project link in the rebuttal and I am glad to raise my score.", "limitations": "Yes, the authors have discussed the limitaions and the potential societal impacts in the supplementary material.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I am gald to see the success of Res-Tuning-Bypass, an efficient memory PET that is the counterpart of LST in visual tasks.\n\n2. Sufficient experiments are conducted and Res-Tuning achieves impressive performance gain compared to state-of-art approaches.\n\n3. The paper is well structured and easy to follow.", "weaknesses": "1. The novelty of unbinded Res-tuning framework seems limited. As for me, Convpass [1] and AdaptFormer adopts a parallel module for PET, which is similar to Res-tuning.\n\n2. The analysis of prompt tuning, prefix tuning, and adapter tuning is similar to that in [2], which is also cited in the paper. For me, the major contribution of this paper is the  Res-tuning-bypass, which brings the success of memory-efficient side tuning to visual tasks.\n\n3. The research of parameter efficient tuning has gained extensive attention recently. More recent related works should be added and described or even compared. In the work, only VPT, SSF, NoAH, and Adaptformer are used as baselines. More recent work such as Convpass[1], SNF[3] should be carefully examined.\n\n4. Lacking some important experiments, such as few-shot experiments on FGVC and Domain Generalization experiments on four ImageNet related datasets (ImageNet-A, ImageNet-R, ImageNet-V2, ImageNet-Sketch).\n\n[1] Jie, Shibo, and Zhi-Hong Deng. \"Convolutional bypasses are better vision transformer adapters.\" arXiv preprint arXiv:2207.07039 (2022).\n\n[2] He J, Zhou C, Ma X, et al. Towards a Unified View of Parameter-Efficient Transfer Learning[C]//International Conference on Learning Representations. 2021.\n\n[3] Wang Y, Shi B, Zhang X, et al. Adapting Shortcut With Normalizing Flow: An Efficient Tuning Framework for Visual Recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15965-15974.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer mYTv,\n\nThank you for the acknowledgement of our contributions and your valuable comments. We address your concern as follows:\n\n**Q1: Novelty of the unbinded Res-Tuning framework.**\n\nAs mentioned in the general response, the novelty of the unbinded Res-Tuning framework when compared to parallel modules such as Convpass and AdaptFormer is the flexibility, as well as its ability to derive new PETL methods with the unbinding formulation. Empirically, we also show that Res-Tuning framework performs favourably against them (in Table 3 and Table R1). \n\nMore importantly, the significance of the Res-Tuning framework in our manuscript is to provide a basis for the Res-Tuning-Bypass framework. With the unbinding formulation, we can now use the tuners derived from existing PETL methods (proven effective in PETL applications) as the basic building block for the bypass network in Res-Tuning-Bypass.\n\n\n**Q2: Similarity to MAM-Adapter in terms of the analysis of existing PETL methods.**\n\nWe agree that the analysis on the existing PETL methods is similar to that of MAM-Adapter. In fact, our Res-Tuning framework is partially inspired by them. However, we believe that our analysis is more rigorous, where we show the equivalence of our framework and the existing methods both theoretically and empirically. \n\nWe also agree that the major contribution of the manuscript is Res-Tuning-Bypass. We sincerely thank you for the acknowledgement of our contribution. In the revisions, we will carefully reorganize the manuscript to show that more clearly. \n\n**Q3: More comparison experiments on recent work**\n\nThanks for the suggestion. Here, we provide an empirical comparison of the mentioned methods in the Table 1 of the extra pdf material. Overall, our Res-Tuning framework shows a competitive performance. We will try to include more recent works in our revision.\n\n**Q4: Extended experiments on Few-Shot Learning and Domain Generalization**\n\nThanks for the suggestion. We have added the experiments as follows:\n\n- Domain Generalization\n\nFollow the setting of NOAH[4], we first train a model on ImageNet using 16 shots per category and test it on four other variants of ImageNet. The list of data also comes from NOAH and all results are averaged over three random seeds.\n\nOur Res-Tuning goes beyond NOAH by 6.54\\% on ImageNet and 2.6\\% on the average accuracy of ImageNet. Surprisingly, Res-Tuning-Bypass also achieved better results than other tuning methods.\n\nTable R1. Results on domain generalization. 'Mean' denotes the average accuracy of four variants of ImageNet.\n|  | Source | Target |  |  |  |  |\n|---|---|:---:|:---:|:---:|:---:|:---:|\n|  | ImageNet | IN-V2 | IN-Sketch | IN-A | IN-R | Mean |\n| Adapter | 70.5 | 59.1 | 16.4 | 5.5 | 22.1 | 34.7 |\n| VPT | 70.5 | 58.0 | 18.3 | 4.6 | 23.2 | 34.9 |\n| LoRA | 70.8 | 59.3 | 20.0 | 6.9 | 23.3 | 36.1 |\n| NOAH | 71.5 | 66.1 | 24.8 | 11.9 | 28.5 | 40.6 |\n| Res-Tuning | 78.04 | 66.58 | 29.23 | 13.15 | 29.01 | 43.20 |\n| LST† | 70.00 | 57.04 | 14.39 | 7.21 | 17.02 | 33.13 |\n| Res-Tuning-Bypass | 77.30 | 65.23 | 27.39 | 10.66 | 26.45 | 41.41 |\n\n- Few-Shot Learning\n\nWe have included the results for few-shot learning in the extra pdf material. Specifically, we include the few-shot performance on FGVC in Figure 1. Compared with existing parameter-efficient tuning methods, our Res-Tuning shows a certain advantage over the few-shot performance on the FGVC dataset.\n\n**Q5: Extension of Res-Tuning-Bypass to CNNs.**\n\nThis is indeed an interesting aspect to explore. We present the results for ConvNeXt pretrained on IN21K and ResNet-101 pretrained on IN1K in Table R2. \nWe observe that two convolutional models have different characteristics, where the performance variations of ConvNeXt is small and that of ResNet is large. \nIn terms of the effectiveness of Res-Tuning-Bypass, it is observed that it outperforms the fully-finetuned version of ConvNeXt and notably improves the performance of linear probing for ResNet. \n\nTable R2. Results on CNNs backbones.\n|  | ConvNext |  |  | ResNet-101 |  |  |\n|---|---|---|---|---|---|---|\n| Method | Accuracy | Params(M) | Mem. (GB) | Accuracy | Params(M) | Mem. (GB) |\n| Full | 90.15 | 87.67 | 11.16 | 77.40 | 42.70 | 7.30 |\n| Linear | 90.06 | 0.11 | 3.36 | 54.96 | 0.20 | 2.83 |\n| Res-Tuning-Bypass | 90.51 | 1.13 | 3.63 | 72.27 | 3.63 | 4.05 |\n\n**Q6: Code release.**\n\nWe are sorry that at the current stage, we are only able to provide the core part of the model implementation as well as relevant documentations (which we have submitted to the AC) due to the disclosure policy on codes within our organization. But we are actively preparing the formal release of the full code for training and evaluating our framework and we promise we will do that in the near future.  \n\n[1] Jie et al. Convolutional bypasses are better vision transformer adapters. arXiv.\n\n[2] Chen et al. AdaptFormer: Adapting vision Transformers for scalable visual recognition. NeurIPS2022.\n\n[3] He et al. Towards a unified view of parameter-efficient transfer learning. ICLR2022.\n\n[4] Wang et al. Adapting Shortcut With Normalizing Flow: An Efficient Tuning Framework for Visual Recognition. CVPR2023.\n\n[5] Zhang et al. Neural Prompt Search. arXiv."}, {"review_id": "TasQthQ2Dc", "reviewer": "Reviewer_Yhf3", "summary": "This paper proposes an unbinding formulation of parameter-efficient methods and further leverage structural disentanglement to develop a memory-efficient variant.\nSufficient experiments on both visual discriminative and text-to-image generative tasks are performed.", "questions": "The authors employ the CLIP model for text-to-image generation, however, the generative model is more difficult to evaluate its performance.\nCLIP-based Parameter efficient approaches have also gained extensive attention recently, such as CoOp[1], PLOT[2], MaPLe[3], and CoPrompt[4]. I am curious whether the proposed Res-tuning-bypass can also reduce the memory cost for vision-language PET approaches with small performance degradation.\n\n[1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022.\n\n[2] Chen G, Yao W, Song X, et al. PLOT: Prompt Learning with Optimal Transport for Vision-Language Models[C]//The Eleventh International Conference on Learning Representations. 2022.\n\n[3] Khattak M U, Rasheed H, Maaz M, et al. Maple: Multi-modal prompt learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 19113-19122.\n\n[4] Roy S, Etemad A. Consistency-guided Prompt Learning for Vision-Language Models[J]. arXiv preprint arXiv:2306.01195, 2023.\n", "limitations": "Yes, the authors have addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. novel implementation.\n2. good motivation, which is reasonable for me.\n3. good evaluation, promising results, and easy to follow.", "weaknesses": "1. The proposed Res-Tuning is not that novel and the analysis of several PET approaches is similar to that in Towards a unified view of parameter-efficient transfer learning (ICLR2022). Parallel adapter design is already a consensus, minor changes seem trivial. \nRather than this boring repetition, I find the memory-efficient variant i.e., Res-tuning-bypass is meaningful and novel. I suggest that the authors make this section a priority and restructure the article.\n2. Following Lader Side Tuning, the proposed Res-tuning-bypass can free the backbone from backpropagation. In this way, the training time will be reduced, however, only the training time for generating tasks is reported. Thus I am curious about the training time for visual recognition tasks.\n3. The authors compute inference time for multi-tasks, although it seems a bit trivial, I recognize the significance of this time reduction.\nHowever, I cannot accept the comparison with just non memory-efficient methods in Figure 4.  The authors should report LST's results and discuss them carefully.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nDear Reviewer Yhf3,\n\nThank you for the acknowledgement of the proposed method and experiments. We address your concerns as follows:\n\n**Q1: Novelty of Res-Tuning and similarity to MAM-Adapter in terms of analysis on existing methods.**\n\nWe totally agree that the priority of the manuscript is the memory-efficient Res-Tuning-Bypass, and we will reorganize the manuscript to make this clearer. \n\nAs mentioned in the general response, the novelty of Res-Tuning lies in its flexibility. It encompasses most popular existing approaches and can derive new ones. Empirically, we found the performance of Res-Tuning stronger than existing PETL methods. \n\nIn terms of the similarity to MAM-Adapter with respect to the analysis on existing methods, we claim that our analysis is more rigorous, proving the equivalance of our formulation and existing works both theoretically and empirically. This also provides the foundation for further research on parallel modules for PETL. \n\nEssentially, we believe that Res-Tuning is an indispensible part of our manuscript as it serves as the basis for Res-Tuning-Bypass. Nevertheless, we will adjust the organization of our manuscript to make the emphasis on Res-Tuning-Bypass clearer in our future revisions. \n\n**Q2: Training time on discriminative tasks**\n\nWe provide the training time on discriminative tasks in Table R1. \n\nTable R1. The training time on CIFAR-100 corresponds to the Table 2c in manuscript.\n\n| Method | Train (Min/Epoch) | Percentage w.r.t. Full |\n|---|---|---|\n| Full | 2.65 | 100.00% |\n| Linear | 1.19 | 45.10% |\n| MAM-Adapter | 3.07 | 115.72% |\n| AdaptFormer  | 2.18 | 82.23% |\n| Res-Tuning | 2.46 | 92.91% |\n| Side-Tuning | 1.33 | 50.15% |\n| LST | 2.22 | 83.65% |\n| Res-Tuning-Bypass | 1.92 | 72.52% |\n\n\n**Q3: Inference time for multi-tasks**\n\nThanks for the suggestion. \n\nFigure 4 mainly demonstrates the advantage of Res-Tuning-Bypass framework in terms of inference efficiency on multiple tasks. In fact, such a property is shared by approaches similar to ours, which includes Side-Tuning, LST, as well as linear probing. Including other memory-efficient approaches would make the figure a bit scattered, and it would be almost impossible to distinguish the memory-efficient approaches in the figure. Hence, we will include another figure for comparing the inference efficiency of memory-efficient approaches.\n\nAdditionally, we provide a table here for reference in terms of the inference efficiency. Generally, the inference time grows linearly with the number of tasks, so we use the inference time per additional task for comparing the efficiency in Table R2. Overall, the inference time for different approaches is similar. The additional inference time introduced on top of linear probing for Res-Tuning-Bypass is higher than Side-Tuning and lower than LST, but since there are around 10K data for each test dataset, the additional latency introduced by memory-efficient approaches could be negelected (less than 1ms per sample). \n\nTable R2. The testing time on CIFAR-100 between different memory-efficient methods.\n| Method | Testing (sec./test dataset) |\n|---|---|\n| Linear | 23.90 |\n| Side-Tuning | 24.41 |\n| LST | 25.28 |\n| Res-Tuning-Bypass | 24.98 |\n\n\n**Q4: Further evaluations on vision-language task.**\n\n\nWe include the results on the vision-language tasks in Figure 2 in the extra pdf material. We noticed that with the same backbone (ViT), the performance is slightly worse than CoOp. In Table R3, we provide the comparison in terms of parameter and the memory consumption between CoOp and Res-Tuning-Bypass. It is observed that the memory consumption of Res-Tuning-Bypass is reduced by 45% when compared to CoOp. In our future revisions, we will include more relevant results and comparisons.\n\nTable R3. Comparison of parameter and memory consumption between CoOp and our Res-Tuning-Bypass. \n|  | Param. (M) | Mem. (GB) | Percentage |\n|---|---|---|---|\n| CoOp | 0.008 | 5768 | 100% |\n| Res-Tuning-Bypass | 0.38 | 3174 | 55% |\n\n[1] Sung et al. LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning. NeurIPS2022.\n\n[2] Zhou et al. Learning to prompt for vision-language models. IJCV2022.\n\n\n"}, {"review_id": "l1QRmV4NKj", "reviewer": "Reviewer_nLwc", "summary": "This paper shows some existing parameter-efficient tuning methods can be decoupled from backbones, which can be formulated as a unified Res-Tuning model. Furthermore, the authors conduct empirical experiments to seek the optimal Res-Tuner. Additionally, a memory-efficient variant of Res-Tuning is introduced by combining outputs of backbone with previous Res-Tuner, which can only compute the gradients for Res-Tuning module. The experiments are conducted on several downstream tasks.", "questions": "Please see paper weaknesses. ", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+: Compared with parameter-efficient tuning, memory and speed efficient tuning methods show more practical in real-world applications. The proposed Res-Tuning-Bypass show clearly superior to memory-efficient counterparts, i.e., Side-Tuning, LST and linear probing. Meanwhile, Res-Tuning can show speed-efficient on multi-task learning.\n\n+: The proposed method is easy to implement, and is clearly described.\n", "weaknesses": "-: The contribution of this work could be further clarified.\n\n(1)\tThe statement on unbinding formulation seems a bit overclaimed. For tuning parameters of large models, parameter-efficient modules can be naturally designed in a cascaded or parallel manner. For parallel structures, outputs of frozen backbones and learnable branch can be fused by element-wise addition or multiplication, and such kinds of structures are often used for design network architectures. For Res-Tuning model, it can be regarded as a parallel structure with element-wise addition for fusion. Therefore, it is hardly regarded as a novel or special structure.\n\n(2)\tThe authors show existing PETL methods have equivalent counterparts in the unbinding formulation, but I feel a bit confused about necessity of this conclusion. Besides, derivation of equivalent on adapter seems be less rigorous. I am not sure could such conclusion help to design Res-Tuner? Because the optimal Res-Tuner seems be decided by empirical experiments.\n\n(3)\tI feel a bit confused about the relationship between the unbinding formulation and memory-efficient variant of Res-Tuning. In my opinion, memory-efficient Res-Tuning can be regarded as an improved LST, where an adapter is used for outputs of frozen backbones before fusing with learnable branches. Besides, learnable modules of LST and Res-Tuning are different. The authors would better give more discussion about above issue. \n\n-: Some issues about experiments.\n\n(1)\tAs shown in Table 3 (c), why Linear probing can be improved when only Bypass is used? The authors would better give more discussion.\n\n(2)\tIn line 172, improvement of 0.94\\% should be corrected to 0.92\\%?\n\n\n-: The writing can be further improved.\n\n(1)\tLine 124, Fig. 3b -> Fig. 3c\n\n(2)\tFFN$_{adapter}$ in Eq. 6 is inconsistent with description of Fig. 2c. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nDear Reviewer nLwc,\n\nThank you for your time and helpful comments. We address your concerns below:\n\n**Q1: The contribution of the unbinding formulation.**\n\nThe essential contribution of the unbinding formulation is to abstract existing PETL methods into a unified formulation of a frozen operation and a learnable tuner. This formulation allows for the **flexible combination** of various approaches, which encompasses existing PETL methods and is able to derive new ones. We would prefer to regard the unbinding formulation as a new perspective towards PETL methods instead of a novel structure. \n\nAdditionally, it also serves as the basis for the memory-efficient version Res-Tuning-Bypass, where the side network is entirely constructed by the Res-Tuners in the Res-Tuning framework.\n\nWe include more detailed discussions in the general response. \n\n**Q2: Equivalence of unbinding formulation**\n\nThe motivation for the unbinding formulation is to provide a unified description to the existing PETL methods. The core reason for us to prove the equivalence between the parallel form and the original form of existing PETL approaches is that the existing PETL methods are proven effective in various tasks, and the equivalence between two forms would ensure the theoretical effectiveness of our formulation. The theoretical proof combined with empirical validation demonstrates that our unbinding formulation is effective and even stronger than existing PETL methods. \n\n**Q3: Relationship between Res-Tuning and Res-Tuning-Bypass.**\n\nSorry for the disorientation. It is indeed that Res-Tuning-Bypass could be viewed as an improved version of LST. From another perspective, the bypass network in the Res-Tuning-Bypass framework is constructed entirely using Res-Tuners, which is the parallel form of existing PETL methods derived in our unbinding formulation. With both the theoretical proof and empirical validation in Res-Tuning, we can safely reduce the design space and rely on the validated structures for constructing the model. Hence, the Res-Tuning framework and its unbinding formulation serves as an important basis for the Res-Tuning-Bypass model. \n\n**Q4: Issues about experiments.**\n\n(1) Improvement of bypass without Res-Tuners.\n\nThanks for spotting this. It is indeed an interesting result that worth discussing. We believe the performance improvement brought by the plain bypasses is because the structure of Res-Tuning-Bypass without Res-Tuners essentially performs feature ensemble for features generated by different layers in the Transformer. \n\n(2) Performance improvement of 0.94%.\n\nThe number referes to the comparison with existing tuning approaches (underlined numbers in Table 2a), where the highest performance is 92.34%, and our Tri-Res-Tuner, where the performance is 93.28%. Hence, we obtain the least performance improvement of 0.94%. \n\n**Q6: Writing issues.**\n\nThanks for the suggestion. We will correct the mistakes in our revisions. \n"}, {"review_id": "fAJxEboTu3", "reviewer": "Reviewer_DEjH", "summary": "This paper proposes a new tuning paradigm, dubbed as Res-Tuning. They first introduce the basic building blocks of foundation models and then unbind three popular tuners from foundation models. They provide theoretical and empirical evidence to support their structural disentanglement. By detaching from the foundation models, they further propose a memory-efficient variant of Res-tuning, dubbed as Res-Tuning-Bypass. They conduct extensive experiments on both discriminative and generative tasks to demonstrate the superiority of their method.", "questions": "- Please see the weaknesses part.\n- There are no queries in FFN/Block,  How did you apply ResPre/ResPro in FFN/Block?\n- The form of adapter tuning in Eq.(6) and Fig. 2c is inconsistent. Is there an FFN before the adapter of the parallel branch?\n- LoRa [1] is an effective method in PETL. Adding a comparison with Lora in discriminative tasks can make this paper more solid.\n- The full fine-tuning and linear probing results in Tab.3 are inconsistent with the results in SSF[2]. The results of the same task (CIFAR-100) and model (ViT-B/16) are relatively low.\n\n#### Reference\n\n[1] E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In Int. Conf. Learn. Represent., 2021.\n\n[2] D. Lian, D. Zhou, J. Feng, and X. Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Adv. Neural Inform. Process. Syst., 2022.", "limitations": "The transfer ability of this method depends to a large extent on the performance of the upstream model. This method shares the same vulnerability as existing PETL solutions.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- This paper is well-written and the illustration of this paper is concise and easy to understand.\n- The idea is simple yet effective.\n- They conduct extensive experiments and the results demonstrate superior efficacy and efficiency on both discriminative and generative tasks.", "weaknesses": "- There are some detail errors in line 124 and line 130. The reference seems to be Fig.3c, not Fig.3b. And there are some punctuation errors in Eq.(8). There are many similar issues in the article, the author should check this paper again to correct them.\n- Code is unavailable now and open-source as soon as possible is beneficial for expanding the influence and credibility of this paper.\n- The author provides detailed derivation in supplementary materials, but the manuscript is hard to understand. It would be helpful if the deriving process have a more detailed explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nDear Reviewer DEjH,\n\nThank you for the acknowledgement of the proposed method and experiments. We address you concerns as follows:\n\n**Q1: Typos.**\n\nThanks for spotting the errors. We will carefully fix them and polish the writing in our revisions.\n\n**Q2: Code release.**\n\nLimited by our organization's disclosure policy, we are unable to provide the full code for training and evaluating the model for now. But we do have submitted a core part of the model implementation to the AC. Additionally, we are actively preparing the release of the full code and will release them in the near future. \n\n**Q3: Detailed explanation for the deriving process.**\n\nThanks for the suggestion. We will add more detailed explanation and make corresponding modifications to make the manuscript more easily understandable in our revisions. \n\n**Q4: How is Res-Pre. and Res-Pro. applied in FFN/Block?**\n\nFor FFN and Block, we apply them directly with the output of FFN or Block as the query. \n\n**Q5: Inconsistency between Eq.6 and Fig.2c.**\n\nThanks for pointing this out. In fact, we set out to use the form in Eq.6, but later we opt for the structure in Fig.2c and remove the FFN before the adapter to avoid the backpropagation through the FFN, especially when it is used in the Res-Tuning-Bypass framework. We will clarify this in the revisions. \n\n**Q6: Experiments for the comparison with LoRA.**\n\nThanks for the suggestion. We will include the following empirical comparisons in the revisions. \n\nTable R1. Performance comparison on FGVC. † denotes our own implementation.\n| Method | CUB_200_2011  | NABirds  | OxfordFlowers  | StanfordCars  | StanfordDogs  | Mean   |\n|---|---|---|---|---|---|---|\n| LoRA† | 86.02 | 80.22 | 99.2 | 85.16 | 88.59 | 87.84 |\n| Res-Tuning | 89.66 | 85.87 | 99.45 | 87.58 | 92.21 | 90.95 |\n| Res-Tuning-Bypass | 88.75 | 83 | 99.61 | 75.41 | 92.4 | 87.83 |\n\nTable R2. Performance comparison on VTAB-1k.\n| Method | Natural Mean | Specialized Mean | Structured Mean | Group Mean |\n|---|---|---|---|---|\n| LoRA | 79.49 | 84.55 | 59.78 | 74.60 |\n| Res-Tuning | 82.29 | 85.46 | 61.19 | 76.32 |\n| Res-Tuning-Bypass | 76.73 | 84.56 | 55.66 | 72.32 |\n\n**Q6: Baseline setting**\n\nFor the baseline performance, we mainly followed the settings in AdaptFormer[1] and obtained a similar performance to theirs. We will try reproducing the result of the baseline of SSF later as well as applying the similar experimental settings on our approach to see the result. \n\n[1] Chen et al. AdaptFormer: Adapting vision Transformers for scalable visual recognition. NeurIPS2022.\n"}, {"review_id": "8ohy4iKvnm", "reviewer": "Reviewer_hNoJ", "summary": "This paper provides a unified framework (called Res-Tuning) to combine different efficient tuning methods. It introduces an unbinding form that integrates existing methods and allows combination flexibility. A memory-efficient variant is also introduced for the sake of training memory efficiency. Experiments are performed on discriminative tasks (CIFAR-100 and VTAB-1K benchmark) and generative tasks (Text-to-image generation on COCO), demonstrating the proposed framework's flexibility and efficiency. \n\n", "questions": "Why does this paper not evaluate the proposed method on NLP tasks that most parameter-efficient tuning methods evaluated on? This paper only includes the reproduced results of MAM-Adapter on CIFAR-100 without any discussion on NLP benchmarks. Since this paper aims to introduce a unified framework for efficient tuning methods, it is necessary to include empirical evaluations on commonly used benchmarks. I would increase my score if sufficient results or discussion were provided. ", "limitations": "Limitations are not adequately discussed in this paper. It is unknown whether the proposed method can generalize well. Negative societal impacts are unknown. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-organized, and the presentation is clear. \n2. The formulation of the unified and unbinding form is sound and straightforward and is novel to this field of research.\n3. The unbinding form allows combination flexibility, which is the main strength of the proposed methods. \n4. Empirical evaluation and theoretical analysis verify the equivalence between the unbinding form and existing approaches. \n5. Empirical evaluation on different standard benchmarks shows better performance among existing approaches.", "weaknesses": "My main concerns about this paper are the comparisons with previous works. \n\nFrom my perspective, this paper partially draws inspiration from [13] (MAM-Adapter), which also struggled to unify parameter-efficient tuning methods such as Adapter, Prefix or Prompt tuning. All these methods are evaluated in NLP tasks rather than vision tasks that this paper used. \n\nOn vision tasks, VPT [18] is particularly proposed to explore PETL methods in vision (rather than providing a unified view), Thus, the evaluation following VPT is not sufficient for this work. Besides, VPT can work well with Convolution-based foundation models and self-supervised pre-trained models (e.g., MAE, MoCo V3). I don't find such discussion in this work.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nDear Reviewer hNoJ,\n\nThank you for the acknowledgement of our contributions and your valuable comments. We address your concern as follows:\n\n**Q1: Comparisons with previous works**\n\nWe would like to first point out that the unified formulation of existing PETL approaches in an unbinding form is only a part of our contribution in the manuscript. The unbinding formulation leads to the Res-Tuning framework, and we discuss its novelty and contribution compared to existing works in detail in the general response. \n\nAnother important contribution of our manuscript that distinguishes our work with existing approaches is the memory-efficient variant called Res-Tuning-Bypass, which is able to reduce the resource and time consumption for both training and inference in many scenarios, and is stronger than other existing memory-efficient methods (Side-Tuning and LST) presented in NLP. \n\n**Q2: Extended experiments on different structure**\n\nFollowing VPT, we provide more analysis on our approach in comparison with the existing PETL approaches. In Table R1, we provide performance comparisons with different pre-training sources. In Table R2, we compare the performances with different convolutional backbones. We will include more relevant results in our revisions. \n\nTable R1. Performance comparison on different pre-trained models on CIFAR-100.\n| Method | MAE | DINO |\n|---|---|---|\n| Full | 85.90 | 87.88 |\n| Linear | 69.83 | 85.51 |\n| Adapter | 85.86 | 89.01 |\n| VPT | 82.44 | 88.33 |\n| Res-Tuning | 86.37 | 89.03 |\n\nTable R2. Performance comparison on different CNN models on CIFAR-100.\n|  | ConvNext |  |  | ResNet-101 |  |  |\n|---|---|---|---|---|---|---|\n| Method | Accuracy | Params(M) | Mem. (GB) | Accuracy | Params(M) | Mem. (GB) |\n| Full | 90.15 | 87.67 | 11.16 | 77.40 | 42.70 | 7.30 |\n| Linear | 90.06 | 0.11 | 3.36 | 54.96 | 0.20 | 2.83 |\n| Res-Tuning | 90.86 | 0.87 | 9.45 | 86.80 | 0.92 | 7.20 |\n| Res-Tuning-Bypass | 90.51 | 1.13 | 3.63 | 72.27 | 3.63 | 4.05 |\n\n\n**Q3: Extended evaluations on NLP benchmarks**\n\nThanks for the suggestion. Limited by time, here, we preliminarily provide the results of text classification in Table R3. It is observed that the performance of our Res-Tuning framework is on par with or better than MAM-Adapter in text classification, with slightly longer training time but lower memory consumption. Our Res-Tuning-Bypass significantly reduces the training time and memory consumption and achieves a mildly lower performance. In our revisions, we will try to include more NLP-based evaluations for more thorough comparisons. \n\nTable R3. Performance comparison with MAM-Adapter on text classification. \n|  | SST2 |  | MNLI |  |  |  |  |\n|---|---|---|---|---|---|---|---|\n|  | Accuracy | Train (Min/Epoch) | Accuracy | Train (Min/Epoch) | Param. (M) (Train） | Param. (M) (Inference） | Mem. (GB) |\n| MAM Adapter | 94.2 | 7.2 | 87.4 | 41.4 | 46.78 (37.4%) | 0.61 (0.5%) | 22416 |\n| Res-Tuning | 94.56 | 7.9 | 87.45 | 47.3 | 0.97 (0.77%) | 0.97 (0.77%) | 19308 |\n| Res-Tuning-Bypass | 92.94 | 4.2 | 82.01 | 24.2 | 0.98 (0.78%) | 0.98 (0.78%) | 4392 |\n\n\n**Q4: Limitations**\n\nSorry for the disorientation. We have discussed the limitations and the potential societal impacts in the supplementary material.\n\n[1] Zhang et al. Side-tuning: Network adaptation via additive side networks. ECCV2019.\n\n[2] Sung et al. LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning. NeurIPS2022.\n\n[3] Jia et al. Visual Prompt Tuning. ECCV2022.\n\n"}]

Author rebuttal:
Dear all, 

We would like to express our gratitude to our reviewers for their valuable comments. For positive comments, 
- memory-efficiency of Res-Tuning-Bypass (R-mYTv, R-nLwc), 
- sufficient and strong experiments (R-mYTv, R-Yhf3, R-DEjH, R-hNoJ), 
- well-structured and easy to follow (R-mYTv, R-nLwc, R-Yhf3, R-DEjH, R-hNoJ), 
- novel (R-Yhf3, R-hNoJ), 
- good motivation (R-Yhf3), 
- and flexible (R-hNoJ), 

we appreciate them and will carry them forward. 

We address the common concern here, which regards the **novelty of the Res-Tuning framework**.

The novelty of the Res-Tuning framework lies in the abstraction of existing PETL methods to the parallel connection of a frozen operation and a learnable Res-Tuner. This allows for the independent development and flexible combination of the structure of the foundation models and the structure of the PETL tuners. 

More importantly, the significance of the Res-Tuning framework is that it serves as the basis for the Res-Tuning-Bypass framework. With the unbinding formulation in the Res-Tuning framework, we can treat existing PETL methods as basic building blocks for constructing the bypass network. Such a formulation also allows for the easy adaptation of the Res-Tuning-Bypass model with the development of new PETL methods, which could be simply achieved by replacing the existing Res-Tuners in the bypass network by new modules developed in the future. 

*Novelty compared to other existing parallel PETL methods.* Our Res-Tuning framework is able to encompass them and derive new methods based on their module. 

*Novelty compared to MAM-Adapter.* The Res-Tuning framework is partially inspired by MAM-Adapter. Besides the modality difference between Res-Tuning and MAM-Adapter, the derivation process for existing PETL methods is more rigorous in Res-Tuning, proving the equivalence of our Tuner with existing methods. We also included the analysis for prompts, which is not included in the analysis of MAM-Adapter. 

Indeed, we agree with Reviewer **mYTv**, **Yhf3**, and **nLwc** that the major contribution of our manuscript is the Res-Tuning-Bypass framework, which we will further clarify in the revisions. 

For other concerns, we address them in the respective comments to the reviewers. 



Thanks and best regards,  

Authors of Submission 12594

Meta-review: This paper initially received 5 borderline accepts. The main concerns about this work are: 1) Limited novelty. The comparisons and differences between previous work need to be further highlighted. 2) The comparisons with other SOTA memory-efficient methods (e.g., Ladder Side-Tuning) are missing. 3) The writing quality needs to be further improved. After the rebuttal, all reviewers think most of their concerns have been addressed, and keep a positive rating for this paper. In summary, I recommend accepting this paper, and hope the authors can incorporate all these discussions into the final camera-ready version.
Decision: Accept (poster)