SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Performance and efficiency comparison on the VTAB-1K benchmark with ViT-B/16 models pre-trained on ImageNet-21K. "Group Mean" denotes the average accuracy of the three subgroups. "All Mean" denotes the average accuracy of 19 downstream tasks.
Natural	Specialized	Structured
CIFAR-100	Caltech101	DTD	Flowers102	Pets	SVHN	Camelyon	EuroSAT	Resisc45	Retinopathy	Clevy-Count	Clevy-Dist	DMLab	KITTI-Dist	dSpr-Loc	dSpr-Ori	sNORB-Azim	sNORB-Elev	Group Mean	All Mean	Param. (M)	Mem. (GB)
Traditional methods
Full	68.9	87.7	64.3	97.2	86.9	87.4	38.8	79.7	95.7	84.2	73.9	56.3	58.6	41.7	65.5	57.5	46.7	25.7	29.1	68.96	65.57	85.84	9.40
Linear	63.4	85.0	63.2	97.0	86.3	36.6	51.0	78.5	87.5	68.6	74.0	34.3	30.6	33.2	55.4	12.5	20.0	9.6	19.2	57.64	52.94	0.04	3.09
Parameter-efficient tuning methods
Adapter [25]	74.2	85.7	62.7	97.8	87.2	36.4	50.7	76.9	89.2	73.5	71.6	45.2	41.8	31.1	56.4	30.4	24.6	13.2	22.0	60.52	56.35	1.82	6.53
LoRA [27]	67.1	91.4	69.4	98.8	90.4	85.3	54.0	84.9	95.3	84.4	73.6	82.9	69.2	49.8	78.5	75.7	47.1	31.0	44.0	74.60	72.30	0.29	6.88
VPT-Deep [28]	78.8	90.8	65.8	98.0	88.3	78.1	49.6	81.8	96.1	83.4	68.4	68.5	60.0	46.5	72.8	73.6	47.9	32.9	37.8	71.96	69.43	0.60	8.13
SSF [42]	69.0	92.6	75.1	99.4	91.8	90.2	52.9	87.4	95.9	87.4	75.5	75.9	62.3	53.3	80.6	77.3	54.9	29.5	37.9	75.69	73.10	0.24	7.47
NOAH [83]	69.6	92.7	70.2	99.1	90.4	86.1	53.7	84.4	95.4	83.9	75.8	82.8	68.9	49.9	81.7	81.8	48.3	32.8	44.2	75.48	73.25	0.42	7.27
Res-Tuning	75.2	92.7	71.9	99.3	91.9	86.7	58.5	86.7	95.6	85.0	74.6	80.2	63.6	50.6	80.2	85.4	55.7	31.9	42.0	76.32	74.10	0.55	8.95
Memory-efficient tuning methods
Side-Tuning [82]	60.7	60.8	53.6	95.5	66.7	34.9	35.3	58.5	87.7	65.2	61.0	27.6	22.6	31.3	51.7	8.2	14.4	9.8	21.8	49.91	45.65	9.59	3.48
LST†[68]	58.0	87.1	66.2	99.1	89.7	63.2	52.6	81.9	92.2	78.5	69.4	68.6	56.1	38.8	73.4	72.9	30.5	16.6	31.0	67.56	64.52	0.89	5.13
Res-Tuning-Bypass	64.5	88.8	73.2	99.4	90.6	63.5	57.2	85.5	95.2	82.4	75.2	70.4	61.0	40.2	66.8	79.2	52.6	26.0	49.3	72.32	69.51	0.42	4.73
$\dagger$  denotes our own implementation since the original approach is proposed for natural language processing.
4.4 Comparisons with existing tuning strategies on different scenarios
Transfer Learning. We mainly evaluate our approach on the basic transfer learning scenario, where pre-trained models are fine-tuned on different downstream tasks. CIFAR-100 [35] is a standard general-purpose image classification dataset. VTAB-1K [80] is composed of 19 various visual classification tasks falling into three categorizations, i.e., natural, specialized, and structured. We compare the performance of our approach and other baseline methods on the following:
CIFAR-100. We present the comparisons to existing tuning strategies in Tab. 3c. For parameter-efficient methods, our Res-Tuning framework notably improves over AdaptFormer [7] by  $1.39\%$  using only  $0.55\%$  extra parameters, which is  $0.91\%$  less than AdaptFormer [7]. For memory-efficient methods, we outperform LST [68] by  $0.61\%$  with memory reduced by 0.54G  $(10\%)$ .
VTAB-1K. In Tab. 4, we present comprehensive evaluation on the 19 datasets on the VTAB-1K benchmark. Both our Res-Tuning and Res-Tuning-Bypass outperform existing approaches respectively within the group of parameter- and memory-efficient tuning methods. Specifically, our Res-Tuning framework achieves a  $0.85\%$  improvement in terms of the average performance on 19 datasets, compared to the previous best performance. For our Res-Tuning-Bypass, we outperform LST [68] in 18 out of 19 tasks and overall by  $4.99\%$  in terms of average performance. This is achieved with even fewer parameters and memory consumption. We further visualize the parameter vs. performance curve and memory vs. performance curve in Fig. 4 to show the advantage of our Res-Tuning and Res-Tuning-Bypass framework on VTAB-1K.
Few-Shot Learning. To evaluate the ability of our approach to adapt with only a few training samples, we follow the few-shot evaluation protocol in NOAA [83], using  $\{1,2,4,8,16\}$ -shots for training and full test data. We conduct experiments on five fine-grained datasets, including FGVC-Aircraft [51], Food-101 [4], Oxford Flowers [55], Oxford Pets [56], Stanford Cars [15].
Figure 5: Results of few-shot learning on five fine-grained visual recognition datasets. The solid line represents the comparison of parameter-efficient tuning methods, and the dashed line represents the comparison of memory-efficient tuning methods. All results are averaged over 3 random seeds.
Table 5: Results on domain generalization. "Mean" denotes the average accuracy of four variants of ImageNet. All results are averaged over 3 random seeds.
Source
ImageNet	Target
IN-V2	IN-Sketch	IN-A	IN-R	Mean
Parameter-efficient tuning methods
Adapter [25]	70.5	59.1	16.4	5.5	22.1	25.8
VPT [28]	70.5	58.0	18.3	4.6	23.2	26.0
LoRA [27]	70.8	59.3	20.0	6.9	23.3	27.4
NOAH [83]	71.5	66.1	24.8	11.9	28.5	32.8
Res-Tuning	78.04	66.58	29.23	13.15	29.01	34.50
Memory-efficient tuning methods
Side-Tuning [82]	74.57	62.52	23.55	10.37	25.06	30.38
LST [68]	70.00	57.04	14.39	7.21	17.02	23.92
Res-Tuning-Bypass	77.30	65.23	27.39	10.66	26.45	32.43
As the results are shown in Fig. 5, in terms of the overall performance (top-left), both Res-Tuning and Res-Tuning-Bypass demonstrate clear advantages over other corresponding parameter-efficient and memory-efficient tuning strategies of few-shot learning on five FGVC datasets. We also observe that Res-Tuning-Bypass performs as well as or even better than the non-memory-efficient methods on one or two shots with low training samples on several datasets.
Domain Generalization. To evaluate the robustness of our approach to distribution shift, we train a model on the source domain using 16 shots per category and test it on both the source and target domain. The source domain uses ImageNet-1K [11]) and the target domains use four other variants of ImageNet, including ImageNet-V2 [62], ImageNet-Sketch [73], ImageNet-A [24], ImageNet-R [23]. The results in Tab. 5 prove that our approach is robust under domain shift. Our Res-Tuning goes beyond NOAA [83] by  $6.54\%$  on ImageNet and  $1.7\%$  on the average accuracy of four variants of ImageNet. Furthermore, Res-Tuning-Bypass also demonstrates stronger robustness than the memory-efficient baselines and outperforms most existing parameter-efficient tuning methods.
5 Empirical evaluations on generative task
5.1 Experimental setups
Downstream tasks. To provide a more comprehensive evaluation of our Res-Tuning framework, we further apply it to the text-to-image generation task. Following [63], we evaluate the text-to-image
Table 6: Comparison of FID and efficiency on COCO2017. Following the default settings of stable diffusion<sup>1</sup>, we sample 10k captions from the validation set for generating images of size  $512^{2}$  using 50 PLMS steps with classifier-free guidance scale 3.0 and compare against the full validation set.
Method	FID	Param. (M)	Mem. (GB)	Train (Hour/Epoch)
SD v1.5	15.48	-	-	-
+ Full	14.85	862 (100%)	72.77	1.98
+ LoRA	14.50	9.96 (1.15%)	61.03	1.42
+ Adapter	14.73	2.51 (0.29%)	54.30	1.30
+ Prefix	15.36	4.99 (0.58%)	64.91	2.20
+ Prompt	14.90	1.25 (0.14%)	63.70	2.17
+ Res-Tuning	13.96	2.54 (0.29%)	54.49	1.38
+ Res-Tuning Bypass	14.89	3.76 (0.44%)	21.35	0.82
Figure 6: Qualitative results of SD v1.5, existing tuning strategies and our Res-Tuning on COCO2017 validation set [43]. We frame our results in green and others in red.
generation performance on COCO2017 dataset [43]. The main quantitative evaluation metric is the FID score and we also perform instance-level fine-tuning transfer on the fine-grained datasets [4, 55].
Baselines. We experiment with the version 1.5 of stable diffusion [63] (SD) model. On COCO2017, we compare our Res-Tuning framework with zero-shot SD, SD with fully fine-tuning as well as SD with other existing tuning methods. On the fine-grained datasets, we employ DreamBooth [65] as our baseline and employ various tuning methods including our own for comparison.
Text-to-image generation on COCO. We show the comparison between our Res-Tuning framework and other approaches both quantitatively and qualitatively. The quantitative comparison is presented in Tab. 6. Compared with the fully fine-tuned baseline, our Res-Tuning framework improves the FID score by 0.89, using only  $0.29\%$  of the parameters. It is noteworthy that our Res-Tuning framework is the only approach that reaches a FID score below 14, while the best existing tuning strategy on the task is LoRA [27] achieving 14.50 with 4x the number of parameters in Res-Tuning. Hence, employing Res-Tuning could greatly facilitate the adaptation of pre-trained text-to-image generation models to downstream tasks.
A highlight is observed that our Res-Tuning-Bypass framework reduces the memory consumption for tuning the SD v1.5 model to only  $29\%$  while maintaining a similar performance (14.89 vs 14.85). Meanwhile, the time consumption for training the model is reduced to  $41\%$ . In terms of the reduction in the memory and time consumption, our Res-Tuning-Bypass framework outperforms the best tuning strategy by  $3.3\mathrm{x}$  ( $70.7\%$  memory reduction of Res-Tuning-Bypass vs.  $25.3\%$  that of adapter) and  $1.7\mathrm{x}$  ( $58.6\%$  reduction in time consumption of Res-Tuning-Bypass vs.  $34.3\%$  that of adapter), respectively.
The qualitative results are presented in Fig. 6. Both Res-Tuning and Res-Tuning-Bypass show a better alignment between the text and the generated image, where the surfboard is propped up in our generated images. Res-Tuning also demonstrates a better fidelity where the feather texture is realistic in the generated black bird.
<sup>1</sup>https://huggingface.co/runwayml/stable-diffusion-v1-5
Figure 7: Qualitative results of SD v1.5, DreamBooth, existing tuning strategies, and our Res-Tuning on Oxford Flowers and Food-101 fine-grained dataset with the same generated seed. We frame our results in green and others in red.
Text-to-image generation on Oxford Flowers and Food-101. Here, we evaluate the transfer ability of existing tuning strategies on specific fine-grained categories. During every evaluation process, we select data from one specific category to train the model. The qualitative results are demonstrated in Fig. 7. It is observed that Res-Tuning presents a better view in terms of fidelity and the correct understanding of ambiguous categories. For example, the balloon flower is bell-shaped and has five petals, while existing approaches generate flowers with the wrong number of petals or even balloons rather than real flowers. Instead, both our Res-Tuning and our Res-Tuning-Bypass retain correct semantics and fine-grained features.
Transformers in computer vision. Transformers [70] have demonstrated strong capabilities in various fields [5, 61, 12, 59, 2, 17, 48, 53]. In computer vision, Vision Transformers (ViT) [13] are widely applied in various applications, such as visual classification [48, 39], object detection [67, 6], segmentation [84, 74] and generation [63, 57]. Owing to the strong scalability of the Transformer backbone [31, 10], recent endeavors either focus on expanding the size of the ViT [81] or training on a larger corpus of data in an unsupervised manner [75, 14].
Parameter-efficient tuning. Despite the strong performance and generalization ability brought by scaling up the Transformers, it also makes the adaptation to the downstream tasks expensive and almost infeasible. Hence, parameter-efficient transfer learning (PETL) emerged [26, 25, 76, 41, 85, 28]. Existing PETL methods could be generally categorized into three types: (i) MHA-based tuning embeds tunable parameters in the multi-head self-attention layers [27, 76, 28, 41, 45, 46]. (ii) FFN-based tuning methods represented by adapters [25] and its generalized versions [58, 33, 32, 19] introduce a multi-layer perceptron to the FFN layer. (iii) Other tuning methods adapt certain existing parameters [79, 42]. Closely related to our work, some recent efforts are devoted to finding out the optimal design paradigm of the tuning modules by neural architecture search [83]. Instead, we show that through our Res-Tuning framework, a stronger tuning strategy can be easily found by the flexible combination of several existing tuning strategies in our unbinding form.
Memory-efficient tuning. Since the structures of the existing PETL methods are deeply embedded in the backbone structure, back-propagation is required through the massive parameters of the pretrained models, leading to unnecessary extra consumption of memory and computation. Hence, Side-Tuning [82] and LST [68] in natural language processing connect a side network in parallel with the pre-trained model to avoid data flow from the trainable parameters to the frozen ones. Our Res-Tuning-Bypass is inspired by the conceptual idea of these approaches. Compared to Side-Tuning and LST, we show that our Res-Tuning-Bypass is more flexible and memory-efficient.
In this work, we unbind the tuners from the backbone and form a flexible and efficient tuning paradigm Res-Tuning. With Res-Tuning, we are able to find stronger tuning strategies compared to existing ones. On top of Res-Tuning, we further extend a memory-efficient Res-Tuning-Bypass, which significantly reduces the memory consumption and multi-task inference cost. We hope our discoveries can facilitate further research in the flexible and efficient tuning of large foundation models.
In the appendix, we provide a detailed derivation process for turning existing PETL methods into our unbinding formulation (appendix A), demonstration of our unbinding formulation as a unified formulation for existing PETL methods (appendix B), more implementation details (appendix C) including the dataset used, architectures for discriminative and generative tasks, and hyperparameters used in training, additional results on discriminative tasks (appendix D) and generative tasks (appendix E).
In this section, we provide the detailed deriving process for the existing prefix [41] and prompt [28] tuning to be turned into our unbinding formulation.
Prefix tuning [41]: The following is the detailed derivation of Eq. (3).
$$
\begin{array}{l} \mathrm {M H A} _ {\mathrm {p r e}} = \operatorname {A t t n} \left(\boldsymbol {x} \boldsymbol {W} _ {q}, \left[ \boldsymbol {K} _ {p r e}; \boldsymbol {x} \boldsymbol {W} _ {k} \right], \left[ \boldsymbol {V} _ {p r e}; \boldsymbol {x} \boldsymbol {W} _ {v} \right]\right) \\ = \mathrm {s o f t m a x} \left(\boldsymbol {x} \boldsymbol {W} _ {q} [ \boldsymbol {K} _ {p r e}; \boldsymbol {x} \boldsymbol {W} _ {k} ] ^ {\top}\right) \left[ \begin{array}{l} \boldsymbol {V} _ {p r e} \\ \boldsymbol {x} \boldsymbol {W} _ {v} \end{array} \right] \\ = (1 - \lambda (\boldsymbol {x})) \operatorname {s o f t m a x} \left(\boldsymbol {x} \boldsymbol {W} _ {q} \boldsymbol {W} _ {k} ^ {\top} \boldsymbol {x} ^ {\top}\right) \boldsymbol {x} \boldsymbol {W} _ {v} + \lambda (\boldsymbol {x}) \operatorname {s o f t m a x} \left(x \boldsymbol {W} _ {q} \boldsymbol {K} _ {p r e} ^ {\top}\right) \boldsymbol {V} _ {p r e} \tag {9} \\ = (1 - \lambda (\pmb {x})) \mathrm {A t t n} (\pmb {x} \pmb {W} _ {q}, \pmb {x} \pmb {W} _ {k}, \pmb {x} \pmb {W} _ {v}) + \lambda (\pmb {x}) \mathrm {A t t n} (\pmb {x} \pmb {W} _ {q}, \pmb {K} _ {p r e}, \pmb {V} _ {p r e}) \\ = \left(1 - \lambda \left(\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {K} _ {p r e}\right)\right) \underbrace {\operatorname {A t t n} \left(\boldsymbol {Q} , \boldsymbol {K} , \boldsymbol {V}\right)} _ {\text {s t a n d a r d a t t e n t i o n}} + \lambda \left(\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {K} _ {p r e}\right) \underbrace {\operatorname {A t t n} \left(\boldsymbol {Q} , \boldsymbol {K} _ {p r e} , \boldsymbol {V} _ {p r e}\right)} _ {\text {i n d e p e n d e n t o f} \boldsymbol {K} _ {p r e}, \boldsymbol {V} _ {p r e}} \\ \end{array}
$$
where  $Q$ ,  $K$  denote the original query and original key,  $K_{pre}$  and  $V_{pre}$  are prefix key tokens and prefix value tokens, and
$$
\lambda (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {K} _ {p r e}) = \frac {\sum_ {i} \exp \left(\boldsymbol {Q} \boldsymbol {K} _ {p r e} ^ {\top}\right) _ {i}}{\sum_ {i} \exp \left(\boldsymbol {Q} \boldsymbol {K} ^ {\top}\right) _ {i} + \sum_ {j} \exp \left(\boldsymbol {Q} \boldsymbol {K} _ {p r e} ^ {\top}\right) _ {j}}, \tag {10}
$$
Prompt tuning [28]: The following is the detailed derivation of Eq. (5).
$$
\begin{array}{l} \mathrm {M H A} _ {\mathrm {p r o}} = \mathrm {A t t n} ([ \pmb {x}; \pmb {x} _ {p r o} ] \pmb {W} _ {q}, [ \pmb {x}; \pmb {x} _ {p r o} ] \pmb {W} _ {k}, [ \pmb {x}; \pmb {x} _ {p r o} ] \pmb {W} _ {v}) \\ = \mathrm {c o n c a t} \left(\mathrm {s o f t m a x} \left(\boldsymbol {x} \boldsymbol {W} _ {q} [ \boldsymbol {x} \boldsymbol {W} _ {k}; \boldsymbol {x} _ {p r o} \boldsymbol {W} _ {k} ] ^ {\top}\right) \left[ \begin{array}{c} \boldsymbol {x} \boldsymbol {W} _ {v} \\ \boldsymbol {x} _ {p r o} \boldsymbol {W} _ {v} \end{array} \right], \right. \\ \operatorname {s o f t m a x} \left(\pmb {x} _ {p r o} \pmb {W} _ {q} [ \pmb {x} \pmb {W} _ {k}; \pmb {x} _ {p r o} \pmb {W} _ {k} ] ^ {\top}\right)\left[\begin{array}{c}\pmb {x} \pmb {W} _ {v}\\\pmb {x} _ {p r o} \pmb {W} _ {v}\end{array}\right]\left. \right) \\ = \operatorname {c o n c a t} \left(\left(1 - \lambda (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {K} _ {p r o})\right) \operatorname {A t t n} (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {V}) + \lambda (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {K} _ {p r o}) \operatorname {A t t n} (\boldsymbol {Q}, \boldsymbol {K} _ {p r o}, \boldsymbol {V} _ {p r o})\right), \\ \left(1 - \beta \left(\boldsymbol {Q} _ {p r o}, \boldsymbol {K} _ {p r o}, \boldsymbol {K}\right)\right) \operatorname {A t t n} \left(\boldsymbol {Q} _ {p r o}, \boldsymbol {K} _ {p r o}, \boldsymbol {V} _ {p r o}\right) \\ + \beta \left(\boldsymbol {Q} _ {p r o}, \boldsymbol {K} _ {p r o}, \boldsymbol {K}\right) \operatorname {A t t n} \left(\boldsymbol {Q} _ {p r o}, \boldsymbol {K}, \boldsymbol {V}\right)) \tag {11} \\ \end{array}
$$
where  $K_{pro}$  and  $V_{pro}$  are prompt key tokens and prompt value tokens, and
$$
\lambda (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {K} _ {p r o}) = \frac {\sum_ {i} \exp \left(\boldsymbol {Q} \boldsymbol {K} _ {p r o} ^ {\top}\right) _ {i}}{\sum_ {i} \exp \left(\boldsymbol {Q} \boldsymbol {K} ^ {\top}\right) _ {i} + \sum_ {j} \exp \left(\boldsymbol {Q} \boldsymbol {K} _ {p r o} ^ {\top}\right) _ {j}}, \tag {12}
$$
$$
\beta \left(\boldsymbol {Q} _ {p r o}, \boldsymbol {K} _ {p r o}, \boldsymbol {K}\right) = \frac {\sum_ {i} \exp \left(\boldsymbol {Q} _ {p r o} \boldsymbol {K} ^ {\top}\right) _ {i}}{\sum_ {i} \exp \left(\boldsymbol {Q} _ {p r o} \boldsymbol {K} _ {p r o} ^ {\top}\right) _ {i} + \sum_ {j} \exp \left(\boldsymbol {Q} _ {p r o} \boldsymbol {K} ^ {\top}\right) _ {j}}, \tag {13}
$$
B Unified formulation
From Eq. (7), we derive a unified formulation for existing PETL methods, which is the combination of the frozen pre-trained operation (OP) and the tuner with learnable parameters (Res-Tuner). The following is the detailed instantiation of this unified formulation (as in Tab. 7). Thanks to the form of unbinding, tuners can be treated as individual and flexibly combined. Our framework can effectively encompass existing tuning methods in a residual form and is not limited to the OP and Res-Tuner mentioned in our work. For example, for MAM-Adapter, we are free to attach Res-Prefix and Res-Adapter to MHA and FFN modules, respectively. In particular, when new PETL methods are proposed, they can be quickly applied (abbreviated as New-Tuning), and different methods can be arbitrarily combined (abbreviated as Mix-Tuning).
Table 7: The combination of Res-Tuning.
Method	OP	Res-Tuner
Adapter [25]	FFN	Res-Adapter
Prefix [41]	MHA	Res-Prefix
Prompt [28]	MHA	Res-Prompt
LoRA [27]	Weight	MLP
BitFit [79]	Bias	Parameter
AdaptFormer [7]	FFN	Res-Adapter
MAM-Adapter [19]	MHA + FFN	Res-Prefix + Res-Adapter
Side-Tuning [82]	All blocks	Side model
LST [68]	Block	ViT
New-Tuning	Any where	New-Tuner
Mix-Tuning	Any combination	Any combination
C.1 Dataset description
In Tab. 8 and Tab. 9, we list the description of each dataset in our experiments, which includes the number of classes and the amount of images in training set and test set for discriminative and generative tasks, respectively.
Table 8: Datasets used for generative tasks.  $\star$  denotes only part of the data is used.
Dataset	Description	Classes	Train	Test
image	prompt	image	prompt
Common Objects in Context (COCO)
COCO2017 Captions	common objects	-	118287	591753	5000	25014
Fine-grained Image Generation
Aircraft [51]*	Fine-grained aircraft	100	3334	-	3333	-
Food-101 [4]*	Fine-grained food	101	75750	-	25250	-
NABirds [69]*	Fine-grained bird	555	23929	-	24633	-
Stanford Cars [15]*	Fine-grained car	196	8144	-	8144	-
Stanford Dogs [34]*	Fine-grained dog	120	12000	-	8580	-
SUN397 [78]*	Fine-grained scene	397	76044	-	16295	-
Table 9: Datasets used for discriminative tasks.
Dataset	Description	Classes	Train	Test
General Image Classification
CIFAR-100 [35]	General	100	50000	10000
Fine-grained Visual Classification (FGVC)
CUB-200-2011 [72]	Bird	200	5994	5794
NABirds [69]	Bird	555	23929	24633
Oxford Flowers [55]	Flower	102	1020	6149
Stanford Cars [15]	Car	196	8144	8041
Stanford Dogs [34]	Dog	120	12000	8580
Visual Task Adaptation Benchmark (VTAB-1K)
CIFAR-100 [35]	100	10000
Caltech101 [38]	102	6084
DTD [9]	47	1880
Flower102 [55]	Natural	102	1000	6149
Pets [56]	37	3669
SVHN [54]	10	26032
SUN397 [78]	397	21750
Camelyon [71]	2	32768
EuroSAT [22]	10	5400
Resisc45 [8]	45	1000	6300
Retinopathy [18]	5	42670
Clevr-Count [30]	8	15000
Clevr-Dist [30]	6	15000
DMLab [3]	6	22735
KITTI-Dist [16]	4	711
dSpr-Loc [52]	Structured	16	1000	73728
dSpr-Ori [52]	16	73728
sNORB-Azim [36]	18	12150
sNORB-Ele [36]	9	12150
Few-Shot Learning
FGVC-Aircraft [51]	Aircraft	100	(1/2/4/8/16) * class	3333
Food-101 [4]	Food	101	(1/2/4/8/16) * class	30300
Oxford Flowers [55]	Flower	102	(1/2/4/8/16) * class	2463
Oxford Pets [56]	Pet	37	(1/2/4/8/16) * class	3669
Stanford Cars [15]	Car	196	(1/2/4/8/16) * class	8041
Domain Generalization
ImageNet-1K [11]	General	1000	16 * class	50000
ImageNet-V2 [62]	General	1000	/	10000
ImageNet-Sketch [73]	General	1000	/	50889
ImageNet-A [24]	General	200	/	7500
ImageNet-R [23]	General	200	/	30000
C.2 Architectures design
We show the complete Res-Tuning-Bypass structural design according to the different task frameworks. For discriminative tasks, we design based on the architecture of Vision Transformers [13], mainly based on main blocks and unbound bypass (as in Fig. 8). For generative tasks, we design based on the stable diffusion [63] framework and innovatively apply Res-Tuner to U-Net [64] structure (as in Fig. 9). We attach Res-Tuner to the U-Net intermediate module and decoder module for more efficient training.
Figure 8: Architecture design for discriminative tasks.
Figure 9: Architecture design for generative tasks.
We list all the hyperparameters for discriminative (see in Tab. 10) and generative (see in Tab. 11) tasks.
Table 10: Hyperparameter selection for discriminative tasks.
Config	Value
Batch size	32
Optimizer	AdamW [50]
Weight decay	0.05
Base learning rate range	{0.05, 0.01, 0.005, 0.001}
Learning rate schedule	Cosine decay
Training epochs range	{50, 100}
Warmup epochs	10
Augmentation	RandomResizedCrop, RandomHorizontalFlip
OP	MHA [70]
FFN [70]
Block [70]
Res-Tuner	Res-Adapter
Res-Prefix
Res-Prompt
Architecture	ViT/B-16 [13]
ViT/L-14 [13]
Pre-trained	ImageNet-21K [11]
CLIP [59]
Device	A100 × 1
Table 11: Hyperparameter selection for generative tasks.
Config	Value
Batch size	8
Optimizer	AdamW [50]
Base learning rate	1e-4
Learning rate schedule	Constant
Training epochs	10
Augmentation	CenterCrop, RandomHorizontalFlip
Resolution	512 × 512
Sampler	PLMS [44]
Guidance	3.0
OP	MHA [70]
FFN [70]
Block [70]
Conditional 2D U-Net [64]
Res-Tuner	Res-Adapter
Res-Prefix
Res-Prompt
Architecture	Stable Diffusion [63]
Pre-trained	Stable Diffusion v1.5 [63]
CLIP [59]
Device	A100 × 8
Library	Diffusers2
$^{2}$ https://github.com/huggingface/diffusers
D Additional experiments on discriminative tasks
D.1 Comparisons on FGVC datasets