SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare the transfer ability of our Res-Tuning framework with different existing approaches for parameter- and memory-efficient transfer learning on FGVC datasets. As shown in Tab. 12, our Res-Tuning outperforms other tuning methods on the average accuracy of five FGVC datasets. For memory-efficient methods, Res-Tuning-Bypass achieves a  $2.89\%$  improvement compared to LST [68] with less memory consumption.
Table 12: Performance and efficiency comparison on FGVC. † denotes our own implementation.
Datasets Method	CUB-200-2011	NABirds	Oxford Flowers	Stanford Cars	Stanford Dogs	Mean	Param. (M)	Mem. (GB)
Full	87.3	82.7	98.8	84.5	89.4	88.54	85.98	9.40
Linear	85.3	75.9	97.9	51.3	86.2	79.32	0.18	3.09
Parameter-efficient tuning methods
Adapter [25]	87.3	84.3	98.4	68.4	88.8	85.46	1.96	6.55
Prefix†[41]	85.4	78.8	99.2	76.4	89.5	85.86	0.36	6.60
VPT-Shallow [28]	86.7	78.8	98.4	68.7	90.7	84.62	0.25	8.14
VPT-Deep [28]	88.5	84.2	99.0	83.6	90.2	89.11	0.85	8.16
LoRA†[27]	86.0	80.2	99.2	85.2	88.6	87.84	0.55	6.78
Convpass†[29]	86.9	81.4	99.3	85.7	89.9	88.66	0.51	7.44
SSF [42]	89.5	85.7	99.6	89.2	89.6	90.72	0.39	7.47
Res-Tuning	89.66	85.87	99.45	87.58	92.21	90.95	0.68	8.98
Memory-efficient tuning methods
Side-Tuning [82]	84.7	75.8	96.9	48.6	85.8	78.35	9.73	3.48
LST†[68]	82.98	76.11	98.83	78.46	87.89	84.85	1.04	5.28
Res-Tuning-Bypass	88.75	83.00	99.61	75.41	92.40	87.83	0.56	4.73
D.2 More ablation studies
Varying the length of dimensions. The length of dimensions represents the hidden dimension for Res-Adapter and the number of tokens of Res-Prompt and Res-Prefix. Altering the length of dimensions affects performances, the number of parameters, and memory consumption simultaneously (see in Tab. 13). By default, we use the length that balances the number of parameters and memory, although there is a slight increase as the length grows to a certain extent.
Table 13: Ablation studies on the length of dimension for Res-Tuning on CIFAR-100. Default setting is marked in gray.
Dim	Res-Tuning	Res-Tuning-Bypass
Acc.	Param.	Mem.	Acc.	Param.	Mem.
5	93.03	0.30M	6.84G	89.16	0.37M	4.65G
10	93.25	0.48M	6.85G	89.33	0.46M	4.72G
20	92.98	0.85M	6.87G	89.28	0.64M	4.83G
50	92.69	1.96M	6.90G	89.22	1.19M	5.08G
100	92.48	3.80M	6.96G	89.52	2.11M	5.68G
Different number of block layers. Based on the ViT/B-16 structure, we change the number of block layers in two ways: (i) increase from head to toe gradually (see in Tab. 14), (ii) discard several intermediate layers (see in Tab. 15), and evaluate the impact of using a different number of layers. It can be seen that the best performance can be achieved by using the full number of layers, but it requires the use of relatively more parameters and memory.
Table 14: Ablation studies on the tuning block layers for Res-Tuning on CIFAR-100. The right arrow denotes use the number of all layers from left to right. Default setting is marked in gray.
Blocks	Res-Tuning	Res-Tuning-Bypass
Acc.	Param.	Mem.	Acc.	Param.	Mem.
1 → 1	92.23	0.11M	6.43G	88.50	0.11M	3.58M
1 → 3	92.78	0.18M	6.52G	88.60	0.17M	3.73G
1 → 6	92.93	0.28M	6.64G	89.05	0.27M	4.09G
1 → 9	93.17	0.38M	6.84G	89.32	0.36M	4.53G
1 → 12	93.25	0.48M	6.85G	89.33	0.46M	4.72G
Table 15: Ablation studies on the drop block layers for Res-Tuning on CIFAR-100. Inside the braces indicate the layer number that is discarded. The default setting is marked in gray.
Drop block layers	Res-Tuning	Res-Tuning-Bypass
Acc.	Param.	Mem.	Acc.	Param.	Mem.
{}	93.25	0.48M	6.85G	89.33	0.46M	4.72G
{3,6,9}	92.88	0.38M	6.73G	89.30	0.36M	4.36G
{2,3,5,7,9,11}	92.96	0.28M	6.68G	88.89	0.27M	3.96G
{2,3,4,5,6,8,9,10,11}	92.71	0.18M	6.56G	88.79	0.17M	3.65G
{1,2,3,4,5,6,7,8,9,10,11}	87.77	0.11M	3.31G	88.49	0.11M	3.48G
Different CNN backbones. We present the results for ConvNeXt [49] and ResNet-101 [20] pretrained on ImageNet-21K in Tab. 16. We observe that two convolutional models have different characteristics, where the performance variation of ConvNeXt is small and that of ResNet is large. In terms of the effectiveness of Res-Tuning-Bypass, it is observed that it outperforms the fully-finetuned version of ConvNeXt and notably improves the performance of linear probing for ResNet.
Table 16: Ablation studies on the CNN-based backbones on CIFAR-100.
Method	ConvNext	ResNet-101
Acc.	Param.	Mem.	Acc.	Param.	Mem.
Full	90.15	87.67	11.16	77.40	42.70	7.30
Linear	90.06	0.11	3.36	54.96	0.20	2.83
Res-Tuning	90.86	0.87	9.45	86.80	0.92	7.20
Res-Tuning-Bypass	90.51	1.13	3.63	72.27	3.63	4.05
Experiments on NLP downstream task. We also conduct experiments on downstream tasks beyond vision. For NLP, we perform experiments on the SST2 [66] and MNLI [77] datasets for text classification tasks. It is observed that the performance of our Res-Tuning framework is on par with or better than MAM-Adapter [19] in text classification, with slightly longer training time but lower memory consumption. Our Res-Tuning-Bypass significantly reduces the training time and memory consumption and achieves a mildly lower performance.
Table 17: Performance comparison on text classification task.
Method	SST2	MNLI	Train Param.	Test Param.	Mem.
Acc.	Train Time	Acc.	Train Time
MAM Adapter [19]	94.2	7.2	87.4	41.4	46.78 (37.4%)	0.61 (0.5%)	22.4G
Res-Tuning	94.56	7.9	87.45	47.3	0.97 (0.77%)	0.97 (0.77%)	19.3G
Res-Tuning-Bypass	92.94	4.2	82.01	24.2	0.98 (0.78%)	0.98 (0.78%)	4.3G
E Additional Experiments on generative tasks
E.1 More visualizations on COCO
We present more visualization results, comparing real images, stable diffusion [63] (SD) v1.5, fully fine-tuning, LoRA [27], Res-Tuning and Res-Tuning-Bypass in Fig. 10 and more detailed generated image in Fig. 11. Specifically, the text conditions of text-to-image task are sampled from COCO Captions. To better demonstrate the advantages of our approach in understanding the concepts of text, the main elements are marked in blue, and the main modifiers or prepositions are marked in green.
E.2 More visualizations on fine-grained datasets
We present more visualization results on fine-grained datasets (see in Fig. 12, Fig. 13, and Fig. 14), including NABirds [69], Stanford Dogs [34], Stanford Cars [15], Aircraft [51], SUN397 [78] and Food-101 [4].
F Limitations and Societal Impacts
This work is a tuning paradigm that is fine-tuned based on the pre-trained foundation models while freezing the backbone network, so its transfer ability depends to a large extent on the performance of the upstream model. However, when the upstream pre-training model contains illegal content training, it will also lead to the illegal use of tuning methods.
Reale
LoRA
Tunrnnn
Rssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
A teddy bear is sitting in front a group of boxes.
Figure 10: Qualitative results of existing tuning strategies and our Res-Tuning on COCO2017 validation set.
A vase with flowers on a display near a wall.
A landscape with water, a boat, trees and mountains.
Bananas, marshmellows, chocolate chips and sprinkles in a bowl.
The rain is pouring on the white car on the street.
Figure 11: Visualization of our Res-Tuning on COCO2017 validation set.
Stanford Dogs
(a) Res-Tuning
(b) Res-Tuning-Bypass
Figure 12: Visualization of our Res-Tuning on NABirds and Stanford Dogs.
Stanford Cars
(b) Res-Tuning-Bypass
Figure 13: Visualization of Res-Tuning on Stanford Cars and Aircraft.
(b) Res-Tuning-Bypass
Figure 14: Visualization of Res-Tuning on SUN397 and Food-101.
SUN397
Food-101