SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zeyinzi Jiang<sup>1</sup> Chaojie Mao<sup>1</sup> Ziyuan Huang<sup>2</sup> Ao Ma<sup>1</sup> Yiliang Lv<sup>1</sup> Yujun Shen<sup>3</sup> Deli Zhao<sup>1</sup> Jingren Zhou<sup>1</sup>
\(^{1}\)Alibaba Group \(^{2}\)National University of Singapore \(^{3}\)Ant Group \(\{\text{zeyinzi.jzyz, chaojie.mcj, dave.ma, yiliang.lyl, jingren.zhou}\} @alibaba-inc.com \(\{\text{ziyuan.huang}\} @u.nus.edu \{\text{shenyujun0302, zhaodeli}\} @gmail.com\)
Parameter-efficient tuning has become a trend in transferring large-scale foundation models to downstream applications. Existing methods typically embed some lightweight tuners into the backbone, where both the design and the learning of the tuners are highly dependent on the base model. This work offers a new tuning paradigm, dubbed Res-Tuning, which intentionally unbinds tuners from the backbone. With both theoretical and empirical evidence, we show that popular tuning approaches have their equivalent counterparts under our unbinding formulation, and hence can be integrated into our framework effortlessly. Thanks to the structural disentanglement, we manage to free the design of tuners from the network architecture, facilitating flexible combination of various tuning strategies. We further propose a memory-efficient variant of Res-Tuning, where the bypass (i.e., formed by a sequence of tuners) is effectively detached from the main branch, such that the gradients are back-propagated only to the tuners but not to the backbone. Such a detachment also allows one-time backbone forward for multi-task inference. Extensive experiments on both discriminative and generative tasks demonstrate the superiority of our method over existing alternatives from the perspectives of efficacy and efficiency. Project page: https://res-tuning.github.io/.
Recently, foundation models have demonstrated strong generalization capability across numerous visual [21, 2], language [47, 60] and multi-modal tasks [40, 1]. Pre-trained on a large corpus of data, a foundation model offers a good initialization for downstream adaptation. Unfortunately, the increasing model scale makes it expensive and almost infeasible to fully fine-tune such a model for every task. Hence, parameter-efficient transfer learning (PETL) [37, 41, 27] is often resorted to as an efficient approach for downstream adaptation without incurring an unaffordable computation burden.
Popular existing approaches for parameter-efficient tuning introduce additional tunable structures (which we term as tuners) to the pre-trained base model [37, 41, 27]. Compared to the fully-fine-tuned counterparts, the light-weight tuners significantly reduce the training cost while maintaining a competitive performance [32, 28]. However, the current designs of tuners are deeply coupled with their base structures, as shown in Fig. 1a, thus restricting the design flexibility and impeding the extension to new approaches. For example, prefix tuning [41] is embedded into the self-attention operation, and prompts [28, 85] could only be introduced at the beginning or between layers, etc.
In this work, we introduce Res-Tuning, a new tuning paradigm for flexible and efficient transfer learning. As in Fig. 1b, our Res-Tuning framework unbinds the tuners from the base model, such that it is possible to decouple both the design and the learning of the tuners from the base structure. Since
Figure 1: Concept comparison between existing methods and our Res-Tuning framework. (a) Existing PETL methods are deeply embedded into original structures. (b) Our Res-Tuning framework can decouple PETL methods from the backbone. (c) Backpropagation only through a bypass consisting of Res-Tuner can be achieved by detaching the connections.
the design of the tuners is no longer dependent on the base structure in our Res-Tuning framework, we explore various possibilities. With both theoretical and empirical evidence, we first show that our framework can seamlessly encompass popular tuning approaches such as prefix-tuning [41], prompt-tuning [28], and adapters [25]. Further, it is demonstrated that the structural disentanglement also allows for flexible combination of various tuners, leading to the discovery of stronger tuning strategies. On top of that, we show that such an unbinding formulation also allows for the detachment of the tuners from the backbone as in Fig. 1c, which further improves the memory efficiency. In this memory-efficient version, i.e., Res-Tuning-Bypass, not only is the training cost reduced because the gradient computation through the massive parameters is avoided in the base model, but it also reduces the number of forward passes of the backbone model to only once during multi-task inference.
We evaluate Res-Tuning framework on both discriminative and generative tasks. On discriminative tasks, we show that our unbinding formulation leads to a tuning strategy that achieves state-of-the-art performance on VTAB-1K with similar learnable parameters. Our Res-Tuning-Bypass framework also performs favorably against the fully-fine-tuned variant, while reducing the training memory consumption by  $49.7\%$  and the multi-task inference time by  $80.9\%$ . In addition, it also obtains better performance in few-shot learning and domain generalization scenarios. On generative tasks, apart from the strong performance achieved by Res-Tuning framework in terms of both FID scores and qualitative results, we further show that our Res-Tuning-Bypass can reduce the memory consumption by  $70.7\%$ , training time by  $58.6\%$ , and multi-task inference time by  $83.6\%$  when maintaining a competitive FID score and generation quality compared to the fully-fine-tuned variant.
2 Unbinding parameter-efficient tuning
In order to reduce the entanglement between the base model and the tuners as well as to increase the flexibility of the tuning strategies, we set out to unbind the tuners from the pre-trained structures. In this section, we provide our unbinding formulation to existing parameter-efficient transfer learning strategies. We start by revisiting the basic building blocks of the foundation models, before diving into unbinding existing tuners from the backbone. In the last part of this section, we further provide empirical proof that our unbinding formulation could seamlessly encompass existing PETL methods like prompt tuning [28], prefix tuning [41], and adapters [25].
2.1 Basic building blocks of foundation models
Existing foundation models in both natural language processing, vision, and vision-language applications mostly adopt Transformers [70] as the backbone. The major building blocks of the Transformers that one usually adapts for the downstream tasks are the multi-head attention (MHA) and the feed-forward network (FFN). Formally, the standard MHA and FFN could be expressed as:
$$
\operatorname {A t t n} (\boldsymbol {Q}, \boldsymbol {K}, \boldsymbol {V}) = \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} \boldsymbol {K} ^ {\mathrm {T}}}{\sqrt {d}}\right) \boldsymbol {V}, \tag {1}
$$
$$
\operatorname {F F N} (\boldsymbol {x}) = \phi (\boldsymbol {x} \boldsymbol {W} _ {1} + \boldsymbol {b} _ {1}) \boldsymbol {W} _ {2} + \boldsymbol {b} _ {2}
$$
(a) Prefix Tuning
(b) Prompt Tuning
Figure 2: The original and the unbinding form of (a) prefix tuning [41], (b) prompt tuning [28], and (c) adapter tuning [25] for parameter-efficient transfer learning.
(c) Adapter Tuning
where  $Q$ ,  $K$  and  $V$  denote the query, key and value, respectively.  $W_{1}$  and  $W_{2}$  are the projection weights,  $b_{1}$  and  $b_{2}$  are the bias terms, and  $\phi$  is the non-linear activation between fully-connected layers. Usually, given input tokens  $x$ , the query, key, and value are obtained through a linear projection  $Q = xW_{q}$ ,  $K = xW_{k}$ , and  $V = xW_{v}$ , where  $W_{q}$ ,  $W_{k}$  and  $W_{v}$  are learnable projection weights.
2.2 Unbinding tuners from foundation models
For the adaptation of the foundation models to downstream applications, the existing PETL approaches mostly resort to adjusting the output of MHA, FFN, or the Transformer block composed of MHA and FFN in various ways. We choose popular and exemplary approaches and unbind their structures from foundation models. Here, we provide the unbinding formulations of prefix tuning [41] and prompt tuning [28] for MHA adaptation, as well as adapter tuning [25] for FFN adaptation.
Prefix tuning [41] prepends learnable parameters, i.e., prefix tokens, to the projected keys and values:
$$
\mathrm {M H A} _ {\text {p r e}} = \operatorname {A t t n} \left(\boldsymbol {x} \boldsymbol {W} _ {q}, \left[ \boldsymbol {K} _ {p r e}; \boldsymbol {x} \boldsymbol {W} _ {k} \right], \left[ \boldsymbol {V} _ {p r e}; \boldsymbol {x} \boldsymbol {W} _ {v} \right]\right), \tag {2}
$$
where  $K_{pre}$  and  $V_{pre}$  are prefix tokens. Essentially, if we view this as performing MHA separately between  $(Q, K, V)$  and between  $(Q, K_{pre}, V_{pre})$ , we unbind prefix tuning as follows:
$$
\mathrm {M H A} _ {\text {p r e}} = (1 - \lambda) \underbrace {\operatorname {A t t n} (\boldsymbol {Q} , \boldsymbol {K} , \boldsymbol {V})} _ {\text {o r i g i n a l a t t e n t i o n}} + \lambda \underbrace {\operatorname {A t t n} (\boldsymbol {Q} , \boldsymbol {K} _ {\text {p r e}} , \boldsymbol {V} _ {\text {p r e}})} _ {\text {p r e f i x a t t e n t i o n i n p a r a l l e l}}, \tag {3}
$$
where  $\lambda$  weighs between the original and prefix attention. Detailed value for  $\lambda$  and the derivation process are included in appendix A. In this way, the original MHA in the foundation model  $\mathrm{Attn}(Q,K,V)$  and the prefix attention  $\mathrm{Attn}(Q,K_{pre},V_{pre})$  can be computed independently. The unbinding formulation of prefix tuning can be seen in Fig. 2a.
Prompt tuning [28] appends latent tokens to the input token before performing MHA in the backbone:
$$
\mathrm {M H A} _ {\text {p r o}} = \operatorname {A t t n} \left(\left[ \boldsymbol {x}; \boldsymbol {x} _ {\text {p r o}} \right] \boldsymbol {W} _ {q}, \left[ \boldsymbol {x}; \boldsymbol {x} _ {\text {p r o}} \right] \boldsymbol {W} _ {k}, \left[ \boldsymbol {x}; \boldsymbol {x} _ {\text {p r o}} \right] \boldsymbol {W} _ {v}\right), \tag {4}
$$
where  $\pmb{x}_{pro}$  are prompt tokens concatenated to the input token  $\pmb{x}$  in the first layer or between multiple layers. Similar to prefix tuning, the unbinding formulation of prompt tuning is as follows:
$$
\mathrm {M H A} _ {\text {p r o}} = \left[ (1 - \lambda) \underbrace {\operatorname {A t t n} \left(\boldsymbol {Q} , \boldsymbol {K} , \boldsymbol {V}\right)} _ {\text {o r i g i n a l a t t e n t i o n}} + \lambda \underbrace {\operatorname {A t t n} \left(\boldsymbol {Q} , \boldsymbol {K} _ {\text {p r o}} , \boldsymbol {V} _ {\text {p r o}}\right)} _ {\text {p r o m p t a t t e n t i o n i n p a r a l l e l}}; \boldsymbol {D} \right], \tag {5}
$$
where  $K_{pro} = x_{pro}W_{k}$  and  $V_{pro} = x_{pro}W_{v}$ .  $D$  denotes disposable parts that would not affect the output of MHA, where  $D = (1 - \beta)\operatorname{Attn}(Q_{pro},K_{pro},V_{pro}) + \beta\operatorname{Attn}(Q_{pro},K,V)$ .  $\lambda$  and  $\beta$  are individual weights. More details can be seen in appendix A. The unbinding formulation of prompt tuning can be seen in Fig. 2b.
Adapter tuning [25] typically inserts a multi-layer perceptron (MLP) after FFN. Since the MLP could be performed independently, we simply re-route the adapter and connect it in parallel to the FFN as in Fig. 2c. The resultant unbinding formulation of the adapters is as follows:
$$
\mathrm {F F N} _ {\text {a d a p t e r}} = \underbrace {\operatorname {F F N} (\boldsymbol {x})} _ {\text {o r i g i n a l m o d u l e}} + \underbrace {\phi \left(\operatorname {F F N} (\boldsymbol {x}) \boldsymbol {W} _ {\text {d o w n}}\right) \boldsymbol {W} _ {\text {u p}}} _ {\text {a d a p t e r m o d u l e i n p a r a l l e}}, \tag {6}
$$
where  $W_{down}$  and  $W_{up}$  denote the weights for the down- and up-projection layers, respectively.
Figure 3: Structure illustration of (a) various Res-Tuners in our unbinding form, (b) our Res-Tuning framework that is flexible and efficient, and (c) Res-Tuning-Bypass, the memory-efficient version of our framework.
Table 1: Empirical equivalence of PETL methods and their counterparts in our unbinding form.
Method	ViT/B-16 (IN-21K)	ViT/L-14 (CLIP)
Original	Unbinding form	Δ	Original	Unbinding form	Δ
Adapter [25]	92.34	92.34	0.00	92.43	92.44	+0.01
Prefix [41]	91.90	91.88	-0.02	90.99	91.01	+0.02
Prompt [28]	92.21	92.24	+0.03	91.86	91.83	-0.03
2.3 Empirical equivalency of the unbinding formulation
After we have obtained the unbinding formulation of popular PETL methods with theoretical derivation, Tab. 1 shows empirical evidence of the equivalency between the original formulation and our unbinding formulation. We carry out the comparison between two formulations on CIFAR-100, using ViT [13] pre-trained on ImageNet-21K and by CLIP [59]. For all cases, we observe that the performance discrepancy between the original and our unbinding form is within a reasonable range  $(\pm 0.03)$ , which shows that our formulation encompasses existing PETL methods effortlessly.
3 Res-Tuning
3.1 Unified formulation of Res-Tuning
Given the unbinding formulation of the existing PETL methods, we can derive a unified formulation as the combination of the frozen pre-trained operation and the tuner with learnable parameters:
$$
\boldsymbol {x} ^ {\prime} = \mathrm {O P} (\boldsymbol {x}) + \operatorname {R e s - T u n e r} (\boldsymbol {x}), \tag {7}
$$
where OP denotes existing operations in the pre-trained backbone such as MHA and FFN, while the Res-Tuner represents the learnable structures that are connected in parallel to the existing operations.
With this unified unbinding formulation, the design of the tuner structure can now be independent of the original operation in the base model. This leads to unprecedented flexibility in parameter-efficient tuning, which enables us to explore various instantiations of the tuner (as in Fig. 3a) for the base structures. For example, we found in Tab. 2a that, instantiating the Res-Tuner with prompt tuning for FFN results in a better performance compared to adapters.
Further, the structural disentanglement also allows for the flexible combination of various tuning strategies. In Fig. 3b, we instantiate our Res-Tuning framework by associating one Res-Tuner with every operation in the base model, including MHA, FFN, and the whole Transformer block.
3.2 Res-Tuning-Bypass: Towards memory-efficient PETL
The unified formulation of Res-Tuning provides a viable solution for flexible and parameter-efficient transfer learning. However, since it is directly derived from existing PETL methods, it shares the
same vulnerability as existing PETL solutions. Despite the parameter efficiency, they require backpropagation through the massive parameters in the pre-trained backbone, which leads to unnecessary extra consumption of memory and computation.
To avoid this, we further present a memory-efficient version of our Res-Tuning framework, dubbed as Res-Tuning-Bypass, as in Fig. 3c. Specifically, we remove the data flow from the Res-Tuner to the backbone, such that the Res-Tuner is detached from the pre-trained architectures. In this way, we form a bypass network constructed by Res-Tuners in parallel with the backbone. Formally, given the tokenized feature  $\boldsymbol{x}_0$  and the output feature of the  $l$ -th layer  $\boldsymbol{x}_l$  from the pre-trained model, our bypass network is formulated as:
$$
\boldsymbol {x} _ {0} ^ {\text {b y p a s s}} = \boldsymbol {x} _ {0}, \tag {8}
$$
$$
\boldsymbol {x} _ {l} ^ {\text {b y p a s s}} = \lambda \operatorname {R e s} - \operatorname {T u n e r} (\boldsymbol {x} _ {l}) + (1 - \lambda) \operatorname {R e s} - \operatorname {T u n e r} (\boldsymbol {x} _ {l - 1} ^ {\text {b y p a s s}}), l \geq 1,
$$
where  $\lambda$  is a learnable parameter followed by a sigmoid function, which is initialized to 0.5. As demonstrated in Fig. 3c, we group the Res-Tuners in the bypass network into horizontal ones and vertical ones, respectively processing the output feature of the  $l$ -th layer from the backbone and the  $(l - 1)$ -th feature from the bypass network. Within each group, we keep the structure identical. Thanks to the flexibility of our unbinding formulation, we can also explore different instantiations of the Res-Tuner and various combinations of the existing tuners.
4 Empirical evaluations on discriminative tasks
Evaluation scenarios. We mainly analyze the flexibility and efficiency of our proposed Res-Tuning framework and evaluate the discriminative capabilities on three different scenarios: transfer learning, few-shot learning, and domain generalization.
Baselines. Apart from the traditional downstream adaptation methods like fully fine-tuning and linear probing, we divide existing tuning approaches into two categories: (i) methods focusing on parameter-efficiency, including adapter tuning [25], prefix tuning [41], VPT [28], LoRA [27], AdaptFormer [7], SSF [42], and NOAA[83]; (ii) methods focusing on memory-efficiency, including Side-Tuning [82], and LST [68]. Since these two categories have distinct characteristics with respect to the parameters, memory, and performance, we mainly compare our Res-Tuning framework within the former category, while the Res-Tuning-Bypass is mainly compared in the latter one.
Implementation details. For most experiments, we adopt ViT-B/16 [13] pre-trained on ImageNet-21K [11] as the backbone model, following VPT [28]. Unless otherwise specified, the middle of the adapter, as well as the number of prefix and prompt tokens in our Res-Tuning are set to 10 for parameter efficiency. We include the training details in appendix C. For all the tasks, we use top-1 accuracy as the main evaluation metric.
4.2 Analysis on flexibility
Flexible combination between backbone structures and tuners. Since our unbinding formulation allows for the structural disentanglement between the frozen structure in the backbone and the tuner with learnable parameters, it enables us to explore various combinations of the operation and the tuners. Here, we experiment with various instantiations of OP and Res-Tuner in our Res-Tuning framework, and the number of tuners is limited to one tuner per block (Single-Res-Tuner). The results are presented in Tab. 2a. We found that the default combination in existing approaches (prefix and prompt for MHA adaptation and adapter for FFN adaptation) is far from optimal, and connecting prompt tuning to FFN results in the best performance.
Flexible combination of multiple tuning strategies. Next, we show that the flexibility brought by our unbinding formulation could also effortlessly lead to stronger tuning strategies by combining various tuners in our unbinding formulation. Here, we consider two tuners per block (Dual-Res-Tuner), respectively connected to MHA and FFN. As in Tab. 2b, employing two tuners for each block brings notable improvements over the Single-Res-Tuner variants, with employing two adapters respectively for MHA and FFN achieving the strongest performance of 93.25. On top of the best performing Dual-Res-Tuner model, we further attach tuners at the block level in Tab. 2c. With
Table 2: Exploration of various combinations of operations in the pre-trained backbone and various Res-Tuners achieves a stronger performance compared to the existing tuning strategies on CIFAR-100. Adapter, prefix, and prompt are abbeviated as Ada., Pre. and Pro., respectively.
(a) Single-Res-Tuner.	(b) Dual-Res-Tuner.	(c) Tri-Res-Tuner.
Tuner\OP	MHA	FFN	Block	MHA\FFN	Res-Ada.	Res-Pre.	Res-Pro.	Block	Dual-Res-Tuner
Res-Ada.	92.46	92.34	92.49	Res-Ada.	93.25	92.95	92.62	Res-Ada.	93.28
Res-Pre.	91.88	92.33	92.39	Res-Pre.	93.22	92.38	92.87	Res-Pre.	93.20
Res-Pro.	92.24	92.68	92.16	Res-Pro.	93.03	92.92	92.91	Res-Pro.	93.16
Table 3: In-depth analysis of our Res-Tuning and Res-Tuning-Bypass in terms of performance, parameter-efficiency and memory efficiency on CIFAR-100.
(a) Parameter efficiency of Res-Tuning on CIFAR-100.
Method	Full	Linear Probing	Single-	Dual-	Tri-
Acc.	89.12	85.95	92.68	93.25	93.28
Param.	85.9M	0.07M	0.17M	0.48M	0.67M
(b) Parameter and memory efficiency of Res-Tuning-Bypass on CIFAR-100.
Method	Bypass	Res-Tuner	Acc.	Param.	Mem.
Linear probing	X	-	85.95	0.07M	2.72G
✓	None	86.34	0.07M	3.48G
↓	✓	Hori.	88.08	0.27M	3.66G
✓	Vert.	87.26	0.27M	3.64G
Res-Tuning-Bypass	✓	Both	89.33	0.46M	4.72G
Fully fine-tuning	X	-	89.12	85.9M	9.02G
(c) Performance and efficiency comparison with existing tuning strategies on CIFAR-100.
Method	Acc.	Param. (M)	Mem.
Full	89.12	85.9 (100%)	9.02G
Linear	85.95	0.07 (0.08%)	2.72G
Parameter-efficient tuning methods
MAM-Adapter†[19]	91.70	10.08 (11.72%)	9.57G
AdaptFormer [7]	91.86	1.26 (1.46%)	6.32G
Res-Tuning	93.25	0.48 (0.55%)	6.85G
Memory-efficient tuning methods
Side-Tuning [82]	87.16	9.62 (11.18%)	3.48G
LST†[68]	88.72	0.93 (1.08%)	5.26G
Res-Tuning-Bypass	89.33	0.46 (0.53%)	4.72G
$\dagger$  denotes our own implementation since the original approach is presented for natural language processing.
adapters on top of the Dual-Res-Tuner model, we observe further slight improvements on the classification performance. Compared to existing tuning strategies of underlining in Tab. 2a, without bells and whistles, the Tri-Res-Tuner version of our Res-Tuning framework achieves at least  $0.94\%$  performance improvement.
4.3 Analysis on parameter-, memory- and multi-task inference-efficiency
Parameter-efficiency. We analyze the parameter efficiency of our Res-Tuning framework in Tab. 3a. Our Single-Res-Tuner version surpasses the performance of the fully-fine-tuned variant with less than  $0.2\%$  learnable parameters. Combining all three tuners on MHA, FFN, and block-level, we manage to outperform the fully-fine-tuned and linear evaluated variants by respectively  $4.16\%$  and  $7.33\%$ , while only using 0.67M parameters.
Memory-efficiency. Here, we present the evolution from linear probing to our Res-Tuning-Bypass in Tab. 3b. It is observed that introducing a bypass network without any tuners can help ensemble the original features obtained from different layers, thereby mildly improving the classification accuracy as compared to the linear probing approach where only the classifiers are trained. On top of that, both horizontal and vertical Res-Tuner bring notable performance improvement with limited parameter and memory overhead. With both horizontal and vertical Res-Tuners in place, our Res-Tuning-Bypass framework achieves stronger performance with only  $52\%$  of the memory consumption when compared with the fully-fine-tuned variant.
Multi-task inference-efficiency. In Fig. 4, our Res-Tuning-Bypass demonstrates superior multi-task inference-efficiency on both discriminative and generative tasks. For discriminative multi-task inference, we combine the validation set of 19 tasks in VTAB-1K, perform 19 tasks on every image, and obtain the overall process time for the whole validation set. For generation multi-task inference, we take one image and provide the model with 10 fixed-length prompts for generation and record the overall generation time for the 10 prompts. For existing parameter-efficient methods, the inference time grows linearly when the number of tasks grows. Compared to the fully fine-tuned variant, all existing parameter-efficient tuning strategies increase the inference time to various extents. In contrast, our Res-Tuning-Bypass framework significantly reduces the inference time on 19 discriminative tasks and 10 generative tasks by respectively  $80.9\%$  and  $83.6\%$  when compared with the fully-fine-tuned variant.
Figure 4: Comparisons of the parameter-, memory-, and multi-task inference-efficiency. For multi-task inference-efficiency, we evaluate Res-Tuning-Bypass on both discriminative and generative tasks. For parameter- and memory-efficiency, here, we show the comparisons on VTAB-1K between our approach and existing tuning strategies.