SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yonatan Dukler Alessandro Achille Hao Yang Benjamin Bowman* Varsha Vivek
Luca Zancato Avinash Ravichandran Charless Fowlkes Ashwin Swaminathan
{dukler, aachille, haoyng, bowmaben, varshviv, zancato, ravinash, fowlkec, swashwin, soattos}@amazon.com
We present a framework for transfer learning that efficiently adapts a large base-model by learning lightweight cross-attention modules attached to its intermediate activations. We name our approach InCA (Introspective-Cross-Attention) and show that it can efficiently survey a network's representations and identify strong performing adapter models for a downstream task. During training, InCA enables training numerous adapters efficiently and in parallel, isolated from the frozen base model. On the ViT-L/16 architecture, our experiments show that a single adapter,  $1.3\%$  of the full model, is able to reach full fine-tuning accuracy on average across 11 challenging downstream classification tasks. Compared with other forms of parameter-efficient adaptation, the isolated nature of the InCA adaptation is computationally desirable for large-scale models. For instance, we adapt ViT-G/14 (1.8B+ parameters) quickly with  $20+$  adapters in parallel on a single V100 GPU (76% GPU memory reduction) and exhaustively identify its most useful representations. We further demonstrate how the adapters learned by InCA can be incrementally modified or combined for flexible learning scenarios and our approach achieves state of the art performance on the ImageNet-to-Sketch multi-task benchmark.
Foundation models promise to achieve top performance with minimal adaptation on any downstream task. In the realm of language, the data and the hypothesis spaces are shared, and many tasks can be unified into a homogeneous representation. Visual inference domains, on the other hand, can be highly heterogeneous and possibly antagonistic. For instance, the hypothesis space for pose estimation is geometric, whereas for scene classification it is semantic and even domains that appear homogeneous, such as image classification into the 10 CIFAR classes, can trigger interference in the trained model in the presence of modest perturbations of the image statistics.
Antagonistic domains may interfere within the activations of the latter layers, which cannot be simultaneously minimal and sufficient for all domains. However, information about a dissimilar domain may be present in earlier layers, and certainly in the input data which is trivially sufficient
*Work conducted while interning at AWS AI Labs.
Figure 1: (Left) Top-1 Test Error for fine-grained classification transfer learning tasks evaluated with the ViT-L/16 architecture. InCA performs comparable to full fine-tuning on each challenging dataset. (Right) Max GPU Memory usage during training for different adaptation approaches and model sizes.
Figure 2: InCA Adaptation In (B), intermediate activation maps are extracted from a pretrained backbone during forward pass. Each activation map is passed to a lightweight InCA adapter (shown in (C)) or Open-InCA adapter (shown in (D)) depending on the task settings. In (A), we illustrate how multiple adapters are trained in parallel and independently, and during inference can be combined or used in parallel. In (C), (D) we present a schema of the InCA and Open-InCA adapters; see Sec. 3 for details.
for any task. Indeed, as opposed to just operating with the final model's representations, the typical approach of addressing domain variability in transfer learning is by applying full fine-tuning of the model on new data. By optimizing all of the model's parameters, each of the model representations can be potentially harnessed as a starting point for the adaptation.
While accurate and robust to a variety of domains, full fine-tuning of large-scale models entails sweeping computational and storage costs. Further, the resultant model can only function for its dedicated task, not allowing for shared computation and parallel execution of potential future tasks. To tackle the problem of efficient, versatile, and modular adaptation we introduce Introspective Cross-Attention (InCA). InCA operates on the base-model by attaching isolated shallow adapter modules utilizing any of the activation maps for downstream tasks.
Since modern architectures are deep and so is the variety of possible downstream tasks, iterating over all the possible candidate activations of a model is prohibitively expensive. Instead in InCA we search for useful model representations exhaustively and in parallel by training numerous isolated adapters attached to different activations. When using InCA, each adapter is light and the base-model is fixed and does not require any backpropagation. This makes InCA computationally efficient in terms of GPU memory which is crucial for scaling to large models. Further, the shallow InCA adapter networks simplify the training dynamics as compared with existing approaches which speeds up training considerably and makes the optimization straightforward and robust (see Appendix C).
In detail, during parallel training of InCA, a set of adapters sharing the same architecture are trained simultaneously and independently on a downstream task. Each adapter accepts an assigned activation and does not feed back to the backbone, leaving the backbone execution unaltered during both training and inference (Fig. 2). At inference, the learned adapters may be combined or a best performing adapter can be selected for downstream prediction. The InCA adapter architecture is simple, consisting of a single cross-attention module followed by a linear classifier for the downstream task. Despite the simplicity of the adapter, we observe that a single top-performing adapter trained for a downstream
task is capable of achieving strong performance across a variety of different architectures and pretrainings when tested on a diverse set of visual domains. Because our approach does not modify the execution of the model or any of the pre-trained layers as done in existing parameter-efficient approaches [28, 34, 27], our method can be automatically applied to any architecture without the hassle of re-implementation and architecture specific modifications. In particular, we present results of InCA for ViT [18], SWIN [48], and CNN architectures [49, 74] on a suite of fine-grained recognition tasks.
Since the adapters learned in InCA are shallow and small (1.3% of the parameters on ViT-L/16), the strong performance of a single adapter implies that many pre-trained models already contain strong representations for a diverse set of downstream tasks. Instead, previous approaches like linear probing fall short in using the representations, not because the task can not be solved using the existing model, but rather because of not having the right "extraction capacity" of cross-attention; we explore this systematically in Sec. 4 and present a theoretical proof for the advantage of the cross-attention layer in Appendix D. For challenging datasets, using intermediate representations as opposed to only adapting the last layer's representation is key to making InCA a versatile method that closes the gap with full fine-tuning on diverse tasks (See Fig. 1).
A byproduct of the exhaustive approach of InCA adaptation is a signature describing the performance of the internal representations of a network on different downstream tasks. This renders InCA as a powerful tool for understanding the underlying representations of pre-trained models and task space [1] (See Sec. 5, Appendix B). Curiously, we observe that in certain downstream datasets, different pre-trained models hold similar performance signatures, even when the pre-trained models use different architectures or pre-training augmentations.
The isolated training of the adapters means no backpropagation through the pre-trained model is taking place. This significantly reduces the correlation between adaptation cost and the pre-trained model size, which makes it possible to leverage very large architectures even with modest compute resources. Fig. 1 shows that one V100 GPU can train InCA with 40 adapters using an architecture as big as ViT-G/14. In contrast, existing parameter efficient approaches that backpropagate through the architecture exhaust GPU memory with any model larger than ViT-B/16.
Our contributions are summarized as
- We introduce InCA, a method that enables versatile downstream adaptation by surveying the internal network representations of any frozen model with lightweight cross-attention based modules as an alternative to full fine-tuning. On ViT-L/16, InCA matches full fine-tuning performance and reaches within  $0.7\%$  accuracy for a SWIN-L backbone on average over 11 diverse downstream domains.
- We demonstrate how the modular adapter architecture of InCA enables flexible learning and inference scenarios and present the Open-InCA adapter. With our approach we unlock powerful pre-trained models for reusable and parallel multi-task inference, and class-incremental learning.
- On the efficiency front, InCA scales to massive scale architectures under typical computation budgets while its implementation can be automatically applied to any new model. For example, training ViT-G/14 using InCA results in  $76\%$  GPU memory reduction as compared with full fine-tuning. Further, InCA is easy to optimize and highly parameter efficient (See Appendix C).
The rest of the paper is organized as follows: In Sec. 2 we review related work, and in Sec. 3 we present our approach. We empirically evaluate InCA on a wide set of visual recognition tasks in Sec. 4. Lastly we provide analysis of intermediate representation signatures (Sec. 5) followed by discussion (Sec. 6). Additional results and analysis including Open-InCA are presented in the Appendix.
Transfer learning Transfer learning in deep neural networks aims at endowing existing models with new "downstream" knowledge. The de-facto approach for transfer learning in deep learning modifies an existing pre-trained network by applying full, partial or linear fine-tuning on the model [37, 66, 45]. Depending on the the variability between the source and downstream task domains,
different approaches aim at delineating different transfer domains [1, 58, 76], extending coverage of transfer via suitable pre-trainings such as meta-learning [42, 19, 30], or by selecting and combining from a set of expert models [15, 21]. More broadly the field of representation learning focuses on learning transferable representations [38] via different pre-training strategies that can be applied for downstream tasks [11].
Efficient adaptation methods In recent times, the top-performing pre-trained model architectures are becoming considerably larger [61, 49], and we arrive at a crossroad as full fine-tuning of such large models is becoming out of reach in many practical settings. Recent works address the storage costs associated with large model transfer by proposing parameter efficient transfer approaches as opposed to storing all of the fine-tuned model parameters [24]. These approaches achieve parameter efficiency by training a subset of existing parameters [6, 45], inserting new weight parameters into existing modules [28, 27, 39, 16] or via additional learnable activations or inputs, known as prompts [34, 65, 43]. Compared with existing work [79, 75], InCA is also parameter efficient, yet we place special emphasis on compute and optimization efficiency, especially in the large-scale model setting. Additional lines of work study learning via selective tuning, enabling multi-domain transfer [70, 22].
Feature extraction with attention Self and cross-attention mechanisms aggregate information from a set of feature tokens [69, 41, 44] and create representations based on relative inter-feature importance as computed by the attention mechanism [12, 69]. Self-attention plays a key in the transformer architecture [69] enabling non-local information aggregation. In settings where the number of inputs and desired outputs differ, cross-attention enables flexible aggregation based on a pair of query and key feature sets. When using cross-attention, one can cross-attend between different sets of activations [44, 17] or between a set of activations and learnable latent parameters [32, 9, 80]. In our settings, the adapter architecture applies cross-attention on extracted representations from a pretrained network, inspired by the cross-attention module of Perceiver [32]. However, we train multiple adapters in parallel and avoid the iterative re-sampling architecture present in their work. More generally, cross-attention layers have been vital in many existing object detection and multi-modal systems that fuse activations [44, 8], or apply cross-attention with learnable latents [3, 32, 77, 9, 80].
Learning with intermediate features The re-use of intermediate representations in deep learning is vast and spans from works on interpretability [78], to state of the art approaches in object detection that harness intermediate layers for multi-resolution feature pyramids [47, 68, 20] and segmentation [23, 29]. For ConvNets, the work of [2] studies classification utilizing intermediate network representations and the authors observe a decrease in accuracy when probing earlier layers.
We introduce InCA, a lightweight and modular transfer learning alternative to full fine-tuning, that avoids backpropagation through the base-model. Let  $f(x) = g_{n} \circ g_{n-1} \circ \ldots \circ g_{1}(x)$  be a pretrained feed-forward neural network of  $n$  layers, with  $g_{j}(\cdot)$  corresponding to the  $j$ -th layer of the network. We denote the activation computed by  $g_{j}$  as  $f_{j}(x) = g_{j} \circ g_{j-1} \circ \ldots \circ g_{1}(x)$ . During network inference, a "forward" computation processes and computes each  $f_{j}(x)$  activation to arrive to the network's final prediction  $f(x)$ . During standard training, all of the intermediate activations  $\{f_{1}(x), \ldots, f_{n-1}(x), f_{n}(x)\}$  are held in GPU memory and are used to compute gradients to update the model. For large models, this incurs large computational and GPU memory costs [63] which limits using the best and largest available pre-trained models under typical computation budgets.
Instead, we attach a set of isolated "models" to the pre-trained model  $f$  at selected activations  $f_{j_k}$  and pass them as input to a set of lightweight and shallow networks  $h_k(a)$  with separate parameters and losses. With this, we can train a set of heterogeneous adapters  $h_k(a)$  in parallel, while computing inference of the pre-trained model  $f$  only once during each update (see Fig. 2). For a set of adapters  $h_k(a)$  that take as input intermediate activations from  $\{f_{j_k}\}$  training follows as:
1. Single inference of  $f$  through a data batch  $x$  which computes  $f(x)$  and selected activations  $\{f_{j_k}(x)\}$ .
We use a callback (though Torch's register_forward hook() or TensorFlow's get_layer().output) to cache the activations of the relevant layers which become leafs of the computational graph
2. Calculate the batch predictions and losses for each adapter  $h_k$ ,  $\ell_k = \ell(h_k(f_{j_k}(x)), y)$ .
3. Computing  $\ell_{\Sigma} = \sum \ell_{k}$  and applying automatic differentiation then efficiently resolves the gradient and updates of each  $h_k$  automatically as desired.
By avoiding backpropagation through the pre-trained  $f$  we decouple the majority of the training costs from depending on the size of the base model  $f$  and instead the costs correlate with the much smaller adapter set  $\{h_k\}$ . Below we demonstrate that even a simple cross-attention module for  $h_k$  makes the overall adaptation sufficiently expressive yet highly efficient.
InCA adapter After extraction of the layer representation  $f_{k}(x)$ , we have access to a high-dimensional activation map at our disposal. To predict a target label  $\hat{y}$  from the high-dimensional  $f_{k}(x)$ , the typical approach is to apply dimension reduction such as averaging (avgpool) or computing maximum values over a subset of the dimensions and then applying a linear classification head:
$$
\hat {y} = \operatorname {h e a d} \circ \operatorname {a v g - p o o l} \circ f _ {m} (x).
$$
Nonetheless, this simple aggregation approach leads to loss of information which we observe empirically in Sec. 4 and theoretically analyze in Appendix D. Instead, we use a cross-attention module to intelligently aggregate information from the entire large-dimensional activation map  $f_{k}(x)$  into a fixed-dimensional representation based on a number of cross-attention queries. Specifically, for standard downstream adaptation, given an intermediate feature map  $\mathbf{z} = [z^{1},\dots,z^{T}] = f_{k}(x)$  with  $T$  tokens or channels we use the following adapter architecture
$$
\begin{array}{l} v _ {\text {c r o s s}} (\mathbf {z}) _ {[ 1: m ]} := \operatorname {c r o s s - a t t n} _ {\theta} ([ z ^ {1}, \dots , z ^ {T} ], [ q _ {1}, \dots , q _ {m} ]) \\ \operatorname {I n C A} _ {\theta} (\mathbf {z}) := \operatorname {h e a d} _ {\theta} \circ \operatorname {n o r m} \left(\operatorname {a v g - p o o l} \left(v _ {\text {c r o s s}} (\mathbf {z}) _ {[ 1: m ]}\right)\right). \\ \end{array}
$$
Note that the query tokens  $[q_1, \ldots, q_m]$  are optimized along with  $\theta$ . The multi-head cross-attention layer outputs  $v_{\mathrm{cross}}$  is produced by surveying the feature map  $f_k(x)$  with the query tokens  $[q_1, \ldots, q_m]$ . Then, the classification output  $\hat{y} = \operatorname{InCA}_{\theta}(\mathbf{z})$  is obtained through averaging the cross-attention outputs (if  $m > 1$ ) followed by a fully-connected classification head after normalizing with LayerNorm [4]. Based on our experiments, using a single query token  $q$  ( $m = 1$ ) achieves strong performance and is computationally efficient and we report results with  $m = 1$  unless otherwise stated.
For more flexible inference such as in the settings of continual and class-incremental learning tasks, we present a modular version of InCA that disentangles the representations learned between different classes, which we refer to as "Open-InCA". For a  $c$ -way classification task, define separate queries  $[q_1,\dots q_c]$  for each class to compute representations separately,
$$
\begin{array}{l} \left[ v _ {\text {c r o s s}} ^ {1} (\mathbf {z}), \dots , v _ {\text {c r o s s}} ^ {c} (\mathbf {z}) \right] := \operatorname {c r o s s - a t t n} _ {\theta} \left(\left[ z ^ {1}, \dots , z ^ {T} \right], \left[ q _ {1}, \dots , q _ {c} \right]\right) \\ \operatorname {O p e n - I n C A} _ {\theta} (\mathbf {z}) := \operatorname {d i a g - h e a d} _ {\theta} \circ \operatorname {n o r m} ([ v _ {\text {c r o s s}} ^ {1} (\mathbf {z}), \dots , v _ {\text {c r o s s}} ^ {c} (\mathbf {z})) ] \\ \end{array}
$$
Above,  $\mathrm{diag - head}_{\theta}$  is a linear operator layer that operates on a matrix input  $[a_1,\dots ,a_c]$  "diagonally". Given a weight parameter  $W$  , the operator is defined as the column-wise dot product,
$$
\operatorname {d i a g - h e a d} _ {\theta} ([ a _ {1}, \dots a _ {c} ]) = [ \langle W _ {1}, a _ {1} \rangle , \dots \langle W _ {c}, a _ {c} \rangle ].
$$
Open-InCA composition In the Open-InCA adapter architecture, unique queries  $[q_1,\dots ,q_c]$  are defined for each class along with diag-head  $\theta$  that independently processes each coordinate prediction. Both diag-head, LayerNorm and the cross-attn module in Open-InCA operate on each input  $q_{i}$  independently which separates the representation learned for each class and enables isolating each adapter output coordinate as
$$
\mathrm {O p e n - I n C A} (\mathbf {z}) _ {i} = \langle W _ {i}, \mathrm {n o r m} (\mathrm {c r o s s - a t t n} _ {\theta} ([ z ^ {1}, \dots , z ^ {T} ], [ q _ {i} ])) \rangle .
$$
Above,  $W_{i}$  corresponds to the  $i$ -th column of diag-head weight. As a result Open-InCA enables class-level modularity with the capabilities of new class insertion, deletion and isolated class updates without regression. For example, deleting class  $i$  from the Open-InCA architecture amounts to simply dropping the query and head parameters  $q_{i}$  and  $W_{i}$  for that coordinate. In the setting of class-incremental learning (CIL) different query-head pairs from Open-InCA can be combined together, as long as the parameters of the norm and cross-attn remain the same. In practice, this leads to the notion of training Open-InCA with fixed norm and cross-attention weight parameters, in what we refer to as "query-only-training". In query-only-training, the learning of a new class
corresponds to learning just 2,  $d$  dimensional parameters per-class and adapter, where  $d$  is the token dimension. Nonetheless, when using pre-trained Open-InCA layer parameters, "query-only-training" performs within the accuracy of InCA on many datasets. In Appendix A we compare results of InCA, Open-InCA and query-only-training in class-incremental learning (CIL). In Tab. 6 of the Appendix we observe that even learning just the query and head parameters is capable of harnessing the large dimensional representation maps  $f_{k}(x)$
Layer branching candidate selection The cross-attention adapters can be applied in parallel over any intermediate layer of the network and we observe that the performance of many tasks hinges on identifying the right intermediate layer to use for that task. When considering intermediate activations  $f_{k}(x)$ , we observe that
- Using activations such that  $f_{j}(x)$  is directly computed from a residual connection yields better adapter accuracy. This reflects that network representations are refined through each residual block.
- The middle and later layers of the network provide stronger input representations for the adapter. This is likely since the representations of the early layers do not have discriminative enough features to be used directly for high-level tasks.
Two-Stage training In settings where the base-model forward-propagation during InCA training is too constraining, one may conduct training in two stages. In the first stage, save the activations that serve the input for the adapter for the entire training set, by running the base-model inference for a single epoch. After saving, the second stage proceeds by training the adapters for  $T$  epochs with loaded activations. Suppose the per-epoch cost of the pre-trained model forward-propagation is  $C_{PT}$  and the per-epoch cost of adapter optimization is  $C_A$ , then two-stage training reduces the time of training from  $O((C_{PT} + C_A) \times T)$  to  $O(C_A \times T + C_{PT})$ , where  $C_{PT} \gg C_A$ . With two-stage training, we are able to reduce a 30-epoch adapter training job to 30 seconds for a cached Stanf. Cars dataset ( $\sim 8,000$  training samples). We speculate that further optimization can reduce training costs to "real-time", enabling an array of user-interactive applications.
Datasets In our experiments, we measure the capabilities of InCA on a diverse set of 11 fine-grained datasets consisting of: CUB-200 [73], Aircrafts, [54], Stanford Cars [40], Stanford Dogs [35], Oxford Flowers 102 [56], MIT-67 [60], Oxford Pets [59], Describable Textures (DTD) [13], European Flood [5], FGVC Herbarium [57], and EuroSAT Land Use dataset [26]. In Table 4 we explore InCA in the settings of multi-task learning and evaluate it on the ImageNet-to-Sketch benchmark that is comprised of 5 datasets: WikiArt [64], Oxford Flowers [59], Sketch [71], Stanford Cars [40], and CUB-200 [73].