Title: Your representations are in the network: composable and parallel adaptation for large scale models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Efficient learning

Compute-efficient deep learning

Deep Learning Theory

class-incremental-learning

downstream adaptation
Abstract: We present a framework for transfer learning that efficiently adapts a large base-model by learning lightweight cross-attention modules attached to its intermediate activations.
We name our approach InCA (Introspective-Cross-Attention) and show that it can efficiently survey a network’s representations and identify strong performing adapter models for a downstream task.
During training, InCA enables training numerous adapters efficiently and in parallel, isolated from the frozen base model. On the ViT-L/16 architecture, our experiments show that a single adapter, 1.3% of the full model, is able to reach full fine-tuning accuracy on average across 11 challenging downstream classification tasks.
Compared with other forms of parameter-efficient adaptation, the isolated nature of the InCA adaptation is computationally desirable for large-scale models. For instance, we adapt ViT-G/14 (1.8B+ parameters) quickly with 20+ adapters in parallel on a single V100 GPU (76% GPU memory reduction) and exhaustively identify its most useful representations.
We further demonstrate how the adapters learned by InCA can be incrementally modified or combined for flexible learning scenarios and our approach achieves state of the art performance on the ImageNet-to-Sketch multi-task benchmark.

Your representations are in the network: composable and parallel adaptation for large scale models

Abstract

We present a framework for transfer learning that efficiently adapts a large basemodel by learning lightweight cross-attention modules attached to its intermediate activations. We name our approach InCA (Introspective-Cross-Attention) and show that it can efficiently survey a network's representations and identify strong performing adapter models for a downstream task. During training, InCA enables training numerous adapters efficiently and in parallel, isolated from the frozen base model. On the ViT-L/16 architecture, our experiments show that a single adapter, 1.3% of the full model, is able to reach full fine-tuning accuracy on average across 11 challenging downstream classification tasks. Compared with other forms of parameter-efficient adaptation, the isolated nature of the InCA adaptation is computationally desirable for large-scale models. For instance, we adapt ViT-G/14 (1.8B+ parameters) quickly with 20+ adapters in parallel on a single V100 GPU (76% GPU memory reduction) and exhaustively identify its most useful representations. We further demonstrate how the adapters learned by InCA can be incrementally modified or combined for flexible learning scenarios and our approach achieves state of the art performance on the ImageNet-to-Sketch multi-task benchmark.

1 Introduction

Foundation models promise to achieve top performance with minimal adaptation on any downstream task. In the realm of language, the data and the hypothesis spaces are shared, and many tasks can be unified into a homogeneous representation. Visual inference domains, on the other hand, can be highly heterogeneous and possibly antagonistic. For instance, the hypothesis space for pose estimation is geometric, whereas for scene classification it is semantic and even domains that appear homogeneous, such as image classification into the 10 CIFAR classes, can trigger interference in the trained model in the presence of modest perturbations of the image statistics.

Antagonistic domains may interfere within the activations of the latter layers, which cannot be simultaneously minimal and sufficient for all domains. However, information about a dissimilar domain may be present in earlier layers, and certainly in the input data which is trivially sufficient

Aircrafts

5.0

10.0

15.0

20.0

25.0

Top-1 Test Error

Herbarium

5.0

10.0

15.0

20.0

25.0

Stanf. Cars

4.0

6.0

8.0

10.0

12.0

Full FT

InCA

InCA (last)

Prompt Tuning (VPT)

ViT-B/16

ViT-L/16

ViT-H/14

ViT-G/14 ViT-Gig/14

10GB

20GB

30GB

40GB

50GB

60GB

1×V100 GPU

InCA (1 adapter)

InCA (20 adapters)

Full fine-tuning

Prompt Tuning (VPT)

Z

x

1

x

2

x

n

…

Input

Cross Attention

Layer Activations

Queries

…

v

1

v

2

v

k

…

Avg. Pool

y

z

1

z

n

…

v

1

v

2

v

k

…

FC

y

= (

y

1

, …,

y

C

)

FC

FC

InCA

Open-InCA

Layer Activations

InCA

loss n

q

k

q

2

q

1

Cross Attention

q

k

q

2

q

1

Queries

…

z

2

z

1

z

n

…

z

2

Layer 1

x

n

Layer M

…

z

1

z

2

z

n

…

…

InCA

loss n-1

InCA

loss 1

…

(B)

(A)

(C)

(D)

for any task. Indeed, as opposed to just operating with the final model's representations, the typical approach of addressing domain variability in transfer learning is by applying full fine-tuning of the model on new data. By optimizing all of the model's parameters, each of the model representations can be potentially harnessed as a starting point for the adaptation.

While accurate and robust to a variety of domains, full fine-tuning of large-scale models entails sweeping computational and storage costs. Further, the resultant model can only function for its dedicated task, not allowing for shared computation and parallel execution of potential future tasks. To tackle the problem of efficient, versatile, and modular adaptation we introduce Introspective Cross-Attention (InCA). InCA operates on the base-model by attaching isolated shallow adapter modules utilizing any of the activation maps for downstream tasks.

Since modern architectures are deep and so is the variety of possible downstream tasks, iterating over all the possible candidate activations of a model is prohibitively expensive. Instead in InCA we search for useful model representations exhaustively and in parallel by training numerous isolated adapters attached to different activations. When using InCA, each adapter is light and the base-model is fixed and does not require any backpropagation. This makes InCA computationally efficient in terms of GPU memory which is crucial for scaling to large models. Further, the shallow InCA adapter networks simplify the training dynamics as compared with existing approaches which speeds up training considerably and makes the optimization straightforward and robust (see Appendix C).

In detail, during parallel training of InCA, a set of adapters sharing the same architecture are trained simultaneously and independently on a downstream task. Each adapter accepts an assigned activation and does not feed back to the backbone, leaving the backbone execution unaltered during both training and inference (Fig. 2). At inference, the learned adapters may be combined or a best performing adapter can be selected for downstream prediction. The InCA adapter architecture is simple, consisting of a single cross-attention module followed by a linear classifier for the downstream task. Despite the simplicity of the adapter, we observe that a single top-performing adapter trained for a downstream

task is capable of achieving strong performance across a variety of different architectures and pretrainings when tested on a diverse set of visual domains. Because our approach does not modify the execution of the model or any of the pre-trained layers as done in existing parameter-efficient approaches [28, 34, 27], our method can be automatically applied to any architecture without the hassle of re-implementation and architecture specific modifications. In particular, we present results of InCA for ViT [18], SWIN [48], and CNN architectures [49, 74] on a suite of fine-grained recognition tasks.

Since the adapters learned in InCA are shallow and small (1.3% of the parameters on ViT-L/16), the strong performance of a single adapter implies that many pre-trained models already contain strong representations for a diverse set of downstream tasks. Instead, previous approaches like linear probing fall short in using the representations, not because the task can not be solved using the existing model, but rather because of not having the right 'extraction capacity' of cross-attention; we explore this systematically in Sec. 4 and present a theoretical proof for the advantage of the cross-attention layer in Appendix D. For challenging datasets, using intermediate representations as opposed to only adapting the last layer's representation is key to making InCA a versatile method that closes the gap with full fine-tuning on diverse tasks (See Fig. 1).

A byproduct of the exhaustive approach of InCA adaptation is a signature describing the performance of the internal representations of a network on different downstream tasks. This renders InCA as a powerful tool for understanding the underlying representations of pre-trained models and task space [1] (See Sec. 5, Appendix B). Curiously, we observe that in certain downstream datasets, different pre-trained models hold similar performance signatures, even when the pre-trained models use different architectures or pre-training augmentations.

The isolated training of the adapters means no backpropagation through the pre-trained model is taking place. This significantly reduces the correlation between adaptation cost and the pre-trained model size, which makes it possible to leverage very large architectures even with modest compute resources. Fig. 1 shows that one V100 GPU can train InCA with 40 adapters using an architecture as big as ViT-G/14. In contrast, existing parameter efficient approaches that backpropagate through the architecture exhaust GPU memory with any model larger than ViT-B/16.

Our contributions are summarized as

We introduce InCA, a method that enables versatile downstream adaptation by surveying the internal network representations of any frozen model with lightweight cross-attention based modules as an alternative to full fine-tuning. On ViT-L/16, InCA matches full fine-tuning performance and reaches within 0 . 7% accuracy for a SWIN-L backbone on average over 11 diverse downstream domains.

We demonstrate how the modular adapter architecture of InCA enables flexible learning and inference scenarios and present the Open-InCA adapter. With our approach we unlock powerful pre-trained models for reusable and parallel multi-task inference, and class-incremental learning.

On the efficiency front, InCA scales to massive scale architectures under typical computation budgets while its implementation can be automatically applied to any new model. For example, training ViT-G/14 using InCA results in 76% GPU memory reduction as compared with full fine-tuning. Further, InCA is easy to optimize and highly parameter efficient (See Appendix C).

The rest of the paper is organized as follows: In Sec. 2 we review related work, and in Sec. 3 we present our approach. We empirically evaluate InCA on a wide set of visual recognition tasks in Sec. 4. Lastly we provide analysis of intermediate representation signatures (Sec. 5) followed by discussion (Sec. 6). Additional results and analysis including Open-InCA are presented in the Appendix.

2 Related works

Transfer learning Transfer learning in deep neural networks aims at endowing existing models with new 'downstream' knowledge. The de-facto approach for transfer learning in deep learning modifies an existing pre-trained network by applying full, partial or linear fine-tuning on the model [37, 66, 45]. Depending on the the variability between the source and downstream task domains,

different approaches aim at delineating different transfer domains [1, 58, 76], extending coverage of transfer via suitable pre-trainings such as meta-learning [42, 19, 30], or by selecting and combining from a set of expert models [15, 21]. More broadly the field of representation learning focuses on learning transferable representations [38] via different pre-training strategies that can be applied for downstream tasks [11].

Efficient adaptation methods In recent times, the top-performing pre-trained model architectures are becoming considerably larger [61, 49], and we arrive at a crossroad as full fine-tuning of such large models is becoming out of reach in many practical settings. Recent works address the storage costs associated with large model transfer by proposing parameter efficient transfer approaches as opposed to storing all of the fine-tuned model parameters [24]. These approaches achieve parameter efficiency by training a subset of existing parameters [6, 45], inserting new weight parameters into existing modules [28, 27, 39, 16] or via additional learnable activations or inputs, known as prompts [34, 65, 43]. Compared with existing work [79, 75], InCA is also parameter efficient, yet we place special emphasis on compute and optimization efficiency, especially in the large-scale model setting. Additional lines of work study learning via selective tuning, enabling multi-domain transfer [70, 22].

Feature extraction with attention Self and cross-attention mechanisms aggregate information from a set of feature tokens [69, 41, 44] and create representations based on relative inter-feature importance as computed by the attention mechanism [12, 69]. Self-attention plays a key in the transformer architecture [69] enabling non-local information aggregation. In settings where the number of inputs and desired outputs differ, cross-attention enables flexible aggregation based on a pair of query and key feature sets. When using cross-attention, one can cross-attend between different sets of activations [44, 17] or between a set of activations and learnable latent parameters [32, 9, 80]. In our settings, the adapter architecture applies cross-attention on extracted representations from a pretrained network, inspired by the cross-attention module of Perceiver [32]. However, we train multiple adapters in parallel and avoid the iterative re-sampling architecture present in their work. More generally, cross-attention layers have been vital in many existing object detection and multi-modal systems that fuse activations [44, 8], or apply cross-attention with learnable latents [3, 32, 77, 9, 80].

Learning with intermediate features The re-use of intermediate representations in deep learning is vast and spans from works on interpretability [78], to state of the art approaches in object detection that harness intermediate layers for multi-resolution feature pyramids [47, 68, 20] and segmentation [23, 29]. For ConvNets, the work of [2] studies classification utilizing intermediate network representations and the authors observe a decrease in accuracy when probing earlier layers.

3 Method

We introduce InCA, a lightweight and modular transfer learning alternative to full fine-tuning, that avoids backpropagation through the base-model. Let f ( x ) = g n ◦ g n -1 ◦ . . . g 1 ( x ) be a pretrained feed-forward neural network of n layers, with g j ( · ) corresponding to the j -th layer of the network. We denote the activation computed by g j as f j ( x ) = g j ◦ g j -1 ◦ . . . g 1 ( x ) . During network inference, a 'forward' computation processes and computes each f j ( x ) activation to arrive to the network's final prediction f ( x ) . During standard training, all of the intermediate activations { f 1 ( x ) , . . . f n -1 ( x ) , f n ( x ) } are held in GPU memory and are used to compute gradients to update the model. For large models, this incurs large computational and GPU memory costs [63] which limits using the best and largest available pre-trained models under typical computation budgets.

Instead, we attach a set of isolated 'models' to the pre-trained model f at selected activations f j k and pass them as input to a set of lightweight and shallow networks h k ( a ) with separate parameters and losses. With this, we can train a set of heterogeneous adapters h k ( a ) in parallel, while computing inference of the pre-trained model f only once during each update (see Fig. 2). For a set of adapters h k ( a ) that take as input intermediate activations from { f j k } training follows as:

Single inference of f through a data batch x which computes f ( x ) and selected activations { f j k ( x ) } . †

Calculate the batch predictions and losses for each adapter h k , glyph[lscript] k = glyph[lscript] ( h k ( f j k ( x )) , y ) .

Computing glyph[lscript] Σ = ∑ glyph[lscript] k and applying automatic differentiation then efficiently resolves the gradient and updates of each h k automatically as desired.

By avoiding backpropagation through the pre-trained f we decouple the majority of the training costs from depending on the size of the base model f and instead the costs correlate with the much smaller adapter set { h k } . Below we demonstrate that even a simple cross-attention module for h k makes the overall adaptation sufficiently expressive yet highly efficient.

InCA adapter After extraction of the layer representation f k ( x ) , we have access to a highdimensional activation map at our disposal. To predict a target label ˆ y from the high-dimensional f k ( x ) , the typical approach is to apply dimension reduction such as averaging (avgpool) or computing maximum values over a subset of the dimensions and then applying a linear classification head:

Nonetheless, this simple aggregation approach leads to loss of information which we observe empirically in Sec. 4 and theoretically analyze in Appendix D. Instead, we use a cross-attention module to intelligently aggregate information from the entire large-dimensional activation map f k ( x ) into a fixed-dimensional representation based on a number of cross-attention queries. Specifically, for standard downstream adaptation, given an intermediate feature map z = [ z 1 , . . . , z T ] = f k ( x ) with T tokens or channels we use the following adapter architecture

Note that the query tokens [ q 1 , . . . q m ] are optimized along with θ . The multi-head cross-attention layer outputs v cross is produced by surveying the feature map f k ( x ) with the query tokens [ q 1 , . . . q m ] . Then, the classification output ˆ y = InCA θ ( z ) is obtained through averaging the cross-attention outputs (if m> 1 ) followed by a fully-connected classification head after normalizing with LayerNorm [4]. Based on our experiments, using a single query token q ( m = 1 ) achieves strong performance and is computationally efficient and we report results with m = 1 unless otherwise stated.

For more flexible inference such as in the settings of continual and class-incremental learning tasks, we present a modular version of InCA that disentangles the representations learned between different classes, which we refer to as 'Open-InCA'. For a c -way classification task, define separate queries [ q 1 , . . . q c ] for each class to compute representations separately,

Above, diag-head θ is a linear operator layer that operates on a matrix input [ a 1 , . . . , a c ] 'diagonally'. Given a weight parameter W , the operator is defined as the column-wise dot product,

Open-InCA composition In the Open-InCA adapter architecture, unique queries [ q 1 , . . . , q c ] are defined for each class along with diag-head θ that independently processes each coordinate prediction. Both diag-head , LayerNorm and the cross-attn module in Open-InCA operate on each input q i independently which separates the representation learned for each class and enables isolating each adapter output coordinate as

Above, W i corresponds to the i -th column of diag-head weight. As a result Open-InCA enables class-level modularity with the capabilities of new class insertion, deletion and isolated class updates without regression. For example, deleting class i from the Open-InCA architecture amounts to simply dropping the query and head parameters q i and W i for that coordinate. In the setting of class-incremental learning (CIL) different query-head pairs from Open-InCA can be combined together, as long as the parameters of the norm and cross-attn remain the same. In practice, this leads to the notion of training Open-InCA with fixed norm and cross-attention weight parameters, in what we refer to as ' query-only-training '. In query-only-training, the learning of a new class

corresponds to learning just 2, d dimensional parameters per-class and adapter, where d is the token dimension. Nonetheless, when using pre-trained Open-InCA layer parameters, 'query-only-training' performs within the accuracy of InCA on many datasets. In Appendix A we compare results of InCA , Open-InCA and query-only-training in class-incremental learning (CIL). In Tab. 6 of the Appendix we observe that even learning just the query and head parameters is capable of harnessing the large dimensional representation maps f k ( x ) .

Layer branching candidate selection The cross-attention adapters can be applied in parallel over any intermediate layer of the network and we observe that the performance of many tasks hinges on identifying the right intermediate layer to use for that task. When considering intermediate activations f k ( x ) , we observe that

Using activations such that f j ( x ) is directly computed from a residual connection yields better adapter accuracy. This reflects that network representations are refined through each residual block.

The middle and later layers of the network provide stronger input representations for the adapter. This is likely since the representations of the early layers do not have discriminative enough features to be used directly for high-level tasks.

Two-Stage training In settings where the base-model forward-propagation during InCA training is too constraining, one may conduct training in two stages. In the first stage, save the activations that serve the input for the adapter for the entire training set, by running the base-model inference for a single epoch. After saving, the second stage proceeds by training the adapters for T epochs with loaded activations. Suppose the per-epoch cost of the pre-trained model forward-propagation is C PT and the per-epoch cost of adapter optimization is C A , then two-stage training reduces the time of training from O (( C PT + C A ) × T ) to O ( C A × T + C PT ) , where C PT glyph[greatermuch] C A . With two-stage training, we are able to reduce a 30-epoch adapter training job to 30 seconds for a cached Stanf. Cars dataset ( ∼ 8,000 training samples). We speculate that further optimization can reduce training costs to 'real-time', enabling an array of user-interactive applications.

4 Experiments

Datasets In our experiments, we measure the capabilities of InCA on a diverse set of 11 fine-grained datasets consisting of: CUB-200 [73], Aircrafts, [54], Stanford Cars [40], Stanford Dogs [35], Oxford Flowers 102 [56], MIT-67 [60], Oxford Pets [59], Describable Textures (DTD) [13], European Flood [5], FGVC Herbarium [57], and EuroSAT Land Use dataset [26]. In Table 4 we explore InCA in the settings of multi-task learning and evaluate it on the ImageNet-to-Sketch benchmark that is comprised of 5 datasets: WikiArt [64], Oxford Flowers [59], Sketch [71], Stanford Cars [40], and CUB-200 [73].

Baselines For downstream transfer experiments, we compare InCA adaptation to other adaptation protocols. 1) paragon : Full fine-tuning is considered as the paragon as it performs well on a diverse set of datasets but incurs steep computational and parameter costs. 2) We compare InCA to other parameter efficient approaches, including 2a) LoRA [28] 2b) Visual Prompt Tuning (VPT) [34] where we apply the top-performing VPT approach, VPT-Deep . 2c) BitFit [6] and 2d) AdaLN [45] which is the LayerNorm analogous, AdaBN approach. Note, each approach we name in 2) requires backpropagating through the entire network's activations to update the learnable parameters and leads to a large computational overhead compared with InCA. 3) In addition, we also compare InCA with a suite of computationally efficient approaches that avoid backbone backpropagation like InCA. These include 3a) Linear Probing (LP), 3b) Intermediate Linear Probing (In. LP) which utilizes the same training procedure as InCA but with a LP classifier on the activations. 3c) MLP-3, which is a feed-forward network that consists of probing the base-model with a 3-layer feed-forward network, and 3d) Intermediate MLP-3 (In. MLP-3), the extension of MLP-3 to intermediate layers.

Training details In all of our results (including multi-task settings) we use the same training configuration for InCA. We only change the adapter architecture input layer to automatically match the dimension of the base-model activation map. InCA is robust to hyper-parameters and our training schedule is consistent for all runs. This amounts to 30 training epochs, AdamW optimizer [52] with 2 learning rates, and cosine annealing learning schedule; we provide the full training details

in Appendix F. While InCA is efficient enough to operate at larger resolutions, we use 224 image resolution unless stated otherwise. Nonetheless, InCA performance improves at 384 resolution while remaining computationally competitive (see Table 2).

Transfer learning on ViT In Table 1, we demonstrate the transfer performance of InCA applied to ViT-L/16. For each dataset we train InCA and extract activations at residual layers of the ViT for the last 12 blocks and output layer. For all baselines and our method we use the ViT DeiT pre-training [67] and additionally report ViT-L/16 pre-training results in Appendix Table 8. In the table, we compare InCA to full fine-tuning as well as applying InCA on the last layer and observe that only InCA is capable of achieving good results on challenging datasets such as Aircraft, Stanf. Cars, etc. and closes the maximal gap to full fine-tuning to -2.3%. The second best adaptation approach is LoRA which achieves a maximum gap of -4.4% to full fine-tuning, yet at additional training costs. For a single dataset we can train the InCA modules with 2 learning rates in parallel which corresponds to 26 InCA modules with identical architectures attached to 13 activation maps. In this case, the total training costs of InCA on a single dataset correspond to one base-model run. In Appendix C, we report the hyper-parameter settings and training cost of InCA and current state of the art adaptation method for transformers, VPT [34] which incurs up to 8 . 7 × the training costs of InCA with a large hyper-parameter search ( 2 vs. 24 settings).

Transfer learning on SWIN In Table 3 we present downstream adaptation results for the SWIN-L pre-trained model. InCA adaptation is applied to the 3rd and 4th stages of the network residual activations. Because of the heterogeneous activation dimensions of the hierarchical SWIN architecture,

the reported adaptation model sizes depend on the activation map used for the selected adapter. InCA achieves the smallest maximum gap to full fine-tuning on SWIN while being computationally efficient. As with ViT-L, in SWIN we observe that challenging datasets require using intermediate activation maps for InCA, closing the maximal gap from (-9.5%) to (-3.6%).

Evaluating InCA on different pre-trained models InCA can be applied to any feed-forward neural network without any implementation changes. We simply specify the intermediate layer names and feature tensor-ordering for any new architecture and InCA can be used directly. We note this is in sharp contrast to methods that rely on specific layers such as convolution filters [7, 55] or self-attention [28, 34, 43]. We illustrate the architecture versatility of our method in Table 2. We report the mean and maximum test error gap from full fine-tuning on the 11 fine-grained dataset suite as studied in Table 1. We test different architecture families, which include vanilla vision transformers i.e. ViTs, SWIN [48], and modern convolutional networks (ConvNext [49], ResNext [74]). In addition, we test models pre-trained via different strategies including supervised learning [18, 67] and vision-language objectives [44, 61, 31]. We also test InCA at different ViT scales from ViT/B-16 (86M) to ViT/G-14 (1.8B). For InCA adaptation, all model sizes were trained on a single V100 GPU with batch size 32, including for the larger input resolutions runs.

Multi-task Experiments InCA's isolated design is suitable for multi-task inference and a single pre-trained-model can efficiently evaluate a batch of samples on multiple tasks, allowing for 'one-tomany' inference. We compare InCA on the ImageNet-to-Sketch multi-task benchmark in Table 4. All methods except BA 2 were trained with a ViT-L/16 model and evaluated with the ImageNet-to-Sketch version of each dataset [55]. For BA 2 [7], we report the adaptation on a ResNet-50 [25] backbone, as the BA 2 approach requires convolutional filters. Overall, InCA is the top performing method reaching near the paragon on the evaluated datasets. Importantly for multi-task, only InCA and LP enable multi-task inference via 'computation sharing' of the base model inference.

Learning efficiency Isolating the learning from the base-model means InCA learns shallow neural networks directly on a downstream task. By avoiding deeply backpropagated gradients through the base model, the adapters receive direct signal which improves the optimization dynamics and speed of training. We compare the number of training steps required to train InCA and VPT-Deep and observe that InCA can be optimized in 4.5 × fewer epochs than VPT. Here we don't take into account the additional GPU memory costs of optimizing VPT in each step, nor the required hyper-parameter sweeps used in VPT. More detailed efficiency comparison results are given in Appendix C.

5 Analysis

We analyze the results of InCA adaptation, focusing on the performance signature of different intermediate representations used as input for the adapter and the relation between the top InCA layers with fine-tuning. Further in Appendix D, we provide a theoretical proof motivating the extraction capabilities of cross-attention as it is used in InCA.

Intermediate representations We consider the intermediate representation signature created by evaluating the accuracy of adapters that utilize different layers. In Figure 3, we review the adapter performance applied to different layer representations. Datasets like CUB-200 and Flood-Depth mostly prefer final representations, whereas for datasets like Aircrafts and Stanf. Cars, the best adaptations use earlier representations with decreasing performance towards the last activations. Curiously, we observe consistency in layer affinity for certain datasets while using different pretrainings for the backbone and even when using different architectures (Appendix Fig. 5).

InCA and partial-tuning In Appendix B we compare InCA with gradually un-freezing the basemodel and applying partial fine-tuning on a growing set of layers. We run a set of experiments where we fine-tune a pre-trained model starting at different freezing points, this means we optimize all layers of the network after the freezing point location. For each dataset we construct a 'partial tuning curve' where we plot the final test accuracy vs. freezing point (Figure 4). Interestingly, we observe a direct correlation between the layer-location of the top InCA adapter and the point where the partial tuning curve saturates. In particular, the partial tuning test accuracy plateaus (to the tuning of more

Res. Block 11

Res. Block 12

Res. Block 13

Res. Block 14

Res. Block 15

Res. Block 16

Res. Block 17

Res. Block 18

Res. Block 19

Res. Block 20

Res. Block 21

Res. Block 22

Res. Block 23

Output

150

100

50

0

50

100

150

Relative test error improvement (%)

Aircrafts

Stanf. Cars

CUB-200

Block 9

Block 10

Block 11

Block 12

Block 13

Block 14

Block 15

Block 16

Block 17

Block 18

Block 19

Block 20

Block 21

Block 22

Block 23

Output

Freeze point

80.0

82.5

85.0

87.5

90.0

92.5

95.0

97.5

100.0

Relative Accuracy (%)

DTD

CUB-200

Herbarium

Aircrafts

MIT-67

layers) at around the same layer location as the top performing InCA adapter layer location. Namely, the point of saturation of the partial tuning curve is where partial-tuning is capable of harnessing the representation found by InCA at that layer. This gives further evidence that 'your representations are in the network' and that fine-tuning surfaces existing representations that can be directly identified by InCA. However, InCA adaptation operates an order of magnitude more efficiently and scales better to large models.

6 Discussion

In this paper, we present an efficient and effective alternative to full fine-tuning for transfer learning, closing the gap to full fine-tuning on a diverse set of downstream datasets. InCA has many benefits: it inherently generalizes to different architectures, efficiently scales to massive models, optimizes effectively, and unlocks modular and flexible adaptation applications including multi-task and incremental learning. Further, through the parallel exhaustive search of InCA we are able to better understand the inner representation dynamics of neural networks and construct illuminating 'representation signatures' of different models and datasets.

References

Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019. 3, 4

Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. 4

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals,

Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022. 4

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. 5

Björn Barz, Kai Schröter, Moritz Münch, Bin Yang, Andrea Unger, Doris Dransch, and Joachim Denzler. Enhancing flood impact analysis using interactive retrieval of social media images. Archives of Data Science, Series A (Online First) , 5(1):A06, 21 S. online, 2018. 6

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. 4, 6, 7, 8, 13

Rodrigo Ferreira Berriel, Stéphane Lathuilière, Moin Nabi, Tassilo Klein, Thiago Oliveira-Santos, Nicu Sebe, and Elisa Ricci. Budget-aware adapters for multi-domain learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pages 382-391. IEEE, 2019. 8, 9

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 2206-2240. PMLR, 17-23 Jul 2022. 4

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020. 4

Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations , 2019. 1, 2

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020. 4

Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attentionbased models for speech recognition. Advances in neural information processing systems , 28, 2015. 4

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2014. 6

John D Cook. Upper and lower bounds for the normal distribution function. John D Cook's Blog . 11

Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. A linearized framework and a new benchmark for model selection for fine-tuning. arXiv preprint arXiv:2102.00084 , 2021. 4

Chaitanya Devaguptapu, Samarth Sinha, K J Joseph, Vineeth N Balasubramanian, and Animesh Garg. δ -patching: A framework for rapid adaptation of pre-trained convolutional networks without base performance loss, 2023. 4

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4171-4186. Association for Computational Linguistics, 2019. 4

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. 3, 7, 8

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017. 4

Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7036-7045, 2019. 4

Raphael Gontijo-Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations , 2022. 4, 11

Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4805-4814, 2019. 4, 9

Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns for object segmentation and fine-grained localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 447-456, 2015. 4

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations , 2022. 4, 1

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016. 8

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2019. 6

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019. 3, 4

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. 3, 4, 6, 7, 8, 13

Vladimir Iglovikov and Alexey Shvets. Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation. arXiv preprint arXiv:1801.05746 , 2018. 4

Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights, 2022. 4

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. 7, 8

Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning , pages 46514664. PMLR, 2021. 4

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021. 5

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV) , 2022. 3, 4, 6, 7, 8, 5

Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition , Colorado Springs, CO, June 2011. 6

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017. 2

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision , pages 491-507. Springer, 2020. 3

Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2661-2671, 2019. 4

Zhi Kou, Kaichao You, Mingsheng Long, and Jianmin Wang. Stochastic normalization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 16304-16314. Curran Associates, Inc., 2020. 4

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) , Sydney, Australia, 2013. 6

Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In International conference on machine learning , pages 3744-3753. PMLR, 2019. 4

Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10657-10665, 2019. 4

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. 4, 8

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems , 34:9694-9705, 2021. 4, 7, 8

Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. 3, 4, 6, 7, 8, 13

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017. 1, 2

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2117-2125, 2017. 4

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021. 3, 7, 8

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 11966-11976, 2022. 3, 4, 7, 8

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. 12

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2017. 12

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019. 6

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV) , pages 181-196, 2018. 7

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013. 6

Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 67-82, 2018. 8

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing , Dec 2008. 6

New York Botanical Garden (NYBG). Fgvc 7 - herbarium 2020, 2020. 6

Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Transferability estimation using bhattacharyya class separability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9172-9182, June 2022. 4

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition , 2012. 6

Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition , pages 413-420. IEEE, 2009. 6

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021. 4, 7, 8

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 12

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-16. IEEE, 2020. 4

Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. International Journal for Digital Art History , (2), Oct. 2016. 6

Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Andrew Jackson. Fine-tuning image transformers using learnable memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12155-12164, 2022. 4

Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9594-9602, 2021. 3

Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV , page 516-533, Berlin, Heidelberg, 2022. Springer-Verlag. 7, 8

Cristina Vasconcelos, Vighnesh Birodkar, and Vincent Dumoulin. Proper reuse of image classification features improves object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13628-13637, 2022. 4

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 4

Matthew Wallingford, Hao Li, Alessandro Achille, Avinash Ravichandran, Charless Fowlkes, Rahul Bhotika, and Stefano Soatto. Task adaptive parameter sharing for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7561-7570, 2022. 4, 9

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems , pages 10506-10518, 2019. 6

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 139-149, 2022. 1, 2

Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010. 6

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492-1500, 2017. 3, 7, 8

Li Yang, Adnan Siraj Rakin, and Deliang Fan. Rep-net: Efficient on-device learning via feature reprogramming. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12277-12286, 2022. 4

Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. Advances in Neural Information Processing Systems , 33:17236-17246, 2020. 4

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research , 2022. 4

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision , pages 818-833. Springer, 2014. 4

Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 698-714. Springer, 2020. 4

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable {detr}: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations , 2021. 4

Appendix

Below we provide additional details and results which are not presented in the main manuscript.

A Continual Learning with Open-InCA

With the Open-InCA adapter, each class prediction is isolated using a different dedicated query and classifier vector. For continual learning tasks, in addition to running multiple adapters in parallel as presented for multi-task results in Table 4, Open-InCA enables an even more granular composition of adapter sub-tasks. Recall the Open-InCA adapter architecture is defined as

with LN denoting LayerNorm. Due to the properties of each operator, each class prediction can be computed separately as

Because of this property we can remove a class prediction or add a new class prediction without any effects on other model predictions (as long as the parameters of cross-attn and LN remain fixed). As presented in Sec. 4 we use 'query-only-training' which trains new adapter classes while freezing cross-attn , LN and enabling compatibility between task predictions.

When training with 'query-only-training' the softmax function, softmax( u ) = exp( u k ) ∑ c i =1 exp( u i ) , indirectly injects information of predictions from all classes due the normalization term in the denominator, which means gradients about a particular class i will include information from other classes j . Instead, we can achieve complete training separation by using a Sigmoid final activation,

σ

(

u

) =

exp(

exp(

u

u

)

) + 1

, and a Binary Cross Entropy (BCE) loss that considers each prediction separately.

Clearly in 'query-only-training' the adapter representation capacity is reduced, since the crossattention weights are not trained. We present an experiment evaluating the performance of InCA, Open-InCA and 'query-only-training' Open-InCA in Table 6 and observe that despite the isolated and reduced parameter set in query-only-training of Open-InCA, the method is still competitive and outperforms Linear Probing on most datasets.

Next we test Open-InCA for class-incremental learning, for which we consider the Split CIFAR-100 incremental learning benchmark. The Split CIFAR-100 dataset is trained with 10 incremental learning episodes each introducing 10 new classes. As in [10], we present the average episode accuracy and forgetting of 'query-only-training' Open-InCa and additional baselines.

In particular we evaluate Open-InCa using a ViT-B/16 along with state of the art methods L2P [72], LwF [46] and EWC [24]. Nonetheless, we do not apply any special routing of our learned episodic models and simply combine their predictions. In contrast L2P is a prompt based approach that, during inference, passes each new sample to an auxiliary classifier to predict its corresponding episode (in this case a 10-way classifier) and the corresponding episode model is up-weighted according to the prediction. We believe that with such an auxiliary classifier Open-InCA performance can significantly improve, nonetheless we observe that Open-InCa can simply leverage a larger model efficiently to achieve state of the art accuracy. We leave routing of samples to different learned sub-models as an interesting avenue for future work.

In addition, Open-InCA has additional benefits as compared to typical class-incremental learning approaches:

Flexible incrementation With Open-InCA different episodes can naturally contain a variable number of classes and episodes can be further decomposed if needed. This is since one can modify the model at the granularity of a single-class predictor via the Open-InCA adapter architecture by introducing or removing additional q i and W i .

Reduced forgetting risk With Open-InCA the ability of adding new classes without forgetting is built-in into the architecture, as prediction of different classes ensures that the

previous class predictions remain the same ( i.e. , no logit regression) which reduces catastrophic forgetting.

Parameter and computation efficient The Open-InCA adapter benefits from the InCA approach, which is parameter efficient and computationally efficient during inference (see Table 4) as well as during training (see Fig. 6 for comparison with prompts).

B Intermediate Representation Signatures

The parallel training of InCA results in the synthesis of tens of models that can run inference with insignificant per-adapter marginal costs. As a result we have the ability to glean highly useful information about the network's different representations and study the network inner representations effectively. This is especially important for recent non-convolutional based architectures that do not have as many inductive biases explaining some of their behavior. In this section we present results showing information we retrieve from the performance of InCA adapters.

Block 9

Block 10

Block 11

Block 12

Block 13

Block 14

Block 15

Block 16

Block 17

Block 18

Block 19

Block 20

Block 21

Block 22

Block 23

Output

Freeze point

80.0

82.5

85.0

87.5

90.0

92.5

95.0

97.5

100.0

Relative Accuracy (%)

DTD

CUB-200

Herbarium

Aircrafts

MIT-67

B.1 Partial fine-tuning and adapter performance

Below we present in detail the experiments discussed in Sec. 5, in particular regarding partial fine-tuning and InCA. The experiments illustrate the relationship between InCA adapters at different layers with partial tuning. We tune the pre-trained model starting from different 'freezing points'. In particular for neural network f ( x ) = g 1 ◦ . . . g l , for each freezing point g m we consider its position m and all of the preceding layers and apply layer freezing to g 1 , . . . g m -1 ( i.e. not updating gradients for those layers). Back-propagation is then only applied for optimizing g m , . . . g l including the network's prediction 'head'. In Figure 4 we show the dynamics of partial tuning, where we optimize the pre-trained network (ViT-L/16) in different runs with each run having a different freezing point. We compile the final Top-1 Test accuracy of each freezing run to create a partial tuning 'curve' for a single dataset. We compare the partial tuning performance curve of each dataset with the corresponding top layer of the InCA adapter trained on that dataset and observe that they are highly aligned, with datasets that prefer later InCA layers plateauing in their test accuracy earlier (at a later freezing point). In particular what we observe is that partial tuning performance plateaus roughly at the same layer where InCA identifies the top adapter representation. This is also the point at which partial fine-tuning is capable of harnessing that representation for the downstream task. Overall this gives further evidence that 'your representations are in the network' and fine-tuning simply surfaces existing representations that are already identified by InCA. When drawing the vertical lines of the top InCA adapters, we refer to output layers, e.g. , the adapter at block 19 means the adapter corresponding to the final output of block 19, or the first input of block 20.

B.2 Task Layer Affinities

In InCA we select top-performing adapters that 'listen' to different intermediate representations of a neural network. In our work we observe that one is able to achieve strong and diverse transfer learning by utilizing intermediate representations, and that for challenging tasks it is often required to use intermediate representations to achieve top results. Indeed the best representation layer for an adapter tends to be highly robust to hyper-parameter variables of the optimization. Even more intriguingly, we find that this representation affinity is preserved across different pre-trainings and even architectures. This is, certain tasks have a strong 'affinity' to a certain range of representation layers even for different architectural circumstances. The majority of the architectures we consider have some pre-trained component on one of the ImageNet datasets (aside from the CLIP ViT-L/14 model). At the same time, the fact that different architectures give rise to similarly helpful representations gives strong clues about the effect of different architectures as compared with the pre-training task during learning over a large diverse dataset such as ImageNet.

0

5

10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100Gap

Rel. pos. %

DeiT

Vit-L

DeiT-384

CLIP ViT-L/14

SWIN-L

ConvNext

Models

5.6

5.0

16.2

4.0

9.5

5.2

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

0

5

10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100Gap

Rel. pos. %

DeiT

Vit-L

DeiT-384

CLIP ViT-L/14

SWIN-L

ConvNext

Models

0.9

0.6

0.4

0.0

0.0

0.4

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0

5

10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100Gap

Rel. pos. %

DeiT

Vit-L

DeiT-384

CLIP ViT-L/14

SWIN-L

ConvNext

Models

2.1

2.5

6.9

5.5

8.4

5.1

0

2

4

6

8

10

In detail, we look at the best-performing InCA adapter for a fixed task on different architectures. The pre-trained models we consider consist of 2 different pre-trainings of the ViT-L/16 architecture (ViToriginal and DeiT), the 384-resolution pre-training of DeiT with the resolution adjusted ViT-L/16, CLIP's ViT-L/14 architecture, the SWIN-L architecture, and the convolutional based ConvNext-Base architecture. All of the vanilla ViTs we consider each have 24 residual transformer blocks so that comparing between blocks is directly aligned. SWIN-L and ConvNext follow the 'Stage' breakdown of blocks, namely SWIN-L has (2,2,18,2) stage breakdown that conveniently also adds up to 24 blocks (hence aligned in the figure) and lastly, ConvNext follows a (3-3-26-3) + head stage block composition, which we rescale in the figures to fit on the same 24 block range. In addition in the plots we also present the test error gap of each architecture with using the InCA adapter applied on its final block representation. Tasks that prefer earlier layers such as Stanf. Cars and Aircraft have a large gap from the performance of the last layer representation adaptation and such later layers lead to sub-optimal results.

We remark that the work of InCA sheds light on the inner representations learned by neural networks showing in some aspects performance is invariant to the architecture and more based on the pretraining dataset. We leave this topic for further research and find it to be an intriguing topic of study.

C Efficiency Results

InCA is highly efficient especially for large models, which is based on the isolated adapter architecture that does not modify the backbone. We delineate the efficiency aspects as follows:

Training memory efficiency The use of a frozen pre-trained model makes the training much more efficient and scalable since not all of the intermediate computations need to be stored as done in standard training or as required by methods that compute gradient information using inner-layers of the network. As soon as any intermediate layer requires a gradient, all subsequent activations must be held in GPU memory after the forward pass. This means methods like LoRA, FitBit and VPT all require storing of all of the activation maps for all of the layer operations in the network since they update parameters based on gradients from the very early layers in the network.

Fast optimization Unlike typical parameter efficient methods that insert some form of trainable parameters in the network, InCA adapters are trained with 'direct gradient' information coming from an isolated loss. Essentially each adapter corresponds to a very shallow neural network trained directly via back-propagation. This makes the training dynamics fast as direct gradient information about the loss easily reaches all of the adapter parameters. On the other hand, to update inserted parameters in the backbone, the gradient information is indirect and needs to be back-propagated through the backbone, with the risk of information loss and making the optimization more challenging, as we and the authors [34] observe regarding prompt tuning.

Efficient multi-task inference As we present in Table 4 the unchanged backbone execution enables efficient and parallel inference efficiency as multiple tasks can be evaluated at once.

C.1 Computational Efficiency of InCA Compared with VPT

In Table 7 we observe that InCA is an order of magnitude more efficient to train than VPT . For the results in the table we consider the VPT-Deep adaptation method, that is trained with 50 prompt tokens in each layer. We report calculated training times in GPU-hours of a standard Nvidia-T4 GPU using a ViT-L/16 architecture and accuracy numbers based on the datasets of Table 1 with the DeiT pre-training. For larger architectures such as ViT-H/14 ('ViT Huge') the difference in training-time is even more striking, as InCA maintains good per-run training time of 2.5, VPT-Deep requires staggering 55.8 GPU-hours per-run for a single GPU. On ViT-H/14 this is exacerbated as we must reduce the batch-size of VPT significantly to fit training on a common-place single GPU (Nvidia-T4). We measure in terms of training InCA and VPT-Deep for the same number of epochs. This however, is inaccurate as InCA trains an order of magnitude faster on a per epoch basis (see Figure 6).

We attribute the difference in training time of InCA to:

InCA does not require back-propagation through the whole model which gives ∼ 50% speed improvement alone.

InCA is robust to hyper-parameters and we optimize it using just 2 learning rates, compared with the hyper-parameter set of VPT (in our experiments we use 24 hyper-parameter configurations per dataset while using the full configuration presented in [33] takes even longer). In addition, with 'one-to-many' training, we train the two hyperparameters of InCA in parallel and report the specific training time in Table 7 denoted by ( ∗ ) for parallel hyper-parameter training.

InCA does not increase the number of propagated tokens in the transformer ( e.g. in VPT with 100 propagated tokens the attention matrix doubles, from ∼ 4 × 10 4 to ∼ 9 × 10 4 entries).

C.2 Optimization dynamics of InCA and VPT

In Fig. 6 we conduct an experiment where we train InCA and state of the art prompting method VPT-Deep [34] for different numbers of epochs and report the final test accuracy. We observe that InCA trains order of magnitude faster than prompting and reaches within 95% relative test accuracy after 3 training epochs.

0

3 5

10

20

30

40

50

Number of epochs in run

50

60

70

80

90

100

Relative Accuracy (%)

InCA

VPT Deep

D Theoretical Analysis

Empirically, we consistently observe that cross-attention as opposed to a linear or MLP-3 architecture enables InCA to better harness the existing model. We present a theoretical result asserting that using the cross-attention layer for aggregation as opposed to linear averaging, or even full-concatenation followed by a large dimensional linear layer is capable of learning over a strictly broader set of data distributions.

We give the precise statement in Theorem D.1 and intuitively argue that cross-attention with learned queries has the ability to sift through irrelevant pieces of the representation that may be at variable positions in different data samples.

Recall in the settings considered thus far, the extracted activation of an image data-point can be viewed as x i ∈ R d × T or T tokens e.g. x i = [ x 1 i , x 2 i . . . x T i ] , with x j i ∈ R d . We argue that in many scenarios, task-pertinent information is a property of individual tokens ( e.g. x j i ) within a data-point x i and not a property of the overall feature map. We present the theorem below. To this end we define a Token-Separability (TS) notion of a dataset.

Definition 1 (Token-separable Dataset) . A dataset D = { ( x 1 , y 1 ) , . . . ( x n , y n ) } with x i = [ x 1 i , x 2 i , . . . x T i ] ∈ R d × T and y i ∈ {-1 , 1 } is said to be linearly-token-separable if there exists a scalar c > 0 and w ∈ R d satisfying ‖ w ‖ 2 = 1 , such that for each data point ( x i , y i ) ∈ D there exists a token x j i i ∈ x i with

We define ( w D , b D ) and c 0 as the maximum margin solution and maximum margin respectively, i.e. c 0 = max {‖ w ‖ =1 ,b ∈ R } min D { y i ( 〈 x j i i , w 〉 + b ) } for D with ( w D , b D ) corresponding to the selected c 0 .

Intuitively, D is a TS-dataset if each of its data points contain a token that leads to linear separability (the same w shared among all points x i ∈ D ). One can further distinguish between aligned -TS Datasets, where the index j i of the linearly separating token is consistent among the n data points, or permutable -TS where j is dependent on i . Further TS datasets can be generalized to k -token separable datasets where k tokens are responsible for separability in each x i , for this theoretical contribution we don't make an assumption on whether the dataset is aligned or permuted, but consider the setting provided by Definition 1 (i.e. not the k -separable generalization). We present an analytical statement for the advantage of cross-attn , the theorem is provided for binary classification via a scalar prediction, but can conventionally extend to C -class classification. For binary classification we define a prediction via the standard scalar binary aggregator as σ ( u ) = sign ( ∑ i u i ) that converts a vector into a binary prediction.

Theorem D.1. Let D be a binary-class, token-separable dataset with max-margin c D and maxmargin solution ( w D , b D ) consisting of n data points. Suppose that D is distributed such that for x i = ( x 1 i , . . . x T i ) with ( x i , y i ) ∈ D are normalized for separating token x j i i ∈ D and that the rest of the tokens correspond to 'noise', x k i ∼ N ( 0 , I/d ) , E ‖ x k i ‖ 2 2 = 1 . Furthermore assume

Then there exists a cross-attention classifier

that separates D with probability at least 1 -δ . In contrast, every fixed member g ( x ; w,b ) = σ ( ∑ T l =1 x i · w + b ) of the linear classifier family will fail to separate D with probability at least 1 √ 2 π s s 2 +1 exp( -s 2 / 2) where s = √ d √ T -1 c D .

As stated above, the failure probability of the simple linear classifier g depends on s which satisfies s ∼ √ d/T . For existing architectures d, T tend to have a similar order of magnitudes, e.g. for ViT-B/16, d = 768 , T = 196 which makes the failure probability non-negligible. Before presenting the proof, we make the following observation: in InCA, we use the same cross-attn layer with latent q , which we show can be simplified via reparameterization.

Observation D.2 (Query Collapse Reparameterization) . A single-head cross-attention parameterization with latent [ q ] is equivalent to the following simplified layer, cross-attn( x , q ) = ∑ softmax( q ∗ x ) glyph[circledot] Wx with q ∗ ∈ R d .

This can be derived by decomposing the attention score which is the input to softmax .

Where q ∗ = q glyph[latticetop] W glyph[latticetop] q W k and q ∗ ∈ R d , hence the cross-attn layer simplifies, which is used in the proof.

As our proof shows, the cross-attn layer can operate on a large data bandwidth, e.g. x ∈ R d × T while still being selective in finding task specific representations. Empirically we also observe that increasing the number of heads of cross-attn also improves the performance of the InCA . This is in part because it enables the learned latent query parameter q to identify more useful token patterns, and since q is fixed using more heads remain stable (as opposed to when q is a data input). We now present the proof of the claim.

Proof of Theorem D.1

Proof. The proof of the theorem has two parts A) the positive condition on the cross-attn layer and B) the negative condition on the linear layer (a non-separability probability lower bound). We start with A) and consider separability of positive and negative data samples in turn. First we simplify and write an equivalent cross-attention binary classifier expression for the cross-attn classifier.

Positive result for the cross-attention model We consider a 'single-head' cross-attention layer and by Observation D.2 we can write the cross-attn layer as follows

Note that softmax is a function of the entire vector {〈 x j i , q ∗ 〉} j ∈ [1 ,T ] , however we write it in the form above to illustrate the summed terms. For simplicity of notation, we drop the asterisk and write q ∗ ∈ R d as q . Combining cross-attn with the binary aggregator, we have aggregation over the output vector of the cross-attn layer.

Define S ( x i , q ) ∈ R 1 × T as the computed softmax argument,

Substituting into the classifier, we have

Let u = ∑ d l =1 ( W v ) [ l, :] be the sum of the rows of W v . Note x j i can be pulled out from the inner summation to give

Thus the cross-attn classifier presented is equivalent to the parameterization above. Next we consider the two terms in the sum, namely S j and 〈 u, x j i 〉 . We will be deriving their distribution in the case where a data point ( x i , y i ) , has prediction labels y i = 1 and y i = -1 separately. We start with y i = 1 and consider

glyph[negationslash]

Take j i to be the separating token for sample x i . By the assumption of the theorem for k = j i the tokens correspond to isotropic noise of expected squared norm 1, i.e. x k i ∼ N (0 , I/d ) . For a fixed u ∈ R d with ‖ u ‖ 2 = 1 , we take η k to be the distribution of the dot product,

For k = j i this is a sum of independent Gaussians and each coordinate is distributed as ∼ N (0 , u 2 l d ) . As such we have

since ‖ u ‖ 2 = 1 . Next we consider k = j i . By the hypothesis we have that

With positive label ( y i = 1 ) this gives 〈 x j i i , w D 〉 + b D ≥ c D . Note that since c D ≥ 2 | b D | we have that 〈 x j i i , w D 〉 ≥ c D / 2 . Since w D is the maximal margin solution, we have ‖ w D ‖ = 1 and c D > 0 . Take q to be of the form q = t · w D for t ∈ R + and u = w D , then

glyph[negationslash]

For η k , k = j i , separating t , we have that 〈 x k i , q 〉 = t · η k . Define M = max k = j i ( | η k | ) . M is a random variable distributed as the maximum of of T -1 i.i.d. Gaussians distributed according to N (0 , 1 /d ) . We bound M by investigating an upper bound of the Gaussian CDF. Recall that the moment generating function of a Gaussian random variable X ∼ N (0 , 1) is given by M X ( r ) = E [ e rX ] = e 1 2 r 2 . Then note that for any s > 0 we have glyph[negationslash]

where the inequality is an application Markov's inequality. Setting s = r this gives the tail bound

For our settings with η k ∼ N (0 , 1 /d )

For a two-sided bound, by symmetry of the distribution we have

Therefore a union bound results in

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

We can bound the bulk of the distribution of M as

Taking r = c D / 4 , then with probability at least 1 -2 T exp( -d ( c D / 4) 2 / 2) = 1 -2 T exp( -d ( c D ) 2 / 32) we have glyph[negationslash]

glyph[negationslash]

With high probability, Eq. (13) holds. This implies that for j i , glyph[negationslash]

glyph[negationslash]

Note that c D > 0 and T is fixed. Nonetheless, the probability bound is independent of t which may take arbitrarily values, e.g. for any glyph[epsilon1] > 0 , take t = 4 /c D log( T/glyph[epsilon1] ) , which gives

glyph[negationslash]

Since S k ≥ 0 for each k and ∑ T k =1 S k = 1 we have for k = j i , glyph[negationslash]

We consider the classifer prediction

where b is a bias parameter we can choose. Recall u = w D . Focusing on the inside of the sign function glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

provided that glyph[epsilon1] < 1 / 3 . If we take b = -c D 4 we have that for q = tw D and u = w D

and thus glyph[negationslash]

Next we address the case where y i = -1 . We consider the classifier prediction

with u = w D . Again for k = j i we have that glyph[negationslash]

On the other hand for j i , y i = -1 we have that

where we have used the hypothesis that c D ≥ 2 | b D | in the last line. We consider the term inside the classifier. We note that glyph[negationslash]

glyph[negationslash]

where in the first inequality we have used the fact that S j i 〈 u, x j i i 〉 < 0 and in the last inequality we have used the fact that ∑ T j =1 S j = 1 . Therefore again with bias term b = -c D / 4 we have that

and f ( x i ; q, u, b ) = -1 = y i . Thus we have just shown that with probability 1 -2 T exp( -d ( c D ) 2 / 32) the model f ( x ; q, u, b ) with u = w D , q = tw D , b = -c D / 4 gives the correct label for x i . Taking the union bound over all n points in D we get with probability at least 1 -2 Tn exp( -d ( c D ) 2 / 32) ≥ 1 -δ the model f ( x ; q, u, b ) with u = w D , q = tw D , b = -c D / 4 separates D .

Negative result for the linear model We consider the linear classifier

where w ∈ R d is restricted to have unit norm ‖ w ‖ = 1 . For an input x i under the aggregation, the term inside the sign function simplifies to

We recall that the x k i for k = j i are distributed according to N (0 , 1 d I ) . Thus we have that glyph[negationslash]

So the problem of classification is equivalent to learning a linear classifier over the separating tokens under the presence of Gaussian noise with distribution N (0 , T -1 d ) . Let i ∗ be the index corresponding to the input x i ∗ with smallest margin, i.e.

glyph[negationslash]

glyph[negationslash]

Then we have that y i ∗ ( 〈 w,x j i ∗ i ∗ 〉 + b D ) ≤ c D . We note that any linear classifier g ( x ; w,b ) with ‖ w ‖ 2 = 1 will fail to classify D whenever y i ∗ α i ∗ < -c D . Thus we will lower bound the probability of P ( y i ∗ α i ∗ < -c D ) . Note for a standard Gaussian random variable η ∼ N (0 , 1) as shown in [14] we have for r > 0

Set s = √ d √ T -1 c D . Then by symmetry of the Gaussian distribution the above bound translates into the following bound for α i ∗

It follows that for any w ∈ R d such that ‖ w ‖ 2 = 1 that the linear classifier g ( x ; w,b ) incorrectly classifies D with probability at least

This completes the second part of the proof.

E Further Results

We present additional experiments below. In Subsection E.1 we present per-dataset results for additional architectures and a discussion about ensembling InCA is given in Subsection E.2.

E.1 Per-dataset results for different architectures as presented in Table 2

Table 8 provides per-dataset results that are presented in aggregate in Table 2. Below we present the results for ConvNext-Base and ViT-L/16 (original pre-training) pre-trained models (with the results for ViT-L/16 DeiT and SWIN-L presented in Table 1 and Table 3 respectively).

E.2 Ensembling learned adapters

Because of 'one-to-many' inference of InCA can take a set of independently learned adapters and ensemble them without a marginal increase to the inference cost. We follow non-parametric equalweight ensembling, by taking the output predictions of two adapters h 1 ( x ) , h 2 ( x ) on a sample image x . Note that the adapters are computed with their relevant representations via a single forward pass, which makes the execution of h 1 ( x ) and h 2 ( x ) together only incrementally higher than computing just h 1 ( x ) . The ensemble is defined as

Given the large combinatorial selection of k adapters from the l learned adapters we consider the case of ensembling with just two adapter members. After training we evaluate all m ( m -1) / 2 such pairs and compare them with the top performing single layer predictor which we present in Figure 7. In the figure, we illustrate the representations and corresponding adapter pairs that lead to best performance and also present the computed ensemble gain which is the difference between the ensembled model accuracy and the top accuracy of any single adapter.

In addition to improving classification accuracy, ensembling can aid in improving robustness and out of distribution performance which we leave as a future work. Further directions of ensembling include ensembling performance when using adapters of different adapter architectures (e.g. an MLP-3 ensembled with an InCA adapter) or adapters that use representations from different neural networks [21].

E.3 Ablation on the number of queries

We apply an ablation to see the effects of using a different number of queries in the InCA adapter architecture. In particular, the InCA adapter is written as,

.

For m> 1 the output of tokens [ q 1 , . . . q m ] through the cross-attn layer are averaged, and we test whether using m> 1 brings additional representational benefit to each adapter. We present the result in Table 9 and observe that using a different m does not have a consistent effect on the accuracy of the learned adapters, and in our experiments we use m = 1 for InCA adapters to be most computationally efficient.

F Implementation details

We present the optimization and augmentation details for training InCA, and note we use standardized procedures for augmentation and training (without extensive hyper-parameter optimization) of the different transfer learning methods we evaluate.

Augmentation Unless otherwise specified we train with input image size 224 and standard augmentation practice [62]. In particular, during training we resize to image-size 256 and apply random cropping, for testing we apply resizing and center cropping. For larger image resolutions we maintain the same resize-crop ratio of 0 . 875 .

Optimization For the linear probing and InCA approaches, we train with the AdamW optimizer [50], cosine annealing learning rate scheduler [51] for 30 epochs and with weight decay 1e -4 . In each method we sweep over 2 learning rates lr = {1e -4 ,3e -4 }. For full fine-tuning, we also train with AdamW optimizer (weight decay 1e -4 ), cosine annealing for 30 epochs, but in addition, identify optimal learning rates for each pre-training and architecture separately. We first identify an

Block 11

Block 12

Block 13

Block 14

Block 15

Block 16

Block 17

Block 18

Block 19

Block 20

Block 21

Block 22

Block 23

Last layer

Ensemble gain

Ensemble pieces

cub200

describabletextures

europeanflooddepth

fgvc_aircrafts

mit67

oxfordflowers

pets

stanfordcars

stanforddogs

Dataset

1.0

1.9

3.1

2.7

0.7

0.1

0.3

3.9

0.4

0

1

2

3

4

5

architecture coarse-range learning rate based on performance on 5 datasets by sweeping over lr = {1e -2 ,1e -3 ,1e -4 ,1e -5 ,1e -6 } followed by a refined sweep with learning rates lr = B, 2B with B being the optimal coarse learning rate.

For the VPT baseline, we follow the details presented in the paper and train with VPT-Deep which was observed to outperforms VPT-Shallow. To train VPT, we use the SGD optimizer with momentum and cosine annealing for 100 epochs. For each dataset we run a sweep on the prompt length {5,20,100}, base learning rate {0.25,0.1,0.05,0.01}, and weight-decay {1e -2 ,1e -4 } for a total of 24 runs with 100-epochs for each dataset. We compare the training cost of InCA and VPT-Deep in Table 7. In general we note that the shallow and small architecture of InCA or linear probing that are separate from the base model makes them straightforward to optimize, compared with adaptation methods that receive back-propagated gradients from a frozen intermediate layer of the network as shown in Fig. 2.

For the LoRA baseline [28] we apply a LoRA modified attention to each block's self-attention layer ( W k , W q , W v ) in ViT based architectures and to each block's WindowAttention for SWIN. For the low rank dimension we sweep over the best value among d = 5 , 10 , 50 . For BitFit we follow the discussion in [6] and train all of the bias-parameters in the network in addition to full training of the head. Analogously for [45] we follow their procedure with LayerNorm which includes training each of the LayerNorm parameters ( γ, β ) for each layer along with training of the head of the pre-trained model. For all of the efficient training methods above we sweep over lr={3e -5 ,1e -4 ,3e -4 ,1e -3 } to identify the best learning rate for the dataset.

Broader Impacts Our method, InCA enables efficient and modular model adaptation that can be applied to any strong available pre-trained backbone. In that sense, InCA reduces the computational barriers to entry for training and evaluating over a large set of (potentially massive scale) models and optimization settings to identify a model to be used for downstream adaptation. This bridges the gap between cutting edge research in general visual representation learning and specific domain applications, especially since the best performing models are computationally expensive to adapt. Given that InCA operates well on fine-grained visual datasets, this can have positive applications in scientific domains such as medical imaging. In many scientific domains, the available datasets are known to be fine-grained yet also with sparse training data. In addition the ease of use and reduced computational costs associated with downstream adaptation with InCA makes it possible for domain experts without machine learning expertise to use InCA without access to large computational resources. This can enable domain researchers solve their domain problems by leveraging various public pre-trained models to achieve competitive results.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a study on the benefits of training a small cross-attention based adapter instead of performing full fine-tuning of a large VIT model. The authors propose a cross-attention layer they dub InCA that has trainable queries to cross attend to intermediate layers in the large pretrained VIT model and then extract the information by average pooling, layer-norm and Linear. They claim a single query is enough in most cases but also propose a multiple query version they call OpenInCA.\n\nSTRENGTHS:\n* There are strong benefits of having a methodical study of adaptation and I think this paper fits the bill.\n* The method is simple, well explained and a priori easy to reproduce and expand.\n* The experiments are broad enough to be interesting to the community.\n* There are clear computational benefits from the two stage training.\n\nWEAKNESSES:\n* The claim in L177 is a bit of a red flag for me. If only one query is enough it may be that the way this is achieved is when the query is very far from the \"latent\" data and thus the attention weights become flat ie performing average pooling. The nice thing is that it would be an easy and i think useful experiment to check this. Redoing figure 3 with avg pooling but with all the extra bits like the layer-norm that is in the model. I presume the LP doesn't have it ?\n* I haven't found any experiments to validate that the class-incremental learning benefits of the Open InCA architecture which is claimed in line 203.\n\nQUESTIONS:\n* Have you tried multiple self-attention layers instead of just one? This would be equivalent to running a small perceiver model [30] and then either average the resulting latent tokens.\n* The average pooling has to be done to make sure that the cross attention IS the right architecture here and not a simpler one. It would also add a bit of depth to the paper and answer an obvious question. I think this is required to recommend acceptance.\n* Why do you include both tables 1 and 3 ?\n\nLIMITATIONS:\nI think there is no obvious negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper proposes a new way to adapt a pretrained deep neural network for downstream tasks called InCA. InCA does not modify the intermediary representations of the pretrained network and thus doesn’t require backproping through it, which makes it memory- and compute-efficient. To use InCA, one first heuristically identifies a few layers, the activations of which are processed through cross-attention with trainable query tokens.\n- The cross-attention setup is more expressive than linear maps, which is supported by empirical and theoretical arguments. It also allows the addition of new classes through the injection of new query tokens.\n- Finally, because the forward pass of the original network is not at all modified, one can cache the relevant activations of the entire dataset once and apply InCA without calling the original model at all. It is also possible to adapt to many tasks in parallel for the same reason.\n- Experiments on many image datasets show that InCA performs competitively with finetuning and other efficient adaptation methods.\n\nSTRENGTHS:\n- This work includes a broad range of baselines in their experiments.\n- The proposal is simple yet appears to be effective. The stated benefits are technically sound.\n\nWEAKNESSES:\n- The paper “[presents] a framework for transfer learning that efficiently adapts a large base model by learning lightweight cross-attention modules attached to its intermediate activations.” However, the experiments are limited to mostly computer vision classification tasks. Some of the baseline approaches, such as LoRA and BitFit, are extensively used for language understanding, language generation, and image diffusion. The result of this paper will be much more convincing if experiments in these domains are included, especially if InCA can outperform existing baselines.\n- A significant portion of the benefit of InCA comes from using a subset of the layer activations. However, this important choice (as described in B.2) is done heuristically according to Section 3. The paper can benefit from more clarity on how to choose such layers. For example, how are these layers chosen for the experiments?\n- The clarity of the writing can be improved. For example, it was not clear during my first pass what the dimension of z is and what T is. The description for diag-head is confusing: it appears to be a simple matrix multiplication between [a_1, …, a_c] and W. Section 3 overall can be better organized by heuristics, methodology, and benefits.\n- The “signature” phenomenon is interesting, but the given analysis doesn’t provide much insight. It would be great to hear more about what we can learn from these “signatures,” especially given that they are presented as one of the key advantages of InCA in the conclusion.\n\nQUESTIONS:\n- How competitive is InCA in other domains where many of the compared baselines, e.g., LoRA and BitFit, dominate? E.g., language understanding, language generation, diffusion, etc.\n- How is the subset of layers chosen for the experiments? Could a similar selection process benefit other baselines?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an efficient fine-tuning method that works parallel to the pre-trained network. Based on cross-attention between intermediate activations, InCA can generalize to various classification tasks from different domains. Additionally, the framework inherently supports class incremental learning, multi-task learning and continual learning objectives. The overall objective is cross-layer feature merging using cross-attention between them while freezing the base model.\n\nSTRENGTHS:\n1. There are various parameter-efficient fine-tuning methods in the literature; the major strength of this work is described in the last part of Sec. 3 - using cached activations to train the model in mere seconds.\n2. The overall formulation of cross-attention is simpler and elegantly extends to multi-task and continual learning paradigms.\n3. Experiments are shown on various vision classification datasets and one multi-task learning benchmark.\n\nWEAKNESSES:\n1. The formulation of Open-InCA is not completely clear and can be better presented; However, I understood the complete idea, and I had to re-read it a couple of times for deeper understanding.\n2. The field of efficient fine-tuning/transfer learning is rapidly moving, and using standard benchmarks helps understand the performance gains more. The authors can give results on VTAB-1k, few-shot learning experiments, etc, as shown in SSF[1], VPT[2], FacT[3]\n3. Also, I found comparison with existing methods a major weakness, as only LoRA, BiTFit and AdaLN are shown.\n\n[1] Scaling & shifting your features: A new baseline for efficient model tuning. [2] Visual Prompt Tuning [3] FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer\n\nQUESTIONS:\nSee the weakness sections.\n1. Standard benchmarks and a thorough comparison with existing works will make this work a lot stronger\n2. A major emphasis on peak training peak memory, inference latency and parameter overhead needs to be provided.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method termed InCA (Introspective Cross-Attention) to learn compact adapter modules for large vision models that can be used for various downstream tasks (image classification domains). The proposed approach has an advantage over the entire model finetuning due to its parameter efficiency resulting in a smaller GPU memory footprint required for training. In addition, the proposed introspective cross-attention module is architecture-agnostic, enabling simple implementation for different intermediate network representations and network types.\n\nSTRENGTHS:\nThe proposed approach (InCA) demonstrates stable performance when applied across different model types (ViT, SWIN, CNN). Unlike previous architecture-specific adaptors, the method can be implemented without modifications with different base model architectures.\n \nThe experiments demonstrate that the method can reach performance comparable to the entire model finetuning. At the same time, the number of trainable parameters constitutes only a tiny fraction (1-2%) of the backbone model weights. Because the training process doesn’t require backpropagation through the backbone, it is GPU memory efficient, making the large pre-trained transformer-based backbones reusable for different downstream tasks. \n \nCompared to the linear probing, the proposed cross-attention architecture of the adaptor module has a significantly larger “extraction capacity” that leads to an improved classification accuracy of the adapted model. When applied in parallel to different activation maps, the method produces a network “signature” - w.r.t. to the downstream tasks. All this highlights the fact that the internal representations of the pre-trained large models have sufficient representation power for many downstream domains.\n\nWEAKNESSES:\nThe paper builds on the large body of literature on efficient transfer learning. While exploring the utility of cross-attention as a choice for adaptor architecture, the authors follow the steps of many prior works, such as Perceiver [30] or [A]. On the other hand, the current study combines empirical results for different backbone architectures (e.g., both CNNs and Transformers) and suggests applying cross-attention modules in parallel to different activation maps to identify the most relevant features for the downstream tasks systematically. Thus, despite being a comprehensive study, the novelty is limited. As discussed in the Appendix, InCA, as an efficient and modular model adaptation framework, can be helpful for domain practitioners (e.g., medical imaging domain) to bridge the gap between cutting-edge research in visual representation learning and real-world applications.\n\n[A] Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation, in EMNLP 2021.\n\nQUESTIONS:\nCould you highlight the most interesting (maybe counterintuitive?) findings? “The second best adaptation approach is LoRa … at additional training costs” invites the comparison along axes other than the error rate. Could you compare the training costs directly?\n\nTable 2: “\\dagger indicates full FT was avoided due to prohibitive computational costs” – could you elaborate on what costs were considered prohibitive and why?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a method called Introspective Cross Attention (InCA), which aims to identify high-performing adapter models for handling downstream tasks using large-scale pre-trained models. InCA achieves competitive performance compared to the well-established baseline of full fine-tuning, while also enabling parallel multi-task inference. The experimental results effectively demonstrate the effectiveness of this method in terms of both performance and efficiency.\n\nSTRENGTHS:\n1. The paper is well-written and easily comprehensible. The Introduction section, in particular, effectively establishes the context for the entire work.\n2. The exploration of introducing adapters for large-scale models in the context of vision transformers is still relatively less explored.\n3. The promising results on well-known transfer learning datasets at a smaller scale indicate the potential of the proposed method.\n\nWEAKNESSES:\n1. The datasets employed in this study may not be the most appropriate test-bed for evaluating large-scale pre-trained models. Previous transfer learning works [1, 2, 3, 4] have achieved comparable accuracies using lightweight CNN models that demand significantly fewer FLOPs and parameters. To gain a comprehensive understanding, it would be valuable to compare the FLOPs and parameter counts.\n2. While I acknowledge that the authors have utilized commonly used datasets for transfer learning tasks, it is worth noting that these datasets may not provide sufficient challenges when employing models such as ViT-G/14 and other large variants. \n\n\nReferences\n1. [Co-Tuning for Transfer Learning](https://proceedings.neurips.cc/paper/2020/hash/c8067ad1937f728f51288b3eb986afaa-Abstract.html)\n2. [Stochastic Normalization](https://proceedings.neurips.cc/paper/2020/hash/bc573864331a9e42e4511de6f678aa83-Abstract.html)\n3. [Bi-tuning of Pre-trained Representations](https://arxiv.org/pdf/2011.06182)\n4. [$\\Delta$-Networks for Efficient Model Patching](https://arxiv.org/pdf/2303.14772) -- Table-7\n\nQUESTIONS:\n1. Could you please provide (or comment) a comparison in terms of FLOPs? \n2. In terms of memory and compute how does CNN based transfer learning approaches compare with ViT based approaches? \n3. This approach might suit very well for tasks like model patching, see [PAINT](https://model-patching.github.io/) and [Delta-Networks](https://arxiv.org/abs/2303.14772)\n\nLIMITATIONS:\nDiscussed in the appendix", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary: Firstly, I would like to kindly point out that this paper proposes an \"adaptation\" method; however, I believe there may be some serious concerns in multiple aspects. Firstly, the paper appears to lack innovation in its methodology. Secondly, the experimental design also seems to have some significant shortcomings, such as lacking adequacy and detailed experimental evidence. Additionally, I would like to express my concerns about the writing aspect of the paper, where some details seem to be missing and where some over-claims are made.\n\nSTRENGTHS:\n(Positive) Although this paper possesses some flaws, including the lack of innovation and rigorous experimental design, I must acknowledge that the approach in this paper appears reasonable to some extent. Additionally, it is commendable that the authors have provided a significant amount of supplementary material. However, I believe these aspects may not fully compensate for the shortcomings observed in other areas of the paper. I genuinely hope that in future research, the authors will earnestly consider these critical points and endeavor to enhance the quality of their work.\n\nWEAKNESSES:\n(Negative) Allow me to further inquire about some of the claims made in the paper. In the first sentence of the article, the authors claim that in natural language tasks, the data and hypothesis space are shared, which appears quite astonishing. Considering the diversity of tasks in natural language, can this assumption be readily applicable to every task? I believe this general statement may lack sufficient basis and in-depth investigation.\n\n(Negative) Furthermore, the authors assert that their model can enhance \"robustness,\" but it is important to note that \"robustness\" is an extremely specialized concept. Therefore, I hope they can provide more detailed experimental evidence to support this claim. A simple assertion may not be sufficient to substantiate such an important proposition.\n\n(Negative) I must emphasize that the approach presented in this paper does not seem to demonstrate significant differences compared to existing methods, including but not limited to LoRA. This apparent lack of innovation in the technical aspects is somewhat disappointing.\n\n(Negative) Moreover, it appears that the authors' method has not undergone sufficiently large-scale model validation to demonstrate its feasibility in terms of large model transferability. Additionally, the model they have chosen may not entirely qualify as a so-called \"foundation model.\" This lack of rigor in the experimental design raises doubts about the viability of this method.\n\n(Negative) Additionally, the authors seem to have not thoroughly validated the capabilities of their method on various diverse tasks. The scope of their experiments appears to be rather narrow, omitting coverage of multiple tasks such as generative models and discriminative models simultaneously. For instance, in generative models, the training cost of the so-called \"stable diffusion model\" may be high, but its adaptation is crucial. Unfortunately, the authors' in-depth research and contributions in this area seem to be limited.\n\n\n(Negative) What is also concerning is that the authors of this paper appear to have a somewhat limited perspective, primarily focusing on a few tasks they studied, while neglecting the introduction of task details. Moreover, they may hold the belief that ImageNet pretraining is the default choice worldwide, a viewpoint that is challenging to comprehend. In reality, a foundation model should not be confined to ImageNet pretraining but should consider larger benchmarks to enhance its generalization ability. I would like to strongly urge the authors to be mindful of this and clearly indicate in the paper where their model was actually pretrained.\n\nQUESTIONS:\nSee *Weaknesses\n\nLIMITATIONS:\nNo. See *Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}