Your representations are in the network: composable and parallel adaptation for large scale models
Authors: Yonatan Dukler
Alessandro Achille
Hao Yang
Varsha Vivek
Luca Zancato
Benjamin Bowman
Avinash Ravichandran
Charless Fowlkes
Ashwin Swaminathan
Stefano Soatto
Venue: NeurIPS 2023 poster (2023)
Keywords: Efficient learning
Compute-efficient deep learning
Deep Learning Theory
class-incremental-learning
downstream adaptation

Abstract: We present a framework for transfer learning that efficiently adapts a large base-model by learning lightweight cross-attention modules attached to its intermediate activations.
We name our approach InCA (Introspective-Cross-Attention) and show that it can efficiently survey a network’s representations and identify strong performing adapter models for a downstream task.
During training, InCA enables training numerous adapters efficiently and in parallel, isolated from the frozen base model. On the ViT-L/16 architecture, our experiments show that a single adapter, 1.3% of the full model, is able to reach full fine-tuning accuracy on average across 11 challenging downstream classification tasks.
Compared with other forms of parameter-efficient adaptation, the isolated nature of the InCA adaptation is computationally desirable for large-scale models. For instance, we adapt ViT-G/14 (1.8B+ parameters) quickly with 20+ adapters in parallel on a single V100 GPU (76% GPU memory reduction) and exhaustively identify its most useful representations.
We further demonstrate how the adapters learned by InCA can be incrementally modified or combined for flexible learning scenarios and our approach achieves state of the art performance on the ImageNet-to-Sketch multi-task benchmark.

Your representations are in the network: composable and parallel adaptation for large scale models
Yonatan Dukler
Alessandro Achille
Hao Yang
Benjamin Bowman"
Varsha Vivek
Luca Zancato
Avinash Ravichandran
Charless Fowlkes
Ashwin Swaminathan
Stefano Soatto
AWS Labs
(dukler, aachille, haoyng, bownaben, varshviv, zancato, ravinash, fowlkee, swashwin, scattos]@amazon com
Abstract
We present a framework for transfer learning that efficiently adapts large base- model by learning lightweight cross-attention modules attached to its intermediate activations. We name our approach InCA (Introspective-Crow Caaeee-rrsssAAA (ross-Attention) and show that it can efficiently survey network's representations and identify strong performing adapter models for a downstream task. During training, InCA enables training numerous adapters efficiently and in parallel, isolated from the frozen base model. On the ViT-L/16 architecture, our experiments show that a single adapter, 1.3% of the full model, is able reach full fine-tuning accuracy average across 11 challenging downstream classification tasks. Compared with other forms of parameter-efficient adaptation, the isolated nature of the InCA adaptation is computationally desirable for large-scale models. For instance, we adapt ViT-G/14 (1.8B+ parameters) quickly with 20+ adapters in parallel a single V100 GPU (76% GPU memory reduction) and exhaustively identify its most useful representations. We further demonstrate how the adapters learned by InCA can be incrementally modified or combined for flexible learning scenarios and our approach achieves state of the art performance on the ImageNet-to-Skettl multi-task benchmark.
Introduction
Foundation models promise achieve top performance with minimal adaptation any downstream task. In the realm of language, the data and the hypothesis spaces are shared, and many tasks can be unified into homogeneous representation. Visual inference domains, on the other hand, can be highly heterogeneous and possibly antagonistic. For instance, the hypothesis space for pose estimation geometrie, whereas for scene classification it semantic and even domains that appear homogeneous, such as image classification into the 10 CIFAR classes, can trigger interference in the trained model in the presence of modest perturbations of the image statistics.
Antagonistic domains may interfere within the activations of the latter layers, which cannot be simultaneously minimal and sufficient for all domains. However, information about dissimilar domain may be present in earlier layers, and certainly in the input data which is trivially sufficient
Work conducted while interning at AWS Al Labs.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

12.0
60GB
---
InCA adaptafi
Top-1 Error
25.0
Torea Tany
Training Memory
50GB
InCA 120 asapterst
Rul fnenuiing
20.0
10.0
-
Porg: Wuring (VPT)
20.0
40GB
15.0
15.0
8.0
30GB
10.0
KV100 GRU
5.0
VIT-8/16 VITU/16 VIT-V14 VIT-G/14 VIT-Gig/14
Figure I: (Left)) Top-1 Test Error for fine-grained classification transfer learning tasks evaluated with the ViT-L/16 architecture. InCA performs comparable to full fine-tuning on each challenging dataset. (Right) Max GPU Memory usage during training for different adaptation approaches and model sizes.
Large Backbone
InCA
Open-InCA
Layer M+1
ArC Paol
Queries
Cucrics
Layer
%
Cross Atantion
Crosse
O:
Layer 1
Layer Activations
Latyr Activallioee
(A)
(C)
(D)
Figure 2: InCA Adaptation In (B), intermediate activation maps are extracted from a pretrained
backbone during forward pass. Each activation map is passed to a lightweight InCA adapter (shown in (C)) or Open-InCA adapter (shown in (D)) depending on the task settings. In (A), we illustrate how multiple adapters trained in parallel and independently, and during inference be combined used parallel. (C), (D) we present schema of the InCA and Open-InCn InCA adapters; see Sec. for details.
for any task. Indeed, opposed to just operating with the final model's representations, the typical approach addressing domain variability in transfer learning is by applying full fine-tuning of the model new data. By optimizing all of the model's parameters, each of the model representations can be potentially harnessed as a starting point for the adaptation.
While accurate and robust to variety of domains, full fine-tuning of large-scale models entails sweeping computational and storage costs. Further, the resultant model can only function for its dedicated task, not allowing for shared computation and parallel execution of potential future tasks. To tackle the problem of efficient, versatile, and modular adaptation we introduce Introspective Cross-Attention (InCA). InCA operates the base-model by attaching isolated shallow adapter modules utilizing any the activation maps for downstream tasks.
Since modern architectures are deep and so is the variety of possible downstream tasks, iterating over all the possible candidate activations of model is prohibitively expensive. Instead in InCA we search for useful model representations exhaustively and in parallel by training numerous isolated adapters attached to different activations. When using InCA, each adapter light and the base-model is fixed and does not require any backpropagation. This makes InCA computationally efficient in terms GPU memory which is crucial for scaling to large models. Further, the shallow InCA adapter networks simplify the training dynamics compared with existing approaches which speeds up training considerably and makes the optimization straightforward and robust (see Appendix C).
In detail, during parallel training of InCA, set of adapters sharing the same architecture are trained simultaneously and independently on a downstream task. Each adapter accepts an assigned activation and does not feed back to the backbone, leaving the backbone execution unaltered during both training and inference (Fig. 2). At inference, the learned adapters may be combined or best performing adapter can be selected for downstream prediction. The InCA adapter architecture is simple, consisting of single artss-attention module followed by linear classifier for the downstream task. Despite the simplicity of the adapter, observe that single popperfformin adapter trained for downstream

---

task is capable of achieving strong performance across variety of different architectures and pre- trainings when tested on diverse set of visual domains. Because our approach does not modify the execution of the model or any of the pre-trained layers done in existing parameter-efficient approaches [28, 34, 27], our method can be automatically applied to any architecture without the hassle of re-implementation and architecture specific modifications. In particular, we present results of InCA for ViT [18], SWIN [48], and CNN architectures [49, 74] on a suite of fine-grained recognition tasks.
Since the adapters learned in InCA are shallow and small (1.3% of the parameters on ViT-L/16), the strong performance of single adapter implies that many pre-trained models already contain strong representations for diverse set of downstream tasks. Instead, previous approaches like linear probing fall short in using the representations, not because the task can not be solved using the existing model, but rather because of not having the right "extraction capacity" of cross-attention; we explore this systematically in Sec. 4 and present a theoretical proof for the advantage of the cross-attention layer in Appendix D. For challenging datasets, using intermediate representations as opposed to only adapting the last layer's 's representation is key to making InCA versatile method that closes the gap with full fine-tuning on diverse tasks (See Fig. 1).
A byproduct of the exhaustive approach of InCA adaptation is a signature describing the performance of the internal representations of a network on different downstream tasks. This renders InCA as powerful tool for understanding the underlying representations of pre-trained models and task space [1] (See Sec. 5, Appendix B). Curiously, we observe that in certain downstream datasets, different pre-trained models hold similar performance signatures, even when the pre-trained models use different architectures or pre-training augmentations.
The isolated training of the adapters means no backpropagation through the pre-trained model is taking place. This significantly reduces the correlation between adaptation cost and the pre-trained model size, which makes it possible to leverage very large architectures even with modest compute resources. Fig. 1 shows that one V100 GPU can train InCA with 40 adapters using an architecture as big as ViT-G/14. In contrast, existing parameter efficient approaches that backpropagate through the architecture exhaust GPU memory with any model larger than ViT-B/16.
Our contributions are summarized as
We introduce InCA, a method that enables versatile downstream adaptation by surveying the internal network representations of any frozen model with lightweight cross-attention based modules as an alternative to full fine-tuning. On ViT-L/16, InCA matches full fine-tuning performance and reaches within 0.7% accuracy for SWIN-L backbone on average over 11 diverse downstream domains.
We demonstrate how the modular adapter architecture of InCA enables flexible learning and inference scenarios and present the Open-InCA adapter. With our approach we unlock pow- erful pre- treitrained models for reusable and parallel multi-task inference, and class-incremental learning.
On the efficiency front, InCA scales to massive scale architectures under typical computation budgets while its implementation can be automatically applied to any new model. For example, training VIT-G/14 using InCA results in 76% GPU memory reduction as compared with full fine-tuning. Further, InCA is easy to optimize and highly parameter efficient (See Appendix C).
The rest of the paper is organized as follows: In Sec. 2 we review related work, and in Sec. 3 we present our approach. We empirically evaluate InCA on wide set of visual recognition tasks in Sec. 4. Lastly we provide analysis of intermediate representation signatures (Sec. 5) followed by discussion (Sec. 6). Additional results and analysis including Open-InCA are presented in the Appendix.
2 Related works
Transfer learning Transfer learning deep neural networks aims at endowing existing models with new "downstream"" knowledge. The de-facto approach for transfer learning in deep learning modifies an existing pre-trained network by applying full, partial or linear fine-tuning on the model [37, 66, 45]. Depending on the the variability between the source and downstream task domains,
3

---

different approaches aim at delineating different transfer domains [1, 58, 76], extending coverage of transfer via suitable pre-trainings such as meta-learning [42, 19, 30], or by selecting and combining from set of expert models [15, 21J. More broadly the field of representation learning focuses on learning transferable representations [38] via different pre-training strategies that can be applied for downstream tasks [11].
Efficient adaptation methods In recent times, the top-performing pre-trained model architectures are becoming considerably larger [61, 49], and we arrive at a crossroad as full fine-tuning of such large models is becoming out of reach in many practical settings. Recent works address the storage costs associated with large model transfer by proposing parameter efficient transfer approaches as opposed to storing all of the tne-tuned model parameters [24]. These approaches achieve parameter efficiency by training a subset of existing parameters [6, 45], inserting new weight parameters into existing modules [28, 27, 39, 16] or via additional learnable activations inputs, known as prompts [34, 65, 43). Compared with existing work [79, 75]. InCA is also parameter efficient, yet we place special emphasis compute and optimization efficiency, especially in the large-scale model setting. Additional lines of work study learning via selective tuning, enabling multi-domain transfer [70, 22].
Feature extraction with attention Self and cross-attention mechanisms aggregate information from set of feature tokens [69, 41, 44] and create representations based on relative inter-feature importance as computed by the attention mechanism [12, 69]. Self-attention plays a key in the transformer architecture [69] enabling non-local information aggregation. In settings where the number of inputs and desired outputs differ, cross-attention enables flexible aggregation based on pair of query and key feature sets. When using cross-attention, one can cross-attend between different sets of activations [44, 17] or between set of activations and learnable latent parameters [32, 9, 80]. In our settings, the adapter architecture applies cross-attention on extracted representations from pre- trained network, inspired by the cross-attention module of Perceiver [32]. However, we train multiple adapters in parallel and avoid the iterative re-sampling architecture present in their work. More generally, cross-attention layers have been vital in many existing object detection and multi-modal systems that fuse activations [44, 8], or apply tross-attention with learnable latents [3, 32, 77, 9, 80].
Learning with intermediate features The re-use of intermediate representations in deep learning is vast and spans from works interpretability [78], to state of the art approaches in object detection that harness intermediate layers for multi-resolution feature pyramids [47, 68, 20] and segmentation [23, 29]. For ConvNets, the work of [2] studies classification utilizing intermediate network representations and the authors observe a decrease in accuracy when probing earlier layers.
3 Method
We introduce InCA, lightweight and modular transfer learning alternative to full fine-tuning, that avoids backpropagation through the base-model. Let f(x) = In 9n-1 ..... 91 (z) be a pre- trained forw-forward neural network of n layers, with gj corresponding to the j-th layer of the network. We denote the activation computed by 9j as fz()) 9j o o network inference, a "forward" computation processes and computes each fz( to the network's 's final prediction (z)) During standard training, all of the intermediate activations {f(()), fn- (Ir), J }x)) are held in GPU memory and are used to compute gradients to update the model. For large models, this incurs large computational and GPU memory costs [63] which limits using the best and largest available pre-trained models under typical computation budgets.
... ... (z)) During
activation to arrive
Instead, we attach a set of isolated "models" to the pre-trained model f at selected activations fix and pass them as input to set of lightweight and shallow networks hy (a) with separate parameters and losses. With this, we can train a set of heterogeneous adapters h2 (a) in parallel, while computing inference of the pre-trained model f only once during each update (see Fig. 2). For set of adapters he (a) that take as input intermediate activations from (fzx) training follows as:
1. Single inference of f through data batch which computes (Ix) and selected activations (S. (x)).
TWe use callback (though Torch's register_ wowward () or TensorFlow's get get_layer() () output) to cache the activations of the relevant layers which become leafs of the computational graph

---

2. Calculate the batch predictions and losses for each adapter h2, lu = Ehp (f. (e))).)). v).
3. Computing bg = Σ lx and applying automatic differentiation then efficiently resolves the gradient and updates of each hi automatically desired.
By avoiding backpropagation through the pre-trained f we decouple the majority of the training costs from depending on the size of the base model f and instead the costs correlate with the much smaller adapter set J. Below we demonstrate that even a simple cross-attention module for hıg makes the overall adaptation sufficiently expressive yet highly efficient.
InCA adapter After extraction of the layer representation fx(x), we have access a high- dimensional activation map at our disposal. To predict target label from the high-dimensional fz),,, the typical approach to apply dimension reduction such averaging (avgpool) or computing maximum values over subset of the dimensions and then applying a linear classification head:
i = head o avg-pool (m))..
Nonetheless, this simple aggregation approach leads to loss of information which we observe empirically in Sec. 4 and theoretically analyze in Appendix D. Instead, we use a cross-attention module to intelligently aggregate information from the entire farge-dimensional activation map f& (x) into a died-dimensional representation based on number of cross-attention queries. Specifically, for standard downstream adaptation, given an intermediate feature map z = ..... fu (x) with 7 tokens or channels we use the following adapter architecture
Ucross (z) [1mm] == cross-attno(("".... ,.... ""..tt, 2²",, la1.-.., 9ml)
InCAs(z) == headg o norm (orrgpooo((urg Ueross (=mm))))
Note that the query tokens [91.... qml are optimized along with A. The multi-head cross-attention layer outputs Eeuss is produced by surveying the feature map (x) with the query tokens [91.... 9m] Then, the classification output ÿ InCAs(z) is obtained through averaging the cross- tross-attention out- puts (if m 1) followed by fully fully-connected classification head after normalizing with LayerNorm [4]. Based on our experiments, using a single query token (m = 1) achieves strong performance and is computationally efficient and report results with = 1 unless otherwise stated.
For more flexible inference such as the settings continual and class-incremental learning tasks, we present modular version of InCA that disentangles the representations learned between different classes, which we refer to as "Open-InCA", For a c-way classification task, define separate queries [y1.... Ue] for each class to compute representations separately.
L (z).... Femes (z) cross-attno ..... 271, 141 , qel)
Open-InCAg(2) :- diag-head, o norm (1 (z), V (z)])
Above, diag-head, is linear operator layer that operates on matrix input [ai. ........aa ao] "diagonally". Given a weight parameter W, the operator is defined as the column-wise dot product.
diag-hoad ((anadd((a.. (lai...... a..) = [ a.)....................... (Wor es (W).
Open-InCA composition In the Open-InCA adapter architecture, unique queries (q....., qe] are defined for each class along with diag-head, that independently processes each coordinate prediction. Both diag-head, LayerNorm and the cross-attn module in Open-InCA operate on each input di independently which separates the representation learned for each class and enables isolating each adapter output coordinate as
Open-InCA(2): = (W,, norm(cross-attnn((2 [[",]"""",,",],,,, "". (q:)))).
Above, W; corresponds to the i-th column of diag-head weight. As result Open-InCA enables class-level modularity with the capabilities of new class insertion, deletion and isolated class updates without regression. For example, deleting class from the Open-InCA architecture amounts to simply dropping the query and head parameters and W. for that coordinate. In the setting of class-incremental learning (CIL) different query-head pairs from Open-InCA can be combined together, as long the parameters of the norm and cross-alin remain the same. In practice, this leads to the notion of training Open-InCA with fixed norm and cross-attention weight parameters, in what we refer to as squery-only-rrainnng In guery-only-trainnng the learning of a new class
5

---

corresponds to learning just 2, d dimensional parameters per-class and adapter, where d is the token dimension. Nonetheless, when using pre- Open-InCA layer parameters, gquery-onl--traninng performs within the accuracy of InCA on many datasets. In Appendix A we compare results of TnCA, Open-InCA and query-only-training we observe that even learning just the query and head parameters is capable of harnessing the large dimensional representation maps (x()..
in class-incremental learning (CII.). In Tab. 6 of the Appendix
Layer branching candidate selection The cross-attention adapters can be applied in parallel over any intermediate layer of the network and we observe that the performance of many tasks hinges identifying the right intermediate layer to use for that task. When considering intermediate activations fi (z), observe that
Using activations such that (z) is directly computed from residual connection yields better adapter accuracy. This reflects that network representations are refined through each residual block.
The middle and later layers of the network provide stronger input representations for the adapter. This is likely since the representations of the early layers do not have discriminative enough features be used directly for high-level tasks.
Two-Stage training In settings where the base-model prooradpooppgation during InCA training is too constraining, one may conduct training in two stages. In the first stage, save the activations that serve the input for the adapter for the entire training set, by running the base-model inference for a single epoch. After saving, the second stage proceeds by training the adapters for ''' epochs with loaded activations. Suppose the per-epoch cost of the pre-trained model pewaardpropaggtion is Cpr and the per-epoch cost of adapter optimization is CA. then two-stage training reduces the time of training from O(CPPT + Ca) x T) O(CA x T + CPT), where CPT > Ca. With two-stage training, we are able to reduce 30-epoch adapter training job to 30 seconds for a cached Stanf. Cars dataset (~88,000 training samples). We speculate that further optimization can reduce training costs to "real-time", enabling an array of user-interactive applications.
4 Experiments
Datasets In our experiments, we measure the capabilities of InCA on diverse set of 11 fine-grained datasets consisting of: CUB-200 [73], Aircrafts, [54]. Stanford Cars [40], Stanford Dogs [35]. Oxford Flowers 102 [56]. MIT-67 [60], Oxford Pets [59], Describable Textures (DTD) [13], European Flood [5], FGVC Herbarium [57], and EuroSAT Land Use dataset [26]. In Table 4 we explore InCA in the settings of multi-task learning and evaluate it on the ImageNet-to-Skeen benchmark that comprised
of 5 datasets: WikiArt [64], Oxford Flowers [59], Sketch [71], Stanford Cars [40], and CUB-200
[73].
Baselines For downstream transfer experiments, we compare InCA adaptation to other adaptation protocols. 1) paragon: Full fine-tuning is considered as the paragon it performs well diverse
set of datasets but incurs steep computational and parameter costs. 2) We compare InCA to other parameter efficient approaches, including 2a) LoRA [28] 2b) Visual Prompt Tuning (VPT) [34] where we apply the for-perfonning VPT approach, VPT-Deep. 2c) BitFit [6] and 2d) AdaLN [45] which is the LayerNorm analogous, AdaBN approach. Note, each approach we name in 2) requires backpropagating through the entire network's 's activations to update the learnable parameters and leads to large computational overhead compared with InCA. 3) In addition, we also compare InCA with a suite of computationally efficient approaches that avoid backbone backpropagation like InCA. These include 3a) Linear Probing (LP). 3b) Intermediate Linear Probing (In. LP) which utilizes the same training procedure as InCA but with LP classifier on the activations. 3c) MLP-3, which is a feed-forward network that consists of probing the base-model with a 3-layer feed-forward network, and 3d) Intermediate MLP-3 3 (In. MLP-3), the extension of MLP-3 to intermediate layers.
Training details In all of our results (including multi-task settings) we use the same training configuration for InCA. We only change the adapter architecture input layer to automatically match the dimension of the base-model activation map. InCA is robust to hyper--arameters and our training schedule is consistent for all runs. This amounts to 30 training epochs, AdamW optimizer [52] with 2 learning rates, and cosine annealing learning schedule; we provide the full training details

---

Top-1
Test Error, VIT-/116
(last)
[34]
[28]
[[S]
(6)
CLBB2000
9.1
8,7
9.4
16.2
16.2
13.9
13.9
10A
12.7
13.6
15.4
DTD
18.2
17.2
18.4
18.9
20.6
17.4
20.1
21A4
19.4
22.2
21.9
Flood Depth
18.9
17.1
19.6
17.8
22.8
20.1
19.0
18.7
EuroAAT
Aireralls
14,9
15.6
21,9
50.6
67.
36.8
47.4
21.7
28.5
27.2
Herbarium
18.8
21.1
24.6
32.6
39,8
29.5
36.4
21
19.2
27.9
28.3
MIT-67
10.4
9.0
9.0
9.7
10.5
10.1
11.2
14.8
14,8
15.1
15.1
Oxford Flowers
0.6
0.4
0.6
1.1
7.2
Oxford
Stanf. Cars
8.1
7.7
10.2
29.2
47.2
20.8
3L44
9.2
16.0
14.7
Stan. Dogs
5.9
5.1
5.8
5.3
5,3
5.7
5.7
7,3
4.3
3.8
3.7
Mean Test Error
(0.0)
(-2.)) (-74) (-35.7)
(-68)
14.4
gop
3 Triiable puram.
100%
1.3%
1.3%
0.1%)
0.19)
2,8%
2.8%
0.8%
2,4%
0.1%
0.1%
No backbone backprop.
|
X
X
X
X
X
Table I: Fine-grained Classification Top-1 Test Error (ViT-L/16) We compare InCA full fine- tuning (Full FT) along with other adaptation approaches for downstream learning. For each method we summarize the maximum gap performance compared with the full fine-tuning paragon. In addition, we report the parameter efficiency and whether the method requires backpropagation through the pre-trained model. The minimum error over the columns excluding Full FT is presented in bold.
in Appendix F. While InCA is efficient enough to operate at larger resolutions, we use 224 image resolution unless stated otherwise. Nonetheless, InCA performance improves 384 resolution while remaining computationally competitive (see Table 2).
Transfer learning on VIT In Table 1, we demonstrate the transfer performance of InCA applied to ViT-L/16. For each dataset we train InCA and extract activations at residual layers the ViT for the last 12 blocks and output layer. For all baselines and method we use the VIT DeiT pre-training [67] and additionally report ViT-L/16 pre-training results in Appendix Table In the table, we compare InCA to full fine-tuning as well as applying InCA the last layer and observe that only InCA capable achieving good results on challenging datasets such as Aireraft, Stanf. Cars, etc. and closes the maximal gap full fine-tuning to -2.3%. The second best adaptation approach is LoRA which achieves maximum gap of -4.4% to full fine-tuning, yet at additional training costs. For single dataset
train the InCA modules with 2 learning rates in parallel which corresponds to 26 InCA modules with identical architectures attached to 13 activation maps. In this case, the total training costs InCA on single dataset correspond to one base-model run. Appendix C, report the hyper-parameter settings and training cost of InCA and current state of the art adaptation method for transformers, VPT [34] which incurs up 8.7x the training costs of InCA with large search (2 vs. 24 settings).
Transfer learning on SWIN In Table 3 we present downstream adaptation results for the SWIN-L pre-trained model. InCA adaptation applied to the 3rd and 4th stages of the network residual activations. Because of the heterogeneous activation dimensions of the hierarchical SWIN architecture,
Mean Top-
Test Ertor (Max. full FT)
Category
Architecture
Pretraining data
PTll
InCA
InCA (las))
inter. 1.P
I.P
Model size
Vanilla
VIT-R/16 [18]
In21K
13.0
(-166) (-7.6))))
23.9.-12.4)
24.3 (-324)
36.5M
[1]]
(CCI-M)
13.8
(4.())3))
VIT-L/16 [67]
b21K (DeiT)
10.0
(-2.5)
(175
(-357)
21.9 (-52.5)
ViT-L/16 8:384 [67]
I22K (DeiT)
9.2
(-9.1-
17.3 (-38.11)
22.0 (-54,4
301.M
CLIP-NIT-L/14633
400M Text
10.6
19.6
21.8
304-2M
[18)
2B Im- Text
9.4
10.4
14.0
15.2
632
VIT-G//44 [51]
28 Ims-Txxt
9.6
10.4
1533
168
1834.9M
Hier.
SWIN-L [SS]
(-13
196.5M
Convolatinal Convolational
ContNext-B [19]
1221
9.1
(-7.4)
(-15
19.1
(-112)
(-M.22
88. 5M
ResNext-100 101 [74]
IG-3.5B 155]
11.4
(-8.7)
(-27
20.1
(-38.8)
2-33
468.5M
Table 2: Mean Top-1 Test Error for transfer learning with variety VIT, SWIN, and convolutional networks, including different network scales and pre-training strategies. Averages reported on the 11 datasets presented in Table 1.f indicates Full FT was avoided due to prohibitive computational costs. For DeiT ViT-L/16 @ 384 the gap is computed with respect to the 224 pre-training.
7

---

the reported adaptation model sizes depend on the activation map used for the selected adapter. InCA achieves the smallest maximum gap full fine-tuning on SWIN while being computationally efficient. As with ViT-I,, in SWIN we observe that challenging datasets require using intermediate activation maps for InCA, closing the maximal gap from (-9.5%) to (-3.6%).
Evaluating InCA on different pre-trained models InCA can be applied to any feed-forward neural network without any implementation changes. We simply specify the intermediate layer names and feature tensor-ordering for any new architecture and InCA can be used directly. We note this is in sharp contrast methods that rely specific layers such as convolution filters [7, 55] self-attention [28, 34, 43]. We illustrate the architecture versatility our method Table 2. We report the mean and maximum test gap from full fine-tuning the 11 fine fine-graine grained dataset suite studied in Table 1. We test different architecture families, which include vanilla vision transformers i.e. ViTss, SWIN [48], and modern convolutional networks (ConvNext [49], ResNext [74]). In addition, test models pre-trained via different strategies including supervised learning [18, 67] and vision-language objectives [44, 61, 31]. We also test InCA at different ViT scales from VIT/B-16 16 (86M) to ViT/G- 14 (1.8B). For InCA adaptation, all model sizes were trained on single V100 GPU with batch size 32, including for the larger input resolutions runs.
Multi-fask Experiments InCA's s isolated design is suitable for multi-task inference and a single pre-trained-man model can efficiently evaluate a batch of samples on multiple tasks, allowing for "one-to- many" inference. We compare InCA on the ImageNet-to-Sketee multi-task benchmark in Table 4. All methods except BA² were trained with a ViT-L/16 model and evaluated with the Beggeeettt--Sktte version of each dataset [55]. For BA² 2 [7]. we report the adaptation on a ResNet-50 [25] backbone, as the BA approach requires convolutional filters. Overall, InCA is the top performing method reaching near the paragon on the evaluated datasets. Importantly for multi-task, only InCA and LP enable multi-task inference via "computation sharing" of the base model inference.
Learning efficiency Isolating the learning from the base-model means InCA learns shallow neural networks directly on downstream task. By avoiding deeply backpropagated gradients through the base model, the adapters receive direct signal which improves the optimization dynamics and speed of training. We compare the number training steps required to train InCA and VPT-Deep and observe that InCA be optimized in 4.5x fewer epochs than VPT. Here don't take into account the additional GPU memory costs of optimizing VPT in each step, nor the required hyper-parameter sweeps used in VPT. More detailed efficiency comparison results given in Appendix C.
Top-1 Error,
Datasct
Full FT]
InCA
InCA (last)
In. LP
LP
In. MLP-3
MLP-3
LoRA[28]
AdaL.N(45]
BitFit[s]
CUB-200
9.6
10.2
10.6
DTD
15.6
17.8
19.1
17.7
19.1
16.7
16.7
15.8
16.7
17.0
Flood Depth
EuroSAT
0.7
1.5
2.4
2.7
3.7
1.6
2.2
0.9
1.1
1.7
Aireralts
12.2
15.8
25.3
43.5
52.7
33.7
34.8
16.1
22.7
26.5
Herbarium
14.9
18.2
23.0
29.2
34.0
24.9
27.6
18.4
21.2
29.7
MIT-67
10.5
10.1
10.1
9.9
10.3
10.2
10.2
9.6
8.9
8.5
Flowers
044
Oxford Pets
4.6
4.7
5.5
5.0
5.5
5.5
5.5
5.2
4,8
4.
Stanf. Cars
7.3
15.0
29.2
39.0
14,2
18.4
Stanf. Dogs
9.1
8.1
8.1
7.1
7.1
9.8
9.8
11.3
9.1
9.0
Top-1 Test Error
(Max, gap to Full FT)
9.3 (0)
(-366) 12.4 (-9.5) 15.8 (-313) 18.3(-40.5) 5) 13.8 (-21.) 14.7 (-22.6) 10.4 (-3.9) 11.4 (-10.5) (-10.8114.8)
Trainable param.
No backbone backprop.
X
x
Table 3: Fine-grained Classification Top-1 Test Error (SWIN-L) We compare InCA to full fine- tuning (Full FT) along with other adaptation approaches for downstream learning. For each method we summarize the maximum gap in performance compared with the full fine-tuning paragon. In addition we report the parameter efficiency and whether the method requires backpropagation through the pre-trained model. For SWIN-L different activations map sizes leads to different % of trainable parameters and we report the maximum for each method. The minimum error over the columns excluding Full FT is in bold.

---

Top-1 Test Error
Adaptation Efficiency
of trainable purameters
GPU Memory (training)
Inference Time (for all tusks)
Method
Avg.
Flowers
WikiArt
Sketch
Cars
CUB-200
Full fine-tuning
10.5
0.6
14.7
14.4
10.8
12.2
5x
1x
5x
Linear probing
29.8
10.9
37.2
29.3
44,5
27.9
0.01x
0.17x
1.0lx
BA [7]
15.9
4,3
27.7
20.7
7.9
18.8
1.03x
1x
5x
TAPS [70]
10.4
0.6
15.8
14,0
10.4
4.12x
1.23x
5x
SpotTune [22]
14.3
3.7
24.2
19.8
7.6
16.0
6,27x
2x
7.3x
InCA
9.8
0,3
15.4
16.8
7.7
8.8
0.06x
0.51x
1.13x
Table 4: Multitask Efficiency and Top-1 Test Error on "ImageNet-to-Skern Skeooh"kech" benchmark. InCA is the top performing method on average and is parameter efficient. Further, only InCA and linear probing "share computation" of the pre-trained model and enable "one-to-many" inference execution measured in the "Inference Time" column. BA? is based ResNet-50 and can not be applied to ViTs. The rest of the methods are based on ViT-L/16.
Analysis
We analyze the results of InCA adaptation, focusing on the performance signature of different intermediate representations used as input for the adapter and the relation between the top InCA layers with fine-tuning. Further in Appendix D, we provide a theoretical proof motivating the extraction capabilities of cross-attention as it is used in InCA.
Intermediate representations We consider the intermediate representation signature created by evaluating the accuracy of adapters that utilize different layers. In Figure 3, we review the adapter performance applied to different layer representations. Datasets like CUB-200 and Flood-Depth mostly prefer final representations, whereas for datasets like Aircrafts and Stanf. Cars, the best adaptations use earlier representations with decreasing performance towards the last activations. Curiously, we observe consistency in layer affinity for certain datasets while using different pre- trainings for the backbone and even when using different architectures (Appendix Fig. 5).
InCA and partial-tuning In Appendix B we compare InCA with gradually un-freezing the base- model and applying partial fine-tuning we fine-tune pre-trained model starting at different freezing points, this means we optimize all layers of the network after the freezing point location. For each dataset we construct "partial tuning curve" where we plot the final test accuracy vs. freezing point (Figure 4). Interestingly, we observe direct correlation between the layer-location of the top InCA adapter and the point where the partial tuning curve saturates. In particular, the partial tuning test accuracy plateaus (to the tuning of more
growing set of layers. We a of experiments where
Relative test error Improvement (%)
150
50
Aircrafts
-100
Stani. Cars CUB.200
Flood Depth
-150
Ren. Block 11
Res. Block 12
Rea. Block 13
Res, Block 24
Res. Block 25
Res, Block 16
Res. Block 17
Res. Block 18
Rea. Black 19
Res. Block 20
Rea. Block 21
Res, Block 22
Res. Block 23
Output
Figure 3: InCA Layer Performance Signature Relative test enror improvement of InCA adapters attached different intermediate layers. We evaluate InCA with ViT-L/16 with adapters each residual block starting from Block 11.

---

97.5
Relative Accuracy (%)
95.0
92.5
90.0
87.5
DTD
CUB-200
Herbarium
82.5
Aircrafts
MIT-67
80.0
Block
Block 10
Slack
Black
Bookk
Block Back
Back 16
Block
Bizz Block 18
Back 2
Back 22
Slack 23
Output
1
20
Freeze point
Figure 4: Partial Fine- fine-tuuing vs. InCA Vertical dashed lines indicate the top InCA layer, curves show final test accuracy for different partial tuning training runs. Each mark indicates of the pre-trained model parameters are trained up to a "freeze point" in the network's s layers. Note partial tuning performance saturates in close proximity the optimal InCA adapter layer. This is aligned with our hypothesis that full fine-tuning attempts to surface existing representations already in the network. In that case, performance improves until the tuning approach unlocks the capacity to utilize an existing relevant representation and performance plateaus afterwards. Note here we refer output layers, e.g,, the adapter block 19 means the adapter corresponding the final output of block 19, or the input to block 20.
where all
layers) at around the same layer location as the top performing InCA adapter layer location. Namely, the point of saturation of the partial tuning curve where partial-tuning is capable of harnessing the representation found by InCA at that layer. This gives further evidence that "your representations are in the network" and that fine tne-tuning surfaces existing representations that can be directly identified by InCA. However, InCA adaptation operates an order of magnitude more efficiently and scales better to large models.
Discussion
In this paper, we present an efficient and effective alternative to full fine-tuning for transfer learning. closing the gap full fine-tuning on diverse set of downstream datasets. InCA has many benefits: inherently generalizes to different architectures, efficiently scales to massive models, optimizes effec- tively, and unlocks modular and flexible adaptation applications including multi-task and incremental learning. Further, through the parallel exhaustive search of InCA we are able to better understand the inner representation dynamics of neural networks and construct illuminating "representation signatures" of different models and datasets.
References
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji. Charless C. Fowikes, Stefano Soatto, and Pictro Perona. Task2vee: Task embedding for meta-learning In Proceedings of the JEFF/CVF International Conference on Computer Vision (ICCV), October 2019. 3,4
[2] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017. Workshop Track Proceedings. OpenReview.net, 2017. 4
[3] Jean- San-Bappiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals,
10

---

Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint wew.w66070640000 2016. 5
[5] Björn Barz, Kai Schröter, Moritz Münch, Bin Yang, Andrea Unger, Doris Dransch, and Joachim Denzler. Enhancing flood impact analysis using interactive retrieval of social media images. Archives Data Science, Series (Online First), 5(1):A06, 21 S. online, 2018.
[6] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficien fine-tuning transformer-basee masked Linguage-models,
Pruceedings of the 60th Annual Meeting of the Asxociation for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland, May
2022. Association for Computational Linguistics. 4, 6, 8, 13
[7] Rodrigo Ferreira Berriel, Stéphane Lathuilière, Moin Nabi, Tassilo Klein, Thiago SliveiraSaantos Nicu Sebe. and Elisa Ricci. Budget-aware adapters for multi domain learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Scoul, Korea (South). October 27 November 2, 2019, pages 382-391. IEEE, 2019. 8,
[8] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katic Millican, George Bm Van Den Driessche, Jean- -an-tBttiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang. Loren Maggiore, Chris Jones, Alhin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Frich Flsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Rexearch, pages 2206-2240, PMLR, 17-23 Jul 2022. 4
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 219-229. Springer, 2020. 4
[10] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Leurning Representations, 2019. 1.2
[11] Ting Chen, Simon Korblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pages 1597-1607. PMLR, 2020. 4
[12] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cha, and Yoshua Bengio. Attention- based models for speech recognition. Advances neural information processing systems, 28, 2015.
[13] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2014. 6
[14] John D Cook. Upper and lower bounds for the normal distribution function. John D Cook's Blog. 11
[15] Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. linearized framework and new benchmark for model selection for fine-tuning. arXiv preprint arXiv:2102.cm 000,000044 2021. 4
[16] Chaitanya Devaguptapu, Samarth Sinha, K Joseph, Vineeth N Balasubramanian, and Animesh Garg. 6-patching: A framework for rapid adaptation of pre-trained convolutional networks without base perfor- mance loss, 2023. 4
[17] Jacob Devlin, Ming-Wes Peig-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019.
[18] Alexey Dosovitskiy. Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 3,7,8

---

[19] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[20] Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7036-7045, 2019, 4
[21] Raphael Gontijo-Lopss. Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations, 2022.
4, 11
[22] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4805-4814, 2019. 4,9
[23] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Hypercolumns for object segmen- tation and fine-grained localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 447 447-456, 2015. 4
[24] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatten and Graham Neubig. Towards a unified view of parameter-cfficient transfer learning. In International Conference on Learning Representations, 2022. 4, 1
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, IEEE Computer Society, 2016. 8
[26] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep lcarning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. 6
[27] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Atariyan, and Sylvain Gelly. Parameter-efficient transfer learning for alp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019. 3,
[28] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022. 3, 4, 6, 7, 8, 13
[29] Vladimir Iglovikov and Alexey Shvets. Ternausnet: U-net with vggll encoder pre-trained on imagenet for image segmentation. arXiv preprint arXiv: /801.5560 2018. 4
[30] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannanch Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights,
2022. 4
[31] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. 7,8
[32] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference machine learning, pages 4651- 4664. PMLR, 2021.4 4
[33] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Zhen Li, and Tom Ducrig. Scaling visual and vasion-language representation Icarning with noisy text supervision. In International Conference Machine Learning, pages 4904-4916 PMLR, 2021. 5
Sung.
[34] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Fumppan Conference on Computer Vision (ECCV), 2022. 3, 4, 6,
7, 8,
[35] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEFE Conference on Computer Vision and Puttern Recognition, Colorado Springs. CO. June 2011. 6
[36] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkkska et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. 114(13):3521-335 3526, 2017.
12

---

[37] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference computer vision, pages 491-507. Springer. 2020.
[38] Simon Kornblith, Jonathon Shlens, and Quoc V Le. better imagenet models transfer better? In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 2661-2677
2019. 4
[39] Zhi Kou, Kaichao You, Mingsheng Long, and Jianmin Wang. Stochastic normalization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16304-16314. Curran Associates, Inc., 2020.
[40] Jonathan Krause, Michael Stark, Deng, and Fei-Fei. object representations fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR- 13), Sydney, Australia, 2013. 6
[41] Juho Lee, Yoonho Lee. Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for blaeetion-based permutation-inyariint neural networks. In International conference on machine learning, pages 3744-3753, PMLR, 2019. 4
[42] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differen- tiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10657-10665, 2019. 4
[43] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameler-efficient prompt tuning. In Proceeding of the 2021 Conference Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. 4, 8
[44] Junnan Li. Ramprasaath Selvaraju. Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 97055 2021. 4, 8
[45] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. 3, 4,4, 7, 8, 13
International Conference on Learning Representations, ICLR 2017,
[46] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2) 2017. 1.2
[47] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017. 4
[48] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021. 3, 7,8
[49] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11966-11976, 2022. 3, 4, 7,
[50] Ilya Loshchilov and Frank Hutter. 1rXiv:1711.05101,200 2017. 12
Decoupled weight decay regularization. arXiv preprint
[51] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. 12
[52] Nly Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[53] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181- 2018. 7
[54] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aireraft. arXiv preprint arXiv:1306.5151, 2013. 6
13

---

[55] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV). pages 67-82, 2018. 8
[56] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dee 2008. 6
[57] New York Botanical Garden (NYBG). Fgve 7- herbarium 2020, 2020. 6
[58] Michal Pândy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Transferability estimation using bhattacharyya class separabilityy Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9172-9182, June 2022. 4
[59] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. 6
[60] Ariadna Quattoni and Antonio Torralha. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition, pages 413-420. IEFE, 2009, 6
[61] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR,
2021. 4, 7,8 8
[62] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 12
[63] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward
training trillion parameter models. In SC20: International Conference for High Perfornance Computing, Networking, Storage and Analysis, pages 1-16. IEFE, 2020. 4
[64] Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric the right feature. International Journal for Digital Art History, (2), Oct. 2016. 6
[65] Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Andrew Jackson. Fine-tuning image transformers using learnable memory. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12155-12164, 2022.
[66] Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang- Ting Cheng. Partial is better than all: Revisiting fine- tine-tuning strategy for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9594 9602, 2021. 3
[67] Hugo Touvron, Matthicu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXIV, page 516-533, Berlin, Heidelberg, 2022. Springer- Verlag. 7,8
[68] Cristina Vasconcelos, Vighnesh Birodkar, and Vincent Dumoulin. Proper reuse of image classification features improves object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13628-13637, 2022.
[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems,
30, 2017. 4
[70] Matthew Wallingford, Hao Li, Alessandro Achille, Avinash Ravichandran, Charless Fowlkes, Rahul Bhotika, and Stefano Soatto. Task adaptive parameter sharing for multi-task learning. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7561-7570, 2022. 4,
[71] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, pages 10506- 10518, 2019.
[72] Zifeng Wang, Zizhao Zhang, Chen- Yu Lec, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 139-149, 2022. 1,2
[73] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 2010.
14

---

[74] Saining Xie. Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings the IEEE conference computer vision and puttern recognition, pages 1492-1500, 2017. 7,7.8
[75] Li Yang, Adnan Siraj Rakin, and Deliang Fan. Rep-net: Efficient on-device learning via feature reprogram- ming. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12277-12286, 2022. 4
[76] Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. Advances Neural Information Processing Systems, 33:1 17336-1724, 2020.
[77] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. 4
[78] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pages 818 833. Springer, 2014. 4
[79] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part III 16, pages 698 714. Springer, 2020.
[80] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable [detr): Deformable transformers for end-to-end object detection, In International Conference Learning Representations, 2021. 4
15

---

Appendix
Below provide additional details and results which are not presented in the main manuscript.
A
Continual Learning with Open-InCA
With the Open-InCA adapter, each class prediction is isolated using a different dedicated query and classifier vector. For continual learning tasks, in addition to running multiple adapters in parallel as presented for multi-task results in Table 4, Open-InCA enables an even more granular composition of adapter sub-tasks. Recall the Open-InCA adapter architecture defined as
[Voross (x), (z),..., Tross (2) cross-attioo(2",......... (,),,,,.,),,,.............,,..,,
Open-InCAg(z) == diag- head, LN [[o (Z) (z).... ................................ e (z)])
with LN denoting LayerNorm. Due to the properties of each operator, each class prediction can be computed separately as
Open-InCA(2); (W,, LN(crosss-atann ([2 . (g:)))).
Because of this property we can remove a class prediction or add a new class prediction without any effects on other model predictions (as long the parameters of cross-alln and LN remain fixed). As presented in Sec. we use gguery-onl-ttraninng which trains new adapter classes while freezing cross-attn, LN and enabling compatibility between task predictions.
exp(u*)
When training with "query-only-training"" the softmax function, softmax(u) -i exp(u*)
indirectly injects information of predictions from all classes due the normalization term in the denominator, which means gradients about a particular class i will include information from other classes j. Instead, we can achieve complete training separation by using Sigmoid final activation, exp(u)
(u) exp(u) + 1 and a Binary Cross Entropy (BCE) loss that considers each prediction separately.
Clearly sotry--onl--trainnng the adapter representation capacity is reduced, since the cross- attention weights are not trained. We present experiment evaluating the performance of InCA, Open-InCA and sglery-only-traininng Open-InCA in Table 6 and observe that despite the isolated and reduced parameter set in query-only-trainnng of Open-InCA, the method still competitive and outperforms Linear Probing on most datasets.
Next we test Open-InCA for class-incremental learning, for which we consider the Split CIFAR-100 incremental learning benchmark. The Split CIFAR- 100 dataset is trained with incremental learning episodes each introducing 10 new classes. As in [10], we present the average episode accuracy and forgetting of "query-ooll-ttriiinn InCa and additional baselines.
In particular we evaluate Open-InCa using ViT-B/16 along with state of the art methods L2P [72]. LwF [46] and EWC [24]. Nonetheless, we do not apply any special routing of our learned episodic models and simply combine their predictions. In contrast 1.2P is prompt based approach that, during inference, passes each new sample to an auxiliary classifier to predict corresponding episode (in this case a 10-way classifier) and the corresponding episode model is up-weighted according to the prediction. We believe that with such an auxiliary classifier Open-InCA performance can significantly improve, nonetheless we observe that Open-InCa can simply leverage a larger model efficiently to achieve state of the art accuracy. We leave routing of samples to different learned sub-models as an interesting avenue for future work.
In addition, Open-InCa InCA has additional benefits compared to typical class-incremental learning approaches:
Flexible incrementation With Ipen different episodes can naturally contain vari- able number of classes and episodes can be further decomposed needed. This is since one can modify the model at the granularity of a single-class predictor via the Open-InCA adapter architecture by introducing or removing additional and W..
Reduced forgetting risk With Open the ability of adding new classes without for- getting is built-in into the architecture, prediction of different classes ensures that the

---

previous class predictions remain the same (i.e., no logit regression) which reduces catas- trophic forgetting.
Parameter and computation efficient The Open-InCA adapter benefits from the InCA approach, which is parameter efficient and computationally efficient during inference (see Table 4) well during training (see Fig. 6 for comparison with prompts).
Method
Average Accuracy (t)
Forgetting
Top-1 Test Error
Dataset
InCA
Open-InCA
Query only Open-InCAA
In. LP
I.P-sqqentiial"
Full-FT-sequentaal
33,6
86.9
CUB-200
9.1
9.5
12.1
16.2
EWC [36]
47.0
33.3
DID
17.8
17.1
19.2
LwF [46]
27.8
Aircrafts
15,8
18.1
38.6
50.6
1.2P [72]
83.8
7.6
M11-67
10.1
9.1
Oxford Flowers
0.3
0.4
0.4
0.6
Open-InCA (ViT-B/16)
83.0
9.1
Oxford Pets
4.7
5.4
Open-InCA (VIT-1/166)
Stanf, Cars
8.4
8.4
22.8
29.2
Open-InCA (VIT-H/14)
86.1
8.2
Stanf. Dogs
8.1
5.3
Table 5: CIFAR-100 Class-Incremental Learning Split CIFAR-100 is trained with 10 episodes of 10 classes in the standard CIL evaluation suite [72]. Average accuracy and forgetting is reported over the 10 episodes according with [10]. "Sequential fine-tuning results are taken from [72].
Average
9.3
Table 6: Open-InCA adapter performance We compare InCA, Open-InCA, "query- training" Open-InCA and Intermediate Linear Probing (In. LP). We observe that Open-InCA is comparable with InCA and that "query- training" significantly out-performs In. LP.
B
Intermediate Representation Signatures
The parallel training of InCA results in the synthesis of tens of models that can run inference with insignificant per-adapter marginal costs. As result we have the ability to glean highly useful information about the network's 's different representations and study the network inner representations effectively. This is especially important for recent non-convolutional based architectures that do not have as many inductive biases explaining some of their behavior. In this section we present results showing information retrieve from the performance of InCA adapters.
100.0
97.5
Relative Accuracy (%)
95.0
90.0
87.5
DTD
CUB-200
85.0
Herbarium
Aircrafts
MIT-67
80.0
Black 9
Black 10
Block
Black
Block 13
Block
Block 15
Blcck
elock
Black
Block
0 20
Block
Blcck 22
Black 23
Cutpus
Freeze point
Figure 4: (repeated) Partial fine-tuning InCA Vertical dashed lines indicate the top InCA layer: curves show final test accuracy for different partial tuming training runs. Each mark indicates run where all of the pre-trained model parameters are trained up to "freeze point" in the network's layers. Note partial tuning performance saturates in close proximity to the optimal InCA adapter layer. This is aligned with our hypothesis that full fine-tuning attempts surface existing representations already in the network. In that case, performance improves until the tuning approach unlocks the capacity to utilize existing relevant representation and performance plateaus afterwards.

---

B.1 Partial fine-tuning and adapter performance
Below we present in detail the experiments discussed in Sec. 5, in particular regarding partial fine-tuning and InCA. The experiments illustrate the relationship between InCA adapters at different layers with partial tuning. We tune the pre-trained model starting from different "freezing points".. In particular for neural network faz 91 .... 9t, for each freezing point Im consider its position m and all of the preceding layers and apply layer freezing to 91. 9m-1 (i.e. not updating gradients for those layers). Back- pock-propagation is then only applied for optimizing Im..... including the network's s prediction "head". In Figure 4 we show the dynamics of partial tuning, where we optimize the pre-trained network (ViT-L/16) in different runs with each having different freezing point. We compile the final Top-1 Test accuracy of each freezing run to create a partial tuning "curve" for single dataset. We compare the partial tuning performance curve of each dataset with the corresponding top layer of the InCA adapter trained on that dataset and observe that they are highly aligned, with datasets that prefer later InCA layers plateauing in their test accuracy earlier (at later freezing point). In particular what we observe is that partial tuning performance plateaus roughly at the same layer where InCA identifies the top adapter representation. This is also the point which partial fine-tuning is capable of harnessing that representation for the downstream task. Overall this gives further evidence that "your representations are in the network" " and fine-tuning simply surfaces existing representations that are already identified by InCA. When drawing the vertical lines of the top InCA adapters, we refer to output layers, e.g., the adapter at block means the adapter corresponding to the final output of block 19, or the first input of block 20.
B.2 Task Layer Affinities
In InCA we select oo--erformmnng adapters that "listen" to different intermediate representations of a neural network. In work observe that one is able to achieve strong and diverse transfer learning by utilizing intermediate representations, and that for challenging tasks it is often required to use intermediate representations to achieve top results. Indeed the best representation layer for an adapter tends to be highly robust to hyper-paramedd variables of the optimization. Even more intriguingly, we find that this representation affinity is preserved across different pre-training and even architectures. This is, certain tasks have strong "affinity" to certain range of representation layers even for different architectural cireumstances. The majority of the architectures we consider have some pre-trained component on one of the ImageNet datasets (aside from the CLIP ViT-L/14 model). At the same time, the fact that different architectures give rise similarly helpful representations gives strong clues about the effect of different architectures as compared with the pre- pre-training task during learning over a large diverse dataset such as ImageNet.
22
DeT
DNI-
VI-
VI-
Models
Dei 344
042
-2a
Models
Def-:384
CUP
SWIN
-s0
SWIN-1
50000
Rel. pos.
Rel. pos.
DET
ViLL
Models
Dei1-384
SWIN-L-
Con-ftect-
03 20 85.4 ao taltaop Rel. pos. 9
Figure Best-performing representation for InCA adapter for Aireraft (Top-left) MIT-67 (Top-right) and Stanf. Cars (Bottom).
3

---

In detail, we look at the best- pest-perfonnin InCA adapter for fixed task on different architectures. The pre-trained models we consider consist of 2 different pre- tre-trainings of the ViT-L/16 architecture (ViT- original and DeiT), the 384-resolution pre-training of DeiTT with the resolution adjusted ViT-L/16, CLIP's ViT-L/14 architecture, the SWIN-L. architecture, and the convolutional based ConvNext- architecture. All of the vanilla ViTs we consider each have 24 residual transformer blocks so that comparing between blocks is directly aligned. SWIN-L and ConvNext follow the "Stage" breakdown of blocks, namely SWIN-L has (2.2,18,2) stage breakdown that conveniently also adds up to 24 blocks (hence aligned in the figure) and lastly, ConvNext follows a (3-3-26-3) + head stage block composition, which we rescale in the figures to fit on the same 24 block range. In addition in the plots we also present the test error gap of each architecture with using the InCA adapter applied on its final block representation. Tasks that prefer earlier layers such Stanf. Cars and Aircraft have large gap from the performance of the last layer representation adaptation and such later layers lead to sub-optimal results.
We remark that the work of InCA sheds light on the inner representations learned by neural networks showing in some aspects performance is invariant to the architecture and more based on the pre- training dataset. We leave this topic for further research and find it to be an intriguing topic of study.
C Efficiency Results
InCA is highly efficient especially for large models, which is based on the isolated adapter architecture that does not modify the backbone. We delineate the efficiency aspects follows:
Training memory efficiency The use of a frozen pre- pre-trained model makes the training much more efficient and scalable since not all of the intermediate computations need to be stored as done in standard training or as required by methods that compute gradient information using inner-layers of the network. As soon any intermediate layer requires gradient, all subsequent activations must be held in GPU memory after the forward pass. This means methods like LoRA, FitBit and VPT all require storing of all of the activation maps for all of the layer operations in the network since they update parameters based on gradients from the very early layers in the network.
Fast optimization Unlike typical parameter efficient methods that insert some form of train- able parameters in the network, InCA adapters are trained with "direct gradient" information coming from an isolated loss. Essentially each adapter corresponds to a very shallow neural network trained direetly via back packkpropagation. This makes the training dynamics fast as direct gradient information about the loss easily reaches all of the adapter parameters. On the other hand, update inserted parameters in the backbone, the gradient information is indirect and needs to be back-propagater through the backbone, with the risk of information loss and making the optimization more challenging, we and the authors [34] observe regarding prompt tuning.
Efficient multi-task inference As present in Table 4 the unchanged backbone execution enables efficient and parallel inference efficiency as multiple tasks can be evaluated at once.
C.1 Computational Efficiency of InCA Compared with VPT
In Table 7 we observe that InCA is an order of magnitude more efficient train than VPT. For the results in the table we consider the VPT-Deep adaptation method, that is trained with 50 prompt tokens in each layer. We report calculated training times in GPU-hours of a standard Nvidia-T4 GPU using a VIT-L/16 architecture and accuracy numbers based on the datasets of Table I with the DeiT pre-training. For larger architectures such as ViT-H/14 ('VIT Huge") the difference training time is even more striking, as InCA maintains good per-run training time of 2.5, VPT-Deep requires staggering 55.8 GPU-hours per-run for a single GPU. On ViT-H/14 this exacerbated as we must reduce the batch-size of VPT significantly to fit training on common-place single GPU (Nvidia-T4). We measure in terms of training InCA and VPT-Deep for the same number of epochs. This however, is inaccurate as InCA trains order of magnitude faster a per epoch basis (see Figure 6).

---

Max. Full-FT
Training time
Train time per datasct (GPU hrs.)
Mcan Test Err.
# Hparam. per dataset
Method
per run (GPU hrs.)
gap
InCA
10.2
2,4
2.0
2 (purallel)
4.0 (2.4")
VPT Deep [34]
12.3
6.8
5,8
24
139.6
Table Computation costs of adaptation We adapt ViT-L/16 to CUB-200 downstream classifica-
tion with the same number of training epochs. We evaluate the training and computational costs of single run and training VPT-Deep and InCA for one training dataset. "Training with learning rates in parallel leads to training time decreasing from 4.0 to 2.4 GPU-hours.
We attribute the difference training time of InCA to:
1. InCA does not require back- proppprppggttion through the whole model which gives ~ 50% speed improvement alone.
2. InCA is robust to hyper-paramene parameters and we optimize it using just 2 learning rates, compared with the hyper-parameter set of VPT (in our experiments use 24 hyper-parameter configurations per dataset while using the full configuration presented in [33] takes even longer). In addition, with "one-to-many" training, we train the two hyperparameters of InCA in parallel and report the specific training time in Table 7 denoted by (*) for parallel super-paramete training.
3. InCA does not increase the number of propagated tokens in the transformer (e.g. in VPT with 100 propagated tokens the attention matrix doubles, from ~ x 10 to ~ 9 x 10 entries).
C.2 Optimization dynamics of InCA and VPT
In Fig. 6 we conduct experiment where we train InCA and state of the art prompting method VPT-Deep [34] for different numbers of epochs and report the final test accuracy. We observe that InCA trains order of magnitude faster than prompting and reaches within 95% relative test accuracy after 3 training epochs.
100
Relative Accuracy (%)
90
70
60
InCA
VPT Deep
50
35 Number of epochs in run
10
20
30
40 50
Figure 6: Optimization Speed for training InCA and prompt tuning (VPT-Deep) on the Airerafts dataset. We train each method until completion with varying numbers of epochs and report the relative final test accuracy to epoch training. The shallow adapter architecture and direct gradient signal in InCA makes the training of the adapter order of magnitude faster (in terms of gradient updates) than prompt tuning approaches. Both methods use batch-size 32 and take the same number of gradient steps in each corresponding run, under the optimal learning rate.
5

---

D Theoretical Analysis
Empirically, we consistently observe that cross-attention as opposed to a linear or MLP-3 architecture enables InCA to better harness the existing model. We present theoretical result asserting that using the coossatttnttonn layer for aggregation as opposed to linear averaging, or even full-concatenation followed by large dimensional linear layer is capable of learning over strictly broader set of data distributions.
We give the precise statement in Theorem D.I and intuitively argue that cross-attention with learned queries has the ability to sift through irrelevant pieces of the representation that may be variable positions in different data samples.
Recall in the settings considered thus far, the extracted activation of an image data-point can be ...],
viewed as x; RRxx or T tokens e.g. X; [xc] ...... with sr E Rdd. We argue that in many scenarios, task-pertinent information a property of individual tokens (e.g. x;)) within a data-point x; and not a property of the overall feature map. We present the theorem below. To this end we define a Booken-Separabllig (TS) notion of a dataset.
(Xn, yni} with x, = [o], .x," E RdxT and Yi E 1- is said to be linearly-token sepparsseaaaable if there exists a scalar с > 0 and w E Rd satisfying ||l|| = 1, such that for each data point (xi, yo) E D there exists a token x²," X2 with
Definition 1 (Token-separable Dataset). A dataset D =
(1) We define (WD, bp) and co as the maximum margin solution and maximum margin respectively, i.e. co (aaeaaa/alllleeele mindty(((s) (x:((r wd) b)} for DD with (WD, bd) corresponding the selected co-
u.(( w) + c.
Intuitively, D is a TS-dataset if each of its data points contain a token that leads to linear separability (the same shared among all points x; D ). One can further distinguish between aligned-TS Datasets, where the index J: of the linearly separating token is consistent among the n data points, or permutable-TS where j dependent on Further TS datasets can be generalized to k-token separable datasets where k tokens are responsible for separability in each x,, for this theoretical contribution we don't 't make an assumption on whether the dataset aligned or permuted, but consider the setting provided by Definition (i.e. not the k-separable generalization). We present an analytical statement for the advantage of cross-attn, the theorem is provided for binary classification via a scalar prediction, but can conventionally extend to C'claass classification. For binary classification define a prediction via the standard scalar binary aggregator as a (u) = sign (2; u;) that converts a vector into binary prediction.
Theorem D.1. Let D be a binary-class, token-separable dataset with max-margin cD and max- margin solution (WD, bp) consisting of data points. Suppose that D is distributed such that for X; = (x).... a!²) with (Xi, Us) E DD are normalized for separating token x!; E D and that the rest of the tokens correspond to "noise", xt ~ (O, T d/d.
B| = 1. Furthermore assume
CD Z max (V 1232 [log(1 (08(1/8) log log((nT)), 2/bol).
(2)
Then there exists cross-attention classifier
f(x; q, ()),b)
Σ cross-attn(x, q)1 +
(3)
that sepurates DD with probability at least 1 - 8. In contrast, every fixed member g(x; w. b) ΣT= w + b) of the linear classifier family will fail to separate 'D with probability at least V2= 241 exp(- /2/2) where VT-TO CD-
Vd
As stated above, the failure probability of the simple linear classifier g depends on s which satisfies 8 /d/T. T. For existing architectures d, T tend to have similar order of magnitudes, e.g. for VIT-B/16, d = 768, T = 196 which makes the failure probability non-negligible. Before presenting the proof, we make the following observation: in InCA, we use the same cross-attn layer with latent g, which we show can be simplified via reparameterization.

---

Observation D.2 (Query Collapse Reparameterization). A single-head cross-attention param- eterization with latent [g] is equivalent to the following simplified layer, cross-attn(x.Q) Σ softmax(g"x) C Wx with q" R
This can be derived by decomposing the attention score which is the input to softmax.
aj = (Wyg. 98 W War") (Wag) (War')
q W Wa (q") 22
Where q" - WT and q" R hence the cross-atin layer simplifies, which is used in proof.
As our proof shows, the cross-attn layer can operate on large data bandwidth, e.g.x R while still being selective in finding task specific representations. Empirically we also observe that increasing the number of heads of pross-atth also improves the performance of the InCA This is in part because it enables the learned latent query parameter to identify more useful token patterns, and since g is fixed using more heads remain stable (as opposed to when q is a data input). We now present the proof of the claim.
Proof of Theorem D.1
Proof. The proof of the theorem has parts A) the positive condition the cross-attn layer and B) the negative condition on the linear layer (a non-separability probability lower bound). We start with A) and consider separability of positive and negative data samples in turn. First simplify and write an equivalent cross-attention binary classifier expression for the cross-attn classifier.
Positive result for the cross-attention model We consider a "single-head" cross-attention layer and by Observation D.2 we can write the cross-attn layer as follows
cross-attn(xx q, (W))
Σ coftmax((s/..))) Wor..
Note that softmax is a function of the entire vector form above to illustrate the summed terms. For simplicity of notation, we drop the asterisk and write 9² Rdd as g. Combining cross-attn with the binary aggregator, we have aggregation over the output vector of the cross-attn layer.
however we write it in the
f(x:99 (.3::,
cross-attn(x;; (W)), +
Define 5(x;, LXTT the computed softmax argument,
S(x;, q) softmax((<2) g)....)................ (2f .)))).
(5)
Substituting into the classifier, we have
d
T
f(x;4, (W),b)
Sj
W Wwr?), +
1=1
j=l
T
S
a +b)
Let = (L,,(W,,)) (W.)p.) be the sum of the rows of W p. Note x can be pulled out from the inner summation to give
T f(x; q. u, =
Σ
(y + +b)
1=1
7

---

Thus the cross-attn classifier presented is equivalent to the parameterization above. Next we consider the two terms in the sum, namely Sj and (u, x²})- We will be deriving their distribution in the case where data point (x,, Vi), has prediction labels y, = and y, separately. We start with y = and consider
exp((at, 4)) S Σ exp((a2,.9)
Take je to be the separating token for sample x, By the assumption of the theorem for k + ji the tokens correspond to isotropic noise of expected squared norm 1, i... x ~ (N, (/d). For fixed E Ed with |/al|2 we take 772 to be the distribution of the dot product,
(u, x
Σ (u)) (x/)).
?i
For + ji this is a sum of independent Gaussians and each coordinate is distributed as ~ N(, ) As such we have
(u,x²) N (0. (0. 1 ut NO. 1211//d)
N (0, 1/d)
since ||u||2 1. Next we consider ls jt- By the hypothesis have that
vi(( wD) + bp) 2 CD.
With positive label (y: = 1) this gives (xF) wD) + bp 2 cp. Note that since cp 2 2|bpl we have that (x/), Wp) 2 cp/2. Since wD is the maximal margin solution, we have ||w Take q to be of the form q t- wD for t R+ and u wD, then
and op > 0.
t. 1.
= (x
Ma7 wD) 2 t. cD/2
For mkk k # jr. separating t, we have that (x 9) = t- 77e- Define M = max (lmxl). M is random variable distributed as the maximum of of - i.i.d. Gaussians distributed according to (0, 1/d). We bound M by investigating an upper bound of the Gaussian CDF. Recall that the moment generating function of Gaussian random variable X N(0 given by Mx (r) [[erX]= etr?. Then note that for any > O we have
P(X 2 r) = P(( x e")) Se M(s) = - 2.2
where the inequality an application Markov's 's inequality. Setting r this gives the tail bound
P(X r) S exp(-r²/2),
(9)
For our settings with π) (0,
P(m: 2 r) S exp(-dr2 12). For wide bound, by symmetry of the distribution we have
(10)
P(mel r) S 2 2exp(- dr /
(11)
Therefore union bound results in
P(M 2 rP
U |me rr ktji
5 P(mmol 2r)
< (T - 1). -xexpp-dr> /2)
< 2T expl- - -dr² 12).
8

---

We can bound the bulk of the distribution of M as
PMM r) 2 1 2T exp( -dr 12).
(12)
Taking r cD/4 then with probability at least 1 - 27 cxpp(-d(cp/4))/2) = 2T exp(-d(co))//0) we have
M max lTp 4 kボ
(13)
and thus
tcp
max (g. .x,) 4
(14)
ky
With high probability, Eq. (13) holds. This implies that for Ji.
exp( (I+t q))
5₁. = Σز= exp((x),
1 Ejts. expl (at 9) (1, g)))
1
+ Litt. exp((xe,,0) q) tep (2)
1
1 2jz,. expl -top/4))
1+ (T - 1) expl - tcp /4))
(T 1) exp( 1lDp/4)
1+ (T 1) exp(-tcp/4p)
Note that ep > 0 and T is fixed. Nonetheless, the probability bound is independent of t which may take arbitrarily values, e.g. for any > 0, take / = 4/cD log(7 (Te), which gives
S₁
(15)
Since S 2 0 for each l and Σi=1 Sy
we have for ls + ji:
Sk Σ SJ
Sj. S 6.
(16)
We consider the classifer prediction
T Stu, +
f(x: q, u. =
(17)
where b is bias parameter we can choose. Recall a WD. Focusing on the inside of the sign function
Σ Sy(wD. x²) Sy. (WD, x79) +
Σ
Sknk
ktj
2 Sy. (WD, 2/1)
Σ
S max(Ing))
2 (1 e)cp/2 - c-cp/4
= (1 (3/2)c)ox/2) 4
CD
provided that e < 1/3, If we take 6
so we have that for q = tw's and u = wp 4
T
f(x;q, u, =
Σ (p x/)) + = =1 yi-
(18)

---

Next we address the case where = -1. We consider the classifier prediction
,x, Σ (u, x²)) +b)
(19)
with u = wp. Again for # Ji we have that
max((u, x41) CD/4.
(20)
kgft
On the other hand for j:. yı -1 we have that
((<a/t wD) + bp) 2 cD
= (of) wD) + bD 5 CcD (1,", wD) S -cp/2 <
where we have used the hypothesis that ep 2 2/bp| in the last line. We consider the term inside the classifier. We note that
T
Σ
Su xxt <
Se (u x1)
Sk (CD)/4
$ cD/4
where in the first inequality we have used the fact that SJ. 3,,u., a4') and in the last inequality we have used the fact that Σ/= Sj = 1. Therefore again with bias term -cp/4 we have that
>
(y, xz) +b<0
(21)
and f(x; q, u, -1 14. Thus we have just shown that with probability
27 exp( (xxp(-((e))²// /²/32) the model f(x; u, b) with u WD, Q = twD, b = -CD//4 gives the cor- rect label for x, Taking the union bound over all n points in we get with probability at least
1 1 2Tn exp(-d(cp) ///0))/33))))) /32) 2 1 - 5 the model f(r; b) with = wp.q = twp, b = -cp/4
separates D.
Negative result for the linear model We consider the linear classifier
T
g(x: w, b) = σ b+
(w, x²))
(22)
j=1
where w E IRd is restricted to have unit norm ||||| = 1. For an input x; under the aggregation, the term inside the sign function simplifies
T
T
Σ
(w.z/) x/) (w,
x
We recall that the af for + ji are distributed according N 41).
Thus we have that
T-
05
N
0,
(23)
d
So the problem of classification is equivalent learning linear classifier over the separating tokens under the presence of Gaussian noise with distribution N(,, T=1). Let be the index corresponding to the input x;- with smallest margin, i.e.
argmin ((w, (w, bD).
10

---

Then we have that Vr- ((w, x/2"" + bp) S cD. We note that any linear classifier g(x; w, b) with
[|wl|, 1 will fail classify D whenever Y/- ax cp. Thus we will lower bound the probability
of P(y) a+- < -cp). Note for standard Gaussian random variable N(0 1) as shown in [14] we have for r
P(1 r)
12 exp(-r212)
(24)
Set VT-1 cD. Then by symmetry of the Gaussian distribution the above bound translates into the following bound for a.
Og- < <cp)) 2 /2 s exp(- (ep(-s2/2)) -2/2).
s
follows that for any w E I such that ||ll|22 = 1 that the linear classifier g(x: w. b) incorrectly classifies D with probability at least
V 2 S² + exp(-s² 22/2) This completes the second part of the proof.
E
Further Results
We present additional experiments below. In Subsection E.1 we present per-dataset results for additional architectures and discussion about ensembling InCA is given in Subsection E.2.
E.1 Per-dataset results for different architectures as presented in Table 2
Table 8 provides per-dataset results that are presented in aggregate in Table 2. Below we present the results for ConvNext-Basse and ViT-L/16 (original pre-training) pre-trained models (with the results for ViT-L/16 DeiT and SWIN-L presented in Table and Table respectively).
E.2 Ensembling learned adapters
Because of "one-to-many" inference of InCA can take a set of independently learned adapters and ensemble them without marginal increase to the inference cost. We follow non-parametric equal- weight ensembling, by taking the output predictions of two adapters ft (x), (2(x)) on sample image x. Note that the adapters are computed with their relevant representations via single forward pass, which makes the execution of h, (x) and hz(x)) together only incrementally higher than computing just h (x). The ensemble is defined
h (xx (1) + h2((1)
(25)
2
Given the large combinatorial selection of & adapters from the learned adapters we consider the case of ensembling with just two adapter members. After training evaluate all m(m 1)/2 such pairs and compare them with the top performing single layer predictor which we present in Figure 7. the figure, illustrate the representations and corresponding adapter pairs that lead best performance and also present the computed ensemble gain which is the difference between the ensembled model accuracy and the top accuracy of any single adapter.
In addition to improving classification accuracy, ensembling can aid in improving robustness and out of distribution performance which we leave as future work. Further directions of ensembling include ensembling performance when using adapters of different adapter architectures (c.g. an MLP-3 ensembled with an InCA adapter) adapters that use representations from different neural networks [21].
E.3 Ablation the number of queries
We apply an ablation to see the effects of using a different number of queries in the InCA adapter architecture. In partieular, the InCA adapter is written as,
Veross (z) [2:m]] cross-attng ([z), . "..,,,"), [gi.... {mm)
(z) = headg norm (avg- wowg-pool(crg (Veross (z) [2)]))

---

Top-1 Test Error for ConyNext-B
Full finc-tuning
InCA (last)
Dataset
InCA
Inter. LP
CUB-200
9.3
9.3
9.3
13.0
13.0
DTD
16.7
17.4
17.4
18.6
18.6
Flood Depth
16.9
16.5
20.5
19.4
19.9
EuroSAT
0.9
1.6
2.2
2.8
Aircrafts
10.5
23.1
54.7
54.7
Herbarium
17.0
22.7
26.4
37.4
39.5
MIT-67
10.9
10.3
10.7
10.2
10.4
Oxford Flowers
0.5
0.4
0.4
0.6
0.6
Oxford Pets
5.2
4.6
5.6
5.y
6.0
Stanf. Cars
6.8
9.3
14.4
39,9
39.9
Dogs
8.9
7.3
Ave. Top-1 Test Brror
9.4
10.7 (-7.4)
12.8 (-12.6)
19.7 (-4.2)
20.0 (-44.2)
Top-1 Test Error for ViT-L/16 (viT pre-training)
Full fine-tuning
InCA (last)
Dataset
InCA
Inter. LP
LP
CUB-200
11.7
10.9
10.9
12.2
12.2
DTD
18.3
18.9
20.1
19.9
20.1
Flood Depth
20.8
18.7
18.7
18.7
EuroSAT
0.8
2.5
3.5
Aircrafts
20.7
23.2
28.2
44.5
46.4
Herbarium
20.3
26.9
31.3
38.9
41.3
MIT-66
12.8
11.3
11.9
10.4
11.1
Oxford Flowers
0.6
0.3
0,4
0.3
0.4
Oxford Pets
5.5
Stanf. Cars
9.3
27.6
30.2
Stanf. Dogs
11.0
10.4
10.4
10.1
10.1
Ave. Top-1 Test Error
12.0
12.5 (-6.6)
(-11.0)
17.4 (-23.8)
18.2 (-25.7)
Table 8: Per-dataset Adaptation Top-1 Test Error various architectures We test transfer learning performance of fine-gratd ain-ggrined datasets applied to different architectures and pre-trainings including, ViTs, SWIN, and convolutional networks. We report the per-dataset Top-1 test for the datasets presented Table
For m > the output of tokens (q1.-. 7m] through the cross-attn layer are averaged, and we test whether using m > 1 brings additional representational benefit each adapter. We present the result in Table 9 and observe that using a different m does not have a consistent effect on the accuracy of the learned adapters, and in our experiments we m for InCA adapters be most computationally efficient.
Implementation details
We present the optimization and augmentation details for training InCA, and note standardized procedures for augmentation and training (without extensive hyper-parameter optimization) of the different transfer learning methods evaluate.
Augmentation Unless otherwise specified we train with input image size 224 and standard augmen- tation practice [62]. In particular, during training we resize to image-size 256 and apply random cropping, for testing we apply resizing and center cropping. For larger image resolutions we maintain the same resize-crop crop ratio of 0.875.
Optimization For the linear probing and InCA approaches, we train with the AdamW W optimizer [50], cosine annealing learning rate scheduler [51] 30 epochs with weight decay le-4. In each method we sweep over learning rates Ir = (1e-4,3 4.3e-4). For full fine-tuning, we also train with AdamW optimizer (weight decay le-4), cosine annealing for 30 epochs, but addition, identify optimal learning rates for each pre-training and architecture separately. We first identify an
12

---

cub200
describabletostues
curppeanlloodephh
fgvc, aircrafts
-3
mit67
oxfordflowers
stanfordcans
8.0
stanfortdags
Block
Block 14
Block 15
Block 16
Block 17
Block 19
Block 21
23
Ensemble gain
Block
Bleck
Blook
Block
Block
Last
Ensemble pieces
Figure 7: Optimal Representation pairings Optimal ensemble pairs of InCA of listeners at different locations of the network; Optimal ensembles improve any single layer. VIT-L/16 DeiT pre-training.
Top-1 Test Error for VIT-L/16 (DeiT pre-training)
# of InCA queries (m)
Dataset
4
16
CUB-200
9.6
DTD
17.8
18.4
19.2
19.1
Aircrafts
15.8
19.3
19.8
16.8
MIT-67
10.1
10.8
11.0
10.9
Oxford Flowers
0.3
0.3
0.3
0.4
Oxford Pets
4.7
4.7
4.5
4.4
Stanf. Cars
8.4
8.7
8.8
8.2
Stanf. Dogs
8.1
6.3
6.3
5.9
Table 9: Varying # of queries in the InCA adapter We run an ablation testing the effect of applying different number of queries ...... 4m and then averaging when using the InCA adapter. We observe that most cases m does not have a big effect on accuracy and that m = 1 has sufficient representation capacity for the adapter.
architecture coarse-range learning rate based on performance 5 datasets by sweeping over Ir 1ec-,,le3,,11 1.---.....111..1-1------ followed by a refined sweep with learning rates B, 2B with B being the optimal coarse learning rate.
For the VPT baseline, we follow the details presented in the paper and train with VPT-Deep which was observed to outperforms VPT-Shallow. To train VPT, we use the SGD optimizer with momentum and cosine annealing for 100 epochs. For each dataset we run sweep on the prompt length (5,20,100), base learning rate [0.25.0.1 1,.00,,00001 and weight-decay 11e-2,1e-1 100-epochs for each dataset. We compare the training cost of InCA and VPT-Deep Table 7. In general we note that the shallow and small architecture of InCA or linear probing that are separate from the base model makes them straightforward to optimize, compared with adaptation methods that receive back propparppee gradients from a frozen intermediate layer of the network shown in Fig, 2.
for total of 24 runs with
For the LoRA baseline [28] apply LoRA modified attention each block's self-attention layer (We, W.. W.) in VIT based architectures and to each block's 's Attentinn for SWIN. For the low rank dimension we sweep over the best value among d 10, 50. For BitFit we follow the discussion in [6] and train all of the bias bias- parameters in the network in addition to full training of the head. Analogously for [45] we follow their procedure with LayerNorm which includes training each of the LayerNorm parameters (7, 3) for each layer along with training of the head of the pre-trained model. For all of the efficient training methods above we sweep over Ir=(3e- 5, 4,3e 4, 4,1e-3) to identify the best learning rate for the dataset.
13

---

Broader Impacts Our method, InCA enables efficient and modular model adaptation that can be applied to any strong available pre-trained backbone. that sense, InCA reduces the computational barriers to entry for training and evaluating over large set of (potentially massive scale) models and optimization settings to identify model be used for downstream adaptation. This bridges the gap between cutting edge research in general visual representation learning and specific domain applications, especially since the best performing models are computationally expensive to adapt. Given that InCA operates well on fine-grained visual datasets, this can have positive applications in scientific domains such as medical imaging. many scientific domains, known to be fine-grained yet also with sparse training data. In addition the reduced computational costs associated with downstream adaptation with InCA makes it possible for domain experts without machine learning expertise resources. This can enable domain researchers solve their domain problems by leveraging various public pre-trained models achieve competitive results.
available datasets of use and
InCA without access to large computational
14


Reviews:
[{"review_id": "ESAXLI7OPV", "reviewer": "Reviewer_gDN4", "summary": "\nSummary: Firstly, I would like to kindly point out that this paper proposes an \"adaptation\" method; however, I believe there may be some serious concerns in multiple aspects. Firstly, the paper appears to lack innovation in its methodology. Secondly, the experimental design also seems to have some significant shortcomings, such as lacking adequacy and detailed experimental evidence. Additionally, I would like to express my concerns about the writing aspect of the paper, where some details seem to be missing and where some over-claims are made.\n", "questions": "\nSee *Weaknesses", "limitations": "\nNo. See *Weaknesses", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "\n\n(Positive) Although this paper possesses some flaws, including the lack of innovation and rigorous experimental design, I must acknowledge that the approach in this paper appears reasonable to some extent. Additionally, it is commendable that the authors have provided a significant amount of supplementary material. However, I believe these aspects may not fully compensate for the shortcomings observed in other areas of the paper. I genuinely hope that in future research, the authors will earnestly consider these critical points and endeavor to enhance the quality of their work.\n", "weaknesses": "\n\n(Negative) Allow me to further inquire about some of the claims made in the paper. In the first sentence of the article, the authors claim that in natural language tasks, the data and hypothesis space are shared, which appears quite astonishing. Considering the diversity of tasks in natural language, can this assumption be readily applicable to every task? I believe this general statement may lack sufficient basis and in-depth investigation.\n\n(Negative) Furthermore, the authors assert that their model can enhance \"robustness,\" but it is important to note that \"robustness\" is an extremely specialized concept. Therefore, I hope they can provide more detailed experimental evidence to support this claim. A simple assertion may not be sufficient to substantiate such an important proposition.\n\n(Negative) I must emphasize that the approach presented in this paper does not seem to demonstrate significant differences compared to existing methods, including but not limited to LoRA. This apparent lack of innovation in the technical aspects is somewhat disappointing.\n\n(Negative) Moreover, it appears that the authors' method has not undergone sufficiently large-scale model validation to demonstrate its feasibility in terms of large model transferability. Additionally, the model they have chosen may not entirely qualify as a so-called \"foundation model.\" This lack of rigor in the experimental design raises doubts about the viability of this method.\n\n(Negative) Additionally, the authors seem to have not thoroughly validated the capabilities of their method on various diverse tasks. The scope of their experiments appears to be rather narrow, omitting coverage of multiple tasks such as generative models and discriminative models simultaneously. For instance, in generative models, the training cost of the so-called \"stable diffusion model\" may be high, but its adaptation is crucial. Unfortunately, the authors' in-depth research and contributions in this area seem to be limited.\n\n\n(Negative) What is also concerning is that the authors of this paper appear to have a somewhat limited perspective, primarily focusing on a few tasks they studied, while neglecting the introduction of task details. Moreover, they may hold the belief that ImageNet pretraining is the default choice worldwide, a viewpoint that is challenging to comprehend. In reality, a foundation model should not be confined to ImageNet pretraining but should consider larger benchmarks to enhance its generalization ability. I would like to strongly urge the authors to be mindful of this and clearly indicate in the paper where their model was actually pretrained.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reviewing this work.  We address each of the reviewer’s points below:\n\n>authors hold the belief that ImageNet pre-training is the default choice worldwide, a viewpoint that is challenging to comprehend\n\nWhile we consider ImageNet pre-training for some of our experiments, we do not believe it to be the default choice. Indeed, we consider diverse pre-training data. In our experiments we use pre-training from different large-scale datasets **including LAION, OpenAI 400M**, and **Instagram 3.5B-17K**. Tab. 1 uses an ImageNet-22K pre-training and we note that ImageNet still serves as a strong pre-training in computer vision [1,2,3,4].\n\n>this paper does not seem to demonstrate significant differences compared to existing methods, including but not limited to LoRA. This apparent lack of innovation in the technical aspects is somewhat disappointing.\n\nAs we note in lines 87-90, InCA has key differences with respect to LoRA and other common adaptation methods, which enable new use cases (continual and multi-task learning). Moreover, unlike LoRA, InCA enables massive parallelization and efficiency boost both at inference and training time (lines 55-57), with 65.3% lower GPU memory footprint than LoRA on ViT-H/14. At the same time, InCA achieves 47.7% lower relative test error than LoRA (Tab. 1). We provide the details on the differences below:\n\n* **Architecture agnostic**: InCA can be automatically extended to any backbone architecture whereas LoRA does not. InCA is simply applied on top of any architecture — including CNNs — without any change (as we show in Sec. 3, Tab. 2). On the other hand, LoRA relies on self-attention layers, limiting its use to transformers, and requires manual and cumbersome changes, replacing self-attention with new custom layers.\n* **Different learning scenarios**\n    * **Continual learning**: Unlike LoRA, the modular Open-InCA (Sec. 3) disentangles the learning of each class. This allows both class incremental learning (CIL) and unlearning (forgetting) of a class without effect on other classes. In App. A we show that Open-InCA achieves competitive results on the Split-CIFAR100 CIL benchmark.\n    * **Multi-task learning**: Unlike methods such as LoRA, VPT, etc. InCA does not alter the pre-trained model execution, therefore allowing computation sharing for multi-task inference. In Sec. 3, Tab. 4, InCA achieves the top result in terms of accuracy and performs well in inference time due to computation sharing.\n* **Computational efficiency**: InCA adapters share almost all forward computations with the backbone, however during training do not require back-propagation through it's layers. For example on a ViT-H/14 architecture, InCA with 20 learned adapters has **65.3% lower GPU memory footprint** than training one LoRA adaptation on ViT-H/14. \n\n>the authors' method has not undergone sufficiently large-scale model validation to demonstrate its feasibility..Additionally, the model they have chosen may not entirely qualify as a so-called foundation model.\n\n* Our study of InCA includes 9 separate pre-trained models each evaluated on 11 vision datasets as well as CIL and multi-task benchmarks. Our experiments include foundation models with diverse pre-trainings that have large architectures and use massive amounts of pre-training data. We provide the specific details below:\n\n* **Diverse pre-trainings:** In the paper, we experiment with models pre-trained on diverse datasets, such as\n    *  **3.5B Instagram images** based pre-training (ResNext101)[5]\n    * **400M image-text pairs** for the OpenAI CLIP pre-training (ViT-L/14)[6]\n    * **LAION 2B+ image-text pairs** used for the OpenCLIP pre-training (ViT-H/14, ViT-Gig/14)[7]\n\n* **Large architectures**: In Tab. 2 we use the ViT Gigantic/14 foundation model of OpenCLIP [7]. With over 1.8B parameters, this makes the ***largest public pre-trained vision backbone*** available today (on `timm` and `Huggingface` vision backbones). \n\n>\"robustness\" is an extremely specialized concept. Therefore, I hope they can provide more detailed experimental evidence to support this claim. \n\nTo clarify, the paper focuses on developing a modular and efficient method generalizing to a variety of visual classification tasks. In this context, robustness of the method refers to the ability of InCA to be applied to a diverse and challenging set of classification tasks *across a variety of domains*. We added a statement in the revised paper to avoid confusion with different notions of “robustness” as defined for example in the adversarial attack or optimization communities. In Table 1, 2, 3 we observe that InCA achieves uniformly better transfer than other methods on average over the 11 different tasks and different architectures. \n\n>Allow me to further inquire about some of the claims made in the paper. In the first sentence of the article, the authors claim that in natural language tasks, “the data and hypothesis space are shared”, which appears quite astonishing.\n\nTo clarify on the statement, the field of NLP has seen great progress in recent times due to LLMs. Part of this success is due to the fact that many different NLP tasks can be recasted as a sequence-to-sequence generation problem (where both the input and output sequence (data and hypothesis) live in the same shared space) [8]. For example, traditional NLP tasks such as entailment, and NER are casted as a seq-to-seq task via prompts.\nWe note that paper does not aim to explore NLP foundation models and this should be taken as introductory to the narrative which then directs the reader towards the vision domain. it is **not** a main claim to be established by this work.\n\n>Using InCA in  generative models / the so-called \"stable diffusion model\" \n\nOur paper is concerned with performing fine-grained visual classification, a discriminative task. We do not make claims for InCA regarding generative tasks for this manuscript including stable diffusion adaptation."}, {"review_id": "DW8yXCtIGn", "reviewer": "Reviewer_st4F", "summary": "This work presents a method called Introspective Cross Attention (InCA), which aims to identify high-performing adapter models for handling downstream tasks using large-scale pre-trained models. InCA achieves competitive performance compared to the well-established baseline of full fine-tuning, while also enabling parallel multi-task inference. The experimental results effectively demonstrate the effectiveness of this method in terms of both performance and efficiency.", "questions": "1. Could you please provide (or comment) a comparison in terms of FLOPs? \n2. In terms of memory and compute how does CNN based transfer learning approaches compare with ViT based approaches? \n3. This approach might suit very well for tasks like model patching, see [PAINT](https://model-patching.github.io/) and [Delta-Networks](https://arxiv.org/abs/2303.14772)", "limitations": "Discussed in the appendix", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easily comprehensible. The Introduction section, in particular, effectively establishes the context for the entire work.\n2. The exploration of introducing adapters for large-scale models in the context of vision transformers is still relatively less explored.\n3. The promising results on well-known transfer learning datasets at a smaller scale indicate the potential of the proposed method.", "weaknesses": "1. The datasets employed in this study may not be the most appropriate test-bed for evaluating large-scale pre-trained models. Previous transfer learning works [1, 2, 3, 4] have achieved comparable accuracies using lightweight CNN models that demand significantly fewer FLOPs and parameters. To gain a comprehensive understanding, it would be valuable to compare the FLOPs and parameter counts.\n2. While I acknowledge that the authors have utilized commonly used datasets for transfer learning tasks, it is worth noting that these datasets may not provide sufficient challenges when employing models such as ViT-G/14 and other large variants. \n\n\nReferences\n1. [Co-Tuning for Transfer Learning](https://proceedings.neurips.cc/paper/2020/hash/c8067ad1937f728f51288b3eb986afaa-Abstract.html)\n2. [Stochastic Normalization](https://proceedings.neurips.cc/paper/2020/hash/bc573864331a9e42e4511de6f678aa83-Abstract.html)\n3. [Bi-tuning of Pre-trained Representations](https://arxiv.org/pdf/2011.06182)\n4. [$\\Delta$-Networks for Efficient Model Patching](https://arxiv.org/pdf/2303.14772) -- Table-7\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review of this work and are encouraged that they find the presentation to have effective context and and the method to be be presented comprehensively including in the realm of large-scale modern architectures. We address the points raised by the reviewer below  \n\n> This approach might suit very well for tasks like model patching, see PAINT and Delta-Networks\n\nWe thank the reviewer for suggesting model patching. This is a very suitable testbed for the modular design of the InCA and Open-InCA adaptation, we add and cite this relevant application during our multi-task discussion in the manuscript. \n\nBy not modifying the pre-trained backbone architecture and by sharing the majority of the computation with the pre-trained backbone, InCA adapters can be used for patching an existing base model. Similar to model patching are the flexible learning frameworks we consider in Sec. 3 (multi-task learning) and Appendix A (Class Incremental Learning and forgetting) which both do not exactly follow the model patching frameworks yet are highly adjacent. Namely through Tab. 4 we observe InCA is the top performing method in the ImageNet-to-Sketch multi-task benchmark and enables efficient multi-task inference which is be crucial for efficient patching networks. Further in Appendix A we show how even at the level of a single adapter and a single class one can add additional classes or unlearn a class flexibly. The corroborated results of InCA on multi-task learning and continual learning make it a great method for model patching and we added this context in the paper.\n\n> Previous transfer learning works [1, 2, 3, 4] have achieved comparable accuracies \n\nWe thank the reviewer for the suggested relevant references, we have incorporated [1,2]  in the related work section as methods that focus on the learning objective and references [3,4] as additional interesting approaches for adaptation (e.g. improving normalization). We agree with the reviewer these works can be complementary to the the adaptation approach presented InCA.  However, we note when discussing similar accuracies it’s important to keep the discussion in the context of the learning task among other details. For example, the suggested work of [2] changes the *learning objective* by introducing additional un-supervised losses, as mentioned these approaches are complementary with InCA yet operate orthogonally. \n\nIn general, final test accuracies may also be influenced by factors such as different data augmentations, modified image resolutions among many other boosting methodologies which improve results.  We refrain from using such boosting methods and our study simply uses the 224 input resolution with traditional random crop augmentation for fair benchmarking and to avoid conflating multiple aspects of the learning task. We can not find those details in some of the suggested work which can also make it more challenging to directly compare. \n\n> In terms of memory and compute how does CNN based transfer learning approaches compare with ViT based approaches?\n\nThe memory and compute of most transfer learning methods will directly correlate with the size of the pre-trained network (especially for methods that require back-propagating through the pre-trained backbone). this is mostly independent of the architectural family between ViT and CNNs. The majority of proposed “efficient” adaptation methods are parameter efficient but are often not compute efficient (e.g. VPT, see Tab. 7 of the Appendix) which makes them challenging to be used with large scale models. On the other hand InCA is compute efficient and does not back-propagate through the pre-trained backbone making it highly scalable to massive architectures. \n\n> Could you please provide (or comment) a comparison in terms of FLOPs?\n\nInCA is computationally efficient with regards to FLOPs as compared to other methods. There is no back-propagation through the pre-trained network, which eliminates the FLOPS associated with computing gradients through each intermediate activation in the model’s operation graph. The only backwards operations being propagated are on the small adapters parameters themselves. During parallel training multiple adapters share the FLOP computations associated with the backbone's forward pass which amortizes the computation between the adapters. For a comparison in terms of training costs (measured in wall clock time) between InCA and state of the art transformer adaptation method VPT see Appendix C Tab. 7. "}, {"review_id": "JqJ8Ck8Tnl", "reviewer": "Reviewer_DnPr", "summary": "The paper presents a method termed InCA (Introspective Cross-Attention) to learn compact adapter modules for large vision models that can be used for various downstream tasks (image classification domains). The proposed approach has an advantage over the entire model finetuning due to its parameter efficiency resulting in a smaller GPU memory footprint required for training. In addition, the proposed introspective cross-attention module is architecture-agnostic, enabling simple implementation for different intermediate network representations and network types.  ", "questions": "Could you highlight the most interesting (maybe counterintuitive?) findings? “The second best adaptation approach is LoRa … at additional training costs” invites the comparison along axes other than the error rate. Could you compare the training costs directly?\n\nTable 2: “\\dagger indicates full FT was avoided due to prohibitive computational costs” – could you elaborate on what costs were considered prohibitive and why? \n\n\n", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach (InCA) demonstrates stable performance when applied across different model types (ViT, SWIN, CNN). Unlike previous architecture-specific adaptors, the method can be implemented without modifications with different base model architectures.\n \nThe experiments demonstrate that the method can reach performance comparable to the entire model finetuning. At the same time, the number of trainable parameters constitutes only a tiny fraction (1-2%) of the backbone model weights. Because the training process doesn’t require backpropagation through the backbone, it is GPU memory efficient, making the large pre-trained transformer-based backbones reusable for different downstream tasks. \n \nCompared to the linear probing, the proposed cross-attention architecture of the adaptor module has a significantly larger “extraction capacity” that leads to an improved classification accuracy of the adapted model. When applied in parallel to different activation maps, the method produces a network “signature” - w.r.t. to the downstream tasks. All this highlights the fact that the internal representations of the pre-trained large models have sufficient representation power for many downstream domains.\n", "weaknesses": "The paper builds on the large body of literature on efficient transfer learning. While exploring the utility of cross-attention as a choice for adaptor architecture, the authors follow the steps of many prior works, such as Perceiver [30] or [A]. On the other hand, the current study combines empirical results for different backbone architectures (e.g., both CNNs and Transformers) and suggests applying cross-attention modules in parallel to different activation maps to identify the most relevant features for the downstream tasks systematically. Thus, despite being a comprehensive study, the novelty is limited. As discussed in the Appendix, InCA, as an efficient and modular model adaptation framework, can be helpful for domain practitioners (e.g., medical imaging domain) to bridge the gap between cutting-edge research in visual representation learning and real-world applications.\n\n[A] Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation, in EMNLP 2021. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the through review of our method noting the parameter/training efficiency, the signatures produced by InCA and the compactness and modularity enabled by the method as well as that InCA \"can be helpful for domain practitioners\". We address the points of the reviewer below.  \n\n> follow the steps of many prior works, such as Perceiver [30] or [A]\n\nThe reviewer is correct that we are not the first to utilize cross attention for it’s extraction capacity and indeed in the related work section we cover different uses of cross attention in the literature. We also thank the reviewer for pointing reference [A] which we have added in the related work section for it’s expressivity of cross attention. *However* regarding [A] we note that the approach by which the cross attention is used is different both in terms of the tasks considered - natural language translation, but also in 2 major ways in which the cross-attention is actually used. In the suggested work, an encoder-decoder transformer is used for machine translation task. In that setting the base architecture already crucially utilizes cross attention in the decoder layers, and the cross attention happens between the encoded tokens (as keys) and the previously generated tokens (as queries). In the work the authors show that just optimizing the cross-attention layers provides the expressivity needed for fine-tuning on the task. However we note that InCA introduces randomly initialized adapters that did not previously exist in the architecture. Further the queries themselves in the cross-attention layer used by InCA are not computed from the activations but rather are learned and optimizable parameters. \n\nRegarding the novelty of InCA, we note that while the architecture of the InCA adapter is simple, the parallel and exhaustive training of InCA does not appear in perceiver (or any other work) and is key for the efficiency and the search for relevant activations existing in the network. The lightweight and parallel approach of InCA enables it’s use to conquer large-scale pre-trained models at extremely efficient adaptation costs (e.g. adapting ViT G/14 on 1 GPU). The parallel training of InCA is also directly responsible for it’s ability to produce intermediate representation signatures in a feasible manner via the computation sharing of the different trained adapters which are all new.\n\nFurther we would like to point out that we extend InCA to Open-InCA (presented in Sec. 3 and additionally in Appendix A with results) which introduces a novel adaptation approach where the learning of each class is done in a disentangled and separate fashion. This makes InCA into a highly modular and flexible adaptation approach for class-incremental learning and unlearning that further illustrates the novelty and the flexibility of the proposed approach.  \n\n> “The second best adaptation approach is LoRa … at additional training costs” invites the comparison along axes other than the error rate. Could you compare the training costs directly?\n\nIn our experiments we observe InCA outperforms LoRA in all architectures tested, with LoRA being the second best approach. With regard to training costs, InCA scales to larger models much more efficiently. In Figure 1 (right) we observe that since InCA does not back-propagate through the pre-trained backboned it makes training even architectures as large as ViT-Gigantic/14 feasible under fairly modest computation and GPU memory constraints. We run an experiment to compare InCA with LoRA on the a ViT-H/14 architecture, where for InCA we use 20 adapters trained in parallel. We observe that even with 20 adapters, InCA has **65.3% lower GPU memory footprint** than training one LoRA adaptation on ViT-H/14. \n\nIn addition InCA and Open-InCA enable computation sharing for multi-task learning, and continual learning which are not feasible with LoRA (since LoRA modifies the backbone execution whenever it is applied to a model). Lastly InCA is architecture agnostic and can be applied to CNNs whereas LoRA dictates the existence of self-attention layers in the architecture. \n\n> dagger indicates full FT was avoided due to prohibitive computational costs” – could you elaborate on what costs were considered prohibitive and why?\n\nFor each entry in Tab. 2 we run multiple training runs with different learning rate on each of the 11 datasets used in Tab. 1 to report the average accuracies on those tasks. For ViT Gigantic/14 which has 1.8B trainable parameters, running training with full-fine tuning on each of the listed datasets incurred prohibitively large costs for the given timeframe."}, {"review_id": "grEaqYSjNf", "reviewer": "Reviewer_KdhN", "summary": "This paper proposes an efficient fine-tuning method that works parallel to the pre-trained network. Based on cross-attention between intermediate activations, InCA can generalize to various classification tasks from different domains. Additionally, the framework inherently supports class incremental learning, multi-task learning and continual learning objectives. The overall objective is cross-layer feature merging using cross-attention between them while freezing the base model. \n\n", "questions": "See the weakness sections.\n1. Standard benchmarks and a thorough comparison with existing works will make this work a lot stronger\n2. A major emphasis on peak training peak memory, inference latency and parameter overhead needs to be provided. ", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. There are various parameter-efficient fine-tuning methods in the literature; the major strength of this work is described in the last part of Sec. 3 - using cached activations to train the model in mere seconds.\n2. The overall formulation of cross-attention is simpler and elegantly extends to multi-task and continual learning paradigms.\n3. Experiments are shown on various vision classification datasets and one multi-task learning benchmark.", "weaknesses": "1. The formulation of Open-InCA is not completely clear and can be better presented; However, I understood the complete idea, and I had to re-read it a couple of times for deeper understanding.\n2. The field of efficient fine-tuning/transfer learning is rapidly moving, and using standard benchmarks helps understand the performance gains more. The authors can give results on VTAB-1k, few-shot learning experiments, etc, as shown in SSF[1], VPT[2], FacT[3]\n3. Also, I found comparison with existing methods a major weakness, as only LoRA, BiTFit and AdaLN are shown.\n\n[1] Scaling & shifting your features: A new baseline for efficient model tuning. [2] Visual Prompt Tuning [3] FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The formulation of Open-InCA is not completely clear and can be better presented; However, I understood the complete idea, and I had to re-read it a couple of times for deeper understanding.\n\nWe thank the reviewer for the engaging review of this work and value their appreciation of the elegance of IncA for modular multi-task and CIL settings. Regarding unclear presentation of Open-InCA in Sec. 3, we first note that we provide a more extensive description and presentation of Open-InCA and class incremental learning experiments in Appendix A.\n\nAs the reviewer is suggesting, we re-wrote and combined the presentation of Open-InCA in Sec. 3 that was previously presented into 2 separate parts (L178 - 184)  and continued in (L194 - 212) to give a complete picture of Open-InCA that fills in all the details directly in the main manuscript. The revised writing includes the discussion on “query only training” and highlights the disentangling of class representations for class incremental learning and forgetting. \n\n> A major emphasis on peak training peak memory, inference latency and parameter overhead needs to be provided.\n\nRegarding peak training memory we refer the reviewer to Fig. 1 (Right) which reports **peak** training GPU Memory with the training of InCA as compared with other adaptation approaches, the GPU memory is measured for the training process with a batch size of 32. \n\nRegarding inference latency, as presented in Tab. 4, the inference speed of the adapter in InCA is dwarfed by the inference of the pre-trained model, with 2.6% additional overhead for an IncA adapter on ViT L/16. This is because the InCA adapter is a small shallow network made out of a few cheap sequential operations with cross attention scaling linearly unlike self-attention. In the case of multi-task learning, the shared backbone inference computation leads to 4.4x speedup on the ImageNet to Sketch benchmark. \n\nFor regular, single task inference, as noted above the overhead of InCA adapter is < 3%. Further, if the top adapter appears at a particular intermediate layer, one can discard subsequent layers and use a truncated pre-trained model.  As an example, taking ViT-L/16 with Stanf. cars (best block, 16) adaptation we run an inference benchmark for the purpose of this review and use InCA with a truncated backbone, from this experiment we observe using InCA with truncated backbone leads to 31% faster inference as compared with a fine-tuned network since InCA benefits from removing the unnecessary later layers.\n\nLastly, regarding parameter efficiency, InCA is parameter efficient with 1.3% of the parameters of a ViT L/16.  Nonetheless as we discuss in the paper (L115-119) we believe that parameter efficiency is only a piece of the picture for an efficient adaptation which should also enable efficient training. "}, {"review_id": "8dOo0c5zvc", "reviewer": "Reviewer_XAoM", "summary": "- This paper proposes a new way to adapt a pretrained deep neural network for downstream tasks called InCA. InCA does not modify the intermediary representations of the pretrained network and thus doesn’t require backproping through it, which makes it memory- and compute-efficient. To use InCA, one first heuristically identifies a few layers, the activations of which are processed through cross-attention with trainable query tokens.\n- The cross-attention setup is more expressive than linear maps, which is supported by empirical and theoretical arguments. It also allows the addition of new classes through the injection of new query tokens.\n- Finally, because the forward pass of the original network is not at all modified, one can cache the relevant activations of the entire dataset once and apply InCA without calling the original model at all. It is also possible to adapt to many tasks in parallel for the same reason.\n- Experiments on many image datasets show that InCA performs competitively with finetuning and other efficient adaptation methods.", "questions": "- How competitive is InCA in other domains where many of the compared baselines, e.g., LoRA and BitFit, dominate? E.g., language understanding, language generation, diffusion, etc.\n- How is the subset of layers chosen for the experiments? Could a similar selection process benefit other baselines?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This work includes a broad range of baselines in their experiments.\n- The proposal is simple yet appears to be effective. The stated benefits are technically sound.", "weaknesses": "- The paper “[presents] a framework for transfer learning that efficiently adapts a large base model by learning lightweight cross-attention modules attached to its intermediate activations.” However, the experiments are limited to mostly computer vision classification tasks. Some of the baseline approaches, such as LoRA and BitFit, are extensively used for language understanding, language generation, and image diffusion. The result of this paper will be much more convincing if experiments in these domains are included, especially if InCA can outperform existing baselines.\n- A significant portion of the benefit of InCA comes from using a subset of the layer activations. However, this important choice (as described in B.2) is done heuristically according to Section 3. The paper can benefit from more clarity on how to choose such layers. For example, how are these layers chosen for the experiments?\n- The clarity of the writing can be improved. For example, it was not clear during my first pass what the dimension of z is and what T is. The description for diag-head is confusing: it appears to be a simple matrix multiplication between [a_1, …, a_c] and W. Section 3 overall can be better organized by heuristics, methodology, and benefits.\n- The “signature” phenomenon is interesting, but the given analysis doesn’t provide much insight. It would be great to hear more about what we can learn from these “signatures,” especially given that they are presented as one of the key advantages of InCA in the conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review of our work and appreciate that they found InCA a simple and comprehensive approach with broad range of baselines and sound benefits. below we address the main points.\n\n>A significant portion of the benefit of InCA comes from using a subset of the layer activations... how are these layers chosen for the experiments? \n\nWe would like to clarify our layer selection process — we do not heuristically select the layers for each experiment, rather we use the parallel training property of InCA (Lines 148-157) to **exhaustively train all layers in parallel**. We test the performance of each adapter using cross validation and finally report the best performing layer (retrained on all the training set). This is one of the key motivations of InCA, we wanted to find a way to exhaustively analyze each layer. Done sequentially, this is very expensive, instead InCA’s parallel training becomes an effective solution since we are capable of training 40+ adapters in parallel on a single GPU. \n\nIn Sec. 3, we note that when running the experiments *we patently* observe that the best performing layers are from activations with residual connections and tend to be from the later half of the network. Therefore we use that simple criteria for the adapter set, however we can, and have trained InCA attached to all of the network’s blocks. \n\n> Could a similar selection process benefit other baselines?\n\nYes, in Tab. 1 and 2 we report the In-LP and In-MLP-3 adaptation results (with “In” refers to intermediate). This corresponds to applying InCA’s parallel training procedure with intermediate representations for linear probing and MLP networks. For other baselines that modify the internal representations of the network (e.g. LoRA, VPT) InCA’s procedure can not be applied since the adaptation modifies the backbone’s execution. \n\n> The clarity of the writing can be improved. For example, it was not clear during my first pass what the dimension of z is and what T is. The description for diag-head is confusing...\n\nThank you for the feedback, we updated the notation issues of T (# of tokens) in line 171 and on, to be consistent in our notation and remove this ambiguity. To clarify regarding, diag-linear, it is not exactly a matrix product, rather computes the diagonal of a matrix product, i.e. \n\ndiag-linear([a_1, ... a_c], [w_1, ... w_c]) → [<a_1, w_1>, ... <a_c, w_c>].\n\n>The “signature” phenomenon is interesting, but the given analysis doesn’t provide much insight. It would be great to hear more about what we can learn from these “signatures,” especially given that they are presented as one of the key advantages of InCA in the conclusion.\n\nWe provide an extensive analysis with insights from the signatures produced by InCA in App. B. These results were relegated to the appendix due to the page limit. Below we provide a self-contained summary of InCA’s signatures and some of the results presented in the appendix. We encourage the reviewer to check out App. B if they would to find more details about the intermediate representation signatures.\n\n+ **Signatures**: Training InCA produces a set of adapters attached to different network’s activations. Evaluating each adapter (in parallel) gives the performance of each particular activation map on a task. We observe unique patterns for the signatures on different tasks indicating which intermediate representations are most helpful.\n\n+ **InCA and partial tuning have matching curves (App. B.1 and Fig. 4)**\n    + In App. B.1 we compare the InCA and partial tuning signatures. In partial tuning, we repeat fine-tuning experiments where we fine-tune all layers up to a “freezing point” e.g.  training the last K layers of the network for varying values of K=1, ... L. \n\n    + In the same fashion as InCA, we produce partial tuning signature by running a set of experiments with different partial tunings and their evaluated performance. By running this for each freezing point we get a signature (albeit at a much larger cost than a single InCA training run).\n\n    + For partial tuning is that as you tune more layers - the tuning is more expressive and the test accuracy increases. Interestingly, the partial tuning plots follow an “elbow behavior” where after a certain number of layers are used for tuning, the performance improves dramatically and the performance roughly saturates afterwards (Fig. 4).\n\n    + Even more exciting, the top InCA adapter matches the location of the “elbow”. Meaning the point of the elbow is at the same location where the representation found by InCA can be harnessed by the partially tuned network.  This systematically shows the behavior of fine-tuning as “surfacing” existing representations. That is, when a particular activation layer is unfrozen it can be leveraged for downstream representation via tuning, or alternatively InCA efficiently finds those representations directly. \n\n+ **Layer affinities (App. B.2, Fig. 5)**\n    + In App. B.2 we review the InCA signature of the same task applied to different backbones and architectures. (Fig. 5).\n    + What we observe is that for many datasets, the intermediate representation signature is highly consistent between different architectures and same type of pre-training to the level of exact layer matches. This intriguing property shows how much of the representations of the network are independent of the architecture and are mostly a function of the pre-training task, even for very different architectures such ViTs vs CNNs.\n\n>Applying InCA for language generation, and image diffusion\n\nApplying InCA to generative tasks is an interesting avenue for future work where we think the modularity and efficiency of InCA can be leveraged. However in the current work we focus on core discriminative tasks.\n\nWe thank the reviewer for the detailed review and hope that by addressing the main concerns regarding layer selection and \"signatures\" that the reviewer considers raising their recommendation."}, {"review_id": "5kt44cZHPY", "reviewer": "Reviewer_d63D", "summary": "The paper presents a study on the benefits of training a small cross-attention based adapter instead of performing full fine-tuning of a large VIT model. The authors propose a cross-attention layer they dub InCA that has trainable queries to cross attend to intermediate layers in the large pretrained VIT model and then extract the information by average pooling, layer-norm and Linear. They claim a single query is enough in most cases but also propose a multiple query version they call OpenInCA.", "questions": "* Have you tried multiple self-attention layers instead of just one? This would be equivalent to running a small perceiver model [30] and then either average the resulting latent tokens.\n* The average pooling has to be done to make sure that the cross attention IS the right architecture here and not a simpler one. It would also add a bit of depth to the paper and answer an obvious question. I think this is required to recommend acceptance.\n* Why do you include both tables 1 and 3 ?", "limitations": "I think there is no obvious negative impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* There are strong benefits of having a methodical study of adaptation and I think this paper fits the bill.\n* The method is simple, well explained and a priori easy to reproduce and expand.\n* The experiments are broad enough to be interesting to the community.\n* There are clear computational benefits from the two stage training.", "weaknesses": "* The claim in L177 is a bit of a red flag for me. If only one query is enough it may be that the way this is achieved is when the query is very far from the \"latent\" data and thus the attention weights become flat ie performing average pooling. The nice thing is that it would be an easy and i think useful experiment to check this. Redoing figure 3 with avg pooling but with all the extra bits like the layer-norm that is in the model. I presume the LP doesn't have it ?\n* I haven't found any experiments to validate that the class-incremental learning benefits of the Open InCA architecture which is claimed in line 203.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and thoughtful review of our work and the insightful questions. We are pleased that the reviewer appreciates the the methodological experimentation presented in the paper, the generality of the method, and the discussion of two-stage-training. Below we address key questions raised by the reviewer and run the experiments suggested by the reviewer.\n\n>If only one query is enough it may be that the way this is achieved is when the query is very far from the \"latent\" data and thus the attention weights become flat ie performing average pooling. [An] Experiment to check this... [is] redoing LP with avg pooling but with all the extra bits.\n\nThank you for the detailed and insightful question regarding the number of queries. The subtlety that can clarify this point is the presence of **multi-head** cross-attention in InCA. With multi-head cross attention, a single query is projected differently for each head. In our case a single query corresponds to 16 different query projections, thereby allowing a single query to surface multiple task specific representations from the activation map.\n\n**Experiment with average pooling**\n\nFollowing the reviewer’s suggestion, we repeat the adapter experiment using average pooling. Specifically, as you suggest we keep all the extra bits of InCA (e.g. LayerNorm, projection layer of cross attention) and simply replace the learnable cross attention layer with average pooling (but still keep the projection layer for additional capacity). We will refer to this as InLPX for intermediate linear probing “extended”, which now has a layer norm, average pooling, projection layer, and a classifier head. We use InLPX applied to the same activations as InCA in the same exact settings of Table 1 ViT L/16. We report the average test error on the 11 datasets presented in Tab. 1 of the paper in the table below. In short, as the experiment shows, InCA outperforms average pooling which given the experimental settings can be attributed to the relative expressivity of the multi-head cross-attention of the InCA adapter.\n\n**Top-1 Test error over the 11 datasets presented in Tab. 1**\n| Method | Full fine-tuning | InCA | Intermediate linear probing |  InPLX (*New*) |\n| -------- | -------- | -------- | -------- | -------- |\n| Average | 10.0  | 9.8 | 17.2 | 14.1 |\n| Maximum gap to full FT | 0 | 2.3 | 35.7 | 26.4 |\n\nAs you suggest we observe that adding the additional bits of InCA to intermediate linear probing does lead to an improvement, namely from 17.2 → 14.1 average test error. **However**, even by repeating the entire procedure aside from cross attention as in InCA we still observe that InLPX is still has a maximum gap of 26.4 points as compared with InCA that has 2.3 point gap (>10x larger maximum gap), and has 44% worse test error relative to InCA.  Given the experimental design of making InLPX match InCA on all aspects except cross attention, the gain can be directly attributed to the cross attention layer.\n\n**Detailed explanation on expressivity**\n\nWe point out 2 key factors when looking at cross attention vs. average pooling\n1. **Multi-head**: The InCA adapter uses a multi-head cross attention module which has 16 heads by default (follows the number of heads in the pre-trained model). In this example, the single query is equivalent to 16 distinct smaller dimensional queries applied on the activation map, enabling diverse querying of task-relevant representations.\n\n2. **Instance-based**: Further another aspect we believe is important in the cross attention layer is that even if the queries are fixed, the aggregation of the activation map will not be constant between images, this is because the computed keys of the activation map are going to depend on the particular input instance which will further result in dynamic aggregation.\n\nLastly note Appendix D is concerned with analytically describing the benefits of cross attention and can be relevant to this discussion.\n\n>I haven't found any experiments to validate that the class-incremental learning benefits of the Open InCA architecture which is claimed in line 203.\n\nThe experiments for Open-InCA are presented in Appendix A. We provide a quick summary of the results below. Recall in Open-InCA, we learn a query specific to each class separately, as well as class specific heads (diag-head). This completely disentangles the learning of additional classes from existing classes which enables extremely flexible class incremental learning and class forgetting. Out of the box, Open-InCA achieves competitive results on the Split-CiFAR100 Class Incremental Learning benchmark without task specific architectural changes (e.g. a split routing classifier) used by state of the art methods. \n\n>have you tried multiple self-attention layers instead of just one? This would be equivalent to running a small perceiver model [30] and then either average the resulting latent tokens.\n\nWe have not trained InCA adapters with access to multiple activation maps, this is an interesting future research. However, we observe that a *single adapter* utilizing a *single relevant activation map* is already capable of outperforming all other efficient adaptation methods, achieving within the same accuracy on average as full fine-tuning for ViT-L/16. We also point the reviewer to Appendix E2 that presents a preliminary ensembling study of InCA adapters.\n>Why do you include both tables 1 and 3 ? \n\nTable 1 and Table 3 of ViT and SWIN architectures are included to demonstrate the generality and diversity of InCA on non-vanilla transformer architectures. \n\n**Conclusion** We thank the reviewer again for the thoughtful review of this work. We hope that by addressing the main concerns raised, specifically 1) the additional details on Open-InCA experiments (presented in Appendix A) and 2) clearing up the query question and repeating the experiment with average pooling, the reviewer considers raising their recommendation."}]

Author rebuttal:

Meta-review: I am recommending to **accept** this paper "Your representations are in the network: composable and parallel adaptation for large scale models".

Six reviewers provided feedback on this paper. The authors provided a response to the reviews and I appreciate the authors' comments and clarifications, specifically around the computational requirements of the presented method. There was some discussion, and two reviewers increased their scores as a result.

Among the 6 reviewers, only one reviewer's assessment falls onto the "borderline reject" side of the threshold and 3 reviewers vote for acceptance, even though only as "weak accept". Based on the review and rebuttal discussion, I would tend to put more weight on some of the reviews that evaluated the paper positively. Altogether, I think there is sufficient support among the reviewers for an acceptance of the paper.

To me personally, the method seems interesting, relatively novel in this form, well-presented, practically relevant, and the presented results are also interesting. 

Overall, I believe the paper is of high enough quality to be accepted at NeurIPS, hence my recommendation to accept.
Decision: Accept (poster)