SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines For downstream transfer experiments, we compare InCA adaptation to other adaptation protocols. 1) paragon: Full fine-tuning is considered as the paragon as it performs well on a diverse set of datasets but incurs steep computational and parameter costs. 2) We compare InCA to other parameter efficient approaches, including 2a) LoRA [28] 2b) Visual Prompt Tuning (VPT) [34] where we apply the top-performing VPT approach, VPT-Deep. 2c) BitFit [6] and 2d) AdaLN [45] which is the LayerNorm analogous, AdaBN approach. Note, each approach we name in 2) requires backpropagating through the entire network's activations to update the learnable parameters and leads to a large computational overhead compared with InCA. 3) In addition, we also compare InCA with a suite of computationally efficient approaches that avoid backbone backpropagation like InCA. These include 3a) Linear Probing (LP), 3b) Intermediate Linear Probing (In. LP) which utilizes the same training procedure as InCA but with a LP classifier on the activations. 3c) MLP-3, which is a feed-forward network that consists of probing the base-model with a 3-layer feed-forward network, and 3d) Intermediate MLP-3 (In. MLP-3), the extension of MLP-3 to intermediate layers.
Training details In all of our results (including multi-task settings) we use the same training configuration for InCA. We only change the adapter architecture input layer to automatically match the dimension of the base-model activation map. InCA is robust to hyper-parameters and our training schedule is consistent for all runs. This amounts to 30 training epochs, AdamW optimizer [52] with 2 learning rates, and cosine annealing learning schedule; we provide the full training details
Top-1 Test Error, ViT-L/16
Dataset	Full FT	InCA	InCA (last)	In. LP	LP	In. MLP-3	MLP-3	VPT [34]	LoRA [28]	AdaLN[45]	BitFit [6]
CUB-200	9.1	8.7	9.4	16.2	16.2	13.9	13.9	10.4	12.7	15.6	15.4
DTD	18.2	17.2	18.4	18.9	20.6	17.4	20.1	21.4	19.4	22.2	21.9
Flood Depth	18.9	17.1	19.6	17.8	22.8	17.6	20.1	19.0	19.6	18.7	18.7
EuroSAT	1.0	1.2	1.9	2.1	3.7	1.5	2.5	1.1	0.9	1.5	1.4
Aircrafts	14.9	15.6	21.9	50.6	67.4	36.8	47.4	21.7	16.6	28.5	27.2
Herbarium	18.8	21.1	24.6	32.6	39.8	29.5	36.4	21.4	19.2	27.9	28.3
MIT-67	10.4	9.0	9.0	9.7	10.5	10.1	11.2	14.8	14.8	15.1	15.1
Oxford Flowers	0.6	0.3	0.4	0.6	1.1	0.5	0.7	2.2	4.0	7.0	7.2
Oxford Pets	4.2	4.0	4.2	6.1	6.4	5.3	5.5	6.9	4.3	5.5	5.3
Stanf. Cars	8.1	7.7	10.2	29.2	47.2	20.8	31.4	9.2	8.4	16.0	14.7
Stanf. Dogs	5.9	5.4	5.8	5.3	5.3	5.7	5.7	7.3	4.3	3.8	3.7
Mean Top-1 Test Error (Max. gap to Full FT)	10.0 (0.0)	9.8 (-2.3)	11.5 (-7.0)	17.2 (-35.7)	21.9 (-52.5)	14.5 (-21.9)	17.7 (-32.5)	12.3 (-6.8)	11.3 (-4.4)	14.7 (-13.6)	14.4 (-12.3)
% Trainable param.	100%	1.3%	1.3%	0.1%	0.1%	2.8%	2.8%	0.8%	2.4%	0.1%	0.1%
No backbone backprop.	X	✓	✓	✓	✓	✓	✓	X	X	X	X
in Appendix F. While InCA is efficient enough to operate at larger resolutions, we use 224 image resolution unless stated otherwise. Nonetheless, InCA performance improves at 384 resolution while remaining computationally competitive (see Table 2).
Transfer learning on ViT In Table 1, we demonstrate the transfer performance of InCA applied to ViT-L/16. For each dataset we train InCA and extract activations at residual layers of the ViT for the last 12 blocks and output layer. For all baselines and our method we use the ViT DeiT pre-training [67] and additionally report ViT-L/16 pre-training results in Appendix Table 8. In the table, we compare InCA to full fine-tuning as well as applying InCA on the last layer and observe that only InCA is capable of achieving good results on challenging datasets such as Aircraft, Stanf. Cars, etc. and closes the maximal gap to full fine-tuning to  $-2.3\%$ . The second best adaptation approach is LoRA which achieves a maximum gap of  $-4.4\%$  to full fine-tuning, yet at additional training costs. For a single dataset we can train the InCA modules with 2 learning rates in parallel which corresponds to 26 InCA modules with identical architectures attached to 13 activation maps. In this case, the total training costs of InCA on a single dataset correspond to one base-model run. In Appendix C, we report the hyper-parameter settings and training cost of InCA and current state of the art adaptation method for transformers, VPT [34] which incurs up to  $8.7\times$  the training costs of InCA with a large hyper-parameter search (2 vs. 24 settings).
Transfer learning on SWIN In Table 3 we present downstream adaptation results for the SWIN-L pre-trained model. InCA adaptation is applied to the 3rd and 4th stages of the network residual activations. Because of the heterogeneous activation dimensions of the hierarchical SWIN architecture,
Table 1: Fine-grained Classification Top-1 Test Error (ViT-L/16) We compare InCA to full fine-tuning (Full FT) along with other adaptation approaches for downstream learning. For each method we summarize the maximum gap in performance compared with the full fine-tuning paragon. In addition, we report the parameter efficiency and whether the method requires backpropagation through the pre-trained model. The minimum error over the columns excluding Full FT is presented in bold.
Category	Architecture	Mean Top-1 Test Error (Max. gap to full FT)
Pretraining data	Full FT	InCA	InCA (last)	inter. LP	LP	Model size
Vanilla	ViT-B/16 [18]	In21K	13.0 (0)	15.9 (-7.6)	17.5 (-16.4)	23.9 (-32.4)	24.3 (-32.4)	86.5M
Transformer	ViT-B/16 [44]	ALBEF (CC14M)	13.8 (0)	13.5 (-4.2)	14.8 (-9.3)	24.7 (-42.6)	25.8 (-42.6)	85.9M
ViT-L/16 [67]	In21K (DeiT)	10.0 (0)	9.8 (-2.3)	11.5 (-7)	17.2 (-35.7)	21.9 (-52.5)	304.3M
ViT-L/16 @384 [67]	In21K (DeiT)	\( -\dagger \)	9.2 (-0.6 \( \dagger \))	11.7 (-9.1 \( \dagger \))	17.3 (-38.1 \( \dagger \))	22.0 (-54.4 \( \dagger \))	304.7M
CLIP-ViT-L/14@336 [61]	400M Im-Text	\( -\dagger \)	9.2	10.6	19.6	21.8	304.2M
ViT-H/14 [18]	2B Im-Text	\( -\dagger \)	9.4	10.4	14.0	15.2	632.8M
ViT-G/14 [31]	2B Im-Text	\( -\dagger \)	9.6	10.4	15.3	16.8	1884.9M
Hier. Transformer	SWIN-L [48]	In21K	9.3 (0)	10 (-3.6)	12.4 (-9.5)	15.8 (-31.3)	18.3 (-40.5)	196.5M
Convolutional	ConvNext-B [49]	In21K	9.4 (0)	10.7 (-7.4)	12.5 (-12.6)	19.1 (-44.2)	19.4 (-44.2)	88.5M
ResNext-101 [74]	IG-3.5B [53]	11.4 (0)	12 (-8.7)	17.3 (-27.1)	20.1 (-38.8)	21.3 (-39.7)	468.5M
Table 2: Mean Top-1 Test Error for transfer learning with a variety of ViT, SWIN, and convolutional networks, including different network scales and pre-training strategies. Averages are reported on the 11 datasets presented in Table  $1.\dagger$  indicates Full FT was avoided due to prohibitive computational costs. For DeiT ViT-L/16 @384 the gap is computed with respect to the 224 pre-training.
the reported adaptation model sizes depend on the activation map used for the selected adapter. InCA achieves the smallest maximum gap to full fine-tuning on SWIN while being computationally efficient. As with ViT-L, in SWIN we observe that challenging datasets require using intermediate activation maps for InCA, closing the maximal gap from  $(-9.5\%)$  to  $(-3.6\%)$ .
Evaluating InCA on different pre-trained models InCA can be applied to any feed-forward neural network without any implementation changes. We simply specify the intermediate layer names and feature tensor-ordering for any new architecture and InCA can be used directly. We note this is in sharp contrast to methods that rely on specific layers such as convolution filters [7, 55] or self-attention [28, 34, 43]. We illustrate the architecture versatility of our method in Table 2. We report the mean and maximum test error gap from full fine-tuning on the 11 fine-grained dataset suite as studied in Table 1. We test different architecture families, which include vanilla vision transformers i.e. ViTs, SWIN [48], and modern convolutional networks (ConvNext [49], ResNext [74]). In addition, we test models pre-trained via different strategies including supervised learning [18, 67] and vision-language objectives [44, 61, 31]. We also test InCA at different ViT scales from ViT/B-16 (86M) to ViT/G-14 (1.8B). For InCA adaptation, all model sizes were trained on a single V100 GPU with batch size 32, including for the larger input resolutions runs.
Multi-task Experiments InCA's isolated design is suitable for multi-task inference and a single pre-trained-model can efficiently evaluate a batch of samples on multiple tasks, allowing for "one-to-many" inference. We compare InCA on the ImageNet-to-Sketch multi-task benchmark in Table 4. All methods except  $\mathrm{BA}^2$  were trained with a ViT-L/16 model and evaluated with the ImageNet-to-Sketch version of each dataset [55]. For  $\mathrm{BA}^2$  [7], we report the adaptation on a ResNet-50 [25] backbone, as the  $\mathrm{BA}^2$  approach requires convolutional filters. Overall, InCA is the top performing method reaching near the paragon on the evaluated datasets. Importantly for multi-task, only InCA and LP enable multi-task inference via "computation sharing" of the base model inference.
Learning efficiency Isolating the learning from the base-model means InCA learns shallow neural networks directly on a downstream task. By avoiding deeply backpropagated gradients through the base model, the adapters receive direct signal which improves the optimization dynamics and speed of training. We compare the number of training steps required to train InCA and VPT-Deep and observe that InCA can be optimized in  $4.5 \times$  fewer epochs than VPT. Here we don't take into account the additional GPU memory costs of optimizing VPT in each step, nor the required hyper-parameter sweeps used in VPT. More detailed efficiency comparison results are given in Appendix C.
Top-1 Test Error, SWIN-L
Dataset	Full FT	InCA	InCA (last)	In. LP	LP	In. MLP-3	MLP-3	LoRA[28]	AdaLN[45]	BitFit[6]
CUB-200	9.0	9.1	9.6	10.2	10.6	9.7	9.7	10.0	9.1	8.8
DTD	15.6	17.8	19.1	17.7	19.1	16.7	16.7	15.8	16.7	17.0
Flood Depth	17.6	16.3	18.3	18.5	18.5	16.7	18.5	17.1	16.9	17.8
EuroSAT	0.7	1.5	2.4	2.7	3.7	1.6	2.2	0.9	1.1	1.7
Aircrafts	12.2	15.8	25.3	43.5	52.7	33.7	34.8	16.1	22.7	26.5
Herbarium	14.9	18.2	23.0	29.2	34.0	24.9	27.6	18.4	21.2	29.7
MIT-67	10.5	10.1	10.1	9.9	10.3	10.2	10.2	9.6	8.9	8.5
Oxford Flowers	0.5	0.3	0.4	0.5	0.5	0.5	0.5	0.4	0.4	0.4
Oxford Pets	4.6	4.7	5.5	5.0	5.5	5.5	5.5	5.2	4.8	4.9
Stanf. Cars	7.3	8.4	15.0	29.2	39.0	22.4	26	9.6	14.2	18.4
Stanf. Dogs	9.1	8.1	8.1	7.1	7.1	9.8	9.8	11.3	9.1	9.0
Mean Top-1 Test Error (Max. gap to Full FT)	9.3 (0)	10.0 (-3.6)	12.4 (-9.5)	15.8 (-31.3)	18.3 (-40.5)	13.8 (-21.5)	14.7 (-22.6)	10.4 (-3.9)	11.4 (-10.5)	13.0 (-14.8)
% Trainable param.\$	100%	3.7%	3.7%	0.1%	0.1%	2.8%	2.8%	0.8%	0.1%	0.1%
No backbone backprop.	✗	✓	✓	✓	✓	✓	✓	✗	✗	✗
Table 3: Fine-grained Classification Top-1 Test Error (SWIN-L) We compare InCA to full fine-tuning (Full FT) along with other adaptation approaches for downstream learning. For each method we summarize the maximum gap in performance compared with the full fine-tuning paragon. In addition we report the parameter efficiency and whether the method requires backpropagation through the pre-trained model.  $\S$  For SWIN-L different activations map sizes leads to different  $\%$  of trainable parameters and we report the maximum for each method. The minimum error over the columns excluding Full FT is in bold.
Top-1 Test Error	Adaptation Efficiency
Method	Avg.	Flowers	WikiArt	Sketch	Cars	CUB-200	# of trainable parameters	GPU Memory (training)	Inference Time (for all 5 tasks)
Full fine-tuning	10.5	0.6	14.7	14.4	10.8	12.2	5×	1×	5×
Linear probing	29.8	10.9	37.2	29.3	44.5	27.9	0.01×	0.17×	1.01×
BA2[7]	15.9	4.3	27.7	20.7	7.9	18.8	1.03×	1×	5×
TAPS [70]	10.4	0.6	15.8	14.0	11.1	10.4	4.12×	1.23×	5×
SpotTune [22]	14.3	3.7	24.2	19.8	7.6	16.0	5.27×	2×	7.3×
InCA	9.8	0.3	15.4	16.8	7.7	8.8	0.06×	0.51×	1.13×
Table 4: Multitask Efficiency and Top-1 Test Error on "ImageNet-to-Sketch" benchmark. InCA is the top performing method on average and is parameter efficient. Further, only InCA and linear probing "share computation" of the pre-trained model and enable "one-to-many" inference execution measured in the "Inference Time" column. BA $^2$  is based on ResNet-50 and can not be applied to ViTs. The rest of the methods are based on ViT-L/16.
We analyze the results of InCA adaptation, focusing on the performance signature of different intermediate representations used as input for the adapter and the relation between the top InCA layers with fine-tuning. Further in Appendix D, we provide a theoretical proof motivating the extraction capabilities of cross-attention as it is used in InCA.
Intermediate representations We consider the intermediate representation signature created by evaluating the accuracy of adapters that utilize different layers. In Figure 3, we review the adapter performance applied to different layer representations. Datasets like CUB-200 and Flood-Depth mostly prefer final representations, whereas for datasets like Aircrafts and Stanf. Cars, the best adaptations use earlier representations with decreasing performance towards the last activations. Curiously, we observe consistency in layer affinity for certain datasets while using different pretrainings for the backbone and even when using different architectures (Appendix Fig. 5).
InCA and partial-tuning In Appendix B we compare InCA with gradually un-freezing the base-model and applying partial fine-tuning on a growing set of layers. We run a set of experiments where we fine-tune a pre-trained model starting at different freezing points, this means we optimize all layers of the network after the freezing point location. For each dataset we construct a "partial tuning curve" where we plot the final test accuracy vs. freezing point (Figure 4). Interestingly, we observe a direct correlation between the layer-location of the top InCA adapter and the point where the partial tuning curve saturates. In particular, the partial tuning test accuracy plateaus (to the tuning of more
Figure 3: InCA Layer Performance Signature Relative test error improvement of InCA adapters attached to different intermediate layers. We evaluate InCA with ViT-L/16 with adapters at each residual block starting from Block 11.
Figure 4: Partial Fine-tuning vs. InCA Vertical dashed lines indicate the top InCA layer; curves show final test accuracy for different partial tuning training runs. Each mark indicates a run where all of the pre-trained model parameters are trained up to a "freeze point" in the network's layers. Note partial tuning performance saturates in close proximity to the optimal InCA adapter layer. This is aligned with our hypothesis that full fine-tuning attempts to surface existing representations already in the network. In that case, performance improves until the tuning approach unlocks the capacity to utilize an existing relevant representation and performance plateaus afterwards. Note here we refer to output layers, e.g., the adapter at block 19 means the adapter corresponding to the final output of block 19, or the input to block 20.
layers) at around the same layer location as the top performing InCA adapter layer location. Namely, the point of saturation of the partial tuning curve is where partial-tuning is capable of harnessing the representation found by InCA at that layer. This gives further evidence that "your representations are in the network" and that fine-tuning surfaces existing representations that can be directly identified by InCA. However, InCA adaptation operates an order of magnitude more efficiently and scales better to large models.
In this paper, we present an efficient and effective alternative to full fine-tuning for transfer learning, closing the gap to full fine-tuning on a diverse set of downstream datasets. InCA has many benefits: it inherently generalizes to different architectures, efficiently scales to massive models, optimizes effectively, and unlocks modular and flexible adaptation applications including multi-task and incremental learning. Further, through the parallel exhaustive search of InCA we are able to better understand the inner representation dynamics of neural networks and construct illuminating "representation signatures" of different models and datasets.
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019. 3, 4
[2] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. 4
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals,
[74] Saining Xie, Ross Girshick, Piotr Dólar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017. 3, 7, 8
[75] Li Yang, Adnan Siraj Rakin, and Deliang Fan. Rep-net: Efficient on-device learning via feature reprogramming. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12277-12286, 2022. 4
[76] Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. Advances in Neural Information Processing Systems, 33:17236-17246, 2020. 4
[77] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. 4
[78] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014. 4
[79] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: a baseline for network adaptation via additive side networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16, pages 698–714. Springer, 2020. 4
[80] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable  $\{\mathrm{detr}\}$ : Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021. 4
Below we provide additional details and results which are not presented in the main manuscript.
A Continual Learning with Open-InCA
With the Open-InCA adapter, each class prediction is isolated using a different dedicated query and classifier vector. For continual learning tasks, in addition to running multiple adapters in parallel as presented for multi-task results in Table 4, Open-InCA enables an even more granular composition of adapter sub-tasks. Recall the Open-InCA adapter architecture is defined as
$$
\left[ v _ {\text {c r o s s}} ^ {1} (\mathbf {z}), \dots , v _ {\text {c r o s s}} ^ {c} (\mathbf {z}) \right] := \operatorname {c r o s s - a t t n} _ {\theta} \left(\left[ z ^ {1}, \dots , z ^ {T} \right], \left[ q _ {1}, \dots , q _ {c} \right]\right)
$$
$$
\operatorname {O p e n - I n C A} _ {\theta} (\mathbf {z}) := \operatorname {d i a g - h e a d} _ {\theta} \circ \mathrm {L N} \left(\left[ v _ {\text {c r o s s}} ^ {1} (\mathbf {z}), \dots , v _ {\text {c r o s s}} ^ {c} (\mathbf {z}) \right]\right)
$$
with LN denoting LayerNorm. Due to the properties of each operator, each class prediction can be computed separately as
$$
\operatorname {O p e n - I n C A} (\mathbf {z}) _ {i} = \langle W _ {i}, \operatorname {L N} (\operatorname {c r o s s - a t t n} _ {\theta} ([ z ^ {1}, \dots , z ^ {T} ], [ q _ {i} ])) \rangle .
$$
Because of this property we can remove a class prediction or add a new class prediction without any effects on other model predictions (as long as the parameters of cross-attn and LN remain fixed). As presented in Sec. 4 we use "query-only-training" which trains new adapter classes while freezing cross-attn, LN and enabling compatibility between task predictions.
When training with "query-only-training" the softmax function,  $\mathrm{softmax}(u) = \frac{\exp(u^k)}{\sum_{i=1}^{c}\exp(u^i)}$ , indirectly injects information of predictions from all classes due the normalization term in the denominator, which means gradients about a particular class  $i$  will include information from other classes  $j$ . Instead, we can achieve complete training separation by using a Sigmoid final activation,  $\sigma(u) = \frac{\exp(u)}{\exp(u) + 1}$ , and a Binary Cross Entropy (BCE) loss that considers each prediction separately. Clearly in "query-only-training" the adapter representation capacity is reduced, since the cross-attention weights are not trained. We present an experiment evaluating the performance of InCA, Open-InCA and "query-only-training" Open-InCA in Table 6 and observe that despite the isolated and reduced parameter set in query-only-training of Open-InCA, the method is still competitive and outperforms Linear Probing on most datasets.
Next we test Open-InCA for class-incremental learning, for which we consider the Split CIFAR-100 incremental learning benchmark. The Split CIFAR-100 dataset is trained with 10 incremental learning episodes each introducing 10 new classes. As in [10], we present the average episode accuracy and forgetting of "query-only-training" Open-InCa and additional baselines.
In particular we evaluate Open-InCa using a ViT-B/16 along with state of the art methods L2P [72], LwF [46] and EWC [24]. Nonetheless, we do not apply any special routing of our learned episodic models and simply combine their predictions. In contrast L2P is a prompt based approach that, during inference, passes each new sample to an auxiliary classifier to predict its corresponding episode (in this case a 10-way classifier) and the corresponding episode model is up-weighted according to the prediction. We believe that with such an auxiliary classifier Open-InCA performance can significantly improve, nonetheless we observe that Open-InCa can simply leverage a larger model efficiently to achieve state of the art accuracy. We leave routing of samples to different learned sub-models as an interesting avenue for future work.
In addition, Open-InCA has additional benefits as compared to typical class-incremental learning approaches:
- Flexible incrementation With Open-InCA different episodes can naturally contain a variable number of classes and episodes can be further decomposed if needed. This is since one can modify the model at the granularity of a single-class predictor via the Open-InCA adapter architecture by introducing or removing additional  $q_{i}$  and  $W_{i}$ .
- Reduced forgetting risk With Open-InCA the ability of adding new classes without forgetting is built-in into the architecture, as prediction of different classes ensures that the
previous class predictions remain the same (i.e., no logit regression) which reduces catastrophic forgetting.
- Parameter and computation efficient The Open-InCA adapter benefits from the InCA approach, which is parameter efficient and computationally efficient during inference (see Table 4) as well as during training (see Fig. 6 for comparison with prompts).
Method	Average Accuracy (↑)	Forgetting (↓)
LP-sequential*	17.7	59.1
Full-FT-sequential*	33.6	86.9
EWC [36]	47.0	33.3
LwF [46]	60.7	27.8
L2P [72]	83.8	7.6
Open-InCA (ViT-B/16)	83.0	9.1
Open-InCA (ViT-L/16)	88.3	7.1
Open-InCA (ViT-H/14)	86.1	8.2