Knowledge Composition using Task Vectors with Learned Anisotropic Scaling
Authors: Frederic Z. Zhang
Paul Albert
Cristian Rodriguez-Opazo
Anton van den Hengel
Ehsan Abbasnejad
Venue: NeurIPS 2024 poster (2024)
Keywords: task vectors
task arithmetic
transfer learning
few-shot learning
test-time adaptation
parameter-efficient fine-tuning
Primary area: machine_vision
TL;DR: We present a learning algorithm where knowledge from different domains can be combined or transferred using task vectors with learned anisotropic scaling.

Abstract: Pre-trained models produce strong generic representations that can be adapted via fine-tuning on specialised datasets. The learned weight difference relative to the pre-trained model, known as a task vector, characterises the direction and stride of fine-tuning that enables the model to capture these specialised representations. The significance of task vectors is such that simple arithmetic operations on them can be used to combine diverse representations from different domains. This paper builds on these properties of task vectors and aims to answer (1) whether components of task vectors, particularly parameter blocks, exhibit similar characteristics, and (2) how such blocks can be used to enhance knowledge composition and transfer. To this end, we introduce aTLAS, an algorithm that linearly combines parameter blocks with different learned coefficients, resulting in anisotropic scaling at the task vector level. We show that such linear combinations explicitly exploit the low intrinsic dimensionality of pre-trained models, with only a few coefficients being the learnable parameters. Furthermore, composition of parameter blocks enables modular learning that effectively leverages the already learned representations, thereby reducing the dependency on large amounts of data. We demonstrate the effectiveness of our method in task arithmetic, few-shot recognition and test-time adaptation, with supervised or unsupervised objectives. In particular, we show that (1) learned anisotropic scaling allows task vectors to be more disentangled, causing less interference in composition; (2) task vector composition excels with scarce or no labelled data and is less prone to domain shift, thus leading to better generalisability; (3) mixing the most informative parameter blocks across different task vectors prior to training can reduce the memory footprint and improve the flexibility of knowledge transfer. Moreover, we show the potential of aTLAS as a parameter-efficient fine-tuning method, particularly with less data, and demonstrate that it can be easily scaled up for higher performance.

Knowledge Composition using Task Vectors with Learned Anisotropic Scaling
Frederic Zhang" Paul Albert"
Cristian Rodriguez-Opazo
Anton van den Hengel
Ehsan Abbasnejad
Australian Institute for Machine Learning
The University of Adelaide
(firstname. llastname)Gdellaideee edu.au http:///h....mmmrrreaaaanaaaaaaa
Abstract
Pre-trained models produce strong generic representations that can be adapted via fine-tuning on specialised datasets. The learned weight difference relative the pre-trained model, known as a task vector, characterises the direction and stride of fine-tuning that enables the model to capture these specialised representations. The significance of task vectors is such that simple arithmetic operations on them can be used to combine diverse representations from different domains. This paper builds on these properties of task vectors and aims to answer (1) whether components of task vectors, particularly parameter blocks, exhibit similar characteristics, and (2) how such blocks can be used to enhance knowledge composition and transfer. To this end, we introduce aTLAS, an algorithm that linearly combines parameter blocks with different learned coefficients, resulting in anisotropic scaling at the task vector level. We show that such linear combinations explicitly exploit the low intrinsic dimensionality of pre-trained models, with only few coefficients being the Iearnable parameters. Furthermore, composition of parameter blocks enables modular learning that effectively leverages the already learned representations, thereby reducing the dependency on large amounts of data. We demonstrate the effectiveness of our method in task arithmetic,, few-shot recognition and test-time adaptation, with supervised or unsupervised objectives. In particular, we show that (1) learned anisotropic scaling allows task vectors to be more disentangled, causing less interference in composition; (2) task vector composition excels with scarce no labelled data and is less prone domain shift, thus leading better generalisability; (3) mixing the most informative parameter blocks across different task vectors prior training can reduce the memory footprint and improve the flexibility of knowledge transfer. Moreover, we show the potential of aTLAS as a parameter--fficient fine-tuning method, particularly with less data, and demonstrate that it can be easily scaled up for higher performance.
1 Introduction
One practical advantage neural networks is the fact that knowledge learned from a previous problem, in the form of network weights, be transferred to solve other related problems. Commonly referred to as transfer learning [6, 73], this technique is often applied when a model trained on general-purpose Imtgset-ImageNet [52] for many years-is fine-tuned on other datasets to improve
"Equal contribution. Listed order was determined by a coin toss. Fred formalised the idea; developed the original codebase; conducted experiments on task arithmetic; and drafted the paper. Paul designed and conducted experiments few-shot recognition, test-time adaptation, parameter-efficiee fine-tuning: investigated numerous properties of task vector compositions; and was crucially involved in every stage of the work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Example loss contour plot scallng coefficients
ApTi 7.
|2 1
a
Go Airi Apt2
1
400
org mni Dinay)
[< (x:00x+ Ein, Am)))))))
300
1/
Learning lask vector compositions
Isotropic unisotropic scaling
Figure I: Illustration of (a) learning task vector compositions (n = 2, On denotes the weights of a pre-trained model) and (b) the flexibility of anisotropic scaling. Assume a task vector T (T(1) -(2) has two parameter
blocks, learning anisotropic scaling grants more flexibility when combining task vectors.
performance on downstream problems. In the past, classification models [18, 53] have been used as the medium for such knowledge transfer, which played a crucial part in the success of detection and segmentation [7, 19, 51, 66-68]. In recent years, foundation models [4] trained on broad data, CLIP [47] particularly, have demonstrated strong performance
a multitude of tasks, even when applied in a zero-shot manner. Besides the conventional way of exploiting the knowledge in these models via fine-tuning, recent works [28. 44, 62] have presented more direct measures to manipulate the network weights. In particular, Ilharco et al. [28] showed that, a task vector, defined as the weight difference between preettrineed and a fine-tuned model, can be used as carrier of the task-specific knowledge learned via fine-tuning. As such, multiple task vectors, when combined with simple arithmetic, can form a multi-tasi model that largely retains its performance across all fine-tuning tasks. Linearisation techniques [44], in addition, have been shown to further enhance this compositionality.
Intrigued by this phenomenon, we investigate the potential of task vectors being knowledge carriers in this paper, by learning linear combinations of them (Figure Ia) for various problems. In particular, parameter blocks, e.g., weights and biases, tend to encode different learned representations in different layers. We thus learn an independent scaling coefficient per block for more precise adjustments tailored to the unique roles of each parameter block. This results in anisotropic scaling of task vectors (Figure 1b), and allows us to exploit their modularity in knowledge composition, granting higher controllability when steering the behaviours of a model for task arithmetic [28].
The potential applications of task vector composition extend beyond model editing. With the coefficients being the only learnable parameters, our method exploits the rich knowledge encapsulated in the task vectors by searching in a low-dimensional coefficient space. As a result, it is a competitive parameter-efficien fine-tuning (PEFT) method, and is particularly effective in cases where labelled data is scarce. This offers new opportunities for few-shot learning [34, 69] and test-time adaptation [35, 57]. Furthermore, for multi-purpos models such as CLIP [47]. variants of the model trained with different data sources or fine-tuned on different downstream tasks are often available [26]. These resources constitute a significant knowledge bank, with task vectors being the knowledge carrier. Many learning problems may be simplified to learning a combination of task vectors.
Our primary contribution is learning algorithm named aTLAS, wherein otherwise complex learning problems can be framed as learning linear combinations of task vectors. The algorithm is broadly applicable to optimising supervised and unsupervised objectives. Its effectiveness is demonstrated in task arithmetic, few-shot recognition, test-time adaptation and parameter-efficient fine-tuning, where we show that (1) learning linear combinations of task vectors directly exploits the low intrinsic dimensionality of pre-trained models [1, 33), resulting in a small number of learnable parameters; (2) standard task vectors, otherwise inferior to linearised variants [44] in task arithmetic, can produce stronger multi-task models with learned anisotropic scaling; (3) aTLSS is effective in low-data regimes, and improves the accuracy of CLIP by 6.5 absolute points averaged over 22 datasets with unlabelled data; (4) aTLAS is complementary to previous few-shot adaptation methods, in that one third of the examples improves upon are unique; (5) aTLAS as a few-shot learning method is less prone to domain shift, and achieves better generalisation on informative parameter blocks from different task vectors can be mixed prior to training, allowing for flexible and efficient knowledge transfer under memory constraints; (7) aTLAS is strong PEFT method when data is limited, and existing PEFT methods such as low-rank adaptations (LoRA) [23] can be scamlessly integrated into aTAS to improve memory efficiency.
datasets; (6) the most

---

2 Models and task vectors
As Ilharco et al. [28] demonstrated, task vectors exhibit many intriguing properties across a wide range of models, such CLIP [47], GPT-2 [46] and T5-based models [48]. To facilitate more in-depth experimentation and analysis, we focus on the CLIP model in this paper, due to its wide availability and manageable size. In particular,
follow previous practice [28, 44] and acquire task vectors by fine-tuning the image encoder, with the text representations frozen. This ensures that image encoders fine-tuned different datasets produce features residing in the same representation space, through common text encoder. The task vectors obtained from these fine-tuned encoders can thus be combined more effectively to form unified multi-task model.
Formally, denote the CLIP image encoder by f X' x θ + Z, such that for input image x E and parameters 0 θ f(; 0) is the learned latent representation for the input image. Denote the weights pre-trained model by Og. and the weights of its fine-tuned variant by 0,, N+, where i indexes dataset D,. We follow Ilharco et al. [28] and define a task vector as т; 0; Ou In addition, we investigate task vectors produced by linearised variants of the image encoder using the first-order Taylor expansion,
g 0) = f(x; 0n) + +0 (0) 0o) Ve 0o).
Ortiz-Jiménez et al. [44] showed that, task vectors obtained from fine-tuning the linearised variants have low disentanglement errors, and exhibit strong compositional properties.
Learning task vector compositions
Parameters neural network, depending on the depth of the layer, often have different significance. For instance, early layers in convolutional neural networks [18, 53] are known for extracting generic, low-level features, such edges, corners, etc., while deeper layers produce features more specific to the task. We recognise the non- non-uniform impacts parameters at different layers can have, and do not perform isotropic scaling on task vectors. Instead, weights, biases and any other forms of parameterisation, which we collectively refer to as parameter blocks, will be scaled independently.
3.1 Proposed method: aTLAS
Formally, denote a task vector with m parameter blocks by = (r..)..... (m)). where each parameter block (() is vectorised, and round brackets denote column vector concatenation. We learn a block diagonal matrix A, parameterised as
( (1)
A
{ (m)
where X E IR is a learnable coefficient; (6) denotes an identity matrix with its number columns matching the dimension 7(); and the superscript j E N+ indexes parameter block. This results in anisotropic scaling of a task vector, that is,
A,T: (1 (1) ,..., (mm)((m)) (m)
(m)
where the subscript E N+ indexes task vector. As such, assuming a supervised objective, finding the optimal composition of task vectors can be defined as the following optimisation problem
arg min Exx.))e,//[[(xx O, + EL AT::).y)1 An....
where C is the loss function for target task; n is the number of task vectors; y is the labels corresponding to inputs x; D, denotes target dataset. The number of learnable parameters, as result, is precisely mm, Let us denote the solution to the aforementioned optimisation problem by ( In inference, model fxx On + Σ T,) will be deployed, which incurs no additional computational cost compared to models trained in the conventional way.

---

100
30
Relative accuracy (%)
uccunccy
50
80
Random
75
Random hasis
Task vector
Task vector
4
10
12
14
0
2
10
12
14
Number of bases
Namber of buses
(a) MNIST
(b) CIFAR100
Recognition accuracy versus the number of bases when optimising in low-dimensional subspace. The accuracy is normalised by that of the fully fine-tuned model. Using task vectors to construct the projection matrix performs consistently better than using random bases (a) MNIST [32], (b) CIFAR100 [31].
Figure 2:
In addition, we investigate the task vectors obtained from fine-tuning linearised variants of the model, i.c. g(x) ) in Eq. 1. Denote such task vectors by F. The learning objective with lincarised task vectors can be derived as follows
arg min E(XY)ED (O + (E A.Ti) vo)(xx)))))) An...nA
(5)
3.2 Relation to intrinsic dimensionality
notable characteristic of aTLAS its parameter efficiency. To offer more intuitions, we refer to previous findings [1, 33] that deep neural networks often produce solutions residing in subspace with much lower intrinsic dimensionality. This is measured by finding a minimum number of parameters, such that learning these parameters (Ô Rd) leads to approximately the same performance as optimising in the full parameter space (0 E R). This can be expressed as follows
0 PO.
(6)
where 0g E R denotes the pre-trained weights and P E Rxd is random projection matrix. We demonstrate that learning task vector compositions leads the same formulation. For brevity of exposition, let us consider compositions at the block level. For the j-th parameter block, we have
( = -000 + Σ (0)00
0b0) + U
(8)
projection matris
learnable parameters
We draw parallel between Eqs. 6 and 8 and note that aTLAS explicitly exploits the low intrinsic dimensionality by learning a small set coefficients. The number task vectors, i.e., n, is much smaller than the dimension of weight vector ( and is analogous the intrinsic dimensionality However, as opposed to using a random projection matrix P, a TLAS constructs the projection matrix from task vectors, making use of the learned representations. To demonstrate its advantage, we use the same number of bases for task vectors² and random bases and show that task vectors consistently achieve higher performance in Figure 2. These results solidify understanding of task vectors being knowledge carriers. We thus set out to apply aTLASS to various applications.
Task arithmetic
Task arithmetic [28] comprised of few tasks aimed editing pre-trained models using task vectors. Following previous practice [28, 44], we conduct experiments under the settings of task negation and task addition on eight image classification datasets (details included in Appendix A).
'A lixed number of task vectors are selected based on blockwise gradient. Details can be found Section 5.2 and Appendix D.6.
'aac random basis of the projection drawn from Gaussian distribution with the mean and standard deviation to match those of the pre- tre-trained weights in the corresponding parameter block, i.e., g(5)

---

Table Performance of task negation averaged eight datasets. Selected results must maintain least 95%
pre-trained accuracy on the control dataset, following previous practice [44]. Best performance in each
section is highlighted in bold. Task vector is abbreviated as L.. Results for each dataset are available in Table 7.
VIT-B/32
VIT-B/16
ViT-L/14
T.V. Methods
Models
Target
Control (t) Target (J)
Control (f)
Target (L)
Control (1)
n/a
Pre-trained
f(x; Ou)
48.14 14
63.35
55.48
68.33
64.89
75.54
Std.
Search
f(x; 8u + ar)) f(x; Ar)
23.22
60.71
19.38
64.66
19.15
72.05
aTLAS (ours)
18.76
61.21
17.34
65.84
17.75
73.28
Search
g(x: Oo a.) g(x AF)
11.54
60.74
10.88
65.54
12.78
72.95
aTLAS (ours)
11.06
61.02
10.16
65.58
12.61
73.14
11
blocke
ba
parameer blocke
001
bo
bP
Wi
Wi
W2
W2
00
11
12
00.6
-0.4
-0.2
0.0
-0.5
0.0
0.5
1.0
0.5
0.0
1.5
learned coef. (lambda)
learned coef. (lambda)
learned coef. (lambda)
(2) Coef. parum. (negation)
(b) param. (addition)
Coef. by layeoleepth (addition)
Figure Box-and-whisker plots for learned coefficients. As each transformer layer consists of fixed set parameter blocks, we visualise the distribution of coefficients for these parameter blocks across all layers, for (a) task negation and (b) task addition, as well as (c) distribution of cocfficients by layer. We denote the learnablc LayerNorm parameters by y and 8. Weights and biases denoted by W and b, respective, with attention layer parameters indexed by superscripts and the MLP parameters indexed by subscripts.
Previous works acquire the optimal isotropic scaling factor task vectors via hyper- pepee-pprmmeter search on validation sets. As such, we learn anisotropic scaling matrices on the same validation sets, and visualise the learned coefficients to shed light on this mechanism.
4.1 Task negation
Task negation aims reduce undesired biases, characterised by the performance, on a target task, while maintaining performance on a control dataset, ImageNet [52] in this case. Denote the validation sets for the target and control tasks by D, and De, respectively. We perform simultaneous gradient ascent on the target task and gradient descent on the control task, described as follows,
arg min E Six.)jepl-L(ff 0, ArT.). [Lx.))eD[[(((x.ee (1
ArT.).)),,
(9)
where те is the task vector for the target dataset, and cross-entropy loss is used. The learning objectives with linearised task vectors can be derived easily based Eq. 5, and so are omitted.
We summarise the task negation results in Table 1, and show that our method significantly improves upon standard task vectors, while the improvement upon linear task vectors is less prominent. In particular, we observe that weights matrices tend to have much larger negative coefficients, as shown Figure 3a. To investigate this, we instead only learn coefficients for the weight matrices, with zero coefficients on other parameter blocks, effectively reducing the number of learnable parameters by two thirds. With ViT-B/32 the backbone, we observe average accuracy of 20.14 (vs. 18.76) on target tasks and 61.23 (vs. 61.21) on the control task, which shows that weight matrices carry majority of the knowledge required for task negation.
4.2 Task addition
Task addition aims at producing a multi-task model using task vectors acquired from range of datasets. We utilise task vectors from the eight image classification datascts, and learn the anisotropic
5

---

Table 2: Performance of task addition averaged across eight datasets. We report the absolute accuracy (Abs.) and the relative accuracy (Rel.) with respect to the fine-tuned model. Best performance each section is highlighted in bold. Task vector is abbreviated as L.v. Results for each dataset are available in Table 8.
VIT-B/32
ViT-B/16
VIT-L/14
T.V.
Methods
Models
Abs. (1)
Rel. (1)
Abs. (t)
Rel. (t)
Abs. (1)
Rel. (1)
n/a
Pre-trained
(xx;0))
48.14
55.48
64.89
Std.
Search
f(x;00 + -. 11x:00 + E, AcTi)
70.12 84.98
77.24
73.63
79.85
82.93
87.92
aTLAS (ours)
93.79
86.08
93.44
91.36
97.07
Search
9x;00 + a +aE, ",.,) 7: 9 0x + A:T.)
74.67
85.17
77.51
86.21
84.75
91.86
aTLAS (ours)
83.42
95.42
85.38
95.10
88.65
96.12
10.1
0.0
5.9
7:5
4.6
4.0
7:2
5.2
0.0
6.7
7.4
6.9
0.0
4.1
6.7
4.1
7.8
0TD
5.4
0.0
10.5
9.5
8.4
9.8
69
0.9
5.2
0.0
9.7
3.7
4.6
7.1
4.8
5.6
4.7
0.0
6.3
5.9
4.6
10.2
5.7
6.6
5.4
0.0
4.5
5.8
4.0
5.7
5.9
4.5
EurOSAT
2.8
4.3
16
3.9
1.3
0.0
0.8
1.5
0.
4.5
3.4
2.0
0.6
211
19
0.9
8.9
12:5
8.5 9.9
8.6
10.3
2.5
2.5
0.0
2.8
1.6
4.0
5.3
3.6
6.9
2.2
2.1
5.3
MN6T
0.9
1.0
2.4
2.0
0.0
4.3
0.8
3.1
0.9 9.9 15
1.7
0.0
11
0.8
2.8
2.8
2.4
2.8
5.0
0.0
2.8
1.5
6.6
0.3 0.3
22
688
9.
0.2
0.1
2.3
RE9ISC45
21
3.3
2
SUNTAT
6.2
6.6
8.0
72
6.8
7.5
0.0
6.9
5.6
5.2
7.3 3.5
2.0
6.5
0.0
3.9
7.4
8.5
6.
6.2
5.0
9.
0.0
5.7
7.9
5 6.0
6.6
0.0
4.3
SVIN
5.2
0.0
10.1 9.1
29.0 13.0
6.5
0.0
1.5
1.6
24
5.0
2.1
0.0
7.0
4.2
Cars
010
Eurodt
NIsST
REstor4s
SUN897
Shig
Care
0TO
EurdAr
GTSR
MRST
RESICCES
SUNE97
Care
Euredar
MAsT
RESIOCEs
SURU97
5Whn
DTD
EuroSAT
oTsa0
MNIST
Swa
(a) Std. [28]. C
6.67%
(b) Lin. [44]. c 5.69%
(c) Std. (nurs), E
4.28%
(d) Lin. (ours) 4.39%
Figure 4: Disentanglement errors between each pair of datasets. Each row reflects the percentage of data in the corresponding dataset that have altered predictions after combining two task vectors. Our method achieves stronger task addition performance result of less interference amongst task vectors.
scaling matrices with the objectives described in Eqs. 4, using the cross-entropy loss. The training data is comprised of the validation sets for all eight dataset, i.e., D, = U D;,
Performance comparison against previous methods is shown in Table 2, where our method yields substantial improvements. Interestingly, we note that with previous methods [28, 44], linear task vectors outperform the standard ones in terms of absolute accuracy, while the converse is true with our method. To investigate this, we compute the pairwise disentanglement error & [44], which measures the percentage of data with inconsistent predictions when task vectors are combined (more details in Appendix C.2). Results in Figure 4 show that standard task vectors with learned anisotropic scaling achieve the lowest average error, indicating less interference in task vector composition. Along with higher fine-tuning accuracy, previously referred to as the non-linear advantage [44], standard task vectors demonstrate stronger performance in task addition.
Furthermore, we again observe that weight matrices have consistently larger coefficients in Figure 3b, and learning coefficients weight matrices alone results in an accuracy of 84.17 (vs. 84.98) using ViT-B/32. This suggests that weight matrices in transformers are the primary knowledge carrier, which enabled knowledge composition and negation. Note that for better clarity in visualisation, we add L regularisation on the learned coefficients during learning, which causes marginal performance drop (84.23 84.98) but significantly improves interpretability. In addition, we observe substantially higher coefficients on deeper layers (Figure 3c). This aligns with our understanding that early layers extract generic features that do not vary significantly across datasets [29], while the deeper layers produce task-specific features and require more careful adaptations.
5 Knowledge transfer in low-data regimes
Beyond model editing for task arithmetic, we explore the idea of transferring existing knowledge in task vectors to previously unseen tasks. To this end, we use the CLIP [47] model and total of 22 image classification datasets, each of which produces a task vector. We defer the details of datasets and the process to acquire task vectors to Appendix A. Denote the set of available task vectors by T = {T tn and the dataset corresponding to task vector T, by D,. For each target dataset D,,

---

ATIAS 15.7%
Tp-Adapher
aTLAS
7G
Tip Acaptor LP++
4.5
-12
72
Accurdyy
69
STLAS Tip
4.1
66
53%
aTLAS w/ LP++
2.0
Life
63
VTLAS WTE
JTT ATTASWIT
Imagelit
imagetott4
ImaggnNet
image
ImageNet
000 mean
60
16
(R)
(a) Fow-shot recognition performance
(b) Images (%) that become conrectly classified
(c) Improvements on OOD datasets
Figure 5: Few-shot experiment results averaged across 22 datasets and three seeds, showing (a) comparison against -fate-of-the-ee tht few-shot methods with ViT TTT-B/32 backbone and (b) percentage of images in the validation sets that become correctly classified after applying few-shot methods. We also show (c) performance difference compared to pre-trained CLIP model on OOD datasets. More detailed results are included in Appendix D.
learn task vector compositions using the subset T \ (Tx), excluding the task vector for the target dataset to avoid information leakage. We test our method in few-shot and test-time adaptation, to demonstrate its effectiveness in low-data regimes. Notably, we observe that task vectors complement existing few-shot methods. Combining a aTLAS with them thus leads to significant improvements.
5.1 Few-shot adaptation
Few-shot recognition requires learning new objects or concepts using a limited amount labelled data-k per class for k-shot. Following previous practice [69], we approach this problem by adapting a pre-trained CLIP model [47] to each target dataset D,. We use the subset task vectors T \ (T₂) and & E 2, 4, 8, 16) images from dataset Dj. During training, we adopt the cross-entropy loss and minimise objectives described in Eqs, 4 and 5 for standard and linear task vectors, respectively.
We compare against Tip-Adapter [69] and LP++ [25] using CLIP with VIT-B/32 backbone, across 22 datasets over three random seeds, and summarise the results Figure 5a. We show that with = 1, approach, aTLAS, significantly outperforms previous methods, demonstrating the effectiveness of knowledge transfer with scarce labelled data. More importantly. we note that the idea task vector composition is highly complementary aTLAS with them results in significant improvements. This is also illustrated in Figure 5b as a Venn diagram, where show the percentage of examples in the validation set that incorrectly classified by the pre-trained model but correctly classified with few-shot methods. Out of the examples a'TLAS improves upon, around half are unique compared against either Tip-Adapter or LP++, demonstrating its complementarity. We also found that standard task vectors generally perform better than their linearised counterparts, and so defer the results of linear task vectors Appendix D.2.
those presented in previous methods. As such, combining
In addition, due to the low number learnable parameters, aTLAS exhibits strong generalisabilliy. To demonstrate this, learn task vector composition ImageNet [52], and test oft-of-domain (OOD) datasets: ImageNet-A [22], ImageNet-R [21], ImageNet-sketch [60] and ImageNetV2 [50]. We summarise the results in Figure 5c, which shows the performance difference against the pre-trained model. Notably, aTTLSS is the only method that consistently improves upon the pre-trained model on OOD datasets, and combining aTLAS with other methods can improve their generalisability.
We also test our method and variants integrated with Tip-Adapter and LP++ using other backbones, including ViT- VTT-(B/16, L/14) and ResNet- (50, 101), J. and find that the results are consistent with those for ViT-B/32. More details can be found Appendix D.3.
5.2 Task vector budget and selection
In practical applications, there may only be a limited number of task vectors available, or the number of task vectors used in training may be restricted due to memory constraints. To this end, we study the influence of task vector budget à on few-shot recognition performance. We experiment with four selection strategies: (1) random selection; (2) feature-based selection; (3) gradient-based selection; and (4) blockwise gradient-base selection. To elaborate, feature-based selection computes the mean image feature representation of each dataset, and selects b task vectors from datasets most similar
7

---

Table 3: Test-time adaptation accuracy averaged over 22 dataset, with x1 standard over 3 random seeds.
LN refers to tuning the LayerNorm layers. CLIP with the ViT-B/32 backbone used. Highest performance highlighted in bold.
Method
Zero-shot
Contrastive (SimCLR)
Entropy (SAR)
Pseudo labelling (UFM)
LN
aTLAS
LN
aTLAS
LN
aTLAS
Accuracy
60.4
60.4 = 0.0 62.7 + 0.1 61.2 + 0.1 62.9 = 0.0 62.2 + 0.1
66.9 = 0.1
to the target dataset. Gradient- selection computes the gradient with respect each of the learnable coefficients, and either select entire task vectors with the highest 71 gradient norm, or select task vectors with the highest blockwise gradient for the corresponding parameter block, and repeat the process for all parameter blocks. The blockwise selection therefore allows parameter blocks across different task vectors be mixed prior to training. More details can be found in Appendix D.6.
For a task vector budget E {1, 2, 5, 10, 21,,1], we summarise the few-shot recognition performance in Figure 6. First, we note that the accuracy of aTLAS does not plateau with the maximum number of task vectors available (21), indicating that more task vectors could be beneficial. Second, we find that selecting task vectors based on feature similarity is a simple yet effec- tive approach with sufficient budgets (b > 5). Selecting whole task vectors with gradient is less effective, gen- erally on par with random selection. Nevertheless, the blockwise variant achieves the best accuracy, particu- larly for very low budgets (b (1, as able to exploit knowledge from more task vectors than the budget dictates. We thus deduce that parameter blocks can function as knowledge carriers in isolation, inde- pendent of the task vectors to which they belong. In fact, a parameter block (11) as part of the task vector T = ..... ((m)) can be considered as task vec- tor by itself, i.e., (7 .,....... 0). This modular nature underscores the potential of task vectors for flexible and efficient knowledge transfer.
70
Accurncy Soummyy
68
Random
Features
Grad. whole
64
Grad, blockwise
2
10
15
21
Task vector budget (b)
Figure 6: Few-shot performance of aTLAS with various task vector budgets. The accuracy is av- eraged across 22 datasets and over three random seeds. Standard deviation 11 overlaid as the етгоr margin. Performance under the 16-shot setting is visualised, while additional detailed results are included in Table 14.
5.3 Test-time adaptation
Test-time adaptation (TTA) [35, 57, 59] assumes no labelled data available for the target task, requiring the model to adapt in an unsupervised fashion. We conduct experiments under the offline adaptation setting, which allows access the target dataset. We consider three categories of self- supervised techniques for TTA: constrastive objectives, entropy objectives and pseudo labelling. Contrastive objectives align representations the same image under different data augmentations. For this category, we adopt SimCLR [9], a simple yet effective method. Entropy objectives encourage the pre-trained model to produce confident predictions on unseen datasets by minimising the entropy over the predictions. This technique was previously explored by Yang et al. [65] in model merging. While effective simpler cases, it can lead to catastrophic collapse in TTA. Therefore, we utilise ofte--of-the-art sharpness-aware entropy minimisation algorithm named SAR [43]. Last, we experiment with an unsupervised pscudo-labelling algorithm inspired by FixMatch [54], which we refer unsupervised FixMatch (UFM). UFM selects equal number of highly confident examples per class as the labelled set, and then employs FixMatch to produce pseudo-labels from rest of the unlabelled examples. Details are available in Appendix E.
We summarise the results in Table and compare our method, i.e., learning task vector compositions, against the conventional approach of tuning the layer normalisation parameters [43, 57, 59]. We show that under all self-supervised objectives, aTLAS achieves higher accuracy than tuning the LayerNorm. In particular, LayerNorm has 30k learnable parameters with VIT-B/32 while our method only has 3.5k learnable parameters. We note that with the UFM objective, TASS performs the best and improves the accuracy by an average of 6.5 absolute points over the zero-shot baseline.
8

---

Table 4: Few-shot recognition performance using standard task vectors LoRAs sparse task vectors. Results are averaged across 22 datasets over three seeds, with standard deviation. The memory consumption for ViT-B/32 backbone is annotated under each variant. For standard task vectors, we learn compositions on all parameter blocks or weight matrices only. For LoRAs as task vectors, we report results with rank 4, 16 and 64,
Standard task
LoRAs as task vectors
Shots (k)
All parameter blocks
Weight matrices
Rank=4 (3.3 GB)
Rank=16 (3.4 GB)
Rank=64 (4.1 GB)
(10.7 GB)
(10.5 GB)
66.0 + 0.2
66.0 + 0.1
64.4 0.1
64.6 + 0.1
65.4 + 0.1
2
67.7 + 0.1
67.0 + 0.2
65.7 = 0.0
66.6+0..2
67.4 0.1
4
70.0 + 0.0
69.4 + 0.2
68.2 = 0.0
6.17+0.1
69.5 + 0.2
8
71.3 + 0.1
70.9 1 0.0
70.2 = 0.2
7.14
70.9 + 0.1
16
72.8 0.1
72.3 1 0.0
7L.7 0.1
7.11
72.0 0.1
6
Relation to parameter-efficient fine-tuning
One of the key advantages of aTLAS is its ability to adapt pre-trained models with few learnable parameters, making it suitable for parameter-efficient fine-tuning (PEFT). Similar to popular PEFT methods such as low-rank adaptation (LoRA) [23], our approach does not introduce additional modules, thereby avoiding an increase in inference complexity. addition, since only the encoded weight matrices in LoRAs have non-zero weight difference, LoRAs are in fact sparse task vectors. They can thus be seamlessly integrated into our method, significantly reducing the memory cost.
6.1 LoRAs task vectors
Due to the sparsity and rank deficiency, LoRAs as task vectors may have limited representation capacity and carry less knowledge. Therefore, they may be inferior standard task vectors for knowledge transfer. We investigate this by learning linear combinations of LoRAs* using our method, under the settings of few-shot recognition. Results summarised in Table 4. We first shed light the impact of sparsity, and compare two variants of our method that either learns linear combinations of all parameter blocks or just the weight matrices. Results show that sparsity results in accuracy decrease of around 0.5% on average, except for the one-shot setting. The rank deficiency, on the other hand, causes more substantial accuracy drop. Nevertheless, this can be largely mitigated by increasing the rank. Using a rank of 64 leads to similar performance compared to learning compositions of only weight matrices in standard task vectors. conclusion, while the sparsity and rank deficiency introduce some performance drops, especially in low-shot settings, LoRAs are competitive alternatives to standard task vectors due their low memory cost.
6.2 Scalability of aTLAS
Despite the parameter efficiency of aTLAS, its performance is not competitive when suf- ficient training data is available. To address this, devise strategy to flexibly scale up the number of learnable parameters as needed. Specifically, we randomly divide each parame- ter block into K partitions, and assign learn- able coefficient to each partition, naturally in- creasing the number of learnable parameters by K-fold. We denote these variants by aTLAS xK. We conduct experiments with these vari- ants using (1,5, 5, 10, 25, 35, 50, 100]% of the to- tal available training data across the 22 datasets used Section The results are summarised in Figure 7, showing that our method consis- tently improves as K increases. Compared to LoRAs, particularly with limited training data,
84
Accuracy (%)
80
76
STLAS X5 (10K)
STLAS (20 (00))
(8021660)
(140)
68
1
5
10
25
35
50
100
Pereentage of training data (%)
Figure 7: Scalability of aTLAS. We compare the accu- racy of our method against LoRAs, and vary the amount of training data. Results averaged over 22 datasets. Detailed results are included in Table 17.
'Details about the process acquire LoRAs included in Appendix D.7.

---

our method achieves higher performance with fewer learnable parameters. With sufficient training data, the variant aTLAS x x1200 leads to higher performance with similar number of learnable parameters, as it is able to exploit the knowledge contained in the task vectors that may otherwise be unobtainable from the target dataset.
Related work
Task vectors and model compositions. Recent studies have demonstrated the possibility of manipulating the behaviours of neural networks directly in the weight space [27, 62, 64]. In particular, task vectors [28], carrier of the domain-speciff knowledge learned through fine-tuning, exhibit strong compositional properties. Such compositionality can be enhanced via linearisation using first-order Taylor expansion [44], and improves model editing with simple arithmetic,, e.g., addition, negation, etc. Yang et al. [65] also investigated the idea of learning layer-wise coefficients to improve task arithmetic. In addition, low-rank adaptations [23], as special forms of task vectors, were shown to also support such arithmetic operations. A recent study [3] also investigated the idea of learning combinations of LoRAs for few-shot recognition.
Mooel-based transfer learning. One interpretation of transfer learning [73] is to exploit the knowledge encapsulated in a pre-trained model for target domain. Amongst various sub-modules of a pre-trained model, transferring the feature extractor is the most extensively studied. This ranges from early convolutional neural networks [18, 53] to modern transformers [58], from vision backbones [14, 37] to language models [13, 46]. For vision applications, classification models trained on ImageNet [52] have been used the medium for knowledge transfer. In recent years, contrastively pre-trained multi-modal models such as CLIP [47] have emerged as a prevelant choice. Such models trained on large volumes of data by aligning image and language representations, leading strong baselines well suited for transfer learning. CLIP representations have since been use for medical imaging [70], semantic segmentation [72], satellite imaging [40], etc.
Model adaptation in low-data regimes. The performance of pre-trained models is often con- strained when applied to specific tasks with limited labelled data. To address this limitation, extensive research has been conducted on few-shot adaptation of CLIP [47]. These studies focus on var- ious techniques, including prompt engineering [71], feature adaptation [16], and more recently classifier adaptation [25, 69]. In addition to few-shot adaptation, test-time adaptation represents an even more challenging scenario where annotated data is available. This typically requires leveraging self-supervised objectives to adapt the model, employing methods such as entropy minimi- sation [35, 43, 59], contrastive learning [8], pseudo labelling [35] and image rotation prediction [57].
8 Conclusion
In this paper, we introduced aTLAS, learning algorithm that leverages the rich knowledge en- capsulated task vectors through learned linear combinations with anisotropic scaling. Unlike conventional methods that learn network parameters, our approach focuses on learning coefficients on task vectors, significantly reducing the number of learnable parameters. We conducted experiments across task arithmetic, few-shot recognition, test-time adaptation and parameter-efficient fine-tuning, demonstrating the effectiveness of our method with supervised and unsupervised objectives. par- ticular, we highlighted several properties of aTLAS, including low disentanglement error, robustness against domain shift, effectiveness low-data regimes, complementarity with existing few-shot methods, etc. These properties paved the way for efficient knowledge composition and transfer.
Limitations. As a task vector is defined with respect to a specific pre-trained model, knowledge composition and transfer are not yet feasible across different architectures. This may become possible with suitable projections and remains part of the future work. In addition, combining large numbers of task vectors can consume substantial amount of GPU memory when training larger models. This can be mitigated by selecting subset of task vectors, using LoRAs task vectors or by offloading the computation of task vector composition to CPU, at the cost of training speed decrease. possible to perform task vector composition at bit-width lower than floating point precision, e.g., 4-bit. Similar features are being tested with popular deep learning frameworks such as Py PyTorch and we expect the memory requirement of larger models to be less of a constraint in the future.
also
10

---

This research is funded in part by the Australian Government through the Australian Research Council (Project DP240103278), and the Centre of Augmented Reasoning the Australian Institute for Machine Learning, established by grant from the Department of Education. We would like to thank Stephen Gould for his valuable feedback on the paper.
Acknowledgements.
References
[1] Armen Aghajanyan. Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing, pages 7319 7328, Association for Computational Linguistics, Aug 2021. URL https: hatp://///laaahoogg/ org 202g/2221.00 mell-long.a0 568..6666... 2,
[2] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pscudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning. In International Joint Conference on Neural Networks (JCNN), 2020. URL httpst //axxii http:///axxvvvvo.o//eeee hwtg/pp///// 1908. 02983. 29
[3] Nader Asadi, Mahdi Beitollahi, Yasser Khalil, Yinchuan Li, Guojun Zhang, and Xi Chen. Does combining parameter--efliieet modules improve few-shot transfer accuracy? arXiv preprint arXiv:2402.1544 2024. URL https://arxiv. org/ h/tpg///////// http//fff/220/ /2402.22222222 15414. 10
[4] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quiney Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt. Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard. Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani. Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and al. On the opportunities and risks of foundation models. arXiv, (tbs/210...// 100.2288.. 2021. URL https: / aarxiv. org/ at/aab/a/22000 2200..221122 21t.//100...0....1 2
[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101-1011--- hol-mining discriminative components with random forests. In Proceedings of European Conference on Computer Vision, pages 446-461. Springer, 2014. URL https: / //innk.ii.////////// springer... com/ chapter /00 10070000777700 /978- 3-919-11-801 119-910999------- 4_29. 16
[6] Stevo Bozinovski and Ante Fulgosi. The influence of pattern similarity and transfer learning upon the training of base perceptron b2. Proceedings of Symposium Informatica, 3(125), 1976.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of European Conference Computer Vision (ECCV), pages 213-229, Cham, 2020. Springer International Publishing. URL https: //aaviv////ivv///////////aa org/ http:ddf/22000 12872. 2
[8] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Fhrahimi. Contrastive test-time adaptation. In Proceedings of IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 295-305, Jun 2022. URL httpp:///artti htt//axivvorrrg prg//pf/ /2204. 10377. 10
[9] T. Chen, Kornblith, M. Norouzi, and G. Hinton. Simple Framework for Contrastive Learning of Visual Representations. In International Conference on Machine Learning (ICML), 2020. URL
attps://arxiv org/ pwtgg/d///002222/ 05709, 8
[10] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification; Benchmark and state of the art. Proceedings of IEEE, 105(10):1865 1880-183,,1888 Oct 2017. URL https://arxiv. www.aaawwwv..ovrraaaaaaaaeee abs// 1703. 00121. 16
[11] Mireea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 3606-3613, Columbus, OH, USA, 24-27 Jun 2014. URL https: //tpri/////////// org/abs. abs/
1311.366 3618. 16
[12] Adam Coates, Andrew Ng, and Honglak Lee. An analysis single-layer networks unsupervised feature learning. In Proceedings of International Conference on Artificial Intelligence and Statistics, pages 215-2225 223. JMLR Workshop and Conference Proceedings, 1. URL https://proceedeee mlr.
press/v15 v1tp wwt.//aaaaeaaaaa waaee/ssaaaaae hatp//aaaa//aeeeeeee moatesllss pdf. 16

---

[13] Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex; Human Language Technologies". pages 4171-41866 Minncapolis, Minnesota, Jun 2019, URL pdf. 10
Arg/N19-11 1423.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of International Conference on Learning Representations (ICLR), 2021. URL https: //openreview htt////rrmidddiiiiieee
10
[15] Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual objective classes challenges: A retrospective. Interational Journal of Computer Vision (IJCV), 2015. URL sattps://limg springer wtt./oooti///eee http//arriiee11.e 1007 0tt0///s11133000 1111111111-11-00011 014-073------00000 16
[16] Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. CLIP-Adapter Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04444 2021. URL https:///araae
org/pdf/ httg//df/244/ /2044. 02285. 10
[17] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech- object category datasct. 2007. URL https://authors
cabbarryccaltedee wwweeehhhedurreeeeeeeeeeeeeeeeee adu/records//ett sasvijyytspe 16
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, Las Vegas, NV, USA, 26 Jun - 1 Jul 2016. URL https: http:p:://aaxi..o//ppt////////// otggp///11//// httpp//////...///..... 03385. 2, 3,
10
[19] Kaiming He, Georgia Gkioxari, Pitor Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pages 2980-2988, Venice, Italy, 22-29 Oct 2017. URL https:/ / wtwaaaavvvvrrr..e/eaeeeee..eea.e org/pdf/ http//pf//.0//////////////111111 17003 06870. 2
[20] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing euroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. In Proceedings of IEEE International Geoscience and Remote Sensing Symposium, pages 204-200 207, Valencia, Spain, 22 27 Jul
2018. URL https: /ttp//aaxvvv//////////////..a.v/ lttpai.wr.g/aa.a/a//ooo..o.oooll aaaaaas///// 10t./909990000..... 16
[21] Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distributio generalization. In IEEE/CVF International Conference Computer Vision (ICCV), URL https: //aziiv wwtps/ooor/////////// /ttppp////22..//////////2 16241. 7
pages 8340-8349, 2021.
[22] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacoh Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 152622 15271, 2021. URL https:/ / arxiiv wtg/pd//190// .07174.7
[23] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. Proceedings of International Conference on Learning Representations (ICLR), 2021. URL https: org/ org/pdf / /2006. 2106.09685.
2,9, 10, 27, 29
[24] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. LoRAhub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307. 13669, 2023, URL /ttpss//aarivvv wwg/po///////// /2307 13269. 28
[25] Yunshi Huang, Fereshteh Shakeri, Jose Dolz, Malik Boudiaf, Houda Bahig, and Ismail Ben Ayed. LP++: A surprisingly strong linear probe for few-shot clip. In Proceedings of TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL https:
org/ wwt/oot//////ooooooooopo /24/4404. 02285. 7, 10, 23
[26] Gabriel Ilbarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and I Ludwig Schmidt. OpenCLIP, Jul 2021. URL https:///aet http:/:://ooi.o/.////////o
wttp///.g//////////// 028.1/enooo....... wtt.eennoo.....3773.. 5143773. 2.29
[27] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open- apen-vocabulary models by interpolating weights In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages
Cur- ran Associates, Inc., 2022. URL / http:////ooo/os...////// neurips. cc/ happer_files/ppper////e file/ http://bcccccbb22222111116666660 cew.aa/eeeeepppeeooooeeeeeeeeeee
Conference, .df. 10
12

---

[28] Gabriel Ilharco, Marco Túlio Ribeiro. Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In Proceedings of International Conference Learning Representations (ICLR), Kigali, Rwanda, 1-5 May 2023. OpenReview.net. URL net/paf?id-6ttKare jxj. 2, 3, 4, 6, 10, 18, 21, 23
[29] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pages 3519-3529, PMLR, 09-15 Jun 2019.
URL https: / /ttp:///ccceeeng...... mlr. press/v97 /kkorbllith99.... html. 6
[30] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In IEEE/CVF International Conference Computer Vision (ICCV) Workshop 3D Representation and Recognition, pages 554 551. Sydney, Australia, 1-8 Dec 2013. URL http://vviii stanford. edu/ paf 3drr13.pdd 16
[31] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009, 4,
16
[32] Yann LeCun, Corinna Cortez, and Christopher C. J. Burges. The mnist handwritten digit datahase, 1998.
URL. http: //ann. lecun. com/ exddbmmiiee . 4, 16
[33] Chunyuan Li, Hecrad Farkhoor, Rosanne Liu, and Jason Yosinski. Mcasuring the instrinsic dimension of objective landscapes. In Proceedings of International Conference on Learnngg Representations (ICLR), Vancouver, Canada, 30 Apr-3 May 2018. URL httpa:::111
WCW. 2, 4
[34] Fei-Fei Li, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Transactions Pattern Analysis and Machine Intelligence (PAMI), 28(4):594-611, 2006. URL http://visioon.
stanford. edu/ /tt///ocuuue/eeeeeeeeeee www.aantte/eee-PPeeeeeeeeeeeee FeiFergguaerona220666 pdf. 2, 16
[35] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Proceedings of International Conference on Machine Learning (ICMI.). volume 119 of Proceedings of Machine learning Research, pages 6028-6033 PMI.R, 13-18 Jul 2020. URI. https: pttpttss://proceedingge mlr http://.........rr/rraeaeaeeeeee
html, 2, 8, 10
[36] Baijiong Lin. LoRA-ToRee Torch: PyTorch reimplementation of LoRA, 2023. URL https: /// teiithub.co com/ Baijiong-B Linng-LinnLenn LoRA- Torch. 27
[37] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of IEEE/CVF International Conference Computer Vision (ICCV), 11-17 Oct 2021. 10
[38] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 6-9 may 2019. OpenReview.net. URL /ttp://////eeeeiee/eeeee wwtp:/eetttteieeeeeeeoeeeeeeraee wwt.oo//o....oookkkooooo..... 16, 23
[39] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.51511 2013.
[40] Sangwoo Mo, Minkyu Kim, Kyungmin Lee, and Jinwoo Shin. S-elip: Semi-supervised lision-language learning using few specialist captions. Advances in Neural Information Processing Systems, 36, 2024. 10
[41] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Audrew Ng. Reading digits in natural images with unsupervised feature learning. In Neural Information Processing Systems (NewrIPS) Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain, 12-17 Dec 201 URL http:/ /ufldllffllll///l/ stanford
http://uuuuuubbrr/////ee https//nnnnnnnnnnnooeee/eea hwwtotouoosseembrr....ooo pdf. 16
[42] Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian conference computer vision, graphics & image processing, pages 722-729. IEEE, 2008. 16
[43] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In International Conference on Learning Representations (ICLR), 2023. 8, 10
[44] Guillermo Jiménzzi Alessandro Favero, and Pscal Frossard. Task arithmetic the tangent space: Improved editing of trained models. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 66727-66754, New Orleans, LA, USA, 10- 16 Dec 2023, Curran Associates, Inc. 2, 3, 4. 5. 6, 10, 18, 21,
13

---

[45] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman. and CV Jawahar. Cats and dogs. In Proceedings of Conference on Computer Vision and Patiern Recognition (CVPR), pages 3498-3505. IEEE, 2012. 16
[46] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog. 2019. 3, 10
[47] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of International Conference on Machine Learning (ICML), volume 139, pages 8748 8763, Proceedings of Machine Learning Research (PMLR), 18 24 Jul 2021. 2, 3, 6, 7, 10, 16,
[48] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67 2020. 3
[49] J. Rapin and O. Teytaud. Nevergrad A gradient-free optimization platform. https: com/ sicebookReeaarch/lleeeegg
2018. 28
[50] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML.), pages 5389-5400. PMLR, 2019.
[51] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN:
Towards real-time object detection with region proposal networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Gareett, editors, Advances in Neural Information Processing Systems (NeurIPS), volume 28, pages 91 99, Montréal, Canada, 7-12 Dec 2015. Curran Associates, Inc. 2
[52] Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michacl Bernstcin, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015. 1, 7, 10, 16, 26
[53] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In Proceedings of International Conference on Learning Representations (ICLR), San Diego, CA,, USA,7 7-9 May 2015. OpenReview.net. 3, 10
[54] K. Sohn, D. Berthelot, C.-L. L, Z. Zhang, N. Carlini, E. Cubuk, A Kurakin, H. Zhang, and C. Raffel. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. arXiv: 2001.07685, 2020. 8, 29
[55] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 12Xiv:1212.0402, 2012. 16
[56] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. In Proceedings of International Joint Conference on Neural Networks (ICNN), pages 1453 1460,, San Jose, CA, USA, 31 Jul-5 Aug 2011. URL https: //1 //eeexxplore.... leee. org/document/ /tt.o//nneeeet3335....o 6033395. 16
[57] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of International Conference on Machine Learning (ICML). volume 119 of Proceedings of Machine Learinng Research, pages 9229-9248. PMIRR 13-18 Jul 2020. httpat http:///pcoceee...././ mlr. press/eeisii
mmr..ng..........mmmmmmm....mmmm prress/vv19/ v119/ httpa///v///////lleeeeeeae sun20b. html, 2, 8, 10
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neu- ral Information Processing Systems (NeurIPS), volume 30, pages 6000-6010. Curran Asso- ciates, Inc., 2017. URL https: htttpp://proceedingg weigigss.euripppp www.ree..cceeeeeeoeeeeeee httppperfileepppprrrrpppp 015664438477ee11aa00304444466644 hate444885aaaaaaaa.aaae
10
[59] Dequan Wang, Evan Shelhamer, Shaoteng Liu. Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In Proceedings of International Conference on Learning Representations (ICLR), 2021. URL https: httpss://pperrvvieeee wttp://oooumm...ddtt33ZZZZL
8, 10
[60] Haohan Wang, Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), pages 10506-10514) 2019. 7
14

---

[61] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report 200-TR-2010--01, California Institute of Technology, 2010. 16
[62] Mitchell Wortsman, Gabriel Ilaarco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong. Ali Farhadi, Yair Carmon, Simon Komblith, and Ludwig Schmidt. Model soups: averaging weights multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of International Conference on Machine Learning (ICML). volume 162, pages 23965-23998, 23-29 Jul 2022. 2, 10
[63] Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories, Interational Journal of Computer Vision (IJCV), 119(1): 3 22, 2016. ISSN 1573- URL https:://dol org/10 1007 s0000/s11133--11 014-0748- 16
[64] Prateck Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. TIES- Merging: Resolving interference when merging models. In Advances in Neural Infor- mation Processing Systems (NeurIPS), volume 36, pages 7093-7115, Curran Associates, Inc., 2023. URL. https: https::/pproceedingg..t neurips. wwtp:/eeeeppppppppppeeeeeeeeeeee www.leee//pperr222///1eeee Conference pdf. 10
[65] Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task Icarning. In Proceedings of International Conference on Leurning Repreventations (ICLR), 2024. 8, 10
[66] Frederic 7. Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting bujan-object interactions. In Proceedings of IEEF/CVF International Conference Computer Vision (ICCV), pages 13319-13327, Oct 2021. URL https: arxiv. org/ wwtgg//a//2/1110 06060.
[67] Frederic Z. Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of hujan-object interactions with novel unary-pairwise transformer. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20104-201 New Orleans, LA, USA, Jun 2022.
URL https: /et:////axiii org/ /t////11....111 01838.
[68] Frederic Z. Zhang, Yuhui Yuan, Dylan Campbell, Zhuoyao Zhong, and Stephen Gould. Exploring predicate visual context detecting of human-object interactions. In Proceedings of IEEE/CVF International Conference on Computer Vision (ICCV), pages 10411-10421 Oct 2023.
[69] Rearui Zhang, Wei Zhang, Rongyao Fang. Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-Adapter: Training-fre frree adaption of CLIP for few-shot classification. In Proceedings of European Conference Computer Vision (ECCV), pages 493-510, Tel Aviv, Israel, 23-27 Oct 2022. Springer Nature Switzerland. ISBN 978-3-031-198333
2, 7, 10, 23,
[70] Zihao Zhao, Yuxiao Liu, Han Wu, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Xiang Li, Zhiming Cui, Qian Wang, al. Clip in medical imaging: comprehensive survey, arXiv preprint arXiv:2312.07355. 2023. 10
[71] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-languee models. Interational Journal of Computer Vision (IVCV,, 130(9):2337- 2348, Sep 2022. ISSN 0920-5691. URL https: //ds::///o... org/ 10. 1007/000///// /tt////11330-00 01653-1. 1. 10
[72] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Linggiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 11 1117-111855 2023. 10
[73] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning, Proceedings of the IEFF, 109(1) (09(1):33-76, 2021. ISSN 0018-9219. 10
15

---

A Datasets and task vectors
We acquire task vectors by fine-tuning CLIP [47] on variety of 22 image recognition datasets:
(1) Stanford Cars [30], (2) DTD [11], (3) EuroSAT [20], (4) GTSRB [56]. (5) MNIST [32], (6)
RESISC45 [10], (7) SUN397 [63]. (8) SVHN [41], (9) CIFARIO [31], (10) CIFARI00 [31], (11)
ImageNet [52], (12) STL10 [12]. (13) Food101 [5]. (14) Caltech 101 [34], (15) Caltech256 [17].
(16) FGVCAircraft [39], (17) Flowers 102 [42], (18) Oxford Pets [45], (19) CUB200 [61], (20)
PascalVOC [15]. (21) Country211 [47], and (22) UCF101 [55]. Fine-tuning was conducted using
AdamW optimiser [38], with learning rate 10-3 batch size of 128 and weight decay of 0.1.
Details of the datasets, additional datasct-specific hyper-parameters, and the accuracy after fine-tuning for an assortment of backbones are shown Table 5, We use the same hyper- eeper-parameters for the
linearised variants of the model.
Table Details the 22 image classification datasets used experiments, number epochs fine-tuning and the final accuracy for different backbones of the CLIP model.
Datasets
Classes
Splits
Epochs
Fine tuned accuracy
Prii
2est
RN50
VIT-B/32
VIT-B/16
()) Cars
196
7,330
814
8,041
35
61.92
68.41
78.26
84,14
91.67
DTD
3,384
376
1,880
76
73.14
72.50
78.
81.91
84.73
(3) EuroSAT
10
21,600
2,700
2,700
12
98.11
98.07
98.89
98.93
99.81
GTSRB
2,664
12,630
97.33
97.51
(5) MNIST
10
$5,000
5,000
10,000
99.62
99.45
99.65
99.69
99.77
(6) RESISCA5
45
17,010
1,890
6,300
15
93.16
93.27
95.94
96.59
97.14
(7) SUN397
397
17,865
1,985
19,850
14
69.65
72.26
75.40
78.12
81,98
(8) SVHN
68,257
5,000
26.032
94.30
94.58
97.38
97.
97.97
CIFARIO
45,000
10,000
93.55
95.43
98.05
98.54
(10) CIFARI00
100
45,000
3,000
10,000
77.55
80.15
89.09
89.95
93.01
(11) ImageNet
1,000
1,2
5,000
50,000
10
76.01
78.19
76.41
81.33
(12) STLI0
10
4,500
500
8,000
90.15
91.35
98.55
99.20
99.62
(13)
Food101
101
70,750
5,000
25,250
15
85.14
87.22
88.68
92.85
95.37
(14) Caltech101
101
6,941
1,736
10
87.62
85.89
94.41
95.22
94,82
(15) Caltech256
257
22,037
2,448
6,122
88.29
90.54
92.60
94,58
97.17
FGVCAireraft
3,333
23.88
40.65
(17)
Flowers102
102
1,020
1,020
6,149
40
60.79
55.47
90.08
94.67
97.884
(18)
OxfordUlIPPet
3,669
75.14
77.49
92.15
93.59
(19)
CUB200
200
5,395
599
5,794
20
58.11
59.56
73,56
77.37
86.35
(20)
Vos
20
7.844
7,818
14,976
10
74.88
76.87
88.42
90.35
92.05
(21)
Country21
211
31,650
10.550
21,100
19.24
20.60
21.99
27.64
38.06
(22)
UCFI01
101
7,639
1,898
3,783
20
81.63
83.00
85.01
89.14
92.55
To shed light on the semantic relationships amongst datasets, we extract the features of images for each dataset, and visualise the distributions as ellipses (Figure 8). Specifically, for each dataset, the mean μ4, E Rd and covariance Σ, E dd image features are computed. Principal component analysis (PCA) is used produce projection matrix / gdx2 from the mean features pe, Sub- sequently, the mean and covariance with reduced dimensionality can be expressed as pP μ, and PTE,P, respectively.
16

---

14
15
1. Cars
2. DTD
11
10
20
3. EuroSAT
4. GTSRB
5. MNIST
6. RESISC45
7. SUN397
SVHN
CIFAR10
CIFAR10D
11. Net
12. STL10
13. Food101
14. Flowers102
15. OxfordllITPet
16. Caltech101
17. Caltech256
18. FGVCAircraft
19. CUB200
20. Pascalvoc
21. Country2.11
22. UCF101
Distributions of datasct features ellipses with standard deviation
13
EuroSAT
11
GTSRB
20
MNIST
9
6. RESISC45
12
7. SUN397
8. SVHN
9. CIFAR10
10. CIFAR100
11. ImageNet
12. STL10
13. Food101
C
14. Flowers1002
C
15. OxfordllIPPet
C
16. Caltech101
18
17. Caltech256
C
18. FGVCAircraR
CUB200
PascalVOC
Country211
UCF101
(b) Distribations of dataset features as ellipses with 3x standard deviation
Figure 8: visualisation of dataset image feature distributions as ellipses. The mean image features for all datascts are visualised the ellipse center, with the dimensionality reduced to 2 using Principal Component Analysis (PCA). The dimensionality of covariance matrices also reduced using the same principal components. We show visualisations with (a) x1 (b) standard deviations. Pre-trained CLIP [47] with ViT-B/32 used extract image features.
17

---

Table 6: Learning rates and training epochs for task negation.
Cars
DID
EuroSAT
GISRB
MNIST
RESISCAS
SUN397
Learning rale Epochs
5 10
10-2
5- 1073
5 10-3
3 10-3
2 --003
3 10-3
5 10
Table 7: Accuracy on larget and control tasks of task negation for each of the eight datasets. Highest performance each section is highlighted in bold. The method search corresponds to model f(x; On + aT..). where a determined via a hyper- syper--arameter parameter search. Our method anixo. corresponds to model [(z; 0g + ACTI), where A, is a learnable scaling matrix.
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Average
Tgt.
Ctr.
Tgt.
Ctr.
Tgt.
Ctr.
Tgt.
Ctr.
Tgr.
Ctr.
Tgt.
Ctr.
Tgt.
Ctr.
Tgt.
Ctr.
Tgt.
Ctr.
Zero-shot
59.73 63.35 43.99
63.35 45.
63.35
32.56
63.35
48.25
63.35
60.65
63.35
63.18 18 63.35
31.61
63.35 48. 18 63.35
1/11/14 VIHB/16 VT-B32
Std. (search)
35.06 60.72
29.41
60,66
11.89
60.68
7.16
60.39
12.67
60.84
31.27
61.26
51.25
60.48
7.03
60.61
23.22
60.71
Std. (amiso.)
28.93 60.52
25.21
60.48
10.44
61.62
5.51
60.67
10.76
62.9
20.95
60.72
46.29
60.82
7.28
62.72
18.76
61.21
Lin. (search)
27.06
60.71
15.27
60.42
0.26
60.63
1.03
61.23
0.06
62.52
6.83
60.68
41.3
59.93
0.54
59.77
11.54 60.74
(aniso.)
42.71
Zero-shot
64.61 68.33
45.11
68.33
55.78
68.33
43.34
68.33
51.79
68.33
65.76
68.33
65.5
68.33
51.98
68.33
55.48
68.33
Std. (search)
24.19
64.41
21.65
63.75
12.41
64.76
7.16
63.95
9.85
65.52
25.48
64.23
47.86
64.16
6.47
66.47
19.38
64.66
Std. (aniso.)
16.63 63.95
20.69
64.6
15,93
67.65
66.37
9.51
68.29
65.17
45.43
6.84
67.93
17.34
65.84
Lin. (search)
23.91
64.74
0.15
64.12
67.41
5.48
64.52
42.39
65.11
06.54
10.88
65.54
Lin. (aniso.)
19.85 64.05
9.68
64.57
0.33
65.91
0.97
65.51
0.01
66.87
7.27
65.39
41.18
65.36
1.97
67.01
10.16
65.58
sero-shut
77.75 75.54 55.32
75.54
61.33
75.54
50.55
75
54
76.36
75.54
71,05
75.54
68.28
75.54
58.45
75.54
64.89
75.54
Std. (scarch)
24.44
71.34
26.91
71.83
8.63
71.46
6.24
71.78
11.15
72.43
17.98
72.07
51.11
71.99
6.72
73.53
72.05
Std. (aniso.)
14.49
71.07
23.94
72.2
12.15
74.81
3.95
71.66
7.29
74.69
25.11
74.29
47.93
72.8
7.16
74.69
17.75
73.28
Lin. (search)
18.57 71.09
13.03
71.92
0.33
73.15
5.57
74.41
5.31
74.32
3.11
72.03
45.79
72.2
10.54
74.51
12.78
72.95
(aniso.)
71.67
1.19
7.6
73.38
B Task negation
The evaluation of task negation is conducted on eight classification datasets (1-8 8 in Table 5), following previous practice [28, 44]. particular, we learn anisotropic scaling using the validation set of each dataset. We also adjust the learning rates and training epochs on the same validation set. The details are shown in Table 6. We report detailed task negation results for each dataset in Table In addition, for more evidence that weight matrices learn large negative coefficients, we show detailed visualisation of the learned coefficients Figure and distribution the coefficients in Figure 10.
18

---

Learned coefficients standard task vectors
Leamed coefficients linear task
Figure 9: visualisation of the learned coefficients for (a) standard and (b) linear task in task negation. Note that coefficients for different datasets are learned independently. despite heing visualised jointly. Large negative coefficients can be observed on weight matrices. CLIP with VIT-B/32 backhone is used.
19

---

blocks
parameter blocks
parameter blecks
bi
by
Wi
We
Wo
W2
-0.5
-0.55
00.5
0.25
learned coel. (lambda)
learned coef. (lambda)
leamned coef (lambela)
DTD (standard)
EuroSAT (standard)
Ren by
1
paramete blocks
parameter blocks
paramete blocks
bf
+
byo
by?
Wi
Wa
Wa
Ws
by
0.5
-0.7
11.50
learned coof. (lambda)
Iearned coef. (lambda)
Icarnod coef. (lambda)
EuroSAT (lincar)
GTSRB (standard)
GTSRB (linear)
Pt
19
blocks
paramtte blocks
by
paramite blocks
bf
Pg Wi
Wi
Wr
W2
Wa
W2
by
be
Icarned caef. (lambda)
learmed cocf. (lambda)
Icarned coef. (lambda)
MNIST (standard)
(h) MNIST (linear)
RESISC4S (standard)
19
blocks
parameter blocks
hy
parameter blocks
by
b9
by
b>
A
Wi
Wi
Wi
by
Wa
b2
b2
learned coef. (lambda)
learned coef. (lambda)
learned coef. (lambda)
RESISCA5 (lineur)
SUN399 (standard)
SUN397 (linear)
parameter blocks
by
b
Wi;
by
Iearned coef. (lambda)
learned coef. (lambda)
SVHN (standard)
(n) SVHIN (lineur)
Figure 10: Additional box-and-whisker plots for the learned coefficients in task negation, beside previous visualisation on Cars (Figure 3a), including results (a, DTD, (c. d) EuroSAT, (e, D GTSRB, (g. h) MNIST, (i.j) RESISC45, (k, D) SUN397 and (m, n) SVHN.
20

---

Table 8: Detailed performance for task addition across all eight datasets. We show additional performance with learned isotropic scaling. which have comparable accuracy to the simple hyper-parameler search used in previous methods [28, 44]. Highest performance in each section is highlighted in bold. The method search corresponds to model f(;; 0u + a Σ T.). where a is determined via a
search. Methods iso, and aniso. use
learned coefficients and correspond to models f(z; Ou + > a.T.) and ((z; 0o + A.T.). respectively.
Cars
DTD
EurSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Average
Abs.
Rel.
Ahs.
Rel.
Ahs.
Rel.
Abs.
Rel.
Abs.
Rel.
Abs.
Rel.
Abs.
Rel.
Abs.
Rel.
Abs.
Rel.
Zero-shot
59.73
43,99
45.19
32.56
48.25
60.65
63.18
31.61
48.14
Std. (scarch)
58.97
75.35
52.29
66.24
80.04
80.94
66.74
67.31
95.96
96.3
70.54
73.53
60.74
80.55
75.66
77.69
70.12
77.24
N5T-B632
Std. (isu.)
56.88
72.68
51.97
65.84
87.96
88.95
73.14
73.77
82.23
N2.52
61.16
63.75
62.88
83.4
87.98
90.34
70.53
77.66
Std. (aniso.)
71.56
91.43
72.66
92.05
93.93
94.98
90.58
96.13
96.47
91.43
68,9
91.94
94.41
84.98
93.79
Lin. (search)
67.14
87.97
58.56
73.2
95.67
97.6 67.62 67.
57
72.13
94.61
95.25
81.25
85.92
63,99
83.78
68. 36
85.5
74.67
85.17
Lin. (iso.)
63.4
83.07
65,59
81,98
93.78
95.69
$2.42
87.97
95.24
95.88
84.78
89.64
58.79
76.96
67.92
84.95
76.49
87.02
Lin. (aniso.)
72.33
94.77
73.03
91.29
95.26
97.2
88.41
94.36
97.27
97.93
88.79
93.89
68.53
89.71
72.53
90.71
83.42
95,42
Zero-shut
64.61
45.11
55.78
43.34
$1.79
65.76
65.5
51.98
55.48
Std. (search)
68.47
81.38
52.82
48
75.0
75.81
71 01
71 86
96.97
97. 27.27
76.35
79.05
66.57
85 23
81.82
83.75
73.63
79.85
VTTW/166
Std. (iso.)
65.55
77.9
50.05
61.1
81.96
82.
74.06
74.93
94.96
95.26
80.94
83.8
60.48
77.43
91.65
93.81
74.96
80.88
Std. (aniso.)
71.79
85,32 67.07
81,88
91.85
92.85
91.9
92.98
97.02
97.32
84.3
87.28
47.28
66.6
85.26
92.66
94.85
86.08
93.44
Lin. (scarch)
72.7
85.26
60,96
73.65
95 0 96 79
70
39
78
37
96.17
83,78
63
71.47
90.67
70.44
84.71
77.51
86.21
Lin. (iso.)
72.6
85.14
70.48
85.15
99.19
93.92
82.43
87.53 58
93.86
94.65
18.14
92.2
61.26
77.72
72.95
87.73
79.24
88.01
(aniso.)
81.78
95.9
93.19
96.33
98.15
94.18
98.6
94.65
72.95
77.11
92.72
85.38
Zero-shot
77.75
$5.32
61.33
50.55
76.36
71.05
68.28
58.45
64.89
Std. (search)
81.69
89.12
64.63
76.27
88.67
88.83
93.88
94.54
98.75
98 98
82.68
85.11
71.3
86.98
81.86
83. 56
82.93
87.92
Std. (iso.)
85.72
93.52
71.38
84.24
83.74
83.9
91.74
92.39
96.5
96.7
91.56
94.25
60.33
73.59
84.49 89.44
Sid. (aniso.)
89,58 97.72 80.85
95,42
98.0
98.18
96.75
97.43
98.48
98.71
93.03
95.77
77.96
95.1
96.24
98.23
91.36 97.07
Lin. (search)
85.13
94.86
74.41
89.28
95.89
97.33
71.82
81 12
98.11
98.75
89.87
93.14
74.29
90.0
82.45
90.38
84.75 91.86
Lin. (iso.)
84.18
93.81
74.41
89.28
94,8
96.32
82.62
86.13
97.16
97.8
91.33
94.65
73.87
89,48
83.01
90.99
88 92.31
Lin. (aniso.)
87.38
97.37
78.51
94.19
95.7
97.14
91.73
95.62
98.39
99.03
93.56
96.96
77.25
93.58
86.7
95.04
88.65
96.12
C Task addition
Task addition is also evaluated on datasets 1-8 shown in Table 5. The hyper-parameten are identical to fine-tuning, except the learning rate is modified to 10-3 We show detailed performance on each dataset in Table 8, where we compare our method against hyper-paramete search used in previous works [28, 44]. and another variant with learned isotropic scaling. We also visualise the learned coefficients with Ln regularisation in Figure 12. It can be easily observed that weight matrices, particularly those the deeper layers, have significantly higher learned coefficients, which conforms to our observations in Figures 3b and 3c.
C.1 Comparison against full full-parameter optimisation
Since our method involves learning the coeffi- cients, unlike previous methods [28, 44] that only require a compare against the direct fine-tuning approach. We fine-tune the pre-trained model on the union of eight datasets, assuming only the validation sets are available. The results are shown in Fig- ure 11. Unsurprisingly, task vector composi- tions, whether the coefficients are searched or learned, are less susceptible to the lack of data, as the accuracy only starts to drop with less than 35% of the data. The performance of full- parameter fine-tuning, however, drops substan- tially as the amount of data available decreases.
85
80
search, we also
Task addition accuracy
75
70
65
aTLAS Search
60
Fine-tune
100
75
50
35
20
Percentage of dataset used (%)
Figure 11: Task addition accuracy averaged across eight datasets (1-8) versus different percentage of validation data used. Standard task vectors are used.
C.2 Disentanglement error
In addition, we provide more technical details and intuitions on the pairwise disentanglement error [44], which was visualised Figure 4. Specifically, we make a few changes to the formulation proposed by Ortiz-Jiménez et al. [44], and evaluate the disentanglement error only with the optimal
21

---

(a) Learned coefficients on standard task vectors
(b) Learned coeflicients on linear task vectors
Figure 12: visualisation of the learned coefficients for (a) standard and (b) linear task vectors in task addition. Note that L1 regularisation has been applied the coefficients during training for better clarity. CLIP with ViT-B/32 backbone used produce the results.
coefficients. Given two datasets D1, D2 and the respective task vectors T1, T2, we overload the definition of function f to denote the mapping from data space X' to the label space V, and define the disentanglement error as
{(T1, T2) EXED, [8 0u AiT1), Ou A/ A5T2))
(10)
{ FI) ExED, 8(f(x: Oo AiTz), O. + Aire))
(11)
where M1, A₁ are the learned coefficients in task addition, and 5 is defined as
8(01, x2)
{
0
σ =1
(12)
+ x2-
The eтrоr metric É(T1, task vector 72 is added to the model, the predicted labels differ from when only using task vector Ti. As task vector
T2) measures the percentage of data in dataset Dj, such that when a second
is acquired from dataset D,. a low disentanglement emorr indicates that most predictions made by Ti highi likely to be correct- will be retained, thus resulting higher performance in task addition.
22

---

Table 9: Average accuracy for few-shot recognition 22 datasets. We report accuracy averaged over random n-shot sample selections, with x standard error. Results are produced using CLIP with ViT-B/32 backbone. For our method, we show results with both standard [28] and linearised [44] task vectors. The best method for each choice of k E {1, 4, 8, 16) is highlighted in bold.
Shors (k)
Tip-Adapter
LP++
aTLAS
aTLAS w/ LP++
aTLAS w/ Tip-Adapter
Std.
Lin.
Std.
Lin.
Std.
Lin.
1
64.3 0.2
64.5
06.2 0.2
0.1
68.9
69.3
68.7 + 0.4
66.7
+ 0.3
2
6.10
0.1
67.6
6.11
0.1
0.2
71.9 0.2
68.9
0.1
4
0.1
69.9
70.0
6.1
0.17
7..8
74.3
0.1
71.6
0.2
8
71.8
0.1
72.3 0.1
0.0
68.2
0.01
73.1
76.5 1 0.1
74.2
0.1
16
73.7
= 0.1
0.1
0.1
6.1
7..8
76.2
78.0
76.7
+ 0.0
D Few-shot learning
D.1 Baselines: Tip-Adapter and LP++
Two variants of Tip-Adapter [69] were proposed for few-shot recognition where the weights of the adaptor are either fixed based features of the few-shot examples further fine-tuned. We only study the fine-tuned variant due to its higher performance. Tip-Adapter has two hyyer-parameterd which in the original paper optimised through typer-parameter search on separate validation set. This practice may not align with the principles of few-shot learning, where access to extensive validation data is typically limited. In addition, Huang al. [25] note that the performance of Tip- Adapter is very sensitive to these syper-parameters. We thus opt to learn these two hyper-parameters parameters together with the feature adaptor through gradient descent. The learning rates for the feature adaptor and the hyper-parametedd are set to 10-3 and 10-1 respectively.
For both Tip-Adapter and LP++ [25], we conduct experiments using the publicly available codebase 5. We train both LP++ and Tip-Adapter for 300 epochs on frozen zero-shot features. We apply a cosine annealing decay for Tip-Adapte and maintain fixed learning rates for LP++ as per the official implementation.
D.2 linearised task vectors
We report the average few-shot accuracy over the 22 datasets in Table 9, which corresponds to results in Figure 5a. In particular, we show results with linearised task vectors, as proposed by Ortiz-Jiménez et al. [44]. As highlighted in Section 4, learned anisotropic scaling allows standard task vectors to achieve stronger performance than the linear variants in task addition. For few-shot recognition, we again observe that standard task vectors result in superior performance in most cases. We, however, note the exception that linear task vectors when combined with LP++ achieve higher performance in the 1-shot setting. Nevertheless, the margin over standard task vectors is not very significant, and aTLAS using standard task vectors when integrated with Tip-Adapter is generally a stronger few-shot model.
D.3 Integrating state-of-the-art methods into aTLAS
We use the AdamW [38] optimiser with learning rate of 10-1 and a weight decay of 10-1, Our method by itself is trained for 10 epochs with ViT backbones and 30 epochs with ResNet backbones.
We show that Wate-of-theear few-shot methods can be seamlessly integrated into our method, since both Tip-Adapter and LP++ focus on the classifier, while aTLAS improves the feature representations. We experiment with two strategies combine aTLAS with previous methods, where we either (1) train our method first and use the frozen representations to train a previous method, or (2) train parameters both methods jointly. Results Table 10 shows that the joint training strategy results in higher performance, particularly in low-shot settings. We therefore adopt the joint training strategy when combing our method with Tip-Adapter. During training, we adopt different learning rates for different parameter groups, that is, 10-1 for learnable coefficients in aTLAS and the in Tip-Adapter, and 10-3 for the adaptor. The joint training takes 20 epochs for ViT backbones and 60 epochs on ResNet backbones, twice the number of epochs when training aTLAS alone.
htt.ah/..coooseeetteeeeaeeeerrea clip- strong-baselinee
23

---

Table 10: Comparison of few-shot recognition accuracy between training our method and Tip-Adapter sequen-
tially and jointly over different shots (k). ViT-B/32 used as the backbone. Results are averaged across three
random seeds. Highest performance in each section is highlighted in bold.
Cars
DID
BumSAT
GESRB
MNIST
RESISC45
SUN397
SVHN
CIFARIO
CIFARIOD
ImageNet
STL10
Food101
Caltec1256
PGVCAirenat
Flowers102
OxfordlUUPPe
CUB200
Pascu/VOC
Country211
Caltech111
UCF101
Averge
Strategy
Seq.
59.7 43.7 64.5 44.6
75.7
63.6
64.0
38.3
89.8
69.2
63.7 97.2 84.0
84,9
19.9
70.2
88.8
$3.0
76.2
17.2
89.5 66.0 64.7
1
Joint
61.4
49.3
72.5
57.5
84.8
70.8
67.5
57.0
91.0
75.5
64.2
97.2
84.0
84.0
22.6
79.0
88.7
54.4
76.2
17.3
89.3
69.9 68.8
Seq.
63.6
48.8
72.6
49.0
85.4
72.8
69.9
64.8
97.2
84.0
86
22.8
78.4
88.8
57.5
76.2
17.2
90.3
69.8 68.0
2
Joint
65.1 55,9 83,4
67.6
86.7
77.3
69.2
56.3
93,4
75,9
64.5
97.5
84.0
88.0
26.2
85.0
89.6
58.8
76.2
18.1 90.4
72,9
71.9
Seq.
68.7 55.5 76.8
58.3
65.8
28.9
86.4
89.0
63.6
76.2
17.2
92.2
73.2 71.0
4
Joint
69.0 63.7 79.6
73.1
90.6
78.9
70.8
67.6
93.9
78.3
64.7
97.5
84.0
87.5
29.3
91.6
89.7
62.8
77.3
18.1
90.4
77.5
74.4
8
Seq.
73.7 64.7 82.3
69.5
91.0
79.8
72.3
43.0
91.9
73.4
66.9
97.7
84.0
89.9
34.7
91.2
90.8
69.3
76.2
18.5
91.9
78.8
74.1
Joint
74.2 69.4 89.5 78.4
94.0
81.1
72.1
60.1
95.0
78.5
65.6
97.8
84.3
88.6
36.6
93.7
90.0 69.0
77.6
19.0
91.7
78.9
76.6
Seq.
77.6 69.0 89.3 79.3
92.8
83.7
74.1
64.6
94.0
79.9
66.8
97.5
84.6
89.2
36.1 94.2
91.4
72.8
80.3
20.1
93.6
80.0
77.8
16
Joint
79.1 68.8 91.4 82.7 93.9
82.8
73.7
63.7
95.1
80.0
67.0
98.1
84.3
88.6 6 39.1
94.5
90.1
74.0
77.0
20.4
91.9
80.4 78.0
Table 11: Detailed few-shot accuracy for each dataset over different shots (k) using ViT-B/32 backbone. We report results averaged over random sceds. In the case where the results are worse than the zero-shot accuracy,
we report zero-shot accuracy. Highest performance and those within range of 0.1 in each section are highlighted
in bold. Tip-Adapter is abbreviated as Tip.
POVCAircrat
OxfordUIPPa
k: Method
Cars
DTT
EuroSAT
GISRB
MNIST
RESISC45
SUNW9
SVHN
CIFARIO
CIFARIOD
ImageNa
STL10
Food101
Caltech256
Flowers102
CUB200
PascalVOC
County211
Caftech
UCF101
Avernge
0 CLIP
|59.7 43.7 42,9 32.7
48.3
60.6
63.2
31.3
89.8 64.2 63.4 97.2 84.0 82.0 19.6 66.6
87,5
53,0
76.2
17.2
84.0
61.6
60.4
Tip
62.6 52.3 55.2
37.5
61.3
67.2
66.5
34.7
89.8
66.4
63.9
97.7
84.0
84.8
22.0
80.6
87.6
55.1
76.7
17.3
87.5
68.2 64.5
LP++
61.5 51.3 60.0
39.5
50.5
68.8
65.8
31.8
89.
66.3
63.9
97.2 84.1 84.7
23.7
81.9
87.5
55.1
76.2
17.2
88.3
69.7
64.3
aTLAS
43.7 74.7
79.5
64.5
8988
70.2
27 84.0
66.6
87.5
76.2
17.2
66.2
aTLAS w/ LP+4
62.2 50,2 72.5
52.8
84.0
69.9
67.2
64.6
92.1
75.6
64.4 97.2 84.0
85.4
24.1
81.6
87.5
56.2
76.2
17.2
89.5
69.4
69.2
aTLAS w/ Tip
61.4 49.3
72.5
57.5
84.8
70.8
67.5
57.0
91.0
75.5
64.2
97.2
84.0
84.0
22.6
79.0
88.7
54.4
76.2
17.3
89.3
69.0
68.8
Tip
64.1 57.4 70.8
43.7
66.0
73.1
68.3
31.8
90.0
66.6
64.5 97.9 84.3 85.6
24.0
87.3
88.0
ST 4
76.2
17.5
88.4
72.5 67.1
LP++
57.3 69.8
69.7
74.7
89.8
67.6
97.4 84.0
86.8
76.2
17.4
67.4
2.
aTL.AS
59.7 43.9 79.5
53.8
86.0
68.0
64.0
0 60.9
91,1 72.5
63.9 97.2 84.0 84.6
21.2
67.4
8
53.0
76.2
17.2
89.8
65.0 67.6
aTLAS w/ LP++
65.8 58.2 80,0
58.4
87.4
74.4
68.5 5 55.1
93.1 76.8
64.9 97.4 84.0 87.0
26.9
87.0 88.5 60.5 76.2
17.5
89.7
72.7
71.4
aTLAS w/ Tip
65.1 55.9 83.4
67.6
86.7
77.3
69.2
56.3
93.4
75.9
64.5
97
84.0
88.0
26.2
85.0
89.6
58.8
76.2
18.1
90.4
72.9
71.9
Tip
65.8 62.3 77.3 48.7
77.9
77.1
70.2
32.8
90.4
68.
65.0
97.2 84.7
87. 28.8
91.5
88.4
62.3
76.2
18.3
90.7
73.9 69.8
I.P++
67.5 61.5 74.2 54.0
72.5
79.1
70.5
34.3
90.8
68.
65.4 98.0 84.3 88.7
27.4
90.1 87.7
62.5 77.3
18.8
91.8
75.5 70.0
aTLAS
60.9 48.6 84.3 66.3
89.6
72.0
65
71.8
91.9
7 64.7
97.2 84.0
85.2
22.9
69.9
88.2 53.5 76.2
17.2
90.7
65.2
70.0
aTLAS w/ LP++
69.9 62.9 85.3 73,6
89.2
76.0
70.8
58.5
93.6
78.0
65.7
97.2 84.4
88.2
28.9
89.2 90.6
64.3 77.0
17.7
91.2
75.8 74.0
.TLAS w/ Tip
69.0 63.7 79.6
73.1
90.6
78.9
70.8
67.6
93.9
78.3
64.7
97.5
84.0
87.5
29.3
91.6
89.7
62.8 77.3
18.1
90.4
77.5
74.4
Tip
71.1 65.6 78.3
58.1
84.9
80.9
71.7
31.3
91.0
68
65.0
97.6
85.0
88,0
31.2
93.1 90.4 66.7 76.5
19.4
91.5
76.5
71.9
LP+4
72.2 65.2 79.4
61.2
82.5
81.8
72.2
31.3
91.0
69.6
97.9
84.7
89.1
92.2
89.9 66.7 78.7
19.3
92.9
76.8
72.4
8
aTLAS
61.6 52.1 90.8
67.2
90.2
74.6
68.9
72.2
93.0
77.3
64.8
97.2
84.0
85.8
24.8
73.0
89,9 55.4 77.0
17.4
91.2
67.8
71.5
aTLAS w/ LP++
73.5 65,2 86.6
73.2
92.8
80.8
72.8
64.2
94.1
79.6
67.3 97.2 84.7
88.0
32.1
91.4
90.2 66.7 76.2
19.0
92.4
77.3 75.7
aTLAS w/ Tip
74.2 69.4
89.5
78.4
94.0
81.1
72.1
60.1
95.0
78.5
65.6
97.8
84.3
88.6
36.6
93.7
90.0
69.0 77.6
19:0
91.7
78.9
76.6
Tip
73.5 67.5 85.8 66.6
89.1
82.3
73
91.3
4
65.5
97.9
84.7
88.6
36.2
94.5
89.6
70.3 76.2
20.3
92.3
73.9
LP+4
75.2 69.7 86.4 64.8
87.4
83.3
74.4
31.3
91.7
70.8
68.2 98.0 85.5
89.1 35.7 92.2 90.6
69.3 76.9
20.4
93.1 79.5 74.2
16
aTLAS
62.9 55.5 92.6 71.3 92.8 77.0
66.5
78.5
93.9
8
65.4
97.3 84.3 86.6
24,9 74.1 90.6 56.5 76.9
17.8 92.8 68.4 72.9
aTLAS w/ LP++
76.9 68.9 89.6
79.3
94.4
84.1
73.5
70.7
94.7
80.6
68.7
97.9
85.1 90.3
34.2
92.0
90.8
69.5
78.4
19.8
93.4
79.9 77.9
.AT.AS w/ Tip
79.1 68.8
91 82.7
93.9
82.8
73.7
63.7
95.1
80.0
67.0
98.1
84.3 88.6
39.1
94.5
90.1
74.0
77.0
20.4
91.9
80.4
78.0
On the other hand, The joint training strategy with LP++ is non-trivial, due to LP++'s super- convergence strategy being designed around frozen feature representations, which would have been updated every iteration by aTLAS. We thus use the sequential strategy to combine aTLSS and LP++. We include detailed results for each dataset with ViT-B/32 in Table 11 and additional results with different backbones in Table 12, where we show our method scales well across different datasets and backbones.
24

---

Table 12: Detailed few-shot accuracy for each dataset across RN50, RNI0I, VIT-B/16, and ViT-L/14 backbones. We report results for same random seed. In the case where the results are worse than the zero-shot accuracy, we report zero-shot accuracy. Highest performance and those within a range of 0.1 in each section are highlighted in bold. Tip-Adapter is abbreviated as Tip.
Methed
MINIST
SVHN
CERAR
CIPARIO
FoodIO
LONCARDER
CUR0000
10F
10
CLIP
54.3
41.2
41.5
35.1
58.1
53.1
(0L)
25.9
71.5
40.3
599
94.2
806
71.3
17.0
66.2
K5.8
46.5
66.2
154
77.6
58.3
55,9
56.3
66.2
14.2
47.2
80.3
7359
51.3
50.2
1.
57.2
64.8
57.4
59.4
63.0
72.3
45.0
64-3
99.6
80.8
31.3
66.0
538
58.0
89.6
64.7
50.9
54.3
41.3
63.7
46.5
75.5
16.0
61.11
9UL7
78.4
54.
60.7
09.3
19.6
79.8
19.1
96.7
85.6
54.5
56-2
15.8
16.88
00.01
63.1
aTTAS Wlp
06.1
-1.7
17.5
78.5
$4.9
62.0
79.0
71.6
-0.8
63.3
97.3
8440
85.2
25.9
75.7
K7.4
535
76.8
17.8
18577
60.4
5009
58.3
66.4
37.5
57.4
66.6
66.5
29.7
77.0
48.0
00.8
95.0
85.7
81.3
23.3
67.2
56.5
67.5
16.2
86.3
06.7
62.3
LP
09.0
52.0
66.1
41.3
51.5
64.5
64.8
29.7
72.1
66.9
94.4
93.6
92.5
21.6
64.2
86.5
56.0
67.5
16.2
68.1
66.1
61.1
ATLAS
55.2
42.0
76.6
52.3
75.0
59.8
60.5
56.8
79.7
57.8
00.7
94.3
90.
90.4
20.6
57.0
85.8
51.0
56.2
15.8
96.9
59.6
51.5
4TLAS
$8.9
63.3
$8.8
62.5
35.9
72.1
41.0
97.1
94.0
$6.5
34.9
67.5
57.3
16.5
17.5
89.0
73.0
64.9
Tip
63.3
61.2
94.4
41.4
68.6
31.0
16.3
29.7
76.9
29.1
61.3
94.4
81.
$2.5
34.3
60.2
86.0
58.0
66.2
16.5
96.5
71.5
54.5
LP
63.2
55.5
76.5
43.4
70.4
69.5
66.8
26.7
74.9
43.0
62.4
94.3
81.1
83.7
24.5
59.2
56.9
50.1
66.7
16.2
$2.0
70.9
64.8
ATLAS
$7.6
43.0
76.7
15.3
82.2
62.0
61.4
53.4
56.9
61.4
94.2
906
905
22.6
69.7
56.5
535
16.8
15
87.1
61.3
64.3
02.9
$8.9
77.9
96.1
70.0
245
71.7
42.1
47.A
84.3
87.
39.3
M87
631
76.3
76.0
67.8
Tip
05.9
63.5
96.5
54.3
57.1
31.5
06.9
20.1
76.7
43.
61.1
95.1
91.6
83.5
304
93.3
15.5
12.
00.6
17.6
69.6
74.0
61.3
LP:1
69.2
64.9
76.7
50.5
76.7
16.7
70.0
25.7
75.5
50.3
6338
95.8
91.6
$4.3
25.8
92.8
$8.0
64.2
18.5
17.0
63.7
74.1
57.5
55.4
51.3
15.4
16.9
84.3
01.2
61.9
01-3
81.1
912
33.6
K76
55.
19-1
15.6
19.6
61-5
66.1
76.4
Tip
71.9
60.1
63.3
63.3
16.0
15.1
69.
32.4
795
51.
67.8
45.0
81.5
85.1
31.1
94.3
67.6
56.9
68.5
18.5
91.9
75.2
700
73.0
17.3
78.0
75.4
71.3
7947
76.6
51.
64.7
96.3
85.1
85.5
31.6
95.5
K9.3
07%
72.5
17.
07.9
77.5
60.
4TLAS
09.4
$7.8
68.5
61.9
67.7
94.9
61.9
95.6
81.5
83.1
34.4
18.7
53.4
68.7
16.1
19.9
60.8
65.3
Tip
69.8
90.6
60.1
41.3
56.7
60.1
19.5
3019
52.1
633
96.1
84,0
$4.8
23.5
71.9
67.6
58.2
69.7
17.4
W7.2
62.0
51.5
61.8
434
56.4
M6.1
10.
K2.
540
618
96.8
81A
$1.0
22.6
16.6
51.
K5.
17.4
16.
69.8
63.0
ATLAS
61.1
136
85.7
507
66.8
63.
61.4
76.0
589
99.8
Tip
67.3
55.4
63.0
37.5
61.9
68.0
65.6
37.0
82.5
52.5
638
97.8
84.8
$5.0
24-1
S7.
87.7
548
64.4
17.6
62.9
70.8
54.5
67.5
67.7
69.5
51.0
53.7
93.3
$4.5
56.5
57.0
72.1
96.6
65.1
58.5
=TEAS
66.3
$1.4
65.3
63.9
85.2
59.9
56.6
R30
60.4
96.3
81.6
$1.8
32.0
K7.1
52.5
68.9
17.5
88.8
61.3
65.7
10/111
122-
05.9
00.0
74.7
43.4
76.8
14.1
67.
15.7
3119
513
647
97-3
84.3
85.1
21-3
15.0
71.
18.1
91.4
74.3
67.
64.0
47.3
A3.6
10.3
10.
60.5
99.4
N7.4
648
83.5
31.5
K7.
51.4
67.7
170
58.5
66-7
ATLAS
70.9
61.5
61.3
13.1
$9.9
15.2
65.9
59.9
852
62.9
6499
97.1
93.3
$4.5
24.3
72.1
$9.0
54.5
14.6
18.0
99.0
67.5
69.4
23.7
54.3
77.3
31.9
69.
52.5
56.
849
85.9
3112
68.4
50.
11.9
18m
9L.
16.3
aTLAS
64.9
40.5
$6.5
66.5
$7.6
73.1
63.2
66.6
18.1
67.0
64.0
95.9
81.9
65.3
24.4
71.
68.1
53.6
71.0
17.1
91.3
66.1
56A
ATLAS WASSw/TT
75.2
63.8
$3.5
79.6
93.0
7K.
67.4
76.6
V3.3
75.2
64.6
47.3
86.1
85.2
35.5
91.3
MA.7
67.8
70.0
17.
91.3
3010
76.9
77.6
62.5
$0.5
15.6
72.1
19.9
02.3
78.3
71.3
LP
84.5
54.8
596
57.6
30.6
(5.1
ATLAS
67.4
55.0
$8.5
70,5
50.5
75.5
66.3
09.0
14.5
77.3
652
V1.8
31.9
85.5
23.1
69.7
91.6
55.5
17.7
17.
06.3
66.2
11.8
ATLAS Tip
90.5
64.9
$5.5
$4.4
93.9
$4.0
72.7
7388
92.9
78.1
65%
07.1
84.1
$9.0
41.0
54.5
91.2
72.8
76.2
20.0
94.5
78.2
10
CLIP
64.0
45.0
56.5
42.9
50.5
67.6
65.5
44.5
50.7
68.2
65.4
97.8
85.7
553
24.4
71.1
$7.2
56.8
17.6
22.9
84A
66.0
54.5
1.P-
67.8
57.8
50.6
15.6
06.0
50.5
9117
660
85.3
86.3
79.6
91.4
61.3
17.6
744
66.9
AS
64.0
45.0
81.3
10.7
$6.5
67.6
KS.
TU.9
V2.8
77.
66.4
47.5
857
ST.1
76.9
71.1
56%
17.6
75.9
96.8
66:1
50.6
WTpp
51.5
64.3
92.1
71.
76.6
94.9
75.0
65.5
96.3
85.9
$4.0
79.1
K5.1
K9.
19.
17.6
70.4
71.9
11.2
3.
aTLAS
76.2
65.3
45.0
68.5
75.5
$4.5
64.5
41.9
92.8
53.0
17.4
58.9
67.
70.0
08.5
52.4
92.9
92.5
24.5
10.8
69.6
69.3
91.3
95.2
95.6
55.
59.
$5.0
31.5
25.2
73.5
91.1
18.6
90.6
56.6
64.3
72.6
72.6
23.2
22.9
91.9
59.8
66.1
14.9
11.4
11.4
aTLAS
70.5
59.6
$6.5
93.6
75.5
72.8
91.6
75.4
69.8
98.4
590
89.2
33.6
89:3
69.4
66.2
71.8
90.5
14.9
74.9
Tip
71.6
60.6
71.4
51.7
16.5
$2.6
12.0
452
92.0
21.5
70.1
89.4
93.5
16.1
45.0
01.5
56.1
75.4
23.8
91.6
75.1
73.5
V11-4016
LP
75.8
62.7
79.3
55.1
$1.9
12.5
73.1
50.
91.6
72.5
768
99.4
99.0
98.
35.8
93.5
91.6
57.5
81.6
33.4
61.9
76.5
14.1
ATLAS
67.2
$1.4
$7.8
07.5
92.7
75.0
(8.
76.3
93.1
76-7
7000
97.8
81.
$9.5
3012
75.1
M64
56.5
17.6
22.9
02.6
66.4
73.3
75.8
66.3
77.9
91.3
$1.4
74.3
90.3
77.0
97.8
95.5
$9/6
35.7
96.7
92.0
70.4
23.5
14.4
74.2
75.0
Hip
73.8
65.6
$4.5
06.2
56.8
$4.1
73.9
48.3
92.9
725
70.6
95.3
$9.1
900
35.3
96.2
92.6
72.9
79.2
34.5
92.6
S1.7
76.5
12
76.4
66.9
44.0
614
$6.5
$4.5
75.
15.6
95.1
711
72.1
05.5
98.
91.2
194
94
92.
12.6
K2.1
24.4
92.7
81.0
76.5
69.3
51.
41A
14.0
$1.5
15.7
MLS
94.3
79.2
70-
56.1
11-1
78.0
90.4
56.5
77.6
919
69.
14.8
30.1
75.1
70.9
Tip
90.4
71.6
$5.0
70.7
$1.5
15.5
75.6
44.6
95.
72.8
71.3
95.5
9500
90.4
44.3
96.5
91.9
76.9
17.7
35.5
93.8
53.0
71.8
90.1
71.5
35.5
68.5
$7.5
$6.8
76.
46.5
03.1
74.3
73.1
95.7
98.2
91.8
41.3
95:2
90.3
75.6
81.5
26.0
04.3
84.2
75.0
ATLAS
09.5
92.6
18.5
95.1
90.0
93.2
89-2
9012
30.3
92.6
59.3
79.2
01.8
71.5
15.6
76.1
68.1
11.8
Thp
79.5
55.3
76.3
15.6
11.5
15.9
72.0
59.5
95.8
77.9
76.3
99.5
93.1
90.1
48.4
95.3
96.5
57.
79.2
32.11
19.1
79.7
15.9
74.5
61.6
16.3
$1.0
$1.5
12.5
528
95.6
74.1
99.4
93.3
40.4
U5.6
70.5
19.3
77.1
16.1
ATLAS
73.5
70.1
50.5
76.0
75.4
90.3
90.5
36.0
99.5
63.1
91.3
74.9
76.5
H1.9
Tip
80.0
63.8
34.3
60.9
57.0
81.7
74.0
58.6
95.7
76.7
76.5
99.4
93.5
91.4
43.8
15.6
95.6
71.3
20.3
32.8
12.3
81.
77.6
LP-+
91.8
64.6
37.9
$1.5
142
793
99.2
93.1
93.8
411
15.5
95.5
73.5
19.9
81.6
17.6
75.5
61.0
$1.2
93.9
$4.8
74.7
14.
95.6
81.5
91.2
92.1
42.
65.8
74.6
99.4
01.
10.5
$6.5
Tip
83.3
67.8
55.3
69.1
56.T
84.7
75%
49%
95.
3000
72.1
49.4
93.1
93.
45.3
97.7
96.9
76.1
79.4
334
91.4
84.3
79.5
VIS1//14
M6.
765
85.1
97.0
84.6
70.3
ATLAS
83.8
68.7
93.9
$4.6
94.1
15.7
76.6
76.7
962
83.9
76.2
99.4
93.2
93.2
45.7
96.2
94.1
76.7
19.7
32.5
94.5
85.1
$2.4
$8.5
16.9
12.4
52.
96.5
50.7
22.1
99.4
93.3
91.8
51.2
16.1
94.5
ML.
19.1
00.2
56.5
$1.0
+TLAS
90.9
62.9
91.4
92.3
$4.6
$3.6
72.5
62.8
97.4
85.8
76.3
99.2
91.0
91.6
42.4
66.1
94.6
58.0
80.0
52.0
02.5
50.3
863
ATLAS W/ Tip
86.1
74.8
95.5
$6.6
9688
67.8
17.1
796
96.7
N5.
76.4
99.2
93.4
93.5
53.8
98.3
94.1
79.7
79.8
32.7
95.5
16.38
$4.1
90.5
10.4
78.
81.5
784
5648
961
95.8
M5.8
$3.4
83.7
56.2
870
94.1
919
$2.0
83.9
58.1
ATLAS
81.4
66.9
94.0
$3.9
$4.4
15.
13.1
63.4
97.9
86.4
77.4
99.2
93.0
93.0
44.3
18.1
94.7
50.
81.5
32.5
04.6
81.3
$1.8
STELAS W/ WTT
89.8
75.0
95.3
$9.1
95.1
$9.9
19.0
18.5
96.9
96.0
764
99.2
91.4
94.2
59.9
04.4
83.4
83.5
55.2
9410
19.1
$5.3
25

---

Table 13: Accuracy of few-shot methods trained on ImageNet [52] and tested on out-of-domain datasets, for
A E (4. 16). Results are produced by CLIP with ViT-B/32 backbone and averaged across three random seeds.
6=4 4
8 16
Method
INet
INer-A
INet-R
INet-Skerch
INerV2
INer
INet-A
INet-R
NNet-Stetch
INetV2
Zero sbot
63.4
31.5
42.3
09.2
62.7
63.4
31.5
42.3
09.2
62.7
50.8 11810.0
69.9
64.1
6..0.10.0
42.9 1.0000
70.4
1171100
Tip Aig-Adatter
64.9
41.3
08.5
03.4
243
40.5
67.7 =00
63.7
LP-+
65.7 0.0
30.1 0.2
4..0..22
67.8 0.1
04.1...0
00.0
2922...11
11.00:00
67.0 1:1
80.0 0.00
ATLAS wr Tip
66.0-0.0
00.2-0011
41.6+-0.11
00.2+00.2
64.5+000
68.0
20-4+00.3
41.4+00.1
68.9 +.2
66.4+00.1
aTLAS w/ Lipe-
06.0 0.0
29.1 022
4..2
67.9 0.4
648=00
68.9 0.0
26.7 + 0.1
41.
67.5 001
6..0=00.0
Cars
2%
0%
9%
DTD
100%
20%
125
2%
5%
15%
2%
11%
7%
2%
25
22%
EuroSAT
20%
MNIST
30%
77%
14.
90%
RESISC45
11%
12%
10%
100%
11%
12%
12%
0%
9%
7%
53%
SUN297
04
0%
7%
7%
0%
115L
11%
10%
11C
5%
13%
RM
25
0%
ON
9%
10%
01%
SVHN
22%
25%
20%
20%
SAY
24%
200%
20%
32%
15%
27%
25%
13%
12%
12%
25%
25%
22%
24%
25%
CIFAR1D
27%
16h
34%
70%
10%
10%
18%
20%
17%
17%
CIFAR100
22%
38h
15%
2:7
2
10%
25%
1/%
25%
20%
80%
ImageNet
STL10
12%
Food101
Caltech256
32%
32%
10%
30%
13%
JUP
22%
FGVCAircraft
74
7%
4%
14
2%
7%
10%
C+
CS
%O%
CNL
TOL
00%
35%
Flowers102
0%
156
0%
2%
2%
2%
3%
0%
4%
3%
2%
4%
2%
ON
DOGS
ON
IN
ON
15
35
0%
OxfordlITPPe
20R
2%
1%
10%
2%
17%
15%
GPS
LES
22%
22
90%
CUB200
9%
2%
0%
2%
X0%
5%
S%
100%
1N
PascalVOC
Country211
Caltech101
UCF101
2%
8%
4%
100%
Average contribution
35%
12%
30h
15%
14h
12h
12%
1.75
1.2%
15%
28%
24
1.%
14%
174
0%
5%
114
1/4
22%
12%
224
010
Ruerdar
07348
RESISOES
GPAR10
Imagedier
S7L18
Foodlol
Cal
OrformirPet
cunzall
Counteg
Calterioz
oorin
Figure 13: Accuracy improvement of aTLSS (16-shot) using one task vector normalised by that of fine- tuning in the full parameter space (all training data). Each column corresponds to unique task vector, and reflects the relative improvement it leads different target datasets. Each row reflects the relative improvement dataset, using different task vectors.
D.4 Out-of-domain generalisation
We show detailed results for out-of-domain generalisation over k E {4, 16) shots in Table 13. These results correspond to those presented in Figure 5c, aTTLAS is the only method that consistently improves test accuracy over the zero-shot model on out-of- images. When combined with LP++ or Tip-Adapter, aTLAS can be observed to improve the out-of-domain generalisation of these methods.
D.5 Relative significance of individual task vectors
In this section, we examine the informativeness of task vector across different target datasets. To this end, we apply aTLAS to each of the 22 datasets using only one task vector. For each dataset, we compute the relative accuracy improvement, that is, the accuracy improvement of aTLAS normalised by that of fine-tuning in the full parameter space. Note that aTLAS is applied under the 16-shot setting, while standard fine-tuning uses all training data available. Results are shown in Figure 13. We first note that certain datasets are more prone to accuracy improvement, such as EuroSAT, MNIST, etc., as indicated by the high percentage across entire rows. This is most likely due to the low intrinsic dimensionality of the task. In addition, we highlight the average improvement in the last row. Notably, certain task vectors, e.g., ImageNet task vector, are particularly informative while others, such as those from Flowers 102 and OxfordPets are much less so. These results illustrate the varying contributions different task vectors can have depending on the target dataset, which also motivated subsequent efforts on careful task vector selection.
26

---

Table 14: Few-shot accuracy when only using budget of b task vectors with different selection strategies. We report results for and shots, The results are averaged over 22 datasets and three random seeds. CLIP with ViT-B/32 backbone is used. Highest performance in each section is highlighted in bold.
Shots (k)
Strategy
b 0
b=1
b=
b ==5
b= 10
b= 15
b = 21
Random
60.39
63.5 = 0.0
61.8 0.1
67.1
69.3
69.7 0.1
70.0 0.0
Features
60.39
65.6 0.1
67.6 0.2
68.8 0.1
69.5
69.8
70.0 0.0
Grad. whole
60.39
0.1
0.2
0.1
69.1 0.11
69.6
7.0
Grad. blockwise
60.39
67.3
68.2
68.9
69.1
69.7
70.0
Random
60.39
6..7
0..0
08.8
72.0 7.0
72.8
Features
60.39
0.0
6.21
7..3
724 0.1
72.8 0.1
16
Grad, whole
60.39
0.1
06.2
0..
0.1
0.12
72.8
Grad. blockwise
60.39
0.1
69.3
7.1
7..6
7.0
72.8 - 0.1
D.6
Task vector budget and selection
In this section, we provide details for selecting budget of b task vectors with feature-based and gradient-based strategies, as introduced in Section 5.2.
Feature based selection. For each dataset Dr, we compute the average image representation ž; of the dataset using the zero-shot model follows
z. = (xcD]f(x 0o)].
(13)
Given target dataset D., we simply compute the cosine similarity between its feature representation 2 and that of each other dataset 2,, i + t. Subsequently, b task vectors corresponding to the datasets with highest similarity will be selected.
Gradient-based selection.
Given a target dataset DD we may directly compute the gradient with respect to the m learnable coefficients for each of the n task vectors. However, as one important motivation behind task vector selection is to reduce memory consumption, using all task vectors to compute the gradient defeats the purpose. Therefore, we instead only load a group of task vectors (b < n),, compute the gradient with respect to their learnable coefficients, and repeat for other groups. With this sequential computation, the gradient across different groups is not calibrated. Nevertheless, we empirically found this strategy to work well. Denote the partial derivative of the loss dataset Dr with respective learnable coefficient X by X(6) such that
x
(xy)ED,
(14)
O1 (5)
(m) and select task vectors with larger gradient. Alternatively, we may select task vectors block by block. Specifically, for the j-th parameter block, we inspect the absolute values of the partial derivatives for the corresponding coefficients, i.e., 2 and select task vectors with higher absolute values. This process is repeated
||500 (..,,...
For the i-th task vector, we may compute its L1 gradient norm, i.c.,
for each parameter block, thus allowing different parameter blocks to have different selections. Crucially, for low budgets, particularly b = 1, this enables our method to effectively exploit more task vectors than the budget specifies. The impact of this can be observed in Table 14 (corresponding to Figure 6), that blockwise selection significantly outperforms other methods when the budget is low.
D.7 LoRAs as task vectors
We fine-tune LoRAs for ViT-B/32 using the LoRA-Torch [36] library with ranks 4, 16 and 64. We stop at rank 64 as we do not observe improvements beyond it. We train LoRAs on attention and MLP layers and use the same settings for full finetuning but with learning rate of 10-3
Table shows additional results using LoRAs as task vectors. We study learning the effect of fine-tuning the LoRAs task vectors on attention layers only (as done in the original LoRA paper [23]) or on the MLPs. Although the original LoRA paper recommendeds training on the attention layers only [23], observe that training on MLP layers is important to produce strong LoRA task vectors.
27

---

Table 15: Additional few-shot recognition results using LoRAs trained attention layers, MLP layers both.
Results are averaged across datasets over three seeds, with x standard deviation. Rank is used for LoRAs.
Task vector type
Method
O-shot
I-shot
2-shot
4-shot
8-sbot
16-shot
LoRAs (Att..)
aTLAS
60.4
63.5 1 0.1
65.1 1. 0.1
66.6 1 0.1
67.9 1 0.1
69.5 1 0.0
(MLP)
66.2 0.11
0..3 68.7
0.1
0.0
LoRAs (Attn, & MLP)
aTLAS
60.4
64.6 = 0.1
66.6 0.2
70.4
718
LoRAs (Ati.. aTLAS
67.1 0.3
0.1
0.14
7.19
78.2 0.1
LoRAs (Atin. & MLP)
aTLAS w/ Tip Adapier
60.4
67.5 1 0.1
10.0.:0.11
0.1
74.9 1..9
77.0 0.1
Standard
L.P++
68.9
0.1
74.1
75.8
77.9 7..0
Standard
aTLAS w/ Tip-Adapter
60.4
08.6 0.4
0.2
7.1
7.1
7.0
Table 16: Few-shot recognition performance with gradient- free optimisation. Results are averaged accuracy over 22 datasets, with x standard error over random seeds.
Scaling
Use gradient
Memory (GB)
Osshot
I-shot
2-shot
4-shot
8-shol
16-shot
Anisotropic
71.7
Isotropic
60.4
63.1 +: 0.45
64.2 + 0.35
65.0 4: 0.12
65.7 +: 0.05
65.4 + 0.14
Anisotropic
60.4
61.3
0.08
61.5
61.5 0.04
61.6 + 0.03
61.6 0.02
D.8
Gradient-free optimisation
An alternative to save memory during training is to utilise gradient-free methods to learn the coeffi- cients. We follow previous work the combination of LoRAs [24] and the nevergrad [49] library. We observe memory usage reduction of 60% from 10GB to 4GB calculated using a dedicated pytorch function Results for few-shot recognition although gradient-free optimisation improves upon the zero-shot model, the performance quickly plateaus as the amount of data increases. In addition, learning anisotropic scaling results in worse performance, most likely due to the relatively high number of parameters.
summarised in Table 16. We show that
http:://wwerabaooooo///ss//a/sss cet.cch.cddammmorr.... w.w.atttealocaaeddoooooomm hhtl
28

---

Table 17: Accuracy after fine-tuning different percentage of training data for variants of aTLAS K and
LoRAs [23]. Results are averaged across 22 datasets. Highest accuracy in each section highlighted bold.
Method
Parums
5%
10%
25%
35%
50%
100%
aTLAS
2k
68.5
71.5
72.6
73.6
74.6
75.4
76.4
72.9
76.2
77.5
78.8
aTLAS x20
40k
69.5
74.0
75.6
77.5
78.2
78.9
80.5
aILAS X80
160%
70.2
74.7
76.2
77.9
78.9
80.0
82.0
aTLAS X1200
2.4M
71.3
75.0
76.6
78.3
80.2
81.5
83,9
(mank16)
83.6
E
Unsupervised FixMatch
We provide more details on the Unsupervised FixMatch (UFM) approach in this section. Fix- Match [54] utilises a labelled set to guide training, which given as part of the semi-supervised learning protocol, while we produce a class-balanced "labelled" set from unlabelled images. Given target dataset D. consisting of unlabelled images, we first rank the examples by the prediction scores from the zero-shot model across C classes. We then select the top min( 100) examples, that is, at most 100 examples per class, as trusted set absence cross-entropy loss is applied the trusted set. For the rest of the unlabelled images, augmented (Open-CLae CLIP [26] validation augmentations) view of an image to produce pseudo-labels, and incur loss the strongly augmented view (Tip-Adapter [69] augmentations). Denote image with weak augmentation by x, its strongly augmented view by x', and the predictions made by network by ý and y', respectively, the unsupervised loss can be expressed
labelled set. The standard
weakly
Ly 8') = -1(mx((r()) > w) o8)) log()'),
(15)
y...5
(y)) 1 30.55
(16)
where 1-) denotes the indicator function, o(-) performs re-normalisation with adjusted temperature scaling, and w is confidence threshold that is linearly adjusted from 0.9 during training. The trusted set is re-estimated at the beginning of each epoch to account for the improving accuracy of model. In training, images the trusted set are over-sampled constitute one fourth of each batch, this practice prevents the model from diverging due to confirmation bias [2, 54].
F Details of aTLAS x K variants
Dividing parameter block into random partitions allows us to introduce more learnable coeffi- cients to each block, thus scaling up our method flexibly. One draw back of this approach, however, that masks for the partitions have to be stored in memory, resulting in a linear memory increase with respect to the size of the parameter block and the value K. To reduce the memory consumption the of aTLAS x K variants, we only apply it to LoRAs task vectors. Nevertheless, these memory requirements could most likely be reduced by exploiting sparse matrices or memory efficient matrix indexing techniques, which we plan to investigate in the future.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the introducing summarise findings in Sections 4, and 6. We clearly enunciate our claims and hypothesis by numbering them by order of appearance in the main body of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalise to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations of our work in a paragraph found in the conclusion, Section 8. The dominant limitation we observe our work is possible memory limitations when applied to larger models with LoRA task vectors. We otherwise tested our approach on larger array of varied image classification datasets, ranging from simple to larger datasets. Some of the datasets we tested on are specialized while others are more generic. This should ensure that our results are generalisable to large array of tasks.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognise that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
30

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not contribute any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data is provided or not)?
Answer: [Yes]
The algorithms we propose is fully explained in Section with all the informa- tion needed to reproduce results being available the appendix Sections A, D.1, D.3, E and F. Furthermore, the pre-trained models and every dataset we use are publicly available.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data is provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognise that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
31

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have released the entirety of our code base and task vector checkpoint under the link provided in the paper. All the data used in this paper is not owned by us and is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wwwppppllcgggpiiiceeeeeeeoooeele httpl///ggiiieee/eeeuuubbiiiii htt///eeeeesssiiiieeeeeeeeeeeeee ssionsolicyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimiser, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All typer-paramteer number of training samples, optimiser used, model architectures and everything else needed to reproduce our results are available the appendix Sections A, D.1, D.3, E and F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We perform 3 independent runs in Figures 5a, ?? and ?? with standard errors being reported. The standard error is computed the standard deviation over 3 runs divided by the number of runs (3). This is calculated with the numpy library.
32

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification:
Although we do not report compute requirements in the paper, on a single
A 1100 4090GPU except for ViT-L/14 experiments that performed 2 A A100. The typical run time for an experiment is 2 hours with ViT-L/14 experiments going up to 8h. We estimate the total compute needed 1000 GPU-hours for repeated results over seeds. The research project includes failed experiements and iterations on the method that are not reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: fctps://neuurpee cctppublccc/hhiccciiieeeeeeee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
33

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We find no direct societal impact for this research paper as research is con- ducted controlled open-sourced datasets. This paper did not study applicability to specialized datasets that can be used to impact society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimising neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognise that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets use are credited in Section A.
34

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
35

---

Justification:
No participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognise that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
36


Reviews:
[{"review_id": "PnDeqZQ2DM", "reviewer": "Reviewer_RiZ9", "summary": "The paper enhances the performance of task arithmetic, a recent model editing technique based on weight interpolation, in vision-language models. Instead of the original task- and parameter-independent scaling coefficients of the task vectors, it proposes to learn anisotropic scaling coefficients from validation data, resulting in significant performance improvements, particularly in task addition. The method also proves effective in few-shot learning and test-time adaptation scenarios and as a parameter-efficient fine-tuning (PEFT) method in low-data regimes.", "questions": "8. **Multi-task learning vs. learning the coefficients.** Differently from the original task arithmetic technique, aTLAS involves learning the optimal merging coefficients for each task and block from data. How would the performance of fine-tuning the pre-trained model on the same number of data and for the same number of steps compare with your task addition results? What about the accuracy of the model on a control dataset that was not used during fine-tuning, e.g., ImageNet?", "limitations": "The paper adequately discusses its limitations. I do not foresee any potential negative societal impacts arising from this study.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Originality:** While some parts of the method are not entirely novel (see Weaknesses section), overall, aTLAS goes beyond previous work. The few-shot adaptation application and its relation to PEFT (e.g., using LoRAs as task vectors) are also original contributions to the field of model editing/merging.\n- **Quality:** The work is generally of good quality.\n- **Significance:** Editing foundation models is an emerging field with promising real-world impact. The results obtained with the proposed method are very good. Specifically, the authors merge the parameters of 8 ViT-L-14 CLIP models while retaining 97.07% of the performance of the single models (see Tab. 2).", "weaknesses": "**Originality and references to previous work**\n\n1. The idea of learning task-wise and layer-wise scaling coefficients for the task vectors is one of the main features of Adaptive Model Merging (*AdaMerging*, Yang et al., 2024). This work, which is not currently referenced, must be duly cited, along with a detailed discussion of the similarities and differences between aTLAS and AdaMerging. \n2. Similarly, the idea of using test-time adaptation techniques such as entropy optimization was also present in Yang et al. (2024).\n\n**Clarity and missing details** \n\n3. The paper does not report the essential methodological details of aTLAS. Specifically, the loss, optimizer, learning rate, and hyperparameters for learning the scaling coefficients are missing.\n4. The disentanglement error (line 143) must be defined and briefly explained. Moreover, as it seems to slightly differ from the original metric defined in Ortiz et al. (2024) – as likely it is computed only for the best set of scaling coefficients – this difference should also be mentioned.\n5. In lines 153-155, some citations could be added, e.g., to the fact that the representation built by neural networks is often hierarchical and increases in complexity with the layer depth.\n6. The experiments with ResNet backbones are never mentioned in the main text. \n7. A comparison of the computational costs compared to standard task arithmetic is missing and should be provided.\n\nYang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X. and Tao, D., 2024. AdaMerging: Adaptive model merging for multi-task learning. The Twelfth International Conference on Learning Representations.\n\nOrtiz-Jimenez, G., Favero, A. and Frossard, P., 2024. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of the novelty, quality and impact of our work, and we are thankful for the feedbacks. In what follows, we now address the questions and concerns.\n\n**W1. Comparison to AdaMerging (Yang et al., 2024)**\n\nWe thank the reviewer for suggesting this comparison, and have added this in our revision (Section 3, Tables 2 and 3) with appropriate citations. In short, the idea of anisotropic scaling is a more general formulation, while layer-wise scaling in AdaMerging is a specific variant. Besides, AdaMerging is designed for model merging, similar to task addition, which is one of the five applications we investigated. Below, we detail the key differences between our work and AdaMerging.\n- **Formulation**. We formulated anisotropic scaling as $\\Lambda \\mathbf{\\tau}$, where $\\mathbf{\\tau}$ denotes a task vector and $\\Lambda$ is a block-diagonal scaling matrix. When parameters in the same layer share one scaling coefficient, this formulation specializes to AdaMerging. However, for complex use cases such as parameter-efficient fine-tuning (Sec. 6.2 and Appendix F), we show that scaling different rows, columns or random partitions of weight matrices differently is necessary to increase the representation power. This variant can also be captured by our formulation. In addition, AdaMerging constrains the learned coefficients to be in $[0,1]$ while aTLAS does not. We observe that larger or negative coefficients can be beneficial for task addition in Fig. 10. **The formulation of anisotropic scaling therefore covers a wider range of use cases**.\n- **Task vector variants**. We additionally studied combining linearized task vectors (Ortiz-Jimenez et al., 2023) in Section 4 and Appendix D.3, and using LoRAs as sparse task vectors to reduce memory consumption (Section 6.1).\n- **Training objectives**. AdaMerging uses entropy as an unsupervised objective, while we experimented with constrastive, regularized entropy and pseudo-labelling objectives in Section 5.3. Note that our unsupervised pseudo-labeling algorithm UFM outperforms the entropy objective for test-time adaptation (Table 3).\n- **Applications**.  AdaMerging focuses on model merging, whereas our work investigates additional applications such as task negation (Sec. 4.1), few-shot recognition (Sec 5.1), test-time adaptation (Sec 5.3) and parameter-efficient fine-tuning (Sec 6.2), which, as noted by the reviewer, are original contributions to the field. \n- **Novel insights for model merging**. As discussed in L141-146, anisotropic scaling reduces interference when merging task vectors (Fig. 3), a novel insight uncovered in our paper, highlighting its advantage over isotropic scaling and negating the need for linearized task vectors.\n\nLast, we kindly note that AdaMerging's publication in ICLR'24 coincides with the submission period of NeurIPS'24. Nevertheless, we have added comparison and acknowledgements where appropriate throughout the paper and thank the reviewer for the suggestion.\n\n**W2. Entropy optimisation in AdaMerging**\n\nWe thank the reviewer for pointing this out and have added a discussion in L224 to acknowledge this. We highlight that UFM, the pseudo-labelling algorithm we designed, outperforms (regularized) entropy optimisation (SAR, Table 3). AdaMerging has also been included as a baseline in this table.\n\n**W3. Technical details**\n\nThey can be found in Appendix A for task addition or negation; Appendix D.1 and D.2 for few-shot learning. In our revision, we have added hyperlinks at L160 for clarity.\n\n**W4. Disentanglement Error**\n\nThe reviewer is correct that the disentanglement error is evaluated at the optimal coefficients. We have added detailed mathematical formulas in our revision for clarity, as follows\n\n$\\xi(\\tau_1, \\tau_2) = \\textbf{E}_{\\mathbf{x} \\in \\mathcal{D}_1} \\big[ \\delta \\big( f(\\mathbf{x}; \\mathbf{\\theta}_0 + \\Lambda^\\star_1 \\mathbf{\\tau}_1), f(\\mathbf{x}; \\mathbf{\\theta}_0 + \\Lambda^\\star_1 \\mathbf{\\tau}_1 + \\Lambda^\\star_2 \\mathbf{\\tau}_2) \\big) \\big]$,\n\nwhere $\\Lambda^\\star$ denotes the optimal coefficients and $\\delta(x_1, x_2)$ is a distance function that returns 1 if $x_1 \\neq x_2$ and 0 otherwise. \n\n**W5. Citations**\n\nWe have added references to Yosinski et al. (2014) and Kornblith et al. (2019) in our revision.\n\n**W6. ResNet Experiments**\n\nWe have added a hyperlink to the results with ResNets (Table 13) in our revision.\n\n**W7. Computational cost**\n\nStandard task addition performs hyper-parameter search on one scaling coefficient ranging from 0 to 1, with an interval of 0.05, and therefore runs inference 21 times. Our method is trained for 10 epochs. Using an RTX 4090, our method takes 12min to train while the hyper-parameter search takes 20min.\n\n**Q1a. Comparison against multi-task fine-tuning**\n\nAs shown in figure in the attached PDF, our method consistently outperforms multi-task fine-tuning using different percentage of the validation data, since the training data is considered unavailable. However, with more data, fine-tuning is expected to achieve better performance.\n\n**Q1b. Performance on a control dataset**\n\nWith our method, the merged model achieves 58.1% accuracy on ImageNet, which retains 91.6% of the zero-shot accuracy 63.4%. In comparison, the model from multi-task fine-tuning achieves 57.3% accuracy on ImageNet, which shows our method generalizes better.\n\nReferences:\n- Yang et al.  AdaMerging: Adaptive model merging for multi-task learning. ICLR'24\n- Ortiz-Jimenez et al. Task arithmetic in the tangent space: Improved editing of pre-trained models. NeurIPS'23.\n- Yosinski et al. How transferable are features in deep neural networks? NeurIPS'14.\n- Kornblith et al. Similarity of neural network representations revisited. ICML'19.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "auJIDtei0M", "reviewer": "Reviewer_GVMe", "summary": "The paper introduces a method named aTLAS, which leverages task vectors and anisotropic scaling to enhance knowledge composition and transfer in pre-trained models. The authors investigate whether components of task vectors, particularly parameter blocks, exhibit similar characteristics and how these can be used to improve knowledge composition and transfer. The effectiveness of the proposed method is demonstrated in various tasks such as task arithmetic, few-shot recognition, and test-time adaptation.", "questions": "- How does the method handle potential conflicts when combining task vectors from very dissimilar domains? Will it negatively affect performance?", "limitations": "- Knowledge composition and transfer are limited to the specific pre-trained model architecture, which may restrict its applicability across diverse model architectures.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The introduction of anisotropic scaling at the task vector level is novel and offers higher controllability in model behavior, particularly for task addition and negation.\n- The method is thoroughly validated across multiple tasks and datasets, showing significant improvements in performance.\n- aTLAS demonstrates strong parameter efficiency, making it suitable for scenarios with limited data.\n- The method complements existing few-shot adaptation techniques, leading to additional improvements in performance when combined.", "weaknesses": "- Knowledge composition and transfer are limited to the specific pre-trained model architecture, which may restrict its applicability across diverse model architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for recognizing the novelty, parameter efficiency and thorough experimentation in our work, and we are thankful for the feedbacks. We now address the questions and concerns as follows.\n\n**W1. Knowledge composition and transfer are limited to specific pre-trained model architecture.**\n\nWhile the experiments primarily use ViT-B/32 as the visual encoder, we showed that the proposed method works consistently across other architectures including ViT-B/16, ViT-L/14 (Table 1, 2), ResNet50 and ResNet101 (Table 13). Nevertheless, we acknowledge that aTLAS cannot combine task vectors obtained from different architectures, or transfer the knowledge in task vectors to a different architecture. This may require finding appropriate projections, and remains part of the future work.\n\n**Q1. Task vectors from dissimilar domains.**\n\nWe studied the potential conflicts between task vectors using disentanglement error (Ortiz-Jimenez et al., 2023), as shown in Figure 3 of the paper. Specifically, Each row reflects the percentage of data in the corresponding dataset that have altered predictions after combining two task vectors. Therefore, low disentanglement error indicates low interference between task vectors. More importantly, this interference also depends on the dataset. For two datasets A and B, combining their corresponding task vectors into one model may hurt the performance on one dataset more than the other. \n\nOur method, particularly with standard task vectors (Figure 3c), generally decreases the disentanglement error across all pairs, which very effectively reduces the conflict between task vectors, whether they are obtained from similar or dissimilar domains.\n\nReferences:\n- Ortiz-Jimenez et al. Task arithmetic in the tangent space: Improved editing of pre-trained models. NeurIPS'23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDzo9FVMVM", "reviewer": "Reviewer_TrF6", "summary": "This paper presents a method called aTLAS, which leverages task vectors to enhance transfer learning in neural networks. Task vectors represent the difference in weights between a pre-trained model and its fine-tuned variant. aTLAS introduces anisotropic scaling to these task vectors by learning different coefficients for each parameter block, which allows for more effective composition of knowledge from different tasks. The method is tested in various scenarios, including task arithmetic, few-shot recognition, and test-time adaptation, demonstrating improvements in performance and efficiency.", "questions": "See weaknesses section", "limitations": "See weaknesses section", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- An interesting way of using task vectors: the paper extends the concept of task vectors by introducing anisotropic scaling, which enhances the flexibility and effectiveness of knowledge transfer. This approach allows for fine-grained control over the composition of different task vectors, leading to better performance in multi-task settings.\n\n- Evaluation and analysis: the method is evaluated across multiple tasks, including task arithmetic, few-shot learning, and test-time adaptation. The comprehensive set of experiments provides strong evidence for the effectiveness of the proposed approach. It provides insights on the behavior and importance of various parts of the neural network during task vector composition.\n\n- Parameter efficiency: aTLAS is shown to be a parameter-efficient method for fine-tuning, which is particularly valuable in scenarios with limited data. The ability to achieve high performance with fewer learnable parameters is a significant advantage for practical applications.", "weaknesses": "- **Only CLIP?** the method is primarily tested on the CLIP model and might not directly generalize to other architectures without significant modifications. Future work should explore the applicability of aTLAS across a broader range of model architectures.\n\n- **Computational complexity (is this scalable)?**  While the method is parameter-efficient, the computation of task vector compositions during training can still be resource-intensive, especially for large models. Strategies to optimize this process or reduce its computational footprint would be beneficial.  The method's scalability with an increasing number of task vectors is not fully explored. While the paper shows that performance improves with more task vectors, it is unclear how this scales with very large sets of task vectors or in more complex multi-task environments. Also, earning different coefficients for each parameter block introduces additional complexity. The benefits of this complexity should be weighed against the potential for simpler approaches that might achieve similar results with less computational overhead.\n\n- **How exactly should one select the task vectors?** The selection of task vectors for composition can significantly impact performance. The paper discusses various selection strategies but does not provide a definitive approach. Further research into more sophisticated selection mechanisms could enhance the method's robustness and effectiveness.\n\n\n- **How to leverage this in real world?** The experiments are conducted in controlled settings with well-defined datasets. Evaluating the method's performance in more diverse and real-world scenarios would provide a better understanding of its practical applicability and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are encouraged that the reviewer find our method interesting and the experiments comprehensive and insightful. We are thankful for the feedbacks. Below, we address the questions and concerns.\n\n**W1. Applicability across different model architectures**\n\nFor task arithmetic, we included results with ViT-{B/32, B/16, L/14} backbones (Table 1, 2) following previous practice (Ilharco et al., 2023 and Ortiz-Jimenez et al., 2023). For transfer learning applications including few-shot adaptation and test-time adaptation, we additionally showed results with ResNet-{50, 101} backbones besides ViTs (Table 13). Beyond CLIP models, Ilharco et al. (2023) showed that the properties of task vectors also apply to GPT-2 and T5 models. We believe task vector composition can also be applied to these models and plan to investigate in future work.\n\n**W2. Scalability**\n\nResearch on larger pools of task vectors remains part of the future work. In such scenarios, LoRA task vectors emerge as the current optimal solution for mitigating memory usage. As shown in Fig. 6a and Table 14 in Appendix F, LoRA task vectors substantially reduce memory requirements while maintaining an accuracy score comparable to full task vectors, demonstrating their feasibility and potential for large number of task vector. This finding underscores the potential of LoRA task vectors as a memory-efficient solution for exploiting extensive collections of task vectors. \n\n**W3. Definitive approach on task vector selection**\n\nThank you for pointing this out. We find in Fig. 5 that block-wise gradient-based selection is the best strategy, especially with low budget on task vectors. Therefore, it is our recommended approach and have been explicitly stated at L213 in our revision.\n\n**W4. Applying aTLAS to the real world**\n\nWe have demonstrated strong results of our method in five applications. In particular, few-shot adaptation, parameter-efficient fine-tuning, etc., have direct use cases in real-world scenarios. We also conducted experiments across 22 datasets that cover a wide range of domains, in order to test its general practicality. We believe the observations we made on these datasets can reasonably reflect the challenges in the real world.\n\nReferences:\n- Ilharco et al. Editing models with task arithmetic. ICLR'23\n- Ortiz-Jimenez et al. Task arithmetic in the tangent space: Improved editing of pre-trained models. NeurIPS'23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank each reviewer for dedicating their time to reviewing our paper. We are encouraged that the reviewers find our work novel/original (Reviewers [RiZ9](https://openreview.net/forum?id=G9OJUgKo4B&noteId=PnDeqZQ2DM), [GVMe](https://openreview.net/forum?id=G9OJUgKo4B&noteId=PnDeqZQ2DM)), our experiment results thorough (Reviewers [GVMe](https://openreview.net/forum?id=G9OJUgKo4B&noteId=PnDeqZQ2DM), [TrF6](https://openreview.net/forum?id=G9OJUgKo4B&noteId=UDzo9FVMVM)) and impactful ([RiZ9](https://openreview.net/forum?id=G9OJUgKo4B&noteId=PnDeqZQ2DM)). In particular, reviewer [RiZ9](https://openreview.net/forum?id=G9OJUgKo4B&noteId=PnDeqZQ2DM) appreciates our contribution in applications such as few-shot adaptation and parameter-efficient fine-tuning, which goes beyond previous work. Reviewers [GVMe](https://openreview.net/forum?id=G9OJUgKo4B&noteId=PnDeqZQ2DM) and [TrF6](https://openreview.net/forum?id=G9OJUgKo4B&noteId=UDzo9FVMVM) appreciate that our experiments provide insights on the behaviors of task vector compositions, such as our method being complementary to existing few-shot methods.

In response to the question from reviewer [RiZ9](https://openreview.net/forum?id=G9OJUgKo4B&noteId=PnDeqZQ2DM), we ran experiments to compare our method aTLAS against multi-task fine-tuning, and have included the new results in the attached PDF. Specifically, we show that our method outperforms multi-task fine-tuning significantly and consistently when using different percentage of data. The validation sets are used in this case following previous practice by Ilharco et al. (2023), because the training data for foundational models may be, and often is unavailable.

We hope these results answer the question. We remain available throughout the discussion period in case of further questions and discussions. Thank you again for the feedbacks. 

References:
- Ilharco et al. Editing models with task arithmetic. ICLR'23

Meta-review: The paper presents an improvement to task arithmetic using different trained coefficients for each block. Overall the reviewers thought that the idea was original, the paper well presented, and the experiments thorough. In the rebuttal, a more thorough discussion of prior work and a multitask finetuning baseline were provided. Otherwise I believe no major weaknesses remain, and I recommend the paper for acceptance. Congratulations to the authors!
Decision: Accept (poster)