Neural Fine-Tuning Search for Few-Shot Learning
Authors: Panagiotis Eustratiadis
Łukasz Dudziak
Da Li
Timothy Hospedales
Venue: ICLR 2024 oral (2024)
Keywords: stochastic
neural
architecture
search
few
shot
learning
adapters
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A stochastic neural architecture search algorithm that searches for the optimal configuration of layers in a pre-trained backbone architecture, to be adapted or fine-tuned.

Abstract: In few-shot recognition, a classifier that has been trained on one set of classes is required to rapidly adapt and generalize to a disjoint, novel set of classes. To that end, recent studies have shown the efficacy of fine-tuning with carefully-crafted adaptation architectures. However this raises the question of: How can one design the optimal adaptation strategy? In this paper, we study this question through the lens of neural architecture search (NAS). Given a pre-trained neural network, our algorithm discovers the optimal arrangement of adapters, which layers to keep frozen, and which to fine-tune. We demonstrate the generality of our NAS method by applying it to both residual networks and vision transformers and report state-of-the-art performance on Meta-Dataset and Meta-Album.

Published as a conference paper at ICLR 2024
NEURAL FINE-TUNING SEARCH FOR FEW-SHOT LEARNING
Panagiotis Eustratiadis', Lukasz Dudziak Da Li', Timothy Hospedales 'University of Edinbuugh 'Samsung AI Center, Cambridge
ABSTRACT
In few-shot recognition, a classifier that has been trained on one set of classes is required to rapidly adapt and generalize disjoint, novel set of classes. To that end, recent studies have shown the efficacy of fine-tuning with carefully-crafted adaptation architectures. However this raises the question of: How can one design the optimal adaptation strategy? In this paper, we study this question through the lens of neural architecture search (NAS). Given a pre-traime pre-trained neural network, our algorithm discovers the optimal arrangement of adapters, which layers to keep frozen, and which to fine-tune. We demonstrate the generality of our NAS method by applying it to both residual networks and vision transformers and report state- of-the-ar art performance on Meta-Datasee and Meta-Album.
INTRODUCTION
Few-shot recognition (Lake et al. 2011 Miller et al. 2000 Wang et al. 2020b) aims to learn novel concepts from few examples, often by rapid adaptation of a model trained on a disjoint set of la- bels. Many solutions adopt meta-learning perspective (Finn et al. 2017 Lee et al., 2019 Ravi & Larochelle 2017 Snell et al. 2017). or train a powerful feature extractor on the source classes (Tian 2019 both of which assume that the training and testing classes are drawn from the same underlying distribution e.g., written characters (Lake et al. 2015). or ImageNet cat- egories (Vinyals et al., 2016). Later work considers more realistic and challenging setting of few-shot adaptation not only across visual categories, but also across diverse visual domains (Tii- antafilou et al. 2020 Ullah et al. 2022). this cross-domain problem variant, customising feature extractor for novel domains is important, and several studies address this through dynamic feature extractors (Bateni et al. 2020 Requeima et al. 2019) or ensembles of features (Dvornik) et al., [2200a] Li et al. 2021 Liu et al. 20)1aa. Another group of studies employ heuristically- motivated fine-tuning strategies for adaptation (Dhillon et al. 2020 Hu et al. 2022 Li et al. 2022] Xu et al.) 2002)). Thus, important question that arises from previous work is: How one design the optimal adaptation strategy? In this paper, we take step towards answering this question.
al. 2020 Wang
Fine-tuning approaches few-shot adaptation must manage trade-off between adapting large or small number of parameters. The former allows for better adaptation, but risks overfitting on few-shot training set. The latter reduces the risk of overfitting, but limits the capacity for adap- lation to novel categories and domains. The recent PMF (Hu et al.) (022)) manages this trade-off through careful tuning of learning rates while fine-tuning the entire feature extractor. TSA (Li al. 2022) and ETT (Xu et al., 2022) manage it by freezing the feature extractor weights, and insert- ing some parameter-efficien adaptation modules, lightweight enough be trained in few-shot manner. FLUTE (Triantafillou et al.) (021) manages it through selective fine-tuning of a tiny set of FILM (Perez et al. 20188) parameters, while keeping most of them fixed. Despite this progress, the best way manage the inaptationngeneralisattt learning (FSL) is still an open question. For example, which layers should fine- tuned? What kind of adapters should be inserted, and where? While PMF, TSA, ETT, FLUTE, and others provide some intuitive recommendations, we propose a more systematic approach to answer these questions.
trade-off in fine-tuning approaches to few-shot
In this paper, advance the cuaptation-based paradigm for FSL by developing neural architecture search (NAS) algorithm to find the optimal adaptation architecture. Given an initial pre-traine trained feature extractor, our NAS determines the subset of the architecture that should be fine-tuned, as

---

Published as a conference paper at ICLR 2024
well as the subset of layers where adaptation modules should be inserted. We draw inspiration from
recent work in NAS (Cai et al., 2020; Chen et al.) 2021: Chu et al. 2021 Guo et al.
2020 Zhang
let al.
2022 that proposes revised versions of the stochastic Single-Path One-Shot (SPOS) Guo
et al. 2020 weight-sharing strategy. Specifically, given a pre-trained ResNet (He et al. 2016) or
Vision Transformer (ViT) (Dosovitskiy al., (02T), we consider search space defined by the inclusion or non-inclusion of task-specific adapters per layer, and the freezing or fine-tuning of learnable parameters per layer. Based on this search space, we construct supernet (Brock et al. 2018) that we train by sampling random path in each forward pass (Guo et al. (020). Our supernet architecture illustrated schematically in Figure [b] where the aforementioned decisions are drawn as decision nodes (o), and possible paths are marked in dotted lines.
While supernet training remains somewhat similar to standard NAS, the subsequent search poses new challenges in the FSL setting. Specifically, as cross-domain FSL. considers novel do- mains/datasets test time, the mainstream NAS paradigm of searching for a single neural archi- tecture (Cai et al.) 2019 Li et al. 2020b Liu et al., 2019 Wang et al. 202I) is sub-optimal, as diverse downstream datasets likely prefer different architectures. On the other hand, conducting full-blown NAS per few-shot episode is too slow and would likely overfit to the small support set. Motivated by these challenges, we propose a novel NAS algorithm that shortlists a small number of architecturally diverse configurations training time, but defers the final selection until the dataset and episode are known at test time. We empirically show that this is not only computationally effi cient, but also improves results noticeably, especially when only one domain is available training time. We term our method Neural Fine-Tumine Search (NFTS). NFTS defines search space that relevant to both convolutional and transformers architectures, and the choice of which specific adapter modules to consider is a hyperparameter, rather than hard constraint.
Our contributions are summarised as follows: (i) We provide the first systematic Auto-ML approach to finding the optimal adaptation strategy to trade off adaptation flexibility and overfitting in multi- domain FSL.. (ii) Our novel NFTS algorithm automatically determines which layers should be frozen or adapted, and where new adaptation parameters should be inserted for best few-shot adaptation. (iii) We advance the statc-of-the -htte-r in the well-established and challenging Meta-Dataset (Triantafil- lou et al., 2020), and the more recent and diverse Meta-Album (Ullah et al. 2022) benchmarks.
2 NEURAL FINE-TUNING SEARCH
2.1 FEW-SHOT LEARNING BACKGROUND
Let D = [ be the set of D classification domains, and D = {X, (X, E D a task containing samples along with their designated true labels {X, Y) {x,, M/ } Few-shot classification is
defined as the problem of learning correctly classify a query set = (Xo. Ye) ~ D by training on support set S = [Xs, Ys} ~ DD that contains very few examples. This can be achieved by finding the parameters 0 of a classifier fo with the objective
arg max II [I(Yelfo(S.xe))
(1)
In practice, if 0 is randomly initialised and trained using stochastic gradient descent small sup- port set S, it will overfit and fail to generalise to Q. To address this issue, one can exploit knowledge transfer from some seen classes to the novel classes. Formally, each domain D is partitioned into two disjoint sets Drain and Dest, which are commonly referred to as "meta-train" and "meta-test", respectively. The labels in these sets are also disjoint, i.e., Yuain n Yest 0. In that case, @ trained by maximising the objective in Eq. [ Jusing the meta-train set, but the overall objective is to perform adequately when transferring enowledge to meta-test.
The knowledge transferred from meta-train to meta-test can take various forms ( (Hospedales et al.) 2022]. As discussed earlier, we aim to generalise family of few-shot methods (Hu et al. [002]]] Li et al. 2022 Xu et al. 2022) where parameters 0 are transferred before a subset of them C 0 are fine-tuned; and possibly extended by attaching additional "adapter" parameters a that are trained for the target task. For meta-test, Eq. [ can therefore be rewritten as
arg max
II (S
(2)
0,0 Denn
2

---

Published as a conference paper at ICLR 2024
Our focus is finding the optimal adaptation strategy in terms of (i) the optimal subset of parame- ters @ C 0 that need to be fine-tuned, and (ii) the optimal task-specific parameters a to add.
2.2 DEFINING THE SEARCH SPACE
Let Io, be the minimal unit for adaptation in architecture. We consider these be the repeated units in contemporary deep architectures, e.g., a convolutional layer in a ResNet, or a self-attention block in VIT. If the feature extractor So comprises of K such units with learnable parameters da. then we denote =
U di. assuming all other parameters are kept fixed. For brevity in notation will now omit the indices and refer to every such layer Ié. Following the state-of-the-art (Hu (et al., 2022 Li et al., 2022| Triantafillou et al., 2022 Xu et al. 2022), let us also assume that task- specific adaptation can be performed either by inserting additional adapter parameters or into Ja or by fine-tuning the layer parameters Φ.
This allows us to define the search space two independent binary decisions per layer: (i) The inclusion or exclusion of an adapter module attached to Iø. and (ii) the decision of whether to use the pre-trained parameters d. replace them with their fine-tuned counterparts d'. The size of the search space is, therefore, (22) = _ For ResNets, we use the proposed adaptation architecture of TSA (Li al. 2022), where a residual adapter ha, parameterised by a, connected go
90.6'.o Io.o (I) + (r)))),
where x E IR W.H.C. For ViTs, we use the proposed adaptation architecture of ETT (Xu et al. 2022). where a tuneable prefix is prepended to the multi-head eeff-attention module Aykes and residual adapter is appended to both Agke and the feed-forward module z in each decoder block
D0.d' (a) =(Agke gkel1 Go.er (æ)) + ha1) + ha2;
(4)
where x R.D and [ denotes the concatenation operation. Note that in the case of ViTs the adapter is not function of the input features, but simply an added offset.
Irrespective of the architecture, every layer Jo.é',a a is parameterised by three sets of parameters, Φ, ø', and denoting the initial parameters. fine-tuned parameters and adapter parameters respectively. Consequently, when sampling a configuration (i.e., path) from that search space, every such layer can be sampled one of the variants listed in Table I
2.3 TRAINING THE SUPERNET
Following SPOS Guo et al. 2020). our search space is actualised in the form of a supernet fo.a. "super" architecture that contains all possible architectures derived from the decisions detailed in Section [.2 It is parameterised by: (i) 0, the frozen parameters from the backbone architecture fo- (ii) a, from the adapters h., and (iii) ø', from the fine-tuned parameters per layer fø.d'.a-
We use prototypical loss L(f,5, Q) as the core objective during supernet training and the subse- quent search and fine-tuning.
1
10]
deus (Cq,
C(f.S, Q)
log プド= (C,,
(5)
where Cg. denotes the embedding of the class centroid that corresponds to the true class of Q., and deos denotes the cosine distance. The class centroids C' are the mean embeddings of support examples that belong to the same class: C₁
[su=l] E ((S)))
L.a
ResNet
no(2) + h (x) =(Aghn 4 (a(x)) =(Apuol 9x(x)) h.r) + haa =(Aguala (x)) z(Agpal9: (= + hor) hwa
(a(a))
(ar)
Ger (ar) (2)(
Table 1: The search space, as described in Section 2.2 A layer Jo.6' o can sampled in one of
the following variants: (i) d: fixed pre-trained parameters, adaptation, (ii) as fixed pre-trained parameters, with adaptation, (ii) d': fine-tuned parameters, no adaptation, (iv) d', a fine-tuned- parameters, with adaptation.
3

---

Published as a conference paper at ICLR 2024
Mcta-training
Meta-lesting
f0
ImageNet pre-trained
backbone architecture
fe o.
Support set $
Supemet training
Best architocture sclection
Trained supernet
Evolutionary search
Evaluation
P1
6
diverse
Query set
architectures
Adapter parameters
fx-4
In
Frozen parameters
Fine-tuned parameters e
a
92
0.0
9k
(a) After a supernet is trained, evolutionary search finds the lop-performing candidates (validation set). During new test episode, the shortlisted candidates evalu- ated on the support (Eq. [[]. and the best architec- ture for that test episode is selected.
(b) The dotted lines represent possible paths that can be sampled during SPOS training. Every adaptablc layer in the architecture (g.) has its own pre pre-traines te-trrined (ø. C 8), ine-tuned (#,), and adapter (as) parameters.
Figure 1: Our proposed NAS paradigm for few-shot adaptation. (a) Overall meta-train/mett--ess workflow. (b) The supernet architecture. f denotes the feature extractor, which is composed of many layers, g. which the minimal unit for adaptation in our search space.
For supernet training, let P be a set of size 4K enumerating all possible sequences of K layers that can be sampled from the search space. Denoting path sampled from the supernet as fo.a. minimise the loss in Eq. oover multiple episodes and paths, the final objective becomes:
arg min Ep-plEs Clb.a,,s 2).
(6)
0.0/
In Appendix we summarise the supernet training algorithm in pseudocode (Algorithm) I
2.4 TWO-STAGE SEARCH FOR AN OPTIMAL PATH
supernet fe. trained with the method described in Section 2.3 contains 4k models, intertwined via weight sharing. As explained in Section | our goal is to search for the best rest-performing one, but the main challenge is related to the fact that do know what data is going be used for adaptation at test time. One extreme approach would be to search for single solution during training and simply use throughout the entire test, regardless of the potential domain shift. Another, would be to defer the search and perform it from scratch each time a new support set is given to at test time. However, both have their shortcomings. As such, we propose a hybrid, where searching is split into two phases one during training, and subsequent
during testing.
Meta-training time The search is responsible for pre-selecting a set of N models from the entire search space. Its main purpose is to mitigate potential overfitting that happen at test time, when only a small amount of data is available, while providing enough diversity to successfully adjust the architecture to the diverse set of test domains. Formally, we search for sequence of paths (p), P2, ......P'''' where:
Pk = arg max Es, ( A(S. 8' S, Q), s.t. PEP
a 0" arg min Li (, S,
(8)
a.
V Vj=l,.nk-1 (pt, pp) 2 T,
(9)
4

---

Published as a conference paper at ICLR 2024
where T denotes scalar threshold for the cosine distance between paths Pk and py, and A is the classification accuracy of nearest centroid classifier (NCC) (Snell et al. 2017).
[
1
A(f.S
arg min (Ce, = Yo.).
(10)
O
j
We measure accuracy of solution using a query set, after fine-tuning on separate support set (Eq.8), then average across multiple episodes to avoid overfitting to a particular support set (Eq. ( We also employ a diversity constraint, in the form of cosine distance between binary encodings of selected paths (Eq. 9 to allow for sufficient flexibility in the following test time search.
To efficiently obtain sequence {pi,.... PN we use evolutionary search to find points that maximise Eq. 7 and afterwards select the N best performers from the evolutionary search history that sat- isfy the constraint Eq. 9
In Appendix C we summarise the training-time search algorithm in
pseudocode ( (Algorithm 2
For given meta-test episode, we decide which one of the pre-selected N models is best-suited for adaptation on the given support set data. acts as a failsafe to counteract the bias of the initial selection made at training time cases when the support set might be particularly out-of-domain. Formally, the final path p" to be used in particular episode is defined as:
Meta-testing time
p
arg min PC (p).....PP
Llfb,a (", s.t.
(11)
a の0 arg min L(J,, S,S).
(12)
a.
We test each of the N models by fine-tuning it on the support set (Eq, 12 and scoring its loss on the same support set (Eq. [T] This is because the support set is the only source of data we have at test time and we cannot extract disjoint validation set from it without risking the fine-tuming quality. important to note that, while this step risks overfitting, the pre-selection of models at training time, as described previously, should already limit the subsequent search to only models that are unlikely to overfit. Since is kept small in our experiments, we use a naive grid search to find p".
This approach is generalization of the existing NAS approaches, as it recovers both when N 1 or N = 44 Our claim that intermediate values of N are more likely to give us better results would expect
than any of the extremes, due to the reasons mentioned earlier. In particular, pre-selecting 1 < N < 4K models to introduce reasonable overhead at test time while improving results, especially in cases when exposure different domains might be limited training time. In our evaluation we compare N 3 and N = to test this hypothesis. We do not include comparison to N = 4 as it is computationally infeasible in setting (performing equivalent of training time search for each test episode would require us to fine-tune ss 14 x 10th models in total).
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Evaluation on Meta-Datased We evaluate NFTS on the extended version of Meta-Dataset Ree- queima et al. 2019 Triantafillou et al. 2020). currently the most commonly used benchmark for Tew-shot classification, consisting of 13 publicly available datasets: FGVC Aircraft, CU Birds, De- scribable Textures (DTD), FGVCx Fungi, ImageNet, Omniglot, QuickDraw. VGG Flowers, CIFAR- 10/100, MNIST, MSCOCO, and Traffic Signs. There are 2 evaluation protocols: single domain (SD) learning and multi-domain (MD) learning. In the single domain setting, only ImageNet is seen dur- ing training and meta-training, while the multi-domain setting the first eight datasets are seen (FGVC Aircraft to VGG Flower). For meta-testing, 600 episodes are sampled for each domain, following the evaluation protocol proposed by Triantafillou et al. (2020).
Evaluation on Meta-Album Further, we evaluate NFTS the more recent Meta-Album (UI- lah et al. 2022). ). which is more diverse than Meta-Dataset. We use the currently available Sets 0-2, which contain over 1000 unique labels across 30 datasets spanning 10 domains including mi- croscopy, remote sensing, manufacturing, plant discase, character recognition, human action recog- nition tasks, etc. Unlike Meta-Dataset, in which the default evaluation protocol is variable-way
5

---

Published as a conference paper at ICLR 2024
Nathed
Birds
D1D
Fregi
brazchei
Obeaiahr:
Quiclirow
Flonex
CEPARIO
CIAARIOO
MSSST
MSCOCC
Ir. Sitie
Averaee
55.4
52.
Proni
08.8
08.6
74.7
50.5
18.0
15.3
-1..
mozm 203 2001
581
7017
08.3
-1.4
11.9
67.5
533
13:3
(880
11.8
1033
(rizzze Sort p00n
768
52.8
T5A
74.9
44.7
676
82.1
267
N770
349
76.5
$1.
51.5
63.7
MIL1
673
94.5
K3
71.5
94.
59.7
11.9
TX.1
733
88.9
$7.
01.3
61.4
76.
02.3
NPT
63.0
855
87.6
63.2
71.0
766
56.0
79.4
72.6
02.6
$7.9
Single domain setting: only ImageNet is seen in Table 2: State-of-th the art methods on Additional data used for training. meta-train. Mean acc. over 600 episodes.
Imagehee
Ohriighe
1:5ip
Anerogg
CNARS
Bacoeacaziin prate
817
71.6
09.5
93.3
535
91.3
71.7
55.4
74.1
16.1
06.5
Stapl
57-3
CNAPS
$1.
76.8
68.0
48.6
58.
65.3
SUA
655
11.3
11.3
66.3
5522
06.1
81.
83.3
06.5
38.9
[6.3
32.0
51.0
71.4
13.3
13.0
UKT
818
71.
41.0
968
12.1
988
FLA 12800194412
02.8
15.3
71.2
48.5
39.6
92.
11.3
98.3
75.4
62.0
1962
32.8
60.0
723
TSA
$3.0
$1.1
17.3
09.3
99.5
81.3
63.2
$2.9
70.4
196.
37.6
62.8
03.5
53.1
18.2
09.5
35.6
15.4
83.8
13.4
76.5
67.3
90.3
57.5
20.4
713
NPTS
98.1
83.8
62.3
03.4
61.4
62.9
02.2
83.8
95.4
38.8
81.9
913
1113
13.5
03.7
12.2
12.6
88.3
NFTS
99.1
50.5
16.3
76.1
74.
92.8
184
93.3
75.9
708
V1.3
62.6
67.3
K34
Table 3: State-of-the art methods on Meta-Datasse Multi-domain setting: the first 8 datasets seen in meta-train. Mean acc. over 600 episodes. Different backbone. Additional training data.
variable-shot, Meta-Album evaluation follows a 5-way variable-shot setting, where the number of shots is typically 5, 10 and 20. For meta-testing, results are averaged over 1800 episodes.
Architectures We employ two different backbone architectures, R8 (He et al. 2016) and a ViT-small (Dosovitskiy et al. 202T). Following TSA (Li et al. 2022h, the ResNet-18 backbone is pre-trained on the seen domains with the knowledge-distillationn method URL (Li et al. 202
and, following ETT (Xu et al. 2022), the ViT-small backbone is pre-trained on the seen portion of ImageNet with the self-supervised method DINO (Caron et al. 202I). We consider TSA residual adapters ([i i et al., 2022] Rebufh et al. (017) for ResNet and Prefix Tuning (Li & Liang, 2021 Xu et al. 2022) adapters for VIT. This mainly to enable direct comparison with prior work the same base architectures that use exactly these same adapter families, without introducing new confounders in terms of mixing adapter types (Li et al. 2022 Xu et al. (0022). However, our framework is flexible, meaning it can accept any adapter type, or even multiple types in its search space.
3.2 COMPARISON TO THE OFATE-OF-THE-AR
Meta- The results on Meta-Dataset are shown in Table and Table for the single domain and multi-domain training settings respectively. We can see that NFTS obtains the best average performance across all the competitor methods for both ResNet and VIT architectures. The margins over prior state-of-the-art are often substantial for this benchmark with +1 -1.9% over TSA in ResNet- 18 single domain, +2.3% in multi-domain and +1 +1.6% increased margin in the multi-domain case is intuitive, as our framework has more data with which learn the optimal path(s).
ETT in VIT-small single domain. The
We re-iterate that PMF, ETT, and TSA are special cases of our search space corresponding respec- tively to: (i) Fine-tune all and include no adapters, (ii) Include ETT adapters every layer while freezing all backbone weights, and (iii) Include TSA adapters at every layer while freezing all back- bone weights. We also share initial pre-trained backbones with ETT and TSA (but not PMF, as they use a stronger pre-trained model with additional data). Thus, the margins achieved over these competitors are attributable to our systematic approach to finding suitable architectures, terms of where to fine-tune and where to insert new adapter parameters.
Meta-Album The results on Meta- Album are shown Table a function of number of shots within the 5-way setting, following Ullah al. (2022). We can see that across the whole range of support set sizes, our NFTS dominates all the well-tuned baselines from Ullah et al. (2022). The margins substantial, greater than 5% at 5- 6-way/5-shol operating point, for example. This result confirms that our framework scales to even more diverse datasets and domains than those considered previously in Meta- Dataset.

---

Published as a conference paper at ICLR 2024
From Scratch
Fine Tuning
Matching Net
ProtoNet
FO-MAMI.
NFTS
1-shot
30.42
40.43
34.49
38.07
33,94
43.76
5-shot
38.31
50.87
44.32
51.17
44.50
57.59
10-shot
39.58
53.42
49.23
55.18
48.62
60.10
20-shot
39.83
55.12
52.99
59.67
51.35
60.97
Table 4: Comparison of our method against Meta-Album baselines, as reported in Fig. 2b of their paper (Ullah al. 2022). The setting is cross-domain 5-way [1, 5, 10, 20]-shot, and accuracy scores are averaged over 1800 tasks drawn from SetO, Set1 and Set2.
ResNet-16 18
ViT-S
Method
SD
MD
SD
MD
67.8
67.8
71.8
71.8
N=1
N=3
N=10
N=100
70.4
76.5
73.8
77.3
Test acc. (Support)
96.5
97.1
99.9
99.8
70.2
76.3
74.0
77.5
Test (Query)
72.8
72.9
71.5
71.4
70.8
76.9
74.4
78.9
S/Q Pearson r
0.31
0.35
0.28
0.18
NFTS-1
73.6
80.1
78.7
83.1
NFTS-N
75.2
80.7
79.2
83.4
Table Ablation study on Meta-Dataser Left: Comparing four special cases of the search space in terms of average accuracy: (i) ø, -: No adaptation, no fine-tuning, (ii) é, a: Adapt all, (iii) o', Fine-tune all, (iv) ø', ac Adapt and fine-tune all. NFTS- (1,N) refer to conventional and deferred episode-wise NAS respectively. Mean accuracy over 600 test episodes. Right: Impact of the number of candidates N in the deferred NAS shortlist. Mean accuracy over 30 test episodes.
3.3 FURTHER ANALYSIS
Ablation Study To analyse the role that our architecture search plays in few-shot performance more precisely, we also conduct an ablation study of our final model against four corners of our search space: (i) Initial model only, using pre-trained feature extractor and simple NCC classi- fier, which loosely corresponds to SimpleShot (Wang et al. 2019), (ii) Full adaptation only, us- ing fixed feature extractor, which loosely corresponds to TSA (Li al.. 2022). ETT (Xu et al.] 2022), FLUTE (Triantafillou et al., (021). and others depending on base architecture and choice of adapter, (iii) Fully fine-tuned model, which loosely corresponds to PMF (Hu et al.) (202)), and (iv) Combination of full fine-tuning and adaptation. Table (left) shows that full adaptation, full fine-tuning, and their combination, give improvement on the linear readout baseline. Our work is the first to note the better performance of combining fine-tuning and adapters (Ø', a).
Most importantly, both variants of our NFTS substantially outperform the baselines, demonstrating the value of overall framework compared mainstream fixed fine-tuning patterns. Next, can compare our top-1 adaptation architecture selection strategy against our progressive approach that defers the final architecture selection an episode-wise decision (i.e., NFTS-1 vs. NFTS-N). Our deferred architecture selection improves upon fixing the top-1 architecture from meta-train, especially for the single source domain case. This is intuitive, because NAS on single source domain (cf., multi-domain) condition is most at risk of over-tuning to that domain, and should benefit the most from learning and transferring a diverse pool of architectures to the target domains.
Discovered Architectures We next analyse: What kind of adaptation architecture is discovered by our NAS strategy? We first summarise results of the entire search space terms of which layers are preferential to fine-tune or not, and which layers are preferential to insert adapters or not in Figure 2a The blocks indicate layers (columns) and indicating whether that architectural decision was positively (green) negatively (red) correlated with validation performance. We can see that the result complex, without a simple pattern, as assumed by existing work (Hu et al.) 2022 Li et al. 2022 Xu et al. 2022). That said, our NAS does discover some interpretable trends. For example, adapters should be included at early/late ResNet- 18 layers and not at layers 5-9.
paatterrs/ineeuunnng (rows), with the color
We see that these follow the strong trends in the search space from Figure For example, they always adapt (o)
We next show the top three performing paths subject to diversity constraint in Figure 2b

---

Published as a conference paper at ICLR 2024
-05
911 012
91
9.4
915
02
8r Sia
911
91
Bix
84 Bi
Vi
(a) Correlation between inclusion/mooniiclusiion of learnable parameters and o', and validation perfor- mance.
(b) Top performing paths subject to diversity con- straint. We expect the green in Fig. 2a be roughly followed.
Figure 2: Qualitative analysis of architecture search. Fig. summarises the whole search space by answering the question: How important is to adapt ((x) or fine-tune (Φ') each block? The color of each square indicates the point-biserial correlation (over all searched architectures) between adapting/fine-tuning layer 9; and validation performance. Fig. Fig.[2]shorr shows the top performing can- didates subject diversity constraint, after 15 generations of evolutionary search. Dark blue indicates that the layer is adapted/fine-tuned and light blue that it is not.
CIFARIO CIFAR100
82.0
81.2
83.3
75.9
75.0
75.1
MNIST MSCOCO
95.5
94.4
95.1
58.1
57.8
56.4
Tr. Signs
81.7
82.2
81.8
Table 6: How the diverse selection of architectures from Fig. 2b perform per unseen downstream domain in Meta-Dataset. Shading indicates episode-wise architecture selection frequency, numbers indicate accuracy using corresponding architecture. The best dataset-wise architecture (bold) most often selected (shading).
block 14 and never adapt block 9. However, due to our diversity constraint, they do include diverse decisions (e.g., whether to fine-tune (ø') block 15) that were not strongly indicated in Figure 2a
Finally, we analyse how our small set of N candidate architectures in Figure 2b is used during meta-test. Recall that this small set allows us perform an efficient minimal episode-wise NAS, including for novel datasets unseen during training. The results in Table 6 show how often each architecture selected by held-out datasets during meta-test (shading), and what the per-dataset performance using only that architecture. It shows how our approach successfully learns to select the most suitable architecture on a per-dataset basis, even for unseen datasets. This unique capability goes beyond prior work (Hu et al. 2022 Li et al. 2022 Xu et al. 2022) where all domains must rely on the same adaptation strategy despite their diverse adaptation needs.
Why N=3? Impact of Candidate Architecture Set Size Our framework admits various design options from N 1, to large N (full-blown NAS per few-shot (esdseot As discussed carlier, N expect
uses the same architecture for all episodes without datasct-specific selection. Meanwhile, large N to suffer overfitting due to ultimately selecting a large number of parameters (N networks, times the number of learnable parameters each) based small support set, defeating the purpose of our whole selective fine-tuning paradigm. To illustrate this we conduct a small experiment on subset of 30 episodes in Table 9 (right), comparing the support/train and query/test accuracy as a function of N. For large N, besides being expensive, we see overfitting with support accuracy approaching perfect, but query accuracy decreasing. This is also reflected in the decreasing Pearson correlation between episode-wise support and query accuracies as N becomes large.
Discussion Overall, our deferred NAS approach where large-scale search is conducted up-front during meta-train and small candidate set search is conducted per meta-test episode, provides reasonable trade-off between per-epi cost and efficacy. While our cost recommended N is slightly higher than competitors with single fine-tuning, it is similar or less than competitors who repeat adaptation with different learning rates during testing (Hu et al.) 2022] (4x cost), or exploit backbone ensemble (8x cost) (Dvorik et al. 2020b; Liu et al. (021a). Where this cost is not acceptable, our single architecture NFTS-I already provides state-of-the-an results.
Selecting high number of episodes is expensive, due substantial per-episode architecture selection
8

---

Published as a conference paper at ICLR 2024
4 RELATED WORK
Gradient-Basee Few-Shot Adaptation Parameter-efficient adaptation modules have been ap- plied for multi-domain learning, and transfer learning. seminal example is the Residual Adapter (Rebutti et al. 2017). a lightweight Ixl convolutional filter added to ResNet blocks. They were initially proposed for multi-domain learning, but were successfully used to achieve state of the art results for CNNs on the meta- meta-dataset benchmark (Trinnafillou et al.] 2020) by enabling fine- tuning of a URL (Li et al., 202T) pre-trained backbone without severe overfitiing in the fho-shot regime Li et al. (022). Meanwhile, prompt (Jia et al. (202)) and prefix (Li & Liang/ 2021) tuning are established examples of parameter-efficiet adaptation for transformer architectures for similar reasons. In FSL, Efficient Transformer Tuning (ETT) (Xu et al., (2022) apply a similar strategy to few-shot VIT adaptation using a DINO (Caron et al.
2021) pre-trained backbone.
PMF (Hu et al.) 2022), FLUTE (Triantafillou et al. 202 and FT (Dhillon al. 2020) focus on adapting existing parameters without inserting new ones. To manage the adaptation/overiitin trade- off in the few-shot regime, PMF fine-tunes the whole ResNet or ViT backbone, but with carefully- managed learning rates. Meanwhile, FLUTE hand-picks a set of FILM parameters with a modified ResNet backbone for few-shot fine-tuning, while keeping the majority of the feature extractor frozen.
All of the methods above make heuristic choices about where to place adapters within the backbone, or for which parameters to allow/disallow fine-tuning. However, as different input layers represent different features (Chen et al., [021]: Zeiler & Fergus] 2014), there is scope for making better de- cisions about which features to update. Furthermore, in the multi-domain setting different target datasets may benefit from different choices about which modules to update. This paper takes an Auto-MI. NAS-based approach to systematically address this issue.
Neural Architecture search Neural Architecture Search (NAS) is a large topic (Elsken et al., 2019) which we do not attempt to review in detail here. Mainstream NAS aims to discover new architectures that achieve high performance when training
single dataset from scratch in many-
shot regime. To this end, research aims to develop faster search algorithms (Abdelfattah et al. 202T]
Guo et al. 2020; Liu et al. 2019 Xiang et al. (023)) and better search spaces (Ci et al. 2021
Fang
ctt al. 2020
Radosavovic et al. 2019 Zhou et al., 202IJ. We build upon the popular SPOS Guo 2020) family of search strategies that encapsulate the entire search space inside supernet that is trained by sampling paths randomly, and a search algorithm then determines the optimal path. We develop an instantiation of the SPOS strategy for multi-domain FSL. We construct search space suited for parameter-efficien adaptation of a prior architecture to a new set of categories, and extend SPOS learn on a suite of datasets, and efficiently generalise novel datasets. This is different than the traditional SPOS paradigm of training and evaluating on the same dataset and label-space.
ctt al.
While there exist some recent NAS works that try address similar "train once, search many times" problem efficiently (Cai et al. 20200 Li et al. [2000a] Molchanov et al. 2022; Moons et al.) 2021). naively using these approaches has two serious shortcomings: i) They assume that after the initial supernet training, subsequent searches do not involve any training (e.g., a search is only performed to consider different FLOPs constraint while accuracy of different configurations is assumed to stay the same) and thus can be done efficiently this is not true in the FSL setting as explained earlier. ii) Even if naively searching for each dataset at test time were computationally feasible. the few-shot nature of our setting poses significant risk of overfitting the architecture to the small support set considered in each episode.
5 CONCLUSIONS
In this paper we present NFTS, novel framework for discovering the optimal adaptation architec- ture for gradient-based few-shot learning. NFTS contains several recent strong heuristic adaptation architectures as special cases within its search space, and we show that by systematic architecture search they are all outperformed, leading to a new state-of-th of-thh--rttttttt the-art on Meta-Datases and Meta-Album While in this paper we simple and coarse search space for easy and direct comparison to prior work's 's hand- designed adaptation strategies, in future work we will extend this framework to include richer range of adaptation strategies, and finer-granularity of search.

---

Published as a conference paper at ICLR 2024
REFERENCES
Mohamed S Abdelfattah, Abhinav Mehrotra, Fukasz Dudziak, and Nicholas Donald Lane. Zero-cost proxies for lightweight NAS. In ICLR, 2021.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In CVPR, 2020.
Peyman Bateni, Jarred Barber, Jan- Wil-Willem van de Meent, and Frank Wood. Enhancing few-shot image classification with unlabelled examples. In WACV, 2022.
Andrew Brock, Theo Lim, James M. Ritchie, and Nick Weston. SMASH: One-shot model architec- ture search through hypernetworks. In ICIR,, 2018.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In ICLR, 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one
network and specialize it for efficient deployment. In ICLR, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In ICCV, 2021.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation faimess of weight sharing neural architecture search. In ICCV, 2021.
Yuanzheng Ci, Chen Lin, Ming Sun, Boyu Chen, Hongwen Zhang, and Wanli Ouyang. Evolving search space for neural architecture search. ICCV, 2021.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In ICLR, 2020.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformer:: spatially- aware few-shot transfer. In NeurlPS, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Nikita Dvomikk Cordelia Schmid, and Julien Mairal. Selecting relevant features from multi- domain representation for few-shot classification. In ECCV, 2020a.
Nikita Dvornik, Cordelia Schmid, and Julien Mairal. Selecting relevant features from a universal representation for few-shot classification. In ECCV, 2020b.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In CVPR, 2020.
Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Densely connected search space for more flexible neural architecture search. In CVPR, 2020.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In ICLR, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In ICML, 2017.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff prun- ing. In ACI., 2021.
10

---

Published as a conference paper at ICLR 2024
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In CVPR, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun.
Single path one-shot neural architecture search with uniform sampling. ECCV, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: survey. IEEE Truns. Pattern Anal. Mach. Intell., 2022.
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M. Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In CVPR, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. One shot learning of simple visual concepts. In CogSci, 2011.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, 2019.
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang. Blockwisely supervised neural architecture search with knowledge distillation. CVPR, 2020a.
Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Muller, Ali Thabet, and Bernard Ghanem. Sgas: Sequential greedy architecture search. In CVPR, 2020b.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Universal representation learning from multiple domains for few-shot classification. In ICCV, 2021.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Cross-domain few-shot learning with task speciffe adapters. In CVPR, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuming: Optimizing continuous prompts for generation. In ACL, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019,
Lu Liu, William L. Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. A universal repre- sentation transformer layer for few-shot image classification. ICLR, 2021a.
Yanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, and Yi Yang. A multi-mode mod- ulator for multi-domain few-shot classification. In ICCV, 2021b.
Erik G. Miller, Nicholas E. Matsakis, and Paul A. Viola. Learning from one example through shared densities on transforms. In CVPR, 2000.
Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolò Fusi, and Arash Vahdat. LANA: latency aware network acceleration. In ECCV, 2022.
Bert Moons, Parham Noorzad, Andrii Skliar, Giovanni Mariani, Dushyant Mehta, Chris Lott, and Tijmen Blankevoort. Distilling optimal neural networks: Rapid search in diverse spaces. In ICCV, 2021.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville, Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
11

---

Published as a conference paper at ICLR 2024
Xiaorong Qin, Xinhang Song, and Shuqiang Jiang. Bi-level meta-learning for few-shot domain generalization. In CVPR, 2023.
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In ICCV, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NeurlPS, 2017.
James Requeima, Jonathan Gordon, John Bronskill, Sehastian Nowozin, and Richard E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In NeurIPS, 2019.
Tonmoy Saikia, Thomas Brox, and Cordelia Schmid. Optimized generic feature learning for few- shot classification across domains. CoRR, abs/2001.07926, 2020.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.
Manogna Sreenivas and Soma Biswas. Similar class style augmentation for efficient cross-domain few-shot learning. In CVPR Workshops, 2023.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding all you need? ECCV, 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: dataset of datasets for learning to learn from few examples. ICLR,, 2020.
Eleni Triantafillou, Hugo Larochelle, Richard S. Zemel, and Vincent Dumoulin. Learning universal template for few-shot dataset generalization. In ICML, 2021.
Ihsan Ullah, Dustin Carrion, Sergio Escalera, Isabelle M Guyon, Mike Huisman, Felix Mohr, Jan N
van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu. Meta-album: Multi-domaii meta- dataset for few-shot image classification. In NeurlPS Datasets and Benchmarks Track, 2022.
Oriol Vinyals, Charles Blundell, Tim Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NeurlPS, 2016.
Jiaxing Wang, Jiaxiang Wu, Haoli Bai, and Jian Cheng. M-nas: Meta neural architecture search. In AAAI, 2020a.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsich. Rethinking architecture selection in differentiable NAS. In ICLR, 2021.
Yan Wang, Wei-Lun Chao, Kilian Q. Weinberger, and Laurens van der Maaten. Simpleshot: Revis- iting neirbot- classification for few-shot learning. CoRR, 1is1191,,0462,, 2019.
Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few exam- ples: A survey on few-shot learning. ACM Comput. Surv.. 2020b.
Lichuan Xiang, Lukasz Dudziak, Mohamed S. Abdelfattah, Thomas Chau, Nicholas D. Lane, and Hongkai Wen. Zero-cost operation scoring in differrentialee architecture search. In AAAI, 2023,
Chengming Xu, Siqian Yang, Yabiao Wang, Zhanxiong Wang, Yanwei Fu, and Xiangyang Xue. Exploring efficient few-shot adaptation for vision transformers. TMLR, 2022.
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. CoRR, abs/2206.04673, 2022.
Daquan Zhou, Xiaojie Jin, Xiaochen Lian, Linjie Yang, Yujing Xue, Qibin Hou, and Jiashi Feng. Autospace: Neural architecture search with less human interference. In ICCV, 2021.
12

---

Published as a conference paper at ICLR 2024
A ADDITIONAL RELATED WORK
Feed-Forward Few-Shot Adaptation Besides the gradient-based few-shot adaptation methods mentioned in Section 4 an alternative line of work (Requeima et al. 2019) Bateni et al. 2020) uses
feed-forward networks to modulate the feature extraction process. However, these dynamic feature extractors are less able to generalise to completely novel domains than gradient- based methods (Finn)
& Levine 2018), the adaptation module itself suffers from out of distribution problem.
Sparse Fine- Toni-Tuning In the many-shot regime there exists a line of work on sparse-fing fine tuning that attempts identify subset of parameters to update for fine-tuning, such as SpotTune (Guo al. 2019) and DiffPrune (Guo et al., (021). This is related to our goal in terms of working with sparse masks, but otherwise are completely different. They optimise for a sparse mask that optimises single-task training loss on large dataset, while we optimise for a small set of architectures that, in expectation. enables the learner to generalise the test/query set of some novel after learning on small train/support set in a few-shot episode.
NAS and Few-Shot Learning There have been few prior attempts NAS-relate few-shot learning methods such as MetaNAS (Elsken et al. 2020) and mNAS (Wang et al. 2020a). These ap- proaches differ from ours in that they attempt to learn architectures that are suited for learning from scratch, rather than how to fine-tune powerful pre-trained feature; and have only been demonstrated on small architectures using a (flawed (Wang et al. 202T DARTS- search, while lacking the ability to scale to the large architectures we consider. EG: Elsken et al. (2020) reports 7 GPU-days to search a < 30k parameter model, while we take 2 GPU-days for I 11M (ResNet- BessNet-18)aa model.
B ADDITIONAL RESULTS
B. META-TRAINING CONVERGENCE
Meta- Training Fig. shows convergence in terms of training and validation accuracy during pernet training (accuracy is averaged across episodes and paths, following the training objective in Eq. [8. This convergence curve corresponds training a supernet with ResNet- 18 backbone in the mulfi-domain setting, for 40 thousand episodes. Similar behaviour occurs during the meta-training time search. The total cost for ResNet- 18 is approximately 10 GPUh for supernet training, 30 GPUh searching, 0.05 GPUh per test episode.
Path Search Process In addition, we illustrate the path search process in Figure This figure shows a 2D t-SNE projection of our 2K dimenssooa architecture search space, where the dots are candidate architectures of the evolutionary search process different iterations. The dots are colored according to their validation accuracy. From the results we can see that: The initial set of candidates broadly dispersed and generally low performing (left), and gradually converge toward a tighter cluster of high performing candidates (right). The top 3 performing paths subject to diversity constraint (also illustrated in Fig. 2b) are annotated in purple outline.
B.2 ADDITIONAL ABLATIONS
Why not ensemble? Our framework involves performing an episode-wise selection from our short- list candidate set of N architectures, where the scoring each architecture involves fine-tuning that architecture. One might therefore reasonably ask how our strategy compares to ensembling the same set of N fine-tuned models? Our guiding hypothesis is that different test-time episodes likely need different architecturev/adaptttion schemes. For example, in single-domain meta-dataset,
trai act.
1000
40000
Figure 3: Meta-training convergence (40,000 train episodes).
13

---

Published as a conference paper at ICLR 2024
0.78
Generation
Generation
Generation 15
Figure 4: Population of paths(candidate architectures) in the search space after 1, 5, and 15 gener- ations evolutionary search. Each dot 2-d TSNE projection of the binary vector representing an architecture, and its color shows the validation performance for that architecture. The supernet contains wide variety of models terms of validation performance, and the search algorithm con- verges to a well-performing population. The top 3 performing paths that are given in Fig. 2b are highlighted in the far right figure (Generation 15) in purple outline.
ResNet
VIT
Base Parameter Count
11M
22M
Number of Blocks
16
Parameter Count per Adapter
+36K-2M
+200K
Min-Max Possible Parameters Added in Search Space
0 11M
0-4M
Min Max Possible Parameters Updated in Search Space
0 22M
0 0-26M 26M
Number of Actual Parameters after NAS (avg. over N=3 architectures)
16M (+46%)
25M (+14%)
Number of Updatable Parameters After NAS (avg. N=3 architectures)
15M
24M
Actual Parameters Updated (TSA/ETT)
11M
4M
Actual Parameters Updated (PMF)
22M
Table 7: Architectural properties in the adaptation scheme of NFTS, for ResNet and VIT backbone architectures.
downstream task more similar to ImageNet might prefer fewer updateable parameters compared downstream task more different ImageNet. An architecture ensemble would take away this flexibility as the prediction architecture would always be a mixture of the N architectures (even if they are episode-wise fine-tuned), rather than selecting the best suited architecture for each episode.
We validate our strategy with an ablation in Tab. (ResNet-188 SD setting), showing that our second stage of architecture selection from the candidate short list outperforms simply predicting based on the ensemble of fine-tuned architectures.
Detailed Results of Search Space Ablation Tables 10 and I1 provide the exact scores per Meta- Dataset domain that are summarised in Table of the main paper, for single domain and multi- domain FSL respectively.
Structure and number of trainable parameters We start with ResNet-18 and VIT, which well-known architectures with well-documented number of parameters (1 (11M and 22M respectively). These are fine-tuned and/or extended with additive modules: Between 0-16 TSA modules are added in the case of ResNet, and between 0-12 ETT modules are added in the case of VIT. The specific architecture of TSA and ETT modules given by Li et al. (002)) and Xu al. (2022) respectively, and already summarised for completeness in Table of our paper. Each TSA module has 100K-2M attached), and each ETT module has 200K
parameters (depending where the architecture parameters. The number of parameters increased by adapting is not a fixed quantity, it depends on the result of the architecture search (e.g..: Fig 2b defines the result of the architecture search by showing which option in Table 1 is used at each layer). With respect to parameter counts of our search space, and of the final models discovered within that search space, we summarise them in Table together with few related methods PMF, TSA, and ETT. Finally, we remark that although NFTS adds some parameters to the base network, our total number parameters is still much less than the ensemble based methods like SUR (Dvornik et al. 2020b) and URT (Liu et al. 2021a), which are roughly 8x the base architecture size. Furthermore, in Table we present a concise summary presenting parameter counts and computational costs between SUR, URL., and NFTS.
14

---

Published as a conference paper at ICLR 2024
Method
Backbone
Parameters
Performance
SUR
Dvornik ct al.
2020b
ResNet-18
82M
71.4
URL
Li et
2021
ResNet-18
82M
76.6
CTX
Doersch et al.
2020)
ResNet-34
31.4M
74.2
NFTS
ResNet-18
16M
80.7
Table 8:
Ablation study for comparison of number of parameters and performances between SUR, setting.
URL, and NFTS. Performance mean accuracy scores Meta-Dataset,
NFTS-N (select best)
NFTS-N (ensemble)
Test acc.
75.2
74.4
Table 9: Ablation: Selection vs ensemble (N=3, 600 test eps.).
C SUPERNET TRAINING AND PATH SEARCH ALGORITHMS
We present the detailed implementation (in pseudocode) for the supernet training and evolutionary search, Algorithms and respectively.
D HYPERPARAMETER SETTING
Table
12
reports the hyperparameters used for all our experiments. Note the following clarifica-
tions:
"Number of epochs" refers to multiple forward passes of the same episode, while "Number of episodes" refers the number of episodes sampled in total.
The batch size is not mentioned, because only conduct episodic learning, where do not split the episode into batches, i.e., we feed the entire support and query set into our neural network architectures.
Learning rate warmup, where applicable, occurs for the first 10% of the episodes.
We further specify something important: While our strongest competitors (Li et al. 2022 Xu et al. 2022) tune their learning rates for meta-testing (e-g.. TSA uses LR=0.1 for seen domains and LR=1.0 for unseen, and ETT uses different learning rate per downstream Meta-Dataset domain), we treat meta-testing episodes as completely unknown, and use the same hyperparameters we used on validation set during search.
E
SOURCE CODE
The source code is available at:
15

---

Published as a conference paper at ICLR 2024
InegrNa
Omnight
Signe
Avonge
M6.4
14.3
15.0
63.9
K2.1
67.8
38
69.5
67.7
73.0
42.5
59.5
71.3
64.9
88.5
77.4
70.0
90.2
53.4
80.1
70.4
69.9
14.7
13.3
39.5
57.3
71.9
65.4
89.0
76.5
66.3
93.6
54.4
81.4
70.2
07.5
69.
77,0
19.3
59.7
77.
66.1
$7.4
31.7
09.5
91.9
55.1
78.7
70,8
NETS-1
73.2
76.
81A
42.1
61.3
80.2
66.9
90.0
$2.9
65.8
94.0
53.4
80.6
73.8
NFTS-N
74.9
76.
61.6
50.5
62.7
80.2
67.2
94.5
$3.0
71.5
94.0
99.7
$1.9
75.2
76.7
79.2
Table 10: Ablation study on Meta-Datasee comparing four special cases of the search space: (i) No adaptation, fine-tuning, (ii) Adapt all, (iii) d', -: Fine-tune all, (iv) ø', a: Adapt and fine-tune all. NFTS-(1,M [1,] refer to conventional and deferred episode-wist NAS respectively. Single domain setring: Only ImageNet is seen during training and search. Reporting mean accuracy over 600 episodes.
Methed
Airrralss
Bints
I/11>
Pungi
InegeNet
Omnngie
Quectiame
Howers
CIFR--IO
CHARIIO0
MINIST
MICOCC
TI: Signt
Averge
64.5
69.6
71.1
41.2
56.4
14.5
64.2
84.6
75.0
63.9
82.1
55.9
71.7
67.8
89.3
18.3
57.2
93.3
76.0
71.8
66.1
58.5
56.9
79.5
26.5
57.
68.3
16.2
08-2
96.1
26.3
75.1
90.7
60.5
71.8
85.7
34.3
81.8
68.7
7044
80.1
77.0
90.2
13.5
61.4
K0.6
53.7
71.4
27.8
83.0
$4.5
$1.1
70.9
72.4
88.6
74.6
90.4
75.1
61.5
87.0
$4.0
75.5
77.5
82.5
35.9
62.7
68.9
73.7
90.4
77.1
94.0
73.4
66.2
85.9
55.9
77.4
76.9
NITS-1
89.1
90.3
M.3
75.1
74.5
92.0
90.5
93.5
75.9
70.8
91.3
62.7
87.2
83.1
NETS-N
89.1
92.5
6633
75.1
76.5
92.0
90.6
90.5
75.9
70.8
91.3
62.8
87.2
83.4
Table 11: Ablation study Meta-Dataset comparing four special cases of the search space: (i)
J, - No adaptation, no fine-tuning, (ii) Φi o: Adapt all, (iii) -: Fine-tune all, (iv) Φ', a: Adapt and fine-tune all. NFTS-(1. (1,N) refer conventional and deferred episode-wise NAS respectively. Multi-domain setting: The first 8 datasets are seen during training and search. Reporting mean accuracy over 600 episodes.
Algorithm 1: Supernet training.
Input: Supernet fa.0.0'. Datasets D. Step sizes 12. Path pool P. Prototypical loss C (5.
Traind supernet fe.a.or. repeat
Output:
Sample dataset D ~ D
Sample episode S. Q D Sample path p ~ with learnable parameters ap, &,, and frozen parameters Φp C
Op Op V., L.B.,, S, Q)
0 6 di V ar C(S. S, Q)
until prototypical loss converges
16

---

Published as a conference paper at ICLR 2024
Algorithm 2: Training time evolutionary search.
Input:
Supernet fa. Datasets D. Step sizes 1/1, 72. Prototypical loss (Eq. 5). NCC accuracy A (Eq. (Oq.
Output: Optimal path p' Initialise population P randomly Initialise fitness of as P 0
repeat
Sample episodes from all datasets S, Q ~ D for each candidate p P do for a small number of epochs do
Ckp
ap
of S S) 12 V % L S,
1J1 Vo, L(
end
A A(f
0 S,Q
end
offspring - recombine the M best candidates of w.r.t. Up P P offspring
eliminate the M worst candidates of wrt P
until convergence or max. iterations
ResNet-18 MDL. (MD)
VIT-S
Hyperparameter
SDL (MD)
MDL (MA)
SDL. (MD)
MDL (MD)
Backhone architecture
URL
URL
Supervised TSA
DINO
DINO
Adapter architecture
TSA
TSA
ETT
EIT
Number of episodes
50000
80000
20000
80000
160000
Number of epochs Optimizer
1
1
1
adadelta 0.05
adadelta 0.05
adadelta 0.05
adamm
adamw
TRAIN
Learning rate
0.00007
0.00007
Learning rate schedule
cosine
cosine
Learning rate warmup
linear
linear
Weight decay
0.0001
0.0001
0.0001
0.01
0.01
Weight decay schedule
cosine
cosine
Number episodes
100
100
100
100
Number epochs
20
20
40
Optimizer
adadelta
adadelta
adadelta 0.1
adamw
adamw
Learning rate
0.1
0.1
O. 0.000003
0.000003
SEARCH
Weight decay
0.0001
0.0001
0.0001
0.1
0.1
Initial population size
64
64
64
64
Top-K crossover
8
8
8
Mutation chance
5%
5%
5%
5%
5%
Top-N paths
3
3
3
3
3
Diversity threshold
0.4
0.4
0.4
0.2
0.2
Number of episodes
600
600
1800
600
600
Number of epochs Optimizer
40
40
40
40
40
TEST
adadelta
adadelta
adadelta
adamw
adamw
Learning rate
0.1
0.1
0.1
0.000003
0.000003
Weight decay
0.0001
0.0001
0.0001
0.1
0.1
Regulariser strength
0.04
0.04
0.04
Hyperparameter setting for all experiments presented in Section of the main pa- per. The notation is as follows: SDL=Single domain learning, MDL=-Mlti-doman learning, MD=Meta-Dataset, MA=Meta-Album, TRAIN=Supernet training phase, SEARCHH-Exolttionay search phase, TEST=Meta-test phase.
Table 12:
17


Reviews:
[{"review_id": "vsgr4JAbRG", "reviewer": "Reviewer_mc8g", "summary": "The paper presents NFTS, a hierarchical method for neural architecture search in the few-shot image classification domain. The proposed framework engages various ways to adapt ResNet and ViT architectures to the support set including fine-tuning and adaptation parameters and then performs a search to identify the best-performing combination amongst each search path. The total number of paths is limited to both address computational limitations and prevent overfitting. Experiments on Meta-Dataset and Meta-Album demonstrate the strong efficacy of the approach when adapted inside a prototypical classifier with ResNet and ViT backbones. Ablation studies provide insights into various aspects of the method.", "questions": "Please address the questions and limitations noted above. Overall, I believe that this is a strong submission, and the broader research community can benefit from it. I believe that the empirical results of the paper need to be verified in terms of statistical significance by providing the appropriate confidence intervals across the reported numbers. This is the only major weakness in the submission, and once addressed with the other limitations noted, I would be more than happy to recommend the paper for acceptance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well-written.\n- NFTS is empirically effective and demonstrates a good balance between enabling adaptation using the support set while preventing overfitting. Clear empirical evidence shows the model's ability to select a more optimal architectural combination than previous baselines.\n- Experiments are extensively performed on large-scale datasets that demonstrate the efficacy of NFTS across both ResNet and ViTs.\n- Ablation studies justify various architectural choices made such the total number of search paths and the granularity of options.", "weaknesses": "- Empirical results reported lack confidence intervals. Although I suspect this is due to space limitations, they should be included to verify the statistical significance of the results report and for better comparison with baselines. If some results do not meet statistical significance, they must be modified when presented in results tables to reflect so accordingly and the claims made need to be adjusted.\n- Ablation study on search paths shows that N=3 not only provides better computational efficiency but additionally prevent overfitting on the support set. How does it compare with N=2 or N=4? I believe that further insights here would be useful as to how this hyperparameter is set. Furthermore, how does performance vary depending on N across ViT and ResNet?\n- Meta-dataset baselines that are compared to omit some recent methods that can be included for completeness of comparison [1, 2, 3].\n\n[1] Improved Few-Shot Visual Classification\n[2] CrossTransformers: spatially-aware few-shot transfer\n[3] Enhancing Few-Shot Image Classification with Unlabelled Examples\n[4] Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AWVZ9su2qx", "reviewer": "Reviewer_eX3h", "summary": "1. This paper provides the first systematic Auto-ML approach for finding the optimal adaptation strategy in few-shot learning.\n2. This method designs a novel strategy for defining the search space.\n3. The proposed method, namely NFTS, outperforms state-of-the-art methods in both Meta-Dataset and Meta-Album benchmarks.", "questions": "Could you offer some insights about their consistent adaptation of (α) block 14 and their lack of adaptation for block 9 in the 'Discovered Architectures' paragraph?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for introducing NAS into FSL is good, as mentioned in this work: current FSL works have started to understand the trade-off between frozen weights and trained parameters. It makes sense to automatically search for the best configuration instead of manual search or \"carefully tuning learning rates.\"\n\n2. The experimental results present the superiority of NFTS; it achieves a significant performance gain on the Meta-Dataset.\n\n3. The analysis is interesting as it shows the trend that the best-searched configuration does perform the best in the unseen downstream.", "weaknesses": "The results lack significance compared to the additional training required to obtain NFTs. The method requires training a supernet, performing an evaluation to find the best subnet. As NFTs achieve only a less than 1% accuracy gain on the Meta-Dataset in a multi-domain setting, the method is excessively computationally expensive and inefficient when compared to the actual performance gain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "STuqdDJf8j", "reviewer": "Reviewer_pojm", "summary": "- The authors propose the optimal adaptation method through the lens of neural architecture search (NAS) in few-shot recognition.\n- Given a pre-trained neural network, the proposed algorithm discovers the optimal arrangement of adapters, which layers to keep frozen, and which to fine-tune.\n- The authors demonstrate the generality of our NAS method by applying it to both residual networks and vision transformers and report state-of-the-art performance on Meta-Dataset and Meta-Album.", "questions": "- What is the most differentiating point from the prior NAS structures?\n- How many parameters increased by adapting?\n- Could authors provide parameter tables comparing NFTS (ResNet18) with others? The detailed architectural layout could be helpful to understand better.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(+) The proposed methods find some interpretable trends using layer-wise adaptations, which include the early/late layers of ResNet and ViT.", "weaknesses": "- (-) The authors stated the superior performances in various experimental settings. However, the author didn’t specify the structure and the number of parameters.\n- (-) There is no ablation study on the two-stage search for optimal path (sec. 2.4): the best-performing path during training time, the searching path at test time, and the proposed hybrid one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)