Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Authors: Haidong Kang
Venue: ICML 2025 poster (2025)
Keywords: Few-shot learning
Training-free
MAML
Representation Learning
Primary area: general_machine_learning->representation_learning

Abstract: Neural Architecture Search (NAS) has recently outperformed hand-designed networks in various artificial intelligence areas. However, previous works only target a pre-defined task. For a new task in few-shot learning (FSL) scenarios, the architecture is either searched from scratch, which is neither efficient nor flexible, or borrowed architecture from the ones obtained on other tasks, which may lead to sub-optimal. Can we select the best neural architectures without involving any training and eliminate a significant portion of the search cost for new tasks in FSL? In this work, we provide an affirmative answer by proposing a novel information bottleneck (IB) theory driven \textit{Few-shot Neural Architecture Search} (dubbed, IBFS) framework to address this issue. We first derive that the global convergence of Model-agnostic meta-learning (MAML) can be guaranteed by only considering the first-order loss landscape. Moreover, motivated by the observation that IB provides a unified view toward understanding machine learning models, we propose a novel Zero-Cost method tailored for FSL to rank and select architectures based on their \textit{expressivity} obtained by IB mechanisms. Extensive experiments show that IBFS achieves state-of-the-art performance in FSL without training, which demonstrates the effectiveness of our IBFS.

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Haidong Kang
Abstract
The best directions:
Neural Architecture Search (NAS) has recently outperformed hand-designed networks in vari- ous artificial intelligence areas. However, pre- vious works only target a pre-defined task. For task in few-shot learning (FSL) scenarios, the architecture either searched from scratch, which is neither efficient nor flexible, or bor- rowed architecture from the ones obtained on other tasks, which may lead to sub-optimal. Can we select the best neural architectures without involving any training and eliminate a signifi- cant portion of the search cost for new tasks in FSL? In this work, we provide affirmative an- swer by proposing novel information bottleneck (IB) theory-driven Few-shot Neural Architecture Search (dubbed, IBFS) framework to address this issue. We first derive that the global convergence of agdel-ag meta-learning (MAMI.) can be guaranteed by only considering the first-order loss landscape. Moreover, motivated by the ob- servation that IB provides unified view toward understanding machine learning models, we pro- pose novel Zero-Cost method tailored for FSL to rank and select architectures based on their ex- pressivity obtained by IB mechanisms. Extensive experiments show that IBFS achieves state-of-the- performance in FSL without training, which demonstrates the effectiveness of our IBFS.
The less costs (search/train/meeooy the better, and the higher accuracy and generalization the better.
IBFS
AutoMeta
MAML
Vanilla NAS
Costs (search/train Generalization /memory)
Accuracy
Figure I. IBFS Peer competitors in terms of costs, accuracy. and generalization.
raises an important concern about whether the frameworks of the DNNs are ideal for those tasks. Naturally, NAS (Liu et al., 2019b: Zoph al., 2018) is straightforward solution for exploring frameworks for those tasks (i.e., FSI.) due to its automated search manner, which can automatically search high- gigg-performance architectures for the aforemen- tioned tasks.
Problem statement. Though promising, conventional NAS has two fundamental limitations. First, conventional NAS methods only focus on single task. As shown in Fig. 2a, task,
vanilla NAS only targets predefined target, for the architecture is either searched from scratch, which is Hime-consuming. or borrowed from the ones obtained on other tasks, which may lead to sub-optimal. Second, as depicted Fig. 2b, despite adaption NAS (Kim et al., 2018; Lian al., 2020; Elsken et al., 2020) based on meta-learning can enhance the generalization of searched architecture on unseen tasks, however, it is extremely time-consuming, quiring over 100 GPU days (Kim et al., 2018) for searching.
1. Introduction
Deep Neural Networks (DNNs) have shown remarkable performance on various challenging real-world tasks, i.c., image recognition (He et al., 2016), and few-shot learn- ing (Finn et al., 2017; Sun & Gao, 2024). However, this
'College of Software, Northeastern University, Shenyang, China. Correspondence to: Haidong Kang <hd- kang @ www.mmi..uu.cdu.cmm
Our goal. In this paper, we aim to address the above two fundamental limitations by revisiting neural networks tai- lored for FSL from a zero-shot NAS perspective. As shown in Fig. our IBFS achieves better performance in terms of
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
T
T
T2
Tp
T1
NAS Learner Meta Leamer
Meta Architecture
Training/ Testing
Training
NAS Learner
Meta Learner
Testing
Training Testing
Tu+11
T
update
Without training
Enem
Specific learner fortask
Meta
NAS learner
T
Tn+2
T n+m
T Ta+1
Tn+2
T Tn+m
(a) Neural Architecture Search (Vanilla)
(b) Adaption Neural Architecture Search (meta- learnin)))))g))
(c) Adaption Neural Architecture Search (Ours)
Figure 2. Illustration of our IBFS and related approaches. (a) Vanilla neural architecture search. (b) Adaption neural architecture search. (c) The proposed IBFS can find the best meta architecture without training multiple unseen tasks, For the scenario of our IBFS, there is no need to restart the training of NAS after updating the meta weights for new task learning.
The correlation of various zero-cost proxies
Motivations. To this end, we analyze which properties of MAML determine the direction of optimization for FSL. By carefully analyzing the global convergence of MAML, we derive Theorem 4.1, which indicates that the global conver- gence of MAML can be guaranteed by only considering the first-order loss landscape. Hence, Theorem 4.1 provides shortcut to the global convergence of MAML that only considers the properties of the architecture sampled from the search space for the FSL at any initialization. Now, the problem of crafting a specialized architecture for FSL can be transferred to finding a suitable proxy to accurately mea- sure the expressivity of architecture without training. Then, we explore the relationship between existing proxies (i.e., NASWOT) and the accuracy of architectures (as shown in Fig. 3) in first-order way. To enhance the effectiveness of the aforementioned empirical validation, simple proxies, i.e., Kullback-Leibler (KL), and Entropy. To be specific, Fig. 3, can yield two key conclusions: No matter whether the networks at initialization or well-trained, we can clearly observe that there positive correlation between zero-cost proxies (Mellor al., 2021; Chen et al., 2021b) and the actual test accuracy. In brief, it is possi- ble to accurately measure the expressivity of architecture sampled from the search space for FSL without training. @ We also observe that existing proxies (i.e., NASWOT) suffer from larger score variance, which will degrade the accuracy. This is because that larger score variance unstable. In contrast, our proposed simple proxy, namely, Entropy, consistently obtains higher accuracy under various architectures, due to smaller score variances. Consequently, these observations suggest that designed proxies toward FSL should obey the principle of smaller score variances.
0.98
Accuracy
0.94
Entropy
0.9
10
Proxy value
Figure 3. Zero-cost proxies u.s. Accuracy. To be specific, sam- ple models (e.g., densenet40, seresnet20, resnet56, pyramidnet110) from PytorchCV, as well as retrieve their real test accuracy. Then, we use various zero-cost proxies (i.c., Entropy, KI., NASWOT) to measure the expressivity of those models. For the convenience of showing. we reduced the proxy value and accuracy by 100 and 10 times, respectively. We clearly observe that existing proxies (i.e., NWOT) suffer from larger score variance. In addition, we provide more discussion in Section 1.
propose two
costs, accuracy, and generalization than peer competitors. Challenges. However, to address the limitations of the aforementioned, we target the following key questions:
Which properties of MAML. impact the global conver- gence of FSL?
Can we find a simple proxy that well evaluates the ex- pressivity of the architecture sampled from the search space for the FSL.?
Motivated by the principle of IB theory (Tishby et al., 2000; Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017b:
2

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Kawaguchi al., 2023; Hu et al., 2024), find that the information entropy varies different architectures, how- ever, Kendall's Tau between accuracy and information en- tropy remains unchanged (as shown in Fig. 4) with the increasing epochs. Notably, the variance of the informa- tion entropy is very small, which shows that information entropy based on the IB theory is promising proxy tailored for FSL. Grounded in empirical observations, in this paper, propose novel method, called IBFS, measuring the expressivity of neural architectures sampled from the prede- fined search space for FSL. by exploring their information entropy properties. The information entropy eliminates the influence of various gradients of architectures from search spaces, making the IBFS pay more attention to the learning of multiple tasks.
using meta-learning (Kim al., 2018; Lian et al., 2020; Elsken et al., 2020). However, these approaches cost more than 100 GPU days in the search stage, which is extremely come-cons By contrast, our proposed IBFS can find FSI--friendly architecture for new tasks without involving any training, which is very efficient. The superiority of IBFS can be attributed to the powerful IB theory, measuring the expressivity of neural architecture.
2.2. Information Bottleneck Theory
The IB theory (Tishby et al., 2000; Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017b; Kawaguchi et al., 2023; Hu al., 2024) offers comprehensive framework for understanding machine learning models that involve latent variables (Shamir et al.. 2010; Shwartz-Ziv & Tishby, 2017 Slonim & Tishby, 1999; Tishby et al., 2000). According to this theory, in the context of input x, true label y, and latent representation 2, the objective of supervised learning is to minimize the mutual information [(2; x)) while preserving as much mutual information T(2:)) as possible. By achiev- ing this, a predictor with such a representation eliminates unnecessary transformations present in while retaining crucial information about y. In the analysis of deep neural networks, the latent variables 2 correspond to the hidden layer activities z(1) at each layer 1. The learning process can be visualized as a trajectory the (T(z; [(:; coordinate space. This approach, known as information plane analysis, has received significant attention in recent years (Yu et al., 2020).
Contributions. The key contributions of this paper are:
To solve the bilevel optimization of MAML, we derive that the global convergence of MAML can be guaran- teed by only considering the first-order approximation of the loss landscape, which transfers the problem of crafting a specialized architecture for FSL to find suitable proxy, accurately measuring the expressivity of architectures without training.
We propose a novel IB driven method, called Few-shot Neural Architecture Search (IBFS), which can learn a meta-architecture tailored for FSL without training.
Extensive experiments on NAS- MAS-Bench-201 (Dong & Yang, 2020) and few-shot tasks show that IBFS achieves comparable performance in FSL with surpris- ingly smallest search costs, which demonstrates the effectiveness and superiority of our IBFS compared with its peer competitors.
2.3. Meta Learning for FSL
Meta-learning family (e.g., MAML (Finn et al., 2017) and its variants (Finn et al., 2019; Xu et al., 2020; Bai et al., 2021; Fallah et al., 2020b; Ji et al., 2020; Rajeswaran et al., 2019; Zhou et al., 2019; Tack et al., 2022; Sun & Gao, 2024; Kang et al., 2023)) is the most promising approach for FSL, which can quickly generalize to a new task from only a few samples by training a meta learner. Despite promis- ing, those approaches rely neural networks/architectures (i.c., ConvNets, ResNets) tailored for supervised learning benchmarks (i.e., ImageNet), and little attention is focused on analyzing the role of architectures for FSL. In this paper, we provide an affirmative answer by proposing a simple yet effective method, called IBFS, to design FSL-friendly architectures from a zero-cost NAS perspective. Notably, our method does not limit the scope of MAML, it can also be generalized to its variants.
2. Related Works
2.1. Neural Architecture Search
Recently, various Neural Architecture Search (NAS) ap- proaches have emerged jointly optimize the weights and the architectures (Wu et al., 2019; Chu et al., 2020b; Chen et al., 2021c; Ye et al., 2022). This is achieved by various search algorithms including Reinforcement learning (Zoph & Le, 2017; Baker et al., 2016), Evolutionary algorithm (Xie et al., 2018; Real et al., 2019b;a), and Gradient de- scent (Liu al. 2019a; Yu et al., 2019; Wang et al., 2020: Chu et al., 2020b; Ye et al., 2022). To accelerate NAS, training-free free strategy (i.e., without training in the search stage), which significantly reduces search costs (Mellor et al., 2021; Xu et al., 2021; Chen et al., 2021b; Abdelfattah et al., 2021). Recently, there have been a few works to explore the generalization of architectures sampled from search spaces by combining NAS with FSI.
lot of works focus
3. Preliminaries
In this section, we introduce the concept and problem for- mulation of the NAS and FSL.
3

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Neural Architecture Search. We consider a NAS searching process. Typically, the whole searching pipeline is organized as Directed Acyclic Graph (DAG), suppose that we have task T and each task is composed of b' data samples X Accordingly, the objective function is defined as:
0.7
0.6
0.5
gatterr (W", a.), S.T. W = arg min lim (W,a), W
Kandail's
0.4
0.3
where W' denotes the weight of the architecture from search spaces, denotes the mixing weight of the operations, and W' denotes the weight of the best architecture.
0.2
0.1
Entropy
Few Shot Learning. Suppose that we have a set of tasks T T T (i.e., multiple datasets, multiple tasks, or multiple loT devices, etc.). Then, each task 7 is composed of ''' data samples (X*, ) and (Xt, Y²) represented by support samples (i.e., training samples of Derain) and query samples (i.e., test samples of Dlest), respectively. The loss function L can be formulated as follows:
10
30
50
80
100
120
Epoch
Figure 4. Correlation E.S. Epochs. The correlation coefficients (Kendall's 7) between zero-cost proxy (namely, entropy) under varying number of epochs and real test accuracy of models. where those models (e.g.. densenet40, seresnet20, resnet56, pyramid- net 10, wrn16) are trained on CIFAR-10 datasets.
Ee(D G(() ( A.
min
.a; A); E (1,
(2)
(second-order gradient) in the first-order Jacobian matrix complicates the optimization problem in MAML. If is pos- sible to quantify the impact of the Hessian term on the over- all optimization within a certain range, it would simplify the analysis of MAML'S convergence properties. Fortunately, several lines (Antoniou et al., 2018; Fallah al., 2020a; Wang et al., 2022b; Ji et al., 2022) demonstrate that MAML can reach a global minimum that gradient descent with sufficiently small learning rate. In this paper, we provide a proof for the global convergence of MAML as follows.
where A and W are the neural architecture and its parame- ters, respectively. G denotes the computation of parameter updates using one or more gradient descent steps, For con- venience, we employ the square loss function E(Y't, y 1 - Y' in this paper. As the gradient-based meta- learning algorithm, we utilize Reptile (Nichol et al., 2018), which has demonstrated comparable performance to MAML (Finn et al., 2017) in few-shot classification tasks by utilizing only first-order gradient information. However, traditional NAS methods that involve gradient-based or evolutionary computation for updates tend have lengthy search pro- cesses. Combining such methods directly with FSL often leads to sub-optimal results.
Theorem 4.1 (Global Convergence of MAML). To clearly illustrate the global convergence of MAML, we borrow one of the conclusions from MetaNTK-NAL (Wang et al., 2022b), which provides a Neural Tangent Kernel (NTK) perspec- live to understand MAML. While MetaNTK-NAS provides valuable theoretical guarantees, its proof is highly intricate and relies heavily on NTK theory, which can be computa- tionally expensive. In contrast, this paper presents a novel and more accessible proof for the global convergence of MAML. Moreover, the detailed proof is provided in App. A. At first, we define F, sampled from search space A, 1 denotes the number of i-th hidden layer. miin ( The min- imum singular value of a matrix tb likely related to the Jacobian of the network's 's output with respect to its parame- ters. nu a constant related to the learning. rate. rate-snner- V F( Xt, Y!), Ir limi 1 Einner Oynner and For arbitrarily small 5 > O. and there exist G > 0, 1" E N., and Xo > O, hence, for width 2 1", "'perrfom gradient descent with learning rates Ir < 1pp and X < da with random initialization, from the 2q.2, we give an upper bound on the above loss function:
4. Design of FSL-Friendly Architectures
The metrics designed for supervised learning may not be suitable for FSL., due their optimization objective and scenario being quite different. In this section, our goal is to build framework that yields the meta architecture A with corresponding weights WA. Then, given a new task T, both a and WA should quickly adapt to T₃ based on a few labeled samples. To fulfill this goal, we propose IBFS, a novel method that naturally combines FSL. and Zero-shot NAS. First, we analyze the convergence of MAML for the FSL. Then, propose IBFS to evaluate the expressiveness of the architecture. In addition, we will describe how the meta architecture encoded by a can be quickly specialized to new task without requiring re-training of WA-
@ Emnal1 Etminff
MAML AMonel-Agnostic Meta-Learning) is a classic meta- learning approach for ESI. However, the Hessian term
4

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Examples include information implicit in image tasks or neural architectures. The main inquiry is to explore those features of æ that play a role in the prediction of the neural architecture's 's output We define this problem as squeezing the information about x provided by through a "bottle- neck" formed by some metric on the neural architecture. Given a neural architecture F; at initialization, Y = (X),,, we try to construct a function for measuring the expressivity of a neural architecture based on the following IB principle. As follows, the amount of information about Y in I is given by:
((W1) S (1 - T. 7J00 =))0min())) (
T E (0, 1)
(3)
Remark. Theorem 4.1 indicates that there exists a global minimum with any initialization parameter W in small region, and the Jacobian of Linner is stable during the updating of parameter W in the episode meta-train and meta-test. According to the theory of flatness of local minima (Keskar et al., 2016; Hoffer et al., 2017; Cha et al., 2020), the flatness of the loss landscape near the global minimum represents MAML having strong generalizability. More importantly, we can drive that the MAML is first-order linner conver- gence problem from Theorem 4. Then, according to the principle of IB, we evaluate the expressivity of the architecture sampled from the search space for the FSL by calculating the information entropy of the distribu- tion of its inner loss Cinner outputs. Fig. illustrates that Kendall's 's Tau values for model accuracy and the expressivity of the model computed by our method re- main unchanged across different epochs. Specifically, we sampled five models (i.e., densenet40, seresnet20, resnet56, pyramidnet 10, wrn16) trained with various epochs, and calculated Kendall values using our met- ric in that epoch. In Fig. 4, entropy represents the score derived from our metric. Each line in the plot with six points indicates that the value of Kendall's Tau between model accuracy and entropy is the same at different epochs for multiple models. In essence, model expressivity, an intrinsic characteristic, remains stable and can be accurately computed without any training, even at initialization. This experiment vali- dates that our proposed metric can accurately measure model expressivity without the need for training.
I(RYY) - ΣE EEy, r) log p(y.7) p(y)p(r)
S I(X;Y) (4)
According to (Tishby et al., 2000), we give an objective function that finds the trade-off between the input x and neural architecture F; output y, the objective function can be formulated as follows:
BI(R)),
(5)
where 3 2 0 is the Lagrange multiplier attached to the con- strained meaningful information, while maintaining the nor- malization of the mapping p(r(az) for every z. Introducing X(x) for the normalization of the conditional distributions p(r(ar), the Eq.5, becomes:
C|p(r|=)] I(x: - BI(R.Y) Σ (az)((((z)
ΣΣ p(r, 1)) log p(r, x)
p(z)p(r)
ΣΣ plu,r) log ply,r) 8 p(y)p(r)
((x)p(1aa) (6)
2,
Σ p(r, log p(r,z)
p(z)p(()
4.1. Learning Rapid Adaption of Neural Architecture via the IB Principle
- -3I(X;Y) Σ ((x)p(1(z),
Design Principles. Given multiple neural architectures F2..,. Fn sampled from search space A with parameter w; at initialization, when randomly sampled batch data X as input, how to correctly rank these neural architectures without training becomes a key challenge. To this end, we bring the IB theoretic view and explain how it helps with ranking neural architectures at initialization. We begin by defining several terms in our analysis: x E X, the output y E Y, the features of neural architectures E Rt and some functions for measuring information of F,, i.c.. entropy (X) = Expaa] log p( and Kullback-Leible (KL) divergence (T.(p(z))lg(z))
Finally, we obtain the final optimal solution by minimizing Eq.6:
8L 5p(r(z)
p(x) log p(r(a) p(r)
+ 3 Σ p(yla) log p(ulz))
X(x) 0,
p(y|r)
y
where M(x) A(x) 3 E, p(y| |2) log
p(vl=) If we
p [lo p((=) q(=)
p(x)
p(0)
choose entropy H(X) = Ep(aa| log p (z) as the func-
We define relevant information in a signal x E X as that signal providing information about another signal y E Y.
tion to measure distortion of neural architecture F,, the p(ra)), becomes:
5

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Table 1. Comparison results on NAS-Rench-201. Red, blue, and orange indicate the best, second-hest, and third-best results, respectively.
CIEAR-10
CIEAR-100
AmageNet-16-122
Method
Year
Search
validation
validation
validation (9)
(S)
Search Methods
ResNet(He at al. 2816)
CVPR2016
93.97
70:86
43.63
Manual
Now wreiht sharing
REAIZoph al. 2018)
CVPR2018
12000
91. 19+0031
93, 9.992++11.. 30
7181+1112
21,86+0099
45.15+0.89
45.54=1.08
BOHBO. et 2018)
ECCV2018
12800
90.82:0.533
93.6 33.61:10.52
1..74
70.85::1.28
44.26:::.36
44.42::1499
HPO
REINFORCE(Realdd et ol, 20196)
AAAI2019
12000
91 0.330
93,85+0,57
7..1+1112
11,71+1.09
45.05 1.02
45.34=1.18
RL.
Weight sharing
SNAS (Xie et al. 2020)
ICLR2018
90 10:1001
92.77200.83
69 69.6912.339
69.3411 1,98
42.8421.79
43.1612.64
GD
ENAS:Phum 2018)
KCML2018
13315
39.77210.00
54,30100.00
15.02:1..00
15,61
16.43:0.000
16.32=00.00
RL
DARTS-V2/Liue al., 20196)
ICLR3019
39.77100.00
51,30.0,00
15.6110.00
16,13.0.000
16.30.00.00
GDASILoog Yang, 2019)
CVPR2019
28926
90.00
93.51
71.14200.27
70.61::0.20
41.70 1.26
41.84=00.0
GD
DSNAS (Hu ot al, 2020)
ICIR2009
89 66-6660022 29
93.08+0.113
30,87+16.4 40
31,01+16.38
40.61+0.09
41.07=0.09
GD
PC-DARTS (X al.
ICLR2000
89 15
67.12.0.39 39
40.63310088
41.311:0:22
RSPS/Li& & Talwalkar, 2020)
UA120200
7587
84, 16.11611:1 1.09
87.66 1,09
59.00114.60
58.33114.34
31.56133.28
31.14
RS+WS
iDARTS (Zhang al., 2021a)
ICML2021
89 09..66.0.0 60
03.58.0.32
70.57.0.24 24
70.8310.48
40.38.0,59
40.89,0.68
GD
(lang al., 2023)
NeurIPS2023
90.88a0.00
93.70=0.15
20.56a0.28
70.40+0.22
44.17200.49
43.97=0.28
IS-DARTS (He al., 2024)
AAAI3024
7200
91. 550.500000
94, 9600000
73.49+0000
73.31+0..0
46,3740:00
46.34-0000
GD
Training free
Random
K3.20 13.28
60.70 12.55
60.83 12.58
33.34 9.99
Random
NASWOT (Mellor et al., 2021)
ICM15501
4.4
88.47 + 1.33
91.53 1.62
66.49 + 3.08
66.63 3.14
38.33 4.9%
38, 5.22
7ero-ont
GradSign (Zhang & Jia, 2022)
ICLR2022
30.38
93.52 0.19
70.57 0.31
41.89 0.69
Zero-col
ZiCo al., 2023)
93.50 0.18
42.04 0.82
2010 cost
AZ-NAS (1.z2& Ham, 2024)
CVPR22044
0.71
93.53 + 0. 0.723
70.75 0.44
45 0.29
zero-ooet
SWAP (Peng 2024)
ICLK2024
4.7
87.31
2.36
90.48 0.94
65.92 4.332
67.13 1.83
33.85 4,98
35.40 3.96
Zero-cost
(BFS(ours)
91.55 0.76
994.77
73.31 2.12
73.09 2.08
45.59 0.32
46 1.2
Zero-one
Opimal (NAS-Bench200)
NIA
94.37
73.51
47.31
N/A
at initialization implies the network architecture obtains a higher accuracy after convergence.
p(r[z) = p(f)
exp( (X)()
{(x)
p(f) exp(- -33 log p (x)))
5. Experiments
X(x)
5.1. Experiment Setup
< H * exp(-
log ())))
To validate the effectiveness of the proposed proxy, first evaluate our IBFS framework on search space of NAS- sench-201 & Yang, 2020) in three supported datasets (CIFAR-10, CIFAR-100, ImageNet- (Chrabaszez et al., 2017)). Then, to validate the effectiveness of our IBFS in designing the FSL-friendly architecture, we conduct comprehensive experiments two popular few-shot image classification datasets, mini-ImageNNt, and tiered- IetgeNet with the wide of peer competitors, which both are subsets of ImageNet (Deng et al., 2009). (1) mini -er (Vinyals et al., 2016): It contains 60,000 RGB images of 84x84 pixels extracted from ImageNetIK (Deng et al., 2009). It includes 100 classes (each with 600 images) that are split into 64 training classes, 16 validation classes, and 20 test classes. (2) fiered-Iaaeete (Ren al., 2018): This dataset contains
where H is scalar.
Given neural architecture F, at initialization, first, we measure the expressivity of a neural architecture based on the above IB principle. Concretely, we define a mapping from X, through untrained architecture F; with parameter wi, construction of kernel Jacobian matrix, input a batch
with B data Zj. where T E X < B, J; = Bu(z.)) To evaluate the performance of the architecture F; on different inputs data xy, we constructed the following Jacobian matrix J:
da;
J (
(ww (21) dx
Ow (IB) ŌxB
T
(9)
The intuition to our method is that different neural architec- tures have different Jacobian J values, based on the N eigen- values $1, S2; ..., Sy of the Jacobian matrix, we construct an entropy-based IB formulation for the neural architecture F,, The untrained architecture F; can be scored as:
779, .65 RGB images of 84x84 pixels extracted from Ima- g K (Deng et al., 2009). It includes 608 classes that are split into 351 training, 97 validation, and 160 test classes.
Peer Competitors. For FSL, we compare our method with a wide scope of the state-of-the-ard NAS-based base- lines, as follows: (1) MAML(Finn et al., 2017); (2) ANIL (Raghu et al., 2020); (3) MetaOptNet(Leee et al., 2019); (4) RFS(Tian et al., 2020); (5) AutoMeta(Kim et al., 2018); (6) T-NAS++(Lian et al., 2020); (7) MetaNAS(Elsken et al., 2020); (8) H-Mela-NAS(Zhao et al., 2022); (9) MetaNTK-
N
N Neepressiiitty
Σ
p plogp
(10)
< H * crp(-Eppsaad log p(sx))).
The intuition for entropy-driven IB is that a higher score
6

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Table 2. Performance comparison the DARTS search space with ImageNer1k dataset, where "Img" denotes network models directly
searched in ImageNetIk, and "C10" and "C100" denotes network models searched in CIFAR- 10 and CIFAR-100, respectively.
Method
Top-1(%)
Top-5 %)
Params
(M)
PLOPS
(MU
Scarch Cost
(GPU-days)
Method
Search
ResNet30 2016)
75.3
922
25.6
4100
Manual
MabulNNeVV (Howard al., 2017)
70.6
39.5
Masual
MobileNatV2 (Sundler al. 2018)
74.7
91.0
3.4
300
Manual
ShuttleNe (Ma et al. 2018)
72.6
3.5
259
Masual
AmadeaNasAA (Zoph 2013)
924
Evolution
20181
Rl.
ERiisennNes-800 (Tnn Le, 2019)
76.3
95.3
390
5:3000
RL.
NASNet-A (Real et 20199)
74.0
91.6
5.3
564
2000
KL
( 20196)
73.3
Gradant
FBNet (Wo
74.9
Gradieut
PC-DARTS(mgg (Xo al., 2019)
75.8
92.7
533
597
3.7
Gradient
P.DARTSICIO0 (Ches al. 2019)
75.3
92.5
5.1
577
0.3
Gradicut
DARTS- (Lang 2019
Gradient
DARTS-(Img) (Chu 2030g)
76.2
93.0
Gradiett
FiiDARRS-B((igg) (Chu al. 20200s)
75.1
92.5
4.8
541
Gradiaut
SNASICIO: (Xe al. 2020)
72.7
90.8
522
L.5
Gradient
DARTS-PTICTI) (Wang
Gradiant
DOTS/C1)) eG al., 2021)
75.7
92.6
0,3
Gradient
A=DARTS(CI00) (Ye 2022)
75.8
92.9
54
597
0,4
Gradient
A DARISS (Movahedi
75.7
Gradicnt
OLDS (ling et al. 2022)
Gradient
FR-DARTSIIO) (Wing et 2023)
75.7
92.7
0.08
Gradient
TAARTS-AFR" (Ting at al. 2023)
76.0
92.8
5.1
578
2.0
Gradicut
[S-DARTS
Gradient
NAO (Lave et at, 2018)
74.3
91X
Proty
SemNNAS (1 sl. 2030)
76.5
93.3
6.3
599
Proxy
WeakNAS (Wa al. 2021)
76.5
93.3
55
591
2.5
Prosy
TENAS (Chen
201ee)
Trsining-free
NASI ADA/C1I (Stu et al., 2022)
25.0
92.2
4.9
559
0.01
Training free
SWAP (Shu et all, ((mg)
75.0
92.4
58
0.006
Trriningfie free
IDFSCCIO)
76.7 110.3
93.5
00042 ((00022)
NAS(Wang et al., 2022a). In addition, also compare our method with a wide scope of the state-of-the-art non-
factor of 10x. As for tieredImageNet, the initial learning rate is set to 0.2. We apply 10x learning rate decay at epochs 20, 40, and 60, 80.
NAS-based baselines, as follows: (1) MAML(Finn et al.,
2017); (2) ANIL (Raghu et al., 2020); (3) COML
Metrics, In line with common practice, we give priority to the relative ranking of architectures and utilize Kendall's Tau (Sen, 1968) and accuracy as the evaluation metrics. Kendall's Tau is widely adopted measure for assessing the quality of performance predictors. It aims to evaluate the rel- ative ranking of candidate architectures and provides values within the range of [-1, 1j. A higher Kendall's Tau indicates a stronger alignment between the predicted ranking and the actual ranking, thereby indicating better performance.
et al., 2022); (4) Meta-AdaM(Sun & Gao, 2024); (5)
GAP(Kang et al., 2023); (6) MetaDiff(Zhang et al., 2024):
(7) MetaOptNettLee al., 2019); (8) CTM(Li et al., 2019):
(9) RFS(Tian al., 2020); (10) MAML+ALFA(Baiİ
al.,
2020); (11) MAMLse Oswald et al., 2021); (12)
McTAL(Baik et al.. 2021); (13) ClassifierBaselieeCCnn
et al., 2021d); (14) MetaQDA(Zhang et al., 2021b); (15)
MAML+SiMTTTTcc et al., 2022); (16) COeLN et al., 2022).
Code. We have implemented our code using the Py ToTch framework (Paszke et al., 2019). Specifically, for the NAS search stage, we have built upon the codebase provided by (Chen et al., 2021b). This serves as the foundation for our implementation. On the other hand, for the training and evaluation stages, we have utilized the code provided by (Tian et al., 2020).
Implementation Details. In our study, we construct the neural architecture by stacking either 5 or 8 searched cells together. Within the architecture, there are specific cells positioned at 1/3 and 2/3 of the total depth of the network, referred to as reduction cells. These reduction cells serve the purpose of decreasing the spatial resolution while doubling the number of channels. The initial number of channels is set to 48 starting point for the architecture.
Hardware. The majority of our experiments were con- ducted using NVIDIA RTX 2080Ti GPUs, while the re- maining experiments were run NVIDIA RTX A100 80G GPUs. Each experiment was executed on single GPU at a time to ensure consistent and reliable results. The search cost of IBFS, specifically, was benchmarked using NVIDIA RTX 2080Ti GPUs.
Optimization Setup. In line with the approach proposed in (Tian et al., 2020), we employ the stochastic gradient descent (SGD) optimizer with a momentum of 0.9 and a weight decay of 0.0005. The training process for all models consists of 120 epochs for minilmageNett and 80 epochs for tieredImagenet. Regarding the specific learning rate sched- ules, for minilmageNet, we start with an initial learning rate of O. At epochs 40 and 80, the learning rate is decayed by
7

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Table Comparison results FSL. henchaarks. Red, blue, and orange indicate the best, second-best, and third-best results, respectively.
mini-ImageNee 5-way
Bered--mageNN S-way
Method
Year
Arch.
#Cells
Train
dParam. Search Cost 1-shot
5-chot
#Param. Search Cost I-shot
5-thot
MAMI /Fin al. 2017)
ICML.17
Cones
MAMI.
30k
48.70++1.84
63.11+0.22
30k
51.67+1881
70.30+1.75
ANIL (Rugha et 2020)
ICLR20
Conel
ANIL
30K
48.0.10.7
62.210.55
COMLNDeeku et.l. 2022)
LCLK22
Coned
53,0110.662
70.4100.54
$4.3011.099
71.2520.77
Miss-AdaMSunn Gao, 2004)
NeurIPS23
Come
Mota Learinng
52.00:+0049
70.70++0.49
53,93+0.499
72:66+0.49
GAP(Kang et al., 2023)
CVPR23
Conel
54,86:10.85
71.55 10.61
57 57.60:10.93
74 74.9020.68
MesaDifi2hang et al., 2024)
AAA24
Conel
Meta Leurning
35.06.10.81
73.18 10.64
59,7740.900
75.1610.69
MasaOyNNetLLe ctal. 2019)
CVPR19
ResNet-12
MetotiptNec
12.5M
7863+0.46
12.7M
05.99::0.72
81.56=0055
CTM(Li 2010))
CVP219
R8
64. 64.12+0..8
80.51+0.13
68. 41 68.41
84 84-28+1.75
RFSITian 2020)
BCCV20
ResNet-12
RES MAML
12:5M
62. 62.02.10.63
79.6110.44
12.7M
69.71.10.72
10.55
MAML-ALPMBBac 2020)
NeurIPS20
ResNet-12
59. 49.7440.49
71.9640.47
64.2240.49
N2.48=0.39
Squne Owwald at, 2021)
NeurIPS21
ResNer-12
MAMI.
56. 39++0.35
73.0+
53. 53.4750..53
66.83-0.65
MrTAL[Baik et al., 2021)
ICCV21
ResNet11
Meta Learning
39, 39.6410..88
76.2010.19
63.8910.48
80.1410:40 40
CassiierBasclleeCCoo et 2021d)
ICLY21
ResNet-12
Meta Learning
61.22::0.44
78.7240.00
09.7110.88
83:87 20.64
MesQDA/Zhang-r of al,, 2021bb
ICCV21
ResNer-12
Beyesion
65. +0.66
$0.96+00.75
69.97 +00.52
85.51 +0.58
MAML-SSMMTBBcc et al. 2023)
NeurIPS22
ResNet12
MAML
51.49.10.18
68.7410.12
52.51110221
69 69.58:10.11
COMLNiDelu 2022)
LCLR22
ResNet-12
39,2611..55
71.2610.49
62.9310.71
$1.13 10.53
Mesa-AdaMtSun & Gan, 2024)
NeurIPS23
ResNer-12
Meta Leanning
59, 59.89+0049
77.92+043
65. 65.31+0.48
KS. 25.24+0.35
MesDiifiZhnng al. 2024)
AAAI24
ResNat112
Meta Leaming
64.99:0.777
21 10,56
02.33+0.92
06-11-062
AmooMcta Kim al. 2013)
NeurlESSS
Cells
Repuie
100k
37.6102
7471022
ENAS==(Lian et al, 2020)
ICLK20
Cells
2
POMAMI
27k
lr
54.11=1.35
09.59+00.85
MisuNAS(Flalendd al. 2020)
CVPR30
Colls
Repelle
1.M
168 hr
63.1+0.3
79.5+0.2
MesNNASEElbkend et al., 2020)re
CVPR20
Cells
RFS
3.5M
63.88.10223
7988810.14
3.70M
he
72.3210.02
06.48
H-Mei-NAS(Zhba et al., 2022)
NeurIPS22
70.28K
$7.36+1.111
77.53+0.77
MesaNKK NAS((Nn al., 2022g)
CVPR22
Colls
RFS
3.1M
1.92
64.26::0.14 14
80.35+:012
4.78M
2.73
72.37.0.79
06.43-0.52
IBFS
Colls
3.0M
81.5220008
4.50M
0.10
723660002
8673+00008
5.2. Results on HAS-Bench-201
Table details the comparison between our IBFS and its peer competitors in terms of NAS benchmark (namely, NAS- Bench-201 search space) in CIFAR-10, CIFAR-100, 100, and ImageNet- 16 120 datasets. To be specific, as depicted in Table 1, the non-weight sharing methods suffer from huge search costs, which search the target architecture with a budget of 12000 seconds. For training-free and non-weight- sharing methods, we report the accuracies are averaged over 500 runs. As shown, it is evident that the IBFS consistently outperforms its peer competitors in all cases. In particular, IBFS only cost 3.82s, finding competitive architecture at initialization than other methods. In addition, our IBFS obtains a higher similarity value in terms of Kendall's s Tau than the existing zero-cost proxy (i.e., NASWOT) designed for NAS (0.752 vs. 0.422). Notably. NASWOT is designed for NAS in classification tasks, not for few-shot learning. While NASWOT and our method utilize the Jacobian ma- trix, however, the computation of the Jacobian matrix is totally different. NASWOT uses the Hamming distance be- tween two binary codes. By contrast, we directly utilize the gradient of the eutwork kernel. To sum up, this experiment provides a piece of critical evidence that proposed IBFS can design FSL-friendly architectures without training.
way 5 shot
8o
480x fower owercosse COSTS
1680x fewer costs
19.2 fower cods
70
munu dn Reccume A
0.67 % higher nce
60-
J0.44 % higher acc
Our IBFS MetaNAS MetaNTIKNAS
35
5 way shot
T-NAS +
0
25
50
75
100
125
150
175
Search costs (hr)
Figure 5. Comparison to SOTAs in terms of accuracy of FSL and Search costs in mini-ImageNet dataset (5 way 5 shot).
method achieves the Top-1 accuracy of 1.7% higher than SWAP, 1.6% higher than NASI-ADA, and 1.2% higher than TENAS. Notably, our method only searches in small CIFAR- 10 dataset, and then can well generalize to large ImageNetlk, which largely reduces search costs. Those empirical results illustrate that our method possesses strong generalization ability in larger dataset.
6. Results in Larger ImageNetlk
To validate the effectiveness of our method on larger dataset, we conduct the experiments in ImageNetlk. From Table 2, we can clearly see that our IBFS method consistently outperforms compared SOTA methods: it achieves highest 76.7 % T accuracy, with 0,0042 (GPU-days) fewest search costs. Compared with its peer competitors, our
1.1. Results for Few Shot Learning
Table 3 presents a comparison of IBFS with a wide scope of meta-learning approaches under the setting of 5-way shw- classification tasks in minilmageNet and tiered- ImageNet datasets. Notably, to keep a fair comparison, we
8

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
only compare with CNN-based methods. If we use trans- formers the backbone in FSL., it will bring unfairness for comparison. Among these approaches, MetaDiff represents the current state-of-the-ar gradient-based based meta-learning ap- proach for FSL., which obtains 64.99-+-0.77 (1 shot) and 81.21 0.56 (5 shot) in minilmageNNet dataset using ResNet- 12 as a feature extractor. Additionally, we present a com- parison of IBFS with a wide scope of NAS-based methods (i.e., AutoMeta, MetaNAS, and MetaNTK-NAS) for FSL. In this case, our FSL consistently surpasses nate-of-the-ar methods in terms of accuracy and search costs, showing su- periority in terms of search speed tailored for FSL. To put it more intuitively, we present the visualization of comparison in Fig. 5. The result of Fig. 5 suggests that the architecture guided by IBFS more effectively learns features for FSL, facilitating adaption unseen tasks. Notably, as shown Fig. 7 and 8, we provide the visualization of the searched architecture, which intuitively validates the effectiveness of our method. Overall, these results validate the effectiveness of IBFS tailored for FSL.
Map manee
c_N
sop, odrr______
dl Al
[1-2)
avg poot
Mg L
EPS
Figure 8. Reduce cell searched by IBFS.
Table Results on AutoFormer benchmark in ImageNet.
Mieihed
Vaa
Tpp11111
Swith Cor
Myp Soath Mrihel
Vierg 2016
213
Morrx
AundFonces T (Cher cial, 2021a)
CVPR2221
74.9
24
Truskkus
Evoattis
TI-AA511 (2tea as sl, aiz))
CVP53311
75.3
015
Traning-ms
VTTAS.C 2025
ECCV2062
747
32
Trussoans
Exduttin
Aute-hro (Wes 1024) aBs
AAAU3M
25.
0.1
Traasteme
26.5
0.33
CNN
Iraneng-ioo
7. Conclusion and Future work
This paper proposes a novel IB driven Few-shot Neural Architecture Search (IBFS) framework to address the chal- lenge of designing high-performance neural architectures for new tasks without involving any training. We first demonstrate that the global convergence of Model-agnostic meta- mneta-learnnng (MAML) can be guaranteed by considering only the first-order loss landscape. Additionally, we lever- age the IB theory to develop a training-free method for FSL, enabling to effectively design and rank neural architectures based on IB theory. We conduct extensive experiments to validate the effectiveness of our IBFS framework. The experimental results show that our IBFS achieves state-of- the-art performance in terms of accuracy and search costs in FSL without the need for training.
6.2. Results for Transformer Design
To further scrutinize the effectiveness of our method for transformer designing, we devote lots of effort & explo- ration the IBFS day and night, conducting additional experiments on AutoFormer (Chen et al., 2021a) a larger ImageNet dataset. The experimental setting is the same as TF-TAS-T (Zhou et al., 2022). We can find that IBFS achieves the highest 1 1 accuracy (76.5%). Those empiri- cal results show the strong generalizability of our method for transformer design.
In the future, we plan deploy IBFS on edge devices (i.e.. loT, and Raspberry Pi), which will significantly benefit the NAS community and real-world applications by pro- viding high-quality services in terms of discovering high- performing architectures. Because edge devices possess the characteristics of limited resources, it highlights the need for more efficient and nigh-performing architectures by deploy- ing our IBFS. However, it is challenging to deploy IBFS on edge devices due the limited resources of edge devices. Therefore, we plan to explore ways to reduce the costs of IBFS in terms of storage and search costs. In addition, to en- hance the generalization of our method under cross-domain settings, we will study cross-domain FSL for more com- plicated and challenging, varying-way, varying-shot tasks beyond the current scope.
t-0 644
3.72
30
09
60
(a)
(b)
Figure 6. The impact of 0 on NAS-Bench-201 search space. (a) and (b) are Kendall's 7 on various hyper-parameters a.
c_
3+3
cony. 545
pooi
500, Gotmosee
di 5xg
<_lk-2)
di alecom.aa Sw5
Impact Statement
0_04
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
smp
Figure? 7. Normal cell scarched by IBFS.

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
References
Chen, X., Xie, L.,, Wu, J., and Tian, Q. Progressive dif- ferentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1294-1303, 2019.
Abdelfattah, M. S., Mehrotra, A., Dudziak, L., and Lane, N. D. Zero-cost proxies for lightweight nas. arXiv preprint arXiv:2101.081:4 2021.
Antoniou, A., Edwards, H., and Storkey, A. How to train your maml. arXiv preprint arxiv: 1810.09502, 2018.
Chen, Y., Liu, Z., Xu, H., Darrell, T., and Wang, X. Meta- baseline: Exploring simple meta-learning for few-shot learning, In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9062-9071, 2021d.
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, R. On exact computation with an infinitely wide neural net. Advances in neural information process- ing systems, 32, 2019.
Chrabaszez, P., Loshchilov, I., and Hutter, F. downsam- pled variant of imagenet datasets. arXiv preprint 1707.88777.88111 2017.
alternative to the cifar
Bai, Y., Chen, M., Zhou, P. Zhao, T., Lee, J., Kakade, S., Wang, H., and Xiong, C. How important is the train- validation split in meta-learning? In International Confer- ence on Machine Learning, pp. 543-553, PMLR, 2021.
Chu, X., Wang, X., Zhang, B., Lu, S., Wei, X., and Yan, J. Darts-: robustly stepping out of performance collapse without indicators. ArXiv, abs/2009.0100
Baik, S., Choi, M., Choi, J., Kim, H., and Lee, K. M. Meta- learning with adaptive hyperparameters. Advances in neural information processing systems, 33:20755-20765, 2020.
2020a.
Chu, X., Zhou, T., Zhang, B., and Li, J. Fair darts: Eliminat- ing unfair advantages in differentiable architecture search. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XV, pp. 465-480 Springer, 2020b.
Baik, S., Choi, J., Kim, H., Cho, D., Min, J., and Lee, K. M. Meta-learning with task-adaptive loss function for few-shot learning. Proceedings of the IEEE/CVF international conference on computer vision, pp. 9465- 9474, 2021.
Deleu, T., Kanaa, D., Feng. L., Kerg, G., Bengio, Y., Lajoie, G., and Bacon, P.-I.. Continuous-time meta-learning with forward mode differentiation. In International Confer- ence Learning Representations, 2022.
Baker, B., Gupta, O., Naik, N., and Raskar, R. Designing neural network architectures using reinforcement learn- ing, arXiv preprint arXiv: 16X1::1611.02167 2016.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei- Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
Cai, H., Zhu, L., and Han, S. Proxylessnas: Direct neural architecture search on target task and hardware. ArXiv, abs/1812.00332, 2018.
Dong, X. and Yang. Y. Searching for robust neural architecture in four gpu hours. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1761-1770, 2019.
Cha, S., Hsu, H., Hwang, T., Calmon, F. P., and Moon, T. Cpr: classifier-projection regularization for continual learning. arXiv preprint arXiv:2006.07326 2020.
Chayti, E. M. and Jaggi, M. A new first-order meta-learning algorithm with convergence guarantees. arXiv preprint arXiv:2409.03688 2024.
Dong, X. and Yang. Y. Nas-bench-201. Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020. URL hutps//oppnreeiew.neefommmm"""" HJxyZkBKDr.
Chen, M., Peng, H., Fu, J., and Ling, H. Autoformer: Searching transformers for visual recognition. In Interna- tional Conference on Computer Vision, 2021a.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Chen, W., Gong, X., and Wang. Z. Neural architecture search on imagenet in four {gpu} hours: A theoret- ically inspired perspective. In International Confer- ence on Learning Representations, 2021b. URL https: Vlopenreview.net/frmee hatpreeeeee...rroooomooooooaaaaa Cid-ConnSeecnHittt
Chen, W., Gong, X., and Wang, Z. Neural architecture search on imagenet in four gpu hours: theoretically inspired perspective. arXiv preprint arXiv:2102.11533, 2021c.
Elsken, T., Staffler, B., Metzen, H., and Hutter, F. Meta- learning of neural architectures for few-shot learning. In CVPR, 2020.
10

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Fallah, A., Mokhtari, A., and Ozdaglar, A. On the con- vergence theory of gradient-based model-agnostic meta- learning algorithms. In International Conference on Arti- ficial Intelligence and Statistics, pp. 1082-1092. PMLR, 2020a.
Ji, K., Yang, J., and Liang, Y. Multi-step model-agnostic meta-learning: Convergence and improved algorithms. arXiv preprint urXiv:2002.07883 2020.
Ji, K., Yang. J., and Liang, Y. Theoretical convergence of multi-step model-agnostic meta- weeta-lerrning. The Journal of Machine Learning Research, 23(1):1317-133 2022.
Fallah, A., Mokhtari, A., and Ozdaglar, A. On the con- vergence theory of gradient-based model-agnostic meta- learning algorithms. In International Conference on Arti-
Jiang, S., Ji, Z.,, Zhu, G., Yuan, C., and Huang. Y. Operation- level early stopping for robustifying differentiable nas. Advances in Neural Information Processing Systems, 36, 2023.
ficial Intelligence and Statistics, pp. 1082-1092, 2020b.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In Proceed- ings of the 34th International Conference on Machine Learning- Searning-Vollmee Volume 70, pp. 1126-1135. JMLR. org, 2017.
Jing, K., Chen, L., and Xu, J. An architecture entropy regularizer for differentiable neural architecture search. Neural Networks, 158:1 11-120, 2023.
Finn, C.,, Rajeswaran, A., Kakade, S., and Levine, S. Online meta-learning, In International Conference on Machine Learning, pp. 1920-1930, 2019.
Kang, S., Hwang, D., Eo, M., Kim, T., and Rhee, W. Meta- learning with a geometry-adaptive preconditioner. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16080-16090, 2023.
Gu, Y.-C., Wang, L.-J., Liu, Y., Yang, Y. Wu, Y.H.,, Lu, S.-P., and Cheng, M.-M. Dots: Decoupling operation and topology in differentiable architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12311-12320, 2021.
Kawaguchi, K., Deng, Z., Ji, X., and Huang, J. How does information bottleneck help deep learning? In Inter- national Conference on Machine Learning, pp. 16049- 16096. PMLR, 2023.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, T. P. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: /609: 04836, 2016.
He, H., Liu, L., Zhang, H., and Zheng, N. Is-darts: Stabi- lizing darts through precise measurement on candidate importance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12367-12375, 2024.
Kim, J., Lee, S., Kim, S., Cha, M., Lee, K., Choi, Y., Choi, Y., Cho. D.-Y., and Kim, J. Auto-meta: Auto- mated gradient based meta learner search. arXiv preprint arXiv: TrXiv:180600692. 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Lee, J. and Ham, B. Az-nas: Assembling zero-cost proxies for network architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5893-5903, 2024.
Hoffer, E., Hubara, I., and Soudry, D. Train longer, general- ize better: closing the generalization gap in large batch training of neural networks. Advances in neural informa- tion processing systems, 30, 2017.
Lee, K., Maji, S., Ravichandran, A., and Soatto. S. Meta- learning with differentiable convex optimization. In CVPR, 2019.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 1704vv17004048661 2017.
Li, G., Yang, Y., Bhardwaj. K., and Marculescu, R. Zico: Zero-shot nas via inverse coefficient of variation on gra- dients. arXiv preprint 2301::2301.113000 2023.
Hu, S., Xie, S., Zheng, H., Liu, C.,, Shi, J., Liu, X., and Lin, D. Dsnas: Direct neural architecture search without parameter retraining. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 12081-12089, 2020.
Li, H., Eigen, D., Dodge, S., Zeiler, M., and Wang, X. Finding task-relevant features for few-shot learning by category traversal. In Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 1-10, 2019.
Hu, S., Lou, Z., Yan, X., and Ye, Y. survey on information bottleneck. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Li, L. and Talwalkar, A. Random search and reproducibility for neural architecture search. In Uncertainty artificial intelligence, pp. 367-377. PMLR, 2020.

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Lian, D., Zheng, Y., Xu, Y., Lu, Y., Lin, L., Zhao, P., Huang, J., and Gao, S. Towards fast adaptation of neu- ral architectures with meta learning. In International Conference on Learning Representations, 2020. URL
Peng, Y., Song, A., Fayek, H. M., Ciesielski, V., and Chang, X. Swap-nas: Sample-wise activation patterns for ultra- fast nas, arXiv preprint X11::2403.0411.1 2024.
Pham, H., Guan, M., Zoph, B., Le, Q., and Dean, J. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pp. 4095- 4104. PMLR, 2018.
Liang, H., Zhang, S., Sun, J., He, X., Huang, W., Zhuang, K., and Li, 7. Darts+: Improved differentiable architecture search with early stopping. ArXiv, hts/19090060355 2019.
Raghu, A., Raghu, M., Bengio, S., and Vinyals, O. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2020.
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li,
L.-J., Fei-Fei, L., Yuille, A., Huang. J., and Murphy, K. Progressive neural architecture search. In Proceedings of the European Conference Computer Vision (ECCV), pp. 19-34,22 2018.
Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, pp. 113-124, 2019.
Liu, H., Simonyan, K., and Yang. Y. DARTS: Differ- entiable architecture search. In International Confer- ence on Learning Representations, 2019a. URL https:
Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Aging evolution for image classifier architecture search, In AAAI Conference on Artificial Intelligence, volume 2, 2019a.
Liu, H., Simonyan, K., and Yang, Y. DARTS: Differentiable architecture search. 2019b.
Luo, R., Tian, F., Qin, T., Chen, E., and Liu, T.-.. Neural ar- chitecture optimization. Advances in Neural Information Processing Systems, 31, 2018.
Real, E., Aggarwal, A., Huang, Y., and Le. Q. V. Regular- ized evolution for image classifier architecture search. In Proceedings of the auai conference artificial intelli- gence, volume 33, pp. 4780-4789, 2019b.
Luo, R., Tan, X., Wang, R., Qin, T., Chen, E.,, and Liu, T.-Y. superviue neural architecture search. Advances in Neural Information Processing Systems, 33:10547- 10557, 2020.
Ren, M., Ravi, S. Triantafillou, E., Snell, J., Swersky, K., Tenenbaum, J. B., Larochelle, H., and Zemel, R. S. Meta- learning for semi-supervised few-shot classification. In International Conference on Learning Representations, 2018. URL http:://ppaaeevvee...errooouuddd
Ma, N., Zhang, X., Zheng, H.-T., and Sun, Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4510- 4520, 2018.
Mellor, J., Turner, J., Storkey, A., and Crowley, E. J. Neural architecture search without training. In International Con- ference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Sen, P. K. Estimates of the regression coefficient based kendall's tau. Journal of the American statistical association, (3(322)::339--338) 1968.
Movahedi, S., Adabinejad, M., Imani, A., Keshavarz, A., Dehghani, M., Shakery, A., and Araabi, B. N. X-darts: Mitigating performance collapse by harmonizing opera- tion selection among cells. In The Eleventh International Conference Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023, 2023.
Shamir, O., Sabato, S., and Tishby, N. Learning and gen- eralization with the information bottleneck. Theoretical Computer Science, 41 411229-30):296-2211 2010.
Nichol, A., Achiam, J., and Schulman, J. On first-order meta-learning algorithms. arXiv preprint arXiv: 1803.0090 2018.
Shu, Y., Cai, S., Dai, Z., Ooi, B. C., and Low, B. K. H. NASI: Label- and data-agnostic neural architecture search at initialization. In International Conference on Learning Representations, 2022.
Paszke, A., Gross, S., Massa, F. Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32:8026 22800268037, 2019.
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv: Learning, 2017a.
12

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint 17Xiv:1703.00810, 2017b.
for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9797-9808, 2022b.
Slonim, N. and Tishby, N. Agglomerative information bot- tleneck. Neural Information Processing Systems, 1999.
Wang. R., Cheng, M., Chen, X., Tang, X., and Hsieh, C.-J. Rethinking architecture selection in differentiable nas. In International Conference on Learning Representations, 2020.
Su, X., You, S., Xie, J., Zheng, M., Wang, F,, Qian, C., Zhang, C., Wang, X., and Xu, C. Vision transformer ar- chitecture search. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXI, pp. 139-157, 2022.
Wang, W., Zhang, X., Cui, H., Yin, H., and Zhang, Y. Fp-darts: Fast parallel differentiable neural architecture search for image classification. Puttern Recognition, 136: 109193, 2023.
Sun, S. and Gao, H. Meta-adam: An meta-learned adaptive optimizer with momentum for few-shot learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wei, Z.., Dong, P., Hui, Z., Li, Li, Lu, M., Pan, H., and Li, D. Auto-prox: Training-fres frre vision transformer architecture search via automatic proxy discovery. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 38, pp. 15814-15822, 2024.
Tack, J., Park, J., Lee, H., Lee, J., and Shin, J. Meta-learning with self- self-improving momentum target. Advances in Neural Information Processing Systems, 35:6318-633 2022.
Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K. Fbnet: Hardware- aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Tan, M. and Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Con- ference on Machine Learning, pp. 6105-6114. PMLR, 2019.
Wu, J., Dai, X., Chen, D., Chen, Y., Liu, M., Yu, Y., Wang, Z., Liu, Z., Chen, M., and Yuan, L. Stronger nas with weaker predictors. Advances in Neural Information Pro- cessing Systems, 34:28904-28916 2021.
Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J. B., and Isola, P. Rethinking few-shot image classification: good embedding is all you need? ECCV, 2020.
Tishby, N. and Zaslavsky, N. Deep learning and the infor- mation bottleneck principle. In 2015 ieee information theory workshop (inw), pp. 1-5. IEEE, 2015.
Xie, S., Zheng, H., Liu, C., and Lin, L. Snas: stochastie neu- ral architecture search. arXiv preprint ar arXiv: 182.0998. 2018.
Tishby, N., Pereira, F,, and Bialek, W. The information bottleneck method. 2000.
Xie, S., Zheng, H., Liu, C., and Lin, L. Snas: Stochastic neural architecture search. ArXiv, abs/1812.09922 2020.
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Xu, J., Zhao, L., Lin, J., Gao, R., Sun, X., and Yang, H. Knas: green neural architecture search. In Inter- national Conference on Machine Learning, pp. 11613 11625. PMLR, 2021.
Von Oswald, J., Zhao, D., Kobayashi, S., Schug. S., Caccia, M.. Zucchet, N., and Sacramento, J. Learning where to learn: Gradient sparsity in meta and continual learning. Advances in Neural Information Processing Systems, 34: 5250-5263, 2021.
Xu, R., Chen, L., and Karbasi, A. Meta learning in the continuous time limit, arXiv preprint XXXiv:2006.1092 2020.
Xu, Y., Xie, I.., Zhang, X., Chen, X., Qi, G.-J,, Tian, Q. and Xiong, H. Pe-darts: Partial channel connec- tions for memory-efficiert architecture search. ArXiv, abs/190 0bs/1907.wg 907.05737, 2019.
Wang, H., Wang, Y., Sun, R., and Li, B. Global convergence of maml and theory-inspired neural architecture search for few-shot learning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA,jun 2022a.
Ye, P., Li, B., Li, Y., Chen, T., Fan, J., and Ouyang, W. Vbeta-darts: Beta-decay regularization for differentiable architecture search. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10864-10873. IEFE, 2022.
Wang, H., Wang, Y., Sun, R., and Li, B. Global convergence of maml and theory-inspired neural architecture search
13

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Yu, K., Sciuto, C., Jaggi, M., Musat, C., and Salzmann, M. Evaluating the search phase of neural architecture search. In International Conference on Learning Representations, 2019.
Yu, S., Wickstrøm, K., Jenssen, R., and Principe, C. Un- derstanding convolutional neural networks with informa- tion theory: An initial exploration. IEEE transactions on neural networks and learning systems, 32(1))433-44 442, 2020.
Zhang, B., Luo, C., Yu, D., Li, X., Lin, H., Ye, Y., and Zhang, B. Metadiff: Meta-learning with conditional diffusion for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16687-16695, 2024.
Zhang. M., Su, S. W., Pan, S., Chang, X., Abbasnejad, E. M., and Haffari, R. idarts:
Differentiable architecture search with stochastic implicit gradients. In International Con- ference on Machine Learning, pp. 1557-12566, a0.
Zhang, X., Meng, D., Gouk, H., and Hospedales, T. M. Shal- low bayesian meta learning for real-world few-shot recog- nition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 651-660, 2021b.
Zhang, Z. and Jia, Z. Gradsign: Model performance infer- ence with theoretical insights. In International Confer- ence on Learning Representations, 2022.
Zhao, Y., Gao, X., Shumailov, I., Fusi, N., and Mullins, D. Rapid model architecture adaption for meta-learning In Advances in Neural Information Processing Systems, 2022.
Zhou, P., Yuan, X., Xu, H., and Yan, S. Efficient meta learn- ing via minibatch proximal update. Neural Information Processing Systems, 2019.
Zhou, Q., Sheng, K., Zheng, X., Li, K., Sun, X., Tian, Y., Chen, J., and Ji, R. Training-free ffre transformer architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10894- 10903, 2022.
Zoph, B. and Le, Q. Neural architecture search with reinforcement learning. ICLR, 2017.
Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697-8710, 2018.
14

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Overview of the Appendix
The overview of this appendix as follows:
App. A: Describes detailed proof of global convergence of MAML.
A. Proof of Global Convergence of MAML
In this section, we will prove the global convergence of MAML.. First, we rethink the process MAMI. under the FSL setting, then, introduce some variables used in the proof. Finally, will introduce proof of global convergence of MAML.
Rethink the Process of MAMI. From the FSL definition, we have set of tasks (T') where each task T consists of b' data samples. These are split into support (training) samples (X (Y) E Ditai and query (test) samples (X', ,Y) E D The loss function for FSL defined as:
m
min
C(D G(Dtaiin o; A)),
(11)
where A is the neural architecture, W are the task task-specific parameters, and G represents parameter updates via gradient descent. From the NAS definition, the search process is Directed Acyclic Graph (DAG), and the objective is:
min L NE (W*, .x), s.t. W = arg min Lrain eTr (W, a))
(12)
W
where W are the weights of the architecture, a are the mixing weights of operations, and W the optimal weight. the paper, MAML is formulated as a bi-level optimization problem:
M
0" = arg min F(0) == Σ OOO), BEB
1
F(()) - ((), (0) = Alg(S,,
(13)
M
where f is the training loss, is the test loss, and Alg is optimization algorithm (e.g., gradient descent) that adapts 0 to task-speeific parameters (0).
Definition of Variable. In this section, we provide the detailed definition of variables in our paper. The details are as follows:
Fí: This represents the loss for task T', sampled from search space Based on the FSL definition, F,(xt, x Y*) is the test loss (0)), where (0) is the adapted parameter for task T
((W) e("', Y 11 ÿ F((X", xt, Xt,'t) (using adapted parameters) and the true output yf. Here, W corresponds to the adapted parameters (0).
The loss is the squared error between the predicted output
YY):)): This is the gradient the task-specific loss with respect to the meta-parameters 0, Cinerr = (X 1o,,,,X,,' computed on the support set.
Ir -o Linner This appears to define a limiting behavior of the inner gradient's 's covariance the width / (number of hidden units in layer i) increases. It's likely intended to capture the spectral properties of the gradient.
Φ Euss (ea Here, Eaux and Emi are the maximum and minimum eigenvalues of foo, suggesting Φo is related to the learning rate or a stability constant.
7o A constant related to the learning rate.
m (b): The minimum singular value of matrix Q likely related to the Jacobian of the network's 's output with respect to its parameters.
15

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
t: The width of the i-th hidden layer in the neural network.
Ir: The learning rate for gradient descent.
À: A regularization parameter.
Detailed Proof. To make the proof clear, in this paper, we provide some assumptions follows. These assumptions can also be found in previous works (Arora et al., 2019).
Assumption (Smoothness and Strong Convexity.) Assume the task-specific training loss ft is Lı-smooth and the inner objective (o) + 1 011 is j-strongly convex (Assumption 3 in the paper). The test loss St is -smoothh
Assumption (Bounded Gradients.) The meta meta-gradient gradient FF()) is bounded (Corollary 3 in the paper).
Assumption 3 (Spectral Properties.) The matrix & represents the Jacobian of the network output with respect to 0, and Omin ( > 0, ensuring the network is expressive enough.
Assumption 4 (Learning Rate and Width.) The learning rate Ir < Lipe, and the width I 1", ensuring the network is sufficiently wide to approximate the limiting behavior.
Theorem A.1 (Theorem 4.1 restated) a neural network with width 2 1", performing gradient descent with learning rate Ir tr and regularization X < 2. the loss E(WY) 2 1 X X Y²) 1/ satisfies:
(WW)) S (1-T min (4)) (4))²
T (0,
(14)
where 710 > 0, and (D) is the minimum singular value of the Jacobian matrix Φ.
First, we will define the loss and update rule of MAML under FSL settings. The loss for task T is:
E(W') _ 1 X", Y²) - -
(15)
where Fi(X'XX' -xxYtt is the predicted output using the adapted parameters V (00)), and Y is the predicted output on the test set. The meta-objective is:
1 M M
F(0)=
Σ fl()(()), Φ.(0) 0 avo((()).
(16)
After that, the meta-update at iteration t is:
e = -
(17)
where ir is the outer learning rate. For each task T the inner loop performs one step of gradient descent:
(0') = 0² (VV
(18)
Since fi is -mooth, the inner gradient (e) is Lipschitz, and the adaptation step reduces the training loss. Then, we introduce the meta-gradient and smoothness in our paper. To be specific, the meta-gradient is defined as VF =
M ΣM v (0)) (o)( According to the previous work (Chayti & Jaggi, 2024), can obtain generalized
smoothness condition as:
[VF(0) VF(0')l min(C(0), ((0')) 0'|],
(19)
where (())
C. + VF)(() Under strong convexity (Assumption 3), FF()) is bounded (Corollary 3), so F
L-smooth in the classical sense:
L L G= XLo
(20)
16

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Since F is C-smooth, the descent lemma can be written
s = VF(0)) -0" - -10 0011
(21)
2
Substituting the update 0²+1 0¹ -+VF((O)):
F(0 = F(0") - 1= + f 1
(22)
If Ir < 2. then:
) =F(0") - Ir (11- -ir Clr
2 |FF(0')11
(23)
When Ir 1. Z. we can obtain:
FF((") =( 2C |VF(('))
(24)
The task loss ((W") (e') is a component of F(e'). Since ft is Lm-smooth, the gradient (f(( relates to the loss via the network's Jacobian OF: The minimum singular value min () ensures that:
80.
Ve((): 2 min ( 2f(
(25)
Thus, f(())(())) v
( 2d @0)2
and since F averages over tasks, we approximate:
M [T
VF
o (2)) (f((
(26)
M
According to the above equation, the decrease in J' implies a decrease in the average task loss. For a single task:
1 [VF(()((0')11 min 10)2-2f((dr(0)
|FFO')
(27)
M
When set Y/o- 2, we can obtain:
(1- mo (2)²
f <
fe fe():(0)))
(28)
2
After iterations, we can obtain:
f(())(0))) < <11-200 min ( (8)))
fe(
(29)
2
Bounding the Initial Loss E(WO).
Note that:
)(W))= (e()) ((())))()))))))) } F
x X2, Y9) - 1'1
where (e( = g - foV)(( is the one-step adapted model.
Under Assumption (bounded gradient norm) and assuming Lipschitz continuity of F, with constant Lr, we obtain:
(X x². yt) 8/112 < T (19) 0- + [IF.(0)) 8
17

---

Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
where 0* is the task-optimal parameter. Then,
e(wo) & (L) - 0-1 + +))²
where e is the irreducible error from the Bayes optimal predictor.
Because (,()) = 8 V. and the gradient is bounded by G, we have:
(l - 00 ll2 S -100 0- + aG.
Therefore, the upper bound becomes:
((W) 1 (Le(1 - 02 + aG) + +c)²
If needed, a lower bound can also be derived assuming some minimum deviation from optimality due to finite adaptation:
e(W0) 2 2 (Lrōmin)²
where 6mnn = inf, [l -0 across tasks.
Final Task Loss Bound. Putting this together, we have:
((W') S (1 mi (o)) e(W").
where E(WO) is bounded as shown above. Therefore, Theorem 4.1 is proved.
18


Reviews:
[{"review_id": "o5yCgRe5XH", "reviewer": "Reviewer_DRyn", "summary": "The paper introduces IBFS (Information Bottleneck-driven Few-shot Neural Architecture Search), a novel framework designed to efficiently select neural architectures for few-shot learning (FSL) without requiring any training. Traditional NAS approaches either search architectures from scratch—resulting in high computational costs—or transfer architectures from other tasks—potentially leading to suboptimal performance. IBFS addresses these limitations by leveraging Information Bottleneck (IB) theory and a Zero-Cost evaluation method.\n\nMain Contributions:\n1.  The paper derives that the global convergence of Model-Agnostic Meta-Learning (MAML) can be ensured by considering only the first-order loss landscape.\n\n2. Information bottleneck provides a unified perspective on understanding machine learning models.\nThe proposed Zero-Cost expressivity ranking method estimates an architecture's effectiveness without training, significantly reducing search costs.\n\n3. IBFS achieves state-of-the-art results in FSL without requiring training, validating its effectiveness.\n\n\nOverall, the paper presents a theoretically grounded and empirically validated approach to optimizing neural architectures for FSL, offering a cost-effective alternative to conventional NAS techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful and insightful review, which is very helpful for us to further improve this paper. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**Q1**: Concern about MAML.\n\n**A1**: Many thanks for your comments! Our paper follows the MetaDiff (AAAI24), MetaNTK-NAS (CVPR22), which only provide 5-1 and 5-5 settings. 5-20 and 5-50 settings are indeed promising, however, the previous method did not conduct the experiments on 5-20 and 5-50 settings. We worry that it is unfair for comparison.\n\n**Q2**: Lack of newer works.\n\n **A2**: Many thanks for your comments! We devote lots of effort & exploration in the IBFS day and night, conducting experiments compared with newer works. The results demonstrate that our method still achieves the best performance. ``-\" indicates that the value is not found in the original paper or the training codes are not provided.\n\n|                             | Year     | Cost(s) | CIFAR10 (val)    | CIFAR10(test)          | CIFAR-100(val)   | CIFAR-100(test)  | ImgeNet(val)     | ImgeNet(test)    |\n| --------------------------- | -------- | ------- | ------------ | ------------------ | ------------ | ------------ | ------------ | ------------ |\n| SNAS [1]                    | ICLR2018 | -       | 90.10±1.04   | 92.77±0.83         | 69.69±2.39   | 69.34±1.98   | 42.84±1.79   | 43.16±2.64   |\n| DSNAS [2]                   | ICLR2019 | -       | 89.66±0.29   | 93.08±0.13         | 30.87±16.40  | 31.01±16.38  | 40.61±0.09   | 41.07±0.09   |\n| DARTS- [3]                  | ICLR2020 | 192     | 91.03±0.44   | 93.80±0.40         | 71.36±1.51   | 71.53±1.51   | 44.87±1.46   | 45.12±0.82   |\n| PC-DARTS [4]                | ICLR2019 | 8.70    | 89.96±0.15   | 93.41±0.30         | 67.12±0.39   | 67.48±0.89   | 40.83±0.08   | 41.31±0.22   |\n| iDARTS [5]                  | ICML2021 | -       | 89.86±0.60   | 93.58±0.32         | 70.57±0.24   | 70.83±0.48   | 40.38±0.59   | 40.89±0.68   |\n| GradSign [6]                | ICLR2022 | 30.38   | -            | 93.52 ± 0.19       | -            | 70.57 ± 0.31 | -            | 41.89 ± 0.69 |\n| ZiCo [7]                    | ICLR2023 | 6.2     | 93.50 ± 0.18 | -                  | 70.62 ± 0.26 | -            | 42.04 ± 0.82 | -            |\n| IS-DARTS [8]                | AAAI2024 | 7200    | 91.55 ± 0.00 | 94.36 ± 0.00       | 73.49 ± 0.00 | 73.51 ± 0.00 | 46.37 ± 0.00 | 46.34 ± 0.00 |\n| AZ-NAS [9]                  | CVPR2024 | 0.71    | -            | 93.53 ± 0.15 0.723 | -            | 70.75 ± 0.48 | -            | 45.43 ± 0.29 |\n| SWAP [10]                   | ICLR2024 | 4.7     | 87.31 ± 2.36 | 90.48 ± 0.94       | 65.92 ± 4.32 | 67.13 ± 1.83 | 33.85 ± 4.98 | 35.40 ± 3.96 |\n| IBFS(ours) ( $\\theta$=0)    |          | 3.36    | 89.58 ± 0.57 | 92.96 ± 0.81       | 69.17 ± 1.81 | 68.94 ± 1.41 | 41.30 ± 1.79 | 41.11 ± 1.51 |\n| IBFS(ours) ( $\\theta$=0.75) |          | 3.82    | 91.55 ± 0.76 | 94.37 ± 0.34       | 73.31 ± 2.12 | 73.09 ± 2.08 | 45.59 ± 0.32 | 46.33 ± 1.27 |\n\n**Q3**: Concern about Figure 4.\n\n **A3**: Many thanks for your comments! Kendall’s correlation coefficient requires two lists, each containing multiple elements, to be computed. For a given epoch, we first calculate the proxy scores for the models DenseNet-40, SE-ResNet-20, ResNet-56, PyramidNet-110, and WRN-16. These scores form one list, while the corresponding true accuracy values at the same epoch form another. We then use Kendall’s coefficient to measure the correlation between these two lists. As our analysis focuses on the correlation between proxy scores and true accuracy at each epoch, it is naturally represented by a single line segment rather than multiple curves.\n\n[1]  Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926, 2018\n\n[2] Dsnas: Direct neural architecture search without parameter retraining. In CVPR 2020\n\n[3]  Darts-: robustly stepping out of performance collapse without indicators. arXiv preprint arXiv:2009.01027, 2020.\n\n[4]  “PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search.” In ICLR 2019\n\n[5]  idarts: Differentiable architecture search with stochastic implicit gradients. arXiv preprint arXiv:2106.10784, 2021\n\n[6]  GradSign: Model performance inference with theoretical insights. In ICLR2022\n\n[7]  ZiCo: Zero-shot NAS via inverse coefficient of variation on gradients. In ICLR2023\n\n[8] \"IS-DARTS: stabilizing DARTS through precise measurement on candidate importance.\" In AAAI 2024.\n\n[9] \"Az-nas: Assembling zero-cost proxies for network architecture search.\" In CVPR 2024.\n\n[10]  \"SWAP-NAS: Sample-wise activation patterns for ultra-fast NAS.\" *arXiv preprint arXiv:2403.04161* (2024).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow, making complex concepts accessible to the reader.\n\n2. It introduces a novel zero-cost NAS framework specifically designed for few-shot learning (FSL), which effectively selects optimal architectures without training, significantly reducing computational overhead.\n\n3. The proposed IBFS framework achieves state-of-the-art performance in FSL tasks without requiring any training, demonstrating both the efficiency and effectiveness of its architecture selection strategy.\n\nWeaknesses:\n\n1. The experimental evaluation is somewhat limited, as it is conducted solely on MAML, an approach that is now considered outdated. Additionally, the FSL evaluation is restricted to 5-1 and 5-5 settings, whereas comparisons with 5-20 and 5-50 settings are typically necessary for a more comprehensive assessment.\n\n2. The choice of baselines is not sufficiently up-to-date, as it lacks comparisons with recent NAS methods from the past two years, which may impact the fairness and relevance of the evaluation.\n\n3. There is a discrepancy in Fig. 4—according to the annotation and the \"Remark\" section, the figure should contain multiple curves, but in its current form, only one curve is presented, which may lead to confusion or misinterpretation of the results.", "other_comments_or_suggestions": "After rebuttal, the authors have addressed all concerns. I will keep my rating.", "questions_for_authors": "See weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ehF1DQQsYG", "reviewer": "Reviewer_masR", "summary": "This paper mainly considers the case that NAS is applied in few-shot learning scenarios, where previous works mainly search for the optimal architecture from scratch or borrow the architecture from other tasks. The paper presents a novel framework called IBFS (Information Bottleneck-driven Few-shot Neural Architecture Search) that addresses two key limitations in conventional Neural Architecture Search (NAS) for few-shot learning scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful and insightful review, which is very helpful for us to further improve this paper. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**Q1**: Concern about optimal architecture for few-shot learning tasks.\n\n**A1**: Many thanks for your comments! As mentioned in Section 1, we claim a statement of whether the frameworks of the DNNs are ideal for those tasks.  This is because that popular networks (i.e., ResNet) are developed on supervised learning, it overfit to those task. This evidence indicates that the popular networks may not be optimal for tasks beyond supervised learning, e.g., few-shot learning (FSL) task. This statement can be supported with MetaNTK-NAS (CVPR2022). Therefore, it is worth and necessary to explore the frameworks of the DNNs tailored for FSL. \n\nRegarding computation cost, traditional methods (e.g., AutoMeta, MetaNAS) are expensive since each architecture requires full training. In contrast, our training-free proxy achieves top performance in ≤0.1hr on miniImageNet and tieredImageNet, demonstrating exceptional efficiency.\n\nFor generality, our searched architecture not only excels in FSL but also achieves optimal performance on NAS-Bench-201 (CIFAR-10, CIFAR-100, ImageNet-16-120). Additional results on ImageNet1k (Appendix B) further confirm strong generalization, reaching 76.6% Top-1 accuracy with the lowest 0.0042 GPU-days search cost.\n\nComapred with Model Parameter Adaptation (i.e., MAML), NAS guided FSL possesses the following advantages: (1) higher accuracy; (2) reduced human labor; (3) breaking limitations of parameter optimization of fixed network (ResNet-12); (4) NAS can design network for any setting of FSL.\n\n**Q2**: Concern about Fig 1. \n\n**A2**: Many thanks for your comments! Fig. 1 is a schematic diagram, visualizing the comparison of different methods in terms of cost, generalization, and accuracy. The data is derived from Table 2. For example, in the cost dimension, AutoMeta is at the top, indicating the highest computational cost. The continuous curves are used to better visualize performance trends across different algorithms, not real data.\n\n**Q3**: Explanations about Eq. (6-7). \n\n**A3**: Many thanks for your comments! Eq. 6 introduces the Lagrange multiplier $\\lambda(x)$ to ensure normalization and utilizes the upper bound on mutual information to simplify the optimization problem. Eq. 7 shows the process of obtaining optimal $p(r|x)$ by taking the derivative and setting it to zero.\n\n**Q4**: The relationship of IB and architecture selection. \n\n**A4**: Many thanks for your comments! In this work, we use the Information Bottleneck (IB) theory to measure the expressivity of neural networks. A stronger neural network maintains more feature information from the input $x$, resulting in a higher $NN_{expressivity}$. During the NAS search process, architectures that preserve more input feature information are more likely to be selected.\n\n**Q5**: The optimal way to balance between expressivity and computational efficiency. \n\n**A5**: Many thanks for your comments! To clearly answer this question, let’s first assume a neural network **NetA**. When fully trained in supervised learning, **NetA** achieves maximum expressivity but also incurs the highest computational cost. To reduce computational cost, one approach is to decrease the number of training iterations, but this does not guarantee an accurate measure of expressivity. Instead, our method takes a different approach: rather than training NetA, we design a proxy that accurately measures its expressivity based solely on its architecture. This allows us to maximize the balance between expressivity and computational efficiency. While this approach may not be strictly optimal, it is the closest to the optimal solution, as demonstrated by our experimental results.\n\n**Q6**: Dicussusion about cross-domain.\n\n**A6**: Many thanks for your comments! First, our paper follows the MetaDiff (AAAI24), MetaNTK-NAS (CVPR22), which only provide 5way-1shot and 5way-5shot settings of few-shot learning tasks. Therefore, we believe our empirical evaluation is sufficient, and not beyond the scope of ICML 2025. Second, we greatly appreciate **masR**'s insights on cross-domain learning, which reinforce our belief that he is an outstanding expert in this field. His perspective provides valuable inspiration for our future research on cross-domain FSL. cross-domain FSL (e.g., Meta-Dataset) presents several challenges, such as differences between training and testing domains and limited data availability. The key challenge in cross-domain learning is extracting **domain-invariant** features. For example, **fo-Proto-MAML** leverages second-order optimization in MAML to enhance cross-domain FSL. Since our method is only related to MAML’s second-order optimization and is independent of the image domain, we believe our approach can generalize well to cross-domain FSL.", "claims_and_evidence": "In this paper, the claims mainly include the following several aspects:\n1. **Zero-Cost Architecture Selection**: The claim that architectures can be selected without training. Such claim is supported by: (1) theoretical analysis of MAML's convergence properties; (2) empirical validation showing correlation between zero-cost proxies and actual test accuracy; (3) Experimental results demonstrating state-of-the-art performance\n\n2. **Information Bottleneck Theory Application**: The claim that IB theory provides a unified view for understanding machine learning models. Such claim is supported by: (1) analysis of information entropy variations across different architectures; (2) consistent Kendall's Tau correlation between accuracy and information entropy; (3) theoretical framework connecting IB principles to architecture selection", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are sound and well-structured:\n\n1. **Theoretical Framework**:\n   - Clear derivation of MAML convergence properties;\n   - Well-motivated connection to Information Bottleneck theory;\n   - Logical progression from theoretical insights to practical implementation\n\n2. **Evaluation Approach**:\n   - Comprehensive comparison with existing methods;\n   - Multiple evaluation metrics (costs, accuracy, generalization);\n   - Validation across different architectures and datasets", "theoretical_claims": "The paper's theoretical contributions are sound:\n\n1. **MAML Convergence Analysis**:\n   - Theorem 4.1 provides a rigorous foundation for the approach;\n   - The connection to first-order loss landscape is well-established;\n   - The theoretical framework supports the practical implementation\n\n2. **Information Bottleneck Integration**:\n   - Clear connection between IB theory and architecture selection;\n   - Well-motivated use of information entropy as a proxy;\n   - Theoretical justification for the proposed metrics", "experimental_designs_or_analyses": "The experimental design is comprehensive and well-executed:\n\n1. **Results Analysis**:\n   - Thorough analysis of zero-cost proxies vs. accuracy;\n   - Clear visualization of results (Figures 1-4);\n   - Statistical validation of findings\n\n2. **Ablation Studies**:\n   - Analysis of different proxy metrics;\n   - Investigation of architecture variations;\n   - Validation of key components", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Pros:\n- The paper is well written.\n- The paper is based on theoretical results, which makes the paper solid.\n- The empirical results are good.\n\nCons:\n- The motivation of the paper is not convincing enough.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- As mentioned in the introduction, a concern raised in FSL is whether the model is ideal for those tasks. However, is it necessary for us to search for an optimal specific architecture for few-shot learning tasks? To some extent, the obtained architecture consumes the computation cost and lacks generality. Can you compare NAS and model parameter adaptation for few-shot classification?\n- Fig. 1 is quite confusing. Could you please provide some detailed information? I mean, how do you describe the cost/generalization and accuracy simultaneously in the same figure with a continuous curve?\n- I am a little bit confused of Eq. (6-7). Could you please provide me with some explanations?\n- How does the Information Bottleneck theory specifically guide the architecture selection process?\n- What is the optimal way to balance between expressivity and computational efficiency in the architecture search?\n- How does the framework handle different types of few-shot learning tasks beyond the current scope? For example, in cross-domain settings (meta-dataset), the vary-way vary-shot tasks are more complicated and challenging.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IsId5aLzPm", "reviewer": "Reviewer_xKah", "summary": "The paper proposes a novel framework called IBFS (Information Bottleneck-driven Few-shot Neural Architecture Search) for few-shot learning (FSL) tasks. IBFS leverages the Information Bottleneck (IB) theory to rank and select neural architectures without requiring any training, significantly reducing search costs. The authors demonstrate that the global convergence of Model-Agnostic Meta-Learning (MAML) can be guaranteed by considering only the first-order loss. Extensive experiments on NAS-Bench-201 and few-shot learning benchmarks show that IBFS achieves state-of-the-art performance with minimal search costs.\n\n## Update After Rebuttal\nThanks for the authors' response! I will keep the rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful and insightful review, which is very helpful for us to further improve this paper. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**Q1**: Sensitivity.\n\n**A1**: Many thanks for your comments! We will provide some evidence to prove the robustness of our method. In Table 3, we present the ablation study to evaluate the influence of $\\theta$ on NAS-Bench-201. As shown, the Kendall’s Tau consistently increases with the increase of $\\theta$  until $\\theta=0.75$,  when $\\theta=0.75$, our method obtains the highest  0.752 Kendall’s Tau and maximum accuracy on NAS-Bench-201. When $\\theta \\in [0.75, 1]$, Kendall’s Tau slightly decreases, for example,  when $\\theta=0.9$,  Kendall’s Tau is 0.739. We can clearly observe that the impact of $\\theta$ is extremely slight. More importantly, our method is only related to the  **expressivity** of the neural networks. The search space included those neural networks that are stable in different devices and scenarios; therefore, even when we deployed our method on different devices and scenarios, our method still achieves optimal performance at $\\theta=0.75$.\n\n**Q2**: Scalability. \n\n**A2**: Many thanks for your comments! Due to the page limit of the main text, we provide additional results on **larger dataset (ImageNet1k)**  in **Appendix**. To be specific, the detailed experimental results on **ImageNet1k** are presented in **App. B**, which can be found in **line 715 - line 763** in our paper. As shown in **Table 4**,  we can see that our IBFS method consistently outperforms compared SOTA methods: it achieves highest 76.6\\% Top-1 accuracy, with 0.0042 (GPU-days) fewest search costs. Compared with its peer competitors, our method achieves the Top-1 accuracy of 1.6\\% higher than SWAP, 1.6\\% higher than NASI-ADA, and 1.1\\% higher than TENAS. Notably, our method only searches in small CIFAR-10 dataset, and then can well generalize to large ImageNet1k, which largely reduces search costs.\n\n**Q3**: Model-agnostic. \n\n**A3**: Many thanks for your comments! The goal of this paper is to design neural networks for few-shot learning tasks. Our paper follows the MetaDiff (AAAI24), MetaNTK-NAS (CVPR22), which only provide 5way-1shot and 5way-5shot settings of few-shot learning tasks in mini-ImageNet and tiered-ImageNet datasets. As shown in Table 1, we can clearly see that all methods (i.e., MetaDiff) in FSL use CNNs as the backbone, therefore, we also use CNNs for fair comparison. If we use transformers as the backbone in FSL, it will bring unfairness for comparison. In addition, to evaluate the effectiveness of the proposed method, we provide the additional experiments on NASBench-201 search space in three datasets (i.e., n CIFAR-10, CIFAR-100, and ImageNet-16-120). Therefore, we believe our empirical evaluation is sufficient, and not beyond the scope of ICML 2025.  \n\nTo further scrutinize the generalizability of our method for **transformer**,  we devote lots of effort \\& exploration in the IBFS day and night, conducting additional experiments on AutoFormer [2] in ImageNet. The experimental setting is the same as TF-TAS-T [3].  We can find that IBFS achieves the highest Top-1 accuracy. Those empirical results demonstrate the strong generalizability of our method for transformer design. \n\n|  **NAS method**  |   Year   | **Top-1 (%)** | **Search Cost**(GPU Days) | Model Type  | Search Method |\n| :--------------: | :------: | :-----------: | :-----------------------: | :---------: | :-----------: |\n|    ViT-Ti [1]    | ICLR2020 |     74.5      |             -             | Transformer |    Manual     |\n| AutoFormer-T [2] | CVPR2021 |     74.9      |            24             | Transformer |   Evolution   |\n|   TF-TAS-T [3]   | CVPR2022 |     75.3      |            0.5            | Transformer | Training-free |\n|   ViTAS-C [4]    | ECCV2022 |     74.7      |            32             | Transformer |   Evolution   |\n|  Auto-Prox [5]   | AAAI2024 |     75.6      |            0.1            | Transformer | Training-free |\n|     **IBFS**     |          |     76.5      |            0.03            |    CNNs     | Training-free |\n\n**Q4**: Line 665: should it be App. B? \n\n**A4**: Many thanks for your comments! This is a typo. Thanks for pointing it out. Line 665 is App. B, we will correct it in the final revision.\n\n[1] An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR2020\n\n[2] Autoformer: Searching transformers for visual recognition. In ICCV 2021\n\n[3] Training free transformer architecture search. In CVPR 2022\n\n[4] Vision transformer architecture search. In ECCV 2022\n\n[5] Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery. AAAI2024", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I did not check the correctness", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The key contribution of the paper is very helpful, especially to reduce the cost of meta-learning.", "essential_references_not_discussed": "I am not in the field of NAS, not sure about this.", "other_strengths_and_weaknesses": "Strength:\n1. The motivation is clear and strong.\n2. IBFS eliminates the need for training during the architecture search phase, reducing computational costs and making it highly efficient compared to traditional NAS methods.\n3. Extensive experiments demonstrate the effectiveness of IBFS.\n\nWeakness:\n1. Sensitivity: The paper mentions that IBFS is slightly sensitive to hyperparameters (e.g., $\\theta$), which could affect its robustness and ease of use in different settings.\n2. Scalability: Since IBFS can reduce cost and works well on small datasets, that would be great if its generalization to large datasets also could demonstrate its good performance.\n3. Model-agnostic: It looks most experiments are conducted based on the traditional CNNs. How about the performance on Transformers or other tasks beyond image classification?", "other_comments_or_suggestions": "Line 665: should it be App. B?", "questions_for_authors": "see the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D9dO6QHr0Q", "reviewer": "Reviewer_3Pd4", "summary": "This paper proposes an entropy-based expressivity metric, namely Few-shot Neural Architecture Search (IBFS), for training-free neural architecture search (training-free NAS), which uses Jacobian eigenvalues at initialization to estimate model performance. The authors claim that the motivation for such formulation is inspired by the theorem of global convergence of model-agnostic meta-learning (MAML) and information bottleneck (IB) theories. The proposed method IBFS is evaluated on NAS-Bench-201, mini-ImageNet 5-way, tiered-ImageNet 5-way, and ImageNet1k (i.e., DARTS search space). Ablation studies about the influence of $\\theta$ are also reported.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful and insightful review, which is very helpful for us to further improve this paper. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**Q1**: Concern about Theorem 4.1. **A1**: Many thanks for your comments! We regret the lack of clarity in Theorem 4.1 and will provide a detailed proof and definitions for terms like \\($ \\ell_{\\text{inner}}$ \\), \\( $\\nabla \\theta$ \\), and \\( $\\phi$ \\). However, we must respectfully correct a misunderstanding: MAML is not a ``first-order \\( $\\ell_{\\text{inner}}$ \\) convergence problem'' as stated. We will provide a detailed definition for $\\theta$, H, etc, in the final version. $\\theta$ is hyperparamter, $IB_{proxy}=- \\theta \\sum_{k=1}^{N} p \\log p$.\n\nProof.\n\n- **MAML Update Rule**: The inner-loop update in MAML is given by:\n\n  $Wt′=Wt−α∇WtL(Dt^{train},Wt),W_t' = W^t - \\alpha \\nabla_{W^t} \\mathcal{L}(D_t^{train}, W^t),$\n\n  where $\\alpha$ is the learning rate for the inner loop.\n\n- **Meta-Update**: The meta-update step updates the parameters by:\n\n  $Wt+1=Wt−η∑t∇WtL(Dt^{test},Wt′),W^{t+1} = W^t - \\eta \\sum_t \\nabla_{W^t} \\mathcal{L}(D_t^{test}, W_t'),$\n\n  where $\\eta$ is the learning rate for the outer loop.\n\n- **Hessian Approximation**: The second-order term is approximated by the Hessian matrix, H=∇Wt2L(Dttrain,Wt)H = \\nabla^2_{W^t} \\mathcal{L}(D_t^{train}, W^t), and the meta-update is simplified using the transformation Φ=I−αH\\Phi = I - \\alpha H.\n\n- **Loss Function**: The loss function is defined as:\n\n  $ℓ(Wt)=12∥Yt^−Yt∥22.\\ell(W^t) = \\frac{1}{2} \\|\\hat{Y^t} - Y^t\\|_2^2.$\n\nThe gradient-based updates for both the inner and outer loops are correctly formulated. The inner loop updates the weights based on the training set $D_t^{train}$, and the outer loop meta-update step minimizes the loss over the test set $D_t^{test}$. The use of the Hessian matrix approximation for second-order terms is standard in MAML, and the transformation $Φ=I−αH\\Phi = I - \\alpha H$ is correctly introduced. This simplification allows for easier analysis of the convergence behavior.The proof shows how the loss function $\\ell(W^t)$ evolves through the meta-learning process. After applying gradient descent, the relationship for $\\ell(W^{t+1})$ is given by:\n\n$ℓ(Wt+1)≤ℓ(Wt)−η∥∇Wℓ(Wt)∥2.\\ell(W^{t+1}) \\leq \\ell(W^t) - \\eta \\|\\nabla_W \\ell(W^t)\\|^2$.\n\nThe bound on the gradient is correct:\n\n$∥∇Wℓ(Wt)∥2≥σmin⁡(Φ)ℓ(Wt),\\|\\nabla_W \\ell(W^t)\\|^2 \\geq \\sigma_{\\min}(\\Phi) \\ell(W^t),$\n\nwhich implies that the loss function decreases at a rate determined by the eigenvalue $\\sigma_{\\min}(\\Phi)$ of the matrix $\\Phi$. The recurrence relation is derived correctly:\n\n$ℓ(Wt+1)≤(1−ησmin⁡(Φ))ℓ(Wt),\\ell(W^{t+1}) \\leq \\left(1 - \\eta \\sigma_{\\min}(\\Phi)\\right) \\ell(W^t)$,\n\nwe can obtain:\n\n$ℓ(Wt)≤ \\left(1 - \\frac{\\eta_0 \\sigma_{\\min}(\\Phi)}{3} \\right)^{2t} R.$\n\n**Q2**: Concern about Experiments. **A2**: Many thanks for your comments! We provide newer works for comparison below, demonstrating that our method still achieves the best performance.\n\n|                             | Year     | Cost(s) | C10 (val)    | C10(test)          | C-100(val)   | C-100(test)  | Img(val)     | Img(test)    |\n| --------------------------- | -------- | ------- | ------------ | ------------------ | ------------ | ------------ | ------------ | ------------ |\n| GradSign [6]                | ICLR2022 | 30.38   | -            | 93.52 ± 0.19       | -            | 70.57 ± 0.31 | -            | 41.89 ± 0.69 |\n| ZiCo [7]                    | ICLR2023 | 6.2     | 93.50 ± 0.18 | -                  | 70.62 ± 0.26 | -            | 42.04 ± 0.82 | -            |\n| IS-DARTS [8]                | AAAI2024 | 7200    | 91.55 ± 0.00 | 94.36 ± 0.00       | 73.49 ± 0.00 | 73.51 ± 0.00 | 46.37 ± 0.00 | 46.34 ± 0.00 |\n| AZ-NAS [9]                  | CVPR2024 | 0.71    | -            | 93.53 ± 0.15 0.723 | -            | 70.75 ± 0.48 | -            | 45.43 ± 0.29 |\n| SWAP [10]                   | ICLR2024 | 4.7     | 87.31 ± 2.36 | 90.48 ± 0.94       | 65.92 ± 4.32 | 67.13 ± 1.83 | 33.85 ± 4.98 | 35.40 ± 3.96 |\n| IBFS(ours) ( $\\theta$=0)    |          | 3.36    | 89.58 ± 0.57 | 92.96 ± 0.81       | 69.17 ± 1.81 | 68.94 ± 1.41 | 41.30 ± 1.79 | 41.11 ± 1.51 |\n| IBFS(ours) ( $\\theta$=0.75) |          | 3.82    | 91.55 ± 0.76 | 94.37 ± 0.34       | 73.31 ± 2.12 | 73.09 ± 2.08 | 45.59 ± 0.32 | 46.33 ± 1.27 |\n\n\n\n**Q3**: Concern about novelty. **A3**: Many thanks! Our paper is not incremental! The reviewer DRyn(**Rating:** 3),  xKah(**Rating:** 3)  and  masR (**Rating:** 3) all appreciate the novelty. While NASWOT uses Jacobian covariance, our entropy-based Jacobian metric introduces a novel perspective by leveraging entropy to capture expressivity, which we will justify with a detailed comparison in the revision. We believe this distinction offers a meaningful contribution to training-free NAS.\n\n**Q4**: Concern about  Algorithm 1. **A4**: We will cite the  Algorithm 1 in the final version.", "claims_and_evidence": "This paper has **severe issues regarding clarity**, which significantly impact readability and make it unclear whether the proposed claims have been fully justified. Specifically:\n\n### 1. Claims about global convergence of MAML (i.e., Theorem 4.1).\nThe paper states\n> ... we derive that the global convergence of MAML can be guaranteed by only considering the first-order approximation of loss landscape, which transfers the problem of crafting a specialized architecture for FSL to find a suitable proxy ... \n\nHowever, it is **surprising** that no proof or reference (if it is a restatement) is given for Theorem 4.1, making it almost impossible to verify whether this claim is valid. Furthermore, it is unclear how is it connected to the flatness of loss landscape, as nothing in Theorem 4.1 actually discusses loss landscape curvature. In fact, MAML is a first-order $\\ell_{inner}$ convergence problem, but Theorem 4.1 does not explicitly distinguish between first-order and second-order effects. In addition, the formulation of Therorem 4.1 itself is problematic. For example, a lot of terms used without definition (e.g., $\\eta_0, \\sigma_{\\text{min}}(\\Phi), \\xi_{\\text{max}}(\\ell_{\\infty})$). Even with defined terms like $\\ell_{\\text{inner}} = \\nabla_{\\theta} F_t(\\hat{X}^t, X^t, Y^t)$, it is unclear what does $\\nabla_{\\theta}$ apply to? Is it the gradient of a loss function or some feature transformation? And there is no explicit definition of how  $\\ell_{inner}$ relates to the meta-learning optimization problem in MAML. Overall, it is **amazing** to see such a poor-quality theorem existing in a manuscript submitted to ICML.\n\n### 2. Problematic derivation between IB theory and the proposed expressivity proxy\nWhile the authors claim their expressivity proxy is IB-driven, and they do provide a derivation in Sec 4.1, however, the derivation seems to be problematic. Specifically:\n-  **Equation (8) does not follow directly from Eq. (7).**  \n   In standard IB, stationarity gives $p(r \\mid x) \\propto p(r) \\exp \\left[ -\\beta \\mathbb{E}_{p(y|x)} (-\\log p(y \\mid r)) \\right] $.  \n   In the paper, the derivation inexplicably substitutes an _input entropy_ $H(X)$ in place of the typical “relevance to $y$.”\n\n- **Label dependence disappears.**  \n   The “$ \\sum_y p(y \\mid x) \\log \\dots $” part in Eq. (7) is crucial to measuring how well $r$ captures label information.  \n   In the next step, it is replaced by $\\exp(-\\beta H(X))$. That is not standard and is never shown in detail.\n\n- **Definition of NN expressivity is an inequation**.\n\n  It is confusing how a definition can contain an inequation in Eq. (10). I suspect authors try to switch $x$ in Eq. (8) to $\\varsigma$ in Eq. (10) so that the formulation of their NN expressivity is \"IB theory-driven\". They might want to say \n> \"Because $p(r \\mid x)$ is bounded by an exponential factor,\ntherefore the resulting entropy over the eigenvalues of the network’s Jacobian is also bounded by a similar exponential factor.\"\n\nBut that chain of logic is never spelled out (and usually, one would need to connect “$r$” to “Jacobians” to “label information” in detail). Even so, this logic itself is problematic. It just shows the authors are using the *same bounding pattern* for two different objects, without providing a rigorous link that ensures we are applying the same bound to the right distributions. In addition, what is $p$ in Eq. (10) is also not defined. \n\n- **Similar to Theorem 4.1, key constants and notations** $( p(\\tilde{r}), \\lambda(x), H, \\text{etc.}) $ **either not defined or simply introduced as a black box.**\n\nOverall, the entire Section 4.1 plus Theorem 4.1 require a major revision for clarity and the current version is *far away* from the acceptance bar of ICML.", "methods_and_evaluation_criteria": "In addition to the problems mentioned in Claims And Evidence, I have three more concerns about the methods.\n\n- Utilization of Jacobian metric is not new in training-free NAS, as authors themselves also mention the NASWOT, which uses the Jacobian covariance of activation layers to score the architectures. However, the authors didn't discuss the relationship between their proposed NN expressivity with NASWOT, nor why they think the eigenvalues would outperform. \n\n- The paper mentions Algorithm 1 which contains IBFS’s implementation details. However, I didn't find any reference to it in the main text. I would like to request authors to point it out. Also, it seems to have no relationship with the NN expressivity.\n\n- The empirical performance seems heavily influenced by a hyperparameter $\\theta$. However, I didn't find a definition of it in the entire paper.", "theoretical_claims": "Yes, as stated in Claims And Evidence, the entire theoretical derivation is problematic and requires significant revision.", "experimental_designs_or_analyses": "Due to the clarity issues, I am unable to justify the validity of the experimental designs, as I don't know the definition of $\\theta$. However, given that it seems to significantly influence the results given in Table 3, I would wonder how to determine a suitable $\\theta$ if we have no prior information about the tasks/benchmarks we are testing on. If it requires the knowledge of choosing a specific $\\theta$ and measuring the corresponding architecture's performance, it will compromise the benefits of low search costs of IBFS due to these trials and errors.\n\nIn addition, the baselines used in this paper are somehow outdated. Training-free NAS has developed a lot since NASWOT and there are many other proxies/metrics available for comparison. I would suggest authors to refer to more recent results. A non-exhaustive list of training-free metrics until 2022 can be found in NAS-Bench-Suite-Zero [1].\n\n[1] NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies, NeurIPS Datasets and Benchmarks Track 2022.", "relation_to_broader_scientific_literature": "The paper does not cite or compare against newer zero-cost NAS methods, making its benchmark evaluation incomplete. See Experimental Designs Or Analyses.", "essential_references_not_discussed": "As mentioned in Methods And Evaluation Criteria, NASWOT is cited but does not carefully discuss the relationship with the proposed NN expressivity given they are both extracted from Jacobian metric.", "other_strengths_and_weaknesses": "Strengths:\n- Introduce a new training-free metric, which is valuable and relevant to training-free NAS community\n- Empirical performance is strong, especially in NAS-Bench-201 and DARTS search space (although the chosen of $\\theta$ might bring an unfair advantage for the method)\n\nWeaknesses:\n-  Severe clarity issues **[major issue that suggests a clear rejection for this paper]**: Undefined notation, missing links between theory and experiments, and inconsistent references (e.g., Algorithm 1 is never cited in the main text) make the paper difficult to follow.\n- Theorem 4.1 is unverifiable: The paper states but does not prove its key theoretical result, raising concerns about its validity.\n- Key hyperparameter ($\\theta$) is undefined: The ablation study shows that $\\theta$ significantly impacts performance, but the paper never explains its theoretical role.\n- Experimental evaluation is outdated: More recent baselines can be included.\n- Weak justification of novelty: The paper does not sufficiently explain why entropy-based Jacobian measures are better than existing alternatives like Jacobian covariance (NASWOT).", "other_comments_or_suggestions": "I strongly encourage the authors to carefully revise Theorem 4.1 and Section 4.1. While some level of logical gap in derivations may be understandable, the omission of key term definitions is completely unacceptable for a submission to ICML. Such omissions make it impossible for readers to properly assess the claims and significantly hinder the paper’s clarity and rigor.", "questions_for_authors": "Please refer to my previous comments. I do not have additional questions at this time. Unless the authors can demonstrate that I have overlooked key parts of the derivation, my evaluation of the current version of the paper is unlikely to change.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed an approach to select the neural architectures without involving any training. They use a novel information bottleneck theory-driven few-shot neural architecture search framework to achieve this. Both theoretical analysis and experimental results are provided in this paper. 

This paper received positive ratings from three of the four reviewers. Most of the concerns have been addressed during the rebuttal period. However, there are still some important concerns from Reviewer 3Pd4. For example, the authors claimed, "MAML is first-order convergence problem, however, this is result obtained Theorem 4.1. In fact, MAML is a second-order convergence problem." This claim is confusing, in my view. Why the result obtained from Theorem 4.1 is not consistent with the claim in Section 3 (Model-agnostic meta-learning (MAML) is a second-order problem) should be carefully explained in this paper. Considering all of these, I recommend a weak acceptance for this paper.
Decision: Accept (poster)