SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS
Authors: Yameng Peng
Andy Song
Haytham M. Fayek
Vic Ciesielski
Xiaojun Chang
Venue: ICLR 2024 spotlight (2024)
Keywords: Neural Architecture Search
Network evaluation
Training-free metric
Deep neural networks
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Training-free metrics (a.k.a. zero-cost proxies) are widely used to avoid resource-intensive neural network training, especially in Neural Architecture Search (NAS). Recent studies show that existing training-free metrics have several limitations, such as limited correlation and poor generalisation across different search spaces and tasks. Hence, we propose Sample-Wise Activation Patterns and its derivative, SWAP-Score, a novel high-performance training-free metric. It measures the expressivity of networks over a batch of input samples. The SWAP-Score is strongly correlated with ground-truth performance across various search spaces and tasks, outperforming 15 existing training-free metrics on NAS-Bench-101/201/301 and TransNAS-Bench-101. The SWAP-Score can be further enhanced by regularisation, which leads to even higher correlations in cell-based search space and enables model size control during the search. For example, Spearman’s rank correlation coefficient between regularised SWAP-Score and CIFAR-100 validation accuracies on NAS-Bench-201 networks is 0.90, significantly higher than 0.80 from the second-best metric, NWOT. When integrated with an evolutionary algorithm for NAS, our SWAP-NAS achieves competitive performance on CIFAR-10 and ImageNet in approximately 6 minutes and 9 minutes of GPU time respectively.

Published as a conference paper at ICLR 2024
S SWAP-NAS: SAMPLE-WISE ACTIVATION PATTERNS FOR ULTRA-FAST NAS
Yameng Pengl Andy Songi Haytham M. Fayek* Vic Ciesielski Xiaojun Chang
TScooo of Computing Technologies. RMIT University, Australia
University of Technology Sydney, MMohamed bin Zayed University of Artificial Intelligence 102 6 com, andy. songermis edu. au, haytham. fayek ieee. org vic. ciesiel skii Ormit.edu au, xiaojun. cwwnghuts. edd..uuueeedduuu
ABSTRACT
Training-free metrics (a.k.a. zero-cost proxies) are widely used to avoid resource- intensive neural network training, especially in Neural Architecture Search (NAS), Recent studies show that existing maining-free metries have several limitations, such as limited correlation and poor generalisation across different search spaces and tasks. Hence, propose Sample-Win Activation Patterns and its derivative, SWAP-Score, a novel high-performance training- free metric. It measures the ex- pressivity of networks over batch of input samples. The SWAP-Scort is strongly correlated with ground-truth performance across various search spaces and tasks, outperforming 15 existing training-free metrics on NAS-Bench- watcch-01/2013300 and TransNAS- BencB-ennlh The SWAP-Score can be further enhanced by regularisa- tion, which leads to even higher correlations in cell-based search space and en- ables model size control during the search. For example, Spearman's 's rank corre- lation coefficient between regularised SWAP- and CIFAR- 100 validation ac- curacies on JAS-Bench-201 networks is 0.90, significantly higher than 0.80 from the second-best best metric, NWOT. When integrated with an evolutionary algorithm for NAS, our SWAP-NAS achieves competitive performance on CIFAR- 10 and ImageNet in approximately 6 minutes and 9 minutes of GPU time respectively.'
INTRODUCTION
Performance evaluation of neural networks critical, especially Neural Architecture Search (NAS) which aims to automatically construct nigh-performing neural networks for given task. The conventional approach evaluates candidate networks by feed-forward and back- pookkpropagation training. This process typically requires every candidate to be trained the target dataset until convergence (Liu et al., 2019; Zoph & Le, 2017), and often leads to prohibitively high computational cost (Ren al., 2022; White al., 2023). To mitigate this cost, several alternatives have been introduced, such as performance predictors, architecture comparators and weight-sharing strategies.
A divergent approach the use of training-free metrics, also known as zero-cost proxies (Chen et al., 2021a; Lin et al., 2021; Lopes et al., 2021; Mellor et al., 2021; Mok et al., 2022; Tanaka et al., 2020b; Li al., 2023). The aim is eliminate the need for network training entirely. These metries are either positively or negatively correlated with the networks' ground-trutt performance. Typically, they only necessitate few forward or backward passes with a mini batch of input data, making their computational costs negligible compared to traditional network performance evaluation. How- ever, training-free metrics face several challenges: (1) Unreliable correlation with the network's ground-truth performance (Chen et al., 202 la; Mok et al., 2022); (2) Limited generalisation across different search spaces and tasks (Krishnakumar et al., 2022), even unable to consistently surpass some computationally simple counterparts like the number of network parameters FLOPs; (3) A bias towards larger models (White et al., 2023), which means they do not naturally lead to smaller models when such models are desirable.
Corresponding author: 'Our code is available at https: / //g thub. com com/pym/ 024/SWAP. All experiments conducted on single Tesla V100 GPU.

---

Published as a conference paper at ICLR 2024
To overcome these limitations, we introduce novel high perffommreme training- free metric, Sample-Wis Wise Activation Patterns expressivity (Montúfar et al., 2014; Xiong et al., 2020), but addresses the above limitations. It correlates with the network ground-truth per- formance much stronger. We rigorously evalu- ate its predictive capabilities across five distinct search spaces - NAS- 100- NAS-Bench- 201, NAS-Bench-301 and TransNAS-Bewa 101-Micro/Macro, validate whether SWAP- Score can generalise well on different types of tasks. is benchmarked against 15 existing training-free metrics to gauge its correlation with networks' ground-truth performance. Further, the correlation of SWAP-Score can be increased by regularisation, and enables model size con- trol during the architecture search. Finally, we demonstrate its capability by integrating SWAP- Score into NAS as a new method, SWAP-NAS. This method combines the efficiency of SWAP- Score with the effectiveness of population- based evolutionary search, which is typically computa- tionally intensive. This work's primary contribu- tions are as follows:
which is inspired by the studies of network
OFFR 10 cocouuo (81
P-DARTS
FairDARTS
BENA
TE-NAS
CARS
272
GDAS
ENAB
05
62
08
(CIPU
Figure I: Search cost and performance compar- ison between SWAP-NAS and other SoTA NAS on CIFAR-10. Methods over GPU day are not included. The dot size indicates the model size.
We introduce Sample-Win Wise Activation Patterns and its derivative, SWAP- Score, a novel high correlation training- free metric. Unlike revealing network expressivity through standard activation patterns, SWAP-Score offers a significantly higher capability to differentiate networks. Compre- hensive experiments validate its robust generalisation and superior performance across five benchmark search spaces, i.e., stack-based and cell-based, and seven tasks, i.e., image classification, object detection, autoencoding and jigsaw puzzle, outperforming 15 existing training- -free metrics including recent proposed NWOT and ZiCo.
We enable model size control in architecture search by adding regularisation to SWAP- Score. Besides, regularised SWAP-Score can more accurately align with the performance distribution of cell-based search spaces.
We propose an ultra-fast NAS algorithm, SWAP-NAS, by integrating regularised SWAP- Score with evolutionary search. It can complete search on CIFAR- in a mere 0.004 GPU days (6 minutes), outperforming SoTA NAS methods in both speed and perfor- mance, as illustrated in Fig. 1. A direct search on ImageNet requires only 0.006 GPU days minutes) achieve SoTA NAS, demonstrating its high efficiency and performance.
RELATED WORK
Early network evaluation approaches often train each candidate network individually. For instance, AmoebaNet (Real et al., 2019) employs an evolutionary search algorithm and trains every sampled network from scratch, requiring approximately 3150 GPU days search on the CIFAR-10 dataset. The resulting architecture, when transferred to the ImageNet dataset, achieves a top-1 accuracy of 74.5%. Performance predictors can reduce evaluation costs, such as training regression models based on aechitecture-accuaayy pairs (Liu et al., 2018; Luo et al., 2020; Shi et al., 2020; Wen et al., 2020; Peng et al., 2023). Another strategy architecture comparator, which selects the better ar- chitecture from pair of candidates through pairwise comparison (Dudziak et al., 2020; Chen et al., 2021b). Nevertheless, both approaches necessitate the preparation of training data consisting of architecture-accuracy pairs. One alternative evaluation strategy is weight-sharing among candidate architectures, eliminating the need to train each candidate individually (Cai et al., 2018; Pham et al., 2018; Liang al., 2019; Liu et al., 2019; Dong & Yang, 2019b; Xu et al., 2020; Chu et al., 2020). With this strategy, the computational overhead in NAS can be substantially reduced from tens of thousands of GPU hours to dozens, less. For example, DARTS (Liu al., 2019) combines the
2

---

Published as a conference paper at ICLR 2024
one-shot model, representative weight-sharing strategy, with a gradient- search algorithm, requiring only GPÚ days to achieve a test accuracy of 97.33% on CIFAR-10 10. Unfortumately, is problematic to share trained weights among heterogeneous networks. In addition, weight-sharing strategies often suffer from an optimisation gap between the ground-truth performance and the ap- proximated performance evaluated by these strategies (Shi et al., 2020; Xie et al., 2021). Further, the one-shot model, treats the entire search space as an over-parameterised super-network and that difficult optimise and work with limited resources.
In comparison with the above strategies, training-free metrics further reduce evaluation cost as no training required (Tanaka et al., 2020b; Chen et al., 2021a; Lin al., 2021; Lopes et al., 2021; Mellor et al., 2021; Mok et al., 2022; Li et al., 2023). For instance, by combining two training- frree metries, the number of linear regions (Xiong et al., 2020) and the spectrum of the neural tangent kernel (Jacot et al., 2018), TE-NAS (Chen et al., 2021a) only requires 0.05 GPU days for CIFAR-10 and 0.17 GPU days for ImageNet. NWOT (Mellor et al., 2021) explores the overlap of activations between data points in untrained networks as an indicator of performance. Zen-NAS (Lin et al., 2021) 1) observed that most training-free NAS were inferior to the training-based state-of-the-arr NAS methods. Thus, they proposed Zen-Score, training- free metric inspired by the network expressivity studies (Jacot et al., 2018; Xiong et al., 2020). With a specialised search space, Zen-NAS achieves 83.6% top-1 accuracy ImageNet in 0.5 GPU day, which is the first training-free NAS that out- performs training-based NAS methods. However, Zen-Score is not mathematically well-defined in irregular design spaces, thus, it cannot be applied to search spaces like cell-based. ZiCo (Li al., 2023) noted that none of the existing training-free metrics could work consistently better than the number of network parameters. By leveraging the mean value and standard deviation of gradients across different training batches as the indicator, they proposed ZiCo, which demonstrates consistent and better performance on several search spaces and tasks, than the number of network parameters. However, it is still inferior to another naive metric, FLOPs. A recent empirical study, NAS-Bench- Suite-Zero (Krishnakumar et al., 2022), evaluates 13 training-fre metrics on multiple tasks. Their results indicate that most training-free metrics do not generalise well across different types of search spaces and tasks. Moreover, simple baselines such as the number of network parameters and FLOPs show better performance than some training-free metrics which are more computationally complex.
SAMPLE-WE WISE ACTIVATION PATTERNS AND SWAP-SCORE
To address the aforementioned challenges, we introduce SWAP-Score. Similar to NWOT and Zen- Score, SWAP-Scome is also inspired by studies on network expressivity, aiming to uncover the ex- pressivity of deep neural networks by examining their activation patterns. What sets SWAP-Score apart from other training-free free metrics is its focus on sample- wise activation patterns, which offers high correlation and robust performance across a wide range of search spaces, including both stack- based and cell-based, well diverse tasks, such as image classification, object detection, scene classification, autoencoding and jigsaw puzzles. The following section introduces the idea of re- vealing network's expressivity by examining the standard activation patterns. Then SWAP-Score is presented which is to better measure the network expressivity through sample-wise activation patterns. Lastly, we add regularisation the SWAP-Score.
3.1 STANDARD ACTIVATION PATTERNS, NETWORK'S 'S EXPRESSIVITY & LIMITATIONS
The studies on exploring the expressivity of deep neural networks (Pascanu et al., 2013; Montúfar et al., 2014; Xiong et al., 2020) demonstrate that, networks employing piecewise linear activation functions such as ReLU (Nair Hinton, 2010), each ReLU function partitions its input space into two regions: either zero or non-zero positive value. These ReI.U activation functions introduce piecewise linearity into the network. Since the composition of piecewise linear functions remains piecewise linear, ReLU neural network can be viewed as piecewise linear function. Conse- quently, the input space of such a network can be divided into multiple distinct segments, each referred to linear region. The number of distinct linear regions serves indicator of the network's 's functional complexity. A network with more linear regions is capable of capturing more complex features in the data, thereby exhibiting higher expressivity.
Following this idea, the network's 's expressivity can be revealed by counting the cardinality of a set composed of standard activation patterns.
3

---

Published as a conference paper at ICLR 2024
Definition 3.1. Given N as ReLU deep neural network, 0 as fixed set of network parameters (randomly initialised weights and biases) of N, batch of inputs containing S samples, the standard activation pattern, AN,0, is defined set of post-activation values shown as follows:
AN = {p(0) ( = W & {1, (L.....s]]
(I)
where V denotes the number of intermediate values feeding into ReLU layers. pt denotes single post-activation value from the nd intermediate value 4th sample. 1(xx) is the indicator function that identifies the unique activation patterns. In the context of ReLU networks, the Signum function can be adopted as the indicator function, that converts positive non-zero values to one while leaving containing the binarised cost-activation
zero values unchanged. Consequently, AN,o represents values produced by network N with parameters and S input samples.
The AN,0 also be viewed matrix, with each element (p:
representing vector ( Y== of binarised post-activation values over all intermediate values in V. Each value cell corresponds to 1(p(") defined in Eq. 1. The upper bound of the cardinality of AN equal the number of imput samples, S. Since V represents the number of intermediate values feeding into the activation layers, define N contains layers, the dimensionality of input is C W H, we have:
El n/.
if is MLP.
(2)
Ele a x
m + 1)
lu + 1)). if CNN.
For multi-layer perceptrons (MLP), ny denotes the number of hidden neurons in 7th layer. For convolutional neural networks (CNN), a denotes the number of convolution kernels, t denotes the
stride of convolution kernels, k denotes the kernel size in 7 layer. Hence, the length of Il (p()) y influenced by the dimensionality of the input samples. Given the same number inputs, higher- dimensional inputs deeper networks will generate more intermediate values, making it more likely to produce distinct vectors and reach the upper bounds of cardinality. Fig. 2 illustrates two examples, where (a) shows the matrix Âv,e derived from low-dimensional inputs, while that of (b) is derived from sebher-dimensionll inputs. Due to pattern duplication, the cardinality in (a) is 4, whereas in (b) it is 5. Note, the latter reaches the upper limit, the number of input samples, S. that is this case. This highlights the limitation of examining standard activation patterns for measuring the network's expressivity. Methods like TE-NAS (Chen al., 2021a) only allow inputs of small dimensions, e.g., 1 x 3 x 3. Otherwise, the metric values from different networks will all approach the number of input samples, making them indistinguishable.
9 %
9 by
1
B 1 by
4
0
1 0
1
1 1
%
0 1 0
%
0 1 0
|AN
=
0 1
AN A uper
0
1
1
1
11
(a)
Figure 2: Two examples of AN, with different inputs. Green denotes duplicate patterns.
3.2 SAMPLE- WISE ACTIVATION PATTERNS
SWAP- SeeAe- addresses the limitation identified above. also uses piecewise linear activation func- tions to measure the expressivity of deep neural networks. However, SWAP-Score does on sample-wise activation patterns, resulting a significantly higher upper bound, providing more space to discriminate separate networks with different performances.
Definition 3.2 (Sample-Wis Wise Activation Patterns). Given a ReLU deep neural network A, as fixed set of network parameters (randomly initialised weights and biases) of N, batch of inputs containing S samples, sample-wise activation patterns ÂNDo o is defined follows:
4

---

Published as a conference paper at ICLR 2024
AN = (p : pl 1(p!")) v (l....,,y
vt]. (Et....,
(3)
where p denotes single post-activation value from the 8t sample at the wh intermediate value.
Note, in comparison with Avro in Eq. 1, the vectors here are now sample-wise rather than interme- diate value-wiss wise as in standard activation patterns. In sample-wisg wise activation patterns, 1 (p: )==)) vector containing binarised post-activation values across all samples in S.
Definition 3.3 (SWAP-Scoon V). Given a SWAP set AN.e. e. the SWAP-Score v of network, / with fixed set of network parameters @ is defined as the cardinality of the set, computed as follows:
L N 0 AN.
(4)
Fig. 3 illustrates the connection and the difference between AN, and Ay,o in simplified form. Both sets are based on the same network with the same input. Hence, they have the same set of binarised post-activation values but are represented differently. The upper bound of the cardinality using standard activation patterns Av.e is 5. In contrast, the upper bound of SWAP-Score y extends 7. According Eq. the number of intermediate values V grows exponentially with either an increase in the dimensionality of the input samples the depth of the neural networks. This implies that the number of intermediate values, V. would be much larger than the number of input samples, S. As result, SWAP has significantly higher capacity for distinct patterns, which allows SWAP- Score to measure the network's expressivity more accurately. Specifically, this characteristic leads to high correlation with the ground-truth performance of network. M (see Section 4 for more details).
$
%
% %
%
%
%
1
1 1
1 0 1
4
1 0 1 0
1 0
0
1
1
0
1
O
1
1
1
0
1
0
5
0
A Upper Bound
%
1
0 1
1
AN plu pper Bownd =
1
15
1
0
O
AN,e
1
0
0
ANDO
Figure 3: Illustration of AN, and ÂNyo from network N. Green denotes the duplicate patterns.
3.3 REGULARISATION
As mentioned earlier, training-free metrics tend to bias towards larger models (White et al., 2023), meaning they do not naturally lead to smaller models when such models are desirable. Using the convolutional neural network or more channels have more parameters while producing more intermediate values compared to operations like skip connection (He al., 2016) pooling layer. Consequently, larger networks typically yield higher metric values, which may not always be desirable. To mitigate this bias, we add regularisation for SWop-Score.
example, the convolution operations with larger kernel sizes
Definition 3.4 (Regularisation). Given the total number of network parameters O coefficients μ and a, SWAP regularisation is defined as follows:
ʃ(0) = (9-2)
(5)
Definition 3.5 (Regularised SWAP-Score). Given regularisation function f(0). SWAP-Score of network N'' with a fixed set of network parameters 0, regularised SWAP-Score V/ is defined as:
N. N f(0).
5

---

Published as a conference paper at ICLR 2024
Regularisation function f() is bell-shaped curve. Coefficient μ controls the centre position of this curve. Coefficient ø adjusts the shape the curve. By explicitly setting the values for and o, the regularised SWAP-Score, V. guide the resulting architectures toward a desired range of model sizes.
EXPERIMENTS AND RESULTS
Comprehensive experiments are conducted confirm the effectiveness of SWAP-Score. Firstly, SWAP and its regularised version are benchmarked against 15 other training- free metrics across five distinct search spaces and seven tasks (Section 4.1). Subsequently, we integrate regu- larised SWAP-Score with evolutionary search as SWAP-NAS, to evaluate its performance in NAS. Further, state-of-the-arl NAS methods are compared in terms of both search performance and ef- ficiency (Section 4.2). Additionally,
ablation study demonstrates the effectiveness of SWAP- Scores, particularly when handling large size inputs and model size control (Section 4.3). The tasks involved in the experiments
Image classification tasks: CIFAR-10 CIFAR-100 (Krizhevsky, 2009), ImageNet- Ik (Deng et al., 2009) and ImageNet16- 120 (Chrabaszez et al., 2017).
2. Object detection task: Taskonomy dataset (Zamir et al., 2018).
3. Scene classification task: MIT Places dataset (Zhou et al., 2018).
4. Jigsaw puzzle: the imput is divided into patches and shuffled according to preset permuta- tions. The objective to classify which permutation is used (Krishnakumar et al., 2022).
5. Autoencoding: a pixel-level prediction task that encodes images into low-dimensional la- tent representation then reconstructs the raw image (Krishnakumar et al., 2022).
Six search spaces are used verify the advantages of SWAP-Score and regularised SWAP-Scone
1. NAS-Bench-101 (Ying et al., 2019): cell-based benchmark search space which tains 423624 unique architectures trained on CIFAR-10. The architectures are designed as ResNet-like and Inception-likk (He al., 2016: Szegedy et al., 2016).
2. NAS-Bench-201 (Dong & Yang, 2020): a cell-based benchmark search space which con- tains 15625 unique architectures trained on CIFAR-10, CIFAR-101 1-0 and ImageNet16-120
3. NAS-Bench-301 (Siems et al., 2020): surrogate benchmark space which contains archi- tectures sampled from the DARTS scarch space.
4. --1-----ennn---------- Macrroo (Duan et al., 2021): consists micro (cell-based) search space of size 4096, and macro (stack-based) search space of size 3256.
5. DARTS (Liu et al., 2019): cell-based search space contains 10 possible architectures.
4.1 SWAP-SCORES v.s. 15 OTHER TRAINING-FREE METRICS ON CORRELATION
Our SWAP-Scores are compared against 15 training-free (TF) metrics (Mellor al., 2021; Abdelfat- lah et al., 2021; Lin et al., 2021; Lopes et al., 2021; Turner et al., 2020; Ning et al., 2021; Wang et al.; Lee al.; Tanaka al., 2020a: Li al., 2023) across different search spaces and tasks, terms of correlation. These extensive studies follow the same setup as mar al., 2022), which is a standardised framework for verifying the effectiveness of training- free metrics. The hyper-parameters, such as batch size, input data, sampled architectures and random seeds are fixed and consistently applied to training-free metrics as NAS-Bench- BBench-Suite-Zen The comparison is shown in Fig. 4, where each column is the Spearman coefficients of all metrics one task with one given search space. They are computed on 1000 randomly sampled architectures. Each value in Fig. 4 is an average of five independent runs with different random seeds. The a and a setups for the regularisation function are determined by the model size distribution based on 1000 randomly sampled architectures. The process only requires a few seconds for each search space.
Zero (Krishnaku-
The results Fig. clearly demonstrate the exceptional predictive capability of SWAP-Scores across diverse types of search spaces and tasks. Notably, both SWAP-Scores outperform 15 other

---

Published as a conference paper at ICLR 2024
plain
0.07
-0.19
-0.30
-0.32
-0.11
-0.19
-0.33
0.35
0.34
0.23
-0.24
-0.25
-0.19
grasp
-0.12
-0.64
-0.27
0.27
-0.02
-0.43
0.34
-0.13
-0.23
-0.27
0.54
0.51
0.53
num Ir
0.00
0.00
0.00
0.00
0.00
0.00
-0.02
0.00
0.00
0.00
0.07
0.07
0.07
fisher
-0.58
-0.30
-0.26
-0.28
-0.18
-0.12
-0.27
0.32
0.46
0.67
0.47
0.49
0.53
grad _norm
-0.32
-0.55
-0.27
-0.24
0.32
-0.34
-0.03
0.37
0.40
0.66
0.55
0.57
0.62
snip
0.26
-0.37
-0.20
-0.19
0.20
-0.14
-0.04
0.43
0.47
0.71
0.55
0.58
0.62
nas
0.00
-0.01
0.02
-0.00
0.00
-0.01
0.00
0.16
0.40
0.52
0.33
0.70
0.58
12 norm
0.05
0.09
0.15
0.51
-0.19
0.29
0.46
0.35
0.33
0.53
0.68
0.68
0.71
zen
0.14
0.10
0.24
0.60
-0.01
0.28
0.44
0.52
0.55
0.72
0.38
0.33
0.34
jacov
0.18
0.07
0.19
-0.29
0.46
0.19
-0.05
0.56
0.51
0.75
0.70
0.74
0.70
params
-0.00
0.17
0.17
0.38
-0.18
0.33
0.46
0.44
0.46
0.63
0.69
0.72
0.73
synflow
0.00
0.12
0.34
0.31
0.00
0.28
0.18
0.48
0.49
0.72
0.75
0.73
0.76
zico
0.13
0.04
0.08
0.37
0.07
0.23
0.53
0.52
0.50
0.70
0.77
0.76
0.79
flops
-0.01
0.79
0.64
0.37
0.76
0.85
0.43
0.45
0.46
0.65
0.67
0.69
0.71
nwot
0.03
0.84
0.76
0.32
0.66
0.89
0.48
0.43
0.40
0.60
0.77
0.77
0.80
swap
-0.09
0.86
0.77
0.47
0.68
0.83
0.58
0.43
0.41
0.57
0.76
0.79
0.81
reg. swap
0.00
0.86
0.77
0.75
0.68
0.83
0.63
0.47
0.48
0.67
0.87
0.88
0.90
TN8101 MICRO AUTOENC
TN8101 MACRO MACRO-OBJECT OBJECT
TNB101 HACRO JIGSAW
TNB101 MACRO AUTOENC
NB101 HB101-CF10
TNB101 MACRO SCENE
TNB101 MICRO MICRO-JIGSAA JIGSAW
NB301- NB301-CF10 CF10
TNB101 MICRO OBJECT
-CCRD-SCENEE SCENE
NB281 HB281-IMGNT IMGNT
NB201 N8201-CF18 -CF1
NB201- N8201-CF100
TNB101 MICRO
Figure 4: Spearman's 's rank correlation coefficients between TF-metric values and networks' ground- truth performance for 15 existing metrics and our two SWAP-Scores. The rows and columns are sorted based on mean scores of five independent experiments for each metric.
metrics in the majority of the evaluations. One interesting observation is the significant enhance- ment in SWAP-Score's performance when regularisation applied (noted as 'reg_swap'), although its original intention is to control the model size during architecture search. This improvement is HAS-Beenc-2201, NAS-
particularly evident in cell-based search spaces, including NAS-Bench-101. Bench-301, 301, and TransNASS SaansNAS--encc----- -Mc-Mo... However, it is worth noting that regularisation does not appear to impact the correlation results in stack-basee space, TransNAS-Bent 1ench-101-Macrce 01-Marro.
4.2 SWAP-NAS ON DARTS SPACE
To further validate the effectiveness of SWAP-Score, utilise it for NAS by integrating the reg- ularised version with evolutionary search as SWAP-NAS. DARTS search space is used for the fol- lowing experiments, given its widespread presence in NAS studies, allowing a fair comparison with SoTA methods. For the evolutionary search, SWAP-NAS is similar to Real et al. (2019). but uses regularised SWAP-Score as the performance measure. Parent architectures generate possible off- spring iteratively in each search cycle, with both mutation and crossover operations. Unlike many training-basse based NAS approaches that initiate the search on CIFAR-10 and later transfer the architec- ture to ImageNet, SWAP-NA -AS conducts direct searches ImageNet. This is made feasible because of the high efficiency of SWAP-Scorc.
4.2. RESULTS ON CIFAR-10
Table shows the results of architectures found by SWAP-NAS from DARTS space for CIFAR- -10. The networks' training strategy and hyper-parameters are exactly following the setup in DARTS (Liu al., 2019). Three variations SWAP-NAS are presented, with different regularisation parameters and a. Regardless of these parameters. SWAP-NAS only requires 0.004 GPU days, or 6 minutes. That 6.5 times faster than the SoTA (TE-NAS). Meanwhile, the architectures found by SWAP- NAS also outperform most of the previous work. In addition, the capability of model size control is demonstrated. SWAP-NAS-A, with small μ and a values, generates smaller networks but also suffers a tiny performance deterioration. While SWAP-NAS-C. with large l and a, achieves the best error rate but at the cost slightly bloated network. This capability allows practitioners to find a balance between performance and model size according the need of the task.

---

Published as a conference paper at ICLR 2024
Test Errorr
Params
GPU
Search Method
Evaluation
(%)
(M)
Days
PNAS 2018)
0.34
SMBO
Predictor
EcoNAS (Zhou et al., 2020)
2.62:10.02
2.9
Evalation
Conventional
DARTS (L.iu al., 2019)f
3.00
Gradient
shot
EvNAS (Sinha & Chen, 2021)f
2.47=00.66
3.6
3.83
Evolution
One-shot
RandomNAS
Talwalkar, 2020
2.85=0.08
4.3
2.7
Random
One-shot
EFNA (Zhu 2019)
2.56x
8.47
0.65
Evolurion
Weights Inherit
PRE-NAS (Pong ot al., 2023)f
2. 2.4920.09
4.5
0.6
Evolution
Predictor
ENAS (Pham
2018)
2,89+
Reinforce
One-shot
FairDARTS (Chu el ol.,, 2020)
2.54+
2.8
0.42
Gradient
One shot
CARS (Yang
2020yf
2.62*
Evolution
P-DARTS (Chen ct al., 2019)T
2.50+
3.4
0.3
Gradicut
One-shot
TNASP (Lu al..
2.57=0004
3.6
0.3
Evolution
Predictor
PINAT (Lu
2023)f
2.54+0.08
Evolution
Predictor
GDAS (Dong & Yang. 2019g)
2,82+
2.5
0.17
Gradient
One-shot
CTNAS (Chen et 2021bb
2.59:00.4
Reinforce
Predictor
TE-NAS (Chen et al., 202
2.63:10.004
3.8
0.03
Pruning
Training free
SWAP-NAS-A (y=009 o=0.9)
2.65
3.06
0.004
Evolution
Training- free
SWAP-NAS-B (j==1.2, m=1.2)
2.507
3.48
0.004
Evalution
Training free
SWAP-NAS-C (um1,, o.5)
2.48:10.09
4.3
0.004
Evolution
Training- free
Table 1: Performance comparison between the networks found by SWAP-NAS and other methods on CIFAR-10. A lower test error rate is better. means the method adopted DARTS space. indicates the original paper only reported their best result.
Test Error Topp/TopS
Params (M)
GPU
Search Method
Evaluation
Searched
Days
Dataset
PrasylesNAAS (Cai et al., 2018)
24.977. 7.5
7.1
8,3
Grudient
One shot
ImageNet
DARTS (Liu et al., 2019)
26.7 1
4.7
Gradicat
One-shot
CIFAR-10
EvNAS (Sinha & Chen, 2021);
2..4/74
5.1
3.83
Evolution
Ohe-shol
CIFAR-IP
PRE-NAS (Peng et al., 202 1)
2..0/7.8
6.2
0.6
Evolution
Predictor
CIFAR-IF
FairDARTS (Chu al. 2020)t
26.3/8.3
2.8
0.42
Gradient
One-shot
CIFAR-10
CARS (Yang ct al., 202011
2..8/7.5
5.1
0.4
Evolution
One-shot
CIFAR-IO
P-DARTS (Chen
2019)1
24.4/74
Grudient
One-shot
CIFAR-10
CTNAS (Chen et al., 202 1b)
22.7/7.5
0.1+50
Reinforce
Predictor
ImageNet
(Li al., 2023)
21.91-
Evolution
Training-fre
ImageNet
PINAT (Lu et 2023)
24.977.5
5.2
0.3
Evolution
Predictor
CIFAR
GDAS (Dong & Yang. 2019t) 9t)
27.5/9.1
4.4
0.17
Gradient
One-shot
CIFAR-10
TE-NAS (Chen
202
24.5/7.5
Pruning
Training-frce
ImageNet
QE-NAS (Sun et al. 2022)1
25.51-
3.2
0.02
Evolution
Training-free
ImageNet
SWAP-NAS (n=25, 6=25)
24.077.6
5.8
0.006
Evolution
Training-free
ImageNet
Table 2: Performance comparison between the networks found by SWAP-NAS and other methods
on ImageNet. A lower test error rate is better. means the method adopted DARTS space.
4.2.2 RESULTS ON IMAGENET
Table 2 shows the NAS results on ImageNet, where training strategy and hyper-parametert setting are also the same in DARTS (Liu al., 2019). The search cost of SWAP-NAS here slightly increased to 0.006 GPU days, or 9 minutes. That is still 2.3 times faster than the SoTA (QE-NAS) yet with better performance.
4.3 ABLATION STUDY
The first ablation study is to further elucidate the limitation of standard activation patterns as dis- cussed in Section 3.1. A mini batch of 32 images is provided compute the metric values using the standard activation patterns, the sample-wise patterns, and the regularised patterns. these architec- tures using mini batch of inputs. The mini batch size aligns with that in JuS--eech-SSite-- (Krishnakumar et al., 2022) and other training- free metrics studies such as TE-NAS (Chen et al., 2021a). Table 3 shows results under three input sizes, 3 x 15 15 and 32 x 32, the latter being the original size of CIFAR-10 images. The mean and standard deviation for each metric are calculated based on their values across 1000 architectures. The corresponding Spearman correlations to true performance are also listed. With the standard activation patterns, |AN, shows tiny variation under input 3 x 3 and zero variation under larger size inputs, indicating its limited capability on distin- guishing the differences between architectures, particularly when the input dimensionality is high. Additionally, the mean value approaches its theoretical upper bound, the number of input samples, 32. This phenomenon confirms our discussion in Section 3.1. In contrast, both SWAP-Score v.
8

---

Published as a conference paper at ICLR 2024
and regularised SWAP-Score V'v, show significantly higher mean values and much more variation across the 1000 architectures. This indicates they have higher upper bounds and better capabilities to differentiate architectures. Notably, the regularised SWAP-Score exhibits even greater diversity and high correlation with the increase in input size. V Na starts from 0.89 and reaches 0.93.
Input
Input size 15
Input size 32
Metric values
Correlation
Metric
Correlation
Metric
Correlation
|Axr
15.90 +
31.0
0.45
32.0
4 A/.0
42.94 + 10.76
0.86
859.73 = 194.33
0.84
3567.02 02 775.45
0.90
37.77 + 11.13 0.89
831.31 + 216.11
0.92
3412.49 + 867.31
0.93
Table 3: Metric values and correlations from standard activation patterns, sample-wised patterns and regularised patterns with three input sizes (means and standard deviations from 1000 architectures).
250
6.5
1.7
2.1
200-
2.5
Model size (MB)
4.0
150
2.5
100
3.0
2.5
0.0
2.0
15
0.5
1.3
2.1
Model size (KB)
(a) Model size distribution 1000 networks
(b) Illustration the model size control of SWAP-
sampled from DARTS for CIFAR-10.
NAS on CIFAR-10.
Figure Illustration of regularised SWAP-Scors capability on model size control NAS.
The second ablation study illustrates regularised SWAP-Score for model size control. Fig. 5a shows the size distribution of 1000 models generated for CIFAR- 10 networks from DARTS space. It is almost Gaussian, ranging from 0.5 KB to 2.5 KB. To simplify the illustration, we set a and increase them simultaneously from 0.5 to 2.5, with step of 0.4. Fig. 5b visualises the relation between a, and the size of models that are found by SWAP-NAS. At each j value, SWAP-NAS runs times. Random jitter introduced in the drawing here reduce overlaps between dots of the same μ. From the figure, it can be clearly seen that values nicely correlate to the model sizes. The same p value leads to almost the same model size. By adjusting μ along with 0, one can control the size of the generated model.
5 CONCLUSION AND FUTURE WORK
In this paper, introduce Sample- Wise Activation Patterns and derivative, SWAP-Scort a novel training- free network evaluation metric. The proposed SWAP-Score and its regularised version show much stronger correlations with ground-truth performance than existing training-free metrics on different spaces, stack-based and cell-based for different tasks, i.e. image classification, object detection, autoencoding, and jigsaw puzzle. In addition, the regularised SWAP-Score can enable model size control during search and can further improve correlation in cell-based search spaces. When integrated with an evolutionary search algorithm as SWAP-NAS, a combination ultra-fast architecture search and highly competitive performance ImageNet, outperforming SoTA NAS methods. Our future work will extend the concept SWAP- Score other activation functions, including other piecewise linear and non-linear types like GELU.
be achieved both CIFAR- 10 and

---

Published as a conference paper at ICLR 2024
REFERENCES
Mohamed S. Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane. Zero- cost proxies for lightweight NAS. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task 2018.
and hardware. CoRR,
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search imagenet in four GPU hours: A theoretically inspired perspective. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Yaofo Chen, Yong Guo, Qi Chen, Minli Li, Wei Zeng, Yaowei Wang, and Mingkui Tan. Contrastive neural architecture search with neural architecture comparators. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 9502-9511. Computer Vision Foundation IEEE, 2021b.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR, abs/1707.088 2017.
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: eliminating unfair ad- vantages in differentiable architecture search. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV, volume 12360, pp. 465- 480. Springer, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pottern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four GPU hours. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 1761-1770. Computer Vision Foundation / IEEE, 2019a.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template net- work. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 -Nvember 2, 2019, pp. 3680-3689, IEEE, 2019b.
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li. Transnas-bench- 101: Improving transferability and generalizability of cross-task neural archi- tecture search. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 25, 2021, pp. 5251-5260. Computer Vision Foundation IEEE, 2021.
Lukasz Dudziak, Thomas Chau, Mohamed S. Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D. Lane. BRP-NAS: prediction-based NAS using gens. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
10

---

Published as a conference paper at ICLR 2024
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kris- ten Grauman, Nicolò Cesa-Bianchi and Roman Gamett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580 8589, 2018
Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, and Frank Hutter. Nas- bench-suite-zero: Accelerating research on zero cost proxies. In Thirty-sixth Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations.
Guihong Li, Yuedong Yang. Kartikeya Bhardwaj, and Radu Marculescu. Zico: Zero-shot NAS via inverse coefficient of variation on gradients. In The Eleventh International Conference on Learning Reprexentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 Pro- ceedings of Machine Learning Research, pp. 367-377, 2020.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping, 2019.
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: zero-shot NAS for high-performance image recognition. 2021 IEEF/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10- 17, 2021, pp. 337-346. IEEE, 2021. doi: 10110991CCV48922222..000
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), September 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019.
Vasco Lopes, Saeid Alirezazadeh, and Luis A Alexandre. Epe-nas: Efficient performance estima- tion without training for neural architecture search. In Artificial Neural Networks and Machine Learning-ICANI ICANN 2021: 30th International Conference Artificial Neural Networks, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part V, pp. 552-563. Springer, 2021.
Shun Lu, Jixiang Li, Jianchao Tan, Sen Yang. and Liu. Tnasp: transformer- based nas predictor with a self-evolutiin framework. Advances in Neural Information Processing Systems, 34:15125- 15137, 2021.
Shun Lu, Yu Hu, Peihao Wang. Yan Han, Jianchao Tan, Jixiang Li, Sen Yang, and Ji Liu. Pinat: a permutation invariance augmented transformer for nas predictor. Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 8957-8965, 2023.
Zhichao Lu, Kalyanmoy Deb, Erik D. Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search. In Pro- ceedings of the European Conference on Computer Vision (ECCV), volume 12346, pp. 35-51. Springer, 2020.
Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tie- Yan Liu. Semi-supervised neural architecture search. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and - Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
11

---

Published as a conference paper at ICLR 2024
Joe Mellor, Jack Turner, Amos J. Storkey, and Elliot J. Crowley. Neural architecture search with- out training. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 7588-7598. PMLR, 2021.
Jisoo Mok, Byunggook Na, Ji-Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural architecture search without training? In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 11851-11860 IEEE, 2022. doi: 10.1109/ 2VPPR52688.0222.0115
Guido Montúfar, Razvan Pascanu, KyungHyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27: Annual Conference Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2924-2932, 2014.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Johannes Fürnkranz and Thorsten Joachims (eds.), Proceedings of the 27th International Confer- ence on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pp. 807-814. Omnipress, 2010.
Xuefei Ning. Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Wang. Evaluating efficient performance estimators of neural architectures. Advances in Neural Information Processing Systems, 34: 112112226-22211 2021.
Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with piece-wise linear activations. arXiv preprint arXiv:1312.6098, 2013.
Yameng Peng, Andy Song, Vic Ciesielski, Haytham M. Fayek, and Xiaojun Chang. Pre-nas: Evo- lutionary neural architecture search with predictor. IEEE Transactions on Evolutionary Compu- tation, 27(1))26-36, 2023, doi: 10. 109/TEVC.200..3225522.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference Machine Learning (ICML), pp. 4095-4100 4104, 2018
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Ma- chine Learning Research, pp. 9902-2911, International Convention Centre, Sydney, Australia, 06-11 Aug 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for im- age classifier architecture search. In AAAI Conference on Artificial Intelligence, volume 33, pp. 4780-4789, 2019.
Pengzhen Ren, Yun Xiao, Xiaojun Chang. Poyao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Comput. Surv., 54(4):76:1-76:34 2022.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T. Kwok, and Tong Zhang. Bridging the gap between basple-based and one-shot neural architecture search with bonas. In Advances in Neural Information Processing Systems, volume 33, 2020.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. CoRR, abs/2008.097777, 2020.
Nilotpal Sinha and Kuan-War Chen. Evolving neural architecture using one shot model. In Fran- cisco Chicano and Krzysztof Krawiec (eds.), GECCO '21: Genetic and Evolutionary Computa- fion Conference, Lille, France, July 10- 10-14, 2021, pp. 910-918. ACM, 2021.
12

---

Published as a conference paper at ICLR 2024
Zhenhong Sun, Ce Ge, Junyan Wang, Ming Lin, Hesen Chen, Hao Li, and Xiuyu Sun. Entropy- driven mixed- preccison quantization for deep network design. In Advances in Neural Information Processing Systems, 2022.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 28 2818-2826, 2016.
Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information pro- cessing systems, 33:6377-6388, 2020a.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florinn Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020b.
Jack Tumnr, Elliot J Crowley, Michael O' O'Boyle, Amos Storkey, and Gavin Gray, Blockswap: Fisher- guided block substitution for network compression on budget. In International Conference on Learning Representations, 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations.
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Helen Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In Computer - 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX, volume 12374, pp. 660- 676. Springer, 2020.
Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, De- badeepta Dey, and Frank Hutter. Neural architecture search: Insights from 1000 papers. CoRR, abs/2301.08727, 2023.
Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang, Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, et al. Weight-sharing neural architecture search: A battle to shrink the optimization gap. ACM Computing Surveys (CSUR), 54(9):1-37,, 2021.
Huan Xiong. Lei Huang, Mengyang Yu, Li Liu, Fan Zhu, and Ling Shao. On the number of linear regions of convolutional neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 105 10514-0523.. PMLR, 2020.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC- DARTS: partial channel connections for memory-efficient differentiable architecture search. In International Conference on Learning Representations (ICLR), 2020.
Zhaohui Yang. Yunhe Wang. Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. CARS: continuous evolution for efficient neural architecture search. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 1826-1835. IEEE, 2020.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- 101c Towards reproducible neural architecture search. In Proceedings of the 36th Interna- tional Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97, pp. 7105-7114, 14. PMLR, 2019.
Amir R. Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In 2018 IEEE Conference Com- puter Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 3712-3722. Computer Vision Foundation IEEE Computer Society, 2018
13

---

Published as a conference paper at ICLR 2024
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 40(6): 1452-1464, 2018
Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 11393-11401. IEEE, 2020.
Hui Zhu, Zhulin An, Chuanguang Yang, Kaiqiang Xu, Erhu Zhao, and Yongjun Xu. EENA: efficient evolution of neural architecture. 2019 IEEE/CVE International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pp. 1891-1899. IEEE, 2019.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Interna- tional Conference on Learning Representations (ICLR), 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architec- tures for scalable image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
14

---

Published as a conference paper at ICLR 2024
A
MODEL SIZE CONTROL DURING ARCHITECTURE SEARCH
U.D.5, C40.1
u=0.7, Uw0.9, .w0.9
fE0)
Figure 6: Illustration of the curves of regularisation function f(@) with different μ and a.
As elucidated in Section 3.2, the regularization function f(O), defined in Equation 5, is influenced by two key parameters: and a. These parameters shape the curve depicted in Fig. 6, where determines the curve's 's central position and modulates its shape. Theoretically, models with o value proximate to will have their L values largely preserved. Conversely, a significant deviation from will result in substantial attenuation of V A smaller value of a sharpens the curve, thereby amplifying the regularization effect on models whose è values are distant from μ.. This leads two results: (1) enables control on the model size during the architecture search, and (2) it enhances the correlation of SWAP-Score in cell-based search spaces. While this section primarily focuses on the first point, the impact of varying l and a on correlation is detailed in Appendix B.
Similar to the model size control on CIFAR-10 (Section 4.3), we can also see the size control capabil- ity of regularised SWAP-SS SWAP-Score through SWAP-NAS ImageNet. Fig. 7 shows the size distribution of cell networks searched from DARTS space for ImageNet. These networks are in the range of 25 KB to 27.5 KB. We still set and to identical values and gradually increase them from 25 to 27.5 with an interval of 0.5. Fig. is the relation between ji and the size of these fully stacked ImageNet networks found by SWAP-NAS with that value. Similar to the previous experiments each lr value repeats the search times. Random jitter is also introduced here. But there are much higher noticeable variations in size at each μ compared to that for CIFAR-10 (Fig. (b)). Neverthe- less, it can still be shown that in general, model size decreases with a reduced μ. Adjusting would have direct impact on the size of the generated model, even for complex tasks like ImageNet.
CIFAR- 10,
160
140
120
Frequency
100
40
20
Model size (KB)
Figure 7: Model size distribution of 1000 cell networks sampled from DARTS space for ImageNet.
15

---

Published as a conference paper at ICLR 2024
6.8
25.0
25.5
26.5
27.0
Model size (MB)
6.4
6.0
5.8
5.6
25.0
25.5
26.0
26.5
27.0
H
Figure 8: Illustration of the model size control SWAP-NAS on ImageNet. The architecture search repeats 5 times at each j value. Random jitter used to reduce overlaps.
B IMPACT OF VARYING AND a TO THE CORRELATION
Firstly we demonstrate the impact of different and a on NAS-Bench-10 space (Ying et al., 2019). Following a similar procedure in SAS-Bench-Suite-Zern (Krishnakumar al., 2022), utilize different random seeds to form 6 groups, with each group comprising 1000 architectures randomly sampled from the NAS-Bench-10 101 space. One of the groups (Group is used to approximate the distribution of model sizes in the NAS-Bench-101 101 space, not participating in the subsequent experiments. The distribution histogram obtained from Group is shown in Fig. 9. The range of model size in this group is 0.3 to 31 megabytes (MB). Most of the models are in 0.3 MB to MB intervals. Leveraging this information, we can better see the impact of μ and a on the other five groups of sampled architectures (Table 4).
Table shows different combinations of μ and values, and the corresponding Spearman's is Rank Conrelation Coefficient between regularised SWAP-Scone and the ground-truth performance of the networks for the five groups, Group to Group 5. There are four blocks in the table. The first block contains only row, which shows the results without regularisation, in other words, the results from SWAP-Score. The second block shows the results of assigning identical values to μ and a, ranging from 0.3 40, the same range of model size in MB, shown Fig. The third block shows the results of varying while fixing μ. The fixed value, 40, is chosen because it leads to the highest correlation the second block, where μ a. The last block in Table shows the results of adjusting μ while fixing a. The fixed value, 30, is chosen because it gets the highest correlation in the third block.
From the results shown in Table 4, we can see that the correlation between regularised SWAP- and ground-truth performance mainly affected by the value of μl, as only minor changes occur in the correlations from = 30 to a 10, when we fix μ at 40 in the third block. On the contrary, reducing μ leads to a significant drop correlation in the last block. This observation consistent across all five groups. It is explainable μ defines the centre position of the regularisation curve and directly determines how the regularisation curve covers the size distribution. Having said that, o not insignificant. A poor choice of e.g. j 40, 0.3, can lead to bad correlations (<c 0.05 different way. A small leads to sharp curve, which
Table 4). The impact of a on search narrows down the coverage of the regularisation function to small area, meaning architectures
16

---

Published as a conference paper at ICLR 2024
600
Frequency
500
200
10
15
30
Model size (MB)
Figure 9: Histogram of model sizes in sampled in one of the six groups.
space, based on 1000 CIFAR-10 networks
Group
Group 2
Group 3
Group 4
Group 5
No regularisation
N/A
N/A
0.46
0.49
0.45
0.45
0.44
0.3
0.3
-0.77
-0.78
-0.77
-0.75
-0.75
5
0.03
0.08
0.07
0.05
0.04
10
0.61
0.63
0.64
0.6
0.64
15
15
0.74
0.75
0.73
0.73
0.73
11 =
20
0.72
0.75
0.75
0.74
0.73
25
25
0.76
0.76
0.76
0.75
0.74
30
0.76
0.77
0.76
0.74
0.74
35
35
0.76
0.76
0.76
0.75
0.74
40
40
0.76
0.76
0.77
0.75
0.74
40
30
0.76
0.76
0.76
0.74
0.75
Varying with fixed μ
40
20
0.76
0.76
0.76
0.74
0.74
40
10
0.75
0.74
0.73
0.73
0.73
40
0.3
0.04
0.04
0.03
0.03
0.03
20
30
0.75
0.76
0.74
0.73
0.74
Varying with a fixed
10
30
0.64
0.64
0.63
0.63
0.66
30
0.04
0.08
0.06
0.04
0.06
0.3
30
-0.77
-0.78
-0.77
-0.74
-0.75
Table 4: Impact of and illustrated on NAS-Bench-109 101 architectures. The values under Group 1-5 represent the Spearman's 's Rank Correlation Coefficients between SWAP-Score and the ground-truth performance of these networks. Each group contains 1000 architectures sampled from 101 space by different random seeds. "N/A" indicates correlations without regularisation.
outside of that size range will heavily penalised, as their regularised SWAP-Scores after applying the regularisation function will be very low. Based on the study, recommendation choosing μi and that μ and a can set large when the target is finding top-performing architectures.
Following the above study on NAS-Bench-101, we secondly demonstrate the impact of different and o on NAS- Bench-201 space. Six groups of architectures are sampled. One of them is used to approximate the distribution of model sizes in the NAS-Bench-201 space. The corresponding histogram shown in Fig. 10. As can be seen in this figure, the range of model size here is 0.1 to 1.5 megabytes (MB). Accordingly, the i values in Table 5 in the range of 0.1 to 1.5.
17

---

Published as a conference paper at ICLR 2024
230
Frequency
150
100
0.2
0.4
0.6
0.0
1.2
1.4
1.6
Model size (MB)
Figure 10: Histogram of model sizes in NAS- space, based on 1000 CIFAR- 10 networks sampled in one of the six groups.
Group
Group 2
Group
Group 4
Group
No regularisation
N/A
N/A
0.81
0.79
0.75
0.77
0.75
0.1
0.1
0.05
0.04
0.03
0.03
0.03
0.7
0.7
0.83
0.82
0.82
0.84
0.81
1.5
1.5
0.89
0.87
0.87
0.89
0.87
Varying
1.5
0.87
0.84
0.86
0,86
0.84
0.74
with a fixed
1.5
0.4
0.83
0.78
0.83
0.79
0.79
1.5
0.1
0.54
0.54
0.53
0.54
0.51
Varying with fixed
0.6
0.7
0.82
0.82
0.82
0.83
0.81
0.3
0.7
0.51
0.49
0.51
0.54
0.5
0.1
0.7
0.23
0.21
0.23
0.24
0.21
Table 5: Impact of μ and illustrated on NAS-Bench-200 architectures. The values under Group 1-5 represent Spearman's 's Rank Correlation Coefficients between SWAP-Score and the ground-truth performance of these networks. Each group contains 1000 architectures sampled from BAS-Bench- 201 space by different random seeds. "N/A" indicates correlations without regularisation.
Similar to Table 4, Table 5 also has four blocks, showing four scenarios of study and a. Fewer combinations are presented as the general trend here is the same as that appears in NAS-Bench- 101.
The third part of the study in Section B is the impact of different f and Similar to the previous two parts, six groups of architectures are sampled, with one for showing the distribution of model sizes in the NAS-Bench-30 space. Fig. 11 is the distribution histogram, where the size range can be seen as 1.0 to 1.8 megabytes (MB). With the same style as in Table and Table 5, Table lists the results of different j and a in four blocks. Again, the general trend here is the same as that in
NASS-Bench-30 space.
101 and NAS-Bench-22 Bench-201.
18

---

Published as a conference paper at ICLR 2024
Frequency
150
Model size (MB)
Figure 11: Histogram of model sizes in NAS-Bench-300 space, based on 1000 CIFAR- networks sampled in one of the six groups
/
Group
Group 2
Group
Group 4
Group
No regularisation
N/A
N/A
0.57
0.58
0.57
0.57
0.54
1.0
1.0
0.57
0.58
0,57
0.57
0,55
/ =
1.5
1.5
0.63
0.61
0.61
0.61
0.60
1.8
1.8
0.63
0.63
0.63
0.63
0.62
Varying
1,8
1.5
0.63
0.63
0.62
0.63
0.62
with fixed
1.8
1.3
0.62
0.62
0.61
0.61
0.62
1.8
1.0
0.62
0.60
0.60
0.61
0.60
Varying
1.3
1.5
0.62
0.62
0.60
0.60
0.60
with fixed ø
1.2
1.5
0.61
0.61
0.61
0.59
0.60
1.0
1.5
0.60
0.59
0.59
0.59
0.59
Table 6: Impact of μ and illustrated on NAS-Bench-301 architectures. The values under Group 1-5 represent Spearman's 's Rank Correlation Coefficients between SWAP-Score and the ground-truth performance of these networks. Each group contains 1000 architectures sampled from NAS-Bench- 301 space by different random seeds. "N/A" indicates correlations without regularisation.
C EVOLUTIONARY SEARCH ALGORITHM
In this section of the Appendix, we elaborate on the evolutionary search algorithm employed in SWAP-NAS. SWAP-NAS adopts cell-based search space, similar DARTS-related works, such as Chen et al. (2019); Chu et al. (2020); Sinha & Chen (2021); Cai et al. (2018); Chen et al. (2021a); Sun al. (2022). In terms of the search algorithm, SWAP-NAS uses evolution-basee based search where each step of the search is performed on a population of candidate networks rather than an individual approach allows for broader coverage of the search space, thereby
network. This population-bast increasing the likelihood of finding nigh-performance architectures. While evolutionary search al- gorithms are generally resource-ings intensive due to the need for multiple evaluations, SWAP-NAS mit- igates this drawback by capitalizing on the low computational cost of SWAP-Score. As result, aim to deliver a NAS algorithm that both efficient and effective, achieving high accuracy without incurring prohibitive computational costs.
The details are presented in two sections. The first section, C.I. explains the cell-based search space, while the evolutionary aspect is explained in the second section, C.2.
19

---

Published as a conference paper at ICLR 2024
3x3 Convolution
0
2
3
Convolution
Average Pooling
0
Skip Connection
Cell network
Matrix encoding
Figure 12:
Illustration of cell network and matrix encoding. Grey circles represent the nodes inside the cell. Arrow lines indicate the flow while network operations are in different colours.
C.1 ARCHITECTURE ENCODING OF CELL-BASED SEARCH SPACE
Fig. 12 illustrates the cell-based network representation (Liu et al., 2019; Shi et al., 2020; Zoph al., 2018). This cell network is widely used in NAS studies (Dong & Yang, 2020; Liu al., 2019; Siems et al., 2020; Ying et al., 2019). With this representation, the search algorithm only needs to focus on finding good micro-structure for one cell, which shallow network. The final model after the search can be easily reconstructed by stacking this cell network together repetitively. The depth of the stack is determined by the difficulty of the task. As shown the right of Fig. 12, a cell is encoded as an adjacency matrix, on which each number represents type of connection, convolution, is
convolution, 2 is a connection. The matrix in the figure is x 4 since the example cell network has four nodes. A zero the matrix means no connection or is not applicable. This matrix is an upper triangular matrix it represents the directed acyclic graph (DAG). With this matrix representation, the subsequent cvolutionary search can be conveniently performed by simply manipulating the matrix, for example alternating the type of connection by changing the number at the particular entry connecting to different node by shifting the position of the number that represents this connection.
for a
average pooling, for a skip
Algorithm SWAP-NAS
Require: Population size P, Search cycle C, Sample size S, TF-metric SWAP, mutation.time
1: population +
2: while population P do
3: model.arch + RandomGenneattNeeeuuen
4: model. + SWAP WWAPPmodel.archh
5: Add model to the population
6: end while
7: for c= 1..,... do
8: candidates S random samples of population
9: parent <- best in candidates OR best from crossover between the best and the second best candidates
10:
while inntation_timen not reached do child + M (paren()aeent) child.scon
11:
12:
SWWP (WAP(child.arch)
13: end while
Add the best chvidd the population
15: Remove the worst from the population
end for
C.2 EVOLUTIONARY SEARCH IN SWAP-NAS
As an effective search paradigm, evolutionary search utilised in large number NAS methods, showing good search performance and flexibility (Lu et al., 2020; Peng et al., 2023; Real et al., 2017: 2019; Yang et al., 2020). For this very reason, SWAP-NAS is also based on an evolutionary search, but SWAP-Scome the detailed steps in SWAP-NAS. The evolutionary search component of SWAP-NAS slightly different from the existing evolutionary search methods used for NAS. The key distinctions the way of producing offspring networks and how the population updated after each search cycle. In SWAP-NAS. tournament-style strategy used sample offspring networks. Half of the networks
is not restricted to certain type of search algorithm. Algorithm is
20

---

Published as a conference paper at ICLR 2024
Parent architecture
Offspring architecture
0 0 0 3 2
0 1 2 0
0
0
3 2
0
0
0
4
0 0 0
Matrix encoding
Matrix encoding
(a) Mutating network operation
Parent architecture
Offspring architecture
1 0 1
3 2
4
0
0 0
Matrix encoding
Matrix encoding
(b) Mutating network connectivity
Figure 13: Illustration of the mutation operators. (a) demonstrates offspring being produced by
mutating the operation of the parent architecture. (b) demonstrates offspring being produced by mutating the connectivity the parent architecture.
are randomly selected from the population during each search cycle (Step 8). Then, in a random fashion, SWAP-NAS decides whether to perform the crossover operation on the selected network or directly use the selected network as the parent (Step 9). Therefore the parent will be either the best individual from the sampled networks network produced by the crossover between the best and the second best networks (Step 9).
In SWAP-NAS, the majority of offspring networks are generated by mutation (Step 11). There two types of mutation, operation mutation, and connectivity mutation. They are illustrated in Fig. 13 respectively. Mutating operation is simply changing number in the cell network matrix, shown in Fig. 13 (a), changing the connection from node to node 3 from 3 x 3 convolution (type number 1, red) to 1 convolution (type number 2, green). Mutating connectivity is shifting connection to different position, as shown in Fig. 13 (b), moving the 1 x convolution connection from node to node 3 to node I node 4. By randomly performing these two types of mutation, SWAP-NAS can stochastically explore possible new architectures from a given parent during each search cycle.
As mentioned early, SWAP-NAS does incorporate crossover, which is a key operation, other than mutation, in evolutionary search. The use of crossover here is to avoid the search being trapped in local optimal. The application of crossover is random as shown Step 9 of Algorithm 1. Fig. 14 illustrates the crossover operation which is slightly different from the crossover often seen in evo- lutionary search. The crossover exchanges "genetic materials" between the two selected networks, c.g. entries of the two matrices. In the example of Fig. 14, crossover swaps the incoming connections to node between the two networks, e.g. exchanging the entries in (1,3) and (2, 3). The two newly generated offspring networks will be evaluated using SWAP-Score. The one that scored higher will become the new parent for mutation.
Note SWAP-Score is utilised for evaluation in three places. Other than the aforementioned evalua- tion during the crossover, it appears in Step 4 and Step 11 in Algorithm 1, for evaluating the initial population and the new network generated by mutation. SWAP-Score is applied the architecture,
21

---

Published as a conference paper at ICLR 2024
Candidate
Candidate 2
1 0
3 0
0 3 1
1 2
0 4
0 2
0 0
Matrix encoding
Matrix encoding
0 4
Choose the better one as parent
0 4 3 0
0 0
Figure 14: Illustration crossover operator. Candidates are the best and second-best networks from the sample. Two networks generated by swapping components of the candidates. The better one will become the parent network for the subsequent mutation.
e.g. mocel.arch child.arch. The generated score saved property of the network, e.g. model.score or child.score.
Steps 14 & 15 of Algorithm are the population updating mechanism of SWAP-NAS. Unlike the aging evolution in AmoebaNet (Real et al., 2019) which removes the oldest individual from the population, SWAP-NAS removes the worst. Theoretically, aging evolution could lead to higher diversity and better exploration of the search space. However, the elitism approach can converge faster, hence reducing the computational cost on the search algorithm side.
22

---

Published as a conference paper at ICLR 2024
D CORRELATION OF METRICS BY INPUTS OF DIFFERENT DIMENSIONS
The correlation between these metrics and the validation accuracies of the networks is measured by Spearman's rank correlation coefficient. The full visualisation of results shown in Fig. 15. The metric based on standard activation patterns drops dramatically when the input dimension increases. This aligns with the observation from Table Both SWAP Score and regularised SWAP- show a strong and consistent correlation with rising input dimensions. In particular, regularised SWAP Score outperforms the other two. regardless of the input size. When using the original dimension of CIFAR-10 10, 32 x 32, regularised SWAP-Scom shows a strong correlation, 0.93.
0.55
0.85
Spearman's rank conslation coefficient
0.50
0.75
0.70
0.68
0.00
0.58
0.50
0.45
0:10
0.30
0.30
Standard Actvation Patlams
0.25
SWAA--Score
Rag. SWAP-Soore
12112
10r10
2h21
2dx2
20277
30ca0
32x52
Input dimension
Figure 15: Spearman's 's coefficient between three metrics and the validation accuracies.
E VISUALISATION OF CORRELATION BETWEEN SWAP-SCORES AND NETWORKS' GROUND-TRUTH PERFORMANCE
Figures 16 and 17 demonstrate the correlation between the SNAP-ScoreRRegullriied SWAP-Score and the ground-truth performance of networks across various search spaces and tasks. In these figures, each dot represents distinct neural network. The visualisations effectively demonstrate strong correlation between the SWAP-Score and the ground-truth performance across the majority of search spaces and tasks. Furthermore, the application of the regularisation function results a more concentrated distribution of dots, indicating an enhanced correlation with the networks' ground-truth performance.
Spes Yaa Coefiic 0.99
Spenmoon COP 3
Spenman Conticierr 3.82
Crouee South Femerranc
50
80
20
56300 62000
66000
20000
40000
60000
20000
40000
60000
(a)
(b)
(c)
23

---

Published as a conference paper at ICLR 2024
SADSESCHOES LIMROO CORTONNE
TRONSEOSCHIDL AUTORMCC05 cotfiieee
94.0
0.7
02.5
03.0
20
92.5
Creunc
02.0
9.4
10
91.5
91.0
0.2
10000 20000
30000
42000 50000
0.8
3.0
3.2
1.4
1.6
1.0
62500
540000
64500
65000
65500
Score
Reg.
TRANSBENCHETL MADRO JOSAM
WHMERSSCHOOLL CLASO OSRECT
TRAMBSSCHHEL MADIO CLASO SCERE
Ground Femormatc
54
12
52
86
0.0
0.2
0.4
0.6
0.8
1.0
1.2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
3.0
SWAr
TRWISBEMCISTL MIORO AUTECNCODERR
"RANGENNHIOLL MIDO JOSAA
TRAEBDS VICTO oupe
0.5
Ground Thun Nutormate
022
0.1
90000
.0000
#O000
0.23
0.50
0.75
1.00
1.25
1.50
0.5
3.0
3.5
2.0
2.5
SWAR--Scm
(k)
TRANGERRCHOO NEDO CLACK SCEME
1:3
Treadiiu Nerbo
30
10
0.5
1.0
1.5
2.0
2.5
14b
(m)
Figure 16: Visualisation of correlation between SWAP-Score and networks' ground-truth perfor- mance, for each search space and task combination. Colors indicate the search spaces.
24

---

Published as a conference paper at ICLR 2024
AASRERCHOOI CIRATE
RASSENCION CRON1O
HAGREMO-GEI FOR113
Spearaan Conffcrrt 3.16
Speerman Conficunt 3.09
Sosummn Conticeet
70
Crnuectuuus Perfrrercs
Grunnhuum Pertomercs
-0
80
10
20
20000
40000
00000
50
150
200
250
100
150
200
May SMaa Sure
Hay 5M60--000
S0A-5-50000 Sure
(c)
THANSEEN AUTDESCOUER Spentnar ConfiNeT
SpnarIin Conficert- 0.60
Sprnorio Conticen
54.55
$4.0
03.5
0.00
10
03.0
20
52.5
Grurgbuu
52.0
10
90.5
0.2
91.0
150
200
100
1.0
2.0
Pag. SWin
SWMR Score
Ang SWAR Stort
le7
Spnamann
Speurns Ceefic est
Graad ferforanne
Crainhhth femoraacs
E
0
3
%
R
82
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
167
(kg. SMAA
107
(h)
0.G
Spoarnan Core Youn BAI
100
Spuarnan wad.ooodde.....o.o.. 2.43
SoNuRTRa Coclliant 2.40
Gound Neformana
20
Goundhut Nutormnnc
04
0.3
02
1.0
1.0
400000
600D0n
105
105
(k)
Spearman Confiearr 0.00
50
80
00
(m)
Figure 17: Visualisation of correlation between regularised SWAP-Score and networks' ground- truth performance, for each search space and task combination. Colors indicate the search spaces.
25

---

Published as a conference paper at ICLR 2024
F ARCHITECTURES FOUND BY SWAP-NAS ON DARTS SEARCH SPACE
Figures 18 to 21 demonstrate the neural architectures discovered by SWAP-NAS under varying model size constraints for the CIFAR-IO and ImageNet datasets. These figures effectively demon- strate the capability of the regularised SWAP-Score to control model size within the context of NAS. Additionally, they highlight the trend of increasing topological complexity in the architectures the model size grows.
sep conv 3x3
skip counect
C_ (k-2)
_onv__x333 3x3
(k)
cop_conv_3xX
avg pool 3x3
3
sep_conv_3ke
skip connect
(k-1)
avg_pool__ _3x3
Figure 18: Cell architecture found by SWAP-NAS on CIFAR- 10 dataset with model size 3.06 MB.
sep_conv_3xx
(k-2)
sep cOnv 3x3
cop. conv_3x3 3x3
2
sep cony 3x3
c_
dil conv..... 3xx3
(_(k-)
skip connect
3
skip ronnect
Figure 19: Cell architecture found by SWAP-NAS on CIFAR- dataset with model size 3.48 MB.
skip cumnec
sep
3x3
(k-2)
Kep- copV 3x3
scp_ sp__cow__30
[_(
sep exp_ooov___0 3x3
sep_ooov_Bx5
sep. conn 3x3
(_-k)
Figure 20: Cell architecture found by SWAP-NAS on CIFAR-10 10 dataset with model size 4.3 MB.
sup_ite 3x3
sep canv: 3x3
c [k-2]
sep pany
sen cony
skip
c (k-1)
SP1 5x5
di CODV 5x5
C (k)
Figure 21: Cell architecture found by SWAP-NAS on ImageNet dataset with model size 5.8 MB.
26


Reviews:
[{"review_id": "ULi7pXE2ht", "reviewer": "Reviewer_pab6", "summary": "This paper proposes a new approach called SWAP-NAS for ultra-fast Neural Architecture Search (NAS) based on Sample-Wise Activation Patterns. SWAP-NAS outperforms existing metrics on various search spaces and tasks, achieving competitive performance on CIFAR-10 and ImageNet in just a few minutes of GPU time. The paper also discusses the limitations of existing training-free metrics in NAS and how regularisation can enhance the SWAP-Score and enable model size control during the search.", "questions": "- How would SWAP-NAS integrate with pruning-based search method such as the one used in TE-NAS?\n- How to intergate regularised SWAP-NAS with FLOPs or Latency budget constrain?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n- The proposed SWAP-Score and regularised SWAP-Score show much stronger correlations with ground-truth performance than existing training-free metrics on different spaces and tasks.\n- The regularised SWAP-Score can enable model size control during search and can further improve correlation in cell-based search spaces.\n- When integrated with an evolutionary search algorithm as SWAP-NAS, a combination of ultra-fast architecture search and highly competitive performance can be achieved on both CIFAR-10 and ImageNet, outperforming SoTA NAS methods.", "weaknesses": "- The paper only give numerical numbers on NAS Search Space, and did not show or analyze any searched neural architectures, to back the claim that Sample-Wise Activation Patterns would measure the network’s expressivity more accurately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TFJvs4Rosl", "reviewer": "Reviewer_AWYQ", "summary": "This paper proposes a training-free metric called SWAP-score, which measures the expressivity of the networks. This metric can be integrated with the evolutionary NAS method to obtain competitive performance on several datasets with the least search cost.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed metric is efficient and has a good correlation with the ground truth performance across various search spaces and tasks, beating all existing training-free metrics in public benchmarks. \n2. The metric combined with the evolutionary search method can achieve good performance with extremely small search costs.", "weaknesses": "1. Missing references: \nPINAT: A Permutation INvariance Augmented Transformer for NAS Predictor  AAAI 2023\nTNASP: A Transformer-based NAS Predictor with a Self-evolution Framework  NeurIPS 2021\nPlease cite and compare with them in Table 1 and Table 2.\n2. The method is proposed to be used in the ReLU activation function-based network, which is not been proven to work well on other nonlinear activation function-based networks or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hYzpQmbkSH", "reviewer": "Reviewer_QHog", "summary": "This paper proposes a novel training-free metric called SWAP-Score, based on Sample-Wise Activation Patterns, for zero-shot Neural Architecture Search (NAS). The SWAP-Score measures the expressivity of networks over a batch of input samples and outperforms 15 existing training-free metrics on various search spaces and tasks. The authors also introduce regularisation to the SWAP-Score for model size control during the search", "questions": "1. How are `sigma` and `mu` related to model parameters, and what mechanisms allow their automatic configuration in SWAP-NAS?\n\n2. In Figure 4, what causes TranBench101 to exhibit inferior performance compared to NAS-Bench-x01? Are there specific factors that contribute to this observed difference?\n\n3. The paper claims versatility of SWAP-NAS across various tasks, including object detection, autoencoding, and the jigsaw puzzle. However, the primary focus seems to be on image classification results, such as ImageNet and CIFAR. Can the authors shed light on the reason for this specific focus? Furthermore, are there any plans or intentions to investigate SWAP's performance in the other mentioned domains in future work?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Robust Experimental Results: Upon meticulously examining and executing the code provided in the supplementary files, I affirmed the robustness of the experimental results by myself. The effectiveness of the proposed SWAP-NAS has been convincingly validated through my independent verification.\n\n- Comprehensive Experimental Validation: The authors have conducted extensive experiments across multiple NAS-Bench datasets, demonstrating the clear superiority of SWAP-NAS. This broad-spectrum analysis significantly strengthens the validity and generalizability of the proposed approach.\n\n- Clarity and Accessibility of Presentation: The paper is exceptionally well-crafted, featuring lucid and coherent exposition. The methodology is articulated in a manner that is not only easy to comprehend but also straightforward to follow. This enhances the paper's accessibility to a broader audience within the field.", "weaknesses": "- Limited Applicability to ReLU-based Networks: One limitation of this approach is its dependency on neural networks using ReLU activations. While ReLU is widely used, it represents a subset of possible activation functions, which narrows the scope of application and might not encompass the full diversity of network architectures.\n\n- Similarity to NWOT Format: It's worth noting that the format of the proposed zero-cost proxy shares similarities with existing methods like NWOT. While not necessarily a weakness, this overlap should be acknowledged and explored further to understand the distinctions and innovations brought by SWAP-NAS.\n\n- Unclear Impact of Dimension: The influence of dimensionality remains somewhat obscure in the paper. It's important to consider that neural networks, especially in shallow layers, may exhibit varying feature map sizes. A more detailed analysis of how these differences affect SWAP-NAS would enhance the completeness of the study.\n\n- Absence of Correlation Visualization: The paper lacks visualizations that illustrate the correlation between SWAP and performance, which could provide valuable insights into the method's behavior and its implications for optimization. The inclusion of such visual aids would strengthen the paper's argument and help readers grasp the methodology's practical significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mF4nAScljg", "reviewer": "Reviewer_6s7y", "summary": "In this paper, the authors introduced a novel training-free network evaluation metric based on Sample-Wise Activation Patterns. The proposed SWAP-Score and regularised SWAP-Score show stronger correlations with ground-truth performance than 15 existing training-free metrics on different spaces, stack-based and cell-based, for different tasks. The regularised SWAP-Score can enable model size\ncontrol during search and can further improve correlation in cell-based search spaces. It is integrated with an evolutionary search algorithm as SWAP-NAS.", "questions": "Can this method apply to transformer-based NAS?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The training-free NAS method shows better performance than previous SOTA and 5 NAS benchmarks. The searching time is reasonably low. The proposed metric looks very promising for model expressivity.", "weaknesses": "The method assumes ReLU as the activation function, which is a big limitation as more and more transformer based networks use GELU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)