NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance
Authors: Raphael T. Husistein
Markus Reiher
Marco Eckhoff
Venue: ICLR 2025 Poster (2025)
Keywords: Network Expressivity by Activation Rank (NEAR)
Effective Rank
Neural Architecture Search
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Artificial neural networks have been shown to be state-of-the-art machine learning models in a wide variety of applications, including natural language processing and image recognition. However, building a performant neural network is a laborious task and requires substantial computing power. Neural Architecture Search (NAS) addresses this issue by an automatic selection of the optimal network from a set of potential candidates. While many NAS methods still require training of (some) neural networks, zero-cost proxies promise to identify the optimal network without training. In this work, we propose the zero-cost proxy Network Expressivity by Activation Rank (NEAR). It is based on the effective rank of the pre- and post-activation matrix, i.e., the values of a neural network layer before and after applying its activation function. We demonstrate the cutting-edge correlation between this network score and the model accuracy on NAS-Bench-101 and NATS-Bench-SSS/TSS. In addition, we present a simple approach to estimate the optimal layer sizes in multi-layer perceptrons. Furthermore, we show that this score can be utilized to select hyperparameters such as the activation function and the neural network weight initialization scheme.

Published as a conference paper at ICLR 2025
NEAR: A TRAINING-FREE PRE-ESTIMATOR OF MA- CHINE LEARNING MODEL PERFORMANCE
Raphael T. Husistein, Markus Reiher, and Marco Eckhoff
Department of Chemistry and Applied Biosciences, ETH Zurich,
Mudimir-Prelogs Frelog-Wegg 2, 8093 Zurich, Switzerland. (raphahus, mreiher, fckhoffm]ee (aeboo.fmmeeth...oo
ABSTRACT
Artificial neural networks have been shown to be state-of-the-art machine learning models in a wide variety of applications, including natural language processing and image recognition. However, building a performant neural network is a labo- rious task and requires substantial computing power. Neural Architecture Search (NAS) addresses this issue by automatic selection of the optimal network from a set of potential candidates. While many NAS methods still require training of (some) neural networks, zero-cost proxies promise identify the optimal network without training. In this work, we propose the zero-cost proxy Network Expres- sivity by Activation Rank (NEAR). It based on the effective rank of the pre- and post-activation matrix, i.e., the values of a neural network layer before and after applying its activation function. We demonstrate the cutting-edee correla- tion between this network score and the model accuracy on NAS- NAS-Bench- 101 and NATS-Bench- Bench-SSS/TSS, In addition, we present simple approach to estimate the optimal layer sizes in multi-layer perceptrons. Furthermore, we show that this score can be utilized to select hyperparameters such as the activation function and the neural network weight initialization scheme.
INTRODUCTION
Originally inspired by the structure and function of biological neural networks, artificial neural net- works are now key component in machine learning (Bishop 2006, Russell & Norvig 2021}. They are applied in many tasks such as natural language processing, speech recognition, and even protein folding (Vaswani al.] 2017 Baevski et al., 2020 Jumper et al. 2021). Artificial neural networks are built by layers of interconnected neurons, which receive inputs, apply a, typically non-linear, activation function, and pass on the result (see Figure A. By tuning the weights the individual inputs to neuron, neural network can learn functional relations from sample data to perform pre- dictions without explicit program instructions. Deep learning is obtained more than one layer is hidden between the first input and last output layer of the neural network. In theory, already a neural network with a single hidden layer and enough neurons is capable of approximating any continuous function up arbitrary precision (Hornik et al. 1989; Cybenko
1989
Hornik 1991).
However, the choice of the model size, i.e., the number of hidden layers and the number of neurons per layer, has huge impact the performance, training time, and computational demand (Tan & Le 2019). A too small network is unable to capture a complex relation in dataset resulting in underfitting. Conversely. a too large model lacks efficiency and carries the risk of overfitting and poor generalization. Determining the optimal model size that strikes balance between accuracy and efficiency remains an important challenge (Ren et al. (02T). In common practise, various models of different sizes are trained and their final performances are compared. This manual process is time- consuming and often leads to sub-optimal model architectures. The goal of Neural Architecture Search (NAS) is to eliminate this manual process and autonomously identify suitable architectures (Elsken et al. 2019). Commonly applied techniques for NAS are based on reinforcement learning, evolutionary algorithms,
gradient-bart based optimization (Zoph & Le 2017 Real et al. 2017 Liu Zero-cost proxies
ctt al. 2019). However, these methods are still time- and resource-intensie have been proposed to improve the efficiency of NAS methods (Mellor al. 2021 Abdelfattah

---

Published as a conference paper at ICLR 2025
et al. 2021: Krishnakumar al. 2022). They exploit properties of the untrained network that correlated with the final accuracy, bypassing the costly training process. In this way, they promise speed-ups of several orders of magnitude compared to traditional NAS algorithms (Abdelfattah et al. 2021].
The performance of previously reported zero-cost proxies is highly dependent the underlying search space (Ning et af.] 2021]. Furthermore, many of them are unable to demonstrate consis- tently higher correlation with the final accuracy than the trivial baseline given by the number of parameters (#Params) (Nig et al. (02T). In addition, these proxies require a search space of known architectures. While the latter not severe limitation
applications such as computer vision, in many other applications such search spaces do not exist. An example is a so-called machine learning potential (Behler Parrinello, 2007; Behler 202T applied in chemistry and materials sci- ence, which employs neural networks to predict the energy of chemical system as function of the molecular structure. In this work, we propose the zero-cost proxy Network Expressivity by Ac- tivation Rank (NEAR) that can be used in conjunction with a search space, but also provides the possibility to estimate the optimal layer size for multi-layer perceptrons without a search space.
Besides the network architecture, the activation function and weight initialization scheme have neural network (LeCun ct al. 1998; Glorot & Bengio 2010). A variety of techniques have been developed to select such hyperparameters, including methods based on Bayesian optimization and evolutionary algorithms (Snoek al. 2012 Lorenzo et al., (017). However, these techniques still require expensive training of multiple networks, and the zero-cost proxies reported so far have not been applied to the selection of the activation function and the weight initialization scheme. A zero-cost proxy that can be applied not only to find optimal architectures, but also to choose the activation function and the weight initialization scheme, would allow one to further reduce the computational burden of constructing a performant artificial neural network. Here, we therefore evaluate the suitability of our zero-cost proxy NEAR for this purpose.
significant impact on the training dynamics and the final performance
In summary, we make the following three key contributions: First, we propose the zero-cost proxy NEAR and test its performance across several search spaces and datasets. Second, we introduce a simple method to estimate the optimal layer size of multi-layer perceptrons and show its effective- ness for machine learning potentials. Third, we empirically demonstrate that our zero-cost proxy NEAR can support the selection of an activation function and suitable weight initialization scheme.
2 RELATED WORK
Zero-cost NAS proxies are designed to identify optimal neural network architectures without requir- ing training. Some of the earliest proxies such as Single-Shot Network Praning Based on Connec- tion Sensitivity (SNIP), Gradient Signal Preservation (GraSP), Fisher, and Synaptic Flow (SynFlow) have been adapted from pruning techniques and are based on gradients of the neural network (Ab- delfattah et al. 2021). While SNIP assigns scores to individual parameters by approximating the change in loss when the parameter is removed (Lee et al.) 2019). the scores of GraSP are based on an approximation of the change in the gradient norm (Wang et al. 2020). The Fisher proxy estimates the importance of an activation by calculating its contribution to the Toss (Turner et al.) 2020). and SynFlow assigns a score to each parameter indicating how much contributes to the information flow in the network (Tanaka et al. 2020). A proxy that has not been adapted from the pruning litera- ture is Grad norm, which is defined as the Euclidean norm of the gradient (Abdelfattah al. (02T). SNIP, GraSP, SynFlow, and Gram nnom are calculated for each individual parameter, whereas Fisher calculated for channels within convolution layer. To obtain a score for the entire network, the values assigned to the parameters or channels are summed. Zero-Shot NAS via Inverse Coefficient of Variation on Gradients (ZiCo) (Li et al., (023) is another proxy that exploits gradient information, linking the mean and the variance of the gradient to the convergence rate and the capacity of the neural network. With the exception of SynFlow, all the aforrmmentooedd proxies require the pres- ence of target output, so-called labels. Therefore, they can only be applied to a limited extent in unsupervised learning for categorization tasks.
Some zero-cost proxies exist that employ properties of the untrained network that do not require backpropagation and hence work without labels. These include Neural Architecture Search Without Training (NASWOT) (Mellor al. 2021), Zen-Score (Lin al. 2021), Sample- Wise Activation
2

---

Published as a conference paper at ICLR 2025
Patterns (swap) Peng et al. 2024). Regularized Sample- Wise Activation Patterns (reg- (Peng et al. 2024). and Minimum Eigenvalue of Correlation (MeCoopt) (Jiang et al. 2023). To certain ex- tent, these proxies can be related to theoretical considerations regarding, for example, the functions that can be represented by a particular neural network or the convergence rate during training. A
proxy that combines information from gradients with the output of the untrained network and works without labels is Training-Fre Free Neural Architecture Search (TE-NAS) (Chen et al.) (222I).
All these proxies assess the performance of a network based on the output of individual untrained layers. However, most of them, namely NASWOT, Zen-Score, swap, reg_swap, and TE-NAS, can only be applied if the Rectified Linear Unit (ReLU) (Nair & Hinton, (010)) activation function is employed. Our zero-cost proxy NEAR utilizes the output of individual layers before and after the application of the activation function for sample inputs without constraints on the type of activation function. Therefore, its applicability is more general.
3 METHODS
3.1 MULTI-LAYER PERCEPTRON
An artificial neural network with the architecture described in the introduction (see Figure A.I is also called multi-layer perceptron. A multi-layer perceptron with L layers can be expressed
F(x) (a WL) (a WL.1)
(ar W2) (a W (x)(x) =
where x and y are the input and output features, is an activation function, and W, E R are the weights of layer I. The weights can also include so-called bias weights, which are added to the output of the neuron without being multiplied by any input value. Hence, they effectively shift the activation function input of the neuron.
We define the pre- antivativttoon of layer as
(Wi) (o W
(x)
(2)
and the post-activation as
h = o(z).
(3)
3.2 NETWORK EXPRESSIVITY BY ACTIVATION RANK (NEAR)
The construction of our zero-cost proxy NEAR is inspired by theoretical considerations on machine learning that attempt to assess the expressivity of artificial neural networks (Pascanu et al. 2013 Montufar et al., 2014). Therefore, we restate here the most important concepts from these works. For neural networks applying the ReLU activation function, each activation function can be concep- tualized as a hyperplane that separates the input space into an active and an inactive region. Both regions are referred to linear regions. The number of such linear regions can be viewed measure of the expressivity of the network (Pascanu et al.) 2203)] Montufar et al.] 2014). Following previous studies (Raghu et al. 2017 Montúfar] [017]] Serra et al., 2018), we define the activation pattern for an input x and a network Ff' with the ReLU activation function by the set of vectors A(x; F) {a,,...,ay } where an 1 if the output of the ith neuron layer / positive and = 0 otherwise. Given that different linear regions correspond to distinct activation patterns, the number of linear regions is equivalent to the number of activation patterns. Consequently, it is natural idea to analyze the activation patterns to assess the expressivity of network. This idea is exploited in several zero-cost proxies (Mellor al., 202T Chen al. 202T Lin et al., 202T Peng et al. 2024). However, constructing a proxy based on the aforementioned definition of the activation pattern will restrict its application to networks applying the ReLU activation function, a drawback that all those proxies have in common.
To build a more general zero-cost proxy, we relax the definition of the activation pattern to enable other activation functions: A(x; (h...... hrJ, with hy defined in Eq. (3) While there is no longer a direct relationship to the number of linear regions, we argue that inputs with a very similar activation pattern more challenging for the network to distinguish, and we leverage this argument to develop our zero-cost proxy. We define the pre-activation matrix for layer as Z, E R K"i where ny is the number of neurons in layer 1. Each of the ny rows of Z₁ contains 2n for some
3

---

Published as a conference paper at ICLR 2025
different sample imput x,. In similar manner, the post-activation matrix for layer is defined as H, E R=**"i, where each row contains h for a different sample input x, Intuitively. we would expect well-performing network when the rows in the pree/post-actiiation matrix very different to each other, while a matrix with all similar rows would indicate a poorly performing network. As an example, we can consider a classification problem in which rows are highly similar yet belong to inputs of different classes. In this case even a minor perturbation can result in a change in the predicted class. Additionally, we would expect that a matrix, in which certain rows have entries that are much larger than the entries in other rows, will result in poorer performance than matrix with entries of similar size. The reason for this behavior is that the differences in size can cause the predictions to become highly dependent on individual weights and hence on individual input values. These two relationships are captured by the effective rank, defined as follows:
Definition 3.1 Effective Rank (Roy & Vetterli 2007). The effective rank of a matrix A E CMNN, with Q min(M, (M, N), is given by
crank(A)
A) = expl (pi..... ... pP)))
(4)
where H H(pi....,Po' p2)) is the Shannon entropy (Shannon 1948).
- Σ pk log(pk). with Pk
Ök
(5)
Σ 1==1 Ōz
k=
where Pı: are normalized singular values and o; are singular values.
To interpret the effective rank of matrix, we can think of the matrix as a linear transformation. The rank of the matrix indicates the number of dimensions of its range. By contrast, the effective rank contains information about the geometrical shaping of the transformation, A linear transformation that causes a strong stretching along one dimension, while all other dimensions remain unchanged, has lower effective rank than a transformation that causes an equally strong stretching along all dimensions. The rank of the matrix of both transformations is, however, identical. In addition, lin- car transformation that stretches each dimension equally is represented by a matrix with orthogonal rows, and rows that orthogonal each other could be considered maximally different. In other words, the effective rank measures what we have intuitively explained above as being indicators of good performance. Consequently, we propose the zero-cost proxy NEAR as the following:
Definition 3.2 Network Expressivity by Activation Rank (NEAR). The NEAR score of a neural network is given by
Σ (Zaan erank(H:).
where Z₁ and Hi denote the pre- and post-activation matrices, respectively.
A higher NEAR score s indicates a better performing network. The score depends on the input sam- ples, which will usually be randomly selected. To prevent large, random fluctuations depending on the selected samples, we recommend to calculate the score multiple times and employ the average. On the NAS benchmarks, we calculated the average 32 repetitions, while for the machine learning potential benchmarks, we employed 400 repetitions due to the more diverse dataset.
3.3 NEAR FOR CONVOLUTIONAL NEURAL NETWORKS
Convolutional Neural Networks (CNNs) are a neural network architecture particularly popular in the field of computer vision (Li et al. 202I). Our description here focuses on networks applying 2D convolutions. However, the approach can be extended to 3D convolutions as well. For 2D convolutions, the input has the dimensions W H C, where C is the number of channels. For example, C' equals 3 for color images, since these consist of red, green, and blue color channels with dimension W x H (number of pixels in width W and height H). To extract features from the input, convolution operation is performed (see Figure A.2 using one or more filters with dimension k x l x C, where l: is an adjustable hyperparameter. These filters are essentially the weights that learned during the training process. The results of the convolutions are summed, that each filter results in a single output matrix, also known as a feature map. The output is passed to the next layer, where new filters are applied, allowing the network to learn increasingly complex features.
4

---

Published as a conference paper at ICLR 2025
We employ the feature maps in the construction of the activation matrix to calculate the NEAR score for CNN. A layer with C' imput channels results in an output of the form Z₁ R W x H xC' with W = W - l + and H' = H - k + 1 (in case of no padding and stride of one). C2 denotes the number of filters in the layer. Since this output is 3D tensor, the NEAR score cannot be calculated directly. Instead, the output is first transformed into a matrix. We consider the number of filters to be analogous the number of neurons in a fully connected layer. Therefore, the maximum possible NEAR score should be larger if there are more filters in a layer. This behavior could be achieved by flattening each feature map into vector and then packing the vectors into a matrix. However, for an image with three channels and 32 x 32 pixels as input, this approach quickly results in a large matrix. For example, when using filter of dimension 3 3 8, the matrix already of dimensions 7200 x 7200, and the effort to compute the NEAR score is no longer negligible. Therefore, we approximate this matrix for the calculation of the NEAR score. First, we flatten the output to get Z E Rw' N''x'' xC' (see Figure A.3). Second, to reduce the size of the first dimension of this matrix, a random selection of C" contiguous rows is made, where the first row contains elements taken from the first row of the feature maps. This approach yields essentially submatrix of the large matrix containing all the feature maps as vectors. We employ this submatrix as proxy to compute the NEAR score. For a comparison between the rank calculated from the full matrix and the submatrix,
see Figures A.8 and A.9
RESULTS AND DISCUSSION
4.1 CORRELATION OF EFFECTIVE RANK AND FINAL MODEL ACCURACY
To examine the performance of our zero-cost proxy NEAR, we evaluated it on the three standard cell-based NAS benchmarks NAS-Bench-100 (Ying et al. 2019), HATS-Bench-SSS, and NATS- Bench-TSS (Dong et al. 2022). We note that "cell-based" refers to the construction of the individual networks, which were obtained by placing stacks of repeated cells in a common skeleton (for details see Ying et al. (2019) and Dong et al. (2021). For comparison, we report the two commonly em- ployed rank correlation measures Kendall's s (Kendall 1938) and Spearman's 's p (Spearman 1904) also for twelve other zero-cost proxies. For Grad _orm_ SNIP, GraSP, Fisher, and SynFlow we relied on the implementations of Abdelfattaah et al. (2021) and for ZiCo (Li et al., 2023), McCOopt (Jiang eet al. 2023), swap (Peng et al. 2022). and reg-swap (Peng et al., 2024) we employed the respec- tive paper's implementation. All results were recalculated by us. In some cases we observed small differences in the correlation coefficients compared to the values of the original publications. To demonstrate that NEAR is not only applicable to image classification tasks, we provide supplemen- tary results ct al. 2024) in Section A.2
PransNAS--encch-100 -eeechh101--iccr (Duan et al. 2021) and HW-GPT-Bench (Sukthanker
NATS-Bench-TSS
The WATS-Bench-TTS. search space consists of 15 625 neural network architectures trained on the datasets C0, CIFAR-100 (Krizhevsky & Hinton 2009), and ImageNet16-120 (Chrabaszez eet al., 2017). The correlation between the different proxies and the final test accuracy is shown in Table ! Some of the previously reported proxies, such as Grrm_ SNIP, and GraSP, yield a lower correlation than the number of parameters. The latter can be seen as trivial baseline for the zero-cost proxies. The more recently developed proxies ZiCo, MeCo swap, reg swap, and our proxy NEAR demonstrate superior performance relative to this baseline. In general, MeCo,gt achieves the highest correlation across all datasets for NATS-Bench-TSS.. Nevertheless, the NEAR and reg_swap results are very close to those of MeCo.pt. However, the comparison to the reg_swap score is not entirely fair because it assumes that architectures with a parameter number close to the chosen parameter μ perform best, which may not be universally true. McCo and NEAR are therefore more generally applicable.
NATS-Bench-SSS
For the larger NATS-Bench- Bench-SSS search space with 32 768 architectures trained on CIFAR-10, CIFAR-100, 100, and ImageNet16-1200 neither McCogg nor reg_swap consistently show higher cor- relation with the final test accuracy than the number of parameters (Table2). Only SynFlow, NEAR, and ZiCo are able to surpass the performance of this baseline on all three datasets, with SynFlow
5

---

Published as a conference paper at ICLR 2025
demonstrating the highest correlation CIFAR-10 and ImageNetl6-122 and NEAR yielding the highest correlation on CIFAR-100.
NAS-Bench-101
The NAS-Bench-Iie 101 search space consists of 423 624 architectures trained on the CIFAR-IC 10 dataset. As shown in Table E NEAR achieves the highest correlation with the final accuracy for both metrics. It significantly outperforms MeCoogt SynFlow, and wgg.s which show competi- tive performance on NATS- MATS--Bnch--SS. and NAS-Bench-109 is shown by the Zen-Score, closely followed by ZiCo. However, Zen-Score yields correlation than the number of parameters NATS-Bench-TSS across all datasets.
The second highest correlation for
Table 1: Kendall's ' т and Spearman's 's correlation coefficients for different zero-cost proxies on the MATS-Bench-TSS. benchmark. The highest correlation is highlighted in bold face.
MATSsBBenc- ISS
Dataset
CIFAR-I0
CIFAR- 100
INI6- 120
Correlation
Proxy
P
norn Atteelaataa al.
2021
0.48
0.65
0.47
0.64
0.41
0.56
SNIP
Abdellatiah el al 2021
0.48
0.64
0.47
0.63
0.43
0.57
GraSP
Abdeliattah et al.
2021
0.39
0.55
0.40
0.57
0.40
0.55
Fisher
Abdelfatiah et al.
2021
0.41
0.55
0.41
0.56
0.35
0.47
SynFlow [Abietrataa
20221
0.58
0.77
0.57
0.76
Zen-Score
0.32
0.43
0.31
0.42
0.32
0.44
FLOPs
0.58
0.75
0.55
0.73
0.51
0.68
#Params
0.58
0.75
0.55
0.73
0.49
0.66
7iCo et. 2023
0.58
0.78
0.60
0.80
0.57
0.76
MeCungt Chang el 2023
0.72
0.73
0.90
0.67
swup (Peng el 2014
0.64
0.82
0.65
0.82
0.58
0.74
reg_ [Peng a.. 2024
0.71
0.88
0.69
0.87
0.62
0.79
NEAR
0.70
0.88
0.69
0.87
0.66
0.84
Table 2: Kendall's r and Spearman's 's correlation coefficients for different zero-cost proxies on the HATS-Bench-Se benchmark. The highest correlation is highlighted in bold face.
ENTS-Bench-SSS CIFAR-TO
Dataset
CIFAR T00
INT6- T20
Correlation
Proxy
Grad_porm Abdelfattah et al. 2021
0.53
0.72
0.38
0.54
0.70
SNIP Abdelfatiah ei al. 2021
0.63
0.82
0.43
0.61
0.61
0.79
GraSP
Abdelfattah et al
2021
0.30
0.43
0.10
0.15
0.33
0.48
Fisher
Abdeliataa et al.
2021
0.47
0.65
0.30
0.43
0.43
0.60
SynFlow [Abiel
0.79
0.59
0.78
0.81
Zen-Scure
0.75
0.92
0.50
0.69
0.72
0.89
FLOPs
0.44
0.61
0.19
0.28
0.41
0.57
#Params
0.69
0.87
0.53
0.72
0.68
0.86
ZiCo (Li al.
2023 2023
0.72
0.59
0.54
0.74
0.73
0.90
McCo Ulang 21 2023
0.74
swap (Peng
0.48
0.67
0.23
0.34
0.43
0.60
swap (Peng el al. 2024
0.62
0.81
0.38
0.54
0.55
0.73
NEAR
0.74
0.91
0.62
0.82
0.76
0.92
To conclude, we provide performance ranking of the zero-cost proxies across all datasets and search spaces. For each combination of search space and dataset, we order the zero-cost proxies according to their Spearman's value. We identify the position in this order as rank, whereby the proxy of highest correlation is assigned rank one. Subsequently, the average over all datasets for each NAS search space and for all search spaces is calculated. For example, NEAR achieves ranks three, one, and two on respectively. These rankings result in an average rank of two on this search space. Of the proxies tested, reg-swap, SynFlow, ZiCo, MeCo and NEAR are on average better than the baseline given by the number of parameters (Table 4 However, only NEAR and ZiCo yield consistently higher correlation than the number parameters for each combination of search space and dataset, whereby NEAR outperforms ZiCo in every case. For each search space, NEAR is either the best second best zero-cost proxy highlighting its broad applicability. Overall, NEAR achieves the best rank, followed by Mc and ZiCo.
for CIFAR- 10, CIFAR-100, and ImageNet16: 120,

---

Published as a conference paper at ICLR 2025
Table Kendall's and Spearman's 's correlation coefficients for different zero-cost proxies on the MAS-Bench-110 101 benchmark. The highest correlation is highlighted in bold face.
NAX Bench-101
Correlation
Proxy
7
Grad norm AAbdelfattah et al.
2021
0.17
0.25
SNIP (Abdeliattah ct al 2021
0.12
-0.17
GraSP [Abdcifattan al 2021
0.17
0.25
Fisher
al.
-011
-..88
SynFlow (Abdelaatan
0.25
0.37
Zen-Score [Lml el
0.47
0.64
FLOPs
0.30
0.43
#Params
0.30
0.43
ZiCo ct 2023
0.45
McCo (Jing 0 2023
0.35
0.49
(Peng et al. 2024
0.31
0.43
reg reg_swap [Feng et al
2024
0.30
0.43
NEAR
0.52
0.70
Table 4: The rank of different zero-cost proxies averaged over all datasets of SATS--ench-SSS, and WAS-Bench-100 and across all search spaces. The best rank is highlighted in bold face.
Average rank
Search space
TSS
SSS
101
All
Pruxy
GraSP (Abdelfattah al. 2021
9.33
12.00
7.00
10.14
Fisher Abuellatian el al. 2021
10.00
9.67
10.00
9.86
Grad_norm (Abdellatah el al. 2021
7.67
8.33
9.00
8.14
FLOPs
5.67
11.00
5,00
7.86
SNIP (Abdelfattah al. 2021
8,00
6.67
5.00
7.43
Zen-Score (Lin et 202T
11.00
4.00
2.00
6.71
swap (Peng et at. 2024
3.33
9.67
5.00
6.29
#Params
6.33
5.00
5.00
5.57
reg seg_swap (Pene ct al 2024
2.00
7.67
5.00
4.86
SynFlow (Abdelfartah
2021
ZiCo ef 2024
3.67
3.67
3.00
3.57
MeCogg: Uaang 2023
1.00
3.67
4.00
2.57
NEAR
1.67
2.00
1.00
1.71
4.2 ESTIMATION OF OPTIMAL LAYER SIZE
Inspired by previous work that suggested a power law scaling for the accuracy with respect to the model size for language models (Kaplan et al. 2020), we analyzed the NEAR score of a single layer of different sizes. We observed that the relative score, i.c., the score divided by the number of neurons, appears to be well represented by a simple power function of the form f(s) = a + +3--7, s", where is the NEAR One is the balanced version of the extended MNIST (EMNIST) dataset (Cohen et al., 2017). contains handwritten digits and letters, with an equal number of examples for each class. The other is a molecular dataset for lifelong machine learning potential (IMLP). This dataset includes target energies and atomic forces for features given by molecular structures, which were obtained from various conformations in 42 different Sx2 reactions (8 600 structures) (Eckhoff & Reiher 2023).
(see Figure A.4). This behavior consistent for two different datasets.
We argue that decreasing relative NEAR score indicates that the network benefits less from additional neurons and, at certain threshold, its performance is no longer limited by its size. Our experiments suggest threshold based on the slope of the power function to work reasonably well. We therefore propose the following approach to estimate the size of individual layers: First, calculate the NEAR score for some different layer sizes. Second, a power function to the calculated relative scores. Third, calculate at which layer size the slope of the fit falls below certain threshold and apply this layer size for the neural network. Since the NEAR score of each layer depends only the previous layers, this approach also be applied to networks with multiple layers. Starting with the first layer, the size of all layers can be determined iteratively.

---

Published as a conference paper at ICLR 2025
Machine Learning Potential
We applied our approach to estimate the layer sizes on a seseeehhdden--yyeee machine learning poten- tial, employing the dataset B of Eckhoff & (2033) Our method predicted a size of 52, 58, and 60 for the first, second, and third hidden layers, respectively, with a threshold of 0.5% of the slope at layer size of 1. These sizes are significantly smaller than applied in Ecchhof & Reiher (2023) (which are 102, 61, and 44). While this change reduces the number of parameters from 22 990 to 13 909, the average test loss only increases from 0.012 + 0.003 to 0.015 + 0.004. This increase is reflected in change of the root mean squared error for the energies from (4.4 = 1.2) meV atom"
to (4.6 + 1.2) meV atom - and for the forces from (100 = 8) meV À to (109 + 8) mcV À A- These numbers represent the mean and standard deviation over 20 individual machine learning po- tentials, for which the training was started from different randomly initialized neural networks. We note that the test loss depends not only on the capacity of the neural network, but also on the training process. However, NEAR only aims to capture the capacity of network and does not take into account the training dynamics. Therefore, NEAR alone cannot be expected to determine the perfect size but rather to give a good estimate. Keeping the size of the first and second hidden layers at 52 and 58, respectively, and only varying the size of the third hidden layer showed that the predicted size achieves good performance. Even increasing the layer size to 100 neurons did not improve the loss (Table 5). However, slightly smaller layer of 40 neurons would still be sufficient to achieve the same performance. We note that smaller networks require fewer computational resources and are therefore more efficient to evaluate. Repeating the experiment for a variable second hidden layer showed again that the predicted size has favorable characteristics in terms of accuracy and number of parameters. Reducing the size of the second layer to 38 neurons (reduction in the number of parameters by 16%), results in an increase in the loss of 20%. Conversely, increasing the neuron number to 78 (increase in the parameter number by ~ 16%). yields loss decrease of only ~ 7% (Table 5)
Balanced Extended MNIST
In a second experiment, we estimated the layer sizes for two-hidden layer neural network on the balanced EMNIST dataset. Again, we applied a threshold of 0.5% of the slope at layer size of 1. The resulting sizes were 200 for both, first and second hidden layer. A comparison of the test loss to a network with reduced second layer size of 120, which decreases the parameter number by 10%, shows an loss increase of ~ 4%. The aforementioned loss values correspond classification accuracy of 86.85% and 87.30%. Conversely, an increase in the neuron number to 280 for the second layer, which increases the parameter number by ~ 9%, results in a decrease in the loss of around 1% and an increase in the accuracy to 87 37%.. This observation suggests that size of 200 represents a reasonable compromise between efficiency and accuracy (Table (). To check whether also the prediction for the first layer was reasonable, we fixed the size of the second layer to 200 and train networks with various sizes for the first layer (Table ()) A comparison of the test loss for network with size of 160 (~ 20% decrease in the parameter number) yields increase in the loss of ~ 1.3% and a decrease in the accuracy from 87 87.30% to 87.14%. Even when the size of the first layer was increased to 260 (~ 30% increase in the parameter number), the loss can only be reduced by- ~ 0.7% and the accuracy is increased to 87.32%. Again, the predicted size appears to offer good trade-off between the number of parameters and the accuracy.
a
Table 5: The test loss of trained IMLPs when only the size of the second or third hidden layer is changed. Marked in bold is the architecture predicted by our approach. In this and all following tables, mean and standard deviation (rounded up) of 20 experiment repetitions are reported.
Variation of second layer
Test loss
Paarmss
Variation of third layer
Test loss
#Params
52
18
0.022 0.004
9389
58
20
0.017 0.003
11
52 38 60
0.018 0.004
11649
52 58
40
0.015 0.003
12 709
52 58- 60
0.015 # 0.004
909
52 58
60
0.015 = 0.004
139
32 78 - 60
0.014 = 0.004
16 169
52 58 880
0.014 = 0.008
15 109
52 98 60
0.014 0.003
18 429
52 58 100
0.015 0.003
16309
4.3 ESTIMATION OF ACTIVATION FUNCTION AND WEIGHT INITIALIZATION PERFORMANCE
Numerous NAS benchmarks, including NATS-Bench- TSS and -SSS and NAS- Be--BBenhhhh. apply the ReLU activation function for all networks. There is lack of attention regarding zero-cost proxies
8

---

Published as a conference paper at ICLR 2025
Table 6: The test loss on the balanced EMNIST dataset when only the size of the first or second hidden layer is changed. Marked in bold is the architecture predicted by our approach.
Variation first layer
Test loss
#aarmss
Variation of second layer
Test loss
#Params
120
200
0.393 0.012
127 817
200
40
0.433 0.017
166 967
140 200
0.380
0.006
1475:)
200
80
0.397 + 0.011
176 887
160- - 200
0.373 0.009
167 247
200 120
0.383 + 0.009
186 807
180 200
0.368 0.007
186947
200 160
0.376 0.009
196 727
200 200
0.368 0.006
206, 647
200
200
0.368 1 0.006
206, 647
220 200
0.363 = 0.007
226
200 240
0.364 0.006
216 567
240 200
0.359 0.006
246 D47
200 280
0.364 + 0.008
226 187
260 - 200
0.366 0.007
265 747
200 320
0.363 + 0.009
236 407
280 200
0.363 0.008
285 447
200 360
0.359 0.007
246 327
to detect favorable activation functions weight initialization schemes. There even proxies specifically designed for the ReLU activation function (Mellor et al.) 2021 Chen et al., 202 Peng
al. 2024). However, it is well known that choosing appropriate activation function and weight
initialization scheme is crucial for achieving good model performance (LeCun et al.) 1998 Glorot
& Bengio 2010 Eckhoff & Reiher 2023). To assess whether NEAR can be employed to select an
activation function and a weight initialization scheme, we calculated the score before training and compared it with the final model performance.
Machine Learning Potential
Eckhoff & Reiher (203)) previously reported that the activation function (Tanh((:) 1.59223
tauh(z) with tailored weight initialization scheme improves the final accuracy of IMLP. We re- peated training for the same dataset and the same neural network architecture (number of neurons per layer: 1 1/////2//// 22/1/ 44//1) employing the CoRe optimizer (Eckhoff & Reihher 2023 2024a b). Table [shows that the sTanh activation function with the tailored weight initialization scheme achieves the lowest test loss, while also exhibiting the largest NEAR score. Comparing the two weight initializa- tion schemes s'aanh and Kaiming uniform (He et al. 2015) demonstrates that the former improves performance and that the NEAR score can be applied as a reliable predictor for the more suitable weight initialization scheme. A comparison of the loss and NEAR score of different activation functions applying the same weight initialization scheme also shows that the score can guide the selection of an appropriate activation function. For example, the activation function sTanh with the customized initialization scheme exhibits the lowest loss and the highest NEAR score, while the activation function Tanhshrink shows the highest loss and lowest NEAR score for this weight initialization scheme. However, the discrepancies
loss for the three best activation functions s' s'anh, Sigmoid Linear Unit (SiLU) (Elfwing et al.] (2018), and Tanh are minimal, whereas the dif- ferences in the NEAR score would suggest larger differences. The lowest NEAR score is obtained by Tanhshrink with Kaiming initialization, which is also consistent with the fact that this combi- nation achieves the highest average loss. Hence, NEAR is capable of identifying more and less effective activation functions, while the ordering in close cases cannot be guaranteed. Since the loss function of the IMLP includes derivative constraints, the second derivative of the activation function appears in the gradient calculation. Consequently, only activation functions with non-zero second derivative can be considered. resulting in the omission of the ReLU function in this experiment. This issue once again highlights the benefit of NEAR that it can be applied to any activation function.
Balanced Extended MNIST
To show that the NEAR score can guide the selection of the activation function and weight initial- ization scheme across various datasets and independent of the optimizer, we report the test loss and the NEAR score on the balanced EMNIST dataset. We employed a multi-layer perceptron with two hidden layers of size 200 and the Adam optimizer (Kingma & Ba 2015). We applied early stopping with a patience of 10 epochs, whereby 10% of the training set served as validation set. We report the test loss and the NEAR score for the initialization schemes Xavier uniform (Glorot & Bengio 2010). Kaiming uniform, and uniform and the activation functions SiLU, ReLU, Tanh, and Tanhshrink in Table 8 The NEAR score indicates that the Xavier initialization is the most effective, followed by the Kaiming initialization and the uniform initialization. While the uniform initialization is undoubt- edly the least favorable in terms of resulting test loss, the distinction between Xavier and Kaiming is not as cuear-cut. For the activation functions under consideration, the average loss is found to be lower with the Kaiming initialization than the Xavier initialization, but the difference is marginal

---

Published as a conference paper at ICLR 2025
and the standard deviations are higher than the difference. For the Kaiming and Xavier initializa- tions, the activation functions SiLU, ReLU, and Tanh result in similar test losses, while the NEAR score gives the impression that there is clear ordering for given initialization. As the orderings of the test losses and NEAR scores do not agree for both initializations, we conclude that these small differences in the resulting loss cannot be captured by the NEAR score. We note here again that the NEAR score only accounts for the expressivity of the network and does not take into account the training process. Still, the NEAR score can reliably predict the worst performing activation function (Tanhshrink). For this activation function, the test loss is also significantly different from those of the others. Analogously, the NEAR score can correctly identify that SiLU, Tanhshrink, and ReLU lead to lower loss than Tanh if the uniform initialization is applied. Hence, NEAR can predict performance of an activation function and weight initialization scheme, though it may not re- solve minor differences in the resulting loss. In addition, Section A.3 shows that NEAR can identify suitable activation functions and weight initialization schemes better than previous proxies.
Table 7: Comparison of the test loss (after training) and the NEAR score (before training) for various activation functions and weight initialization schemes.
a, initialization
Test loss
NEAR score
TTanh, s'Tanh
0.012 1 0.003
70.8 0.2
SiLU, sTann
0.013 : 0.004
63.1
Tanh, sTanh
0.015 = 0.003
6.3
Tanhshrink, sTanh
0.027 = 0.006
45.2 0.2
SiLU, Kaiming
0.031 = 0.008
57.1
2.5
<Tanh, Kaiming
0.041 + 0.019
60.3
+ 2.5
Tanh, Kaiming
0.050 1 0.022
58.5
2.3
Tanhshrink, Kaiming
0.082 + 0.014
38.7
1.8
Table Comparison of the test loss (after training) and the NEAR score (before training) for various activation functions and weight initialization schemes on the balanced EMNIST dataset. For "SiLU, uniform" one outlier was excluded from the mean and standard deviation.
initialization
Test loss
NEAR score
SILUU Kaiming
0.367 + 0.008
382.8 + 5.0
SiLU, Xavier
0.368 0.008
417.7 1 0.6
ReLU, Kaiming
0.372 0.010
403.7 =
Xavier
0.374 0.008
421.1
Tanh, Kaiming
0.392 0.010
398.2 5.1
Tanh, Xavier
0.396 + 0.011
415.5 = 0.4
Tanhshrink, Kaiming
0.447 + 0.022
278.4 3.2
Tanhshrink. Xavier
0.449 0.018
401.4 =
SiLU. uniform
0.525
13.3
Tanhshrink, uniform
4602++0.080
13.4 0.1
ReLU, uniform
0.651 0.047
13.3 1 0.1
Tanh, uniform
3.858 1: 0.030
9.0 1 0.1
5 CONCLUSION
In this work, we have introduced NEAR which a zero-cost proxy to pre-estimate the performance of a machine learning model without training. For this purpose, the NEAR score estimates the expressivity of neural networks by calculating the effective rank of the pre- and post-activation matrix. We have demonstrated its strong correlation with the final model accuracy on NATS-Bende SSS/TSS and MAS--ench--O0. In a ranking of 13 zero-cost proxies by their Spearman's 's correlation coefficient, NEAR achieves an average rank of 1.7 across all datasets and search spaces, while the second-best proxy MeCo.p has only an average rank of 2.6. In contrast to other proxies, NEAR only requires sample input data, but no output labels and it not bound to a specific activation function. In fact, we have shown that it can even be applied to choose well performing activation function and weight initialization scheme. Moreover, NEAR is not limited to a given search space and can be applied to estimate the optimal layer size in multi-layer perceptrons. This task represents significant challenge in the construction of neural networks, and hence, NEAR can help substantially reduce invested human time and computational demand.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was created as part of NCCR Catalysis (grant number 180544), a National Centre of Competence Research funded by the Swiss National Science Foundation. M.E. was supported by an ETH Zurich Postdoctoral Fellowship.
REPRODUCIBILITY STATEMENT
In order make calculations reproducible, we provide the code including all hyperparameter settings as well as the raw data on Zenodo (Husistein et 20.2)) The NEAR software is available on GitHub (https: /ectps://giie tww.. com/co/eeeeeeeeee herGroup/ NEAR).
REFERENCES
Mohamed S. Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas D. Lane. Zero-cost proxies for lightweight NAS. In 9th International Conference Learning Representations (ICLR), 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Adv. Neural Inf. Process. Syst. volume 33, pp. 12449-12460, 2020.
Jörg Behler. Four generations of high-dimensional neural network potentials. Chem. Rev. 121: 10037-10072, 2021. doi:
Jörg Behler and Michele Parrinello. Generalized neural-network representation of high nigh-dimensional potential-energy surfaces. 146401.
Phys. Rev. Lett., 98:146401, 2007. doi: 10.1103/hhssReeLeet.....
Christopher M. Bishop. Pottern Recognition and Machine Learning. Springer, New York, NY, USA, 2006. BtRLlattps///iikk..... com.://......eee..ooooooooo/ooaa http://geeccc///o//////333333333
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search ImageNet in four GPU hours: A theoretically inspired perspective. 9th International Conference on Learning Representations (ICLR), 2021.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the CIFAR datasets. arXiv:1707.0880 [es. CV), 2017.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre van Schaik. EMNIST: Extending MNIST to handwritten letters. In International Joint Conference on Neural Networks (IJCNN,, pp. 2921-2926, Anchorage, United States of America, 2017. IEEE. doi: N0.1109/UCNN2201. 7966217.
George Cybenko. Approximation by superpositions a sigmoidal function. Math. Control Signals Syst. 2(4):303-314, 1989.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. algorithms for architecture topology and size. IEEE Trans. Patlern Anal. Mach. Intell., 44(7): 3634-3646, 2021. doi: 10.1 109/TPAMM..0011 3054824.
Benchmarking NAS
Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li. TransNAS-Benchh
Improving transferability and generalizability of cross-task neural archi- tecture search. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Marco Eckhoff and Markus Reiher. Lifelong machine learning potentials. J. Chem. Theory Comput., 19:3509-3525, 2023. doi: 10. 10.1021/assjjerrc.c00777
Marco Eckhoff and Markus Reiher. CoRe optimizer: an all-in-one solution for machine learning. Mach. Learn.: Sei. Technol, 5018, 2024a. doi: 10. 10.1088/2632-21551a

---

Published as a conference paper at ICLR 2025
Marco Eckhoff and Markus Reiher. RelherCrrupCookk.pptiiiiez Release 1.1.0. Zenodo, 2024b.
doi: 10.5281/2emod..1111 11551858.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Nerw. 107:3 11, 2018.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. J. Mach. Learn. Res., 20(55):1-21, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In 13th International Conference on Artificial Intelligence and Statistics (AISTATS), pp- 249-256, Sardinia, Italy, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proc. IEEE Int. Conf. Comput. Vis., pp- 1026-1034, 2015.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs).
arXiv:1606.08115
[cs.LG], 2016.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Netw., 4(2): 251-257, 1991.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural Netw., 2(5):359-366 1989.
Raphael T. Husistein, Markus Reiher, and Marco Eckhoff. NEAR: A training-free pre-estimator of machine learning model performance (version 1.0). Zenodo, 2024. doi: 10.5281/zemodo 13332011.
Tangyu Jiang, Haodi Wang, and Rongfang Bie. McCo: Zero-shot NAS with one data and single for- ward pass via minimum eigenvalue correlation. In Adv. Neural Inf. Process. Syst., volume 36, pp. 61020-61047, 2023.
John Jumper, Richard Evans, Alexander Pritzel, and et al. Highly accurate protein structure predic- tion with AlphaFold. Nature, 596(7873):583-588 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.0833 [Cs, 2020.
Maurice G. Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In 3rd Inter- national Conference Learning Representations ICLR, San Diego, United States of America, 2015.
Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, and Frank Hutter. NAS- Fench-Suite--e Accelerating research on zero proxies. In Adv. Neural Inf. Process. Syst., volume 35, pp. 28037-28051 2022.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-50. Springer Berlin Hcidelberg, 1998, doi: 10.1007/ 3-540-49430-87 2.
Namhoon Lee, Thalaiyasingar Ajanthan, and Philip HS Torr. SNIP: Single-shot network pruning based on connection sensitivity. In 7th International Conference on Learning Representations (ICLR), New Orleans, United States of America, 2019.
Guihong Li. Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. ZiCo: Zero-shot NAS
via inverse coefficient of variation on gradients. In 1 International Conference on Learning Representations (ICLR), Kigali, Rwanda, 2023.
12

---

Published as a conference paper at ICLR 2025
Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Trans. Neural Netw. Learn. Syst., 33(12): 6999-7019, 2021.
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-NAS: A zero-shot NAS for high-performance image recognition. In Proc. IEEE Int. Conf. Comput, Vis., pp. 347-356, 2021. doi: 10.1 109/1C144992220002222 .00040.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In 7th International Conference Learning Representations (ICLR), New Orleans, United States of America, 2019.
Pablo Ribalta Lorenzo, Jakub Nalepa, Michal Kawulok, Luciano Sanchez Ramos, and José Ranilla Pastor. Particle swarm optimization for hyper-parameter selection in deep neural networks. In Generic and Evolutionary Computation Conference, pp. 481-488, Berlin, Germany, 2017. doi:
00.1144//071178.307222
Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training. In 38th International Conference on Machine Learning (ICML), pp. 7588-7598, 2021.
Jisoo Mok, Byunggook Na, Ji-Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural architecture search without training? In Conference on Computer Vision and Pattern Recognition (CVPR), pp-
11851-11860, 2022.
Guido Montúfar. Notes the number of linear regions of deep neural networks. In h Interna- tional Conference on Sampling Theory and Applications (SampTA), Tallinn, Estonia, 2017.
Guido F. Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Adv. Neural Inf. Process. Syst., volume 27, 2014.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In 27th International Conference Machine Learning (ICML), pp. 807-814, Haifa, Israel, 2010.
Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang. Evaluating efficient performance estimators of neural architectures. In Adv. Neural Inf. Process. Syst., volume 34, pp. 12265-12277, 2021.
Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of inference regions of deep feed forward networks with piece-wise linear activations. In 2nd International Conference Learning Representations (ICLR), Banff National Park, Canada, 2013.
Yameng Peng, Andy Song, Haytham M. Fayek. Vie Ciesielski, and Xiaojun Chang. SWAP-NAS Sample-wise activation patterns for ultra-fast NAS. 12th International Conference on Learning Representations (ICLR), Vienna, Austria, 2024.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the ex- pressive power of deep neural networks. 34th International Conference on Machine Learning (ICML), pp. 2847-2854, Sydney, Australia, 2017.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In 34th Interna- tional Conference on Machine Learning (ICML), pp. 2902-2911, Sydney, Australia, 2017. doi: 10.5555/3305800333099
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Comput, Surv., 54(4):1-34, 2021.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. 15th European signal processing conference, pp. 606-610, 2007.
13

---

Published as a conference paper at ICLR 2025
Stuart J. Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Pearson, Harlow, United Kingdom, 4th edition, 2021. URL o de 99.
15C005/378129240111
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions of deep neural networks. In 35th International Conference on Machine Learning (ICML), pp. 4558 4566, Stockholm, Sweden, 2018.
Claude E. Shannon. A mathematical theory of communication. Bell Syst. Tech. J., 27(3):379 423, 1948.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Adv. Neural Inf. Process. Syst. volume 25, 2012.
C. Spearman. The proof and measurement of association between two things. Am. J. Psychol., 15: 72-101, 1904. doi: 10.2307/1412159.
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Aaron Klein, Jorg Franke, and Frank Hutter. HW-GPT-Bench: Hardware-aware architecture benchmark for language models. In Adv. Neural Inf. Process. Syst., volume 38, 2024.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In 36th International Conference Machine Learning (ICML), pp. 6105-6114, Long Beach, United States of America, 2019.
Hidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Adv. Neural Inf. Process. Syst., vol- ume 33, pp. 6377-6389, 2020.
Jack Turner, Elliot J. Crowley, Michael O' O'Boyle, Amos Storkey, and Gavin Gray. BlockSwap: Fisher-guided block substitution for network compression on a budget. 8 International Con- ference Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inf. Process. Syst., volume 30, 2017.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In 8th International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- Bench- 101: Towards reproducible neural architecture search. In 36th International Conference Machine Learning (ICML), pp. 7105-7114, Long Beach, United States America, 2019.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In 5th Inter- national Conference on Learning Representations (ICLR), Toulon, France, 2017.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 SUPPORTING FIGURES
V1
22
y/2
F3
Figure A.I: Illustration of artificial neural network with three inputs {z;} and two outputs {y )- The single hidden layer consists of five neurons. The activation functions are shown within the neurons. The solid lines represent the weights, while the dashed lines indicate input and output without weights.
200
map
Three
Figure A.2: A convolution is performed on the input of dimension 4 x33 with filter of dimension x resulting in a feature map of dimension 3 x 3.
15

---

Published as a conference paper at ICLR 2025
10
11
12
10
23
27
22
(a] Feature mape
14
23
(b) Rechaped Scoture map
(c) Two
Figure A.3: Process of reshaping convolutional neural network feature maps. (a) The process begins with four 3 x 3 feature maps. (b) These feature maps subsequently reshaped to 9 x matrix. (c) An activation matrix given by four contiguous rows, whereby the first row contains elements extracted from the top of the feature maps. From all possible activation matrices one is randomly selected.
Rolative NEAR Seare
0.5
1000
Figure A.4: A power function fitted to the NEAR score divided by the total number of neurons in the layer. The star marks the first time where the slope smaller or equal 0.5% of the slope at 1. The plot has been generated for the experiments on the IMLP.
A.2 ADDITIONAL NAS-BENCIMARKS
TransNAS-RRench-001-
To demonstrate that NEAR is not only applicable to image classification tasks, we show here results seven different tasks from the -eensNs---ench---- 101-Miro benchmark (Duan al., 2021). The tasks include classification, regression, pixel-level prediction, and self-supervised tasks. NEAR outperforms the baseline given by the number of parameters and the number of FLOPs on each task (Table 9)
Table 9: Kendall's and Spearman's p correlation coefficients for NEAR different tasks of TransNAS -ench-101- 101-Micro.
Bakk
Cl. Oljjcc
Cls. Scone
Astocaending
Nurf. Ncemal
Scm. Segment.
Rooen Lapont
Jigaww
Prony
#Params
0.36
0.53
0.62
-0.03
-0.03
0.46
0.05
0.63
0.35
0.30
0.45
0.80
0.37
16

---

Published as a conference paper at ICLR 2025
HW-GPT-Bench
One of the strengths of NEAR is its ability to work with networks which do not employ the ReLU activation function. To explore this capability, applied NEAR to language models with the Gaussian Error Linear Unit (GELU) Henddrycks & Gimpel 2016) activation function using the HW- GPT benchmark (Sukthanker et al. 2024). Specifically, we randomly sampled 20 000 architectures from the small variant of the search space and computed the correlation between the NEAR score and perplexity. While NEAR did not outperform the strong baseline given by the correlation between model parameters and perplexity, it still showed a higher correlation than the MeCo proxy (Table 10
Table 10: Kendall's r and Spearman's 's correlation between zero-cost proxy and perplexity for 20 000 randomly sampled architectures of the small HW-GPT-Bench search space. As lower perplexity values indicate better models, the correlations are expected to be negative.
Correlation
Pruxy
p
#Params
-0.99
FLOPS
-0.92
-0.99
MeCogg Jiang el al. 2023
-0688
-0.88
NEAR
-0.90
-0.9
A.3 PROXIES TO SELECT WEIGHT INITIALIZATION AND ACTIVATION FUNCTION
To show that NEAR outperforms previous proxies in identifying suitable weight initialization schemes and activation functions, we calculated the proxy scores for the same combinations listed in Table 8 Subsequently, we calculated Spearman's o and Kendall's correlation coefficients for the average accuracy as function of the average proxy score. The results indicate that NEAR exhibits a significantly higher correlation compared to the other proxies evaluated (see Table TT]. Since the swap and reg_swap scores are only defined for the ReI.U activation function, they have been omitted from this analysis.
Table 11: Kendall's 7 and Spearman's ' p correlation coefficients between the average model accu- racy and the average proxy scores evaluated on the combinations of activation functions and weight initialization methods listed in Table Averages are obtained from 20 repetitions. A higher cor- relation means that the proxy is more effective at identifying good activation functions and weight initializations.
Correlation
Proxy
Grad.norm (Abdelfattah et al.
2021
0.53
0.32
SNIP (Abdeltattah et al. 2021
0.53
-0.32
GraSP [Abdeifattan
2021
0.53
0.32
Fisher (Alxitaataah
4121
0.53
-0.35
SynFlow (Atuellataan et
4021
0.15
-0.88
Zen-Score [Lm el
2021
0.35
0.24
at 2024
0.07
0.00
McCo., (Jaang
2023
0.67
0.45
NEAR
0.84
0.70
A.4 STABILITY OF NEAR SCORE AFTER MINIMAL TRAINING
Similar to the analysis Mok et al. 2022), we computed the NEAR score after 0, 1, 3, 5, and 10 epochs of training the NATS SSS benchmark using the CIFAR-10 dataset. While many of the tested proxies show a decrease in correlation after some training. the correlation of NEAR only decreases slightly after the first training epoch and increases steadily for the following epochs (Figure A..5). Remarkably, NEAR is the only proxy that achieves a higher correlation after ten epochs of training than it had prior to training.
17

---

Published as a conference paper at ICLR 2025
NEAR
MoCo opt
ZiCo
SynFlow
0.930
6.00
0.33
0.010
Speamaa's
Spsaman's
Sptaman's
66n
230
0.822
682
3.99
0.80
16
Training epoch reg. swap
Training spoch
Traninn opoch Zen-Spore
Training apoch SNIP
#Params
2.40
Spearman's
0.82
Spaarman's
Spearmenn's
Spearman's
0.0
6.4
16
Training epoch FLOPs
Training epoch
Training spoch Fisher
Training epoch GraSP
Grad norm
9.7
0.6
Speerman's
Spearann'
Sprarman's
2/4
Training epoch
Training epoch
Training epoch
Training epoch
Figure A.5: Spearman's ' correlation on NATS-Bench-SSS using the CIFAR-10 dataset, evaluated
after 0, 3, 5, and 10 training epochs.
A.5 INTERPRETATION OF COIOELATIONS
To provide further insight into the interpretation of the correlations between proxy scores and accu- racy, we calculated the probability that, given two randomly selected networks Nj and N2, if the accuracy of N₁ higher than that of N2, then the proxy score of N₁ will also be higher than the proxy score of N2. These probabilities are presented in Table 12
Search Space
NATS Bench TSS
NAIS Bench SSS
NAS Bench 101
Correlation
CIFAR-10
CIPAR-100
IN16-120
CIFAR-10
CIFAR-100
IN16-120
CIFAR-10
Proxy
swap Peng et al.
reg_swap (Pene
2024
0.85
0.85
0.81
0.81
0.09
0.77
0.65
McCous.
(Jin
ZiCb
Li HIM4
0.79
0.50
0.79
0.86
0.77
0.86
0.73
2021
SynFlow
AAroiaatal 21
202
0.79
0.78
0.75
0.90
0.79
0.91
0.63
AAfieitataan
GraSP
2021
0.69
0.70
0.70
0.65
0.55
0.67
0.59
SNIP
Arxieiraran
0.74
0.82
0.44
jammm (Abdelatah al
202
0.74
0.74
0.71
0.77
0.69
0.76
0.42
#Params
0.85
0.84
0.05
FL.PPs
0.73
0.72
0.71
0.72
0.60
0.71
0.65
NEAR
0.83
0.81
0.88
0.76
Table 12: The probability that for two randomly chosen networks the network with the higher ac- curacy also shows the higher zero-cost proxy score. This probability was estimated by randomly sampling one million pairs of networks. The highest probability is highlighted in bold face.
A.6 WORST PREDICTIONS
An examination of NEAR's 100 most inaccurate predictions reveals that its main challenge is in evaluating networks with a large number of parameters. However, unlike other proxies, NEAR does not misclassify a network with low accuracy as one with high accuracy, thus reliably excluding networks with poor performance (see Figure/A.6)
18

---

Published as a conference paper at ICLR 2025
NBAR
Neço Apt
2ICa
SydFoow
02
02-
Namber Farameiera
Number Nauncles
SanturoFeeedaa
107
humber Porarster
1el
reg swap
#Foroms
Zcore
SNIP
02-
62-
Namber Piramriiers
1x3
Number Naumoiss
147
Namber ofntenenn Niterr
147
humber Paramater
26l
ILOPs
Grad nom
GraSP
Set DETAIMPEE
12-
02-
6.2
Parareters
Lx3
Number Parurctes
347
of
167
Numbor Rooamtte
Figure A.6: Each point represents a neural network from the NAS-Bench-101 benchmark. The points highlighted orange indicate the 100 neural networks where the proxy's 's predictions deviated most significantly from actual performance.
A.7 CONTRIBUTIONS OF INDIVIDUAL LAYERS
To investigate how individual layers contribute to the overall correlation, performed a layer- specific correlation analysis. This analysis involved generating over a billion random subsets, each comprising 6 layers. For each subset, we calculated the NEAR score. Then, for each layer, we summed the NEAR scores from all subsets containing that layer. Finally, we divided this sum by the total number of subsets in which each layer appeared. This analysis revealed no layer with a significantly lower or higher average correlation than the others. In other words, the results indicated no individual layer exerted disproportionately strong or weak influence on the overall correlation. The average correlation per layer is shown in Figure A.7
A.8 SAMPLING OF FEATURE MAPS
we explained how NEAR can be calculated using only subsample of the full matrix, which consists of all feature maps represented vectors (see also Figure A.3) For NEAR to yield meaningful scores despite this approximation, the subsample matrix needs to remain sensitive to variations in kernel size and the number of channels. As shown Figure A.8 and Figure [g9 the effective rank derived from both, the full matrix and the subsample matrix, exhibit similar trends, indicating that these parameters are accurately captured even within the approximation. In addition, Table 13 shows the impact of the sample size on the estimation of the rank of the full matrix.
In Section 3.3
19

---

Published as a conference paper at ICLR 2025
0.8
0.4
0.3
0.2
0.1
0.0
10
20
30
40
50
60
70
Layer number
Figure A.7: The average correlation for each layer on NATS-Bench-SSS for CIFAR-IO, determined
by computing the correlations for random subsets of six layers and then averaging the results.
Effectivv rank max rank
0g
Effective rank max affective
0.8
12
Number of output channels
Number of output channels
(a) Rank calculated from full matrix.
(b) Rank calculated from subsample matrix.
Figure A.8: Comparison of the rank calculated from the full matrix and the subsample matrix as function of the number of channels. This evaluation is based the CIFAR-10 dataset.
Purameler
Kernel size
Channel nurnber
# Samples
0.95
0.92
0.98
0.98
Table 13: The first column shows Spearman's 's correlation between the rank calculated from the full matrix and the rank calculated from the subsample matrix employing kernel sizes of 1, 3, 5, 7, and 9 averaged over 10 000 iterations. The second column shows Spearman's correlation between the rank calculated from the full matrix and the rank calculated from the subsample matrix employing between 1 and 13 output channels averaged over 10 000 iterations.
20

---

Published as a conference paper at ICLR 2025
0.7
Effective rank max effective rank
0.6
0.30
0.25
Effective
03
0.0
02
015
Kernel size
Kernel size
(a) Rank calculated from full matrix.
(b) Rank calculated from subsample matrix.
Figure A.9: Comparison of the rank calculated from the full matrix and the subsample matrix
function of the kernel size. This evaluation based on the CIFAR-10 dataset.
21


Reviews:
[{"review_id": "jorFsO6bbu", "reviewer": "Reviewer_he4q", "summary": "This paper presents a method for identifying optimal architectures in Neural Architecture Search (NAS) using a zero-cost proxy score derived from the activation rank, without requiring separate training. To achieve this, the authors calculate both pre- and post-activation ranks and introduce a sampling method to approximate matrices for efficiency. Additionally, the paper thoroughly discusses the potential of the proposed approach using effective rank.", "questions": "* **Theoretical Advantages over Existing Methods**: Could the authors provide a more detailed theoretical analysis comparing the NEAR score to existing zero-cost proxies? Specifically, what theoretical advantages does the NEAR score offer that may account for its superior performance? Highlighting any novel insights or mechanisms by which NEAR improves upon prior methods would strengthen the contribution.\n\n* **Justification for Redefining Definition 3.1**: Since Definition 3.1 (Effective Rank) was established in previous studies, can the authors clarify why redefining it here is necessary or beneficial in the context of their method? Providing justification or modifications that are specific to their approach would help readers understand the significance of this definition in their work.\n\n* **Layer-wise Contribution to NEAR Score**: To understand the impact of each layer on the NEAR score, could the authors conduct experiments exploring how the score changes when calculated on different subsets of layers? For instance, does excluding certain layers significantly affect the correlation with final model accuracy? Discussing these findings could provide insights into optimizing computational efficiency without compromising accuracy.\n\n* **Stability of NEAR Score After Minimal Training**: In line with observations from Mok et al. (2022) (1), have the authors considered evaluating the stability of the NEAR score after a few epochs of training? Conducting experiments that compare the zero-shot NEAR score to scores obtained after 2-3 epochs could demonstrate the robustness of the method and whether minimal training affects its predictive capabilities.\n\n* **Comparison with Few-Shot Performance**: How does the zero-shot NEAR score compare with few-shot methods in terms of computational cost and accuracy? Including a comparison could help position the NEAR score within the broader context of NAS methods and provide evidence for its practical advantages.\n\n**References**\n1. “Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?” (Mok et al., 2022)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* **Applicability Across Architectures**: A key contribution of this work is that the proxy score is applicable regardless of architecture structure or activation function, providing robustness and versatility.\n* **Benchmark Performance**: The authors demonstrated the method’s efficacy across standard NAS benchmarks, showing its comparative strength against other zero-cost proxies.\n* **Efficiency**: By not requiring separate training, the method significantly reduces computational overhead, which is valuable for practical NAS applications.", "weaknesses": "* **Lack of Theoretical Clarification**: While the experimental results demonstrate the effectiveness of the NEAR score, the theoretical significance of using the effective rank (erank) in Definition 3.2 remains somewhat unclear. Numerous NAS studies have previously explored concepts related to rank. It would strengthen the paper if the authors could provide a more detailed theoretical analysis comparing NEAR to existing methods, highlighting any novel insights their approach offers. Additionally, clarifying why redefining Definition 3.1 is necessary or beneficial in this context would enhance the theoretical foundation of the work.\n\n* **Evaluation Across Layers**: It is not entirely clear whether calculating the NEAR score across all layers is necessary to achieve optimal performance. The paper does not present experiments exploring how the NEAR score changes when calculated on different subsets of layers. Conducting such experiments could provide insights into the contributions of individual layers to the overall score and discuss the implications for computational efficiency and accuracy.\n\n* **Comparison with Few-Shot Methods**: The paper focuses on zero-shot performance but does not compare the NEAR score with few-shot methods that involve minimal training (e.g., a few epochs). As noted in \"Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?\" (Mok et al., 2022), it would be beneficial to conduct experiments comparing the zero-shot NEAR score to scores obtained after a few epochs of training. This comparison could substantiate the robustness of the NEAR score and reinforce the claim that it can effectively identify optimal neural networks without training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNLLqKkZDp", "reviewer": "Reviewer_6bBR", "summary": "Zero-cost proxies in Neural Architecture Search aim to predict the performance of neural networks on a target task without training them (hence, zero-cost). The authors point out that the current zero-cost proxies share a few common limitations. First, they show a weaker correlation with the network performance than the number of parameters, which is a naive baseline. Second, they require a pre-defined search space, which may be unclear in many machine learning applications. Third, they have only been applied for finding optimal architectures, and they cannot be used to tune or select other important hyperparameters, such as the activation function and weight initialization scheme. To overcome these limitations, the authors propose a new zero-cost proxy called NEAR (Network Expressivity by Activation Rank) that estimate the optimal layer size for MLP networks without a search space. It can also be used to tune other miscellaneous hyperparameters. The idea behind NEAR consists of two key technical contributions 1) relaxing the definition of the activation patterns to encompass non-ReLU activation functions and 2) utilizing the effective rank of pre- and post-activation matrices to compute the NEAR score. The effectiveness of NEAR is verified empirically through experiments on several NAS benchmarks.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The authors propose a novel zero-cost proxy score that can be utilized to predict not only the optimal architecture of the neural network but also the optimal set of hyperparameters, such as the weight initialization scheme and activation functions. \n\n- The proposed network utilizes the effective rank of pre- and post-activation matrices to predict the final performance of a neural network, which appears to be theoretically sound. Such an approach, however, has been used in previous literature, as noted by the authors in lines 152-154. Therefore, the main technical contribution of NEAR is relaxing the definition of the activation patterns to include various activation functions.\n\n- The authors present sufficient empirical evidence to show the effectiveness of the proposed approach from three main perspectives: 1) high rank correlation on NAS benchmarks, 2) optimal layer size estimation, and 3) activation function and weight initialization selection.", "weaknesses": "- Since the proposed method can be used to predict the performance of non-ReLU-based architectures, it can theoretically be applied to predict the performance of LLMs with GELU activations. Have the authors attempted this? It would be a strong extension of the proposed method especially since the authors argue that the major advantage of the proposed method is its applicability to tasks that do not have a fixed search space. \n\n- The authors argue that this method enables estimation of optimal weight initialization and activation function, but it was not clear after reading the paper why other zero-cost proxies cannot be used for this purpose. It seems that although previous metrics did not explicitly take weight initialization or activation function into consideration, they can theoretically be used for the same purpose. Therefore, it would be helpful if the authors could add a comparison with previous approaches in Tables 7 and 8 to better highlight that only NEAR score can do this.\n\n- I didn't quite follow how relaxing the definition of activation patterns affects the definition of the NEAR score. If the definition isn't relaxed and is constrained only to ReLU activations, how would the NEAR score be formulated? Or would it be impossible to formulate this score in the first place? If there is no difference between the two (theoretically speaking), why did this definition have to be relaxed? Does the NEAR score before and after relaxing this definition show lower/higher correlation with performance? Because this is unclear, I didn't understand how the NEAR is designed to serve all of its intended purposes, such as choosing the optimal number of layers, weight initialization, and activation function. \n\n- In Tables 1-3, The NEAR score only outperforms other metrics only on NAS-Bench-101 and NATS-Bench-SSS (CIFAR-100). Tables 5-8 do not compare the NEAR score against previous approaches. This makes me question whether the NEAR score is as reliable as the authors claim. I would like to see at least one application in which the NEAR score is clearly far more advantageous than other metrics. \n\nIn general, the authors make very strong arguments about the theoretical novelty of the proposed proxy and its usability. However, because of the points stated above, I did not find the current manuscript convincing enough. After reading this paper, I am left wondering in what exact practical application the NEAR score would actually be more useful than previous proxies. If the authors can explicate both theoretically and empirically that the NEAR score does indeed have distinct advantages, I would be willing to adjust my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c810CRlQgL", "reviewer": "Reviewer_qDTu", "summary": "This paper introduces NEAR (Network Expressivity by Activation Rank), a zero-cost proxy (ZCP) that estimates the rank of neural network architectures via the effective rank of pre- and post-activation matrices at initialization (without training). The method is quite generic and is agnostic to the type of activation function or labelled data. Experiments on benchmarks such NAS-Bench-101 and NATS-Bench demonstrate NEAR’s high correlation with model accuracy, often outperforming other proxies.", "questions": "- What is the cost of computing NEAR and how does it compare to the cost of computing the other ZCPs?\n\n- Can the authors comment on the relationship between their findings on the optimal initializaiton scheme and activation functions, and the maximal update parameterization (muP) [1], which enables effective transfer of hyperaparameters across model scales?\n\n- As the efficiency of transformer-based models is becoming more and more relevant as their capacity keeps growing, I would suggest the authors assess the performance of their proxy method on approximating the rank of pruned transformer models. Could the authors evaluate their method on the recent HW-GPT-Bench [2]?\n\n- Do the authors have any more theoretical justification of the effective matrix rank and its ability to assess the neural network generalization, trainability and expressivity at initialization? Is the effective rank of the pre- and post-activation matrices somehow related to the number of linear separable regions in TE-NAS [3]?\n\n-- References --\n\n[1] Yang et al. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. 2022\n\n[2] Sukthanker et al. HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models. In NeurIPS 2024 DBT\n\n[3] Chen et al. Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective. In ICLR 2021", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "I think the direction of using the effective matrix rank as a zero-cost proxy (ZCP) for neural net accuracy is a useful idea and worth pursuing. Some of the results are interesting and the NEAR proxy seems to filter out architectures that perform worst and does provide somehow reliable information on which architectures are in the top 10% best performing ones.", "weaknesses": "Despite the aforementioned positives, I have the following major concerns related to this submission:\n\n- The differences in test loss in Table 7 are not statistically significant and it seems to me that even for these small networks the NEAR score can't determine correctly the ranking of the models. Furthermore, there are only a few tasks and datasets considered for this evaluation, and despite NEAR being able to filter out the bad architectures, typically one is interested in finding the best architecture among the top 10% of architectures in the search space, which this is clearly not the case from the results in Table 8.\n\n- I think that the usage of the standard image classification benchmarks in NAS such as NATS-Bench or NAS-Bench-101, as in this paper, poses the risk of overfitting new proposed methods on these benchmarks. For instance, I would recommend the authors running their method on benchmarks of NAS-Bench-Suite-Zero [1].\n\n- The contributions of the proposed approach are marginal regarding novelty, and despite I find the idea behind using effective rank useful, I still do not fully understand how it can be used to determine effective training and generalization of neural nets at initialization. Can the authors comment on this point further (see questions below as well)?\n\n- In terms of significance, I do believe that the zero-cost proxy research could be beneficial to the community if it is rephrased in the current landscape of LLM compression [2]. I do find the standard NAS benchmarks useful for prototyping accuracy proxies, however I do not see anymore the practical relevance of these benchmarks outside prototyping and fast experimentation. This is my personal opinion though and my recommendation would be to think of ways how ZCP research on CNN models could be rephrased to structural pruning of LLMs (actually the ZCPs were inspired from network pruning methods in the first place).\n\n- Finally, I think the paper could be made easier to read and gain more clarity with less text and a few more figures summarizing some of the main results described in the text.\n\n-- References --\n\n[1] Krishnakumar et al. NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies. In NeurIPS 2022 DBT\n\n[2] Zhu et al. A Survey on Model Compression for Large Language Models. In TACL 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dhbEOCoeGA", "reviewer": "Reviewer_o26N", "summary": "NEAR is a zero-cost proxy that predicts machine learning model performance without training by estimating neural network expressivity through the effective rank of activation matrices. It demonstrates a strong correlation with final model accuracy on NATS-Bench and NAS-Bench-101, ranking first among 13 zero-cost proxies. NEAR requires only input data, not labels, and works with any activation function. It also helps select optimal activation functions, weight initializations, and layer sizes in multi-layer perceptrons.", "questions": "In the case of using ranks of pre and post-activation matrices as indicators of accuracy, a study that verifies this claim would be helpful. The same goes for the convolution part.\n\nSpecifically for the convolution part how accurate is the predicted rank using sampling? What is the minimum sample size required?\n\nFor the results, in cases where the estimations are off, why does that happen? A post-analysis would provide some insights into cases where NEAR is not optimal\n\nIt has been shown in the literature that randomly generated matrices in high dimensions contain almost linearly independent vectors, how does this impact the prediction if we only derive the rank hypothesis only from a few input samples?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper attempts to address a well-established and critical problem of correctly predicting the accuracies of different model architectures using zero-cost proxies.\n\nApplies a good intuition of using ranks of pre and post-activation matrices as indicators of accuracy. \n\nNovel analysis of weight initialization and activation functions.\n\nComprehensive comparison with existing state-of-the-art on relevant benchmarks.", "weaknesses": "Although the authors show good performance of their techniques across benchmarks, it is not clear if the metrics used to determine the model performance compare different architectures correctly. For example, given two model architectures A1 and A2 with test accuracy of A1 being greater than A2,  how often is the case where the rank of pre and post-matrices of A1 greater than A2?\n\nHow well does this generalize to larger models and more complex datasets?\n\nFor convolution operations, there are many hyperparameters like kernel size, stride, dilation, in-channels, out-channels, grouped convolutions, etc. How does the flattening of these matrices and calculating the score take into account all of the above factors? The intuition in this case is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)