NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance

## Abstract

Abstract Artificial neural networks have been shown to be state-of-the-art machine learning models in a wide variety of applications, including natural language processing and image recognition. However, building a performant neural network is a laborious task and requires substantial computing power. Neural Architecture Search (NAS) addresses this issue by an automatic selection of the optimal network from a set of potential candidates. While many NAS methods still require training of (some) neural networks, zero-cost proxies promise to identify the optimal network without training. In this work, we propose the zero-cost proxy Network Expressivity by Activation Rank (NEAR). It is based on the effective rank of the pre- and post-activation matrix, i.e., the values of a neural network layer before and after applying its activation function. We demonstrate the cutting-edge correlation between this network score and the model accuracy on NAS-Bench-101 and NATS-Bench-SSS/TSS. In addition, we present a simple approach to estimate the optimal layer sizes in multi-layer perceptrons. Furthermore, we show that this score can be utilized to select hyperparameters such as the activation function and the neural network weight initialization scheme.

## 1 Introduction

Originally inspired by the structure and function of biological neural networks, artificial neural networks are now a key component in machine learning [, ]. They are applied in many tasks such as natural language processing, speech recognition, and even protein folding [, , ]. Artificial neural networks are built by layers of interconnected neurons, which receive inputs, apply a, typically non-linear, activation function, and pass on the result (see Figure 1). By tuning the weights of the individual inputs to a neuron, a neural network can learn functional relations from sample data to perform predictions without explicit program instructions. Deep learning is obtained if more than one layer is hidden between the first input and last output layer of the neural network. In theory, already a neural network with a single hidden layer and enough neurons is capable of approximating any continuous function up to arbitrary precision [, , ].

Figure: Figure 1: Illustration of an artificial neural network with three inputs $\{x_{i}\}$ and two outputs $\{y_{i}\}$. The single hidden layer consists of five neurons. The activation functions are shown within the neurons. The solid lines represent the weights, while the dashed lines indicate input and output without weights.
Refer to caption: x1.png

However, the choice of the model size, i.e., the number of hidden layers and the number of neurons per layer, has a huge impact on the performance, training time, and computational demand []. A too small network is unable to capture a complex relation in a dataset resulting in underfitting. Conversely, a too large model lacks efficiency and carries the risk of overfitting and poor generalization. Determining the optimal model size that strikes a balance between accuracy and efficiency remains an important challenge []. In common practise, various models of different sizes are trained and their final performances are compared. This manual process is time-consuming and often leads to sub-optimal model architectures.

The goal of Neural Architecture Search (NAS) is to eliminate this manual process and autonomously identify suitable architectures []. Commonly applied techniques for NAS are based on reinforcement learning, evolutionary algorithms, or gradient-based optimization [, , ]. However, these methods are still time- and resource-intensive. Zero-cost proxies have been proposed to improve the efficiency of NAS methods [, , ]. They exploit properties of the untrained network that are correlated with the final accuracy, bypassing the costly training process. In this way, they promise speed-ups of several orders of magnitude compared to traditional NAS algorithms [].

The performance of previously reported zero-cost proxies is highly dependent on the underlying search space []. Furthermore, many of them are unable to demonstrate consistently a higher correlation with the final accuracy than the trivial baseline given by the number of parameters (#Params) []. In addition, these proxies require a search space of known architectures. While the latter is not a severe limitation in applications such as computer vision, in many other applications such search spaces do not exist. An example is a so-called machine learning potential [, ] applied in chemistry and materials science, which employs neural networks to predict the energy of a chemical system as a function of the molecular structure. In this work, we propose the zero-cost proxy Network Expressivity by Activation Rank (NEAR) that can be used in conjunction with a search space, but also provides the possibility to estimate the optimal layer size for multi-layer perceptrons without a search space.

Besides the network architecture, the activation function and weight initialization scheme have a significant impact on the training dynamics and the final performance of a neural network [, ]. A variety of techniques have been developed to select such hyperparameters, including methods based on Bayesian optimization and evolutionary algorithms [, ]. However, these techniques still require expensive training of multiple networks, and the zero-cost proxies reported so far have not been applied to the selection of the activation function and the weight initialization scheme. A zero-cost proxy that can be applied not only to find optimal architectures, but also to choose the activation function and the weight initialization scheme, would allow one to further reduce the computational burden of constructing a performant artificial neural network. Here, we therefore evaluate the suitability of our zero-cost proxy NEAR for this purpose.

In summary, we make the following key contributions:

- •
We propose the zero-cost proxy NEAR and test its performance across several search spaces and datasets.
- •
We introduce a simple method to estimate the optimal layer size of multi-layer perceptrons and show its effectiveness for machine learning potentials.
- •
We empirically demonstrate that our zero-cost proxy NEAR can support the selection of an activation function and a suitable weight initialization scheme.

This work is organized as follows: In Section 2, we discuss previously introduced zero-cost proxies. In Section 3, we show how the effective rank previously published in Ref. [] can be employed to construct a zero-cost proxy. We evaluate the performance of our zero-cost proxy NEAR on several benchmarks in Section 4. Further, we show that NEAR can detect not only optimal architectures in a search space, but also estimate optimal layer sizes in multi-layer perceptrons. Additionally, in Section 4 we demonstrate that NEAR can be applied to choose performant activation functions and weight initializations. Conclusions are then given in Section 5.

## 2 Related Work

Zero-cost NAS proxies are designed to identify optimal neural network architectures without requiring training. Some of the earliest proxies such as Single-Shot Network Pruning Based on Connection Sensitivity (SNIP), Gradient Signal Preservation (GraSP), Fisher, and Synaptic Flow (SynFlow) have been adapted from pruning techniques and are based on gradients of the neural network []. While SNIP assigns scores to individual parameters by approximating the change in loss when the parameter is removed [], the scores of GraSP are based on an approximation of the change in the gradient norm []. The Fisher proxy estimates the importance of an activation by calculating its contribution to the loss [], and SynFlow assigns a score to each parameter indicating how much it contributes to the information flow in the network []. A proxy that has not been adapted from the pruning literature is Grad_norm, which is defined as the Euclidean norm of the gradient []. SNIP, GraSP, SynFlow, and Grad_norm are calculated for each individual parameter, whereas Fisher is calculated for channels within a convolution layer. To obtain a score for the entire network, the values assigned to the parameters or channels are summed. Zero-Shot NAS via Inverse Coefficient of Variation on Gradients (ZiCo) [] is another proxy that exploits gradient information, linking the mean and the variance of the gradient to the convergence rate and the capacity of the neural network. With the exception of SynFlow, all the aforementioned proxies require the presence of target output, so-called labels. Therefore, they can only be applied to a limited extent in unsupervised learning for categorization tasks.

Some zero-cost proxies exist that employ properties of the untrained network that do not require backpropagation and hence work without labels. These include Neural Architecture Search Without Training (NASWOT) [], Zen-Score [], Sample-Wise Activation Patterns (swap) [], Regularized Sample-Wise Activation Patterns (reg_swap) [], and Minimum Eigenvalue of Correlation (MeCo${}_{\text{opt}}$) []. To a certain extent, these proxies can be related to theoretical considerations regarding, for example, the functions that can be represented by a particular neural network or the convergence rate during training. A proxy that combines information from gradients with the output of the untrained network and works without labels is Training-Free Neural Architecture Search (TE-NAS) [].

All these proxies assess the performance of a network based on the output of individual untrained layers. However, most of them, namely NASWOT, Zen-Score, swap, reg_swap, and TE-NAS, can only be applied if the Rectified Linear Unit (ReLU) [] activation function is employed. Our zero-cost proxy NEAR utilizes the output of individual layers before and after the application of the activation function for sample inputs without constraints on the type of activation function. Therefore, its applicability is more general.

## 3 Methods

### 3.1 Multi-Layer Perceptron

An artificial neural network with the architecture described in the introduction (see Figure 1) is also called multi-layer perceptron. A multi-layer perceptron with $L$ layers can be expressed as

$$ $\mathcal{F}(\mathbf{x})=(\sigma\circ\mathbf{W}_{L})\circ(\sigma\circ\mathbf{W} _{L-1})\circ\\ \dotsm\circ(\sigma\circ\mathbf{W}_{2})\circ(\sigma\circ\mathbf{W}_{1})(\mathbf {x})\coloneq\mathbf{y}\ ,$ (1) $$

where $\mathbf{x}$ and $\mathbf{y}$ are the input and output features, $\sigma$ is an activation function, and $\mathbf{W}_{l}\in\mathbb{R}^{N_{l}\times N_{l-1}}$ are the weights of layer $l$. The weights can also include so-called bias weights, which are added to the output of the neuron without being multiplied by any input value. Hence, they effectively shift the activation function input of the neuron.

We define the pre-activation of layer $l$ as

$$ $\mathbf{z}_{l}=(\mathbf{W}_{l})\circ(\sigma\circ\mathbf{W}_{l-1})\circ\dotsm \circ(\sigma\circ\mathbf{W}_{2})\circ(\sigma\circ\mathbf{W}_{1})(\mathbf{x})$ (2) $$

and the post-activation as

$$ $\mathbf{h}_{l}=\sigma(\mathbf{z}_{l})\ .$ (3) $$

### 3.2 Network Expressivity by Activation Rank (NEAR)

The construction of our zero-cost proxy NEAR is inspired by theoretical considerations on machine learning that attempt to assess the expressivity of artificial neural networks [, ]. Therefore, we restate here the most important concepts from these works. For neural networks applying the ReLU activation function, each activation function can be conceptualized as a hyperplane that separates the input space into an active and an inactive region. Both regions are referred to as linear regions. The number of such linear regions can be viewed as a measure of the expressivity of the network [, ]. Following previous studies [, , ], we define the activation pattern for an input $\mathbf{x}$ and a network $\mathcal{F}$ with the ReLU activation function by the set of vectors $\mathcal{A}(\mathbf{x};\mathcal{F})=\{\mathbf{a}_{1},\ldots,\mathbf{a}_{L}\}$ where ${a_{l}}_{i}=1$ if the output of the $i$th neuron in layer $l$ is positive and ${a_{l}}_{i}=0$ otherwise. Given that different linear regions correspond to distinct activation patterns, the number of linear regions is equivalent to the number of activation patterns. Consequently, it is a natural idea to analyze the activation patterns to assess the expressivity of a network. This idea is exploited in several zero-cost proxies [, , , ]. However, constructing a proxy based on the aforementioned definition of the activation pattern will restrict its application to networks applying the ReLU activation function, a drawback that all those proxies have in common.

To build a more general zero-cost proxy, we relax the definition of the activation pattern to enable other activation functions: $\mathcal{A}(\mathbf{x};\mathcal{F})=\{\mathbf{h}_{1},\ldots,\mathbf{h}_{L}\}$, with $\mathbf{h}_{l}$ defined in Eq. (3). While there is no longer a direct relationship to the number of linear regions, we argue that inputs with a very similar activation pattern are more challenging for the network to distinguish, and we leverage this argument to develop our zero-cost proxy. We define the pre-activation matrix for layer $l$ as $\mathbf{Z}_{l}\in\mathbb{R}^{n_{l}\times n_{l}}$, where $n_{l}$ is the number of neurons in layer $l$. Each of the $n_{l}$ rows of $\mathbf{Z}_{l}$ contains $\mathbf{z}_{l}$ for some different sample input $\mathbf{x}_{i}$. In a similar manner, the post-activation matrix for layer $l$ is defined as $\mathbf{H}_{l}\in\mathbb{R}^{n_{l}\times n_{l}}$, where each row contains $\mathbf{h}_{l}$ for a different sample input $\mathbf{x}_{i}$. Intuitively, we would expect a well-performing network when the rows in the pre-/post-activation matrix are very different to each other, while a matrix with all similar rows would indicate a poorly performing network. As an example, we can consider a classification problem in which rows are highly similar yet belong to inputs of different classes. In this case even a minor perturbation can result in a change in the predicted class. Additionally, we would expect that a matrix, in which certain rows have entries that are much larger than the entries in other rows, will result in poorer performance than a matrix with entries of similar size. The reason for this behavior is that the differences in size can cause the predictions to become highly dependent on individual weights and hence on individual input values. These two relationships are captured by the effective rank, defined as follows:

###### Definition 3.1 Effective Rank [ 25 ] .

The effective rank of a matrix $\mathbf{A}\in\mathbb{C}^{M\times N}$, with $Q=\min\{M,N\}$, is given by

$$ $\operatorname{erank}(\mathbf{A})=\exp[H(p_{1},\ldots,p_{Q})]\ ,$ (4) $$

where $H(p_{1},\ldots,p_{Q})$ is the Shannon entropy [],

$$ $H(p_{1},\ldots,p_{Q})=-\sum_{k=1}^{Q}p_{k}\log(p_{k})\ ,$ (5) $$

and $p_{k}$ are normalized singular values,

$$ $p_{k}=\frac{\tilde{\sigma}_{k}}{\sum_{i=1}^{Q}\tilde{\sigma}_{i}}\ ,$ (6) $$

with the singular values $\tilde{\sigma}_{i}$.

In order to interpret the effective rank of a matrix, we can think of the matrix as a linear transformation. The rank of the matrix indicates the number of dimensions of its range. By contrast, the effective rank contains information about the geometrical shaping of the transformation. A linear transformation that causes a strong stretching along one dimension, while all other dimensions remain unchanged, has a lower effective rank than a transformation that causes an equally strong stretching along all dimensions. The rank of the matrix of both transformations is, however, identical. In addition, a linear transformation that stretches each dimension equally is represented by a matrix with orthogonal rows, and rows that are orthogonal to each other could be considered maximally different. In other words, the effective rank measures what we have intuitively explained above as being indicators of good performance. Consequently, we propose the zero-cost proxy NEAR as the following:

###### Definition 3.2 Network Expressivity by Activation Rank (NEAR).

The NEAR score of a neural network is given by

$$ $s=\sum_{l=1}^{L}\operatorname{erank}(\mathbf{Z}_{l})+\operatorname{erank}( \mathbf{H}_{l})\ ,$ $$

where $\mathbf{Z}_{l}$ and $\mathbf{H}_{l}$ denote the pre- and post-activation matrices, respectively.

A higher NEAR score $s$ indicates a better performing network. The calculated score depends on the input samples, which will usually be randomly selected. To prevent large, random fluctuations depending on the selected samples, we recommend to calculate the score multiple times and employ the average. On the NAS benchmarks, we calculated the average of $32$ repetitions, while for the machine learning potential benchmarks, we employed $400$ repetitions due to the more diverse dataset.

###### Definition 3.1 Effective Rank [ 25 ] .

###### Definition 3.2 Network Expressivity by Activation Rank (NEAR).

### 3.3 NEAR for Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are a neural network architecture particularly popular in the field of computer vision []. Our description here focuses on networks applying 2D convolutions. However, the approach can be extended to 3D convolutions as well. For 2D convolutions, the input has the dimensions $W\times H\times C$, where $C$ is the number of channels. For example, $C$ equals $3$ for color images, since these consist of red, green, and blue color channels with dimension $W\times H$ (number of pixels in width $W$ and height $H$). To extract features from the input, a convolution operation is performed (see Figure 2) using one or more filters with dimension $k\times k\times C$, where $k$ is an adjustable hyperparameter. These filters are essentially the weights that are learned during the training process. The results of the convolutions are summed, so that each filter results in a single output matrix, also known as a feature map. The output is passed to the next layer, where new filters are applied, allowing the network to learn increasingly complex features.

Figure: Figure 2: A convolution is performed on the input of dimension $4\times 4\times 3$ with a filter of dimension $2\times 2\times 3$ resulting in a feature map of dimension $3\times 3$.
Refer to caption: x2.png

We employ the feature maps in the construction of the activation matrix to calculate the NEAR score for a CNN. A layer with $C$ input channels results in an output of the form $\mathbf{Z}_{l}\in\mathbb{R}^{W^{\prime}\times H^{\prime}\times C^{\prime}}$, with $W^{\prime}=W-k+1$ and $H^{\prime}=H-k+1$ (in case of no padding and stride of one). $C^{\prime}$ denotes the number of filters in the layer. Since this output is a 3D tensor, the NEAR score cannot be calculated directly. Instead, the output is first transformed into a matrix. We consider the number of filters to be analogous to the number of neurons in a fully connected layer. Therefore, the maximum possible NEAR score should be larger if there are more filters in a layer. This behavior could be achieved by flattening each feature map into a vector and then packing the vectors into a matrix. However, for an image with three channels and $32\times 32$ pixels as input, this approach quickly results in a large matrix. For example, when using a filter of dimension $3\times 3\times 8$, the matrix already is of dimensions $7200\times 7200$, and the effort to compute the NEAR score is no longer negligible. Therefore, we approximate this matrix for the calculation of the NEAR score. First, we flatten the output to get $\mathbf{Z}_{l}\in\mathbb{R}^{W^{\prime}\cdot H^{\prime}\times C^{\prime}}$ (see Figure 3). Second, to reduce the size of the first dimension of this matrix, a random selection of $C^{\prime}$ contiguous rows is made, where the first row contains elements taken from the first row of the feature maps. This approach yields essentially a submatrix of the large matrix containing all the feature maps as vectors. We employ this submatrix as a proxy to compute the NEAR score.

Figure: Figure 3: Process of reshaping convolutional neural network feature maps. (a) The process begins with four $3\times 3$ feature maps. (b) These feature maps are subsequently reshaped to a $9\times 4$ matrix. (c) An activation matrix is given by four contiguous rows, whereby the first row contains elements extracted from the top row of the feature maps. From all possible activation matrices one is randomly selected.
Refer to caption: x3.png

## 4 Results and Discussion

### 4.1 Correlation of Effective Rank and Final Model Accuracy

To examine the performance of our zero-cost proxy NEAR, we evaluated it on the three standard cell-based NAS benchmarks NAS-Bench-101 [], NATS-Bench-SSS, and NATS-Bench-TSS []. We note that “cell-based” refers to the construction of the individual networks, which were obtained by placing stacks of repeated cells in a common skeleton (for details see Refs. [] and []). For comparison, we report the two commonly employed rank correlation measures Kendall’s $\tau$ [] and Spearman’s $\rho$ [] also for twelve other zero-cost proxies. For Grad_norm, SNIP, GraSP, Fisher, and SynFlow we relied on the implementations of Ref. [] and for ZiCo [], MeCo${}_{\text{opt}}$ [], swap [], and reg_swap [] we employed the respective paper’s implementation. All results were recalculated by us. In some cases we observed small differences in the correlation coefficients compared to the values of the original publications. To make our calculations reproducible, the code including all hyperparameter settings is available on Zenodo [].

#### NATS-Bench-TSS

The NATS-Bench-TSS search space consists of $15\,625$ neural network architectures trained on the datasets CIFAR-10, CIFAR-100 [], and ImageNet16-120 []. The correlation between the different proxies and the final test accuracy is shown in Table 1. Some of the previously reported proxies, such as Grad_norm, SNIP, and GraSP, yield a lower correlation than the number of parameters. The latter can be seen as trivial baseline for the zero-cost proxies. The more recently developed proxies ZiCo, MeCo${}_{\text{opt}}$, swap, reg_swap, and our proxy NEAR demonstrate superior performance relative to this baseline. In general, MeCo${}_{\text{opt}}$ achieves the highest correlation across all datasets for NATS-Bench-TSS. Nevertheless, the NEAR and reg_swap results are very close to those of MeCo${}_{\text{opt}}$. However, the comparison to the reg_swap score is not entirely fair because it assumes that architectures with a parameter number close to the chosen parameter $\mu$ perform best, which may not be universally true. MeCo${}_{\text{opt}}$ and NEAR are therefore more generally applicable.

**Table 1: Kendall’s $\tau$ and Spearman’s $\rho$ correlation coefficients for different zero-cost proxies on the NATS-Bench-TSS benchmark. The highest correlation is highlighted in bold face.**
| NATS-Bench-TSS |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | CIFAR-10 | CIFAR-100 | IN16-120 |  |  |  |
| Proxy<br>Correlation | $\tau$ | $\rho$ | $\tau$ | $\rho$ | $\tau$ | $\rho$ |
| Grad_norm [] | $0.48$ | $0.65$ | $0.47$ | $0.64$ | $0.41$ | $0.56$ |
| SNIP [] | $0.48$ | $0.64$ | $0.47$ | $0.63$ | $0.43$ | $0.57$ |
| GraSP [] | $0.39$ | $0.55$ | $0.40$ | $0.57$ | $0.40$ | $0.55$ |
| Fisher [] | $0.41$ | $0.55$ | $0.41$ | $0.56$ | $0.35$ | $0.47$ |
| SynFlow [] | $0.58$ | $0.77$ | $0.57$ | $0.76$ | $0.49$ | $0.67$ |
| Zen-Score [] | $0.32$ | $0.43$ | $0.31$ | $0.42$ | $0.32$ | $0.44$ |
| FLOPs | $0.58$ | $0.75$ | $0.55$ | $0.73$ | $0.51$ | $0.68$ |
| #Params | $0.58$ | $0.75$ | $0.55$ | $0.73$ | $0.49$ | $0.66$ |
| ZiCo [] | $0.58$ | $0.78$ | $0.60$ | $0.80$ | $0.57$ | $0.76$ |
| MeCo${}_{\text{opt}}$ [] | $\mathbf{0.72}$ | $\mathbf{0.90}$ | $\mathbf{0.73}$ | $\mathbf{0.90}$ | $\mathbf{0.67}$ | $\mathbf{0.84}$ |
| swap [] | $0.64$ | $0.82$ | $0.65$ | $0.82$ | $0.58$ | $0.74$ |
| reg_swap [] | $0.71$ | $0.88$ | $0.69$ | $0.87$ | $0.62$ | $0.79$ |
| NEAR | $0.70$ | $0.88$ | $0.69$ | $0.87$ | $0.66$ | $\mathbf{0.84}$ |

#### NATS-Bench-SSS

For the larger NATS-Bench-SSS search space with $32\,768$ architectures trained on CIFAR-10, CIFAR-100, and ImageNet16-120, neither MeCo${}_{\text{opt}}$ nor reg_swap consistently show a higher correlation with the final test accuracy than the number of parameters (Table 2). Only SynFlow, NEAR, and ZiCo are able to surpass the performance of this baseline on all three datasets, with SynFlow demonstrating the highest correlation on CIFAR-10 and ImageNet16-120, and NEAR yielding the highest correlation on CIFAR-100.

**Table 2: Kendall’s $\tau$ and Spearman’s $\rho$ correlation coefficients for different zero-cost proxies on the NATS-Bench-SSS benchmark. The highest correlation is highlighted in bold face.**
| NATS-Bench-SSS |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | CIFAR-10 | CIFAR-100 | IN16-120 |  |  |  |
| Proxy<br>Correlation | $\tau$ | $\rho$ | $\tau$ | $\rho$ | $\tau$ | $\rho$ |
| Grad_norm [] | $0.53$ | $0.72$ | $0.38$ | $0.54$ | $0.52$ | $0.70$ |
| SNIP [] | $0.63$ | $0.82$ | $0.43$ | $0.61$ | $0.61$ | $0.79$ |
| GraSP [] | $0.30$ | $0.43$ | $0.10$ | $0.15$ | $0.33$ | $0.48$ |
| Fisher [] | $0.47$ | $0.65$ | $0.30$ | $0.43$ | $0.43$ | $0.60$ |
| SynFlow [] | $\mathbf{0.79}$ | $\mathbf{0.94}$ | $0.59$ | $0.78$ | $\mathbf{0.81}$ | $\mathbf{0.95}$ |
| Zen-Score [] | $0.75$ | $0.92$ | $0.50$ | $0.69$ | $0.72$ | $0.89$ |
| FLOPs | $0.44$ | $0.61$ | $0.19$ | $0.28$ | $0.41$ | $0.57$ |
| #Params | $0.69$ | $0.87$ | $0.53$ | $0.72$ | $0.68$ | $0.86$ |
| ZiCo [] | $0.72$ | $0.89$ | $0.54$ | $0.74$ | $0.73$ | $0.90$ |
| MeCo${}_{\text{opt}}$ [] | $0.74$ | $0.91$ | $\mathbf{0.62}$ | $0.81$ | $0.61$ | $0.80$ |
| swap [] | $0.48$ | $0.67$ | $0.23$ | $0.34$ | $0.43$ | $0.60$ |
| reg_swap [] | $0.62$ | $0.81$ | $0.38$ | $0.54$ | $0.55$ | $0.73$ |
| NEAR | $0.74$ | $0.91$ | $\mathbf{0.62}$ | $\mathbf{0.82}$ | $0.76$ | $0.92$ |

#### NAS-Bench-101

The NAS-Bench-101 search space consists of $423\,624$ architectures trained on the CIFAR-10 dataset. As shown in Table 3, NEAR achieves the highest correlation with the final accuracy for both metrics. It significantly outperforms MeCo${}_{\text{opt}}$, SynFlow, and reg_swap, which show competitive performance on NATS-Bench-TSS and NATS-Bench-SSS. The second highest correlation for NAS-Bench-101 is shown by the Zen-Score, closely followed by ZiCo. However, Zen-Score yields worse correlation than the number of parameters on NATS-Bench-TSS across all datasets.

**Table 3: Kendall’s $\tau$ and Spearman’s $\rho$ correlation coefficients for different zero-cost proxies on the NAS-Bench-101 benchmark. The highest correlation is highlighted in bold face.**
| NAS-Bench-101 |  |  |
| --- | --- | --- |
| Proxy<br>Correlation | $\tau$ | $\rho$ |
| Grad_norm [] | $-0.17$ | $-0.25$ |
| SNIP [] | $-0.12$ | $-0.17$ |
| GraSP [] | $0.17$ | $0.25$ |
| Fisher [] | $-0.19$ | $-0.28$ |
| SynFlow [] | $0.25$ | $0.37$ |
| Zen-Score [] | $0.47$ | $0.64$ |
| FLOPs | $0.30$ | $0.43$ |
| #Params | $0.30$ | $0.43$ |
| ZiCo [] | $0.45$ | $0.63$ |
| MeCo${}_{\text{opt}}$ [] | $0.35$ | $0.49$ |
| swap [] | $0.31$ | $0.43$ |
| reg_swap [] | $0.30$ | $0.43$ |
| NEAR | $\mathbf{0.52}$ | $\mathbf{0.70}$ |

#### Summary

To compare the performance of the zero-cost proxies across all datasets and search spaces, we provide a ranking. For each combination of search space and dataset, we order the zero-cost proxies according to their Spearman’s $\rho$ value. We identify the position in this order as rank, whereby the proxy of highest correlation is assigned a rank of one. Subsequently, the average over all datasets for each NAS search space and for all search spaces is calculated. For example, NEAR achieves ranks three, one, and two on NATS-Bench-SSS for CIFAR-10, CIFAR-100, and ImageNet16-120, respectively. These rankings result in an average rank of two on this search space. Of the proxies tested, reg_swap, SynFlow, ZiCo, MeCo${}_{\text{opt}}$, and NEAR are on average better than the baseline given by the number of parameters (Table 4). However, only NEAR and ZiCo yield consistently higher correlation than the number of parameters for each combination of search space and dataset, whereby NEAR outperforms ZiCo in every case. For each search space, NEAR is either the best or second best zero-cost proxy highlighting its broad applicability. Overall, NEAR achieves the best rank, followed by MeCo${}_{\text{opt}}$ and ZiCo.

**Table 4: The rank of different zero-cost proxies averaged over all datasets of NATS-Bench-TSS, NATS-Bench-SSS, and NAS-Bench-101 and across all search spaces. The best rank is highlighted in bold face.**
| Average rank |  |  |  |  |
| --- | --- | --- | --- | --- |
| Proxy<br>Search space | TSS | SSS | 101 | All |
| GraSP [] | $9.33$ | $12.00$ | $7.00$ | $10.14$ |
| Fisher [] | $10.00$ | $9.67$ | $10.00$ | $9.86$ |
| Grad_norm [] | $7.67$ | $8.33$ | $9.00$ | $8.14$ |
| FLOPs | $5.67$ | $11.00$ | $5.00$ | $7.86$ |
| SNIP [] | $8.00$ | $6.67$ | $8.00$ | $7.43$ |
| Zen-Score [] | $11.00$ | $4.00$ | $2.00$ | $6.71$ |
| swap [] | $3.33$ | $9.67$ | $5.00$ | $6.29$ |
| #Params | $6.33$ | $5.00$ | $5.00$ | $5.57$ |
| reg_swap [] | $2.00$ | $7.67$ | $5.00$ | $4.86$ |
| SynFlow [] | $5.33$ | $\mathbf{1.67}$ | $6.00$ | $3.86$ |
| ZiCo [] | $3.67$ | $3.67$ | $3.00$ | $3.57$ |
| MeCo${}_{\text{opt}}$ [] | $\mathbf{1.00}$ | $3.67$ | $4.00$ | $2.57$ |
| NEAR | $1.67$ | $2.00$ | $\mathbf{1.00}$ | $\mathbf{1.71}$ |

### 4.2 Estimation of Optimal Layer Size

Inspired by previous work that suggested a power law scaling for the accuracy with respect to the model size for language models [], we analyzed the NEAR score of a single layer of different sizes. We observed that the relative score, i.e., the score divided by the number of neurons, appears to be well represented by a simple power function of the form $f(s)=\alpha+\beta\cdot s^{\gamma}$, where $s$ is the NEAR score (see Figure 4). This behavior is consistent for two different datasets. One is the balanced version of the extended MNIST (EMNIST) dataset []. It contains handwritten digits and letters, with an equal number of examples for each class. It is therefore more challenging than the original MNIST dataset including only digits []. The other is a molecular dataset for a lifelong machine learning potential. This dataset includes target energies and atomic forces for features given by molecular structures. These molecular structures were obtained from various conformations in $42$ different $\mathrm{S}_{\mathrm{N}}2$ reactions ($8\,600$ structures) [].

Figure: Figure 4: A power function fitted to the NEAR score divided by the total number of neurons in the layer. The star marks the first time where the slope is smaller or equal to $0.5\%$ of the slope at $x=1$. The plot has been generated for the experiments on the machine learning potential.
Refer to caption: x4.png

We argue that a decreasing relative NEAR score indicates that the network benefits less from additional neurons and, at a certain threshold, its performance is no longer limited by its size. Our experiments suggest a threshold based on the slope of the power function to work reasonably well. We therefore propose the following approach to estimate the size of individual layers:

- 1.
Calculate the NEAR score for some different layer sizes.
- 2.
Fit a power function to the calculated relative scores.
- 3.
Calculate at which layer size the slope of the fit falls below a certain threshold and apply this layer size for the neural network.

Since the NEAR score of each layer depends only on the previous layers, this approach can also be applied to networks with multiple layers. Starting with the first layer, the size of all layers can be determined iteratively.

#### Machine Learning Potential

We applied our approach to estimate the layer sizes on a three-hidden-layer lifelong machine learning potential, employing the dataset B of Ref. []. Our method predicted a size of $52$, $58$, and $60$ for the first, second, and third hidden layers, respectively, with a threshold of $0.5\%$ of the slope at a layer size of $1$. These sizes are significantly smaller than applied in Ref. [] (which are $102$, $61$, and $44$). While this change reduces the number of parameters from $22\,990$ to $13\,909$, the average test loss only increases from $0.012\pm 0.003$ to $0.015\pm 0.004$. This increase is reflected in a change of the root mean squared error for the energies from $(4.4\pm 1.2)\,\mathrm{meV\,atom}^{-1}$ to $(4.6\pm 1.2)\,\mathrm{meV\,atom}^{-1}$ and for the forces from $(100\pm 8)\,\mathrm{meV}\,\text{\AA}^{-1}$ to $(109\pm 8)\,\mathrm{meV}\,\text{\AA}^{-1}$. These numbers represent the mean and standard deviation over $20$ individual lifelong machine learning potentials, for which the training was started from different randomly initialized neural networks. We note that the test loss depends not only on the capacity of the neural network, but also on the training process. However, NEAR only aims to capture the capacity of a network and does not take into account the training dynamics. Therefore, NEAR alone cannot be expected to determine the perfect size but rather to give a good estimate.

Keeping the size of the first and second hidden layers at $52$ and $58$, respectively, and only varying the size of the third hidden layer showed that the predicted size achieves good performance. Even increasing the layer size to $100$ neurons did not improve the loss (Table 5). However, a slightly smaller layer of $40$ neurons would still be sufficient to achieve the same performance. Another point to note here is that smaller networks require fewer computational resources and are therefore more efficient to evaluate.

**Table 5: The test loss of trained lifelong machine learning potentials with different sizes of the third hidden layer, while the first and second layers are fixed at $52$ and $58$ neurons, respectively. Marked in bold is the architecture predicted by our approach. All experiments were repeated $20$ times with different randomly initialized neural networks. We show the mean and standard deviation (rounded up).**
| Layer size | Test loss | #Params |
| --- | --- | --- |
| $20$ | $0.017\pm 0.003$ | $11\,509$ |
| $40$ | $0.015\pm 0.003$ | $12\,709$ |
| $\mathbf{60}$ | $\mathbf{0.015\pm 0.004}$ | $\mathbf{13\,909}$ |
| $80$ | $0.014\pm 0.003$ | $15\,109$ |
| $100$ | $0.015\pm 0.003$ | $16\,309$ |

Repeating the experiment for a variable second hidden layer showed again that the predicted size has favorable characteristics in terms of accuracy and number of parameters. Reducing the size of the second layer to $38$ neurons, which corresponds to a reduction in the number of parameters by $\sim 16\%$, results in an increase in the loss of $20\%$. Conversely, increasing the neuron number to $78$, which corresponds to an increase in the parameter number by $\sim 16\%$, yields a loss decrease of only $\sim 7\%$ (Table 6).

**Table 6: The test loss of trained lifelong machine learning potentials with different sizes of the second hidden layer, while the first and third layers are fixed at $52$ and $60$ neurons, respectively. Marked in bold is the architecture predicted by our approach. All experiments were repeated $20$ times with different randomly initialized neural networks. We show the mean and standard deviation (rounded up).**
| Layer size | Test loss | #Params |
| --- | --- | --- |
| $18$ | $0.022\pm 0.004$ | $9\,389$ |
| $38$ | $0.018\pm 0.004$ | $11\,649$ |
| $\mathbf{58}$ | $\mathbf{0.015\pm 0.004}$ | $\mathbf{13\,909}$ |
| $78$ | $0.014\pm 0.004$ | $16\,169$ |
| $98$ | $0.014\pm 0.003$ | $18\,429$ |

#### Balanced Extended MNIST

In a second experiment, we estimated the layer sizes for a two-hidden layer neural network on the balanced EMNIST dataset. Again, we applied a threshold of $0.5\%$ of the slope at a layer size of $1$. The resulting sizes were $200$ for both, first and second hidden layer. A comparison of the test loss to a network with a reduced second layer size of $120$, which decreases the parameter number by $\sim 10\%$, shows an loss increase of $\sim 4\%$. The aforementioned loss values correspond to a classification accuracy of $86.85\%$ and $87.30\%$. Conversely, an increase in the neuron number to $280$ for the second layer, which increases the parameter number by $\sim 9\%$, results in a decrease in the loss of around $1\%$ and an increase in the accuracy to $87.34\%$. This observation suggests that a size of 200 represents a reasonable compromise between efficiency and accuracy (Table 7).

**Table 7: The test loss on the balanced EMNIST dataset for different sizes of the second layer, while the first layer has a fixed size of $200$ neurons. Marked in bold is the architecture predicted by our approach. All experiments were repeated $20$ times with different randomly initialized neural networks. We show the mean and the standard deviation (rounded up).**
| Layer size | Test loss | #Params |
| --- | --- | --- |
| $40$ | $0.433\pm 0.017$ | $166\,967$ |
| $80$ | $0.397\pm 0.011$ | $176\,887$ |
| $120$ | $0.383\pm 0.009$ | $186\,807$ |
| $160$ | $0.376\pm 0.009$ | $196\,727$ |
| $\mathbf{200}$ | $\mathbf{0.368\pm 0.006}$ | $\mathbf{206\,647}$ |
| $240$ | $0.364\pm 0.006$ | $216\,567$ |
| $280$ | $0.364\pm 0.008$ | $226\,487$ |
| $320$ | $0.363\pm 0.009$ | $236\,407$ |
| $360$ | $0.359\pm 0.007$ | $246\,327$ |

To check whether also the prediction for the first layer was reasonable, we fixed the size of the second layer to $200$ and train networks with various sizes for the first layer (Table 8). A comparison of the test loss for a network with a size of $160$ ($\sim 20\%$ decrease in the parameter number) yields an increase in the loss of $\sim 1.3\%$ and a decrease in the accuracy from $87.30\%$ to $87.14\%$. Even when the size of the first layer was increased to $260$ ($\sim 30\%$ increase in the parameter number), the loss can only be reduced by $\sim 0.7\%$ and the accuracy is increased to $87.32\%$. Again, the predicted size appears to offer a good trade-off between the number of parameters and the accuracy.

**Table 8: The test loss on the balanced EMNIST dataset for different sizes of the first layer, while the second layer has a fixed size of $200$ neurons. Marked in bold is the architecture predicted by our approach. All experiments were repeated $20$ times with different randomly initialized neural networks. We show the mean and the standard deviation (rounded up).**
| Layer size | Test loss | #Params |
| --- | --- | --- |
| $120$ | $0.393\pm 0.012$ | $127\,847$ |
| $140$ | $0.380\pm 0.006$ | $147\,547$ |
| $160$ | $0.373\pm 0.009$ | $167\,247$ |
| $180$ | $0.368\pm 0.007$ | $186\,947$ |
| $\mathbf{200}$ | $\mathbf{0.368\pm 0.006}$ | $\mathbf{206\,647}$ |
| $220$ | $0.363\pm 0.007$ | $226\,347$ |
| $240$ | $0.359\pm 0.006$ | $246\,047$ |
| $260$ | $0.366\pm 0.007$ | $265\,747$ |

### 4.3 Estimation of Activation Function and Weight Initialization Performance

Numerous NAS benchmarks, including NATS-Bench-TSS, NATS-Bench-SSS, and NAS-Bench-101, apply the ReLU activation function for all networks. There is no attention for zero-cost proxies to detect favorable activation functions or weight initialization schemes. There are even proxies that are specifically designed for the ReLU activation function [, , ]. However, it is well known that choosing an appropriate activation function and weight initialization scheme is crucial for achieving good model performance [, , ]. To assess whether NEAR can be employed to select an activation function and a weight initialization scheme, we calculated the score before training and compared it with the final model performance.

#### Machine Learning Potential

We previously reported that the activation function $\operatorname{sTanh}(x)=1.59223\cdot\tanh(x)$ with a tailored weight initialization scheme improves the final accuracy of a lifelong machine learning potential []. We repeated the training for the same dataset and the same neural network architecture (number of neurons per layer: $133-102-61-44-1$) employing the CoRe optimizer [, , ].

Table 9 shows that the sTanh activation function with the tailored weight initialization scheme achieves the lowest test loss, while also exhibiting the largest NEAR score. Comparing the two weight initialization schemes sTanh and Kaiming uniform [] demonstrates that the former improves performance and that the NEAR score can be applied as a reliable predictor for the more suitable weight initialization scheme.

A comparison of the loss and NEAR score of different activation functions applying the same weight initialization scheme also shows that the score can guide the selection of an appropriate activation function. For example, the activation function sTanh with the customized initialization scheme exhibits the lowest loss and the highest NEAR score, while the activation function Tanhshrink shows the highest loss and lowest NEAR score for this weight initialization scheme. However, the discrepancies in loss for the three best activation functions sTanh, Sigmoid Linear Unit (SiLU) [], and Tanh are minimal, whereas the differences in the NEAR score would suggest larger differences. The lowest NEAR score is obtained by Tanhshrink with Kaiming initialization, which is also consistent with the fact that this combination achieves the highest average loss. Hence, NEAR is capable of identifying more and less effective activation functions, while the ordering in close cases cannot be guaranteed.

Since the loss function of the lifelong machine learning potential includes derivative constraints, the second derivative of the activation function appears in the gradient calculation. Consequently, only activation functions with a non-zero second derivative can be considered, resulting in the omission of the ReLU function in this experiment. This issue once again highlights the benefit of NEAR that it can be applied to any activation function.

**Table 9: Comparison of the test loss (after training) and the NEAR score (before training) for various activation functions $\sigma$ and weight initialization schemes. The table is sorted by ascending test loss. All experiments were repeated $20$ times with different randomly initialized neural networks and a random selection of inputs employed to calculate the NEAR score in each repetition. We report the mean and the standard deviation (rounded up).**
| $\sigma$, initialization | Test loss | NEAR score |
| --- | --- | --- |
| sTanh, sTanh | $0.012\pm 0.003$ | $70.8\pm 0.2$ |
| SiLU, sTanh | $0.013\pm 0.004$ | $63.1\pm 0.3$ |
| Tanh, sTanh | $0.015\pm 0.003$ | $67.2\pm 0.3$ |
| Tanhshrink, sTanh | $0.027\pm 0.006$ | $45.2\pm 0.2$ |
| SiLU, Kaiming | $0.031\pm 0.008$ | $57.1\pm 2.5$ |
| sTanh, Kaiming | $0.041\pm 0.019$ | $60.3\pm 2.5$ |
| Tanh, Kaiming | $0.050\pm 0.022$ | $58.5\pm 2.3$ |
| Tanhshrink, Kaiming | $0.082\pm 0.014$ | $38.7\pm 1.8$ |

#### Balanced Extended MNIST

To show that the NEAR score can be utilized to guide the selection of the activation function and weight initialization scheme across various datasets and independent of the optimizer, we report the test loss and the NEAR score on the balanced EMNIST dataset. We employed a multi-layer perceptron with two hidden layers of size $200$ in combination with the Adam optimizer []. We applied early stopping with a patience of $10$ epochs, whereby $10\%$ of the training set served as validation set.

We report the test loss and the NEAR score for the initialization schemes Xavier uniform [], Kaiming uniform, and uniform and the activation functions SiLU, ReLU, Tanh, and Tanhshrink in Table 10. The NEAR score indicates that the Xavier initialization is the most effective, followed by the Kaiming initialization and the uniform initialization. While the uniform initialization is undoubtedly the least favorable in terms of resulting test loss, the distinction between Xavier and Kaiming is not as clear-cut. For the activation functions under consideration, the average loss is found to be lower with the Kaiming initialization than the Xavier initialization, but the difference is marginal and the standard deviations are higher than the difference.

For the Kaiming and Xavier initializations, the activation functions SiLU, ReLU, and Tanh result in similar test losses, while the NEAR score gives the impression that there is a clear ordering for a given initialization. As the orderings of the test losses and NEAR scores do not agree for both initializations, we conclude that these small differences in the resulting loss cannot be captured by the NEAR score. We note here again that the NEAR score only accounts for the expressivity of the network and does not take into account the training process. Still, the NEAR score can reliably predict the worst performing activation function (Tanhshrink). For this activation function, the test loss is also significantly different from those of the others. Analogously, the NEAR score can correctly identify that SiLU, Tanhshrink, and ReLU lead to a lower loss than Tanh if the uniform initialization is applied. Hence, NEAR is able to predict whether an activation function and weight initialization scheme performs well or bad, and only small differences in the resulting loss are not resolved.

**Table 10: Comparison of the test loss (after training) and the NEAR score (before training) for various activation functions $\sigma$ and weight initialization schemes on the balanced EMNIST dataset. The table is sorted by ascending test loss. All experiments were repeated $20$ times with different randomly initialized neural networks and a random selection of inputs employed to calculate the NEAR score in each repetition. We report the mean and the standard deviation (rounded up). For “SiLU, uniform” one outlier was excluded from the mean and standard deviation.**
| $\sigma$, initialization | Test loss | NEAR score |
| --- | --- | --- |
| SiLU, Kaiming | $0.367\pm 0.008$ | $382.8\pm 5.0$ |
| SiLU, Xavier | $0.368\pm 0.008$ | $417.7\pm 0.6$ |
| ReLU, Kaiming | $0.372\pm 0.010$ | $403.7\pm 7.4$ |
| ReLU, Xavier | $0.374\pm 0.008$ | $421.1\pm 0.8$ |
| Tanh, Kaiming | $0.392\pm 0.010$ | $398.2\pm 5.1$ |
| Tanh, Xavier | $0.396\pm 0.011$ | $415.5\pm 0.4$ |
| Tanhshrink, Kaiming | $0.447\pm 0.022$ | $278.4\pm 3.2$ |
| Tanhshrink, Xavier | $0.449\pm 0.018$ | $401.4\pm 1.2$ |
| SiLU, uniform | $0.525\pm 0.040$ | $13.3\pm 0.1$ |
| Tanhshrink, uniform | $0.602\pm 0.030$ | $13.4\pm 0.1$ |
| ReLU, uniform | $0.651\pm 0.047$ | $13.3\pm 0.1$ |
| Tanh, uniform | $3.858\pm 0.030$ | $9.0\pm 0.1$ |

## 5 Conclusion

In this work, we have introduced NEAR which is a zero-cost proxy to pre-estimate the performance of a machine learning model without training. For this purpose, the NEAR score estimates the expressivity of neural networks by calculating the effective rank of the pre- and post-activation matrix. We have demonstrated its strong correlation with the final model accuracy on NATS-Bench-SSS/TSS and NAS-Bench-101. In a ranking of 13 zero-cost proxies by their Spearman’s correlation coefficient, NEAR achieves an average rank of $1.7$ across all datasets and search spaces, while the second-best proxy MeCo${}_{\text{opt}}$ has only an average rank of $2.6$. In contrast to other proxies, NEAR only requires sample input data, but no output labels and it is not bound to a specific activation function. In fact, we have shown that it can even be applied to choose a well performing activation function and weight initialization scheme. Moreover, NEAR is not limited to a given search space and can be applied to estimate the optimal layer size in multi-layer perceptrons. This task represents a significant challenge in the construction of neural networks, and hence, NEAR can help substantially reduce invested human time and computational demand.

## Code Availability

The NEAR software is available on GitHub ([https://github.com/ReiherGroup/NEAR](https://github.com/ReiherGroup/NEAR)).

## Acknowledgments

This work was created as part of NCCR Catalysis (grant number 180544), a National Centre of Competence in Research funded by the Swiss National Science Foundation. M.E. was supported by an ETH Zurich Postdoctoral Fellowship.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * **Theoretical Advantages over Existing Methods**: Could the authors provide a more detailed theoretical analysis comparing the NEAR score to existing zero-cost proxies? Specifically, what theoretical advantages does the NEAR score offer that may account for its superior performance? Highlighting any novel insights or mechanisms by which NEAR improves upon prior methods would strengthen the contribution.

* **Justification for Redefining Definition 3.1**: Since Definition 3.1 (Effective Rank) was established in previous studies, can the authors clarify why redefining it here is necessary or beneficial in the context of their method? Providing justification or modifications that are specific to their approach would help readers understand the significance of this definition in their work.

* **Layer-wise Contribution to NEAR Score**: To understand the impact of each layer on the NEAR score, could the authors conduct experiments exploring how the score changes when calculated on different subsets of layers? For instance, does excluding certain layers significantly affect the correlation with final model accuracy? Discussing these findings could provide insights into optimizing computational efficiency without compromising accuracy.

* **Stability of NEAR Score After Minimal Training**: In line with observations from Mok et al. (2022) (1), have the authors considered evaluating the stability of the NEAR score after a few epochs of training? Conducting experiments that compare the zero-shot NEAR score to scores obtained after 2-3 epochs could demonstrate the robustness of the method and whether minimal training affects its predictive capabilities.

* **Comparison with Few-Shot Performance**: How does the zero-shot NEAR score compare with few-shot methods in terms of computational cost and accuracy? Including a comparison could help position the NEAR score within the broader context of NAS methods and provide evidence for its practical advantages.

**References**
1. “Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?” (Mok et al., 2022)
soundness: 3
strengths: * **Applicability Across Architectures**: A key contribution of this work is that the proxy score is applicable regardless of architecture structure or activation function, providing robustness and versatility.
* **Benchmark Performance**: The authors demonstrated the method’s efficacy across standard NAS benchmarks, showing its comparative strength against other zero-cost proxies.
* **Efficiency**: By not requiring separate training, the method significantly reduces computational overhead, which is valuable for practical NAS applications.
summary: This paper presents a method for identifying optimal architectures in Neural Architecture Search (NAS) using a zero-cost proxy score derived from the activation rank, without requiring separate training. To achieve this, the authors calculate both pre- and post-activation ranks and introduce a sampling method to approximate matrices for efficiency. Additionally, the paper thoroughly discusses the potential of the proposed approach using effective rank.
weaknesses: * **Lack of Theoretical Clarification**: While the experimental results demonstrate the effectiveness of the NEAR score, the theoretical significance of using the effective rank (erank) in Definition 3.2 remains somewhat unclear. Numerous NAS studies have previously explored concepts related to rank. It would strengthen the paper if the authors could provide a more detailed theoretical analysis comparing NEAR to existing methods, highlighting any novel insights their approach offers. Additionally, clarifying why redefining Definition 3.1 is necessary or beneficial in this context would enhance the theoretical foundation of the work.

* **Evaluation Across Layers**: It is not entirely clear whether calculating the NEAR score across all layers is necessary to achieve optimal performance. The paper does not present experiments exploring how the NEAR score changes when calculated on different subsets of layers. Conducting such experiments could provide insights into the contributions of individual layers to the overall score and discuss the implications for computational efficiency and accuracy.

* **Comparison with Few-Shot Methods**: The paper focuses on zero-shot performance but does not compare the NEAR score with few-shot methods that involve minimal training (e.g., a few epochs). As noted in "Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?" (Mok et al., 2022), it would be beneficial to conduct experiments comparing the zero-shot NEAR score to scores obtained after a few epochs of training. This comparison could substantiate the robustness of the NEAR score and reinforce the claim that it can effectively identify optimal neural networks without training.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses section.
soundness: 3
strengths: - The authors propose a novel zero-cost proxy score that can be utilized to predict not only the optimal architecture of the neural network but also the optimal set of hyperparameters, such as the weight initialization scheme and activation functions. 

- The proposed network utilizes the effective rank of pre- and post-activation matrices to predict the final performance of a neural network, which appears to be theoretically sound. Such an approach, however, has been used in previous literature, as noted by the authors in lines 152-154. Therefore, the main technical contribution of NEAR is relaxing the definition of the activation patterns to include various activation functions.

- The authors present sufficient empirical evidence to show the effectiveness of the proposed approach from three main perspectives: 1) high rank correlation on NAS benchmarks, 2) optimal layer size estimation, and 3) activation function and weight initialization selection.
summary: Zero-cost proxies in Neural Architecture Search aim to predict the performance of neural networks on a target task without training them (hence, zero-cost). The authors point out that the current zero-cost proxies share a few common limitations. First, they show a weaker correlation with the network performance than the number of parameters, which is a naive baseline. Second, they require a pre-defined search space, which may be unclear in many machine learning applications. Third, they have only been applied for finding optimal architectures, and they cannot be used to tune or select other important hyperparameters, such as the activation function and weight initialization scheme. To overcome these limitations, the authors propose a new zero-cost proxy called NEAR (Network Expressivity by Activation Rank) that estimate the optimal layer size for MLP networks without a search space. It can also be used to tune other miscellaneous hyperparameters. The idea behind NEAR consists of two key technical contributions 1) relaxing the definition of the activation patterns to encompass non-ReLU activation functions and 2) utilizing the effective rank of pre- and post-activation matrices to compute the NEAR score. The effectiveness of NEAR is verified empirically through experiments on several NAS benchmarks.
weaknesses: - Since the proposed method can be used to predict the performance of non-ReLU-based architectures, it can theoretically be applied to predict the performance of LLMs with GELU activations. Have the authors attempted this? It would be a strong extension of the proposed method especially since the authors argue that the major advantage of the proposed method is its applicability to tasks that do not have a fixed search space. 

- The authors argue that this method enables estimation of optimal weight initialization and activation function, but it was not clear after reading the paper why other zero-cost proxies cannot be used for this purpose. It seems that although previous metrics did not explicitly take weight initialization or activation function into consideration, they can theoretically be used for the same purpose. Therefore, it would be helpful if the authors could add a comparison with previous approaches in Tables 7 and 8 to better highlight that only NEAR score can do this.

- I didn't quite follow how relaxing the definition of activation patterns affects the definition of the NEAR score. If the definition isn't relaxed and is constrained only to ReLU activations, how would the NEAR score be formulated? Or would it be impossible to formulate this score in the first place? If there is no difference between the two (theoretically speaking), why did this definition have to be relaxed? Does the NEAR score before and after relaxing this definition show lower/higher correlation with performance? Because this is unclear, I didn't understand how the NEAR is designed to serve all of its intended purposes, such as choosing the optimal number of layers, weight initialization, and activation function. 

- In Tables 1-3, The NEAR score only outperforms other metrics only on NAS-Bench-101 and NATS-Bench-SSS (CIFAR-100). Tables 5-8 do not compare the NEAR score against previous approaches. This makes me question whether the NEAR score is as reliable as the authors claim. I would like to see at least one application in which the NEAR score is clearly far more advantageous than other metrics. 

In general, the authors make very strong arguments about the theoretical novelty of the proposed proxy and its usability. However, because of the points stated above, I did not find the current manuscript convincing enough. After reading this paper, I am left wondering in what exact practical application the NEAR score would actually be more useful than previous proxies. If the authors can explicate both theoretically and empirically that the NEAR score does indeed have distinct advantages, I would be willing to adjust my score.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - What is the cost of computing NEAR and how does it compare to the cost of computing the other ZCPs?

- Can the authors comment on the relationship between their findings on the optimal initializaiton scheme and activation functions, and the maximal update parameterization (muP) [1], which enables effective transfer of hyperaparameters across model scales?

- As the efficiency of transformer-based models is becoming more and more relevant as their capacity keeps growing, I would suggest the authors assess the performance of their proxy method on approximating the rank of pruned transformer models. Could the authors evaluate their method on the recent HW-GPT-Bench [2]?

- Do the authors have any more theoretical justification of the effective matrix rank and its ability to assess the neural network generalization, trainability and expressivity at initialization? Is the effective rank of the pre- and post-activation matrices somehow related to the number of linear separable regions in TE-NAS [3]?

-- References --

[1] Yang et al. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. 2022

[2] Sukthanker et al. HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models. In NeurIPS 2024 DBT

[3] Chen et al. Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective. In ICLR 2021
soundness: 2
strengths: I think the direction of using the effective matrix rank as a zero-cost proxy (ZCP) for neural net accuracy is a useful idea and worth pursuing. Some of the results are interesting and the NEAR proxy seems to filter out architectures that perform worst and does provide somehow reliable information on which architectures are in the top 10% best performing ones.
summary: This paper introduces NEAR (Network Expressivity by Activation Rank), a zero-cost proxy (ZCP) that estimates the rank of neural network architectures via the effective rank of pre- and post-activation matrices at initialization (without training). The method is quite generic and is agnostic to the type of activation function or labelled data. Experiments on benchmarks such NAS-Bench-101 and NATS-Bench demonstrate NEAR’s high correlation with model accuracy, often outperforming other proxies.
weaknesses: Despite the aforementioned positives, I have the following major concerns related to this submission:

- The differences in test loss in Table 7 are not statistically significant and it seems to me that even for these small networks the NEAR score can't determine correctly the ranking of the models. Furthermore, there are only a few tasks and datasets considered for this evaluation, and despite NEAR being able to filter out the bad architectures, typically one is interested in finding the best architecture among the top 10% of architectures in the search space, which this is clearly not the case from the results in Table 8.

- I think that the usage of the standard image classification benchmarks in NAS such as NATS-Bench or NAS-Bench-101, as in this paper, poses the risk of overfitting new proposed methods on these benchmarks. For instance, I would recommend the authors running their method on benchmarks of NAS-Bench-Suite-Zero [1].

- The contributions of the proposed approach are marginal regarding novelty, and despite I find the idea behind using effective rank useful, I still do not fully understand how it can be used to determine effective training and generalization of neural nets at initialization. Can the authors comment on this point further (see questions below as well)?

- In terms of significance, I do believe that the zero-cost proxy research could be beneficial to the community if it is rephrased in the current landscape of LLM compression [2]. I do find the standard NAS benchmarks useful for prototyping accuracy proxies, however I do not see anymore the practical relevance of these benchmarks outside prototyping and fast experimentation. This is my personal opinion though and my recommendation would be to think of ways how ZCP research on CNN models could be rephrased to structural pruning of LLMs (actually the ZCPs were inspired from network pruning methods in the first place).

- Finally, I think the paper could be made easier to read and gain more clarity with less text and a few more figures summarizing some of the main results described in the text.

-- References --

[1] Krishnakumar et al. NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies. In NeurIPS 2022 DBT

[2] Zhu et al. A Survey on Model Compression for Large Language Models. In TACL 2024

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: In the case of using ranks of pre and post-activation matrices as indicators of accuracy, a study that verifies this claim would be helpful. The same goes for the convolution part.

Specifically for the convolution part how accurate is the predicted rank using sampling? What is the minimum sample size required?

For the results, in cases where the estimations are off, why does that happen? A post-analysis would provide some insights into cases where NEAR is not optimal

It has been shown in the literature that randomly generated matrices in high dimensions contain almost linearly independent vectors, how does this impact the prediction if we only derive the rank hypothesis only from a few input samples?
soundness: 3
strengths: The paper attempts to address a well-established and critical problem of correctly predicting the accuracies of different model architectures using zero-cost proxies.

Applies a good intuition of using ranks of pre and post-activation matrices as indicators of accuracy. 

Novel analysis of weight initialization and activation functions.

Comprehensive comparison with existing state-of-the-art on relevant benchmarks.
summary: NEAR is a zero-cost proxy that predicts machine learning model performance without training by estimating neural network expressivity through the effective rank of activation matrices. It demonstrates a strong correlation with final model accuracy on NATS-Bench and NAS-Bench-101, ranking first among 13 zero-cost proxies. NEAR requires only input data, not labels, and works with any activation function. It also helps select optimal activation functions, weight initializations, and layer sizes in multi-layer perceptrons.
weaknesses: Although the authors show good performance of their techniques across benchmarks, it is not clear if the metrics used to determine the model performance compare different architectures correctly. For example, given two model architectures A1 and A2 with test accuracy of A1 being greater than A2,  how often is the case where the rank of pre and post-matrices of A1 greater than A2?

How well does this generalize to larger models and more complex datasets?

For convolution operations, there are many hyperparameters like kernel size, stride, dilation, in-channels, out-channels, grouped convolutions, etc. How does the flattening of these matrices and calculating the score take into account all of the above factors? The intuition in this case is unclear.

Meta-review: The paper presents a zero-cost proxy technique (NEAR) for predicting neural network performances. The proposed method leverages the effective rank of pre- and post-activation matrices to estimate model accuracy. The authors demonstrate good correlations across benchmarks like NAS-Bench-101 and NATS-Bench-SSS/TSS. The authors provide extensive experiments showing NEAR’s computational efficiency and robustness.

The paper is a borderline case and reviewers are mixed in their assessments of the merits and weaknesses of the work. 

In terms of weaknesses, the theoretical basis for NEAR’s correlation with performance is underdeveloped, with only superficial connections to rank collapse and linear region theory. Its generalizability to large-scale architectures, such as transformers in LLMs, needs more validation, though preliminary results on HW-GPT-Bench are promising. The intuition behind the method’s handling of convolutional layers and its reliance on correlations without deeper ranking consistency analyses also leave room for improvement. 

However, on the positive side, the paper offers a compelling empirical contribution with practical relevance, justifying acceptance despite areas needing further exploration.

As a result, I recommend accepting the paper, but I urge the reviewers to consider the comments in the camera ready.

Decision: Accept (Poster)