EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
Authors: Yash Akhauri
Juan Pablo Munoz
Nilesh Jain
Ravishankar Iyer
Venue: NeurIPS 2022 Accept (2022)
Keywords: Zero Shot Neural Architecture Search
Neural Architecture Search
Program Synthesis
Evolutionary Search
Genetic Programming

Abstract: Neural Architecture Search (NAS) has significantly improved productivity in the design and deployment of neural networks (NN). As NAS typically evaluates multiple models by training them partially or completely, the improved productivity comes at the cost of significant carbon footprint. To alleviate this expensive training routine, zero-shot/cost proxies analyze an NN at initialization to generate a score, which correlates highly with its true accuracy. Zero-cost proxies are currently designed by experts conducting multiple cycles of empirical testing on possible algorithms, datasets, and neural architecture design spaces. This experimentation lowers productivity and is an unsustainable approach towards zero-cost proxy design as deep learning use-cases diversify in nature. Additionally, existing zero-cost proxies fail to generalize across neural architecture design spaces. In this paper, we propose a genetic programming framework to automate the discovery of zero-cost proxies for neural architecture scoring. Our methodology efficiently discovers an interpretable and generalizable zero-cost proxy that gives state of the art score-accuracy correlation on all datasets and search spaces of NASBench-201 and Network Design Spaces (NDS). We believe that this research indicates a promising direction towards automatically discovering zero-cost proxies that can work across network architecture design spaces, datasets, and tasks.

EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
Yash A
J. Pablo Muñoz² Nilesh Jain²
Ravi Iyer?
Cornell University ya255@corne11 edu (pablo. mwnoz, milesh jashh.jjaaae jain ravi iyer]
Intel Labs
Abstract
Neural Architecture Search (NAS) has significantly improved productivity in the design and deployment of neural networks (NN). As NAS typically evaluates multiple models by training them partially or completely, the improved productivity comes at the cost significant carbon footprint. To alleviate this expensive training routine, zero-shot/cost proxies analyze an NN initialization generate a score, which correlates highly with its true accuracy. Zero-cost proxies are currently designed by experts conducting multiple cycles of empirical testing possible algorithms, datasets, and neural architecture design spaces. This experimentation lowers productivity and is an unsustainable approach towards zero-cost proxy design as deep learning use-cases diversify in nature. Additionally, existing zero- cost proxies fail generalize across neural architecture design spaces. In this paper, we propose genetic programming framework to automate the discovery of zero-cost proxies for neural architecture scoring. Our methodology efficiently discovers an interpretable and generalizable zero-cost proxy that gives state of the art score-accuracy correlation on all datasets and search spaces NASBench- 201 and Network Design Spaces (NDS). We believe that this research indicates a promising direction towards automatically discovering zero-cost proxies that can work across network architecture design spaces, datasets, and tasks.
Introduction
The manual trial-and-error method of designing neural network architectures and assessing whether it meets performance and accuracy requirements is not scalable to complex neural architecture design spaces and hardware deployment scenarios. Neural Architecture Search is intended to address this problem of iterative and expensive design by automating the search of neural network architectures. Enabling such automation requires accurate and efficient prediction of the accuracy of candidate neural network architectures. The process of predicting the accuracy often involves partial or complete training of many neural network architectures in the search space. Such methods infeasible for very large architecture design spaces, as considerable sampling and training of neural networks would be required sufficiently describe the entire design space. Popular NAS techniques often involve generating and training neural network architectures and using this accuracy to train accuracy predictors serve feedback in the search algorithm. This architecture generation and feedback methodology can be leveraged by algorithms such as reinforcement learning (Luo et al. (019)) and
evolutionary algorithms (Real et al. (2019). One-shot methods (Cai al. [2019] 2020 Xie al.
22000, Yang et al. [2020], Liu et al. 2019 maintain a super-network, from which sub-networks are sampled. The weights are shared between the sub-networks which reduces NAS cost as there are lesser parameters to train.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The primary challenge of Neural Architecture Search is the evaluation of candidate architectures, which can take hours to days to estimate (Abdelfattah et al. (202T). Further, practical deployment of neural networks is no longer limited to accuracy-oriented neural architecture design. The need for efficient deployment has given rise to significant literature in co-design of hardware and neural network architectures (Choi et al. [202T]. + Akhauri et al. (202T]. Zhang et al. (2020)) However, hardware performance metrics in such co-design tasks are often non-differentiable in nature due to factors such as memory access patterns and cache hierarchy. The interaction between the workload and the hardware makes this optimization problem significantly more complex, and being able to score neural network architectures without the expensive training step would allow for the development of more efficient co-design algorithms.
Most zero-cost proxies utilize a single minibatch of data and a single forward/backward propagation pass to score a neural network (Abdelfattah et al. (202T]. )) We refer to algorithms that score neural network architectures without training as Zero-Cost Neural Architecture Scoring Metrics (ZC-NASM). Design of existing ZC-NASMs are driven by human intuition or are theoretically inspired. These algorithms attempt to quantify the trainability and expressivity of neural networks. The primary hurdle with existing ZC-NASM design is that they are not generalizable to different neural architecture design spaces.
In this paper, we introduce a genetic programming driven methodology automatically discover ZC-NASMs that are interpretable, generalizable and deliver state of the art score-accuracy correlation. We refer to our method as EZNAS (Evolutionarily Generated Zero-Cost Neural Architecture Scoring Metric). Our approach is:
Interpretable: We discover ZC-NASMs expression trees that explicitly indicate which NN features and mathematical operations are being utilized.
Generalizable: The ZC-NASM discovered by EZNAS delivers SoTA score-accuracy correlation on unseen neural architecture design spaces and datasets (NASBench-201, NDS & NATS-Bench). Our framework utilizes simple mathematical operations and an expression tree structure that can be trivially extended to implement arbitrarily complex ZC-NASMs
Efficient: We are able to discover our SoTA ZC-NASM on an Intel(R) Xeon(R) Gold 6242 CPU in under 24 hours. This requires 12.5x lesser CO2e than a NAS search. Our 7C-NASM is generalizable to multiple design spaces, increasing the end-to-end efficiency of NN architecture search by over two orders of magnitude.
2 Related Work
Neural Architecture Search (NAS) was proposed to reduce the human effort that goes into manually designing complex neural network architectures. Early efforts in the field of NAS adopted brute force methods by training candidate architectures and using the obtained accuracy as proxy to discover better architectures. AmoebaNet (Real et al. (2019) utilized evolutionary algorithms (EA) and 3150 GPU days of compute to achieve 74.5% top- accuracy on the ImageNet dataset. Many EA and Reinforcement Learning (RL) driven methods have since significantly improved the efficiency of the search process (Yang et al. [2020], Liu et al. (2018). Tan et al. [2019] Zoph et al.] (2018). These methods often require sampling and training of several candidate architectures. One-shor shot (Hu et al.. [2020). Xie et al. 2200)) Cai et al. (2019) methods of NAS do not require training of candidate architectures to completion, but train large super-networks and identify sub-networks that can give high accuracy. Such super-networks can be generated automatically from pre-trained models (Muñoz) ct al. [202I)) These improvements have significantly reduced the cost of NAS. However, as the search space continues to get larger with new architectural innovations, need more efficient methods to predict the accuracy neural networks in intractably large design spaces.
Zero-Cost Neural Architecture Scoring is a promising paradigm which explores zero-cost proxies for estimating the true accuracy of neural networks. The majority of research in this field introduces novel methods of scoring neural networks at initialization along with theoretical or intuitive explanation of their scoring strategy. For instance, (Abdelfattah et al. (22T]) empirically studies metries such as sow, SNIP and FISHER. Figure[]]epiect the expression tree representations of these metrics. NASWOT (Mellor et al. (2221) works by treating the output of each layer as a binary
2

---

function
ZC-NASM
clwwise
abslog
sign
14G
(B)
alwas Hum
eltwise sum
alolog
sign
T4 (AND
abolog
sign
T3G
T4G
T3
T40p
Network lapet Type
BotelNoom2
Burchoorma
RonchNorm:3
(2)
Rai 7
Cont
ReLU
Com
ReLL
Conva
Thaxa T1G
12 120
Bodwaad Teasor
(A)
Figure 1: (A): Collecting neural network statistics for the ZC-NASM. The D input tensor represents single mini-batch from the dataset serving as input to the neural network. / represents a randomly initialized noise tensor. P represents an input to the neural network which is data-sample perturbed by noise, NN statisties are collected for each of the D. / and P inputs. (B): The ZC-NASM applied to each ReLU-Conv2D-BB Baaeev2DDBBachhhee BatchNorm2D (RCB) instance of the neural network. The ZC-NASM has 22 tensors available it as arguments in each RCB instance, generated by collecting intermediate tensors of the neural network for the three types of input (D, N. P). The ZC-NASM depicted above only utilizes 2 intermediate tensors (T' and T4Gp) p) in each layer to generate a score.
indicator (zero if value is negative, one if value is positive) and using the hamming distance between two binary codes induced by untrained network from two inputs measure of dissimilarity. The intuition here is that the more similar two inputs are, the more challenging it is for the network to learn how to separate them. Works like TENAS (Chen al. (021)) ). GradSign (Zhang and Jia create proxies for trainability & expressivity of neural networks to rank them.
2021
Program Synthesis: Program synthesis is the task of automatically discovering programs that satisfy user constraints. AutoML-Zero (Real et al. (0200) evolves entire machine learning gorithms from scratch with little restrictions form and using only simple mathematical opera- learnt symbolically, rep- resenting programs as sequence of instructions. We posit that the current endeavor of identifying fundamental architectural properties that correlate strongly with testing accuracy can benefit from a methodology that minimizes human bias and in- tervention in a similar fashion. This stems from the observation that majority of the existing ZC- NASMs can be represented as simple programs that utilize the neural network statistics as inputs which can be discovered with genetic programming
SNIP abs
FISHER
SYNFLOW
LI Norn
Power
eltwisc prod
eltwise prod
eltwise prod
T3G
T3g
T3G, T3p
tions. The algorithms
14G
14p
Figure 2: Expression tree representation of existing ZC ZC-NASMs. 13G6p represents the Conv2D weight gradient for a mini-batch of data, T3p represents the weights of Conv2D. T'G'p represents the activa- tion gradient and T4p represents the corresponding activation. T3G represents the Conv2D weight gradient for mini-batch of random noisy tensor, T3N represents the weights Conv2D, Each pro- gram is applied to all layers of the neural network, and the output is aggregated to generate the final score. Further details of this program representation explained later.
3 Evolutionary Framework
EZNAS uses evolutionary search to discover programs that can score neural network architectures such that correlates highly with accuracy. In this section, we detail the specifics of how ZC-NASM programs constructed, evaluated and evolved with EZNAS.

---

3.1 Program Representation
Each individual ZC-NASM has to be evaluated on tens of gigabytes of tensors of NN architectures with no approximations to generate precise ZC- NASM score-accuracy correlation to serve as the measure of fitness. To make such search compu- tationally tractable, it is crucial to not introduce redundant operations in the program. Our ini- tial attempt described in the appendix resembled
Algorithm EZNAS Search Algorithm
1: evol_space [[00 rena, ennae ItN [nne)
2: population = gen random sardddm_allddddd populattioo)))
3: Evaluate and assign fitmess over evol_space.
population = evaluate(populaaion)) for gen=l:T
offspring =
while len(odispring)) [n//2] do
8:
# Variation Function
that of AutoML-Zero (Real et al.] [2020]) where
9:
children VarOr(populationn
ZC-NASMs were represented as a sequence of instructions and a memory space to store interme-
10:
ollspring.append Velecc/aliddcchikdeen))
11:
new_e= gen_random saaprin_allii_ldeelleeee
diate tensors. This led to discovery of ZC-NASMs 12:
offspring.appenddnees
with many redundant computations due to pro- gram length bloating and an intractably large time. To reduce the complexity of search, we necessitate a expression free structure on the ZC-NASM program capture the executional ordering of the program. The program output appears at root node, and the child (terminal) nodes are the arguments of the expression tree, These arguments are the network statistics. The advantage of this program representation is that there is only single root node with dense connectivity from the root to the terminal nodes, leading to fewer redundancies.
13:
population cvaluate(oftspring)
3.2 Neural Network Statistics Generation
Each ZC-NASM requires set of arguments inputs. These arguments the intermediate tensors of the neural network. As depicted in Figure I for any sampled neural network from the - NDS or NASBench-201 spaces, identify every BeLU-Conv2D- BatchNorm2D (RCB) instance at initialization, and register the activations, weights and the corresponding gradients for three types of network inputs (a mini-batch of data (D), random noisy tensor (N), and mini-batch of data perturbed by random noise (P)).
We collect 71 T.P. T2D.N.P.. T3, TAID.N.PY (10 tensors) and TIG T2G T3G [4G(D.N.P) (12 tensors) from RCB instance. T3 represents the Conv2D weights and does not change for the network input type, thus giving 22 tensors for each RCB instance. We present an alternate formulation by identifying network statistics generation to demonstrate that the evolutionary framework is not restricted to RCB structure in the appendix.
(CBR) instance for
3.3 Mathematical Operations
The expression tree describes the execution order of the mathematical operations available to us. It is crucial to provide a varied set of operations to process the neural network statistics effectively. We provide 34 unique operations in our program search space. We include basic mathematical operations (Addition, Product) as well as some advanced operations such as Cosine Similarity and Hamming Distance. We provide the full list of mathematical operations in the supplementary material.
3.4 Program Application
Majority of the neural network architectures available in NASBench-201 and NDS have over 100 instances of SaU---ConnDD-Batcchorrreree
This may mean that an expression tree would have as many as 2200 (22>) 100) possible arguments (terminal nodes), each of which can used multiple times. This would result in a computationally intractable expression tree. To simplify the search problem, we generate single expression tree with 22 possible inputs. It is not necessary that the root node of an expression tree would give a scalar value, so we add a to p_sscllar operation above the root node of the expression tree. As seen Figure TggueeThhe The expression tree is then applied on every instance and the output is aggregated across all instances using an MelU--Conv2D-Bttchlloo This serves as the 'score' of the the sampled neural network architecture. eggregaation_funct.e In EZNAS-A, the aggregation_fune onnotntioooooooooo and to so_scalar are both mean. In the appendix, we explore 1.2-Norm as a to so_scalar function as well and find that we are able to discover effective ZC-NASMs.

---

3.5 Evolutionary Algorithm
Our search algorithm discovers programs by mod- ifying the expression tree representation of the population by poriattio__functai ness score is generated for every expression
Pregrant
Praguee
Tregrse
Prograe
A fit-
Coertikke
Baam
ligin
tree in the population at each generation. The variation_ funcctionn (VarOr) is then used generate new offspring. We describe our search algorithm in Algorithm
TXy
3.5.1 Population Initialization
Concrati
Itam
Ages
Idwio.
Ei zere
We initialize population of n programs. We do not impose any restrictions on the operations the nodes can in the expression tree. Due this, the number of valid expression trees is several orders of magnitude lesser than the total number of expression trees that can be generated with our mathematical operators and network statisties. To increase search efficiency, we would like to ensure that we sample and evolve only valid expression trees. To enable this, we ensure that all individuals in the population are valid programs by testing program execution on a small sub-set of network statistics data. All programs that produce outputs with in.,, nan or fail to execute are replaced by new randomly initialized valid programs.
Figure 3: Variation of expression trees every gen- eration. The signs denote point of cross-over or mutation.
3.5.2 Fitness Objective
As the goal of zero-cost NAS be able to rank neural network architectures well, we utilize the Kendall Tau rank correlation coefficient as the fitness objective. The search objective is to maximize the Kendall Tau rank correlation coefficient between the scores generated by program and the test accuracy of the neural networks in the evol_space.
3.5.3 Variation Algorithms
In our tests, we utilize the VarOr implementation from Distributed Evolutionary Algorithms in Python (DEAP) (Fortin et al. [2012) framework for the variation of individual programs. We generate (hyper-parameter) offspring programs at each generation. [n/2] offspring are generated as result of three operations; crossover, mutation or reproduction. These variations are depicted in Figure 3 For crossover, two individual programs are randomly selected from the population and mated. Our mating function randomly selects a crossover point from each individual and exchanges the sub-trees with the selected point as root between each individual. The first child is appended to the offspring population. For mutation, we randomly select a point in the individual program, and replace the sub-tre tree at that point by randomly generated expression tree. We repeat the variation algorithm the population till n/2 valid individuals are generated. To encourage diversity, we also randomly generate [n/2] valid individuals. We have placed static limits on the depth of all expression trees 10.
3.6 Program Evaluation Methodology
At each generation, the fitness of the entire population is invalidated and recalculated. Calculating the fitness of each program on the entire dataset (which can be approximately 1 TB) is computationally infeasible. Further, we may want to find generalized programs that give high fitness on many different architecture design spaces and datasets. This would translate to evaluating each individual on over 7 TB of data on the NDS and NASBench-200 search spaces.
Reducing the computation by evaluating the fitness of the population on a single small fixed sub-set of neural networks from the search space causes discovered programs to trivially over-fit to the sub-set statistics in our tests. To address over-fitting of programs to small datasets of network statistics while minimizing compute resources required for evaluating on the entire dataset of network statisties, generate evolution task dataset. This is generated by randomly sampling 80 neural networks
5

---

SPEARMAN p CFIO
CF100 IN16-120
FZNAS-A
0.83
0.82
KENDALL
CFI0
CF100
IN16-120
synflow
0.74
0.76
0.75
EZNAS-A
0.65
0.65
0.61
jacob_ cov
0.73
0.71
0.71
NASWOT
0.57
0.61
0.55
FLOPs
0.75
0.72
0.69
AngleNAS
0.57
0.60
0.54
Params
0.75
0.72
0.69
FLOPs
0.56
0.54
0,50
grad_norm
0,58
0.64
0.58
Params
0.56
0.54
0.50
snip
0.58
0.63
grasp
0,48
0.54
0.56
Table 3: Kendall for NASBench-201.
fisher
0.36
0.39
KENDALL DARTS Amocha ENAS PNAS NASNet
Table 1: Spearman p for NASBench- 201.
EZNAS-A 0.56
0.45
0.52
0,51
0.44
NASWOT
0.47
0.22
0.37
0.38
0.30
grad_nome norm
0.28
-0.1
-0.02
-0.01
-0.08
SPEARMAN p CF10 CF100
IN16-120
synflow
0.37
-0.06
0.02
0.03
-0.03
EZNAS-A
0.89
0.74
0.81
FLOPs
0.51
0.26
0.47
0.34
0.20
Params
0.50
0.26
0.47
0.32
0.21
NASWOT 0.45
0.18
0.41
Table 4: Kendall for NDS CIFAR-10.
Table 2: Spearman p for SSS.
from each available search space (NASBench-22) and NDS) and dataset (CIFAR-10. CIFAR- 100, ImageNet- 16- 16-10).. We evaluate individuals by randomly choosing search spaces, and sampling 20 neural networks from each of the chosen search spaces. We take the minimum fitness achieved by the individual program the spaces. We consider hyper-parameter. In tests, this is consistently kept at 4.
3.7 Program Testing Methodology
At the end of the evolutionary search, our primary goal is to test whether the programs discovered able provide high fitness on previously unseen neural network architectures. We test the fittest program from our final population as well as the two fittest programs encountered through-out the evolutionary search. At test time, take the program and find the score-accuracy correlation 4000 neural network architectures sampled from the NASBench-201 and NDS design spaces.
4 Results
As described in Algorithm D search for effective ZC-NASMs every design space from NDS CIFAR-IO and the NASBench-201 datasets. Each search takes approximately 24 hours on a Intel(R) Xeon(R) Gold 6242 CPU with terabyte of RAM.
We choose the most consistent ZC-NASM (EZNAS-A, discovered by evolving pro- grams on the NDS-DARTS CIFAR-10 search space) from our evolutionary search and compare it with existing ZC-NASMs from recent works. We report both the Kendall 7 rank correlation coefficient and the Spearman p ronk-order correlation co- efficient to fairly compare EZNAS-A with broad set of ZC-NASMs from recent lit- erature. We provide our Spearman P cor- relation the NATS-Bend' SSS search space and NDS ImageNet design spaces as well.
Aging Evolution Search
91.50
EZNAS-A Jecob,Com
SNIP
Avg. accurscy
93.45
Synd low
91.35
91.30
91.20
Tained Models
Figure 4: Search speedup with EZNAS-A on CIFAR- 10 MAS-Bench-201 The average best test accuracy is taken over 10 repeated experiments.
report our score- accuracy correlation by scoring all neural
NASBench-201:

---

networks in the NASBench-201 design space. As seen in Table B we obtain the highest score- accuracy Kendall Tau Ranking Correlation on all datasets. We also benchmark our work against the pruning based saliency metrics formalized in (Abdelfattah et al. (021 and find that our program EZNAS-A has the highest Spearman p among the ZC-NASMs in T
We report our score-accuracy correlation by scoring all neural networks in the NDS design space for the CIFAR-10 dataset and scoring 40 random neural networks over 5 seeds for the ImageNet dataset (Table 5 We find that EZNAS-A has the highest score accuracy Kendall Tau Ranking Correlation on ea ch of the design spaces NDS CIFAR-10 in Table 4
NDS:
NATS-Bench: We report our score-accuracy correlation by scoring 200 randomly sampled neural networks in the NATS- Bench TSS (same as NB-201) and SSS (Dong al. [202T] design space averaged over 5 seeds for the CIFAR-10, CIFAR-100 and ImageNet16- 120 dataset in Table 2
Neural Architecture Search Integration: We integrate our ZC-NASMs EZNAS-A with the Aging Evolution (AE) Search algorithm from (Real et al. (2019)). Our evolutionary search discovers ZC-NASM that is competitive in search efficiency with existing ZC-NASMs in Figure
Examination Of Best Programs
SPEARMAN p DARTS Amocba PNAS
ENAS
NASNet
EZNAS discovers a set of ZC-NASMs at the end of the evolutionary search. In this sec- tion, we analyze two of the best ZC-NASMs discovered by our method to give deeper un- derstanding of the nature of programs. We refer to these programs tensors of varying sizes and average the ZC-NASM score 5000 times. This is an approximate method understand how the ZC-NASM responds to change in architectural parameters (kernel size, number of channels, activation size).
EZNAS-A
0.70
0.58
0.43
0.43
0.31
Figure S: Spearman for NDS ImageNet.
EZNAS-A and EZNAS-B. For each program, generate random input
EZNAS- A: The evolution task dataset for discovering this program NDS-DARTS. The input to the ZC-NASM in Figure Glis the T3G (Weight Gradient with Random Noise Input) tensor. We find that the score increases as the number of channels depth increases, sizes of and 7 give higher scores than 3 and with the lowest score being assigned to kernel of size that the expectation value of EZNAS-A in Figure Seguur/@translaarn translates to a weighted
also observe that kernel
3. It is interesting form of parameter counting, with a non- non-linear monotonically increasing scaling of score with the number of input/output channels and locally parabolic relationship between the score and the kernel size with the minimum score at kernel size of 3.
The evolution task dataset for discovering this program was NDS-ENAS. The input to the ZC-NASM in Figure 6 ii TLGp p (Difference in pre-activation gradients for a noise perturbed data mini-batch). We find that the activation map size is exponentially more influential to the score when compared to the number of channels.
EZNAS-B:
Through this analysis, we see that EZNAS-A & EZNAS-B generate score which is correlated with the activation or weight sizes. is interesting to note that when compared ZC-NASMs from recent literature, this form of weighted parameter counting works more consistently. This is supported by our finding that FLOPs and Params are more generalizable than ZC-NASMs which work on the NASBench-201 space but fail on the NDS space.
Discussion and Future Work
With the EZNAS formulation, we discover effective ZC-NASMs which generalize across design spaces and datasets, as well give SoTA score-accuracy correlation. This is promising new direction, but there is much work to be done. In this section, we discuss the current limitations of EZNAS with scope for future work.
Program Design:
To simplify the search problem, we take mean (aggregation_funce) of the scores generated across all RelU--ConYD-BBaccNornnn instances. This formulation does not take layer connectivity patterns into account explicitly. Extending evolutionary
7

---

EZNAS-A
EZNAS-B
ei invent
Score
eltwise
Score
Gob zorm
frob nonnt
exp
sigmed
TIGp
Reb noom
Channel Count
P0
logdat
log
Channel Count
sgmoid
sign
Score
Score
lig
T3G
sigmoid
TIG,
Kemel Size
Activation Size
Figure (Left) Analysis of our best program (EZNAS-A A) on the Image Classification task. so_scalar is not required as the output is already scalar. (Right) Analysis of our second best program (EZNAS- on the Image Classification task.
search to take into account the global connectivity pattern of the neural networks or learning weighted averaging techniques for the individual layer (RCB instance) scores could help us discover better ZC-NASMs. Further, we impose structural restrictions on our program by ne- cessitating fixed structure (ReLU-ConvDD- (onoo2DD-atccNoom222) We have truncate instances of ReLU--ConvDD-Con222 BeF-DD--oooO---BBacclloiomii Batchhorm22 from the NDS space by ignoring the second convolution. As more diverse architectures are introduced in the field, we must re-formulate collection of network statistics in more robust manner. Note that NASWOT measures the similarity in the binary codes generated from the ReLU units for two different inputs. Our program design limits us to only using single mini-bater of inputs and does not compare inputs directly. Thus, would not be able to discover the NASWOT ZC-NASM metric with our approach. Fortunately, integrating such functionality into the program design space is feasible and is an interesting line of future work to generate more complex metrics.
It is also important to have robust and diverse set of mathematical operations to discover effective ZC-NASMs. We utilize mathematical operations inspired by AutoML--ero (Real et al. (020 and operations found in existing ZC-NASMs. None of our operations or network statistics generation have any scalar hyper parameters. For example, the Power operation fixed do an element- wise square, and the noise generation for input tensor is fixed to (O, 1). Optimizing these values dynamically and ensuring a sufficiently diverse set of operations may enable discovery of better ZC-NASMs.
Due to the computational resources required to generate network statistics, we have to pre-compute them and use the generated data as an evolutionaay task dataset. We only use network statistics with input batch-size of for evolution. A single sample statistic of neural network is insufficient describe the architecture, as evidenced by our increase in score-accuracy correlation with the batch size in Figure 7 Further, from Figure 7 we observe high variance in the fitness of ZC-NASM when evaluated for multiple seeds with batch size of 1. Testing over large number of seeds or increasing the batch-size would cause linear increase in memory requirement as well as increase in run-time of the evolutionary search.
Network Statistics:
Ranking Architectures: It is interesting to observe that in the entire neural architecture design space, FLOPs and Params are competitive proxies and sometimes better than many existing ZC- NASMs. A deeper investigation reveals that ranking the top 10% of the neural networks in the design space significantly more difficult task. In the top 10% of neural networks, FLOPs and Params are extremely weak proxies. Further, (Abdelfattah et al. (202T) finds that among their ZC-NASMs, synflow is the only one which able serve as a weak correlator neural networks on the NASBench-201 CIFAR- 100 and ImageNet 6- 120 design spaces. However, synflox low is not able to rank networks effective in the top 10% of CIFAR-10 NASBench-20 and the entire NDS design space.
performance in top 10% the

---

10:8
A 206
0:46
Kcpdall
0.5
0:42
04
L6
NR00 CFL0
NBA0
NR201
NDS
NDS
NDS
NDS
NDS
Basch Size
CF100
IN16120
Amoebo
DARTS
ENAS
PNAS
NASNa
Figure 7: (Left) Effect on score-accuracy correlation of EZNAS- with respect to batch size. This test done the NDS PNAS design space over 7 seeds for 400 Neural Networks.(Righ)) Effect of seed on our score-accuracy correlation of EZNAS-A with a batch size 1. CIFAR and ImageNet abbreviated CF and IN respectively. This test was done each design space Neural Networks.
seeds for 400
EZNAS-A is able to rank neural networks robustly across both the NASBench-22 and NDS design spaces, but does not rank the top 10% of neural networks effectively. We discover alternative ZC-NASMs that can weakly rank the top 10% of neural networks, but such ZC-NASMs fail to generalize across neural architecture design search spaces. This ability identify good architectures but inability to distinguish between the best architectures may be an inherent limitation of MC-NAMMs. Alternatively, improvements to our evolutionary framework may result robust ZC-NASMs that can rank the top 10% neural networks effectively. This is an interesting research direction and may require creation of more neural architecture design spaces like NASBench-20I/NDI which exhibit lower correlation with FLOPs/Parame
Evolutionary Search: In this paper, introduce intuitive program representation and appropriate variation methodologies to enable discovery of ZC-NASMs with minimal human intervention.
However, recent advances the field of Neuro-Symbolic Program Synthesis (Parisotto al. [2017) to learn mappings from input-output examples (neural network statistics to neural architecture scores) motivate improvements in evolutionary search. Further, exploring architecture connectivity encodings (White et al. (021)) for neural architecture search and discovering programs to rank neural architecture encodings may enable discovery of effective connectivity patterns enable deeper understanding of important features in neural architecture design beyond those contained in activations maps and weights.
Efficiency: EZNAS is able to discover a set of ZC-NASMs on a single CPU under 24 hours, which translates to 358.6 of CO2e (Lannelongue et al. [020]). In comparison, single NAS search (assuming 8 GPU days (Zhou et al. 2)20))) can take over 4,49 kg of CO2e. Recent works integrate their zero-cost proxies with differentiable architecture search (Xiang al. [2021) deliver 40x NAS speed-up. As the architecture design spaces that ZC-NASMs need to be discovered on diversify, the efficiency of our methodology must be improved scale larger problems. Seeding the initial population with viable candidates can enable faster discovery of robust ZC-NASMs. The search process can be made more efficient by using lower precision numerical formats, exploring proxy tasks on smaller datasets (down-sampled image datasets, smaller neural network design architecture etc.).
7 Conclusion
In this paper, we present EZNAS, novel genetic programming driven approach to discover Zero-Cost Neural Architecture Scoring Metrics (ZC-NASMs). The key advantage EZNAS is that it is an interpretable approach discover generalizable methods to rank neural networks. Generalizability of ZC-NASM crucial for its practical utility, as robust ZC-NASM should be able to rank NNs in previously unseen neural architecture search spaces.
With our approach, we are able to discover ZC-NASM (EZNAS-A) which evolved only on the NDS DARTS space, but delivers state of the art score-accuracy correlation across both the NASBench- 201 and NDS design space. We also demonstrate the generalizability of EZNAS-A A by providing extremely strong correlation results on competitive search efficiency EZNAS- by integrating our metric with the Aging Evolution (AE) search algorithm from (Abdelfattah et al. (021)). Further, we provide an in-depth analysis of the nature of programs we have discovered, along with a detailed account of existing limitations of our methodology to motivate future work in this direction. EZNAS has the potential to reduce human bias in design of ZC-NASMs, and aid in discovery of programs that provide deeper insights into properties that make neural network perform well.
as well as NDS ImageNet. We demonstrate

---

References
[1] Mohamed S. Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas D. Lane. cero-cost proxies for lightweight nas, 2021.
[2] Yash Akhauri, Adithya Niranjan, J. Pablo Munoz, Suvadeep Banerjee, Abhijit Davare, Pasquale Cocchini, Anton A. Sorokin, Ravi Iyer, and Nilesh Jain. Rhnas: Realizable hardware and neural architecture search, 2021.
[3] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware, 2019.
[4] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize for efficient deployment, 2020.
[5] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective, 2021.
[6] Kanghyun Choi, Deokki Hong, Hojae Yoon, Joonsang Yu, Youngsok Kim, and Jinho Lee. Dance: Differentiable accelerator/network co-exploration, 2021.
[7] Xuanyi Dong, Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-bench: Benchmarking NAS algorithms for architecture topology and size. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, pages 1-1, 2021. doi: 10.1009ppami2221..354488. URL https
//doi. 100.0....1999 11092Ftpami 2021 3054824
[8] Félix-Antoine Fortin, François-Michel De Rainville, Marc-André Gardner, Mare Parizeau, and Christian Gagné. DEAP: Evolutionary algorithms made easy. Journal of Machine Learning Research, 13:2171-2175, jul 2012.
[9] Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua
Lin. Dsnas: Direct neural architecture search without parameter retraining, 2020.
[10] Loic Lannelongue, Jason Grealey, and Michael Inouye. Green algorithms: Quantifying the carbon footprint computation, 2020.
[11] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search, 2018.
[12] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: 2019.
Differentiable architecture search,
[13] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu, Neural architecture optimiza- tion, 2019.
[14] Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training, 2021.
[15] J. Pablo Muñoz, Nikolay Lyalyushkin, Yash Akhauri, Anastasia Senina, Alexander Kozlov, and Nilesh Jain. Enabling NAS with automated network generation. CoRR, abs/2112.10878 2021. URL Utt.ttttsss///// //axxiv wwrrzivv.orr// abs///as//211/ 2112. 10878.
[16] Emilio Parisotto, Abdelrahman Mohamed, Rishabh Singh, Lihong Li, Denny Zhou, and Push- meet Kohli. Neuro-symbolic program synthesis. In 5th International Conference on Learning Representationns (ICLR 2017), February 2017. URL/httss:/ //wpttss::/www... http::::s///ww.ii...oeetee microsoft. wttoooott.ooooooooooooooooooooot en us/ program-ooaa-mmmmm sytprram-syttessss21
[17] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc Le. Regularized evolution for image classifier architecture search, 2019.
[18] Esteban Real, Chen Liang, David R. So, and Quoe V. Le. Automl-zero: Evolving machine learning algorithms from scratch, 2020.
[19] Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile, 2019.
10

---

[20] Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study on encodings for neural architecture search, 2021.
[21] Lichuan Xiang, Lukasz Dudziak, Mohamed S. Abdelfattah, Thomas Chau, Nicholas D. Lane, and Hongkai Wen. Zero-cost proxies meet differentiable architecture search, 2021.
[22] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: Stochastic neural architecture search, 2020.
[23] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. Cars: Continuous evolution for efficient neural architecture search, 2020.
[24] Yongan Zhang, Yonggan Fu, Weiwen Jiang. Chaojian Li, Haoran You, Meng Li, Vikas Chandra, and Yingyan Lin. Dna: Differentiable network- accellracce co-search, 2020.
[25] Zhihao Zhang and Zhihao Jia. Gradsign: Model performance inference with theoretical insights. 08616
2021. doi: 1.44550/AAXXV.21100000666 URL htappe///axxii.rr/bbbbeees
[26] Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search, 2020.
[27] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition, 2018.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] Our method increases the efficiency of NAS significantly, and is aimed at reducing the negative societal impact of Al.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as a URL)? [No] We plan to open-source our work as part of a larger framework in the future.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report error bars for the score-accuracy correlation of our discovered metric against variation of bate hsize as well as seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [No] Crediting the curators was sufficient specified in their open-source license.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "gf9HFLxbvgp", "reviewer": "Reviewer_UZds", "summary": "This work proposes a framework, namely EZNAS, searching for zero-cost proxies based on evolutionary algorithm. Specifically, the authors utilize an expression tree to represent the computation of a zero-cost proxy and adopt DEAP evolutionary algorithm framework. Extensive experiments on NAS benchmarks show the effectiveness of the method.", "questions": "1. How do you obtain the accuracy of architectures sampled from NDS (Line 253-256)? Do you train theses networks from scratch or directly use the accuracy in NAS-Bench-301?", "limitations": "The limitations are addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The motivation is interesting and sound. This work attempts to search an effective zero-cost proxy rather than manually design one like prior works.\n+ The experimental results are pretty good.\n\nWeaknesses:\n- The expression tree representation and the evolutionary algorithm is not new, which slightly limits the novelty.\n- The layout is informal. \na) Fig. 1 should be placed behind the Fig. 2.\nb) I suggest the authors providing a detailed legend in Fig. 2 to explain the meanings of T1(G), T2(G), T3(G), and T4(G).\nc) Figure 5 should be Table 5.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FWISXnGsV_1", "reviewer": "Reviewer_DDGZ", "summary": "This work concentrates on crafting training-free measurement for neural architecture at initialization. Instead of being designed by human experts, the authors propose a genetic programming framework to automate the discovery of zero-cost proxies. Each candidate is represented by a computational graph of operation trees. The initial population is randomly constructed. At each iteration, an evolution task subset is sampled from the full datasets to evaluate, and the candidates with higher ranking correlation with the final performance is preserved. Experiments on a series of NAS benchmarks shows that the searched zero-cost proxies are more robust than the manually designed scoring functions.", "questions": "What does the searched EZNAS-A and EZNAS-B tree look like? I think it is important to visualize them or comprare them with typical zero-shot proxies. ", "limitations": "No limitation is discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n1. I believe that this paper provides an insightful idea for the AutoML community that we can adopt evolutionary search on computational graphs for a specific operation with infinite searching space. Zero shot NAS is one example.\n2. Experiments show that the searched proxy can generalize to different datasets, with extremely small searching cost (a single CPU in under 24 hours)\n# Weaknesses:\n1. Actually, this paper can be regarded as a new predictor-based NAS method, where the predictor in this work is searched by genetic algorithm. A general framework for predictor-based NAS is to construct a surrogate predictor (e.g. a meta network) to predict the final performance. However, none prior work on predictor-based NAS  has been referenced in this paper, nor does the author discuss its difference with them. I suggest the author should include them in your discussion.\n2. Some of the very important baselines are missing. There exists a new trend for zero-cost NAS on neural tangent kernels[1, 2], which is cited in the paper but not presented in the quantitative comparison. I highly recommend the authors to compare with them.\n3. The search space is quite restricted, \"ReLU-Conv2D-BatchNorm2D\", which hinders its application to more advanced architecture nowadays. \n[1] Shu Y, Cai S, Dai Z, et al. NASI: Label-and Data-agnostic Neural Architecture Search at Initialization[J]. arXiv preprint arXiv:2109.00817, 2021.\n[2] Chen W, Gong X, Wang Z. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective[J]. arXiv preprint arXiv:2102.11535, 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FV74i1reqL5", "reviewer": "Reviewer_gsE5", "summary": "Although NAS automates the manual design process of networks, it still requires expensive computing costs for evaluating multiple models. Zero cost proxies have been studied for alleviating the issues but it has several limitations, low productivity and poor ability of generalization. To tackle the current limitation of it, this paper introduces a genetic programming driven methodology, EZNAS, to automatically discover Zero-Cost Neural Architecture Scoring Metrics (ZC-NASMs)  that are interpretable, generalizable and deliver state of the art score-accuracy correlation.\n", "questions": "I can only see the score-accuracy correlation in the main paper and the supplementary documents. \n\nIf possible, please provide actual test accuracy and statistics, and detailed information, (i.e. training times, search times, FLOPs, model sizes, etc) of sampled networks for Table 1-5. \n", "limitations": "No negative societal impact is expected.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Clarity: This paper is well-organized and easy-to-read. The authors smoothly develop their logic and key concepts throughout the paper. The concept figures are illustrated clearly. \n\nQuality: This paper smoothly develops its logic throughout the paper. The experiments seem carefully designed.  \n\nOriginality: The contribution of this work seems solid, since unlike the existing methods that use ZC-NASMs that are driven by human intuition or theoretical inspiration, this work devises a method for automatically discovering ZC-NASMs which lead to state-of-the-art score-accuracy correlations. This work has the potential to reduce human bias in the design of ZC-NASMs. \n\nSignificance: The achievement of this work seems significant, sinc it shows better score-accuracy correlation compared to the existing metrics and methods, i.e. DARTS, Amoeba, etc. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5JSzWLvzDP0", "reviewer": "Reviewer_anAK", "summary": "This paper automates the discovery of zero-cost proxies in NAS. Very low-level mathematical operations are selected and combined through genetic programming to form proper zero-cost proxies. The effectiveness of the proposed method is validated on NASBench-201 and NDS.", "questions": "None", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1.\tThe paper is well motivated and clearly written. The paper is clear about limitations too.\n\n2.\tI think this paper will be valuable to the community. Zero Cost Proxies discovery is important for efficient neural architecture search. Evolving the proxies automatically from low-level mathematical operations also follows the trend of auto-ml. \n\n3.\tThe searched zero Cost Proxy can better capture the order of candidate models than previous hand-crafted proxies, and is reported can generalize better on different search spaces.\n\nWeakness:\n\n1.\tThe proposed method looks quite complicated, considering the program representation, NN statistics generation as well as the evolutionary scheme. The inclusion of code would be a large benefit to this work. \n\n2.\tThis paper borrows ideas from [1], providing the program synthesis is also considered before, which weakens the novelty of this paper. So I recommend a weak accept.\n\nReferences:\n\nE. Real et.al. Automl-zero: Evolving machine learning algorithms from scratch. ICML ‘20.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept