Neural Network Surgery: Combining Training with Topology Optimization
Authors: Elisabeth Schiessler
Roland Aydin
Kevin Linka
Christian Cyron
Venue:  (2021)
Keywords: Neural Architecture Search
Genetic Algorithm
SVD

Abstract: With ever increasing computational capacities, neural networks become more and more proficient at solving complex tasks. However, picking a sufficiently good network topology usually relies on expert human knowledge. Neural architecture search aims to reduce the extent of expertise that is needed. Modern architecture search techniques often rely on immense computational power, or apply trained meta controllers for decision making. We develop a framework for a genetic algorithm that is both computationally cheap and makes decisions based on mathematical criteria rather than trained parameters. It is a hybrid approach that fuses training and topology optimization together into one process. Structural modifications that are performed include adding or removing layers of neurons, with some re-training applied to make up for incurred change in input-output behaviour. Our ansatz is tested on both the SVHN and (augmented) CIFAR-10 datasets with limited computational overhead compared to training only the baseline. This algorithm can achieve a significant increase in accuracy (as compared to a fully trained baseline), rescue insufficient topologies that in their current state are only able to learn to a limited extent, and dynamically reduce network size without loss in achieved accuracy.

Under review as a conference paper at ICLR 2021
NEURAL NETWORK SURGERY: COMBINING TRAINING WITH TOPOLOGY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
With ever increasing computational capacities, neural networks become more and more proficient at solving complex tasks. However, picking a sufficiently good network topology usually relies on expert human knowledge. Neural architecture search aims to reduce the extent of expertise that is needed. Modern architecture search techniques often rely on immense computational power, or apply trained meta-controllers for decision making. We develop framework for genetic al- gorithm that is both computationally cheap and makes decisions based on math- ematical criteria rather than trained parameters. It is a hybrid approach that fuses training and topology optimization together into one process. Structural modi- fications that are performed include adding or removing layers of neurons, with some re-training applied to make up for incurred change in input-output behaviour. Our ansatz is tested on both the SVHN and (augmented) CIFAR-10 datasets with limited computational overhead compared to training only the baseline. This al- fully
gorithm can achieve a significant increase in accuracy (as compared trained baseline). rescue insufficient topologies that in their current state are only able learn limited extent, and dynamically reduce network size without loss in achieved accuracy.
INTRODUCTION
A common problem for any given machine learning task be addressed with artificial neural net- works (ANNs) is how to choose sufficiently good network topology. Picking one that is too small may not yield acceptable prediction accuracy. To improve results, one can keep adding structural el- ements to the network until the desired accuracy value has been reached. However, at the same time too large networks may cause an explosion in computational cost for both training and evaluation. Where the sweet spot in between lies is unclear, and heavily dependent optimization is not easily possible, since reliable estimates on network behaviour already require training results, and also there exists generalization for which topology will fit which problem. Researchers have applied a number of search strategies such as random search (Li & Talwalkar 2019), Bayesian optimization (Kandasamy al. 2018), reinforcement learning (Zoph & Le| 2017). and gradient-bart based methods (Dong & Yang. 2019). Another technique applied since at least Miller al. (1989) are so called (neuro-) evolutionary algorithms. These algorithms serve evolve the network architecture, often also training network weights at the same time (Elsken et al. 2019).
the given task. A priori
In this paper propose novel training regime incorporating genetic algorithm that reduces computational cost compared to state of the art approaches of this kind (Dong & Yang 2019: Li & Talwalkar 2019). We achieve this by re-using network weights for competing modification candi- dates instead of retraining each net from scratch, branching off modification candidates during train- ing, and letting them compete against each other until a new main branch selected. This better fuses the evolutionary optimization paradigm with the ANN training into an integrated framework that folds both processes into a single training/topology optimization hybrid. As such, evolutionary steps are not carried out by meta-controller or other black-box-like inplementations. We make use of mathematical tools such as singular value decomposition (SVD) and the Bayesian informa- tion criterion (BIC) (Schwarz 1978) for network weight analysis, decision making, and structural modifications. Network modifications are performed by adapting existing weights such as to incur minimal changes onput-output behaviour.

---

Under review as a conference paper at ICLR 2021
Our framework for combined ANN training and neural architecture search consists of three main components: a module that can perform a number of minimally invasive network operations ("surg- eries"), module that analyses network weights and can give recommendations which modifica- tions are most likely to increase (validation) accuracy, and finally a module that serves as genetic algorithm (the "Surgeon")), containing the former two while gradually evolving any given starting network. With the Surgeon, we are able to evolve and improve models for the SVHN (Netzer et al., 2011] and CIFAR-10 (Krizhevsky (200) datasets for varying starting topologies. We achieve par- ticularly good results on starting topologies that would a posteriori have proven to be suboptimal. great benefit of our approach is that it adds topology optimization to the ML training at a very limited additional computational cost. Convergence is reached for all test cases within a few CPU hours.
This paper contributes computationally cheap ansatz for genetic neural architecture search algo- rithm that makes evolutionary decisions based on mathematical analysis.
2 RELATED WORK
Neural architecture search (NAS) has been an increasingly popular research topic for many years (Elsken et al. 2019), starting as early as Miller et al. (1989), who presented one of the earliest neuro- evolutionary algorithms to search for suitable network topologies. Recent approaches by Dong & Yang (2019): Li & Talwalkar (209)), and Zoph & Le (017)) reach competitive performance on benchmark datasets such as CIFAR-IO. However, this often comes at the cost of vast computational resources, with Zoph & Le (017)) making use of up to 800 GPUs for several weeks.
Cai et al. 2018 attempt to reduce computational costs by re-using network weights, as well as training and applying reinforcement meta-controller for structural decisions. They make use of a number of seactton-preserving transformations (net2net) introduced by Chen et al. (2016), and extend them to allow also non-sequential network structures, such DenseNet (Huang al. 2017). DiMattina & Zhang (2010) introduce and rigorously prove conditions, under which gradual changes of the parametrization of neural network are possible, while keeping the input-output behaviour constant.
There are also number of neural architecture search strategies that do not depend on manual net- work modifications. Dong & Yang (2019) represent the search space as a directed acyclic graph and propose an efficient search algorithm. Irsoy Alpaydın (2020) learn the network structure via so-called "budding perceptrons", in which an extra parameter is Iearned for each layer, that indi- cates whether or not any given node needs to branch out again or be removed altogether. Their method focuses on growing the network to the required size from a minimal starting topology. [Fran- kle Carbin (2019) present a method to identify particularly good network initializations that can train sparse networks to competitive accuracy. Another approach in NAS is to prune down from larger starting topology (Blalock et al. 2020 Popular pruning techniques include applying SVD to existing network weights (Girshick] 2015 Denton et al. 2014 Xue et al. 2013).
The novelty of our research lies in combining existing tools such as net2net (Chen ct al. 2016) and SVD with genetic algorithm that modifies the given network in a decision based process instead of utilizing a black-box like decision module. To our best knowledge no such method has yet been proposed.
3 METHODS
This work introduces and utilizes three main modules:
performs network modifications ("surgeries") so as to incur minimal changes to input-output behaviour.
modification module:
recommendation module: analyzes network weights and gives recommendations on which operations are most likely to improve network accuracy.
"the Surgeon": : a genetic algorithm that links the above two modules, and gradually evolves given starting network.
2

---

Under review as a conference paper at ICLR 2021
3.1 THE MODIFICATION MODULE
We need to be able to perform a number of different modifications that can restructure network architecture. Ideally, these operations should not change all the input-output behaviour. In cases where this is not possible, we aim to perform minimal impact changes instead. We perform four different types of modifications: adding or removing neurons or whole layers.
Adding neurons and layers. Under piecewise linear activation functions such as ReLu, one can always add neurons to a hidden layer, or even add whole layers, without any change the overall network behaviour. Adding a whole layer is done by using an identity matrix as a new weight matrix for this inserted layer. We add new neuron any layer by dividing all weights for one existing neuron by two, and duplicating the respective connections in the next layer (cf. appendix (.I))
Removing neurons. Removing neurons from a layer is rarely possible without changing the input- output behavior unless some of the units are degenerate to begin with, i.e. the weight matrix is of reduced rank. For a modification with minimal impact on input-output behaviour we need the closes possible projection onto lower rank subspace. The Eckart- Eckart-Younge 'foung(-Mirsk) theorem states that this projection can be found by applying Singular Value Decomposition (SVD) to the weight matrix:
Let A E Rm*x" be the (weight) matrix of some layer of interest in a neural network, then there exists representation
A = UVVT
with orthogonal U Rmxm V E RRxn and rectangular diagonal Σ E Rmxn with k S min(m, n) non-negative real entries a; along its diagonal. called singular values of A (cf. appendix [A.)). In order reduce the rank of A r < k, we set c; = 0 Vi r, and drop associated columns/rows of U and V, which is feasible since by definition o; 2 a:+1- We then project smaller matrix Á by computing UE. To project back to the original layer size, so that the layer weight shapes match again, we multiply the weights of the subsequent layer with V..
Projecting onto a lower rank subspace by setting singular values to zero is sometimes called trun- cated SVD, and is used in network pruning (Denton et al.] [2044] Girshick 2015) Xue et al., 2013). This technique adds an intermediate layer of (potentially much fewer) neurons, which in case of very large weight matrices can drastically reduce the overall connection count.
Note that this projection method is not concerned at all with a potential activation function after the network's modified layer. In particular, changing a layer's activation function is in general not possible without changes to input/output behaviour, since activation functions are usually linelr. We therefore do not perform any additional modifications to counterbalance a change of activation.
Removing layers. We remove whole layers in similar fashion, by multiplying the layer's weights onto the weights of the next layer. This again ignores any activation function between the two layers, and will thus cause change in input-output behaviour.
Recuperation from surgery. As we have seen, network modifications will in general cause a small change in input-output behaviour, typically leading to loss in prediction accuracy. In order to make up for this, all network modifications are given a small amount of recuperative training (sev- eral batches) before any comparison is made. The retraining amount determined by trials and statistical analysis based on a datase which we do not otherwise use in our results, to avoid over- fitting the search algorithm a specific dataset.
3.2 THE RECOMMENDATION MODULE
For the decision when to execute which network modification, we perform a two-step analysis. As a first order criterion, we compute the amount of information carried by each neuron by looking at the layer's singular values. The number of neurons that may be removed from layer depends on the count of singular values that are (close to) zero, or are several orders of magnitude smaller than the layer's largest singular value:
removable neurons = (o,, i = 1...k k olo < <10 Vo < < E1, 62 ER}
(2)
I "under Frohenius norm ||
MNIST; (LeCun al. 1998
3

---

Under review as a conference paper at ICLR 2021
The number of removable neurons in relation to the layer's total neuron count gives an indication whether addition or removal operations are preferable (cf. appendix A..
The second order decision basis is the Bayesian information criterion (BIC). It weighs the number of model parameters and training samples against the likelihood function of predicted vs. true target data. Since all operations chosen incur minimal changes to predicted values, the difference in the likelihood function between two modifications becomes negligible compared to the number of parameters, which allows reduce the formula the parameter count of the modified network (cf. appendix A.4).
Thus, all potential network modifications ranked by two parameters. The Surgeon has two modes: it can either pick the top n ranking operations per decision step (with n being a tunable hy- perparameter), select the highest ranking operation of each type. Our experiments are performed with the latter option.
3.3 "THE SURGEON"; A GENETIC ALGORITIM FOR NEURAL ARCHITECTURE SEARCH
The final module is the Surgeon, the genetic algorithm that searches for an optimized network archi- tecture while training network weights at the same time. In this first paper about our ansatz, limit our focus on perhaps the most ubiquitous type of architecture: sequential networks (i.e. with- recurrent skip connections) consisting only of fully connected layers. As Cai al. (2018) have shown, however, number of the above described tools can be generalized easily also to non- sequential networks as well as convolutional layers.
Algorithm
The Surgeon
1: function
2: train model [m] for initial number of epochs
3: current branches [mi]
4: while termination criteria not met do
Dr e.g. max. number of epochs
5: for all current branches do
6:
determine potential modification candidates [Meana]
7:
repeat
8:
select [maei] from [Meand]
e.g. competitors e.g. 15 batches
9:
re-train [m] for small number of batches
10:
compare set] keep top scoring as new current branches > e.g. conc. branches train current branches full epoch step
e.g. 10 epochs
12:
until improvement achieved or max re-tries reached
13:
select final best scoring branch
14:
return optimized network Mopt
The overall structure of the Surgeon can be seen in algorithm I trained for an initial number of epochs, then the list of current branches is initialized with it. The choice of how many branches at most are being kept concurrently is hyperparameter setting, and has great influence on the total computational cost of the Surgeon.
First, the provided model is pre-
We then evolve and continuously update the list of concurrent branches until termination criteria, such the maximal number of epochs (if limited by computational resources) or minimum accu- racy threshold (if the topology optimization used without a computational resource bottleneck), met. At each decision point, the recommendation module analyses all networks in the list of current branches, and ranks all potential modifications. From these, in line g of algorithm | we select the most promising candidates, and perform the selected operations using the modification module.
The generated candidates re- re-trained for several batches, to make up for lost performance, and then scored according to their validation accuracy a, validation accuracy gain Aa, and parameter count increase A fraction of its parent's parameter count:
exp (Aa)
= a +
(3)
exp (Af)
4

---

Under review as a conference paper at ICLR 2021
This composite score serves to reduce greediness of the Surgeon, and is adapted from Cai et al. 2018) (cf. appendix A.5).. The top scoring candidates are then kept as new concurrent branches, and trained up to the full epoch step.
As an option to alleviate greediness, the Surgeon can keep a one cycle memory. In this case, the newly selected concurrent branches are compared previously best scoring ones, and retained only their achieved accuracy is at least as good as the previous value. Should this not be the case, they are discarded and we backtrack one step. New potential candidates are provided by the recommendation module, and we train and compare these. Finally the best scoring branch is returned as the optimized network.
EXPERIMENTS
4.1 DATA
We evaluate the performance of the Surgeon on two standard benchmark datasets: CIFAR-10, and SVHN. We pick three starting topologies that are described in [able and perform several runs of the Surgeon on each one. The small one is consciously chosen to insufficient, to mimic case where model is unknowingly trained with an inadequate network architecture. We average our results over several runs and several random seeds, and compare to results achieved by simply training starting topology for the same number of epochs. Detailed machine properties and hyperparameter settings are listed in appendix
Table 1: Starting topologies for the Surgeon, excluding input and output layers
Name Hidden layer count
Hidden layer sizes
Activation
Small
10
ReLu
Medium
3
10 10-10-10
ReLu
Large
3
300 100 100
ReLu
The CIFAR-10 dataset. The Canadian Institute For Advanced Research (CIFAR)-10 dataset was first published by Krizhevsky (2000) It contains 60,000 32 x2 color images that are evenly divided into ten classes. 10,000 of the images are set aside for validation purposes. Following related works (Cai et al. 2018 Huang al. 2017), we perform data augmentation by flipping images horizontally, as well as translating by pixels in either up, down, left or right direction. This yields a dataset of 360,000 images. We then normalize these images using channel means and standard deviation.
The SVHN dataset. The (Google) Street View House Number (SVHN) dataset was first published by Netzer et al. (201T). It contains 73,257 training, well 26.032 validation color images. We use the cropped version, where images are of size 32 32, and fall into 10 classes according to the numbers 0-9, Additionally, 531,131 extra images of lower difficulty are available but currently not used. We do not apply data augmentation, but divide by 255 in order to normalize the data.
4.2 RESULTS
We apply the Surgeon to each combination of the above datasets and starting topologies (cf table []. We do so a total of 15 times, re-initializing the numpy and tensorflow modules with a new random seed after every 3 runs, and report average statistics (cf table 2 As baseline, we train the starting topology for the same number epochs with each random seed. Note that throughout the entire section, unless otherwise stated, report achieved validation accuracies rather than training accuracies.
To avoid overfitting on any specific dataset, fix some hyperparameters for training (such batch size, optimizer, and learning rate) before starting any runs with the Surgeon (cf. appendix B). No additional fine tuning is performed any model.
Overall Surgeon performance. Table 2 and figure I both show an overview of average Surgeon performance for all starting topologies and datasets. We that in all cases, the Surgeon reaches
5

---

Under review as a conference paper at ICLR 2021
0.60
0.28
cloline
60/3
0.30
0.45
0.45
0.74
0.00
0.22
0.30
0.39
0.35
0.20
Sagoun Bradee
0.70
100
25
50
75
100
spoch
0.30
0.33
Valteattoon accunccy
0.31
0.34
0.30
0.23
0.52
0.14
0.23
0.30
6.22
0.40
25
50
75
300
23
50
75
100
hil
100
sporh
Spoch
Figure 1: Average performance of the Surgeon over all 15 runs on the SVHN (top row) and CIFAR- 10 (bottom row) datasets for small (left), medium (center), and large (right) starting topology, com- pared training the starting topologies as a baseline.
outperforms the result of the baseline. As expected, achieved accuracies for the SVHN dataset are higher than for the (almost 5 times larger) CIFAR-I0 dataset.
In case of the SVHN dataset (figure D top row), the baseline for the small starting topology is not able to learn at all, whereas the only slightly larger medium topology is just about sufficient for at least a little bit of learning. The Surgeon able improve training in both cases, reaching 58% accuracy in case of the medium topology. At first glance it may seem peculiar that the Surgeon on average able to reach only around 29% accuracy for the small one, given how similar the two of them are. This is due to averaging over a bi-modal distribution of results, where those branches that could rescued (27 %) reach an average accuracy of 54% (in line with the medium topology), whereas the rest remain on baseline level (cf. table 2).
The large topology trained on the SVHN dataset yields similar accuracies with or without the aid of the Surgeon. What we can not see in figure 2 (top right) however, is that the Surgeon on average is able to reduce the total number of trainable parameters by around 7%, with an std of around 32% (cf. table 2 and figure rright).
For the CIFAR-10 set (figure boottom row), both the small and medium starting topologics are too small for good results, even with the aid of the Surgeon, achieving average of 34% and 36% accuracy respectively. On the large starting topology, the Surgeon is able to achieve on average at least a little improvement compared the baseline. Parameter count increasing all three cases, which is indicative of the fact that neither of the chosen topologies is sufficient for a dataset of this size.
We will subsequently highlight a few interesting cases.
Topology rescue. We recall that the small starting topology, consisting of only one hidden layer with 10 neurons, was purposefully chosen to be insufficient for convergence. In fact, as we can see in the left column of figure D the baseline learns for neither SVHN nor CIFAR-I0 The Surgeon on average manages to improve the topology and learn at least little, though the achieved validation accuracy of around 29% and 34% respectively seems not great. In the case of SVHN however, the global average over all runs contains several instances where even with the Surgeon, the model is not able to learn at all and stays stuck in the initial state, as well number of runs where the Surgeon is able to very quickly leave this local sink and then in fact provides model that learns very well.
In figure (left), we can see a single run of the Surgeon, where an early Add Layer operation allows the network to train properly. The total parameter increase in this case is less than 2%, with the Surgeon preferring to add several small layers rather than widening existing layers. Note that due to the shape of our training data and choice of starting topology, large portion of the network's parameters is required to connect the input layer to the first hidden layer. Adding a whole layer after the 10-unit layer causes only small overall increase, whereas widening the first hidden layer might

---

Under review as a conference paper at ICLR 2021
Table 2: Statistics all baseline and Surgeon runs. We report means and standard deviations.
Topology
small
medium
large
SVHN
Baseline accuracy
0.20 = 0.00
0.34 + 0.12 0.58 0.04
0.78 = 0.01
Surgeon accuracy
0.29 = 0.16
0.79 = 0.01
Relative accuracy increase [%]
45.00
70.59 1.97 +: 7.13
1.28 -7.23 + 32.48
Parameter fraction increase [%]
16.48 +: 52.0
CIFAR-10
Baseline accuracy
0.28 + 0.01
0.32 = 0.01
0.49 + 0.00
Surgeon accuracy
0.34 + 0.01
0.38 + 0.06
0.51 + 0.00
Relative accuracy increase [%]
21.43
18.75
4.08 2.12 3.13
Parameter fraction increase [%]
1.12 1 = 2.85
203.87 + 417.45
SVHN dataset, small starting topology
0.60
Surgeon
0.55
Baseline
Achieved accuracy
0.50
Input layer
Outpua layer
0.40
0.35
Input Inyor
10
10
10
10
10
Ootput layer
0.30
0.25
0.20
Epoch
Figure 2: Left: Single run of the Surgeon on the SVHN dataset and small starting topology. Right top: Small starting topology. Right bottom: Topology after 100 epochs training with the Surgeon.
cause greater increment in parameter count. Figure (right) shows the starting topology (top) and the result after training with the Surgeon for 100 epochs (bottom). Note that up until epoch 50, only two layers have been added. In particular in this case, earlier stopping point would have yielded similar performance with lower parameter count.
Accuracy increase. The Surgeon is able detect and improve sub-optimally sized network archi- tectures. This works in cases such as above (figure left), where very obvious that little to no learning happens. but also in less apparent ones, where the base topology does learn to certain extent. In figure 3 (left) we can see a single run of the Surgeon trained on CIFAR-10 where both the large starting topology well the Surgeon start learning in similar fashion and reach a plateau. After a while however, the Surgeon is able to perform an Add Layer operation that allows the topology overcome the local optimum which had been reached.
Parameter reduction. As mentioned above, using the large topology as a starting point for the Surgeon does improve achieved accuracy by any large margin (figure [ right column). The baseline in this case is already performing quite well given that we are regarding a very basic network architecture. In this case we can observe pruning by the Surgeon. The composite scoring function given in eq. 3 prevents any big jumps in accuracy since they would most likely come at the cost of a (potentially rather large) increase network size. The scoring function much rather prefers network operations that keep the accuracy more less constant while reducing network size. On average, for the SVHN dataset, we are able to reduce the network's parameter count by around 7%. In figure 3 (right) we see example where with early Remove Layer operation, the Surgeon is able to reduce the total parameter count by almost 66%. The resulting loss validation accuracy is compensated within just few epochs of training.
Computational costs. All of our experiments are performed on a standard office computer, without making use of any GPU acceleration (cf. appendix B for detailed specs). In trials the Surgeon

---

Under review as a conference paper at ICLR 2021
large topology
SVHN dataset. large starting topology
0.52
0.80
Achieved accuracy
0.50
Average accuracy
0.75
800000
Paramater count
0.48
0.46
0.70
Surgeon
Bascline
0.44
0.65
Param,
0.42
0.40
Beseline
+
400000
20
60
80
100
Epoch
Epoch
Single run of the CIFAR-10 dataset and large starting topology. The Surgeon man- ages overcome the local optimum at around epoch 30. Right: Single run the Surgeon on the SVHN dataset and large starting topology. The Surgeon is able to reduce the total parameter count by almost 2/3 while still reaching the same overall validation accuracy.
Figure 3: Left:
is configured to produce a resulting topology that has been trained for exactly 100 epochs, with decision points every 10 epochs. We allow for maximum of two concurrent branches, well as re-drawing from potential branches up to two times. On average, 0.66 re-draws are necessary per decision step, resulting in an average total training amount of around 310 epochs per run of the Surgeon.
We can see from figure that 100 epochs seems to be longer than is necessary for the Surgeon to reach convergence. With appropriate early stopping techniques, and/or more dynamic training schedule, the total training amount for the Surgeon could be considerably reduced, and the resulting model (including its trained weights) used either as is, or further fine tuned.
SVHN dataset, medium starting topology
0.7
Add layer
Oscillation between configurations
Achieved accuracy
0.6
0.5
0.4
Remove layer
0.3
Remove neurons
Surgeon acc.
0.2
Baseline
0
20
40
60
80
100
Epoch
Figure 4: Example of a topology evolution performed by the Surgeon on the SVHN dataset and medium starting topology. Early reducing the network size helps increase the accuracy level (at epochs 10 and 30). Adding a whole layer in epoch 50 still able to achieve some increase in accuracy. From epoch 60 the network oscillates between very similar states. Ideally this behaviour can be used as an indicator for early stopping in future versions of the Surgeon.
DISCUSSION AND OUTLOOK
In this paper, we presented the Surgeon, a ANNievolutionary algorithm hybrid optimization de- signed for neural architecture search. The algorithm utilizes modification module, that is able to perform minimally invasive network surgeries, where the topology of the network modified with little change to overall input-output-behane behavit- possible. Additionally, it uses recommenda- tion module, that analyses given neural network and indicates which structural changes may be most beneficial. Those changes can be either increases of network width depth in case of high
8

---

Under review as a conference paper at ICLR 2021
information density, respective decreases, in case network size can be reduced without fear of too high am accuracy loss. Both modules do not utilize any black box behaviour but are based on mathematical tools, which we presented in section
We put the Surgeon to the test on several combinations of starting topologies and the SVHN and 10 datasets. We that the network generated by the Surgeon is able outperform the baseline in case of suboptimal topologies, or reach comparable accuracies while pruning the under- lying network structure less resource-intensive topologies.
A very important feature of the Surgeon is that itself is computationally cheap, with little overhead compared baseline training. Via hyperparameter settings, it is possible to make of larger computational power if required/available. For this proof of concept work, we limited ourselves to the most basic and ubiquitous network structures dense layers linked strictly in sequence.
For future work we wish to extend and improve both the Surgeon as well as the underlying modules. Currently, the Surgeon follows static routine, with hyperparameters such as epochs steps, number of selected candidates, or number of concurrent branches staying constant throughout the whole process. This could be changed to a more dynamic approach, and could maybe integrate further features such as adaptive learning rates, or more sophisticated memory, as well as early stopping mechanisms improve performance gains compared baseline training even further. The recom- mendation module can be improved by finding closer approximation for the BIC.
Lastly we want to expand the modification module to allow more complex topologies as well, and include option to cross architecture types. This would allow us to include e.g. convolutional or recurrent elements, change a network from one type to another, or even freely mix and match as required.
REFERENCES
Davis Blalock. Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In I. Dhillon, D. Papailiopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2, pp. 129-146, 2020.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture scarch by network transformation. In 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp- 2787-2794. AAAI press, jul 2018. ISBN 9781577358008.
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. In Y. Bengio and LeCun (eds.), 4th International Conference on Learning Represen- tations, ICLR, 2016.
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 27, pp. 1269-1277 Curran Associates, Inc., 2014.
Christopher DiMattina and Kechen Zhang. How modify a neural network gradually without changing its input-output functionality. Neural Computation, 22(1):1-47, 2010. doi: 10. 16.1 2.cc..009005-08.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceed- ings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR, 2019.
Ross Girshick. Fast R-CNN. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440-1448. IEBE Computer Society, dec 2015. doi: 10. 10911.1....11...6....

---

Under review as a conference paper at ICLR 2021
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. doi: 10.1109/CVPR.2011.224
Ozan Irsoy and Ethem Alpaydın. Continuously constructive deep neural networks. IEEE Transac- fions Neural Networks and Learning Systems, 31(4): 124-1133 1133, 2020. doi: 10.1109/TNNIS, 2019.2918222 .
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing- Neural architecture search with bayesian optimisation and optimal transport. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 2016-2022 Curran Associates, Inc., 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 86(11):2278-2324 1998.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Conference Uncertainty in Artificial Intelligence (UAI), 2019.
Geoffrey F. Miller, Peter M. Todd, and Shailesh U. Hegde. Designing neural networks using genetic algorithms. In Proceedings of the Third International Conference on Genetic Algorithms, pp- 379-384. Morgan Kaufmann Publishers Inc., 1989.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Gideon Schwarz. Estimating the dimension of model. Ann. Statist., 6(2))661-466 03 1978. doi: 10. 10.1214/aos/|1 1atp//1116444366..
Jian Xue, Jinyu Li, and Yifan Gong. Restructuring of deep neural network acoustic models with singular value decomposition. In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, pp. 2365-2369, 2013.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th Inter- national Conference Learning Representations ICLR, 2017.
A APPENDIX A MATHEMATICAL BACKGROUND
A.1 ADDING LAYERS AND NEURONS WITHOUT CHANGE TO INPUT/OUTPUT BEHAVIOUR UNDER PIECEWISE LINEAR ACTIVATION FUNCTIONS
Adding whole layers to feed forward neural networks without change to the network's input-output behaviour is possible, if and only if the layer's activation function is at least piecewise linear (Chen) et al. 2016). The number n of units in the given layer has to be identical to the number of units in the following layer. We initialize the weights an identity matrix I E R". Let & denote the activation function of dense layer and æ an arbitrary input, then
$ (xx) = Φb (x))
(4)
and only if & is at least piecewise linear. In particular, for ReLu activation, adding layers without changing the input-output behaviour is possible.
For adding neurons to an existing layer, consider the following example. Let x E R" the input to layer with weights A RRmx and B km the next layer's weights. Assume the activation
10

---

Under review as a conference paper at ICLR 2021
function @ acting between the two layers to be an identity mapping. Then the output given by
R
is
y B (Axx) B (Ax)
(5)
In particular, the 9th entry of is
K 2
Sim1 22222
yj (b31, by2,-... -... bjm)
(6)
1=
a4ami
n
byb
1,01; +
bjm
T,aim
where di is the ij element of A, and bij the ij element of B. We now arbitrarily pick unit and duplicate its incoming and outgoing weights. Note that we also have to divide by in order to keep the total sum constant.
yj = bرg
J:01 +
b jm
1m0 +
Djm
1m01m
(8)
2
1=1
Tiali
1= 2,02i
bym 2
bim 2
0311.b2......
(9)
ami
1= 3. a,ami
This methods yields the exact same output E R given input E IR", but the weight matrices A and are now of dimension (m + 1, and (ki m + 1) respectively.
Recall now that we chose the activation & to be the identity mapping. For any other activation function, the equation in line (8) holds true, and only if this activation least piecewise linear.
Chen et al. (016) base their Net2WiderNet and Net2DeeperNet transformation on these steps.
A.2 SINGULAR VALUE DECOMPOSITION AND PROJECTION ON LOWER RANK SUBSPACES
Removing units from a neural network's 's dense layer can never be done completely without change to input-output behaviour, unless the weight matrix is of reduced rank to begin with. Assuming that this is not the case, we need method to generate the closest possible lower rank representation of weight matrix, and find projection onto it. Here introduce such method and specify what we mean by closest possible.
Singular value decomposition. Let A mm be an arbitrary, real-valued matrix of rank k < mm,, n)). Then there exist orthogona matrices U E R V E Rnx=, as well as rectangular diagonal matrix Σ E mxn with non-negative real entries along its diagonal, such that
A AEVV.
(10)
This representation is called the Singular Value Decomposition of A. The non-negative diagonal entries o, of are called singular values of A, and are usually given descending order, i.e. 22 ... σ2 O.
For some applications, slightly different notation chosen by dropping any rows and columns of Σ that consist entirely of zeros. This yields quadratic diagonal matrix Σ k with & non- negative diagonal entries. U E | and V E Rgn are cropped accordingly by dropping the respective columns.
"UU-=UC =
11

---

Under review as a conference paper at ICLR 2021
A measure for closeness. Let further
||
||I denote the Frobenius norm
m
||||
l trace( A² A).
(II)
j=1
It can be shown that ||||| oo (A), since (;( corresponds to the square root of the ith non-zero eigenvalue of A A.
The closest lower rank representation. The matrices U and act as projectors to and from a rank k subspace of A, spanned by the k non-7ero entries o; of Σ. If we now wish to find projection from A a matrix rank r < k, all we need to do set the singular values 0411.... Fn to zero, yielding matrix A, of rank The Eckart- Eckart-Younge Young((Mirs(()) theorem states that this matrix is the closest rank r representation of A under the Frobenius norm, or
A A,I|F < ||A
B AB of rank [)
(12)
This property is the core for a number of rank reduction techniques such as principal component analysis (PCA).
Application for network modifications. Think now of A as the weight matrix for a dense layer. As we already stated, the singular values d; are always non-negative and ordered by descending magnitude. In particular, the magnitudes and magnitude distribution of the singular values allows us infer how many units are actually needed to earry all of this layer's information.
This number may be surprisingly small compared to the number of units present, since (given a large enough dense layer) normally the first few singular values make up for the majority of contained
information (Xue et al. 2013). A number of publications (Denton et al., 2014 Girshick, 2015) make use of this fact for various pruning techniques.
A.3 ANALYTICAL CRITERIA FOR MODEL. SELECTION
The recommendation module analyses each potential network topology modification, and scores it according to two criteria. For the first order decision criterion, the number of layer's removable neurons calculated
n2, = Ho,, = 1.... k l, < 61 Vo, < 62, 61, 62 E RR)
(13)
This number is compared to the layer's total neuron count nt. Let be the number of hidden layers the network, and 4 be the index the layer in which the modification was performed. Then each modification candidate is given score between 0 and 1, calculated as follows:
add neurons: - n1//1
add layer: (1 - -_,//ne)) Li/c
remove neurons: ny/nu
remove layer: 1, if n,, ny, otherwise (n,/ni) 44/4
In particular, the higher the layer number, the more likely a layer is added or removed. For further refinement in the future, we aim to replace this selection criterion with more sophisticated formula, which would improve the optimization process beyond the results presented herein.
The second order decision criterion is discussed in the next section.
A.4 STATISTICAL CRITERIA FOR MODEL SELECTION
The Bayesian information criterion, also known as Schwarz information criterion (BIC), was derived by |Shwwarz 1978) to address the problem of selecting between (statistical) models of different dimension. It takes into account the number of parameters of the given model, the sample size of the input data, as well as the a-posteriori model eттоr computed from a likelihood function of the model given its parameters and input data. Since methods from Bayesian statistics applied, assumed that the underlying data are independent and identically distributed from a family of allowed distributions.
12

---

Under review as a conference paper at ICLR 2021
The Bayesian information criterion is given by
BIC = Inn - k -
2 In L,
(14)
where l is the number of model parameters, n the sample size, and L the likelihood function.
In our application, sample size is constant throughout the whole process. L needs to be estimated or calculated a-posteriori after a network operation has been performed. Our modifications are intended keep the change in input-output behaviour minimal, therefore the difference in L. will be very small between any two modifications. Eq. 414) thus becomes
BIC R 01 +1 AL
(15)
where 61 and C2 are constant and AL the (small) change in error depending on the performed operation. Since the number of parameters k may be well above 10" and c In n. > 1 is dependent on the sample size, we neglect AL and the constants, and directly use it as a second order constraint when deciding which modifications to apply.
A.5 EVALUATING COMPETING BRANCHES
Our main objective function is to maximize the accuracy achieved by any given neural network through continuous network surgery. It is therefore sensible retain those network candidates in algorithm I line T0 that reach the highest validation accuracy score. Note that we are scoring on validation accuracy instead of accuracy to avoid overfitting.
Focusing on accuracy alone is a greedy approach and carries the risk of getting stuck in local optima. We overcome this by additionally rewarding modification candidates that show greater accuracy gain. However we need to make a distinction when rewarding this gain. We share Cai et al. (2018) 's rationale that an increase of 1% needs to be weighed higher in case it happens from 90 to 91 % accuracy rather than 70 to 71 %. At the same time, if an operation keeps the accuracy constant at e.g. 95% we can assume that a local optimum may have been reached. Therefore an operation that leads to an accuracy increase from 90% to 94%, showing potential for further improvement, should be regarded higher even though the reached accuracy is lower. Lastly we do not wish to neglect network size. A 1% accuracy increase might never be favourable at all, if the required increase in network size is "too big".
Note that it is hard to define when network has indeed become "too big". fact that is emphasized by the large number of publications dealing with pruning techniques (Blalock et al.) 2020).
We need to find way to balance these three components (accuracy, accuracy gain, network size), and to create a composite score by which we can rank the performance of current branches. As an additional restriction, the composite score should not depend on any global candidate statisties, nor do we want to set a global limit for network size. Therefore we cannot in any meaningful way regard the total number of parameters of modification candidate, since we lack overall compari- son. Instead, for each candidate, we store the network size as a fraction of the candidate's parent's s network size. Thus, a network size fraction greater 1 indicates growth, a fraction smaller indicates shrinking, and the identity operation yields a size fraction of exactly 1.
We want the scoring function as well its first order derivative to be strictly increasing with accu- racy gain, but decreasing with size fraction. This behaviour needs to hold even when accuracy gain becomes network size fraction becomes 1. The following scoring function fulfills all specified requirements:
$ a + exp (Aa)
(16)
exp (f)
where a is the current accuracy, Aa the current accuracy gain, and Af the current network size fraction.
13

---

Under review as a conference paper at ICLR 2021
Table 3: Scoring example. The layer number is given in reference the hidden layers. Winning operations, that are kept new concurrent branches, are indicated with an asterisk.
Operation
Position
Accuracy
Acc. gain
Param. frac.
Score
Remove layer Identity
layer
0.6023
0.0274
0.9977
0.9812
0.6016
0.0268
1.0000
0.9795
Remove Neuron
layer 5
0.5829
0.0081
0.9997
0.9539
Add layer Add 20 Neurons
before layer 4
0.5773
0.0025
1.0030
0.9450
layer
0.5678
-0.0070
2.6544
0.6377
B APPENDIX B HYPERPARAMETER SETTINGS AND MACHINE SPECIFICATIONS
All our simulations were performed Windows 10 machine with an Intel(R) Core(TM) 17-9700K
CPU 3.6GHz processor and 64,0 GB RAM.
All our code was implemented in python 3.7.7 using tensorflow 2.1.0. We chose the following hyperparameters:
The modification module.
Amount of re-training per modification type:
Identity: re-training batches
Add Neuron: re-training batches
Add Layer: 10 re-training batches
Remove Neuron: 10 re-training batches
Remove Layer: 10 re-training batches
Truncated SVD: re-training batches
The recommendation module.
Absolute singular value threshold: <2 0.3
Relative singular value threshold: 0.005
Recommendation style: best scoring per modification type
Include additional random draw: True
The Surgeon.
Training time for winning branch: 100 Epochs
Initial pre-training: 10 Epochs
Interval between decision points: 10 Epochs
Concurrent branches:
Maximum re-draws per decision step:
Number of batches for re- re-training: 25 batches
The testing setup.
Random seeds: 6, 63, 72, 77, 97
Loss function: Sparse categorical crossentropy
Optimizer: SGD
learning rate 0.005
Metrics: accuracy
Batch size: 16
14


Reviews:
[{"review_id": "cLPFQSLgaM0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a neural network training framework based on a genetic algorithm called Surgeon. The idea consists of two modules. First, a series of network structure changes that aims to minimize the network input and output. Second, a heuristic (with regard to the objective function) for ranking and selecting the potentially good network modifications. The algorithm works iteratively between the former propose candidate network changes, and the later decides which candidate to accept and evolve into.\n\nThe paper is well written and easy to follow. As the authors stated, the paper is the first of the series of work. The framework presented is rigorous and flexible. I am fairly optimistic that the idea itself is pointing to a promising direction for exploring the neural network structure. However, I think the theoretical basis and experiments are a bit weak in their current form.\n\nI have two main concerns. First, the heuristic used by Surgeon, either by weights or by BIC, does not establish direct association to the objective function that the network is optimizing. Therefore there is no any form of guarantee that the evolution will converge to some optima. Second, the experiments are initiated from rather simple baseline, without comparing against more sophisticated work or discussing why other state-of-the-art results are not relevant in this context.\n\nOverall I like this paper, but for the two reasons listed above I think it may fall short to be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DgjgMZvsBke", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "this paper presents the Surgeon, a ANN/evolutionary algorithm hybrid optimization de- signed for neural architecture search. On SVHN and CIFAR-10, the network generated by the Surgeon is able to outperform the baseline in case of suboptimal topologies, or reach comparable accuracies while pruning the underlying network structure to less resource-intensive topologies. My major concern is the novelty, as the SVD technique and net2net techniques have all be proposed before. The experiment is not solid. The accuracy on Cifar is far from the state of the art. There is no large scale datasets. There's no performance comparison with related work, making the paper less convincing. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5knkbiyADxo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis study aims to search for topology jointly with the network training. The topology is optimized by a heuristic search in structural modifications including adding/removing neurons/layers. Experiments on SVHN and CIFAR-10 are conducted to show the effectiveness. \n\nPros:\nThe paper is well organized. The proposed method enjoys good interpretability because the decision of modification is made based on the weight matrix properties by SVD and projecting onto its lower-rank subspaces. \n\nCons:\n1.\tThe major problem of this study is its extendibility. The method is only tested on fully-connected networks with very limited network depths and layer sizes. When the network is deep with much more non-linear activations, is the proposed method still valid? Besides, the authors only claim that the method can be generalized to convolutional network. But there is no corresponding description about how to generalize to convolutional network and its experimental results. \n2.\tThe current experiments are not sufficient. The network and datasets used to test are too simple and the performance is far from being acceptable. It is weird to use fully-connected network on real-image datasets, CIFAR-10 and SVHN. Besides, the training time, computational and memory cost should also be reported and compared with the baseline method. \n3.\tThe novelties are limited. The methods proposed in this paper are mainly heuristic implementations. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0v6kpi4hV1d", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper seems to devise a set of modification modules and develops an overall genetic algorithm that utilizes the recommendation module to identify optimized topologies for networks.\n\nThe paper is a combination of genetic algorithm with recommendation rule which builds on some tools like BIC. While these tools are organized in a reasonable manner, the overall idea seems rather simplistic and the contributions seem rather humble.\n\nImportantly, the main concerns about the paper revolves around the evaluation. For example, the surgeon is tested on very small networks. As such, the evaluation seems very primitive. Without the results of networks for large datasets like ImageNet, it is hard to give credit to their evaluation nor gain confidence about the performance of this method in real scenarios.\n\nAlso, it would be very interesting to see the evolution of the topology. Maybe some figures may help the readers visualize the overall procedure. While this may be excessive for small networks like the ones authors rely on, I believe this may lead to new and interesting observations.\n\nFurthermore, it would be very inspiring if the paper presents some prospect of the work in improving the mainstream neural architecture search algorithms. I believe such evaluation with some empirical results would make this a much more relevant paper for people working on improving neural architecture search.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a framework to search for the topology of an artificial neural network jointly with the network training, via a genetic algorithm that can decide structural actions, such as addition or removal of neurons and layers. An extra heuristic based on Bayesian information criterion helps the optimization process decide on its decisions about the topology. They demonstrate improvements over baseline fully-connected networks on SVHN and (augmented) CIFAR-10.

Reviewers and myself agree that this is an interesting idea, and that the paper is easy to follow. While I may not agree that we need to achieve SOTA on these datasets, or see large scale ImageNet-type experiments for novel ideas, I agree with the reviewers, esp R1's point that the current experiments are not satisfactory to meet the bar for acceptance at ICLR.

CIFAR-10 and SVHN are well-established tasks, and showing baseline accuracy of 75%/48% on them respectively doesn't seem to do them justice, especially when most methods (even with low compute requirements) can get > 95% on both, for the past few years. For this work to be of interest to the broader community, it needs to be improved to incorporate at least respectable baselines on these small datasets, and perhaps be improved to work beyond fully connected networks.

At this stage, we need to see a revision of the method and see improvements before an acceptance decision can be made.
Decision: Reject