Neural Network Surgery: Combining Training with Topology Optimization

1 INTRODUCTION . A common problem for any given machine learning task to be addressed with artificial neural networks ( ANNs ) is how to choose a sufficiently good network topology . Picking one that is too small may not yield acceptable prediction accuracy . To improve results , one can keep adding structural elements to the network until the desired accuracy value has been reached . However , at the same time too large networks may cause an explosion in computational cost for both training and evaluation . Where the sweet spot in between lies is unclear , and heavily dependent on the given task . A priori optimization is not easily possible , since reliable estimates on network behaviour already require training results , and also there exists no generalization for which topology will fit which problem . Researchers have applied a number of search strategies such as random search ( Li & Talwalkar , 2019 ) , Bayesian optimization ( Kandasamy et al. , 2018 ) , reinforcement learning ( Zoph & Le , 2017 ) , and gradient-based methods ( Dong & Yang , 2019 ) . Another technique applied since at least Miller et al . ( 1989 ) are so called ( neuro- ) evolutionary algorithms . These algorithms serve to evolve the network architecture , often also training network weights at the same time ( Elsken et al. , 2019 ) . In this paper we propose a novel training regime incorporating a genetic algorithm that reduces computational cost compared to state of the art approaches of this kind ( Dong & Yang , 2019 ; Li & Talwalkar , 2019 ) . We achieve this by re-using network weights for competing modification candidates instead of retraining each net from scratch , branching off modification candidates during training , and letting them compete against each other until a new main branch is selected . This better fuses the evolutionary optimization paradigm with the ANN training into an integrated framework that folds both processes into a single training/topology optimization hybrid . As such , evolutionary steps are not carried out by a meta-controller or other black-box-like implementations . We make use of mathematical tools such as singular value decomposition ( SVD ) and the Bayesian information criterion ( BIC ) ( Schwarz , 1978 ) for network weight analysis , decision making , and structural modifications . Network modifications are performed by adapting existing weights such as to incur minimal changes to input-output behaviour . Our framework for a combined ANN training and neural architecture search consists of three main components : a module that can perform a number of minimally invasive network operations ( “ surgeries ” ) , a module that analyses network weights and can give recommendations which modifications are most likely to increase ( validation ) accuracy , and finally a module that serves as a genetic algorithm ( the “ Surgeon ” ) , containing the former two while gradually evolving any given starting network . With the Surgeon , we are able to evolve and improve models for the SVHN ( Netzer et al. , 2011 ) and CIFAR-10 ( Krizhevsky , 2009 ) datasets for varying starting topologies . We achieve particularly good results on starting topologies that would a posteriori have proven to be suboptimal . A great benefit of our approach is that it adds topology optimization to the ML training at a very limited additional computational cost . Convergence is reached for all test cases within a few CPU hours . This paper contributes a computationally cheap ansatz for a genetic neural architecture search algorithm that makes evolutionary decisions based on mathematical analysis . 2 RELATED WORK . Neural architecture search ( NAS ) has been an increasingly popular research topic for many years ( Elsken et al. , 2019 ) , starting as early as Miller et al . ( 1989 ) , who presented one of the earliest neuroevolutionary algorithms to search for suitable network topologies . Recent approaches by Dong & Yang ( 2019 ) ; Li & Talwalkar ( 2019 ) , and Zoph & Le ( 2017 ) reach competitive performance on benchmark datasets such as CIFAR-10 . However , this often comes at the cost of vast computational resources , with Zoph & Le ( 2017 ) making use of up to 800 GPUs for several weeks . Cai et al . ( 2018 ) attempt to reduce computational costs by re-using network weights , as well as training and applying a reinforcement meta-controller for structural decisions . They make use of a number of function-preserving transformations ( net2net ) introduced by Chen et al . ( 2016 ) , and extend them to allow also non-sequential network structures , such as DenseNet ( Huang et al. , 2017 ) . DiMattina & Zhang ( 2010 ) introduce and rigorously prove conditions , under which gradual changes of the parametrization of a neural network are possible , while keeping the input-output behaviour constant . There are also a number of neural architecture search strategies that do not depend on manual network modifications . Dong & Yang ( 2019 ) represent the search space as a directed acyclic graph and propose an efficient search algorithm . İrsoy & Alpaydın ( 2020 ) learn the network structure via so-called “ budding perceptrons ” , in which an extra parameter is learned for each layer , that indicates whether or not any given node needs to branch out again or be removed altogether . Their method focuses on growing the network to the required size from a minimal starting topology . Frankle & Carbin ( 2019 ) present a method to identify particularly good network initializations that can train sparse networks to competitive accuracy . Another approach in NAS is to prune down from a larger starting topology ( Blalock et al. , 2020 ) . Popular pruning techniques include applying SVD to existing network weights ( Girshick , 2015 ; Denton et al. , 2014 ; Xue et al. , 2013 ) . The novelty of our research lies in combining existing tools such as net2net ( Chen et al. , 2016 ) and SVD with a genetic algorithm that modifies the given network in a decision based process instead of utilizing a black-box like decision module . To our best knowledge no such method has yet been proposed . 3 METHODS . This work introduces and utilizes three main modules : • modification module : performs network modifications ( “ surgeries ” ) so as to incur minimal changes to input-output behaviour . • recommendation module : analyzes network weights and gives recommendations on which operations are most likely to improve network accuracy . • “ the Surgeon ” : a genetic algorithm that links the above two modules , and gradually evolves a given starting network . 3.1 THE MODIFICATION MODULE . We need to be able to perform a number of different modifications that can restructure network architecture . Ideally , these operations should not change at all the input-output behaviour . In cases where this is not possible , we aim to perform minimal impact changes instead . We perform four different types of modifications : adding or removing neurons or whole layers . Adding neurons and layers . Under piecewise linear activation functions such as ReLu , one can always add neurons to a hidden layer , or even add whole layers , without any change to the overall network behaviour . Adding a whole layer is done by using an identity matrix as a new weight matrix for this inserted layer . We add a new neuron to any layer by dividing all weights for one existing neuron by two , and duplicating the respective connections in the next layer ( cf . appendix A.1 ) . Removing neurons . Removing neurons from a layer is rarely possible without changing the inputoutput behavior unless some of the units are degenerate to begin with , i.e . the weight matrix is of reduced rank . For a modification with minimal impact on input-output behaviour we need the closest1 possible projection onto a lower rank subspace . The Eckart-Young ( -Mirsky ) theorem states that this projection can be found by applying Singular Value Decomposition ( SVD ) to the weight matrix : Let A ∈ Rm×n be the ( weight ) matrix of some layer of interest in a neural network , then there exists a representation A = UΣV T ( 1 ) with orthogonal U ∈ Rm×m , V ∈ Rn×n and rectangular diagonal Σ ∈ Rm×n with k ≤ min ( m , n ) non-negative real entries σi along its diagonal , called singular values of A ( cf . appendix A.2 ) . In order to reduce the rank of A to r < k , we set σi = 0 ∀i > r , and drop associated columns/rows of U and V , which is feasible since by definition σi ≥ σi+1 . We then project to a smaller matrix Ã by computing UΣ . To project back to the original layer size , so that the layer weight shapes match again , we multiply the weights of the subsequent layer with V T . Projecting onto a lower rank subspace by setting singular values to zero is sometimes called truncated SVD , and is used in network pruning ( Denton et al. , 2014 ; Girshick , 2015 ; Xue et al. , 2013 ) . This technique adds an intermediate layer of ( potentially much fewer ) neurons , which in case of very large weight matrices can drastically reduce the overall connection count . Note that this projection method is not concerned at all with a potential activation function after the network ’ s modified layer . In particular , changing a layer ’ s activation function is in general not possible without changes to input/output behaviour , since activation functions are usually non-linear . We therefore do not perform any additional modifications to counterbalance a change of activation . Removing layers . We remove whole layers in a similar fashion , by multiplying the layer ’ s weights onto the weights of the next layer . This again ignores any activation function between the two layers , and will thus cause a change in input-output behaviour . Recuperation from surgery . As we have seen , network modifications will in general cause a small change in input-output behaviour , typically leading to a loss in prediction accuracy . In order to make up for this , all network modifications are given a small amount of recuperative training ( several batches ) before any comparison is made . The retraining amount was determined by trials and statistical analysis based on a dataset2 which we do not otherwise use in our results , to avoid overfitting the search algorithm to a specific dataset . 3.2 THE RECOMMENDATION MODULE . For the decision when to execute which network modification , we perform a two-step analysis . As a first order criterion , we compute the amount of information carried by each neuron by looking at the layer ’ s singular values . The number of neurons that may be removed from a layer depends on the count of singular values that are ( close to ) zero , or are several orders of magnitude smaller than the layer ’ s largest singular value : removable neurons = { σi , i = 1 . . . k | σi < 1σ1 ∨ σi < 2 , 1 , 2 ∈ R+ } ( 2 ) 1under Frobenius norm ‖ · ‖F 2MNIST , ( LeCun et al. , 1998 ) The number of removable neurons in relation to the layer ’ s total neuron count gives an indication whether addition or removal operations are preferable ( cf . appendix A.3 ) . The second order decision basis is the Bayesian information criterion ( BIC ) . It weighs the number of model parameters and training samples against the likelihood function of predicted vs. true target data . Since all operations are chosen to incur minimal changes to predicted values , the difference in the likelihood function between two modifications becomes negligible compared to the number of parameters , which allows us to reduce the formula to the parameter count of the modified network ( cf . appendix A.4 ) . Thus , all potential network modifications are ranked by two parameters . The Surgeon has two modes : it can either pick the top n ranking operations per decision step ( with n being a tunable hyperparameter ) , or select the highest ranking operation of each type . Our experiments are performed with the latter option .

Summary: this paper presents the Surgeon, a ANN/evolutionary algorithm hybrid optimization de- signed for neural architecture search. On SVHN and CIFAR-10, the network generated by the Surgeon is able to outperform the baseline in case of suboptimal topologies, or reach comparable accuracies while pruning the underlying network structure to less resource-intensive topologies. My major concern is the novelty, as the SVD technique and net2net techniques have all be proposed before. The experiment is not solid. The accuracy on Cifar is far from the state of the art. There is no large scale datasets. There's no performance comparison with related work, making the paper less convincing.