RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks

1 INTRODUCTION . Deep neural networks solve a variety of problems using multiple layers to progressively extract higher level features from the raw input . The commonly adopted method to train deep neural networks is backpropagation ( Rumelhart et al . ( 1985 ) ) and it has been around for the past 35 years . Backpropagation assumes that the function is differentiable and leverages the partial derivative w.r.t the weight wi for minimizing the function f ( x , w ) as follows , wi+1 = wi − η∇f ( x , w ) ∆wi , where η is the learning rate . Also , the method is efficient as it makes a single functional estimate to update all the weights of the network . As in , the partial derivative for some weight wj , where j 6= i would change once wi is updated , still this change is not factored into the weight update rule for wj . Moreover , it may not even be optimal for all weights to move in the same direction as obtained from the gradients in the previous layer . Although deep neural networks are non-convex ( and the weight update rule measures approximate gradients ) , this update rule works surprisingly well in practice . To explain the above observation , recent literature ( Du et al . ( 2019 ) ; Li & Liang ( 2018 ) ) argues that because the network is over-parametrized , the initial set of weights are very close to the final solution and even a little bit of nudging using gradient descent around the initialization point leads to a very good solution . We take this argument to another extreme - instead of using gradient based optimizers - which provide strong direction and magnitude signals for updating the weights ; we explore the region around the initialization point by sampling weight changes to minimize the objective function . Formally , our weight update rule is wi+1 = { wi , f ( x , wi ) < = f ( x , wi + ∆wi ) wi + ∆wi , f ( x , wi ) > f ( x , wi + ∆wi ) , where ∆wi is the weight change hypothesis . Here , we explicitly test the region around the initial set of weights by computing the function and update a weight if it minimizes the loss , see Fig . 1 . Surprisingly , our experiments demonstrate that the above update rule requires fewer weight updates compared to backpropagation to find good minimizers for deep neural networks , strongly suggesting that just exploring regions around randomly initialized networks is sufficient , even without explicit gradient computation . We evaluate this weight update scheme ( called RSO ; random search optimization ) on classification datasets like MNIST and CIFAR-10 with deep convolutional neural networks ( 6-10 layers ) and obtain competitive accuracy numbers . For example , RSO obtains 99.1 % accuracy on MNIST and 81.8 % accuracy on CIFAR-10 using just the random search optimization algorithm . We do not use any other optimizers for optimizing the final classification layer . Although RSO is computationally expensive ( because it requires updates which are linear in the number of network parameters ) , our hope is that as we develop better intuition about structural properties of deep neural networks , we will be able to accelerate RSO ( using Hebbian principles , Gabor filters , depth-wise convolutions ) . If the number of trainable parameters are reduced drastically ( Frankle et al . ( 2020 ) ) , search based methods could be a viable alternative to back-propagation . Furthermore , since architectural innovations which have happened over the past decade use backpropagation by default , a different optimization algorithm could potentially lead to a different class of architectures , because minimizers of an objective function via different greedy optimizers could potentially be different . 2 RELATED WORK . Multiple optimization techniques have been proposed for training deep neural networks . When gradient based methods were believed to get stuck in local minima with random initialization , layer wise training was popular for optimizing deep neural networks ( Hinton et al . ( 2006 ) ; Bengio et al . ( 2007 ) ) using contrastive methods ( Hinton ( 2002 ) ) . In a similar spirit , recent work , Greedy InfoMax by Löwe et al . ( 2019 ) maximizes mutual information between adjacent layers instead of training a network end to end . Taylor et al . ( 2016 ) finds the weights of each layer independently by solving a sequence of optimization problems which can be solved globally in closed form . Weight perturbation ( Werfel et al . ( 2004 ) ) based methods have been used for approximate gradients estimation in situations where gradient estimation is expensive . However , these training methods do not generalize to deep neural networks which have more than 2-3 layers and its not shown that the performance increases as we make the network deeper . Hence , back-propagation with SGD or other gradient based optimizers ( Duchi et al . ( 2011 ) ; Sutskever et al . ( 2013 ) ; Kingma & Ba ( 2014 ) ) are commonly used for optimizing deep neural networks . Recently , multiple works have proposed that because these networks are heavily over-parametrized , the initial set of random filters is already close to the final solution and gradient based optimizers only nudge the parameters to obtain the final solution ( Du et al . ( 2019 ) ; Li & Liang ( 2018 ) ) . For example , only training batch-norm parameters and keeping the random filters fixed can obtain very good results with heavily parametrized very deep neural networks ( > 800 layers ) as shown in Frankle et al . ( 2020 ) . It was also shown that networks can be trained by just masking out some weights without modifying the original set of weights by Ramanujan et al . ( 2020 ) - although one can argue that masking is a very powerful operator and can be used to represent an exponential number of output spaces . The network pruning literature covers more on optimizing subsets of an over-parametrized randomly initialized neural network ( Frankle & Carbin ( 2019 ) ; Li et al . ( 2017 ) ) . Our method , RSO , is also based on the hypothesis that the initial set of weights is close to the final solution . Here we show that gradient based optimizers may not even be necessary for training deep networks and when starting from randomly initialized weights , even search based algorithms can be a feasible option . Recently , search based algorithms have gained traction in the deep learning community . Since the design space of network architectures is huge , search based techniques are used to explore the placement of different neural modules to find better design spaces which lead to better accuracy ( Zoph & Le ( 2016 ) ; Liu et al . ( 2018 ) ) . This is done at a block level and each network is still trained with gradient descent . Similar to NAS based methods , weight agnostic neural networks ( WANN ) ( Gaier & Ha ( 2019 ) ) also searches for architectures , but uses a fixed set of weight values [ −2 , −1 , −0.5 , +0.5 , +1 , +2 ] . WANN operates at a much finer granularity as compared to NAS based methods while searching for connections between neurons and does not use gradient descent for optimization . Algorithms like Deep Neuroevolution by Such et al . ( 2017 ) and evolution strategies ( ES ) by Salimans et al . ( 2017 ) are search based optimization algorithms which have been used for training neural networks for reinforcement learning . ES is comprehensively reviewed by Beyer & Schwefel ( 2004 ) . Both Deep Neuroevolution and Salimans et al . ( 2017 ) create multiple replicas ( children ) of an initial neural network by adding small perturbations to all the weight parameters and then update the parameters by either selecting the best candidate or by performing a weighted average based on the reward . Both the methods update all the parameters of the network in each update . The problem with changing all the weights is that updating all the parameters of the network at once leads to random directions which are unlikely to contain a direction which will minimize the objective function and slows down learning ( results shown in Section 4.5 ) . Also , these methods were only trained on networks with 2-3 hidden layers , which is fairly shallow when compared to modern deep architectures . 3 APPROACH . Consider a deep neural network with D layers , where the weights of a layer d with nd neurons is represented byWd = { w1 , w2 .. , wid , .. wnd } . For an input activationAd−1 = { a1 , a2 , ... and−1 } , Wd generates an activation Ad = { a1 , a2 , ... and } , see Fig 2 . Each weight tensor wid ∈ Wd generates an activation ai ∈ Ad , where ai can be a scalar or a tensor depending on whether the layer is fully connected , convolutional , recurrent , batch-norm etc . The objective of the training process is to find the best set of weights W , which minimize a loss function F ( X , L ; W ) given some input data X and labels L. To this end , we initialize the weights of the network with a Gaussian distribution N ( 0 , √ 2/|wid | ) , like He et al . ( 2015 ) . The input data is also normalized to have zero mean and unit standard deviation . Once the weights of all layers are initialized , we compute the standard deviation σd of all elements in the weight tensor Wd . In the weight update step for a weight wj ∈ wid , ∆wj is sampled from ∼ N ( 0 , σd ) . We call this ∆Wj which is zero for all weights of the network but for wj , where j ∈ id . For a randomly sampled mini-batch ( x , l ) ∈ X , L , we compute the loss F ( x , l ; W ) for W , W + ∆Wj and W −∆Wj . If adding or subtracting ∆Wj reduces F , W is updated , otherwise the original weight is retained . This process is repeated for all the weights in the network , i.e. , to update all the weights of the network once , F needs to be computed three times the number of weight parameters in the network , 3|W | . We first update the weights of the layer closest to the labels and then sequentially move closer to the input . This is typically faster than optimizing the other way , but both methods lead to good results . This algorithm is described in Algorithm 1 . In Algorithm 1 , in line 12 , we sample change in weights from a Gaussian distribution whose standard deviation is the same as the standard deviation of the layer . This is to ensure that the change in weights is within a small range . The Gaussian sampling can also be replaced with other distributions like uniform sampling from ( −2σd , 2σd ) or just sampling values from a template like ( −σd , 0 , σd ) and these would also be effective in practice . The opposite direction of a randomly sampled weight is also tested because often it leads to a better hypothesis when one direction does not decrease the loss . However , in quite a few cases ( close to 10 % as per our experiments ) , not changing the weight at all is better . Note that there is no concept of learning rate in this algorithm . We also do not normalize the loss if the batch size increases or decreases as the weight update step is independent of the magnitude of the loss . There is widespread belief in the literature that randomly initialized deep neural networks are already close to the final solution ( Du et al . ( 2019 ) ; Li & Liang ( 2018 ) ) . Hence , we use this prior and explore regions using bounded step sizes ( N ( 0 , σd ) ) in a single dimension . We chose to update one weight at a time instead of sampling all the weights of the network as this would require sampling an exponential number of samples to estimate their joint distribution . RSO will be significantly faster even if prior knowledge about the distribution of the weights of individual neurons is used .

Summary: Instead of back-propagation, the authors consider a randomized search heuristic to train the parameters of neural networks. The proposed work is based on the hypothesis that the initial set of neural network weights is close to the final solution. The authors identify the problem that existing randomized search methods update all the parameters of the network in each update. Thus the proposed method updates only a single weight per iteration. Experimental results on MNIST and CIFAR10 show that the proposed method delivers competitive results.