Deep Gradient Boosting -- Layer-wise Input Normalization of Neural Networks

1 INTRODUCTION . Boosting , along side deep learning , has been a very successful machine learning technique that consistently outperforms other methods on numerous data science challenges . In a nutshell , the basic idea of boosting is to sequentially combine many simple predictors in such a way that their combined performance is better than each individual predictor . Frequently , these so called weak learners are implemented as simple decision trees and one of the first successful embodiment of this idea was AdaBoost proposed by Freund & Schapire ( 1997 ) . No too long after this , Breiman et al . ( 1998 ) and Friedman ( 2001 ) made the important observation that AdaBoost performs in fact a gradient descent in functional space and re-derived it as such . Friedman ( 2001 ) went on to define a general statistical framework for training boosting-like classifiers and regressors using arbitrary loss functions . Together with Mason et al . ( 2000 ) they showed that boosting minimizes a loss function by iteratively choosing a weak learner that approximately points in the negative gradient direction of a functional space . Neural networks , in particular deep neural nets with many layers , are also trained using a form of gradient descent . Stochastic gradient descent ( SGD ) ( Robbins & Monro , 1951 ) has been the main optimization method for deep neural nets due to its many desirable properties like good generalization error and ability to scale well with large data sets . At a basic level , neural networks are composed of stacked linear layers with differentiable non-linearities in between . The output of the last layer is then compared to a target value using a differentiable loss function . Training such a model using SGD involves updating the network parameters in the direction of the negative gradient of the loss function . The crucial step of this algorithm is calculating the parameter gradients and this is efficiently done by the backpropagation algorithm ( Rumelhart et al. , 1988 ; Werbos , 1974 ) . Backpropagation has many variations that try to achieve either faster convergence or better generalization through some form of regularization . However , despite superior training outcomes , accelerated optimization methods such as Adam ( Kingma & Ba , 2015 ) , Adagrad ( Duchi et al. , 2011 ) or RMSprop ( Graves , 2013 ) have been found to generalize poorly compared to stochastic gradient descent ( Wilson et al. , 2017 ) . Therefore even before using an explicit regularization method , like dropout ( Srivastava et al. , 2014 ) or batch normalization ( Ioffe & Szegedy , 2015 ) , SGD shows very good performance on validation data sets when compared to other methods . The prevalent explanation for this empirical observation has been that SGD prefers ” flat ” over ” sharp ” minima , which in turn makes these states robust to perturbations . Despite its intuitive appeal , recent work by ( Dinh et al. , 2017 ) cast doubt on this explanation . This work introduces a simple extension of SGD by combining backpropagation with gradient boosting . We propose that each iteration of the backpropagation algorithm can be reinterpreted by solving , at each layer , a regularized linear regression problem where the independent variables are the layer inputs and the dependent variables are gradients at the output of each layer , before non-linearity is applied . We call this approach deep gradient boosting ( DGB ) , since it is effectively a layer-wise boosting approach where the typical decision trees are replaced by linear regressors . Under this model , SGD naturally emerges as an extreme case where the network weights are highly regularized , in the L2 norm sense . We hypothesize that for some learning problems the regularization criteria doesn ’ t need to be too strict . These could be cases where the data domain is more restricted or the learning task is posed as a matrix decomposition or another coding problem . Based on this idea we further introduce INN , a novel layer normalization method free of learnable parameters and show that it achieves competitive results on benchmark image recognition problems when compared to batch normalization ( BN ) . 2 DEEP GRADIENT BOOSTING . Consider a learning problem where training examples ( x , y ) are generated from a probability distribution P ( x , y ) where x ∈ RN is the space of measurements and y ∈ R is the space of labels . The goal of gradient boosting is to find an approximation F̂ ( x ) to a function F ( x ) that minimizes a specified loss function L ( F ( x ) , y ) : F̂ = arg min F L ( F ( x ) , y ) ( 1 ) Gradient boosting solves Eq . 1 by seeking functions ht ( x ) , so called weak learners , from a differentiable class H such that F̂ = ∑ t ht ( x ) . Starting from a constant function F0 and applying steepest descent we can find F̂ by iteratively updating with learning rate γ : Ft ( x ) = Ft−1 ( x ) − γ∇Ft−1L ( Ft−1 ( x ) , y ) ( 2 ) The weak learners ht are thus found by fitting the training data to the pseudo-residuals rt = ∇Ft−1L ( Ft−1 ( x ) , y ) hence in practice : Ft ( x ) = Ft−1 ( x ) − γht ( x ) ( 3 ) Next , consider a neural network F ( x ) with parameters w [ l ] comprised of L successive dense linear layers F [ l ] ( x ) = xTw [ l ] with non-linearities ψ : F ( x ) = F [ 0 ] ( ψ ( F [ 1 ] ( ... ψ ( F [ L ] ( x ) ) ) ) ) . Similarly , starting from parameter w0 and following steepest descent the network weights are updated at iteration t : wt = wt−1 − γ∇wt−1L ( F ( xt ) , yt ) ( 4 ) At each iteration , for a fixed layer l the gradient of the parameter w [ l ] ij can be calculated using the chain rule : ∂Lt ∂w [ l ] ij = ∂Lt ∂F [ l ] t ∂F [ l ] t ∂w [ l ] ij = ∂Lt ∂F [ l ] t ψ ( F [ l−1 ] t ) i = ∂Lt ∂F [ l ] t o [ l ] ti ( 5 ) where o [ l ] i is the output of the layer l − 1 at iteration t and position i and the gradients ∂Lt ∂F [ l ] t are calculated using the backpropagation algorithm . Alternatively , using the gradient boosting formula from Eq . 3 and plugging in Ft = F [ l ] ( xt ) = xTt w [ l ] for input xt = o [ l ] t we obtain , after dropping the layer indexes [ l ] for simplicity : oTt wt = o T t wt−1 + γh ( ot ) ( 6 ) We choose the weak learner h ( ot ) in this case to be a linear transformation as well h ( ot ) = oTt vt and fit it to the pseudo-residuals rt = ∂Lt ∂F [ l ] t by solving the associated regularized linear regression : v̂t = arg min v ( ‖rt − oTt v‖22 + α‖v‖22 ) , α > 0 ( 7 ) The primal form solution of Eq . 7 is v̂t = ( oTt ot + αI ) −1otrt and plugging it into Eq . 6 yields the following update rule : wt = wt−1 − γ ( oTt ot + αI ) −1otrt where I ∈ RN×N is the identity matrix . For a given input batch of size B , the update rule becomes : Wt = Wt−1 − γ ( OTt Ot + αI ) −1OTt Rt ( 8 ) where Wt ∈ Rn×m is the weight matrix for a given layer , Ot ∈ RB×n is the input matrix at the same layer while Rt ∈ RB×m is the pseudo-residual matrix ∇FtLt . In practice , the inverse of matrix OTt Ot can be hard to calculate for neural networks with large number of neurons per layer . In consequence , it is often more convenient to use the dual form formulation of the ridge regression solution of Eq . 7 that involves calculating the inverse of OtOTt instead . This is a well known formulation ( e.g . see Chapter 1 of ( Camps-Valls et al. , 2006 ) ) and leads to the analogous weight update rule : Wt = Wt−1 − γOTt ( OtOTt + αI ) −1Rt ( 9 ) The weight update formulas from Eq . 8 and 9 have the drawback that they change in magnitude with either the number of neurons in the neural network layer or batch size . This is an undesirable behavior that can lead to differences in scale between weights from layers of different sizes . In consequence , the terms OTt Ot and OtO T t are scaled accordingly . For large values of α the terms OTt Ot + αI and OtO T t + αI can be approximated by a constant diagonal matrix which leads to the classic SGD solution Wt ≈ Wt−1 − γOTt Rt . This shows that back-propagation with SGD is an extreme case of DGB that implicitly minimizes the magnitude of weight updates . 2.1 EXPERIMENTS . In the following we will compare the performance of DGB and SGD on a variety of reference data sets and explore the effect of the α parameter using simple dense layer neural networks . A summary of the data sets used is given in Table 1 : The first data set is the MNIST database of handwritten digits ( LeCun et al. , 1999 ) comprised of 60,000 28x28 grayscale images for training and 10,000 for testing . In the case of the Higgs data set ( Baldi et al. , 2014 ) we used a randomly chosen subset with 1 million samples out of 10 million for training , and 500 thousands samples for testing . This data comes from the field of high energy physics and was generated by Monte Carlo simulations . The third data set ( Reuters-21578 ) is comprised of 11,228 english language news articles from Reuters international news agency , labeled over 46 topics . The forth data set is the Human Resource ( HR ) data set ( Stacker IV , 2015 ) which was put together by IBM Inc. to predict employee attrition based on 47 features and was randomly split into 1,029 training samples and 441 test samples . The fifth data set is a regression problem for predicting a measure of air quality based on the outputs from an array of chemical sensors together with air temperature and humidity ( De Vito et al. , 2008 ) . All experiments were performed until test set convergence using a gradient update momentum of 0.9 and no decay . For simplicity , network architecture was fixed for each data set and all activation functions were rectified linear units ( ReLU ) ( Nair & Hinton , 2010 ) . The network weights were initialized using a uniform distribution according to ( He et al. , 2015 ) . The loss functions used were cross-entropy loss for classification problems and mean squared loss for regression problems . The performance metrics used were accuracy for classification tasks and root mean squared error ( RMSE ) for regression tasks . In addition , standard deviations were calculated and reported for all performance measures by repeating each experiment 10 times with different random seeds .

Summary: This paper propose an extention method of SGD, deep gradient boosting (DGB), which views the back-propagation procedure as a pseudo-residual targets of a gradient boosting problem. To apply DGB to the real CNNs, DGB is simplified to a input normalization layer, conditioned on the assumption that the convolution kernels should be small. After applying the input normalization layer to CNNs, the model could achieve comparable performance to the model with BN on CIFAR-10 and ImageNet recognition.