Deep Gradient Boosting -- Layer-wise Input Normalization of Neural Networks

1 INTRODUCTION . Boosting , along side deep learning , has been a very successful machine learning technique that consistently outperforms other methods on numerous data science challenges . In a nutshell , the basic idea of boosting is to sequentially combine many simple predictors in such a way that their combined performance is better than each individual predictor . Frequently , these so called weak learners are implemented as simple decision trees and one of the first successful embodiment of this idea was AdaBoost proposed by Freund & Schapire ( 1997 ) . No too long after this , Breiman et al . ( 1998 ) and Friedman ( 2001 ) made the important observation that AdaBoost performs in fact a gradient descent in functional space and re-derived it as such . Friedman ( 2001 ) went on to define a general statistical framework for training boosting-like classifiers and regressors using arbitrary loss functions . Together with Mason et al . ( 2000 ) they showed that boosting minimizes a loss function by iteratively choosing a weak learner that approximately points in the negative gradient direction of a functional space . Neural networks , in particular deep neural nets with many layers , are also trained using a form of gradient descent . Stochastic gradient descent ( SGD ) ( Robbins & Monro , 1951 ) has been the main optimization method for deep neural nets due to its many desirable properties like good generalization error and ability to scale well with large data sets . At a basic level , neural networks are composed of stacked linear layers with differentiable non-linearities in between . The output of the last layer is then compared to a target value using a differentiable loss function . Training such a model using SGD involves updating the network parameters in the direction of the negative gradient of the loss function . The crucial step of this algorithm is calculating the parameter gradients and this is efficiently done by the backpropagation algorithm ( Rumelhart et al. , 1988 ; Werbos , 1974 ) . Backpropagation has many variations that try to achieve either faster convergence or better generalization through some form of regularization . However , despite superior training outcomes , accelerated optimization methods such as Adam ( Kingma & Ba , 2015 ) , Adagrad ( Duchi et al. , 2011 ) or RMSprop ( Graves , 2013 ) have been found to generalize poorly compared to stochastic gradient descent ( Wilson et al. , 2017 ) . Therefore even before using an explicit regularization method , like dropout ( Srivastava et al. , 2014 ) or batch normalization ( Ioffe & Szegedy , 2015 ) , SGD shows very good performance on validation data sets when compared to other methods . The prevalent explanation for this empirical observation has been that SGD prefers ” flat ” over ” sharp ” minima , which in turn makes these states robust to perturbations . Despite its intuitive appeal , recent work by ( Dinh et al. , 2017 ) cast doubt on this explanation . This work introduces a simple extension of SGD by combining backpropagation with gradient boosting . We propose that each iteration of the backpropagation algorithm can be reinterpreted by solving , at each layer , a regularized linear regression problem where the independent variables are the layer inputs and the dependent variables are gradients at the output of each layer , before non-linearity is applied . We call this approach deep gradient boosting ( DGB ) , since it is effectively a layer-wise boosting approach where the typical decision trees are replaced by linear regressors . Under this model , SGD naturally emerges as an extreme case where the network weights are highly regularized , in the L2 norm sense . We hypothesize that for some learning problems the regularization criteria doesn ’ t need to be too strict . These could be cases where the data domain is more restricted or the learning task is posed as a matrix decomposition or another coding problem . Based on this idea we further introduce INN , a novel layer normalization method free of learnable parameters and show that it achieves competitive results on benchmark image recognition problems when compared to batch normalization ( BN ) . 2 DEEP GRADIENT BOOSTING . Consider a learning problem where training examples ( x , y ) are generated from a probability distribution P ( x , y ) where x ∈ RN is the space of measurements and y ∈ R is the space of labels . The goal of gradient boosting is to find an approximation F̂ ( x ) to a function F ( x ) that minimizes a specified loss function L ( F ( x ) , y ) : F̂ = arg min F L ( F ( x ) , y ) ( 1 ) Gradient boosting solves Eq . 1 by seeking functions ht ( x ) , so called weak learners , from a differentiable class H such that F̂ = ∑ t ht ( x ) . Starting from a constant function F0 and applying steepest descent we can find F̂ by iteratively updating with learning rate γ : Ft ( x ) = Ft−1 ( x ) − γ∇Ft−1L ( Ft−1 ( x ) , y ) ( 2 ) The weak learners ht are thus found by fitting the training data to the pseudo-residuals rt = ∇Ft−1L ( Ft−1 ( x ) , y ) hence in practice : Ft ( x ) = Ft−1 ( x ) − γht ( x ) ( 3 ) Next , consider a neural network F ( x ) with parameters w [ l ] comprised of L successive dense linear layers F [ l ] ( x ) = xTw [ l ] with non-linearities ψ : F ( x ) = F [ 0 ] ( ψ ( F [ 1 ] ( ... ψ ( F [ L ] ( x ) ) ) ) ) . Similarly , starting from parameter w0 and following steepest descent the network weights are updated at iteration t : wt = wt−1 − γ∇wt−1L ( F ( xt ) , yt ) ( 4 ) At each iteration , for a fixed layer l the gradient of the parameter w [ l ] ij can be calculated using the chain rule : ∂Lt ∂w [ l ] ij = ∂Lt ∂F [ l ] t ∂F [ l ] t ∂w [ l ] ij = ∂Lt ∂F [ l ] t ψ ( F [ l−1 ] t ) i = ∂Lt ∂F [ l ] t o [ l ] ti ( 5 ) where o [ l ] i is the output of the layer l − 1 at iteration t and position i and the gradients ∂Lt ∂F [ l ] t are calculated using the backpropagation algorithm . Alternatively , using the gradient boosting formula from Eq . 3 and plugging in Ft = F [ l ] ( xt ) = xTt w [ l ] for input xt = o [ l ] t we obtain , after dropping the layer indexes [ l ] for simplicity : oTt wt = o T t wt−1 + γh ( ot ) ( 6 ) We choose the weak learner h ( ot ) in this case to be a linear transformation as well h ( ot ) = oTt vt and fit it to the pseudo-residuals rt = ∂Lt ∂F [ l ] t by solving the associated regularized linear regression : v̂t = arg min v ( ‖rt − oTt v‖22 + α‖v‖22 ) , α > 0 ( 7 ) The primal form solution of Eq . 7 is v̂t = ( oTt ot + αI ) −1otrt and plugging it into Eq . 6 yields the following update rule : wt = wt−1 − γ ( oTt ot + αI ) −1otrt where I ∈ RN×N is the identity matrix . For a given input batch of size B , the update rule becomes : Wt = Wt−1 − γ ( OTt Ot + αI ) −1OTt Rt ( 8 ) where Wt ∈ Rn×m is the weight matrix for a given layer , Ot ∈ RB×n is the input matrix at the same layer while Rt ∈ RB×m is the pseudo-residual matrix ∇FtLt . In practice , the inverse of matrix OTt Ot can be hard to calculate for neural networks with large number of neurons per layer . In consequence , it is often more convenient to use the dual form formulation of the ridge regression solution of Eq . 7 that involves calculating the inverse of OtOTt instead . This is a well known formulation ( e.g . see Chapter 1 of ( Camps-Valls et al. , 2006 ) ) and leads to the analogous weight update rule : Wt = Wt−1 − γOTt ( OtOTt + αI ) −1Rt ( 9 ) The weight update formulas from Eq . 8 and 9 have the drawback that they change in magnitude with either the number of neurons in the neural network layer or batch size . This is an undesirable behavior that can lead to differences in scale between weights from layers of different sizes . In consequence , the terms OTt Ot and OtO T t are scaled accordingly . For large values of α the terms OTt Ot + αI and OtO T t + αI can be approximated by a constant diagonal matrix which leads to the classic SGD solution Wt ≈ Wt−1 − γOTt Rt . This shows that back-propagation with SGD is an extreme case of DGB that implicitly minimizes the magnitude of weight updates . 2.1 EXPERIMENTS . In the following we will compare the performance of DGB and SGD on a variety of reference data sets and explore the effect of the α parameter using simple dense layer neural networks . A summary of the data sets used is given in Table 1 : The first data set is the MNIST database of handwritten digits ( LeCun et al. , 1999 ) comprised of 60,000 28x28 grayscale images for training and 10,000 for testing . In the case of the Higgs data set ( Baldi et al. , 2014 ) we used a randomly chosen subset with 1 million samples out of 10 million for training , and 500 thousands samples for testing . This data comes from the field of high energy physics and was generated by Monte Carlo simulations . The third data set ( Reuters-21578 ) is comprised of 11,228 english language news articles from Reuters international news agency , labeled over 46 topics . The forth data set is the Human Resource ( HR ) data set ( Stacker IV , 2015 ) which was put together by IBM Inc. to predict employee attrition based on 47 features and was randomly split into 1,029 training samples and 441 test samples . The fifth data set is a regression problem for predicting a measure of air quality based on the outputs from an array of chemical sensors together with air temperature and humidity ( De Vito et al. , 2008 ) . All experiments were performed until test set convergence using a gradient update momentum of 0.9 and no decay . For simplicity , network architecture was fixed for each data set and all activation functions were rectified linear units ( ReLU ) ( Nair & Hinton , 2010 ) . The network weights were initialized using a uniform distribution according to ( He et al. , 2015 ) . The loss functions used were cross-entropy loss for classification problems and mean squared loss for regression problems . The performance metrics used were accuracy for classification tasks and root mean squared error ( RMSE ) for regression tasks . In addition , standard deviations were calculated and reported for all performance measures by repeating each experiment 10 times with different random seeds .

Summary: The paper proposes an a new idea of treating the back propagated gradients using chain rule as pseudo residual targets of a gradient boosting problem. Then the weight update is done by solving the boosting problem using a linear base learner. Furthermore, to reduce computational cost incurred by solving the boosting problem, an idea proposed is only keep the diagonal terms of the matrix inversion involved.