Deep Gradient Boosting -- Layer-wise Input Normalization of Neural Networks
Authors: Erhan Bilal
Venue:  (2020)
Keywords: sgd
dgb
boosting
batch norm
input norm

Abstract: Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due to its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well
on unseen data even when the neural network has millions of parameters. We hypothesize that in certain cases it is desirable to relax its intrinsic generalization properties and introduce an extension of SGD called deep gradient boosting (DGB). The key idea of DGB is that back-propagated gradients inferred using the chain rule can be viewed as pseudo-residual targets of a gradient boosting problem. Thus at each layer of a neural network the weight update is calculated by solving the corresponding boosting problem using a linear base learner. The resulting weight update formula can also be viewed as a normalization procedure of the data that arrives at each layer during the forward pass. When implemented as a separate input normalization layer (INN) the new architecture shows improved performance on image recognition tasks when compared to the same architecture without normalization layers. As opposed to batch normalization (BN), INN has no learnable parameters however it matches its performance on CIFAR10 and ImageNet classification tasks.

Under review as a conference paper at ICLR 2020
DEEP GRADIENT BOOSTING LAYER-WISE INPUT NORMALIZATION OF NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Stochastic gradient descent (SGD) has been the dominant optimization method for training deep neural networks due its many desirable properties. One of the more remarkable and least understood quality of SGD is that it generalizes relatively well on unseen data even when the neural network has millions pa- rameters. We hypothesize that in certain cases it is desirable to relax its intrinsic generalization properties and introduce an extension SGD called deep gradient boosting (DGB). The key idea of DGB is that back- back-propagated gradients inferred using the chain rule can be viewed as pseudo-residung targets of a gradient boost- ing problem. Thus each layer of neural network the weight update is calculated by solving the corresponding boosting problem using a linear base learner. The re- sulting weight update formula can also be viewed the data that arrives at each layer during the forward pass. When implemented as separate input normalization layer (INN) the new architecture shows improved performance on image recognition tasks when compared to the same architecture without normalization layers. As opposed to batch normalization (BN), INN has no learnable parameters however it matches its performance on CIFARIO and Im- ageNet classification tasks.
normalization procedure of
INTRODUCTION
Boosting, along side deep learning, has been very successful machine learning technique that consistently outperforms other methods on numerous data science challenges. In nutshell, the basic idea of boosting to sequentially combine many simple predictors such way that their combined performance is better than each individual predictor. Frequently, these so called weak learners are implemented as simple decision trees and one of the first successful embodiment of this idea was AdaBoost proposed by Freund & Schapire (197)). No too long after this, Breiman et al. (1998) and Friedman (2011) made the important observation that AdaBoost performs in fact a gradient descent in functional space and re-derived it as such. |Friedman (2001) went to define general statistical framework for training boosting-lite classifiers and regressors using arbitrary loss functions. Together with Mason al. (2000) they showed that boosting minimizes a loss function by iteratively choosing a weak learner that approximately points in the negative gradient direction of functional space.
Neural networks, in particular deep neural nets with many layers, are also trained using a form of gradient descent. Stochastic gradient descent (SGD) (Robbins & Monro 195T) has been the main optimization method for deep neural nets due to its many desirable properties like good general- ization error and ability to scale well with large data sets. At basic level, neural networks are composed of stacked linear layers with differentiable non-linearities in between. The output of the last layer then compared to target value using a differentiable loss function. Training such a model using SGD involves updating the network parameters in the direction of the negative gradient of the loss function. The crucial step of this algorithm is calculating the parameter gradients and this is efficiently done by the backpropagation algorithm (Rumelhart et al. 1988 Werbos 1974).
Backpropagation has many variations that try to achieve either faster convergence or better gener- alization through some form of regularization. However, despite superior training outcomes, celerated optimization methods such as Adam (Kingma Bal (2013), Adagrad (Duchi et al. 201T) or RMSprop (Graves 2013) have been found generalize poorly compared to stochastic gradient

---

Under review as a conference paper at ICLR 2020
descent Wilssnn al., 2017). Therefore even before using an explicit regularization method, like dropout (Srivastava et al. (0144) or batch normalization (loffe Szegedy) (013). SGD shows very good performance validation data sets when compared to other methods. The prevalent explana- tion for this empirical observation has been that SGD prefers "flat" over "sharp" minima, which in turn makes these states robust to perturbations. Despite its intuitive appeal, recent work by (Dinh) et al. 2017) cast doubt on this explanation.
This work introduces simple extension of SGD by combining backpropagation with gradient boost- ing. We propose that each iteration of the backpropagation algorithm can be reinterpreted by solving, each layer, a regularized linear regression problem where the independent variables are the layer inputs and the dependent variables are gradients at the output of cach layer, before non non-linearity is applied. We call this approach deep gradient boosting (DGB), since it is effectively a layer-wise boosting approach where the typical decision trees are replaced by linear regressors. Under this model, SGD naturally emerges as an extreme case where the network weights are highly regular- ized, in the L2 norm sense. We hypothesize that for some learning problems the regularization criteria doesn't 't need to be too strict. These could be cases where the data domain is more restricted or the learning task is posed
matrix decomposition or another coding problem. Based this idea we further introduce INN, a novel layer normalization method free of learnable parameters and show that it achieves competitive results on benchmark image recognition problems when compared to batch normalization (BN).
2 DEEP GRADIENT BOOSTING
Consider a learning problem where training examples (x. y) are generated from a probability dis- tribution P(r., y) where R.N is the space of measurements and y R is the space of labels. The goal of gradient boosting find an approximation (xx) function F(x)) that minimizes specified loss function L(F(r).
F= arg min mig L(F(x),
Gradient boosting solves Eq. bb seeking functions h, (2)),, so called weak learners, from differen- tiable class H. such that F= E. hz))). Starting from constant function Fo and applying steepest descent we can find f by iteratively updating with learning rate yy:
Fx(x) (x) VVr. (FFF-1()),)))
The weak learners h4 are thus found by fitting the training data the pseudo-residuals VF,,, (FF:- y) hence in practice:
=
(x)
Next, consider a neural network F(x) with parameters "I comprised of successive dense linear layers Flll (x) = xf w with non-linearities vh: F(x) = (..(()))))))))) Similarly, starting from parameter we and following steepest descent the network weights are updated at iter- ation t:
Wi w 7V -w (FF(t)) m.)
(4)
At each iteration, for fixed layer / the gradient of the parameter w!/ can be calculated using the chain rule:
OL, oL, aF!" ÖL,
w(rl = dL, m dn
(5)
ow R
OF Ow!y or
where of is the output of the layer / at iteration t and position and the gradients calculated using the backpropagation algorithm.
8L, are ar!e
Alternatively, using the gradient boosting formula from Eq. 3 and plugging in F. = F
=
of wlll for input म, = o," we obtain, after dropping the layer indexes [[] for simplicity:
of wt of w + (h(o))
(6)
2

---

Under review as a conference paper at ICLR 2020
We choose the weak learner h((o)) in this case to be a linear transformation well h(o;) of'0 i2 and fit it to the pseudo-residuail aF!
B by solving the associated regularized linear regression:
0 arg megmin((1r of 2112 all001)) 0
The primal form solution of Eq. is in = (o o + a ' our and plugging it into Eq. 6 yields the following update rule: W) = wi 7(0 or + al) O,T, where I E R RNN is the identity matrix.
For given input batch of size B, the update rule becomes:
W. W 10 +
where W, E IRxx is the weight matrix for a given layer, O. E RBxn is the input matrix at the same layer while Rg E Bx is the pseudo-residual matrix Vr, L. In practice, the inverse of matrix Of On can be hard to calculate for neural networks with large number of neurons per layer. In consequence, it is often more convenient use the dual form formulation of the ridge regression solution of Eq. 7 that involves calculating the inverse of 0,0, instead. This is a well known formulation (e.g. see Chapter of (Camps- Valls al., 2006)) and leads to the analogous weight update rule:
W₁ = W - -0, +al)-
The weight update formulas from Eq. 8 and 9 have the drawback that they change magnitude with either the number of neurons in the neural network layer or batch size. This is an undesirable behavior that can lead to differences in scale between weights from layers of different sizes. In consequence, the terms O O, and O,0, scaled accordingly.
For large values of a the terms O + al and O,O, + al can be approximated by a constant diagonal matrix which leads the classic SGD solution W2 W -00? R4. This shows that back-propagation with SGD is an extreme case of DGB that implicitly minimizes the magnitude of weight updates.
2.1 EXPERIMENTS
In the following we will compare the performance of DGB and SGD on variety of reference data sets and explore the effect of the a parameter using simple dense layer neural networks. summary of the data sets used is given in Table I
Data set
No samples
No features
Task
MNIST
70,000
784
Image classification
Higgs
1.5M
28
Higgs boson event classification
Reuters
11,228
1,000
Newswire topic classification
HR 1470
47
Employee attrition classification Air quality regression
Air 7576
Table 1: Data sets used in experiments.
The first data set is the MNIST database of handwritten digits (LeCun et al., 1999) comprised of 60,000 28x28 grayscale images for training and 10,000 for testing. In the case of the Higgs data set (Baldi et al. 2014) we used randomly chosen subset with million samples out of 10 million for training, and 500 thousands samples for testing. This data comes from the field of high energy physics and was generated by Monte Carlo simulations. The third data set (Reuters-22 -15r8)) is com- prised of 11,228 english language news articles from Reuters international news agency, labeled over 46 topics. The forth data set is the Human Resource (HR) data set Stackerr IV 2015) which was put together by IBM Inc. to predict employee attrition based on 47 features and was randomly split into 1,029 training samples and 441 test samples. The fifth data set is regression problem for predicting a measure air quality based on the outputs from an array of chemical sensors together with air temperature and humidity (De Vito al.) 2008)_
All experiments were performed until test set convergence using a gradient update momentum of 0.9 and decay. For simplicity, network architecture was fixed for each data set and all activation
3

---

Under review as a conference paper at ICLR 2020
functions were rectified linear units (ReLU) (Narr & Hinton, 2010). The network weights were initialized using a uniform distribution according to (He et al., 2011)). The loss functions used cross-entropy loss for classification problems and mean squared loss for regression problems. The performance metrics used were accuracy for classification tasks and mot mean squared error (RMSH) for regression tasks. In addition, standard deviations were calculated and reported for all performance measures by repeating each experiment 10 times with different random seeds.
2.1.1 MNIST
For this experiment found that results relatively invariant to network architecture passed certain complexity, in consequence we chose a dense neural network with three hidden layers each with 500 ReLU nodes. All models fixed 0.1 learning rate and achieved 100% accuracy on the training set. The two DGB variants were labeled DGB()) if it used the left pseudo-inverse from Eq. 8 and DGB(r) corresponding the right pseudo-inverse from Eg.
trained using 100 sample batch size for 25 epochs using
Model
Learning rate
Performance
SGD
0.1[0-25]
98.41% 0.12
DGB(r)
1.0
0.1[0-25]
98.59% = 0.04
DGB(r)
0.1
0.1[0-25]
98.35% 0.10
DGB(r)
0.05
0.1[0-25]
97.99% 0.11
DGB(r)
0.01
0.1[0-25]
96.25% 0.29
DGB()
1.0
0.1[0-25]
08.45%
DGB()
0.1
0.1[0-25]
98.25% 0.12
DGB()
0.05
0.1[0-25]
98.03% + 0.10
DGB())
0.01
0.1[0-25]
96.86% + 0.13
Table 2: Performance the MNIST data set measured as accuracy.
Both left and right variants achieve increasingly better test performance with larger a values culminating with 98.59% average accuracy for the DGB(() model with a = 1.0. This marginally better than 98.41% average accuracy for SGD but more importantly this result shows how increasing the regularization parameter alpha successfully improves test set performance for an image classifi- cation problem prone to over-fitting.
Overall DGB()) and DGB(r) show similar performance with the drawback that for relatively small batch sizes DGB(I) considerably slower. This is because it needs to calculate the inverse of 500x500 matrix as opposed to calculating the inverse of 100x matrix, as is the case for DGB(r). When training on a single V-100 Nvidia GPU one SGD iteration took on average 0.0166 see, one DGB(r) iteration took 0.0188 see while in contrast DGB(1) took 1390 sec. In consequence, for the rest of experiments we will just use DGB(r) and simply label DGB.
2.1.2 HIGGS
Higgs data set was created using Monte Carlo simulations of particle collisions and has 11 mil samples with 28 features. This a binary classification problem designed distinguish between signal process which produces Higgs bosons and a background process which does not. For this exercise we kept only 1 mil samples for training and 500k for testing. All 28 features were normalized to be between 0 and 1. We used architecture composed of two hidden linear layers with 500 ReLU activations each that was trained using a small batch size of 30 samples over 50 epochs. The learning rate kept constant at 0.01 for SGD and DGB for high values and was reduced to 0.001 for a 5 0.1.
This data set poses considerable challenge with its relatively few features and large sample size. Performance on the training set relatively poor and doesn't 't approach 100% accuracy like in the other experiments. The best test set accuracy is obtained for smaller a value ATable/A]
4

---

Under review as a conference paper at ICLR 2020
2.1.3 REUTERS
Reuters is a text categorization data set comprised or articles organized into 46 news topics. was randomly divided into 8982 training and 2246 test samples that were processed using binary tokenizer for the top 1,000 most common words. This data was then fed into a dense neural network with two hidden layers each with 500 ReL.U nodes and trained for 100 epochs with learning rate fixed at 0.01.
Table A2 shows the results for experiments run using the SGD optimizer and DGB with values 1.0, 0.1 and 0.01. This is a relatively small data set with a large number of features that is usually more successfully addressed using more advanced approaches like recurrent neural networks or convolutional neural nets. In this case, the results are close for all experiments and only marginally better at 78.05% mean accuracy for DGB with a 0.1.
2.1.4 HR
Similar to the other data sets, min-max normalization was used on all 47 features of the Human re- source attrition data (HR). This is a binary classification task designed to predict employee attrition. To this end we employed a dense neural network with one hidden layer composed of 100 ReLU nodes. As before, we used a batch size of 30 samples for speed. All models were trained for 500 epochs with a fixed learning rate 0.001.
With only 1029 training samples and 441 test samples the error bars (i.e. standard deviation) are too big to make a claim that DGB outperforms SGD in this case (see Table A3).
2.5.5 AIR
The Air quality data set contains 9358 instances of hourly averaged reads from chemical sensors de- signed to measure air quality. After removing samples with missing values we divided the remaining data equally into 3788 training and 3788 test samples, and all the features were max normalized to be between 0 and 1. Unlike the previous experiments this of predicting concentrations of benzene, a common pollutant associated with road traffic. For this we employed neural network with two hidden layers with 100 ReLU activations each, trained for 1000 epochs using a fixed learning rate of 0.1 and a batch size of 30 samples.
regression problem with the goal
Just as the previous experiments, DGB with larger values were closer in performance SGD (Table A4). In this case, relaxing the regularization parameter led to a gradual increase in test set performance with almost half the root mean squared error for DGB with 0.001 when compared to SGD.
3 FAST APPROXIMATION OF DGB
We showed in the previous section that deep gradient boosting outperforms regular stochastic gra- dient descent in some cases. However, this comes at a cost because DGB gradient updates involve calculating the inverse of a matrix at each step and for each layer. The computational cost for matrix inversion is roughly (n²²) and this can be significant even for small batch sizes as the ones used in the previous experiments. A straight forward way of speeding matrix inversion is keep just the diagonal terms. This is equivalent of making the simplified assumption that for layer input matrix O E Bxn the rows are independent (i.e. Σn OukOjk for i + j) in the case of DGB(r) or the columns are independent (i.e. Σx Ont0k for + j) in the case of DGB()).
For layer with n input dimensions and sample hatch of size B. plugging the diagonal approx- 9 the gradient updates become:
imations of matrices O,O and Of O, into Eq. 8 &
W. = Wi1 7O R
(10)
where
Ojj
6.j
(11)
2j=1 2 +
5

---

Under review as a conference paper at ICLR 2020
in the case of DGB(r) and
Oij B
(12)
2i= 01 + a
B
in the case of DGB(I).
In the above notation Ö is the normalized version of layer input matrix O with elements diy. In the following paragraphs, FDGB(r) will refer to fast DGB(r) from Eq. I1 and FDGB()) will refer to fast DGB(I) from Eq. 12
3.1 CONVOLUTIONAL NEURAL NETWORKS
Convolutional neural networks (CNN) are class of neural networks particularly suited to analyzing visual imagery. Typically the input at each CNN layer is tensor of size (B x x w h) where B is the batch size, is the number of feature maps while and h are the width and height of the map. Instead of using a dense network with different weight connecting each point the (c x w x hi) map, CNNs work by dividing the feature map into usually overlapping sections by scanning the input with a kernel a size much smaller than (w x f),, usually (3x3). In this way CNNs effectively increase the number of input samples and simultaneously decrease the number of weights that need to be learned. This process can be viewed as a form of regularization and it has proven to greatly outperform regular dense networks that usually prone to over-fitting on image recognition problems.
For given CNN layer, assume we have an input of size (B X Co x wo x ho) and an output of size (B x c1 X wi x h1) that was generated with kernel of size (p X q). In this case, the equivalent DGB formulation based on Eq. & would have an input matrix O2 of size (B W1 x h1, co x x Q). would be extremely costly to first unfold the input as described and then calculate the matrix inverse of either O O, or O,O In consequence, it is advisable to use the fast versions of DGB however, one would still have to unfold the input matrix which is still a costly operation.
The key observation for extending FDGB(I) to convolutional neural networks is that for small kernels with a small step size each position in the kernel matrix will "visit" roughly the same pixels in feature map. Hence, calculating the raw second moment of each feature needs to be done only once per map. After resizing the input tensor O2 to (B wo x ho, co) the new normalized input Ö becomes:
Oij Bx Xhe
(13)
of
+ a
Tx X hp
It is not unreasonable assume small kernel size given that most modern convolutional network architectures use kernels of size (3,3) for image recognition.
4 INPUT NORMALIZATION
It is clear from Eq. 10 and previous paragraphs that DGB's gradient update formulation can also be viewed as input normalization defined N(X) = x(x al)-1 N(X) := (XX* X + a followed by a regular SGD update. Hence this process can also be implemented as separate normalization layer, similar to how other layer normalization work like batch norm (BN) (Toffe & Szegedy 2015), layer norm (LN) (Ba et al., 2016), instance norm (IN) (Ulyanov et al.) 2016) and others.
The advantages of these normalization methods are well documented: they accelerate descent by enabling higher learning rates without the risk of divergence which in turn may help avoid local sharp minima (Bjorck et al. 2018). make training more resilient to parameter (Ba al. 2016), and they stabilize layer output variance which prevents gradient exposition in very deep networks (He et al.) 2016a). Interestingly, for formulations that operate across samples like batch norm they also have a regularization effect and virtually eliminate the need for other regularization methods like dropout (Srivastava al. 2014]. This is because training examples seen in conjunction with other examples in the mini- batch, hence the layer outputs are no longer deterministic. This added noise found to be beneficial the generalization of the network.
input data scale
Batch norm, layer norm and instance norm perform a z-score standardization of features/inputs by subtracting the mean and dividing by standard deviation which is typically applied to the output

---

Under review as a conference paper at ICLR 2020
of layer, before the activation function. This procedure runs the risk of keeping the output of layer inside the linear regime of activation functions that are symmetric around zero. order to recover the expressive power of the network, after normalization is performed two new parameters are introduced that scale and shift the output. These parameters are in turn learned during training like the rest of the network weights. Interestingly, the scale and shift parameters help convergence even for neural networks with non-symmetric activation functions (c.g. ReLU).
In contrast to these methods, the normalization procedure described here is applied the layer input, it is not scale invariant and has only one parameter a which is predefined before training, not learned. In addition, "normalization" in our case doesn't 't refer the process of making the data look more like a standard normal distribution but instead refers to an algebraic transformation of the data that improves its condition number.
4.1 INPUT NORMALIZATION MULTIPLE LINEAR REGRESSION
In the following we will study the effect of input normalization as defined in the previous paragraph to the simple problem of ordinary least squares solved by gradient descent. Consider a multiple linear regression model arg min Xw)² (y Xw) where X E IR is input matrix, y E R is the output vector and w E R. is the vector of parameters. It is well known that the rate of convergence of gradient descent in this case depends the condition number of the Hessian H = X X where high values lead to slow convergence. X = UVV" is the singular value decomposition of input matrix X, where U and V are orthogonal matrices and E is diagonal matrix, then the condition number of H = x = VΣ is:
o mar
<(H)=
(14)
where o mar and min are the largest and respectively smallest singular values of X.
Let N(X) X( + al)- and arg my ((X)x)) N(X)w) be the resulting linear regression problem after applying input normalization. The new Hessian matrix is H = (X"X- + a X X (X"X al)-1. After plugging in X = UEVT and some linear algebra manipulation the new condition number becomes:
max
(o +a)²
2 + 0
2
<(H))
k(H)
(((H)
(15)
(o + a)²
mar + a
This result shows that input normalization leads smaller condition number of the Hessian matrix in the case of multiple linear regression. The same result can be obtained when using the alternative
input normalization N (X) (XXT + al)- -Ix -I (see Prop. A.1 & A.2 in the appendix).
4.2 FORMULATION
In practice, we will define Input Normalization (INN) using the faster diagonal approximations from Eq. [1 & 12 In addition, similar to other normalization layers will also mean center the input data. For a single layer network, centering further improves conditioning of the Hessian of the loss [LeCun al. 1991), it brings the input data near the critical point of most activation functions thus removing the bias towards certain neurons and, it has a regularization effect by shifting the data in nondeterministic way during training based on the content of the mini-batch.
For an input matrix X E B.aa with elements Tij left input normalization is labeled INN(I) and defined as
N(ray) Zij EE
Zij
(16)
5 1== 2. + while right input normalization labeled INN(r) and defined as
B
1 12 Σ
Fij
N(wx)) = 1 Σ +
(17)
In the case of INN()) formulation the batch statisties m. Σe Tay and m2 Σe= 2 can cause non-deterministic predictions at test time. Instead, running exponential averages are calculated during training for both my and m2. These new values then used test time.

---

Under review as a conference paper at ICLR 2020
4.3 EXPERIMENTS
We tested the implicit (i.c. FDGB) and explicit (i.e. INN) imput normalization models the CI- FARI0 data set (Krizhevsky et al. 2009) and compared them to Batch Normalization (BN). We then validated the best performing models Ruusakkvvky et al. 2015). In both cases we used versions of the VGG architecture first introduced by Simonyan Zisserman (2014) and then modified them to accommodate the different models as described in Appendix A.
the ImageNet object recognition data set (ILSVRC2015)
4.3.1 CIFARIO
The II data set consists of 60000 images organized into 10 classes. There are 50000 train- ing and 10000 test images. According to Table [ the original VGGI model achieves a baseline performance of 89.50% accuracy this problem. Adding batch normalization layers to this archi- tecture significantly improves the performance to 91.18% accuracy. The INN()) model outperforms both FDGB()) and VGGI1 while matching the performance of batch normalized VGG11 architec- ture (VGG11 _BN). In the next section we validate the performance INN(!) on a larger image classification data set using a deeper convolutional network.
Model
Learning rate
Performance
VGG11
0.1[0-250), 0.01[250-350]
89.50% + 0.20
VGG11_BA
0.1[0-250), 0.01[250-350)
91.18% + 0.16
FDGB()
1.0
0.1[0-250), 0.01[250-350]
87.91% + 0.30
FDGB()
3.0
0.1[0-250), 0.01[250-350]
89.10% + 0.25
INN())
0.5
0.1[0-250). (201(250-300
91.14% + 0.14
INN())
1.0
0.1[0-250), 0.01[250-350]
91.34% + 0.18
Table 3: Performance on the CIFAR10 data set measured as accuracy.
4.3.2 IMAGENET
The ImageNet data set is part of the Large Scale Image Recognition Challenge 2015 (Russakovsky et al. 2015) and consists of 1,281,167 training and 50,000 validation images organized into 1,000) categories. For this experiment
compared the performance of VGG19 and ResNet101 with batch and respectively ResNet101_Be _BN) to the equivalent architecture with
normalization (VGGI9_BN)) input normalization layers (VGG19 INN()) and respectively ResNet101 (NN())) (See Appendix A for more details). In addition, for VGG19 VGG19_BN(b) we disabled the learnable affine parameters which reduces BN basic standardization operation.
Model
a
Top-5 error
Top-I I error
VGGI9 BN(a)
6.79%
23.64%
VGG19 BN(b)
12.58%
33.96%
VGG19 (NN((
1.0
6.59%
23.27%
ResNet101 _N
5.29%
20.69%
ResNer101 (NN(()
0.5
5.74%
21.63%
Table 4: Performance on the ImageNet data set measured as top-1 and top-5 errors.
Similar to CIFAR10 the performance of VGGJINN()) is very close to that of VGG19_BNG (Na() although during early epochs VGG (IGG)NN(() shows better convergence (Figure [T]. For the BN model without affine parameters (VGG19 (N()) performance drops dramatically when trained using the same learning rate schedule showing that extra parameters are essential for the performance of VGG19 with batch normalization (Table 4). When employing architecture with residual connections ResNet101, replacing batch normalization layers with INN operations leads to a slight decrease in performance.
8

---

Under review as a conference paper at ICLR 2020
80
80
Top-6 emor
60
Top-1
20
20
20
100
20
100
Epochs
Epochs
(a) VGG19
(b) VGG19
BN
BN INN
70
INN
error
Top-5 error
40
20
10-
20
20
120
Epochs
Epochs
(c) ResNet101
(d) ResNet101
Figure 1: Top-1 (left) and top-5 (right) validation set errors of VGG_BN(a) and ResNet101.BB labeled as BN and INN(I) labeled as INN over 120 epochs.
CONCLUSIONS
This work introduces Deep Gradient Boosting (DGB), a simple extension of Stochastic Gradient Descent (SGD) that allows for finer control over the intrinsic generalization properties of SGD. We empirically show how DGB can outperform SGD in certain cases among a variety classification and regression tasks. We then propose faster approximation of DGB and extend it to convolutional layers (FDGB). Finally, we reinterpret DGB as a layer-wise algebraic manipulation of the input data and implement it separate normalization layer (INN). We then test INN image classification tasks where its performance proves to be on par with batch normalization without the need for additional parameters.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450. 2016.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles high-energy physics with deep learning. Nature communications, 5:4308, 2014.
Johan Bjorck, Carla Gomes, Bart Selman, and Kilian Q. Weinberger. Understanding batch normal- ization. 2018.
Leo Breiman et al. Arcing classifier (with discussion and a rejoinder by the author). The annals of statistics, 26(3):801-849, 1998.
Gustavo Camps- Valls, Josee L Rojo-Alvarez, and Martin Martinez-Ramon. Kernel methods in bioengineering, signal and image processing. Idea Group Pub., 2006.

---

Under review as a conference paper at ICLR 2020
Saverio De Vito, Ettore Massera, M Piga, L Martinotto, and G Di Francia. On field calibration of an electronic nose for benzene estimation in an urban pollution monitoring scenario. Sensors and Actuators B: Chemical, 129(2):50--557, 2008. Online at attps::/neetton.ai/ att.s//tttooo.aasssssss// archives/ais ....p
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1019-1028. JMLR. org. 2017.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 120ul):2121-2159 2011.
Yoav Freund and Robert E Schapire. A -eeision-theoreticc generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1): 19-139, 1997.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232,22 2001.
Alex Graves. Generating sequences with recurrent neural networks. arXiv: 0850,000000. 2013.
arXiv preprint
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing
human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi: 10.1109/icc.22015.222 URL phttp://de doi.
Exg/10.1109/I00 201...01..222
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), Jun 2016a. doi: 10.1 10.1109//pr
doi, http:////oo.oo..ooo/1111////////
2016. 90
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016b.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 456, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. The International Conference on Learning Representations (ICLR),, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Ido Kanter, and Sara A Solla. Second order properties of error surfaces: Learning time and generalization. In Advances in neural information processing systems, pp. 918-924, 1991.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. The mnist database of handwritten digits. 1999. Online at wthttp://yaann. lecun com/ wwweexmixmma..
Llew Mason, Jonathan Baxter, Peter L Bartlett, and Marcus R Frean. Boosting algorithms as gradient descent. Advances in neural information processing systems, pp. 512-518, 2000.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICMI.-10), pp. 807-814, 2010.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. In-place activated batchnorm for memory- memory-optimizee training of dnns. In Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 5639-5647, 2018.
10

---

Under review as a conference paper at ICLR 2020
David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, al. Learning representations by back-propagating errors. Cognitive modeling. 5(3):1, 1988.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):111-255 2015. doi: 10.1007/s1 126000611222----1111---6333
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409: 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, (5(1):1929-19) 1958, 2014.
McKinley Stacker IV. Hr employee attrition and performance. 2015. Online at https:
//neuton. al assets /ttpa//ssttttttt..seee/ttttteeee
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization, 2016.
Paul Werbos, Beyond regression: New tools for prediction and analysis the behavioral sciences.
Ph. D. dissertation, Harvard University, 1974.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4148-4158, 2017.
A APPENDIX
Model
Learning rate
Performance
SGD
0.01[0-50]
74.10% + 0.12
DGB
1.0
0.01[0-50]
73.53% 0.42
DGB
0.1
0.001[0-50]
74.60% 0.19
DGB
0.01
0.001[0-50]
74.99 % = 0.09
DGB
0.001
0.001[0-50]
73.98% 1.22
Table Al: Performance on the Higgs data set measured as accuracy.
Model
a
Learning rate
Performance
SGD
0.01[0-100]
77.54% + 0.29
DGB
1.0
0.01[0-100]
77.31% 0..27
DGB
0.1
0.01[0-100]
78.05% + 0..23
DGB
0.01
0.01[0-100]
77.39% = 0.24
Table A2: Performance the Reuters data set measured as accuracy.
Model
a
Learning rate
Performance
SGD
0.001[0-500]
86.51% + 0.44
DGB
1.0
0.001[0-500]
86.87% + 0.21
DGB
0.1
0.001[0-500]
86.92% + 0.48
DGB
0.01
0.001[0-500]
85.24% = 0.59
Table A3: Performance the HR data set measured accuracy.

---

Under review as a conference paper at ICLR 2020
Model
a
Learning rate
Performance
SGD
0.1[0-1000]
0.0020 + 0.00017
DGB
1.0
0.1[0-1000]
0.0020 + 0.00020
DGB
0.1
0.1[0-1000]
0.0013 + 0.00007
DGB
0.01
0.1[0-1000]
0.0012 + 0.00028
DGB
0.001
0.1[0-1000]
0.0010 = 0.00015
Table A4: Performance on the Air data measured root mean squared error.
Proposition A.1. If a is a singular value of matrix X then (x + +
has singular values of the form (o=+a)=
o²
Let X = -UVV be the singular value decomposition of X then:
(V " +al)-1 (x x -xx--xxXXX(( xxx"XXXXXXx xxxxx++a)" +al)
= =VVEV" (V -aVV?)-
=V(>² VV +al)+-1
=V (2') 4al)
Proposition A.2. If a is a singular value of matrix X then X (XX" aal- X2 has singular o²
values of the form ta"
Let X U be the singular value decomposition of X then:
("XX"" + 2x VEU QUE'U + 2
VEU
=VEU*U(E? -
= =VE(E² -
A.1 CIFAR10
For this experiment we used a version of the VGG1I network introduced by Simonyan & Zisserman (2014) that has 8 convolutional layers followed by a linear layer with 512 ReLU nodes, dropout layer with probability 0.5 and then a final softmax layer for assigning the classification probabilities. second version of this architecture (VGG11 1 (_BN) has batch normalization applied at the output of each convolutional layer, before the ReLU activation recommended by Toffe & Szegedy (2015) We modified this architecture by first removing all the batch normalization and dropout layers. We then either replaced all convolutional and linear layers with ones that implement the fast version of DGB for the FDGB() architecture or added INN(I) layers in front each of the original convo- lutional and linear layers. Both (1GB() and INN(1) models implement input normalization based on the left pseudo-inverse (see Eq. 12 & 16 in order to take advantage its regularization effect. All weights were initialized according stochastic gradient descent with momentum 0.9 and batch size 128. For the FDGB(I) model the gra- dients were calculated according to Eq. 12 for linear and 13 for convolutional layers. Training was started with learning rate 0.1 and reduced to 0.01 after 250 epochs and continued for 350 epochs. All experiments were repeated 10 times with different random seeds and performance was reported on the validation set as mean accuracy + standard deviation.
Simonyyn & Zisserman (2014) and were trained using
A.2 IMAGENET
Similar to the CIFAR10 data set we based these experiments on the larger VGG19 architecture of Simonyan & Zisserman (2014). The VGG19 network has 16 convolutional layers interspersed with max-pool operations, tollowed by two linear layers with 4096 ReLU nodes and a final softmax layer with 1000 outputs. The original VGG19 model uses dropout regularization after each rectified linear
12

---

Under review as a conference paper at ICLR 2020
layer. A second version of this architecture (VGG19_BB) BNN has batch normalization applied at the out- put each convolutional layer, before the ReLU activation. We tested VGGI9_BB with and without affine parameters (VGG19 BN(a) and respectively VGG19 BN(b)). After removing all the dropout and batch normalization layers we created the INN()) version by adding input normalizations in front of each remaining layer.
In order to explore the performance of INN on network architectures with residual connections we employed ResNet101 proposed by He al. (2016b). The original network uses batch normaliza- tion between convolutional layers by default (ResNet 101Nett01_BB) The INN version of this network (ResNet101 _NN())) was created by first removing all BN layers, then adding INN operations in front of each convolutional and dense layers, with the exception of down-sample convolutions.
Stochastic gradient descent was used minimize the cross entropy loss of each network over 120 epochs, using an initial learning rate of 0.05, 0.9 momentum, batch size of 128 images, and 0.0001 weight decay. Every 30 epochs the learning rate was decreased by factor of 10. All networks were evaluated by computing the top-1 and top-5 validation errors on held out data set using 10 crops per image.
13


Reviews:
[{"review_id": "rkgI6MHnKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper propose an extention method of SGD, deep gradient boosting (DGB), which views the back-propagation procedure as a pseudo-residual targets of a gradient boosting problem. To apply DGB to the real CNNs, DGB is simplified to a input normalization layer, conditioned on the assumption that the convolution kernels should be small. After applying the input normalization layer to CNNs, the model could achieve comparable performance to the model with BN on CIFAR-10 and ImageNet recognition.\n\nThere are several concerns influencing my rating:\n* I cannot catch the advantages of this input normalization layer compared to BN. For example, could this input normalization layer help to address the problem that BN performs bad when batch size is small? The authors mention that this layer does not have additional parameters. But as I know, the parameter size of BN is small, which downgrades the significance of the proposed method. \n\n* In the CIFAR-10 and ImageNet experiments, only the VGG model is adopted, which obviously limits the application scope. Could the proposed method work well on ResNet, DenseNet or other more recent deep architectures?\n\n* In the DGB experiments, the improvements of DGB compared with SGD in four datasets all seem marginal, in which DGB is slower than SGD. \n\nOverall, I recognize the exploration of this method. But the advantages of DGB compared to SGD seem marginal.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkeAqkjTFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new gradient update rule for neural net weights, which is a result of relaxing the usual SGD to a regularized linear regression between the descent direction and the original gradient. The method is interpreted as some kind of input normalization similar to Batch Norm later in the paper.\n\nAlthough the method seems to work and the idea is interesting, I’m not entirely convinced by the experimental evidence. There are some technical concerns:\n\n- In section 2.1, I’m only familiar with the MNIST dataset. Although the proposed method achieves slightly better performance than vanilla SGD, it’s pretty weird the accuracy is only 98.xx% as a very simple CNN could easily achieve >99% accuracy on MNIST. This might be due to the simple network architecture used. A more modern architecture would be preferable in those experiments. Also the improvement on CIFAR seems to close to tell.\n\n- The writing seems to need some improvement. It’s sometimes hard to follow the text, and some result tables (Table A1,A2,A3) are in the appendix not the main text. It might be better to reorganize the text and present everything about the method (fast approximation, CNN, interpretation as normalization) together.\n\n- The authors name the proposed method “deep gradient boosting”, but I’m not entirely sure how it is related to gradient boosting. Eq. 3 concerns sum of weak predictors, but how is this related to Eq. 4? The neural net F(x) doesn’t seem to be exactly a sum of weak learners. I don’t quite follow Eq. 5 – There seems to be missing something.\n\nTypos:\n- Eq. 5, d L_t instead of d L_T, and should there be a d \\phi / d w_ij term?\n- Eq. 7, v_t => v on the right-hand side.\n- Eq. 12, should be \\sum_i=1^B\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eGyKJTKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an a new idea of treating the back propagated gradients using chain rule as pseudo residual targets of a gradient boosting problem. Then the weight update is done by solving the boosting problem using a linear base learner. Furthermore, to reduce computational cost incurred by solving the boosting problem, an idea proposed is only keep the diagonal terms of the matrix inversion involved. \n\nIt is an interesting idea to look at gradients different and update weight accordingly. The experimental results shows some improvements. \n\nThe paper is written in a way that it looks like two papers and connected with the same idea DGB. It is a bit confusing during the first pass of reading. \n\nIn all experiments, can you also report running times? how much is the overhead? and  how much reduction in training time if only using diagonal terms? \n\nDoes only using diagonal terms hurt model performance?\n\nFor 2.1.1 MNIST, why is alpha the larger the better while the others are not this case?\n\nFor 2.1.5 AIR, why spit training and test equality?\n\nFor 4.3 why choose heavy model VGG? not ResNet? \n\nIn Figure 1, is the spike of BN normal?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a neat idea that an SGD update can be written as a solution of the linear least squares problem with a given backpropagated output; this is generalized to a larger batch size, giving a sort of "block" gradient-type update. Some notes that the columns of $O_t$ have to be scaled are made, but not clear why. The paper then goes into the experiments, and then gets back to the fast approximation of DGB. It really looks like bad organization of the paper, which was noted. 
The reviewers agree that the actual computational improvements are marginal, and all recommend rejection. As a recommendation, I would suggest to restructure the paper for a more coherent view, and also the improvements in Top-1 are not very stimulating. The general view is interesting, but it is not clear what insight it brings.
Decision: Reject