Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities
Authors: Tommi Vatanen, Tapani Raiko, Harri Valpola, Yann LeCun
ICLR 2013

Abstract: Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scale of the outputs of each hidden neuron, and secondly by analyzing the connections to second order optimization methods. We show that the transformations make a simple stochastic gradient behave closer to second-order optimization methods and thus speed up learning. This is shown both in theory and with experiments. The experiments on the third transformation show that while it further increases the speed of learning, it can also hurt performance by converging to a worse local optimum, where both the inputs and outputs of many hidden neurons are close to zero.

Pushing Stochastic Gradient towards Second-Order
Methods – Backpropagation Learning with
Transformations in Nonlinearities
Tommi Vatanen, Tapani Raiko, Harri Valpola
Department of Information and Computer Science
Aalto University School of Science
P.O.Box 15400, FI-00076, Aalto, Espoo, Finland
first.last@aalto.fi
Yann LeCun
New York University
715 Broadway, New York, NY 10003, USA
firstname@cs.nyu.edu
Abstract
Recently, we proposed to transform the outputs of each hidden neuron in a multi-
layer perceptron network to have zero output and zero slope on average, and use
separate shortcut connections to model the linear dependencies instead. We con-
tinue the work by ﬁrstly introducing a third transformation to normalize the scale
of the outputs of each hidden neuron, and secondly by analyzing the connections
to second order optimization methods. We show that the transformations make
a simple stochastic gradient behave closer to second-order optimization methods
and thus speed up learning. This is shown both in theory and with experiments.
The experiments on the third transformation show that while it further increases
the speed of learning, it can also hurt performance by converging to a worse local
optimum, where both the inputs and outputs of many hidden neurons are close to
zero.
1 Introduction
Learning deep neural networks has become a popular topic since the invention of unsupervised
pretraining [4]. Some later works have returned to traditional back-propagation learning in deep
models and noticed that it can also provide impressive results [6] given either a sophisticated learning
algorithm [9] or simply enough computational power [2]. In this work we study back-propagation
learning in deep networks with up to ﬁve hidden layers, continuing on our earlier results in [10].
In learning multi-layer perceptron (MLP) networks by back-propagation, there are known transfor-
mations that speed up learning [8, 11, 12]. For instance, inputs are recommended to be centered to
zero mean (or even whitened), and nonlinear functions are proposed to have a range from -1 to 1
rather than 0 to 1 [8]. Schraudolph [12, 11] proposed centering all factors in the gradient to have
zero mean, and further adding linear shortcut connections that bypass the nonlinear layer. Gradient
factor centering changes the gradient as if the nonlinear activation functions had zero mean and zero
slope on average. As such, it does not change the model itself. It is assumed that the discrepancy
between the model and the gradient is not an issue, since the errors will be easily compensated
by the linear shortcut connections in the proceeding updates. Gradient factor centering leads to a
signiﬁcant speed-up in learning.
1
arXiv:1301.3476v3  [cs.LG]  11 Mar 2013
In this paper, we transform the nonlinear activation functions in the hidden neurons such that they
have on average 1) zero mean, 2) zero slope, and 3) unit variance. Our earlier results in [10] in-
cluded the ﬁrst two transformations and here we introduce the third one. cdsaasd We explain the
usefulness of these transformations by studying the Fisher information matrix and the Hessian, e.g.
by measuring the angle between the traditional gradient and a second order update direction with
and without the transformations.
It is well known that second-order optimization methods such as the natural gradient [1] or New-
ton’s method decrease the number of required iterations compared to the basic gradient descent,
but they cannot be easily used with high-dimensional models due to heavy computations with large
matrices. In practice, it is possible to use a diagonal or block-diagonal approximation [7] of the
Fisher information matrix or the Hessian. Gradient descent can be seen as an approximation of the
second-order methods, where the matrix is approximated by a scalar constant times a unit matrix.
Our transformations aim at making the Fisher information matrix as close to such matrix as possible,
thus diminishing the difference between ﬁrst and second order methods. Matlab code for replicating
the experiments in this paper is available at
https://github.com/tvatanen/ltmlp-neuralnet
2 Proposed Transformations
Let us study a MLP-network with a single hidden layer and shortcut mapping, that is, the output
column vectors yt for each sample tare modeled as a function of the input column vectors xt with
yt = Af (Bxt) + Cxt + ϵt, (1)
where f is a nonlinearity (such as tanh) applied to each component of the argument vector sepa-
rately, A, B, and C are the weight matrices, and ϵt is the noise which is assumed to be zero mean
and Gaussian, that is, p(ϵit) = N
(
ϵit; 0,σ2
i
)
. In order to avoid separate bias vectors that complicate
formulas, the input vectors are assumed to have been supplemented with an additional component
that is always one.
Let us supplement the tanh nonlinearity with auxiliary scalar variables αi, βi, and γi for each
nonlinearity fi. They are updated before each gradient evaluation in order to help learning of the
other parameters A, B , and C. We deﬁne
fi(bixt) = γi[tanh(bixt) + αibixt + βi] , (2)
where bi is the ith row vector of matrix B. We will ensure that
T∑
t=1
fi(bixt) = 0 (3)
T∑
t=1
f′
i(bixt) = 0 (4)
[ T∑
t=1
fi(bixt)2
T
][ T∑
t=1
f′
i(bixt)2
T
]
= 1 (5)
by setting αi, βi, and γi to
αi = −1
T
T∑
t=1
tanh′(bixt) (6)
βi = −1
T
T∑
t=1
[tanh(bixt) + αibixt] (7)
γi =
{1
T
T∑
t=1
[tanh(bixt) + αibixt + βi]2
}1/4{1
T
T∑
t=1
[
tanh′(bixt) + αi
]2 }1/4
. (8)
2
One way to motivate the ﬁrst two transformations in Equations (3) and (4), is to study the expected
output yt and its dependency of the input xt:
1
T
∑
t
yt = A 1
T
∑
t
f(Bxt) + C 1
T
∑
t
xt (9)
1
T
∑
t
∂yt
∂xt
= A
[
1
T
∑
t
f′(Bxt)
]
BT + C. (10)
We note that by making nonlinear activations f(·) zero mean in Eq. (3), we disallow the nonlinear
mapping Af (B·) to affect the expected output yt, that is, to compete with the bias term. Similarly,
by making the nonlinear activations f(·) zero slope in Eq. (4), we disallow the nonlinear mapping
Af (B·) to affect the expected dependency of the input, that is, to compete with the linear mapping
C. In traditional neural networks, the linear dependencies (expected∂yt/∂xt) are modeled by many
competing paths from an input to an output (e.g. via each hidden unit), whereas our architecture
gathers the linear dependencies to be modeled only by C. We argue that less competition between
parts of the model will speed up learning. Another explanation for choosing these transformations is
that they make the nondiagonal parts of the Fisher information matrix closer to zero (see Section 3).
The goal of Equation (5) is to normalize both the output signals (similarly as data is often normalized
as a preprocessing step – see,e.g., [8]) and the slopes of the output signals of each hidden unit at
the same time. This is motivated by observing that the diagonal of the Fisher information matrix
contains elements with both the signals and their slopes. By these normalizations, we aim pushing
these diagonal elements more similar to each other. As we cannot normalize both the signals and
the slopes to unity at the same time, we normalize their geometric mean to unity.
The effect of the ﬁrst two transformations can be compensated exactly by updating the shortcut
mapping C by
Cnew = Cold −A(αnew −αold)B
−A(βnew −βold) [0 0 ... 1] , (11)
where α is a matrix with elements αi on the diagonal and one empty row below for the bias term,
and β is a column vector with components βi and one zero below for the bias term. The third
transformation can be compensated by
Anew = Aoldγoldγ−1
new, (12)
where γ is a diagonal matrix with γi as the diagonal elements.
Schraudolph [12, 11] proposed centering the factors of the gradient to zero mean. It was argued that
deviations from the gradient fall into the linear subspace that the shortcut connections operate in, so
they do not harm the overall performance. Transforming the nonlinearities as proposed in this paper
has a similar effect on the gradient. Equation (3) corresponds to Schraudolph’s activity centering
and Equation (4) corresponds to slope centering.
3 Theoretical Comparison to a Second-Order Method
Second-order optimization methods, such as the natural gradient [1] or Newton’s method, decrease
the number of required iterations compared to the basic gradient descent, but they cannot be easily
used with large models due to heavy computations with large matrices. The natural gradient is
the basic gradient multiplied from the left by the inverse of the Fisher information matrix. Using
basic gradient descent can thus be seen as using the natural gradient while approximating the Fisher
information with a unit matrix multiplied by the inverse learning rate. We will show how the ﬁrst two
proposed transformations move the non-diagonal elements of the Fisher information matrix closer
to zero, and the third transformation makes the diagonal elements more similar in scale, thus making
the basic gradient behave closer to the natural gradient.
The Fisher information matrix contains elements
Gij =
∑
t
⟨∂2 log p(yt |xt,A,B,C)
∂θi∂θj
⟩
, (13)
3
where ⟨·⟩is the expectation over the Gaussian distribution of noise ϵt in Equation (1), and vector θ
contains all the elements of matrices A, B, and C. Note that here yt is a random variable and thus
the Fisher information does not depend on the output data. The Hessian matrix is closely related to
the Fisher information, but it does depend on the output data and contains more terms, and therefore
we show the analysis on the simpler Fisher information matrix.
The elements in the Fisher information matrix are:
∂
∂aij
∂
∂ai′j′
log p=
{ 0 i′̸= i
−1
σ2
i
∑
tfj(bjxt)fj′ (bj′ xt) i′= i, (14)
where aij is the ijth element of matrix A, fj is the jth nonlinearity, and bj is the jth row vector of
matrix B. Similarly
∂
∂bjk
∂
∂bj′k′
log p= −
∑
i
1
σ2
i
aijaij′
∑
t
f′
j(bjxt)f′
j′ (bj′ xt)xktxk′t (15)
and
∂
∂cik
∂
∂ci′k′
log p=
{ 0 i′̸= i
−1
σ2
i
∑
txktxk′t i′= i. (16)
The cross terms are
∂
∂aij
∂
∂bj′k
log p= −1
σ2
i
aij′
∑
t
fj(bjxt)f′
j′ (bj′ xt)xkt (17)
∂
∂cik
∂
∂ai′j
log p=
{ 0 i′̸= i
−1
σ2
i
∑
tfj(bjxt)xkt i′= i (18)
∂
∂cik
∂
∂bjk′
log p= −1
σ2
i
aij
∑
t
f′
j(bjxt)xktxk′t. (19)
Now we can notice that Equations (14–19) contain factors such as fj(·), f′
j(·), and xit. We ar-
gue that by making the factors as close to zero as possible, we help in making nondiagonal ele-
ments of the Fisher information closer to zero. For instance, E[fj(·)fj′ (·)] = E[fj(·)]E[fj′ (·)] +
Cov[fj(·),fj′ (·)], so assuming that the hidden unitsjand j′are representing different things, that is,
fj(·) and fj′ (·) are uncorrelated, the nondiagonal element of the Fisher information in Equation (14)
becomes exactly zero by using the transformations. When the units are not completely uncorrelated,
the element in question will be only approximately zero. The same argument applies to all other
elements in Equations (15–19), some of them also highlighting the beneﬁt of making the input data
xt zero-mean. Naturally, it is unrealistic to assume that inputs xt, nonlinear activations f(·), and
their slopes f′(·) are all uncorrelated, so the goodness of this approximation is empirically evaluated
in the next section.
The diagonal elements of the Fisher can be found in Equations (14–16) when i = i′, j = j′, and
k = k′. There we ﬁnd f(·)2 and f′(·)2 that we aim to keep similar in scale by using the third
transformation in Equation (5).
4 Empirical Comparison to a Second-Order Method
Here we investigate how linear transformations affect the gradient by comparing it to a second-order
method, namely Newton’s algorithm with a simple regularization to make the Hessian invertible.
We compute an approximation of the Hessian matrix using ﬁnite difference method, in which case
k-th row vector hk of the Hessian matrix H is given by
hk = ∂(∇E(θ))
∂θk
≈∇E(θ + δφk) −∇E(θ −δφk)
2δ , (20)
where φk = (0,0,..., 1,..., 0) is a vector of zeros and 1 at the k-th position, and the error func-
tion E(θ) = −∑
tlog p(yt |xt,θ). The resulting Hessian might still contain some very small or
4
500 1500 2500
−6
−4
−2
0
Eigenvalue order
Log10 eigenvalue
 
 
LTMLP
no−gamma
regular
(a) Eigenvalues
0.01 0.03 0.1 0.5
0
20
40
60
80
100
µ
Angle / degrees
 
 
LTMLP
no−gamma
regular (b) Angles
Figure 1: Comparison of (a) distributions of the eigenvalues of Hessians ( 2600 ×2600 matrix) and
(b) angles compared to the second-order update directions using LTMLP and regular MLP. In (a),
the eigenvalues are distributed most evenly when using LTMLP. (b) shows that gradients of the
transformed networks point to the directions closer to the second-order update.
even negative eigenvalues which cause its inversion to blow up. Therefore we do not use the Hes-
sian directly, but include a regularization term similarly as in the Levenberg-Marquardt algorithm,
resulting in a second-order update direction
∆θ = (H + µI)−1∇E(θ), (21)
where I denotes the unit matrix. Basically, Equation (21) combines the steepest descent and the
second-order update rule in such a way, that when µgets small, the update direction approaches the
Newton’s method and vice versa.
Computing the Hessian is computationally demanding and therefore we have to limit the size of
the network used in the experiment. We study the MNIST handwritten digit classiﬁcation problem
where the dimensionality of the input data has been reduced to 30 using PCA with a random rotation
[10]. We use a network with two hidden layers with architecture 30–25–20–10. The network was
trained using the standard gradient descent with weight decay regularization. Details of the training
are given in the appendix.
In what follows, networks with all three transformations ( LTMLP, linearly transformed multi-layer
perceptron network), with two transformations ( no-gamma where all γi are ﬁxed to unity) and a
network with no transformations (regular, where we ﬁxαi = 0, βi = 0, and γi = 1) were compared.
The Hessian matrix was approximated according to Equation (20) 10 times in regular intervals
during the training of networks. All ﬁgures are shown using the approximation after 4000 epochs of
training, which roughly corresponds to the midpoint of learning. However, the results were parallel
to the reported ones all along the training.
We studied the eigenvalues of the Hessian matrix (2600×2600) and the angles between the methods
compared and second-order update direction. The distribution of eigenvalues in Figure 1a for the
networks with transformations are more even compared to the regular MLP. Furthermore, there
are fewer negative eigenvalues, which are not shown in the plot, in the transformed networks. In
Figure 1b, the angles between the gradient and the second-order update direction are compared as
a function of µin Equation (21). The plots are cut when H + µI ceases to be positive deﬁnite as
µdecreases. Curiously, the update directions are closer to the second-order method, when γ is left
out, suggesting that γs are not necessarily useful in this respect.
Figure 2 shows histograms of the diagonal elements of the Hessian after 4000 epochs of training.
All the distributions are bimodal, but the distributions are closer to unimodal when transformations
are used (subﬁgures (a) and (b)) 1. Furthermore, the variance of the diagonal elements in log-scale
is smaller when using LTMLP, σ2
a = 0 .90, compared to the other two, σ2
b = 1 .71 and σ2
c =
1.43. This suggests that when transformations are used, the second-order update rule in Equation
(21) corrects different elements of the gradient vector more evenly compared to a regular back-
propagation learning, implying that the gradient vector is closer to the second-order update direction
when using all the transformations.
1It can be also argued whether (a) is more unimodal compared to (b).
5
10
−4
10
−20
50
100
150
200
(a) LTMLP
10
−4
10
−2
10
00
50
100
150
200 (b) no-gamma
10
−4
10
−20
50
100
150
200
250
 
 
1−2
1−3
2−3
1−4
2−4
3−4 (c) regular
Figure 2: Comparison of distributions of the diagonal elements of Hessians. Coloring according to
legend in (c) shows which layers to corresponding weights connect (1 = input, 4 = output). Diagonal
elements are most concentrated in LTMLP and most spread in the regular MLP network. Notice the
logarithmic x-axis.
To conclude this section, there is no clear evidence in way or another whether the addition ofγben-
eﬁts the back-propagation learning with only αand β. However, there are some differences between
these two approaches. In any case, it seems clear that transforming the nonlinearities beneﬁts the
learning compared to the standard back-propagation learning.
5 Experiments: MNIST Classiﬁcation
We use the proposed transformations for training MLP networks for MNIST classiﬁcation task.
Experiments are conducted without pretraining, weight-sharing, enhancements of the training set or
any other known tricks to boost the performance. No weight decay is used and as only regularization
we add Gaussian noise with σ = 0 .3 to the training data. Networks with two and three hidden
layers with architechtures 784–800–800–10 (solid lines) and 784–400–400–400–10 (dashed lines)
are used. Details are given in the appendix.
Figure 3 shows the results as number of errors in classifying the test set of 10 000 samples. The
results of the regular back-propagation without transformations, shown in blue, are well in line with
previously published result for this task. When networks with same architecture are trained using
the proposed transformations, the results are improved signiﬁcantly. However, adding γin addition
to previously proposed αand βdoes not seem to affect results on this data set. The best results, 112
errors, is obtained by the smaller architecture without γ and for the three-layer architecture with γ
the result is 114 errors. The learning seems to converge faster, especially in the three-layer case,
with γ. The results are in line what was obtained in [10] where the networks were regularized more
thoroughly. These results show that it is possible to obtain results comparable to dropout networks
(see [5]) using only minimal regularization.
6 Experiments: MNIST Autoencoder
Previously, we have studied an auto-encoder network using two transformations, αand β, in [10].
Now we use the same auto-encoder architecture, 784–500–250–30–250–500–784. Adding the third
transformation γ for training the auto-encoder poses problems. Many hidden neurons in decoding
layers (i.e., 4th and 5th hidden layers) tend to be relatively inactive in the beginning of training,
which induces corresponding γs to obtain very large values. In our experiments, auto-encoder with
γs eventually diverge despite simple constraint we experimented with, such as γi ≤100. This be-
havior is illustrated in Figure 4. The subﬁgure (a) shows the distribution of variances of outputs of all
hidden neurons in MNIST classiﬁcation network used in Section 5 given the MNIST training data.
The corresponding distribution for hidden neurons in the decoder part of the auto-encoder is shown
in the subﬁgure (b). The “dead neurons“ can be seen as a peak in the origin. The corresponding γs,
constrained γi ≤100, can be seen in the subﬁgure (c). We hypothesize that this behavior is due to
the fact, that in the beginning of the learning there is not much information reaching the bottleneck
layer through the encoder part and thus there is nothing to learn for the decoding neurons. According
to our tentative experiments, the problem described above may be overcome by disabling γs in the
6
0 50 100 150
100
120
140
160
180
200
220
240
Epochs
Number of errors
 
 
LTMLP
no−gamma
regular
Figure 3: The error rate on the MNIST test set for LTMLP training, LTMLP without γ and regular
back-propagation. The solid lines show results for networks with two hidden layers of 800 neurons
and the dashed lines for networks with three hidden layers of 400 neurons.
0 1 2 3
0
50
100
150
200
250
Variance
(a) MNIST classiﬁcation
0 0.5 1 1.5 2
0
50
100
150
200
Variance (b) MNIST auto-encoder
0 50 100
0
100
200
300
γ (c) MNIST auto-encoder
Figure 4: Histograms of (a-b) variation of output of hidden neurons given the MNIST training data
and (c) γs of the decoder part (4th and 5th hidden layer) in the MNIST auto-encoder. (a) shows
a healthy distributions of variances, whereas in (b), which includes only variances of the decoder
part, there are many “dead neurons”. These neurons induce corresponding γs, histogram of which
is shown in (c), to blow up which eventually lead to divergence.
decoder network (i.e., ﬁx γ = 1). However, this does not seem to speed up the learning compared
to our earlier results with only two transformations in [10]. It is also be possible to experiment with
weight-sharing or other constraints to overcome the difﬁculties with γs.
7 Discussion and Conclusions
We have shown that introducing linear transformation in nonlinearities signiﬁcantly improves the
back-propagation learning in (deep) MLP networks. In addition to two transformation proposed
earlier in [10], we propose adding a third transformation in order to push the Fisher information
matrix closer to unit matrix (apart from its scale). The hypothesis proposed in [10], that the transfor-
mations actually mimic a second-order update rule, was conﬁrmed by experiments comparing the
networks with transformations and regular MLP network to a second-order update method. How-
ever, in order to ﬁnd out whether the third transformation, γ, we proposed in this paper, is really
useful, more experiments ought to be conducted. It might be useful to design experiments where
convergence is usually very slow, thus revealing possible differences between the methods. As hy-
perparameter selection and regularization are usually nuisance in practical use of neural networks,
it would be interesting to see whether combining dropouts [5] and our transformations can provide
a robust framework enabling training of robust neural networks in reasonable time.
7
The effect of the ﬁrst two transformations is very similar to gradient factor centering [12, 11], but
transforming the model instead of the gradient makes it easier to generalize to other contexts: When
learning by by MCMC, variational Bayes, or by genetic algorithms, one would not compute the
basic gradient at all. For instance, consider using the Metropolis algorithm on the weight matrices,
and expecially matrices A and B. Without transformations, the proposed jumps would affect the
expected output yt and the expected linear dependency∂yt/∂xt in Eqs. (9)–(10), thus often leading
to low acceptance probability and poor mixing. With the proposed transformations included, longer
proposed jumps in A and B could be accepted, thus mixing the nonlinear part of the mapping faster.
For further discussion, see [10], Section 6. The implications of the proposed transformations in
these other contexts are left as future work.
References
[1] S. Amari. Natural gradient works efﬁciently in learning. Neural Computation, 10(2):251–276,
1998.
[2] D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber. Deep big simple neural nets
excel on handwritten digit recognition. CoRR, abs/1003.0358, 2010.
[3] X. Glorot and Y . Bengio. Understanding the difﬁculty of training deep feedforward neural
networks. In Proceedings of the Thirteenth International Conference on Artiﬁcial Intelligence
and Statistics (AISTATS), pages 249–256, 2010.
[4] G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural net-
works. Science, 313(5786):504–507, 2006.
[5] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut-
dinov. Improving neural networks by preventing co-adaptation of feature detectors. CoRR,
abs/1207.0580, 2012.
[6] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classiﬁcation with deep convolutional
neural networks. 2012.
[7] N. Le Roux, P. A. Manzagol, and Y . Bengio. Topmoumoute online natural gradient algorithm.
In Advances in Neural Information Processing Systems 20 (NIPS*2007), 2008.
[8] Y . LeCun, L. Bottou, G. B. Orr, and K.-R. M ¨uller. Efﬁcient backprop. In Neural Networks:
tricks of the trade. Springer-Verlag, 1998.
[9] J. Martens. Deep learning via Hessian-free optimization. In Proceedings of the 27th Interna-
tional Conference on Machine Learning (ICML), 2010.
[10] Tapani Raiko, Harri Valpola, and Yann LeCun. Deep learning made easier by linear transfor-
mations in perceptrons. Journal of Machine Learning Research - Proceedings Track, 22:924–
932, 2012.
[11] N. N. Schraudolph. Accelerated gradient descent by factor-centering decomposition. Technical
Report IDSIA-33-98, Istituto Dalle Molle di Studi sull’Intelligenza Artiﬁciale, 1998.
[12] N. N. Schraudolph. Centering neural network gradient factors. In Genevieve Orr and Klaus-
Robert Mller, editors, Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in
Computer Science, pages 548–548. Springer Berlin / Heidelberg, 1998.
Appendix
Details of Section 4
In experiments of Section 4, networks with all three transformations (LTMLP), only αand β (no-
gamma) and network with no transformations (regular) were compared. Full batch training without
momentum was used to make things as simple as possible. The networks were regularized using
weight decay and adding Gaussian noise to the training data. Three hyperparameters, weight de-
cay term, input noise variance and learning rate, were validated for all networks separately. The
input data was normalized to zero mean and the network was initialized as proposed in [3], that is,
the weights were drawn from a uniform distribution between ±
√
6/√nj + nj+1, where nj is the
number of neurons on the jth layer.
8
20 40 60 80 100
0
2
4
6
8
10Training error / %
Training time / %
 
 
LTMLP
no−gamma
regular
(a) Training error
20 40 60 80 100
0
2
4
6
8
10Test error / %
Training time / %
 
 
LTMLP
no−gamma
regular (b) Test error
Figure 5: Comparison of (a) training and (b) test errors of the algorithms using the MNIST data
in the experiment comparing them to the second-order method. Note how the best learning for the
regular MLP is relatively high, leading to oscillations until it is annealed towards the end.
We sampled the three hyperparameters randomly (given our best guess intervals) for 500 runs and
selected the median of the runs that resulted in the best 50 validation errors as the hyperparameters.
Resulting hyperparameters are listed in Table 1. Notable differences occur in step sizes, as it seems
that networks with transformations allow using signiﬁcantly larger step size which in turn results in
more complete search in the weight space.
Our weight updates are given by
θτ = θτ−1 −ετ∇θτ. (22)
where the learning rate on iteration τ, ετ, is given by
ετ =
{
ε0 τ ≤T/2
2(1 −τ
T)ε0 τ >T/2 (23)
that is, the learning rate starts decreasing linearly after the midpoint of the given training time T.
Furthermore, the learning rate ετ is dampened for shortcut connection weights by multiplying with(1
2
)s
, where sis number of skipped layers as proposed in [10]. 2 Figure 5 shows training and test
errors for the networks. The LTMLP obtains the best results although there is no big difference
compared to training without γ.
Details of Section 5
The MNIST dataset consists of 28 ×28 images of hand-drawn digits. There are 60 000 training
samples and 10 000 test samples. We experimented with two networks with two and three hidden
layers and number of hidden neurons by arbitrary choice. Training was done in minibatch mode with
1000 samples in each batch and transformations are updated on every iteration using the current
minibatch with using (6)–(8). This seems to speed up learning compared to the approach in [10]
where transformations were updated only occasionally with the full training data. Random Gaussian
noise with σ= 0.3 was injected to the training data in the beginning of each epoch.
2This heuristic is not well supported by analysis of Figure 2 and could be re-examined.
Table 1: Hyperparameters for the neural networks
LTMLP no-gamma regular
weight decay 4.6 ×10−5 1.3 ×10−5 3.9 ×10−5
noise 0.31 0.36 0.29
step size 1.2 2.5 0.45
9
Our weight update equations are given by:
∆θτ = ∇θ + pτ∆θt−1, (24)
θτ = θτ−1 −ετ∆θτ, (25)
where
ετ =
{ ε0 τ ≤T
ε0fτ−T τ >T (26)
pτ =
{ τ
Tpf + (1 −τ
T)p0 τ ≤T
pf τ >T (27)
In the equations above, T is a “burn-in time” where momentum pτ is increased from starting value
p0 = 0.5 to pf = 0.9 and learning rate ε = ε0 is kept constant. When τ > Tmomentum is kept
constant and learning rate starts decreasing exponentially with f = 0.9. Hyperparameters were not
validated but chosen by arbitrary guess such that learning did not diverge. For the regular training,
ε0 = 0 .05 was selected since it diverged with higher learning rates. Then according to lessons
learned, e.g. in Section 4, ε0 = 0.3 was set for LTMLP with γand ε0 = 0.7 for the variant with no
γ. Basically, it seems that transformations allow using higher learning rates and thus enable faster
convergence.
10


Peer review:
[Review]:
First of all we would like to thank you for your informed, thorough and kind comments. We realize that there is major overlap with our previous paper [10]. We hope that these two papers could be combined in a journal paper later on. It was mentioned that we use some text verbatim from [10]. There is some basic methodology which is necessary to explain before going to deeper explanations and we felt that it is not a big violation to use our own text. However, we have now modified the sections in question with your comments and proposals in mind. If you feel that it is necessary to check every sentence for verbatim, please consider conditional acceptance with this condition.

We agree that the evidence supporting the use of the third transformation is rather weak. We have tried to report our findings as honestly as possible and also express our doubts in the paper (see, e.g.,  end of Section 4).

To reviewer 'Anonymous 1567':

You argue that Eqs. (12-17) are slightly accurate. However, we have computed the expectation over epsilon with pen and paper and epsilon does vanish from the Eqs. Thus, the Eqs. in question are exact. Would you think that we should still write down the gradients explicitly?

We had considered using the df/dx notation, but decided to use the f' notation, since the derivative is taken with respect to Bx and using the df/dx notation would require us to define a new variable u = Bx and denote df/du. We think, this would further clutter the equations. Would you think this is acceptable?

We tried to clarify the meaning of 'transforming the model instead of the gradient ...' in Discussion.

To reviewer 'Anonymous b670':

We have now explained the relationship to Schraudolph's method in more detail. We provide an example and refer to Discussion of [10].

When writing about 'many competing paths' and 'linear dependencies', we have added explanations with equations in the updated version.

The question, whether the arguments in Section 3 can be applied to networks with more than one hidden layer: We have presented the theory with this simplified case in order to convey the understanding to the reader. We assume that the idea could be formulated in the general (deep) case, but writing it out would substantially complicate the equations. Our experimental results support this assumption.

About the uncorrelatedness assumption, we have added the following explanation: 'Naturally, it is unrealistic to assume that inputs $x_t$, nonlinear activations $f(cdot)$, and their slopes $f^prime(cdot)$ are all uncorrelated, so the goodness of this approximation is empirically evaluated in the next section.'

We do realize that it is possible and more elegant to compute exact solution for the Hessian matrix. However, as being more error prone, it would require careful checking by, e.g., some approximative method. As the mere approximation suits our needs well, we refrained from doing the extra work for the exact solution. We have also acknowledged this in the paper.

Regarding mu in Eq. 19: Thanks for this remark. We have reformulated the text surrounding Eq. 19. Could you kindly provide further suggestions and/or references if you still find it unsatisfactory.

Experiments on eigenvalue distribution: Fig. 1(a) suggest that there is no clear difference between the eigenvalue distributions with our without gamma (the vertical position of the plot is irrelevant since it corresponds to choosing a different learning rate).

We show histogram of diagonal elements in order to distinguish between weights. For instance, the colors in Fig.2 could not have been used otherwise.

Fisher Information vs. Hessian matrix: This is a relevant point for the future work. The Hessian describes the curvature of the actual optimization problem. We chose Fisher information matrix in the theoretical part simply because it has more compact equations. As we note in the paper, 'the hessian matrix is closely related to the Fisher information matrix, but it does depend on the output data and contains more term'. We argue that the terms present in Fisher information matrix will make our point clear and adding the other terms included in the Hessian would just be additional clutter.

Tommi Vatanen and Tapani Raiko