Title: DEMYSTIFYING RESNET

Abstract: We provide a theoretical explanation for the great performance of ResNet via the study of deep linear networks and some nonlinear variants. We show that with or without nonlinearities, by adding shortcuts that have depth two, the condition number of the Hessian of the loss function at the zero initial point is depthinvariant, which makes training very deep models no more difficult than shallow ones. Shortcuts of higher depth result in an extremely flat (high-order) stationary point initially, from which the optimization algorithm is hard to escape. The 1shortcut, however, is essentially equivalent to no shortcuts. Extensive experiments are provided accompanying our theoretical results. We show that initializing the network to small weights with 2-shortcuts achieves significantly better results than random Gaussian (Xavier) initialization, orthogonal initialization, and shortcuts of deeper depth, from various perspectives ranging from final loss, learning dynamics and stability, to the behavior of the Hessian along the learning process.

1. INTRODUCTION:

Residual network (ResNet) was first proposed in He et al. (2015a) and extended in He et al. (2016) . It followed a principled approach to add shortcut connections every two layers to a VGG-style network (Simonyan & Zisserman, 2014) . The new network becomes easier to train, and achieves both lower training and test errors. Using the new structure, He et al. (2015a) managed to train a network with 1001 layers, which was virtually impossible before. Unlike Highway Network (Srivastava et al., 2015a; b) which not only has shortcut paths but also borrows the idea of gates from LSTM (Sainath et al., 2013) , ResNet does not have gates. Later He et al. (2016) found that by keeping a clean shortcut path, residual networks will perform even better.

Many attempts have been made to improve ResNet to a further extent. "ResNet in ResNet" (Targ et al., 2016) adds more convolution layers and data paths to each layer, making it capable of representing several types of residual units. "ResNets of ResNets" (Zhang et al., 2016 ) construct multilevel shortcut connections, which means there exist shortcuts that skip multiple residual units. Wide Residual Networks (Zagoruyko & Komodakis, 2016) makes the residual network shorter but wider, and achieves state of the art results on several datasets while using a shallower network. Moreover, some existing models are also reported to be improved by shortcut connections, including Inception-v4 (Szegedy et al., 2016) , in which shortcut connections make the deep network easier to train.

Why are residual networks so easy to train? He et al. (2015a) suggests that layers in residual networks are learning residual mappings, making them easier to represent identity mappings, which prevents the networks from degradation when the depths of the networks increase. However, Veit et al. (2016) claims that ResNets are actually ensembles of shallow networks, which means they do not solve the problem of training deep networks completely.

We propose a theoretical explanation for the great performance of ResNet. We concur with He et al. (2015a) that the key contribution of ResNet should be some special structure of the loss function that makes training very deep models no more difficult than shallow ones. Analysis, however, seems non-trivial. Quoting He et al. (2015a) :

"Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy from increased depth (as shown on CIFAR-10), but are not as economical as the bottleneck ResNets. So the usage of bottleneck designs is mainly due to practical considerations. We further note that the degradation problem of plain nets is also witnessed for the bottleneck designs. "

Their empirical observations are inspiring. First, the 1-shortcuts mentioned in the first paragraph do not work. Second, noting that the non-bottleneck ResNets have 2-shortcuts, but the bottleneck ResNets use 3-shortcuts, one sees that shortcuts with depth three also do not work. Hence, a reasonable theoretical explanation must be able to distinguish the 2-shortcut from shortcuts of other depths, and clearly demonstrate why the 2-shortcuts are special and are able to ease the optimization process so significantly for deep models, while shortcuts of other depths may not do the job.

FIX:

Aiming at explaining the performance of 2-shortcuts, we need to eliminate other variables that may contribute to the success of ResNet. Indeed, one may argue that the deep structure of ResNet may give it better representation power (lower approximation error), which contributes to lower training errors. To eliminate this effect, we focus on deep linear networks, where deeper models do not have better approximation properties. The special role of 2-shortcuts naturally arises in the study.

2. MAIN RESULTS:

Our work reveals that non-degenerate depth-invariant initial condition numbers, a unique property of residual networks with 2-shortcuts, contributed to the success of ResNet. In fact, in a linear network that will be defined rigorously later, the condition number of Hessian of the Frobenius loss function at the zero initial point is

EQUATION

which is independent of the number of layers. Here Σ XX and Σ Y X denote the input-input and the output-input correlation matrices, defined in Section 3.3. The condition number of a possibly non-PSD matrix is defined as: Definition 1. The condition number of a matrix A is defined as

EQUATION

where σ max (A) and σ min (A) are the maximum and minimum of singular values of A. In particular, if A is normal, i.e. A T A = AA T , the definition can be simplified to 1

EQUATION

where |λ(A)| max and |λ(A)| min are the maximum and minimum of the absolute values of eigenvalues of A.

Moreover, the zero initial point for ResNet with 2-shortcuts is in fact a so-called strict saddle point (Ge et al., 2015) , which are proved to be easy to escape from.

Why shortcuts of other depths do not work? We show that the Hessian at the zero initial point for the 1-shortcut ResNet has condition number growing unboundedly for deep nets. As is well known in convex optimization theory, large condition numbers can have enormous adversarial impact on the convergence of first order methods (Nemirovski, 2005) . Hence, it is quite clear that starting training at a point with a huge condition number would make the algorithm very difficult to escape from the initial point, making 1-shortcut ResNet no better than conventional approaches.

1 The equivalence of Equation (2) and Equation (3) can be proved easily using the eigenvalue decomposition of A. Note that as Hessians are symmetric (if all the second derivatives are continuous), we will use Equation (3) for their condition numbers. As the |λ|min of Hessian is usually very unstable, we calculated |λ|max |λ| (0.1) to represent condition numbers instead, where |λ| (0.1) is the 10 th percentile of the absolute values of eigenvalues. Note that the Hessian at zero initial point for 2-shortcut ResNet also has a nice structure of spectrum: see Theorem 1 for details.

For shortcuts with depth deeper than two, the Hessian at the zero initial point is a zero matrix, making it a higher-order stationary point. Intuitively, the higher order the stationary point is, the harder it is to escape from it. Indeed, it is supported both in theory (Anandkumar & Ge, 2016) and by our experiments.

One may still ask: why are we interested in the Hessian at the zero initial point? It is because in order for the outputs of deep neural networks not explode, the singular values of the mapping of each layer are not supposed to be deviating too much from one. Indeed, it is because it is extremely challenging to keep num of layers i=1 λ i from exploding or vanishing without keeping all of the λ i having unit norm. However, by design ResNet with shortcuts already have an identity mapping every few layers, which forces the mappings inside the shortcuts to have small operator norms. Hence, analyzing the network at zero initial point gives a decent characterization of the searching environment of the optimization algorithm.

Our results also shows that the form of Hessian is more important than the existance of nonlinearities when training the networks. The behaviors of the networks we studied are consistent across both linear and nonlinear structures, where networks with clearer Hessians are much easier to achieve lower training errors.

On the other hand, our experiments reveal that orthogonal initialization (Saxe et al., 2013) is suboptimal. Although better than Xavier initialization (Glorot & Bengio, 2010) , the initial condition numbers of the networks still explode as the networks become deeper, which means the networks are still initialized on "bad" submanifolds that are hard to optimize using gradient descent.

3. MODEL:



3.1. DEEP LINEAR NETWORKS:

Deep linear networks are feed-forward neural networks that only contain linear units, which means their input-output mappings are simply linear transformations. Apparently, increasing their depths will not affect the representational power of the networks. However, linear networks with depth deeper than one show nonlinear dynamics of training (Saxe et al., 2013) . As a result, analyzing the training of deep linear networks gives us a better understanding of the training of non-linear networks.

Much theoretical work has been done on deep linear networks. Kawaguchi (2016) extended the work of Choromanska et al. (2015a; b) and proved that with few assumptions, every local minimum point in deep linear networks is a global minimum point. This means that the difficulties in the training of deep linear networks mostly come from saddle points on the loss surfaces, which are also the main causes of slow learning in nonlinear networks (Pascanu et al., 2014) . Saxe et al. (2013) studied the dynamics of training using gradient descent. They found that for a special class of initial conditions, which could be obtained from greedy layerwise pre-training, the training time for a deep linear network with an infinity depth can still be finite. Furthermore, they found that by setting the initial weights to random orthogonal matrices (produced by performing QR or SVD decompositions on random Gaussian matrices), the network will still have a depth independent learning time. They argue that it is caused by the eigenvalue and singular value spectra of the end-to-end linear transformation. When using orthogonal initialization, the overall transformation is an orthogonal matrix, which has all the singular values equal to 1. In the meantime, when using scaled Gaussian initialization, most of the singular values are close to zero, making the network unsuitable for backpropagating errors. However, this explanation is not sufficient to prove that the training difficulty of orthogonal initialized networks is depth-invariant. It only gives us an intuition on why orthogonal initialization performs better than scaled Gaussian initialization. Thus, we use deep linear networks to study the effect of shortcut connections. After adding the shortcuts, the overall model is still linear and the global minimum does not change.

3.2. NETWORK STRUCTURE:

We first generalize a linear network by adding shortcuts to it to make it a linear residual network. We organize the network into R residual units. The r-th residual unit consists of L r layers whose weights are W r,1 , . . . , W r,Lr-1 , denoted as the transformation path, as well as a shortcut S r connecting from the first layer to the last one, denoted as the shortcut path. The input-output mapping can be written as

EQUATION

where

x ∈ R dx , y ∈ R dy , W ∈ R dy×dx . Here if b ≥ a, b i=a W i denotes W b W (b-1) • • • W (a+1)

W a , otherwise it denotes an identity mapping. The matrix W represents the combination of all the linear transformations in the network. Note that by setting all the shortcuts to zeros, the network will go back to a ( r (L r -1) + 1)-layer plain linear network.

Instead of analyzing the general form, we concentrate on a special kind of linear residual networks, where all the residual units are the same.

Definition 2. A linear residual network is called an n-shortcut linear network if 1. its layers have the same dimension (so that d x = d y ); 2. its shortcuts are identity matrices;

3. its shortcuts have the same depth n.

The input-output mapping for such a network becomes

EQUATION

where W r,l ∈ R dx×dx .

Then we add some activation functions to the networks. We concentrate on the case where activation functions are on the transformation paths, which is also the case in the latest ResNet (He et al., 2016) .

Definition 3. An n-shortcut linear network becomes an n-shortcut network if element-wise activation functions σ pre (x), σ mid (x), σ post (x) are added at the transformation paths, where on a transformation path, σ pre (x) is added before the first weight matrix, σ mid (x) is added between two weight matrixes and σ post (x) is added after the last weight matrix. Note that n-shortcut linear networks are special cases of n-shortcut networks, where all the activation functions are identity mappings.

3.3. OPTIMIZATION:

We denote the collection of all the variable weight parameters in an n-shortcut linear network as w. Consider m training samples {x µ , y µ }, µ = 1, . . . , m. Using Frobenius loss, for an n-shortcut linear network, we define the loss function as follows:

EQUATION

where x µ , y µ are the µ-th columns of X, Y , and • F denotes the Frobenius norm. Using gradient descent with learning rate α, we have the weights updating rules as

EQUATION

where Σ XX and Σ Y X denote the input-input and the output-input correlation matrices, defined as

EQUATION

Here W r before , W r after denote the linear mappings before and after the r-th residual unit, W r,l before , W r,l after denote the linear mappings before and after W r,l within the transformation path of the r-th residual unit. In other words, the overall transformation can be represented as

EQUATION

)

4 THEORETICAL STUDY

4.1. INITIAL POINT PROPERTIES:

Before we analyze the initial point properties of n-shortcut networks, we have to choose the way to initialize them. NEW

ResNet uses MSRA initialization (He et al., 2015b) . It is a kind of scaled Gaussian initialization that tries to keep the variances of signals along a transformation path, which is also the idea behind Xavier initialization (Glorot & Bengio, 2010) . However, because of the shortcut paths, the output variance of the entire network will actually explode as the network becomes deeper. Batch normalization units partly solved this problem in ResNet, but still they cannot prevent the large output variance in a deep network. NEW A simple idea is to zero initialize all the weights, so that the output variances of residual units stay the same along the network. It is worth noting that as found in He et al. (2015a) , the deeper ResNet has smaller magnitudes of layer responses. This phenomenon has been confirmed in our experiments. As illustrated in Figure 2 and Figure 3 , the deeper a residual network is, the small its average Frobenius norm of weight matrixes is, both during the traning process and when the training ends. Also, Hardt & Ma (2016) proves that if all the weight matrixes have small norms, a linear residual network will have no critical points other than the global optimum. NEW All these evidences indicate that zero is spacial in a residual network: as the network becomes deeper, the training tends to end up around it. Thus, we are looking into the Hessian at zero. As the zero is a saddle point, in our experiments we use zero initialization with small random perturbations to escape from it. We first Xavier initialize the weight matrixes, and then multiply a small constant (0,01) to them.

We begin with the definition of k-th order stationary point.

Definition 4. Suppose function f (x) admits k-th order Taylor expansion at point x 0 . We say that the point x 0 is a k-th order stationary point of f (x) if the corresponding k-th order Taylor expansion of f (x) at x = x 0 is a constant:

f (x) = f (x 0 ) + o( x -x 0 k 2 ).

Now we state our main theorem, whose proof can be found in Appendix A.

Theorem 1. Assume that σ mid (0) = σ post (0) = 0 and all of σ

(k) pre (0), σ (k) mid (0), σ (k) post (0), 1 ≤ k ≤ max(n -1, 2) exist.

For the loss function of an n-shortcut network, at point zero, 1. if n ≥ 2, it is an (n -1)th-order stationary point. In particular, if n ≥ 3, the Hessian is a zero matrix; 2. if n = 2, the Hessian can be written as

EQUATION

whose condition number is

EQUATION

where A only depends on the training set and the activation functions. Except for degenerate cases, it is a strict saddle point (Ge et al., 2015) .

3. if n = 1, the Hessian can be written as

H =         B A T A T • • • A T A B A T • • • A T A A B . . . . . . . . . . . . A T A A • • • A B         (13)

where A, B only depend on the training set and the activation functions.

Theorem 1 shows that the condition numbers of 2-shortcut networks are depth-invariant with a nice structure of eigenvalues. Indeed, the eigenvalues of the Hessian H at the zero initial point are multiple copies of ± eigs(A T A), and the number of copies is equal to the number of shortcut connections.

The Hessian at zero initial point for the 1-shortcut linear network follows block Toeplitz structure, which has been well studied in the literature. In particular, its condition number tends to explode as the number of layers increase (Gray, 2006) .

The assumptions hold for most activation functions including tanh, symmetric sigmoid and ReLU (Nair & Hinton, 2010) . Note that although ReLU does not have derivatives at zero, one may do a local polynomial approximation to yield σ

(k) , 1 ≤ k ≤ max(n -1, 2).

To get intuitive explanations of the theorems, imagine changing parameters in an n-shortcut network.

One has to change at least n parameters to make any difference in the loss. So zero is an (n -1)thorder stationary point. Notice that the higher the order of a stationary point, the more difficult for a first order method to escape from it.

On the other hand, if n = 2, one will have to change two parameters in the same residual unit but different weight matrices to affect the loss, leading to a clear block diagonal Hessian.

4.2. LEARNING DYNAMICS:

To understand Equation ( 7) better, we can take n-shortcut linear networks to two extremes. First, when n = 1, let V r,1 = W r,1 + I dx , r = 1, . . . , R -1. As I dx is a constant, we have

EQUATION

which can be seen as a linear network with identity initialization, a special case of orthogonal initialization, if the original 1-shortcut network is zero initialized. 

4.3. LEARNING RESULTS:

The optimal weights of an n-shortcut linear network can be easily computed via least squares, which leads to

EQUATION

)

and the minimum of the loss function is

EQUATION

where • F denotes the Frobenius norm and (Σ XX ) -1 denotes any kind of generalized inverse of Σ XX . So given a training set, we can pre-compute its L min and use it to evaluate any n-shortcut linear network.

5. EXPERIMENTS:

We compare networks with Xavier initialization (Glorot & Bengio, 2010) , networks with orthogonal initialization (Saxe et al., 2013) and 2-shortcut networks with zero initialization. The training dynamics of 1-shortcut networks are similar to that of linear networks with orthogonal initialization in our experiments. Setup details can be found in Appendix B.

5.1. INITIAL POINT:

We first compute the initial condition numbers for different kinds of linear networks with different depths.

As can be seen in Figure 5 , 2-shortcut linear networks have constant condition numbers as expected. On the other hand, when using Xavier or orthogonal initialization in linear networks, the initial condition numbers will go to infinity as the depths become infinity, making the networks hard to train. This also explains why orthogonal initialization is helpful for a linear network, as its initial condition number grows slower than the Xavier initialization.

Having a good beginning does not guarantee an easy trip on the loss surface. In order to depict the loss surfaces encountered from different initial points, we plot the maxima and 10 th percentiles (instead of minima, as they are very unstable) of the absolute values of Hessians eigenvalues at different losses.

As shown in Figure 6 and Figure 7 , the condition numbers of 2-shortcut networks at different losses are always smaller, especially when the loss is large. Also, notice that the condition numbers roughly evolved to the same value for both orthogonal and 2-shortcut linear networks. This may be explained by the fact that the minimizers, as well as any point near them, have similar condition numbers.

2-Shortcut:

O r t h o g o n a l X a v i e r 

lg(condition number):





O rt h o g o n a l Maximum 1 0 t h p e r c e n t i l e:

Figure 6 : Maxima and 10 th percentiles of absolute values of eigenvalues at different losses when the depth is 16. For each run, eigenvalues at different losses are calculated using linear interpolation. Another observation is the changes of negative eigenvalues ratios. Index (ratio of negative eigenvalues) is an important characteristic of a critical point. Usually for the critical points of a neural network, the larger the loss the larger the index (Dauphin et al., 2014) . In our experiments, the index of a 2-shortcut network is always smaller, and drops dramatically at the beginning, as shown in Figure 8 , left. This might make the networks tend to stop at low critical points.

EQUATION

Orthogonal index lg( gr ad ien t)

Figure 8 : Left: ratio of negative eigenvalues at different losses when the depth is 16. For each run, indexes at different losses are calculated using linear interpolation. Right: the dynamics of gradient and index of a 2-shortcut linear network in a single run. The gradient reaches its maximum while the index drops dramatically, indicating moving toward negative curvature directions. This is because the initial point is near a saddle point, thus it tends to go towards negative curvature directions, eliminating some negative eigenvalues at the beginning. This phenomenon matches the observation that the gradient reaches its maximum when the index drops dramatically, as shown in Figure 8 , right.

We run different networks for 1000 epochs using different learning rates at log scale, and compare the average final losses of the optimal learning rates. 9 : Left: Optimal Final losses of different linear networks. Right: Corresponding optimal learning rates. When the depth is 96, the final losses of Xavier with different learning rates are basically the same, so the optimal learning rate is omitted as it is very unstable.

Figure 9 shows the results for linear networks. Just like their depth-invariant initial condition numbers, the final losses of 2-shortcut linear networks stay close to optimal as the networks become deeper. Higher learning rates can also be applied, resulting in fast learning in deep networks.

Then we add ReLUs to the mid positions of the networks. To make a fair comparison, the numbers of ReLU units in different networks are the same when the depths are the same, so 1-shortcut and 3-shortcut networks are omitted. The result is shown in Figure 10 . Corresponding optimal learning rates. Note that as it is hard to compute the minimum losses with ReLUs, we plot the log 10 (final loss) instead of log 10 (final lossoptimal loss). When the depth is 64, the final losses of Xavier-ReLU and orthogonal-ReLU with different learning rates are basically the same, so the optimal learning rates are omitted as they are very unstable.

Note that because of the nonlinearities, the optimal losses vary for different networks with different depths. It is usually thought that deeper networks can represent more complex models, leading to smaller optimal losses. However, our experiments show that linear networks with Xavier or orthogonal initialization have difficulties finding these optimal points, while 2-shortcut networks find these optimal points easily as they did without nonlinear units.

A PROOFS OF THEOREMS Definition 5. The elements in Hessian of an n-shortcut network is defined as

EQUATION

)

where L is the loss function, and the indices ind(•) is ordered lexicographically following the four indices (r, l, j, i) of the weight variable w r,l i,j . In other words, the priority decreases along the index of shortcuts, index of weight matrix inside shortcuts, index of column, and index of row.

Note that the collection of all the weight variables in the n-shortcut network is denoted as w. We study the behavior of the loss function in the vicinity of w = 0.

Lemma 1. Assume that w 1 = w r1,l1 i1,j1 , • • • , w N = w r N ,l N i N ,j N are N parameters of an n-shortcut network. If ∂ 2 L ∂w1•••∂w N w=0

is nonzero, there exists r and k 1 , • • • , k n such that r km = r and l km = m for m = 1, • • • , n.

Proof. Assume there does not exist such r and k 1 , • • • , k n , then for all the shortcut units r = 1, • • • , R, there exists a weight matrix l such that none of w 1 , • • • , w N is in W r,l , so all the transformation paths are zero, which means W = I dx . Then

∂ 2 L ∂w1•••∂w N w=0 = 0, leading to a contradic- tion.

Lemma 2. Assume that w 1 = w r1,l1 i1,j1 , w 2 = w r2,l2 i2,j2 , r 1 ≤ r 2 . Let L 0 (w 1 , w 2 ) denotes the loss function with all the parameters except w 1 and w 2 set to 0,

EQUATION

.

Then ∂ 2 L0(w1,w2) ∂w1∂w2 | (w1,w2)=0 = ∂ 2 L0(w 1 ,w 2 ) ∂w 1 ∂w 2 | (w 1 ,w 2 )=0 .

Proof. As all the residual units expect unit r 1 and r 2 are identity transformations, reordering residual units while preserving the order of units r 1 and r 2 will not affect the overall transformation, i.e. L 0 (w 1 , w

2 )| w1=a,w2=b = L 0 (w 1 , w 2 )| w 1 =a,w 2 =b . So ∂ 2 L0(w1,w2) ∂w1∂w2 | (w1,w2)=0 = ∂ 2 L0(w 1 ,w 2 ) ∂w 1 ∂w 2 | (w 1 ,w 2 )=0 .

Proof of Theorem 1. Now we can prove Theorem 1 with the help of the previously established lemmas.

1. Using Lemma 1, for an n-shortcut network, at zero, all the k-th order partial derivatives of the loss function are zero, where k ranges from 1 to n -1. Hence, the initial point zero is a (n -1)th-order stationary point of the loss function.

2. Consider the Hessian in n = 2 case. Using Lemma 1 and Lemma 2, the form of Hessian can be directly written as Equation ( 11), as illustrated in Figure 11 .

So we have

EQUATION

Thus cond(H) = cond(A T A), which is depth-invariant. Note that the dimension of A is d 2 x × d 2 x . To get the expression of A, consider two parameters that are in the same residual unit but different weight matrices, i.e. w 1 = w r,2 i1,j1 , w 2 = w r,1 i2,j2 . From Lemma 2, we conclude the translation invariance and that all blocks marked in orange (slash) (resp. blue (chessboard)) are the same. Given that the Hessian is symmetric, the blocks marked in blue and orange are transposes of each other, and thus it can be directly written as Equation ( 11).

If j 1 = i 2 , we have

EQUATION

Else, we have A (j1-1)dx+i1,(j2-1)dx+i2 = 0.

Noting that A (j1-1)dx+i1,(j2-1)dx+i2 in fact only depends on the two indices i 1 , j 2 (with a small difference depending on whether j 1 = i 2 ), we make a d x × d x matrix with rows indexed by i 1 and columns indexed by j 2 , and the entry at (i 1 , j 2 ) equal to A (j1-1)dx+i1,(j2-1)dx+i2 . Apparently, this matrix is equal to σ mid (0)σ post (0)(Σ Xσpre(X) -Σ Y σpre(X) ) when j 1 = i 2 , and equal to the zero matrix when j 1 = i 2 .

To simplify the expression of A, we rearrange the columns of A by a permutation matrix, i.e.

EQUATION

where P ij = 1 if and only if i = ((j -1) mod d x )d x + j dx . Basically it permutes the i-th column of A to the j-th column. X) . . .

Then we have:

A = σ mid (0)σ post (0)    Σ Xσpre(X) -Σ Y σpre(

EQUATION

So the eigenvalues of H becomes eigs(H) = ±σ mid (0)σ post (0) eigs((Σ Xσpre(X) -Σ Y σpre(X) ) T (Σ Xσpre(X) -Σ Y σpre(X) )),

which leads to Equation (12).

3. Now consider the Hessian in the n = 1 case. Using Lemma 2, the form of Hessian can be directly written as Equation (13).

To get the expressions of A and B in σ pre (x) = σ post (x) = x case, consider two parameters that are in the same residual units, i.e. w 1 = w r,1 i1,j1 , w 2 = w r,1 i2,j2 . We have

B (j1-1)dx+i1,(j2-1)dx+i2 = ∂ 2 L ∂w 1 ∂w 2 w=0 (23) = 1 m m µ=1 x µ j1 x µ j2 i 1 = i 2 0 i 1 = i 2 (24)

Rearrange the order of variables using P , we have

EQUATION

)

Then consider two parameters that are in different residual units, i.e. w 1 = w r1,1 i1,j1 , w 2 = w r2,1 i2,j2 , r 1 > r 2 . We have A (j1-1)dx+i1,(j2-1)dx+i2 = ∂ 2 L ∂w 1 ∂w 2 w=0

(26)

EQUATION

In the same way, we can rewrite A as

EQUATION

B EXPERIMENT SETUP:

We took the experiments on whitened versions of MNIST. 10 greatest principal components are kept for the dataset inputs. The dataset outputs are represented using one-hot encoding. The network was trained using gradient descent. For every epoch, the Hessians of the networks were calculated using the method proposed in (Bishop, 1992) . As the |λ| min of Hessian is usually very unstable, we calculated |λ|max |λ| (0.1) to represent condition number instead, where |λ| (0.1) is the 10 th percentile of the absolute values of eigenvalues.

As pre, mid or post positions are not defined in linear networks without shortcuts, when comparing Xavier or orthogonal initialized linear networks to 2-shortcut networks, we added ReLUs at the same positions in linear networks as in 2-shortcuts networks.

:

"But if F has only a single layer, Eqn.(1) is similar to a linear layer: y = W1x + x, for which we have not observed advantages. "

& $ & $ '#% $ '#%

& $ '#% $() '#%

FUTURE DIRECTIONSFurther studies should concentrate on the behavior of shortcut connections on convolution networks, as well as the influences of batch normalization units(Ioffe & Szegedy, 2015) in ResNet. Meanwhile, it would be very interesting to extend the insights obtained in this paper to recurrent neural networks such as LSTM(Sainath et al., 2013).

Figures and Tables:

Figure 1: An example of different position for nonlinearities in a residual unit of a 2-shortcut network.

The average Frobenius norms of ResNets of different depths during the training process. The pre-ResNet implementation in https://github.com/facebook/fb.resnet.torch is used. The learning rate is initialized to 0.1, decreased to 0.01 at the 81 st epoch (marked with circles) and decreased to 0.001 at the 122 nd epoch (marked with triangles). Each model is trained for 200 epochs.

The average Frobenius norms of 2-shortcut networks of different depths during the training process when zero initialized. Left: Without nonlinearities. Right: With ReLUs at mid positions.

Figure 4: Equivalents of two extremes of n-shortcut linear networks. 1-shortcut linear networks are equivalent to linear networks with identity initialization, while skip-all shortcuts will only change the effective dataset outputs.

Figure 5: Initial condition numbers of Hessians for different linear networks as the depths of the networks increase. Means and standard deviations are estimated based on 10 runs.

Maxima and 10 th percentiles of absolute values of eigenvalues at different losses when the depth is 16. Eigenvalues at different losses are calculated using linear interpolation.

Figure 10: Left: Optimal Final losses of different networks with ReLUs in mid positions. Right:Corresponding optimal learning rates. Note that as it is hard to compute the minimum losses with ReLUs, we plot the log 10 (final loss) instead of log 10 (final lossoptimal loss). When the depth is 64, the final losses of Xavier-ReLU and orthogonal-ReLU with different learning rates are basically the same, so the optimal learning rates are omitted as they are very unstable.

Figure11: The Hessian in n = 2 case. It follows from Lemma 1 that only off-diagonal subblocks in each diagonal block, i.e., the blocks marked in orange (slash) and blue (chessboard), are non-zero. From Lemma 2, we conclude the translation invariance and that all blocks marked in orange (slash) (resp. blue (chessboard)) are the same. Given that the Hessian is symmetric, the blocks marked in blue and orange are transposes of each other, and thus it can be directly written as Equation (11).

Peer review (iclr2017):
This paper studies the optimization issue of linear ResNet, and shows mathematically that for 2-shortcuts and zero initialization, the Hessian has condition number independent of depth. I skimmed through the proof but have not checked them carefully. This result is a nice observation for training deep linear networks. But I do not think the paper has fully resolved the linear vs nonlinear issue. Some question: 1. Though the revision has added some results using ReLU units, it seems it is only added to the mid positions of the network (sec 5.3), is this how it is typically done in ResNet? Moreover, ReLU is not differentiable at zero point, which does not satisfy the condition you had in Theorem 1. Why not use differentiable activations like sigmoid or tanh? 2. From equation (22) in the appendix, it seems for nonlinear activations, the condition number depends on the derivative sigma^prime at 0. Therefore, if we use tanh which has derivative 1 at zero, the condition number is the same for linear and tanh activations. But this probably is not enough to explain the bit difference in performance or optimization for linear and nonlinear networks, or how the situations evolve after learning the 0 point. 3. As for the success of ResNet (or convnets in general) in computer vision, I believe there are more types of nonlinearity such as pooling? Can the result here generalizes to pooling as well? Minor: - sec 1 last paragraph, low approximation error typically means more powerful model class and better training error, but not necessarily better test error - sec 4.1 what do you mean by *zero initialization with small random perturbations*? why not exactly zero initialization, how large is the random perturbation?