Title: On skip connections and normalisation layers in deep optimisation
Venue: NeurIPS 2023
Accepted: yes
Keywords: optimisation

optimization

skip

connection

normalisation

normalization

deep

learning

polyak

lojasiewicz

lipschitz
Abstract: We introduce a general theoretical framework, designed for the study of gradient optimisation of deep neural networks, that encompasses ubiquitous architecture choices including batch normalisation, weight normalisation and skip connections.  Our framework determines the curvature and regularity properties of multilayer loss landscapes in terms of their constituent layers, thereby elucidating the roles played by normalisation layers and skip connections in globalising these properties. We then demonstrate the utility of this framework in two respects. First, we give the only proof of which we are aware that a class of deep neural networks can be trained using gradient descent to global optima even when such optima only exist at infinity, as is the case for the cross-entropy cost.  Second, we identify a novel causal mechanism by which skip connections accelerate training, which we verify predictively with ResNets on MNIST, CIFAR10, CIFAR100 and ImageNet.

On skip connections and normalisation layers in deep optimisation

Abstract

We introduce a general theoretical framework, designed for the study of gradient optimisation of deep neural networks, that encompasses ubiquitous architecture choices including batch normalisation, weight normalisation and skip connections. Our framework determines the curvature and regularity properties of multilayer loss landscapes in terms of their constituent layers, thereby elucidating the roles played by normalisation layers and skip connections in globalising these properties. We then demonstrate the utility of this framework in two respects. First, we give the only proof of which we are aware that a class of deep neural networks can be trained using gradient descent to global optima even when such optima only exist at infinity, as is the case for the cross-entropy cost. Second, we identify a novel causal mechanism by which skip connections accelerate training, which we verify predictively with ResNets on MNIST, CIFAR10, CIFAR100 and ImageNet.

1 Introduction

Deep, overparameterised neural networks are efficiently trainable to global optima using simple first order methods. That this is true is immensely surprising from a theoretical perspective: modern datasets and deep neural network architectures are so complex and irregular that they are essentially opaque from the perspective of classical (convex) optimisation theory. A recent surge in inspired theoretical works [22, 14, 13, 1, 52, 51, 36, 28, 29, 33, 32] has elucidated this phenomenon, showing linear convergence of gradient descent on certain classes of neural networks, with certain cost functions, to global optima. The formal principles underlying these works are identical. By taking width sufficiently large, one guarantees uniform bounds on curvature (via a Lipschitz gradientstype property) and regularity (via a Polyak-Łojasiewicz-type inequality) in a neighbourhood of initialisation. Convergence to a global optimum in that neighbourhood then follows from a wellknown chain of estimates [23].

Despite significant progress, the theory of deep learning optimisation extant in the literature presents at least three significant shortcomings:

It lacks a formal framework in which to compare common practical architecture choices . Indeed, none of the aforementioned works consider the impact of ubiquitous (weight/batch) normalisation layers. Moreover, where common architectural modifications such as skip connections are studied, it is unclear exactly what impact they have on optimisation. For instance, while in [13] it is shown that skip connections enable convergence with width polynomial in the number of layers, as compared with exponential width for chain networks, in [1] polynomial width is shown to be sufficient for convergence of both chain and residual networks.

It lacks theoretical flexibility . The consistent use of uniform curvature and regularity bounds are insufficiently flexible to enable optimisation guarantees too far away from initialisation, where the local, uniform bounds used in previous theory no longer hold. In particular, proving globally optimal convergence for deep neural nets with the cross-entropy cost was (until now) an open problem [5].

It lacks practical utility . Although it is presently unreasonable to demand quantitatively predictive bounds on practical performance, existing optimisation theory has been largely unable to inform architecture design even qualitatively. This is in part due to the first item, since practical architectures typically differ substantially from those considered for theoretical purposes.

Our purpose in this article is to take a step in addressing these shortcomings. Specifically:

We provide a formal framework, inspired by [45], for the study of multilayer optimisation. Our framework is sufficiently general to include all commonly used neural network layers, and contains formal results relating the curvature and regularity properties of multilayer loss landscapes to those of their constituent layers. As instances, we prove novel results on the global curvature and regularity properties enabled by normalisation layers and skip connections respectively, in contrast to the local bounds provided in previous work.

Using these novel, global bounds, we identify a class of weight-normalised residual networks for which, given a linear independence assumption on the data, gradient descent can be provably trained to a global optimum arbitrarily far away from initialisation. From a regularity perspective, our analysis is strictly more flexible than the uniform analysis considered in previous works, and in particular solves the open problem of proving global optimality for the training of deep nets with the cross-entropy cost.

Using our theoretical insight that skip connections aid loss regularity, we conduct a systematic empirical analysis of singular value distributions of layer Jacobians for practical layers. We are thereby able to predict that simple modifications to the classic ResNet architecture [20] will improve training speed. We verify our predictions on MNIST, CIFAR10, CIFAR100 and ImageNet.

2 Background

In this section we give a summary of the principles underlying recent theoretical advances in neural network optimisation. We discuss related works after this summary for greater clarity.

2.1 Smoothness and the PŁ-inequality

Gradient descent on a possibly non-convex function ℓ : R p → R ≥ 0 can be guaranteed to converge to a global optimum by insisting that ℓ have Lipschitz gradients and satisfy the Polyak-Łojasiewicz inequality . We recall these well-known properties here for convenience.

Definition 2.1. Let β > 0 . A continuously differentiable function ℓ : R p → R is said to have β -Lipschitz gradients , or is said to be β -smooth over a set S ⊂ R p if the vector field ∇ ℓ : R p → R p is β -Lipschitz. If S is convex, ℓ having β -Lipschitz gradients implies that the inequality

holds for all θ 1 , θ 2 ∈ S .

The β -smoothness of ℓ over S can be thought of as a uniform bound on the curvature of ℓ over S : if ℓ is twice continuously differentiable, then it has Lipschitz gradients over any compact set K with (possibly loose) Lipschitz constant given by

where D 2 ℓ is the Hessian and ∥ · ∥ denotes any matrix norm.

Definition 2.2. Let µ > 0 . A differentiable function ℓ : R p → R ≥ 0 is said to satisfy the µ -PolyakŁojasiewicz inequality , or is said to be µ -PŁ over a set S ⊂ R p if

for all θ ∈ S .

The PŁ condition on ℓ over S is a uniform guarantee of regularity , which implies that all critical points of ℓ over S are S -global minima; however, such a function need not be convex . Synthesising these definitions leads easily to the following result (cf. Theorem 1 of [23]).

Theorem 2.3. Let ℓ : R p → R ≥ 0 be a continuously differentiable function that is β -smooth and µ -PŁ over a convex set S . Suppose that θ 0 ∈ S and let { θ t } ∞ t =0 be the trajectory taken by gradient descent, with step size η < 2 β -1 , starting at θ 0 . If { θ t } ∞ t =0 ⊂ S , then ℓ ( θ t ) converges to an S -global minimum ℓ ∗ at a linear rate:

for all t ∈ N

Essentially, while the Lipschitz constant of the gradients controls whether or not gradient descent with a given step size can be guaranteed to decrease the loss at each step, the PŁ constant determines by how much the loss will decrease. These ideas can be applied to the optimisation of deep neural nets as follows.

2.2 Application to model optimisation

The above theory can be applied to parameterised models in the following fashion. Let f : R p × R d 0 → R d L be a differentiable, R p -parameterised family of functions R d 0 → R d L (in later sections, L will denote the number of layers of a deep neural network). Given N training data { ( x i , y i ) } N i =1 ⊂ R d 0 × R d L , let F : R p → R d L × N be the corresponding parameter-function map defined by

Any differentiable cost function c : R d L × R d L → R ≥ 0 , convex in the first variable, extends to a differentiable, convex function γ : R d L × N → R ≥ 0 defined by

and one is then concerned with the optimisation of the composite ℓ := γ ◦ F : R p → R ≥ 0 via gradient descent.

To apply Theorem 2.3, one needs to determine the smoothness and regularity properties of ℓ . By the chain rule, the former can be determined given sufficient conditions on the derivatives Dγ ◦ F and DF (cf. Lemma 2 of [45]). The latter can be bounded by Lemma 3 of [45], which we recall below and prove in the appendix for the reader's convenience.

Theorem 2.4. Let S ⊂ R p be a set. Suppose that γ : R d L × N → R ≥ 0 is µ -PŁ over F ( S ) with minimum γ ∗ S . Let λ ( DF ( θ )) denote the smallest eigenvalue of DF ( θ ) DF ( θ ) T . Then

for all

Note that Theorem 2.4 is vacuous ( λ ( DF ( θ )) = 0 for all θ ) unless in the overparameterised regime ( p ≥ d L N ). Even in this regime, however, Theorem 2.4 does not imply that ℓ is PŁ unless λ ( θ ) can be uniformly lower bounded by a positive constant over S . Although universally utilised in previous literature, such a uniform lower bound will not be possible in our global analysis, and our convergence theorem does not follow from Theorem 2.3, in contrast to previous work. Our theorem requires additional argumentation, which we believe may be of independent utility.

3 Related works

Convergence theorems for deep linear networks with the square cost are considered in [4, 2]. In [22], it is proved that the tangent kernel of a multi-layer perceptron (MLP) becomes approximately constant over all of parameter space as width goes to infinity, and is positive-definite for certain data distributions, which by Theorem 2.4 implies that all critical points are global minima. Strictly speaking, however, [22] does not prove convergence of gradient descent: the authors consider only gradient flow, and leave Lipschitz concerns untouched. The papers [14, 13, 1, 52, 51, 36] prove that overparameterized neural nets of varying architectures can be optimised to global minima close to initialisation by assuming sufficient width of several layers. While [1] does consider the cross-entropy cost, convergence to a global optimum is not proved: it is instead shown that perfect classification accuracy can be achieved close to initialisation during training. Improvements on these works have been made in [33, 32, 7], wherein large width is required of only a single layer.

It is identified in [28] that linearity of the final layer is key in establishing the approximate constancy of the tangent kernel for wide networks that was used in [14, 13]. By making explicit the implicit use of the PŁ condition present in previous works [14, 13, 1, 52, 51, 36], [29] proves a convergence theorem even with nonlinear output layers. The theory explicated in [29] is formalised and generalised in [45]. A key weakness of all of the works mentioned thus far (bar the purely formal [45]) is that their hypotheses imply that optimisation trajectories are always close to initialisation. Without this, there is no obvious way to guarantee the PŁ-inequality along the optimisation trajectory, and hence no way to guarantee one does not converge to a suboptimal critical point. However such training is not possible with the cross-entropy cost, whose global minima only exist at infinity. There is also evidence to suggest that such training must be avoided for state-of-the-art test performance [8, 15, 26]. In contrast, our theory gives convergence guarantees even for trajectories that travel arbitrarily far from initialisation, and is the only work of which we are aware that can make this claim.

Among the tools that make our theory work are skip connections [20] and weight normalisation [41]. The smoothness properties of normalisation schemes have previously been studied [42, 40], however they only give pointwise estimates comparing normalised to non-normalised layers, and do not provide a global analysis of Lipschitz properties as we do. The regularising effect of skip connections on the loss landscape has previously been studied in [35], however this study is not tightly linked to optimisation theory. Skip connections have also been shown to enable the interpretation of a neural network as coordinate transformations of data manifolds [19]. Mean field analyses of skip connections have been conducted in [49, 30] which necessitate large width; our own analysis does not. A similarly general framework to that which we supply is given in [47, 48]; while both encapsulate all presently used architectures, that of [47, 48] is designed for the study of infinite-width tangent kernels, while ours is designed specifically for optimisation theory. Our empirical singular value analysis of skip connections complements existing theoretical work using random matrix theory [18, 37, 39, 34, 16]. These works have not yet considered the shifting effect of skip connections on layer Jacobians that we observe empirically.

Our theory also links nicely to the intuitive notions of gradient propagation [20] and dynamical isometry already present in the literature. In tying Jacobian singular values rigorously to loss regularity in the sense of the Polyak-Lojasiewicz inequality, our theory provides a new link between dynamical isometry and optimisation theory [43, 38, 46]: specifically dynamical isometry ensures better PŁ conditioning and therefore faster and more reliable convergence to global minima. In linking this productive section of the literature to optimisation theory, our work may open up new possibilities for convergence proofs in the optimisation theory of deep networks. We leave further exploration of this topic to future work.

Due to this relationship with the notion of dynamical isometry, our work also provides optimisationtheoretic support for the empirical analyses of [24, 6] that study the importance of layerwise dynamical isometry for trainability and neural architecture search [27, 44, 25]. Recent work on deep kernel shaping shows via careful tuning of initialisation and activation functions that while skip connections and normalisation layers may be sufficient for good trainability, they are not necessary [31, 50]. Other recent work has also shown benefits to inference performance by removing skip connections from the trained model using a parameter transformation [11] or by removing them from the model altogether and incorporating them only into the optimiser [12].

Finally, a line of work has recently emerged on training in the more realistic, large learning rate regime known as 'edge of stability" [9, 3, 10]. This intriguing line of work diverges from ours, and its integration into the framework we present is a promising future research direction.

4 Formal framework

In this section we will define our theoretical framework. We will use R n × m to denote the space of n × m matrices, and vectorise rows first. We use Id m to denote the m × m identity matrix, and 1 n to denote the vector of ones in R n . We use ⊗ to denote the Kronecker (tensor) product of matrices, and given a vector v ∈ R n we use diag ( v ) to denote the n × n matrix whose leading diagonal is v . Given a matrix A ∈ R n × m , and a seminorm ∥ · ∥ on R m , ∥ A ∥ row will be used to denote the vector of ∥ · ∥ -seminorms of each row of A . The smallest singular value of A is denoted σ ( A ) , and the smallest eigenvalue of AA T denoted λ ( A ) . Full proofs of all of our results can be found in the appendix.

Our theory is derived from the following formal generalisation of a deep neural network.

Definition 4.1. By a multilayer parameterised system (MPS) we mean a family { f l : R p l × R d l -1 × N → R d l × N } L l =1 of functions. Given a data matrix X ∈ R d 0 × N , we denote by F : R ∑ L i =1 p i → R d L × N the parameter-function map 2 defined by

Definition 4.1 is sufficiently general to encompass all presently used neural network layers, but we will assume without further comment from here on in that all layers are continuously differentiable . Before we give examples, we record the following result giving the form of the derivative of the parameter-function map, which follows easily from the chain rule. It will play a key role in the analysis of the following subsections.

Proposition 4.2. Let { f l : R p l × R d l -1 × N → R d l × N } L l =1 be a MPS, and X ∈ R d 0 × N a data matrix. For 1 ≤ l ≤ L , denote the derivatives of f l with respect to the R p l and R d l -1 variables by Df l and Jf l respectively, and let f <l ( ⃗ θ, X ) denote the composite

The derivative DF of the associated parameter-function map is given by

where for 1 ≤ l < L , D θ l F ( ⃗ θ ) is given by the formula

with the product taken so that indices are arranged in descending order from left to right.

All common differentiable neural network layers fit into this framework; we record some examples in detail in the appendix. We will see in the next section that insofar as one wishes to guarantee global smoothness, the usual parameterisation of affine layers is poor, although this defect can be ameliorated to differing extents by normalisation strategies.

4.1 Smoothness

In this subsection, we give sufficient conditions for the derivative of the parameter-function map of a MPS to be bounded and Lipschitz. We are thereby able to give sufficient conditions for any associated loss function to have Lipschitz gradients on its sublevel sets. We begin with a formal proposition that describes the Lipschitz properties of the derivative of the parameter-function map of a MPS in terms of those of its constituent layers.

Proposition 4.3. Let { f l : R p l × R d l -1 × N → R d l × N } L l =1 , and let { S l ⊂ R p l } L l =1 be subsets of the parameter spaces. Suppose that for each bounded set B l ⊂ R d l -1 × N , the maps f l , Df l and Jf l are all bounded and Lipschitz on S l × B l . Then for any data matrix X ∈ R d 0 × N , the derivative DF of the associated parameter-function map F : R p → R d L × N is bounded and Lipschitz on S := ∏ L j =1 S j ⊂ R p .

Proposition 4.3 has the following immediate corollary, whose proof follows easily from Lemma B.1.

Corollary 4.4. Let c : R d L → R be any cost function whose gradient is bounded and Lipschitz on its sublevel sets Z α := { z ∈ R d L : c ( z ) ≤ α } . If { f l : R p l × R d l -1 × N → R d l × N } L l =1 is any MPS satisfying the hypotheses of Proposition 4.3 and X ∈ R d 0 × N , then the associated loss function ℓ := γ ◦ F has Lipschitz gradients over S := ∏ L l =1 S l .

The most ubiquitous cost functions presently in use (the mean square error and cross entropy functions) satisfy the hypotheses of Corollary 4.4. We now turn to an analysis of common layer types in deep neural networks and indicate to what extent they satisfy the hypotheses of Proposition 4.3.

Theorem 4.5. Fix ϵ > 0 . The following layers satisfy the hypotheses of Proposition 4.3 over all of parameter space.

Continuously differentiable nonlinearities.

Bias-free ϵ -weight-normalised or ϵ -entry-normalised affine layers 3 .

Any residual block whose branch is a composite of any of the above layer types.

Consequently, the loss function of any neural network composed of layers as above, trained with a cost function satisfying the hypotheses of Corollary 4.4, has globally Lipschitz gradients along any sublevel set.

The proof we give of Theorem 4.5 also considers composites bn ◦ aff of batch norm layers with affine layers. Such composites satisfy the hypotheses of Proposition 4.3 only over sets in data-space R d × N which consist of matrices with nondegenerate covariance. Since such sets are generic (probability 1 with respect to any probability measure that is absolutely continuous with respect to Lebesgue measure), batch norm layers satisfy the hypotheses of Proposition 4.3 with high probability over random initialisation.

Theorem 4.5 says that normalisation of parameters enables global analysis of the loss, while standard affine layers, due to their unboundedness, are well-suited to analysis only over bounded sets in parameter space.

4.2 Regularity

Having examined the Lipschitz properties of MPS and given examples of layers with global Lipschitz properties, let us now do the same for the regularity properties of MPS. Formally one has the following simple result.

Proposition 4.6. Let { f l : R p l × R d l -1 × N → R d l × N } L i =1 be a MPS and X ∈ R d 0 × N a data matrix. Then

is a lower bound for λ ( DF ( ⃗ θ ) ) .

Proposition 4.6 tells us that to guarantee good regularity, it suffices to guarantee good regularity of the constituent layers. In fact, since Equation (12) is a sum of non-negative terms, it suffices merely to guarantee good regularity of the parameter-derivative of the first layer 4 , and of the input-output Jacobians of every subsequent layer. Our next theorem says that residual networks with appropriately normalised branches suffice for this.

Theorem 4.7. Let { g l : R p l × R d l -1 × N → R d l × N } L i =1 be a MPS and X ∈ R d 0 × N a data matrix for which the following hold:

d l -1 ≥ d l , and ∥ Jg l ( θ l , Z ) ∥ 2 < 1 for all θ l ∈ R p l , Z ∈ R d l -1 × N and l ≥ 2 .

N ≤ d 0 , X is full rank, p 1 ≥ d 1 d 0 and f 1 : R p 1 × R d 0 × N → R d 1 × N is a P -parameterised affine layer, for which DP ( θ 1 ) is full rank for all θ 1 ∈ R p 1 .

For any sequence { I l : R d l -1 × N → R d l × N } L l =2 of linear maps whose singular values are all equal to 1, define a new MPS { f l } L l =1 by f 1 := g 1 , and f l ( θ l , X ) := I l X + g l ( θ l , X ) . Let F : R p → R d L × N be the parameter-function map associated to { f l } L l =1 and X . Then λ ( DF ( ⃗ θ )) > 0 (but not uniformly so) for all ⃗ θ ∈ R p .

In the next and final subsection we will synthesise Theorem 4.5 and Theorem 4.7 into our main result: a global convergence theorem for gradient descent on appropriately normalised residual networks.

5 Main theorem

We will consider normalised residual networks , which are MPS of the following form. The first layer is an ϵ -entry-normalised, bias-free affine layer f 1 = affen : R d 1 × d 0 × R d 0 × N → R d 1 × N (cf. Example A.1). Every subsequent layer is a residual block

Here, for all l ≥ 2 , we demand that d l -1 ≥ d l , I i : R d i -1 × N → R d i × N is some linear map with all singular values equal to 1, and the residual branch g ( θ i , X ) is a composite of weight- or entrynormalised, bias-free 5 affine layers aff P (cf. Example A.1), rescaled so that ∥ P ( w ) ∥ 2 < 1 uniformly for all parameters w , and elementwise nonlinearities Φ for which ∥ Dϕ ∥ ≤ 1 everywhere (Example A.2). These hypotheses ensure that Theorem 4.7 holds for { f i } L i =1 and X : see the Appendix for a full proof.

We emphasise again that our main theorem below does not follow from the usual argumentation using smoothness and the PŁ inequality, due to the lack of a uniform PŁ bound. Due to the novelty of our technique, which we believe may be of wider utility where uniform regularity bounds are unavailable, we include an idea of the proof below.

Theorem 5.1. Let { f i } L i =1 be a normalised residual network; X a data matrix of linearly independent data, with labels Y ; and c any continuously differentiable, convex cost function. Then there exists a learning rate η > 0 such that gradient descent on the associated loss function converges from any initialisation to a global minimum.

Idea of proof. One begins by showing that Theorems 4.5 and 4.7 apply to give globally β -Lipschitz gradients and a positive smallest eigenvalue of the tangent kernel at all points in parameter space. Thus for learning rate η < 2 β -1 , there exists a positive sequence ( µ t = µλ ( DF ( θ t ))) t ∈ N (see Theorem 2.4) such that ∥∇ ℓ t ∥ 2 ≥ µ t ℓ t , for which the loss iterates ℓ t therefore obey

where α = η (1 -2 β -1 η ) > 0 . To show global convergence one must show that ∏ ∞ t =0 (1 -µ t α ) = 0 .

If µ t can be uniformly lower bounded (e.g. for the square cost) then Theorem 2.3 applies to give convergence as in all previous works. However, µ t cannot be uniformly lower bounded in general (e.g. for the cross-entropy cost). We attain the general result by showing that, despite admitting no non-trivial lower bound in general, µ t can always be guaranteed to vanish sufficiently slowly that global convergence is assured.

We conclude this section by noting that practical deep learning problems typically do not satisfy the hypotheses of Theorem 5.1: frequently there are more training data than input dimensions (such as for MNIST and CIFAR), and many layers are not normalised or skip-connected. Moreover our Lipschitz bounds are worst-case, and will generally lead to learning rates much smaller than are used in practice. Our strong hypotheses are what enable a convergence guarantee from any initialisation, whereas in practical settings initialisation is of key importance. Despite the impracticality of Theorem 5.1, in the next section we show that the ideas that enable its proof nonetheless have practical implications.

6 Practical implications

Our main theorem is difficult to test directly, as it concerns only worst-case behaviour which is typically avoided in practical networks which do not satisfy its hypotheses. However, our framework more broadly nonetheless yields practical insights. Informally, Theorem 4.7 gives conditions under which:

skip connections aid optimisation by improving loss regularity.

In this section, we conduct an empirical analysis to demonstrate that this insight holds true even in practical settings , and thereby obtain a novel, causal understanding of the benefits of skip connections in practice. With this causal mechanism in hand, we recommend simple architecture changes to practical ResNets that consistently (albeit modestly) improve convergence speed as predicted by theory. All code is available at https://github.com/lemacdonald/skip-connections-normalisation/ .

6.1 Singular value distributions

Let us again recall the setting ℓ = γ ◦ F of Theorem 2.4. In the overparameterised setting, the smallest singular value of DF gives a pessimistic lower bound on the ratio ∥∇ ℓ ∥ 2 / ( ℓ -ℓ ∗ ) , and hence a pessimistic lower bound on training speed (cf. Theorems 2.4, 2.3). Indeed, this lower bound is only attained when the vector ∇ γ perfectly aligns with the smallest singular subspace of DF : a probabilistically unlikely occurence. In general, since ∇ ℓ = DF T ∇ γ , the entire singular value distribution of DF at a given parameter will play a role in determining the ratio ∥∇ ℓ ∥ 2 / ( ℓ -ℓ ∗ ) , and hence training speed.

Now DF is partly determined by products of layer Jacobians (cf. Proposition 4.2). As such, the distribution of singular values of DF is determined in part by the distribution of singular values of such layer Jacobian products, which are themselves determined by the singular value distributions of each of the individual layer Jacobians.

0

1

2

Sing. Val.

0

20

40

Frequency

0

1

2

Sing. Val.

0

20

40

Frequency

In particular, if through some architectural intervention each layer Jacobian could have its singular value distribution shifted upwards, we would expect the singular value distribution of DF to be shifted upwards, too. Our argument in the previous paragraph then suggests that such an intervention will result in faster training, provided of course that the upwards-shifting is not so large as to cause exploding gradients.

Figure 1 shows in the linear setting that a skip connection constitutes precisely such an intervention. We hypothesise that this continues to hold even in the nonlinear setting.

Hypothesis 6.1. The addition of a deterministic skip connection, all of whose singular values are 1, across a composite of possibly nonlinear random layers, shifts upwards the singular value distribution of the corresponding composite Jacobian 6 , thereby improving convergence speed at least initially.

We test Hypothesis 6.1 in the next subsections.

6.2 MNIST

Recall that the ResNet architecture [20] consists in part of a composite of residual blocks

where A is either the identity transformation, in the case when the dimension of the output of g ( θ, X ) is the same as the dimension of its input; or a randomly initialised 1x1 convolution otherwise.

0

2

4

6

Sing. Val.

0

10

20

30

40

Frequency

(a) Chain. Mean: 2.36

0

2

4

6

Sing. Val.

0

10

20

30

40

Frequency

(b) Res. Mean: 2.91

0

2

4

6

Sing. Val.

0

10

20

30

40

Frequency

(c) ResAvg. Mean: 3.17

0

10

20

Epoch

0.00

1.25

2.50

Loss

(a)

(b)

(c)

(d) Loss

0

20

40

60

80

Epoch

10

2

10

1

10 0

Loss

orig.

mod.

(a) ResNet18, CIFAR10

0

20

40

60

80

Epoch

10

2

10

1

10 0

Loss

orig.

mod.

(b) ResNet18, CIFAR100

0

20

40

60

80

Epoch

1

2

3

4

5

Loss

orig.

mod.

(c) ResNet50, ImageNet

Hypothesis 6.1 predicts that the additions of the identity skip connections will shift upwards the singular value distributions of composite layer Jacobians relative to the equivalent chain network, thus improving convergence speed. It also predicts that replacing 1x1 convolutional skip connections A with I + A , where I is deterministic with all singular values equal to 1, will do the same.

We first test this hypothesis by doing gradient descent, with a learning rate of 0.1, on 32 randomly chosen data points from the MNIST training set 7 . We compare three models, all with identical initialisations for each trial run using the default PyTorch initialisation:

(Chain) A batch-normed convolutional chain network, with six convolutional layers.

(Res) The same as (1), but with convolutional layers 2-3 and 5-6 grouped into two residual blocks, and with an additional 1x1 convolution as the skip connection in the second residual block, as in Equation (14).

(ResAvg) The same as (2), but with the 1x1 convolutional skip connection of Equation (14) replaced by

where I is an average pool, rescaled to have all singular values equal to 1.

Hypothesis 6.1 predicts that the singular value distributions of the composite layer Jacobians will be more positively shifted going down the list, resulting in faster convergence. This is indeed what we observe (Figure 2).

6.3 CIFAR and ImageNet

We now test Hypothesis 6.1 on CIFAR and ImageNet. We replace all of the convolution-skip connections in the ResNet architecture [20] with sum(average pool, convolution)-skip connections as in Equation (15) above, leaving all else unchanged. Hypothesis 6.1 predicts that these modifications will improve training speed, which we verify using default PyTorch initialisation schemes.

Wetrained PreAct-ResNet18 on CIFAR10/100 and PreAct-ResNet50 [20] on ImageNet using standard training regimes (details can be found in the appendix), performing respectively 10 and 3 trial runs

on CIFAR and ImageNet. We performed the experiments at a range of different learning rates. We have included figures for the best performing learning rates on the original model (measured by loss value averaged over the final epoch) in Figure 3, with additional plots and validation accuracies in the appendix. Validation accuracies were not statistically significantly different between the two models on CIFAR10/100. Although the modified version had statistically significantly better validation accuracy in the ImageNet experiment, we believe this is only due to the faster convergence, as the training scheme was not sufficient for the model to fully converge.

7 Discussion

Our work suggests some open research problems. First, the recently-developed edge of stability theory [9] could be used in place of our Lipschitz bounds to more realistically characterise training of practical nets with large learning rates. Second, like in pervious works [7], the heavy-lifting for our PŁ-type bounds is all done by the parameter-derivative of a single layer, and the bounds would be significantly improved by an analysis that considers all layers. Third, extension of the theory to SGD is desirable. Fourth, the dependence of Hypothesis 6.1 on weight variance should be investigated. Fifth, our empirical results on the impact of skip connections on singular value distributions suggests future work using random matrix theory [18].

Beyond these specifics, our formal framework provides a setting in which all neural network layers can be analysed in terms of their effect on the key loss landscape properties of smoothness and regularity, and is the first to demonstrate that a uniform bound on regularity is not necessary to prove convergence. We hope the tools we provide in this paper will be of use in extending deep learning optimisation theory to more practical settings than has so far been the case.

8 Conclusion

We gave a formal theoretical framework for studying the optimisation of multilayer systems. We used the framework to give the first proof that a class of deep neural networks can be trained by gradient descent even to global optima at infinity. Our theory generates the novel insight that skip connections aid optimisation speed by improving loss regularity, which we verified empirically using practical datasets and architectures.

References

Z. Allen-Zhu, Y. Li, and Z. Song. A Convergence Theory for Deep Learning via OverParameterization. In ICML , pages 242-252, 2019.

S. Arora, N. Cohen, N. Golowich, and W. Hu. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. In ICLR , 2019.

S. Arora, Z. Li, and A. Panigrahi. Understanding Gradient Descent on Edge of Stability in Deep Learning. In ICML , 2022.

P. L. Bartlett, D. P. Hembold, and P. M. Long. Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks. In ICML , 2018.

M. Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica , 30:203-248, 2021.

K. Bhardwaj, G. Li, and R. Marculescu. How does topology influence gradient propagation and model performance of deep networks with densenet-type skip connections? In CVPR , 2021.

S. Bombari, M. H. Amani, and M. Mondelli. Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization. In NeurIPS , 2022.

L. Chizat, E. Oyallon, and F. Bach. On Lazy Training in Differentiable Programming . In NeurIPS , 2019.

J. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. In ICLR , 2021.

A. Damian, E. Nichani, and J. Lee. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. In NeurIPS , 2022.

X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun. Repvgg: Making vgg-style convnets great again. In CVPR , 2021.

X. Ding, H. Chen, X. Zhang, K. Huang, J. Han, and G. Ding. Re-parameterizing your optimizers rather than architectures. In ICLR , 2023.

S. S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient Descent Finds Global Minima of Deep Neural Networks. In ICML , pages 1675-1685, 2019.

S. S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient Descent Provably Optimizes Overparameterized Neural Networks. In ICLR , 2019.

F. He and T. Liu and D. Tao. Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence . In NeurIPS , 2019.

Z. Fan and Z. Wang. Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks. In NeurIPS , 2020.

E. Hairer, S. Norsett, and G. Wanner. Solving Ordinary Differential Equations I: Nonstiff Problems . Springer, 2008.

B. Hanin and M. Nica. Products of Many Large Random Matrices and Gradients in Deep Neural Networks. Communications in Mathematical Physics , 376:287-322, 2020.

M. Hauser and A. Ray. Principles of riemannian geometry in neural networks. In NeurIPS , 2017.

K. He, X. Zhang, S. Ren, and J. Sun. Identity Mappings in Deep Residual Networks. In ECCV , 2016.

S. Ioffe and C. Szegedy. Batch Normalization: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML , page 448-456, 2015.

A. Jacot, F. Gabriel, and C. Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In NeurIPS , pages 8571-8580, 2018.

H. Karimi, J. Nutini, and M. Schmidt. Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition. In ECML PKDD , pages 795--811, 2016.

N. Lee, T. Ajanthan, S. Gould, and P. H. S. Torr. A signal propagation perspective for pruning neural networks at initialization. In ICLR , 2020.

G. Li, Y. Yang, K. Bhardwaj, and R. Marculescu. Zico: Zero-shot nas via inverse coefficient of variation on gradients. In ICLR , 2023.

Y. Li, C. Wei, and T. Ma. Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks. In NeurIPS , 2019.

M. Lin, P. Wang, Z. Sun, H. Chen, X. Sun, Q. Q, H. Li, and R. Jin. Zen-nas: A zero-shot nas for high-performance deep image recognition. In ICCV , 2021.

C. Liu, L. Zhu, and M. Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. In NeurIPS , 2020.

C. Liu, L. Zhu, and M. Belkin. Loss landscapes and optimization in over-parameterized nonlinear systems and neural networks. Applied and Computational Harmonic Analysis , 59:85-116, 2022.

Y. Lu, C. Ma, Y. Lu, J. Lu, and L. Ying. A Mean Field Analysis Of Deep ResNet And Beyond: Towards Provably Optimization Via Overparameterization From Depth. In ICML , 2020.

J. Martens, A. Ballard, G. Desjardins, G. Swirszcz, V. Dalibard, J. Sohl-Dickstein, and S. S. Schoenholz. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. arXiv:2110.01765, 2021.

Q. Nguyen. On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths. In NeurIPS , 2021.

Q. Nguyen and M. Mondelli. Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology. In NeurIPS , 2020.

Q. Nguyen, M. Mondelli, and G. Montufar. Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks. In ICML , 2021.

A. E. Orhan and X. Pitkow. Skip Connections Eliminate Singularities. In ICLR , 2018.

S. Oymak and M. Soltanolkotabi. Towards moderate overparameterization: global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory , 1:84-105, 2020.

J. Pennington and P. Worah. Nonlinear random matrix theory for deep learning. In NeurIPS , 2017.

J. Pennington, S. S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In NeurIPS , 2017.

J. Pennington, S. Schoenholz, and S. Ganguli. The emergence of spectral universality in deep networks. In AISTATS , 2018.

S. Qiao, H. Wang, C. Liu, W. Shen, and A. Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv:1903.10520, 2019.

T. Salismans and D. Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In NeurIPS , 2016.

S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry. How does batch normalization help optimization? In NeurIPS , 2018.

A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR , 2014.

Z. Sun, M. Lin, X. Sun, Z. Tan, H. Li, and R. Jin. Mae-det: Revisiting maximum entropy principle in zero-shot nas for efficient object detection. In ICML , 2022.

D. Terjék and D. González-Sánchez. A framework for overparameterized learning, 2022. arXiv:2205.13507.

L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In ICML , 2018.

G. Yang. Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes. In NeurIPS , 2019.

G. Yang. Tensor Programs II: Neural Tangent Kernel for Any Architecture . arXiv:2006.14548, 2020.

G. Yang and S. S. Schoenholz. Mean Field Residual Networks: On the Edge of Chaos. In NeurIPS , 2017.

G. Zhang, A. Botev, and J. Martens. Deep learning without shortcuts: Shaping the kernel with tailored rectifiers. In ICLR , 2022.

D. Zou and Q. Gu. An improved analysis of training over-parameterized deep neural networks. In NeurIPS , 2019.

D. Zou, Y. Cao, D. Zhou, and Q. Gu. Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks. Machine Learning , 109:467-492, 2020.

Computational resources: Both the exploratory and final experiments for this paper were conducted using a desktop machine with two Nvidia RTX A6000 GPUs, for a total running time of approximately 150 hours.

A Examples of MPS

All commonly used neural network layers fit into the framework of multilayer parameterised systems (MPS). Here we list the examples of most relevance to our work.

A routine calculation shows that

while

More generally, if P : R p → R d 1 × d 0 denotes any continuously differentiable map, then one obtains a P -parameterised affine layer

where DP ( w ) ∈ R d 0 N × p is the derivative of P at w . Common examples include ϵ -weight normalisation wn ( w ) := ( ϵ + ∥ w ∥ 2 row ) -1 2 w [41] and convolutions, which send convolutional kernels to associated Toeplitz matrices. We will also consider ϵ -entry normalisation en ( w ) := ( ϵ + w 2 ) -1 2 w , with operations applied entrywise.

Example A.2. A (parameter-free) elementwise nonlinearity Φ : R d 0 × N → R d 0 × N defined by a continuously differentiable function ϕ : R → R is given by applying ϕ to every component of a matrix X ∈ R d 0 × N . Extension to the parameterised case is straightforward.

Example A.3. A (parameter-free) batch normalisation (BN) layer bn : R d 0 × N → R d 0 × N is given by the formula

where ϵ > 0 is some fixed hyperparameter and E and σ denote the row-wise mean and standard deviation. The parameterised BN layer from [21], with scaling and bias parameters γ and β respectively, is given simply by postcomposition affdiag ( γ, β, · ) ◦ bn with a diag-parameterised affine layer (Example A.1).

Example A.4. A residual block f : R p × R d 0 × N → R d 1 × N can be defined given any other layer (or composite thereof) g : R p × R d 0 × N → R d 1 × N by the formula

where I : R d 0 × N → R d 1 × N is some linear transformation. In practice, I is frequently the identity map [20]; our main theorem will concern the case where I has all singular values equal to 1.

B Proofs

Proof of Theorem 2.4. Using the fact that Dℓ = ( Dγ ◦ F ) · DF , we compute:

One has and, by the chain rule,

where the first inequality follows from the standard estimate ⟨ v, AA T v ⟩ ≥ λ min ( AA T ) ∥ v ∥ 2 , and the final inequality follows from the fact that γ is µ -PŁ over the set { F ( θ ) : θ ∈ R p } .

Our proof of Proposition 4.3 requires the following standard lemma.

̸

Lemma B.1. Let { g i : R p → R m i × m i -1 } n i =1 be a family of matrix-valued functions. If, with respect to some submultiplicative matrix norm, each g i is bounded by b i and Lipschitz with constant c i on a set S ⊂ R p , then their pointwise matrix product θ ↦→ ∏ n i =1 g i ( θ ) is also bounded and Lipschitz on S , with bound ∏ n i =1 b i and Lipschitz constant ∑ n i =1 c i ( ∏ j = i b j ) .

Proof of Lemma B.1. We prove the lemma by induction. When n = 2 , adding and subtracting a copy of g 1 ( θ ) g 2 ( θ ′ ) and using the triangle inequality implies that ∥ g 1 g 2 ( θ ) -g 1 g 2 ( θ ′ ) ∥ is bounded by

Applying submultiplicativity of the matrix norm and the bounds provided by the b i and c i gives

̸

Now suppose we have the result for n = k . Writing ∏ k +1 i =1 g i as g 1 ∏ k +1 i =2 g i and applying the above argument, the induction hypothesis tells us that ∏ k +1 i =1 g i is indeed bounded by ∏ k +1 i =1 b i and Lipschitz with Lipschitz constant ∑ k +1 i =1 c i ( ∏ j = i b j ) . The result follows.

Proof of Proposition 4.3. By Proposition 4.2, it suffices to show that for each 1 ≤ l ≤ L , the function

is bounded and Lipschitz on S . To show this, we must first prove that each map ⃗ θ ↦→ f <j ( ⃗ θ, X ) is bounded and Lipschitz on S . This we prove by induction.

By hypothesis, ⃗ θ ↦→ f 1 ( ⃗ θ, X ) = f 1 ( θ 1 , X ) is bounded and Lipschitz on S . Suppose now that for j > 1 , one has ⃗ θ ↦→ f <j ( ⃗ θ, X ) bounded and Lipschitz on S . Then the range of S ∋ θ ↦→ f <j ( ⃗ θ, X ) is a bounded subset of R d j × N . By hypothesis on f j , it then follows that θ ↦→ f <j +1 ( ⃗ θ, X ) = f j ( θ j , f <j ( ⃗ θ, X ) ) is bounded and Lipschitz on S .

The hypothesis on the Jf j and Df j now implies that the maps ⃗ θ ↦→ Jf j ( θ j , f <j ( ⃗ θ, X ) ) , l +1 ≤ j ≤ L , and ⃗ θ ↦→ Df l ( θ l , f <l ( ⃗ θ, X ) ) are all bounded and Lipschitz on S . In particular, as a product of bounded and Lipschitz functions, the map given in Equation (24) is also bounded and Lipschitz on S . Therefore DF is bounded and Lipschitz on S .

Proof of Corollary 4.4. By hypothesis, F ( S ) is a bounded subset of R d L × N . Continuity of γ then implies that γ ( F ( S )) is a bounded subset of R , so that F ( S ) is contained in a sublevel set of γ . The result now follows from the hypotheses.

To prove Theorem 4.5 it will be convenient to recall some tensor calculus. If f : R n 1 × n 2 → R m 1 × m 2 is a matrix-valued, differentiable function of a matrix-valued variable, its derivative Df can be regarded as a map R n 1 × n 2 → R m 1 × m 2 × n 1 × n 2 whose components are given by

where 1 ≤ i α ≤ m α and 1 ≤ j α ≤ n α are the indices, α = 1 , 2 . It is easily deduced from the chain rule of ordinary calculus that if f : R n 1 × n 2 → R m 1 × m 2 and g : R m 1 × m 2 → R l 1 × l 2 are differentiable, then g ◦ f is differentiable with derivative ( Dg ◦ f ) · Df : R n 1 × n 2 → R l 1 × l 2 × n 1 × n 2 , where here · denotes contraction over the m 1 × m 2 indices. The following lemmata then follow from routine calculation.

Lemma B.2. Let bn : R d × N → R d × N be an ϵ -batchnorm layer. Then one can write bn = v ◦ m , where v, m : R d × N → R d × N are given respectively by

One has and

with and all second derivatives of m

being zero.

Lemma B.3. Let wn : R d 1 × d 0 → R d 1 × d 0 be an ϵ -weight normalised parameterisation (Example A.1). Then one has

and

Similarly, if en : R d 1 × d 0 → R d 1 × d 0 is an ϵ -entry-normalised parameterisation, then

and

Proof of Theorem 4.5. (1) follows from continuity of the nonlinearity and its derivative, implying boundedness of both over bounded sets in R d × N .

(2) and (3) follow from a similar argument to the following argument for batch norm, which we give following Lemma B.2. Specifically, for the composite f := bn ◦ aff : R d 1 × d 0 × R d 0 × N → R d 1 × N defined by an ϵ -BN layer and an affine layer, we will prove that over any set B ⊂ R d 0 × N consisting of matrices X whose covariance matrix is nondegenerate, one has f , Df and Jf all globally bounded and Lipschitz. Indeed, v (Equation (25)) is clearly globally bounded, while Dv (Equation (27)) is globally bounded, decaying like ∥ Y ∥ -1 row out to infinity, and D 2 v (Equation (28)) is globally bounded, decaying like ∥ Y ∥ -2 row out to infinity. Consequently,

and similarly the derivatives of D ( bn ◦ aff ) and J ( bn ◦ aff ) are all globally bounded over R d 1 × d 0 × B . The hypothesis that B consist of matrices with nondegenerate covariance matrix is needed here because while Jv ◦ m ◦ aff decays like ∥ A ( X -E [ X ]) ∥ -1 row out to infinity, the row-norm ∥ ( A ( X -E [ X ])) i ) ∥ 2 = ( A i ( X -E [ X ])( X -E [ X ]) T ( A i ) T ) = ∥ A i ∥ 2 Cov ( X ) can only be guaranteed to increase with A if Cov ( X ) is nondegenerate. Thus, for instance, without the nondegeneracy hypothesis on Cov ( X ) , A ↦→ J ( bn ◦ aff )( A,X ) grows unbounded like J aff ( A,X ) = A ⊗ Id N in any direction of degeneracy of Cov ( X ) . Nonetheless, with the nondegenerate covariance assumption on elements of B , bn ◦ aff satisfies the hypotheses of Proposition 4.3 over R d 1 × d 0 × B .

(2) and (3) now follow from essentially the same boundedness arguments as for batch norm, using Lemma B.3 in the place of Lemma B.2. However, since the row norms in this case are always defined by the usual Euclidean inner product on row-vectors, as opposed to the possibly degenerate inner product coming from the covariance matrix of the input vectors, one does not require any hypotheses aside from boundedness on the set B . Thus entry- and weight-normalised affine layers satisfy the hypotheses of Proposition 4.3.

Finally, (5) follows from the above arguments. More specifically, if g : R p × R d × N → R d × N is any composite of layers of the above form, then g satisfies the hypotheses of Proposition 4.3. Consequently, so too does the residual block f ( θ, X ) := X + g ( θ, X ) , for which Jf ( θ, X ) = Id d ⊗ Id N + Jg ( θ, X ) and Df ( θ, X ) = Dg ( θ, X ) .

Proof of Proposition 4.6. In the notation of Proposition 4.2, the product DF ( ⃗ θ ) DF ( ⃗ θ ) T is the sum of the positive-semidefinite matrices D θ l F ( ⃗ θ ) D θ l F ( ⃗ θ ) T . Therefore λ ( DF ( ⃗ θ )) ≥ ∑ l λ ( D θ l F ( ⃗ θ )) . The result now follows from the inequality λ ( AB ) ≥ λ ( A ) λ ( B ) applied inductively using Equation (11). Note that λ ( AB ) ≥ λ ( A ) λ ( B ) is either trivial if one or both of A and B have more rows than columns (in which case the right hand side is zero), and follows from the well-known inequality σ ( AB ) ≥ σ ( A ) σ ( B ) for the smallest singular values if both A and B have at least as many columns as rows.

Theorem 4.7 follows from the following two lemmata.

Lemma B.4. Let g : R p × R d 0 × N → R d 1 × N be a layer for which there exists δ > 0 such that ∥ Jg ( θ, X ) ∥ 2 < (1 -δ ) for all θ and X . Let I : R d 0 × N → R d 1 × N be a linear map whose singular values are all equal to 1. Then the residual block f ( θ, X ) := IX + g ( θ, X ) has σ ( Jf ( θ, X )) > δ for all θ and X .

Proof. Observe that for all θ and X .

Lemma B.5. Let P : R p → R d 1 × d 0 be a parameterisation. Then

for all w ∈ R p and X ∈ R d 0 × N .

Proof. Follows from Equation (21) and the inequality σ ( AB ) ≥ σ ( A ) σ ( B ) .

Proof of Theorem 4.7. Hypothesis 1 in Theorem 4.7 says that the residual branches of the f l , l ≥ 2 , satisfy the hypotheses of Lemma B.4, so that σ ( Jf l ( θ l , f <l ( ⃗ θ, X ))) > 0 for all l ≥ 2 . By the assumption that d l -1 ≥ d l , this means that λ ( Jf l ( θ l , f <l ( ⃗ θ, X ))) = σ ( Jf l ( θ l , f <l ( ⃗ θ, X ))) 2 > 0 . On the other hand, hypothesis 2 together with Lemma B.5 implies that λ ( Df 1 ( θ 1 , X )) ≥ σ ( Df 1 ( θ 1 , X )) 2 > 0 . The result now follows from Proposition 4.6.

.

The result then follows from Weyl's inequality: all singular values of I ⊗ Id N are equal to 1, so that

Proof of Theorem 5.1. By Theorem 4.5, all layers satisfy the Hypotheses of Proposition 4.3 and so by Corollary 4.4, the associated loss function is globally Lipschitz, with Lipschitz constant some β > 0 . Take η > 0 to be any number smaller than 2 β -1 ; thus the loss can be guaranteed to be decreasing with every gradient descent step.

We now show that the network satisfies the hypotheses of Theorem 4.7. The dimension constraints in item (1) are encoded directly into the definition of the network, while the operator-norm of each of the residual branches, as products of P ( w ) and D Φ matrices, are globally bounded by 1 by our hypotheses on these factors. For item (2), our data matrix is full-rank since it consists of linearly independent data, while by definition we have p 1 = d 1 d 0 with Df 1 = D affen being everywhere full-rank since ϵ -entry-normalisation is a diffeomorphism onto its image for any ϵ > 0 . Its hypotheses being satisfied by our weight-normalised residual network, Theorem 4.7 implies the parameter-function map F associated to { f l } L l =1 and X satisfies λ ( DF ( ⃗ θ )) = σ ( DF ( ⃗ θ )) 2 > 0 for all parameters ⃗ θ . There are now two cases to consider.

In the first case, the gradient descent trajectory never leaves some ball of finite radius in R d 1 × d 0 , the parameter space for the first layer. In any such ball, recalling that the first layer's parameterisation is entry-normalisation (Example A.1), the smallest singular value

of D en ( w ) is uniformly lower bounded by some positive constant. Thus by Lemmas B.5 and B.4 8 , the smallest singular value of DF is also uniformly lower bounded by a positive constant in any such ball. It follows from Theorem 2.4 that the loss satisfies the PŁ-inequality over such a ball, so that gradient descent converges in this case at a linear rate to a global minimum.

The second and only other case that must be considered is when for each R > 0 , there is some time T for which the weight norm ∥ w t ∥ of the parameters in the first layer is greater than R for all t ≥ T . That is, the parameter trajectory in the first layer is unbounded in time. In this case, inspection of Equation (36) reveals that the smallest singular value of DF cannot be uniformly bounded below by a positive constant over all of parameter space. Theorem 2.4 then says that there is merely a sequence ( µ t ) t ∈ N , with µ t proportional to σ ( D en ( w t )) , for which

where α = η (1 -2 β -1 η ) > 0 . To guarantee convergence in this case, therefore, it suffices to show that ∏ ∞ t =0 (1 -µ t α ) = 0 ; equivalently, it suffices to show that the infinite series

diverges.

The terms of the series (38) form a sequence of negative numbers which converges to zero. Hence, for the series (38) to diverge, it is necessary that µ t decrease sufficiently slowly with time. By the integral test, therefore, it suffices to find an integrable function m : [ t 0 , ∞ ) → R ≥ 0 such that µ t ≥ m ( t ) for each integer t ≥ t 0 , for which the integral ∫ ∞ t 0 log(1 -m ( t ) α ) dt diverges.

We construct m by considering the worst possible case: where each gradient descent step is in exactly the same direction going out to ∞ , thereby decreasing σ ( D en ( w )) at the fastest possible rate. By applying an orthogonal-affine transformation to R d 1 d 0 , we can assume without loss of generality that the algorithm is initialised at, and consistently steps in the direction of, the first canonical basis vector e 1 in R d 1 d 0 . Specifically, letting ⃗ θ be the vector of parameters for all layers following the first and w ∈ R d 1 d 0 the first layer parameters, for r ∈ R ≥ 1 we may assume that

with ∂ w 1 1 ℓ ( ⃗ θ, re 1 ) ≥ 0 for all ( ⃗ θ, r ) . Let γ denote the convex function defined by the cost c (cf. Equation (6)), and let A ( ⃗ θ, re 1 ) denote the d 1 d 0 -dimensional row vector

Then, in this worst possible case, the single nonzero partial derivative defining the loss gradient with respect to w at the point ( ⃗ θ, re 1 ) is given by

where A ( ⃗ θ, re 1 ) 1 denotes the first component of the row vector A ( ⃗ θ, re 1 ) (cf. Equation (21)). By Theorem 4.5, however, the magnitude of A ( ⃗ θ, re 1 ) can be globally upper bounded by some constant C . Thus for all ⃗ θ and r ≥ 1 .

Let us therefore consider the Euler method, with step size η , applied over R ≥ 1 , starting from r 0 = 1 , with respect to the vector field V ( r ) = Cϵr -3 . Labelling the iterates ( r t ) t ∈ N , we claim that there exist constants γ 1 , γ 2 and γ 3 such that 0 < r t ≤ γ 1 + γ 2 ( t + γ 3 ) 1 4 for all t ∈ N . Indeed, observe that the solution to the flow equation ˙ r ( t ) = Cϵr ( t ) -3 is r ( t ) = (4 Cϵt +1) 1 4 , so the claim follows if we can show that there exists a constant B such that | r t -r ( t ) | < B for all integer t ≥ 0 . However this follows from Theorem 10.6 of [17].

Now, since for each t , r t is an upper bound for the magnitude of the parameter vector w t ∈ R d 1 d 0 , we see from Equation (36) that the smallest singular value σ ( D en ( w t )) admits the lower bound

for all t ∈ N . Clearly, ( ϵ +( γ 1 + γ 2 ( t + γ 3 ) 1 4 ) 2 ) 3 2 = O ( t 3 4 ) . Hence there exist t 0 > 0 and Γ > 0 such that

for all integer t ≥ t 0 . Then, setting m ( t ) := Γ t -3 4 , the integral

diverges. It follows that gradient descent converges as t →∞ to a global minimum.

C Experimental details

For all our experiments, the data was standardised channel-wise using the channel-wise mean and standard deviation over the training set.

On CIFAR10/100, our models 9 were trained using SGD with a batch size of 128 and random crop/horizontal flip data augmentation. We ran 10 trials over each of the learning rates 0.2, 0.1, 0.05 and 0.02. The only exception to this is for CIFAR10 with a learning rate of 0.2, where training diverged 6 our of 10 times on the original network, so we plotted only those 4 trials where training did not diverge. Mean and standard deviation test accuracies, as well as averaged-over-final epoch loss values, are given in Tables 1 and 2.

The plots at the optimal learning rate, 0.1 for CIFAR10 and 0.05 for CIFAR100, are in Figure 3, while we provide the plots for the other learning rates in Figures 4 and 5.

0

20

40

60

80

Epoch

10 0

10 1

Loss

orig.

mod.

(a)

lr

= 0

.

2

0

20

40

60

80

Epoch

10

2

10

1

10 0

Loss

orig.

mod.

(b)

lr

= 0

.

05

0

20

40

60

80

Epoch

10

2

10

1

10 0

Loss

orig.

mod.

(c)

lr

= 0

.

02

0

20

40

60

80

Epoch

10

2

10

1

10 0

Loss

orig.

mod.

(a)

lr

= 0

.

2

0

20

40

60

80

Epoch

10

2

10

1

10 0

Loss

orig.

mod.

(b)

lr

= 0

.

05

0

20

40

60

80

Epoch

10

2

10

1

10 0

Loss

orig.

mod.

(c)

lr

= 0

.

02

0

10

20

30

40

50

60

70

80

Epoch

1

2

3

4

5

Loss

orig. 0.2

mod. 0.2

orig. 0.1

orig. 0.05

mod. 0.05

orig. 0.02

mod. 0.02

On ImageNet, the models were trained using the default PyTorch ImageNet example 10 , using SGD with weight decay of 1 e -4 and momentum of 0 . 9 , batch size of 256, and random crop/horizontal flip data augmentation. See Table 3 for the validation accuracies, and Figure 6 for the plots of the training losses at different learning rates.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies optimisation in deep architectures, and presents a framework for theoretically capturing the role of architectural components like skip connections and normalisation layers/weight normalisation. This framework considers the curvature (smootness) and regularity properties of multi-layer networks (networks that are compositions of per-layer blocks), but differs from existing results for deep NN optimisation as it enables convergence to be shown for trajectories far away from initialisation (useful for settings when global minima only exist at infinity, like with Cross entropy loss). Finally, the authors use their theoretical results to argue that the empirically observed optimisation benefits of skip connections may be due to improving the conditioning of the loss landscape, and reinforce this argument through motivating modified skip connections which (slightly) improve performance empirically in practice.\n\nSTRENGTHS:\n- The paper is well motivated; it address an important question (that of bridging the gap between theory and practice in deep learning optimisation)\n- The paper is well written and places itself clearly and fairly within the existing literature (in section 3)\n- The paper makes contributions towards answering this question, such as: 1) building upon existing work in optimisation theory to enable global analyses that can hold for convergence far from initialisation, and 2) connecting their theoretical results to practical considerations concerning architectural design, e.g. skip connections.\n\nWEAKNESSES:\n- I'm not sure the extent to which the theoretical bounds e.g. showing the benefits of skips in terms of regularity (i.e. looking at minimum eigenvalues) translate into practice, due to the assumptions made that may not hold in practice, and also the fact that they are worst-case bounds (see lines 265-271 for the authors own words on this). I should add that this isn't really a huge criticism of this work as it applies to most works in this area.\n- I'm not sure if the statement in section 6 'skip connections aid optimisation by improving loss regularity' is itself a big contribution. As the authors write in lines 138-146, there have been several works to make this statement/similar statements. Moreover, as far as I understand the theoretical arguments in Theorems 4.7 and 5.1, the arguments goes like: \"Each residual layer's jacobian can be written as <identity + something bound to have eigenvalues norms less than 1>, which will have positive minimum eigenvalue, and by the chain rule their product will be too\", which is quite a simple argument (forgive me if I have misunderstood, and of course it is good to formalise these things).\n- The experimental results seem a bit thin, particularly I'm a bit hesistant to read too much into comparisons of training speed on ImageNet if learning rates haven't been tuned (and indeed it seems most hyperparameters were not tuned for the CIFAR results, in Appendix A.1). It would also be good to verify the empirical gains of the proposed skip architectures across different optimisers (e.g. Adam), and more depths/different residual architectures like PostAct, and also other data settings (e.g. language tasks with residual transformers), to be more convincing empirically.\n- I think the Deep Kernel Shaping recent line of work (https://arxiv.org/abs/2110.01765, https://arxiv.org/abs/2203.08120) is highly relevant, and complements the current paper, so probably should be cited. Likewise, the reparameterised VGG works (https://arxiv.org/abs/2101.03697, https://arxiv.org/abs/2205.15242).\n\nQUESTIONS:\n- Can the authors comment on the fact that their results are designed for parameters converging to global minima at infinity, with the fact that they require bounded weights (or normalised weights/normalisation layers) in their analysis?\n- Can the authors clarify what they mean by 'I is an average pool', do you pool over all pixels or only within a set filter size? It would be good just to provide a formula for this.\n- Please can the authors respond to my comments in the weaknesses section. I will be happy to update my score if the rebuttal can address these.\n\n**Update: after clarifications in the rebuttal, additional experiments, and reading the other reviews, I am increasing my score to 7**\n\nLIMITATIONS:\nPlease see weaknesses, though the authors are quite fair about the limitations of their work and also future directions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper deals with convergence of deep learning making use of neural Tangent Kernels (NTK) theory and its recent advances. It is a theoretical work, claiming to provide ... \"l: 7-11 Abstract: ... the only proof of which we are aware that a class of deep neural networks can be trained using gradient descent to global optima even when such optima only exist at infinity, as is the case for the cross-entropy cost.\" The proof, that holds for normalised networks with dense skip connections, uses an interesting approach showing that there exists sufficiently small learning rate that induces both gradient Lipschitzity and PL lower bound on gradient norm.\nInterestingly the paper admits that main theorem (claiming above) is unrealistic for many reasons, i.e., more data points than input dimensions, not skipping every layer etc. The experimental results are to demonstrate that some ideas used in the main theorem could hold in practical settings as well, forming the Hypothesis 6.1.\n\nSTRENGTHS:\nFirst I'd like to thank authors for the submission of interesting paper. It has well written introduction and background knowledge sections, recalling the key building blocks (Lipschitz gradients and PL inequality) and how are they treated in related works.\n\nAlso the approach of proving the main theorem as highlighted in summary is interesting. However its 'globality' is less surprising when normalisation layers are required (than any point is weight space is translated and rescaled to lie nearby origin in probability and thus near origin NTK theory applies).\n\nWEAKNESSES:\nImpact: From my perspective and despite the nice work of proving the Main Theorem, the main weakness of this work lies in its impact. The Theorem relies on unrealistic assumptions, admitted by authors as well between lines 265-275. Then paper reduces to \"just another\" hypothesis why in \"practice\" deep learning converges to global optima. \nRecall that regularisation of skip connections have been reported and theoretically argued in multitude of previous works (see, Related Work sections or Principles of riemannian geometry in neural networks, Hauser, Michael and Ray, Asok, Advances in neural information processing systems, vol. 30, 2017 just to mention one out of many uncited, for instance).\n\nClarity: To improve the impact the work would benefit from more accessible presentation. I leave it to authors to consider that for next iteration.\n\nQUESTIONS:\nQ1: The paper strives to establish new toolkit for a convergence theory in DL. One way to increase the impact and importance for ML community would be to elaborate and suggest research directions newly opened up by this approach. Could authors elaborate on open problems briefly mentioned in Discussion and suggest the ways to leverage results in the paper to advance them?\n\nLIMITATIONS:\nAs per above, the main results relies on highly unrealistic assumptions and thus impact is limited mostly to 'proof technique' for certain problems within NTK realm. This could be powerful, but should be elaborated more, see Questions section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a novel proof strategy showing the guaranteed convergence to a global optima if a particular class of deep neural networks with non-linear activation functions provided there are skip connections.  Such results have been known for linear networks so it is nice that it also applies to non-linear networks even when using a cross-entropy loss where some of the weights might diverge.  The paper goes on to show empirically that by adding in skip connections where the do not currently exist it is possible to speed up convergence of Resnets albeit marginally.\n\nSTRENGTHS:\nThis is a very accomplished paper, clearly written by people who know what they are doing.  The paper provides a novel proof strategy to prove convergence.\n\nWEAKNESSES:\nAlthough technically elegant, the proof strategy only currently works in the strictly over-parameterised regime.  This is not a regime where most people work and it seems unlikely that convergence fails when we have more data-points.  I would therefore class this as a proof strategy that unfortunately doesn't really work.  Obviously, I might have misunderstood and would change my judgement if the authors could convince me that I am wrong.\n\nThe proof also seems to sacrifice linear convergence, which makes the proof slightly less interesting.  Again, maybe I am wrong about this.\n\nMore generally it is a difficult judgement call as to the value of this paper.  Clearly in deep networks people run in a regime where there is most likely many local and global optima and where they never run long enough to reach an optima (given the number of parameters, there use of mini-batches and typical size of the momentum parameter).  It is also not clear that reaching a global optimum is desirable (early stopping was a common regularisation strategy in small neural networks---it maybe unnecessary in deep neural networks because we always stop early).  In addition, Relu activation functions are usually found to give superior performance to those with Lipschitz gradients.  It thus seems that the direction of this line of research is telling us little about deep learning optimisaton.  I accept that it is interesting whether there exists a deep network with a unique minima that can be solved efficiently that demonstrates some of the power of deep learning.  This could be shown empirically, but conditions such as strict over-parameterisation makes this less useful.  I also appreciate the challenge involved in proving anything and I admire the attempt, but I still need some convincing that the direction of research is leading somewhere.\n\nI found the empirical part of the paper slightly underwhelming.  The effect of adding a skip connection did lead to systematic improvement, but rather minor.  However, there are many explanations out there of why skip connections are beneficial and while it is plausible that the improvement is due to improved regularisation, I suspect other explanations could also be put forward (e.g. further breaking of symmetry).\n\nThis was by far the hardest paper I had to review and it took an order of magnitude longer to do it.  Even then I am at my limit of understanding so my critique might be unfounded.  If you can convince me then I am happy to change my score.\n\nQUESTIONS:\nIs the regime where there are more training data than free parameters physically different (i.e. we would expect in some cases we would not converge)?  If not is not the success of the proof a mathematical artefact rather than capturing the physics?\n\nIs it the case that convergence might be sub-linear?  If so are there any guarantees on the speed of convergence?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a formal theoretical framework to analyze gradient and convergence properties of multilayer deep networks. The authors make the following contributions: (1) they provide the first proof that certain deep networks can be trained to global optima at infinity using gradient descent; (2) the authors show that skip connections improve the loss regularity and thus improve the optimization speed; (3) they empirically show that improving the mean singular values of layer Jacobians can practically improve the convergence of ResNets on practical datasets like MNIST/CIFAR/ImageNet.\n\nSTRENGTHS:\nThe paper has following strengths:\n\n1. The authors present a detailed study of smoothness and regularity of deep networks with skip connections and normalization layers. By connecting these properties to singular values of layer Jacobians, authors demonstrate that skip connections improve loss regularity. This insight is interesting, and the authors verified this empirically on practical networks on MNIST, CIFAR, ImageNet datasets.\n\n2. The proposed theoretical framework may be useful for further theoretical studies on models with skip connections.\n\n3. The authors also conducted an interesting experiment where they augment the 1x1 convolutions with a deterministic mapping which has singular values = 1 (they used average pool that was rescaled to give singular values = 1). The authors showed that this improved training convergence as well as the final loss for all networks considered.\n\nWEAKNESSES:\nThe paper has following weaknesses:\n\n1. Some of the assumptions (as identified by the authors themselves) are not practical (lines 265-271). However, this is to be expected due to difficulty of this kind of theoretical studies. A few other assumptions include: d_{l-1} >= d_{l} (line 241) which generally does not hold (models usually have increasing number of neurons/feature maps). Also, in Theorem 5.1, can authors elaborate more on the implications of using convex cost function on the practicality of their theoretical results (line 252)?\n\n2. Instead of using a deterministic map (avg pool rescaled to give singular values = 1), did the authors try directly using orthogonal initialization on the 1x1 convolutions in the skip connections? Would that significantly improve convergence compared to regular ResNets with default 1x1 initializations? Why did the authors use avg pool based parameterization of the skip connections?\n\n3. There are a few important related works that the authors should discuss in section 3. Specifically, one paper defines “Layerwise Dynamical Isometry” [ICLR 2020, https://arxiv.org/abs/1906.06307] that talks about singular values of layer Jacobians and shows how improving those singular values can significantly improve training convergence of highly sparse networks. Another paper that is highly related defines NN-Mass [CVPR 2021, https://arxiv.org/abs/1910.00780] which explicitly relates singular values of layer Jacobians to skip connections. Although their theory is formulated for DenseNet-type networks, empirically, they show that their metric works for ResNets and MobileNets too. These methods show that such theory-grounded “training-free” metrics can identify efficient models directly at initialization without expensive training-based search methods. Other newer papers from this domain conduct full-blown zero-shot (training-free) Neural Architecture Search like ZenNAS (ICCV 2021, https://arxiv.org/abs/2102.01063), ZiCo (ICLR 2023, https://arxiv.org/abs/2301.11300), MAE-DET (ICML 2022, https://arxiv.org/abs/2111.13336), etc. These papers provide more insight towards expressive power or gradient properties of complex deep networks. Although the above papers may not have as much formal theory as the proposed work, it would be good to see relationship between more practical papers and the present theoretical study.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}