GENERALIZATION BOUNDS FOR DEEP CONVOLUTIONAL NEURAL NETWORKS
Year: 2020

Abstract:
We prove bounds on the generalization error of convolutional networks. The bounds are in terms of the training loss, the number of parameters, the Lipschitz constant of the loss and the distance from the weights to the initial weights. They are independent of the number of pixels in the input, and the height and width of hidden feature maps. We present experiments using CIFAR-10 with varying hyperparameters of a deep convolutional network, comparing our bounds with practical generalization gaps.

text: We prove bounds on the generalization error of convolutional networks. The bounds are in terms of the training loss, the number of parameters, the Lipschitz constant of the loss and the distance from the weights to the initial weights. They are independent of the number of pixels in the input, and the height and width of hidden feature maps. We present experiments using CIFAR-10 with varying hyperparameters of a deep convolutional network, comparing our bounds with practical generalization gaps.

heading: 1 INTRODUCTION
text: Recently, substantial progress has been made regarding theoretical analysis of the generalization of deep learning models (see Neyshabur et al., 2015; Zhang et al., 2016; Dziugaite and Roy, 2017; Bartlett et al., 2017; Neyshabur et al., 2017; 2018; Arora et al., 2018; Golowich et al., 2018; Neyshabur et al., 2019; Wei and Ma, 2019a; Cao and Gu, 2019; Daniely and Granot, 2019). One interesting point that has been explored, with roots in (Bartlett, 1998), is that even if there are many parameters, the set of models that can be represented using weights with small magnitude is limited enough to provide leverage for induction (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018). Intuitively, if the weights start small, since the most popular training algorithms make small, incremental updates that get smaller as the training accuracy improves, there is a tendency for these algorithms to produce small weights. (For some deeper theoretical exploration of implicit bias in deep learning and related settings, see (Gunasekar et al., 2017; 2018a;b; Ma et al., 2018).) Even more recently, authors have proved generalization bounds in terms of the distance from the initial setting of the weights instead of the size of the weights (Dziugaite and Roy, 2017; Bartlett et al., 2017; Neyshabur et al., 2019; Nagarajan and Kolter, 2019). This is important because small initial weights may promote vanishing gradients; it is advisable instead to choose initial weights that maintain a strong but non-exploding signal as computation flows through the network (see LeCun et al., 2012; Glorot and Bengio, 2010; Saxe et al., 2013; He et al., 2015). A number of recent theoretical analyses have shown that, for a large network initialized in this way, accurate models can be found by traveling a short distance in parameter space (see Du et al., 2019b;a; Allen-Zhu et al., 2019; Zou et al., 2018; Lee et al., 2019). Thus, the distance from initialization may be expected to be significantly smaller than the magnitude of the weights. Furthermore, there is theoretical reason to expect that, as the number of parameters increases, the distance from initialization decreases. This motivates generalization bounds in terms of distance from initialization (Dziugaite and Roy, 2017; Bartlett et al., 2017).
Convolutional layers are used in all competitive deep neural network architectures applied to image processing tasks. The most influential generalization analyses in terms of distance from initialization have thus far concentrated on networks with fully connected layers. Since a convolutional layer has an alternative representation as a fully connected layer, these analyses apply in the case of convolutional networks, but, intuitively, the weight-tying employed in the convolutional layer constrains the set of functions computed by the layer. This additional restriction should be expected to aid generalization.
∗Authors are ordered alphabetically.
In this paper, we prove new generalization bounds for convolutional networks that take account of this effect. As in earlier analyses for the fully connected case, our bounds are in terms of the distance from the initial weights, and the number of parameters. Additionally, our bounds independent of the number of pixels in the input, or the height and width of the hidden feature maps.
Our most general bounds apply to networks including both convolutional and fully connected layers, and, as such, they also apply for purely fully connected networks. In contrast with earlier bounds for settings like the one considered here, our bounds are in terms of a sum over layers of the distance from initialization of the layer. Earlier bounds were in terms of product of these distances which led to an exponential dependency on depth. Our bounds have linear dependency on depth which is more aligned with practical observations.
As is often the case for generalization analyses, the central technical lemmas are bounds on covering numbers. Borrowing a technique due to Barron et al. (1999), these are proved by bounding the Lipschitz constant of the mapping from the parameters to the loss of the functions computed by the networks. (Our proof also borrows ideas from the analysis of the fully connected case, especially (Bartlett et al., 2017; Neyshabur et al., 2018).) Covering bounds may be applied to obtain a huge variety of generalization bounds. We present two examples for each covering bound. One is a standard bound on the difference between training and test error. Perhaps the more relevant bound has the flavor of “relative error”; it is especially strong when the training loss is small, as is often the case in modern practice. Our covering bounds are polynomial in the inverse of the granularity of the cover. Such bounds seem to be especially useful for bounding the relative error.
In particular, our covering bounds are of the form (B/ )W , where is the granularity of the cover, B is proportional to the Lipschitz constant of a mapping from parameters to functions, and W is the number of parameters in the model. We apply a bound from the empirical process literature in terms of covering bounds of this form due to Giné and Guillou (2001), who paid particular attention to the dependence of estimation error on B. This bound may be helpful for other analyses of the generalization of deep learning in terms of different notions of distance from initialization. (Applying bounds in terms of Dudley’s entropy integral in the standard way leads to an exponentially worse dependence on B.)
Related previous work. Du et al. (2018) proved bounds for CNNs in terms of the number of parameters, for two-layer networks. Arora et al. (2018) analyzed the generalization of networks output by a compression scheme applied to CNNs. Zhou and Feng (2018) provided a generalization guarantee for CNNs satisfying a constraint on the rank of matrices formed from their kernels. Li et al. (2018) analyzed the generalization of CNNs under other constraints on the parameters. Lee and Raginsky (2018) provided a size-free bound for CNNs in a general unsupervised learning framework that includes PCA and codebook learning.
Related independent work. Ledent et al. (2019) proved bounds for CNNs that also took account of the effect of weight-tying. (Their bounds retain the exponential dependence on the depth of the network from earlier work, and are otherwise qualitatively dissimilar to ours.) Wei and Ma (2019b) obtained bounds for fully connected networks with an improved dependence on the depth of the network. Daniely and Granot (2019) obtained improved bounds for constant-depth fully-connected networks. Jiang et al. (2019) conducted a wide-ranging empirical study of the dependence of the generalization gap on a variety of quantities, including distance from initialization.
Notation. If K(i) is the kernel of convolutional layer number i, then op(K(i)) refers to its operator matrix 1 and vec(K(i)) denotes the vectorization of the kernel tensor K(i). For matrix M , ‖M‖2 denotes the operator norm of M . For vectors, || · || represents the Euclidian norm, and || · ||1 is the L1 norm. For a multiset S of elements of some set Z, and a function g from Z to R, let ES [g] = 1m ∑m t=1 g(zt). We will denote the function parameterized by Θ by fΘ.

heading: 2 BOUNDS FOR A BASIC SETTING
text: In this section, we provide a bound for a clean and simple setting.
1Convolution is a linear operator and can thus be written as a matrix-vector product. The operator matrix of kernel K, refers to the matrix that describes convolving the input with kernel K. For details, see (Sedghi et al., 2018).

heading: 2.1 THE SETTING AND THE BOUNDS
text: In the basic setting, the input and all hidden layers have the same number c of channels. Each input x ∈ Rd×d×c satisfies ‖ vec(x)‖ ≤ 1. We consider a deep convolutional network, whose convolutional layers use zero-padding (see Goodfellow et al., 2016). Each layer but the last consists of a convolution followed by an activation function that is applied componentwise. The activations are 1-Lipschitz and nonexpansive (examples include ReLU and tanh). The kernels of the convolutional layers are K(i) ∈ Rk×k×c×c for i ∈ {1, . . . , L}. Let K = (K(1), ...,K(L)) be the L× k × k × c× c-tensor obtained by concatening the kernels for the various layers. Vector w represents the last layer; the weights in the last layer are fixed with ||w|| = 1. Let W = Lk2c2 be the total number of trainable parameters in the network.
We let K(1)0 , . . . ,K (L) 0 take arbitrary fixed values (interpreted as the initial values of the kernels) subject to the constraint that, for all layers i, || op(K(i)0 )||2 = 1. (This is often the goal of initialization schemes.) Let K0 be the corresponding L× k× k× c× c tensor. We provide a generalization bound in terms of distance from initialization, along with other natural parameters of the problem. The distance is measured with ||K −K0||σ def = ∑L i=1 || op(K(i))− op(K (i) 0 )||2.
For β > 0, define Kβ to be the set of kernel tensors within || · ||σ distance β of K0, and define Fβ to be set of functions computed by CNNs with kernels in Kβ . That is, Fβ = {fK : ||K −K0||σ ≤ β}. Let ` : R× R→ [0, 1] be a loss function such that `(·, y) is λ-Lipschitz for all y. An example is the 1/λ-margin loss.
For a function f from Rd×d×c to R, let `f (x, y) = `(f(x), y).
We will use S to denote a set {(x1, y1), . . . , (xm, ym)} = {z1, . . . zm} of random training examples where each zt = (xt, yt). Theorem 2.1 (Basic bounds). For any η > 0, there is a C > 0 such that for any β, δ > 0, λ ≥ 1, for any joint probability distribution P over Rd×d×c × R, if a training set S of n examples is drawn independently at random from P , then, with probability at least 1− δ, for all f ∈ Fβ ,
Ez∼P [`f (z)] ≤ (1 + η)ES [`f ] + C(W (β + log(λn)) + log(1/δ))
n
and, if β ≥ 5, then Ez∼P [`f (z)] ≤ ES [`f ] + C √ W (β + log(λ)) + log(1/δ)
n
and otherwise
Ez∼P [`f (z)] ≤ ES [`f ] + C ( βλ √ W
n +
√ log(1/δ)
n
) .
If Theorem 2.1 is applied with the margin loss, then Ez∼P [`f (z)] is in turn an upper bound on the probability of misclassification on test data. Using the algorithm from (Sedghi et al., 2018), || · ||σ may be efficiently computed. Since ||K −K0||σ ≤ || vec(K) − vec(K0)||1 (Sedghi et al., 2018), Theorem 2.1 yields the same bounds as a corollary if the definition of Fβ is replaced with the analogous definition using || vec(K)− vec(K0)||1.

heading: 2.2 TOOLS
text: Definition 2.2. For d ∈ N , a set G of real-valued functions with a common domain Z, we say that G is (B, d)-Lipschitz parameterized if there is a norm || · || on Rd and a mapping φ from the unit ball w.r.t. || · || in Rd to G such that, for all θ and θ′ such that ||θ|| ≤ 1 and ||θ′|| ≤ 1, and all z ∈ Z, |(φ(θ))(z)− (φ(θ′))(z)| ≤ B||θ − θ′||.
The following lemma is essentially known. Its proof, which uses standard techniques (see Pollard, 1984; Talagrand, 1994; 1996; Barron et al., 1999; Van de Geer, 2000; Giné and Guillou, 2001; Mohri et al., 2018), is in Appendix A.
Lemma 2.3. Suppose a set G of functions from a common domain Z to [0,M ] is (B, d)-Lipschitz parameterized for B > 0 and d ∈ N. Then, for any η > 0, there is a C such that, for all large enough n ∈ N, for any δ > 0, for any probability distribution P over Z, if S is obtained by sampling n times independently from P , then, with probability at least 1− δ, for all g ∈ G,
Ez∼P [g(z)] ≤ (1 + η)ES [g] + CM(d log(Bn) + log(1/δ))
n
and if B ≥ 5,
Ez∼P [g(z)] ≤ ES [g] + CM
√ d logB + log 1δ
n ,
and, for all B,
Ez∼P [g(z)] ≤ ES [g] + C B√ d n +M √ log 1δ n  .

heading: 2.3 PROOF OF THEOREM 2.1
text: Definition 2.4. Let `F = {`f : f ∈ F}.
We will prove Theorem 2.1 by showing that `Fβ is ( βλeβ ,W ) -Lipschitz parameterized. This will be achieved through a series of lemmas.
Lemma 2.5. Choose K ∈ Kβ and a layer j. Suppose K̃ ∈ Kβ satisfies K(i) = K̃(i) for all i 6= j. Then, for all examples (x, y), |`(fK(x), y)− `(fK̃(x), y)| ≤ λeβ ∥∥∥op(K(j))− op(K̃(j))∥∥∥ 2 .
Proof. For each layer i, let βi = || op(K(i))− op(K(i)0 )||2. Since ` is λ-Lipschitz w.r.t. its first argument, we have that |`(fK(x), y)−`(fK̃(x), y)| ≤ λ|fK(x)− fK̃(x)|, so it suffices to bound |fK(x) − fK̃(x)|. Let gup be the function from the inputs to the whole network with parameters K to the inputs to the convolution in layer j, and let gdown be the function from the output of this convolution to the output of the whole network, so that fK = gdown ◦ fop(K(j)) ◦ gup. Choose an input x to the network, and let u = gup(x). Recalling that ||x|| ≤ 1, and the non-linearities are nonexpansive, we have ‖u‖ ≤
∏ i<j ∥∥∥op(K(i))∥∥∥ 2 . Since the non-linearities
are 1-Lipschitz, and, recalling that K(i) = K̃(i) for i 6= j, we have
|fK(x)− fK̃(x)| = |gdown(op(K (j))u)− gdown(op(K̃(j))u)|
≤ ∏ i>j ∥∥∥op(K(i))∥∥∥ 2 ∥∥∥op(K(j))u− op(K̃(j))u∥∥∥ ≤
∏ i>j ∥∥∥op(K(i))∥∥∥ 2 ∥∥∥op(K(j))− op(K̃(j))∥∥∥ 2 ‖u‖
≤ ∏ i 6=j ∥∥∥op(K(i))∥∥∥ 2 ∥∥∥op(K(j))− op(K̃(j))∥∥∥ 2
≤ ∏ i 6=j (1 + βi) ∥∥∥op(K(j))− op(K̃(j))∥∥∥ 2
where the last inequality uses the fact that || op(K(i))−op(K(i)0 )||2 ≤ βi for all i and || op(K (i) 0 )||2 = 1 for all i.
Now ∏ i 6=j(1 + βi) ≤ ∏L i=1(1 + βi), and the latter is maximized over the nonnegative βi’s subject
to ∑ i6=j βi ≤ β when each of them is β/L. Since (1 + β/L)L ≤ eβ , this completes the proof.
Now we prove a bound when all the layers can change between K and K̃.
Lemma 2.6. For any K, K̃ ∈ Kβ , for any input x to the network, |`(fK(x), y) − `(fK̃(x), y)| ≤ λeβ ∥∥∥K − K̃∥∥∥ σ .
Proof. Consider transforming K to K̃ by replacing one layer of K at a time with the corresponding layer in K̃. Applying Lemma 2.5 to bound the distance traversed with each replacement and combining this with the triangle inequality gives
|`(fK(x), y)− `(fK̃(x), y)| ≤ λe β L∑ j=1 ∥∥∥op(K(j))− op(K̃(j))∥∥∥ 2 = λeβ ∥∥∥K − K̃∥∥∥ σ .
Now we are ready to prove our basic bound.
Proof (of Theorem 2.1). Consider the mapping φ from the ball w.r.t. ‖·‖σ of radius 1 in RLk 2c2 centered at vec(K0) to `Fβ defined by φ(θ) = `fK0+β vec−1(θ) , where vec −1(θ) is the reshaping of θ into a L× k × k × c× c-tensor. Lemma 2.6 implies that this mapping is βλeβ-Lipschitz. Applying Lemma 2.3 completes the proof.

heading: 2.4 COMPARISONS
text: Since a convolutional network has an alternative parameterization as a fully connected network, the bounds of (Bartlett et al., 2017) have consequences for convolutional networks. To compare our bound with this, first, note that Theorem 2.1, together with standard model selection techniques, yields a
O
(√ W (||K −K0||σ + log(λ)) + log(1/δ)
n
) (1)
bound on Ez∼P [`f (z)]−ES [`f (z)] (For more details, please see Appendix B.) Translating the bound of (Bartlett et al., 2017) to our setting and notation directly yields a bound on Ez∼P [`f (z)]−ES [`f (z)] whose main terms are proportional to
λ (∏L i=1 || op(K(i))||2 )(∑L i=1 || op(K(i))>−op(K(i)0 ) >||2/32,1
|| op(K(i))||2/32
)3/2 log(d4c2L) + √ log(1/δ)
√ n
(2)
where, for a p × q matrix A, ||A||2,1 = ||(||A:,1||2, ..., ||A:,q||2)||1. One can get an idea of how this bound relates to (1) by comparing the bounds in a simple concrete case. Suppose that each of the convolutional layers of the network parameterized by K0 computes the identity function, and that K is obtained from K0 by adding to each entry. In this case, disregarding edge effects, for all i, || op(K(i))||2 = 1 + k2c and ||K − K0||σ = k2cL (as proved in Appendix C). Also, || op(K(i))> − op(K(i)0 )>||2,1 = (cd2)( √ ck2) = c3/2d2k. We get additional simplification if we set = 1k2 . In this case, (2) gives a constant times
(c+ 1)L √ cd(d/k)L3/2λ log(dcL) + √ log(1/δ)√
n
where (1) gives a constant times c3/2kL+ ck √ log(λ) + √
log(1/δ)√ n .
In this scenario, the new bound is independent of d, and grows more slowly with λ, c and L. Note that k ≤ d (and, typically, it is much less). This specific case illustrates a more general phenomenon that holds when the initialization is close to the identity, and changes to the parameters are on a similar scale.
Golowich et al. (2017) established bounds that improve on the bounds of (Bartlett et al., 2017) in some cases. Their bound requires a restriction on the activation function (albeit one that is satisfied by the ReLU). For large n, the main term of the natural consequence of Corollary 1 of their paper in the setting of this section, with the required additional assumption on the activation function, grows like
λ √ L ∏L i=1 || op(K(i))||F√
n ≈ λ(cd)
L √ L√
n ,
when = 1/k2. (We note, however, that, in addition to not trying to take account of the convolutional structure, Golowich et al. (2017) also did not make an effort to obtain stronger bounds in the case that the distance from initialization is small. On the other hand, we suspect that modifying their proof analogously to (Bartlett et al., 2017) to do so would not remove the exponential dependence on L.)

heading: 3 A MORE GENERAL BOUND
text: In this section, we generalize Theorem 2.1.

heading: 3.1 THE SETTING
text: The more general setting concerns a neural network where the input is a d × d × c tensor whose flattening has Euclidian norm at most χ, and network’s output is a m-dimensional vector, which may be logits for predicting a one-hot encoding of an m-class classification problem.
The network is comprised of Lc convolutional layers followed by Lf fully connected layers. The ith convolutional layer includes a convolution, with kernel K(i) ∈ Rki×ki×ci−1×ci , followed by a componentwise non-linearity and an optional pooling operation. We assume that the non-linearity and any pooling operations are 1-Lipschitz and nonexpansive. Let V (i) be the matrix of weights for the ith fully connected layer. Let Θ = (K(1), ...,K(Lc), V (1), ..., V (Lf )) be all of the parameters of the network. Let L = Lc + Lf .
We assume that, for all y, `(·, y) is λ-Lipschitz for all y and that `(ŷ, y) ∈ [0,M ] for all ŷ and y. An example (x, y) includes a d× d× c-tensor x and y ∈ Rm.
We let K(1)0 , . . . ,K (Lc) 0 , V (1) 0 , ..., V (Lf ) 0 take arbitrary fixed values subject to the constraint that, for all convolutional layers i, || op(K(i)0 )||2 ≤ 1+ν, and for all fully connected layers i, ||V (i) 0 ||2 ≤ 1+ν. Let Θ0 = (K (1) 0 , ...,K (Lc) 0 , V (1) 0 , ..., V (Lf ) 0 ).
For Θ = (K(1), ...,K(Lc), V (1), ..., V (Lf )) and Θ̃ = (K̃(1), ..., K̃(Lc), Ṽ (1), ..., Ṽ (Lf )). define
||Θ− Θ̃||N = ( Lc∑ i=1 || op(K(i))− op(K̃(i))||2 ) + Lf∑ i=1 ||V (i) − Ṽ (i)||2.
For β, ν ≥ 0, define Fβ,ν to be set of functions computed by CNNs as described in this subsection with parameters within || · ||N -distance β of Θ0. Let Oβ,ν be the set of their parameterizations. Theorem 3.1 (General Bound). For any η > 0, there is a constant C such that the following holds. For any β, ν, χ > 0, for any δ > 0, for any joint probability distribution P over Rd×d×c × Rm such that, with probability 1, (x, y) ∼ P satisfies || vec(x)||2 ≤ χ, under the assumptions of this section,
if a training set S of n examples is drawn independently at random from P , then, with probability at least 1− δ, for all f ∈ Fβ,ν ,
Ez∼P [`f (z)] ≤ (1 + η)ES [`f ] + CM (W (β + νL+ log(χλβn)) + log(1/δ))
n
and, if χλβ(1 + ν + β/L)L ≥ 5, Ez∼P [`f (z)] ≤ ES [`f ] + CM √ W (β + νL+ log(χλβ)) + log(1/δ)
n
and a bound of
Ez∼P [`f (z)] ≤ ES [g] + C χλβ(1 + ν + β/L)L√W n +M √ log 1δ n  holds for all χ, λ, β > 0.

heading: 3.2 PROOF OF THEOREM 3.1
text: We will prove Theorem 3.1 by using ||·||N to witness the fact that `Fβ,ν is ( χλβ(1 + ν + β/L)L,W ) - Lipschitz parameterized.
The first two lemmas concern the effect of changing a single layer. Their proofs are very similar to the proof of Lemma 2.5, and are in the Appendices D and E.
Lemma 3.2. Choose Θ = (K(1), ...,K(Lc), V (1), ..., V (Lf )), Θ̃ = (K̃(1), ..., K̃(Lc), Ṽ (1), ..., Ṽ (Lf )) ∈ Oβ,ν and a convolutional layer j. Suppose that K(i) = K̃(i) for all convolutional layers i 6= j and V (i) = Ṽ (i) for all fully connected layers i. Then, for all examples (x, y),
|`(fΘ(x), y)− `(fΘ̃(x), y)| ≤ χλ(1 + ν + β/L) L ∥∥∥op(K(j))− op(K̃(j))∥∥∥
2 .
Lemma 3.3. Choose Θ = (K(1), ...,K(Lc), V (1), ..., V (Lf )), Θ̃ = (K̃(1), ..., K̃(Lc), Ṽ (1), ..., Ṽ (Lf )) ∈ Oβ,ν and a fully connected layer j. Suppose that K(i) = K̃(i) for all convolutional layers i and V (i) = Ṽ (i) for all fully connected layers i 6= j. Then, for all examples (x, y),
|`(fΘ(x), y)− `(fΘ̃(x), y)| ≤ χλ (1 + ν + β/L) L ∥∥∥V (j) − Ṽ (j)∥∥∥
2 .
Now we prove a bound when all the layers can change between Θ and Θ̃.
Lemma 3.4. For any Θ = (K(1), ...,K(Lc), V (1), ..., V (Lf )), Θ̃ = (K̃(1), ..., K̃(Lc), Ṽ (1), ..., Ṽ (Lf )) ∈ Oβ,ν , for any input x,
|`(fΘ(x), y)− `(fΘ̃(x), y)| ≤ χλ (1 + ν + β/L) L ∥∥∥Θ− Θ̃∥∥∥
N .
Proof. Consider transforming Θ to Θ̃ by replacing one layer at a time of Θ with the corresponding layer in Θ̃. Applying Lemma 3.2 to bound the distance traversed with each replacement of a convolutional layer, and Lemma 3.3 to bound the distance traversed with each replacement of a fully connected layer, and combining this with the triangle inequality gives the lemma.
Now we are ready to prove our more general bound.
Proof (of Theorem 3.1). Consider the mapping φ from the ball of || · ||N -radius 1 centered at Θ0 to `Fβ,ν defined by φ(Θ) = `fΘ0+βΘ . Lemma 2.6 implies that this mapping is( χλβ (1 + ν + β/L) L ,W ) -Lipschitz. Applying Lemma 2.3 completes the proof.

heading: 3.3 MORE COMPARISONS
text: Theorem 3.1 applies in the case that there are no convolutional layers, i.e. for a fully connected network. In this subsection, we compare its bound in this case with the bound of (Bartlett et al., 2017). Because the bounds are in terms of different quantities, we compare them in a simple concrete case. In this case, for D = cd2, each hidden layer has D components, and there are D classes. For all i, V (i)0 = I and V (i) = I + H/ √ D, where H is a Hadamard matrix (using the Sylvester construction), and χ = M = 1. Then, dropping the superscripts, each layer V has ||V ||2 = 2, ||V − V0||2 = 1, ||V − V0||2,1 = D.
Further, in the notation of Theorem 3.1, W = D2L, and β = L, and ν = 0. Plugging into to Theorem 3.1 yields a bound on the generalization gap proportional to
DL+D √ L log(λ) + √ log(1/δ)√
n
where, in this case, the bound of (Bartlett et al., 2017) is proportional to λ2LL3/2D log(DL) + √
log(1/δ)√ n
and, when D is large relative to L, Corollary 1 of (Golowich et al., 2017) (approximately) gives
λ2L/2 √ LDL/2 + √ log(1/δ)√
n .

heading: 4 EXPERIMENTS
text: We trained a 10-layer all-convolutional model on the CIFAR-10 dataset. The architecture was similar to VGG (Simonyan and Zisserman, 2014). The network was trained with dropout regularization and an exponential learning rate schedule. We define the generalization gap as the difference between train error and test error. In order to analyze the effect of the number of network parameters on the generalization gap, we scaled up the number of channels in each layer, while keeping other elements of the architecture, including the depth, fixed. Each network was trained repeatedly, sweeping over different values of the initial learning rate and batch sizes 32, 64, 128. For each setting the results were averaged over five different random initializations. Figure 1 shows the generalization gap for different values of W ||K −K0||σ . As in the bound of Theorem 3.1, the generalization gap increases with W ||K − K0||σ. Figure 2 shows that as the network becomes more over-parametrized, the generalization gap remains almost flat with increasing W . This is expected due to role of overparametrization on generalization (Neyshabur et al., 2019). An explanation of this phenomenon that is consistent with the bound presented here is that, ultimately, increasing W leads to a decrease in value of ||K −K0||σ; see Figure 3a. The fluctuations in Figure 3a are partly due to the fact that training neural networks is not an stable process. We provide the medians ||K −K0||σ for different values of W in Figure 3b.

heading: ACKNOWLEDGMENTS
text: We thank Peter Bartlett, Jaeho Lee and Sam Schoenholz for valuable conversations, and anonymous reviewers for helpful comments.

heading: A PROOF OF LEMMA 2.3
text: Definition A.1. If (X, ρ) is a metric space and H ⊆ X , we say that G is an -cover of H with respect to ρ if every h ∈ H has a g ∈ G such that ρ(g, h) ≤ . ThenNρ(H, ) denotes the size of the smallest -cover of H w.r.t. ρ.
Definition A.2. For a domain Z, define a metric ρmax on pairs of functions from Z to R by ρmax(f, g) = supz∈Z |f(z)− g(z)|.
We need two lemmas in terms of these covering numbers. The first is by now a standard bound from Vapnik-Chervonenkis theory (Vapnik and Chervonenkis, 1971; Vapnik, 1982; Pollard, 1984). For example, it is a direct consequence of (Haussler, 1992, Theorem 3).
Lemma A.3. For any η > 0, there is a constant C depending only on η such that the following holds. Let G be an arbitrary set of functions from a common domain Z to [0,M ]. If there are constants B and d such that, Nρmax(G, ) ≤ ( B )d for all > 0, then, for all large enough n ∈ N, for any δ > 0, for any probability distribution P over Z, if S is obtained by sampling n times independently from P , then, with probability at least 1− δ, for all g ∈ G,
Ez∼P [g(z)] ≤ (1 + η)ES [g] + CM(d log(Bn) + log(1/δ))
n .
We will also use the following, which is the combination of (2.5) and (2.7) of (Giné and Guillou, 2001).
Lemma A.4. Let G be an arbitrary set of functions from a common domain Z to [0,M ]. If there are constants B ≥ 5 and d such that Nρmax(G, ) ≤
( B )d for all > 0, for all large enough
n ∈ N, for any δ > 0, for any probability distribution P over Z, if S is obtained by sampling n times independently from P , then, with probability at least 1− δ, for all g ∈ G,
Ez∼P [g(z)] ≤ ES [g] + CM
√ d logB + log 1δ
n ,
where C is an absolute constant.
The above bound only holds for B ≥ 5. The following, which can be obtained by combining Talagrand’s Lemma with the standard bound on Rademacher complexity in terms of the Dudley entropy integral (see (Van de Geer, 2000; Bartlett, 2013)), yields a bound for all B.
Lemma A.5. Let G be an arbitrary set of functions from a common domain Z to [0,M ]. If there are constants B > 0 and d such that Nρmax(G, ) ≤
( B )d for all > 0, then, for all large enough
n ∈ N, for any δ > 0, for any probability distribution P over Z, if S is obtained by sampling n times independently from P , then, with probability at least 1− δ, for all g ∈ G,
Ez∼P [g(z)] ≤ ES [g] + C B√ d n +M √ log 1δ n  , where C is an absolute constant.
So now we want a bound on Nρmax(G, ) for Lipschitz-parameterized classes. For this, we need the notion of a packing which we now define. Definition A.6. For any metric space (X, ρ) and any H ⊆ S, letMρ(H, ) be the size of the largest subset of H whose members are pairwise at a distance greater than w.r.t. ρ.
Lemma A.7 ((Kolmogorov and Tikhomirov, 1959)). For any metric space (X, ρ), any H ⊆ X , and any > 0, we have
Nρ(H, ) ≤Mρ(H, ).
We will also need a lemma about covering a ball by smaller balls. This is probably also already known, and uses a standard proof (see Pollard, 1990, Lemma 4.1), but we haven’t found a reference for it. Lemma A.8. Let
• d be a positive integer,
• || · || be a norm
• ρ be the metric induced by || · ||, and
• κ, > 0.
A ball in Rd of radius κ w.r.t. ρ can be covered by ( 3κ )d balls of radius .
Proof. We may assume without loss of generality that κ > . Let q > 0 be the volume of the unit ball w.r.t. ρ in Rd. Then the volume of any α-ball with respect to ρ is αdq. Let B be the ball of radius r in Rd. The /2-balls centered at the members of any -packing of B are disjoint. Since these centers are contained in B, the balls are contained in a ball of radius κ+ /2. Thus
Mρ(B, ) (
2
)d q ≤ ( κ+
2
)d q ≤ ( 3κ
2
)d q.
Solving forMρ(B, ) and applying Lemma A.7 completes the proof.
We now prove Lemma 2.3. Let || · || be the norm witnessing the fact that G is (B, d)-Lipschitz parameterized, and let B be the unit ball in Rd w.r.t. || · || and let ρ be the metric induced by || · ||. Then, for any , an /B-cover of B w.r.t. ρ induces an -cover of G w.r.t. ρmax, so
Nρmax(G, ) ≤ Nρ(B, /B).
Applying Lemma A.8, this implies Nρmax(G, ) ≤ ( 3B )d .
Then applying Lemma A.3, Lemma A.4 and Lemma A.5 completes the proof.

heading: B PROOF OF (1)
text: For δ > 0, and for each j ∈ N, let βj = 5 × 2j let δj = 12j2 . Taking a union bound over an application of Theorem 2.1 for each value of j, with probability at least 1− ∑ j δj ≥ 1− δ, for all j, and all f ∈ Fβj
Ez∼P [`f (z)] ≤ (1 + η)ES [`f (z)] + C(W (βj + log(λn)) + log(j/δ))
n
and
Ez∼P [`f (z)] ≤ ES [`f (z)] + C √ W (βj + log(λ)) + log(j/δ)
n .
For any K, if we apply these bounds in the case of the least j such that ||K −K0||σ ≤ βj , we get
Ez∼P [`f (z)] ≤ (1 + η)ES [`f (z)] + C(W (2||K −K0||σ + log(λn)) + log(log(||K −K0||σ)/δ))
n
and
Ez∼P [`f (z)] ≤ ES [`f (z)] + C √ W (2||K −K0||σ + log(λ)) + log(log(||K −K0||σ)/δ)
n ,
and simplifying completes the proof.
C THE OPERATOR NORM OF op(K(i))
Let J = K(i) −K(i)0 . Since || op(K(i))||2 = 1 + || op(J)||2, it suffices to find || op(J)||2. For the rest of this section, we number indices from 0, let [d] = {0, ..., d − 1}, and define ω = exp(2πi/d). To facilitate the application of matrix notation, pad the k × k × c× c tensor J out with zeros to make a d× d× c× c tensor J̃ . The following lemma is an immediate consequence of Theorem 6 of Sedghi et al. (2018).
Lemma C.1 (Sedghi et al. (2018)). Let F be the complex d× d matrix defined by Fij = ωij .
For each u, v ∈ [d]× [d], let P (u,v) be the c× c matrix given by P (u,v)k` = (FT J̃:,:,k,`F )uv . Then
|| op(J)||2 = max u,v ||P (u,v)||2.
First, note that, by symmetry, for each u and v, all components of P (u,v) are the same. Thus,
||P (u,v)||2 = c|P (u,v)00 |. (3)
For any u, v,
|P (u,v)00 | = ∣∣∣∣∣∑ p,q ωupωvqJ̃p,q,0,0 ∣∣∣∣∣ ≤ k2 and P (0,0)00 = k
2. Combining this with (3) and Lemma C.1, || op(J)||2 = ck2, which implies || op(K)||2 = 1 + ck2.

heading: D PROOF OF LEMMA 3.2
text: For each convolutional layer i, let βi = || op(K(i)) − op(K(i)0 )||2, and, for each fully connected layer i, let γi = ||V (i) − V (i)0 ||2. Since ` is λ-Lipschitz w.r.t. its first argument, we have that |`(fΘ(x), y)− `(fΘ̃(x), y)| ≤ λ|fΘ(x)− fΘ̃(x)|. Let gup be the function from the inputs to the whole network with parameters Θ to the inputs to the convolution in layer j, and let gdown be the function from the output of this convolution to the output of the whole network, so that fΘ = gdown ◦ fop(K(j)) ◦ gup. Choose an input x to the network, and let u = gup(x). Recalling that ||x|| ≤ χ, and that the non-linearities and pooling operations are non-expansive, we have ‖u‖ ≤ χ
∏ i<j ∥∥∥op(K(i))∥∥∥ 2 . Using the fact that the non-linearities are
1-Lipschitz, we have
|fΘ(x)− fΘ̃(x)| = |gdown(op(K (j))u)− gdown(op(K̃(j))u)|
≤ ∏ i>j ∥∥∥op(K(i))∥∥∥ 2 (∏ i ∥∥∥V (i)∥∥∥ 2 )∥∥∥op(K(j))u− op(K̃(j))u∥∥∥ ≤
∏ i>j ∥∥∥op(K(i))∥∥∥ 2 (∏ i ∥∥∥V (i)∥∥∥ 2 )∥∥∥op(K(j))− op(K̃(j))∥∥∥ 2 ‖u‖
≤ χ ∏ i 6=j ∥∥∥op(K(i))∥∥∥ 2 (∏ i ∥∥∥V (i)∥∥∥ 2 )∥∥∥op(K(j))− op(K̃(j))∥∥∥ 2
≤ χ ∏ i 6=j (1 + ν + βi) (∏ i (1 + ν + γi) )∥∥∥op(K(j))− op(K̃(j))∥∥∥ 2
where the last inequality uses the fact that || op(K(i))−op(K(i)0 )||2 ≤ βi for all i, ||V (i)−V (i) 0 ||2 ≤ γi for all i, || op(K(i)0 )||2 ≤ 1 + ν for all i and ||V (i) 0 ||2 ≤ 1 + ν for all i.
Since (∏ i 6=j(1 + ν + βi) ) ( ∏ i(1 + ν + γi)) ≤ ( ∏ i(1 + ν + βi)) ( ∏ i(1 + ν + γi)), and the latter
is maximized subject to ( ∑ i βi) + ∑ i γi ≤ β when each summand is β/L, this completes the proof.

heading: E PROOF OF LEMMA 3.3
text: For each convolutional layer i, let βi = || op(K(i)) − op(K(i)0 )||2, and, for each fully connected layer i, let γi = ||V (i) − V (i)0 ||2. Since ` is λ-Lipschitz w.r.t. its first argument, we have that |`(fΘ(x), y)− `(fΘ̃(x), y)| ≤ λ|fΘ(x)− fΘ̃(x)|. Let gup be the function from the inputs to the whole network with parameters Θ to the inputs to fully connected layer layer j, and let gdown be the function from the output of this layer to the output of the whole network, so that fΘ = gdown ◦ fV (j) ◦ gup. Choose an input x to the network, and let u = gup(x). Recalling that ||x|| ≤ χ, and that the non-linearities and pooling operations are
non-expansive, we have ‖u‖ ≤ χ (∏ i ∥∥∥op(K(i))∥∥∥ 2 )∏ i<j ∥∥∥V (i)∥∥∥ 2  . Thus |fΘ(x)− fΘ̃(x)| = |gdown(V (j)u)− gdown(Ṽ (j)u)|
≤ ∏ i>j ∥∥∥V (i)∥∥∥ 2 ∥∥∥V (j)u− Ṽ (j)u∥∥∥ ≤
∏ i>j ∥∥∥V (i)∥∥∥ 2 ∥∥∥V (j) − Ṽ (j)∥∥∥ 2 ‖u‖
≤ χ (∏ i ∥∥∥op(K(i))∥∥∥ 2 )∏ i6=j ∥∥∥V (i)∥∥∥ 2 ∥∥∥V (j) − Ṽ (j)∥∥∥ 2
≤ χ (∏ i (1 + ν + βi) )∏ i 6=j (1 + ν + γi) ∥∥∥V (j) − Ṽ (j)∥∥∥ 2 .
Since ( ∏ i(1 + ν + βi)) (∏ i 6=j(1 + ν + γi) ) ≤ ( ∏ i(1 + ν + βi)) ( ∏ i(1 + ν + γi)), and the latter
is maximized subject to ( ∑ i βi) + ∑ i γi ≤ β when each summand is β/L, this completes the proof.

Review summaries:
The paper describes new norm-based generalization bounds that were specifically adapted to convolutional neural networks. Since convolutional neural networks do not explicitly depend on the input dimension, these bounds share the same property. Further additional improvement over Bartlett et al. ‘17 bound, is that this new bound depends on the sum of the operator norms of the parameter matrices, rather than the product.

This paper studied the generalization power of CNNs and showed several upper bounds of generalization errors. Their results have two characteristics. First, the bounds are in terms of the quantity that is independent of the input dimension (size-free). Second, the upper bounds involve the distance between initial and learned parameters. These results improved the upper bounds that we can derive by naively applying the results of Bartlett et al. (2017) or Neushubar et al. (2017), because the dominant term of the existing upper bounds contained $l_{2, 1}$ or $l_2$ norms, which could depend on the input dimensions in the worst case. The authors empirically showed that there is a correlation between the generalization error of learned CNNs and the dominant term of the upper bound (i.e., the product of the parameter size and the distance from the set of initial parameters).