Title: How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learning
Venue: NeurIPS 2024
Accepted: no
Keywords: Generalization

Rademacher complexity

Compositionality

Feature Learning
Abstract: We show that deep neural networks (DNNs) can efficiently learn any
composition of functions with bounded $F_{1}$-norm, which allows
DNNs to break the curse of dimensionality in ways that shallow networks
cannot. More specifically, we derive a generalization bound that combines
a covering number argument for compositionality, and the $F_{1}$-norm
(or the related Barron norm) for large width adaptivity. We show that
the global minimizer of the regularized loss of DNNs can fit for example
the composition of two functions $f^{*}=h\circ g$ from a small number
of observations, assuming $g$ is smooth/regular and reduces the dimensionality
(e.g. $g$ could be the modulo map of the symmetries of $f^{*}$),
so that $h$ can be learned in spite of its low regularity. The measures
of reguarity we consider is the Sobolev norm with different levels
of differentiability, which is well adapted to the $F_{1}$ norm.
We compute scaling laws empirically, and observe phase transitions
depending on whether $g$ or $h$ is harder to learn, as predicted
by our theory.

Abstract

We show that deep neural networks (DNNs) can efficiently learn any composition of functions with bounded F 1 -norm, which allows DNNs to break the curse of dimensionality in ways that shallow networks cannot. More specifically, we derive a generalization bound that combines a covering number argument for compositionality, and the F 1 -norm (or the related Barron norm) for large width adaptivity. We show that the global minimizer of the regularized loss of DNNs can fit for example the composition of two functions f ∗ = h ◦ g from a small number of observations, assuming g is smooth/regular and reduces the dimensionality (e.g. g could be the modulo map of the symmetries of f ∗ ), so that h can be learned in spite of its low regularity. The measures of regularity we consider is the Sobolev norm with different levels of differentiability, which is well adapted to the F 1 norm. We compute scaling laws empirically and observe phase transitions depending on whether g or h is harder to learn, as predicted by our theory.

1 Introduction

One of the fundamental features of DNNs is their ability to generalize even when the number of 15 neurons (and of parameters) is so large that the network could fit almost any function [46]. Actually 16 DNNs have been observed to generalize best when the number of neurons is infinite [8, 21, 20]. 17 The now quite generally accepted explanation to this phenomenon is that DNNs have an implicit 18 bias coming from the training dynamic where properties of the training algorithm lead to networks 19 that generalize well. This implicit bias is quite well understood in shallow networks [11, 36], in 20 linear networks [24, 30], or in the NTK regime [28], but it remains ill-understood in the general deep 21 nonlinear case. 22

In both shallow networks and linear networks, one observes a bias towards small parameter norm 23

(either implicit [12] or explicit in the presence of weight decay [42]). Thanks to tools such as the 24 F 1 -norm [5], or the related Barron norm [44], or more generally the representation cost [14], it is 25 possible to describe the family of functions that can be represented by shallow networks or linear 26 networks with a finite parameter norm. This was then leveraged to prove uniform generalization 27 bounds (based on Rademacher complexity) over these sets [5], which depend only on the parameter 28 norm, but not on the number of neurons or parameters. 29

Similar bounds have been proposed for DNNs [7, 6, 39, 33, 25, 40], relying on different types of 30 norms on the parameters of the network. But it seems pretty clear that we have not yet identified 31 the 'right' complexity measure for deep networks, as there remains many issues: these bounds are 32

typically orders of magnitude too large [29, 23], and they tend to explode as the depth L grows [40]. 33

Two families of bounds are particularly relevant to our analysis: bounds based on covering numbers 34

which rely on the fact that one can obtain a covering of the composition of two function classes from 35

covering of the individual classes [7, 25], and path-norm bounds which extend the techniques behind 36 the F 1 -norm bound from shallow networks to the deep case [32, 6, 23]. 37

Another issue is the lack of approximation results to accompany these generalization bounds: many 38 different complexity measures R ( θ ) on the parameters θ of DNNs have been proposed along with 39 guarantees that the generalization gap will be small as long as R ( θ ) is bounded, but there are often 40 little to no result describing families of functions that can be approximated with a bounded R ( θ ) 41 norm. The situation is much clearer in shallow networks, where we know that certain Sobolev spaces 42 can be approximated with bounded F 1 -norm [5]. 43

We will focus on approximating composition of Sobolev functions, and obtaining close to optimal 44 rates. This is quite similar to the family of tasks considered [39], though the complexity measure we 45 consider is quite different, and does not require sparsity of the parameters. 46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

1.1 Contribution

We consider Accordion Networks (AccNets), which are the composition of multiple shallow networks f L :1 = f L ◦ · · · ◦ f 1 , we prove a uniform generalization bound L ( f L :1 ) -˜ L N ( f L :1 ) ≲ R ( f 1 , . . . , f L ) log N √ N , for a complexity measure

that depends on the F 1 -norms ∥ f ℓ ∥ F 1 and Lipschitz constanst Lip ( f ℓ ) of the subnetworks, and the intermediate dimensions d 0 , . . . , d L . This use of the F 1 -norms makes this bound independent of the widths w 1 , . . . , w L of the subnetworks, though it does depend on the depth L (it typically grows linearly in L which is still better than the exponential growth often observed).

Any traditional DNN can be mapped to an AccNet (and vice versa), by spliting the middle weight matrices W ℓ with SVD USV T into two matrices U √ S and √ SV T to obtain an AccNet with dimensions d ℓ = Rank W ℓ , so that the bound can be applied to traditional DNNs with bounded rank.

We then show an approximation result: any composition of Sobolev functions f ∗ = f ∗ L ∗ ◦ · · · ◦ f ∗ 1 can be approximated with a network with either a bounded complexity R ( θ ) or a slowly growing one. Thus under certain assumptions one can show that DNNs can learn general compositions of Sobolev functions. This ability can be interpreted as DNNs being able to learn symmetries, allowing them to avoid the curse of dimensionality in settings where kernel methods or even shallow networks suffer heavily from it.

Empirically, we observe a good match between the scaling laws of learning and our theory, as well as qualitative features such as transitions between regimes depending on whether it is harder to learn the symmetries of a task, or to learn the task given its symmetries.

2 Accordion Neural Networks and ResNets

Our analysis is most natural for a slight variation on the traditional fully-connected neural networks (FCNNs), which we call Accordion Networks, which we define here. Nevertheless, all of our results can easily be adapted to FCNNs.

Accordion Networks (AccNets) are simply the composition of L shallow networks, that is f L :1 = f L ◦ · · · ◦ f 1 where f ℓ ( z ) = W ℓ σ ( V ℓ z + b ℓ ) for the nonlinearity σ : R → R , the d ℓ × w ℓ matrix W ℓ , w ℓ × d ℓ -1 matrix V ℓ , and w ℓ -dim. vector b ℓ , and for the widths w 1 , . . . , w L and dimensions d 0 , . . . , d L . We will focus on the ReLU σ ( x ) = max { 0 , x } for the nonlinearity. The parameters θ are made up of the concatenation of all ( W ℓ , V ℓ , b ℓ ) . More generally, we denote f ℓ 2 : ℓ 1 = f ℓ 2 ◦ · · · ◦ f ℓ 1 for any 1 ≤ ℓ 1 ≤ ℓ 2 ≤ L .

We will typically be interested in settings where the widths w ℓ is large (or even infinitely large), while 77 the dimensions d ℓ remain finite or much smaller in comparison, hence the name accordion. 78

If we add residual connections, i.e. f res 1: L = ( f L + id ) ◦ · · · ◦ ( f 1 + id ) for the same shallow nets 79 f 1 , . . . , f L we recover the typical ResNets. 80

Remark. The only difference between AccNets and FCNNs is that each weight matrix M ℓ of the 81 FCNN is replaced by a product of two matrices M ℓ = V ℓ W ℓ -1 in the middle of the network (such a 82 structure has already been proposed [34]). Given an AccNet one can recover an equivalent FCNN by 83 choosing M ℓ = V ℓ W ℓ -1 , M 0 = V 0 and M L +1 = W L . In the other direction there could be multiple 84 ways to split M ℓ into the product of two matrices, but we will focus on taking V ℓ = U √ S and 85 W ℓ -1 = √ SV T for the SVD decomposition M ℓ = USV T , along with the choice d ℓ = Rank M ℓ . 86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

2.1 Learning Setup

We consider a traditional learning setup, where we want to find a function f : Ω ⊂ R d in → R d out that minimizes the population loss L ( f ) = E x ∼ π [ ℓ ( x, f ( x ))] for an input distribution π and a ρ -Lipschitz and ρ -bounded loss function ℓ ( x, y ) ∈ [0 , B ] . Given a training set x 1 , . . . , x N of size N we approximate the population loss by the empirical loss ˜ L N ( f ) = 1 N ∑ N i =1 ℓ ( x i , f ( x i )) that can be minimized.

To ensure that the empirical loss remains representative of the population loss, we will prove high probability bounds on the generalization gap ˜ L N ( f ) -L ( f ) uniformly over certain functions families f ∈ F .

For regression tasks , we assume the existence of a true function f ∗ and try to minimize the distance ℓ ( x, y ) = ∥ f ∗ ( x ) -y ∥ p for p ≥ 1 . If we assume that f ∗ ( x ) and y are uniformly bounded then one can easily show that ℓ ( x, y ) is bounded and Lipschitz. We are particularly interested in the cases p ∈ { 1 , 2 } , with p = 2 representing the classical MSE, and p = 1 representing a L 1 distance. The p = 2 case is amenable to 'fast rates' which take advantage of the fact that the loss increases very slowly around the optimal solution f ∗ , We do not prove such fast rates (even though it might be possible) so we focus on the p = 1 case.

̸

For classification tasks on k classes, we assume the existence of a 'true class' function f ∗ : Ω → { 1 , . . . , k } and want to learn a function f : Ω → R k such that the largest entry of f ( x ) is the f ∗ ( k ) -th entry. One can consider the hinge cost ℓ ( x, y ) = max { 0 , 1 -( y f ∗ ( k ) -max i = f ∗ ( x ) y i ) } , which is zero whenever the margin y f ∗ ( k ) -max i = f ∗ ( x ) y i is larger than 1 and otherwise equals 1 minus the margin. The hinge loss is Lipschitz and bounded if we assume bounded outputs y = f ( x ) . The cross-entropy loss also fits our setup.

̸

3 Generalization Bound for DNNs

The reason we focus on accordion networks is that there exists generalization bounds for shallow networks [5, 44], that are (to our knowledge) widely considered to be tight, which is in contrast to the deep case, where many bounds exist but no clear optimal bound has been identified. Our strategy is to extend the results for shallow nets to the composition of multiple shallow nets, i.e. AccNets. Roughly speaking, we will show that the complexity of an AccNet f θ is bounded by the sum of the complexities of the shallow nets f 1 , . . . , f L it is made of.

We will therefore first review (and slightly adapt) the existing generalization bounds for shallow networks in terms of their so-called F 1 -norm [5], and then prove a generalization bound for deep AccNets.

3.1 Shallow Networks

We call this the F 1 -ball since it can be thought of as the unit ball w.r.t. the F 1 -norm ∥ f ∥ F 1 which we 126 define as the smallest positive scalar s such that 1 1 s f ∈ B F 1 . For more details in the single output 127 case, see [5]. 128

129

The generalization gap over any F 1 -ball can be uniformly bounded with high probability:

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

Theorem 1. For any input distribution π supported on the L 2 ball B (0 , b ) with radius b , we have with probability 1 -δ , over the training samples x 1 , . . . , x N , that for all f ∈ B F 1 (0 , R ) = R · B F 1

This theorem is a slight variation of the one found in [5]: we simply generalize it to multiple outputs, and also prove it using a covering number argument instead of a direct computation of the Rademacher complexity, which will be key to obtaining a generalization bound for the deep case. But due to this change of strategy we pay a log N cost here and in our later results. We know that the log N term can be removed in Theorem 1 by switching to a Rademacher argument, but we do not know whether it can be removed in deep nets.

Notice how this bound does not depend on the width w , because the F 1 -norm (and the F 1 -ball) themselves do not depend on the width. This matches with empirical evidence that shows that increasing the width does not hurt generalization [8, 21, 20].

To use Theorem 1 effectively we need to be able to guarantee that the learned function will have a small enough F 1 -norm. The F 1 -norm is hard to compute exactly, but it is bounded by the parameter norm: if f ( x ) = Wσ ( V x + b ) , then ∥ f ∥ F 1 ≤ 1 2 ( ∥ W ∥ 2 F + ∥ V ∥ 2 F + ∥ b ∥ 2 ) , and this bound is tight if the width w is large enough and the parameters are chosen optimally. Adding weight decay/ L 2 -regularization to the cost then leads to bias towards learning with small F 1 norm.

3.2 Deep Networks

Since an AccNet is simply the composition of multiple shallow nets, the functions represented by an AccNet is included in the set of composition of F 1 balls. More precisely, if ∥ W ℓ ∥ 2 + ∥ V ℓ ∥ 2 + ∥ b ℓ ∥ 2 ≤ 2 R ℓ then f L :1 belongs to the set { g L ◦ · · · ◦ g 1 : g ℓ ∈ B F 1 (0 , R ℓ ) } for some R ℓ , which is width agnostic.

As already noticed in [7], the covering number number is well-behaved under composition, this allows us to bound the complexity of AccNets in terms of the individual shallow nets it is made up of:

Theorem 2. Consider an accordion net of depth L and widths d L , . . . , d 0 , with corresponding set of functions F = { f L :1 : ∥ f ℓ ∥ F 1 ≤ R ℓ , Lip ( f ℓ ) ≤ ρ ℓ } . With probability 1 -δ over the sampling of the training set X from the distribution π supported in B (0 , b ) , we have for all f ∈ F

Theorem 2 can be extended to ResNets ( f L + id ) ◦ · · · ◦ ( f 1 + id ) by simply replacing the Lipschitz constant Lip ( f ℓ ) by Lip ( f ℓ + id ) .

The Lipschitz constants Lip ( f ℓ ) are difficult to compute exactly, so it is easiest to simply bound it by the product of the operator norms Lip ( f ℓ ) ≤ ∥ W ℓ ∥ op ∥ V ℓ ∥ op , but this bound can often be quite loose. The fact that our bound depends on the Lipschitz constants rather than the operator norms ∥ W ℓ ∥ op , ∥ V ℓ ∥ op is thus a significant advantage.

This bound can be applied to a FCNNs with weight matrices M 1 , . . . , M L +1 , by replacing the middle 162 M ℓ with SVD decomposition USV T in the middle by two matrices W ℓ -1 = √ SV T and V ℓ = U √ S , 163 so that the dimensions can be chosen as the rank d ℓ = Rank M ℓ +1 . The Frobenius norm of the new 164 matrices equal the nuclear norm of the original one ∥ W ℓ -1 ∥ 2 F = ∥ V ℓ ∥ 2 F = ∥ M ℓ ∥ ∗ . Some bounds 165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

Vg=8,vh=1

vg=9,vh=9

WESAD

103

103

103

Model

AccNets

Complexity

102

102

102

101

Model

101

101

DNN

100

ACC

100

100

100

test_error

10-2.

Model

DNN

10-1

10-4

AccNets

10-1

Shallow

10-6

Kernel

102

103

104

102

103

104

102

103

104

105

N

N

N

assuming rank sparsity of the weight matrices also appear in [41]. And several recent results have shown that weight-decay leads to a low-rank bias on the weight matrices of the network [27, 26, 19] and replacing the Frobenius norm regularization with a nuclear norm regularization (according to the above mentioned equivalence) will only increase this low-rank bias.

We compute in Figure 1 the upper bound of Theorem 2 for both AccNets and DNNs, and even though we observe a very large gap (roughly of order 10 3 ), we do observe that it captures rate/scaling of the test error (the log-log slope) well. So this generalization bound could be well adapted to predicting rates, which is what we will do in the next section.

Remark. Note that if one wants to compute this upper bound in practical setting, it is important to train with L 2 regularization until the parameter norm also converges (this often happens after the train and test loss have converged). The intuition is that at initialization, the weights are initialized randomly, and they contribute a lot to the parameter norm, and thus lead to a larger generalization bound. During training with weight decay, these random initial weights slowly vanish, thus leading to a smaller parameter norm and better generalization bound. It might be possible to improve our generalization bounds to take into account the randomness at initialization to obtain better bounds throughout training, but we leave this to future work.

4 Breaking the Curse of Dimensionality with Compositionality

In this section we study a large family of functions spaces, obtained by taking compositions of Sobolev balls. We focus on this family of tasks because they are well adapted to the complexity measure we have identified, and because kernel methods and even shallow networks do suffer from the curse of dimensionality on such tasks, whereas deep networks avoid it (e.g. Figure 1).

More precisely, we will show that these sets of functions can be approximated by a AccNets with 187 bounded (or in some cases slowly growing) complexity measure 188

This will then allow us show that AccNets can (assuming global convergence) avoid the curse of 189 dimensionality, even in settings that should suffer from the curse of dimensionality, when the input 190 dimension is large and the function is not very smooth (only a few times differentiable). 191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

EmpericalErrorRate

TheoreticalErrorRate

DifferenceinEmperical&TheoreticalErrorRate

0.5

0.0

1.0

1.5

0.4

2.0

2.5

-0.1

3.0

3.5

0.2

4.0

4.5

-0.2

5.0

5.5

- 0.0

6.0

6.5

-0.3

7.0

7.5

0.2

8.0

8.5

-0.4

9.5

9.0

0.4

10.0

0.5

2

un

c

5

6

5

6

10.

0.51.52.53.54.55.56.57.58.59.5

Vh

0.51.52.53.54.55.56.57.58.59.5

.

4.1 Composition of Sobolev Balls

The family of Sobolev norms capture some notion of regularity of a function, as it measures the size of its derivatives. The Sobolev norm of a function f : R d in → R is defined in terms of its derivatives ∂ α x f for some d in -multi-index α , namely the W ν,p ( π ) -Sobolev norm with integer ν and p ≥ 1 is defined as

Note that the derivative ∂ α x f only needs to be defined in the 'weak' sense, which means that even non-differentiable functions such as the ReLU functions can actually have finite Sobolev norm.

The Sobolev balls B W ν,p ( π ) (0 , R ) = { f : ∥ f ∥ W ν,p ( π ) ≤ R } are a family of function spaces with a range of regularity (the larger ν , the more regular). This regularity makes these spaces of functions learnable purely from the fact that they enforce the function f to vary slowly as the input changes. Indeed we can prove the following generalization bound:

Proposition 3. Given a distribution π with support the L 2 ball with radius b, we have that with probability 1 -δ for all functions f ∈ F = { f : ∥ f ∥ W ν, 2 ≤ R, ∥ f ∥ ∞ ≤ R }

But this result also illustrates the curse of dimensionality: the differentiability ν needs to scale with the input dimension d in to obtain a reasonable rate. If instead ν is constant and d in grows, then the number of datapoints N needed to guarantee a generalization gap of at most ϵ scales exponentially in d in , i.e. N ∼ ϵ -d in ν . One way to interpret this issue is that regularity becomes less and less useful the larger the dimension: knowing that similar inputs have similar outputs is useless in high dimension where the closest training point x i to a test point x is typically very far away.

4.1.1 Breaking the Curse of Dimensionality with Compositionality

To break the curse of dimensionality, we need to assume some additional structure on the data or task which introduces an 'intrinsic dimension' that can be much lower than the input dimension d in :

Manifold hypothesis : If the input distribution lies on a d surf -dimensional manifold, the error rates typically depends on d surf instead of d in [38, 10].

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

Error rate for Shallow Net

Error rate for Kernel method

0.0

Difference in Shallow Net & AccNet

Difference in Kernel Method & AccNet

0.4

0.4

0.1

m

- 0.2

- 0.2

0.2

- 0.0

- 0.0

0.3

-0.2

0.2

0.4

-0.4

-0.4

0.5

2

8

6

10

6

8

6

10

6

7

8

6

10

8

6

10

Vh

.

Known Symmetries: If f ∗ ( g · x ) = f ∗ ( x ) for a group action · w.r.t. a group G , then f ∗ can be written as the composition of a modulo map g ∗ : R d in → R d in / G which maps pairs of inputs which are equivalent up to symmetries to the same value (pairs x, y s.t. y = g · x for some g ∈ G ), and then a second function h ∗ : R d in / G → R d out , then the complexity of the task will depend on the dimension of the modulo space R d in / G which can be much lower. If the symmetry is known, then one can for example fix g ∗ and only learn h ∗ (though other techniques exist, such as designing kernels or features that respect the same symmetries) [31].

Symmetry Learning: However if the symmetry is not known then both g ∗ and h ∗ have to be learned, and this is where we require feature learning and/or compositionality. Shallow networks are able to learn translation symmetries, since they can learn so-called low-index functions which satisfy f ∗ ( x ) = f ∗ ( Px ) for some projection P (with a statistical complexity that depends on the dimension of the space one projects into, not the full dimension [5, 2]). Low-index functions correspond exactly to the set of functions that are invariant under translation along the kernel ker P . To learn general symmetries, one needs to learn both h ∗ and the modulo map g ∗ simultaneously, hence the importance of feature learning.

For g ∗ to be learnable efficiently, it needs to be regular enough to not suffer from the curse of dimensionality, but many traditional symmetries actually have smooth modulo maps, for example the modulo map g ∗ ( x ) = ∥ x ∥ 2 for rotation invariance. This can be understood as a special case of composition of Sobolev functions, whose generalization gap can be bounded:

Theorem 4. Consider the function set F = F L ◦ · · · ◦ F 1 where F ℓ = { f ℓ : R d ℓ -1 → R d ℓ s.t. ∥ f ℓ ∥ W ν ℓ , 2 ≤ R ℓ , ∥ f ℓ ∥ ∞ ≤ b ℓ , Lip ( f ℓ ) ≤ ρ ℓ } , and let r min = min ℓ r ℓ for r ℓ = ν ℓ d ℓ -1 , then with probability 1 -δ we have for all f ∈ F

where C ℓ depends only on d ℓ -1 , d ℓ , ν ℓ , b ℓ -1 .

We see that only the smallest ratio r min matters when it comes to the rate of learning. And actually the above result could be slightly improved to show that the sum over all layers could be replaced by a sum over only the layers where the ratio r ℓ leads to the worst rate E r ℓ ( N ) = E r min ( N ) (and the other layers contribute an asymptotically subdominant amount).

Coming back to the symmetry learning example, we see that the hardness of learning a function of the type f ∗ = h ◦ g with inner dimension d mid and regularities ν g and ν h , the error rate will be (up to log terms) N -min { 1 2 , νg d in , ν h d mid } . This suggests the existence of three regimes depending on which term attains the minimum: a regime where both g and h are easy to learn and we have N -1 2 learning, a regime g is hard, and a regime where h is hard. The last two regimes differentiate between tasks

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

where learning the symmetry is hard and those where learning the function knowing its symmetries is hard.

In contrast, without taking advantage of the compositional structure, we expect f ∗ to be only min { ν g , ν h } times differentiable, so trying to learn it as a single Sobolev function would lead to an error rate of N -min { 1 2 , min { νg ,ν h } d in } = N -min { 1 2 , νg d in , ν h d in } which is no better than the compositional rate, and is strictly worse whenever ν h < ν g and ν h d in < 1 2 (we can always assume d mid ≤ d in since one could always choose d = id ).

Furthermore, since multiple compositions are possible, one can imagine a hierarchy of symmetries that slowly reduce the dimensionality with less and less regular modulo maps. For example one could imagine a composition f L ◦ · · · ◦ f 1 with dimensions d ℓ = d 0 2 -ℓ and regularities ν ℓ = d 0 2 -ℓ so that the ratios remain constant r ℓ = d 0 2 -ℓ d 0 2 -ℓ +1 = 1 2 , leading to an almost parametric rate of N -1 2 log N even though the function may only be d 0 2 -L times differentiable. Without compositionality, the rate would only be N -2 -L .

Remark. In the case of a single Sobolev function, one can show that the rate E ν / d ( N ) is in some sense optimal, by giving an information theoretic lower bound with matching rate. A naive argument suggests that the rate of E min { r 1 ,...,r L } ( N ) should similarly be optimal: assume that the minimum r ℓ is attained at a layer ℓ , then one can consider the subset of functions such that the image f ℓ -1:1 ( B (0 , r )) contains a ball B ( z, r ′ ) ⊂ R d ℓ -1 and that the function f L : ℓ +1 is β -non-contracting ∥ f L : ℓ +1 ( x ) -f L : ℓ +1 ( y ) ∥ ≥ β ∥ x -y ∥ , then learning f L :1 should be as hard as learning f ℓ over the ball B ( z, r ′ ) (more rigorously this could be argued from the fact that any ϵ -covering of f L :1 can be mapped to an ϵ / β -covering of f ℓ ), thus forcing a rate of at least E r ℓ ( N ) = E min { r 1 ,...,r L } ( N ) .

An analysis of minimax rates in a similar setting has been done in [22].

4.2 Breaking the Curse of Dimensionality with AccNets

Now that we know that composition of Sobolev functions can be easily learnable, even in settings where the curse of dimensionality should make it hard to learn them, we need to find a model that can achieve those rates. Though many models are possible 2 , we focus on DNNs, in particular AccNets. Assuming convergence to a global minimum of the loss of sufficiently wide AccNets with two types of regularization, one can guarantee close to optimal rates:

Theorem 5. Given a true function f ∗ L ∗ :1 = f ∗ L ∗ ◦ · · · ◦ f ∗ 1 going through the dimensions d ∗ 0 , . . . , d ∗ L ∗ , along with a continuous input distribution π 0 supported in B (0 , b 0 ) , such that the distributions π ℓ of f ∗ ℓ ( x ) (for x ∼ π 0 ) are continuous too and supported inside B (0 , b ℓ ) ⊂ R d ∗ ℓ . Further assume that there are differentiabilities ν ℓ and radii R ℓ such that ∥ f ∗ ℓ ∥ W ν ℓ , 2 ( B (0 ,b ℓ )) ≤ R ℓ , and ρ ℓ such that Lip ( f ∗ ℓ ) ≤ ρ ℓ . For an infinite width AccNet with L ≥ L ∗ and dimensions d ℓ ≥ d ∗ 1 , . . . , d ∗ L ∗ -1 , we have for the ratios ˜ r ℓ = ν ℓ d ∗ ℓ +3 :

At a global minimizer ˆ f L :1 of the regularized loss f 1 , . . . , f L ↦→ ˜ L N ( f L :1 ) + λ ∏ L ℓ =1 Lip ( f ℓ ) ∑ L ℓ =1 ∥ f ℓ ∥ F 1 Lip ( f ℓ ) √ d ℓ -1 + d ℓ , we have L ( ˆ f L :1 ) = ˜ O ( N -min { 1 2 , ˜ r 1 ,..., ˜ r L ∗ } ) .

At a global minimizer ˆ f L :1 of the regularized loss f 1 , . . . , f L ↦→ ˜ L N ( f L :1 )+ λ ∏ L ℓ =1 ∥ f ℓ ∥ F 1 , we have L ( ˆ f L :1 ) = ˜ O ( N -1 2 + ∑ L ∗ ℓ =1 max { 0 , ˜ r ℓ -1 2 } ) .

There are a number of limitations to this result. First we assume that one is able to recover the global minimizer of the regularized loss, which should be hard in general 3 (we already know from [5] that this is NP-hard for shallow networks and a simple F 1 -regularization). Note that it is sufficient to recover a network f L :1 whose regularized loss is within a constant of the global minimum, which

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

might be easier to guarantee, but should still be hard in general. The typical method of training with GD on the regularized loss is a greedy approach, which might fail in general but could recover almost optimal parameters under the right conditions (some results suggest that training relies on first order correlations to guide the network in the right direction [2, 1, 35]).

We propose two regularizations because they offer a tradeoff:

First regularization: The first regularization term leads to almost optimal rates, up to the change from r ℓ = ν ℓ d ∗ ℓ to r ℓ = ν ℓ d ∗ ℓ +3 which is negligible for large dimensions d ℓ and differentiabilities ν ℓ . The first problem is that it requires an infinite width at the moment, because we were not able to prove that a function with bounded F 1 -norm and Lipschitz constant can be approximated by a sufficiently wide shallow networks with the same (or close) F 1 -norm and Lipschitz constant (we know from [5] that it is possible without preserving the Lipschitzness). We are quite hopeful that this condition might be removed in future work.

The second and more significant problem is that the Lipschitz constants Lip ( f ℓ ) are difficult to optimize over. For finite width networks it is in theory possible to take the max over all linear regions, but the complexity might be unreasonable. It might be more reasonable to leverage an implicit bias instead, such as a large learning rate, because a large Lipschitz constant implies that the nework is sensible to small changes in its parameters, so GD with a large learning rate should only converge to minima with a small Lipschitz constant (such a bias is described in [26]). It might also be possible to replace the Lipschitz constant in our generalization bounds, possibly along the lines of [43].

Second regularization: The second regularization term actually does not require an infinite width, only a sufficiently large one. Also its regularization term is equivalent to ∏ ( ∥ W ℓ ∥ 2 + ∥ V ℓ ∥ 2 + ∥ b ℓ ∥ 2 ) which is much closer to the traditional L 2 -regularization (and actually one could prove the same or very similar rates for L 2 -regularization). The issue is that it lead to rates that could be far from optimal depending on the ratios ˜ r ℓ : it recovers the same rate as the first regularization term if no more than one ratio ˜ r ℓ is smaller than 1 2 , but if many of these ratios are above 1 2 , it can be arbitrarily smaller.

In Figure 2, we compare the empirical rates (by doing a linear fit on a log-log plot of test error as a function of N ) and the predicted optimal rates min { 1 2 , ν g d in , ν h d mid } and observe a pretty good match. Though surprisingly, it appears the the empirical rates tend to be slightly better than the theoretical ones.

Remark. As can be seen in the proof of Theorem5, when the depth L is strictly larger than the true depth L ∗ , one needs to add identity layers, leading to a so-called Bottleneck structure, which was proven to be optimal and observed empirically in [27, 26, 45]. These identity layers add a term that scales linearly in the additional depth ( L -L ∗ ) d ∗ min √ to the first regularization, and an exponential prefactor (2 d min ) bottleneck structure, or simply by switching to ResNets.

N ∗ L -L ∗ to the second. It might be possible to remove these factors by leveraging the

5 Conclusion

We have given a generalization bound for Accordion Networks and as an extension Fully-Connected networks. It depends on F 1 -norms and Lipschitz constants of its shallow subnetworks. This allows us to prove under certain assumptions that AccNets can learn general compositions of Sobolev functions efficiently, making them able to break the curse of dimensionality in certain settings, such as in the presence of unknown symmetries.

References

Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory , pages 4782-4887. PMLR, 2022.

Emmanuel Abbe, Enric Boix-Adserà, Matthew Stewart Brennan, Guy Bresler, and Dheeraj Mysore Nagaraj. The staircase property: How hierarchical structure can guide deep learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via 341 over-parameterization. pages 242-252, 2019. 342

Kendall Atkinson and Weimin Han. Spherical harmonics and approximations on the unit 343 sphere: an introduction , volume 2044. Springer Science & Business Media, 2012. 344

Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal 345 of Machine Learning Research , 18(1):629-681, 2017. 346

Andrew R Barron and Jason M Klusowski. Complexity, statistical risk, and metric entropy of 347 deep nets using total path variation. stat , 1050:6, 2019. 348

349

350

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

M. S. Birman and M. Z. Solomjak. Piecewise-polynomial approximations of functions of the classes W α p . Mathematics of The USSR-Sbornik , 2:295-317, 1967.

Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Nonparametric regression on low-dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: A Journal of the IMA , 11(4):1203-1253, 2022.

Lénaïc Chizat and Francis Bach. On the Global Convergence of Gradient Descent for Overparameterized Models using Optimal Transport. In Advances in Neural Information Processing Systems 31 , pages 3040-3050. Curran Associates, Inc., 2018.

Lénaïc Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 1305-1338. PMLR, 09-12 Jul 2020.

Feng Dai. Approximation theory and harmonic analysis on spheres and balls . Springer, 2013.

Zhen Dai, Mina Karzand, and Nathan Srebro. Representation costs of linear neural networks: Analysis and design. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Andreas Damianou and Neil D. Lawrence. Deep Gaussian processes. In Carlos M. Carvalho and Pradeep Ravikumar, editors, Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics , volume 31 of Proceedings of Machine Learning Research , pages 207-215, Scottsdale, Arizona, USA, 29 Apr-01 May 2013. PMLR.

Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations , 2019.

I. Dumer, M.S. Pinsker, and V.V. Prelov. On coverings of ellipsoids in euclidean spaces. IEEE Transactions on Information Theory , 50(10):2348-2356, 2004.

Lawrence C Evans. Partial differential equations , volume 19. American Mathematical Society, 2022.

Tomer Galanti, Zachary S Siegel, Aparna Gupte, and Tomaso Poggio. Sgd and weight decay provably induce a low-rank bias in neural networks. arXiv preprint arXiv:2206.05794 , 2022.

Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics: Theory and Experiment , 2020(2):023401, 2020.

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

Mario Geiger, Stefano Spigler, Stéphane d'Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart. Jamming transition as a paradigm to understand the loss landscape of deep neural networks. Physical Review E , 100(1):012115, 2019.

Matteo Giordano, Kolyan Ray, and Johannes Schmidt-Hieber. On the inability of gaussian process regression to optimally learn compositional functions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Antoine Gonon, Nicolas Brisebarre, Elisa Riccietti, and Rémi Gribonval. A path-norm toolkit for modern networks: consequences, promises and challenges. In The Twelfth International Conference on Learning Representations , 2023.

Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1832-1841. PMLR, 10-15 Jul 2018.

Daniel Hsu, Ziwei Ji, Matus Telgarsky, and Lan Wang. Generalization bounds via distillation. In International Conference on Learning Representations , 2021.

Arthur Jacot. Bottleneck structure in learned features: Low-dimension vs regularity tradeoff. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 23607-23629. Curran Associates, Inc., 2023.

Arthur Jacot. Implicit bias of large depth networks: a notion of rank for nonlinear functions. In The Eleventh International Conference on Learning Representations , 2023.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In Advances in Neural Information Processing Systems 31 , pages 8580-8589. Curran Associates, Inc., 2018.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178 , 2019.

Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations , 2020.

Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics , 65(10):1331-1398, 2012.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on learning theory , pages 1376-1401. PMLR, 2015.

Atsushi Nitanda and Taiji Suzuki. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime. In International Conference on Learning Representations , 2020.

Greg Ongie and Rebecca Willett. The role of linear layers in nonlinear interpolating networks. arXiv preprint arXiv:2202.00856 , 2022.

Leonardo Petrini, Francesco Cagnetta, Umberto M Tomasini, Alessandro Favero, and Matthieu Wyart. How deep neural networks learn compositional data: The random hierarchy model. arXiv preprint arXiv:2307.02129 , 2023.

Grant Rotskoff and Eric Vanden-Eijnden. Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks. In Advances in Neural Information Processing Systems 31 , pages 7146-7155. Curran Associates, Inc., 2018.

Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM International Conference on Multimodal Interaction , ICMI '18, page 400-408, New York, NY, USA, 2018. Association for Computing Machinery.

Johannes Schmidt-Hieber. Deep relu network approximation of functions on a manifold. arXiv 435 preprint arXiv:1908.00695 , 2019. 436

Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU 437 activation function. The Annals of Statistics , 48(4):1875 - 1897, 2020. 438

Mark Sellke. On size-independent sample complexity of relu networks. Information Processing 439 Letters , page 106482, 2024. 440

441

442

443

Taiji Suzuki, Hiroshi Abe, and Tomoaki Nishimura. Compression based bound for noncompressed network: unified generalization error analysis of large compressible deep neural network. In International Conference on Learning Representations , 2020.

444

445

446

447

448

449

450

451

452

Zihan Wang and Arthur Jacot. Implicit bias of SGD in l 2 -regularized linear DNNs: Oneway jumps from high to low rank. In The Twelfth International Conference on Learning Representations , 2024.

Colin Wei and Tengyu Ma. Data-dependent sample complexity of deep neural networks via lipschitz augmentation. Advances in Neural Information Processing Systems , 32, 2019.

E Weinan, Chao Ma, and Lei Wu. Barron spaces and the compositional function spaces for neural network models. arXiv preprint arXiv:1906.08039 , 2019.

Yuxiao Wen and Arthur Jacot. Which frequencies do cnns need? emergent bottleneck structure in feature learning. to appear at ICML , 2024.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding 453 deep learning requires rethinking generalization. ICLR 2017 proceedings , Feb 2017. 454

The Appendix is structured as follows: 455

456

In Section A, we describe the experimental setup and provide a few additional experiments.

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

In Section B, we prove Theorems 1 and 2 from the main.

In Section C, we prove Proposition 3 and Theorem 4.

In Section D, we prove Theorem 5 and other approximation results concerning Sobolev functions.

In Section E, we prove a few technical results on the covering number.

A Experimental Setup 4

In this section, we review our numerical experiments and their setup both on synthetic and real-world datasets in order to address theoretical results more clearly and intuitively.

A.1 Dataset

A.1.1 Emperical Dataset

The Matérn kernel is considered a generalization of the radial basis function (RBF) kernel. It controls the differentiability, or smoothness, of the kernel through the parameter ν . As ν →∞ , the Matérn kernel converges to the RBF kernel, and as ν → 0 , it converges to the Laplacian kernel, a 0-differentiable kernel. In this study, we utilized the Matérn kernel to generate Gaussian Process (GP) samples based on the composition of two Matérn kernels, K g and K h , with varying differentiability in the range [0.5,10]×[0.5,10]. The input dimension ( d in ) of the kernel, the bottleneck mid-dimension ( d mid ), and the output dimension ( d out ) are 15, 3, and 20, respectively.

This outlines the general procedure of our sampling method for synthetic data:

Sample the training dataset X ∈ R D × d in 2. From X, compute the D × D kernel K g with given ν g 3. From K g , sample Z ∈ R D × d mid with columns sampled from the Gaussian N (0 , K g ) . 4. From Z , compute K g with given ν h D × d

From K h , sample the test dataset Y ∈ R out with columns sampled from the Gaussian N (0 , K h ) .

We utilized four AMD Opteron 6136 processors (2.4 GHz, 32 cores) and 128 GB of RAM to generate our synthetic dataset. The maximum possible dataset size for 128 GB of RAM is approximately 52,500; however, we opted for a synthetic dataset size of 22,000 due to the computational expense associated with sampling the Matérn kernel. This decision was made considering the time complexity of O ( n 3 ) and the space complexity of O ( n 2 ) involved. Out of the 22,000 dataset points, 20,000 were allocated for training data, and 2,000 were used for the test dataset

A.1.2 Real-world dataset: WESAD

In our study, we utilized the Wearable Stress and Affect Detection (WESAD) dataset to train our AccNets for binary classification. The WESAD dataset, which is publicly accessible, provides multimodal physiological and motion data collected from 15 subjects using devices worn on the wrist and chest. For the purpose of our experiment, we specifically employed the Empatica E4 wrist device to distinguish between non-stress (baseline) and stress conditions, simplifying the classification task to these two categories.

After preprocessing, the dataset comprised a total of 136,482 instances. We implemented a train-test split ratio of approximately 75:25, resulting in 100,000 instances for the training set and 36,482 instances for the test set. The overall hyperparameters and architecture of the AccNets model applied to the WESAD dataset were largely consistent with those used for our synthetic data. The primary differences were the use of 100 epochs for each iteration of Ni from Ns, and a learning rate set to 1e-5.

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

A.2 Model setups

To investigate the scaling law of test error for our synthetic data, we trained models using N i datapoints from our training data, where N = [100 , 200 , 500 , 1000 , 2000 , 5000 , 10000 , 20000] . The models employed for this analysis included the kernel method, shallow networks, fully connected deep neural networks (FC DNN), and AccNets. For FC DNN and AccNets, we configured the network depth to 12 layers, with the layer widths set as [ d in , 500 , 500 , ..., 500 , d out ] for DNNs, and [ d i n, 900 , 100 , 900 , ..., 100 , 900 , d out ] for AccNets.

To ensure a comparable number of neurons, the width for the shallow networks was set to 50,000, resulting in dimensions of [ d in , 50000 , d out ] .

We utilized ReLU as the activation function and L 1 -norm as the cost function, with the Adam optimizer. The total number of batch was set to 5, and the training process was conducted over 3600 epochs, divided into three phases. The detailed optimizer parameters are as follows:

For the first 1200 epochs: learning rate ( lr ) = 1 . 5 ∗ 0 . 001 , weight decay = 0

For the second 1200 epochs: lr = 0 . 4 ∗ 0 . 001 , weight decay = 0 . 002

For the final 1200 epochs: lr = 0 . 1 ∗ 0 . 001 , weight decay = 0 . 005

We conducted experiments utilizing 12 NVIDIA V100 GPUs (each with 32GB of memory) over a period of 6.3 days to train the synthetic dataset. In contrast, training the WESAD dataset required only one hour on a single V100 GPU.

A.3 Additional experiments

B AccNet Generalization Bounds

The proof of generalization for shallow networks (Theorem 1) is the special case L = 1 of the proof of Theorem 2, so we only prove the second:

Theorem 6. Consider an accordion net of depth L and widths d L , . . . , d 0 , with corresponding set of functions F = { f L :1 : ∥ f ℓ ∥ F 1 ≤ R ℓ , Lip ( f ℓ ) ≤ ρ ℓ } with input space Ω = B (0 , r ) . For any ρ -Lipschitz loss function ℓ ( x, f ( x )) with | ℓ ( x, y ) | ≤ c 0 , we know that with probability 1 -δ over the sampling of the training set X from the distribution π , we have for all f ∈ F

Proof. The strategy is: (1) prove a covering number bound on F (2) use it to obtain a Rademacher 526 complexity bound, (3) use the Rademacher complexity to bound the generalization error. 527

DNN N 10000 vg 2.0, vh2.0

AccNet N 10000 vg 2.0, vh2.0

DNN N 200 vg 10.0, vh10.0

AccNet N 200 vg 10.0, vh10.0

2.5

2.0

1.5

1.0

0.5

0.0

0

2

4

6

8

10

12

0

2

4

6

8

10

0

2

4

6

8

10

12

0

2

4

6

8

10

.

(1) We define f ℓ = V ℓ ◦ σ ◦ W ℓ so that f θ = f L :1 = f L ◦ · · · ◦ f 1 . First notice that we can write each 528 f ℓ as convex combination of its neurons: 529

for ¯ w ℓ,i = w ℓ,i ∥ w ℓ,i ∥ , ¯ v ℓ,i = v ℓ,i ∥ v ℓ,i ∥ , R ℓ = ∑ ℓ i =1 ∥ v ℓ,i ∥ ∥ w ℓ,i ∥ and c ℓ,i = 1 R ℓ ∥ v ℓ,i ∥ ∥ w ℓ,i ∥ .

530

Let us now consider a sequence ϵ k = 2 -k for k = 0 , . . . , K and define ˜ v ( k ) ℓ,i , ˜ w ( k ) ℓ,i to be the ϵ k -covers 531 of ¯ v ℓ,i , ¯ w ℓ,i , furthermore we may choose ˜ v (0) ℓ,i = ˜ w (0) ℓ,i = 0 since every unit vector is within a ϵ 0 = 1 532 distance of the origin. We will now show that on can approximate f θ by approximating each of the f ℓ 533 by functions of the form 534

for indices i ( k ) ℓ,m = 1 , . . . , w ℓ choosen adequately. Notice that the number of functions of this type 535 equals the number of M k,ℓ quadruples (˜ v ( k ) ℓ,i ( k ) ℓ,m , ˜ w ( k ) T ℓ,i ( k ) ℓ,m , ˜ v ( k -1) ℓ,i ( k ) ℓ,m , ˜ w ( k -1) T ℓ,i ( k ) ℓ,m ) where these vectors belong 536 to the ϵ k - resp. ϵ k -1 -coverings of the d in - resp. d out -dimensional unit sphere. Thus the number of 537 such functions is bounded by 538

and we have this choice for all ℓ = 1 , . . . , L . We will show that with sufficiently large M k,ℓ this set 539 of functions ϵ -covers F which then implies that 540

We will use the probabilistic method to find the right indices i ( k ) ℓ,m to approximate a function f ℓ = 541 R ℓ ∑ w ℓ i =1 c ℓ,i ¯ v ℓ,i σ ( ¯ w T ℓ,i x ) with a function ˜ f ℓ . We take all i ( k ) ℓ,m to be i.i.d. equal to the index i = 542 1 , · · · , w ℓ with probability c ℓ,i , so that in expectation 543

We will show that this expectation is O ( ϵ K ℓ ) -close to f ℓ and that the variance of ˜ f ℓ goes to zero as 544 the M ℓ,k s grow, allowing us to bound the expected error E ∥ ∥ ∥ f L :1 -˜ f L :1 ∥ ∥ ∥ 2 π ≤ ϵ 2 which then implies 545 that there must be at least one choice of indices i ( k ) ℓ,m such that ∥ ∥ ∥ f L :1 -˜ f L :1 ∥ ∥ ∥ π ≤ ϵ . 546

Let us first bound the distance 547

Then we bound the trace of the covariance of ˜ f ℓ which equals the expected square distance between 548 ˜ f ℓ and its expectation: 549

Putting both together, we obtain 550

We will now use this bound, together with the Lipschitzness of f ℓ to bound the error 551 E ∥ ∥ ∥ f L :1 ( x ) -˜ f L :1 ( x ) ∥ ∥ ∥ 2 . We will do this by induction on the distances E ∥ ∥ ∥ f ℓ :1 ( x ) -˜ f ℓ :1 ( x ) ∥ ∥ ∥ 2 . 552 We start by 553

And for the induction step, we condition on the layers f ℓ -1:1 554

Now since 555

we obtain that 556

We define ˜ ρ 2 ℓ = ρ 2 ℓ [ 1 + 4 R 2 ℓ ρ 2 ℓ ( ϵ 2 K ℓ +9 ∑ K ℓ k =1 ϵ 2 k M k,ℓ )] and obtain 557

Thus for any distribution π over the ball B (0 , r ) , there is a choice of indices i ( k ) ℓ,m such that 558

We now simply need to choose K ℓ and M k,ℓ adequately. To reach an error of 2 ϵ , we choose 559

Given s 0 = ∑ ∞ k =1 k 2 -k ≈ 1 . 3473 < ∞ , we define 561

where ρ L :1 = ∏ L ℓ =1 ρ ℓ . Notice that that ϵ 2 K ℓ ≤ 1 4 ρ 2 L :1 r 2 ( ∑ L ℓ ′ =1 R ℓ ′ ρ ℓ ′ √ d ℓ ′ + d ℓ ′ -1 ) ρ ℓ d ℓ + d ℓ -1 R ℓ ϵ 2 . 560 √

So that for all ℓ 562

Now this also implies that 563

and thus 564

Putting it all together, we obtain that 565

Now since log N 2 ( S d ℓ -1 , ϵ ) = d ℓ log ( 1 ϵ +1 ) and 566

we have 567

The diameter of F is smaller than ρ L :1 r , so for all δ ≥ ρ L :1 r , log N 2 ( F , δ ) = 0 . For all δ ≤ ρ L :1 r 568 we choose ϵ = δ √ 2 e so that √ 2 exp ( ϵ 2 ρ 2 L :1 r 2 ) ϵ ≤ δ and therefore 569

(2) Our goal now is to use chaining / Dudley's theorem to bound the Rademacher complexity 570 R ( F ( X )) evaluated on a set X of size N (e.g. Lemma 27.4 in [Understanding Machine Learning]) 571 of our set: 572

Lemma 7. Let c = max f ∈F 1 √ N ∥ f ( X ) ∥ , then for any integer M > 0 , 573

To apply it to our setting, first note that for all x ∈ B (0 , r ) , ∥ f L :1 ( x ) ∥ ≤ ρ L :1 r so that c = 574 max f ∈F 1 √ N ∥ f ( X ) ∥ ≤ ρ L :1 r , we then have 575

Taking M = ⌈ -log 2 ( 72 √ N s 0 e ∑ L ℓ ′ =1 R ℓ ′ ρ ℓ ′ √ d ℓ ′ + d ℓ ′ -1 )⌉ , we obtain 576

(3) For any ρ -Lipschitz loss function ℓ ( x, f ( x )) with | ℓ ( x, y ) | ≤ c 0 , we know that with probability 577 1 -δ over the sampling of the training set X from the distribution π , we have for all f ∈ F 578

579

C Composition of Sobolev Balls 580

Proposition 8 (Proposition 3 from the main.) . Given a distribution π with support in B (0 , r ) , we 581 have that with probability 1 -δ for all functions f ∈ F = { f : ∥ f ∥ W ν, 2 ≤ R, ∥ f ∥ ∞ ≤ R } 582

where E r ( N ) = N -1 2 if r > 1 2 , E r ( N ) = N -1 2 log N if r = 1 2 , and E r ( N ) = N -r if r < 1 2 . 583

Proof. (1) We know from Theorem 5.2 of [9] that the Sobolev ball B W ν, 2 (0 , R ) over any d -584 dimensional hypercube Ω satisfies 585

for a constant c and any measure π supported in the hypercube. 586

(2) By Dudley's theorem we can bound the Rademacher complexity of our function class B ( X ) 587 evaluated on any training set X : 588

If 2 ν = d , we take M = 1 2 log N and obtain the bound 589

If 2 ν > d , we take M = ∞ and obtain the bound 590

If 2 ν < d , we take M = ν d log N and obtain the bound 591

Putting it all together, we obtain that R ( B ( X )) ≤ C 1 E ν / d ( N ) . 592

(3) For any ρ -Lipschitz loss function ℓ ( x, f ( x )) with | ℓ ( x, y ) | ≤ c 0 , we know that with probability 593 1 -δ over the sampling of the training set X from the distribution π , we have for all f ∈ F 594

595

Proposition 9. Let F 1 , . . . , F L be set of functions mapping through the sets Ω 0 , . . . , Ω L , then if all 596 functions in F ℓ are ρ ℓ -Lipschitz, we have 597

Proof. For any distribution π 0 on Ω there is a ϵ 1 -covering ˜ F 1 of F 1 with ∣ ∣ ∣ ˜ F 1 ∣ ∣ ∣ ≤ N 2 ( F 1 , ϵ 1 ) then 598 for any ˜ f 1 ∈ ˜ F 1 we choose a ϵ 2 -covering ˜ F 2 w.r.t. the measure π 1 which is the measure of f 1 ( x ) if 599 x ∼ π 0 of F 2 with ∣ ∣ ∣ ˜ F 2 ∣ ∣ ∣ ≤ N 2 ( F 2 , ϵ ) , and so on until we obtain coverings for all ℓ . Then the set 600 ˜ F = { ˜ f L ◦ · · · ◦ ˜ f 1 : ˜ f 1 ∈ ˜ F 1 , . . . , ˜ f L ∈ ˜ F L } is a ∑ L ℓ =1 ρ L : ℓ +1 ϵ ℓ -covering of F = F L ◦ · · · ◦ F 1 , 601

indeed for any f = f L :1 we choose ˜ f 1 ∈ ˜ F 1 , . . . , ˜ f L ∈ ˜ F L that cover f 1 , . . . , f L , then ˜ f L :1 covers 602 f L :1 : 603

and log cardinality of the set ˜ F is bounded ∑ L ℓ =1 log N 2 ( F ℓ , ϵ ℓ ) .

Theorem 10. Let F = F L ◦ · · · ◦ F 1 where F ℓ = { f ℓ : R d ℓ -1 → R d ℓ s.t. ∥ f ℓ ∥ W ν ℓ , 2 ≤ R ℓ , ∥ f ℓ ∥ ∞ ≤ b ℓ , Lip ( f ℓ ) ≤ ρ ℓ } , and let r ∗ = min ℓ r ℓ for r ℓ = ν ℓ d ℓ -1 , then with probability 1 -δ we have for all f ∈ F

where C ℓ depends only on d ℓ -1 , d ℓ , ν ℓ , b ℓ -1 .

Proof. (1) We know from Theorem 5.2 of [9] that the Sobolev ball B W ν ℓ , 2 (0 , R ℓ ) over any d ℓ -dimensional hypercube Ω satisfies

for a constant C ℓ that depends on the size of hypercube and the dimension d ℓ and the regularity ν ℓ and any measure π ℓ -1 supported in the hypercube.

604

605

606

607

608

609

610

611

612

Thus Proposition 9 tells us that the composition of the Sobolev balls satisfies 613

Given r ∗ = min ℓ r ℓ , we can bound it by ∑ L ℓ =1 ( C ℓ R ℓ ϵ ℓ ) 1 r ∗ and by then choosing ϵ ℓ = 614 ρ -1 L : ℓ +1 ( ρ L : ℓ +1 C ℓ R ℓ ) 1 r ∗ +1 ∑ ℓ ( ρ L : ℓ +1 C ℓ R ℓ ) 1 r ∗ +1 ϵ , we obtain that 615

(2,3) It the follows by a similar argument as in points (2, 3) of the proof of Proposition 8 that there is 616 a constant C 0 such that with probability 1 -δ for all f ∈ F 617

618

D Generalization at the Regularized Global Minimum 619

In this section, we first give the proof of Theorem 5 and then present detailed proofs of lemmas used 620 in the proof. The lemmas are largely inspired by [5] and may be of independent interest. 621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

D.1 Theorem 5 in Section 4.2

Theorem 11 (Theorem 5 in the main) . Given a true function f ∗ L ∗ :1 = f ∗ L ∗ ◦· · · ◦ f ∗ 1 going through the dimensions d ∗ 0 , . . . , d ∗ L ∗ , along with a continuous input distribution π 0 supported in B (0 , b 0 ) , such that the distributions π ℓ of f ∗ ℓ ( x ) (for x ∼ π 0 ) are continuous too and supported inside B (0 , b ℓ ) ⊂ R d ∗ ℓ . Further assume that there are differentiabilities ν ℓ and radii R ℓ such that ∥ f ∗ ℓ ∥ W ν ℓ , 2 ( B (0 ,b ℓ )) ≤ R ℓ , and ρ ℓ such that Lip ( f ∗ ℓ ) ≤ ρ ℓ . For a infinite width AccNet with L ≥ L ∗ and constant width d ≥ d ∗ 1 , . . . , d ∗ L ∗ -1 , we have for the ratios ˜ r ℓ = ν ℓ d ∗ ℓ +3 :

At a global minimizer ˆ f L :1 of the regularized loss f 1 , . . . , f L ↦→ ˜ L N ( f L :1 )+ λR ( f 1 , . . . , f L ) , we have L ( ˆ f L :1 ) = ˜ O ( N -min { 1 2 , ˜ r 1 ,..., ˜ r L ∗ } ) .

At a global minimizer ˆ f L :1 of the regularized loss f 1 , . . . , f L ↦→ ˜ L N ( f L :1 )+ λ ∏ L ℓ =1 ∥ f ℓ ∥ F 1 , we have L ( ˆ f L :1 ) = ˜ O ( N -1 2 + ∑ L ∗ ℓ =1 max { 0 , ˜ r ℓ -1 2 } ) .

Proof. If f ∗ = f ∗ L ∗ ◦ · · · ◦ f ∗ 1 with L ∗ ≤ L , intermediate dimensions d ∗ 0 , . . . , d ∗ L ∗ , along with a continuous input distribution π 0 supported in B (0 , b 0 ) , such that the distributions π ℓ of f ∗ ℓ ( x ) (for x ∼ π 0 ) are continuous too and supported inside B (0 , b ℓ ) ⊂ R d ∗ ℓ . Further assume that there are differentiabilities ν ∗ ℓ and radii R ℓ such that ∥ f ∗ ℓ ∥ W ν ∗ ℓ , 2 ( B (0 ,b ℓ )) ≤ R ℓ .

We first focus on the L = L ∗ case and then extend to the L > L ∗ case.

Each f ∗ ℓ can be approximated by another function ˜ f ℓ with bounded F 1 -norm and Lipschitz constant. Actually if 2 ν ∗ ℓ ≥ d ∗ ℓ -1 +3 one can choose ˜ f ℓ = f ∗ ℓ since ∥ f ∗ ℓ ∥ F 1 ≤ C ℓ R ℓ by Lemma 14, and by assumption Lip ( ˜ f ℓ ) ≤ ρ ℓ . If 2 ν ∗ ℓ < d ∗ ℓ -1 +3 , then by Lemma 13 we know that there is a ˜ f ℓ with ∥ ∥ ∥ ˜ f ℓ ∥ ∥ ∥ F 1 ≤ C ℓ R ℓ ϵ -1 2˜ r ℓ +1 ℓ and Lip ( ˜ f ℓ ) ≤ C ℓ Lip ( f ∗ ℓ ) ≤ C ℓ ρ ℓ and error

Therefore the composition ˜ f L :1 satisfies 642

643

644

645

646

647

648

649

650

651

652

653

654

For any L ≥ L ∗ , dimensions d ℓ ≥ d ∗ ℓ and widths w ℓ ≥ N , we can build an AccNet that fits eactly ˜ f L :1 , by simply adding zero weights along the additional dimensions and widths, and by adding identity layers if L > L ∗ , since it is possible to represent the identity on R d with a shallow network with 2 d neurons and F 1 -norm 2 d (by having two neurons e i σ ( e T i · ) and -e i σ ( -e T i · ) for each basis e i ). Since the cost in parameter norm of representing the identity scales with the dimension, it is best to add those identity layers at the minimal dimension min { d ∗ 0 , . . . , d ∗ L ∗ } . We therefore end up with a AccNet with L -L ∗ identity layers (with F 1 norm 2 min { d ∗ 0 , . . . , d ∗ L ∗ } ) and L ∗ layers that approximate each of the f ∗ ℓ with a bounded F 1 -norm function ˜ f ℓ .

term) is therefore bounded by 655

Taking ϵ ℓ = E ˜ r min ( N ) and λ = N -1 2 log N , this is upper bounded by 656

which implies that at the globla minimizer of the regularized loss, the (unregularized) train loss is of 657 order E ˜ r min ( N ) and the complexity measure R ( ˆ f 1 , . . . , ˆ f L ) is of order 1 N E ˜ r min ( N ) which implies 658 that the test error will be of order 659

(2) Let us now consider adding the closer to traditional L 2 -regularization L λ ( f L :1 ) = L ( f L :1 ) + 660 λ ∏ L ℓ =1 ∥ f ℓ ∥ F 1 . ,We see that the global minimizer ˆ f L :1 of the L 2 -regularized loss is upper bounded 661 by 662

Which for ϵ ℓ = E ˜ r min ( N ) and λ = N -1 N is upper bounded by 663

Which implies that both the train error is of order N -1 2 ∏ L ∗ ℓ =1 √ NE ˜ r min ( N ) and the product of the 664 F 1 -norms is of order ∏ L ∗ ℓ =1 √ NE ˜ r min ( N ) . 665

Now note that the product of the F 1 -norms bounds the complexity measure up to a constant since 666 Lip ( f ) ≤ ∥ f ∥ F 1 667

And since at the global minimum the product of the F 1 -norms is of order ∏ L ∗ ℓ =1 √ NE ˜ r min ( N ) the 668 test error will of order ( ∏ L ∗ ℓ =1 √ NE ˜ r ℓ ( N ) ) log N √ N . 669

670

671

Note that if there is at a most one ℓ where ˜ r ℓ > 1 2 then the rate is up to log term the same as E ˜ r min ( N ) .

672

D.2 Lemmas on approximating Sobolev functions

Now we present the lemmas used in this proof above that concern the approximation errors and 673 Lipschitz constants of Sobolev functions and compositions of them. We will bound the F 2 -norm and 674 note that the F 2 -norm is larger than the F 1 -norm, cf. [5, Section 3.1]. 675

Lemma 12 (Approximation for Sobolev function with bounded error and Lipschitz constant) . 676 Suppose g : S d → R is an even function with bounded Sobolev norm ∥ g ∥ 2 W ν, 2 ≤ R with 2 ν ≤ d +2 , 677 with inputs on the unit d -dimensional sphere. Then for every ϵ > 0 , there is ˆ g ∈ G 2 with small 678 approximation error ∥ g -ˆ g ∥ L 2 ( S d ) = C ( d, ν, R ) ϵ , bounded Lipschitzness Lip(ˆ g ) ≤ C ′ ( d )Lip( g ) , 679 and bounded norm 680

681

682

683

684

685

686

687

688

689

690

691

692

Proof. Given our assumptions on the target function g , we may decompose g ( x ) = ∑ ∞ k =0 g k ( x ) along the basis of spherical harmonics with g 0 ( x ) = ∫ S d g ( y )d τ d ( y ) being the mean of g ( x ) over the uniform distribution τ d over S d . The k -th component can be written as

with N ( d, k ) = 2 k + d -1 k ( k + d -2 d -1 ) and a Gegenbauer polynomial of degree k and dimension d +1 :

known as Rodrigues' formula. Given the assumption that the Sobolev norm ∥ g ∥ 2 W ν, 2 is upper bounded, we have ∥ f ∥ 2 L 2 ( S d ) ≤ C 0 ( d, ν ) R for f = ∆ ν/ 2 g where ∆ is the Laplacian on S d [18, 5]. Note that g k are eigenfunctions of the Laplacian with eigenvalues k ( k + d -1) [4], thus

where the last inequality holds because ∥ f ∥ 2 L 2 ( S d ) = ∑ k ≥ 0 ∥ f k ∥ 2 L 2 ( S d ) converges. Note using the Hecke-Funk formula, we can also write g k as scaled p k for the underlying density p of the F 1 and F 2 -norms:

where λ k = ω d -1 ω d ∫ 1 -1 σ ( t ) P k ( t )(1 -t 2 ) ( d -2) / 2 d t = Ω( k -( d +3) / 2 ) [5, Appendix D.2] and ω d denotes the surface area of S d . Then by definition of ∥ · ∥ F 2 , for some probability density p ,

Now to approximate g , consider function ˆ g defined by truncating the 'high frequencies' of g , i.e. 693 setting ˆ g k = ✶ [ k ≤ m ] g k for some m> 0 we specify later. Then we can bound the norm with 694

̸

where (a) uses Eq 1 and λ k = Ω( k -( d +3) / 2 ) ; (b) approximates by integral. 695

To bound the approximation error, 696

≤ C 5 ( d, ν, R ) m by integral approximation.

Finally, choosing m = ϵ -1 ν , we obtain ∥ g -ˆ g ∥ L 2 ( S d ) ≤ C ( d, ν, R ) ϵ and 697

̸

Then it remains to bound Lip(ˆ g ) for our constructed approximation. By construction and by [13, 698 Theorem 2.1.3], we have ˆ g = g ∗ h with now 699

by orthogonality of the Gegenbauer polynomial P k 's and the convolution is defined as 700

The coefficients for 0 ≤ k ≤ m given by [13, Theorem 2.1.3] are 701

where (a) follows from the (inverse of) weighted L 2 norm of P k ; (b) plugs in the unit constant 702 P k (1) = Γ( k + d -1) Γ( d -1) k ! and suppresses the dependence on d . Note that the constant factor Γ( d -1) Γ( d -1+ k ) 703 comes from the difference in the definitions of the Gegenbauer polynomials here and in [13]. Then 704 we can bound 705

by orthogonality of P 's w.r.t. this measure

706

707

708

709

710

711

for some constant C ( d ) that only depends on d . Hence Lip(ˆ g ) = C ′ ( d )Lip( g ) .

The next lemma adapts Lemma 12 to inputs on balls instead of spheres following the construction in [5, Proposition 5].

Lemma 13. Suppose f : B (0 , b ) → R has bounded Sobolev norm ∥ f ∥ 2 W ν, 2 ≤ R with ν ≤ ( d +2) / 2 even, where B (0 , b ) = { x ∈ R d : ∥ x ∥ 2 ≤ b } is the radiusb ball. Then for every ϵ > 0 there exists f ϵ ∈ F 2 such that ∥ f -f ϵ ∥ L 2 ( B (0 ,b )) = C ( d, ν, b, R ) ϵ , Lip( f ϵ ) ≤ C ′ ( b, d )Lip( f ) , and

Proof. Define g ( z, a ) = f ( 2 bz a ) a on ( z, a ) ∈ S d with z ∈ R d and 1 √ 2 ≤ a ∈ R . One may 712 verify that unit-norm ( z, a ) with a ≥ 1 √ 2 is sufficient to cover B (0 , b ) by setting x = bz a and 713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

solve for ( z, a ) . Then we have bounded ∥ g ∥ 2 W ν, 2 ≤ b ν R and may apply Lemma 12 to get ˆ g with ∥ g -ˆ g ∥ L 2 ( S d ) ≤ C ( d, ν, b, R ) ϵ. Letting f ϵ ( x ) = ˆ g ( ax b , a ) a -1 for the corresponding ( ax b , a ) ∈ S d gives the desired upper bounds.

Lemma 14. Suppose f : B (0 , b ) → R has bounded Sobolev norm ∥ f ∥ 2 W ν, 2 ≤ R with ν ≥ ( d +3) / 2 even. Then f ∈ F 2 and ∥ f ∥ F 2 ≤ C ( d, ν ) b ν R .

In particular, W ν, 2 ⊆ F 2 for ν ≥ ( d +3) / 2 even.

Proof. This lemma reproduces [5, Proposition 5] to functions with bounded Sobolev L 2 norm instead of L ∞ norm. The proof follows that of Lemma 12 and Lemma 13 and noticing that by Eq 1,

̸

Finally, we remark that the above lemmas extend straightforward to functions f : B (0 , b ) → R d ′ with multi-dimensional outputs, where the constants then depend on the output dimension d ′ too.

D.3 Lemma on approximating compositions of Sobolev functions

With the lemmas given above and the fact that the F 2 -norm upper bounds the F 1 -norm, we can find infinite-width DNN approximations for compositions of Sobolev functions, which is also pointed out in the proof of Theorem 5.

Lemma 15. Assume the target function f : Ω → R d out , with Ω ⊆ B (0 , b ) ⊆ R d in , satisfies:

f = g k ◦ · · · ◦ g 1 a composition of k Sobolev functions g i : R d i → R d i +1 with bounded norms ∥ g i ∥ 2 W ν i , 2 ≤ R for i = 1 , . . . , k , with d 1 = d in ;

f is Lipschitz, i.e. Lip( g i ) < ∞ for i = 1 , . . . , k .

If ν i ≤ ( d i +2) / 2 for any i , i.e. less smooth than needed, for depth L ≥ k and any ϵ > 0 , there is an infinite-width DNN ˜ f such that

Lip( ˜ f ) ≤ C 1 ∏ k i =1 Lip( g i ) ;

∥ ˜ f -f ∥ L 2 ≤ C 2 ϵ ;

the constants C 1 depends on all of the input dimensions d i (to g i ) and d out , and C 2 depends on d i , d out , ν i , b, R, k , and Lip( g i ) for all i .

If otherwise ν i ≥ ( d i +3) / 2 for all i , we can have ˜ f = f where each layer has a parameter norm bounded by C 3 R , with C 3 depending on d i , d out , ν i , and b .

Proof. Note that by Lipschitzness,

i.e. the pre-image of each component lies in a ball. By Lemma 12, for each g i , if ν i ≤ ( d i +2) / 2 , 742 we have an approximation ˆ g i on a slightly larger ball b ′ i = b ∏ i -1 j =1 C ′′ ( d j , d j +1 )Lip( g j ) such that 743

744

745

746

747

748

749

750

751

752

753

764

765

766

767

768

769

770

771

772

773

∥ g i -ˆ g i ∥ L 2 ≤ C ( d i , d i +1 , ν i , b ′ i , R ) ϵ ;

Lip(ˆ g i ) ≤ C ′′ ( d i , d i +1 )Lip( g i ) ;

where d i is the input dimension of g i . Write the constants as C i , C ′ i , and C ′′ i for notation simplicity. Note that the Lipschitzness of the approximations ˆ g i 's guarantees that, when they are composed, (ˆ g i -1 ◦ · · · ◦ ˆ g 1 )(Ω) lies in a ball of radius b ′ i = b ∏ i -1 j =1 C ′′ j Lip( g j ) , hence the approximation error remains bounded while propagating. While each ˆ g i is a (infinite-width) layer, for the other L -k layers, we may have identity layers 5 .

Let ˜ f be the composed DNN of these layers. Then we have

and approximation error

where the last equality suppresses the dependence on d i , d out , ν i , b, R, k , and Lip( g i ) for i = 754 1 , . . . , k . 755

In particular, by Lemma 14, if ν i ≥ ( d i +3) / 2 for any i = 1 , . . . , k , we can take ˆ g i = g i . If this 756 holds for all i , then we can have ˜ f = f while each layer has a F 2 -norm bounded by O ( R ) . 757

758

759

E Technical results

Here we show a number of technical results regarding the covering number.

First, here is a bound for the covering number of Ellipsoids, which is a simple reformulation of 760 Theorem 2 of [17]: 761

Theorem 16. The d -dimensional ellipsoid E = { x : x T K -1 x ≤ 1 } with radii √ λ i for λ i the i -th 762 eigenvalue of K satisfies log N 2 ( E,ϵ ) = M ϵ (1 + o (1)) for 763

For our purpose, we will want to cover a unit ball B = { w : ∥ w ∥ ≤ 1 } w.r.t. to a non-isotropic norm ∥ w ∥ 2 K = w T Kw , but this is equivalent to covering E with an isotropic norm:

Corollary 17. The covering number of the ball B = { w : ∥ w ∥ ≤ 1 } w.r.t. the norm ∥ w ∥ 2 K = w T Kw satisfies log N ( B, ∥·∥ K , ϵ ) = M ϵ (1 + o (1)) for the same M ϵ as in Theorem 16 and under the same condition.

Proof. If ˜ E is an ϵ -covering of E w.r.t. to the L 2 -norm, then ˜ B = K -1 2 ˜ E is an ϵ -covering of B w.r.t. the norm ∥·∥ K , because if w ∈ B , then √ Kw ∈ E and so there is an ˜ x ∈ ˜ E such that ∥ ∥ ∥ x -√ Kw ∥ ∥ ∥ ≤ ϵ , but then ˜ w = √ K -1 x covers w since ∥ ˜ w -w ∥ K = ∥ ∥ ∥ x -√ Kw ∥ ∥ ∥ K ≤ ϵ .

Since λ i ≤ Tr K i , we have K ≤ ¯ K for ¯ K the matrix obtained by replacing the i -th eigenvalue λ i of 774 K by Tr K i , and therefore N ( B, ∥·∥ K , ϵ ) ≤ N ( B, ∥·∥ ¯ K , ϵ ) since ∥·∥ K ≤ ∥·∥ ¯ K . We now have the 775 a[proximation log N ( B, ∥·∥ ¯ K , ϵ ) = ¯ M ϵ (1 + o (1)) for 776

We now have the simplification 777

where the o (1) term vanishes as ϵ ↘ 0 . Furthermore, this allows us to check that as long as 778 log d = o ( √ Tr K 4 ϵ log √ Tr ϵ K ) , the condition is satisfied 779

780

Second we prove how to obtain the covering number of the convex hull of a function set F : 781

Theorem 18. Let F be a set of B -uniformly bounded functions, then for all ϵ K = B 2 -K 782

Proof. Define ϵ k = B 2 -k and the corresponding ϵ k -coverings ˜ F k (w.r.t. some measure π ). For any 783 f , we write ˜ f k [ f ] for the function ˜ f k [ f ] ∈ ˜ F k that covers f . Then for any functions f in Conv F , we 784 have 785

We may assume that ˜ f 0 [ f i ] = 0 since the zero function ϵ 0 -covers the whole F since ϵ 0 = B . 786

We will now use the probabilistic method to show that the sums ∑ m i =1 β i ( ˜ f k [ f i ] -˜ f k -1 [ f i ] ) 787 can be approximated by finite averages. Consider the random functions ˜ g ( k ) 1 , . . . , ˜ g ( k ) m k 788 sampled iid with P [ ˜ g ( k ) j ] = ( ˜ f k [ f i ] -˜ f k -1 [ f i ] ) with probability β i . We have E [˜ g ( k ) j ] = 789 ∑ m i =1 β i ( ˜ f k [ f i ] -˜ f k -1 [ f i ] ) and 790

Thus if we take m k = 1 a k ( 3 ϵ k ϵ K ) 2 with ∑ a k = 1 we know that there must exist a choice of ˜ g ( k ) j s such 791 that 792

This implies that finite the set ˜ C = { ∑ K k =1 1 m k ∑ m k j =1 ˜ g ( k ) j : ˜ g ( k ) j ∈ ˜ F k -˜ F k -1 } is an 2 ϵ K covering 793 of C = Conv F , since we know that for all f = ∑ m i =1 β i f i there are ˜ g ( k ) j such that 794

Since ∣ ∣ ∣ ˜ C ∣ ∣ ∣ = ∏ K k =1 ∣ ∣ ∣ ˜ F k ∣ ∣ ∣ m k ∣ ∣ ∣ ˜ F k -1 ∣ ∣ ∣ m k , we have 795

This is minimized for the choice 796

which yields the bound 797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contribution section accurately describes our contributions, and all theorems are proven in the appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations of our Theorems after we state them.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are either stated in the Theorem statements, except for a few recurring assumptions that are stated in the setup section.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental setup is described in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use openly available data or synthetic data, with a description of how to build this synthetic data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In the experimental setup section in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The numerical experiments are mostly there as a visualization of the theoretical results, our main goal is therefore clarity, which would be hurt by putting error bars everywhere.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

Answer: [Yes]

Justification: In the experimental setup section of the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the Code of Ethics and see no issue.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is theoretical in nature, so it has no direct societal impact that can be meaningfully discussed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible 1025 release of data or models that have a high risk for misuse (e.g., pretrained language models, 1026 image generators, or scraped datasets)? 1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

Answer: [NA]

Justification: Not relevant to our paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In the experimental setup section of the Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not relevant to this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not relevant to this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Accordion Networks (AccNets), a novel neural network structure composed of multiple shallow networks. The authors propose a generalization bound for AccNets that leverages the F1-norms and Lipschitz constants of the subnetworks, demonstrating that these networks can break the curse of dimensionality by efficiently learning compositions of Sobolev functions. The paper also provides theoretical insights and empirical validation, showcasing the superior performance of AccNets in learning complex compositional tasks compared to shallow networks and kernel methods.\n\nSTRENGTHS:\nThe introduction of Accordion Networks (AccNets) as a novel neural network structure is a creative and original contribution. The paper provides a thorough theoretical analysis supported by empirical evidence, ensuring the soundness of its claims. The ability of AccNets to break the curse of dimensionality by learning compositional functions efficiently addresses a fundamental challenge in high-dimensional learning tasks.\n\nWEAKNESSES:\n1. The practical implementation of the proposed regularization methods might be challenging, particularly the first one requiring infinite width. \n\n2. The paper mentions the difficulty in optimizing Lipschitz constants, which could be a limitation in practical applications.\n\n3.  Additional experiments on more diverse real-world datasets could further demonstrate the robustness and generalizability of AccNets.\n\n4. Although the author has discussed the differences between DNN and AccNet, there is still not enough information for me to be sure in which settings to use AccNet and in which settings to use DNN. More clear differences and applicable conditions, especially the shortcomings of each need to be pointed out.\n\nQUESTIONS:\nCan the authors provide more details on the computational complexity of training Accordion Networks compared to traditional DNNs?\n\nHow sensitive are the generalization bounds to the choice of hyperparameters, particularly the Lipschitz constants and F1-norms?\n\nAre there any specific types of tasks or datasets where Accordion Networks might not perform as well as traditional methods?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a generalization bound for deep neural networks that describes how depth enables models to learn functions that are compositions of Sobolev functions. To do this, they both prove a generalization bound for compositions of accordion networks (densely connected networks with a low-rank weight structure) and for compositions of Sobolev functions. They then present a sample efficiency result for different kinds of regularization on accordion networks.\n\nSTRENGTHS:\nI really liked this paper and would like to see it accepted to NeurIPS. It addresses an important question: how does depth change generalization bounds for deep neural networks? To my knowledge, not many papers so far have addressed this question and I found the findings presented here very interesting and well embedded within prior methodology.\n\nI also found the paper very well written. I found it easy to follow along despite the highly technical nature of the results (note that I did not check the proofs in particular detail). I especially appreciated the remarks explaining different potential extensions and limitations.\n\nFinally, the theory appears to be able to explain certain empirical phenomena (in networks trained under realistic paradigms) at least qualitatively (though note that I had a few questions I will mention under weaknesses and questions). This indicates to me that it is a promising way for thinking about generalization in deep neural networks.\n\nWEAKNESSES:\n1. I would like to see a more thorough comparison with shallow networks and generalization bounds, as this comparison is a central argument for the usefulness of the presented theory. While it is clear how the findings for the shallow network are a special case of the findings on the deep networks (as presented in Thm. 1), it remains a bit unclear to me how the theory can explain improved generalization in deep compared to shallow networks. The authors certainly present different several pieces of evidence on this: both Fig. 1 and Fig. 3 demonstrate that shallow networks exhibit worse scaling. I also appreciated the theoretical explanation of a particular contrast in l. 256-261. However, I think it would be really useful to provide a general theoretical explanation for this difference and test it empirically: would it be possible to extend the theoretical comparison in l. 256-261 to the general experimental setup studied in the figures --- and if so, would this theoretical comparison predict the conditions under which deep networks have the strongest advantages over shallow networks (or perhaps the conditions under which they don't perform that much better)? Not only would this serve as a useful validation of the theory, I think it would also provide a more extensive intuition for the authors' findings.\n\n2. I appreciated the fact that the authors compare their findings with related work wherever this becomes relevant. However, I think a (potentially brief) section comparing the results here to other theoretical investigations of depth in deep networks (perhaps using different approaches) would be useful. \n\n3. The linked codebase does not contain the notebooks indicated in the README as far as I can tell and therefore currently can't be used to directly reproduce the findings.\n\n4. I believe the figures would still benefit from error bars or some other indication of the overall statistical error in the findings. I agree that the main contribution of this paper is theoretical, but since the experiments test the empirical validity of the theory, I believe it is nevertheless important to get a sense for the overall deviation in these findings (e.g. across model seeds). If the authors are concerned about a lack of clarity, they could leave the bars out of the main figures but add supplementary figures with error bars. Moreover, some of the lines in Fig. 1 do contain error bars and it would be good to clarify what these error bars represent.\n\nQUESTIONS:\n1. Do you think my suggestion in point 1 of the weaknesses make sense or do you have a reason why you see it as unnecessary?\n\n2. As far as I understand, the reason for the asymmetry between $\\nu_g$ and $\\nu_h$ in Fig. 2 is the different dimensionality, correct? It would be good to mention these dimensionalities, as I was only able to find them in the appendix.\n\n3. Could you clarify why in Fig. 2, you're using the scalings from Prop 3 rather than from Thm. 5?\n\nLIMITATIONS:\nThe authors adequately discuss the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce accordion networks (AccNets), which are compositions of multiple shallow networks. By leveraging prior workthat computes norm-based generalization bounds for shallow two-layer networks, the authors bound the complexity of a deep AccNet (as measured by its F1 norm) but the sum of the complexities of the individual shallow networks. They empirically observe that the rates predicted on real-world data are roughly representative of the trained networks, and are indeed much better than those for kernels trained on the same tasks. They put forth a nontrivial scaling law for the excess risk: $N^{-\\mathrm{min}(1/2, \\nu_g/d_{in}, \\nu_h/d_{mid})}$ for an Acc Net compared to $\\mathcal L \\sim N^{-\\mathrm{min}(1/2, \\nu_g/d_{in}, \\nu_h/d_{in})}$ for a kernel in terms of the dimensionalities $d$ and Sobolev constants $\\nu$ of the respective spaces and functions. From this, the authors obtain predictions of several phases, that they put forth experiments to verify.\n\nSTRENGTHS:\nThe paper tackles a very important open question in the theory of deep learning, for which not much progress has been made. By creatively leveraging results for shallow network in composition, the authors arrive at a nontrivial bound for deep nets. The empirics are a very compelling and welcome part of the paper. The phase diagrams illustrate the nontrivial predictivity of the theory, especially at the level of the rates. This may have important implications for scaling laws. Modulo minor revisions in discussion and exposition, the whole paper is quite readable for a relatively broad audience.\n\nWEAKNESSES:\nI am not sure how compelling the phase plots in Figure 2 are. The bounds in general are extremely loose, however the comparison of the rates in Figure 2c and Figure 3 is very promising. In general, however, it is the experience of the reviewer that measuring a rate is an extremely finicky business. It is therefore important to add a section in the appendix explicitly stating how the rates were obtained and measured. I also strongly encourage the authors to make the code for all figures public. \n\nBecause they are used very early on throughout the paper, it is the opinion of the reviewer that the notions of F1 distance and Sobolev norm should be defined earlier on in the paper. Without this, it seems like the audience will be constrained to the set of learning theorists familiar with these terms. However, if these terms are defined early on, the paper becomes remarkably accessible to a much broader audience.\n\nQUESTIONS:\nThe plot labels in Figures 2 and 3 are very difficult to read. \n\nA small comment: I have not seen the term \"modulo space\" used before. Often the term is \"quotient space\" \n\nThe sentence defining the $F_1$ ball (above theorem 1) is confusing, circular, and difficult to read. Please rewrite it.\n\nThe excess rate formula $\\mathcal L \\sim N^{-\\mathrm{min}(1/2, \\nu_g/d_{in}, \\nu_h/d_{mid})}$ is a very important result and I recommend that it be formatted for display, not inline.\n\nHow are you measuring \"dimension\" in 4.1.1? A high-dimensional gaussian with spectral decay of its covariance going as $k^{-\\alpha}$ for capacity exponent $\\alpha$ is nominally \"full dimensional\" since it is not strictly speaking constrained to a sub-manifold, and yet basic results in kernel theory and high-dimensional linear regression can show that the generalization error achieves a much better rate at larger values of $\\alpha$. Specifically, a model with capacity exponent $\\alpha$ and source exponent $r$ achieves a rate of $N^{-2\\alpha min(r, 1)}$. See, e.g. https://arxiv.org/abs/2105.15004 .  Such power law anisotropy is in abundant in natural data. In particular shallow two layer networks in the lazy limit can achieve this scaling for such 'easy tasks' with quick spectral decay. On the other hand, the bounds that you state cannot decay faster than $N^{-1/2}$. \n* In this sense, it seems that the bounds (shallow or deep) presented are certainly not tight for some datasets. Am I incorrect in concluding this? Do you have an intuition for what causes the breakdown in correctly predicting the error rates in this case?\n* Given that they breakdown in that setting, what about the datasets that you study makes it so that the scaling law predictions seem to hold?\n\nLIMITATIONS:\nGiven the theoretical nature of this work, it is unlikely to have major social implications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}