Title: Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets
Venue: NeurIPS 2024
Accepted: no
Keywords: Feature Learning

Bottleneck Structure

NeuralODE

Hamiltonian mechanics
Abstract: We study Leaky ResNets, which interpolate between ResNets ($\tilde{L}=0$)
and Fully-Connected nets ($\tilde{L}\to\infty$) depending on an 'effective
depth' hyper-parameter $\tilde{L}$. In the infinite depth limit,
we study 'representation geodesics' $A_{p}$: continuous paths in
representation space (similar to NeuralODEs) from input $p=0$ to
output $p=1$ that minimize the parameter norm of the network. We
give a Lagrangian and Hamiltonian reformulation, which highlight the
importance of two terms: a kinetic energy which favors small layer
derivatives $\partial_{p}A_{p}$ and a potential energy that favors
low-dimensional representations, as measured by the 'Cost of Identity'.
The balance between these two forces offers an intuitive understanding
of feature learning in ResNets. We leverage this intuition to explain
the emergence of a bottleneck structure, as observed in previous work:
for large $\tilde{L}$ the potential energy dominates and leads to
a separation of timescales, where the representation jumps rapidly
from the high dimensional inputs to a low-dimensional representation,
move slowly inside the space of low-dimensional representations, before
jumping back to the potentially high-dimensional outputs. Inspired
by this phenomenon, we train with an adaptive layer step-size
to adapt to the separation of timescales.

1

Abstract

We study Leaky ResNets, which interpolate between ResNets ( ˜ L = 0 ) and FullyConnected nets ( ˜ L →∞ ) depending on an 'effective depth' hyper-parameter ˜ L . In the infinite depth limit, we study 'representation geodesics' A p : continuous paths in representation space (similar to NeuralODEs) from input p = 0 to output p = 1 that minimize the parameter norm of the network. We give a Lagrangian and Hamiltonian reformulation, which highlight the importance of two terms: a kinetic energy which favors small layer derivatives ∂ p A p and a potential energy that favors low-dimensional representations, as measured by the 'Cost of Identity'. The balance between these two forces offers an intuitive understanding of feature learning in ResNets. We leverage this intuition to explain the emergence of a bottleneck structure, as observed in previous work: for large ˜ L the potential energy dominates and leads to a separation of timescales, where the representation jumps rapidly from the high dimensional inputs to a low-dimensional representation, move slowly inside the space of low-dimensional representations, before jumping back to the potentially high-dimensional outputs. Inspired by this phenomenon, we train with an adaptive layer step-size to adapt to the separation of timescales.

1 Introduction

Feature learning is generally considered to be at the center of the recent successes of deep neural 18 networks (DNNs), but it also remains one of the least understood aspects of DNN training. 19

20

21

There is a rich history of empirical analysis of the features learned by DNNs, for example the appearance of local edge detections in CNNs with a striking similarity to the biological visual cortex

22

23

24

[19], feature arithmetic properties of word embeddings [22], similarities between representations

at different layers [18, 20], or properties such as Neural Collapse [24] to name a few. While some

of these phenomenon have been studied theoretically [3, 8, 27], a more general theory of feature

learning in DNNs is still lacking. 25

For shallow networks, there is now strong evidence that the first weight matrix is able to recognize a 26

low-dimensional projection of the inputs that determines the output (assuming this structure is present) 27

[4, 2, 1]. A similar phenomenon appears in linear networks, where the network is biased towards 28

learning low-rank functions and low-dimensional representations in its hidden layers [13, 21, 29]. 29

But in both cases the learned features are restricted to depend linearly on the inputs, and the feature 30

learning happens in the very first weight matrix, whereas it has been observed that features increase 31

in complexity throughout the layers [31]. 32

The linear feature learning ability of shallow networks has inspired a line of work that postulates that 33

the weight matrices learn to align themselves with the backward gradients and that by optimizing for 34

this alignment directly, one can achieve similar feature learning abilities even in deep nets [5, 25]. 35

For deep nonlinear networks, a theory that has garnered a lot of interest is the Information Bottleneck 36 [28], which observed amongst other things that the inner representations appear to maximize their 37 mutual information with the outputs, while minimizing the mutual information with the inputs. A 38 limitation of this theory is its reliance on the notion of mutual information which has no obvious 39 definition for empirical distributions, which lead to some criticism [26]. 40

A recent theory that is similar to the Information Bottleneck but with a focus on the 41 dimensionality/rank of the representations and weight matrices rather than the mutual information is 42 the Bottleneck rank/Bottleneck structure [16, 15, 30]: which describes how, for large depths, most of 43 the representations will have approximately the same low dimension, which equals the Bottleneck 44 rank of the task (the minimal dimension that the inputs can be projected to while still allowing 45 for fitting the outputs). The intuitive explanation for this bias is that a smaller parameter norm is 46 required to (approximately) represent the identity on low-dimensional representations rather than 47 high dimensional ones. Some other types of low-rank bias have been observed in recent work [9, 14]. 48

49 50

51

52

53

In this paper we will focus on describing the Bottleneck structure in ResNets, and formalize the notion of 'cost of identity' as a driving force for the bias towards low dimensional representation. The ResNet setup allows us to consider the continuous paths in representation space from input to output, similar to the NeuralODE [6], and by adding weight decay, we can analyze representation geodesics, which are paths that minimize parameter norm, as already studied in [23].

54

55

56

57

58

59

60

61

62

63

64

65

1.1 Leaky ResNets

Our goal is to study a variant of the NeuralODE [6, 23] approximation of ResNet with leaky skip connections and with L 2 -regularization. The classical NeuralODE describes the continuous evolution of the activations α p ( x ) ∈ R w starting from α 0 ( x ) = x at the input layer p = 0 and then follows

for the w × ( w +1) matrices W p and the nonlinearity σ : R w → R w +1 which maps a vector z to σ ( z ) = ( [ z 1 ] + . . . [ z w ] + 1 ) , applying the ReLU nonlinearity entrywise and appending a new entry with value 1 . Thanks to the appended 1 we do not need any explicit bias, since the last column W p, · w +1 of the weights replaces the bias.

This can be thought of as a continuous version of the traditional ResNet with activations α ℓ ( x ) for ℓ = 1 , . . . , L : α ℓ +1 ( x ) = α ℓ ( x ) + W ℓ σ ( α ℓ ( x )) .

We will focus on Leaky ResNets , a variant of ResNets that interpolate between ResNets and FCNNs, by tuning the strength of the skip connections leading to the following ODE with parameter ˜ L :

This can be thought of as the continuous version of α ℓ +1 ( x ) = (1 -˜ L ) α ℓ ( x ) + W ℓ σ ( α ℓ ( x )) . As we 66 will see, the parameter ˜ L plays a similar role as the depth in a FCNN. 67

Finally we will be interested describing the paths that minimize a cost with L 2 -regularization 68

The scaling of λ ˜ L for the regularization term will be motivated in Section 1.2.

This type of optimization has been studied in [23] without leaky connections, but we will describe in this paper large ˜ L behavior which leads to a so-called Bottleneck structure [16, 15] as a result of a separation of time scales in p .

1.2 A Few Symmetries

69

70

71

72

73

Changing the leakage parameter ˜ L is equivalent (up to constants) to changing the integration range 74 [0 , 1] or to scaling the outputs. 75

Integration range: Consider the weights W p on the range [0 , 1] and leakage parameter ˜ L , leading 76 to activations α p . Then stretching the weights to a new range [0 , c ] , by defining W ′ q = 1 c W q/c for 77

q ∈ [0 , c ] , and dividing the leakage parameter by c , stretches the activations α ′ q = α p/c : 78

and the parameter norm is simply divided by c : ∫ c 0 ∥ ∥ W ′ q ∥ ∥ 2 dq = 1 c ∫ 1 0 ∥ W p ∥ 2 dp .

79

This implies that a path on the range [0 , c ] with leakage parameter ˜ L = 1 is equivalent to a path on 80 the range [0 , 1] with leakage parameter ˜ L = c up to a factor of c in front of the parameter weights. 81 For this reason, instead of modeling different depths as changing the integration range, we will keep 82 the integration range to [0 , 1] for convenience but change the leakage parameter ˜ L instead. To get rid 83 of the factor in front of the integral, we choose a regularization term of the form λ ˜ L . From now on, we 84 call ˜ L the (effective) depth of the network. 85

86

87

88

Note that this also suggests that in the absence of leakage ( ˜ L = 0 ), changing the range of integration has no effect on the effective depth, since 2 ˜ L = 0 too. Instead, in the absence of leakage, the effective depth can be increased by scaling the outputs as we now show.

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

where ( · ) + is the pseudo-inverse.

We therefore consider the equivalent optimization over the activations A p :

This is our first encounter with the norm ∥ M ∥ K p = ∥ Mσ ( A p ) + ∥ F corresponding to the scalar product ⟨ A,B ⟩ K p = Tr [ AK + p B ] for K p = σ ( A p ) T σ ( A p ) that will play a central role in our upcoming analysis. By convention, we say that ∥ M ∥ K p = ∞ if M does not lie in the image of K p , i.e. Im M T ⊈ Im K p .

It can be helpful to decompose this loss along the different neurons

Leading to a particle flow behavior, where the neurons A p,i · ∈ R N are the particles. At first glance, it 109 appears that there is no interaction between the particles, but remember that the norm ∥·∥ K p depends 110 on the covariance K p = ∑ w i =1 σ ( A i · ) σ ( A i · ) T , leading to a global interaction between the neurons. 111

Output scaling: Given a path W p on the [0 , 1] (for simplicity, we assume that there are no bias, i.e. W p, · w +1 = 0 ), then increasing the leakage by a constant ˜ L → ˜ L + c leads to a scaled down path α ′ p = e -cp α p . Indeed we have α ′ 0 ( x ) = α 0 ( x ) and

Thus a nonleaky ResNet ˜ L = 0 with very large outputs α 1 ( x ) is equivalent to a leaky ResNet ˜ L > 0 with scaled down outputs e -˜ L α 1 ( x ) . Such large outputs are common when training on cross-entropy loss, and other similar losses that are only minimized at infinitely large outputs. When trained on such losses, it has been shown that the outputs of neural nets will keep on growing during training [12, 7], suggesting that when training ResNets on such a loss, the effective depth increases during training (though quite slowly).

1.3 Lagrangian Reformulation

The optimization of Leaky ResNets can be reformulated, leading to a Lagrangian form.

First observe that the weights W p at any minimizer can be expressed in terms of the matrix of activations A p = α p ( X ) ∈ R w × N over the whole training set X ∈ R w × N (similar to [17]):

If we assume that Im A T p ⊂ Im σ ( A p ) T , we can decompose the inside of the integral as three terms: 112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

The middle term ⟨ ∂ p A p , A p ⟩ K + p plays a relatively minor role in our analysis 1 , so we focus more on the two other terms:

Cost of identity ∥ A p ∥ 2 K p / potential energy -˜ L 2 ∥ A p ∥ 2 K p : This term can be interpreted as a form of potential energy, since it only depends on the representation A p and not its derivative ∂ p A p . We call it the cost of identity (COI), since it is the Frobenius norm of the smallest weight matrix W p such that W p σ ( A p ) = A p . The COI can be interpreted as measuring the dimensionality of the representation, inspired by the fact if the representations A p is non-negative (and there is no bias β = 0 ), then A p = σ ( A p ) and the COI simply equals the rank ∥ A p ∥ 2 K p = Rank A p (this interpretation is further justified in Section 1.4). We follow the convention of defining the potential energy as the negative of the term that appears in the Lagrangian, so that the Hamiltonian equals the sum of these two energies.

Kinetic energy 1 2 ˜ L ∥ ∂ p A p ∥ 2 K p : This term measures the size of the representation derivative ∂ p A p w.r.t. the K p norm. It favors paths p ↦→ A p that do not move too fast, especially along directions where σ ( A p ) is small.

This suggests that the local optimal paths must balance two objectives that are sometimes opposed: the kinetic energy favors going from input representation to output representation in a 'straight line' that minimizes the path length, the COI on the other hand favors paths that spends most of the path in low-dimensional representations that have a low COI. The balance between these two goals shifts as the depth ˜ L grows, and for large depths it becomes optimal for the network to rapidly move to a representation of smallest possible dimension (not too small that it becomes impossible to map back to the outputs), remain for most of the layers inside the space of low-dimensional representations, and finally move rapidly to the output representation; even if this means doing a large 'detour' and having a large kinetic energy. The main goal of this paper is to describe this general behavior.

Note that one could imagine that as ˜ L →∞ it would always be optimal to first go to the minimal COI representation which is the zero representation A p = 0 , but once the network reaches a zero representation, it can only learn constant representations afterwards (the matrix K p = 11 T is then rank 1 and its image is the space of constant vectors). So the network must find a representation that minimizes the COI under the condition that there is a path from this representation to the outputs.

Remark. While this interpretation and decomposition is a pleasant and helpful intuition, it is rather difficult to leverage for theoretical proofs directly. The problem is that we will focus on regimes where the representations A p and σ ( A p ) are approximately low-dimensional (since those are the representations that locally minimize the COI), leading to an unbounded pseudo-inverse σ ( A p ) + . This is balanced by the fact that ( ˜ LA p + ∂ p A p ) is small along the directions where σ ( A p ) + explodes, ensuring a finite weight matrix norm ∥ ∥ ∥ ˜ LA p + ∂ p A p ∥ ∥ ∥ 2 K + p . But the suppression of ( ˜ LA p + ∂ p A p ) along these bad directions usually comes from cancellations, i.e. ∂ p A p ≈ -˜ LA p . In such cases, the decomposition in three terms of the Lagrangian is ill adapted since all three terms are infinite and cancel each other to yield a finite sum ∥ ∥ ∥ ˜ LA p + ∂ p A p ∥ ∥ ∥ 2 K p . One of our goal is to save this intuition and prove a similar decomposition with stable equivalent to the cost of identity and kinetic energy where K + p is replaced by the bounded ( K p + γI ) + for the right choice of γ .

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

1.4 Cost of Identity as a Measure of Dimensionality

The cost of identity can be thought of as a measure of dimensionality of the representation. It is obvious for non-negative representations because ∥ A p ∥ 2 K + p = ∥ A p A p + ∥ 2 F = Rank A p , but in general, it can be shown to upper bound a notion of 'stable rank':

Proposition 1. ∥ Aσ ( A ) + ∥ 2 F ≥ ∥ A ∥ 2 ∗ ∥ A ∥ 2 F for the nuclear norm ∥ A ∥ ∗ = ∑ Rank A i =1 s i ( A ) .

Proof. Weknow that ∥ σ ( A ) ∥ F ≤ ∥ A ∥ F , therefore ∥ Aσ ( A ) + ∥ 2 F ≥ min ∥ B ∥ F ≤∥ A ∥ F ∥ AB + ∥ 2 F which is minimized when B = ∥ A ∥ F √ ∥ A ∥ ∗ √ A , yielding the result.

The stable rank ∥ A ∥ 2 ∗ ∥ A ∥ 2 F is upper bounded by Rank A , with equality if all non-zero singular values of A are equal, and it is lower bound by the more common notion of stable rank ∥ A ∥ 2 F ∥ A ∥ 2 op , because ∑ s i max s i ≥ ∑ s 2 i for the singular values s i .

Note that in contrast to the COI which is a very unstable quantity because of the pseudo-inverse, the ratio ∥ A ∥ 2 ∗ ∥ A ∥ 2 F is continuous except at A = 0 . This also makes it much easier to compute empirically than the COI itself.

We know that the COI matches the dimension or rank for positive representations, but it turns out that the local minima of the COI that are stable under the addition of a new neuron are all positive:

Proposition 2. A local minimum of A ↦→ ∥ Aσ ( A ) + ∥ 2 F is said to be stable if it remains a local minimum after concatenating a zero vector A ′ = ( A 0 ) ∈ R ( w +1) × N . All stable minima are non-negative, and satisfy ∥ Aσ ( A ) + ∥ 2 F = Rank A .

̸

Assume by contradiction that there is a i = 1 , . . . , N such that σ ( A · i ) = A · i , then choosing z = σ ( A ) T σ ( A · i ) we have σ ( z ) = z and the two ϵ 2 terms are negative:

which implies that A ′ it is not a local minimum.

These stable minima will play a significant role in the rest of our analysis, as we will see that for large ˜ L the representations A p of most layers will be close to one such local minimum. Now we are not able to rule out the existence of non-stable local minima (nor guarantee that they are avoided with high probability), but one can show that all strict local minima of wide enough networks are stable. Actually we can show something stronger, starting from any non-stable local minimum there is a constant loss path that connects it to a saddle:

Proposition 3. If w > N ( N +1) then if ˆ A ∈ R w × N is local minimum of A ↦→∥ Aσ ( A ) + ∥ 2 F that is not non-negative, then there is a continuous path A t of constant COI such that A 0 = ˆ A and A 1 is a saddle.

This could explain why a noisy GD would avoid such negative/non-stable minima, since there is no 'barrier' between the minima and a lower one, one could diffuse along the path described in Proposition 3 until reaching a saddle and going towards a lower COI minima. But there seems to be something else that pushes away from such non-negative minima, as in our experiments with full population GD we have only observed stable/non-negative local minimas.

(a) Hamiltonian measures across ˜ L

(b) Bottleneck structure

(c) Hamiltonian dynamics

10

Q

10

15

True k

COI

20

25

3

5

effective depth

10°

()s

10

10°

10-1

14

12

(°M)s

10

8

6

2

0.0

0.4

0.6

0.8

1.0

p

15.0

Stable Hamiltonian (igamma=0.001)

12.5

Stable Hamiltonian (Igamma=0.0001)

Kinetic Energy

CO1

10.0

True k*

7.5

5.0

2.5

0.0

2.5

5.0

0.2

0.4

0.6

0.8

1.0

P

187

188

189

1.5 Hamiltonian Reformulation

We can further reformulate the evolution of the optimal representations A p in terms of a Hamiltonian, similar to Pontryagin's maximum principle.

Let us define the backward pass variables B p = -1 λ ∂ A p C ( A 1 ) for the cost C ( A ) = 1 2 ∥ f ∗ ( X ) -A ∥ 2 F , 190 which play the role of the 'momenta' of A p in this Hamiltonian interpretation, which follows the 191 backward differential equation 192

Now at any critical point, we have that ∂ W p C ( A 1 ) + λ ˜ L W p = 0 and thus W p = 193 -˜ L λ ∂ A p C ( A 1 ) σ ( A p ) T = ˜ LB p σ ( A p ) T , leading to joint dynamics for A p and B p : 194

These are Hamiltonian dynamics ∂ p A p = ∂ B p H and -∂ p B p = ∂ A p H w.r.t. the Hamiltonian 195

The Hamiltonian is a conserved quantity, i.e. it is constant in p . It will play a significant role in 196 describing a separation of timescales that appears for large depths ˜ L . Another significant advantage 197 of the Hamiltonian reformulation over the Lagrangian approach is the absence of the unstable 198 pseudo-inverses σ ( A p ) + . 199

Remark. Note that the Lagrangian and Hamiltonian reformulations have already appeared in previous 200 work [23] for non-leaky ResNets. Our main contributions are the description in the next section of the 201 Hamiltonian as the network becomes leakier ˜ L →∞ , the connection to the cost of identity, and the 202 appearance of a separation of timescales. These structures are harder to observe in non-leaky ResNets 203 (though they could in theory still appear since increasing the scale of the outputs is equivalent to 204 increasing the effective depth ˜ L as shown in Section 1.2). 205

The Lagrangian and Hamiltonian are also very similar to the ones in [10, 11], and the separation of 206 timescales and rapid jumps that we will describe also bear a strong similarity. Though a difference 207 with our work is that the norm ∥·∥ K p depends on A p and can be degenerate. 208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

2 Bottleneck Structure in Representation Geodesics

A recent line of work [16, 15] studies the appearance of a so-called Bottleneck structure in large depth fully-connected networks, where the weight matrices and representations of 'almost all' layers of the layers are approximately low-rank/low-dimensional as the depth grows. This dimension k is consistent across layers, and can be interpreted as being equal to the so-called Bottleneck rank of the learned function. This structure has been shown to extend to CNNs in [30], and we will observe a similar structure in our leaky ResNets, further showcasing its generality.

More generally, our goal is to describe the 'representation geodesics' of DNNs: the paths in representation space from input to output representation. The advantage of ResNets (leaky or not) over FCNNs is that these geodesics can be approximated by continuous paths and are described by differential equations (as described by the Hamiltonian reformulation).

This section provides an approximation of the Hamiltonian that illustrates the separation of timescales that appears for large depths, with slow layers with low COI/dimension, and fast layers with high COI/dimension.

2.1 Separation of Timescales

If Im A T p ⊂ Im σ ( A p ) T , then the Hamiltonian equals the sum of the kinetic and potential energies:

This implies that ∥ ∂ p A p ∥ K p = ˜ L √ ∥ A p ∥ 2 K p + 2 ˜ L H which implies that for large ˜ L , the derivative ∂ p A p is only finite at p s where the COI ∥ A p ∥ 2 K p is close to -2 ˜ L H . On the other hand, ∂ p A p will blow up for all p with a finite gap √ ∥ A p ∥ 2 K p + 2 ˜ L H > 0 between the COI and the Hamiltonian. This suggests a separation of timescales as ˜ L →∞ , with slow dynamics in layers whose COI/dimension is close to -2 ˜ L H and fast dynamics in the high COI/dimension layers.

But the assumption Im A T p ⊂ Im σ ( A p ) T seems to rarely be true in practice, and both kinetic and COI appear to be often infinite in practice. But up to a few approximations, the same argument can be made for stable versions of the kinetic energy/COI:

Theorem 4. For sequence A ˜ L p of geodesics with ∥ ∥ ∥ B ˜ L p ∥ ∥ ∥ 2 ≤ c < ∞ , and any γ > 0 , we have

for the path length ℓ γ, ˜ L = ∫ 1 0 ∥ ∥ ∥ ∂ p A ˜ L p ∥ ∥ ∥ ( K p + γI ) dp . Finally 234

235

236

237

238

239

240

241

242

243

244

Note that the size of ∥ B ˜ L p ∥ 2 can vary a lot throughout the layers, we therefore suggest choosing a p -dependent γ : γ p = γ 0 ∥ σ ( A ˜ L p ) ∥ 2 op = γ 0 ∥ K p ∥ 2 op . There are two motivations for this: first it is natural to have γ scale with K p , ; and second, since W p = ˜ LB p σ ( A p ) T is of approximately constant size (thanks to balancedness, see Appendix A.3), we typically have that the size of B p is inversely proportional to that of σ ( A p ) , so that γ p ∥ B p ∥ 2 should keep roughly the same size for all p .

Theorem 4 shows that for large ˜ L (and choosing e.g. γ = ˜ L -1 ), the Hamiltonian is close to the minimal COI along the path. Second, the norm of the derivative ∥ ∂ p A p ∥ ( K p + γi ) is close to ˜ L times the 'extra-COI' √ ∥ A p ∥ 2 ( K p + γI ) + 2 ˜ L H ≈ √ ∥ A p ∥ 2 ( K p + γI ) -min q ∥ A q ∥ 2 ( K q + γI ) , which describes the separation of timescales, with slow ( ∼ 1 ) dynamics at layers p where the COI is almost optimal and fast ( ∼ ˜ L ) dynamics everywhere the COI is far from optimal.

(a) Test performance versus depth

(b) Bottleneck structure and adaptivity.

(c) Paths

adaptive

irregular

equidistant

I~J

10-1

6 × 10-2

4 × 102

3 ×102

5.0

7.5

10.0

12.5

15.0

17.5

20.0

22.5

L

10

8

(M)s

0.0

0.2

0.4

0.6

0.8

1.0

p

adaptive

irregular

equidistant

input

Assuming a finite length ℓ γ, ˜ L < ∞ , the norm of the derivative must be finite at almost all layers, 245 meaning that the COI/dimensionality is optimal in almost all layers, with only a countable number 246 of short high COI/dimension jumps. These jumps typically appear at the beginning and end of the 247 network, because the input and output dimensionality and COI are (mostly) fixed, so it will typically 248 be non-optimal, and so there will often be fast regions close to the beginning and end of the network. 249 We have actually never observed any jump in the middle of the network, though we are not able to 250 rule them out theoretically. 251

252

253

If we assume that the paths A ˜ L p are stable under adding a neuron, then we can additionally guarantee that the representations in the slow layers ('inside the Bottleneck') will be non-negative:

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

Proposition 5. Let A ˜ L p be a uniformly bounded sequence of local minima for increasing ˜ L , at any p 0 ∈ (0 , 1) such that ∥ ∂ p A p ∥ is uniformly bounded in a neighborhood of p 0 for all ˜ L , then A ∞ p 0 = lim ˜ L A ˜ L p 0 is non-negative.

We therefore know that the optimal COI min q ∥ A q ∥ 2 ( K q + γI ) is close to the dimension of the limiting representations A ∞ p 0 , i.e. it must be an integer k ∗ which we call the Bottleneck rank of the sequence of minima since it is closely related to the Bottleneck rank introduced in [16]. The Hamiltonian H is then close to -˜ L 2 k ∗ .

Figure 1 illustrates these phenomena: the Hamiltonian (and the stable Hamiltonians H γ = 1 2 ˜ L ∥ ∂ p A p ∥ 2 ( K p + γI ) -˜ L 2 ∥ A p ∥ 2 ( K p + γI ) ) approach the rank k ∗ = 3 from below, while the minimal COI approaches it from above; The kinetic energy is proportional to the extra COI, and they are both large towards the beginning and end of the network where the weights W p are higher dimensional. We see in Figure 1c that the (stable) Hamiltonian are not exactly constant, but it still varies much less than its components, the kinetic and potential energies.

Because of the non-convexity of the loss we are considering, one can imagine that there could exist distinct sequences of local minima as ˜ L →∞ , which could have different rank, depending on what low-dimension they reach inside their bottleneck. Indeed in our experiments we have seen that the number of dimensions that are kept inside the bottleneck can vary by 1 or 2, and in FCNN distinct sequences of depth increasing minima with different ranks have been observed in [15].

3 Discretization Scheme

To use such Leaky ResNets in practice, we need to discretize over the range [0 , 1] . For this we 273 choose a set of layer-steps ρ 1 , . . . , ρ L with ∑ ρ ℓ = 1 , and define the activations at the locations 274

p ℓ = ρ 1 + · · · + ρ ℓ ∈ [0 , 1] recursively as 275

and the regularized cost L ( θ ) = C ( α 1 ( X )) + λ 2 ˜ L ∑ L ℓ =1 ρ ℓ ∥ W p ℓ ∥ 2 , for the parameters θ = ( W p 1 , . . . , W p L ) . Note that it is best to ensure that ρ ℓ ˜ L remains smaller than 1 so that the prefactor (1 -ρ ℓ ˜ L ) does not become negative, though we will also discuss certain setups where it might be okay to take larger layer-steps.

Now comes the question of how to choose the ρ ℓ s. We consider three options:

Equidistant: The simplest choice is to choose equidistant points ρ ℓ = 1 L . Note that the condition ρ ℓ L < 1 then becomes L > ˜ L . But this choice might be ill adapted in the presence of a Bottleneck structure due to the separation of timescales.

Irregular: Since we typically observe that the fast layers appear close to the inputs and outputs with a slow bottleneck in the middle, one could simply choose the ρ ℓ to be go from small to large and back to small as ℓ ranges from 1 to L . This way there are many discretized layers in the fast regions close to the input and output and not too many layers inside the Bottleneck where the representations are changing less. More concretely one can choose ρ ℓ = 1 L + a L ( 1 4 -∣ ∣ ℓ L -1 2 ∣ ∣ ) for a ∈ [0 , 1) , the choice a = 0 leads to an equidistant mesh, but increasing a will lead to more points close to the inputs and outputs. To guarantee ρ ℓ ˜ L < 1 , we need L > (1 + a 1 4 ) ˜ L .

Adaptive: But this can be further improved by choosing the ρ ℓ to guarantee that the distances ∥ A ℓ -A ℓ -1 ∥ / ∥ A p ∥ are approximately the same for all ℓ (we divide by the size of A p since it can vary a lot throughout the layers). Since the rate of change of A is proportional to ρ

p ℓ ( ∥ A ℓ -A ℓ -1 ∥ / ∥ A p ∥ = ρ ℓ c ℓ ), it is optimal to choose ρ ℓ = c -1 ℓ ∑ c -1 ℓ for c ℓ = ∥ A ℓ -A ℓ -1 ∥ / ρ ℓ ∥ A p ∥ . The update ρ ℓ ← c -1 i ∑ c -1 i can be done at every training step or every few training steps.

Note that the condition ρ ℓ ˜ L < 1 might not be necessary inside the bottleneck since we have the approximation W p σ ( A p ℓ -1 ) ≈ ˜ LA p ℓ -1 , canceling out the negative direction. In particular with the adaptive layer-steps that we propose, a large ρ ℓ is only possible for layers where c ℓ is small, which is only possible when W p σ ( A p ℓ -1 ) ≈ ˜ LA p ℓ -1 .

Figure 2 illustrates the effect of the choice of ρ ℓ for different depths L , we see a small but consistent advantage in the test error when using adaptive or irregular ρ ℓ s. Looking at the resulting Bottleneck structure, we see that the adaptive ρ ℓ s result in more steps especially in the beginning of the network, but also at the end. This because the 'true function' f ∗ : R 30 → R 30 we are fitting in these experiments is of the form f ∗ = g 3 ◦ g 2 ◦ g 1 where the first inner dimension is 6 and the second is 3, thus resulting in a rank of k ∗ = 3 . But before reaching this minimal dimension, the network needs to represent g 2 ◦ g 1 , which requires more layers, and one can almost see that the weight matrices are roughly 6 -dimensional around p = 0 . 3 . The adaptivity to this structure could explain the advantage in the test error.

4 Conclusion

We have given a description of the representation geodesics A p of Leaky ResNets. We have identified an invariant, the Hamiltonian, which is the sum of the kinetic and potential energy, where the kinetic energy measures the size of the derivative ∂ p A p , while the potential energy is inversely proportional to the cost of identity, which is a measure of dimensionality of the representations. As the effective depth of the network grows, the potential energy dominates and we observe a separation of timescales. At layers with minimal dimensionality over the path, the kinetic energy (and thus the derivative ∂ p A p ) is finite. Conversely, at layers where the representation is higher-dimensional, the kinetic energy must scale with ˜ L . This leads to a Bottleneck structure, with a short, high-dimensional jump from the input representation to a low dimensional representation, followed by slow dynamics inside the space of low-dimensional representations followed by a final high-dimensional jump to the high dimensional outputs.

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

References

Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory , pages 4782-4887. PMLR, 2022.

Emmanuel Abbe, Enric Boix-Adserà, Matthew Stewart Brennan, Guy Bresler, and Dheeraj Mysore Nagaraj. The staircase property: How hierarchical structure can guide deep learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics , 4:385-399, 2016.

Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research , 18(1):629-681, 2017.

Daniel Beaglehole, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Mechanism of feature learning in convolutional neural networks. arXiv preprint arXiv:2309.00570 , 2023.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Lénaïc Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 1305-1338. PMLR, 09-12 Jul 2020.

Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards understanding linear word analogies. arXiv preprint arXiv:1810.04882 , 2018.

Tomer Galanti, Zachary S Siegel, Aparna Gupte, and Tomaso Poggio. Sgd and weight decay provably induce a low-rank bias in neural networks. arXiv preprint arXiv:2206.05794 , 2022.

Tobias Grafke, Rainer Grauer, T Schäfer, and Eric Vanden-Eijnden. Arclength parametrized hamilton's equations for the calculation of instantons. Multiscale Modeling & Simulation , 12(2):566-580, 2014.

Tobias Grafke and Eric Vanden-Eijnden. Numerical computation of rare events via large deviation theory. Chaos: An Interdisciplinary Journal of Nonlinear Science , 29(6):063118, 06 2019.

Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1832-1841. PMLR, 10-15 Jul 2018.

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Florentin Guth, Brice Ménard, Gaspar Rochette, and Stéphane Mallat. A rainbow in deep network black boxes. arXiv preprint arXiv:2305.18512 , 2023.

Arthur Jacot. Bottleneck structure in learned features: Low-dimension vs regularity tradeoff. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 23607-23629. Curran Associates, Inc., 2023.

Arthur Jacot. Implicit bias of large depth networks: a notion of rank for nonlinear functions. In The Eleventh International Conference on Learning Representations , 2023.

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

Arthur Jacot, Eugene Golikov, Clément Hongler, and Franck Gabriel. Feature learning in l 2 -regularized dnns: Attraction/repulsion and sparsity. In Advances in Neural Information Processing Systems , volume 36, 2022.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning , pages 3519-3529. PMLR, 2019.

A. Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60:84 - 90, 2012.

Jianing Li and Vardan Papyan. Residual alignment: Uncovering the mechanisms of residual networks. Advances in Neural Information Processing Systems , 36, 2024.

Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations , 2020.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Houman Owhadi. Do ideas have shape? plato's theory of forms as the continuous limit of artificial neural networks. arXiv preprint arXiv:2008.03920 , 2020.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mechanism for feature learning in neural networks and backpropagation-free machine learning models. Science , 383(6690):1461-1467, 2024.

Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference on Learning Representations , 2018.

Peter Súkeník, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. Advances in Neural Information Processing Systems , 36, 2024.

Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw) , pages 1-5. IEEE, 2015.

Zihan Wang and Arthur Jacot. Implicit bias of SGD in l 2 -regularized linear DNNs: Oneway jumps from high to low rank. In The Twelfth International Conference on Learning Representations , 2024.

Yuxiao Wen and Arthur Jacot. Which frequencies do cnns need? emergent bottleneck structure in feature learning. to appear at ICML , 2024.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In 405 Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 406 6-12, 2014, Proceedings, Part I 13 , pages 818-833. Springer, 2014. 407

A Proofs 408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

A.1 Cost of Identity

Proposition 6 (Proposition 3 in the main.) . If w > N ( N +1) then if ˆ A ∈ R w × N is local minimum of A ↦→∥ Aσ ( A ) + ∥ 2 F that is not non-negative, then there is a continuous path A t of constant COI such that A 0 = ˆ A and A 1 is a saddle.

̸

Proof. The local minimum ˆ A leads to a pair of N × N covariance matrices ˆ K = ˆ A T ˆ A and ˆ K σ = σ ( ˆ A ) T σ ( ˆ A ) . The pair ( ˆ K, ˆ K σ ) belongs to the conical hull Cone { ( ˆ A i · ˆ A T i · , σ ( ˆ A i · ) σ ( ˆ A i · ) T ) : i = 1 , . . . , w } . Since this cones lies in a N ( N +1) -dimensional space (the space of pairs of symmetric N × N matrices), we know by Caratheodory's theorem (for convex cones) that there is a conical combination ( ˆ K, ˆ K σ -β 2 1 N × N ) = ∑ w i =1 a i ( ˆ A i · ˆ A T i · , σ ( ˆ A i · ) σ ( ˆ A i · ) T ) such that no more than N ( N + 1) of the coefficients are nonzero. We now define A t to have lines A t,i · = √ (1 -t ) + ta i ˆ A i · , so that A t =0 = ˆ A and at t = 1 at least one line of A t =1 is zero (since at least one of the a i s is zero). First note that the covariance pairs remain constant over the path: K t = A T t A t = ∑ w i =1 ((1 -t ) + ta i ) ˆ A i · ˆ A T i · = (1 -t ) ˆ K + t ˆ K = ˆ K and similarly K σ t = ˆ K σ , which implies that the cost ∥ A t σ ( A t ) + ∥ 2 F = Tr [ K t K σ + t ] is constant too. Second, since a representation A is non-negative iff the covariances satisfy K = K σ , the representation path A t cannot be non-negative either since it has the same kernel pairs ( ˆ K, ˆ K σ ) with ˆ K = ˆ K σ .

Now (the converse of) Proposition 2 tells us that if A t =1 is not non-negative and has a zero line, then it is not a local minimum, which implies that it is a saddle.

A.2 Bottleneck

Theorem 7. For any uniformly bounded sequence A ˜ L p of geodesics, i.e. ∥ ∥ ∥ A ˜ L p ∥ ∥ ∥ 2 , ∥ ∥ ∥ B ˜ L p ∥ ∥ ∥ 2 ≤ c < ∞ , and any γ > 0 , we have

for the path length ℓ γ, ˜ L = ∫ 1 0 ∥ ∥ ∥ ∂ p A ˜ L p ∥ ∥ ∥ ( K p + γI ) dp . Finally 431

Proof. First observe that 432

and thus we have 433

(1) The upper bound -2 ˜ L Hmin p ∥ ∥ ∥ A ˜ L p ∥ ∥ ∥ 2 ( K p + γI ) ≤ γc then follows from the fact that ∥ B p ∥ 2 ≤ c . 434 For the lower bound, first observe that 435

and therefore 436

which implies the lower bound. 437

(2) The lower bound follows from equation 1. The upper bound follows from 438

439

440

441

442

443

444

445

446

447

448

449

450

451

Proposition 8 (Proposition 5 in the main.) . Let A ˜ L p be a uniformly bounded sequence of local minima for increasing ˜ L , at any p 0 ∈ (0 , 1) such that ∥ ∂ p A p ∥ is uniformly bounded in a neighborhood of p 0 for all ˜ L , then A ∞ p 0 = lim ˜ L A ˜ L p 0 is non-negative.

Proof. Given a path A p with corresponding weight matrices W p corresponding to a width w , then ( A 0 ) is a path with weight matrix ( W p 0 0 0 ) . Our goal is to show that for sufficiently large depths, one can under certain assumptions slightly change the weights to obtain a new path with the same endpoints but a slightly lower loss, thus ensuring that if certain assumptions are not satisfied then the path cannot be locally optimal.

Let us assume that ∥ ∂ p A p ∥ ≤ c 1 in a neighborhood of a p 0 ∈ (0 , 1) , and assume by contradiction that there is an input index i = 1 , . . . , N such that A p 0 , · i has at least one negative entry, and therefore ∥ A p 0 , · i ∥ 2 -∥ σ ( A p 0 , · i ) ∥ 2 = c 0 > 0 for all ˜ L .

We now consider the new weights

for t ( p ) = max { 0 , 1 -| p -p 0 | r } a triangular function centered in p 0 and for an ϵ > 0 . 452

For ϵ and r small enough, the parameter norm will decrease: 453

Now since W p σ ( A p, · i ) = ∂ p A p, · i + ˜ LA p, · i , this simplifies to 454

455

456

457

458

459

460

461

By taking r small enough, we can guarantee that -∥ A p, · i ∥ 2 + ∥ σ ( A p, · i ) ∥ 2 < -c 0 2 for all p such that t ( p ) > 0 , and for ˜ L large enough we can guarantee that ∣ ∣ ∣ 1 ˜ L A T p, · i ∂ p A p, · i ∣ ∣ ∣ is smaller then c 0 4 , so that we can guarantee that the parameter norm will be strictly smaller for ϵ small enough.

We will now show that with these new weights the path becomes approximately ( A p ϵa p ) where

Note that a p is positive for all p since K p has only positive entries. Also note that as ˜ L → ∞ , a p → t ( p ) K p,i · and so that a 0 → 0 and a 1 → 1 .

On one hand, we have the time derivative

On the other hand the actual derivative as determined by the new weights: 462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

The only difference is the two terms

One can guarantee with a Grönwall type of argument that the representation path resulting from the new weights must be very close to the path ( A p ϵa p ) .

A.3 Balancedness

This paper will heavily focus on the Hamiltonian H p that is constant throughout the layers p ∈ [0 , 1] , and how it can be interpreted. Note that the Hamiltonian we introduce is distinct from an already known invariant, which arises as the result of so-called balancedness, which we introduce now.

Though this balancedness also appears in ResNets, it is easiest to understand in fullyconnected networks. First observe that for any neuron i ∈ 1 , . . . , w at a layer ℓ one can multiply the incoming weights ( W ℓ,i · , b ℓ,i ) by a scalar α and divide the outcoming weights W ℓ +1 , · i by the same scalar α without changing the subsequent layers. One can easily see that the scaling that minimize the contribution to the parameter norm is such that the norm of incoming weights equals the norm of the outcoming weights ∥ W ℓ,i · ∥ 2 + ∥ b ℓ,i ∥ 2 = ∥ W ℓ +1 , · i ∥ 2 . Summing over the i s we obtain ∥ W ℓ ∥ 2 F + ∥ b ℓ ∥ 2 = ∥ W ℓ +1 ∥ 2 F and thus ∥ W ℓ ∥ 2 F = ∥ W 1 ∥ 2 F + ∑ ℓ -1 k =1 ∥ b k ∥ 2 F , which means that the norm of the weights is increasing throughout the layers, and in the absence of bias, it is even constant.

Leaky ResNet exhibit the same symmetry: 478

Proposition 9. At any critical W p , we have ∥ W p ∥ 2 = ∥ W 0 ∥ 2 + ˜ L ∫ p 0 ∥ W p, · w +1 ∥ 2 dq . 479

480

481

Proof. This proofs handles the bias W p, · ( w +1) differently to the rest of the weights W p, · (1: w ) , to simplify notations, we write V p = W p, · (1: w ) and b p = W p, · ( w +1) for the bias.

482

483

484

First let us show that choosing the weight matrices ˜ V q = r ′ ( q ) V r ( q ) and bias ˜ b q = r ′ ( q ) e ˜ L ( r ( q ) -q ) b r ( q ) leads to the path ˜ A q = e ˜ L ( r ( q ) -q ) A r ( q ) . Indeed the path ˜ A q = e ˜ L ( r ( q ) -q ) A r ( q ) has the right value when p = 0 and it then satisfies the right differential equation:

The optimal reparametrization r ( q ) is therefore the one that minimizes 485

For the identity reparametrization r ( q ) = q to be optimal, we need 486

for all dr ( q ) with dr (0) = dr (1) = 0 . Since

we need

and thus for all p

Integrating, we obtain as needed

B Experimental Setup

Our experiments make use of synthetic data to train leaky ResNets so that the Bottleneck rank k ∗ is known for our experiments. The synthetic data is generated by teacher networks for a given true rank k ∗ . To construct a bottleneck, the teacher network is a composition of networks for which the the inner-dimension is k ∗ . Our experiments used an input and output dimension of 30, and a bottleneck of k ∗ = 3 . For data, we sampled a thousand data points for training, and another thousand for testing which are collectively augmented by demeaning and normalization.

To train the leaky ResNets, it is important for them to be wide, usually wider than the input or output dimension, we opted for a width of 100. However, the width of the representation must be constant to implement leaky residual connections, so we introduce a single linear mapping at the start, and another at the end, of the forward pass to project the representations into a higher dimension for the paths. These linear mappings can be either learned or fixed.

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

∂

p

(

∥

W

p

∥

2

+

∥

b

p

∥

2

)

= ˜

L

∥

b

p

∥

2

.

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

p p p p p from the Hamiltonian reformulation. Figure 3 shows how their respective magnitudes remains relatively constant as the effective depth ˜ L grows.

For compute resources, these small networks are not particularly resource intensive. Even on a CPU, 522 it only takes a couple minutes to fully train a leaky ResNet. 523

maxl|o(A,lloll8l

103

102

101

100

3

4

5

6

8

Effective depth L

To achieve a tight convergence in training, we train primarily using Adam using Mean Squared Error as a loss function, and our custom weight decay function. After training on Adam (we found 5000 epochs to work well), we then train briefly (usually 1000 epochs) using SGD with a smaller learning rate to tighten the convergence.

The bottleneck structure of a trained network, as seen in Figure 1b and 2b, can be observed in the spectra of both the representations A p and the weight matrices W p at each layer. As long as the training is not over-regularized ( λ too large) then the spectra reveals a clear separation between k ∗ number of large values as the rest decay. In our experiments, λ = 0 . 001 ˜ L to get good results. To facilitate the formation of the bottleneck structure, L should be large, for our experiments we usually use L = 20 . Figure 2a shows how larger L , which have better separation between large and small singular values, lead to improved test performance.

As first noted in section 1.3, solving for the Cost Of Identity, the kinetic energy, and the Hamiltonian H is difficult due to the instability of the pseudo-inverse. Although the relaxation ( K p + γI ) improves the stability, we also utilize the solve function to avoid computing a pseudo-inverse altogether. The stability of these computations rely on the boundedness of some additional properties: the path length ∫ || ∂ A || dp , as well as the magnitudes of B , and B σ ( A ) T

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contribution section accurately describes our contributions, and all theorems/propositions are proven in the main or the appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations of our results and approach after we state them.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are stated in the Theorem statements.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental setup is described in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We use synthetic data, with a description of how to build this synthetic data. The code is not the main contribution of the paper, so there is little reason to publish it.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Most details are given in the experimental setup section in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The numerical experiments are mostly there as a visualization of the theoretical results, our main goal is therefore clarity, which would be hurt by putting error bars everywhere.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the experimental setup section of the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the Code of Ethics and see no issue.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is theoretical in nature, so it has no direct societal impact that can be meaningfully discussed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not relevant to our paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We only use our own synthetic data.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not relevant to this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not relevant to this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the so-called \"Leaky ResNet\" ordinary differential equation. Leaky ResNets are a variant of the NeuralODE with an additional vector field that attracts trajectories to the origin, the strength of which is governed by a parameter $\\tilde{L}$ that is later shown to correspond to a separation of timescales. \n\nThe authors consider a system of ODEs resulting from the optimization of the empirical risk with a regularization term inversely proportional to $\\tilde{L}$. \n\nThey define the \"Cost of Identity\" (COI), a quantity that expresses the coupling of the ODEs and evolves along the trajectories of the ODEs with initial values depending on the training dataset. They then proceed to show that the solutions spend most time in regions where the COI is close to an optimal value.\n\nThe final part of the paper proposes three different discretization schemes for the ODE and provides numerical results.\n\nSTRENGTHS:\n* The idea of showing that neural networks have a certain property by constructing a model where trajectories spend most of their time in regions with that property is interesting.\n\n* The authors explain their intuition as well as the underlying assumptions of their derivations and highlight the limitations of their analysis.\n\n* The COI seems to be novel and reflect some interesting properties of ODE models for neural-networks.\n\nWEAKNESSES:\n* The main results are not clearly stated in the abstract or introduction. The author's stated goal is to study Leaky ResNets, it would be nice to have a rigorous statement of the results of that study at the beginning of the paper.\n\n* In the abstract, the authors state that the paper explains the emergence of a bottleneck structure in ResNets. It is not clear how this claim can be derived from the results in the paper.\n\n* There is no rigorous justification that the results from the study apply to neural networks with a finite number of layers.\n\n* One of the main ingredients used in the derivations is quoted from another paper in (l.101). It would be nice to have a short proof in the appendix or some explanation to make the paper more self-contained.\n\n* In general the paper lacks rigor, as the authors themselves note in many positions, it is not guaranteed that all the quantities are finite and that the decompositions are justified. It would be welcome if some of the informal discussions could be replaced by rigorous statements and proofs.\n\n* The propositions/theorems in general lack clear statements of which assumptions are used. For example, at the top of page 4 it is stated that the decomposition only holds under a certain assumption (which is rarely satisfied in practice as noted later in the paper). Most of the discussion in the paper relies on this decomposition, but it is not immediately clear whether the formal results require the assumption.\n\n* In general there is a lot of discussion mixing formal definitions and informal arguments, for example reasoning in terms of quantities that don't exist in most cases of interest. This makes the paper somewhat hard to read. I can understand the author's intention of providing some intuition, which might perhaps be better served by a combination of rigorous definitions together with a toy example where all the rigorous quantities simplify.\n\nQUESTIONS:\n* What are the precise assumptions required in each of the Propositions/Theorems?\n* How can this framework be used to analyze neural networks with a finite number of layers?\n\nLIMITATIONS:\nThe authors appropriately discuss the limitations of their results next to their statements. However, the extrapolation to feature learning in ResNets made in the abstract is not justified by the results of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper maps the dynamics of representations across layers of leaky ResNets to a Largrangian and Hamiltonian formulation, giving an intuitive picture of a balance between two terms: a kinetic energy term which favors small layer derivatives and a potential energy\nthat favors low-dimensional representations. This intuition to explain the emergence of a bottleneck structure, as observed in previous work.\n\nSTRENGTHS:\n1. The paper addresses a timely and important topic: feature learning in DNNs. \n\n2. The introduction provides a good connection to previous work.\n\n3. The mapping to a Hamiltonian formulation is interesting and provides a valuable intuition. \n\n4. The propositions and theorems are mostly clearly stated and the proofs seem sound.\n\nWEAKNESSES:\n1. Numerical experiments:\na. Many of the figures are poorly explained and have missing labels etc, e.g. in Figs 1b, 2b what is the color code? \nb. I failed to find a mention of what data the models were trained / evaluated on. \nc. Fig 2c - what is the projection on to? \n\n2. It is sometimes hard to follow the rationale and motivation for the \"storyline\" of the paper and its different sections could be better connected to each other. \n\n\n3. Novelty wrt previous works - in lines 206-208 a difference from similar works is mentioned but this seems is very brief and looks not very significant on the face of it. It would be better to highlight and elaborate on what is new here relative to these previous works. \n\n\n\nTechnical comments: \na. Eqs are not numbered. \nb. I found the notations to be confusing and non standard, e.g. $\\alpha_p \\in \\mathbb{R}^w$ which make it harder to follow the derivations.\n\n\n\nTypos:\na. Lines 211-212: \"layers of the layers...\"\n\nQUESTIONS:\n1. Does $K_p$ bare some analogy with the mass matrix from classical mechanics? \n\n2. The backward pass variables $B_p$ play the role of momenta but why are they defined as in line 190? why not via a Legendre transformation of the velocities? or are these equivalent? \n\n3. Line 224: the analogous condition in classical mechanics is that the potential energy depends only on the coordinates and not on the momenta, and that the kinetic energy is a quadratic form in the velocities. Is there some intuition that can be imported here? \n\n4. In Fig 1c - why are the Hamiltonians not exactly conserved? these are quite substantial deviations.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies feature learning in Leaky ResNets and shows the emergence of the previously studied Bottleneck structure under certain assumptions. In particular the paper provides a Hamiltonian formulation of the features and their dynamics to show that the ResNet will prefer low dimensional features (low potential energy) when the effective depth of the ResNet is large, which gives the Bottleneck structure. The paper also has a final section on choosing the scales of the residual layers across depths motivated by their theory.\n\nSTRENGTHS:\n1. Studies the problem of understanding feature learning in NNs, which is of broader interest in the NeurIPS community.\n2. The paper identifies the effect of “effective depth” in Leaky ResNets on the previously observed Bottleneck structure, through a Hamiltonian decomposition into kinetic and potential energy. \n3. In particular, the authors provide a nice intuition that the potential energy is minimised at large effective depths, which corresponds to low rank solutions.\n\nWEAKNESSES:\n1. The paper is unclear in several important moments which compromises readability. For example, is the leakage parameter $\\tilde{L}$ suppose to lie in [0,1] (as suggested by line 80) or in [0,\\infty] (as is necessary for the “separation of timescales” arguments in section 2.1). Moreover, in line 224 the authors write closed forms for the Hamiltonian but it is not clear how they obtain this object, from the previously stated Hamiltonian on linear 195.\n2. Theory seems tied to several simplifying assumptions which reduces its generality in describing/understanding feature learning in standard NNs, e.g. the reliance on ReLU activation, the need for weight decay to minimise parameter norms (though it has been called into question if the role of weight decay is actually to find minimal parameter norms in practice (https://arxiv.org/abs/2310.04415), or also the omission of normalisation layers.\n3. On a related note, the theory in the paper seems to have a limited relevance for practice. The one glimmer of this is that the paper suggests changing the weighting on the residual branches in order to evenly balance the difference layers in terms of how much the representations are changing, but this seems underdeveloped at present. It would be worth investigating if this can improve training in practice. Moreover, the paper (and the works on the Bottleneck structure in general) seem to argue that the representations should be changing a lot at late layers because the representation shifts back from being low rank. But this is counter to existing practical works that suggest one can prune late residual layers for improved efficiency https://arxiv.org/abs/2207.07061. This represents a gap between this theory (of Bottleneck structures) and practice to me.\n4. The paper studies properties of optimal solutions (e.g. geodesics with minimal parameter norm) in terms of Hamiltonian energies etc (Theorem 4) but does not seem to discuss whether training dynamics will lead to such solutions in practice.\n\nQUESTIONS:\n1. The motivation to study the Bottleneck structure seems not fully convincing to me. I appreciate that the authors spend quite a lot of the introduction to justify the (information) Bottleneck setting from various angles, but has it been shown that practical networks display such a bottleneck structure? Especially in the modern age of large pre-training datasets when the models are underparameterised. It seems like the theory relies on assuming that a low rank structure exists in the function being learnt?\n2. What is the effect of $\\gamma$ in the inverse in line 257 in terms of meaning that the Bottleneck rank is an integer? It seems that will only hold if $\\gamma=0$, in which case is it necessary for the Bottleneck rank to be an integer?\n\n\nTypos:\n\n1. q/2 in equation below line 78\n2. B^T in line 105.\n3. no $\\tilde{L}$ in middle term of equation below line 112.\n4. \"of the layers\" repeated in line 212\n\nLIMITATIONS:\nThere is no limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the feature learning dynamics in Leaky ResNets using Hamiltonian mechanics. By introducing the concept of 'representation geodesics',  the authors analyze continuous paths in representation space that minimize the parameter norm of the network. The study provides a Lagrangian and Hamiltonian reformulation that highlights the importance of balancing kinetic and potential energy during feature learning. The potential energy, which favors low-dimensional representations, becomes dominant as the network depth increases, leading to a bottleneck structure. This structure involves rapid transitions from high-dimensional inputs to low-dimensional representations, slow evolution within the low-dimensional space, and eventual rapid transitions back to high-dimensional outputs. The paper also proposes training with adaptive layer step-size to better handle the separation of timescales observed in the representation dynamics. These insights offer a novel perspective on the mechanisms underlying feature learning in deep neural networks and suggest potential improvements in network training methodologies.\n\nSTRENGTHS:\n- This paper offers a novel approach for understanding feature learning by applying Hamiltonian mechanics to Leaky ResNets, bridging a gap between theoretical physics and machine learning.\n- This paper conducts experiments to validate the findings. Based on experiments, some interesting observations are obtained, which may give some new insights for future works.\n- The insights gained from this study have the potential to influence future research in neural network optimization and feature learning, advancing the state of the art in deep learning theory.\n\nWEAKNESSES:\n1. There are multiple typos in the article, which affect readability. Below are several obvious typos, and it is recommended that the authors carefully polish the language of the article.\n\t-  The third word in line 24, \"phenomenon\"$\\rightarrow$ \"phenomena\".\n\t-   In line 27, \"determines\" $\\rightarrow$ \"determine\".\n\t-   In line 40, \"lead\" $\\rightarrow$ \"leads\".\n\t-   In line 68, the preposition \"in\" should be added after \"interested\".\n\t-  The formula at the end of line 78  should be$$\\alpha_{q}^{'}=\\alpha_{q/c}.$$\n\t-  The formula between lines 78 and 79 is also incorrect. The last term should be $$\\frac{1}{c}\\partial_{p}\\alpha_{q/c}(x).$$\n\t-  The expression of $K_p$  in line 111 is incorrect, because it should depend on $p$.\n\t-  The formula between lines 112 and 113 is incorrect. The coefficient of the middle term on the right side of the equation should be 1 instead of $\\tilde{L}$.\n\t- In line 145, \" $||\\tilde{L}A_{p}+\\partial_{p}A_{p}||_{K_{p}^{+}}^{2}$ \"  $\\rightarrow$  \"$||\\tilde{L}A_{p}+\\partial_{p}A_{p}||_{K_{p}}^{2}$\".\n\t-  In line 159, \"bound\" $\\rightarrow$ \"bounded\".\n\t-  The expression for $C(A)$ in line 190 is incorrect, which should be $$C(A)=\\frac{1}{N}||f^{*}(X)-A||_{F}^{2} .$$\n\t- In line 282, \"$\\rho_{l}L <1$ \" $\\rightarrow$ \"$\\rho_{l}\\tilde{L}<1$\".\n\t- In line 415, \"cones\"  $\\rightarrow$  \"cone\".\n\t- In Theorem 7 and its proof, it seems that all instances of $\\sqrt{\\gamma c}$ should be changed to $\\gamma \\sqrt{c}$.\n\t- In proposition 9, $\\tilde{Z}_{q}$ should be $\\tilde{A}_{q}$ in the formula above line 485.\n2. The formula between lines 101 and 102 is a crucial one, so it is recommended that the authors provide the derivation process for this formula.\n3. In line 59 of the paper, the definition of $\\sigma$ is given. First, the \"+\" in this formula should be replaced with a comma. Secondly, my question is about the last component \"1\" in the definition of $\\sigma$. In the proofs of some propositions, is it necessary that $\\sigma$ does not include this last component \"1\"?\n\t- In line 153, why $||A_{p}||_{K_{p}}^{2}=||A_{p}A_{p}^{+}||_{F}^{2}$ for non-negative $A_{p}$ ?\n\t- In line 156, why $||\\sigma(A)||_{F}\\le ||A||_{F}$ ?\n\t- In line 450, why $||A_{p_{0}, \\cdot i}||^{2}-||\\sigma(A_{p_{0}, \\cdot i})||^{2}=c_{0}>0$ for all $\\tilde{L}$ ?\n4. In the proof of Proposition 8, the authors did not explain why the limit exists. Secondly, the formula above line 454 is missing $O(\\epsilon^4)$. Additionally, the formula above line 464 is incorrect.\n\nQUESTIONS:\n1. In practical applications, the training of neural networks is often affected by random initialization and noise. Could the authors elaborate on how the Hamiltonian maintains its stability and robustness in the presence of such randomness and noise? Are there theoretical or experimental supports demonstrating the effectiveness of the Hamiltonian in describing the system dynamics under different training conditions?\n2. The authors propose an adaptive layer step-size to adapt to the separation of timescales observed in Leaky ResNets. How does this adaptive training approach impact the computational complexity and training time compared to standard training methods? Can the authors provide benchmarks or case studies demonstrating the trade-offs between performance gains and computational costs?\n\nLIMITATIONS:\nThe theory they proposed aims to provide guidance for better training of networks, and their experimental and implementation details have been documented.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}