Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets

## Abstract

Abstract We study Leaky ResNets, which interpolate between ResNets ( L ~ = 0 ~ 𝐿 0 \tilde{L}=0 over~ start_ARG italic_L end_ARG = 0 )
and Fully-Connected nets ( L ~ → ∞ → ~ 𝐿 \tilde{L}\to\infty over~ start_ARG italic_L end_ARG → ∞ ) depending on an ’effective
depth’ hyper-parameter L ~ ~ 𝐿 \tilde{L} over~ start_ARG italic_L end_ARG . In the infinite depth limit,
we study ’representation geodesics’ A p subscript 𝐴 𝑝 A_{p} italic_A start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT : continuous paths in
representation space (similar to NeuralODEs) from input p = 0 𝑝 0 p=0 italic_p = 0 to
output p = 1 𝑝 1 p=1 italic_p = 1 that minimize the parameter norm of the network. We
give a Lagrangian and Hamiltonian reformulation, which highlight the
importance of two terms: a kinetic energy which favors small layer
derivatives ∂ p A p subscript 𝑝 subscript 𝐴 𝑝 \partial_{p}A_{p} ∂ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT italic_A start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT and a potential energy that favors
low-dimensional representations, as measured by the ’Cost of Identity’.
The balance between these two forces offers an intuitive understanding
of feature learning in ResNets. We leverage this intuition to explain
the emergence of a bottleneck structure, as observed in previous work:
for large L ~ ~ 𝐿 \tilde{L} over~ start_ARG italic_L end_ARG the potential energy dominates and leads to
a separation of timescales, where the representation jumps rapidly
from the high dimensional inputs to a low-dimensional representation,
move slowly inside the space of low-dimensional representations, before
jumping back to the potentially high-dimensional outputs. Inspired
by this phenomenon, we train with an adaptive layer step-size
to adapt to the separation of timescales.

## 1 Introduction

Feature learning is generally considered to be at the center of the
recent successes of deep neural networks (DNNs), but it also remains
one of the least understood aspects of DNN training.

There is a rich history of empirical analysis of the features learned
by DNNs, for example the appearance of local edge detections in CNNs
with a striking similarity to the biological visual cortex Krizhevsky et al. (),
feature arithmetic properties of word embeddings Mikolov et al. (),
similarities between representations at different layers Kornblith et al. (); Li and Papyan (),
or properties such as Neural Collapse Papyan et al. ()
to name a few. While some of these phenomenon have been studied theoretically
Arora et al. (); Ethayarajh et al. (); Súkeník et al. (),
a more general theory of feature learning in DNNs is still lacking.

For shallow networks, there is now strong evidence that the first
weight matrix is able to recognize a low-dimensional projection of the inputs
that determines the output (assuming this structure is present) Bach (); Abbe et al. (, ).
A similar phenomenon appears in linear networks, where the network
is biased towards learning low-rank functions and low-dimensional
representations in its hidden layers Gunasekar et al. (); Li et al. (); Wang and Jacot ().
But in both cases the learned features are restricted to depend linearly
on the inputs, and the feature learning happens in the very first
weight matrix, whereas it has been observed that features increase
in complexity throughout the layers Zeiler and Fergus ().

The linear feature learning ability of shallow networks has inspired
a line of work that postulates that the weight matrices learn to align
themselves with the backward gradients and that by optimizing for
this alignment directly, one can achieve similar feature learning
abilities even in deep nets Beaglehole et al. (); Radhakrishnan et al. ().

For deep nonlinear networks, a theory that has garnered a lot of interest
is the Information Bottleneck Tishby and Zaslavsky (),
which observed amongst other things that the inner representations
appear to maximize their mutual information with the outputs, while
minimizing the mutual information with the inputs. A limitation of
this theory is its reliance on the notion of mutual information which
has no obvious definition for empirical distributions, which lead
to some criticism Saxe et al. ().

A recent theory that is similar to the Information Bottleneck but
with a focus on the dimensionality/rank of the representations and
weight matrices rather than the mutual information is the Bottleneck
rank/Bottleneck structure Jacot (, ); Wen and Jacot ():
which describes how, for large depths, most of the representations
will have approximately the same low dimension, which equals the Bottleneck
rank of the task (the minimal dimension that the inputs can be projected
to while still allowing for fitting the outputs). The intuitive explanation
for this bias is that a smaller parameter norm is required to (approximately)
represent the identity on low-dimensional representations rather than
high dimensional ones. Some other types of low-rank bias have been
observed in recent work Galanti et al. (); Guth et al. ().

In this paper we will focus on describing the Bottleneck structure
in ResNets, and formalize the notion of ‘cost of identity’ as a driving
force for the bias towards low dimensional representation. The ResNet
setup allows us to consider the continuous paths in representation
space from input to output, similar to the NeuralODE Chen et al. (),
and by adding weight decay, we can analyze representation geodesics,
which are paths that minimize parameter norm, as already studied in
Owhadi ().

### 1.1 Leaky ResNets

Our goal is to study a variant of the NeuralODE Chen et al. (); Owhadi ()
approximation of ResNet with leaky skip connections and with $L_{2}$-regularization.
The classical NeuralODE describes the continuous evolution of the
activations $\alpha_{p}(x)\in\mathbb{R}^{w}$ starting from $\alpha_{0}(x)=x$
at the input layer $p=0$ and then follows

$$ $\partial_{p}\alpha_{p}(x)=W_{p}\sigma(\alpha_{p}(x))$ $$

for the $w\times(w+1)$ matrices $W_{p}$ and the nonlinearity $\sigma:\mathbb{R}^{w}\to\mathbb{R}^{w+1}$
which maps a vector $z$ to $\sigma(z)=\left(\begin{array}[]{cccc}[z_{1}]_{+}&\dots&[z_{w}]_{+}&1\end{array
}\right),$ applying the ReLU nonlinearity entrywise and appending a new entry
with value $1$. Thanks to the appended $1$ we do not need any explicit
bias, since the last column $W_{p,\cdot w+1}$ of the weights replaces the bias.

This can be thought of as a continuous version of the traditional
ResNet with activations $\alpha_{\ell}(x)$ for $\ell=1,\dots,L$: $\alpha_{\ell+1}(x)=\alpha_{\ell}(x)+W_{\ell}\sigma(\alpha_{\ell}(x)).$

We will focus on Leaky ResNets, a variant of ResNets that
interpolate between ResNets and FCNNs, by tuning the strength of the
skip connections leading to the following ODE with parameter $\tilde{L}$:

$$ $\partial_{p}\alpha_{p}(x)=-\tilde{L}\alpha_{p}(x)+W_{p}\sigma(\alpha_{p}(x)).$ $$

This can be thought of as the continuous version of $\alpha_{\ell+1}(x)=(1-\tilde{L})\alpha_{\ell}(x)+W_{\ell}\sigma(\alpha_{\ell}(
x))$.
As we will see, the parameter $\tilde{L}$ plays a similar role as
the depth in a FCNN.

Finally we will be interested describing the paths that minimize a
cost with $L_{2}$-regularization

$$ $\min_{W_{p}}\frac{1}{N}\sum_{i=1}^{N}\left\|f^{*}(x_{i})-\alpha_{1}(x_{i}) \right\|^{2}+\frac{\lambda}{2\tilde{L}}\int_{0}^{1}\left\|W_{p}\right\|_{F}^{2 }dp.$ $$

The scaling of $\frac{\lambda}{\tilde{L}}$ for the regularization
term will be motivated in Section 1.2.

This type of optimization has been studied in Owhadi ()
without leaky connections, but we will describe in this paper large
$\tilde{L}$ behavior which leads to a so-called Bottleneck structure
Jacot (, ) as a result of a
separation of time scales in $p$.

### 1.2 A Few Symmetries

Changing the leakage parameter $\tilde{L}$ is equivalent (up to constants)
to changing the integration range $[0,1]$ or to scaling the outputs.

Integration range: Consider the weights $W_{p}$ on the range
$[0,1]$ and leakage parameter $\tilde{L}$, leading to activations
$\alpha_{p}$. Then stretching the weights to a new range $[0,c]$,
by defining $W_{q}^{\prime}=\frac{1}{c}W_{q/c}$ for $q\in[0,c]$, and dividing
the leakage parameter by $c$, stretches the activations $\alpha^{\prime}_{q}=\alpha_{p/c}$:

$$ $\partial_{q}\alpha^{\prime}_{q}(x)=-\frac{\tilde{L}}{c}\alpha^{\prime}_{q}(x)+ \frac{1}{c}W_{q/c}\sigma(\alpha^{\prime}_{q}(x))=\frac{1}{c}\partial_{p}\alpha _{q/2}(x),$ $$

and the parameter norm is simply divided by $c$:$\int_{0}^{c}\left\|W^{\prime}_{q}\right\|^{2}dq=\frac{1}{c}\int_{0}^{1}\left\|
W_{p}\right\|^{2}dp$.

This implies that a path on the range $[0,c]$ with leakage parameter
$\tilde{L}=1$ is equivalent to a path on the range $[0,1]$ with
leakage parameter $\tilde{L}=c$ up to a factor of $c$ in front of
the parameter weights. For this reason, instead of modeling different
depths as changing the integration range, we will keep the integration
range to $[0,1]$ for convenience but change the leakage parameter
$\tilde{L}$ instead. To get rid of the factor in front of the integral,
we choose a regularization term of the form $\frac{\lambda}{\tilde{L}}$.
From now on, we call $\tilde{L}$ the (effective) depth of the network.

Note that this also suggests that in the absence of leakage ($\tilde{L}=0$),
changing the range of integration has no effect on the effective depth,
since $2\tilde{L}=0$ too. Instead, in the absence of leakage, the
effective depth can be increased by scaling the outputs as we now
show.

Output scaling: Given a path $W_{p}$ on the $[0,1]$ (for
simplicity, we assume that there are no bias, i.e. $W_{p,\cdot w+1}=0$),
then increasing the leakage by a constant $\tilde{L}\to\tilde{L}+c$
leads to a scaled down path $\alpha^{\prime}_{p}=e^{-cp}\alpha_{p}$. Indeed
we have $\alpha^{\prime}_{0}(x)=\alpha_{0}(x)$ and

$$ $\displaystyle\partial_{p}\alpha^{\prime}_{p}(x)$ $\displaystyle=-(\tilde{L}+c)\alpha^{\prime}_{p}(x)+W_{p}\sigma(\alpha^{\prime} _{p}(x))=e^{-cp}\left(\partial_{p}\alpha_{p}(x)-c\alpha_{p}(x)\right)=\partial _{p}(e^{-cp}\alpha_{p}(x)).$ $$

Thus a nonleaky ResNet $\tilde{L}=0$ with very large outputs $\alpha_{1}(x)$
is equivalent to a leaky ResNet $\tilde{L}>0$ with scaled down outputs
$e^{-\tilde{L}}\alpha_{1}(x)$. Such large outputs are common when
training on cross-entropy loss, and other similar losses that are
only minimized at infinitely large outputs. When trained on
such losses, it has been shown that the outputs of neural nets will
keep on growing during training Gunasekar et al. (); Chizat and Bach (),
suggesting that when training ResNets on such a loss, the effective
depth increases during training (though quite slowly).

### 1.3 Lagrangian Reformulation

The optimization of Leaky ResNets can be reformulated, leading to
a Lagrangian form.

First observe that the weights $W_{p}$ at any minimizer can be expressed
in terms of the matrix of activations $A_{p}=\alpha_{p}(X)\in\mathbb{R}^{w\times N}$
over the whole training set $X\in\mathbb{R}^{w\times N}$ (similar
to Jacot et al. ()):

$$ $W_{p}=(\tilde{L}A_{p}+\partial_{p}A_{p})\sigma(A_{p})^{+}$ $$

where $(\cdot)^{+}$ is the pseudo-inverse.

We therefore consider the equivalent optimization over the activations
$A_{p}$:

$$ $\min_{A_{p}:A_{0}=X}\frac{1}{N}\left\|f^{*}(X)-A_{1}\right\|^{2}+\frac{\lambda }{2\tilde{L}}\int_{0}^{1}\left\|\tilde{L}A_{p}+\partial_{p}A_{p}\right\|_{K_{p }}^{2}dp.$ $$

This is our first encounter with the norm $\left\|M\right\|_{K_{p}}=\left\|M\sigma(A_{p})^{+}\right\|_{F}$
corresponding to the scalar product $\left\langle A,B\right\rangle_{K_{p}}=\mathrm{Tr}\left[AK_{p}^{+}B\right]$
for $K_{p}=\sigma(A_{p})^{T}\sigma(A_{p})$ that will play a central
role in our upcoming analysis. By convention, we say that $\left\|M\right\|_{K_{p}}=\infty$
if $M$ does not lie in the image of $K_{p}$, i.e. $\mathrm{Im}M^{T}\nsubseteq\mathrm{Im}K_{p}$.

It can be helpful to decompose this loss along the different neurons

$$ $\min_{A_{p}:A_{0}=X}\sum_{i=1}^{w}\frac{1}{N}\left\|f_{i}^{*}(X)-A_{1,i}\right \|^{2}+\frac{\lambda}{2\tilde{L}}\int_{0}^{1}\left\|\tilde{L}A_{p,i\cdot}+ \partial_{p}A_{p,i\cdot}\right\|_{K_{p}}^{2}dp,$ $$

Leading to a particle flow behavior, where the neurons $A_{p,i\cdot}\in\mathbb{R}^{N}$
are the particles. At first glance, it appears that there is no interaction
between the particles, but remember that the norm $\left\|\cdot\right\|_{K_{p}}$
depends on the covariance $K_{p}=\sum_{i=1}^{w}\sigma(A_{i\cdot})\sigma(A_{i\cdot})^{T}$, leading to a global interaction between the neurons.

If we assume that $\mathrm{Im}A_{p}^{T}\subset\mathrm{Im}\sigma(A_{p})^{T}$, we can decompose the inside of the integral as three terms:

$$ $\frac{1}{2\tilde{L}}\left\|\tilde{L}A_{p}+\partial_{p}A_{p}\right\|_{K_{p}^{+} }^{2}=\frac{\tilde{L}}{2}\left\|A_{p}\right\|_{K_{p}}^{2}+\tilde{L}\left \langle\partial_{p}A_{p},A_{p}\right\rangle_{K_{p}^{+}}+\frac{1}{2\tilde{L}} \left\|\partial_{p}A_{p}\right\|_{K_{p}}^{2}.$ $$

The middle term $\left\langle\partial_{p}A_{p},A_{p}\right\rangle_{K_{p}^{+}}$
plays a relatively minor role in our analysis(^1^11In linear networks $\sigma=id$ it can actually be discarded, since it is integrable $\int_{0}^{1}\mathrm{Tr}\left[\partial_{p}A_{p}\sigma(A_{p})^{+}\sigma(A_{p})^{ +T}A_{p}^{T}\right]dp=\log\left|A_{1}\right|_{+}-\log\left|A_{0}\right|_{+}$, where $\left|\cdot\right|_{+}$ is pseudo-determinant, the product of the non-zero singular values. Since its integral only depends on the endpoints, it has no impact on the representation path in between, which is the focus of this paper. In nonlinear networks, we are not able to discard in such a manner, but we will see that in the rest of analysis the two other terms play a central role, while the second term plays less role.), so we focus more on the two other terms:

Cost of identity $\left\|A_{p}\right\|_{K_{p}}^{2}$
/ potential energy $-\frac{\tilde{L}}{2}\left\|A_{p}\right\|_{K_{p}}^{2}$:
This term can be interpreted as a form of potential energy, since
it only depends on the representation $A_{p}$ and not its derivative
$\partial_{p}A_{p}$. We call it the cost of identity (COI), since
it is the Frobenius norm of the smallest weight matrix $W_{p}$ such
that $W_{p}\sigma(A_{p})=A_{p}$. The COI can be interpreted as measuring
the dimensionality of the representation, inspired by the fact if
the representations $A_{p}$ is non-negative (and there is no bias
$\beta=0$), then $A_{p}=\sigma(A_{p})$ and the COI simply equals
the rank $\left\|A_{p}\right\|_{K_{p}}^{2}=\mathrm{Rank}A_{p}$
(this interpretation is further justified in Section 1.4).
We follow the convention of defining the potential energy as the negative of the
term that appears in the Lagrangian, so that the Hamiltonian equals the sum of these two energies.

Kinetic energy $\frac{1}{2\tilde{L}}\left\|\partial_{p}A_{p}\right\|_{K_{p}}^{2}$:
This term measures the size of the representation derivative
$\partial_{p}A_{p}$ w.r.t. the $K_{p}$ norm. It favors paths $p\mapsto A_{p}$
that do not move too fast, especially along directions where $\sigma(A_{p})$
is small.

This suggests that the local optimal paths must balance two objectives
that are sometimes opposed: the kinetic energy favors going from input
representation to output representation in a ‘straight line’ that
minimizes the path length, the COI on the other hand favors paths
that spends most of the path in low-dimensional representations that
have a low COI. The balance between these two goals shifts as the
depth $\tilde{L}$ grows, and for large depths it becomes optimal
for the network to rapidly move to a representation of smallest possible
dimension (not too small that it becomes impossible to map back to
the outputs), remain for most of the layers inside the space of low-dimensional
representations, and finally move rapidly to the output representation;
even if this means doing a large ‘detour’ and having a large kinetic
energy. The main goal of this paper is to describe this general behavior.

Note that one could imagine that as $\tilde{L}\to\infty$ it would
always be optimal to first go to the minimal COI representation which
is the zero representation $A_{p}=0$, but once the network reaches
a zero representation, it can only learn constant representations
afterwards (the matrix $K_{p}=\mathbf{1}\mathbf{1}^{T}$ is then rank
$1$ and its image is the space of constant vectors). So the network
must find a representation that minimizes the COI under the
condition that there is a path from this representation to the outputs.

###### Remark .

While this interpretation and decomposition is a pleasant and helpful
intuition, it is rather difficult to leverage for theoretical proofs
directly. The problem is that we will focus on regimes where the representations
$A_{p}$ and $\sigma(A_{p})$ are approximately low-dimensional (since
those are the representations that locally minimize the COI), leading
to an unbounded pseudo-inverse $\sigma(A_{p})^{+}$. This is balanced
by the fact that $(\tilde{L}A_{p}+\partial_{p}A_{p})$ is small along
the directions where $\sigma(A_{p})^{+}$ explodes, ensuring a finite
weight matrix norm $\left\|\tilde{L}A_{p}+\partial_{p}A_{p}\right\|_{K_{p}^{+}}^{2}$.
But the suppression of $(\tilde{L}A_{p}+\partial_{p}A_{p})$ along
these bad directions usually comes from cancellations, i.e. $\partial_{p}A_{p}\approx-\tilde{L}A_{p}$.
In such cases, the decomposition in three terms of the Lagrangian
is ill adapted since all three terms are infinite and cancel each
other to yield a finite sum $\left\|\tilde{L}A_{p}+\partial_{p}A_{p}\right\|_{K_{p}}^{2}$.
One of our goal is to save this intuition and prove a similar decomposition
with stable equivalent to the cost of identity and kinetic energy
where $K_{p}^{+}$
is replaced by the bounded $\left(K_{p}+\gamma I\right)^{+}$
for the right choice of $\gamma$.

###### Remark .

### 1.4 Cost of Identity as a Measure of Dimensionality

The cost of identity can be thought of as a measure of dimensionality
of the representation. It is obvious for non-negative representations
because $\left\|A_{p}\right\|_{K_{p}^{+}}^{2}=\left\|A_{p}A_{p}{}^{+}\right\|_{F}^{2}=
\mathrm{Rank}A_{p}$,
but in general, it can be shown to upper bound a notion of ‘stable
rank’:

###### Proposition 1 .

$\left\|A\sigma(A)^{+}\right\|_{F}^{2}\geq\frac{\left\|A\right\|_{*}^{2}}{\left
\|A\right\|_{F}^{2}}$
for the nuclear norm $\left\|A\right\|_{*}=\sum_{i=1}^{\mathrm{Rank}A}s_{i}(A)$.

###### Proof.

We know that $\left\|\sigma(A)\right\|_{F}\leq\left\|A\right\|_{F}$,
therefore $\left\|A\sigma(A)^{+}\right\|_{F}^{2}\geq\min_{\left\|B\right\|_{F}\leq\left\|
A\right\|_{F}}\left\|AB^{+}\right\|_{F}^{2}$
which is minimized when $B=\frac{\left\|A\right\|_{F}}{\sqrt{\left\|A\right\|_{*}}}\sqrt{A}$,
yielding the result.
∎

The stable rank $\frac{\left\|A\right\|_{*}^{2}}{\left\|A\right\|_{F}^{2}}$
is upper bounded by $\mathrm{Rank}A$, with equality if all non-zero
singular values of $A$ are equal, and it is lower bound by the more
common notion of stable rank $\frac{\left\|A\right\|_{F}^{2}}{\left\|A\right\|_{op}^{2}}$,
because $\sum s_{i}\max s_{i}\geq\sum s_{i}^{2}$ for the singular
values $s_{i}$.

Note that in contrast to the COI which is a very unstable quantity
because of the pseudo-inverse, the ratio $\frac{\left\|A\right\|_{*}^{2}}{\left\|A\right\|_{F}^{2}}$
is continuous except at $A=0$. This also makes it much easier to
compute empirically than the COI itself.

We know that the COI matches the dimension or rank for positive representations,
but it turns out that the local minima of the COI that are stable
under the addition of a new neuron are all positive:

###### Proposition 2 .

A local minimum of $A\mapsto\left\|A\sigma(A)^{+}\right\|_{F}^{2}$
is said to be stable if it remains a local minimum after concatenating
a zero vector $A^{\prime}=\left(\begin{array}[]{c}A\\
0\end{array}\right)\in\mathbb{R}^{(w+1)\times N}$. All stable minima are non-negative, and satisfy $\left\|A\sigma(A)^{+}\right\|_{F}^{2}=\mathrm{Rank}A$.

###### Proof.

The COI of the nearby point $\left(\begin{array}[]{c}A\\
\epsilon z\end{array}\right)$ for $z\in\mathrm{Im}\sigma(A)^{T}$ equals

$$ $\displaystyle\mathrm{Tr}\left[(A^{T}A+\epsilon^{2}zz^{T})\left((\sigma(A)^{T} \sigma(A)+\epsilon^{2}\sigma(z)\sigma(z)^{T}\right)^{+}\right]$ $\displaystyle=\left\|A\sigma(A)^{+}\right\|^{2}+\epsilon^{2}\left\|z^{T}\sigma (A)^{+}\right\|^{2}-\epsilon^{2}\left\|\sigma(z)^{T}\sigma(A)^{+}\sigma(A)^{+T }A^{T}\right\|^{2}+O(\epsilon^{4}).$ $$

Assume by contradiction that there is a $i=1,\dots,N$ such that $\sigma(A_{\cdot i})\neq A_{\cdot i}$,
then choosing $z=\sigma(A)^{T}\sigma(A_{\cdot i})$ we have $\sigma(z)=z$
and the two $\epsilon^{2}$ terms are negative:

$$ $\epsilon^{2}\left\|\sigma(A_{i})\right\|^{2}-\epsilon^{2}\left\|A_{i}\right\|^ {2}<0,$ $$

which implies that $A^{\prime}$ it is not a local minimum.
∎

These stable minima will play a significant role in the rest of our
analysis, as we will see that for large $\tilde{L}$ the representations
$A_{p}$ of most layers will be close to one such local minimum. Now
we are not able to rule out the existence of non-stable local minima
(nor guarantee that they are avoided with high probability), but
one can show that all strict local minima of wide enough networks
are stable. Actually we can show something stronger, starting from
any non-stable local minimum there is a constant loss path that connects
it to a saddle:

###### Proposition 3 .

If $w>N(N+1)$ then if
$\hat{A}\in\mathbb{R}^{w\times N}$ is local minimum of $A\mapsto\left\|A\sigma(A)^{+}\right\|_{F}^{2}$
that is not non-negative, then there is a continuous path $A_{t}$
of constant COI such that $A_{0}=\hat{A}$ and $A_{1}$ is a saddle.

This could explain why a noisy GD would avoid such negative/non-stable
minima, since there is no ‘barrier’ between the minima and a lower
one, one could diffuse along the path described in Proposition 3
until reaching a saddle and going towards a lower COI minima. But
there seems to be something else that pushes away from such non-negative
minima, as in our experiments with full population GD we have only
observed stable/non-negative local minimas.

Figure: (a) Hamiltonian measures across $\tilde{L}$
Refer to caption: extracted/5624314/figures/hamiltonian.png

###### Proposition 1 .

###### Proof.

###### Proposition 2 .

###### Proof.

###### Proposition 3 .

### 1.5 Hamiltonian Reformulation

We can further reformulate the evolution of the optimal representations
$A_{p}$ in terms of a Hamiltonian, similar to Pontryagin’s maximum
principle.

Let us define the backward pass variables $B_{p}=-\frac{1}{\lambda}\partial_{A_{p}}C(A_{1})$ for the cost $C(A)=\frac{1}{N}\|f^{*}(X)-A\|^{2}_{F}$,
which play the role of the ‘momenta’ of $A_{p}$ in this Hamiltonian
interpretation, which follows the backward differential equation

$$ $\displaystyle B_{1}$ $\displaystyle=-\frac{1}{\lambda}\partial_{A_{1}}C(A_{1})=\frac{2}{\lambda N}(f ^{*}(X)-A_{1})$ $\displaystyle-\partial_{p}B_{p}$ $\displaystyle=\dot{\sigma}(A_{p})\odot\left[W_{p}^{T}B_{p}\right]-\tilde{L}B_{ p}.$ $$

Now at any critical point, we have that $\partial_{W_{p}}C(A_{1})+\frac{\lambda}{\tilde{L}}W_{p}=0$
and thus $W_{p}=-\frac{\tilde{L}}{\lambda}\partial_{A_{p}}C(A_{1})\sigma(A_{p})^{T}=
\tilde{L}B_{p}\sigma(A_{p})^{T}$,
leading to joint dynamics for $A_{p}$ and $B_{p}$:

$$ $\displaystyle\partial_{p}A_{p}$ $\displaystyle=\tilde{L}(B_{p}\sigma(A_{p})^{T}\sigma(A_{p})-A_{p})$ $\displaystyle-\partial_{p}B_{p}$ $\displaystyle=\tilde{L}\left(\dot{\sigma}(A_{p})\odot\left[\sigma(A_{p})B_{p}^ {T}B_{p}\right]-B_{p}\right).$ $$

These are Hamiltonian dynamics $\partial_{p}A_{p}=\partial_{B_{p}}\mathcal{H}$ and $-\partial_{p}B_{p}=\partial_{A_{p}}\mathcal{H}$ w.r.t. the Hamiltonian

$$ $\mathcal{H}(A_{p},B_{p})=\frac{\tilde{L}}{2}\left\|B_{p}\sigma(A_{p})^{T} \right\|^{2}-\tilde{L}\mathrm{Tr}\left[B_{p}A_{p}^{T}\right].$ $$

The Hamiltonian is a conserved quantity, i.e. it is constant in $p$.
It will play a significant role in describing a separation of timescales
that appears for large depths $\tilde{L}$. Another significant advantage
of the Hamiltonian reformulation over the Lagrangian approach is the
absence of the unstable pseudo-inverses $\sigma(A_{p})^{+}$.

###### Remark .

Note that the Lagrangian and Hamiltonian reformulations have already
appeared in previous work Owhadi () for non-leaky
ResNets. Our main contributions are the description in the next section
of the Hamiltonian as the network becomes leakier $\tilde{L}\to\infty$,
the connection to the cost of identity, and the appearance of
a separation of timescales. These structures are harder to observe
in non-leaky ResNets (though they could in theory still appear since
increasing the scale of the outputs is equivalent to increasing the
effective depth $\tilde{L}$ as shown in Section 1.2).

The Lagrangian and Hamiltonian are also very similar to the ones in
Grafke et al. (); Grafke and Vanden-Eijnden (),
and the separation of timescales and rapid jumps that we will describe
also bear a strong similarity. Though a difference with our work is
that the norm $\left\|\cdot\right\|_{K_{p}}$ depends on $A_{p}$
and can be degenerate.

###### Remark .

## 2 Bottleneck Structure in Representation Geodesics

A recent line of work Jacot (, )
studies the appearance of a so-called Bottleneck structure in large
depth fully-connected networks, where the weight matrices and representations
of ‘almost all’ layers of the layers are approximately low-rank/low-dimensional
as the depth grows. This dimension $k$ is consistent across layers,
and can be interpreted as being equal to the so-called Bottleneck
rank of the learned function. This structure has been shown to extend
to CNNs in Wen and Jacot (), and we will observe a similar
structure in our leaky ResNets, further showcasing its generality.

More generally, our goal is to describe the ‘representation geodesics’
of DNNs: the paths in representation space from input
to output representation. The advantage of ResNets (leaky or not)
over FCNNs is that these geodesics can be approximated by continuous paths and are described
by differential equations (as described by the Hamiltonian reformulation).

This section provides an approximation of the Hamiltonian that illustrates
the separation of timescales that appears for large depths, with slow
layers with low COI/dimension, and fast layers with high COI/dimension.

### 2.1 Separation of Timescales

If $\mathrm{Im}A_{p}^{T}\subset\mathrm{Im}\sigma(A_{p})^{T}$, then
the Hamiltonian equals the sum of the kinetic and potential energies:

$$ $\mathcal{H}=\frac{1}{2\tilde{L}}\left\|\partial_{p}A_{p}\right\|_{K_{p}}^{2}- \frac{\tilde{L}}{2}\left\|A_{p}\right\|_{K_{p}}^{2}.$ $$

This implies that $\left\|\partial_{p}A_{p}\right\|_{K_{p}}=\tilde{L}\sqrt{\left\|A_{p}\right\|_{
K_{p}}^{2}+\frac{2}{\tilde{L}}\mathcal{H}}$
which implies that for large $\tilde{L}$, the derivative $\partial_{p}A_{p}$
is only finite at $p$s where the COI $\left\|A_{p}\right\|_{K_{p}}^{2}$
is close to $-\frac{2}{\tilde{L}}\mathcal{H}$. On the other hand,
$\partial_{p}A_{p}$ will blow up for all $p$ with a finite gap $\sqrt{\left\|A_{p}\right\|_{K_{p}}^{2}+\frac{2}{\tilde{L}}\mathcal{H}}>0$
between the COI and the Hamiltonian. This suggests a separation of
timescales as $\tilde{L}\to\infty$, with slow dynamics in layers whose COI/dimension is close to $-\frac{2}{\tilde{L}}\mathcal{H}$
and fast dynamics in the high COI/dimension layers.

But the assumption $\mathrm{Im}A_{p}^{T}\subset\mathrm{Im}\sigma(A_{p})^{T}$
seems to rarely be true in practice, and both kinetic and COI appear
to be often infinite in practice. But up to a few approximations,
the same argument can be made for stable versions of the kinetic energy/COI:

###### Theorem 4 .

For
sequence $A_{p}^{\tilde{L}}$ of geodesics with $\left\|B_{p}^{\tilde{L}}\right\|^{2}\leq c<\infty$,
and any $\gamma>0$, we have

$$ $-\left(\frac{1}{\tilde{L}}\ell_{\gamma,\tilde{L}}+\sqrt{\gamma}c\right)^{2} \leq-\frac{2}{\tilde{L}}\mathcal{H}-\min_{p}\left\|A_{p}^{\tilde{L}}\right\|_{ (K_{p}+\gamma I)}^{2}\leq\gamma c,$ $$

for the path length $\ell_{\gamma,\tilde{L}}=\int_{0}^{1}\left\|\partial_{p}A_{p}^{\tilde{L}}\right
\|_{(K_{p}+\gamma I)}dp$.
Finally

$$ $-\tilde{L}\sqrt{\gamma c}\leq\left\|\partial_{p}A_{p}\right\|_{(K_{p}+\gamma i )}-\tilde{L}\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+\frac{2}{\tilde{ L}}\mathcal{H}}\leq 2\tilde{L}\sqrt{\gamma c}.$ $$

Note that the size of $\|B_{p}^{\tilde{L}}\|^{2}$ can vary a lot throughout the layers, we therefore suggest choosing a $p$-dependent $\gamma$: $\gamma_{p}=\gamma_{0}\|\sigma(A_{p}^{\tilde{L}})\|^{2}_{op}=\gamma_{0}\|K_{p}
\|^{2}_{op}$. There are two motivations for this: first it is natural to have $\gamma$ scale with $K_{p}$, ; and second, since $W_{p}=\tilde{L}B_{p}\sigma(A_{p})^{T}$ is of approximately constant size (thanks to balancedness, see Appendix A.3), we typically have that the size of $B_{p}$ is inversely proportional to that of $\sigma(A_{p})$, so that $\gamma_{p}\|B_{p}\|^{2}$ should keep roughly the same size for all $p$.

Theorem 4 shows that for large $\tilde{L}$ (and choosing
e.g. $\gamma=\tilde{L}^{-1}$), the Hamiltonian is close to the minimal
COI along the path. Second, the norm of the derivative $\left\|\partial_{p}A_{p}\right\|_{(K_{p}+\gamma i)}$
is close to $\tilde{L}$ times the ‘extra-COI’ $\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+\frac{2}{\tilde{L}}\mathcal{
H}}\approx\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}-\min_{q}\left\|A_{
q}\right\|_{(K_{q}+\gamma I)}^{2}}$,
which describes the separation of timescales, with slow ($\sim 1$)
dynamics at layers $p$ where the COI is almost optimal and fast ($\sim\tilde{L}$)
dynamics everywhere the COI is far from optimal.

Assuming a finite length $\ell_{\gamma,\tilde{L}}<\infty$, the norm
of the derivative must be finite at almost all layers, meaning
that the COI/dimensionality is optimal in almost all layers, with
only a countable number of short high COI/dimension jumps. These jumps
typically appear at the beginning and end of the network, because
the input and output dimensionality and COI are (mostly) fixed, so
it will typically be non-optimal, and so there will often be fast
regions close to the beginning and end of the network. We have actually
never observed any jump in the middle of the network, though we are
not able to rule them out theoretically.

If we assume that the paths $A_{p}^{\tilde{L}}$ are stable under
adding a neuron, then we can additionally guarantee that the representations
in the slow layers (‘inside the Bottleneck’) will be non-negative:

###### Proposition 5 .

Let $A_{p}^{\tilde{L}}$ be a uniformly
bounded sequence of local minima for increasing $\tilde{L}$, at any
$p_{0}\in(0,1)$ such that $\left\|\partial_{p}A_{p}\right\|$
is uniformly bounded in a neighborhood of $p_{0}$ for all $\tilde{L}$,
then $A_{p_{0}}^{\infty}=\lim_{\tilde{L}}A_{p_{0}}^{\tilde{L}}$ is
non-negative.

We therefore know that the optimal COI $\min_{q}\left\|A_{q}\right\|_{(K_{q}+\gamma I)}^{2}$
is close to the dimension of the limiting representations $A_{p_{0}}^{\infty}$,
i.e. it must be an integer $k^{*}$ which we call the Bottleneck rank
of the sequence of minima since it is closely related to the Bottleneck
rank introduced in Jacot (). The Hamiltonian $\mathcal{H}$
is then close to $-\frac{\tilde{L}}{2}k^{*}$.

Figure 1 illustrates these phenomena: the Hamiltonian (and the stable Hamiltonians $\mathcal{H}_{\gamma}=\frac{1}{2\tilde{L}}\left\|\partial_{p}A_{p}\right\|_{(K_
{p}+\gamma I)}^{2}-\frac{\tilde{L}}{2}\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^
{2}$) approach the rank $k^{*}=3$ from below, while the minimal COI approaches it from above; The kinetic energy is proportional to the extra COI, and they are both large towards the beginning and end of the network where the weights $W_{p}$ are higher dimensional. We see in Figure 1(c) that the (stable) Hamiltonian are not exactly constant, but it still varies much less than its components, the kinetic and potential energies.

Because of the non-convexity of the loss we are considering, one can imagine that there could exist distinct sequences of local minima as $\tilde{L}\to\infty$, which could have different rank, depending on what low-dimension they reach inside their bottleneck. Indeed in our experiments we have seen that the number of dimensions that are kept inside the bottleneck can vary by 1 or 2, and in FCNN distinct sequences of depth increasing minima with different ranks have been observed in Jacot ().

Figure: (a) Test performance versus depth
Refer to caption: extracted/5624314/figures/test_error_L3.png

###### Theorem 4 .

###### Proposition 5 .

## 3 Discretization Scheme

To use such Leaky ResNets in practice, we need to discretize over
the range $[0,1]$. For this we choose a set of layer-steps $\rho_{1},\dots,\rho_{L}$
with $\sum\rho_{\ell}=1$, and define the activations at the locations
$p_{\ell}=\rho_{1}+\dots+\rho_{\ell}\in[0,1]$ recursively as

$$ $\displaystyle\alpha_{p_{0}}(x)$ $\displaystyle=x$ $\displaystyle\alpha_{p_{\ell}}(x)$ $\displaystyle=(1-\rho_{\ell}\tilde{L})\alpha_{p_{\ell-1}}(x)+\rho_{\ell}W_{p_{ \ell}}\sigma\left(\alpha_{p_{\ell-1}}(x)\right)$ $$

and the regularized cost $\mathcal{L}(\theta)=C(\alpha_{1}(X))+\frac{\lambda}{2\tilde{L}}\sum_{\ell=1}^{
L}\rho_{\ell}\left\|W_{p_{\ell}}\right\|^{2}$,
for the parameters $\theta=(W_{p_{1}},\dots,W_{p_{L}})$. Note that
it is best to ensure that $\rho_{\ell}\tilde{L}$ remains smaller
than $1$ so that the prefactor $(1-\rho_{\ell}\tilde{L})$ does not
become negative, though we will also discuss certain setups where
it might be okay to take larger layer-steps.

Now comes the question of how to choose the $\rho_{\ell}$s. We consider
three options:

Equidistant: The simplest choice is to choose equidistant
points $\rho_{\ell}=\frac{1}{L}$. Note that the condition $\rho_{\ell}L<1$
then becomes $L>\tilde{L}$. But this choice might be ill adapted
in the presence of a Bottleneck structure due to the separation of
timescales.

Irregular: Since we typically observe that the fast layers
appear close to the inputs and outputs with a slow bottleneck in the
middle, one could simply choose the $\rho_{\ell}$ to be go from small
to large and back to small as $\ell$ ranges from $1$ to $L$. This
way there are many discretized layers in the fast regions close to
the input and output and not too many layers inside the Bottleneck
where the representations are changing less. More concretely one can
choose $\rho_{\ell}=\frac{1}{L}+\frac{a}{L}(\frac{1}{4}-\left|\frac{\ell}{L}-\frac{1}{
2}\right|)$
for $a\in[0,1)$, the choice $a=0$ leads to an equidistant mesh,
but increasing $a$ will lead to more points close to the inputs and
outputs. To guarantee $\rho_{\ell}\tilde{L}<1$, we need $L>(1+a\frac{1}{4})\tilde{L}$.

Adaptive: But this can be further improved by choosing the
$\rho_{\ell}$ to guarantee that the distances $\left\|A_{\ell}-A_{\ell-1}\right\|/\|A_{p}\|$
are approximately the same for all $\ell$ (we divide by the size of $A_{p}$ since it can vary a lot throughout the layers). Since the rate of change of $A_{p}$ is proportional to $\rho_{\ell}$ ($\left\|A_{\ell}-A_{\ell-1}\right\|/\|A_{p}\|=\rho_{\ell}c_{\ell}$),
it is optimal to choose $\rho_{\ell}=\frac{c_{\ell}^{-1}}{\sum c_{\ell}^{-1}}$ for $c_{\ell}=\nicefrac{{\left\|A_{\ell}-A_{\ell-1}\right\|}}{{\rho_{\ell}\|A_{p}\|}}$.
The update $\rho_{\ell}\leftarrow\frac{c_{i}^{-1}}{\sum c_{i}^{-1}}$
can be done at every training step or every few training steps.

Note that the condition $\rho_{\ell}\tilde{L}<1$ might not be necessary inside
the bottleneck since we have the approximation $W_{p}\sigma(A_{p_{\ell-1}})\approx\tilde{L}A_{p_{\ell-1}}$,
canceling out the negative direction. In particular with the adaptive
layer-steps that we propose, a large $\rho_{\ell}$ is only possible
for layers where $c_{\ell}$ is small, which is only possible when
$W_{p}\sigma(A_{p_{\ell-1}})\approx\tilde{L}A_{p_{\ell-1}}$.

Figure 2 illustrates the effect of the choice of $\rho_{\ell}$ for different depths $L$, we see a small but consistent advantage in the test error when using adaptive or irregular $\rho_{\ell}$s. Looking at the resulting Bottleneck structure, we see that the adaptive $\rho_{\ell}$s result in more steps especially in the beginning of the network, but also at the end. This because the ‘true function’ $f^{*}:\mathbb{R}^{30}\to\mathbb{R}^{30}$ we are fitting in these experiments is of the form $f^{*}=g_{3}\circ g_{2}\circ g_{1}$ where the first inner dimension is 6 and the second is 3, thus resulting in a rank of $k^{*}=3$. But before reaching this minimal dimension, the network needs to represent $g_{2}\circ g_{1}$, which requires more layers, and one can almost see that the weight matrices are roughly $6$-dimensional around $p=0.3$. The adaptivity to this structure could explain the advantage in the test error.

## 4 Conclusion

We have given a description of the representation geodesics $A_{p}$
of Leaky ResNets. We have identified an invariant, the Hamiltonian,
which is the sum of the kinetic and potential energy, where the kinetic
energy measures the size of the derivative $\partial_{p}A_{p}$, while
the potential energy is inversely proportional to the cost of identity,
which is a measure of dimensionality of the representations. As the
effective depth of the network grows, the potential energy dominates
and we observe a separation of timescales. At layers with minimal dimensionality over the path, the kinetic energy (and thus the derivative $\partial_{p}A_{p}$) is finite. Conversely, at layers where the representation is higher-dimensional, the kinetic energy must scale with $\tilde{L}$. This leads to a Bottleneck structure,
with a short, high-dimensional jump from the input representation
to a low dimensional representation, followed by slow dynamics inside
the space of low-dimensional representations followed by a final high-dimensional
jump to the high dimensional outputs.

## Appendix A Proofs

### A.1 Cost of Identity

###### Proposition 6 (Proposition 3 in the main.) .

If $w>N(N+1)$ then if $\hat{A}\in\mathbb{R}^{w\times N}$ is local
minimum of $A\mapsto\left\|A\sigma(A)^{+}\right\|_{F}^{2}$
that is not non-negative, then there is a continuous path $A_{t}$
of constant COI such that $A_{0}=\hat{A}$ and $A_{1}$ is a saddle.

###### Proof.

The local minimum $\hat{A}$ leads to a pair of $N\times N$ covariance
matrices $\hat{K}=\hat{A}^{T}\hat{A}$ and $\hat{K}^{\sigma}=\sigma(\hat{A})^{T}\sigma(\hat{A})$.
The pair $(\hat{K},\hat{K}^{\sigma})$ belongs to the conical hull
$\mathrm{Cone}\left\{(\hat{A}_{i\cdot}\hat{A}_{i\cdot}^{T},\sigma(\hat{A}_{i
\cdot})\sigma(\hat{A}_{i\cdot})^{T}):i=1,\dots,w\right\}$.
Since this cones lies in a $N(N+1)$-dimensional space (the space
of pairs of symmetric $N\times N$ matrices), we know by Caratheodory’s
theorem (for convex cones) that there is a conical combination $(\hat{K},\hat{K}^{\sigma}-\beta^{2}\mathbf{1}_{N\times N})=\sum_{i=1}^{w}a_{i}
(\hat{A}_{i\cdot}\hat{A}_{i\cdot}^{T},\sigma(\hat{A}_{i\cdot})\sigma(\hat{A}_{
i\cdot})^{T})$
such that no more than $N(N+1)$ of the coefficients are non-zero.
We now define $A_{t}$ to have lines $A_{t,i\cdot}=\sqrt{(1-t)+ta_{i}}\hat{A}_{i\cdot}$,
so that $A_{t=0}=\hat{A}$ and at $t=1$ at least one line of $A_{t=1}$
is zero (since at least one of the $a_{i}$s is zero). First note
that the covariance pairs remain constant over the path: $K_{t}=A_{t}^{T}A_{t}=\sum_{i=1}^{w}((1-t)+ta_{i})\hat{A}_{i\cdot}\hat{A}_{i
\cdot}^{T}=(1-t)\hat{K}+t\hat{K}=\hat{K}$
and similarly $K_{t}^{\sigma}=\hat{K}^{\sigma}$, which implies that
the cost $\left\|A_{t}\sigma(A_{t})^{+}\right\|_{F}^{2}=\mathrm{Tr}\left[K_{t}K_{t}^{
\sigma+}\right]$
is constant too. Second, since a representation $A$ is non-negative
iff the covariances satisfy $K=K^{\sigma}$, the representation path
$A_{t}$ cannot be non-negative either since it has the same kernel
pairs $(\hat{K},\hat{K}^{\sigma})$ with $\hat{K}\neq\hat{K}^{\sigma}$.

Now (the converse of) Proposition 2
tells us that if $A_{t=1}$ is not non-negative and has a zero line,
then it is not a local minimum, which implies that it is a saddle.
∎

###### Proposition 6 (Proposition 3 in the main.) .

###### Proof.

### A.2 Bottleneck

###### Theorem 7 .

For any uniformly bounded sequence $A_{p}^{\tilde{L}}$ of geodesics,
i.e. $\left\|A_{p}^{\tilde{L}}\right\|^{2},\left\|B_{p}^{\tilde{L}}\right\|^{2}\leq c<\infty$,
and any $\gamma>0$, we have

$$ $-\left(\frac{1}{\tilde{L}}\ell_{\gamma,\tilde{L}}+\sqrt{\gamma}c\right)^{2} \leq-\frac{2}{\tilde{L}}\mathcal{H}-\min_{p}\left\|A_{p}^{\tilde{L}}\right\|_{ (K_{p}+\gamma I)}^{2}\leq\gamma c,$ $$

for the path length $\ell_{\gamma,\tilde{L}}=\int_{0}^{1}\left\|\partial_{p}A_{p}^{\tilde{L}}\right
\|_{(K_{p}+\gamma I)}dp$.
Finally

$$ $-\tilde{L}\sqrt{\gamma c}\leq\left\|\partial_{p}A_{p}\right\|_{(K_{p}+\gamma i )}-\tilde{L}\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+\frac{2}{\tilde{ L}}\mathcal{H}}\leq 2\tilde{L}\sqrt{\gamma c}.$ $$

###### Proof.

First observe that

$$ $\displaystyle\left\|\frac{1}{\tilde{L}}\partial_{p}A_{p}+\gamma B_{p}\right\|_ {(K_{p}+\gamma I)}^{2}$ $\displaystyle=\left\|B_{p}(K_{p}+\gamma)-A_{p}\right\|_{(K_{p}+\gamma I)}^{2}$ $\displaystyle=\left\|B_{p}\sigma(A_{p})^{T}\right\|^{2}+\gamma\left\|B_{p} \right\|^{2}-2\mathrm{Tr}\left[B_{p}A_{p}^{T}\right]+\left\|A_{p}\right\|_{(K_ {p}+\gamma I)}^{2}$ $\displaystyle=\frac{2}{\tilde{L}}\mathcal{H}+\gamma\left\|B_{p}\right\|^{2}+ \left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}$ $$

and thus we have

$$ $-\frac{2}{\tilde{L}}\mathcal{H}=\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}- \left\|\frac{1}{\tilde{L}}\partial_{p}A_{p}+\gamma B_{p}\right\|_{(K_{p}+ \gamma I)}^{2}+\gamma\left\|B_{p}\right\|^{2}.$ $$

(1) The upper bound $-\frac{2}{\tilde{L}}\mathcal{H}-\min_{p}\left\|A_{p}^{\tilde{L}}\right\|_{(K_{
p}+\gamma I)}^{2}\leq\gamma c$
then follows from the fact that $\left\|B_{p}\right\|^{2}\leq c$.
For the lower bound, first observe that

$$ $\displaystyle\frac{1}{\tilde{L}}\left\|\partial_{p}A_{p}\right\|_{(K_{p}+ \gamma I)}$ $\displaystyle\geq\left\|\frac{1}{\tilde{L}}\partial_{p}A_{p}+\gamma B_{p} \right\|_{(K_{p}+\gamma I)}-\left\|\gamma B_{p}\right\|_{(K_{p}+\gamma I)}$ $\displaystyle\geq\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+\frac{2}{ \tilde{L}}\mathcal{H}+\gamma\left\|B_{p}\right\|^{2}}-\sqrt{\gamma c}$ $\displaystyle\geq\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+\frac{2}{ \tilde{L}}\mathcal{H}}-\sqrt{\gamma c},$ (1) $$

and therefore

$$ $\displaystyle\frac{1}{\tilde{L}}\ell_{\gamma,\tilde{L}}$ $\displaystyle=\frac{1}{\tilde{L}}\int_{0}^{1}\left\|\partial_{p}A_{p}\right\|_ {(K_{p}+\gamma I)}dp$ $\displaystyle\geq\int_{0}^{1}\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2} +\frac{2}{\tilde{L}}\mathcal{H}}-\sqrt{\gamma c}dp$ $\displaystyle\geq\sqrt{\min_{p}\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+ \frac{2}{\tilde{L}}\mathcal{H}}-\sqrt{\gamma c}$ $$

which implies the lower bound.

(2) The lower bound follows from equation 1.
The upper bound follows from

$$ $\displaystyle\frac{1}{\tilde{L}}\left\|\partial_{p}A_{p}\right\|_{(K_{p}+ \gamma I)}$ $\displaystyle\leq\left\|\frac{1}{\tilde{L}}\partial_{p}A_{p}+\gamma B_{p} \right\|_{(K_{p}+\gamma I)}+\left\|\gamma B_{p}\right\|_{(K_{p}+\gamma I)}$ $\displaystyle\leq\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+\frac{2}{ \tilde{L}}\mathcal{H}+\gamma\left\|B_{p}\right\|^{2}}+\sqrt{\gamma c}$ $\displaystyle\leq\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+\frac{2}{ \tilde{L}}\mathcal{H}}+\sqrt{\gamma}\left\|B_{p}\right\|+\sqrt{\gamma c}$ $\displaystyle\leq\sqrt{\left\|A_{p}\right\|_{(K_{p}+\gamma I)}^{2}+\frac{2}{ \tilde{L}}\mathcal{H}}+2\sqrt{\gamma c}.$ $$

∎

###### Proposition 8 (Proposition 5 in the main.) .

Let $A_{p}^{\tilde{L}}$ be a uniformly bounded sequence of local
minima for increasing $\tilde{L}$, at any $p_{0}\in(0,1)$ such that
$\left\|\partial_{p}A_{p}\right\|$ is uniformly bounded in
a neighborhood of $p_{0}$ for all $\tilde{L}$, then $A_{p_{0}}^{\infty}=\lim_{\tilde{L}}A_{p_{0}}^{\tilde{L}}$
is non-negative.

###### Proof.

Given a path $A_{p}$ with corresponding weight matrices $W_{p}$
corresponding to a width $w$, then $\left(\begin{array}[]{c}A\\
0\end{array}\right)$ is a path with weight matrix $\left(\begin{array}[]{cc}W_{p}&0\\
0&0\end{array}\right)$. Our goal is to show that for sufficiently large depths, one can
under certain assumptions slightly change the weights to obtain a
new path with the same endpoints but a slightly lower loss, thus ensuring
that if certain assumptions are not satisfied then the path cannot
be locally optimal.

Let us assume that $\left\|\partial_{p}A_{p}\right\|\leq c_{1}$
in a neighborhood of a $p_{0}\in(0,1)$, and assume by contradiction
that there is an input index $i=1,\dots,N$ such that $A_{p_{0},\cdot i}$
has at least one negative entry, and therefore $\left\|A_{p_{0},\cdot i}\right\|^{2}-\left\|\sigma(A_{p_{0},\cdot i})\right\|^
{2}=c_{0}>0$
for all $\tilde{L}$.

We now consider the new weights

$$ $\left(\begin{array}[]{cc}W_{p}-\tilde{L}\epsilon^{2}t(p)A_{p,\cdot i}\sigma(A_ {p,\cdot i})^{T}&\epsilon\tilde{L}t(p)A_{p,\cdot i}\\ \epsilon\tilde{L}t(p)\sigma(A_{p,\cdot i})&0\end{array}\right)$ $$

for $t(p)=\max\{0,1-\frac{\left|p-p_{0}\right|}{r}\}$ a triangular
function centered in $p_{0}$ and for an $\epsilon>0$.

For $\epsilon$ and $r$small enough, the parameter norm will decrease:

$$ $\displaystyle\int_{0}^{1}\left\|\begin{array}[]{cc}W_{p}-\tilde{L}\epsilon^{2} t(p)A_{p,\cdot i}\sigma(A_{p,\cdot i})^{T}&\epsilon\tilde{L}t(p)A_{p,\cdot i} \\ \epsilon\tilde{L}t(p)\sigma(A_{p,\cdot i})&0\end{array}\right\|^{2}dp$ $\displaystyle=\int_{0}^{1}\left\|W_{p}\right\|^{2}+\tilde{L}^{2}\epsilon^{2}t( p)^{2}\left(-\frac{2}{\tilde{L}}A_{p,\cdot i}^{T}W_{p}\sigma(A_{p,\cdot i})+ \left\|A_{p,\cdot i}\right\|^{2}+\left\|\sigma(A_{p,\cdot i})\right\|^{2} \right)dp.$ $$

Now since $W_{p}\sigma(A_{p,\cdot i})=\partial_{p}A_{p,\cdot i}+\tilde{L}A_{p,\cdot i}$,
this simplifies to

$$ $\int_{0}^{1}\left\|W_{p}\right\|^{2}+\tilde{L}^{2}\epsilon^{2}t(p)^{2}\left(- \left\|A_{p,\cdot i}\right\|^{2}+\left\|\sigma(A_{p,\cdot i})\right\|^{2}- \frac{1}{\tilde{L}}A_{p,\cdot i}^{T}\partial_{p}A_{p,\cdot i}\right)dp+O( \epsilon^{4}).$ $$

By taking $r$ small enough, we can guarantee that $-\left\|A_{p,\cdot i}\right\|^{2}+\left\|\sigma(A_{p,\cdot i})\right\|^{2}<-
\frac{c_{0}}{2}$
for all $p$ such that $t(p)>0$, and for $\tilde{L}$ large enough
we can guarantee that $\left|\frac{1}{\tilde{L}}A_{p,\cdot i}^{T}\partial_{p}A_{p,\cdot i}\right|$
is smaller then $\frac{c_{0}}{4}$, so that we can guarantee that
the parameter norm will be strictly smaller for $\epsilon$ small
enough.

We will now show that with these new weights the path becomes approximately
$\left(\begin{array}[]{c}A_{p}\\
\epsilon a_{p}\end{array}\right)$ where

$$ $a_{p}=\tilde{L}\int_{0}^{p}t(q)K_{p,i\cdot}e^{\tilde{L}(q-p)}dq.$ $$

Note that $a_{p}$ is positive for all $p$ since $K_{p}$ has only
positive entries. Also note that as $\tilde{L}\to\infty$, $a_{p}\to t(p)K_{p,i\cdot}$
and so that $a_{0}\to 0$ and $a_{1}\to 1$.

On one hand, we have the time derivative

$$ $\partial_{p}\left(\begin{array}[]{c}A_{p}\\ \epsilon a_{p}\end{array}\right)=\left(\begin{array}[]{c}W_{p}\sigma(A_{p})- \tilde{L}A_{p}\\ \epsilon\tilde{L}\left(t(p)K_{p,i\cdot}-a_{p}\right)\end{array}\right).$ $$

On the other hand the actual derivative as determined by the new weights:

$$ $\displaystyle\left(\begin{array}[]{cc}W_{p}-\tilde{L}\epsilon^{2}t(p)A_{p, \cdot i}\sigma(A_{p,\cdot i})^{T}&\epsilon\tilde{L}t(p)A_{p,\cdot i}\\ \epsilon\tilde{L}t(p)\sigma(A_{p,\cdot i})&0\end{array}\right)\left(\begin{ array}[]{c}\sigma(A_{p})\\ \epsilon\sigma(a_{p})\end{array}\right)-\tilde{L}\left(\begin{array}[]{c}A_{p} \\ \epsilon a_{p}\end{array}\right)$ $\displaystyle=\left(\begin{array}[]{c}W_{p}\sigma(A_{p})-\tilde{L}A_{p}-\tilde {L}\epsilon^{2}t(p)^{2}A_{p,\cdot i}K_{p,i\cdot}+\tilde{L}\epsilon^{2}t(p)A_{p ,\cdot i}a_{p}\\ \epsilon\tilde{L}t(p)K_{p,i\cdot}-\epsilon\tilde{L}a(p)\end{array}\right).$ $$

The only difference is the two terms

$$ $-\tilde{L}\epsilon^{2}t(p)^{2}A_{p,\cdot i}K_{i\cdot}+\tilde{L}\epsilon^{2}t(p )A_{p,\cdot i}a_{p}=\tilde{L}\epsilon^{2}t(p)A_{p,\cdot i}\left(t(p)K_{i\cdot} -a_{p}\right).$ $$

One can guarantee with a Grönwall type of argument that the representation
path resulting from the new weights must be very close to the path
$\left(\begin{array}[]{c}A_{p}\\
\epsilon a_{p}\end{array}\right)$.
∎

###### Theorem 7 .

###### Proof.

###### Proposition 8 (Proposition 5 in the main.) .

###### Proof.

### A.3 Balancedness

This paper will heavily focus on the Hamiltonian $\mathcal{H}_{p}$
that is constant throughout the layers $p\in[0,1]$, and how it can
be interpreted. Note that the Hamiltonian we introduce is distinct
from an already known invariant, which arises as the result of so-called
balancedness, which we introduce now.

Though this balancedness also appears in ResNets, it is easiest to
understand in fullyconnected networks. First observe that for any
neuron $i\in 1,\dots,w$ at a layer $\ell$ one can multiply the incoming
weights $(W_{\ell,i\cdot},b_{\ell,i})$ by a scalar $\alpha$ and
divide the outcoming weights $W_{\ell+1,\cdot i}$ by the same scalar
$\alpha$ without changing the subsequent layers. One can easily see
that the scaling that minimize the contribution to the parameter norm
is such that the norm of incoming weights equals the norm of the outcoming
weights $\left\|W_{\ell,i\cdot}\right\|^{2}+\left\|b_{\ell,i}\right\|^{2}=\left\|W_{
\ell+1,\cdot i}\right\|^{2}$.
Summing over the $i$s we obtain $\left\|W_{\ell}\right\|_{F}^{2}+\left\|b_{\ell}\right\|^{2}=\left\|W_{\ell+1}
\right\|_{F}^{2}$
and thus $\left\|W_{\ell}\right\|_{F}^{2}=\left\|W_{1}\right\|_{F}^{2}+\sum_{k=1}^{\ell-
1}\left\|b_{k}\right\|_{F}^{2}$,
which means that the norm of the weights is increasing throughout
the layers, and in the absence of bias, it is even constant.

Leaky ResNet exhibit the same symmetry:

###### Proposition 9 .

At any critical $W_{p}$, we have $\left\|W_{p}\right\|^{2}=\left\|W_{0}\right\|^{2}+\tilde{L}\int_{0}^{p}\left\|
W_{p,\cdot w+1}\right\|^{2}dq$.

###### Proof.

This proofs handles the bias $W_{p,\cdot(w+1)}$ differently
to the rest of the weights $W_{p,\cdot(1:w)}$, to simplify notations,
we write $V_{p}=W_{p,\cdot(1:w)}$ and $b_{p}=W_{p,\cdot(w+1)}$ for
the bias.

First let us show that choosing the weight matrices $\tilde{V}_{q}=r^{\prime}(q)V_{r(q)}$
and bias $\tilde{b}_{q}=r^{\prime}(q)e^{\tilde{L}(r(q)-q)}b_{r(q)}$ leads
to the path $\tilde{A}_{q}=e^{\tilde{L}\left(r(q)-q\right)}A_{r(q)}$.
Indeed the path $\tilde{A}_{q}=e^{\tilde{L}\left(r(q)-q\right)}A_{r(q)}$
has the right value when $p=0$ and it then satisfies the right differential
equation:

$$ $\displaystyle\partial_{q}\tilde{A}_{q}$ $\displaystyle=\tilde{L}(r^{\prime}(q)-1)\tilde{A}_{q}+e^{\tilde{L}(r(q)-q)}r^{ \prime}(q)\partial_{p}A_{r(q)}$ $\displaystyle=\tilde{L}(r^{\prime}(q)-1)\tilde{A}_{q}+e^{\tilde{L}(r(q)-q)}r^{ \prime}(q)\left(-\tilde{L}A_{r(q)}+V_{r(q)}\sigma(A_{r(q)})+b_{r(q)}\right)$ $\displaystyle=-\tilde{L}\tilde{Z}_{q}+r^{\prime}(q)A_{r(q)}\sigma\left(\tilde{ Z}_{q}\right)+e^{\tilde{L}(r(q)-q)}r^{\prime}(q)b_{r(q)}$ $\displaystyle=\tilde{V}_{q}\sigma\left(\tilde{A}_{q}\right)+\tilde{b}_{q}- \tilde{L}\tilde{A}_{q}$ $$

The optimal reparametrization $r(q)$ is therefore the one that minimizes

$$ $\displaystyle\int_{0}^{1}\left\|\tilde{W}_{q}\right\|^{2}+\left\|\tilde{b}_{q} \right\|^{2}dq$ $\displaystyle=\int_{0}^{1}r^{\prime}(q)^{2}\left(\left\|W_{r(q)}\right\|^{2}+e ^{2\tilde{L}(r(q)-q)}\left\|b_{r(q)}\right\|^{2}\right)dq$ $$

For the identity reparametrization $r(q)=q$ to be optimal, we need

$$ $\int_{0}^{1}2dr^{\prime}(p)\left(\left\|W_{p}\right\|^{2}+\left\|b_{p}\right\| ^{2}\right)+2\tilde{L}dr(p)\left\|b_{p}\right\|^{2}dp=0$ $$

for all $dr(q)$ with $dr(0)=dr(1)=0$. Since

$$ $\int_{0}^{1}dr^{\prime}(p)\left(\left\|W_{p}\right\|^{2}+\left\|b_{p}\right\|^ {2}\right)dp=-\int_{0}^{1}dr(p)\partial_{p}\left(\left\|W_{p}\right\|^{2}+ \left\|b_{p}\right\|^{2}\right)dq,$ $$

we need

$$ $\int_{0}^{1}dr(p)\left[-\partial_{p}\left(\left\|W_{p}\right\|^{2}+\left\|b_{p }\right\|^{2}\right)+\tilde{L}\left\|b_{p}\right\|^{2}\right]dp=0$ $$

and thus for all $p$

$$ $\partial_{p}\left(\left\|W_{p}\right\|^{2}+\left\|b_{p}\right\|^{2}\right)= \tilde{L}\left\|b_{p}\right\|^{2}.$ $$

Integrating, we obtain as needed

$$ $\left\|W_{p}\right\|^{2}+\left\|b_{p}\right\|^{2}=\left\|W_{0}\right\|^{2}+ \left\|b_{0}\right\|^{2}+\tilde{L}\int_{0}^{p}\left\|b_{q}\right\|^{2}dq.$ $$

∎

###### Proposition 9 .

###### Proof.

## Appendix B Experimental Setup

Our experiments make use of synthetic data to train leaky ResNets so that the Bottleneck rank $k^{*}$ is known for our experiments. The synthetic data is generated by teacher networks for a given true rank $k^{*}$. To construct a bottleneck, the teacher network is a composition of networks for which the the inner-dimension is $k^{*}$. Our experiments used an input and output dimension of 30, and a bottleneck of $k^{*}=3$. For data, we sampled a thousand data points for training, and another thousand for testing which are collectively augmented by demeaning and normalization.

To train the leaky ResNets, it is important for them to be wide, usually wider than the input or output dimension, we opted for a width of 100. However, the width of the representation must be constant to implement leaky residual connections, so we introduce a single linear mapping at the start, and another at the end, of the forward pass to project the representations into a higher dimension for the paths. These linear mappings can be either learned or fixed.

To achieve a tight convergence in training, we train primarily using Adam using Mean Squared Error as a loss function, and our custom weight decay function. After training on Adam (we found 5000 epochs to work well), we then train briefly (usually 1000 epochs) using SGD with a smaller learning rate to tighten the convergence.

The bottleneck structure of a trained network, as seen in Figure 1(b) and 2(b), can be observed in the spectra of both the representations $A_{p}$ and the weight matrices $W_{p}$ at each layer. As long as the training is not over-regularized ($\lambda$ too large) then the spectra reveals a clear separation between $k^{*}$ number of large values as the rest decay. In our experiments, $\lambda=\frac{0.001}{\tilde{L}}$ to get good results. To facilitate the formation of the bottleneck structure, $L$ should be large, for our experiments we usually use $L=20$. Figure 2(a) shows how larger $L$, which have better separation between large and small singular values, lead to improved test performance.

As first noted in section 1.3, solving for the Cost Of Identity, the kinetic energy, and the Hamiltonian $\mathcal{H}$ is difficult due to the instability of the pseudo-inverse. Although the relaxation $(K_{p}+\gamma I)$ improves the stability, we also utilize the solve function to avoid computing a pseudo-inverse altogether. The stability of these computations rely on the boundedness of some additional properties: the path length $\int||\partial_{p}A_{p}||\;dp$, as well as the magnitudes of $B_{p}$, and $B_{p}\sigma(A_{p})^{T}$ from the Hamiltonian reformulation. Figure 3 shows how their respective magnitudes remains relatively constant as the effective depth $\tilde{L}$ grows.

For compute resources, these small networks are not particularly resource intensive. Even on a CPU, it only takes a couple minutes to fully train a leaky ResNet.

Figure: Figure 3: Various properties of the Hamiltonian dynamics of Leaky ResNets which remain bounded
Refer to caption: extracted/5624314/figures/bounded_props.png

## NeurIPS Paper Checklist

- 1.
Claims
- Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
- Answer: [Yes]
- Justification: The contribution section accurately describes our contributions, and all theorems/propositions are proven in the main or the appendix.
- Guidelines:
•
The answer NA means that the abstract and introduction do not include the claims made in the paper.
•
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
•
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
•
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
- 2.
Limitations
- Question: Does the paper discuss the limitations of the work performed by the authors?
- Answer: [Yes]
- Justification: We discuss limitations of our results and approach after we state them.
- Guidelines:
•
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
•
The authors are encouraged to create a separate "Limitations" section in their paper.
•
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
•
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
•
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
•
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
•
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
•
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
- 3.
Theory Assumptions and Proofs
- Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
- Answer: [Yes]
- Justification: All assumptions are stated in the Theorem statements.
- Guidelines:
•
The answer NA means that the paper does not include theoretical results.
•
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
•
All assumptions should be clearly stated or referenced in the statement of any theorems.
•
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
•
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
•
Theorems and Lemmas that the proof relies upon should be properly referenced.
- 4.
Experimental Result Reproducibility
- Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
- Answer: [Yes]
- Justification: The experimental setup is described in the Appendix.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
•
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
•
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
•
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a)
If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b)
If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
- 5.
Open access to data and code
- Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
- Answer: [No]
- Justification: We use synthetic data, with a description of how to build this synthetic data. The code is not the main contribution of the paper, so there is little reason to publish it.
- Guidelines:
•
The answer NA means that paper does not include experiments requiring code.
•
Please see the NeurIPS code and data submission guidelines ([https://nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details.
•
While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
•
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ([https://nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details.
•
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
•
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
•
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
•
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
- 6.
Experimental Setting/Details
- Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
- Answer: [Yes]
- Justification: Most details are given in the experimental setup section in the Appendix.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
•
The full details can be provided either with the code, in appendix, or as supplemental material.
- 7.
Experiment Statistical Significance
- Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
- Answer: [No]
- Justification: The numerical experiments are mostly there as a visualization of the theoretical results, our main goal is therefore clarity, which would be hurt by putting error bars everywhere.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
•
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
•
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
•
The assumptions made should be given (e.g., Normally distributed errors).
•
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
•
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
•
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
•
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
- 8.
Experiments Compute Resources
- Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
- Answer: [Yes]
- Justification: In the experimental setup section of the Appendix.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
•
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
•
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
- 9.
Code Of Ethics
- Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics [https://neurips.cc/public/EthicsGuidelines](https://neurips.cc/public/EthicsGuidelines)?
- Answer: [Yes]
- Justification: We have read the Code of Ethics and see no issue.
- Guidelines:
•
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
•
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
•
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
- 10.
Broader Impacts
- Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
- Answer: [N/A]
- Justification: The paper is theoretical in nature, so it has no direct societal impact that can be meaningfully discussed.
- Guidelines:
•
The answer NA means that there is no societal impact of the work performed.
•
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
•
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
•
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
•
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
•
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
- 11.
Safeguards
- Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
- Answer: [N/A]
- Justification: Not relevant to our paper.
- Guidelines:
•
The answer NA means that the paper poses no such risks.
•
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
•
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
•
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
- 12.
Licenses for existing assets
- Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
- Answer: [N/A]
- Justification: We only use our own synthetic data.
- Guidelines:
•
The answer NA means that the paper does not use existing assets.
•
The authors should cite the original paper that produced the code package or dataset.
•
The authors should state which version of the asset is used and, if possible, include a URL.
•
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
•
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
•
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, [paperswithcode.com/datasets](paperswithcode.com/datasets) has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
•
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
•
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
- 13.
New Assets
- Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
- Answer: [N/A]
- Justification: We do not release any new assets.
- Guidelines:
•
The answer NA means that the paper does not release new assets.
•
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
•
The paper should discuss whether and how consent was obtained from people whose asset is used.
•
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
- 14.
Crowdsourcing and Research with Human Subjects
- Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
- Answer: [N/A]
- Justification: Not relevant to this paper.
- Guidelines:
•
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
•
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
•
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
- 15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
- Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
- Answer: [N/A]
- Justification: Not relevant to this paper.
- Guidelines:
•
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
•
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
•
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
•
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Please expand why is it natural to scale $\gamma_p$ with $\|K_p\|$ and how this relates 
2. Please describe the scaling in \tilde L of the quantities in Theorem 4
3. Is the "naive" intuition presented in pt. 4. of the "weaknesses" section valid? If no, why? If yes, is the bound optimal in some sense, i.e., why can we not get rid of that $O(1)$ term?
4. Why can we safely assume that the length $\ell_{\gamma, \tilde L}$ is uniformly bounded in \tilde L if we choose $\gamma = 1/\tilde L$?
5. Can you please expand on how generic are the assumptions made in the main body of section 2?
soundness: 3
strengths: 1. The paper introduces the reader to the main concepts in an incremental and understandable way, relating these quantities in a clear way, and clearly explains the results it presents, clearly stating their limitations. The results are proven rigorously.
2. The paper features the occurrence of the "bottleneck phenomenon" in leaky resnets by elegantly framing the representation geodesics into a Lagrangian and Hamiltonian framework and analyzing the resulting dynamics in the large L limit. While the Hamiltonian formulation was admittedly developed in a previous reference, the definition of a family of networks indexed by the parameter \tilde L and the identification of a separation of timescales in its dynamics is both nontrivial and original.
3. The decomposition of the Lagrangian into a "potential energy" and "kinetic energy" terms appeals to the intuition and allows to identify different regimes in the propagation of representations in the network. Furthermore, the "cost of identity" is clearly related to the potential energy, and the landscape is studiet in some detail.
4. The authors provide numerical experiments that illustrate nicely some of the results presented in the paper.
summary: This paper investigates the feature learning dynamics in Leaky ResNets, a variant of ResNets with tunable skip connections, by examining how the network’s representations evolve under an "effective depth" parameter, \tilde L. The authors identify a bottleneck phenomenon, where which high-dimensional inputs compress into low-dimensional representations before expanding back to high-dimensional outputs. The study cast the depth-dynamics of the network representation in Lagrangian and Hamiltonian form, defining kinetic and potential energy terms to explain the aforementioned bottleneck phenomenon. In particular, the authors introduce the "cost of identity" (COI) as a measure of dimensionality within the network, relating it to the Hamiltonian through a separation of timescales argument in the limit of large \tilde L, identifying a "fast" initial/final phase of compression/decompression and a "slow", low-dimensional phase for the representation geodesics of the network.
weaknesses: 1. The paper could benefit from a revision of the main results' statements. For example, in Theorem 4 "for sequence" could be interpreted as "for A sequence" or "for ANY sequence". It is also unclear if the second display should hold for all $p$'s. Furthermore, section 2.1 feels a bit rushed with respect to the nicely paced exposition in the rest of the paper, especially in the motivation for some statements.
2. Theorem 4 is stated for constant \gamma, but the authors mention that it might be of interest to adapt $\gamma$ as a function of $p$. In this sense, it is unclear what the relationship between the two statements is, and if a similar results as in Theorem 4 should hold when $\gamma$ depends on $p$. 
3. The decomposition of the Hamiltonian at the beginning of Section 2.1 seems to follow from its definition at the end of section 1.5. This decomposition is central to the following analysis, so it would be helpful if the authors could show the main steps of this derivation.
4. It is not always transparent from the statement of Theorem 4 and the subsequent discussion what the scaling of the various quantities are in $\tilde L$, so it is not always clear if the bounds that the theorem presents are tight. For instance, one may naively expect that the term $\ell_{\gamma,\tilde L}$ might be of order $1/\gamma$, inheriting from the (worst-case) scaling of the norm in the integral. Choosing $\gamma = 1/\tilde L$ would result in term on the LHS of the first expression in Theorem 4 that is $O(1)$ (killing the term \gamma c). 
5. It is not clear how much of the assumptions made in the discussion contained in Section 2 are actually respected in practical scenarios. The simulations seem to corroborate the results presented in the main theorems, but it is difficult to assess how generic this behavior is in practice.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could you further clarify the relationship between the Leaky ReLU activation and the emergence of the bottleneck structure?  
2. In the context of practical applications, what are the potential advantages of explicitly considering the bottleneck structure and the proposed training method? Are there specific domains or tasks where this could have a more significant impact?
soundness: 2
strengths: 1. **Theoretical novelty**: The paper offers a novel theoretical perspective on understanding the behavior of neural networks, specifically the bottleneck structure, through the lens of Hamiltonian mechanics. This provides a new framework for analyzing and potentially optimizing network architectures.
2. **Clear explanations**: The concepts of cost of identity, kinetic energy, and potential energy are well-defined and effectively related to the network's dynamics. This helps in building an intuitive understanding of the forces at play during feature learning.
summary: This paper focuses on Leaky ResNets, which interpolate between ResNets and FCNNs. It reformulates the optimization of Leaky ResNets into Lagrangian and Hamiltonian forms, highlighting the significance of kinetic and potential energy terms related to the cost of identity. The work provides an explanation for the emergence of the bottleneck structure in ResNets and proposes an adaptive layer step-size training approach. However, the connection between the analysis of the bottleneck structure and the Leaky ReLU network could be more explicit, and the practical implications of the proposed method are not entirely clear.
weaknesses: **Limited empirical evaluation**: The experimental setup is relatively simplistic and based on synthetic data. The impact of the proposed method appears to be marginal as indicated by the figures, raising questions about its practical significance and generalizability to more complex real-world scenarios.
2. **Connection to Leaky ReLU network**: The link between the in-depth analysis of the bottleneck structure and the Leaky ReLU network is not clearly elucidated. It is not evident how the unique characteristics of the Leaky ReLU activation function contribute to or interact with the observed phenomena. 
3. **Lack of practical problem focus**: It is unclear what practical problem the paper is primarily addressing. While the exploration of the bottleneck structure is interesting theoretically, the practical benefits and applications of enhancing or understanding this behavior are not well-articulated.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * See the Weakness part of the review.
* Could you explain the derivation (the backward equation and the joint dynamics) at the top of page 7?
soundness: 3
strengths: The observations and the interpretation of the COI and kinetic energy are interesting. In particular, by leveraging the 
conservation of the Hamiltonian, the authors uncover a trade-off between a large intermediate dimension and a 
rapidly evolving intermediate representation. This trade-off appears fundamental and could potentially lead to 
new algorithm (such as the discretization scheme the authors provide in Section 3) and/or offer new insights into 
the inner feature representation of Leaky ResNets.
summary: In this paper, the authors study the evolution of the feature representation $(A_p)_p$ across layers of an infinite-depth 
Leaky ResNet. First, they show that the $l^2$-regularization term in the loss function can be approximately decomposed into
two terms --- Cost of Identity (COI) and kinetic energy --- which favor low-dimensional $A_p$ and $A_p$ with a small 
layer derivative $\partial_p A_p$, respectively. By analyzing the interplay of these two quantities and the corresponding 
Hamiltonian, the authors show that when the effective depth is large, the behavior of $(A_p)_p$ can be classified into 
two types: when the COI/rank of $A_p$ is small (resp. large), $A_p$ will change quickly (resp. slowly). Finally, they 
propose a discretization scheme for infinite-depth Leaky ResNet based on this observation.
weaknesses: * This paper seems to be a collection of observations based on the intuitions surrounding the COI and kinetic energy. 
  It is unclear whether these observations can lead to meaningful implications for any theoretical or empirical problem. 
  In addition, the formal results in this paper require additional assumptions and/or certain approximations and do 
  not precisely match the intuitive arguments. 
  It would be helpful if the authors could at least provide a toy example to illustrate the validity of the intuitions 
  and the usefulness of the results in this paper. 
* The argument and results mainly focus on the $l^2$-regularization term and do consider the regression part of the loss. 
  It is unclear if the observations in this paper would still hold when the network needs to actually 
  use the representations to fit the target function. 
* The structure of the paper could be improved. It might be good to gather the main results in one place and 
  explain their relationship, rather than having them scattered across different sections, separated by more technical 
  results.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Why does the limiting representation have to be an integer?
2. Can we derive the Hamiltonian directly from the Lagrangian in the normal way?
3. Are the introduction of the action and two Hamiltonians novel (in that they either haven't appeared in this way before, or that the derivation is particularly transparent, or in some other way)? 
4. Why are the behaviors of the representations at the endpoints unimportant when the representations are not necessarily fixed at the endpoints as $A_1$ is a dynamical quantity?
5. What is the "physical" interpretation of a negative (unbounded from below) potential energy?
6. Is there a standard way to solve the equations of notion at 331 given that we have boundary conditions for $A$ at 0 and $B$ at 1?
soundness: 2
strengths: This paper develops the Lagrangian and Hamiltonian perspectives on leaky ResNets in a transparent way and offers important qualitative arguments about their structure. Furthermore the setting they consider isn't so removed from a realistic case -- in particular the data is not treated with undue approximation and results are general.
summary: The authors consider a particular limit of residual networks which they term "Leaky ResNets" with an $L_2$ weight regularization term, which they relate to an effective depth of this network. By a change of variables into the representation space, they rewrite the behavior of the representations into a Lagrangian formulation with certain boundary conditions. They split the Lagrangian into three terms
1. A potential term which they call "cost of identity" (COI) which prefers small representations
2. A kinetic term that prefers slowly changing representations
3. A cross-term that they claim to be unimportant. 

They then show that the COI has some relationship to well-known norms and show that strict local minima might be preferred due to their separation from saddle points. 

At this stage they derive a Hamiltonian which explains both the forward and reverse dynamics of the network. They use the form of $H$ to argue that some layers will correspond to slow and others to fast dynamics, while the dimension of the limiting representations then allow an approximation of the Hamiltonian.

They close with a discussion of adaptive discretization schemes to convert these ODEs into neural networks, along with a student-teacher example showing that adaptive size leads to better test accuracy.
weaknesses: The critical weakness of this paper is its presentation. For one, it is not clear what is novel, particularly in light of the fact that the Hamiltonian and Lagrangian formulations have been discussed before (though the derivation is the most transparent I have seen in my review). Additionally, the idea of the "cost of identity" in general is well known, so more care could be taken to contextualize the background and new contribution. The second impediment to understanding this contribution is the somewhat poor presentation of the work. 

1. Some claims are made without a clear foundation, such as those about the "middle term " on line 171 or the source of cancellations in the Lagrangian on line 207.
2. The relationship between the action at line 161 and the Hamiltonian at line 335 and the other at line 372 is unclear. 
3. The discussion in the final three pages of the "main contributions" (line 342) is incomplete. For example, it's unclear why better discretization is better and in what cases we may expect this kind of adaptive re-discretization to be helpful.