SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For any notion of rank,  $\operatorname{Rank}(id; \Omega)$  defines a notion of dimensionality of  $\Omega$ . The Jacobian rank  $\operatorname{Rank}_J(id; \Omega) = \max_{x \in \Omega} \dim T_x \Omega$  is the maximum tangent space dimension, while the Bottleneck rank  $\operatorname{Rank}_{BN}(id; \Omega)$  is the smallest dimension  $\Omega$  can be embedded into. For example, the circle  $\Omega = \mathbb{S}^{2-1}$  has  $\operatorname{Rank}_J(id; \Omega) = 1$  and  $\operatorname{Rank}_{BN}(id; \Omega) = 2$ .
On a domain  $\Omega$  where the two notions of dimensionality match  $\mathrm{Rank}_J(id;\Omega) = \mathrm{Rank}_{BN}(id;\Omega) = k$ , the first correction  $R^{(1)}(id;\Omega)$  is non-negative since for any  $x$  with  $\dim T_x\Omega = k$ , we have  $R^{(1)}(id;\Omega) \geq \log |P_{T_x}|_+ = 0$ . The  $R^{(1)}(id;\Omega)$  value measures how non-planar the domain  $\Omega$  is, being 0 only if  $\Omega$  is  $k$ -planar, i.e. its linear span is  $k$ -dimensional:
Proposition 5. For a domain with  $\mathrm{Rank}_J(id;\Omega) = \mathrm{Rank}_{BN}(id;\Omega) = k$ , then  $R^{(1)}(id;\Omega) = 0$  if and only if  $\Omega$  is  $k$ -planar and completely positive.
This proposition shows that the  $R^{(1)}$  term does not only bound the Jacobian of  $f$  as shown in Theorem 3, but also captures properties of the curvature of the domain/function.
Thus at ratios  $p$  where the representation geodesics converge continuously, the representations  $\tilde{\alpha}_p(\Omega)$  are  $k = R^{(0)}(f;\Omega)$ -planar, proving the Bottleneck structure that was only observed empirically in [Jac23]. But the assumption of convergence over which we build this argument does not hold in general, actually we give in the appendix an example of a simple function  $f$  whose optimal representations diverges in the infinite depth limit. This is in stark contrast to the linear case, where the second correction  $R^{(2)}(A) = \frac{1}{2}\left\| \log_{+}A^{T}A\right\|^{2}$  guarantees convergence, since it bounds the operator norm of  $A$ . To prove and describe the bottleneck structure in nonlinear DNNs, we therefore need to turn to another strategy.
4 Bottleneck Structure in Large Depth Networks
Up to now we have focused on one aspect of the Bottleneck structure observed in [Jac23]: that the representations  $\alpha_{\ell}(X)$  inside the Bottleneck are approximately  $k$ -planar. But another characteristic of this phenomenon is that the weight matrices  $W_{\ell}$  in the bottleneck have  $k$  dominating singular values, all close to 1. This property does not require the convergence of the geodesics and can be proven with finite depth rates:
Theorem 6. Given parameters  $\theta$  of a depth  $L$  network, with  $\| \theta \| ^2\leq kL + c_1$  and a point  $x$  such that  $\mathrm{Rank}Jf_{\theta}(x) = k$ , then there are  $w_{\ell}\times k$  (semi-)orthonormal  $V_{\ell}$  such that  $\sum_{\ell = 1}^{L}\left\| W_{\ell} - V_{\ell}V_{\ell -1}^{T}\right\|_{F}^{2}\leq c_{1} - 2\log |Jf_{\theta}(x)|_{+}$  thus for any  $p\in (0,1)$  there are at least  $(1 - p)L$  layers  $\ell$  with
$$
\left\| W _ {\ell} - V _ {\ell} V _ {\ell - 1} ^ {T} \right\| _ {F} ^ {2} \leq \frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L}.
$$
Note how we not only obtain finite depth rates, but our result has the advantage of being applicable to any parameters with a sufficiently small parameter norm (close to the minimal norm solution). The bound is tighter at optimal parameters in which case  $c_{1} = R^{(1)}(f_{\theta})$ , but the theorem shows that the Bottleneck structure generalizes to points that are only almost optimal.
To prove that the pre-activations  $\tilde{\alpha}_{\ell}(X)$  are approximately  $k$ -dimensional for some dataset  $X$  (that may or may not be the training set) we simply need to show that the activations  $\alpha_{\ell -1}(X)$  do not diverge, since  $\tilde{\alpha}_{\ell}(X) = W_{\ell}\alpha_{\ell -1}(X) + b_{\ell}$  (and one can show that the bias will be small at almost every layer too). By our counterexample we know that we cannot rule out such explosion in general, however if we assume that the NTK [JGH18]  $\Theta^{(L)}(x,x)$  is of order  $O(L)$ , then we can guarantee to convergence of the activations  $\alpha_{\ell -1}(X)$  at almost every layer:
Theorem 7. Given balanced parameters  $\theta$  of a depth  $L$  network, with  $\| \theta \| ^2\leq kL + c_1$  and a point  $x$  such that  $\mathrm{Rank}Jf_{\theta}(x) = k$  then if  $\operatorname {Tr}\left[\Theta^{(L)}(x,x)\right]\leq cL$  , then  $\sum_{\ell = 1}^{L}\| \alpha_{\ell -1}(x)\| _2^2\leq \frac{c\max\{1,e^{\frac{c_1}{k}}\}}{k|Jf_{\theta}(x)|_+^{2 / k}} L$  and thus for all  $p\in (0,1)$  there are at least  $(1 - p)L$  layers such that
$$
\left\| \alpha_ {\ell - 1} (x) \right\| _ {2} ^ {2} \leq \frac {1}{p} \frac {c \operatorname* {m a x} \left\{1 , e ^ {\frac {c _ {1}}{k}} \right\}}{k \left| J f _ {\theta} (x) \right| _ {+} ^ {2 / k}}.
$$
Note that the balancedness assumption is not strictly necessary and could easily be weakened to some form of approximate balancedness, since we only require the fact that the parameter norm  $\| W_{\ell}\|_{F}^{2}$  is well spread out throughout the layers, which follows from balancedness.
The NTK describes the narrowness of the minima [JGH20], and the assumption of bounded NTK is thus related to stability under large learning rates. There are multiple notions of narrowness that have been considered:
- The operator norm of the Hessian  $H$  (which is closely related to the top eigenvalue of the NTK Gram matrix  $\Theta^{(L)}(X,X)$  especially in the MSE case where at any interpolating function  $\| H\|_{op} = \frac{1}{N}\left\| \Theta^{(L)}(X,X)\right\|_{op}$ ) which needs to be bounded by  $2 / \eta$  to have convergence when training with gradient descent with learning rate  $\eta$ .
- The trace of the Hessian (in the MSE case  $\mathrm{Tr}H = \frac{1}{N}\mathrm{Tr}\Theta^{(L)}(X,X)$ ) which has been shown to describe the bias of stochastic gradient descent or approximation thereof [DML21, LWA21].
Thus boundedness of almost all activations as  $L \to \infty$  can be guaranteed by assuming either  $\frac{1}{N} \left\| \Theta^{(L)}(X, X) \right\|_{op} \leq cL$  (which implies  $d_{out} \operatorname{Tr} \Theta^{(L)}(X, X) \leq cL$ ) or  $\frac{1}{N} \operatorname{Tr} \Theta^{(L)}(X, X) \leq cL$  directly, corresponding to either gradient descent with  $\eta = 2 / cL$  or stochastic gradient descent with a similar scaling of  $\eta$ ).
Note that one can find parameters that learn a function with a NTK that scales linearly in depth, but it is not possible to represent non-trivial functions with a smaller  $\mathrm{NTK}\Theta^{(L)}\ll L$ . This is why we consider a linear scaling in depth to be the 'normal' size of the NTK, and anything larger to be narrow.
Putting the two Theorems together, we can prove the Bottleneck structure for almost all representations  $\tilde{\alpha}_{\ell}(X)$ :
Corollary 8. Given balanced parameters  $\theta$  of a depth  $L$  network with  $\| \theta \| ^2\leq kL + c_1$  and a set of points  $x_{1},\ldots ,x_{N}$  such that  $\mathrm{Rank}Jf_{\theta}(x_i) = k$  and  $\frac{1}{N}\mathrm{Tr}\left[\Theta^{(L)}(X,X)\right]\leq cL$ , then for all  $p\in (0,1)$  there are at least  $(1 - p)L$  layers such that
$$
s _ {k + 1} \left(\frac {1}{\sqrt {N}} \tilde {\alpha} _ {\ell} (X)\right) \leq \sqrt {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}} \left(\sqrt {\frac {1}{N} \sum_ {i = 1} ^ {N} \frac {c \max \{1 , e ^ {\frac {c _ {1}}{k}} \}}{k | J f _ {\theta} (x _ {i}) | _ {+} ^ {2 / k}}} + \sqrt {p}\right) \frac {1}{p \sqrt {L}}.
$$
4.1 Narrowness of rank-underestimating minima
We know that the large  $R^{(1)}$  value of BN-rank 1 fitting functions is related to the explosion of its derivative, but a large Jacobian also leads to a blow up of the NTK:
Proposition 9. For any point  $x$ , we have
$$
\left\| \partial_ {x y} ^ {2} \Theta (x, x) \right\| _ {o p} \geq 2 L \| J f _ {\theta} (x) \| _ {o p} ^ {2 - 2 / L}
$$
where  $\partial_{xy}^2\Theta (x,x)$  is understood as a  $d_{in}d_{out}\times d_{in}d_{out}$  matrix.
Furthermore, for any two points  $x, y$  such that the pre-activations of all neurons of the network remain constant on the segment  $[x, y]$ , then either  $\| \Theta(x, x) \|_{op}$  or  $\| \Theta(y, y) \|_{op}$  is lower bounded by
$$
\frac {L}{4} \left\| x - y \right\| ^ {2} \left\| J f _ {\theta} (x) \frac {y - x}{\| x - y \|} \right\| _ {2} ^ {2 - 2 / L}.
$$
(a) Spectra of the  $W_{\ell}\mathrm{s}$
(b) PCA of  $\alpha_{6}(X)$
(c) Dist.  $\| \alpha_{2}(x) - \alpha_{2}(x_{0})\|$
Figure 2: A depth  $L = 25$  network with a width of 200 trained on the task described in Section 5 with a ridge  $\lambda = 0.0002$ . (a) Singular values of the weight matrices of the network, showing two outliers in the bottleneck, which implies that the network has recovered the true rank of 2. (b) Hidden representation of the 6-th layer projected to the first two dimensions, we see how images of GD paths do not cross in this space, showing that the dynamics on these two dimensions are self-consistent. (c) The distance  $\| \alpha_{2}(x_{0}) - \alpha_{2}(x) \|$  in the second hidden layer between the representations at a fixed point  $x_{0}$  (at the white pixel) and another point  $x$  on a plane orthogonal to the axis  $w$  of rotation, we see that all points on the same symmetry orbit are collapsed together, proving that the network has learned the rotation symmetry.
With some additional work, we can show the TTK of such rank-underestimating functions will blow up, suggesting a narrow minimum:
Theorem 10. Let  $f^{*}:\Omega \to \mathbb{R}^{d_{out}}$  be a function with Jacobian rank  $k^{*} > 1$  (i.e. there is a  $x\in \Omega$  with  $\mathrm{Rank}Jf^{*}(x) = k^{*}$ ), then with high probability over the sampling of a training set  $x_{1},\ldots ,x_{N}$  (sampled from a distribution with support  $\Omega$ ), we have that for any parameters  $\theta$  of a deep enough network that represent a BN-rank 1 function  $f_{\theta}$  that fits the training set  $f_{\theta}(x_i) = f^{*}(x_i)$  with norm  $\| \theta \| ^2 = L + c_1$  then there is a point  $x\in \Omega$  where the NTK satisfies
$$
\operatorname {T r} \left[ \Theta^ {(L)} (x, x) \right] \geq c ^ {\prime \prime} L e ^ {- c _ {1}} N ^ {4 - \frac {4}{k ^ {*}}}.
$$
One the one hand, we know that  $c_{1}$  must satisfy  $c_{1} \geq R^{(1)}(f_{\theta}) \geq 2(1 - \frac{1}{k^{*}}) \log N$  but if  $c_{1}$  is within a factor of 2 of this lower bound  $c_{1} < 4(1 - \frac{1}{k^{*}}) \log N$ , then the above shows that the NTK will blow up a rate  $N^{\alpha} L$  for a positive  $\alpha$ .
The previous explanation for why GD avoids BN-rank 1 fitting functions was that when  $N$  is much larger than the depth  $L$  (exponentially larger), there is a rank-recovering function with a lower parameter norm than any rank-underestimating functions. But this relies on the assumption that GD converges to the lower norm minima, and it is only true for sufficiently small depths. In contrast the narrowness argument applies for any large enough depth and does not assume global convergence.
Of course the complete explanation might be a mix of these two reasons and possibly some other phenomenon too. Proving why GD avoids minima that underestimate the rank with a rank  $1 < k < k^{*}$  also remains an open question.
5 Numerical Experiment: Symmetry Learning
In general, functions with a lot of symmetries have low BN-rank since a function  $f$  with symmetry group  $G$  can be decomposed as mapping the inputs  $\Omega$  to the inputs module symmetries  $\Omega / G$  and then mapping it to the outputs, thus  $\operatorname{Rank}_{BN}(f; \Omega) \leq \dim \Omega / G$  where  $\dim \Omega / G$  is the smallest dimension  $\Omega / G$  can be embedded into. Thus the bias of DNNs to learn function with a low BN-rank can be interpreted as the network learning symmetries of the task. With this interpretation, overestimating the rank corresponds to failing to learn all symmetries of the task, while underestimating the rank can be interpreted as the network learning spurious symmetries that are not actual symmetries of the task.
To test this idea, we train a network to predict high dimensional dynamics with high dimensional symmetries. Consider the loss  $C(v) = \left\| vv^T - (ww^T + E) \right\|_F^2$  where  $w \in \mathbb{R}^d$  is a fixed unit vector and  $E$  is a small noise  $d \times d$  matrix. We optimize  $v$  with gradient descent to try and fit the true vector  $w$  (up to a sign). One can think of these dynamics as learning a shallow linear network  $vv^T$  with a single hidden neuron. We will train a network to predict the evolution of the cost in time  $C(v(t))$ .
For small noise matrix  $E$ , the GD dynamics of  $v(t)$  are invariant under rotation around the vector  $w$ . As a result, the high-dimensional dynamics of  $v(t)$  can be captured by only two summary statistics  $u(v) = ((w^T v)^2, \|(I - ww^T)v\|^2)$ : the first measures the position along the axis formed by  $w$  and the second the distance to this axis [AGJ22]. The evolution of the summary statistics is (approximately) self-consistent (using the fact that  $\|v\|^2 = (w^T v)^2 + \|(I - ww^T)v\|^2$ ):
$$
\partial_ {t} \left(w ^ {T} v\right) ^ {2} = - 8 \left(\| v \| ^ {2} - 1\right) \left(w ^ {T} v\right) ^ {2} + O (\| E \|)
$$
$$
\partial_ {t} \left\| (I - w w ^ {T}) v \right\| ^ {2} = - 8 \left\| v \right\| ^ {2} \left\| (I - w w ^ {T}) v \right\| ^ {2} + O (\| E \|).
$$
Our goal now is to see whether a DNN can learn these summary statistics, or equivalently learn the underlying rotation symmetry. To test this, we train a network on the following supervised learning problem: given the vector  $v(0)$  at initialization, predict the loss  $(C(v(1)),\ldots ,C(v(T)))$  over the next  $T$  GD steps. For  $E = 0$ , the function  $f^{*}:\mathbb{R}^{d}\to \mathbb{R}^{T}$  that is to be learned has BN-rank 2, since one can first map  $v(0)$  to the corresponding summary statistics  $u(v(0))\in \mathbb{R}^2$ , and then solve the differential equation on the summary statistics  $(u(1),\dots,u(T))$  over the next  $T$  steps, and compute the cost  $C(v) = \| v\| ^4 -2(w^T v)^2 +1 + O(\| E\|)$  from  $u$ .
We observe in Figure 2 that a large depth  $L_{2}$ -regularized network trained on this task learns the rotation symmetry of the task and learns two dimensional hidden representations that are summary statistics (summary statistics are only defined up to bijections, so the learned representation match  $u(v)$  only up to bijection but they can be recognized from the fact that the GF paths do not cross on the 2D representation).
We have computed corrections to the infinite depth description of the representation cost of DNNs given in [Jac23], revealing two regularity  $R^{(1)}, R^{(2)}$  measures that balance against the dominating low rank/dimension bias  $R^{(0)}$ . We have also partially described another regularity inducing bias that results from large learning rates. We argued that these regularity bias play a role in stopping the network from underestimating the 'true' BN-rank of the task (or equivalently overfitting symmetries).
We have also proven the existence of a bottleneck structure in the weight matrices and under the condition of a bounded NTK of the learned representations, where most hidden representations are approximately  $k = R^{(0)}(f_{\theta})$ -dimensional, with only a few high-dimensional representations.
[ABMM18] Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. In International Conference on Learning Representations, 2018.
[AGJ22] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for SGD: Effective dynamics and critical scaling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[Bac17] Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-681, 2017.
[BHH59] Jillian Beardwood, J. H. Halton, and J. M. Hammersley. The shortest path through many points. Mathematical Proceedings of the Cambridge Philosophical Society, 55(4):299-327, 1959.
A Properties of the Corrections
A.1 First Correction
Theorem 11 (Theorem 3 from the main). For all inputs  $x$  where  $\operatorname{Rank} Jf(x) = R^{(0)}(f; \Omega)$ , we have  $R^{(1)}(f) \geq 2\log |Jf(x)|_{+}$ , furthermore:
1. If  $R^{(0)}(f \circ g) = R^{(0)}(f) = R^{(0)}(g)$ , then  $R^{(1)}(f \circ g) \leq R^{(1)}(f) + R^{(1)}(g)$ .
2. If  $R^{(0)}(f + g) = R^{(0)}(f) + R^{(0)}(g)$ , then  $R^{(1)}(f + g) \leq R^{(1)}(f) + R^{(1)}(g)$ .
3. If  $P_{\mathrm{Im}A^T}\Omega$  and  $A\Omega$  are  $k = \mathrm{Rank}A$  dimensional and completely positive (i.e. they can be embedded with an isometric linear map into  $\mathbb{R}_+^m$  for some  $m$ ), then  $R^{(1)}(x \mapsto Ax; \Omega) = 2\log |A|_+$ .
Proof. For the first bound, we remember that  $R(f; \Omega, L) \geq L \| Jf \|_{2/L}^{2/L}$ , therefore
$$
\begin{array}{l} R ^ {(1)} (f; \Omega) = \lim _ {L \to \infty} R (f; \Omega , L) - L R ^ {(0)} (f; \Omega) \\ \geq \lim  _ {L \rightarrow \infty} L \sum_ {i = 1} ^ {\operatorname {R a n k} J f (x)} s _ {i} (J f (x)) ^ {\frac {2}{L}} - 1 \\ \geq \sum_ {i = 1} ^ {\operatorname {R a n k} J f (x)} 2 \log s _ {i} (J f (x)) \\ \end{array}
$$
where we used  $s^{\frac{2}{L}} - 1 = e^{\frac{2}{L}\log s} - 1\geq \frac{2}{L}\log s$
(1) Since  $R(f\circ g;\Omega ,L_1 + L_2)\leq R(f;L_1) + R(g;L_2)$ , we have
$$
\begin{array}{l} R ^ {(1)} (f \circ g; \Omega) = \lim _ {L _ {1} + L _ {2} \to \infty} R (f \circ g; \Omega , L _ {1} + L _ {2}) - (L _ {1} + L _ {2}) R ^ {(0)} (f \circ g; \Omega) \\ \leq \lim  _ {L _ {1} \rightarrow \infty} R (f; \Omega , L _ {1}) - L _ {1} R ^ {(0)} (f; \Omega) + \lim  _ {L _ {2} \rightarrow \infty} R (g; \Omega , L _ {2}) - L _ {2} R ^ {(0)} (f; \Omega) \\ = R ^ {(1)} (f; \Omega) + R ^ {(1)} (g; \Omega). \\ \end{array}
$$
(2) Since  $R(f + g;\Omega ,L)\leq R(f;\Omega ,L) + R(g;\Omega ,L)$ , we have
$$
\begin{array}{l} R ^ {(1)} (f + g; \Omega) = \lim  _ {L \rightarrow \infty} R (f + g; \Omega , L) - L R ^ {(0)} (f + g; \Omega) \\ \leq \lim  _ {L \rightarrow \infty} R (f; \Omega , L) - L R ^ {(0)} (f; \Omega) + \lim  _ {L \rightarrow \infty} R (g; \Omega , L) - L R ^ {(0)} (g; \Omega) \\ = R ^ {(1)} (f; \Omega) + R ^ {(1)} (g; \Omega). \\ \end{array}
$$
(3) By the first bound, we know that  $R^{(1)}(x \mapsto Ax; \Omega) \geq 2\log |A|_+$ , we now need to show  $R^{(1)}(x \mapsto Ax; \Omega) \leq 2\log |A|_+$ . Let us define the set of completely positive representations as the set of bilinear kernels  $K(x,y) = x^T B^T By$  such that  $Bx$  has non-negative entries for all  $x \in \Omega$  (we say that a kernel  $K$  is completely positive over  $\Omega$  if it can be represented in this way for some choice of  $B$ ). The set of completely positive representations is convex, since for  $K(x,y) = x^T B^T By$  and  $\tilde{K}(x,y) = x^T \tilde{B}^T \tilde{B}y$ , we have
$$
\frac {K (x , y) + \tilde {K} (x , y)}{2} = x ^ {T} \left( \begin{array}{c} \frac {1}{\sqrt {2}} B \\ \frac {1}{\sqrt {2}} \tilde {B} \end{array} \right) ^ {T} \left( \begin{array}{c} \frac {1}{\sqrt {2}} B \\ \frac {1}{\sqrt {2}} \tilde {B} \end{array} \right) y.
$$
The conditions that there are  $O_{in}$  and  $O_{out}$  with  $O_{in}^{T}O_{in} = P_{\mathrm{Im}A^{T}}$  and  $O_{out}^{T}O_{out} = P_{\mathrm{Im}A}$  such that  $O_{in}\Omega \in \mathbb{R}_{+}^{k_{1}}$  and  $O_{out}A\Omega \in \mathbb{R}_{+}^{k_{2}}$  is equivalent to saying that the kernels  $K_{in}(x,y) = x^{T}P_{\mathrm{Im}A^{T}}y$  and  $K_{out}(x,y) = x^{T}A^{T}Ax$  are completely positive over  $\Omega$ .
By the convexity of completely positive representations, the interpolation  $K_{p} = pK_{in} + (1 - p)K_{out}$  is completely positive for all  $p\in [0,1]$ . Now choose for all depths  $L$  and all layers  $\ell = 1,\dots ,L - 1$
a matrix  $B_{L,\ell}$  such that  $K_{p = \frac{\ell}{L}}(x,y) = x^T B_{L,\ell}^T B_{L,\ell}y$  and then choose the weights  $W_{\ell}$  of the depth  $L$  network as
$$
W _ {\ell} = B _ {L, \ell} B _ {L, \ell - 1} ^ {+},
$$
using the convention  $B_{L,0} = I_{d_{in}}$  and  $B_{L,L} = I_{out}$ . By induction, we show that for any input  $x \in \Omega$  the activation of the  $\ell$ -th hidden layer is  $B_{L,\ell}x$ . This is true for  $\ell = 1$ , since  $W_{1} = B_{L,1}$  and therefore  $p^{(1)}(x) = B_{L,1}x$  which has positive entries so that  $q^{(1)}(x) = \sigma\left(p^{(1)}(x)\right) = B_{L,1}x$ . Then by induction
$$
p ^ {(\ell)} (x) = W _ {\ell} q ^ {(\ell - 1)} (x) = B _ {L, \ell} B _ {L, \ell - 1} ^ {+} B _ {L, \ell - 1} x = B _ {L, \ell} x,
$$
which has positive entries, so that again  $q^{(\ell)}(x) = \sigma \left(p^{(\ell)}(x)\right) = B_{L,\ell}x$ . In the end, we get  $p^{(L)}(x) = Ax$  as needed.
Let us now compute the Frobenius norms of the weight matrices  $\| W_{\ell}\| _F^2 = \mathrm{Tr}\left[B_{L,\ell}^T B_{L,\ell}\left(B_{L,\ell -1}^T B_{L,\ell -1}\right)^+\right]$  as  $L\to \infty$  , remember that  $B_{L,\ell}^{T}B_{L,\ell} = \frac{\ell}{L} P_{\mathrm{Im}A^{T}} + (1 - \frac{\ell}{L})A^{T}A$  therefore the matrices  $B_{L,\ell}^{T}B_{L,\ell}$  and  $B_{L,\ell -1}^{T}B_{L,\ell -1}$  converge to each other, so that at first order  $B_{L,\ell}^{T}B_{L,\ell}\left(B_{L,\ell -1}^{T}B_{L,\ell -1}\right)^{+}$  converges to  $P_{\mathrm{Im}A^T}$  , so that  $\| W_{\ell}\| _F^2\rightarrow \mathrm{Rank}A$  , so that  $\sum_{\ell = 1}^{L}\| W_{\ell}\| _F^2 -L\mathrm{Rank}A$  converges to a finite value as  $L\to \infty$  . To obtain this finite limit, we need to study approximate the next order
$$
\begin{array}{l} \| W _ {\ell} \| _ {F} ^ {2} - \mathrm {R a n k} A = \sum_ {i = 1} ^ {\mathrm {R a n k} A} 2 \log s _ {i} (W _ {i}) + O (L ^ {- 2}) \\ = \log \left| B _ {L, \ell} ^ {T} B _ {L, \ell} \left(B _ {L, \ell - 1} ^ {T} B _ {L, \ell - 1}\right) ^ {+} \right| _ {+} + O (L ^ {- 2}) \\ = \log \left| B _ {L, \ell} ^ {T} B _ {L, \ell} \right| _ {+} - \log \left| B _ {L, \ell - 1} ^ {T} B _ {L, \ell - 1} \right| _ {+} + O (L ^ {- 2}). \\ \end{array}
$$
But as we sum all these second order terms, they cancel out, and we are left with
$$
\sum_ {\ell = 1} ^ {L} \| W _ {\ell} \| _ {F} ^ {2} - L \operatorname {R a n k} A = 2 \log | A | _ {+} - 2 \log | I _ {\operatorname {I m} A ^ {T}} | _ {+} + O (L ^ {- 1}).
$$
We have therefore build parameters  $\theta$  that represent the function  $x\mapsto Ax$  with parameter norm  $\| \theta \| ^2 = L\mathrm{Rank}A + 2\log |A|_+ + O(L^{-1})$ , which upper bounds the representation cost, thus implying that  $R^{(1)}(x\mapsto Ax;\Omega)\leq 2\log |A|_+$  as needed.
A.2 Identity
Proposition 12 (Proposition 5 from the main). For a domain with  $\operatorname{Rank}_J(id; \Omega) = \operatorname{Rank}_{BN}(id; \Omega) = k$ , then  $R^{(1)}(id; \Omega) = 0$  if and only if  $\Omega$  is  $k$ -planar and completely positive.
Proof. First if  $\Omega$  is completely positive and  $k$ -planar one can represent the identity with a depth  $L$  network of parameter norm  $Lk$ , by taking  $W_{1} = O, W_{\ell} = P_{\mathrm{Im}O}, W_{L} = O^{T}$  where  $O$  is the  $m \times d$  so that  $O\Omega \subset \mathbb{R}_{+}^{m}$  and  $O^{T}O = P_{\mathrm{span}\Omega}$ . Thus  $R^{(1)}(id; \Omega) = 0$  (and all other corrections as well).
We will show that for any two points  $x, y \in \Omega$  with  $k$ -dim tangent spaces, their tangent spaces must match if  $R^{(1)}(id; \Omega) = 0$ .
Let  $A = J\alpha^{(L - 1)}(x)_{|T_x\Omega}$  and  $B = J\alpha^{(L - 1)}(y)_{|T_y\Omega}$  be the be the Jacobian of the last hidden activations restricted to the tangent spaces, we know that
$$
P _ {T _ {x} \Omega} = W _ {L} A
$$
$$
P _ {T _ {y} \Omega} = W _ {L} B
$$
so that given any weight matrix  $W_{L}$  whose image contains  $T_{x}\Omega$  and  $T_{y}\Omega$ , we can write
$$
A = W _ {L} ^ {+} P _ {T _ {x} \Omega}
$$
$$
B = W _ {L} ^ {+} P _ {T _ {y} \Omega}.
$$
Without loss of generality, we may assume that the span of  $T_{x}\Omega$  and  $T_{y}\Omega$  is full output space, and therefore that  $W_{L}W_{L}^{T}$  is invertible.