SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now we now that any parameters that represent the identity on  $\Omega$  and has  $A = J\alpha^{(L - 1)}(x)_{|T_x\Omega}$  and  $B = J\alpha^{(L - 1)}(y)_{|T_y\Omega}$  must have parameter norm at least
$$
\left\| W _ {L} \right\| _ {F} ^ {2} + k (L - 1) + \max \left\{2 \log | A | _ {+}, 2 \log | B | _ {+} \right\}.
$$
Subtracting  $kL$  and taking  $L \to \infty$ , we obtain that
$$
R ^ {(1)} (i d; \Omega) \geq \min  _ {W _ {L}} \| W _ {L} \| _ {F} ^ {2} - k + \max  \left\{2 \log \left| W _ {L} ^ {+} P _ {T _ {x} \Omega} \right| _ {+}, 2 \log \left| W _ {L} ^ {+} P _ {T _ {y} \Omega} \right| _ {+} \right\}.
$$
If we optimize  $W_{L}$  only up to scaling (i.e. optimize  $aW_{L}$  over  $a$ ) we see that at the optimum, we always have  $\| W_{L}\|_{F}^{2} = k$ . This allows us to rewrite the optimization as
$$
R ^ {(1)} (i d; \Omega) \geq \min _ {\| W _ {L} \| _ {F} ^ {2} = k,} \max \left\{2 \log \left| W _ {L} ^ {+} P _ {T _ {x} \Omega} \right| _ {+}, 2 \log \left| W _ {L} ^ {+} P _ {T _ {y} \Omega} \right| _ {+} \right\}.
$$
The only way to put the first term inside the maximum to 0 is to have  $W_{L}W_{L}^{T} = P_{T_{x}\Omega}$ , but this leads to an exploding second term if  $P_{T_x\Omega} \neq P_{T_y\Omega}$ .
Under the assumption of  $C$ -uniform Lipschitzness of the representations (that for all  $\ell$ , the functions  $\tilde{\alpha}_{\ell}$  and  $(\alpha_{\ell} \to f_{\theta})$  are  $C$ -Lipschitz), one can show a stronger version of the above:
Proposition 13. For a  $C$ -uniformly Lipschitz sequence of ReLU networks representing the function  $f$ , we have
$$
R ^ {(1)} (f) \geq \log | J f (x) | _ {+} + \log | J f (y) | _ {+} + C ^ {- 2} \| J f _ {\theta} (x) - J f _ {\theta} (y) \| _ {*}.
$$
Proof. The decomposition of the difference
$$
J f _ {\theta} (x) - J f _ {\theta} (y) = \sum_ {\ell = 1} ^ {L - 1} W _ {L} D _ {L - 1} (y) \dots W _ {\ell + 1} \left(D _ {\ell} (x) - D _ {\ell} (y)\right) W _ {\ell} D _ {\ell - 1} (x) \dots D _ {1} (x) W _ {1},
$$
for the  $w_{\ell} \times w_{\ell}$  diagonal matrices  $D_{\ell}(x) = \mathrm{diag}(\hat{\sigma} (\tilde{\alpha}_{\ell}(x)))$  , implies the bound
$$
\begin{array}{l} \| J f _ {\theta} (x) - J f _ {\theta} (y) \| _ {*} \leq \sum_ {\ell = 1} ^ {L - 1} \| W _ {L} D _ {L - 1} (y) \dots D _ {\ell + 1} (y) \| _ {o p} \| W _ {\ell + 1} \left(D _ {\ell} (x) - D _ {\ell} (y)\right) W _ {\ell} \| _ {*} \| D _ {\ell - 1} (x) \dots D _ {1} (x) W _ {1} \| _ {o p} \\ \leq \frac {C ^ {2}}{2} \sum_ {\ell = 1} ^ {L - 1} \left(\| W _ {\ell + 1} (D _ {\ell} (x) - D _ {\ell} (y)) \| _ {F} ^ {2} + \| (D _ {\ell} (x) - D _ {\ell} (y)) W _ {\ell} \| _ {F} ^ {2}\right) \\ \end{array}
$$
since  $\| AB\|_{*}\leq \frac{\|A\|_{F}^{2} + \|B\|_{F}^{2}}{2}$  and  $(D_{\ell}(x) - D_{\ell}(y))^{2} = (D_{\ell}(x) - D_{\ell}(y)).$
Now since
$$
L \| J f _ {\theta} (x) \| _ {2 / L} ^ {2 / L} \leq \sum_ {\ell = 1} ^ {L} \| W _ {\ell} D _ {\ell - 1} (x) \| _ {F} ^ {2}
$$
$$
L \| J f _ {\theta} (x) \| _ {2 / L} ^ {2 / L} \leq \sum_ {\ell = 1} ^ {L} \| D _ {\ell} (x) W _ {\ell} \| _ {F} ^ {2}
$$
with the convention  $D_0(x) = I_{d_{in}}$  and  $D_L(x) = I_{d_{out}}$ . We obtain that
$$
\begin{array}{l} L\| Jf_{\theta}(x)\|_{2 / L}^{2 / L} + L\| Jf_{\theta}(y)\|_{2 / L}^{2 / L}\leq \frac{1}{2}\sum_{\ell = 1}^{L}\| W_{\ell}D_{\ell -1}(x)\|_{F}^{2} + \| W_{\ell}D_{\ell -1}(y)\|_{F}^{2} + \| D_{\ell}(x)W_{\ell}\|_{F}^{2} + \| D_{\ell}(y)W_{\ell}\|_{F}^{2} \\ \leq \sum_ {\ell = 1} ^ {L} 2 \| W _ {\ell} \| _ {F} ^ {2} - \frac {1}{2} \| W _ {\ell} (D _ {\ell - 1} (x) - D _ {\ell - 1} (y)) \| _ {F} ^ {2} - \frac {1}{2} \| (D _ {\ell} (x) - D _ {\ell} (y)) W _ {\ell} \| _ {F} ^ {2}. \\ \end{array}
$$
This implies the bound
$$
\left\| \theta \right\| ^ {2} \geq \frac {L \left\| J f _ {\theta} (x) \right\| _ {2 / L} ^ {2 / L} + L \left\| J f _ {\theta} (y) \right\| _ {2 / L} ^ {2 / L}}{2} + C ^ {- 2} \left\| J f _ {\theta} (x) - J f _ {\theta} (y) \right\| _ {*}
$$
A.3 Second Correction
Proposition 14 (Proposition 4 from the main). If there is a limiting representation as  $L \to 0$  in the optimal representation of  $f$ , then  $R^{(2)}(f) \geq 0$ . Furthermore:
Proof. We start from the inequality
$$
R (f \circ g; \Omega , L _ {f} + L _ {g}) \leq R (f; g (\Omega), L _ {f}) + R (g; \Omega , L _ {g}).
$$
We subtract  $(L_{f} + L_{g})R^{(0)}(f\circ g) + R^{(1)}(f\circ g)$  divide by  $L_{f} + L_{g}$  and take the limit of increasing depths  $L_{f},L_{g}$  with  $\lim_{L_g,L_f\to \infty}\frac{L_f}{L_f + L_g} = p\in (0,1)$  to obtain
$$
R ^ {(2)} (f \circ g; \Omega) \leq \frac {1}{1 - p} R ^ {(2)} (f; g (\Omega)) + \frac {1}{p} R ^ {(2)} (g; \Omega). \tag {1}
$$
If  $K_{p}$  is the limiting representation at a ratio  $p \in (0,1)$ , we have  $R^{(2)}(f;\Omega) = \frac{1}{p} R^{(2)}(K_p;\Omega) + \frac{1}{1 - p} R^{(2)}(K_p \to f; \Omega)$  and  $p$  must minimize the RHS since if it was instead minimized at a different ratio  $p' \neq p$ , one could find a lower norm representation by mapping to  $K_{p}$  in the first  $p' L$  layers and then back to the outputs. Now there are two possibilities, either  $R^{(2)}(K_p;\Omega)$  and  $R^{(2)}(K_p \to f; \Omega)$  are non-negative in which case the minimum is attained at some  $p \in (0,1)$  and  $R^{(2)}(f;\Omega) \geq 0$ , or one or both is negative in which case the above is minimized at  $p \in \{0,1\}$  and  $R^{(2)}(f;\Omega) = -\infty$ . Since we assumed  $p \in (0,1)$ , we are in the first case.
(1) To prove the first property, we optimize the RHS of 1 over all possible choices of  $p$  (and assuming that  $R^{(2)}(f; g(\Omega))$ ,  $R^{(2)}(g; \Omega) \geq 0$ ) we obtain
$$
\sqrt {R ^ {(2)} (f \circ g ; \Omega)} \leq \sqrt {R ^ {(2)} (f ; g (\Omega))} + \sqrt {R ^ {(2)} (g ; \Omega)}.
$$
(2) This follows from the inequality  $R(f + g; \Omega, L) \leq R(f; g(\Omega), L) + R(g; \Omega, L)$  after subtracting the  $R^{(0)}$  and  $R^{(1)}$  terms, dividing by  $L$  and taking  $L \to \infty$ .
(3) If  $A = USV^T$ , one chooses  $W_{\ell} = U_{\ell}S^{\frac{1}{L}}U_{\ell -1}^{T}$  with  $U_0 = V$ ,  $U_{L} = U$  and  $U_{\ell}$  chosen so that  $U_{\ell}S^{\frac{\ell}{L}}V^{T}\Omega \in \mathbb{R}_{+}^{n_{\ell}}$ , choosing large enough widths  $n_{\ell}$ . This choice of representation of  $A$  is optimal, i.e. its parameter norm matches the representation cost  $L\mathrm{Tr}\left[S^{\frac{2}{L}}\right] = L\mathrm{Rank}A + 2\log |A|_{+} + \frac{1}{2L}\left\| \log_+A^TA\right\|^2 + O(L^{-2})$ .
We know that
$$
\lim _ {L \to \infty} R ^ {(1)} (\alpha_ {\ell_ {1}} \to \alpha_ {\ell_ {2}}; \Omega) = R ^ {(1)} (f _ {\theta}; \Omega) \lim _ {L \to \infty} \frac {\ell_ {2} - \ell_ {1}}{L}
$$
$$
\frac {1}{p} R ^ {(2)} (\alpha ; \Omega) + \frac {1}{1 - p} R ^ {(2)} (\alpha \rightarrow f; \Omega) \geq R ^ {(2)} (f; \Omega)
$$
B Bottleneck Structure
This first result shows the existence of a Bottleneck structure on the weight matrices:
Theorem 15 (Theorem 6 from the main). Given parameters  $\theta$  of a depth  $L$  network, with  $\| \theta \| ^2\leq kL + c_1$  and a point  $x$  such that  $\mathrm{Rank}Jf_{\theta}(x) = k$ , then there are  $w_{\ell}\times k$  (semi-)orthonormal  $U_{\ell},V_{\ell}$  such that
$$
\sum_ {\ell = 1} ^ {L} \left\| W _ {\ell} - U _ {\ell} V _ {\ell + 1} ^ {T} \right\| _ {F} ^ {2} + \left\| b _ {\ell} \right\| ^ {2} \leq c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}
$$
thus for any  $p\in (0,1)$  there are at least  $(1 - p)L$  layers  $\ell$  with
$$
\left\| W _ {\ell} - U _ {\ell} V _ {\ell + 1} ^ {T} \right\| _ {F} ^ {2} + \| b _ {\ell} \| ^ {2} \leq \frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L}
$$
Proof. Since
$$
\begin{array}{l} J f _ {\theta} (x) = W _ {L} D _ {L - 1} (x) \dots D _ {1} (x) W _ {1} \\ = W _ {L} P _ {\operatorname {I m} J \alpha_ {L - 1} (x)} D _ {L - 1} (x) \dots P _ {\operatorname {I m} J \alpha_ {1} (x)} D _ {1} (x) W _ {1} \\ \end{array}
$$
If the preimage of  $A$  matches the image of  $B$  then  $|AB|_{+} = |A|_{+}|B|_{+}$ . We therefore have
$$
\begin{array}{l} \log \left| J f _ {\theta} (x) \right| _ {+} = \log \left| W _ {L} P _ {\operatorname {I m} J \alpha_ {L - 1} (x)} \right| _ {+} \\ + \log \left| P _ {\operatorname {I m} J \left(\alpha_ {L - 1} \rightarrow f _ {\theta}\right) (x) ^ {T}} D _ {L - 1} (x) P _ {\operatorname {I m} J \tilde {\alpha} _ {L - 1} (x)} \right| _ {+} + \log \left| P _ {\operatorname {I m} J \left(\tilde {\alpha} _ {L - 1} \rightarrow f _ {\theta}\right) (x) ^ {T}} W _ {L - 1} P _ {\operatorname {I m} J \alpha_ {L - 2} (x)} \right| _ {+} \\ + \dots \\ + \log \left| P _ {\operatorname {I m} J \left(\alpha_ {1} \rightarrow f _ {\theta}\right) (x) ^ {T}} D _ {1} (x) P _ {\operatorname {I m} J \tilde {\alpha} _ {1} (x)} \right| _ {+} + \log \left| P _ {J \left(\tilde {\alpha} _ {1} \rightarrow f _ {\theta}\right) (x) ^ {T}} W _ {1} \right| _ {+} \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {\ell = 1} ^ {L} \left\| W _ {\ell} \right\| _ {F} ^ {2} - k - 2 \log \left| P _ {\operatorname {I m} J (\tilde {\alpha} _ {\ell} \rightarrow f _ {\theta}) (x) ^ {T}} W _ {\ell} P _ {\operatorname {I m} J \alpha_ {\ell - 1} (x)} \right| _ {+} \\ - 2 \log \left| P _ {\operatorname {I m} J \left(\alpha_ {\ell} \rightarrow f _ {\theta}\right) (x) ^ {T}} D _ {\ell} (x) P _ {\operatorname {I m} J \tilde {\alpha} _ {\ell} (x)} \right| \\ = \| \theta \| ^ {2} - k L - 2 \log | J f _ {\theta} (x) | _ {+} \\ \leq c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+} \\ \end{array}
$$
with the convention  $D_L(x) = I_{w_{out}}$ .
Our goal is to show that the LHS is a sum of positive value which sum up to a finite positive value, which will imply that most of the summands must be very small.
First observe that
$$
- 2 \log \left| P _ {\operatorname {I m} J (\alpha_ {\ell} \rightarrow f _ {\theta}) (x) ^ {T}} D _ {\ell} (x) P _ {\operatorname {I m} J \tilde {\alpha} _ {\ell} (x)} \right| \geq k - \left\| P _ {\operatorname {I m} J (\alpha_ {\ell} \rightarrow f _ {\theta}) (x) ^ {T}} D _ {\ell} (x) P _ {\operatorname {I m} J \tilde {\alpha} _ {\ell} (x)} \right\| _ {F} ^ {2}
$$
which is positive since the eigenvalues of  $D_{\ell}(x)$  are  $\leq 1$
To show that the other part of the summands  $\| W_{\ell}\| _F^2 -k - 2\log \left|P_{\mathrm{Im}J(\tilde{\alpha}_\ell \to f_\theta)(x)^T}W_\ell P_{\mathrm{Im}J\alpha_{\ell -1}(x)}\right|_+$  is positive, we give lower bound it. First, it can be rewritten as
$$
\begin{array}{l} \left(\left\| P _ {\operatorname {I m} J (\tilde {\alpha} _ {\ell} \rightarrow f _ {\theta}) (x) ^ {T}} W _ {\ell} P _ {\operatorname {I m} J \alpha_ {\ell - 1} (x)} \right\| _ {F} ^ {2} - k - 2 \log \left| P _ {\operatorname {I m} J (\tilde {\alpha} _ {\ell} \rightarrow f _ {\theta}) (x) ^ {T}} W _ {\ell} P _ {\operatorname {I m} J \alpha_ {\ell - 1} (x)} \right| _ {+}\right) \\ + \left\| W _ {\ell} - P _ {\operatorname {I m} J (\tilde {\alpha} _ {\ell} \rightarrow f _ {\theta}) (x) ^ {T}} W _ {\ell} P _ {\operatorname {I m} J \alpha_ {\ell - 1} (x)} \right\| _ {F} ^ {2} \\ \end{array}
$$
Now for a general matrix  $A$ , we have
$$
\begin{array}{l} \left\| A \right\| _ {F} ^ {2} - \operatorname {R a n k} A - 2 \log | A | _ {+} = \sum_ {i = 1} ^ {\operatorname {R a n k} A} s _ {i} (A) ^ {2} - 1 - 2 \log s _ {i} (A) \\ \geq \sum_ {i = 1} ^ {\operatorname {R a n k} A} s _ {i} (A) ^ {2} - 1 - 2 \left(s _ {i} (A) - 1\right) \\ = \sum_ {i = 1} ^ {\operatorname {R a n k} A} \left(s _ {i} (A) - 1\right) ^ {2} \\ = \left\| A - U V ^ {T} \right\| _ {F} ^ {2} \\ \end{array}
$$
for the SVD decomposition  $A = USV^T$ . We can thus lower bound
$$
\left\| W _ {\ell} \right\| _ {F} ^ {2} - k - 2 \log \left| P _ {\operatorname {I m} J (\tilde {\alpha} _ {\ell} \rightarrow f _ {\theta}) (x) ^ {T}} W _ {\ell} P _ {\operatorname {I m} J \alpha_ {\ell - 1} (x)} \right| _ {+} \geq \left\| W _ {\ell} - U _ {\ell} V _ {\ell} ^ {T} \right\| _ {F} ^ {2}
$$
where  $U_{\ell}S_{\ell}V_{\ell}^{T}$  is the SVD decomposition of  $P_{\mathrm{Im}J(\tilde{\alpha}_{\ell}\rightarrow f_{\theta})(x)^T}W_{\ell}P_{\mathrm{Im}J\alpha_{\ell -1}(x)}$  (which we know has rank  $k$  since it must match the rank of  $Jf_{\theta}(x)$ ).
Since  $\| \theta \| ^2 = \sum_{\ell}\| W_\ell \| _F^2 +\| b_\ell \| ^2\leq kL + c_1$  , we have
$$
\sum_ {\ell = 1} ^ {L} \left\| W _ {\ell} - U _ {\ell} V _ {\ell + 1} ^ {T} \right\| _ {F} ^ {2} + \| b _ {\ell} \| ^ {2} \leq c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}.
$$
And for any  $p\in (0,1)$  there are at most  $pL$  layers  $\ell$  with
$$
\left\| W _ {\ell} - U _ {\ell} V _ {\ell + 1} ^ {T} \right\| _ {F} ^ {2} + \| b _ {\ell} \| ^ {2} \leq \frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L}.
$$
The fact that almost all weight matrices  $W_{\ell}$  are approximately  $k$ -dim would imply that the pre-activations  $\tilde{\alpha}_{\ell}(X) = W_{\ell}\alpha_{\ell -1}(X)$  are  $k$ -dim too under the condition that the activations  $\alpha_{\ell -1}(X)$  do not diverge. Assuming a bounded NTK is sufficient to guarantee that these activations converge at almost every layer:
Theorem 16 (Theorem 7 from the main). Given balanced parameters  $\theta$  of a depth  $L$  network, with  $\| \theta \| ^2\leq kL + c_1$  and a point  $x$  such that  $\mathrm{Rank}Jf_{\theta}(x) = k$  then if  $\frac{1}{N}\mathrm{Tr}\left[\Theta^{(L)}(x,x)\right]\leq cL$ , then  $\sum_{\ell = 1}^{L}\| \alpha_{\ell -1}(x)\| _2^2\leq \frac{c\max\{1,e^{\frac{c_1}{k}}\}}{k|Jf_\theta(x)|_+^{2 / k}} L$  and thus for all  $p\in (0,1)$  there are at least  $(1 - p)L$  layers such that
$$
\| \alpha_ {\ell - 1} (x) \| _ {2} ^ {2} \leq \frac {1}{p} \frac {c \operatorname* {m a x} \left\{1 , e ^ {\frac {c _ {1}}{k}} \right\}}{k \left| J f _ {\theta} (x) \right| _ {+} ^ {2 / k}}.
$$
$$
\operatorname {T r} \left[ \Theta^ {(L)} (x, x) \right] = \sum_ {\ell = 1} ^ {L} \| \alpha_ {\ell - 1} (x) \| _ {2} ^ {2} \| J (\tilde {\alpha} _ {\ell} \to \alpha_ {L}) (x) \| _ {F} ^ {2},
$$
we therefore need to lower bound  $\| J(\tilde{\alpha}_{\ell}\to \alpha_{L})(x)\|_{F}^{2}$  to show that the activations  $\| \alpha_{\ell -1}(x)\| _2^2$  must be bounded at almost every layer.
We will lower bound  $\| J(\tilde{\alpha}_{\ell}\to \alpha_{L})(x)\|_{F}^{2}$  by  $\| J(\tilde{\alpha}_{\ell}\to \alpha_{L})(x)P_{\ell}\|_{F}^{2}$  for  $P_{\ell}$  the orthogonal projection to the image  $\operatorname{Im}J\tilde{\alpha}_{\ell}(x)$ . Note  $J(\tilde{\alpha}_{\ell}\to \alpha_{L})(x)P_{\ell}$  and  $Jf_{\theta}(x)$  have the same rank.
By the arithmetic-geometric mean inequality, we have  $\| A\| _F^2\geq \mathrm{Rank}A|A|_{+}^{2 / k}$ , yielding
$$
\left\| J \left(\tilde {\alpha} _ {\ell} \rightarrow \alpha_ {L}\right) (x) P _ {\ell} \right\| _ {F} ^ {2} \geq k \left| J \left(\tilde {\alpha} _ {\ell} \rightarrow \alpha_ {L}\right) (x) P _ {\ell} \right| _ {+} ^ {2 / k}.
$$
Now the balancedness of the parameters (i.e.  $\| W_{\ell ,i}\| ^2 +b_{\ell ,i}^2 = \| W_{\ell +1,\cdot i}\| ^2$ $\forall \ell ,i)$  implies that the parameter norms are increasing  $\| W_{\ell +1}\| _F^2\geq \| W_\ell \| _F^2$  and thus
$$
\frac {\left\| W _ {\ell + 1} \right\| ^ {2} + \cdots + \left\| W _ {L} \right\| ^ {2}}{L - \ell} \geq \frac {\left\| W _ {1} \right\| ^ {2} + \cdots + \left\| W _ {\ell} \right\| ^ {2}}{\ell}.
$$
$$
\begin{array}{l} \frac {\left\| W _ {1} \right\| ^ {2} + \cdots + \left\| W _ {\ell} \right\| ^ {2}}{\ell} = \frac {\left(\left\| W _ {1} \right\| ^ {2} + \cdots + \left\| W _ {\ell} \right\| ^ {2}\right)}{L} + \frac {L - \ell}{L} \frac {\left(\left\| W _ {1} \right\| ^ {2} + \cdots + \left\| W _ {\ell} \right\| ^ {2}\right)}{\ell} \\ \leq \frac {\| \theta \| ^ {2}}{L} \\ \end{array}
$$
$$
\left| J \tilde {\alpha} _ {\ell} (x) \right| _ {+} ^ {2 / k \ell} \leq \frac {1}{k} \| J \tilde {\alpha} _ {\ell} (x) \| _ {2 / \ell} ^ {2 / \ell} \leq \frac {\left\| W _ {1} \right\| ^ {2} + \cdots + \left\| W _ {\ell} \right\| ^ {2}}{k \ell} \leq \frac {\left\| \theta \right\| ^ {2}}{k L} \leq 1 + \frac {c _ {1}}{k L}
$$
$$
\begin{array}{l} \left\| J \left(\tilde {\alpha} _ {\ell} \rightarrow f _ {\theta}\right) (x) P _ {\ell} \right\| _ {F} ^ {2} \geq k \left| J \left(\tilde {\alpha} _ {\ell} \rightarrow f _ {\theta}\right) (x) P _ {\ell} \right| _ {+} ^ {2 / k} \\ = k \frac {\left| J f _ {\theta} (x) \right| _ {+} ^ {2 / k}}{\left| J \tilde {\alpha} (x) \right| _ {+} ^ {2 / k}} \\ \geq k \frac {\left| J f _ {\theta} (x) \right| _ {+} ^ {2 / k}}{\left(1 + \frac {c _ {1}}{L}\right) ^ {\ell}} \\ \geq k \left| J f _ {\theta} (x) \right| _ {+} ^ {2 / k} e ^ {- \frac {\ell}{L} \frac {c _ {1}}{k}} \\ \geq k \left| J f _ {\theta} (x) \right| _ {+} ^ {2 / k} \min  \{1, e ^ {- \frac {c _ {1}}{k}} \}. \\ \end{array}
$$
$$
\sum_ {\ell = 1} ^ {L} \left\| \alpha_ {\ell - 1} (x) \right\| _ {2} ^ {2} \leq \frac {c \max  \left\{1 , e ^ {\frac {c _ {1}}{k}} \right\}}{k | J f _ {\theta} (x) | _ {+} ^ {2 / k}} L
$$
which implies that there are at most  $pL$  layers  $\ell$  with
$$
\| \alpha_ {\ell - 1} (x) \| _ {2} ^ {2} \geq \frac {1}{p} \frac {c \max  \{1 , e ^ {\frac {c _ {1}}{k}} \}}{k | J f _ {\theta} (x) | _ {+} ^ {2 / k}}.
$$
Note that for the MSE loss in the limit  $\lambda \searrow 0$ , the Hessian at a global minimum (i.e.  $f_{\theta}$  interpolates the training set) equals  $\operatorname{Tr}[\mathcal{H}\mathcal{L}(\theta)] = \frac{1}{N}\operatorname{Tr}\left[\Theta^{(L)}(X,X)\right]$ . If we then assume that the trace of the Hessian is bounded by  $cL$ , we get that  $\operatorname{Tr}\left[\Theta^{(L)}(X,X)\right] \leq cNL$  and thus there are at least  $(1 - p)L$  layers where
$$
\frac {1}{N} \left\| \alpha_ {\ell - 1} (X) \right\| _ {F} ^ {2} \leq \frac {1}{p} \frac {c \max  \left\{1 , e ^ {\frac {c _ {1}}{k}} \right\}}{k | J f _ {\theta} (x) | _ {+} ^ {2 / k}},
$$
thus guaranteeing the infinite depth convergence of training set activations  $\alpha_{\ell -1}(X)$  on those layers.
Putting the two above theorems together, we can prove that the pre-activations are  $k$ -dim at almost every layer:
Corollary 17 (Corollary 8 from the main). Given balanced parameters  $\theta$  of a depth  $L$  network with  $\| \theta \| ^2\leq kL + c_1$  and a set of points  $x_{1},\ldots ,x_{N}$  such that  $\mathrm{Rank}Jf_{\theta}(x_i) = k$  and  $\frac{1}{N}\mathrm{Tr}\left[\Theta^{(L)}(X,X)\right]\leq cL$ , then for all  $p\in (0,1)$  there are at least  $(1 - p)L$  layers such that
$$
s _ {k + 1} \left(\frac {1}{\sqrt {N}} \tilde {\alpha} _ {\ell} (X)\right) \leq \sqrt {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}} \left(\sqrt {\frac {1}{N} \sum_ {i = 1} ^ {N} \frac {c \max \{1 , e ^ {\frac {c _ {1}}{k}} \}}{k | J f _ {\theta} (x _ {i}) | _ {+} ^ {2 / k}}} + \sqrt {p}\right) \frac {1}{p \sqrt {L}},
$$
Proof. Since  $\tilde{\alpha}_{\ell}(X) = W_{\ell}\alpha_{\ell -1}(X) + b_{\ell}\mathbf{1}_N^T$  we know that
$$
s _ {k + 1} \left(\frac {1}{\sqrt {N}} \tilde {\alpha} _ {\ell} (X)\right) \leq s _ {k + 1} (W _ {\ell}) \left\| \frac {1}{\sqrt {N}} \alpha_ {\ell - 1} (X) \right\| _ {o p} + \| b _ {\ell} \|.
$$
By Theorems 15 and 16, there are for any  $p \in (0, \frac{1}{2})$  at least  $(1 - 2p)L$  layers such that
$$
\begin{array}{l} \left\| W _ {\ell} - V _ {\ell} V _ {\ell - 1} ^ {T} \right\| _ {F} ^ {2} + \| b _ {\ell} \| ^ {2} \leq \frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L} \\ \left\| \frac {1}{\sqrt {N}} \alpha_ {\ell - 1} (X) \right\| _ {F} ^ {2} \leq \sum_ {i = 1} ^ {N} \frac {1}{p} \frac {c \max \{1 , e ^ {\frac {c _ {1}}{k}} \}}{k | J f _ {\theta} (x _ {i}) | _ {+} ^ {2 / k}} \\ \end{array}
$$
$$
s _ {k + 1} \left(\frac {1}{\sqrt {N}} \tilde {\alpha} _ {\ell} (X)\right) \leq \sqrt {\frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p}} \left(\sqrt {\frac {1}{p N} \sum_ {i = 1} ^ {N} \frac {c \max \{1 , e ^ {\frac {c _ {1}}{k}} \}}{k | J f _ {\theta} (x _ {i}) | _ {+} ^ {2 / k}}} + 1\right) \frac {1}{\sqrt {L}}.
$$
C Local Minima Stability
In this section we motivate the assumption that the Jacobian  $J\tilde{\alpha}_{\ell}(x)$  is uniformly bounded in operator norm as  $L \to \infty$ . The idea is that solutions with a blowing up Jacobian  $J\tilde{\alpha}_{\ell}(x)$  correspond to very narrow local minima.
The narrowness of a local minimum is related to the Neural Tangent Kernel (or Fisher matrix). We have that
$$
\operatorname {T r} \left[ \Theta^ {(L)} (x, x) \right] = \sum_ {\ell = 1} ^ {L} \| \alpha_ {\ell - 1} (x) \| ^ {2} \| J (\tilde {\alpha} _ {\ell} \rightarrow f _ {\theta}) (x) \| _ {F} ^ {2}.
$$
A large Jacobian  $Jf_{\theta}(x)$  leads to a blow up of the derivative of the NTK:
Proposition 18 (Proposition 9 from the main). For any point  $x$ , we have
$$
\left| \left| \partial_ {x y} ^ {2} \Theta (x, x) \right| \right| _ {o p} \geq 2 L \left. \left| \left| J f _ {\theta} (x) \right| \right| _ {o p} ^ {2 - 2 / L} \right.
$$
Proof. (1) For any point  $x$ , we have
$$
\begin{array}{l} \partial_ {x, y} \left(v ^ {T} \Theta (x, x) v\right) [ u, u ] = \sum_ {\ell = 1} ^ {L} u ^ {T} W _ {1} ^ {T} D _ {1} (x) \dots D _ {\ell - 1} (x) ^ {2} \dots D _ {1} (x) W _ {1} u v ^ {T} W _ {L} D _ {L - 1} (x) \dots D _ {\ell} (x) ^ {2} \dots D _ {L - 1} (x) W _ {L} ^ {T} v \\ = \sum_ {\ell = 1} ^ {L} \| D _ {\ell - 1} (x) \dots D _ {1} (x) W _ {1} u \| _ {2} ^ {2} \| D _ {\ell} (x) \dots D _ {L - 1} (x) W _ {L} v \| _ {2} ^ {2}. \\ \end{array}
$$
$$
\begin{array}{l} \left| v ^ {T} J f _ {\theta} (x) u \right| = \left| v ^ {T} W _ {L} D _ {L - 1} (x) \dots D _ {1} (x) W _ {1} u \right| \\ \leq \left\| D _ {\ell} (x) \dots D _ {1} (x) W _ {1} u \right\| _ {2} \left\| D _ {\ell} (x) \dots D _ {L - 1} (x) W _ {L} v \right\| _ {2}, \\ \end{array}
$$
where we used the fact that  $D_{\ell}(x)D_{\ell}(x) = D_{\ell}(x)$ . This applies to the case  $\ell = L$  and  $\ell = 1$  too, using the definition  $D_{L}(x) = I_{d_{out}}$  and  $D_0(x) = I_{d_{in}}$ . This implies
$$
\begin{array}{l} \partial_ {x y} ^ {2} \left(v ^ {T} \Theta (x, x) v\right) [ u, u ] \geq \left| v ^ {T} J f _ {\theta} (x) u \right| ^ {2} \sum_ {\ell = 1} ^ {L} \frac {\left\| D _ {\ell - 1} (x) \cdots D _ {1} (x) W _ {1} u \right\| _ {2} ^ {2}}{\left\| D _ {\ell} (x) \cdots D _ {1} (x) W _ {1} u \right\| _ {2} ^ {2}} \\ \geq \left| v ^ {T} J f _ {\theta} (x) u \right| ^ {2} L \left(\frac {\| u \| _ {2} ^ {2}}{\| W _ {L} D _ {L - 1} (x) \cdots D _ {1} (x) W _ {1} u \| _ {2} ^ {2}}\right) ^ {\frac {1}{L}} \\ \geq L \frac {\left| v ^ {T} J f _ {\theta} (x) u \right| ^ {2}}{\left\| J f _ {\theta} (x) u \right\| _ {2} ^ {2 / L}}. \\ \end{array}
$$
where we used the geometric/arithmetic mean inequality for the second inequality.
If  $u, v$  are right and left singular vectors of  $Jf_{\theta}(x)$  with singular value  $s$ , then the above bound equals  $Ls^{2 - \frac{2}{L}}$ .
(2) Now let us consider a segment  $\gamma(t) = (1 - t)x + ty$  between two points  $x, y$  with no changes of activations on these paths (i.e.  $D_{\ell}(\gamma(t))$  is constant for all  $t \in [0,1]$ ). Defining  $u = \frac{y - x}{\|y - x\|}$  and  $v = \frac{Jf_{\theta}(x)u}{\|Jf_{\theta}(x)u\|}$ , we have
$$
\partial_ {t} v ^ {T} \Theta (\gamma (t), \gamma (t)) v = \| x - y \| \partial_ {x} \left(v ^ {T} \Theta (\gamma (t), \gamma (t)) v\right) [ u ] + \| x - y \| \partial_ {y} \left(v ^ {T} \Theta (\gamma (t), \gamma (t)) v\right) [ u ]
$$
and since  $\partial_{xx}\Theta (\gamma (t),\gamma (t)) = 0$  and  $\partial_{yy}\Theta (\gamma (t),\gamma (t)) = 0$  for all  $t\in [0,1]$  , we have
$$
\partial_ {t} ^ {2} \left(v ^ {T} \Theta (\gamma (t), \gamma (t)) v\right) = 2 \left\| x - y \right\| ^ {2} \partial_ {x y} ^ {2} \left(v ^ {T} \Theta (\gamma (t), \gamma (t)) v\right) [ u, u ] \geq 2 L \left\| x - y \right\| ^ {2} \left\| J f _ {\theta} (x) u \right\| _ {2} ^ {2 - 2 / L}.
$$
Since  $v^T\Theta(\gamma(t), \gamma(t))v \geq 0$  for all  $t \in [0,1]$  then either
$$
v ^ {T} \Theta (x, x) v \geq \frac {L}{4} \| x - y \| ^ {2} \| J f _ {\theta} (x) u \| _ {2} ^ {2 - 2 / L}
$$
$$
v ^ {T} \Theta (y, y) v \geq \frac {L}{4} \left\| x - y \right\| ^ {2} \left\| J f _ {\theta} (x) u \right\| _ {2} ^ {2 - 2 / L}.
$$
Rank-underestimating fitting functions typically feature exploding derivatives, which was used to show in [Jac23] that BN-rank 1 fitting functions must have a  $R^{(1)}$  term that blows up with the number of datapoints  $N$ . With some additional work, we can show that the NTK will blow up at some  $x$ :