SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 19 (Theorem 10 from the main). Let  $f^{*}:\Omega \to \mathbb{R}^{d_{out}}$  be a function with Jacobian rank  $k^{*} > 1$  (i.e. there is a  $x\in \Omega$  with  $\mathrm{Rank}Jf^{*}(x) = k^{*}$ ), then with high probability over the sampling of a training set  $x_{1},\ldots ,x_{N}$  (sampled from a distribution with support  $\Omega$ ), we have that for any parameters  $\theta$  of a deep enough network that represent a BN-rank 1 function  $f_{\theta}$  that fits the training set  $f_{\theta}(x_i) = f^{*}(x_i)$  with norm  $\| \theta \| ^2 = L + c_1$  then there is a point  $x\in \Omega$  where the NTK satisfies
$$
\Theta^ {(L)} (x, x) \geq c ^ {\prime \prime} L e ^ {- c _ {1}} N ^ {4 - \frac {4}{k ^ {*}}}
$$
Proof. For all  $i$  we define  $d_{1,i}$  and  $d_{2,i}$  to be the distance between  $y_{i}$  and its closest and second closest point in  $y_{1},\ldots ,y_{N}$ . Following the argument in [BHH59], the shortest path that goes through all points must be at least  $\sum_{i = 1}^{N}\frac{d_{1,i} + d_{2,i}}{2}$  (which would be tight if it is possible to always jump to the closest or second closest point along the path). Since the expected distances  $d_{1,i}$  and  $d_{2,i}$  are  $N^{-\frac{1}{k^*}}$  since the  $y_{i}$  are sampled from a  $k^*$ -dimensional distribution, the expected length of the shortest path is of order  $N^{1 - \frac{1}{k^*}}$ . Actually most of the distance  $d_{1,i}$  and  $d_{2,i}$  will be of order  $N^{-\frac{1}{k^*}}$  with only a few outliers with larger or smaller distances, thus for any subset of indices  $I\subset [1,\dots ,N]$  that contains a finite ratio of all indices, the sum  $\sum_{i\in I}\frac{d_{1,i} + d_{2,i}}{2}$  will be of order  $N^{1 - \frac{1}{k^*}}$  too.
Following the argument in the proof Theorem 2 from [Jac23], we can reorder the indices so that the segment  $[x_1, x_N]$  will mapped  $f_\theta$  to a path that goes through  $y_1, \ldots, y_N$ . We can therefore define the points  $\tilde{x}_1, \ldots, \tilde{x}_N$  that are preimages of  $y_1, \ldots, y_N$  on the segment.
On the interval  $[\tilde{x}_i, \tilde{x}_{i+1}]$  there must be a point  $x$  with  $\|Jf_{\theta}(x)\|_{op} \geq \frac{\|y_{i+1}-y_i\|}{\|\tilde{x}_{i+1}-\tilde{x}_i\|}$ . Now since  $\sum \| \tilde{x}_{i+1} - \tilde{x}_i \| = \| x_N - x_1 \| \leq \mathrm{diag} \Omega$  there must be at least  $p(N-1)$  intervals  $i$  with  $\| \tilde{x}_{i+1} - \tilde{x}_i \| \leq \frac{\mathrm{diag} \Omega}{(1-p)(N-1)}$ , and amongst those  $i$  they would all satisfy  $\| y_{i+1} - y_i \| \geq c N^{-\frac{1}{k^*}}$  except for a few outliers. Thus we can for example guarantee that there are at least  $\frac{5}{6}(N-1)$  intervals  $[\tilde{x}_i, \tilde{x}_{i+1}]$  that contain a point  $x$  with  $\| Jf_{\theta}(x)\|_{op} \geq c N^{1-\frac{1}{k^*}}$ .
First observe that by Theorem 2 of [Jac23], there must be a point  $x \in \Omega$  such that  $\| Jf_{\theta}(x)\|_{op} \geq N^{1 - \frac{1}{k}}$  thus by Theorem 6, there are at least  $pL$  layers where
$$
\left\| W _ {\ell} - u _ {\ell} v _ {\ell} ^ {T} \right\| ^ {2} \leq \frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L}.
$$
Consider one of those  $pL$  layers  $\ell$ , with activators  $z_{i} = \alpha_{\ell -1}(\tilde{x}_{i})$ . Let  $i_1,\ldots ,i_N$  be the ordering of the indices so that  $u_{\ell}^{T}z_{i_m}$  is increasing in  $m$ . Then the hidden representations must satisfy
$$
\frac {\left\| W _ {\ell} (z _ {i _ {m}} - z _ {i _ {m - 1}}) \right\| + \left\| W _ {\ell} (z _ {i _ {m}} - z _ {i _ {m + 1}}) \right\|}{2} \geq e ^ {- \frac {\left\| \theta^ {(\ell + 1 : L)} \right\| ^ {2} - (L - \ell)}{2}} \frac {d _ {1 , i _ {m}} + d _ {2 , i _ {m}}}{2}
$$
$$
\left\| W _ {\ell} (z _ {i _ {m}} - z _ {i _ {m - 1}}) \right\| \leq u _ {\ell} ^ {T} (z _ {i _ {m}} - z _ {i _ {m - 1}}) + \sqrt {\frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L}} \left\| z _ {i _ {m}} - z _ {i _ {m - 1}} \right\|.
$$
For any two indices  $m_1 < m_2$  separated by  $pN$  indices (where  $p > 0$  remains finite), we have
$$
\begin{array}{l} u _ {\ell} ^ {T} \left(z _ {i _ {m _ {2}}} - z _ {i _ {m _ {1}}}\right) \geq \sum_ {m = m _ {1} + 1} ^ {m _ {2}} \frac {\left\| W _ {\ell} \left(z _ {i _ {m}} - z _ {i _ {m - 1}}\right) \right\| + \left\| W _ {\ell} \left(z _ {i _ {m}} - z _ {i _ {m + 1}}\right) \right\|}{2} \\ - \sqrt {\frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L}} \sum_ {m = m _ {1} + 1} ^ {m _ {2}} \frac {\left\| z _ {i _ {m}} - z _ {i _ {m - 1}} \right\| + \left\| z _ {i _ {m}} - z _ {i _ {m + 1}} \right\|}{2} \\ \geq e ^ {- \frac {\left\| _ {\theta (\ell + 1 : L)} \right\| ^ {2} - (L - \ell)}{2}} \sum_ {m = m _ {1} + 1} ^ {m _ {2}} \frac {d _ {1 , i _ {m}} + d _ {2 , i _ {m}}}{2} \\ - \left(m _ {2} - m _ {1}\right) e ^ {\frac {\left\| \theta^ {(1 : \ell - 1)} \right\| ^ {2} - (\ell - 1)}{2}} \sqrt {\frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L}} \mathrm {d i a m} \Omega \\ \geq \left(m _ {2} - m _ {1}\right) \left[ c e ^ {- \frac {\left\| \theta^ {(\ell + 1 : L)} \right\| ^ {2} - (L - \ell)}{2}} N ^ {- \frac {1}{k ^ {*}}} - e ^ {\frac {\left\| \theta^ {(1 : \ell - 1)} \right\| ^ {2} - (\ell - 1)}{2}} \sqrt {\frac {c _ {1} - 2 \log | J f _ {\theta} (x) | _ {+}}{p L}} \mathrm {d i a m} \Omega \right], \\ \end{array}
$$
where we used the fact that up to a few outliers  $\frac{d_{1,i} + d_{2,i}}{2} = \Omega\left(N^{\frac{1}{k^{*}}}\right)$ .
Thus for  $L \geq \frac{4}{c^2} e^{c_1 + 1} N^{\frac{2}{k^*}} \frac{c_1 - 2\log|Jf_\theta(x)|_+}{p} (\mathrm{diam}\Omega)^2$ , we have  $u_\ell^T (z_{i_{m_2}} - z_{i_{m_1}}) \geq (m_2 - m_1)\frac{c}{2} e^{-\frac{\left\|\theta^{(\ell + 1:L)}\right\|^2 - (L - \ell)}{2}}N^{-\frac{1}{k^*}}$ . which implies that at least half of the activations  $z_i$  have norm larger than  $\frac{c}{8} e^{-\frac{\left\|\theta^{(\ell + 1:L)}\right\|^2 - (L - \ell)}{2}}N^{1 - \frac{1}{k^*}}$ .
This implies that at least one fourth of the indices  $i$  satisfy for at least one fourth of the  $pL$  layers  $\ell$
$$
\left\| \alpha_ {\ell - 1} (\tilde {x} _ {i}) \right\| \geq \frac {c}{8} e ^ {- \frac {\left\| \theta^ {(\ell + 1 : L)} \right\| ^ {2} - (L - \ell)}{2}} N ^ {1 - \frac {1}{k ^ {*}}}.
$$
Now amongst these indices there are at least some such that there is a point  $x$  in the interval  $[\tilde{x}_i, \tilde{x}_{i+1}]$  with  $\| Jf_{\theta}(x)\|_{op} \geq cN^{1 - \frac{1}{k^*}}$ . Since  $x$  is  $O(N^{-1})$ -close to  $\tilde{x}_i$  one can guarantee that  $\| \alpha_{\ell-1}(x)\| \geq c' e^{-\frac{\|\theta^{(\ell+1:L)}\|^2 - (L-\ell)}{2}} N^{1 - \frac{1}{k^*}}$  for some constant  $c'$ .
But the Jacobian  $J(\tilde{\alpha}_{\ell}\to f_{\theta})$  also explodes at  $x$  since
$$
\| J (\tilde {\alpha} _ {\ell} \to f _ {\theta}) (x) \| _ {o p} \geq \frac {\| J f _ {\theta} (x) \| _ {o p}}{\| J \tilde {\alpha} _ {\ell} (x) \| _ {o p}} \geq e ^ {- \frac {\left\| _ {\theta^ {(1 : \ell)}} \right\| ^ {2} - _ {\ell}}{2}} c N ^ {1 - \frac {1}{k ^ {*}}}.
$$
We can now lower bound the NTK
$$
\begin{array}{l} \mathrm {T r} \left[ \Theta^ {(L)} (x, x) \right] = \sum_ {\ell = 1} ^ {L} \| \alpha_ {\ell - 1} (x) \| ^ {2} \| J (\tilde {\alpha} _ {\ell} \to f _ {\theta}) (x _ {1}) \| _ {F} ^ {2} \\ \geq \frac {p L}{4} c ^ {\prime 2} e ^ {- \left\| \theta^ {(\ell + 1: L)} \right\| ^ {2} + (L - \ell)} N ^ {2 - \frac {2}{k ^ {*}}} e ^ {- \left\| \theta^ {(1: \ell)} \right\| ^ {2} + \ell} c ^ {2} N ^ {2 - \frac {2}{k ^ {*}}} \\ = c ^ {\prime \prime} L e ^ {- c _ {1}} N ^ {4 - \frac {4}{k ^ {*}}} \\ \end{array}
$$
This suggests that such points are avoided not only because they have a large  $R^{(1)}$  value, but also (if not mostly) because they lie at the bottom of a very narrow valley.
D Technical Results
D.1 Regularity Counterexample
We gave here an example of a simple function whose optimal representation geodesic does not converge, due to it being not uniformly Lipschitz:
Example 20. The function  $f: \Omega \to \mathbb{R}^3$  with  $\Omega = [0,1]^3$  defined by
$$
f (x, y, z) = \left\{ \begin{array}{l l} (x, y, z) & \text {i f} x \leq y \\ (x, y, z + a (x - y)) & \text {i f} x > y \end{array} \right.
$$
satisfies  $R^{(0)}(f;\Omega) = 3$  and  $R^{(1)}(f;\Omega) = 0$ . The optimal representations of  $f$  are not uniformly Lipschitz as  $L\to \infty$ .
Proof. While we are not able to identify exactly the optimal representation geodesic for the function  $f$ , we will first show that  $R^{(1)}(f;\Omega) = 0$ , and then show that the uniform Lipschitzness of the optimal representations would contradict with Proposition 13.
(1) Since the Jacobian takes two values inside  $\mathbb{R}_+^3$ , either the identity  $I_3$  or  $\left( \begin{array}{ccc}1 & 0 & 0\\ 0 & 1 & 0\\ 1 & -1 & 1 \end{array} \right)$ , we
know by Theorem 11 that  $R^{(1)}(f;\Omega) \geq 2\log |I_3|_+ = 0$ . We therefore only need to construct a sequence of parameters of different depths that represent  $f$  with a squared parameter norm of order  $3L + o(1)$ . For simplicity, we only do this construction for even depths (the odd case can be constructed similarly). We define:
$$
\begin{array}{l} W _ {\ell} = \left( \begin{array}{c c c} e ^ {\epsilon} & 0 & 0 \\ 0 & e ^ {\epsilon} & 0 \\ 0 & 0 & e ^ {- 2 \epsilon} \end{array} \right) \text {f o r} \ell = 1, \ldots , \frac {L}{2} - 1 \\ W _ {\frac {L}{2}} = \left( \begin{array}{c c c} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \sqrt {a} e ^ {- \frac {L - 2}{2} \epsilon} & - \sqrt {a} e ^ {- \frac {L - 2}{2} \epsilon} & 0 \end{array} \right) \\ W _ {\frac {L}{2} + 1} = \left( \begin{array}{c c c c} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & \sqrt {a} e ^ {- (L - 2) \epsilon} \end{array} \right) \\ W _ {\ell} = \left( \begin{array}{c c c} e ^ {- \epsilon} & 0 & 0 \\ 0 & e ^ {- \epsilon} & 0 \\ 0 & 0 & e ^ {2 \epsilon} \end{array} \right) \text {f o r} \ell = \frac {L}{2} + 2, \ldots , L \\ \end{array}
$$
We have for all  $x\in \mathbb{R}_+^3$
$$
\alpha_ {\frac {L}{2} - 1} (x) = \left( \begin{array}{c} e ^ {\frac {L - 2}{2} \epsilon} x _ {1} \\ e ^ {\frac {L - 2}{2} \epsilon} x _ {2} \\ e ^ {- (L - 2) \epsilon} x _ {3} \end{array} \right)
$$
$$
\alpha_ {\frac {L}{2}} (x) = \left( \begin{array}{c} e ^ {\frac {L - 2}{2} \epsilon} x _ {1} \\ e ^ {\frac {L - 2}{2} \epsilon} x _ {2} \\ e ^ {- (L - 2) \epsilon} x _ {3} \\ \sigma (x _ {1} - x _ {2}) \end{array} \right)
$$
$$
\alpha_ {\frac {L}{2} + 1} (x) = \left( \begin{array}{c} e ^ {\frac {L - 2}{2} \epsilon} x _ {1} \\ e ^ {\frac {L - 2}{2} \epsilon} x _ {2} \\ e ^ {- (L - 2) \epsilon} \left(x _ {3} + \sigma (x _ {1} - x _ {2})\right) \end{array} \right)
$$
$$
f _ {\theta} (x) = \left( \begin{array}{c} x _ {1} \\ x _ {2} \\ x _ {3} + \sigma (x _ {1} - x _ {2}) \end{array} \right).
$$
The norm of the parameters is
$$
\begin{array}{l} \frac {L - 2}{2} \left(2 e ^ {2 \epsilon} + e ^ {- 4 \epsilon}\right) + \left(3 + 2 e ^ {- (L - 2) \epsilon}\right) + \left(3 + e ^ {- 2 (L - 2) \epsilon}\right) + \frac {L - 2}{2} \left(2 e ^ {- 2 \epsilon} + e ^ {4 \epsilon}\right) \\ = 3 L + 2 \left(e ^ {2 \epsilon} - 1\right) + \left(e ^ {- 4 \epsilon} - 1\right) + 2 e ^ {- (L - 2) \epsilon} + e ^ {- 2 (L - 2) \epsilon} + 2 \left(e ^ {- 2 \epsilon} - 1\right) + \left(e ^ {4 \epsilon} - 1\right) \\ \end{array}
$$
If we take  $\epsilon = L^{-\gamma}$  for  $\gamma \in (\frac{1}{2}, 1)$ , then the terms  $2e^{-(L - 2)\epsilon}$  and  $e^{-2(L - 2)\epsilon}$  decay exponentially (at a rate of  $e^{L^{1 - \gamma}}$ ), in addition the terms  $2(e^{2\epsilon} - 1) + (e^{-4\epsilon} - 1)$  and  $2(e^{-2\epsilon} - 1) + (e^{4\epsilon} - 1)$  are of order  $L^{-2\gamma}$ . This proves that  $R^{(1)}(f; \Omega) = 0$ .
(2) Let us now assume that the optimal representation of  $f$  is  $C$ -uniform Lipschitz for some constant  $C$ , then by Proposition 13, we have that
$$
R^{(1)}(f;\Omega)\geq \log |I_{3}|_{+} + \log \left|\left( \begin{array}{ccc}1 & 0 & 0\\ 0 & 1 & 0\\ 1 & -1 & 1 \end{array} \right)\right|_{+} + C^{-2}\left\| I_{3} - \left( \begin{array}{ccc}1 & 0 & 0\\ 0 & 1 & 0\\ 1 & -1 & 1 \end{array} \right)\right\|_{*} > 0,
$$
which contradicts with the fact that  $R^{(1)}(f;\Omega) = 0$
D.2 Extension outside FPLFs
Since all functions represented by finite depth and width networks are FPLFs, the representation cost of any such function is infinite. But we can define the representation cost of a function  $f$  that is the limit of a sequence of FPLF as the infimum over all sequences  $f_{i} \to f$  converging of  $\lim_{i \to \infty} R(f_{i}; \Omega)$  (for some choice of convergence type that implies convergence of the Jacobians  $Jf_{i}(x) \to Jf(x)$ ). Note that since the representation cost  $R(f; \Omega)$  is lower semi-continuous, i.e.  $\lim_{f \to f_0} R(f; \Omega) \geq R(f_0; \Omega)$ , this does not change the definition of the representation cost on the space of FPLFs.
E Numerical Experiments
For the first numerical experiment, the data pairs  $(x,y)$  were generated as follows. First we sample a 8-dimensional 'latent vector'  $z$ , from which we define  $x = g(z_{1},\ldots ,z_{8})\in \mathbb{R}^{20}$  and  $y = h(z_{1},z_{2})\in$ $\mathbb{R}^{20}$  for two random functions  $g:\mathbb{R}^8\to \mathbb{R}^{20}$  and  $h:\mathbb{R}^2\to \mathbb{R}^{20}$  given by two shallow networks with random parameters. Assuming that  $g$  is injective (which it is with high probability), the function  $f^{*} = h\circ g^{-1}$  which maps  $x$  to  $y$  has BN-rank 2.