SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Follows from  $\pmb{v}_{\alpha} = \frac{1}{2} (\pmb{w}_{\alpha} - \tilde{\pmb{w}}_{\alpha})$ . Recall that  $\pmb{w}_{\alpha}(t) > 0$  and notice that  $\tilde{w}_{\alpha}(t) > 0$ . In the first case,  $w_{\alpha,i}(t) < \alpha^{1 + c}$  and  $\tilde{w}_{\alpha,i}(t) > \alpha^{1 - c / 2}$ . In the latter case  $w_{\alpha,i}(t) > \alpha^{1 - c}$  and  $\tilde{w}_{\alpha,i}(t) < \alpha^{1 + c / 2}$ .
D.2 Sign of gradients on coordinates that leave support
Lemma D.4. For any  $k \geq 1$  and  $i \in S_k^c$ , if  $b_i^k \in \{0,2\}$  then we must have  $i \in \operatorname{supp}(\boldsymbol{u}^{k-1}) \setminus \operatorname{supp}(\boldsymbol{u}^k)$ , and we must have  $g_i(\boldsymbol{u}^k) < 0$  if  $b_i^k = 0$  and  $g_i(\boldsymbol{\theta}^k) > 0$  if  $b_i^k = 2$ . In particular,  $\Delta_k(i_k) > 0$  for all  $k$ .
Proof. This is by induction on  $k$  and using the non-degeneracy Assumption 4.3.
D.3 Local lipschitzness and smoothness
We provide several technical lemmas on the local Lipschitzness and smoothness of  $\ell, h$ , and  $\pmb{g}$ .
Lemma D.5. The function  $\ell(\boldsymbol{y}, \cdot)$  is locally Lipschitz and smooth in its second argument: for any  $R > 0$ , there exists  $K_R$  such that for any  $\zeta, \zeta' \in B(0, R)$
$$
\left| \ell (\boldsymbol {y}, \boldsymbol {\zeta}) - \ell (\boldsymbol {y}, \boldsymbol {\zeta} ^ {\prime}) \right| \leq K _ {R} \| \boldsymbol {\zeta} - \boldsymbol {\zeta} ^ {\prime} \|
$$
$$
\left\| D \ell (\boldsymbol {y}, \boldsymbol {\zeta}) - D \ell (\boldsymbol {y}, \boldsymbol {\zeta} ^ {\prime}) \right\| \leq K _ {R} \| \boldsymbol {\zeta} - \boldsymbol {\zeta} ^ {\prime} \|,
$$
almost surely over  $\pmb{y}$ . Here  $D\ell(\pmb{y}, \cdot)^{\top} \in \mathbb{R}^{d_{out}}$  is the derivative in the second argument.
Proof. Since  $\ell$  is continuously twice-differentiable, for each  $\pmb{y} \in \mathbb{R}^{d_y}$ ,  $\pmb{\zeta} \in \mathbb{R}^{d_{out}}$  there is  $K_{\pmb{y},\pmb{\zeta}} < \infty$  such that for all  $\pmb{y} \in B(\pmb{y},1 / K_{\pmb{y},\pmb{\zeta}})$  and  $\zeta' \in B(\zeta,1 / K_{\pmb{y},\pmb{\zeta}})$  we have
$$
\left\| D \ell \left(\boldsymbol {y} ^ {\prime}, \boldsymbol {\zeta} ^ {\prime}\right) \right\| \leq K _ {\boldsymbol {y}, \boldsymbol {\zeta}} \quad \text {a n d} \quad \left\| D ^ {2} \ell \left(\boldsymbol {y} ^ {\prime}, \boldsymbol {\zeta} ^ {\prime}\right) \right\| \leq K _ {\boldsymbol {y}, \boldsymbol {\zeta}},
$$
where  $D\ell$  and  $D^{2}\ell$  denote the first and second derivative in the second argument. So for all such  $\pmb{y}^{\prime}\in B(\pmb {y},1 / K_{\pmb{y},\pmb {\zeta}})$  and  $\zeta^\prime ,\zeta ''\in B(\zeta ,1 / K_{\pmb{y},\pmb {\zeta}})$  we have
$$
| \ell (\boldsymbol {y} ^ {\prime}, \boldsymbol {\zeta} ^ {\prime}) - \ell (\boldsymbol {y} ^ {\prime}, \boldsymbol {\zeta} ^ {\prime \prime}) | \leq K _ {\boldsymbol {y}, \boldsymbol {\zeta}} \| \boldsymbol {\zeta} ^ {\prime} - \boldsymbol {\zeta} ^ {\prime \prime} \| \quad \text {a n d} \quad | D \ell (\boldsymbol {y} ^ {\prime}, \boldsymbol {\zeta} ^ {\prime}) - D \ell (\boldsymbol {y} ^ {\prime}, \boldsymbol {\zeta} ^ {\prime \prime}) | \leq K _ {\boldsymbol {y}, \boldsymbol {\zeta}} \| \boldsymbol {\zeta} ^ {\prime} - \boldsymbol {\zeta} ^ {\prime \prime} \|.
$$
Cover the set  $\{(\pmb {y},\pmb {\zeta}):\| \pmb {y}\| \leq C,\| \pmb {\zeta}\| \leq R\}$  with the balls  $\cup_{\pmb{y}}B(\pmb {y},1 / K_{\pmb{y},\pmb{\zeta}})$ . By compactness, there is a finite subcover  $(\pmb {y}_1,\pmb {\zeta}_1),\dots ,(\pmb {y}_r,\pmb {\zeta}_r)$ , so we can take  $K_{R} = \max_{i\in [r]}K_{\pmb{y}_{i},\pmb{\zeta}_{i}} < \infty$  and the lemma holds since  $\| \pmb {y}\| \leq C$  almost surely by Assumption 2.1.
Lemma D.6. The function  $h(\pmb{x};\cdot)$  is locally bounded, Lipschitz and smooth in its second argument: for any  $R > 0$  there exists  $K_R$  such that for any  $\psi, \psi' \in B(0,R)$ ,
$$
\left\| h (\boldsymbol {x}; \boldsymbol {\psi}) \right\| \leq K _ {R}
$$
$$
\left\| h (\boldsymbol {x}; \boldsymbol {\psi}) - h (\boldsymbol {x}; \boldsymbol {\psi} ^ {\prime}) \right\| \leq K _ {R} \| \boldsymbol {\psi} - \boldsymbol {\psi} ^ {\prime} \|
$$
$$
\left\| D h (\boldsymbol {x}; \psi) - D h (\boldsymbol {x}; \psi^ {\prime}) \right\| \leq K _ {R} \| \psi - \psi^ {\prime} \|,
$$
almost surely over  $\mathbf{x}$ . Here  $Dh(\mathbf{x},\cdot)\in \mathbb{R}^{d_{out}}\times R^p$  is the derivative in the second argument.
Proof. Analogous to proof of Lemma D.5, using continuous twice-differentiability of  $h$  and boundedness of  $\| x\|$ .
Lemma D.7 (Local Lipschitzness of loss and loss derivative). When  $\pmb{\theta} = (\pmb{u},\pmb{v})\in \mathbb{R}^{2p}$  and  $f_{\mathbb{N}\mathbb{N}}(\pmb {x};\pmb {\theta}) = h(\pmb {x};\pmb {u}\odot \pmb {u})$  the following holds for  $\pmb {g}(\pmb {\theta})$  defined in (4). For any  $R > 0$ , there exists  $K_{R} < \infty$  such that for any  $\pmb {\theta},\pmb{\theta}^{\prime}\in B(0,K_{R})$
$$
\left\| \boldsymbol {g} (\boldsymbol {\theta}) - \boldsymbol {g} \left(\boldsymbol {\theta} ^ {\prime}\right) \right\| \leq K _ {R} \left\| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \right\|
$$
$$
\left\| \nabla_ {\boldsymbol {\theta}} \mathcal {L} (\boldsymbol {\theta}) - \nabla_ {R} \mathcal {L} \left(\boldsymbol {\theta} ^ {\prime}\right) \right\| \leq K _ {\boldsymbol {\theta}} \left\| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \right\|
$$
$$
\left| \mathcal {L} (\boldsymbol {\theta}) - \mathcal {L} \left(\boldsymbol {\theta} ^ {\prime}\right) \right| \leq K _ {R} \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \|.
$$
Proof. Let  $\pmb{\theta} = (\pmb{u},\pmb{v}),\pmb{\theta}' = (\pmb{u}',\pmb{v}')$ . This follows immediately from the local Lipschitzness and smoothness of  $h$  and  $\ell$  in Lemmas D.5 and D.6, as well as
$$
\left\| \boldsymbol {g} (\boldsymbol {\theta}) - \boldsymbol {g} \left(\boldsymbol {\theta} ^ {\prime}\right) \right\| = \left\| \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \left[ D h \left(\boldsymbol {x}; \boldsymbol {u} \odot \boldsymbol {v}\right) ^ {\top} D \ell \left(\boldsymbol {y}, h \left(\boldsymbol {x}; \boldsymbol {u} \odot \boldsymbol {v}\right)\right) ^ {\top} - D h \left(\boldsymbol {x}; \boldsymbol {u} ^ {\prime} \odot \boldsymbol {v} ^ {\prime}\right) ^ {\top} D \ell \left(\boldsymbol {y}, h \left(\boldsymbol {x}; \boldsymbol {u} ^ {\prime} \odot \boldsymbol {v} ^ {\prime}\right)\right) ^ {\top} \right] \right\|.
$$