SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \bar {L} (\hat {\mu}) + \lambda^ {(K)} \mathrm {K L} (\nu , \hat {\mu}) - (\bar {L} (\mu^ {*}) + \lambda^ {(K)} \mathrm {K L} (\nu , \mu^ {*})) \\ \leq L (\hat {\mu}) + \lambda^ {(K)} \mathrm {K L} (\nu , \hat {\mu}) - (L (\mu^ {*}) + \lambda^ {(K)} \mathrm {K L} (\mu^ {*}, \nu)) + (\bar {L} (\hat {\mu}) - L (\hat {\mu}) + L (\mu^ {*}) - \bar {L} (\mu^ {*})) \\ \leq \epsilon^ {*} + (\bar {L} (\hat {\mu}) - L (\hat {\mu}) + L (\mu^ {*}) - \bar {L} (\mu^ {*})). \\ \end{array}
$$
By the optimality of  $\mu^{*}$ , the right hand side can be lower bounded as
$$
\begin{array}{l} \bar {L} (\hat {\mu}) + \lambda^ {(K)} \mathrm {K L} (\nu , \hat {\mu}) - \left(\bar {L} (\mu^ {*}) + \lambda^ {(K)} \mathrm {K L} (\nu , \mu^ {*})\right) \\ \geq \bar {L} (\hat {\mu}) - \bar {L} (\mu^ {*}) - (\hat {\mu} - \mu^ {*}) \frac {\delta \bar {L} (\mu^ {*})}{\delta \mu} + \lambda^ {(K)} \mathrm {K L} (\mu^ {*}, \hat {\mu}). \\ \end{array}
$$
Then, we have  $\mathrm{KL}(\mu^{*},\hat{\mu})\leq (\epsilon^{*} + 2(\bar{R} +1)) / \lambda^{(K)}$ . Hence, the uniform bound on the Rademacher complexity (Lemma 3) with  $M_0 = (\epsilon^* +2(\bar{R} +1)) / \lambda^{(K)}$ , it holds that
$$
\begin{array}{l} \bar {L} (\hat {\mu}) - L (\hat {\mu}) + L \left(\mu^ {*}\right) - \bar {L} \left(\mu^ {*}\right) \\ \leq 2 \left[ \mathrm {R a d} (\mathcal {F} _ {2 \mathrm {K L} (\mu^ {*}, \hat {\mu}) \vee \frac {1}{n}} (\mu^ {*})) + (2 \| f _ {\hat {\mu}} - f _ {\mu^ {*}} \| _ {\infty} \vee n ^ {- 1}) \sqrt {\frac {t + \log \log_ {2} (8 n ^ {2} M _ {0} \bar {R})}{n}} \right], \\ \end{array}
$$
with probability  $1 - \exp (-t)$  for any  $t > 0$ . Moreover, Lemma 1 gives
$$
\operatorname {R a d} \left(\mathcal {F} _ {2 \operatorname {K L} (\hat {\mu}, \mu^ {*}) \vee \frac {1}{n}} \left(\mu^ {*}\right)\right) \leq 2 \bar {R} \sqrt {\frac {2 \operatorname {K L} \left(\mu^ {*} , \hat {\mu}\right) \vee n ^ {- 1}}{n}}.
$$
Therefore, by setting  $t = s\sqrt{n}$  with  $s < 1$ , we obtain that
$$
\begin{array}{l} \bar {L} (\hat {\mu}) - \bar {L} (\mu^ {*}) - (\hat {\mu} - \mu^ {*}) \frac {\delta \bar {L} (\mu^ {*})}{\delta \mu} + \lambda^ {(K)} \mathrm {K L} (\mu^ {*}, \hat {\mu}) \\ \leq 2 \left[ 2 \bar {R} \sqrt {\frac {2 \mathrm {K L} (\mu^ {*} , \hat {\mu}) \lor n ^ {- 1}}{n}} + (2 \| f _ {\hat {\mu}} - f _ {\mu^ {*}} \| _ {\infty} \lor n ^ {- 1}) \sqrt {\frac {s \sqrt {n} + \log \log_ {2} (8 n ^ {2} M _ {0} \bar {R})}{n}} \right] \\ \leq \frac {1}{4} \lambda^ {(K)} \mathrm {K L} (\mu^ {*}, \hat {\mu}) + \frac {3 2 \bar {R} ^ {2}}{n \lambda^ {(K)}} + \frac {4 \bar {R}}{n} \\ + \lambda^ {(K)} \frac {\| f _ {\hat {\mu}} - f _ {\mu^ {*}} \| _ {\infty} ^ {2}}{8 \bar {R} ^ {2}} + \frac {\lambda^ {(K)}}{3 2 \bar {R} ^ {2} n ^ {2}} + \frac {8 \bar {R} ^ {2}}{\lambda^ {(K)}} \frac {s \sqrt {n} + \log \log_ {2} (8 n ^ {2} M _ {0} \bar {R})}{n}, \\ \end{array}
$$
with probability  $1 - \exp (-s\sqrt{n})$ , where a arithmetic-geometric mean relation and an inequality  $(a\vee b)^2\leq a^2 +b^2$ $(a,b\geq 0)$  are used in the last inequality. Here, we know that the Hellinger distance between  $\mu^{*}$  and  $\hat{\mu}$  can be bounded as
$$
2 \int (\sqrt {\mu^ {*}} - \sqrt {\hat {\mu}}) ^ {2} \mathrm {d} x = 2 \mathrm {d} _ {\mathrm {H}} (\mu^ {*}, \hat {\mu}) \leq \mathrm {K L} (\mu^ {*}, \hat {\mu}).
$$
$$
\begin{array}{l} \left\| f _ {\hat {\mu}} - f _ {\mu^ {*}} \right\| _ {\infty} ^ {2} = \left\| \int h _ {x} (\cdot) \left(\hat {\mu} (x) - \mu^ {*} (x)\right) d x \right\| _ {\infty} ^ {2} \\ = \left\| \int h _ {x} (\cdot) \left(\sqrt {\hat {\mu} (x)} + \sqrt {\mu^ {*} (x)}\right) \left(\sqrt {\hat {\mu} (x)} - \sqrt {\mu^ {*} (x)}\right) d x \right\| _ {\infty} ^ {2} \\ \leq \left\| \int h _ {x} (\cdot) ^ {2} \left(\sqrt {\hat {\mu} (x)} + \sqrt {\mu^ {*} (x)}\right) ^ {2} \mathrm {d} x \right\| _ {\infty} \int \left(\sqrt {\hat {\mu} (x)} - \sqrt {\mu^ {*} (x)}\right) ^ {2} \mathrm {d} x \\ \leq \left\| 2 \int h _ {x} (\cdot) ^ {2} \left(\hat {\mu} (x) + \mu^ {*} (x)\right) \mathrm {d} x \right\| _ {\infty} \int \left(\sqrt {\hat {\mu} (x)} - \sqrt {\mu^ {*} (x)}\right) ^ {2} \mathrm {d} x \\ \leq 4 \bar {R} ^ {2} \mathrm {d} _ {\mathrm {H}} \left(\mu^ {*}, \hat {\mu}\right) \leq 2 \bar {R} ^ {2} \mathrm {K L} \left(\mu^ {*}, \hat {\mu}\right). \\ \end{array}
$$
By summarizing the argument above and noticing  $\bar{L} (\hat{\mu}) - \bar{L} (\mu^{*}) - (\hat{\mu} -\mu^{*})\frac{\delta\bar{L}(\mu^{*})}{\delta\mu}\geq 0$  , it holds that
$$
\begin{array}{l} \bar {L} (\hat {\mu}) - \bar {L} (\mu^ {*}) - (\hat {\mu} - \mu^ {*}) \frac {\delta \bar {L} (\mu^ {*})}{\delta \mu} + \frac {1}{2} \lambda^ {(K)} \mathrm {K L} (\mu^ {*}, \hat {\mu}) \\ \leq \frac {1}{2} \lambda^ {(K)} \mathrm {K L} (\mu^ {*}, \hat {\mu}) \\ \leq \frac {3 2 \bar {R} ^ {2}}{n \lambda^ {(K)}} + \frac {4 \bar {R}}{n} + \frac {\lambda^ {(K)}}{3 2 \bar {R} ^ {2} n ^ {2}} + \frac {8 \bar {R} ^ {2}}{\lambda^ {(K)}} \frac {s \sqrt {n} + \log \log_ {2} (8 n ^ {2} M _ {0} \bar {R})}{n}, \\ \end{array}
$$
with probability  $1 - \exp (-s\sqrt{n})$ . Then, letting  $s = \tau \lambda^{(K)}$ , then we have
$$
\begin{array}{l} \left\| f _ {\hat {\mu}} - f _ {\mu^ {*}} \right\| _ {\infty} ^ {2} \\ \leq 4 \bar {R} ^ {2} \left(\frac {3 2 \bar {R} ^ {2}}{n \lambda^ {(K) 2}} + \frac {4 \bar {R}}{n \lambda^ {(K)}} + \frac {1}{3 2 \bar {R} ^ {2} n ^ {2}} + 8 \bar {R} ^ {2} \frac {\tau \lambda^ {(K)} \sqrt {n} + \log \log_ {2} (8 n ^ {2} M _ {0} \bar {R})}{n \lambda^ {(K) 2}}\right) \\ \leq \frac {4 \bar {R} ^ {2}}{n \lambda^ {(K) 2}} \left(3 2 \bar {R} ^ {2} + 4 \bar {R} \lambda^ {(K)} + \frac {\lambda^ {(K) 2}}{3 2 \bar {R} ^ {2} n} + 8 \bar {R} ^ {2} [ \tau \sqrt {n \lambda^ {(K) 2}} + \log \log_ {2} (8 n ^ {2} M _ {0} \bar {R}) ]\right), \\ \end{array}
$$
with probability  $1 - \exp (-\tau \lambda^{(K)}\sqrt{n})$  for any  $\tau >0$ . Hence, if we take  $n$  sufficiently large and  $\tau$  sufficiently small so that the right hand side is smaller than  $c_0^2$ , then we have
$$
P \left(Y f _ {\hat {\mu}} (Z) > 0\right) = 1.
$$
More precisely, if we let
$$
s _ {n} = \frac {n \lambda^ {(K) 2}}{3 2 \bar {R} ^ {4}} \left\{c _ {0} ^ {2} - \frac {4 \bar {R} ^ {2}}{n \lambda^ {(K) 2}} \left[ \lambda^ {(K)} \left(4 \bar {R} + \frac {\lambda^ {(K)}}{3 2 \bar {R} ^ {2} n}\right) + 8 \bar {R} ^ {2} (4 + \log \log_ {2} (8 n ^ {2} M _ {0} \bar {R})) \right] \right\},
$$
and  $s_n > 0$ , then  $P(Yf_{\hat{\mu}}(Z) > 0) = 1$  with probability  $1 - \exp (-s_n)$ .
Let  $\hat{\mu} \coloneqq \mu^{(K)}$  and  $\mu^* \coloneqq \mu_{[\lambda^{(K)}}$ . From the proof of Theorem 1, with probability  $1 - \exp(-t)$ , it holds that
$$
\begin{array}{l} \frac {3 2 \bar {R} ^ {2}}{n \lambda^ {(K)}} + \frac {4 \bar {R}}{n} + \frac {\lambda^ {(K)}}{3 2 \bar {R} ^ {2} n ^ {2}} + \frac {8 \bar {R} ^ {2}}{n \lambda^ {(K)}} \left(t + \log \log_ {2} (8 n ^ {2} M _ {0} \bar {R})\right) \\ \geq \bar {L} (\hat {\mu}) - \bar {L} \left(\mu^ {*}\right) - \left(\hat {\mu} - \mu^ {*}\right) \frac {\delta \bar {L} \left(\mu^ {*}\right)}{\delta \mu} + \frac {1}{2} \lambda^ {(K)} \mathrm {K L} \left(\mu^ {*}, \hat {\mu}\right) \\ \geq \frac {1}{2} \left[ \bar {L} (\hat {\mu}) - \bar {L} \left(\mu^ {*}\right) - \left(\hat {\mu} - \mu^ {*}\right) \frac {\delta \bar {L} \left(\mu^ {*}\right)}{\delta \mu} + \lambda^ {(K)} \mathrm {K L} \left(\mu^ {*}, \hat {\mu}\right) \right] \\ \end{array}
$$
$$
\geq \frac {1}{2} \left[ \bar {L} (\hat {\mu}) - \bar {L} (\mu^ {*}) - (\hat {\mu} - \mu^ {*}) \frac {\delta \bar {L} (\mu^ {*})}{\delta \mu} \right].
$$
Since  $Yf_{\mu^*}(X) \geq c_0$  almost surely, the Markov's inequality yields that
$$
\begin{array}{l} P (Y f _ {\hat {\mu}} <   0) \\ \leq \frac {1}{\ell (0) - (- c _ {0} \ell^ {\prime} (c _ {0}) + \ell (c _ {0}))} \left(\bar {L} (\hat {\mu}) - \bar {L} (\mu^ {*}) - (\hat {\mu} - \mu^ {*}) \frac {\delta \bar {L} (\mu^ {*})}{\delta \mu}\right) \\ = 2 \psi (c _ {0}) \frac {1}{2} \left(\bar {L} (\hat {\mu}) - \bar {L} (\mu^ {*}) - (\hat {\mu} - \mu^ {*}) \frac {\delta \bar {L} (\mu^ {*})}{\delta \mu}\right). \\ \end{array}
$$
This yields the assertion.
In addition to the 2-sparse parity problem, here we give additional experiments for the  $k$ -sparse parity problems with  $k = 3$  and  $k = 4$ . The setting is the same with the main text, except that the width, step size, the number of iterations are modified to  $N = 2,000$ ,  $\eta = 0.2$ , and  $T = 10,000$ , and we take different grids of  $n$  and  $d$ . We ran the experiment 10 times and plotted the mean for each  $n$  and  $d$ . The results are provided in Figure 2.
For  $k = 3$ , which is shown in Figure 2 (a), we can see that there appear phase-transition lines of  $n = \Theta(d^2)$  and  $n = \Theta(d)$ , which correspond to the test accuracy of around  $90\%$  and the test accuracy of around  $70\%$ , respectively. The former almost perfect classification means the Type I convergence in 4.2, where the classification error gets exponentially small with respect to  $n / d^2$ . The latter means the Type II polynomial convergence.
For  $k = 4$ , presented in Figure 2 (b), only the linear phase-transition, that corresponds to the test accuracy of around  $70\%$ , is observed.
Together with Section 5, we conclude that the our theoretical results are experimentally verified for the 2-sparse parity problem as well as higher-order sparse parity problems. The reason why the quadratic line was not observed in  $k = 4$  would be partially because the hidden coefficients and the required number of steps are exponentially large with respect to  $k$ .
(a) Result with  $k = 3$ .
(b) Result with  $k = 4$ .
Figure 2: Test accuracy of two-layer neural network trained by MFLD to learn a  $d$ -dimensional  $k$ -sparse parity problems.