SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For any integer  $J \leq J_{\alpha}$  define  $\bar{\theta}_0^J = \int \theta_0^J(y)\nu(dy)$ , where  $\theta_0^J(\cdot) = \sum_{j=1}^{J} \sum_{k=1}^{2^j d} \theta_{0;j,k}\vartheta_{j,k}(\cdot)$ . Since,  $\theta_0^J(\cdot) \in \mathrm{Im}(\mathcal{C}^{1/2})$  and  $S$  is a symmetric and HS operator, we invoke Theorem 5 in [22], to write
$$
\begin{array}{l} \operatorname {K L} \left(\mathcal {N} \left(\bar {\theta} _ {0} ^ {J}, \mathcal {C} _ {q}\right) \| \mathcal {N} (0, \mathcal {C})\right) = \frac {1}{2} \| \mathcal {C} ^ {- 1 / 2} \bar {\theta} _ {0} ^ {J} \| ^ {2} - \frac {1}{2} \log \det  (I - S) + \frac {1}{2} t r (- S), \\ = \frac {1}{2} \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {2 ^ {j d}} \frac {\theta_ {0 ; k , j} ^ {2}}{\mu_ {j} ^ {2}} - \frac {1}{2} \log \prod_ {j = 1} ^ {J} \prod_ {k = 1} ^ {2 ^ {j d}} (1 - \kappa_ {j} ^ {2}) - \frac {1}{2} \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {2 ^ {j d}} \kappa_ {j} ^ {2} \\ = \frac {1}{2} \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {2 ^ {j d}} \frac {\theta_ {0 ; k , j} ^ {2}}{\mu_ {j} ^ {2}} - \frac {1}{2} \log \prod_ {j = 1} ^ {J} (1 - \kappa_ {j} ^ {2}) ^ {2 ^ {j d}} - \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j d} \kappa_ {j} ^ {2} \\ = \frac {1}{2} \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {2 ^ {j d}} \frac {\theta_ {0 ; k , j} ^ {2}}{\mu_ {j} ^ {2}} - \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j d} \log (1 - \kappa_ {j} ^ {2}) - \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j d} \kappa_ {j} ^ {2}. \\ \end{array}
$$
Now for  $\mu_j 2^{jd/2} = 2^{-ja}$ , and using the definition of Besov norm of  $\theta_0$  denoted as  $\| \theta_0 \|_{\beta, \infty, \infty}^2$ , and denoting  $1 - \kappa_j^2 = \frac{1}{1 + n \epsilon_n^2 \tau_j^2}$ , we have
$$
\begin{array}{l} \mathrm {K L} \left(\mathcal {N} \left(\bar {\theta} _ {0} ^ {J}, \mathcal {C} _ {q}\right) \| \mathcal {N} (0, \mathcal {C})\right) \\ \leq \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j (2 a - 2 \beta + d)} \| \theta_ {0} \| _ {\beta , \infty , \infty} ^ {2} - \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j d} \log (1 - \kappa_ {j} ^ {2}) - \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j d} \kappa_ {j} ^ {2} \\ = \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j (2 a - 2 \beta + d)} \| \theta_ {0} \| _ {\beta , \infty , \infty} ^ {2} - \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j d} \left(\log \left(1 - \kappa_ {j} ^ {2}\right) + \kappa_ {j} ^ {2}\right) \\ = \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j (2 a - 2 \beta + d)} \| \theta_ {0} \| _ {\beta , \infty , \infty} ^ {2} + \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j d} \left(\log \left(1 + n \epsilon_ {n} ^ {2} \tau_ {j} ^ {2}\right) - \frac {n \epsilon_ {n} ^ {2} \tau_ {j} ^ {2}}{1 + n \epsilon_ {n} ^ {2} \tau_ {j} ^ {2}}\right) \\ \leq \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j (2 a - 2 \beta + d)} \| \theta_ {0} \| _ {\beta , \infty , \infty} ^ {2} + \frac {1}{2} \sum_ {j = 1} ^ {J} 2 ^ {j d} \left(n \epsilon_ {n} ^ {2} \tau_ {j} ^ {2}\right), \\ \end{array}
$$
where the last inequality follows from the fact that,  $\log (1 + x) - \frac{x}{1 + x} \leq \frac{x^2}{1 + x} \leq x$  for  $x > 0$ . Substituting  $\tau_j^2 = 2^{-2ja - jd}$ , we have
$$
\begin{array}{l} \frac {1}{n} \mathrm {K L} (\mathcal {N} (\bar {\theta} _ {0} ^ {J}, \mathcal {C} _ {q}) \| \mathcal {N} (0, \mathcal {C})) \leq \frac {1}{2 n} \sum_ {j = 1} ^ {J} 2 ^ {j (2 a - 2 \beta + d)} \| \theta_ {0} \| _ {\beta , \infty , \infty} ^ {2} + \frac {\epsilon_ {n} ^ {2}}{2} \sum_ {j = 1} ^ {J} 2 ^ {- 2 j a} \\ \leq \frac {\| \theta_ {0} \| _ {\beta , \infty , \infty} ^ {2}}{2 n} \sum_ {j = 1} ^ {J} 2 ^ {j (2 a - 2 \beta + d)} + \frac {2 ^ {- 2 a}}{2} \frac {1 - 2 ^ {- 2 J a}}{1 - 2 ^ {- 2 a}} \epsilon_ {n} ^ {2}. \\ \end{array}
$$
The summation in the first term above is bounded by  $\epsilon_n^2$  as derived in [30, Theorem 4.5]. Therefore,
$$
\frac {1}{n} \mathrm {K L} \left(\mathcal {N} \left(\bar {\theta} _ {0} ^ {J}, \mathcal {C} _ {q}\right) \| \mathcal {N} (0, \mathcal {C})\right) \leq \max  \left(\| \theta_ {0} \| _ {\beta , \infty , \infty} ^ {2}, \frac {2 ^ {- 2 a} - 2 ^ {- 2 J a - 2 a}}{1 - 2 ^ {- 2 a}}\right) \epsilon_ {n} ^ {2}. \tag {103}
$$
Now consider the second term
$$
\begin{array}{l} \frac {1}{n} \mathbb {E} _ {Q _ {n}} \operatorname {K L} \left(P _ {0} ^ {n} \| P _ {\theta} ^ {n}\right) \\ = \mathbb {E} _ {Q _ {n}} \int \left(\Psi_ {1} (\theta_ {0} (y)) \log \frac {\Psi_ {1} (\theta_ {0} (y))}{\Psi_ {1} (\theta (y))} + \Psi_ {- 1} (\theta_ {0} (y)) \log \frac {\Psi_ {- 1} (\theta_ {0} (y))}{\Psi_ {- 1} (\theta (y))}\right) \nu (d y) \\ \leq \mathbb {E} _ {Q _ {n}} \int \langle \theta (y) - \theta_ {0} (y), \theta (y) - \theta_ {0} (y) \rangle \nu (d y) \\ = \mathbb {E} _ {Q _ {n}} \int \| \theta (y) - \theta_ {0} ^ {J} (y) - (\theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {2} ^ {2} \nu (d y) \\ = \mathbb {E} _ {Q _ {n}} \int \| \theta (y) - \theta_ {0} ^ {J} (y) \| _ {2} ^ {2} + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {2} ^ {2} - 2 \langle \theta (y) - \theta_ {0} ^ {J} (y), \theta_ {0} (y) - \theta_ {0} ^ {J} (y) \rangle \nu (d y) \\ \leq \mathbb {E} _ {Q _ {n}} \int \| \theta (y) - \theta_ {0} ^ {J} (y) \| _ {2} ^ {2} \nu (d y) + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ = \mathbb {E} _ {Q _ {n}} \int | \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {2 ^ {j d}} \zeta_ {j} Z _ {j, k} \vartheta_ {j, k} (y) | ^ {2} \nu (d y) + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ \leq \mathbb {E} _ {Q _ {n}} \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {2 ^ {j d}} \zeta_ {j} ^ {2} Z _ {j, k} ^ {2} \int \vartheta_ {j, k} (y) ^ {2} \nu (d y) + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ = \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {2 ^ {j d}} \zeta_ {j} ^ {2} \mathbb {E} _ {Q _ {n}} \left[ Z _ {j, k} ^ {2} \right] + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ = \sum_ {j = 1} ^ {J} \sum_ {k = 1} ^ {2 ^ {j d}} \mu_ {j} ^ {2} (1 - \kappa_ {j} ^ {2}) + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ = \sum_ {j = 1} ^ {J} 2 ^ {j d} \frac {\mu_ {j} ^ {2}}{1 + n \epsilon_ {n} ^ {2} \tau_ {j} ^ {2}} + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ \leq \frac {1}{n \epsilon_ {n} ^ {2}} \sum_ {j = 1} ^ {J} \frac {2 ^ {- 2 j a}}{\tau_ {j} ^ {2}} + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ = \frac {1}{n \epsilon_ {n} ^ {2}} \sum_ {j = 1} ^ {J} 2 ^ {j d} + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ = \frac {2 ^ {d}}{n \epsilon_ {n} ^ {2}} \frac {2 ^ {d J} - 1}{2 ^ {d} - 1} + \| \theta_ {0} (y) - \theta_ {0} ^ {J} (y)) \| _ {\infty} ^ {2} \\ \leq \frac {2 ^ {d} / (2 ^ {d} - 1)}{(\log n) ^ {2}} + C ^ {\prime} \epsilon_ {n} ^ {2}, \\ \end{array}
$$
where in the second inequality, we used the second assertion of Lemma 3.2 [30] for logistic function, the fifth inequality uses the fact that  $\theta(y) - \theta_0^J(y)$  is orthogonal to  $\theta_0(y) - \theta_0^J(y)$ . For any  $a \leq \alpha$  fix  $J = J_{\alpha}$  otherwise  $J = J_{a}$ , and then it is straightforward to check from the definition of  $\epsilon_n$  given in the assertion of the theorem that  $(2^{dJ-1}/n\epsilon_n^2) \leq (\log n)^{-2}$ . The term  $\|\theta_0(y) - \theta_0^J(y)\|_\infty^2$  is also bounded by  $C' \epsilon_n^2$  as shown in the proof of Theorem 4.5 in [30]. Consequently, the term  $\frac{1}{n} \mathbb{E}_{Q_n} \mathrm{KL}(P_0^n \| P_\theta^n)$  is bounded above by  $\epsilon_n^2$  (upto a constant) for sufficiently large  $n$  since  $(\log n)^{-2} < \epsilon_n^2$  and the result follows.
Proof of Theorem 4.2. The proof is a direct consequence of Theorem 3.2, Lemmas B.11, B.12, B.13, and Proposition 3.2.