SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\hat{\theta}_n = \frac{\sum_{i=1}^n \xi_i}{n}$ . Note that  $\hat{\theta}_n - \theta_0 \sim \mathcal{N}(\cdot | 0, \frac{1}{n} \Sigma)$ , where  $\frac{1}{n} \Sigma$  is a symmetric positive definite matrix. Therefore it can be decomposed as  $\Sigma = Q^T \Lambda Q$ , where  $Q$  is an orthogonal matrix and  $\Lambda$  is a diagonal matrix consisting of respective eigen values and consequently  $\hat{\theta}_n - \theta_0 \sim Q \mathcal{N}(\cdot | 0, \frac{1}{n} \Lambda)$ . So, we have  $\| \hat{\theta}_n - \theta_0 \|^2 \sim \| \mathcal{N}(\cdot | 0, \frac{1}{n} \Lambda) \|^2$ . Notice that  $\| \mathcal{N}(\cdot | 0, \frac{1}{n} \Lambda) \|^2$  is a linear combination of  $d \chi_{(1)}^2$  random variable weighted by elements of the diagonal matrix  $\frac{1}{n} \Lambda$ . Using this observation, we first verify that  $\phi_{n,\epsilon}$  satisfies condition (i) of the Lemma. Observe that
$$
\mathbb {E} _ {P _ {0} ^ {n}} \left[ \phi_ {n} \right] = P _ {0} ^ {n} \left(\tilde {X} _ {n}: \left\| \hat {\theta} _ {n} - \theta_ {0} \right\| ^ {2} > C \epsilon^ {2}\right) = P _ {0} ^ {n} \left(\tilde {X} _ {n}: \| \mathcal {N} (\cdot | 0, \Lambda) \| ^ {2} > C n \epsilon^ {2}\right).
$$
Note that  $\chi_{(1)}^2$  is  $\Gamma$  distributed with shape  $1/2$  and scale 2, which implies  $\chi_{(1)}^2 - 1$  is a sub-gamma random variable with scale factor 2 and variance factor 2. Now observe that for  $\hat{\Lambda} = \max_{i \in \{1,2,\ldots,d\}} \Lambda_{ii}$ ,
$$
\begin{array}{l} P _ {0} ^ {n} \left(\tilde {X} _ {n}: \| \mathcal {N} (\cdot | 0, \Lambda) \| ^ {2} > C n \epsilon^ {2}\right) \leq P _ {0} ^ {n} \left(\tilde {X} _ {n}: \chi_ {(1)} ^ {2} > \frac {1}{d \hat {\Lambda}} C n \epsilon^ {2}\right) \\ \leq P _ {0} ^ {n} \left(\tilde {X} _ {n}: \chi_ {(1)} ^ {2} > \frac {1}{d \hat {\Lambda}} C n \epsilon^ {2}\right) \\ = P _ {0} ^ {n} \left(\tilde {X} _ {n}: \chi_ {(1)} ^ {2} - 1 > \frac {1}{d \hat {\Lambda}} C n \epsilon^ {2} - 1\right) \\ \leq e ^ {- \frac {\left(\frac {1}{d \hat {\Lambda}} C n \epsilon^ {2} - 1\right) ^ {2}}{2 \left(2 + 2 \left(\frac {1}{d \hat {\Lambda}} C n \epsilon^ {2} - 1\right)\right)}} \\ \leq e ^ {- 1 / 8 \frac {1}{d \bar {\Lambda}} C n \epsilon^ {2} + 1 / 8} \leq e ^ {- 1 / 8 \left(\frac {C}{d \bar {\Lambda}} - 1\right) n \epsilon^ {2}}, \tag {82} \\ \end{array}
$$
where in the third inequality we used the well known tail bound for sub-gamma random variable (Lemma 3.12 [5]) assuming that  $C$  is sufficiently large such that  $\left(\frac{1}{d\Lambda} C n \epsilon^2 - 1\right) > 1$  and in the last inequality follows from the assumption that  $n \epsilon^2 > n \epsilon_n^2 \geq 1$ .
Now, we fix the alternate set to be  $\{\theta \in \mathbb{R}^d : \| \theta - \theta_0 \| \geq 2\sqrt{C\epsilon^2}\}$ . Next, we verify that  $\phi_{n,\epsilon}$  satisfies condition (ii) of the lemma. First, observe that
$$
\mathbb {E} _ {P _ {\theta} ^ {n}} \left[ 1 - \phi_ {n} \right] = P _ {\theta} ^ {n} \left(\tilde {X} _ {n}: \left\| \hat {\theta} _ {n} - \theta_ {0} \right\| ^ {2} \leq C \epsilon^ {2}\right) \leq P _ {\theta} ^ {n} \left(\tilde {X} _ {n}: \| \hat {\theta} _ {n} - \theta \| \geq \| \theta - \theta_ {0} \| - \sqrt {C \epsilon^ {2}}\right), \tag {83}
$$
where in the last inequality, we used the fact that  $\| \theta -\theta_0\| \leq \| \hat{\theta}_n - \theta \| +\left\| \hat{\theta}_n - \theta_0\right\|$ . Now on alternate set  $\{\theta \in \mathbb{R}^d:\| \theta -\theta_0\| \geq 2\sqrt{C\epsilon^2}\}$ ,
$$
\begin{array}{l} \mathbb {E} _ {P _ {\theta} ^ {n}} \left[ 1 - \phi_ {n} \right] \leq P _ {\theta} ^ {n} \left(\tilde {X} _ {n}: \| \hat {\theta} _ {n} - \theta \| \geq \| \theta - \theta_ {0} \| - \sqrt {C \epsilon^ {2}}\right) \\ \leq P _ {\theta} ^ {n} \left(\tilde {X} _ {n}: \| \hat {\theta} _ {n} - \theta \| \geq \| \theta - \theta_ {0} \| - \sqrt {C \epsilon^ {2}}\right) \\ \leq P _ {\theta} ^ {n} \left(\tilde {X} _ {n}: \| \hat {\theta} _ {n} - \theta \| \geq \sqrt {C \epsilon^ {2}}\right). \tag {84} \\ \end{array}
$$
Now, it follows from (82) and  $\Theta \subset \mathbb{R}^d$  that
$$
\mathbb {E} _ {P _ {0} ^ {n}} \left[ \phi_ {n} \right] \leq e ^ {- 1 / 8 \left(\frac {C}{d \Lambda} - 1\right) n \epsilon^ {2}},
$$
$$
\sup_{\{\theta \in \Theta : \| \theta -\theta_{0}\| \geq 2\sqrt{C\epsilon^{2}}\}}\mathbb{E}_{P_{\theta}^{n}}\bigl[1 - \phi_{n}\bigr ]\leq \sup_{\{\theta \in \mathbb{R}^{d}: \| \theta -\theta_{0}\| \geq 2\sqrt{C\epsilon^{2}}\}}\mathbb{E}_{P_{\theta}^{n}}\bigl[1 - \phi_{n}\bigr ]\leq e^{-1 / 8\bigl(\frac{C}{d\Lambda} -1\bigr)n\epsilon^{2}}.
$$
Using Lemma C.13,  $\{\theta \in \Theta : n^{-1/2} \sqrt{L_n^{MNV}(\theta, \theta_0)} \geq 2K \sqrt{C\epsilon^2}\} = \{\theta \in \Theta : d_L^{MNV}(\theta, \theta_0) \geq 2K \sqrt{C\epsilon^2}\} \subseteq \{\theta \in \Theta : \| \theta - \theta_0 \| \geq 2\sqrt{C\epsilon^2}\}$ , which implies that
$$
\sup_{\{\theta \in \Theta :L_{n}^{M N V}(\theta ,\theta_{0})\geq 4K^{2}C n\epsilon^{2}\}}\mathbb{E}_{P_{\theta}^{n}}[1 - \phi_{n}]\leq \sup_{\{\theta \in \Theta :\| \theta -\theta_{0}\| \geq 2\sqrt{C\epsilon^{2}}\}}\mathbb{E}_{P_{\theta}^{n}}[1 - \phi_{n}].
$$
Therefore,  $P_{\theta}$  for  $\theta \in \Theta$ , satisfies Assumptions 2.1 for  $L_{n}(\theta ,\theta_{0}) = L_{n}^{MNV}(\theta ,\theta_{0})$  for  $C_0 = 1$ ,  $C_1 = 4K^2 C$  and  $C = 1 / 8\left(\frac{C}{d\Lambda} -1\right)$ .
Proof. Proof of Lemma B.7:
First, we write the Rényi divergence between two multivariate Gaussian distribution with known  $\Sigma$  as
$$
D _ {1 + \lambda} \left(\mathcal {N} (\cdot | \theta_ {0}) \| \mathcal {N} (\cdot | \theta)\right) = \frac {\lambda + 1}{2} \left(\theta - \theta_ {0}\right) ^ {T} \Sigma \left(\theta - \theta_ {0}\right), \tag {85}
$$
and  $D_{1 + \lambda}(\mathcal{N}(\cdot |\theta)\| \mathcal{N}(\cdot |\theta_0)) < \infty$  if and only if  $\Sigma^{-1}$  is positive definite [13].
Since, we assumed that the sequence of models are iid, therefore,  $D_{1 + \lambda}(P_0^n\| P_\theta^n) = \frac{1}{\lambda}\log \int \left(\frac{dP_0^n}{dP_\theta^n}\right)^\lambda dP_0^n = n\frac{1}{\lambda}\log \int \left(\frac{dP_0}{dP_\theta}\right)^\lambda dP_0 = n\left(\frac{\lambda + 1}{2} (\theta -\theta_0)^T\Sigma (\theta -\theta_0)\right)$ , when  $\Sigma^{-1}$  is positive definite and  $D_{1 + \lambda}(P_0^n\| P_\theta^n) = \infty$  otherwise. Now observe that
$$
\begin{array}{l} \Pi \big (D _ {1 + \lambda} \left(P _ {0} ^ {n} \| P _ {\theta} ^ {n}\right) \leq n C _ {3} \epsilon_ {n} ^ {2} \big) = \Pi \left(\left(\left(\theta - \theta_ {0}\right) ^ {T} \Sigma (\theta - \theta_ {0})\right) \leq \frac {2}{\lambda + 1} C _ {3} \epsilon_ {n} ^ {2}\right) \\ = \Pi \left(\left[ (\theta - \theta_ {0}) Q \right] ^ {T} \Lambda [ Q (\theta - \theta_ {0}) ]\right) \leq \frac {2}{\lambda + 1} C _ {3} \epsilon_ {n} ^ {2}\left. \right) \\ \geq \Pi \left(\left[ (\theta - \theta_ {0}) Q \right] ^ {T} [ Q (\theta - \theta_ {0}) ]\right) \leq \frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}\left. \right), \\ = \Pi \left(\left[ (\theta - \theta_ {0}) \right] ^ {T} \left[ (\theta - \theta_ {0}) \right]\right) \leq \frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}\left. \right), \tag {86} \\ \end{array}
$$
where  $\hat{\Lambda} = \max_{i\in \{1,2,\ldots d\}}\Lambda_{ii}$  and in the second equality we used eigen value decomposition of  $\Sigma = Q^T\Lambda Q$ . Next, observe that,
$$
\begin{array}{l} \Pi \left(D _ {1 + \lambda} \left(P _ {0} ^ {n} \| P _ {\theta} ^ {n}\right) \leq n C _ {3} \epsilon_ {n} ^ {2}\right) = \Pi \left(\left[ (\theta - \theta_ {0}) \right] ^ {T} [ (\theta - \theta_ {0}) ]\right) \leq \frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}\left. \right) \\ = \Pi \left(\| (\theta - \theta_ {0}) \| \leq \sqrt {\frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}}\right) \\ \geq \Pi \left(\| (\theta - \theta_ {0}) \| _ {\infty} \leq \sqrt {\frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}}\right) \\ = \prod_ {i = 1} ^ {d} \Pi_ {i} \left(| (\theta_ {i} - \theta_ {0} ^ {i}) | \leq \sqrt {\frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}}\right), \\ \end{array}
$$
where in the last equality we used the fact that the prior distribution is uncorrelated. Now, the result follows immediately for sufficiently large  $n$ , if the prior distribution is uncorrelated and uniformly
distributed on the compact set  $\Theta_{i}$ , for each  $i\in \{1,2,\ldots ,d\}$ . In particular observe that for large enough  $n$ , we have
$$
\begin{array}{l} \Pi \left(D _ {1 + \lambda} \left(P _ {0} ^ {n} \| P _ {\theta} ^ {n}\right) \leq n C _ {3} \epsilon_ {n} ^ {2}\right) \geq \prod_ {i = 1} ^ {d} \frac {\theta_ {0} ^ {i} + \sqrt {\frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}} - \theta_ {0} ^ {i} + \sqrt {\frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}}}{m (\Theta_ {i})} \\ = \frac {2 ^ {d} \left(\frac {2}{\hat {\Lambda} (\lambda + 1)} C _ {3} \epsilon_ {n} ^ {2}\right) ^ {d / 2}}{\prod_ {i = 1} ^ {d} m (\Theta_ {i})} = \left(\frac {8}{(\hat {\Lambda} (\lambda + 1))} \left(\prod_ {i = 1} ^ {d} m (\Theta_ {i})\right) ^ {- 2 / d} C _ {3} \epsilon_ {n} ^ {2}\right) ^ {d / 2}, \\ \end{array}
$$
where  $m(A)$  is the Lebesgue measure (volume) of any set  $A\subset \mathbb{R}$ . Now if  $\epsilon_n^2 = \frac{\log n}{n}$ , then for  $\frac{8}{\hat{\Lambda}(\lambda + 1)\left(\prod_{i = 1}^{d}m(\Theta_i)\right)^{2 / d}} C_3 > 2$ ,  $\frac{8}{\hat{\Lambda}(\lambda + 1)\left(\prod_{i = 1}^{d}m(\Theta_i)\right)^{2 / d}} C_3\epsilon_n^2\geq e^{-\frac{8}{\hat{\Lambda}(\lambda + 1)\left(\prod_{i = 1}^{d}m(\Theta_i)\right)^{2 / d}}C_3n\epsilon_n^2}$  for all  $n\geq 2$ , therefore,
$$
\Pi (D _ {1 + \lambda} \left(P _ {0} ^ {n} \| P _ {\theta} ^ {n}\right) \leq n C _ {3} \epsilon_ {n} ^ {2}) \geq e ^ {- \frac {4 d}{\hat {\Lambda} (\lambda + 1) \left(\prod_ {i = 1} ^ {d} m (\Theta_ {i})\right) ^ {2 / d}} C _ {3} n \epsilon_ {n} ^ {2}}.
$$
Proof. Proof of Lemma B.9: Since family  $\mathcal{Q}$  contains all uncorrelated Gaussian distributions restricted to  $\Theta$ , observe that  $\{q_n(\cdot) \in \mathcal{Q}\} \forall n \geq 1$ . By definition,  $q_n^i(\theta) \propto \frac{1}{\sqrt{2\pi\sigma_{i,n}^2}} e^{-\frac{1}{2\sigma_{i,n}^2} (\theta - \mu_{i,n})^2} \mathbb{1}_{\Theta_i} = \frac{\mathcal{N}(\theta_i | \mu_{i,n}, \sigma_{i,n}) \mathbb{1}_{\Theta_i}}{\mathcal{N}(\Theta_i | \mu_{i,n}, \sigma_{i,n})}$  and fix  $\sigma_{i,n} = 1 / \sqrt{n}$  and  $\theta_i = \theta_0^i$  for all  $i \in \{1, 2, \ldots, d\}$ . Now consider the first term; using the definition of the KL divergence it follows that
$$
\operatorname {K L} \left(q _ {n} (\theta) \| \pi (\theta)\right) = \int q _ {n} (\theta) \log \left(q _ {n} (\theta)\right) d \theta - \int q _ {n} (\theta) \log (\pi (\theta)) d \theta . \tag {87}
$$
$$
\begin{array}{l} \int q _ {n} (\theta) \log (q _ {n} (\theta)) d \theta = \sum_ {i = 1} ^ {d} \int q _ {n} ^ {i} (\theta_ {i}) \log (q _ {n} ^ {i} (\theta_ {i})) d \theta_ {i} \\ \leq \sum_ {i = 1} ^ {d} \int \mathcal {N} \left(\theta_ {i} \mid \mu_ {i, n}, \sigma_ {i, n}\right) \log \mathcal {N} \left(\theta_ {i} \mid \mu_ {i, n}, \sigma_ {i, n}\right) d \theta_ {i} \\ = - \sum_ {i = 1} ^ {d} \left[ \log \left(\sqrt {2 \pi e}\right) + \log \sigma_ {i, n} \right], \tag {88} \\ \end{array}
$$
where in the last equality, we used the well known expression for the differential entropy of Gaussian distributions. Recall  $\pi (\theta) = \prod_{i = 1}^{d}\frac{1}{m(\Theta_i)}$ . Now consider the second term in (87). It is straightforward to observe that,
$$
- \int q _ {n} (\theta) \log (\pi (\theta)) d \theta = \sum_ {i = 1} ^ {d} \log (m \left(\Theta_ {i}\right)). \tag {89}
$$
Substituting (89) and (88) into (87) and dividing either sides by  $n$  and substituting  $\sigma_{i,n}$ , we obtain
$$
\begin{array}{l} \frac {1}{n} \mathrm {K L} (q _ {n} (\theta) \| \pi (\theta)) \leq - \frac {1}{n} \sum_ {i = 1} ^ {d} [ \log (\sqrt {2 \pi e}) - \log (m (\Theta_ {i})) - \frac {1}{2} \log n ] \\ = \frac {d}{2} \frac {\log n}{n} - \frac {1}{n} \sum_ {i = 1} ^ {d} \left[ \log \left(\sqrt {2 \pi e}\right) - \log \left(m \left(\Theta_ {i}\right)\right) \right]. \tag {90} \\ \end{array}
$$
Now, consider the second term in the assertion of the lemma. Since  $\xi_i, i \in \{1, 2 \dots n\}$  are independent and identically distributed, we obtain
Now using the expression for KL divergence between the two multivariate Gaussian distributions, we have
$$
\begin{array}{l} \frac {1}{n} \mathbb {E} _ {q _ {n} (\theta)} \left[ \mathrm {K L} \left(d P _ {0} ^ {n} \| p (\tilde {X} _ {n} | \theta)\right) \right] = \frac {1}{2} \mathbb {E} _ {q _ {n} (\theta)} \left[ (\theta - \theta_ {0}) ^ {T} \Sigma^ {- 1} (\theta - \theta_ {0}) \right] \\ \leq \frac {\tilde {\Lambda} ^ {- 1}}{2} \mathbb {E} _ {q _ {n} (\theta)} \left[ \left(\theta - \theta_ {0}\right) ^ {T} \left(\theta - \theta_ {0}\right) \right] \\ \leq \frac {d}{n} \frac {\check {\Lambda} ^ {- 1}}{2} \tag {91} \\ \end{array}
$$
where  $\check{\Lambda} = \min_{i\in \{1,2,\dots d\}}\Lambda_{ii}$ , and  $\Sigma^{-1} = Q^T\Lambda^{-1}Q$ , where  $Q$  is an orthogonal matrix and  $\Lambda$  is a diagonal matrix consisting of the respective eigen values of  $\Sigma$ . Combined together (91) and (90) implies that
$$
\begin{array}{l} \frac {1}{n} \left[ \operatorname {K L} \left(q _ {n} (\theta) \| \pi (\theta)\right) + \mathbb {E} _ {q _ {n} (\theta)} \left[ \operatorname {K L} \left(d P _ {0} ^ {n}\right) \| p (\tilde {X} _ {n} | \theta)\right) \right] \\ \leq \frac {d}{2} \frac {\log n}{n} - \frac {1}{n} \sum_ {i = 1} ^ {d} \left[ \log \left(\sqrt {2 \pi e}\right) - \log \left(m \left(\Theta_ {i}\right)\right) \right] + \frac {d}{n} \frac {\check {\Lambda} ^ {- 1}}{2} \leq C _ {9} \frac {\log n}{n}. \tag {92} \\ \end{array}
$$
where  $C_9 \coloneqq \frac{d}{2} + \max \left(0, -\sum_{i=1}^{d} [\log(\sqrt{2\pi e}) - \log(m(\Theta_i))] + \frac{d}{2} \breve{\Lambda}^{-1}\right)$  and the result follows.
C.8 Gaussian process classification
Proof of Lemma B.11. In view of Theorem 7.1 in [11], it suffices to show that
$$
N \left(\epsilon , \Theta_ {n} (\epsilon), d _ {\mathrm {T V}}\right) \leq e ^ {\bar {C} n \epsilon^ {2}},
$$
for some  $\bar{C} > 0$ . Now, first observe that
$$
\begin{array}{l} d _ {\mathrm {T V}} \left(P _ {\theta (y)}, P _ {\theta_ {0} (y)}\right) = \frac {1}{2} \mathbb {E} _ {\nu} \left(\left| \Psi_ {1} (\theta (y)) - \Psi_ {1} \left(\theta_ {0} (y)\right) \right| + \left| \Psi_ {- 1} (\theta (y)) - \Psi_ {- 1} \left(\theta_ {0} (y)\right) \right|\right) \\ = \mathbb {E} _ {\nu} \left(\left| \Psi_ {1} (\theta (y)) - \Psi_ {1} \left(\theta_ {0} (y)\right) \right|\right) \\ \leq \mathbb {E} _ {\nu} \left(| \theta (y) - \theta_ {0} (y) |\right) \leq \| \theta (y) - \theta_ {0} (y) \| _ {\infty}, \tag {93} \\ \end{array}
$$
where the second equality uses the definition of  $\Psi_{-1}(\cdot)$ . Since, total-variation distance above is bounded above by supremum norm, there exists a constant  $0 < c' < 1/2$ , such that
$$
N \left(\epsilon , \Theta_ {n} (\epsilon), d _ {\mathrm {T V}}\right) \leq N \left(c ^ {\prime} \epsilon , \Theta_ {n} (\epsilon), \| \cdot \| _ {\infty}\right) \leq e ^ {\frac {2}{3} c ^ {\prime 2} C _ {1 0} n \epsilon^ {2}}, \tag {94}
$$
where the last inequality follows from (13) in Lemma B.10. Then if follows from Theorem 7.1 in [11] that for every  $\epsilon >\epsilon_{n}$ , there exists a test  $\phi_{n}$  (depending on  $\epsilon >0$ ) such that, for every  $j\geq 1$
$$
\mathbb {E} _ {P _ {0} ^ {n}} [ \phi_ {n} ] \leq e ^ {\frac {2}{3} c ^ {\prime 2} C _ {1 0} n \epsilon^ {2}} e ^ {- \frac {1}{2} n \epsilon^ {2}} \frac {1}{1 - \exp \left(- \frac {1}{2} n \epsilon^ {2}\right)}, \mathrm {a n d}
$$
$$
\sup_{\{\theta \in \Theta_{n}(\epsilon):d_{TV}(P_{\theta},P_{\theta_{0}}) > j\epsilon \}}\mathbb{E}_{P_{\theta}^{n}}[1 - \phi_{n}]\leq \exp \left(-\frac{1}{2} n\epsilon^{2}j\right).
$$
Now for all  $n$  such that  $n\epsilon^2 > n\epsilon_n^2 > 2\log 2$  and  $C_{10} = c'^{-2} / 4 > 1$  and  $j = 1$ , we have
$$
\mathbb {E} _ {P _ {0} ^ {n}} \left[ \phi_ {n} \right] \leq 2 e ^ {- \frac {1}{3} n \epsilon^ {2}}, \text {a n d} \tag {95}
$$
$$
\sup  _ {\left\{\theta \in \Theta_ {n} (\epsilon): d _ {T V} \left(P _ {\theta}, P _ {\theta_ {0}}\right) > \epsilon \right\}} \mathbb {E} _ {P _ {\theta} ^ {n}} \left[ 1 - \phi_ {n} \right] \leq e ^ {- \frac {1}{2} n \epsilon^ {2}} \leq e ^ {- \frac {1}{3} n \epsilon^ {2}}. \tag {96}
$$
Now observe that
$$
\begin{array}{l} \sup  _ {a \in \mathcal {A}} | G (a, \theta) - G (a, \theta_ {0}) | \\ = \max  \left(c _ {+} | \mathbb {E} _ {\nu} [ \Psi_ {- 1} (\theta (y)) ] - \mathbb {E} _ {\nu} [ \Psi_ {- 1} (\theta_ {0} (y)) ] |, c _ {-} | \mathbb {E} _ {\nu} [ \Psi_ {1} (\theta (y)) ] - \mathbb {E} _ {\nu} [ \Psi_ {1} (\theta_ {0} (y)) ] |\right) \\ = \max  \left(c _ {+} | \mathbb {E} _ {\nu} \left[ \Psi_ {1} \left(\theta_ {0} (y)\right) \right] - \mathbb {E} _ {\nu} \left[ \Psi_ {1} (\theta (y)) \right] \mid , c _ {-} | \mathbb {E} _ {\nu} \left[ \Psi_ {1} \left(\theta (y)\right) \right] - \mathbb {E} _ {\nu} \left[ \Psi_ {1} \left(\theta_ {0} (y)\right) \right] |\right) \\ = \max  \left(c _ {+}, c _ {-}\right) \left| \mathbb {E} _ {\nu} \left[ \Psi_ {1} \left(\theta_ {0} (y)\right) \right] - \mathbb {E} _ {\nu} \left[ \Psi_ {1} (\theta (y)) \right] \right| \\ \leq \max  \left(c _ {+}, c _ {-}\right) \mathbb {E} _ {\nu} \left[ \left| \Psi_ {1} \left(\theta_ {0} (y)\right) - \Psi_ {1} (\theta (y)) \right| \right] \\ \leq \max  \left(c _ {+}, c _ {-}\right) d _ {T V} \left(P _ {\theta}, P _ {\theta_ {0}}\right) \tag {97} \\ \end{array}
$$
where the second equality uses the fact that  $\Psi_{-1}(\cdot) = 1 - \Psi_1(\cdot)$ .
$$
\{\theta \in \Theta_ {n} (\epsilon): \sup  _ {a \in \mathcal {A}} | G (a, \theta) - G (a, \theta_ {0}) | > \max  (c _ {+}, c _ {-}) \epsilon \} \subseteq \{\theta \in \Theta_ {n} (\epsilon): d _ {T V} (P _ {\theta}, P _ {\theta_ {0}}) > \epsilon \}
$$
Therefore, it follows from (95) and (96) and the definition of  $L_{n}(\theta ,\theta_{0})$  that
$$
\mathbb {E} _ {P _ {0} ^ {n}} \left[ \phi_ {n} \right] \leq 2 e ^ {- \frac {1}{3} n \epsilon^ {2}}, \text {a n d} \tag {98}
$$
$$
\sup  _ {\left\{\theta \in \Theta_ {n} (\epsilon): L _ {n} \left(\theta , \theta_ {0}\right) > \left(\max  \left(c _ {+, c _ {-}}\right)\right) ^ {2} n \epsilon^ {2} \right\}} \mathbb {E} P _ {\theta} ^ {n} \left[ 1 - \phi_ {n} \right] \leq e ^ {- \frac {1}{2} n \epsilon^ {2}} \leq e ^ {- \frac {1}{3} n \epsilon^ {2}}. \tag {99}
$$
Finally, the result follows for  $C = 1/3$ ,  $C_0 = 2$  and  $C_1 = (\max(c_+, c_-))^2$ .
Proof of Lemma B.12. The Rényi divergence
$$
\begin{array}{l} D _ {1 + \lambda} \left(P _ {0} ^ {n} \| P _ {\theta} ^ {n}\right) \\ = n \frac {1}{\lambda} \ln \int (\Psi_ {1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {- \lambda} + \Psi_ {- 1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {- \lambda}) \nu (d y) \\ = n \frac {1}{\lambda} \ln \int e ^ {\lambda \frac {1}{\lambda} \ln \left(\Psi_ {1} \left(\theta_ {0} (y)\right) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {- \lambda} + \Psi_ {- 1} \left(\theta_ {0} (y)\right) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {- \lambda}\right)} \nu (d y). \tag {100} \\ \end{array}
$$
Note that the derivative of the exponent in the integrand above with respect to  $\theta(y)$  is
$$
\begin{array}{l} \begin{array}{c} \big (- \lambda \Psi_ {1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {- \lambda - 1} \psi (\theta (y)) + \lambda \Psi_ {- 1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {- \lambda - 1} \psi (\theta (y)) \big) \\ \hline (\Psi_ {1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {- \lambda} + \Psi_ {- 1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {- \lambda}) \end{array} \\ = \lambda \psi (\theta (y)) \frac {\left(- \Psi_ {1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {- \lambda - 1} + \Psi_ {- 1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {- \lambda - 1}\right)}{(\Psi_ {1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {- \lambda} + \Psi_ {- 1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {- \lambda})} \\ = \lambda \frac {\psi (\theta (y))}{\Psi_ {1} (\theta (y)) \Psi_ {- 1} (\theta (y))} \frac {\left(- \Psi_ {1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {\lambda + 1} + \Psi_ {- 1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {\lambda + 1}\right)}{\left(\Psi_ {1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {\lambda} + \Psi_ {- 1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {\lambda}\right)} \\ = \lambda \frac {\left(- \Psi_ {1} \left(\theta_ {0} (y)\right) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {\lambda + 1} + \Psi_ {- 1} \left(\theta_ {0} (y)\right) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {\lambda + 1}\right)}{\left(\Psi_ {1} \left(\theta_ {0} (y)\right) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {\lambda} + \Psi_ {- 1} \left(\theta_ {0} (y)\right) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {\lambda}\right)} \\ = \lambda \frac {\left(- e ^ {- (\lambda + 1) \theta (y)} + e ^ {- (1 + \lambda) \theta_ {0} (y)}\right)}{\left(e ^ {- \lambda \theta (y)} + e ^ {- (\lambda + 1) \theta_ {0} (y)}\right) \left(1 + e ^ {- \theta (y)}\right)} \\ = \lambda \frac {e ^ {- (1 + \lambda) \theta_ {0} (y)} \left(1 - e ^ {- (\lambda + 1) (\theta (y) - \theta_ {0} (y))}\right)}{\left(e ^ {- \lambda \theta (y)} + e ^ {- (\lambda + 1) \theta_ {0} (y)}\right) \left(1 + e ^ {- \theta (y)}\right)} \\ \leq \lambda \frac {(\lambda + 1) (\theta (y) - \theta_ {0} (y))}{(e ^ {- \lambda \theta (y) + (\lambda + 1) \theta_ {0} (y)} + 1) (1 + e ^ {- \theta (y)})} \\ \leq \lambda (\lambda + 1) | \theta (y) - \theta_ {0} (y) |, \tag {101} \\ \end{array}
$$
where in the fourth equality we used definition of the logistic function and the penultimate inequality follows from the well known inequality that  $1 - e^{-x} \leq x$ . Consequently, using Taylor's theorem it follows that the exponent in the integrand of the Rényi divergence in (100) is bounded above by  $\lambda (\lambda + 1)|\theta(y) - \theta_0(y)|^2$  and thus by  $\lambda (\lambda + 1)\|\theta(y) - \theta_0(y)\|_\infty^2$ . Therefore,
$$
\begin{array}{l} D _ {1 + \lambda} \left(P _ {0} ^ {n} \| P _ {\theta} ^ {n}\right) \\ = n \frac {1}{\lambda} \ln \int (\Psi_ {1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {1} (\theta (y)) ^ {- \lambda} + \Psi_ {- 1} (\theta_ {0} (y)) ^ {1 + \lambda} \Psi_ {- 1} (\theta (y)) ^ {- \lambda}) \nu (d y) \\ \leq n \frac {1}{\lambda} \ln \int e ^ {\lambda (\lambda + 1) \| \theta (y) - \theta_ {0} (y) \| _ {\infty} ^ {2}} \nu (d y) \\ = n (\lambda + 1) \| \theta (y) - \theta_ {0} (y) \| _ {\infty} ^ {2}. \\ \end{array}
$$
Now using the inequality for  $C_3 = 16(\lambda + 1)$  above observe that
$$
\begin{array}{l} \Pi (A _ {n}) = \Pi (D _ {1 + \lambda} \left(P _ {0} ^ {n} \| P _ {\theta} ^ {n}\right) \leq C _ {3} n \epsilon_ {n} ^ {2}) \\ \geq \Pi (n (\lambda + 1) \| \theta (y) - \theta_ {0} (y) \| _ {\infty} ^ {2} \leq C _ {3} n \epsilon_ {n} ^ {2}) \\ = \Pi \left(\left\| \theta (y) - \theta_ {0} (y) \right\| _ {\infty} \leq 4 \epsilon_ {n}\right) \geq e ^ {- n \epsilon_ {n} ^ {2}} \tag {102} \\ \end{array}
$$
and the result follows from (15) of Lemma B.10.
Proof of Lemma B.13. Let us first analyze the KL divergence between the prior distribution and variational family. Recall that two Gaussian measures on infinite dimensional spaces are either equivalent or singular. [27, Theorem 6.13] specify the condition required for the two Gaussian measures to be equivalent. In particular, note that  $\theta_0^J (\cdot)\in \mathrm{Im}(\mathcal{C}^{1 / 2})$ . Now observe that the covariance operator of  $Q_{n}$  has eigenvalues  $\{\zeta_j^2\}_{j = 1}^J$  therefore operator  $S$  in the definition of  $\mathcal{C}_q$  has eigenvalues  $\{1 - \zeta_j^2 /\mu_j^2\}_{j = 1}^J$  For any  $a > 0$ ,  $\sum_{j = 1}^{J}2^{jd}\left(\frac{n\epsilon_n^22^{-2ja - jd}}{1 + n\epsilon_n^22^{-2ja - jd}}\right)^2 = \sum_{j = 1}^{J}2^{-jd}\left(\frac{n\epsilon_n^22^{-2ja}}{1 + n\epsilon_n^22^{-2ja - jd}}\right)^2 < \infty$ , therefore  $S$  is an HS operator.