SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left. U _ {8, 2} \epsilon_ {n} ^ {2} / a _ {n} H _ {n 2} (U _ {8, 2} ^ {\prime} I _ {n} ^ {(2)}) ^ {H _ {n 2}} (k _ {n 2} r _ {n 2} / H _ {n 2}) ^ {H _ {n 2}} \right\}
$$
for some constants  $U_{8,1}, U_{8,2} > 0$  and  $U_{8,1}'$ ,  $U_{8,2}' > 1$ .
Thus, the log covering number of  $\mathcal{P}_n$ ,  $\log N(\epsilon_n / 36, \mathcal{P}_n, d_n)$ , can be upper bounded by  $\log \mathcal{K}_n = \log \mathcal{K}_{n1} + \log \mathcal{K}_{n2} + \log \mathcal{K}_{n3} + \log \mathcal{K}_{n4}$ , with  $\mathcal{K}_{nj}$  given in (25) for  $j = 1, 2, 3, 4$ .
Considering the fact that  $\log \chi^{(1)}(j) \leq \log \binom{p_n}{k_{n1}^{\prime}s_{n1}}\binom{k_{n1}^{\prime}s_{n1}+H_{n1}\overline{L}_{n1}}{j} \leq k_{n1}^{\prime}s_{n1}\log(p_n)+$ $j\log(k_{n1}^{\prime}s_{n1} + H_{n1}\overline{L}_{n1}^{2}) \leq k_{n1}^{\prime}s_{n1}\log(p_n) + k_{n1}r_{n1}\log\{H_{n1}(k_{n1}^{\prime}s_{n1} + \overline{L}_{n1})^{2}\}$ , and by choosing  $\log I_n^{(1)} = O(\log n)$ ,  $\log I_n^{(2)} = O(\log n)$ ,  $a_n \asymp \sqrt{n\epsilon_n^2}$  and  $b_n \asymp \sqrt{n\epsilon_n^2}$ , we have that
$\log \mathcal{K}_{n1}$
$$
\begin{array}{l} \leq \log (k _ {n 1} r _ {n 1}) + \log \chi^ {(1)} (k _ {n 1} r _ {n 1}) + k _ {n 1} r _ {n 1} \log \left(\frac {I _ {n} ^ {(1)}}{\rho_ {n}} + 1\right) \\ \leq \log \left(k _ {n 1} r _ {n 1}\right) + k _ {n 1} ^ {\prime} s _ {n 1} \log \left(p _ {n}\right) + k _ {n 1} r _ {n 1} \log H _ {n 1} + 2 k _ {n 1} r _ {n 1} \log \left(k _ {n 1} ^ {\prime} s _ {n 1} + \bar {L} _ {n 1}\right) \\ + k _ {n 1} r _ {n 1} \left\{\text {c o n s t a n t} + \log I _ {n} ^ {(1)} + \log b _ {n} + \log H _ {n 1} + H _ {n 1} \log \left(U _ {7, 1} ^ {\prime} I _ {n} ^ {(1)}\right) + H _ {n 1} \log \left(k _ {n 1} r _ {n 1} / H _ {n 1}\right) \right. \\ \left. + \log a _ {n} + \log H _ {n 2} + H _ {n 2} \log (U _ {7, 2} ^ {\prime} I _ {n} ^ {(2)}) + H _ {n 2} \log (k _ {n 2} r _ {n 2} / H _ {n 2}) + \log \frac {1}{\epsilon_ {n} ^ {2}} \right\} \\ = k _ {n 1} ^ {\prime} s _ {n 1} \log (p _ {n}) + k _ {n 1} r _ {n 1} \cdot O \left(H _ {n 1} \log n + H _ {n 2} \log n + \log \bar {L} _ {n 1} + \log \bar {L} _ {n 2}\right), \\ \end{array}
$$
where the last equality holds since  $k_{n1}r_{n1}\leq n$ ,  $k_{n2}r_{n2}\leq n$ ,  $k_{n1}'s_{n1}\leq n$  and  $k_{n1}'s_{n1}\leq n$ . By choosing  $k_{n1}'$  and  $k_{n1}$  such that  $k_{n1}r_{n1}\left(H_{n1}\log n + H_{n2}\log n + \log \overline{L}_{n1} + \log \overline{L}_{n2}\right)$ $\asymp k_{n1}'s_{n1}\log (p_n)\asymp n\epsilon_n^2$ , we have  $n\epsilon_n^2 = O\big\{r_{n1}H_{n1}\log n + r_{n1}H_{n2}\log n + r_{n1}\log \overline{L}_{n1} + r_{n1}\log \overline{L}_{n2} + s_{n1}\log (p_n)\big\}$ . By applying similar discussion on  $\log \mathcal{K}_{n2}$ ,  $\log \mathcal{K}_{n3}$  and  $\log \mathcal{K}_{n4}$ , we further obtain that  $\log N(\mathcal{P}_n,\epsilon_n)\leq \log \mathcal{K}_n\leq n\epsilon_n^2$ , where the sequence
$$
\begin{array}{l}\{\epsilon_{n}\}_{n = 1}^{\infty}\text{satisfies that} n\epsilon_{n}^{2} = O\big\{(r_{n1} + r_{n2})(H_{n1} + H_{n2})\log n + (r_{n1} + r_{n2})(\log \overline{L}_{n1} + \log \overline{L}_{n2})\\ +(s_{n1} + s_{n2})\log p_{n}\big\} . \end{array}
$$
B.2.4 Verification of Condition (b) in Lemma 3
Proof. Since  $\Pi(\Theta_n^c) \leq \Pi(\mathcal{S}_{n1}^c) + \Pi(\mathcal{S}_{n2}^c) + \Pi(\mathcal{B}_n^c) + \Pi(\mathcal{A}_n^c)$ , we first examine  $\Pi(\mathcal{B}_n^c)$  and  $\Pi(\mathcal{A}_n^c)$ . For any  $b_n > 0$ , by the prior assumption given in (6), we have that
$$
\begin{array}{l} \Pi \left(\mathcal {B} _ {n} ^ {c}\right) = \Pi \left\{\mathbf {B} ^ {(k)} = \left(\beta_ {1} ^ {(k)},..., \beta_ {K} ^ {(k)}\right) \text {f o r} k \in [ K ]: \left| \beta_ {l, j} ^ {(k)} \right| \leq b _ {n} \text {f o r} l, k \in [ K ], \text {a n d} j \in [ M ^ {(2)} ] \right\} ^ {c} \\ = \Pi \left(\cup_ {k, l \in [ K ], j \in [ M ^ {(2)} ]} \left| \beta_ {l, j} ^ {(k)} \right| > b _ {n}\right) \\ \leq \sum_ {k = 1} ^ {K} \sum_ {l = 1} ^ {K} \sum_ {j = 1} ^ {M ^ {(2)}} \pi_ {B} ^ {(k)} \left(\left| \beta_ {l, j} ^ {(k)} \right| > b _ {n}\right) \\ = K \sum_ {k = 1} ^ {K} \sum_ {j = 1} ^ {M ^ {(2)}} \pi_ {B} ^ {(k)} \left(\left| \beta_ {1, j} ^ {(k)} \right| > b _ {n}\right), \\ \end{array}
$$
where  $\pi_B^{(k)}(\cdot)$  denotes the measure induced by the prior distribution of  $\mathbf{B}^{(k)}$  as defined in Section 3.2.
Applying the bounds on the Mills ratio [76] for the standard normal distribution:  $\frac{1}{x} -\frac{1}{x^3} <  \frac{1 - \Phi(x)}{\varphi(x)} <  \frac{1}{x} -\frac{1}{x^3} +\frac{3}{x^5}$  for  $-\infty < x < \infty$  , where  $\Phi (\cdot)$  and  $\varphi (\cdot)$  denote the CDF and probability density function (PDF) of the standard normal distribution, respectively, we further obtain that, by taking  $b_{n} = \max_{k\in [K]}\sqrt{5\overline{B}_{n}^{(k)}n\epsilon_{n}^{2}}$
$$
\begin{array}{l} \pi_ {B} ^ {(k)} (| \beta_ {1, j} ^ {(k)} | > b _ {n}) <   \frac {2 \varphi (b _ {n} / \sqrt {(\pmb {\Sigma} _ {\beta} ^ {(k)}) _ {j , j}})}{b _ {n} / \sqrt {(\pmb {\Sigma} _ {\beta} ^ {(k)}) _ {j , j}}} = \frac {2}{\sqrt {2 \pi b _ {n} ^ {2} / (\pmb {\Sigma} _ {\beta} ^ {(k)}) _ {j , j}}} \cdot \exp \left\{- \frac {1}{2} (b _ {n} ^ {2} / (\pmb {\Sigma} _ {\beta} ^ {(k)}) _ {j, j}) \right\} \\ \leq \frac {2}{\sqrt {2 \pi b _ {n} ^ {2} / \overline {{B}} _ {n} ^ {(k)}}} \cdot \exp \left\{- \frac {1}{2} (b _ {n} ^ {2} / \overline {{B}} _ {n} ^ {(k)}) \right\} \prec \exp \left(- \frac {5}{2} n \epsilon_ {n} ^ {2}\right) \\ \end{array}
$$
where  $A_{j,j}$  represents the  $(j,j)$  element of matrix  $A$ , the first step is due to the Mills ratio bounds, the second step comes from the definition of the PDF of the standard normal distribution, the third step is due to the fact that the sum of the eigenvalues of a matrix equals the trace of the matrix and the positivity of  $\left(\Sigma_{\beta}^{(k)}\right)_{j,j}$ , and the last step comes from the choice of  $b_{n}$ . Therefore, it follows that  $\Pi (\mathcal{B}_n^c) < K^2 M^{(2)}\exp (-\frac{5}{2} n\epsilon_n^2)$  for large enough  $n$ .
Similarly, we can obtain that  $\Pi (\mathcal{A}_n^c) < K^2 M^{(1)}\exp (-\frac{5}{2} n\epsilon_n^2)$  for large enough  $n$  by taking  $a_{n} = \max_{r\in [R]}\sqrt{5\overline{A}_{n}^{(r)}n\epsilon_{n}^{2}}.$
We then consider  $\Pi(S_{n1}^{c})$ , which can be upper bounded by
$$
\begin{array}{l} \Pi \left(\mathcal {S} _ {n 1} ^ {c}\right) \leq \Pi \left(\boldsymbol {\theta} ^ {(1)}: \cup_ {j = 1} ^ {J _ {n 1}} \mid \theta_ {j} ^ {(1)} \mid > I _ {n} ^ {(1)}\right) + \Pi \left(\boldsymbol {\theta} ^ {(1)}: \mid \boldsymbol {\gamma} _ {\theta} ^ {(1)} \mid = \mid \{j: \mid \theta_ {j} ^ {(1)} \mid \geq \delta_ {n 1} ^ {\prime} \} \mid > k _ {n 1} r _ {n 1}\right) \\ + \pi \left(\boldsymbol {\theta} ^ {(1)}: \left| \boldsymbol {\gamma} _ {\theta} ^ {(1)} \right| _ {\text {i n}} > k _ {n 1} ^ {\prime} s _ {n 1}\right) \\ = J _ {n 1} \pi_ {\theta} ^ {(1)} \left(\left| \theta_ {1} ^ {(1)} \right| > I _ {n} ^ {(1)} |\right) + \mathbb {P} \left\{\operatorname {B i n o m i a l} \left(J _ {n 1}, v _ {n}\right) > k _ {n 1} r _ {n 1} \right\} \tag {26} \\ + \mathbb {P} \{\mathrm {B i n o m i a l} (p _ {n} L _ {1} ^ {(1)}, \nu_ {n}) > k _ {n 1} ^ {\prime} s _ {n 1} \} \\ \end{array}
$$
where  $\nu_{n} = 1 - \pi_{\theta}^{(1)}([-\delta_{n1}^{\prime},\delta_{n1}^{\prime}])\leq \frac{1}{J_{n1}}\exp \{(H_{n1} + H_{n2})\log n + \log \overline{L}_{n1} + \log \overline{L}_{n2} + \log p_n\}$  according to Condition B.1.2 in Section A.3. For the first term in (26), we have that  $J_{n1}\pi_{\theta}^{(1)}(|\theta_1^{(1)}| > I_n^{(1)}|)\prec \exp \{-(2 + \epsilon_3^{(k)})n\epsilon_n^2\}$  following Condition B.1.3 in Section A.3, where  $\epsilon_3^{(k)} > 0$  is a
constant. The second term can be upper bounded by
$$
\begin{array}{l} \mathbb {P} \left\{\operatorname {B i n o m i a l} \left(J _ {n 1}, \nu_ {n}\right) > k _ {n 1} r _ {n 1} \right\} \\ = \sum_ {j = k _ {n 1} r _ {n 1} + 1} ^ {J _ {n 1}} \binom {J _ {n 1}} {j} \nu_ {n} ^ {j} (1 - \nu_ {n}) ^ {J _ {n 1} - j} \\ \leq \sum_ {j = k _ {n 1} r _ {n 1} + 1} ^ {J _ {n 1}} J _ {n 1} ^ {j} \nu_ {n} ^ {j} (1 - \nu_ {n}) ^ {J _ {n 1} - j} \leq J _ {n 1} \left(J _ {n 1} \nu_ {n}\right) ^ {k _ {n 1} r _ {n 1}} \\ \leq J _ {n 1} \exp \left[ - S _ {0} ^ {(1)} k _ {n 1} r _ {n 1} \left\{\left(H _ {n 1} + H _ {n 2}\right) \log n + \log \bar {L} _ {n 1} + \log \bar {L} _ {n 2} + \log p _ {n} \right\} \right] \\ \prec \exp \left\{- S _ {0} ^ {(1)} n \epsilon_ {n} ^ {2} \right\}, \\ \end{array}
$$
where the last inequality is due to the choice of  $k_{n1}$  in Appendix B.2.3, and  $S_0^{(1)} > 2$  is a constant given in Condition B.1.2 in Section A.3.
Similarly, for the third term in (26) we have that
$$
\begin{array}{l} \mathbb {P} \left\{\operatorname {B i n o m i a l} \left(p _ {n} L _ {1} ^ {(1)}, v _ {n}\right) > k _ {n 1} ^ {\prime} s _ {n 1} \right\} \\ = \sum_{j = k^{\prime}_{n1}s_{n1} + 1}^{p_{n}L_{1}^{(1)}}j\binom {p_{n}L_{1}^{(1)}}{j}\nu_{n}^{j}(1 - \nu_{n})^{p_{n}L_{1}^{(1)} - j} \\ \leq \sum_ {j = k _ {n 1} ^ {\prime} s _ {n 1} + 1} ^ {p _ {n} L _ {1} ^ {(1)}} (p _ {n} L _ {1} ^ {(1)}) ^ {j} \nu_ {n} ^ {j} (1 - \nu_ {n}) ^ {p _ {n} L _ {1} ^ {(1)} - j} = p _ {n} L _ {1} ^ {(1)} \left(J _ {n 1} \nu_ {n}\right) ^ {k _ {n 1} ^ {\prime} s _ {n 1}} \left(\frac {p _ {n} L _ {1} ^ {(1)}}{J _ {n 1}}\right) ^ {k _ {n 1} ^ {\prime} s _ {n 1}} \\ \leq p _ {n} L _ {1} ^ {(1)} \exp \left[ - S _ {0} ^ {(1)} k _ {n 1} ^ {\prime} s _ {n 1} \left\{\left(H _ {n 1} + H _ {n 2}\right) \log n + \log \bar {L} _ {n 1} + \log \bar {L} _ {n 2} + \log p _ {n} \right\} \right. \\ \left. - k _ {n 1} ^ {\prime} s _ {n 1} \log \left(\frac {J _ {n 1}}{p _ {n} L _ {1} ^ {(1)}}\right) \right] \\ \prec \exp \{- S _ {0} ^ {(1)} n \epsilon_ {n} ^ {2} \}. \\ \end{array}
$$
Thus, Condition (b) in Lemma 3 is verified.
We now prove Theorem 2. The proof of the upper bound adapts the techniques in [46], and the proof of the lower bound employs a general mechanism for lower bounding the Bayes risk [40], which is tighter than the Fano's inequality [44].
Proof. Upper Bound. The Bayes risk can be written as
$$
\begin{array}{l} \mathfrak{R}_{\text{Bayes}} = \inf_{\overline{\mathbf{y}}}\left\{\sum_{\mathbf{y}\in [K]^{\bar{n}}} \hbar (\mathbf{y})\mathbb{E}(\mathcal{L}(\overline{\mathbf{y}},\mathbf{y})|\mathbf{y};\boldsymbol {\tau})\right\} \\ = \inf  _ {\overline {{\mathbf {y}}}} \left\{\sum_ {\mathbf {y} \in [ K ] ^ {\bar {n}}} \hbar (\mathbf {y}) \frac {1}{\bar {n}} \sum_ {i = 1} ^ {\bar {n}} \mathbb {P} \left(\bar {\mathbf {y}} _ {i} \neq \mathbf {y} _ {i} | \mathbf {y} _ {i}; \boldsymbol {\tau}\right) \right\} \\ = \inf  _ {\overline {{\mathbf {y}}}} \left\{\frac {1}{\bar {n}} \sum_ {i = 1} ^ {\bar {n}} \sum_ {\mathbf {y} \in [ K ] ^ {\bar {n}}} \hbar (\mathbf {y}) \mathbb {P} \left(\bar {y} _ {i} \neq y _ {i} | \mathrm {y} _ {i}; \boldsymbol {\tau}\right) \right\} \\ = \inf  _ {\overline {{\mathbf {y}}}} \left\{\frac {1}{\bar {n}} \sum_ {i = 1} ^ {\bar {n}} \sum_ {g = 1} ^ {K} \hbar_ {i, g} \mathbb {P} (\overline {{\mathbf {y}}} _ {i} \neq g | \mathbf {y} _ {i} = g; \boldsymbol {\tau}) \right\}. \\ \end{array}
$$
We first consider the situation  $g = 1$ :
$$
\mathbb {P} \left(\bar {y} _ {i} \neq 1 | y _ {i} = 1; \boldsymbol {\tau}\right) \leq \sum_ {g ^ {\prime} = 2} ^ {K} \mathbb {P} \left(\bar {y} _ {i} = g ^ {\prime} | y _ {i} = 1; \boldsymbol {\tau}\right).
$$
For each  $g^{\prime}\geq 2$ , considering the inference procedure of  $\overline{y}_i$ , we have that
$$
\begin{array}{l} \mathbb {P} \left(\bar {\mathbf {y}} _ {i} = g ^ {\prime} | \mathbf {y} _ {i} = 1; \boldsymbol {\tau}\right) \\ \leq \mathbb {P} \left(\frac {\hbar_ {i , g ^ {\prime}}}{\hbar_ {i , 1}} \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left(\frac {\tau_ {i , g ^ {\prime} l} ^ {(r)}}{\tau_ {i , 1 l} ^ {(r)}}\right) ^ {\mathbb {I} \left(\mathrm {y} _ {i} ^ {(r)} = l\right)} > \Omega \Bigg | \mathrm {y} _ {i} = 1; \boldsymbol {\tau}\right) \\ = \mathbb {P} \left(\left(\frac {\hbar_ {i , g ^ {\prime}}}{\Omega \hbar_ {i , 1}}\right) ^ {\lambda} \prod_ {r = 1} ^ {R} \prod_ {l = 1} ^ {K} \left(\frac {\tau_ {i , g ^ {\prime} l} ^ {(r)}}{\tau_ {i , 1 l} ^ {(r)}}\right) ^ {\lambda \mathbb {I} \left(\mathbf {y} _ {i} ^ {(r)} = l\right)} > 1 \mid \mathrm {y} _ {i} = 1; \boldsymbol {\tau}\right) \\ \leq \min  _ {\lambda \geq 0} \left(\frac {\hbar_ {i , g ^ {\prime}}}{\Omega \hbar_ {i , 1}}\right) ^ {\lambda} \prod_ {r = 1} ^ {R} \mathbb {E} \left\{\prod_ {l = 1} ^ {K} \left(\frac {\tau_ {i , g ^ {\prime} l} ^ {(r)}}{\tau_ {i , 1 l} ^ {(r)}}\right) ^ {\lambda \mathbb {I} (\mathrm {y} _ {i} ^ {(r)} = l)} \Bigg | \mathrm {y} _ {i} = 1; \boldsymbol {\tau} \right\} \\ = \min  _ {\lambda \geq 0} \left(\frac {\hbar_ {i , g ^ {\prime}}}{\Omega \hbar_ {i , 1}}\right) ^ {\lambda} \prod_ {r = 1} ^ {R} \sum_ {l = 1} ^ {K} \left(\tau_ {i, g ^ {\prime} l} ^ {(r)}\right) ^ {\lambda} \left(\tau_ {i, 1 l} ^ {(r)}\right) ^ {1 - \lambda} \\ \leq \exp \left\{- R \left\{- \min  _ {0 \leq \lambda \leq 1} \frac {1}{R} \left\{- \lambda \log \frac {\Omega \hbar_ {i , 1}}{\hbar_ {i , g ^ {\prime}}} + \sum_ {r = 1} ^ {R} \log \left\{\sum_ {l = 1} ^ {K} \left(\tau_ {i, g ^ {\prime} l} ^ {(r)}\right) ^ {\lambda} \left(\tau_ {i, 1 l} ^ {(r)}\right) ^ {1 - \lambda} \right\} \right\} \right\} \right\}, \\ \end{array}
$$
where the inequality in the third line follows from Markov's inequality. Denote  $C_{gg'}^{(i)} = -\min_{0 \leq \lambda \leq 1} \frac{1}{R} \left\{ -\lambda \log \frac{\Omega \hbar_{i,g}}{\hbar_{i,g'}} + \sum_{r=1}^{R} \log \left\{ \sum_{l=1}^{K} \left( \tau_{i,gl}' \right)^{1-\lambda} \left( \tau_{i,g'l}' \right)^{\lambda} \right\} \right\}$  and  $I_{\Omega}^{(g)}(\pmb{\tilde{h}}_i, \pmb{\tau}_i) = \min_{g' \neq g} C_{gg'}^{(i)}$ , we further obtain that
$$
\begin{array}{l} \mathbb {P} \left(\bar {\mathrm {y}} _ {i} = l \mid \mathrm {y} _ {i} = 1; \boldsymbol {\tau}\right) \leq \exp \left\{- R C _ {1 g ^ {\prime}} ^ {(i)} \right\} \\ \leq \exp \left\{- R \min  _ {g ^ {\prime} \neq 1} C _ {g g ^ {\prime}} ^ {(i)} \right\} = \exp \left\{- R I _ {\Omega} ^ {(1)} (\boldsymbol {\hbar} _ {i}, \boldsymbol {\tau} _ {i}) \right\}. \\ \end{array}
$$
Thus, we can obtain the upper bound of the Bayes risk:
$$
\mathfrak {R} _ {\text {B a y e s}} \leq \frac {K - 1}{\bar {n}} \sum_ {i = 1} ^ {\bar {n}} \sum_ {g = 1} ^ {K} \hbar_ {i, g} e ^ {- R I _ {\Omega} ^ {(g)} \left(\boldsymbol {\hbar} _ {i}, \boldsymbol {\tau} _ {i}\right)}.
$$
Lower Bound. Using Markov's inequality, we obtain that
$$
\begin{array}{l} \mathfrak {R} _ {\text {B a y e s}} = \inf  _ {\overline {{\mathbf {y}}}} \left\{\sum_ {\mathbf {y} \in [ K ] ^ {\bar {n}}} \hbar (\mathbf {y}) \mathbb {E} \left(\mathcal {L} (\overline {{\mathbf {y}}}, \mathbf {y}) | \mathbf {y}; \boldsymbol {\tau}\right) \right\} \\ \geq \inf  _ {\overline {{\mathbf {y}}}} \left\{\sum_ {\mathbf {y} \in [ K ] ^ {\bar {n}}} \hbar (\mathbf {y}) \frac {1}{\bar {n}} \mathbb {P} \left(\mathcal {L} (\overline {{\mathbf {y}}}, \mathbf {y}) > \frac {1}{\bar {n}} | \mathbf {y}; \tau\right) \right\} \\ = \frac {1}{\bar {n}} \inf  _ {\overline {{\mathbf {y}}}} \left\{\sum_ {\mathbf {y} \in [ K ] ^ {\bar {n}}} \hbar (\mathbf {y}) \mathbb {P} (\overline {{\mathbf {y}}} \neq \mathbf {y} | \mathbf {y}; \boldsymbol {\tau}) \right\} \\ = \frac {1}{\bar {n}} \inf  _ {\overline {{\mathbf {y}}}} \left[ \sum_ {\mathbf {y} \in [ K ] ^ {\bar {n}}} \hbar (\mathbf {y}) \mathbb {E} \left\{\mathbf {1} (\bar {\mathbf {y}} \neq \mathbf {y}) | \mathbf {y}; \tau \right\} \right] \\ \triangleq \frac {1}{\bar {n}} \overline {{\mathfrak {R}}} \\ \end{array}
$$
where the second step is due to Markov's inequality. Using Lemma 7, we can obtain that
$$
I _ {f} (\hbar , \mathcal {P}) \geq \phi_ {f} (\overline {{\mathfrak {R}}}, \mathfrak {R} _ {0}),
$$
where  $\mathcal{P}$  is the set of distributions of  $\{\tilde{\mathbf{y}}_i\}_{i=1}^{\bar{n}}$  induced by  $\mathbf{y}$ , and for 0-1 loss,  $\Re_0$  has the expression in [40]:  $\Re_0 = 1 - \sup_{\mathbf{a} \in [K]^{\bar{n}}} \hbar(B(\mathbf{a}))$  with  $B(\mathbf{a}) = \{\mathbf{y} \in [K]^{\bar{n}} : \mathbf{1}(\mathbf{y} \neq \mathbf{a}) = 0\}$ . Then, it can be easily verified that  $\Re_0 = 1 - \prod_{i=1}^{\bar{n}} \max_{k \in [K]} \hbar_{i,k}$ .
Let  $g(t) = \phi_f(t, \mathfrak{R}_0)$ , which is, by Lemma 6, non-increasing for  $t \in [0, \mathfrak{R}_0]$ , convex, and continuous in  $t$ . Using the convexity of  $g(t)$ , we obtain that for every  $t \in (0, \mathfrak{R}_0]$
$$
\phi_ {f} (\overline {{\mathfrak {R}}}, \mathfrak {R} _ {0}) \geq \phi_ {f} (t, \mathfrak {R} _ {0}) + \phi_ {f} ^ {\prime} (t -, \mathfrak {R} _ {0}) (\overline {{\mathfrak {R}}} - t),
$$
where  $\phi_f'(t - ,\Re_0)$  denotes the left derivative of  $x\mapsto \phi_f(x,\Re_0)$  at  $x = t$  . Then we can obtain that
$$
\overline {{\mathfrak {R}}} \geq t + \frac {\phi_ {f} (\overline {{\mathfrak {R}} , \mathfrak {R} _ {0}}) - \phi_ {f} (t , \mathfrak {R} _ {0})}{\phi_ {f} ^ {\prime} (t -, \mathfrak {R} _ {0})} \geq t + \frac {I _ {f} (\hbar , \mathcal {P}) - \phi_ {f} (t , \mathfrak {R} _ {0})}{\phi_ {f} ^ {\prime} (t -, \mathfrak {R} _ {0})},
$$
where the inequalities come from the fact that  $I(\hbar, \mathcal{P}) \geq \phi_f(\overline{\Re}, \Re_0)$  by Lemma 7 and that  $\phi_f'(t -, \Re_0) \leq 0$  due to the non-increasing function  $g(t)$  over  $t \in [0, \Re_0]$ . By taking  $f(t) = t \log t$  and  $t = \frac{\Re_0}{1 + \Re_0}$ , we obtain that
$$
\overline {{\mathfrak {R}}} \geq 1 + \frac {I (\hbar , \mathcal {P}) + \log (1 + \mathfrak {R} _ {0})}{\log (1 - \mathfrak {R} _ {0})},
$$
where  $I(\hbar, \mathcal{P})$  is the mutual information of  $\hbar$  and  $\mathcal{P}$ .
Let  $Y$  and  $Z$  denote two independent random variables such that  $Y \sim \hbar$  and  $Z \sim \mathcal{P}$ , and let  $D_{KL}(\cdot ||\cdot)$  denote the KL divergence of the associated distributions. For ease of notation, we use  $P(Y), P(Z), P(Y,Z)$  and  $P(Z|\hat{Y})$  to denote the  $\hbar$ ,  $\mathcal{P}$ , the joint distribution of  $Y$  and  $Z$ , and the conditional distribution of  $Z$ , given  $Y$ . Then  $I(\hbar, \mathcal{P})$  can be evaluated as follows [43]:
$$
\begin{array}{l} I (\hbar , \mathcal {P}) = I (Y; Z) \\ = D _ {K L} (P (Y, Z) \| P (Y) P (Z)) \\ = \sum_ {Y \in [ K ] ^ {\bar {n}}, Z \in [ K ] ^ {R \bar {n}}} P (Y) P (Z | Y) \log \left\{\frac {P (Y) P (Z | Y)}{P (Y) P (Z)} \right\} \\ = \sum_ {Y \in [ K ] ^ {\bar {n}}} P (Y) D _ {K L} \left(P (Z | Y) \| P (Z)\right) \\ = \sum_ {Y \in [ K ] ^ {\bar {n}}} P (Y) D _ {K L} (P (Z | Y) \| \sum_ {Y ^ {\prime} \in [ K ] ^ {\bar {n}}} P (Z | Y ^ {\prime}) P (Y ^ {\prime})). \\ \end{array}
$$
Using the log sum inequality in Lemma 4, we further obtain that
$$
\begin{array}{l} I (\hbar , \mathcal {P}) = \sum_ {Y \in [ K ] ^ {\bar {n}}} P (Y) \sum_ {Y ^ {\prime} \in [ K ] ^ {\bar {n}}} P \left(Y ^ {\prime}\right) D _ {K L} \left(P (Z | Y) \| P (Z | Y ^ {\prime})\right) \\ = \sum_ {Y \in [ K ] ^ {\bar {n}}} \sum_ {Y ^ {\prime} \in [ K ] ^ {\bar {n}}} P (Y) P \left(Y ^ {\prime}\right) \left\{\sum_ {i = 1} ^ {\bar {n}} \sum_ {r = 1} ^ {R} D _ {K L} \left(P \left(Z _ {i} ^ {(r)} \mid Y\right) \| P \left(Z _ {i} ^ {(r)} \mid Y ^ {\prime}\right)\right) \right\} \\ = \sum_ {Y \in [ K ] ^ {\bar {n}}} \sum_ {Y ^ {\prime} \in [ K ] ^ {\bar {n}}} \sum_ {i = 1} ^ {\bar {n}} \sum_ {r = 1} ^ {R} P (Y) P (Y ^ {\prime}) D _ {K L} \left(\tau_ {i, Y _ {i} *} ^ {(r)} \| \tau_ {i, Y _ {i} ^ {\prime} *} ^ {(r)}\right) \\ = \sum_ {i = 1} ^ {\bar {n}} \sum_ {r = 1} ^ {R} \sum_ {g = 1} ^ {K} \sum_ {g ^ {\prime} = 1} ^ {K} P \left(Y _ {i} = g\right) P \left(Y _ {i} ^ {\prime} = g ^ {\prime}\right) D _ {\kappa L} \left(\tau_ {i, g *} ^ {(r)} \| \tau_ {i, g ^ {\prime} *} ^ {(r)}\right) \\ = \sum_ {i = 1} ^ {\bar {n}} \sum_ {r = 1} ^ {R} \sum_ {g = 1} ^ {K} \sum_ {g ^ {\prime} = 1} ^ {K} \hbar_ {i, g} \hbar_ {i, g ^ {\prime}} D _ {K L} \left(\tau_ {i, g *} ^ {(r)} \| \tau_ {i, g ^ {\prime} *} ^ {(r)}\right) \\ \triangleq \bar {n} \bar {D} _ {K L} (\boldsymbol {\hbar}, \tau). \\ \end{array}
$$
Combining the discussion above, we obtain that
$$
\mathfrak {R} _ {\text {B a y e s}} \geq \frac {1}{\bar {n}} \left[ 1 - \frac {\overline {{D}} _ {K L} (\boldsymbol {\hbar} , \boldsymbol {\tau}) + \frac {1}{\bar {n}} \log (2 - \prod_ {i = 1} ^ {\bar {n}} \max  _ {k \in [ K ]} \hbar_ {i , k})}{\left\{\sum_ {i = 1} ^ {\bar {n}} \log (\max  _ {k \in [ K ]} \hbar_ {i , k}) \right\} / \bar {n}} \right].
$$
B.4 Proof of Corollary 3
Proof. For  $k \in [K]$ ,  $r \in [R]$ , and consider the following set:
$$
\mathcal {S} ^ {c} = \left\{\boldsymbol {\theta}: \cup_ {k \in [ K ], r \in [ R ]} \left\{\max  _ {\tilde {\mathbf {y}} \in [ K ], \mathbf {x} \in \overline {{\mathcal {D}}} _ {\mathbf {x}}} \left| f ^ {(k, r)} (\tilde {\mathbf {y}} | \boldsymbol {\omega} ^ {(k, r)}) - f _ {0} ^ {(k, r)} (\tilde {\mathbf {y}} | \boldsymbol {\omega} _ {0} ^ {(k, r)}) \right| > M _ {n} \epsilon_ {n} \right\} \right\},
$$
where  $\theta, \epsilon_{n}$  and  $M_{n}$  satisfy the conditions in Theorem 1,  $\omega^{(k,r)}$  and  $\omega_0^{(k,r)}$  depend on  $\mathbf{x}$ , and  $\overline{\mathcal{D}}_{\mathbf{x}}$  represents the set of observations of the instances in  $\overline{\mathcal{D}}$ . Then, according to Theorem 1 and using the union bound, we have that
$$
\begin{array}{l} \Pi (\mathcal {S} ^ {c} | \overline {{\mathcal {D}}} _ {0}) \leq \sum_ {k \in [ K ], r \in [ R ]} \Pi \left\{\boldsymbol {\theta}: \max  _ {\tilde {\mathbf {y}} \in [ K ], \mathbf {x} \in \overline {{\mathcal {D}}} _ {\mathbf {x}}} \left| f ^ {(k, r)} (\tilde {\mathbf {y}} | \boldsymbol {\omega} ^ {(k, r)}) - f _ {0} ^ {(k, r)} (\tilde {\mathbf {y}} | \boldsymbol {\omega} _ {0} ^ {(k, r)}) \right| > M _ {n} \epsilon_ {n} \middle | \overline {{\mathcal {D}}} _ {0} \right\} \\ \leq \sum_ {k \in [ K ], r \in [ R ]} \Pi \left(\boldsymbol {\theta}: \| f ^ {(k, r)} - f _ {0} ^ {(k, r)}) \| _ {\mathrm {T V}} > U _ {9} M _ {n} \epsilon_ {n} \Big | \overline {{\mathcal {D}}} _ {0}\right) \\ \leq \sum_ {k \in [ K ], r \in [ R ]} \Pi \left(\boldsymbol {\theta}: d \left(f ^ {(k, r)}, f _ {0} ^ {(k, r)}\right) > \sqrt {2} U _ {9} M _ {n} \epsilon_ {n} \mid \overline {{\mathcal {D}}} _ {0}\right) \longrightarrow 0 \\ \end{array}
$$
in  $\mathbb{P}_0^n$  probability as  $n\to \infty$ , where  $\| \cdot \|_{\mathrm{TV}}$  denotes the total variation distance of the associated distributions as defined in Example 1, and  $U_{9}$  is a positive constant. Here, the second inequality is due to the definition of the total variation distance of discrete distributions and the fact that  $f(\mathbf{x}) > \varsigma_2$  for  $\mathbf{x}\in \overline{\mathcal{D}}_{\mathbf{x}}$ , and the third inequality follows the (i) in Lemma 5. Thus,  $\Pi (\mathcal{S}|\overline{\mathcal{D}}_0)\longrightarrow 1$  in  $\mathbb{P}_0^n$  probability as  $n\to \infty$ , where  $\mathcal{S}$  can be written as
$$
\mathcal {S} = \left\{\boldsymbol {\theta}: \max  _ {l \in [ K ], \mathbf {x} \in \overline {{\mathcal {D}}} _ {\mathbf {x}}} \left| \overline {{\tau}} _ {k l} ^ {(r)} (\mathbf {x}) - \tau_ {k l} ^ {(r)} (\mathbf {x}) \right| \leq M _ {n} \epsilon_ {n}, k \in [ K ], r \in [ R ] \right\}.
$$
We now consider the Bayes risk  $\Re_{\mathrm{Bayes}}$ , which, according to the proof of Theorem 2, can be written as  $\Re_{\mathrm{Bayes}} = \inf_{\overline{\mathbf{y}}}\left\{\frac{1}{\bar{n}}\sum_{i = 1}^{\bar{n}}\sum_{g = 1}^{K}\hbar_{i,g}\mathbb{P}(\overline{\mathbf{y}}_i\neq g|\mathbf{y}_i = g;\pmb {\tau})\right\}$ . We first consider  $g = 1$ :