SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 2 ([37], Lemma S2). Consider a sparse DNN  $\phi(\pmb{\theta},\mathbf{x})$  with indicator vector  $\gamma$  satisfying the conditions in Lemma 1, and a DNN  $\phi(\bar{\pmb{\theta}},\mathbf{x})$  with  $\bar{\pmb{\theta}} \in \mathcal{A}$ , where  $\mathcal{A}$  is defined as
$$
\mathcal {A} = \{\tilde {\pmb {\theta}} = (\tilde {\theta} _ {1}, \dots , \tilde {\theta} _ {q}) ^ {\top}: | \theta_ {j} - \tilde {\theta} _ {j} | <   \delta_ {1} \mathrm {f o r} j \in \pmb {\gamma} \mathrm {a n d} | \theta_ {j} - \tilde {\theta} _ {j} | <   \delta_ {2} \mathrm {f o r} j \notin \pmb {\gamma} \}
$$
for given  $\delta_1 > 0$  and  $\delta_{2} > 0$ . Then
$$
\begin{array}{l} \max  _ {| \mathbf {x} | _ {\infty} \leq 1} \| \phi (\boldsymbol {\theta}, \mathbf {x}) - \phi (\bar {\boldsymbol {\theta}}, \mathbf {x}) \| _ {1} \leq \delta_ {1} H _ {n} (E _ {n} + \delta_ {1}) ^ {H _ {n} - 1} \prod_ {h = 1} ^ {H _ {n}} r _ {h} \\ + \delta_ {2} \left(p _ {n} L _ {1} + \sum_ {h = 1} ^ {H _ {n}} L _ {h}\right) \prod_ {h = 1} ^ {H _ {n}} \left\{\left(E _ {n} + \delta_ {1}\right) r _ {h} + \delta_ {2} L _ {h} \right\}. \\ \end{array}
$$
B.1.4 A useful lemma
Assume  $\mathcal{P}_n$  is a sequence of sets of probability densities and  $\mathcal{P}_n^c$  denotes the complement of  $\mathcal{P}_n$  for each  $n$ . Let  $\epsilon_n$  be a sequence of positive numbers. An  $\epsilon_n$ -cover of  $\mathcal{P}_n$  with respect to (w.r.t.)
distance  $d$  is a set  $\{f_1,\dots,f_k\} \subset \mathcal{P}_n$  such that for each  $f\in \mathcal{P}_n$ , there exists  $j\in \{1,\ldots ,k\}$  such that  $d(f,f_j)\leq \epsilon_n$ . The  $\epsilon_{n}$ -covering number is the cardinality of the smallest  $\epsilon_{n}$ -cover [72]. Let  $N(\epsilon_{n},\mathcal{P}_{n},d)$  denote the  $\epsilon_{n}$ -covering number of  $\mathcal{P}_n$  w.r.t. the distance  $d$ .
Consider a vector of independently (not necessarily identically) distributed observations  $\mathcal{D}^n$ , where the  $i$ th component is generated from distribution  $\mathbb{P}_{\theta,i}$  with the density  $p_{\theta,i}$  relative to a  $\sigma$ -finite measure  $\nu_i$  on  $(\mathcal{X}_i, \mathcal{A}_i)$  for  $i \in [n]$ , and  $\pmb{\theta}$  is the vector of parameters in the parameter space  $\Theta$ . We define  $\mathbb{P}_{\pmb{\theta}}^n$  to be the product measure  $\bigotimes_{i=1}^{n} \mathbb{P}_{\pmb{\theta},i}$  on the corresponding product measurable space  $\bigotimes_{i=1}^{n} (\mathcal{X}_i, \mathcal{A}_i)$ . Assume that  $\mathcal{D}^n$  is generated from the true distribution  $\mathbb{P}_{\pmb{\theta}_0}^n$ . We define the square of the semimetric  $d_n$  as in [73]:
$$
d _ {n} ^ {2} (\boldsymbol {\theta}, \boldsymbol {\theta} _ {0}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \int \left(\sqrt {p _ {\boldsymbol {\theta} , i}} - \sqrt {p _ {\boldsymbol {\theta} _ {0} , i}}\right) ^ {2} d \nu_ {i}, \tag {16}
$$
which can be seen as the average of the squares of the Hellinger distances. For  $\epsilon > 0$ , we define the  $\epsilon$ -neighborhood around  $\theta_0$ :
$$
G _ {n} ^ {*} (\boldsymbol {\theta} _ {0}, \epsilon) = \left\{\boldsymbol {\theta}: \frac {1}{n} \sum_ {i = 1} ^ {n} K _ {i} (\boldsymbol {\theta}, \boldsymbol {\theta} _ {0}) \leq \epsilon^ {2}; \frac {1}{n} \sum_ {i = 1} ^ {n} V _ {i} (\boldsymbol {\theta}, \boldsymbol {\theta} _ {0}) \leq \epsilon^ {2} \right\}, \tag {17}
$$
where  $K_{i}(\pmb {\theta},\pmb{\theta}_{0}) = K(\mathbb{P}_{\pmb{\theta},i},\mathbb{P}_{\pmb{\theta}_{0},i})$  and  $V_{i}(\pmb {\theta},\pmb{\theta}_{0}) = V(\mathbb{P}_{\pmb{\theta},i},\mathbb{P}_{\pmb{\theta}_{0},i})$ , defined in Definition 1.
Let  $\Pi(\cdot)$  denote the prior probability measure on  $\theta$ , and let  $\Pi(\cdot|\mathcal{D}^n)$  represent the associated posterior measure given the data  $\mathcal{D}^n$ . For ease of exposition, we put  $\Pi(\Theta^*)$  for  $\Pi(\{\theta \in \Theta^*\})$  for any  $\Theta^* \subset \Theta$ . The following lemma is modified from Theorem 4 in [73] and will be used in the proof of Theorem 1.
Lemma 3. Suppose that for a sequence of sets  $\Theta_{n} \subset \Theta$  and for a sequence of positive numbers  $\{\epsilon_{n}\}_{n=1}^{\infty}$  such that  $\epsilon_{n} \to 0$  as  $n \to \infty$  and  $n\epsilon_{n}^{2}$  is bounded away from zero, the following conditions hold for large enough  $n$ :
(a)  $\sup_{\epsilon >\epsilon_n}\log N(\epsilon /36,\{\pmb {\theta}\in \Theta_n:d_n(\pmb {\theta},\pmb {\theta}_0) <   \epsilon \} ,d_n)\leq n\epsilon_n^2;$
(b)  $\Pi (\Theta \backslash \Theta_n) = o(\exp \{-(r + 2)n\epsilon_n^2\})$
(c)  $\Pi (G_n^* (\pmb {\theta}_0,\epsilon_n))\geq \exp (-rn\epsilon_n^2)$
for some constant  $r > 0$ . Then for any  $M_n > 0$ $\mathbb{P}_{\pmb{\theta}_0}^n \Pi(\pmb{\theta}: d_n(\pmb{\theta}, \pmb{\theta}_0) \geq M_n \epsilon_n | \mathcal{D}^n) \to 0$  as  $M_n \to \infty$ .
B.1.5 Useful information-theoretical definitions and lemmas
We give some useful information-theoretical definitions and lemmas in this subsection, which will be used in the proof of Theorem 2 and Corollary 3.
Lemma 4 (Log sum inequality, [45]). For positive numbers,  $a_1, \ldots, a_n$  and  $b_1, \ldots, b_n$ ,
$$
\sum_ {i = 1} ^ {n} a _ {i} \log \left(\frac {a _ {i}}{b _ {i}}\right) \geq \left(\sum_ {i = 1} ^ {n} a _ {i}\right) \log \left(\frac {\sum_ {i = 1} ^ {n} a _ {i}}{\sum_ {i = 1} ^ {n} b _ {i}}\right),
$$
with equality if and only if  $\frac{a_i}{b_i}$  equals for  $i = 1, \dots, n$ .
Definition 2 ([74, 75]). Let  $P$  and  $Q$  be probability distributions on the set  $\mathcal{X}$ , and let  $f: \mathbb{R}_+ \longrightarrow \mathbb{R}$  be a convex function satisfying  $f(1) = 0$ . Without loss of generality, assume that  $P$  and  $Q$  are absolutely continuous with respect to the base measure  $\mu$ . The  $f$ -divergence between  $P$  and  $Q$  is then defined as
$$
D _ {f} (P \| Q) := \int_ {\mathcal {X}} q (x) f \left(\frac {p (x)}{q (x)}\right) d \mu (x) + f ^ {\prime} (\infty) P \{q = 0 \},
$$
where  $p$  and  $q$  are the densities of  $P$  and  $Q$  with respect to the measure  $\mu$ , respectively, and  $f'(\infty)$  represents  $\lim_{x \to \infty} f(x) / x$ .
Example 1. By taking different  $f$  functions, we provide some popular examples of  $f$ -divergences.
- Kullback-Leibler (KL) divergence: taking  $f(t) = t\log t$  gives  $D_{f}(P\| Q)\triangleq D_{\mathrm{KL}}(P\| Q) = \int p\log (p / q)d\mu$ , which is also denoted  $d_0(p,q)$  in Definition 1.
- The total variation distance: taking  $f(t) = \frac{1}{2} |t - 1|$  yields  $D_{f}(P\| Q)\triangleq \| P - Q\|_{\mathrm{TV}} = \frac{1}{2}\int \left|\frac{p}{q} -1\right|qd\mu = \sup_{A\subset \mathcal{X}}|P(A) - Q(A)|$ , which is also denoted  $d_0(p,q)$  in Definition 1.
- The Hellinger distance: taking  $f(t) = (\sqrt{t} - 1)^2 = t - 2\sqrt{t} + 1$  leads to the squared Hellinger distance  $D_f(P \| Q) \triangleq H^2(P \| Q) = \int (\sqrt{p} - \sqrt{q})^2 d\mu$ , which is also denoted  $d^2(p, q)$  in Definition 1.
- The  $\chi^2$ -divergence: taking  $f(t) = \frac{1}{2} (t - 1)^2$  produces the  $\chi^2$ -divergence  $D_f(P\| Q)\triangleq \chi^2 (P\| Q) = \frac{1}{2}\int (\frac{p}{q} -1)^2 d\mu .$
Lemma 5 ([75]). For the quantities defined in Example 1, the following relationships hold:
(i) For the Hellinger distance,
$$
\frac {1}{2} H ^ {2} (P, Q) \leq \| P - Q \| _ {T V} \leq H (P, Q) \sqrt {1 - H ^ {2} (P , Q) / 4}.
$$
(ii) Pinsker's inequality: for any distributions  $P$  and  $Q$ ,
$$
\| P - Q \| _ {T V} ^ {2} \leq \frac {1}{2} D _ {K L} (P \| Q).
$$
Definition 3 ([41, 40]). Let  $\mathcal{P} = \{P_{\theta} : \theta \in \Theta\}$  be a family of probability measures on a space  $\mathcal{X}$ , indexed by  $\theta \in \Theta$ , and let  $\omega$  be a probability measure on  $\Theta$ . For each  $f$  satisfying the conditions in Definition 2, the  $f$ -informativity,  $I_{f}(\omega, \mathcal{P})$ , is defined as
$$
I _ {f} (\omega , \mathcal {P}) := \inf  _ {Q} \int D _ {f} (P _ {\theta} \| Q) \omega (d \theta),
$$
where the infimum is taken over all possible probability measures  $Q$  on  $\mathcal{X}$ . In particular, when  $f(t) = t\log t$ , the  $f$ -informativity is equal to the mutual information and is denoted by  $I(\omega, \mathcal{P})$ .
For each  $f$  satisfying the conditions in Definition 2, let  $\phi_f:[0,1]^2\to \mathbb{R}$  be the function defined as follows: for  $a,b\in [0,1]^2$ ,  $\phi_f(a,b)$  is the  $f$ -divergence between the two probability measures  $P$  and  $Q$  on  $\{0,1\}$  given by  $P\{1\} = a$  and  $Q\{1\} = b$ . Then,  $\phi_f(a,b)$  has the following expression:
$$
\phi_ {f} (a, b) = \left\{ \begin{array}{l l} b f \left(\frac {a}{b}\right) + (1 - b) f \left(\frac {1 - a}{1 - b}\right) & \text {f o r} 0 <   b <   1; \\ f (1 - a) + a f ^ {\prime} (\infty) & \text {f o r} b = 0; \\ f (a) + (1 - a) f ^ {\prime} (\infty) & \text {f o r} b = 1. \end{array} \right. \tag {18}
$$
The following lemma implies monotonicity and convexity properties of  $\phi_f$  by taking it as a univariate function with one argument of  $\phi_f$  fixed at a given value.
Lemma 6 ([40]). For each  $f$  satisfying the conditions in Definition 2, consider  $\phi_f$  defined in (18). Then
(a). for every fixed  $b > 0$ , the map  $g(a): a \mapsto \phi_f(a, b)$  is non-increasing for  $a \in [0, b]$  and  $g(a)$  is convex and continuous in  $a$ ;
(b). for every fixed  $a < 1$ , the map  $h(b): b \mapsto \phi_f(a, b)$  is non-decreasing for  $b \in [a, 1]$ .
Lemma 7 ([40]). Let  $\mathcal{P} = \{P_{\theta} : \theta \in \Theta\}$  be a family of probability measures on a space  $\mathcal{X}$  and let  $\omega$  be a probability measure on  $\Theta$ . Suppose that the loss function  $\mathcal{L}$  is zero-one valued. Define the Bayes error as  $\Re_{Bayes}(\omega) = \inf_{\mathfrak{d}} \int_{\Theta} \mathbb{E}\mathcal{L}(\theta, \mathfrak{d}(X))\omega(d\theta)$  with  $\mathfrak{d} : \mathcal{X} \to \Theta$  denoting a mapping from the sample space to the parameter space. For any  $f$  satisfying the conditions in Definition 2, we have that
$$
I _ {f} (\omega , \mathcal {P}) \geq \phi_ {f} \left(\Re_ {\text {B a y e s}} (\omega), \Re_ {0}\right),
$$
where  $\phi_f$  is given by (18), and  $\Re_0$  is defined as  $\Re_0 \triangleq \inf_{a \in \Theta} \int_{\Theta} \mathcal{L}(\theta, a) \omega(d\theta)$ .
We now establish the proof of Theorem 1 by checking the three conditions in Lemma 3 established for the sparse Bayesian deep learning framework. We first provide a useful result on the discrepancy measures for generalized linear models in Section B.2.1, and then verify the three conditions in Sections B.2.2-B.2.4 following the proof techniques in [37, 67].
B.2.1 Discrepancy measures for generalized linear models
Paired variables in the set of anchor points  $\overline{\mathcal{D}}_0 = \overline{\mathcal{D}}_{0,1} \cup \ldots \cup \overline{\mathcal{D}}_{0,K}$  in Section 3.1 can be seen as independently (not necessarily identically) distributed, and we let  $\mathcal{D}^n$  denote  $\overline{\mathcal{D}}_0$  in the following derivations to emphasize its dependence on the sample size  $n$ . For  $\{\mathbf{x}, \tilde{\mathbf{y}}\} \in \overline{\mathcal{D}}_{0,k}$  with  $k \in [K]$ , we write the  $r$ th element of  $\tilde{\mathbf{y}}, \tilde{\mathbf{y}}^{(r)}$ , in the 1-of- $K$  fashion, i.e., only the  $j$ th element is equal to 1 while others are all 0 if  $\tilde{\mathbf{y}}^{(r)}$  is the  $j$ th class, and then, the conditional probability density/mass function of  $\tilde{\mathbf{y}}^{(r)}$  induced by  $\boldsymbol{\theta}$  is given by
$$
f _ {\boldsymbol {\theta}} ^ {(k, r)} \left(\tilde {\mathbf {y}} ^ {(r)}\right) = G ^ {\top} \left(\boldsymbol {\omega} ^ {(k, r)}\right) \tilde {\mathbf {y}} ^ {(r)} = \left(G _ {1} \left(\boldsymbol {\omega} ^ {(k, r)}\right), \dots , \left(G _ {K} \left(\boldsymbol {\omega} ^ {(k, r)}\right)\right) \tilde {\mathbf {y}} ^ {(r)} \right. \tag {19}
$$
for  $r \in [R]$ . We denote the joint conditional probability density/mass function of  $\tilde{\mathbf{y}}$  as  $f_{\theta}^{(k)}(\tilde{\mathbf{y}})$ , given by
$$
f _ {\boldsymbol {\theta}} ^ {(k)} (\tilde {\mathbf {y}}) = \prod_ {r = 1} ^ {R} f _ {\boldsymbol {\theta}} ^ {(k, r)} (\tilde {\mathbf {y}} ^ {(r)}), \tag {20}
$$
and in contrast, we denote the underlying true conditional probability density/mass functions for  $\tilde{\mathbf{y}}^{(r)}$  and  $\tilde{\mathbf{y}}$  as  $f_0^{(k,r)}$  and  $f_0^{(k)}$ , respectively.
Result 1. If Conditions A.4 in Section A.2 are satisfied, then, for any  $k \in [K]$ ,
$$
K \left(f _ {\boldsymbol {\theta}} ^ {(k)}, f _ {0} ^ {(k)}\right) \leq C _ {K} \sum_ {r = 1} ^ {R} \mathbb {E} _ {\mathbf {x}} \| \boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)} \| _ {1}; \tag {21}
$$
$$
V \left(f _ {\boldsymbol {\theta}} ^ {(k)}, f _ {0} ^ {(k)}\right) \leq C _ {V} \sum_ {r = 1} ^ {R} \mathbb {E} _ {\mathbf {x}} \| \boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)} \| _ {1}, \tag {22}
$$
for some positive constants  $C_K$  and  $C_V$  in the neighbourhood of  $\pmb{\theta}_0^4$ .
Proof. By Definition (1) (iii) and (20), we have that
$$
\begin{array}{l} K (f _ {\boldsymbol {\theta}} ^ {(k)}, f _ {0} ^ {(k)}) = \int \int f _ {0} ^ {(k)} \log \left(\frac {f _ {0} ^ {(k)}}{f _ {\boldsymbol {\theta}} ^ {(k)}}\right) \nu_ {2} (d \tilde {\mathbf {y}}) \nu_ {1} (d \mathbf {x}) \\ = \sum_ {r = 1} ^ {R} \int \int f _ {0} ^ {(k)} \log \left(\frac {f _ {0} ^ {(k , r)}}{f _ {\boldsymbol {\theta}} ^ {(k , r)}}\right) \nu_ {2} (d \tilde {\mathbf {y}}) \nu_ {1} (d \mathbf {x}) \\ = \sum_ {r = 1} ^ {R} \int \int f _ {0} ^ {(k, r)} \log \left(\frac {f _ {0} ^ {(k , r)}}{f _ {\boldsymbol {\theta}} ^ {(k , r)}}\right) \nu_ {2, r} (d \tilde {\mathbf {y}} ^ {(r)}) \nu_ {1} (d \mathbf {x}). \\ \end{array}
$$
These two distances can also be upper bounded by the  $L_{2}$  norm of  $\omega_{2} - \omega_{1}$  since the  $L_{1}$  norm and  $L_{2}$  norm are equivalent on  $\mathbb{R}^p$  in the sense that  $|\boldsymbol {\omega}|_2\leq |\boldsymbol {\omega}|_1\leq \sqrt{p} |\boldsymbol {\omega}|_2$
According to (19) and by taking the Taylor's expansion of  $\log \left(f_{\pmb{\theta}}^{k,r}\right)$  at  $\omega_0^{(k,r)}$ , we obtain that
$$
\begin{array}{l} \int f _ {0} ^ {(k, r)} \log \left(\frac {f _ {0} ^ {(k , r)}}{f _ {\pmb {\theta}} ^ {(k , r)}}\right) \nu_ {2, r} (d \tilde {\mathbf {y}} ^ {(r)}) \\ = - \int f _ {0} ^ {(k, r)} \left[ \left(\frac {\partial \log f _ {\boldsymbol {\theta}} ^ {(k , r)}}{\partial \omega} \Big | _ {\boldsymbol {\omega} = \bar {\boldsymbol {\omega}}}\right) ^ {\top} (\boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)}) \right] \nu_ {2, r} (d \tilde {\mathbf {y}} ^ {(r)}) \\ = - \int f _ {0} ^ {(k, r)} \left[ \left(\frac {\partial \log (G (\boldsymbol {\omega}) ^ {\top} \tilde {\mathbf {y}} ^ {(r)})}{\partial \boldsymbol {\omega}} \Big | _ {\boldsymbol {\omega} = \overline {{\boldsymbol {\omega}}}}\right) ^ {\top} (\boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)}) \right] \nu_ {2, r} (d \tilde {\mathbf {y}} ^ {(r)}) \\ = - \int f _ {0} ^ {(k, r)} \left[ \left(\frac {1}{G (\bar {\omega}) ^ {\top}} \frac {\partial G (\bar {\omega}) ^ {\top}}{\partial \boldsymbol {\omega}}\right) ^ {\top} (\boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)}) \right] \nu_ {2, r} (d \tilde {\mathbf {y}} ^ {(r)}), \\ \end{array}
$$
where  $\bar{\omega} = (\bar{\omega}_1,\dots,\bar{\omega}_K)^\top$  is between  $\pmb{\omega}^{(k,r)}$  and  $\pmb{\omega}_{0}^{(k,r)}$  and
$$
\frac {\partial G (\bar {\omega}) ^ {\top}}{\partial \boldsymbol {\omega}} = \left(\frac {\partial G _ {1} (\bar {\omega})}{\partial \bar {\omega}},..., \frac {\partial G _ {K} (\bar {\omega})}{\partial \bar {\omega}}\right) = \left( \begin{array}{c c c} \frac {\partial G _ {1} (\bar {\omega})}{\partial \bar {\omega} _ {1}} & \dots & \frac {\partial G _ {K} (\bar {\omega})}{\partial \bar {\omega} _ {1}} \\ \vdots & & \vdots \\ \frac {\partial G _ {1} (\bar {\omega})}{\partial \bar {\omega} _ {K}} & \dots & \frac {\partial G _ {K} (\bar {\omega})}{\partial \bar {\omega} _ {K}} \end{array} \right).
$$
Since  $\tilde{y}^{(r)}$  is discretely distributed, according to (19) and Condition A.4.1, we further obtain that
$$
\begin{array}{l} \int f _ {0} ^ {(k, r)} \log \left(\frac {f _ {0} ^ {(k , r)}}{f _ {\boldsymbol {\theta}} ^ {(k , r)}}\right) \nu_ {2, r} (d \tilde {\mathbf {y}} ^ {(r)}) \\ = - \sum_ {j = 1} ^ {K} \frac {G _ {j} \left(\omega_ {0} ^ {(k , r)}\right)}{G _ {j} (\bar {\omega})} \left\{\frac {\partial G _ {j} (\bar {\omega})}{\partial \omega} \right\} ^ {\top} \left(\omega^ {(k, r)} - \omega_ {0} ^ {(k, r)}\right) \\ \leq \frac {C _ {3} K}{1 - \epsilon_ {2}} \left\| \boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)} \right\| _ {1}. \\ \end{array}
$$
Thus, the proof of inequality (21) is completed.
Similarly, we write  $V(f_{\pmb{\theta}}^{(k)}, f_0^{(k)})$  as:
$$
\begin{array}{l} V (f _ {\pmb {\theta}} ^ {(k)}, f _ {0} ^ {(k)}) = \int \int f _ {0} ^ {(k)} \left\{\log \left(\frac {f _ {0} ^ {(k)}}{f _ {\pmb {\theta}} ^ {(k)}}\right) \right\} ^ {2} \nu_ {2} (d \tilde {\mathbf {y}}) \nu_ {1} (d \mathbf {x}) \\ = \int \int f _ {0} ^ {(k)} \left\{\sum_ {r = 1} ^ {R} \log \left(\frac {f _ {0} ^ {(k , r)}}{f _ {\boldsymbol {\theta}} ^ {(k , r)}}\right) \right\} ^ {2} \nu_ {2} (d \tilde {\mathbf {y}}) \nu_ {1} (d \mathbf {x}) \\ = \sum_ {r = 1} ^ {R} \int \int f _ {0} ^ {(k, r)} \left\{\log \left(\frac {f _ {0} ^ {(k , r)}}{f _ {\boldsymbol {\theta}} ^ {(k , r)}}\right) \right\} ^ {2} \nu_ {2, r} (d \tilde {\mathbf {y}} ^ {(r)}) \nu_ {1} (d \mathbf {x}) \\ + \sum_ {r _ {1} \neq r _ {2}} \int \left[ \int f _ {0} ^ {(k, r _ {1})} \left\{\log \left(\frac {f _ {0} ^ {(k , r _ {1})}}{f _ {\boldsymbol {\theta}} ^ {(k , r _ {1})}}\right) \right\} \nu_ {2, r _ {1}} \left(d \tilde {\mathbf {y}} ^ {(r _ {1})}\right) \right] \\ \cdot \left[ \int f _ {0} ^ {(k, r _ {2})} \left\{\log \left(\frac {f _ {0} ^ {(k , r _ {2})}}{f _ {\theta} ^ {(k , r _ {2})}}\right) \right\} \nu_ {2, r _ {2}} (d \tilde {\mathrm {y}} ^ {(r _ {2})}) \right] \nu_ {1} (d \mathbf {x}), \\ \end{array}
$$
where the second term can be upper bounded by
$$
\left\{\frac {C _ {3} K}{1 - \epsilon_ {2}} \right\} ^ {2} \sum_ {r _ {1} \neq r _ {2}} \mathbb {E} _ {\mathbf {x}} \| \boldsymbol {\omega} ^ {(k, r _ {1})} - \boldsymbol {\omega} _ {0} ^ {(k, r _ {1})} \| _ {1} \| \boldsymbol {\omega} ^ {(k, r _ {2})} - \boldsymbol {\omega} _ {0} ^ {(k, r _ {2})} \| _ {1},
$$
and for the first term, we have
$$
\int f _ {0} ^ {(k, r)} \left\{\log \left(\frac {f _ {0} ^ {(k , r)}}{f _ {\pmb {\theta}} ^ {(k , r)}}\right) \right\} ^ {2} \nu_ {2, r} (d \tilde {\mathbf {y}} ^ {(r)}) = \sum_ {j = 1} ^ {K} G _ {j} (\pmb {\omega} _ {0} ^ {(k, r)}) \left\{\log \left(\frac {G _ {j} (\pmb {\omega} _ {0} ^ {(k , r)})}{G _ {j} (\pmb {\omega} ^ {(k , r)})}\right) \right\} ^ {2}.
$$
If  $G_{j}(\pmb{\omega}_{0}^{(k,r)}) \leq \| \pmb{\omega}^{(k,r)} - \pmb{\omega}_{0}^{(k,r)}\|_{1}$ , we have that
$$
G _ {j} \left(\omega_ {0} ^ {(k, r)}\right) \left\{\log \left(\frac {G _ {j} \left(\omega_ {0} ^ {(k , r)}\right)}{G _ {j} \left(\omega^ {(k , r)}\right)}\right) \right\} ^ {2} \leq \max  \left\{\left| \log (1 - \epsilon_ {2}) \right|, \left| \log (1 + \epsilon_ {2}) \right| \right\} \| \omega^ {(k, r)}) - \omega_ {0} ^ {(k, r)}) \| _ {1}
$$
according to Condition A.4.1. If  $G_{j}(\omega_{0}^{(k,r)}) > \|\omega^{(k,r)} - \omega_{0}^{(k,r)}\|_{1}$ , similar to the proof for (21), we obtain that
$$
\begin{array}{l} G _ {j} \left(\boldsymbol {\omega} _ {0} ^ {(k, r)}\right) \left\{\log \left(\frac {G _ {j} \left(\boldsymbol {\omega} _ {0} ^ {(k , r)}\right)}{G _ {j} \left(\boldsymbol {\omega} ^ {(k , r)}\right)}\right) \right\} ^ {2} = G _ {j} \left(\boldsymbol {\omega} _ {0} ^ {(k, r)}\right) \left[ \frac {1}{G _ {j} (\overline {{\boldsymbol {\omega}}})} \left\{\frac {\partial G _ {j} (\overline {{\boldsymbol {\omega}}})}{\partial \boldsymbol {\omega}} \right\} ^ {\top} \left(\boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)}\right) \right] ^ {2} \\ \leq \frac {G _ {j} \left(\omega_ {0} ^ {(k , r)}\right)}{\left\lbrace (1 - \epsilon_ {2}) G _ {j} \left(\omega_ {0} ^ {(k , r)}\right) \right\rbrace^ {2}} \cdot \left(C _ {3} \| \omega^ {(k, r)} - \omega_ {0} ^ {(k, r)} \| _ {1}\right) ^ {2} \\ \leq \left(\frac {C _ {3}}{1 - \epsilon_ {2}}\right) ^ {2} \| \boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)} \| _ {1}, \\ \end{array}
$$
where  $\bar{\omega}$  is between  $\omega_0^{(k,r)}$  and  $\omega^{(k,r)}$ . Thus, the inequality (22) holds in the neighbourhood of  $\theta_0$ .
B.2.2 Verification of Condition (c) in Lemma 3
Proof. For the sequence  $\{\epsilon_n\}_{n=1}^{\infty}$  in Lemma 3, by (21) and (22) in Result 1 and (17), we obtain that
$$
\Pi \left(G _ {n} ^ {*} \left(\boldsymbol {\theta} _ {0}, \epsilon_ {n}\right)\right) \geq \Pi \left\{\boldsymbol {\theta}: \mathbb {E} _ {\mathbf {x}} \| \boldsymbol {\omega} ^ {(k, r)} - \boldsymbol {\omega} _ {0} ^ {(k, r)} \| _ {1} \leq \Delta_ {n} \text {f o r} k \in [ K ], r \in [ R ] \right\},
$$
where  $\Delta_{n} = U_{1}\epsilon_{n}^{2}$  for some positive constant  $U_{1}$ .
For simplicity of presentation, we now omit  $k$  and  $r$  in  $f_0^{(k,r)}$ ,  $f_{\pmb{\theta}}^{(k,r)}$ ,  $\omega_0^{(k,r)}$ ,  $\omega^{(k,r)}$ ,  $\mathbf{B}^{(k)}$  and  $\mathbf{A}^{(r)}$  in this proof here. Note that
$$
\begin{array}{l} \| \boldsymbol {\omega} - \boldsymbol {\omega} _ {0} \| _ {1} = \| \mathbf {B} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) + \mathbf {A} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}) - \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}) \| _ {1} \\ \leq \left\| \mathbf {B} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}) \right\| _ {1} + \left\| \mathbf {A} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) - \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}) \right\| _ {1} \\ \leq \| \mathbf {B} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) \| _ {1} + \| \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) \| _ {1} \\ + \left\| \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2) *}) - \mathbf {B} _ {0} ^ {\top} \psi_ {2} (\mathbf {x}) \right\| _ {1} + \left\| \mathbf {A} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) - \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) \right\| _ {1} \\ + \left\| \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}) - \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1) *}) \right\| _ {1} + \left\| \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1) *}) - \mathbf {A} _ {0} ^ {\top} \psi_ {1} (\mathbf {x}) \right\| _ {1}. \\ \end{array}
$$
Corresponding to each term above, we consider the following six terms: