SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[65] Philipp Petersen and Felix Voigtlaender. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks, 108:296-330, 2018.
[66] Helmut Bolcskei, Philipp Grohs, Gitta Kutyniok, and Philipp Petersen. Optimal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science, 1(1):8-45, 2019.
[67] Minerva Mukhopadhyay and David B Dunson. Targeted random projection for prediction from high-dimensional features. Journal of the American Statistical Association, 115(532):1998-2010, 2020.
[68] Michael R Sampford. Some inequalities on mill's ratio and related functions. The Annals of Mathematical Statistics, 24(1):130-132, 1953.
[69] Rajarshi Guhaniyogi and David B Dunson. Bayesian compressed regression. Journal of the American Statistical Association, 110(512):1500-1514, 2015.
[70] Abdulhamid A Alzaid and Maha A Omair. On theoisson difference distribution inference and applications. Bulletin of the Malaysian Mathematical Sciences Society. Second Series, 33(1):17-45, 2010.
[71] C. M. Joshi and S. K. Bissu. Some inequalities of bessel and modified bessel functions. Journal of the Australian Mathematical Society, 50(2):333-342, 1991.
[72] Martin J Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press, 2019.
[73] Subhashis Ghosal and Aad Van Der Vaart. Convergence rates of posterior distributions for noniid observations. The Annals of Statistics, 1:192-223, 2007.
[74] Syed Mumtaz Ali and Samuel D Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131-142, 1966.
[75] John Duchi. Information theory and statistics. Lecture Notes for Statistics, 311, 2019.
[76] Michael Short. Improved inequalities for the poisson and binomial distribution and upper tail quantile functions. International Scholarly Research Notices, 2013.
[77] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
In the supplementary materials, we first summarize the regularity conditions on the underlying true model and prior distributions of the network parameters in Section A. The proofs of Theorem 1, Theorem 2, and Corollary 3 are provided in Sections B.2-B.4 with all the needed preliminaries presented in Section B.1. Implementation details of the proposed method and additional experiment results are exhibited in Section C, including the accuracy and number of selected labels using the proposed label correction algorithm (10), the test accuracy on MNIST, the hyperparameter analysis on CIFAR100, the classification accuracy on CIFAR100 with varying number of annotators, and the average estimation error on CIFAR10 and CIFAR100 with varying number of annotators.
A Regularity Conditions
A.1 Network structure
To incorporate the sparse high dimensional setting [59], we utilize sparse Bayesian DNNs to reconstruct  $\psi_{1}(\mathbf{x})$  and  $\psi_{2}(\mathbf{x})$  in (3) [37, 60]. Specifically, to approximate  $\psi_{j}(\mathbf{x})$  with  $j = 1,2$ , we consider a network with  $H_{nj} - 1$  hidden layers and a width vector  $L^{(j)} = (L_0^{(j)}, L_1^{(j)}, \dots, L_{H_{nj}}^{(j)})^\top$ , where the width of the  $h$ th layer is denoted  $L_h^{(j)}$  for  $h = 0, \dots, H_{nj}$  with  $L_0^{(j)} = p_n$  for the input layer and  $L_{H_{nj}}^{(j)} \triangleq M^{(j)}$  for the output layer. Then the DNN with network architecture  $\{H_{nj}, L^{(j)}\}$  is the nonlinear function of the form:
$$
\psi_ {j} (\mathbf {x}; \boldsymbol {\theta} ^ {(j)}) = \mathbf {W} ^ {(j, H _ {n j})} \sigma \left(\dots \sigma \left[ \mathbf {W} ^ {(j, h)} \sigma \left\{\dots \sigma (\mathbf {W} ^ {(j, 1)} \mathbf {x} + \mathbf {b} ^ {(j, 1)}) \dots \right\} + \mathbf {b} ^ {(j, h)} \right] \dots\right) + \mathbf {b} ^ {(j, H _ {n j})},
$$
where for  $h = 1,\dots,H_{nj}$ ,  $\mathbf{W}^{(j,h)}$  is a  $L_h^{(j)}\times L_{h - 1}^{(j)}$  weight matrix,  $\mathbf{b}^{(j,h)}\in \mathbb{R}^{L_h^{(j)}}$  is the bias of layer  $h$ ,  $\sigma (\cdot)$  is a nonlinear activation function, and  $\pmb{\theta}^{(j)}$  represents the  $J_{nj}\times 1$  vector formed from stacking  $\{\mathbf{W}^{(j,h)},\mathbf{b}^{(j,h)}\}_{h = 1}^{H_{nj}}$  from bottom to the top, with  $J_{nj}\triangleq \sum_{h = 1}^{H_{nj}}(L_{h - 1}^{(j)}\times L_h^{(j)} + L_h^{(j)})$ . We treat weights and biases equally without distinguishing them in  $\pmb{\theta}^{(j)}$ , and write  $\pmb{\theta}^{(j)}$  as  $\pmb{\theta}^{(j)} = (\theta_1^{(j)},\dots,\theta_{J_{nj}}^{(j)})^\top$ . Let  $\gamma^{(j)} = (\gamma_1^{(j)},\dots,\gamma_{J_{nj}}^{(j)})^\top$  denote the indicator vector, with  $\gamma_k^{(j)} = \mathbf{1}(\theta_k^{(j)}\neq 0)$  for  $k = 1,\dots,J_{nj}$ . For ease of presentation, we use  $\psi_j(\mathbf{x};\pmb{\theta}^{(j)})$  and  $\psi_j(\mathbf{x};\pmb{\theta}^{(j)},\pmb{\gamma}^{(j)})$  exchangeable to represent the model for  $\psi_j(\mathbf{x})$ , and let  $\mathcal{F}_n = \mathcal{F}(H_{n1},H_{n2},L^{(1)},L^{(2)},C_1,C_2,\epsilon_1)$  denote the space of all sparse networks that satisfy Condition A.3 in Appendix A.2 and are constrained by positive constants  $C_1,C_2$  and  $\epsilon_{1}$ .
To determine the parameters of the sparse DNNs  $\psi_j(\mathbf{x};\pmb{\theta}^{(j)},\pmb{\gamma}^{(j)})$  that best approximate  $\psi_j(\mathbf{x})$  for  $j = 1,2$ , we define
$$
\begin{array}{l} \left(\boldsymbol {\theta} ^ {(1) *}, \boldsymbol {\gamma} ^ {(1) *}, \boldsymbol {\theta} ^ {(2) *}, \boldsymbol {\gamma} ^ {(2) *}\right) = \underset {\left(\boldsymbol {\theta} ^ {(1)}, \boldsymbol {\gamma} ^ {(1)}, \boldsymbol {\theta} ^ {(2)}, \boldsymbol {\gamma} ^ {(2)}\right) \in \mathcal {F} _ {n}} {\arg \min } \quad \left\{\left| \boldsymbol {\gamma} ^ {(1)} \right| + \left| \boldsymbol {\gamma} ^ {(2)} \right| \right\}, \tag {14} \\ \left\| \psi_ {1} (\mathbf {x}; \boldsymbol {\theta} ^ {(1)}, \boldsymbol {\gamma} ^ {(1)}) - \psi_ {1} (\mathbf {x}) \right\| _ {L ^ {2} (\Omega)} \leq \varpi_ {n 1} \\ \left\| \psi_ {2} (\mathbf {x}; \boldsymbol {\theta} ^ {(2)}, \boldsymbol {\gamma} ^ {(2)}) - \psi_ {2} (\mathbf {x}) \right\| _ {L ^ {2} (\Omega)} \leq \varpi_ {n 2} \\ \end{array}
$$
where for  $j = 1,2$ ,  $\varpi_{nj}$  is an  $n$ -dependent positive constant satisfying  $\varpi_{nj} \to 0$  as  $n \to \infty^3$ . We call  $\pmb{\theta}^{(1)*}, \pmb{\gamma}^{(1)*}, \pmb{\theta}^{(2)*}$ , and  $\pmb{\gamma}^{(2)*}$  the true parameters of  $\pmb{\theta}^{(1)}, \pmb{\gamma}^{(1)}, \pmb{\theta}^{(2)}$ , and  $\pmb{\gamma}^{(2)}$ , respectively.
A.2 Conditions for the sparse GLM
A.1 The input vector is standardized so that  $\mathbf{x} \in \Omega \subset [-1, 1]^{p_n}$ , where  $\Omega$  is the support of  $\mathbf{x}$  and the probability density  $f(\cdot)$  of  $\mathbf{x}$  satisfies that  $\sup_{\mathbf{x} \in \Omega} |f(\mathbf{x})| \leq C_0$  for some positive constant  $C_0$ .
A.2 The activation function  $\sigma (\cdot)$  is 1-Lipschitz.
<sup>3</sup>In this definition, the  $L_{2}$  norm of the network approximation error is bounded. Consider a measure space  $(\Omega, \mathcal{G}, \mu)$  and  $0 < p < q \leq \infty$ , we have that  $\|f\|_p \leq \mu(\Omega)^{1/p-1/q} \|f\|_q$  using Hölder's inequality and therefore,  $\|f\|_1 \leq \|f\|_2$  and  $L^1(\Omega, \mu) \subset L^2(\Omega, \mu)$  for probability measure  $\mu$ . Thus, the  $L_1$  norm of the network approximation error is also bounded, which will be used in the following proofs.
A.3 The true sparse DNN model satisfies the following conditions.
A.3.1 For  $j = 1,2$ , let  $r_{nj} = \| \pmb{\gamma}^{(j)}\|_1$  denote the connectivity of  $\pmb{\gamma}^{(j)}$ , let  $\overline{L}_{nj} = \max_{1\leq h\leq H_{nj}}L_h^{(j)}$  denote the maximum layer width, and let  $s_{nj}$  represent the input dimension of  $\pmb{\gamma}^{(j)}$ . Let  $\zeta_n = \left\{(r_{n1} + r_{n2})(H_{n1} + H_{n2})\log n + (r_{n1} + r_{n2})(\log \overline{L}_{n1} + \log \overline{L}_{n2}) + (s_{n1} + s_{n2})\log p_n\right\} / n$ . The true sparse DNN model satisfies that  $\zeta_n\leq C_1n^{-\epsilon_1}$  for some constants  $C_1 > 0$  and  $0 < \epsilon_1 < 1$ .
A.3.2 For  $j = 1,2$ ,  $\| \pmb{\theta}^{(j)}\|_{\infty}\leq E_{nj}$ , where positive constant  $E_{nj}\leq n^{C_2}$  for some constant  $C_2 > 0$ .
A.4 Write the function  $G(\cdot)$  in (3) as  $(G_{1}(\cdot),\dots,G_{K}(\cdot))^{\top}$ . For  $\mathbf{x}\in \Omega$ ,  $r\in [R]$  and  $k\in [K]$ , write  $\pmb{\omega}^{(k,r)} = \mathbf{A}^{(r)\mathrm{T}}\psi_1(\mathbf{x};\pmb{\theta}^{(1)}) + \mathbf{B}^{(k)\mathrm{T}}\psi_2(\mathbf{x};\pmb{\theta}^{(2)})$ .
A.4.1 For  $k \in [K]$  and  $r \in [R]$ , in the neighbourhood of  $\theta_0$
$$
\sup  _ {j, l \in [ K ]} \left| \frac {\partial G _ {j} (\boldsymbol {\omega})}{\partial \omega_ {l}} \right| _ {\boldsymbol {\omega} = \boldsymbol {\omega} ^ {(k, r)}} \Big | = C _ {3} \text {a n d} \sup  _ {j \in [ K ]} \left| \frac {G _ {j} \left(\boldsymbol {\omega} ^ {(k , r)}\right)}{G _ {j} \left(\boldsymbol {\omega} _ {0} ^ {(k , r)}\right)} - 1 \right| = \epsilon_ {2}
$$
for some constants  $C_3 > 0$  and  $\epsilon_{2}\in (0,1)$ , where the latter requirement can be achieved if  $G_{j}(\omega_{0}^{k,r}) > \varsigma$  for some positive constant  $\varsigma >0$ .
A.4.2 For  $r \in [R]$  and  $j, k \in [K]$ ,  $\| \alpha_{j0}^{(r)} \|_{\infty} \leq F_1$  and  $\| \beta_{j0}^{(k)} \|_{\infty} \leq F_2$ , where  $F_1$  and  $F_2$  are positive constants.
A.3 Conditions for the prior
B.1 For  $j = 1,2$ , assume each element of  $\pmb{\theta}^{(j)}$  has independent continuous prior distribution, denoted  $\pi_{\theta}^{(j)}(\cdot)$ . Thus, its minimum value on the interval  $[-E_{nj} - 1,E_{nj} + 1]$  exists, and let  $\underline{\pi}_{\theta}^{(j)}$  denote it. For a sequence of positive constants  $I_n^{(j)}$  with  $\log I_n^{(j)} = O(\log n)$ , let  $\delta_{nj}$  and  $\delta_{nj}'$  be two sequences of constants satisfying that  $\delta_{nj} < 1 / nJ_{nj}(c_0I_n^{(j)})^{H_{nj}}(n / H_{nj})^{H_{nj}}$  and  $\delta_{nj}' < 1 / nJ_{nj}(c_0E_{nj})^{H_{nj}}(r_{nj} / H_{nj})^{H_{nj}}$  for some constant  $c_0 > 1$ , respectively. Assume that:
B.1.1  $\log (1 / \underline{\pi}_{\theta}^{(j)}) = O(H_{nj}\log n + \log \overline{L}_{nj})$
B.1.2  $\pi_{\theta}^{(j)}([- \delta_{nj}, \delta_{nj}]) \geq 1 - \frac{1}{J_{nj}} \exp \left[ -S_0^{(j)} \left\{(H_{n1} + H_{n2}) \log n \right.\right]$
$$
\left. \left. + \log \overline {{L}} _ {n 1} + \log \overline {{L}} _ {n 2} + \log p _ {n} \right\} \right]
$$
for some constant  $S_0^{(j)} > 2$
$$
\pi_ {\theta} ^ {(j)} ([ - \delta_ {n j} ^ {\prime}, \delta_ {n j} ^ {\prime} ]) \geq 1 - \frac {1}{J _ {n j}};
$$
B.1.3  $-\log \left\{J_{nj}\pi_{\theta}^{(j)}(|\theta_1^{(j)}| > I_n^{(j)})\right\} \succ (2 + \epsilon_3^{(j)})n\epsilon_n^2$  for some positive constant  $\epsilon_3^{(j)}$ .
B.2 For  $k \in [K]$ , let  $\overline{B}_n^{(k)}$  and  $\underline{B}_n^{(k)}$  denote the largest and the smallest eigenvalues of  $\Sigma_{\beta}^{(k)}$ , respectively, and for  $r \in [R]$ , let  $\overline{A}_n^{(r)}$  and  $\underline{A}_n^{(r)}$  denote the largest and the smallest eigenvalues of  $\Sigma_{\alpha}^{(r)}$ , respectively. Assume that for large enough  $n$ ,
B.2.1  $\overline{A}_n^{(r)}\leq S_1^{(1)}M^{(1)q_1}$  and  $\underline{A}_n^{(r)}\geq S_1^{(1)}\{\log M^{(1)}\}^{-1}$  for some positive constants  $S_{1}^{(1)}$ $S_{2}^{(1)}$  and  $q_{1}$
B.2.2  $\overline{B}_n^{(k)}\leq S_1^{(2)}M^{(2)q_1}$  and  $\underline{B}_n^{(k)}\geq S_2^{(2)}\{\log M^{(2)}\}^{-1}$  for some positive constants  $S_{1}^{(2)}$ $S_{2}^{(2)}$  and  $q_{2}$
A.4 Remark
1. Assumption A.1 specifies that the hypothesis set we consider is a class of DNNs, which is a common setting in the literature [61, 35].
2. Assumption A.2 ensures that the underlying noise transition probability can be approximated by a sparse model. Existing works [62-64] empirically show that large DNNs often contain a large number of redundant parameters and propose methods for compressing neural networks without affecting performance. Moreover, theoretical works [65, 66] in approximation theory provide theories that guarantee uniform approximation rates for a broad family of function classes. Similar assumptions can be found in [35, 67].
3. Assumption A.3 specifies the constraints on the prior distribution we use. As in our experiments in Section 5, we may employ the spike-and-slab prior  $\lambda_{n}\mathcal{N}(0,\sigma_{1n}^{2}) + (1 - \lambda_{n})\mathcal{N}(0,\sigma_{0n}^{2})$  for each element of the parameter vector  $\pmb{\theta}^{(j)}$  of the sparse Bayesian DNN  $\psi_j(\mathrm{x};\pmb{\theta}^{(j)})$ , with  $j = 1,2$ ; we take the normal prior  $\mathcal{N}(0,\sigma_n^2)$  for each element of the regression weights. It can be verified that Condition B.1 in A.3 is satisfied if the values of  $\lambda_{n}$ ,  $\sigma_{1n}$ , and  $\sigma_{0n}$  are properly chosen [35]. In particular, the value of  $\lambda_{n}$  is related to the sparsity of the model and we require it to satisfy that  $\lambda_{n} = O(1 / J_{nj}[n^{H_{n1} + H_{n2}}(\overline{L}_{n1} + \overline{L}_{n2})p_n])^c)$  for some positive constant  $c$  and  $j = 1,2$ , which should be chosen by considering the network structure and the number of data points,  $n$ . Moreover, by using techniques such as Mill's ratio [68], Condition B.2 in A.3 is satisfied if  $c_{1} < \sigma_{n} < c_{2}$  for some positive constants  $c_{1}$  and  $c_{2}$ , which are related to  $S_1^{(j)}, S_2^{(j)}, q_j$  and  $M^{(j)}$  for  $j = 1,2$  in Condition B.2 [60, 67]. Similar assumptions can be found in [60, 35, 67].
In this section, we present the proofs of Theorem 1, Theorem 2, and Corollary 3. Specifically, we provide all the need preliminaries in Section B.1, where the definitions and results in Sections B.1.1-B.1.4 will be used in the proof of Theorem 1, and the information-theoretical definitions and lemmas provided in Section B.1.5 will be utilized in the proof of Theorem 2 and Corollary 3.
B.1.1 Definitions of some discrepancy measures
Definition 1 ([69]). Let  $f$  and  $f_0$  denote two conditional probability density/mass functions of  $\tilde{\mathbf{y}}$ , given  $\mathbf{x}$ . Let  $\nu_1(d\mathbf{x})$  denote the probability measure for  $\mathbf{x}$  associated with the density  $f(\mathbf{x})$  and let  $\nu_2(d\tilde{\mathbf{y}}) = \bigotimes_{r=1}^{R} \nu_{2,r}(d\tilde{\mathbf{y}}^{(r)})$  be a dominating measure for  $f$  and  $f_0$ , and hence, a dominating measure of  $(\mathbf{x}, \tilde{\mathbf{y}})$  is taken as the product  $\nu_1(d\mathbf{x})\nu_2(d\tilde{\mathbf{y}})$ .
(i) The Hellinger distance between  $f$  and  $f_0$  is defined as
$$
d (f, f _ {0}) = \sqrt {\int \int (\sqrt {f} - \sqrt {f _ {0}}) ^ {2} \nu_ {2} (d \tilde {\mathbf {y}}) \nu_ {1} (d \mathbf {x})}.
$$
(ii) For any  $t > 0$ , define
$$
d _ {t} (f, f _ {0}) = \frac {1}{t} \left\{\int \int f _ {0} \left(\frac {f _ {0}}{f}\right) ^ {t} \nu_ {2} (d \tilde {\mathbf {y}}) \nu_ {1} (d \mathbf {x}) - 1 \right\}.
$$
(iii) The Kullback-Leibler divergence between  $f$  and  $f_0$  is defined as
$$
d _ {0} (f, f _ {0}) \triangleq K (f, f _ {0}) = \int \int f _ {0} \log \left(\frac {f _ {0}}{f}\right) \nu_ {2} (d \tilde {\mathbf {y}}) \nu_ {1} (d \mathbf {x}).
$$
(iv) For  $q > 1$ , define
$$
V _ {q} (f, f _ {0}) = \int \int f _ {0} \left| \log \left(\frac {f _ {0}}{f}\right) \right| ^ {q} \nu_ {2} (d \tilde {\mathbf {y}}) \nu_ {1} (d \mathbf {x}).
$$
For  $q = 2$ , the index of  $V_{2}(\cdot, \cdot)$  is omitted and the discrepancy measure is denoted  $V(\cdot, \cdot)$ . [60] shows that (1)  $d(f, f_0) \leq \sqrt{d_0(f, f_0)}$ ; (2)  $d_t(f, f_0)$  decreases to  $d_0(f, f_0)$  as  $t$  decreases to 0.
B.1.2 Mathematical details about the regression weights
Adapting the proof in [69], we prove the following proposition.
Proposition 1. Assume that  $\beta \sim \mathcal{N}(\mathbf{0},\pmb{D}_{\beta})$ , where  $\pmb{D}_{\beta}$  is a positive definite matrix. Then, for any given  $\dim (\beta)\times 1$  vector of functions  $\phi (\mathbf{x};\pmb {\theta})$  of  $\mathbf{x}$  and  $\pmb{\theta}$ , and for any constant  $\Delta >0$
$$
\begin{array}{l} P \left\{\left| (\phi (\mathbf {x}; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {\beta} - \phi (\mathbf {x}; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {\beta} _ {0} \right| <   \Delta \right\} \\ > 8 \exp \left\{- \frac {\{\phi (\mathbf {x} ; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {\beta} _ {0} \} ^ {2} + \Delta^ {2}}{2 \underline {{B}} \| \phi (\mathbf {x} ; \boldsymbol {\theta}) \| ^ {2}} \right\} \frac {\Delta^ {4}}{\overline {{B}} ^ {2} \| \phi (\mathbf {x} ; \boldsymbol {\theta}) \| ^ {4}}, \\ \end{array}
$$
where  $\overline{B}$  and  $\underline{B}$  are the largest and the smallest eigenvalues of  $D_{\beta}$ , respectively.
Proof. The proof is established in two steps.
Step 1. We first prove that
$$
P \left\{\left| (\phi (\mathbf {x}; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {\beta} - \phi (\mathbf {x}; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {\beta} _ {0} \right| <   \Delta \right\} > P (X - Y \geq 2), \tag {15}
$$
where  $X \sim Pois\left(\frac{\Delta_1}{2}\right)$  and  $Y \sim Pois\left(\frac{\lambda}{2}\right)$ , with  $\Delta_1 = \frac{\Delta^2}{\phi(\mathbf{x};\boldsymbol{\theta})^{\mathrm{T}}D_{\beta}\phi(\mathbf{x};\boldsymbol{\theta})}$  and  $\lambda = \frac{\left\{\phi(\mathbf{x};\boldsymbol{\theta})^{\mathrm{T}}\boldsymbol{\beta}_0\right\}^2}{\phi(\mathbf{x};\boldsymbol{\theta})^{\mathrm{T}}D_{\beta}\phi(\mathbf{x};\boldsymbol{\theta})}$ , and  $X$  and  $Y$  are independent.
By the definition of the noncentral chi-squared distribution, it can be easily seen that  $T \triangleq \frac{\left\{\phi(\mathbf{x};\boldsymbol{\theta})^{\mathrm{T}}\boldsymbol{\beta} - \phi(\mathbf{x};\boldsymbol{\theta})^{\mathrm{T}}\boldsymbol{\beta}_0\right\}^2}{\phi(\mathbf{x};\boldsymbol{\theta})^{\mathrm{T}}D_{\boldsymbol{\beta}}\phi(\mathbf{x};\boldsymbol{\theta})}$  is distributed according to the noncentral chi-squared distribution  $\chi_1^2 (\lambda)$ . Thus, by utilizing the cumulative distribution function (CDF) of  $\chi_1^2 (\lambda)$ , we obtain that
$$
\begin{array}{l} P \left\{\left| (\phi (\mathbf {x}; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {\beta} - \phi (\mathbf {x}; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {\beta} _ {0} \right| <   \Delta \right\} \\ = P \left\{T <   \Delta_ {1} \right\} \\ = \sum_ {j = 0} ^ {\infty} \frac {\exp \left(- \frac {\lambda}{2}\right) (\frac {\lambda}{2}) ^ {j}}{j !} Q \left(\Delta_ {1}; 1 + 2 j\right), \\ \end{array}
$$
where  $Q(\cdot; 1 + 2j)$  is the CDF of  $\chi_{1 + 2j}^2$ , the central chi-squared distribution with  $1 + 2j$  degrees of freedom.
Noting that  $Q(\Delta_1; 1 + 2j) > Q(\Delta_1; 2 + 2j)$ , by the result that  $Z_{j1} + \ldots + Z_{j,j+1} \sim \chi_{2+2j}^2$  if  $Z_{j1}, \ldots, Z_{j,j+1} \stackrel{\text{i.i.d.}}{\sim} \chi_2^2 \stackrel{\text{d}}{=} \exp\left(\frac{1}{2}\right)$ , we obtain that  $Q(\Delta_1; 1 + 2j) > P\{Z_{j1} + \ldots + Z_{j,j+1} < \Delta_1\}$ . According to the relationship between the Poisson counting process and exponential variables, we have that the counting process with  $Z_{j1}, \ldots, Z_{j,j+1}$  as inter-arrival times is the Poisson process with rate  $\frac{1}{2}$ . Let  $N(\Delta_1)$  denote the total number of occurrences or events that have happened up to time  $\Delta_1$ . Then,  $N(\Delta_1)$  follows the Poisson distribution with parameter  $\frac{\Delta_1}{2}$ , i.e.,  $N(\Delta_1) \stackrel{\text{d}}{=} X$ , and  $\{Z_{j1} + \ldots + Z_{j,j+1} < \Delta_1\} = \{N(\Delta_1) > j + 1\}$ . Consequently, we have that
$$
\begin{array}{l} \sum_ {j = 0} ^ {\infty} \frac {\exp \left(- \frac {\lambda}{2}\right) (\frac {\lambda}{2}) ^ {j}}{j !} Q \left(\Delta_ {1}; 1 + 2 j\right) > \sum_ {j = 0} ^ {\infty} \frac {\exp \left(- \frac {\lambda}{2}\right) (\frac {\lambda}{2}) ^ {j}}{j !} P \left\{Z _ {j 1} + \dots + Z _ {j, j + 1} <   \Delta_ {1} \right\} \\ = \sum_ {j = 0} ^ {\infty} \frac {\exp \left(- \frac {\lambda}{2}\right) (\frac {\lambda}{2}) ^ {j}}{j !} P \{N (\Delta_ {1}) > j + 1 \} \\ = \sum_ {j = 0} ^ {\infty} \frac {\exp \left(- \frac {\lambda}{2}\right) \left(\frac {\lambda}{2}\right) ^ {j}}{j !} P \{X \geq j + 2 \} \\ = \sum_ {j = 0} ^ {\infty} \mathbb {P} (Y = j) \mathbb {P} (X \geq j + 2) \\ = \sum_ {j = 0} ^ {\infty} \mathbb {P} (X \geq j + 2, Y = j) \\ = P \{X - Y \geq 2 \}, \\ \end{array}
$$
where we use the distribution assumptions and the independence assumption for  $X$  and  $Y$ . Thus, (15) is proved.
Step 2. Since  $X \sim \text{Pois}(\frac{\Delta_1}{2})$  and  $Y \sim \text{Pois}(\frac{\lambda}{2})$ , and they are independent, we have that  $X - Y$  follows the Poisson difference distribution (aka the Skellam distribution): for  $k = 0, \pm 1, \pm 2, \ldots$ ,
$$
P \{X - Y = k \} = \exp \left(- \frac {\Delta_ {1} + \lambda}{2}\right) \left(\frac {\Delta_ {1}}{\lambda}\right) ^ {\frac {k}{2}} I _ {k} (\sqrt {\Delta_ {1} \lambda}),
$$
where  $I_{\nu}(z) = \sum_{m=0}^{\infty} \frac{(z/2)^{\nu+2m}}{m! \Gamma(\nu+m+1)}$  is the modified Bessel function of the first kind [70]. Using the fact that  $I_{\nu}(z) > z^{\nu} 2^{\nu} \Gamma(\nu+1)$  for  $z > 0$  [71] and plugging in  $\Delta_1$  and  $\lambda$ , we obtain that
$$
\begin{array}{l} P \{X - Y \geq 2 \} > P \{X - Y = 2 \} > 8 \exp \left(- \frac {\Delta_ {1} + \lambda}{2}\right) \Delta_ {1} ^ {2} \\ = 8 \exp \left\{- \frac {\left\{\phi (\mathbf {x} ; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {\beta} _ {0} \right\} ^ {2} + \Delta^ {2}}{2 \phi (\mathbf {x} ; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {D} _ {\boldsymbol {\beta}} \phi (\mathbf {x} ; \boldsymbol {\theta})} \right\} \cdot \frac {\Delta^ {4}}{\left\{\phi (\mathbf {x} ; \boldsymbol {\theta}) ^ {\mathrm {T}} \boldsymbol {D} _ {\boldsymbol {\beta}} \phi (\mathbf {x} ; \boldsymbol {\theta}) \right\} ^ {2}} \\ \geq 8 \exp \left\{- \frac {\{\phi (\mathbf {x} ; \pmb {\theta}) ^ {\mathrm {T}} \pmb {\beta} _ {0} \} ^ {2} + \Delta^ {2}}{2 \underline {{B}} \| \phi (\mathbf {x} ; \pmb {\theta}) \| ^ {2}} \right\} \cdot \frac {\Delta^ {4}}{\overline {{B}} ^ {2} \| \phi (\mathbf {x} ; \pmb {\theta}) \| ^ {4}}, \\ \end{array}
$$
where the last inequality follows from the fact that for a symmetric matrix  $D$ ,  $\sup_{\| u \| = 1} u^{\mathrm{T}} Du = \lambda_{max}(D)$  and  $\inf_{\| u \| = 1} u^{\mathrm{T}} Du = \lambda_{min}(D)$ , where  $\lambda_{max}(D)$  and  $\lambda_{min}(D)$  represent the largest and the smallest eigenvalues of  $D$ , respectively. This completes the proof.
B.1.3 Mathematical details about the sparse Bayesian DNNs
Consider a DNN with network architecture  $(H_{n},L)$ , where  $H_{n} - 1$  is the number of hidden layers, and  $\pmb {L} = (L_0,L_1,\dots ,L_{H_n})^{\mathrm{T}}$  is the width vector with  $L_{0} = p_{n}$  for the input layer and  $L_{H_n} = M$  for the output layer. For the corresponding indicator vector  $\gamma$  and for  $h = 1,\ldots ,H_{n}$ , let  $r_h$  denote the number of nonzero connections to the  $h$ th hidden layer which includes the bias for the  $h$ th hidden layer and the weights between the  $(h - 1)$ th and the  $h$ th layer, such that  $\sum r_h = |\pmb {\gamma}|_1\dot{=} r_n$ . For a parameter vector  $\pmb{\theta}$ , let  $O_{h,j}(\pmb {\theta},\mathbf{x})$  represent the output value of the  $j$ th node of the  $h$ th hidden layer for  $j = 1,\dots ,L_{h}$ .
Lemma 1 ([37], Lemma S1). Consider a sparse DNN with parameter vector  $\pmb{\theta} = (\theta_{1},\dots,\theta_{q})^{\top}$  with dimension  $q$ , the corresponding indicator vector  $\gamma$  and the network architecture as mentioned above. Suppose that Conditions A.1-A.2 in Section A.2 are satisfied and  $|\pmb{\theta}|_{\infty} \leq E_n$  for a positive constant  $E_{n}$ . Then, for  $1 \leq h \leq H_{n}$ , the summation of the outputs of the  $h$ th layer is upper bounded by
$$
\sum_ {j = 1} ^ {L _ {h}} O _ {h, j} (\boldsymbol {\theta}, \mathbf {x}) \leq E _ {n} ^ {h} \prod_ {k = 1} ^ {h} r _ {k}.
$$