SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {j = 1} ^ {L _ {1}} | \check {O} _ {1, j} ^ {t} - O _ {1, j} ^ {t} | \leq r _ {w _ {1}} \delta_ {1} + r _ {v _ {1}} \delta_ {1} \sum_ {k = 1} ^ {L _ {1}} | O _ {1, k} ^ {t - 1} | + r _ {v _ {1}} (E _ {n} + \delta_ {1}) \sum_ {k = 1} ^ {L _ {1}} | \check {O} _ {1, k} ^ {t - 1} - O _ {1, k} ^ {t - 1} |.
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {i}} \left| \check {O} _ {i, j} ^ {t} - O _ {i, j} ^ {t} \right| \leq r _ {w _ {i}} \delta_ {1} \sum_ {k = 1} ^ {L _ {i - 1}} \left| O _ {i - 1, k} ^ {t} \right| + r _ {w _ {i}} \left(E _ {n} + \delta_ {1}\right) \sum_ {k = 1} ^ {L _ {i - 1}} \left| \check {O} _ {i - 1, k} ^ {t} - O _ {i - 1, k} ^ {t} \right| \\ + r _ {v _ {i}} \delta_ {1} \sum_ {k = 1} ^ {L _ {i}} \left| O _ {i, k} ^ {t - 1} \right| + r _ {v _ {i}} \left(E _ {n} + \delta_ {1}\right) \sum_ {k = 1} ^ {L _ {i}} \left| \check {O} _ {i, k} ^ {t - 1} - O _ {i, k} ^ {t - 1} \right|. \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {H _ {n}}} \left| \check {O} _ {H _ {n}, j} ^ {t} - O _ {H _ {n}, j} ^ {t} \right| \leq r _ {w _ {H _ {n}}} \delta_ {1} \sum_ {k = 1} ^ {L _ {H _ {n} - 1}} \left| O _ {H _ {n} - 1, k} ^ {t} \right| \\ + r _ {w _ {H _ {n}}} \left(E _ {n} + \delta_ {1}\right) \sum_ {k = 1} ^ {L _ {H _ {n} - 1}} \left| \check {O} _ {H _ {n} - 1, k} ^ {t} - O _ {H _ {n} - 1, k} ^ {t} \right|. \\ \end{array}
$$
For  $1\leq i\leq H_{n} - 1$
$$
\sum_ {j = 1} ^ {L _ {i}} \left| \check {O} _ {i, j} ^ {t} - O _ {i, j} ^ {t} \right| \leq t ^ {i + 1} i \delta_ {1} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n} + \delta_ {1}) ^ {(i + 1) t - 2}.
$$
For  $i = H_{n}$
$$
\begin{array}{l} | \mu (\boldsymbol {x} _ {t: 1}, \boldsymbol {\beta}) - | \mu (\boldsymbol {x} _ {t: 1}, \check {\boldsymbol {\beta}}) | = \sum_ {j = 1} ^ {L _ {H _ {n}}} | \check {O} _ {H _ {n}, j} ^ {t} - O _ {H _ {n}, j} ^ {t} | \\ \leq t ^ {H _ {n}} H _ {n} \delta_ {1} \left(\prod_ {k = 1} ^ {H _ {n}} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {H _ {n} - 1} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n} + \delta_ {1}) ^ {(H _ {n} + 1) t - 2}. \\ \end{array}
$$
We can verify the above conclusion by plugging it into all recursive relationships. We show the steps to verify the case when  $t \geq 2$  and  $2 \leq i \leq H_{n} - 1$ , since other cases are trivial to
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {i}} | \check {O} _ {i, j} ^ {t} - O _ {i, j} ^ {t} | \leq r _ {w _ {i}} \delta_ {1} \sum_ {k = 1} ^ {L _ {i - 1}} | O _ {i - 1, k} ^ {t} | + r _ {w _ {i}} (E _ {n} + \delta_ {1}) \sum_ {k = 1} ^ {L _ {i - 1}} | \check {O} _ {i - 1, k} ^ {t} - O _ {i - 1, k} ^ {t} | \\ + r _ {v _ {i}} \delta_ {1} \sum_ {k = 1} ^ {L _ {i}} | O _ {i, k} ^ {t - 1} | + r _ {v _ {i}} (E _ {n} + \delta_ {1}) \sum_ {k = 1} ^ {L _ {i}} | \check {O} _ {i, k} ^ {t - 1} - O _ {i, k} ^ {t - 1} | \\ \leq \delta_ {1} t ^ {i - 1} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i - 1} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n} + \delta_ {1}) ^ {t i - t} \\ + \delta_ {1} t ^ {i} (i - 1) \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i - 1} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n} + \delta_ {1}) ^ {t i - 1} \\ + \delta_ {1} (t - 1) ^ {i} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 2} r _ {v _ {i}} (E _ {n} + \delta_ {1}) ^ {t i - i} \\ + \delta_ {1} (t - 1) ^ {i + 1} i \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 2} r _ {v _ {i}} (E _ {n} + \delta_ {1}) ^ {t i - i + t - 2} \\ \leq \delta_ {1} t ^ {i - 1} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n} + \delta_ {1}) ^ {(i + 1) t - 2} \\ + \delta_ {1} t ^ {i} (i - 1) \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} \left(E _ {n} + \delta_ {1}\right) ^ {(i + 1) t - 2} \\ + \delta_ {1} (t - 1) ^ {i} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} \left(E _ {n} + \delta_ {1}\right) ^ {(i + 1) t - 2} \\ + \delta_ {1} (t - 1) ^ {i + 1} i \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} \left(E _ {n} + \delta_ {1}\right) ^ {(i + 1) t - 2} \\ \leq t ^ {i + 1} i \delta_ {1} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n} + \delta_ {1}) ^ {(i + 1) t - 2}, \\ \end{array}
$$
where the last inequality is due to the fact that for  $t \geq 2$  and  $i \geq 2$ , it is easy to see that
$$
\begin{array}{l} \frac {t ^ {i - 1} + (i - 1) t ^ {i} + (t - 1) ^ {i} + i (t - 1) ^ {i + 1}}{i t ^ {i + 1}} = \frac {1}{i t ^ {2}} + \frac {i - 1}{i} \frac {1}{t} + \frac {1}{i t} \left(\frac {t - 1}{t}\right) ^ {i} + \left(\frac {t - 1}{t}\right) ^ {i + 1} \\ \leq \frac {1}{2 t ^ {2}} + \frac {1}{t} + \frac {(t - 1) ^ {2}}{2 t ^ {3}} + \left(\frac {t - 1}{t}\right) ^ {3} = \frac {t + 2 t ^ {2} + t ^ {2} - 2 t + 1 + 2 (t - 1) ^ {3}}{2 t ^ {3}} \\ = 1 + \frac {- 3 t ^ {2} + 5 t - 1}{2 t ^ {3}} \leq 1. \\ \end{array}
$$
Now let  $\widetilde{O}_{i,j}^{t}$  denotes  $O_{i,j}^{t}(\widetilde{\beta},\pmb{x}_{1:t})$ , then from the facts that
For  $1\leq t\leq T$  and  $1\leq i\leq H_{n}$
$$
\sum_ {j = 1} ^ {L _ {i}} \left| \check {O} _ {i, j} ^ {t} \right| \leq t ^ {i} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n} + \delta_ {1}) ^ {t i}, \quad 1 \leq i \leq H _ {n} - 1, \quad 1 \leq t \leq T,
$$
$$
\sum_ {j = 1} ^ {L _ {H _ {n}}} \left| \check {O} _ {H _ {n}, j} ^ {t} \right| \leq t ^ {H _ {n}} \left(\prod_ {k = 1} ^ {H _ {n}} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {H _ {n} - 1} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n} + \delta_ {1}) ^ {t (H _ {n} - 1) + 1}, \quad 1 \leq t \leq T.
$$
This can be easily derived by Lemma A.1 and the fact that  $\left\| \breve{\beta} \right\|_{\infty} \leq E_n + \delta_1$ .
$$
\sum_ {j = 1} ^ {L _ {i}} \left| \check {O} _ {i, j} ^ {1} - \widetilde {O} _ {i, j} ^ {1} \right| \leq \delta_ {2} \left(p _ {n} L _ {1} + \sum_ {k = 1} ^ {i} L _ {k}\right) \left(\prod_ {k = 1} ^ {i} \left[ \delta_ {2} L _ {k} + r _ {w _ {k}} \left(E _ {n} + \delta_ {1}\right) \right]\right).
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {1}} \left| \check {O} _ {1, j} ^ {t} - \widetilde {O} _ {1, j} ^ {t} \right| \leq \delta_ {2} L _ {1} p _ {n} + \delta_ {2} L _ {1} \sum_ {k = 1} ^ {L _ {1}} \left| \check {O} _ {1, k} ^ {t - 1} - \widetilde {O} _ {1, k} ^ {t - 1} \right| + r _ {v _ {1}} (E _ {n} + \delta_ {1}) \sum_ {k = 1} ^ {L _ {1}} \left| \check {O} _ {1, k} ^ {t - 1} - \widetilde {O} _ {1, k} ^ {t - 1} \right| \\ + \delta_ {2} L _ {1} \sum_ {k = 1} ^ {L _ {1}} | \check {O} _ {1, k} ^ {t - 1} |. \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {i}} \left| \check {O} _ {i, j} ^ {t} - \widetilde {O} _ {i, j} ^ {t} \right| \leq \delta_ {2} L _ {i} \sum_ {k = 1} ^ {L _ {i - 1}} \left| \check {O} _ {i - 1, k} ^ {t} - \widetilde {O} _ {i - 1, k} ^ {t} \right| + r _ {w _ {i}} \left(E _ {n} + \delta_ {1}\right) \sum_ {k = 1} ^ {L _ {i - 1}} \left| \check {O} _ {i - 1, k} ^ {t} - \widetilde {O} _ {i - 1, k} ^ {t} \right| \\ + \delta_ {2} L _ {i} \sum_ {k = 1} ^ {L _ {i - 1}} | \check {O} _ {i - 1, k} ^ {t} | + \delta_ {2} L _ {i} \sum_ {k = 1} ^ {L _ {i}} | \check {O} _ {i, k} ^ {t - 1} - \widetilde {O} _ {i, k} ^ {t - 1} | \\ + r _ {v _ {i}} \left(E _ {n} + \delta_ {1}\right) \sum_ {k = 1} ^ {L _ {i}} \left| \check {O} _ {i, k} ^ {t - 1} - \widetilde {O} _ {i, k} ^ {t - 1} \right| + \delta_ {2} L _ {i} \sum_ {k = 1} ^ {L _ {i}} \left| \check {O} _ {i, k} ^ {t - 1} \right|. \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {H _ {n}}} \left| \check {O} _ {H _ {n}, j} ^ {t} - \widetilde {O} _ {H _ {n}, j} ^ {t} \right| \leq \delta_ {2} L _ {H _ {n}} \sum_ {k = 1} ^ {L _ {H _ {n} - 1}} \left| \check {O} _ {H _ {n} - 1, k} ^ {t} - \widetilde {O} _ {H _ {n} - 1, k} ^ {t} \right| \\ + r _ {w _ {H _ {n}}} \left(E _ {n} + \delta_ {1}\right) \sum_ {k = 1} ^ {L _ {H _ {n} - 1}} \left| \check {O} _ {H _ {n} - 1, k} ^ {t} - \widetilde {O} _ {H _ {n} - 1, k} ^ {t} \right| \\ + \delta_ {2} L _ {H _ {n}} \sum_ {k = 1} ^ {L _ {H _ {n} - 1}} | \check {O} _ {H _ {n} - 1, k} ^ {t} |. \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {i}} \left| \check {O} _ {i, j} ^ {t} - \widetilde {O} _ {i, j} ^ {t} \right| \leq t ^ {i + 1} \delta_ {2} \left(p _ {n} L _ {1} + \sum_ {k = 1} ^ {i} L _ {k}\right) \left(\prod_ {k = 1} ^ {i} \left[ \delta_ {2} L _ {k} + r _ {w _ {k}} \left(E _ {n} + \delta_ {1}\right) \right]\right) \\ \times \left(\prod_ {k = 1} ^ {i} \left[ \delta_ {2} L _ {k} + r _ {v _ {k}} \left(E _ {n} + \delta_ {1}\right) \right]\right) ^ {t - 1}. \\ \end{array}
$$
$$
\begin{array}{l} | \mu (\boldsymbol {x} _ {t: 1}, \breve {\beta}) - \mu (\boldsymbol {x} _ {t: 1}, \widetilde {\beta}) | = \sum_ {j = 1} ^ {L _ {H _ {n}}} | \breve {O} _ {H _ {n}, j} ^ {t} - O _ {H _ {n}, j} ^ {t} | \\ \leq t ^ {H _ {n}} \delta_ {2} \left(p _ {n} L _ {1} + \sum_ {k = 1} ^ {H _ {n}} L _ {k}\right) \left(\prod_ {k = 1} ^ {H _ {n}} \left[ \delta_ {2} L _ {k} + r _ {w _ {k}} \left(E _ {n} + \delta_ {1}\right) \right]\right) \left(\prod_ {k = 1} ^ {H _ {n} - 1} \left[ \delta_ {2} L _ {k} + r _ {v _ {k}} \left(E _ {n} + \delta_ {1}\right) \right]\right) ^ {t - 1}. \\ \end{array}
$$
We can verify the above conclusion in a similar approach. The proof is completed by summation of the bound for  $|\mu(\pmb{x}_{t:1}, \pmb{\beta}) - |\mu(\pmb{x}_{t:1}, \check{\pmb{\beta}})|$  and  $|\mu(\pmb{x}_{t:1}, \check{\pmb{\beta}}) - \mu(\pmb{x}_{t:1}, \widetilde{\pmb{\beta}})|$ .
B Proofs on Posterior Consistency: A Single Time Series
To establish posterior consistency for DNNs with i.i.d. data, [20] utilized Proposition 1 from [53]. This lemma provides three sufficient conditions for proving posterior consistency for general statistical models with i.i.d. data, along with a posterior contraction rate. In this paper, we aim to establish posterior consistency for DNNs with stochastic processes, specifically time series, that are strictly stationary and  $\alpha$ -mixing with an exponentially decaying mixing coefficient [54, 42].
Consider a time series  $D_{n} = (z_{1},z_{2},\ldots ,z_{n})$  defined on a probability space  $(\Omega ,\mathcal{F},P^{*})$ , which satisfies the assumptions outlined in Assumption 1. For simplicity, we assume that the initial values  $z_{1 - l},\dots ,z_0$  are fixed and given.
Let  $\mathbb{P}_n$  denote a set of probability densities, let  $\mathbb{P}_n^c$  denote the complement of  $\mathbb{P}_n$ , and let  $\epsilon_{n}$  denote a sequence of positive numbers. Let  $N(\epsilon_n,\mathbb{P}_n)$  be the minimum number of Hellinger balls of radius  $\epsilon_{n}$  that are needed to cover  $\mathbb{P}_n$ , i.e.,  $N(\epsilon_n,\mathbb{P}_n)$  is the minimum of all number  $k$ 's such that there exist sets  $S_{j} = \{p:d(d(p,p_{j})\leq \epsilon_{n}\}, j = 1,\dots,k$ , with  $P_{n}\subset \cup_{j = 1}^{k}S_{j}$  holding, where
$$
d (p, q) = \sqrt {\iint (\sqrt {p (z | \boldsymbol {x})} - \sqrt {q (z | \boldsymbol {x})}) ^ {2} v (\boldsymbol {x}) d z d \boldsymbol {x}},
$$
denotes the integrated Hellinger distance [42, 54] between the two conditional densities  $p(z|\pmb{x})$  and  $q(z|\pmb{x})$ , where  $\pmb{x} \in \mathbb{R}^l$  contains the history up to  $l$  time steps of  $z$ , and  $v(\pmb{x})$  is the probability density function of the marginal distribution of  $\pmb{x}$ . Note that  $v$  is invariant with respect to time index  $i$  due to the strictly stationary assumption.
For  $D_{n}$ , denote the corresponding true conditional density by  $p^{*}$ . Define  $\pi(\cdot)$  as the prior density, and  $\pi(\cdot|D_{n})$  as the posterior. Define  $\hat{\pi}(\epsilon) = \pi[d(p,p^{*}) > \epsilon|D_{n}]$  for each  $\epsilon > 0$ . Assume the conditions:
(a)  $\log N(\epsilon_n,\mathbb{P}_n)\leq n\epsilon_n^2$  for all sufficiently large  $n$
(b)  $\pi (\mathbb{P}_n^c)\leq e^{-bn\epsilon_n^2}$  for some  $b > 0$  and all sufficiently large  $n$
(c) Let  $\| f\| _s = \left(\int |f|^s dr\right)^{1 / s}$ , then  $\pi (\| p - p^{*}\|_{s}\leq b^{\prime}\epsilon_{n})\geq e^{-\gamma n\epsilon_{n}^{2}}$  for some  $b^{\prime},\gamma >0,$ $s > 2$ , and all sufficiently large  $n$ .
Lemma B.1. Under the conditions (a), (b) and (c), given sufficiently large  $n$ ,  $\lim_{n\to \infty}\epsilon_n = 0$  and  $\lim_{n\to \infty}n\epsilon_n^2 = \infty$ , we have for some large  $C > 0$
$$
\hat {\pi} \left(C \epsilon_ {n}\right) = \pi \left(d \left(p, p ^ {*}\right) \geq C \epsilon_ {n} \mid D _ {n}\right) = O _ {P ^ {*}} \left(e ^ {- n \epsilon_ {n} ^ {2}}\right). \tag {7}
$$
Proof. This Lemma can be proved with similar arguments used in Section 9.5.3, Theorem 8.19, and Theorem 8.29 of [54].  $\square$
B.1 Posterior Consistency with a General Shrinkage Prior
Let  $\beta$  denote the vector of all connection weights of a RNN. To prove Theorem 3.9, we first consider a general shrinkage prior that all entries of  $\beta$  are subject to an independent continuous prior  $\pi_{b}$ , i.e.,  $\pi (\beta) = \prod_{j = 1}^{K_n}\pi_b(\beta_j)$ , where  $K_{n}$  denotes the total number of elements of  $\beta$ . Theorem B.2 provides a sufficient condition for posterior consistency.
Theorem B.2. (Posterior consistency) Suppose Assumptions 3.1 - 3.7 hold. If the prior  $\pi_b$  satisfies that
$$
\begin{array}{l} \log \left(1 / \underline {{\pi}} _ {b}\right) = O \left(H _ {n} \log n + \log \bar {L}\right), (8) \\ \pi_ {b} \left\{\left[ - \eta_ {n}, \eta_ {n} \right] \right\} \geq 1 - \frac {1}{K _ {n}} \exp \left\{- \tau \left[ H _ {n} \log n + \log \bar {L} + \log p _ {n} \right] \right\}, (9) \\ \pi_ {b} \left\{\left[ - \eta_ {n} ^ {\prime}, \eta_ {n} ^ {\prime} \right] \right\} \geq 1 - \frac {1}{K _ {n}}, (10) \\ - \log \left[ K _ {n} \pi_ {b} \left(\left| \beta_ {j} \right| > M _ {n}\right) \right] \succ n \epsilon_ {n} ^ {2}, (11) \\ \end{array}
$$
for some  $\tau > 0$ , where
$$
\eta_ {n} <   1 / \left[ \sqrt {n} M _ {l} ^ {H _ {n}} K _ {n} (n / H _ {n}) ^ {2 M _ {l} H _ {n}} \left(c _ {0} M _ {n}\right) ^ {2 M _ {l} H _ {n}} \right],
$$
$$
\eta_ {n} ^ {\prime} <   1 / \left[ \sqrt {n} M _ {l} ^ {H _ {n}} K _ {n} \left(r _ {n} / H _ {n}\right) ^ {2 M _ {l} H _ {n}} \left(c _ {0} E _ {n}\right) ^ {2 M _ {l} H _ {n}} \right],
$$
with some  $c_0 > 1$ ,  $\underline{\pi}_b$  is the minimal density value of  $\pi_b$  within the interval  $[-E_n - 1, E_n + 1]$ , and  $M_n$  is some sequence satisfying  $\log(M_n) = O(\log(n))$ . Then, there exists a sequence  $\epsilon_n$ , satisfying  $n\epsilon_n^2 \asymp r_nH_n\log n + r_n\log \bar{L} + s_n\log p_n + n\varpi_n^2$  and  $\epsilon_n \prec 1$ , such that
$$
\hat {\pi} (M \epsilon_ {n}) = \pi (d (p, p ^ {*}) \geq M \epsilon_ {n} | D _ {n}) = O _ {P ^ {*}} (e ^ {- n \epsilon_ {n} ^ {2}})
$$
for some large  $M > 0$
Proof. To prove this theorem, it suffices to check all three conditions listed in Lemma B.1.
Checking condition (c):
Consider the set  $A = \{\beta : \max_{j \in \gamma^*} |\beta_j - \beta_j^*| \leq \omega_n, \max_{j \notin \gamma^*} |\beta_j - \beta_j^*| \leq \omega_n'\}$ , where
$$
\omega_ {n} \leq \frac {c _ {1} \epsilon_ {n}}{M _ {l} ^ {H _ {n}} H _ {n} (r _ {n} / H _ {n}) ^ {2 M _ {l} H _ {n}} (c _ {0} E _ {n}) ^ {2 M _ {l} H _ {n}}},
$$
$$
\omega_ {n} ^ {\prime} \leq \frac {c _ {1} \epsilon_ {n}}{M _ {l} ^ {H _ {n}} K _ {n} (r _ {n} / H _ {n}) ^ {2 M _ {l} H _ {n}} (c _ {0} E _ {n}) ^ {2 M _ {l} H _ {n}}},
$$
for some  $c_{1} > 0$  and  $c_{0} > 1$ . If  $\beta \in A$ , by Lemma A.2, we have
$$
\left| \mu \left(\boldsymbol {x} _ {M _ {l}: 1}, \boldsymbol {\beta}\right) - \mu \left(\boldsymbol {x} _ {M _ {l}: 1}, \boldsymbol {\beta} ^ {*}\right) \right| \leq 3 c _ {1} \epsilon_ {n}.
$$
By the definition (4), we have
$$
\left| \mu \left(\boldsymbol {x} _ {M _ {l}: 1}, \beta\right) - \mu^ {*} \left(\boldsymbol {x} _ {M _ {l}: 1}\right) \right| \leq 3 c _ {1} \epsilon_ {n} + \varpi_ {n}
$$
Finally for some  $s > 2$  (for simplicity, we take  $s$  to be an even integer),
$$
\begin{array}{l} \left\| p _ {\boldsymbol {\beta}} - p _ {\mu^ {*}} \right\| _ {s} = \left(\int \left(| \mu (\boldsymbol {x} _ {M _ {l}: 1}, \boldsymbol {\beta}) - \mu^ {*} (\boldsymbol {x} _ {M _ {l}: 1}) |\right) ^ {s} v (\boldsymbol {x} _ {M _ {l}: 1}) d \boldsymbol {x} _ {M _ {l}: 1}\right) ^ {1 / s} \\ \leq \left(\int \left(3 c _ {1} \epsilon_ {n} + \varpi_ {n}\right) ^ {s} v \left(\boldsymbol {x} _ {M _ {l}: 1}\right) d \boldsymbol {x} _ {M _ {l}: 1}\right) ^ {1 / s} \\ \leq 3 c _ {1} \epsilon_ {n} + \varpi_ {n}. \\ \end{array}
$$
For any small  $b' > 0$ , condition (c) is satisfied as long as  $c_{1}$  is sufficiently small,  $\epsilon_{n} \geq C_{0}\varpi_{n}$  for some large  $C_0$ , and the prior satisfies  $-\log \pi (A) \leq \gamma n\epsilon_n^2$ . Since
$$
\begin{array}{l} \pi (A) \geq (\pi_ {b} ([ E _ {n} - \omega_ {n}, E _ {n} + \omega_ {n} ])) ^ {r _ {n}} \times \pi (\{\max  _ {j \notin \gamma^ {*}} | \beta_ {j} | \leq \omega_ {n} ^ {\prime} \}) \\ \geq \left(2 \pi_ {b} \omega_ {n}\right) ^ {r _ {n}} \times \pi_ {b} \left(\left[ - \omega_ {n} ^ {\prime}, \omega_ {n} ^ {\prime} \right]\right) ^ {K _ {n} - r _ {n}} \\ \geq \left(2 \pi_ {b} \omega_ {n}\right) ^ {r _ {n}} \left(1 - 1 / K _ {n}\right) ^ {K _ {n}}, \\ \end{array}
$$
where the last inequality is due to the fact that  $\omega_{n}^{\prime}\gg \eta_{n}^{\prime}$ . Note that  $\lim_{n\to \infty}(1 - 1 / K_n)^{K_n} = e^{-1}$ . Since  $\log (1 / \omega_{n})\asymp 2M_{l}H_{n}\log (E_{n}) + 2M_{l}H_{n}\log (r_{n} / H_{n}) + \log (1 / \epsilon_{n}) + \mathrm{constant} = O(H_{n}\log n)$  (note that  $\log (1 / \epsilon_n) = O(\log n)$ ), then for sufficiently large  $n$
$$
\begin{array}{l} - \log \pi (A) \leq r _ {n} \log \left(\frac {1}{\underline {{\pi}} _ {b}}\right) + r _ {n} O (H _ {n} \log (n)) + r _ {n} \log \left(\frac {1}{2}\right) + 1 \\ = O \left(r _ {n} H _ {n} \log (n) + r _ {n} \log \bar {L}\right). \\ \end{array}
$$
Thus, the prior satisfies  $-\log \pi(A) \leq \gamma n \epsilon_n^2$  for sufficiently large  $n$ , when  $n \epsilon_n^2 \geq C_0 (r_n H_n \log n + r_n \log \bar{L})$  for some sufficiently large constant  $C_0$ . Thus condition (c) holds.
Checking condition (a):
Let  $\mathbb{P}_n$  denote the set of probability densities for the RNNs whose parameter vectors satisfy
$$
\beta \in B _ {n} = \left\{\left| \beta_ {j} \right| \leq M _ {n}, \Gamma_ {\beta} = \left\{i: \left| \beta_ {i} \right| \geq \delta_ {n} ^ {\prime} \right\} \text {s a t i s f i e s} \left| \Gamma_ {\beta} \right| \leq k _ {n} r _ {n}, \left| \Gamma_ {\beta} \right| _ {i n} \leq k _ {n} ^ {\prime} s _ {n} \right\},
$$
where  $|\Gamma_{\beta}|_{in}$  denotes the number of input connections with the absolute weights greater than  $\delta_n'$ ,  $k_n(\leq n / r_n)$  and  $k_n'(\leq n / s_n)$  will be specified later, and
$$
\delta_ {n} ^ {\prime} = \frac {c _ {1} \epsilon_ {n}}{M _ {l} ^ {H _ {n}} K _ {n} (k _ {n} r _ {n} / H _ {n}) ^ {2 M _ {l} H _ {n}} (c _ {0} M _ {n}) ^ {2 M _ {l} H _ {n}}},
$$
for some  $c_{1} > 0, c_{0} > 0$ . Let
$$
\delta_ {n} = \frac {c _ {1} \epsilon_ {n}}{M _ {l} ^ {H _ {n}} H _ {n} (k _ {n} r _ {n} / H _ {n}) ^ {2 M _ {l} H _ {n}} (c _ {0} M _ {n}) ^ {2 M _ {l} H _ {n}}}.
$$
Consider two parameter vectors  $\beta^u$  and  $\beta^v$  in set  $B_{n}$ , such that there exists a structure  $\gamma$  with  $|\gamma| \leq k_n r_n$  and  $|\gamma|_{in} \leq k_n' s_n$ , and  $|\beta_j^u - \beta_j^v| \leq \delta_n$  for all  $j \in \gamma$ ,  $\max(|\beta_j^v|, |\beta_j^u|) \leq \delta_n'$  for all  $j \notin \gamma$ . Hence, by Lemma A.2, we have that  $|\mu(\pmb{x}_{1:M_l}, \pmb{\beta}^u) - \mu(\pmb{x}_{1:M_l}, \pmb{\beta}^v)|^2 \leq 9c_1^2 \epsilon_n^2$ . For two normal distributions  $N(\mu_1, \sigma^2)$  and  $N(\mu_2, \sigma^2)$ , define the corresponding Kullback-Leibler divergence as
$$
d _ {0} (p _ {\mu_ {1}}, p _ {\mu_ {2}}) = \frac {1}{2 \sigma^ {2}} | \mu_ {2} - \mu_ {1} | ^ {2}.
$$
Together with the fact that  $2d(p_{\mu_1},p_{\mu_2})^2\leq d_0(p_{\mu_1},p_{\mu_2})$  , we have
$$
d (p _ {\beta^ {u}}, p _ {\beta^ {v}}) \leq \sqrt {d _ {0} (p _ {\beta^ {u}} , p _ {\beta^ {v}})} \leq \sqrt {C (9 + o (1)) c _ {1} ^ {2} \epsilon_ {n} ^ {2}} \leq \epsilon_ {n}
$$
for some  $C > 0$ , given a sufficiently small  $c_{1}$ .
Given the above results, one can bound the packing number  $N(\mathbb{P}_n,\epsilon_n)$  by  $\sum_{j = 1}^{k_n r_n}\chi_{H_n}^j (\frac{2M_n}{\delta_n})^j$  where  $\chi_{H_n}^j$  denotes the number of all valid networks who has exact  $j$  connections and has no more than  $k_{n}^{\prime}s_{n}$  inputs. Since  $\log \chi_{H_n}^j\leq k's_n\log p_n + j\log \bigl (k's_nL_1 + 2H_n\bar{L}^2\bigr)$ ,  $\log M_{n} = O(\log n)$ ,  $k_{n}r_{n}\leq n$ , and  $k_{n}^{\prime}s_{n}\leq n$ , then
$$
\begin{array}{l} \log N (\mathbb {P} _ {n}, \epsilon_ {n}) \leq \log \left(k _ {n} r _ {n} \chi_ {H _ {n}} ^ {k _ {n} r _ {n}} \left(\frac {2 M _ {n}}{\delta_ {n}}\right) ^ {k _ {n} r _ {n}}\right) \\ \leq \log k _ {n} r _ {n} + k ^ {\prime} s _ {n} \log p _ {n} + k _ {n} r _ {n} \log \left(2 H _ {n}\right) + 2 k _ {n} r _ {n} \log \left(\bar {L} + k _ {n} ^ {\prime} s _ {n}\right) \\ + k _ {n} r _ {n} \log \frac {2 M _ {n} M _ {l} ^ {H _ {n}} H _ {n} \left(k _ {n} r _ {n} / H _ {n}\right) ^ {2 M _ {l} H _ {n}} \left(c _ {0} M _ {n}\right) ^ {2 M _ {l} H _ {n}}}{c _ {1} \epsilon_ {n}} \\ = k _ {n} r _ {n} O \left(H _ {n} \log n + \log \bar {L}\right) + k _ {n} ^ {\prime} s _ {n} \log p _ {n}. \\ \end{array}
$$
We can choose  $k_{n}$  and  $k_{n}^{\prime}$  such that for sufficiently large  $n$ ,  $k_{n}r_{n}\{H_{n}\log n + \log \bar{L}\} \asymp k_{n}^{\prime}s_{n}\log p_{n} \asymp n\epsilon_{n}^{2}$  and then  $\log N(\mathbb{P}_n,\epsilon_n)\leq n\epsilon_n^2$
Checking condition (b):
Lemma B.3. (Theorem 1 of [55]) Let  $X \sim \text{Binomial}(n,p)$  be a Binomial random variable. For any  $1 < k < n - 1$
$$
P r (X \geq k + 1) \leq 1 - \Phi (s i g n (k - n p) [ 2 n H (p, k / n) ] ^ {1 / 2}),
$$
where  $\Phi$  is the cumulative distribution function (CDF) of the standard Gaussian distribution and  $H(p, k/n) = (k/n)\log(k/np) + (1 - k/n)\log[(1 - k/n)/(1 - p)]$ .
$$
\pi (\mathbb {P} _ {n} ^ {c}) \leq P r (\operatorname {B i n o m i a l} (K _ {n}, v _ {n}) > k _ {n} r _ {n}) + K _ {n} \pi_ {b} (| \beta_ {j} | > M _ {n}) + P r (| \Gamma_ {\beta} | _ {i n} > k _ {n} ^ {\prime} s _ {n}),
$$
where  $v_{n} = 1 - \pi_{b}([-\delta_{n}^{\prime},\delta_{n}^{\prime}])$ . By the condition on  $\pi_b$  and the fact that  $\delta_n^\prime \gg \eta_n^\prime$ , it is easy to see that  $v_{n}\leq \exp \{\tau [H_{n}\log n + \log \bar{L} +\log p_{n}] - \log K_{n}\}$  for some constant  $\tau >0$ . Thus, by Lemma B.3,  $-\log Pr(\mathrm{Binomial}(K_n,v_n) > k_n r_n)\approx \tau k_n r_n[H_n\log n + \log \bar{L} +\log p_n]\gtrsim n\epsilon_n^2$  due to the choice of  $k_{n}$ , and  $-\log Pr(|\Gamma_{\beta}|_{in}\geq k_n's_n)\approx k_n's_n[\tau (H_n\log n + \log \bar{L} +\log p_n) + \log (p_nK_n / L_1)]\gtrsim n\epsilon_n^2$  due to the choice of  $k_{n}^{\prime}$ . Thus, condition (b) holds as well. Then by lemma B.1, the proof is completed.
B.2 Proof of Theorem 3.9
Proof. To prove Theorem 3.9 in the main text, it suffices to verify the four conditions on  $\pi_b$  listed in Theorem B.2. Let  $M_n = \max(\sqrt{2n}\sigma_{1,n}, E_n)$ . Condition 8 can be verified by choosing  $\sigma_{1,n}$  such that  $E_n^2 / 2\sigma_{1,n}^2 + \log \sigma_{1,n}^2 = O(H_n \log n + \log \bar{L})$ . Conditions 9 and 10 can be verified by setting  $\lambda_n = 1 / \{M_l^{H_n}K_n[n^{2M_lH_n}(\bar{L}p_n)]^\tau\}$  and  $\sigma_{0,n} \prec 1 / \{M_l^{H_n}\sqrt{n}K_n(n / H_n)^{2M_lH_n}(c_0M_n)^{2M_lH_n}\}$ . Finally, condition 11 can be verified by  $M_n \geq 2n\sigma_{0,n}^2$  and  $\tau[H_n \log n + \log \bar{L} + \log p_n] + M_n^2 / 2\sigma_{1,n}^2 \geq n$ . Finally, based on the proof above we see that  $n\epsilon_n^2 \asymp r_nH_n \log n + r_n \log \bar{L} + s_n \log p_n + n\varpi_n^2$  and  $\lim_{n \to \infty} \epsilon_n = 0$ .
B.3 Posterior Consistency for Multilayer Perceptrons