SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As highlighted in Section 3, the MLP can be formulated as a regression problem. Here, the input is  $\boldsymbol{x}_i = y_{i-1:i-R_l}$  for some  $l \leq R_l \ll n$ , with the corresponding output being  $y_i$ . Thus, the dataset  $D_n$  can be represented as  $(\boldsymbol{x}_i, y_i)_{i=1+R_l}^n$ . We apply the same assumptions as for the MLP, specifically 3.1, 3.4, and 3.7. We also use the same definitions and notations for the MLP as those in [20, 21].
Leveraging the mathematical properties of sparse MLPs presented in [20] and the proof of sparse RNNs for a single time series discussed above, one can straightforwardly derive the following Corollary. Let  $d(p_{1},p_{2})$  denote the integrated Hellinger distance between two conditional densities  $p_1(y|\boldsymbol {x})$  and  $p_2(y|\boldsymbol {x})$ . Let  $\pi (\cdot |D_n)$  be the posterior probability of an event.
Corollary B.4. Suppose Assumptions 3.1, 3.4, and 3.7 hold. If the mixture Gaussian prior (3) satisfies the conditions:  $\lambda_{n} = O(1 / [K_{n}[n^{H_{n}}(\bar{L} p_{n})]^{\tau}])$  for some constant  $\tau >0$ ,  $E_{n} / [H_{n}\log n + \log \bar{L}]^{1 / 2}\leq \sigma_{1,n}\leq n^{\alpha}$  for some constant  $\alpha$ , and  $\sigma_{0,n}\leq \min \{1 / [\sqrt{n} K_n(n^{3 / 2}\sigma_{1,n} / H_n)^{H_n}],1 / [\sqrt{n} K_n(nE_n / H_n)^{H_n}]\}$ , then there exists an error sequence  $\epsilon_{n}^{2} = O(\varpi_{n}^{2}) + O(\zeta_{n}^{2})$  such that  $\lim_{n\to \infty}\epsilon_n = 0$  and  $\lim_{n\to \infty}n\epsilon_n^2 = \infty$ , and the posterior distribution satisfies
$$
\pi \left(d \left(p _ {\beta}, p _ {\mu^ {*}}\right) \geq C \epsilon_ {n} \mid D _ {n}\right) = O _ {P ^ {*}} \left(e ^ {- n \epsilon_ {n} ^ {2}}\right) \tag {12}
$$
for sufficiently large  $n$  and  $C > 0$ , where  $\zeta_n^2 = [r_nH_n\log n + r_n\log \bar{L} +s_n\log p_n] / n$ ,  $p_{\mu^*}$  denotes the underlying true data distribution, and  $p_\beta$  denotes the data distribution reconstructed by the Bayesian MLP based on its posterior samples.
C Asymptotic Normality of Connection Weights and Predictions
This section provides detailed assumptions and proofs for Theorem 3.12 and 3.13. For simplicity, we assume  $y_{0:1 - R_l}$  is also given, and we let  $M_{l} = R_{l} + 1$ . Let  $l_{n}(\beta) = \frac{1}{n}\sum_{i = 1}^{n}\log (p_{\beta}(y_i|y_{i - 1:i - R_l}))$  denote the likelihood function, and let  $\pi (\beta)$  denote the density of the mixture Gaussian prior (3). Let  $h_{i_1,i_2,\ldots ,i_d}(\beta) = \frac{\partial^d l_n(\beta)}{\partial\beta_{i_1}\cdots\partial\beta_{i_d}}$  which denotes the  $d$ -th order partial derivatives. Let  $H_{n}(\beta)$  denote the Hessian matrix of  $l_{n}(\beta)$ , and let  $h_{i,j}(\beta), h^{i,j}(\beta)$  denote the  $(i,j)$ -th component of  $H_{n}(\beta)$  and  $H_{n}^{-1}(\beta)$ , respectively. Let  $B_{\lambda ,n} = \bar{\lambda}_n^{1 / 2}(\beta^*) / \underline{\lambda}_n(\beta^*)$  and  $b_{\lambda ,n} = \sqrt{r_n / n} B_{\lambda ,n}$ . For a RNN with  $\beta$ , we define the weight truncation at the true model structure  $\gamma^{*}:(\beta_{\gamma^{*}})_{i} = \beta_{i}$  for  $i\in \gamma^{*}$  and  $(\beta_{\gamma^{*}})_i = 0$  for  $i\notin \gamma^{*}$ . For the mixture Gaussian prior (3), let  $B_{\delta_n}(\beta^*) = \{\beta :|\beta_i - \beta_i^* |\leq \delta_n,\forall i\in \gamma^*,|\beta_i - \beta_i^* |\leq 2\sigma_{0,n}\log \bigg(\frac{\sigma_{1,n}}{\lambda_n\sigma_{0,n}}\bigg),\forall i\notin \gamma^*\}$ .
In addition, we make the following assumptions for Theorem 3.12 and Theorem 3.13.
Assumption C.1. Assume the conditions of Lemma 3.10 hold with  $\rho(\epsilon_n) = o\left(\frac{1}{K_n}\right)$  and the  $C_1 \geq \frac{2}{3}$  defined in Assumption 3.4. For some  $\delta_n$  s.t.  $\frac{r_n}{\sqrt{n}} \lesssim \delta_n \lesssim \frac{1}{\sqrt[3]{n}r_n}$ , let  $A(\epsilon_n, \delta_n) =$
$\{\beta : \max_{i \in \gamma^*} |\beta_i - \beta_i^*| > \delta_n, d(p_\beta, p_\mu^*) \leq \epsilon_n\}$ , where  $\epsilon_n$  is the posterior contraction rate as defined in Theorem 3.9. Assume there exists some constants  $C > 2$  and  $M > 0$  such that
C.1  $\beta^{*} = (\beta_{1},\dots ,\beta_{K_{n}})$  is generic,  $\min_{i\in \gamma^*}|\beta_i^* | > C\delta_n$  and  $\pi (A(\epsilon_n,\delta_n)|D_n)\to 0$  as  $n\to \infty$
C.2  $|h_i(\beta^*)| < M, |h_{j,k}(\beta^*)| < M, |h^{j,k}(\beta^*)| < M, |h_{i,j,k}(\beta)| < M, |h_l(\beta)| < M$  hold for any  $i,j,k \in \gamma^*, l \notin \gamma^*$ , and  $\beta \in B_{2\delta_n}(\beta^*)$ .
C.3  $\sup \{|E_{\beta}(a'U^3)|: \| \beta_{\gamma^*} - \beta^*\| \leq 1.2b_{\lambda,n}, \| a \| = 1 \leq 0.1\sqrt{n / r_n}\underline{\lambda}_n^2(\beta^*) / \bar{\lambda}_n^{1/2}(\beta^*)\}$  and  $B_{\lambda,n} = O(1)$ , where  $U = Z - E_{\beta_{\gamma^*}}(Z)$ ,  $Z$  denotes a random variable drawn from a neural network model parameterized by  $\beta_{\gamma^*}$ , and  $E_{\beta_{\gamma^*}}$  denotes the mean of  $Z$ .
C.4  $r_n^2 / n \to 0$  and the conditions for Theorem 2 of [56] hold.
Conditions  $C.1$  to  $C.3$  align with the assumptions made in [21]. An additional assumption,  $C.4$ , is introduced to account for the dependency inherent in time series data. This assumption is crucial and employed in conjunction with  $C.3$  to establish the consistency of the maximum likelihood estimator (MLE) of  $\beta_{\gamma^*}$  for a given structure  $\gamma^*$ . While the assumption  $r_n / n \to 0$  is sufficient for independent data, which is implied by Assumption 3.4, a stronger restriction, specifically  $r_n^2 / n \to 0$ , is necessary for dependent data such as time series. It is worth noting that the conditions used in Theorem 2 of [56] pertain specifically to the time series data itself.
Let  $\mu_{i_1,i_2,\dots ,i_d}(\cdot ,\beta) = \frac{\partial^d\mu(\cdot,\beta)}{\partial\beta_{i_1}\cdots\partial\beta_{i_d}}$  denotes the d-th order partial derivative for some input.
Assumption C.2.  $|\mu_i(\cdot, \beta^*)| < M, |\mu_{i,j}(\cdot, \beta)| < M, |\mu_k(\cdot, \beta)| < M$  for any  $i, j \in \gamma^*$ ,  $k \notin \gamma^*$ , and  $\beta \in B_{2\delta_n}(\beta^*)$ , where  $M$  is as defined in Assumption C.1.
Proof of Theorem 3.12 and Theorem 3.13 The proof of these two theorems can be conducted following the approach in [21]. The main difference is that in [21], they rely on Theorem 2.1 of [57], which assumes independent data. However, the same conclusions can be extended to time series data by using Theorem 2 of [56], taking into account assumption C.4. This allows for the consideration of dependent data in the analysis.
D Computation
Algorithm 1 gives the prior annealing procedure[21]. In practice, the following implementation can be followed based on Algorithm 1:
- For  $0 < t < T_{1}$ , perform initial training.
- For  $T_1 \leq t \leq T_2$ , set  $\sigma_{0,n}^{(t)} = \sigma_{0,n}^{init}$  and gradually increase  $\eta^{(t)} = \frac{t - T_1}{T_2 - T_1}$ .
- For  $T_2 \leq t \leq T_3$ , set  $\eta^{(t)} = 1$  and gradually decrease  $\sigma_{0,n}^{(t)}$  according to the formula:  $\sigma_{0,n}^{(t)} = \left(\frac{T_3 - t}{T_3 - T_2}\right)(\sigma_{0,n}^{init})^2 + \left(\frac{t - T_2}{T_3 - T_2}\right)(\sigma_{0,n}^{end})^2$ .
- For  $t > T_3$ , set  $\eta^{(t)} = 1$ ,  $\sigma_{0,n}^{(t)} = \sigma_{0,n}^{end}$ , and gradually decrease the temperature  $\tau$  according to the formula:  $\tau = \frac{c}{t - T_3}$ , where  $c$  is a constant.
Please refer to Appendix E for intuitive explanations of the prior annealing algorithm and further details on training a model to achieve the desired sparsity.
E Mixture Gaussian Prior
The mixture Gaussian prior imposes a penalty on the model parameters by acting as a piecewise L2 penalty, applying varying degrees of penalty in different regions of the parameter space. Given values for  $\sigma_{0,n}$ ,  $\sigma_{1,n}$ , and  $\lambda_{n}$ , a threshold value can be computed using Algorithm
Algorithm 1 Prior Annealing: Frequentist
[1] (Initial Training) Train a neural network satisfying condition  $(S^{*})$  such that a global optimal solution  $\beta_0 = \arg \max_{\beta} l_n(\beta)$  is reached, this stage can be accomplished by using SGD or Adam.
[2] (Prior Annealing) Initialize  $\beta$  at  $\beta_0$ , and simulate from a sequence of distributions  $\pi(\beta|D_n, \tau, \eta^{(k)}, \sigma_{0,n}^{(k)}) \propto e^{nl_n(\beta)/\tau} \pi_k^{\eta^{(k)}/\tau}(\beta)$  for  $k = 1, 2, \ldots, m$ , where  $0 < \eta^{(1)} \leq \eta^{(2)} \leq \dots \leq \eta^{(m)} = 1$ ,  $\pi_k = \lambda_n N(0, \sigma_{1,n}^2) + (1 - \lambda_n) N(0, (\sigma_{0,n}^{(k)})^2)$ , and  $\sigma_{0,n}^{init} = \sigma_{0,n}^{(1)} \geq \sigma_{0,n}^{(2)} \geq \dots \geq \sigma_{0,n}^{(m)} = \sigma_{0,n}^{end}$ . This can be done by using stochastic gradient MCMC algorithms [58]. After the stage  $m$  has been reached, continue to run the simulated annealing algorithm by gradually decreasing the temperature  $\tau$  to a very small value. Denote the resulting model by  $\hat{\beta}$ .
[3] (Structure Sparsification) For each connection weight  $i \in \{1, 2, \ldots, K_n\}$ , set  $\tilde{\gamma}_i = 1$ , if  $|\beta_i| > \frac{\sqrt{2} \sigma_{0,n} \sigma_{1,n}}{\sqrt{\sigma_{1,n}^2 - \sigma_{0,n}^2}} \sqrt{\log \left( \frac{1 - \lambda_n}{\lambda_n} \frac{\sigma_{1,n}}{\sigma_{0,n}} \right)}$  and 0 otherwise, where the threshold value is determined by solving  $\pi(\gamma_i = 1 | \beta) > 0.5$ , and  $\sigma_{0,n} = \sigma_{0,n}^{end}$ . Denote the structure of the sparse RNN by  $\tilde{\gamma}$ .
[4] (Nonzero-weights Refining) Refine the nonzero weights of the sparse model  $(\hat{\beta},\tilde{\gamma})$  by maximizing  $l_{n}(\beta)$ . Denote the resulting estimate by  $(\tilde{\beta},\tilde{\gamma})$ .
Figure 3: Changes of the negative log prior density function in prior annealing of  $\sigma_{0,n}^2$ , where the values of  $\lambda_{n}$  and  $\sigma_{1,n}^2$  are fixed.
1. Parameters whose absolute values are below this threshold will receive a large penalty, hence constituting the "penalty region", while parameters whose absolute values are above the threshold will receive relatively minimal penalty, forming the "free space". The severity of the penalty in the free space largely depends on the value of  $\sigma_{1,n}$ .
For instance, as depicted in Figure 3, based on the simple practice implementation detailed in Appendix D, we fix  $\lambda_{n} = 1e - 7$ ,  $\sigma_{1,n}^{2} = 0.05$ , and we set  $(\sigma_{0,n}^{init})^2 = 1e - 5$ ,  $(\sigma_{0,n}^{end})^2 = 1e - 7$ , and gradually reduce  $\sigma_{0,n}^2$  from the initial value to the end value. Initially, the free space is relatively small and the penalty region is relatively large. However, the penalty imposed on the parameters within the initial penalty region is also minor, making it challenging to shrink these parameters to zero. As we progressively decrease  $\sigma_{0,n}^2$ , the free space enlarges,
the penalty region diminishes, and the penalty on the parameters within the penalty region intensifies simultaneously. Once  $\sigma_{0,n}^2$  equals  $(\sigma_{0,n}^{end})^2$ , the parameters within the penalty region will be proximate to zero, and the parameters outside the penalty region can vary freely in almost all areas of the parameter space.
For model compression tasks, achieving the desired sparsity involves several steps post the initial training phase outlined in Algorithm 1. First, determine the initial threshold value based on pre-set values for  $\lambda_{n}$ ,  $\sigma_{1,n}^{2}$ , and  $(\sigma_{0,n}^{init})^{2}$ . Then, compute the proportion of the parameters in the initial model whose absolute values are lower than this threshold. This value serves as an estimate for anticipated sparsity. Adjust the  $(\sigma_{0,n}^{init})^{2}$  value until the predicted sparsity aligns closely with the desired sparsity level.
F Construction of Multi-Horizon Joint Prediction Intervals
To construct valid joint prediction intervals for multi-horizon forecasting, we estimate the individual variances for each time step in the prediction horizon using procedures similar to one-step-ahead forecasting. We then adjust the critical value by dividing  $\alpha$  by the number of time steps  $m$  in the prediction horizon and use  $z_{\alpha/(2m)}$  as the adjusted critical value. This Bonferroni correction ensures the desired coverage probability across the entire prediction horizon.
As an example, we refer to the experiments conducted in 5.1.2, where we work with a set of training sequences denoted by  $D_{n} = \{y_{1}^{(i)},\ldots ,y_{T}^{(i)},\ldots y_{T + m}^{(i)}\}_{i = 1}^{n}$ . Here,  $y_{T + 1}^{(i)},y_{T + 2}^{(i)},\dots ,y_{T + m}^{(i)}$  represent the prediction horizon for each sequence, while  $T$  represents the length of the observed sequence.
- Train a model by the proposed algorithm, and denote the trained model by  $(\hat{\beta},\hat{\gamma})$
Calculate  $\hat{\pmb{\sigma}}^2$  as an estimator of  $\pmb {\sigma}^2\in \mathbb{R}^{m\times 1}$
$$
\hat {\boldsymbol {\sigma}} ^ {2} = \frac {1}{n - 1} \sum_ {i = 1} ^ {n} \left(\boldsymbol {\mu} (\hat {\boldsymbol {\beta}}, y _ {1: T} ^ {(i)}) - y _ {T + 1: T + m} ^ {(i)}\right) \otimes \left(\boldsymbol {\mu} (\hat {\boldsymbol {\beta}}, y _ {1: T} ^ {(i)}) - y _ {T + 1: T + m} ^ {(i)}\right),
$$
where  $\pmb{\mu}(\hat{\pmb{\beta}},y_{1:T}^{(i)}) = \hat{y}_{T + 1:T + m}^{(i)}$ , and  $\otimes$  denotes elementwise product.
- For a test sequence  $y_{1:T}^{(0)}$ , calculate
$$
\hat {\pmb {\Sigma}} = \nabla_ {\hat {\pmb {\gamma}}} \pmb {\mu} (\hat {\pmb {\beta}}, y _ {1: T} ^ {(0)}) ^ {\prime} (- \nabla_ {\hat {\pmb {\gamma}}} ^ {2} l _ {n} (\hat {\pmb {\beta}})) ^ {- 1} \nabla_ {\hat {\pmb {\gamma}}} \pmb {\mu} (\hat {\pmb {\beta}}, y _ {1: T} ^ {(0)}).
$$
Let  $\hat{\pmb{\varsigma}}\in \mathbb{R}^{m\times 1}$  denote the vector formed by the diagonal elements of  $\hat{\Sigma}$ .
- The Bonferroni simultaneous prediction intervals for all elements of  $y_{T + 1:T + m}^{(0)}$  are given by
$$
\boldsymbol {\mu} (\hat {\boldsymbol {\beta}}, y _ {1: T} ^ {(0)}) \pm z _ {\alpha / (2 m)} \left(\frac {1}{n} \hat {\boldsymbol {\varsigma}} + \hat {\boldsymbol {\sigma}} ^ {2}\right) ^ {\circ \frac {1}{2}},
$$
where  $\circ \frac{1}{2}$  represents the element-wise square root operation.
The Bayesian method is particularly advantageous when dealing with a large number of non-zero connection weights in  $(\hat{\beta},\hat{\gamma})$ , making the computation of the Hessian matrix of the log-likelihood function costly or unfeasible. For detailed information on utilizing the Bayesian method for constructing prediction intervals, please refer to [21].
G.1 French Electricity Spot Prices
Dataset. The given dataset contains the spot prices of electricity in France that were established over a period of four years, from 2016 to 2020, using an auction market. In
this market, producers and suppliers submit their orders for the next day's 24-hour period, specifying the electricity volume in MWh they intend to sell or purchase, along with the corresponding price in €/MWh. At midnight, the Euphemia algorithm, as described in [59], calculates the 24 hourly prices for the next day, based on the submitted orders and other constraints. This hourly dataset consists of 35064 observations, covering  $(3 \times 365 + 366) \times 24$  periods. Our main objective is to predict the 24 prices for the next day by considering different explanatory variables such as the day-ahead forecast consumption, day-of-the-week, as well as the prices of the previous day and the same day in the previous week, as these variables are crucial in determining the spot prices of electricity. Refer to Figure 4 for a visual representation of the dataset.
The prediction models and training settings described below are the same for all 24 hours.
(b)
Figure 4: (a): French electricity spot prices (2016 to 2019). (b): Online learning method used for ACI and AgACI in [31], where the prediction range is assumed to be between  $T_{\mathrm{pred start}}$  and  $T_{\mathrm{pred end}}$ .
Prediction Model. We use an MLP with one hidden layer of size 100 and the sigmoid activation function as the underlying prediction model for all methods and all hours.
Training: Baselines. For all CP baselines, we train the MLP for 300 epochs using SGD with a constant learning rate of 0.001 and momentum of 0.9, the batch size is set to be 100. For the ACI, we use the same list of \(\gamma \in \{0,0.000005,0.00005,0.0001,0.0002,0.0003,0.0004,0.0005,0.0001,0.0002,0.0003,0.0004, 0.0005,0.0006,0.0007,0.0008,0.0009,0.001,0.002,0.003,0.004,0.005,0.006,0.007,0.008,0.009, 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09\} as in [31].
Training: Our Method. For our method PA, we train a total of 300 epochs with the same learning rate, momentum, and batch size. We use  $T_{1} = 150$ ,  $T_{2} = 160$ ,  $T_{3} = 260$ . We run SGD with momentum for  $t < T_{1}$  and SGHMC with temperature= 1 for  $t >= T_{1}$ . For the mixture Gaussian prior, we fix  $\sigma_{1,n}^{2} = 0.01$ ,  $(\sigma_{1,n}^{init})^{2} = 1e - 5$ ,  $(\sigma_{1,n}^{end})^{2} = 1e - 6$ , and  $\lambda_{n} = 1e - 7$ .
Table 5: Uncertainty quantification results produced by different methods for the French electricity spot prices dataset. This table corresponds to the Figure 2.
Methods	Coverage	Average PI Lengths (standard deviation)	Median PI Lengths (interquartile range)
PA offline (ours)	90.06	20.19(3.23)	20.59(1.82)
AgACI online	91.05	23.27(14.49)	21.38(2.62)
ACI γ = 0.01 online	90.21	22.13(8.50)	20.68(2.70)
ACI γ = 0.0 online	92.89	24.39(9.17)	22.99(2.91)
EnbPI V2 online	91.23	27.99(10.17)	26.55(5.09)
NexCP online	91.18	24.41(10.40)	22.88(2.86)
G.2 EEG, MIMIC-III, and COVID-19
EEG The EEG dataset, available at here, served as the primary source for the EEG signal time series. This dataset contains responses from both control and alcoholic subjects who were presented with visual stimuli of three different types. To maintain consistency with
previous work [33], we utilized the medium-sized version, consisting of 10 control and 10 alcoholic subjects. We focused solely on the control subjects for our experiments, as the dataset summaries indicated their EEG responses were more difficult to predict. For detailed information on our processing steps, please refer to [33].
COVID-19 We followed the processing steps of previous research [33] and utilized COVID-19 data from various regions within the same country to minimize potential distribution shifts while adhering to the exchangeability assumption. The lower tier local authority split provided us with 380 sequences, which we randomly allocated between the training, calibration, and test sets over multiple trials. The dataset can be found at here.
MIMIC-III We collect patient data on the use of antibiotics (specifically Levofloxacin) from the MIMIC-III dataset [46]. However, the subset of the MIMIC-III dataset used in [33] is not publicly available to us, and the authors did not provide information on the processing steps, such as SQL queries and Python code. Therefore, we follow the published processing steps provided in [60]. We remove sequences with fewer than 5 visits or more than 47 visits, resulting in a total of 2992 sequences. We randomly split the dataset into train, calibration, and test sets, with corresponding proportions of  $43\%$ ,  $47\%$ , and  $10\%$ . We use the white blood cell count (high) as the feature for the univariate time series from these sequences. Access to the MIMIC-III dataset requires PhysioNet credentialing.
Table 6 presents the training hyperparameters and prediction models used for the baselines in the three datasets, while Table 7 presents the training details used for our method.
We hypothesize that the mediocre performance on the EEG dataset is due to the small size of the prediction model, which has only 2025 parameters and is substantially smaller than the number of training sequences. Hence, we conducted experiments on the EEG dataset using a slightly larger prediction model for different methods. The results are presented in Table 8, and the corresponding training details are provided in Table 6. Again, our method outperforms other baselines as well.
Table 6: Training details for the baselines, where LSTM-  $d$  refers to a single hidden layer LSTM network with a size of  $d$ .
MIMIC-III	EEG	COVID-19
model	LSTM-500	LSTM-20	LSTM-20
learning rate	2e-4	0.01	0.01
Epochs	65	100	1000
Optimizer	Adam	Adam	Adam
Batch size	150	150	150
Table 7: Training details for our method, where LSTM-  $d$  refers to a single hidden layer LSTM network with a size of  $d$ .
MIMIC-III	EEG	COVID-19
Model	LSTM-500	LSTM-20	LSTM-20
Learning rate	2e-4	0.01	0.01
Epochs	65	100	1000
Optimizer	Adam	Adam	Adam
Batch size	150	150	150
σ21,n	0.05	0.1	0.1
(σ0,n)2	1e-5	1e-7	1e-7
(σ0,n)2	1e-6	1e-8	1e-8
λn	1e-7	1e-7	1e-7
Temperature	1	1	1
T1	40	50	800
T2	40	55	850
T3	55	80	950
Table 8: Uncertainty quantification results produced by various methods for the EEG data using a larger prediction model. The "coverage" represents the average joint coverage rate over different random seeds. The prediction interval (PI) lengths are averaged across prediction horizons and random seeds. The values in parentheses indicate the standard deviations of the respective means.
EEG
Model	Coverage	PI lengths
PA-RNN	91.0%(0.8%)	40.84(5.69)
CF-RNN	90.8%(1.8%)	43.4(6.79)
MQ-RNN	45.2%(2.5%)	20.63(2.07)
DP-RNN	0.6%(0.1%)	5.02(0.53)
Table 9: Training details for the EEG data with a slightly larger prediction model for all methods, where "- " denotes that the information is not applicable.
PA-RNN	CF-RNN	MQ-RNN	DP-RNN
model	LSTM-100	LSTM-100	LSTM-100	LSTM-100
learning rate	1e-3	1e-3	1e-3	1e-3
Epochs	150	150	150	150
Optimizer	Adam	Adam	Adam	Adam
Batch size	150	150	150	150
σ21,n	0.01	-	-	-
(σint0,n)2	1e-5	-	-	-
(σend0,n)2	1e-6	-	-	-
λn	1e-7	-	-	-
temperature	1	-	-	-
T1	100	-	-	-
T2	105	-	-	-
T3	130	-	-	-
G.3 Autoregressive Order Selection
Model An Elman RNN with one hidden layer of size 1000. Different window sizes (i.e., 1 or 15) will result in a different total number of parameters..
Hyperparameters All training hyperparameters are given in Table 10
Table 10: Training details for the autoregressive order selection experiment.
PA-RNN 1	PA-RNN 15	RNN 1	RNN 15
Learning rate	4e-3	1e-4	1e-4	1e-4
Iterations	25000	25000	25000	25000
Optimizer	SGHMC	SGHMC	SGHMC	SGHMC
Batch size	36	36	36	36
Subsample size per iteration	50	50	50	50
Predicton horizon	1	1	1	1
σ21,n	0.05	0.05	-	-
(σ0,n)2	2e-6	4e-6	-	-
(σend0,n)2	1e-7	1e-7	-	-
λn	1e-7	1e-7	-	-
Temperature	0.1	0.1	-	-
T1	5000	5000	-	-
T2	10000	10000	-	-
T3	25000	25000	-	-
G.4 Large-Scale Model Compression
As pointed out by recent summary/survey papers on sparse deep learning [61, 62], the lack of standardized benchmarks and metrics that provide guidance on model structure, task, dataset, and sparsity levels has caused difficulties in conducting fair and meaningful comparisons with previous works. For example, the task of compressing Penn Tree Bank (PTB) word language model [63] is a popular comparison task. However, many previous works [64, 65, 66, 67] have avoided comparison with the state-of-the-art method by either not using the standard baseline model, not reporting, or conducting comparisons at different sparsity levels. Therefore, we performed an extensive search of papers that reported performance on this task, and to the best of our knowledge, the state-of-the-art method is the Automated Gradual Pruning (AGP) by [68].