SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.12. Assume the conditions of Lemma 3.10 hold with  $\rho(\epsilon_n) = o\left(\frac{1}{K_n}\right)$  and additional assumptions hold given in Appendix C, then  $\pi(\sqrt{n}(\tilde{\nu}(\beta) - \beta^*)|D_n) \rightsquigarrow N(\mathbf{0}, \mathbf{V})$  in  $P^*$ -probability as  $n \to \infty$ , where  $\mathbf{V} = (v_{i,j})$ , and  $v_{i,j} = E(h^{i,j}(\beta^*))$  if  $i, j \in \gamma^*$  and 0 otherwise.
Theorem 3.13 establishes asymptotic normality of the sparse RNN prediction, which implies prediction consistency and forms the theoretical basis for prediction uncertainty quantification as well.
Theorem 3.13. Assume the conditions of Theorem 3.12 and additional assumptions hold given in Appendix C. Then  $\pi (\sqrt{n} (\mu (\cdot ,\beta) - \mu (\cdot ,\beta^{*})|D_{n})\rightsquigarrow N(0,\Sigma)$ , where  $\Sigma = \nabla_{\gamma^{*}}\mu (\cdot ,\beta^{*})^{\prime}H^{-1}\nabla_{\gamma^{*}}\mu (\cdot ,\beta^{*})$  and  $H = E(-\nabla_{\gamma^{*}}^{2}l_{n}(\beta^{*}))$  is the Fisher information matrix.
4 Computation
In the preceding section, we established a theoretical foundation for sparse deep learning with time series data under the Bayesian framework. Building on [20], it is straightforward to show that Bayesian computation can be simplified by invoking the Laplace approximation theorem at the maximum a posteriori (MAP) estimator. This essentially transforms the proposed Bayesian method into a regularization method by interpreting the log-prior density function as a penalty for the log-likelihood function in RNN training. Consequently, we can train the regularized RNN model using an optimization algorithm, such as SGD or Adam. To address the local-trap issue potentially suffered by these methods, we train the regularized RNN using a prior annealing algorithm [21], as described in Algorithm 1. For a trained RNN, we sparsify its structure by truncating the weights less than a threshold to zero and further refine the nonzero weights for attaining the MAP estimator. For algorithmic
specifics, refer to Appendix D. Below, we outline the steps for constructing prediction intervals for one-step-ahead forecasts, where  $\mu (y_{k - 1:k - R_l},\hat{\beta})$  is of one dimension, and  $\hat{\beta}$  and  $\hat{\gamma}$  represent the estimators of the network parameters and structure, respectively, as obtained by Algorithm 1:
Estimate  $\sigma^2$  by  $\hat{\sigma}^2 = \frac{1}{n - R_l - 1}\sum_{i = R_l + 1}^{n}\| y_i - \mu (y_{i - 1:i - R_l},\hat{\beta})\| ^2.$
- For a test point  $y_{k - 1:k - R_l}$ , estimate  $\Sigma$  in Theorem 3.13 by
$$
\hat {\varsigma} ^ {2} = \nabla_ {\hat {\boldsymbol {\gamma}}} \mu (y _ {k - 1: k - R _ {l}}, \hat {\boldsymbol {\beta}}) ^ {\prime} (- \nabla_ {\hat {\boldsymbol {\gamma}}} ^ {2} l _ {n} (\hat {\boldsymbol {\beta}})) ^ {- 1} \nabla_ {\hat {\boldsymbol {\gamma}}} \mu (y _ {k - 1: k - R _ {l}}, \hat {\boldsymbol {\beta}}).
$$
- The corresponding  $(1 - \alpha)\%$  prediction interval is given by
$$
\mu \left(y _ {k - 1: k - R _ {l}}, \hat {\beta}\right) \pm z _ {\alpha / 2} \sqrt {\hat {\varsigma} ^ {2} / (n - R _ {l}) + \hat {\sigma} ^ {2}},
$$
where there are  $n - R_{l}$  observations used in training, and  $z_{\alpha /2}$  denotes the upper  $\alpha /2$  -quantile of the standard Gaussian distribution.
For construction of multi-horizon prediction intervals, see Appendix F.
5.1 Uncertainty Quantification
As mentioned in Section 3, we will consider two types of time series datasets: the first type comprises a single time series, and the second type consists of a set of time series. We will compare the performance of our method against the state-of-the-art Conformal Prediction (CP) methods for both types of datasets. We set  $\alpha = 0.1$  (the error rate) for all uncertainty quantification experiments in the paper, and so the nominal coverage level of the prediction intervals is  $90\%$ .
5.1.1 A Single Time Series: French Electricity Spot Prices
We perform one-step-ahead forecasts on the French electricity spot prices data from 2016 to 2020, which consists of 35,064 observations. A detailed description and visualization of this time series are given in Appendix G.1. Our goal is to predict the 24 hourly prices of the following day, given the prices up until the end of the current day. As the hourly prices exhibit distinct patterns, we fit one model per hour as in the CP baseline [31]. We follow the data splitting strategy used in [31], where the first three years 2016 - 2019 data are used as the (initial) training set, and the prediction is made for the last year 2019 - 2020.
For all the methods considered, we use the same underlying neural network model: an MLP with one hidden layer of size 100 and the sigmoid activation function. More details on the training process are provided in Appendix G.1. For the state-of-the-art CP methods, EnbPI-V2 [29], NEX-CP [30], ACI [32] and AgACI [31], we conduct experiments in an online fashion, where the model is trained using a sliding window of the previous three years of data (refer to Figure 4 in the Appendix). Specifically, after constructing the prediction interval for each time step in the prediction period, we add the ground truth value to the training set and then retrain the model with the updated training set. For ACI, we conduct experiments with various values of  $\gamma$  and present the one that yields the best performance. In the case of AgACI, we adopt the same aggregation approach as used in [31], namely, the Bernstein Online Aggregation (BOA) method [44] with a gradient trick. We also report the performance of ACI with  $\gamma = 0$  as a reference. For NEX-CP, we use the same weights as those employed in their time-series experiments. For EnbPI-V2, we tune the number of bootstrap models and select the one that offers the best performance.
Since this time series exhibits no or minor distribution shift, our method PA is trained in an offline fashion, where the model is fixed for using only the observations between 2016 and 2019, and the observations in the prediction range are used only for final evaluation. That is, our method uses less data information in training compared to the baseline methods.
Figure 2: Comparisons of our method and baselines on hourly spot electricity prices in France. Left: Average prediction length with vertical error bars indicating the standard deviation of the prediction interval lengths. Right: Median prediction length with vertical error bars indicating the interquartile range of the prediction interval lengths. The precise values for these metrics are provided in Table 5 of Appendix G.1.
The results are presented in Figure 2, which includes empirical coverage (with methods that are positioned closer to 90.0 being more effective), median/average prediction interval length, and corresponding interquartile range/standard deviation. As expected, our method is able to train and calibrate the model by using only the initial training set, i.e., the data for 2016-2019, and successfully produces faithful prediction intervals. In contrast, all CP methods produce wider prediction intervals than ours and higher coverage rates than the nominal level of  $90\%$ . In addition, ACI is sensitive to the choice of  $\gamma$  [31].
5.1.2 A Set of Time Series
We conduct experiments using three publicly available real-world datasets: Medical Information Mart for Intensive Care (MIMIC-III), electroencephalography (EEG) data, and daily COVID-19 case numbers within the United Kingdom's local authority districts (COVID-19) [33]. A concise overview of these datasets is presented in Table 1. Our method, denoted as PA-RNN (since the underlying prediction model is an LSTM [45]), is compared to three benchmark methods: CF-RNN [33], MQ-RNN [35], and DP-RNN [36], where LSTM is used for all these methods. To ensure a fair comparison, we adhere to the same model structure, hyperparameters, and data processing steps as specified in [33]. Detailed information regarding the three datasets and training procedures can be found in Appendix G.2.
The numerical results are summarized in Tables 2. Note that the baseline results for EEG and COVID-19 are directly taken from the original paper [33]. We reproduce the baseline results for MIMIC-III, as the specific subset used by [33] is not publicly available. Table 2 indicates that our method consistently outperforms the baselines. In particular, Our method consistently generates shorter prediction intervals compared to the conformal baseline CF-RNN while maintaining the same or even better coverage rate as CF-RNN. Both the MQ-RNN and DP-RNN methods fail to generate prediction intervals that accurately maintain a faithful coverage rate.
Table 1: A brief description of the datasets used in Section 5.1.2, where "Train size" and "Test size" indicate the numbers of training and testing sequences, respectively, and "Length" represents the length of the input sequence.
Dataset	Train size	Test size	Length	Prediction horizon
MIMIC-III [46]	2692	300	[3, 47]	2
EEG [47]	19200	19200	40	10
COVID-19 [48]	300	80	100	50
Table 2: Uncertainty quantification results by different methods for the MIMIC-III, EEG, and COVID-19 data, where "Coverage" represents the joint coverage rate averaged over five different random seeds, the prediction interval length is averaged across prediction horizons and five random seeds, and the numbers in the parentheses indicate the standard deviations (across five random seeds) of the respective means.
MIMIC-III	EEG	COVID-19
Model	Coverage	PI length	Coverage	PI length	Coverage	PI length
PA-RNN	90.8%(0.7%)	2.35(0.26)	94.69%(0.4%)	51.02(10.50)	90.5%(1.4%)	444.93(203.28)
CF-RNN	92.0%(0.3%)	3.01(0.17)	96.5%(0.4%)	61.86(18.02)	89.7%(2.4%)	733.95(582.52)
MQ-RNN	85.3%(0.2%)	2.64(0.11)	48.0%(1.8%)	21.39(2.36)	15.0%(2.6%)	136.56(63.32)
DP-RNN	1.2%(0.3%)	0.12(0.01)	3.3%(0.3%)	7.39(0.74)	0.0%(0.0%)	61.18(32.37)
5.2 Autoregressive Order Selection
In this section, we evaluate the performance of our method in selecting the autoregressive order for two synthetic autoregressive processes. The first is the non-linear autoregressive (NLAR) process [49, 50, 51]:
$$
y _ {i} = - 0. 1 7 + 0. 8 5 y _ {i - 1} + 0. 1 4 y _ {i - 2} - 0. 3 1 y _ {i - 3} + 0. 0 8 y _ {i - 7} + 1 2. 8 0 G _ {1} \left(y _ {i - 1: i - 7}\right) + 2. 4 4 G _ {2} \left(y _ {i - 1: i - 7}\right) + \eta_ {i},
$$
where  $\eta_i \sim N(0,1)$  represents i.i.d. Gaussian random noises, and the functions  $G_1$  and  $G_2$  are defined as:
$$
\begin{array}{l} G _ {1} \left(y _ {i - 1: i - 7}\right) = \left(1 + \exp \{- 0. 4 6 \left(0. 2 9 y _ {i - 1} - 0. 8 7 y _ {i - 2} + 0. 4 0 y _ {i - 7} - 6. 6 8\right) \}\right) ^ {- 1}, \\ G _ {2} \left(y _ {i - 1: i - 7}\right) = \left(1 + \exp \left\{- 1. 1 7 \times 1 0 ^ {- 3} \left(0. 8 3 y _ {i - 1} - 0. 5 3 y _ {i - 2} - 0. 1 8 y _ {i - 7} + 0. 3 8\right) \right\}\right) ^ {- 1}. \\ \end{array}
$$
The second is the exponential autoregressive process [52]:
$$
y _ {i} = \left(0. 8 - 1. 1 \exp \left\{- 5 0 y _ {i - 1} ^ {2} \right\}\right) y _ {i - 1} + \eta_ {i},
$$
where, again,  $\eta_{i}\sim N(0,1)$  denotes i.i.d. Gaussian random noises.
For both synthetic processes, we generate five datasets. Each dataset consists of training, validation, and test sequences. The training sequence has 10000 samples, while the validation and test sequences each contain 1000 samples. For training, we employ a single-layer RNN with a hidden layer width of 1000. Further details on the experimental setting can be found in Appendix G.3.
For the NLAR process, we consider two different window sizes for RNN modeling: 15 (with input as  $y_{i - 1:i - 15}$ ) and 1 (with input as  $y_{i - 1}$ ). Notably, the NLAR process has an order of 7. In the case where the window size is 15, the input information suffices for RNN modeling, rendering the past information conveyed by the hidden states redundant. However, when the window size is 1, this past information becomes indispensable for the RNN. In contrast, the exponential autoregressive process has an order of 1. For all window sizes we explored, namely 1,3,5,7,10,15, the input information is always sufficient for RNN modeling.
We evaluate the predictive performance using mean square prediction error (MSPE) and mean square fitting error (MSFE). The model selection performance is assessed by two metrics: the false selection rate (FSR) and the negative selection rate (NSR). We define
$\mathrm{FSR} = \frac{\sum_{j=1}^{5} |\hat{S}_j / S|}{\sum_{j=1}^{5} |\hat{S}_j|}$  and  $\mathrm{NSR} = \frac{\sum_{j=1}^{5} |S / \hat{S}_j|}{\sum_{i=j}^{5} |S|}$ , where  $S$  denotes the set of true variables, and
$\hat{S}_j$  represents the set of selected variables for dataset  $j$ . Furthermore, we provide the final number of nonzero connections for hidden states and the estimated autoregressive orders. The numerical results for the NLAR process are presented in Table 3, while the numerical results for the exponential autoregressive process are given in Table 4.
Our results are promising. Specifically, when the window size is equal to or exceeds the true autoregressive order, all connections associated with the hidden states are pruned, effectively converting the RNN into an MLP. Conversely, if the window size is smaller than the true autoregressive order, a significant number of connections from the hidden states
are retained. Impressively, our method accurately identifies the autoregressive order—a noteworthy achievement considering the inherent dependencies in time series data. Although our method produces a nonzero FSR for the NLAR process, it is quite reasonable considering the relatively short time sequence and the complexity of the functions  $G_{1}$  and  $G_{2}$ .
Table 3: Numerical results for the NLAR process: numbers in the parentheses are standard deviations of the respective means, "-" indicates not applicable,  $\downarrow$  means lower is better, and "  $\#$  hidden link" denotes the number of nonzero connections from the hidden states. All results are obtained from five independent runs.
Model	Window size	FSR ↓	NSR ↓	AR order	#hidden link	MSPE ↓	MSFE ↓
PA-RNN	1	0	0	-	357(21)	1.056(0.001)	1.057(0.006)
PA-RNN	15	0.23	0	7.4(0.25)	0(0)	1.017(0.008)	1.020(0.010)
Table 4: Numerical results for the exponential autoregressive process.
Model	Window size	FSR ↓	NSR ↓	AR order	#hidden link	MSPE ↓	MSFE ↓
PA-RNN	1	0	0	1(0)	0(0)	1.004(0.004)	1.003(0.005)
PA-RNN	3	0	0	1(0)	0(0)	1.006(0.005)	0.999(0.004)
PA-RNN	5	0	0	1(0)	0(0)	1.000(0.004)	1.007(0.005)
PA-RNN	7	0	0	1(0)	0(0)	1.006(0.005)	1.000(0.003)
PA-RNN	10	0	0	1(0)	0(0)	1.002(0.004)	1.002(0.006)
PA-RNN	15	0	0	1(0)	0(0)	1.001(0.004)	1.002(0.007)
5.3 RNN Model Compression
We have also applied our method to RNN model compression, achieving state-of-the-art results. Please refer to Section G.4 in the Appendix for details.
This paper has established the theoretical groundwork for sparse deep learning with time series data, including posterior consistency, structure selection consistency, and asymptotic normality of predictions. Our empirical studies indicate that sparse deep learning can outperform current cutting-edge approaches, such as conformal predictions, in prediction uncertainty quantification. More specifically, compared to conformal methods, our method maintains the same coverage rate, if not better, while generating significantly shorter prediction intervals. Furthermore, our method effectively determines the autoregression order for time series data and surpasses state-of-the-art techniques in large-scale model compression. The theory developed in this paper has included LSTM [45] as a special case, and some numerical examples have been conducted with LSTM; see Section G of the Appendix for the detail. Furthermore, there is room for refining the theoretical study under varying mixing assumptions for time series data, which could broaden applications of the proposed method. Also, the efficacy of the proposed method can potentially be further improved with the elicitation of different prior distributions.
In summary, this paper represents a significant advancement in statistical inference for deep RNNs, which, through sparsing, has successfully integrated the RNNs into the framework of statistical modeling. The superiority of our method over the conformal methods shows further the criticality of consistently approximating the underlying mechanism of the data generation process in uncertainty quantification.
In terms of limitations of the proposed method, one potential concern pertains to the calculation of the inverse of the Fisher information matrix. For large-scale problems, the sparsified model could still retain a large number of non-zero parameters. In such instances, the computational feasibility of calculating the Hessian matrix might become compromised. Nonetheless, an alternative avenue exists in the form of the Bayesian approach, which circumvents the matrix inversion challenge. A concise description of this Bayesian strategy can be found in [21].
[63] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014.
[64] Nadezhda Chirkova, Ekaterina Lobacheva, and Dmitry Vetrov. Bayesian compression for natural language processing. arXiv preprint arXiv:1810.10927, 2018.
[65] Maxim Kodryan, Artem Grachev, Dmitry Ignatov, and Dmitry Vetrov. Efficient language modeling with automatic relevance determination in recurrent neural networks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 40-48, 2019.
[66] Ekaterina Lobacheva, Nadezhda Chirkova, and Dmitry Vetrov. Bayesian sparsification of gated recurrent neural networks. arXiv preprint arXiv:1812.05692, 2018.
[67] Artem M Grachev, Dmitry I Ignatov, and Andrey V Savchenko. Compression of recurrent neural networks for efficient language modeling. Applied Soft Computing, 79:354-362, 2019.
[68] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
Appendix for "Sparse Deep Learning for Time Series Data: Theory and Applications"
A Mathematical Facts of Sparse RNNs
For a sparse RNN model with  $H_{n} - 1$  recurrent layers and a single output layer, several mathematical facts can be established. Let's denote the number of nodes in each layer as  $L_{1},\ldots ,L_{H_{n}}$ . Additionally, let  $r_{w_i}$  represent the number of non-zero connection weights for  $\pmb{w}^i$ , and let  $r_{v_i}$  denote the number of non-zero connection weights for  $\pmb{v}^i$ , where  $\pmb{w}_i$  and  $\pmb{v}_i$  denote the connection weights at two layers of a sparse RNN.
Furthermore, we define  $O_{i,j}^{t}(\beta ,\pmb{x}_{t:1})$  as the output value of the  $j$ -th neuron of the  $i$ -th layer at time  $t$ . This output depends on the parameter vector  $\beta$  and the input sequence  $\pmb{x}_{t:1} = (\pmb{x}_t,\dots ,\pmb{x}_1)$ , where  $\pmb{x}_t$  represents the generic input to the sparse RNN at time step  $t$ .
Lemma A.1. Under Assumption 3.7, if a sparse RNN has at most  $r_n$  non-zero connection weights (i.e.,  $r_n = \sum_{i=1}^{H_n} r_{w_i} + \sum_{i=1}^{H_n-1} r_{v_i}$ ) and  $\|\beta\|_{\infty} = E_n$ , then the summation of the absolute outputs of the  $i$ th layer at time  $t$  is bounded by
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {i}} \left| O _ {i, j} ^ {t} (\boldsymbol {\beta}, \boldsymbol {x} _ {t: 1}) \right| \leq t ^ {i} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n}) ^ {t i}, \quad 1 \leq i \leq H _ {n} - 1, \\ \sum_ {j = 1} ^ {L _ {H _ {n}}} \left| O _ {H _ {n}, j} ^ {t} (\boldsymbol {\beta}, \boldsymbol {x} _ {t: 1}) \right| \leq t ^ {H _ {n}} \left(\prod_ {k = 1} ^ {H _ {n}} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {H _ {n} - 1} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n}) ^ {t (H _ {n} - 1) + 1}. \\ \end{array}
$$
Proof. For simplicity, we rewrite  $O_{i,j}^{t}(\beta, \boldsymbol{x}_{t:1})$  as  $O_{i,j}^{t}$  when appropriate. The lemma is the result from the following facts:
- For  $t = 1$  and  $1 \leq i \leq H_{n}$  (Lemma S4 from [20])
$$
\sum_ {j = 1} ^ {L _ {i}} \left| O _ {i, j} ^ {1} \right| \leq E _ {n} ^ {i} \prod_ {k = 1} ^ {i} r _ {w _ {k}}.
$$
- For  $t \geq 2$  and  $i = 1$  (recursive relationship)
$$
\sum_ {j = 1} ^ {L _ {1}} \left| O _ {1, j} ^ {t} \right| \leq E _ {n} r _ {w _ {1}} + E _ {n} r _ {v _ {1}} \sum_ {j = 1} ^ {L _ {1}} \left| O _ {1, j} ^ {t - 1} \right|.
$$
- For  $t \geq 2$  and  $2 \leq i \leq H_{n} - 1$  (recursive relationship)
$$
\sum_ {j = 1} ^ {L _ {i}} | O _ {i, j} ^ {t} | \leq E _ {n} r _ {w _ {i}} \sum_ {j = 1} ^ {L _ {i - 1}} | O _ {i - 1, j} ^ {t} | + E _ {n} r _ {v _ {i}} \sum_ {j = 1} ^ {L _ {i}} | O _ {i, j} ^ {t - 1} |.
$$
- For  $t \geq 2$  and  $i = H_{n}$  (recursive relationship)
$$
\sum_ {j = 1} ^ {L _ {H _ {n}}} \left| O _ {H _ {n}, j} ^ {t} \right| \leq E _ {n} r _ {w _ {H _ {n}}} \sum_ {j = 1} ^ {L _ {H _ {n} - 1}} \left| O _ {H _ {n} - 1, j} ^ {t} \right|.
$$
We can verify this lemma by plugging the conclusion into all recursive relationships. We show the steps to verify the case when  $t \geq 2$  and  $2 \leq i \leq H_{n} - 1$ , since other cases are trivial.
to verify.
$$
\begin{array}{l} \sum_ {j = 1} ^ {L _ {i}} \left| O _ {i, j} ^ {t} \right| \leq E _ {n} r _ {w _ {i}} \sum_ {j = 1} ^ {L _ {i - 1}} \left| O _ {i - 1, j} ^ {t} \right| + E _ {n} r _ {v _ {i}} \sum_ {j = 1} ^ {L _ {i}} \left| O _ {i, j} ^ {t - 1} \right| \\ \leq t ^ {i - 1} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i - 1} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n}) ^ {t (i - 1) + 1} \\ + (t - 1) ^ {i} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 2} r _ {v _ {i}} \left(E _ {n}\right) ^ {(t - 1) i + 1} \\ \leq t ^ {i - 1} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n}) ^ {t i} + (t - 1) ^ {i} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n}) ^ {t i} \\ \leq t ^ {i} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right) \left(\prod_ {k = 1} ^ {i} r _ {v _ {k}}\right) ^ {t - 1} (E _ {n}) ^ {t i}, \\ \end{array}
$$
where the last inequality is due to the fact that
$$
\frac {t ^ {i - 1} + (t - 1) ^ {i}}{t ^ {i}} = \frac {1}{t} + \left(\frac {t - 1}{t}\right) ^ {i} \leq \frac {1}{t} + \frac {t - 1}{t} = 1.
$$
Lemma A.2. Under Assumption 3.7, consider two RNNs,  $\beta$  and  $\widetilde{\beta}$ , where the former one is a sparse network satisfying  $\| \beta \|_0 = r_n$  and  $\| \beta \|_{\infty} = E_n$ , and it's network structure vector is denoted by  $\gamma$ . Assume that if  $|\beta_i - \widetilde{\beta}_i| < \delta_1$  for all  $i \in \gamma$  and  $|\beta_i - \widetilde{\beta}_i| < \delta_2$  for all  $i \notin \gamma$ , then
$$
\begin{array}{l} \max  _ {\boldsymbol {x} _ {t: 1}} | \mu (\boldsymbol {x} _ {t: 1}, \beta) - \mu (\boldsymbol {x} _ {t: 1}, \widetilde {\beta}) | \\ \leq t ^ {H _ {n}} \delta_ {1} H _ {n} \left(\prod_ {k = 1} ^ {H _ {n}} r _ {w _ {k}}\right) (E _ {n} + \delta_ {1}) ^ {t (H _ {n} + 1) - 2} \left(\prod_ {k = 1} ^ {H _ {n} - 1} r _ {v _ {k}}\right) ^ {t - 1} \\ + t ^ {H _ {n}} \delta_ {2} (p _ {n} L _ {1} + \sum_ {k = 1} ^ {H _ {n}} L _ {k}) \left(\prod_ {k = 1} ^ {H _ {n}} [ \delta_ {2} L _ {k} + r _ {w _ {k}} (E _ {n} + \delta_ {1}) ]\right) \left(\prod_ {k = 1} ^ {H _ {n} - 1} [ \delta_ {2} L _ {k} + r _ {v _ {k}} (E _ {n} + \delta_ {1}) ]\right) ^ {t - 1}. \\ \end{array}
$$
Proof. Define  $\breve{\beta}$  such that  $\breve{\beta}_i = \widetilde{\beta}_i$  for all  $i \in \gamma$  and  $\breve{\beta}_i = 0$  for all  $i \notin \gamma$ . Let  $\check{O}_{i,j}^t$  denote  $O_{i,j}^t (\breve{\beta},\boldsymbol{x}_{t:1})$ . Then, from the following facts that
$$
\sum_ {j = 1} ^ {L _ {i}} \left| \check {O} _ {i, j} ^ {1} - O _ {i, j} ^ {1} \right| \leq i \delta_ {1} \left(E _ {n} + \delta_ {1}\right) ^ {i - 1} \left(\prod_ {k = 1} ^ {i} r _ {w _ {k}}\right).
$$