Title: Sparse Deep Learning for Time Series Data: Theory and Applications
Venue: NeurIPS 2023
Accepted: yes
Keywords: Sparse Deep Learning

Uncertainty Quantification

Model Compression

Variable Selection

Dependent Data
Abstract: Sparse deep learning has become a popular technique for improving the performance of deep neural networks in areas such as uncertainty quantification, variable selection, and large-scale network compression. However, most existing research has focused on problems where the observations are independent and identically distributed (i.i.d.), and there has been little work on the problems where the observations are dependent, such as time series data and sequential data in natural language processing. This paper aims to address this gap by studying the theory for sparse deep learning with dependent data. We show that sparse recurrent neural networks (RNNs) can be consistently estimated, and their predictions are asymptotically normally distributed under appropriate assumptions, enabling the prediction uncertainty to be correctly quantified. Our numerical results show that sparse deep learning outperforms state-of-the-art methods, such as conformal predictions, in prediction uncertainty quantification for time series data. Furthermore, our results indicate that the proposed method can consistently identify the autoregressive order for time series data and outperform existing methods in large-scale model compression. Our proposed method has important practical implications in fields such as finance, healthcare, and energy, where both accurate point estimates and prediction uncertainty quantification are of concern.

Sparse Deep Learning for Time Series Data: Theory and Applications

Abstract

Sparse deep learning has become a popular technique for improving the performance of deep neural networks in areas such as uncertainty quantification, variable selection, and large-scale network compression. However, most existing research has focused on problems where the observations are independent and identically distributed (i.i.d.), and there has been little work on the problems where the observations are dependent, such as time series data and sequential data in natural language processing. This paper aims to address this gap by studying the theory for sparse deep learning with dependent data. We show that sparse recurrent neural networks (RNNs) can be consistently estimated, and their predictions are asymptotically normally distributed under appropriate assumptions, enabling the prediction uncertainty to be correctly quantified. Our numerical results show that sparse deep learning outperforms state-of-the-art methods, such as conformal predictions, in prediction uncertainty quantification for time series data. Furthermore, our results indicate that the proposed method can consistently identify the autoregressive order for time series data and outperform existing methods in large-scale model compression. Our proposed method has important practical implications in fields such as finance, healthcare, and energy, where both accurate point estimates and prediction uncertainty quantification are of concern.

1 Introduction

Over the past decade, deep learning has experienced unparalleled triumphs across a multitude of domains, such as time series forecasting [1, 2, 3, 4, 5], natural language processing [6, 7], and computer vision [8, 9]. However, challenges like generalization and miscalibration [10] persist, posing potential risks in critical applications like medical diagnosis and autonomous vehicles.

In order to enhance the performance of deep neural networks (DNNs), significant research efforts have been dedicated to exploring optimization methods and the loss surface of the DNNs, see, e.g., [11, 12, 13, 14, 15, 16], which have aimed to expedite and direct the convergence of DNNs towards regions that exhibit strong generalization capabilities. While these investigations are valuable, effectively addressing both the challenges of generalization

Faming Liang

and miscalibration require additional and perhaps more essential aspects: consistent estimation of the underlying input-output mapping and complete knowledge of the asymptotic distribution of predictions. As a highly effective method that addresses both challenges, sparse deep learning has been extensively studied, see, e.g., [17, 18, 19, 20, 21]. Nevertheless, it is important to note that all the studies have been conducted under the assumption of independently and identically distributed (i.i.d.) data. However, in practice, we frequently encounter situations where the data exhibits dependence, such as time series data.

The primary objective of this paper is to address this gap by establishing a theoretical foundation for sparse deep learning with time series data. Specifically, we lay the foundation within the Bayesian framework. For RNNs, by letting their parameters be subject to a mixture Gaussian prior, we establish posterior consistency, structure selection consistency, inputoutput mapping estimation consistency, and asymptotic normality of predicted values. We validate our theory through numerical experiments on both synthetic and real-world datasets. Our approach outperforms existing state-of-the-art methods in uncertainty quantification and model compression, highlighting its potential for practical applications where both accurate point prediction and prediction uncertainty quantification are of concern.

Additionally, we would elaborate the contribution of this paper in a broader context of statistical modeling. As discussed in [22], two distinct cultures exist for statistical modeling: the 'data modeling culture' and the 'algorithmic modeling culture'. The former focuses on simple generative models that explain the data, potentially lacking a consistent estimate of the true data-generating mechanism due to the model's inherent simplicity. The latter, on the other hand, aims to find models that can predict the data regardless of complexity. Our proposed method occupies a middle ground between these two cultures. It seeks to identify a parsimonious model within the realm of complex models, while also ensuring a consistent estimation of the true data-generating mechanism. From this perspective, this work and related ones, e.g., [17, 18, 19, 20, 21], represent a hybridization of the 'algorithmic modeling culture' and the 'data modeling culture', which holds the potential to expedite advancements in modern data science.

2 Related Works

Sparse deep learning . Theoretical investigations have been conducted on the approximation power of sparse DNNs across different classes of functions [23, 24]. Recently, [20] has made notable progress by integrating sparse DNNs into the framework of statistical modeling, which offers a fundamentally distinct neural network approximation theory. Unlike traditional theories that lack data involvement and allow connection weights to assume values in an unbounded space to achieve arbitrarily small approximation errors with small networks [25], their theory [20] links network approximation error, network size, and weight bounds to the training sample size. They show that a sparse DNN of size O ( n/ log( n )) can effectively approximate various types of functions, such as affine and piecewise smooth functions, as n → ∞ , where n denotes the training sample size. Additionally, sparse DNNs exhibit several advantageous theoretical guarantees, such as improved interpretability, enabling the consistent identification of relevant variables for high-dimensional nonlinear systems. Building upon this foundation, [21] establishes the asymptotic normality of connection weights and predictions, enabling valid statistical inference for predicting uncertainties. This work extends the sparse deep learning theory of [20, 21] from the case of i.i.d data to the case of time series data.

Uncertainty quantification . Conformal Prediction (CP) has emerged as a prominent technique for generating prediction intervals, particularly for black-box models like neural networks. A key advantage of CP is its capability to provide valid prediction intervals for any data distribution, even with finite samples, provided the data meets the condition of exchangeability [26, 27]. While i.i.d. data easily satisfies this condition, dependent data, such as time series, often doesn't. Researchers have extended CP to handle time series data by relying on properties like strong mixing and ergodicity [28, 29]. In a recent work, [30] introduced a random swapping mechanism to address potentially non-exchangeable data, allowing conformal prediction to be applied on top of a model trained with weighted samples. The main focus of this approach was to provide a theoretical basis for the

differences observed in the coverage rate of the proposed method. Another recent study by [31] took a deep dive into the Adaptive Conformal Inference (ACI) [32], leading to the development of the Aggregation Adaptive Conformal Inference (AgACI) method. In situations where a dataset contains a group of similar and independent time series, treating each time series as a separate observation, applying a CP method becomes straightforward [33]. For a comprehensive tutorial on CP methods, one can refer to [34]. Beyond CP, other approaches for addressing uncertainty quantification in time series datasets include multi-horizon probabilistic forecasting [35], methods based on dropout [36], and recursive Bayesian approaches [37].

3 Sparse Deep Learning for Time Series Data: Theory

Let D n = { y 1 , . . . , y n } denote a time series sequence, where y i ∈ R . Let (Ω , F , P ∗ ) be the probability space of D n , and let α k = sup {| P ∗ ( y j ∈ A,y k + j ∈ B ) -P ∗ ( y j ∈ A ) P ∗ ( y k + j ∈ B ) | : A,B ∈ F , j ∈ N + } be the k -th order α -mixing coefficient.

Assumption 3.1. The time series D n is (strictly) stationary and α -mixing with an exponentially decaying mixing coefficient and follows an autoregressive model of order l

where µ ∗ is a non-linear function, y i -1: i -l = ( y i -1 , . . . , y i -l ), u i contains optional exogenous variables, and η i i.i.d. ∼ N (0 , σ 2 ) with σ 2 being assumed to be a constant.

Remark 3.3 . Extending the results of [20, 21] to the case that the dataset includes a set of i.i.d. time series, with each time series regarded as an individual observation, is straightforward, this is because all observations are independent and have the same distribution.

Remark 3.2 . Similar assumptions are commonly adopted to establish asymptotic properties of stochastic processes [38, 39, 40, 41, 42, 29]. For example, the asymptotic normality of the maximum likelihood estimator (MLE) can be established under the assumption that the time series is strictly stationary and ergodic, provided that the model size is fixed [38]. A posterior contraction rate of the autoregressive (AR) model can be obtained by assuming it is α -mixing with ∑ ∞ k =0 a 1 -2 /s k < ∞ for some s > 2 which is implied by an exponentially decaying mixing coefficient [42]. For stochastic processes that are strictly stationary and β -mixing, results such as uniform laws of large numbers and convergence rates of the empirical processes [39, 40] can also be obtained.

3.1 Posterior Consistency

Both the MLP and RNN can be used to approximate µ ∗ as defined in (1), and for simplicity, we do not explicitly denote the exogenous variables u i unless it is necessary. For the MLP, we can formulate it as a regression problem, where the input is x i = y i -1: i -R l for some l ≤ R l /lessmuch n , and the corresponding output is y i , then the dataset D n can be expressed as { ( x i , y i ) } n i =1+ R l . Detailed settings and results for the MLP are given in Appendix B.3. In what follows, we will focus on the RNN, which serves as an extension of the previous studies. For the RNN, we can rewrite the training dataset as D n = { y i : i -M l +1 } n i = M l for some l ≤ R l < M l /lessmuch n , i.e., we split the entire sequence into a set of shorter sequences, where R l denotes an upper bound for the exact AR order l , and M l denotes the length of these shorter sequences (see Figure 1). We assume R l is known but not l since, in practice, it is unlikely that we know the exact order l .

For simplicity of notations, we do not distinguish between weights and biases of the RNN. In this paper, the presence of the subscript n in the notation of a variable indicates its potential to increase with the sample size n . To define an RNN with H n -1 hidden layers, for h ∈ { 1 , 2 , . . . , H n } , we let ψ h and L h denote, respectively, the nonlinear activation function and the number of hidden neurons at layer h . We set L H n = 1 and L 0 = p n , where p n denotes a generic input dimension. Because of the existence of hidden states from the past, the input x i can contain only y i -1 or y i -1: i -r for some r > 1. Let w h ∈ R L h × L h -1 and v h ∈ R L h × L h denote the weight matrices at layer h . With these notations, the output of the step i of an RNN model can be expressed as

giM+R+1

yi

yiM+k:iM+1

yi-1:ik

where z h i = ψ h [ w h z h -1 i + v h z h i -1 ] denotes the hidden state of layer h at step i with z h 0 = 0 ; and β is the collection of all weights, consisting of K n = ∑ H n h =1 ( L h × L h -1 )+ ∑ H n -1 h =1 ( L 2 h )+ L h elements. To represent the structure for a sparse RNN, we introduce an indicator variable for each weight in β . Let γ = { γ j ∈ { 0 , 1 } : j = 1 , . . . , K n } , which specifies the structure of a sparse RNN. To include information on the network structure γ and keep the notation concise, we redenote µ ( x i , { z h i -1 } H n -1 h =1 , β ) by µ ( x i : i -M l +1 , β , γ ), as { z h i -1 } H n -1 h =1 depends only on ( β , γ ) and up to x i -1: i -M l +1 .

Posterior consistency is an essential concept in Bayesian statistics, which forms the basis of Bayesian inference. While posterior consistency generally holds for low-dimensional problems, establishing it becomes challenging in high-dimensional scenarios. In such cases, the dimensionality often surpasses the sample size, and if the prior is not appropriately elicited, prior information can overpower the data information, leading to posterior inconsistency.

Following [20, 21, 17], we let each connection weight be subject to a mixture Gaussian prior, i.e.,

by integrating out the structure information γ , where λ n ∈ (0 , 1) is the mixture proportion, σ 2 0 ,n is typically set to a very small number, while σ 2 1 ,n is relatively large. Visualizations of the mixture Gaussian priors for different λ n , σ 2 0 ,n , and σ 2 1 ,n are given in the Appendix E.

We assume µ ∗ can be well approximated by a sparse RNN given enough past information, and refer to this sparse RNN as the true RNN model. To be more specific, we define the true RNN model as

where G n := G ( C 0 , C 1 , /epsilon1,p n , L 1 , L 2 , . . . , L H n ) denotes the space of all valid networks that satisfy the Assumption 3.4 for the given values of H n , p n , and L h 's, and /pi1 n is some sequence converging to 0 as n → ∞ . For any given RNN ( β , γ ), the error | µ ∗ ( · ) -µ ( · , β , γ ) | can be decomposed as the approximation error | µ ∗ ( · ) -µ ( · , β ∗ , γ ∗ ) | and the estimation error | µ ( · , β ∗ , γ ∗ ) -µ ( · , β , γ ) | . The former is bounded by /pi1 n , and the order of the latter will be given in Theorem 3.9. For the sparse RNN, we make the following assumptions:

Assumption 3.4. The true sparse RNN model ( β ∗ , γ ∗ ) satisfies the following conditions:

The network structure satisfies: r n H n log n + r n log ¯ L + s n log p n ≤ C 0 n 1 -/epsilon1 , where 0 < /epsilon1 < 1 is a small constant, r n = | γ ∗ | denotes the connectivity of γ ∗ , ¯ L = max 1 ≤ j ≤ H n -1 L j denotes the maximum hidden state dimension, and s n denotes the input dimension of γ ∗ .

The network weights are polynomially bounded: ‖ β ∗ ‖ ∞ ≤ E n , where E n = n C 1 for some constant C 1 > 0.

Remark 3.5 . Assumption 3.4 is identical to assumption A.2 of [20], which limits the connectivity of the true RNN model to be of o ( n 1 -/epsilon1 ) for some 0 < /epsilon1 < 1. Then, as implied by

Lemma 3.10, an RNN of size O ( n/ log( n )) has been large enough for modeling a time series sequence of length n . Refer to [20] for discussions on the universal approximation ability of the neural network under this assumption; the universal approximation ability can still hold for many classes of functions, such as affine function, piecewise smooth function, and bounded α -Hölder smooth function.

Remark 3.6 . The existence of the sparse RNN model stems from Lemma 4.1 of [39], which, through the trick of independent block sequence construction, shows that many properties of the i.i.d processes can be extended to mixing processes. While the lemma was proven for the case of β -mixing, the author did mention her doubts about its applicability to α -mixing. Therefore, at least for the sequences of β -mixing, which implies α -mixing, the non-empty of the sparse RNN set in (4) can be guaranteed for many classes of functions as mentioned in Remark 3.5.

Assumption 3.7. The activation function ψ h is bounded for h = 1 (e.g., sigmoid and tanh), and is Lipschitz continuous with a Lipschitz constant of 1 for 2 ≤ h ≤ H n (e.g., ReLU, sigmoid and tanh).

Remark 3.8 . Assumption 3.7 mirrors [20, 21], except that we require the activation function for the first layer to be bounded. This extra assumption can be viewed as a replacement for the boundedness assumption for the input variables of a conventional DNN.

Let d ( p 1 , p 2 ) denote the integrated Hellinger distance between two conditional densities p 1 ( y | x ) and p 2 ( y | x ). Let π ( ·| D n ) be the posterior probability of an event. Theorem 3.9 establishes posterior consistency for the RNN model with the mixture Gaussian prior (3).

Theorem 3.9. Suppose Assumptions 3.1, 3.4, and 3.7 hold. If the mixture Gaussian prior (3) satisfies the conditions : λ n = O (1 / [ M H n l K n [ n 2 M l H n ( ¯ Lp n )] τ ]) for some constant τ > 0 , E n / [ H n log n + log ¯ L ] 1 / 2 ≤ σ 1 ,n ≤ n α for some constant α , and σ 0 ,n ≤ min { 1 / [ M H n l √ nK n ( n 3 / 2 σ 1 ,n /H n ) 2 M l H n ] , 1 / [ M H n l √ nK n ( nE n /H n ) 2 M l H n ] } , then there exists an an error sequence /epsilon1 2 n = O ( /pi1 2 n )+ O ( ζ 2 n ) such that lim n →∞ /epsilon1 n = 0 and lim n →∞ n/epsilon1 2 n = ∞ , and the posterior distribution satisfies

for sufficiently large n and C > 0 , where ζ 2 n = [ r n H n log n + r n log ¯ L + s n log p n ] /n , p µ ∗ denotes the underlying true data distribution, and p β denotes the data distribution reconstructed by the Bayesian RNN based on its posterior samples.

3.2 Uncertainty Quantification with Sparse RNNs

As mentioned previously, posterior consistency forms the basis for Bayesian inference with the RNN model. Based on Theorem 3.9, we further establish structure selection consistency and asymptotic normality of connection weights and predictions for the sparse RNN. In particular, the asymptotic normality of predictions enables the prediction intervals with correct coverage rates to be constructed.

Structure Selection Consistency It is known that the neural network often suffers from a non-identifiability issue due to the symmetry of its structure. For instance, one can permute certain hidden nodes or simultaneously change the signs or scales of certain weights while keeping the output of the neural network invariant. To address this issue, we follow [20] to define an equivalent class of RNNs, denoted by Θ, which is a set of RNNs such that any possible RNN for the problem can be represented by one and only one RNN in Θ via appropriate weight transformations. Let ν ( β , γ ) ∈ Θ denote an operator that maps any RNN to Θ. To serve the purpose of structure selection in the space Θ, we consider the marginal posterior inclusion probability (MIPP) approach. Formally, for each connection weight i = 1 , . . . , K n , we define its MIPP as q i = ∫ ∑ γ e i | ν ( β , γ ) π ( γ | β , D n ) π ( β | D n ) d β , where e i | ν ( β , γ ) is the indicator of i in ν ( β , γ ). The MIPP approach selects the connections whose MIPPs exceed a threshold ˆ q . Let ˆ γ ˆ q = { i : q i > ˆ q, i = 1 , . . . , K n } denote an estimator of γ ∗ = { i : e i | ν ( β ∗ , γ ∗ ) = 1 , i = 1 , . . . , K n } . Let A ( /epsilon1 n ) = { β : d ( p β , p µ ∗ ) ≤ /epsilon1 n } and ρ ( /epsilon1 n ) = max 1 ≤ i ≤ K n ∫ A ( /epsilon1 n ) ∑ γ | e i | ν ( β , γ ) -e i | ν ( β ∗ , γ ∗ ) | π ( γ | β , D n ) π ( β | D n ) d β , which measures

the structure difference on A ( /epsilon1 ) for the true RNN from those sampled from the posterior. Then we have the following Lemma:

Lemma 3.10. If the conditions of Theorem 3.9 are satisfied and ρ ( /epsilon1 n ) → 0 as n →∞ , then

(i) max 1 ≤ i ≤ K n {| q i -e i | ν ( β ∗ , γ ∗ ) |} p → 0 as n →∞ ;

(ii) (sure screening) P ( γ ∗ ⊂ ˆ γ ˆ q ) p → 1 as n →∞ , for any prespecified ˆ q ∈ (0 , 1) ;

(iii) (consistency) P ( γ ∗ = ˆ γ 0 . 5 ) p → 1 as n →∞ ;

where p → denotes convergence in probability.

Remark 3.11 . This lemma implies that we can filter out irrelevant variables and simplify the RNN structure when appropriate. Please refer to Section 5.2 for a numerical illustration.

Asymptotic Normality of Connection Weights and Predictions The following two theorems establish the asymptotic normality of ˜ ν ( β ) and predictions, where ˜ ν ( β ) denotes a transformation of β which is invariant with respect to µ ( · , β , γ ) while minimizing ‖ ˜ ν ( β ) -β ∗ ‖ ∞ .

We follow the same definition of asymptotic normality as in [21, 43, 19]. The posterior distribution for the function g ( β ) is asymptotically normal with center g ∗ and variance G if, for d β the bounded Lipschitz metric for weak convergence, and φ n the mapping φ n : β → √ n ( g ( β ) -g ∗ ), it holds, as n →∞ , that

in P ∗ -probability, which we also denote π ( ·| D n ) ◦ φ -1 n ⇝ N (0 , G ).

The detailed assumptions and setups for the following two theorems are given in Appendix C. For simplicity, we let M l = R l +1, and let l n ( β ) = 1 n -R l ∑ n i = R l +1 log( p β ( y i | y i -1: i -R l )) denote the averaged log-likelihood function. Let H n ( β ) denote the Hessian matrix of l n ( β ), and let h i,j ( β ) and h i,j ( β ) denote the ( i, j )-th element of H n ( β ) and H -1 n ( β ), respectively.

Theorem 3.12. Assume the conditions of Lemma 3.10 hold with ρ ( /epsilon1 n ) = o ( 1 K n ) and additional assumptions hold given in Appendix C, then π ( √ n (˜ ν ( β ) -β ∗ ) | D n ) ⇝ N ( 0 , V ) in P ∗ -probability as n → ∞ , where V = ( v i,j ) , and v i,j = E ( h i,j ( β ∗ )) if i, j ∈ γ ∗ and 0 otherwise.

Theorem 3.13 establishes asymptotic normality of the sparse RNN prediction, which implies prediction consistency and forms the theoretical basis for prediction uncertainty quantification as well.

Theorem 3.13. Assume the conditions of Theorem 3.12 and additional assumptions hold given in Appendix C. Then π ( √ n ( µ ( · , β ) -µ ( · , β ∗ ) | D n ) ⇝ N (0 , Σ) , where Σ = ∇ γ ∗ µ ( · , β ∗ ) ′ H -1 ∇ γ ∗ µ ( · , β ∗ ) and H = E ( -∇ 2 γ ∗ l n ( β ∗ )) is the Fisher information matrix.

4 Computation

In the preceding section, we established a theoretical foundation for sparse deep learning with time series data under the Bayesian framework. Building on [20], it is straightforward to show that Bayesian computation can be simplified by invoking the Laplace approximation theorem at the maximum a posteriori (MAP) estimator. This essentially transforms the proposed Bayesian method into a regularization method by interpreting the log-prior density function as a penalty for the log-likelihood function in RNN training. Consequently, we can train the regularized RNN model using an optimization algorithm, such as SGD or Adam. To address the local-trap issue potentially suffered by these methods, we train the regularized RNN using a prior annealing algorithm [21], as described in Algorithm 1. For a trained RNN, we sparsify its structure by truncating the weights less than a threshold to zero and further refine the nonzero weights for attaining the MAP estimator. For algorithmic

specifics, refer to Appendix D. Below, we outline the steps for constructing prediction intervals for one-step-ahead forecasts, where µ ( y k -1: k -R l , ˆ β ) is of one dimension, and ˆ β and ˆ γ represent the estimators of the network parameters and structure, respectively, as obtained by Algorithm 1:

For a test point y k -1: k -R l , estimate Σ in Theorem 3.13 by

The corresponding (1 -α )% prediction interval is given by

where there are n -R l observations used in training, and z α/ 2 denotes the upper α/ 2-quantile of the standard Gaussian distribution.

For construction of multi-horizon prediction intervals, see Appendix F.

5 Numerical Experiments

5.1 Uncertainty Quantification

As mentioned in Section 3, we will consider two types of time series datasets: the first type comprises a single time series, and the second type consists of a set of time series. We will compare the performance of our method against the state-of-the-art Conformal Prediction (CP) methods for both types of datasets. We set α = 0 . 1 (the error rate) for all uncertainty quantification experiments in the paper, and so the nominal coverage level of the prediction intervals is 90%.

5.1.1 A Single Time Series: French Electricity Spot Prices

We perform one-step-ahead forecasts on the French electricity spot prices data from 2016 to 2020, which consists of 35,064 observations. A detailed description and visualization of this time series are given in Appendix G.1. Our goal is to predict the 24 hourly prices of the following day, given the prices up until the end of the current day. As the hourly prices exhibit distinct patterns, we fit one model per hour as in the CP baseline [31]. We follow the data splitting strategy used in [31], where the first three years 2016 -2019 data are used as the (initial) training set, and the prediction is made for the last year 2019 -2020.

For all the methods considered, we use the same underlying neural network model: an MLP with one hidden layer of size 100 and the sigmoid activation function. More details on the training process are provided in Appendix G.1. For the state-of-the-art CP methods, EnbPI-V2 [29], NEX-CP [30], ACI [32] and AgACI [31], we conduct experiments in an online fashion, where the model is trained using a sliding window of the previous three years of data (refer to Figure 4 in the Appendix). Specifically, after constructing the prediction interval for each time step in the prediction period, we add the ground truth value to the training set and then retrain the model with the updated training set. For ACI, we conduct experiments with various values of γ and present the one that yields the best performance. In the case of AgACI, we adopt the same aggregation approach as used in [31], namely, the Bernstein Online Aggregation (BOA) method [44] with a gradient trick. We also report the performance of ACI with γ = 0 as a reference. For NEX-CP, we use the same weights as those employed in their time-series experiments. For EnbPI-V2, we tune the number of bootstrap models and select the one that offers the best performance.

Since this time series exhibits no or minor distribution shift, our method PA is trained in an offline fashion, where the model is fixed for using only the observations between 2016 and 2019, and the observations in the prediction range are used only for final evaluation. That is, our method uses less data information in training compared to the baseline methods.

40

35

35

30

length

30

Median length

25

25

Average

20

20

15

15

10

5

10

90.0

90.5

91.0

91.5

92.0

92.5

93.0

90.0

90.5

91.0

91.5

92.0

92.5

93.0

Coverage

PA offline

AgACI online

ACI y=0.01 online

ACIy=0online

+

EnbPI V2

NEX-CP

The results are presented in Figure 2, which includes empirical coverage (with methods that are positioned closer to 90.0 being more effective), median/average prediction interval length, and corresponding interquartile range/standard deviation. As expected, our method is able to train and calibrate the model by using only the initial training set, i.e., the data for 2016-2019, and successfully produces faithful prediction intervals. In contrast, all CP methods produce wider prediction intervals than ours and higher coverage rates than the nominal level of 90%. In addition, ACI is sensitive to the choice of γ [31].

5.1.2 A Set of Time Series

We conduct experiments using three publicly available real-world datasets: Medical Information Mart for Intensive Care (MIMIC-III), electroencephalography (EEG) data, and daily COVID-19 case numbers within the United Kingdom's local authority districts (COVID-19) [33]. A concise overview of these datasets is presented in Table 1. Our method, denoted as PA-RNN (since the underlying prediction model is an LSTM [45]), is compared to three benchmark methods: CF-RNN [33], MQ-RNN [35], and DP-RNN [36], where LSTM is used for all these methods. To ensure a fair comparison, we adhere to the same model structure, hyperparameters, and data processing steps as specified in [33]. Detailed information regarding the three datasets and training procedures can be found in Appendix G.2.

The numerical results are summarized in Tables 2. Note that the baseline results for EEG and COVID-19 are directly taken from the original paper [33]. We reproduce the baseline results for MIMIC-III, as the specific subset used by [33] is not publicly available. Table 2 indicates that our method consistently outperforms the baselines. In particular, Our method consistently generates shorter prediction intervals compared to the conformal baseline CF-RNN while maintaining the same or even better coverage rate as CF-RNN. Both the MQ-RNN and DP-RNN methods fail to generate prediction intervals that accurately maintain a faithful coverage rate.

5.2 Autoregressive Order Selection

In this section, we evaluate the performance of our method in selecting the autoregressive order for two synthetic autoregressive processes. The first is the non-linear autoregressive (NLAR) process [49, 50, 51]:

where η i ∼ N (0 , 1) represents i.i.d. Gaussian random noises, and the functions G 1 and G 2 are defined as:

The second is the exponential autoregressive process [52]:

where, again, η i ∼ N (0 , 1) denotes i.i.d. Gaussian random noises.

For both synthetic processes, we generate five datasets. Each dataset consists of training, validation, and test sequences. The training sequence has 10000 samples, while the validation and test sequences each contain 1000 samples. For training, we employ a single-layer RNN with a hidden layer width of 1000. Further details on the experimental setting can be found in Appendix G.3.

For the NLAR process, we consider two different window sizes for RNN modeling: 15 (with input as y i -1: i -15 ) and 1 (with input as y i -1 ). Notably, the NLAR process has an order of 7. In the case where the window size is 15, the input information suffices for RNN modeling, rendering the past information conveyed by the hidden states redundant. However, when the window size is 1, this past information becomes indispensable for the RNN. In contrast, the exponential autoregressive process has an order of 1. For all window sizes we explored, namely 1 , 3 , 5 , 7 , 10 , 15, the input information is always sufficient for RNN modeling.

We evaluate the predictive performance using mean square prediction error (MSPE) and mean square fitting error (MSFE). The model selection performance is assessed by two metrics: the false selection rate (FSR) and the negative selection rate (NSR). We define

ˆ S j represents the set of selected variables for dataset j . Furthermore, we provide the final number of nonzero connections for hidden states and the estimated autoregressive orders. The numerical results for the NLAR process are presented in Table 3, while the numerical results for the exponential autoregressive process are given in Table 4.

Our results are promising. Specifically, when the window size is equal to or exceeds the true autoregressive order, all connections associated with the hidden states are pruned, effectively converting the RNN into an MLP. Conversely, if the window size is smaller than the true autoregressive order, a significant number of connections from the hidden states

are retained. Impressively, our method accurately identifies the autoregressive order-a noteworthy achievement considering the inherent dependencies in time series data. Although our method produces a nonzero FSR for the NLAR process, it is quite reasonable considering the relatively short time sequence and the complexity of the functions G 1 and G 2 .

5.3 RNN Model Compression

We have also applied our method to RNN model compression, achieving state-of-the-art results. Please refer to Section G.4 in the Appendix for details.

6 Discussion

This paper has established the theoretical groundwork for sparse deep learning with time series data, including posterior consistency, structure selection consistency, and asymptotic normality of predictions. Our empirical studies indicate that sparse deep learning can outperform current cutting-edge approaches, such as conformal predictions, in prediction uncertainty quantification. More specifically, compared to conformal methods, our method maintains the same coverage rate, if not better, while generating significantly shorter prediction intervals. Furthermore, our method effectively determines the autoregression order for time series data and surpasses state-of-the-art techniques in large-scale model compression. The theory developed in this paper has included LSTM [45] as a special case, and some numerical examples have been conducted with LSTM; see Section G of the Appendix for the detail. Furthermore, there is room for refining the theoretical study under varying mixing assumptions for time series data, which could broaden applications of the proposed method. Also, the efficacy of the proposed method can potentially be further improved with the elicitation of different prior distributions.

In summary, this paper represents a significant advancement in statistical inference for deep RNNs, which, through sparsing, has successfully integrated the RNNs into the framework of statistical modeling. The superiority of our method over the conformal methods shows further the criticality of consistently approximating the underlying mechanism of the data generation process in uncertainty quantification.

In terms of limitations of the proposed method, one potential concern pertains to the calculation of the inverse of the Fisher information matrix. For large-scale problems, the sparsified model could still retain a large number of non-zero parameters. In such instances, the computational feasibility of calculating the Hessian matrix might become compromised. Nonetheless, an alternative avenue exists in the form of the Bayesian approach, which circumvents the matrix inversion challenge. A concise description of this Bayesian strategy can be found in [21].

References

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural information processing systems , 31, 2018.

Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. arXiv preprint arXiv:2201.12886 , 2022.

Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting , 37(1):388-427, 2021.

Gábor Melis, Tomáš Kočisk` y, and Phil Blunsom. Mogrifier lstm. arXiv preprint arXiv:1909.01792 , 2019.

Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines , 30:681-694, 2020.

Yuki Tatsunami and Masato Taki. Sequencer: Deep lstm for image classification. arXiv preprint arXiv:2205.01972 , 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In International conference on machine learning , pages 2603-2612. PMLR, 2017.

Marco Gori and Alberto Tesi. On the problem of local minima in backpropagation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 14(1):76-86, 1992.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning , pages 242-252. PMLR, 2019.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019.

Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over-parameterized deep relu networks. Machine learning , 109(3):467-492, 2020.

Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. Advances in neural information processing systems , 32, 2019.

Faming Liang, Qizhai Li, and Lei Zhou. Bayesian neural networks for selection of drug sensitive genes. Journal of the American Statistical Association , 113(523):955-972, 2018.

Nicholas G Polson and Veronika Ročková. Posterior concentration for sparse deep learning. Advances in Neural Information Processing Systems , 31, 2018.

Yuexi Wang and Veronika Rocková. Uncertainty quantification for sparse deep learning. In International Conference on Artificial Intelligence and Statistics , pages 298-308. PMLR, 2020.

Yan Sun, Qifan Song, and Faming Liang. Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association , pages 1-15, 2021.

Yan Sun, Wenjun Xiong, and Faming Liang. Sparse deep learning: A new framework immune to local traps and miscalibration. Advances in Neural Information Processing Systems , 34:22301-22312, 2021.

L. Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science , 16:199-231, 2001.

Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics , 48(4):1916-1921, 2020.

Helmut Bolcskei, Philipp Grohs, Gitta Kutyniok, and Philipp Petersen. Optimal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science , 1(1):8-45, 2019.

Vitaly Maiorov and Allan Pinkus. Lower bounds for approximation by mlp neural networks. Neurocomputing , 25(1-3):81-91, 1999.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13 , pages 345-356. Springer, 2002.

Victor Chernozhukov, Kaspar Wüthrich, and Zhu Yinchu. Exact and robust conformal inference methods for predictive machine learning with dependent data. In Conference On learning theory , pages 732-749. PMLR, 2018.

Chen Xu and Yao Xie. Conformal prediction interval for dynamic time-series. In International Conference on Machine Learning , pages 11559-11569. PMLR, 2021.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics , 51(2):816-845, 2023.

Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning , pages 25834-25866. PMLR, 2022.

Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

Kamile Stankeviciute, Ahmed M Alaa, and Mihaela van der Schaar. Conformal timeseries forecasting. Advances in Neural Information Processing Systems , 34:6216-6228, 2021.

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.

Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053 , 2017.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Derrick T Mirikitani and Nikolay Nikolaev. Recursive bayesian recurrent neural networks for time-series modeling. IEEE Transactions on Neural Networks , 21(2):262-274, 2009.

Shiqing Ling and Michael McAleer. A general asymptotic theory for time-series models. Statistica Neerlandica , 64(1):97-111, 2010.

Bin Yu. Rates of convergence for empirical processes of stationary mixing sequences. The Annals of Probability , pages 94-116, 1994.

Ron Meir. Performance bounds for nonlinear time series prediction. In Proceedings of the tenth annual conference on computational learning theory , pages 122-129, 1997.

Cosma Shalizi and Aryeh Kontorovich. Predictive pac learning and process decompositions. Advances in neural information processing systems , 26, 2013.

Subhashis Ghosal and Aad van der Vaart. Convergence rates of posterior distributions for non-i.i.d. observations. Annals of Statistics , 35:192-223, 2007.

Ismael Castillo and Judith Rousseau. A bernstein-von mises theorem for smooth functionals in semiparametric models. Annals of Statistics , 43:2353-2383, 2013.

Olivier Wintenberger. Optimal learning with bernstein online aggregation. Machine Learning , 106:119-141, 2017.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput. , 9(8):1735-1780, nov 1997.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Catherine Blake. Uci repository of machine learning databases. http://www. ics. uci. edu/˜ mlearn/MLRepository. html , 1998.

GOV UK et al. Coronavirus (covid-19) in the uk, 2020.

Souhaib Ben Taieb and Amir F Atiya. A bias and variance analysis for multistep-ahead time series forecasting. IEEE transactions on neural networks and learning systems , 27(1):62-76, 2015.

Marcelo C Medeiros, Timo Teräsvirta, and Gianluigi Rech. Building neural network models for time series: a statistical approach. Journal of Forecasting , 25(1):49-75, 2006.

Timo Terasvirta and Anders Kock. Forecasting with nonlinear time series models. Econometrics: Multiple Equation Models eJournal , 2010.

Bjørn Auestad and Dag Tjøstheim. Identification of nonlinear time series: First order characterization and order determination. Biometrika , 77(4):669-687, 1990.

Wenxin Jiang. Bayesian variable selection for high dimensional generalized linear models: convergence rates of the fitted densities. The Annals of Statistics , 35(4):1487-1511, 2007.

Subhashis Ghosal and Aad Van der Vaart. Fundamentals of nonparametric Bayesian inference , volume 44. Cambridge University Press, 2017.

Andre M Zubkov and Aleksandr A Serov. A complete proof of universal inequalities for the distribution function of the binomial law. Theory of Probability & Its Applications , 57(3):539-544, 2013.

Pentti Saikkonen. Dependent versions of a central limit theorem for the squared length of a sample mean. Statistics & probability letters , 22(3):185-194, 1995.

Stephen Portnoy. Asymptotic behavior of likelihood methods for exponential families when the number of parameters tends to infinity. The Annals of Statistics , pages 356-366, 1988.

Yi-An Ma, Tianqi Chen, and Emily Fox. A complete recipe for stochastic gradient mcmc. Advances in neural information processing systems , 28, 2015.

Grigoris A Dourbois and Pandelis N Biskas. European market coupling algorithm incorporating clearing conditions of block and complex orders. In 2015 IEEE Eindhoven PowerTech , pages 1-6. IEEE, 2015.

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Conformal prediction with temporal quantile adjustments. arXiv preprint arXiv:2205.09940 , 2022.

Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems , 2:129-146, 2020.

Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res. , 22(241):1-124, 2021.

Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 , 2014.

Nadezhda Chirkova, Ekaterina Lobacheva, and Dmitry Vetrov. Bayesian compression for natural language processing. arXiv preprint arXiv:1810.10927 , 2018.

Maxim Kodryan, Artem Grachev, Dmitry Ignatov, and Dmitry Vetrov. Efficient language modeling with automatic relevance determination in recurrent neural networks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP2019) , pages 40-48, 2019.

Ekaterina Lobacheva, Nadezhda Chirkova, and Dmitry Vetrov. Bayesian sparsification of gated recurrent neural networks. arXiv preprint arXiv:1812.05692 , 2018.

Artem M Grachev, Dmitry I Ignatov, and Andrey V Savchenko. Compression of recurrent neural networks for efficient language modeling. Applied Soft Computing , 79:354-362, 2019.

Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 , 2017.

Appendix for 'Sparse Deep Learning for Time Series Data: Theory and Applications'

A Mathematical Facts of Sparse RNNs

For a sparse RNN model with H n -1 recurrent layers and a single output layer, several mathematical facts can be established. Let's denote the number of nodes in each layer as L 1 , . . . , L H n . Additionally, let r w i represent the number of non-zero connection weights for w i , and let r v i denote the number of non-zero connection weights for v i , where w i and v i denote the connection weights at two layers of a sparse RNN.

Furthermore, we define O t i,j ( β , x t :1 ) as the output value of the j -th neuron of the i -th layer at time t . This output depends on the parameter vector β and the input sequence x t :1 = ( x t , . . . , x 1 ), where x t represents the generic input to the sparse RNN at time step t .

Lemma A.1. Under Assumption 3.7, if a sparse RNN has at most r n non-zero connection weights (i.e., r n = ∑ H n i =1 r w i + ∑ H n -1 i =1 r v i ) and ‖ β ‖ ∞ = E n , then the summation of the absolute outputs of the i th layer at time t is bounded by

Proof. For simplicity, we rewrite O t i,j ( β , x t :1 ) as O t i,j when appropriate. The lemma is the result from the following facts:

For t = 1 and 1 ≤ i ≤ H n (Lemma S4 from [20])

For t ≥ 2 and i = 1 (recursive relationship)

For t ≥ 2 and 2 ≤ i ≤ H n -1 (recursive relationship)

For t ≥ 2 and i = H n (recursive relationship)

We can verify this lemma by plugging the conclusion into all recursive relationships. We show the steps to verify the case when t ≥ 2 and 2 ≤ i ≤ H n -1, since other cases are trivial

to verify.

where the last inequality is due to the fact that

Lemma A.2. Under Assumption 3.7, consider two RNNs, β and ˜ β , where the former one is a sparse network satisfying ‖ β ‖ 0 = r n and ‖ β ‖ ∞ = E n , and it's network structure vector is denoted by γ . Assume that if | β i -˜ β i | < δ 1 for all i ∈ γ and | β i -˜ β i | < δ 2 for all i / ∈ γ , then

Proof. Define ˇ β such that ˇ β i = ˜ β i for all i ∈ γ and ˇ β i = 0 for all i / ∈ γ . Let ˇ O t i,j denote O t i,j ( ˇ β , x t :1 ). Then, from the following facts that

For t = 1 and 1 ≤ i ≤ H n (Lemma S4 from [20]) L i ∑ j =1 | ˇ O 1 i,j -O 1 i,j | ≤ iδ 1 ( E n + δ 1 ) i -1 ( i ∏ k =1 r w k ) .

For t ≥ 2 and i = 1 (recursive relationship)

For t ≥ 2 and 2 ≤ i ≤ H n -1 (recursive relationship) |

For t ≥ 2 and i = H n (recursive relationship)

We have

For 1 ≤ i ≤ H n -1

For i = H n

We can verify the above conclusion by plugging it into all recursive relationships. We show the steps to verify the case when t ≥ 2 and 2 ≤ i ≤ H n -1, since other cases are trivial to

verify.

where the last inequality is due to the fact that for t ≥ 2 and i ≥ 2, it is easy to see that

Now let ˜ O t i,j denotes O t i,j ( ˜ β , x 1: t ), then from the facts that

For 1 ≤ t ≤ T and 1 ≤ i ≤ H n

This can be easily derived by Lemma A.1 and the fact that ∥ ∥ ∥ ˇ β ∥ ∥ ∥ ∞ ≤ E n + δ 1 .

We have

We can verify the above conclusion in a similar approach. The proof is completed by summation of the bound for | µ ( x t :1 , β ) -| µ ( x t :1 , ˇ β ) | and | µ ( x t :1 , ˇ β ) -µ ( x t :1 , ˜ β ) | .

B Proofs on Posterior Consistency: A Single Time Series

To establish posterior consistency for DNNs with i.i.d. data, [20] utilized Proposition 1 from [53]. This lemma provides three sufficient conditions for proving posterior consistency for general statistical models with i.i.d. data, along with a posterior contraction rate. In this paper, we aim to establish posterior consistency for DNNs with stochastic processes, specifically time series, that are strictly stationary and α -mixing with an exponentially decaying mixing coefficient [54, 42].

Consider a time series D n = ( z 1 , z 2 , . . . , z n ) defined on a probability space (Ω , F , P ∗ ), which satisfies the assumptions outlined in Assumption 1. For simplicity, we assume that the initial values z 1 -l , . . . , z 0 are fixed and given.

Let P n denote a set of probability densities, let P c n denote the complement of P n , and let /epsilon1 n denote a sequence of positive numbers. Let N ( /epsilon1 n , P n ) be the minimum number of Hellinger balls of radius /epsilon1 n that are needed to cover P n , i.e., N ( /epsilon1 n , P n ) is the minimum of all number k 's such that there exist sets S j = { p : d ( p, p j ) ≤ /epsilon1 n } , j = 1 , ..., k , with P n ⊂ ∪ k j =1 S j holding, where

denotes the integrated Hellinger distance [42, 54] between the two conditional densities p ( z | x ) and q ( z | x ), where x ∈ R l contains the history up to l time steps of z , and v ( x ) is the probability density function of the marginal distribution of x . Note that v is invariant with respect to time index i due to the strictly stationary assumption.

For D n , denote the corresponding true conditional density by p ∗ . Define π ( · ) as the prior density, and π ( ·| D n ) as the posterior. Define ˆ π ( /epsilon1 ) = π [ d ( p, p ∗ ) > /epsilon1 | D n ] for each /epsilon1 > 0. Assume the conditions:

(a) log N ( /epsilon1 n , P n ) ≤ n/epsilon1 2 n for all sufficiently large n .

(b) π ( P c n ) ≤ e -bn/epsilon1 2 n for some b > 0 and all sufficiently large n .

(c) Let ‖ f ‖ s = ( ∫ | f | s dr ) 1 /s , then π ( ‖ p -p ∗ ‖ s ≤ b ′ /epsilon1 n ) ≥ e -γn/epsilon1 2 n for some b ′ , γ > 0, s > 2, and all sufficiently large n .

Lemma B.1. Under the conditions (a), (b) and (c), given sufficiently large n , lim n →∞ /epsilon1 n = 0 , and lim n →∞ n/epsilon1 2 n = ∞ , we have for some large C > 0 ,

Proof. This Lemma can be proved with similar arguments used in Section 9 . 5 . 3, Theorem 8 . 19, and Theorem 8 . 29 of [54].

B.1 Posterior Consistency with a General Shrinkage Prior

Let β denote the vector of all connection weights of a RNN. To prove Theorem 3.9, we first consider a general shrinkage prior that all entries of β are subject to an independent continuous prior π b , i.e., π ( β ) = ∏ K n j =1 π b ( β j ), where K n denotes the total number of elements of β . Theorem B.2 provides a sufficient condition for posterior consistency.

Theorem B.2. (Posterior consistency) Suppose Assumptions 3.1 - 3.7 hold. If the prior π b satisfies that

for some τ > 0 , where

with some c 0 > 1 , π b is the minimal density value of π b within the interval [ -E n -1 , E n +1] , and M n is some sequence satisfying log( M n ) = O (log( n )) . Then, there exists a sequence /epsilon1 n , satisfying n/epsilon1 2 n /equivasymptotic r n H n log n + r n log ¯ L + s n log p n + n/pi1 2 n and /epsilon1 n ≺ 1 , such that

for some large M > 0 .

Proof. To prove this theorem, it suffices to check all three conditions listed in Lemma B.1 Checking condition (c):

Consider the set A = { β : max j ∈ γ ∗ | β j -β ∗ j | ≤ ω n , max j / ∈ γ ∗ | β j -β ∗ j | ≤ ω ′ n } , where

for some c 1 > 0 and c 0 > 1. If β ∈ A , by Lemma A.2, we have

By the definition (4), we have

Finally for some s > 2 (for simplicity, we take s to be an even integer),

For any small b ′ > 0, condition (c) is satisfied as long as c 1 is sufficiently small, /epsilon1 n ≥ C 0 /pi1 n for some large C 0 , and the prior satisfies -log π ( A ) ≤ γn/epsilon1 2 n . Since

where the last inequality is due to the fact that ω ′ n /greatermuch η ′ n . Note that lim n →∞ (1 -1 /K n ) K n = e -1 . Since log(1 /ω n ) /equivasymptotic 2 M l H n log( E n ) + 2 M l H n log( r n /H n ) + log(1 //epsilon1 n ) + constant = O ( H n log n ) (note that log(1 //epsilon1 n ) = O (log n )), then for sufficiently large n ,

Thus, the prior satisfies -log π ( A ) ≤ γn/epsilon1 2 n for sufficiently large n , when n/epsilon1 2 n ≥ C 0 ( r n H n log n + r n log ¯ L ) for some sufficiently large constant C 0 . Thus condition (c) holds. Checking condition (a):

Let P n denote the set of probability densities for the RNNs whose parameter vectors satisfy

where | Γ β | in denotes the number of input connections with the absolute weights greater than δ ′ n , k n ( ≤ n/r n ) and k ′ n ( ≤ n/s n ) will be specified later, and

for some c 1 > 0 , c 0 > 0. Let

Consider two parameter vectors β u and β v in set B n , such that there exists a structure γ with | γ | ≤ k n r n and | γ | in ≤ k ′ n s n , and | β u j -β v j | ≤ δ n for all j ∈ γ , max( | β v j | , | β u j | ) ≤ δ ′ n for all j / ∈ γ . Hence, by Lemma A.2, we have that | µ ( x 1: M l , β u ) -µ ( x 1: M l , β v ) | 2 ≤ 9 c 2 1 /epsilon1 2 n . For two normal distributions N ( µ 1 , σ 2 ) and N ( µ 2 , σ 2 ), define the corresponding Kullback-Leibler divergence as

Together with the fact that 2 d ( p µ 1 , p µ 2 ) 2 ≤ d 0 ( p µ 1 , p µ 2 ), we have

for some C > 0, given a sufficiently small c 1 .

Given the above results, one can bound the packing number N ( P n , /epsilon1 n ) by ∑ k n r n j =1 χ j H n ( 2 M n δ n ) j where χ j H n denotes the number of all valid networks who has exact j connections and has no more than k ′ n s n inputs. Since log χ j H n ≤ k ′ s n log p n + j log ( k ′ s n L 1 +2 H n ¯ L 2 ) , log M n = O (log n ), k n r n ≤ n , and k ′ n s n ≤ n , then

We can choose k n and k ′ n such that for sufficiently large n , k n r n { H n log n + log ¯ L } /equivasymptotic k ′ n s n log p n /equivasymptotic n/epsilon1 2 n and then log N ( P n , /epsilon1 n ) ≤ n/epsilon1 2 n .

Checking condition (b):

Lemma B.3. (Theorem 1 of [55]) Let X ∼ Binomial ( n, p ) be a Binomial random variable. For any 1 < k < n -1

where Φ is the cumulative distribution function (CDF) of the standard Gaussian distribution and H ( p, k/n ) = ( k/n ) log( k/np ) + (1 -k/n ) log[(1 -k/n ) / (1 -p )] .

Now,

where v n = 1 -π b ([ -δ ′ n , δ ′ n ]). By the condition on π b and the fact that δ ′ n /greatermuch η ′ n , it is easy to see that v n ≤ exp { τ [ H n log n +log ¯ L +log p n ] -log K n } for some constant τ > 0. Thus, by Lemma B.3, -log Pr (Binomial( K n , v n ) > k n r n ) ≈ τk n r n [ H n log n +log ¯ L +log p n ] ≳ n/epsilon1 2 n due to the choice of k n , and -log Pr ( | Γ β | in ≥ k ′ n s n ) ≈ k ′ n s n [ τ ( H n log n +log ¯ L +log p n ) + log ( p n K n /L 1 )] ≳ n/epsilon1 2 n due to the choice of k ′ n . Thus, condition (b) holds as well. Then by lemma B.1, the proof is completed.

B.2 Proof of Theorem 3.9

Proof. To prove Theorem 3.9 in the main text, it suffices to verify the four conditions on π b listed in Theorem B.2. Let M n = max( √ 2 nσ 1 ,n , E n ). Condition 8 can be verified by choosing σ 1 .n such that E 2 n / 2 σ 2 1 .n + log σ 2 1 ,n = O ( H n log n + log ¯ L ). Conditions 9 and 10 can be verified by setting λ n = 1 / { M H n l K n [ n 2 M l H n ( ¯ Lp n )] τ } and σ 0 ,n ≺ 1 / { M H n l √ nK n ( n/H n ) 2 M l H n ( c 0 M n ) 2 M l H n } . Finally, condition 11 can be verified by M n ≥ 2 nσ 2 0 .n and τ [ H n log n +log ¯ L +log p n ] + M 2 n / 2 σ 2 1 ,n ≥ n . Finally, based on the proof above we see that n/epsilon1 2 n /equivasymptotic r n H n log n + r n log ¯ L + s n log p n + n/pi1 2 n and lim n →∞ /epsilon1 n = 0.

B.3 Posterior Consistency for Multilayer Perceptrons

As highlighted in Section 3, the MLP can be formulated as a regression problem. Here, the input is x i = y i -1: i -R l for some l ≤ R l /lessmuch n , with the corresponding output being y i . Thus, the dataset D n can be represented as ( x i , y i ) n i =1+ R l . We apply the same assumptions as for the MLP, specifically 3.1, 3.4, and 3.7. We also use the same definitions and notations for the MLP as those in [20, 21].

Leveraging the mathematical properties of sparse MLPs presented in [20] and the proof of sparse RNNs for a single time series discussed above, one can straightforwardly derive the following Corollary. Let d ( p 1 , p 2 ) denote the integrated Hellinger distance between two conditional densities p 1 ( y | x ) and p 2 ( y | x ). Let π ( ·| D n ) be the posterior probability of an event.

Corollary B.4. Suppose Assumptions 3.1, 3.4, and 3.7 hold. If the mixture Gaussian prior (3) satisfies the conditions : λ n = O (1 / [ K n [ n H n ( ¯ Lp n )] τ ]) for some constant τ > 0 , E n / [ H n log n + log ¯ L ] 1 / 2 ≤ σ 1 ,n ≤ n α for some constant α , and σ 0 ,n ≤ min { 1 / [ √ nK n ( n 3 / 2 σ 1 ,n /H n ) H n ] , 1 / [ √ nK n ( nE n /H n ) H n ] } , then there exists an an error sequence /epsilon1 2 n = O ( /pi1 2 n ) + O ( ζ 2 n ) such that lim n →∞ /epsilon1 n = 0 and lim n →∞ n/epsilon1 2 n = ∞ , and the posterior distribution satisfies

for sufficiently large n and C > 0 , where ζ 2 n = [ r n H n log n + r n log ¯ L + s n log p n ] /n , p µ ∗ denotes the underlying true data distribution, and p β denotes the data distribution reconstructed by the Bayesian MLP based on its posterior samples.

C Asymptotic Normality of Connection Weights and Predictions

This section provides detailed assumptions and proofs for Theorem 3.12 and 3.13. For simplicity, we assume y 0:1 -R l is also given, and we let M l = R l + 1. Let l n ( β ) = 1 n ∑ n i =1 log( p β ( y i | y i -1: i -R l )) denote the likelihood function, and let π ( β ) denote the den-∂ d l ( β )

the d -th order partial derivatives. Let H n ( β ) denote the Hessian matrix of l n ( β ), and let h i,j ( β ) , h i,j ( β ) denote the ( i, j )-th component of H n ( β ) and H -1 n ( β ), respectively. Let B λ,n = ¯ λ 1 / 2 n ( β ∗ ) /λ n ( β ∗ ) and b λ,n = √ r n /nB λ,n . For a RNN with β , we define the weight truncation at the true model structure γ ∗ : ( β γ ∗ ) i = β i for i ∈ γ ∗ and ( β γ ∗ ) i = 0 for i / ∈ γ ∗ . B β ∗ { β | β -β ∗ | ≤ δ , ∀ i ∈ γ ∗ , | β -β ∗ | ≤

sity of the mixture Gaussian prior (3). Let h i 1 ,i 2 ,...,i d ( β ) = n ∂β i 1 · · · ∂β i d which denotes

For the mixture Gaussian prior (3), let δ n ( ) = : i i n i i ( )

In addition, we make the following assumptions for Theorem 3.12 and Theorem 3.13.

Assumption C.1. Assume the conditions of Lemma 3.10 hold with ρ ( /epsilon1 n ) = o ( 1 K n ) and

{ β : max i ∈ γ ∗ | β i -β ∗ i | > δ n , d ( p β , p µ ∗ ) ≤ /epsilon1 n } , where /epsilon1 n is the posterior contraction rate as defined in Theorem 3.9. Assume there exists some constants C > 2 and M > 0 such that

C.1 β ∗ = ( β 1 , . . . , β K n ) is generic, min i ∈ γ ∗ | β ∗ i | > Cδ n and π ( A ( /epsilon1 n , δ n ) | D n ) → 0 as n →∞ .

C.2 | h i ( β ∗ ) | < M, | h j,k ( β ∗ ) | < M, | h j,k ( β ∗ ) | < M, | h i,j,k ( β ) | < M, | h l ( β ) | < M hold for any i, j, k ∈ γ ∗ , l / ∈ γ ∗ , and β ∈ B 2 δ n ( β ∗ ).

C.3 sup {| E β ( a ′ U 3 ) | : ∥ ∥ β γ ∗ -β ∗ ∥ ∥ ≤ 1 . 2 b λ,n , ‖ a ‖ = 1 ≤ 0 . 1 √ n/r n λ 2 n ( β ∗ ) / ¯ λ 1 / 2 n ( β ∗ ) } and B λ,n = O (1), where U = Z -E β γ ∗ ( Z ), Z denotes a random variable drawn from a neural network model parameterized by β γ ∗ , and E β γ ∗ denotes the mean of Z .

C.4 r 2 n /n → 0 and the conditions for Theorem 2 of [56] hold.

Conditions C. 1 to C. 3 align with the assumptions made in [21]. An additional assumption, C. 4, is introduced to account for the dependency inherent in time series data. This assumption is crucial and employed in conjunction with C. 3 to establish the consistency of the maximum likelihood estimator (MLE) of β γ ∗ for a given structure γ ∗ . While the assumption r n /n → 0 is sufficient for independent data, which is implied by Assumption 3.4, a stronger restriction, specifically r 2 n /n → 0, is necessary for dependent data such as time series. It is worth noting that the conditions used in Theorem 2 of [56] pertain specifically to the time series data itself.

Assumption C.2. | µ i ( · , β ∗ ) | < M, | µ i,j ( · , β ) | < M, | µ k ( · , β ) | < M for any i, j ∈ γ ∗ , k / ∈ γ ∗ , and β ∈ B 2 δ n ( β ∗ ), where M is as defined in Assumption C.1.

Proof of Theorem 3.12 and Theorem 3.13 The proof of these two theorems can be conducted following the approach in [21]. The main difference is that in [21], they rely on Theorem 2 . 1 of [57], which assumes independent data. However, the same conclusions can be extended to time series data by using Theorem 2 of [56], taking into account assumption C. 4. This allows for the consideration of dependent data in the analysis.

D Computation

Algorithm 1 gives the prior annealing procedure[21]. In practice, the following implementation can be followed based on Algorithm 1:

For 0 < t < T 1 , perform initial training.

For T 1 ≤ t ≤ T 2 , set σ ( t ) 0 ,n = σ init 0 ,n and gradually increase η ( t ) = t -T 1 T 2 -T 1 .

For T 2 ≤ t ≤ T 3 , set η ( t ) = 1 and gradually decrease σ ( t ) 0 ,n according to the formula: σ ( t ) 0 ,n = ( T 3 -t T 3 -T 2 ) ( σ init 0 ,n ) 2 + ( t -T 2 T 3 -T 2 ) ( σ end 0 ,n ) 2 .

For t > T 3 , set η ( t ) = 1, σ ( t ) 0 ,n = σ end 0 ,n , and gradually decrease the temperature τ according to the formula: τ = c t -T 3 , where c is a constant.

Please refer to Appendix E for intuitive explanations of the prior annealing algorithm and further details on training a model to achieve the desired sparsity.

E Mixture Gaussian Prior

The mixture Gaussian prior imposes a penalty on the model parameters by acting as a piecewise L2 penalty, applying varying degrees of penalty in different regions of the parameter space. Given values for σ 0 ,n , σ 1 ,n , and λ n , a threshold value can be computed using Algorithm

Algorithm 1 Prior Annealing: Frequentist

(Initial Training) Train a neural network satisfying condition ( S ∗ ) such that a global optimal solution β 0 = arg max β l n ( β ) is reached, this stage can be accomplished by using SGD or Adam.

[2] (Prior Annealing) Initialize β at β 0 , and simulate from a sequence of distributions π ( β | D n , τ, η ( k ) , σ ( k ) 0 ,n ) ∝ e nl n ( β ) /τ π η ( k ) /τ k ( β ) for k = 1 , 2 , . . . , m , where 0 < η (1) ≤ η (2) ≤ · · · ≤ η ( m ) = 1, π k = λ n N (0 , σ 2 1 ,n ) + (1 -λ n ) N (0 , ( σ ( k ) 0 ,n ) 2 ), and σ init 0 ,n = σ (1) 0 ,n ≥ σ (2) 0 ,n ≥ · · · ≥ σ ( m ) 0 ,n = σ end 0 ,n . This can be done by using stochastic gradient MCMC algorithms [58]. After the stage m has been reached, continue to run the simulated annealing algorithm by gradually decreasing the temperature τ to a very small value. Denote the resulting model by ˆ β .

[3] (Structure Sparsification) For each connection weight i ∈ { 1 , 2 , . . . , K n } , set ˜ γ i = 1, √

determined by solving π ( γ i = 1 | β ) > 0 . 5, and σ 0 ,n = σ end 0 ,n . Denote the the structure of the sparse RNN by ˜ γ .

(Nonzero-weights Refining) Refine the nonzero weights of the sparse model ( ˆ β , ˜ γ ) by maximizing l n ( β ). Denote the resulting estimate by ( ˜ β , ˜ γ ).

入n = 1e - 7,0²,n = 0.05

Init "free space"

Init "free space"

Negative log probability

10

Mid "free space"

Mid "free space"

End "free space"

End "free space"

(nt)2 = 1e - 4

-5

(oid)2 = 1e - 5

(0ng)2 = 1e - 7

-0.20

0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20

β

Parameters whose absolute values are below this threshold will receive a large penalty, hence constituting the "penalty region", while parameters whose absolute values are above the threshold will receive relatively minimal penalty, forming the "free space". The severity of the penalty in the free space largely depends on the value of σ 1 ,n .

For instance, as depicted in Figure 3, based on the simple practice implementation detailed in Appendix D, we fix λ n = 1 e -7, σ 2 1 ,n = 0 . 05, and we set ( σ init 0 ,n ) 2 = 1 e -5, ( σ end 0 ,n ) 2 = 1 e -7, and gradually reduce σ 2 0 ,n from the initial value to the end value. Initially, the free space is relatively small and the penalty region is relatively large. However, the penalty imposed on the parameters within the initial penalty region is also minor, making it challenging to shrink these parameters to zero. As we progressively decrease σ 2 0 ,n , the free space enlarges,

the penalty region diminishes, and the penalty on the parameters within the penalty region intensifies simultaneously. Once σ 2 0 ,n equals ( σ end 0 ,n ) 2 , the parameters within the penalty region will be proximate to zero, and the parameters outside the penalty region can vary freely in almost all areas of the parameter space.

For model compression tasks, achieving the desired sparsity involves several steps post the initial training phase outlined in Algorithm 1. First, determine the initial threshold value based on pre-set values for λ n , σ 2 1 ,n , and ( σ init 0 ,n ) 2 . Then, compute the proportion of the parameters in the initial model whose absolute values are lower than this threshold. This value serves as an estimate for anticipated sparsity. Adjust the ( σ init 0 ,n ) 2 value until the predicted sparsity aligns closely with the desired sparsity level.

F Construction of Multi-Horizon Joint Prediction Intervals

To construct valid joint prediction intervals for multi-horizon forecasting, we estimate the individual variances for each time step in the prediction horizon using procedures similar to one-step-ahead forecasting. We then adjust the critical value by dividing α by the number of time steps m in the prediction horizon and use z α/ (2 m ) as the adjusted critical value. This Bonferroni correction ensures the desired coverage probability across the entire prediction horizon.

As an example, we refer to the experiments conducted in 5.1.2, where we work with a set of training sequences denoted by D n = { y ( i ) 1 , . . . , y ( i ) T , . . . y ( i ) T + m } n i =1 . Here, y ( i ) T +1 , y ( i ) T +2 , . . . , y ( i ) T + m represent the prediction horizon for each sequence, while T represents the length of the observed sequence.

Train a model by the proposed algorithm, and denote the trained model by ( ˆ β , ˆ γ ).

Calculate ˆ σ 2 as an estimator of σ 2 ∈ R m × 1 :

where µ ( ˆ β , y ( i ) 1: T ) = ˆ y ( i ) T +1: T + m , and ⊗ denotes elementwise product.

For a test sequence y (0) 1: T , calculate

Let ˆ ς ∈ R m × 1 denote the vector formed by the diagonal elements of ˆ Σ .

The Bonferroni simultaneous prediction intervals for all elements of y (0) T +1: T + m are given by

where ◦ 1 2 represents the element-wise square root operation.

The Bayesian method is particularly advantageous when dealing with a large number of non-zero connection weights in ( ˆ β , ˆ γ ), making the computation of the Hessian matrix of the log-likelihood function costly or unfeasible. For detailed information on utilizing the Bayesian method for constructing prediction intervals, please refer to [21].

G Numerical Experiments

G.1 French Electricity Spot Prices

Dataset. The given dataset contains the spot prices of electricity in France that were established over a period of four years, from 2016 to 2020, using an auction market. In

this market, producers and suppliers submit their orders for the next day's 24-hour period, specifying the electricity volume in MWh they intend to sell or purchase, along with the corresponding price in €/MWh. At midnight, the Euphemia algorithm, as described in [59], calculates the 24 hourly prices for the next day, based on the submitted orders and other constraints. This hourly dataset consists of 35064 observations, covering (3 × 365 + 366) × 24 periods. Our main objective is to predict the 24 prices for the next day by considering different explanatory variables such as the day-ahead forecast consumption, day-of-the-week, as well as the prices of the previous day and the same day in the previous week, as these variables are crucial in determining the spot prices of electricity. Refer to Figure 4 for a visual representation of the dataset.

The prediction models and training settings described below are the same for all 24 hours.

(a)

FrenchElectricitySpotPrices

800

Price (E/MWh)

600

400

200

0

2016

2017

2018

2019

2020

Date

Proper training set

Calibration set t = Tpeed start

(b)

Prediction Model. We use an MLP with one hidden layer of size 100 and the sigmoid activation function as the underlying prediction model for all methods and all hours.

Training: Baselines. For all CP baselines, we train the MLP for 300 epochs using SGD with a constant learning rate of 0 . 001 and momentum of 0 . 9, the batch size is set to be 100. For the ACI, we use the same list of γ ∈ { 0 , 0 . 000005 , 0 . 00005 , 0 . 0001 , 0 . 0002 , 0 . 0003 , 0 . 0004 , 0 . 0005 , 0 . 0001 , 0 . 0002 , 0 . 0003 , 0 . 0004 , 0 . 0005 , 0 . 0006 , 0 . 0007 , 0 . 0008 , 0 . 0009 , 0 . 001 , 0 . 002 , 0 . 003 , 0 . 004 , 0 . 005 , 0 . 006 , 0 . 007 , 0 . 008 , 0 . 009 , 0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 , 0 . 06 , 0 . 07 , 0 . 08 , 0 . 09 } as in [31].

Training: Our Method. For our method PA, we train a total of 300 epochs with the same learning rate, momentum, and batch size. We use T 1 = 150, T 2 = 160, T 3 = 260. We run SGD with momentum for t < T 1 and SGHMC with temperature= 1 for t > = T 1 . For the mixture Gaussian prior, we fix σ 2 1 ,n = 0 . 01, ( σ init 1 ,n ) 2 = 1 e -5, ( σ end 1 ,n ) 2 = 1 e -6, and λ n = 1 e -7.

G.2 EEG, MIMIC-III, and COVID-19

EEG The EEG dataset, available at here, served as the primary source for the EEG signal time series. This dataset contains responses from both control and alcoholic subjects who were presented with visual stimuli of three different types. To maintain consistency with

Unused data

Test data point t = Tpred end

0 =

previous work [33], we utilized the medium-sized version, consisting of 10 control and 10 alcoholic subjects. We focused solely on the control subjects for our experiments, as the dataset summaries indicated their EEG responses were more difficult to predict. For detailed information on our processing steps, please refer to [33].

COVID-19 We followed the processing steps of previous research [33] and utilized COVID19 data from various regions within the same country to minimize potential distribution shifts while adhering to the exchangeability assumption. The lower tier local authority split provided us with 380 sequences, which we randomly allocated between the training, calibration, and test sets over multiple trials. The dataset can be found at here.

MIMIC-III We collect patient data on the use of antibiotics (specifically Levofloxacin) from the MIMIC-III dataset [46]. However, the subset of the MIMIC-III dataset used in [33] is not publicly available to us, and the authors did not provide information on the processing steps, such as SQL queries and Python code. Therefore, we follow the published processing steps provided in [60]. We remove sequences with fewer than 5 visits or more than 47 visits, resulting in a total of 2992 sequences. We randomly split the dataset into train, calibration, and test sets, with corresponding proportions of 43%, 47%, and 10%. We use the white blood cell count (high) as the feature for the univariate time series from these sequences. Access to the MIMIC-III dataset requires PhysioNet credentialing.

Table 6 presents the training hyperparameters and prediction models used for the baselines in the three datasets, while Table 7 presents the training details used for our method.

We hypothesize that the mediocre performance on the EEG dataset is due to the small size of the prediction model, which has only 2025 parameters and is substantially smaller than the number of training sequences. Hence, we conducted experiments on the EEG dataset using a slightly larger prediction model for different methods. The results are presented in Table 8, and the corresponding training details are provided in Table 6. Again, our method outperforms other baselines as well.

G.3 Autoregressive Order Selection

Model An Elman RNN with one hidden layer of size 1000. Different window sizes (i.e., 1 or 15) will result in a different total number of parameters..

Hyperparameters All training hyperparameters are given in Table 10

G.4 Large-Scale Model Compression

As pointed out by recent summary/survey papers on sparse deep learning [61, 62], the lack of standardized benchmarks and metrics that provide guidance on model structure, task, dataset, and sparsity levels has caused difficulties in conducting fair and meaningful comparisons with previous works. For example, the task of compressing Penn Tree Bank (PTB) word language model [63] is a popular comparison task. However, many previous works [64, 65, 66, 67] have avoided comparison with the state-of-the-art method by either not using the standard baseline model, not reporting, or conducting comparisons at different sparsity levels. Therefore, we performed an extensive search of papers that reported performance on this task, and to the best of our knowledge, the state-of-the-art method is the Automated Gradual Pruning ( AGP ) by [68].

In our experiments, we train large stacked LSTM language models on the PTB dataset at different sparsity levels. The model architecture follows the same design as in [63], comprising an embedding layer, two stacked LSTM layers, and a softmax layer. The vocabulary size for the model is 10000, the embedding layer size is 1500, and the hidden layer size is 1500 resulting in a total of 66 million parameters. We compare our method with AGP at different sparsity levels, including 80%, 85%, 90%, 95%, and 97 . 5% as in [68]. The results are summarized in Table 11, and our method achieves better results consistently. For the AGP , numbers are taken directly from the original paper, and since they only provided one experimental result for each sparsity level, no standard deviation is reported. For our method, we run three independent trials and provide both the mean and standard deviation for each sparsity level. During the initial training stage, we follow the same training procedure as in [63]. The details of the prior annealing and fine-tuning stages of our method for different sparsity levels are provided below.

During the initial training stage, we follow the same training procedure as in [63] for all sparsity, and hence T 1 = 55.

During the prior annealing stage, we train a total of 60 epochs using SGHMC. For all levels of sparsity considered, we fix σ 2 1 ,n = 0 . 5, λ n = 1 e -6, momentum 1 -α = 0 . 9, minibatch size = 20, and we fix T 2 = 5 + T 1 = 60 , T 3 = T 2 +20 = 80. We set the initial temperature τ = 0 . 01 for t ≤ T 3 and and gradually decrease τ by τ = 0 . 01 t -T 3 for t > T 3 .

During the fine tune stage, we apply a similar training procedure as the initial training stage, i.e., we use SGD with gradient clipping and we decrease the learning rate by a constant factor after certain epochs. The minibatch size is set to be 20, and we apply early stopping based on the validation perplexity with a maximum of 30 epochs.

Table 12 gives all hyperparameters (not specified above) for different sparsity levels.

Note that, the mixture Gaussian prior by nature is a regularization method, so we lower the dropout ratio during the prior annealing and fine tune stage for models with relatively high sparsity.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an expansion of sparse deep learning theory, specifically tailored for the analysis of time series data. The authors provide a detailed explanation of the theoretical foundation behind this theory, employing recurrent neural networks (RNNs), particularly long short-term memory (LSTM) networks. The authors demonstrate that their novel approach yields competitive outcomes when compared to other cutting-edge methodologies from diverse research domains, including conformal prediction.\n\nSTRENGTHS:\nThe paper is well-written and grammatically correct with no evident typos. The narrative is clear: related work, theoretical grounding, computation and experiments. \n\nIt is clear that the authors have well investigated the theory of sparse deep learning and it is evident that they know it quite well the field. Hence, the derivations, in principle, seem correct — although I must admit that I found the ideas and derivations presented in the paper to be somewhat challenging to grasp, even though I devoted a considerable amount of time to understanding them. Plus, newcomers to the sparse deep learning theory can find the complete derivations in the Appendix, which is quite extensive.\n\nThe proposed method is tested in three relevant tasks related to time series data, and it is evaluated on at least four real-world datasets. This thorough evaluation approach provides strong empirical evidence to support the theoretical foundations of the method, demonstrating its validity and applicability.\n\nWEAKNESSES:\nBefore outlining the weaknesses, I repeat again that I am a new reader of sparse deep learning theory. However, I still find that the paper might exhibit some weaknesses:\n\n1. First of all, while I believe the theoretical side of the paper is of good quality, I encountered difficulties in understanding the concepts due to the overwhelming presence of complex notation. Sometimes there are many variables, and many subscripts, which in temporal series can be, e.g. $y_{k-1:k-R_l}$. This makes the reading difficult and sometimes cumbersome.\n2. After reading the paper, I still cannot see why do you choose conformal prediction as the main comparison field. I lack some more context or motivation on the paper. The paper might be already interesting for the natural extension of sparse deep learning for time series data, but I also believe there should be some kind of justification of why these methods are a good alternative. \n3. The point above leads me to the following point: I worry that the literature review for Conformal Prediction might be scarce. Authors comment on the work by [1], but there is no reference to other works in the same line of this work [3], or other contemporary works such as [2]. \n4. And following this line, I wonder if this theory might be very challenging to apply in real-world scenarios compared to CP. The beauty of CP is that you can basically apply it on top of any ML model in a “plug-and-play” fashion, with the cost of very strong assumptions such as the ****************exchangeability**************** assumption. But it is more straightforward to apply and to easily understand than this sparse deep learning technique. I would like to know what the authors think about this.\n\n[1] Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34:1660–1672, 2021.\n\n[2] Barber, R. F., Candes, E. J., Ramdas, A., & Tibshirani, R. J. (2023). Conformal prediction beyond exchangeability. *The Annals of Statistics*, *51*(2), 816-845.\n\n[3] Romano, Y., Patterson, E., & Candes, E. (2019). Conformalized quantile regression. *Advances in neural information processing systems*, *32*.\n\nQUESTIONS:\nHere are some questions/suggestions for the authors:\n\n- In Figure 1, the circles shouldn’t contain some variables or some naming? Are these the states of the RNN? This figure is a bit misleading to me.\n- In line 239, what is ******local-trap issue?:****** A bit more context could help the reader.\n- Is there any reason why you choose LSTM against other architectures? Some study on whether this applies to GRU or other RNN could be interesting.\n- Do you believe that the LSTM is providing good results because of the forget gate? Or do you think there is some alignment between your results and the use of LSTM, that might actually shadow the developed theroy?\n- In Table 3, when considering the autoregressive order selection, it appears that the errors of the proposed model are comparable to those of the baselines that only employ RNN. Notably, the absence of standard errors in the results raises some questions. It is possible that I may have overlooked something, or perhaps the authors could provide additional clarity regarding the significance of these findings. One noteworthy aspect is whether the reduced number of hidden links is the primary factor contributing to the performance of the proposed method.\n\nMinor questions:\n\n- What is PI length? Prediction Interval length?  It is just presented in the table with no extra explanation.\n- Why the method is called PA? When do you choose the name?\n\nLIMITATIONS:\nThe authors did not explicitly address the limitations of their work, which raises concerns about the feasibility of deploying their proposed model in practical settings. It is crucial to understand the potential challenges and constraints associated with implementation. Considering the computational time and power required, it is essential to evaluate the trade-off involved and compare it to the relatively inexpensive nature of conformal prediction. Understanding these aspects would provide valuable insights into the practical implications and potential drawbacks of the proposed model. Plus, I lack some direct comments on the limitations with respect to conformal prediction, since my prior belief is that the their presented comparison w.r.t conformal prediction methods is scarce. \n\nWith all this in mind, I consider that the paper needs some further improvement, and I await for the author’s and other reviewers’ comments for the possibility of increasing the score.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the theoretical analysis of sparse deep learning to time series data.  Statistical propoerties of sparse RNNs are investigated including consistency and asymptotical behaviour.  The paper presents some numerical results showing that sparse deep learning outperforms existing methods in predicting uncertainty quantification for time series data, as well as in identifying the autoregressive order for time series data and large-scale model compression. The proposed method has practical implications in some fields, such as finance, healthcare, and energy.\n\nSTRENGTHS:\n(1) The paper addresses an important and underexplored topic by studying sparse deep learning for dependent time series data. This fills a gap in existing research, which has mostly focused on i.i.d. data.\n\n(2) The paper presents the empirical results that show the superiority of sparse deep learning over existing methods in predicting uncertainty quantification and autoregressive order identification for time series data. This demonstrates the practical effectiveness of the proposed method.\n\nWEAKNESSES:\n(1)Some of the theorems in references are not given in the text and some symbols are confused, e.g.,\n 1)Please give the concrete content of Theorem 2 from [1] that the authors cite.\n 2)Please give the definition of $O_{P^*}$ on line 174.\n 3)The lemma used on line 575 should be lemma S1 of section 4 in [2]. The modifications of the above weaknesses can enhance the readability of the article.\n\n(2)There are related references that study the theoretical analysis of deep neural networks for temporally dependent observations, such as [3]. I encourage the authors to check them out and see if they should be included in the related work.\n\n[1]Pentti Saikkonen. Dependent versions of a central limit theorem for the squared length of a sample mean. Statistics & probability letters, 1995.\n[2]Yan Sun, Qifan Song, and Faming Liang. Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association, 2021.\n[3]M. Ma and A. Safikhani. Theoretical analysis of deep neural networks\nfor temporally dependent observations, in NeurIPS, 2022.\n\nQUESTIONS:\n(1)The authors assume that the time series are $\\alpha$-mixing process, but in fact there are many other mixing processes, such as $\\beta$-mixing and $\\tau$-mixing. Please explain how the $\\alpha$-mixing affects the theoretical analysis of algorithms. Whether the different mixing will cause different results?\n\n(2)Whether the data used in the experiment satisfy the $\\alpha$-mixing? If yes, please provide some supporting materials. Some auto-regressive models are not $\\alpha$-mixed, e.g.,\\begin{equation}\\label{1}\n\\mathbf{x}_t=f_0(\\mathbf{x}_{t-1})+\\epsilon_t,~t\\in \\mathbb{Z},\n\\end{equation},\nwhere $f_0:[-K,K]^d\\rightarrow[-K,K]^d$ is a Lipschitz function with Lipschitz constant $\\leq$ 2 and $\\epsilon_t\\sim\\mathcal{B}(0.5)$ [4].\n\n(3)Please give the proof of Lemma B.1 for completeness.\n\n(4)What is the biggest difference between the algorithm the authors proposed in section 4 and the algorithm proposed in [5]？\n\n(5)This article is too similar to [5] and [2]. It seems that the result of this paper is an extension from i.i.d. observations to dependent data. Please clarify what are the main difficulties in the theoretical analysis for time series data compared with i.i.d. data.\n\n[2]Yan Sun, Qifan Song, and Faming Liang. Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association,  2021.\n[4]D. W. Andrews. Non-strong mixing autoregressive processes. Journal of Applied Probability, 1984.\n[5]Yan Sun, Wenjun Xiong, and Faming Liang. Sparse deep learning: A new framework immune to local traps and miscalibration. Advances in Neural Information Processing Systems, 2021.\n\nLIMITATIONS:\nThis is a theoretical work. But part of the proof is a little vague such as proof of Theorem 3.11 and Theorem 3.12. Authors should highlight the key differences  of analysis techniques  between their work and [2] [5] (instead of just using the different tools).\n\n[2]Yan Sun, Qifan Song, and Faming Liang. Consistent sparse deep learning: Theory and Computation.\n[5]Yan Sun, Wenjnd computation. Journal of the American Statistical Association,  2021.un Xiong, and Faming Liang. Sparse deep learning: A new framework immune to local traps and miscalibration. Advances in Neural Information Processing Systems, 2021.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFor iid data, sparse DL has been shown as a way towards consistency of input-output mappings and well understood distribution of model predictions. This theory is missing for time-series data however, which the authors thus introduce here. In particular, the following results for RNNs with Gaussian mixture parameters are presented:\n\n- Consistency of posteriors, structure selection and input-output mappings\n- (asymptotic) normality of predicted values\n\nSTRENGTHS:\nThe paper was well organized and written. \n\nExtending the recent results from sparse DL work to time-series is valuable and a novel contribution.\n\nThe consistency results presented here are convincing and valuable (at least to this reviewer who is not an expert in this field) and should pave way for a lot further work in the area -- the assumptions made in the theoretical parts are fair for this nascent area of study and provides a good starting point.\n\n\nThe authors show how the method can be framed as regularization in the way of laplace approximation at the MAP estimator which is useful in practice. \n\nThe experiments provide evidence of superior performance in comparison to CP methods.\n\nWEAKNESSES:\nComparison to other possible approaches to uncertainty quantification is lacking: there is no comparisons or discussion of this method to the alternatives mentioned in the introduction: \"include multi-horizon probabilistic forecasting [33], dropout-based methods [34], and recursive Bayesian methods [35].\". This is the case with the experiments, but also on theoretical side. Naturally theory for the other methods may not exist in the same vain, but it would have been useful if the authors tried to make some further comparisons. \n\nOne limiting assumption here is that $\\mu*{\\star}$ can be approximated by a sparse RNN arbitrarily well (i.e. ground-truth sparse model).   Especially since the connectivity of this *true* RNN model is assumed to be limited. Also, as the authors point out, universal approximation does not hold fully (but can still hold for some nonlinear function classes).  Relatedly, theorem 3.8 relates to the estimation error only, but what about the approximation error?\n\nThe assumptions and conditions required for Theorems 3.8 onward are not discussed in sufficient depth.\n\nThe AR order selection experiments, while promising, are quite narrow (only 5 datasets from a single model). More different datasets would be needed with different settings (of window sizes etc.)\n\nQUESTIONS:\nMinor things:\n\nThere could be a bit more discussion about mixture priors: if and why they are a reasonable choice\n\nConsider adding a comma around in line 117 to make the meaning of \"only\" and \"or\" more clear in sentence \"the input $x_i$ can contain only $y_{i-1}$ or $y_{i-1:i-r}$ for some r>1\". Now it's not clear if only refers to \"only either of a or b\", or \"only a, or if not a then b is the only alternative\".\n\nI found it  confusing that $H$ was used to denote the layers and $L$ the number of hidden neurons. I would have found them more logical the other way around.\n\nAs a misc. note: I found the font of the paper odd, but that could be a local issue on my side.\n\nLIMITATIONS:\nIn general limitations are discussed fairly well --  especially given the limited space. As always, this type of theoretical works is limited by assumptions that are made in particular. Here e.g. Gaussian mixture prior is assumed and I would be curious to hear more about the limitations of this choice of prior in more detail. What alternatives are there? The mixing assumptions can be limiting too, but the authors do acknowledge that much.\n\nAs mentioned above, it would also be nice to have bit more discussion on the assumptions and conditions of Theorems 3.8 and the other consequent theoretical results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to extend sparse deep learning in the context of time series data. This context is different from the classical one as samples are not *i.i.d.* but dependent. The paper shows theoretical results -- posterior consistency and asymptotic normality of the weights --, a computation method based on posterior annealing and some numerical experiments that showcase the performance of the method on real datasets.\n\nSTRENGTHS:\n- The paper extend sparse deep learning result to the challenging context of dependent data, applicable for time series.\n- The many reported results show improvement of the method compared to existing state-of-the-art methods and seems to yield interpretable models for time series.\n\nWEAKNESSES:\n- It is not totally clear what is really different between the results in [17] and the results presented here (see **Q1**).\n- The writing is not always clear (**Q2**), in particular for non-expert in sparse deep learning. It is not clear why `Eq.(4)` admits an non-empty set of minimizers (**Q3**).\n- There are many hyperparameters in the methods and while there value is reported, it is not clear how to select them (**Q4**).\n\nQUESTIONS:\n- **Q1** - The theoretical results are very similar to the ones of [17], which considers *iid* data with DNN. As the proposed method consider the input of the network as fixed segments $x_{i:i-M_l+1}$, the considered networks can be assimilated as weight-tied DNN, so the class of architecture is not really different. And in `Theorem3.8`, it is not clear where the dependence between the samples intervenes. In particular, I am surprised that we get similar sample efficiency rates between the two settings, and I wonder what is lost by dropping the iid assumption. Could the authors comment on this point?\n- **Q2** -  Many details are deferred to the appendix or referred to in other articles. This makes the paper sometimes hard to read. In particular, `Section.4` is very hard to follow, in particular the part on the transformation of the Bayesian method into a regularization method. As this constitutes the core of the proposed method, I think this could be better explained in the main article.\n- **Q3** - In equation (4), it is unclear what are the conditions for  $\\varpi_n$ so that the argmin is not empty. In particular, as the size of the neural network in the set $\\mathcal G_n$ is constrained in `Assumption3.4`. This specific assumption that there exists a minimizer is used in Eq. bellow l.657, which relies on the fact that for all $n$, $\\|\\mu(X, \\beta, \\gamma) - \\mu^*\\|_{L^2(\\Omega)} \\le \\varpi_n$ is not empty. This is not guaranteed I think, in particular as $\\varpi_n$ goes to 0 at the same rate as the network size increases, while the data point to interpolate increases much faster. Note that a similar question arises in the proof for [17]. It seems that the results in [17] show that universal approximation holds for a specific class of functions even for sparse DNN, but it is unclear if the constraints in `Assumption3.4` allows for this. But maybe this is due to a misunderstanding on my side.\n- **Q4** The method relies on many hyperparameters, that are specified in the appendix. However, the process for their selection is not reported, and it would be nice to understand how sensitive the hyperparameter selection of the model is.\n\n**Minor remarks and typos:**\n- **m1** - `Section 5.2` - Isn't it possible to estimate the AR order of window size =1 by looking at the average dependency length? Or at least estimating if changing one sample changes the result?\n- **m2** - The proposed results seem to relate to the compress sensing and sparse coding theory, it would be nice to draw the connection between these two fields in the introduction. But I have no specific results in mind, so maybe it is too far-fetched.\n\n*unnumbered remarks don't call for answers*\n- Eq. (2) - Missing $+ v^{H_n}z^{H_n}_{i-1}$ at the end to show the structure that is not just feedforward?\n- l.123 - $\\gamma = \\{\\gamma_i \\in \\{0, 1\\} : j=1\\dots K_n\\}$ -> $\\gamma \\in \\{0, 1\\}^{K_n}$?\n- l.155 - an RNN of size ... has been\" -> \"... is\".\n- l.192 - mismatch between name and acronym `MIPP` and `marginal posterior inclusion probability`.\n- l.218 - $H_n$ is already the number of layers (l.113), finding a different notation would help the reader follow the arguments.\n- l305 - \"In particular, Our method\" -> \"our\".\n\nLIMITATIONS:\n* see weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the sparse DNN theory from [1, 2] to time series data and proposes the sparsity method for RNNs. This method aims to improve uncertainty quantification on time-series tasks and provide state-of-the-art RNN compression.\n\nSTRENGTHS:\n* The combination of DNN sparsification and uncertainty quantification for time series data is an interesting direction.\n* The theoretical groundwork seems complete and convincing.\n\nWEAKNESSES:\n**Novelty**\n\n* **Theory** - The theoretical results for RNNs and time series seem almost identical to [1, 2] aside from minor time series specifics. I appreciate that the authors explicitly discuss the connection to these works. However, I have concerns if the theoretical contribution is significant enough. \n* **Computation** - the prior annealing algorithm (PA) for model sparsification is identical to [2]. Constructing the prediction intervals for time series might be novel, but it is a minor contribution from my perspective. \n\n**Large-scale model compression**\n\nThe authors state that one of their contributions is s.o.t.a. large-scale RNN compression method. \n\n1. In my opinion, the scope of the model compression area is much larger than the weight sparsification. One should either reformulate \"compression\" to \"weight sparsification\" or compare with low-rank approximation, quantization, and knowledge distillation methods. \n2. The proposed method is compared only with AGP[5] on PTB. The authors state that this area has problems with benchmarks and baselines. I agree that it makes the fair comparison challenging. On the other hand, I think the results on a single dataset against a single baseline from 2017 are not sufficient to support such a strong claim. I would consider at least one more dataset and compare with, e.g., GraNet[6], and some sparsification methods for RNNs: e.g., [7, 12]. Otherwise, please revise the claims and contributions.\n3. (Minor) All experiments are entirely deferred to the appendix. If the authors declared this as one of the contributions, I would expect it in the main paper. \n\n**Related work** \n\n* **Uncertainty quantification:** please add DropConnect[3] and deep ensembles[4] as uncertainty estimation methods and consider them for comparison in the multi-horizon setting.\n* **Sparse DNN** should be largely extended by discussing different before-training [8], post-training [9] and during-training [5,6,7,10,11] approaches. Regarding the RNN sparsification, I would consider citing and discussing the following works [12, 13, 14]. \n\n**Clarity**\n\nIt was hard to follow some theoretical parts of the paper due to large equations and overloaded notation, e.g., Theorem 3.8. I can see that the authors put effort into simplifying the notation and supporting the theory with discussions in some places. Nevertheless, I think it would be great to do an extra pass to make reading more clear. \n\n(Minor) Figure 1 takes a lot of space but just aims to explain M_l and R_l notation and the RNN setting. I would suggest making it more informative or maybe consider moving it to the appendix. The model compression results arguably seem more important than this figure. \n\n---\n\n[1] Consistent sparse deep learning: Theory and computation. JASA, 2021\n\n[2] Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration, NeurIPS 2021\n\n[3] Dropconnect is effective in modeling uncertainty of bayesian deep networks 2019\n\n[4] Simple and scalable predictive uncertainty estimation using deep ensembles NIPS2017\n\n[5] To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017\n\n[6] Sparse training via boosting pruning plasticity with neuroregeneration, NeurIPS2021 \n\n[7] Bayesian Compression for Natural Language Processing, EMNLP2018\n\n[8] SNIP: Single-shot network pruning based on connection sensitivity, ICLR2019\n\n[9] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ICLR2019\n\n[10] Learning Sparse Neural Networks through L0 Regularization, ICLR2018\n\n[11] Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science, Nature 2018\n\n[12] Spectral Pruning for Recurrent Neural Networks, AISTATS2020\n\n[13] Structured Sparsification of Gated Recurrent Neural Networks, AAAI2020\n\n[14] Stage-Wise Magnitude-Based Pruning for Recurrent Neural Networks. TNNLS2022.\n\nQUESTIONS:\n1. Could the authors address the novelty and model compression evaluation concerns? \n2. In my understanding, deep ensembles are s.o.t.a. for uncertainty estimation. Is it a reasonable baseline in the multi-horizon setting?\n\nLIMITATIONS:\nThe authors addressed the limitations to some extent. One can also discuss more limitations of the proposed algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-19"}