SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mingxuan Zhang
zhan3692@purdue.edu
Department of Biostatistics, Epidemiology, and Informatics
yan.sun@pennmedicine.upenn.edu
Faming Liang
fmliang@purdue.edu
Sparse deep learning has become a popular technique for improving the performance of deep neural networks in areas such as uncertainty quantification, variable selection, and large-scale network compression. However, most existing research has focused on problems where the observations are independent and identically distributed (i.i.d.), and there has been little work on the problems where the observations are dependent, such as time series data and sequential data in natural language processing. This paper aims to address this gap by studying the theory for sparse deep learning with dependent data. We show that sparse recurrent neural networks (RNNs) can be consistently estimated, and their predictions are asymptotically normally distributed under appropriate assumptions, enabling the prediction uncertainty to be correctly quantified. Our numerical results show that sparse deep learning outperforms state-of-the-art methods, such as conformal predictions, in prediction uncertainty quantification for time series data. Furthermore, our results indicate that the proposed method can consistently identify the autoregressive order for time series data and outperform existing methods in large-scale model compression. Our proposed method has important practical implications in fields such as finance, healthcare, and energy, where both accurate point estimates and prediction uncertainty quantification are of concern.
Over the past decade, deep learning has experienced unparalleled triumphs across a multitude of domains, such as time series forecasting [1, 2, 3, 4, 5], natural language processing [6, 7], and computer vision [8, 9]. However, challenges like generalization and miscalibration [10] persist, posing potential risks in critical applications like medical diagnosis and autonomous vehicles.
In order to enhance the performance of deep neural networks (DNNs), significant research efforts have been dedicated to exploring optimization methods and the loss surface of the DNNs, see, e.g., [11, 12, 13, 14, 15, 16], which have aimed to expedite and direct the convergence of DNNs towards regions that exhibit strong generalization capabilities. While these investigations are valuable, effectively addressing both the challenges of generalization
and miscalibration require additional and perhaps more essential aspects: consistent estimation of the underlying input-output mapping and complete knowledge of the asymptotic distribution of predictions. As a highly effective method that addresses both challenges, sparse deep learning has been extensively studied, see, e.g., [17, 18, 19, 20, 21]. Nevertheless, it is important to note that all the studies have been conducted under the assumption of independently and identically distributed (i.i.d.) data. However, in practice, we frequently encounter situations where the data exhibits dependence, such as time series data.
The primary objective of this paper is to address this gap by establishing a theoretical foundation for sparse deep learning with time series data. Specifically, we lay the foundation within the Bayesian framework. For RNNs, by letting their parameters be subject to a mixture Gaussian prior, we establish posterior consistency, structure selection consistency, input-output mapping estimation consistency, and asymptotic normality of predicted values. We validate our theory through numerical experiments on both synthetic and real-world datasets. Our approach outperforms existing state-of-the-art methods in uncertainty quantification and model compression, highlighting its potential for practical applications where both accurate point prediction and prediction uncertainty quantification are of concern.
Additionally, we would elaborate the contribution of this paper in a broader context of statistical modeling. As discussed in [22], two distinct cultures exist for statistical modeling: the 'data modeling culture' and the 'algorithmic modeling culture'. The former focuses on simple generative models that explain the data, potentially lacking a consistent estimate of the true data-generating mechanism due to the model's inherent simplicity. The latter, on the other hand, aims to find models that can predict the data regardless of complexity. Our proposed method occupies a middle ground between these two cultures. It seeks to identify a parsimonious model within the realm of complex models, while also ensuring a consistent estimation of the true data-generating mechanism. From this perspective, this work and related ones, e.g., [17, 18, 19, 20, 21], represent a hybridization of the 'algorithmic modeling culture' and the 'data modeling culture', which holds the potential to expedite advancements in modern data science.
Sparse deep learning. Theoretical investigations have been conducted on the approximation power of sparse DNNs across different classes of functions [23, 24]. Recently, [20] has made notable progress by integrating sparse DNNs into the framework of statistical modeling, which offers a fundamentally distinct neural network approximation theory. Unlike traditional theories that lack data involvement and allow connection weights to assume values in an unbounded space to achieve arbitrarily small approximation errors with small networks [25], their theory [20] links network approximation error, network size, and weight bounds to the training sample size. They show that a sparse DNN of size  $O(n / \log (n))$  can effectively approximate various types of functions, such as affine and piecewise smooth functions, as  $n\to \infty$ , where  $n$  denotes the training sample size. Additionally, sparse DNNs exhibit several advantageous theoretical guarantees, such as improved interpretability, enabling the consistent identification of relevant variables for high-dimensional nonlinear systems. Building upon this foundation, [21] establishes the asymptotic normality of connection weights and predictions, enabling valid statistical inference for predicting uncertainties. This work extends the sparse deep learning theory of [20, 21] from the case of i.i.d data to the case of time series data.
Uncertainty quantification. Conformal Prediction (CP) has emerged as a prominent technique for generating prediction intervals, particularly for black-box models like neural networks. A key advantage of CP is its capability to provide valid prediction intervals for any data distribution, even with finite samples, provided the data meets the condition of exchangeability [26, 27]. While i.i.d. data easily satisfies this condition, dependent data, such as time series, often doesn't. Researchers have extended CP to handle time series data by relying on properties like strong mixing and ergodicity [28, 29]. In a recent work, [30] introduced a random swapping mechanism to address potentially non-exchangeable data, allowing conformal prediction to be applied on top of a model trained with weighted samples. The main focus of this approach was to provide a theoretical basis for the
differences observed in the coverage rate of the proposed method. Another recent study by [31] took a deep dive into the Adaptive Conformal Inference (ACI) [32], leading to the development of the Aggregation Adaptive Conformal Inference (AgACI) method. In situations where a dataset contains a group of similar and independent time series, treating each time series as a separate observation, applying a CP method becomes straightforward [33]. For a comprehensive tutorial on CP methods, one can refer to [34]. Beyond CP, other approaches for addressing uncertainty quantification in time series datasets include multi-horizon probabilistic forecasting [35], methods based on dropout [36], and recursive Bayesian approaches [37].
3 Sparse Deep Learning for Time Series Data: Theory
Let  $D_{n} = \{y_{1},\ldots ,y_{n}\}$  denote a time series sequence, where  $y_{i}\in \mathbb{R}$ . Let  $(\Omega ,\mathcal{F},P^{*})$  be the probability space of  $D_{n}$ , and let  $\alpha_{k} = \sup \{|P^{*}(y_{j}\in A,y_{k + j}\in B) - P^{*}(y_{j}\in A)P^{*}(y_{k + j}\in B)|:A,B\in \mathcal{F},j\in \mathbb{N}^{+}\}$  be the  $k$ -th order  $\alpha$ -mixing coefficient.
Assumption 3.1. The time series  $D_{n}$  is (strictly) stationary and  $\alpha$ -mixing with an exponentially decaying mixing coefficient and follows an autoregressive model of order  $l$
$$
y _ {i} = \mu^ {*} \left(y _ {i - 1: i - l}, \boldsymbol {u} _ {i}\right) + \eta_ {i}, \tag {1}
$$
where  $\mu^{*}$  is a non-linear function,  $y_{i - 1:i - l} = (y_{i - 1},\ldots ,y_{i - l})$ $\pmb{u}_i$  contains optional exogenous variables, and  $\eta_i\stackrel {i.i.d.}{\sim}N(0,\sigma^2)$  with  $\sigma^2$  being assumed to be a constant.
Remark 3.2. Similar assumptions are commonly adopted to establish asymptotic properties of stochastic processes [38, 39, 40, 41, 42, 29]. For example, the asymptotic normality of the maximum likelihood estimator (MLE) can be established under the assumption that the time series is strictly stationary and ergodic, provided that the model size is fixed [38]. A posterior contraction rate of the autoregressive (AR) model can be obtained by assuming it is  $\alpha$ -mixing with  $\sum_{k=0}^{\infty} a_k^{1-2/s} < \infty$  for some  $s > 2$  which is implied by an exponentially decaying mixing coefficient [42]. For stochastic processes that are strictly stationary and  $\beta$ -mixing, results such as uniform laws of large numbers and convergence rates of the empirical processes [39, 40] can also be obtained.
Remark 3.3. Extending the results of [20, 21] to the case that the dataset includes a set of i.i.d. time series, with each time series regarded as an individual observation, is straightforward, this is because all observations are independent and have the same distribution.
3.1 Posterior Consistency
Both the MLP and RNN can be used to approximate  $\mu^{*}$  as defined in (1), and for simplicity, we do not explicitly denote the exogenous variables  $\mathbf{u}_i$  unless it is necessary. For the MLP, we can formulate it as a regression problem, where the input is  $\pmb{x}_i = y_{i - 1:i - R_l}$  for some  $l\leq R_l\ll n$ , and the corresponding output is  $y_{i}$ , then the dataset  $D_{n}$  can be expressed as  $\{(x_i,y_i)\}_{i = 1 + R_l}^n$ . Detailed settings and results for the MLP are given in Appendix B.3. In what follows, we will focus on the RNN, which serves as an extension of the previous studies. For the RNN, we can rewrite the training dataset as  $D_{n} = \{y_{i:i - M_{l} + 1}\}_{i = M_{l}}^{n}$  for some  $l\leq R_l < M_l\ll n$ , i.e., we split the entire sequence into a set of shorter sequences, where  $R_{l}$  denotes an upper bound for the exact AR order  $l$ , and  $M_{l}$  denotes the length of these shorter sequences (see Figure 1). We assume  $R_{l}$  is known but not  $l$  since, in practice, it is unlikely that we know the exact order  $l$ .
For simplicity of notations, we do not distinguish between weights and biases of the RNN. In this paper, the presence of the subscript  $n$  in the notation of a variable indicates its potential to increase with the sample size  $n$ . To define an RNN with  $H_{n} - 1$  hidden layers, for  $h\in \{1,2,\ldots ,H_n\}$ , we let  $\psi^h$  and  $L_{h}$  denote, respectively, the nonlinear activation function and the number of hidden neurons at layer  $h$ . We set  $L_{H_n} = 1$  and  $L_0 = p_n$ , where  $p_n$  denotes a generic input dimension. Because of the existence of hidden states from the past, the input  $\pmb{x}_i$  can contain only  $y_{i - 1}$  or  $y_{i - 1:i - r}$  for some  $r > 1$ . Let  $\pmb{w}^{h}\in \mathbb{R}^{L_{h}\times L_{h - 1}}$  and  $\pmb{v}^{h}\in \mathbb{R}^{L_{h}\times L_{h}}$  denote the weight matrices at layer  $h$ . With these notations, the output of the step  $i$  of an RNN model can be expressed as
$$
\mu \left(\boldsymbol {x} _ {i}, \left\{\boldsymbol {z} _ {i - 1} ^ {h} \right\} _ {h = 1} ^ {H _ {n} - 1}, \beta\right) = \boldsymbol {w} ^ {H _ {n}} \psi^ {H _ {n} - 1} \left[ \dots \psi^ {1} \left[ \boldsymbol {w} ^ {1} \boldsymbol {x} _ {i} + \boldsymbol {v} ^ {1} \boldsymbol {z} _ {i - 1} ^ {1} \right] \dots \right], \tag {2}
$$
Figure 1: A multi-layer RNN with an input window size of  $k$ . We restrict the use of the RNN's outputs until the hidden states have accumulated a sufficient quantity of past information to ensure accurate predictions.
where  $\pmb{z}_i^h = \psi^h [\pmb{w}^h\pmb{z}_i^{h - 1} + \pmb{v}^h\pmb{z}_{i - 1}^h]$  denotes the hidden state of layer  $h$  at step  $i$  with  $z_0^h = 0$ ; and  $\beta$  is the collection of all weights, consisting of  $K_{n} = \sum_{h = 1}^{H_{n}}(L_{h}\times L_{h - 1}) + \sum_{h = 1}^{H_{n} - 1}(L_{h}^{2}) + L_{h}$  elements. To represent the structure for a sparse RNN, we introduce an indicator variable for each weight in  $\beta$ . Let  $\gamma = \{\gamma_j\in \{0,1\} :j = 1,\dots ,K_n\}$ , which specifies the structure of a sparse RNN. To include information on the network structure  $\gamma$  and keep the notation concise, we redenote  $\mu (\pmb {x}_i,\{z_{i - 1}^h\}_{h = 1}^{H_n - 1},\beta)$  by  $\mu (\pmb{x}_{i:i - M_l + 1},\pmb {\beta},\pmb {\gamma})$ , as  $\{z_{i - 1}^h\}_{h = 1}^{H_n - 1}$  depends only on  $(\beta ,\gamma)$  and up to  $x_{i - 1:i - M_l + 1}$ .
Posterior consistency is an essential concept in Bayesian statistics, which forms the basis of Bayesian inference. While posterior consistency generally holds for low-dimensional problems, establishing it becomes challenging in high-dimensional scenarios. In such cases, the dimensionality often surpasses the sample size, and if the prior is not appropriately elicited, prior information can overpower the data information, leading to posterior inconsistency.
Following [20, 21, 17], we let each connection weight be subject to a mixture Gaussian prior, i.e.,
$$
\beta_ {j} \sim \lambda_ {n} N \left(0, \sigma_ {1, n} ^ {2}\right) + \left(1 - \lambda_ {n}\right) N \left(0, \sigma_ {0, n} ^ {2}\right), \quad j \in \{1, 2, \dots , K _ {n} \}, \tag {3}
$$
by integrating out the structure information  $\gamma$ , where  $\lambda_{n} \in (0,1)$  is the mixture proportion,  $\sigma_{0,n}^{2}$  is typically set to a very small number, while  $\sigma_{1,n}^{2}$  is relatively large. Visualizations of the mixture Gaussian priors for different  $\lambda_{n}$ ,  $\sigma_{0,n}^{2}$ , and  $\sigma_{1,n}^{2}$  are given in the Appendix E.
We assume  $\mu^{*}$  can be well approximated by a sparse RNN given enough past information, and refer to this sparse RNN as the true RNN model. To be more specific, we define the true RNN model as
$$
\left(\boldsymbol {\beta} ^ {*}, \boldsymbol {\gamma} ^ {*}\right) = \quad \arg \min  | \boldsymbol {\gamma} |, \tag {4}
$$
$$
(\beta , \gamma) \in \mathcal {G} _ {n}, \left\| \mu (\boldsymbol {x} _ {i: i - M _ {l} + 1}, \beta , \gamma) - \mu^ {*} (y _ {i - 1: i - l}) \right\| _ {L _ {2} (\Omega)} \leq \varpi_ {n}
$$
where  $\mathcal{G}_n\coloneqq \mathcal{G}(C_0,C_1,\epsilon ,p_n,L_1,L_2,\ldots ,L_{H_n})$  denotes the space of all valid networks that satisfy the Assumption 3.4 for the given values of  $H_{n}$ ,  $p_n$ , and  $L_{h}$ 's, and  $\varpi_{n}$  is some sequence converging to 0 as  $n\to \infty$ . For any given RNN  $(\beta ,\gamma)$ , the error  $|\mu^{*}(\cdot) - \mu (\cdot ,\beta ,\gamma)|$  can be decomposed as the approximation error  $|\mu^{*}(\cdot) - \mu (\cdot ,\beta^{*},\gamma^{*})|$  and the estimation error  $|\mu (\cdot ,\beta^{*},\gamma^{*}) - \mu (\cdot ,\beta ,\gamma)|$ . The former is bounded by  $\varpi_{n}$ , and the order of the latter will be given in Theorem 3.9. For the sparse RNN, we make the following assumptions:
Assumption 3.4. The true sparse RNN model  $(\beta^{*},\gamma^{*})$  satisfies the following conditions:
- The network structure satisfies:  $r_nH_n\log n + r_n\log \bar{L} +s_n\log p_n\leq C_0n^{1 - \epsilon}$ , where  $0 < \epsilon < 1$  is a small constant,  $r_n = |\gamma^*|$  denotes the connectivity of  $\gamma^{*}$ ,  $\bar{L} = \max_{1\leq j\leq H_{n - 1}}L_j$  denotes the maximum hidden state dimension, and  $s_n$  denotes the input dimension of  $\gamma^{*}$ .
- The network weights are polynomially bounded:  $\| \beta^{*}\|_{\infty}\leq E_{n}$ , where  $E_{n} = n^{C_{1}}$  for some constant  $C_1 > 0$ .
Remark 3.5. Assumption 3.4 is identical to assumption A.2 of [20], which limits the connectivity of the true RNN model to be of  $o(n^{1 - \epsilon})$  for some  $0 < \epsilon < 1$ . Then, as implied by
Lemma 3.10, an RNN of size  $O(n / \log (n))$  has been large enough for modeling a time series sequence of length  $n$ . Refer to [20] for discussions on the universal approximation ability of the neural network under this assumption; the universal approximation ability can still hold for many classes of functions, such as affine function, piecewise smooth function, and bounded  $\alpha$ -Hölder smooth function.
Remark 3.6. The existence of the sparse RNN model stems from Lemma 4.1 of [39], which, through the trick of independent block sequence construction, shows that many properties of the i.i.d processes can be extended to mixing processes. While the lemma was proven for the case of  $\beta$ -mixing, the author did mention her doubts about its applicability to  $\alpha$ -mixing. Therefore, at least for the sequences of  $\beta$ -mixing, which implies  $\alpha$ -mixing, the non-empty of the sparse RNN set in (4) can be guaranteed for many classes of functions as mentioned in Remark 3.5.
Assumption 3.7. The activation function  $\psi^h$  is bounded for  $h = 1$  (e.g., sigmoid and tanh), and is Lipschitz continuous with a Lipschitz constant of 1 for  $2 \leq h \leq H_n$  (e.g., ReLU, sigmoid and tanh).
Remark 3.8. Assumption 3.7 mirrors [20, 21], except that we require the activation function for the first layer to be bounded. This extra assumption can be viewed as a replacement for the boundedness assumption for the input variables of a conventional DNN.
Let  $d(p_{1}, p_{2})$  denote the integrated Hellinger distance between two conditional densities  $p_{1}(y|\pmb{x})$  and  $p_{2}(y|\pmb{x})$ . Let  $\pi(\cdot|D_{n})$  be the posterior probability of an event. Theorem 3.9 establishes posterior consistency for the RNN model with the mixture Gaussian prior (3).
Theorem 3.9. Suppose Assumptions 3.1, 3.4, and 3.7 hold. If the mixture Gaussian prior (3) satisfies the conditions:  $\lambda_{n} = O(1 / [M_{l}^{H_{n}}K_{n}[n^{2M_{l}H_{n}}(\bar{L} p_{n})]^{\tau}])$  for some constant  $\tau >0$ ,  $E_{n} / [H_{n}\log n + \log \bar{L}]^{1 / 2}\leq \sigma_{1,n}\leq n^{\alpha}$  for some constant  $\alpha$ , and  $\sigma_{0,n}\leq \min \{1 / [M_l^{H_n}\sqrt{n} K_n(n^{3 / 2}\sigma_{1,n} / H_n)^{2M_lH_n}],1 / [M_l^{H_n}\sqrt{n} K_n(nE_n / H_n)^{2M_lH_n}]\}$ , then there exists an error sequence  $\epsilon_{n}^{2} = O(\varpi_{n}^{2}) + O(\zeta_{n}^{2})$  such that  $\lim_{n\to \infty}\epsilon_n = 0$  and  $\lim_{n\to \infty}n\epsilon_n^2 = \infty$ , and the posterior distribution satisfies
$$
\pi \left(d \left(p _ {\beta}, p _ {\mu^ {*}}\right) \geq C \epsilon_ {n} \mid D _ {n}\right) = O _ {P ^ {*}} \left(e ^ {- n \epsilon_ {n} ^ {2}}\right), \tag {5}
$$
for sufficiently large  $n$  and  $C > 0$ , where  $\zeta_n^2 = [r_nH_n\log n + r_n\log \bar{L} +s_n\log p_n] / n$ ,  $p_{\mu^*}$  denotes the underlying true data distribution, and  $p_\beta$  denotes the data distribution reconstructed by the Bayesian RNN based on its posterior samples.
3.2 Uncertainty Quantification with Sparse RNNs
As mentioned previously, posterior consistency forms the basis for Bayesian inference with the RNN model. Based on Theorem 3.9, we further establish structure selection consistency and asymptotic normality of connection weights and predictions for the sparse RNN. In particular, the asymptotic normality of predictions enables the prediction intervals with correct coverage rates to be constructed.
Structure Selection Consistency It is known that the neural network often suffers from a non-identifiability issue due to the symmetry of its structure. For instance, one can permute certain hidden nodes or simultaneously change the signs or scales of certain weights while keeping the output of the neural network invariant. To address this issue, we follow [20] to define an equivalent class of RNNs, denoted by  $\Theta$ , which is a set of RNNs such that any possible RNN for the problem can be represented by one and only one RNN in  $\Theta$  via appropriate weight transformations. Let  $\nu(\beta, \gamma) \in \Theta$  denote an operator that maps any RNN to  $\Theta$ . To serve the purpose of structure selection in the space  $\Theta$ , we consider the marginal posterior inclusion probability (MIPP) approach. Formally, for each connection weight  $i = 1, \ldots, K_n$ , we define its MIPP as  $q_i = \int \sum_{\gamma} e_{i|\nu(\beta, \gamma)} \pi(\gamma|\beta, D_n) \pi(\beta|D_n)d\beta$ , where  $e_{i|\nu(\beta, \gamma)}$  is the indicator of  $i$  in  $\nu(\beta, \gamma)$ . The MIPP approach selects the connections whose MIPPs exceed a threshold  $\hat{q}$ . Let  $\hat{\gamma}_{\hat{q}} = \{i : q_i > \hat{q}, i = 1, \ldots, K_n\}$  denote an estimator of  $\gamma_* = \{i : e_{i|\nu(\beta^*, \gamma^*)} = 1, i = 1, \ldots, K_n\}$ . Let  $A(\epsilon_n) = \{\beta : d(p_\beta, p_{\mu^*}) \leq \epsilon_n\}$  and  $\rho(\epsilon_n) = \max_{1 \leq i \leq K_n} \int_{A(\epsilon_n)} \sum_{\gamma} |e_{i|\nu(\beta, \gamma)} - e_{i|\nu(\beta^*, \gamma^*)}| \pi(\gamma|\beta, D_n) \pi(\beta|D_n)d\beta$ , which measures
the structure difference on  $A(\epsilon)$  for the true RNN from those sampled from the posterior. Then we have the following Lemma:
Lemma 3.10. If the conditions of Theorem 3.9 are satisfied and  $\rho (\epsilon_n)\to 0$  as  $n\to \infty$ , then
(i)  $\max_{1\leq i\leq K_n}\{|q_i - e_{i|\nu (\pmb{\beta}^*,\pmb{\gamma}^*)}|\}\xrightarrow{p}0$  as  $n\to \infty$
(ii) (sure screening)  $P(\pmb{\gamma}_* \subset \hat{\pmb{\gamma}}_{\hat{q}}) \stackrel{p}{\to} 1$  as  $n \to \infty$ , for any prespecified  $\hat{q} \in (0,1)$ ;
(iii) (consistency)  $P(\gamma_{*} = \hat{\gamma}_{0.5})\xrightarrow{p}1$  as  $n\to \infty$
where  $\xrightarrow{p}$  denotes convergence in probability.
Remark 3.11. This lemma implies that we can filter out irrelevant variables and simplify the RNN structure when appropriate. Please refer to Section 5.2 for a numerical illustration.
Asymptotic Normality of Connection Weights and Predictions The following two theorems establish the asymptotic normality of  $\tilde{\nu} (\beta)$  and predictions, where  $\tilde{\nu} (\beta)$  denotes a transformation of  $\beta$  which is invariant with respect to  $\mu (\cdot ,\beta ,\gamma)$  while minimizing  $\| \tilde{\nu} (\beta) - \beta^{*}\|_{\infty}$ .
We follow the same definition of asymptotic normality as in [21, 43, 19]. The posterior distribution for the function  $g(\beta)$  is asymptotically normal with center  $g^*$  and variance  $G$  if, for  $d_{\beta}$  the bounded Lipschitz metric for weak convergence, and  $\phi_n$  the mapping  $\phi_n: \beta \to \sqrt{n}(g(\beta) - g^*)$ , it holds, as  $n \to \infty$ , that
$$
d _ {\boldsymbol {\beta}} \left(\pi (\cdot | D _ {n}) \circ \phi_ {n} ^ {- 1}, N (0, G)\right)\rightarrow 0, \tag {6}
$$
in  $P^*$ -probability, which we also denote  $\pi(\cdot|D_n) \circ \phi_n^{-1} \rightsquigarrow N(0,G)$ .
The detailed assumptions and setups for the following two theorems are given in Appendix C. For simplicity, we let  $M_{l} = R_{l} + 1$ , and let  $l_{n}(\beta) = \frac{1}{n - R_{l}}\sum_{i = R_{l} + 1}^{n}\log (p_{\beta}(y_{i}|y_{i - 1:i - R_{l}}))$  denote the averaged log-likelihood function. Let  $H_{n}(\beta)$  denote the Hessian matrix of  $l_{n}(\beta)$  and let  $h_{i,j}(\beta)$  and  $h^{i,j}(\beta)$  denote the  $(i,j)$ -th element of  $H_{n}(\beta)$  and  $H_{n}^{-1}(\beta)$ , respectively.