Theoretical analysis of deep neural networks for temporally dependent observations
Authors: Mingliang Ma
Abolfazl Safikhani
Venue: NeurIPS 2022 Accept (2022)

Abstract: Deep neural networks are powerful tools to model observations over time with non-linear patterns. Despite the widespread use
of neural networks in such settings, most theoretical developments of deep neural networks are under the assumption of independent observations, and theoretical results for temporally dependent observations are scarce. To bridge this gap, we study theoretical properties of deep neural networks on modeling non-linear time series data. Specifically, non-asymptotic bounds for prediction error of (sparse) feed-forward neural network with ReLU activation function is established under mixing-type assumptions. These assumptions are mild such that they include a wide range of time series models including auto-regressive models. Compared to independent observations, established convergence rates have additional logarithmic factors to compensate for additional complexity due to dependence among data points. The theoretical results are supported via various numerical simulation settings as well as an application to a macroeconomic data set.

Theoretical analysis of deep neural networks for temporally dependent observations
Mingliang Ma Department of Statistics University of Florida Gainesville, FL 32611 maminglian edu
Abolfazl Safikhani
Department of Statistics George Mason University
Fairfax, VA 22030 asaf ikha@guu...... edu
Abstract
Deep neural networks are powerful tools model observations over time with non-linear patterns, Despite the widespread use of neural networks in such settings, most theoretical developments of deep neural networks are under the assumption of independent observations, and theoretical results for temporally dependent observa- tions are scarce. To bridge this gap, we study theoretical properties of deep neural networks on modeling non- non-linear time series data. Specifically, non-asymptotic bounds for prediction error of (sparse) wred-forward neural network with ReLU activation function is established under mixing-ty terpiggttppeeeee assumptions. These assump- tions are mild such that they include a wide range of time series models including auto-regressive models. Compared to independent observations, established con- vergence rates have additional logarithmic factors to compensate for additional complexity due to dependence among data points. The theoretical results are supported via various numerical simulation settings as well as an application to a macroeconomic data set.
Introduction
Neural networks have the ability to model highly complex relationship among data. If input data are observed data in past with future observations as response, neural networks can be utilized to perform time series forecasting. Examples of application of neural networks in forecasting include biotechnology [.. finance [24], health sciences [20]. and business (29). just to name a selected few. Compared more traditional time series forecasting methods such ARIMA models [9) neural networks have the ability to detect highly non-linear trend and seasonality. In this work, we analyse the prediction error consistency of (deep) feed-forward neural networks fit stationary (non-linear) time series models.
The property of the single hidden layer neural network is well studied in the past few decades. For example, [21] use single hidden layer neural network with a transformed cosine activation function to show that sufficiently complex single hidden layer need-forward network approximate any member a specific class of functions to any desired degree of accuracy. Such an approximation property for neural networks with sigmoidal activation function was also analyzed in 44.)) Further, 3 use monotonic homogeneous activation function (general version of ReLU activation function) and show that both the input dimension and number of hidden units have effect the convergence rate when using single layer neural networks.
There many recent works which shed some light on the reasoning behind the good performance of multi-layer (or deep) neural networks. The performance is evaluated via computing mean squared predictive error which is also called the statistical risk. For example, [6] show that the statistical risk of a multi-layer neural network depends on the number of layers and the input dimension of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

each layer. The problem of applying deep neural network in high- foth-ddmmnsionll settings is that the nigh-dimensional imput vector nonparametric regression leads to slow convergence rate 8 while the complexity often scales exponentially with the depth number of units per layer 77 16 23
Further, convergence rate for prediction error also related to the regression function and fast rate can be obtained in special classes of regression functions such as additive and/or composition functions [8] [7 [8]. To avoid the curse of dimensionality and achieving faster rates, [25] work under hierarchical composition assumption with sparse neural network. It is shown that under the independence assumption over input vectors, the estimator utilizing a sparse network achieves nearly optimal convergence rates for prediction error. Finally, [26] use neural network classifier for temporally dependent observations which are based on Markov processes. We refer to [4] for an overview of deep learning methods.
Most of theoretical developments related to prediction error consistency of neural networks are under the assumption that either the input variables are independent or they are independent with the error (noise) term, or both. However, these assumptions are restrictive and may not hold in time series models. To bridge this gap, the main goal of this paper is to establish consistency rates for prediction error of deep feed-forward neural networks for temporally dependent observations. To that end, we focus multivariate nonparametric regression model with bounded and composite regression functions and apply sparse neural networks with ReLU activation functions for estimation (see more details in Section ( The modeling framework is similar to (25] while the independence assumption is relaxed. Specifically, we show that given temporally dependent observations, under certain mixing condition, the statistical risk coincides with the result of independent observations setting with additional log" (n) factor where is the sample size (Theorem [)) Moreover, utilizing the Wold decomposition, this result is extended to a general family of stationary time series models in which it is shown that the decay rate of AR( (c)) representation coefficients plays an important role on the consistency rate for prediction error of neural networks (Theorem 2). These results give some insights on the effect of temporal dependence the performance of neural networks by specifically quantifying the prediction error in such settings. Finally, the prediction performance of neural networks in time series settings is investigated empirically via several simulation scenarios and a real data application (Sections 5)
Notation. For two random variables X and Y, X 2 Y implies that X and Y have the same distribution. For a matrix W, || == max:,; Wayl. | lo is the number of nonzero entries of W, and [W]l = Eis [Wyl- For vector v. (o/o [u|] and |o/so are defined by the same way. We write |x| for the smallest integer 2 x. For two sequences (at)121 and (b:)121, we write at S b, if there exists a constant e 2 1 such that a, I cb, for all t. If both a, S by and b, 2 ar, we write a x b. Also, Oe = o(be) implies a. -> as I. - Do. For multi-dimension random variable X, X ~ N(u, Σ) implies that X has multivariate Gaussian distribution with mean μr and covariance matrix Σ. For two functions f.g. we use f a g(x) to denote [g(())). Also, |I oc == If is the sup-norm of f. and (a)_ max(a, 0) for E R.
2 Setup
In this section, a brief presentation of feed-forward neural networks is provided in Section 2.1 followed by discussing the modeling framework under consideration in Section 2.2
2.1 Background about multi-layer neural networks
Multi-layer neural network is composed of three parts: input layer, hidden layers and output layer. We denote the depth of multi- neural network by L. which implies that there are L + 1 layers in total consisting of hidden layers, one input layer and one output layer. We refer to the input layer as the 0-th layer and the output layer as the L-tth layer. The multi-layer neural network can be written as
f(x) Rd - R= W1ow, ..... Wiovo (Wo(x))))
(1)
where W, is the matrix of weights between (i 1) and i-th layers of the network (i 1,.... L.) and αy is modified ReLU activation function in each layer. Specifically, for the shift parameter

---

V= ( (.) E IR,, the activation function ov I -
R is defined as
I1
[1 01)+
Ov
(Ir wr)+
Let p; denote the number of units in the i-th layer (note that Po = and Pz. = 1). For fully connected multi-layer neural network, the total number of parameters is Σi=0 P:pi+1 that is defined as the size of multi-layer neural network [TO]. Similar to [25], the entries of all weight matrices (Wi.e...... LL and shift parameters {vi]e= are assumed to be uniformly bounded. The sparsity level is defined as the number of all non-zero parameters in [W.)=1,.... and {vi .L. We also assume that the value of lfl is bounded by some constant F. In summary, we focus on the collection of s-sparse multi-layer neural networks with bounded parameters which is denoted by F(L, 8, F) and defined as
Σ 2
Willo + Ivslo S 8, 1,11/llo S F},
where F(L,P) := (f form
max 0.1,," L
|W V /vvloo - 13.
This restriction neural networks to the ones with sparse connections and bounded parameters is common on deep learning (see e.g. [25] and references therein) since neural networks are typically trained using certain penalization methods and dropouts.
2.2
Model
The modeling framework considered is similar to [25] while allowing for temporal dependence among observations. Let (E))22 be sequence of independent random variables with [[e, Let [X,hz21 be p-dimensional stationary process with X, == (X,,,,"',, Xt,p). We assume Y, is generated as
Y = fo, + +
(2)
with measurable function fo R of several functions, specifically
We assume that the regression function fo is composition
So = 9q 1 o 0,00
with 9: [a;, b,]a _ (a:+1, bet1]d44 where do = p. dg+1 = 1. Each Ji has d -imeensionl vector output 9; = (g J..d.41) We assume that the multivariate function 9t,5 depends on at most &; variables while is far less than d, i.e. &; < d, As mentioned in (25] for B- smooth function fo. the minimax estimation rate for the prediction error is n (2/((29+d)) Since the dimensionality do can be large in applications, the rate can be slow. To mitigate this issue, the sparse structure avoids the effect of input dimension on the convergence rate and improves the rate.
Let T be region in IR. Let 8 and L be two positive numbers. The Hölder class E(B, L) is defined as the of [8] times differentiable functions f T - R" whose derivative d f(x) satisfies
| fxx)
8 f(y)
S I.
x y 8- [8]
where we used the notation 0 = 0....... 8 with = (a). ,a,, and |a| := |al Further, we
define the ball of 3-Hölder functions with radius K as
(D -
[00) - s(y)
Σ
100 S1xx +
Σ
sup xyED
X - 8-188
< K}.
a:/a|<3 KB
a:ja [=|a|=13]
We assume that all functions J5 for = 0,... ".... , and j = 1, 1,... dd+1 belong to B,-Hölder class
? (Dug, K). From the model (3) we know that Dij [a,, 6.1% Hence, the class of fo focus
3

---

belongs to
g(g,d,,,333k) = {fu =94
o 9o : 9: (gis)) [a,, b;I"
[as+1, bi+1 1 Dij E C₁ ([4, 4,]%, K), for some layl. [b,1 S K}.
(4)
with d (do. d.+1).t:= == (to. .. tq) and [ (So. .
Main result
In this section, we present consistency results for prediction error of deep neural networks applied to model (2) followed by providing time series model examples satisfying the assumptions in Section 3.1 First, need to introduce some notations and state the main assumptions under which the theoretical developments are established. For any estimator In in the class F(L, p, 8, F), we define (similar to
25 A, (Fx fo) to measure the difference between the expected empirical risk of fn and the global minimum over all networks in the class F(L.p. p, 8, F) as
An An(Sn fo)
1 n E fxxx)))²
n Σ (V: f(x))²
= -Efo
inf FEF(L.P.))
The quantity An Frr fo) plays pivotal role in consistency properties of neural networks. The performance of fr is evaluated by the prediction error defined as
R(Fn fo) = 18 [fx(()) - fo(x))
(5)
with X 2 X, and X is independent with (Xe Recall from Section 2.2 the regression function
fo is the class G(g, d, t, B, K). To simplify notations, we define 3r := B, 1 (Be 1), on ==
287
max
The following assumptions are needed to present the first theorem.
Assumption For all: i 1,2..... Ble,] = 0, E[e = o², and there exists some positive constant e such that B[[es[""] & o² m
3, 4.
Assumption 2 (X+) is strictly stationary and exponentially a-mixing process. Recall that the a-mixing coefficient of a stationary process (X,} is define as
a(s) sup( B) - (PA)) -c0 < <t 00, AE
o(
where X₃ consists of the entire past of the process including Xt, and X consists of its entire future. The process (X,) is said to be exponentially c-mixing if there exists some constant >> 0 such that log(a(t)) < - et,
Assumption The error term e, is independent with {X,. $5.SSh tt.
Assumption is known as the Bernstein condition and implies that e, is a sub-exponential variable. This assumption is often used when cannot assume E; is bounded. Assumption 2 to control the dependence among input variables and holds for wide range of time series models, see e.g. auto-regressive models in [12. Assumption controls dependence between input variables and error terms. A more stringent condition is to assume the whole error process fc112 is independent with (X,)s2a- However, this assumption is restrictive since it excludes auto-regressive model which is an important family of time series models. To avoid this, we only assume the current error term is independent of current and past input variables. Further, this assumption ensures that Σ. (X,,) martingale which helps in verifying certain concentration inequalities needed in the proof of main results. All three assumptions are common in non-linear time series analysis, see e.g. [T] Now, we ready to state the main result.

---

Theorem Consider the d-variate nonparametric regression model ②2 for composite regression
function (3) in the class G(g, d, G, Suppose Assumptions 1-3 hold. Let fn be estimator taking values in the network class F(L, (p:)i=0,....L+
satisfying (i) F Z max(K, 1); (ii) E'=o log (4t, V 43,))ogg) I L S noni (iii) non X min _LPsi and (iv)
no,logn. Then, there exist positive constants C, C' depending only on d, such that if (n (f, S Co,,, n, then
R(fn, fo) S C' LnLlog"n,
(6)
and if (n. ConLlog then
R( (o) R(m c' C'nn(Fn,Jo)
Based on Theorem the prediction error defined in 5 is controlled by &h Llog" 6 n. From condition (ii), L is at least of the order of lo Thus, the rate in Theorem becomes on, log' n. This rate for the case of independent observations is of order l, 'n based on Theorem in [25]. Compared to latter, our rate has an extra log' 'n factor, which compensates for additional complexity in verifying the prediction error consistency in the presence of temporal dependence among input variables. Further, note that for fully connected neural network, the number of parameters is Σ P:P++1 Z nndd, L. We can see that this number is greater than the sparsity level 8 which is of order no,,loge based on condition (iv) in Theorem Thus, it can seen that condition (iv) restricts the neural network class to the ones with sparse architecture. In other words, at least E1-1 PiP++1 - 5 units of the neural network is completely inactive.
3.1 Time series model examples
In this section, we introduce some (well-known) examples of time series models that satisfy the assumptions of Theorem [ The first example is to let (G) and (X;) be two independent processes. This independence assumption implies that the input variables X, are exogenous, thus Assumption is automatically satisfied. Further, assume satisfy the moment conditions in Assump- tion I (for example, they have normal distribution) and X, is stationary and geometrically a-mixing process. There are many examples of such processes including certain finite-order auto-regressive processes, see e.g. [5 12] assume
The second example is to consider non-linear auto-regressive models, i.e.
X; = 9==((X1-1, (.,,--) +
(8)
This is special case of model (2) by setting X, (X X1-1) and Y, X,, Assuming Er's is are i.idd random variables with positive density in the real line and the boundedness of the function g. it can be shown that there exists a stationary solution equation (8) while the solution is
exponentially a-mixing as well [2] [2]. In both examples, assumptions of Theorem I are satisfied, thus the results of this Theorem are applicable.
Now, we consider a more general time series model. Recall that by the well-known Wold representa- tion, every purely nondeterministic stationary and vero-mean stochastic process X, can be expressed as X, Ei=o (sét-i where e, is mean-zero white noise. Further, if X, has a non-vanishing spectral density and absolute summable auto-regressive coefficients, i.c. Σ [6.] < oc, has the AR(oc) representation X, -im o + Ge (see e.g. (27).. Motivated by this discussion, we consider a general family of times series models satisfying
X, Σ X +
where é are i.i.d. errors. Independence among et's is a strong assumption compared to only assuming that they are uncorrelated, but this is required for our theoretical analysis. The interesting fact about model (9) is that a linear model. However, since there are infinite covariates this AR(co) representation, training neural networks directly is impossible. The common solution is to
5

---

truncate the covariates and only consider the first few, i.c. approximate model ② by an AR(d) model for some d. This approximation can successfully estimate second order structures of the original model (i.e. spectral density or auto-correlation function) if d is selected carefully and under certain assumptions on the AR coefficients 0; 8 (see e-g. (27)). Therefore, we follow this path and fit a neural network to the d-dimensional input variables (Xi-1--...) X,_a) with proper choice of d while keeping in mind that the true regression function is in fact fo(X.) = Ein &.Xe- To establish the prediction error consistency of neural networks on truncated input variables, we need two additional assumptions.
Assumption There exists a > 0, M > 0 such that Ei=( (1+1) S M oc.
Assumption For some constant K 0. [xa] S K for all t 2
Assumption 4 controls the decay rate of the AR(2)) coefficients in the true model and a can be treated as a decreasing rate of "s. This assumption is needed to compensate for approximating general time series of form (9) with finite lag AR process. Further, it plays an important role restricting the first derivative of fo. which corresponds the 3,-smoothness assumption on 9ij in 4 Note that Assumption 4 is satisfied if the spectral density function is strictly positive and continuous, and the auto-covariance function of X has some bounded property [19]. Moreover, since the model is linear (i.e. the regression function is unbounded), Assumption 5 becomes necessary to make (X,, bounded, a property needed for Theorem [ as well.
Theorem 2 Consider model (9) with fo(X,) Di=10 Let fn be estimator taking values in the network class F(L, (pi)i=o,.. ,+118, 8, F) satisfying (i) F KM; (ii L 2 4; (iii) 8 = Ld,
and (iv) d S minj-1,". I.Pi. Assume that nat., Under the Assumptions 1 there exist positive constants C, C' such that if (n fo) S Cn oF Llog"n n, then R(Fn, fo) S C'n कh Llog n,
(10)
and if An Ar. > On w+I Llog n, then
ã _m((f. & B0) S C' SO)
(1I)
Based on Theorem the best convergence rate of R fo) for model ( is n of log"n. Compared with the convergence rate in Theorem D we can see that the convergence rate in model 9 depends on the decreasing rate of coefficients Ø 's instead of the smoothness of regression function. Also the logarithmic factor changes from log" " since the truncated model is linear, a shallower and sparser neural network can be used in the proof of Theorem 2 (as seen from conditions (ii) and (iii) in the statement of Theorem 2) Also, note that for a simple linear AR(d) model where Φ, = 0 for j > d, Assumption 4 is satisfied for any large a. Thus, in this case, the best convergence rate of prediction error becomes n 'log"n.
log"n. Such subtle decrease is due the fact that
To fit the neural network to truncated model @.. shallow neural networks are sufficient. In fact, L 4 is enough based Theorem This because model has a simple linear structure compared to composition functions funnciionn
Remark
Remark 2 For a general selection of input vector d, the result Theorem 2 becomes Rff, fo) S C +C' C' d'log'n + +AD,(f,Jo)) With the choice of d = natt, we balance the first two terms and establish proper upper bound for R[ Thus, this selection can be regarded as the "optimal selection" of lag d when applying neural networks for estimation in model 9
4 Simulation experiments
In this section, we conduct two simulation settings to illustrate the performance of neural networks applied to temporally dependent data (see also Section C in the supplementary materials for numerical comparisons between feed-forward neural networks and LSTM). In the first simulation in Section 4.1 we aim to compare the convergence rate in dependent observations setting with the rate in independent

---

observations setting. The convergence rate is explained as how fast the mean squared predictive eтror decreases as the sample size grows. In the second simulation (Section 4.2). motivated by time series model examples introduced in Section 3.1 we use neural networks to fit linear and non-linear auto-regressive models and compare the performance with the result of linear regression method (least squares method). All simulations are repeated 200 times.
To train the neural network, we split the data into three parts: training Ti validation 72. and lesting set T3. We set Ti = {I..... n//2}. T2 {n/2,... 33/44, and Ta = (3n/4, 4.... nj. We
penalize parameters of weight matrix of the neural network and use mean square error as our loss. To
be more specific, the loss function is defined as
loss: = - + IWil
(12)
BETI
where À is the sparsity tuning parameter. The value of À does not have significant effect sim- ulation results. In this section, we set À = 0.1. We apply gradient descent method to update parameters of neural network and stop the iteration when the neural network gives the minimum mean square error for T2. The prediction error R(fn: fo) defined in 5 is empirically estimated by R = EIET, (f) - 8,)3/(m/44) We use R to evaluate the performance of the trained neural network.
4.1 Dependent vs. independent observations
In this experiment, we consider a nonlinear additive model Jo(X,) = -2 cos(X Thus, Y, is from the following model
Y 2 Σ (1.)) + T/s, 1,,,,....
(13)
with ni's 's independent standard normal random variables, and X, = (Xe,1," Xe,1) is generated from an AR(4) model. Specifically,
0
0
0
0
X,-
X,-1 + ce. ~~ I.),
0
0
0
0
0
0
100. 400, 1600, 6400. A three layer neural network is selected to fit model (13) while ReLU is used as the activation function. The network requires 4- dimensional input vector and has 20 units in each hidden layer. To make fair comparison with the performance of neural network in independent observations setting, we still use model 13 but with independent observations, i.c. we generate X X X NO, Σ) i.i.d and Σ
where p takes two value, == 0.2, 0.6. The sample size
derives from x 2 XI.
R against various sample sizes for both temporally dependent case and independent case by running 200 replications for each. As can be seen, the prediction performance of neural network improves as sample size increases. Further, we observe that the prediction error for the neural network with temporally dependent data and independent one are similar. This may imply that the additional logarithmic terms appearing in Theorem [ might an artifact of the proof.
Figure iillustrates log
4.2 Auto-regressive examples
In this experiment, we test the performance of the neural network and compare its result with simple linear regression (least squares method) for several linear and non-linear auto-regressive
models. Specifically, we consider the following four models: (1) X, = 0.6X,, + 6; (2) X; = 0.6X,-1 0.4X,-2 + 0.2Xx-3 + exi (3) X, -0.5 + (4) X; X. The enror term is generated as 61, ... 6n N(0, 1) i.i.d. in all models. The sample size 100, 400, 1600, 6400.
7

---

temporally dependent endeperdent
temporally dependent indepeedent
lopanitin Wit-gg
Vogettam of oe
125
8400
120
(a) p
0.2
p 0.6
Figure I: Box plots of logarithm of the mean square error the testing set function of sample size.
We again use four layer neural network with 20 units each hidden layer. This time, we have no prior knowledge the input dimension of the neural network, since lags of these four time series models are unknown. To determine the input dimension (lag time series), we apply linear regression with AIC criterion. The AIC is defined as AIC nlog(SSE)) + 2d, in which d is the input dimension and SSE is the summation of squared errors.
al
Iituall reteork
model
Breer model
nigr of RV 6
logarithm R.5.3
0000
300
100
1603 1000
302
120
1200
5930
0400
Sammlle size
semple size
(a) model (1)
(b) model (2)
suunal network Innar model
meand work Innor model
loganthm of Aif. 6.)
-2-
Iegarithm Go
-12
123
100
400
2043
1000
660
600
200
106
dlu
1800
1800 6M0
sampla 522
sample site
(c) model (3)
(d) model (4)
Figure 2: Box plots of logarithm of the mean square error on the testing set as function of sample size.
Figure 2 displays the logarithm of R(f, fo) as sample size increases. Linear regression method has a fast convergence rate AR(1) and AR(3) models, i.e. models (1) and (2). For non-linear auto- regressive models (models (3) and (4)), the convergence rate of linear regression method becomes slow while neural network outperforms the linear method in these cases. In summary, neural network has the advantage to deal with estimation in both linear and non- -on-linear auto-regressive models.
8

---

Real data analysis
In this section, we apply neural network on macroeconomic data to predict monthly inflation rate. The data consist of 132 monthly macroeconomic variables from January 1960 to December 2011, total of 624 time periods. The inflation rate measured by the percentage changes of the Consumer Price Index (CPI). To be more specific, we are interested in forecasting
[CI++1
1200 x log
(14)
CPI,
we use the AR(4) model as the benchmark model, i.e. T++1 = op Ei=o ap Thus, the overall model can be written as
Similar to 22 28
*++1 = ( Te + 04+1,
(15)
where x, consists of other 131 predictors in the data set along with three of their lags. Therefore, X is 524-dimensional vector. We consider one step ahead forecasts computed in rolling window scheme with 453 observations. Since there are some missing values in the first four years, we start from 1964 to fit model and predict the inflation rate. We the observations between May 1964 and Jan 2002 to fit (13) and predict the monthly inflation rate starting at time Feb 2002.
We apply the feature screening method Sure Independent Screening (SIS) proposed in 13 to reduce the input dimension and select only the top 7% important variables from x, for y = 0, 5, 7.5, 10. The case of 0 implies that we do not select any covariates from X and only use four lags of n. as predictors. Then we use the selected covariates and four lags of n. as predictors or the input vector of our neural network. The neural network is selected to have ReLU as the activation function and 100 units in each hidden layers. Two different depths neural network are selected, one with 3 layers (1 hidden layers) and the other with 6 layers (4 hidden layers). To train the neural network, dropout method for hidden layers and gradient descent to update parameters. In each rolling window, 453 observations are divided into two parts: the first 300 observations is treated as training set while the last 153 observations is treated as validation set. The loss function is again [2) with À = 0.1. We stop training when MSE over the validation set reaches its minimum. The performance is evaluated by MSE of one step ahead forecast. We also show the performance of linear regression (i.e. least
squares method) with the same input as neural network.
The results are summarized in Table while Figure 3 in the supplementary materials plots the predicted values against the truth. As seen from Table I neural network has a better performance compared to linear regression method. In other words, the overall prediction error of neural network with three layers is small compared to the ones for linear regression (except for y 7.5). Specifically, in the case of = 10 (i.e. including 52 covariates from vector xt), the performance neural network remains satisfactory while linear regression method suffers from large input vector dimension. Finally, note that three layers seems to be enough for this data as increasing the number of layers does not reduce the prediction error.
neural network (3 layers)
neural network layers)
linear regression
7% 0%
15.71
17.44
18.05*
7 7 5%
15.65
18.37
17.05
7 7% 7.5%
16.40
19.67
15.42
7% = 10%
16.51
20.70
172.49
Table 1: Prediction error of inflation rate, i.e. the mean square of the forecasting error. Since the result of neural network depends on initialization, we averaged results across 10 replicates. The entry with asterisk corresponds to the benchmark model (i.e. AR(4) model).
Conclusion
Considering nonparametric regression model with the regression function belonging to a specific family of bounded composite functions, we analyzed the performance of deep feed-forward neural
link : hhtps//wwwwssddnnylllvvgoo..oo.a appennides

---

networks with ReLU activation function on estimating such functions. Consistency of prediction error established under mild conditions on the imput data which can include temporal dependence among observations. Interestingly, the consistency rate matches the one for independent data with additional logarithmic factors with respect to sample size. This result applicable wide range of linear and non-linear time series models including finite lag non-linear auto-regressive models. Then, the result is extended to include a general family of stationary time series models utilizing the Wold decomposition while the consistency rate depends on the decay rate of AR(oo) representation coefficients. Relaxing some assumptions the theoretical analysis including the mixing condition and boundedness of observations in the case of general time series models are interesting future directions. Another limitation of the work is considering only the ReLU as the activation function while extending the results to more general family of activation functions is fruitful research direction.
References
[1] Jonas S Almeida. Predictive non-linear modeling of complex data by artificial neural networks. Current opinion in biotechnology, 13(1):72-76, 2002.
[2] HZ An and FC Huang. The geometrical ergodicity of nonlinear autoregressive models. Statistica Sinica, pages 943-9566 956, 1996.
[3] Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-61,, 2017.
[4] Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory. 39(3):930-944) 1993.
[5] Andrew R Barron, Approximation and estimation bounds for artificial neural networks, Machine learning, 14(1):155-133, 1994.
[6] Andrew R Barron and Jason M Klusowski, Approximation and estimation for high-dimensional deep learning networks. arXiv preprint arXiv: 180900009, 2018.
[7] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
[8] Benedikt Bauer and Michael Kohler. On deep learning as a remedy for the curse of dimension- ality in nonparametric regression. The Annals of Statistics, 47(4)22261 2285,
[9] Peter Brockwell and Richard A Davis. Introduction to time series and forecasting. Springer, 2002.
[10] Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pages 192-204. PMLR, 2015.
[11] Richard A Davis and Mikkel Nielsen. Modeling of time series using random forests: Theoret- ical developments. Electronic Journal of Statistics, 14(2):3644-361) 2020.
[12] Paul Doukhan. Mixing: properties and examples, volume 85. Springer Science & Business Media, 2012.
[13] Jianqing Fan and Jinchi Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(5):849-911, 2008.
[14] Jianqing Fan, Cong Ma and Yiqiao Zhong. A selective overview of deep learning. arXiv preprint 1rX 2019.
[15] Jianqing Fan and Qiwei Yao. Nonlinear time series: nonparametric and parametric methods. Springer Science & Business Media, 2008
[16] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pages 297-299, PMLR, 2018.
10

---

[17] Anatoli B Juditsky, Oleg V Lepski. and Alexandre B Tsybakov. Nonparametric estimation of composite functions. The Annals of Statistics, 37(3):1360-1408) 2009.
[18] Michael Kohler and Adam Krzyžak, Adaptive regression estimation with multilayer feedforward neural networks. Nonparametric Statistics, 17(8):891-913, 2005.
[19] Jens-Peter Kreiss, Efstathios Paparoditis, and Dimitris N Politis. On the range of validity of the autoregressive sieve bootstrap. The Annals of Statistics, 3944))2103-2130, 2011.
[20] Paulo JG Lisboa. A review of evidence of health benefit from artificial neural networks in medical intervention. Neural networks, 15(1):11-39, 2002.
[21] Daniel McCaffrey and A Ronald Gallant. Convergence rates for single hidden layer feedfor- ward networks. Neural Networks, 7(1):147-158, 1994.
[22] Marcelo C Medeiros and Eduardo F Mendes. l f--regularization of high-dimensional time-series models with non-gaussian and heteroskedastic error.. Journal of Econometrics, 191(1):255-271, 2016.
[23] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. Conference on Learning Theory, pages 1376-1401. PMLR, 2015.
[24] Marcus D Odom and Ramesh Sharda. A neural network model for bankruptcy prediction. In 1990 IJCNN International Joint Conference on neural networks, pages 163-168. IEEE, 1990.
[25] Johannes Schmidt-Hieber, Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 48(4):1875-1897, 2020.
[26] Lan V Truong. Generalization error bounds on deep learning with markov datasets. arXiv preprint arXiv:2201. 11059, 2021.
[27] Jiang Wang and Dimitris Politis. Consistent autoregressive spectral estimates: Nonlinear time series and large autocovariance matrices. Journal of Time Series Analysis, 42(5-6))580-59 2021.
[28] Wei-Biao Wu and Ying Nian Wu. Performance bounds for parameter estimates of high- dimensional linear models with correlated errors. Electronic Journal of Statistics, 10(1):352- 379, 2016.
[29] G Peter Zhang. Neural networks in business forecasting. IGI global, 2004.


Reviews:
[{"review_id": "u-XoDTTBUnX", "reviewer": "Reviewer_KBnt", "summary": "This paper establishes some non-asymptotic bounds on the prediction error of using feedforward neural network to learn models when the data are timeseries with temporal dependence. These bounds are established under assumptions on uniform bounds on the network parameters, with the ReLU activation function, and assumptions of the model (which is rich enough to contain finite AR process).\n\nPerformance of this structure is then tested on a few experiments and compared with regression.", "questions": "The comparison of temporally dependent and independent data suggests there is not the large difference expected from the theorem. And it is postulated that the additional logarithmic term may be an artifact. This implies the bounds obtained are not tight, how might this result be tightened further?\n\nFor the linear AR examples, wouldn't the linear regression be a rich enough class to learn the true model (or am I misinterpreting what would be the predictor)? If so, why does the approximation error remain high for model 1 and 2 in Figure 2?", "limitations": "The authors briefly discuss in the conclusion the limitation of their work in terms of the assumptions they have placed, i.e. they are studying a rather specific family of problems bounded observations and ReLU activation function for the neural network.\n\nAlso, as mentioned above the bounds do not appear tight. How might this result be tightened further in the restricted case examined in this paper.\n\nAlthough not discussed, this paper assumes that the output dimension is 1, how might the analysis change for greater dimensions? ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is fairly well written and adds to the literature by giving theoretical bounds for the case of temporal dependent data in the simple case of a feedforward neural network. The code for the experiments are included for full reproducibility.\n\nHowever, it appears the bound is not tight, so the experiments did not show as big a difference between temporally dependent and independent cases as expected from the theory.\n\nIt is interesting to be able to obtain optimality bounds. However, as in machine learning a lot of the results are empirical, I would be interested in seeing how the model performs on prediction tasks against something like LSTM/GRU which has structure to capture time-dependence. I would expect that LSTM would perform better in which case, the applicability/significance of the results of this paper would be lower given the alternative, better neural network structure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TCubXUq0dVw", "reviewer": "Reviewer_D8dK", "summary": "This paper provides a theoretical analysis of nonparametric regression for dependent observations using deep neural networks. Non-asymptotic bounds for prediction errors are established. Numerical studies and empirical applications confirms the validity of the theoretical results. ", "questions": "To find the DNN estimator, how to perform optimization under the constraints $\\|f\\|_{\\infty}\\leq F$, $\\max\\|W_i\\|_0+\\|v_i\\|_0\\leq s$ given tuning parameters $F$ and $s$.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weakness: Even though the authors provide upper bounds for the DNN estimator, it seems that the result is an extension from I.I.D observations to dependent data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-btcPVdEFnK", "reviewer": "Reviewer_FsYW", "summary": "In this paper, the authors derive the expected generalization bounds on (sparse) feed-forward neural networks with ReLU activation functions. This model was considered in Johannes Schmidt-Hieber (2020) [reference 25]. However, different from Johannes Schmidt-Hieber's model, the authors assume that the dataset is non-i.i.d. More specifically, the authors assume that the observation sequence is a strictly stationary and exponentially $\\alpha$-mixing process, which includes time series models such as auto-regressive models. The results show that a multiplicative term of order $\\log^3 n$ is added to the generalization bound in [25] (for i.i.d. dataset) which explains for the additional difficulty of prediction caused by the correlation among samples in the dataset. Some numerical results and an application to a macroeconomic dataset are given to support the theoretical results. \n\nThe main idea of this paper (including proofs) is originated from [25]. However, there are some changes in Beinstein inequalities to account for the dependence among samples. More specifically, the Beinstein inequality in [25] for i.i.d. process (dataset) is replaced by two versions in this paper: (1) Beinstein inequality for $\\alpha$-mixing process and (cf. line  68) (2) Beinstein inequality for martingale difference sequence (cf. line 26) thanks to the assumptions (1)-(3) in this paper and some algebraic manipulations. Lemma 3 is the key contribution in this paper.  ", "questions": "In general, all the results in this paper are well proved and look correct. There are some typos and unclear parts. Please see the following list for some typos or concerning problems. \n\n1. Please explain in details why we obtain the expression $(2k-1)var(Y_{n_0})$ in the line 23? It should be $var(Y_{n_0})+2\\sum_{i=1}^{k-1} |cov(Y_{n_0},Y_{n_i})|$. \n2. The sum in Lemma 2 should be $2\\sum_{n\\geq i>0}|cov(Y_{n_0},Y_{n_i})|$ since the fact in line 22 only holds for $i\\leq n$. Please check whether these limit affects your results. \n3.  In Lemma 3, $\\mathcal{N}(\\delta,\\mathcal{F},\\lVert \\cdot\\rVert_{\\infty})$ should be defined ($\\delta$-cover of $\\mathcal{F}$ under $\\|\\cdot\\|_{\\infty}$ norm.)\n4. In the proof of (II) in Lemma 3 proof, please change $\\hat{f}$ to $\\tilde{f}$. \n5. The $\\sigma$-algebra $\\mathcal{F_t}$ in line 89 should be defined as $\\mathcal{F_t}:=\\sigma((\\mathbf{X_i}):i\\leq t)$ instead of $\\mathcal{F_t}:=\\sigma((\\mathbf{X_i},\\epsilon_i):i\\leq t)$. Otherwise, the inequalities such as between line 92 and 93 are hard to explain except when you put an additional asssumption that $(\\epsilon_i: i \\in [n])$ are independent in Assumption 1. Besides, you also need to assume that $\\mathbb{E}[\\epsilon_i]=0$ in Assumption 1 such that some expressions in the proof hold (for example, $\\mathbb{E}[\\epsilon_i f_0(\\mathbf{X_i})]=0$ in line 83). \n", "limitations": "This is a theoretical work. The authors already finished the checklist as well as stated the limitations of theorems and results. Some additional constraints as mentioned above should be added. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work can be considered as a combination of well-known techniques with some new elements in using concentration inequalities as mentioned in the summary part. The strong point is that these results are new and have not appeared in the research literature (to my knowledge). In addition, as mentioned, the authors had some new contributions in making use of some new Beinstein inequalities from the technical viewpoint.  However, most of the arguments resemble [25], and the changes look not novel since they are much dependent on assumptions 1-3, which is the main weak point of this paper. \n\nThe deep neural (worst-case) generalization networks with temporally dependent observation, including the Auto-regressive model, was considered in some existing results (for example, https://arxiv.org/abs/2201.11059). Please cite these related results. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept