Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper establishes some non-asymptotic bounds on the prediction error of using feedforward neural network to learn models when the data are timeseries with temporal dependence. These bounds are established under assumptions on uniform bounds on the network parameters, with the ReLU activation function, and assumptions of the model (which is rich enough to contain finite AR process).

Performance of this structure is then tested on a few experiments and compared with regression.
strengths_and_weaknesses: The paper is fairly well written and adds to the literature by giving theoretical bounds for the case of temporal dependent data in the simple case of a feedforward neural network. The code for the experiments are included for full reproducibility.

However, it appears the bound is not tight, so the experiments did not show as big a difference between temporally dependent and independent cases as expected from the theory.

It is interesting to be able to obtain optimality bounds. However, as in machine learning a lot of the results are empirical, I would be interested in seeing how the model performs on prediction tasks against something like LSTM/GRU which has structure to capture time-dependence. I would expect that LSTM would perform better in which case, the applicability/significance of the results of this paper would be lower given the alternative, better neural network structure.
questions: The comparison of temporally dependent and independent data suggests there is not the large difference expected from the theorem. And it is postulated that the additional logarithmic term may be an artifact. This implies the bounds obtained are not tight, how might this result be tightened further?

For the linear AR examples, wouldn't the linear regression be a rich enough class to learn the true model (or am I misinterpreting what would be the predictor)? If so, why does the approximation error remain high for model 1 and 2 in Figure 2?
limitations: The authors briefly discuss in the conclusion the limitation of their work in terms of the assumptions they have placed, i.e. they are studying a rather specific family of problems bounded observations and ReLU activation function for the neural network.

Also, as mentioned above the bounds do not appear tight. How might this result be tightened further in the restricted case examined in this paper.

Although not discussed, this paper assumes that the output dimension is 1, how might the analysis change for greater dimensions?
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper provides a theoretical analysis of nonparametric regression for dependent observations using deep neural networks. Non-asymptotic bounds for prediction errors are established. Numerical studies and empirical applications confirms the validity of the theoretical results. 
strengths_and_weaknesses: Weakness: Even though the authors provide upper bounds for the DNN estimator, it seems that the result is an extension from I.I.D observations to dependent data.
questions: To find the DNN estimator, how to perform optimization under the constraints $\|f\|_{\infty}\leq F$, $\max\|W_i\|_0+\|v_i\|_0\leq s$ given tuning parameters $F$ and $s$.
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors derive the expected generalization bounds on (sparse) feed-forward neural networks with ReLU activation functions. This model was considered in Johannes Schmidt-Hieber (2020) [reference 25]. However, different from Johannes Schmidt-Hieber's model, the authors assume that the dataset is non-i.i.d. More specifically, the authors assume that the observation sequence is a strictly stationary and exponentially $\alpha$-mixing process, which includes time series models such as auto-regressive models. The results show that a multiplicative term of order $\log^3 n$ is added to the generalization bound in [25] (for i.i.d. dataset) which explains for the additional difficulty of prediction caused by the correlation among samples in the dataset. Some numerical results and an application to a macroeconomic dataset are given to support the theoretical results. 

The main idea of this paper (including proofs) is originated from [25]. However, there are some changes in Beinstein inequalities to account for the dependence among samples. More specifically, the Beinstein inequality in [25] for i.i.d. process (dataset) is replaced by two versions in this paper: (1) Beinstein inequality for $\alpha$-mixing process and (cf. line  68) (2) Beinstein inequality for martingale difference sequence (cf. line 26) thanks to the assumptions (1)-(3) in this paper and some algebraic manipulations. Lemma 3 is the key contribution in this paper.  
strengths_and_weaknesses: This work can be considered as a combination of well-known techniques with some new elements in using concentration inequalities as mentioned in the summary part. The strong point is that these results are new and have not appeared in the research literature (to my knowledge). In addition, as mentioned, the authors had some new contributions in making use of some new Beinstein inequalities from the technical viewpoint.  However, most of the arguments resemble [25], and the changes look not novel since they are much dependent on assumptions 1-3, which is the main weak point of this paper. 

The deep neural (worst-case) generalization networks with temporally dependent observation, including the Auto-regressive model, was considered in some existing results (for example, https://arxiv.org/abs/2201.11059). Please cite these related results. 
questions: In general, all the results in this paper are well proved and look correct. There are some typos and unclear parts. Please see the following list for some typos or concerning problems. 

1. Please explain in details why we obtain the expression $(2k-1)var(Y_{n_0})$ in the line 23? It should be $var(Y_{n_0})+2\sum_{i=1}^{k-1} |cov(Y_{n_0},Y_{n_i})|$. 
2. The sum in Lemma 2 should be $2\sum_{n\geq i>0}|cov(Y_{n_0},Y_{n_i})|$ since the fact in line 22 only holds for $i\leq n$. Please check whether these limit affects your results. 
3.  In Lemma 3, $\mathcal{N}(\delta,\mathcal{F},\lVert \cdot\rVert_{\infty})$ should be defined ($\delta$-cover of $\mathcal{F}$ under $\|\cdot\|_{\infty}$ norm.)
4. In the proof of (II) in Lemma 3 proof, please change $\hat{f}$ to $\tilde{f}$. 
5. The $\sigma$-algebra $\mathcal{F_t}$ in line 89 should be defined as $\mathcal{F_t}:=\sigma((\mathbf{X_i}):i\leq t)$ instead of $\mathcal{F_t}:=\sigma((\mathbf{X_i},\epsilon_i):i\leq t)$. Otherwise, the inequalities such as between line 92 and 93 are hard to explain except when you put an additional asssumption that $(\epsilon_i: i \in [n])$ are independent in Assumption 1. Besides, you also need to assume that $\mathbb{E}[\epsilon_i]=0$ in Assumption 1 such that some expressions in the proof hold (for example, $\mathbb{E}[\epsilon_i f_0(\mathbf{X_i})]=0$ in line 83). 

limitations: This is a theoretical work. The authors already finished the checklist as well as stated the limitations of theorems and results. Some additional constraints as mentioned above should be added.
review_title: null

Meta-review: metareview: The reviewers consensus is that this manuscript is a near the borderline between acceptance and rejection due in large part to the results being somewhat natural extensions of existing results from [25] with the appropriate extension of the bounds as pointed out by reviewer FsYW.  While this limits the overall excitement about the results, it is an important extension that would be of interest to many readers as time-series data is far more realistic in many applications than is i.i.d. data.  There is a useful discussion between reviewer KBnt and the author about some numerical experiments.  The author is encouraged to consider if is possible to expand upon the numerical experiments in the manuscript to highlight as much as possible the difference achieve between the dependent entries in the time-series and those in the i.i.d. setting.
confidence: Less certain
Decision: Accept