Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose, theoretically analyze and empirically evaluate a sparse additive huber additive model (SpHAM), which is an additive model where each component function in the model is a kernel function that is fitted by minimizing a huber huber loss function with a functional sparsity inducing penalty term.  Statistical properties when estimating this model on stationary data with non gaussian noise and also on non-stationary data are developed. Finally, the model and fitting routines are demonstrated empirically on 2 synthetic and 1 real world time series.    
main_review: Overall, I thought this was a solid paper.  The SpHAM model is not that novel, but the analysis of the model appeared novel, correct and of interest to those working with real world data (e.g. financial data) that is non i.i.d., non-stationary and heavy tailed for which "practical" theoretical findings are not that common.

Pros:
-I think this was a well written paper. The development was clean and logical and the authors nicely illustrated their contributions.
-I think Theorem 1/Corollary 1 is a tidy result.  Expressing the rate of convergence via a moment condition, c, and the huber threshold param provides some nice intuition for situations where the method will/will not be effective.
-The experiments nicely demonstrated the utility of the SpHAM approach for heavy tailed, stationary data and the adaptive SpHAM for non-stationary data.

Cons:
-How restrictive is assumption 3 in practice? Since we don't know the true f*, it that condition testable/verifiable?
-How do we know that a q being a uniform distribution over the most recent observations satisfies Assumption 4?  Seems like if l is to large its not likely to hold, but if l is too small then we can't capture any of the non-stationarity.
-Can you provide a bit more color on how Step A/eq (9) is solved? 
-Can you comment on the scalability/space, time complexity of algorithm 1?  Could the method be scaled to longer time series and do you envision an incremental/online version of the procedure? 
-How does performance change if t/400 is changed in experiment B?  i.e. as the drift to noise scale ratio is changed.

summary_of_the_review: Overall, I think this was a good paper with some nice theoretical developments and an easily understandable and interpretable model.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this manuscript, the authors proposed an adaptive sparse Huber additive model and provide an algorithm to handle non-parametric estimation of the underlying unknown function. Theoretical guarantees are provided under certain conditions and some empirical comparison with competing methods are provided.
main_review: - Assumption 1 on boundedness of $|Y^t|$ is very strong and may not hold even for simple distributions such as Gaussian. The authors are encouraged to relax this assumption, otherwise the results of the manuscript will be limited and not general enough.

- Authors need to either verify or give references to why certain kernels would satisfy Assumption 2.

- The authors claim in the abstract that `"... we propose an adaptive sparse Huber additive model for robust
forecasting and inference (e.g., Granger causal discovery) …". However, (a) there is no results about inference in the manuscript which would quantify the uncertainty of estimation; (b) The Granger causal network is only mentioned in the appendix for a real data set while it is not discussed how this network is estimated using the proposed algorithm and whether the estimated network makes sense scientifically. The authors need to add results related to inference, or at least modify the abstract accordingly.

- The upper bound in Theorem 2 is unusual. It is appropriate for the authors to interpret the results of this theorem. Also, are the results sharp?


- There are several hyper-parameters in the proposed algorithm, and as mentioned at top of page 9, the authors did not attempt to find data-driven methods to select such parameters. In is not clear whether the proposed methodology would still work well under different simulation settings using the same fixed hyper-parameters. A better treatment of hyper-parameter selection is needed for the method to become practical.

- Experiments in Section 4.1 are repeated only 10 times which is quite low. These studies need to be replicated more, say 100 times.

- The code for reproducing empirical results is not provided.




- The sentence above Definition 2 in page 7 needs rewriting.

- Statement of Theorem 3 needs rewriting: "Let Assumption 4 the conditions in Theorem 2 be true".


- Typo at the beginning of section 4.2 on page 9: "We test our algorithm on nonlinear Nonlinear-VAR dataset".


summary_of_the_review: - A more comprehensive discussion on Assumptions are needed.
- More interpretations about the theoretical results need to be provided.
- Some claims in the abstract need to be modified.
- A better treatment of hyper-parameter selection is required.
- Some typos need to be fixed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This ms tackles a timer series problem where each observation is the sum of p hidden functions and a noise (termed as innovation in ms). Traditionally the noise is Gaussian, but this ms makes the noise into heavy tailed distribution such as t distribution. The L2 loss function of the original problem is then replaced by so called Huber loss, which introduces a cutoff to the deviations and is less affected by outliers than L2 loss. Based on this new loss function, this ms sets up the sparse Huber additive model and derive the bounds of the learned hidden functions, under stationary processes. In case of non-stationary processes, the ms introduces different weights to each time points, which can be used to calculate a discrepancy that reflects the non-stationarity. With different weights to different time points, the ms come up with new error bounds. Inference algorithms are provided in the end.
main_review: The novelty of this ms is mainly theoretical. It introduces a non-Gaussian noise to the additive model can come up with the error bounds. The experiments are not comprehensive. It only compares with two other auto regressive methods. I will be interested to see its performance in non-autoregressive setting, i.e. simply sum several non-linear functions and Gaussian / non-Gaussian noise and see if the ground truth could be recovered. Also, the real data experiment (F.1) result is only a figure without any interpretation. It will be nice to see if this method provides new insights compared with previous methods.
summary_of_the_review: My general feeling is that there is theoretical contribution but the experiments could be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a robust Huber additive model for non-stationary time series prediction. They combine the idea of robust Huber regression (against non-Gaussian innovations) with a linear additive model formulated with the aid of the representer theorem in RKHS; sparsity is imposed on the weights, and the overall weights optimization problem can be solved efficiently through the classic FISTA algorithm due to the special problem structure. The major contributions lie in a robust (due to the Huber loss function) and interpretable (due to the use of linear additive model) time series prediction model as well as some theoretical supports on the performance. 
main_review: Strengths: 
1. The proposed time series model is both robust and interpretable. 
2. Some novel performance lower bounds for the proposed model. 

Weaknesses:
1. Although the simple linear additive model is well interpretable, its modeling capacity is quite weak on the other hand. As a remedy not destroying your optimization framework, you may consider adding some interacting terms, like Y2*Y3. 
2. The proposed robust time series prediction model with the three theorems constitutes the main contribution of the paper. But to me, the meanings of the lower bound (namely each part of the right-hand side) are hard to interpret. Are they tight bounds in general? Besides, are $\hat{f}$ solutions of the corresponding optimization problems? 
3. It is well known that robust Huber criterion function can handle non-Gaussian innovations/errors, but it remains unclear in the paper how is it used in the proposed model to handle non-i.i.d. data, which is regarded as one major contribution if I were right.
4. Could the authors comment on the impact of the kernel selection? I see in the experiments only a Gaussian kernel (which is a stationary kernel) is demonstrated. Don't you need a non-stationary kernel instead?  
5. Experimental validation is somewhat insufficient. The selected examples (A&B) well match the linear additive model assumption. It would be interesting to see some model mismatching examples and the performance thereof. Besides, in both examples, the innovations/errors are i.i.d., right? 
6. As benchmarks for comparison, only a simple AR model is adopted. The authors should compare the proposed model with more state-of-the-art models, such as LSTM, Gaussian process (GP) regression, and even some latest attention-based models. Actually, your model is sort of similar to a GP model when replacing Huber loss back to L2 loss. 
7. Code is not made publicly available. 
8. Many typos and grammatical errors. 
   
summary_of_the_review: To conclude, I find the paper interesting and easy to follow, the technical contributions are fine but obviously need further refinement. The literature review is insufficient, and some SOTA methods, such as LSTM, GPR, etc., are missing. The section of experimental results can be improved.
review_title: null

Meta-review: This paper proposes an adaptive sparse Huber additive model for for forecasting non-stationary time series. The prior work has considered similar models for Gaussian innovations which is overly restrictive for a variety of applications such as finance. The results are supported both by theory and experiments. The results are novel and are of interest to ICLR and machine learning communities in general.
Decision: Accept (Poster)