Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work developed light-weighted ts modeling, time-index meta learning scheme for time series forecasting.  
strength_and_weaknesses: ## Strength
- enhanced deep-time class of method via INR and 
- extensive multivariate real data experiments and ablation study


## Weakness
- clearly define what class of non-stationrity thei work target. It just generally mention covariate shift, conditional distribution shift, concept drift without formal definition. The figure 1 is not sufficient, except visuallizing locally-stationary (as high-level concept)
- need to provide more direct evidence on 'whereas existing neural forecasting methods, which are historical-value models, are unable to take full advantage of this formulation,' in the introduction
- need to clarify 'include having a stronger smoothness prior' in the abstract
- In figure 2, need to specify the method of naive deep-time method. At the same time, usually, historical-value based model can capture such a seonsonalty well.
- In section 2, what is the use of forecaster $h$?. the $\mathcal{L}$ is poorly defined, seems to $R^m \rightarrow R^m$ in eq (1) , which is different in problem formultation. 
- What is $\Theta, \Phi$? how do the author decide? is it depending on model types? any rule of thumb? it is too general and vague with too much degree of freedom and little intuition.
- meta learning framework seesm to be nothing but a fitting some part of parameters like hyperaparemters in additional validation datasets. why this is specifically robust to non-stationary data?
- why only applied for long-term and multivariate forecaster? what about medium-term forecasting and univariate (global) forecasters  like DeepAR, MQ-RNN, ConvTrans, etc.
- lack of literature study on distribution shift, non-stationarity context. For example AdaRNN https://arxiv.org/abs/2108.04443
clarity,_quality,_novelty_and_reproducibility: Lack of clarity on motivation on why it is strong under non-strationtity and historical-based models are not sufficient. Poor mathematical description on problem formulation. Marginal novelty in chosing specific INR and hyperparameter for meta learning. only limited to multivarate modeling, not global modeling modeling even with its modeling capacity, where the baseline are known to be very poor.
summary_of_the_review: Need to better highlight the motivation on why Deep-time class is considered over historical based classes, and why meta learning helps with theoretical analysis or strong intuition. The writing can be enhanced significally especially in problem formulation. More solid experiments beyond multivariate modeling can be done to demonstrate the need of this method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a meta-learning formulation for learning to adapt a deep time-index model to the look-back window. INR was the choice of time-index models, and meta-learning was formulated by using samples from look-back window as context set and forecasting horizon as query set. The goal of the meta-learning was mainly motivated for dealing with non-stationary time series. Experiments were conducted on synthetic as well as real datasets demonstrating performance improvement of forecasting over selected baselines.

strength_and_weaknesses: Strengths:

1. Addressing non-stationary time-series by learning to adapt the time-series model is an important and interesting direction of research. 
2. The use of INR for the deep time-index model is interesting. 
3. The results demonstrated improvement over a good number of baselines used.


Weakness:

1. The presented work is heavily related to switching state-space models [1] and sequential neural processes [2]. In particular, it seems that it can be formulated as a special case of the SNP for learning y = f(t) where Bayesian meta-learning is learned to adapt f to observed frames from the lookback window for prediction in the forecasting horizon. Neither of these two works were discussed in the paper. Relations with these two works, and comparisons in terms of performance, should be provided. 

2. By formulating the meta-learning with optimization-based approaches, the model needs to be optimized to the query set before being used for forecasting. To mitigate this, the paper opted to restrict the optimization to only the last layer of the INR. The effect of this restriction should be demonstrated, empirically at least. In an ideal setting (where resource or time is not a constraint) where the full INR can be optimized to the lookback window each time, how does the performance look like, and how does it compare with the assumption of restricting the optimization to the last layer? Note that, with SNP, this was avoided by formulating the meta-learning with feedforward based approaches. This again stresses the need to compare with SNP.

3. Details on meta-training is needed. It is not clear how the current tasks are defined -- by treating each pair of lookback window and forecasting horizon as a task, it means that the meta-training is looking at a large number of tasks and each task has only one set of context and query sets? Is the task boundary known assumed to be known ahead of time? If it is, it seems to be an unrealistic assumption. It it is not, this seems to be an unconventional setting of meta-learning -- Does the training follow the typical episodic training then? Further, how does the method apply if either the look-back window or the forecasting horizon falls within the transition of tasks boundaries? 

4. Experimental details are missing. It is not clear how the baseline models are trained. Since most of them do not use any meta-learning formulation, are they trained on the meta-training set? In that case, the comparison may not be fair as the presented model -- at meta-test time, actually is optimized to the look-back window (while the baseline models are simply applied to the look-back window without optimization). A fair comparison would be to fine-tune the baseline models (after training on the meta-training set) to the same samples (look-back window) used at meta-test time. If the argument is that these models cannot be fine-tuned this way (since they are history-value based models), at least in ablation study, the "-RR" version need to be fine-tuned at meta-test time.

5. Since the look-back window represents context set in meta-learning, its size may have an important effect in "optimizing" the base model. The value of L in synthetic experiments seems to be large, and was not specified in real-data. Please add such details, and provide experimental evidence about the effect of the size of the context set L on meta-learning.

6. While the paper was heavily motivated for better learning non-stationary time series, the experimental evaluation is limited in demonstrating how or whether the proposed solutions achieved the stated goal. In synthetic experiments, there lack details on how many tasks were used to meta-train the model and how many tasks were used in meta-testing. Since tasks/segments are generated with random sampling of the parameters, it'd be good to get a sense of the distance between the meta-train and meta-testing tasks. Finally, it was stated that "A total of 400 time steps are sampled, with a lookback window length of 200 and forecast horizon of 200." In Appendix D, it was then stated that "each function/task consists of 400 evenly spaced points". So assuming each context-target set pairs are 400 time points, it was not clear how many such samples of length 400 were used. It was also not clear how does the forecasting work in such segment of 400. Does the model take 200 context samples and forecast for 200 context samples, and then it moves to the next window of 400? i.e., the task boundary is assumed to be known?

In real data experiments, it is understandable that the truth about the "non-stationary" nature of the data is not always available, but to the extent it's possible, it'd be desirable to say some level of analyses that link the model performance with the "non-stationary" nature of the underlying time series. 



7. The methodology is presented in a general fashion for forecasting m-dimensional observations. It'd be good to understand to what extend the value of m could be, i.e., what types of observations can be modeled by the presented method. Are we looking at multivariate data with relatively lower number of dimensions, or are we looking at image series?






[1] Variational Learning for Switching State-Space Models, Zoubin Ghahramani and Geoffrey E. Hinton, Neural Computation 12, 831–864 (2000)
[2] Sequential Neural Processes, Gautam Singh, Jaesik Yoon, Youngsung Son, Sungjin Ahn, NeurIPS 2019


clarity,_quality,_novelty_and_reproducibility: The writing was overall clear and relatively easy to follow, although the methodology and experiment descriptions lack many critical details (as detailed above) that make it difficult to assess or reproduce the method. I have questions on the novelty of the work due to its unclear and not-discussed relation with switching systems and more importantly, SNP. 

Code is submitted and will be released for reproducibility.
summary_of_the_review: This paper tackles an important problem of time-series forecasting (i..e, non-stationary series) with an interesting solution (meta-learning of time-index models). The novelty however is unclear with respect to some existing works especially sequential neural processes. The choice or benefit of meta-learning method and the simplification of optimizing only the last layer of the INR, in comparison to alternative meta-learning method such as feed-forward model based method used in SNP that can bypass such simplifications, is not clear. The writing lacks many critical details on methodology and experiments, such as how are tasks handled during meta-training, whether the task boundary needs to be known, and how to address lookback window or forecasting horizon that includes task boundaries. It is also not clear how large a L is needed as the context set, and how baseline models utilized these context data at meta-test time. Overall it is an interesting idea, but can be improved in clarification of novelty and many methodological/experimental details.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new deep learning model for forecasting under non-stationary conditions – utilising a network that builds global representations of random Fourier features and projects it using meta-learning framework that recalibrates the output layer with each window.
strength_and_weaknesses: Strengths
---

As neural networks notoriously underperform in the presence of covariate shifts – which are common in non-stationary time series datasets – the creation of methods that allow neural networks to adapt to changing regimes is extremely important. The use of meta-learning here is an interesting approach, and the out-performance of other state-of-the-art transformers makes it a promising method.

Weaknesses
---

However, I do have several concerns regarding experimental evaluation, in particular:

1.	Lack of hyperparameter optimisation – while the paper does show out-performance across benchmarks, we note that all the hyperparameters of DeepTime and benchmarks have been pre-specified based on “suggested” settings. Given that optimal hyperparameters can vary largely across time-series datasets – for example smaller models in data-limited regimes vs larger models where data is abundant – the lack of hyperparameter optimisation does beg the question of whether the performance differences here are due to improperly selected hyperparams and not model improvements.

2.	Are improvements due to differences in features used? – While the use of random Fourier features helps to make the time index range bound (which would otherwise introduce its own train-test mismatch), the final output of the INR would essentially be a set of non-linear seasonal features which repeat over time. Out-performance hence could be due to these seasonal relationships being more prevalent in the datasets used. As such, it would be good to see 1) how properly tuned transformer/tabular models perform when given the same random Fourier features + observations, and 2) whether a rolling linear regression on random Fourier features is sufficient for performance.

3.	Does the meta learner handle non-stationarity? – while promising in theory, it is unclear whether there are in fact concept drifts in the time series datasets used for benchmarking. Tests on simulated data that incorporates regime changes over time could be useful here, with performance changes/decays across regimes documented for all models.

clarity,_quality,_novelty_and_reproducibility: On the whole, the paper was clearly presented and the motivations were very intuitive as well. A few questions remain, however, which would be very helpful with reproducibility.

Questions
---
1.	Is any feature engineering applied to inputs for benchmarks?
2.	How are lookback windows determined?
3.	How much data was used for training in each case?

summary_of_the_review: I do think the idea and intuition behind the model is compelling, but would require improvements in experiments and benchmarks to be more convincing for me.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes DeepTime, a deep time-index model trained via meta-learning, with the motivations for stronger smoothness prior, avoiding the problem of covariate shift, and having better sample efficiency. Experimentation on various time series benchmarks are shown.
strength_and_weaknesses: Strengths

- Forecasting as meta learning framework is an interesting and canonical idea. 

- Extensive ablation analyses

Weaknesses

- There is not sufficient explanation on what non-stationarity is. 

- The method is proposed for non-stationary time-series, but the experiments are not specifically obtained for non-stationary time-series data. The synthetic datasets are simple functions and most of the real-world benchmarks are not from applications where non-stationary dynamics are prominent, maybe except Exchange, on which the results are weaker. 

- It is unclear why Fourier features would be a good idea for non-stationary time series, as opposed to time-varying frequency representations. 

- There is no analysis on how much different datasets contain covariate or distribution shifts. 

- Some relevant baselines are skipped:
https://arxiv.org/abs/2205.14415
https://arxiv.org/abs/2205.13504
https://arxiv.org/abs/1912.09363

- For computational efficiency, the impact of parallelization is ignored.

- The impact of lookback length is not sufficiently analyzed. 

- For a non-stationary dataset, training and validation distributions would have differences. How would it affect the hparam selection? This aspect is ignored. 
clarity,_quality,_novelty_and_reproducibility: The paper clarity is okay. The methods section could be improved. I suggest including the pseudocode for training and inference. 

E.g. for INR, "We make use a of them as they are a natural fit for time-index models," - such motivations are not well supported. 

The quality is mediocre. Some key aspects have issues as mentioned. 

The idea is novel in time-series. 

Reproducibility is concerning, as not all details are included. 
summary_of_the_review: Because of the issues above, mainly lack of convincing empirical validation appropriate for non-stationarity, the paper cannot be accepted in its current form in my opinion.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a deep time-index meta-learning  for non-stationary time-series data. It argues that meta-learning helps to alleviate co-variate shift, while it addresses conditional distribution shifts. 

All reviewers indicate:
(i) a lack of clear definition of covariate and conditional distribution shifts in the paper. Time-series data has several drifts (covariate, prior probability, conditional probability, concept etc). Esp for the conditional distribution drifts, it is unclear if the paper is addressing the prior probability drift or the conditional probability drift. 
(ii) a lack of clear definition of tasks, 
(iii) a lack of empirical evidence to substantiate the claims made, while the paper is heavily motivated.

Thus, all reviewers have voted for a reject and this AC encourages the authors to take the reviewer's feedback and improve the work.




justification_for_why_not_higher_score: All reviewers indicate:
(i) a lack of clear definition of covariate and conditional distribution shifts in the paper. Time-series data has several drifts (covariate, prior probability, conditional probability, concept etc). Esp for the conditional distribution drifts, it is unclear if the paper is addressing the prior probability drift or the conditional probability drift. 
(ii) a lack of clear definition of tasks, 
(iii) a lack of empirical evidence to substantiate the claims made, while the paper is heavily motivated.

Thus, all reviewers have voted for a reject and this AC encourages the authors to take the reviewer's feedback and improve the work.
Decision: Reject