Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an autoregressive framework, Meta-GLAR, comprising of a global RNN model and a local linear model (learned using a closed form solution) for the task of meta-forecasting of time series (TS). The linear model is the task-specific model varies for each TS in the dataset, whereas the RNN model is the meta model that is shared across al time series. The authors perform comparisons with local models as well as NBEATS and DeepAR, and show that Meta-GLAR is competitive with some of them on particular metrics. They also perform an ablation study to show that each component in their model is important to the overall zero-shot transfer in TS forecasting.
main_review: ## Strengths

* The authors describe the problem setting for meta forecasting as well as the Meta-GLAR approach quite clearly. 

* The experiments conducted on Meta-GLAR are quite comprehensive (large hyper-parameter sweeps, comparisons with models directly trained on target TS). Furthemore, the experimental setup is described in sufficient detail along with supplementary code to ensure reproducibility.

* The ablation study conducted to determine the importance of each component in the Meta-GLAR model is comprehensive and shows that each component (RNN, Meta closed-form adaptation, and iterated forecasts) in the model adds to the performance although it does not clarify why RNN with iterated forecasts leads to worse performance than an RNN baseline. 

## Weaknesses

* The novelty of the approach itself is quite limited. The local model (*differentiable local adaptation layer*) is essentially a linear layer applied on top of the RNN in the context window. The Meta-GLAR approach could be (approximately) described as training an RNN on the full time series without training the linear layer in the forecast horizon, and then using the linear weights from the context window only during transfer to new tasks. 

* In Table 1, why did the authors chose to report three different metrics across the four datasets (ND on ELECTR and TRAFF, sMAPE on M3, MAPE on M4) in their comparisons? If all three metrics were calculated across all baselines and the authors' proposed models then these results need to be included (at least in the appendix). Since the authors have not reported their reasoning behind chosing different metrics for different datasets, it raises the suspicion that these comparisons might be cherry picked.

* In their comparison with fine-tuning, did the authors try fine-tuning both the final layer and RNN? That seems like a fairer comparison versus keeping the RNN model fixed and only training the final layer. The authors used closed form solutions for the final layer which might get more expensive to compute if the RNN model wasn't fixed. A comparison with a global linear layer fine-tuned alongside the RNN model would provide a better picture of the importance of their local adaptive layer.

* One broader question that this research raises is about the scope and relevance of the meta learning framework to TS forecasting. The framework lends itself very well to domains like computer vision due to relationship between visual tasks [1]. However, it is not immediately obvious why it would be relevant to time series forecasting where the underlying domains and granularities can vary widely. I'd like to see the authors comment on the scope and limits of this work in their claims:
> Meta-GLAR is trained on a source dataset and can then generate accurate forecasts for new time series that may **differ significantly** from the source dataset


## Errata/Clarifications

*  Did the authors mean to say generalization in the Into (para 2)? 
> Global-local approaches ..., exhibit a greater level of **specialization** as they learn parameters that are shared by all TS in the training set.  
* The reference regarding the use of closed form solvers in spatial regression seems incorrect in the Related Work (end of para 2)
> ...and spatial regression (Iwata and Kumagai, 2020), while we are not aware of any application in forecasting.

## References

[1] Taskonomy: Disentangling Task Transfer Learning. Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, Silvio Savarese; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018,
summary_of_the_review: The problem of zero-shot transfer in time series forecasting is important but it is also necessary to define its scope. The method presented by the authors is novel but incremental. There are concerns regarding the reporting of results (see Weaknesses) which cast doubt on the model performance, which is already lower than NBEATS on some tasks. Overall the paper presents an interesting approach to performing meta-forecasting but it needs more detailed analysis in terms of its applicability as well as completeness in terms of reporting its results. I would be willing to consider an improvement in the score if these issues are addressed.  

**Post-rebuttal:** My concerns regarding scope and cherry-picking have been sufficiently addressed in the revised submission and the authors' comments. However, I am still concerned about the novelty of the work as well as the extent of ablations. I have increased my overall score to 5 (marginally below the acceptance threshold) and provided additional feedback to the authors in comments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a new meta-learning framework to tackle the zero-shot learning problem for time series data – through the combination of:
1.	A standard autoregressive architecture encoding historical information (e.g. DeepAR).
2.	An adaptive linear output layer whose weights are calibrated in closed-form using encoder history.

This allows the model to be trained end-to-end on a source task, while automatically performing domain adaptation when applied to a new target task.

main_review: Strengths
---
1.	The automatic output layer tuning they propose is an interesting idea. Although not explored in this paper, this could potentially be adopted to cope with concept drifts or structural breaks which are prevalent in non-stationary time series datasets.

Weaknesses
---
However, I do have some concerns with the paper in its current form, which has a couple of areas that need to be addressed. 
1.	The authors use a couple of technical terms in non-standard ways, making the introduction slightly confusing:
  * Out-of-sample -- in the time series forecasting domain, out-of-sample data typically refers to data forwards in time that is unavailable during training. Generalisation forwards in time is the primary purpose of training forecasting models, whether statistical or machine learning approaches. The authors use this to refer to generalisation performance on a different time series entirely, however – which depends critically on similarities between source and target tasks. For instance, it is not unreasonable for a forecasting model that predicts patient survival times to fail to predict retail sales.
  * Multi-task learning – global forecasting models are typically not referred to as multi-task learning, as the goal is to achieve good forecasting performance across all entities and time steps, as measured by a single evaluation metric. See [1] for more on multi-task learning.
2.	Suitability of benchmarks – while local (statistical) models can be useful for **data-limited** settings, I am not sure these are suitable comparisons for zero-shot models which cannot access the target dataset before prediction time. Assuming some data is available for training, there are few global network options available for small data regimes (e.g. [2]), and comparisons to other standard transfer learning/domain adaptation techniques should be evaluated. Similarly, any differences vs NBEATS could have arisen from the choice of encoder -- a NBEATS style encoder should be used within the proposed meta-learning framework for a fair comparison.
3.	How does the framework handle transfer learning between datasets with a different number of covariates? The utility of the approach would very limited if only univariate target inputs can be used, as a lot of useful information would also need to be discarded.
4.	Complexity of matrix inversion -- assuming that h(t) has dimensions d, inverting the dxd matrix typically is O(d^3). As it is possible for d > t (see optimal hidden state sizes in DeepAR and Temporal Fusion Transformer vs encoder length), this can be greater than the sequential matrix multiplications for the LSTM (O(d^2 t) I believe). As such, computational complexity may indeed be a concern in some scenarios for this layer.

References
1.	Crawshaw. Multi-task learning with deep neural networks. arXiv:2009.09796.
2.	Rangapuram et al. Deep State Space Models for Time Series Forecasting. NeurIPS 2018.

summary_of_the_review: While the raw idea has promise, many corrections/clarifications need to be made before it is ready for publication -- specifically related to the terminology used, whether inputs between datasets can differ, and the complexity claims of their proposed layer. In addition, more suitable benchmarks are required to fully evaluate performance claims.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a new forecasting method for jointly learning from a large pool of related time-series. The method called Meta Global-Local Auto-Regression (Meta-GLAR) learns the mapping from representations produced by an RNN to one-step ahead forecasts where the parameters are learned across multiple time-series through backpropagation. This work studies the zero-shot transfer learning problem and proposes a method for it. The method is somewhat incremental and evaluation has some issues, see below for details.

main_review: This work develops a new method for out-of-sample time-series forecasting (that is, transfer setting). Instead of calling the method a meta-forecasting method, which seems problematic. It would be better to use the more standard and frequently used term of zero-shot transfer learning, which fits much better with the method proposed.

Meta-GLAR is shown to have similar runtime and memory overhead compared to a global one-step ahead RNN with similar structure. However, it is unclear how it compares with the other baselines shown in Table 1? In fact, the only result showing training time I noticed, is in Figure 2 (left). However, this only compares Meta-GLAR to RNN. What about the other methods, especially NBEATS? I was hoping to see a table similar to Table 1, but with training time for all the baseline methods across all the datasets. That said, I would remove the last contribution mentioned, since it is only in terms of RNN, but not the actual state-of-the-art methods.  

Deep Factors and the more recent Graph Deep Factors were both shown to outperform DeepAR, NBEATS, and the other baselines used. How do these methods compare to Meta-GLAR? It seems they would be trivial to use in this setting as well. Nevertheless, they should also be included as baselines and discussed appropriately since both utilize global and local components and can be trained with a large corpus of time-series. 

The caption of Table 3 and Table 4 have a typo, “Table 3: sMAPE on M3. All All models”. The reproducibility details mentioned in the Appendix are good and easy to follow. 

summary_of_the_review: The method proposed is interesting (but somewhat incremental) and the problem is important. However, there are several issues that need to be addressed. The method typically doesn’t perform well compared to the state-of-the-art as shown in Table 1. It is also strange why different metrics are used for different datasets, for instance, of the four datasets used, there are 3 different metrics used in the evaluation ND for Electr/Traff, SMAPE for M3, and MAPE for Tourism. It would have been better to report all 3 metrics for each dataset (or just show one of the metrics, and the others in the appendix). It currently seems to be cherry picked a bit. Furthermore, the training time for each baseline and dataset is missing, please include a table like Table 1 but with the runtime for each baseline and dataset. Please see above for other points. Overall, the contribution and novelty of this work is limited.
review_title: null

Meta-review: This paper proposes an autoregressive framework that combines RNN and local linear component for the problem of meta-forecasting of time series. The linear model can domain-adapt to different time series while the RNN component is shared across series. Reviewers thought the problem was important, the paper was generally clear and the experiments extensive. However they found the significance to be limited and all took issue with some of the ways that the comparisons were done. FZbR also raised the issue of complexity of the matrix inversion component of the method.  I believe this paper does fall on the rejection side of the fence due to the issues of complexity, significance and evaluations. With some development, the paper could certainly be ready for acceptance.
Decision: Reject