Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies an efficient way to use the transformer-based model in time-series forecasting tasks. The proposed model, PatchTST, first folds the sequence into several patches which significantly reduces the total sequence length and then splits the multi-channel forecasting signal into independent tasks. In numerical tests on several real datasets, the proposed model reaches the SOTA performance. The authors also test their model on self-supervised learning and verify a promising sign that the transformer-based models could be fit on time-series pretraining tasks.
strength_and_weaknesses: Strength:
1. Good numerical performance forecasting, reaching the SOTA results on several real datasets.

Weakness:
1. Based on the current presentation, the proposed model is analogous to the standard vision transformer (VIT). The technical novelty is limited.
2. The pretraining datasets are relatively small compared to datasets in CV/NLP fields and the domains of the datasets merely share the mutual underlying knowledge. Combining with marignal improvements reported in Table 4 and Table 5, I tends to believe that the pretraining stage isn't meaningful to the considered datasets.


clarity,_quality,_novelty_and_reproducibility: 
Clarity: The paper is well written and I enjoy reading it.

Quality: The overall quality is good. After the introduction section, the authors first discuss the model settings and then switch to several experiments to illustrate the efficiency of the proposed model.

Novelty: Despite the strong numerical performance, the technical novelty is kind of limited as the proposed model structure is mainly borrowed from the standard VIT.

Reproducibility: The paper doesn't include the codes/git. The model parameters are reported but some other parameters, such as learning rate, and optimizer are not reported. At the current stage, I haven't checked the reproducibility.
summary_of_the_review: This paper proposes an efficient transformer-based model in time-series forecasting tasks and strong numerical performance is presented. My major concern is the technical novelty as the proposed model is mainly a direct application of standard VIT.


Minor issues:

1. The "64 words" in the title is a little bit misleading. In the famous VIT paper "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", the patch size is 16 x 16. However, the patch size/length in the current paper is around 8-16 which doesn't match 64. I suggest authors modify the title.

2. More ablation studies. In the current presentation, the ablation without instance norm is not included. In the literature, using the instance norm or the equivalent RevIN in [1] could significantly improve the numerical performance. In order to further highlight the effectiveness of the patch and the channel independence ideas, I'm wondering if adding one more ablation study related to the instance norm will be helpful.

3. Robustness test. In time-series forecasting tasks, usually, the results are very sensitive to the model/training parameters.  How about the proposed model? Can we also include a section to discuss the robustness of the proposed model?


[1] Kim, T., Kim, J., Tae, Y., Park, C., Choi, J. H., & Choo, J. (2021, September). Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposed a transformer architecture that forecast multivariate time series channel-wise. In particular, the model employs patching by grouping values in temporal neighborhoods, so that the locality is enhanced and efficiency is improved. The model can also be trained in a self-supervised way by masking random patches, and it is shown such self-supervised pre-training can lead to superior performance with proper fine-tuning.
strength_and_weaknesses: Strength
1. The essential idea of the paper is well motivated and presented.
2. Extensive experiments are done to compare the proposed model with a variety of baselines.

Weakness:
1. The proposed patching is not novel as LogTrans (Li et al. 2019) has already proposed to use convolution before transformer to enhance locality. The linear processing in PatchTST with fixed path size and stride is not different from 1D convolution, even though a larger grid of configuration is investigated in the experiments.
2. The idea of masked modeling in representation learning is also marginally novel. The motivation of such a strategy is vague in certain parts. For example, in section 3.2, paragraph 2, _Since time series is often temporally redundant, the masked values at the current time step can be easily referred from neighboring values, which makes the reconstruction process trivial and thus the representation may not carry important abstract information_, the triviality here is not clear: in which case ("redundancy") the current time step can be too "easily" to refer? Masked modeling itself aims to refer the masked token/patch from contextual objects, why is it deemed trivial in time series? Moreover, in paragraph 3, the prediction in multivariate time series can also employ certain degree of weight sharing to avoid $(L\cdot D) \times (M\cdot T)$ parameter matrices.
3. Detailed questions: 
+ in supervised learning, the model is said to have a flatten layer with linear head to get univariate time series. When the patches have overlapping, how is the prediction within the overlapping determined?
+ how is the linear probing and end-to-end fine-tuning done after representation learning? Is linear probing simply reconstructing masked patches again but not update the backbone layers except the final linear head? 
clarity,_quality,_novelty_and_reproducibility: Clarity: limited.

Quality: fair.

Novelty: limited.

Reproducibility: fair
summary_of_the_review: Overall I think the current version of this paper does not meet the bar of ICLR due to limited novelty and insufficient clarity.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an efficient design of Transformer-based models (PatchTST) for multivariate time series forecasting and self-supervised representation learning. It segments time series into patches following similar philosophy of ViT and assume channels are independent. Extensive experiments on 8 eight datasets show that it can outperform state-of-the-art models while requires less compute.
strength_and_weaknesses: Strength

*  The idea of utilizing patch like ViT to reduce sequence length and better capture locality is quite simple and effective
* Extensive experiments on multiple datasets show that it can outperform recently state-of-the-art methods in different forecasting horizons and ablation study regarding patches and channel independence is conducted across multiple datasets.
*  This paper is well-organized and easy to understand

Weaknesses

* Channel independence is hardly novel as previous work, e.g. DeepAR [1] and LogTrans, also have this assumption. These two work model multivariate time series by sharing backbone parameters (LSTM/Transformer) of different channels during training, which can reduce parameter numbers and overfitting issue. 

* In experimental setups, "For Transformer-based models, the default lookback window is L = 96". However, this look-back window length can be quite short and may not be ideal especially when forecasting window is long, e.g. 336. In figure 2 (Electricity-T=720), best look-back window for Autoformer is  192/336 rather than 96. This setting make me have some concerns that baselines may be under-estimated.

* Self-supervised methods are only a little better and standard supervised fine-tuning. I am wondering if authors can have multiple runs and report mean and standard deviation so that readers may know how significant it is.

[1] Flunkert, Valentin et al. “DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks.” ArXiv abs/1704.04110 (2020): n. pag.


clarity,_quality,_novelty_and_reproducibility: This paper is clearly motivated and well-written. The idea of utilizing patches to deal with time series is interesting although channel independence is hardly novel given that previous work have explored this setting in time series forecasting. The paper provides some details of its implementation, however, it may need some efforts to reproduce its results as there is no sufficient details, e.g. hyperparameters and code, for reproduction.
summary_of_the_review: 

 I update my score to 8 since the authors' response resolve concerns and provide source code on Github.

\------------------------------------------------------------------------------

The idea of utilizing patch to deal with time series is interesting and have good empirical results. However, channel independence is hardly novel. In addition, I have concerns regarding experimental settings and am not sure if baselines are under-estimated and results statistically significant.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. There are two main components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. The experiment results show the effectiveness of their method.
strength_and_weaknesses: Strength: Multivariate time series forecasting and self-supervised representation learning are important tasks for time series. The proposed methods have some improvements on both tasks.

Weaknesses: I have the following questions.
(i) Segmentation of time series is a naive extension from computer vision (MAE). Thus, it lacks novelty in my perspective.
(ii) In my view, channel-independence is an orthogonal technic compared with other methods. In other words, the authors can perform channel-independence on other baselines. Can the authors show the performances with channel-independence in Table 3 and Table 4?
(iii) Can you provide some analysis for channel-independence? Since different channels have correlations compared with others, why channel-indpendence can improve performance?
clarity,_quality,_novelty_and_reproducibility: The paper is well-orgonized and clarified. However, its novelty is somewhat weak.
summary_of_the_review: The paper proposes two main technics for time series tasks. Even though the experiment results are better than baselines, the improvement is marginal and there miss some deep analysis. Thus, I vote for marginally below the acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper applies transformer to long term forecasting problems of multi-dimensional time series. The method is very simple: Take channels independently, break them into patches and predict the patches into the future using the transformer. The main advantage of this paper is that previous papers have applied transformers to this problem but it resulted in a very weak performance, being beaten by a simple linear methods. This paper found a way to apply the transformer successfully, beating the previous methods. 

While there isn't a large amount of novelty, the simplicity is actually a plus, given that the previous approaches using transformers failed and that the method achieves state of the art. To strengthen the paper, the authors should design a transformer architecture that takes in all the channels (not considering them independently) while preserving the advantage of the patch setup they are using and thus improving the performance further (beyond the independent method). 
justification_for_why_not_higher_score: Not large amount of novelty and not a large scale problem.
justification_for_why_not_lower_score: Simple method of applying transformer where previous such applications achieved poor results.
Decision: Accept: poster