Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an adaptation of the popular MaskedAE model for timeseries data. 
strength_and_weaknesses: S

- solid experimental work in terms of datasets and models compared
- good results with multiple ablation tests

W 

- lack of connection between motivation (disentanglement) and the actual proposed method
- non-significant result compared to TS2Vec baseline (figure 6)


clarity,_quality,_novelty_and_reproducibility: This work adapts an existing model to timeseries data. The work is of high-quality and clarity.
summary_of_the_review: I am positive about the contribution of this paper which might be considered a bit incremental, however, it's the first work in this area. I am reluctant about the comparison with TS2Vec in Figure 6 which shows that both methods are identical.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper propose a masked time series autoencoders which can learn strong representations with less inductive bias or hierarchical trick. The proposed Ti-MAE bridges the connection between contrastive representation learning and generative Transformer-based methods. Ti-MAE adequately leverages all the input sequence and alleviates the distribution shift problem. The flexible setting of masking ratio makes Ti-MAE more adaptive to various prediction scenarios with different time steps. Experimental results demonstrate the efficacy of proposed method.
strength_and_weaknesses: Strengths:
1. It's novel to propose the masked time series auto encoders which can learn strong representations with less inductive bias or hierarchical trick.
2. The experiments are evaluated on various of settings, e.g. multivariate time series forecasting, classification


Weaknesses:
1. The masking ratio is tricky. It highly depends on the dataset. When dataset changes, the masking ratio may change. The paper didn't mention how to deal with this issue.
2. The experiments do not compare with some state-of-the-art method, such as NHITS, FedFormer and etc.
3. No ablation study on the effect of different backbone models, e.g. LSTM, Transformer, TCN and etc.
clarity,_quality,_novelty_and_reproducibility: This paper is pretty clear. The figures are easy to read. The novelty is marginal. It looks reproducible.
summary_of_the_review: This paper proposes Ti-MAE to bridge the connection between contrastive representation learning and generative Transformer-based methods. It can learn strong representations with less inductive bias or hierarchical trick. Ti-MAE adequately leverages all the input sequence and alleviates the distribution shift problem. The flexible setting of masking ratio makes Ti-MAE more adaptive to various prediction scenarios with different time steps. Experimental results demonstrate the efficacy of proposed method.

However, the masking ratio is tricky. It highly depends on the dataset. When dataset changes, the masking ratio may change. The paper didn't mention how to deal with this issue. The experiments do not compare with some state-of-the-art method, such as NHITS, FedFormer and etc. No ablation study on the effect of different backbone models, e.g. LSTM, Transformer, TCN and etc.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work propose a simple and novel self-supervised learning framework for time series representation learning. Instead of using contrastive representation learning, this work directly follow the masked data modeling and optimize the reconstruction loss of the randomly masked data points. Empirical studies on real-world datasets and ablation studies show the effectiveness and scalability of the proposed framework.
strength_and_weaknesses: Pros:

1. The proposed MAE based self-supervised learning framework is interesting and reasonable. Experimental results also demonstrate improvement on most real-world datasets.

2. The paper is well-organized, and the writing is easy to follow.


Cons:

1. The experiments are not comprehensive. The baselines reported in the current version are not SOTA. How are the comparison results with the recent transformer based models, like FEDformer and ETSformer as the authors mentioned? Note that there are some lightweight models (Dlinear, DeepTIME, etc.) which can beat the transformer based models. It would be more convincing to compare these models.


2. The proposed method is reasonable, but the novelty is limited. It is quite straightforward to apply MAE on time series data. Could the authors elaborate more on the challenges when applying MAE on time series data?

Questions:

1. Does the proposed method follow the pre-training + fine-tuning paradigm? It seems the randomly mask strategy could align the reconstruction and prediction tasks and eliminate the effort to build an extra forecasting model.

2. How is the model complexity compared with other transformed based forecasting models?
clarity,_quality,_novelty_and_reproducibility: The idea is clear and reasonable, but novelty is limited. The empirical results would be more convincing to include the STOA results on long sequence forecasting task.
summary_of_the_review: see above
____

I would like to thank the author's great effort for the detailed response and additional experiments. However, I would like to keep the original score due to the following concerns. First, the novelty of methodology and technique are limited. I appreciate the explanation of MAE and the proposed method, but I still feel the proposed method is a straightforward solution to apply it on time series data.  Second, I understand the point from the author that there is no need to compare works unpublished, like deeptime, dlinear. But there exists some published paper which achieves much better results than the baselines selected in this manuscript, for example [1,2]. It would be promising  to see a significant improvement with a simple method in this time series forecasting benchmark. Third, the proposed method has quadratic complexity which is worse than the $L\log L$ complexity of the baseline transformers.

[1] Learning Latent Seasonal-Trend Representations for Time Series Forecasting
[2] FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised framework for time series representation learning. It randomly masks out tokenized time series and learns an autoencoder to reconstruct them at the point-level. The proposed model is applied to time series forecasting and clasification.


strength_and_weaknesses: Strength
1. The authors clearly presents the idea of using masked time series autoencoders for time series forecasting and classification.
2. They also did extensive experiments on multiplier real datasets and compared the proposed method with SOTA ones.

Weaknesses
1. "input time series are assumed to follow an integrate distribution"
What does integrate distribution mean? 
2. In (1), the polynomial series and the Fourier series are not orthogonal thus the learning might have no unique solution. How are the number of polynomial orders and the Fourier orders decided? 
3. It is not clear why 'randomly masks out parts of embedded time series data' can help 'alleviate the distribution shift problem'. Why periodic masking cannot address the same issue? Why do we need randomness here?
4. The way the time series data is randomly masked will affect the performance and is worth being discussed.
5. How can the proposed Ti-MAE make forecasting 'for multiple time windows with various sizes without re-training'? This claim in Section 1 is not justified neither structurally or experimentally.
6. The input time series and the output one has the same dimensionality. This is a quite strong but not necessary assumption. Can the proposed model be applied to forecasting time series with different dimensionality?
7. The ablation study is not complete. Why does data augmentation give worse results? How the way that masks are randomly sampled (as mentioned in 4) affect the performance?
8. It will be helpful to show the training and testing errors and the generalization gap between them.

clarity,_quality,_novelty_and_reproducibility: 1. The authors give a relatively clear high-level description of the proposed method.
2. The idea is marginally novel considering that auto-encoder has been widely used in supervised learning tasks including time series forecasting.
3. There are many unclear details, e.g, how the parameters of encoder and decoder are specified. Without these details, it is hard to reproduce the results in the paper.
summary_of_the_review: The idea is marginally novel and the applications on time series forecasting and classification are interesting.
There are some claims not well justified and need to be addressed.


------------
I have read the authors' responses and would like to keep the same score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a MAE based self-supervised learning framework for time-series representation learning. A random masking strategy was adopted on the time-series data and then the authors learned the model by optimizing the reconstruction loss of those randomly masked data points. 

Strengths: 
1. The paper is clearly written. 
2. The model performance on time series forecasting is very promising. 
3. The experiments are quite comprehensive. After the rebuttal, it is very nice that the authors added experiments to show the effect of different backbones and the effect of the masking ratio. The authors also included additional baselines as suggested by the reviewers.

Weaknesses: 
1. I agree with two reviewers (6CpA and 5eEm) that this work is a straightforward application of MAE on time series data. The novelty is not sufficient for a top-conference like ICLR. 
2. Performance improvement on classification is marginal. To make this model general for time series analytics (forecasting, classification and regression), the authors should address challenges specific to time series. Following the weakness above, additional modules to address these challenges will improve the novelty. 
3. Although the current experiments are comprehensive, one reviewer (5eEm) still raised a valid point that the baselines reported in the paper are not SOTA. The authors can include the comparison with FiLM, LaST,  DeepTime, DLinear and NHITS in the future version.

justification_for_why_not_higher_score: Although this paper has an average score of 5.25, I still don't consider it as a borderline paper and recommend a rejection for two main reasons. First, majority of the reviewers (3 out of 4) have negative ratings. Second, I agree with two of the reviewers that this paper is a straightforward application of MAE on time series data and thus the novelty is limited.
Decision: Reject