Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors targeted an important data format, multivariate time series, and extended the usage of the transformer to this format. The effort is appreciated as multivariate time series data is an important problem while the researches there is limited comparing to other sequences problems, e.g. language. Due to the simple idea of extension and similar works before [1], I expected a higher quality of experiments and presentation of this paper. However, experiments and writing need significant improvement for the next submission. 

Major Concerns: 

1. The experiment's presentation is confusing and I just picked a few examples from Table 1: 
a. There is no standard deviation of the result. 
b. Some methods are consistently worse than others, like 1-NN-DTWD comparing to 5-NN-DTWD. I am not sure why they are needed in the table. 
c. Averaging the RMSE does not deliver much information, especially when there is no normalization on each dataset. For example, the BeijingPM10 will dominate the averaged result. 

2. The structure of writing only has a lot of problems. For example, usually, the fully supervised should be presented before semi-supervised while the paper did in a reverse way. Another problem is the classification result, which seems an important part but the table is shown in the Appendix. 

3. The selection of datasets needs more justifications. 

Reference: 
1. https://arxiv.org/pdf/2001.08317.pdf




review_title: The paper extends the usage of transformer from univariate to multivariate sequential data  

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper aims to develop a transformer-based pre-trained model for multivariate time series representation learning. Specifically, the transformer’s encoder is only used and a time-series imputation task is constructed as their unsupervised learning objective. This is a bit similar to the BERT model in NLP. But authors added a mask for each variable of the time series. After pretraining with this imputation loss, the transformer can be used for downstream tasks, such as regression and classification. As the authors mentioned on page 6, this is achieved by further fine-tuning all weights of the pre-trained transformer. 

It is natural to use the transformer model from NLP for time series modeling since both sentences and time series are sequential data. In this work, the authors’ contributions or changes should include two-points: 1) constructing that imputation task for multivariate time series data. 2) using a learned positional encoding (page 4). I think these two things seem a bit interesting. 
My concerns mainly include:

1. Actually, there are existing works that have tried to use the transformer for time series. But you didn’t compare them in your experiments. At least, I think you should clarify what’s your advantages comparing to these existing works:
- 2019 NeurIPS Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
- 2020 CVPR Sketch-BERT: Learning Sketch Bidirectional Encoder Representation From Transformers by self-supervised Learning of Sketch Gestalt 

3. As a core topic in this work, I encourage authors to clarify the definition of the time series representation. What’s a good representation in time series? There is a related work for time series representation learning: 
- 2019 NeurIPS Unsupervised Scalable Representation Learning for Multivariate Time Series 
I notice that they also train a dilated CNN model to get a single feature for a segment of time series. But in your case, you directly concatenate states from all-time steps to form your final vector. I just wonder if your strategy is reasonable. Because when we deal with long time series, concatenating them will create another long time-series/hidden state sequence. This may not be representation learning and it cannot handle the segment-level tasks such as classification. Thus, I encourage authors to give more insights into the questions what’s a good representation of the time series. 

3. Besides, in the above NeurIPS representation learning paper, I find they didn’t fine-tune their model’s parameters. They just added SVM to test the performance of learned representations. But in this work, the authors fine-tuning their model. Thus, I wonder what’s the performance of your model without additional fine-tuning. What's your parameter settings for the fine-tuning procedure. 

4. Another question I am concerned about is your learnable position encoding. It is with a shape of w-by-d where w should be your window length and d is your input dimension. Since time series is dynamic data and its length would be much longer, this design maybe not good as it largely increases the number of parameters.  

5. For experimental results, authors mainly consider their model’s performance on regression and classification tasks. But I would like to see more analysis of their learned presentations. Adding more visualization analysis would be helpful to demonstrate your framework’s effectiveness. 

6. Experimental settings are unclear. For example, since your datasets only contain train/test sets. How to pick your hyperparameters? Is it based on that performance on that test set? What kind of regression task is used in your regression experiments? Is it a one-step-ahead prediction task? 

7. In Table 5, you claim your model is faster than the ROCKET model. But the running time reported for your model is per-epoch training time, not the total training time. I think this seems a bit unreasonable. Please check it. 

review_title: a transformer is used for multivariate time series representation learning, but there are some unclear points about the proposed model and experimental settings.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper uses transformer to improve mutlivariate time series classification and regression using a BERT inspired self-supervised loss. The authors show improvement over multiple standard datasets. The use of self-supervision improves performance in lower (labeled) data regime.


#### Strong points

- The method is well explained and take good inspiration of BERT pretraining.
- The data-available experiment shows that training with self-supervision helps for classification.
- This work is well positioned in the literature of the field.

#### Weak points

- The timeserie representation is obtained by concatenation of the tokens' representations. This means that the representations are not of fixed size (or contain padding representations).
- The introduced masking loss could suffer from discrepancy between training and inference where all the covariates at a given timesteps need to be predicted. Moreover, the model could rely on very correlated covariates (which frequently happens) to recover the masked variables limiting the effect of self-supervised training. I would like to see if masking all variables at some times step helps/hurts the pretraining.
- The paper does not consider if finetuning is needed or if the network could be frozen and only a small MLP could be learned on top of the "pre-extracted times-series representations".
- The hyperparameters of the network are chosen per dataset but are not reported. It is not clear if the tuning is done with cross validation or looking at the test error. The authors should report the performance of the transformer with common hyperparameters specified in the Appendix. This would be more fair to compare with Franceschi et al., who do not tune the hyperparameters of their encoder.
- The claim in the abstract that the method offers "computational efficiency" is not backed with evidence. Section A.4 "Execution Time" does not report execution time of the method compared to the baselines.


#### Decision

I tend to reject this paper. The idea is interesting but some design decisions of the method (representation pooling, self-supervised loss, need for finetuning) should be better justified. Secondly the evaluation of the´ method should be more precise (hyperparameters tuning, speed).

#### Questions/Remarks

- In section 3.1 "because the computational complexity and the number of parameters of the model
scale as $O(w^2)$ with the input sequence length $w$". The number of parameters of the transformer architecture does not scale with the input length. Only the memory footprint and the computation scales quadratically.

- "we note that similar to the case of word embeddings, the positional encodings generally appear not to interfere with the numerical information of the time series". Do you have evidence for this?

- Have you tried PowerNorm by Shen et al. (2020) that you cite in 3.1?

- You could mention Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting (Lim et al., 2019) arXiv:1912.09363 for another use of transformers for univariate (quantile) forecasting.

- Add the dataset you are experimenting with in the caption of Figure 2.

#### Additional feedback

- Figure 2: chose different colors for left and right figures.
- Table 1: consider using booktabs for table format and transpose dataset and models to make it fit in the margin.
- Section 3.2: The sentence starting with "The reason ..." that spans 6 lines could be split and reworded.
review_title: Interesting adaptation of BERT objective to timeseries self-supervised training but performance and design decisions not rigorously evaluated.

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
# Summary:

The paper proposes an unsupervised learning framework, similar to BERT idea but for multivariate time series.
### Architecture: 
- Inputs are projected into d-dimensional vector space where d is the dimension of the transformer model sequence element representations.
- A learned positional encoding is added to the input which is passed to the transformer encoder.
-The output of the encoder is then passed to another projection layer that depends on the application (i.e classification or regression)
-The transformer architecture is similar to Vaswani et al. (2017), however, they replaced layer norm with batch norm.

###  Unsupervised Learning Training:
For unsupervised learning, the paper proposed taking the input data masking it, and predicting an output sequence, the loss is then calculated as the mean square error between the actual input and the predicted input.

###  Evaluation and Results:
- The paper considered two tasks both regression and classification.  For regression, the proposed method was evaluated on datasets from the Monash University, UEA, UCR Time Series Regression Archive Tan et al. (2020a); for classification, the paper used the UEA Time Series Classification Archive (Bagnall et al., 2018).
- For each dataset, they showed the results of using the proposed transformer architecture in a supervised manner and then using the same dataset for pretraining the transformer by the proposed masking approach in an unsupervised manner followed by fine-tuning using labels. Both methods were compared to other state of the art methods for the same task.
- For multivariate regression, the pretraining followed by supervised training showed improvements in 3 out of 6 datasets. For classification 7 out of 11 datasets showed improvements.

# Strength:
- The paper focues  an important yet a relatively unexplored area.
- The paper is clear, well written and well motivated.
- The paper benchmarked accorss multiple datasets on both classification and regression tasks.

# Weakness:
- My main concern is the lack of novelty the paper is basically suggesting to use a  transformer encoder and add a dense layer before and after, and if we use unsupervised training of the transformer with the same dataset we ***may*** achieve better results.
- The main success of BERT is the ability to transfer and improve the performance on unretaleted task however here they did not include any experiments showing that if we train on model we can use to transfer on another model. I believe critical experiments that are missing is unsupervised training say with dataset 1 and the fine tuning on dataset 2 and showing improvements on multiple task (However, even if this experiments are provided I still believe the paper lacks novelty, maybe invistaginting what properties are being transferred in multivariate time series and showing difference between transferring from unvariate to multivariate and vice versa will help).
- The paper replaced layer norm with batch norm and only stated "here we instead use batch normalization, because it can mitigate the effect of outlier values in time series, an issue that does not arise in NLP word embedding" they did not show the effect of using batch norm on the accuracy or gave any insights on why it *mitigate the effect of outlier values in time series*.
- The code implementing the paper is not provided.
- The effect of changing values variable r in masking is not investigated.



 
review_title: Representation learning on same dataset without transfer

Meta-review: The authors extends the transformer to multivariate time series. The proposed extension is simple, and lacks novelty. Some design decisions of the proposed method should be better justified. Similar works that also use the transformer for timeseries are not compared.

Experimental results are not convincing. The settings are unclear, and the selection of datasets needs more justifications. Some important experiments are missing.

Finally, writing can also be improved.
Decision: Reject