Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes the application of ViT models for the task of timeseries classification. 
strength_and_weaknesses: S

- extensive experiments with multiple datasets and setups
- straightforward application of ViT to timeseries

W

- lack of discussion wrt better results in baselines (TST, Rocket)
- limited novelty compared to existing models (ViT)
clarity,_quality,_novelty_and_reproducibility: The paper is clear, of high quality, and reproducible. The novelty is limited considering that it's applying an existing model.
summary_of_the_review: It's unclear whether this model improves against other sota models in the literature based on Table 3. It seems that baselines perform equally well or even better here. The proposed model seems to perform better only on the internal dataset (private?) which makes the comparison with public benchmarks really difficult. I would urge the authors to expand more on the public benchmark experimentation and discuss (and expand) Table 3.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed the `VitMTSC` model, a vision Transformer model that learns latent features from raw time-series data for classification tasks, and could be applied on large-scale time series data with variable lengths. The model can reach comparable performance on UEA datasets against previous state-of-the-art methods and outperform them by a large margin on real-world dataset.

The contributions of this paper are from proposing a VitTransformer based approach to model long-range contextual relationship in MTS data, and demonstrate its superior performance on real-world application data.

strength_and_weaknesses: This paper provides an interesting idea of using a Vision Transformer for a Multivariate Time-Series Classification problem. The strengths of the paper are:
1. The figures are well-made and improve the clarity of the model description.
2. Details on tokenization, feature encoding, padding and truncation are discussed in the paper, improving the paper's reproducibility.
3. This paper did some interesting ablation study on layer normalization that change its position from after `MSA/FNN block + Residual connection` which is also called `POSTNORM` to before `MSA/FFN block` which is also called `PRENORM`. Experiments show it can improve applications' performance greatly on real-world dataset.
4. The experiments section includes some real world dataset, including offline and online tests, which is interesting to see.

On the other hand, the weaknesses of the paper are:

1. Even though the author(s) claimed this paper is based on Vision Transformer, there's no concept of "patch" applied in this feature extraction stage. The input was naturally given in different time stamps and features were extracted in a one-by-one fashion. After tokenization and feature encoding, there's only a linear projection applied onto each of the feature vectors. I tend to consider this process as some kind of feature engineering for the input of a Transformer. Can you please further explain how this is related to vision transformer's patch concept and whether the novelty comes from this process?
2. Covariates (time features in this paper) like month, day, hour have proven to be very useful in time series, while using them as position encoding could help, have you tried combining it into the original feature vector before linear projection and combine with learnable position encoding?
3. In the UEA dataset, the performance of the proposed model looks a bit worse than state-of-the-art models, but performance a lot better then these models on real world dataset. There seems to be not enough case study and analysis to further understand where the difference comes from and how we can mitigate the gap. I suggest the author(s) to provide more insights on the cause of this difference.
4. The bolded numbers in tables are a bit confusing, for example in Table 3, VitMTSC's validation set results were bolded but other results are from the test set. In Table 5, some of the results in the same ablation study groups are bolded but some are not, e.g., the `padding mask` group where the dataset is `Large`. Can you please add some explanation on the bolded number meaning?
5. The author(s) claimed to have `experimented with different aggregation strategies` but there were no conclusions or results mentioned. Can you please show the conclusion or performance with different aggregation strategies' impact?

clarity,_quality,_novelty_and_reproducibility: The clarity and reproducibility of this paper is in OK level, most of the notations can match up, and the figures are informative. Details of this model are well discussed, including various aspects like feature encoding, padding, position embedding. The Algorithm in page 6 looks a bit confusing because there are some notations unintroduced. Experiment section includes both UEA dataset and real world application, where there's not enough in-depth analysis on the gap of analysis between them. The most concerning point comes from the design of this architecture, where the concept "patch" of Vision Transformer was nowhere to be seen. It seems the whole input processing and embedding process is just a feature engineering for Transformer input. The use of `PRENORM` seems interesting but it's already proposed in GPT-2 (Radford et al., 2019) and ViT (Dosovitskiy et al., 2020). Therefore, the paper seems to lack novelty and hard to find impact other than the performance on the real-world application dataset, which I'm not sure if it's gonna be open sourced. Therefore, the quality and novelty of this paper is low.

summary_of_the_review: To summarize, this paper introduced a Vision Transformer based model called `VitMTSC` to improve multivariate time series classification. However, the novelty in this paper seems to be limited and the results have a huge gap between open-source UEA dataset and real-world application dataset. Even though the performance on real-world dataset looks great and the model can handle large scale dataset, the engineering part seems out of the scope of this paper and there's a lot more to explain on the experiment results. Therefore, I recommend this paper to be rejected.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper applies VisionTransformer for Multivariate Time-Series Classification (VitMTSC) model that learns latent features from raw time-series data for classification tasks and is applicable to large-scale time-series data with millions of data samples of variable lengths. Their results show that it's can obtain near state-of-the-art results on public benchmark and obtain significant improvements on real-world commercial datasets.
strength_and_weaknesses: Strength
* The idea is quite simple and cast time series as an "Image" and apply ViT to time series classification tasks.
* Results on real-word commercial datasets are significantly better than baselines.

Weaknesses
* The idea is hardly novel and it's more like a pure application of Vision Transformer without any deep insight.
* It's unclear why VitMTSC is significantly better than state-of-the-art baselines on commercial datasets. Authors don't discuss these insights in their papers and it's unclear when practitioners should utilize vision transformer for time series classification.
* The accuracy is pretty high on PenDigits, SpokenArabicDigits and CharacterTrajectories. Is it due to data imbalance? Could authors provide other metric, e.g. F1?
* This paper mainly explores binary classifications and multiple-class classification, however, is not fully considered. 
clarity,_quality,_novelty_and_reproducibility: The idea is quite simple and obtain promising results on real-world commercial datasets. However, it's unclear why VitMTSC can outperform state-of-the-art methods in a large margin in real-world commercial datasets while slightly left behind on public benchmarks. Without further insights, it's hardly novel in term of both technical and empirical perspectives. In addition, code link seems to be empty although authors claim that it's publicly available.
summary_of_the_review: This paper applies vision transformer in Multivariate Time-Series Classification problems. Results show that it can outperform state-of-the-art on commercial datasets while slightly left behind on public benchmark datasets. It's unclear why this performance gap exist, making this paper less valuable.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is an interesting application of the vision transformer for time-series data set, but has conducted extensive experiments to demonstrate the effectiveness on time-series data sets

The Novelty is limited and the superiority of the ViT to baseline methods, or the superiority of the baseline methods on some data sets are not adequately discussed.
justification_for_why_not_higher_score: The paper lacks novelty in method, and is a direct demonstration of ViT to several time-series data sets. At least a theory on why the ViT works superior to baseline methods or vice-versa would have enhanced novelty. 
justification_for_why_not_lower_score: The authors can enhance the novelty and resubmit the manuscript to another venue. Hope the review comments are useful for revision.
Decision: Reject