Title: Time Series as Images: Vision Transformer for Irregularly Sampled Time Series
Venue: NeurIPS 2023
Accepted: yes
Keywords: irregularly sampled time series

vision transformer

healthcare

time series classification
Abstract: Irregularly sampled time series are increasingly prevalent, particularly in medical domains. While various specialized methods have been developed to handle these irregularities, effectively modeling their complex dynamics and pronounced sparsity remains a challenge. 
This paper introduces a novel perspective by converting irregularly sampled time series into line graph images, then utilizing powerful pre-trained vision transformers for time series classification in the same way as image classification. This method not only largely simplifies specialized algorithm designs but also presents the potential to serve as a universal framework for time series modeling. Remarkably, despite its simplicity, our approach outperforms state-of-the-art specialized algorithms on several popular healthcare and human activity datasets. Especially in the rigorous leave-sensors-out setting where a portion of variables is omitted during testing, our method exhibits strong robustness against varying degrees of missing observations, achieving an impressive improvement of 42.8% in absolute F1 score points over leading specialized baselines even with half the variables masked. Code and data are available at https://github.com/Leezekun/ViTST.

Time Series as Images: Vision Transformer for Irregularly Sampled Time Series

Abstract

Irregularly sampled time series are increasingly prevalent, particularly in medical domains. While various specialized methods have been developed to handle these irregularities, effectively modeling their complex dynamics and pronounced sparsity remains a challenge. This paper introduces a novel perspective by converting irregularly sampled time series into line graph images, then utilizing powerful pretrained vision transformers for time series classification in the same way as image classification. This method not only largely simplifies specialized algorithm designs but also presents the potential to serve as a universal framework for time series modeling. Remarkably, despite its simplicity, our approach outperforms state-of-the-art specialized algorithms on several popular healthcare and human activity datasets. Especially in the rigorous leave-sensors-out setting where a portion of variables is omitted during testing, our method exhibits strong robustness against varying degrees of missing observations, achieving an impressive improvement of 42.8% in absolute F1 score points over leading specialized baselines even with half the variables masked. Code and data are available at https://github.com/Leezekun/ViTST.

1 Introduction

Time series data are ubiquitous in a wide range of domains, including healthcare, finance, traffic, and climate science. With the advances in deep learning architectures such as LSTM [13], Temporal Convolutional Network (TCN) [18], and Transformer [38], numerous algorithms have been developed for time series modeling. However, these methods typically assume fully observed data at regular intervals and fixed-size numerical inputs. Consequently, these methods encounter difficulties when faced with irregularly sampled time series, which consist of a sequence of samples with irregular intervals between their observation times. To address this challenge, highly specialized models have been developed, which require substantial prior knowledge and efforts in model architecture selection and algorithm design [24, 20, 3, 16, 49, 34, 48].

In parallel, pre-trained transformer-based vision models, most notably vision transformers, 1 have emerged and demonstrated strong abilities in various vision tasks such as image classification and object detection, nearly approaching human-level performance. Motivated by the flexible and effective manner in which humans analyze complex numerical time series data through visualization, we raise the question: Can these powerful pre-trained vision transformers capture temporal patterns in visualized time series data, similar to how humans do?

To investigate this question, we propose a minimalist approach called ViTST (Vision Time Series Transformer), which involves transforming irregularly sampled multivariate time series into line graphs, organizing them into a standard RGB image format, and finetuning a pre-trained vision

Time

00:30

00:34

02:13

03:25

…

Variable

PaO2

HCT

PaCO2

SysABP

…

Value

155.0

29.1

33.0

143.0

…

Multivariate Time Series

PaCO2                 SysABP

HCT                    PaO2

…

Vision Transformer Encoder

(e.g. ViT, Swin)

Prediction Head

Class

Class 1

Class 2

…

transformer for classification using the resulting image as input. An illustration of our method can be found in Figure 1.

Line graphs serve as an effective and efficient visualization technique for time series data, regardless of irregularity, structures, and scales. They can capture crucial patterns, such as the temporal dynamics represented within individual line graphs and interrelations between variables throughout separate graphs. Such a visualization technique benefits our approach because it is both simple and intuitively comprehensible to humans, enabling straightforward decisions for time series-to-image transformations. Leveraging vision models for time series modeling in this manner mirrors the concept of prompt engineering , wherein individuals can intuitively comprehend and craft prompts to potentially enhance the processing efficiency of language models.

We conduct a comprehensive investigation and validation of the proposed approach, ViTST, which has demonstrated its superior performance over state-of-the-art (SoTA) methods specifically designed for irregularly sampled time series. Specifically, ViTST exceeded prior SoTA by 2.2% and 0.7% in absolute AUROC points, and 1.3% and 2.9% in absolute AUPRC points for healthcare datasets P19 [29] and P12 [12], respectively. For the human activity dataset, PAM [28], we observed improvements of 7.3% in accuracy, 6.3% in precision, 6.2% in recall, and 6.7% in F1 score (absolute points) over existing SoTA methods. Our approach also exhibits strong robustness to missing observations, surpassing previous leading solutions by a notable 42.8% in absolute F1 score points under the leave-sensors-out scenario, where half of the variables in the test set are masked during testing. Furthermore, when tested on regular time series data including those with varying numbers of variables and extended sequence lengths, ViTST still achieves excellent results comparable to the specialized algorithms designed for regular time series modeling. This underscores the versatility of our approach, as traditional methods designed for regularly sampled time series often struggle with irregularly sampled data, and vice versa.

In summary, the contributions of this work are three-fold: (1) We propose a simple yet highly effective approach for multivariate irregularly sampled time series classification. Despite its simplicity, our approach achieves strong results against the highly specialized SoTA methods. (2) Our approach demonstrates excellent results on both irregular and regular time series data, showcasing its versatility and potential as a general-purpose framework for time series modeling. It offers a robust solution capable of handling diverse time series datasets with varying characteristics. (3) Our work demonstrates the successful transfer of knowledge from vision transformers pre-trained on natural images to synthetic visualized time series line graph images. We anticipate that this will facilitate the utilization of fast-evolving and well-studied computer vision techniques in the time series domains, such as better model architecture [23], data augmentation [32], interpretability [4], and self-supervised pre-training [15].

2 Related work

Irregularly sampled time series. An irregularly sampled time series is a sequence of observations with varying time intervals between them. In a multivariate setting, different variables within the

same time series may not align. These characteristics present a significant challenge to standard time series modeling methods, which usually assume fully observed and regularly sampled data points. A common approach to handling irregular sampling is to convert continuous-time observations into fixed time intervals [24, 20]. To account for the dynamics between observations, several models have been proposed, such as GRU-D [3], which decays the hidden states based on gated recurrent units (GRU) [6]. Similarly, [46] proposed an approach based on multi-directional RNN, which can capture the inter- and intra-steam patterns. Besides the recurrent and differential equationbased model architectures, recent work has explored attention-based models. Transformer [38] is naturally able to handle arbitrary sequences of observations. ATTAIN [50] incorporates attention mechanism with LSTM to model time irregularity. SeFT [16] maps the irregular time series into a set of observations based on differentiable set functions and utilizes an attention mechanism for classification. mTAND [34] learns continuous-time embeddings coupled with a multi-time attention mechanism to deal with continuous-time inputs. UTDE [48] integrates embeddings from mTAND and classical imputed time series with learnable gates to tackle complex temporal patterns. Raindrop [49] models irregularly sampled time series as graphs and utilizes graph neural networks to model relationships between variables. While these methods are specialized for irregular time series, our work explores a simple and general vision transformer-based approach for irregularly sampled time series modeling.

Numerical time series modeling with transformers. Transformers have gained significant attention in time series modeling due to their exceptional ability to capture long-range dependencies in sequential data. A surge of transformer-based methods have been proposed and successfully applied to various time series modeling tasks, such as forecasting [19, 51, 41, 52], classification [47], and anomaly detection [45]. However, these methods are usually designed for regular time series settings, where multivariate numerical values at the same timestamp are viewed as a unit, and temporal interactions across different units are explicitly modeled. A recent work [25] suggests dividing each univariate time series into a sequence of sub-series and modeling their interactions independently. These methods all operate on numerical values and assume fully observed input, while our proposed method deals with time series data in the visual modality. By transforming the time series into visualized line graphs, we can effectively handle irregularly sampled time series and harness the strong visual representation learning abilities of pre-trained vision transformers.

Imaging time series. Previous studies have explored transforming time series data into different types of images, such as Gramian fields [39], recurring plots [14, 37], and Markov transition fields [40]. These approaches typically employ convolutional neural networks (CNNs) for classification tasks. However, they often require domain expertise to design specialized imaging techniques and are not universally applicable across domains. Another approach [35] involves utilizing convolutional autoencoders to complete images transformed from time series, specifically for forecasting purposes. Similarly, [31] utilized CNNs to encode images converted from time series and use a regressor for numerical forecasting. These approaches, however, still need plenty of specialized designs and modifications to adapt to time series modeling. Furthermore, they still lag behind the current leading numerical techniques. In contrast, our proposed method leverages the strong abilities of pre-trained vision transformer to achieve superior results by transforming the time series into line graph images, sidestepping the need of prior knowledge and specific modifications and designs.

3 Approach

As illustrated in Fig. 1, ViTST consists of two main steps: (1) transforming multivariate time series into a concatenated line graph image, and (2) utilizing a pre-trained vision transformer as an image classifier for the classification task. To begin with, we present some basic notations and problem formulation.

Notation. Let D = { ( S i , y i ) | i = 1 , · · · , N } denote a time series dataset containing N samples. Every data sample is associated with a label y i ∈ { 1 , · · · , C } , where C is the number of classes. Each multivariate time series S i consists of observations of D variables at most (some might have no observations). The observations for each variable d are given by a sequence of tuples with observed time and value [( t d 1 , v d 1 ) , ( t d 2 , v d 2 ) , · · · , ( t d n d , v d n d )] , where n d is the number of observations for variable d . If the intervals between observation times [ t d 1 , t d 2 , · · · , t d n d ] are different across variables or samples, S i is an irregularly sampled time series. Otherwise, it is a regular time series.

Problem formulation. Given the dataset D = { ( S i , y i ) | i = 1 , · · · , N } containing N multivariate time series, we aim to predict the label ˆ y i ∈ { 1 , · · · , C } for each time series S i . There are mainly two components in our framework: (1) a function that transforms the time series S i into an image x i ; (2) an image classifier that takes the line graph image x i as input and predicts the label ˆ y i .

3.1 Time Series to Image Transformation

Time series line graph. The line graph is a prevalent method for visualizing temporal data points. In this representation, each point signifies an observation marked by its time and value: the horizontal axis captures timestamps, and the vertical axis denotes values. Observations are connected with straight lines in chronological order, with any missing values interpolated seamlessly. This graphing approach allows for flexibility for users in plotting time series as images, intuitively suited for the processing efficiency of vision transformers. The practice mirrors prompt engineering when using language models, where users can understand and adjust natural language prompts to enhance the model performance.

In our practice, we use marker symbols ' ∗ ' to indicate the observed data points in the line graph. Since the scales of different variables may vary significantly, we plot the observations of each variable in an individual line graph, as shown in Fig. 1. The scales of each line graph g i,d are kept the same across different time series S i . We employ distinct colors for each line graph for differentiation. Our initial experiments indicated that tick labels and other graphical components are superfluous, as an observation's position inherently signals its relative time and value magnitude. We investigated the influences of different choices of time series-to-image transformation in Section 4.3.

Image Creation. Given a set of time series line graphs G i = g 1 , g 2 , · · · , g D for a time series S i , we arrange them in a single image x i using a pre-defined grid layout. We adopt a square grid by default, following [10]. Specifically, we arrange the D time series line graphs in a grid of size l × l if l × ( l -1) < D ≤ l × l , and a grid of size l × ( l +1) if l × l < D ≤ l × ( l +1) . For example, the P19, P12, and PAM datasets contain 34, 36, and 17 variables, respectively, and the corresponding default grid layouts are 6 × 6 , 6 × 6 , and 4 × 5 . Any grid spaces not occupied by a line graph remain empty. Figure 6 showcases examples of the resulting images. As for the order of variables, we sort them according to the missing ratios for irregularly sampled time series. We explored the effects of different grid layouts and variable orders in Section 4.3.

3.2 Vision Transformers for Time Series Modeling

Given the image x i transformed from time series S i , we leverage an image classifier to perceive the image and perform the classification task. The time series patterns in a line graph image involve both local ( i.e. , the temporal dynamics of a single variable in a line graph) and global (the correlation among variables across different line graphs) contexts. To better capture these patterns, we choose the recently developed vision transformers. Unlike the predominant CNNs, vision transformers are proven to excel at maintaining spatial information and have stronger abilities to capture local and global dependencies [9, 22].

Preliminary. Vision Transformer (ViT) [9] is originally adapted from NLP. The input image is split into fixed-sized patches, and each patch is linearly embedded and augmented with position embeddings. The resulting sequence of vectors is then fed into a standard Transformer encoder consisting of a stack of multi-head attention modules (MSA) and MLP to obtain patch representations. An extra classification token is added to the sequence to perform classification or other tasks. ViT models global inter-unit interactions between all pairs of patches, which can be computationally expensive for high-resolution images. Swin Transformer, on the other hand, adopts a hierarchical architecture with multi-level feature maps and performs self-attention locally within non-overlapping windows, reducing computational complexity while improving performance. We use Swin Transformer as the default backbone vision model unless otherwise specified, but any other vision model can also be used within this framework.

Swin Transformer captures the local and global information by constructing a hierarchical representation starting from small-sized patches in earlier layers and gradually merging neighboring patches in deeper layers. Specifically, in the W-MSA block, self-attention is calculated within each non-overlapping window, allowing for the capture of local intra-variable interactions and temporal dynamics of a single line graph for a variable d . The shifted window block SW-MSA then enables

connections between different windows, which span across different line graphs, to capture global interactions. Fig. 2 illustrates this process. Mathematically, the consecutive Swin Transformer blocks are calculated as:

where ˆ z l and z l denote the output features of the (S)W-MSA module and the MLP module for block l , respectively; LN stands for the layer normalization [1]. After multiple stages of blocks, the global interactions among patches from all line graphs can be captured, enabling the modeling of correlations between different variables. We have also explored the use of additional positional embeddings, including local positional embeddings to indicate the position of each patch within its corresponding line graph, and global positional embeddings to represent the index of the associated line graph in the entire image. However, we didn't observe consistent improvement over the already highly competitive performance, which might suggest that the original pre-trained positional embeddings have already been able to capture the information regarding local and global patch positions.

shifted

Inference. We use vision transformers to predict the labels of time series in the same way as image classification. The outputs of Swin Transformer blocks at the final stage are used as the patch representations, upon which a flattened layer with a linear head is applied to obtain the prediction ˆ y i .

4 Experiments

4.1 Experimental Setup

Datasets and metrics. We conducted experiments using three widely used healthcare and human activity datasets, as outlined in Table 1. The P19 dataset [29] contains information from 38,803 patients, with 34 sensor variables and a binary label indicating sepsis. The P12 dataset [12] comprises data from 11,988 patients, including 36 sensor variables and a binary label indicating survival during hospitalization. Lastly, the PAM dataset [28] includes 5,333 samples from 8 distinct human activities, with 17 sensor variables provided for each sample. We used the processed data provided by Raindrop [49]. To ensure consistency, we employed identical data splits across all comparison baselines, and evaluated the performance using standard metrics such as Area Under a ROC Curve (AUROC) and Area Under Precision-Recall Curve (AUPRC) for the imbalanced P12 and P19 datasets. For the more balanced PAM dataset, we reported Accuracy, Precision, Recall, and F1 score.

Implementation. We utilized the Matplotlib package to plot line graphs and save them as standard RGB images. For the P19, P12, and PAM datasets, we implemented grid layouts of 6 × 6 , 6 × 6 , and 4 × 5 , respectively. For a fair comparison, we assigned a fixed size of 64 × 64 to each grid cell (line graph), resulting in image sizes of 384 × 384 , 384 × 384 , and 256 × 320 , respectively. It is important to note that image sizes can also be directly set to any size, irrespective of grid cell dimensions. We plot the images according to the value scales on training sets. We employed the checkpoint of Swin

Transformer pre-trained on the ImageNet-21K dataset 2 . The default patch size and window size are 4 and 7 , respectively.

Training. Given the highly imbalanced nature of the P12 and P19 datasets, we employed upsampling of the minority class to match the size of the majority class. We finetuned the Swin Transformer for 2 and 4 epochs on the upsampled P19 and P12 datasets, respectively, and for 20 epochs on the PAM dataset. The batch sizes used for training were 48 for P19 and P12, and 72 for PAM, while the learning rate was set to 2e-5. The models were trained using A6000 GPUs with 48G memory.

Incorporating static features. In real-world applications, especially in the healthcare domain, irregular time series data often accompanies additional information such as categorical or textual features. In the P12 and P19 datasets, each patient's demographic information, including weight, height, and ICU type, is provided. This static information remains constant over time and can be expressed using natural language. To incorporate this information into our framework, we employed a template to convert it into natural language sentences and subsequently encoded the resulting text using a RoBERTa-base [21] text encoder. The resulting text embeddings were concatenated with the image embeddings obtained from the vision transformer to perform classification. The static features were also applied to all compared baselines.

4.2 Main Results

Comparison to state-of-the-art. Wecompare our approach with several state-of-the-art methods that are specifically designed for irregularly sampled time series, including Transformer [38], Trans-mean (Transformer with an imputation method that replaces the missing value with the average observed value of the variable), GRU-D [3], SeFT [16], mTAND [34], IP-Net [33], and Raindrop [49]. In addition, we also compared our method with two approaches initially designed for forecasting tasks, namely DGM 2 -O [42] and MTGNN [43]. The implementation and hyperparameter settings of these baselines were kept consistent with those used in Raindrop [49]. Specifically, a batch size of 128 was employed, and all compared models were trained for 20 epochs. To ensure fairness in our evaluation, we averaged the performance of each method over 5 data splits that were kept consistent across all compared approaches.

As shown in Table 2, our proposed approach demonstrates strong performance against the specialized state-of-the-art algorithms on all three datasets. Specifically, on the P19 and P12 datasets, ViTST achieves improvements of 2.2% and 0.7% in absolute AUROC points, and 1.3% and 2.9% in absolute AUPRC points over the state-of-the-art results, respectively. For the PAM dataset, the improvement is even more significant, with an increase of 7.3% in Accuracy, 6.3% in Precision, 6.2% in Recall, and 6.7% in absolute F1 score points.

Leaving-sensors-out settings. We conducted additional evaluations to assess the performance of our model under more challenging scenarios, where the observations of a subset of sensors (variables) were masked during testing. This setting simulates real-world scenarios when some sensors fail or become unreachable. Following the approach adopted in [49], we experimented with two setups using the PAM dataset: (1) leavefixed -sensors-out , which drops a fixed set of sensors across all

fixed

-sensors-out

random

-sensors-out

samples and compared methods, and (2) leaverandom -sensors-out , which drops sensors randomly. It is important to note that only the observations in the validation and test sets were dropped, while the training set was kept unchanged. To ensure a fair comparison, we dropped the same set of sensors in the leavefixed -sensors-out setting as in [49].

The results are presented in Fig. 3, from which we observe that our approach consistently achieves the best performance and outperforms all the specialized baselines by a large margin. With the missing ratio increasing from 10% to 50%, our approach maintains strong performance, staying above 80%. In contrast, the comparison baseline shows a marked drop. The advantage of ViTST over the comparison baselines becomes increasingly significant. Even when half of the variables were dropped, our approach was still able to achieve acceptable performance over 80, surpassing the best-performing baseline Raindrop by 33.1% in Accuracy, 40.9% in Precision, 39.4% in Recall, and 42.8% in F1 score in the leavefixed -sensors-out setting (all in absolute points). We also notice that the variances in our results are notably lower compared to the baselines. These results suggest that our approach is highly robust to varying degrees of missing observations in time series.

4.3 Additional Analysis

(a) P19

(b) P12

(c) PAM

90

100

90

ResNet

ResNet

II

工I

工

工

工I

ViT

80

ViT

80

70

Swin-scratch

70

Swin-scratch

80

ResNet

60

60

60

ViT

50

50

Swin-scratch

40

40

40

30

30

20

20

AUROC

AUPRC

AUROC

AUPRC

AccuracyPrecision

Recall

Where does the performance come from? Our approach transforms time series into line graph images, allowing the use of vision transformers for time series modeling. We hypothesize that vision transformers can leverage their general-purpose image recognition ability acquired from large-scale pre-training on natural images (such as ImageNet [30]) to capture informative patterns in the line graph images. To validate that, we compared the performance of a pre-trained Swin

Transformer with a Swin Transformer trained from scratch, as shown in Fig. 4. The significant drop in performance without pre-training provides evidence that Swin transformer could transfer the knowledge obtained from pre-training on natural images to our synthetic time series line graph images, achieving impressive performance. Nevertheless, the underlying mechanism needs further exploring and probing in future studies.

How do different vision models perform? We benchmarked several backbone vision models within our framework. Specifically, we tried another popular pre-trained vision transformer ViT 3 and a pre-trained CNN-based model, ResNet 4 . The results are presented in Fig. 4. The pre-trained transformer-based ViT and Swin Transformer demonstrate comparable performance, both outperforming the previous state-of-the-art method, Raindrop. In contrast, the pre-trained CNN-based ResNet lagged considerably behind the vision transformer models. This performance gap is consistent with observations in image classification tasks on datasets like ImageNet, where vision transformers have been shown to excel in preserving spatial information compared to conventional CNN models. This advantage enables vision transformers to effectively capture the positions of patches within each line graph sub-image and the entire image and facilitates the modeling of complex dynamics and relationships between variables.

How to create time series line graph images? Using line graphs to visualize time series offers us an intuitive way to interpret the data and adjust the visualization strategy for enhanced clarity and potentially augment the performance. To offer insights on effective time series-to-image transformation, we analyze the effects of several key elements in practice: (1) the default linear interpolation to connect partially observed data points on the line graphs; (2) markers to indicate observed data points; (3) variable-specific colors to differentiate between line graphs representing different variables; (4) the order determined by missing ratios when organizing multiple line graphs in a single image.

The results are presented in Table 3. Given the balanced missing ratios in the PAM dataset, we excluded results without the sorting order. Interestingly, plotting only observed data points without linear interpolation led to better results on P19 and P12 datasets. This could be attributed to the potential inaccuracies introduced by interpolation, blurring distinctions between observed and interpolated points. Additionally, omitting markers complicates the model's task of discerning observed data points from interpolated ones, degrading its performance. The absence of distinctive colors led to the most significant performance drop, underlining the necessity of using varied hues for individual line graphs to help the model to distinguish them. While a specific sorting order may not ensure optimal outcomes across all datasets, it does provide relatively stable results over multiple datasets and evaluation metrics. For the PAM dataset, these nuances seem to have minimal impact, indicating the robustness of our approach against these variations in some scenarios.

Effects of grid layouts and image sizes. We explored the influence of grid layouts and image dimensions on our approach's efficacy. For a fair comparison across grid layouts, we fixed the size of each grid cell as 64 × 64 and altered the grid layouts. As shown in Figure 5. we observed our approach's robustness to variations in grid layouts, with square layout consistently yielding good results across different datasets and metrics, which was particularly evident for the P12 dataset. Regarding image size, when we maintained the grid layout but reduced the overall image dimensions, a noticeable performance decline was observed on the P12 and PAM datasets, which complies with our intuition.

Robustness against varied plotting parameters. To gauge the robustness of our approach against different plotting parameters, we assessed aspects including line style/width and marker style/size,

(a) P19

(b) P12

(c) PAM

90

100

90

4x9 (256x576)

4x9 (256x576)

2x9 (128x576)

4x5 (256x320)

80

5x7(320x448)

6x6(384x384)

6x6 (224x224)

80

6x6 (224x224)

5x8(320x512)

6x6(384x384)

98

3x6(192x384)

4x5(224×224)

70

96

70

60

94

60

50

92

50

40

90

AUROC

AUPRC

AUROC

AUPRC

AccuracyPrecision

Recall

primarily using the P19 dataset. As shown in Table 4, our approach demonstrates robustness against changes in these parameters, maintaining strong performance across different plotting configurations.

What does ViTST capture? To gain insights into the patterns captured by ViTST in the time series line graph images, we analyzed the averaged attention map of a ViTST model with ViT as the backbone, as depicted in Fig. 6. The attention map reveals that the model consistently attends to the informative part, i.e., line graph contours within the image. Furthermore, we observed that the model appropriately focuses on observed data points and areas where the slopes of the lines change. Conversely, flat line graphs that lack dynamic patterns seem to receive less attention. This demonstrates that ViTST might be able to discern between informative and uninformative features in the line graph images, enabling it to extract meaningful patterns.

Attention                              Input

P19                                        P12                                               PAM

A

4.4 Regular Time Series Classification

An advantage of our approach is its ability to model time series of diverse shapes and scales, whether they are regular or irregular. To evaluate the performance of our approach on regular time series data,

we conducted experiments on ten representative multivariate time series datasets from the UEA Time Series Classification Archive [2]. These datasets exhibit diverse characteristics, as summarized in Table 5. It is worth noting that the PS dataset in our evaluation contains an exceptionally high number of variables (963), while the EW dataset has extremely long time series (17984). We specifically selected these two datasets to assess the effectiveness of our approach in handling large numbers of variables and long time series. We follow [47] to use these baselines for comparison: DTW D which stands for dimension-Dependent DTW combined with dilation-CNN [11], LSTM [13], XGBoost [5], Rocket [7], and a transformer-based TST [47] which operates on fully observed numerical time series.

The performance of our approach on regular time series datasets is consistently strong, as demonstrated in Table 5. With an average accuracy that is second-best and closely aligned with the top-performing baseline method TST, our approach showcases its competitive capabilities. Notably, it excels on challenging datasets PS and EW with massive variables and observation length. These results were achieved using the same image resolution ( 384 × 384 ) as the other datasets, indicating the effectiveness and efficiency of our approach. The ability of our approach to handle both irregular and regular time series data further emphasizes its versatility and broad applicability.

5 Conclusion

In this paper, we introduce a novel perspective on modeling irregularly sampled time series. By transforming time series data into line graph images, we could effectively leverage the strengths of pre-trained vision transformers. This approach is straightforward yet effective and versatile, enabling the modeling of time series of diverse characteristics, regardless of irregularity, different structures, and scales. Through extensive experiments, we demonstrate that our approach surpasses state-of-the-art methods designed for irregular time series and maintains strong robustness to varying degrees of missing observations. Additionally, our approach achieves promising results on regular time series data. We envision its potential as a general-purpose framework for various time series tasks. Our results underscore the potential of adapting rapidly advancing computer vision techniques to time series modeling.

6 Limitations and Future Work

In this work, we utilized a straightforward method to image multivariate time series by converting them into line graph images using matplotlib and then saving them as RGB images. While our results are promising and exhibit robustness against variations in the time series-to-image transformation process, there may be alternative ways to visualize the data. This includes potentially more controllable and accurate plotting methods or different image representations beyond line graphs. Our findings also highlight the efficacy of pre-trained vision transformers for time series classification, suggesting that these models might leverage knowledge acquired from pre-training on natural images. Yet, the underlying reasons for their remarkable success still need deeper exploration and investigation. This research serves as a promising starting point in this domain, suggesting various potential directions. We leave these further explorations and investigations for future work.

References

Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. ArXiv , abs/1607.06450, 2016.

Bagnall, A., Dau, H. A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., and Keogh, E. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

Che, Z., Purushotham, S., Cho, K., Sontag, D., and Liu, Y. Recurrent neural networks for multivariate time series with missing values. Scientific reports , 8(1):1-12, 2018.

Chefer, H., Gur, S., and Wolf, L. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 782-791, 2021.

Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , pp. 785-794, 2016.

Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Dempster, A., Petitjean, F., and Webb, G. I. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery , 34 (5):1454-1495, 2020.

DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Fan, Q., Chen, C.-F., and Panda, R. Can an image classifier suffice for action recognition? In International Conference on Learning Representations , 2021.

Franceschi, J.-Y., Dieuleveut, A., and Jaggi, M. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems , 32, 2019.

Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. E. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation , 101(23): e215-e220, 2000.

Graves, A. Long short-term memory. Supervised sequence labelling with recurrent neural networks , pp. 37-45, 2012.

Hatami, N., Gavet, Y., and Debayle, J. Classification of time-series images using deep convolutional neural networks. In Tenth international conference on machine vision (ICMV 2017) , volume 10696, pp. 242-249. SPIE, 2018.

He, K., Chen, X., Xie, S., Li, Y ., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 16000-16009, June 2022.

Horn, M., Moor, M., Bock, C., Rieck, B., and Borgwardt, K. Set functions for time series. In International Conference on Machine Learning , pp. 4353-4363. PMLR, 2020.

Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , pp. 1207-1216, Stanford, CA, 2000. Morgan Kaufmann.

Lea, C., Flynn, M. D., Vidal, R., Reiter, A., and Hager, G. D. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 156-165, 2017.

Li, S., Jin, X., Xuan, Y ., Zhou, X., Chen, W., Wang, Y.-X., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems , 32, 2019.

Lipton, Z. C., Kale, D., and Wetzel, R. Directly modeling missing data in sequences with rnns: Improved classification of clinical time series. In Machine learning for healthcare conference , pp. 253-270. PMLR, 2016.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 10012-10022, 2021.

Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y ., Ning, J., Cao, Y ., Zhang, Z., Dong, L., et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 12009-12019, 2022.

Marlin, B. M., Kale, D. C., Khemani, R. G., and Wetzel, R. C. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In Proceedings of the 2nd ACM SIGHIT international health informatics symposium , pp. 389-398, 2012.

Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 , 2022.

OpenAI. Gpt-4 technical report, 2023.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pp. 8748-8763. PMLR, 2021.

Reiss, A. and Stricker, D. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers , pp. 108-109. IEEE, 2012.

Reyna, M. A., Josef, C., Seyedi, S., Jeter, R., Shashikumar, S. P., Westover, M. B., Sharma, A., Nemati, S., and Clifford, G. D. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. In 2019 Computing in Cardiology (CinC) , pp. Page-1. IEEE, 2019.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Semenoglou, A.-A., Spiliotis, E., and Assimakopoulos, V. Image-based time series forecasting: A deep convolutional neural network approach. Neural Networks , 157:39-53, 2023.

Shorten, C. and Khoshgoftaar, T. M. A survey on image data augmentation for deep learning. Journal of big data , 6(1):1-48, 2019.

Shukla, S. N. and Marlin, B. Interpolation-prediction networks for irregularly sampled time series. In International Conference on Learning Representations , 2018.

Shukla, S. N. and Marlin, B. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations , 2020.

Sood, S., Zeng, Z., Cohen, N., Balch, T., and Veloso, M. Visual time series forecasting: an image-driven approach. In Proceedings of the Second ACM International Conference on AI in Finance , pp. 1-9, 2021.

Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training dataefficient image transformers & distillation through attention. In International Conference on Machine Learning , pp. 10347-10357. PMLR, 2021.

Tripathy, R. and Acharya, U. R. Use of features from rr-time series and eeg signals for automated classification of sleep stages in deep neural network framework. Biocybernetics and Biomedical Engineering , 38(4):890-902, 2018.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Wang, Z. and Oates, T. Imaging time-series to improve classification and imputation. In Twenty-Fourth International Joint Conference on Artificial Intelligence , 2015.

Wang, Z. and Oates, T. Spatially encoding temporal correlations to classify temporal data using convolutional neural networks. arXiv preprint arXiv:1509.07481 , 2015.

Wu, H., Xu, J., Wang, J., and Long, M. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021.

Wu, Y., Ni, J., Cheng, W., Zong, B., Song, D., Chen, Z., Liu, Y., Zhang, X., Chen, H., and Davidson, S. B. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pp. 651-659, 2021.

Wu, Z., Pan, S., Long, G., Jiang, J., Chang, X., and Zhang, C. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pp. 753-763, 2020.

Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., and Hu, H. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9653-9663, 2022.

Xu, J., Wu, H., Wang, J., and Long, M. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:2110.02642 , 2021.

Yoon, J., Zame, W. R., and van der Schaar, M. Multi-directional recurrent neural networks: A novel method for estimating missing data. In Time series workshop in international conference on machine learning , 2017.

Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., and Eickhoff, C. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pp. 2114-2124, 2021.

Zhang, X., LI, S., Chen, Z., Yan, X., and Petzold, L. Improving medical predictions by irregular multimodal electronic health records modeling. ArXiv , abs/2210.12156, 2022.

Zhang, X., Zeman, M., Tsiligkaridis, T., and Zitnik, M. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Representations , 2022.

Zhang, Y. Attain: Attention-based time-aware lstm networks for disease progression modeling. In In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI2019), pp. 4369-4375, Macao, China. , 2019.

Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pp. 11106-11115, 2021.

Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. arXiv preprint arXiv:2201.12740 , 2022.

A More Details on Time Series Line Graph Image Creation

Implementation The time-series-to-image transformation can be implemented using the Matplotlib package 5 with the following few lines of code.

Axis Limits of Line Graphs The axis limits determine the plot area of the line graphs and the range of displayed timestamps and values. By default, we set the limits of the x-axis and y-axis as the ranges of all the observed timestamps and values. However, we found that some extreme observed values for some variables can largely expand the range of the y-axis, causing most plotted points of observations to cluster in a small area and resulting in flat line graphs. Common normalization and standardization methods will not solve this issue, as the relative magnitudes remain unchanged in the created images. We thus tried the following strategies to remove extreme values and narrow the range of the y-axis in our preliminary experiments:

Interquartile Range (IQR): IQR is one of the most extensively used methods for outlier detection and removal. The interquartile range is calculated based on the first and third quartiles of all the observed values of each variable in the dataset and then used to calculate the upper and lower limits.

Standard Deviation (SD): The upper and lower boundaries are calculated by taking 3 standard deviations from the mean of observed values for each variable across the dataset. This method usually assumes the data is normally distributed.

Modified Z-score (MZ): A z-score measures how many standard deviations away a value is from the mean and is similar to the standard deviation method to detect outliers. However, z-scores can be influenced by extreme values, which modified z-scores can better handle. We set the upper and lower limits as the values whose modified z-scores are 3.5 and -3.5.

We show examples of the created images with these strategies in Figure 7.

The performance comparison of models trained on images created with different strategies is shown in Table 6. We observe that the methods that remove extreme values hurt the performance, except for SD on the PAM dataset. Although these methods narrow the value range and highlight the dynamic patterns of line graphs, they discard the extreme values that might be informative themselves. Therefore, we stick with the default way that sets the axis limits as the range of all observed values.

Computation cost We list the inference time (in seconds) of different methods on the test sets of three datasets in Table 7. All the inferences are made in a single Nvidia A6000 GPU. It is observed that our vision-based method consumes more inference time than the non-vision baselines. However, we believe this cost remains within an acceptable range in the context of today's ML practice and medical applications, considering the inference on each sample only costs around 0.01 seconds.

Default

IQR

Standard Deviation

Modified Z-score

(a) P19

(b) P12

(c) PAM

<

h

M

B More Experimental Details

B.1 Datasets

We used the datasets processed by [49], whose details are given below.

P19: PhysioNet Sepsis Early Prediction Challenge 2019. 6 The P19 dataset [29] consists of clinical data for 38,803 patients, and aims to predict whether sepsis will occur within the next 6 hours. The dataset includes 34 irregularly sampled sensors with 8 vital signs and 26 laboratory values for each patient, as well as 6 demographic features. To process the static features, we use templates outlined in Table 8, and utilize a pre-trained Roberta-base model to extract textual features. These textual features are then combined with visual features obtained from the vision transformer to perform binary classification. The dataset is highly imbalanced with only 4% of samples being positive, and has a missing ratio of 94.9%.

P12: PhysioNet Mortality Prediction Challenge 2012. 7 P12 dataset [12] includes clinical data from 11,988 ICU patients, with 36 irregularly sampled sensor observations and 6 static demographic features provided for each patient. The goal is to predict patient mortality, which is a binary classification task. The dataset is highly imbalanced, with around 86% of samples being negative. The missing ratio of the dataset is 88.4%.

PAM: PAMAP2 Physical Activity Monitoring. 8 The PAM dataset originally contains data of 18 physical activities with 9 subjects wearing 3 inertial measurement units. However, to make it suitable for irregular time series classification, [49] excluded the ninth subject due to its short length of sensor readouts, and 10 out of the 18 activities that had less than 500 samples were also excluded. As a result, the task is an 8-way classification with 5,333 samples, each with 600 continuous observations. To simulate the irregular time series setting, 60% of the observations are randomly removed. No static features are provided, and the 8 categories are approximately balanced. The missing ratio is 60.0%.

B.2 Experiments on Static Features

Time series data is often associated with information from other modalities, such as the textual clinical notes in electronic health records (EHRs) in the healthcare domain. Our approach is naturally suitable for incorporating such information since we convert time series data to images, and thus various vision-language and multi-modal techniques can be utilized to incorporate the visual (time series) information and information from other modalities. For example, the CLIP [27] learns a shared

hidden feature space where the paired image and text stay close. Under our framework, such a shared space can also be learned for the paired visual time series images and textual clinical notes, which is our future direction. It also paves the way for the application of multi-modal models such as GPT4 [26] to handle the visualized time series data and the clinical notes simultaneously. In our current experiments, we used a text encoder, Roberta-base, to encode textual demographic information in the P19 and P12 datasets. We also experimented with normalizing the original categorical features and encoding them using an MLP as in previous work, and compare with the strong baseline, Raindrop. The results are shown in Table 9. We observe that even without using static features, our method has already outperformed Raindrop. In addition, utilizing Roberta to encode and incorporate the textual feature is more effective than applying MLP over categorical features.

B.3 Experiments on Image Fusion

In our preliminary experiments, we examined two distinct approaches to fusing the line graph subimages in each multivariate time series data. First, we processed each sub-image independently to learn the patch/image representations and then concatenated their respective patch representations to input into the final prediction layer. In contrast, our default method aggregates all sub-images into a single image to learn the patch embeddings. The key distinction between these strategies is whether a patch can attend to those from other sub-images and how position embedding factors in during the representation learning phase. Aside from this, other parameters were consistent across both methods. We tested on ViT and the performance comparisons with these two settings are shown in Table 10. It is evidenced that attending to patches from other sub-images offers advantages. This likely allows for capturing cross-variable correlations at a granular level within the self-attention layers, as opposed to only at the final linear prediction layer.

B.4 Experiment on Regular Time Series

We selected ten representative multivariate time series datasets from the UEA Time Series Classification Archive [2] with diverse characteristics, including the number of classes, variables, and time series length. The datasets we chose are EthanolConcentration (EC), Handwriting (HW), UWaveGestureLibrary (UW), SelfRegulationSCP1 (SCP1), SelfRegulationSCP2 (SCP2), JapaneseVowels (JV), SpokenArabicDigits (SAD), Heartbeat (HB), FaceDetection (FD), PEMS-SF (PS), and EigenWorms (EW). Notably, the PS dataset has an exceptionally high number of variables (963), while the EW dataset has extremely long time series (17984). These two datasets allow us to assess the effectiveness of our approach when dealing with large numbers of variables and long time series. We applied different image sizes according to the grid layouts for these datasets. The hyperparameter settings are

provided in Table 11, and we applied cutout [8] data augmentation methods to SCP1, SCP2, and JV datasets due to the small size of their training sets.

B.5 Self-supervised Learning

We preliminary explored masked image modeling self-supervised pre-training on the time series line graph images. We randomly mask columns of patches with a width of 32 on each line graph within a grid cell. The masking ratio is set as 50%. We finetuned the Swin Transformer model for 10 epochs with batch size 48. The learning rate is 2e-5. Following [44], we use a linear layer to reconstruct the pixel values and employ an ℓ 1 loss on the masked pixels:

where p M and ˆ p M are the masked and reconstructed pixels, respectively; Ω( · ) denotes the number of elements. With self-supervised masked image modeling, the performance improves by 1.0 in AUPRC points (percentage) from 52.8 ( ± 3.8) to 53.8 ( ± 3.2). The AUROC points (percentage) slightly dropped from 89.4 ( ± 1.9) to 88.9 ( ± 2.1).

B.6 Full Experimental Results

We presented the full experimental results in the leave-sensors-out settings in Table 12.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores a very interesting direction to represent time series as plotted images and then stack vision transformers to perform representation learning. Following this idea, this paper has conducted empirical studies on some classification benchmarks of both irregularly sampled time series and regularly sampled ones. By leveraging this new data representation, the proposed method has obtained the state-of-the-art performance on three irregular time-series benchmarks and also produced competitive results on multiple regular time-series dataests.\n\nSTRENGTHS:\nIn my view, the biggest strength of this paper is the first successful demonstration of applying image-driven methods on representation learning of irregularly sampled time series. While after reading the related work section, I have also learned that this paper is not the first to represent time series as real images in general. Neverthess, I think the successful demonstration of image-based time-series representation learning on time-series classification tasks is interesting and has sufficient novelty.\n\nWEAKNESSES:\nHowever, I do have several concerns on the experimental results at the current stage.\n\n- Given the relatively high performance variations when employing different plotting configurations and specific early stopping epochs on different datasets, I would like to know the detailed hyper-parameter tuning procedure when you develop ViTST. Are you tuning these options directly based on the feedback from the test set? If not, can you elaborate on how to divide the validation part and please report the details of performance on both validation and test sets if possible.\n- This paper includes some ablation tests, but I think the existing results are far from enough to convince me about the robustness of ViTST. For example, as table 3 shows, this method is sensitive to the existences of interpolation, markers, colors. I want to know that whether ViTST is robust to 1) different line styles and widths given the existence of interpolation, 2) different marker sizes and types given the existence of markers, and 3) different color permutations assigned to multivariate time series. My central confusions lie in which aspects play indispenable roles in making ViTST work and whether ViTST is robust to those irrelevant options. Moreover, I would like to see more ablation results about grid layouts and image sizes. Tables 2, 3, 4 only cover a few, and there seem to large performance variations, too.\n- Another question bothering me is that why ViTST can obtain the best performance for irregularly sampled time series but is only roughly comparable to existing solutions for regular time series.\n- I have tried the code but cannot reproduce the results.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nThe major limitation is that we are not clear many other critical factors ensuring the success of ViTST and whether it is robust to different plotting setups.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the use of pre-trained Vision Transformers (ViT) for both regularly and irregularly sampled time series classification. Two primary aspects are discussed: Transforming time series into images and performing time series classification using pre-trained ViT. Through a series of experiments, the authors illustrate the feasibility of this research direction.\n\nSTRENGTHS:\nThe paper is well-organized and presents an engaging perspective on time series classification, particularly focusing on the potential of ViT in relevant tasks. The authors conducted extensive empirical evaluations to showcase the effectiveness of the proposed framework, which is a strong aspect of the work.\n\nWEAKNESSES:\n+ The paper falls short in providing an in-depth analysis of several critical aspects, such as the advantages of using ViT over other foundational models for time series, factors contributing to successful cross-domain transfer when employing ViT for time series classification, and how key design factors impact performance. See the listed questions for details.\n+ In addition, some crucial technical details are missing. For example, it is unclear which layers in ViT are frozen during downstream fine-tuning. Justification for the use of line graphs over other visualization strategies (e.g., frequency maps) is not adequately discussed in Section 3.1. The model's efficiency compared to specialized models in time series modeling is also not clear.\n+ There are some writing issues. For example, the title should be “…for Irregularly Sampled Time Series Classification” unless the effectiveness of ViTST is evident on other time series tasks such as forecasting and imputation. Besides, it is unknown whether the backbone ViT model are frozen in Fig. 2.\n\nQUESTIONS:\n+ What distinguishes the use of ViT for time series modeling from other foundational models like language or acoustic models (e.g., Voice2Series)? Are there unique advantages, beyond performance, to using ViT?\n+ What theoretical factors contribute to such a successful cross-domain transfer? According to the experiments, some factors like line colors may significantly affect the model performance. A follow-up question is how to claim the robustness of the proposed framework in general time series classification when changing line shapes, colors, or even time series datasets to other domains other than healthcare.\n+ What types of interpolation are used in the experiments?\n+ What is the motivation behind Section 4.5? Is it for a second pre-training based on pre-trained ViT or is it for pre-training from scratch?\n+ In the transformation of time series to images, why do the authors opt to combine different univariate time series into a single image, as opposed to potential alternative implementations like introducing an additional layer to fuse different input images before feeding them into ViT?\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations and potential negative societal impacts of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on learning from irregularly sampled time series data. The paper presents a simple approach that converts irregularly sampled time series into an image where different input channels are line graphs. The converted images are then modeled using a standard Transformer model. Experiments show that the proposed approach outperforms the SOTA approaches on several datasets.\n\nSTRENGTHS:\n- The paper focuses on the task of learning from irregularly sampled data which is important in many domains.\n- Experimental results show the effectiveness of the approach when compared to baselines and other recent approaches.\n- The paper is clear and well written.\n\nWEAKNESSES:\n- The irregularly sampled time series data are represented by a line graph where observed points are connected using a straight line which is a very adhoc way to deal with missing values and irregularity.\n- The novelty is marginal as the proposed approach mainly uses a standard Transformer model where time series data is fed as images.\n- The standard deviation of results in Table 2 seems very high. It is not immediately clear if the proposed approach achieves statistically significant performance or it's just noise.\n- The proposed approaches uses static features present with the data. It is not immediately clear how these static features were used with the baseline approaches. \n- The paper notes that all the methods were trained for 20 epochs. Did they all converge in 20 epochs? The baseline approach should be trained till convergence. \n- The paper is missing experiments on MIMIC-III dataset, most commonly used dataset in this space.\n- The paper is missing comparisons with recent ODE based approaches which achieve SOTA performance. \n- Does the size of image change if the sequence length changes?\n\nQUESTIONS:\nI have mentioned my concerns in the weaknesses section.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a surprisingly simple and effective approach for applying computer vision Transformer models to time series classification. Multivariate time series inputs are plotted in a grid to produce images that are used to fine-tune pretrained vision Transformer models. This approach achieves state-of-the-art results on irregularly sampled datasets and competitive results on regularly sampled datasets. Aside from the empirical efficacy, the results are relevant to understanding the capabilities of Transformers and the effectiveness of cross-domain pretraining.\n\nSTRENGTHS:\nI think the basic finding of the paper - that pre-trained vision Transformers are effective at performing time series classification - is original and significant. It raises very interesting questions about the general efficacy of pre-trained Transformers and the relationship between time series and visual data that could inform future work. In a practical sense, I have some concerns detailed below but I generally appreciate the simplicity and efficacy of the proposed approach.\n\nThe basic results are convincing and the ablation results are extensive and informative. Most of the questions I would have had about the model are already addressed in the main paper or Appendix.\n\nThe paper was well-written and clear. Results are reproducible since full code was provided.\n\nWEAKNESSES:\nTo use the proposed method requires making a number of somewhat arbitrary decisions on how to convert time series to images: what layout to use, what order to put the time series in, what colours to use, line width/style, etc. The results in Table 3 and the Appendix show that decisions like these can affect performance even though they don't strictly change the information contained in the input. While these factors can be empirically tuned, the lack of robustness to seemingly arbitrary choices is concerning.\n\nIn particular, the use of matplotlib to generate inputs makes the input construction somewhat poorly controlled. While I appreciate the simplicity of using an existing library, this leaves many implementation details about how exactly plots are rendered up to the library. Again, given that purely visual variations can affect model performance, it's not clear how important these implementation details are. I would have preferred to see a more precise approach that formally determined what numeric value is assigned to each pixel, making these details apparent.\n\nThe range of irregularly sampled time series datasets is somewhat small to draw general conclusions from, but this is in line with existing work in this area.\n\nQUESTIONS:\nHow were line colours selected? If random, what distribution of colours was used? Is there any justification for the specific choice made?\n\nFor your self-supervised learning experiments, did you start with a pretrained model?\n\nLIMITATIONS:\nLimitations are not explicitly discussed. No societal impact concerns come to mind, but I think technical limitations should have been addressed. These could include comparing training and inference resource requirements to other models and acknowledging points like those I raised in the Weaknesses section if the authors agree they are limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

comments: SUMMARY:
For the classification task based on irregular time series data, the authors introduced Vision Time Series Transformer (ViTST) approach where irregular time series data is displayed as line graph then fed to pretrained transformer type models. The authors tested this approach using several time series data from medical domain and human activity domain, and the result shows confident advantages of the proposed approach.

STRENGTHS:
Simple idea but very interesting. Could contribute to many domains dealing with time series data.

WEAKNESSES:
Time series data is everywhere but the authors tested only a few dataset from medical domain and human activity domain. Thus "any shape" in line 357 sounds too strong at this time.

QUESTIONS:
Have you tested multi-class classification tasks and/or regression tasks?

LIMITATIONS:
Since this approach could have a potential to expand widely, it would be better to state "Limitation" of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to model irregularly sampled time series. The method is based on transforming numerical time series data to line graphs and then applying pretrained vision transformers to that data. For multivariate datasets, every variable is plotted separately, and plots are aggregated in a grid. The authors provide exhaustive evaluation on different datasets, including leaving-sensors-out settings and regularly sampled time series data.\n\nSTRENGTHS:\nThe paper is well written and easy to understand. The method proposed by the authors is extremely simple, yet surprisingly effective on many time series modeling tasks, as shown by exhaustive experimental evaluation.\n\nWEAKNESSES:\n-\tI don’t see the authors claims made in the abstract and introduction backed by their empirical results. It is difficult to judge, whether their proposed method indeed outperforms previous approaches “significantly” (L. 10, L. 43, L. 67) as such an analysis is missing. On most of the metrics for the P19 and P12 data at least, improvements do not look significant to me. On the PAM dataset results are likely significant (analysis is also missing here).\n-\tThe PAM dataset (the only one where we likely actually see a significant improvement over Raindrop) is also the one with the fewest number of variables (17 vs 34 & 36 for P19 and P12, respectively). I assume that the proposed method performs much worse for datasets with many variables, as also shown in the authors experiments on the FD, and PS datasets, where ViTST performs indeed much worse than other methods. \n-\tThe authors do not comment on the computational cost of their method, which is likely much higher than for most non-vision based approaches.\n-\tRecently, Semenoglou et al. proposed a very similar method, albeit limited to single-variable time series. They plot a univariate time series and feed this plot into a CNN to predict future observations of the same series. While I think that the present work improves on the work by Semenoglou et al. in several regards (multivariate, pretrained ViT outperforming pretrained ResNet, …), the authors should include a reference to Semenoglou et al. in their related work. Additionally, I don't think statements such as \"This paper studies the problem from a whole new perspective by transforming irregularly sampled time series into line graph images and leveraging powerful vision transformers for time series classification in the same way as image classification.\" (L. 4-7) are valid, given that the idea to transform time series to images and then applying image classification networks is not novel.\n-\tThe authors do not comment on limitations of their method.\n\nQUESTIONS:\n-\tFor the PS dataset, the authors must have used much smaller (I assume 12x12) sizes for their plots, am I correct?\n-\tWhat is the computational cost of the proposed method?\n-\tDid the authors conduct experiments on whether the order of the plots in the grid matters? \n-\tBased on the ablation studies provided by the authors it seems likely that most of the performance comes from pretraining of the transformer models. Wouldn’t a large pretrained transformer for numerical time series modeling be likely to perform similar or better than the method proposed by the authors? If the authors agree, then I would like to see a brief discussion of this in an updated version of the manuscript.\n\nLIMITATIONS:\nUnfortunately, the authors do not comment on limitations of their work. I see the following possible limitations and would like to see an open discussion of those in an updated version of the manuscript:\n-\tAbility to handle extremely large number of variables\n-\tComputational costs of the method", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}